580 lines
26 KiB
Markdown
580 lines
26 KiB
Markdown
|
|
# N=3 sim-test battery — behavioral specification
|
|||
|
|
|
|||
|
|
Companion to `N3_POSTMORTEM_2026-05-25.md`, `N3_DATA_GAPS.md`,
|
|||
|
|
`N3_DEPLOYMENT_REPORT.md`, `SIM_HARDENING_SPEC.md`, and the simulator's
|
|||
|
|
`SIM_SPEC.md`. This document is the contract for a separate coding agent
|
|||
|
|
that will land a battery of simulator tests covering the general failure
|
|||
|
|
shapes the latest deployment exposed.
|
|||
|
|
|
|||
|
|
This is a *behavioral* spec. It names the failure shapes, the contracts
|
|||
|
|
each test must establish, and the verdicts each must produce. It does
|
|||
|
|
not prescribe file layout, TOML field values, or internal helper code.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 0. Motivation and framing
|
|||
|
|
|
|||
|
|
The 2026-05-25 deployment surfaced one new failure shape (`stage-2`'s
|
|||
|
|
relay-mediated path died at ~5 s and never recovered, while its tunnel
|
|||
|
|
to the relay apparently survived) layered on top of failure shapes
|
|||
|
|
prior deploys also exhibited (silent-worker subprocess, gossip-only
|
|||
|
|
membership view, asymmetric host reachability, bundle-recovery only
|
|||
|
|
via staging-file scrape). Together these are the **general** failure
|
|||
|
|
cases the battery must cover — not one scenario per postmortem, but a
|
|||
|
|
*family* per shape, as `SIM_HARDENING_SPEC.md §5` requires.
|
|||
|
|
|
|||
|
|
The simulator has now landed every observability and sim-cross-
|
|||
|
|
pollination contract those postmortems demanded (`F1`–`F3`, `S-A1`
|
|||
|
|
through `S-E2`; see `.loop/verdict.md`). The pieces needed to express
|
|||
|
|
these scenarios all exist: `MutationKind::RelayPeerConnDown`, the
|
|||
|
|
`stage` host kind with `WorkerExit`, the `relay` vertex with policy
|
|||
|
|
mutations, and the §10.1 assertion catalog. **The battery is the
|
|||
|
|
exercise of those pieces against the latest deployment's known shapes,
|
|||
|
|
expressed end-to-end through scenario files and verdicts — not new
|
|||
|
|
sim machinery.**
|
|||
|
|
|
|||
|
|
Why a *battery* rather than one test per shape: the
|
|||
|
|
`SIM_HARDENING_SPEC §5` family rule. A fix that resolves the
|
|||
|
|
2026-05-25 incident's specific timing (relay-peer-down at +5 s) but
|
|||
|
|
regresses a sibling instance of the family (relay-peer-down at +30 s,
|
|||
|
|
or during partition heal, or on only the inbound leg) is a regression
|
|||
|
|
the battery must catch.
|
|||
|
|
|
|||
|
|
A diagnostic deployment is running concurrently to gather data we
|
|||
|
|
don't yet have for the silent-worker class. This spec is written
|
|||
|
|
against the evidence already in the bundle from 2026-05-25; the
|
|||
|
|
implementing agent should not block on that deploy's results. When
|
|||
|
|
results land they will sharpen the parameters of family **B**
|
|||
|
|
(silent-worker) but will not change the shape of the battery.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 1. Cross-cutting requirements
|
|||
|
|
|
|||
|
|
These hold for every family in §3.
|
|||
|
|
|
|||
|
|
### 1.1 No white-box / structural tests
|
|||
|
|
|
|||
|
|
A test in the battery passes or fails based on the *bundle* the
|
|||
|
|
scenario produces and the verdicts the §10.1 assertion catalog
|
|||
|
|
returns against that bundle. No test reads simulator internals, no
|
|||
|
|
test inserts a value via one API path and reads it back via another,
|
|||
|
|
no test asserts that an internal Rust struct has a particular field
|
|||
|
|
shape. A test that would survive a refactor of the engine, the
|
|||
|
|
network, or any host kind, but fail when the *deployment-relevant
|
|||
|
|
behavior* drifts, is a test that belongs.
|
|||
|
|
|
|||
|
|
Litmus test: if removing the assertion would change the bundle's
|
|||
|
|
prose summary in a way a deployment investigator would notice, the
|
|||
|
|
assertion belongs. If removing it would not, the assertion is
|
|||
|
|
echoing internals and does not belong.
|
|||
|
|
|
|||
|
|
### 1.2 Test taxonomy and priority
|
|||
|
|
|
|||
|
|
Each family ships at least one **scenario test** (story-shape:
|
|||
|
|
declared scenario + declared assertion + declared expected verdict)
|
|||
|
|
and where the parameter space is large, at least one **property
|
|||
|
|
test** (a parameterized scenario whose `seed` ranges over the §1.3
|
|||
|
|
family axes). Scenario tests are mandatory; property tests are
|
|||
|
|
required only where §3 names them.
|
|||
|
|
|
|||
|
|
A small number of **contract tests** sit alongside the families: they
|
|||
|
|
assert that the bundle's event schema matches the production
|
|||
|
|
diagnostics schema for the event kinds the battery exercises (the
|
|||
|
|
`SubprocessSpawned`/`SubprocessExited`, `RelaySessionStateChanged`,
|
|||
|
|
`GossipReceived`, and `Tier2RelaySession` shapes the observability
|
|||
|
|
upgrade landed). The contract tests are not per-family; they live
|
|||
|
|
once and protect every family from sim/prod drift.
|
|||
|
|
|
|||
|
|
### 1.3 Family-based, not single-seed
|
|||
|
|
|
|||
|
|
Every family in §3 declares its **mutation axes** — the dimensions
|
|||
|
|
along which the postmortem's parameters are "plausibly variable in
|
|||
|
|
the wild" per `SIM_HARDENING_SPEC §5`. The family's scenario tests
|
|||
|
|
cover the central case (the specific incident's parameters) and the
|
|||
|
|
named extreme cases (e.g., "session closes at +1 s" and "session
|
|||
|
|
closes at +5 min" for family A). The family's property test ranges
|
|||
|
|
over the axes within their declared bounds.
|
|||
|
|
|
|||
|
|
### 1.4 Deterministic replay
|
|||
|
|
|
|||
|
|
Every scenario test's `(scenario, seed)` is recorded in the test
|
|||
|
|
itself; running the test produces a byte-identical bundle to any
|
|||
|
|
previous run on any supported architecture. A property-test failure
|
|||
|
|
prints the seed; running the scenario with that seed reproduces the
|
|||
|
|
failure. This is mechanical — the simulator already guarantees it
|
|||
|
|
(`SIM_SPEC.md §7`); the battery must not undo it. No test reads any
|
|||
|
|
wall-clock or system source of randomness.
|
|||
|
|
|
|||
|
|
### 1.5 Sub-second per scenario
|
|||
|
|
|
|||
|
|
A 3-node scenario test (including bundle assembly and verdict
|
|||
|
|
evaluation) completes in under one second on the developer's
|
|||
|
|
machine. The full battery completes in under thirty seconds locally
|
|||
|
|
and under three minutes in CI. A scenario that grows above this
|
|||
|
|
budget is a regression in the test, not in the simulator; the test
|
|||
|
|
author tightens the scenario rather than relaxing the budget.
|
|||
|
|
|
|||
|
|
### 1.6 Verdict-first
|
|||
|
|
|
|||
|
|
Every test in the battery declares its **expected verdict on the
|
|||
|
|
current source** before it lands: `Pass` (the simulator already
|
|||
|
|
satisfies the contract; the test guards against regression), `Fail`
|
|||
|
|
(the simulator currently violates the contract; landing the test
|
|||
|
|
makes the failure visible, and the test is expected to pass after a
|
|||
|
|
fix names in §4), or `Mixed` (some seeds pass, some fail — typical
|
|||
|
|
for property tests against a probabilistic shape).
|
|||
|
|
|
|||
|
|
A test landing as `Fail` is **not** a build break in the test
|
|||
|
|
binary; it is a verdict in the bundle's `verdicts.json` whose CI
|
|||
|
|
exposure is named in §1.7. A test landing as `Pass` runs with
|
|||
|
|
`#[test]` semantics — a regression in the simulator is a CI break.
|
|||
|
|
|
|||
|
|
### 1.7 CI exposure
|
|||
|
|
|
|||
|
|
Tests with expected verdict `Pass` run as standard `cargo test`
|
|||
|
|
binaries under `crates/simulation/tests/`. Tests with expected
|
|||
|
|
verdict `Fail` or `Mixed` run as a separate
|
|||
|
|
`cargo test --package simulation --test battery_expected_failures`
|
|||
|
|
binary that asserts the verdict matches expectation (`Fail` →
|
|||
|
|
`Fail`, `Mixed` → at least one `Fail` across the seed range, at
|
|||
|
|
least one `Pass`). Promoting a `Fail` test to `Pass` after a fix is
|
|||
|
|
a one-line move between binaries and a deletion from the expected-
|
|||
|
|
failures registry; the implementer should make this move trivial.
|
|||
|
|
|
|||
|
|
### 1.8 Library layout
|
|||
|
|
|
|||
|
|
The battery's scenarios live under
|
|||
|
|
`crates/simulation/scenarios/reproduction/n3_2026_05_25/`, one
|
|||
|
|
subdirectory per family. Each family directory contains:
|
|||
|
|
|
|||
|
|
- A `README.md` naming the family, pointing at the postmortem, and
|
|||
|
|
listing the family's mutation axes.
|
|||
|
|
- One scenario file per named central or extreme case
|
|||
|
|
(`central.toml`, `extreme_*.toml`).
|
|||
|
|
- A `property.toml` file declaring the property-test seed range and
|
|||
|
|
axis bounds where §3 requires a property test.
|
|||
|
|
|
|||
|
|
This layout is the existing `scenarios/reproduction/` convention
|
|||
|
|
extended one level. No new top-level directories.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 2. The shared scenario shape
|
|||
|
|
|
|||
|
|
Every scenario in the battery has the following shape unless its
|
|||
|
|
family in §3 names a divergence:
|
|||
|
|
|
|||
|
|
- **Three peers**: one orchestrator-kind, two stage-kind. IDs
|
|||
|
|
`orch`, `stage-0`, `stage-2` (the latter named to match the
|
|||
|
|
postmortem's victim peer). The third stage from production is
|
|||
|
|
omitted only when its absence does not change the shape of the
|
|||
|
|
failure under test; families that require N=4 to manifest must say
|
|||
|
|
so explicitly. (`stage-1` may appear as a peer in families that
|
|||
|
|
need it; otherwise the simulator's N=3 minimum is the target.)
|
|||
|
|
- **One relay vertex** `R`, with policy seeded from the
|
|||
|
|
`vastai-N3-2` calibration scenario (own-relay shape — widened
|
|||
|
|
egress, modest queue depth). Per-family scenarios may tighten or
|
|||
|
|
loosen this; the central case for each family uses the calibration
|
|||
|
|
defaults.
|
|||
|
|
- **Routing**: all host-to-host edges declared `via = R`. The 2026-
|
|||
|
|
05-25 incident exercised the relay path exclusively; no direct
|
|||
|
|
edges in the battery's central cases. Extreme cases that need
|
|||
|
|
direct edges declare them per `SIM_SPEC.md §8.1`.
|
|||
|
|
- **Duration**: 10 simulated minutes (`duration_ns = 600_000_000_000`)
|
|||
|
|
matching the 2026-05-25 run's wall-clock budget. Scenarios may
|
|||
|
|
shorten but not lengthen — long scenarios violate the sub-second
|
|||
|
|
budget in §1.5.
|
|||
|
|
- **Snapshots**: at least one snapshot per peer per simulated
|
|||
|
|
minute, plus a snapshot one virtual nanosecond before and one
|
|||
|
|
after every named fault, so the bundle reader can see the state
|
|||
|
|
on each side of each transition. (This is a property of the
|
|||
|
|
scenario, not of the engine: the scenario's `[[snapshots]]` array
|
|||
|
|
declares these.)
|
|||
|
|
- **Assertions**: each family in §3 names its required assertions.
|
|||
|
|
Scenarios may add further assertions from §10.1 to tighten the
|
|||
|
|
contract; they may not remove or relax the named ones.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 3. The families
|
|||
|
|
|
|||
|
|
Six families, each named for the failure shape it covers. Families
|
|||
|
|
A, B, and C are derived directly from the 2026-05-25 incident.
|
|||
|
|
Families D, E, and F are derived from the broader N≥3 deployment
|
|||
|
|
history that the latest run did not contradict and should not
|
|||
|
|
regress.
|
|||
|
|
|
|||
|
|
### Family A — Relay-mediated peer-connection drop with surviving tunnel
|
|||
|
|
|
|||
|
|
**Source**: `N3_POSTMORTEM_2026-05-25.md` "iroh state — orchestrator's
|
|||
|
|
view of stage-2"; `N3_DATA_GAPS.md` gaps 1, 2, 3.
|
|||
|
|
|
|||
|
|
**Shape**: A peer-to-peer path through a relay opens, succeeds for a
|
|||
|
|
short window, then dies. The relay's tunnel to the victim peer
|
|||
|
|
remains apparently healthy — the victim's `Tier2RelaySession.status`
|
|||
|
|
stays `connected` or is reported as such by the relay, while the
|
|||
|
|
orchestrator's `connection_cache[victim].last_failure_reason` shows
|
|||
|
|
the path closed. iroh does not re-establish.
|
|||
|
|
|
|||
|
|
**Central case** (`central.toml`): `RelayPeerConnDown { relay: R,
|
|||
|
|
from: orch, to: stage-2, at_ns: 5_000_000_000, duration_ns: 0 }`
|
|||
|
|
(permanent until run end), inserted shortly after SWIM convergence.
|
|||
|
|
No other faults.
|
|||
|
|
|
|||
|
|
**Mutation axes** (the family's parameter space):
|
|||
|
|
|
|||
|
|
1. `at_ns`: when the cut fires. Central +5 s; extremes +1 s, +30 s,
|
|||
|
|
+1 min, +5 min.
|
|||
|
|
2. `duration_ns`: how long the cut persists. Central permanent;
|
|||
|
|
extremes 100 ms, 5 s, 30 s.
|
|||
|
|
3. Direction: cut on `(orch → stage-2)` only, on `(stage-2 → orch)`
|
|||
|
|
only, or on both. The 2026-05-25 evidence is ambiguous about
|
|||
|
|
direction; the battery covers all three.
|
|||
|
|
4. Flap: a sequence of `RelayPeerConnDown` mutations interleaved with
|
|||
|
|
their natural recovery — close, reopen, close. Inter-flap durations
|
|||
|
|
100 ms, 1 s, 5 s.
|
|||
|
|
5. Phase: cut during SWIM convergence (before all peers Alive); cut
|
|||
|
|
during steady-state after convergence; cut during a
|
|||
|
|
`Partition`+`Heal` cycle's heal phase (per `SIM_HARDENING_SPEC §9`).
|
|||
|
|
|
|||
|
|
**Required assertions**:
|
|||
|
|
|
|||
|
|
- `no_flap_while_probes_ok { peer: stage-2, window_start_ns:
|
|||
|
|
at_ns, window_end_ns: duration_ns_end }` — the family asserts the
|
|||
|
|
*observability* contract that a relay-peer cut produces a typed
|
|||
|
|
event chain (`RelayPeerConnDown` mutation record →
|
|||
|
|
`RelaySessionStateChanged` or equivalent on the victim's view →
|
|||
|
|
`connection-closed` in the observer's cache). What it does *not*
|
|||
|
|
assert is that the simulator's SWIM tolerates the cut — the
|
|||
|
|
current simulator does not.
|
|||
|
|
- `event_count { kind: "RelaySessionStateChanged", min: 1 }` on
|
|||
|
|
the central case — a cut must produce at least one transition
|
|||
|
|
event for the bundle reader to see.
|
|||
|
|
- `dead_peer_resurrects_within { peer: stage-2, after_ns:
|
|||
|
|
heal_at_ns, within_ns: 30_000_000_000 }` on the finite-duration
|
|||
|
|
extreme cases — once the cut lifts, the cluster must reconverge.
|
|||
|
|
|
|||
|
|
**Property test**: `property.toml` ranges seeds 0..256 over axes 1,
|
|||
|
|
2, and 5. The seed search reports any seed whose run violates
|
|||
|
|
`no_flap_while_probes_ok` while the cut is *not* active (a
|
|||
|
|
false-flap during a healthy window — the bug class the family
|
|||
|
|
exists to catch).
|
|||
|
|
|
|||
|
|
**Expected verdict on current source**: `Mixed`. The central case
|
|||
|
|
is expected `Fail` against the current SWIM source (the
|
|||
|
|
deployment's actual failure mode); the flap extreme and the
|
|||
|
|
phase-during-heal extreme are also expected `Fail`. The finite-
|
|||
|
|
duration extremes with short cuts may pass.
|
|||
|
|
|
|||
|
|
**Family closes when**: a fix lands that lets the central case
|
|||
|
|
pass and at least the flap and phase-during-heal extremes pass,
|
|||
|
|
with no other family regressing.
|
|||
|
|
|
|||
|
|
### Family B — Silent stage subprocess (never spawned, spawned-and-stuck, spawned-and-exited)
|
|||
|
|
|
|||
|
|
**Source**: `N3_POSTMORTEM_2026-05-25.md` "Custom (worker) events"
|
|||
|
|
table (`stage-2` emitted zero `worker_starting`, zero `worker_ready`);
|
|||
|
|
`N3_DATA_GAPS.md` gap 4; `SIM_HARDENING_SPEC §5`.
|
|||
|
|
|
|||
|
|
**Shape**: A stage's worker subprocess fails to reach the
|
|||
|
|
`worker_ready` state. The stage actor itself is alive — snapshots
|
|||
|
|
still arrive, events still flow — but no work begins. The failure
|
|||
|
|
splits into three buckets per the §4 spec the observability upgrade
|
|||
|
|
already landed: never-spawned, spawned-and-stalled-before-ready,
|
|||
|
|
spawned-and-exited-before-ready.
|
|||
|
|
|
|||
|
|
**Central case** (`central.toml`): install a `SubprocessFakeSpec`
|
|||
|
|
on `stage-2` with `never_ready = true`, no `exit_after_ns`. The
|
|||
|
|
orchestrator's view: `SubprocessSpawned` arrives, no `worker_ready`
|
|||
|
|
Custom event ever does. The sim already supports this via `F1`.
|
|||
|
|
|
|||
|
|
**Mutation axes**:
|
|||
|
|
|
|||
|
|
1. Bucket: `never_spawned` (no `SubprocessFakeSpec` installed at
|
|||
|
|
all; stage actor never registers); `stalled` (spawned, never
|
|||
|
|
ready); `early_exit` (spawned, exits before ready with named
|
|||
|
|
exit code / signal).
|
|||
|
|
2. `exit_after_ns` for the `early_exit` bucket: 100 ms (faster than
|
|||
|
|
any plausible ready), 1 s, 10 s.
|
|||
|
|
3. Number of victim stages: one (central), two (whole stage layer
|
|||
|
|
silent), zero (control — all stages reach `worker_ready` —
|
|||
|
|
sanity).
|
|||
|
|
4. Whether SWIM convergence completes before or after the worker
|
|||
|
|
silence is observable.
|
|||
|
|
|
|||
|
|
**Required assertions**:
|
|||
|
|
|
|||
|
|
- The bundle must make the three buckets distinguishable at the
|
|||
|
|
verdict level. The discriminator is the joint state of
|
|||
|
|
`SubprocessSpawned`, `SubprocessExited`, and the `worker_ready`
|
|||
|
|
Custom event for the victim peer, with the buckets mapping as:
|
|||
|
|
- `never_spawned`: `SubprocessSpawned == 0`, `worker_ready == 0`.
|
|||
|
|
- `stalled`: `SubprocessSpawned == 1`, `worker_ready == 0`, no
|
|||
|
|
`SubprocessExited` for the run's duration.
|
|||
|
|
- `early_exit`: `SubprocessSpawned == 1`, `worker_ready == 0`,
|
|||
|
|
`SubprocessExited == 1` with the declared reason.
|
|||
|
|
- `name_resolves_within { name: "pp-entry", observers: [orch],
|
|||
|
|
within_ns: 300_000_000_000, from_ns: 0 }` — the orchestrator's
|
|||
|
|
resolution of the pipeline entry name must fail when any victim
|
|||
|
|
stage is silent. The contract: `Inconclusive` is **not**
|
|||
|
|
acceptable — the bundle must clearly say "the orchestrator looked
|
|||
|
|
and the name was absent," not "we don't know if the orchestrator
|
|||
|
|
looked."
|
|||
|
|
|
|||
|
|
**Property test**: not required for B. The bucket count is small
|
|||
|
|
enough that all combinations land as scenario tests.
|
|||
|
|
|
|||
|
|
**Expected verdict on current source**: per-bucket. `never_spawned`
|
|||
|
|
and `stalled` expected `Fail` on the `name_resolves_within`
|
|||
|
|
assertion (correct — the cluster cannot resolve `pp-entry` if a
|
|||
|
|
stage is silent). `early_exit` expected `Fail` on the same plus
|
|||
|
|
`event_count { kind: "SubprocessExited", min: 1 }` with the
|
|||
|
|
correct exit code observable in the bundle.
|
|||
|
|
|
|||
|
|
The battery's job here is to **prove the bucket is observable**, not
|
|||
|
|
to prove the cluster recovers. Recovery from a silent worker is a
|
|||
|
|
product question, not a sim contract.
|
|||
|
|
|
|||
|
|
**Family closes when**: the bundle's `summary.md` (rendered through
|
|||
|
|
`swactor-diag-postproc`) names which bucket the victim stage is in,
|
|||
|
|
in human-readable prose, for every scenario in the family.
|
|||
|
|
|
|||
|
|
### Family C — Gossip-arrival absence (control-plane vs data-plane discriminator)
|
|||
|
|
|
|||
|
|
**Source**: `N3_POSTMORTEM_2026-05-25.md` "iroh state — stage-2's
|
|||
|
|
view of itself" (`peers: [orchestrator only]`); `N3_DATA_GAPS.md`
|
|||
|
|
gap 10; `SIM_HARDENING_SPEC` §1 and §2.
|
|||
|
|
|
|||
|
|
**Shape**: A victim peer's local membership view contains only the
|
|||
|
|
orchestrator, never its siblings. Two possible causes are
|
|||
|
|
indistinguishable from the postmortem bundle: gossip about siblings
|
|||
|
|
never arrived (control-plane failure), or gossip arrived but the
|
|||
|
|
dials based on it never connected (data-plane failure). The battery
|
|||
|
|
must let a single scenario+verdict pair disambiguate these.
|
|||
|
|
|
|||
|
|
**Central case** (`central.toml`): a `Partition` mutation that
|
|||
|
|
isolates `stage-2` from `stage-0` and `stage-1` at the
|
|||
|
|
network-graph layer (no direct, no relayed route between them),
|
|||
|
|
while leaving each stage's path to `orch` intact. Stage-2 should
|
|||
|
|
never receive gossip naming stage-0 / stage-1.
|
|||
|
|
|
|||
|
|
**Mutation axes**:
|
|||
|
|
|
|||
|
|
1. Topology: full isolation (central); one-way isolation (stage-2
|
|||
|
|
receives gossip, dials silently dropped); periodic gossip drops
|
|||
|
|
modulated by `LossBurst`.
|
|||
|
|
2. Whether the orchestrator's gossip-piggyback ever names the
|
|||
|
|
siblings (which depends on its own membership view at the time
|
|||
|
|
stage-2 boots and receives its first ping).
|
|||
|
|
|
|||
|
|
**Required assertions**:
|
|||
|
|
|
|||
|
|
- `event_count { kind: "GossipReceived", peer: stage-2,
|
|||
|
|
payload_kind: "NameRegistry", min: N }` where `N` depends on the
|
|||
|
|
axis: for the central case, `N >= 1` (gossip must reach
|
|||
|
|
stage-2); the assertion lets us prove the discriminator. A
|
|||
|
|
scenario in which gossip *did* arrive but dials failed produces
|
|||
|
|
`GossipReceived >= 1` and `DialOutcome` with failure reasons for
|
|||
|
|
the siblings; a scenario in which gossip never arrived produces
|
|||
|
|
`GossipReceived == 0`. The two bundles are now distinguishable
|
|||
|
|
by the verdict.
|
|||
|
|
- `event_count { kind: "DialStarted", peer: stage-2, target: stage-0,
|
|||
|
|
min: 1 }` on the one-way-isolation axis: dials must be observable
|
|||
|
|
in the data-plane-failure case.
|
|||
|
|
|
|||
|
|
**Property test**: not required.
|
|||
|
|
|
|||
|
|
**Expected verdict on current source**: `Pass` for all cases — the
|
|||
|
|
observability upgrade landed `GossipReceived` (`S-E2`) and the
|
|||
|
|
per-peer dial rollup (`S-A3`), so the discriminator is already
|
|||
|
|
expressible. The battery's job is to *guard* this contract against
|
|||
|
|
regression in the simulator or in the post-processor.
|
|||
|
|
|
|||
|
|
**Family closes when**: a probe-by-grep against the bundle's
|
|||
|
|
`summary.md` confirms the discriminator is named in prose, not
|
|||
|
|
buried in raw event counts.
|
|||
|
|
|
|||
|
|
### Family D — Asymmetric host reachability (NAT / mapping pathology)
|
|||
|
|
|
|||
|
|
**Source**: `N3_POSTMORTEM_2026-05-25.md` "UDP echo probes" (stage-2
|
|||
|
|
1/12 timeout while others were clean); `N3_DATA_GAPS.md` gaps 8 and
|
|||
|
|
11; `SIM_HARDENING_SPEC §2` host-environment-level faults.
|
|||
|
|
|
|||
|
|
**Shape**: One peer's host network behaves correctly *most* of the
|
|||
|
|
time, but exhibits asymmetric loss, NAT-rebind, or kernel-UDP-buffer
|
|||
|
|
overflow in a pattern that downstream iroh layers cannot
|
|||
|
|
distinguish from a relay-side issue or a peer-software issue. The
|
|||
|
|
postmortem could not tell which.
|
|||
|
|
|
|||
|
|
**Central case** (`central.toml`): a `LossBurst` on
|
|||
|
|
`(stage-2 → R)` with `prob_ppm = 80_000` (8% loss) lasting 30 s
|
|||
|
|
during steady state. This is the smallest fault that produces the
|
|||
|
|
postmortem's "one peer flaky, others clean" symptom.
|
|||
|
|
|
|||
|
|
**Mutation axes**:
|
|||
|
|
|
|||
|
|
1. Symmetry: loss on outbound from victim, on inbound to victim,
|
|||
|
|
on both directions, none (control).
|
|||
|
|
2. Burst shape: continuous low-rate loss vs short high-rate burst.
|
|||
|
|
3. Co-occurrence: loss alone vs loss + clock skew on the same peer
|
|||
|
|
(compound — per `SIM_HARDENING_SPEC §7`).
|
|||
|
|
|
|||
|
|
**Required assertions**:
|
|||
|
|
|
|||
|
|
- The bundle's UDP echo probe records must show the victim's
|
|||
|
|
outcome distribution (`ok` / `timeout` / `refused` / `unresolved`
|
|||
|
|
/ `error`) differing from the other peers' by a margin evident
|
|||
|
|
to a human reader.
|
|||
|
|
- Across the run, the victim's
|
|||
|
|
`Tier3InterfaceCounters.rx_packets_dropped` or
|
|||
|
|
`Tier3UdpKernelStats.in_errors` is non-zero in the bundle, while
|
|||
|
|
the other peers' is zero. This is the "kernel saw the loss, not
|
|||
|
|
just iroh" contract gap 11 demanded.
|
|||
|
|
|
|||
|
|
**Property test**: required, seeds 0..128. Range over axes 1 and
|
|||
|
|
2. The property: for every seed in which the victim's UDP echo
|
|||
|
|
shows >5% loss, the bundle must surface a non-zero kernel-counter
|
|||
|
|
delta on the same peer. (This is the discriminator gap 11 asked
|
|||
|
|
for.)
|
|||
|
|
|
|||
|
|
**Expected verdict on current source**: `Mixed`. The observability
|
|||
|
|
upgrade landed kernel counters in the bundle (`S-A4`); the
|
|||
|
|
simulator's stage host needs to emit `Tier3InterfaceCounters` under
|
|||
|
|
the loss-burst mutation for the discriminator to hold. If it does
|
|||
|
|
not, that is a sim-coverage gap belonging in `SIM_BLIND_SPOTS.md`
|
|||
|
|
per `SIM_HARDENING_SPEC §10`, not a reason to relax the assertion.
|
|||
|
|
|
|||
|
|
**Family closes when**: the property test runs to 128 seeds with
|
|||
|
|
the loss-discriminator holding on every seed it sees loss; the
|
|||
|
|
sim-coverage gap, if it exists, is filed.
|
|||
|
|
|
|||
|
|
### Family E — Bundle integrity under operator SIGKILL
|
|||
|
|
|
|||
|
|
**Source**: `N3_POSTMORTEM_2026-05-25.md` "Bundle recovery";
|
|||
|
|
`N3_DATA_GAPS.md` gap 7; observability upgrade `S-D` (bundle
|
|||
|
|
without finalize).
|
|||
|
|
|
|||
|
|
**Shape**: The orchestrator is killed ungracefully (SIGKILL via
|
|||
|
|
TaskStop, not graceful shutdown). No finalize record is written.
|
|||
|
|
The diagnostic bundle must still be assemblable from staging files
|
|||
|
|
on disk, with `manifest.finalize_received: false`.
|
|||
|
|
|
|||
|
|
**Central case** (`central.toml`): a `PeerKill { peer: orch,
|
|||
|
|
at_ns: 60_000_000_000 }` mutation 60 s into the run. No
|
|||
|
|
`PeerResurrect`. The scenario's `duration_ns` extends 30 s past
|
|||
|
|
the kill so the collector has time to observe and the bundle has
|
|||
|
|
time to coalesce.
|
|||
|
|
|
|||
|
|
**Mutation axes**:
|
|||
|
|
|
|||
|
|
1. Timing of kill: during convergence, during steady state, during
|
|||
|
|
a partition heal.
|
|||
|
|
2. Which peer: orchestrator, a stage, the relay.
|
|||
|
|
|
|||
|
|
**Required assertions**:
|
|||
|
|
|
|||
|
|
- The bundle's `manifest.json` must exist and contain
|
|||
|
|
`finalize_received: false`.
|
|||
|
|
- Every peer's pre-kill events and snapshots must be present in
|
|||
|
|
the bundle (the kill must not erase prior records).
|
|||
|
|
- The `verdicts.json` must contain a verdict for every declared
|
|||
|
|
assertion, with `Inconclusive` for any assertion whose
|
|||
|
|
preconditions did not fire (e.g., a steady-state assertion when
|
|||
|
|
steady state was never reached).
|
|||
|
|
|
|||
|
|
**Property test**: not required.
|
|||
|
|
|
|||
|
|
**Expected verdict on current source**: `Pass`. The observability
|
|||
|
|
upgrade landed `S-D` (bundle assembly without finalize). This
|
|||
|
|
family guards that contract against regression.
|
|||
|
|
|
|||
|
|
**Family closes when**: every scenario in the family produces a
|
|||
|
|
parseable bundle whose `summary.md` renders cleanly through
|
|||
|
|
`swactor-diag-postproc`.
|
|||
|
|
|
|||
|
|
### Family F — Compound faults under recovery
|
|||
|
|
|
|||
|
|
**Source**: `SIM_HARDENING_SPEC §7` and §9.
|
|||
|
|
|
|||
|
|
**Shape**: Two or more faults active during a single recovery
|
|||
|
|
window — a partition heal during a relay-peer-down, a clock skew
|
|||
|
|
during a worker respawn, a kernel UDP overflow during SWIM gossip
|
|||
|
|
burst. The 2026-05-25 incident is consistent with at least two
|
|||
|
|
overlapping faults (relay-peer-down + silent-worker); the battery
|
|||
|
|
must cover the next overlap before it lands in prod.
|
|||
|
|
|
|||
|
|
**Central case** (`central.toml`): a `Partition` cutting `stage-2`
|
|||
|
|
from `stage-0` from t=10 s to t=30 s; a `RelayPeerConnDown { from:
|
|||
|
|
orch, to: stage-2, at_ns: 20_000_000_000, duration_ns:
|
|||
|
|
20_000_000_000 }` overlapping the partition's last 10 s and
|
|||
|
|
extending 10 s past its heal. The scenario tests whether SWIM
|
|||
|
|
behaves under the *overlap* and the *heal* sequence the postmortem
|
|||
|
|
mentions but did not isolate.
|
|||
|
|
|
|||
|
|
**Mutation axes**:
|
|||
|
|
|
|||
|
|
1. Which two faults overlap (cross product of the four single-fault
|
|||
|
|
families above, restricted to combinations that produce
|
|||
|
|
distinguishable bundles).
|
|||
|
|
2. Overlap geometry: full overlap, partial overlap, abutting (one
|
|||
|
|
ends as the other begins).
|
|||
|
|
3. Recovery phase: which recovery phase the second fault hits, per
|
|||
|
|
`SIM_HARDENING_SPEC §9`.
|
|||
|
|
|
|||
|
|
**Required assertions**: family-dependent — each compound test
|
|||
|
|
combines the assertions of its constituent families. The compound
|
|||
|
|
test passes only if every constituent assertion holds.
|
|||
|
|
|
|||
|
|
**Property test**: required, seeds 0..512. Range over all three
|
|||
|
|
axes. The property: any seed in which a compound bundle violates
|
|||
|
|
*more* assertions than the sum of the constituents' individual
|
|||
|
|
violations is a true compound bug, reported separately.
|
|||
|
|
|
|||
|
|
**Expected verdict on current source**: `Mixed`. Compound failures
|
|||
|
|
are the under-tested corner; the implementing agent should expect
|
|||
|
|
to find at least one new sim-coverage gap during this family's
|
|||
|
|
implementation and file it.
|
|||
|
|
|
|||
|
|
**Family closes when**: at least one compound bug is either fixed
|
|||
|
|
or filed as a sim-coverage gap with a structural reason.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 4. Out of scope
|
|||
|
|
|
|||
|
|
- Tuning the simulator's existing scenarios under
|
|||
|
|
`scenarios/calibration/` or `scenarios/smoke/`.
|
|||
|
|
- Adding new failure shapes the 2026-05-25 deployment did not
|
|||
|
|
surface (the diagnostic deployment running in parallel may; if
|
|||
|
|
so, those land as a new spec, not as an amendment to this one).
|
|||
|
|
- Changes to the simulator's engine, network, host kinds, bundle
|
|||
|
|
writer, or post-processor. The battery exercises them; it does
|
|||
|
|
not modify them.
|
|||
|
|
- Changes to the production diagnostics code path. The
|
|||
|
|
observability upgrade landed; the battery consumes its output.
|
|||
|
|
- Documentation of the simulator beyond `SIM_BLIND_SPOTS.md`
|
|||
|
|
amendments. `SIM_HARDENING_SPEC.md` and `SIM_SPEC.md` already
|
|||
|
|
exist; this document is the only new prose required.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 5. References
|
|||
|
|
|
|||
|
|
- `N3_POSTMORTEM_2026-05-25.md` — source for families A, B, C, D, E.
|
|||
|
|
- `N3_DATA_GAPS.md` — source for the gap-named contracts each
|
|||
|
|
family asserts the simulator's bundle must satisfy.
|
|||
|
|
- `N3_DEPLOYMENT_REPORT.md` — historical context: Layers A/B/C from
|
|||
|
|
the prior eight deploys.
|
|||
|
|
- `N3_OBSERVABILITY_UPGRADE_SPEC.md` — the contract the bundle
|
|||
|
|
*already* satisfies. The battery consumes that contract.
|
|||
|
|
- `SIM_HARDENING_SPEC.md` — the family / mutation-axis discipline
|
|||
|
|
that §1 and §3 above enforce.
|
|||
|
|
- `crates/simulation/SIM_SPEC.md` — the simulator's behavioral
|
|||
|
|
surface. §3.1 components, §5.5 mutations, §6A stage host, §10.1
|
|||
|
|
assertion catalog, §8 scenario format are the load-bearing
|
|||
|
|
references.
|
|||
|
|
- `.loop/notes.md`, `.loop/verdict.md` — the observability-upgrade
|
|||
|
|
iteration log and verdict, current as of 2026-05-25; STATUS:
|
|||
|
|
DONE, VERDICT: PASS.
|