swactor/examples/pipeline-parallel-inference/N3_SIM_TEST_BATTERY_SPEC.md

N=3 sim-test battery — behavioral specification

Companion to N3_POSTMORTEM_2026-05-25.md, N3_DATA_GAPS.md, N3_DEPLOYMENT_REPORT.md, SIM_HARDENING_SPEC.md, and the simulator's SIM_SPEC.md. This document is the contract for a separate coding agent that will land a battery of simulator tests covering the general failure shapes the latest deployment exposed.

This is a behavioral spec. It names the failure shapes, the contracts each test must establish, and the verdicts each must produce. It does not prescribe file layout, TOML field values, or internal helper code.

---

0. Motivation and framing

The 2026-05-25 deployment surfaced one new failure shape (stage-2's relay-mediated path died at ~5 s and never recovered, while its tunnel to the relay apparently survived) layered on top of failure shapes prior deploys also exhibited (silent-worker subprocess, gossip-only membership view, asymmetric host reachability, bundle-recovery only via staging-file scrape). Together these are the general failure cases the battery must cover — not one scenario per postmortem, but a family per shape, as SIM_HARDENING_SPEC.md §5 requires.

The simulator has now landed every observability and sim-cross- pollination contract those postmortems demanded (F1–F3, S-A1 through S-E2; see .loop/verdict.md). The pieces needed to express these scenarios all exist: MutationKind::RelayPeerConnDown, the stage host kind with WorkerExit, the relay vertex with policy mutations, and the §10.1 assertion catalog. The battery is the exercise of those pieces against the latest deployment's known shapes, expressed end-to-end through scenario files and verdicts — not new sim machinery.

Why a battery rather than one test per shape: the SIM_HARDENING_SPEC §5 family rule. A fix that resolves the 2026-05-25 incident's specific timing (relay-peer-down at +5 s) but regresses a sibling instance of the family (relay-peer-down at +30 s, or during partition heal, or on only the inbound leg) is a regression the battery must catch.

A diagnostic deployment is running concurrently to gather data we don't yet have for the silent-worker class. This spec is written against the evidence already in the bundle from 2026-05-25; the implementing agent should not block on that deploy's results. When results land they will sharpen the parameters of family B (silent-worker) but will not change the shape of the battery.

---

1. Cross-cutting requirements

These hold for every family in §3.

1.1 No white-box / structural tests

A test in the battery passes or fails based on the bundle the scenario produces and the verdicts the §10.1 assertion catalog returns against that bundle. No test reads simulator internals, no test inserts a value via one API path and reads it back via another, no test asserts that an internal Rust struct has a particular field shape. A test that would survive a refactor of the engine, the network, or any host kind, but fail when the deployment-relevant behavior drifts, is a test that belongs.

Litmus test: if removing the assertion would change the bundle's prose summary in a way a deployment investigator would notice, the assertion belongs. If removing it would not, the assertion is echoing internals and does not belong.

1.2 Test taxonomy and priority

Each family ships at least one scenario test (story-shape: declared scenario + declared assertion + declared expected verdict) and where the parameter space is large, at least one property test (a parameterized scenario whose seed ranges over the §1.3 family axes). Scenario tests are mandatory; property tests are required only where §3 names them.

A small number of contract tests sit alongside the families: they assert that the bundle's event schema matches the production diagnostics schema for the event kinds the battery exercises (the SubprocessSpawned/SubprocessExited, RelaySessionStateChanged, GossipReceived, and Tier2RelaySession shapes the observability upgrade landed). The contract tests are not per-family; they live once and protect every family from sim/prod drift.

1.3 Family-based, not single-seed

Every family in §3 declares its mutation axes — the dimensions along which the postmortem's parameters are "plausibly variable in the wild" per SIM_HARDENING_SPEC §5. The family's scenario tests cover the central case (the specific incident's parameters) and the named extreme cases (e.g., "session closes at +1 s" and "session closes at +5 min" for family A). The family's property test ranges over the axes within their declared bounds.

1.4 Deterministic replay

Every scenario test's (scenario, seed) is recorded in the test itself; running the test produces a byte-identical bundle to any previous run on any supported architecture. A property-test failure prints the seed; running the scenario with that seed reproduces the failure. This is mechanical — the simulator already guarantees it (SIM_SPEC.md §7); the battery must not undo it. No test reads any wall-clock or system source of randomness.

1.5 Sub-second per scenario

A 3-node scenario test (including bundle assembly and verdict evaluation) completes in under one second on the developer's machine. The full battery completes in under thirty seconds locally and under three minutes in CI. A scenario that grows above this budget is a regression in the test, not in the simulator; the test author tightens the scenario rather than relaxing the budget.

1.6 Verdict-first

Every test in the battery declares its expected verdict on the current source before it lands: Pass (the simulator already satisfies the contract; the test guards against regression), Fail (the simulator currently violates the contract; landing the test makes the failure visible, and the test is expected to pass after a fix names in §4), or Mixed (some seeds pass, some fail — typical for property tests against a probabilistic shape).

A test landing as Fail is not a build break in the test binary; it is a verdict in the bundle's verdicts.json whose CI exposure is named in §1.7. A test landing as Pass runs with #[test] semantics — a regression in the simulator is a CI break.

1.7 CI exposure

Tests with expected verdict Pass run as standard cargo test binaries under crates/simulation/tests/. Tests with expected verdict Fail or Mixed run as a separate cargo test --package simulation --test battery_expected_failures binary that asserts the verdict matches expectation (Fail → Fail, Mixed → at least one Fail across the seed range, at least one Pass). Promoting a Fail test to Pass after a fix is a one-line move between binaries and a deletion from the expected- failures registry; the implementer should make this move trivial.

1.8 Library layout

The battery's scenarios live under crates/simulation/scenarios/reproduction/n3_2026_05_25/, one subdirectory per family. Each family directory contains:

• A README.md naming the family, pointing at the postmortem, and listing the family's mutation axes.
• One scenario file per named central or extreme case (central.toml, extreme_*.toml).
• A property.toml file declaring the property-test seed range and axis bounds where §3 requires a property test.

This layout is the existing scenarios/reproduction/ convention extended one level. No new top-level directories.

---

2. The shared scenario shape

Every scenario in the battery has the following shape unless its family in §3 names a divergence:

• Three peers: one orchestrator-kind, two stage-kind. IDs orch, stage-0, stage-2 (the latter named to match the postmortem's victim peer). The third stage from production is omitted only when its absence does not change the shape of the failure under test; families that require N=4 to manifest must say so explicitly. (stage-1 may appear as a peer in families that need it; otherwise the simulator's N=3 minimum is the target.)
• One relay vertex R, with policy seeded from the vastai-N3-2 calibration scenario (own-relay shape — widened egress, modest queue depth). Per-family scenarios may tighten or loosen this; the central case for each family uses the calibration defaults.
• Routing: all host-to-host edges declared via = R. The 2026- 05-25 incident exercised the relay path exclusively; no direct edges in the battery's central cases. Extreme cases that need direct edges declare them per SIM_SPEC.md §8.1.
• Duration: 10 simulated minutes (duration_ns = 600_000_000_000) matching the 2026-05-25 run's wall-clock budget. Scenarios may shorten but not lengthen — long scenarios violate the sub-second budget in §1.5.
• Snapshots: at least one snapshot per peer per simulated minute, plus a snapshot one virtual nanosecond before and one after every named fault, so the bundle reader can see the state on each side of each transition. (This is a property of the scenario, not of the engine: the scenario's [[snapshots]] array declares these.)
• Assertions: each family in §3 names its required assertions. Scenarios may add further assertions from §10.1 to tighten the contract; they may not remove or relax the named ones.

---

3. The families

Six families, each named for the failure shape it covers. Families A, B, and C are derived directly from the 2026-05-25 incident. Families D, E, and F are derived from the broader N≥3 deployment history that the latest run did not contradict and should not regress.

Family A — Relay-mediated peer-connection drop with surviving tunnel

Source: N3_POSTMORTEM_2026-05-25.md "iroh state — orchestrator's view of stage-2"; N3_DATA_GAPS.md gaps 1, 2, 3.

Shape: A peer-to-peer path through a relay opens, succeeds for a short window, then dies. The relay's tunnel to the victim peer remains apparently healthy — the victim's Tier2RelaySession.status stays connected or is reported as such by the relay, while the orchestrator's connection_cache[victim].last_failure_reason shows the path closed. iroh does not re-establish.

Central case (central.toml): RelayPeerConnDown { relay: R, from: orch, to: stage-2, at_ns: 5_000_000_000, duration_ns: 0 } (permanent until run end), inserted shortly after SWIM convergence. No other faults.

Mutation axes (the family's parameter space):

1. at_ns: when the cut fires. Central +5 s; extremes +1 s, +30 s, +1 min, +5 min.
2. duration_ns: how long the cut persists. Central permanent; extremes 100 ms, 5 s, 30 s.
3. Direction: cut on (orch → stage-2) only, on (stage-2 → orch) only, or on both. The 2026-05-25 evidence is ambiguous about direction; the battery covers all three.
4. Flap: a sequence of RelayPeerConnDown mutations interleaved with their natural recovery — close, reopen, close. Inter-flap durations 100 ms, 1 s, 5 s.
5. Phase: cut during SWIM convergence (before all peers Alive); cut during steady-state after convergence; cut during a Partition+Heal cycle's heal phase (per SIM_HARDENING_SPEC §9).

Required assertions:

• no_flap_while_probes_ok { peer: stage-2, window_start_ns: at_ns, window_end_ns: duration_ns_end } — the family asserts the observability contract that a relay-peer cut produces a typed event chain (RelayPeerConnDown mutation record → RelaySessionStateChanged or equivalent on the victim's view → connection-closed in the observer's cache). What it does not assert is that the simulator's SWIM tolerates the cut — the current simulator does not.
• event_count { kind: "RelaySessionStateChanged", min: 1 } on the central case — a cut must produce at least one transition event for the bundle reader to see.
• dead_peer_resurrects_within { peer: stage-2, after_ns: heal_at_ns, within_ns: 30_000_000_000 } on the finite-duration extreme cases — once the cut lifts, the cluster must reconverge.

Property test: property.toml ranges seeds 0..256 over axes 1, 2, and 5. The seed search reports any seed whose run violates no_flap_while_probes_ok while the cut is not active (a false-flap during a healthy window — the bug class the family exists to catch).

Expected verdict on current source: Mixed. The central case is expected Fail against the current SWIM source (the deployment's actual failure mode); the flap extreme and the phase-during-heal extreme are also expected Fail. The finite- duration extremes with short cuts may pass.

Family closes when: a fix lands that lets the central case pass and at least the flap and phase-during-heal extremes pass, with no other family regressing.

Family B — Silent stage subprocess (never spawned, spawned-and-stuck, spawned-and-exited)

Source: N3_POSTMORTEM_2026-05-25.md "Custom (worker) events" table (stage-2 emitted zero worker_starting, zero worker_ready); N3_DATA_GAPS.md gap 4; SIM_HARDENING_SPEC §5.

Shape: A stage's worker subprocess fails to reach the worker_ready state. The stage actor itself is alive — snapshots still arrive, events still flow — but no work begins. The failure splits into three buckets per the §4 spec the observability upgrade already landed: never-spawned, spawned-and-stalled-before-ready, spawned-and-exited-before-ready.

Central case (central.toml): install a SubprocessFakeSpec on stage-2 with never_ready = true, no exit_after_ns. The orchestrator's view: SubprocessSpawned arrives, no worker_ready Custom event ever does. The sim already supports this via F1.

Mutation axes:

1. Bucket: never_spawned (no SubprocessFakeSpec installed at all; stage actor never registers); stalled (spawned, never ready); early_exit (spawned, exits before ready with named exit code / signal).
2. exit_after_ns for the early_exit bucket: 100 ms (faster than any plausible ready), 1 s, 10 s.
3. Number of victim stages: one (central), two (whole stage layer silent), zero (control — all stages reach worker_ready — sanity).
4. Whether SWIM convergence completes before or after the worker silence is observable.

Required assertions:

• The bundle must make the three buckets distinguishable at the verdict level. The discriminator is the joint state of SubprocessSpawned, SubprocessExited, and the worker_ready Custom event for the victim peer, with the buckets mapping as:
  • never_spawned: SubprocessSpawned == 0, worker_ready == 0.
  • stalled: SubprocessSpawned == 1, worker_ready == 0, no SubprocessExited for the run's duration.
  • early_exit: SubprocessSpawned == 1, worker_ready == 0, SubprocessExited == 1 with the declared reason.
• name_resolves_within { name: "pp-entry", observers: [orch], within_ns: 300_000_000_000, from_ns: 0 } — the orchestrator's resolution of the pipeline entry name must fail when any victim stage is silent. The contract: Inconclusive is not acceptable — the bundle must clearly say "the orchestrator looked and the name was absent," not "we don't know if the orchestrator looked."

Property test: not required for B. The bucket count is small enough that all combinations land as scenario tests.

Expected verdict on current source: per-bucket. never_spawned and stalled expected Fail on the name_resolves_within assertion (correct — the cluster cannot resolve pp-entry if a stage is silent). early_exit expected Fail on the same plus event_count { kind: "SubprocessExited", min: 1 } with the correct exit code observable in the bundle.

The battery's job here is to prove the bucket is observable, not to prove the cluster recovers. Recovery from a silent worker is a product question, not a sim contract.

Family closes when: the bundle's summary.md (rendered through swactor-diag-postproc) names which bucket the victim stage is in, in human-readable prose, for every scenario in the family.

Family C — Gossip-arrival absence (control-plane vs data-plane discriminator)

Source: N3_POSTMORTEM_2026-05-25.md "iroh state — stage-2's view of itself" (peers: [orchestrator only]); N3_DATA_GAPS.md gap 10; SIM_HARDENING_SPEC §1 and §2.

Shape: A victim peer's local membership view contains only the orchestrator, never its siblings. Two possible causes are indistinguishable from the postmortem bundle: gossip about siblings never arrived (control-plane failure), or gossip arrived but the dials based on it never connected (data-plane failure). The battery must let a single scenario+verdict pair disambiguate these.

Central case (central.toml): a Partition mutation that isolates stage-2 from stage-0 and stage-1 at the network-graph layer (no direct, no relayed route between them), while leaving each stage's path to orch intact. Stage-2 should never receive gossip naming stage-0 / stage-1.

Mutation axes:

1. Topology: full isolation (central); one-way isolation (stage-2 receives gossip, dials silently dropped); periodic gossip drops modulated by LossBurst.
2. Whether the orchestrator's gossip-piggyback ever names the siblings (which depends on its own membership view at the time stage-2 boots and receives its first ping).

Required assertions:

• event_count { kind: "GossipReceived", peer: stage-2, payload_kind: "NameRegistry", min: N } where N depends on the axis: for the central case, N >= 1 (gossip must reach stage-2); the assertion lets us prove the discriminator. A scenario in which gossip did arrive but dials failed produces GossipReceived >= 1 and DialOutcome with failure reasons for the siblings; a scenario in which gossip never arrived produces GossipReceived == 0. The two bundles are now distinguishable by the verdict.
• event_count { kind: "DialStarted", peer: stage-2, target: stage-0, min: 1 } on the one-way-isolation axis: dials must be observable in the data-plane-failure case.

Property test: not required.

Expected verdict on current source: Pass for all cases — the observability upgrade landed GossipReceived (S-E2) and the per-peer dial rollup (S-A3), so the discriminator is already expressible. The battery's job is to guard this contract against regression in the simulator or in the post-processor.

Family closes when: a probe-by-grep against the bundle's summary.md confirms the discriminator is named in prose, not buried in raw event counts.

Family D — Asymmetric host reachability (NAT / mapping pathology)

Source: N3_POSTMORTEM_2026-05-25.md "UDP echo probes" (stage-2 1/12 timeout while others were clean); N3_DATA_GAPS.md gaps 8 and 11; SIM_HARDENING_SPEC §2 host-environment-level faults.

Shape: One peer's host network behaves correctly most of the time, but exhibits asymmetric loss, NAT-rebind, or kernel-UDP-buffer overflow in a pattern that downstream iroh layers cannot distinguish from a relay-side issue or a peer-software issue. The postmortem could not tell which.

Central case (central.toml): a LossBurst on (stage-2 → R) with prob_ppm = 80_000 (8% loss) lasting 30 s during steady state. This is the smallest fault that produces the postmortem's "one peer flaky, others clean" symptom.

Mutation axes:

1. Symmetry: loss on outbound from victim, on inbound to victim, on both directions, none (control).
2. Burst shape: continuous low-rate loss vs short high-rate burst.
3. Co-occurrence: loss alone vs loss + clock skew on the same peer (compound — per SIM_HARDENING_SPEC §7).

Required assertions:

• The bundle's UDP echo probe records must show the victim's outcome distribution (ok / timeout / refused / unresolved / error) differing from the other peers' by a margin evident to a human reader.
• Across the run, the victim's Tier3InterfaceCounters.rx_packets_dropped or Tier3UdpKernelStats.in_errors is non-zero in the bundle, while the other peers' is zero. This is the "kernel saw the loss, not just iroh" contract gap 11 demanded.

Property test: required, seeds 0..128. Range over axes 1 and 2. The property: for every seed in which the victim's UDP echo shows >5% loss, the bundle must surface a non-zero kernel-counter delta on the same peer. (This is the discriminator gap 11 asked for.)

Expected verdict on current source: Mixed. The observability upgrade landed kernel counters in the bundle (S-A4); the simulator's stage host needs to emit Tier3InterfaceCounters under the loss-burst mutation for the discriminator to hold. If it does not, that is a sim-coverage gap belonging in SIM_BLIND_SPOTS.md per SIM_HARDENING_SPEC §10, not a reason to relax the assertion.

Family closes when: the property test runs to 128 seeds with the loss-discriminator holding on every seed it sees loss; the sim-coverage gap, if it exists, is filed.

Family E — Bundle integrity under operator SIGKILL

Source: N3_POSTMORTEM_2026-05-25.md "Bundle recovery"; N3_DATA_GAPS.md gap 7; observability upgrade S-D (bundle without finalize).

Shape: The orchestrator is killed ungracefully (SIGKILL via TaskStop, not graceful shutdown). No finalize record is written. The diagnostic bundle must still be assemblable from staging files on disk, with manifest.finalize_received: false.

Central case (central.toml): a PeerKill { peer: orch, at_ns: 60_000_000_000 } mutation 60 s into the run. No PeerResurrect. The scenario's duration_ns extends 30 s past the kill so the collector has time to observe and the bundle has time to coalesce.

Mutation axes:

1. Timing of kill: during convergence, during steady state, during a partition heal.
2. Which peer: orchestrator, a stage, the relay.

Required assertions:

• The bundle's manifest.json must exist and contain finalize_received: false.
• Every peer's pre-kill events and snapshots must be present in the bundle (the kill must not erase prior records).
• The verdicts.json must contain a verdict for every declared assertion, with Inconclusive for any assertion whose preconditions did not fire (e.g., a steady-state assertion when steady state was never reached).

Property test: not required.

Expected verdict on current source: Pass. The observability upgrade landed S-D (bundle assembly without finalize). This family guards that contract against regression.

Family closes when: every scenario in the family produces a parseable bundle whose summary.md renders cleanly through swactor-diag-postproc.

Family F — Compound faults under recovery

Source: SIM_HARDENING_SPEC §7 and §9.

Shape: Two or more faults active during a single recovery window — a partition heal during a relay-peer-down, a clock skew during a worker respawn, a kernel UDP overflow during SWIM gossip burst. The 2026-05-25 incident is consistent with at least two overlapping faults (relay-peer-down + silent-worker); the battery must cover the next overlap before it lands in prod.

Central case (central.toml): a Partition cutting stage-2 from stage-0 from t=10 s to t=30 s; a RelayPeerConnDown { from: orch, to: stage-2, at_ns: 20_000_000_000, duration_ns: 20_000_000_000 } overlapping the partition's last 10 s and extending 10 s past its heal. The scenario tests whether SWIM behaves under the overlap and the heal sequence the postmortem mentions but did not isolate.

Mutation axes:

1. Which two faults overlap (cross product of the four single-fault families above, restricted to combinations that produce distinguishable bundles).
2. Overlap geometry: full overlap, partial overlap, abutting (one ends as the other begins).
3. Recovery phase: which recovery phase the second fault hits, per SIM_HARDENING_SPEC §9.

Required assertions: family-dependent — each compound test combines the assertions of its constituent families. The compound test passes only if every constituent assertion holds.

Property test: required, seeds 0..512. Range over all three axes. The property: any seed in which a compound bundle violates more assertions than the sum of the constituents' individual violations is a true compound bug, reported separately.

Expected verdict on current source: Mixed. Compound failures are the under-tested corner; the implementing agent should expect to find at least one new sim-coverage gap during this family's implementation and file it.

Family closes when: at least one compound bug is either fixed or filed as a sim-coverage gap with a structural reason.

---

4. Out of scope

• Tuning the simulator's existing scenarios under scenarios/calibration/ or scenarios/smoke/.
• Adding new failure shapes the 2026-05-25 deployment did not surface (the diagnostic deployment running in parallel may; if so, those land as a new spec, not as an amendment to this one).
• Changes to the simulator's engine, network, host kinds, bundle writer, or post-processor. The battery exercises them; it does not modify them.
• Changes to the production diagnostics code path. The observability upgrade landed; the battery consumes its output.
• Documentation of the simulator beyond SIM_BLIND_SPOTS.md amendments. SIM_HARDENING_SPEC.md and SIM_SPEC.md already exist; this document is the only new prose required.

---

5. References

• N3_POSTMORTEM_2026-05-25.md — source for families A, B, C, D, E.
• N3_DATA_GAPS.md — source for the gap-named contracts each family asserts the simulator's bundle must satisfy.
• N3_DEPLOYMENT_REPORT.md — historical context: Layers A/B/C from the prior eight deploys.
• N3_OBSERVABILITY_UPGRADE_SPEC.md — the contract the bundle already satisfies. The battery consumes that contract.
• SIM_HARDENING_SPEC.md — the family / mutation-axis discipline that §1 and §3 above enforce.
• crates/simulation/SIM_SPEC.md — the simulator's behavioral surface. §3.1 components, §5.5 mutations, §6A stage host, §10.1 assertion catalog, §8 scenario format are the load-bearing references.
• .loop/notes.md, .loop/verdict.md — the observability-upgrade iteration log and verdict, current as of 2026-05-25; STATUS: DONE, VERDICT: PASS.