zacheryasc/swactor
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
swactor/crates/simulation/SIM_SPEC.md

70 KiB
Raw Blame History

SIM_SPEC — simulator MVP

Status: draft, pre-implementation. Lives in examples/pipeline-parallel-inference/ so it is not touched by the ongoing simplification of crates/simulation. It moves alongside the implementation once that cleanup lands.

This spec describes an architecture and an implementation strategy concretely enough that two independent implementers, working from this document alone, produce code that meshes. To that end it names components, the data that crosses every boundary between them, and the behaviour each component owes the others. Internal data structures, module layout, and helper types are the implementer's call; the named boundaries are not.

The MVP's first consumer is SWIM, because SWIM is the algorithm whose production failures motivated the simulator. The architecture is not SWIM- specific: the engine knows about hosts, not about SWIM. A host is a piece of code that consumes ticks and inbound messages and emits actions. SWIM is the first such host kind; the second is whatever we need next.

0. Motivation

The pipeline-parallel-inference example failed eight live N≥3 vast.ai deploys. The session report (N3_DEPLOYMENT_REPORT.md, next to this file) decomposes the failure into three independent bugs stacked:

  • Layer A — relay-mediated head-of-line blocking. iroh 0.96's RelayMode::Default routed gossip through n0's canary relay, which buffered SWIM traffic for 187 seconds. A shared queue servicing multiple peers couples otherwise-independent traffic: a 9.8 KB Ack from one peer delays every probe behind it on the same egress.
  • Layer B — SWIM gossip-flap (B1). In a seven-minute run, the orchestrator refuted Suspect claims against itself 228 times — roughly once every 1.8 seconds — and one peer ended the run marked Dead despite probes succeeding in both directions on both sides of the link. The bug only appears with three or more peers, multi- region latency, and enough cumulative gossip state for piggybacked membership updates to grow into the multi-kilobyte range.
  • Layer C — internal-cause peer death. Pipeline stages died 51–191 seconds into run #2 from worker-process exits, not from network. The orchestrator recorded "connect timeout" because the peer was gone. The failure mode is internal: the host emits a diagnostic record and then halts of its own accord.

Catching any of these in production costs about two dollars of GPU rental per attempt, forty-five to ninety minutes of engineer time per iteration, and produces one non-reproducible bundle of evidence per run. The same source, run twice, produces different outcomes.

The simulator exists to make the iteration loop sub-second and the outcomes byte-identical for a fixed seed. It is not a complete model of production; it is the smallest model that lets us reproduce A, B, and C deterministically and tune SWIM (and the relay, and the stage lifecycle) without deploying. Future algorithms layer onto the same engine without changing the behaviour this MVP guarantees.

1. What "done" means

The MVP ships when these are simultaneously true.

A property test reproduces the gossip-flap bug (Layer B) deterministically against the current SWIM source. The same scenario with the same seed produces byte-identical output across runs and across the architectures we claim to support.

A property test reproduces the relay-HOL flap (Layer A) deterministically against the current SWIM source: a multi-peer scenario routed through one relay with bounded egress, with cumulative gossip state growing into the multi-kilobyte range, exhibits probe timeouts that disappear when the relay's egress capacity is widened or the message-size cap from §10.1 is enforced.

A scenario reproduces the run-#2 stage-death failure mode (Layer C): a stage declared with a scheduled internal exit dies on schedule, emits a worker_exited record with the declared reason, halts, and is observed by the rest of the cluster through the same diagnostic channel production uses.

The fix workflow does not deploy. A developer writes a property, runs the sim, sees it fail, edits the SWIM source (or the relay policy, or the stage-supervisor lifecycle), re-runs, sees it pass — all locally, in under a second per iteration.

The simulator is calibration-grounded against the three N3 bundles we have (vastai-N3-1 canary relay + real worker; vastai-N3-2 own relay + real worker; vastai-N3-stub own relay + stub worker). Distributions emitted by the sim, when configured to mirror a given N3 run, are within declared tolerances of the corresponding live bundle.

Each known live failure mode has at least one scenario in the library, with a prose comment naming what it reproduces.

3. Architecture

3.1 Components

The simulator is one process holding six named components. Each component has one responsibility and one set of inbound and outbound message types. The boundaries between them are the contract two independent implementers must agree on; nothing inside a component is.

            scenario.toml
                 │
                 ▼
          ┌──────────────┐
          │ scenario     │   parses and validates the input
          │ loader       │
          └──────┬───────┘
                 │ parsed scenario
                 ▼
          ┌──────────────┐    send queries     ┌──────────────┐
          │              ├────────────────────►│   network    │
          │   engine     │◄────────────────────┤              │
          │              │   arrival times,    └──────────────┘
          │              │   drop reasons
          │              │
          │              │    ticks, recv       ┌──────────────┐
          │              ├────────────────────► │   hosts      │
          │              │◄──────────────────── │  (per peer)  │
          │              │    actions           └──────────────┘
          │              │
          │              │    events, snapshots ┌──────────────┐
          │              ├────────────────────► │   bundle     │
          │              │                      │   writer     │
          └──────────────┘                      └──────┬───────┘
                                                       │
                                                       ▼
                                              bundle on disk
                                                       │
                                                       ▼
                                              ┌──────────────┐
                                              │ assertion    │
                                              │ evaluator    │
                                              └──────┬───────┘
                                                     │
                                                     ▼
                                              verdicts on disk

Scenario loader. Reads a TOML scenario, validates it, returns a parsed scenario value (§8). No state; pure function from path to validated scenario.

Engine. Owns the virtual clock, the scheduling queue, the table of hosts, references to the network and the bundle writer. Single entry point: given a parsed scenario, run to completion. §4 specifies behaviour.

Network. A directed-graph link model. Answers send queries deterministically and accepts mutations on a timeline. Holds no schedule of its own; the engine pops events, the network answers questions. §5 specifies behaviour. The network also owns the relay vertices (§5A): a relay is a first-class vertex of the network graph that is not a host. It has one ingress queue and one egress queue per outbound link, drains them at policy-declared capacities, and applies head-of-line ordering. Hosts do not see whether their traffic was direct or relayed; the relay is opaque to the host layer.

Host. An instance of some host kind, one per peer in the scenario. The host kinds for the MVP are two: the production SWIM state machine wrapped in a thin adapter (§6.2) and the pipeline-stage host kind (§6A) that wraps the production stage supervisor lifecycle and emits the diagnostic records the N3 deployment report's C.1–C.3 findings require. The host trait — what the engine calls and what the host returns — is §6.

Bundle writer. The only filesystem-touching component. Receives event and snapshot records from the engine, serializes them to the production diagnostics schema, writes them to a bundle directory. §9 specifies output.

Assertion evaluator. Post-run reader of the bundle. Evaluates each declared assertion against the event stream and snapshot directory. Emits a verdict file. §10 specifies behaviour.

3.2 Data flow

Each boundary is named below with the data that crosses it.

scenario.toml → scenario loader. A TOML file. §8 names the schema.

scenario loader → engine. A validated parsed scenario value. The fields are exactly those §8 names; the engine consumes nothing else from outside.

engine ⇄ network. Two query methods, no others.

  • send(from, to, byte_len, sent_at_ns) → SendOutcome. SendOutcome is either Arrive { at_ns } or Drop { reason }. The network mutates its per-link state (last-send-time, warm/cold) as it answers.
  • apply_mutation(mutation, at_ns) → Vec<InvalidatedDelivery>. The network updates its internal state and returns a list of currently-scheduled deliveries the mutation invalidates. The engine removes those from its queue.

engine ⇄ host. Three call sites, no others. The host responds to each with Vec<Action>; the engine processes actions in returned order.

  • tick(now_ns) — fired at the host's tick instants.
  • recv(message, now_ns) — fired when a delivery event for this host pops.
  • snapshot() → SnapshotBytes — fired on snapshot dispatch.

Action is a closed enum the engine handles exhaustively:

  • Send { to: HostId, message: HostMessage } — engine asks the codec for the message's byte length, asks the network for arrival time, schedules a Deliver event or notifies the sender of SendFailed (next bullet).
  • RecordEvent { event: EventBytes } — engine forwards to the bundle writer with the current virtual time.
  • ScheduleTimer { at_ns, token: TimerToken } — engine schedules a TimerFired event delivered through recv at at_ns.
  • Halt — engine stops issuing further ticks to this host. Inbound recv still flows (so the host can observe drains) until the host's recv itself returns Halt.

The engine never invents actions; the host produces them. The engine never silently drops actions; an unknown variant aborts the run.

engine → bundle writer. A single method, write_record(record). The record is one of:

  • EventRecord { virtual_time_ns, host_id, kind_tag, event_bytes } — produced by host RecordEvent actions and by engine-synthesized events (drops, cold-dial penalty firings, cache-invalidate firings, send-failure notifications).
  • SnapshotRecord { virtual_time_ns, host_id, snapshot_bytes } — produced by snapshot dispatch.
  • MutationRecord { virtual_time_ns, mutation } — produced when a mutation pops.

The bundle writer is append-only and accepts records in arbitrary order. It is the writer's responsibility to organize records into the bundle layout in §9.

bundle → assertion evaluator. The evaluator reads the finished bundle from disk. Its input is the bundle path; its output is a verdicts.json written into the same bundle. §10 names the verdict format.

3.3 Host kinds and the codec contract

A host kind is a (host_trait_impl, codec) pair. The codec exists because the bandwidth model needs to know how many bytes a host's outgoing message will occupy on the wire, and that number must equal what the production transport would put on the wire for the same message, or the simulator's bandwidth-driven failure modes diverge from the live ones.

For each host kind, the codec exposes:

  • encode(message) → bytes.
  • decode(bytes) → message.
  • kind_tag() → str.

A contract test asserts byte-equality between the sim's encode path and the production transport's encode path for a representative message set; divergence breaks the build.

For the MVP, the SWIM host kind reuses the production transport's encoding logic directly. If production code is too tangled with iroh to import cleanly, the encoding is factored into a small shared module that both the production transport and the sim host call; that refactor is part of the MVP, not deferred.

3.4 Independent buildability

Each component in §3.1 can be built by an independent agent against the contracts in §3.2 and §3.3 alone. Specifically:

  • The scenario loader is built against §8.
  • The network is built against §5 and the substream rule in §7.
  • The engine is built against §4 plus the network and host call signatures.
  • The SWIM host is built against §6 and the production SWIM API.
  • The bundle writer is built against §9.
  • The assertion evaluator is built against §10.

A change to a component's internal structure is invisible to the others. A change to a contract in §3.2 / §3.3 is a spec amendment.

4. The engine

The engine owns the virtual clock, the scheduling queue, the host table, the network reference, and the bundle-writer reference.

4.1 The scheduling queue

The queue is a priority queue over (virtual_time_ns, sequence_number) keys. The sequence number is assigned monotonically on enqueue; it is the only tie-break mechanism the engine permits. Two events at the same virtual time pop in enqueue order.

Each entry carries one of:

  • Tick { host_id }.
  • Deliver { host_id, message_bytes }.
  • TimerFired { host_id, token }.
  • Mutation { mutation }.
  • Snapshot.
  • Terminate.

Pre-population: at engine start, one Tick is enqueued per host at the host's first-tick virtual time, one Mutation per scenario mutation, one Snapshot per scenario snapshot request, and one Terminate at duration_ns.

4.2 The main loop

Pop the smallest entry. Advance the virtual clock to its time. Dispatch by kind (§4.3 through §4.7). Repeat until Terminate pops or the early-termination condition fires (§4.8). After termination, finalize the bundle and run the assertion evaluator.

The clock advances only on pop. Nothing in the engine reads any other clock, virtual or real.

4.3 Tick dispatch

For Tick { host_id }:

  1. Call host[host_id].tick(now_ns). If the host has Halt-ed, skip the call but still schedule the next tick (the host may un-halt only via a PeerResurrect mutation).
  2. Process the returned action list in order (§4.6).
  3. Enqueue the next Tick { host_id } at now_ns + tick_period_ns[host_id].

The first tick for each host is offset by a stable, seed-derived per-host offset. The offset is drawn from the host's RNG substream (§7) so peers do not tick on the same virtual instants and silent symmetry artefacts do not mask real timing bugs.

4.4 Delivery dispatch

For Deliver { host_id, message_bytes }:

  1. If host[host_id] is halted or killed, drop the delivery and emit a DropOnDelivery engine-synthesized event.
  2. Otherwise, decode the bytes via the host kind's codec.
  3. Call host[host_id].recv(message, now_ns).
  4. Process the returned action list in order (§4.6).

4.5 Other dispatches

  • TimerFired { host_id, token } is delivered through recv with a TimerFired(token) envelope and processed identically to a network delivery.
  • Mutation { mutation } is forwarded to network.apply_mutation. The returned list of invalidated deliveries is removed from the queue (or tombstoned — the visible behaviour is identical). The engine emits a MutationRecord to the bundle writer.
  • Snapshot asks every live host for its snapshot() and forwards each result to the bundle writer as a SnapshotRecord.
  • Terminate ends the main loop.

4.6 Action processing

For each action returned by a host:

  • Send { to, message } — encode via the codec, ask the network with the resulting byte length. On Arrive, enqueue a Deliver at the returned time. On Drop, emit a DropOnSend engine-synthesized event and deliver a SendFailed envelope to the sender via the same recv path the production transport would.
  • RecordEvent { event } — forward to the bundle writer with current virtual time.
  • ScheduleTimer { at_ns, token } — enqueue TimerFired at at_ns.
  • Halt — mark the host halted (§4.3 covers re-entry).

4.7 Engine-synthesized events

The engine emits records for behaviours hosts do not see directly:

  • DropOnSend { from, to, reason } — a send the network refused.
  • DropOnDelivery { to, reason } — a delivery the engine refused at arrival time (host killed mid-flight).
  • CacheStateChange { from, to, transition } — the link warmed, the link went cold by idle, the link was invalidated by a mutation. The network surfaces these to the engine through a side-channel on send and apply_mutation.
  • DialStart / DialOutcome — emitted whenever the cold-dial penalty fires on a send.

These records exist so the bundle's observable surface matches what production diagnostics emit for the same activity. The engine never suppresses them and never emits them for activity that did not happen.

4.8 Early termination

A scenario may set early_terminate_on_all_assertions_resolved = true. If set, after each event dispatch the engine polls the assertion evaluator's streaming side (§10.4); if every declared assertion has a resolved verdict, the engine fast-forwards to Terminate. The bundle still records every event that fired up to that point.

4.9 Time unit

The engine's virtual clock is in integer nanoseconds. The manifest records the unit so post-processors render times consistently.

4.10 Behavioral tests

The engine's contract is the dispatch and action-processing behaviour of §4. Its tests assert that it has the properties below; how each is verified is the test author's call.

Tick cadence. Each host receives ticks at its declared period, starting at a stable seed-derived offset. The offset is identical across runs and differs across hosts in the same scenario.

Action ordering. A host's emitted actions are processed in returned order. The effects of action N are fully observable before action N+1's effects begin.

Send semantics. A Send whose network query returns Arrive results in the recipient's recv being called at the returned arrival time with the codec-produced bytes. A Send the network drops results in the sender's recv being called with SendFailed, a DropOnSend record in the bundle, and no recipient call.

Timer fidelity. A ScheduleTimer { at_ns, token } causes a TimerFired(token) envelope to reach the host's recv at exactly at_ns.

Halt. A halted host receives no further ticks; it continues to receive deliveries.

Closed action set. An action outside the closed set §4.6 names aborts the run with a structured error. The engine never silently ignores or invents an action.

Tie-break. Events scheduled at the same virtual time pop in enqueue order. The order is identical across runs and architectures.

Mutation propagation. Deliveries the network invalidates are removed from the engine's queue; each emits a DropOnDelivery record at the mutation's virtual time. No invalidated delivery reaches a host's recv.

Snapshot fanout. A scheduled snapshot produces exactly one record per live host at the scheduled virtual time.

Early termination is clean. Every record emitted before the termination time is preserved; no record carries a later virtual time.

Determinism. Same scenario, same seed ⇒ byte-identical events.ndjson.

5. The network

5.1 Topology

A directed graph. Vertices are the host IDs declared in the scenario. Edges carry link policies. An ordered pair with no declared edge is permanently partitioned; the network returns Drop(NoRoute) for any send on it. This is distinct from a temporary partition mutation, which can heal.

Asymmetry is allowed and intended: policy(A→B) and policy(B→A) are independent.

5.2 Link policy

Each edge carries the following integer-valued fields:

  • latency_ns — base one-way delivery time.
  • jitter_stddev_ns — symmetric jitter; samples are drawn from a precomputed integer lookup table approximating a standard-normal distribution scaled by this stddev (§7 forbids float math in decisions).
  • loss_prob_ppm — independent drop probability per send, parts-per- million.
  • reorder_prob_ppm — probability of inserting extra delay sufficient to swap delivery order with the next message on the same edge.
  • bandwidth_bps — bytes per second. A message of N bytes occupies the link for (N * 1_000_000_000) / bandwidth_bps ns.
  • cold_dial_penalty_ns — extra latency added when the link is cold.
  • cache_warm_after_ns — wall of warm-time after first contact before subsequent sends are warm.
  • cache_invalidate_after_idle_ns — idle duration after which the link returns to cold.

5.3 Per-link state

Each edge tracks:

  • last_send_ns — virtual time of the last send that returned Arrive.
  • last_arrive_ns — virtual time of the latest scheduled arrival; used by the bandwidth model for the next message's serialization start.
  • cache_state — Cold, Warming(since_ns), or Warm.

State transitions happen inside send and apply_mutation.

5.4 The send algorithm

send(from, to, byte_len, sent_at_ns) → SendOutcome. Steps:

  1. If the edge does not exist, return Drop(NoRoute).
  2. If the active partition set cuts (from, to), return Drop(Partitioned).
  3. If a LossBurst mutation is active for this edge, use its override probability; otherwise use the edge's loss_prob_ppm. Draw a u32 from the edge's RNG substream; if draw % 1_000_000 < prob_ppm, return Drop(Lossy).
  4. Compute serialization_start = max(sent_at_ns, last_arrive_ns). Compute serialization_end = serialization_start + (byte_len * 1e9 / bandwidth_bps).
  5. Compute arrival = serialization_end + latency_ns + jitter_sample, where jitter_sample is one draw from the per-link substream into the integer-Gaussian table, scaled by jitter_stddev_ns.
  6. If a LatencySpike mutation is active, multiply the additive latency contribution (latency + jitter) by the spike factor before adding.
  7. If a RelayBuffer mutation is active, take arrival = max(arrival, sent_at_ns + floor_ns).
  8. If cache_state is Cold, add cold_dial_penalty_ns to arrival and transition cache_state to Warming(now). Emit a DialStart side- channel notification to the engine and a DialOutcome at the arrival time.
  9. If the reorder draw fires, add enough delay so this message arrives after the next message scheduled on this edge.
  10. Update last_send_ns = sent_at_ns, last_arrive_ns = arrival. If cache_state is Warming(since) and now - since >= cache_warm_after_ns, transition to Warm and emit a CacheStateChange.
  11. Return Arrive(arrival).

If last_send_ns - now > cache_invalidate_after_idle_ns at the start of a send, the link returns to Cold and emits a CacheStateChange before proceeding.

5.5 Mutations

Supported kinds:

  • Partition { peers_a, peers_b } — set the active partition to cut every edge between the two groups in both directions. In-flight messages on cut edges are returned in the invalidated-deliveries list.
  • Heal — clear the active partition set. No in-flight invalidations.
  • LatencySpike { links, factor_x100, duration_ns } — multiply additive latency on named links by factor_x100 / 100 for a duration. In-flight messages are not retroactively delayed.
  • LossBurst { links, prob_ppm, duration_ns } — override loss probability on named links for a duration.
  • RelayBuffer { links, floor_ns, duration_ns } — impose a minimum delivery delay on named links for a duration. (Legacy per-edge floor; the relay vertex of §5A models shared-queue HOL more faithfully.)
  • PeerKill { peer } — drop the peer's inbox. In-flight deliveries to the peer are invalidated. The peer's ticks are stopped by the engine.
  • PeerResurrect { peer, preserve_state } — restart the peer. If preserve_state, the engine reuses the host instance; otherwise a fresh host of the same kind is instantiated from the scenario's peer declaration.
  • WorkerExit { peer, reason, status_code?, signal? } — delivers a WorkerExit envelope to the named peer at the mutation's time via the same recv path that TimerFired and SendFailed use today. The host kind decides what to do with it; the stage host (§6A) emits a worker_exited event and returns Halt. Targeting a host kind that does not accept the envelope (e.g. the SWIM kind) aborts the run with a structured error — silent fallback is the bug class the simulator exists to prevent.
  • RelayKill { relay } — the relay stops forwarding. All currently queued messages are returned in the invalidated-deliveries list. Subsequent sends through the relay drop with reason RelayDown.
  • RelayBoot { relay } — the relay returns to service. Its queues are empty; the next forwarded message pays the cold_start_penalty_ns.
  • RelayCapacityChange { relay, ingress_capacity_bps?, egress_capacity_bps_per_link?, queue_depth_bytes? } — at the named time, replace any subset of the three relay policy fields (§5A.2). In-flight messages already past ingress complete at their previously- computed arrival times; messages that arrive after the mutation use the new policy.

5.6 Determinism within the network

Every random draw the network makes comes from a substream keyed by ("link", from_id, to_id) (§7). Editing one link's policy must not perturb the draws on any other link.

5.7 Behavioral tests

The network is a pure function of (state, query). Its tests assert that it has the properties §5 names; how each property is verified is the test author's call.

Reachability. A send over an ordered pair returns Arrive iff that pair is declared as an edge and the active partition set does not cut it. A send the network refuses leaves the network's state unchanged.

Partition heals to identity. A Partition followed by a Heal at later virtual times leaves the network indistinguishable on subsequent sends from one that experienced neither.

Mutation invalidation is exact. Every delivery a mutation renders impossible appears in the mutation's invalidated-deliveries return. No delivery the mutation does not invalidate appears in that return.

Loss is Bernoulli. Drops on a link are independent draws with the link's declared probability. A LossBurst substitutes its override probability for the duration it names and only the duration it names.

Bandwidth serializes. A link with finite bandwidth never overlaps two messages' wire-occupancy intervals: each message's arrival is delayed at least until the previous message's arrival plus that message's transmission time.

Latency is additive. A send's arrival decomposes into base latency, serialization delay, jitter, cold-dial penalty when applicable, and the contributions of active mutations. The terms are independent in the policy and combine without interaction beyond what §5.4 specifies.

Jitter is symmetric and integer-valued. Jitter samples come from the precomputed table of §7, are symmetric around zero, and are never non-integer.

Cache state follows traffic. The link's cache_state reflects recent traffic: warm after sufficient activity, cold again after sufficient idleness, with the thresholds the policy names. The cold-dial penalty is paid by exactly the sends the network classifies cold.

Cache transitions are observable. Every cold↔warm transition emits exactly one CacheStateChange notification at the transition's virtual time. No transition is silent and no notification fires without a transition.

Mutation scoping. A mutation affects exactly the links its links field names and exactly the duration it declares. Sends on other links, or on the named links outside the duration, are unaffected.

Substream isolation. Editing one link's policy does not change any draw the network makes on any other link. This is the property that makes bisecting scenario edits possible.

Determinism. Same topology, same seed, same query sequence ⇒ identical SendOutcome sequence and identical invalidated-deliveries returns.

5A. The relay vertex

A relay is a vertex of the network's directed graph that is not a host. Relay IDs share the ID namespace with host IDs (§5.1 and §8 of this spec together require uniqueness across both sets). An edge's from and to may name either.

A route between two hosts is either direct — exactly one edge from A to B — or relayed — an edge A→R, the relay R, and an edge R→B. Multi-hop relayed routes (A→R₁→R₂→B) are out of scope; the scenario loader rejects them in §8.2.

If a scenario declares both a direct A→B edge and a relayed A→R→B route, the scenario loader rejects the ambiguity. Each ordered host pair has at most one route.

5A.2 Relay policy

Each relay declares the following integer-valued fields:

  • ingress_capacity_bps — maximum bytes per second the relay accepts across all inbound links combined.
  • egress_capacity_bps_per_link — maximum bytes per second the relay serves per outbound link.
  • queue_depth_bytes — maximum bytes buffered across all egress queues combined. A message that would push the total beyond this limit at enqueue time is dropped.
  • queue_discipline — Fifo is the only value the MVP accepts.
  • cold_start_penalty_ns — extra latency added to the first message the relay forwards after a RelayBoot mutation.

A relay has no jitter, loss, or cache-state fields of its own; the edges feeding it carry their own such fields per §5.2. A relay's drop reasons are exclusively queue-overflow and RelayDown; lossy drops remain a property of edges.

5A.3 The forward algorithm

When the network receives send(from, to, byte_len, sent_at_ns) and the configured route is relayed through R, the composition is:

  1. Resolve the inbound edge from → R. Apply §5.4 for the inbound leg. If that leg drops, the composed send drops with the same reason; the relay's state is not consulted.
  2. At time arrival_at_R, attempt to enqueue at the relay. If enqueued_bytes + byte_len > queue_depth_bytes, the composed send drops with reason RelayQueueFull and the relay emits a RelayDrop record (§9.1).
  3. Otherwise, compute the ingress serialization end: max(arrival_at_R, ingress_queue_tail_ns) + (byte_len * 1_000_000_000) / ingress_capacity_bps.
  4. Compute the egress serialization start on the outbound link to to: max(ingress_end, egress_queue_tail_ns[to]). If the relay is Booting, add cold_start_penalty_ns and transition to Booted. Compute egress_end = egress_start + (byte_len * 1_000_000_000) / egress_capacity_bps_per_link.
  5. Resolve the outbound edge R → to per §5.4 with sent_at_ns = egress_end. Its arrival is the composed send's arrival.
  6. The relay emits a RelayEnqueue at arrival_at_R and a RelayDequeue at egress_end (§9.1).

The composed send returns one Arrive(at_ns) or one Drop(reason); the relay's internal events are surfaced through the side channel that §3.2 already names for cache state changes.

A relay's egress and ingress are independent: the ingress can be serializing a new message while the egress is still draining an old one. Head-of-line blocking arises only when two messages share an egress link (or the single ingress).

5A.4 Determinism within the relay

The relay draws no randomness in the MVP. Iteration over per-egress- link state inside a relay is by destination host ID in lexicographic order, per §7.3.

5A.5 Behavioral tests

The relay's tests assert that it has the properties below; how each is verified is the test author's call. (Tests live in tests/relay_invariants.rs.)

  • Composition is transparent to hosts. A send through a relayed route returns one SendOutcome shaped identically to a direct send's. The receiving host cannot distinguish a relayed delivery from a direct one by the message it sees.
  • HOL is observable and bounded. Two messages sharing an egress arrive in send order, separated by at least the first message's egress serialization time.
  • Ingress and egress are independent. A message destined for peer X does not delay a message destined for peer Y on the egress side.
  • Queue overflow is exact. A send that would push enqueued_bytes strictly above queue_depth_bytes at its arrival-at-relay time drops with RelayQueueFull and emits a RelayDrop record. A send that exactly fills the queue is accepted.
  • Cold-start penalty is paid once per boot. After a RelayBoot, the first forwarded message includes the cold-start penalty; the second does not.
  • Mutation invalidation is exact. A RelayKill returns exactly the deliveries in-flight through the relay at the mutation's virtual time and no others. A RelayCapacityChange invalidates no deliveries.
  • Ambiguous routes are rejected statically. A scenario declaring both a direct edge A→B and a relayed route A→R→B, or two relayed routes for the same ordered host pair, is rejected by the loader.
  • No multi-hop in MVP. A scenario declaring a route that traverses two relays is rejected by the loader.
  • Determinism. Same topology, same seed, same query sequence ⇒ identical composed SendOutcome sequence and identical RelayEnqueue / RelayDequeue / RelayDrop record streams.

6. Hosting an entity

6.1 The host trait

A host kind implements:

  • fn id(&self) -> HostId.
  • fn kind_tag() -> &'static str. Used for routing and bundle tagging.
  • fn tick(&mut self, now_ns: u64) -> Vec<Action>.
  • fn recv(&mut self, message: HostMessage, now_ns: u64) -> Vec<Action>.
  • fn snapshot(&self) -> SnapshotBytes.

HostMessage is either a decoded inbound application message (the host kind's own type, dispatched via the codec) or a TimerFired(token) or a SendFailed { to, reason } envelope.

A host kind also exposes:

  • fn new_from_config(id: HostId, config: HostKindConfig, rng: HostRng) -> Self.
  • A codec (§3.3).
  • A validation routine for its HostKindConfig (used by the scenario loader; §8).

6.2 The SWIM host kind

The SWIM host wraps the production SWIM state machine without re- implementing it. It:

  • Constructs the production state machine with the scenario's per-peer config.
  • Installs the production diagnostics emitter against a shim that pushes every emission into a per-tick / per-recv Vec<Action> as RecordEvent actions.
  • Installs the production tier-2 introspector against the host's snapshot() method so the snapshot bytes are exactly what production emits.
  • Dispatches recv to the production handler matching the message kind (ping, ack, ping-request, indirect-ack, join-request, join-response).
  • Translates production state-machine output (outgoing messages, timer requests) into Send and ScheduleTimer actions.

When the production state machine emits an output the host adapter does not know how to route — a new message kind in a future SWIM version, for example — the adapter panics. Silent fallback is exactly the class of bug the simulator is meant to prevent.

The adapter does not substitute for any production logic. Its job is purely translation between production data types and the host trait.

6.3 Adding a new host kind

Adding a new host kind is a strict superset operation:

  1. Implement the host trait against the new algorithm.
  2. Provide a codec.
  3. Add a kind arm to the scenario loader's peer-declaration parser.
  4. Register the kind with the engine's host-instantiation factory.

Existing host kinds continue to work without change. The network, the bundle writer, the engine main loop, and the determinism contract are host-kind-agnostic.

6.4 Behavioral tests

Host-kind tests come in two layers: kind-agnostic properties every registered kind must satisfy, and per-kind properties specific to the algorithm a kind hosts. The list below is what the tests must assert; how is the test author's call.

Trait conformance (every kind). The kind exposes the §6.1 surface with the §6.1 signatures. kind_tag() is a non-empty string unique among registered kinds.

Codec is invertible (every kind). Encoding then decoding a message is the identity on the kind's message type.

Host determinism (every kind). Same HostKindConfig, same RNG seed, same tick/recv sequence ⇒ identical action sequence.

SWIM emits no novel kinds. Every event a SWIM host emits is of a kind production's diagnostics also emits. The simulator invents no SWIM event kind for itself.

SWIM codec parity with production. The SWIM host's encoding of an outgoing message is byte-identical to the production transport's encoding of the same message. Drift breaks the build.

SWIM snapshot parity with production. A SWIM host's snapshot() conforms to the production tier-2 SWIM-state schema.

SWIM unknown-output is loud. A production state-machine output the SWIM adapter does not route aborts the run with a structured error. Silent fallback is a test failure.

6A. The pipeline-stage host kind

The stage host wraps the production stage supervisor — the worker lifecycle implemented in examples/pipeline-parallel-inference and running today as pp-gpu-node. The wrap is structurally analogous to §6.2's SWIM host.

The MVP stage host does not engage in inter-stage application traffic (activation forwarding, KV-cache updates). The failures Layer C exhibits are lifecycle failures, not application-protocol failures; inter-stage traffic is a strict superset and is out of scope for this extension.

6A.2 Lifecycle

The stage host moves through four states, each transition emitting exactly one diagnostic record:

  • Cold — initial state on new_from_config. No actions until the first tick.
  • Registering — entered on the first tick. The host emits a register_name event (§9.1) with the host's declared name and address, then transitions to Running.
  • Running — steady state. The host emits no further state- transition records on its own.
  • Halted — entered on receipt of a WorkerExit { reason } envelope or after a PeerKill mutation. The host emits a worker_exited event (§9.1) with the reason, then returns Halt.

Transitions are linear: Cold → Registering → Running → Halted. There is no resurrection. A PeerResurrect mutation against a stage host produces a fresh instance from the scenario's peer declaration per §5.5; the resurrected instance starts in Cold.

6A.3 Internal-cause exit

A WorkerExit { reason } envelope arrives via the same recv path as TimerFired and SendFailed. The stage host's recv for that envelope returns exactly three actions, in order:

  1. RecordEvent carrying a worker_exited event whose payload names the reason verbatim.
  2. RecordEvent carrying a stage_lifecycle event from the current state to Halted.
  3. Halt.

The order is normative: the event must reach the bundle writer before the engine acts on the halt.

A WorkerExit mutation is dispatched synchronously — the engine calls the target host's recv and processes the returned actions inside the same dispatch_mutation call that records the mutation, before the main loop pops the next event. This is the only way to guarantee §6A.6's boundary case: a WorkerExit whose at_ns equals the scenario's duration_ns must still produce a worker_exited record. Routing the envelope through the queue (an enqueued LocalRecv) would race with the construction-time Terminate at the same virtual time and could silently lose the event — the exact failure mode the boundary clause forbids.

The SWIM host kind has no WorkerExit semantics. A WorkerExit mutation targeting a SWIM-kind peer aborts the run with a structured error (EngineAbort::WorkerExitOnWrongKind). Silent acceptance is the failure mode this simulator exists to prevent.

6A.4 Diagnostic surface

The stage host emits three event kinds in addition to anything the production stage supervisor already emits:

  • register_name { name, address, peer_node_id } — emitted exactly once per stage instance, at the transition Registering → Running.
  • worker_exited { reason, status_code, signal } — emitted exactly once per stage instance, immediately before Halt.
  • stage_lifecycle { from, to } — emitted on every state transition the §6A.2 lifecycle declares.

Each kind's payload schema is named here for the production schema to follow. The deployment report's items C.1 and C.3 name register_name and worker_exited respectively; this spec fixes their shapes so the sim and production cannot drift.

The stage host's snapshot() returns a JSON object with the fields:

  • state — one of the four §6A.2 lifecycle states.
  • name_registry — a map of name → address for every name this host has registered. (The deployment report's item C.2 names the absence of this field as a debugging gap; this spec requires it.)
  • last_exit_reason — present only when state == Halted.

6A.5 Codec

The stage host kind's codec exists for parity with §3.3. In the MVP its message-type is empty: the stage host produces no Send actions during its lifecycle. When inter-stage traffic enters scope in a later revision, the codec gains the production wire format.

6A.6 Behavioral tests

The stage host's tests assert that it has the properties below. (Tests live in tests/stage_host_invariants.rs.)

  • Trait conformance. kind_tag() returns "stage", distinct from every other registered kind's.
  • Lifecycle linearity. Every stage instance traverses the §6A.2 states in declared order, never revisits a state, and never skips one. Each transition emits exactly one stage_lifecycle event.
  • register_name exactly once. Across the lifetime of one instance, exactly one register_name event is emitted, at the Registering → Running transition.
  • worker_exited exactly once. Across the lifetime of one instance, exactly one worker_exited event is emitted, immediately before the Halt action that ends the instance. The event's reason field is byte-equal to the mutation's reason.
  • Event-before-halt is observable. A scenario whose duration_ns is the same nanosecond as a WorkerExit mutation's at_ns produces a bundle containing the worker_exited event.
  • WorkerExit against SWIM aborts. A WorkerExit mutation targeting a SWIM-kind peer aborts the run with a structured error (EngineAbort::WorkerExitOnWrongKind) naming the host's kind and the mutation's index.
  • Snapshot contains the registry. A stage host's snapshot() includes a name_registry field; every register_name event the host has emitted appears in the map at every subsequent snapshot.
  • Determinism. Same HostKindConfig, same RNG seed, same envelope sequence ⇒ identical action sequence and identical emitted event stream.

7. Determinism

This section is normative. A violation is a ship-blocker.

7.1 Forbidden inputs

No part of the simulator reads any of: host wall clock, host monotonic clock, host process or thread ID, host hostname, environment variables outside a documented sim-internal prefix, /dev/urandom or any host RNG source, network interface state, filesystem state outside the bundle output path.

The repo's existing lint scanner catches the static cases. Drift is caught by the cross-architecture parity test (§7.6).

7.2 The randomness tree

All randomness derives from one root random stream seeded by the scenario's seed field. Substreams are derived by hashing a fixed, documented tuple with a constant-key siphash:

  • ("link", from_id, to_id) — per-edge substream used by the network.
  • ("host", host_id, label) — per-host substream used for tick offsets and any RNG the host kind needs.
  • ("mutation", index) — per-mutation substream if a mutation needs randomness (none currently do).

The hash function and the key are fixed in code. Substreams are stable across runs and across host architectures.

Substream layout matters: editing one link's policy must not perturb the draws on any other link, or every test edit becomes a new random universe and bisection is impossible.

7.3 No hash-randomized iteration

Wherever any component iterates a collection, the order is determined by the natural key order (host IDs sort lexicographically; pairs sort lexicographically on the pair) or by an insertion-order-preserving structure. Point lookups into hash maps remain allowed; iteration is the divergence source.

7.4 No floating point in decisions

Latencies are integer nanoseconds. Probabilities are integer parts-per- million of a fixed denominator. Jitter samples come from a precomputed integer lookup table approximating a standard-normal distribution; the table is checked in. Bandwidth math uses integer arithmetic with explicit scaling; the precise formula is in §5.4.

Floating point is permitted in post-hoc calibration tools that read a bundle. It is forbidden in the engine, the network, the host adapter, the bundle writer, and the assertion evaluator's verdict computation.

7.5 No work outside the scheduler

No background thread, no async runtime, no timer that fires without the engine's knowledge. Every effect is the consequence of a popped event.

7.6 The cross-architecture parity test

The test suite contains a reference scenario whose bundle output has a known checksum, checked in. The test runs the simulator on the reference scenario on every supported architecture and compares the checksum to the stored value. A mismatch is either a deliberate spec amendment (with justification) or a bug.

8. The scenario format

A scenario is a single TOML file. The file is the simulator's only input and is byte-for-byte sufficient to reproduce any run. The format is TOML because the repo already uses it; the choice is not load-bearing.

8.1 Schema

Top-level fields:

  • name: String.
  • seed: u64.
  • duration_ns: u64.
  • early_terminate_on_all_assertions_resolved: bool (default false).

A [default_tick] table:

  • period_ns: u64.

A [default_link] table containing every field §5.2 names; per-edge overrides under [[links]] may override any subset.

A [[peers]] array, each entry:

  • id: String.
  • kind: String — selects the host kind. The MVP recognises "swim" and "stage"; the relay extension's stage kind_config is { name: String, address: String } (RELAY/STAGE §6A).
  • kind_config: { ... } — host-kind-specific opaque table.
  • initial_state: String — host-kind-specific. For stage, the only legal value is "cold" (mirrors §6A.2's Cold initial state).
  • tick_period_ns_override: u64 (optional).

A [[relays]] array (relay extension), each entry:

  • id: String — unique across the union of peer ids and relay ids.
  • ingress_capacity_bps: u64 — combined ingress bytes/sec.
  • egress_capacity_bps_per_link: u64 — per-outbound-link bytes/sec.
  • queue_depth_bytes: u64 — shared egress buffer bound.
  • cold_start_penalty_ns: u64 (default 0).

A [[links]] array, each entry:

  • from: String.
  • to: String.
  • via: String (optional) — the ID of a relay through which this edge is routed. When via is set, the edge's from and to must both be host IDs; the loader synthesizes the from → via and via → to inbound and outbound legs and registers a relayed HostRoute for the host pair. Multiple via shorthands through the same relay are allowed iff their leg policies agree.
  • Any subset of the §5.2 fields (overrides on top of [default_link]).

A [[mutations]] array, each entry:

  • at_ns: u64.
  • kind: String — one of the §5.5 variants.
  • Variant-specific parameters.

A [[snapshots]] array, each entry:

  • at_ns: u64.

A [[assertions]] array, each entry:

  • kind: String — one of the §10.1 variants.
  • Variant-specific parameters.

A [base] table with a single optional extends: String pointing to another scenario file; the merge is deep, with child entries overriding parent at the leaf.

8.2 Validation

The loader rejects:

  • Duplicate peer IDs.
  • Link, mutation, snapshot, or assertion references to undeclared peers.
  • duration_ns < max(mutation.at_ns) or similar for snapshots / assertions.
  • Any host-kind config that fails the host kind's own validation routine. For the SWIM kind, this includes probe_interval < suspicion_timeout.
  • A default_link field that is non-integer, negative, or in a unit other than the §5.2 names (e.g., latency_ms is rejected; only latency_ns).

Relay extension rules (added by RELAY/STAGE):

  • Duplicate IDs across the union of [[peers]] and [[relays]].
  • A via reference to a peer ID rather than a relay ID.
  • A peer pair declared with both a direct edge and a relayed route (the loader-level ambiguity check).
  • A [[links]] entry whose from or to is a relay and that also carries a via field.
  • A multi-hop relayed route (any direct relay→relay edge, which would make a chained route possible).
  • A worker_exit mutation whose target peer is not stage-kind.
  • A relay_capacity_change mutation with no fields set (the mutation must change at least one of the three policy fields).
  • A stage peer whose kind_config.name is missing or whose kind_config.address is missing or empty (delegated to StageHostKindValidator).

Validation failures produce a structured error with the file path, the offending field, and a one-line explanation.

8.3 Library structure

Scenarios live under a scenarios/ directory next to the simulator, in four subdirectories:

  • smoke/ — happy-path scenarios. Three-node mesh no impairments; eight-node ring; chain; star. Each asserts continuous Alive.
  • reproduction/ — known live failures. Each is expected to fail until its cause is fixed.
  • topology/ — partition-and-heal, rolling restart, peer churn. No specific bug targeted.
  • calibration/ — paired with a captured production bundle. §11.

Every scenario carries a top-of-file prose comment naming what it reproduces, the expected verdict (pass-now / fail-until-fix / sensitivity-study), and any base scenario it extends.

8.4 Behavioral tests

The loader's contract is the schema and validation rules of §8. Its tests assert that it has the properties below.

Examples are well-formed. Every shipped example scenario parses and satisfies every rule in §8.2.

Parse is invertible. Parsing, re-emitting to TOML, and re-parsing is the identity on scenario values.

Validation is complete. Every rule §8.2 names is enforced. A scenario violating any rule is rejected; a scenario violating none is accepted.

Errors are structured. A rejection names the file, the offending field, and the violated rule in one line each. Generic errors are a test failure.

Host-kind validation is delegated. A host-kind-config error surfaces the kind's own rule, not a loader-generic one.

Merge is leaves-override, lists-append. Extending a base scenario replaces leaf values and appends list entries, with no other effect.

Loading is pure. Loading the same file twice produces equal values and performs no filesystem writes.

9. The bundle

9.1 Layout

A bundle is a directory laid out as follows:

<bundle_root>/
  manifest.json
  scenario.toml          # echo of the scenario that produced the run
  events.ndjson          # newline-delimited JSON event stream
  snapshots/
    <host_id>/
      <snapshot_seq>.json
  verdicts.json          # produced by the assertion evaluator (§10)

9.2 The event stream

Each line is one JSON object with the envelope:

{
  "virtual_time_ns": <u64>,
  "host_id": <string or null>,        # null for engine-synth events not bound to a host
  "kind_tag": <string>,               # "swim", "engine", etc.
  "event": <event-payload>
}

The event payload schema is exactly the production diagnostics schema for that event kind. The simulator must not invent new event kinds; an event the simulator emits is one production also emits.

Event kinds the MVP emits:

  • SWIM state transitions, message-send and receive accounting, probe lifecycle (sent / received / timed out), self-incarnation bumps.
  • Engine-synthesized cache state changes, dial start and outcome, drop on send, drop on delivery, send-failure errors.
  • Mutation records.
  • Stage host (RELAY/STAGE §6A.4): register_name, worker_exited, stage_lifecycle. Each carries kind_tag = "stage" and the emitting host's id.
  • Relay subsystem (RELAY/STAGE §5A.3): RelayEnqueue, RelayDequeue, RelayDrop. Each carries kind_tag = "relay" and host_id = null (the relay is not a host).

The stage host's three kinds are introduced ahead of production on the basis of the deployment report's items C.1–C.3. The production stage supervisor must adopt the same schemas as part of landing this spec; otherwise the contract test against the production diagnostics emitter fails and the sim cannot reproduce Layer C without divergence.

9.3 Snapshots

A snapshot is exactly the production tier-2 SWIM-state JSON for that host at that virtual time. The schema is unchanged from production.

9.4 The manifest

manifest.json records:

  • Simulator version (commit hash).
  • Path and SHA-256 of the scenario file.
  • Seed.
  • Duration in ns.
  • Host architecture the run executed on.
  • SHA-256 of events.ndjson.
  • SHA-256 of each snapshot file, keyed by relative path.

Wall-time fields are derived from the virtual clock; the manifest declares the unit so post-processors do not confuse virtual time with real time.

9.5 Renderability

The bundle is renderable by the same post-processor production uses. If the renderer requires inputs the simulator does not have (collector-side receive timestamps, for instance), the simulator substitutes the virtual- clock equivalent and records the substitution in the manifest.

9.6 Behavioral tests

The writer's contract is the layout and schema of §9. Its tests assert that it has the properties below.

Layout. Every produced bundle has the §9.1 entries. (verdicts.json is the assertion evaluator's responsibility; §10.5.)

Envelope conformance. Every line of events.ndjson is valid JSON conforming to the §9.2 envelope shape.

Hash integrity. Every manifest-recorded hash equals the actual hash of the file it names.

Snapshot organization. Each SnapshotRecord corresponds to exactly one file at snapshots/<host_id>/<snapshot_seq>.json. Sequence numbers are monotonically increasing per host from zero.

Ordering is deterministic and documented. Identical record streams produce byte-identical bundles. The ordering rule is named in §9 and the writer obeys it.

Arrival-order independence. Records may arrive in any order; the produced bundle depends only on the multiset of records and the documented ordering rule, not on arrival order.

Idempotency. Writing the same record stream to a fresh output path twice produces byte-identical bundles.

10. Assertions

10.1 The assertion catalog

Each assertion kind has a name and a parameter shape. The MVP catalog:

  • all_alive_at { at_ns, peers } — at the given virtual time, every named peer's view of every other named peer is Alive.
  • all_alive_throughout { window_start_ns, window_end_ns, peers } — the above, continuously, across a window.
  • convergence_after { after_ns, within_ns, peers } — after the named time, the cluster reaches a consistent membership view within the bounded duration.
  • no_flap_while_probes_ok { peer, window_start_ns, window_end_ns } — no peer transitions Suspect → Alive → Suspect within a window in which the peer's bidirectional probes are succeeding.
  • no_dead_when_probes_ok { peer, window_start_ns, window_end_ns } — the peer is never marked Dead in any other peer's view while bidirectional probes are succeeding.
  • self_incarnation_bounded { peer, max_value } — the peer's self- incarnation counter never exceeds the bound.
  • message_size_bounded { kind, max_bytes } — no sent message of the named kind exceeds the byte threshold.
  • dead_peer_resurrects_within { peer, after_ns, within_ns } — after the peer becomes reachable again, the cluster marks it Alive within a duration.
  • event_count { kind, max } — bounds the absolute count of an event kind across the run.
  • event_rate { kind, window_ns, max_per_window } — bounds the rate of an event kind.
  • relay_queue_depth_bounded { relay, max_bytes, window_start_ns?, window_end_ns? } (RELAY/STAGE) — across the window (defaults to the whole run), the named relay's enqueued_bytes never exceeds max_bytes. Fails on the first RelayEnqueue that pushes the total above the bound.
  • worker_alive_throughout { peer, window_start_ns, window_end_ns } (RELAY/STAGE) — the named peer's lifecycle state remains Running throughout the window. Fails on any stage_lifecycle event into Halted whose time falls in the window. The window is inclusive on both ends: a halt at window_end_ns (or at duration_ns when the window spans the whole run) fails the assertion, because the §6A.3 synchronous dispatch rule guarantees the stage_lifecycle → Halted event appears in the bundle even at the boundary.
  • name_resolves_within { name, observers, within_ns, from_ns } (RELAY/STAGE) — starting at from_ns, every observer in observers produces a snapshot whose name_registry contains name within within_ns. Inconclusive if no observer produces a snapshot in the window.

Adding a kind is a deliberate amendment to this section.

10.2 The evaluator interface

The evaluator reads the bundle's events.ndjson and snapshots/ and evaluates each assertion. Per assertion it emits a verdict:

{
  "name": "<assertion-name>",
  "kind": "<assertion-kind>",
  "parameters": { ... },
  "outcome": "Pass" | "Fail" | "Inconclusive",
  "evidence": [
    { "virtual_time_ns": <u64>, "event_or_snapshot_ref": "<path>" }
  ]
}

Inconclusive is reserved for assertions whose preconditions did not fire during the run (e.g., a peer the assertion names never became reachable).

verdicts.json is an array of verdict objects, one per declared assertion, in the order the scenario declared them.

10.3 Library properties

A library property is a parameterized assertion kind evaluated across a generated distribution of scenarios. The MVP ships one such property: the gossip-flap detector. The property generates scenarios from a declared space (peer count, latency range, jitter range, loss range, duration) and applies no_flap_while_probes_ok to each.

The framework records the random seed that produced any failing scenario so the failure is reproducible. The current SWIM source must fail this property. The proposed fix must pass it.

Library properties are otherwise identical to per-scenario assertions in output shape; their verdicts go into the property runner's own output, not into a single bundle's verdicts.json.

10.4 Streaming evaluation (for early termination)

For early termination (§4.8) the evaluator exposes a streaming side: as events are emitted, the evaluator may resolve assertions whose verdicts are determinable from the prefix. The engine polls this side after each event dispatch. The streaming side is an optimization; the post-run side remains the authoritative source for verdicts.json.

10.5 Behavioral tests

The evaluator's contract is the assertion catalog of §10.1, the verdict shape of §10.2, and the streaming side of §10.4. Its tests assert that it has the properties below.

Per-kind soundness. For every kind in §10.1: Pass is returned exactly when the kind's stated condition holds over the bundle; Fail exactly when the condition is violated; Inconclusive exactly when the preconditions did not fire.

Verdict shape. Every verdict conforms to §10.2. Fail verdicts carry evidence referencing the event or snapshot responsible.

Verdict order is scenario-declared. verdicts.json lists verdicts in the order the scenario declared the corresponding assertions.

Streaming agrees with post-run. On any bundle, the streaming side either does not resolve or resolves to the same verdict the post-run side will return. The two are never inconsistent.

Streaming resolves as early as possible. When a verdict is determinable from a prefix, the streaming side resolves no later than the end of that prefix.

Property failures replay exactly. A library-property failure recorded with seed S, replayed with seed S, produces the identical scenario and the identical Fail verdict.

11. Calibration

Calibration measures the simulator's fidelity against captured production bundles.

11.1 The corpus

The MVP corpus is the three N3 vast.ai bundles described in the deployment report. Each pairs with a scenario in scenarios/calibration/ that approximates the conditions under which the bundle was produced:

  • vastai-N3-1 ↔ n3_canary_relay_real_worker.toml. Canary relay, real worker. The relay's policy is set to the canary's measured shape (low egress per link, large queue, multi-hundred-millisecond cold start). Stage hosts carry the worker-exit timing observed in the bundle.
  • vastai-N3-2 ↔ n3_own_relay_real_worker.toml. Same topology with the relay's policy widened to the own-relay's measured shape. Stage hosts carry the same worker-exit timing.
  • vastai-N3-stub ↔ n3_own_relay_stub.toml. Same topology, widened relay, stage hosts with no WorkerExit mutations (full- run survival). Reproduces the gossip-flap pathology without stage death.

Each scenario carries a top-of-file prose comment naming the bundle it pairs with, the placeholder tolerances that the first calibration pass will replace with measured numbers, and any base scenario it extends.

11.2 The procedure

For each pair, the calibration tool:

  1. Runs the simulator with the declared scenario.
  2. Loads the corresponding production bundle.
  3. Computes the comparison metrics (§11.3) on both bundles.
  4. Reports per-metric pass/fail against per-metric tolerances.

11.3 The metrics

  • Per-peer event-timespan summaries (p50, p90, p99) per event kind.
  • State-transition reason distribution across all peers.
  • Message-size distribution per message kind.
  • Message-count per kind, per peer pair.
  • Self-incarnation trajectory per peer.
  • Connection-cache hit count.
  • Dial-started count.
  • Per-peer fraction of run time in the Alive state.

Relay extension metrics (RELAY/STAGE §10.2):

  • Relay queue-depth distribution over time (p50, p90, p99 of enqueued_bytes).
  • Per-link HOL delay decomposition: each delivery's delay attributed to base latency, edge-bandwidth serialization, jitter, cold-dial penalty, relay ingress, relay egress, and active mutations.
  • Probe-success-vs-transition ratio: fraction of Suspect transitions in which the observer's and the peer's bidirectional probes were succeeding at the transition time.
  • Worker-exit reason distribution per peer.
  • Name-resolution latency per registered name (time from register_name to first observer snapshot containing the name).

Tolerances ship as placeholders informed by intuition; the first calibration pass against the N3 corpus sets the real numbers.

11.4 CI integration

Calibration runs on every change that touches the simulator or the SWIM state machine. A regression — a previously-in-tolerance metric goes out of tolerance — blocks merge. A widening of a tolerance is a separate, justified commit.

12. Implementation phasing

Each phase ships independently. At every phase boundary the partial simulator does something useful and is testable.

Phase Adds Verifiable outcome
1 Scenario loader Every shipped scenario parses; malformed input is rejected with a structured error.
2 Network (no engine, no host) A microbenchmark queries send at scenario-scale rates and produces deterministic outputs.
3 Engine skeleton + a trivial echo host kind The cross-architecture parity test passes on a reference scenario.
4 SWIM host kind + codec contract test The smoke scenario runs to completion and asserts continuous Alive.
5 Assertion evaluator The gossip-flap reproduction scenario fails on current SWIM, passes after the fix.
6 Bundle writer A sim bundle renders through the production post-processor.
7 Calibration tool At least one N3 pair passes calibration.
R1 Relay vertex (§5A) without mutations A direct send through a relay arrives later than the same send over a direct edge of equal policy by the relay's ingress + egress serialization time.
R2 Relay mutations (§5.5 additions) A RelayKill followed by a RelayBoot produces a bundle in which the in-flight messages at kill-time appear in the invalidated-deliveries list and no others.
R3 Stage host kind (§6A) and WorkerExit mutation A scenario with one stage host and one WorkerExit mutation produces a bundle containing exactly one register_name, one worker_exited, and three stage_lifecycle records, in §6A.2 order.
R4 Assertion catalog additions (§10.1 R-tail) The N3-1 calibration scenario fails relay_queue_depth_bounded against the canary-relay policy and passes against the own-relay policy.
R5 Scenario loader additions (§8) and bundle additions (§9) Every shipped calibration scenario parses; bundles render through the production post-processor with the new event kinds passed through unchanged.
R6 Calibration tool extensions (§11) At least one of the three N3 pairs passes calibration on every declared metric.

MVP exit is the end of phase R6. Subsequent phases — proptest catalog expansion, scenario library growth — are post-MVP.

Phases 1, 2, 6 are independently buildable by separate agents from this spec alone; phases 3 onwards require the prior phase as input. The relay phases R1–R3 are independently buildable from the §5A / §6A sections alone; R4–R6 require their predecessors.

13. Open questions

These are deliberately unanswered; they are expected to resolve during phases 1–3.

  • SWIM's reactive probe mode introduces an internal safety-sweep timer. Whether the engine needs a separate event kind for the safety sweep, or whether driving the host on its tick interval is enough, depends on details inside the production probe code that are easier to resolve once the engine skeleton exists.
  • One peer corresponds to one host ID in the MVP. Production permits a single host to expose multiple endpoints. The MVP punts; if a calibration scenario needs the multi-endpoint shape, the scenario grows a per-peer endpoint list and the engine dispatches by endpoint.
  • The starting tolerances in §11.3 are placeholders. The first calibration run against the N3 corpus sets the real numbers; those numbers replace the placeholders in a follow-up commit.
  • The simulator's code lands under crates/simulation/ after the current cleanup of that crate. If the cleanup renames or relocates the simulator, the phase plan in §12 needs a one-pass path refresh; nothing else in this spec depends on the path.

14. Spec change protocol

Changes that relax a contract in §3 / §4 / §5 / §6 / §7 — a widened tolerance, a removed assertion, a relaxed determinism rule — are behaviour-changing and require a deliberate commit whose subject names the relaxation and whose body justifies it in prose. A tightening change — a new assertion kind, a narrower tolerance, a more restrictive determinism rule — can land in any commit. Adding new scenarios, new properties, or new phases does not require special treatment.

The lint scanner, the determinism digest, the cross-architecture parity test, the encoding-symmetry contract test, and the calibration tolerances are the load-bearing artefacts that enforce this spec. If any of them is short-circuited — disabled in CI, allow-listed at the call site, silenced with an exemption — the spec is being worked around, and the workaround must surface in code review.

15. References

  • examples/pipeline-parallel-inference/N3_DEPLOYMENT_REPORT.md — source of truth for the live failures the simulator must reproduce. Items A, B2, C.1, C.2, and C.3 of that report map to §5A, §11.3 (metrics), §6A.4 + §9.1 (register_name), §6A.4 (snapshot registry), and §6A.4 (worker_exited) respectively.
  • crates/simulation/NORTH_STAR.md — the long-term simulator vision. This MVP is a strict subset and does not retract any of its claims.
  • crates/simulation/BLOCKED.md — the staged plan this MVP supersedes for the immediate iteration. The deeper goals there (real quinn / iroh on a sim facade, detector as a peer, full distribution-crate facade migration) remain on the roadmap, just not gating SWIM-tuning.
  • crates/distribution/src/swim/ — the production SWIM state machine the SWIM host kind wraps.
  • crates/distribution/src/diagnostics/ — the schema the bundle must match and the renderer it must render through.
  • crates/distribution/src/bin/swactor-iroh-relay.rs — the production relay binary the §5A relay vertex models. The §9.2 parity contract names this binary's emissions as the production side once it gains matching schemas.
  • examples/pipeline-parallel-inference/src/bin/pp_gpu_node.rs — the production stage supervisor the §6A host kind wraps.