swactor/examples/pipeline-parallel-inference/N3_SWIM_TUNING_SPEC.md

# N=3 SWIM retuning against deployed latency — behavioral spec

Companion to `N3_POSTMORTEM_2026-05-25_1779733878.md`,
`N3_COVERAGE_EXTENSION_SPEC.md`, and the simulator's
`SWIM_TUNING_REPORT.md`. This document is the contract for a SWIM
retuning pass that uses the `1779733878` deployment's observed
latency and churn data as the evidence the tune is calibrated
against — rather than the simulated 60 ms latency the prior tune
used.

This is a *behavioral* spec. It names the targets the retuning
must hit, the evidence each target is calibrated against, and the
prerequisites that must be in place before a retuning pass can be
evidence-driven rather than guess-driven. It does not prescribe
specific knob values.

---

## 0. Motivation

`SWIM_TUNING_REPORT.md` documented a prior tuning pass against the
§10.3 gossip-flap property and the three N3 calibration scenarios.
That pass collapsed `self_incarnation_peak` from 86–94 to 7–10 — a
significant win — but its calibration latency was **60 ms RTT with
15 ms jitter** (per §2 of that report). The simulator's calibration
scenarios used these numbers because the live bundles available at
the time did not surface per-SWIM-probe RTT.

The `1779733878` run exposes a different reality:

- **Tier-2 (host-level) UDP echo RTT**: orchestrator 293 ms,
  stage-0 181 ms, stage-1 405 ms, stage-2 184 ms. p99 spread is
  multi-hundred-millisecond and asymmetric across peers.
- **Topology**: every peer connection is `conn_type=Relay`. No
  hole-punching succeeded. SWIM probes ride a relay-mediated path
  whose RTT is strictly higher than the tier-2 floor and is
  subject to relay-side HOL queueing.
- **Observed churn**: 1701 `SwimTransition` events over a ~7-minute
  run while iroh continued to exchange messages (`connect-timeout
  count = 0`). The chosen `probe_timeout = 15 ticks = 3 s` was
  selected against 60 ms RTT; against a relay-mediated path with
  p99 multi-hundred-millisecond tier-2 floor and load-driven
  queueing on top, that budget may be marginal or worse.
- **Configuration**: the prior tune's knobs landed at
  `probe_interval=10, probe_timeout=15, suspicion_timeout=75,
  indirect_probes=2, dead_reprobe_interval=50` ticks plus
  `max_piggyback=6`. These are committed defaults; the question
  this spec opens is whether they hold under the observed
  deployment shape, not whether the prior tuning method was
  correct.

The previous postmortem (run `1779720002`) could not have driven
this retune: its bundle lacked the data the observability upgrade
landed afterward. The `1779733878` bundle is the first one rich
enough to retune against. This spec captures the contract that
retuning must satisfy.

---

## 1. Prerequisites

Retuning is not evidence-driven until the data the tune calibrates
against is in the bundle. Three prerequisites are explicit.

### 1.1 Coverage 2.6 from `N3_COVERAGE_EXTENSION_SPEC.md`

Per-SWIM-probe RTT events and the post-processor's `## Probe RTT
distribution` section must land in production *and* in the sim
adapter. Until this coverage is in place:

- The deployed evidence is tier-2 RTT (UDP echo to the collector),
  which understates the relay-mediated SWIM RTT by an unknown
  factor.
- The simulator's `no_flap_while_probes_ok` assertion is
  `Inconclusive` on every SWIM scenario (per
  `SWIM_TUNING_REPORT.md` §6.3), so the assertion can neither
  pass nor fail the retune.

A retuning pass that lands without 2.6 is a guess against
tier-2 latency — the same mistake the prior tune made against
60 ms simulated latency, only with a different proxy for the real
number.

### 1.2 Determinism fix from `SWIM_TUNING_REPORT.md` §6.7

`MemberList`'s `HashMap<NodeId, _>` randomises iteration order per
process; the prior tune reports ±20 % run-to-run variance as a
result. A retuning pass that has to average across five samples
per grid point to estimate variance is exactly five times slower
and five times noisier than one against deterministic substream
selection. `HashMap` → `BTreeMap` is the one-line fix the prior
report names; it must land before retuning, not after, so the
retune's results have signal-to-noise high enough to read.

### 1.3 The Layer-B1 refute-on-stale-Suspect bug (`SWIM_TUNING_REPORT.md` §6.1)

`crates/distribution/src/swim/node.rs::apply_membership_update`
refutes against `self_id()` whenever
`update.state ∈ {Suspect, Dead}` regardless of whether
`update.incarnation` is current. This creates a non-zero floor on
`self_incarnation_peak` that no tuning can collapse. A retuning
pass against the `1779733878` shape, where the relay-mediated path
keeps stale Suspect entries in the dissemination queue for many
probe cycles, will hit this floor and conclude — incorrectly —
that further tuning gain is unavailable.

The one-condition gate the prior report names is the
priority-1 follow-up the prior tune deferred. It is a
prerequisite for evidence-driven retuning against this deployment,
not a downstream cleanup.

---

## 2. Calibration data the retune is driven by

The retuning pass's evidence comes from the `1779733878` bundle
(and any subsequent N=3 deploy bundles that land before the
retune). Three numbers anchor the calibration:

### 2.1 Observed tier-2 RTT distribution

| node         | RTT (ms) | echo success |
|--------------|----------|--------------|
| orchestrator | 293      | 34/35        |
| stage-0      | 181      | 55/55        |
| stage-1      | 405      | 27/28        |
| stage-2      | 184      | 38/38        |

The tier-2 echo path is collector-bound, not peer-bound. It
establishes the floor below which a relay-mediated SWIM probe
cannot land.

### 2.2 Per-SWIM-probe RTT distribution (post coverage 2.6)

After coverage 2.6 lands, the bundle will carry per-probe RTT
distributions per (observer, target) pair, plus per-bucket
distributions over the run window. The retune calibrates
`probe_timeout` such that the configured budget exceeds the
observed p99 of legitimate (non-failure) probe RTT with a margin
the retune explicitly justifies. Until 2.6 is collected against a
live run, the retune uses §2.1 as a lower-bound proxy and is
explicit about that.

### 2.3 SWIM churn and dial outcomes

`SwimTransition: 1701` across a ~7-minute run is the load-bearing
churn signal. The retune is calibrated such that a scenario
configured to mirror the `1779733878` shape produces a churn
count within a stated factor (target: <300, an order-of-magnitude
collapse comparable to the prior tune's `self_incarnation`
collapse).

Per-peer dials from the postmortem (orchestrator 7/11 timeout,
inter-stage 19/19 success) are the discriminator the retune must
not undo: a retuned SWIM that makes inter-stage probes flap is a
regression even if it makes orchestrator-bound probes more stable.

---

## 3. Retuning targets

Six, ordered by load-bearing impact.

### 3.1 `probe_timeout` against relay-mediated p99 RTT

**Target**: `probe_timeout` exceeds the bundle's observed p99 of
legitimate probe RTT (post-2.6) by a margin the retune justifies
in prose — the margin must account for relay-side HOL queueing
peaks the steady-state distribution does not capture.

**Anti-target**: the budget cannot be set so high that suspicion
takes longer than the operator's deadstop threshold. The
postmortem named ~7 min as the operator's deadstop budget; SWIM's
detection time (`probe_timeout + suspicion_timeout`) must remain
well under that, with a documented headroom.

**Evidence**: per-probe RTT histogram from §2.2; SwimTransition
churn count from §2.3.

### 3.2 `suspicion_timeout` under relay-mediated reachability

**Target**: a peer whose relay-mediated path is intermittently
unreachable (the `1779733878` shape — repeated probe failures
interleaved with successes) does not flap between Alive and
Suspect more than the prior tune's bound on the gossip-flap
property, when the scenario mirrors the deployment's latency
distribution.

**Anti-target**: a peer whose path is genuinely dead is not
falsely held Alive past the operator's deadstop window.

**Evidence**: §2.3 churn count; the `no_flap_while_probes_ok`
assertion (now resolvable post-2.6) against the calibration
scenario.

### 3.3 `indirect_probes` count against relay HOL behavior

**Target**: indirect probes still provide redundant coverage when
the direct probe times out, but their cumulative bandwidth
contribution to the relay's egress queue does not push the
`relay_queue_depth_bounded` assertion to fail under own-relay
policy.

**Anti-target**: dropping the count below the prior tune's 2
collapses indirect coverage, which the prior tune's §5 already
documents.

**Evidence**: `relay_queue_depth_bounded` under own-relay calibration;
churn count from §2.3.

### 3.4 `probe_interval` against the dial-rate signal

**Target**: probe rate is set such that the orchestrator-bound
dial failures the `1779733878` run exhibited (7/11 timeout) do
not bottleneck convergence beyond a tolerance the spec names.

**Anti-target**: probe rate is not lifted so high that
`message_size_bounded` regresses against own-relay policy.

**Evidence**: per-peer dial table from §2.3; piggyback byte
totals from the bundle.

### 3.5 `max_piggyback` against observed gossip-receipt sizes

**Target**: piggyback gossip stays within the
`message_size_bounded` envelope under own-relay policy, given
the `1779733878` per-node piggyback byte totals (806–1589
piggybacks per node, 194–522 KB total).

**Anti-target**: lowering `max_piggyback` below the prior tune's
6 stops convergence within the property's window
(`SWIM_TUNING_REPORT.md` §5).

**Evidence**: gossip-receipt totals from the postmortem's "Gossip
receipts" section; `message_size_bounded` assertion under
own-relay.

### 3.6 `LifeguardConfig` wiring (formerly out of scope)

**Target**: the dynamic suspicion-timeout formula in
`crates/distribution/src/swim/lifeguard.rs` is wired into
`SwimNode`'s suspicion state machine. Until wiring lands, the
constants in `lifeguard.rs` have no observable effect — per
`SWIM_TUNING_REPORT.md` §6.5, the prior tune could not sweep "the
lifeguard band" because it was dead code.

This target is the only one that requires code beyond a knob
change. It is included here because the prior tune named it as a
priority follow-up and because the `1779733878` data motivates
adaptive suspicion: a path whose RTT varies 2× under load benefits
from adaptive timeouts more than a static budget can capture.

**Anti-target**: landing the wiring without sweeping its
parameters reproduces the prior tune's dead-code condition for the
new fields. The wiring must come with a sweep against the
calibration scenarios.

**Evidence**: the new dynamic-suspicion code path is exercised by
at least one scenario whose assertion verdict changes when the
multiplier changes.

---

## 4. Calibration scenario updates

The three N3 calibration scenarios under
`crates/simulation/scenarios/calibration/` were last updated to
mirror the prior tune's defaults at the scenario's 200 ms tick
(`SWIM_TUNING_REPORT.md` §3). The retune updates these scenarios
along two axes:

- **Latency distribution**: per-link latency is set against the
  `1779733878` per-peer tier-2 RTT distribution, not the prior
  60 ms baseline. Heavy-tailed per `SIM_HARDENING_SPEC §8` (the
  prior battery spec's reference); the distribution's median, p95,
  and p99 fall within tolerance of the live bundle's after
  coverage 2.6 lands.
- **Topology**: every host-to-host link is routed through the
  relay vertex (`via = R` in scenario syntax). The
  `1779733878` shape had `conn_type=Relay` everywhere; the
  calibration scenarios must reflect that to be evidence-faithful.

The scenarios' `kind_config` blocks are updated to the retune's
chosen operating point. The current calibration block (per the
prior report) gives probes a 333 ms budget against 60 ms RTT;
against multi-hundred-millisecond relay-mediated RTT, the same
budget under-budgets by an order of magnitude. The retune's new
budget is the §3.1 target.

---

## 5. Acceptance

The retune is complete when:

1. Every prerequisite in §1 is in place (coverage 2.6, the
   determinism fix, the Layer-B1 gate).
2. Each target in §3 has a chosen operating point and a one-line
   prose justification anchored to the §2 evidence.
3. The `1779733878` calibration scenario, configured to mirror
   the deployment's latency and topology, produces fewer than
   300 `SwimTransition` events in a 7-minute virtual run (an
   order-of-magnitude reduction from 1701).
4. Inter-stage dial outcomes in the calibration bundle remain at
   the `1779733878` shape (≥95 % success on inter-stage edges)
   — the retune does not improve orchestrator-bound stability at
   the cost of inter-stage flakiness.
5. The §10.3 gossip-flap property's `self_incarnation_peak` does
   not regress from the prior tune's 7–10 band.
6. A retuning report (a successor to `SWIM_TUNING_REPORT.md`)
   documents the new operating point, the evidence each knob
   choice was calibrated against, the before/after numbers across
   every calibration scenario, and the limits the retune could
   not move.

---

## 6. Out of scope

- **Adding new SWIM features.** The retune adjusts existing knobs
  and lands the Layer-B1 gate / Lifeguard wiring the prior report
  named. New algorithmic features (push-pull anti-entropy,
  alternative failure detectors) are not in scope.
- **Relay-side fixes.** The `relay_queue_depth_bounded` failure
  on the canary topology is structurally out-of-reach for SWIM
  tuning (`SWIM_TUNING_REPORT.md` §6.2). The retune does not
  attempt to make canary pass; it does not regress own-relay.
- **Orchestrator-topology changes.** Running the orchestrator on
  a reachable host (the `1779733878` postmortem's item 6) is a
  deployment-shape change, not a SWIM-tuning change. The retune
  is calibrated against the NAT'd-orchestrator shape because that
  is the deployment we have, but the conclusion may be "even
  optimally-tuned SWIM cannot stabilize this topology" — that
  conclusion is a valid retune outcome.
- **The gossip-flap property's `self_incarnation_bounded`
  assertion.** The prior tune collapsed it from 86–94 to 7–10
  without removing the non-zero floor; the retune holds that
  result. Removing the floor is the Layer-B1 fix's job (a §1.3
  prerequisite, not a §3 target).
- **Scenarios beyond the calibration corpus.** The reproduction
  and topology scenarios remain on their current SWIM config.
  Retuning them is a follow-up that should wait for the
  calibration retune to converge.

---

## 7. References

- `N3_POSTMORTEM_2026-05-25_1779733878.md` — source of the
  observed latency distribution (§"UDP echo probes"), the churn
  signal (§"SWIM churn and relay events"), the dial outcomes
  (§"Per-peer dials"), and the topology context
  (`conn_type=Relay` everywhere, NAT'd orchestrator).
- `N3_COVERAGE_EXTENSION_SPEC.md §2.6` — the data surface this
  spec consumes. §1.1 of this spec is a hard prerequisite.
- `crates/simulation/SWIM_TUNING_REPORT.md` — the prior tuning
  pass. §3 (configuration), §5 (tradeoff curve), §6 (limits) are
  the load-bearing prior art the retune does not re-derive. §6.1,
  §6.5, §6.7 limits are §1.3, §3.6, §1.2 prerequisites
  respectively in this spec.
- `crates/simulation/SIM_SPEC.md` — the simulator's calibration
  contract (§11) and the assertion catalog (§10.1) the retune is
  scored against.
- `N3_SIM_TEST_BATTERY_SPEC.md` — the sim-test battery. A
  retuned SWIM that regresses any battery family is a retune
  regression, not a battery regression.