swactor/crates/simulation/BLOCKED.md

Simulator BLOCKED — Stage 6 / §10.2 / §4.4 contracts unfulfilled

This file is filed per TESTING_SPEC §14. The local v1 parity bar is green for the surface this implementation chose to bind to, but three load-bearing contracts named by the plan and TESTING_SPEC have not been satisfied at full scope. The remaining work is not a v1 "out-of-scope" item under TESTING_SPEC §15, so the only honest artifact to ship is this BLOCKED.md plus the qualified BUILD_REPORT.md next to it.

Iteration 14 fixed one of the four findings (Finding D — facade surface descriptor now derived from trait declarations by build.rs); the other three remain open.

Note on layout (added in path-refresh pass): since this file was first filed, the workspace consolidated several crates (runtime-facade, sim-detector, sim-driver, lint-deterministic) into crates/simulation (PRs #39, #50, #52). Paths below reflect the post-consolidation layout. The underlying contracts and the substantive gaps they call out are unchanged.

Symbol	Was	Now
Runtime facade traits	crates/runtime-facade/src/lib.rs	crates/simulation/src/runtime/mod.rs
Surface-lock build	crates/runtime-facade/build.rs	crates/simulation/build.rs
Surface fingerprint	crates/runtime-facade/surface.lock	crates/simulation/surface.lock
Lint config	crates/lint-deterministic/banned.toml	crates/simulation/banned.toml
Lint scanner	crates/lint-deterministic/src/lib.rs	crates/simulation/src/lint.rs
Detector probes	crates/sim-detector/src/lib.rs	crates/simulation/src/detector.rs
Sim binary entry	crates/sim-driver/src/main.rs	crates/simulation/src/bin/sim-driver.rs

---

A. plan §"Stage 6 — TIER CHECKPOINT" — distribution migration

Contract

Migrate crates/distribution to the facade: every direct Instant, SystemTime, tokio::spawn, tokio::time::sleep, rand::thread_rng, HashMap iteration, env::var, std::fs, std::net use is rewritten against the runtime facade. Lint scope widens to include distribution; scan reports zero violations.

What's actually happening

The lint scanner's audit scope (crates/simulation/banned.toml:: [scope]::include) is exactly one directory:

include = [
    "crates/simulation/src",
]

crates/distribution is not in scope. The crate still calls the real runtime directly across at least these files:

• crates/distribution/src/swim/member_list.rs
• crates/distribution/src/diagnostics/aggregator.rs
• crates/distribution/src/diagnostics/sink.rs
• crates/distribution/src/diagnostics/probes.rs
• crates/distribution/src/diagnostics/host_introspect.rs
• crates/distribution/src/diagnostics/collector/udp_echo.rs
• crates/distribution/src/diagnostics/collector/state.rs
• crates/distribution/src/diagnostics/collector/handlers.rs
• crates/distribution/src/cache.rs
• crates/distribution/src/iroh_driver.rs
• crates/distribution/src/peer_auth.rs
• crates/distribution/src/node_metadata.rs
• crates/distribution/src/kademlia/repair.rs
• crates/distribution/src/kademlia/directory.rs

These call sites use tokio::spawn, tokio::time::sleep, std::time::Instant, std::time::SystemTime, rand::thread_rng, std::net::UdpSocket, and std::collections::HashMap directly.

Live-fire reproduces the gap: drop a use std::time::SystemTime; into crates/distribution/src/_violator_test.rs; the lint scan still exits 0, because that directory is not in scope.

The problem, plainly

The whole point of the runtime facade is so that peer code becomes a pure function of (spec, seed) — same inputs produce byte-identical bundle outputs, in prod and in sim. Every direct Instant::now(), tokio::spawn, or HashMap iteration in distribution breaks that guarantee on the production diagnostics path: the diagnostics aggregator's per-peer state lives in a HashMap, so its iteration order varies run-to-run with hash randomisation. A replay can't reproduce a recorded run if a HashMap::iter shows up between input and output.

Until distribution is migrated, the "sim and prod are the same code with a different runtime" claim only holds for the simulation crate itself. The peer code that actually does the work is exempt.

Potential action items

The migration is big and has a forced order. A reasonable plan:

1. Grow the facade Spawn surface to be async-capable. The current trait (crates/simulation/src/runtime/mod.rs:91) only takes Box<dyn FnOnce() + Send + 'static>. Distribution's diagnostics pipeline is built on tokio::spawn(async move {...}). Add fn spawn_future(&self, fut: BoxFuture<'static, ()>) (or equivalent), implement on runtime::prod with tokio::spawn and on runtime::sim against the engine's fiber queue. This is a surface.lock bump.

2. Decide on the HashMap policy and apply it crate-wide. Replace prod-path HashMap / HashSet with IndexMap / IndexSet (preserves insertion order, fast) or BTreeMap / BTreeSet (preserves sort order, slower but no extra dep). Aggregator, member-list, and kademlia routing-table state are the highest-leverage call sites — diagnostics aggregator first because it's the largest source of replay divergence.

3. Migrate the call sites in waves. Suggested order, smallest blast radius first:

   • cache.rs and node_metadata.rs (mostly time reads).
   • peer_auth.rs (time + RNG).
   • swim/member_list.rs (spawn + time + map iteration).
   • kademlia/ (spawn + time + map iteration).
   • diagnostics/* (spawn + time + UDP + map iteration — biggest).
   • iroh_driver.rs (depends on Finding C's facade growth — defer).

4. Widen lint scope. Add "crates/distribution/src" to banned.toml::[scope]::include and fix the violations the scanner reports until it exits 0. Do this last, after the migration waves above land — otherwise the scanner blocks every intermediate commit.

5. Add a live-fire regression test. After (4) is green, drop a throwaway use std::time::SystemTime; somewhere in distribution/src/ and confirm the scanner catches it; revert. Record the result in the iteration notes.

This is a multi-iteration project. Budget at least three.

---

B. TESTING_SPEC §10.2 — detector must run on the sim facade

Contract

The detector binary must run inside the reference scenario as an additional peer.

What's actually happening

crates/simulation/src/facade/sim/bundle.rs:81 calls crate::detector::run_all() synchronously from inside the bundle writer. The detector code lives at crates/simulation/src/detector.rs and every D01–D12 probe body reaches for the real std::* / tokio surface (SystemTime::now(), Instant::now(), std::thread::spawn, std::net::UdpSocket, std::net::TcpStream, File::open("/dev/urandom"), gethostname, pid, allocator addresses, ThreadId). Each violation is silenced with an inline // lint-deterministic: allow … marker.

The detector is therefore observing the host runtime, not the sim runtime. techniques.md admits this verbatim for D01 ("The v1 sim does not intercept std::time::*; the detector running in-process therefore reads the host wall clock") and D02 ("v1 reads the host directly, so the two deltas are microseconds apart in practice").

t_detector::sim_indistinguishable consequently passes whether or not the sim leaks — there is no sim facade between the probe and the host to leak through.

The problem, plainly

NORTH_STAR's load-bearing claim is "no means, statistical or otherwise, of detecting that it is in a sim." The test that's supposed to enforce that claim is currently a no-op: the probes are literally not pointed at the simulated runtime. It's the same as running an integrity test against cat and concluding the data is intact — there's no system under test.

The lint-deterministic: allow markers are load-bearing audit evidence that this work was not done. Removing them without doing the work would just move the failure from a semantic check to a mechanical one.

Potential action items

Two changes have to land together; doing only one of them is worse than doing neither (it advertises a guarantee that isn't there).

1. Make the sim backend actually intercept the std/tokio surface. crates/simulation/src/runtime/sim.rs (133 lines) is mostly stubs. Concretely:

   • Clock::now() must return a virtual-time-derived SystemTime, not pass through to SystemTime::now().
   • Udp::bind must return a socket whose send_to / recv_from ride the engine's link graph instead of OS sockets.
   • Spawn must enqueue onto the engine's fiber scheduler instead of tokio::spawn.
   • ProcessMeta::hostname / pid must return per-node sim values, not the host's.

2. Route the detector probes through the facade. Open crates/simulation/src/detector.rs and, for each probe:

   • Replace SystemTime::now() with facade.clock().now().
   • Replace Instant::now() with facade.clock().monotonic().
   • Replace std::thread::spawn with facade.spawn().spawn(...).
   • Replace std::net::UdpSocket::bind with facade.udp().bind(...).
   • Drop the // lint-deterministic: allow … markers as each site converts. By the end the file should be allow-marker-free.

3. Run the detector as a real peer. Add a host of kind detector to the reference scenario spec. In the bundle writer (crates/simulation/src/facade/sim/bundle.rs:79–88), delete the in-process detector::run_all() call. Instead, have the engine schedule a fiber on that host that runs the same probes via the sim facade and writes its verdict stream into the bundle just like any other host's records.

4. Tighten the test. Once the above is in, modify the sim facade in a controlled way to leak (e.g., have Clock::now() return wall-clock instead of virtual). The relevant probe verdict should flip from Indistinguishable to DetectedSim and t_detector::sim_indistinguishable should fail.

This depends on Finding A only for the Spawn async surface growth, so it can begin in parallel with A's migration waves.

---

C. plan §"Stage 6 — TIER CHECKPOINT" — real transports on the sim facade

Contract

Real quinn linked against sim UDP — no QUIC fork. Real iroh stack (MagicSock, discovery, NodeMap, connection cache) linked against sim facade — no iroh fork.

And TESTING_SPEC §5.2:

The same quinn / iroh / iroh-relay / distribution / swactor source files run in both binaries.

What's actually happening

crates/simulation/src/engine.rs (885 lines) is a discrete-event record synthesiser. Its imports are std::cmp::Ordering, std::collections::{BTreeMap, BTreeSet, BinaryHeap}, serde::{Deserialize, Serialize}, and types from crate::spec. grep -n 'quinn\|iroh\|distribution\|swactor' crates/simulation/src/engine.rs returns zero matches.

The engine emits HostStart / HostStop / HostCrash / Snapshot / Mutation events on a synthetic loopback host and writes them to the bundle. The corpus is shaped to look like real peer-code output, but no production peer code is in the call graph.

(Update from the original BLOCKED.md: the sim-driver binary used to pin #[used] static references to defeat dead-code stripping — the t_same_binary::symbol_overlap test exists to detect that gimmick. The pin gimmick was removed during the consolidation; symbol_overlap is now listed as removed in tests/parity-bar/expected_failures.txt:33. The deeper claim is unchanged: peer code is not exercised through the sim engine.)

The problem, plainly

§5.2's whole point is that there should be one implementation of QUIC, one implementation of iroh, one implementation of distribution — and the only difference between prod and sim is which backend implements the runtime facade. If the sim binary runs synthesised events that mimic peer-code output, then the sim is not testing peer code — it's testing the simulator's model of peer code, which is exactly the thing the parity bar is supposed to make unnecessary.

This is why the dependency chain matters: real quinn needs an async UDP socket, real iroh needs quinn plus a tokio runtime and DNS, real distribution needs iroh::Endpoint. Each layer sits on the one below it.

Potential action items

This is the largest body of work in the BLOCKED set. A reasonable sequencing:

1. Async UDP on the facade. Grow traits::Udp / traits::UdpSocket (crates/simulation/src/runtime/mod.rs:64– 73) from the current sync bind / send_to / recv_from shape to something that implements quinn::AsyncUdpSocket. In prod this wraps tokio::net::UdpSocket; in sim it wraps a queue tied to the engine's link graph and clock. surface.lock bump.

2. Virtual tokio::time::Instant projection. iroh's internals call tokio::time::Instant::now() and tokio::time::sleep directly (it does not consult our facade). Either (a) build a sim-tokio runtime so tokio::time reads from the engine's virtual clock when iroh is linked into the sim binary, or (b) accept that iroh-as-shipped can't run under the sim and patch iroh upstream. (a) is the path TESTING_SPEC §5.2 implies.

3. Async Spawn. Same growth as Finding A (1); A and C share this prerequisite.

4. DNS table in sim. traits::Dns (crates/simulation/src/runtime/mod.rs:85) needs to answer relay-URL queries from a per-node DNS table the spec configures, not from the host resolver. iroh's relay discovery won't work otherwise.

5. Build a quinn::Endpoint from facade pieces. Once (1) and (2) are in, construct a quinn::Endpoint from the facade's UDP socket and feed it to iroh's MagicSock setup.

6. Re-point distribution. Wire distribution's transport construction off direct iroh::Endpoint::builder() calls and onto the facade-built endpoint. (Finding A's migration is a precondition.)

7. Replace synthetic engine emission with real peer-code execution. This is the big one. engine.rs currently is the event source; under (6), real peer code becomes the event source and the engine becomes scheduling + link graph + clock. Most of engine.rs's synthetic emit logic deletes; the schema-floor parity bars then bind to events real code produced.

8. Re-enable the parity-bar checks currently in expected_failures.txt. t_replay::*, the §6/§7 schema coverage suite, and t_same_binary::shared_load_bearing are parked behind Stage 6/7. They light up as the real-peer-code path comes online.

Expect this to span four to six iterations. Don't try to land it as one PR.

---

D. TESTING_SPEC §4.4 — facade surface lock — fixed in iteration 14

Contract

Each trait's method signatures hashed in declaration order.

Resolution

Was: SURFACE_DESCRIPTOR = include_str!("surface_descriptor.txt"), a 38-line hand-edited file. A trait edit that did not also touch the text file passed the lock unchanged.

Now: crates/simulation/build.rs parses crates/simulation/src/runtime/mod.rs with syn at build time, walks pub mod traits, emits one line per trait method signature into $OUT_DIR/surface_descriptor.txt, and the runtime module embeds the build output via include_str!. Live-fire verified: adding fn stealth_method(&self) -> u32 to pub trait Clock flips the descriptor hash and fails live_fingerprint_matches_locked. Reverted.

This finding is closed. Listed here for the record because the other three are open against the same TIER CHECKPOINT.

---

What the next agent should do

Either:

1. Pick up the chain at A → B → C in order. A is the prerequisite for the lint widening; A's facade growth (async Spawn) and B's sim-facade interception together unlock C's real-transport linkage. Plan on multiple tier-checkpoint-sized iterations; this is not a single-iteration cleanup.

2. Raise a TESTING_SPEC defect under §14.2 if a specific check is itself wrong — e.g., if §10.2's "additional peer" wording proves impractical given the workspace constraint no-modify src/ and the facade should instead intercept at the bundle-writer boundary. A defect would need to quote NORTH_STAR / SPEC / OBSERVABILITY and ship as a separate commit titled TESTING_SPEC: correct §N.M per §14.

What the next agent should not do is widen the lint-deterministic: allow markers, leave the engine wholly synthetic, or paper over symbol-presence checks. The bar in this repo is what the spec says.