swactor/docs/development_history/distribution/SIMULATION_TESTING.md

# Simulation Testing — Development History

> Covers the addition of network fault injection to the simulation harness
> and 15 new cluster scenario tests, informed by research into production
> distributed systems testing practices.
>
> 4 files changed · ~950 insertions
>
> *Branch: `distribution-realization`*

---

## Table of Contents

1. [Overview & Motivation](#1-overview--motivation)
2. [What Was Built](#2-what-was-built)
3. [Research Phase](#3-research-phase)
4. [Network Fault Injection](#4-network-fault-injection)
5. [Cluster Scenario Tests](#5-cluster-scenario-tests)
6. [Key Findings](#6-key-findings)
7. [Design Decisions](#7-design-decisions)
8. [Known Gaps & Future Work](#8-known-gaps--future-work)

---

## 1. Overview & Motivation

The simulation crate (`crates/simulation/`) had 6 distribution tests covering
happy-path scenarios: cluster convergence, node death detection, node rejoin,
and actor resolution. All tests assumed a perfect network — 100% delivery,
zero latency variation, no partitions.

Real networks drop packets, partition nodes, and deliver messages out of order.
The SWIM protocol's correctness under these conditions was untested. This work
adds network fault simulation and exercises the protocol under adversarial
conditions drawn from established testing methodologies.

---

## 2. What Was Built

| Component | Location | Description |
|-----------|----------|-------------|
| Network fault model | `crates/simulation/src/distribution/sim.rs` | Partition, heal, and message drop simulation |
| 15 cluster scenario tests | `crates/simulation/tests/cluster_scenarios.rs` | Behavioral tests for failure modes |
| Research notes | `CLAUDE/notes/research_simulation_testing.md` | Survey of 7 codebases/frameworks |

All 15 new tests run in ~1.4s total (well under the 2-minute cap).
The original 6 distribution_sim tests are unaffected.

---

## 3. Research Phase

Seven codebases and frameworks were studied for their simulation testing
methodology:

| Source | Key Takeaway |
|--------|-------------|
| **FoundationDB** | Deterministic simulation: single-threaded, seeded PRNG, simulated time. BUGGIFY injects faults inside production code at ~25% activation × 25% firing probability. |
| **Hashicorp memberlist** | ~80 test functions. Lifeguard extensions: suspicion timer with log(k+1) decay, health-aware probe timeouts, dogpile confirmation. |
| **Antithesis** | Categorized fault injection: network, process, disk, timing. Emphasis on property-based invariant checking. |
| **TigerBeetle** | VOPR simulation + Vortex TCP proxy. Runs millions of seeds nightly. |
| **Turmoil** (tokio-rs) | Rust DST: `sim.partition(a,b)`, `sim.hold(a,b)`, `sim.repair(a,b)`. Seeded RNG, simulated time. |
| **MadSim** | Rust DST used by RisingWave. FIRO scheduling, libc interception for true determinism. |
| **Jepsen** | Standard nemesis catalog: partition, kill, pause, clock skew, membership change. |

Full notes: `CLAUDE/notes/research_simulation_testing.md`

---

## 4. Network Fault Injection

Three new types model network conditions:

```rust
pub struct Partition {
    pub side_a: Vec<usize>,   // node indices
    pub side_b: Vec<usize>,
    pub asymmetric: bool,     // if true, only side_a→side_b is blocked
}

pub enum NetworkFault {
    Partition { round: usize, partition: Partition },
    Heal { round: usize },
    SetDropRate { round: usize, rate: f64 },
}
```

`NetworkState` tracks blocked pairs (as a `HashSet<(usize, usize)>`) and
applies probabilistic message dropping via a deterministic LCG PRNG
(seed `0x853c49e6748fea9b`). The `should_deliver(from, to)` method checks
both partition membership and drop rate before allowing message delivery.

Faults are applied per-round in `run_simulation` before the tick/deliver
cycle. Initial join and settle phases always use a clean `NetworkState`
(no faults during cluster formation).

### Backward Compatibility

`DistributionSimConfig` gained a `network_faults: Vec<NetworkFault>` field
defaulting to an empty vec. Existing tests that don't set this field
see no behavior change — the renamed `deliver_actions_tagged_with_net`
function with a clean `NetworkState` is functionally identical to the
original `deliver_actions_tagged`.

---

## 5. Cluster Scenario Tests

15 tests organized by failure category:

### Partitions
| Test | Scenario | Assertion |
|------|----------|-----------|
| `symmetric_partition_splits_membership_views` | 6 nodes split {0,1,2} vs {3,4,5} | Each side forms sub-cluster; dead-declared nodes not auto-rediscovered |
| `asymmetric_partition_causes_one_sided_suspicion` | 5 nodes, one-way block | Recovery after heal |
| `partition_plus_kill_in_minority_side` | 6 nodes, partition + kill in minority | Compound failure handled |
| `sequential_partitions_fragment_cluster` | Sequential partition events | Creates sub-clusters |
| `actor_resolution_degrades_during_partition` | Actors registered pre-partition | Cached resolutions survive partition |

### Message Loss
| Test | Scenario | Assertion |
|------|----------|-----------|
| `cluster_converges_under_10_percent_message_loss` | 10% drop rate | Some membership maintained |
| `heavy_message_loss_causes_membership_instability` | 30% drop rate | Degrades but doesn't crash |
| `cluster_survives_brief_message_loss` | 15% loss for 15 rounds then heals | ≥2 well-connected survivors |

### Node Failures
| Test | Scenario | Assertion |
|------|----------|-----------|
| `cluster_survives_seed_node_death` | Kill node 0 (seed) | 4 survivors maintain ≥60% accuracy |
| `simultaneous_two_node_failure_detected` | Kill 2 of 7 at once | Both deaths detected |
| `cascading_failures_leave_quorum_intact` | Kill 3 of 7 sequentially | Survivors maintain membership |
| `graceful_leave_detected_faster_than_crash` | Crash detection timing | Bounded detection rounds |

### Scale & Churn
| Test | Scenario | Assertion |
|------|----------|-----------|
| `cluster_of_fifty_converges` | 50-node cluster | ≥90% accuracy |
| `rapid_churn_maintains_partial_membership` | 8 nodes, 4 kill/revive cycles | Partial membership maintained |
| `membership_changes_disseminate_to_all_nodes` | 10-node cluster, verify propagation | All survivors detect death |

---

## 6. Key Findings

1. **SWIM does not auto-rediscover dead-declared nodes.** Once the suspicion
   timeout expires and a node is declared dead, it is permanently removed.
   Re-joining requires the join protocol. This is correct SWIM behavior,
   not a bug — but tests must account for it.

2. **Message loss is highly destabilizing for SWIM** because it affects both
   the direct probe AND indirect probes in the same cycle. Default config
   (`suspicion_timeout=5`, `indirect_probes=1`) cannot tolerate even 15%
   loss. Tuned config (`suspicion_timeout=15–20`, `indirect_probes=2`,
   `probe_timeout=5`) tolerates ~10%.

3. **The LCG PRNG for message dropping needs a non-zero seed** to avoid
   correlated early values (seed 0 always produces 0.0 as first output,
   causing deterministic first-message drop).

4. **50-node clusters converge quickly** with the simulation's
   topology-aware join strategy, achieving ≥90% accuracy.

---

## 7. Design Decisions

| Decision | Rationale |
|----------|-----------|
| LCG instead of `rand` crate | Keeps simulation deterministic without adding dependencies; 64-bit LCG with Knuth constants is sufficient for drop-rate testing |
| Blocked pairs in HashSet | O(1) lookup per message; partition model maps directly to real network behavior |
| Clean NetworkState for join/settle | Faults during initial cluster formation would conflate test setup with test assertions |
| Loose accuracy thresholds for loss tests | SWIM's sensitivity to message loss means tight thresholds create flaky tests; the behavioral property being tested is "degrades gracefully" not "maintains perfect accuracy" |
| Tests verify SWIM's actual semantics | Rather than expecting auto-recovery after partition heal (which SWIM doesn't support), tests verify the sub-cluster formation that actually occurs |

---

## 8. Known Gaps & Future Work

| Gap | Priority | Notes |
|-----|----------|-------|
| Property-based invariant checking | High | Formal completeness/accuracy as automated checks |
| Message reordering | Medium | Out-of-order delivery in network model |
| Kademlia-specific scenarios | Medium | Routing table convergence under churn, directory repair |
| Suspicion refutation tests | Medium | Incarnation bump prevents false death |
| Graceful leave protocol | Medium | Wire `node.leave()` into simulation |
| BUGGIFY-style injection | Low | Probabilistic faults at protocol decision points |
| Re-join after partition heal | Low | Auto-rediscovery mechanism (not standard SWIM) |