zacheryasc/swactor
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
swactor/CLAUDE/notes/progress.md
Developer 8d8e33c0e2 feat(simulation): network fault injection and 15 cluster scenario tests 
Add network partition, asymmetric partition, and message loss simulation
to the distribution test harness. 15 new behavioral tests covering
split-brain, cascading failure, seed node death, rapid churn, 50-node
clusters, and actor resolution under partition.

Research notes from studying FoundationDB DST, Hashicorp memberlist,
Antithesis, TigerBeetle VOPR, Turmoil/MadSim, and Jepsen nemeses.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-13 07:45:17 +00:00

3.3 KiB
Raw Blame History

Progress

Session 1 — Simulation Test Breadth (2026-02-12)

Completed

  1. Research phase: Studied Hashicorp memberlist, FoundationDB DST, Antithesis, TigerBeetle VOPR, Turmoil/MadSim, Jepsen nemeses
    • Notes in CLAUDE/notes/research_simulation_testing.md
  2. Enhanced simulation harness (crates/simulation/src/distribution/sim.rs):
    • Added NetworkFault enum: Partition, Heal, SetDropRate
    • Added Partition struct with side_a, side_b, asymmetric fields
    • Added NetworkState with blocked-pair tracking and LCG-based message dropping
    • Modified deliver_actions_tagged → deliver_actions_tagged_with_net (respects network faults)
    • Existing 6 distribution tests unaffected (backward compatible)
  3. 15 new cluster scenario tests (crates/simulation/tests/cluster_scenarios.rs):
    • Symmetric partition (split-brain, each side forms sub-cluster)
    • Asymmetric partition (one-way communication)
    • 10% message loss (converges with tuned timeouts)
    • 30% message loss (degrades but doesn't crash)
    • Seed node death (cluster survives without seed)
    • Simultaneous 2-node failure
    • Cascading sequential failure (3 nodes killed over time)
    • Large cluster (50 nodes)
    • Rapid churn (kill/revive cycles)
    • Crash detection speed (bounded detection time)
    • Partition + kill in minority side
    • Actor resolution during partition
    • Dissemination completeness (10-node cluster, all detect death)
    • Sequential partitions (fragment cluster)
    • Brief message loss recovery

Key Findings

  • SWIM does not auto-rediscover dead-declared nodes after partition heals. Once the suspicion timeout expires and a node is declared dead, it's permanently removed. Re-discovery requires the join protocol.
  • Message loss is highly destabilizing for SWIM because it affects both the direct probe AND the indirect probe simultaneously. Even 15% loss with default config can cause false deaths.
  • Tuning suspicion_timeout and indirect_probes is critical for lossy networks. Higher values tolerate more loss but increase detection latency.
  • The LCG PRNG for message dropping needs a non-zero seed to avoid correlated early values.

Next Steps

  1. Depth: Property-based invariant checking — Add formal SWIM invariants (completeness, accuracy) as automated property checks
  2. Message reordering — Add out-of-order delivery to the network model
  3. Kademlia-specific scenarios — Test routing table convergence under churn, directory repair after death
  4. Suspicion refutation tests — Verify incarnation bump prevents false death declarations
  5. Graceful leave protocol — Wire node.leave() into the simulation (currently only crash-stop)
  6. BUGGIFY-style injection — Add probabilistic fault injection at protocol decision points
  7. Study more codebases — tikv/raft-rs test harness, al8n/memberlist (Rust port)

Open Questions

  • Should we add a re-join mechanism that fires automatically when a partition heals? (FoundationDB does this; standard SWIM doesn't)
  • Are the 3 pre-existing gossip MT test failures worth investigating? (convergence_curve_is_monotonic_mt, all_nodes_receive_all_keys_in_ring_1000_mt, partition_heals_and_converges_mt)
  • How to model clock skew in a tick-based simulation?

Blockers

  • None currently