Dead-node reprobe mechanism allows partition-healed nodes to rejoin the
cluster automatically. When a reprobe ping reaches a dead-declared node,
the piggyback exchange triggers incarnation-bump refutation, transitioning
the node back to Alive. Death declarations are re-enqueued fresh before
each reprobe to ensure piggyback carries useful membership info.
Added SWIM property invariant checks (completeness, accuracy, convergence)
as reusable post-condition validators for simulation tests.
Investigated 3 flaky MT gossip tests: rewrote convergence_curve_is_monotonic_mt
(strict monotonicity invalid under non-atomic MT snapshots), tuned
partition_heals_and_converges_mt (reduced nodes, relaxed threshold),
documented all_nodes_receive_all_keys_in_ring_1000_mt (stable in isolation).
Authored by Claude, lovingly guided by Zachery Aaron Shores-Chmielewski
Add network partition, asymmetric partition, and message loss simulation
to the distribution test harness. 15 new behavioral tests covering
split-brain, cascading failure, seed node death, rapid churn, 50-node
clusters, and actor resolution under partition.
Research notes from studying FoundationDB DST, Hashicorp memberlist,
Antithesis, TigerBeetle VOPR, Turmoil/MadSim, and Jepsen nemeses.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
