# Simulation Testing Research

## Sources Studied
- Hashicorp memberlist (Go SWIM) — test methodology, Lifeguard extensions
- FoundationDB — deterministic simulation, BUGGIFY fault injection
- Antithesis — fault injection categories
- TigerBeetle — VOPR simulation, Vortex TCP proxy testing
- Turmoil / MadSim — Rust DST frameworks
- Jepsen — standard nemeses for distributed systems
- Academic: SWIM paper, gossip protocol convergence properties

## Key Concepts

### FoundationDB DST Pattern
- Single-threaded, seeded PRNG, simulated time (discrete-event)
- Same binary for simulation and production (interface abstraction)
- BUGGIFY: two-phase internal fault injection (25% activation, 25% firing)
  - 5 patterns: minimal work, error forcing, concurrency delays, knob randomization, damage control
- Test oracle: reference impl comparison, operation replay, invariant workloads

### Hashicorp Memberlist Test Coverage
- **Probe cycle**: direct ping → indirect ping (PingReq) → TCP fallback → suspect
- **Lifeguard**: Suspicion timer with log(k+1) decay, health-aware probe timeouts, Dogpile confirmation
- **State machine**: Alive → Suspect → Dead with incarnation-based conflict resolution
- **Tests**: ~80 test functions covering join/leave, probe, state transitions, encryption, labels, metadata, PushPull sync
- **Key missing from swactor**: awareness/health scoring, nack-based probing, PushPull full state sync

### Standard Failure Modes (from Jepsen/Antithesis/TigerBeetle)
1. Network partition (symmetric)
2. Asymmetric partition (A→B works, B→A drops)
3. Message loss (random % drop)
4. Message delay/reorder
5. Process crash + restart
6. Slow/degraded node (CPU starvation)
7. Cascading failure (sequential kills)
8. Split-brain (minority vs majority partition)
9. Clock skew (not applicable to our tick-based sim)

### Invariants to Check (SWIM+Kademlia)
- **Completeness**: Every failed node eventually detected by all survivors
- **Accuracy**: No healthy node permanently marked dead
- **Convergence**: Membership views agree within O(log N) rounds
- **Dissemination**: Membership updates reach all nodes
- **Routing table consistency**: k-buckets maintain closest-node invariant
- **Directory repair**: Dead node's entries re-replicated to surviving nodes
- **Cache coherence**: Dead node's cached locations invalidated

## Gaps in Current Test Suite
| Gap | Priority | Notes |
|-----|----------|-------|
| Network partition / split-brain | High | No partition testing exists |
| Message loss (% drop) | High | Sim delivers 100% reliably |
| Asymmetric partition | Medium | One-way failures |
| Seed node failure | High | Current tests only kill non-seed |
| Simultaneous multi-node failure | Medium | Only single kills tested |
| Cascading sequential failure | Medium | Real-world pattern |
| Large cluster (50+) | Medium | Only 5 and 20 tested |
| Rapid churn (join+leave+kill) | High | Realistic workload |
| Graceful leave protocol | Medium | leave() untested in sim |
| Dissemination completeness | High | Not directly verified |
| Suspicion refutation | Medium | Incarnation bump logic |
| Directory repair after death | Medium | repair_queue untested |
| Cache invalidation correctness | Low | Simple but important |