75 lines
3.2 KiB
Markdown
75 lines
3.2 KiB
Markdown
|
# Dead-Node Reprobe — Design & Rationale
|
|||
|
|
|
|||
|
|
## Problem
|
|||
|
|
|
|||
|
|
When a network partition heals, SWIM nodes on both sides may have declared each
|
|||
|
|
other Dead. The `alive_members()` filter excludes Dead nodes from probing
|
|||
|
|
targets, so neither side initiates communication — creating a **permanent split**
|
|||
|
|
even after connectivity is restored.
|
|||
|
|
|
|||
|
|
The existing refutation mechanism (incarnation bump on learning of own death
|
|||
|
|
declaration) handles resurrection correctly, but depends on someone *telling*
|
|||
|
|
the dead-declared node about its status. With no probes to dead nodes, nobody
|
|||
|
|
does.
|
|||
|
|
|
|||
|
|
## Solution: Independent Dead-Node Reprobe Cycle
|
|||
|
|
|
|||
|
|
Added a lightweight reprobe mechanism to `SwimProbe` that periodically pings
|
|||
|
|
dead nodes. The existing piggyback + refutation mechanism handles the rest:
|
|||
|
|
|
|||
|
|
1. **Reprober** pings dead node with piggybacked "you are Dead(inc=N)"
|
|||
|
|
2. **Target** receives piggyback, sees it's declared Dead → refutes → bumps incarnation
|
|||
|
|
3. **Target** replies with Ack carrying piggybacked "I'm Alive(new_inc)"
|
|||
|
|
4. **Reprober** applies piggyback → target transitions Dead→Alive
|
|||
|
|
|
|||
|
|
Key insight: we re-enqueue the death declaration in the dissemination queue
|
|||
|
|
before packing the reprobe ping's piggyback. Without this, the original death
|
|||
|
|
declaration's transmit budget would be long exhausted, and the piggyback would
|
|||
|
|
carry no useful membership info.
|
|||
|
|
|
|||
|
|
## Design Decisions
|
|||
|
|
|
|||
|
|
### Why inside SwimProbe (not SwimNode)?
|
|||
|
|
|
|||
|
|
- SwimProbe owns the tick counter, sequence counter, and member list access
|
|||
|
|
- All probe-related logic stays in one place
|
|||
|
|
- The reprobe is a simple independent cycle — doesn't interfere with ProbePhase
|
|||
|
|
|
|||
|
|
### Why no new wire messages?
|
|||
|
|
|
|||
|
|
- `SwimAction::SendPing` works identically for normal probes and reprobes
|
|||
|
|
- The ack from a reprobe targets a different sequence than the current probe
|
|||
|
|
cycle, so the probe state machine ignores it — but the piggyback is applied
|
|||
|
|
at the SwimNode layer before the probe state machine sees the ack
|
|||
|
|
|
|||
|
|
### Configuration
|
|||
|
|
|
|||
|
|
- `dead_reprobe_interval: u64` (default: 50 ticks, ~5× probe_interval)
|
|||
|
|
- Set to 0 to disable completely
|
|||
|
|
- Existing test configs use 0 to avoid interference with timing
|
|||
|
|
|
|||
|
|
## Files Changed
|
|||
|
|
|
|||
|
|
| File | Change |
|
|||
|
|
|------|--------|
|
|||
|
|
| `crates/distribution/src/swim/probe.rs` | `dead_reprobe_interval` in config, `maybe_reprobe_dead()` method |
|
|||
|
|
| `crates/distribution/src/swim/member_list.rs` | Added `dead_members()` |
|
|||
|
|
| `crates/distribution/src/swim/node.rs` | Re-enqueue death declaration in `translate_probe_actions` |
|
|||
|
|
| All `SwimConfig` struct literals | Added `dead_reprobe_interval` field |
|
|||
|
|
|
|||
|
|
## Edge Cases
|
|||
|
|
|
|||
|
|
- **Truly dead nodes**: Reprobe ping is lost (no ack), no harm done
|
|||
|
|
- **All members dead**: Reprobe cycles through them round-robin
|
|||
|
|
- **Concurrent reprobe + normal probe**: Independent, different sequences
|
|||
|
|
- **Dissemination budget**: Death re-enqueued fresh each reprobe, not stale
|
|||
|
|
|
|||
|
|
## Alternatives Considered
|
|||
|
|
|
|||
|
|
1. **Dead node grace period + resurrection timer**: More complex, adds new
|
|||
|
|
state tracking alongside suspicion timers. Rejected for simplicity.
|
|||
|
|
2. **Periodic re-join via seed nodes**: Requires seed node availability,
|
|||
|
|
doesn't work when seed is itself dead-declared. Rejected.
|
|||
|
|
3. **Direct liveness inference from Ping reception**: Would require changing
|
|||
|
|
`handle_ping` to special-case pings from dead nodes. More invasive.
|