swactor-development-history/cfuzz/CYCLE_18_SUPERVISOR_STRATEGIES.md

Cycle 18: OneForAll and RestForOne Supervisor Strategies — Development History

Commit: 771c38c · 4 files · 277 insertions, 2 deletions

---

Motivation

Cycle 17 introduced supervision with the OneForOne strategy (only the failed child is restarted). Erlang/OTP defines two additional coordinated restart strategies that handle interdependent children:

• one_for_all — when one child fails, ALL children are restarted (for tightly coupled children that share state assumptions)
• rest_for_one — when one child fails, it and all children started AFTER it are restarted (for chains where later children depend on earlier ones)

These strategies require coordinated shutdown: the supervisor must stop living siblings, wait for all of them to die, then restart the affected set in the original spec order.

Research Detour: SmallBox/InlineAny Optimization

Before choosing this cycle's topic, investigated SmallBox optimization for message dispatch — a 44% queue throughput improvement was measured. However, it was deferred because:

• Requires unsafe code in a core path
• Would touch 32+ call sites across the codebase
• Violates the "src/ structure frozen" constraint

Extended the Supervisor with coordinated strategies instead — higher value, zero risk.

Competitor Analysis

Framework	OneForAll	RestForOne	Coordinated Shutdown
Erlang/OTP	Yes	Yes	Built into supervisor behaviour
Akka	No (different model: Resume/Restart/Stop/Escalate)	No	N/A
Ractor	No	No	N/A
Bastion	Implicit (redundancy groups)	No	Implicit
Swactor	Yes	Yes	Phase-based state machine

Erlang's Coordinated Restart

In Erlang, one_for_all and rest_for_one stop affected children in reverse start order, wait for all to terminate, then restart in start order. This guarantees initialization dependencies are respected.

Implementation

SupervisorPhase State Machine

• Normal — steady state, processing handle_down events normally
• Stopping { awaiting: HashSet<ActorAddress>, restart_set: Vec<usize> } — coordinated shutdown in progress

SupervisorStrategy Extensions

• SupervisorStrategy::OneForAll — all children restarted when one fails
• SupervisorStrategy::RestForOne — failed child + all children after it (in spec order) restarted

Coordinated Restart Flow

1. Child dies → handle_down called
2. Strategy determines affected indices (OneForAll: all, RestForOne: failed + later)
3. begin_coordinated_restart(ctx, indices):
   • Sends stop signals to living siblings in the restart set
   • Transitions to Stopping phase with awaiting set
   • Already-dead children handled: if all targets are already dead, skip to immediate restart
4. Subsequent handle_down calls during Stopping phase:
   • Remove from awaiting set
   • When awaiting is empty → all stopped
5. finish_restart(ctx):
   • Restart all children in the restart set, in spec order
   • Transition back to Normal phase

Refactoring

• check_intensity() factored out of handle_down for restart budget checking — shared by all strategies

Key files modified: src/actor.rs, tests/runtime_api.rs, docs/runtime.md

Design Decisions

• Phase-based state machine — the Stopping phase cleanly separates "waiting for siblings to die" from "normal operation." This prevents races where a new death arrives while a coordinated restart is in progress.
• Stop signals (not kill) — affected siblings are stopped gracefully (PoisonPill semantics), giving them a chance to run on_stop for cleanup. This matches Erlang's terminate/2 being called during supervised shutdown.
• Restart in spec order — children are restarted in the order they appear in the ChildSpec list, regardless of which child triggered the restart. This preserves initialization dependencies.
• Already-dead optimization — if all children in the restart set are already dead (e.g., cascading failures), skip the Stopping phase entirely and restart immediately. Without this, the supervisor would wait forever for Down messages that already arrived.
• Meltdown protection shared — the same max_restarts budget applies across all strategies. OneForAll restarts count as one restart event (not N), matching Erlang's behavior.

Tests Added

3 new tests (138 → 141 total):

• supervisor_one_for_all_restarts_all_on_single_failure — one child panics, all 3 get new addresses
• supervisor_rest_for_one_restarts_rest_after_failed — child_b panics, child_a unchanged, child_b + child_c restarted
• supervisor_one_for_all_waits_for_all_downs_before_restart — verifies coordinated shutdown completes before restart begins

Result

• 141 tests pass (133 behavioral + 7 proptest + 1 doctest)
• Zero warnings, full workspace compiles
• All three Erlang-standard supervision strategies now available: OneForOne, OneForAll, RestForOne