swactor/docs/os-design/05-supervision.md

Supervision — User-Space Self-Healing

Problem

When nodes churn (spot instances preempted, hardware rebooted, network partitions), actors on those nodes are lost. Something needs to detect the loss and re-spawn the actors on surviving nodes. This is the "self-healing" property of a distributed OS.

Design Principle: Supervision Is User-Space

Supervision is not a runtime primitive. It is a pattern built from:

• Actor Watching (01) — detect death
• Cluster Registry (02) — re-register under the same name
• Node Capabilities (03) — find a suitable replacement node

The runtime provides the low-level mechanisms. Supervision is a library actor that composes them. This keeps the kernel minimal and lets users customize supervision policy without forking the runtime.

Supervisor Actor

/// A supervised child definition.
struct SupervisedChild {
    /// Human-readable name (registered in cluster registry).
    name: String,
    /// Factory function to create the actor.
    factory: Box<dyn ActorFactory>,
    /// Placement constraints for the child.
    constraints: PlacementRequirement,
    /// Current address (None if not yet spawned or dead).
    current_addr: Option<ActorAddress>,
    /// Number of restarts so far.
    restart_count: u32,
    /// Maximum restarts before giving up (0 = unlimited).
    max_restarts: u32,
    /// Backoff state for restart delays.
    last_restart: Option<Instant>,
}

/// Restart strategy for a supervisor.
#[derive(Debug, Clone)]
enum RestartStrategy {
    /// Restart only the failed child.
    OneForOne,
    /// If any child fails, restart all children.
    AllForOne,
    /// Don't restart — just notify (for monitoring supervisors).
    Notify,
}

/// The supervisor actor.
struct Supervisor {
    children: Vec<SupervisedChild>,
    strategy: RestartStrategy,
}

Message Protocol

/// Messages the supervisor handles.
enum SupervisorMsg {
    /// A watched child died.
    Exited(ActorExited),
    /// External request to add a child.
    AddChild {
        name: String,
        factory: Box<dyn ActorFactory>,
        constraints: PlacementRequirement,
    },
    /// External request to remove a child.
    RemoveChild { name: String },
    /// Query: what children are running?
    Status { reply_to: ActorAddress },
}

Lifecycle

Startup

impl Supervisor {
    fn start(&mut self, ctx: &Ctx) {
        for child in &mut self.children {
            match self.spawn_child(ctx, child) {
                Ok(addr) => {
                    child.current_addr = Some(addr);
                    ctx.watch(addr);
                    ctx.register_as(&child.name); // or register the child
                }
                Err(e) => {
                    eprintln!("supervisor: failed to spawn {}: {e}", child.name);
                }
            }
        }
    }
}

Death Handling

impl ActorInterface for Supervisor {
    type Incoming = SupervisorMsg;
    type Response = ();

    fn handle(&mut self, ctx: &Ctx, msg: SupervisorMsg) {
        match msg {
            SupervisorMsg::Exited(exited) => {
                match self.strategy {
                    RestartStrategy::OneForOne => {
                        self.restart_one(ctx, &exited);
                    }
                    RestartStrategy::AllForOne => {
                        self.restart_all(ctx);
                    }
                    RestartStrategy::Notify => {
                        // Just log — don't restart
                    }
                }
            }
            // ... other messages ...
        }
    }
}

Restart Flow (OneForOne)

1. Receive ActorExited { addr: X, reason: NodeDown }
2. Find child with current_addr == X → child "worker-3"
3. Check restart_count < max_restarts
4. Query capabilities: find_suitable_nodes(child.constraints)
5. Pick best node (least loaded, or local if possible)
6. Spawn child on selected node via factory
7. Watch new address
8. Register child.name → new address in cluster registry
9. Update child.current_addr
10. Increment child.restart_count

impl Supervisor {
    fn restart_one(&mut self, ctx: &Ctx, exited: &ActorExited) {
        let child = match self.children.iter_mut()
            .find(|c| c.current_addr == Some(exited.addr))
        {
            Some(c) => c,
            None => return, // not our child
        };

        child.current_addr = None;

        if child.max_restarts > 0 && child.restart_count >= child.max_restarts {
            eprintln!(
                "supervisor: child {} exceeded max restarts ({}), giving up",
                child.name, child.max_restarts,
            );
            return;
        }

        // Spawn replacement
        match self.spawn_child(ctx, child) {
            Ok(addr) => {
                child.current_addr = Some(addr);
                child.restart_count += 1;
                ctx.watch(addr);
                // Re-register name → new address
                // (done via the registry, which gossips to all nodes)
            }
            Err(e) => {
                eprintln!("supervisor: failed to restart {}: {e}", child.name);
            }
        }
    }
}

Restart Flow (AllForOne)

When any child dies:

1. Stop all other children (send stop signal).
2. Wait for all ActorExited notifications.
3. Restart all children in order.

This is useful for interdependent actor groups where partial restart doesn't make sense.

Capability-Aware Placement

The supervisor uses find_suitable_nodes() from the distribution crate:

fn spawn_child(&self, ctx: &Ctx, child: &SupervisedChild)
    -> Result<ActorAddress, String>
{
    // If constraints are empty, spawn locally
    if child.constraints.is_empty() {
        return child.factory.spawn(ctx);
    }

    // Find suitable remote nodes
    let nodes = dist_node.find_suitable_nodes(&child.constraints);
    if nodes.is_empty() {
        return Err("no nodes satisfy placement constraints".into());
    }

    // Pick the least-loaded suitable node
    let target_node = &nodes[0]; // TODO: sort by load

    // Spawn remotely (requires remote spawn protocol — future work)
    // For now: if current node satisfies, spawn locally
    // Otherwise: send spawn request to target node
    todo!("remote spawn")
}

Note: Remote spawn (telling another node to create an actor) is not yet part of the runtime. The supervisor design accounts for it, but the initial implementation will only support local spawn + re-registration.

Relationship to Existing spawn_restartable

The core runtime already has spawn_restartable with factory and max_restarts. This is a local-only recovery mechanism — when an actor panics, the same worker restarts it.

The supervisor pattern extends this to cluster-wide recovery:

Feature	spawn_restartable	Supervisor
Scope	Single worker	Cluster-wide
Trigger	Panic	Panic, stop, or node death
Placement	Same worker	Capability-aware, any node
Naming	No	Yes (cluster registry)
Strategy	Always restart	OneForOne, AllForOne, Notify
Implementation	Runtime internal	User-space actor

They complement each other: spawn_restartable handles fast local recovery (no network round-trip); the supervisor handles node-level failures.

Future Extensions

• Restart backoff: exponential backoff between restarts to avoid thrashing.
• Health checks: periodic health probes (not just death detection).
• Cascading supervisors: supervisor trees (supervisor watches sub-supervisor).
• Declarative spec: TOML/YAML file defining supervision topology, loaded at startup.
• Migration (not restart): move a running actor's state to another node (requires persistence, out of scope).

Files

This is a library actor, not a runtime change. Implementation lives in:

File	Content
crates/supervision/src/lib.rs	Supervisor, SupervisedChild, RestartStrategy
crates/supervision/src/factory.rs	ActorFactory trait, factory registry
crates/supervision/Cargo.toml	Depends on swactor, swactor-capabilities, distribution

Or, if the scope doesn't warrant a separate crate, it can live in src/supervision.rs behind a feature flag.

Tests

• one_for_one_restart: supervisor with 3 children, kill one, verify only that one restarts
• all_for_one_restart: supervisor with 3 children, kill one, verify all restart
• max_restarts_exceeded: child dies repeatedly, verify supervisor gives up after max
• name_re_registration: child dies and restarts, verify name resolves to new address
• capability_placement: child with GPU constraint, verify spawned on GPU node (or error if none available)
• supervisor_itself_dies: verify children are stopped (or orphaned — design decision)