swactor/docs/os-design/05-supervision.md

# Supervision — User-Space Self-Healing

## Problem

When nodes churn (spot instances preempted, hardware rebooted, network partitions), actors on those nodes are lost. Something needs to detect the loss and re-spawn the actors on surviving nodes. This is the "self-healing" property of a distributed OS.

## Design Principle: Supervision Is User-Space

Supervision is **not** a runtime primitive. It is a pattern built from:

- **Actor Watching** (01) — detect death
- **Cluster Registry** (02) — re-register under the same name
- **Node Capabilities** (03) — find a suitable replacement node

The runtime provides the low-level mechanisms. Supervision is a library actor that composes them. This keeps the kernel minimal and lets users customize supervision policy without forking the runtime.

## Supervisor Actor

```rust
/// A supervised child definition.
struct SupervisedChild {
    /// Human-readable name (registered in cluster registry).
    name: String,
    /// Factory function to create the actor.
    factory: Box<dyn ActorFactory>,
    /// Placement constraints for the child.
    constraints: PlacementRequirement,
    /// Current address (None if not yet spawned or dead).
    current_addr: Option<ActorAddress>,
    /// Number of restarts so far.
    restart_count: u32,
    /// Maximum restarts before giving up (0 = unlimited).
    max_restarts: u32,
    /// Backoff state for restart delays.
    last_restart: Option<Instant>,
}

/// Restart strategy for a supervisor.
#[derive(Debug, Clone)]
enum RestartStrategy {
    /// Restart only the failed child.
    OneForOne,
    /// If any child fails, restart all children.
    AllForOne,
    /// Don't restart — just notify (for monitoring supervisors).
    Notify,
}

/// The supervisor actor.
struct Supervisor {
    children: Vec<SupervisedChild>,
    strategy: RestartStrategy,
}
```

## Message Protocol

```rust
/// Messages the supervisor handles.
enum SupervisorMsg {
    /// A watched child died.
    Exited(ActorExited),
    /// External request to add a child.
    AddChild {
        name: String,
        factory: Box<dyn ActorFactory>,
        constraints: PlacementRequirement,
    },
    /// External request to remove a child.
    RemoveChild { name: String },
    /// Query: what children are running?
    Status { reply_to: ActorAddress },
}
```

## Lifecycle

### Startup

```rust
impl Supervisor {
    fn start(&mut self, ctx: &Ctx) {
        for child in &mut self.children {
            match self.spawn_child(ctx, child) {
                Ok(addr) => {
                    child.current_addr = Some(addr);
                    ctx.watch(addr);
                    ctx.register_as(&child.name); // or register the child
                }
                Err(e) => {
                    eprintln!("supervisor: failed to spawn {}: {e}", child.name);
                }
            }
        }
    }
}
```

### Death Handling

```rust
impl ActorInterface for Supervisor {
    type Incoming = SupervisorMsg;
    type Response = ();

    fn handle(&mut self, ctx: &Ctx, msg: SupervisorMsg) {
        match msg {
            SupervisorMsg::Exited(exited) => {
                match self.strategy {
                    RestartStrategy::OneForOne => {
                        self.restart_one(ctx, &exited);
                    }
                    RestartStrategy::AllForOne => {
                        self.restart_all(ctx);
                    }
                    RestartStrategy::Notify => {
                        // Just log — don't restart
                    }
                }
            }
            // ... other messages ...
        }
    }
}
```

### Restart Flow (OneForOne)

```
1. Receive ActorExited { addr: X, reason: NodeDown }
2. Find child with current_addr == X → child "worker-3"
3. Check restart_count < max_restarts
4. Query capabilities: find_suitable_nodes(child.constraints)
5. Pick best node (least loaded, or local if possible)
6. Spawn child on selected node via factory
7. Watch new address
8. Register child.name → new address in cluster registry
9. Update child.current_addr
10. Increment child.restart_count
```

```rust
impl Supervisor {
    fn restart_one(&mut self, ctx: &Ctx, exited: &ActorExited) {
        let child = match self.children.iter_mut()
            .find(|c| c.current_addr == Some(exited.addr))
        {
            Some(c) => c,
            None => return, // not our child
        };

        child.current_addr = None;

        if child.max_restarts > 0 && child.restart_count >= child.max_restarts {
            eprintln!(
                "supervisor: child {} exceeded max restarts ({}), giving up",
                child.name, child.max_restarts,
            );
            return;
        }

        // Spawn replacement
        match self.spawn_child(ctx, child) {
            Ok(addr) => {
                child.current_addr = Some(addr);
                child.restart_count += 1;
                ctx.watch(addr);
                // Re-register name → new address
                // (done via the registry, which gossips to all nodes)
            }
            Err(e) => {
                eprintln!("supervisor: failed to restart {}: {e}", child.name);
            }
        }
    }
}
```

### Restart Flow (AllForOne)

When any child dies:
1. Stop all other children (send stop signal).
2. Wait for all `ActorExited` notifications.
3. Restart all children in order.

This is useful for interdependent actor groups where partial restart doesn't make sense.

## Capability-Aware Placement

The supervisor uses `find_suitable_nodes()` from the distribution crate:

```rust
fn spawn_child(&self, ctx: &Ctx, child: &SupervisedChild)
    -> Result<ActorAddress, String>
{
    // If constraints are empty, spawn locally
    if child.constraints.is_empty() {
        return child.factory.spawn(ctx);
    }

    // Find suitable remote nodes
    let nodes = dist_node.find_suitable_nodes(&child.constraints);
    if nodes.is_empty() {
        return Err("no nodes satisfy placement constraints".into());
    }

    // Pick the least-loaded suitable node
    let target_node = &nodes[0]; // TODO: sort by load

    // Spawn remotely (requires remote spawn protocol — future work)
    // For now: if current node satisfies, spawn locally
    // Otherwise: send spawn request to target node
    todo!("remote spawn")
}
```

**Note**: Remote spawn (telling another node to create an actor) is not yet part of the runtime. The supervisor design accounts for it, but the initial implementation will only support local spawn + re-registration.

## Relationship to Existing `spawn_restartable`

The core runtime already has `spawn_restartable` with `factory` and `max_restarts`. This is a **local-only** recovery mechanism — when an actor panics, the same worker restarts it.

The supervisor pattern extends this to **cluster-wide** recovery:

| Feature | `spawn_restartable` | Supervisor |
|---|---|---|
| Scope | Single worker | Cluster-wide |
| Trigger | Panic | Panic, stop, or node death |
| Placement | Same worker | Capability-aware, any node |
| Naming | No | Yes (cluster registry) |
| Strategy | Always restart | OneForOne, AllForOne, Notify |
| Implementation | Runtime internal | User-space actor |

They complement each other: `spawn_restartable` handles fast local recovery (no network round-trip); the supervisor handles node-level failures.

## Future Extensions

- **Restart backoff**: exponential backoff between restarts to avoid thrashing.
- **Health checks**: periodic health probes (not just death detection).
- **Cascading supervisors**: supervisor trees (supervisor watches sub-supervisor).
- **Declarative spec**: TOML/YAML file defining supervision topology, loaded at startup.
- **Migration (not restart)**: move a running actor's state to another node (requires persistence, out of scope).

## Files

This is a library actor, not a runtime change. Implementation lives in:

| File | Content |
|------|---------|
| `crates/supervision/src/lib.rs` | `Supervisor`, `SupervisedChild`, `RestartStrategy` |
| `crates/supervision/src/factory.rs` | `ActorFactory` trait, factory registry |
| `crates/supervision/Cargo.toml` | Depends on `swactor`, `swactor-capabilities`, `distribution` |

Or, if the scope doesn't warrant a separate crate, it can live in `src/supervision.rs` behind a feature flag.

## Tests

- **one_for_one_restart**: supervisor with 3 children, kill one, verify only that one restarts
- **all_for_one_restart**: supervisor with 3 children, kill one, verify all restart
- **max_restarts_exceeded**: child dies repeatedly, verify supervisor gives up after max
- **name_re_registration**: child dies and restarts, verify name resolves to new address
- **capability_placement**: child with GPU constraint, verify spawned on GPU node (or error if none available)
- **supervisor_itself_dies**: verify children are stopped (or orphaned — design decision)