swactor/docs/os-design/04-command-interface.md
Developer 473999d1df feat: actor watching — local death notifications
Add watch/unwatch API to the actor system so actors can monitor each
other's liveness. When a watched actor dies (panic or stop), watchers
receive an ActorExited notification via on_actor_exit().

- ExitReason enum (Stopped, Panicked, NodeDown) and ActorExited struct
- ContextInner::watch()/unwatch() + Ctx typed wrappers
- ActorInterface::on_actor_exit() default method (system message fallback)
- WatchRegistry in worker with bidirectional tracking
- Death notification dispatch as phase 5b in tick_once
- Runtime-level watch for external callers
- 10 behavioral tests in tests/watch_api.rs
- Design documents for OS features in docs/os-design/

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-13 07:25:27 +00:00

15 KiB

Command Interface — Frontend-Agnostic Dispatch

Problem

The runtime has an investigate/REPL protocol (crates/runtime-dashboard/src/investigate.rs) that accepts text commands and returns JSON. It works, but it's hardcoded to stdin/stdout and tightly coupled to the dashboard crate. We need the same commands accessible from:

  • Terminal CLI (stdin/stdout)
  • TUI (the existing ratatui dashboard)
  • REST API (the existing HTTP server)
  • Future: WebSocket, remote CLI, programmatic SDK

And we need write commands (spawn, stop, drain) — not just read-only inspection.

Design

Architecture

 ┌──────────────────────────────────────────────────────────┐
 │                     Frontends                            │
 │                                                          │
 │  ┌──────────┐  ┌─────────┐  ┌──────────┐  ┌──────────┐ │
 │  │CLI / REPL│  │   TUI   │  │REST /api/│  │ Future   │ │
 │  │(stdin/   │  │(ratatui │  │cmd?name= │  │(websocket│ │
 │  │ stdout)  │  │ events) │  │&arg=val  │  │ etc.)    │ │
 │  └────┬─────┘  └────┬────┘  └────┬─────┘  └────┬─────┘ │
 │       │             │            │              │        │
 │       └─────────────┴─────┬──────┴──────────────┘        │
 └───────────────────────────┼──────────────────────────────┘
                             │
                    ┌────────▼────────┐
                    │  CommandRouter  │
                    │                 │
                    │  name → handler │
                    │  dispatch()     │
                    └────────┬────────┘
                             │
              ┌──────────────┼──────────────┐
              │              │              │
       ┌──────▼──────┐ ┌────▼────┐ ┌───────▼───────┐
       │  Built-in   │ │Built-in │ │    Custom      │
       │  Read Cmds  │ │Write    │ │  (actor-       │
       │  (overview, │ │Cmds     │ │   registered)  │
       │   workers,  │ │(spawn,  │ │                │
       │   actors,   │ │ stop,   │ │                │
       │   hot, ...) │ │ drain)  │ │                │
       └─────────────┘ └─────────┘ └────────────────┘

Core Types

// crates/command/src/lib.rs

use std::collections::HashMap;
use std::sync::Arc;

/// A command request.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CommandRequest {
    /// Command name (e.g., "overview", "spawn", "actors").
    pub command: String,
    /// Named arguments. Values are JSON for flexibility.
    pub args: HashMap<String, serde_json::Value>,
}

/// A command response. Always JSON-serializable.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CommandResponse {
    pub ok: bool,
    pub command: String,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub data: Option<serde_json::Value>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub error: Option<String>,
}

impl CommandResponse {
    pub fn ok(command: &str, data: impl Serialize) -> Self {
        Self {
            ok: true,
            command: command.to_string(),
            data: Some(serde_json::to_value(data).unwrap_or(serde_json::Value::Null)),
            error: None,
        }
    }

    pub fn err(command: &str, msg: impl Into<String>) -> Self {
        Self {
            ok: false,
            command: command.to_string(),
            data: None,
            error: Some(msg.into()),
        }
    }

    /// Serialize to a single JSON line (for REPL protocol).
    pub fn to_json_line(&self) -> String {
        serde_json::to_string(self).unwrap_or_else(|e| {
            format!(r#"{{"ok":false,"command":"","error":"serialization: {e}"}}"#)
        })
    }
}

CommandHandler Trait

/// Metadata about a command, used for help text and validation.
pub struct CommandMeta {
    pub name: &'static str,
    pub description: &'static str,
    pub usage: &'static str,
    /// Whether this command mutates state (spawn, stop, etc.)
    pub is_write: bool,
}

/// A command handler. Implementations are stateless — all state
/// comes through CommandContext.
pub trait CommandHandler: Send + Sync {
    fn meta(&self) -> CommandMeta;
    fn handle(&self, args: &HashMap<String, serde_json::Value>, ctx: &CommandContext) -> CommandResponse;
}

CommandContext

/// Context available to command handlers.
///
/// Contains references to runtime subsystems. Optional fields allow
/// commands to work in both standalone and distributed configurations.
pub struct CommandContext {
    pub runtime: Arc<swactor::runtime::Runtime>,
    pub stats_collector: Option<Arc<crate::StatsCollector>>,
    // Distribution (only present when running distributed)
    pub dist_node: Option<Arc<std::sync::Mutex<distribution::node::DistributedNode>>>,
    // Extensibility: arbitrary typed data that custom commands can access
    extensions: HashMap<std::any::TypeId, Box<dyn std::any::Any + Send + Sync>>,
}

impl CommandContext {
    /// Retrieve a typed extension.
    pub fn get_ext<T: 'static + Send + Sync>(&self) -> Option<&T> {
        self.extensions.get(&std::any::TypeId::of::<T>())
            .and_then(|b| b.downcast_ref())
    }

    /// Add a typed extension.
    pub fn set_ext<T: 'static + Send + Sync>(&mut self, val: T) {
        self.extensions.insert(std::any::TypeId::of::<T>(), Box::new(val));
    }
}

CommandRouter

/// Central command dispatch.
pub struct CommandRouter {
    handlers: HashMap<String, Box<dyn CommandHandler>>,
}

impl CommandRouter {
    pub fn new() -> Self {
        Self { handlers: HashMap::new() }
    }

    /// Register all built-in commands.
    pub fn with_builtins(mut self) -> Self {
        self.register(Box::new(builtins::HelpCommand));
        self.register(Box::new(builtins::OverviewCommand));
        self.register(Box::new(builtins::WorkersCommand));
        self.register(Box::new(builtins::WorkerCommand));
        self.register(Box::new(builtins::ActorsCommand));
        self.register(Box::new(builtins::ActorCommand));
        self.register(Box::new(builtins::HotCommand));
        self.register(Box::new(builtins::PhasesCommand));
        self.register(Box::new(builtins::DiffCommand));
        // Write commands
        self.register(Box::new(builtins::SpawnCommand));
        self.register(Box::new(builtins::StopCommand));
        self.register(Box::new(builtins::ShutdownCommand));
        // Distribution-aware commands (no-op if dist_node is None)
        self.register(Box::new(builtins::NodesCommand));
        self.register(Box::new(builtins::RegistryCommand));
        self.register(Box::new(builtins::ResolveCommand));
        self.register(Box::new(builtins::DrainCommand));
        self
    }

    /// Register a custom command handler.
    pub fn register(&mut self, handler: Box<dyn CommandHandler>) {
        let name = handler.meta().name.to_string();
        self.handlers.insert(name, handler);
    }

    /// Dispatch a command request.
    pub fn dispatch(&self, req: &CommandRequest, ctx: &CommandContext) -> CommandResponse {
        match self.handlers.get(&req.command) {
            Some(handler) => handler.handle(&req.args, ctx),
            None => CommandResponse::err(
                &req.command,
                format!("unknown command `{}` — try `help`", req.command),
            ),
        }
    }

    /// List all registered commands (for help text).
    pub fn commands(&self) -> Vec<&CommandMeta> {
        // sorted by name for stable output
        let mut metas: Vec<_> = self.handlers.values()
            .map(|h| h.meta())
            .collect();
        metas.sort_by_key(|m| m.name);
        metas
    }
}

Built-in Commands (MVP)

Read commands (extracted from existing investigate.rs):

Command Args Description Source
help — List all commands cmd_help()
overview — Runtime summary cmd_overview()
workers — Per-worker stats cmd_workers()
worker id: int Single worker detail cmd_worker()
actors sort, limit, worker List actors cmd_actors()
actor prefix: str Find by address prefix cmd_actor()
hot n: int Top N by mailbox depth cmd_hot()
phases worker: int? Tick phase breakdown cmd_phases()
diff seconds: float Snapshot delta cmd_diff()
nodes — Cluster member list new
registry — All registered names new
resolve name: str Look up a name new

Write commands (new):

Command Args Description
stop prefix or name Stop an actor (poison + cleanup)
drain node: str? Stop accepting new actors on a node, let existing drain
shutdown node: str? Graceful shutdown (drain + stop all)
spawn factory, node?, constraints? Spawn from a registered factory

spawn requires a factory registry — actors register factory functions that can be invoked by name:

pub trait ActorFactory: Send + Sync {
    fn name(&self) -> &str;
    fn spawn(&self, runtime: &Runtime, args: &HashMap<String, serde_json::Value>)
        -> Result<ActorAddress, String>;
}

Migration from investigate.rs

The existing crates/runtime-dashboard/src/investigate.rs has 9 command functions. Migration strategy:

  1. Create crates/command/src/builtins/ with one file per command (or grouped by category).
  2. Each cmd_* function becomes a CommandHandler impl. The logic is identical — just restructured.
  3. dispatch_repl becomes CommandRouter::dispatch with a text-to-CommandRequest parser.
  4. dispatch_command (HTTP) becomes CommandRouter::dispatch with query-param-to-CommandRequest parser.
  5. run_investigate remains in the dashboard crate as a thin loop over CommandRouter.

Example extraction:

// crates/command/src/builtins/overview.rs

pub struct OverviewCommand;

impl CommandHandler for OverviewCommand {
    fn meta(&self) -> CommandMeta {
        CommandMeta {
            name: "overview",
            description: "Summary: worker count, actor count, total messages, panics",
            usage: "overview",
            is_write: false,
        }
    }

    fn handle(&self, _args: &HashMap<String, serde_json::Value>, ctx: &CommandContext) -> CommandResponse {
        let stats = ctx.enriched_stats();
        let total_msgs: u64 = stats.workers.iter().map(|w| w.messages_processed).sum();
        // ... same logic as existing cmd_overview ...
        CommandResponse::ok("overview", serde_json::json!({
            "workers": stats.num_workers,
            "actors": stats.actor_details.len(),
            "total_messages_processed": total_msgs,
            // ...
        }))
    }
}

Frontend Adapters

Each frontend is a thin adapter that converts its input format into CommandRequest and CommandResponse back to its output format.

REPL adapter (stdin/stdout):

// crates/command/src/adapters/repl.rs

pub fn parse_line(line: &str) -> CommandRequest {
    let parts: Vec<&str> = line.split_whitespace().collect();
    let command = parts.first().unwrap_or(&"help").to_string();
    let args = parse_positional_and_flags(&parts[1..]);
    CommandRequest { command, args }
}

REST adapter (HTTP query params):

// crates/command/src/adapters/rest.rs

pub fn from_query_params(params: &HashMap<String, String>) -> CommandRequest {
    let command = params.get("cmd").cloned().unwrap_or_else(|| "help".into());
    let args: HashMap<String, serde_json::Value> = params.iter()
        .filter(|(k, _)| *k != "cmd")
        .map(|(k, v)| (k.clone(), serde_json::Value::String(v.clone())))
        .collect();
    CommandRequest { command, args }
}

TUI adapter: TUI input field text → parse_line() → dispatch() → render response in panel.

Custom Commands (actor-registered)

Actors can register command handlers at runtime through the Ctx:

impl Ctx<'_> {
    /// Register a command that routes to this actor.
    /// When the command is invoked, a CommandInvocation message
    /// is sent to this actor's mailbox.
    pub fn register_command(&self, name: &str, description: &str) {
        self.inner.register_command(self.self_addr, name, description);
    }
}

When a custom command is dispatched:

  1. Router finds it's actor-registered.
  2. Sends CommandInvocation { command, args, reply_addr } to the actor's mailbox.
  3. The actor processes it and sends CommandResult { data } back to reply_addr.
  4. Router waits on a one-shot inbox (with timeout, e.g. 5 seconds).
  5. Returns the response.
/// Sent to an actor when its registered command is invoked.
#[derive(Debug, Clone)]
pub struct CommandInvocation {
    pub command: String,
    pub args: HashMap<String, serde_json::Value>,
    pub reply_addr: ActorAddress,
}

/// Sent back by the actor with the command result.
#[derive(Debug, Clone)]
pub struct CommandResult {
    pub data: serde_json::Value,
}

This mechanism allows any actor to expose operational endpoints without modifying the command crate.

Files Modified

File Change
crates/command/ New crate
crates/command/Cargo.toml Dependencies: swactor, serde, serde_json
crates/command/src/lib.rs CommandRouter, CommandHandler, CommandRequest/Response, CommandContext
crates/command/src/builtins/ Built-in command handlers (mod.rs + per-command files)
crates/command/src/adapters/ REPL and REST input parsers
crates/runtime-dashboard/src/investigate.rs Refactored: thin REPL loop over CommandRouter
crates/runtime-dashboard/src/server.rs REST endpoints use CommandRouter
crates/runtime-dashboard/Cargo.toml Depends on crates/command
Cargo.toml Add crates/command to workspace

Tests

  • dispatch_known_command: overview returns ok: true with expected fields
  • dispatch_unknown_command: returns ok: false with helpful error
  • help_lists_all: help response includes all registered command names
  • parse_repl_line: "actors --sort mailbox --limit 5" → correct CommandRequest
  • parse_query_params: {cmd: "actor", prefix: "a1b2"} → correct CommandRequest
  • custom_command_dispatch: register actor command, invoke, verify response
  • custom_command_timeout: registered actor doesn't respond, verify timeout error
  • write_command_stop: stop an actor via command, verify it's poisoned