Zachery Aaron Shores-Chmielewski
d19dd91324
`cargo xtask provisioning-reconciler-demo [--port n] [--nodes n]` boots a
lightweight orchestrator for visual, human-checked E2E confirmation of the
provisioning reconciler: swactor engine + real ClusterDriver + demo
provider, with node children re-exec'ing the same xtask binary in node
role and joining the supervisor over real iroh connections.
- supervisor actor owns driver/provider/shape on a 250ms wall-clock tick,
mirroring the production ClusterReconciler poll semantics; emits
prov.reconciler.events/snapshot plus per-node lifecycle/status streams
- k8s-styled reconciler view: ready/desired header, node stage cards,
commands-out and events-in feeds
- dashboard `demo-control` feature: POST /control/{kill,provision,remove}
+ Fleet Control view; regular builds compile none of it (symbol-verified)
- fleet cards fold proc.<node>.lifecycle and node.status heartbeats into
per-node pid/state pills that stay live
- hardening: exe resolution survives binary replacement by rebuilds,
spawn failures feed back as BootstrapFailed so the reconciler retries
instead of wedging at SshReady, teardown skips exit waits for
never-started children
Verified in-browser: boot 3/3 converged with real joins; dashboard kill
dips and fully recovers with a replacement; provision +1 → 4/4; remove −2
graceful teardown → 2/2; child process count matches reconciler nodes.
44 lines
1.6 KiB
Rust
44 lines
1.6 KiB
Rust
//! Demo-only control plane: write actions out of the dashboard.
|
|
//!
|
|
//! This module exists only under the `demo-control` feature. The dashboard's
|
|
//! data path stays read-only in every regular build; the provisioning
|
|
//! reconciler demo turns this feature on so a human can kill provisioned
|
|
//! processes and request new ones from the fleet view.
|
|
//!
|
|
//! Routes (only present when the feature is enabled and a control sink is
|
|
//! installed):
|
|
//! - `POST /control/kill` body `{"node": "<stream node id>"}`
|
|
//! - `POST /control/provision` body `{"count": 1}`
|
|
|
|
use std::sync::mpsc::Sender;
|
|
use std::sync::OnceLock;
|
|
|
|
use serde::Deserialize;
|
|
|
|
/// A control command issued from the dashboard UI.
|
|
#[derive(Clone, Debug, Deserialize)]
|
|
pub enum ControlCommand {
|
|
/// Kill the process backing the fleet card identified by its stream node.
|
|
Kill { node: String },
|
|
/// Ask the reconciler to provision `count` additional nodes.
|
|
Provision { count: u32 },
|
|
/// Lower the desired cluster size by `count` nodes (graceful scale
|
|
/// down: teardown through the reconciler, not a kill).
|
|
Remove { count: u32 },
|
|
}
|
|
|
|
static CONTROL_SENDER: OnceLock<Sender<ControlCommand>> = OnceLock::new();
|
|
|
|
/// Install the sink that receives dashboard-issued control commands.
|
|
///
|
|
/// Called once by the embedding demo before the HTTP server starts. Without a
|
|
/// sink the control routes answer `503 Service Unavailable`.
|
|
pub fn set_control_sender(sender: Sender<ControlCommand>) {
|
|
let _ = CONTROL_SENDER.set(sender);
|
|
}
|
|
|
|
pub(crate) fn dispatch(command: ControlCommand) -> bool {
|
|
CONTROL_SENDER
|
|
.get()
|
|
.is_some_and(|sender| sender.send(command).is_ok())
|
|
}
|