Trim the datastream crate to its dumb-pipe core: drop the frame-timing sidecar, make mux positions gap-free for accepted frames, and simplify the endpoint fanout.
- mux: defer position assignment from submit to drain so an overflowed submission no longer consumes a position (no synthetic gaps); submit now returns bool and the Mutex<Receiver> is removed since positions are assigned only to accepted frames
- endpoint/mux: switch from std::sync::mpsc to crossbeam-channel and drop per-event event_matches_request filtering — request filters now apply only to the initial catalog snapshot, and future events broadcast to all subscribers
- endpoint (DeliveryFanout): snapshot sender handles under the lock and deliver outside it via FanoutTarget/FanoutReport, so large batches or slow subscribers no longer block subscribe/snapshot control-plane ops
- emit/endpoint/producer: drop set_frame_timing_enabled/frame_timing_enabled and the Position return from submit_record/submit_text/submit_bytes, and add submit_text_owned taking owned String
- timing/lib/spec: delete the timing module and FRAME_TIME_CHANNEL/FRAME_TIME_CHANNEL_ID/FrameTimeSample re-exports (including the auto-registered timing channel in ChannelCatalogState) and renumber the DATASTREAM_SPEC.md section references across frame/ingest/store/mux
- tests: remove the 567-line shared datastream_support/mod.rs harness
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Make single-threaded tick driving observable so callers can tell whether a tick actually performed work.
- src/runtime.rs: add try_tick() returning whether any worker did work, add has_work() reporting schedulable work, and reduce tick() to a thin wrapper that ignores try_tick()'s result
- src/worker.rs: extract the fast-idle predicate into a reusable Worker::has_work() (backlog plus non-empty spawn/transfer/admin queues plus pending extension work) and reuse it from both the new runtime has_work() and the idle short-circuit in tick_once()
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add a RuntimeAdmin control plane for out-of-band inspection and mutation of live actors, bypassing their normal message handlers.
- src/admin.rs: add RuntimeAdmin + Admin<T> reply handle, the AdminCommand enum (ListActors, InspectActor, GetActorState, ReplaceActorState, StopActor, SuspendActor, ResumeActor), typed result types (ActorSummary/ActorStatus/ActorStateSnapshot/ListActorsResponse), and AdminError (ActorNotFound/AddressMismatch/TypeMismatch/Timeout)
- src/actor.rs: expose type-erased accessors on AnyActor — metadata() returning ActorTypeMetadata plus as_any()/as_any_mut() for downcasting — and Actor::inner()/replace_inner() so admin can snapshot or swap concrete actor state
- src/runtime.rs: wire a per-worker admin channel (admin_txs), expose Runtime::admin(), and implement the RuntimeAdmin ops that resolve an address to its owning worker, send the command, notify that worker, and return an Admin<T> handle; Admin::recv_ticking drives ticks and collects the reply
- src/worker.rs: add an admin_rx receiver and a drain_admin tick phase run before actor handlers, plus ActorPool helpers (actor_summary, get_actor_erased[_mut], suspend/resume/stop_actor_admin) that apply commands on the worker thread; fold admin_rx into the fast-idle has_work/tick_once checks
- tests/runtime_admin.rs: add 608 lines of end-to-end coverage for list/inspect/get/replace/stop/suspend/resume and the type-mismatch and not-found error paths
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Replace stdout-parsed runtime-ready detection with an explicit, plugin-driven bootstrap-completion step and actorize SSH bootstrap teardown.
- provisioning: drop the `PluginObservation::RuntimeReady` variant and add `ProvisionPlugin::complete_bootstrap`, an explicit per-node completion hook (no-op for `LocalDockerPlugin`)
- bootstrap_datastream: remove `parse_runtime_ready`/`RuntimeReadyLine` so bootstrap no longer infers readiness from a parsed stdout JSON line
- vastai_provisioning: drop the `ReadyTrackingSink` ready-flag wrapper; the SSH retry loop now runs purely `while !stopping`, and `complete_bootstrap` stops the node's bootstrap with `BootstrapStopReason::RuntimeReady`
- vastai_provisioning: actorize teardown as `SshBootstrapActor` on the swactor `Runtime` (handle holds an `ActorAddress`), with `stop_bootstrap(handle, reason)` delivering a `Stop` message; add `BootstrapStopReason::{RuntimeReady,NodeStop}`
- actors/provisioner: replace the `RuntimeReady` observation arm with a `ProvisionerMsg::RuntimeReady` handler that calls `complete_bootstrap` then `mark_live`/emits NodeLive (or NodeFailed on error)
- callers/tests: wire the new explicit ready flow through node_agent, the orchestrator/worker_node binaries, and `mvp_one_node_chat`; add the `ssh_bootstrap_actor_stop_kills_child` test
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Stand up an interactive end-to-end chat over a CUDA GPU, provisioning a Dockerized node that loads a GGUF model and serves prompts over TCP.
- prompt_rpc: add the newline-JSON prompt protocol (`SubmitPrompt` + `PromptEvent::{TextDelta,Done,Fault}`) carried over TCP
- mvp_chat: add an interactive REPL client connecting to the prompt RPC port (default 127.0.0.1:19777)
- mvp_orch_one_node / mvp_one_node_chat: add the single-node orchestrator that provisions a `LocalDockerPlugin` node, loads `bartowski/Llama-3.2-1B-Instruct-GGUF` (Q4_K_M), and exposes the prompt RPC listener with boot/route/weight timeouts
- mvp_node: add the GPU worker binary that spawns `tinygrad_worker.py` (default device CUDA) and ships runtime telemetry via a `ClusterFrameSink`
- vastai_provisioning / bootstrap_datastream: add the vast.ai provider adapter (`VastAiProvisioningConfig`, `VastAiLeaseClient`) wrapping `swactor_vastai`, plus a bridge that folds provision stdout onto a per-node datastream
- apps/mvp-node: add CUDA base/runtime Dockerfiles (nvidia/cuda 12.6.3, tinygrad 0.12.0, sshd), `mvp_entrypoint.sh` (sshd + mvp-node, held for postmortem), `local_docker_e2e.sh`, the GGUF tinygrad worker, and one-node-chat/bootstrap/vastai guarantee tests
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Treat Docker as a first-class provisioning provider for the local e2e cluster and add the weight-shard fetch/validate/bind lifecycle behind it.
- docker_cluster_provisioning: model Docker as a provider adapter behind a `DockerCli` boundary (`run_container`/`inspect_ssh_endpoint`/`remove_force`) exposing only per-node ownership primitives
- local_e2e_cluster: wire Docker provisioning into the e2e driver (default `swactor-mvp-local-e2e-cluster` image) across the bootstrap/teardown flow
- weight_shards: add `ModelArtifactRef` (parses `hf://repo@rev/path`), `ShardAssignment`, `ShardManifest`, and `ValidatedShard` with digest-based validation
- shard_fetch: add `ShardLocator` (digest/split/stage -> uri + cache key), `ShardCache`/`ShardFetcher` traits, a `ShardFetchCoordinator`, and typed `FetchError`s
- shard_weight_lifecycle: add the `ShardWeightLifecycle` state machine (Idle->Assigned->Located->Fetching->Fetched->Validating->Binding->Ready/Faulted) with a `WorkerShardBinder` trait
- provisioner/telemetry: route provision logs onto the datastream via per-node/stream channels (`submit_bytes`) and add shard_fetch/shard_weight_lifecycle/weight_shards guarantee tests
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add a universal live datastream explorer view to the HTTP dashboard and point the e2e cluster at it by default.
- live_explorer: add the `LiveDatastreamExplorer` `DashboardView`, registered in `start_dashboard`, served at `/view/datastream/live` over the existing `/events` and `/api/frames` endpoints
- live_explorer_page: ship a 500-line self-contained page that auto-connects, renders streams/channels/frames metrics, and supports level drill-down (streams->channels->frames), payload-kind filtering, and per-channel retention limits
- local_e2e_cluster: switch the dashboard's default opened URL from `/view/mvp/cluster` to `/view/datastream/live`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
node_provisioning: NodeManager record FSM and transient BootstrapSession SSH core per
the provisioning spec. docker_cluster_provisioning provider; add guarantee tests for
both.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Introduce pool/planner/launcher/runtime_stack/model/roles primitives for topology
construction and cluster launch. Drop the core guarantees module entirely; rework
worker bootstrap.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Fold the fragmented distribution swim/routing/gossip tests into swim_core, routing,
and swim_actor. Prune the dashboard tui and command surfaces. Add mvp-system actors
(node_agent, orchestrator, stage_controller), the local_e2e harness, and gpu worker
e2e.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Move iroh_driver and the relay binary out of distribution into a dedicated
crates/iroh-driver (lib re-exports IrohDriver; relay bin renamed). Remove the node crate
and the single-gpu-inference example; drop the docker/datastream demo. Slim
pipeline-parallel vastai.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Remove the datastore crate, the top-level design/orchestration/ring specs, the
benches, and the ci config. Add the dashboard host telemetry sampler
(cpu/disk/net/gpu/mem). Localize the pipeline-parallel e2e stub/mock paths.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
SwimTelemetry observer records probe RTTs (p50), recent targets, and membership
transitions for the node's telemetry tick. Expand registry/registry_actor, datastream
catalog/emit/source, and the pipeline-parallel cluster/fleet. Wire the node main loop
to emit host/runtime/transport/membership frames.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Promote pipeline-parallel-inference to a first-class app and consolidate observability on the datastream wire, decoupling the dashboard crate from `distribution`.
- apps/pipeline-parallel-inference: move the example out of `examples/` into `apps/` as its own workspace, rename binaries to `pp-worker`/`pp-orchestrator`, and strip release binaries
- cluster: add `ClusterNode`, a synchronous facade over the actorized distribution protocol (IrohDriver + per-node Runtime hosting Swim/Registry/Metadata/Directory actors with a `MembershipFanout`), replacing ad-hoc `driver.node()`/`tick()` call sites
- fleet: add per-node fleet telemetry that ships identity/resource records as `DatastreamFrame`s over the cluster transport to the orchestrator's `DatastreamSink`, folded into a `FleetView` on a 3s tick
- provision: add best-effort, opt-in SSH boot-phase telemetry (`PP_DEPLOY_KEY`) that streams rented-node boot logs onto the orchestrator's datastream as `proc.boot.<stage>.*`
- dashboard: rewire the crate dependency from `distribution` to `datastream`, drop the standalone `swactor-datastream-dashboard` binary, and rewrite `datastream_source.rs` to demux per-node frames into Overview/Distribution/Fleet views with live-node TTL filtering
- distribution: refresh dist/netmap plugin copy and README from "Kademlia routing" to gossip-directory terminology
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Strip the collector/aggregator/postproc/snapshot, vastai sampler+shipper,
host/iroh/subprocess/swim introspection, relay observability, sink/spool, and the diag
binaries; drop the t_diag_* tests. Remove DiagEvent emission from iroh_driver. Add
datastream emit/wire (mux + NoopSink/UdpFrameSink/ClusterFrameSink) and rewire the
dashboard onto datastream_source.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Adds the pipeline-stage host kind wrapping the production stage-supervisor lifecycle
(Cold/Registering/Running/Halted per RELAY_SPEC), N3 relay calibration scenarios
(own-relay and canary-relay, real-worker and stub), relay invariant and
scenario-validation tests, and an adversarial judge test suite hunting spec/code gaps.
Expands SIM_SPEC to a three-layer failure decomposition.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Restructures the crate to the SIM_SPEC six-component model (engine, network, host,
bundle, evaluator, scenario) and relocates SIM_SPEC.md into the crate. Adds the SWIM
host adapter wrapping production distribution::swim::SwimNode, the swim codec, a
virtual network model, the scenario loader, an assertion evaluator, deterministic RNG,
property-based tests, and reproduction/smoke/topology scenario TOMLs; drops the
detector/divergence/lint/postproc/spec/runtime modules and prior spec docs.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Appends section 7.7 clarifying that the determinism contracts bind only the
simulator's own components: hidden entropy inside a wrapped production dependency
(allocator state, HashMap RandomState, process-local entropy) is out of scope, and
property tests comparing two independently-allocated wrapped state machines should
not be written.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Six per-component property-test subsections (§4.10 engine, §5.8 network,
§6.4 hosting, §8.4 scenario loader, §9.6 bundle writer, §10.5 assertion
evaluator) written in property-language form. Each names the property the
component must have; how the property is verified is the test author's
call.
These were drafted in conversation but never committed, causing the
boundary check to flag SIM_SPEC.md on iteration 1 of the implementation
loop. Committing now puts them at HEAD so future loop iterations diff
cleanly.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Upgrades iroh/iroh-relay across datastore, distribution, node, and integration,
adapts iroh_driver to the new Endpoint::builder(Minimal).relay_mode(...) API, and
removes the vendored ed25519-dalek patch now that 0.98 resolves the upstream compile
errors. Adds the SIM_SPEC.md simulator MVP spec under pipeline-parallel-inference.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Replaces the generic SimNode/gossip/dashboard framework with a virtual-time
discrete-event engine (priority queue ordered by time/node/fiber/seq), a TOML spec
parser, bundle writer, replay, divergence detector, lint, and postproc, plus the
parity-bar test harness with fixtures and xtask parity-lock tooling. Rewrites
transport identity/crypto and adds the SPEC/TESTING_SPEC/OBSERVABILITY/NORTH_STAR docs.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Structured observability for the iroh/SWIM layer: Aggregator, typed Event/Snapshot
types, Sink (NoopSink default), ProbeScheduler, process stats, and host/iroh/swim
introspection, plus the swactor-diag-collector, -postproc, and -iroh-relay binaries
that assemble and render per-run bundles. Generalizes the pipeline-parallel-inference
example to N stages and adds the topology-planner spec.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Extend the single-GPU example into a two-node pipeline-parallel run that splits llama3.2:1b across two rented vast.ai GPUs and closes the autoregressive decode loop over iroh.
- topology: add linear-chain helpers where each stage derives its neighbours locally from `STAGE`/`NUM_STAGES`, registering `pp-entry`/`pp-exit`/`pp-stage-{i}` SWIM names
- messages: add `StageActivation` (bf16 hidden-state hand-off carrying position/seq_len/is_prefill) and `NextToken` (sampled-token feedback with a `done` flag) that close the autoregressive loop between stage 0 and stage 1
- stage_actor: add `Stage0Actor` (tokenize -> embed_and_forward -> prefill activation; decode_step on each NextToken) and `Stage1Actor` (forward_and_sample -> NextToken back; emit InferenceResponse on EOS/max_tokens)
- vastai: fork the client and add `create_pipeline_instances` (rents one instance per stage, threading `STAGE`/`NUM_STAGES`, best-effort destroys on partial failure) and `destroy_all_instances`
- pp_tinygrad_worker.py: per-stage worker slicing `model.blk[start:end]` in stub and real (GGUF) modes, plus new `pp_gpu_node`/`pp_smoke_run` binaries and ROADMAP/SPEC/TEST_SPEC docs
- reuse: build on the single-GPU example's iroh transport and process bridge unchanged; add actor/codec/topology/integration test suites
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add a complete single-GPU distributed-inference example that rents a vast.ai GPU, boots a worker container, and runs a prompt end-to-end over iroh/SWIM.
- examples/single-gpu-inference: add the `single_gpu_inference` orchestrator binary that starts a local iroh node, waits for the remote gpu-node to register the `"inference"` SWIM name, then sends an `InferenceRequest` and prints the response
- examples/single-gpu-inference: add the `gpu_node` binary that joins the cluster via `SEED_ADDR`, spawns an `InferenceActor` over `tinygrad_worker.py`, and registers the `"inference"` bridge
- inference_actor: bridge swactor messaging to a Python child process via stdin/stdout JSON, with `ProcessBridge`/`RequestBridge` adapters that satisfy the single-`Incoming` actor constraint
- iroh_transport: add `IrohActorTransport` that sends `WireEnvelope`s over iroh QUIC uni-streams (connection-cached against early close), plus wire encode/decode and an inbound drain helper
- vastai: add a vast.ai REST client (`find_offer` with reliability/cuda/geo filters excluding CN, `create_instance`, `wait_for_running`, `destroy_instance`) parameterised by a mockable `base_url`
- worker/docs/tests: ship `tinygrad_worker.py`/`echo_worker.py` (newline-JSON, `--stub`/`--model` defaulting to llama3.2:1b), a Dockerfile, Makefile, SPEC, and actor/codec/cluster/integration/vastai test suites
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Using a bounded model checker to provide absolute guarantees on runtime behavior.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Remove the unused TCP transport and dead CLI/simulation scaffolding and collapse scattered single-purpose modules into consolidated files across the dashboard, process, and simulation crates.
- crates/transport: drop the TCP transport (src/tcp.rs and its tcp feature), leaving only ed25519 identity and encoding utilities behind the iroh transport
- crates/dashboard: collapse actor_detail_html/actors_html/dashboard_html/topology_html into a single html.rs, drop command/parse.rs, and inline the trace types into lib.rs
- crates/process: fold driver, pipeline_types, queue, subscriber, waker, and local pipes/signal/wait into local/mod.rs, pipeline.rs, and a unified types.rs, consolidating the public re-exports
- crates/simulation: remove the ci subdirectory (local_sim, sim) and dead node/config/trace modules, and flatten the distribution subdirectory into top-level files
- crates/datastore: remove the unused store_cli, cli, and in-memory storage, and deduplicate crypto.rs across datastore and distribution (about 190 lines of shared code removed)
- crates/distribution: drop dead codec code and trim the messages module
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Reduce idle cpu usage on my main machine from 17% to 1%. Made SWIM gossip more lazy.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Remove co-dependencies for different modules found in `crates` and migrate the development history to a new repository. The docs were stale, and largely not getting used, so simply deleted for now. When code stabilizes more, they will become useful again.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Initial experiments in 'formal verification' of authorization tooling. Current state is not to be trusted, but we are not in a security critical situation, so that is fine.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
