Route cargo lint through repository timeout validation and strict Clippy, reject unregistered timing primitives with compiler-resolved call budgets, bound Nextest concurrency, and replace polling waits in test infrastructure.
Deny workspace warnings and lint suppressions, consolidate Rust and Python coverage under cargo xtask test with 60-second per-test limits, and remove stale flaky, stateful, Docker, and orphaned test artifacts.
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
Add actor-backed manual node provisioning, control-plane endpoints, and fleet UI assets with durable provider lifecycle handling.
Simplify Myelin orchestration, node runtime, staging, and telemetry paths while removing obsolete engine-builder, dashboard-view, and local-mock implementations.
Align runtime delivery, data-plane, distribution, job-runner, process, telemetry, dashboard, Vast.ai integrations, and their tests with the revised actor and transport contracts.
Replace the single-purpose chat runtime with a persistent fleet daemon that provisions, adopts, and controls nodes through the dashboard.
Add distributed job-runner actors and provider-backed deployment so jobs can materialize workspaces, execute remotely, and return outputs over iroh.
Introduce a pure, level-triggered reconciler in `crates/provisioning`
that drives a declared cluster shape toward convergence over the
existing node lifecycle, replacing the edge-triggered imperative node
orchestration in `apps/myelin`.
- `reconcile`/`reconcile_node`/`observe`: pure decider and observation
folder with stable logical-node identity, per-attempt operation
identity, and deterministic retry backoff; `ClusterDriver` is the sole
writer of observed state, coalescing triggers, recording operations as
pending before dispatch, and scheduling timed requeues.
- `IdempotentEffectExecutor`: deduplicates submissions by
`(run_id, logical_node_id, attempt)` and runs provider work on the
engine-hosted blocking substrate, never blocking a reconcile pass.
- Myelin integration: `MyelinEffectBackend` bridges `ProvisionPlugin` to
the executor contract; `LocalProcessPlugin`/`LocalDockerPlugin`
provider adapters; `ProvisionedClusterGuard` pumps triggers,
observations, and due operations.
- Retire the imperative acquire/bootstrap/teardown sequencing across
`apps/myelin` orchestration, staging, observability, and provider
adapters in favor of the declarative driver.
- Move the reconciler specification to `docs/specs/archive`.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add project licensing and a full README, plus a minimal WebAssembly ping-pong example that exercises the actor runtime.
- `LICENSE`: add the full GNU AGPL-3.0 text and set `license = "AGPL-3.0-only"` on every package (`swactor`, each crate, `apps/myelin`, `tools/vastai`, `xtask`)
- `README.md`: rewrite from a stub into a full project overview, covering features (actor_id routing, WASM, iroh QUIC/SWIM, OTP-style std, process manager, zero-copy objects, datastream metrics, dashboard), architecture, examples, developing, status, and license
- `examples/ping-pong`: new standalone workspace (`pingpong` cdylib) where two actors volley on a single-threaded `wasm` runtime driven by a Node host via `tick()`, demonstrating spawning, message passing, and death monitoring (`watching`)
- `examples/ping-pong`: add a host/run harness -- `run.sh` (wasm-pack build + `run.mjs`), `serve.sh` (dashboard + static demo), `index.html`, and a pinned `Cargo.lock`
- `.gitignore`: stop ignoring `.loop/`, `.deployment-notes/`, and `.omp/`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add read-only actor overview and per-actor dossier pages to the dashboard, fed by enriched per-actor runtime snapshots.
- `dashboard/swactor/actor_view`: new `ActorPanelView`, a tolerant frame consumer over `runtime.actors`/`runtime.stats` that folds per-actor snapshots and serves `/view/swactor/actor-overview` (roster) and `/view/swactor/actor-dossier` (per-actor detail), each backed by an embedded HTML template (`actor_overview.html`, `actor_dossier.html`)
- `dashboard`: register both views in `DashboardHandle` and export `actor_overview_view()`/`actor_dossier_view()` from the swactor module
- `swactor` core: enrich `ActorSnapshot` with `actor_type` and `message_type` (populated from `slot.actor.metadata()` in `ActorPool`) and add `ActorAddress::to_full_hex()` for untruncated display
- `myelin/orchestration`: publish actor stats to the dashboard via a `runtime.actors` channel producer (`stats_hook_on`) threaded through the distribution stack, and carry the orchestrator actor address into readiness signaling
- workspace `Cargo.toml`: add `default-members` for native iteration and a centralized `[workspace.dependencies] tokio` so members share one feature set; `dashboard/Cargo.toml` switches to `tokio.workspace = true`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Speed up VastAI provisioning by creating instances directly from a cached offer pool, and replace best-effort SSH readiness with classified, post-grace bootstrap-failure detection plus a dedicated per-node provider-status monitor.
- vastai_provisioning: provision_one now iterates a cached candidate_pool of offers calling create_instance directly (with host blacklist and failed-host dedup) instead of re-running client.provision; plan_first_wave_offers caches planned_offer_pool/planned_offer_ids for reuse
- vastai_provisioning: add VastAiProviderMonitor (background thread + AtomicBool stop + Drop) spawned per node via the new spawn_provider_monitor trait method, polling instance_status and emitting VastAiProviderStatusObserved/PollRetry/StatusFailure and terminal-start failures
- vastai_provisioning: add classify_ssh_observation (auth_denied/refused/timeout) with spawn_classifying_stderr_reader; spawn_retrying_ssh_bootstrap aborts after POST_GRACE_BOOTSTRAP_FAILURE_LIMIT repeated classified failures past the grace window instead of retrying forever
- vastai_provisioning: ssh_endpoint delegates to client.wait_for_ssh_endpoint; VastAiNode carries run_id/node_id/label/sink and emits structured VastAiLeaseReady/SshEndpointDiscoveryStarted/SshEndpointReady/RuntimeReadyAccepted/ContractCleanup events; LifecyclePolicy is threaded into start_bootstrap
- tools/vastai: add fetch_instance_status and wait_for_ssh_endpoint_with_policy, refactor wait_for_running onto fetch_instance_status, and export both plus ProviderInstanceStatus from lib.rs
- tools/vastai/types: add ProviderInstanceStatus (actual/intended status, status_msg, public_ipaddr, ssh_port, disk_usage) with ssh_endpoint() and From<InstanceStatus>
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Complete an 8-stage pipeline-parallel run over VastAI by provisioning stages high-to-low, adding per-stage/per-step metrics, host anti-colocation, and provider state-timeout guardrails.
- orchestrator_app: select the next weight-load stage by max index (provision stages high-to-low for parallel spread), add a throttled "loaded N of M; waiting on stage X" stage_provision_wait headline, and surface min_compute_cap/state_timeout_secs in the config dump.
- orchestrator_app: enrich pipeline_token_in/out and tokenizer_decode events with token_count/token_ids/generated_index.
- worker_node: add timing metrics across the data path (helper_execute_ms, egress_ring_read_ms, send_ms, ingress_ring_write_ms, object_load_ms), refactor take_complete_ingress_record into IngressRecordBytes (object_id/sequence/extent/flags), and emit a new object_loaded event.
- vastai_provisioning: track leased host_ids and blacklist already-leased hosts in later ProvisionRequests so stages don't co-locate, and tag SSH-bootstrap retry logs with the attempt number.
- tools/vastai: add min_compute_cap (PP_MIN_COMPUTE_CAP) filter/search query and a LifecyclePolicy state_timeout (PP_STATE_TIMEOUT_SECS) that fails instances stuck in a non-running status instead of polling forever.
- xtask: raise the check timeout to 1800s/30s grace, drop --skip-rebuild for VastAI, aggregate per-stage StepExecuted metrics, add a vastai summary section, and write failure artifacts on abort.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Land the first working two-stage pipeline-parallel run over VastAI, wiring a real inter-stage data path with observability, a max-price offer cap, and remote-image reuse.
- orchestrator_app: raise the VastAI pipeline-stage cap from 1 to 2 and let VastAI pipeline planning resolve the HuggingFace GGUF from the default local cached-model metadata path instead of requiring host mounts; add --vastai-max-dph-total (CLI/env/TOML) config.
- vastai_provisioning: make complete_bootstrap a no-op so the SSH bootstrap log tail stays alive past runtime-ready until node stop, preserving post-ready worker logs; add a test asserting the tail is only stopped on NodeStop.
- worker_node: emit data-path NodeEvents across the pipeline (iroh_edge_stream_arrived/bytes_read/bytes_sent, egress_ring_read, ingress_ring_write) with edge/byte metadata.
- tools/vastai: add max_dph_total (PP_MAX_DPH_TOTAL) to SelectionPolicy, the reachable-offer filter, and the search query, and improve the empty-pool error message.
- xtask: pass --skip-rebuild for the VastAI scenario and gate it on a new require_vastai_data_path_facts plus GPU facts (ring install, activation object load/step, interstage handoff, iroh edge read/sent).
- mvp_chat: add ChatModelConfig (model id/gguf/tokenizer/max-context) forwarded to the orchestrator; for VastAI + skip-rebuild, emit skip events and reuse the remote node image without a local build.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Move iroh_driver and the relay binary out of distribution into a dedicated
crates/iroh-driver (lib re-exports IrohDriver; relay bin renamed). Remove the node crate
and the single-gpu-inference example; drop the docker/datastream demo. Slim
pipeline-parallel vastai.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Remove co-dependencies for different modules found in `crates` and migrate the development history to a new repository. The docs were stale, and largely not getting used, so simply deleted for now. When code stabilizes more, they will become useful again.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Extract the timer wheel and watch registry from the worker into a reusable std-extension crate, generalize the worker around a RuntimeExtension factory, and decompose the monolithic runtime test file into focused suites.
- crates/std: extract TimerWheel (deterministic tick-counted one-shot/interval timers) and WatchRegistry (target-to-watcher death-notification index) out of src/worker.rs into reusable modules
- crates/std: add Ctx extension traits (CtxMonitoring, CtxNaming, CtxWatching, CtxTimers) and Runtime extension traits (RuntimeNaming, RuntimeWatching, RuntimeGroups) wiring monitor/name/watch/timer/group support
- src/worker.rs: replace the hard-coded timer/watch fields with a generic RuntimeExtension factory and add route_to_pool_or_remote for message routing (local pool, then cross-worker address map, then external inboxes)
- tests: split the 4622-line tests/runtime_api.rs into focused suites (actor_lifecycle, message_delivery, runtime_stress, std_extension) plus a shared tests/common/mod.rs harness, and drop watch_api.rs
- benches/fuzz: add runtime_benchmarks and adjust the runtime fuzz target
- tools/docs: add fn_complexity.py and loc_analysis.py analysis scripts and refresh the runtime and worker-thread docs
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Split the monolithic crate into a Cargo workspace with the Python and Wasm bindings as separate member crates.
- Cargo.toml: declare a `[workspace]` with members `.`/`crates/swactor-python`/`crates/swactor-wasm`, remove the `python` feature and pyo3 dependency, and change root crate-type from `["cdylib","rlib"]` to `["rlib"]`
- crates/swactor-python: new cdylib crate re-exporting the PyO3 bindings (Runtime/RuntimeConfig/RuntimeHandle/Inbox/Ctx/ActorAddress/RuntimeStats), depending on `swactor` + pyo3; pyproject.toml and uv.lock relocated here from the root
- crates/swactor-wasm: new cdylib crate moved from top-level `wasm/`, depending on `swactor` with `no_random` features
- src/actor.rs: widen `Actor::new`, `AnyActor`, `ContextInner`, and `Ctx::raw_inner` to `pub` so the separate binding crates can drive the runtime
- src/lib.rs: delete the in-tree `python` module and `#[pymodule]`, and gate the `no_random` RNG behind `all(feature = "no_random", not(feature = "getrandom"))`
- tools/: relocate package.json/package-lock.json; drop the now-duplicate `wasm/Cargo.lock`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Asking the agent to refactor to reduce spectral complexity, it worked. Trivial change, but this did reduce code complexity.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Generates a report on the connectivity properties of the code interdependency DAG.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
