Integrate namespace and source-route lifecycle changes, contextual process cleanup, Python binding updates, and Myelin worker/orchestrator recovery. Keep the shared control contracts, deployment identity fencing, SSH bootstrap adapters, paid admission accounting, and VastAI cleanup implementation together with their consumers.
Migrate Iroh dependencies and telemetry transport/collection with dashboard and demo callsites, workspace build configuration, and actor-control-flow policy updates. This is an intermediate development checkpoint, not paid-provider qualification.
Review verification: contextual_process_guarantees (4 tests), telemetry_transport (4 tests), and shared control contracts (5 tests) passed. Historical five-node redeployment and campaign execution passed individually; complete ordered qualification remains pending.
Duty mixing between iroh-driver and data-plane is resolved: the transport
crate now owns only byte pumping, and the data-plane owns every edge
semantic.
data-plane:
- ids.rs: single EdgeId/RingId/StreamId/NodeId/RunId/LeaseRequestId/
ActorAddress definitions; arena, edge_lifecycle, and ring re-export them
(previously duplicated per module)
- edge_wire.rs: the whole transport contract — WireEvent, EdgeWriter, and
the EdgeTransport port (associated Writer/PeerAddr types)
- edge_runtime.rs: EdgeRuntime composition engine absorbing iroh-driver's
driver_pumps bookkeeping, the EdgeEstablisher lifecycle drive, arena
leasing, ingress stream buffering with object-record parsing, and ring
writes; effects go through a WorkerPort trait; progress surfaces as
structured Observations the application maps to telemetry/agent messages
- delete superseded test-only layers: actor.rs (DataPlaneNodeActor),
edge_actor.rs, ingress.rs, egress.rs and their guarantee tests
- fold ObjectIdAllocator into object_record (now edge-free, starts at 1)
iroh-driver:
- edge_transport speaks pure data_plane::edge_wire vocabulary; EdgeSendHandle
implements EdgeWriter; IrohDriver implements EdgeTransport (PeerAddr =
EndpointAddr) — the entire edge surface is open_writer + drain_events
- delete driver_pumps.rs; new dependency on data-plane (no cycle)
- IROH_DRIVER_SPEC §5 updated for the new module set and edge boundary
myelin:
- WorkerEdgeRuntime shrinks from ~830 lines of hand glue to an EdgeRuntime
holder plus a tinygrad WorkerPort impl and observation reporting; the
driver-event/edge-event translation layers and newtype re-wrapping are
gone
- orchestration/app.rs and job edge drains consume WireEvent
Tests: data-plane 31 (5 new EdgeRuntime contract tests), iroh-driver 13,
myelin 65 — all green.
The crate is the per-node metrics/logging pipe with a universal
subscriber endpoint, but "datastream" kept getting misread as a general
messaging plane. Rename crate, module paths, and public API
(`DatastreamEndpoint` → `TelemetryEndpoint`, etc.) so misuse is visible
on sight.
Renamed contracts (all in-repo producers/consumers migrated):
- env vars `MYELIN_DATASTREAM_*` → `MYELIN_TELEMETRY_*`
- artifact `datastream.ndjson` → `telemetry.ndjson`
- actor names `telemetry-publisher` / `telemetry-sink`
- wire ALPN `swactor/telemetry/0`
- `DATASTREAM_SPEC.md` → `TELEMETRY_SPEC.md`
Also fixes two latent test breaks: `process` and `iroh-driver` tests
imported `DatastreamEvent` from the crate root, which was never
re-exported; they now use the observer path `telemetry::frame::`.
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
Specs gain a monotonic Id and relocate by their true status: drafts
(WIP/aspirational) to docs/specs/drafts/, and accurate code-behavior
references stay in their crate dirs. docs/specs/archive/ is reserved for
superseded docs (currently empty).
Dispositions were cross-referenced against code, not the specs' own headers.
IROH_DRIVER claimed "current-state" but ~30% is unbuilt redesign, so it moves
to drafts. DATA_PLANE_ACTOR's central integration claim is unrealized (myelin
bypasses its node actor), and ACTOR_PANEL is not a reference; both are dropped
rather than reviewed or archived. DATASTREAM and MANAGED_PROCESS stay as
references; MYELIN stays in place (stale, flagged for review).
From here, commit titles reference a spec by [N] when one applies. This
bootstrap commit does not carry one.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add the iroh-driver spec; trim and restructure the mvp-system orchestrator and
mvp-chat specs. Reframe prompt RPC as the prompt-engine contract in mvp_chat.md; add an
iroh-driver dependency; minor datastream_transport cleanup.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
