zacheryasc/swactor
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
swactor/crates/data-plane/src/lib.rs

29 lines
769 B
Rust
Raw Normal View History

refactor: data movement handled by an independant crate Promote the data-plane into its own crate with a swactor-facing DataPlaneNodeActor that owns wire-edge lifecycle, and move the edge/ring/egress/ingress machinery out of mvp-system into crates/data-plane. - data-plane/src/actor.rs: add the 769-line DataPlaneNodeActor implementing ActorInterface, owning WireEdgeEndpoint provisioning and the EdgeEstablisher lifecycle and draining commands to arena/worker/transport actors (DataPlaneArenaMsg/WorkerMsg/TransportMsg) with DataPlaneReportMsg back to a report sink - data-plane/src/lib.rs: expand the crate surface to expose actor, arena, edge_actor, edge_lifecycle, egress, and ingress alongside object_record/ring, and reframe it as actor-oriented wire-edge / ring / arena / object-movement contracts - data-plane: move edge_actor and edge_lifecycle (from node/), egress (from worker/), and ingress (from node_data/) into the crate, and add ArenaSample (serde Record, ARENA_SAMPLE_CHANNEL/INTERVAL) to arena.rs - data-plane: add DATA_PLANE_ACTOR_SPEC.md and the data_plane_actor_guarantees/edge_lifecycle_guarantees/egress_guarantees tests - mvp-system/node: add data_plane_bridge.rs wiring the node runtime to the DataPlaneNodeActor, drop the old node_data/arena.rs and node_data/mod.rs (now in data-plane), and remove the arena_manager_guarantees test - mvp-system: drop node_data from the lib.rs pub surface and repoint node/mod.rs to consume the data-plane crate Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-28 17:30:56 +00:00
//! Reusable actor-oriented data-plane contracts for wire edges, local IPC rings,
//! arena-backed byte movement, and GPU worker object movement.
Move all edge logic into data-plane; reduce iroh-driver to a byte-transport port Duty mixing between iroh-driver and data-plane is resolved: the transport crate now owns only byte pumping, and the data-plane owns every edge semantic. data-plane: - ids.rs: single EdgeId/RingId/StreamId/NodeId/RunId/LeaseRequestId/ ActorAddress definitions; arena, edge_lifecycle, and ring re-export them (previously duplicated per module) - edge_wire.rs: the whole transport contract — WireEvent, EdgeWriter, and the EdgeTransport port (associated Writer/PeerAddr types) - edge_runtime.rs: EdgeRuntime composition engine absorbing iroh-driver's driver_pumps bookkeeping, the EdgeEstablisher lifecycle drive, arena leasing, ingress stream buffering with object-record parsing, and ring writes; effects go through a WorkerPort trait; progress surfaces as structured Observations the application maps to telemetry/agent messages - delete superseded test-only layers: actor.rs (DataPlaneNodeActor), edge_actor.rs, ingress.rs, egress.rs and their guarantee tests - fold ObjectIdAllocator into object_record (now edge-free, starts at 1) iroh-driver: - edge_transport speaks pure data_plane::edge_wire vocabulary; EdgeSendHandle implements EdgeWriter; IrohDriver implements EdgeTransport (PeerAddr = EndpointAddr) — the entire edge surface is open_writer + drain_events - delete driver_pumps.rs; new dependency on data-plane (no cycle) - IROH_DRIVER_SPEC §5 updated for the new module set and edge boundary myelin: - WorkerEdgeRuntime shrinks from ~830 lines of hand glue to an EdgeRuntime holder plus a tinygrad WorkerPort impl and observation reporting; the driver-event/edge-event translation layers and newtype re-wrapping are gone - orchestration/app.rs and job edge drains consume WireEvent Tests: data-plane 31 (5 new EdgeRuntime contract tests), iroh-driver 13, myelin 65 — all green.
2026-08-16 17:49:45 +00:00
//!
//! Composition lives in [`edge_runtime`]: [`edge_runtime::EdgeRuntime`] drives
//! the edge lifecycle ([`edge_lifecycle`]) over a byte transport
//! ([`edge_wire`]), the arena ([`arena`]), and an application worker port.
refactor(mvp-system): establish module boundaries - Split arena/ring/object-record into a new data-plane crate and node/plugin contracts into a provisioning crate. - Reorganize mvp-system into orchestration, staging, node, chat, and worker modules; extract binaries into chat/runtime and node/worker_node_runtime. - Add MVP_SYSTEM_MODULE_BOUNDARY_SPEC. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-28 07:29:31 +00:00
pub mod arena;
feat(myelin): add actor-backed job data plane and uploads Replace the eventfd/ring job bootstrap with one inherited arena descriptor, actor-owned sessions, sealed blob leases, awaitable inbox wakeups, and zero-copy Python mappings. Route VastAI mock provisioning through image-backed local Docker workers and preserve pinned child and controller routes across directory updates. Add TOML job-file submission to the Fleet UI with generic started, running, and completed feedback, reusable remote job controller routing, cancellation and kill invariants, Tinygrad fixture and image support, and comprehensive Rust, Python, CUDA, and lifecycle-ordering coverage.
2026-08-21 14:45:10 +00:00
pub mod blob;
feat(data-plane): complete virtual blob namespace Add durable namespace control, Iroh-backed source transfer, session-independent publication lifetime, and wire managed Myelin jobs and bindings through the reusable data-plane API.
2026-08-22 17:16:56 +00:00
pub mod blob_transfer;
feat(data-plane): add exec bootstrap ABI for job processes Define the spawn-time handoff a job process consumes before any application code runs: two inherited descriptors named by SWACTOR_ARENA_FD and SWACTOR_WAKE_FD, and a fixed 48-byte little-endian header at arena offset 0 naming one control-ring region. - Canonical header layout, shared parser, and a host writer that leases the header region and control ring disjointly under the arena's own placement law, zeroes the ring, stamps its generation, and returns the handoff (env map, inheritable arena and wake descriptors, host wake eventfd) only after every write completes. - Python binding gains swactor.run(main): map the arena read-only, take ground truth from fstat, validate through the shared parser, arm FD_CLOEXEC on the wake descriptor, close the arena descriptor after mapping, and drive main on asyncio with Context.data carrying the resolved state. Fail-fast BootstrapError before main on any defect. - Rewrite jobs/tiny_linear_inference.py against the approved path API and record the slice, invariants, and remaining bridge work in the handoff doc. Verified with 12 new data-plane bootstrap guarantee tests (parser defect table, fuzzed pages, writer round-trip), 29 binding tests across in-process and exec boundaries, and full data-plane and iroh-driver suites with no regressions.
2026-08-20 16:43:44 +00:00
pub mod bootstrap;
feat(myelin): add actor-backed job data plane and uploads Replace the eventfd/ring job bootstrap with one inherited arena descriptor, actor-owned sessions, sealed blob leases, awaitable inbox wakeups, and zero-copy Python mappings. Route VastAI mock provisioning through image-backed local Docker workers and preserve pinned child and controller routes across directory updates. Add TOML job-file submission to the Fleet UI with generic started, running, and completed feedback, reusable remote job controller routing, cancellation and kill invariants, Tinygrad fixture and image support, and comprehensive Rust, Python, CUDA, and lifecycle-ordering coverage.
2026-08-21 14:45:10 +00:00
pub mod byte_ring;
feat(data-plane): complete virtual blob namespace Add durable namespace control, Iroh-backed source transfer, session-independent publication lifetime, and wire managed Myelin jobs and bindings through the reusable data-plane API.
2026-08-22 17:16:56 +00:00
pub mod control;
feat(myelin): add actor-backed job data plane and uploads Replace the eventfd/ring job bootstrap with one inherited arena descriptor, actor-owned sessions, sealed blob leases, awaitable inbox wakeups, and zero-copy Python mappings. Route VastAI mock provisioning through image-backed local Docker workers and preserve pinned child and controller routes across directory updates. Add TOML job-file submission to the Fleet UI with generic started, running, and completed feedback, reusable remote job controller routing, cancellation and kill invariants, Tinygrad fixture and image support, and comprehensive Rust, Python, CUDA, and lifecycle-ordering coverage.
2026-08-21 14:45:10 +00:00
pub mod data_plane;
refactor: data movement handled by an independant crate Promote the data-plane into its own crate with a swactor-facing DataPlaneNodeActor that owns wire-edge lifecycle, and move the edge/ring/egress/ingress machinery out of mvp-system into crates/data-plane. - data-plane/src/actor.rs: add the 769-line DataPlaneNodeActor implementing ActorInterface, owning WireEdgeEndpoint provisioning and the EdgeEstablisher lifecycle and draining commands to arena/worker/transport actors (DataPlaneArenaMsg/WorkerMsg/TransportMsg) with DataPlaneReportMsg back to a report sink - data-plane/src/lib.rs: expand the crate surface to expose actor, arena, edge_actor, edge_lifecycle, egress, and ingress alongside object_record/ring, and reframe it as actor-oriented wire-edge / ring / arena / object-movement contracts - data-plane: move edge_actor and edge_lifecycle (from node/), egress (from worker/), and ingress (from node_data/) into the crate, and add ArenaSample (serde Record, ARENA_SAMPLE_CHANNEL/INTERVAL) to arena.rs - data-plane: add DATA_PLANE_ACTOR_SPEC.md and the data_plane_actor_guarantees/edge_lifecycle_guarantees/egress_guarantees tests - mvp-system/node: add data_plane_bridge.rs wiring the node runtime to the DataPlaneNodeActor, drop the old node_data/arena.rs and node_data/mod.rs (now in data-plane), and remove the arena_manager_guarantees test - mvp-system: drop node_data from the lib.rs pub surface and repoint node/mod.rs to consume the data-plane crate Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-28 17:30:56 +00:00
pub mod edge_lifecycle;
Move all edge logic into data-plane; reduce iroh-driver to a byte-transport port Duty mixing between iroh-driver and data-plane is resolved: the transport crate now owns only byte pumping, and the data-plane owns every edge semantic. data-plane: - ids.rs: single EdgeId/RingId/StreamId/NodeId/RunId/LeaseRequestId/ ActorAddress definitions; arena, edge_lifecycle, and ring re-export them (previously duplicated per module) - edge_wire.rs: the whole transport contract — WireEvent, EdgeWriter, and the EdgeTransport port (associated Writer/PeerAddr types) - edge_runtime.rs: EdgeRuntime composition engine absorbing iroh-driver's driver_pumps bookkeeping, the EdgeEstablisher lifecycle drive, arena leasing, ingress stream buffering with object-record parsing, and ring writes; effects go through a WorkerPort trait; progress surfaces as structured Observations the application maps to telemetry/agent messages - delete superseded test-only layers: actor.rs (DataPlaneNodeActor), edge_actor.rs, ingress.rs, egress.rs and their guarantee tests - fold ObjectIdAllocator into object_record (now edge-free, starts at 1) iroh-driver: - edge_transport speaks pure data_plane::edge_wire vocabulary; EdgeSendHandle implements EdgeWriter; IrohDriver implements EdgeTransport (PeerAddr = EndpointAddr) — the entire edge surface is open_writer + drain_events - delete driver_pumps.rs; new dependency on data-plane (no cycle) - IROH_DRIVER_SPEC §5 updated for the new module set and edge boundary myelin: - WorkerEdgeRuntime shrinks from ~830 lines of hand glue to an EdgeRuntime holder plus a tinygrad WorkerPort impl and observation reporting; the driver-event/edge-event translation layers and newtype re-wrapping are gone - orchestration/app.rs and job edge drains consume WireEvent Tests: data-plane 31 (5 new EdgeRuntime contract tests), iroh-driver 13, myelin 65 — all green.
2026-08-16 17:49:45 +00:00
pub mod edge_runtime;
pub mod edge_wire;
feat(myelin): add actor-backed job data plane and uploads Replace the eventfd/ring job bootstrap with one inherited arena descriptor, actor-owned sessions, sealed blob leases, awaitable inbox wakeups, and zero-copy Python mappings. Route VastAI mock provisioning through image-backed local Docker workers and preserve pinned child and controller routes across directory updates. Add TOML job-file submission to the Fleet UI with generic started, running, and completed feedback, reusable remote job controller routing, cancellation and kill invariants, Tinygrad fixture and image support, and comprehensive Rust, Python, CUDA, and lifecycle-ordering coverage.
2026-08-21 14:45:10 +00:00
pub mod host;
Move all edge logic into data-plane; reduce iroh-driver to a byte-transport port Duty mixing between iroh-driver and data-plane is resolved: the transport crate now owns only byte pumping, and the data-plane owns every edge semantic. data-plane: - ids.rs: single EdgeId/RingId/StreamId/NodeId/RunId/LeaseRequestId/ ActorAddress definitions; arena, edge_lifecycle, and ring re-export them (previously duplicated per module) - edge_wire.rs: the whole transport contract — WireEvent, EdgeWriter, and the EdgeTransport port (associated Writer/PeerAddr types) - edge_runtime.rs: EdgeRuntime composition engine absorbing iroh-driver's driver_pumps bookkeeping, the EdgeEstablisher lifecycle drive, arena leasing, ingress stream buffering with object-record parsing, and ring writes; effects go through a WorkerPort trait; progress surfaces as structured Observations the application maps to telemetry/agent messages - delete superseded test-only layers: actor.rs (DataPlaneNodeActor), edge_actor.rs, ingress.rs, egress.rs and their guarantee tests - fold ObjectIdAllocator into object_record (now edge-free, starts at 1) iroh-driver: - edge_transport speaks pure data_plane::edge_wire vocabulary; EdgeSendHandle implements EdgeWriter; IrohDriver implements EdgeTransport (PeerAddr = EndpointAddr) — the entire edge surface is open_writer + drain_events - delete driver_pumps.rs; new dependency on data-plane (no cycle) - IROH_DRIVER_SPEC §5 updated for the new module set and edge boundary myelin: - WorkerEdgeRuntime shrinks from ~830 lines of hand glue to an EdgeRuntime holder plus a tinygrad WorkerPort impl and observation reporting; the driver-event/edge-event translation layers and newtype re-wrapping are gone - orchestration/app.rs and job edge drains consume WireEvent Tests: data-plane 31 (5 new EdgeRuntime contract tests), iroh-driver 13, myelin 65 — all green.
2026-08-16 17:49:45 +00:00
pub mod ids;
feat(myelin): add actor-backed job data plane and uploads Replace the eventfd/ring job bootstrap with one inherited arena descriptor, actor-owned sessions, sealed blob leases, awaitable inbox wakeups, and zero-copy Python mappings. Route VastAI mock provisioning through image-backed local Docker workers and preserve pinned child and controller routes across directory updates. Add TOML job-file submission to the Fleet UI with generic started, running, and completed feedback, reusable remote job controller routing, cancellation and kill invariants, Tinygrad fixture and image support, and comprehensive Rust, Python, CUDA, and lifecycle-ordering coverage.
2026-08-21 14:45:10 +00:00
pub mod mapped_arena;
feat(data-plane): complete virtual blob namespace Add durable namespace control, Iroh-backed source transfer, session-independent publication lifetime, and wire managed Myelin jobs and bindings through the reusable data-plane API.
2026-08-22 17:16:56 +00:00
pub mod namespace;
pub mod namespace_store;
refactor(mvp-system): establish module boundaries - Split arena/ring/object-record into a new data-plane crate and node/plugin contracts into a provisioning crate. - Reorganize mvp-system into orchestration, staging, node, chat, and worker modules; extract binaries into chat/runtime and node/worker_node_runtime. - Add MVP_SYSTEM_MODULE_BOUNDARY_SPEC. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-28 07:29:31 +00:00
pub mod object_record;
feat(myelin): add actor-backed job data plane and uploads Replace the eventfd/ring job bootstrap with one inherited arena descriptor, actor-owned sessions, sealed blob leases, awaitable inbox wakeups, and zero-copy Python mappings. Route VastAI mock provisioning through image-backed local Docker workers and preserve pinned child and controller routes across directory updates. Add TOML job-file submission to the Fleet UI with generic started, running, and completed feedback, reusable remote job controller routing, cancellation and kill invariants, Tinygrad fixture and image support, and comprehensive Rust, Python, CUDA, and lifecycle-ordering coverage.
2026-08-21 14:45:10 +00:00
pub mod path;
pub mod protocol;
refactor(mvp-system): establish module boundaries - Split arena/ring/object-record into a new data-plane crate and node/plugin contracts into a provisioning crate. - Reorganize mvp-system into orchestration, staging, node, chat, and worker modules; extract binaries into chat/runtime and node/worker_node_runtime. - Add MVP_SYSTEM_MODULE_BOUNDARY_SPEC. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-28 07:29:31 +00:00
pub mod ring;
feat(data-plane): complete virtual blob namespace Add durable namespace control, Iroh-backed source transfer, session-independent publication lifetime, and wire managed Myelin jobs and bindings through the reusable data-plane API.
2026-08-22 17:16:56 +00:00
pub mod source;
feat(data-plane): add routed SPSC stream transport Implement namespace-owned stream rendezvous with incarnation isolation, cancellation, recovery, bounded local byte rings, and host/child actor protocols. Add a replaceable stream transport boundary plus an Iroh ALPN adapter with framed records, backpressure, reconnect handling, and terminal propagation. Wire stream sources and sinks through Myelin job deployment and Python bindings, with guarantee coverage across blob, namespace, job, byte-ring, and transport paths. Sanitize nested Cargo build context in xtask so lint, Nextest, doctest, and maturin invocations retain stable fingerprints.
2026-08-23 08:47:31 +00:00
pub mod stream_transport;
Reference in a new issue Copy permalink