From 91cc8a4b26f4f03b11ef8eb18bb0836ce2b6f2e4 Mon Sep 17 00:00:00 2001 From: Zachery Aaron Shores-Chmielewski Date: Sun, 21 Jun 2026 23:54:58 +0400 Subject: [PATCH] draft spec stash --- DESIGN_DIRECTIVES.md | 367 ++++++++ GPU_WORKER_INTERFACE_SPEC.md | 1397 +++++++++++++++++++++++++++++++ ORCHESTRATION_SPEC.md | 518 ++++++++++++ RING_BACKPRESSURE_SPEC.md | 1522 ++++++++++++++++++++++++++++++++++ 4 files changed, 3804 insertions(+) create mode 100644 DESIGN_DIRECTIVES.md create mode 100644 GPU_WORKER_INTERFACE_SPEC.md create mode 100644 ORCHESTRATION_SPEC.md create mode 100644 RING_BACKPRESSURE_SPEC.md diff --git a/DESIGN_DIRECTIVES.md b/DESIGN_DIRECTIVES.md new file mode 100644 index 0000000..3586157 --- /dev/null +++ b/DESIGN_DIRECTIVES.md @@ -0,0 +1,367 @@ +# Design Directives + + +Captured design directives for the swactor data-movement / coordination-plane work. +These are the constraints and decisions as directed — not proposals. Resume from here. + +## Problem & domain +- Building a **library for neural nets over heterogeneous, WAN-connected, churning hardware**. +- The thing being designed: how nodes **coordinate transfer/streaming of large blobs** — weights, activations, data — as a **reusable primitive**. +- **Nodes are trusted.** All trustless/P2P-style concerns (incentives, adversarial verification, Sybil, DHT discovery) are irrelevant. +- Focus is **ML workloads**, not general P2P. + +## Scope discipline +- **Control plane first.** Performance (pre-alloc, buffering, chunked streams) is for later — *we don't know our bottlenecks yet*. +- The problem is **much smaller than what an object store solves** — don't over-engineer toward Ray/Plasma scope. +- Want **a solid set of abstractions that make designing distributed graphs over a WAN easy.** + +## The library goal (ought) & lifecycle +- (For directional purposes only, we are only building one piece currently) +- Input: **a workload graph + a pool of hardware.** The library **distributes the work according to the graph across the hardware.** +- The meta lifecycle: + 1. **define graph and pool in top-level code** + 2. graph is **broken up per optimal placement** onto the extant resource pool + 3. nodes **establish edges/networking** + 4. nodes **fetch the weights/data they need** + 5. **data flows end-to-end** + 6. **teardown**, etc. + +## Layers +- **Total compute graph** (e.g. inference over the transformer weights in a GGUF). +- **Roles created strategically** from **model shape + available hardware**. +- **Distribution of the model.** +- **Piping the data around.** + +## Roles, graph, edges +- **Shape is owned by the graph, decided before roles are assigned.** +- A **role = a portion of the compute graph**: tensors flow in through the network boundary, get processed, new tensors flow back out. +- The graph is **known ahead of time** (pipelined inference). Nodes know ahead of time: they'll **send** activations, the **role they send to**, and that they'll **receive**. +- Each node knows: its **incoming tensors, outgoing tensors, and its own held weights.** +- The control plane specifies **size + shape + abstracted role endpoints** ("I am X sending Y to Z; I am Z receiving Y from X"). +- **Graph-specifying code references a role abstractly** — not how the data reaches it or is received. +- Assume a **placement algorithm exists** (hand-defined at first, real later) — assignment/boundaries are already decided. + +## swactor's mandate & boundaries +- **swactor is the runtime: the coordination plane for data movement and resource provisioning.** +- swactor actors are **strongly-typed FSMs passing messages**, to allow strict design of what the data looks like. +- The **CP is owned by an orchestrator** that: **runs and observes SWIM**, **handles resource provisioning**, and **stages execution on the nodes**. +- swactor actor paradigm = the **CP/dataflow layer**; **tinygrad = GPU execution.** +- **swactor does NOT handle the stream. iroh handles the streaming** — forget the nuts and bolts of the bytes on the wire. +- swactor is **not** doing flow control / credit / RTS-CTS — that solves a different problem, because **everything but the transport route is known ahead of time.** + +## What's known vs. unknown +- **Known ahead of time:** the blob's **shape/dtype family** and its **`capacity`** + (the per-chunk byte ceiling, `max_seq_len × hidden × dtype`), plus the abstracted + endpoint (role). +- **Unknown at runtime:** the **transport** — who role X physically is, how to reach + it — *and* each chunk's actual **`extent`** (e.g. the prompt length: prefill + carries many rows, decode one), a per-step runtime fact `≤ capacity`. +- Runtime job: **resolve the transport problem** and bind the abstract role to a concrete route. + +## swactor's concrete job (the gate) +- Assume over-the-wire streams are solved: **bytes travel fast and safe and arrive in a known buffer location.** +- **Receive side:** (zero-)deserialize the already-buffered bytes; **signal that we have the tensors** so they can be piped to the GPU. +- **Send side:** bytes are cooked from the GPU; swactor **populates a send buffer** and **resolves the sink.** +- What swactor tells the process (tinygrad) is **an actual location of the bytes on-device.** +- swactor **handles the stages** of this handoff, **not the actual movement.** +- **swactor actors are simple gates.** +- → Concretely realized in **The process-facing layer (stream & sink)** below. + +## The process-facing layer (stream & sink) + +Framing: we are designing the **typed send/receive infrastructure**, not the process. +The process view is the motivating lens — start from what the local compute sees, then +build the CP that drives it. + +- **The role is a function:** typed chunks come in, get computed, typed chunks go out, + through ports the process exposes. For the CP these ports are an **arbitrary address + + handler space**; the first iteration uses one inbound and one outbound, but nothing + assumes that count. Inbound and outbound ports are **independent** — not a coupled 1:1 + pass-through; the source and the sink in a single process are unrelated. +- **The process is driven, not autonomous.** It waits for the CP to stage a chunk, + computes, hands the result back. (We lean toward a blocking pull as the natural shape + for a single straight-line GPU worker, but that's a hedge, not a committed API — the + load-bearing point is that the CP drives.) +- **A chunk is a typed tensor, not bytes** — fully parsed, sanitized, shaped, i.e. + everything except the handoff to tinygrad and the GPU copy, which is the one step the + process performs. Shape/dtype are already determined inside the process by its role; + they aren't designed here, and we stay abstract about the type machinery for now. +- **The control plane handles ordering** and correlation between inbound and outbound + chunks. The process tags and tracks nothing. +- **The sink carries its destination.** The process knows the next address and binds it + to the sink — the sink, as a type/struct, has the *next address* built in. **Resolving + that address, serializing the chunk, and streaming it out is the CP's job.** This is + where we reach for **swactor's distributed address space**: the CP outside the process + receives and parses typed chunks, feeds them into the local process (tinygrad + GPU), + then on the way out resolves the sink's address and streams the outbound chunk. +- **Statefulness** (KV cache, position) is worker-internal and out of scope for this + layer. +- **Handle/payload representation and zero-copy staging are deferred** — performance + lives there, behind an unchanged process-facing surface. + +This is an **abstract in-process surface** — a good vantage to design from, not a locked +API: +- an inbound typed-chunk source the process is driven from, +- an outbound typed sink the process pushes to, carrying its next address, +- both shaped by the role, both independent. + +The CP's mandate from here: **receive + parse typed chunks into a process, and resolve + +serialize + stream typed chunks out of it.** + +## End-to-end single pass (abstract) + +Assumes SWIM converged and a start signal received. The pass is one repeating **edge** +(role A → role B) plus two ends; the **orchestrator is just another participant** (sink → +role0, source ← roleN), and **tokens are typed chunks** like activations. + +- **Egress (role A, CP):** take the typed chunk from the sink; serialize → wire bytes; + resolve the sink's abstract next-address → concrete route. Serialize and resolve are + independent operations. +- **Transport:** stream bytes A → B (iroh; not swactor's concern). +- **Ingress (role B, CP):** reassemble → deserialize + validate against the role-known + spec → stage → drive the local process. +- **Ends:** orch → role0 ships *tokens*; roleN → orch ships a *token*. Not special cases + — edges whose endpoint is the orch and whose chunk type is tokens. + +The open seam, designed next: **addressing** — binding the sink's abstract next-address to +a concrete remote endpoint. + +## The edge: addressing, signals, and the byte boundary + +How an edge is established and how chunks flow across it. Steady state assumes SWIM +converged and edges established. + +**Addressing — orchestrator-direct, no inter-end handshake** + +- An edge is a **pair of stream actors** (distinct from the process-driving actor): a + `Tx` (send) and an `Rx` (receive). **Neither end knows the other's actor address.** + The data plane is addressed by **`(node_id, edge_id)`**: the Tx sends to the + consumer's node by its `node_id`, and the `edge_id` at the head of the stream + demuxes it to the right `Rx`. +- **There is no inter-end negotiation** — the two ends never exchange a message; each + is handed everything it needs at provisioning (the orchestrator owns placement). + This still avoids per-chunk negotiation and the RTS-CTS/credit flow-control the + directives rule out ("everything but the route is known ahead of time") — it just + avoids the per-*edge* handshake too. +- The data endpoint is **handed down as the stable `node_id`** (the orchestrator knows + it from placement); iroh resolves the live path from it — so there's no stale + mapping to rot. (Churn is deferred; we design the happy case where both ends are + resolvable.) +- **Byte-level backpressure is pushed into the streaming logic**, not the actor layer. + Actors hold the edge; the streaming layer owns moving the bytes. + +**The buffer-ownership baton & signals** + +- The sink buffer is owned by either the process or the actor system at any instant; + signals are the handoffs. +- Egress: process **`done`** (buffer filled) → egress actor hands `(buffer, endpoint)` to + streaming → streaming **`released`** returns the slot to the alloc pool. +- Ingress: streaming **`landed`** → ingress actor inspects → **`ready`** drives the + process. +- **`done` is non-blocking.** The process never blocks after signalling it produced a + chunk. It blocks only on **alloc** (acquiring a send slot) and on **recv** (a chunk + arriving). With a single buffer, chunk *k+1*'s alloc blocks until chunk *k*'s + `released`. (Double-buffering deferred.) + +**The byte boundary — egress trusts, ingress verifies** + +- **Egress:** no actor-level parsing. The buffer is correct by construction, the endpoint + is bound from setup, and the receiver knows how to decode (type known a priori). The + egress actor hands the buffer straight to the streaming layer. **swactor touches zero + bytes on egress.** +- **Ingress:** swactor enters the byte path only to **read/check, never to transform**. + The **`Rx` and its edge service double as the inspector** — no new actor. + - Behind the **stream abstraction (data-plane integrity):** the framed `[extent]` + prefix is read and its `extent` bytes arrive complete (`extent ≤ capacity`). + Size/length lives here; a short or torn frame means no `landed`. + - The **ingress actor (control-plane gate):** given a complete chunk, clears it to + drive the process (belongs to this edge, expected in sequence), then flags `ready`. + The designated home for any sanity/terms check; thin in the happy path, but where + checking lives so the process is never handed an unvetted chunk. +- Net swactor byte-contract, both sides: **never transforms payload bytes; reads them + only to inspect, and only on ingress.** + +**Open (not yet decided):** the depth of the ingress check — pure terms/sequence gate vs. +cracking the payload for a content-level (shape/dtype) sanity check before `ready`. + +--- + +## Ingress check depth — decided (resolves "Open" above) + +**Optimistic ingress: a chunk is accepted on its framed length alone.** If the +`[extent]` prefix reads cleanly, `extent ≤ capacity`, and that many bytes arrive +complete, they go to the process as-is — no peeking inside, no +deserialize-to-validate, no shape/dtype content check. Correct-by-construction +egress + a clean framed read on ingress is the entire gate. Nodes are trusted; +content trust is total. + +## The transport, minimal + +- Bytes move over **iroh** (already fixed). One **ordered, reliable stream per + edge** is the whole mechanism. Striping, chunk hashing/verification, and resume + are **deferred**. +- We are **not** building on the existing `crates/datastore/src/streams/` module — + treated as not-ready; design fresh. + +## The transfer actors — a reusable primitive + +- An edge's ends are **two actor types: `Tx` (send) and `Rx` (receive)**, one per + edge-end. They are a **general blob-moving primitive** — no notion of role, + compute, or the graph. +- Each is **pre-told the blob's `capacity`** (a `BlobSpec`; the per-chunk byte + ceiling — `dtype`/`shape` live in the role layer above, not the transport) and owns + **zero-copy (de)serialization**: Tx views the producer's buffer as bytes (no + transform); Rx views landed bytes back as a typed value. Rx **pre-allocates its + landing buffer** from `capacity` at setup; each chunk's actual `extent ≤ capacity` + varies per step and rides the wire as a length-prefix. + +## Edge establishment — orchestrator-direct (contract #1) + +- **No derived/hashed addresses, no gossip discovery, no polling.** The + **orchestrator owns placement and wires edges directly** — it hands each end + everything it needs. Addresses stay runtime-assigned (random); identities are + *handed over*, never computed or discovered. +- **The data plane is addressed by `(node_id, edge_id)`.** The orchestrator hands the + `Tx` its consumer's stable `node_id` (known from placement) in the provision + message; iroh resolves the live path from it, so no static transport mapping can + rot. There is no peer-to-peer endpoint exchange. +- **Per-node `Provisioner`** spawns the local `Tx`/`Rx`. The node's **edge service** + (the ALPN-aware `IrohDriver`) demuxes incoming streams to the right `Rx` by the + run-global **`edge_id`** at the head of the stream — not a separate `Listener` + actor; the demux is a tokio task on the existing endpoint. +- **No inter-end handshake; race-free by a single barrier.** The two ends never + exchange a message — each is fully equipped at provisioning, which fans out **in + parallel** (no Tx-before-Rx ordering). Race-freedom is one barrier: a node acks + `Provisioned` only after its `Rx` ends have **registered** their landing, and the + orchestrator injects the prompt only after **every** node has acked — so no stream + can arrive before its `Rx` is registered, with no per-edge ordering. +- **The establishment "exchange" is just the `edge_id` stream preamble (Tx→Rx).** No + `EndpointOffer`, no `Ready`, no `EdgeReady`, no `tx_addr` relay. READY is a local + terminal state: `Tx` is ready on spawn; `Rx` is ready once it has pre-allocated and + registered its landing buffer. +- **Kickoff, not broadcast.** The orchestrator is just another participant; once all + nodes are `Provisioned` it injects the driving prompt into role0 on its own + outbound edge. Every other node derives its own state from arriving data. +- **Asymmetry:** `Tx` never needs `Rx`'s actor address — and now neither end needs + the other's; they are coupled only by the shared `edge_id`. + +## Still deferred (unchanged stance) + +Churn/failure policy, teardown, the fan-in **join**, and the per-chunk zero-copy +baton (contract #4) remain out of scope. (The **start signal** is no longer here — +it's decided: there is none; see "Kickoff, not broadcast" above.) + +--- + +# Blob streaming & host allocation + +Detailing blob streaming + host allocation (contract #4). Full flow in +`BLOB_STREAMING.md` (draft). + +## The host substrate — one sparse arena per node +- Blob bytes live on the host in a single `memfd` arena, mapped by **both** the + node process and the Python GPU worker. +- The arena is reserved **big and sparse** (lazy tmpfs backing) and mapped **once**; + the mapping is never moved. Edges are **regions sub-allocated** from it and + returned on teardown, so topology is **dynamic without touching the fd**. + +## The slot handoff +- Each edge owns a **ring of N `capacity`-sized slots** (default 2). A slot is owned + at any instant by exactly one of {iroh, GPU worker}; ownership passes by **signal, + two per direction**, over the existing stdin/stdout pipe. Actors never touch a + payload byte; the node process is the sole authority on slot state. + +## The GPU boundary +- tinygrad reads/writes slots **in place** via a `memoryview` (`copyin`/`copyout`); + a blob **never enters Python's heap**. The host↔device DMA is the worker's only + copy. + +## Streaming +- **One long-lived iroh uni-stream per edge**; `edge_id` preamble once; then + length-prefixed chunks `[extent: u32][extent bytes]`, so the **prefix is the + frame** (each chunk's `extent ≤ capacity` varies per step; the slot is sized once + to `capacity`). Refines establishment's per-call `open_uni` into a persistent + stream, and its single landing buffer into the ring. + +## Deferred to their own passes +- Host-pinning + removing tinygrad's CUDA `copyin` bounce (perf). + +--- + +# Activation stream transport — the iroh ↔ swactor boundary + +How activation tensors actually cross a READY edge, and how the swactor actors, +the iroh driver, and the GPU worker are wired to move them. Full spec in +`STREAM_TRANSPORT.md`. Scoped to activations (not gossip, not weights). + +## The decision in one line +- **One persistent uni-stream per edge; the bytes ride it in place, in the arena; + swactor passes only slot indices, never bytes.** + +## Stream shape — persistent, length-framed +- **One long-lived uni-stream per edge**, not a stream per tensor. `edge_id` + preamble once; then back-to-back length-prefixed tensors. Because each tensor's + size varies per step (prefill many rows, decode one), each rides a fixed-width + `u32` `extent` prefix — `[extent][extent bytes]`, `extent ≤ capacity` — and **the + prefix *is* the frame**; the slot is sized once to `capacity`. Stream-per-message + was rejected: it pays a task spawn + alloc + a `max_concurrent_uni_streams` slot + per tensor and buys nothing. +- **QUIC owns reliability.** No app-level fragmentation (the ring already pipelines + a whole-tensor object) and no striping (one connection over one path shares a + single congestion window — striping needs multipath we don't have). + +## Zero-copy — bytes never enter an actor +- Bytes live in the **shared arena** from the worker's `copyout` to the far + worker's `copyin`, moved **in place**: the wire `read_exact`/`write_all`s arena + slots directly. Huge tensors are never copied into a `Vec` or an actor message. +- **swactor moves slot indices (`usize`), not bytes.** The only payload-byte + touchers are the **GPU worker** and a per-edge **byte-pump task**. + +## Roles — tokio stays behind the driver wall +- **Driver** owns the endpoint, the connection cache, the `edge_id` demux, and + **spawns/owns the byte-pump tasks** — the one place tokio lives, async byte + readers and writers. +- **Reads are tokio-native** — `read_exact`/`write_all` only advance when polled on + the runtime — so a byte-pump *task* is unavoidable while iroh is the transport. + +## MVP +- **MVP = one dedicated byte-pump task per edge-end**, driver-owned. The + **actor↔driver contract is slot-indices-in, slot-indices-up**, so the pump + *mechanism* is a driver-internal detail. Toward removing tokio, it can later + collapse to one-task-per-connection or to the node loop polling the stream + futures — a driver refactor that **touches no actor**. +- **No single-threaded-tick assumption:** all cross-thread traffic is `deliver_raw` + + the slot channels + actor isolation, so the design survives a multi-threaded + runtime. + +## Deferred +- The swactor ↔ GPU-worker pipe **mechanism** (async-Python rework) — its + `ready/consumed/filled/drained` signals are fixed here, the transport is not. + +--- + +# Remaining orchestration decisions before implementation + +The first concrete workload is **sharded inference of large models across +prosumer GPUs**. Do not prematurely generalize this into a broad graph IR. The +next spec layer should describe only the workload/role shape needed for that use +case: model partitioning, role boundaries, edge object specs, weight/shard +ownership, and the runtime sequence for prefill/decode. + +The **orchestrator remains the authority** for resource provisioning, placement, +and run staging. Nodes do not need to advertise capabilities after boot as part +of this design; the orchestrator provisions the pool and already knows the +resource inventory it is placing onto. + +The missing spec surface is therefore: +- how the orchestrator decides roles and placement from model shape plus + provisioned hardware, +- what a role provisioning message contains, +- how model weights/shards are assigned, fetched, loaded, and declared ready, +- what execution semantics the first inference path guarantees, +- and what behavioral contracts are required for reliable tests. + +Observability should be added when the descriptive specs are converted into +behavioral contracts for testing. The goal is not just prose architecture, but +testable run behavior: provisioned, loaded, ready, object produced/consumed, +completed, faulted, and torn down. diff --git a/GPU_WORKER_INTERFACE_SPEC.md b/GPU_WORKER_INTERFACE_SPEC.md new file mode 100644 index 0000000..2cb445f --- /dev/null +++ b/GPU_WORKER_INTERFACE_SPEC.md @@ -0,0 +1,1397 @@ +# GPU Worker Interface and Control Specification + +**Status:** draft design specification for review. + +**Relationship to other documents.** This document extends +`RING_BACKPRESSURE_SPEC.md`. The ring spec remains the authority for shared +memory rings, arena leases, QUIC pumps, ring cursor safety, object records, and +backpressure. `DESIGN_DIRECTIVES.md` remains steering context. This document +defines the process-facing control surface: how the Rust node process starts, +drives, supervises, and tears down the Python/tinygrad GPU worker. + +The design is intentionally a swactor-integrated worker, not a custom runtime. +The Rust-side interface is a small actor message FSM. The process boundary uses +the existing `swactor_process` actor pattern: a domain actor owns a +`ProcessActor`, writes process input through `ProcessCommand::WriteStdin`, and +receives lifecycle/output through `ProcessNotification`. + +Current code is pattern guidance for this document, especially the +`InferenceActor` and `StageActor` process supervision shape. Current payload +movement code is not otherwise authoritative. + +--- + +## 1. Scope + +This document specifies: + +- the node process to GPU worker process boundary +- the swactor messages used to drive the worker +- worker boot, ring installation, execution, teardown, and crash handling +- GPU worker responsibilities and non-responsibilities +- device object lifetime +- how tinygrad execution is driven by the control plane +- correctness and safety invariants + +This document does not specify: + +- placement policy +- model graph partitioning +- role provisioning beyond the minimal control hook needed by execution +- QUIC stream implementation details +- actor runtime internals +- trustless verification +- high-performance host pinning policy beyond the required safety contract + +--- + +## 2. Design Commitments + +The GPU worker is driven by the node control plane. It is not an autonomous graph +scheduler. + +The Rust-side worker controller is a swactor actor. Other actors send it typed +messages; it handles those messages serially according to its current state. + +The worker process does not run a separate swactor runtime. It is treated as a +process-backed mailbox endpoint owned by the controller actor. + +The worker never receives payload bytes in actor/control messages. Payload bytes +move through shared-memory rings only. + +Actors and the worker controller exchange coarse lifecycle events, terminal +events, and wake hints. They do not exchange byte ranges, free byte counts, host +pointers, tensor payloads, or fragments. + +The worker owns GPU-visible state. The node process owns arena allocation, +transport, edge lifecycle, process supervision, and graph ordering. + +The role layer drives compute explicitly with `ExecuteStep`. Loading an ingress +object into GPU memory does not automatically run tinygrad. + +--- + +## 3. Process Topology + +Each node has: + +```text +node process (Rust) GPU worker process +------------------- ------------------ +swactor runtime Python control loop +GpuWorkerCtl actor native ring helper +swactor_process::ProcessActor tinygrad role code +ArenaManager device bridge +Driver and QUIC pumps + +shared memfd arena mapped once by both processes +ProcessActor-owned stdin/stdout/stderr pipes for process control and diagnostics +``` + +`GpuWorkerCtl` is the Rust-side owner of the GPU worker process. It is the only +domain actor that sends worker commands or interprets worker events. + +Driver pumps and edge actors do not talk to the worker process directly. They +send worker-bound messages to `GpuWorkerCtl`. `GpuWorkerCtl` serializes those +commands onto the process actor. + +The process actor remains the only code that owns OS pipes, child process +lifecycle, and process notifications. This matches the existing +`InferenceActor`/`StageActor` pattern: + +```text +domain actors + -> GpuWorkerCtlMsg + -> GpuWorkerCtl actor + -> ProcessCommand::WriteStdin { data } + -> ProcessActor + -> worker stdin + +worker stdout/stderr + -> ProcessNotification::Output + -> ProcessBridge + -> GpuWorkerCtlMsg::Process + -> GpuWorkerCtl actor + -> domain events +``` + +The worker process maps the arena once at startup. The worker uses the native +ring helper for all cross-process ring cursor operations. Python code does not +implement shared atomics, wrap arithmetic, cursor publication, or ring span +calculation. + +--- + +## 4. Components and Ownership + +### 4.1 GpuWorkerCtl + +`GpuWorkerCtl` owns: + +- worker process spawn and termination through `swactor_process` +- worker generation numbering +- arena fd inheritance or fd passing setup +- the process actor address and process notification bridge +- the Rust-side table of installed rings for the current worker generation +- routing worker events to EdgeEstablisher, Driver, Tx/Rx actors, and the role + layer +- crash detection and crash fanout +- restart policy + +`GpuWorkerCtl` does not own: + +- arena leases +- QUIC streams +- ring payload bytes +- device allocations +- tinygrad execution +- graph placement + +### 4.2 GPU Worker Process + +The worker process owns: + +- its mapped view of the arena +- worker-side ring handles +- per-ring parser and producer state +- device allocations +- device object handles +- tinygrad role state +- worker-internal state such as KV cache +- host-to-device and device-to-host copy scheduling +- copy completion tracking + +The worker process does not own: + +- arena allocation or lease reuse +- edge establishment +- peer routing +- QUIC stream creation +- actor addresses +- graph-level ordering outside the received `ExecuteStep` commands + +### 4.3 Native Ring Helper + +The native ring helper is linked or loaded by the worker process and used by Rust +hot-path code. It exposes the operations needed to: + +- open the shared arena and open/close rings by layout +- read committed spans +- advance consume after bytes are safe to release +- read writable spans +- advance commit after bytes are valid +- inspect ring state for fault/debug handling + +The helper may also expose wake pending-bit helpers, but wake ownership remains +outside payload ownership. Ring cursors are the authority. + +### 4.4 Device Bridge + +USER: If it is determined this is an outsized amount of engineering work, we will defer and require objects to be complete before gpu io. + +The worker must have a backend-specific device bridge capable of range copies: + +```text +alloc_device(ObjectSpec, extent) -> DeviceAllocation +free_device(DeviceAllocation) +host_to_device(arena_ptr, len, DeviceAllocation, device_offset) -> CopyEvent +device_to_host(DeviceAllocation, device_offset, arena_ptr, len) -> CopyEvent +copy_event_complete(CopyEvent) -> bool +wrap_for_tinygrad(DeviceAllocation, TensorViewSpec) -> tinygrad object +``` + +A high-level API that can only copy a complete host buffer into a complete tensor +is insufficient. Objects may be larger than a ring, so ingress and egress require +partial range copies. + +For the MVP, copies may be synchronous. If copies are asynchronous, the worker +must not advance a ring cursor past bytes still used by DMA. + +--- + +## 5. Actor and Process Message Boundary + +The public Rust-side interface is a normal swactor actor message enum. + +```rust +enum GpuWorkerCtlMsg { + StartWorker, + InstallRing(InstallRing), + UninstallRing(UninstallRing), + RingReadable { ring_id: RingId }, + RingWritable { ring_id: RingId }, + ConfigureRole(ConfigureRole), + ExecuteStep(ExecuteStep), + ReleaseDeviceObject { device_handle: DeviceObjectHandle }, + ShutdownWorker(ShutdownWorker), + Process(ProcessNotification), +} +``` + +`Process(ProcessNotification)` is delivered by a small `ProcessBridge` actor, +exactly like the existing process-backed inference and pipeline actors. + +There is no custom length-prefixed frame, no control envelope, no protocol +version field on every message, and no generic command id/reply id layer. +Correlation uses the domain identifiers already present in the command: +`ring_id`, `edge_id`, `port_id`, `object_id`, `sequence`, `step_id`, and +`device_handle`. + +### 5.1 Process Adapter Encoding + +The process adapter may encode worker commands and events as one JSON object per +line on stdin/stdout. This is an implementation adapter for a Python subprocess, +not a separate protocol layer. + +Rules: + +- one serialized command or event per line +- stdout is reserved for worker events +- stderr is reserved for logs and diagnostics +- payload bytes are forbidden in command/event JSON +- invalid JSON or unknown event shape is a worker/process fault from + `GpuWorkerCtl`'s perspective + +This keeps the Python worker easy to inspect while preserving the actual swactor +boundary: actors exchange typed Rust messages with `GpuWorkerCtl`, and +`GpuWorkerCtl` uses `ProcessActor` as the subprocess mailbox adapter. + +### 5.2 Process Environment + +At spawn, the node makes the arena fd available to the worker and passes the fd +number through environment: + +```text +SWACTOR_ARENA_FD memfd for the shared arena +SWACTOR_ARENA_BYTES arena reservation ceiling +``` + +The worker reads commands from stdin and writes events to stdout. It may write +logs to stderr. It must not write logs to stdout. + +--- + +## 6. Identifiers and Minimal Data Types + +```rust +struct WorkerGeneration(u64); +struct RoleId(u64); +struct PortId(u64); +struct RingId(u64); +struct EdgeId(u64); +struct ObjectId(u64); +struct Sequence(u64); +struct StepId(u64); +``` + +`RingId` is unique for the node lifetime, as specified by the ring spec. + +`JsonValue` means a `serde_json::Value`-style opaque configuration value used +only for low-frequency app metadata. It must not carry payload bytes. + +`DeviceObjectHandle` is opaque to the node process: + +```rust +struct DeviceObjectHandle { + worker_generation: u64, + id: u64, +} +``` + +A device handle is valid only inside the worker generation that created it. A +worker restart invalidates every previous handle, even if a later worker maps the +same arena. + +### 6.1 Ring Layout and Object Spec + +`RingLayout`, `ObjectSpec`, and `ObjectHeader` are defined by +`RING_BACKPRESSURE_SPEC.md`. This document only relies on these facts: + +- `RingLayout` contains arena-relative offsets and ring capacity; it never + contains process-local pointers +- `ObjectHeader` supplies runtime facts: `object_id`, `sequence`, `extent`, and + flags +- `ObjectSpec` supplies the role-known validation contract: object kind, max + extent, dtype/shape/layout, alignment, and sequence policy + +### 6.2 Role Configuration + +Role provisioning is intentionally not designed here. The current worker may be +single-role and may choose its role through `ProcessSpec` args/env at spawn time. + +`ConfigureRole` exists only as a low-frequency control hook for deployments that +need runtime role configuration before execution: + +```rust +struct ConfigureRole { + role_id: RoleId, + config: JsonValue, +} +``` + +`config` is opaque to this spec. It must not carry payload bytes. Detailed module +loading, factory selection, versioning, and persistent binding APIs are deferred +until role provisioning is actually implemented. + +--- + +## 7. Commands Sent To The Worker + +These are the worker commands that solve current control problems. Anything not +listed here is deferred until a concrete caller needs it. + +### 7.1 InitializeWorker + +Sent once after the process starts. + +```rust +struct InitializeWorker { + worker_generation: WorkerGeneration, + arena_ceiling: u64, + required_ring_helper_abi: u16, + backend: JsonValue, +} +``` + +The worker maps the arena fd, initializes the native helper and backend, and +emits `WorkerReady` or `WorkerFatal`. + +### 7.2 ConfigureRole + +Optional. Sent only when the app needs runtime role configuration. + +```rust +struct ConfigureRole { + role_id: RoleId, + config: JsonValue, +} +``` + +The worker records enough role state to execute later `ExecuteStep` commands. It +does not run compute. + +### 7.3 InstallRing + +```rust +struct InstallRing { + ring_id: RingId, + edge_id: EdgeId, + port_id: PortId, + direction: RingDirection, + layout: RingLayout, + object_spec: ObjectSpec, +} + +enum RingDirection { + Ingress, + Egress, +} +``` + +`edge_id` and `port_id` are required. The worker must be able to report +`ObjectLoaded`, `ObjectProduced`, and `RingFault` in role/edge terms without +guessing from `ring_id`. + +For ingress, the worker creates a parser state machine in `NeedHeader`. + +For egress, the worker creates a producer state machine waiting for an +`ExecuteStep` output that names the ring. + +Terminal events: + +```rust +RingInstalled { ring_id, edge_id, port_id } +RingFault { ring_id, edge_id, port_id, reason } +``` + +### 7.4 UninstallRing + +```rust +struct UninstallRing { + ring_id: RingId, + reason: UninstallReason, +} +``` + +The worker removes the ring from active state, stops parsing or producing on the +ring, waits for copy lifetimes to end, and closes the native helper handle. + +Terminal event: + +```rust +RingQuiesced { ring_id } +``` + +### 7.5 RingReadable + +```rust +RingReadable { ring_id: RingId } +``` + +Wake hint to the worker. For ingress rings, it tells the worker that committed +bytes may be available. The worker must reload ring cursors from shared memory. + +Duplicate hints may be coalesced. + +### 7.6 RingWritable + +```rust +RingWritable { ring_id: RingId } +``` + +Wake hint to the worker. For egress rings, it tells the worker that free space +may be available. The worker must reload ring cursors from shared memory. + +Duplicate hints may be coalesced. + +### 7.7 ExecuteStep +USER: Why do we need this explicit? Why doesnt it fall out naturally from the I/O? This whole thing is async message driven, so why do we need a start message? + +```rust +struct ExecuteStep { + role_id: RoleId, + step_id: StepId, + inputs: Vec, + outputs: Vec, + runtime: JsonValue, + release_inputs_after: bool, +} + +struct InputBinding { + port_id: PortId, + object_id: ObjectId, + sequence: Sequence, + device_handle: DeviceObjectHandle, +} + +struct OutputBinding { + port_id: PortId, + ring_id: RingId, + object_id: ObjectId, + sequence: Sequence, + extent: u64, + flags: u32, +} +``` + +The role layer assigns output `object_id` and `sequence`. The worker does not +invent graph-visible ordering. + +Execution requirements: + +1. Validate the role is available. +2. Validate input handles belong to the current worker generation. +3. Wrap input handles as tinygrad-compatible views. +4. Run the role code. +5. Validate returned outputs against the declared output bindings. +6. Write each output object to the named egress ring. +7. Emit `ObjectProduced` after each full output object has been committed. +8. Emit `StepCompleted` after all declared outputs are produced and role state + updates are complete. + +Terminal events: + +```rust +StepCompleted { role_id, step_id } +StepFailed { role_id, step_id, reason } +``` + +If `release_inputs_after` is true, the worker releases each input device object +after the step reaches a terminal event and no backend work still references the +object. + +There is no separate step FSM requirement. The step behavior follows from serial +message handling, the per-ring FSMs, and backpressure from writable ring space. + +### 7.8 ReleaseDeviceObject + +```rust +ReleaseDeviceObject { + device_handle: DeviceObjectHandle, +} +``` + +The worker frees the device allocation after no tinygrad computation or copy +event still references it. + +Terminal events: + +```rust +DeviceObjectReleased { device_handle } +ReleaseFailed { device_handle, reason } +``` + +### 7.9 ShutdownWorker + +```rust +struct ShutdownWorker { + mode: ShutdownMode, +} + +enum ShutdownMode { + Graceful, + AbortInFlight, +} +``` + +For `Graceful`, the worker rejects new work, finishes accepted operations if +possible, quiesces rings, releases device objects, emits `WorkerStopped`, and +exits. + +For `AbortInFlight`, the worker stops accepting new commands, faults in-flight +work, quiesces rings as far as possible, emits `WorkerStopped`, and exits. + +The timeout policy belongs to `GpuWorkerCtl` and `ProcessActor`, not to the +worker command schema. + +### 7.10 Deferred Commands + +These controls are intentionally not part of the MVP: + +- generic command accepted/rejected acknowledgements +- `BindDeviceObject` / `UnbindDeviceObject` +- `CancelStep` +- `AbortObject` +- `Ping` +- role module/factory provisioning + +They can be added when a concrete caller needs them. Until then, the existing +domain terminal events carry enough state to route success and failure. + +--- + +## 8. Events Sent By The Worker + +### 8.1 Worker Lifecycle Events + +```rust +WorkerReady { + pid: u32, + worker_generation: WorkerGeneration, + ring_helper_abi: u16, + backend: JsonValue, +} + +WorkerFatal { + reason: WorkerFatalReason, +} + +WorkerStopped { + reason: WorkerStoppedReason, +} +``` + +`WorkerReady` means the arena is mapped, the native helper ABI is compatible, and +the backend can accept ring installation and execution commands. It does not mean +any role or ring is installed. + +### 8.2 Role Events + +Only needed if `ConfigureRole` is used: + +```rust +RoleConfigured { role_id: RoleId } +RoleFailed { role_id: RoleId, reason: RoleFailure } +``` + +### 8.3 Ring Events + +```rust +RingInstalled { + ring_id: RingId, + edge_id: EdgeId, + port_id: PortId, +} + +RingFault { + ring_id: RingId, + edge_id: EdgeId, + port_id: PortId, + reason: RingFaultReason, +} + +RingQuiesced { + ring_id: RingId, +} +``` + +### 8.4 Object Events + +```rust +ObjectLoaded { + ring_id: RingId, + edge_id: EdgeId, + port_id: PortId, + object_id: ObjectId, + sequence: Sequence, + extent: u64, + device_handle: DeviceObjectHandle, +} + +ObjectProduced { + ring_id: RingId, + edge_id: EdgeId, + port_id: PortId, + object_id: ObjectId, + sequence: Sequence, + extent: u64, +} + +ObjectFailed { + ring_id: RingId, + edge_id: EdgeId, + port_id: PortId, + object_id: Option, + sequence: Option, + reason: ObjectFailure, +} +``` + +`ObjectLoaded` means a full ingress object has been copied into device memory and +all copy events for that object have completed. + +`ObjectProduced` means the full egress object header and payload have been +committed to the egress ring. It does not mean the remote node received it. + +### 8.5 Step Events +USER: Why is this here? It seems useless. + + +```rust +StepCompleted { + role_id: RoleId, + step_id: StepId, +} + +StepFailed { + role_id: RoleId, + step_id: StepId, + reason: StepFailure, +} +``` + +### 8.6 Device Lifetime Events + +```rust +DeviceObjectReleased { + device_handle: DeviceObjectHandle, +} + +ReleaseFailed { + device_handle: DeviceObjectHandle, + reason: ReleaseFailure, +} +``` + +### 8.7 Wake Hints Sent By The Worker + +```rust +RingWritable { ring_id: RingId } +RingReadable { ring_id: RingId } +``` + +For ingress rings, the worker sends `RingWritable` after advancing `consume`. + +For egress rings, the worker sends `RingReadable` after advancing `commit`. + +Wake hints carry no byte counts, ranges, or ownership. + +### 8.8 GpuWorkerCtl Synthetic Events + +Some events are generated by `GpuWorkerCtl`, not read from the worker: + +```rust +WorkerCrashed { + worker_generation: WorkerGeneration, + status: ExitStatus, +} + +RingFault { + ring_id: RingId, + edge_id: EdgeId, + port_id: PortId, + reason: RingFaultReason, +} +``` + +`ExitStatus` is `swactor_process::ExitStatus`. + +`GpuWorkerCtl` synthesizes these events after process exit, process error, or +stdout control-stream failure. The worker cannot emit them because it is already +gone or no longer trustworthy. For process exit, the synthetic ring fault reason +is `WorkerProcessExited`. + +--- + +## 9. GpuWorkerCtl Actor FSM + +`GpuWorkerCtl` is a normal swactor actor. Its FSM is advanced by incoming +`GpuWorkerCtlMsg` values and by process notifications delivered through +`ProcessBridge`. + +```text +NotStarted + on StartWorker -> Spawning + +Spawning + spawn ProcessActor with ProcessSpec + spawn ProcessBridge + subscribe bridge to ProcessActor + wait for ProcessNotification::Started + send InitializeWorker through ProcessCommand::WriteStdin + -> Initializing + +Initializing + on WorkerReady -> Running + on WorkerFatal/process exit/timeout -> Failed + +Running + on actor command -> validate state, write worker command, update local tables + on worker event -> route event to local actors and driver + on ShutdownWorker -> Stopping + on process exit/error -> Crashed + +Stopping + send ShutdownWorker if process is still alive + wait for WorkerStopped and process exit + on timeout -> Killing + +Killing + send ProcessCommand::Close or stop/kill ProcessActor according to ProcessSpec + reap process through ProcessActor notification + mark installed rings faulted + -> Stopped + +Crashed + mark current-generation device handles invalid + mark installed rings faulted + ask driver to stop pumps for all installed rings + wait for teardown to release rings + -> Failed or Restarting + +Restarting + increment worker_generation + -> Spawning +``` + +`GpuWorkerCtl` must not claim a ring is quiesced merely because the worker exited. +Quiescence for arena reuse still requires the edge teardown proof from the ring +spec: pump stopped, worker process reaped or ring quiesced, and copy lifetimes +ended or the process that owns them is gone. + +--- + +## 10. Worker Process FSM + +The worker process has one state owner and handles process commands serially. It +may maintain multiple internal FSMs, especially one per installed ring, but there +is no separate worker scheduler runtime requirement. + +```text +Booting + read environment + wait for InitializeWorker on stdin + +Initializing + map arena + initialize native helper + initialize backend/tinygrad + emit WorkerReady + -> Running + +Running + on InstallRing -> install ring state + on RingReadable/RingWritable -> reload cursors and advance affected ring FSMs + on ExecuteStep -> run explicit role step subject to ring backpressure + on ReleaseDeviceObject -> release handle when safe + on ShutdownWorker -> Draining + +Draining + reject new work + finish or abort in-flight work according to shutdown mode + quiesce rings + emit WorkerStopped + exit + +Fatal + emit WorkerFatal if possible + exit non-zero +``` + +The worker should greedily make progress after any relevant command or wake hint: +drain readable ingress prefixes, advance egress output if writable space exists, +observe copy completions, and emit resulting events. The concrete internal data +structures used to remember ready rings or pending copies are implementation +details, provided duplicate wake hints cannot lose liveness. + +--- + +## 11. Per-Ring Worker State + +### 11.1 Ingress Ring FSM + +```text +Uninstalled + on InstallRing(direction = Ingress) -> NeedHeader + +NeedHeader + on committed bytes < header_len -> wait + on complete header -> validate + valid -> allocate device object -> NeedPayload + invalid -> Faulted + +NeedPayload + copy committed payload prefixes to device allocation + advance consume only after copied bytes are safe to release + if copied == extent -> ObjectComplete + on close before copied == extent -> Faulted + +ObjectComplete + wait for copy completion + emit ObjectLoaded + -> NeedHeader + +Faulted + emit ObjectFailed or RingFault + stop consuming until UninstallRing + +Uninstalling + stop consuming + wait for copy lifetimes to end + emit RingQuiesced + -> Uninstalled +``` + +### 11.2 Egress Ring FSM + +```text +Uninstalled + on InstallRing(direction = Egress) -> WaitingForOutput + +WaitingForOutput + on ExecuteStep output binding naming this ring -> NeedWritableForHeader + +NeedWritableForHeader + wait for writable span + write ObjectHeader + advance commit + emit RingReadable + -> NeedWritableForPayload + +NeedWritableForPayload + copy device payload prefixes into writable ring spans + advance commit only after host bytes are valid + emit/coalesce RingReadable + if produced == extent -> ObjectProduced + -> WaitingForOutput + +Faulted + emit RingFault + stop producing until UninstallRing + +Uninstalling + stop accepting outputs for this ring + wait for copy lifetimes to end + emit RingQuiesced + -> Uninstalled +``` + +The egress producer may block an `ExecuteStep` while waiting for ring space. That +is the intended backpressure path. + +--- + +## 12. Role and Step State + +Role state is intentionally minimal in this spec. + +```text +Unavailable + on WorkerReady with fixed boot role -> Ready + on ConfigureRole -> Configuring + +Configuring + load/record app-defined role config + on success -> Ready + on failure -> Failed + +Ready + on ExecuteStep -> run one explicit step + on Shutdown -> Stopping + +Stopping + reject new steps + release role-local resources +``` + +The MVP role concurrency policy is one active `ExecuteStep` per role. A later +worker may support multiple active steps only if the role declares that its state +is reentrant or the steps are explicitly independent. + +No separate step FSM is specified. The step either reaches `StepCompleted` or +`StepFailed`, and any partial egress object faults its ring. + +--- + +## 13. Message Flows + +### 13.1 Boot + +```text +GpuWorkerCtl -> swactor_process: spawn ProcessActor +ProcessActor -> OS: spawn worker with arena fd/env +ProcessActor -> GpuWorkerCtl: ProcessNotification::Started +GpuWorkerCtl -> ProcessActor: WriteStdin(InitializeWorker JSON line) +worker: map arena once +worker: initialize native helper and backend +worker -> stdout: WorkerReady JSON line +ProcessActor -> GpuWorkerCtl: ProcessNotification::Output(stdout) +GpuWorkerCtl: parse WorkerReady; node may now install rings and execute steps +``` + +Failure before `WorkerReady` is a boot failure. No rings are installed, so no +arena lease can be corrupted by the worker. + +### 13.2 Install Receive Edge + +```text +EdgeEstablisher receives RingLeased +EdgeEstablisher -> GpuWorkerCtl: InstallRing(direction = Ingress, edge_id, port_id) +GpuWorkerCtl -> worker: InstallRing +worker: open native ring handle and create ingress parser +worker -> GpuWorkerCtl: RingInstalled +GpuWorkerCtl -> EdgeEstablisher: RingInstalled +EdgeEstablisher -> Driver: EstablishRecv +Driver: pair recv spec with stream when available +Driver: spawn recv-pump +``` + +The driver does not start a recv-pump until the worker has installed the ring. + +### 13.3 Ingress Object Load + +```text +recv-pump reads QUIC bytes into free ingress ring span +recv-pump advances commit +recv-pump -> GpuWorkerCtl: RingReadable(ring_id) +GpuWorkerCtl -> worker: RingReadable(ring_id) + +worker reloads commit/consume +worker parses ObjectHeader +worker validates extent and sequence +worker allocates device object +worker copies committed payload prefixes to device +worker advances consume after copies are safe +worker -> GpuWorkerCtl: RingWritable(ring_id) +GpuWorkerCtl -> recv-pump: RingWritable(ring_id) + +when full object is copied and copy events complete: +worker -> GpuWorkerCtl: ObjectLoaded(..., device_handle) +GpuWorkerCtl routes ObjectLoaded to the role/Rx layer +``` + +No actor receives byte ranges. The ring cursors are the only byte ownership +state. + +### 13.4 Execute Step and Produce Output + +```text +role layer has all required input device handles +role layer chooses output object_id and sequence +role layer -> GpuWorkerCtl: ExecuteStep(inputs, outputs) +GpuWorkerCtl -> worker: ExecuteStep + +worker validates handles +worker calls tinygrad role code +worker writes output ObjectHeader to egress ring +worker copies output payload to egress ring as space becomes available +worker advances commit after host bytes are valid +worker -> GpuWorkerCtl: RingReadable(egress_ring) +GpuWorkerCtl -> send-pump: RingReadable + +send-pump writes committed bytes to QUIC +send-pump advances consume after write_all accepts bytes +send-pump -> GpuWorkerCtl: RingWritable(egress_ring) +GpuWorkerCtl -> worker: RingWritable + +worker -> GpuWorkerCtl: ObjectProduced +worker -> GpuWorkerCtl: StepCompleted +``` + +`StepCompleted` follows `ObjectProduced` for all declared outputs. For a step +with no outputs, `StepCompleted` follows successful compute and state update. + +### 13.5 Release Device Object + +```text +role layer decides object is no longer needed +role layer -> GpuWorkerCtl: ReleaseDeviceObject(device_handle) +GpuWorkerCtl -> worker: ReleaseDeviceObject +worker waits until no copy or compute references the object +worker frees device allocation +worker -> GpuWorkerCtl: DeviceObjectReleased +``` + +The node process never dereferences a device handle. It can only pass it back to +the worker. + +### 13.6 Stop Edge + +```text +EdgeEstablisher -> Driver: StopEdge +Driver stops pump +Driver -> EdgeEstablisher: PumpStopped + +EdgeEstablisher -> GpuWorkerCtl: UninstallRing +GpuWorkerCtl -> worker: UninstallRing +worker removes ring from active state +worker waits for copy lifetimes to end +worker -> GpuWorkerCtl: RingQuiesced +GpuWorkerCtl -> EdgeEstablisher: RingQuiesced + +EdgeEstablisher -> ArenaManager: ReleaseRing(proof) +``` + +The arena lease is released only after both the driver pump and worker ring are +quiesced, or after the worker process is dead and reaped. + +### 13.7 Worker Crash + +```text +worker process exits or process actor reports error +GpuWorkerCtl marks all current-generation device handles invalid +GpuWorkerCtl emits WorkerCrashed to role layer +GpuWorkerCtl synthesizes RingFault for every installed ring +GpuWorkerCtl asks Driver to stop pumps for those rings +EdgeEstablisher waits for PumpStopped +worker side is considered gone only after process reap +EdgeEstablisher releases arena rings after teardown proof +``` + +The arena survives because it is owned by the node process. Device allocations do +not survive because they were owned by the worker process/backend context. + +Restart creates a new worker generation. Rings and roles must be reinstalled. + +--- + +## 14. Error Taxonomy + +### 14.1 RingFaultReason + +```rust +enum RingFaultReason { + UnsupportedRingVersion, + RingLayoutInvalid, + RingStateInvalid, + DeviceOutOfMemory, + DeviceCopyFailed, + SequenceViolation, + HeaderMalformed, + WorkerProcessExited, + WorkerShuttingDown, + WorkerInternal, +} +``` + +### 14.2 ObjectFailure + +```rust +enum ObjectFailure { + HeaderMalformed, + UnsupportedObjectVersion, + ExtentExceedsMax, + ExtentAlignmentInvalid, + SequenceViolation, + DeviceAllocationFailed, + DeviceCopyFailed, + EofMidObject, +} +``` + +### 14.3 StepFailure + +```rust +enum StepFailure { + RoleUnavailable, + InvalidInputHandle, + InvalidOutputRing, + TinygradError, + DeviceOutOfMemory, + OutputExtentInvalid, + OutputCopyFailed, + WorkerProcessExited, + WorkerShuttingDown, +} +``` + +### 14.4 WorkerFatalReason + +```rust +enum WorkerFatalReason { + ArenaMapFailed, + RingHelperAbiMismatch, + BackendInitFailed, + MalformedControlMessage, + UnhandledException, +} +``` + +### 14.5 Role and Release Failures + +```rust +enum RoleFailure { + InvalidRoleConfig, + BackendUnsupported, + WorkerRejectedRole, +} + +enum ReleaseFailure { + InvalidDeviceHandle, + ObjectInUse, + BackendFreeFailed, +} +``` + +### 14.6 Control Reasons + +```rust +enum UninstallReason { + EdgeStopped, + RingFaulted, + WorkerShutdown, +} + +enum WorkerStoppedReason { + Graceful, + AbortInFlight, + Fatal, +} +``` + +MVP policy: device OOM during ingress object allocation faults the ring. The +worker does not silently skip a payload and continue to later objects. Replanning +or retry is a higher-level policy. + +--- + +## 15. Correctness and Safety Invariants + +**Control messages carry no payload bytes.** Payload movement occurs only through +arena-backed rings. This keeps actors and process stdio out of the byte hot path. + +**The worker cannot access an unleased arena range.** `InstallRing` is sent only +after ArenaManager emits `RingLeased`; the message contains the only usable ring +layout. + +**The driver cannot pump into a ring the worker has not installed.** +EdgeEstablisher sends `EstablishRecv` or `EstablishSend` only after +`RingInstalled`. + +**Python does not own ring atomics.** All cursor loads, cursor stores, span +calculation, and wrap arithmetic use the native ring helper. + +**Ingress compute visibility starts at `ObjectLoaded`.** The worker emits +`ObjectLoaded` only after header validation, exact extent copy, copy completion, +and device handle creation. The role layer cannot pass a partial object to +`ExecuteStep` because no handle exists before then. + +**The recv-pump cannot overwrite host bytes still needed by DMA.** The worker +advances ingress `consume` only after the corresponding host bytes are no longer +read by synchronous copy or asynchronous DMA. + +**The send-pump cannot read unwritten egress bytes.** The worker advances egress +`commit` only after header or payload bytes are valid in host memory. + +**Backpressure is absence of writable ring space.** If the worker is slow, +ingress `consume` does not advance and the recv-pump stops reading QUIC. If the +network is slow, egress `consume` does not advance and the worker stops producing +more bytes into the egress ring. + +**Wake hints are not ownership.** `RingReadable` and `RingWritable` only cause +the receiver to reload cursors. Duplicate hints may be coalesced only while the +ring remains discoverable. + +**Device handles are generation-scoped.** A handle from an old worker generation +is rejected by the new worker. This prevents stale handles from aliasing new +device allocations after restart. + +**Object id and sequence ownership is explicit.** Ingress object id and sequence +come from the received `ObjectHeader` and are validated against `ObjectSpec`. +Egress object id and sequence are assigned by the role/control layer in +`ExecuteStep`; the worker serializes them but does not invent graph-visible +ordering. + +**Role execution is explicit.** Loading an object does not run compute. The role +layer must issue `ExecuteStep` with concrete input handles and output bindings. + +**Partial egress objects fault the ring.** If the worker commits an output header +but cannot commit the full declared payload, the egress ring is faulted. The +receiver must not interpret later bytes as the next object. + +**A released arena range has no live users.** `ReleaseRing` is sent only after +the driver pump is stopped and the worker has emitted `RingQuiesced`, or after +the worker process is reaped. Copy lifetimes must have ended or the process that +owned them must be gone. + +**Worker restart does not preserve GPU state.** Roles, rings, steps, and device +handles are all current-generation state. Restart requires reconfiguration. + +--- + +## 16. Implementation Requirements + +### 16.1 Actor Integration + +The Rust-side implementation must follow the existing process-actor idiom: + +- `GpuWorkerCtl` is an `ActorInterface` implementation with one incoming enum +- it spawns a `ProcessActor` through `swactor_process` +- it spawns a private bridge actor for `ProcessNotification` +- it sends worker input with `ProcessCommand::WriteStdin` +- it closes the worker with `ProcessCommand::Close` and normal actor teardown +- it never exposes process stdin/stdout handles to driver pumps or edge actors + +### 16.2 Copy Safety + +For synchronous copies: + +```text +copy returns -> safe to advance relevant cursor +``` + +For asynchronous copies: + +```text +start copy -> record CopyEvent and host/device ranges +event complete -> advance relevant cursor or release device object +``` + +The worker must track host ranges for ring cursor release and device ranges for +device object lifetime. + +### 16.3 Output Commit Discipline + +For egress: + +1. The worker must know the output extent before writing the object header. +2. The worker writes and commits the header. +3. The worker copies payload prefixes into writable spans. +4. The worker commits each prefix only after host bytes are valid. +5. The worker emits `ObjectProduced` only after all `extent` bytes are committed. + +If the worker cannot determine output extent before serialization, it must first +materialize the output into a device object with known byte extent. The wire +header cannot contain a placeholder extent. + +### 16.4 Logs and Diagnostics + +Worker events go to stdout as process-adapter messages. Diagnostic logs go to +stderr. + +`GpuWorkerCtl` may keep a stderr tail for crash diagnostics. This is diagnostic +only and does not change safety rules. + +--- + +## 17. Resolutions To Known Design Issues + +### 17.1 No Custom Control Frames + +Decision: remove length-prefixed frames, generic envelopes, per-message protocol +versions, command ids, and reply ids. Rust code uses normal swactor messages. +The Python subprocess adapter may use newline-delimited JSON because the +codebase already uses `ProcessActor` stdin/stdout for process-backed workers. + +Reason: this worker is integrated into swactor. A second local control protocol +duplicates actor semantics without solving a current problem. + +### 17.2 Ring Install Needs Edge and Port Identity + +Decision: `InstallRing` includes `edge_id` and `port_id`. The worker reports all +object and ring events with `ring_id`, `edge_id`, and `port_id`. + +Reason: `ring_id` is a node-local transport identifier. The role and edge layers +need graph-facing identity without maintaining an ambiguous reverse lookup in +the worker. + +### 17.3 Device Handle Lifetime + +Decision: device handles are opaque, generation-scoped, and explicitly released +with `ReleaseDeviceObject`. + +Reason: this prevents stale handle aliasing after restart and gives the role +layer explicit control over activation and cache lifetimes. + +### 17.4 Compute Driving API + +Decision: compute is driven by `ExecuteStep`; `ConfigureRole` is optional and +opaque until role provisioning exists. `ObjectLoaded` is a data-ready event, not +an execution trigger. + +Reason: the control plane owns ordering and correlation. + +### 17.5 Object Id And Sequence Ownership + +Decision: ingress validates object id and sequence from the stream header. +Egress object id and sequence are assigned by the role/control layer and supplied +in `ExecuteStep.OutputBinding`. + +Reason: the worker should not invent graph-visible ordering. The control plane +already knows the expected ordering and correlation. + +### 17.6 Weight Loading + +Decision: weight/persistent-state specifics are deferred with role provisioning. +For now, weights can be loaded as ordinary ingress objects and passed to +`ExecuteStep`, or a fixed-role worker can load them during its own initialization +from app-specific config. + +Reason: this avoids designing binding APIs before the role provisioning path +exists. + +### 17.7 Multi-Port And Fan-In + +Decision: rings bind to ports, and `ExecuteStep` lists all input handles and +output rings explicitly. Fan-in is represented at the role layer by waiting for +multiple `ObjectLoaded` events before issuing one `ExecuteStep`. + +Reason: the ring remains SPSC and byte-level simple. Join semantics stay out of +the ring and pump layers. + +### 17.8 GPU OOM And Allocation Pressure + +Decision: ingress allocation failure faults the ring in the MVP. Step-time OOM +fails the step. If an egress object was partially committed, the egress ring is +faulted. + +Reason: skipping a payload or continuing after a partial object requires more +protocol machinery and can violate stream alignment. Replanning or retry belongs +above the worker boundary. + +### 17.9 Error Separation + +Decision: use separate failure classes for worker fatal errors, ring faults, +object failures, release failures, and step failures. + +Reason: recovery action differs. A malformed object is not the same as a process +crash, and a tinygrad exception is not the same as a corrupt ring cursor. + +### 17.10 tinygrad Range-Copy Gap + +Decision: the worker implementation must provide a `DeviceBridge` with range +copy support. If tinygrad does not expose the necessary primitive directly, the +bridge must use a native backend helper or tinygrad raw-buffer API. The ring +protocol must not be weakened to require `extent <= ring_capacity`. + +Reason: the ring spec explicitly allows objects larger than rings. Partial +prefix copy is required for liveness and backpressure. + +### 17.11 Restart Semantics + +Decision: restart creates a new worker generation and invalidates all device +handles, role instances, ring installs, and in-flight steps. The node may reuse +the same arena only after normal ring teardown proves quiescence. + +Reason: GPU backend state is process-local. Treating restart as transparent +would risk stale handles, leaked DMA, and mismatched role state. diff --git a/ORCHESTRATION_SPEC.md b/ORCHESTRATION_SPEC.md new file mode 100644 index 0000000..f6f084f --- /dev/null +++ b/ORCHESTRATION_SPEC.md @@ -0,0 +1,518 @@ +# GGUF Pipeline Orchestration - MVP Specification + +**Status:** draft buildout specification. + +**Relationship to other documents.** `DESIGN_DIRECTIVES.md` remains steering +context. `RING_BACKPRESSURE_SPEC.md` defines edge establishment, object records, +rings, pumps, and teardown. `GPU_WORKER_INTERFACE_SPEC.md` defines worker +startup, shard/weight binding, and `ExecuteStep`. This document defines the +missing layer above them: how the orchestrator plans and stages one linear GGUF +pipeline inference run. + +This document is intentionally not a general graph specification. + +--- + +## 1. Scope + +The MVP workload is pipeline-parallel inference from a GGUF model: + +```text +orchestrator --tokens--> stage 0 --activations--> stage 1 --activations--> +... --activations--> stage N-1 --tokens--> orchestrator +``` + +The orchestrator is a control participant and token endpoint. It does not run GPU +compute. GPU compute happens only inside provisioned stages. + +This spec covers: + +- building a linear stage plan from a GGUF model and a provisioned GPU pool +- assigning GGUF shard/layer ranges to stages +- assigning run-scoped edge ids +- provisioning each stage with the facts it needs +- defining the readiness barrier before prompt injection +- defining the orchestrator and stage FSMs +- defining security and correctness guarantees for the MVP behavior + +This spec does not cover: + +- arbitrary graph execution +- automatic placement optimization +- batching, speculative decoding, or continuous serving +- failure recovery by re-placement +- detailed ring, pump, or worker internals already specified elsewhere +- behavioral test contracts; those come after the full system shape is drafted + +--- + +## 2. Core Responsibilities + +The orchestrator owns: + +- run ids +- stage count and stage order +- GGUF shard/layer assignment +- edge id assignment +- stage provisioning +- the global readiness barrier +- prompt injection +- final token consumption +- EOS and `max_tokens` stop policy +- run-level fault and teardown + +Each stage owns: + +- loading its assigned GGUF shard/layer range +- configuring its local GPU worker +- establishing its local edge ends +- converting loaded inbound objects into local `ExecuteStep` calls +- producing the next object on its outbound edge +- reporting readiness and faults to the orchestrator + +The ring and worker specs own the byte movement and GPU worker command details. +This spec only decides which stages and edges exist and what sequence of control +events makes the run progress. + +--- + +## 3. Run Plan + +The orchestrator builds one `RunPlan` before provisioning: + +```rust +struct RunPlan { + run_id: RunId, + model: GgufModelPlan, + stages: Vec, + edges: Vec, + max_tokens: u32, +} + +struct GgufModelPlan { + model_id: String, + gguf_source: GgufSource, + num_layers: u32, + hidden_dim: u32, + dtype_family: DTypeFamily, + dtype_width_bytes: u32, + max_seq_len: u32, + eos_token_id: u32, +} +``` + +`gguf_source` may identify a whole GGUF file, a pre-split shard collection, or a +cache key. The orchestration contract is the assigned layer range. Whether the +node reads only part of a whole GGUF file or receives a physically pre-split +artifact is a local loading detail. + +Each stage receives a contiguous layer range: + +```rust +struct StagePlan { + run_id: RunId, + stage_index: u32, + stage_count: u32, + node_id: NodeId, + gguf_source: GgufSource, + layer_start: u32, + layer_end_exclusive: u32, + inbound_edge: EdgeId, + outbound_edge: EdgeId, +} +``` + +For stage `0`, `inbound_edge` is the token edge from the orchestrator. For the +last stage, `outbound_edge` is the token edge back to the orchestrator. Interior +edges carry activations. + +Each edge has exactly one producer and one consumer: + +```rust +struct EdgePlan { + run_id: RunId, + edge_id: EdgeId, + kind: EdgeKind, + producer: EdgeEndpoint, + consumer: EdgeEndpoint, + object_spec: ObjectSpec, + ring_spec: RingSpec, +} + +enum EdgeKind { + TokenIn, + Activation, + TokenOut, +} + +enum EdgeEndpoint { + Orchestrator { node_id: NodeId }, + Stage { node_id: NodeId, stage_index: u32 }, +} +``` + +The orchestrator assigns all `edge_id`s. Stage code never derives edge ids from +names, layer ranges, peer ids, or hashes. + +--- + +## 4. Stage Provisioning Message + +The orchestrator sends one provision message to each stage node: + +```rust +ProvisionStage { + run_id: RunId, + stage_index: u32, + stage_count: u32, + gguf_source: GgufSource, + layer_start: u32, + layer_end_exclusive: u32, + inbound: InboundEdgeProvision, + outbound: OutboundEdgeProvision, + model: StageModelFacts, +} + +struct InboundEdgeProvision { + edge_id: EdgeId, + kind: EdgeKind, + object_spec: ObjectSpec, + ring_spec: RingSpec, +} + +struct OutboundEdgeProvision { + edge_id: EdgeId, + kind: EdgeKind, + consumer_node_id: NodeId, + object_spec: ObjectSpec, + ring_spec: RingSpec, +} + +struct StageModelFacts { + model_id: String, + hidden_dim: u32, + dtype_family: DTypeFamily, + dtype_width_bytes: u32, + max_seq_len: u32, +} +``` + +The inbound edge is established locally as a receive edge. The outbound edge is +established locally as a send edge to `consumer_node_id`. For the last stage, +`consumer_node_id` is the orchestrator node. + +The node-local stage controller translates this provision message into the lower +level operations: + +```text +configure worker for assigned stage +load/bind assigned GGUF shard or layer range +provision receive edge for inbound.edge_id +provision send edge for outbound.edge_id +report StageReady when all required local work is complete +``` + +--- + +## 5. Orchestrator FSM + +The orchestrator has one run-level FSM: + +```text +Planning + build RunPlan + validate layer ranges and edge ids + -> Provisioning + +Provisioning + send ProvisionStage to every stage node + create local token producer for token-in edge + create local token consumer for token-out edge + -> WaitingReady + +WaitingReady + on StageReady for every stage and local token endpoints ready + -> Running + on StageFault or timeout + -> Faulted + +Running + inject prompt token object on token-in edge, sequence 0 + consume token objects from token-out edge in sequence order + after token sequence k: + if EOS or max_tokens reached -> Completed + else inject token object sequence k + 1 on token-in edge + on StageFault, edge fault, local token endpoint fault, or timeout + -> Faulted + +Completed + stop injecting tokens + finalize output text + -> TearingDown + +Faulted + stop injecting tokens + record one run-level failure reason + -> TearingDown + +TearingDown + send StopRun to all provisioned stages + tear down local token endpoints + wait for StageStopped from every stage or timeout + -> Done + +Done + terminal +``` + +There is no broadcast start message. The start signal is the first token object +written by the orchestrator after the readiness barrier. + +--- + +## 6. Stage FSM + +Each provisioned stage has one node-local stage controller. It is control-path +only: it watches local worker and edge events and issues worker commands. It does +not move payload bytes. + +```text +Unprovisioned + on ProvisionStage from authorized orchestrator + validate run_id and stage assignment + -> Preparing + +Preparing + configure local worker for assigned GGUF range + start GGUF shard/range load and bind + establish inbound receive edge + establish outbound send edge + when worker configured, shard bound, and both edge ends ready + emit StageReady + -> Ready + on any required setup failure + emit StageFault + -> Faulted + +Ready + on inbound ObjectLoaded(sequence = s) + if s is the next expected sequence + issue ExecuteStep for sequence s + -> Executing + else + emit StageFault(sequence violation) + -> Faulted + on StopRun + -> Stopping + +Executing + worker runs exactly one step for the loaded inbound object + worker writes the output object to outbound edge with the same sequence + on StepCompleted + release any per-step input handle that is no longer needed + -> Ready + on StepFailed or output fault + emit StageFault + -> Faulted + +Faulted + reject new run work + wait for StopRun + -> Stopping + +Stopping + stop local edges + release per-run device objects + stop or reset worker according to local policy + emit StageStopped + -> Stopped + +Stopped + terminal for this run +``` + +The controller is the component that decides when `ExecuteStep` is called. The +orchestrator does not issue per-stage execute commands during the run. Once the +prompt object is injected, stage execution is driven by object arrival and local +readiness. + +--- + +## 7. Execution Semantics + +Sequence `0` is prefill. + +```text +orchestrator writes prompt token object sequence 0 +stage 0 executes prefill over prompt tokens +stage 0 writes activation sequence 0 +each interior stage executes prefill over activation sequence 0 +last stage executes prefill and writes token sequence 0 +orchestrator consumes token sequence 0 +``` + +Decode sequences are `1..`: + +```text +orchestrator writes one-token object sequence k +stage 0 executes decode for sequence k +each downstream stage executes decode for sequence k +last stage writes token sequence k +orchestrator consumes token sequence k +``` + +The orchestrator sends sequence `k + 1` only after consuming token sequence `k` +and deciding the run should continue. + +For every stage: + +- the inbound object sequence is the output object sequence +- one active `ExecuteStep` per stage is allowed in the MVP +- a stage cannot execute before its assigned GGUF shard/range is loaded and bound +- a stage cannot execute before its inbound object is loaded +- a stage cannot produce to an edge that is not ready + +The last stage samples or otherwise produces token ids as part of its GPU worker +step. The orchestrator consumes those token ids, accumulates output, applies EOS +and `max_tokens`, and writes the next token object only when continuing. + +--- + +## 8. Control Messages And Events + +These are schematic message shapes, not final Rust APIs. + +Orchestrator to stage: + +```rust +ProvisionStage { ... } + +StopRun { + run_id: RunId, + reason: StopReason, +} +``` + +Stage to orchestrator: + +```rust +StageReady { + run_id: RunId, + stage_index: u32, + node_id: NodeId, +} + +StageFault { + run_id: RunId, + stage_index: u32, + node_id: NodeId, + reason: StageFaultReason, +} + +StageStopped { + run_id: RunId, + stage_index: u32, + node_id: NodeId, +} +``` + +Optional setup progress events may exist for diagnostics, but `StageReady`, +`StageFault`, and `StageStopped` are the only required run-level events in this +draft. + +--- + +## 9. Object Specs + +Token edges carry token objects. The prompt token object may contain multiple +token ids for prefill. Decode token objects contain one token id. + +Activation edges carry activation objects with runtime extent bounded by model +shape: + +```text +max_extent = max_seq_len * hidden_dim * dtype_width_bytes +``` + +The object record and ring behavior are defined by `RING_BACKPRESSURE_SPEC.md`. +This orchestration spec only requires that all stage plans for a run agree on the +model facts used to build those object specs. + +--- + +## 10. Security Model + +Nodes are trusted. The system does not attempt trustless verification, +adversarial tensor validation, Sybil defense, or incentive enforcement. + +The orchestrator is the authority for run topology. A stage accepts run +provisioning only from the authorized orchestrator for its node. + +Stages reject: + +- unknown `run_id` +- stale `run_id` +- duplicate provisioning for an already-active run unless explicitly stopped +- edge ids not present in the provision message +- peer rewiring requests from another stage + +`edge_id`s are run-scoped capabilities for wiring and demux. They are not a +cryptographic trust boundary between trusted nodes, but a stage must still reject +objects and stream setup that do not match its active run plan. + +The orchestrator may tear down a run at any time. Stages must treat `StopRun` for +their active `run_id` as authoritative. + +--- + +## 11. Correctness Guarantees + +Layer assignment: + +- stage layer ranges are contiguous +- stage layer ranges do not overlap +- the union of stage layer ranges covers the intended GGUF block range +- every stage has exactly one assigned range + +Edge assignment: + +- every `edge_id` is unique within a run +- every edge has exactly one producer and one consumer +- token-in is produced by the orchestrator and consumed by stage `0` +- token-out is produced by stage `N - 1` and consumed by the orchestrator +- activation edge `i` is produced by stage `i` and consumed by stage `i + 1` + +Readiness: + +- the orchestrator does not inject prompt tokens before every stage reports + `StageReady` +- a stage does not report `StageReady` before its worker, shard/range binding, + inbound edge, and outbound edge are ready + +Execution: + +- prefill is sequence `0` +- decode sequences are strictly increasing +- a stage executes sequence `s` only after loading inbound object sequence `s` +- a stage output uses the same sequence as its input +- the orchestrator injects sequence `s + 1` only after consuming token sequence + `s` + +Termination: + +- each run has one terminal outcome: completed, faulted, or torn down +- after a run faults, the orchestrator stops injecting new token objects +- teardown is sent to every stage that was provisioned for the run + +--- + +## 12. Deferred + +- placement optimization +- physical GGUF shard format +- multiple concurrent runs on one stage chain +- batching and speculative decoding +- direct stage-to-stage token feedback that bypasses the orchestrator +- warm reuse policy across prompts +- re-placement after node failure +- behavioral test matrix and observability schema diff --git a/RING_BACKPRESSURE_SPEC.md b/RING_BACKPRESSURE_SPEC.md new file mode 100644 index 0000000..9b93807 --- /dev/null +++ b/RING_BACKPRESSURE_SPEC.md @@ -0,0 +1,1522 @@ +# Ring-Backpressured Data Movement - Canonical Specification + +**Status:** design specification. This document supersedes the previous root +workflow drafts for edge establishment, blob streaming, driver streams, stream +transport, and arena management: + +- `EDGE_ESTABLISHMENT.md` +- `BLOB_STREAMING.md` +- `STREAM_TRANSPORT.md` +- `DRIVER_STREAMS.md` +- `ARENA_MANAGEMENT.md` + +`DESIGN_DIRECTIVES.md` remains steering context. This file is the buildout +specification. + +**Scope.** Node-local host memory, process-crossing rings, worker loading, +edge establishment, persistent QUIC streams, actor/control messages, safety +contracts, backpressure, and teardown. This covers activations, weights, and +other large objects moving between disk, network, host memory, and GPU memory. + +**Out of scope.** Compute overlap with partial tensors, placement policy, +churn placement, trustless verification, object-store semantics, and high-level +model graph scheduling. This spec permits transfer and VRAM upload to overlap +with network streaming. It deliberately does not permit matmuls or other compute +to observe an object until the whole logical object is loaded. + +--- + +## 1. Shape + +A node is an async backpressure machine built from one stable host-memory arena +and many bounded byte rings. + +The arena is address space. It is one large sparse `memfd`, created by the Rust +node process, mapped once by the Rust node process, and mapped once by the GPU +worker process. It does not define flow control and it does not define object +ownership. + +Rings define flow control. Every payload-moving boundary is represented as a +bounded single-producer/single-consumer byte ring backed by a lease inside the +arena: + +- QUIC ingress stream -> host ingress ring -> worker -> GPU memory +- GPU memory -> worker -> host egress ring -> QUIC egress stream +- disk reader -> host ring -> worker/GPU memory +- future GPU download/upload and network paths follow the same ring contract + +Actors establish, supervise, and tear down rings. Actors do not move payload +bytes, do not relay per-range readiness, and do not track per-byte ownership. +Once a ring is active, the hot path is shared ring metadata plus coalesced wake +hints: + +```text +producer writes bytes into ring +producer advances commit cursor +producer sends/coalesces a readable wake hint + +consumer wakes +consumer drains every committed byte it can use +consumer advances consume cursor +consumer sends/coalesces a writable wake hint if space was released +``` + +The wake hint is a signal that ring state may have changed. It carries a +`ring_id` and a reason such as readable or writable, not byte counts or ranges. +The producer may send it as soon as bytes are committed; it does not wait for the +ring to fill. Duplicate hints may be coalesced through a ready set or pending bit, +but a ring that has become readable or writable must remain discoverable until +the other side reloads the cursors. The cursors are the state. + +The worker can load ingress into VRAM while the stream is still arriving. It does +that by greedily consuming the committed prefix of the ingress byte ring and +copying those bytes into the correct offsets of a device allocation. The object +becomes compute-visible only after all expected bytes have been copied and the +worker has observed object completion. + +--- + +## 2. Vocabulary + +**node process** - The Rust process hosting the swactor runtime, actors, driver, +arena manager, iroh endpoint, pump tasks, and worker supervisor. + +**GPU worker** - A separate OS process, normally Python plus tinygrad and a small +native ring helper. It maps the arena, receives coalesced ring wake hints, parses +object streams, copies payload bytes into or out of GPU memory, and emits coarse +events. + +**arena** - One sparse `memfd` reservation per node. The node process and worker +process map the same bytes. Arena offsets are stable for the node lifetime. + +**arena lease** - A non-overlapping byte range in the arena assigned to one ring. +The ArenaManager mints leases and releases them after quiescence. + +**ring** - A bounded single-producer/single-consumer byte stream backed by an +arena lease. A ring has shared metadata, data bytes, one producer, one consumer, +and coalesced wake hints. + +**edge** - A one-way typed conduit from one role to another role. An edge has one +producer node, one consumer node, one `edge_id`, and one persistent QUIC +uni-stream once data begins flowing. + +**`edge_id`** - A run-global edge identifier assigned by the orchestrator. It is +the control-plane name of the edge and the fixed-width stream preamble used by +the receiver's driver demux. + +**object** - One logical payload on an edge, such as an activation tensor, a +weight tensor, a token batch, or a model shard. Objects are sent as object +records inside the edge byte stream. + +**extent** - The actual byte length of an object. It is a runtime fact and may be +smaller than the edge's maximum object capacity. + +**object spec** - The role-known contract for objects on an edge: maximum extent, +dtype family, shape/layout rules, object kind, and any alignment requirements. +The worker uses this spec to turn raw bytes into a correctly shaped device +allocation. + +**ring spec** - The size and operating parameters of a ring: data capacity, +alignment, optional host-pinning requirement, wake coalescing mode, and whether +the ring is ingress or egress. + +**pump** - A driver-owned async task. A recv-pump copies QUIC bytes into an +ingress ring. A send-pump copies egress ring bytes onto QUIC. Pumps do not parse +payload objects after the `edge_id` preamble. + +**Tx/Rx edge actors** - Small swactor actors representing the local edge end. +They hold edge identity and lifecycle state, receive coarse completion/fault +events, and integrate with the role layer. They do not receive per-byte or +per-range messages. + +--- + +## 3. Process Topology + +Each node has two payload-relevant OS processes: + +```text +node process (Rust) GPU worker process +------------------- ------------------ +swactor runtime tinygrad / CUDA +ArenaManager actor ring parser +EdgeEstablisher actor device allocator +Tx/Rx edge actors host-to-device / device-to-host copies +Driver mailbox +iroh endpoint +recv/send pumps + + shared memfd arena, mapped by both processes + actor/control messages for install, wake hints, and coarse events +``` + +Host-to-device means copying from arena-backed host memory into GPU memory. +Device-to-host means copying from GPU memory into arena-backed host memory. + +The node process creates the `memfd` without `CLOEXEC` before spawning the worker, +or otherwise passes the fd explicitly during worker startup. The worker maps the +same reservation once. Neither process remaps the arena during node lifetime. + +For a process-crossing ring: + +- `RingReadable { ring_id }` wakes the consumer after the producer commits bytes. +- `RingWritable { ring_id }` wakes the producer after the consumer releases bytes. + +For ingress, the producer is the Rust recv-pump and the consumer is the worker. +For egress, the producer is the worker and the consumer is the Rust send-pump. + +The worker control pipe carries lifecycle messages, wake hints, and coarse events +such as `InstallRing`, `RingInstalled`, `ObjectLoaded`, and `RingFault`. It does +not carry payload bytes, byte counts, or per-range ownership. + +--- + +## 4. Arena Manager + +The ArenaManager is the single per-node authority for arena layout. + +### 4.1 Responsibilities + +The ArenaManager owns: + +- the arena `memfd` +- the node process mapping base pointer +- the reservation ceiling +- the arena free-list +- the pending lease queue +- the table of live ring leases + +The ArenaManager does not own: + +- any worker process +- any QUIC stream +- any pump task +- any object parser +- any payload byte + +It never reads or writes payload bytes. It only mints stable offsets. + +### 4.2 Boot + +At node boot: + +1. The ArenaManager creates the `memfd`. +2. It truncates it to a generous sparse ceiling. +3. It maps the whole reservation once in the node process. +4. It exposes the base pointer to the driver and ring constructors. +5. It makes the fd available to the worker process at spawn. + +The reservation costs virtual address space. Physical pages are backed lazily by +the kernel when touched. The mapping is not moved or resized. Any offset minted +by the ArenaManager remains meaningful until the node shuts down. + +### 4.3 FSM + +```text +Booting + on ConstructArena{ceiling} + -> Ready if memfd, truncate, and mmap succeed + -> Failed if any boot resource cannot be created + +Ready + on LeaseRing{request_id, requester, edge_id, direction, ring_spec} + -> lease immediately and emit RingLeased if a range fits + -> enqueue request if the request is satisfiable but no current range fits + -> emit RingLeaseRejected if the ring_spec can never fit in the ceiling + +Ready + on CancelLease{request_id} + -> remove queued request if it has not been leased yet + +Ready + on ReleaseRing{ring_id, proof} + -> return range to free-list + -> retry queued leases serially + +Ready + on Shutdown + -> ShuttingDown + +ShuttingDown + no new leases are accepted +``` + +### 4.4 Messages + +Inbound: + +```rust +ConstructArena { + ceiling: u64, +} + +LeaseRing { + request_id: LeaseRequestId, + requester: ActorAddress, + edge_id: EdgeId, + direction: RingDirection, + ring_spec: RingSpec, +} + +CancelLease { + request_id: LeaseRequestId, +} + +ReleaseRing { + ring_id: RingId, + proof: QuiescenceProof, +} + +Shutdown +``` + +Outbound: + +```rust +ArenaReady { + base_ptr: NonNull, + ceiling: u64, +} + +RingLeased { + request_id: LeaseRequestId, + requester: ActorAddress, + edge_id: EdgeId, + ring_id: RingId, + direction: RingDirection, + arena_offset: u64, + layout: RingLayout, +} + +RingLeaseRejected { + request_id: LeaseRequestId, + requester: ActorAddress, + edge_id: EdgeId, + reason: LeaseRejectReason, +} +``` + +There is no temporary allocation-failure message. Temporary pressure is encoded +by absence of `RingLeased`: the request waits in the lease queue. Permanent +impossibility is explicit because no future release can make an oversized ring +fit. A queued request can be cancelled by `request_id` if the edge establishment +record stops before the lease is granted. + +### 4.5 Correctness + +Two live leases cannot overlap because all lease and release operations pass +through one ArenaManager mailbox. A handler mutates the free-list to completion +before the next handler runs. The allocator either removes one complete range +from the free-list and records it in the live table, or it leaves the free-list +unchanged and queues/rejects the request. There is no state in which a partial +lease is visible downstream. + +A pump or worker cannot observe an unleased range because `RingLeased` is the +only message that contains a usable ring offset. Edge establishment does not +install a worker ring or spawn a pump until that message exists. + +A range cannot be reused under a live pump or worker because `ReleaseRing` is a +proof, not a request. The EdgeEstablisher emits it only after the driver has +stopped the pump, the worker has uninstalled the ring, and in-flight DMA for the +ring has completed or been abandoned with the worker process dead. The +ArenaManager does not infer quiescence; it relies on the upstream teardown FSM +to earn the proof. + +Queued lease requests cannot corrupt establishment because the requester receives +nothing while queued. No ring offset exists, so no driver or worker hot-path state +can be created for that ring. If a stop races with a grant, the +EdgeEstablisher accepts `RingLeased` only when the matching edge record is still +waiting on the same `request_id`; otherwise it releases the unused lease without +installing a worker ring or spawning a pump. + +--- + +## 5. Ring Contract + +A ring is a bounded SPSC byte stream in shared memory. It is the universal +payload handoff primitive. + +The process-crossing ring is a fixed shared-memory ABI, not a Rust collection +placed inside the arena. In-process queues such as `crossbeam_queue::ArrayQueue` +may be used for local actor channels, ready sets, or wake scheduling, but the +arena ABI stores only offsets, cursors, state bits, and payload bytes. This keeps +the mapped bytes valid even when the node process and worker process map the +same `memfd` at different virtual addresses. + +### 5.1 Single Producer, Single Consumer + +Each ring has exactly one producer and one consumer. + +Ingress: + +```text +producer = recv-pump +consumer = worker +``` + +Egress: + +```text +producer = worker +consumer = send-pump +``` + +Disk or future GPU rings follow the same rule. Fan-in or fan-out is represented +by multiple rings or by a higher-level mux/demux component that itself owns one +side of a ring. A ring never has multiple hot-path producers or consumers. + +### 5.2 Shared-Arena ABI + +The ring header lives in shared memory and is aligned for cross-process atomic +operations. + +```rust +#[repr(C, align(64))] +struct RingHeader { + magic: u32, + version: u16, + header_len: u16, + ring_id: u64, + capacity: u64, + commit: AtomicU64, + consume: AtomicU64, + state: AtomicU32, + wake: AtomicU32, +} +``` + +`ring_id` is unique for the node lifetime. Arena ranges may be reused after +quiescence, but ring identifiers are not reused. Stale control or wake events +therefore cannot alias a later ring that happens to occupy the same arena range. + +`commit` is the first byte after the committed readable prefix. Bytes with +logical positions `< commit` are valid for the consumer to read. + +`consume` is the first byte not yet released by the consumer. Bytes with logical +positions `< consume` are free for the producer to reuse. + +The producer also keeps a local `write` cursor. `write` is the first byte after +the producer's reserved or in-progress write prefix. It is not shared with the +consumer because bytes in `commit..write` are not readable yet. + +The readable interval is: + +```text +consume .. commit +``` + +The producer-owned reserved interval is: + +```text +commit .. write +``` + +Cursor values are monotonic logical byte positions. The physical byte index is: + +```text +physical_index = cursor % capacity +``` + +The ring is empty when `consume == commit`. The ring is full when: + +```text +write - consume == capacity +``` + +At producer startup for a fresh ring, `write == commit == consume`. After a +producer reserves a span for an async read or copy, `write` may be greater than +`commit`. The consumer still cannot read the reserved bytes because `commit` has +not advanced. + +The `state` word records coarse lifecycle state such as active, closing, closed, +or faulted. The `wake` word records coalescing bits such as readable-pending and +writable-pending. Wake bits are hints for scheduling; `commit` and `consume` are +the ownership authority. + +### 5.3 Native Helper Surface + +Both Rust hot-path code and the Python worker access process-crossing rings +through the same native implementation. Python does not implement shared atomics, +wrap arithmetic, or cursor publication directly. + +The helper exposes operations equivalent to: + +```text +ring_readable_span(handle) -> ptr, len +ring_advance_consume(handle, len) +ring_writable_span(handle) -> ptr, len +ring_advance_commit(handle, len) +ring_state(handle) -> state +``` + +Returned pointers are process-local addresses derived from the caller's mapped +arena base plus arena offsets. The shared header never stores process-local +pointers. + +### 5.4 Producer Rule + +The producer may write only into free space: + +```text +free = capacity - (write - consume) +``` + +Before writing, the producer acquires `consume`. It computes free space against +its local `write` cursor, reserves a contiguous physical span by advancing +`write`, and then writes into that reserved span. If the ring wraps, the producer +reserves at most to the end of the physical buffer, commits that prefix after it +is written, and then reserves the wrapped span. + +After bytes are written, the producer stores the new `commit` with release +ordering and sends or coalesces a readable wake hint. + +The producer may advance `commit` after any successful network read, disk read, +or GPU copy. It does not wait for a complete tensor, complete object, complete +range, or complete frame before committing newly valid bytes. + +### 5.5 Consumer Rule + +The consumer acquires `commit` and may read any byte in: + +```text +consume .. commit +``` + +The consumer greedily drains all bytes it can make progress on. For the worker, +"make progress" means: + +- parse complete control headers when enough bytes are available +- copy available payload bytes into the correct device allocation +- stop only when the ring is empty, the next header is incomplete, the target + device allocation cannot currently accept more bytes, or the ring is faulted + +After the consumer no longer needs a prefix of bytes, it stores the new `consume` +with release ordering and sends or coalesces a writable wake hint if the release +may unblock the producer. + +If the consumer uses asynchronous DMA from host memory, it cannot release bytes +until the DMA no longer reads those bytes. The MVP uses synchronous copy or +event-tracked asynchronous copy that advances `consume` only after the copy event +completes. + +### 5.6 Wakeups + +Wakeups are edge-trigger hints, not state. + +The producer sends `RingReadable { ring_id }` after advancing `commit` when the +consumer may be asleep. The consumer must still load `commit` from the ring +header, because several commits may have coalesced into one wakeup. + +The consumer sends `RingWritable { ring_id }` after advancing `consume` when the +producer may be blocked on free space. The producer must still load `consume` +from the ring header, because several releases may have coalesced into one +wakeup. + +Wake coalescing must preserve liveness. Dropping duplicate wake hints is allowed +only while a pending bit, ready-set entry, or equivalent durable scheduler state +still makes the ring discoverable. Losing the only transition from empty to +readable, or from full to writable, is a liveness bug even though it does not +corrupt memory. The cursors remain the source of truth for ownership. + +### 5.7 Safety + +The consumer cannot read unwritten bytes because the producer publishes +`commit` only after the bytes have been written, and the consumer reads `commit` +with acquire ordering before reading the bytes. + +The producer cannot overwrite unread bytes because it computes free space from +the consumer-owned `consume` cursor and the producer-local `write` cursor. Bytes +at or after `consume` remain unavailable for reuse until the consumer advances +`consume`; bytes in `commit..write` are also unavailable because the producer has +reserved them but not yet published them. + +Two consumers cannot both consume the same byte because a ring has one consumer +and only that consumer writes `consume`. Any design needing two consumers must +split the stream into two rings or insert an explicit fan-out component. + +Wraparound cannot cause stale bytes to be mistaken for new bytes because +ownership is determined by monotonic logical cursors, not by physical indices. +The same physical index can be reused only after `consume` has advanced past the +previous logical byte range that occupied it. + +Cursor overflow cannot occur during a node lifetime if the implementation treats +the `u64` logical cursor space as a runtime ceiling and tears the ring down before +approaching wrap. A ring carrying 1 TB/s would take centuries to exhaust `u64` +byte positions, so this is an operationally unreachable limit for the intended +workloads. + +--- + +## 6. Object Stream Protocol + +The ring carries a byte stream. The stream contains small object headers and raw +payload bytes. Headers define how the worker maps subsequent payload bytes into +VRAM. Payload bytes are not actor messages. + +### 6.1 Edge Stream + +Each edge uses one persistent QUIC uni-stream from producer node to consumer +node. + +Wire shape: + +```text +[edge_id preamble] +[object record] +[object record] +... +``` + +The receiver's edge-demux reader consumes the fixed-width `edge_id` preamble and +hands the stream to the driver rendezvous. After that preamble, the recv-pump is +byte-blind. It copies stream bytes into the ingress ring and advances `commit`. +The worker parses object records from the ring. + +### 6.2 Object Record + +An object record is: + +```text +ObjectHeader +payload bytes, exactly header.extent bytes +``` + +The header is fixed-size in the MVP so the worker can parse it without heap +allocation: + +```rust +struct ObjectHeader { + magic: u32, + version: u16, + header_len: u16, + object_id: u64, + sequence: u64, + extent: u64, + flags: u32, + reserved: u32, +} +``` + +The edge's `ObjectSpec` supplies dtype, shape family, max extent, layout, and +alignment. The header supplies runtime facts: which object this is, its sequence +on the edge, and its actual extent. + +`extent` may be larger than the ingress ring capacity. That is normal. The ring +is a transfer window, not the object storage location. + +### 6.3 Parsing Partial Records + +Headers and payloads may arrive partially. The worker parser therefore has two +states per ingress ring: + +```text +NeedHeader +NeedPayload{object_id, remaining, device_offset} +``` + +In `NeedHeader`, the worker waits until at least `header_len` committed bytes are +available. It may copy those header bytes into a small parser scratch buffer and +release them. Header bytes are control, not payload. + +In `NeedPayload`, the worker copies any committed payload prefix into the target +device allocation. It does not wait for the full payload. If 60 percent of a +large object has arrived and the worker is free, it can copy that 60 percent to +VRAM, release the ring space, and let the recv-pump continue reading the +remaining 40 percent. + +An object is complete when the worker has copied exactly `extent` payload bytes +for that object and any required copy-completion event has fired. + +### 6.4 Validation + +The worker rejects a record before allocating or exposing a device object if: + +- `magic` or `version` is not supported +- `extent > ObjectSpec.max_extent` +- `extent` violates the edge's alignment/layout rules +- the sequence violates the edge ordering policy +- the ring closes before `extent` bytes arrive + +These checks prevent malformed control bytes from becoming a visible tensor. The +payload content itself is trusted. The worker does not inspect tensor values. + +--- + +## 7. Worker Design + +The worker is the consumer of ingress rings and the producer of egress rings. It +is responsible for translating raw stream bytes into GPU-resident objects. + +### 7.1 Worker Control Surface + +The control pipe carries lifecycle messages: + +```rust +InstallRing { + ring_id: RingId, + direction: RingDirection, + layout: RingLayout, + object_spec: ObjectSpec, +} + +UninstallRing { + ring_id: RingId, +} + +AbortObject { + ring_id: RingId, + object_id: u64, + reason: AbortReason, +} + +RingReadable { + ring_id: RingId, +} + +RingWritable { + ring_id: RingId, +} + +Shutdown +``` + +Worker outbound control events and wake hints: + +```rust +RingInstalled { + ring_id: RingId, +} + +ObjectLoaded { + ring_id: RingId, + edge_id: EdgeId, + object_id: u64, + device_handle: DeviceObjectHandle, +} + +ObjectProduced { + ring_id: RingId, + edge_id: EdgeId, + object_id: u64, +} + +ObjectFailed { + ring_id: RingId, + edge_id: EdgeId, + object_id: u64, + reason: ObjectFailure, +} + +RingFault { + ring_id: RingId, + reason: RingFaultReason, +} + +RingQuiesced { + ring_id: RingId, +} + +RingReadable { + ring_id: RingId, +} + +RingWritable { + ring_id: RingId, +} +``` + +There is no `ReadyRange`, `ConsumedRange`, `LandedRange`, or `FreeBytes` control +message. Wake hints name a ring, not a byte range. Per-range messages would put +the actor/control path back into the hot loop. + +### 7.2 Ingress Worker FSM + +Each ingress ring has an independent worker-side FSM: + +```text +Uninstalled + on InstallRing -> NeedHeader + +NeedHeader + on committed bytes < header_len -> wait + on complete header -> validate and allocate device object -> NeedPayload + on invalid header -> Faulted + +NeedPayload + on committed payload bytes -> copy greedy prefix to device + on copied bytes == extent -> ObjectComplete + on stream/ring abort -> Faulted + +ObjectComplete + wait for outstanding copy completion + emit ObjectLoaded + -> NeedHeader + +Faulted + emit RingFault or ObjectFailed + stop consuming until control resolves or uninstalls the ring +``` + +The worker greedily drains. Wake delivery maintains a ready set, typically backed +by an in-process `ArrayQueue` plus per-ring pending bits, so the worker +drains rings that were reported readable instead of scanning every installed +ring on each wake. It continues until no ready ring can make progress. + +### 7.3 Device Assembly + +On a valid `ObjectHeader`, the worker creates an assembly record: + +```rust +struct ObjectAssembly { + object_id: u64, + sequence: u64, + extent: u64, + object_spec: ObjectSpec, + device_allocation: DeviceAllocation, + bytes_copied: u64, + copies_in_flight: CopyTracker, +} +``` + +`ObjectSpec` tells the worker how to interpret the raw bytes: + +- dtype width and dtype family +- shape rule for turning `extent` into rows/tokens/elements +- memory layout expected by the compute role +- alignment requirements +- maximum extent + +For activations, the shape family is known at provisioning and `extent` selects +the runtime row/token count. For weights, the extent and shape may be fixed by +the role or model shard. The worker does not deserialize through Python objects; +it creates or reserves a device buffer whose byte layout matches the role's +expected tensor layout. + +The required primitive is a range copy into a device allocation: + +```text +arena[ring_span] -> device_allocation[object_offset .. object_offset + len] +``` + +The worker may implement this through a lower-level tinygrad device buffer API, +a native CUDA helper, or another backend-specific range-copy primitive. A +high-level API that only supports "copy this entire host buffer into this entire +tensor" is insufficient for ingress streaming, because the object may be larger +than the host ring and the worker must copy partial committed prefixes. + +### 7.4 Host Memory and DMA Safety + +If the worker uses synchronous host-to-device copies, it may advance `consume` +immediately after the copy call returns. + +If it uses asynchronous copies, it may advance `consume` for a byte range only +after the copy no longer depends on that host memory. With pinned host memory, +that means recording the CUDA/event backend completion and releasing the range +after the event fires. Without this rule, the recv-pump could overwrite a ring +span still being read by DMA. The overwrite cannot occur when `consume` is held +back until copy completion, because producer free space is computed from +`consume`. + +### 7.5 Compute Visibility + +The role/compute layer receives an object only after `ObjectLoaded`. + +`ObjectLoaded` is emitted after: + +1. a valid object header was parsed +2. exactly `extent` payload bytes were copied into the device allocation +3. all copy events for those bytes completed +4. the resulting device allocation was associated with the role's expected dtype, + shape, and layout + +Because no compute-visible handle exists before `ObjectLoaded`, compute cannot +observe a partially loaded object. + +--- + +## 8. Driver and Transport + +The driver is the node's swactor-to-iroh boundary. It owns the endpoint, +connection cache, edge demux, and pump tasks. + +### 8.1 Connection Model + +The node uses one iroh endpoint. Edge streams use a dedicated ALPN, for example: + +```text +swactor/edge/1 +``` + +Connections are cached per `(peer_node_id, ALPN)`. A QUIC connection is the +transport object that performs a real handshake. A uni-stream is opened +unilaterally by the sender and is cheap relative to the connection. + +All edges between the same node pair and ALPN reuse the same connection. Each +edge has one persistent uni-stream within that connection. + +### 8.2 Driver Mailbox + +Driver inbound messages: + +```rust +EstablishSend { + edge_id: EdgeId, + rx_node_id: NodeId, + ring_id: RingId, + ring_layout: RingLayout, + tx_addr: ActorAddress, +} + +EstablishRecv { + edge_id: EdgeId, + ring_id: RingId, + ring_layout: RingLayout, + rx_addr: ActorAddress, +} + +StreamArrived { + edge_id: EdgeId, + stream: RecvStream, +} + +StopEdge { + edge_id: EdgeId, +} +``` + +Driver outbound actor events: + +```rust +DriverEdgeReady { + edge_id: EdgeId, +} + +StreamClosed { + edge_id: EdgeId, +} + +StreamFault { + edge_id: EdgeId, + reason: StreamFaultReason, +} + +PumpStopped { + edge_id: EdgeId, + ring_id: RingId, +} +``` + +There are no per-object, per-range, or per-buffer-fragment driver mailbox messages in the +hot path. + +### 8.3 Receive Demux Rendezvous + +A recv-pump needs two resources: + +1. local receive establishment state, including the ingress ring +2. the arriving QUIC stream + +They can arrive in either order. The driver stores both halves: + +```text +recv_specs: HashMap +pending_streams: HashMap +``` + +On `EstablishRecv`, if a pending stream exists, the driver spawns the recv-pump. +Otherwise it stores the spec. + +On `StreamArrived`, if a recv spec exists, the driver spawns the recv-pump. +Otherwise it stores the stream. + +The orchestrator therefore does not need an inter-end readiness handshake. If a +stream arrives before the receiver is locally established, it waits in +`pending_streams`. Because no recv-pump reads from it yet, QUIC flow control +eventually stalls the sender instead of dropping bytes. + +### 8.4 Recv-Pump FSM + +```text +WaitingForSpecAndStream + -> Streaming when ring and stream are both present + +Streaming + read free ring span + read QUIC bytes into that span + advance commit + send/coalesce RingReadable{ring_id} to worker + repeat + +Backpressured + entered when no ring free space exists + wait for RingWritable{ring_id} + return to Streaming + +Closed + entered on stream EOF or edge teardown + +Faulted + entered on read error, protocol edge failure, or ring fault +``` + +The recv-pump is byte-blind after stream demux. It does not parse `ObjectHeader` +and it does not know where object boundaries are. Its only correctness +responsibility is to copy bytes into free ring space and publish `commit` after +those bytes are valid. + +Because the recv-pump is byte-blind, it does not classify EOF as object-aligned +or mid-object. It reports stream closure. The worker parser classifies the close +against its per-ring parser state: EOF with no partial record is a clean stream +close; EOF while a header or payload is incomplete is an object failure. + +If the worker is slow, `consume` stops advancing. The recv-pump computes no free +space, stops reading QUIC, and waits. Since the recv-pump stops reading the +stream, QUIC's stream flow control stalls the remote sender. No actor credit +protocol is needed. + +### 8.5 Send-Pump FSM + +```text +WaitingForConnection + ensure or await cached edge-ALPN connection + +WaitingForBytes + wait for RingReadable{ring_id} + +OpenStream + open one uni-stream + write edge_id preamble + -> Streaming + +Streaming + acquire commit + write committed egress bytes to QUIC + advance consume after bytes are accepted by write_all + send/coalesce RingWritable{ring_id} + repeat + +Backpressured + write_all is pending because network/QUIC flow control is slow + keep ownership of unread ring bytes until write completes + +Closed/Faulted + emit coarse driver event +``` + +If the network is slow, the send-pump stops advancing `consume`. The worker then +runs out of free egress ring space and stalls before producing more outbound +bytes. This is egress backpressure. + +--- + +## 9. Edge Establishment + +Establishment is local actor setup plus transport rendezvous. The two remote edge +ends do not exchange actor messages with each other. + +### 9.1 Orchestrator + +The orchestrator owns graph placement. For each edge, it sends local provision +messages to the producer node and consumer node: + +```rust +ProvisionTx { + edge_id: EdgeId, + rx_node_id: NodeId, + object_spec: ObjectSpec, + ring_spec: RingSpec, +} + +ProvisionRx { + edge_id: EdgeId, + object_spec: ObjectSpec, + ring_spec: RingSpec, +} +``` + +The producer needs the consumer's `node_id`, not the consumer's actor address. +The consumer needs the shared `edge_id`, not the producer's actor address. + +### 9.2 EdgeEstablisher Records + +Each node has one EdgeEstablisher actor. The actor itself stays live and able to +receive new messages. It does not enter `WaitingForLease` globally. Instead, it +owns a table of per-edge establishment records: + +```rust +edge_records: HashMap +ring_to_edge: HashMap + +struct EdgeRecord { + edge_id: EdgeId, + direction: RingDirection, + state: EdgeProvisionState, + lease_request_id: Option, + ring_id: Option, + local_edge_actor: ActorAddress, +} +``` + +The record FSM is: + +```text +New + on ProvisionTx/ProvisionRx + spawn local Tx/Rx edge actor + create LeaseRequestId + send LeaseRing{request_id, ...} to ArenaManager + -> WaitingForLease + +WaitingForLease + on RingLeased matching request_id + record ring_id + send InstallRing to WorkerCtl + -> WaitingForWorkerRing + on RingLeaseRejected matching request_id + notify local edge actor failure + -> Failed + on StopEdge + send CancelLease{request_id} + notify local edge actor stopped + -> Stopped + +WaitingForWorkerRing + on RingInstalled + send EstablishSend/EstablishRecv to Driver + -> WaitingForDriver + on RingFault or StopEdge + -> Stopping + +WaitingForDriver + on DriverEdgeReady + notify local Tx/Rx edge actor Ready + -> Ready + on StreamFault/RingFault/StopEdge + -> Stopping + +Ready + hot path runs without this actor + coarse ObjectLoaded/ObjectProduced/StreamFault events may pass through + +Stopping + stop pump if one exists + uninstall worker ring if installed + wait for quiescence proofs + release ring if leased + -> Stopped +``` + +An edge cannot become `Ready` without a ring lease because the record transition +out of `WaitingForLease` requires a `RingLeased` carrying the same +`LeaseRequestId`. + +An edge cannot spawn a pump for an unmapped worker ring because +`EstablishSend/Recv` is sent only after the same record observes `RingInstalled`. + +If a stale `RingLeased`, `RingInstalled`, `RingFault`, or `PumpStopped` arrives +for a record that has already stopped or for a ring id no longer present in +`ring_to_edge`, the EdgeEstablisher ignores it except for releasing an unused +fresh lease that was granted after cancellation raced with allocation. + +### 9.3 Tx and Rx Edge Actors + +Tx and Rx actors are role-facing lifecycle gates. + +Tx actor state: + +```text +Provisioning -> Ready -> Producing -> Stopping -> Stopped + -> Faulted +``` + +Rx actor state: + +```text +Provisioning -> Ready -> LoadingObject -> ObjectReady -> Stopping -> Stopped + -> Faulted +``` + +They receive: + +```rust +EdgeReady { edge_id } +ObjectLoaded { edge_id, object_id, device_handle } +ObjectProduced { edge_id, object_id } +ObjectFailed { edge_id, object_id, reason } +StreamFault { edge_id, reason } +StopEdge { edge_id } +``` + +They do not receive: + +- bytes +- host pointers to payload +- per-range readiness +- per-range consumed events +- free-space events + +This keeps actor execution deterministic and bounded by coarse workflow events, +while ring cursors handle the high-frequency byte path. + +### 9.4 Race Freedom + +A sender may open its stream before the receiver has completed local +establishment. This does not lose bytes because the receiving driver demux stores +the stream by `edge_id` until `EstablishRecv` supplies a ring. The stream is not +read until the recv-pump exists. Unread QUIC streams apply transport flow control +to the sender. + +A receiver may establish before the sender opens its stream. This does not need a +remote ack because the receive spec waits in the driver demux table. When the +stream arrives, `edge_id` pairs the two halves. + +The two sides do not need each other's actor addresses because data-plane routing +is `(node_id, edge_id)`: the sender dials the consumer node and writes `edge_id` +as the stream preamble; the receiver demuxes by `edge_id`. + +--- + +## 10. Ingress Flow + +Ingress is the path: + +```text +remote worker/disk/GPU -> remote egress ring -> QUIC -> local ingress ring + -> local worker -> local GPU memory -> ObjectLoaded +``` + +### 10.1 End-to-End Sequence + +On the receiving node: + +1. The driver demux reads the stream's `edge_id` preamble. +2. The driver rendezvous pairs the stream with the local ingress ring. +3. The recv-pump waits for free ring space. +4. The recv-pump reads QUIC bytes directly into the ring's free span. +5. The recv-pump advances `commit` after each successful read. +6. The recv-pump sends or coalesces `RingReadable { ring_id }`. +7. The worker wakes and reads `consume..commit`. +8. The worker parses object headers as soon as enough committed bytes exist. +9. The worker allocates the target device object after validating the header. +10. The worker copies every committed payload prefix it can into VRAM. +11. The worker advances `consume` after copies are safe to release. +12. The recv-pump sees free space and continues reading. +13. When all bytes for an object are copied, the worker emits `ObjectLoaded`. + +### 10.2 Partial Ring Fill + +If an object is larger than the ingress ring, the ring may fill with only a +prefix of the object. This is normal. + +Example: + +```text +object extent = 10 GiB +ingress ring = 512 MiB + +recv-pump fills 512 MiB and stalls +worker receives a readable wake and copies committed bytes to VRAM +worker advances consume +recv-pump resumes and reads the next bytes +``` + +The pipeline remains live because the worker consumes the committed prefix, not +completed ranges. The object is not compute-visible until the worker has copied +all 10 GiB and emitted `ObjectLoaded`. + +### 10.3 Why No Per-Range Actor Messages Are Needed + +The worker does not need `Rx` to tell it that bytes landed. `RingReadable` and +the ring cursors already provide that information at the process boundary. The +worker does not need `Rx` to return buffer space. Advancing `consume` releases +bytes to the producer; a coalesced `RingWritable` wake only tells the producer to +reload the cursor. + +Removing per-range actor messages is correct because actor state is not the +authority for ring ownership. The producer and consumer cursors are the +authority. An actor message would be a slower duplicate of state the worker can +read directly. + +--- + +## 11. Egress Flow + +Egress is the path: + +```text +local GPU memory -> local worker -> local egress ring -> QUIC + -> remote ingress ring -> remote worker/GPU +``` + +### 11.1 Sequence + +1. The worker receives or creates a compute result in GPU memory. +2. It creates an `ObjectHeader` according to the edge's `ObjectSpec`. +3. It waits for free egress ring space. +4. It writes header bytes into the egress ring and advances `commit`. +5. It copies payload bytes from the device object into free egress ring spans. +6. It advances `commit` as host bytes become valid. +7. It sends or coalesces `RingReadable { ring_id }`. +8. The send-pump wakes and opens the persistent edge uni-stream on first bytes. +9. The send-pump writes the `edge_id` preamble once. +10. It writes committed egress bytes to QUIC. +11. It advances `consume` after bytes have been accepted by `write_all`. +12. It sends or coalesces `RingWritable { ring_id }`, allowing the worker to + produce more bytes. + +### 11.2 Egress Backpressure + +If QUIC or the remote receiver is slow, `write_all` stops completing. The +send-pump cannot advance `consume`. The egress ring fills. The worker eventually +finds no free space and stops copying more bytes out of GPU memory. + +This cannot overwrite egress bytes because the worker computes free space from +the send-pump-owned `consume` cursor. Until the send-pump advances `consume`, +those bytes remain owned by the send-pump. + +--- + +## 12. Backpressure Model + +There are no credits, no RTS/CTS, and no per-range acknowledgements. + +Backpressure is always absence of writable ring space. + +### 12.1 Worker Slow on Ingress + +```text +worker slow + -> consume does not advance + -> recv-pump sees no free ingress space + -> recv-pump stops reading QUIC + -> QUIC flow control stalls sender + -> sender send-pump stops draining its egress ring + -> sender worker eventually stalls on egress free space +``` + +No data is dropped because each layer stops before overwriting unread bytes. + +### 12.2 Network Slow on Egress + +```text +network slow + -> send-pump write_all remains pending + -> egress consume does not advance + -> worker sees no free egress space + -> worker stops copying more outbound bytes +``` + +No actor credit protocol is needed because the bounded ring and QUIC flow control +already encode the pressure. + +### 12.3 Arena Pressure + +```text +arena temporarily exhausted + -> ArenaManager queues LeaseRing + -> the EdgeEstablisher edge record remains WaitingForLease + -> no worker ring is installed + -> no pump is spawned + -> no hot-path state exists for that edge +``` + +When a ring is released, the ArenaManager retries queued leases. Allocation +pressure is establishment backpressure. + +--- + +## 13. Teardown and Churn + +Teardown must prove quiescence before releasing an arena lease. + +### 13.1 Teardown FSM + +For one edge: + +```text +Ready + on StopEdge or fault + -> StoppingPump + +StoppingPump + driver stops recv/send pump + driver emits PumpStopped + -> StoppingWorkerRing + +StoppingWorkerRing + WorkerCtl sends UninstallRing + worker removes the ring from ready queues + worker waits for in-flight copies or abandons them by process death + worker emits RingQuiesced + -> ReleasingArena + +ReleasingArena + EdgeEstablisher sends ReleaseRing{proof} + ArenaManager returns lease to free-list + -> Stopped +``` + +### 13.2 Why Release Is Safe + +The ArenaManager can safely reuse the range after `ReleaseRing` because the proof +requires both hot-path owners to be gone: + +- the driver pump has stopped, so no Rust task will write/read the ring bytes +- the worker has uninstalled the ring or died, so no worker loop will read/write + the ring bytes +- copy events have completed or the process owning them is gone, so no DMA will + read/write the ring bytes + +Since every live user of the lease is stopped before the release message is sent, +the next lease cannot alias a live user. + +### 13.3 Churn + +Churn replaces edges. It does not mutate a live edge into a different peer. + +A replacement edge gets a new `edge_id`, a new establishment sequence, and either +a fresh ring lease or a reused lease after the old edge proves quiescence. This +keeps old streams, old ring cursors, and old object sequence numbers from merging +with replacement state. + +--- + +## 14. Failure Handling + +### 14.1 Boot Failure + +If `memfd_create`, `ftruncate`, or `mmap` fails, the node does not enter steady +state. No arena offsets have been minted, no rings exist, and no payload state can +be corrupted. + +### 14.2 Oversized Ring Request + +If a `RingSpec` can never fit in the arena ceiling, the ArenaManager emits +`RingLeaseRejected`. The edge fails before worker install or pump spawn. Since no +ring offset is emitted, no hot-path state can reference invalid memory. + +### 14.3 Temporary Arena Exhaustion + +Temporary exhaustion queues the lease. The edge waits before establishment. This +does not lose data because no stream pump has been created for the edge. If a +remote stream arrives early, the receiver's driver holds it in `pending_streams` +and QUIC flow control stalls the sender until local establishment catches up. + +### 14.4 Malformed Object Header + +The worker faults the object if header validation fails. It does not allocate a +compute-visible object. Since compute receives only `ObjectLoaded`, malformed +objects cannot be consumed by the role layer. + +### 14.5 EOF Mid-Object + +If the stream closes before `extent` bytes are copied, the worker emits +`ObjectFailed`. The driver emits `StreamClosed` for transport EOF or +`StreamFault` for a read error; it does not decide whether the close was +object-aligned. The partially allocated device object is discarded. It is not +exposed because completion requires exactly `extent` copied bytes and copy +completion. + +### 14.6 Worker Crash + +If the worker process exits, WorkerCtl marks installed rings faulted and asks the +driver to stop their pumps. The ArenaManager does not release leases until the +worker process is reaped and the driver reports `PumpStopped`. The arena itself +survives worker restart because it is owned by the node process. + +### 14.7 Pump Failure + +If a pump fails, the driver emits `StreamFault` and stops touching the ring. The +worker is told to uninstall the matching ring. The arena lease is released only +after both sides quiesce. + +--- + +## 15. Implementation Requirements + +### 15.1 Shared Atomics + +Ring cursors must be accessed with atomic acquire/release semantics across the +process boundary. The Python worker must use the native ring helper for ring +header access, span calculation, cursor publication, and wake coalescing. Python +may own parser and tinygrad logic, but it does not directly implement +cross-process atomic cursor operations. + +### 15.2 Alignment + +Ring headers must be aligned for atomic cursor operations. Ring data must be +aligned to the largest required host-copy and device-copy alignment for the +backend. Alignment is a performance and DMA requirement; non-overlap correctness +still comes from ArenaManager leases. + +### 15.3 Host Pinning + +The MVP may use pageable host memory and synchronous copies. Host pinning is the +path to efficient asynchronous DMA. If host memory is pinned, it must be pinned +per ring lease and unpinned only after ring quiescence. A pinned range cannot be +returned to the free-list while a backend may still DMA from it. + +### 15.4 Object Layout + +The object byte layout must be fixed by `ObjectSpec`. The worker may parse +headers and validate sizes, but it does not reinterpret or transform payload +values. Payload bytes are copied into device memory in the layout the compute +role expects. + +### 15.5 Persistent Streams + +Each edge uses one persistent stream, not one stream per object. The `edge_id` +preamble is written once. Object records follow back-to-back. This avoids +per-object stream allocation and preserves the ring's natural pipeline. + +--- + +## 16. Invariants and Why They Hold + +**Arena offsets remain valid.** The arena mapping is created once and never moved +or resized. Offsets are relative to that mapping and remain meaningful until node +shutdown. + +**Live rings do not overlap.** The ArenaManager serializes all lease/release +operations through one mailbox and records every live lease. A range is removed +from the free-list before `RingLeased` is emitted and returned only after +`ReleaseRing`. + +**Stale ring events cannot alias replacement rings.** `ring_id` values are unique +for the node lifetime. Churn may reuse arena ranges after quiescence, but it does +not reuse the identifier that control messages and wake hints carry. + +**The worker cannot read bytes before the recv-pump writes them.** The recv-pump +writes bytes first, then advances `commit` with release ordering. The worker +loads `commit` with acquire ordering and never reads beyond it. + +**The recv-pump cannot overwrite bytes still needed by the worker.** Free space is +computed from the worker-owned `consume` cursor and the producer-local `write` +cursor. The worker advances `consume` only after it has parsed/copied the bytes +and, for DMA, after the copy no longer depends on that host memory. Until then, +the producer's free-space calculation cannot include that physical range. + +**Wakeups do not own data.** Wakeups carry no ownership information. The ring +cursors are durable shared state. A consumer that wakes late still sees all bytes +in `consume..commit`. Liveness requires the wake implementation to preserve a +pending ring in a ready set, pending bit, or equivalent durable scheduler state +until the other side reloads the relevant cursor. + +**Partial objects cannot reach compute.** The only compute-visible event is +`ObjectLoaded`, and the worker emits it only after the full declared extent has +been copied into device memory and copy completion is known. + +**A stream can arrive before local receive establishment without dropping bytes.** +The driver demux stores the stream by `edge_id`. It does not read the stream +payload until a recv-pump exists. QUIC flow control stalls the sender if buffers +fill while the stream is pending. + +**Actor scheduling cannot corrupt payload state.** Actors do not own payload +bytes or per-byte cursors. The hot-path state is in ring atomics owned by exactly +one producer and one consumer. + +**A released arena range has no live users.** `ReleaseRing` is sent only after the +driver pump has stopped, the worker ring has quiesced, and copy lifetimes have +ended. The ArenaManager reuses ranges only after that proof. + +---