swactor/RING_BACKPRESSURE_SPEC.md

Ring-Backpressured Data Movement - Canonical Specification

Status: design specification. This document supersedes the previous root workflow drafts for edge establishment, blob streaming, driver streams, stream transport, and arena management:

• EDGE_ESTABLISHMENT.md
• BLOB_STREAMING.md
• STREAM_TRANSPORT.md
• DRIVER_STREAMS.md
• ARENA_MANAGEMENT.md

DESIGN_DIRECTIVES.md remains steering context. This file is the buildout specification.

Scope. Node-local host memory, process-crossing rings, worker loading, edge establishment, persistent QUIC streams, actor/control messages, safety contracts, backpressure, and teardown. This covers activations, weights, and other large objects moving between disk, network, host memory, and GPU memory.

Out of scope. Compute overlap with partial tensors, placement policy, churn placement, trustless verification, object-store semantics, and high-level model graph scheduling. This spec permits transfer and VRAM upload to overlap with network streaming. It deliberately does not permit matmuls or other compute to observe an object until the whole logical object is loaded.

---

1. Shape

A node is an async backpressure machine built from one stable host-memory arena and many bounded byte rings.

The arena is address space. It is one large sparse memfd, created by the Rust node process, mapped once by the Rust node process, and mapped once by the GPU worker process. It does not define flow control and it does not define object ownership.

Rings define flow control. Every payload-moving boundary is represented as a bounded single-producer/single-consumer byte ring backed by a lease inside the arena:

• QUIC ingress stream -> host ingress ring -> worker -> GPU memory
• GPU memory -> worker -> host egress ring -> QUIC egress stream
• disk reader -> host ring -> worker/GPU memory
• future GPU download/upload and network paths follow the same ring contract

Actors establish, supervise, and tear down rings. Actors do not move payload bytes, do not relay per-range readiness, and do not track per-byte ownership. Once a ring is active, the hot path is shared ring metadata plus coalesced wake hints:

producer writes bytes into ring
producer advances commit cursor
producer sends/coalesces a readable wake hint

consumer wakes
consumer drains every committed byte it can use
consumer advances consume cursor
consumer sends/coalesces a writable wake hint if space was released

The wake hint is a signal that ring state may have changed. It carries a ring_id and a reason such as readable or writable, not byte counts or ranges. The producer may send it as soon as bytes are committed; it does not wait for the ring to fill. Duplicate hints may be coalesced through a ready set or pending bit, but a ring that has become readable or writable must remain discoverable until the other side reloads the cursors. The cursors are the state.

The worker can load ingress into VRAM while the stream is still arriving. It does that by greedily consuming the committed prefix of the ingress byte ring and copying those bytes into the correct offsets of a device allocation. The object becomes compute-visible only after all expected bytes have been copied and the worker has observed object completion.

---

2. Vocabulary

node process - The Rust process hosting the swactor runtime, actors, driver, arena manager, iroh endpoint, pump tasks, and worker supervisor.

GPU worker - A separate OS process, normally Python plus tinygrad and a small native ring helper. It maps the arena, receives coalesced ring wake hints, parses object streams, copies payload bytes into or out of GPU memory, and emits coarse events.

arena - One sparse memfd reservation per node. The node process and worker process map the same bytes. Arena offsets are stable for the node lifetime.

arena lease - A non-overlapping byte range in the arena assigned to one ring. The ArenaManager mints leases and releases them after quiescence.

ring - A bounded single-producer/single-consumer byte stream backed by an arena lease. A ring has shared metadata, data bytes, one producer, one consumer, and coalesced wake hints.

edge - A one-way typed conduit from one role to another role. An edge has one producer node, one consumer node, one edge_id, and one persistent QUIC uni-stream once data begins flowing.

edge_id - A run-global edge identifier assigned by the orchestrator. It is the control-plane name of the edge and the fixed-width stream preamble used by the receiver's driver demux.

object - One logical payload on an edge, such as an activation tensor, a weight tensor, a token batch, or a model shard. Objects are sent as object records inside the edge byte stream.

extent - The actual byte length of an object. It is a runtime fact and may be smaller than the edge's maximum object capacity.

object spec - The role-known contract for objects on an edge: maximum extent, dtype family, shape/layout rules, object kind, and any alignment requirements. The worker uses this spec to turn raw bytes into a correctly shaped device allocation.

ring spec - The size and operating parameters of a ring: data capacity, alignment, optional host-pinning requirement, wake coalescing mode, and whether the ring is ingress or egress.

pump - A driver-owned async task. A recv-pump copies QUIC bytes into an ingress ring. A send-pump copies egress ring bytes onto QUIC. Pumps do not parse payload objects after the edge_id preamble.

Tx/Rx edge actors - Small swactor actors representing the local edge end. They hold edge identity and lifecycle state, receive coarse completion/fault events, and integrate with the role layer. They do not receive per-byte or per-range messages.

---

3. Process Topology

Each node has two payload-relevant OS processes:

node process (Rust)                         GPU worker process
-------------------                         ------------------
swactor runtime                             tinygrad / CUDA
ArenaManager actor                          ring parser
EdgeEstablisher actor                       device allocator
Tx/Rx edge actors                           host-to-device / device-to-host copies
Driver mailbox
iroh endpoint
recv/send pumps

          shared memfd arena, mapped by both processes
          actor/control messages for install, wake hints, and coarse events

Host-to-device means copying from arena-backed host memory into GPU memory. Device-to-host means copying from GPU memory into arena-backed host memory.

The node process creates the memfd without CLOEXEC before spawning the worker, or otherwise passes the fd explicitly during worker startup. The worker maps the same reservation once. Neither process remaps the arena during node lifetime.

For a process-crossing ring:

• RingReadable { ring_id } wakes the consumer after the producer commits bytes.
• RingWritable { ring_id } wakes the producer after the consumer releases bytes.

For ingress, the producer is the Rust recv-pump and the consumer is the worker. For egress, the producer is the worker and the consumer is the Rust send-pump.

The worker control pipe carries lifecycle messages, wake hints, and coarse events such as InstallRing, RingInstalled, ObjectLoaded, and RingFault. It does not carry payload bytes, byte counts, or per-range ownership.

---

4. Arena Manager

The ArenaManager is the single per-node authority for arena layout.

4.1 Responsibilities

The ArenaManager owns:

• the arena memfd
• the node process mapping base pointer
• the reservation ceiling
• the arena free-list
• the pending lease queue
• the table of live ring leases

The ArenaManager does not own:

• any worker process
• any QUIC stream
• any pump task
• any object parser
• any payload byte

It never reads or writes payload bytes. It only mints stable offsets.

4.2 Boot

At node boot:

1. The ArenaManager creates the memfd.
2. It truncates it to a generous sparse ceiling.
3. It maps the whole reservation once in the node process.
4. It exposes the base pointer to the driver and ring constructors.
5. It makes the fd available to the worker process at spawn.

The reservation costs virtual address space. Physical pages are backed lazily by the kernel when touched. The mapping is not moved or resized. Any offset minted by the ArenaManager remains meaningful until the node shuts down.

4.3 FSM

Booting
  on ConstructArena{ceiling}
    -> Ready if memfd, truncate, and mmap succeed
    -> Failed if any boot resource cannot be created

Ready
  on LeaseRing{request_id, requester, edge_id, direction, ring_spec}
    -> lease immediately and emit RingLeased if a range fits
    -> enqueue request if the request is satisfiable but no current range fits
    -> emit RingLeaseRejected if the ring_spec can never fit in the ceiling

Ready
  on CancelLease{request_id}
    -> remove queued request if it has not been leased yet

Ready
  on ReleaseRing{ring_id, proof}
    -> return range to free-list
    -> retry queued leases serially

Ready
  on Shutdown
    -> ShuttingDown

ShuttingDown
  no new leases are accepted

4.4 Messages

Inbound:

ConstructArena {
    ceiling: u64,
}

LeaseRing {
    request_id: LeaseRequestId,
    requester: ActorAddress,
    edge_id: EdgeId,
    direction: RingDirection,
    ring_spec: RingSpec,
}

CancelLease {
    request_id: LeaseRequestId,
}

ReleaseRing {
    ring_id: RingId,
    proof: QuiescenceProof,
}

Shutdown

Outbound:

ArenaReady {
    base_ptr: NonNull<u8>,
    ceiling: u64,
}

RingLeased {
    request_id: LeaseRequestId,
    requester: ActorAddress,
    edge_id: EdgeId,
    ring_id: RingId,
    direction: RingDirection,
    arena_offset: u64,
    layout: RingLayout,
}

RingLeaseRejected {
    request_id: LeaseRequestId,
    requester: ActorAddress,
    edge_id: EdgeId,
    reason: LeaseRejectReason,
}

There is no temporary allocation-failure message. Temporary pressure is encoded by absence of RingLeased: the request waits in the lease queue. Permanent impossibility is explicit because no future release can make an oversized ring fit. A queued request can be cancelled by request_id if the edge establishment record stops before the lease is granted.

4.5 Correctness

Two live leases cannot overlap because all lease and release operations pass through one ArenaManager mailbox. A handler mutates the free-list to completion before the next handler runs. The allocator either removes one complete range from the free-list and records it in the live table, or it leaves the free-list unchanged and queues/rejects the request. There is no state in which a partial lease is visible downstream.

A pump or worker cannot observe an unleased range because RingLeased is the only message that contains a usable ring offset. Edge establishment does not install a worker ring or spawn a pump until that message exists.

A range cannot be reused under a live pump or worker because ReleaseRing is a proof, not a request. The EdgeEstablisher emits it only after the driver has stopped the pump, the worker has uninstalled the ring, and in-flight DMA for the ring has completed or been abandoned with the worker process dead. The ArenaManager does not infer quiescence; it relies on the upstream teardown FSM to earn the proof.

Queued lease requests cannot corrupt establishment because the requester receives nothing while queued. No ring offset exists, so no driver or worker hot-path state can be created for that ring. If a stop races with a grant, the EdgeEstablisher accepts RingLeased only when the matching edge record is still waiting on the same request_id; otherwise it releases the unused lease without installing a worker ring or spawning a pump.

---

5. Ring Contract

A ring is a bounded SPSC byte stream in shared memory. It is the universal payload handoff primitive.

The process-crossing ring is a fixed shared-memory ABI, not a Rust collection placed inside the arena. In-process queues such as crossbeam_queue::ArrayQueue may be used for local actor channels, ready sets, or wake scheduling, but the arena ABI stores only offsets, cursors, state bits, and payload bytes. This keeps the mapped bytes valid even when the node process and worker process map the same memfd at different virtual addresses.

5.1 Single Producer, Single Consumer

Each ring has exactly one producer and one consumer.

Ingress:

producer = recv-pump
consumer = worker

Egress:

producer = worker
consumer = send-pump

Disk or future GPU rings follow the same rule. Fan-in or fan-out is represented by multiple rings or by a higher-level mux/demux component that itself owns one side of a ring. A ring never has multiple hot-path producers or consumers.

5.2 Shared-Arena ABI

The ring header lives in shared memory and is aligned for cross-process atomic operations.

#[repr(C, align(64))]
struct RingHeader {
    magic: u32,
    version: u16,
    header_len: u16,
    ring_id: u64,
    capacity: u64,
    commit: AtomicU64,
    consume: AtomicU64,
    state: AtomicU32,
    wake: AtomicU32,
}

ring_id is unique for the node lifetime. Arena ranges may be reused after quiescence, but ring identifiers are not reused. Stale control or wake events therefore cannot alias a later ring that happens to occupy the same arena range.

commit is the first byte after the committed readable prefix. Bytes with logical positions < commit are valid for the consumer to read.

consume is the first byte not yet released by the consumer. Bytes with logical positions < consume are free for the producer to reuse.

The producer also keeps a local write cursor. write is the first byte after the producer's reserved or in-progress write prefix. It is not shared with the consumer because bytes in commit..write are not readable yet.

The readable interval is:

consume .. commit

The producer-owned reserved interval is:

commit .. write

Cursor values are monotonic logical byte positions. The physical byte index is:

physical_index = cursor % capacity

The ring is empty when consume == commit. The ring is full when:

write - consume == capacity

At producer startup for a fresh ring, write == commit == consume. After a producer reserves a span for an async read or copy, write may be greater than commit. The consumer still cannot read the reserved bytes because commit has not advanced.

The state word records coarse lifecycle state such as active, closing, closed, or faulted. The wake word records coalescing bits such as readable-pending and writable-pending. Wake bits are hints for scheduling; commit and consume are the ownership authority.

5.3 Native Helper Surface

Both Rust hot-path code and the Python worker access process-crossing rings through the same native implementation. Python does not implement shared atomics, wrap arithmetic, or cursor publication directly.

The helper exposes operations equivalent to:

ring_readable_span(handle) -> ptr, len
ring_advance_consume(handle, len)
ring_writable_span(handle) -> ptr, len
ring_advance_commit(handle, len)
ring_state(handle) -> state

Returned pointers are process-local addresses derived from the caller's mapped arena base plus arena offsets. The shared header never stores process-local pointers.

5.4 Producer Rule

The producer may write only into free space:

free = capacity - (write - consume)

Before writing, the producer acquires consume. It computes free space against its local write cursor, reserves a contiguous physical span by advancing write, and then writes into that reserved span. If the ring wraps, the producer reserves at most to the end of the physical buffer, commits that prefix after it is written, and then reserves the wrapped span.

After bytes are written, the producer stores the new commit with release ordering and sends or coalesces a readable wake hint.

The producer may advance commit after any successful network read, disk read, or GPU copy. It does not wait for a complete tensor, complete object, complete range, or complete frame before committing newly valid bytes.

5.5 Consumer Rule

The consumer acquires commit and may read any byte in:

consume .. commit

The consumer greedily drains all bytes it can make progress on. For the worker, "make progress" means:

• parse complete control headers when enough bytes are available
• copy available payload bytes into the correct device allocation
• stop only when the ring is empty, the next header is incomplete, the target device allocation cannot currently accept more bytes, or the ring is faulted

After the consumer no longer needs a prefix of bytes, it stores the new consume with release ordering and sends or coalesces a writable wake hint if the release may unblock the producer.

If the consumer uses asynchronous DMA from host memory, it cannot release bytes until the DMA no longer reads those bytes. The MVP uses synchronous copy or event-tracked asynchronous copy that advances consume only after the copy event completes.

5.6 Wakeups

Wakeups are edge-trigger hints, not state.

The producer sends RingReadable { ring_id } after advancing commit when the consumer may be asleep. The consumer must still load commit from the ring header, because several commits may have coalesced into one wakeup.

The consumer sends RingWritable { ring_id } after advancing consume when the producer may be blocked on free space. The producer must still load consume from the ring header, because several releases may have coalesced into one wakeup.

Wake coalescing must preserve liveness. Dropping duplicate wake hints is allowed only while a pending bit, ready-set entry, or equivalent durable scheduler state still makes the ring discoverable. Losing the only transition from empty to readable, or from full to writable, is a liveness bug even though it does not corrupt memory. The cursors remain the source of truth for ownership.

5.7 Safety

The consumer cannot read unwritten bytes because the producer publishes commit only after the bytes have been written, and the consumer reads commit with acquire ordering before reading the bytes.

The producer cannot overwrite unread bytes because it computes free space from the consumer-owned consume cursor and the producer-local write cursor. Bytes at or after consume remain unavailable for reuse until the consumer advances consume; bytes in commit..write are also unavailable because the producer has reserved them but not yet published them.

Two consumers cannot both consume the same byte because a ring has one consumer and only that consumer writes consume. Any design needing two consumers must split the stream into two rings or insert an explicit fan-out component.

Wraparound cannot cause stale bytes to be mistaken for new bytes because ownership is determined by monotonic logical cursors, not by physical indices. The same physical index can be reused only after consume has advanced past the previous logical byte range that occupied it.

Cursor overflow cannot occur during a node lifetime if the implementation treats the u64 logical cursor space as a runtime ceiling and tears the ring down before approaching wrap. A ring carrying 1 TB/s would take centuries to exhaust u64 byte positions, so this is an operationally unreachable limit for the intended workloads.

---

6. Object Stream Protocol

The ring carries a byte stream. The stream contains small object headers and raw payload bytes. Headers define how the worker maps subsequent payload bytes into VRAM. Payload bytes are not actor messages.

6.1 Edge Stream

Each edge uses one persistent QUIC uni-stream from producer node to consumer node.

Wire shape:

[edge_id preamble]
[object record]
[object record]
...

The receiver's edge-demux reader consumes the fixed-width edge_id preamble and hands the stream to the driver rendezvous. After that preamble, the recv-pump is byte-blind. It copies stream bytes into the ingress ring and advances commit. The worker parses object records from the ring.

6.2 Object Record

An object record is:

ObjectHeader
payload bytes, exactly header.extent bytes

The header is fixed-size in the MVP so the worker can parse it without heap allocation:

struct ObjectHeader {
    magic: u32,
    version: u16,
    header_len: u16,
    object_id: u64,
    sequence: u64,
    extent: u64,
    flags: u32,
    reserved: u32,
}

The edge's ObjectSpec supplies dtype, shape family, max extent, layout, and alignment. The header supplies runtime facts: which object this is, its sequence on the edge, and its actual extent.

extent may be larger than the ingress ring capacity. That is normal. The ring is a transfer window, not the object storage location.

6.3 Parsing Partial Records

Headers and payloads may arrive partially. The worker parser therefore has two states per ingress ring:

NeedHeader
NeedPayload{object_id, remaining, device_offset}

In NeedHeader, the worker waits until at least header_len committed bytes are available. It may copy those header bytes into a small parser scratch buffer and release them. Header bytes are control, not payload.

In NeedPayload, the worker copies any committed payload prefix into the target device allocation. It does not wait for the full payload. If 60 percent of a large object has arrived and the worker is free, it can copy that 60 percent to VRAM, release the ring space, and let the recv-pump continue reading the remaining 40 percent.

An object is complete when the worker has copied exactly extent payload bytes for that object and any required copy-completion event has fired.

6.4 Validation

The worker rejects a record before allocating or exposing a device object if:

• magic or version is not supported
• extent > ObjectSpec.max_extent
• extent violates the edge's alignment/layout rules
• the sequence violates the edge ordering policy
• the ring closes before extent bytes arrive

These checks prevent malformed control bytes from becoming a visible tensor. The payload content itself is trusted. The worker does not inspect tensor values.

---

7. Worker Design

The worker is the consumer of ingress rings and the producer of egress rings. It is responsible for translating raw stream bytes into GPU-resident objects.

7.1 Worker Control Surface

The control pipe carries lifecycle messages:

InstallRing {
    ring_id: RingId,
    direction: RingDirection,
    layout: RingLayout,
    object_spec: ObjectSpec,
}

UninstallRing {
    ring_id: RingId,
}

AbortObject {
    ring_id: RingId,
    object_id: u64,
    reason: AbortReason,
}

RingReadable {
    ring_id: RingId,
}

RingWritable {
    ring_id: RingId,
}

Shutdown

Worker outbound control events and wake hints:

RingInstalled {
    ring_id: RingId,
}

ObjectLoaded {
    ring_id: RingId,
    edge_id: EdgeId,
    object_id: u64,
    device_handle: DeviceObjectHandle,
}

ObjectProduced {
    ring_id: RingId,
    edge_id: EdgeId,
    object_id: u64,
}

ObjectFailed {
    ring_id: RingId,
    edge_id: EdgeId,
    object_id: u64,
    reason: ObjectFailure,
}

RingFault {
    ring_id: RingId,
    reason: RingFaultReason,
}

RingQuiesced {
    ring_id: RingId,
}

RingReadable {
    ring_id: RingId,
}

RingWritable {
    ring_id: RingId,
}

There is no ReadyRange, ConsumedRange, LandedRange, or FreeBytes control message. Wake hints name a ring, not a byte range. Per-range messages would put the actor/control path back into the hot loop.

7.2 Ingress Worker FSM

Each ingress ring has an independent worker-side FSM:

Uninstalled
  on InstallRing -> NeedHeader

NeedHeader
  on committed bytes < header_len -> wait
  on complete header -> validate and allocate device object -> NeedPayload
  on invalid header -> Faulted

NeedPayload
  on committed payload bytes -> copy greedy prefix to device
  on copied bytes == extent -> ObjectComplete
  on stream/ring abort -> Faulted

ObjectComplete
  wait for outstanding copy completion
  emit ObjectLoaded
  -> NeedHeader

Faulted
  emit RingFault or ObjectFailed
  stop consuming until control resolves or uninstalls the ring

The worker greedily drains. Wake delivery maintains a ready set, typically backed by an in-process ArrayQueue<RingId> plus per-ring pending bits, so the worker drains rings that were reported readable instead of scanning every installed ring on each wake. It continues until no ready ring can make progress.

7.3 Device Assembly

On a valid ObjectHeader, the worker creates an assembly record:

struct ObjectAssembly {
    object_id: u64,
    sequence: u64,
    extent: u64,
    object_spec: ObjectSpec,
    device_allocation: DeviceAllocation,
    bytes_copied: u64,
    copies_in_flight: CopyTracker,
}

ObjectSpec tells the worker how to interpret the raw bytes:

• dtype width and dtype family
• shape rule for turning extent into rows/tokens/elements
• memory layout expected by the compute role
• alignment requirements
• maximum extent

For activations, the shape family is known at provisioning and extent selects the runtime row/token count. For weights, the extent and shape may be fixed by the role or model shard. The worker does not deserialize through Python objects; it creates or reserves a device buffer whose byte layout matches the role's expected tensor layout.

The required primitive is a range copy into a device allocation:

arena[ring_span] -> device_allocation[object_offset .. object_offset + len]

The worker may implement this through a lower-level tinygrad device buffer API, a native CUDA helper, or another backend-specific range-copy primitive. A high-level API that only supports "copy this entire host buffer into this entire tensor" is insufficient for ingress streaming, because the object may be larger than the host ring and the worker must copy partial committed prefixes.

7.4 Host Memory and DMA Safety

If the worker uses synchronous host-to-device copies, it may advance consume immediately after the copy call returns.

If it uses asynchronous copies, it may advance consume for a byte range only after the copy no longer depends on that host memory. With pinned host memory, that means recording the CUDA/event backend completion and releasing the range after the event fires. Without this rule, the recv-pump could overwrite a ring span still being read by DMA. The overwrite cannot occur when consume is held back until copy completion, because producer free space is computed from consume.

7.5 Compute Visibility

The role/compute layer receives an object only after ObjectLoaded.

ObjectLoaded is emitted after:

1. a valid object header was parsed
2. exactly extent payload bytes were copied into the device allocation
3. all copy events for those bytes completed
4. the resulting device allocation was associated with the role's expected dtype, shape, and layout

Because no compute-visible handle exists before ObjectLoaded, compute cannot observe a partially loaded object.

---

8. Driver and Transport

The driver is the node's swactor-to-iroh boundary. It owns the endpoint, connection cache, edge demux, and pump tasks.

8.1 Connection Model

The node uses one iroh endpoint. Edge streams use a dedicated ALPN, for example:

swactor/edge/1

Connections are cached per (peer_node_id, ALPN). A QUIC connection is the transport object that performs a real handshake. A uni-stream is opened unilaterally by the sender and is cheap relative to the connection.

All edges between the same node pair and ALPN reuse the same connection. Each edge has one persistent uni-stream within that connection.

8.2 Driver Mailbox

Driver inbound messages:

EstablishSend {
    edge_id: EdgeId,
    rx_node_id: NodeId,
    ring_id: RingId,
    ring_layout: RingLayout,
    tx_addr: ActorAddress,
}

EstablishRecv {
    edge_id: EdgeId,
    ring_id: RingId,
    ring_layout: RingLayout,
    rx_addr: ActorAddress,
}

StreamArrived {
    edge_id: EdgeId,
    stream: RecvStream,
}

StopEdge {
    edge_id: EdgeId,
}

Driver outbound actor events:

DriverEdgeReady {
    edge_id: EdgeId,
}

StreamClosed {
    edge_id: EdgeId,
}

StreamFault {
    edge_id: EdgeId,
    reason: StreamFaultReason,
}

PumpStopped {
    edge_id: EdgeId,
    ring_id: RingId,
}

There are no per-object, per-range, or per-buffer-fragment driver mailbox messages in the hot path.

8.3 Receive Demux Rendezvous

A recv-pump needs two resources:

1. local receive establishment state, including the ingress ring
2. the arriving QUIC stream

They can arrive in either order. The driver stores both halves:

recv_specs: HashMap<EdgeId, RecvSpec>
pending_streams: HashMap<EdgeId, RecvStream>

On EstablishRecv, if a pending stream exists, the driver spawns the recv-pump. Otherwise it stores the spec.

On StreamArrived, if a recv spec exists, the driver spawns the recv-pump. Otherwise it stores the stream.

The orchestrator therefore does not need an inter-end readiness handshake. If a stream arrives before the receiver is locally established, it waits in pending_streams. Because no recv-pump reads from it yet, QUIC flow control eventually stalls the sender instead of dropping bytes.

8.4 Recv-Pump FSM

WaitingForSpecAndStream
  -> Streaming when ring and stream are both present

Streaming
  read free ring span
  read QUIC bytes into that span
  advance commit
  send/coalesce RingReadable{ring_id} to worker
  repeat

Backpressured
  entered when no ring free space exists
  wait for RingWritable{ring_id}
  return to Streaming

Closed
  entered on stream EOF or edge teardown

Faulted
  entered on read error, protocol edge failure, or ring fault

The recv-pump is byte-blind after stream demux. It does not parse ObjectHeader and it does not know where object boundaries are. Its only correctness responsibility is to copy bytes into free ring space and publish commit after those bytes are valid.

Because the recv-pump is byte-blind, it does not classify EOF as object-aligned or mid-object. It reports stream closure. The worker parser classifies the close against its per-ring parser state: EOF with no partial record is a clean stream close; EOF while a header or payload is incomplete is an object failure.

If the worker is slow, consume stops advancing. The recv-pump computes no free space, stops reading QUIC, and waits. Since the recv-pump stops reading the stream, QUIC's stream flow control stalls the remote sender. No actor credit protocol is needed.

8.5 Send-Pump FSM

WaitingForConnection
  ensure or await cached edge-ALPN connection

WaitingForBytes
  wait for RingReadable{ring_id}

OpenStream
  open one uni-stream
  write edge_id preamble
  -> Streaming

Streaming
  acquire commit
  write committed egress bytes to QUIC
  advance consume after bytes are accepted by write_all
  send/coalesce RingWritable{ring_id}
  repeat

Backpressured
  write_all is pending because network/QUIC flow control is slow
  keep ownership of unread ring bytes until write completes

Closed/Faulted
  emit coarse driver event

If the network is slow, the send-pump stops advancing consume. The worker then runs out of free egress ring space and stalls before producing more outbound bytes. This is egress backpressure.

---

9. Edge Establishment

Establishment is local actor setup plus transport rendezvous. The two remote edge ends do not exchange actor messages with each other.

9.1 Orchestrator

The orchestrator owns graph placement. For each edge, it sends local provision messages to the producer node and consumer node:

ProvisionTx {
    edge_id: EdgeId,
    rx_node_id: NodeId,
    object_spec: ObjectSpec,
    ring_spec: RingSpec,
}

ProvisionRx {
    edge_id: EdgeId,
    object_spec: ObjectSpec,
    ring_spec: RingSpec,
}

The producer needs the consumer's node_id, not the consumer's actor address. The consumer needs the shared edge_id, not the producer's actor address.

9.2 EdgeEstablisher Records

Each node has one EdgeEstablisher actor. The actor itself stays live and able to receive new messages. It does not enter WaitingForLease globally. Instead, it owns a table of per-edge establishment records:

edge_records: HashMap<EdgeId, EdgeRecord>
ring_to_edge: HashMap<RingId, EdgeId>

struct EdgeRecord {
    edge_id: EdgeId,
    direction: RingDirection,
    state: EdgeProvisionState,
    lease_request_id: Option<LeaseRequestId>,
    ring_id: Option<RingId>,
    local_edge_actor: ActorAddress,
}

The record FSM is:

New
  on ProvisionTx/ProvisionRx
    spawn local Tx/Rx edge actor
    create LeaseRequestId
    send LeaseRing{request_id, ...} to ArenaManager
    -> WaitingForLease

WaitingForLease
  on RingLeased matching request_id
    record ring_id
    send InstallRing to WorkerCtl
    -> WaitingForWorkerRing
  on RingLeaseRejected matching request_id
    notify local edge actor failure
    -> Failed
  on StopEdge
    send CancelLease{request_id}
    notify local edge actor stopped
    -> Stopped

WaitingForWorkerRing
  on RingInstalled
    send EstablishSend/EstablishRecv to Driver
    -> WaitingForDriver
  on RingFault or StopEdge
    -> Stopping

WaitingForDriver
  on DriverEdgeReady
    notify local Tx/Rx edge actor Ready
    -> Ready
  on StreamFault/RingFault/StopEdge
    -> Stopping

Ready
  hot path runs without this actor
  coarse ObjectLoaded/ObjectProduced/StreamFault events may pass through

Stopping
  stop pump if one exists
  uninstall worker ring if installed
  wait for quiescence proofs
  release ring if leased
  -> Stopped

An edge cannot become Ready without a ring lease because the record transition out of WaitingForLease requires a RingLeased carrying the same LeaseRequestId.

An edge cannot spawn a pump for an unmapped worker ring because EstablishSend/Recv is sent only after the same record observes RingInstalled.

If a stale RingLeased, RingInstalled, RingFault, or PumpStopped arrives for a record that has already stopped or for a ring id no longer present in ring_to_edge, the EdgeEstablisher ignores it except for releasing an unused fresh lease that was granted after cancellation raced with allocation.

9.3 Tx and Rx Edge Actors

Tx and Rx actors are role-facing lifecycle gates.

Tx actor state:

Provisioning -> Ready -> Producing -> Stopping -> Stopped
                         -> Faulted

Rx actor state:

Provisioning -> Ready -> LoadingObject -> ObjectReady -> Stopping -> Stopped
                         -> Faulted

They receive:

EdgeReady { edge_id }
ObjectLoaded { edge_id, object_id, device_handle }
ObjectProduced { edge_id, object_id }
ObjectFailed { edge_id, object_id, reason }
StreamFault { edge_id, reason }
StopEdge { edge_id }

They do not receive:

• bytes
• host pointers to payload
• per-range readiness
• per-range consumed events
• free-space events

This keeps actor execution deterministic and bounded by coarse workflow events, while ring cursors handle the high-frequency byte path.

9.4 Race Freedom

A sender may open its stream before the receiver has completed local establishment. This does not lose bytes because the receiving driver demux stores the stream by edge_id until EstablishRecv supplies a ring. The stream is not read until the recv-pump exists. Unread QUIC streams apply transport flow control to the sender.

A receiver may establish before the sender opens its stream. This does not need a remote ack because the receive spec waits in the driver demux table. When the stream arrives, edge_id pairs the two halves.

The two sides do not need each other's actor addresses because data-plane routing is (node_id, edge_id): the sender dials the consumer node and writes edge_id as the stream preamble; the receiver demuxes by edge_id.

---

10. Ingress Flow

Ingress is the path:

remote worker/disk/GPU -> remote egress ring -> QUIC -> local ingress ring
    -> local worker -> local GPU memory -> ObjectLoaded

10.1 End-to-End Sequence

On the receiving node:

1. The driver demux reads the stream's edge_id preamble.
2. The driver rendezvous pairs the stream with the local ingress ring.
3. The recv-pump waits for free ring space.
4. The recv-pump reads QUIC bytes directly into the ring's free span.
5. The recv-pump advances commit after each successful read.
6. The recv-pump sends or coalesces RingReadable { ring_id }.
7. The worker wakes and reads consume..commit.
8. The worker parses object headers as soon as enough committed bytes exist.
9. The worker allocates the target device object after validating the header.
10. The worker copies every committed payload prefix it can into VRAM.
11. The worker advances consume after copies are safe to release.
12. The recv-pump sees free space and continues reading.
13. When all bytes for an object are copied, the worker emits ObjectLoaded.

10.2 Partial Ring Fill

If an object is larger than the ingress ring, the ring may fill with only a prefix of the object. This is normal.

Example:

object extent = 10 GiB
ingress ring = 512 MiB

recv-pump fills 512 MiB and stalls
worker receives a readable wake and copies committed bytes to VRAM
worker advances consume
recv-pump resumes and reads the next bytes

The pipeline remains live because the worker consumes the committed prefix, not completed ranges. The object is not compute-visible until the worker has copied all 10 GiB and emitted ObjectLoaded.

10.3 Why No Per-Range Actor Messages Are Needed

The worker does not need Rx to tell it that bytes landed. RingReadable and the ring cursors already provide that information at the process boundary. The worker does not need Rx to return buffer space. Advancing consume releases bytes to the producer; a coalesced RingWritable wake only tells the producer to reload the cursor.

Removing per-range actor messages is correct because actor state is not the authority for ring ownership. The producer and consumer cursors are the authority. An actor message would be a slower duplicate of state the worker can read directly.

---

11. Egress Flow

Egress is the path:

local GPU memory -> local worker -> local egress ring -> QUIC
    -> remote ingress ring -> remote worker/GPU

11.1 Sequence

1. The worker receives or creates a compute result in GPU memory.
2. It creates an ObjectHeader according to the edge's ObjectSpec.
3. It waits for free egress ring space.
4. It writes header bytes into the egress ring and advances commit.
5. It copies payload bytes from the device object into free egress ring spans.
6. It advances commit as host bytes become valid.
7. It sends or coalesces RingReadable { ring_id }.
8. The send-pump wakes and opens the persistent edge uni-stream on first bytes.
9. The send-pump writes the edge_id preamble once.
10. It writes committed egress bytes to QUIC.
11. It advances consume after bytes have been accepted by write_all.
12. It sends or coalesces RingWritable { ring_id }, allowing the worker to produce more bytes.

11.2 Egress Backpressure

If QUIC or the remote receiver is slow, write_all stops completing. The send-pump cannot advance consume. The egress ring fills. The worker eventually finds no free space and stops copying more bytes out of GPU memory.

This cannot overwrite egress bytes because the worker computes free space from the send-pump-owned consume cursor. Until the send-pump advances consume, those bytes remain owned by the send-pump.

---

12. Backpressure Model

There are no credits, no RTS/CTS, and no per-range acknowledgements.

Backpressure is always absence of writable ring space.

12.1 Worker Slow on Ingress

worker slow
  -> consume does not advance
  -> recv-pump sees no free ingress space
  -> recv-pump stops reading QUIC
  -> QUIC flow control stalls sender
  -> sender send-pump stops draining its egress ring
  -> sender worker eventually stalls on egress free space

No data is dropped because each layer stops before overwriting unread bytes.

12.2 Network Slow on Egress

network slow
  -> send-pump write_all remains pending
  -> egress consume does not advance
  -> worker sees no free egress space
  -> worker stops copying more outbound bytes

No actor credit protocol is needed because the bounded ring and QUIC flow control already encode the pressure.

12.3 Arena Pressure

arena temporarily exhausted
  -> ArenaManager queues LeaseRing
  -> the EdgeEstablisher edge record remains WaitingForLease
  -> no worker ring is installed
  -> no pump is spawned
  -> no hot-path state exists for that edge

When a ring is released, the ArenaManager retries queued leases. Allocation pressure is establishment backpressure.

---

13. Teardown and Churn

Teardown must prove quiescence before releasing an arena lease.

13.1 Teardown FSM

For one edge:

Ready
  on StopEdge or fault
    -> StoppingPump

StoppingPump
  driver stops recv/send pump
  driver emits PumpStopped
  -> StoppingWorkerRing

StoppingWorkerRing
  WorkerCtl sends UninstallRing
  worker removes the ring from ready queues
  worker waits for in-flight copies or abandons them by process death
  worker emits RingQuiesced
  -> ReleasingArena

ReleasingArena
  EdgeEstablisher sends ReleaseRing{proof}
  ArenaManager returns lease to free-list
  -> Stopped

13.2 Why Release Is Safe

The ArenaManager can safely reuse the range after ReleaseRing because the proof requires both hot-path owners to be gone:

• the driver pump has stopped, so no Rust task will write/read the ring bytes
• the worker has uninstalled the ring or died, so no worker loop will read/write the ring bytes
• copy events have completed or the process owning them is gone, so no DMA will read/write the ring bytes

Since every live user of the lease is stopped before the release message is sent, the next lease cannot alias a live user.

13.3 Churn

Churn replaces edges. It does not mutate a live edge into a different peer.

A replacement edge gets a new edge_id, a new establishment sequence, and either a fresh ring lease or a reused lease after the old edge proves quiescence. This keeps old streams, old ring cursors, and old object sequence numbers from merging with replacement state.

---

14. Failure Handling

14.1 Boot Failure

If memfd_create, ftruncate, or mmap fails, the node does not enter steady state. No arena offsets have been minted, no rings exist, and no payload state can be corrupted.

14.2 Oversized Ring Request

If a RingSpec can never fit in the arena ceiling, the ArenaManager emits RingLeaseRejected. The edge fails before worker install or pump spawn. Since no ring offset is emitted, no hot-path state can reference invalid memory.

14.3 Temporary Arena Exhaustion

Temporary exhaustion queues the lease. The edge waits before establishment. This does not lose data because no stream pump has been created for the edge. If a remote stream arrives early, the receiver's driver holds it in pending_streams and QUIC flow control stalls the sender until local establishment catches up.

14.4 Malformed Object Header

The worker faults the object if header validation fails. It does not allocate a compute-visible object. Since compute receives only ObjectLoaded, malformed objects cannot be consumed by the role layer.

14.5 EOF Mid-Object

If the stream closes before extent bytes are copied, the worker emits ObjectFailed. The driver emits StreamClosed for transport EOF or StreamFault for a read error; it does not decide whether the close was object-aligned. The partially allocated device object is discarded. It is not exposed because completion requires exactly extent copied bytes and copy completion.

14.6 Worker Crash

If the worker process exits, WorkerCtl marks installed rings faulted and asks the driver to stop their pumps. The ArenaManager does not release leases until the worker process is reaped and the driver reports PumpStopped. The arena itself survives worker restart because it is owned by the node process.

14.7 Pump Failure

If a pump fails, the driver emits StreamFault and stops touching the ring. The worker is told to uninstall the matching ring. The arena lease is released only after both sides quiesce.

---

15. Implementation Requirements

15.1 Shared Atomics

Ring cursors must be accessed with atomic acquire/release semantics across the process boundary. The Python worker must use the native ring helper for ring header access, span calculation, cursor publication, and wake coalescing. Python may own parser and tinygrad logic, but it does not directly implement cross-process atomic cursor operations.

15.2 Alignment

Ring headers must be aligned for atomic cursor operations. Ring data must be aligned to the largest required host-copy and device-copy alignment for the backend. Alignment is a performance and DMA requirement; non-overlap correctness still comes from ArenaManager leases.

15.3 Host Pinning

The MVP may use pageable host memory and synchronous copies. Host pinning is the path to efficient asynchronous DMA. If host memory is pinned, it must be pinned per ring lease and unpinned only after ring quiescence. A pinned range cannot be returned to the free-list while a backend may still DMA from it.

15.4 Object Layout

The object byte layout must be fixed by ObjectSpec. The worker may parse headers and validate sizes, but it does not reinterpret or transform payload values. Payload bytes are copied into device memory in the layout the compute role expects.

15.5 Persistent Streams

Each edge uses one persistent stream, not one stream per object. The edge_id preamble is written once. Object records follow back-to-back. This avoids per-object stream allocation and preserves the ring's natural pipeline.

---

16. Invariants and Why They Hold

Arena offsets remain valid. The arena mapping is created once and never moved or resized. Offsets are relative to that mapping and remain meaningful until node shutdown.

Live rings do not overlap. The ArenaManager serializes all lease/release operations through one mailbox and records every live lease. A range is removed from the free-list before RingLeased is emitted and returned only after ReleaseRing.

Stale ring events cannot alias replacement rings. ring_id values are unique for the node lifetime. Churn may reuse arena ranges after quiescence, but it does not reuse the identifier that control messages and wake hints carry.

The worker cannot read bytes before the recv-pump writes them. The recv-pump writes bytes first, then advances commit with release ordering. The worker loads commit with acquire ordering and never reads beyond it.

The recv-pump cannot overwrite bytes still needed by the worker. Free space is computed from the worker-owned consume cursor and the producer-local write cursor. The worker advances consume only after it has parsed/copied the bytes and, for DMA, after the copy no longer depends on that host memory. Until then, the producer's free-space calculation cannot include that physical range.

Wakeups do not own data. Wakeups carry no ownership information. The ring cursors are durable shared state. A consumer that wakes late still sees all bytes in consume..commit. Liveness requires the wake implementation to preserve a pending ring in a ready set, pending bit, or equivalent durable scheduler state until the other side reloads the relevant cursor.

Partial objects cannot reach compute. The only compute-visible event is ObjectLoaded, and the worker emits it only after the full declared extent has been copied into device memory and copy completion is known.

A stream can arrive before local receive establishment without dropping bytes. The driver demux stores the stream by edge_id. It does not read the stream payload until a recv-pump exists. QUIC flow control stalls the sender if buffers fill while the stream is pending.

Actor scheduling cannot corrupt payload state. Actors do not own payload bytes or per-byte cursors. The hot-path state is in ring atomics owned by exactly one producer and one consumer.

A released arena range has no live users. ReleaseRing is sent only after the driver pump has stopped, the worker ring has quiesced, and copy lifetimes have ended. The ArenaManager reuses ranges only after that proof.

---