swactor/crates/mvp-system/MVP_SYSTEM_SPEC.md

# MVP System Specification
 ***STALE! FOR HISTORICAL REFERENCE ONLY*** 

**Status:** draft consolidated system specification.

This document is the single MVP reference for the swactor GGUF pipeline system.
It folds the system behavior previously split across the orchestration,
ring/backpressure, and GPU-worker drafts into one end-to-end specification.

The focused root specs may remain as design history while this document is being
stabilized. For the MVP drafting process, this document is the place where the
complete system, its components, guarantees, and systemic behavior are described
together.

---

## 1. Purpose

The MVP is a trusted distributed ML runtime for one linear GGUF inference
pipeline over a rented pool of GPU nodes.

The first deployment shape is:

```text
operator provisions rented GPU nodes
  -> each node runs a barebones Docker image
  -> the image contains swactor runtime + tinygrad/CUDA worker
  -> nodes join membership and reach SWIM convergence
  -> orchestrator assigns pipeline roles and edges
  -> nodes download/load assigned GGUF weights
  -> nodes provision arena-backed data edges
  -> nodes report readiness
  -> orchestrator injects a "hello world" prompt
  -> pipeline returns tokens
  -> run completes or faults and tears down
```

The document is intentionally about the first complete feature, not a broad
graph engine. Later features can generalize from this system after the MVP
behavior is stable.

---

## 2. Scope

In scope:

- trusted node pool boot and readiness
- SWIM membership convergence as the planning gate
- orchestrator authority, planning, staging, faults, and teardown
- one linear GGUF pipeline
- stage/layer assignment and edge id assignment
- stage-local weight download, load, and bind readiness
- arena-backed rings and iroh/QUIC edge transport
- Rust node process to Python/tinygrad worker process control
- GPU worker object loading, step execution, and object production
- prompt injection, prefill, decode, token return
- completed, faulted, and operator-stopped terminal outcomes
- observability events needed for behavioral contracts and tests

Out of scope:

- arbitrary graph IR
- automatic placement optimization
- batching, speculative decoding, continuous serving
- re-placement after churn or node failure
- trustless verification, incentives, Sybil defense, adversarial tensor checks
- production artifact layout and cache eviction policy
- VastAI account automation, bidding, billing, or provider-specific control
  machinery
- high-performance host pinning policy beyond the safety contract

---

## 3. Design Commitments

Nodes are trusted. Payload content is trusted after structural checks. The
system does not defend against malicious tensor values or adversarial peers.

The orchestrator is the run authority. It owns placement, run planning,
provisioning, prompt injection, terminal outcome, and teardown.

swactor owns the control plane. Actors establish, observe, and tear down
components. Actors do not move tensor payload bytes.

Payload bytes move through shared-memory rings and iroh/QUIC streams. The hot
path is ring cursors, wake hints, and byte-pump tasks, not actor mailboxes.

The GPU worker is a supervised process, not a separate swactor runtime. It maps
the shared arena, uses a native ring helper, owns device allocations, and runs
tinygrad role code when explicitly commanded.

Compute is explicit. `ObjectLoaded` means data is ready on device.
`ExecuteStep` is the local compute admission command. Loading an object does not
automatically run tinygrad.

Backpressure is absence of writable ring space plus QUIC flow control. There is
no actor-level credit protocol, RTS/CTS exchange, or per-range acknowledgement.

One persistent iroh uni-stream carries all objects for one edge. The stream
starts with an `edge_id` preamble and then object records.

The MVP permits transfer and device upload/download to overlap with network
streaming. It does not permit compute to observe a logical object until the
whole object has loaded.

---

## 4. Trust And Authority Model

The orchestrator owns:

- intended node pool
- resource inventory used by placement
- run ids
- model and runtime configuration
- tokenizer/vocabulary facts
- stage count and stage order
- stage-to-node assignment
- layer range assignment
- edge id assignment
- object specs and ring specs
- weight source assignment
- global readiness barrier
- prompt tokenization and prompt injection
- output token consumption
- EOS and `max_tokens` stop policy
- run-level fault and teardown decisions

Each node owns:

- stable node identity for the run
- node process lifecycle
- GPU worker process lifecycle
- local arena and arena leases
- local iroh endpoint and byte pumps
- SWIM participant
- local stage controller
- local edge establishment records
- assigned weight download/load work
- local worker/device resources

Nodes do not rewire the graph. A node accepts run provisioning only from the
authorized orchestrator and rejects edge data or provisioning that does not match
its active run plan.

Other stages cannot redirect a node to a new peer, edge id, or layer range.

Resource inventory is known to the orchestrator before placement in the MVP. A
node may report boot health and readiness, but there is no distributed
capability negotiation protocol after boot.

---

## 5. System Topology

Each GPU node runs one Rust node process and one Python/tinygrad GPU worker
process:

```text
GPU node container
------------------
Rust node process
  swactor runtime
  SWIM participant
  iroh endpoint and driver
  ArenaManager
  EdgeEstablisher
  Tx/Rx edge actors
  GpuWorkerCtl
  StageController

Python/tinygrad worker process
  mapped shared arena
  native ring helper
  device bridge
  role code
  device allocations and KV/state
```

The orchestrator may run in its own container or on one of the provisioned GPU
machines. It is a control participant and token endpoint. It does not run model
compute for the MVP.

When the orchestrator participates in token edges over the data plane, it has a
stable `node_id`, an iroh endpoint, and local token edge endpoints like any
other producer or consumer node. Co-locating the orchestrator with a GPU node is
an implementation choice, not a change to edge semantics.

The system has two network-facing planes:

- membership/control observation: SWIM provides node liveness/membership state
  used by the orchestrator as the pool readiness gate
- data movement: iroh/QUIC carries persistent uni-streams for data edges,
  addressed by `(node_id, edge_id)`

Provisioning, readiness, faults, and teardown are swactor messages. Tensor bytes
do not travel in actor messages.

---

## 6. Identifiers

Identifier types are schematic. Concrete Rust APIs may wrap these as newtypes.

```rust
struct RunId(u64);
struct NodeId(u64);
struct RoleId(u64);
struct StageIndex(u32);
struct EdgeId(u64);
struct RingId(u64);
struct ObjectId(u64);
struct Sequence(u64);
struct StepId(u64);
struct WorkerGeneration(u64);
```

`EdgeId` is unique within a run and assigned only by the orchestrator.

`Sequence` names the inference-step index carried on object records. It is not a
token id, token-array position, byte offset, stream packet number, or transport
ordering primitive.

`RingId` is unique for the node lifetime. Arena ranges may be reused after
quiescence, but ring ids are not reused.

`DeviceObjectHandle` is opaque to the Rust node process:

```rust
struct DeviceObjectHandle {
    worker_generation: u64,
    id: u64,
}
```

A device handle is valid only in the worker generation that created it. Worker
restart invalidates all prior handles, roles, rings, and in-flight steps.

---

## 7. System Lifecycle

The complete MVP lifecycle is:

```text
Deployment
  -> NodeBoot
  -> MembershipConvergence
  -> RunPlanning
  -> StageProvisioning
  -> WeightLoading
  -> EdgeProvisioning
  -> GlobalReadiness
  -> PromptInjection
  -> PipelineExecution
  -> CompletionOrFault
  -> Teardown
  -> Done
```

Phase ownership:

- node boot is local to each node
- membership convergence is observed by the orchestrator
- run planning is owned by the orchestrator
- local provisioning is owned by each node
- global readiness is owned by the orchestrator
- execution progress is driven by object arrival at stages
- terminal outcome and teardown are owned by the orchestrator

There is no broadcast start message. The first prompt object written by the
orchestrator after the readiness barrier starts execution.

---

## 8. Node Boot

At container start, each node runs:

```text
ContainerStarting
  -> NodeProcessStarting
  -> ArenaReady
  -> WorkerReady
  -> TransportReady
  -> MembershipJoining
  -> NodeAvailable
```

`NodeAvailable` means:

- Rust node process is alive
- swactor runtime can receive control messages
- node has a stable `node_id` known to the orchestrator
- arena is created and mapped in the node process
- GPU worker has emitted `WorkerReady`
- iroh endpoint is initialized and associated with the node identity
- SWIM participant has joined or is joining the intended pool
- node can accept run provisioning

`NodeAvailable` does not mean weights are present, a role is configured, or any
run edge is established.

If implementation defers worker startup until run provisioning, the same
run-level gate must still hold: a stage cannot report `StageReady` before its
worker is ready and assigned weights are usable.

Boot failure before `ArenaReady` or `WorkerReady` means the node is unavailable.
No run edge should be provisioned to that node.

---

## 9. Membership And Pool Readiness

The orchestrator starts run planning only after the intended candidate pool is
ready.

For the MVP, `PoolReady` means:

- every node in the intended candidate pool is known to the orchestrator
- every candidate node is live in the SWIM membership view
- every candidate node has reported `NodeAvailable`
- every candidate node has data-plane identity/endpoint material the
  orchestrator can provision into edges
- no candidate node is currently suspect or faulted in the orchestrator view
- the pool view has remained stable for the configured convergence window

The convergence window is an implementation parameter.

Nodes do not compute placement and do not need to agree on graph state. SWIM is a
membership/liveness input to the orchestrator, not a distributed graph protocol.

If pool readiness is lost before a `RunPlan` is committed, the orchestrator keeps
waiting or aborts before provisioning according to local policy.

If pool readiness is lost after provisioning begins, the run faults. The MVP does
not re-place an active run after a node disappears.

---

## 10. GGUF Pipeline Workload

The MVP workload is one linear pipeline:

```text
orchestrator --tokens--> stage 0 --activations--> stage 1 --activations-->
... --activations--> stage N-1 --tokens--> orchestrator
```

The orchestrator is a token endpoint and control participant. It does not run GPU
compute.

Each stage owns a contiguous GGUF layer range. A stage receives typed input
objects, executes its assigned layer range, and writes typed output objects.

Stage `0` consumes token objects from the orchestrator and produces activation
objects. Interior stages consume and produce activation objects. The last stage
consumes activation objects and produces token objects for the orchestrator.

The last stage samples or otherwise produces token ids as part of its worker
step if sampling is delegated to the stage. If sampling is not delegated, the
worker output object must contain enough logits/token data for the orchestrator
to apply the configured policy. The `RunPlan` must state which policy is used.

---

## 11. Run Plan

The orchestrator builds exactly one `RunPlan` before provisioning:

```rust
struct RunPlan {
    run_id: RunId,
    model: GgufModelPlan,
    runtime: RuntimePlan,
    stages: Vec<StagePlan>,
    edges: Vec<EdgePlan>,
    max_tokens: u32,
}

struct GgufModelPlan {
    model_id: String,
    gguf_source: GgufSource,
    num_layers: u32,
    hidden_dim: u32,
    dtype_family: DTypeFamily,
    dtype_width_bytes: u32,
    max_seq_len: u32,
    eos_token_id: u32,
    tokenizer: TokenizerSource,
}

struct RuntimePlan {
    prompt: PromptSource,
    sampling: SamplingPolicy,
    token_output_policy: TokenOutputPolicy,
}
```

`gguf_source` may identify a whole GGUF file, a pre-split shard collection, a
cache key, a local path, or an HTTP/object-store artifact. The system contract is
the assigned layer range and readiness after that range is usable by the worker.

Each stage receives one contiguous layer range:

```rust
struct StagePlan {
    run_id: RunId,
    stage_index: u32,
    stage_count: u32,
    node_id: NodeId,
    gguf_source: GgufSource,
    layer_start: u32,
    layer_end_exclusive: u32,
    inbound_edge: EdgeId,
    outbound_edge: EdgeId,
}
```

Each edge has exactly one producer and one consumer:

```rust
struct EdgePlan {
    run_id: RunId,
    edge_id: EdgeId,
    kind: EdgeKind,
    producer: EdgeEndpoint,
    consumer: EdgeEndpoint,
    object_spec: ObjectSpec,
    ring_spec: RingSpec,
}

enum EdgeKind {
    TokenIn,
    Activation,
    TokenOut,
}

enum EdgeEndpoint {
    Orchestrator { node_id: NodeId },
    Stage { node_id: NodeId, stage_index: u32 },
}
```

Layer assignment guarantees:

- stage layer ranges are contiguous
- stage layer ranges do not overlap
- the union of stage layer ranges covers the intended GGUF block range
- every stage has exactly one assigned range

Edge assignment guarantees:

- every edge id is unique within the run
- token-in is produced by the orchestrator and consumed by stage `0`
- token-out is produced by stage `N - 1` and consumed by the orchestrator
- activation edge `i` is produced by stage `i` and consumed by stage `i + 1`
- stages never derive edge ids from names, layer ranges, peer ids, or hashes

---

## 12. Provisioning Messages

The orchestrator sends one stage provision message to each stage node:

```rust
struct ProvisionStage {
    run_id: RunId,
    stage_index: u32,
    stage_count: u32,
    gguf_source: GgufSource,
    layer_start: u32,
    layer_end_exclusive: u32,
    inbound: InboundEdgeProvision,
    outbound: OutboundEdgeProvision,
    model: StageModelFacts,
    runtime: StageRuntimeFacts,
}

struct InboundEdgeProvision {
    edge_id: EdgeId,
    kind: EdgeKind,
    object_spec: ObjectSpec,
    ring_spec: RingSpec,
}

struct OutboundEdgeProvision {
    edge_id: EdgeId,
    kind: EdgeKind,
    consumer_node_id: NodeId,
    object_spec: ObjectSpec,
    ring_spec: RingSpec,
}

struct StageModelFacts {
    model_id: String,
    hidden_dim: u32,
    dtype_family: DTypeFamily,
    dtype_width_bytes: u32,
    max_seq_len: u32,
}

struct StageRuntimeFacts {
    role_id: RoleId,
    input_port: PortId,
    output_port: PortId,
    sampling: Option<SamplingPolicy>,
}
```

The inbound edge is established locally as a receive edge. The outbound edge is
established locally as a send edge to `consumer_node_id`.

For the last stage, `consumer_node_id` is the orchestrator node.

Provisioning fan-out is parallel. There is no Tx-to-Rx actor handshake and no
peer-to-peer endpoint exchange.

---

## 13. Orchestrator FSM

The orchestrator subsystem owns planning, validation, and control. The
run-level FSM described here receives an already-valid `RunPlan`; plan
validation failures are planner errors and are not FSM states.

```text
Planning outside this FSM
  build RunPlan
  validate layer ranges, edge ids, object specs, node ids
  emit PlanAvailable(valid RunPlan)

Provisioning
  after PoolReady and PlanAvailable(valid RunPlan)
  send ProvisionStage to every stage node
  create local token-in producer
  create local token-out consumer
  -> WaitingReady

WaitingReady
  on StageReady for every stage and local token endpoints ready
    -> Running
  on StageFault, endpoint fault, membership loss, or timeout
    -> Faulted

Running
  inject input object for inference step 0
  consume output objects from token-out edge in inference-step order
  after output object for inference step k:
    if run policy stops -> Completed
    else inject next input object for inference step k + 1
  on StageFault, edge fault, endpoint fault, membership loss, or timeout
    -> Faulted

Completed
  stop injecting objects
  finalize the run output defined by the run contract
  -> TearingDown

Faulted
  stop injecting objects
  record one run-level failure reason
  -> TearingDown

OperatorStopped
  stop injecting objects
  record operator stop
  -> TearingDown

TearingDown
  send StopRun to all provisioned stages
  tear down local token endpoints
  wait for StageStopped from every stage or teardown timeout
  -> Done

Done
  terminal
```

The orchestrator does not issue per-stage `ExecuteStep` during a run. After
prompt injection, stage execution is driven by object arrival and local stage
readiness.

---

## 14. StageController FSM

Each provisioned stage has one node-local StageController. It is control-path
only: it watches worker and edge events, issues worker commands, and reports
run-level events. It does not move payload bytes.

```text
Unprovisioned
  on ProvisionStage from authorized orchestrator
    validate run_id and stage assignment
    -> Preparing

Preparing
  configure local worker role path
  start GGUF shard/range load and bind
  establish inbound receive edge
  establish outbound send edge
  when worker configured, weights bound, and both edge ends ready
    emit StageReady
    -> Ready
  on required setup failure
    emit StageFault
    -> Faulted

Ready
  on inbound ObjectLoaded(sequence = s)
    if s is next expected sequence and local state is ready
      issue ExecuteStep for sequence s
      -> Executing
    else
      emit StageFault(sequence violation)
      -> Faulted
  on StopRun
    -> Stopping

Executing
  worker runs exactly one step for the loaded input object
  worker writes output object to outbound edge with same sequence
  on StepCompleted
    release per-step input handles according to policy
    -> Ready
  on StepFailed, ObjectFailed, output fault, or worker crash
    emit StageFault
    -> Faulted

Faulted
  reject new run work
  wait for StopRun
  -> Stopping

Stopping
  stop local edges
  release per-run device objects
  stop or reset worker role state according to local policy
  emit StageStopped
  -> Stopped

Stopped
  terminal for this run
```

The MVP allows one active `ExecuteStep` per stage.

---

## 15. Weight Lifecycle

Weights are stage-local persistent state for the run.

The StageController starts weight work from the assigned `GgufSource` and layer
range. A stage may download:

- a whole GGUF and load only its range
- one or more physical shards containing its range
- a cached artifact that already exists on the node

The physical loading mechanism may be:

- worker startup configuration
- a `ConfigureRole` command
- a local loader path owned by the StageController
- weight objects moved through the same object/ring machinery

The system-visible contract is `WeightsReady` before `StageReady`.

`WeightsReady` means:

- assigned artifact bytes are locally available or already cached
- assigned layer range has been validated against the run plan
- GPU worker has loaded or bound the range needed for execution
- failures in download, parse, device allocation, or binding have surfaced as
  stage faults

Warm model reuse across runs is deferred.

---

## 16. Execution Semantics

Inference step `0` is the initial/prefill step:

```text
orchestrator writes input object for inference step 0
stage 0 executes over the input object for inference step 0
stage 0 writes output object for inference step 0
each interior stage executes over its input object for inference step 0
last stage writes output object for inference step 0
orchestrator consumes output object for inference step 0
```

Continuation steps are `1..`:

```text
orchestrator writes input object for inference step k
stage 0 executes for inference step k
each downstream stage executes for inference step k
last stage writes output object for inference step k
orchestrator consumes output object for inference step k
```

Payload bytes are opaque to this protocol. A run plan's `ObjectSpec`, runtime
policy, and role contract define what an input or output object means and how
the orchestrator decides whether to continue.

For every stage:

- inbound object sequence equals outbound object sequence for the same
  inference step
- a stage cannot execute before weights are loaded and bound
- a stage cannot execute before inbound object is loaded
- a stage cannot produce to an edge that is not ready
- `ObjectLoaded` is data readiness, not compute completion
- `ObjectProduced` is output object committed to egress ring
- `StepCompleted` is the compute transaction terminal success event

The orchestrator writes inference step `k + 1` only after consuming and accepting
the output object for inference step `k` and deciding the run should continue.

---

## 17. Object Specs And Object Records

Objects are logical byte payloads on an edge. Their payload meaning is defined
by the run plan's `ObjectSpec` and role contract; transport, edge, and lifecycle
components treat payload bytes as opaque.
Object specs are role-known validation contracts:

```rust
struct ObjectSpec {
    kind: ObjectKind,
    max_extent: u64,
    dtype_family: DTypeFamily,
    dtype_width_bytes: u32,
    shape: ShapeRule,
    layout: LayoutRule,
    alignment: u32,
    sequence_policy: SequencePolicy,
}
```

The stream carries object records:

```text
ObjectHeader
payload bytes, exactly header.extent bytes
```

The MVP header is fixed-size:

```rust
struct ObjectHeader {
    magic: u32,
    version: u16,
    header_len: u16,
    object_id: u64,
    sequence: u64,
    extent: u64,
    flags: u32,
    reserved: u32,
}
```

The header supplies runtime facts: object id, inference-step sequence, extent,
and flags. `ObjectSpec` supplies dtype, shape/layout family, max extent,
alignment, and sequence policy.

Activation maximum extent:

```text
max_extent = max_seq_len * hidden_dim * dtype_width_bytes
```

`extent` may vary by inference step and object contract. It must satisfy
`extent <= ObjectSpec.max_extent`.

The worker rejects a record before exposing a device object if:

- magic or version is unsupported
- header length is malformed
- `extent > ObjectSpec.max_extent`
- extent violates alignment/layout rules
- sequence violates the edge inference-step ordering policy
- ring or stream closes before `extent` bytes arrive

Payload content itself is trusted. The worker does not inspect tensor values.

---

## 18. Ring Specs And Layouts

`RingSpec` is the requested operating shape for a ring:

```rust
struct RingSpec {
    data_capacity: u64,
    alignment: u32,
    direction: RingDirection,
    host_pinning: HostPinning,
    wake_coalescing: WakeCoalescing,
}

enum RingDirection {
    Ingress,
    Egress,
}

enum HostPinning {
    Pageable,
    PinnedRequired,
}

enum WakeCoalescing {
    PendingBit,
    ReadySet,
}
```

`RingLayout` is the arena-relative layout minted by ArenaManager after a lease:

```rust
struct RingLayout {
    ring_id: RingId,
    arena_offset: u64,
    total_len: u64,
    header_offset: u64,
    data_offset: u64,
    data_capacity: u64,
    alignment: u32,
}
```

`RingLayout` never contains process-local pointers. Each process derives local
pointers from its own arena mapping base plus arena offsets.

The ring header and data region must be aligned for cross-process atomics and
backend copy requirements. Host pinning is optional for the MVP, but if a range
is pinned, it cannot be unpinned or returned to the free list until ring
quiescence.

---

## 19. Arena Model

Each node owns one sparse host-memory arena.

The arena is a `memfd`, created by the Rust node process, truncated to a large
sparse ceiling, and mapped once in the Rust process. The GPU worker maps the same
fd once. Neither process remaps or resizes the arena during node lifetime.

The arena is address space. It does not define flow control and does not define
object ownership.

The ArenaManager owns:

- arena fd
- node process mapping base pointer
- reservation ceiling
- arena free list
- pending lease queue
- table of live ring leases

The ArenaManager does not own:

- worker process
- QUIC stream
- pump task
- object parser
- payload byte

The ArenaManager only mints stable offsets and releases ranges after proof of
quiescence.

ArenaManager FSM:

```text
Booting
  on ConstructArena{ceiling}
    -> Ready if memfd, truncate, mmap succeed
    -> Failed if any boot resource fails

Ready
  on LeaseRing{request_id, requester, edge_id, direction, ring_spec}
    -> RingLeased if a range fits
    -> queue request if satisfiable but no current range fits
    -> RingLeaseRejected if request can never fit in the ceiling

Ready
  on CancelLease{request_id}
    remove queued request if not yet leased

Ready
  on ReleaseRing{ring_id, proof}
    return range to free list
    retry queued leases serially

Ready
  on Shutdown
    -> ShuttingDown

ShuttingDown
  reject new leases
```

Temporary arena pressure is represented by a queued lease request. Permanent
impossibility is explicit through `RingLeaseRejected`.

Two live leases cannot overlap because lease/release operations pass through one
ArenaManager mailbox.

---

## 20. Ring Contract

A ring is a bounded single-producer/single-consumer byte stream backed by an
arena lease.

Rings define flow control for every payload-moving boundary:

- QUIC ingress stream -> host ingress ring -> worker -> GPU memory
- GPU memory -> worker -> host egress ring -> QUIC egress stream
- disk reader -> host ring -> worker/GPU memory, if used

Each ring has exactly one producer and one consumer.

Ingress:

```text
producer = recv-pump
consumer = GPU worker
```

Egress:

```text
producer = GPU worker
consumer = send-pump
```

Fan-in and fan-out are not ring features. They are represented by multiple rings
or by a higher-level component that owns one side of a ring.

The shared arena ABI is not a Rust collection. It stores offsets, cursors, state
bits, wake bits, and payload bytes. It never stores process-local pointers.

Ring header:

```rust
#[repr(C, align(64))]
struct RingHeader {
    magic: u32,
    version: u16,
    header_len: u16,
    ring_id: u64,
    capacity: u64,
    commit: AtomicU64,
    consume: AtomicU64,
    state: AtomicU32,
    wake: AtomicU32,
}
```

`commit` is the first byte after the committed readable prefix. Bytes with
logical positions `< commit` are valid for the consumer.

`consume` is the first byte not yet released by the consumer. Bytes with logical
positions `< consume` are free for producer reuse.

The producer keeps a local `write` cursor. `write` is the first byte after the
producer's reserved or in-progress write prefix.

Readable interval:

```text
consume .. commit
```

Reserved but not readable interval:

```text
commit .. write
```

Physical index:

```text
physical_index = cursor % capacity
```

The ring is empty when `consume == commit`. The ring is full when:

```text
write - consume == capacity
```

Cursor values are monotonic logical byte positions.

---

## 21. Ring Producer, Consumer, And Wake Rules

Producer rule:

- acquire `consume`
- compute free space as `capacity - (write - consume)`
- reserve a contiguous physical span by advancing local `write`
- write bytes into that span
- publish new `commit` with release ordering after bytes are valid
- send or coalesce `RingReadable { ring_id }`

Consumer rule:

- acquire `commit`
- read bytes in `consume .. commit`
- greedily drain all bytes it can use
- publish new `consume` with release ordering after bytes are safe to release
- send or coalesce `RingWritable { ring_id }` if producer may be unblocked

Wake hints are edge-trigger hints, not ownership:

```rust
RingReadable { ring_id: RingId }
RingWritable { ring_id: RingId }
```

The receiver of a wake must reload cursors from shared memory. Wake hints carry
no byte ranges, byte counts, host pointers, or free-space counts.

Duplicate wake hints may be coalesced only while a pending bit, ready-set entry,
or equivalent durable scheduler state still makes the ring discoverable. Losing
the only transition from empty to readable or full to writable is a liveness bug.

If asynchronous DMA reads host ring memory, the consumer cannot advance
`consume` until the DMA no longer depends on those bytes.

Safety:

- consumer cannot read unwritten bytes because producer publishes `commit` only
  after writing
- producer cannot overwrite unread bytes because free space is computed from
  consumer-owned `consume`
- wraparound cannot alias stale bytes because ownership uses monotonic logical
  cursors
- stale wake/control events cannot alias replacement rings because `RingId` is
  unique for node lifetime

---

## 22. Native Ring Helper

Both Rust hot-path code and the Python worker access process-crossing rings
through the same native implementation.

Python does not implement shared atomics, wrap arithmetic, span calculation, or
cursor publication directly.

The helper exposes operations equivalent to:

```text
ring_open(arena_base, RingLayout) -> RingHandle
ring_close(handle)
ring_readable_span(handle) -> ptr, len
ring_advance_consume(handle, len)
ring_writable_span(handle) -> ptr, len
ring_advance_commit(handle, len)
ring_state(handle) -> state
```

Returned pointers are process-local addresses derived from the caller's mapped
arena base plus arena offsets.

Ring cursors must use atomic acquire/release semantics across the process
boundary.

---

## 23. Edge Stream Protocol

Each edge uses one persistent iroh/QUIC uni-stream from producer node to
consumer node.

Wire shape:

```text
[edge_id preamble]
[object record]
[object record]
...
```

The receiver's edge-demux reader consumes the fixed-width `edge_id` preamble and
hands the stream to the driver rendezvous. After the preamble, the recv-pump is
byte-blind. It copies stream bytes into the ingress ring and advances `commit`.
The worker parses object records from the ring.

The stream is persistent for the edge. It is not opened per object.

QUIC owns reliability and stream ordering. The MVP does not add app-level
fragment hashes, striping, resume, or per-object streams.

---

## 24. Driver And Transport

The driver is the node's swactor-to-iroh boundary. It owns:

- one iroh endpoint
- connection cache
- edge ALPN
- edge demux
- receive stream rendezvous
- send/recv pump tasks

Example edge ALPN:

```text
swactor/edge/1
```

Connections are cached per `(peer_node_id, ALPN)`. All edges between the same
node pair and ALPN reuse the same connection. Each edge has one persistent
uni-stream within that connection.

Driver inbound messages:

```rust
EstablishSend {
    edge_id: EdgeId,
    rx_node_id: NodeId,
    ring_id: RingId,
    ring_layout: RingLayout,
    tx_addr: ActorAddress,
}

EstablishRecv {
    edge_id: EdgeId,
    ring_id: RingId,
    ring_layout: RingLayout,
    rx_addr: ActorAddress,
}

StreamArrived {
    edge_id: EdgeId,
    stream: RecvStream,
}

StopEdge {
    edge_id: EdgeId,
}
```

Driver outbound events:

```rust
DriverEdgeReady { edge_id: EdgeId }
StreamClosed { edge_id: EdgeId }
StreamFault { edge_id: EdgeId, reason: StreamFaultReason }
PumpStopped { edge_id: EdgeId, ring_id: RingId }
```

No per-object, per-range, or fragment messages exist in the driver hot path.

---

## 25. Receive Demux And Pump FSMs

A recv-pump needs two resources:

- local receive establishment state, including ingress ring
- arriving QUIC stream

They may arrive in either order. The driver stores both halves:

```text
recv_specs: HashMap<EdgeId, RecvSpec>
pending_streams: HashMap<EdgeId, RecvStream>
```

On `EstablishRecv`, if a pending stream exists, the driver spawns the recv-pump.
Otherwise it stores the spec.

On `StreamArrived`, if a recv spec exists, the driver spawns the recv-pump.
Otherwise it stores the stream.

This removes the need for an inter-end readiness handshake. If a stream arrives
before local receive establishment, it waits in `pending_streams`; because no
recv-pump reads from it, QUIC flow control eventually stalls the sender.

Recv-pump FSM:

```text
WaitingForSpecAndStream
  -> Streaming when ring and stream are both present

Streaming
  read free ring span
  read QUIC bytes into that span
  advance commit
  send/coalesce RingReadable{ring_id} to worker
  repeat

Backpressured
  entered when no ring free space exists
  wait for RingWritable{ring_id}
  return to Streaming

Closed
  entered on stream EOF or edge teardown

Faulted
  entered on read error, protocol edge failure, or ring fault
```

The recv-pump does not parse `ObjectHeader` and does not know object boundaries.
EOF alignment is classified by the worker parser, not by the pump.

Send-pump FSM:

```text
WaitingForConnection
  ensure or await cached edge-ALPN connection

WaitingForBytes
  wait for RingReadable{ring_id}

OpenStream
  open one uni-stream
  write edge_id preamble
  -> Streaming

Streaming
  acquire commit
  write committed egress bytes to QUIC
  advance consume after bytes are accepted by write_all
  send/coalesce RingWritable{ring_id}
  repeat

Backpressured
  write_all is pending because network/QUIC flow control is slow
  keep ownership of unread ring bytes until write completes

Closed/Faulted
  emit coarse driver event
```

---

## 26. Edge Establishment

Establishment is local actor setup plus transport rendezvous. Remote edge ends
do not exchange actor messages.

For each edge, the orchestrator provisions the producer and consumer:

```rust
struct ProvisionTx {
    edge_id: EdgeId,
    rx_node_id: NodeId,
    object_spec: ObjectSpec,
    ring_spec: RingSpec,
}

struct ProvisionRx {
    edge_id: EdgeId,
    object_spec: ObjectSpec,
    ring_spec: RingSpec,
}
```

The producer needs consumer `node_id`, not consumer actor address. The consumer
needs shared `edge_id`, not producer actor address.

Each node has one EdgeEstablisher actor. It owns per-edge records:

```rust
struct EdgeRecord {
    edge_id: EdgeId,
    direction: RingDirection,
    state: EdgeProvisionState,
    lease_request_id: Option<LeaseRequestId>,
    ring_id: Option<RingId>,
    local_edge_actor: ActorAddress,
}
```

Record FSM:

```text
New
  on ProvisionTx/ProvisionRx
    spawn local Tx/Rx edge actor
    create LeaseRequestId
    send LeaseRing to ArenaManager
    -> WaitingForLease

WaitingForLease
  on RingLeased matching request_id
    record ring_id
    send InstallRing to GpuWorkerCtl or token endpoint
    -> WaitingForWorkerRing
  on RingLeaseRejected matching request_id
    notify local edge actor failure
    -> Failed
  on StopEdge
    send CancelLease
    notify local edge actor stopped
    -> Stopped

WaitingForWorkerRing
  on RingInstalled
    send EstablishSend/EstablishRecv to Driver
    -> WaitingForDriver
  on RingFault or StopEdge
    -> Stopping

WaitingForDriver
  on DriverEdgeReady
    notify local Tx/Rx actor Ready
    -> Ready
  on StreamFault/RingFault/StopEdge
    -> Stopping

Ready
  hot path runs without EdgeEstablisher
  coarse object/fault events may pass through

Stopping
  stop pump if one exists
  uninstall worker ring if installed
  wait for quiescence proofs
  release ring if leased
  -> Stopped
```

Stale events for stopped records are ignored except that an unused fresh lease
granted after cancellation must be released without installing worker or pump
state.

---

## 27. Tx And Rx Edge Actors

Tx and Rx actors are role-facing lifecycle gates.

Tx actor state:

```text
Provisioning -> Ready -> Producing -> Stopping -> Stopped
                         -> Faulted
```

Rx actor state:

```text
Provisioning -> Ready -> LoadingObject -> ObjectReady -> Stopping -> Stopped
                         -> Faulted
```

They receive coarse events:

```rust
EdgeReady { edge_id: EdgeId }
ObjectLoaded { edge_id: EdgeId, object_id: ObjectId, device_handle: DeviceObjectHandle }
ObjectProduced { edge_id: EdgeId, object_id: ObjectId }
ObjectFailed { edge_id: EdgeId, object_id: Option<ObjectId>, reason: ObjectFailure }
StreamFault { edge_id: EdgeId, reason: StreamFaultReason }
StopEdge { edge_id: EdgeId }
```

They do not receive:

- bytes
- host pointers
- per-range readiness
- consumed ranges
- free-space events
- flow-control credits

---

## 28. GPU Worker Process Boundary

Each node has one `GpuWorkerCtl` actor. It owns:

- worker process spawn and termination through `swactor_process`
- worker generation numbering
- arena fd inheritance or passing setup
- process actor address and notification bridge
- table of installed rings for current worker generation
- routing worker events to EdgeEstablisher, Driver, Tx/Rx actors, and role layer
- crash detection and crash fanout
- restart policy

`GpuWorkerCtl` does not own:

- arena leases
- QUIC streams
- ring payload bytes
- device allocations
- tinygrad execution
- graph placement

The worker process owns:

- mapped arena view
- worker-side ring handles
- per-ring parser/producer state
- device allocations
- device object handles
- tinygrad role state
- KV cache and worker-internal state
- host-to-device and device-to-host copy scheduling
- copy completion tracking

The public Rust-side interface is a normal actor message enum:

```rust
enum GpuWorkerCtlMsg {
    StartWorker,
    InstallRing(InstallRing),
    UninstallRing(UninstallRing),
    RingReadable { ring_id: RingId },
    RingWritable { ring_id: RingId },
    ConfigureRole(ConfigureRole),
    ExecuteStep(ExecuteStep),
    ReleaseDeviceObject { device_handle: DeviceObjectHandle },
    ShutdownWorker(ShutdownWorker),
    Process(ProcessOutput),
    WorkerAdapter(WorkerAdapterEvent),
}
```

`Process(ProcessOutput)` is delivered by the configured upstream process owner;
there is no process-local notification subscription bridge. If the
production worker still uses a subprocess stdin/stdout protocol, `GpuWorkerCtl`
talks to a separate worker I/O adapter. That adapter owns the child stdio handles
and is outside managed-process core.

The optional worker I/O adapter may use newline-delimited JSON for worker
commands/events. This is an adapter-local protocol, not part of
`crates/process` and not a second distributed protocol.

Adapter rules:

- one command/event JSON object per line;
- adapter-owned stdout may carry worker events;
- adapter-owned stderr may carry logs and diagnostics;
- payload bytes are forbidden in JSON;
- invalid JSON or unknown event shape is a worker/adapter fault.

Worker environment:

```text
SWACTOR_ARENA_FD      memfd for the shared arena
SWACTOR_ARENA_BYTES   arena reservation ceiling
```

---

## 29. GPU Worker Commands

`InitializeWorker`:

```rust
struct InitializeWorker {
    worker_generation: WorkerGeneration,
    arena_ceiling: u64,
    required_ring_helper_abi: u16,
    backend: JsonValue,
}
```

Maps the arena, initializes native helper and backend, then emits `WorkerReady`
or `WorkerFatal`.

`ConfigureRole` is optional:

```rust
struct ConfigureRole {
    role_id: RoleId,
    config: JsonValue,
}
```

`config` is low-frequency app metadata and must not carry payload bytes.

`InstallRing`:

```rust
struct InstallRing {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
    direction: RingDirection,
    layout: RingLayout,
    object_spec: ObjectSpec,
}
```

`edge_id` and `port_id` are required so worker events can be reported in
graph-facing terms.

`UninstallRing`:

```rust
struct UninstallRing {
    ring_id: RingId,
    reason: UninstallReason,
}
```

The worker removes ring state, waits for copy lifetimes to end, closes the
helper handle, and emits `RingQuiesced`.

Wake hints:

```rust
RingReadable { ring_id: RingId }
RingWritable { ring_id: RingId }
```

`ExecuteStep`:

```rust
struct ExecuteStep {
    role_id: RoleId,
    step_id: StepId,
    inputs: Vec<InputBinding>,
    outputs: Vec<OutputBinding>,
    runtime: JsonValue,
    release_inputs_after: bool,
}

struct InputBinding {
    port_id: PortId,
    object_id: ObjectId,
    sequence: Sequence,
    device_handle: DeviceObjectHandle,
}

struct OutputBinding {
    port_id: PortId,
    ring_id: RingId,
    object_id: ObjectId,
    sequence: Sequence,
    extent: u64,
    flags: u32,
}
```

Execution requirements:

1. Validate role is available.
2. Validate input handles belong to current worker generation.
3. Wrap inputs as tinygrad-compatible views.
4. Run role code.
5. Validate returned outputs against declared output bindings.
6. Write each output object to the named egress ring.
7. Emit `ObjectProduced` after each full output object is committed.
8. Emit `StepCompleted` after all declared outputs are produced and role state
   updates are complete.

`ReleaseDeviceObject` frees a device object after no compute or copy event still
references it.

`ShutdownWorker`:

```rust
struct ShutdownWorker {
    mode: ShutdownMode,
}

enum ShutdownMode {
    Graceful,
    AbortInFlight,
}
```

Deferred worker commands:

- generic command accepted/rejected acks
- `BindDeviceObject` / `UnbindDeviceObject`
- `CancelStep`
- `AbortObject`
- `Ping`
- role module/factory provisioning

---

## 30. GPU Worker Events

Worker lifecycle:

```rust
WorkerReady {
    pid: u32,
    worker_generation: WorkerGeneration,
    ring_helper_abi: u16,
    backend: JsonValue,
}

WorkerFatal {
    reason: WorkerFatalReason,
}

WorkerStopped {
    reason: WorkerStoppedReason,
}
```

Role events, only if `ConfigureRole` is used:

```rust
RoleConfigured { role_id: RoleId }
RoleFailed { role_id: RoleId, reason: RoleFailure }
```

Ring events:

```rust
RingInstalled { ring_id: RingId, edge_id: EdgeId, port_id: PortId }
RingFault { ring_id: RingId, edge_id: EdgeId, port_id: PortId, reason: RingFaultReason }
RingQuiesced { ring_id: RingId }
```

Object events:

```rust
ObjectLoaded {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
    object_id: ObjectId,
    sequence: Sequence,
    extent: u64,
    device_handle: DeviceObjectHandle,
}

ObjectProduced {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
    object_id: ObjectId,
    sequence: Sequence,
    extent: u64,
}

ObjectFailed {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
    object_id: Option<ObjectId>,
    sequence: Option<Sequence>,
    reason: ObjectFailure,
}
```

Step events:

```rust
StepCompleted { role_id: RoleId, step_id: StepId }
StepFailed { role_id: RoleId, step_id: StepId, reason: StepFailure }
```

Device lifetime:

```rust
DeviceObjectReleased { device_handle: DeviceObjectHandle }
ReleaseFailed { device_handle: DeviceObjectHandle, reason: ReleaseFailure }
```

Wake hints emitted by worker:

- ingress rings: `RingWritable` after advancing `consume`
- egress rings: `RingReadable` after advancing `commit`

`GpuWorkerCtl` may synthesize `WorkerCrashed` and `RingFault` after process exit,
process error, or worker I/O adapter control-stream failure.

---

## 31. GpuWorkerCtl And Worker FSMs

`GpuWorkerCtl` FSM:

```text
NotStarted
  on StartWorker -> Spawning

Spawning
  spawn ProcessActor with ProcessSpec and upstream = GpuWorkerCtl/process owner
  wait for ProcessOutput::Started
  if subprocess worker protocol is enabled:
    start/connect worker I/O adapter
    send InitializeWorker through worker I/O adapter
  -> Initializing

Initializing
  on WorkerReady -> Running
  on WorkerFatal/process exit/timeout -> Failed

Running
  on actor command -> validate state, write worker command, update tables
  on worker event -> route event
  on ShutdownWorker -> Stopping
  on process exit/error -> Crashed

Stopping
  send ShutdownWorker if process alive
  wait for WorkerStopped and process exit
  on timeout -> Killing

Killing
  close/kill ProcessActor according to policy
  reap process notification
  mark installed rings faulted
  -> Stopped

Crashed
  mark current-generation device handles invalid
  mark installed rings faulted
  ask driver to stop pumps for installed rings
  wait for teardown to release rings
  -> Failed or Restarting

Restarting
  increment worker_generation
  -> Spawning
```

Worker process FSM:

```text
Booting
  read environment
  wait for InitializeWorker on stdin

Initializing
  map arena
  initialize native helper
  initialize backend/tinygrad
  emit WorkerReady
  -> Running

Running
  on InstallRing -> install ring state
  on RingReadable/RingWritable -> reload cursors and advance ring FSMs
  on ExecuteStep -> run explicit role step subject to backpressure
  on ReleaseDeviceObject -> release handle when safe
  on ShutdownWorker -> Draining

Draining
  reject new work
  finish or abort in-flight work according to mode
  quiesce rings
  emit WorkerStopped
  exit

Fatal
  emit WorkerFatal if possible
  exit non-zero
```

The worker greedily makes progress after relevant commands or wake hints: drain
readable ingress prefixes, advance egress output if writable space exists,
observe copy completions, and emit resulting events.

---

## 32. Worker Ring FSMs

Ingress ring FSM:

```text
Uninstalled
  on InstallRing(direction = Ingress) -> NeedHeader

NeedHeader
  on committed bytes < header_len -> wait
  on complete header -> validate
    valid -> allocate device object -> NeedPayload
    invalid -> Faulted

NeedPayload
  copy committed payload prefixes to device allocation
  advance consume only after copied bytes are safe to release
  if copied == extent -> ObjectComplete
  on close before copied == extent -> Faulted

ObjectComplete
  wait for copy completion
  emit ObjectLoaded
  -> NeedHeader

Faulted
  emit ObjectFailed or RingFault
  stop consuming until UninstallRing

Uninstalling
  stop consuming
  wait for copy lifetimes to end
  emit RingQuiesced
  -> Uninstalled
```

Egress ring FSM:

```text
Uninstalled
  on InstallRing(direction = Egress) -> WaitingForOutput

WaitingForOutput
  on ExecuteStep output binding naming this ring -> NeedWritableForHeader

NeedWritableForHeader
  wait for writable span
  write ObjectHeader
  advance commit
  emit RingReadable
  -> NeedWritableForPayload

NeedWritableForPayload
  copy device payload prefixes into writable ring spans
  advance commit only after host bytes are valid
  emit/coalesce RingReadable
  if produced == extent -> ObjectProduced
  -> WaitingForOutput

Faulted
  emit RingFault
  stop producing until UninstallRing

Uninstalling
  stop accepting outputs
  wait for copy lifetimes to end
  emit RingQuiesced
  -> Uninstalled
```

The egress producer may block an `ExecuteStep` while waiting for egress ring
space. This is the intended backpressure path.

---

## 33. Device Bridge

The worker must provide a backend-specific device bridge:

```text
alloc_device(ObjectSpec, extent) -> DeviceAllocation
free_device(DeviceAllocation)
host_to_device(arena_ptr, len, DeviceAllocation, device_offset) -> CopyEvent
device_to_host(DeviceAllocation, device_offset, arena_ptr, len) -> CopyEvent
copy_event_complete(CopyEvent) -> bool
wrap_for_tinygrad(DeviceAllocation, TensorViewSpec) -> tinygrad object
```

The canonical contract supports range copies so objects may be larger than a
ring. Ingress and egress can then stream prefixes through bounded host rings
while the logical object lives in device memory.

If an implementation uses a constrained complete-object fallback, that mode must
be explicit and must preserve system-visible object, sequence, readiness, and
backpressure semantics. It must also constrain object/ring sizing accordingly.

For synchronous copies:

```text
copy returns -> safe to advance relevant cursor
```

For asynchronous copies:

```text
start copy -> record CopyEvent and host/device ranges
event complete -> advance cursor or release device object
```

---

## 34. Ingress And Egress Flow

Ingress path:

```text
remote worker/GPU -> remote egress ring -> QUIC -> local ingress ring
  -> local worker -> local GPU memory -> ObjectLoaded
```

Receiving sequence:

1. Driver demux reads stream `edge_id` preamble.
2. Driver pairs stream with local ingress ring.
3. Recv-pump waits for free ring space.
4. Recv-pump reads QUIC bytes into free ring span.
5. Recv-pump advances `commit`.
6. Recv-pump sends/coalesces `RingReadable`.
7. Worker wakes and reads `consume..commit`.
8. Worker parses object headers when enough committed bytes exist.
9. Worker allocates device object after header validation.
10. Worker copies committed payload prefixes into device memory.
11. Worker advances `consume` after copies are safe to release.
12. Recv-pump sees free space and continues.
13. Worker emits `ObjectLoaded` after full object and copy completion.

Egress path:

```text
local GPU memory -> local worker -> local egress ring -> QUIC
  -> remote ingress ring -> remote worker/GPU
```

Sending sequence:

1. Worker receives or creates compute result in GPU memory.
2. Worker creates `ObjectHeader` according to `ObjectSpec`.
3. Worker waits for free egress ring space.
4. Worker writes and commits header bytes.
5. Worker copies payload bytes from device into writable ring spans.
6. Worker advances `commit` as host bytes become valid.
7. Worker sends/coalesces `RingReadable`.
8. Send-pump opens persistent uni-stream on first bytes.
9. Send-pump writes `edge_id` preamble once.
10. Send-pump writes committed bytes to QUIC.
11. Send-pump advances `consume` after `write_all` accepts bytes.
12. Send-pump sends/coalesces `RingWritable`.

An object may be larger than the ring. The ring is a transfer window, not object
storage. Compute visibility still begins only at `ObjectLoaded`.

---

## 35. Backpressure

No credits or per-range acknowledgements exist.

Worker slow on ingress:

```text
worker slow
  -> ingress consume does not advance
  -> recv-pump sees no free ingress space
  -> recv-pump stops reading QUIC
  -> QUIC stream flow control stalls remote sender
  -> sender send-pump stops draining egress ring
  -> sender worker stalls on egress free space
```

Network slow on egress:

```text
network slow
  -> send-pump write_all remains pending
  -> egress consume does not advance
  -> worker sees no free egress space
  -> worker stops producing more bytes into egress ring
```

Arena pressure:

```text
arena temporarily exhausted
  -> ArenaManager queues LeaseRing
  -> EdgeEstablisher record remains WaitingForLease
  -> no worker ring is installed
  -> no pump is spawned
  -> no hot-path state exists for that edge
```

No data is dropped because each layer stops before overwriting unread bytes.

---

## 36. Fault Semantics

A run has one execution terminal outcome:

- completed
- faulted
- operator-stopped before completion

Run fault sources include:

- node unavailable before or during run
- membership loss for a required node
- provisioning rejection
- arena boot failure
- oversized ring request
- temporary pressure timeout
- weight download/load/bind failure
- edge establishment failure
- malformed object header
- EOF mid-object
- stream fault
- pump failure
- ring fault
- worker fatal error
- worker crash
- device OOM
- device copy failure
- inference-step ordering violation
- step failure

Teardown failures are recorded in teardown outcome/status. They do not rewrite
a completed inference into a faulted run outcome.

MVP recovery policy is fail-stop at the run level. The orchestrator records the
first run-level failure reason, stops injecting objects, and begins teardown. It
does not re-place the run.

After a stage faults, it rejects new run work until stopped.

---

## 37. Error Taxonomy

Ring faults:

```rust
enum RingFaultReason {
    UnsupportedRingVersion,
    RingLayoutInvalid,
    RingStateInvalid,
    DeviceOutOfMemory,
    DeviceCopyFailed,
    SequenceViolation,
    HeaderMalformed,
    WorkerProcessExited,
    WorkerShuttingDown,
    WorkerInternal,
}
```

Object failures:

```rust
enum ObjectFailure {
    HeaderMalformed,
    UnsupportedObjectVersion,
    ExtentExceedsMax,
    ExtentAlignmentInvalid,
    SequenceViolation,
    DeviceAllocationFailed,
    DeviceCopyFailed,
    EofMidObject,
}
```

Step failures:

```rust
enum StepFailure {
    RoleUnavailable,
    InvalidInputHandle,
    InvalidOutputRing,
    TinygradError,
    DeviceOutOfMemory,
    OutputExtentInvalid,
    OutputCopyFailed,
    WorkerProcessExited,
    WorkerShuttingDown,
}
```

Worker fatal reasons:

```rust
enum WorkerFatalReason {
    ArenaMapFailed,
    RingHelperAbiMismatch,
    BackendInitFailed,
    MalformedControlMessage,
    UnhandledException,
}
```

Other control reasons:

```rust
enum UninstallReason {
    EdgeStopped,
    RingFaulted,
    WorkerShutdown,
}

enum WorkerStoppedReason {
    Graceful,
    AbortInFlight,
    Fatal,
}
```

---

## 38. Teardown

Teardown starts after completion, fault, or operator stop.

Teardown has its own outcome, separate from the run execution outcome:

```text
RunOutcome = Completed | Faulted | OperatorStopped
TeardownOutcome = Clean | Faulted | TimedOut
```

Orchestrator teardown:

1. Stop injecting objects.
2. Send `StopRun` to every provisioned stage.
3. Stop local token endpoints.
4. Wait for `StageStopped` from every stage or timeout.
5. Record run teardown outcome/status.

Stage teardown:

1. Stop accepting new run work.
2. Stop local edges.
3. Stop driver pumps.
4. Uninstall worker rings.
5. Release arena leases after quiescence proof.
6. Release per-run device objects.
7. Stop or reset worker role state according to local policy.
8. Emit `StageStopped`.

Edge teardown FSM:

```text
Ready
  on StopEdge or fault -> StoppingPump

StoppingPump
  driver stops recv/send pump
  driver emits PumpStopped
  -> StoppingWorkerRing

StoppingWorkerRing
  GpuWorkerCtl sends UninstallRing
  worker removes ring state
  worker waits for in-flight copies or process death
  worker emits RingQuiesced
  -> ReleasingArena

ReleasingArena
  EdgeEstablisher sends ReleaseRing{proof}
  ArenaManager returns lease to free list
  -> Stopped
```

Arena release is safe only after:

- driver pump stopped
- worker ring quiesced or worker process reaped
- copy lifetimes ended or owning process is gone

Teardown is required after success and fault.

---

## 39. System Guarantees

Authority:

- orchestrator is the only component that assigns topology, stages, layer
  ranges, edge ids, and object specs
- nodes reject provisioning and edge data that do not match active run plan

Readiness:

- orchestrator does not inject prompt before global readiness barrier
- a stage does not report `StageReady` before worker, weights, inbound edge,
  outbound edge, and StageController are ready

Weight use:

- a stage cannot execute before assigned weights are loaded and bound
- weight loading failures become stage faults

Edge identity:

- every edge id is unique within run
- every edge has one producer and one consumer
- data plane is addressed by `(node_id, edge_id)`
- remote actor addresses are not needed for data flow

Inference-step ordering:

- prefill is inference step `0`
- continuation steps are strictly increasing
- a stage executes step `s` only after loading inbound object sequence `s`
- stage output object uses sequence `s`
- orchestrator injects `s + 1` only after consuming and accepting output object
  sequence `s`
- workers do not invent graph-visible object ids or sequence numbers

Payload isolation:

- payload bytes never travel in actor messages, process control messages, logs,
  or JSON command/event lines
- actors carry identities, lifecycle events, wake hints, and opaque handles

Compute visibility:

- compute receives only complete logical objects
- `ObjectLoaded` is emitted only after valid header, exact extent copy, copy
  completion, and device handle creation

Backpressure:

- slow worker, slow network, or downstream stall propagates by ring and QUIC
  flow control
- unread bytes are not overwritten

Terminal outcome:

- each run records exactly one terminal outcome
- after terminal outcome begins, no new run work is accepted except teardown

Quiescence:

- arena memory cannot be reused under a live pump, worker ring, or copy
  operation
- release requires teardown proof

Worker restart:

- restart creates a new generation
- old device handles, roles, rings, and steps are invalid
- roles and rings must be reinstalled

---

## 40. Observability Surface

The system must emit stable lifecycle events for behavioral contracts and tests.
Event transport and storage are implementation details.

Required event identities:

- `run_id`
- `node_id`
- `stage_index`
- `edge_id`
- `ring_id`
- `object_id`
- `sequence`
- `step_id`
- `worker_generation`

Required lifecycle events:

```text
node_started
node_available
node_faulted
pool_ready
run_planned
stage_provision_started
weights_download_started
weights_downloaded
weights_loaded
edge_provision_started
edge_ready
stage_ready
readiness_barrier_passed
prompt_injected
object_loaded
execute_step_started
object_produced
step_completed
token_received
run_completed
run_faulted
stop_run_sent
stage_stopped
run_torn_down
```

Fault events must include a stable reason enum and the component that detected
the fault. Tests and operators should not need to scrape free-form logs to
determine lifecycle progress.

---

## 41. Behavioral Contract Inventory

The next specification layer should extract API surfaces and behavioral
contracts for these components and required behaviors:

- node boot lifecycle
- membership/SWIM convergence
- resource inventory and run planner
- orchestrator run FSM
- StageController
- stage-local weight loading path
- orchestrator token endpoint behavior
- ArenaManager
- shared ring helper/ABI
- EdgeEstablisher
- Tx and Rx edge actors
- iroh driver and recv/send pumps
- GpuWorkerCtl
- GPU worker process adapter
- GPU worker ingress parser
- GPU worker egress producer
- device bridge
- observability/event surface

Required test layers:

1. Component behavioral contract tests.
2. Local end-to-end mock tests without real networking.
3. Local Docker cluster tests with real ports and real networking.

VastAI smoke/integration validation comes after local Docker cluster behavior is
stable.

---

## 42. Canonical MVP Scenario

The first end-to-end scenario is one "hello world" prompt over a rented GPU
pool:

1. Operator provisions `N` GPU nodes with the barebones swactor + tinygrad/CUDA
   image.
2. Every node starts the Rust node process and GPU worker process.
3. Nodes join SWIM membership.
4. Orchestrator observes `PoolReady`.
5. Orchestrator builds a linear GGUF `RunPlan`.
6. Orchestrator provisions every stage.
7. Each stage downloads and loads assigned weights.
8. Each stage provisions inbound and outbound edges.
9. Every stage reports `StageReady`.
10. Orchestrator observes the global readiness barrier.
11. Orchestrator injects the initial input object for inference step `0`.
12. Stages execute the initial step and return output object sequence `0`.
13. Orchestrator continues until the run policy stops or `max_tokens` is reached.
14. Orchestrator records `run_completed`.
15. Orchestrator tears down edges, worker run state, and token endpoints.
16. Orchestrator records `run_torn_down`.

Success requires one terminal run outcome and every arena range leased by the run
to be quiesced or released by teardown.

---

## 43. Deferred

- automatic placement optimization
- arbitrary graph execution
- multi-input joins and fan-out beyond explicit role-layer handling
- multiple concurrent runs on one stage chain
- batching, speculative decoding, continuous serving
- warm model/weight reuse across prompts
- production artifact layout and weight cache eviction
- re-placement after node failure
- trustless verification or adversarial payload defense
- host pinning and asynchronous DMA performance policy
- removing backend copy limitations
- VastAI-specific provisioning automation