# The Datastream — Specification > **Status.** This is the canonical specification for the `datastream` crate. > It supersedes the implicit spec the old `crates/distribution/src/datastream` > doc-comments referred to (`DATASTREAM_SPEC.md §4.1`, etc.). Those section > numbers are not preserved; updating the stale `(spec §…)` references is part > of the migration this document drives. --- ## 1. What the datastream is The datastream is a **per-node, append-only telemetry pipe**. Every node produces exactly one stream of its own observations; one consumer reconstructs those streams and reads them. Its entire value comes from one rule: > **Nothing between a producer and a view ever interprets a payload.** Producers tag bytes with a channel name and hand them off. A single per-node **mux** stamps each with a position and interleaves them into one ordered stream. A best-effort **transport** carries the stream. **Ingest** reconstructs each node's stream by position into a **store**. **Views** decode bytes back into meaning — and only here, at read time, does anything look inside a payload. ```text producers tag bytes with a channel │ ▼ per-node MUX stamp position, interleave into one ordered stream │ ▼ transport best-effort: may drop / reorder / delay, never corrupt │ ▼ ingest reconstruct each node's stream by position │ ▼ store (the truth) whole, append-only, position-keyed │ ▼ views read-time projections — the only place bytes are decoded ``` This collapses the usual telemetry zoo — counters, gauges, histograms, logs, events, traces — into **one** thing: positioned bytes on a named channel. Collection, transport, and storage are uniform because none of them know which of those a frame "is." That distinction does not exist in the pipe. It exists, if at all, in a view: "the latest frame on this channel" is a gauge, "every frame on this channel" is an event log, and they are the same bytes read two ways. ### 1.1 Consequences that the rest of this spec just spells out - There is **one way data enters** (§4). No per-channel "kind of producer." - A channel is **a name, not a resource** (§5). Nothing to allocate or free. - All **semantics live in views** (§8). The pipe is mechanism only. --- ## 2. The data model ### 2.1 The frame is the only unit ```rust pub struct Frame { pub channel: ChannelId, // the named lane these bytes belong to pub position: Position, // the mux-assigned order within the node's stream pub payload: Vec, // opaque bytes — never interpreted by the pipe } ``` A typed record, a log line, and a binary blob are the same kind of thing here: bytes on a channel. The `payload` is opaque to the mux, the transport, and the store. There is **no per-frame wall-clock timestamp.** Frames are ordered and correlated by position alone. A producer that wants wall-clock time puts it *inside* the payload, as a field of its record — it is data, not a property of the pipe. ### 2.2 The coordinate Every frame in the system has exactly one address: ```text Frame @ (node, life, channel, position) └── stream ──┘ source order ``` | Part | Type | Answers | |------------|--------------|------------------| | `node` | `NodeId` | *who* produced it | | `life` | `Lifetime` | *which incarnation* of that node | | `channel` | `ChannelId` | *which source* within that node | | `position` | `Position` | *where in order* within that stream | `(node, life)` together are the **stream**; `channel` is the **source** within it; `position` is the **order**. This four-tuple locates any datum the system has ever produced. Addressing (§3) is built entirely on it. ### 2.3 Position: per-stream, monotonic, gap-free ```rust pub struct Position(pub u64); ``` - **One sequence per stream**, not per channel. A node's mux holds a single counter shared across every channel, so a given channel's frames carry *non-contiguous* positions — a sparse projection of the node's one global sequence, interleaved with every other channel. - **Monotonic and gap-free in assignment.** The mux never reuses a position and never skips one when numbering. A position that is assigned but never delivered surfaces downstream as a *missing* position — a detectable gap (§4.4, §6.3). - **Not comparable across streams.** Positions order frames within one `(node, life)` only. There is no global clock. **Why one counter and not one per channel.** It keeps the mux a single position authority and makes gap detection a whole-stream property: a hole means *something* was lost. The cost is that a drop **cannot be attributed to a specific channel** — you know a frame is missing, not which channel it carried. That is the right trade for telemetry. > **Decision of record — drop attribution.** If a particular source ever needs > guaranteed contiguous accounting (e.g. "did I lose a datastore op?"), it > carries its *own* sequence number as a field in its record. The pipe's > position stays global and dumb; domain counting is domain data. ### 2.4 Stream identity — incarnations never merge ```rust pub struct StreamId { pub node: NodeId, pub life: Lifetime } ``` `StreamId` is the ingest key. Two streams with the same `node` but different `life` are **different streams and must never merge.** A node that dies and is restarted (re-rented, re-scheduled) begins a new `life`, so its fresh stream does not append to — or collide with — its prior one. Restart is visible, not silently glued over. --- ## 3. Addressing ### 3.1 The address is the coordinate A frame is addressed by `(node, life, channel, position)`. A *source* is addressed by the prefix `(node, life, channel)` — drop `position` and you are naming a lane rather than a single datum. A *node's whole stream* is `(node, life)`. Nothing else is needed; there is no separate registry of producer identities. ### 3.2 Channel paths: opaque on the wire, structured at the edges ```rust pub struct ChannelId(Arc); // an opaque token to the pipe ``` To the mux, transport, ingest, and store a `ChannelId` is an uninterpreted string. They never split it, match it, or validate it. This is what keeps the pipe dumb and lets a brand-new channel flow end to end with zero pipe changes. Its **structure is a read-side convention** — known only to producers (to mint paths they own) and to the catalog and consumers (to match and decode them). It is therefore *both*: a flat string on the wire, a structured path at the edges. There is no conflict, because the two views never meet inside the pipe. ### 3.3 The path grammar A channel path is a dotted sequence of segments: ```text channel := namespace ( "." qualifier )* namespace := the owning subsystem e.g. host, transport, swim, runtime, dist, datastore, proc, identity qualifier := instance-key | leaf instance-key identifies *which* of a dynamic source; leaf names the signal ``` Examples, current and proposed: | Path | Namespace | Instance | Leaf | |-------------------------------|-------------|---------------|------------| | `identity` | identity | — | — | | `host.resource` | host | — | resource | | `transport.internals` | transport | — | internals | | `proc.trainer.stdout` | proc | `trainer` | stdout | | `transport.peer..conn` *(proposed)* | transport | `peer/` | conn | A single-segment path (`identity`, `membership`) is the degenerate case: a namespace with one global signal and no multiplicity. Instance segments appear only where a source has many instances (one per process, per peer, per actor). ### 3.4 Source selectors A consumer addresses sources by a **stream scope plus a channel prefix**: ```text selector := (node?, life?) channel-prefix [ ".*" ] ``` - `(*, *) membership` — every node's membership signal. - `(X, *) proc.trainer.*` — everything the trainer process on node `X` ever emitted, across that node's restarts. - `(X, Y) host.resource` — one exact source. - `(*, *) proc.*` — all process output, fleet-wide. Prefix matching over the path **is** the query model. Because instances live in the path, "all processes" or "one peer's view" is a prefix, never a side table of what exists. This is the whole reason the path is structured. --- ## 4. Population — the one way in ### 4.1 A producer holds a submission handle There is exactly one means of population. A producer holds a **submission handle** to its node's mux and calls: ```rust fn submit(&self, channel: impl Into, payload: Vec); ``` Everything else is sugar over this. A typed record encodes itself and submits (`record.emit_to(&handle)`); a text source submits a line; a binary source submits bytes. There is no second mechanism, no "event vs sample vs one-shot" path. *When* a producer calls `submit` — on a timer, in a callback, once at boot — is the producer's own business and is invisible to the stream. The handle is cheaply cloneable and `Send`, so any number of producers, on any threads, feed the same mux. The mux serializes them into one order (§4.3). ### 4.2 One observer per subsystem Every subsystem that produces telemetry exposes **one observer trait**, and the node wires that observer to a submission handle. This is the uniform "means of population" applied across the codebase: ```rust // e.g. datastore trait DatastoreEventObserver { fn on_event(&self, ev: DatastoreEvent); } // the node installs an observer whose body is `handle.submit(...)` ``` This replaces today's inconsistent wiring (some subsystems install a sink, some install `None`, some route through the node as an adapter). The rule: **a subsystem emits by handing its observer a submission handle — nothing else.** Existing observer-shaped seams (`DatastoreEventObserver`, `ProcessOutputObserver`, the unused `SwimObserver`) all conform to this one shape. ### 4.3 The mux: single position authority ```rust pub struct Mux { stream: StreamId, next: AtomicU64, buffer: Mutex> } ``` One mux per node. It is the sole assigner of positions and the outgoing buffer: 1. On `submit`, it atomically takes the next position **before** buffering the frame. Positions are consumed in submission order and never reused. 2. The frame goes into a bounded buffer. 3. The transport `drain`s the buffer each tick, emptying it in insertion order. Because the position is taken before buffering, the numbering is gap-free even when the buffer is not. ### 4.4 Drops are gaps (the back-pressure policy) The buffer is **bounded**. On overflow the mux **drops the frame but keeps its position consumed.** The producer never blocks. Downstream, the dropped position is simply missing — a detectable gap, never a silent renumber. > **Decision of record — lossy, not blocking.** Telemetry must never apply > back-pressure to the work it observes. A datastream is allowed to lose frames; > it is not allowed to stall a producer or to hide that a loss happened. Anything > that cannot tolerate loss does not belong on the datastream (§9). --- ## 5. Channels ### 5.1 Channels are names, not resources This is the core of channel management, and it makes the rest of this section short. A channel is **a name attached to frames** — nothing more. There is: - no channel object, no `open`/`close`, - no per-channel counter (positions are per-stream, §2.3), - no per-channel buffer (one buffer per mux, §4.3), - no registration call at runtime. A channel **exists** the instant a frame bears its name, and not before. Everything below follows from that. ### 5.2 The catalog: a registry of *meaning*, not of *existence* The catalog maps a channel path to how it should be decoded. It is static, compile-time, and finite. It answers *"given this path, what does it mean?"* — never *"what paths exist?"* (that is the store's job, §5.8). ```rust pub enum ChannelKind { Typed, Text, Opaque } pub fn classify(channel: &ChannelId) -> ChannelKind; ``` The pipe never consults the catalog. Only producers (to tag) and views (to decode) do. Adding a channel or teaching a view a new codec changes nothing in the mux, transport, ingest, or store. ### 5.3 Static channels and dynamic families - **Static channel** — a fully literal path known at compile time (`host.resource`). Registered as a constant. - **Dynamic family** — a path *template* with parameter segments (`proc.{label}.stdout`). The template is registered statically; concrete instances are minted at runtime by binding parameters: ```rust pub fn process_output(label: &str, stream: ProcStream) -> ChannelId; // proc.