Zachery Aaron Shores-Chmielewski
ae9ca3bcf3
Promote pipeline-parallel-inference to a first-class app and consolidate observability on the datastream wire, decoupling the dashboard crate from `distribution`. - apps/pipeline-parallel-inference: move the example out of `examples/` into `apps/` as its own workspace, rename binaries to `pp-worker`/`pp-orchestrator`, and strip release binaries - cluster: add `ClusterNode`, a synchronous facade over the actorized distribution protocol (IrohDriver + per-node Runtime hosting Swim/Registry/Metadata/Directory actors with a `MembershipFanout`), replacing ad-hoc `driver.node()`/`tick()` call sites - fleet: add per-node fleet telemetry that ships identity/resource records as `DatastreamFrame`s over the cluster transport to the orchestrator's `DatastreamSink`, folded into a `FleetView` on a 3s tick - provision: add best-effort, opt-in SSH boot-phase telemetry (`PP_DEPLOY_KEY`) that streams rented-node boot logs onto the orchestrator's datastream as `proc.boot.<stage>.*` - dashboard: rewire the crate dependency from `distribution` to `datastream`, drop the standalone `swactor-datastream-dashboard` binary, and rewrite `datastream_source.rs` to demux per-node frames into Overview/Distribution/Fleet views with live-node TTL filtering - distribution: refresh dist/netmap plugin copy and README from "Kademlia routing" to gossip-directory terminology Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
51 lines
2.1 KiB
Rust
51 lines
2.1 KiB
Rust
//! `DatastreamSink` — the cluster-side consumer of [`DatastreamFrame`] messages.
|
|
//!
|
|
//! This is the counterpart to [`ClusterFrameSink`](super::emit::ClusterFrameSink):
|
|
//! a node ships its ordered telemetry as `DatastreamFrame` actor messages over the
|
|
//! regular swactor transport, and this actor — registered under a well-known name
|
|
//! on the collector (e.g. the orchestrator) — receives them, decodes each back
|
|
//! into a `(StreamId, Frame)` delivery, and hands it to a caller-supplied fold.
|
|
//!
|
|
//! It deliberately knows nothing about the dashboard: folding a delivery into a
|
|
//! `FleetView` lives in the `dashboard` crate, which `distribution` must not
|
|
//! depend on. The actor therefore owns an opaque callback, and the binary that
|
|
//! has both crates in scope wires a `FleetView` into it. Malformed payloads are
|
|
//! dropped silently — the same best-effort tolerance the UDP ingest had.
|
|
|
|
use swactor::actor::ActorInterface;
|
|
use swactor::runtime::Ctx;
|
|
|
|
use super::frame::{Frame, StreamId};
|
|
use super::wire::{decode_delivery, DatastreamFrame};
|
|
|
|
/// Receives [`DatastreamFrame`] cluster messages and folds each decoded delivery
|
|
/// through `on_frame`. Spawn it, then publish its address under
|
|
/// [`DATASTREAM_SINK_NAME`] so emitters can resolve and ship to it.
|
|
pub struct DatastreamSink {
|
|
on_frame: Box<dyn FnMut(StreamId, Frame) + Send>,
|
|
}
|
|
|
|
/// The cluster name a [`DatastreamSink`] is published under. Emitters resolve
|
|
/// this to fill their `ClusterFrameSink` destination.
|
|
pub const DATASTREAM_SINK_NAME: &str = "datastream-sink";
|
|
|
|
impl DatastreamSink {
|
|
/// Build a sink that folds every decoded delivery through `on_frame`.
|
|
pub fn new(on_frame: impl FnMut(StreamId, Frame) + Send + 'static) -> Self {
|
|
Self {
|
|
on_frame: Box::new(on_frame),
|
|
}
|
|
}
|
|
}
|
|
|
|
impl ActorInterface for DatastreamSink {
|
|
type Incoming = DatastreamFrame;
|
|
type Response = ();
|
|
|
|
fn handle(&mut self, _ctx: &Ctx, msg: DatastreamFrame) {
|
|
// Best-effort: a malformed datagram is dropped, never panics the sink.
|
|
if let Ok((stream, frame)) = decode_delivery(&msg.payload) {
|
|
(self.on_frame)(stream, frame);
|
|
}
|
|
}
|
|
}
|