Zachery Aaron Shores-Chmielewski
b06583f598
Trim the datastream crate to its dumb-pipe core: drop the frame-timing sidecar, make mux positions gap-free for accepted frames, and simplify the endpoint fanout. - mux: defer position assignment from submit to drain so an overflowed submission no longer consumes a position (no synthetic gaps); submit now returns bool and the Mutex<Receiver> is removed since positions are assigned only to accepted frames - endpoint/mux: switch from std::sync::mpsc to crossbeam-channel and drop per-event event_matches_request filtering — request filters now apply only to the initial catalog snapshot, and future events broadcast to all subscribers - endpoint (DeliveryFanout): snapshot sender handles under the lock and deliver outside it via FanoutTarget/FanoutReport, so large batches or slow subscribers no longer block subscribe/snapshot control-plane ops - emit/endpoint/producer: drop set_frame_timing_enabled/frame_timing_enabled and the Position return from submit_record/submit_text/submit_bytes, and add submit_text_owned taking owned String - timing/lib/spec: delete the timing module and FRAME_TIME_CHANNEL/FRAME_TIME_CHANNEL_ID/FrameTimeSample re-exports (including the auto-registered timing channel in ChannelCatalogState) and renumber the DATASTREAM_SPEC.md section references across frame/ingest/store/mux - tests: remove the 567-line shared datastream_support/mod.rs harness Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
69 lines
2.8 KiB
Rust
69 lines
2.8 KiB
Rust
//! The per-node telemetry **datastream** (see `DATASTREAM_SPEC.md`).
|
|
//!
|
|
//! A deliberately dumb pipe: producers dump bytes tagged by stream-local
|
|
//! channel id, a single per-node mux accepts those bytes and assigns canonical
|
|
//! positions during drain, the endpoint broadcasts catalog-aware events to
|
|
//! subscribers, ingest reconstructs streams by position, and views are
|
|
//! read-time projections over stored frames. Nothing between a producer and a
|
|
//! view interprets the payload.
|
|
//!
|
|
//! ```text
|
|
//! producers (caller-owned records + text)
|
|
//! │ bytes tagged by registered ChannelId
|
|
//! ▼
|
|
//! endpoint / catalog → [`endpoint`]
|
|
//! │ channel metadata + producer handles
|
|
//! ▼
|
|
//! per-node MUX → [`mux::Mux`]
|
|
//! │ positioned [`frame::Frame`]s
|
|
//! ▼
|
|
//! endpoint fanout → [`endpoint::DeliveryFanout`]
|
|
//! │ catalog-aware events, maybe dropped per subscriber
|
|
//! ▼
|
|
//! ingest / store → [`ingest`], [`store`]
|
|
//! │ position-keyed frame truth
|
|
//! ▼
|
|
//! views → [`views`]
|
|
//! ```
|
|
//!
|
|
//! The data model ([`frame`]), extension contract ([`record`]), endpoint/fanout
|
|
//! seam ([`endpoint`]), and compatibility wire helpers ([`wire`]) are the seams
|
|
//! tests observe. Channel meanings live in producer/consumer crates, not in a
|
|
//! datastream-wide global registry.
|
|
|
|
pub mod emit;
|
|
pub mod endpoint;
|
|
pub mod frame;
|
|
pub mod hardware;
|
|
pub mod health;
|
|
pub mod ingest;
|
|
pub mod mux;
|
|
pub mod publisher_actor;
|
|
pub mod record;
|
|
pub mod sink_actor;
|
|
pub mod store;
|
|
pub mod transport;
|
|
pub mod views;
|
|
pub mod wire;
|
|
|
|
pub use endpoint::{
|
|
CatalogSnapshot, ChannelRegistrationError, DatastreamEndpoint, DatastreamProducer,
|
|
DatastreamSnapshot, DatastreamSubscription, DeliveryFanout, EndpointTick, SubscriberSnapshot,
|
|
SubscriptionId, frame_event_to_delivery,
|
|
};
|
|
pub use frame::{
|
|
ChannelContent, ChannelContentKind, ChannelDescriptor, ChannelFilter, ChannelId, ChannelRef,
|
|
DatastreamEvent, Frame, FrameDelivery, Lifetime, NodeId, Position, SourceFilter,
|
|
StreamDescriptor, StreamId, StreamOrigin, SubscriptionRequest,
|
|
};
|
|
pub use ingest::Consumer;
|
|
pub use mux::Mux;
|
|
pub use publisher_actor::{
|
|
DATASTREAM_PUBLISHER_NAME, DatastreamPublisherActor, DatastreamPublisherMsg,
|
|
DatastreamSubscribe, register_datastream_publisher_codec,
|
|
};
|
|
pub use record::{ChannelKind, ChannelRegistry, Record};
|
|
pub use sink_actor::{DATASTREAM_SINK_NAME, DatastreamSink};
|
|
pub use store::{GapSpan, Store, StoredStream};
|
|
pub use transport::{Delivery, Reorder, ScriptedTransport, StreamScript};
|
|
pub use views::{Body, LogEntry, MergedFrame};
|