swactor/crates/telemetry/src/lib.rs

93 lines
3.7 KiB
Rust
Raw Normal View History

//! The per-node telemetry **telemetry** (see `TELEMETRY_SPEC.md`).
//!
enforce datastream telemetry-only invariant: ban frame types from control code The datastream is metrics/logging only; control decisions must never branch on a frame. This was a recurring cultural problem with no structural enforcement. This change makes it a compile-time and CI-enforced fact. datastream crate (lib.rs): - Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root. is now a compile error (E0425). These types live only in datastream::frame::* and are documented as the observer surface. - Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain re-exported at root for producer-side callers. orchestration/app.rs: - Extracted all frame-touching code (CollectedDatastreamFrame, drain_datastream_connections, update_load_progress_from_frame, drain_frames, archive_collected_frame, pump, OrchDatastream, DashboardSupport) into two new observability modules: frame_collector.rs and orch_datastream.rs. - The orchestrator now interacts through a FrameCollector whose drain/drain_with_progress methods take closures; it never names Frame, DatastreamEvent, or CollectedDatastreamFrame. - StageLoadProgress (the one control-relevant signal previously scraped from frame payloads) is extracted inside FrameCollector and handed to the control loop as plain data. xtask: - New check-telemetry-isolation command scans control-plane modules (orchestration/, distribution/, data-plane/, provisioning/) for forbidden frame-type references and fails the build if any are found. Verified: workspace builds (myelin + dashboard feature), datastream 29 tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes clean. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
//! A deliberately dumb pipe: producers dump bytes tagged with a stream-local
2026-07-18 09:38:16 +00:00
//! channel id, a single per-node mux accepts those bytes and assigns canonical
//! positions during drain, the endpoint broadcasts catalog-aware events to
//! subscribers, ingest reconstructs streams by position, and views are
//! read-time projections over stored frames. Nothing between a producer and a
//! view interprets the payload.
//!
enforce datastream telemetry-only invariant: ban frame types from control code The datastream is metrics/logging only; control decisions must never branch on a frame. This was a recurring cultural problem with no structural enforcement. This change makes it a compile-time and CI-enforced fact. datastream crate (lib.rs): - Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root. is now a compile error (E0425). These types live only in datastream::frame::* and are documented as the observer surface. - Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain re-exported at root for producer-side callers. orchestration/app.rs: - Extracted all frame-touching code (CollectedDatastreamFrame, drain_datastream_connections, update_load_progress_from_frame, drain_frames, archive_collected_frame, pump, OrchDatastream, DashboardSupport) into two new observability modules: frame_collector.rs and orch_datastream.rs. - The orchestrator now interacts through a FrameCollector whose drain/drain_with_progress methods take closures; it never names Frame, DatastreamEvent, or CollectedDatastreamFrame. - StageLoadProgress (the one control-relevant signal previously scraped from frame payloads) is extracted inside FrameCollector and handed to the control loop as plain data. xtask: - New check-telemetry-isolation command scans control-plane modules (orchestration/, distribution/, data-plane/, provisioning/) for forbidden frame-type references and fails the build if any are found. Verified: workspace builds (myelin + dashboard feature), datastream 29 tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes clean. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
//! ## Producer vs observer surface
//!
//! This crate has two surfaces:
//!
//! - **Producer** — re-exported at the crate root ([`TelemetryEndpoint`],
//! [`TelemetryProducer`], [`Record`], [`ChannelId`], [`StreamId`], …).
enforce datastream telemetry-only invariant: ban frame types from control code The datastream is metrics/logging only; control decisions must never branch on a frame. This was a recurring cultural problem with no structural enforcement. This change makes it a compile-time and CI-enforced fact. datastream crate (lib.rs): - Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root. is now a compile error (E0425). These types live only in datastream::frame::* and are documented as the observer surface. - Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain re-exported at root for producer-side callers. orchestration/app.rs: - Extracted all frame-touching code (CollectedDatastreamFrame, drain_datastream_connections, update_load_progress_from_frame, drain_frames, archive_collected_frame, pump, OrchDatastream, DashboardSupport) into two new observability modules: frame_collector.rs and orch_datastream.rs. - The orchestrator now interacts through a FrameCollector whose drain/drain_with_progress methods take closures; it never names Frame, DatastreamEvent, or CollectedDatastreamFrame. - StageLoadProgress (the one control-relevant signal previously scraped from frame payloads) is extracted inside FrameCollector and handed to the control loop as plain data. xtask: - New check-telemetry-isolation command scans control-plane modules (orchestration/, distribution/, data-plane/, provisioning/) for forbidden frame-type references and fails the build if any are found. Verified: workspace builds (myelin + dashboard feature), datastream 29 tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes clean. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
//! Everything control-plane and actor code needs to *emit* telemetry.
//!
//! - **Observer** — in submodules ([`frame::Frame`], [`frame::TelemetryEvent`],
enforce datastream telemetry-only invariant: ban frame types from control code The datastream is metrics/logging only; control decisions must never branch on a frame. This was a recurring cultural problem with no structural enforcement. This change makes it a compile-time and CI-enforced fact. datastream crate (lib.rs): - Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root. is now a compile error (E0425). These types live only in datastream::frame::* and are documented as the observer surface. - Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain re-exported at root for producer-side callers. orchestration/app.rs: - Extracted all frame-touching code (CollectedDatastreamFrame, drain_datastream_connections, update_load_progress_from_frame, drain_frames, archive_collected_frame, pump, OrchDatastream, DashboardSupport) into two new observability modules: frame_collector.rs and orch_datastream.rs. - The orchestrator now interacts through a FrameCollector whose drain/drain_with_progress methods take closures; it never names Frame, DatastreamEvent, or CollectedDatastreamFrame. - StageLoadProgress (the one control-relevant signal previously scraped from frame payloads) is extracted inside FrameCollector and handed to the control loop as plain data. xtask: - New check-telemetry-isolation command scans control-plane modules (orchestration/, distribution/, data-plane/, provisioning/) for forbidden frame-type references and fails the build if any are found. Verified: workspace builds (myelin + dashboard feature), datastream 29 tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes clean. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
//! [`store::Store`], [`views`], [`ingest::Consumer`]). Everything a sink
//! (dashboard, archive, transport) needs to *read* telemetry.
//!
//! The crate root deliberately does **not** re-export [`frame::Frame`] or
//! [`frame::TelemetryEvent`]. `use telemetry::Frame` is a compile error; the
//! full path `telemetry::frame::Frame` compiles but is banned in control-plane
enforce datastream telemetry-only invariant: ban frame types from control code The datastream is metrics/logging only; control decisions must never branch on a frame. This was a recurring cultural problem with no structural enforcement. This change makes it a compile-time and CI-enforced fact. datastream crate (lib.rs): - Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root. is now a compile error (E0425). These types live only in datastream::frame::* and are documented as the observer surface. - Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain re-exported at root for producer-side callers. orchestration/app.rs: - Extracted all frame-touching code (CollectedDatastreamFrame, drain_datastream_connections, update_load_progress_from_frame, drain_frames, archive_collected_frame, pump, OrchDatastream, DashboardSupport) into two new observability modules: frame_collector.rs and orch_datastream.rs. - The orchestrator now interacts through a FrameCollector whose drain/drain_with_progress methods take closures; it never names Frame, DatastreamEvent, or CollectedDatastreamFrame. - StageLoadProgress (the one control-relevant signal previously scraped from frame payloads) is extracted inside FrameCollector and handed to the control loop as plain data. xtask: - New check-telemetry-isolation command scans control-plane modules (orchestration/, distribution/, data-plane/, provisioning/) for forbidden frame-type references and fails the build if any are found. Verified: workspace builds (myelin + dashboard feature), datastream 29 tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes clean. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
//! modules by `cargo xtask check-telemetry-isolation`.
//!
//! ```text
//! producers (caller-owned records + text)
2026-07-18 09:38:16 +00:00
//! │ bytes tagged by registered ChannelId
//! ▼
2026-07-18 09:38:16 +00:00
//! endpoint / catalog → [`endpoint`]
//! │ channel metadata + producer handles
//! ▼
2026-07-18 09:38:16 +00:00
//! per-node MUX → [`mux::Mux`]
//! │ positioned [`frame::Frame`]s
//! ▼
2026-07-18 09:38:16 +00:00
//! endpoint fanout → [`endpoint::DeliveryFanout`]
//! │ catalog-aware events, maybe dropped per subscriber
//! ▼
2026-07-18 09:38:16 +00:00
//! ingest / store → [`ingest`], [`store`]
//! │ position-keyed frame truth
//! ▼
2026-07-18 09:38:16 +00:00
//! views → [`views`]
//! ```
//!
2026-07-18 09:38:16 +00:00
//! The data model ([`frame`]), extension contract ([`record`]), endpoint/fanout
//! seam ([`endpoint`]), and compatibility wire helpers ([`wire`]) are the seams
//! tests observe. Channel meanings live in producer/consumer crates, not in a
//! telemetry-wide global registry.
pub mod emit;
pub mod endpoint;
pub mod frame;
pub mod hardware;
pub mod health;
pub mod ingest;
pub mod mux;
pub mod publisher_actor;
pub mod record;
pub mod sink_actor;
pub mod store;
pub mod transport;
pub mod views;
pub mod wire;
enforce datastream telemetry-only invariant: ban frame types from control code The datastream is metrics/logging only; control decisions must never branch on a frame. This was a recurring cultural problem with no structural enforcement. This change makes it a compile-time and CI-enforced fact. datastream crate (lib.rs): - Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root. is now a compile error (E0425). These types live only in datastream::frame::* and are documented as the observer surface. - Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain re-exported at root for producer-side callers. orchestration/app.rs: - Extracted all frame-touching code (CollectedDatastreamFrame, drain_datastream_connections, update_load_progress_from_frame, drain_frames, archive_collected_frame, pump, OrchDatastream, DashboardSupport) into two new observability modules: frame_collector.rs and orch_datastream.rs. - The orchestrator now interacts through a FrameCollector whose drain/drain_with_progress methods take closures; it never names Frame, DatastreamEvent, or CollectedDatastreamFrame. - StageLoadProgress (the one control-relevant signal previously scraped from frame payloads) is extracted inside FrameCollector and handed to the control loop as plain data. xtask: - New check-telemetry-isolation command scans control-plane modules (orchestration/, distribution/, data-plane/, provisioning/) for forbidden frame-type references and fails the build if any are found. Verified: workspace builds (myelin + dashboard feature), datastream 29 tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes clean. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
// ── Producer surface (re-exported at root; safe for control-plane code) ──
pub use endpoint::{
CatalogSnapshot, ChannelRegistrationError, TelemetryEndpoint, TelemetryProducer,
TelemetrySnapshot, TelemetrySubscription, DeliveryFanout, EndpointTick, SubscriberSnapshot,
enforce datastream telemetry-only invariant: ban frame types from control code The datastream is metrics/logging only; control decisions must never branch on a frame. This was a recurring cultural problem with no structural enforcement. This change makes it a compile-time and CI-enforced fact. datastream crate (lib.rs): - Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root. is now a compile error (E0425). These types live only in datastream::frame::* and are documented as the observer surface. - Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain re-exported at root for producer-side callers. orchestration/app.rs: - Extracted all frame-touching code (CollectedDatastreamFrame, drain_datastream_connections, update_load_progress_from_frame, drain_frames, archive_collected_frame, pump, OrchDatastream, DashboardSupport) into two new observability modules: frame_collector.rs and orch_datastream.rs. - The orchestrator now interacts through a FrameCollector whose drain/drain_with_progress methods take closures; it never names Frame, DatastreamEvent, or CollectedDatastreamFrame. - StageLoadProgress (the one control-relevant signal previously scraped from frame payloads) is extracted inside FrameCollector and handed to the control loop as plain data. xtask: - New check-telemetry-isolation command scans control-plane modules (orchestration/, distribution/, data-plane/, provisioning/) for forbidden frame-type references and fails the build if any are found. Verified: workspace builds (myelin + dashboard feature), datastream 29 tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes clean. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
SubscriptionId,
};
pub use frame::{
ChannelContent, ChannelContentKind, ChannelDescriptor, ChannelFilter, ChannelId, ChannelRef,
enforce datastream telemetry-only invariant: ban frame types from control code The datastream is metrics/logging only; control decisions must never branch on a frame. This was a recurring cultural problem with no structural enforcement. This change makes it a compile-time and CI-enforced fact. datastream crate (lib.rs): - Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root. is now a compile error (E0425). These types live only in datastream::frame::* and are documented as the observer surface. - Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain re-exported at root for producer-side callers. orchestration/app.rs: - Extracted all frame-touching code (CollectedDatastreamFrame, drain_datastream_connections, update_load_progress_from_frame, drain_frames, archive_collected_frame, pump, OrchDatastream, DashboardSupport) into two new observability modules: frame_collector.rs and orch_datastream.rs. - The orchestrator now interacts through a FrameCollector whose drain/drain_with_progress methods take closures; it never names Frame, DatastreamEvent, or CollectedDatastreamFrame. - StageLoadProgress (the one control-relevant signal previously scraped from frame payloads) is extracted inside FrameCollector and handed to the control loop as plain data. xtask: - New check-telemetry-isolation command scans control-plane modules (orchestration/, distribution/, data-plane/, provisioning/) for forbidden frame-type references and fails the build if any are found. Verified: workspace builds (myelin + dashboard feature), datastream 29 tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes clean. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
Lifetime, NodeId, Position, SourceFilter, StreamDescriptor, StreamId, StreamOrigin,
SubscriptionRequest,
};
pub use mux::Mux;
pub use publisher_actor::{
TELEMETRY_PUBLISHER_NAME, TelemetryPublisherActor, TelemetryPublisherMsg,
TelemetrySubscribe, register_telemetry_publisher_codec,
};
pub use record::{ChannelKind, ChannelRegistry, Record};
pub use sink_actor::{TELEMETRY_SINK_NAME, TelemetrySink};
enforce datastream telemetry-only invariant: ban frame types from control code The datastream is metrics/logging only; control decisions must never branch on a frame. This was a recurring cultural problem with no structural enforcement. This change makes it a compile-time and CI-enforced fact. datastream crate (lib.rs): - Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root. is now a compile error (E0425). These types live only in datastream::frame::* and are documented as the observer surface. - Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain re-exported at root for producer-side callers. orchestration/app.rs: - Extracted all frame-touching code (CollectedDatastreamFrame, drain_datastream_connections, update_load_progress_from_frame, drain_frames, archive_collected_frame, pump, OrchDatastream, DashboardSupport) into two new observability modules: frame_collector.rs and orch_datastream.rs. - The orchestrator now interacts through a FrameCollector whose drain/drain_with_progress methods take closures; it never names Frame, DatastreamEvent, or CollectedDatastreamFrame. - StageLoadProgress (the one control-relevant signal previously scraped from frame payloads) is extracted inside FrameCollector and handed to the control loop as plain data. xtask: - New check-telemetry-isolation command scans control-plane modules (orchestration/, distribution/, data-plane/, provisioning/) for forbidden frame-type references and fails the build if any are found. Verified: workspace builds (myelin + dashboard feature), datastream 29 tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes clean. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
// ── Observer surface (in submodules; NOT re-exported at root) ──
//
// frame::Frame, frame::TelemetryEvent, frame::FrameDelivery,
enforce datastream telemetry-only invariant: ban frame types from control code The datastream is metrics/logging only; control decisions must never branch on a frame. This was a recurring cultural problem with no structural enforcement. This change makes it a compile-time and CI-enforced fact. datastream crate (lib.rs): - Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root. is now a compile error (E0425). These types live only in datastream::frame::* and are documented as the observer surface. - Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain re-exported at root for producer-side callers. orchestration/app.rs: - Extracted all frame-touching code (CollectedDatastreamFrame, drain_datastream_connections, update_load_progress_from_frame, drain_frames, archive_collected_frame, pump, OrchDatastream, DashboardSupport) into two new observability modules: frame_collector.rs and orch_datastream.rs. - The orchestrator now interacts through a FrameCollector whose drain/drain_with_progress methods take closures; it never names Frame, DatastreamEvent, or CollectedDatastreamFrame. - StageLoadProgress (the one control-relevant signal previously scraped from frame payloads) is extracted inside FrameCollector and handed to the control loop as plain data. xtask: - New check-telemetry-isolation command scans control-plane modules (orchestration/, distribution/, data-plane/, provisioning/) for forbidden frame-type references and fails the build if any are found. Verified: workspace builds (myelin + dashboard feature), datastream 29 tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes clean. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
// store::Store, ingest::Consumer, views::*, transport::Delivery
//
// Access these via their module paths (e.g. `telemetry::frame::Frame`).
enforce datastream telemetry-only invariant: ban frame types from control code The datastream is metrics/logging only; control decisions must never branch on a frame. This was a recurring cultural problem with no structural enforcement. This change makes it a compile-time and CI-enforced fact. datastream crate (lib.rs): - Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root. is now a compile error (E0425). These types live only in datastream::frame::* and are documented as the observer surface. - Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain re-exported at root for producer-side callers. orchestration/app.rs: - Extracted all frame-touching code (CollectedDatastreamFrame, drain_datastream_connections, update_load_progress_from_frame, drain_frames, archive_collected_frame, pump, OrchDatastream, DashboardSupport) into two new observability modules: frame_collector.rs and orch_datastream.rs. - The orchestrator now interacts through a FrameCollector whose drain/drain_with_progress methods take closures; it never names Frame, DatastreamEvent, or CollectedDatastreamFrame. - StageLoadProgress (the one control-relevant signal previously scraped from frame payloads) is extracted inside FrameCollector and handed to the control loop as plain data. xtask: - New check-telemetry-isolation command scans control-plane modules (orchestration/, distribution/, data-plane/, provisioning/) for forbidden frame-type references and fails the build if any are found. Verified: workspace builds (myelin + dashboard feature), datastream 29 tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes clean. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
// Control-plane modules must not import them — enforced by CI.