zacheryasc/swactor
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
swactor/crates/telemetry/Cargo.toml

26 lines
539 B
TOML
Raw Permalink Normal View History

feat: datastream feature cleaning Promote pipeline-parallel-inference to a first-class app and consolidate observability on the datastream wire, decoupling the dashboard crate from `distribution`. - apps/pipeline-parallel-inference: move the example out of `examples/` into `apps/` as its own workspace, rename binaries to `pp-worker`/`pp-orchestrator`, and strip release binaries - cluster: add `ClusterNode`, a synchronous facade over the actorized distribution protocol (IrohDriver + per-node Runtime hosting Swim/Registry/Metadata/Directory actors with a `MembershipFanout`), replacing ad-hoc `driver.node()`/`tick()` call sites - fleet: add per-node fleet telemetry that ships identity/resource records as `DatastreamFrame`s over the cluster transport to the orchestrator's `DatastreamSink`, folded into a `FleetView` on a 3s tick - provision: add best-effort, opt-in SSH boot-phase telemetry (`PP_DEPLOY_KEY`) that streams rented-node boot logs onto the orchestrator's datastream as `proc.boot.<stage>.*` - dashboard: rewire the crate dependency from `distribution` to `datastream`, drop the standalone `swactor-datastream-dashboard` binary, and rewrite `datastream_source.rs` to demux per-node frames into Overview/Distribution/Fleet views with live-node TTL filtering - distribution: refresh dist/netmap plugin copy and README from "Kademlia routing" to gossip-directory terminology Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-09 09:29:07 +00:00
[package]
refactor: `datastream` crate is now `telemetry` The crate is the per-node metrics/logging pipe with a universal subscriber endpoint, but "datastream" kept getting misread as a general messaging plane. Rename crate, module paths, and public API (`DatastreamEndpoint` → `TelemetryEndpoint`, etc.) so misuse is visible on sight. Renamed contracts (all in-repo producers/consumers migrated): - env vars `MYELIN_DATASTREAM_*` → `MYELIN_TELEMETRY_*` - artifact `datastream.ndjson` → `telemetry.ndjson` - actor names `telemetry-publisher` / `telemetry-sink` - wire ALPN `swactor/telemetry/0` - `DATASTREAM_SPEC.md` → `TELEMETRY_SPEC.md` Also fixes two latent test breaks: `process` and `iroh-driver` tests imported `DatastreamEvent` from the crate root, which was never re-exported; they now use the observer path `telemetry::frame::`.
2026-08-15 08:17:48 +00:00
name = "telemetry"
feat: datastream feature cleaning Promote pipeline-parallel-inference to a first-class app and consolidate observability on the datastream wire, decoupling the dashboard crate from `distribution`. - apps/pipeline-parallel-inference: move the example out of `examples/` into `apps/` as its own workspace, rename binaries to `pp-worker`/`pp-orchestrator`, and strip release binaries - cluster: add `ClusterNode`, a synchronous facade over the actorized distribution protocol (IrohDriver + per-node Runtime hosting Swim/Registry/Metadata/Directory actors with a `MembershipFanout`), replacing ad-hoc `driver.node()`/`tick()` call sites - fleet: add per-node fleet telemetry that ships identity/resource records as `DatastreamFrame`s over the cluster transport to the orchestrator's `DatastreamSink`, folded into a `FleetView` on a 3s tick - provision: add best-effort, opt-in SSH boot-phase telemetry (`PP_DEPLOY_KEY`) that streams rented-node boot logs onto the orchestrator's datastream as `proc.boot.<stage>.*` - dashboard: rewire the crate dependency from `distribution` to `datastream`, drop the standalone `swactor-datastream-dashboard` binary, and rewrite `datastream_source.rs` to demux per-node frames into Overview/Distribution/Fleet views with live-node TTL filtering - distribution: refresh dist/netmap plugin copy and README from "Kademlia routing" to gossip-directory terminology Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-09 09:29:07 +00:00
version = "0.1.0"
edition = "2024"
feat: proper `README`, `LICENSE`, and our first quick example Add project licensing and a full README, plus a minimal WebAssembly ping-pong example that exercises the actor runtime. - `LICENSE`: add the full GNU AGPL-3.0 text and set `license = "AGPL-3.0-only"` on every package (`swactor`, each crate, `apps/myelin`, `tools/vastai`, `xtask`) - `README.md`: rewrite from a stub into a full project overview, covering features (actor_id routing, WASM, iroh QUIC/SWIM, OTP-style std, process manager, zero-copy objects, datastream metrics, dashboard), architecture, examples, developing, status, and license - `examples/ping-pong`: new standalone workspace (`pingpong` cdylib) where two actors volley on a single-threaded `wasm` runtime driven by a Node host via `tick()`, demonstrating spawning, message passing, and death monitoring (`watching`) - `examples/ping-pong`: add a host/run harness -- `run.sh` (wasm-pack build + `run.mjs`), `serve.sh` (dashboard + static demo), `index.html`, and a pinned `Cargo.lock` - `.gitignore`: stop ignoring `.loop/`, `.deployment-notes/`, and `.omp/` Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-05 20:29:27 +00:00
license = "AGPL-3.0-only"
feat: datastream feature cleaning Promote pipeline-parallel-inference to a first-class app and consolidate observability on the datastream wire, decoupling the dashboard crate from `distribution`. - apps/pipeline-parallel-inference: move the example out of `examples/` into `apps/` as its own workspace, rename binaries to `pp-worker`/`pp-orchestrator`, and strip release binaries - cluster: add `ClusterNode`, a synchronous facade over the actorized distribution protocol (IrohDriver + per-node Runtime hosting Swim/Registry/Metadata/Directory actors with a `MembershipFanout`), replacing ad-hoc `driver.node()`/`tick()` call sites - fleet: add per-node fleet telemetry that ships identity/resource records as `DatastreamFrame`s over the cluster transport to the orchestrator's `DatastreamSink`, folded into a `FleetView` on a 3s tick - provision: add best-effort, opt-in SSH boot-phase telemetry (`PP_DEPLOY_KEY`) that streams rented-node boot logs onto the orchestrator's datastream as `proc.boot.<stage>.*` - dashboard: rewire the crate dependency from `distribution` to `datastream`, drop the standalone `swactor-datastream-dashboard` binary, and rewrite `datastream_source.rs` to demux per-node frames into Overview/Distribution/Fleet views with live-node TTL filtering - distribution: refresh dist/netmap plugin copy and README from "Kademlia routing" to gossip-directory terminology Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-09 09:29:07 +00:00
[dependencies]
swactor = { path = "../..", features = ["serde", "transport"] }
swactor-transport = { path = "../transport" }
fix(dashboard): improve control and hardware views Refine fleet and provisioning interactions, expand control-plane projections, and add CPU, memory, pressure, and storage telemetry for the local dashboard surfaces.
2026-08-22 17:17:42 +00:00
swactor-engine = { path = "../engine" }
futures-channel = "0.3"
feat: datastream feature cleaning Promote pipeline-parallel-inference to a first-class app and consolidate observability on the datastream wire, decoupling the dashboard crate from `distribution`. - apps/pipeline-parallel-inference: move the example out of `examples/` into `apps/` as its own workspace, rename binaries to `pp-worker`/`pp-orchestrator`, and strip release binaries - cluster: add `ClusterNode`, a synchronous facade over the actorized distribution protocol (IrohDriver + per-node Runtime hosting Swim/Registry/Metadata/Directory actors with a `MembershipFanout`), replacing ad-hoc `driver.node()`/`tick()` call sites - fleet: add per-node fleet telemetry that ships identity/resource records as `DatastreamFrame`s over the cluster transport to the orchestrator's `DatastreamSink`, folded into a `FleetView` on a 3s tick - provision: add best-effort, opt-in SSH boot-phase telemetry (`PP_DEPLOY_KEY`) that streams rented-node boot logs onto the orchestrator's datastream as `proc.boot.<stage>.*` - dashboard: rewire the crate dependency from `distribution` to `datastream`, drop the standalone `swactor-datastream-dashboard` binary, and rewrite `datastream_source.rs` to demux per-node frames into Overview/Distribution/Fleet views with live-node TTL filtering - distribution: refresh dist/netmap plugin copy and README from "Kademlia routing" to gossip-directory terminology Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-09 09:29:07 +00:00
serde = { version = "1", features = ["derive"] }
serde_json = "1"
runtime: checkpoint distributed execution and retained-node deployment Integrate namespace and source-route lifecycle changes, contextual process cleanup, Python binding updates, and Myelin worker/orchestrator recovery. Keep the shared control contracts, deployment identity fencing, SSH bootstrap adapters, paid admission accounting, and VastAI cleanup implementation together with their consumers. Migrate Iroh dependencies and telemetry transport/collection with dashboard and demo callsites, workspace build configuration, and actor-control-flow policy updates. This is an intermediate development checkpoint, not paid-provider qualification. Review verification: contextual_process_guarantees (4 tests), telemetry_transport (4 tests), and shared control contracts (5 tests) passed. Historical five-node redeployment and campaign execution passed individually; complete ordered qualification remains pending.
2026-09-14 09:20:56 +00:00
rmp-serde = "1"
iroh-base.workspace = true
refactor(datastream): cleanup Trim the datastream crate to its dumb-pipe core: drop the frame-timing sidecar, make mux positions gap-free for accepted frames, and simplify the endpoint fanout. - mux: defer position assignment from submit to drain so an overflowed submission no longer consumes a position (no synthetic gaps); submit now returns bool and the Mutex<Receiver> is removed since positions are assigned only to accepted frames - endpoint/mux: switch from std::sync::mpsc to crossbeam-channel and drop per-event event_matches_request filtering — request filters now apply only to the initial catalog snapshot, and future events broadcast to all subscribers - endpoint (DeliveryFanout): snapshot sender handles under the lock and deliver outside it via FanoutTarget/FanoutReport, so large batches or slow subscribers no longer block subscribe/snapshot control-plane ops - emit/endpoint/producer: drop set_frame_timing_enabled/frame_timing_enabled and the Position return from submit_record/submit_text/submit_bytes, and add submit_text_owned taking owned String - timing/lib/spec: delete the timing module and FRAME_TIME_CHANNEL/FRAME_TIME_CHANNEL_ID/FrameTimeSample re-exports (including the auto-registered timing channel in ChannelCatalogState) and renumber the DATASTREAM_SPEC.md section references across frame/ingest/store/mux - tests: remove the 567-line shared datastream_support/mod.rs harness Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-18 09:38:16 +00:00
crossbeam-channel = "0.5"
feat: datastream feature cleaning Promote pipeline-parallel-inference to a first-class app and consolidate observability on the datastream wire, decoupling the dashboard crate from `distribution`. - apps/pipeline-parallel-inference: move the example out of `examples/` into `apps/` as its own workspace, rename binaries to `pp-worker`/`pp-orchestrator`, and strip release binaries - cluster: add `ClusterNode`, a synchronous facade over the actorized distribution protocol (IrohDriver + per-node Runtime hosting Swim/Registry/Metadata/Directory actors with a `MembershipFanout`), replacing ad-hoc `driver.node()`/`tick()` call sites - fleet: add per-node fleet telemetry that ships identity/resource records as `DatastreamFrame`s over the cluster transport to the orchestrator's `DatastreamSink`, folded into a `FleetView` on a 3s tick - provision: add best-effort, opt-in SSH boot-phase telemetry (`PP_DEPLOY_KEY`) that streams rented-node boot logs onto the orchestrator's datastream as `proc.boot.<stage>.*` - dashboard: rewire the crate dependency from `distribution` to `datastream`, drop the standalone `swactor-datastream-dashboard` binary, and rewrite `datastream_source.rs` to demux per-node frames into Overview/Distribution/Fleet views with live-node TTL filtering - distribution: refresh dist/netmap plugin copy and README from "Kademlia routing" to gossip-directory terminology Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-09 09:29:07 +00:00
[target.'cfg(target_os = "linux")'.dependencies]
libc = "0.2"
fix(dashboard): improve control and hardware views Refine fleet and provisioning interactions, expand control-plane projections, and add CPU, memory, pressure, and storage telemetry for the local dashboard surfaces.
2026-08-22 17:17:42 +00:00
perf(telemetry-codec): harden contracts and add benchmarks Reject duplicate codec registrations atomically, add stateful invariant suites, prune obsolete tests, and introduce engine-neutral benchmark workloads behind a thin Criterion adapter.
2026-08-24 09:51:42 +00:00
[dev-dependencies]
proptest = "1"
fix(dashboard): improve control and hardware views Refine fleet and provisioning interactions, expand control-plane projections, and add CPU, memory, pressure, and storage telemetry for the local dashboard surfaces.
2026-08-22 17:17:42 +00:00
[lints]
workspace = true
Reference in a new issue Copy permalink