Commit graph

7 commits

Author SHA1 Message Date
ae9ca3bcf3 feat: datastream feature cleaning
Promote pipeline-parallel-inference to a first-class app and consolidate observability on the datastream wire, decoupling the dashboard crate from `distribution`.

- apps/pipeline-parallel-inference: move the example out of `examples/` into `apps/` as its own workspace, rename binaries to `pp-worker`/`pp-orchestrator`, and strip release binaries
- cluster: add `ClusterNode`, a synchronous facade over the actorized distribution protocol (IrohDriver + per-node Runtime hosting Swim/Registry/Metadata/Directory actors with a `MembershipFanout`), replacing ad-hoc `driver.node()`/`tick()` call sites
- fleet: add per-node fleet telemetry that ships identity/resource records as `DatastreamFrame`s over the cluster transport to the orchestrator's `DatastreamSink`, folded into a `FleetView` on a 3s tick
- provision: add best-effort, opt-in SSH boot-phase telemetry (`PP_DEPLOY_KEY`) that streams rented-node boot logs onto the orchestrator's datastream as `proc.boot.<stage>.*`
- dashboard: rewire the crate dependency from `distribution` to `datastream`, drop the standalone `swactor-datastream-dashboard` binary, and rewrite `datastream_source.rs` to demux per-node frames into Overview/Distribution/Fleet views with live-node TTL filtering
- distribution: refresh dist/netmap plugin copy and README from "Kademlia routing" to gossip-directory terminology

Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-09 13:29:07 +04:00
2748c86be6 feat(distribution): add telemetry datastream module
New datastream module (catalog/frame/mux/transport/ingest/store/views/wire) as a
per-node byte-channel dumb pipe. Dashboard datastream_source + datastream-dashboard
binary; node swactor-datastream-collector; docker demo compose + tests.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-05 11:25:43 +04:00
28bc3c0c95 chore: bump iroh to 0.98, drop vendored patch
Upgrades iroh/iroh-relay across datastore, distribution, node, and integration,
adapts iroh_driver to the new Endpoint::builder(Minimal).relay_mode(...) API, and
removes the vendored ed25519-dalek patch now that 0.98 resolves the upstream compile
errors. Adds the SIM_SPEC.md simulator MVP spec under pipeline-parallel-inference.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-22 11:06:58 +04:00
3fd7c624bd fix: reduce idle cpu, gossip noise, stability (#51)
Reduce idle cpu usage on my main machine from 17% to 1%. Made SWIM gossip more lazy.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-02-25 11:11:03 +00:00
19fabb707e refactor: consolidate crate functions (#50)
Remove co-dependencies for different modules found in `crates` and migrate the development history to a new repository. The docs were stale, and largely not getting used, so simply deleted for now. When code stabilizes more, they will become useful again.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-02-24 09:12:28 +00:00
03d40c42b6 refactor: consolidate crates (#39)
Crates continued to grow in number, but most are still quite small and feature specific. This refactor consolidates them.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-02-13 17:34:01 +00:00
c1feddcab4 feat: realize distribution crate (#33)
Stand up a runnable distribution stack on top of the core node logic.

- distribution: add NodeDriver bridging DistributedNode to real TCP I/O
  (TcpTransport/TcpAcceptor), translating NodeActions to/from wire messages;
  refine swim probe timing and transport wiring.
- node: new swactor-node binary (crates/node) hosting a real node over TCP.
- tests/docker: multi-host LAN cluster harness (compose, run-lan-cluster.sh,
  cluster + lan_cluster integration tests) exercising the full SWIM path.
- simulation: cluster_scenarios integration + distribution property coverage.
- docs: reorganize into distribution/, runtime/, diagrams/, connectome/; add
  DOCKER_REALIZATION + SIMULATION_TESTING realization notes.

Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-02-13 07:55:12 +00:00