zacheryasc/swactor
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
swactor/examples/pipeline-parallel-inference/Cargo.toml

43 lines
1.2 KiB
TOML
Raw Normal View History

feat: two-node pipeline parallel example Extend the single-GPU example into a two-node pipeline-parallel run that splits llama3.2:1b across two rented vast.ai GPUs and closes the autoregressive decode loop over iroh. - topology: add linear-chain helpers where each stage derives its neighbours locally from `STAGE`/`NUM_STAGES`, registering `pp-entry`/`pp-exit`/`pp-stage-{i}` SWIM names - messages: add `StageActivation` (bf16 hidden-state hand-off carrying position/seq_len/is_prefill) and `NextToken` (sampled-token feedback with a `done` flag) that close the autoregressive loop between stage 0 and stage 1 - stage_actor: add `Stage0Actor` (tokenize -> embed_and_forward -> prefill activation; decode_step on each NextToken) and `Stage1Actor` (forward_and_sample -> NextToken back; emit InferenceResponse on EOS/max_tokens) - vastai: fork the client and add `create_pipeline_instances` (rents one instance per stage, threading `STAGE`/`NUM_STAGES`, best-effort destroys on partial failure) and `destroy_all_instances` - pp_tinygrad_worker.py: per-stage worker slicing `model.blk[start:end]` in stub and real (GGUF) modes, plus new `pp_gpu_node`/`pp_smoke_run` binaries and ROADMAP/SPEC/TEST_SPEC docs - reuse: build on the single-GPU example's iroh transport and process bridge unchanged; add actor/codec/topology/integration test suites Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-16 05:49:43 +00:00
[workspace]
[package]
name = "pipeline-parallel-inference"
version = "0.1.0"
edition = "2024"
publish = false
[dependencies]
swactor = { path = "../..", features = ["transport", "serde", "std"] }
swactor-process = { path = "../../crates/process" }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
refactor: rename pp binaries and rewrite orchestrator pp_smoke_run->pp_orchestrator (seed/vastai/hold/teardown, N stages), pp_gpu_node->pp_worker. fleet_plugin subscribes collector SSE + folds locally; iroh_driver drops verbose eprintln connection/relay logging. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-30 07:51:22 +00:00
reqwest = { version = "0.12", features = ["json", "stream"] }
futures-util = "0.3"
feat: two-node pipeline parallel example Extend the single-GPU example into a two-node pipeline-parallel run that splits llama3.2:1b across two rented vast.ai GPUs and closes the autoregressive decode loop over iroh. - topology: add linear-chain helpers where each stage derives its neighbours locally from `STAGE`/`NUM_STAGES`, registering `pp-entry`/`pp-exit`/`pp-stage-{i}` SWIM names - messages: add `StageActivation` (bf16 hidden-state hand-off carrying position/seq_len/is_prefill) and `NextToken` (sampled-token feedback with a `done` flag) that close the autoregressive loop between stage 0 and stage 1 - stage_actor: add `Stage0Actor` (tokenize -> embed_and_forward -> prefill activation; decode_step on each NextToken) and `Stage1Actor` (forward_and_sample -> NextToken back; emit InferenceResponse on EOS/max_tokens) - vastai: fork the client and add `create_pipeline_instances` (rents one instance per stage, threading `STAGE`/`NUM_STAGES`, best-effort destroys on partial failure) and `destroy_all_instances` - pp_tinygrad_worker.py: per-stage worker slicing `model.blk[start:end]` in stub and real (GGUF) modes, plus new `pp_gpu_node`/`pp_smoke_run` binaries and ROADMAP/SPEC/TEST_SPEC docs - reuse: build on the single-GPU example's iroh transport and process bridge unchanged; add actor/codec/topology/integration test suites Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-16 05:49:43 +00:00
tokio = { version = "1", features = ["full"] }
refactor(distribution): prune diagnostics subsystem Strip the collector/aggregator/postproc/snapshot, vastai sampler+shipper, host/iroh/subprocess/swim introspection, relay observability, sink/spool, and the diag binaries; drop the t_diag_* tests. Remove DiagEvent emission from iroh_driver. Add datastream emit/wire (mux + NoopSink/UdpFrameSink/ClusterFrameSink) and rewire the dashboard onto datastream_source. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-06 17:53:25 +00:00
distribution = { path = "../../crates/distribution", features = ["iroh"] }
# Live runtime dashboard (HTTP overview/actors/topology pages + the
# datastream-fed Fleet tab, served on localhost when PP_DASHBOARD is set).
dashboard = { path = "../../crates/dashboard" }
feat(distribution): add n3 observability introspectors Relay/subprocess/registry introspectors, build-time dep-version + git-sha, postproc RTT/inference render, new swactor-iroh-relay binary; dashboard replay-viewer, sim SWIM tuning report + repro tests, N3 postmortem docs. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-25 18:19:06 +00:00
iroh = "0.98"
feat: two-node pipeline parallel example Extend the single-GPU example into a two-node pipeline-parallel run that splits llama3.2:1b across two rented vast.ai GPUs and closes the autoregressive decode loop over iroh. - topology: add linear-chain helpers where each stage derives its neighbours locally from `STAGE`/`NUM_STAGES`, registering `pp-entry`/`pp-exit`/`pp-stage-{i}` SWIM names - messages: add `StageActivation` (bf16 hidden-state hand-off carrying position/seq_len/is_prefill) and `NextToken` (sampled-token feedback with a `done` flag) that close the autoregressive loop between stage 0 and stage 1 - stage_actor: add `Stage0Actor` (tokenize -> embed_and_forward -> prefill activation; decode_step on each NextToken) and `Stage1Actor` (forward_and_sample -> NextToken back; emit InferenceResponse on EOS/max_tokens) - vastai: fork the client and add `create_pipeline_instances` (rents one instance per stage, threading `STAGE`/`NUM_STAGES`, best-effort destroys on partial failure) and `destroy_all_instances` - pp_tinygrad_worker.py: per-stage worker slicing `model.blk[start:end]` in stub and real (GGUF) modes, plus new `pp_gpu_node`/`pp_smoke_run` binaries and ROADMAP/SPEC/TEST_SPEC docs - reuse: build on the single-GPU example's iroh transport and process bridge unchanged; add actor/codec/topology/integration test suites Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-16 05:49:43 +00:00
urlencoding = "2"
base64 = "0.22"
feat(distribution): add diagnostics subsystem Structured observability for the iroh/SWIM layer: Aggregator, typed Event/Snapshot types, Sink (NoopSink default), ProbeScheduler, process stats, and host/iroh/swim introspection, plus the swactor-diag-collector, -postproc, and -iroh-relay binaries that assemble and render per-run bundles. Generalizes the pipeline-parallel-inference example to N stages and adds the topology-planner spec. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-20 07:41:30 +00:00
libc = "0.2"
feat: two-node pipeline parallel example Extend the single-GPU example into a two-node pipeline-parallel run that splits llama3.2:1b across two rented vast.ai GPUs and closes the autoregressive decode loop over iroh. - topology: add linear-chain helpers where each stage derives its neighbours locally from `STAGE`/`NUM_STAGES`, registering `pp-entry`/`pp-exit`/`pp-stage-{i}` SWIM names - messages: add `StageActivation` (bf16 hidden-state hand-off carrying position/seq_len/is_prefill) and `NextToken` (sampled-token feedback with a `done` flag) that close the autoregressive loop between stage 0 and stage 1 - stage_actor: add `Stage0Actor` (tokenize -> embed_and_forward -> prefill activation; decode_step on each NextToken) and `Stage1Actor` (forward_and_sample -> NextToken back; emit InferenceResponse on EOS/max_tokens) - vastai: fork the client and add `create_pipeline_instances` (rents one instance per stage, threading `STAGE`/`NUM_STAGES`, best-effort destroys on partial failure) and `destroy_all_instances` - pp_tinygrad_worker.py: per-stage worker slicing `model.blk[start:end]` in stub and real (GGUF) modes, plus new `pp_gpu_node`/`pp_smoke_run` binaries and ROADMAP/SPEC/TEST_SPEC docs - reuse: build on the single-GPU example's iroh transport and process bridge unchanged; add actor/codec/topology/integration test suites Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-16 05:49:43 +00:00
[[bin]]
refactor: rename pp binaries and rewrite orchestrator pp_smoke_run->pp_orchestrator (seed/vastai/hold/teardown, N stages), pp_gpu_node->pp_worker. fleet_plugin subscribes collector SSE + folds locally; iroh_driver drops verbose eprintln connection/relay logging. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-30 07:51:22 +00:00
name = "pp-worker"
path = "src/bin/pp_worker.rs"
feat: two-node pipeline parallel example Extend the single-GPU example into a two-node pipeline-parallel run that splits llama3.2:1b across two rented vast.ai GPUs and closes the autoregressive decode loop over iroh. - topology: add linear-chain helpers where each stage derives its neighbours locally from `STAGE`/`NUM_STAGES`, registering `pp-entry`/`pp-exit`/`pp-stage-{i}` SWIM names - messages: add `StageActivation` (bf16 hidden-state hand-off carrying position/seq_len/is_prefill) and `NextToken` (sampled-token feedback with a `done` flag) that close the autoregressive loop between stage 0 and stage 1 - stage_actor: add `Stage0Actor` (tokenize -> embed_and_forward -> prefill activation; decode_step on each NextToken) and `Stage1Actor` (forward_and_sample -> NextToken back; emit InferenceResponse on EOS/max_tokens) - vastai: fork the client and add `create_pipeline_instances` (rents one instance per stage, threading `STAGE`/`NUM_STAGES`, best-effort destroys on partial failure) and `destroy_all_instances` - pp_tinygrad_worker.py: per-stage worker slicing `model.blk[start:end]` in stub and real (GGUF) modes, plus new `pp_gpu_node`/`pp_smoke_run` binaries and ROADMAP/SPEC/TEST_SPEC docs - reuse: build on the single-GPU example's iroh transport and process bridge unchanged; add actor/codec/topology/integration test suites Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-16 05:49:43 +00:00
[[bin]]
refactor: rename pp binaries and rewrite orchestrator pp_smoke_run->pp_orchestrator (seed/vastai/hold/teardown, N stages), pp_gpu_node->pp_worker. fleet_plugin subscribes collector SSE + folds locally; iroh_driver drops verbose eprintln connection/relay logging. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-30 07:51:22 +00:00
name = "pp-orchestrator"
path = "src/bin/pp_orchestrator.rs"
feat: two-node pipeline parallel example Extend the single-GPU example into a two-node pipeline-parallel run that splits llama3.2:1b across two rented vast.ai GPUs and closes the autoregressive decode loop over iroh. - topology: add linear-chain helpers where each stage derives its neighbours locally from `STAGE`/`NUM_STAGES`, registering `pp-entry`/`pp-exit`/`pp-stage-{i}` SWIM names - messages: add `StageActivation` (bf16 hidden-state hand-off carrying position/seq_len/is_prefill) and `NextToken` (sampled-token feedback with a `done` flag) that close the autoregressive loop between stage 0 and stage 1 - stage_actor: add `Stage0Actor` (tokenize -> embed_and_forward -> prefill activation; decode_step on each NextToken) and `Stage1Actor` (forward_and_sample -> NextToken back; emit InferenceResponse on EOS/max_tokens) - vastai: fork the client and add `create_pipeline_instances` (rents one instance per stage, threading `STAGE`/`NUM_STAGES`, best-effort destroys on partial failure) and `destroy_all_instances` - pp_tinygrad_worker.py: per-stage worker slicing `model.blk[start:end]` in stub and real (GGUF) modes, plus new `pp_gpu_node`/`pp_smoke_run` binaries and ROADMAP/SPEC/TEST_SPEC docs - reuse: build on the single-GPU example's iroh transport and process bridge unchanged; add actor/codec/topology/integration test suites Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-16 05:49:43 +00:00
[dev-dependencies]
wiremock = "0.6"
feat(distribution): add probe rtt coverage and sim battery Thread SwimDiagEvent (ProbeAcked/TimedOut) for per-probe RTT; postproc renders RTT distribution + inference outcomes. N3 sim test battery: scenario families A-F + battery/evaluator invariants; split iroh-relay client/server features. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-26 08:09:22 +00:00
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
feat: wire stage roster and binary-swap verify Orchestrator resolves per-stage SWIM roster + emits pp_stage_roster; pp_gpu_node verifies fetched binary sha-256 digest (sha2 dep, strip release). Collector gains /diag/runs + /diag/stream SSE; vastai/dockerfile redeploy hardening. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-28 07:05:41 +00:00
feat(distribution): add vastai telemetry layer and synth New diagnostics::vastai (record/sampler/shipper/logs) + vastai-synth crate for reproducible synthetic fleet telemetry. Dashboard live_collector unifies collector + fleet SSE UI; pp example adds vastai_mon, profiles, docker base image, scripts. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-29 08:55:10 +00:00
# These binaries are shipped to rented GPU nodes over the docker image on a
# cold lease, and over scp through vast.ai's throttled SSH proxy for manual
# swaps. Stripping debug symbols (~26MB -> ~18MB) trims both at no runtime cost.
feat: wire stage roster and binary-swap verify Orchestrator resolves per-stage SWIM roster + emits pp_stage_roster; pp_gpu_node verifies fetched binary sha-256 digest (sha2 dep, strip release). Collector gains /diag/runs + /diag/stream SSE; vastai/dockerfile redeploy hardening. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-28 07:05:41 +00:00
[profile.release]
strip = true
Reference in a new issue Copy permalink