zacheryasc/swactor
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
swactor/examples/pipeline-parallel-inference/Cargo.toml

32 lines
727 B
TOML
Raw Normal View History

feat: two-node pipeline parallel example Extend the single-GPU example into a two-node pipeline-parallel run that splits llama3.2:1b across two rented vast.ai GPUs and closes the autoregressive decode loop over iroh. - topology: add linear-chain helpers where each stage derives its neighbours locally from `STAGE`/`NUM_STAGES`, registering `pp-entry`/`pp-exit`/`pp-stage-{i}` SWIM names - messages: add `StageActivation` (bf16 hidden-state hand-off carrying position/seq_len/is_prefill) and `NextToken` (sampled-token feedback with a `done` flag) that close the autoregressive loop between stage 0 and stage 1 - stage_actor: add `Stage0Actor` (tokenize -> embed_and_forward -> prefill activation; decode_step on each NextToken) and `Stage1Actor` (forward_and_sample -> NextToken back; emit InferenceResponse on EOS/max_tokens) - vastai: fork the client and add `create_pipeline_instances` (rents one instance per stage, threading `STAGE`/`NUM_STAGES`, best-effort destroys on partial failure) and `destroy_all_instances` - pp_tinygrad_worker.py: per-stage worker slicing `model.blk[start:end]` in stub and real (GGUF) modes, plus new `pp_gpu_node`/`pp_smoke_run` binaries and ROADMAP/SPEC/TEST_SPEC docs - reuse: build on the single-GPU example's iroh transport and process bridge unchanged; add actor/codec/topology/integration test suites Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-16 05:49:43 +00:00
[workspace]
[package]
name = "pipeline-parallel-inference"
version = "0.1.0"
edition = "2024"
publish = false
[dependencies]
swactor = { path = "../..", features = ["transport", "serde", "std"] }
swactor-process = { path = "../../crates/process" }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
reqwest = { version = "0.12", features = ["json"] }
tokio = { version = "1", features = ["full"] }
feat(distribution): add diagnostics subsystem Structured observability for the iroh/SWIM layer: Aggregator, typed Event/Snapshot types, Sink (NoopSink default), ProbeScheduler, process stats, and host/iroh/swim introspection, plus the swactor-diag-collector, -postproc, and -iroh-relay binaries that assemble and render per-run bundles. Generalizes the pipeline-parallel-inference example to N stages and adds the topology-planner spec. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-20 07:41:30 +00:00
distribution = { path = "../../crates/distribution", features = ["iroh", "collector"] }
feat: two-node pipeline parallel example Extend the single-GPU example into a two-node pipeline-parallel run that splits llama3.2:1b across two rented vast.ai GPUs and closes the autoregressive decode loop over iroh. - topology: add linear-chain helpers where each stage derives its neighbours locally from `STAGE`/`NUM_STAGES`, registering `pp-entry`/`pp-exit`/`pp-stage-{i}` SWIM names - messages: add `StageActivation` (bf16 hidden-state hand-off carrying position/seq_len/is_prefill) and `NextToken` (sampled-token feedback with a `done` flag) that close the autoregressive loop between stage 0 and stage 1 - stage_actor: add `Stage0Actor` (tokenize -> embed_and_forward -> prefill activation; decode_step on each NextToken) and `Stage1Actor` (forward_and_sample -> NextToken back; emit InferenceResponse on EOS/max_tokens) - vastai: fork the client and add `create_pipeline_instances` (rents one instance per stage, threading `STAGE`/`NUM_STAGES`, best-effort destroys on partial failure) and `destroy_all_instances` - pp_tinygrad_worker.py: per-stage worker slicing `model.blk[start:end]` in stub and real (GGUF) modes, plus new `pp_gpu_node`/`pp_smoke_run` binaries and ROADMAP/SPEC/TEST_SPEC docs - reuse: build on the single-GPU example's iroh transport and process bridge unchanged; add actor/codec/topology/integration test suites Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-16 05:49:43 +00:00
iroh = "0.96"
urlencoding = "2"
base64 = "0.22"
feat(distribution): add diagnostics subsystem Structured observability for the iroh/SWIM layer: Aggregator, typed Event/Snapshot types, Sink (NoopSink default), ProbeScheduler, process stats, and host/iroh/swim introspection, plus the swactor-diag-collector, -postproc, and -iroh-relay binaries that assemble and render per-run bundles. Generalizes the pipeline-parallel-inference example to N stages and adds the topology-planner spec. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-20 07:41:30 +00:00
libc = "0.2"
feat: two-node pipeline parallel example Extend the single-GPU example into a two-node pipeline-parallel run that splits llama3.2:1b across two rented vast.ai GPUs and closes the autoregressive decode loop over iroh. - topology: add linear-chain helpers where each stage derives its neighbours locally from `STAGE`/`NUM_STAGES`, registering `pp-entry`/`pp-exit`/`pp-stage-{i}` SWIM names - messages: add `StageActivation` (bf16 hidden-state hand-off carrying position/seq_len/is_prefill) and `NextToken` (sampled-token feedback with a `done` flag) that close the autoregressive loop between stage 0 and stage 1 - stage_actor: add `Stage0Actor` (tokenize -> embed_and_forward -> prefill activation; decode_step on each NextToken) and `Stage1Actor` (forward_and_sample -> NextToken back; emit InferenceResponse on EOS/max_tokens) - vastai: fork the client and add `create_pipeline_instances` (rents one instance per stage, threading `STAGE`/`NUM_STAGES`, best-effort destroys on partial failure) and `destroy_all_instances` - pp_tinygrad_worker.py: per-stage worker slicing `model.blk[start:end]` in stub and real (GGUF) modes, plus new `pp_gpu_node`/`pp_smoke_run` binaries and ROADMAP/SPEC/TEST_SPEC docs - reuse: build on the single-GPU example's iroh transport and process bridge unchanged; add actor/codec/topology/integration test suites Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-16 05:49:43 +00:00
[[bin]]
name = "pp-gpu-node"
path = "src/bin/pp_gpu_node.rs"
[[bin]]
name = "pp-smoke-run"
path = "src/bin/pp_smoke_run.rs"
[dev-dependencies]
wiremock = "0.6"
Reference in a new issue Copy permalink