Land end-to-end canonical benchmark observability and a synthetic datastream-connectivity preflight across the orchestrator, chat, worker-node, and Python tinygrad worker, plus an xtask validator, so a 4-stage 7B pipeline run is fully diagnosable.
- benchmark_observability: expand stamp() with canonical producer fields (producer_component/instance_id/process_id/sequence, wall_clock_unix_ms, monotonic_ms, clock_source) and schema_version so every event shares one envelope shape
- orchestrator_app + bin/{mvp_chat,worker_node}: stamp OrchBootstrap/OrchPromptEvent/ChatProgress/NodeEvent/SamplerHealth with the canonical fields plus span_id/parent_span_id, and add a 4-phase synthetic datastream preflight (ProducerConfigured/Connected/SyntheticEventSent/Observed) plus an endpoint_config_snapshot event on each process
- apps/mvp-node/tinygrad_worker: add apply_canonical_envelope()/datastream_endpoint_snapshot() and emit_python_datastream_preflight() mirroring the Rust preflight, and enrich benchmark_stamp() with the same producer fields
- bin/worker_node: pass MVP_DATASTREAM_ENDPOINT_ID/MVP_BENCHMARK_PRODUCER_INSTANCE/MVP_IROH_ENDPOINT_ADDR_MASK/MVP_IROH_RELAY_MODE env to the spawned tinygrad worker so its stamps identify the stage
- bin/mvp_chat: add --pipeline-parallel as an alias for --pipeline-stages (with a duplicate-guard) and bump recursion_limit
- xtask: add a benchmark-observability validator (ValidatorFinding/BenchmarkValidation, validate_benchmark_observability, canonical-stamp and stage/edge checks, evidence + gap-report builders) with tests for missing python datastream connectivity, wrong run_id, and missing span_id
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Complete an 8-stage pipeline-parallel run over VastAI by provisioning stages high-to-low, adding per-stage/per-step metrics, host anti-colocation, and provider state-timeout guardrails.
- orchestrator_app: select the next weight-load stage by max index (provision stages high-to-low for parallel spread), add a throttled "loaded N of M; waiting on stage X" stage_provision_wait headline, and surface min_compute_cap/state_timeout_secs in the config dump.
- orchestrator_app: enrich pipeline_token_in/out and tokenizer_decode events with token_count/token_ids/generated_index.
- worker_node: add timing metrics across the data path (helper_execute_ms, egress_ring_read_ms, send_ms, ingress_ring_write_ms, object_load_ms), refactor take_complete_ingress_record into IngressRecordBytes (object_id/sequence/extent/flags), and emit a new object_loaded event.
- vastai_provisioning: track leased host_ids and blacklist already-leased hosts in later ProvisionRequests so stages don't co-locate, and tag SSH-bootstrap retry logs with the attempt number.
- tools/vastai: add min_compute_cap (PP_MIN_COMPUTE_CAP) filter/search query and a LifecyclePolicy state_timeout (PP_STATE_TIMEOUT_SECS) that fails instances stuck in a non-running status instead of polling forever.
- xtask: raise the check timeout to 1800s/30s grace, drop --skip-rebuild for VastAI, aggregate per-stage StepExecuted metrics, add a vastai summary section, and write failure artifacts on abort.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Lift the VastAI two-stage cap and let mvp-chat-check drive an arbitrary N-stage deployment through a new --pipeline-stages flag.
- orchestrator_app: remove the provider=vastai >2 stage cap and replace the two-stage plan test with an eight-stage plan test (8 specs, no mounts, remote GGUF, max-context).
- xtask: refactor scenario parsing into MvpChatCheckInvocation carrying an optional pipeline_stages, parse a `--pipeline-stages n` flag (rejecting 0/missing values), and default multinode/docker to 2 while VastAI uses the explicit count.
- xtask: thread the invocation through run_mvp_chat_check/run_mvp_chat_check_process and update the usage text and scenario tests (including `--vastai --pipeline-stages 8`).
- gitignore: ignore .deployment-notes/.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Turn mvp-chat-check into a benchmarking harness with per-stage latency capture, a run envelope, summary artifacts, and a mvp-chat-compare command for delta analysis.
- xtask: add mvp-chat-compare <baseline> <candidate> that checks comparability (schema/scenario/workload/model/provider/pipeline_stages) and prints deltas for total, prepare, standup-to-RPC, and per-prompt roundtrip/first-token/decode/text-decode ms.
- xtask: build_benchmark_summary now writes stdout/stderr/prompts/redacted-config/summary artifacts with per-artifact byte counts and a vastai summary section; rename the dump log to datastream.ndjson.
- xtask: add write_failure_artifacts so failed checks still emit a failure summary with the prompt-corpus blake3 and artifact sizes.
- mvp_chat: add the mvp.chat.benchmark channel and emit_benchmark_envelope (BenchmarkRunEnvelope with model/runtime/provider/workload detail), and tag prompt events with prompt_index and a blake3 prompt_hash.
- tinygrad_worker: add per-phase latency metrics (encode/decode/text-decode/first-token elapsed_ms; stage_execution_ms/record_write_ms on execute_step; ring_readable/encode_prompt/decode_tokens elapsed_ms plus payload sizes).
- vastai_provisioning: emit VastAiLeaseReady and VastAiSshEndpointReady provider lines (contract/offer/host/gpu/dph, ssh host/port/user) for observability.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Land the first working two-stage pipeline-parallel run over VastAI, wiring a real inter-stage data path with observability, a max-price offer cap, and remote-image reuse.
- orchestrator_app: raise the VastAI pipeline-stage cap from 1 to 2 and let VastAI pipeline planning resolve the HuggingFace GGUF from the default local cached-model metadata path instead of requiring host mounts; add --vastai-max-dph-total (CLI/env/TOML) config.
- vastai_provisioning: make complete_bootstrap a no-op so the SSH bootstrap log tail stays alive past runtime-ready until node stop, preserving post-ready worker logs; add a test asserting the tail is only stopped on NodeStop.
- worker_node: emit data-path NodeEvents across the pipeline (iroh_edge_stream_arrived/bytes_read/bytes_sent, egress_ring_read, ingress_ring_write) with edge/byte metadata.
- tools/vastai: add max_dph_total (PP_MAX_DPH_TOTAL) to SelectionPolicy, the reachable-offer filter, and the search query, and improve the empty-pool error message.
- xtask: pass --skip-rebuild for the VastAI scenario and gate it on a new require_vastai_data_path_facts plus GPU facts (ring install, activation object load/step, interstage handoff, iroh edge read/sent).
- mvp_chat: add ChatModelConfig (model id/gguf/tokenizer/max-context) forwarded to the orchestrator; for VastAI + skip-rebuild, emit skip events and reuse the remote node image without a local build.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add a `--vastai` acceptance scenario to mvp-chat-check that provisions real VastAI nodes and verifies the remote provider data path, and standardize the VastAI API key on VAST_API_KEY.
- xtask: add the VastAi variant and `--vastai` flag to the mvp-chat-check scenario, passing `--vastai --yes --endpoint-addr-mask relay-only` (no --cached-model) and gating it on a new require_vastai_network_facts check (node_spec workers, ProvisionStart, provider_start, iroh_driver ready).
- xtask: track VastAI dump-log facts (vastai_node_spec_worker_count, vastai_provision_start_nodes, vastai_provider_start_nodes) via record_vastai_provision_dump_log_event and skip the local ChatProgress span assertion for the remote scenario.
- config: rename the VastAI key env var to VAST_API_KEY in ResolvedVastAiConfig validation while keeping MVP_VASTAI_API_KEY/VASTAI_API_KEY fallbacks.
- orchestrator_app: resolve the api key with VAST_API_KEY first, then MVP_VASTAI_API_KEY/VASTAI_API_KEY, and update the missing-key error messages.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Get the multinode pipeline-parallel prompt loop actually running locally by supporting both tinygrad LLM backends and switching the Docker scenario to direct addressing.
- tinygrad_worker.py (load_pipeline_stage_model): try the modern tinygrad.llm gguf/model modules first and fall back to the legacy tinygrad.apps.llm TransformerBlock on ModuleNotFoundError, with PipelineStageTinygradModel constructing blocks positionally when no TransformerConfig exists
- tinygrad_worker.py (load_weights): drop the Transformer.from_gguf whole-model branch and its TinygradAppsLlmPartialStageUnsupported fatal, so partial pipeline stages build through load_pipeline_stage_model on either backend
- xtask (MultinodeDocker): stop forcing --relay-mode default --endpoint-addr-mask relay-only, so the scenario runs over the Docker network with full/direct addresses
- xtask (dump-log facts): rename relay_masked_* facts to docker_*, assert multiple workers join the coordinator via direct addresses (direct_addr_count > 0), and relax the benchmark report to skip the ensure_worker_binary span for the Docker scenario
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add endpoint-address masking and a relay-only advertisement path so the multinode Docker mvp-chat scenario can run with direct addresses stripped.
- endpoint_advertisement: add EndpointAddrMask (Full/RelayOnly) parsed from --endpoint-addr-mask/MVP_IROH_ENDPOINT_ADDR_MASK, and advertised_endpoint that rebuilds an EndpointAddr from relay URLs only, rejecting relay-only without a relay URL
- orchestrator_app: mask the coordinator endpoint before advertising it, thread the masked collector endpoint into datastream subscribe/runtime-ready acks, surface endpoint_addr_mask/has_relay/direct_addr_count in iroh_driver and node_spec events, forward the mask env to workers, and add a 60s RUNTIME_READY_TIMEOUT to the runtime-ready barriers
- worker_node: advertise the masked self endpoint in the iroh_driver ready and coordinator_join events and propagate it through runtime_ready_local and PendingRuntimeReady
- mvp-chat: add --relay-mode/--relay-url/--endpoint-addr-mask plus a [relay] toml section, require (with a Vast.ai fallback) a relay URL when relay-only, and forward all three to the orchestrator CLI
- node_image: resolve the worker binary to a workspace-relative path for the Docker COPY via docker_build_context_path, rejecting paths outside the build context
- xtask/specs: run MultinodeDocker with --relay-mode default --endpoint-addr-mask relay-only, add dump-log fact checks for relay-masked orchestrator/node/coordinator advertisement, and document the mask/relay flags in mvp_chat.md
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
- Pull ~7.4k lines out of the orchestrator bin into a new orchestrator_app library
module.
- Wire a local single-node GPU prompt loop into the mvp_chat bin; touch
gpu_worker_ingress_parser.
- Grow xtask and the mvp-node tinygrad worker.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add end-to-end timing instrumentation and an xtask benchmark report for mvp-chat runs.
- benchmark_observability: add a shared stamping module — stamp(component) emitting schema/pid/monotonic+wall ms from a process-global start and sequence counter, plus unix_ms_now() — stamped onto every mvp-chat/orchestrator/worker-node event and frame-archive record
- mvp-chat: thread a run_id (new --run-id, defaults to 1) through config and the orchestrator CLI, add per-phase started/ready/failed emits for ensure_orch_binary/ensure_worker_binary/prepare_node_image, and a prompt_complete record carrying tokens_generated/elapsed_ms/final_text bytes
- orchestrator/worker-node: stamp bootstrap and prompt events, add arrival_unix_ms to archived frames, propagate MVP_RUN_ID/MVP_LOGICAL_NODE_ID/MVP_STAGE_INDEX into the tinygrad worker, default the device to CPU for the process provider, emit a prompt_rpc started span, and drop the MVP_TINYGRAD_TEST_MODE passthrough
- tinygrad_worker.py: stamp every control() event and tag it with run/node/stage env, add a CPU:X86 fallback when clang is absent, and remove the test_mode() short-circuits
- xtask: replace the flat dump-log fact assertions with a benchmark report builder (build_benchmark_report) that requires named spans (prepare_runtime, ensure_*_binary, weights_loaded, prompt_rpc) and emits per-prompt first-token/decode/tokens-per-second latency; wrap the cargo run in XtaskBenchmark synthetic frames and pass a unix-ms --run-id
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Move iroh_driver and the relay binary out of distribution into a dedicated
crates/iroh-driver (lib re-exports IrohDriver; relay bin renamed). Remove the node crate
and the single-gpu-inference example; drop the docker/datastream demo. Slim
pipeline-parallel vastai.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Remove the datastore crate, the top-level design/orchestration/ring specs, the
benches, and the ci config. Add the dashboard host telemetry sampler
(cpu/disk/net/gpu/mem). Localize the pipeline-parallel e2e stub/mock paths.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Replaces the generic SimNode/gossip/dashboard framework with a virtual-time
discrete-event engine (priority queue ordered by time/node/fiber/seq), a TOML spec
parser, bundle writer, replay, divergence detector, lint, and postproc, plus the
parity-bar test harness with fixtures and xtask parity-lock tooling. Rewrites
transport identity/crypto and adds the SPEC/TESTING_SPEC/OBSERVABILITY/NORTH_STAR docs.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Remove co-dependencies for different modules found in `crates` and migrate the development history to a new repository. The docs were stale, and largely not getting used, so simply deleted for now. When code stabilizes more, they will become useful again.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Initial experiments in 'formal verification' of authorization tooling. Current state is not to be trusted, but we are not in a security critical situation, so that is fine.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Introduce a gossip-converged pooled datastore protocol layered on SWIM piggybacking, backed by a reusable gossip-channel abstraction, plus a Docker-free multi-process cluster test runner.
- crates/distribution/src/gossip_channel.rs: add the GossipChannel trait (piggyback on SWIM messages) and a budget-limited DisseminationBuffer<T> that replaces the four duplicated Lambda*ceil(log2(n)) dissemination copies
- crates/datastore/src/pool: add PoolDisseminator (CRDT state for membership/capacity/content-location/ACL with join/leave/announce) and PoolCoordinator (placement-aware CRUD actor delegating to the co-located DatastoreNode)
- crates/shared-types/src/pool.rs: add shared pool protocol types (PoolId plus member/capacity/content-location/ACL entries and PoolConfig) consumed by both distribution and datastore
- crates/dashboard/src/pool_html.rs: add a live pool dashboard page (membership, capacity, content locations) and add pool_tests integration coverage
- xtask/src/sim_cluster.rs: add the sim-cluster runner that spawns N swactor nodes over iroh through a local relay server, reusing the docker cluster scenarios without Docker
- crates/distribution/src/iroh_driver.rs: add relay-URL resolution (cache, then SWIM gossip, then home relay) with a 2s connect timeout to back the relay-based connections
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Make distribution and deployment more stable. Consolidate the logic for a generic swactor node.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Content addressable datastore. Allows you to configure a node to store and stream large blobs of data, and retrieve them from any swactor-connected node.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
