Add end-to-end timing instrumentation and an xtask benchmark report for mvp-chat runs.
- benchmark_observability: add a shared stamping module — stamp(component) emitting schema/pid/monotonic+wall ms from a process-global start and sequence counter, plus unix_ms_now() — stamped onto every mvp-chat/orchestrator/worker-node event and frame-archive record
- mvp-chat: thread a run_id (new --run-id, defaults to 1) through config and the orchestrator CLI, add per-phase started/ready/failed emits for ensure_orch_binary/ensure_worker_binary/prepare_node_image, and a prompt_complete record carrying tokens_generated/elapsed_ms/final_text bytes
- orchestrator/worker-node: stamp bootstrap and prompt events, add arrival_unix_ms to archived frames, propagate MVP_RUN_ID/MVP_LOGICAL_NODE_ID/MVP_STAGE_INDEX into the tinygrad worker, default the device to CPU for the process provider, emit a prompt_rpc started span, and drop the MVP_TINYGRAD_TEST_MODE passthrough
- tinygrad_worker.py: stamp every control() event and tag it with run/node/stage env, add a CPU:X86 fallback when clang is absent, and remove the test_mode() short-circuits
- xtask: replace the flat dump-log fact assertions with a benchmark report builder (build_benchmark_report) that requires named spans (prepare_runtime, ensure_*_binary, weights_loaded, prompt_rpc) and emits per-prompt first-token/decode/tokens-per-second latency; wrap the cargo run in XtaskBenchmark synthetic frames and pass a unix-ms --run-id
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Replace stdout-parsed runtime-ready detection with an explicit, plugin-driven bootstrap-completion step and actorize SSH bootstrap teardown.
- provisioning: drop the `PluginObservation::RuntimeReady` variant and add `ProvisionPlugin::complete_bootstrap`, an explicit per-node completion hook (no-op for `LocalDockerPlugin`)
- bootstrap_datastream: remove `parse_runtime_ready`/`RuntimeReadyLine` so bootstrap no longer infers readiness from a parsed stdout JSON line
- vastai_provisioning: drop the `ReadyTrackingSink` ready-flag wrapper; the SSH retry loop now runs purely `while !stopping`, and `complete_bootstrap` stops the node's bootstrap with `BootstrapStopReason::RuntimeReady`
- vastai_provisioning: actorize teardown as `SshBootstrapActor` on the swactor `Runtime` (handle holds an `ActorAddress`), with `stop_bootstrap(handle, reason)` delivering a `Stop` message; add `BootstrapStopReason::{RuntimeReady,NodeStop}`
- actors/provisioner: replace the `RuntimeReady` observation arm with a `ProvisionerMsg::RuntimeReady` handler that calls `complete_bootstrap` then `mark_live`/emits NodeLive (or NodeFailed on error)
- callers/tests: wire the new explicit ready flow through node_agent, the orchestrator/worker_node binaries, and `mvp_one_node_chat`; add the `ssh_bootstrap_actor_stop_kills_child` test
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Stand up an interactive end-to-end chat over a CUDA GPU, provisioning a Dockerized node that loads a GGUF model and serves prompts over TCP.
- prompt_rpc: add the newline-JSON prompt protocol (`SubmitPrompt` + `PromptEvent::{TextDelta,Done,Fault}`) carried over TCP
- mvp_chat: add an interactive REPL client connecting to the prompt RPC port (default 127.0.0.1:19777)
- mvp_orch_one_node / mvp_one_node_chat: add the single-node orchestrator that provisions a `LocalDockerPlugin` node, loads `bartowski/Llama-3.2-1B-Instruct-GGUF` (Q4_K_M), and exposes the prompt RPC listener with boot/route/weight timeouts
- mvp_node: add the GPU worker binary that spawns `tinygrad_worker.py` (default device CUDA) and ships runtime telemetry via a `ClusterFrameSink`
- vastai_provisioning / bootstrap_datastream: add the vast.ai provider adapter (`VastAiProvisioningConfig`, `VastAiLeaseClient`) wrapping `swactor_vastai`, plus a bridge that folds provision stdout onto a per-node datastream
- apps/mvp-node: add CUDA base/runtime Dockerfiles (nvidia/cuda 12.6.3, tinygrad 0.12.0, sshd), `mvp_entrypoint.sh` (sshd + mvp-node, held for postmortem), `local_docker_e2e.sh`, the GGUF tinygrad worker, and one-node-chat/bootstrap/vastai guarantee tests
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
