Speed up VastAI provisioning by creating instances directly from a cached offer pool, and replace best-effort SSH readiness with classified, post-grace bootstrap-failure detection plus a dedicated per-node provider-status monitor.
- vastai_provisioning: provision_one now iterates a cached candidate_pool of offers calling create_instance directly (with host blacklist and failed-host dedup) instead of re-running client.provision; plan_first_wave_offers caches planned_offer_pool/planned_offer_ids for reuse
- vastai_provisioning: add VastAiProviderMonitor (background thread + AtomicBool stop + Drop) spawned per node via the new spawn_provider_monitor trait method, polling instance_status and emitting VastAiProviderStatusObserved/PollRetry/StatusFailure and terminal-start failures
- vastai_provisioning: add classify_ssh_observation (auth_denied/refused/timeout) with spawn_classifying_stderr_reader; spawn_retrying_ssh_bootstrap aborts after POST_GRACE_BOOTSTRAP_FAILURE_LIMIT repeated classified failures past the grace window instead of retrying forever
- vastai_provisioning: ssh_endpoint delegates to client.wait_for_ssh_endpoint; VastAiNode carries run_id/node_id/label/sink and emits structured VastAiLeaseReady/SshEndpointDiscoveryStarted/SshEndpointReady/RuntimeReadyAccepted/ContractCleanup events; LifecyclePolicy is threaded into start_bootstrap
- tools/vastai: add fetch_instance_status and wait_for_ssh_endpoint_with_policy, refactor wait_for_running onto fetch_instance_status, and export both plus ProviderInstanceStatus from lib.rs
- tools/vastai/types: add ProviderInstanceStatus (actual/intended status, status_msg, public_ipaddr, ssh_port, disk_usage) with ssh_endpoint() and From<InstanceStatus>
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add endpoint-address masking and a relay-only advertisement path so the multinode Docker mvp-chat scenario can run with direct addresses stripped.
- endpoint_advertisement: add EndpointAddrMask (Full/RelayOnly) parsed from --endpoint-addr-mask/MVP_IROH_ENDPOINT_ADDR_MASK, and advertised_endpoint that rebuilds an EndpointAddr from relay URLs only, rejecting relay-only without a relay URL
- orchestrator_app: mask the coordinator endpoint before advertising it, thread the masked collector endpoint into datastream subscribe/runtime-ready acks, surface endpoint_addr_mask/has_relay/direct_addr_count in iroh_driver and node_spec events, forward the mask env to workers, and add a 60s RUNTIME_READY_TIMEOUT to the runtime-ready barriers
- worker_node: advertise the masked self endpoint in the iroh_driver ready and coordinator_join events and propagate it through runtime_ready_local and PendingRuntimeReady
- mvp-chat: add --relay-mode/--relay-url/--endpoint-addr-mask plus a [relay] toml section, require (with a Vast.ai fallback) a relay URL when relay-only, and forward all three to the orchestrator CLI
- node_image: resolve the worker binary to a workspace-relative path for the Docker COPY via docker_build_context_path, rejecting paths outside the build context
- xtask/specs: run MultinodeDocker with --relay-mode default --endpoint-addr-mask relay-only, add dump-log fact checks for relay-masked orchestrator/node/coordinator advertisement, and document the mask/relay flags in mvp_chat.md
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
