refactor(myelin): rework control and runtime integration
Add actor-backed manual node provisioning, control-plane endpoints, and fleet UI assets with durable provider lifecycle handling.
Simplify Myelin orchestration, node runtime, staging, and telemetry paths while removing obsolete engine-builder, dashboard-view, and local-mock implementations.
Align runtime delivery, data-plane, distribution, job-runner, process, telemetry, dashboard, Vast.ai integrations, and their tests with the revised actor and transport contracts.
2026-08-19 10:20:01 +00:00
|
|
|
// NOTE: This module is currently excluded from compilation (its `mod` entry
|
|
|
|
|
// was removed from orchestration/mod.rs). Backburner code — deprecated for
|
|
|
|
|
// now but not yet staged for pruning. Re-wire or delete when node-image
|
|
|
|
|
// provisioning is revisited.
|
|
|
|
|
|
2026-07-05 09:59:51 +00:00
|
|
|
use std::collections::{BTreeMap, BTreeSet};
|
|
|
|
|
use std::fs::{self, File};
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
use std::io::{BufRead, BufReader, Read};
|
2026-07-05 09:59:51 +00:00
|
|
|
use std::path::{Path, PathBuf};
|
|
|
|
|
use std::process::{Command, Stdio};
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
use std::sync::mpsc;
|
|
|
|
|
use std::thread;
|
|
|
|
|
use std::time::{Duration, Instant};
|
2026-07-05 09:59:51 +00:00
|
|
|
|
2026-08-17 22:51:08 +00:00
|
|
|
const NODE_IMAGE_CONTENT_INPUTS: &[&str] = &["apps/myelin/node-image/Dockerfile"];
|
2026-07-05 09:59:51 +00:00
|
|
|
|
|
|
|
|
const BASE_IMAGE_SOURCE_INPUTS: &[&str] = &[
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
"apps/myelin/node-image/Dockerfile.base",
|
|
|
|
|
"apps/myelin/node-image/myelin_entrypoint.sh",
|
2026-07-05 09:59:51 +00:00
|
|
|
];
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
const NODE_IMAGE_TAG_LABEL: &str = "org.swactor.myelin.node-image-tag";
|
|
|
|
|
const NODE_IMAGE_SOURCE_HASH_LABEL: &str = "org.swactor.myelin.node.source-hash";
|
2026-08-17 22:51:08 +00:00
|
|
|
const NODE_IMAGE_AGENT_HASH_LABEL: &str = "org.swactor.myelin.node.agent-hash";
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
const NODE_IMAGE_BASE_HASH_LABEL: &str = "org.swactor.myelin.node.base-hash";
|
|
|
|
|
const BASE_IMAGE_SOURCE_HASH_LABEL: &str = "org.swactor.myelin.base.source-hash";
|
2026-07-05 09:59:51 +00:00
|
|
|
|
refactor: prune public api
Collapse mvp-system's public surface to three binary entrypoints and make every domain module private, deleting dead provider/worker/membership implementations and inlining provider config.
- lib.rs: expose only run_chat_from_args/run_orchestrator_from_args/run_worker_node_from_env (plus a crate-private in-process helper) and the cached-model consts, and demote chat/node/observability/orchestration/prompt/staging/transport to private mods
- orchestration/mod.rs: make app private, gate engine_builder behind cfg(test), drop docker_cluster from provider_adapters, tighten vastai to pub(super), and replace pub re-exports with pub(super) run_from_args/run_in_process_from_args
- orchestration/config.rs: inline VastAiConfig/ResolvedVastAiConfig/looks_remote_image (removing provider_adapters/vastai/config.rs) and drop the DEFAULT_PIPELINE_CACHED_MODEL_* consts (hoisted to lib.rs)
- orchestration/provider_adapters/vastai: delete the ProviderPlugin impl VastAiProviderPlugin and all client/bootstrap/config accessors; repoint call sites to crate-level #[path] mods for provisioning/node_provisioning/node_actor/gguf_shard/run_fsm/run_plan
- delete orchestration/{membership_readiness,token_endpoint,resource_inventory}, node/{boot_lifecycle,data_plane_bridge(-74)}, and the worker crate-internal modules (control/device_bridge/process_adapter) along with their guarantees tests
- chat/node: narrow node_image and worker_node_runtime to private and expose only pub(super) run_from_args / run_worker_node_from_env
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-29 10:02:50 +00:00
|
|
|
pub(super) struct NodeImageRequest {
|
|
|
|
|
pub(super) requested_image: String,
|
|
|
|
|
pub(super) base_image: String,
|
|
|
|
|
pub(super) node_bin: PathBuf,
|
2026-07-29 17:09:26 +00:00
|
|
|
pub(super) requires_registry_image: bool,
|
refactor: prune public api
Collapse mvp-system's public surface to three binary entrypoints and make every domain module private, deleting dead provider/worker/membership implementations and inlining provider config.
- lib.rs: expose only run_chat_from_args/run_orchestrator_from_args/run_worker_node_from_env (plus a crate-private in-process helper) and the cached-model consts, and demote chat/node/observability/orchestration/prompt/staging/transport to private mods
- orchestration/mod.rs: make app private, gate engine_builder behind cfg(test), drop docker_cluster from provider_adapters, tighten vastai to pub(super), and replace pub re-exports with pub(super) run_from_args/run_in_process_from_args
- orchestration/config.rs: inline VastAiConfig/ResolvedVastAiConfig/looks_remote_image (removing provider_adapters/vastai/config.rs) and drop the DEFAULT_PIPELINE_CACHED_MODEL_* consts (hoisted to lib.rs)
- orchestration/provider_adapters/vastai: delete the ProviderPlugin impl VastAiProviderPlugin and all client/bootstrap/config accessors; repoint call sites to crate-level #[path] mods for provisioning/node_provisioning/node_actor/gguf_shard/run_fsm/run_plan
- delete orchestration/{membership_readiness,token_endpoint,resource_inventory}, node/{boot_lifecycle,data_plane_bridge(-74)}, and the worker crate-internal modules (control/device_bridge/process_adapter) along with their guarantees tests
- chat/node: narrow node_image and worker_node_runtime to private and expose only pub(super) run_from_args / run_worker_node_from_env
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-29 10:02:50 +00:00
|
|
|
pub(super) extra_tag: Option<String>,
|
|
|
|
|
pub(super) force_refresh: bool,
|
|
|
|
|
pub(super) enabled: bool,
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
|
|
|
|
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
#[derive(Clone, Debug, PartialEq, Eq)]
|
refactor: prune public api
Collapse mvp-system's public surface to three binary entrypoints and make every domain module private, deleting dead provider/worker/membership implementations and inlining provider config.
- lib.rs: expose only run_chat_from_args/run_orchestrator_from_args/run_worker_node_from_env (plus a crate-private in-process helper) and the cached-model consts, and demote chat/node/observability/orchestration/prompt/staging/transport to private mods
- orchestration/mod.rs: make app private, gate engine_builder behind cfg(test), drop docker_cluster from provider_adapters, tighten vastai to pub(super), and replace pub re-exports with pub(super) run_from_args/run_in_process_from_args
- orchestration/config.rs: inline VastAiConfig/ResolvedVastAiConfig/looks_remote_image (removing provider_adapters/vastai/config.rs) and drop the DEFAULT_PIPELINE_CACHED_MODEL_* consts (hoisted to lib.rs)
- orchestration/provider_adapters/vastai: delete the ProviderPlugin impl VastAiProviderPlugin and all client/bootstrap/config accessors; repoint call sites to crate-level #[path] mods for provisioning/node_provisioning/node_actor/gguf_shard/run_fsm/run_plan
- delete orchestration/{membership_readiness,token_endpoint,resource_inventory}, node/{boot_lifecycle,data_plane_bridge(-74)}, and the worker crate-internal modules (control/device_bridge/process_adapter) along with their guarantees tests
- chat/node: narrow node_image and worker_node_runtime to private and expose only pub(super) run_from_args / run_worker_node_from_env
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-29 10:02:50 +00:00
|
|
|
pub(super) enum NodeImageProgressEventKind {
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
ImageReference {
|
|
|
|
|
role: String,
|
|
|
|
|
image_ref: String,
|
|
|
|
|
},
|
|
|
|
|
CommandStarted {
|
|
|
|
|
program: String,
|
|
|
|
|
args: Vec<String>,
|
|
|
|
|
},
|
|
|
|
|
CommandStdout {
|
|
|
|
|
line: String,
|
|
|
|
|
},
|
|
|
|
|
CommandStderr {
|
|
|
|
|
line: String,
|
|
|
|
|
},
|
|
|
|
|
CommandExited {
|
|
|
|
|
status: String,
|
|
|
|
|
code: Option<i32>,
|
|
|
|
|
success: bool,
|
|
|
|
|
},
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#[derive(Clone, Debug, PartialEq, Eq)]
|
refactor: prune public api
Collapse mvp-system's public surface to three binary entrypoints and make every domain module private, deleting dead provider/worker/membership implementations and inlining provider config.
- lib.rs: expose only run_chat_from_args/run_orchestrator_from_args/run_worker_node_from_env (plus a crate-private in-process helper) and the cached-model consts, and demote chat/node/observability/orchestration/prompt/staging/transport to private mods
- orchestration/mod.rs: make app private, gate engine_builder behind cfg(test), drop docker_cluster from provider_adapters, tighten vastai to pub(super), and replace pub re-exports with pub(super) run_from_args/run_in_process_from_args
- orchestration/config.rs: inline VastAiConfig/ResolvedVastAiConfig/looks_remote_image (removing provider_adapters/vastai/config.rs) and drop the DEFAULT_PIPELINE_CACHED_MODEL_* consts (hoisted to lib.rs)
- orchestration/provider_adapters/vastai: delete the ProviderPlugin impl VastAiProviderPlugin and all client/bootstrap/config accessors; repoint call sites to crate-level #[path] mods for provisioning/node_provisioning/node_actor/gguf_shard/run_fsm/run_plan
- delete orchestration/{membership_readiness,token_endpoint,resource_inventory}, node/{boot_lifecycle,data_plane_bridge(-74)}, and the worker crate-internal modules (control/device_bridge/process_adapter) along with their guarantees tests
- chat/node: narrow node_image and worker_node_runtime to private and expose only pub(super) run_from_args / run_worker_node_from_env
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-29 10:02:50 +00:00
|
|
|
pub(super) struct NodeImageProgressEvent {
|
|
|
|
|
pub(super) command_label: Option<String>,
|
|
|
|
|
pub(super) image_ref: Option<String>,
|
|
|
|
|
pub(super) elapsed_ms: Option<u128>,
|
|
|
|
|
pub(super) kind: NodeImageProgressEventKind,
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
}
|
|
|
|
|
|
refactor: prune public api
Collapse mvp-system's public surface to three binary entrypoints and make every domain module private, deleting dead provider/worker/membership implementations and inlining provider config.
- lib.rs: expose only run_chat_from_args/run_orchestrator_from_args/run_worker_node_from_env (plus a crate-private in-process helper) and the cached-model consts, and demote chat/node/observability/orchestration/prompt/staging/transport to private mods
- orchestration/mod.rs: make app private, gate engine_builder behind cfg(test), drop docker_cluster from provider_adapters, tighten vastai to pub(super), and replace pub re-exports with pub(super) run_from_args/run_in_process_from_args
- orchestration/config.rs: inline VastAiConfig/ResolvedVastAiConfig/looks_remote_image (removing provider_adapters/vastai/config.rs) and drop the DEFAULT_PIPELINE_CACHED_MODEL_* consts (hoisted to lib.rs)
- orchestration/provider_adapters/vastai: delete the ProviderPlugin impl VastAiProviderPlugin and all client/bootstrap/config accessors; repoint call sites to crate-level #[path] mods for provisioning/node_provisioning/node_actor/gguf_shard/run_fsm/run_plan
- delete orchestration/{membership_readiness,token_endpoint,resource_inventory}, node/{boot_lifecycle,data_plane_bridge(-74)}, and the worker crate-internal modules (control/device_bridge/process_adapter) along with their guarantees tests
- chat/node: narrow node_image and worker_node_runtime to private and expose only pub(super) run_from_args / run_worker_node_from_env
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-29 10:02:50 +00:00
|
|
|
pub(super) trait NodeImageProgressSink {
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
fn emit(&mut self, event: NodeImageProgressEvent);
|
|
|
|
|
}
|
|
|
|
|
|
refactor: prune public api
Collapse mvp-system's public surface to three binary entrypoints and make every domain module private, deleting dead provider/worker/membership implementations and inlining provider config.
- lib.rs: expose only run_chat_from_args/run_orchestrator_from_args/run_worker_node_from_env (plus a crate-private in-process helper) and the cached-model consts, and demote chat/node/observability/orchestration/prompt/staging/transport to private mods
- orchestration/mod.rs: make app private, gate engine_builder behind cfg(test), drop docker_cluster from provider_adapters, tighten vastai to pub(super), and replace pub re-exports with pub(super) run_from_args/run_in_process_from_args
- orchestration/config.rs: inline VastAiConfig/ResolvedVastAiConfig/looks_remote_image (removing provider_adapters/vastai/config.rs) and drop the DEFAULT_PIPELINE_CACHED_MODEL_* consts (hoisted to lib.rs)
- orchestration/provider_adapters/vastai: delete the ProviderPlugin impl VastAiProviderPlugin and all client/bootstrap/config accessors; repoint call sites to crate-level #[path] mods for provisioning/node_provisioning/node_actor/gguf_shard/run_fsm/run_plan
- delete orchestration/{membership_readiness,token_endpoint,resource_inventory}, node/{boot_lifecycle,data_plane_bridge(-74)}, and the worker crate-internal modules (control/device_bridge/process_adapter) along with their guarantees tests
- chat/node: narrow node_image and worker_node_runtime to private and expose only pub(super) run_from_args / run_worker_node_from_env
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-29 10:02:50 +00:00
|
|
|
pub(super) fn prepare_node_image_with_progress(
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
request: NodeImageRequest,
|
|
|
|
|
progress: Option<&mut dyn NodeImageProgressSink>,
|
2026-07-29 13:17:57 +00:00
|
|
|
) -> Result<String, String> {
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
let mut progress = progress;
|
2026-07-29 17:09:26 +00:00
|
|
|
prepare_node_image_inner(request, &mut progress)
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn prepare_node_image_inner(
|
|
|
|
|
request: NodeImageRequest,
|
|
|
|
|
progress: &mut Option<&mut dyn NodeImageProgressSink>,
|
2026-07-29 13:17:57 +00:00
|
|
|
) -> Result<String, String> {
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
emit_image_reference(progress, "requested", &request.requested_image);
|
2026-07-05 09:59:51 +00:00
|
|
|
if !request.enabled {
|
2026-07-29 13:17:57 +00:00
|
|
|
return Ok(request.requested_image);
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
|
|
|
|
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
emit_image_reference(progress, "base", &request.base_image);
|
2026-07-05 09:59:51 +00:00
|
|
|
let root = workspace_root()?;
|
|
|
|
|
let image = ImageName::parse(&request.requested_image)?;
|
2026-07-29 17:09:26 +00:00
|
|
|
let first_repository_component = image
|
|
|
|
|
.repository
|
|
|
|
|
.split('/')
|
|
|
|
|
.next()
|
|
|
|
|
.unwrap_or(&image.repository);
|
|
|
|
|
let registry_reachable = image.repository.contains('/')
|
|
|
|
|
|| first_repository_component.contains('.')
|
|
|
|
|
|| first_repository_component.contains(':')
|
|
|
|
|
|| first_repository_component == "localhost";
|
|
|
|
|
if request.requires_registry_image && !registry_reachable {
|
2026-07-05 09:59:51 +00:00
|
|
|
return Err(format!(
|
|
|
|
|
"VastAI node image {:?} must include a registry namespace",
|
|
|
|
|
image.repository
|
|
|
|
|
));
|
|
|
|
|
}
|
|
|
|
|
|
2026-07-31 08:04:09 +00:00
|
|
|
run_status_command(
|
fix: faster provisioning, better ssh checks
Speed up VastAI provisioning by creating instances directly from a cached offer pool, and replace best-effort SSH readiness with classified, post-grace bootstrap-failure detection plus a dedicated per-node provider-status monitor.
- vastai_provisioning: provision_one now iterates a cached candidate_pool of offers calling create_instance directly (with host blacklist and failed-host dedup) instead of re-running client.provision; plan_first_wave_offers caches planned_offer_pool/planned_offer_ids for reuse
- vastai_provisioning: add VastAiProviderMonitor (background thread + AtomicBool stop + Drop) spawned per node via the new spawn_provider_monitor trait method, polling instance_status and emitting VastAiProviderStatusObserved/PollRetry/StatusFailure and terminal-start failures
- vastai_provisioning: add classify_ssh_observation (auth_denied/refused/timeout) with spawn_classifying_stderr_reader; spawn_retrying_ssh_bootstrap aborts after POST_GRACE_BOOTSTRAP_FAILURE_LIMIT repeated classified failures past the grace window instead of retrying forever
- vastai_provisioning: ssh_endpoint delegates to client.wait_for_ssh_endpoint; VastAiNode carries run_id/node_id/label/sink and emits structured VastAiLeaseReady/SshEndpointDiscoveryStarted/SshEndpointReady/RuntimeReadyAccepted/ContractCleanup events; LifecyclePolicy is threaded into start_bootstrap
- tools/vastai: add fetch_instance_status and wait_for_ssh_endpoint_with_policy, refactor wait_for_running onto fetch_instance_status, and export both plus ProviderInstanceStatus from lib.rs
- tools/vastai/types: add ProviderInstanceStatus (actual/intended status, status_msg, public_ipaddr, ssh_port, disk_usage) with ssh_endpoint() and From<InstanceStatus>
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-27 08:30:49 +00:00
|
|
|
&root,
|
|
|
|
|
"cargo",
|
feat(provisioning): add level-triggered cluster reconciler
Introduce a pure, level-triggered reconciler in `crates/provisioning`
that drives a declared cluster shape toward convergence over the
existing node lifecycle, replacing the edge-triggered imperative node
orchestration in `apps/myelin`.
- `reconcile`/`reconcile_node`/`observe`: pure decider and observation
folder with stable logical-node identity, per-attempt operation
identity, and deterministic retry backoff; `ClusterDriver` is the sole
writer of observed state, coalescing triggers, recording operations as
pending before dispatch, and scheduling timed requeues.
- `IdempotentEffectExecutor`: deduplicates submissions by
`(run_id, logical_node_id, attempt)` and runs provider work on the
engine-hosted blocking substrate, never blocking a reconcile pass.
- Myelin integration: `MyelinEffectBackend` bridges `ProvisionPlugin` to
the executor contract; `LocalProcessPlugin`/`LocalDockerPlugin`
provider adapters; `ProvisionedClusterGuard` pumps triggers,
observations, and due operations.
- Retire the imperative acquire/bootstrap/teardown sequencing across
`apps/myelin` orchestration, staging, observability, and provider
adapters in favor of the declarative driver.
- Move the reconciler specification to `docs/specs/archive`.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 07:49:18 +00:00
|
|
|
&["build", "--quiet", "-p", "myelin", "--bin", "myelin-worker"],
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
"build myelin-worker",
|
fix: faster provisioning, better ssh checks
Speed up VastAI provisioning by creating instances directly from a cached offer pool, and replace best-effort SSH readiness with classified, post-grace bootstrap-failure detection plus a dedicated per-node provider-status monitor.
- vastai_provisioning: provision_one now iterates a cached candidate_pool of offers calling create_instance directly (with host blacklist and failed-host dedup) instead of re-running client.provision; plan_first_wave_offers caches planned_offer_pool/planned_offer_ids for reuse
- vastai_provisioning: add VastAiProviderMonitor (background thread + AtomicBool stop + Drop) spawned per node via the new spawn_provider_monitor trait method, polling instance_status and emitting VastAiProviderStatusObserved/PollRetry/StatusFailure and terminal-start failures
- vastai_provisioning: add classify_ssh_observation (auth_denied/refused/timeout) with spawn_classifying_stderr_reader; spawn_retrying_ssh_bootstrap aborts after POST_GRACE_BOOTSTRAP_FAILURE_LIMIT repeated classified failures past the grace window instead of retrying forever
- vastai_provisioning: ssh_endpoint delegates to client.wait_for_ssh_endpoint; VastAiNode carries run_id/node_id/label/sink and emits structured VastAiLeaseReady/SshEndpointDiscoveryStarted/SshEndpointReady/RuntimeReadyAccepted/ContractCleanup events; LifecyclePolicy is threaded into start_bootstrap
- tools/vastai: add fetch_instance_status and wait_for_ssh_endpoint_with_policy, refactor wait_for_running onto fetch_instance_status, and export both plus ProviderInstanceStatus from lib.rs
- tools/vastai/types: add ProviderInstanceStatus (actual/intended status, status_msg, public_ipaddr, ssh_port, disk_usage) with ssh_endpoint() and From<InstanceStatus>
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-27 08:30:49 +00:00
|
|
|
None,
|
2026-07-31 08:04:09 +00:00
|
|
|
progress,
|
fix: faster provisioning, better ssh checks
Speed up VastAI provisioning by creating instances directly from a cached offer pool, and replace best-effort SSH readiness with classified, post-grace bootstrap-failure detection plus a dedicated per-node provider-status monitor.
- vastai_provisioning: provision_one now iterates a cached candidate_pool of offers calling create_instance directly (with host blacklist and failed-host dedup) instead of re-running client.provision; plan_first_wave_offers caches planned_offer_pool/planned_offer_ids for reuse
- vastai_provisioning: add VastAiProviderMonitor (background thread + AtomicBool stop + Drop) spawned per node via the new spawn_provider_monitor trait method, polling instance_status and emitting VastAiProviderStatusObserved/PollRetry/StatusFailure and terminal-start failures
- vastai_provisioning: add classify_ssh_observation (auth_denied/refused/timeout) with spawn_classifying_stderr_reader; spawn_retrying_ssh_bootstrap aborts after POST_GRACE_BOOTSTRAP_FAILURE_LIMIT repeated classified failures past the grace window instead of retrying forever
- vastai_provisioning: ssh_endpoint delegates to client.wait_for_ssh_endpoint; VastAiNode carries run_id/node_id/label/sink and emits structured VastAiLeaseReady/SshEndpointDiscoveryStarted/SshEndpointReady/RuntimeReadyAccepted/ContractCleanup events; LifecyclePolicy is threaded into start_bootstrap
- tools/vastai: add fetch_instance_status and wait_for_ssh_endpoint_with_policy, refactor wait_for_running onto fetch_instance_status, and export both plus ProviderInstanceStatus from lib.rs
- tools/vastai/types: add ProviderInstanceStatus (actual/intended status, status_msg, public_ipaddr, ssh_port, disk_usage) with ssh_endpoint() and From<InstanceStatus>
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-27 08:30:49 +00:00
|
|
|
)?;
|
|
|
|
|
|
|
|
|
|
let base_hash = content_hash_for_inputs(&root, BASE_IMAGE_SOURCE_INPUTS)?;
|
|
|
|
|
let image_content_hash = node_image_content_hash(&root, &request.node_bin, &base_hash)?;
|
|
|
|
|
let tag = image_version_tag(&root, &image_content_hash)?;
|
2026-07-05 09:59:51 +00:00
|
|
|
let image_ref = image.ref_for_tag(&tag);
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
emit_image_reference(progress, "resolved", &image_ref);
|
2026-07-29 17:09:26 +00:00
|
|
|
let expected_node_labels = vec![
|
|
|
|
|
(NODE_IMAGE_TAG_LABEL, tag.as_str()),
|
|
|
|
|
(NODE_IMAGE_SOURCE_HASH_LABEL, image_content_hash.as_str()),
|
2026-08-17 22:51:08 +00:00
|
|
|
(NODE_IMAGE_AGENT_HASH_LABEL, image_content_hash.as_str()),
|
2026-07-29 17:09:26 +00:00
|
|
|
(NODE_IMAGE_BASE_HASH_LABEL, base_hash.as_str()),
|
|
|
|
|
];
|
|
|
|
|
let expected_base_labels = vec![(BASE_IMAGE_SOURCE_HASH_LABEL, base_hash.as_str())];
|
2026-07-05 09:59:51 +00:00
|
|
|
let alias_tags = alias_tags(&image, request.extra_tag.as_deref(), &tag)?;
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
for alias in alias_refs(&image, &alias_tags) {
|
|
|
|
|
emit_image_reference(progress, "alias", &alias);
|
|
|
|
|
}
|
2026-07-29 17:09:26 +00:00
|
|
|
let remote_required = request.requires_registry_image;
|
2026-07-05 09:59:51 +00:00
|
|
|
|
2026-07-29 17:09:26 +00:00
|
|
|
let local_image_matches = docker_image_labels_match(&root, &image_ref, &expected_node_labels)?;
|
|
|
|
|
let remote_available = remote_required && docker_manifest_exists(&root, &image_ref);
|
2026-07-05 09:59:51 +00:00
|
|
|
if !request.force_refresh && remote_required && remote_available {
|
2026-07-29 17:09:26 +00:00
|
|
|
ensure_aliases_for_remote(progress, &root, &image_ref, &image, &alias_tags)?;
|
|
|
|
|
prune_old_dirty_images(&root, &image, &tag);
|
2026-07-29 13:17:57 +00:00
|
|
|
return Ok(image_ref);
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
|
|
|
|
if !request.force_refresh && remote_required && local_image_matches {
|
2026-07-29 17:09:26 +00:00
|
|
|
ensure_aliases_local(progress, &root, &image_ref, &image, &alias_tags)?;
|
|
|
|
|
push_image(progress, &root, &image_ref)?;
|
2026-07-05 09:59:51 +00:00
|
|
|
for alias in alias_refs(&image, &alias_tags) {
|
2026-07-29 17:09:26 +00:00
|
|
|
push_image(progress, &root, &alias)?;
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
2026-07-29 17:09:26 +00:00
|
|
|
prune_old_dirty_images(&root, &image, &tag);
|
2026-07-29 13:17:57 +00:00
|
|
|
return Ok(image_ref);
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
|
|
|
|
if !request.force_refresh && !remote_required && local_image_matches {
|
2026-07-29 17:09:26 +00:00
|
|
|
ensure_aliases_local(progress, &root, &image_ref, &image, &alias_tags)?;
|
|
|
|
|
prune_old_dirty_images(&root, &image, &tag);
|
2026-07-29 13:17:57 +00:00
|
|
|
return Ok(image_ref);
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
|
|
|
|
let base_image_matches =
|
2026-07-29 17:09:26 +00:00
|
|
|
docker_image_labels_match(&root, &request.base_image, &expected_base_labels)?;
|
2026-07-05 09:59:51 +00:00
|
|
|
if !base_image_matches {
|
2026-07-31 08:04:09 +00:00
|
|
|
let base_source_hash_label = format!("{BASE_IMAGE_SOURCE_HASH_LABEL}={base_hash}");
|
2026-07-29 17:09:26 +00:00
|
|
|
run_status_command(
|
2026-07-05 09:59:51 +00:00
|
|
|
&root,
|
|
|
|
|
"docker",
|
2026-07-31 08:04:09 +00:00
|
|
|
&[
|
|
|
|
|
"build",
|
|
|
|
|
"-f",
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
"apps/myelin/node-image/Dockerfile.base",
|
2026-07-31 08:04:09 +00:00
|
|
|
"--label",
|
|
|
|
|
base_source_hash_label.as_str(),
|
|
|
|
|
"-t",
|
|
|
|
|
request.base_image.as_str(),
|
|
|
|
|
".",
|
2026-07-05 09:59:51 +00:00
|
|
|
],
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
"build myelin node base image",
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
Some(&request.base_image),
|
2026-07-29 17:09:26 +00:00
|
|
|
progress,
|
2026-07-05 09:59:51 +00:00
|
|
|
)?;
|
|
|
|
|
}
|
|
|
|
|
|
2026-07-31 08:04:09 +00:00
|
|
|
let node_bin = {
|
|
|
|
|
let full = if request.node_bin.is_absolute() {
|
|
|
|
|
request.node_bin.to_path_buf()
|
|
|
|
|
} else {
|
|
|
|
|
root.join(&request.node_bin)
|
|
|
|
|
};
|
|
|
|
|
relative_path(&root, &full).map(|relative| relative.to_string_lossy().to_string())
|
|
|
|
|
}?;
|
2026-07-05 09:59:51 +00:00
|
|
|
let mut build_args = vec![
|
|
|
|
|
"build".to_owned(),
|
|
|
|
|
"-f".to_owned(),
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
"apps/myelin/node-image/Dockerfile".to_owned(),
|
2026-07-05 09:59:51 +00:00
|
|
|
"--build-arg".to_owned(),
|
|
|
|
|
format!("BASE_IMAGE={}", request.base_image),
|
|
|
|
|
"--build-arg".to_owned(),
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
format!("MYELIN_NODE_BIN={node_bin}"),
|
2026-07-05 09:59:51 +00:00
|
|
|
];
|
|
|
|
|
for (key, value) in &expected_node_labels {
|
|
|
|
|
build_args.push("--label".to_owned());
|
|
|
|
|
build_args.push(format!("{key}={value}"));
|
|
|
|
|
}
|
|
|
|
|
build_args.extend(["-t".to_owned(), image_ref.clone(), ".".to_owned()]);
|
2026-07-29 17:09:26 +00:00
|
|
|
run_status_command(
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
&root,
|
|
|
|
|
"docker",
|
2026-07-31 08:04:09 +00:00
|
|
|
&build_args.iter().map(String::as_str).collect::<Vec<_>>(),
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
"build myelin node image",
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
Some(&image_ref),
|
2026-07-29 17:09:26 +00:00
|
|
|
progress,
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
)?;
|
2026-07-29 17:09:26 +00:00
|
|
|
ensure_aliases_local(progress, &root, &image_ref, &image, &alias_tags)?;
|
2026-07-05 09:59:51 +00:00
|
|
|
|
|
|
|
|
if remote_required {
|
2026-07-29 17:09:26 +00:00
|
|
|
push_image(progress, &root, &image_ref)?;
|
2026-07-05 09:59:51 +00:00
|
|
|
for alias in alias_refs(&image, &alias_tags) {
|
2026-07-29 17:09:26 +00:00
|
|
|
push_image(progress, &root, &alias)?;
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2026-07-29 17:09:26 +00:00
|
|
|
prune_old_dirty_images(&root, &image, &tag);
|
2026-07-29 13:17:57 +00:00
|
|
|
Ok(image_ref)
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn workspace_root() -> Result<PathBuf, String> {
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
let output = swactor_process::command_output(&mut Command::new("git")
|
2026-07-05 09:59:51 +00:00
|
|
|
.args(["rev-parse", "--show-toplevel"])
|
refactor: final filetree shape
Reorganize crates/mvp-system from flat files into domain module trees (chat, node, node_data, observability, orchestration, prompt, staging, transport, worker) with documented mod.rs boundaries, and drop the stale inline spec docs.
- lib.rs: replace ~20 flat mod declarations with one pub-mod-per-domain (chat/node/node_data/observability/orchestration/prompt/staging/transport/worker)
- node/, chat/, observability/, orchestration/, staging/, prompt/, transport/, worker/: add mod.rs files with module-boundary doc comments and re-exports (e.g. chat re-exports run_from_args; orchestration re-exports RunConfig/RunId/GgufSource/TokenizerSource/ProviderKind)
- orchestration: group providers under provider_adapters/{docker_cluster,relay,vastai} and fold engine_builder/, config, run_fsm, run_plan, provisioning, resource_inventory, membership_readiness, and token_endpoint under orchestration/
- transport: consolidate codec registration into transport/codec_registry::register_mvp_actor_codecs (was crate::actors::register_mvp_actor_codecs) and rename actors/codec.rs to transport/json_codec.rs
- rename and relocate files into their domains (arena_manager->node_data/arena, actors/node_agent->node/actor, actors/orchestrator->orchestration/actor, stage_controller->staging/actor, telemetry/dashboard_view/etc->observability/, benchmark_observability->observability::benchmark, edge_establisher->node::edge_lifecycle, prompt_rpc->prompt::rpc) and update all crate:: imports accordingly
- remove the stale crates/mvp-system/specs/*.md (MVP_SYSTEM_MODULE_BOUNDARY_SPEC, mvp_chat, orchestrator) now that module boundaries live in mod.rs docs
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-28 09:04:13 +00:00
|
|
|
.current_dir(env!("CARGO_MANIFEST_DIR"))
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
.stdin(Stdio::null()))
|
2026-07-05 09:59:51 +00:00
|
|
|
.map_err(|e| format!("locate repository root with git: {e}"))?;
|
|
|
|
|
if !output.status.success() {
|
|
|
|
|
return Err(format!(
|
|
|
|
|
"git rev-parse --show-toplevel failed with {}: {}",
|
|
|
|
|
output.status,
|
|
|
|
|
String::from_utf8_lossy(&output.stderr).trim()
|
|
|
|
|
));
|
|
|
|
|
}
|
|
|
|
|
Ok(PathBuf::from(
|
|
|
|
|
String::from_utf8_lossy(&output.stdout).trim(),
|
|
|
|
|
))
|
|
|
|
|
}
|
|
|
|
|
|
fix: faster provisioning, better ssh checks
Speed up VastAI provisioning by creating instances directly from a cached offer pool, and replace best-effort SSH readiness with classified, post-grace bootstrap-failure detection plus a dedicated per-node provider-status monitor.
- vastai_provisioning: provision_one now iterates a cached candidate_pool of offers calling create_instance directly (with host blacklist and failed-host dedup) instead of re-running client.provision; plan_first_wave_offers caches planned_offer_pool/planned_offer_ids for reuse
- vastai_provisioning: add VastAiProviderMonitor (background thread + AtomicBool stop + Drop) spawned per node via the new spawn_provider_monitor trait method, polling instance_status and emitting VastAiProviderStatusObserved/PollRetry/StatusFailure and terminal-start failures
- vastai_provisioning: add classify_ssh_observation (auth_denied/refused/timeout) with spawn_classifying_stderr_reader; spawn_retrying_ssh_bootstrap aborts after POST_GRACE_BOOTSTRAP_FAILURE_LIMIT repeated classified failures past the grace window instead of retrying forever
- vastai_provisioning: ssh_endpoint delegates to client.wait_for_ssh_endpoint; VastAiNode carries run_id/node_id/label/sink and emits structured VastAiLeaseReady/SshEndpointDiscoveryStarted/SshEndpointReady/RuntimeReadyAccepted/ContractCleanup events; LifecyclePolicy is threaded into start_bootstrap
- tools/vastai: add fetch_instance_status and wait_for_ssh_endpoint_with_policy, refactor wait_for_running onto fetch_instance_status, and export both plus ProviderInstanceStatus from lib.rs
- tools/vastai/types: add ProviderInstanceStatus (actual/intended status, status_msg, public_ipaddr, ssh_port, disk_usage) with ssh_endpoint() and From<InstanceStatus>
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-27 08:30:49 +00:00
|
|
|
fn image_version_tag(root: &Path, image_content_hash: &str) -> Result<String, String> {
|
2026-07-29 17:09:26 +00:00
|
|
|
if git_capture(root, &["status", "--porcelain"])?
|
|
|
|
|
.trim()
|
|
|
|
|
.is_empty()
|
|
|
|
|
{
|
2026-07-05 09:59:51 +00:00
|
|
|
let sha = git_capture(root, &["rev-parse", "--short=12", "HEAD"])?;
|
|
|
|
|
Ok(format!("git-{}", sha.trim()))
|
|
|
|
|
} else {
|
fix: faster provisioning, better ssh checks
Speed up VastAI provisioning by creating instances directly from a cached offer pool, and replace best-effort SSH readiness with classified, post-grace bootstrap-failure detection plus a dedicated per-node provider-status monitor.
- vastai_provisioning: provision_one now iterates a cached candidate_pool of offers calling create_instance directly (with host blacklist and failed-host dedup) instead of re-running client.provision; plan_first_wave_offers caches planned_offer_pool/planned_offer_ids for reuse
- vastai_provisioning: add VastAiProviderMonitor (background thread + AtomicBool stop + Drop) spawned per node via the new spawn_provider_monitor trait method, polling instance_status and emitting VastAiProviderStatusObserved/PollRetry/StatusFailure and terminal-start failures
- vastai_provisioning: add classify_ssh_observation (auth_denied/refused/timeout) with spawn_classifying_stderr_reader; spawn_retrying_ssh_bootstrap aborts after POST_GRACE_BOOTSTRAP_FAILURE_LIMIT repeated classified failures past the grace window instead of retrying forever
- vastai_provisioning: ssh_endpoint delegates to client.wait_for_ssh_endpoint; VastAiNode carries run_id/node_id/label/sink and emits structured VastAiLeaseReady/SshEndpointDiscoveryStarted/SshEndpointReady/RuntimeReadyAccepted/ContractCleanup events; LifecyclePolicy is threaded into start_bootstrap
- tools/vastai: add fetch_instance_status and wait_for_ssh_endpoint_with_policy, refactor wait_for_running onto fetch_instance_status, and export both plus ProviderInstanceStatus from lib.rs
- tools/vastai/types: add ProviderInstanceStatus (actual/intended status, status_msg, public_ipaddr, ssh_port, disk_usage) with ssh_endpoint() and From<InstanceStatus>
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-27 08:30:49 +00:00
|
|
|
Ok(format!("dirty-{image_content_hash}"))
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn git_capture(root: &Path, args: &[&str]) -> Result<String, String> {
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
let output = swactor_process::command_output(&mut Command::new("git")
|
2026-07-05 09:59:51 +00:00
|
|
|
.current_dir(root)
|
|
|
|
|
.args(args)
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
.stdin(Stdio::null()))
|
2026-07-05 09:59:51 +00:00
|
|
|
.map_err(|e| format!("run git {}: {e}", args.join(" ")))?;
|
|
|
|
|
if output.status.success() {
|
|
|
|
|
Ok(String::from_utf8_lossy(&output.stdout).to_string())
|
|
|
|
|
} else {
|
|
|
|
|
Err(format!(
|
|
|
|
|
"git {} failed with {}: {}",
|
|
|
|
|
args.join(" "),
|
|
|
|
|
output.status,
|
|
|
|
|
String::from_utf8_lossy(&output.stderr).trim()
|
|
|
|
|
))
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
fix: faster provisioning, better ssh checks
Speed up VastAI provisioning by creating instances directly from a cached offer pool, and replace best-effort SSH readiness with classified, post-grace bootstrap-failure detection plus a dedicated per-node provider-status monitor.
- vastai_provisioning: provision_one now iterates a cached candidate_pool of offers calling create_instance directly (with host blacklist and failed-host dedup) instead of re-running client.provision; plan_first_wave_offers caches planned_offer_pool/planned_offer_ids for reuse
- vastai_provisioning: add VastAiProviderMonitor (background thread + AtomicBool stop + Drop) spawned per node via the new spawn_provider_monitor trait method, polling instance_status and emitting VastAiProviderStatusObserved/PollRetry/StatusFailure and terminal-start failures
- vastai_provisioning: add classify_ssh_observation (auth_denied/refused/timeout) with spawn_classifying_stderr_reader; spawn_retrying_ssh_bootstrap aborts after POST_GRACE_BOOTSTRAP_FAILURE_LIMIT repeated classified failures past the grace window instead of retrying forever
- vastai_provisioning: ssh_endpoint delegates to client.wait_for_ssh_endpoint; VastAiNode carries run_id/node_id/label/sink and emits structured VastAiLeaseReady/SshEndpointDiscoveryStarted/SshEndpointReady/RuntimeReadyAccepted/ContractCleanup events; LifecyclePolicy is threaded into start_bootstrap
- tools/vastai: add fetch_instance_status and wait_for_ssh_endpoint_with_policy, refactor wait_for_running onto fetch_instance_status, and export both plus ProviderInstanceStatus from lib.rs
- tools/vastai/types: add ProviderInstanceStatus (actual/intended status, status_msg, public_ipaddr, ssh_port, disk_usage) with ssh_endpoint() and From<InstanceStatus>
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-27 08:30:49 +00:00
|
|
|
fn node_image_content_hash(
|
|
|
|
|
root: &Path,
|
|
|
|
|
node_bin: &Path,
|
|
|
|
|
base_hash: &str,
|
|
|
|
|
) -> Result<String, String> {
|
|
|
|
|
let mut files = Vec::new();
|
|
|
|
|
for input in NODE_IMAGE_CONTENT_INPUTS {
|
|
|
|
|
let path = root.join(input);
|
|
|
|
|
collect_hash_inputs(root, &path, &mut files)?;
|
|
|
|
|
}
|
|
|
|
|
let node_bin = if node_bin.is_absolute() {
|
|
|
|
|
node_bin.to_path_buf()
|
|
|
|
|
} else {
|
|
|
|
|
root.join(node_bin)
|
|
|
|
|
};
|
|
|
|
|
files.push(relative_path(root, &node_bin)?);
|
|
|
|
|
files.sort();
|
|
|
|
|
files.dedup();
|
|
|
|
|
hash_relative_files_with_salts(root, files, &[("base", base_hash)])
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn content_hash_for_inputs(root: &Path, inputs: &[&str]) -> Result<String, String> {
|
|
|
|
|
let mut files = Vec::new();
|
|
|
|
|
for input in inputs {
|
|
|
|
|
let path = root.join(input);
|
|
|
|
|
collect_hash_inputs(root, &path, &mut files)?;
|
|
|
|
|
}
|
|
|
|
|
files.sort();
|
|
|
|
|
files.dedup();
|
fix: faster provisioning, better ssh checks
Speed up VastAI provisioning by creating instances directly from a cached offer pool, and replace best-effort SSH readiness with classified, post-grace bootstrap-failure detection plus a dedicated per-node provider-status monitor.
- vastai_provisioning: provision_one now iterates a cached candidate_pool of offers calling create_instance directly (with host blacklist and failed-host dedup) instead of re-running client.provision; plan_first_wave_offers caches planned_offer_pool/planned_offer_ids for reuse
- vastai_provisioning: add VastAiProviderMonitor (background thread + AtomicBool stop + Drop) spawned per node via the new spawn_provider_monitor trait method, polling instance_status and emitting VastAiProviderStatusObserved/PollRetry/StatusFailure and terminal-start failures
- vastai_provisioning: add classify_ssh_observation (auth_denied/refused/timeout) with spawn_classifying_stderr_reader; spawn_retrying_ssh_bootstrap aborts after POST_GRACE_BOOTSTRAP_FAILURE_LIMIT repeated classified failures past the grace window instead of retrying forever
- vastai_provisioning: ssh_endpoint delegates to client.wait_for_ssh_endpoint; VastAiNode carries run_id/node_id/label/sink and emits structured VastAiLeaseReady/SshEndpointDiscoveryStarted/SshEndpointReady/RuntimeReadyAccepted/ContractCleanup events; LifecyclePolicy is threaded into start_bootstrap
- tools/vastai: add fetch_instance_status and wait_for_ssh_endpoint_with_policy, refactor wait_for_running onto fetch_instance_status, and export both plus ProviderInstanceStatus from lib.rs
- tools/vastai/types: add ProviderInstanceStatus (actual/intended status, status_msg, public_ipaddr, ssh_port, disk_usage) with ssh_endpoint() and From<InstanceStatus>
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-27 08:30:49 +00:00
|
|
|
hash_relative_files_with_salts(root, files, &[])
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn hash_relative_files_with_salts(
|
|
|
|
|
root: &Path,
|
|
|
|
|
files: Vec<PathBuf>,
|
|
|
|
|
salts: &[(&str, &str)],
|
|
|
|
|
) -> Result<String, String> {
|
2026-07-05 09:59:51 +00:00
|
|
|
let mut hasher = blake3::Hasher::new();
|
fix: faster provisioning, better ssh checks
Speed up VastAI provisioning by creating instances directly from a cached offer pool, and replace best-effort SSH readiness with classified, post-grace bootstrap-failure detection plus a dedicated per-node provider-status monitor.
- vastai_provisioning: provision_one now iterates a cached candidate_pool of offers calling create_instance directly (with host blacklist and failed-host dedup) instead of re-running client.provision; plan_first_wave_offers caches planned_offer_pool/planned_offer_ids for reuse
- vastai_provisioning: add VastAiProviderMonitor (background thread + AtomicBool stop + Drop) spawned per node via the new spawn_provider_monitor trait method, polling instance_status and emitting VastAiProviderStatusObserved/PollRetry/StatusFailure and terminal-start failures
- vastai_provisioning: add classify_ssh_observation (auth_denied/refused/timeout) with spawn_classifying_stderr_reader; spawn_retrying_ssh_bootstrap aborts after POST_GRACE_BOOTSTRAP_FAILURE_LIMIT repeated classified failures past the grace window instead of retrying forever
- vastai_provisioning: ssh_endpoint delegates to client.wait_for_ssh_endpoint; VastAiNode carries run_id/node_id/label/sink and emits structured VastAiLeaseReady/SshEndpointDiscoveryStarted/SshEndpointReady/RuntimeReadyAccepted/ContractCleanup events; LifecyclePolicy is threaded into start_bootstrap
- tools/vastai: add fetch_instance_status and wait_for_ssh_endpoint_with_policy, refactor wait_for_running onto fetch_instance_status, and export both plus ProviderInstanceStatus from lib.rs
- tools/vastai/types: add ProviderInstanceStatus (actual/intended status, status_msg, public_ipaddr, ssh_port, disk_usage) with ssh_endpoint() and From<InstanceStatus>
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-27 08:30:49 +00:00
|
|
|
for (key, value) in salts {
|
|
|
|
|
hasher.update(key.as_bytes());
|
|
|
|
|
hasher.update(b"\0");
|
|
|
|
|
hasher.update(value.as_bytes());
|
|
|
|
|
hasher.update(b"\0");
|
|
|
|
|
}
|
2026-07-05 09:59:51 +00:00
|
|
|
for relative in files {
|
|
|
|
|
let full = root.join(&relative);
|
|
|
|
|
hasher.update(relative.to_string_lossy().as_bytes());
|
|
|
|
|
hasher.update(b"\0");
|
|
|
|
|
hash_file_content(root, &full, &mut hasher)?;
|
|
|
|
|
hasher.update(b"\0");
|
|
|
|
|
}
|
|
|
|
|
let hash = hasher.finalize().to_hex().to_string();
|
|
|
|
|
Ok(hash[..16].to_owned())
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn hash_file_content(root: &Path, path: &Path, hasher: &mut blake3::Hasher) -> Result<(), String> {
|
|
|
|
|
let display = display_workspace_path(root, path);
|
|
|
|
|
let mut file = File::open(path).map_err(|e| format!("open {display}: {e}"))?;
|
|
|
|
|
let mut buf = [0_u8; 64 * 1024];
|
|
|
|
|
loop {
|
|
|
|
|
let n = file
|
|
|
|
|
.read(&mut buf)
|
|
|
|
|
.map_err(|e| format!("read {display}: {e}"))?;
|
|
|
|
|
if n == 0 {
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
hasher.update(&buf[..n]);
|
|
|
|
|
}
|
|
|
|
|
Ok(())
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn collect_hash_inputs(root: &Path, path: &Path, out: &mut Vec<PathBuf>) -> Result<(), String> {
|
|
|
|
|
if !path.exists() {
|
|
|
|
|
return Ok(());
|
|
|
|
|
}
|
|
|
|
|
let display = display_workspace_path(root, path);
|
|
|
|
|
let metadata = fs::metadata(path).map_err(|e| format!("stat {display}: {e}"))?;
|
|
|
|
|
if metadata.is_file() {
|
2026-07-29 17:09:26 +00:00
|
|
|
if !matches!(path.extension().and_then(|ext| ext.to_str()), Some("pyc")) {
|
2026-07-05 09:59:51 +00:00
|
|
|
out.push(relative_path(root, path)?);
|
|
|
|
|
}
|
|
|
|
|
return Ok(());
|
|
|
|
|
}
|
feat(provisioning): add level-triggered cluster reconciler
Introduce a pure, level-triggered reconciler in `crates/provisioning`
that drives a declared cluster shape toward convergence over the
existing node lifecycle, replacing the edge-triggered imperative node
orchestration in `apps/myelin`.
- `reconcile`/`reconcile_node`/`observe`: pure decider and observation
folder with stable logical-node identity, per-attempt operation
identity, and deterministic retry backoff; `ClusterDriver` is the sole
writer of observed state, coalescing triggers, recording operations as
pending before dispatch, and scheduling timed requeues.
- `IdempotentEffectExecutor`: deduplicates submissions by
`(run_id, logical_node_id, attempt)` and runs provider work on the
engine-hosted blocking substrate, never blocking a reconcile pass.
- Myelin integration: `MyelinEffectBackend` bridges `ProvisionPlugin` to
the executor contract; `LocalProcessPlugin`/`LocalDockerPlugin`
provider adapters; `ProvisionedClusterGuard` pumps triggers,
observations, and due operations.
- Retire the imperative acquire/bootstrap/teardown sequencing across
`apps/myelin` orchestration, staging, observability, and provider
adapters in favor of the declarative driver.
- Move the reconciler specification to `docs/specs/archive`.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 07:49:18 +00:00
|
|
|
if !metadata.is_dir()
|
|
|
|
|
|| matches!(
|
|
|
|
|
path.file_name().and_then(|name| name.to_str()),
|
|
|
|
|
Some(".git" | "target" | "__pycache__")
|
|
|
|
|
)
|
|
|
|
|
{
|
2026-07-05 09:59:51 +00:00
|
|
|
return Ok(());
|
|
|
|
|
}
|
|
|
|
|
let entries = fs::read_dir(path).map_err(|e| format!("read dir {display}: {e}"))?;
|
|
|
|
|
for entry in entries {
|
|
|
|
|
let entry = entry.map_err(|e| format!("read dir entry {display}: {e}"))?;
|
|
|
|
|
collect_hash_inputs(root, &entry.path(), out)?;
|
|
|
|
|
}
|
|
|
|
|
Ok(())
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn relative_path(root: &Path, path: &Path) -> Result<PathBuf, String> {
|
|
|
|
|
path.strip_prefix(root).map(Path::to_path_buf).map_err(|e| {
|
|
|
|
|
format!(
|
|
|
|
|
"make {} relative to {}: {e}",
|
|
|
|
|
display_workspace_path(root, path),
|
|
|
|
|
"."
|
|
|
|
|
)
|
|
|
|
|
})
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn display_workspace_path(root: &Path, path: &Path) -> String {
|
|
|
|
|
match path.strip_prefix(root) {
|
|
|
|
|
Ok(relative) if relative.as_os_str().is_empty() => ".".to_owned(),
|
|
|
|
|
Ok(relative) => format!("./{}", relative.display()),
|
|
|
|
|
Err(_) => path.display().to_string(),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn alias_tags(
|
|
|
|
|
image: &ImageName,
|
|
|
|
|
extra_tag: Option<&str>,
|
|
|
|
|
version_tag: &str,
|
|
|
|
|
) -> Result<BTreeSet<String>, String> {
|
|
|
|
|
let mut tags = BTreeSet::new();
|
|
|
|
|
if let Some(tag) = image.requested_tag.as_deref() {
|
|
|
|
|
insert_alias_tag(&mut tags, tag, version_tag)?;
|
|
|
|
|
}
|
|
|
|
|
if let Some(tag) = extra_tag {
|
|
|
|
|
insert_alias_tag(&mut tags, tag, version_tag)?;
|
|
|
|
|
}
|
|
|
|
|
Ok(tags)
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn insert_alias_tag(
|
|
|
|
|
tags: &mut BTreeSet<String>,
|
|
|
|
|
tag: &str,
|
|
|
|
|
version_tag: &str,
|
|
|
|
|
) -> Result<(), String> {
|
|
|
|
|
let tag = tag.trim();
|
|
|
|
|
if tag.is_empty() {
|
|
|
|
|
return Err("node image tag must not be empty".to_owned());
|
|
|
|
|
}
|
|
|
|
|
if tag != version_tag {
|
|
|
|
|
tags.insert(tag.to_owned());
|
|
|
|
|
}
|
|
|
|
|
Ok(())
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn ensure_aliases_local(
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
progress: &mut Option<&mut dyn NodeImageProgressSink>,
|
2026-07-05 09:59:51 +00:00
|
|
|
root: &Path,
|
|
|
|
|
source_ref: &str,
|
|
|
|
|
image: &ImageName,
|
|
|
|
|
alias_tags: &BTreeSet<String>,
|
|
|
|
|
) -> Result<(), String> {
|
|
|
|
|
for alias in alias_refs(image, alias_tags) {
|
|
|
|
|
if alias != source_ref {
|
2026-07-31 08:04:09 +00:00
|
|
|
run_status_command(
|
2026-07-05 09:59:51 +00:00
|
|
|
root,
|
|
|
|
|
"docker",
|
|
|
|
|
&["tag", source_ref, &alias],
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
"tag myelin node image",
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
Some(&alias),
|
2026-07-31 08:04:09 +00:00
|
|
|
progress,
|
2026-07-05 09:59:51 +00:00
|
|
|
)?;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
Ok(())
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn ensure_aliases_for_remote(
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
progress: &mut Option<&mut dyn NodeImageProgressSink>,
|
2026-07-05 09:59:51 +00:00
|
|
|
root: &Path,
|
|
|
|
|
source_ref: &str,
|
|
|
|
|
image: &ImageName,
|
|
|
|
|
alias_tags: &BTreeSet<String>,
|
|
|
|
|
) -> Result<bool, String> {
|
|
|
|
|
if alias_tags.is_empty() {
|
|
|
|
|
return Ok(false);
|
|
|
|
|
}
|
2026-07-29 17:09:26 +00:00
|
|
|
if !docker_image_exists(root, source_ref) {
|
2026-07-31 08:04:09 +00:00
|
|
|
run_status_command(
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
root,
|
|
|
|
|
"docker",
|
|
|
|
|
&["pull", source_ref],
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
"pull myelin node image",
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
Some(source_ref),
|
2026-07-31 08:04:09 +00:00
|
|
|
progress,
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
)?;
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
2026-07-29 17:09:26 +00:00
|
|
|
ensure_aliases_local(progress, root, source_ref, image, alias_tags)?;
|
2026-07-05 09:59:51 +00:00
|
|
|
for alias in alias_refs(image, alias_tags) {
|
2026-07-29 17:09:26 +00:00
|
|
|
push_image(progress, root, &alias)?;
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
|
|
|
|
Ok(true)
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn alias_refs(image: &ImageName, alias_tags: &BTreeSet<String>) -> Vec<String> {
|
|
|
|
|
alias_tags
|
|
|
|
|
.iter()
|
|
|
|
|
.map(|tag| image.ref_for_tag(tag))
|
|
|
|
|
.collect()
|
|
|
|
|
}
|
|
|
|
|
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
fn push_image(
|
|
|
|
|
progress: &mut Option<&mut dyn NodeImageProgressSink>,
|
|
|
|
|
root: &Path,
|
|
|
|
|
image_ref: &str,
|
|
|
|
|
) -> Result<(), String> {
|
2026-07-31 08:04:09 +00:00
|
|
|
run_status_command(
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
root,
|
|
|
|
|
"docker",
|
|
|
|
|
&["push", image_ref],
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
"push myelin node image",
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
Some(image_ref),
|
2026-07-31 08:04:09 +00:00
|
|
|
progress,
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
)
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn docker_image_labels_match(
|
|
|
|
|
root: &Path,
|
|
|
|
|
image_ref: &str,
|
|
|
|
|
expected: &[(&str, &str)],
|
|
|
|
|
) -> Result<bool, String> {
|
2026-07-29 17:09:26 +00:00
|
|
|
let Some(labels) = docker_image_labels(root, image_ref)? else {
|
2026-07-12 06:14:34 +00:00
|
|
|
return Ok(false);
|
|
|
|
|
};
|
|
|
|
|
Ok(expected
|
|
|
|
|
.iter()
|
|
|
|
|
.all(|(key, value)| labels.get(*key).map(String::as_str) == Some(*value)))
|
|
|
|
|
}
|
|
|
|
|
|
2026-07-29 17:09:26 +00:00
|
|
|
fn prune_old_dirty_images(root: &Path, image: &ImageName, keep_tag: &str) {
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
let prune_enabled = std::env::var("MYELIN_NODE_IMAGE_PRUNE")
|
2026-07-29 17:09:26 +00:00
|
|
|
.map(|value| {
|
|
|
|
|
let value = value.trim().to_ascii_lowercase();
|
|
|
|
|
!matches!(value.as_str(), "0" | "false" | "no" | "off")
|
|
|
|
|
})
|
|
|
|
|
.unwrap_or(true);
|
|
|
|
|
if !prune_enabled {
|
2026-07-12 06:14:34 +00:00
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
2026-07-29 17:09:26 +00:00
|
|
|
let tags = match docker_image_tags(root, &image.repository) {
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
Ok(tags) => tags,
|
2026-07-12 06:14:34 +00:00
|
|
|
Err(error) => {
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
eprintln!("myelin-node-image: prune old dirty images skipped: {error}");
|
2026-07-12 06:14:34 +00:00
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
};
|
|
|
|
|
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
let keep_old = std::env::var("MYELIN_NODE_IMAGE_PRUNE_KEEP")
|
2026-07-29 17:09:26 +00:00
|
|
|
.ok()
|
|
|
|
|
.and_then(|value| value.trim().parse::<usize>().ok())
|
|
|
|
|
.unwrap_or(3);
|
2026-07-12 06:14:34 +00:00
|
|
|
let mut retained_old = 0_usize;
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
for (repository, tag) in tags {
|
2026-07-12 06:14:34 +00:00
|
|
|
if repository != image.repository
|
|
|
|
|
|| !tag.starts_with("dirty-")
|
|
|
|
|
|| tag == keep_tag
|
|
|
|
|
|| tag == "<none>"
|
|
|
|
|
{
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
let image_ref = image.ref_for_tag(&tag);
|
2026-07-29 17:09:26 +00:00
|
|
|
let Ok(Some(labels)) = docker_image_labels(root, &image_ref) else {
|
2026-07-12 06:14:34 +00:00
|
|
|
continue;
|
|
|
|
|
};
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
if labels.get(NODE_IMAGE_TAG_LABEL).map(String::as_str) != Some(tag.as_str())
|
2026-07-12 06:14:34 +00:00
|
|
|
|| !labels.contains_key(NODE_IMAGE_SOURCE_HASH_LABEL)
|
2026-08-17 22:51:08 +00:00
|
|
|
|| !labels.contains_key(NODE_IMAGE_AGENT_HASH_LABEL)
|
2026-07-12 06:14:34 +00:00
|
|
|
|| !labels.contains_key(NODE_IMAGE_BASE_HASH_LABEL)
|
|
|
|
|
{
|
|
|
|
|
continue;
|
|
|
|
|
}
|
2026-07-29 17:09:26 +00:00
|
|
|
if docker_image_has_container(root, &image_ref) {
|
2026-07-12 06:14:34 +00:00
|
|
|
eprintln!(
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
"myelin-node-image: prune old dirty image {image_ref} skipped: container exists"
|
2026-07-12 06:14:34 +00:00
|
|
|
);
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
if retained_old < keep_old {
|
|
|
|
|
retained_old += 1;
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
eprintln!("myelin-node-image: prune old dirty image {image_ref}");
|
2026-07-29 17:09:26 +00:00
|
|
|
if let Err(error) = docker_image_remove(root, &image_ref) {
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
eprintln!("myelin-node-image: prune old dirty image {image_ref} skipped: {error}");
|
2026-07-12 06:14:34 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
fn emit_image_reference(
|
|
|
|
|
progress: &mut Option<&mut dyn NodeImageProgressSink>,
|
|
|
|
|
role: &str,
|
|
|
|
|
image_ref: &str,
|
|
|
|
|
) {
|
|
|
|
|
emit_progress(
|
|
|
|
|
progress,
|
|
|
|
|
NodeImageProgressEvent {
|
|
|
|
|
command_label: None,
|
|
|
|
|
image_ref: Some(image_ref.to_owned()),
|
|
|
|
|
elapsed_ms: None,
|
|
|
|
|
kind: NodeImageProgressEventKind::ImageReference {
|
|
|
|
|
role: role.to_owned(),
|
|
|
|
|
image_ref: image_ref.to_owned(),
|
|
|
|
|
},
|
|
|
|
|
},
|
|
|
|
|
);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn emit_progress(
|
|
|
|
|
progress: &mut Option<&mut dyn NodeImageProgressSink>,
|
|
|
|
|
event: NodeImageProgressEvent,
|
|
|
|
|
) {
|
|
|
|
|
if let Some(sink) = progress.as_deref_mut() {
|
|
|
|
|
sink.emit(event);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn emit_command_progress(
|
|
|
|
|
progress: &mut Option<&mut dyn NodeImageProgressSink>,
|
|
|
|
|
label: &str,
|
|
|
|
|
image_ref: Option<&str>,
|
|
|
|
|
elapsed_ms: u128,
|
|
|
|
|
kind: NodeImageProgressEventKind,
|
|
|
|
|
) {
|
|
|
|
|
emit_progress(
|
|
|
|
|
progress,
|
|
|
|
|
NodeImageProgressEvent {
|
|
|
|
|
command_label: Some(label.to_owned()),
|
|
|
|
|
image_ref: image_ref.map(str::to_owned),
|
|
|
|
|
elapsed_ms: Some(elapsed_ms),
|
|
|
|
|
kind,
|
|
|
|
|
},
|
|
|
|
|
);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
enum CommandOutputLine {
|
|
|
|
|
Stdout(String),
|
|
|
|
|
Stderr(String),
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn spawn_line_reader<R>(
|
|
|
|
|
reader: R,
|
|
|
|
|
to_line: fn(String) -> CommandOutputLine,
|
|
|
|
|
tx: mpsc::Sender<CommandOutputLine>,
|
|
|
|
|
) -> thread::JoinHandle<()>
|
|
|
|
|
where
|
|
|
|
|
R: Read + Send + 'static,
|
|
|
|
|
{
|
|
|
|
|
thread::spawn(move || {
|
|
|
|
|
for line in BufReader::new(reader).lines().map_while(Result::ok) {
|
|
|
|
|
if tx.send(to_line(line)).is_err() {
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
})
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn drain_command_lines(
|
|
|
|
|
rx: &mpsc::Receiver<CommandOutputLine>,
|
|
|
|
|
progress: &mut Option<&mut dyn NodeImageProgressSink>,
|
|
|
|
|
label: &str,
|
|
|
|
|
image_ref: Option<&str>,
|
|
|
|
|
started: Instant,
|
|
|
|
|
) {
|
|
|
|
|
while let Ok(line) = rx.try_recv() {
|
|
|
|
|
let kind = match line {
|
|
|
|
|
CommandOutputLine::Stdout(line) => NodeImageProgressEventKind::CommandStdout { line },
|
|
|
|
|
CommandOutputLine::Stderr(line) => NodeImageProgressEventKind::CommandStderr { line },
|
|
|
|
|
};
|
|
|
|
|
emit_command_progress(
|
|
|
|
|
progress,
|
|
|
|
|
label,
|
|
|
|
|
image_ref,
|
|
|
|
|
started.elapsed().as_millis(),
|
|
|
|
|
kind,
|
|
|
|
|
);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2026-07-29 17:09:26 +00:00
|
|
|
fn run_status_command(
|
|
|
|
|
root: &Path,
|
|
|
|
|
program: &str,
|
2026-07-31 08:04:09 +00:00
|
|
|
args: &[&str],
|
2026-07-29 17:09:26 +00:00
|
|
|
label: &str,
|
|
|
|
|
image_ref: Option<&str>,
|
|
|
|
|
progress: &mut Option<&mut dyn NodeImageProgressSink>,
|
|
|
|
|
) -> Result<(), String> {
|
2026-07-31 08:04:09 +00:00
|
|
|
let args: Vec<String> = args.iter().map(|arg| (*arg).to_owned()).collect();
|
refactor: `mvp-system` is now a standalone app, `myelin`
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-01 09:46:55 +00:00
|
|
|
eprintln!("myelin-node-image: {label}");
|
2026-07-29 17:09:26 +00:00
|
|
|
if progress.is_none() {
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
let status = swactor_process::command_status(&mut Command::new(program)
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
.current_dir(root)
|
2026-07-31 08:04:09 +00:00
|
|
|
.args(&args)
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
.stdin(Stdio::null())
|
2026-07-29 17:09:26 +00:00
|
|
|
.stdout(Stdio::inherit())
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
.stderr(Stdio::inherit()))
|
2026-07-29 17:09:26 +00:00
|
|
|
.map_err(|e| format!("run {label}: {e}"))?;
|
|
|
|
|
return if status.success() {
|
|
|
|
|
Ok(())
|
|
|
|
|
} else {
|
|
|
|
|
Err(format!("{label} failed with {status}"))
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
let started = Instant::now();
|
|
|
|
|
emit_command_progress(
|
|
|
|
|
progress,
|
|
|
|
|
label,
|
|
|
|
|
image_ref,
|
|
|
|
|
0,
|
|
|
|
|
NodeImageProgressEventKind::CommandStarted {
|
|
|
|
|
program: program.to_owned(),
|
|
|
|
|
args: args.to_vec(),
|
|
|
|
|
},
|
|
|
|
|
);
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
let mut child = match swactor_process::command_spawn(&mut Command::new(program)
|
2026-07-29 17:09:26 +00:00
|
|
|
.current_dir(root)
|
feat(provisioning): add level-triggered cluster reconciler
Introduce a pure, level-triggered reconciler in `crates/provisioning`
that drives a declared cluster shape toward convergence over the
existing node lifecycle, replacing the edge-triggered imperative node
orchestration in `apps/myelin`.
- `reconcile`/`reconcile_node`/`observe`: pure decider and observation
folder with stable logical-node identity, per-attempt operation
identity, and deterministic retry backoff; `ClusterDriver` is the sole
writer of observed state, coalescing triggers, recording operations as
pending before dispatch, and scheduling timed requeues.
- `IdempotentEffectExecutor`: deduplicates submissions by
`(run_id, logical_node_id, attempt)` and runs provider work on the
engine-hosted blocking substrate, never blocking a reconcile pass.
- Myelin integration: `MyelinEffectBackend` bridges `ProvisionPlugin` to
the executor contract; `LocalProcessPlugin`/`LocalDockerPlugin`
provider adapters; `ProvisionedClusterGuard` pumps triggers,
observations, and due operations.
- Retire the imperative acquire/bootstrap/teardown sequencing across
`apps/myelin` orchestration, staging, observability, and provider
adapters in favor of the declarative driver.
- Move the reconciler specification to `docs/specs/archive`.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 07:49:18 +00:00
|
|
|
.args(&args)
|
2026-07-29 17:09:26 +00:00
|
|
|
.stdin(Stdio::null())
|
|
|
|
|
.stdout(Stdio::piped())
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
.stderr(Stdio::piped()))
|
2026-07-29 17:09:26 +00:00
|
|
|
{
|
|
|
|
|
Ok(child) => child,
|
|
|
|
|
Err(error) => {
|
|
|
|
|
emit_command_progress(
|
|
|
|
|
progress,
|
|
|
|
|
label,
|
|
|
|
|
image_ref,
|
|
|
|
|
started.elapsed().as_millis(),
|
|
|
|
|
NodeImageProgressEventKind::CommandExited {
|
|
|
|
|
status: format!("spawn error: {error}"),
|
|
|
|
|
code: None,
|
|
|
|
|
success: false,
|
|
|
|
|
},
|
|
|
|
|
);
|
|
|
|
|
return Err(format!("run {label}: {error}"));
|
|
|
|
|
}
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
let (tx, rx) = mpsc::channel();
|
|
|
|
|
let mut readers = Vec::new();
|
|
|
|
|
if let Some(stdout) = child.stdout.take() {
|
|
|
|
|
readers.push(spawn_line_reader(
|
|
|
|
|
stdout,
|
|
|
|
|
CommandOutputLine::Stdout,
|
|
|
|
|
tx.clone(),
|
|
|
|
|
));
|
|
|
|
|
}
|
|
|
|
|
if let Some(stderr) = child.stderr.take() {
|
|
|
|
|
readers.push(spawn_line_reader(
|
|
|
|
|
stderr,
|
|
|
|
|
CommandOutputLine::Stderr,
|
|
|
|
|
tx.clone(),
|
|
|
|
|
));
|
|
|
|
|
}
|
|
|
|
|
drop(tx);
|
|
|
|
|
|
|
|
|
|
let status = loop {
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
match swactor_process::child_try_wait(&mut child) {
|
2026-07-29 17:09:26 +00:00
|
|
|
Ok(Some(status)) => break status,
|
|
|
|
|
Ok(None) => {
|
|
|
|
|
drain_command_lines(&rx, progress, label, image_ref, started);
|
|
|
|
|
thread::sleep(Duration::from_millis(10));
|
|
|
|
|
}
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
Err(error) => {
|
2026-07-29 17:09:26 +00:00
|
|
|
drain_command_lines(&rx, progress, label, image_ref, started);
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
emit_command_progress(
|
|
|
|
|
progress,
|
|
|
|
|
label,
|
|
|
|
|
image_ref,
|
|
|
|
|
started.elapsed().as_millis(),
|
|
|
|
|
NodeImageProgressEventKind::CommandExited {
|
2026-07-29 17:09:26 +00:00
|
|
|
status: format!("wait error: {error}"),
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
code: None,
|
|
|
|
|
success: false,
|
|
|
|
|
},
|
|
|
|
|
);
|
|
|
|
|
return Err(format!("run {label}: {error}"));
|
|
|
|
|
}
|
|
|
|
|
}
|
2026-07-29 17:09:26 +00:00
|
|
|
};
|
|
|
|
|
for reader in readers {
|
|
|
|
|
let _ = reader.join();
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
}
|
2026-07-29 17:09:26 +00:00
|
|
|
drain_command_lines(&rx, progress, label, image_ref, started);
|
|
|
|
|
let status_text = status.to_string();
|
|
|
|
|
let success = status.success();
|
|
|
|
|
emit_command_progress(
|
|
|
|
|
progress,
|
|
|
|
|
label,
|
|
|
|
|
image_ref,
|
|
|
|
|
started.elapsed().as_millis(),
|
|
|
|
|
NodeImageProgressEventKind::CommandExited {
|
|
|
|
|
status: status_text.clone(),
|
|
|
|
|
code: status.code(),
|
|
|
|
|
success,
|
|
|
|
|
},
|
|
|
|
|
);
|
|
|
|
|
if success {
|
|
|
|
|
Ok(())
|
|
|
|
|
} else {
|
|
|
|
|
Err(format!("{label} failed with {status_text}"))
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
}
|
2026-07-29 17:09:26 +00:00
|
|
|
}
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
|
2026-07-29 17:09:26 +00:00
|
|
|
fn docker_image_exists(root: &Path, image_ref: &str) -> bool {
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
swactor_process::command_status(&mut Command::new("docker")
|
2026-07-29 17:09:26 +00:00
|
|
|
.current_dir(root)
|
|
|
|
|
.args(["image", "inspect", image_ref])
|
|
|
|
|
.stdin(Stdio::null())
|
|
|
|
|
.stdout(Stdio::null())
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
.stderr(Stdio::null()))
|
2026-07-29 17:09:26 +00:00
|
|
|
.map(|status| status.success())
|
|
|
|
|
.unwrap_or(false)
|
|
|
|
|
}
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
|
2026-07-29 17:09:26 +00:00
|
|
|
fn docker_image_labels(
|
|
|
|
|
root: &Path,
|
|
|
|
|
image_ref: &str,
|
|
|
|
|
) -> Result<Option<BTreeMap<String, String>>, String> {
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
let output = swactor_process::command_output(&mut Command::new("docker")
|
2026-07-29 17:09:26 +00:00
|
|
|
.current_dir(root)
|
|
|
|
|
.args([
|
|
|
|
|
"image",
|
|
|
|
|
"inspect",
|
|
|
|
|
"--format",
|
|
|
|
|
"{{ json .Config.Labels }}",
|
|
|
|
|
image_ref,
|
|
|
|
|
])
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
.stdin(Stdio::null()))
|
2026-07-29 17:09:26 +00:00
|
|
|
.map_err(|e| format!("inspect docker image {image_ref}: {e}"))?;
|
|
|
|
|
if !output.status.success() {
|
|
|
|
|
return Ok(None);
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
}
|
2026-07-29 17:09:26 +00:00
|
|
|
let stdout = String::from_utf8_lossy(&output.stdout);
|
|
|
|
|
let labels: Option<BTreeMap<String, String>> = serde_json::from_str(stdout.trim())
|
|
|
|
|
.map_err(|e| format!("parse docker labels for {image_ref}: {e}"))?;
|
|
|
|
|
Ok(Some(labels.unwrap_or_default()))
|
|
|
|
|
}
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
|
2026-07-29 17:09:26 +00:00
|
|
|
fn docker_manifest_exists(root: &Path, image_ref: &str) -> bool {
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
swactor_process::command_status(&mut Command::new("docker")
|
2026-07-29 17:09:26 +00:00
|
|
|
.current_dir(root)
|
|
|
|
|
.args(["manifest", "inspect", image_ref])
|
|
|
|
|
.stdin(Stdio::null())
|
|
|
|
|
.stdout(Stdio::null())
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
.stderr(Stdio::null()))
|
2026-07-29 17:09:26 +00:00
|
|
|
.map(|status| status.success())
|
|
|
|
|
.unwrap_or(false)
|
|
|
|
|
}
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
|
2026-07-29 17:09:26 +00:00
|
|
|
fn docker_image_has_container(root: &Path, image_ref: &str) -> bool {
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
swactor_process::command_output(&mut Command::new("docker")
|
2026-07-29 17:09:26 +00:00
|
|
|
.current_dir(root)
|
|
|
|
|
.args([
|
|
|
|
|
"ps",
|
|
|
|
|
"-a",
|
|
|
|
|
"--filter",
|
|
|
|
|
&format!("ancestor={image_ref}"),
|
|
|
|
|
"--format",
|
|
|
|
|
"{{.ID}}",
|
|
|
|
|
])
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
.stdin(Stdio::null()))
|
2026-07-29 17:09:26 +00:00
|
|
|
.map(|output| output.status.success() && !output.stdout.is_empty())
|
|
|
|
|
.unwrap_or(true)
|
|
|
|
|
}
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
|
2026-07-29 17:09:26 +00:00
|
|
|
fn docker_image_tags(root: &Path, repository: &str) -> Result<Vec<(String, String)>, String> {
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
let output = swactor_process::command_output(&mut Command::new("docker")
|
2026-07-29 17:09:26 +00:00
|
|
|
.current_dir(root)
|
|
|
|
|
.args([
|
|
|
|
|
"image",
|
|
|
|
|
"ls",
|
|
|
|
|
"--format",
|
|
|
|
|
"{{.Repository}}\t{{.Tag}}",
|
|
|
|
|
repository,
|
|
|
|
|
])
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
.stdin(Stdio::null()))
|
2026-07-29 17:09:26 +00:00
|
|
|
.map_err(|error| format!("docker image ls failed: {error}"))?;
|
|
|
|
|
if !output.status.success() {
|
|
|
|
|
return Err(format!("docker image ls failed with {}", output.status));
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
2026-07-29 17:09:26 +00:00
|
|
|
let stdout = String::from_utf8_lossy(&output.stdout);
|
|
|
|
|
Ok(stdout
|
|
|
|
|
.lines()
|
|
|
|
|
.filter_map(|line| {
|
|
|
|
|
let (repository, tag) = line.split_once('\t')?;
|
|
|
|
|
Some((repository.to_owned(), tag.to_owned()))
|
|
|
|
|
})
|
|
|
|
|
.collect())
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
|
|
|
|
|
2026-07-29 17:09:26 +00:00
|
|
|
fn docker_image_remove(root: &Path, image_ref: &str) -> Result<(), String> {
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
let status = swactor_process::command_status(&mut Command::new("docker")
|
2026-07-29 17:09:26 +00:00
|
|
|
.current_dir(root)
|
|
|
|
|
.args(["image", "rm", image_ref])
|
|
|
|
|
.stdin(Stdio::null())
|
|
|
|
|
.stdout(Stdio::null())
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
.stderr(Stdio::null()))
|
2026-07-29 17:09:26 +00:00
|
|
|
.map_err(|error| format!("docker image rm failed: {error}"))?;
|
|
|
|
|
if status.success() {
|
|
|
|
|
Ok(())
|
|
|
|
|
} else {
|
|
|
|
|
Err(format!("docker image rm failed with {status}"))
|
|
|
|
|
}
|
2026-07-05 09:59:51 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#[derive(Clone, Debug)]
|
|
|
|
|
struct ImageName {
|
|
|
|
|
repository: String,
|
|
|
|
|
requested_tag: Option<String>,
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl ImageName {
|
|
|
|
|
fn parse(raw: &str) -> Result<Self, String> {
|
|
|
|
|
let raw = raw.trim();
|
|
|
|
|
if raw.is_empty() {
|
|
|
|
|
return Err("node image must not be empty".to_owned());
|
|
|
|
|
}
|
|
|
|
|
if raw.contains('@') {
|
|
|
|
|
return Err(format!(
|
|
|
|
|
"node image {raw:?} uses a digest; use a repository/tag base for image preparation"
|
|
|
|
|
));
|
|
|
|
|
}
|
|
|
|
|
let last_slash = raw.rfind('/');
|
|
|
|
|
let last_colon = raw.rfind(':');
|
|
|
|
|
let has_tag = match (last_slash, last_colon) {
|
|
|
|
|
(_, None) => false,
|
|
|
|
|
(None, Some(_)) => true,
|
|
|
|
|
(Some(slash), Some(colon)) => colon > slash,
|
|
|
|
|
};
|
|
|
|
|
let (repository, requested_tag) = if has_tag {
|
|
|
|
|
let colon = last_colon.expect("has tag colon");
|
|
|
|
|
let repository = raw[..colon].to_owned();
|
|
|
|
|
let tag = raw[colon + 1..].to_owned();
|
|
|
|
|
if tag.is_empty() {
|
|
|
|
|
return Err(format!("node image {raw:?} has an empty tag"));
|
|
|
|
|
}
|
|
|
|
|
(repository, Some(tag))
|
|
|
|
|
} else {
|
|
|
|
|
(raw.to_owned(), None)
|
|
|
|
|
};
|
|
|
|
|
if repository.is_empty() {
|
|
|
|
|
return Err(format!("node image {raw:?} has an empty repository"));
|
|
|
|
|
}
|
|
|
|
|
Ok(Self {
|
|
|
|
|
repository,
|
feat: per-stage GGUF weight sharding and deploy hardening
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-26 09:01:18 +00:00
|
|
|
|
2026-07-05 09:59:51 +00:00
|
|
|
requested_tag,
|
|
|
|
|
})
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn ref_for_tag(&self, tag: &str) -> String {
|
|
|
|
|
format!("{}:{tag}", self.repository)
|
|
|
|
|
}
|
|
|
|
|
}
|