Zachery Aaron Shores-Chmielewski
c6af8e0a5d
Complete an 8-stage pipeline-parallel run over VastAI by provisioning stages high-to-low, adding per-stage/per-step metrics, host anti-colocation, and provider state-timeout guardrails. - orchestrator_app: select the next weight-load stage by max index (provision stages high-to-low for parallel spread), add a throttled "loaded N of M; waiting on stage X" stage_provision_wait headline, and surface min_compute_cap/state_timeout_secs in the config dump. - orchestrator_app: enrich pipeline_token_in/out and tokenizer_decode events with token_count/token_ids/generated_index. - worker_node: add timing metrics across the data path (helper_execute_ms, egress_ring_read_ms, send_ms, ingress_ring_write_ms, object_load_ms), refactor take_complete_ingress_record into IngressRecordBytes (object_id/sequence/extent/flags), and emit a new object_loaded event. - vastai_provisioning: track leased host_ids and blacklist already-leased hosts in later ProvisionRequests so stages don't co-locate, and tag SSH-bootstrap retry logs with the attempt number. - tools/vastai: add min_compute_cap (PP_MIN_COMPUTE_CAP) filter/search query and a LifecyclePolicy state_timeout (PP_STATE_TIMEOUT_SECS) that fails instances stuck in a non-running status instead of polling forever. - xtask: raise the check timeout to 1800s/30s grace, drop --skip-rebuild for VastAI, aggregate per-stage StepExecuted metrics, add a vastai summary section, and write failure artifacts on abort. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
79 lines
2.3 KiB
Rust
79 lines
2.3 KiB
Rust
pub use swactor_vastai::SelectionPolicy;
|
|
use swactor_vastai::{Offer, VastClient};
|
|
|
|
#[derive(Clone, Debug, PartialEq)]
|
|
pub struct OfferPreview {
|
|
pub offer_id: u64,
|
|
pub host_id: Option<u64>,
|
|
pub gpu_name: String,
|
|
pub gpu_ram_mb: Option<u64>,
|
|
pub dollars_per_hour: f64,
|
|
}
|
|
|
|
impl OfferPreview {
|
|
pub fn from_offer(offer: Offer) -> Self {
|
|
Self {
|
|
offer_id: offer.id,
|
|
host_id: offer.host_id,
|
|
gpu_name: offer.gpu_name,
|
|
gpu_ram_mb: offer.gpu_ram.map(|gb| (gb * 1024.0).round() as u64),
|
|
dollars_per_hour: offer.dph_total,
|
|
}
|
|
}
|
|
}
|
|
|
|
pub trait OfferPreviewer {
|
|
fn preview(&self, api_key: &str, policy: &SelectionPolicy) -> Result<OfferPreview, String>;
|
|
}
|
|
|
|
#[derive(Clone, Copy, Debug, Default)]
|
|
pub struct VastAiOfferPreviewer;
|
|
|
|
impl OfferPreviewer for VastAiOfferPreviewer {
|
|
fn preview(&self, api_key: &str, policy: &SelectionPolicy) -> Result<OfferPreview, String> {
|
|
let runtime = tokio::runtime::Builder::new_current_thread()
|
|
.enable_all()
|
|
.build()
|
|
.map_err(|e| format!("vastai offer preview runtime: {e}"))?;
|
|
let client = VastClient::new(api_key.to_owned());
|
|
let offers = runtime.block_on(client.search_offers(policy, 1))?;
|
|
let offer = offers
|
|
.into_iter()
|
|
.next()
|
|
.ok_or_else(|| "no Vast.ai offers available".to_owned())?;
|
|
Ok(OfferPreview::from_offer(offer))
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
use swactor_vastai::Offer;
|
|
|
|
#[test]
|
|
fn vastai_offer_preview_preserves_offer_identity_price_and_vram_megabytes() {
|
|
let preview = OfferPreview::from_offer(Offer {
|
|
id: 42,
|
|
gpu_name: "RTX 4090".to_owned(),
|
|
dph_total: 0.375,
|
|
gpu_ram: Some(23.5),
|
|
compute_cap: 890,
|
|
geolocation: Some("US".to_owned()),
|
|
inet_down_cost_per_tb: 0.0,
|
|
inet_up_cost_per_tb: 0.0,
|
|
host_id: Some(9001),
|
|
verification: Some("verified".to_owned()),
|
|
});
|
|
|
|
assert_eq!(
|
|
preview,
|
|
OfferPreview {
|
|
offer_id: 42,
|
|
host_id: Some(9001),
|
|
gpu_name: "RTX 4090".to_owned(),
|
|
gpu_ram_mb: Some(24_064),
|
|
dollars_per_hour: 0.375,
|
|
}
|
|
);
|
|
}
|
|
}
|