feat(engine): substrate-neutral execution engine abstraction
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
2026-08-10 20:23:03 +00:00
|
|
|
//! Shared probe actors and bounded-wait helpers for the engine contract tests.
|
|
|
|
|
//!
|
|
|
|
|
//! Imports only public `swactor` APIs and exposes no private engine state. See
|
|
|
|
|
//! `ENGINE_SPEC.md`.
|
|
|
|
|
|
|
|
|
|
#![allow(dead_code)]
|
|
|
|
|
|
|
|
|
|
use std::sync::Arc;
|
|
|
|
|
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
|
|
|
|
|
use std::time::{Duration, Instant};
|
|
|
|
|
|
|
|
|
|
use swactor::actor::{ActorInterface, Ctx};
|
2026-08-11 12:08:06 +00:00
|
|
|
use swactor::runtime::{Runtime, RuntimeConfig, RuntimeParts};
|
Enforce actor-owned Myelin control flow
Add a repository-owned rustc wrapper that enforces execution ownership and dependency boundaries during ordinary Cargo commands, with compile-pass and compile-fail policy contracts.
Move scheduling, timers, provider polling, provisioning, recovery, supervision, and shutdown decisions behind engine and actor APIs. Add deterministic component properties, stateful Myelin lifecycle coverage, persisted regression cases, and the bounded CI workflow.
Tighten resource ownership by cancelling telemetry collectors, terminating reply observers, bounding dashboard projections, and releasing process file descriptors, child observers, and inode-verified Unix socket paths on every exit path.
2026-08-19 21:38:14 +00:00
|
|
|
use swactor_engine::SteppingBackend;
|
2026-08-11 12:08:06 +00:00
|
|
|
|
|
|
|
|
pub fn runtime_parts(config: RuntimeConfig) -> (RuntimeParts, Runtime) {
|
|
|
|
|
let parts = RuntimeParts::new(config);
|
|
|
|
|
let runtime = parts.runtime().clone();
|
|
|
|
|
(parts, runtime)
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
pub fn default_runtime_parts() -> (RuntimeParts, Runtime) {
|
|
|
|
|
runtime_parts(RuntimeConfig::default())
|
|
|
|
|
}
|
|
|
|
|
pub fn runtime_parts_with_workers(worker_count: usize) -> (RuntimeParts, Runtime) {
|
|
|
|
|
let mut config = RuntimeConfig::default();
|
|
|
|
|
config.worker_count = worker_count;
|
|
|
|
|
runtime_parts(config)
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
pub fn default_parts() -> RuntimeParts {
|
|
|
|
|
RuntimeParts::new(RuntimeConfig::default())
|
|
|
|
|
}
|
|
|
|
|
|
feat(engine): substrate-neutral execution engine abstraction
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
2026-08-10 20:23:03 +00:00
|
|
|
// ── Probe message ───────────────────────────────────────────────────────────
|
|
|
|
|
|
|
|
|
|
/// A minimal message delivered to probe actors.
|
|
|
|
|
#[derive(Clone, Debug)]
|
|
|
|
|
pub struct Probe;
|
|
|
|
|
|
|
|
|
|
// ── Probe actors ────────────────────────────────────────────────────────────
|
|
|
|
|
|
|
|
|
|
/// Records how many messages it has received into a shared counter.
|
|
|
|
|
pub struct RecordingProbe {
|
|
|
|
|
pub received: Arc<AtomicUsize>,
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl ActorInterface for RecordingProbe {
|
|
|
|
|
type Incoming = Probe;
|
|
|
|
|
type Response = ();
|
|
|
|
|
fn handle(&mut self, _ctx: &Ctx, _msg: Probe) {
|
|
|
|
|
self.received.fetch_add(1, Ordering::SeqCst);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Detects concurrent or reentrant handler entry. A violation is recorded if
|
|
|
|
|
/// `handle` is entered while a previous invocation is still in flight — which
|
|
|
|
|
/// can only happen if two ticks run the same worker concurrently.
|
|
|
|
|
pub struct ReentrancyGuardProbe {
|
|
|
|
|
pub entered: Arc<AtomicBool>,
|
|
|
|
|
pub violations: Arc<AtomicUsize>,
|
|
|
|
|
pub handled: Arc<AtomicUsize>,
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl ActorInterface for ReentrancyGuardProbe {
|
|
|
|
|
type Incoming = Probe;
|
|
|
|
|
type Response = ();
|
|
|
|
|
fn handle(&mut self, _ctx: &Ctx, _msg: Probe) {
|
|
|
|
|
if self.entered.swap(true, Ordering::SeqCst) {
|
|
|
|
|
self.violations.fetch_add(1, Ordering::SeqCst);
|
|
|
|
|
}
|
|
|
|
|
self.handled.fetch_add(1, Ordering::SeqCst);
|
|
|
|
|
self.entered.store(false, Ordering::SeqCst);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// ── Wait helpers ────────────────────────────────────────────────────────────
|
|
|
|
|
|
|
|
|
|
/// Block until `cond` holds, polling every 2 ms up to `timeout`. Returns the
|
|
|
|
|
/// final value of `cond` (true on success).
|
|
|
|
|
pub fn wait_for<F: Fn() -> bool>(cond: F, timeout: Duration) -> bool {
|
|
|
|
|
const POLL: Duration = Duration::from_millis(2);
|
|
|
|
|
let deadline = Instant::now() + timeout;
|
|
|
|
|
loop {
|
|
|
|
|
if cond() {
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
if Instant::now() >= deadline {
|
|
|
|
|
return cond();
|
|
|
|
|
}
|
|
|
|
|
std::thread::sleep(POLL);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// A substrate-agnostic cooperative yield: suspend the current task for one
|
|
|
|
|
/// scheduler turn (giving other engine work — including the core driver — a
|
|
|
|
|
/// chance to run), then resume. Uses only `std`, so it works on any substrate
|
|
|
|
|
/// without coupling the test to Tokio.
|
|
|
|
|
pub async fn yield_once() {
|
|
|
|
|
// The closure is stored inside `poll_fn`'s future and polled via `&mut`,
|
|
|
|
|
// so its captured `yielded` flag persists across polls: the first poll
|
|
|
|
|
// reschedules and suspends, the next poll resumes.
|
|
|
|
|
let mut yielded = false;
|
|
|
|
|
std::future::poll_fn(move |cx| {
|
|
|
|
|
if yielded {
|
|
|
|
|
std::task::Poll::Ready(())
|
|
|
|
|
} else {
|
|
|
|
|
yielded = true;
|
|
|
|
|
cx.waker().wake_by_ref();
|
|
|
|
|
std::task::Poll::Pending
|
|
|
|
|
}
|
|
|
|
|
})
|
|
|
|
|
.await;
|
|
|
|
|
}
|
Enforce actor-owned Myelin control flow
Add a repository-owned rustc wrapper that enforces execution ownership and dependency boundaries during ordinary Cargo commands, with compile-pass and compile-fail policy contracts.
Move scheduling, timers, provider polling, provisioning, recovery, supervision, and shutdown decisions behind engine and actor APIs. Add deterministic component properties, stateful Myelin lifecycle coverage, persisted regression cases, and the bounded CI workflow.
Tighten resource ownership by cancelling telemetry collectors, terminating reply observers, bounding dashboard projections, and releasing process file descriptors, child observers, and inode-verified Unix socket paths on every exit path.
2026-08-19 21:38:14 +00:00
|
|
|
|
|
|
|
|
pub fn drive_steps(backend: &SteppingBackend, count: usize) {
|
|
|
|
|
for _ in 0..count {
|
|
|
|
|
backend.step();
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
pub fn advance_and_drive(backend: &SteppingBackend, duration: Duration, count: usize) {
|
|
|
|
|
backend.advance_time(duration);
|
|
|
|
|
drive_steps(backend, count);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
pub fn assert_no_poison(runtime: &Runtime) {
|
|
|
|
|
let stats = runtime.stats();
|
|
|
|
|
let panics = stats
|
|
|
|
|
.workers
|
|
|
|
|
.iter()
|
|
|
|
|
.map(|worker| worker.panics)
|
|
|
|
|
.sum::<u64>();
|
|
|
|
|
let poisoned = stats
|
|
|
|
|
.actor_details
|
|
|
|
|
.iter()
|
|
|
|
|
.filter(|actor| actor.poisoned)
|
|
|
|
|
.collect::<Vec<_>>();
|
|
|
|
|
assert!(
|
|
|
|
|
panics == 0 && poisoned.is_empty(),
|
|
|
|
|
"runtime contains poisoned actors: worker_panics={panics}, poisoned={poisoned:?}, \
|
|
|
|
|
actors={:?}, details={:?}",
|
|
|
|
|
stats.actors,
|
|
|
|
|
stats.actor_details,
|
|
|
|
|
);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
pub fn assert_actor_delta_at_most(runtime: &Runtime, baseline: usize, limit: usize) {
|
|
|
|
|
let stats = runtime.stats();
|
|
|
|
|
assert!(
|
|
|
|
|
stats.actors.len() <= baseline.saturating_add(limit),
|
|
|
|
|
"actor count grew from {baseline} to {}, limit={limit}: {:?}",
|
|
|
|
|
stats.actors.len(),
|
|
|
|
|
stats.actors,
|
|
|
|
|
);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
pub fn assert_mailboxes_drained(runtime: &Runtime) {
|
|
|
|
|
let stats = runtime.stats();
|
|
|
|
|
let mailbox_depth = stats
|
|
|
|
|
.workers
|
|
|
|
|
.iter()
|
|
|
|
|
.map(|worker| worker.mailbox_depth)
|
|
|
|
|
.sum::<usize>();
|
|
|
|
|
assert_eq!(
|
|
|
|
|
mailbox_depth, 0,
|
|
|
|
|
"mailboxes did not drain: workers={:?}, details={:?}",
|
|
|
|
|
stats.workers, stats.actor_details,
|
|
|
|
|
);
|
|
|
|
|
}
|