test(provisioning): stateful conformance kit for reconciler and plugins
Replace pointwise scenario testing with a reusable conformance kit in
tests/common: a deterministic trace harness (input alphabet, seeded
generator, naive shrinker), an invariant oracle covering twenty black-box
guarantees (identity, correlation, dead-hold, attempt-fact ownership,
quiescence no-op, monotonic generation, fair convergence, bounded
replacement), and a fair-scheduler tail asserting eventual reconciliation.
Three conformance levels run the same battery:
- FakeBackend: the reference in-memory substrate (256 seeds x 2 modes)
- PluginBackendAdapter over FakePlugin: seam contracts plus the battery
- ProcessPlugin: real child processes, faults as real signals/errors;
"no double-create" and "converged leaks nothing" verified by counting
live PIDs (16 seeds)
Also documents two seam findings the battery surfaced: ProvisionPlugin
cannot express ambiguity (kit convention: AMBIGUOUS_FAULT_MARKER error
reclassified by the adapter; definite classification leaks provider
resources) and spawn_effect closures form a spawner Arc cycle that leaks
backends under queue-based spawners (kit breaks it at harness drop).
2026-08-14 15:21:46 +00:00
|
|
|
//! Process-level conformance: a `ProvisionPlugin` whose resources are
|
|
|
|
|
//! real OS processes (`sleep infinity` children). The kit battery and
|
|
|
|
|
//! seam contracts run against it, so "no double-create", "destroy
|
|
|
|
|
//! releases", "ambiguous create adopts", and "converged leaks nothing"
|
|
|
|
|
//! are verified by counting actual live PIDs.
|
|
|
|
|
|
|
|
|
|
mod common;
|
|
|
|
|
|
|
|
|
|
use std::collections::{BTreeMap, VecDeque};
|
|
|
|
|
use std::process::{Child, Command};
|
|
|
|
|
use std::sync::Arc;
|
|
|
|
|
|
|
|
|
|
use parking_lot::Mutex;
|
2026-08-17 22:51:08 +00:00
|
|
|
use provisioning::plugin::{NodeProvisionSpec, PluginNodeHandle, PluginSink, ProvisionPlugin};
|
Enforce actor-owned Myelin control flow
Add a repository-owned rustc wrapper that enforces execution ownership and dependency boundaries during ordinary Cargo commands, with compile-pass and compile-fail policy contracts.
Move scheduling, timers, provider polling, provisioning, recovery, supervision, and shutdown decisions behind engine and actor APIs. Add deterministic component properties, stateful Myelin lifecycle coverage, persisted regression cases, and the bounded CI workflow.
Tighten resource ownership by cancelling telemetry collectors, terminating reply observers, bounding dashboard projections, and releasing process file descriptors, child observers, and inode-verified Unix socket paths on every exit path.
2026-08-19 21:38:14 +00:00
|
|
|
use swactor_process::{child_kill, child_wait, command_spawn};
|
test(provisioning): stateful conformance kit for reconciler and plugins
Replace pointwise scenario testing with a reusable conformance kit in
tests/common: a deterministic trace harness (input alphabet, seeded
generator, naive shrinker), an invariant oracle covering twenty black-box
guarantees (identity, correlation, dead-hold, attempt-fact ownership,
quiescence no-op, monotonic generation, fair convergence, bounded
replacement), and a fair-scheduler tail asserting eventual reconciliation.
Three conformance levels run the same battery:
- FakeBackend: the reference in-memory substrate (256 seeds x 2 modes)
- PluginBackendAdapter over FakePlugin: seam contracts plus the battery
- ProcessPlugin: real child processes, faults as real signals/errors;
"no double-create" and "converged leaks nothing" verified by counting
live PIDs (16 seeds)
Also documents two seam findings the battery surfaced: ProvisionPlugin
cannot express ambiguity (kit convention: AMBIGUOUS_FAULT_MARKER error
reclassified by the adapter; definite classification leaks provider
resources) and spawn_effect closures form a spawner Arc cycle that leaks
backends under queue-based spawners (kit breaks it at harness drop).
2026-08-14 15:21:46 +00:00
|
|
|
|
|
|
|
|
use common::{
|
2026-08-17 22:51:08 +00:00
|
|
|
AMBIGUOUS_FAULT_MARKER, Fault, PluginBackendAdapter, TestablePlugin, assert_plugin_contracts,
|
|
|
|
|
run_trace_battery,
|
test(provisioning): stateful conformance kit for reconciler and plugins
Replace pointwise scenario testing with a reusable conformance kit in
tests/common: a deterministic trace harness (input alphabet, seeded
generator, naive shrinker), an invariant oracle covering twenty black-box
guarantees (identity, correlation, dead-hold, attempt-fact ownership,
quiescence no-op, monotonic generation, fair convergence, bounded
replacement), and a fair-scheduler tail asserting eventual reconciliation.
Three conformance levels run the same battery:
- FakeBackend: the reference in-memory substrate (256 seeds x 2 modes)
- PluginBackendAdapter over FakePlugin: seam contracts plus the battery
- ProcessPlugin: real child processes, faults as real signals/errors;
"no double-create" and "converged leaks nothing" verified by counting
live PIDs (16 seeds)
Also documents two seam findings the battery surfaced: ProvisionPlugin
cannot express ambiguity (kit convention: AMBIGUOUS_FAULT_MARKER error
reclassified by the adapter; definite classification leaks provider
resources) and spawn_effect closures form a spawner Arc cycle that leaks
backends under queue-based spawners (kit breaks it at harness drop).
2026-08-14 15:21:46 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
struct ProcessPluginState {
|
|
|
|
|
faults: VecDeque<Fault>,
|
|
|
|
|
/// attempt -> live child, present until stopped.
|
|
|
|
|
children: BTreeMap<u64, Child>,
|
|
|
|
|
created: usize,
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// A process provisioner: create spawns a real child keyed by attempt,
|
|
|
|
|
/// stop kills and reaps it, ambiguous faults spawn-then-fail.
|
|
|
|
|
struct ProcessPlugin {
|
|
|
|
|
state: Arc<Mutex<ProcessPluginState>>,
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl Default for ProcessPlugin {
|
|
|
|
|
fn default() -> Self {
|
|
|
|
|
Self {
|
|
|
|
|
state: Arc::new(Mutex::new(ProcessPluginState {
|
|
|
|
|
faults: VecDeque::new(),
|
|
|
|
|
children: BTreeMap::new(),
|
|
|
|
|
created: 0,
|
|
|
|
|
})),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl Drop for ProcessPlugin {
|
|
|
|
|
fn drop(&mut self) {
|
|
|
|
|
// CI hygiene: never leave children behind, even on failure.
|
|
|
|
|
let mut state = self.state.lock();
|
|
|
|
|
let children: Vec<Child> = std::mem::take(&mut state.children).into_values().collect();
|
|
|
|
|
for mut child in children {
|
Enforce actor-owned Myelin control flow
Add a repository-owned rustc wrapper that enforces execution ownership and dependency boundaries during ordinary Cargo commands, with compile-pass and compile-fail policy contracts.
Move scheduling, timers, provider polling, provisioning, recovery, supervision, and shutdown decisions behind engine and actor APIs. Add deterministic component properties, stateful Myelin lifecycle coverage, persisted regression cases, and the bounded CI workflow.
Tighten resource ownership by cancelling telemetry collectors, terminating reply observers, bounding dashboard projections, and releasing process file descriptors, child observers, and inode-verified Unix socket paths on every exit path.
2026-08-19 21:38:14 +00:00
|
|
|
let _ = child_kill(&mut child);
|
|
|
|
|
let _ = child_wait(&mut child);
|
test(provisioning): stateful conformance kit for reconciler and plugins
Replace pointwise scenario testing with a reusable conformance kit in
tests/common: a deterministic trace harness (input alphabet, seeded
generator, naive shrinker), an invariant oracle covering twenty black-box
guarantees (identity, correlation, dead-hold, attempt-fact ownership,
quiescence no-op, monotonic generation, fair convergence, bounded
replacement), and a fair-scheduler tail asserting eventual reconciliation.
Three conformance levels run the same battery:
- FakeBackend: the reference in-memory substrate (256 seeds x 2 modes)
- PluginBackendAdapter over FakePlugin: seam contracts plus the battery
- ProcessPlugin: real child processes, faults as real signals/errors;
"no double-create" and "converged leaks nothing" verified by counting
live PIDs (16 seeds)
Also documents two seam findings the battery surfaced: ProvisionPlugin
cannot express ambiguity (kit convention: AMBIGUOUS_FAULT_MARKER error
reclassified by the adapter; definite classification leaks provider
resources) and spawn_effect closures form a spawner Arc cycle that leaks
backends under queue-based spawners (kit breaks it at harness drop).
2026-08-14 15:21:46 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl TestablePlugin for ProcessPlugin {
|
|
|
|
|
fn apply_fault(&self, fault: Fault) {
|
|
|
|
|
let mut state = self.state.lock();
|
|
|
|
|
match fault {
|
|
|
|
|
Fault::Heal => state.faults.clear(),
|
|
|
|
|
other => state.faults.push_back(other),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn leaked_resources(&self, live_handles: &[u64]) -> Vec<String> {
|
|
|
|
|
let state = self.state.lock();
|
|
|
|
|
state
|
|
|
|
|
.children
|
|
|
|
|
.keys()
|
|
|
|
|
.filter(|attempt| !live_handles.contains(attempt))
|
|
|
|
|
.map(|attempt| {
|
|
|
|
|
let pid = state
|
|
|
|
|
.children
|
|
|
|
|
.get(attempt)
|
|
|
|
|
.map(|child| child.id())
|
|
|
|
|
.unwrap_or_default();
|
|
|
|
|
format!("attempt={attempt} pid={pid}")
|
|
|
|
|
})
|
|
|
|
|
.collect()
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn resources_created(&self) -> usize {
|
|
|
|
|
self.state.lock().created
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl ProvisionPlugin for ProcessPlugin {
|
|
|
|
|
fn create_node(
|
|
|
|
|
&mut self,
|
|
|
|
|
spec: NodeProvisionSpec,
|
|
|
|
|
_sink: PluginSink,
|
|
|
|
|
) -> Result<PluginNodeHandle, String> {
|
|
|
|
|
let mut state = self.state.lock();
|
|
|
|
|
let attempt = spec.attempt_id;
|
|
|
|
|
if let Some(child) = state.children.get(&attempt) {
|
|
|
|
|
// Adoption: the child for this attempt already exists.
|
|
|
|
|
return Ok(PluginNodeHandle {
|
|
|
|
|
id: attempt,
|
|
|
|
|
provider_process_id: Some(child.id()),
|
|
|
|
|
});
|
|
|
|
|
}
|
|
|
|
|
let fault = state.faults.pop_front();
|
|
|
|
|
if matches!(fault, Some(Fault::Panic)) {
|
|
|
|
|
panic!("scripted process plugin panic");
|
|
|
|
|
}
|
|
|
|
|
if matches!(fault, Some(Fault::Definite)) {
|
|
|
|
|
return Err("scripted definite failure".to_owned());
|
|
|
|
|
}
|
Enforce actor-owned Myelin control flow
Add a repository-owned rustc wrapper that enforces execution ownership and dependency boundaries during ordinary Cargo commands, with compile-pass and compile-fail policy contracts.
Move scheduling, timers, provider polling, provisioning, recovery, supervision, and shutdown decisions behind engine and actor APIs. Add deterministic component properties, stateful Myelin lifecycle coverage, persisted regression cases, and the bounded CI workflow.
Tighten resource ownership by cancelling telemetry collectors, terminating reply observers, bounding dashboard projections, and releasing process file descriptors, child observers, and inode-verified Unix socket paths on every exit path.
2026-08-19 21:38:14 +00:00
|
|
|
let mut command = Command::new("sleep");
|
|
|
|
|
command.arg("infinity");
|
|
|
|
|
let child =
|
|
|
|
|
command_spawn(&mut command).map_err(|error| format!("spawn failed: {error}"))?;
|
test(provisioning): stateful conformance kit for reconciler and plugins
Replace pointwise scenario testing with a reusable conformance kit in
tests/common: a deterministic trace harness (input alphabet, seeded
generator, naive shrinker), an invariant oracle covering twenty black-box
guarantees (identity, correlation, dead-hold, attempt-fact ownership,
quiescence no-op, monotonic generation, fair convergence, bounded
replacement), and a fair-scheduler tail asserting eventual reconciliation.
Three conformance levels run the same battery:
- FakeBackend: the reference in-memory substrate (256 seeds x 2 modes)
- PluginBackendAdapter over FakePlugin: seam contracts plus the battery
- ProcessPlugin: real child processes, faults as real signals/errors;
"no double-create" and "converged leaks nothing" verified by counting
live PIDs (16 seeds)
Also documents two seam findings the battery surfaced: ProvisionPlugin
cannot express ambiguity (kit convention: AMBIGUOUS_FAULT_MARKER error
reclassified by the adapter; definite classification leaks provider
resources) and spawn_effect closures form a spawner Arc cycle that leaks
backends under queue-based spawners (kit breaks it at harness drop).
2026-08-14 15:21:46 +00:00
|
|
|
let pid = child.id();
|
|
|
|
|
state.children.insert(attempt, child);
|
|
|
|
|
state.created += 1;
|
|
|
|
|
if matches!(fault, Some(Fault::Ambiguous)) {
|
|
|
|
|
// The child exists but the caller cannot know; a retry with
|
|
|
|
|
// the same attempt must adopt it.
|
|
|
|
|
return Err(AMBIGUOUS_FAULT_MARKER.to_owned());
|
|
|
|
|
}
|
|
|
|
|
Ok(PluginNodeHandle {
|
|
|
|
|
id: attempt,
|
|
|
|
|
provider_process_id: Some(pid),
|
|
|
|
|
})
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn start_bootstrap(&mut self, _handle: &PluginNodeHandle) -> Result<(), String> {
|
|
|
|
|
Ok(())
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn cancel_bootstrap(&mut self, _handle: &PluginNodeHandle) -> Result<(), String> {
|
|
|
|
|
Ok(())
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn complete_bootstrap(&mut self, _handle: &PluginNodeHandle) -> Result<(), String> {
|
|
|
|
|
Ok(())
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn stop_node(&mut self, handle: &PluginNodeHandle) -> Result<(), String> {
|
|
|
|
|
let mut state = self.state.lock();
|
|
|
|
|
if let Some(mut child) = state.children.remove(&handle.id) {
|
Enforce actor-owned Myelin control flow
Add a repository-owned rustc wrapper that enforces execution ownership and dependency boundaries during ordinary Cargo commands, with compile-pass and compile-fail policy contracts.
Move scheduling, timers, provider polling, provisioning, recovery, supervision, and shutdown decisions behind engine and actor APIs. Add deterministic component properties, stateful Myelin lifecycle coverage, persisted regression cases, and the bounded CI workflow.
Tighten resource ownership by cancelling telemetry collectors, terminating reply observers, bounding dashboard projections, and releasing process file descriptors, child observers, and inode-verified Unix socket paths on every exit path.
2026-08-19 21:38:14 +00:00
|
|
|
child_kill(&mut child).map_err(|error| format!("kill failed: {error}"))?;
|
|
|
|
|
child_wait(&mut child).map_err(|error| format!("reap failed: {error}"))?;
|
test(provisioning): stateful conformance kit for reconciler and plugins
Replace pointwise scenario testing with a reusable conformance kit in
tests/common: a deterministic trace harness (input alphabet, seeded
generator, naive shrinker), an invariant oracle covering twenty black-box
guarantees (identity, correlation, dead-hold, attempt-fact ownership,
quiescence no-op, monotonic generation, fair convergence, bounded
replacement), and a fair-scheduler tail asserting eventual reconciliation.
Three conformance levels run the same battery:
- FakeBackend: the reference in-memory substrate (256 seeds x 2 modes)
- PluginBackendAdapter over FakePlugin: seam contracts plus the battery
- ProcessPlugin: real child processes, faults as real signals/errors;
"no double-create" and "converged leaks nothing" verified by counting
live PIDs (16 seeds)
Also documents two seam findings the battery surfaced: ProvisionPlugin
cannot express ambiguity (kit convention: AMBIGUOUS_FAULT_MARKER error
reclassified by the adapter; definite classification leaks provider
resources) and spawn_effect closures form a spawner Arc cycle that leaks
backends under queue-based spawners (kit breaks it at harness drop).
2026-08-14 15:21:46 +00:00
|
|
|
}
|
|
|
|
|
Ok(())
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#[test]
|
|
|
|
|
fn process_plugin_passes_seam_contracts() {
|
|
|
|
|
let mut plugin = ProcessPlugin::default();
|
|
|
|
|
assert_plugin_contracts(&mut plugin);
|
|
|
|
|
// Belt and braces: contracts released everything.
|
|
|
|
|
assert!(plugin.leaked_resources(&[]).is_empty());
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#[test]
|
|
|
|
|
fn process_plugin_battery_holds_invariants_and_converges() {
|
2026-08-17 22:51:08 +00:00
|
|
|
run_trace_battery(
|
|
|
|
|
|| PluginBackendAdapter::new(ProcessPlugin::default()),
|
|
|
|
|
16,
|
|
|
|
|
28,
|
|
|
|
|
);
|
test(provisioning): stateful conformance kit for reconciler and plugins
Replace pointwise scenario testing with a reusable conformance kit in
tests/common: a deterministic trace harness (input alphabet, seeded
generator, naive shrinker), an invariant oracle covering twenty black-box
guarantees (identity, correlation, dead-hold, attempt-fact ownership,
quiescence no-op, monotonic generation, fair convergence, bounded
replacement), and a fair-scheduler tail asserting eventual reconciliation.
Three conformance levels run the same battery:
- FakeBackend: the reference in-memory substrate (256 seeds x 2 modes)
- PluginBackendAdapter over FakePlugin: seam contracts plus the battery
- ProcessPlugin: real child processes, faults as real signals/errors;
"no double-create" and "converged leaks nothing" verified by counting
live PIDs (16 seeds)
Also documents two seam findings the battery surfaced: ProvisionPlugin
cannot express ambiguity (kit convention: AMBIGUOUS_FAULT_MARKER error
reclassified by the adapter; definite classification leaks provider
resources) and spawn_effect closures form a spawner Arc cycle that leaks
backends under queue-based spawners (kit breaks it at harness drop).
2026-08-14 15:21:46 +00:00
|
|
|
}
|