Replace pointwise scenario testing with a reusable conformance kit in
tests/common: a deterministic trace harness (input alphabet, seeded
generator, naive shrinker), an invariant oracle covering twenty black-box
guarantees (identity, correlation, dead-hold, attempt-fact ownership,
quiescence no-op, monotonic generation, fair convergence, bounded
replacement), and a fair-scheduler tail asserting eventual reconciliation.
Three conformance levels run the same battery:
- FakeBackend: the reference in-memory substrate (256 seeds x 2 modes)
- PluginBackendAdapter over FakePlugin: seam contracts plus the battery
- ProcessPlugin: real child processes, faults as real signals/errors;
"no double-create" and "converged leaks nothing" verified by counting
live PIDs (16 seeds)
Also documents two seam findings the battery surfaced: ProvisionPlugin
cannot express ambiguity (kit convention: AMBIGUOUS_FAULT_MARKER error
reclassified by the adapter; definite classification leaks provider
resources) and spawn_effect closures form a spawner Arc cycle that leaks
backends under queue-based spawners (kit breaks it at harness drop).
Introduce a pure, level-triggered reconciler in `crates/provisioning`
that drives a declared cluster shape toward convergence over the
existing node lifecycle, replacing the edge-triggered imperative node
orchestration in `apps/myelin`.
- `reconcile`/`reconcile_node`/`observe`: pure decider and observation
folder with stable logical-node identity, per-attempt operation
identity, and deterministic retry backoff; `ClusterDriver` is the sole
writer of observed state, coalescing triggers, recording operations as
pending before dispatch, and scheduling timed requeues.
- `IdempotentEffectExecutor`: deduplicates submissions by
`(run_id, logical_node_id, attempt)` and runs provider work on the
engine-hosted blocking substrate, never blocking a reconcile pass.
- Myelin integration: `MyelinEffectBackend` bridges `ProvisionPlugin` to
the executor contract; `LocalProcessPlugin`/`LocalDockerPlugin`
provider adapters; `ProvisionedClusterGuard` pumps triggers,
observations, and due operations.
- Retire the imperative acquire/bootstrap/teardown sequencing across
`apps/myelin` orchestration, staging, observability, and provider
adapters in favor of the declarative driver.
- Move the reconciler specification to `docs/specs/archive`.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
