swactor/docs/specs/drafts/JOB_RUNNER_SPEC.md

Synaptic Job Runner Specification

Id: 4 Last modified: b8aff00dc1 Last reviewed:

Any edit to this spec must update Last modified above to the current git HEAD commit.

---

1. Purpose

Run an arbitrary command job on a rented GPU node, end to end, through swactor: provision a node → push the workspace → setup → run → collect outputs → teardown. swactor is the substrate: provisioning, the node runtime, the data plane, and observability are reused; the job runner is built on top of them.

v1 is the single-node command job: one job → one node → one attempt. Multi-node, sharded, and pipeline jobs are future work-descriptions over the same core.

---

2. What swactor grows, what swactor reuses

New — the job runner's content:

• a node primitive that runs an arbitrary command as a supervised process and reports exit via orchestrator↔node messages;
• a bulk-transfer service on the shared iroh endpoint for workspace push (operator → node) and output pull (node → operator) — a new service paralleling the existing edge and telemetry ALPN services, not the arena/tensor data plane;
• a job FSM and the orchestrator↔node protocol that sequence the above.

Reused unchanged: the VastAI provider adapter (provision/teardown), SWIM membership, the telemetry event infrastructure, the shared iroh endpoint (which already multiplexes the edge and telemetry services), and the swactor actor runtime.

---

3. Design principle: job = work, gpu-agnostic

A job describes what to do, not what hardware to run on — no provider, no GPU, no node. Hardware (provider, GPU type/count/VRAM, disk, image, selection) lives in a separate cluster config that is bootstrapped before any job is submitted. This keeps a future sharded-model job the same shape against different placement; sharding is not a v1 feature.

---

4. Job description

field	required	meaning
name	yes	job identity
run	yes	the command that does the work
setup	no	one-time command run before run (env install)
workspace	no	{ workdir, exclude } — pushed to the node first
outputs	no	paths to collect back after run
env	no	environment variables injected into setup and run

Nothing else. No resources, no provider, no GPU, no cleanup policy.

[job]
name = "airfrans_smoke_01"
setup = "uv sync --no-dev"
run = "uv run remote-run smoke-train configs/aggressive_smoke.toml"
workspace = { workdir = ".", exclude = ["/artifacts", "/.venv", "__pycache__"] }
outputs = ["metrics.jsonl", "final_metrics.json", "checkpoint_latest.pt"]
env = { HF_TOKEN = "..." }

Outputs-root convention. Every job writes its artifacts under one fixed, node-known outputs root (conventionally the job's workdir), and outputs lists paths relative to that root. The node resolves each declared output against the root; collection pulls whatever of them exists. This avoids the fragility of globbing timestamped run directories: the job decides where under the root it writes, and the runner never has to discover a fresh, unpredictable output path. Paths outside the root are not collected.

A job is submitted alone — no hardware, no pool. It runs on a ready node from a separately bootstrapped cluster (§3).

5. Roles

• Orchestrator (job authority): places the job on a ready node, drives the lifecycle, owns the terminal outcome.
• Node (executor): runs setup and run as supervised processes, materializes the workspace, exposes outputs, reports lifecycle via orchestrator↔node messages.

Node existence — provisioning and teardown — is a separate cluster-config concern driven by the reconciler, not part of the job runner. A job assumes at least one ready node; it neither provisions nor tears down nodes.

---

6. Lifecycle

States: PENDING → RUNNING → COMPLETED | FAILED.

Ownership: the job runner does not own node existence — that is the separate cluster-config/reconciler concern. The orchestrator drives only what executes on an already-ready node. The telemetry is observability-only — metrics and logging; lifecycle events are mirrored to it but it is never a control channel.

Command routing — what the orchestrator emits, and where it goes:

command	target	kind
MaterializeWorkspace	node	orchestrator↔node message
RunSetup	node	orchestrator↔node message
RunJob	node	orchestrator↔node message
CollectOutputs	node	orchestrator↔node message

Event sourcing — what the orchestrator observes, and where it comes from:

event	source	channel
NodeReady	reconciler — node reached ready	reconciler observed state
WorkspaceMaterialized	node	orchestrator↔node message
SetupCompleted	node	orchestrator↔node message
JobExited{code}	node	orchestrator↔node message
OutputsCollected	node	orchestrator↔node message
NodeFault	node — supervised process	orchestrator↔node message
NodeLost	SWIM membership	membership signal
OperatorStop	operator	orchestrator API

Transition table — events and commands are sourced and routed as above:

event observed	command emitted	transition
JobSubmitted{job}	—	→ PENDING
NodeReady	MaterializeWorkspace	→ RUNNING
WorkspaceMaterialized	RunSetup
SetupCompleted	RunJob
JobExited{0}	CollectOutputs
OutputsCollected	—	→ COMPLETED
JobExited{non-zero}	CollectOutputs (best-effort)	→ FAILED
NodeFault / NodeLost / OperatorStop	—	→ FAILED

• A job claims one ready node from the bootstrapped cluster and waits for that node's readiness, not cluster-wide convergence.
• No setup → WorkspaceMaterialized goes straight to RunJob. No workspace → materialization is skipped.
• The exit code is authoritative: 0 → COMPLETED, non-zero → FAILED. Declared outputs are collected either way; collection on failure is best-effort.
• The job does not tear down its node; cluster teardown is a separate operator/reconciler concern.
• Every transition emits a telemetry event.

OperatorStop is the operator kill switch for a running job.

---

7. Behavior contracts

The orchestrator commands these contracts; the node owns the local mechanism. Control and lifecycle travel as orchestrator↔node messages; the telemetry mirrors them for observability (metrics and logging) and is never a control channel. Bulk workspace and output bytes travel over a dedicated transfer service on the shared iroh endpoint, not in actor messages.

Supervised command execution (node). On RunSetup / RunJob, the node spawns the command in workdir with the declared env, supervises it, and reports SetupCompleted / JobExited{code} (or NodeFault) via orchestrator↔node messages. A failed setup is a failed job.

Workspace materialization (operator → node). On MaterializeWorkspace, the workdir tree (with exclude applied) is pushed from the operator to the node over the iroh transfer service, landing at the node's workdir. The node reports WorkspaceMaterialized.

Output collection (node → operator). On CollectOutputs, declared outputs are pulled from the node to a per-run operator landing directory over the iroh transfer service. The node reports OutputsCollected. Missing outputs do not change the outcome — the exit code already decided it; whatever exists is gathered.

---

8. Relationship to the broader work model

This spec defines only the job plugin. The intended generalization is a work description: a graph of work-units with typed ports, placed by a planner. The v1 job resolves to a single work-unit with no edges; multi-unit, sharded, and pipeline jobs are future work-descriptions over the same core. The gpu-agnostic job shape is what keeps that path open.