swactor/CLAUDE/notes/research_synthesis.md

Research Synthesis: Competitor Analysis

Frameworks Studied

1. Ractor (Rust) — async task-per-actor on tokio
2. Tokio (Rust) — work-stealing async runtime
3. Erlang/OTP BEAM — reduction-counted preemptive scheduler
4. Linux CFS/EEVDF — vruntime fairness, work stealing, adaptive ticks
5. libuv/Node.js — single-threaded event loop with phase-based execution

Critical Finding: Swactor Fairness Bug

tick_all in worker.rs drains the ENTIRE mailbox for each actor before moving to the next:

while let Some(msg) = slot.mailbox.pop_front() {
    // processes ALL messages for actor A before moving to actor B
}

If actor A has 10,000 queued messages, all other actors on the same worker are completely starved until A finishes. Every other runtime studied prevents this:

• BEAM: 4000 reductions per process, then preempt
• Tokio: 128-256 operation cooperative budget per task
• libuv: Round-robin across handlers; no single handler drains completely
• Linux CFS: vruntime-based fairness; time slices enforced

Ranked Improvement Opportunities

P0: Per-Actor Message Budget (Fairness)

• Impact: Prevents starvation; critical for production workloads
• Effort: Small — modify tick_all loop in worker.rs
• Source: BEAM reductions, tokio coop budget
• Design: Process up to N messages per actor per tick, configurable via RuntimeConfig

P1: Improved Benchmarks

• Impact: Can't improve what you can't measure
• Effort: Medium — new benchmark scenarios
• Source: Ractor benchmarks, tokio benchmarks
• New scenarios needed:
  • Fairness: imbalanced load (1 hot actor + 99 cold actors)
  • Message size sensitivity (8B, 64B, 256B, 1KB)
  • Contention: many-to-one fanin
  • Latency percentiles (p50, p99, p999)
  • Cross-worker vs same-worker message delivery

P2: Better Testing Coverage

• Impact: Catches regressions, validates fairness guarantees
• Effort: Medium
• Source: BEAM testing patterns, tokio Loom
• New tests needed:
  • Fairness: hot actor doesn't starve cold actors
  • Backpressure: tiny buffer under load
  • Concurrent spawn+send races
  • Multi-threaded delivery guarantees

P3: Adaptive Backoff with Thread Parking

• Impact: Better latency under varying load; power savings
• Effort: Medium — modify run loop in worker.rs
• Source: Tokio parker, Linux NO_HZ
• Design: Replace spinning with condvar-based parking; use notification to wake

P4: Work Stealing (Future)

• Impact: Dynamic load balancing
• Effort: Large — significant architectural change
• Source: Tokio steal-half, BEAM migration plans
• Note: Would require stealing actors between workers, which changes ownership

Key Design Comparisons

Dimension	Swactor	Ractor	Tokio	BEAM
Scheduling	Sync tick	Async task-per-actor	Work-stealing	Reduction preemption
Fairness	None (drain all)	N/A (1 task = 1 actor)	Coop budget (128)	4000 reductions
Backpressure	Bounded crossbeam ring	None (unbounded)	Bounded MPSC	Off-heap mailbox
Work stealing	None	Tokio handles it	Steal-half, N/2 searchers	Steal + migrate
Panic handling	catch_unwind + poison	AssertUnwindSafe + supervisor	N/A	Process isolation
Message passing	Box downcast	Box downcast	Typed channels	Term copying

Ractor Bug History Lessons

• Destructive get_children() — snapshot methods must not mutate state
• OutputPort silent drops — bounded channels need explicit backpressure, not silent overflow
• Remote actor latency regression — cross-runtime messaging needs careful ordering
• Memory bloat per actor — each channel/structure per actor adds up at scale

Tokio Patterns to Adopt

1. LIFO slot for same-worker sends (cache locality)
2. Searcher count limiting (N/2 max) for cross-worker stealing
3. Steal-half strategy (amortize overhead)
4. Global queue interval checking (reduce contention)
5. Loom-style testing for lock-free code
6. Single allocation per actor context (hot/cold layout)

Additional Frameworks Studied (Cycle 2)

Kameo (v0.19)

• Fully async on tokio, one task per actor
• Dual mailbox: bounded (default 64) or unbounded tokio mpsc channels
• Proper backpressure via bounded mpsc sender blocking
• Typed signals (no Box) — vtable dispatch, no downcast failures
• Erlang-style links for supervision (on_link_died)
• on_panic hook can restart actor (vs swactor's permanent poisoning)
• Bugs: deadlocks in link establishment, leaked ActorRef preventing stop

Actix (v0.13)

• Context-as-Future model — each actor is a single pollable Future on an Arbiter
• Custom Vyukov lock-free MPSC queue (not tokio channels) — push is single atomic_swap
• Default mailbox capacity: 16 (tiny!)
• do_send() bypasses capacity for internal notifications
• Mailbox has 256-message assertion guard (similar to our budget approach!)
• vtable dispatch via Box<dyn EnvelopeProxy<A>> — no Any downcast
• SyncArbiter: crossbeam_channel thread pool for blocking actors
• WHY FAST: custom MPSC queue, no async overhead for message processing, same-thread actors avoid cross-thread coordination, SmallVec for futures

Swactor Advantages (confirmed)

• Synchronous tick model: deterministic, no async overhead, simulation-friendly
• Hybrid channel: bounded ring + unbounded overflow = no message loss
• Per-actor message budget: validated by BEAM (4000 reds), tokio (128 ops), actix (256 assert)
• No tokio dependency: could run on bare metal
• Detailed per-phase timing stats (6-phase TickTiming)

Swactor Weaknesses to Address

• Box downcast can fail silently → type mismatch tracking needed (have it)
• No backpressure → FIXED: optional bounded mailbox with DropNewest/DropOldest (Cycle 6)
• Panicked actors permanently poisoned → no recovery path
• Spin/sleep backoff wastes CPU → FIXED: thread parking with instant wakeup (Cycle 3)
• No supervision trees

Work Stealing Deep Dive (Cycle 5)

Cross-Runtime Comparison

Aspect	Tokio	Go	BEAM	ForkJoinPool
Queue	Fixed 256-slot ring	256-slot ring + runnext	Per-priority linked	Growable array deque
Steal granularity	Half victim's queue	Half victim's runq	Individual processes	One task at a time
LIFO fast-path	Dedicated slot (3-use cap)	runnext (stealable 4th try)	None	Owner pops from top
Global queue	Mutex intrusive list	Checked 1/61 ticks	Per-priority migration	Even-indexed submit queues
Searcher limit	N/2 workers	GOMAXPROCS/2	N/A (proactive migration)	Idle stack in ctl field
Balance strategy	Reactive steal	Reactive steal	Proactive migration + reactive	Reactive scan
Load compaction	No (spread)	No (spread)	Yes (min schedulers)	No (spread)

Key Patterns

1. LIFO slot: Every runtime has one. Improves cache locality by running the recipient immediately after the sender. Tokio caps at 3 consecutive uses to prevent starvation.

2. Steal-half: Tokio and Go both steal half the victim's queue. This amortizes the overhead of cross-thread coordination — O(1) per stolen item instead of O(1) per steal.

3. N/2 searcher limit: Both Tokio and Go cap concurrent searchers to prevent thundering herd. Without it, all N workers scanning causes O(N²) cache-line bouncing.

4. BEAM's migration: Unique dual approach — reactive stealing when idle, proactive migration via periodic check_balance() that computes migration paths based on average max queue length.

Feasibility for Swactor

• Full actor migration: Mechanically possible (ActorSlot is Send), but has 1-tick message loss window and requires push-based donation (ActorPool not Sync → no pull stealing)
• Message stealing without actors: Impossible — actor IS the state, messages without the actor are meaningless
• Transfer queue snooping: Pointless without actor migration
• Load-aware placement ✅ IMPLEMENTED: Placement reads per-worker stats to bias toward lighter workers, with round-robin fallback when stats are equal

Decision: Load-Aware Placement over Work Stealing

Chose load-aware placement because:

• Zero correctness risk (no message loss, no ordering changes)
• O(N) atomic loads per spawn (trivial for N≤8 workers)
• Handles the primary source of imbalance: uneven spawn distribution
• Full work stealing deferred — would require migration channels, address map coordination, and forwarding tombstones