# Research Synthesis: Competitor Analysis ## Frameworks Studied 1. **Ractor** (Rust) — async task-per-actor on tokio 2. **Tokio** (Rust) — work-stealing async runtime 3. **Erlang/OTP BEAM** — reduction-counted preemptive scheduler 4. **Linux CFS/EEVDF** — vruntime fairness, work stealing, adaptive ticks 5. **libuv/Node.js** — single-threaded event loop with phase-based execution ## Critical Finding: Swactor Fairness Bug `tick_all` in `worker.rs` drains the ENTIRE mailbox for each actor before moving to the next: ```rust while let Some(msg) = slot.mailbox.pop_front() { // processes ALL messages for actor A before moving to actor B } ``` If actor A has 10,000 queued messages, all other actors on the same worker are completely starved until A finishes. Every other runtime studied prevents this: - **BEAM**: 4000 reductions per process, then preempt - **Tokio**: 128-256 operation cooperative budget per task - **libuv**: Round-robin across handlers; no single handler drains completely - **Linux CFS**: vruntime-based fairness; time slices enforced ## Ranked Improvement Opportunities ### P0: Per-Actor Message Budget (Fairness) - **Impact**: Prevents starvation; critical for production workloads - **Effort**: Small — modify `tick_all` loop in `worker.rs` - **Source**: BEAM reductions, tokio coop budget - **Design**: Process up to N messages per actor per tick, configurable via RuntimeConfig ### P1: Improved Benchmarks - **Impact**: Can't improve what you can't measure - **Effort**: Medium — new benchmark scenarios - **Source**: Ractor benchmarks, tokio benchmarks - **New scenarios needed**: - Fairness: imbalanced load (1 hot actor + 99 cold actors) - Message size sensitivity (8B, 64B, 256B, 1KB) - Contention: many-to-one fanin - Latency percentiles (p50, p99, p999) - Cross-worker vs same-worker message delivery ### P2: Better Testing Coverage - **Impact**: Catches regressions, validates fairness guarantees - **Effort**: Medium - **Source**: BEAM testing patterns, tokio Loom - **New tests needed**: - Fairness: hot actor doesn't starve cold actors - Backpressure: tiny buffer under load - Concurrent spawn+send races - Multi-threaded delivery guarantees ### P3: Adaptive Backoff with Thread Parking - **Impact**: Better latency under varying load; power savings - **Effort**: Medium — modify `run` loop in `worker.rs` - **Source**: Tokio parker, Linux NO_HZ - **Design**: Replace spinning with condvar-based parking; use notification to wake ### P4: Work Stealing (Future) - **Impact**: Dynamic load balancing - **Effort**: Large — significant architectural change - **Source**: Tokio steal-half, BEAM migration plans - **Note**: Would require stealing actors between workers, which changes ownership ## Key Design Comparisons | Dimension | Swactor | Ractor | Tokio | BEAM | |-----------|---------|--------|-------|------| | Scheduling | Sync tick | Async task-per-actor | Work-stealing | Reduction preemption | | Fairness | None (drain all) | N/A (1 task = 1 actor) | Coop budget (128) | 4000 reductions | | Backpressure | Bounded crossbeam ring | None (unbounded) | Bounded MPSC | Off-heap mailbox | | Work stealing | None | Tokio handles it | Steal-half, N/2 searchers | Steal + migrate | | Panic handling | catch_unwind + poison | AssertUnwindSafe + supervisor | N/A | Process isolation | | Message passing | Box downcast | Box downcast | Typed channels | Term copying | ## Ractor Bug History Lessons - Destructive `get_children()` — snapshot methods must not mutate state - OutputPort silent drops — bounded channels need explicit backpressure, not silent overflow - Remote actor latency regression — cross-runtime messaging needs careful ordering - Memory bloat per actor — each channel/structure per actor adds up at scale ## Tokio Patterns to Adopt 1. LIFO slot for same-worker sends (cache locality) 2. Searcher count limiting (N/2 max) for cross-worker stealing 3. Steal-half strategy (amortize overhead) 4. Global queue interval checking (reduce contention) 5. Loom-style testing for lock-free code 6. Single allocation per actor context (hot/cold layout) ## Additional Frameworks Studied (Cycle 2) ### Kameo (v0.19) - Fully async on tokio, one task per actor - Dual mailbox: bounded (default 64) or unbounded tokio mpsc channels - Proper backpressure via bounded mpsc sender blocking - Typed signals (no Box) — vtable dispatch, no downcast failures - Erlang-style links for supervision (`on_link_died`) - `on_panic` hook can restart actor (vs swactor's permanent poisoning) - Bugs: deadlocks in link establishment, leaked ActorRef preventing stop ### Actix (v0.13) - Context-as-Future model — each actor is a single pollable Future on an Arbiter - **Custom Vyukov lock-free MPSC queue** (not tokio channels) — push is single atomic_swap - Default mailbox capacity: 16 (tiny!) - `do_send()` bypasses capacity for internal notifications - Mailbox has 256-message assertion guard (similar to our budget approach!) - vtable dispatch via `Box>` — no Any downcast - SyncArbiter: crossbeam_channel thread pool for blocking actors - WHY FAST: custom MPSC queue, no async overhead for message processing, same-thread actors avoid cross-thread coordination, SmallVec for futures ### Swactor Advantages (confirmed) - Synchronous tick model: deterministic, no async overhead, simulation-friendly - Hybrid channel: bounded ring + unbounded overflow = no message loss - Per-actor message budget: validated by BEAM (4000 reds), tokio (128 ops), actix (256 assert) - No tokio dependency: could run on bare metal - Detailed per-phase timing stats (6-phase TickTiming) ### Swactor Weaknesses to Address - Box downcast can fail silently → type mismatch tracking needed (have it) - No backpressure: senders never block → unbounded queue growth under sustained load - Panicked actors permanently poisoned → no recovery path - Spin/sleep backoff wastes CPU → condvar-based parking would be better - No supervision trees