swactor/CLAUDE/notes/research_synthesis.md

# Research Synthesis: Competitor Analysis

## Frameworks Studied
1. **Ractor** (Rust) — async task-per-actor on tokio
2. **Tokio** (Rust) — work-stealing async runtime
3. **Erlang/OTP BEAM** — reduction-counted preemptive scheduler
4. **Linux CFS/EEVDF** — vruntime fairness, work stealing, adaptive ticks
5. **libuv/Node.js** — single-threaded event loop with phase-based execution

## Critical Finding: Swactor Fairness Bug

`tick_all` in `worker.rs` drains the ENTIRE mailbox for each actor before moving to the next:
```rust
while let Some(msg) = slot.mailbox.pop_front() {
    // processes ALL messages for actor A before moving to actor B
}
```

If actor A has 10,000 queued messages, all other actors on the same worker are completely starved
until A finishes. Every other runtime studied prevents this:
- **BEAM**: 4000 reductions per process, then preempt
- **Tokio**: 128-256 operation cooperative budget per task
- **libuv**: Round-robin across handlers; no single handler drains completely
- **Linux CFS**: vruntime-based fairness; time slices enforced

## Ranked Improvement Opportunities

### P0: Per-Actor Message Budget (Fairness)
- **Impact**: Prevents starvation; critical for production workloads
- **Effort**: Small — modify `tick_all` loop in `worker.rs`
- **Source**: BEAM reductions, tokio coop budget
- **Design**: Process up to N messages per actor per tick, configurable via RuntimeConfig

### P1: Improved Benchmarks
- **Impact**: Can't improve what you can't measure
- **Effort**: Medium — new benchmark scenarios
- **Source**: Ractor benchmarks, tokio benchmarks
- **New scenarios needed**:
  - Fairness: imbalanced load (1 hot actor + 99 cold actors)
  - Message size sensitivity (8B, 64B, 256B, 1KB)
  - Contention: many-to-one fanin
  - Latency percentiles (p50, p99, p999)
  - Cross-worker vs same-worker message delivery

### P2: Better Testing Coverage
- **Impact**: Catches regressions, validates fairness guarantees
- **Effort**: Medium
- **Source**: BEAM testing patterns, tokio Loom
- **New tests needed**:
  - Fairness: hot actor doesn't starve cold actors
  - Backpressure: tiny buffer under load
  - Concurrent spawn+send races
  - Multi-threaded delivery guarantees

### P3: Adaptive Backoff with Thread Parking
- **Impact**: Better latency under varying load; power savings
- **Effort**: Medium — modify `run` loop in `worker.rs`
- **Source**: Tokio parker, Linux NO_HZ
- **Design**: Replace spinning with condvar-based parking; use notification to wake

### P4: Work Stealing (Future)
- **Impact**: Dynamic load balancing
- **Effort**: Large — significant architectural change
- **Source**: Tokio steal-half, BEAM migration plans
- **Note**: Would require stealing actors between workers, which changes ownership

## Key Design Comparisons

| Dimension | Swactor | Ractor | Tokio | BEAM |
|-----------|---------|--------|-------|------|
| Scheduling | Sync tick | Async task-per-actor | Work-stealing | Reduction preemption |
| Fairness | None (drain all) | N/A (1 task = 1 actor) | Coop budget (128) | 4000 reductions |
| Backpressure | Bounded crossbeam ring | None (unbounded) | Bounded MPSC | Off-heap mailbox |
| Work stealing | None | Tokio handles it | Steal-half, N/2 searchers | Steal + migrate |
| Panic handling | catch_unwind + poison | AssertUnwindSafe + supervisor | N/A | Process isolation |
| Message passing | Box<dyn Any> downcast | Box<dyn Any> downcast | Typed channels | Term copying |

## Ractor Bug History Lessons
- Destructive `get_children()` — snapshot methods must not mutate state
- OutputPort silent drops — bounded channels need explicit backpressure, not silent overflow
- Remote actor latency regression — cross-runtime messaging needs careful ordering
- Memory bloat per actor — each channel/structure per actor adds up at scale

## Tokio Patterns to Adopt
1. LIFO slot for same-worker sends (cache locality)
2. Searcher count limiting (N/2 max) for cross-worker stealing
3. Steal-half strategy (amortize overhead)
4. Global queue interval checking (reduce contention)
5. Loom-style testing for lock-free code
6. Single allocation per actor context (hot/cold layout)

## Additional Frameworks Studied (Cycle 2)

### Kameo (v0.19)
- Fully async on tokio, one task per actor
- Dual mailbox: bounded (default 64) or unbounded tokio mpsc channels
- Proper backpressure via bounded mpsc sender blocking
- Typed signals (no Box<dyn Any>) — vtable dispatch, no downcast failures
- Erlang-style links for supervision (`on_link_died`)
- `on_panic` hook can restart actor (vs swactor's permanent poisoning)
- Bugs: deadlocks in link establishment, leaked ActorRef preventing stop

### Actix (v0.13)
- Context-as-Future model — each actor is a single pollable Future on an Arbiter
- **Custom Vyukov lock-free MPSC queue** (not tokio channels) — push is single atomic_swap
- Default mailbox capacity: 16 (tiny!)
- `do_send()` bypasses capacity for internal notifications
- Mailbox has 256-message assertion guard (similar to our budget approach!)
- vtable dispatch via `Box<dyn EnvelopeProxy<A>>` — no Any downcast
- SyncArbiter: crossbeam_channel thread pool for blocking actors
- WHY FAST: custom MPSC queue, no async overhead for message processing,
  same-thread actors avoid cross-thread coordination, SmallVec for futures

### Swactor Advantages (confirmed)
- Synchronous tick model: deterministic, no async overhead, simulation-friendly
- Hybrid channel: bounded ring + unbounded overflow = no message loss
- Per-actor message budget: validated by BEAM (4000 reds), tokio (128 ops), actix (256 assert)
- No tokio dependency: could run on bare metal
- Detailed per-phase timing stats (6-phase TickTiming)

### Swactor Weaknesses to Address
- Box<dyn Any> downcast can fail silently → type mismatch tracking needed (have it)
- ~~No backpressure~~ → FIXED: optional bounded mailbox with DropNewest/DropOldest (Cycle 6)
- ~~Panicked actors permanently poisoned~~ → FIXED: factory-based restart with max_restarts limit (Cycle 7)
- ~~Spin/sleep backoff wastes CPU~~ → FIXED: thread parking with instant wakeup (Cycle 3)
- ~~No lifecycle hooks~~ → FIXED: on_start/on_stop with default no-ops (Cycle 9)
- ~~No graceful stop~~ → FIXED: ctx.stop_self() (immediate) + runtime.stop_actor() (queued) (Cycle 9)
- No supervision trees (factory restart + lifecycle hooks are steps toward this)

## Lifecycle Hooks Deep Dive (Cycle 9)

| Framework | on_start | on_stop | on_panic | Self-stop | External stop |
|-----------|----------|---------|----------|-----------|---------------|
| Erlang | init/1 | terminate/2 (NOT on crash) | N/A | {stop,Reason,State} | gen_server:stop |
| Akka | preStart | postStop (always) | preRestart | context.stop(self) | PoisonPill / stop |
| Actix | started | stopped | N/A | ctx.stop() | addr.do_send(Stop) |
| Kameo | on_start | on_stop | on_panic | Context::stop() | stop_gracefully/kill |
| Ractor | pre_start | post_stop (NOT on kill/panic) | N/A | stop() | Signal::Kill |
| **Swactor** | **on_start** | **on_stop (NOT on panic)** | N/A | **ctx.stop_self()** | **runtime.stop_actor()** |

Design decisions:
- on_stop NOT called on panic (matches Erlang/Ractor — corrupt state is unsafe)
- ctx.stop_self() is immediate (after current message) via request_stop buffer
- runtime.stop_actor() uses StopSignal message (PoisonPill semantics — queued after existing msgs)
- on_start panics → immediate poison (no restart attempted — init failure is fatal)
- Restarted actors get on_start called again on fresh instance

## Work Stealing Deep Dive (Cycle 5)

### Cross-Runtime Comparison

| Aspect | Tokio | Go | BEAM | ForkJoinPool |
|--------|-------|-----|------|-------------|
| Queue | Fixed 256-slot ring | 256-slot ring + runnext | Per-priority linked | Growable array deque |
| Steal granularity | Half victim's queue | Half victim's runq | Individual processes | One task at a time |
| LIFO fast-path | Dedicated slot (3-use cap) | runnext (stealable 4th try) | None | Owner pops from top |
| Global queue | Mutex intrusive list | Checked 1/61 ticks | Per-priority migration | Even-indexed submit queues |
| Searcher limit | N/2 workers | GOMAXPROCS/2 | N/A (proactive migration) | Idle stack in ctl field |
| Balance strategy | Reactive steal | Reactive steal | **Proactive migration** + reactive | Reactive scan |
| Load compaction | No (spread) | No (spread) | **Yes** (min schedulers) | No (spread) |

### Key Patterns

1. **LIFO slot**: Every runtime has one. Improves cache locality by running the recipient immediately after the sender. Tokio caps at 3 consecutive uses to prevent starvation.

2. **Steal-half**: Tokio and Go both steal half the victim's queue. This amortizes the overhead of cross-thread coordination — O(1) per stolen item instead of O(1) per steal.

3. **N/2 searcher limit**: Both Tokio and Go cap concurrent searchers to prevent thundering herd. Without it, all N workers scanning causes O(N²) cache-line bouncing.

4. **BEAM's migration**: Unique dual approach — reactive stealing when idle, proactive migration via periodic `check_balance()` that computes migration paths based on average max queue length.

### Feasibility for Swactor

- **Full actor migration**: Mechanically possible (ActorSlot is Send), but has 1-tick message loss window and requires push-based donation (ActorPool not Sync → no pull stealing)
- **Message stealing without actors**: Impossible — actor IS the state, messages without the actor are meaningless
- **Transfer queue snooping**: Pointless without actor migration
- **Load-aware placement** ✅ IMPLEMENTED: Placement reads per-worker stats to bias toward lighter workers, with round-robin fallback when stats are equal

### Decision: Load-Aware Placement over Work Stealing

Chose load-aware placement because:
- Zero correctness risk (no message loss, no ordering changes)
- O(N) atomic loads per spawn (trivial for N≤8 workers)
- Handles the primary source of imbalance: uneven spawn distribution
- Full work stealing deferred — would require migration channels, address map coordination, and forwarding tombstones