acacc1b758
Replace thread::sleep with thread::park_timeout in worker backoff loop.
Workers register their thread handle via OnceLock<Thread> on startup.
When send_to or spawn routes work to a worker, Thread::unpark() wakes
it instantly instead of waiting for the sleep timer to expire.
Inspired by tokio's parker state machine and Linux NO_HZ adaptive ticks.
Implementation:
- Runtime stores Vec<OnceLock<Thread>> for worker thread handles
- Workers call OnceLock::set(thread::current()) on startup
- Runtime::send_any, spawn_any, and WorkerContext cross-worker sends
call notify_worker() → Thread::unpark() on the target worker
- TickContext carries worker_threads reference for cross-worker notification
- Zero new dependencies (std::sync::OnceLock + std:🧵:park_timeout)
Benefits:
- Parked workers wake instantly when work arrives (vs up to 1ms sleep delay)
- No overhead on hot path — unpark() is no-op if thread isn't parked
- Single-threaded tick() mode unaffected (OnceLock never set)
All 58 tests pass (52 runtime_api + 5 transport + 1 doctest).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
87 lines
4.6 KiB
Markdown
87 lines
4.6 KiB
Markdown
# Progress Log
|
||
|
||
## Current Stage: Phase 1 — Research + First Improvement Cycle
|
||
|
||
### Status: Cycle 3 COMPLETE
|
||
|
||
## Plan Overview
|
||
1. **Phase 0**: Codebase audit — understand current swactor architecture, existing tests, benchmarks ✅
|
||
2. **Phase 1**: Broad survey + interleaved improvements
|
||
3. **Phase 2**: Deeper improvements based on findings
|
||
4. **Phase 3**: Testing methodology improvements
|
||
5. **Phase 4**: Final evaluation & documentation
|
||
|
||
## Completed This Session
|
||
|
||
### Cycle 1: Fairness (Message Budget)
|
||
- **Research**: Studied ractor, tokio, Erlang/OTP BEAM, Linux CFS/EEVDF, libuv
|
||
- **Finding**: `tick_all` drained ENTIRE mailbox per actor per tick — critical fairness bug
|
||
- BEAM uses 4000 reduction budget, tokio uses 128-op cooperative budget
|
||
- Swactor had zero budget — one hot actor could starve all others on same worker
|
||
- **Implementation**: Added `actor_message_budget` to `RuntimeConfig` (default: 64)
|
||
- Modified `tick_all` to break after `budget` messages per actor
|
||
- `budget=0` means unlimited (backward compatible)
|
||
- **Tests**: 3 new fairness tests (hot_actor_does_not_starve_cold_actor, unlimited_budget_drains_all, budget_messages_drain_across_multiple_ticks)
|
||
- **Benchmarks**: Added fairness benchmark group (cold_latency_under_pressure, throughput_by_budget)
|
||
- **Fixes**: Updated RuntimeConfig struct literals across crates (python, runtime-dashboard, mt_benchmarks)
|
||
- **Result**: 45 tests pass (42 original + 3 new), all workspace crates compile
|
||
|
||
### Cycle 2: Stress Tests, Benchmarks, Research Expansion
|
||
- **Research**: Added Kameo and Actix analysis to synthesis
|
||
- Actix uses custom Vyukov lock-free MPSC queue (why it's fastest)
|
||
- Kameo has dual bounded/unbounded mailbox, default capacity 64
|
||
- Both use vtable dispatch (not Box<dyn Any> downcast)
|
||
- Actix has 256-message assertion guard (validates our budget approach)
|
||
- **Stress tests**: 6 new tests
|
||
- `message_ordering_preserved_under_budget` — FIFO order with budget=8
|
||
- `mt_stress_many_senders_one_receiver` — 50 senders × 100 msgs, 4 threads
|
||
- `mt_stress_concurrent_spawn_and_send` — 200 concurrent spawn+send, 4 threads
|
||
- `mt_chain_spawning_under_load` — 50-level chain across 2 workers
|
||
- `mt_panic_isolation_under_load` — 10 panicking + 10 healthy actors, 4 threads
|
||
- `sustained_throughput_does_not_drop_messages` — 10 batches × 100 msgs
|
||
- **Benchmarks**: 2 new benchmark groups
|
||
- `msg_size`: throughput and send_latency by message size (8B-4KB)
|
||
- `contention`: fanin (1-100 senders to 1 sink), cross_worker (1-4 threads)
|
||
- **Result**: 51 tests pass (42 original + 3 fairness + 6 stress), all workspace compiles
|
||
|
||
### Cycle 3: Thread Parking (Adaptive Backoff)
|
||
- **Implementation**: Replaced `thread::sleep` with `thread::park_timeout` in worker run loop
|
||
- Workers register `thread::current()` via `OnceLock<Thread>` on startup
|
||
- `send_to` and `spawn` call `Thread::unpark()` on target worker
|
||
- Cross-worker sends from `WorkerContext` also unpark target
|
||
- Zero new dependencies (uses `std::sync::OnceLock` + `std::thread::park_timeout`)
|
||
- **Design source**: Tokio's parker state machine, Linux NO_HZ adaptive ticks
|
||
- **Benefits**: Parked workers wake instantly when work arrives (vs waiting for sleep timer)
|
||
- Reduces idle-to-active latency from up to 1ms to near-zero
|
||
- No overhead on hot path — `unpark()` is no-op if thread isn't parked
|
||
- **Tests**: 1 new test (`mt_parked_worker_wakes_on_send`)
|
||
- **Result**: 52 tests pass (51 + 1 new), all workspace compiles
|
||
|
||
### Research Notes
|
||
- Full analysis in `CLAUDE/notes/research_synthesis.md`
|
||
- Baseline benchmarks in `CLAUDE/notes/baseline_benchmarks.md`
|
||
- Constraints in `CLAUDE/notes/constraints.md`
|
||
|
||
## Next Steps
|
||
- [ ] **Cycle 2: Stress testing + property-based tests**
|
||
- Concurrent spawn+send stress tests
|
||
- Multi-threaded fairness validation
|
||
- Property: message ordering preserved under budget
|
||
- Property: all messages eventually delivered with budget > 0
|
||
- [x] **Cycle 3: Adaptive backoff with thread parking** ✅
|
||
- [ ] **Cycle 4: Enhanced benchmarks**
|
||
- Message size sensitivity (8B, 64B, 256B, 1KB)
|
||
- Latency percentiles (p50, p99, p999)
|
||
- Many-to-one fanin contention
|
||
- Cross-worker vs same-worker delivery comparison
|
||
- [ ] **Cycle 5: Work stealing exploration**
|
||
- Evaluate feasibility of actor migration between workers
|
||
- BEAM two-tier approach: reactive steal + periodic migration
|
||
|
||
## Open Questions
|
||
- Should budget be configurable per-actor (not just per-runtime)?
|
||
- Is 64 the right default budget? Benchmarks show budget=32 slightly faster for throughput
|
||
- Thread parking: how to handle the notification mechanism without adding deps?
|
||
|
||
## Blockers
|
||
- (none)
|