zacheryasc/swactor
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
swactor/CLAUDE/notes/progress.md
Developer acacc1b758 feat: thread parking for instant worker wakeup 
Replace thread::sleep with thread::park_timeout in worker backoff loop.
Workers register their thread handle via OnceLock<Thread> on startup.
When send_to or spawn routes work to a worker, Thread::unpark() wakes
it instantly instead of waiting for the sleep timer to expire.

Inspired by tokio's parker state machine and Linux NO_HZ adaptive ticks.

Implementation:
- Runtime stores Vec<OnceLock<Thread>> for worker thread handles
- Workers call OnceLock::set(thread::current()) on startup
- Runtime::send_any, spawn_any, and WorkerContext cross-worker sends
  call notify_worker() → Thread::unpark() on the target worker
- TickContext carries worker_threads reference for cross-worker notification
- Zero new dependencies (std::sync::OnceLock + std:🧵:park_timeout)

Benefits:
- Parked workers wake instantly when work arrives (vs up to 1ms sleep delay)
- No overhead on hot path — unpark() is no-op if thread isn't parked
- Single-threaded tick() mode unaffected (OnceLock never set)

All 58 tests pass (52 runtime_api + 5 transport + 1 doctest).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 11:31:34 +00:00

4.6 KiB
Raw Blame History

Progress Log

Current Stage: Phase 1 — Research + First Improvement Cycle

Status: Cycle 3 COMPLETE

Plan Overview

  1. Phase 0: Codebase audit — understand current swactor architecture, existing tests, benchmarks ✅
  2. Phase 1: Broad survey + interleaved improvements
  3. Phase 2: Deeper improvements based on findings
  4. Phase 3: Testing methodology improvements
  5. Phase 4: Final evaluation & documentation

Completed This Session

Cycle 1: Fairness (Message Budget)

  • Research: Studied ractor, tokio, Erlang/OTP BEAM, Linux CFS/EEVDF, libuv
  • Finding: tick_all drained ENTIRE mailbox per actor per tick — critical fairness bug
    • BEAM uses 4000 reduction budget, tokio uses 128-op cooperative budget
    • Swactor had zero budget — one hot actor could starve all others on same worker
  • Implementation: Added actor_message_budget to RuntimeConfig (default: 64)
    • Modified tick_all to break after budget messages per actor
    • budget=0 means unlimited (backward compatible)
  • Tests: 3 new fairness tests (hot_actor_does_not_starve_cold_actor, unlimited_budget_drains_all, budget_messages_drain_across_multiple_ticks)
  • Benchmarks: Added fairness benchmark group (cold_latency_under_pressure, throughput_by_budget)
  • Fixes: Updated RuntimeConfig struct literals across crates (python, runtime-dashboard, mt_benchmarks)
  • Result: 45 tests pass (42 original + 3 new), all workspace crates compile

Cycle 2: Stress Tests, Benchmarks, Research Expansion

  • Research: Added Kameo and Actix analysis to synthesis
    • Actix uses custom Vyukov lock-free MPSC queue (why it's fastest)
    • Kameo has dual bounded/unbounded mailbox, default capacity 64
    • Both use vtable dispatch (not Box downcast)
    • Actix has 256-message assertion guard (validates our budget approach)
  • Stress tests: 6 new tests
    • message_ordering_preserved_under_budget — FIFO order with budget=8
    • mt_stress_many_senders_one_receiver — 50 senders × 100 msgs, 4 threads
    • mt_stress_concurrent_spawn_and_send — 200 concurrent spawn+send, 4 threads
    • mt_chain_spawning_under_load — 50-level chain across 2 workers
    • mt_panic_isolation_under_load — 10 panicking + 10 healthy actors, 4 threads
    • sustained_throughput_does_not_drop_messages — 10 batches × 100 msgs
  • Benchmarks: 2 new benchmark groups
    • msg_size: throughput and send_latency by message size (8B-4KB)
    • contention: fanin (1-100 senders to 1 sink), cross_worker (1-4 threads)
  • Result: 51 tests pass (42 original + 3 fairness + 6 stress), all workspace compiles

Cycle 3: Thread Parking (Adaptive Backoff)

  • Implementation: Replaced thread::sleep with thread::park_timeout in worker run loop
    • Workers register thread::current() via OnceLock<Thread> on startup
    • send_to and spawn call Thread::unpark() on target worker
    • Cross-worker sends from WorkerContext also unpark target
    • Zero new dependencies (uses std::sync::OnceLock + std::thread::park_timeout)
  • Design source: Tokio's parker state machine, Linux NO_HZ adaptive ticks
  • Benefits: Parked workers wake instantly when work arrives (vs waiting for sleep timer)
    • Reduces idle-to-active latency from up to 1ms to near-zero
    • No overhead on hot path — unpark() is no-op if thread isn't parked
  • Tests: 1 new test (mt_parked_worker_wakes_on_send)
  • Result: 52 tests pass (51 + 1 new), all workspace compiles

Research Notes

  • Full analysis in CLAUDE/notes/research_synthesis.md
  • Baseline benchmarks in CLAUDE/notes/baseline_benchmarks.md
  • Constraints in CLAUDE/notes/constraints.md

Next Steps

  • Cycle 2: Stress testing + property-based tests
    • Concurrent spawn+send stress tests
    • Multi-threaded fairness validation
    • Property: message ordering preserved under budget
    • Property: all messages eventually delivered with budget > 0
  • Cycle 3: Adaptive backoff with thread parking ✅
  • Cycle 4: Enhanced benchmarks
    • Message size sensitivity (8B, 64B, 256B, 1KB)
    • Latency percentiles (p50, p99, p999)
    • Many-to-one fanin contention
    • Cross-worker vs same-worker delivery comparison
  • Cycle 5: Work stealing exploration
    • Evaluate feasibility of actor migration between workers
    • BEAM two-tier approach: reactive steal + periodic migration

Open Questions

  • Should budget be configurable per-actor (not just per-runtime)?
  • Is 64 the right default budget? Benchmarks show budget=32 slightly faster for throughput
  • Thread parking: how to handle the notification mechanism without adding deps?

Blockers

  • (none)