swactor/CLAUDE/notes/progress.md

Progress Log

Current Stage: Phase 1 — Research + First Improvement Cycle

Status: Cycle 16 COMPLETE

Plan Overview

1. Phase 0: Codebase audit — understand current swactor architecture, existing tests, benchmarks ✅
2. Phase 1: Broad survey + interleaved improvements
3. Phase 2: Deeper improvements based on findings
4. Phase 3: Testing methodology improvements
5. Phase 4: Final evaluation & documentation

Completed This Session

Cycle 1: Fairness (Message Budget)

• Research: Studied ractor, tokio, Erlang/OTP BEAM, Linux CFS/EEVDF, libuv
• Finding: tick_all drained ENTIRE mailbox per actor per tick — critical fairness bug
  • BEAM uses 4000 reduction budget, tokio uses 128-op cooperative budget
  • Swactor had zero budget — one hot actor could starve all others on same worker
• Implementation: Added actor_message_budget to RuntimeConfig (default: 64)
  • Modified tick_all to break after budget messages per actor
  • budget=0 means unlimited (backward compatible)
• Tests: 3 new fairness tests (hot_actor_does_not_starve_cold_actor, unlimited_budget_drains_all, budget_messages_drain_across_multiple_ticks)
• Benchmarks: Added fairness benchmark group (cold_latency_under_pressure, throughput_by_budget)
• Fixes: Updated RuntimeConfig struct literals across crates (python, runtime-dashboard, mt_benchmarks)
• Result: 45 tests pass (42 original + 3 new), all workspace crates compile

Cycle 2: Stress Tests, Benchmarks, Research Expansion

• Research: Added Kameo and Actix analysis to synthesis
  • Actix uses custom Vyukov lock-free MPSC queue (why it's fastest)
  • Kameo has dual bounded/unbounded mailbox, default capacity 64
  • Both use vtable dispatch (not Box downcast)
  • Actix has 256-message assertion guard (validates our budget approach)
• Stress tests: 6 new tests
  • message_ordering_preserved_under_budget — FIFO order with budget=8
  • mt_stress_many_senders_one_receiver — 50 senders × 100 msgs, 4 threads
  • mt_stress_concurrent_spawn_and_send — 200 concurrent spawn+send, 4 threads
  • mt_chain_spawning_under_load — 50-level chain across 2 workers
  • mt_panic_isolation_under_load — 10 panicking + 10 healthy actors, 4 threads
  • sustained_throughput_does_not_drop_messages — 10 batches × 100 msgs
• Benchmarks: 2 new benchmark groups
  • msg_size: throughput and send_latency by message size (8B-4KB)
  • contention: fanin (1-100 senders to 1 sink), cross_worker (1-4 threads)
• Result: 51 tests pass (42 original + 3 fairness + 6 stress), all workspace compiles

Cycle 3: Thread Parking (Adaptive Backoff)

• Implementation: Replaced thread::sleep with thread::park_timeout in worker run loop
  • Workers register thread::current() via OnceLock<Thread> on startup
  • send_to and spawn call Thread::unpark() on target worker
  • Cross-worker sends from WorkerContext also unpark target
  • Zero new dependencies (uses std::sync::OnceLock + std::thread::park_timeout)
• Design source: Tokio's parker state machine, Linux NO_HZ adaptive ticks
• Benefits: Parked workers wake instantly when work arrives (vs waiting for sleep timer)
  • Reduces idle-to-active latency from up to 1ms to near-zero
  • No overhead on hot path — unpark() is no-op if thread isn't parked
• Tests: 1 new test (mt_parked_worker_wakes_on_send)
• Result: 52 tests pass (51 + 1 new), all workspace compiles

Cycle 4: Shutdown Fix + Bug-Inspired Tests

• Shutdown improvement: shutdown() now unparks all workers for immediate exit
  • Previously, parked workers wouldn't notice shutdown until park_timeout expired
• Bug-inspired tests (5 new, from competitor bug reports):
  • stats_snapshot_is_read_only — from ractor #310 (destructive get_children)
  • stats_under_load_do_not_interfere_with_processing — stats don't affect msg processing
  • shutdown_wakes_parked_workers_immediately — validates fast shutdown with parking
  • mt_send_after_run_delivers_to_running_actors — from kameo #185 (startup delivery)
  • budget_respected_even_with_self_sends — from actix #515 (mailbox bypass)
• Result: 57 tests pass, all workspace compiles

Cycle 5: Work Stealing Research + Load-Aware Placement

• Research: Deep analysis of work stealing in Tokio, Go, BEAM, ForkJoinPool
  • Tokio: fixed 256-slot ring, steal-half, LIFO slot (3-use starvation cap), N/2 searcher limit
  • Go: M:N scheduler, runnext + 256-slot local queue, steal-half, 4 tries with random permutation
  • BEAM: unique dual approach — reactive stealing + proactive migration via check_balance()
  • ForkJoinPool: owner LIFO / thief FIFO deque, even/odd queue indexing
• Feasibility analysis: Full actor migration IS mechanically possible (ActorSlot is Send), but:
  • Requires push-based donation (ActorPool not Sync → no pull stealing)
  • 1-tick message loss window during migration
  • Significant complexity for uncertain benefit
• Implementation: Load-aware placement replaces blind round-robin
  • Placement::next_worker() now reads per-worker stats (num_actors + mailbox_depth)
  • Scan starts from rotating position → round-robin when all stats equal (initial burst)
  • O(N) relaxed atomic loads per spawn, trivial for N≤8 workers
• Tests: 3 new tests
  • load_aware_placement_prefers_lighter_worker — imbalanced load biases toward lighter worker
  • load_aware_placement_single_worker_degrades_gracefully — single-thread works correctly
  • load_aware_placement_falls_back_to_round_robin_on_fresh_runtime — even distribution before ticks
• Benchmarks: 1 new group — placement/spawn_under_load (2t, 4t)
• Result: 60 tests pass, all workspace compiles

Cycle 9: Lifecycle Hooks + Graceful Stop

• Research: Cross-framework lifecycle analysis (Erlang init/terminate, Akka preStart/postStop, Actix started/stopping/stopped, Kameo on_start/on_stop/on_panic, Ractor pre_start/post_stop, Stakker state-based Prep/Ready/Zombie, CAF on_exit)
  • Also researched graceful stop across Erlang (gen_server:stop, exit, kill), Akka (stop, PoisonPill, Kill, gracefulStop), Actix (ctx.stop, Running::Stop), Kameo (stop_gracefully, kill), Go (context.Done)
  • Key finding: most frameworks have on_stop NOT called on panic (state may be corrupt)
  • Key finding: self-stop should be immediate (after current message), external stop is queued
• Implementation: Lifecycle hooks + dual-mode graceful stop
  • ActorInterface::on_start() and on_stop() — default no-ops, backward compatible
  • AnyActor::on_start()/on_stop() forwarded from Actor<A> impl
  • ctx.stop_self() — immediate stop after current message via request_stop buffer
  • runtime.stop_actor(addr) — external stop via StopSignal message (PoisonPill semantics)
  • ActorSlot gains started: bool and stopping: bool flags
  • on_start called in tick_all before first message; panic in on_start → immediate poison
  • on_stop called in cleanup_dead for stopping (not poisoned) actors, wrapped in catch_unwind
  • Restarted actors get started=false so on_start fires again on fresh instance
  • stops: AtomicU64 added to WorkerStats and WorkerInfo
  • ContextInner::request_stop() method for same-worker immediate stop
  • Phase 7 cleanup_dead now handles both poisoned AND stopping actors, with on_stop context
• Tests: 12 new tests
  • on_start_called_before_first_message — on_start fires on first tick, before messages
  • on_start_called_per_actor — 5 actors each get one on_start call
  • on_start_panic_poisons_actor — panic in on_start → poisoned, no messages processed
  • actor_can_stop_self — 5 msgs sent, stops after 3, only 3 processed, on_stop called
  • runtime_can_stop_actor — external stop via runtime, on_stop called, actor removed
  • send_to_stopped_actor_returns_error — stopped actor gone from address map
  • stop_vs_panic_tracked_separately_in_stats — stops and panics counted independently
  • on_stop_can_send_messages — farewell message sent during on_stop is delivered
  • on_start_called_again_after_restart — restartable actor gets on_start on fresh instance
  • external_stop_is_queued_after_pending_messages — PoisonPill semantics for external stop
  • external_stop_before_new_messages_prevents_processing — stop before send blocks msgs
  • stop_nonexistent_actor_returns_error — stop on bad address returns Err
• Result: 82 tests pass, all workspace compiles

Cycle 16: Benchmark New Features

• Scope: Added benchmark group for features from Cycles 12-15 (named actors, groups, monitors, ask)
• New benchmarks (5 total in registry group):
  • named_spawn_lookup — spawn_named + where_is roundtrip: ~2.4µs (vs bare spawn 1.9µs → +0.5µs overhead for name registration)
  • where_is_100_names — lookup in 100-name registry: ~9.0µs (includes setup overhead)
  • group_publish/{10,50,100} — broadcast to N members: 4.8µs/15.5µs/60µs (linear with O(N) clones)
  • monitor_setup — monitor + stop + cleanup: ~13.4µs
  • ask_roundtrip — ask + recv_ticking: ~4.5µs (vs manual roundtrip 3.0µs → +1.5µs for inbox creation)
• Analysis: All registry operations are efficient. Named lookup adds <1µs over bare spawn. Ask adds ~50% overhead vs manual inbox pattern (acceptable for convenience). Group publish scales linearly — expected for O(N) message cloning. No optimization needed.
• Result: All benchmarks run cleanly, 127 tests pass, zero warnings

Cycle 15: Ask Pattern (Request-Response)

• Research: Studied ask/call/request-response patterns across Erlang gen_server:call (From + reply), Akka ask (temporary actor + Future), Ractor call (RpcReplyPort), Kameo ask (async + Reply trait), xactor Handler (return value auto-routing)
  • Key finding: swactor's synchronous tick model requires explicit reply_to, not implicit routing
  • Decision: convenience wrapper over existing inbox pattern, not implicit auto-reply
• Implementation: Ask<R> struct + Runtime::ask() method
  • Ask<R>: wraps Inbox<R> with try_recv() and recv_ticking(rt, max_ticks)
  • rt.ask(addr, |reply_to| Msg { reply_to }) — creates inbox, builds message, sends, returns Ask
  • ask.recv_ticking(&rt, max_ticks) — ticks until response or timeout (single-threaded only)
  • ask.try_recv() — poll without ticking (works in both modes)
  • ask.reply_addr() — access inbox address for manual use
  • Purely sugar over new_inbox → send_to → tick → try_recv pattern
  • Zero changes to ContextInner or ActorInterface — no implicit auto-reply magic
• Tests: 5 new behavioral tests
  • ask_recv_ticking_returns_response — basic PingPong ask roundtrip
  • ask_multiple_times_tracks_state — 3 sequential asks to CounterActor
  • ask_timeout_when_no_response — ask dead actor → timeout error
  • ask_try_recv_returns_none_before_tick — poll before tick → None, after tick → Some
  • ask_reply_addr_is_accessible — reply address is valid
• Result: 127 tests pass (120 behavioral + 7 proptest), all workspace compiles, zero warnings

Cycle 14: Actor Groups (Pub-Sub)

• Research: Studied group/pub-sub patterns across Erlang pg (scopes, join/leave/get_members), Akka DistributedPubSub (mediator, topics), Ractor pg (join/leave/broadcast), Bastion (Dispatcher), Redis pub/sub (channels, patterns)
  • Common patterns: auto-cleanup on death, at-most-once delivery, string-based naming, flat groups (not hierarchical), lazy creation/deletion
  • Decision: Erlang pg-style flat groups, string keys, auto-cleanup, RwLock pattern
• Implementation: GroupRegistry in delivery.rs with forward + reverse maps
  • groups: RwLock<HashMap<String, HashSet<ActorAddress>>> — group→members
  • memberships: RwLock<HashMap<ActorAddress, HashSet<String>>> — actor→groups (reverse for cleanup)
  • Groups auto-create on first join, auto-delete when empty
  • Runtime API: join_group(addr, name), leave_group(addr, name), publish_to(group, msg), group_members(group), groups()
  • Ctx API: join_group(name), leave_group(name), publish(group, msg), group_members(group)
  • publish clones at the typed level (Message: Clone), sends to each member via normal routing
  • Auto-cleanup: group_registry.cleanup(&addr) in cleanup_dead phase removes dead actor from all groups
  • ContextInner extended: join_group(), leave_group(), group_members() (publish is Ctx-level only)
• Tests: 9 new behavioral tests
  • group_members_returns_joined_actors — join + query
  • empty_group_returns_no_members — nonexistent group → empty
  • publish_broadcasts_to_all_members — 2 members, both receive
  • leave_group_stops_receiving_publishes — leave → excluded from broadcast
  • dead_actor_auto_removed_from_group — stop → removed from group
  • actor_removed_from_all_groups_on_death — multi-group membership cleanup
  • empty_group_auto_deleted — last member leaves → group removed from groups()
  • ctx_join_group_from_handler — join via on_start
  • ctx_publish_broadcasts_from_handler — publish via handler
• Result: 122 tests pass (115 behavioral + 7 proptest), all workspace compiles, zero warnings

Cycle 13: Actor Monitoring / Death Watch

• Research: Studied monitoring across Erlang (monitor/2, DOWN messages), Akka (watch/Terminated), Ractor (link, SupervisionEvent), Actix (none), Kameo (link, on_link_died callback)
  • Key finding: Erlang's unidirectional monitor + message delivery is the best fit for swactor (reuses existing type-erased handler, zero trait changes, composable)
  • Callbacks (Ractor/Kameo style) rejected: would require adding to AnyActor/ActorInterface traits
  • Bidirectional links deferred: can layer on top of monitors later
• Implementation: MonitorRegistry in delivery.rs + Down/StopReason/MonitorRef in actor.rs
  • MonitorRegistry: RwLock<HashMap<ActorAddress, Vec<(MonitorRef, ActorAddress)>>> (watched→watchers)
    • reverse RwLock<HashMap<MonitorRef, ActorAddress>> for O(1) demonitor
  • MonitorRef(u64): unique token from AtomicU64 counter
  • Down { addr: ActorAddress, reason: StopReason }: delivered as normal mailbox message
  • StopReason: Normal (graceful stop) | Panicked (panic, not restartable)
  • ctx.monitor(target) → MonitorRef — subscribe to death notifications
  • ctx.demonitor(mref) — cancel a subscription
  • cleanup_dead now returns Vec<(ActorAddress, StopReason)> instead of Vec<ActorAddress>
  • After cleanup_dead: iterate dead actors, take_monitors from registry, route Down through normal delivery (pool.deliver for same-worker, transfer_txs for cross-worker, inbox_registry for inboxes)
  • Dead watcher cleanup: remove_watcher() strips monitor subscriptions for dead watchers
  • Multiple monitors of same target produce independent notifications (stacking, like Erlang)
• Tests: 7 new behavioral tests
  • monitor_notifies_on_graceful_stop — Down{reason: Normal} on stop
  • monitor_notifies_on_panic — Down{reason: Panicked} on panic
  • multiple_watchers_all_notified — two watchers both get Down
  • demonitor_cancels_notification — demonitor → no Down delivered
  • dead_watcher_does_not_receive_down — dead watcher's monitors cleaned up
  • down_delivered_to_external_inbox — Down forwarded through inbox
  • stacked_monitors_produce_multiple_notifications — two monitors on same target → two Downs
• Result: 113 tests pass (106 behavioral + 7 proptest), all workspace compiles, zero warnings

Cycle 12: Named Actor Registry

• Research: Studied named actor/service discovery across Erlang (register/2, whereis/1, global, pg), Actix (Registry, SystemRegistry — TypeId keys), Bastion (hierarchy-based), Ractor (String keys, DashMap, global static), xactor (TypeId singleton), Akka (Receptionist, ServiceKey[T])
  • Key findings: TypeId keys (Actix/xactor) don't fit swactor's type-erased model; global static (Ractor) breaks multi-runtime scenarios; Erlang's register/whereis is the gold standard
  • Decision: String keys, RwLock (matches existing AddressMap/InboxRegistry pattern), per-runtime scope, error on collision, auto-unregister on death
• Implementation: NameRegistry in delivery.rs with forward + reverse maps
  • NameRegistry: RwLock<HashMap<String, ActorAddress>> + RwLock<HashMap<ActorAddress, String>>
  • Forward map for O(1) name→addr lookup, reverse map for O(1) addr→name cleanup
  • Added to Runtime as Arc<NameRegistry>, threaded through TickContext
  • Runtime API: spawn_named(name, actor), where_is(name), unregister(name), registered_names()
  • Ctx API: spawn_named(name, actor), where_is(name) — usable from inside handlers
  • ContextInner trait extended: where_is() + register_name() (private, supports both Runtime and WorkerContext)
  • Auto-unregister on death: cleanup_dead phase calls name_registry.unregister_by_addr() for each dead actor
  • Name reservation is immediate (before spawn queue push) — prevents TOCTOU race
  • Collision returns Err("Name already registered") — original binding preserved
• Tests: 11 new behavioral tests
  • named_actor_lookup_returns_spawn_address — spawn_named → where_is roundtrip
  • named_actor_receives_messages_via_lookup — send to looked-up address works
  • duplicate_name_returns_error — collision error, original preserved
  • where_is_returns_none_for_unknown_name — nonexistent name → None
  • name_auto_unregistered_on_actor_death — stop_actor → name freed
  • name_can_be_reused_after_actor_death — death → respawn with same name
  • name_auto_unregistered_on_panic — panic → name freed
  • registered_names_lists_all — all registered names returned
  • manual_unregister_frees_name_but_actor_lives — unregister doesn't kill actor
  • ctx_where_is_resolves_inside_handler — where_is from handler context
  • ctx_spawn_named_registers_from_handler — spawn_named from handler context
• Result: 106 tests pass (99 behavioral + 7 proptest), all workspace compiles, zero warnings

Cycle 11: Property-Based Testing (proptest + fuzz extension)

• Research: Studied testing approaches across tokio (loom), Erlang (PropEr, QuickCheck, Concuerror), Rust property-based testing (proptest vs quickcheck), cargo-fuzz, and actor-specific testing patterns.
  • Ranked approaches: #1 proptest-state-machine (perfect fit for deterministic ticks), #2 extend cargo-fuzz, #3 simple proptest, #4 shuttle, #5 loom, #6 DST
  • Also researched remaining feature gaps: named actors, monitoring/death watch, groups, ask pattern
• Implementation: Property-based testing suite with proptest-state-machine
  • Added proptest and proptest-state-machine to dev-dependencies
  • New test file: tests/proptest_runtime.rs with 7 tests:
    • fifo_ordering_for_any_message_sequence — FIFO preserved for 1-100 random messages
    • budget_limits_per_actor_processing — budget caps per-tick processing for 2-10 actors
    • one_shot_timer_fires_at_correct_tick — timer with delay 1-20 fires at exact right tick
    • interval_timer_fires_at_correct_period — period 1-10, verifies 3 consecutive fires
    • bounded_mailbox_never_exceeds_capacity — capacity 1-20, 1-200 messages, never exceeds
    • spawn_n_actors_all_tracked — 1-50 actors, all unique, all in stats
    • swactor_state_machine — stateful property test: random Spawn/Send/Tick/Stop/CheckStats sequences (up to 40 transitions, 128 cases), verifies runtime invariants after each step
  • State machine test defines SwactorModel (reference) vs SwactorTest (SUT) with:
    • Reference model: HashMap<id, alive> tracking expected actor lifecycle
    • Invariants checked after every transition: worker count, actor placement, mailbox safety
    • Automatic shrinking finds minimal failing sequences
  • Extended fuzz targets (fuzz_runtime.rs) with 4 new RawAction variants:
    • StopActor — graceful stop via runtime.stop_actor
    • SpawnRestartable — spawn_restartable with configurable max_restarts
    • ScheduleTimer — one-shot timer via TimerSchedulerActor
    • ScheduleInterval — interval timer via IntervalSchedulerActor
  • Added 3 new actor types to fuzz: TimerSchedulerActor, IntervalSchedulerActor, RestartableEchoActor
• Bug found: State machine test immediately caught invariant mismatch: address map tracks spawned actors immediately, but per-worker num_actors lags until first tick. Fixed invariant to use <= check.
• Result: 95 tests pass (88 behavioral + 7 proptest), fuzz targets compile, zero warnings

Cycle 10: Actor Timers (Tick-Counting)

• Research: Studied timer/scheduling patterns across Erlang (timer:send_after, erlang:start_timer), Akka (scheduleOnce, scheduler), Actix (ctx.run_later, ctx.run_interval), Kameo (tokio::time::sleep), Tokio (tokio::time), Go (time.After, time.NewTicker)
  • Also researched priority messages (REJECTED: lifecycle hooks cover 95% of use cases)
  • Also researched SmallBox optimization (DEFERRED: measure allocation cost first)
  • Key finding: per-worker tick-counting is ideal for swactor's synchronous model (deterministic)
• Implementation: Per-worker TimerWheel with deterministic tick-based scheduling
  • OnceTimer: fire once at fire_at tick, consumed after firing
  • IntervalTimer: fire every period ticks, message cloned via CloneMsg trait
  • CloneMsg trait: type-erased clone for interval timer messages (blanket impl for Message)
  • TimerRequest enum: Once { dest, msg, ticks } | Interval { dest, msg, period }
  • ctx.send_after_ticks(addr, msg, ticks) — one-shot timer API
  • ctx.send_interval_ticks(addr, msg, period) — interval timer API
  • Phase 2.5 in tick_once: fire due timers, route through full delivery system (pool.deliver for local actors, transfer_txs for cross-worker, inbox_registry for inboxes)
  • Phase 5.5: drain timer requests from handler buffer into TimerWheel
  • GC: interval timers for removed actors cleaned up after cleanup_dead
  • schedule_timer on Runtime's ContextInner: no-op with warning (timers are per-worker only)
• Bug fixed: gc_dead_intervals was over-aggressive — removed timers for ANY address not in the local pool, including inboxes and cross-worker actors. Fixed to only GC timers for addresses in the dead set from cleanup_dead.
• Tests: 6 new tests
  • one_shot_timer_fires_after_n_ticks — timer with delay=3 fires on tick 4
  • handler_can_schedule_one_shot_timer — timer scheduled from handler, fires correctly
  • one_shot_timer_fires_only_once — consumed after firing, doesn't repeat
  • interval_timer_fires_repeatedly — period=2, fires every 2 ticks (3 fires verified)
  • interval_timer_cleaned_up_when_actor_dies — GC removes orphaned timers
  • timer_with_zero_delay_fires_next_tick — delay=0 fires on next tick
• Result: 88 tests pass, all workspace compiles, zero warnings

Cycle 8: Dead Actor Cleanup (Memory Leak Fix)

• Research: Audited remaining improvement gaps, found ActorPool and AddressMap both leak permanently poisoned actors. Known class of bug in Akka (#22990), CAF (#420).
• Implementation: Automatic cleanup of poisoned actors after tick_all
  • AddressMap::remove() added to delivery.rs
  • ActorPool::cleanup_dead() collects and removes poisoned actors, returns their addresses
  • Phase 7 in tick_once: cleanup_dead → remove from address_map → update num_actors stat
  • Re-publish num_actors after cleanup so stats immediately reflect removal
• Behavior change: Sends to poisoned actors now return Err (address not found) instead of silently discarding. This is better — callers learn the actor is gone.
• Tests: 2 new tests + 2 existing tests updated
  • dead_actor_cleaned_up_from_stats_and_address_map — good actor persists, bad actor removed
  • bulk_dead_actor_cleanup — 20 panicked actors all cleaned up
  • Updated send_to_poisoned_actor_is_a_silent_black_hole — now asserts send returns Err
  • Updated poisoned_actor_messages_not_counted_as_processed — sends fail to cleaned-up actor
• Result: 70 tests pass, all workspace compiles

Cycle 7: Actor Recovery (Factory Restart)

• Research: Deep analysis of supervision/recovery across Erlang (supervision trees, restart intensity), Akka (Resume/Restart/Stop/Escalate), Kameo (on_panic hook), Actix (Supervised trait), Ractor (SupervisionEvent)
  • Erlang: fresh process via factory (MFA tuple), mailbox lost, PID changes
  • Akka: replace internals but keep ActorRef stable, mailbox preserved (docs say this is usually wrong)
  • Kameo: on_panic(&mut self) — risky with corrupt state after panic
  • Decision: factory-based restart (Erlang-style), safest approach
• Implementation: spawn_restartable(actor, factory, max_restarts) on Runtime and Ctx
  • Actor<A> expanded from tuple struct to named fields: inner, restart_factory, max_restarts, restart_count
  • AnyActor::try_restart(&self) trait method (default None, backward compatible)
  • Factory stored as Arc<dyn Fn() -> A + Send + Sync> — cloned into fresh Actor on restart
  • tick_all panic handler: try_restart before poisoning, clear mailbox, fresh state
  • restarts counter added to WorkerStats and WorkerInfo
• Safety: Factory fields are "cold" (never touched by handle_any), safe to read after catch_unwind
• Tests: 4 new tests
  • restartable_actor_recovers_after_panic — basic restart works
  • restartable_actor_resets_state_on_restart — fresh state post-restart
  • restartable_actor_respects_max_restarts — 2 restarts then permanent poison
  • non_restartable_actor_still_poisons_on_panic — backward compatibility
• Result: 68 tests pass, all workspace compiles

Cycle 6: Mailbox Backpressure

• Research: Compared backpressure across Erlang (unbounded, pobox), Actix (cap 16, do_send bypass), Kameo (cap 64, bounded), Tokio mpsc (bounded, permit pattern), Go channels (blocking)
  • Consensus: bounded by default, configurable overflow policy
• Implementation: Per-actor bounded mailboxes with configurable overflow
  • Added MailboxOverflow enum: DropNewest (discard incoming) and DropOldest (evict oldest)
  • Added default_mailbox_capacity and mailbox_overflow to RuntimeConfig
  • Default: capacity=0 (unbounded) — 100% backward compatible
  • ActorSlot stores per-actor capacity and policy (from runtime defaults)
  • deliver() enforces bounds; dropped messages tracked via drops_this_tick counter
  • messages_dropped: AtomicU64 added to WorkerStats and WorkerInfo
• Tests: 4 new tests
  • bounded_mailbox_drop_newest_caps_at_capacity — 50 msgs, cap 10 → only 10 delivered
  • bounded_mailbox_drop_oldest_keeps_newest — 10 msgs, cap 5 → newest 5 kept
  • unbounded_mailbox_delivers_all_messages — backward compatibility check
  • bounded_mailbox_refills_after_processing — cap 5, process, refill works
• Result: 64 tests pass, all workspace compiles

Research Notes

• Full analysis in CLAUDE/notes/research_synthesis.md
• Baseline benchmarks in CLAUDE/notes/baseline_benchmarks.md
• Constraints in CLAUDE/notes/constraints.md

Next Steps

• Cycle 2: Stress testing + property-based tests ✅
• Cycle 3: Adaptive backoff with thread parking ✅
• Cycle 4: Enhanced benchmarks + bug-inspired tests ✅
• Cycle 5: Work stealing research + load-aware placement ✅
• Cycle 6: Mailbox backpressure ✅
• Cycle 7: Actor recovery (factory restart) ✅
• Cycle 8: Dead actor cleanup ✅
• Cycle 9: Lifecycle hooks + graceful stop ✅
• Cycle 10: Actor timers (tick-counting) ✅
• Cycle 11: Property-based testing (proptest-state-machine + fuzz extension) ✅
• Cycle 12: Next improvement
  • Candidates: named actors/registry (small effort, high value), actor monitoring/death watch, actor groups/pub-sub, SmallBox optimization
  • Priority messages REJECTED (lifecycle hooks cover 95% of cases)
  • LIFO slot rejected (0-5% benefit for typical workloads, not worth complexity)

Open Questions

• Should budget be configurable per-actor (not just per-runtime)?
• Is 64 the right default budget? Benchmarks show budget=32 slightly faster for throughput
• Thread parking: notification mechanism RESOLVED: OnceLock + unpark()
• Should load-aware placement weight mailbox depth more than actor count?
• LIFO slot for same-worker sends: worth the complexity?

Blockers

• (none)