Compare commits

..

13 commits

Author SHA1 Message Date
Claude
d9489ccac2 feat: consolidate tests 2026-02-13 14:48:35 +00:00
Claude
41af749bb4 refactor: decompose tick_once into helper methods
Extract drain_spawns() (phases 1+4, dedup) and cleanup_dead_actors()
    (phase 7, 48 lines of complex death notification + GC logic).

    tick_once: 196 → 133 lines (−32%), max nesting depth: 5 → 3.

    Authored by Claude, lovingly guided by Zachery Aaron Shores-Chmielewski
2026-02-13 14:48:35 +00:00
Claude
5a8aeaf428 bench: add allocation decomposition benchmark for send_to
Measures Box::new+type-erasure (2-160ns) vs full send_to (1.9-7.2us) to
identify the real send latency bottleneck. Finding: allocation is only
1-5% of total cost; the dominant overhead is RwLock address_map lookup
and HybridChannel operations.

Authored by Claude, lovingly guided by Zachery Aaron Shores-Chmielewski
2026-02-13 14:48:35 +00:00
Claude
d370a41cf1 feat: add fn_complexity.py and loc_analysis.py analysis tools
fn_complexity: per-function line count, nesting depth, and unsafe detection.
loc_analysis: per-file breakdown of logic vs comment vs blank vs string lines.
Both support --json, --top N, directory recursion.

Key finding: tick_once is 196 lines (5x runner-up), core is 1,615 logic lines.

Authored by Claude, lovingly guided by Zachery Aaron Shores-Chmielewski
2026-02-13 14:48:35 +00:00
Claude
e21a4ae19c chore: clean up unused imports in test common module
Remove redundant AtomicUsize/Ordering/Arc re-exports from common/mod.rs
and runtime_mechanics.rs. Zero warnings now.

Authored by Claude, lovingly guided by Zachery Aaron Shores-Chmielewski
2026-02-13 14:48:35 +00:00
Claude
133314903a test: add 11 scenario tests for runtime-dashboard (from 0)
EventStore cursor semantics (streaming, overflow, future cursor), recording
pipeline (JSON roundtrip, destructive drain), StatsCollector multi-worker
aggregation, and event sequence monotonicity.

Authored by Claude, lovingly guided by Zachery Aaron Shores-Chmielewski
2026-02-13 14:48:35 +00:00
Claude
803289c7dc docs: update runtime docs to reflect architecture changes from Cycles 1-3
Worker struct: TimerWheel → generic WorkerExtension. TickContext: registries
moved behind RuntimeExtension. Phases 2.5/5.5/7 generalized. Ctx API split
into core (6 methods) vs extension traits (swactor-std).

Authored by Claude, lovingly guided by Zachery Aaron Shores-Chmielewski
2026-02-13 14:48:35 +00:00
Claude
33df021afd refactor: split runtime_api.rs (4,622 lines) into 7 focused test files
Split monolithic test file into domain-specific test modules:
- common/mod.rs: shared messages, actors, helpers
- runtime_lifecycle.rs (57 tests): spawn, FIFO, threading, panic safety
- runtime_mechanics.rs (10): placement, backpressure, cleanup, recovery
- runtime_hooks.rs (12): on_start/on_stop, graceful stop
- runtime_timers.rs (6): one-shot and interval timers
- runtime_registry.rs (32): naming, monitoring, groups, ask pattern
- runtime_supervision.rs (20): supervisor + router
- runtime_hasher.rs (3): identity hasher correctness

All 140 tests pass.

Authored by Claude, lovingly guided by Zachery Aaron Shores-Chmielewski
2026-02-13 14:48:35 +00:00
Claude
edea0aac2a refactor: extract 3,432 lines of embedded HTML to include_str!()
Move HTML content from Rust raw string literals into proper .html
files, replacing each _html.rs with a 1-line include_str!() call.

- runtime-dashboard: dashboard.html (511), actors.html (739),
  distribution.html (746)
- simulation-dashboard: dashboard.html (1,436)

Enables proper HTML syntax highlighting and linting in editors.

Authored by Claude, lovingly guided by Zachery Aaron Shores-Chmielewski
2026-02-13 14:48:35 +00:00
Claude
00e1fce80d refactor: deduplicate message routing pattern in tick_once
Extract route_to_pool_or_remote() helper that unifies the routing
logic (local pool → address_map → inbox_registry) used in phases
2.5 (worker extension delivery) and 7 (death notifications).

Completes P0 core debloating: 2636→2319 (−317, −12.0%).
worker.rs: 845→598 (−247, −29.2%).

Authored by Claude, lovingly guided by Zachery Aaron Shores-Chmielewski
2026-02-13 21:45:44 +07:00
Claude
09d5c75c39 feat: move TimerWheel from core to std via WorkerExtension
Add WorkerExtension trait (on_tick, handle_request, gc_dead) as a
per-worker counterpart to the shared RuntimeExtension. This enables
per-worker state like timer wheels to live outside core.

- TimerWheel, TimerRequest, CloneMsg → crates/std/src/timer_wheel.rs
- CtxTimers extension trait replaces Ctx::send_after_ticks/send_interval_ticks
- schedule_timer → generic post_worker_request on ContextInner
- Worker.timers → Worker.worker_ext (Option<Box<dyn WorkerExtension>>)
- StdExtension factory creates TimerWheel per worker

Core: 2440→2320 (−120). worker.rs: 696→599 (−97).
Cumulative: 2636→2320 (−316, −12.0%). worker.rs: 845→599 (−29.1%).

Authored by Claude, lovingly guided by Zachery Aaron Shores-Chmielewski
2026-02-13 21:45:44 +07:00
Claude
93b1b1fa1e feat: move WatchRegistry from core to std extension
Merge Watch death-notification system into swactor-std, reducing core
surface. WatchRegistry now lives alongside MonitorRegistry in StdExtension,
with CtxWatching and RuntimeWatching extension traits for the public API.
Phase 5b removed from tick_once; notifications delivered via on_actor_death
in phase 7. Core shrinks from 2,636 to 2,440 lines (-196, -7.4%).

Authored by Claude, lovingly guided by Zachery Aaron Shores-Chmielewski
2026-02-13 21:45:44 +07:00
1169f0322d feat: dashboard updates (#38)
Adds some more polish to the dashboard. Further modification will have to wait until swactor is used for more complex projects.
2026-02-13 14:42:10 +00:00
34 changed files with 4997 additions and 4831 deletions

View file

@ -0,0 +1,403 @@
pub const ACTOR_DETAIL_HTML: &str = r##"<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Actor Detail — Swactor Dashboard</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: 'Menlo', 'Consolas', 'Monaco', monospace; background: #0f1117; color: #e0e0e0; font-size: 13px; }
.header {
display: flex; align-items: center; justify-content: space-between;
padding: 12px 20px; background: #161822; border-bottom: 1px solid #2a2d3e;
}
.header h1 { font-size: 16px; font-weight: 600; color: #fff; }
.status-dot {
width: 10px; height: 10px; border-radius: 50%; background: #4caf50;
display: inline-block; margin-left: 8px; vertical-align: middle;
}
.status-dot.disconnected { background: #f44336; }
.header-left { display: flex; align-items: center; }
.nav-links { display: flex; gap: 4px; margin-left: 20px; }
.nav-link {
color: #888; text-decoration: none; font-size: 12px;
padding: 4px 10px; border-radius: 3px; transition: color 0.2s;
}
.nav-link:hover { color: #e0e0e0; }
.nav-link.active { color: #fff; background: #2a2d3e; }
.content { padding: 16px 20px; max-width: 900px; }
.breadcrumb { color: #555; font-size: 12px; margin-bottom: 12px; }
.breadcrumb a { color: #888; text-decoration: none; }
.breadcrumb a:hover { color: #e0e0e0; }
.info-card {
background: #161822; border: 1px solid #2a2d3e; border-radius: 6px;
padding: 16px; margin-bottom: 12px;
}
.info-row { display: flex; gap: 24px; margin-bottom: 6px; flex-wrap: wrap; }
.info-label { color: #888; font-size: 11px; text-transform: uppercase; }
.info-value { color: #fff; font-weight: 700; font-size: 15px; }
.info-value.healthy { color: #4caf50; }
.info-value.poisoned { color: #f44336; }
.stats-cards {
display: grid; grid-template-columns: repeat(4, 1fr); gap: 10px;
margin-bottom: 12px;
}
.stat-card {
background: #161822; border: 1px solid #2a2d3e; border-radius: 4px;
padding: 12px; text-align: center;
}
.stat-card .value { font-size: 22px; font-weight: 700; color: #fff; }
.stat-card .label { font-size: 10px; color: #888; text-transform: uppercase; margin-top: 2px; }
.sparkline-panel {
background: #161822; border: 1px solid #2a2d3e; border-radius: 6px;
padding: 14px; margin-bottom: 12px;
}
.sparkline-panel h3 { font-size: 11px; color: #888; text-transform: uppercase; letter-spacing: 1px; margin-bottom: 8px; }
.sparkline-panel svg { width: 100%; height: 50px; }
.type-breakdown {
background: #161822; border: 1px solid #2a2d3e; border-radius: 6px;
padding: 14px; margin-bottom: 12px;
}
.type-breakdown h3 { font-size: 11px; color: #888; text-transform: uppercase; letter-spacing: 1px; margin-bottom: 8px; }
.type-row { display: flex; align-items: center; gap: 8px; margin-bottom: 4px; font-size: 11px; }
.type-name { color: #e0e0e0; min-width: 180px; text-overflow: ellipsis; overflow: hidden; white-space: nowrap; }
.type-bar-bg { flex: 1; background: #1e2030; height: 14px; border-radius: 2px; overflow: hidden; }
.type-bar-fill { height: 100%; border-radius: 2px; }
.type-count { color: #888; min-width: 60px; text-align: right; }
.type-pct { color: #555; min-width: 40px; text-align: right; }
.logs-panel {
background: #161822; border: 1px solid #2a2d3e; border-radius: 6px;
padding: 14px; margin-bottom: 12px;
}
.logs-panel h3 {
font-size: 11px; color: #888; text-transform: uppercase; letter-spacing: 1px;
margin-bottom: 8px; display: flex; align-items: center; gap: 12px;
}
.level-filter { display: flex; gap: 4px; }
.level-btn {
background: #1e2030; border: 1px solid #2a2d3e; border-radius: 3px;
color: #888; font-size: 10px; padding: 1px 6px; cursor: pointer;
font-family: inherit;
}
.level-btn.active { border-color: #555; color: #fff; }
.level-btn.error { color: #f44336; }
.level-btn.warn { color: #ff9800; }
.level-btn.info { color: #2196f3; }
.level-btn.debug { color: #888; }
.level-btn.trace { color: #555; }
.log-list {
max-height: 400px; overflow-y: auto; font-size: 11px; line-height: 1.6;
}
.log-entry { display: flex; gap: 8px; padding: 1px 0; border-bottom: 1px solid #1a1c2e; }
.log-time { color: #555; white-space: nowrap; min-width: 80px; }
.log-level { font-weight: 700; min-width: 50px; }
.log-level.ERROR { color: #f44336; }
.log-level.WARN { color: #ff9800; }
.log-level.INFO { color: #2196f3; }
.log-level.DEBUG { color: #888; }
.log-level.TRACE { color: #555; }
.log-msg { color: #e0e0e0; word-break: break-all; }
::-webkit-scrollbar { width: 6px; }
::-webkit-scrollbar-track { background: #0f1117; }
::-webkit-scrollbar-thumb { background: #2a2d3e; border-radius: 3px; }
</style>
</head>
<body>
<div class="header">
<div class="header-left">
<h1>
Swactor Runtime Dashboard
<span id="statusDot" class="status-dot"></span>
</h1>
<nav class="nav-links">
<a href="/" class="nav-link">Overview</a>
<a href="/actors" class="nav-link">Actors</a>
</nav>
</div>
</div>
<div class="content">
<div class="breadcrumb">
<a href="/">Overview</a> / <a href="/actors">Actors</a> / <span id="addrBreadcrumb">—</span>
</div>
<div class="info-card">
<div class="info-row">
<div><div class="info-label">Address</div><div class="info-value" id="addrFull">—</div></div>
<div><div class="info-label">Worker</div><div class="info-value" id="addrWorker">—</div></div>
<div><div class="info-label">Status</div><div class="info-value" id="addrStatus">—</div></div>
</div>
<div class="info-row">
<div><div class="info-label">Last Message Type</div><div class="info-value" id="addrLastMsg" style="color:#4caf50;font-size:13px;">—</div></div>
</div>
</div>
<div class="stats-cards">
<div class="stat-card"><div class="value" id="addrMsgs">0</div><div class="label">Messages</div></div>
<div class="stat-card"><div class="value" id="addrMailbox">0</div><div class="label">Mailbox</div></div>
<div class="stat-card"><div class="value" id="addrRate">0</div><div class="label">Msg/s</div></div>
<div class="stat-card"><div class="value" id="addrWorkerLoad">—</div><div class="label">Worker Load</div></div>
</div>
<div class="sparkline-panel">
<h3>Message Rate</h3>
<svg id="rateSpark" viewBox="0 0 400 50" preserveAspectRatio="none"></svg>
</div>
<div class="sparkline-panel">
<h3>Mailbox Depth</h3>
<svg id="mboxSpark" viewBox="0 0 400 50" preserveAspectRatio="none"></svg>
</div>
<div class="type-breakdown" id="typeBreakdown" style="display:none;">
<h3>Message Types</h3>
<div id="typeRows"></div>
</div>
<div class="logs-panel">
<h3>
<span>Logs</span>
<span id="logCount" style="color:#555;">(0)</span>
<div class="level-filter">
<button class="level-btn error active" data-level="ERROR" onclick="toggleLevel(this)">ERR</button>
<button class="level-btn warn active" data-level="WARN" onclick="toggleLevel(this)">WARN</button>
<button class="level-btn info active" data-level="INFO" onclick="toggleLevel(this)">INFO</button>
<button class="level-btn debug active" data-level="DEBUG" onclick="toggleLevel(this)">DBG</button>
<button class="level-btn trace active" data-level="TRACE" onclick="toggleLevel(this)">TRC</button>
</div>
</h3>
<div class="log-list" id="logList"></div>
</div>
</div>
<script>
(function() {
var targetAddr = '__ACTOR_ADDR__';
var dot = document.getElementById('statusDot');
var history = { rates: [], mailbox: [], prev_msgs: 0 };
function formatAddr(addr) {
if (!addr) return '';
var bytes = Array.isArray(addr) ? addr : Object.values(addr);
var hex = '';
for (var i = 0; i < bytes.length; i++) {
hex += ('0' + bytes[i].toString(16)).slice(-2);
}
return hex;
}
function shortAddr(hex) {
return hex.length > 16 ? hex.substring(0, 16) + '\u2026' : hex;
}
function shortTypeName(full) {
if (!full) return '\u2014';
var parts = full.split('::');
return parts[parts.length - 1];
}
function updateSparklineSvg(svgEl, data, color) {
if (!data || data.length < 2) { svgEl.innerHTML = ''; return; }
var max = Math.max.apply(null, data);
if (max === 0) max = 1;
var w = 400, h = 50;
var step = w / (data.length - 1);
var points = data.map(function(v, i) {
return (i * step).toFixed(1) + ',' + (h - (v / max) * (h - 4) - 2).toFixed(1);
}).join(' ');
svgEl.innerHTML = '<polyline fill="none" stroke="' + color + '" stroke-width="2" points="' + points + '"/>';
}
function findActor(stats) {
if (!stats.actor_details) return null;
for (var i = 0; i < stats.actor_details.length; i++) {
var a = stats.actor_details[i];
var hex = formatAddr(a.address);
if (hex === targetAddr || hex.indexOf(targetAddr) === 0) return a;
}
return null;
}
function updateDetail(stats) {
var actor = findActor(stats);
if (!actor) return;
var hex = formatAddr(actor.address);
document.getElementById('addrBreadcrumb').textContent = shortAddr(hex);
document.getElementById('addrFull').textContent = hex;
document.getElementById('addrWorker').textContent = 'W' + actor.worker_id;
var statusEl = document.getElementById('addrStatus');
if (actor.poisoned) {
statusEl.textContent = 'POISONED';
statusEl.className = 'info-value poisoned';
} else {
statusEl.textContent = 'Healthy';
statusEl.className = 'info-value healthy';
}
document.getElementById('addrLastMsg').textContent = shortTypeName(actor.last_msg_type);
document.getElementById('addrMsgs').textContent = actor.messages_processed.toLocaleString();
document.getElementById('addrMailbox').textContent = actor.mailbox_depth;
// Compute rate
var rate = actor.messages_processed - history.prev_msgs;
if (rate < 0) rate = 0;
history.prev_msgs = actor.messages_processed;
history.rates.push(rate);
history.mailbox.push(actor.mailbox_depth);
if (history.rates.length > 300) { history.rates.shift(); history.mailbox.shift(); }
// Rate per second (SSE interval is ~200ms, so multiply by 5)
document.getElementById('addrRate').textContent = (rate * 5).toLocaleString();
// Worker load
if (stats.workers) {
var w = stats.workers.find(function(w) { return w.id === actor.worker_id; });
if (w) {
document.getElementById('addrWorkerLoad').textContent =
w.num_actors + ' actors, mbox ' + w.mailbox_depth;
}
}
updateSparklineSvg(document.getElementById('rateSpark'), history.rates, '#4caf50');
updateSparklineSvg(document.getElementById('mboxSpark'), history.mailbox, '#2196f3');
// Update message type breakdown
updateTypeBreakdown(actor.message_type_counts);
}
var typeColors = ['#4caf50','#2196f3','#ff9800','#9c27b0','#00bcd4','#f44336','#ffeb3b','#e91e63'];
function updateTypeBreakdown(types) {
var panel = document.getElementById('typeBreakdown');
var container = document.getElementById('typeRows');
if (!types || types.length === 0) { panel.style.display = 'none'; return; }
panel.style.display = '';
var total = 0;
for (var i = 0; i < types.length; i++) total += types[i][1];
if (total === 0) { panel.style.display = 'none'; return; }
var html = '';
for (var i = 0; i < types.length; i++) {
var name = types[i][0];
var count = types[i][1];
var pct = (count / total * 100).toFixed(1);
var barPct = (count / types[0][1] * 100).toFixed(1);
var color = typeColors[i % typeColors.length];
var shortName = name.split('::').pop();
html += '<div class="type-row">' +
'<span class="type-name" title="' + escapeHtml(name) + '">' + escapeHtml(shortName) + '</span>' +
'<div class="type-bar-bg"><div class="type-bar-fill" style="width:' + barPct + '%;background:' + color + ';"></div></div>' +
'<span class="type-count">' + count.toLocaleString() + '</span>' +
'<span class="type-pct">' + pct + '%</span>' +
'</div>';
}
container.innerHTML = html;
}
// ─── Logging ─────────────────────────────────────────────────
var activeLevels = { ERROR: true, WARN: true, INFO: true, DEBUG: true, TRACE: true };
var allLogs = [];
var logList = document.getElementById('logList');
var logCount = document.getElementById('logCount');
var autoScroll = true;
window.toggleLevel = function(btn) {
var lvl = btn.getAttribute('data-level');
activeLevels[lvl] = !activeLevels[lvl];
btn.classList.toggle('active');
renderLogs();
};
function formatLogTime(ms) {
var d = new Date(ms);
return ('0' + d.getHours()).slice(-2) + ':' +
('0' + d.getMinutes()).slice(-2) + ':' +
('0' + d.getSeconds()).slice(-2) + '.' +
('00' + d.getMilliseconds()).slice(-3);
}
function renderLogs() {
var visible = allLogs.filter(function(e) { return activeLevels[e.level]; });
logCount.textContent = '(' + visible.length + ')';
var html = '';
for (var i = 0; i < visible.length; i++) {
var e = visible[i];
html += '<div class="log-entry">' +
'<span class="log-time">' + formatLogTime(e.timestamp_ms) + '</span>' +
'<span class="log-level ' + e.level + '">' + e.level + '</span>' +
'<span class="log-msg">' + escapeHtml(e.message) + '</span>' +
'</div>';
}
logList.innerHTML = html;
if (autoScroll) {
logList.scrollTop = logList.scrollHeight;
}
}
function escapeHtml(s) {
if (!s) return '';
return s.replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;');
}
function addLogEntries(events) {
for (var i = 0; i < events.length; i++) {
var e = events[i];
if (!e.actor_addr) continue;
// Match if event's actor_addr starts with or contains target
var a = e.actor_addr.toLowerCase();
if (a.indexOf(targetAddr.toLowerCase()) === 0 || a === targetAddr.toLowerCase()) {
allLogs.push(e);
}
}
// Keep bounded
while (allLogs.length > 500) allLogs.shift();
renderLogs();
}
// Fetch initial logs
fetch('/api/logs?actor=' + targetAddr + '&limit=200')
.then(function(r) { return r.json(); })
.then(function(data) { if (Array.isArray(data)) addLogEntries(data); })
.catch(function() {});
logList.addEventListener('scroll', function() {
autoScroll = (logList.scrollTop + logList.clientHeight >= logList.scrollHeight - 20);
});
// ─── SSE ────────────────────────────────────────────────────
var es = new EventSource('/events');
es.addEventListener('stats', function(e) {
try { updateDetail(JSON.parse(e.data)); } catch(err) { console.error(err); }
});
es.addEventListener('activity', function(e) {
try { addLogEntries(JSON.parse(e.data)); } catch(err) { console.error(err); }
});
es.addEventListener('done', function() {
dot.className = 'status-dot disconnected';
es.close();
});
es.onerror = function() { dot.className = 'status-dot disconnected'; };
es.onopen = function() { dot.className = 'status-dot'; };
})();
</script>
</body>
</html>
"##;

View file

@ -60,7 +60,7 @@
.panel {
background: #161822; border: 1px solid #2a2d3e; border-radius: 6px;
padding: 14px; overflow: hidden;
padding: 14px; overflow: visible;
}
.panel h2 { font-size: 12px; color: #888; text-transform: uppercase; letter-spacing: 1px; margin-bottom: 10px; }
@ -75,7 +75,7 @@
.stat-card .value { font-size: 22px; font-weight: 700; color: #fff; }
.stat-card .label { font-size: 10px; color: #888; text-transform: uppercase; margin-top: 2px; }
canvas { width: 100%; height: 200px; }
canvas { width: 100%; height: 220px; }
.search-wrap { margin-bottom: 10px; display: flex; align-items: center; gap: 12px; }
.search-input {
@ -109,7 +109,12 @@
tr.clickable { cursor: pointer; }
tr.clickable:hover { background: #1a1d2c; }
.detail-panel { display: none; }
.detail-panel {
display: none; position: fixed; bottom: 12px; right: 12px;
width: 420px; max-height: 320px; overflow-y: auto;
background: #161822; border: 1px solid #2a2d3e; border-radius: 6px;
padding: 14px; z-index: 100; box-shadow: 0 4px 24px rgba(0,0,0,0.5);
}
.detail-panel.visible { display: block; }
.detail-header { display: flex; justify-content: space-between; align-items: center; margin-bottom: 10px; }
.detail-close {
@ -118,17 +123,17 @@
}
.detail-close:hover { color: #e0e0e0; border-color: #555; }
.detail-grid {
display: grid; grid-template-columns: repeat(3, 1fr); gap: 10px; margin-bottom: 12px;
display: grid; grid-template-columns: repeat(2, 1fr); gap: 8px; margin-bottom: 10px;
}
.detail-item { background: #1c1f2e; border-radius: 4px; padding: 8px 10px; }
.detail-item { background: #1c1f2e; border-radius: 4px; padding: 6px 8px; }
.detail-item .d-label { font-size: 10px; color: #888; text-transform: uppercase; }
.detail-item .d-value { font-size: 14px; font-weight: 700; color: #fff; margin-top: 2px; word-break: break-all; }
.detail-item .d-value { font-size: 13px; font-weight: 700; color: #fff; margin-top: 2px; word-break: break-all; }
.poisoned-badge {
background: #f44336; color: #fff; font-size: 10px; font-weight: 700;
padding: 2px 6px; border-radius: 3px; letter-spacing: 0.5px;
}
.healthy-badge { color: #4caf50; font-size: 12px; }
canvas.sparkline { width: 100%; height: 60px; }
canvas.sparkline { width: 100%; height: 50px; }
::-webkit-scrollbar { width: 6px; }
::-webkit-scrollbar-track { background: #0f1117; }
@ -187,7 +192,16 @@
<div class="panel full-width">
<h2>All Actors <span id="actorCount" style="color:#555;font-weight:400;"></span></h2>
<div class="search-wrap">
<input type="text" id="actorSearch" class="search-input" placeholder="Filter by address or worker..." />
<input type="text" id="actorSearch" class="search-input" placeholder="Filter by address, type, or worker..." />
<select id="workerFilter" class="search-input" style="width:120px;">
<option value="">All Workers</option>
</select>
<select id="statusFilter" class="search-input" style="width:120px;">
<option value="">All Status</option>
<option value="healthy">Healthy</option>
<option value="poisoned">Poisoned</option>
</select>
<input type="number" id="minDepth" class="search-input" style="width:100px;" placeholder="Min depth" min="0" />
<span id="searchInfo" class="search-info"></span>
</div>
<div class="actor-list-wrap">
@ -207,23 +221,25 @@
</div>
</div>
<!-- Actor detail panel -->
<div id="detailPanel" class="panel full-width detail-panel">
<div class="detail-header">
<h2>Actor Detail <span id="detailAddr" style="color:#aaa;font-weight:400;"></span></h2>
<button class="detail-close" id="detailClose">Close</button>
</div>
<div class="detail-grid">
<div class="detail-item"><div class="d-label">Full Address</div><div class="d-value" id="detailFullAddr" style="font-size:11px;"></div></div>
<div class="detail-item"><div class="d-label">Worker</div><div class="d-value" id="detailWorker"></div></div>
<div class="detail-item"><div class="d-label">Mailbox Depth</div><div class="d-value" id="detailMailbox"></div></div>
<div class="detail-item"><div class="d-label">Messages Processed</div><div class="d-value" id="detailMsgCount"></div></div>
<div class="detail-item"><div class="d-label">Last Message Type</div><div class="d-value" id="detailLastMsg"></div></div>
<div class="detail-item"><div class="d-label">Status</div><div class="d-value" id="detailStatus"></div></div>
</div>
<h2>Mailbox Depth History</h2>
<canvas id="sparkline" class="sparkline"></canvas>
</div>
<!-- Actor detail overlay (fixed position, doesn't affect grid layout) -->
<div id="detailPanel" class="detail-panel">
<div class="detail-header">
<h2 style="font-size:12px;color:#888;text-transform:uppercase;letter-spacing:1px;">Actor <span id="detailAddr" style="color:#aaa;font-weight:400;"></span></h2>
<button class="detail-close" id="detailClose">&times;</button>
</div>
<div class="detail-grid">
<div class="detail-item"><div class="d-label">Address</div><div class="d-value" id="detailFullAddr" style="font-size:10px;"></div></div>
<div class="detail-item"><div class="d-label">Worker</div><div class="d-value" id="detailWorker"></div></div>
<div class="detail-item"><div class="d-label">Mailbox</div><div class="d-value" id="detailMailbox"></div></div>
<div class="detail-item"><div class="d-label">Messages</div><div class="d-value" id="detailMsgCount"></div></div>
<div class="detail-item"><div class="d-label">Last Type</div><div class="d-value" id="detailLastMsg"></div></div>
<div class="detail-item"><div class="d-label">Status</div><div class="d-value" id="detailStatus"></div></div>
</div>
<div style="font-size:10px;color:#888;text-transform:uppercase;letter-spacing:1px;margin-bottom:4px;">Mailbox History</div>
<canvas id="sparkline" class="sparkline"></canvas>
</div>
<script>
@ -374,11 +390,12 @@
var maxCount = Math.max(1, Math.max.apply(null, counts));
var barW = Math.max(12, Math.floor((W - 40) / buckets.length) - 8);
var topPad = 20;
var chartH = H - 35;
for (var i = 0; i < buckets.length; i++) {
var x = 20 + i * (barW + 8);
var h = (counts[i] / maxCount) * (chartH - 10);
var h = (counts[i] / maxCount) * (chartH - topPad);
depthCtx.fillStyle = bucketColors[i];
depthCtx.globalAlpha = 0.85;
depthCtx.fillRect(x, chartH - h, barW, h);
@ -424,11 +441,12 @@
var maxCount = Math.max(1, Math.max.apply(null, entries.map(function(e) { return e.count; })));
var barW = Math.max(12, Math.floor((W - 40) / entries.length) - 8);
var topPad = 20;
var chartH = H - 35;
for (var i = 0; i < entries.length; i++) {
var x = 20 + i * (barW + 8);
var h = (entries[i].count / maxCount) * (chartH - 10);
var h = (entries[i].count / maxCount) * (chartH - topPad);
workerCtx.fillStyle = colors[entries[i].id % colors.length];
workerCtx.globalAlpha = 0.85;
workerCtx.fillRect(x, chartH - h, barW, h);
@ -449,13 +467,29 @@
function renderActorTable() {
var filter = document.getElementById('actorSearch').value.toLowerCase();
var filtered = currentActors;
if (filter) {
filtered = currentActors.filter(function(a) {
var workerFilter = document.getElementById('workerFilter').value;
var statusFilter = document.getElementById('statusFilter').value;
var minDepthVal = document.getElementById('minDepth').value;
var minDepth = minDepthVal ? parseInt(minDepthVal, 10) : 0;
var filtered = currentActors.filter(function(a) {
// Text search
if (filter) {
var hex = addrToHex(a.address).toLowerCase();
return hex.indexOf(filter) >= 0 || ('w' + a.worker_id).indexOf(filter) >= 0;
});
}
var msgType = (a.last_msg_type || '').toLowerCase();
if (hex.indexOf(filter) < 0 && ('w' + a.worker_id).indexOf(filter) < 0 && msgType.indexOf(filter) < 0) {
return false;
}
}
// Worker filter
if (workerFilter && a.worker_id !== parseInt(workerFilter, 10)) return false;
// Status filter
if (statusFilter === 'healthy' && a.poisoned) return false;
if (statusFilter === 'poisoned' && !a.poisoned) return false;
// Min depth
if (minDepth > 0 && a.mailbox_depth < minDepth) return false;
return true;
});
// Sort
filtered.sort(function(a, b) {
@ -531,14 +565,14 @@
// ── Focus / detail panel ───────────────────────────────
function focusActor(fullHex) {
focusedAddrHex = fullHex;
document.getElementById('detailPanel').className = 'panel full-width detail-panel visible';
document.getElementById('detailPanel').className = 'detail-panel visible';
updateDetailPanel();
renderActorTable();
}
function clearFocus() {
focusedAddrHex = null;
document.getElementById('detailPanel').className = 'panel full-width detail-panel';
document.getElementById('detailPanel').className = 'detail-panel';
renderActorTable();
}
@ -558,8 +592,8 @@
return;
}
document.getElementById('detailAddr').textContent = addrToHex(actor.address);
document.getElementById('detailFullAddr').textContent = focusedAddrHex;
document.getElementById('detailAddr').innerHTML = '<a href="/actor/' + focusedAddrHex + '" style="color:#aaa;text-decoration:none;">' + escapeHtml(addrToHex(actor.address)) + '</a>';
document.getElementById('detailFullAddr').innerHTML = '<a href="/actor/' + focusedAddrHex + '" style="color:#fff;text-decoration:none;">' + escapeHtml(focusedAddrHex) + '</a>';
document.getElementById('detailWorker').textContent = 'W' + actor.worker_id;
document.getElementById('detailMailbox').textContent = actor.mailbox_depth;
document.getElementById('detailMsgCount').textContent = (actor.messages_processed || 0).toLocaleString();
@ -647,11 +681,17 @@
});
}
// ── Search handler ─────────────────────────────────────
// ── Search/filter handlers ─────────────────────────────
document.getElementById('actorSearch').addEventListener('keyup', function() {
clearTimeout(searchTimer);
searchTimer = setTimeout(renderActorTable, 150);
});
document.getElementById('workerFilter').addEventListener('change', renderActorTable);
document.getElementById('statusFilter').addEventListener('change', renderActorTable);
document.getElementById('minDepth').addEventListener('input', function() {
clearTimeout(searchTimer);
searchTimer = setTimeout(renderActorTable, 150);
});
// ── Status helpers ─────────────────────────────────────
function setStatus(s) {
@ -697,6 +737,21 @@
if (!liveAddrs[key]) delete depthHistory[key];
}
// Update worker filter dropdown
var wSelect = document.getElementById('workerFilter');
var curVal = wSelect.value;
var workerIds = {};
for (var i = 0; i < currentActors.length; i++) workerIds[currentActors[i].worker_id] = true;
var wids = Object.keys(workerIds).sort(function(a,b) { return +a - +b; });
wSelect.innerHTML = '<option value="">All Workers</option>';
wids.forEach(function(wid) {
var opt = document.createElement('option');
opt.value = wid;
opt.textContent = 'W' + wid;
wSelect.appendChild(opt);
});
wSelect.value = curVal;
renderActorTable();
if (focusedAddrHex) updateDetailPanel();
} catch(err) { console.error('stats parse error', err); }

View file

@ -1 +1 @@
pub const ACTORS_HTML: &str = include_str!("actors.html");
pub const ACTORS_HTML: &str = include_str\!("actors.html");

View file

@ -60,6 +60,7 @@ impl StatsHook for StatsCollector {
last_msg_type: s.last_msg_type.map(|t| t.to_string()),
messages_processed: s.messages_processed,
poisoned: s.poisoned,
message_type_counts: s.message_type_counts.iter().map(|(k, v)| (k.to_string(), *v)).collect(),
}));
}
}

View file

@ -77,6 +77,17 @@
.chart-panel { grid-row: span 2; }
canvas#workerChart { width: 100%; height: 200px; }
.worker-cards { display: flex; flex-direction: column; gap: 6px; }
.worker-card {
display: flex; align-items: center; gap: 10px;
background: #1c1f2e; border-radius: 4px; padding: 6px 10px;
}
.worker-card .wc-id { font-weight: 700; min-width: 32px; }
.worker-card .wc-bar-wrap { flex: 1; height: 14px; background: #0f1117; border-radius: 2px; overflow: hidden; display: flex; }
.worker-card .wc-bar-seg { height: 100%; }
.worker-card .wc-stats { font-size: 11px; color: #888; min-width: 200px; text-align: right; }
.worker-card .wc-spark { display: inline-flex; gap: 4px; margin-left: 6px; }
.actor-table-wrap { max-height: 200px; overflow-y: auto; }
.actor-table-wrap table { width: 100%; border-collapse: collapse; }
.actor-table-wrap th, .actor-table-wrap td {
@ -113,6 +124,8 @@
.worker-group-header .wid { font-weight: 700; }
.worker-group-header .summary { color: #888; font-size: 11px; }
.worker-group-header .toggle { color: #555; font-size: 14px; }
.worker-group-header .sparkline-wrap { display: inline-flex; gap: 8px; margin-left: 12px; }
.worker-group-header .sparkline-wrap svg { vertical-align: middle; }
.worker-group-body { display: none; }
.worker-group.open .worker-group-body { display: block; }
.worker-group-body table { width: 100%; border-collapse: collapse; }
@ -152,11 +165,17 @@
<div id="progressFill" class="progress-fill"></div>
</div>
<div id="warningBanner" style="display:none;padding:8px 20px;background:#1c1f2e;border-bottom:1px solid #2a2d3e;font-size:12px;"></div>
<div class="grid">
<div class="panel chart-panel">
<h2>Worker Distribution</h2>
<canvas id="workerChart"></canvas>
<div id="workerLegend" style="margin-top:8px;font-size:11px;color:#888;"></div>
<h2>Worker Utilization</h2>
<div id="workerCards" class="worker-cards"></div>
<div style="margin-top:6px;font-size:10px;color:#555;">
<span style="color:#4caf50;">\u25A0</span> processing
<span style="color:#2196f3;">\u25A0</span> delivery
<span style="color:#00bcd4;">\u25A0</span> spawns
<span style="color:#f44336;">\u25A0</span> overhead
</div>
</div>
<div class="panel">
@ -203,6 +222,7 @@
var isReplay = (DASHBOARD_MODE === 'replay');
var lastUptimeMs = null;
var lastStatsTime = null;
var workerHistory = {}; // { id: { message_rates: [], mailbox_depths: [] } }
var dot = document.getElementById('statusDot');
var uptimeLabel = document.getElementById('uptimeLabel');
@ -240,48 +260,83 @@
}
setInterval(updateUptime, 1000);
var canvas = document.getElementById('workerChart');
var ctx = canvas.getContext('2d');
var colors = ['#4caf50','#2196f3','#ff9800','#f44336','#9c27b0','#00bcd4','#ffeb3b','#e91e63'];
function drawWorkerChart(workers) {
var dpr = window.devicePixelRatio || 1;
var rect = canvas.getBoundingClientRect();
canvas.width = rect.width * dpr;
canvas.height = rect.height * dpr;
ctx.scale(dpr, dpr);
var W = rect.width, H = rect.height;
ctx.clearRect(0, 0, W, H);
var phaseColors = ['#4caf50', '#2196f3', '#00bcd4', '#f44336'];
// Group tick phases: processing=2, delivery=1+4, spawns=0+3, overhead=5
if (!workers || workers.length === 0) return;
function computePhases(timings) {
if (!timings || timings.length === 0) return [0.25, 0.25, 0.25, 0.25];
var sums = [0,0,0,0,0,0];
var active = 0;
for (var i = 0; i < timings.length; i++) {
var t = timings[i];
if (t.did_work) active++;
for (var p = 0; p < 6 && p < t.phase_us.length; p++) sums[p] += t.phase_us[p];
}
var total = sums.reduce(function(a,b) { return a+b; }, 0);
if (total === 0) return [0.25, 0.25, 0.25, 0.25];
var processing = sums[2] / total;
var delivery = (sums[1] + sums[4]) / total;
var spawns = (sums[0] + sums[3]) / total;
var overhead = sums[5] / total;
var load = timings.length > 0 ? active / timings.length : 0;
return { fracs: [processing, delivery, spawns, overhead], load: load };
}
var maxActors = Math.max(1, Math.max.apply(null, workers.map(function(w) { return w.num_actors; })));
var barW = Math.max(8, Math.floor((W - 40) / workers.length) - 6);
var chartH = H - 30;
function renderWorkerCards(data) {
var container = document.getElementById('workerCards');
if (!data.workers) return;
container.innerHTML = '';
workers.forEach(function(w, i) {
var x = 20 + i * (barW + 6);
var h = (w.num_actors / maxActors) * (chartH * 0.45);
ctx.fillStyle = colors[i % colors.length];
ctx.globalAlpha = 0.8;
ctx.fillRect(x, chartH * 0.5 - h, barW, h);
data.workers.forEach(function(w, idx) {
var timings = data.tick_timings ? data.tick_timings[idx] : null;
var phases = computePhases(timings);
var load = phases.load || 0;
var fracs = phases.fracs || [0.25, 0.25, 0.25, 0.25];
var mh = Math.min(w.mailbox_depth * 2, chartH * 0.4);
ctx.globalAlpha = 0.4;
ctx.fillRect(x, chartH * 0.55, barW, mh);
var card = document.createElement('div');
card.className = 'worker-card';
ctx.globalAlpha = 1;
ctx.fillStyle = '#888';
ctx.font = '10px monospace';
ctx.textAlign = 'center';
ctx.fillText('W' + w.id, x + barW / 2, H - 2);
// ID
var idSpan = document.createElement('span');
idSpan.className = 'wc-id';
idSpan.style.color = colors[w.id % colors.length];
idSpan.textContent = 'W' + w.id;
card.appendChild(idSpan);
// Phase bar
var barWrap = document.createElement('span');
barWrap.className = 'wc-bar-wrap';
var filledPct = Math.round(load * 100);
for (var p = 0; p < 4; p++) {
var seg = document.createElement('span');
seg.className = 'wc-bar-seg';
seg.style.width = (fracs[p] * filledPct) + '%';
seg.style.background = phaseColors[p];
barWrap.appendChild(seg);
}
card.appendChild(barWrap);
// Sparklines
var sparkWrap = document.createElement('span');
sparkWrap.className = 'wc-spark';
var wh = workerHistory[w.id];
if (wh) {
sparkWrap.innerHTML = renderSparklineSvg(wh.message_rates, 60, 14, '#4caf50');
}
card.appendChild(sparkWrap);
// Stats
var statsSpan = document.createElement('span');
statsSpan.className = 'wc-stats';
statsSpan.textContent = w.num_actors + ' actors ' +
w.messages_processed.toLocaleString() + ' msgs mbox ' + w.mailbox_depth +
' ' + Math.round(load * 100) + '%';
card.appendChild(statsSpan);
container.appendChild(card);
});
var legend = document.getElementById('workerLegend');
legend.innerHTML = workers.map(function(w, i) {
return '<span style="color:' + colors[i % colors.length] + '">W' + w.id +
': ' + w.num_actors + ' actors, ' + w.messages_processed + ' msgs, mbox ' + w.mailbox_depth + '</span>';
}).join(' &nbsp;|&nbsp; ');
}
function updateStats(data) {
@ -298,7 +353,7 @@
document.getElementById('statWorkers').textContent = data.num_workers || 0;
document.getElementById('statMailbox').textContent = totalMailbox;
drawWorkerChart(data.workers);
renderWorkerCards(data);
var tbody = document.getElementById('actorTableBody');
tbody.innerHTML = '';
@ -316,7 +371,7 @@
hex += '\u2026';
}
var tr = document.createElement('tr');
tr.innerHTML = '<td style="color:#aaa;font-size:11px;">' + hex + '</td><td>W' + wid + '</td>';
tr.innerHTML = '<td style="color:#aaa;font-size:11px;"><a href="/actor/' + hex + '" style="color:#aaa;text-decoration:none;">' + hex + '</a></td><td>W' + wid + '</td>';
tbody.appendChild(tr);
});
if (data.actors.length > 200) {
@ -347,6 +402,34 @@
return parts[parts.length - 1];
}
function renderSparklineSvg(data, w, h, color) {
if (!data || data.length < 2) return '';
var max = Math.max.apply(null, data);
if (max === 0) max = 1;
var step = w / (data.length - 1);
var points = data.map(function(v, i) {
return (i * step).toFixed(1) + ',' + (h - (v / max) * (h - 2) - 1).toFixed(1);
}).join(' ');
return '<svg width="' + w + '" height="' + h + '" style="vertical-align:middle">' +
'<polyline fill="none" stroke="' + color + '" stroke-width="1.5" points="' + points + '"/></svg>';
}
function pushHistorySample(stats) {
if (!stats.workers) return;
stats.workers.forEach(function(w) {
if (!workerHistory[w.id]) {
workerHistory[w.id] = { message_rates: [], mailbox_depths: [], prev_msgs: w.messages_processed };
}
var wh = workerHistory[w.id];
var rate = w.messages_processed - wh.prev_msgs;
if (rate < 0) rate = 0;
wh.prev_msgs = w.messages_processed;
wh.message_rates.push(rate);
wh.mailbox_depths.push(w.mailbox_depth);
if (wh.message_rates.length > 300) { wh.message_rates.shift(); wh.mailbox_depths.shift(); }
});
}
function updateWorkerDetails(data) {
var container = document.getElementById('workerDetailContainer');
if (!data.workers) return;
@ -377,8 +460,17 @@
var hdr = document.createElement('div');
hdr.className = 'worker-group-header';
var panicHtml = g.info.panics > 0 ? ', <span style="color:#f44336">' + g.info.panics + ' panics</span>' : '';
var wh = workerHistory[wid];
var sparkHtml = '';
if (wh) {
sparkHtml = '<span class="sparkline-wrap">' +
renderSparklineSvg(wh.message_rates, 80, 16, '#4caf50') +
renderSparklineSvg(wh.mailbox_depths, 80, 16, '#2196f3') +
'</span>';
}
hdr.innerHTML =
'<span class="wid" style="color:' + colors[wid % colors.length] + '">W' + wid + '</span>' +
sparkHtml +
'<span class="summary">' + g.actors.length + ' actors, ' +
g.info.messages_processed.toLocaleString() + ' msgs, mbox ' + g.info.mailbox_depth + panicHtml + '</span>' +
'<span class="toggle">' + (isOpen ? '\u25BC' : '\u25B6') + '</span>';
@ -400,7 +492,7 @@
var msgShort = hasMsg ? shortTypeName(a.last_msg_type) : 'none';
var msgClass = hasMsg ? 'msg-type' : 'msg-type none';
var title = hasMsg ? ' title="' + escapeHtml(a.last_msg_type) + '"' : '';
rows += '<tr><td style="color:#aaa;">' + hex + '</td>' +
rows += '<tr><td style="color:#aaa;"><a href="/actor/' + hex + '" style="color:#aaa;text-decoration:none;">' + hex + '</a></td>' +
'<td>' + a.mailbox_depth + '</td>' +
'<td class="' + msgClass + '"' + title + '>' + escapeHtml(msgShort) + '</td></tr>';
});
@ -467,7 +559,44 @@
var es = new EventSource('/events');
es.addEventListener('stats', function(e) {
try { updateStats(JSON.parse(e.data)); } catch(err) { console.error('stats parse error', err); }
try {
var data = JSON.parse(e.data);
pushHistorySample(data);
updateStats(data);
} catch(err) { console.error('stats parse error', err); }
});
es.addEventListener('history', function(e) {
try {
var data = JSON.parse(e.data);
if (data.workers) {
data.workers.forEach(function(w) {
workerHistory[w.id] = {
message_rates: w.message_rates || [],
mailbox_depths: w.mailbox_depths || [],
prev_msgs: 0
};
});
}
} catch(err) { console.error('history parse error', err); }
});
es.addEventListener('warnings', function(e) {
try {
var warnings = JSON.parse(e.data);
var banner = document.getElementById('warningBanner');
if (warnings.length === 0) {
banner.style.display = 'none';
return;
}
banner.style.display = 'block';
var sevColors = {critical:'#f44336',high:'#ff5722',medium:'#ff9800',low:'#888'};
var html = warnings.map(function(w) {
var c = sevColors[w.severity] || '#888';
return '<span style="color:' + c + ';">\u26A0 ' + w.description + '</span>';
}).join(' &nbsp; ');
banner.innerHTML = '<span style="color:#ff9800;font-weight:700;">WARNINGS (' + warnings.length + ')</span> &nbsp; ' + html;
} catch(err) { console.error('warnings parse error', err); }
});
es.addEventListener('activity', function(e) {

View file

@ -1 +1 @@
pub const DASHBOARD_HTML: &str = include_str!("dashboard.html");
pub const DASHBOARD_HTML: &str = include_str\!("dashboard.html");

View file

@ -40,9 +40,9 @@
.graph-panel {
grid-row: 1 / 3; border-right: 1px solid #2a2d3e; position: relative;
min-height: 0;
min-height: 0; overflow: hidden;
}
.graph-panel canvas { width: 100%; height: 100%; display: block; }
.graph-panel canvas { position: absolute; top: 0; left: 0; width: 100%; height: 100%; display: block; }
.side-panel { display: flex; flex-direction: column; overflow: hidden; min-height: 0; }
@ -176,8 +176,17 @@
</div>
</div>
<div class="bottom-section">
<h2>Recent Probes</h2>
<div class="scroll-wrap" id="probesWrap"></div>
<h2>Recent Probes <span id="probeCount" style="color:#555;font-weight:400;"></span></h2>
<div class="scroll-wrap">
<table style="width:100%;border-collapse:collapse;">
<thead><tr>
<th style="color:#888;font-weight:500;font-size:11px;">State</th>
<th style="color:#888;font-weight:500;font-size:11px;">Node</th>
<th style="color:#888;font-weight:500;font-size:11px;">Address</th>
</tr></thead>
<tbody id="probesBody"></tbody>
</table>
</div>
</div>
<div class="bottom-section">
<h2>Routing Buckets</h2>
@ -653,14 +662,26 @@
cacheBody.appendChild(tr);
}
// Recent probes
var probesWrap = document.getElementById('probesWrap');
probesWrap.innerHTML = '';
// Recent probes — cross-reference with members for state + address
var memberMap = {};
for (var i = 0; i < d.members.length; i++) {
memberMap[d.members[i].node_id] = d.members[i];
}
var probesBody = document.getElementById('probesBody');
probesBody.innerHTML = '';
document.getElementById('probeCount').textContent = '(' + d.recent_probe_targets.length + ')';
for (var i = d.recent_probe_targets.length - 1; i >= 0; i--) {
var div = document.createElement('div');
div.style.cssText = 'padding:2px 0;color:#aaa;font-size:11px;border-bottom:1px solid #1c1f2e;';
div.textContent = d.recent_probe_targets[i].substring(0, 16) + '\u2026';
probesWrap.appendChild(div);
var pid = d.recent_probe_targets[i];
var mem = memberMap[pid];
var state = mem ? mem.state : 'unknown';
var addr = mem ? mem.addr : '\u2014';
var cls = 'state-' + state;
var tr = document.createElement('tr');
tr.innerHTML =
'<td class="' + cls + '" style="font-size:10px;">' + state + '</td>' +
'<td style="color:#aaa;font-size:10px;">' + pid.substring(0, 12) + '\u2026</td>' +
'<td style="font-size:10px;">' + addr + '</td>';
probesBody.appendChild(tr);
}
// Routing bucket histogram

View file

@ -1 +1 @@
pub const DISTRIBUTION_HTML: &str = include_str!("distribution.html");
pub const DISTRIBUTION_HTML: &str = include_str\!("distribution.html");

View file

@ -0,0 +1,344 @@
//! In-process time-series history for dashboard sparklines and trend detection.
//!
//! Stores bounded ring buffers of per-worker and per-actor stats, sampled at
//! a configurable interval. All data is kept in memory with automatic eviction
//! of the oldest samples when capacity is reached.
use std::collections::{HashMap, VecDeque};
use std::sync::RwLock;
use swactor::actor::ActorAddress;
use swactor::stats::{ActorInfo, RuntimeStats};
/// Configuration for history collection.
#[derive(Debug, Clone)]
pub struct HistoryConfig {
/// Maximum samples per worker (default: 300 = 5 min at 1/sec).
pub max_worker_samples: usize,
/// Maximum samples per actor (default: 300).
pub max_actor_samples: usize,
/// Maximum number of actors tracked (LRU eviction). Default: 1000.
pub max_tracked_actors: usize,
}
impl Default for HistoryConfig {
fn default() -> Self {
Self {
max_worker_samples: 300,
max_actor_samples: 300,
max_tracked_actors: 1000,
}
}
}
/// Time-series data for a single worker.
#[derive(Debug, Clone)]
pub struct WorkerHistory {
pub message_rates: VecDeque<f64>,
pub mailbox_depths: VecDeque<u64>,
pub actor_counts: VecDeque<u32>,
prev_messages: u64,
}
impl WorkerHistory {
fn new() -> Self {
Self {
message_rates: VecDeque::new(),
mailbox_depths: VecDeque::new(),
actor_counts: VecDeque::new(),
prev_messages: 0,
}
}
fn push(&mut self, messages_processed: u64, mailbox_depth: usize, num_actors: usize, cap: usize) {
let rate = messages_processed.saturating_sub(self.prev_messages) as f64;
self.prev_messages = messages_processed;
push_bounded(&mut self.message_rates, rate, cap);
push_bounded(&mut self.mailbox_depths, mailbox_depth as u64, cap);
push_bounded(&mut self.actor_counts, num_actors as u32, cap);
}
}
/// Time-series data for a single actor.
#[derive(Debug, Clone)]
pub struct ActorHistory {
pub mailbox_depths: VecDeque<u64>,
pub message_rates: VecDeque<f64>,
prev_messages: u64,
last_seen_sample: u64,
}
impl ActorHistory {
fn new(sample_counter: u64) -> Self {
Self {
mailbox_depths: VecDeque::new(),
message_rates: VecDeque::new(),
prev_messages: 0,
last_seen_sample: sample_counter,
}
}
fn push(&mut self, info: &ActorInfo, cap: usize, sample_counter: u64) {
let rate = info.messages_processed.saturating_sub(self.prev_messages) as f64;
self.prev_messages = info.messages_processed;
self.last_seen_sample = sample_counter;
push_bounded(&mut self.mailbox_depths, info.mailbox_depth as u64, cap);
push_bounded(&mut self.message_rates, rate, cap);
}
}
fn push_bounded<T>(buf: &mut VecDeque<T>, val: T, cap: usize) {
if buf.len() >= cap {
buf.pop_front();
}
buf.push_back(val);
}
/// Thread-safe history store. Written by the sampler, read by SSE/TUI.
pub struct DashboardHistory {
inner: RwLock<HistoryInner>,
config: HistoryConfig,
}
struct HistoryInner {
workers: Vec<WorkerHistory>,
actors: HashMap<ActorAddress, ActorHistory>,
sample_counter: u64,
}
impl DashboardHistory {
pub fn new(config: HistoryConfig) -> Self {
Self {
inner: RwLock::new(HistoryInner {
workers: Vec::new(),
actors: HashMap::new(),
sample_counter: 0,
}),
config,
}
}
/// Record a stats snapshot. Called by the sampler thread.
pub fn record(&self, stats: &RuntimeStats) {
let mut inner = self.inner.write().unwrap();
inner.sample_counter += 1;
let counter = inner.sample_counter;
// Resize workers vec if needed
while inner.workers.len() < stats.workers.len() {
inner.workers.push(WorkerHistory::new());
}
// Record per-worker data
for w in &stats.workers {
if let Some(wh) = inner.workers.get_mut(w.id) {
wh.push(
w.messages_processed,
w.mailbox_depth,
w.num_actors,
self.config.max_worker_samples,
);
}
}
// Record per-actor data
for a in &stats.actor_details {
let ah = inner.actors.entry(a.address).or_insert_with(|| ActorHistory::new(counter));
ah.push(a, self.config.max_actor_samples, counter);
}
// LRU eviction: remove actors not seen recently if over capacity
if inner.actors.len() > self.config.max_tracked_actors {
let mut entries: Vec<(ActorAddress, u64)> = inner
.actors
.iter()
.map(|(addr, ah)| (*addr, ah.last_seen_sample))
.collect();
entries.sort_by_key(|&(_, seen)| seen);
let to_remove = inner.actors.len() - self.config.max_tracked_actors;
for (addr, _) in entries.into_iter().take(to_remove) {
inner.actors.remove(&addr);
}
}
}
/// Get a snapshot of worker history for rendering sparklines.
/// Returns Vec indexed by worker_id, each containing recent message rates.
pub fn worker_sparklines(&self) -> Vec<Vec<u64>> {
let inner = self.inner.read().unwrap();
inner
.workers
.iter()
.map(|wh| wh.message_rates.iter().map(|r| *r as u64).collect())
.collect()
}
/// Get worker mailbox depth history.
pub fn worker_mailbox_sparklines(&self) -> Vec<Vec<u64>> {
let inner = self.inner.read().unwrap();
inner
.workers
.iter()
.map(|wh| wh.mailbox_depths.iter().copied().collect())
.collect()
}
/// Get sparkline data for a specific actor.
pub fn actor_sparkline(&self, addr: &ActorAddress) -> Option<(Vec<u64>, Vec<u64>)> {
let inner = self.inner.read().unwrap();
inner.actors.get(addr).map(|ah| {
let mailbox: Vec<u64> = ah.mailbox_depths.iter().copied().collect();
let rates: Vec<u64> = ah.message_rates.iter().map(|r| *r as u64).collect();
(mailbox, rates)
})
}
/// Get total sample count (useful for knowing if history is available).
pub fn sample_count(&self) -> u64 {
self.inner.read().unwrap().sample_counter
}
/// Serialize worker history as JSON for the SSE initial payload.
pub fn worker_history_json(&self) -> String {
let sparklines = self.worker_sparklines();
let mailbox = self.worker_mailbox_sparklines();
serde_json::json!({
"workers": sparklines.iter().enumerate().map(|(i, rates)| {
serde_json::json!({
"id": i,
"message_rates": rates,
"mailbox_depths": mailbox.get(i).unwrap_or(&Vec::new()),
})
}).collect::<Vec<_>>(),
})
.to_string()
}
}
#[cfg(test)]
mod tests {
use super::*;
use swactor::stats::{ActorInfo, WorkerInfo};
fn make_stats(workers: Vec<(u64, usize, usize)>, actors: Vec<ActorInfo>) -> RuntimeStats {
RuntimeStats {
num_workers: workers.len(),
uptime_ms: 0,
actors: actors.iter().map(|a| (a.address, a.worker_id)).collect(),
workers: workers
.into_iter()
.enumerate()
.map(|(id, (msgs, depth, n_actors))| WorkerInfo {
id,
num_actors: n_actors,
mailbox_depth: depth,
messages_processed: msgs,
local_sends: 0,
cross_sends: 0,
inbox_sends: 0,
type_mismatches: 0,
panics: 0,
messages_dropped: 0,
restarts: 0,
stops: 0,
})
.collect(),
actor_details: actors,
tick_timings: Vec::new(),
}
}
fn make_actor(id: u8, worker: usize, depth: usize, msgs: u64) -> ActorInfo {
ActorInfo {
address: ActorAddress([id; 32]),
worker_id: worker,
mailbox_depth: depth,
last_msg_type: None,
messages_processed: msgs,
poisoned: false,
message_type_counts: Vec::new(),
}
}
#[test]
fn worker_rates_accumulate_over_samples() {
let history = DashboardHistory::new(HistoryConfig::default());
// First sample: establishes baseline (rate will be the raw value since prev=0)
let stats1 = make_stats(vec![(100, 5, 2)], vec![]);
history.record(&stats1);
// Second sample: delta = 150 - 100 = 50
let stats2 = make_stats(vec![(150, 3, 2)], vec![]);
history.record(&stats2);
let sparklines = history.worker_sparklines();
assert_eq!(sparklines.len(), 1);
assert_eq!(sparklines[0].len(), 2);
assert_eq!(sparklines[0][0], 100); // first sample: 100 - 0
assert_eq!(sparklines[0][1], 50); // second sample: 150 - 100
}
#[test]
fn bounded_eviction_drops_oldest() {
let config = HistoryConfig {
max_worker_samples: 3,
..Default::default()
};
let history = DashboardHistory::new(config);
for i in 0..5u64 {
let stats = make_stats(vec![(i * 10, 0, 0)], vec![]);
history.record(&stats);
}
let sparklines = history.worker_sparklines();
assert_eq!(sparklines[0].len(), 3); // capped at 3
}
#[test]
fn actor_lru_eviction_keeps_most_recent() {
let config = HistoryConfig {
max_tracked_actors: 2,
..Default::default()
};
let history = DashboardHistory::new(config);
// Sample 1: actors A and B
let stats1 = make_stats(
vec![(0, 0, 2)],
vec![make_actor(1, 0, 0, 0), make_actor(2, 0, 0, 0)],
);
history.record(&stats1);
// Sample 2: actors B and C (A not seen)
let stats2 = make_stats(
vec![(0, 0, 2)],
vec![make_actor(2, 0, 0, 0), make_actor(3, 0, 0, 0)],
);
history.record(&stats2);
// A should be evicted (LRU), B and C kept
assert!(history.actor_sparkline(&ActorAddress([1; 32])).is_none());
assert!(history.actor_sparkline(&ActorAddress([2; 32])).is_some());
assert!(history.actor_sparkline(&ActorAddress([3; 32])).is_some());
}
#[test]
fn actor_rates_track_deltas() {
let history = DashboardHistory::new(HistoryConfig::default());
let stats1 = make_stats(vec![(0, 0, 1)], vec![make_actor(1, 0, 5, 100)]);
history.record(&stats1);
let stats2 = make_stats(vec![(0, 0, 1)], vec![make_actor(1, 0, 3, 175)]);
history.record(&stats2);
let (mailbox, rates) = history.actor_sparkline(&ActorAddress([1; 32])).unwrap();
assert_eq!(mailbox, vec![5, 3]);
assert_eq!(rates[0], 100); // first: 100 - 0
assert_eq!(rates[1], 75); // second: 175 - 100
}
}

View file

@ -20,6 +20,7 @@ pub struct DashboardEvent {
pub level: String,
pub message: String,
pub worker_id: Option<usize>,
pub actor_addr: Option<String>,
pub fields: serde_json::Map<String, serde_json::Value>,
}
@ -83,6 +84,28 @@ impl EventStore {
(batch, new_cursor)
}
/// Read recent events for a specific actor address (hex prefix match).
/// Returns up to `limit` most recent matching events.
pub fn read_for_actor(&self, actor_hex: &str, limit: usize) -> Vec<DashboardEvent> {
let events = self.events.lock().unwrap();
let lower = actor_hex.to_lowercase();
events
.iter()
.rev()
.filter(|e| {
e.actor_addr
.as_ref()
.map(|a| a.to_lowercase().starts_with(&lower) || a.to_lowercase().contains(&lower))
.unwrap_or(false)
})
.take(limit)
.cloned()
.collect::<Vec<_>>()
.into_iter()
.rev()
.collect()
}
/// Drains the full recording log. Only available when recording is enabled.
/// This is destructive — events are consumed. Intended for `save_trace()`.
pub fn all_events(&self) -> Option<Vec<DashboardEvent>> {
@ -183,24 +206,39 @@ where
let mut visitor = FieldVisitor::new();
event.record(&mut visitor);
// Walk span context to find worker_id
// Walk span context to find worker_id and actor_addr
let mut worker_id = None;
let mut actor_addr = None;
if let Some(scope) = ctx.event_scope(event) {
for span in scope {
let exts = span.extensions();
if let Some(wid) = exts.get::<WorkerIdField>() {
worker_id = Some(wid.0);
if worker_id.is_none() {
if let Some(wid) = exts.get::<WorkerIdField>() {
worker_id = Some(wid.0);
}
}
if actor_addr.is_none() {
if let Some(aa) = exts.get::<ActorAddrField>() {
actor_addr = Some(aa.0.clone());
}
}
if worker_id.is_some() && actor_addr.is_some() {
break;
}
}
}
// Also check if worker_id was a field on the event itself
// Also check if worker_id or actor_addr was a field on the event itself
if worker_id.is_none() {
if let Some(serde_json::Value::Number(n)) = visitor.fields.get("worker_id") {
worker_id = n.as_u64().map(|v| v as usize);
}
}
if actor_addr.is_none() {
if let Some(serde_json::Value::String(s)) = visitor.fields.get("actor_addr") {
actor_addr = Some(s.clone());
}
}
let dashboard_event = DashboardEvent {
seq: 0, // filled by push()
@ -208,6 +246,7 @@ where
level: event.metadata().level().to_string(),
message: visitor.message,
worker_id,
actor_addr,
fields: visitor.fields,
};
@ -215,19 +254,25 @@ where
}
fn on_new_span(&self, attrs: &span::Attributes<'_>, id: &span::Id, ctx: Context<'_, S>) {
// Extract worker_id from span fields and store in extensions
// Extract worker_id and actor_addr from span fields and store in extensions
let mut visitor = FieldVisitor::new();
attrs.record(&mut visitor);
if let Some(serde_json::Value::Number(n)) = visitor.fields.get("worker_id") {
if let Some(wid) = n.as_u64() {
if let Some(span) = ctx.span(id) {
if let Some(span) = ctx.span(id) {
if let Some(serde_json::Value::Number(n)) = visitor.fields.get("worker_id") {
if let Some(wid) = n.as_u64() {
span.extensions_mut().insert(WorkerIdField(wid as usize));
}
}
if let Some(serde_json::Value::String(s)) = visitor.fields.get("actor_addr") {
span.extensions_mut().insert(ActorAddrField(s.clone()));
}
}
}
}
/// Stored in span extensions to propagate worker_id to child events.
struct WorkerIdField(usize);
/// Stored in span extensions to propagate actor_addr to child events.
struct ActorAddrField(String);

View file

@ -1,10 +1,15 @@
pub mod collector;
pub mod history;
pub mod investigate;
pub mod layer;
pub mod trace;
pub mod warnings;
mod actor_detail_html;
mod actors_html;
mod dashboard_html;
mod server;
pub mod topology;
mod topology_html;
#[cfg(feature = "tui")]
pub mod tui;
@ -27,6 +32,7 @@ use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;
use crate::collector::StatsCollector;
use crate::history::{DashboardHistory, HistoryConfig};
use crate::layer::{now_ms, DashboardLayer, EventStore};
use crate::trace::{RuntimeTrace, TimestampedStats};
@ -80,6 +86,7 @@ pub struct DashboardHandle {
collector: Arc<Mutex<Option<Arc<StatsCollector>>>>,
shutdown: Arc<AtomicBool>,
stats_timeline: Arc<ArrayQueue<TimestampedStats>>,
history: Arc<DashboardHistory>,
recording: bool,
#[cfg(feature = "distribution")]
distribution: Arc<Mutex<Option<Arc<dyn distribution_collector::DistributionStatsProvider>>>>,
@ -115,6 +122,11 @@ impl DashboardHandle {
*self.distribution.lock().unwrap() = Some(provider);
}
/// Access the time-series history store (for TUI sparklines, etc.).
pub fn history(&self) -> &Arc<DashboardHistory> {
&self.history
}
/// Signal the dashboard to shut down (SSE clients receive "done").
pub fn shutdown(&self) {
self.shutdown.store(true, Ordering::Release);
@ -159,6 +171,7 @@ pub fn start_dashboard(config: DashboardConfig) -> DashboardHandle {
let collector: Arc<Mutex<Option<Arc<StatsCollector>>>> = Arc::new(Mutex::new(None));
let shutdown = Arc::new(AtomicBool::new(false));
let stats_timeline = Arc::new(ArrayQueue::new(config.record_stats_capacity.max(1)));
let history = Arc::new(DashboardHistory::new(HistoryConfig::default()));
#[cfg(feature = "distribution")]
let distribution: Arc<Mutex<Option<Arc<dyn distribution_collector::DistributionStatsProvider>>>> =
@ -169,6 +182,7 @@ pub fn start_dashboard(config: DashboardConfig) -> DashboardHandle {
Arc::clone(&runtime),
Arc::clone(&collector),
Arc::clone(&shutdown),
Arc::clone(&history),
config.port,
#[cfg(feature = "distribution")]
Arc::clone(&distribution),
@ -210,6 +224,7 @@ pub fn start_dashboard(config: DashboardConfig) -> DashboardHandle {
collector,
shutdown,
stats_timeline,
history,
recording: config.record,
#[cfg(feature = "distribution")]
distribution,

View file

@ -8,11 +8,16 @@ use std::collections::HashMap;
use swactor::runtime::Runtime;
use crate::actor_detail_html::ACTOR_DETAIL_HTML;
use crate::actors_html::ACTORS_HTML;
use crate::collector::StatsCollector;
use crate::dashboard_html::DASHBOARD_HTML;
use crate::history::DashboardHistory;
use crate::layer::EventStore;
use crate::topology;
use crate::topology_html::TOPOLOGY_HTML;
use crate::trace::RuntimeTrace;
use crate::warnings::{WarningConfig, WarningDetector};
#[cfg(feature = "distribution")]
use crate::distribution_collector::DistributionStatsProvider;
@ -119,6 +124,7 @@ pub(crate) fn spawn_http_server(
runtime: Arc<Mutex<Option<Arc<Runtime>>>>,
collector: Arc<Mutex<Option<Arc<StatsCollector>>>>,
shutdown: Arc<AtomicBool>,
history: Arc<DashboardHistory>,
port: u16,
#[cfg(feature = "distribution")]
distribution: Arc<Mutex<Option<Arc<dyn DistributionStatsProvider>>>>,
@ -134,6 +140,7 @@ pub(crate) fn spawn_http_server(
let runtime = Arc::clone(&runtime);
let collector = Arc::clone(&collector);
let shutdown = Arc::clone(&shutdown);
let history = Arc::clone(&history);
let cmd_router = Arc::clone(&cmd_router);
#[cfg(feature = "distribution")]
let distribution = Arc::clone(&distribution);
@ -149,6 +156,7 @@ pub(crate) fn spawn_http_server(
match path {
"/" => respond_html(request, DASHBOARD_HTML, "live"),
"/actors" => respond_html(request, ACTORS_HTML, "live"),
"/topology" => respond_html(request, TOPOLOGY_HTML, "live"),
#[cfg(feature = "distribution")]
"/distribution" => respond_html(request, DISTRIBUTION_HTML, "live"),
"/events" => {
@ -158,6 +166,7 @@ pub(crate) fn spawn_http_server(
Arc::clone(&runtime),
Arc::clone(&collector),
Arc::clone(&shutdown),
Arc::clone(&history),
#[cfg(feature = "distribution")]
Arc::clone(&distribution),
);
@ -169,6 +178,16 @@ pub(crate) fn spawn_http_server(
Arc::clone(&collector),
);
}
"/api/history" => {
handle_history_api(request, Arc::clone(&history));
}
"/api/topology" => {
handle_topology_api(
request,
Arc::clone(&runtime),
Arc::clone(&collector),
);
}
"/api/investigate" => {
handle_investigate_api(
request,
@ -185,6 +204,13 @@ pub(crate) fn spawn_http_server(
Arc::clone(&distribution),
);
}
"/api/logs" => {
handle_logs_api(request, &url, Arc::clone(&store));
}
_ if path.starts_with("/actor/") => {
let hex = &path[7..]; // strip "/actor/"
respond_actor_detail(request, hex);
}
_ => respond_404(request),
}
}
@ -192,12 +218,25 @@ pub(crate) fn spawn_http_server(
}
}
fn respond_actor_detail(request: tiny_http::Request, hex_addr: &str) {
let html = ACTOR_DETAIL_HTML
.replace("__DASHBOARD_MODE__", "live")
.replace("__ACTOR_ADDR__", hex_addr);
let response = tiny_http::Response::from_string(html).with_header(
"Content-Type: text/html; charset=utf-8"
.parse::<tiny_http::Header>()
.unwrap(),
);
let _ = request.respond(response);
}
fn handle_live_sse(
request: tiny_http::Request,
store: Arc<EventStore>,
runtime: Arc<Mutex<Option<Arc<Runtime>>>>,
collector: Arc<Mutex<Option<Arc<StatsCollector>>>>,
shutdown: Arc<AtomicBool>,
history: Arc<DashboardHistory>,
#[cfg(feature = "distribution")]
distribution: Arc<Mutex<Option<Arc<dyn DistributionStatsProvider>>>>,
) {
@ -207,6 +246,14 @@ fn handle_live_sse(
// Spawn producer thread
thread::spawn(move || {
let mut cursor: u64 = 0;
let mut warning_detector = WarningDetector::new(WarningConfig::default());
let mut tick_count: u64 = 0;
// Send initial history snapshot so sparklines render immediately
if history.sample_count() > 0 {
let json = history.worker_history_json();
let _ = tx.send(format_sse("history", &json));
}
loop {
// Send stats if runtime is available
@ -217,10 +264,33 @@ fn handle_live_sse(
if let Some(col) = collector.lock().unwrap().as_ref() {
col.enrich(&mut stats);
}
history.record(&stats);
// Run warning detection
let warnings = warning_detector.check(&stats);
if !warnings.is_empty() {
if let Ok(wjson) = serde_json::to_string(&warnings) {
if tx.send(format_sse("warnings", &wjson)).is_err() {
return;
}
}
}
let json = serde_json::to_string(&stats).unwrap();
if tx.send(format_sse("stats", &json)).is_err() {
return;
}
// Send topology every 5th tick (~1/sec)
tick_count += 1;
if tick_count % 5 == 0 {
let topo = topology::worker_topology(&stats);
if let Ok(tjson) = serde_json::to_string(&topo) {
if tx.send(format_sse("topology", &tjson)).is_err() {
return;
}
}
}
}
}
@ -353,6 +423,67 @@ fn handle_distribution_api(
let _ = request.respond(response);
}
fn handle_topology_api(
request: tiny_http::Request,
runtime: Arc<Mutex<Option<Arc<Runtime>>>>,
collector: Arc<Mutex<Option<Arc<StatsCollector>>>>,
) {
let maybe_rt = runtime.lock().unwrap().clone();
let json = match maybe_rt {
Some(rt) => {
let mut stats = rt.stats();
if let Some(col) = collector.lock().unwrap().as_ref() {
col.enrich(&mut stats);
}
let topo = topology::worker_topology(&stats);
serde_json::to_string(&topo).unwrap_or_else(|_| "{}".into())
}
None => "{}".to_string(),
};
let response = tiny_http::Response::from_string(json).with_header(
"Content-Type: application/json"
.parse::<tiny_http::Header>()
.unwrap(),
);
let _ = request.respond(response);
}
fn handle_logs_api(request: tiny_http::Request, url: &str, store: Arc<EventStore>) {
let params = parse_query_string(url);
let actor = params.get("actor").cloned().unwrap_or_default();
let limit: usize = params
.get("limit")
.and_then(|s| s.parse().ok())
.unwrap_or(200);
let level = params.get("level").cloned();
let mut events = store.read_for_actor(&actor, limit);
// Filter by level if specified
if let Some(ref lvl) = level {
let lvl_upper = lvl.to_uppercase();
events.retain(|e| e.level == lvl_upper);
}
let json = serde_json::to_string(&events).unwrap_or_else(|_| "[]".into());
let response = tiny_http::Response::from_string(json).with_header(
"Content-Type: application/json"
.parse::<tiny_http::Header>()
.unwrap(),
);
let _ = request.respond(response);
}
fn handle_history_api(request: tiny_http::Request, history: Arc<DashboardHistory>) {
let json = history.worker_history_json();
let response = tiny_http::Response::from_string(json).with_header(
"Content-Type: application/json"
.parse::<tiny_http::Header>()
.unwrap(),
);
let _ = request.respond(response);
}
fn parse_query_string(url: &str) -> HashMap<String, String> {
let mut params = HashMap::new();
if let Some(qs) = url.split('?').nth(1) {

View file

@ -0,0 +1,82 @@
//! Actor-to-actor (and worker-to-worker) message flow topology.
//!
//! Currently derives topology from per-worker cross_sends/local_sends stats.
//! Future: sample-based per-actor source→destination tracking with core instrumentation.
use swactor::stats::RuntimeStats;
/// An edge in the topology graph.
#[derive(Debug, Clone, serde::Serialize)]
pub struct TopologyEdge {
pub source: String,
pub target: String,
pub weight: u64,
pub label: String,
}
/// A node in the topology graph.
#[derive(Debug, Clone, serde::Serialize)]
pub struct TopologyNode {
pub id: String,
pub label: String,
pub actor_count: usize,
pub group: usize,
}
/// A snapshot of the current topology.
#[derive(Debug, Clone, serde::Serialize)]
pub struct TopologySnapshot {
pub nodes: Vec<TopologyNode>,
pub edges: Vec<TopologyEdge>,
}
/// Build a worker-level topology from RuntimeStats.
///
/// Workers are nodes, edges represent message flow:
/// - Self-loops for local_sends
/// - Cross-edges distributed proportionally (until per-destination tracking exists)
pub fn worker_topology(stats: &RuntimeStats) -> TopologySnapshot {
let mut nodes = Vec::new();
let mut edges = Vec::new();
for w in &stats.workers {
nodes.push(TopologyNode {
id: format!("w{}", w.id),
label: format!("W{}", w.id),
actor_count: w.num_actors,
group: w.id,
});
// Local sends = self-loop
if w.local_sends > 0 {
edges.push(TopologyEdge {
source: format!("w{}", w.id),
target: format!("w{}", w.id),
weight: w.local_sends,
label: format!("{} local", w.local_sends),
});
}
// Cross sends — without per-destination data, distribute evenly to other workers
if w.cross_sends > 0 && stats.workers.len() > 1 {
let others: Vec<&swactor::stats::WorkerInfo> =
stats.workers.iter().filter(|o| o.id != w.id).collect();
let per_worker = w.cross_sends / others.len() as u64;
let remainder = w.cross_sends % others.len() as u64;
for (i, other) in others.iter().enumerate() {
let count = per_worker + if (i as u64) < remainder { 1 } else { 0 };
if count > 0 {
edges.push(TopologyEdge {
source: format!("w{}", w.id),
target: format!("w{}", other.id),
weight: count,
label: format!("{} cross", count),
});
}
}
}
}
TopologySnapshot { nodes, edges }
}

View file

@ -0,0 +1,280 @@
pub const TOPOLOGY_HTML: &str = r##"<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Topology — Swactor Dashboard</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: 'Menlo', 'Consolas', 'Monaco', monospace; background: #0f1117; color: #e0e0e0; font-size: 13px; }
.header {
display: flex; align-items: center; justify-content: space-between;
padding: 12px 20px; background: #161822; border-bottom: 1px solid #2a2d3e;
}
.header h1 { font-size: 16px; font-weight: 600; color: #fff; }
.status-dot {
width: 10px; height: 10px; border-radius: 50%; background: #4caf50;
display: inline-block; margin-left: 8px; vertical-align: middle;
}
.status-dot.disconnected { background: #f44336; }
.header-left { display: flex; align-items: center; }
.nav-links { display: flex; gap: 4px; margin-left: 20px; }
.nav-link {
color: #888; text-decoration: none; font-size: 12px;
padding: 4px 10px; border-radius: 3px;
}
.nav-link:hover { color: #e0e0e0; }
.nav-link.active { color: #fff; background: #2a2d3e; }
.content { padding: 0; display: flex; flex-direction: column; height: calc(100vh - 49px); }
canvas#topoCanvas { flex: 1; width: 100%; cursor: grab; }
canvas#topoCanvas:active { cursor: grabbing; }
.legend {
padding: 8px 20px; background: #161822; border-top: 1px solid #2a2d3e;
font-size: 11px; color: #888;
}
</style>
</head>
<body>
<div class="header">
<div class="header-left">
<h1>Swactor Runtime Dashboard <span id="statusDot" class="status-dot"></span></h1>
<nav class="nav-links">
<a href="/" class="nav-link">Overview</a>
<a href="/actors" class="nav-link">Actors</a>
<a href="/topology" class="nav-link active">Topology</a>
<a href="/distribution" class="nav-link">Distribution</a>
</nav>
</div>
</div>
<div class="content">
<canvas id="topoCanvas"></canvas>
<div class="legend">
Node size = actor count. Edge thickness = message volume. Green = local sends. Blue = cross-worker sends.
</div>
</div>
<script>
(function() {
var canvas = document.getElementById('topoCanvas');
var ctx = canvas.getContext('2d');
var dot = document.getElementById('statusDot');
var colors = ['#4caf50','#2196f3','#ff9800','#f44336','#9c27b0','#00bcd4','#ffeb3b','#e91e63'];
var nodes = [];
var edges = [];
var positions = {};
function resize() {
var dpr = window.devicePixelRatio || 1;
var rect = canvas.getBoundingClientRect();
canvas.width = rect.width * dpr;
canvas.height = rect.height * dpr;
ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
}
window.addEventListener('resize', resize);
resize();
function initPositions() {
var W = canvas.getBoundingClientRect().width;
var H = canvas.getBoundingClientRect().height;
var cx = W / 2, cy = H / 2;
var r = Math.min(W, H) * 0.3;
nodes.forEach(function(n, i) {
if (!positions[n.id]) {
var angle = (2 * Math.PI * i) / Math.max(1, nodes.length);
positions[n.id] = {
x: cx + r * Math.cos(angle),
y: cy + r * Math.sin(angle),
vx: 0, vy: 0
};
}
});
}
function simulate() {
var W = canvas.getBoundingClientRect().width;
var H = canvas.getBoundingClientRect().height;
var cx = W / 2, cy = H / 2;
// Repulsion between nodes
for (var i = 0; i < nodes.length; i++) {
var pi = positions[nodes[i].id];
if (!pi) continue;
for (var j = i + 1; j < nodes.length; j++) {
var pj = positions[nodes[j].id];
if (!pj) continue;
var dx = pi.x - pj.x;
var dy = pi.y - pj.y;
var dist = Math.sqrt(dx * dx + dy * dy) || 1;
var force = 8000 / (dist * dist);
pi.vx += dx / dist * force;
pi.vy += dy / dist * force;
pj.vx -= dx / dist * force;
pj.vy -= dy / dist * force;
}
}
// Attraction along edges
edges.forEach(function(e) {
if (e.source === e.target) return;
var ps = positions[e.source];
var pt = positions[e.target];
if (!ps || !pt) return;
var dx = pt.x - ps.x;
var dy = pt.y - ps.y;
var dist = Math.sqrt(dx * dx + dy * dy) || 1;
var force = (dist - 150) * 0.01;
ps.vx += dx / dist * force;
ps.vy += dy / dist * force;
pt.vx -= dx / dist * force;
pt.vy -= dy / dist * force;
});
// Gravity toward center
for (var i = 0; i < nodes.length; i++) {
var p = positions[nodes[i].id];
if (!p) continue;
p.vx += (cx - p.x) * 0.002;
p.vy += (cy - p.y) * 0.002;
}
// Apply velocity with damping
for (var i = 0; i < nodes.length; i++) {
var p = positions[nodes[i].id];
if (!p) continue;
p.vx *= 0.85;
p.vy *= 0.85;
p.x += p.vx;
p.y += p.vy;
p.x = Math.max(30, Math.min(W - 30, p.x));
p.y = Math.max(30, Math.min(H - 30, p.y));
}
}
function draw() {
var W = canvas.getBoundingClientRect().width;
var H = canvas.getBoundingClientRect().height;
ctx.clearRect(0, 0, W, H);
if (nodes.length === 0) {
ctx.fillStyle = '#555';
ctx.font = '14px monospace';
ctx.textAlign = 'center';
ctx.fillText('Waiting for topology data...', W / 2, H / 2);
return;
}
// Draw edges
var maxWeight = Math.max(1, Math.max.apply(null, edges.map(function(e) { return e.weight; })));
edges.forEach(function(e) {
var ps = positions[e.source];
var pt = positions[e.target];
if (!ps || !pt) return;
var isSelf = e.source === e.target;
var thickness = Math.max(1, (e.weight / maxWeight) * 6);
var color = isSelf ? 'rgba(76, 175, 80, 0.5)' : 'rgba(33, 150, 243, 0.5)';
if (isSelf) {
// Self-loop: small arc above the node
ctx.beginPath();
ctx.arc(ps.x, ps.y - 25, 15, 0.3, Math.PI - 0.3);
ctx.strokeStyle = color;
ctx.lineWidth = thickness;
ctx.stroke();
} else {
ctx.beginPath();
ctx.moveTo(ps.x, ps.y);
ctx.lineTo(pt.x, pt.y);
ctx.strokeStyle = color;
ctx.lineWidth = thickness;
ctx.stroke();
// Arrow
var angle = Math.atan2(pt.y - ps.y, pt.x - ps.x);
var headLen = 8;
var mx = (ps.x + pt.x) / 2;
var my = (ps.y + pt.y) / 2;
ctx.beginPath();
ctx.moveTo(mx, my);
ctx.lineTo(mx - headLen * Math.cos(angle - 0.3), my - headLen * Math.sin(angle - 0.3));
ctx.moveTo(mx, my);
ctx.lineTo(mx - headLen * Math.cos(angle + 0.3), my - headLen * Math.sin(angle + 0.3));
ctx.strokeStyle = color;
ctx.lineWidth = 1.5;
ctx.stroke();
// Edge label
ctx.fillStyle = '#666';
ctx.font = '9px monospace';
ctx.textAlign = 'center';
ctx.fillText(e.label, mx, my - 6);
}
});
// Draw nodes
nodes.forEach(function(n) {
var p = positions[n.id];
if (!p) return;
var r = Math.max(12, 8 + n.actor_count * 2);
var color = colors[n.group % colors.length];
ctx.beginPath();
ctx.arc(p.x, p.y, r, 0, 2 * Math.PI);
ctx.fillStyle = color;
ctx.globalAlpha = 0.7;
ctx.fill();
ctx.globalAlpha = 1;
ctx.strokeStyle = '#fff';
ctx.lineWidth = 1.5;
ctx.stroke();
ctx.fillStyle = '#fff';
ctx.font = 'bold 11px monospace';
ctx.textAlign = 'center';
ctx.textBaseline = 'middle';
ctx.fillText(n.label, p.x, p.y);
ctx.fillStyle = '#888';
ctx.font = '9px monospace';
ctx.fillText(n.actor_count + ' actors', p.x, p.y + r + 12);
});
}
function updateTopology(data) {
nodes = data.nodes || [];
edges = data.edges || [];
initPositions();
}
function tick() {
simulate();
draw();
requestAnimationFrame(tick);
}
tick();
var es = new EventSource('/events');
es.addEventListener('topology', function(e) {
try { updateTopology(JSON.parse(e.data)); } catch(err) { console.error(err); }
});
es.addEventListener('done', function() {
dot.className = 'status-dot disconnected';
es.close();
});
es.onerror = function() { dot.className = 'status-dot disconnected'; };
es.onopen = function() { dot.className = 'status-dot'; };
})();
</script>
</body>
</html>
"##;

View file

@ -1,9 +1,14 @@
use std::collections::{HashMap, VecDeque};
use std::sync::Arc;
use std::time::Instant;
use crossterm::event::{KeyCode, KeyEvent, KeyModifiers};
use swactor::actor::ActorAddress;
use swactor::stats::{RuntimeStats, TickTiming};
use crate::layer::{DashboardEvent, EventStore};
use crate::warnings::{Warning, WarningConfig, WarningDetector};
/// Per-worker data prepared for rendering.
pub struct WorkerView {
pub id: usize,
@ -15,6 +20,10 @@ pub struct WorkerView {
/// Fraction of bar for each phase group: [processing, delivery, spawns, overhead]
pub phase_fractions: [f64; 4],
pub panics: u64,
/// Recent message rates for sparkline rendering.
pub sparkline_rates: Vec<u64>,
/// Recent mailbox depths for sparkline rendering.
pub sparkline_mailbox: Vec<u64>,
}
/// One row in the actor table.
@ -25,6 +34,12 @@ pub struct ActorRow {
pub last_msg_type: Option<String>,
pub messages_processed: u64,
pub poisoned: bool,
/// Per-actor mailbox sparkline (from local ring buffer).
pub sparkline_mailbox: Vec<u64>,
/// Per-actor message rate sparkline.
pub sparkline_rates: Vec<u64>,
/// Per-message-type counts, sorted descending.
pub message_type_counts: Vec<(String, u64)>,
}
#[derive(Clone, Copy, PartialEq, Eq)]
@ -62,6 +77,7 @@ impl SortColumn {
pub enum ViewMode {
Overview,
WorkerDetail,
ActorDetail,
#[cfg(feature = "distribution")]
Distribution,
}
@ -79,7 +95,13 @@ pub struct App {
pub total_panics: u64,
pub num_workers: usize,
pub view_mode: ViewMode,
pub prev_view_mode: ViewMode,
pub focused_worker: usize,
pub focused_actor: Option<ActorAddress>,
pub search_active: bool,
pub search_query: String,
pub search_locked: bool,
pub warnings: Vec<Warning>,
#[cfg(feature = "distribution")]
pub distribution: Option<distribution::snapshot::DistributionNodeSnapshot>,
@ -90,6 +112,21 @@ pub struct App {
prev_time: Instant,
/// Rolling msg rates (smoothed)
msg_rates: Vec<f64>,
/// Per-worker sparkline history (message rate deltas).
sparkline_rates: Vec<VecDeque<u64>>,
/// Per-worker sparkline history (mailbox depths).
sparkline_mailbox: Vec<VecDeque<u64>>,
/// Per-actor sparkline history: address → (prev_msgs, rates, mailbox_depths).
actor_sparklines: HashMap<ActorAddress, (u64, VecDeque<u64>, VecDeque<u64>)>,
warning_detector: WarningDetector,
/// Event store for fetching per-actor logs.
event_store: Option<Arc<EventStore>>,
/// Cached log entries for the focused actor.
pub actor_logs: Vec<DashboardEvent>,
/// Scroll offset for actor log view.
pub log_scroll: usize,
/// Active log level filter (all enabled by default).
pub log_levels: [bool; 5], // ERROR, WARN, INFO, DEBUG, TRACE
}
impl App {
@ -107,7 +144,13 @@ impl App {
total_panics: 0,
num_workers: 0,
view_mode: ViewMode::Overview,
prev_view_mode: ViewMode::Overview,
focused_worker: 0,
focused_actor: None,
search_active: false,
search_query: String::new(),
search_locked: false,
warnings: Vec::new(),
#[cfg(feature = "distribution")]
distribution: None,
#[cfg(feature = "distribution")]
@ -115,9 +158,22 @@ impl App {
prev_messages: Vec::new(),
prev_time: Instant::now(),
msg_rates: Vec::new(),
sparkline_rates: Vec::new(),
sparkline_mailbox: Vec::new(),
actor_sparklines: HashMap::new(),
warning_detector: WarningDetector::new(WarningConfig::default()),
event_store: None,
actor_logs: Vec::new(),
log_scroll: 0,
log_levels: [true; 5],
}
}
/// Set the event store for per-actor log retrieval.
pub fn set_event_store(&mut self, store: Arc<EventStore>) {
self.event_store = Some(store);
}
#[cfg(feature = "distribution")]
pub fn update_distribution(&mut self, snapshot: distribution::snapshot::DistributionNodeSnapshot) {
let max = if snapshot.members.is_empty() { 0 } else { snapshot.members.len() - 1 };
@ -125,6 +181,62 @@ impl App {
self.distribution = Some(snapshot);
}
/// Get visible actor rows (filtered by search query if active).
pub fn visible_actor_rows(&self) -> Vec<&ActorRow> {
if self.search_query.is_empty() {
self.actor_rows.iter().collect()
} else {
let q = self.search_query.to_lowercase();
self.actor_rows.iter().filter(|r| {
let addr = format!("{}", r.address).to_lowercase();
let msg = r.last_msg_type.as_deref().unwrap_or("").to_lowercase();
let worker = format!("w{}", r.worker_id);
addr.contains(&q) || msg.contains(&q) || worker.contains(&q)
}).collect()
}
}
/// Get the focused actor's data (for actor detail view).
pub fn focused_actor_row(&self) -> Option<&ActorRow> {
self.focused_actor.as_ref().and_then(|addr| {
self.actor_rows.iter().find(|r| r.address == *addr)
})
}
/// Refresh the actor log cache from the event store.
pub fn refresh_actor_logs(&mut self) {
if let (Some(addr), Some(store)) = (&self.focused_actor, &self.event_store) {
let hex = format!("{}", addr);
self.actor_logs = store.read_for_actor(&hex, 200);
} else {
self.actor_logs.clear();
}
}
/// Get visible log entries (filtered by level).
pub fn visible_logs(&self) -> Vec<&DashboardEvent> {
self.actor_logs
.iter()
.filter(|e| {
match e.level.as_str() {
"ERROR" => self.log_levels[0],
"WARN" => self.log_levels[1],
"INFO" => self.log_levels[2],
"DEBUG" => self.log_levels[3],
"TRACE" => self.log_levels[4],
_ => true,
}
})
.collect()
}
/// Toggle a log level filter (0=ERROR, 1=WARN, 2=INFO, 3=DEBUG, 4=TRACE).
pub fn toggle_log_level(&mut self, idx: usize) {
if idx < 5 {
self.log_levels[idx] = !self.log_levels[idx];
}
}
/// Actor rows filtered to the focused worker (for worker detail view).
pub fn focused_actor_rows(&self) -> Vec<&ActorRow> {
self.actor_rows
@ -144,6 +256,8 @@ impl App {
if self.prev_messages.len() != stats.workers.len() {
self.prev_messages = stats.workers.iter().map(|w| w.messages_processed).collect();
self.msg_rates = vec![0.0; stats.workers.len()];
self.sparkline_rates.resize_with(stats.workers.len(), VecDeque::new);
self.sparkline_mailbox.resize_with(stats.workers.len(), VecDeque::new);
}
// Compute per-worker views
@ -163,8 +277,19 @@ impl App {
self.msg_rates[i] * 0.6 + rate * 0.4
};
self.msg_rates[i] = smoothed;
// Track sparkline history
let delta = w.messages_processed.saturating_sub(self.prev_messages[i]);
self.prev_messages[i] = w.messages_processed;
let spark_rates = &mut self.sparkline_rates[i];
if spark_rates.len() >= 60 { spark_rates.pop_front(); }
spark_rates.push_back(delta);
let spark_mbox = &mut self.sparkline_mailbox[i];
if spark_mbox.len() >= 60 { spark_mbox.pop_front(); }
spark_mbox.push_back(w.mailbox_depth as u64);
// Load % and phase fractions from tick timings
let timings = stats.tick_timings.get(i).map(|v| v.as_slice()).unwrap_or(&[]);
let (load_pct, phase_fractions) = compute_load_and_phases(timings);
@ -178,6 +303,8 @@ impl App {
load_pct,
phase_fractions,
panics: w.panics,
sparkline_rates: spark_rates.iter().copied().collect(),
sparkline_mailbox: spark_mbox.iter().copied().collect(),
});
}
@ -187,9 +314,20 @@ impl App {
self.total_mailbox = stats.workers.iter().map(|w| w.mailbox_depth).sum();
self.total_panics = stats.workers.iter().map(|w| w.panics).sum();
// Build actor table
// Build actor table with sparkline history
self.actor_rows.clear();
for a in &stats.actor_details {
let (prev, rates_buf, mbox_buf) = self.actor_sparklines
.entry(a.address)
.or_insert_with(|| (0, VecDeque::new(), VecDeque::new()));
let rate_delta = a.messages_processed.saturating_sub(*prev);
*prev = a.messages_processed;
if rates_buf.len() >= 60 { rates_buf.pop_front(); }
rates_buf.push_back(rate_delta);
if mbox_buf.len() >= 60 { mbox_buf.pop_front(); }
mbox_buf.push_back(a.mailbox_depth as u64);
self.actor_rows.push(ActorRow {
address: a.address,
worker_id: a.worker_id,
@ -197,10 +335,21 @@ impl App {
last_msg_type: a.last_msg_type.clone(),
messages_processed: a.messages_processed,
poisoned: a.poisoned,
sparkline_rates: rates_buf.iter().copied().collect(),
sparkline_mailbox: mbox_buf.iter().copied().collect(),
message_type_counts: a.message_type_counts.clone(),
});
}
self.sort_actors();
// Run warning detection
self.warnings = self.warning_detector.check(&stats);
// Refresh actor logs if in detail view
if self.view_mode == ViewMode::ActorDetail {
self.refresh_actor_logs();
}
// Clamp selection
if !self.actor_rows.is_empty() {
self.selected = self.selected.min(self.actor_rows.len() - 1);
@ -246,12 +395,45 @@ impl App {
}
pub fn handle_key(&mut self, key: KeyEvent) {
// Search mode input handling
if self.search_active {
match key.code {
KeyCode::Esc => {
self.search_active = false;
if !self.search_locked {
self.search_query.clear();
}
return;
}
KeyCode::Enter => {
self.search_active = false;
self.search_locked = !self.search_query.is_empty();
return;
}
KeyCode::Backspace => {
self.search_query.pop();
return;
}
KeyCode::Char(c) => {
self.search_query.push(c);
return;
}
_ => return,
}
}
// Global keys
match key.code {
KeyCode::Char('q') => { self.should_quit = true; return; }
KeyCode::Char('c') if key.modifiers.contains(KeyModifiers::CONTROL) => {
self.should_quit = true; return;
}
KeyCode::Char('/') => {
self.search_active = true;
self.search_locked = false;
self.search_query.clear();
return;
}
KeyCode::Char('s') => {
self.sort_column = self.sort_column.next();
self.sort_actors();
@ -265,6 +447,7 @@ impl App {
KeyCode::Tab => {
self.view_mode = match self.view_mode {
ViewMode::Overview => ViewMode::WorkerDetail,
ViewMode::ActorDetail => ViewMode::Overview,
#[cfg(feature = "distribution")]
ViewMode::WorkerDetail => ViewMode::Distribution,
#[cfg(not(feature = "distribution"))]
@ -280,13 +463,15 @@ impl App {
match self.view_mode {
ViewMode::Overview => self.handle_key_overview(key),
ViewMode::WorkerDetail => self.handle_key_worker_detail(key),
ViewMode::ActorDetail => self.handle_key_actor_detail(key),
#[cfg(feature = "distribution")]
ViewMode::Distribution => self.handle_key_distribution(key),
}
}
fn handle_key_overview(&mut self, key: KeyEvent) {
let max = if self.actor_rows.is_empty() { 0 } else { self.actor_rows.len() - 1 };
let visible = self.visible_actor_rows();
let max = if visible.is_empty() { 0 } else { visible.len() - 1 };
match key.code {
KeyCode::Up | KeyCode::Char('k') => {
self.selected = self.selected.saturating_sub(1);
@ -303,11 +488,14 @@ impl App {
KeyCode::Home => { self.selected = 0; }
KeyCode::End => { self.selected = max; }
KeyCode::Enter | KeyCode::Char('l') | KeyCode::Right => {
// Enter worker detail for the selected actor's worker
if let Some(row) = self.actor_rows.get(self.selected) {
self.focused_worker = row.worker_id;
// Enter actor detail for the selected visible actor
if let Some(&row) = visible.get(self.selected) {
self.focused_actor = Some(row.address);
self.prev_view_mode = ViewMode::Overview;
self.view_mode = ViewMode::ActorDetail;
self.log_scroll = 0;
self.refresh_actor_logs();
}
self.view_mode = ViewMode::WorkerDetail;
}
_ => {}
}
@ -337,6 +525,38 @@ impl App {
}
}
fn handle_key_actor_detail(&mut self, key: KeyEvent) {
let max_scroll = self.visible_logs().len().saturating_sub(1);
match key.code {
KeyCode::Esc | KeyCode::Char('h') | KeyCode::Left => {
self.view_mode = self.prev_view_mode;
self.focused_actor = None;
self.actor_logs.clear();
self.log_scroll = 0;
}
KeyCode::Down | KeyCode::Char('j') => {
self.log_scroll = (self.log_scroll + 1).min(max_scroll);
}
KeyCode::Up | KeyCode::Char('k') => {
self.log_scroll = self.log_scroll.saturating_sub(1);
}
KeyCode::PageDown => {
self.log_scroll = (self.log_scroll + 20).min(max_scroll);
}
KeyCode::PageUp => {
self.log_scroll = self.log_scroll.saturating_sub(20);
}
KeyCode::Home => { self.log_scroll = 0; }
KeyCode::End => { self.log_scroll = max_scroll; }
KeyCode::Char('1') => self.toggle_log_level(0),
KeyCode::Char('2') => self.toggle_log_level(1),
KeyCode::Char('3') => self.toggle_log_level(2),
KeyCode::Char('4') => self.toggle_log_level(3),
KeyCode::Char('5') => self.toggle_log_level(4),
_ => {}
}
}
#[cfg(feature = "distribution")]
fn handle_key_distribution(&mut self, key: KeyEvent) {
let max = self

View file

@ -18,6 +18,7 @@ use swactor::runtime::Runtime;
use crate::collector::StatsCollector;
#[cfg(feature = "distribution")]
use crate::distribution_collector::DistributionStatsProvider;
use crate::layer::EventStore;
use self::app::App;
use self::event::{AppEvent, EventLoop};
use self::types::RuntimeEndpoint;
@ -47,6 +48,7 @@ pub fn start_tui(
#[cfg(feature = "distribution")]
distribution: Option<Arc<dyn DistributionStatsProvider>>,
config: TuiConfig,
event_store: Option<Arc<EventStore>>,
) -> io::Result<()> {
// Set up terminal
crossterm::terminal::enable_raw_mode()?;
@ -72,6 +74,7 @@ pub fn start_tui(
#[cfg(feature = "distribution")]
distribution,
config,
event_store,
);
// Restore terminal
@ -92,8 +95,12 @@ fn run_loop(
#[cfg(feature = "distribution")]
distribution: Option<Arc<dyn DistributionStatsProvider>>,
config: TuiConfig,
event_store: Option<Arc<EventStore>>,
) -> io::Result<()> {
let mut app = App::new();
if let Some(store) = event_store {
app.set_event_store(store);
}
let mut table_state = TableState::default();
let events = EventLoop::new(config.poll_interval_ms);

View file

@ -2,7 +2,7 @@ use ratatui::Frame;
use ratatui::layout::{Constraint, Layout, Rect};
use ratatui::style::{Color, Modifier, Style};
use ratatui::text::{Line, Span};
use ratatui::widgets::{Block, Borders, Cell, Paragraph, Row, Table, TableState};
use ratatui::widgets::{Block, Borders, Cell, Paragraph, Row, Sparkline, Table, TableState};
use super::app::{App, SortColumn, ViewMode};
@ -11,6 +11,7 @@ pub fn draw(f: &mut Frame, app: &App, table_state: &mut TableState) {
match app.view_mode {
ViewMode::Overview => draw_overview(f, app, table_state),
ViewMode::WorkerDetail => draw_worker_detail(f, app, table_state),
ViewMode::ActorDetail => draw_actor_detail(f, app),
#[cfg(feature = "distribution")]
ViewMode::Distribution => draw_distribution(f, app, table_state),
}
@ -20,17 +21,23 @@ pub fn draw(f: &mut Frame, app: &App, table_state: &mut TableState) {
fn draw_overview(f: &mut Frame, app: &App, table_state: &mut TableState) {
let num_workers = app.workers.len().max(1);
let has_sparkline_data = app.workers.iter().any(|w| w.sparkline_rates.len() > 1);
let sparkline_height = if has_sparkline_data { 4u16 } else { 0u16 };
let chunks = Layout::vertical([
Constraint::Length(num_workers as u16),
Constraint::Length(sparkline_height),
Constraint::Length(1),
Constraint::Fill(1),
])
.split(f.area());
draw_worker_bars(f, app, chunks[0]);
draw_summary(f, app, chunks[1]);
draw_actor_table(f, app, table_state, chunks[2]);
if has_sparkline_data {
draw_worker_sparklines(f, app, chunks[1]);
}
draw_summary(f, app, chunks[2]);
draw_actor_table(f, app, table_state, chunks[3]);
}
/// Render htop-style worker bars.
@ -45,6 +52,36 @@ fn draw_worker_bars(f: &mut Frame, app: &App, area: Rect) {
f.render_widget(paragraph, area);
}
/// Render per-worker sparklines showing message rate trends.
fn draw_worker_sparklines(f: &mut Frame, app: &App, area: Rect) {
if app.workers.is_empty() {
return;
}
// Split area horizontally: one sparkline per worker
let constraints: Vec<Constraint> = app
.workers
.iter()
.map(|_| Constraint::Ratio(1, app.workers.len() as u32))
.collect();
let cols = Layout::horizontal(constraints).split(area);
for (i, w) in app.workers.iter().enumerate() {
if let Some(&col_area) = cols.get(i) {
let block = Block::default()
.borders(Borders::NONE)
.title(Span::styled(
format!(" W{} ", w.id),
Style::default().fg(Color::DarkGray),
));
let sparkline = Sparkline::default()
.block(block)
.data(&w.sparkline_rates)
.style(Style::default().fg(Color::Green));
f.render_widget(sparkline, col_area);
}
}
}
fn build_worker_line(w: &super::app::WorkerView, total_width: usize) -> Line<'static> {
let id_str = format!("{:>3}", w.id);
let suffix = format!(
@ -129,7 +166,7 @@ fn draw_summary(f: &mut Frame, app: &App, area: Rect) {
Style::default().fg(Color::DarkGray)
};
let line = Line::from(vec![
let mut spans = vec![
Span::styled(" Workers: ", Style::default().fg(Color::DarkGray)),
Span::styled(
format!("{}", app.num_workers),
@ -152,13 +189,47 @@ fn draw_summary(f: &mut Frame, app: &App, area: Rect) {
),
Span::styled(" Panics: ", Style::default().fg(Color::DarkGray)),
Span::styled(format!("{}", app.total_panics), panics_style),
]);
];
if !app.warnings.is_empty() {
spans.push(Span::styled(" Warnings: ", Style::default().fg(Color::DarkGray)));
spans.push(Span::styled(
format!("{}", app.warnings.len()),
Style::default().fg(Color::Yellow).add_modifier(Modifier::BOLD),
));
}
let line = Line::from(spans);
f.render_widget(Paragraph::new(line), area);
}
/// Render the actor table with scrolling and selection.
/// Render the actor table with scrolling, selection, and search.
fn draw_actor_table(f: &mut Frame, app: &App, table_state: &mut TableState, area: Rect) {
// Split area: optional search bar + table
let has_search = app.search_active || !app.search_query.is_empty();
let search_height = if has_search { 1u16 } else { 0 };
let chunks = Layout::vertical([
Constraint::Length(search_height),
Constraint::Fill(1),
])
.split(area);
// Draw search bar
if has_search {
let search_style = if app.search_active {
Style::default().fg(Color::Yellow)
} else {
Style::default().fg(Color::DarkGray)
};
let cursor = if app.search_active { "\u{2588}" } else { "" };
let prefix = if app.search_locked { " [locked] /" } else { " /" };
let line = Line::from(vec![
Span::styled(prefix, Style::default().fg(Color::DarkGray)),
Span::styled(format!("{}{}", app.search_query, cursor), search_style),
]);
f.render_widget(Paragraph::new(line), chunks[0]);
}
let sort_arrow = if app.sort_desc { " \u{25bc}" } else { " \u{25b2}" };
let columns = [
@ -181,8 +252,8 @@ fn draw_actor_table(f: &mut Frame, app: &App, table_state: &mut TableState, area
});
let header = Row::new(header_cells).height(1);
let rows: Vec<Row> = app
.actor_rows
let visible = app.visible_actor_rows();
let rows: Vec<Row> = visible
.iter()
.map(|a| actor_row_cells(a))
.collect();
@ -190,7 +261,22 @@ fn draw_actor_table(f: &mut Frame, app: &App, table_state: &mut TableState, area
table_state.select(Some(app.selected));
let help_text =
" q: quit \u{2191}\u{2193}: scroll s: sort column r: reverse Tab: worker view Enter: drill in";
" q: quit /: search \u{2191}\u{2193}: scroll s: sort r: reverse Tab: worker view Enter: detail";
let title = if !app.search_query.is_empty() {
format!(
" Actors ({} of {} matching \"{}\") ",
visible.len(),
app.actor_rows.len(),
app.search_query,
)
} else {
format!(
" Actors (sorted by {}{}) ",
app.sort_column.label(),
sort_arrow
)
};
let table = Table::new(
rows,
@ -206,11 +292,7 @@ fn draw_actor_table(f: &mut Frame, app: &App, table_state: &mut TableState, area
.block(
Block::default()
.borders(Borders::ALL)
.title(format!(
" Actors (sorted by {}{}) ",
app.sort_column.label(),
sort_arrow
))
.title(title)
.title_bottom(Line::from(help_text).centered()),
)
.row_highlight_style(
@ -443,6 +525,231 @@ fn draw_focused_actor_table(f: &mut Frame, app: &App, table_state: &mut TableSta
f.render_stateful_widget(table, area, table_state);
}
// ─── Actor Detail View ──────────────────────────────────────────────────────
fn draw_actor_detail(f: &mut Frame, app: &App) {
let actor = match app.focused_actor_row() {
Some(a) => a,
None => {
let msg = Paragraph::new(" No actor selected. Press Esc to go back.")
.style(Style::default().fg(Color::DarkGray));
f.render_widget(msg, f.area());
return;
}
};
let has_types = !actor.message_type_counts.is_empty();
let type_height = if has_types {
(actor.message_type_counts.len() as u16 + 2).min(10)
} else {
0
};
let chunks = Layout::vertical([
Constraint::Length(5), // Info card
Constraint::Length(5), // Rate sparkline
Constraint::Length(5), // Mailbox sparkline
Constraint::Length(type_height), // Type breakdown
Constraint::Length(1), // Help bar
Constraint::Fill(1), // Logs panel
])
.split(f.area());
// Info card
let addr_str = format!("{}", actor.address);
let status = if actor.poisoned { "POISONED" } else { "Healthy" };
let status_color = if actor.poisoned { Color::Red } else { Color::Green };
let msg_type = actor.last_msg_type.as_deref()
.map(|s| short_type_name(Some(s)))
.unwrap_or_else(|| "\u{2014}".to_string());
let info_lines = vec![
Line::from(vec![
Span::styled(" Address: ", Style::default().fg(Color::DarkGray)),
Span::styled(addr_str, Style::default().fg(Color::White).add_modifier(Modifier::BOLD)),
]),
Line::from(vec![
Span::styled(" Worker: ", Style::default().fg(Color::DarkGray)),
Span::styled(format!("W{}", actor.worker_id), Style::default().fg(Color::Cyan)),
Span::styled(" Status: ", Style::default().fg(Color::DarkGray)),
Span::styled(status, Style::default().fg(status_color).add_modifier(Modifier::BOLD)),
]),
Line::from(vec![
Span::styled(" Messages: ", Style::default().fg(Color::DarkGray)),
Span::styled(
format_num(actor.messages_processed),
Style::default().fg(Color::Cyan).add_modifier(Modifier::BOLD),
),
Span::styled(" Mailbox: ", Style::default().fg(Color::DarkGray)),
Span::styled(
format!("{}", actor.mailbox_depth),
Style::default().fg(Color::Blue).add_modifier(Modifier::BOLD),
),
Span::styled(" Last Msg: ", Style::default().fg(Color::DarkGray)),
Span::styled(msg_type, Style::default().fg(Color::Green)),
]),
];
let info_block = Block::default()
.borders(Borders::ALL)
.title(" Actor Detail ");
let info = Paragraph::new(info_lines).block(info_block);
f.render_widget(info, chunks[0]);
// Rate sparkline
let rate_block = Block::default()
.borders(Borders::ALL)
.title(Span::styled(" Msg Rate ", Style::default().fg(Color::Green)));
let rate_sparkline = Sparkline::default()
.block(rate_block)
.data(&actor.sparkline_rates)
.style(Style::default().fg(Color::Green));
f.render_widget(rate_sparkline, chunks[1]);
// Mailbox sparkline
let mbox_block = Block::default()
.borders(Borders::ALL)
.title(Span::styled(" Mailbox Depth ", Style::default().fg(Color::Blue)));
let mbox_sparkline = Sparkline::default()
.block(mbox_block)
.data(&actor.sparkline_mailbox)
.style(Style::default().fg(Color::Blue));
f.render_widget(mbox_sparkline, chunks[2]);
// Message type breakdown
if has_types {
draw_type_breakdown(f, &actor.message_type_counts, chunks[3]);
}
// Help bar
let level_names = ["ERR", "WARN", "INFO", "DBG", "TRC"];
let level_colors = [Color::Red, Color::Yellow, Color::Blue, Color::DarkGray, Color::DarkGray];
let mut help_spans: Vec<Span> = vec![
Span::styled(
" Esc: back \u{2191}\u{2193}: scroll logs ",
Style::default().fg(Color::DarkGray),
),
];
for (i, &name) in level_names.iter().enumerate() {
let active = app.log_levels[i];
let color = if active { level_colors[i] } else { Color::DarkGray };
let style = if active {
Style::default().fg(color).add_modifier(Modifier::BOLD)
} else {
Style::default().fg(color)
};
help_spans.push(Span::styled(format!("{}:{} ", i + 1, name), style));
}
f.render_widget(Paragraph::new(Line::from(help_spans)), chunks[4]);
// Logs panel
draw_actor_logs(f, app, chunks[5]);
}
fn draw_type_breakdown(f: &mut Frame, types: &[(String, u64)], area: Rect) {
let total: u64 = types.iter().map(|(_, c)| *c).sum();
let max_count = types.first().map(|(_, c)| *c).unwrap_or(1).max(1);
let inner_height = area.height.saturating_sub(2) as usize;
let bar_colors = [Color::Green, Color::Blue, Color::Yellow, Color::Magenta, Color::Cyan, Color::Red];
let lines: Vec<Line> = types
.iter()
.take(inner_height)
.enumerate()
.map(|(i, (name, count))| {
let short = name.rsplit("::").next().unwrap_or(name);
let pct = if total > 0 { *count as f64 / total as f64 * 100.0 } else { 0.0 };
let bar_width = 20usize;
let filled = ((*count as f64 / max_count as f64) * bar_width as f64).round() as usize;
let color = bar_colors[i % bar_colors.len()];
Line::from(vec![
Span::styled(
format!(" {:>16} ", short),
Style::default().fg(Color::White),
),
Span::styled(
"\u{2588}".repeat(filled),
Style::default().fg(color),
),
Span::styled(
" ".repeat(bar_width.saturating_sub(filled)),
Style::default(),
),
Span::styled(
format!(" {:>8} ", count),
Style::default().fg(Color::Cyan).add_modifier(Modifier::BOLD),
),
Span::styled(
format!("{:>5.1}%", pct),
Style::default().fg(Color::DarkGray),
),
])
})
.collect();
let block = Block::default()
.borders(Borders::ALL)
.title(format!(" Message Types ({}) ", types.len()));
let paragraph = Paragraph::new(lines).block(block);
f.render_widget(paragraph, area);
}
fn draw_actor_logs(f: &mut Frame, app: &App, area: Rect) {
let visible = app.visible_logs();
let log_count = visible.len();
let inner_height = area.height.saturating_sub(2) as usize; // borders
// Compute scroll window
let scroll = app.log_scroll.min(log_count.saturating_sub(inner_height));
let lines: Vec<Line> = visible
.iter()
.skip(scroll)
.take(inner_height)
.map(|e| {
let level_color = match e.level.as_str() {
"ERROR" => Color::Red,
"WARN" => Color::Yellow,
"INFO" => Color::Blue,
"DEBUG" => Color::DarkGray,
"TRACE" => Color::DarkGray,
_ => Color::White,
};
let ts = {
let secs = e.timestamp_ms / 1000;
let ms = e.timestamp_ms % 1000;
let h = (secs / 3600) % 24;
let m = (secs / 60) % 60;
let s = secs % 60;
format!("{:02}:{:02}:{:02}.{:03}", h, m, s, ms)
};
Line::from(vec![
Span::styled(
format!(" {} ", ts),
Style::default().fg(Color::DarkGray),
),
Span::styled(
format!("{:<5} ", e.level),
Style::default().fg(level_color).add_modifier(Modifier::BOLD),
),
Span::styled(
e.message.clone(),
Style::default().fg(Color::White),
),
])
})
.collect();
let title = format!(" Logs ({}) ", log_count);
let block = Block::default()
.borders(Borders::ALL)
.title(title);
let paragraph = Paragraph::new(lines).block(block);
f.render_widget(paragraph, area);
}
// ─── Distribution View ──────────────────────────────────────────────────────
#[cfg(feature = "distribution")]

View file

@ -0,0 +1,392 @@
//! Automated anomaly detection for the runtime dashboard.
//!
//! Runs on each stats sample, comparing consecutive snapshots to detect
//! growing mailboxes, stalled actors, worker imbalance, and other conditions.
use std::collections::HashMap;
use swactor::actor::ActorAddress;
use swactor::stats::RuntimeStats;
/// Types of warnings the detector can produce.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, serde::Serialize)]
#[serde(rename_all = "snake_case")]
pub enum WarningType {
GrowingMailbox,
StalledActor,
PoisonedActor,
WorkerImbalance,
EmptyWorker,
MailboxOverflow,
}
/// Severity levels for warnings.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, serde::Serialize)]
#[serde(rename_all = "lowercase")]
pub enum Severity {
Low,
Medium,
High,
Critical,
}
/// An active warning.
#[derive(Debug, Clone, serde::Serialize)]
pub struct Warning {
pub warning_type: WarningType,
pub severity: Severity,
pub entity: String,
pub description: String,
}
/// Configuration for warning thresholds.
#[derive(Debug, Clone)]
pub struct WarningConfig {
/// Consecutive samples with increasing mailbox depth before warning.
pub growing_mailbox_threshold: usize,
/// Consecutive ticks with no message processing while mailbox > 0.
pub stalled_actor_threshold: usize,
/// A worker is "imbalanced" if it has > this ratio times the average load.
pub worker_imbalance_ratio: f64,
}
impl Default for WarningConfig {
fn default() -> Self {
Self {
growing_mailbox_threshold: 5,
stalled_actor_threshold: 10,
worker_imbalance_ratio: 2.0,
}
}
}
/// Per-actor tracking state.
struct ActorState {
prev_mailbox: usize,
prev_messages: u64,
growing_streak: usize,
stalled_streak: usize,
}
/// Warning detection engine. Call `check()` on each stats sample.
pub struct WarningDetector {
config: WarningConfig,
actors: HashMap<ActorAddress, ActorState>,
}
impl WarningDetector {
pub fn new(config: WarningConfig) -> Self {
Self {
config,
actors: HashMap::new(),
}
}
/// Analyze a stats snapshot and return all active warnings.
pub fn check(&mut self, stats: &RuntimeStats) -> Vec<Warning> {
let mut warnings = Vec::new();
// Track which actors are still alive
let mut live_addrs: std::collections::HashSet<ActorAddress> =
std::collections::HashSet::new();
for actor in &stats.actor_details {
live_addrs.insert(actor.address);
// Poisoned actor — immediate critical warning
if actor.poisoned {
warnings.push(Warning {
warning_type: WarningType::PoisonedActor,
severity: Severity::Critical,
entity: format!("{}", actor.address),
description: "Actor is poisoned (panicked)".to_string(),
});
}
let state = self.actors.entry(actor.address).or_insert(ActorState {
prev_mailbox: actor.mailbox_depth,
prev_messages: actor.messages_processed,
growing_streak: 0,
stalled_streak: 0,
});
// Growing mailbox detection
if actor.mailbox_depth > state.prev_mailbox && actor.mailbox_depth > 0 {
state.growing_streak += 1;
} else {
state.growing_streak = 0;
}
if state.growing_streak >= self.config.growing_mailbox_threshold {
warnings.push(Warning {
warning_type: WarningType::GrowingMailbox,
severity: Severity::Medium,
entity: format!("{}", actor.address),
description: format!(
"Mailbox growing for {} consecutive samples (depth: {})",
state.growing_streak, actor.mailbox_depth,
),
});
}
// Stalled actor detection
if actor.messages_processed == state.prev_messages && actor.mailbox_depth > 0 {
state.stalled_streak += 1;
} else {
state.stalled_streak = 0;
}
if state.stalled_streak >= self.config.stalled_actor_threshold {
warnings.push(Warning {
warning_type: WarningType::StalledActor,
severity: Severity::High,
entity: format!("{}", actor.address),
description: format!(
"No messages processed for {} ticks with {} pending",
state.stalled_streak, actor.mailbox_depth,
),
});
}
state.prev_mailbox = actor.mailbox_depth;
state.prev_messages = actor.messages_processed;
}
// Clean up dead actors
self.actors.retain(|addr, _| live_addrs.contains(addr));
// Mailbox overflow detection
for w in &stats.workers {
if w.messages_dropped > 0 {
warnings.push(Warning {
warning_type: WarningType::MailboxOverflow,
severity: Severity::Medium,
entity: format!("Worker {}", w.id),
description: format!("{} messages dropped", w.messages_dropped),
});
}
}
// Worker imbalance and empty worker detection
if stats.workers.len() > 1 {
let total_actors: usize = stats.workers.iter().map(|w| w.num_actors).sum();
let avg = total_actors as f64 / stats.workers.len() as f64;
for w in &stats.workers {
if avg > 0.0 && w.num_actors as f64 > avg * self.config.worker_imbalance_ratio {
warnings.push(Warning {
warning_type: WarningType::WorkerImbalance,
severity: Severity::Low,
entity: format!("Worker {}", w.id),
description: format!(
"{} actors vs {:.0} average ({:.1}x)",
w.num_actors, avg, w.num_actors as f64 / avg,
),
});
}
if w.num_actors == 0 && total_actors > 0 {
warnings.push(Warning {
warning_type: WarningType::EmptyWorker,
severity: Severity::Low,
entity: format!("Worker {}", w.id),
description: "Worker has no actors while others do".to_string(),
});
}
}
}
// Sort by severity (critical first)
warnings.sort_by(|a, b| b.severity.cmp(&a.severity));
warnings
}
}
#[cfg(test)]
mod tests {
use super::*;
use swactor::stats::{ActorInfo, WorkerInfo};
fn make_worker(id: usize, actors: usize, dropped: u64) -> WorkerInfo {
WorkerInfo {
id,
num_actors: actors,
mailbox_depth: 0,
messages_processed: 0,
local_sends: 0,
cross_sends: 0,
inbox_sends: 0,
type_mismatches: 0,
panics: 0,
messages_dropped: dropped,
restarts: 0,
stops: 0,
}
}
fn make_actor(id: u8, depth: usize, msgs: u64, poisoned: bool) -> ActorInfo {
ActorInfo {
address: ActorAddress([id; 32]),
worker_id: 0,
mailbox_depth: depth,
last_msg_type: None,
messages_processed: msgs,
poisoned,
message_type_counts: Vec::new(),
}
}
fn make_stats(workers: Vec<WorkerInfo>, actors: Vec<ActorInfo>) -> RuntimeStats {
RuntimeStats {
num_workers: workers.len(),
uptime_ms: 0,
actors: actors.iter().map(|a| (a.address, a.worker_id)).collect(),
workers,
actor_details: actors,
tick_timings: Vec::new(),
}
}
#[test]
fn poisoned_actor_triggers_critical_warning() {
let mut detector = WarningDetector::new(WarningConfig::default());
let stats = make_stats(
vec![make_worker(0, 1, 0)],
vec![make_actor(1, 0, 10, true)],
);
let warnings = detector.check(&stats);
assert!(warnings.iter().any(|w| w.warning_type == WarningType::PoisonedActor));
assert!(warnings.iter().any(|w| w.severity == Severity::Critical));
}
#[test]
fn growing_mailbox_triggers_after_threshold() {
let config = WarningConfig {
growing_mailbox_threshold: 3,
..Default::default()
};
let mut detector = WarningDetector::new(config);
// 4 samples with increasing mailbox: should trigger at sample 4
for depth in 1..=4 {
let stats = make_stats(
vec![make_worker(0, 1, 0)],
vec![make_actor(1, depth, 0, false)],
);
let warnings = detector.check(&stats);
if depth < 4 {
assert!(!warnings.iter().any(|w| w.warning_type == WarningType::GrowingMailbox),
"should not trigger at depth {}", depth);
} else {
assert!(warnings.iter().any(|w| w.warning_type == WarningType::GrowingMailbox),
"should trigger at depth {}", depth);
}
}
}
#[test]
fn growing_mailbox_resets_on_decrease() {
let config = WarningConfig {
growing_mailbox_threshold: 3,
..Default::default()
};
let mut detector = WarningDetector::new(config);
// Grow for 2 samples, then decrease, then grow again
for depth in [1, 2, 1, 2, 3, 4] {
let stats = make_stats(
vec![make_worker(0, 1, 0)],
vec![make_actor(1, depth, 0, false)],
);
detector.check(&stats);
}
// After 1,2 → streak=2; then 1 → streak=0; then 2,3,4 → streak=3 → triggers
let stats = make_stats(
vec![make_worker(0, 1, 0)],
vec![make_actor(1, 5, 0, false)],
);
let warnings = detector.check(&stats);
assert!(warnings.iter().any(|w| w.warning_type == WarningType::GrowingMailbox));
}
#[test]
fn stalled_actor_triggers_when_not_processing() {
let config = WarningConfig {
stalled_actor_threshold: 3,
..Default::default()
};
let mut detector = WarningDetector::new(config);
// Same messages_processed, nonzero mailbox for 4 ticks
for _ in 0..4 {
let stats = make_stats(
vec![make_worker(0, 1, 0)],
vec![make_actor(1, 5, 100, false)],
);
let warnings = detector.check(&stats);
// Last one should trigger
if warnings.iter().any(|w| w.warning_type == WarningType::StalledActor) {
return; // test passed
}
}
panic!("expected StalledActor warning");
}
#[test]
fn worker_imbalance_detected() {
let mut detector = WarningDetector::new(WarningConfig::default());
// Worker 0: 10 actors, Worker 1: 1 actor. Avg=5.5, ratio=10/5.5=1.8
// With ratio threshold 2.0, this should NOT trigger
let stats = make_stats(
vec![make_worker(0, 10, 0), make_worker(1, 1, 0)],
vec![],
);
let warnings = detector.check(&stats);
assert!(!warnings.iter().any(|w| w.warning_type == WarningType::WorkerImbalance));
// Worker 0: 20 actors, Worker 1: 1 actor. Avg=10.5, ratio=20/10.5=1.9 — still no
// Worker 0: 30 actors, Worker 1: 1 actor. Avg=15.5, ratio=30/15.5=1.9 — still no
// Worker 0: 100 actors, Worker 1: 1 actor. Avg=50.5, ratio=100/50.5=1.98 — almost
// Worker 0: 100 actors, Worker 1: 0 actor. Avg=50, ratio=100/50=2.0 — at threshold
let stats2 = make_stats(
vec![make_worker(0, 100, 0), make_worker(1, 1, 0)],
vec![],
);
let warnings2 = detector.check(&stats2);
// 100 / 50.5 = 1.98 — not > 2.0
assert!(!warnings2.iter().any(|w| w.warning_type == WarningType::WorkerImbalance));
// Now 200 vs 1: 200/100.5 = ~1.99 — still not. Let's do 300 vs 1: 300/150.5 = ~2.0
// Actually need > 2x. Let's do 50 vs 1: avg=25.5, ratio=50/25.5=1.96. Nope.
// 10 vs 1 vs 1: avg=4, ratio=10/4=2.5 — triggers!
let stats3 = make_stats(
vec![make_worker(0, 10, 0), make_worker(1, 1, 0), make_worker(2, 1, 0)],
vec![],
);
let warnings3 = detector.check(&stats3);
assert!(warnings3.iter().any(|w| w.warning_type == WarningType::WorkerImbalance));
}
#[test]
fn empty_worker_detected() {
let mut detector = WarningDetector::new(WarningConfig::default());
let stats = make_stats(
vec![make_worker(0, 5, 0), make_worker(1, 0, 0)],
vec![],
);
let warnings = detector.check(&stats);
assert!(warnings.iter().any(|w| w.warning_type == WarningType::EmptyWorker));
}
#[test]
fn mailbox_overflow_detected() {
let mut detector = WarningDetector::new(WarningConfig::default());
let stats = make_stats(
vec![make_worker(0, 1, 42)],
vec![],
);
let warnings = detector.check(&stats);
assert!(warnings.iter().any(|w| w.warning_type == WarningType::MailboxOverflow));
}
}

View file

@ -0,0 +1,28 @@
# Dashboard Improvements — Research Phase
## Summary
Researched 10 comparable monitoring/dashboard systems to inform swactor's dashboard improvement plan.
## Systems Analyzed
- **Actor runtimes**: Erlang Observer (GUI/CLI/Web), Akka Insights, Ray Dashboard, Orleans Dashboard
- **Async/runtime tools**: tokio-console, Lunatic
- **Message/infrastructure**: RabbitMQ Management, Consul UI, Nomad UI
- **Web frameworks**: Phoenix LiveDashboard
## Key Findings
1. **Time-series history** is table-stakes — every system provides it
2. **Actor detail drill-down** is universal (Observer has 6-tab process info, Orleans has grain state inspection)
3. **Search/filter** exists in every system
4. **Warning/anomaly detection** (tokio-console's lint system) is a high-value differentiator
5. **Topology visualization** (Consul golden metrics, Observer supervision tree) is rare but powerful
## Implementation Plan
8 feature stages defined (see `CLAUDE/notes/feature-stages/`):
1. Time-Series History Infrastructure
2. Actor Detail Drill-Down
3. Search and Filter
4. Per-Worker Utilization Visualization
5. Warning/Anomaly Detection
6. Actor-to-Actor Message Flow Topology
7. Per-Actor Logging
8. Per-Message-Type Breakdown

View file

@ -120,6 +120,8 @@ pub struct ActorSnapshot {
pub last_msg_type: Option<&'static str>,
pub messages_processed: u64,
pub poisoned: bool,
/// Per-message-type counts, sorted descending by count.
pub message_type_counts: Vec<(&'static str, u64)>,
}
/// Observer hook called by workers after productive ticks.
@ -150,6 +152,9 @@ pub struct ActorInfo {
/// Whether the actor has panicked and is no longer processing messages.
#[cfg_attr(feature = "serde", serde(default))]
pub poisoned: bool,
/// Per-message-type counts, sorted descending by count. Top 32 types.
#[cfg_attr(feature = "serde", serde(default))]
pub message_type_counts: Vec<(String, u64)>,
}
/// Snapshot of overall runtime state.

View file

@ -372,6 +372,8 @@ struct ActorSlot {
started: bool,
last_msg_type: Option<&'static str>,
messages_processed: u64,
/// Per-message-type counters (bounded to 32 entries).
msg_type_counts: HashMap<&'static str, u64>,
/// Per-actor mailbox capacity. 0 = unbounded.
mailbox_capacity: usize,
overflow_policy: MailboxOverflow,
@ -407,6 +409,7 @@ impl ActorPool {
started: false,
last_msg_type: None,
messages_processed: 0,
msg_type_counts: HashMap::new(),
mailbox_capacity: self.default_mailbox_capacity,
overflow_policy: self.default_overflow_policy,
});
@ -459,6 +462,9 @@ impl ActorPool {
continue;
}
#[cfg(feature = "tracing")]
let _actor_span = tracing::trace_span!("actor.tick", actor_addr = %addr).entered();
let ctx = Ctx::new(inner, addr);
// Call on_start once, before first message
@ -521,6 +527,10 @@ impl ActorPool {
Ok(Some(type_name)) => {
slot.last_msg_type = Some(type_name);
slot.messages_processed += 1;
// Track per-type counts (bounded to 32 distinct types)
if slot.msg_type_counts.len() < 32 || slot.msg_type_counts.contains_key(type_name) {
*slot.msg_type_counts.entry(type_name).or_insert(0) += 1;
}
}
}
count += 1;
@ -592,12 +602,16 @@ impl ActorPool {
pub fn mailbox_depths_into(&self, out: &mut Vec<ActorSnapshot>) {
out.clear();
out.extend(self.actors.iter().map(|(&addr, slot)| {
let mut type_counts: Vec<(&'static str, u64)> =
slot.msg_type_counts.iter().map(|(&k, &v)| (k, v)).collect();
type_counts.sort_by(|a, b| b.1.cmp(&a.1));
ActorSnapshot {
address: addr,
mailbox_depth: slot.mailbox.len(),
last_msg_type: slot.last_msg_type,
messages_processed: slot.messages_processed,
poisoned: slot.poisoned,
message_type_counts: type_counts,
}
}));
}

790
tests/actor_lifecycle.rs Normal file
View file

@ -0,0 +1,790 @@
//! Actor Lifecycle Tests — birth, life, death of individual actors.
//!
//! Covers: spawning, on_start, parent-child delegation, graceful stop,
//! panic isolation, dead actor cleanup, watching (ActorExited), and
//! monitoring (Down notifications).
mod common;
use common::*;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
// ── Local actors ────────────────────────────────────────────────────────────
/// Records lifecycle events to shared counters.
struct LifecycleActor {
started: Arc<AtomicUsize>,
stopped: Arc<AtomicUsize>,
handled: Arc<AtomicUsize>,
}
impl ActorInterface for LifecycleActor {
type Incoming = Ping;
type Response = Pong;
fn on_start(&mut self, _ctx: &Ctx) {
self.started.fetch_add(1, Ordering::Relaxed);
}
fn on_stop(&mut self, _ctx: &Ctx) {
self.stopped.fetch_add(1, Ordering::Relaxed);
}
fn handle(&mut self, ctx: &Ctx, msg: Ping) {
self.handled.fetch_add(1, Ordering::Relaxed);
let _ = ctx.send(msg.reply_to, Pong);
}
}
/// Stops itself after processing `stop_after` messages.
struct SelfStopActor {
count: usize,
stop_after: usize,
stopped: Arc<AtomicUsize>,
}
impl ActorInterface for SelfStopActor {
type Incoming = Forward;
type Response = Done;
fn on_stop(&mut self, _ctx: &Ctx) {
self.stopped.fetch_add(1, Ordering::Relaxed);
}
fn handle(&mut self, ctx: &Ctx, msg: Forward) {
self.count += 1;
let _ = ctx.send(msg.reply_to, Done(msg.value));
if self.count >= self.stop_after {
ctx.stop_self();
}
}
}
/// Sends a farewell Pong in on_stop.
struct FarewellActor {
farewell_to: ActorAddress,
}
impl ActorInterface for FarewellActor {
type Incoming = Ping;
type Response = Pong;
fn on_stop(&mut self, ctx: &Ctx) {
let _ = ctx.send(self.farewell_to, Pong);
}
fn handle(&mut self, ctx: &Ctx, msg: Ping) {
let _ = ctx.send(msg.reply_to, Pong);
}
}
/// Panics in on_start.
struct PanicOnStartActor {
handled: Arc<AtomicUsize>,
}
impl ActorInterface for PanicOnStartActor {
type Incoming = Ping;
type Response = Pong;
fn on_start(&mut self, _ctx: &Ctx) {
panic!("on_start panic");
}
fn handle(&mut self, _ctx: &Ctx, _msg: Ping) {
self.handled.fetch_add(1, Ordering::Relaxed);
}
}
/// Spawns a DoubleActor child, sends it work, then panics.
struct SpawnThenPanicActor;
impl ActorInterface for SpawnThenPanicActor {
type Incoming = Forward;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: Forward) {
let child = ctx.spawn(DoubleActor).unwrap();
let _ = ctx.send(child, Forward { value: msg.value, reply_to: msg.reply_to });
panic!("intentional panic after spawn+send");
}
}
/// Sends a Pong reply, then panics.
struct SendThenPanicActor;
impl ActorInterface for SendThenPanicActor {
type Incoming = Ping;
type Response = Pong;
fn handle(&mut self, ctx: &Ctx, msg: Ping) {
let _ = ctx.send(msg.reply_to, Pong);
panic!("intentional panic after send");
}
}
/// Processes `remaining_good` messages then panics.
struct PanicAfterNActor {
remaining_good: usize,
counter: Arc<AtomicUsize>,
}
impl ActorInterface for PanicAfterNActor {
type Incoming = Ping;
type Response = ();
fn handle(&mut self, _ctx: &Ctx, _msg: Ping) {
if self.remaining_good == 0 {
panic!("intentional delayed panic");
}
self.remaining_good -= 1;
self.counter.fetch_add(1, Ordering::SeqCst);
}
}
/// Stops on a trigger message.
struct StopOnTrigger(Arc<AtomicUsize>);
#[derive(Clone)]
struct Trigger(bool);
impl ActorInterface for StopOnTrigger {
type Incoming = Trigger;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: Trigger) {
self.0.fetch_add(1, Ordering::Relaxed);
if msg.0 {
ctx.stop_self();
}
}
}
/// Watches targets and counts exit notifications via on_actor_exit.
struct ExitWatcher {
exit_count: Arc<AtomicUsize>,
last_reason: Arc<std::sync::Mutex<Option<ExitReason>>>,
last_addr: Arc<std::sync::Mutex<Option<ActorAddress>>>,
}
#[derive(Clone)]
enum WatcherCmd {
WatchThis(ActorAddress),
UnwatchThis(ActorAddress),
}
impl ActorInterface for ExitWatcher {
type Incoming = WatcherCmd;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: WatcherCmd) {
match msg {
WatcherCmd::WatchThis(target) => ctx.watch(target),
WatcherCmd::UnwatchThis(target) => ctx.unwatch(target),
}
}
fn on_actor_exit(&mut self, _ctx: &Ctx, exited: ActorExited) {
self.exit_count.fetch_add(1, Ordering::SeqCst);
*self.last_reason.lock().unwrap() = Some(exited.reason);
*self.last_addr.lock().unwrap() = Some(exited.addr);
}
}
struct WatcherState {
exit_count: Arc<AtomicUsize>,
last_reason: Arc<std::sync::Mutex<Option<ExitReason>>>,
}
impl WatcherState {
fn count(&self) -> usize {
self.exit_count.load(Ordering::SeqCst)
}
fn last_reason(&self) -> Option<ExitReason> {
self.last_reason.lock().unwrap().clone()
}
}
fn new_exit_watcher() -> (ExitWatcher, WatcherState) {
let exit_count = Arc::new(AtomicUsize::new(0));
let last_reason = Arc::new(std::sync::Mutex::new(None));
let last_addr = Arc::new(std::sync::Mutex::new(None));
let state = WatcherState {
exit_count: exit_count.clone(),
last_reason: last_reason.clone(),
};
(
ExitWatcher { exit_count, last_reason, last_addr },
state,
)
}
/// Monitors a target and forwards Down to a reply address.
struct MonitorWatcherActor {
watch_target: ActorAddress,
reply_to: ActorAddress,
mref: Option<MonitorRef>,
}
impl ActorInterface for MonitorWatcherActor {
type Incoming = Down;
type Response = ();
fn on_start(&mut self, ctx: &Ctx) {
self.mref = Some(ctx.monitor(self.watch_target));
}
fn handle(&mut self, ctx: &Ctx, msg: Down) {
ctx.send(self.reply_to, msg).unwrap();
}
}
/// Demonitors on Ping.
struct DemonitorActor {
watch_target: ActorAddress,
mref: Option<MonitorRef>,
}
impl ActorInterface for DemonitorActor {
type Incoming = Ping;
type Response = ();
fn on_start(&mut self, ctx: &Ctx) {
self.mref = Some(ctx.monitor(self.watch_target));
}
fn handle(&mut self, ctx: &Ctx, _msg: Ping) {
if let Some(mref) = self.mref.take() {
ctx.demonitor(mref);
}
}
}
/// A silent actor that does nothing (target for watching tests).
struct Sleeper;
#[derive(Clone)]
struct Noop;
impl ActorInterface for Sleeper {
type Incoming = Noop;
type Response = ();
fn handle(&mut self, _ctx: &Ctx, _msg: Noop) {}
}
// ═══════════════════════════════════════════════════════════════════════════
// Tests
// ═══════════════════════════════════════════════════════════════════════════
/// Actors are spawned, on_start fires exactly once per instance before any
/// message, then state accumulates across messages.
#[test]
fn actor_from_birth_to_first_message() {
let started = Arc::new(AtomicUsize::new(0));
let stopped = Arc::new(AtomicUsize::new(0));
let handled = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
// Spawn one tracked actor + 4 more sharing the same counters
let addr = rt.spawn(LifecycleActor {
started: started.clone(),
stopped: stopped.clone(),
handled: handled.clone(),
}).unwrap();
for _ in 0..4 {
rt.spawn(LifecycleActor {
started: started.clone(),
stopped: stopped.clone(),
handled: handled.clone(),
}).unwrap();
}
// First tick: all 5 on_start fire, no messages processed yet
rt.tick();
assert_eq!(started.load(Ordering::Relaxed), 5, "on_start per instance");
assert_eq!(handled.load(Ordering::Relaxed), 0, "no messages before first send");
// Send 3 Increments to a CounterActor to verify state accumulation
let counter_addr = rt.spawn(CounterActor { count: 0 }).unwrap();
let count_inbox = rt.new_inbox::<Count>().unwrap();
for _ in 0..3 {
rt.send_to(counter_addr, Increment { reply_to: *count_inbox.addr() }).unwrap();
}
let replies = tick_and_drain(&rt, &count_inbox, 10);
assert_eq!(replies, vec![Count(1), Count(2), Count(3)], "state accumulates");
// on_start must not fire again on subsequent ticks
rt.tick();
rt.tick();
assert_eq!(started.load(Ordering::Relaxed), 5, "on_start not repeated");
// Verify the first actor still responds normally
rt.send_to(addr, Ping { reply_to: *inbox.addr() }).unwrap();
let reply = tick_until_recv(&rt, &inbox, 10);
assert!(reply.is_some(), "actor handles messages after on_start");
}
/// Delegation chains: parent spawns child, child spawns grandchild, fan-out
/// distributes work. Spawn+send interleaving in a single handler works.
#[test]
fn parent_child_delegation_and_spawn_chains() {
let rt = std_runtime(RuntimeConfig {
max_actors: 2000,
..Default::default()
});
// Act 1: DelegatorActor spawns child, forwards value 7 → Done(14)
let delegator = rt.spawn(DelegatorActor).unwrap();
let inbox = rt.new_inbox::<Done>().unwrap();
rt.send_to(delegator, Forward { value: 7, reply_to: *inbox.addr() }).unwrap();
let reply = tick_until_recv(&rt, &inbox, 20);
assert_eq!(reply, Some(Done(14)), "delegator child doubles value");
// Act 2: Chain of depth 20
let chain = rt.spawn(ChainActor).unwrap();
rt.send_to(chain, ChainMsg { remaining: 20, depth: 0, reply_to: *inbox.addr() }).unwrap();
let reply = tick_until_recv(&rt, &inbox, 200);
assert_eq!(reply, Some(Done(20)), "chain reaches depth 20");
// Act 3: Fan-out to 20 children
let fan = rt.spawn(FanOutActor).unwrap();
rt.send_to(fan, FanOut { count: 20, reply_to: *inbox.addr() }).unwrap();
let replies = tick_and_drain(&rt, &inbox, 50);
assert_eq!(replies.len(), 20, "all 20 fan-out children reply");
}
/// The full graceful-stop story: self-stop with on_stop, farewell messages,
/// external stop ordering vs pending messages, mid-mailbox stop trigger.
#[test]
fn graceful_stop_lifecycle() {
// --- Part A: SelfStopActor ---
let stopped = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Done>().unwrap();
let addr = rt.spawn(SelfStopActor {
count: 0,
stop_after: 3,
stopped: stopped.clone(),
}).unwrap();
for i in 0..5 {
let _ = rt.send_to(addr, Forward { value: i, reply_to: *inbox.addr() });
}
tick_n(&rt, 10);
let mut replies = Vec::new();
while let Some(Done(v)) = inbox.try_recv() {
replies.push(v);
}
assert_eq!(replies.len(), 3, "only 3 messages processed before self-stop");
assert_eq!(stopped.load(Ordering::Relaxed), 1, "on_stop fired");
assert!(rt.send_to(addr, Forward { value: 99, reply_to: *inbox.addr() }).is_err(),
"send to stopped actor fails");
// --- Part B: FarewellActor sends farewell in on_stop ---
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let addr = rt.spawn(FarewellActor { farewell_to: *inbox.addr() }).unwrap();
rt.tick();
rt.stop_actor(addr).unwrap();
tick_n(&rt, 5);
assert_eq!(inbox.try_recv(), Some(Pong), "farewell message delivered from on_stop");
// --- Part C: External stop after pending messages ---
let stopped = Arc::new(AtomicUsize::new(0));
let handled = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let addr = rt.spawn(LifecycleActor {
started: Arc::new(AtomicUsize::new(0)),
stopped: stopped.clone(),
handled: handled.clone(),
}).unwrap();
for _ in 0..10 {
let _ = rt.send_to(addr, Ping { reply_to: *inbox.addr() });
}
rt.stop_actor(addr).unwrap();
tick_n(&rt, 10);
assert_eq!(handled.load(Ordering::Relaxed), 10, "all pending messages processed before stop");
assert_eq!(stopped.load(Ordering::Relaxed), 1, "on_stop fires after messages");
// --- Part D: External stop before messages → 0 processed ---
let handled = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let addr = rt.spawn(LifecycleActor {
started: Arc::new(AtomicUsize::new(0)),
stopped: Arc::new(AtomicUsize::new(0)),
handled: handled.clone(),
}).unwrap();
rt.tick(); // on_start
rt.stop_actor(addr).unwrap();
for _ in 0..5 {
let _ = rt.send_to(addr, Ping { reply_to: *inbox.addr() });
}
tick_n(&rt, 10);
assert_eq!(handled.load(Ordering::Relaxed), 0, "stop before messages prevents processing");
// --- Part E: Mid-mailbox stop trigger ---
let processed = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let addr = rt.spawn(StopOnTrigger(processed.clone())).unwrap();
rt.tick();
rt.send_to(addr, Trigger(false)).unwrap();
rt.send_to(addr, Trigger(false)).unwrap();
rt.send_to(addr, Trigger(true)).unwrap(); // stop trigger
rt.send_to(addr, Trigger(false)).unwrap();
rt.send_to(addr, Trigger(false)).unwrap();
tick_n(&rt, 5);
assert_eq!(processed.load(Ordering::Relaxed), 3,
"only messages up to and including stop trigger processed");
assert!(rt.send_to(addr, Trigger(false)).is_err());
}
/// Panics are caught: healthy siblings survive, panicked actors are poisoned
/// and cleaned from stats/address map, mid-batch panic discards remaining,
/// child spawned before parent panic survives, message sent before panic is
/// delivered, bulk cleanup, on_start panic also poisons.
#[test]
fn panic_isolation_and_cleanup() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let count_inbox = rt.new_inbox::<Count>().unwrap();
// Spawn a healthy counter, a PanicActor, and a PanicOnStartActor
let good = rt.spawn(CounterActor { count: 0 }).unwrap();
let bad = rt.spawn(PanicActor).unwrap();
let bad_start_handled = Arc::new(AtomicUsize::new(0));
let bad_start = rt.spawn(PanicOnStartActor { handled: bad_start_handled.clone() }).unwrap();
// Trigger panics
rt.send_to(bad, PanicMsg).unwrap();
let _ = rt.send_to(bad_start, Ping { reply_to: *inbox.addr() });
tick_n(&rt, 10);
// Healthy actor still works
rt.send_to(good, Increment { reply_to: *count_inbox.addr() }).unwrap();
rt.send_to(good, Increment { reply_to: *count_inbox.addr() }).unwrap();
let replies = tick_and_drain(&rt, &count_inbox, 10);
assert_eq!(replies, vec![Count(1), Count(2)], "healthy actor unaffected by peer panics");
// Poisoned actors are cleaned from address map
assert!(rt.send_to(bad, PanicMsg).is_err(), "send to cleaned-up actor fails");
assert_eq!(bad_start_handled.load(Ordering::Relaxed), 0, "on_start panic prevents messages");
// Stats track panics vs stops separately
let stats = rt.stats();
let panics: u64 = stats.workers.iter().map(|w| w.panics).sum();
assert!(panics >= 2, "at least 2 panics recorded (PanicActor + PanicOnStartActor)");
// --- Mid-batch panic discards remaining ---
let counter = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let dummy = rt.new_inbox::<Pong>().unwrap();
let addr = rt.spawn(PanicAfterNActor { remaining_good: 2, counter: counter.clone() }).unwrap();
for _ in 0..5 {
rt.send_to(addr, Ping { reply_to: *dummy.addr() }).unwrap();
}
tick_n(&rt, 20);
assert_eq!(counter.load(Ordering::SeqCst), 2, "only messages before panic processed");
// --- Child spawned before parent panic survives ---
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Done>().unwrap();
let parent = rt.spawn(SpawnThenPanicActor).unwrap();
rt.send_to(parent, Forward { value: 5, reply_to: *inbox.addr() }).unwrap();
let reply = tick_until_recv(&rt, &inbox, 30);
assert_eq!(reply, Some(Done(10)), "child survives parent panic");
// --- Message sent before panic is delivered ---
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let addr = rt.spawn(SendThenPanicActor).unwrap();
rt.send_to(addr, Ping { reply_to: *inbox.addr() }).unwrap();
let reply = tick_until_recv(&rt, &inbox, 20);
assert!(reply.is_some(), "message sent before panic still delivered");
// --- Bulk cleanup: 20 panicking actors all cleaned ---
let rt = std_runtime(RuntimeConfig::default());
let mut addrs = Vec::new();
for _ in 0..20 {
addrs.push(rt.spawn(PanicActor).unwrap());
}
for &addr in &addrs {
let _ = rt.send_to(addr, PanicMsg);
}
tick_n(&rt, 10);
let stats = rt.stats();
assert_eq!(stats.workers[0].num_actors, 0, "all poisoned actors cleaned up");
}
/// Watch API contract: watchers are notified on death, unwatch cancels,
/// double-watch is idempotent, multiple watchers all notified,
/// runtime-level watch works.
#[test]
fn watch_notification_contract() {
let rt = std_runtime(RuntimeConfig::default());
// Spawn target + 3 watchers + 1 that unwatches
let target = rt.spawn(PanicActor).unwrap();
let (w1, s1) = new_exit_watcher();
let (w2, s2) = new_exit_watcher();
let (w3, s3) = new_exit_watcher();
let (w4, s4) = new_exit_watcher(); // will unwatch
let w1_addr = rt.spawn(w1).unwrap();
let w2_addr = rt.spawn(w2).unwrap();
let w3_addr = rt.spawn(w3).unwrap();
let w4_addr = rt.spawn(w4).unwrap();
// All watch the target
rt.send_to(w1_addr, WatcherCmd::WatchThis(target)).unwrap();
rt.send_to(w2_addr, WatcherCmd::WatchThis(target)).unwrap();
rt.send_to(w3_addr, WatcherCmd::WatchThis(target)).unwrap();
rt.send_to(w4_addr, WatcherCmd::WatchThis(target)).unwrap();
tick_n(&rt, 3);
// w2 double-watches (idempotent test)
rt.send_to(w2_addr, WatcherCmd::WatchThis(target)).unwrap();
tick_n(&rt, 3);
// w4 unwatches
rt.send_to(w4_addr, WatcherCmd::UnwatchThis(target)).unwrap();
tick_n(&rt, 3);
// Kill target
rt.send_to(target, PanicMsg).unwrap();
tick_n(&rt, 5);
assert_eq!(s1.count(), 1, "watcher 1 notified");
assert_eq!(s2.count(), 1, "double-watch still only one notification");
assert_eq!(s3.count(), 1, "watcher 3 notified");
assert_eq!(s4.count(), 0, "unwatched watcher not notified");
assert_eq!(s1.last_reason(), Some(ExitReason::Panicked));
// --- Runtime-level watch ---
let rt = std_runtime(RuntimeConfig::default());
let target = rt.spawn(PanicActor).unwrap();
let (w, s) = new_exit_watcher();
let w_addr = rt.spawn(w).unwrap();
tick_n(&rt, 2);
rt.watch(w_addr, target);
rt.send_to(target, PanicMsg).unwrap();
tick_n(&rt, 5);
assert_eq!(s.count(), 1, "runtime-level watch delivers notification");
}
/// Watch edge cases: watcher dies before target (no crash), self-watch (no
/// crash), watcher reacts to death by spawning a replacement.
#[test]
fn watch_edge_cases() {
// Watcher dies before target — no crash
let rt = std_runtime(RuntimeConfig::default());
let target = rt.spawn(PanicActor).unwrap();
let target2 = rt.spawn(PanicActor).unwrap();
rt.watch(target2, target);
tick_n(&rt, 3);
rt.send_to(target2, PanicMsg).unwrap(); // kill watcher first
tick_n(&rt, 5);
rt.send_to(target, PanicMsg).unwrap(); // kill target — no crash
tick_n(&rt, 5);
// Self-watch — no crash
let rt = std_runtime(RuntimeConfig::default());
let (w, _s) = new_exit_watcher();
let addr = rt.spawn(w).unwrap();
rt.send_to(addr, WatcherCmd::WatchThis(addr)).unwrap();
tick_n(&rt, 5);
// Watcher reacts to death by spawning replacement
let rt = std_runtime(RuntimeConfig::default());
let spawned = Arc::new(AtomicUsize::new(0));
struct SupervisorWatcher {
spawned_count: Arc<AtomicUsize>,
}
#[derive(Clone)]
enum SupCmd {
WatchThis(ActorAddress),
}
impl ActorInterface for SupervisorWatcher {
type Incoming = SupCmd;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: SupCmd) {
match msg {
SupCmd::WatchThis(target) => ctx.watch(target),
}
}
fn on_actor_exit(&mut self, ctx: &Ctx, _exited: ActorExited) {
let _ = ctx.spawn(Sleeper);
self.spawned_count.fetch_add(1, Ordering::SeqCst);
}
}
let target = rt.spawn(PanicActor).unwrap();
let sup = rt.spawn(SupervisorWatcher { spawned_count: spawned.clone() }).unwrap();
rt.send_to(sup, SupCmd::WatchThis(target)).unwrap();
tick_n(&rt, 3);
rt.send_to(target, PanicMsg).unwrap();
tick_n(&rt, 5);
assert_eq!(spawned.load(Ordering::SeqCst), 1, "watcher spawned replacement");
}
/// Monitor API contract: Down on stop (Normal) and panic (Panicked), multiple
/// monitors, demonitor cancels, dead watcher cleanup, stacked monitors,
/// external inbox, handle_down dispatch.
#[test]
fn monitor_death_notification_contract() {
let rt = std_runtime(RuntimeConfig::default());
// --- Stop → Down(Normal) ---
let inbox = rt.new_inbox::<Down>().unwrap();
let target = rt.spawn(PingPongActor).unwrap();
rt.spawn(MonitorWatcherActor {
watch_target: target,
reply_to: *inbox.addr(),
mref: None,
}).unwrap();
rt.tick();
rt.stop_actor(target).unwrap();
tick_n(&rt, 3);
let down = inbox.try_recv().expect("Down on graceful stop");
assert_eq!(down.addr, target);
assert_eq!(down.reason, StopReason::Normal);
// --- Panic → Down(Panicked) ---
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Down>().unwrap();
let target = rt.spawn(PanicActor).unwrap();
rt.spawn(MonitorWatcherActor {
watch_target: target,
reply_to: *inbox.addr(),
mref: None,
}).unwrap();
rt.tick();
rt.send_to(target, PanicMsg).unwrap();
tick_n(&rt, 3);
let down = inbox.try_recv().expect("Down on panic");
assert_eq!(down.reason, StopReason::Panicked);
// --- Multiple monitors ---
let rt = std_runtime(RuntimeConfig::default());
let inbox1 = rt.new_inbox::<Down>().unwrap();
let inbox2 = rt.new_inbox::<Down>().unwrap();
let target = rt.spawn(PingPongActor).unwrap();
rt.spawn(MonitorWatcherActor {
watch_target: target, reply_to: *inbox1.addr(), mref: None,
}).unwrap();
rt.spawn(MonitorWatcherActor {
watch_target: target, reply_to: *inbox2.addr(), mref: None,
}).unwrap();
rt.tick();
rt.stop_actor(target).unwrap();
tick_n(&rt, 3);
assert!(inbox1.try_recv().is_some(), "watcher 1 notified");
assert!(inbox2.try_recv().is_some(), "watcher 2 notified");
// --- Demonitor cancels ---
let rt = std_runtime(RuntimeConfig::default());
let down_inbox = rt.new_inbox::<Down>().unwrap();
let target = rt.spawn(PingPongActor).unwrap();
let watcher = rt.spawn(DemonitorActor { watch_target: target, mref: None }).unwrap();
rt.tick();
rt.send_to(watcher, Ping { reply_to: ActorAddress::default() }).unwrap();
rt.tick(); // demonitor
rt.stop_actor(target).unwrap();
tick_n(&rt, 3);
assert!(down_inbox.try_recv().is_none(), "demonitored: no Down delivered");
// --- Dead watcher cleaned up ---
let rt = std_runtime(RuntimeConfig::default());
let target = rt.spawn(PingPongActor).unwrap();
let watcher = rt.spawn(MonitorWatcherActor {
watch_target: target,
reply_to: ActorAddress::default(),
mref: None,
}).unwrap();
rt.tick();
rt.stop_actor(watcher).unwrap();
rt.tick(); // watcher dies
rt.stop_actor(target).unwrap();
tick_n(&rt, 3); // target dies — no crash trying to deliver to dead watcher
// --- Stacked monitors produce multiple notifications ---
struct DoubleMonitor {
target: ActorAddress,
reply_to: ActorAddress,
}
impl ActorInterface for DoubleMonitor {
type Incoming = Down;
type Response = ();
fn on_start(&mut self, ctx: &Ctx) {
ctx.monitor(self.target);
ctx.monitor(self.target);
}
fn handle(&mut self, ctx: &Ctx, msg: Down) {
ctx.send(self.reply_to, msg).unwrap();
}
}
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Down>().unwrap();
let target = rt.spawn(PingPongActor).unwrap();
rt.spawn(DoubleMonitor { target, reply_to: *inbox.addr() }).unwrap();
rt.tick();
rt.stop_actor(target).unwrap();
tick_n(&rt, 3);
assert!(inbox.try_recv().is_some(), "first Down from stacked monitor");
assert!(inbox.try_recv().is_some(), "second Down from stacked monitor");
assert!(inbox.try_recv().is_none(), "no more");
// --- handle_down dispatch ---
struct MonitoringTracker {
target: ActorAddress,
downs: Vec<Down>,
inbox: ActorAddress,
}
impl ActorInterface for MonitoringTracker {
type Incoming = Ping;
type Response = ();
fn on_start(&mut self, ctx: &Ctx) {
ctx.monitor(self.target);
}
fn handle(&mut self, ctx: &Ctx, _msg: Ping) {
let _ = ctx.send(self.inbox, Count(self.downs.len()));
}
fn handle_down(&mut self, _ctx: &Ctx, down: Down) {
self.downs.push(down);
}
}
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Count>().unwrap();
let target = rt.spawn(PanicActor).unwrap();
let tracker = rt.spawn(MonitoringTracker {
target,
downs: vec![],
inbox: *inbox.addr(),
}).unwrap();
rt.tick();
rt.send_to(target, PanicMsg).unwrap();
tick_n(&rt, 3);
rt.send_to(tracker, Ping { reply_to: ActorAddress::default() }).unwrap();
rt.tick();
assert_eq!(inbox.try_recv(), Some(Count(1)), "handle_down received exactly one Down");
// --- When Incoming=Down, handle_down is NOT called ---
struct DownAsIncoming {
target: ActorAddress,
inbox: ActorAddress,
}
impl ActorInterface for DownAsIncoming {
type Incoming = Down;
type Response = ();
fn on_start(&mut self, ctx: &Ctx) {
ctx.monitor(self.target);
}
fn handle(&mut self, ctx: &Ctx, msg: Down) {
let _ = ctx.send(self.inbox, msg);
}
fn handle_down(&mut self, _ctx: &Ctx, _down: Down) {
panic!("handle_down must not be called when Incoming=Down");
}
}
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Down>().unwrap();
let target = rt.spawn(PanicActor).unwrap();
rt.spawn(DownAsIncoming { target, inbox: *inbox.addr() }).unwrap();
rt.tick();
rt.send_to(target, PanicMsg).unwrap();
tick_n(&rt, 3);
let received = inbox.try_recv().expect("Down delivered via handle(), not handle_down");
assert_eq!(received.reason, StopReason::Panicked);
}

View file

@ -5,11 +5,14 @@
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
pub use swactor::actor::{ActorAddress, ActorInterface, Down, MonitorRef, StopReason};
pub use swactor::actor::{
ActorAddress, ActorExited, ActorInterface, Down, ExitReason, MonitorRef, StopReason,
};
pub use swactor::runtime::{Ctx, Inbox, MailboxOverflow, Runtime, RuntimeConfig};
pub use swactor_std::{
ChildSpec, CtxGroups, CtxMonitoring, CtxNaming, CtxTimers, RestartPolicy, Router,
RoutingStrategy, RuntimeGroups, RuntimeNaming, StdExtension, Supervisor, SupervisorStrategy,
ChildSpec, CtxGroups, CtxMonitoring, CtxNaming, CtxTimers, CtxWatching, RestartPolicy, Router,
RoutingStrategy, RuntimeGroups, RuntimeNaming, RuntimeWatching, StdExtension, Supervisor,
SupervisorStrategy,
};
// ── Messages ────────────────────────────────────────────────────────────────
@ -232,6 +235,13 @@ pub fn tick_until_recv<M: swactor::actor::Message>(
None
}
/// Tick exactly `n` times (no inbox polling).
pub fn tick_n(rt: &Runtime, n: usize) {
for _ in 0..n {
rt.tick();
}
}
/// Tick `n` times, then drain all messages from the inbox.
pub fn tick_and_drain<M: swactor::actor::Message>(
rt: &Runtime,

489
tests/message_delivery.rs Normal file
View file

@ -0,0 +1,489 @@
//! Message Delivery Tests — how data flows through the system.
//!
//! Covers: FIFO ordering, routing correctness at scale, delivery from within
//! handlers, address error handling, fairness/budgets, timers, and mailbox
//! backpressure policies.
mod common;
use common::*;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
// ── Local actors ────────────────────────────────────────────────────────────
/// Sends a countdown message to itself, then replies Done(0).
struct SelfSendActor;
#[derive(Clone)]
struct Countdown {
remaining: usize,
reply_to: ActorAddress,
}
impl ActorInterface for SelfSendActor {
type Incoming = Countdown;
type Response = Done;
fn handle(&mut self, ctx: &Ctx, msg: Countdown) {
if msg.remaining == 0 {
let _ = ctx.send(msg.reply_to, Done(0));
} else {
let _ = ctx.send(
ctx.self_addr(),
Countdown { remaining: msg.remaining - 1, reply_to: msg.reply_to },
);
}
}
}
/// Schedules a one-shot timer in on_start.
struct TimerStartActor {
target: ActorAddress,
delay_ticks: u64,
}
impl ActorInterface for TimerStartActor {
type Incoming = Ping;
type Response = Pong;
fn on_start(&mut self, ctx: &Ctx) {
ctx.send_after_ticks(self.target, Ping { reply_to: ctx.self_addr() }, self.delay_ticks);
}
fn handle(&mut self, _ctx: &Ctx, _msg: Ping) {}
}
/// Schedules a one-shot timer from a handler.
struct DelayPingPongActor;
impl ActorInterface for DelayPingPongActor {
type Incoming = Forward;
type Response = Done;
fn handle(&mut self, ctx: &Ctx, msg: Forward) {
ctx.send_after_ticks(msg.reply_to, Done(msg.value), 3);
}
}
/// Schedules an interval timer on start.
struct HeartbeatActor {
target: ActorAddress,
period: u64,
}
impl ActorInterface for HeartbeatActor {
type Incoming = Ping;
type Response = Pong;
fn on_start(&mut self, ctx: &Ctx) {
ctx.send_interval_ticks(self.target, Ping { reply_to: ctx.self_addr() }, self.period);
}
fn handle(&mut self, _ctx: &Ctx, _msg: Ping) {}
}
/// NumberedMsg/Reply for routing correctness tests.
#[derive(Clone)]
struct NumberedMsg {
n: usize,
reply_to: ActorAddress,
}
#[derive(Clone, Debug, PartialEq)]
struct NumberedReply {
from: ActorAddress,
n: usize,
}
struct NumberedActor;
impl ActorInterface for NumberedActor {
type Incoming = NumberedMsg;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: NumberedMsg) {
let _ = ctx.send(msg.reply_to, NumberedReply { from: ctx.self_addr(), n: msg.n });
}
}
/// Ring node for routing chain test.
#[derive(Clone)]
struct RingHop {
hops_remaining: usize,
final_dest: ActorAddress,
}
#[derive(Clone, Debug, PartialEq)]
struct RingDone(usize);
struct RingNode {
next: ActorAddress,
}
impl ActorInterface for RingNode {
type Incoming = RingHop;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: RingHop) {
if msg.hops_remaining == 0 {
let _ = ctx.send(msg.final_dest, RingDone(100));
} else {
let _ = ctx.send(self.next, RingHop {
hops_remaining: msg.hops_remaining - 1,
final_dest: msg.final_dest,
});
}
}
}
// ═══════════════════════════════════════════════════════════════════════════
// Tests
// ═══════════════════════════════════════════════════════════════════════════
/// Messages arrive in FIFO order even with small buffers, budget constraints,
/// and independent mailboxes isolate actors from each other.
#[test]
fn fifo_ordering_and_mailbox_isolation() {
// FIFO with small buffer and budget
let rt = std_runtime(RuntimeConfig {
channel_buffer_size: 1,
actor_message_budget: 8,
..Default::default()
});
let addr = rt.spawn(CounterActor { count: 0 }).unwrap();
let inbox = rt.new_inbox::<Count>().unwrap();
for _ in 0..100 {
rt.send_to(addr, Increment { reply_to: *inbox.addr() }).unwrap();
}
let replies: Vec<_> = tick_and_drain(&rt, &inbox, 50);
assert_eq!(replies.len(), 100, "all messages delivered");
for (i, reply) in replies.iter().enumerate() {
assert_eq!(*reply, Count(i + 1), "FIFO order preserved at position {i}");
}
// Mailbox isolation: 3 actors each get exactly their own messages
let rt = std_runtime(RuntimeConfig::default());
let mut inboxes = Vec::new();
for _ in 0..3 {
let addr = rt.spawn(PingPongActor).unwrap();
let inbox = rt.new_inbox::<Pong>().unwrap();
rt.send_to(addr, Ping { reply_to: *inbox.addr() }).unwrap();
inboxes.push(inbox);
}
tick_n(&rt, 10);
for (i, inbox) in inboxes.iter().enumerate() {
assert!(inbox.try_recv().is_some(), "actor {i} replied");
assert!(inbox.try_recv().is_none(), "actor {i} has exactly one reply");
}
}
/// 200 actors each get a unique numbered message and reply correctly.
/// A 100-hop ring traversal completes.
#[test]
fn message_routing_at_scale() {
// 200-actor numbered routing
let rt = std_runtime(RuntimeConfig {
max_actors: 300,
channel_buffer_size: 1024,
num_threads: 1,
..Default::default()
});
let inbox = rt.new_inbox::<NumberedReply>().unwrap();
let inbox_addr = *inbox.addr();
let mut addrs = Vec::new();
for _ in 0..200 {
addrs.push(rt.spawn(NumberedActor).unwrap());
}
rt.tick();
for (i, addr) in addrs.iter().enumerate() {
rt.send_to(*addr, NumberedMsg { n: i, reply_to: inbox_addr }).unwrap();
}
tick_n(&rt, 3);
let replies: Vec<NumberedReply> = std::iter::from_fn(|| inbox.try_recv()).collect();
assert_eq!(replies.len(), 200, "all 200 actors replied");
for (i, addr) in addrs.iter().enumerate() {
let reply = replies.iter().find(|r| r.n == i);
assert!(reply.is_some(), "missing reply for actor #{i}");
assert_eq!(reply.unwrap().from, *addr, "reply #{i} came from correct actor");
}
// 100-hop ring
let rt = std_runtime(RuntimeConfig {
max_actors: 200,
channel_buffer_size: 1024,
num_threads: 1,
..Default::default()
});
let inbox = rt.new_inbox::<RingDone>().unwrap();
let inbox_addr = *inbox.addr();
let mut ring_addrs = Vec::new();
let mut next = inbox_addr;
for _ in (0..100).rev() {
let addr = rt.spawn(RingNode { next }).unwrap();
ring_addrs.push(addr);
next = addr;
}
ring_addrs.reverse();
rt.tick();
rt.send_to(ring_addrs[0], RingHop { hops_remaining: 99, final_dest: inbox_addr }).unwrap();
let result = tick_until_recv(&rt, &inbox, 110);
assert_eq!(result, Some(RingDone(100)), "ring message traverses all 100 hops");
}
/// Messages sent in handlers are delivered: delegation, self-send chains,
/// rapid spawn+immediate-send, multiple inbox types coexist.
#[test]
fn delivery_from_within_handlers() {
let rt = std_runtime(RuntimeConfig::default());
// Delegation: spawn+send in handler
let delegator = rt.spawn(DelegatorActor).unwrap();
let inbox = rt.new_inbox::<Done>().unwrap();
rt.send_to(delegator, Forward { value: 5, reply_to: *inbox.addr() }).unwrap();
let reply = tick_until_recv(&rt, &inbox, 20);
assert_eq!(reply, Some(Done(10)), "child spawned during handler receives message");
// Self-send countdown of 20
let self_sender = rt.spawn(SelfSendActor).unwrap();
rt.send_to(self_sender, Countdown { remaining: 20, reply_to: *inbox.addr() }).unwrap();
let reply = tick_until_recv(&rt, &inbox, 50);
assert_eq!(reply, Some(Done(0)), "self-send chain completes");
// Multiple senders reach same actor
let counter = rt.spawn(CounterActor { count: 0 }).unwrap();
let inbox_a = rt.new_inbox::<Count>().unwrap();
let inbox_b = rt.new_inbox::<Count>().unwrap();
rt.send_to(counter, Increment { reply_to: *inbox_a.addr() }).unwrap();
rt.send_to(counter, Increment { reply_to: *inbox_b.addr() }).unwrap();
tick_n(&rt, 10);
assert!(inbox_a.try_recv().is_some());
assert_eq!(inbox_b.try_recv(), Some(Count(2)), "both senders reach same actor");
// 50 rapid spawn+immediate-send pairs
let rt = std_runtime(RuntimeConfig::default());
let pong_inbox = rt.new_inbox::<Pong>().unwrap();
for _ in 0..50 {
let addr = rt.spawn(PingPongActor).unwrap();
rt.send_to(addr, Ping { reply_to: *pong_inbox.addr() }).unwrap();
}
let replies = tick_and_drain(&rt, &pong_inbox, 50);
assert_eq!(replies.len(), 50, "all spawn+send pairs complete");
// Multiple inbox types coexist
let rt = std_runtime(RuntimeConfig::default());
let counter_addr = rt.spawn(CounterActor { count: 0 }).unwrap();
let pinger_addr = rt.spawn(PingPongActor).unwrap();
let count_inbox = rt.new_inbox::<Count>().unwrap();
let pong_inbox = rt.new_inbox::<Pong>().unwrap();
rt.send_to(counter_addr, Increment { reply_to: *count_inbox.addr() }).unwrap();
rt.send_to(pinger_addr, Ping { reply_to: *pong_inbox.addr() }).unwrap();
tick_n(&rt, 10);
assert_eq!(count_inbox.try_recv(), Some(Count(1)));
assert_eq!(pong_inbox.try_recv(), Some(Pong));
}
/// Sending to nonexistent address returns error, wrong type increments
/// type_mismatch counter.
#[test]
fn address_error_handling() {
let rt = std_runtime(RuntimeConfig::default());
// Nonexistent address
let bogus = ActorAddress::new_random();
assert!(rt.send_to(bogus, Pong).is_err(), "send to unknown address fails");
// Wrong type
let addr = rt.spawn(PingPongActor).unwrap();
rt.send_to(addr, Count(42)).unwrap(); // Count instead of Ping
rt.send_to(addr, Count(0)).unwrap();
rt.send_to(addr, Count(0)).unwrap();
tick_n(&rt, 10);
let stats = rt.stats();
let mismatches: u64 = stats.workers.iter().map(|w| w.type_mismatches).sum();
assert_eq!(mismatches, 3, "3 wrong-type messages counted as mismatches");
}
/// Budget fairness: hot actor doesn't starve cold actor, budget is respected
/// with self-sends, unlimited budget drains all.
#[test]
fn fairness_budget_prevents_starvation() {
// Hot (1000 msgs) vs cold (1 msg), budget=64
let rt = std_runtime(RuntimeConfig::default());
let hot_counter = Arc::new(AtomicUsize::new(0));
let cold_inbox = rt.new_inbox::<Pong>().unwrap();
let hot = rt.spawn(CountingPingActor { counter: hot_counter.clone() }).unwrap();
let cold = rt.spawn(PingPongActor).unwrap();
let dummy = rt.new_inbox::<Pong>().unwrap();
for _ in 0..1000 {
rt.send_to(hot, Ping { reply_to: *dummy.addr() }).unwrap();
}
rt.send_to(cold, Ping { reply_to: *cold_inbox.addr() }).unwrap();
rt.tick();
assert!(cold_inbox.try_recv().is_some(), "cold actor not starved by hot actor");
assert!(hot_counter.load(Ordering::SeqCst) <= 64, "hot capped at budget");
// Budget=4 with self-send chain of 20 → completes across multiple ticks
let rt = std_runtime(RuntimeConfig { actor_message_budget: 4, ..Default::default() });
let addr = rt.spawn(SelfSendActor).unwrap();
let inbox = rt.new_inbox::<Done>().unwrap();
rt.send_to(addr, Countdown { remaining: 20, reply_to: *inbox.addr() }).unwrap();
tick_n(&rt, 30);
assert_eq!(inbox.try_recv(), Some(Done(0)), "self-send chain completes despite budget");
// Unlimited budget (0) drains all
let rt = std_runtime(RuntimeConfig { actor_message_budget: 0, ..Default::default() });
let counter = Arc::new(AtomicUsize::new(0));
let dummy = rt.new_inbox::<Pong>().unwrap();
let addr = rt.spawn(CountingPingActor { counter: counter.clone() }).unwrap();
for _ in 0..500 {
rt.send_to(addr, Ping { reply_to: *dummy.addr() }).unwrap();
}
rt.tick();
rt.tick();
assert_eq!(counter.load(Ordering::SeqCst), 500, "unlimited budget drains all");
}
/// One-shot timers fire at the right tick and only once. Interval timers fire
/// repeatedly at the right period. Timers are cleaned up when actors die.
#[test]
fn timer_one_shot_and_interval() {
// One-shot: delay=3 from on_start
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Ping>().unwrap();
rt.spawn(TimerStartActor { target: *inbox.addr(), delay_ticks: 3 }).unwrap();
rt.tick(); // tick 1: on_start schedules
assert!(inbox.try_recv().is_none(), "no delivery tick 1");
rt.tick(); // tick 2
assert!(inbox.try_recv().is_none(), "no delivery tick 2");
rt.tick(); // tick 3
assert!(inbox.try_recv().is_none(), "no delivery tick 3");
rt.tick(); // tick 4: fires
assert!(inbox.try_recv().is_some(), "timer fires after 3-tick delay");
// One-shot from handler
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Done>().unwrap();
let addr = rt.spawn(DelayPingPongActor).unwrap();
rt.send_to(addr, Forward { value: 42, reply_to: *inbox.addr() }).unwrap();
rt.tick(); // process Forward, schedule timer
assert!(inbox.try_recv().is_none());
rt.tick(); // tick 2
rt.tick(); // tick 3
assert!(inbox.try_recv().is_none());
rt.tick(); // tick 4: fires
assert_eq!(inbox.try_recv(), Some(Done(42)), "delayed reply from handler timer");
// One-shot does NOT repeat
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Ping>().unwrap();
rt.spawn(TimerStartActor { target: *inbox.addr(), delay_ticks: 1 }).unwrap();
rt.tick(); // schedule
rt.tick(); // fires
assert!(inbox.try_recv().is_some(), "first fire");
tick_n(&rt, 5);
assert!(inbox.try_recv().is_none(), "one-shot doesn't repeat");
// Zero-delay fires next tick
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Ping>().unwrap();
rt.spawn(TimerStartActor { target: *inbox.addr(), delay_ticks: 0 }).unwrap();
rt.tick(); // schedule
assert!(inbox.try_recv().is_none(), "not immediate — fires next tick");
rt.tick(); // fires
assert!(inbox.try_recv().is_some(), "zero-delay fires next tick");
// Interval: period=2, fires on ticks 3, 5, 7
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Ping>().unwrap();
rt.spawn(HeartbeatActor { target: *inbox.addr(), period: 2 }).unwrap();
rt.tick(); // tick 1: schedule
assert!(inbox.try_recv().is_none());
rt.tick(); // tick 2
assert!(inbox.try_recv().is_none());
rt.tick(); // tick 3: first fire
assert!(inbox.try_recv().is_some(), "fire on tick 3");
rt.tick(); // tick 4
assert!(inbox.try_recv().is_none());
rt.tick(); // tick 5: second fire
assert!(inbox.try_recv().is_some(), "fire on tick 5");
rt.tick(); // tick 6
assert!(inbox.try_recv().is_none());
rt.tick(); // tick 7: third fire
assert!(inbox.try_recv().is_some(), "fire on tick 7");
// Timer cleanup when target actor dies
let rt = std_runtime(RuntimeConfig::default());
let counter_addr = rt.spawn(CounterActor { count: 0 }).unwrap();
rt.spawn(HeartbeatActor { target: counter_addr, period: 1 }).unwrap();
tick_n(&rt, 3);
rt.stop_actor(counter_addr).unwrap();
tick_n(&rt, 5);
let stats = rt.stats();
assert_eq!(stats.workers[0].num_actors, 1, "only heartbeat actor remains");
}
/// Bounded mailboxes: DropNewest caps at capacity, DropOldest keeps newest,
/// unbounded delivers all, mailbox refills after processing.
#[test]
fn mailbox_backpressure_policies() {
// DropNewest: capacity=10, send 50 → only 10 delivered
let rt = std_runtime(RuntimeConfig {
default_mailbox_capacity: 10,
mailbox_overflow: MailboxOverflow::DropNewest,
..Default::default()
});
let inbox = rt.new_inbox::<Count>().unwrap();
let addr = rt.spawn(CounterActor { count: 0 }).unwrap();
for _ in 0..50 {
let _ = rt.send_to(addr, Increment { reply_to: *inbox.addr() });
}
tick_n(&rt, 20);
let mut replies = 0;
while inbox.try_recv().is_some() { replies += 1; }
assert_eq!(replies, 10, "DropNewest caps at mailbox capacity");
let drops: u64 = rt.stats().workers.iter().map(|w| w.messages_dropped).sum();
assert_eq!(drops, 40, "40 messages dropped");
// DropOldest: capacity=5, send 10 → newest 5 kept
let rt = std_runtime(RuntimeConfig {
default_mailbox_capacity: 5,
mailbox_overflow: MailboxOverflow::DropOldest,
..Default::default()
});
let inbox = rt.new_inbox::<Done>().unwrap();
let addr = rt.spawn(DoubleActor).unwrap();
for i in 0..10 {
let _ = rt.send_to(addr, Forward { value: i, reply_to: *inbox.addr() });
}
tick_n(&rt, 10);
let mut replies = Vec::new();
while let Some(Done(v)) = inbox.try_recv() { replies.push(v); }
assert_eq!(replies.len(), 5, "only 5 kept");
assert_eq!(replies, vec![10, 12, 14, 16, 18], "newest values kept (5-9 doubled)");
// Unbounded: 200 messages all delivered
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Count>().unwrap();
let addr = rt.spawn(CounterActor { count: 0 }).unwrap();
for _ in 0..200 {
let _ = rt.send_to(addr, Increment { reply_to: *inbox.addr() });
}
tick_n(&rt, 50);
let mut count = 0;
while inbox.try_recv().is_some() { count += 1; }
assert_eq!(count, 200, "unbounded delivers all");
// Refill after processing
let rt = std_runtime(RuntimeConfig {
default_mailbox_capacity: 5,
actor_message_budget: 5,
mailbox_overflow: MailboxOverflow::DropNewest,
..Default::default()
});
let inbox = rt.new_inbox::<Count>().unwrap();
let addr = rt.spawn(CounterActor { count: 0 }).unwrap();
for _ in 0..5 {
let _ = rt.send_to(addr, Increment { reply_to: *inbox.addr() });
}
rt.tick(); // process batch 1
for _ in 0..5 {
let _ = rt.send_to(addr, Increment { reply_to: *inbox.addr() });
}
rt.tick(); // process batch 2
let mut count = 0;
while inbox.try_recv().is_some() { count += 1; }
assert_eq!(count, 10, "mailbox refills after draining");
}

View file

@ -1,224 +0,0 @@
mod common;
use common::*;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
// ── Identity Hasher Correctness ────────────────────────────────────────────
/// Given: 200 actors each expecting a unique numbered message
/// When: Each actor receives its number and replies with (self_addr, number)
/// Then: All 200 replies match -- no message was misrouted by the identity hasher
#[test]
fn many_actors_all_receive_correct_messages() {
#[derive(Clone)]
struct NumberedMsg {
n: usize,
reply_to: ActorAddress,
}
#[derive(Clone, Debug, PartialEq)]
struct NumberedReply {
from: ActorAddress,
n: usize,
}
struct NumberedActor;
impl ActorInterface for NumberedActor {
type Incoming = NumberedMsg;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: NumberedMsg) {
let _ = ctx.send(
msg.reply_to,
NumberedReply {
from: ctx.self_addr(),
n: msg.n,
},
);
}
}
let rt = std_runtime(RuntimeConfig {
max_actors: 300,
channel_buffer_size: 1024,
num_threads: 1,
..Default::default()
});
let inbox = rt.new_inbox::<NumberedReply>().unwrap();
let inbox_addr = *inbox.addr();
// Spawn 200 actors
let mut addrs = Vec::new();
for _ in 0..200 {
addrs.push(rt.spawn(NumberedActor).unwrap());
}
rt.tick(); // on_start
// Send unique numbered message to each
for (i, addr) in addrs.iter().enumerate() {
rt.send_to(
*addr,
NumberedMsg {
n: i,
reply_to: inbox_addr,
},
)
.unwrap();
}
rt.tick(); // process + reply
rt.tick(); // deliver replies
// Verify all 200 replies
let mut replies: Vec<NumberedReply> = Vec::new();
while let Some(reply) = inbox.try_recv() {
replies.push(reply);
}
assert_eq!(replies.len(), 200, "should receive exactly 200 replies");
// Verify each reply came from the correct actor with the correct number
for (i, addr) in addrs.iter().enumerate() {
let reply = replies.iter().find(|r| r.n == i);
assert!(
reply.is_some(),
"missing reply for actor #{i}"
);
assert_eq!(
reply.unwrap().from, *addr,
"reply #{i} came from wrong actor"
);
}
}
/// Given: A 100-actor ring where each actor forwards to the next
/// When: A message enters the ring and traverses all 100 hops
/// Then: The message completes the full circuit (address_map lookups all correct)
#[test]
fn ring_routing_unchanged_after_hasher_optimization() {
#[derive(Clone)]
struct RingHop {
hops_remaining: usize,
final_dest: ActorAddress,
}
#[derive(Clone, Debug, PartialEq)]
struct RingDone(usize); // total hops completed
struct RingNode {
next: ActorAddress,
}
impl ActorInterface for RingNode {
type Incoming = RingHop;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: RingHop) {
if msg.hops_remaining == 0 {
let _ = ctx.send(msg.final_dest, RingDone(100));
} else {
let _ = ctx.send(
self.next,
RingHop {
hops_remaining: msg.hops_remaining - 1,
final_dest: msg.final_dest,
},
);
}
}
}
let rt = std_runtime(RuntimeConfig {
max_actors: 200,
channel_buffer_size: 1024,
num_threads: 1,
..Default::default()
});
let inbox = rt.new_inbox::<RingDone>().unwrap();
let inbox_addr = *inbox.addr();
// Build chain backwards: last node sends to inbox, first node receives
let mut addrs = Vec::new();
let mut next = inbox_addr;
for _ in (0..100).rev() {
let node = RingNode { next };
let addr = rt.spawn(node).unwrap();
addrs.push(addr);
next = addr;
}
addrs.reverse(); // addrs[0] is start of chain
rt.tick(); // on_start
// Inject message at the start
rt.send_to(
addrs[0],
RingHop {
hops_remaining: 99,
final_dest: inbox_addr,
},
)
.unwrap();
// Tick enough times for the message to traverse all 100 actors
for _ in 0..110 {
rt.tick();
}
let result = inbox.try_recv();
assert!(result.is_some(), "ring message should complete all 100 hops");
assert_eq!(result.unwrap(), RingDone(100));
}
/// Given: An actor that calls ctx.stop_self() upon receiving a trigger message
/// When: The trigger is sent, then 5 more messages are sent, then ticked
/// Then: The actor is removed, only messages before stop are processed
#[test]
fn stop_self_with_pending_messages_still_works() {
let processed = Arc::new(AtomicUsize::new(0));
#[derive(Clone)]
struct Msg(bool); // true = trigger stop
struct StopOnTrigger(Arc<AtomicUsize>);
impl ActorInterface for StopOnTrigger {
type Incoming = Msg;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: Msg) {
self.0.fetch_add(1, Ordering::Relaxed);
if msg.0 {
ctx.stop_self();
}
}
}
let rt = std_runtime(RuntimeConfig::default());
let p = processed.clone();
let addr = rt.spawn(StopOnTrigger(p)).unwrap();
rt.tick(); // on_start
// Send: 2 normal, 1 trigger, 5 more normal
rt.send_to(addr, Msg(false)).unwrap();
rt.send_to(addr, Msg(false)).unwrap();
rt.send_to(addr, Msg(true)).unwrap(); // stop trigger
rt.send_to(addr, Msg(false)).unwrap();
rt.send_to(addr, Msg(false)).unwrap();
rt.send_to(addr, Msg(false)).unwrap();
rt.send_to(addr, Msg(false)).unwrap();
rt.send_to(addr, Msg(false)).unwrap();
rt.tick(); // process messages -- stops after trigger
rt.tick(); // cleanup
// Only 3 messages should be processed (2 normal + 1 trigger)
assert_eq!(
processed.load(Ordering::Relaxed),
3,
"should process exactly the messages up to and including the stop trigger"
);
// Subsequent sends should fail
assert!(rt.send_to(addr, Msg(false)).is_err());
}

View file

@ -1,454 +0,0 @@
mod common;
use common::*;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
// ── Lifecycle Hook Helpers ────────────────────────────────────────────────
/// An actor that records lifecycle events to shared counters.
struct LifecycleActor {
started: Arc<AtomicUsize>,
stopped: Arc<AtomicUsize>,
handled: Arc<AtomicUsize>,
}
impl ActorInterface for LifecycleActor {
type Incoming = Ping;
type Response = Pong;
fn on_start(&mut self, _ctx: &Ctx) {
self.started.fetch_add(1, Ordering::Relaxed);
}
fn on_stop(&mut self, _ctx: &Ctx) {
self.stopped.fetch_add(1, Ordering::Relaxed);
}
fn handle(&mut self, ctx: &Ctx, msg: Ping) {
self.handled.fetch_add(1, Ordering::Relaxed);
let _ = ctx.send(msg.reply_to, Pong);
}
}
/// An actor that stops itself after processing N messages.
struct SelfStopActor {
count: usize,
stop_after: usize,
stopped: Arc<AtomicUsize>,
}
impl ActorInterface for SelfStopActor {
type Incoming = Forward;
type Response = Done;
fn on_stop(&mut self, _ctx: &Ctx) {
self.stopped.fetch_add(1, Ordering::Relaxed);
}
fn handle(&mut self, ctx: &Ctx, msg: Forward) {
self.count += 1;
let _ = ctx.send(msg.reply_to, Done(msg.value));
if self.count >= self.stop_after {
ctx.stop_self();
}
}
}
/// An actor that sends a farewell message in on_stop.
struct FarewellActor {
farewell_to: ActorAddress,
}
impl ActorInterface for FarewellActor {
type Incoming = Ping;
type Response = Pong;
fn on_stop(&mut self, ctx: &Ctx) {
let _ = ctx.send(self.farewell_to, Pong);
}
fn handle(&mut self, ctx: &Ctx, msg: Ping) {
let _ = ctx.send(msg.reply_to, Pong);
}
}
/// An actor whose on_start panics.
struct PanicOnStartActor {
handled: Arc<AtomicUsize>,
}
impl ActorInterface for PanicOnStartActor {
type Incoming = Ping;
type Response = Pong;
fn on_start(&mut self, _ctx: &Ctx) {
panic!("on_start panic");
}
fn handle(&mut self, _ctx: &Ctx, _msg: Ping) {
self.handled.fetch_add(1, Ordering::Relaxed);
}
}
/// Handles Forward messages, replies Done(value * 2), panics on the panic_at-th message.
/// (Needed for on_start_called_again_after_restart and stop_vs_panic tests.)
struct RestartTestActor {
count: usize,
panic_at: usize,
}
impl ActorInterface for RestartTestActor {
type Incoming = Forward;
type Response = Done;
fn handle(&mut self, ctx: &Ctx, msg: Forward) {
self.count += 1;
if self.count >= self.panic_at {
panic!("intentional panic at message {}", self.count);
}
let _ = ctx.send(msg.reply_to, Done(msg.value * 2));
}
}
// ── Lifecycle Hook Tests ──────────────────────────────────────────────────
/// Given an actor with on_start implemented,
/// when it is spawned and the runtime ticks,
/// then on_start is called exactly once before the first message.
#[test]
fn on_start_called_before_first_message() {
let started = Arc::new(AtomicUsize::new(0));
let stopped = Arc::new(AtomicUsize::new(0));
let handled = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let addr = rt.spawn(LifecycleActor {
started: started.clone(),
stopped: stopped.clone(),
handled: handled.clone(),
}).unwrap();
// First tick — should call on_start
rt.tick();
assert_eq!(started.load(Ordering::Relaxed), 1, "on_start called on first tick");
assert_eq!(handled.load(Ordering::Relaxed), 0, "no messages processed yet");
// Send messages and tick more
let _ = rt.send_to(addr, Ping { reply_to: *inbox.addr() });
rt.tick();
assert_eq!(started.load(Ordering::Relaxed), 1, "on_start not called again");
assert_eq!(handled.load(Ordering::Relaxed), 1, "message processed after on_start");
}
/// Given an actor with on_start,
/// when multiple actors are spawned,
/// then each gets its own on_start call exactly once.
#[test]
fn on_start_called_per_actor() {
let started = Arc::new(AtomicUsize::new(0));
let stopped = Arc::new(AtomicUsize::new(0));
let handled = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
for _ in 0..5 {
let _ = rt.spawn(LifecycleActor {
started: started.clone(),
stopped: stopped.clone(),
handled: handled.clone(),
}).unwrap();
}
rt.tick();
assert_eq!(started.load(Ordering::Relaxed), 5, "on_start called for each of 5 actors");
// Subsequent ticks don't repeat on_start
rt.tick();
rt.tick();
assert_eq!(started.load(Ordering::Relaxed), 5, "on_start still 5 after more ticks");
}
/// Given an actor whose on_start panics,
/// when it is spawned and the runtime ticks,
/// then it is immediately poisoned and never processes messages.
#[test]
fn on_start_panic_poisons_actor() {
let handled = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let addr = rt.spawn(PanicOnStartActor { handled: handled.clone() }).unwrap();
let _ = rt.send_to(addr, Ping { reply_to: *inbox.addr() });
for _ in 0..5 { rt.tick(); }
assert_eq!(handled.load(Ordering::Relaxed), 0, "actor never processed messages");
let stats = rt.stats();
let total_panics: u64 = stats.workers.iter().map(|w| w.panics).sum();
assert_eq!(total_panics, 1, "on_start panic counted");
}
// ── Graceful Stop Tests ───────────────────────────────────────────────────
/// Given an actor that calls ctx.stop_self() after 3 messages,
/// when 5 messages are sent,
/// then only 3 are processed, the actor is removed, and on_stop is called.
#[test]
fn actor_can_stop_self() {
let stopped = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Done>().unwrap();
let addr = rt.spawn(SelfStopActor {
count: 0,
stop_after: 3,
stopped: stopped.clone(),
}).unwrap();
for i in 0..5 {
let _ = rt.send_to(addr, Forward { value: i, reply_to: *inbox.addr() });
}
for _ in 0..10 { rt.tick(); }
// Only 3 messages should be processed (stop_self after 3rd)
let mut replies = Vec::new();
while let Some(Done(v)) = inbox.try_recv() {
replies.push(v);
}
assert_eq!(replies.len(), 3, "only 3 messages processed before stop");
assert!(replies.contains(&0));
assert!(replies.contains(&1));
assert!(replies.contains(&2));
assert_eq!(stopped.load(Ordering::Relaxed), 1, "on_stop called exactly once");
// Actor should be removed from address map
let stats = rt.stats();
assert_eq!(stats.actors.len(), 0, "stopped actor removed from address map");
}
/// Given a running actor,
/// when runtime.stop_actor(addr) is called,
/// then the actor stops, on_stop is called, and it's removed from the pool.
#[test]
fn runtime_can_stop_actor() {
let started = Arc::new(AtomicUsize::new(0));
let stopped = Arc::new(AtomicUsize::new(0));
let handled = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let addr = rt.spawn(LifecycleActor {
started: started.clone(),
stopped: stopped.clone(),
handled: handled.clone(),
}).unwrap();
// Let it start and process a message
let _ = rt.send_to(addr, Ping { reply_to: *inbox.addr() });
for _ in 0..3 { rt.tick(); }
assert_eq!(handled.load(Ordering::Relaxed), 1);
// Stop it externally
rt.stop_actor(addr).unwrap();
for _ in 0..3 { rt.tick(); }
assert_eq!(stopped.load(Ordering::Relaxed), 1, "on_stop called");
// Actor should be gone
let stats = rt.stats();
assert_eq!(stats.actors.len(), 0, "stopped actor removed");
assert_eq!(stats.workers[0].num_actors, 0);
}
/// Given a stopped actor,
/// when new messages are sent to it,
/// then sends return Err (address not found).
#[test]
fn send_to_stopped_actor_returns_error() {
let stopped = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Done>().unwrap();
let addr = rt.spawn(SelfStopActor {
count: 0,
stop_after: 1,
stopped: stopped.clone(),
}).unwrap();
// One message triggers stop
let _ = rt.send_to(addr, Forward { value: 1, reply_to: *inbox.addr() });
for _ in 0..10 { rt.tick(); }
// Actor is now removed — send should fail
let result = rt.send_to(addr, Forward { value: 2, reply_to: *inbox.addr() });
assert!(result.is_err(), "send to stopped actor should return Err");
}
/// Given a gracefully stopped actor and a panicked actor,
/// then stats.stops and stats.panics track them separately.
#[test]
fn stop_vs_panic_tracked_separately_in_stats() {
let stopped = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Done>().unwrap();
// Actor that stops itself after 1 message
let _stop_addr = rt.spawn(SelfStopActor {
count: 0,
stop_after: 1,
stopped: stopped.clone(),
}).unwrap();
// Actor that panics on first message
let panic_addr = rt.spawn(RestartTestActor { count: 0, panic_at: 1 }).unwrap();
let _ = rt.send_to(_stop_addr, Forward { value: 1, reply_to: *inbox.addr() });
let _ = rt.send_to(panic_addr, Forward { value: 1, reply_to: *inbox.addr() });
for _ in 0..10 { rt.tick(); }
let stats = rt.stats();
let total_stops: u64 = stats.workers.iter().map(|w| w.stops).sum();
let total_panics: u64 = stats.workers.iter().map(|w| w.panics).sum();
assert_eq!(total_stops, 1, "one graceful stop");
assert_eq!(total_panics, 1, "one panic");
}
/// Given an actor with on_stop that sends a farewell message,
/// when the actor is stopped,
/// then the farewell message is delivered.
#[test]
fn on_stop_can_send_messages() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let addr = rt.spawn(FarewellActor {
farewell_to: *inbox.addr(),
}).unwrap();
// Let it start
rt.tick();
// Stop it
rt.stop_actor(addr).unwrap();
for _ in 0..5 { rt.tick(); }
// Should receive farewell Pong from on_stop
let farewell = inbox.try_recv();
assert_eq!(farewell, Some(Pong), "farewell message delivered from on_stop");
}
/// Given a supervisor with a child that panics and is restarted,
/// when the child is respawned by the supervisor,
/// then on_start is called again on the fresh instance.
#[test]
fn on_start_called_again_after_restart() {
let started = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let started_c = started.clone();
let _sup_addr = rt.spawn(Supervisor::new(
SupervisorStrategy::OneForOne,
5,
vec![ChildSpec::new("child", RestartPolicy::Permanent, move |ctx| {
ctx.spawn(LifecycleActor {
started: started_c.clone(),
stopped: Arc::new(AtomicUsize::new(0)),
handled: Arc::new(AtomicUsize::new(0)),
})
})],
)).unwrap();
// First tick: supervisor starts, spawns child, on_start called
for _ in 0..3 { rt.tick(); }
assert_eq!(started.load(Ordering::Relaxed), 1, "on_start called once");
}
/// Given an actor stopped via stop_actor() with messages already queued,
/// when the stop signal arrives after the queued messages (PoisonPill semantics),
/// then messages ahead of the signal are processed, then the actor stops.
#[test]
fn external_stop_is_queued_after_pending_messages() {
let stopped = Arc::new(AtomicUsize::new(0));
let handled = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let started = Arc::new(AtomicUsize::new(0));
let addr = rt.spawn(LifecycleActor {
started: started.clone(),
stopped: stopped.clone(),
handled: handled.clone(),
}).unwrap();
let inbox = rt.new_inbox::<Pong>().unwrap();
// Queue 10 messages, then stop — StopSignal is queued AFTER the 10
for _ in 0..10 {
let _ = rt.send_to(addr, Ping { reply_to: *inbox.addr() });
}
rt.stop_actor(addr).unwrap();
for _ in 0..10 { rt.tick(); }
// All 10 messages processed (they were ahead of StopSignal in the queue)
let total_handled = handled.load(Ordering::Relaxed);
assert_eq!(total_handled, 10, "all messages processed before stop signal");
assert_eq!(stopped.load(Ordering::Relaxed), 1, "on_stop called");
// Actor is removed
let stats = rt.stats();
assert_eq!(stats.actors.len(), 0, "stopped actor removed");
}
/// Given a running actor with no pending messages,
/// when stop_actor() is called and then new messages are sent,
/// then the stop takes priority and new messages are not processed.
#[test]
fn external_stop_before_new_messages_prevents_processing() {
let stopped = Arc::new(AtomicUsize::new(0));
let handled = Arc::new(AtomicUsize::new(0));
let started = Arc::new(AtomicUsize::new(0));
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let addr = rt.spawn(LifecycleActor {
started: started.clone(),
stopped: stopped.clone(),
handled: handled.clone(),
}).unwrap();
// Let actor start
rt.tick();
// Stop first, then send messages
rt.stop_actor(addr).unwrap();
for _ in 0..5 {
let _ = rt.send_to(addr, Ping { reply_to: *inbox.addr() });
}
for _ in 0..10 { rt.tick(); }
// Stop signal was first in queue, so no messages processed
assert_eq!(handled.load(Ordering::Relaxed), 0, "no messages processed after stop");
assert_eq!(stopped.load(Ordering::Relaxed), 1, "on_stop called");
}
/// Given stop_actor is called on a nonexistent address,
/// then it returns Err.
#[test]
fn stop_nonexistent_actor_returns_error() {
let rt = std_runtime(RuntimeConfig::default());
let fake_addr = swactor::actor::ActorAddress::default();
let result = rt.stop_actor(fake_addr);
assert!(result.is_err(), "stop_actor on nonexistent address should return Err");
}

File diff suppressed because it is too large Load diff

View file

@ -1,362 +0,0 @@
mod common;
use common::*;
// ── Load-Aware Placement Tests ─────────────────────────────────────────────
/// Given a multi-threaded runtime where one worker has many more actors,
/// when new actors are spawned after a few ticks (so stats propagate),
/// then they should be placed on the lighter worker.
#[test]
fn load_aware_placement_prefers_lighter_worker() {
// 2 threads: intentionally imbalance by spawning many actors first
let rt = std_runtime(RuntimeConfig {
num_threads: 2,
..Default::default()
});
// Phase 1: Spawn 20 actors. With round-robin, they split ~10/10.
let mut addrs = Vec::new();
for _ in 0..20 {
addrs.push(rt.spawn(CounterActor { count: 0 }).unwrap());
}
// Run so stats propagate, then bombard worker 0's actors with messages
// to create mailbox depth imbalance.
let handle = rt.run().unwrap();
std::thread::sleep(std::time::Duration::from_millis(10));
// Send 500 messages to the first 10 actors (likely on worker 0).
for addr in &addrs[..10] {
for _ in 0..50 {
let _ = handle.runtime.send_to(*addr, Increment {
reply_to: *addr, // self-reply to keep mailbox depth up
});
}
}
std::thread::sleep(std::time::Duration::from_millis(20));
// Phase 2: Spawn 10 more actors. With load-aware placement,
// they should bias toward the lighter worker.
let mut late_addrs = Vec::new();
for _ in 0..10 {
late_addrs.push(handle.runtime.spawn(CounterActor { count: 0 }).unwrap());
}
std::thread::sleep(std::time::Duration::from_millis(20));
let stats = handle.runtime.stats();
handle.shutdown();
handle.join();
// Verify the system is operational — both workers should have actors
let total_actors: usize = stats.workers.iter().map(|w| w.num_actors).sum();
assert!(total_actors >= 20, "expected at least 20 actors, got {}", total_actors);
// The lighter worker should have gotten more of the late actors.
assert!(
stats.workers.iter().all(|w| w.num_actors > 0),
"both workers should have actors, got {:?}",
stats.workers.iter().map(|w| w.num_actors).collect::<Vec<_>>()
);
}
/// Given a single-threaded runtime (1 worker),
/// when many actors are spawned,
/// then all go to worker 0 regardless of load (no panic, no error).
#[test]
fn load_aware_placement_single_worker_degrades_gracefully() {
let rt = std_runtime(RuntimeConfig::default());
for _ in 0..50 {
rt.spawn(CounterActor { count: 0 }).unwrap();
}
// Tick several times to let stats update
for _ in 0..10 {
rt.tick();
}
let stats = rt.stats();
assert_eq!(stats.workers.len(), 1);
assert_eq!(stats.workers[0].num_actors, 50);
}
/// Given a fresh runtime with no prior ticks,
/// when actors are spawned in a burst,
/// then they distribute evenly (round-robin fallback when stats are all zero).
#[test]
fn load_aware_placement_falls_back_to_round_robin_on_fresh_runtime() {
let rt = std_runtime(RuntimeConfig {
num_threads: 4,
..Default::default()
});
// Spawn 100 actors before any ticks (all stats are zero)
for _ in 0..100 {
rt.spawn(CounterActor { count: 0 }).unwrap();
}
let handle = rt.run().unwrap();
std::thread::sleep(std::time::Duration::from_millis(20));
let stats = handle.runtime.stats();
handle.shutdown();
handle.join();
// With 4 workers and 100 actors, each should have ~25 (+-5).
for w in &stats.workers {
assert!(
w.num_actors >= 20 && w.num_actors <= 30,
"worker {} has {} actors, expected ~25 (round-robin)",
w.id, w.num_actors
);
}
}
// ── Mailbox Backpressure Tests ─────────────────────────────────────────────
/// Given a runtime with bounded mailboxes (capacity=10, DropNewest),
/// when 50 messages are sent to an actor before any ticks,
/// then only the first 10 are delivered and the rest are dropped.
#[test]
fn bounded_mailbox_drop_newest_caps_at_capacity() {
let rt = std_runtime(RuntimeConfig {
default_mailbox_capacity: 10,
mailbox_overflow: MailboxOverflow::DropNewest,
..Default::default()
});
let inbox = rt.new_inbox::<Count>().unwrap();
let addr = rt.spawn(CounterActor { count: 0 }).unwrap();
// Send 50 messages — only first 10 should be queued
for _ in 0..50 {
let _ = rt.send_to(addr, Increment { reply_to: *inbox.addr() });
}
// Tick enough times to process all queued messages
for _ in 0..20 {
rt.tick();
}
// Count replies — should be exactly 10 (the mailbox capacity)
let mut replies = 0;
while inbox.try_recv().is_some() {
replies += 1;
}
assert_eq!(replies, 10, "should deliver exactly mailbox_capacity messages");
// Stats should show drops
let stats = rt.stats();
let total_drops: u64 = stats.workers.iter().map(|w| w.messages_dropped).sum();
assert_eq!(total_drops, 40, "40 messages should have been dropped");
}
/// Given a runtime with bounded mailboxes (capacity=5, DropOldest),
/// when 10 messages are sent before any tick,
/// then only the 5 most recent messages are delivered.
#[test]
fn bounded_mailbox_drop_oldest_keeps_newest() {
let rt = std_runtime(RuntimeConfig {
default_mailbox_capacity: 5,
mailbox_overflow: MailboxOverflow::DropOldest,
..Default::default()
});
let inbox = rt.new_inbox::<Done>().unwrap();
let addr = rt.spawn(DoubleActor).unwrap();
// Send messages with values 0..10. DoubleActor replies Done(value * 2).
for i in 0..10 {
let _ = rt.send_to(addr, Forward {
value: i,
reply_to: *inbox.addr(),
});
}
for _ in 0..10 {
rt.tick();
}
// Collect all replies
let mut replies = Vec::new();
while let Some(Done(v)) = inbox.try_recv() {
replies.push(v);
}
assert_eq!(replies.len(), 5, "should deliver exactly 5 messages");
// The 5 most recent: values 5,6,7,8,9 -> doubled: 10,12,14,16,18
assert_eq!(replies, vec![10, 12, 14, 16, 18], "should keep the newest messages");
}
/// Given a runtime with unbounded mailboxes (capacity=0, the default),
/// when many messages are sent,
/// then all are delivered (backward compatibility).
#[test]
fn unbounded_mailbox_delivers_all_messages() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Count>().unwrap();
let addr = rt.spawn(CounterActor { count: 0 }).unwrap();
for _ in 0..200 {
let _ = rt.send_to(addr, Increment { reply_to: *inbox.addr() });
}
for _ in 0..50 {
rt.tick();
}
let mut replies = 0;
while inbox.try_recv().is_some() {
replies += 1;
}
assert_eq!(replies, 200, "all 200 messages should be delivered with unbounded mailbox");
let stats = rt.stats();
let total_drops: u64 = stats.workers.iter().map(|w| w.messages_dropped).sum();
assert_eq!(total_drops, 0, "no drops with unbounded mailbox");
}
/// Given bounded mailboxes with budget, when an actor processes messages
/// and frees mailbox space, then new messages should be accepted on subsequent ticks.
#[test]
fn bounded_mailbox_refills_after_processing() {
let rt = std_runtime(RuntimeConfig {
default_mailbox_capacity: 5,
actor_message_budget: 5,
mailbox_overflow: MailboxOverflow::DropNewest,
..Default::default()
});
let inbox = rt.new_inbox::<Count>().unwrap();
let addr = rt.spawn(CounterActor { count: 0 }).unwrap();
// Send first batch of 5 — fills mailbox exactly
for _ in 0..5 {
let _ = rt.send_to(addr, Increment { reply_to: *inbox.addr() });
}
// Tick to process all 5 (budget=5, capacity=5)
rt.tick();
// Send second batch of 5 — mailbox is empty, so all 5 should be accepted
for _ in 0..5 {
let _ = rt.send_to(addr, Increment { reply_to: *inbox.addr() });
}
rt.tick();
let mut replies = 0;
while inbox.try_recv().is_some() {
replies += 1;
}
assert_eq!(replies, 10, "all 10 messages across 2 batches should be processed");
let stats = rt.stats();
let total_drops: u64 = stats.workers.iter().map(|w| w.messages_dropped).sum();
assert_eq!(total_drops, 0, "no drops when mailbox drains between batches");
}
// ── Dead Actor Cleanup Tests ───────────────────────────────────────────────
/// Given an actor that panics and is poisoned,
/// when ticks continue,
/// then the actor is removed from stats and sends to its address fail.
#[test]
fn dead_actor_cleaned_up_from_stats_and_address_map() {
let rt = std_runtime(RuntimeConfig::default());
let good = rt.spawn(PingPongActor).unwrap();
let bad = rt.spawn(PanicActor).unwrap();
// Trigger panic
let _ = rt.send_to(bad, PanicMsg);
for _ in 0..5 { rt.tick(); }
let stats = rt.stats();
// Good actor still present, bad actor cleaned up
assert_eq!(stats.workers[0].num_actors, 1, "only the healthy actor should remain");
assert!(
stats.actors.iter().any(|(a, _)| *a == good),
"good actor should be in address map"
);
assert!(
!stats.actors.iter().any(|(a, _)| *a == bad),
"poisoned actor should be removed from address map"
);
// Sends to cleaned-up actor fail
let result = rt.send_to(bad, PanicMsg);
assert!(result.is_err(), "send to cleaned-up actor should fail");
}
/// Given many actors that all panic,
/// when ticks proceed,
/// then all are cleaned up and stats reflect zero actors.
#[test]
fn bulk_dead_actor_cleanup() {
let rt = std_runtime(RuntimeConfig::default());
let mut addrs = Vec::new();
for _ in 0..20 {
addrs.push(rt.spawn(PanicActor).unwrap());
}
// Trigger all panics
for &addr in &addrs {
let _ = rt.send_to(addr, PanicMsg);
}
for _ in 0..10 { rt.tick(); }
let stats = rt.stats();
assert_eq!(stats.workers[0].num_actors, 0, "all poisoned actors should be cleaned up");
assert_eq!(
stats.actors.len(), 0,
"address map should be empty after all actors poisoned"
);
}
// ── Actor Recovery Helpers ──────────────────────────────────────────────────
/// Handles Forward messages, replies Done(value * 2), panics on the panic_at-th message.
struct RestartTestActor {
count: usize,
panic_at: usize,
}
impl ActorInterface for RestartTestActor {
type Incoming = Forward;
type Response = Done;
fn handle(&mut self, ctx: &Ctx, msg: Forward) {
self.count += 1;
if self.count >= self.panic_at {
panic!("intentional panic at message {}", self.count);
}
let _ = ctx.send(msg.reply_to, Done(msg.value * 2));
}
}
// ── Actor Recovery Tests ───────────────────────────────────────────────────
/// Given an actor that panics,
/// when it panics,
/// then it is poisoned and future messages are discarded.
#[test]
fn non_restartable_actor_still_poisons_on_panic() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Done>().unwrap();
// Normal spawn — not restartable
let addr = rt.spawn(RestartTestActor { count: 0, panic_at: 1 }).unwrap();
let _ = rt.send_to(addr, Forward { value: 42, reply_to: *inbox.addr() });
for _ in 0..5 { rt.tick(); }
let stats = rt.stats();
let total_panics: u64 = stats.workers.iter().map(|w| w.panics).sum();
let total_restarts: u64 = stats.workers.iter().map(|w| w.restarts).sum();
assert_eq!(total_panics, 1, "should panic");
assert_eq!(total_restarts, 0, "should not restart (not restartable)");
}

View file

@ -1,758 +0,0 @@
mod common;
use common::*;
// ── Named Actor Registry ────────────────────────────────────────────────────
/// Given a named actor is spawned,
/// when I look it up by name,
/// then I get the same address that spawn returned.
#[test]
fn named_actor_lookup_returns_spawn_address() {
let rt = std_runtime(RuntimeConfig::default());
let addr = rt.spawn_named("greeter", PingPongActor).unwrap();
assert_eq!(rt.where_is("greeter"), Some(addr));
}
/// Given a named actor exists,
/// when I send a message to the looked-up address,
/// then the actor receives and processes it.
#[test]
fn named_actor_receives_messages_via_lookup() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let addr = rt.spawn_named("ponger", PingPongActor).unwrap();
assert_eq!(rt.where_is("ponger"), Some(addr));
rt.send_to(addr, Ping { reply_to: *inbox.addr() }).unwrap();
rt.tick();
assert!(inbox.try_recv().is_some(), "named actor should process message");
}
/// Given a name is already registered,
/// when I try to spawn another actor with the same name,
/// then I get an error and the original binding is preserved.
#[test]
fn duplicate_name_returns_error() {
let rt = std_runtime(RuntimeConfig::default());
let first_addr = rt.spawn_named("singleton", PingPongActor).unwrap();
let result = rt.spawn_named("singleton", PingPongActor);
assert!(result.is_err(), "duplicate name should fail");
assert_eq!(rt.where_is("singleton"), Some(first_addr), "original binding preserved");
}
/// Given no actors are registered,
/// when I look up a nonexistent name,
/// then I get None.
#[test]
fn where_is_returns_none_for_unknown_name() {
let rt = std_runtime(RuntimeConfig::default());
assert_eq!(rt.where_is("ghost"), None);
}
/// Given a named actor is stopped,
/// when the next tick runs cleanup,
/// then the name is automatically unregistered.
#[test]
fn name_auto_unregistered_on_actor_death() {
let rt = std_runtime(RuntimeConfig::default());
let addr = rt.spawn_named("ephemeral", PingPongActor).unwrap();
rt.tick(); // on_start
rt.stop_actor(addr).unwrap();
rt.tick(); // process StopSignal + cleanup
assert_eq!(rt.where_is("ephemeral"), None, "name should be freed after stop");
}
/// Given a named actor died and its name was freed,
/// when I spawn a new actor with the same name,
/// then registration succeeds with a new address.
#[test]
fn name_can_be_reused_after_actor_death() {
let rt = std_runtime(RuntimeConfig::default());
let first = rt.spawn_named("worker", PingPongActor).unwrap();
rt.tick();
rt.stop_actor(first).unwrap();
rt.tick(); // cleanup frees the name
let second = rt.spawn_named("worker", PingPongActor).unwrap();
assert_ne!(first, second, "new actor should have a different address");
assert_eq!(rt.where_is("worker"), Some(second));
}
/// Given a named actor panics (and is not restartable),
/// when the next tick runs cleanup,
/// then the name is freed.
#[test]
fn name_auto_unregistered_on_panic() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let _addr = rt.spawn_named("fragile", PanicActor).unwrap();
rt.tick(); // on_start
rt.send_to(_addr, PanicMsg).unwrap();
rt.tick(); // panic -> poison -> cleanup
assert_eq!(rt.where_is("fragile"), None, "name freed after panic");
// Can reuse the name
let _new = rt.spawn_named("fragile", PingPongActor).unwrap();
assert!(rt.where_is("fragile").is_some());
drop(inbox);
}
/// Given multiple named actors are registered,
/// when I call registered_names(),
/// then all names are returned.
#[test]
fn registered_names_lists_all() {
let rt = std_runtime(RuntimeConfig::default());
rt.spawn_named("alpha", PingPongActor).unwrap();
rt.spawn_named("beta", PingPongActor).unwrap();
rt.spawn_named("gamma", PingPongActor).unwrap();
let mut names = rt.registered_names();
names.sort();
assert_eq!(names, vec!["alpha", "beta", "gamma"]);
}
/// Given a named actor exists,
/// when I manually unregister the name,
/// then the name is freed but the actor continues running.
#[test]
fn manual_unregister_frees_name_but_actor_lives() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let addr = rt.spawn_named("temp-name", PingPongActor).unwrap();
rt.tick(); // on_start
let removed = rt.unregister("temp-name");
assert_eq!(removed, Some(addr));
assert_eq!(rt.where_is("temp-name"), None, "name freed");
// Actor still alive and can receive messages
rt.send_to(addr, Ping { reply_to: *inbox.addr() }).unwrap();
rt.tick();
assert!(inbox.try_recv().is_some(), "actor still processes messages");
}
/// An actor that looks up a peer by name using ctx.where_is().
struct NameLookupActor {
target_name: &'static str,
reply_to: ActorAddress,
}
impl ActorInterface for NameLookupActor {
type Incoming = Ping;
type Response = ();
fn handle(&mut self, ctx: &Ctx, _msg: Ping) {
if let Some(peer) = ctx.where_is(self.target_name) {
ctx.send(self.reply_to, MyAddr(peer)).unwrap();
}
}
}
/// Given a named actor exists,
/// when another actor calls ctx.where_is() from inside a handler,
/// then it resolves the correct address.
#[test]
fn ctx_where_is_resolves_inside_handler() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<MyAddr>().unwrap();
let target = rt.spawn_named("target", PingPongActor).unwrap();
let looker = rt.spawn(NameLookupActor {
target_name: "target",
reply_to: *inbox.addr(),
}).unwrap();
rt.tick(); // on_start
rt.send_to(looker, Ping { reply_to: ActorAddress::default() }).unwrap();
rt.tick(); // handle -> where_is -> send
rt.tick(); // deliver reply
let result = inbox.try_recv();
assert_eq!(result, Some(MyAddr(target)), "ctx.where_is found the named actor");
}
/// An actor that spawns a named child using ctx.spawn_named().
struct NamedSpawnerActor {
reply_to: ActorAddress,
}
impl ActorInterface for NamedSpawnerActor {
type Incoming = Ping;
type Response = ();
fn handle(&mut self, ctx: &Ctx, _msg: Ping) {
match ctx.spawn_named("child", PingPongActor) {
Ok(addr) => { ctx.send(self.reply_to, MyAddr(addr)).unwrap(); }
Err(_) => {}
}
}
}
/// Given an actor calls ctx.spawn_named("child", ...),
/// when the child is spawned,
/// then where_is("child") returns the correct address.
#[test]
fn ctx_spawn_named_registers_from_handler() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<MyAddr>().unwrap();
let spawner = rt.spawn(NamedSpawnerActor {
reply_to: *inbox.addr(),
}).unwrap();
rt.tick(); // on_start
rt.send_to(spawner, Ping { reply_to: ActorAddress::default() }).unwrap();
rt.tick(); // handle -> spawn_named
rt.tick(); // deliver reply
let child_addr = inbox.try_recv().expect("should receive child address");
assert_eq!(rt.where_is("child"), Some(child_addr.0), "name registered from handler");
}
// ── Actor Monitoring / Death Watch ──────────────────────────────────────────
/// An actor that monitors a target and forwards Down notifications to a reply address.
struct WatcherActor {
watch_target: ActorAddress,
reply_to: ActorAddress,
mref: Option<MonitorRef>,
}
impl ActorInterface for WatcherActor {
type Incoming = Down;
type Response = ();
fn on_start(&mut self, ctx: &Ctx) {
self.mref = Some(ctx.monitor(self.watch_target));
}
fn handle(&mut self, ctx: &Ctx, msg: Down) {
// Forward the Down notification to the test inbox
ctx.send(self.reply_to, msg).unwrap();
}
}
/// Given actor A monitors actor B,
/// when B is gracefully stopped,
/// then A receives a Down { reason: Normal } message.
#[test]
fn monitor_notifies_on_graceful_stop() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Down>().unwrap();
let target = rt.spawn(PingPongActor).unwrap();
let _watcher = rt.spawn(WatcherActor {
watch_target: target,
reply_to: *inbox.addr(),
mref: None,
}).unwrap();
rt.tick(); // on_start -> watcher sets up monitor
rt.stop_actor(target).unwrap();
rt.tick(); // target receives StopSignal -> cleanup_dead emits Down
rt.tick(); // watcher receives Down -> forwards to inbox
let down = inbox.try_recv().expect("should receive Down notification");
assert_eq!(down.addr, target);
assert_eq!(down.reason, StopReason::Normal);
}
/// Given actor A monitors actor B,
/// when B panics,
/// then A receives a Down { reason: Panicked } message.
#[test]
fn monitor_notifies_on_panic() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Down>().unwrap();
let target = rt.spawn(PanicActor).unwrap();
let _watcher = rt.spawn(WatcherActor {
watch_target: target,
reply_to: *inbox.addr(),
mref: None,
}).unwrap();
rt.tick(); // on_start
rt.send_to(target, PanicMsg).unwrap();
rt.tick(); // target panics -> cleanup_dead emits Down
rt.tick(); // watcher receives Down -> forwards to inbox
let down = inbox.try_recv().expect("should receive Down on panic");
assert_eq!(down.addr, target);
assert_eq!(down.reason, StopReason::Panicked);
}
/// Given two actors both monitor the same target,
/// when the target dies,
/// then both watchers receive independent Down notifications.
#[test]
fn multiple_watchers_all_notified() {
let rt = std_runtime(RuntimeConfig::default());
let inbox1 = rt.new_inbox::<Down>().unwrap();
let inbox2 = rt.new_inbox::<Down>().unwrap();
let target = rt.spawn(PingPongActor).unwrap();
rt.spawn(WatcherActor {
watch_target: target,
reply_to: *inbox1.addr(),
mref: None,
}).unwrap();
rt.spawn(WatcherActor {
watch_target: target,
reply_to: *inbox2.addr(),
mref: None,
}).unwrap();
rt.tick(); // on_start for all
rt.stop_actor(target).unwrap();
rt.tick(); // cleanup -> Down emitted to both watchers
rt.tick(); // watchers forward Down to inboxes
assert!(inbox1.try_recv().is_some(), "watcher 1 should receive Down");
assert!(inbox2.try_recv().is_some(), "watcher 2 should receive Down");
}
/// An actor that demonitors in response to a Ping message.
struct DemonitorActor {
watch_target: ActorAddress,
mref: Option<MonitorRef>,
}
impl ActorInterface for DemonitorActor {
type Incoming = Ping;
type Response = ();
fn on_start(&mut self, ctx: &Ctx) {
self.mref = Some(ctx.monitor(self.watch_target));
}
fn handle(&mut self, ctx: &Ctx, _msg: Ping) {
// Cancel the monitor
if let Some(mref) = self.mref.take() {
ctx.demonitor(mref);
}
}
}
/// Given actor A monitors actor B then demonitors,
/// when B dies,
/// then A does NOT receive a Down notification.
#[test]
fn demonitor_cancels_notification() {
let rt = std_runtime(RuntimeConfig::default());
let down_inbox = rt.new_inbox::<Down>().unwrap();
let target = rt.spawn(PingPongActor).unwrap();
let watcher = rt.spawn(DemonitorActor {
watch_target: target,
mref: None,
}).unwrap();
rt.tick(); // on_start -> monitor set up
// Trigger demonitor
rt.send_to(watcher, Ping { reply_to: ActorAddress::default() }).unwrap();
rt.tick(); // handle -> demonitor
// Now kill the target
rt.stop_actor(target).unwrap();
rt.tick(); // cleanup -- no Down should be emitted
rt.tick(); // extra tick to be sure
assert!(down_inbox.try_recv().is_none(), "demonitored -- should NOT receive Down");
}
/// Given actor A monitors B, and A dies before B,
/// when B dies,
/// then no Down is delivered (dead watcher cleaned up).
#[test]
fn dead_watcher_does_not_receive_down() {
let rt = std_runtime(RuntimeConfig::default());
let target = rt.spawn(PingPongActor).unwrap();
let watcher = rt.spawn(WatcherActor {
watch_target: target,
reply_to: ActorAddress::default(), // won't matter, watcher dies first
mref: None,
}).unwrap();
rt.tick(); // on_start -> monitor set up
rt.stop_actor(watcher).unwrap();
rt.tick(); // watcher dies -> its monitors are cleaned up
// Now kill the target -- the dead watcher's subscription should be gone
rt.stop_actor(target).unwrap();
rt.tick(); // cleanup -- should not panic or try to deliver to dead watcher
// If we get here without panic, the test passes
}
/// Given an external inbox monitors via the runtime,
/// when the target dies,
/// then the inbox receives a Down message.
#[test]
fn down_delivered_to_external_inbox() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Down>().unwrap();
let target = rt.spawn(PingPongActor).unwrap();
let _watcher = rt.spawn(WatcherActor {
watch_target: target,
reply_to: *inbox.addr(),
mref: None,
}).unwrap();
rt.tick(); // on_start
rt.stop_actor(target).unwrap();
rt.tick(); // cleanup -> Down to watcher
rt.tick(); // watcher forwards to inbox
let down = inbox.try_recv().expect("inbox should receive forwarded Down");
assert_eq!(down.addr, target);
assert_eq!(down.reason, StopReason::Normal);
}
/// Given actor A monitors B with two independent monitors,
/// when B dies,
/// then A receives two Down messages (one per monitor).
#[test]
fn stacked_monitors_produce_multiple_notifications() {
/// An actor that creates two monitors on the same target.
struct DoubleWatcherActor {
target: ActorAddress,
reply_to: ActorAddress,
}
impl ActorInterface for DoubleWatcherActor {
type Incoming = Down;
type Response = ();
fn on_start(&mut self, ctx: &Ctx) {
ctx.monitor(self.target);
ctx.monitor(self.target);
}
fn handle(&mut self, ctx: &Ctx, msg: Down) {
ctx.send(self.reply_to, msg).unwrap();
}
}
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Down>().unwrap();
let target = rt.spawn(PingPongActor).unwrap();
rt.spawn(DoubleWatcherActor {
target,
reply_to: *inbox.addr(),
}).unwrap();
rt.tick(); // on_start -> 2 monitors
rt.stop_actor(target).unwrap();
rt.tick(); // cleanup -> 2 Down messages to watcher
rt.tick(); // watcher forwards both to inbox
assert!(inbox.try_recv().is_some(), "first Down");
assert!(inbox.try_recv().is_some(), "second Down");
assert!(inbox.try_recv().is_none(), "no more");
}
// ── Actor Groups / Pub-Sub ──────────────────────────────────────────────────
/// Given actors join a group,
/// when I query group_members,
/// then all joined actors are listed.
#[test]
fn group_members_returns_joined_actors() {
let rt = std_runtime(RuntimeConfig::default());
let a = rt.spawn(PingPongActor).unwrap();
let b = rt.spawn(PingPongActor).unwrap();
rt.join_group(a, "workers");
rt.join_group(b, "workers");
let mut members = rt.group_members("workers");
members.sort_by_key(|addr| addr.0);
let mut expected = vec![a, b];
expected.sort_by_key(|addr| addr.0);
assert_eq!(members, expected);
}
/// Given no actors have joined a group,
/// when I query group_members,
/// then the result is empty.
#[test]
fn empty_group_returns_no_members() {
let rt = std_runtime(RuntimeConfig::default());
assert!(rt.group_members("nonexistent").is_empty());
}
/// Given actors in a group,
/// when a message is published to the group,
/// then all members receive the message.
#[test]
fn publish_broadcasts_to_all_members() {
let rt = std_runtime(RuntimeConfig::default());
let inbox1 = rt.new_inbox::<Pong>().unwrap();
let inbox2 = rt.new_inbox::<Pong>().unwrap();
let a = rt.spawn(PingPongActor).unwrap();
let b = rt.spawn(PingPongActor).unwrap();
rt.join_group(a, "pongers");
rt.join_group(b, "pongers");
rt.tick(); // on_start
// Publish a Ping with inbox1's addr as reply_to.
let count = rt.publish_to("pongers", Ping { reply_to: *inbox1.addr() });
assert_eq!(count, 2, "two members, two messages sent");
rt.tick(); // actors handle Ping -> send Pong to inbox1
// Both actors send to inbox1
assert!(inbox1.try_recv().is_some(), "first Pong");
assert!(inbox1.try_recv().is_some(), "second Pong");
assert!(inbox1.try_recv().is_none(), "no more");
drop(inbox2);
}
/// Given an actor leaves a group,
/// when a message is published,
/// then the leaver does not receive it.
#[test]
fn leave_group_stops_receiving_publishes() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let a = rt.spawn(PingPongActor).unwrap();
let b = rt.spawn(PingPongActor).unwrap();
rt.join_group(a, "pool");
rt.join_group(b, "pool");
rt.leave_group(b, "pool");
rt.tick(); // on_start
let count = rt.publish_to("pool", Ping { reply_to: *inbox.addr() });
assert_eq!(count, 1, "only one member after leave");
rt.tick();
assert!(inbox.try_recv().is_some(), "one Pong from remaining member");
assert!(inbox.try_recv().is_none(), "no second Pong");
}
/// Given a group member dies,
/// when a message is published,
/// then the dead member is not included.
#[test]
fn dead_actor_auto_removed_from_group() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let a = rt.spawn(PingPongActor).unwrap();
let b = rt.spawn(PingPongActor).unwrap();
rt.join_group(a, "team");
rt.join_group(b, "team");
rt.tick(); // on_start
rt.stop_actor(b).unwrap();
rt.tick(); // b dies, cleaned up from group
let count = rt.publish_to("team", Ping { reply_to: *inbox.addr() });
assert_eq!(count, 1, "dead actor removed from group");
rt.tick();
assert!(inbox.try_recv().is_some());
assert!(inbox.try_recv().is_none());
}
/// Given an actor is in multiple groups,
/// when the actor dies,
/// then it is removed from all groups.
#[test]
fn actor_removed_from_all_groups_on_death() {
let rt = std_runtime(RuntimeConfig::default());
let actor = rt.spawn(PingPongActor).unwrap();
rt.join_group(actor, "alpha");
rt.join_group(actor, "beta");
rt.join_group(actor, "gamma");
rt.tick();
rt.stop_actor(actor).unwrap();
rt.tick(); // cleanup removes from all groups
assert!(rt.group_members("alpha").is_empty());
assert!(rt.group_members("beta").is_empty());
assert!(rt.group_members("gamma").is_empty());
}
/// Given a group becomes empty after its last member leaves,
/// then the group name disappears from the active groups list.
#[test]
fn empty_group_auto_deleted() {
let rt = std_runtime(RuntimeConfig::default());
let actor = rt.spawn(PingPongActor).unwrap();
rt.join_group(actor, "temp");
assert!(rt.groups().contains(&"temp".to_string()));
rt.leave_group(actor, "temp");
assert!(!rt.groups().contains(&"temp".to_string()), "empty group should be removed");
}
/// Given actors join groups from handlers using ctx.join_group(),
/// when group_members is queried,
/// then the joining actors are listed.
#[test]
fn ctx_join_group_from_handler() {
struct GroupJoinerActor;
impl ActorInterface for GroupJoinerActor {
type Incoming = Ping;
type Response = ();
fn on_start(&mut self, ctx: &Ctx) {
ctx.join_group("auto-joined");
}
fn handle(&mut self, _ctx: &Ctx, _msg: Ping) {}
}
let rt = std_runtime(RuntimeConfig::default());
let a = rt.spawn(GroupJoinerActor).unwrap();
let b = rt.spawn(GroupJoinerActor).unwrap();
rt.tick(); // on_start -> both join "auto-joined"
let members = rt.group_members("auto-joined");
assert_eq!(members.len(), 2);
assert!(members.contains(&a));
assert!(members.contains(&b));
}
/// Given an actor uses ctx.publish() from inside a handler,
/// when the published message is processed,
/// then all group members receive it.
#[test]
fn ctx_publish_broadcasts_from_handler() {
#[derive(Clone)]
struct BroadcastCmd {
reply_to: ActorAddress,
}
struct BroadcasterActor;
impl ActorInterface for BroadcasterActor {
type Incoming = BroadcastCmd;
type Response = ();
fn on_start(&mut self, ctx: &Ctx) {
ctx.join_group("broadcast-test");
}
fn handle(&mut self, ctx: &Ctx, msg: BroadcastCmd) {
ctx.publish("broadcast-test", Ping { reply_to: msg.reply_to });
}
}
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
// Spawn 3 PingPongActors and one Broadcaster, all in the same group
let _p1 = rt.spawn(PingPongActor).unwrap();
let _p2 = rt.spawn(PingPongActor).unwrap();
rt.join_group(_p1, "broadcast-test");
rt.join_group(_p2, "broadcast-test");
let broadcaster = rt.spawn(BroadcasterActor).unwrap();
rt.tick(); // on_start (broadcaster joins group too)
// Send BroadcastCmd to broadcaster
rt.send_to(broadcaster, BroadcastCmd { reply_to: *inbox.addr() }).unwrap();
rt.tick(); // broadcaster handles -> publish Ping to all 3 members (including self)
rt.tick(); // PingPong actors handle Ping -> send Pong to inbox
// At least 2 Pongs from the PingPongActors
let mut pong_count = 0;
while inbox.try_recv().is_some() {
pong_count += 1;
}
assert!(pong_count >= 2, "at least 2 PingPong members should reply, got {pong_count}");
}
// ── Ask Pattern ─────────────────────────────────────────────────────────────
/// Given a PingPong actor,
/// when I ask with recv_ticking,
/// then I get the Pong response.
#[test]
fn ask_recv_ticking_returns_response() {
let rt = std_runtime(RuntimeConfig::default());
let actor = rt.spawn(PingPongActor).unwrap();
rt.tick(); // on_start
let pong: Pong = rt.ask(actor, |reply_to| Ping { reply_to })
.unwrap()
.recv_ticking(&rt, 10)
.unwrap();
assert_eq!(pong, Pong);
}
/// Given a CounterActor,
/// when I ask multiple times,
/// then each response reflects the updated state.
#[test]
fn ask_multiple_times_tracks_state() {
let rt = std_runtime(RuntimeConfig::default());
let actor = rt.spawn(CounterActor { count: 0 }).unwrap();
rt.tick(); // on_start
let c1: Count = rt.ask(actor, |reply_to| Increment { reply_to })
.unwrap().recv_ticking(&rt, 10).unwrap();
let c2: Count = rt.ask(actor, |reply_to| Increment { reply_to })
.unwrap().recv_ticking(&rt, 10).unwrap();
let c3: Count = rt.ask(actor, |reply_to| Increment { reply_to })
.unwrap().recv_ticking(&rt, 10).unwrap();
assert_eq!(c1, Count(1));
assert_eq!(c2, Count(2));
assert_eq!(c3, Count(3));
}
/// Given a dead actor,
/// when I ask and tick,
/// then recv_ticking returns a timeout error.
#[test]
fn ask_timeout_when_no_response() {
let rt = std_runtime(RuntimeConfig::default());
let actor = rt.spawn(PingPongActor).unwrap();
rt.tick();
rt.stop_actor(actor).unwrap();
rt.tick(); // actor dies
// Ask the dead actor -- message is undeliverable, no response
let result = rt.ask::<Ping, Pong>(actor, |reply_to| Ping { reply_to });
// send_to may succeed or fail
if let Ok(ask) = result {
let err = ask.recv_ticking(&rt, 5);
assert!(err.is_err(), "should timeout with no response");
}
}
/// Given an ask handle,
/// when I use try_recv before ticking,
/// then it returns None (response hasn't arrived yet).
#[test]
fn ask_try_recv_returns_none_before_tick() {
let rt = std_runtime(RuntimeConfig::default());
let actor = rt.spawn(PingPongActor).unwrap();
rt.tick(); // on_start
let ask = rt.ask::<Ping, Pong>(actor, |reply_to| Ping { reply_to }).unwrap();
assert!(ask.try_recv().is_none(), "no response before ticking");
rt.tick(); // process message
assert_eq!(ask.try_recv(), Some(Pong));
}
/// Given an ask, the reply_addr() returns the inbox address for manual use.
#[test]
fn ask_reply_addr_is_accessible() {
let rt = std_runtime(RuntimeConfig::default());
let actor = rt.spawn(PingPongActor).unwrap();
rt.tick();
let ask = rt.ask::<Ping, Pong>(actor, |reply_to| Ping { reply_to }).unwrap();
let addr = *ask.reply_addr();
// The address should be valid (non-zero)
assert_ne!(addr, ActorAddress::default());
}

364
tests/runtime_stress.rs Normal file
View file

@ -0,0 +1,364 @@
//! Runtime Stress Tests — multi-threaded execution, parking, placement, and scale.
//!
//! Covers: single vs multi-threaded processing, high-volume MT delivery,
//! panic isolation under load, worker parking/shutdown, sustained throughput,
//! and load-aware actor placement.
mod common;
use common::*;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
// ── Helpers ──────────────────────────────────────────────────────────────────
/// Poll `inbox` until a message arrives or `timeout` elapses.
fn poll_inbox<M: swactor::actor::Message>(
inbox: &Inbox<M>,
timeout: Duration,
) -> Option<M> {
let deadline = Instant::now() + timeout;
loop {
if let Some(msg) = inbox.try_recv() {
return Some(msg);
}
if Instant::now() > deadline {
return None;
}
std::thread::sleep(Duration::from_millis(1));
}
}
/// Wait until `counter` reaches `target` or `timeout` elapses.
fn wait_for_count(counter: &AtomicUsize, target: usize, timeout: Duration) -> usize {
let deadline = Instant::now() + timeout;
loop {
let n = counter.load(Ordering::SeqCst);
if n >= target {
return n;
}
if Instant::now() > deadline {
return n;
}
std::thread::sleep(Duration::from_millis(5));
}
}
// ── Tests ────────────────────────────────────────────────────────────────────
/// Single-threaded vs multi-threaded runtime basics.
///
/// Story: We start with a single-threaded runtime driven by tick(), confirm
/// nothing happens without ticking, then graduate to a multi-threaded runtime
/// with run() and verify background processing, cross-worker delegation,
/// custom thread counts, and clean shutdown.
#[test]
fn single_vs_multi_threaded_basics() {
// ── Part A: Single-threaded requires tick() ──
let rt = std_runtime(RuntimeConfig::default());
let addr = rt.spawn(PingPongActor).unwrap();
let inbox = rt.new_inbox::<Pong>().unwrap();
rt.send_to(addr, Ping { reply_to: *inbox.addr() }).unwrap();
assert!(inbox.try_recv().is_none(), "no processing before tick");
tick_n(&rt, 2);
assert!(inbox.try_recv().is_some(), "tick() drives single-threaded processing");
// ── Part B: Multi-threaded processes without ticking ──
let rt_mt = std_runtime(RuntimeConfig { num_threads: 4, ..Default::default() });
let addr = rt_mt.spawn(PingPongActor).unwrap();
let inbox = rt_mt.new_inbox::<Pong>().unwrap();
rt_mt.send_to(addr, Ping { reply_to: *inbox.addr() }).unwrap();
let handle = rt_mt.run().unwrap();
let reply = poll_inbox(&inbox, Duration::from_secs(5));
assert!(reply.is_some(), "background workers process without manual ticking");
// ── Part C: Cross-worker delegation (2 threads, spawn child from handler) ──
let addr2 = handle.runtime.spawn(DelegatorActor).unwrap();
let done_inbox = handle.runtime.new_inbox::<Done>().unwrap();
handle.runtime.send_to(addr2, Forward { value: 3, reply_to: *done_inbox.addr() }).unwrap();
let reply = poll_inbox(&done_inbox, Duration::from_secs(5));
assert_eq!(reply, Some(Done(6)), "cross-worker delegation delivers reply");
// ── Part D: Custom thread count reflected in stats ──
let stats = handle.runtime.stats();
assert_eq!(stats.num_workers, 4, "runtime respects requested thread count");
// ── Part E: Clean shutdown ──
handle.shutdown();
handle.join();
// Test passes by not hanging.
}
/// High-volume multi-threaded delivery.
///
/// Story: We throw large workloads at a 4-thread runtime — 50 senders
/// each firing 100 messages at one receiver, 200 concurrent spawn+send
/// pairs, and a 50-level chain that must hop across workers.
#[test]
fn mt_high_volume_delivery() {
let cfg = || RuntimeConfig {
num_threads: 4,
max_actors: 5_000,
channel_buffer_size: 10_000,
..Default::default()
};
// ── Part A: 50 senders × 100 messages → one receiver ──
{
let rt = std_runtime(cfg());
let counter = Arc::new(AtomicUsize::new(0));
let dummy = rt.new_inbox::<Pong>().unwrap();
let receiver = rt.spawn(CountingPingActor { counter: counter.clone() }).unwrap();
let total_expected = 50 * 100;
for _ in 0..50 {
for _ in 0..100 {
rt.send_to(receiver, Ping { reply_to: *dummy.addr() }).unwrap();
}
}
let handle = rt.run().unwrap();
let processed = wait_for_count(&counter, total_expected, Duration::from_secs(5));
handle.shutdown();
handle.join();
assert_eq!(processed, total_expected, "all 5000 messages delivered to single receiver");
}
// ── Part B: 200 concurrent spawn+send pairs ──
{
let rt = std_runtime(cfg());
let counter = Arc::new(AtomicUsize::new(0));
let dummy = rt.new_inbox::<Pong>().unwrap();
for _ in 0..200 {
let a = rt.spawn(CountingPingActor { counter: counter.clone() }).unwrap();
rt.send_to(a, Ping { reply_to: *dummy.addr() }).unwrap();
}
let handle = rt.run().unwrap();
let received = wait_for_count(&counter, 200, Duration::from_secs(5));
handle.shutdown();
handle.join();
assert_eq!(received, 200, "all 200 spawn+send pairs complete");
}
// ── Part C: 50-level chain across workers ──
{
let rt = std_runtime(RuntimeConfig { num_threads: 2, max_actors: 5_000, ..Default::default() });
let addr = rt.spawn(ChainActor).unwrap();
let inbox = rt.new_inbox::<Done>().unwrap();
rt.send_to(addr, ChainMsg { remaining: 50, depth: 0, reply_to: *inbox.addr() }).unwrap();
let handle = rt.run().unwrap();
let reply = poll_inbox(&inbox, Duration::from_secs(5));
handle.shutdown();
handle.join();
assert_eq!(reply, Some(Done(50)), "50-level chain completes across workers");
}
}
/// Panic isolation under multi-threaded load.
///
/// Story: 10 panicking actors and 10 healthy actors on 4 threads — every
/// panic is isolated and all 1000 healthy messages are still processed.
#[test]
fn mt_panic_isolation_under_load() {
let rt = std_runtime(RuntimeConfig {
num_threads: 4,
max_actors: 5_000,
channel_buffer_size: 10_000,
..Default::default()
});
let counter = Arc::new(AtomicUsize::new(0));
let dummy = rt.new_inbox::<Pong>().unwrap();
let mut panic_addrs = Vec::new();
let mut healthy_addrs = Vec::new();
for _ in 0..10 {
panic_addrs.push(rt.spawn(PanicActor).unwrap());
healthy_addrs.push(rt.spawn(CountingPingActor { counter: counter.clone() }).unwrap());
}
// Trigger panics and flood healthy actors.
for &addr in &panic_addrs {
rt.send_to(addr, PanicMsg).unwrap();
}
for &addr in &healthy_addrs {
for _ in 0..100 {
rt.send_to(addr, Ping { reply_to: *dummy.addr() }).unwrap();
}
}
let handle = rt.run().unwrap();
let expected = 10 * 100;
let processed = wait_for_count(&counter, expected, Duration::from_secs(5));
handle.shutdown();
handle.join();
assert_eq!(
processed, expected,
"all {expected} healthy messages processed despite panicking peers"
);
}
/// Worker parking and shutdown latency.
///
/// Story: Workers park after idle time. We verify they wake quickly on new
/// messages, that messages sent after run() are delivered, and that shutdown
/// wakes all parked workers promptly.
#[test]
fn worker_parking_and_shutdown() {
// ── Part A: Parked workers wake on send ──
let rt = std_runtime(RuntimeConfig { num_threads: 2, ..Default::default() });
let addr = rt.spawn(PingPongActor).unwrap();
let inbox = rt.new_inbox::<Pong>().unwrap();
let handle = rt.run().unwrap();
std::thread::sleep(Duration::from_millis(50)); // Let workers park.
let before = Instant::now();
handle.runtime.send_to(addr, Ping { reply_to: *inbox.addr() }).unwrap();
let reply = poll_inbox(&inbox, Duration::from_secs(1));
let latency = before.elapsed();
assert!(reply.is_some(), "parked worker should wake and process");
assert!(latency.as_millis() < 100, "wake latency should be <100ms, was {:?}", latency);
// ── Part B: Send after run() delivers ──
let addr2 = handle.runtime.spawn(PingPongActor).unwrap();
let inbox2 = handle.runtime.new_inbox::<Pong>().unwrap();
std::thread::sleep(Duration::from_millis(10));
handle.runtime.send_to(addr2, Ping { reply_to: *inbox2.addr() }).unwrap();
let reply2 = poll_inbox(&inbox2, Duration::from_secs(5));
assert!(reply2.is_some(), "message sent after run() must be delivered");
handle.shutdown();
handle.join();
// ── Part C: Shutdown wakes parked workers quickly ──
let rt2 = std_runtime(RuntimeConfig { num_threads: 4, ..Default::default() });
let h2 = rt2.run().unwrap();
std::thread::sleep(Duration::from_millis(50)); // Let workers park.
let before = Instant::now();
h2.shutdown();
h2.join();
let shutdown_time = before.elapsed();
assert!(
shutdown_time.as_millis() < 500,
"shutdown should complete quickly with parked workers, took {:?}",
shutdown_time
);
}
/// Sustained throughput with no message loss.
///
/// Story: We send 10 batches of 100 messages, ticking between batches on a
/// single-threaded runtime. Each batch must make forward progress, and after
/// draining, all 1000 messages are accounted for.
#[test]
fn sustained_throughput_no_message_loss() {
let rt = std_runtime(RuntimeConfig::default());
let counter = Arc::new(AtomicUsize::new(0));
let dummy = rt.new_inbox::<Pong>().unwrap();
let addr = rt.spawn(CountingPingActor { counter: counter.clone() }).unwrap();
for batch in 0..10 {
for _ in 0..100 {
rt.send_to(addr, Ping { reply_to: *dummy.addr() }).unwrap();
}
tick_n(&rt, 5);
let processed = counter.load(Ordering::SeqCst);
assert!(
processed > batch * 50,
"batch {batch}: expected progress, only {processed} processed"
);
}
// Drain remaining.
tick_n(&rt, 100);
let total = counter.load(Ordering::SeqCst);
assert_eq!(total, 1000, "sustained load should not drop any messages");
}
/// Load-aware actor placement.
///
/// Story: A fresh runtime falls back to round-robin (even distribution).
/// Under imbalanced load, new actors bias toward the lighter worker.
/// A single-worker runtime degrades gracefully.
#[test]
fn load_aware_actor_placement() {
// ── Part A: Round-robin fallback on fresh runtime (4 workers, 100 actors) ──
let rt = std_runtime(RuntimeConfig { num_threads: 4, ..Default::default() });
for _ in 0..100 {
rt.spawn(CounterActor { count: 0 }).unwrap();
}
let handle = rt.run().unwrap();
std::thread::sleep(Duration::from_millis(20));
let stats = handle.runtime.stats();
handle.shutdown();
handle.join();
for w in &stats.workers {
assert!(
w.num_actors >= 20 && w.num_actors <= 30,
"worker {} has {} actors, expected ~25 (round-robin)",
w.id, w.num_actors
);
}
// ── Part B: Imbalanced load biases toward lighter worker ──
let rt2 = std_runtime(RuntimeConfig { num_threads: 2, ..Default::default() });
let mut addrs = Vec::new();
for _ in 0..20 {
addrs.push(rt2.spawn(CounterActor { count: 0 }).unwrap());
}
let h2 = rt2.run().unwrap();
std::thread::sleep(Duration::from_millis(10));
// Bombard the first 10 actors (likely worker 0) with messages.
for addr in &addrs[..10] {
for _ in 0..50 {
let _ = h2.runtime.send_to(*addr, Increment { reply_to: *addr });
}
}
std::thread::sleep(Duration::from_millis(20));
// Spawn 10 more — should bias toward lighter worker.
for _ in 0..10 {
h2.runtime.spawn(CounterActor { count: 0 }).unwrap();
}
std::thread::sleep(Duration::from_millis(20));
let stats2 = h2.runtime.stats();
h2.shutdown();
h2.join();
let total_actors: usize = stats2.workers.iter().map(|w| w.num_actors).sum();
assert!(total_actors >= 20, "expected at least 20 actors, got {total_actors}");
assert!(
stats2.workers.iter().all(|w| w.num_actors > 0),
"both workers should have actors: {:?}",
stats2.workers.iter().map(|w| w.num_actors).collect::<Vec<_>>()
);
// ── Part C: Single-worker degrades gracefully ──
let rt3 = std_runtime(RuntimeConfig::default());
for _ in 0..50 {
rt3.spawn(CounterActor { count: 0 }).unwrap();
}
tick_n(&rt3, 10);
let stats3 = rt3.stats();
assert_eq!(stats3.workers.len(), 1);
assert_eq!(stats3.workers[0].num_actors, 50);
}

View file

@ -1,866 +0,0 @@
mod common;
use common::*;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
// ─── Supervisor Helpers ────────────────────────────────────────────────────
/// Actor that panics after receiving a configurable number of messages.
struct PanicAfterN {
trigger: usize,
count: usize,
counter: Arc<AtomicUsize>,
}
impl ActorInterface for PanicAfterN {
type Incoming = Ping;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: Ping) {
self.count += 1;
self.counter.fetch_add(1, Ordering::SeqCst);
let _ = ctx.send(msg.reply_to, Pong);
if self.count >= self.trigger {
panic!("intentional panic at message {}", self.count);
}
}
}
// --- handle_down tests ---
/// Given an actor with handle_down and a monitored target,
/// when the target dies, the watcher receives a Down via handle_down.
#[test]
fn handle_down_receives_death_notification() {
struct MonitoringTracker {
target: ActorAddress,
downs: Vec<Down>,
inbox: ActorAddress,
}
impl ActorInterface for MonitoringTracker {
type Incoming = Ping;
type Response = ();
fn on_start(&mut self, ctx: &Ctx) {
ctx.monitor(self.target);
}
fn handle(&mut self, ctx: &Ctx, _msg: Ping) {
let _ = ctx.send(self.inbox, Count(self.downs.len()));
}
fn handle_down(&mut self, _ctx: &Ctx, down: Down) {
self.downs.push(down);
}
}
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Count>().unwrap();
let inbox_addr = *inbox.addr();
let target = rt.spawn(PanicActor).unwrap();
let tracker = rt.spawn(MonitoringTracker {
target,
downs: vec![],
inbox: inbox_addr,
}).unwrap();
rt.tick(); // on_start for both
// Kill the target
rt.send_to(target, PanicMsg).unwrap();
rt.tick(); // target panics
rt.tick(); // Down delivered to tracker via handle_down
// Ask tracker how many downs it saw
rt.send_to(tracker, Ping { reply_to: inbox_addr }).unwrap();
rt.tick();
assert_eq!(inbox.try_recv(), Some(Count(1)));
}
/// Given an actor whose Incoming type IS Down, handle_down is NOT called --
/// the Down goes through the normal handle() method (backward compatibility).
#[test]
fn handle_down_skipped_when_incoming_is_down() {
struct DownAsIncoming {
target: ActorAddress,
inbox: ActorAddress,
}
impl ActorInterface for DownAsIncoming {
type Incoming = Down;
type Response = ();
fn on_start(&mut self, ctx: &Ctx) {
ctx.monitor(self.target);
}
fn handle(&mut self, ctx: &Ctx, msg: Down) {
let _ = ctx.send(self.inbox, msg);
}
fn handle_down(&mut self, _ctx: &Ctx, _down: Down) {
panic!("handle_down must not be called when Incoming=Down");
}
}
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Down>().unwrap();
let inbox_addr = *inbox.addr();
let target = rt.spawn(PanicActor).unwrap();
let _watcher = rt.spawn(DownAsIncoming { target, inbox: inbox_addr }).unwrap();
rt.tick(); // on_start
rt.send_to(target, PanicMsg).unwrap();
rt.tick(); // panic
rt.tick(); // Down delivered through handle(), not handle_down
let received = inbox.try_recv().expect("Down should be delivered via handle()");
assert_eq!(received.reason, StopReason::Panicked);
}
// --- ctx.stop_actor tests ---
/// Given two actors, one can stop the other via ctx.stop_actor().
#[test]
fn ctx_stop_actor_stops_target() {
#[derive(Clone)]
struct StopCmd {
target: ActorAddress,
}
struct Stopper;
impl ActorInterface for Stopper {
type Incoming = StopCmd;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: StopCmd) {
let _ = ctx.stop_actor(msg.target);
}
}
let rt = std_runtime(RuntimeConfig::default());
let target = rt.spawn(PingPongActor).unwrap();
let stopper = rt.spawn(Stopper).unwrap();
rt.tick(); // on_start
rt.send_to(stopper, StopCmd { target }).unwrap();
rt.tick(); // stopper handles StopCmd -> stop_actor(target)
rt.tick(); // StopSignal delivered to target, target stops
rt.tick(); // cleanup
assert!(rt.send_to(target, Ping { reply_to: ActorAddress::default() }).is_err());
// Stopper should still be alive
assert!(rt.send_to(stopper, StopCmd { target }).is_ok());
}
// --- Supervisor tests ---
/// Given a supervisor with one permanent child,
/// when the child panics, the supervisor restarts it.
#[test]
fn supervisor_restarts_permanent_child_on_panic() {
let counter = Arc::new(AtomicUsize::new(0));
let counter_c = counter.clone();
let inbox_holder: Arc<std::sync::Mutex<Option<ActorAddress>>> =
Arc::new(std::sync::Mutex::new(None));
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let inbox_addr = *inbox.addr();
*inbox_holder.lock().unwrap() = Some(inbox_addr);
let sup = Supervisor::new(
SupervisorStrategy::OneForOne,
5,
vec![ChildSpec::new("worker", RestartPolicy::Permanent, move |ctx| {
ctx.spawn(PanicAfterN {
trigger: 2, // panics on 2nd message
count: 0,
counter: counter_c.clone(),
})
})],
);
let _sup_addr = rt.spawn(sup).unwrap();
rt.tick(); // supervisor on_start -> spawns child
rt.tick(); // child on_start
// Find the child by checking stats
let stats = rt.stats();
assert_eq!(stats.workers[0].num_actors, 2); // supervisor + child
// Discover child address from stats
let child_addr = stats.actors.iter()
.find(|(addr, _)| *addr != _sup_addr)
.map(|(addr, _)| *addr)
.unwrap();
// First message: child processes, increments counter
rt.send_to(child_addr, Ping { reply_to: inbox_addr }).unwrap();
rt.tick();
assert_eq!(counter.load(Ordering::SeqCst), 1);
// Second message: child panics (trigger=2)
rt.send_to(child_addr, Ping { reply_to: inbox_addr }).unwrap();
rt.tick(); // child panics and is poisoned
rt.tick(); // cleanup: Down delivered to supervisor via handle_down
rt.tick(); // supervisor restarts child (spawns new one)
rt.tick(); // new child on_start
// Supervisor is still alive, and a new child exists
let stats = rt.stats();
assert_eq!(stats.workers[0].num_actors, 2); // supervisor + new child
}
/// Given a supervisor with a transient child,
/// when the child stops normally, it is NOT restarted.
#[test]
fn supervisor_does_not_restart_transient_child_on_normal_stop() {
let rt = std_runtime(RuntimeConfig::default());
struct StopsAfterFirst;
impl ActorInterface for StopsAfterFirst {
type Incoming = Ping;
type Response = ();
fn handle(&mut self, ctx: &Ctx, _msg: Ping) {
ctx.stop_self();
}
}
let sup = Supervisor::new(
SupervisorStrategy::OneForOne,
5,
vec![ChildSpec::new("worker", RestartPolicy::Transient, |ctx| {
ctx.spawn(StopsAfterFirst)
})],
);
let sup_addr = rt.spawn(sup).unwrap();
rt.tick(); // supervisor on_start -> child spawned
rt.tick(); // child on_start
let stats = rt.stats();
assert_eq!(stats.workers[0].num_actors, 2); // sup + child
// Find child address
let child_addr = stats.actors.iter()
.find(|(addr, _)| *addr != sup_addr)
.map(|(addr, _)| *addr)
.unwrap();
// Send message -- child stops itself
rt.send_to(child_addr, Ping { reply_to: ActorAddress::default() }).unwrap();
rt.tick(); // child handles, stops self
rt.tick(); // cleanup: Down(Normal) delivered to supervisor
rt.tick(); // supervisor sees Transient + Normal -> no restart
let stats = rt.stats();
assert_eq!(stats.workers[0].num_actors, 1); // only supervisor remains
}
/// Given a supervisor with a transient child,
/// when the child panics, it IS restarted.
#[test]
fn supervisor_restarts_transient_child_on_panic() {
let rt = std_runtime(RuntimeConfig::default());
let counter = Arc::new(AtomicUsize::new(0));
let counter_c = counter.clone();
let sup = Supervisor::new(
SupervisorStrategy::OneForOne,
5,
vec![ChildSpec::new("worker", RestartPolicy::Transient, move |ctx| {
ctx.spawn(PanicAfterN {
trigger: 1, // panics on first message
count: 0,
counter: counter_c.clone(),
})
})],
);
let sup_addr = rt.spawn(sup).unwrap();
rt.tick(); // supervisor starts, spawns child
rt.tick(); // child on_start
let child_addr = rt.stats().actors.iter()
.find(|(addr, _)| *addr != sup_addr)
.map(|(addr, _)| *addr)
.unwrap();
// Send message -- child panics
let inbox = rt.new_inbox::<Pong>().unwrap();
rt.send_to(child_addr, Ping { reply_to: *inbox.addr() }).unwrap();
rt.tick(); // child panics
rt.tick(); // Down(Panicked) -> supervisor restarts
rt.tick(); // new child spawned
rt.tick(); // new child on_start
// Supervisor + new child alive
let stats = rt.stats();
assert_eq!(stats.workers[0].num_actors, 2);
}
/// Given a supervisor with a temporary child,
/// when the child dies (any reason), it is never restarted.
#[test]
fn supervisor_never_restarts_temporary_child() {
let rt = std_runtime(RuntimeConfig::default());
let sup = Supervisor::new(
SupervisorStrategy::OneForOne,
5,
vec![ChildSpec::new("worker", RestartPolicy::Temporary, |ctx| {
ctx.spawn(PanicActor)
})],
);
let sup_addr = rt.spawn(sup).unwrap();
rt.tick(); // supervisor starts, spawns child
rt.tick(); // child on_start
let child_addr = rt.stats().actors.iter()
.find(|(addr, _)| *addr != sup_addr)
.map(|(addr, _)| *addr)
.unwrap();
// Kill the child
rt.send_to(child_addr, PanicMsg).unwrap();
rt.tick(); // panic
rt.tick(); // Down -> supervisor sees Temporary -> no restart
rt.tick(); // settle
let stats = rt.stats();
assert_eq!(stats.workers[0].num_actors, 1); // only supervisor
}
/// Given a supervisor with max_restarts=2,
/// when more than 2 restarts occur, the supervisor stops itself (meltdown).
#[test]
fn supervisor_meltdown_after_max_restarts() {
let rt = std_runtime(RuntimeConfig::default());
let counter = Arc::new(AtomicUsize::new(0));
let sup = Supervisor::new(
SupervisorStrategy::OneForOne,
2, // only 2 restarts allowed
vec![ChildSpec::new("crasher", RestartPolicy::Permanent, {
let counter = counter.clone();
move |ctx| {
ctx.spawn(PanicAfterN {
trigger: 1,
count: 0,
counter: counter.clone(),
})
}
})],
);
let sup_addr = rt.spawn(sup).unwrap();
rt.tick(); rt.tick(); // supervisor + child started
// Crash the child 3 times
for _ in 0..3 {
if let Some((child_addr, _)) = rt.stats().actors.iter()
.find(|(addr, _)| *addr != sup_addr)
{
let inbox = rt.new_inbox::<Pong>().unwrap();
let _ = rt.send_to(*child_addr, Ping { reply_to: *inbox.addr() });
rt.tick(); // child panics
rt.tick(); // Down delivered -> restart or meltdown
rt.tick(); // new child spawned (or supervisor stopped)
rt.tick(); // settle
}
}
// After 3 crashes with max_restarts=2, supervisor should have stopped itself
let stats = rt.stats();
let sup_alive = stats.actors.iter().any(|(addr, _)| *addr == sup_addr);
assert!(!sup_alive, "supervisor should have stopped after exceeding max_restarts");
}
/// Given a supervisor with multiple children,
/// when one child panics, only that child is restarted (OneForOne).
#[test]
fn supervisor_one_for_one_only_restarts_failed_child() {
let rt = std_runtime(RuntimeConfig::default());
let counter_a = Arc::new(AtomicUsize::new(0));
let counter_b = Arc::new(AtomicUsize::new(0));
let sup = Supervisor::new(
SupervisorStrategy::OneForOne,
5,
vec![
ChildSpec::new("crasher", RestartPolicy::Permanent, {
let c = counter_a.clone();
move |ctx| ctx.spawn_named("child_a", PanicAfterN {
trigger: 1, count: 0, counter: c.clone(),
})
}),
ChildSpec::new("stable", RestartPolicy::Permanent, {
let c = counter_b.clone();
move |ctx| ctx.spawn_named("child_b", CountingPingActor { counter: c.clone() })
}),
],
);
let _sup_addr = rt.spawn(sup).unwrap();
rt.tick(); rt.tick(); // start up
let child_a = rt.where_is("child_a").expect("child_a should be named");
let child_b = rt.where_is("child_b").expect("child_b should be named");
// Send to child_b to prove it's alive
let inbox = rt.new_inbox::<Pong>().unwrap();
rt.send_to(child_b, Ping { reply_to: *inbox.addr() }).unwrap();
rt.tick();
let b_processed_before = counter_b.load(Ordering::SeqCst);
assert!(b_processed_before >= 1);
// Crash child_a
rt.send_to(child_a, Ping { reply_to: *inbox.addr() }).unwrap();
rt.tick(); // child_a panics
rt.tick(); // Down -> supervisor restarts child_a
rt.tick(); rt.tick(); // new child spawned + on_start
// child_b should still be alive (same address, same name)
let child_b_after = rt.where_is("child_b").expect("child_b should still exist");
assert_eq!(child_b, child_b_after, "child_b address should be unchanged");
rt.send_to(child_b, Ping { reply_to: *inbox.addr() }).unwrap();
rt.tick();
assert!(counter_b.load(Ordering::SeqCst) > b_processed_before,
"child_b should still be processing messages");
// Supervisor + 2 children should be alive
assert_eq!(rt.stats().workers[0].num_actors, 3);
}
/// Given a OneForAll supervisor with 3 children,
/// when one child panics, ALL children are stopped and restarted in spec order.
#[test]
fn supervisor_one_for_all_restarts_all_on_single_failure() {
let rt = std_runtime(RuntimeConfig::default());
let counter_a = Arc::new(AtomicUsize::new(0));
let counter_b = Arc::new(AtomicUsize::new(0));
let counter_c = Arc::new(AtomicUsize::new(0));
let sup = Supervisor::new(
SupervisorStrategy::OneForAll,
5,
vec![
ChildSpec::new("a", RestartPolicy::Permanent, {
let c = counter_a.clone();
move |ctx| ctx.spawn_named("ofa_a", PanicAfterN {
trigger: 1, count: 0, counter: c.clone(),
})
}),
ChildSpec::new("b", RestartPolicy::Permanent, {
let c = counter_b.clone();
move |ctx| ctx.spawn_named("ofa_b", CountingPingActor { counter: c.clone() })
}),
ChildSpec::new("c", RestartPolicy::Permanent, {
let c = counter_c.clone();
move |ctx| ctx.spawn_named("ofa_c", CountingPingActor { counter: c.clone() })
}),
],
);
let _sup_addr = rt.spawn(sup).unwrap();
rt.tick(); rt.tick(); // startup
let old_b = rt.where_is("ofa_b").expect("ofa_b exists");
let old_c = rt.where_is("ofa_c").expect("ofa_c exists");
let child_a = rt.where_is("ofa_a").expect("ofa_a exists");
// Crash child_a
let inbox = rt.new_inbox::<Pong>().unwrap();
rt.send_to(child_a, Ping { reply_to: *inbox.addr() }).unwrap();
rt.tick(); // child_a panics
// supervisor receives Down(a) -> OneForAll -> stops b and c
for _ in 0..8 { rt.tick(); }
// All 3 children should be alive with NEW addresses
let stats = rt.stats();
assert_eq!(stats.workers[0].num_actors, 4); // sup + 3 new children
let new_b = rt.where_is("ofa_b").expect("ofa_b re-registered after restart");
let new_c = rt.where_is("ofa_c").expect("ofa_c re-registered after restart");
assert_ne!(old_b, new_b, "child_b should have a new address after restart");
assert_ne!(old_c, new_c, "child_c should have a new address after restart");
}
/// Given a RestForOne supervisor with children [a, b, c],
/// when child b panics, children b and c are restarted.
/// Child a is unaffected.
#[test]
fn supervisor_rest_for_one_restarts_rest_after_failed() {
let rt = std_runtime(RuntimeConfig::default());
let counter_a = Arc::new(AtomicUsize::new(0));
let counter_b = Arc::new(AtomicUsize::new(0));
let counter_c = Arc::new(AtomicUsize::new(0));
let sup = Supervisor::new(
SupervisorStrategy::RestForOne,
5,
vec![
ChildSpec::new("a", RestartPolicy::Permanent, {
let c = counter_a.clone();
move |ctx| ctx.spawn_named("rfo_a", CountingPingActor { counter: c.clone() })
}),
ChildSpec::new("b", RestartPolicy::Permanent, {
let c = counter_b.clone();
move |ctx| ctx.spawn_named("rfo_b", PanicAfterN {
trigger: 1, count: 0, counter: c.clone(),
})
}),
ChildSpec::new("c", RestartPolicy::Permanent, {
let c = counter_c.clone();
move |ctx| ctx.spawn_named("rfo_c", CountingPingActor { counter: c.clone() })
}),
],
);
let _sup_addr = rt.spawn(sup).unwrap();
rt.tick(); rt.tick(); // startup
let old_a = rt.where_is("rfo_a").expect("rfo_a exists");
let old_c = rt.where_is("rfo_c").expect("rfo_c exists");
let child_b = rt.where_is("rfo_b").expect("rfo_b exists");
// Crash child_b
let inbox = rt.new_inbox::<Pong>().unwrap();
rt.send_to(child_b, Ping { reply_to: *inbox.addr() }).unwrap();
rt.tick(); // child_b panics
for _ in 0..8 { rt.tick(); }
// All 3 children should be alive
let stats = rt.stats();
assert_eq!(stats.workers[0].num_actors, 4); // sup + 3 children
// child_a should be UNCHANGED
let new_a = rt.where_is("rfo_a").expect("rfo_a still exists");
assert_eq!(old_a, new_a, "child_a should not be restarted in RestForOne when b fails");
// child_c should have a NEW address
let new_c = rt.where_is("rfo_c").expect("rfo_c re-registered");
assert_ne!(old_c, new_c, "child_c should have a new address after RestForOne restart");
}
/// Given a OneForAll supervisor, when the last child of the failed set confirms death,
/// all children are restarted in spec order.
#[test]
fn supervisor_one_for_all_waits_for_all_downs_before_restart() {
let rt = std_runtime(RuntimeConfig::default());
let sup = Supervisor::new(
SupervisorStrategy::OneForAll,
5,
vec![
ChildSpec::new("x", RestartPolicy::Permanent, |ctx| ctx.spawn(PingPongActor)),
ChildSpec::new("y", RestartPolicy::Permanent, |ctx| ctx.spawn(PingPongActor)),
],
);
let sup_addr = rt.spawn(sup).unwrap();
rt.tick(); rt.tick(); // startup
assert_eq!(rt.stats().workers[0].num_actors, 3); // sup + 2 children
// Stop one child
let actors: Vec<_> = rt.stats().actors.iter()
.filter(|(addr, _)| *addr != sup_addr)
.map(|(addr, _)| *addr)
.collect();
rt.stop_actor(actors[0]).unwrap();
// Tick enough times for full cycle
for _ in 0..10 { rt.tick(); }
// Should have supervisor + 2 new children
assert_eq!(rt.stats().workers[0].num_actors, 3);
}
/// Given a supervisor that stops, its children also stop.
#[test]
fn supervisor_on_stop_kills_children() {
let rt = std_runtime(RuntimeConfig::default());
let sup = Supervisor::new(
SupervisorStrategy::OneForOne,
5,
vec![
ChildSpec::new("a", RestartPolicy::Permanent, |ctx| ctx.spawn(PingPongActor)),
ChildSpec::new("b", RestartPolicy::Permanent, |ctx| ctx.spawn(PingPongActor)),
],
);
let sup_addr = rt.spawn(sup).unwrap();
rt.tick(); rt.tick(); // start up
assert_eq!(rt.stats().workers[0].num_actors, 3); // sup + 2 children
// Stop the supervisor
rt.stop_actor(sup_addr).unwrap();
rt.tick(); // StopSignal delivered to supervisor, on_stop sends stop to children
rt.tick(); // supervisor cleaned up, stop signals delivered to children
rt.tick(); // children stop
rt.tick(); // children cleaned up
assert_eq!(rt.stats().workers[0].num_actors, 0);
}
// ── Router tests ─────────────────────────────────────────────────────────────
#[test]
fn router_round_robin_distributes_across_workers() {
let rt = std_runtime(RuntimeConfig::default());
let collected = Arc::new(std::sync::Mutex::new(Vec::new()));
struct Collector(Arc<std::sync::Mutex<Vec<(ActorAddress, usize)>>>);
#[derive(Clone)]
struct Work(usize);
impl ActorInterface for Collector {
type Incoming = Work;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: Work) {
self.0.lock().unwrap().push((ctx.self_addr(), msg.0));
}
}
let c = collected.clone();
let router = Router::<Work>::new(
RoutingStrategy::RoundRobin,
3,
move |ctx| ctx.spawn(Collector(c.clone())),
10,
);
let router_addr = rt.spawn(router).unwrap();
rt.tick(); // on_start spawns 3 workers
for i in 0..6 {
rt.send_to(router_addr, Work(i)).unwrap();
}
rt.tick(); // router receives 6 Work messages, forwards to workers
rt.tick(); // workers process their messages
let data = collected.lock().unwrap();
assert_eq!(data.len(), 6);
// Count how many unique workers received messages
let mut per_worker = std::collections::HashMap::new();
for (addr, _) in data.iter() {
*per_worker.entry(*addr).or_insert(0usize) += 1;
}
// All 3 workers should have received exactly 2 messages each
assert_eq!(per_worker.len(), 3);
for count in per_worker.values() {
assert_eq!(*count, 2);
}
}
#[test]
fn router_broadcast_sends_to_all_workers() {
let rt = std_runtime(RuntimeConfig::default());
let count = Arc::new(AtomicUsize::new(0));
struct Counter(Arc<AtomicUsize>);
#[derive(Clone)]
struct Ping;
impl ActorInterface for Counter {
type Incoming = Ping;
type Response = ();
fn handle(&mut self, _ctx: &Ctx, _msg: Ping) {
self.0.fetch_add(1, Ordering::Relaxed);
}
}
let c = count.clone();
let router = Router::<Ping>::new(
RoutingStrategy::Broadcast,
3,
move |ctx| ctx.spawn(Counter(c.clone())),
10,
);
let router_addr = rt.spawn(router).unwrap();
rt.tick(); // on_start spawns workers
rt.send_to(router_addr, Ping).unwrap();
rt.tick(); // router broadcasts
rt.tick(); // workers process
assert_eq!(count.load(Ordering::Relaxed), 3);
}
#[test]
fn router_random_delivers_to_some_worker() {
let rt = std_runtime(RuntimeConfig::default());
let collected = Arc::new(std::sync::Mutex::new(Vec::new()));
struct Collector(Arc<std::sync::Mutex<Vec<ActorAddress>>>);
#[derive(Clone)]
struct Work;
impl ActorInterface for Collector {
type Incoming = Work;
type Response = ();
fn handle(&mut self, ctx: &Ctx, _msg: Work) {
self.0.lock().unwrap().push(ctx.self_addr());
}
}
let c = collected.clone();
let router = Router::<Work>::new(
RoutingStrategy::Random,
3,
move |ctx| ctx.spawn(Collector(c.clone())),
10,
);
let router_addr = rt.spawn(router).unwrap();
rt.tick();
for _ in 0..30 {
rt.send_to(router_addr, Work).unwrap();
}
rt.tick();
rt.tick();
let data = collected.lock().unwrap();
assert_eq!(data.len(), 30);
let unique: std::collections::HashSet<_> = data.iter().collect();
assert!(unique.len() >= 2, "expected at least 2 workers used, got {}", unique.len());
}
#[test]
fn router_replaces_dead_worker() {
let rt = std_runtime(RuntimeConfig::default());
let spawn_count = Arc::new(AtomicUsize::new(0));
struct PanicOnFirst {
first: bool,
}
#[derive(Clone)]
struct Work;
impl ActorInterface for PanicOnFirst {
type Incoming = Work;
type Response = ();
fn handle(&mut self, _ctx: &Ctx, _msg: Work) {
if self.first {
self.first = false;
panic!("first message panic");
}
}
}
let sc = spawn_count.clone();
let router = Router::<Work>::new(
RoutingStrategy::RoundRobin,
3,
move |ctx| {
let n = sc.fetch_add(1, Ordering::Relaxed);
ctx.spawn(PanicOnFirst { first: n == 0 })
},
10,
);
let router_addr = rt.spawn(router).unwrap();
rt.tick(); // spawn workers (3 spawned)
assert_eq!(spawn_count.load(Ordering::Relaxed), 3);
// Send a message that will hit worker 0
rt.send_to(router_addr, Work).unwrap();
rt.tick(); // router forwards to worker 0
rt.tick(); // worker 0 panics
rt.tick(); // cleanup + Down delivered to router
rt.tick(); // router spawns replacement
rt.tick(); // replacement starts
// Should have spawned 4 total (3 original + 1 replacement)
assert_eq!(spawn_count.load(Ordering::Relaxed), 4);
// Verify all 3 slots are live
assert_eq!(rt.stats().workers[0].num_actors, 4);
}
#[test]
fn router_meltdown_after_max_restarts() {
let rt = std_runtime(RuntimeConfig::default());
struct AlwaysPanics;
#[derive(Clone)]
struct Work;
impl ActorInterface for AlwaysPanics {
type Incoming = Work;
type Response = ();
fn handle(&mut self, _ctx: &Ctx, _msg: Work) {
panic!("always");
}
}
let router = Router::<Work>::new(
RoutingStrategy::RoundRobin,
1,
|ctx| ctx.spawn(AlwaysPanics),
2, // max 2 restarts
);
let router_addr = rt.spawn(router).unwrap();
rt.tick(); // on_start
// Kill the worker 3 times (> max_restarts=2)
for _ in 0..3 {
rt.send_to(router_addr, Work).unwrap();
for _ in 0..5 {
rt.tick();
}
}
// After 3 restarts, router should have shut down
for _ in 0..5 {
rt.tick();
}
assert_eq!(rt.stats().workers[0].num_actors, 0);
}
#[test]
fn router_on_stop_kills_workers() {
let rt = std_runtime(RuntimeConfig::default());
struct Dummy;
#[derive(Clone)]
struct Work;
impl ActorInterface for Dummy {
type Incoming = Work;
type Response = ();
fn handle(&mut self, _ctx: &Ctx, _msg: Work) {}
}
let router = Router::<Work>::new(
RoutingStrategy::RoundRobin,
3,
|ctx| ctx.spawn(Dummy),
10,
);
let router_addr = rt.spawn(router).unwrap();
rt.tick(); // on_start
assert_eq!(rt.stats().workers[0].num_actors, 4); // router + 3 workers
rt.stop_actor(router_addr).unwrap();
for _ in 0..5 {
rt.tick();
}
assert_eq!(rt.stats().workers[0].num_actors, 0);
}
#[test]
fn router_broadcast_multiple_messages_all_received() {
let rt = std_runtime(RuntimeConfig::default());
let total = Arc::new(AtomicUsize::new(0));
struct Sink(Arc<AtomicUsize>);
#[derive(Clone)]
struct Tick;
impl ActorInterface for Sink {
type Incoming = Tick;
type Response = ();
fn handle(&mut self, _ctx: &Ctx, _msg: Tick) {
self.0.fetch_add(1, Ordering::Relaxed);
}
}
let t = total.clone();
let router = Router::<Tick>::new(
RoutingStrategy::Broadcast,
3,
move |ctx| ctx.spawn(Sink(t.clone())),
10,
);
let router_addr = rt.spawn(router).unwrap();
rt.tick();
for _ in 0..5 {
rt.send_to(router_addr, Tick).unwrap();
}
rt.tick(); // router broadcasts
rt.tick(); // workers process
assert_eq!(total.load(Ordering::Relaxed), 15);
}

View file

@ -1,213 +0,0 @@
mod common;
use common::*;
// ── Timer Helpers ─────────────────────────────────────────────────────────
/// Actor that schedules a one-shot timer in on_start: sends a Ping to target after N ticks.
struct TimerStartActor {
target: ActorAddress,
delay_ticks: u64,
}
impl ActorInterface for TimerStartActor {
type Incoming = Ping;
type Response = Pong;
fn on_start(&mut self, ctx: &Ctx) {
ctx.send_after_ticks(self.target, Ping { reply_to: ctx.self_addr() }, self.delay_ticks);
}
fn handle(&mut self, _ctx: &Ctx, _msg: Ping) {}
}
/// Actor that schedules a one-shot timer when it receives a Forward message.
struct DelayPingPongActor;
impl ActorInterface for DelayPingPongActor {
type Incoming = Forward;
type Response = Done;
fn handle(&mut self, ctx: &Ctx, msg: Forward) {
ctx.send_after_ticks(msg.reply_to, Done(msg.value), 3);
}
}
/// Actor that schedules an interval timer on start: sends Ping every N ticks.
struct HeartbeatActor {
target: ActorAddress,
period: u64,
}
impl ActorInterface for HeartbeatActor {
type Incoming = Ping;
type Response = Pong;
fn on_start(&mut self, ctx: &Ctx) {
ctx.send_interval_ticks(self.target, Ping { reply_to: ctx.self_addr() }, self.period);
}
fn handle(&mut self, _ctx: &Ctx, _msg: Ping) {}
}
// ── Timer Tests ───────────────────────────────────────────────────────────
/// Given an actor that schedules a one-shot timer in on_start,
/// when enough ticks pass,
/// then the timer message is delivered to the target.
#[test]
fn one_shot_timer_fires_after_n_ticks() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Ping>().unwrap();
let _timer_actor = rt.spawn(TimerStartActor {
target: *inbox.addr(),
delay_ticks: 3,
}).unwrap();
// Tick 1: on_start schedules timer (fire_at = current_tick + 3 = 4)
rt.tick(); // tick 1: on_start, timer scheduled
assert!(inbox.try_recv().is_none(), "no delivery before delay");
rt.tick(); // tick 2
assert!(inbox.try_recv().is_none(), "no delivery on tick 2");
rt.tick(); // tick 3
assert!(inbox.try_recv().is_none(), "no delivery on tick 3");
rt.tick(); // tick 4: timer fires
let msg = inbox.try_recv();
assert!(msg.is_some(), "timer message delivered after 3-tick delay");
}
/// Given an actor that schedules a one-shot timer from a message handler,
/// when enough ticks pass after the triggering message,
/// then the delayed response arrives.
#[test]
fn handler_can_schedule_one_shot_timer() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Done>().unwrap();
let addr = rt.spawn(DelayPingPongActor).unwrap();
let _ = rt.send_to(addr, Forward { value: 42, reply_to: *inbox.addr() });
rt.tick(); // process Forward, schedule timer (delay=3)
assert!(inbox.try_recv().is_none(), "no immediate reply");
rt.tick(); // tick 2
rt.tick(); // tick 3
assert!(inbox.try_recv().is_none(), "not yet");
rt.tick(); // tick 4: timer fires
let reply = inbox.try_recv();
assert_eq!(reply, Some(Done(42)), "delayed reply arrives after 3 ticks");
}
/// Given a one-shot timer,
/// when it fires,
/// then it does NOT fire again on subsequent ticks (consumed).
#[test]
fn one_shot_timer_fires_only_once() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Ping>().unwrap();
let _timer_actor = rt.spawn(TimerStartActor {
target: *inbox.addr(),
delay_ticks: 1,
}).unwrap();
rt.tick(); // on_start schedules timer
rt.tick(); // timer fires
assert!(inbox.try_recv().is_some(), "first fire");
// Subsequent ticks should NOT fire again
for _ in 0..5 { rt.tick(); }
assert!(inbox.try_recv().is_none(), "one-shot does not repeat");
}
/// Given an interval timer with period 2,
/// when multiple ticks pass,
/// then the timer fires repeatedly every 2 ticks.
#[test]
fn interval_timer_fires_repeatedly() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Ping>().unwrap();
let _heartbeat = rt.spawn(HeartbeatActor {
target: *inbox.addr(),
period: 2,
}).unwrap();
rt.tick(); // tick 1: on_start, interval scheduled (next_fire = current + 2 = 3)
assert!(inbox.try_recv().is_none(), "no fire on tick 1");
rt.tick(); // tick 2
assert!(inbox.try_recv().is_none(), "no fire on tick 2");
rt.tick(); // tick 3: first fire
assert!(inbox.try_recv().is_some(), "fire on tick 3");
rt.tick(); // tick 4
assert!(inbox.try_recv().is_none(), "no fire on tick 4");
rt.tick(); // tick 5: second fire
assert!(inbox.try_recv().is_some(), "fire on tick 5");
rt.tick(); // tick 6
assert!(inbox.try_recv().is_none(), "no fire on tick 6");
rt.tick(); // tick 7: third fire
assert!(inbox.try_recv().is_some(), "fire on tick 7");
}
/// Given an interval timer targeting an actor that gets stopped,
/// when the actor is removed,
/// then the interval timer is cleaned up (no orphan timers).
#[test]
fn interval_timer_cleaned_up_when_actor_dies() {
let rt = std_runtime(RuntimeConfig::default());
let _inbox = rt.new_inbox::<Ping>().unwrap();
// Heartbeat sends to a counter that we'll kill
let counter_addr = rt.spawn(CounterActor { count: 0 }).unwrap();
// HeartbeatActor sends Ping to counter every tick
let _hb = rt.spawn(HeartbeatActor {
target: counter_addr,
period: 1,
}).unwrap();
// Let it run a few ticks
for _ in 0..3 { rt.tick(); }
// Stop the counter
rt.stop_actor(counter_addr).unwrap();
for _ in 0..5 { rt.tick(); }
// Counter is gone, interval timer should be GC'd.
let stats = rt.stats();
// Only the heartbeat actor should remain
assert_eq!(stats.workers[0].num_actors, 1);
}
/// Given a timer with delay 0,
/// when the next tick fires,
/// then the message is delivered immediately on the next tick.
#[test]
fn timer_with_zero_delay_fires_next_tick() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Ping>().unwrap();
let _timer_actor = rt.spawn(TimerStartActor {
target: *inbox.addr(),
delay_ticks: 0,
}).unwrap();
rt.tick(); // on_start schedules timer with delay=0
// Timer requests are processed after tick_all (phase 5.5)
// Timer fires on the NEXT tick (phase 2.5)
assert!(inbox.try_recv().is_none(), "not yet -- timer fires next tick");
rt.tick(); // timer fires
assert!(inbox.try_recv().is_some(), "zero-delay timer fires on next tick");
}

742
tests/std_extension.rs Normal file
View file

@ -0,0 +1,742 @@
//! StdExtension Tests — higher-level patterns from swactor-std.
//!
//! Covers: naming registry, groups/pub-sub, ask pattern, supervision
//! strategies and restart policies, and router work distribution.
mod common;
use common::*;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
// ── Local actors ────────────────────────────────────────────────────────────
/// Looks up a peer by name using ctx.where_is().
struct NameLookupActor {
target_name: &'static str,
reply_to: ActorAddress,
}
impl ActorInterface for NameLookupActor {
type Incoming = Ping;
type Response = ();
fn handle(&mut self, ctx: &Ctx, _msg: Ping) {
if let Some(peer) = ctx.where_is(self.target_name) {
ctx.send(self.reply_to, MyAddr(peer)).unwrap();
}
}
}
/// Spawns a named child from a handler.
struct NamedSpawnerActor {
reply_to: ActorAddress,
}
impl ActorInterface for NamedSpawnerActor {
type Incoming = Ping;
type Response = ();
fn handle(&mut self, ctx: &Ctx, _msg: Ping) {
if let Ok(addr) = ctx.spawn_named("child", PingPongActor) {
ctx.send(self.reply_to, MyAddr(addr)).unwrap();
}
}
}
/// Panics after `trigger` messages.
struct PanicAfterN {
trigger: usize,
count: usize,
counter: Arc<AtomicUsize>,
}
impl ActorInterface for PanicAfterN {
type Incoming = Ping;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: Ping) {
self.count += 1;
self.counter.fetch_add(1, Ordering::SeqCst);
let _ = ctx.send(msg.reply_to, Pong);
if self.count >= self.trigger {
panic!("intentional panic at message {}", self.count);
}
}
}
/// Stops itself on first message.
struct StopsAfterFirst;
impl ActorInterface for StopsAfterFirst {
type Incoming = Ping;
type Response = ();
fn handle(&mut self, ctx: &Ctx, _msg: Ping) {
ctx.stop_self();
}
}
// ═══════════════════════════════════════════════════════════════════════════
// Naming Registry
// ═══════════════════════════════════════════════════════════════════════════
/// Full naming lifecycle: register, lookup, send, duplicate fails, auto-unregister
/// on stop and panic, name reuse, registered_names list, manual unregister.
#[test]
fn naming_registry_lifecycle() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
// Register "alice", lookup, send Ping → Pong
let alice = rt.spawn_named("alice", PingPongActor).unwrap();
assert_eq!(rt.where_is("alice"), Some(alice));
rt.send_to(alice, Ping { reply_to: *inbox.addr() }).unwrap();
rt.tick();
assert!(inbox.try_recv().is_some(), "named actor processes messages");
// Duplicate fails, original binding preserved
assert!(rt.spawn_named("alice", PingPongActor).is_err());
assert_eq!(rt.where_is("alice"), Some(alice));
// Unknown name → None
assert_eq!(rt.where_is("ghost"), None);
// Stop "alice" → name freed
rt.stop_actor(alice).unwrap();
rt.tick();
assert_eq!(rt.where_is("alice"), None, "name freed after stop");
// Reuse the name
let alice2 = rt.spawn_named("alice", PingPongActor).unwrap();
assert_ne!(alice, alice2);
assert_eq!(rt.where_is("alice"), Some(alice2));
// Panic also frees the name
let bob = rt.spawn_named("bob", PanicActor).unwrap();
rt.tick();
rt.send_to(bob, PanicMsg).unwrap();
rt.tick();
assert_eq!(rt.where_is("bob"), None, "name freed after panic");
let _bob2 = rt.spawn_named("bob", PingPongActor).unwrap();
assert!(rt.where_is("bob").is_some());
// registered_names enumerates all
rt.spawn_named("gamma", PingPongActor).unwrap();
let mut names = rt.registered_names();
names.sort();
assert!(names.contains(&"alice".to_string()));
assert!(names.contains(&"bob".to_string()));
assert!(names.contains(&"gamma".to_string()));
// Manual unregister: name freed but actor lives
let charlie_inbox = rt.new_inbox::<Pong>().unwrap();
let charlie = rt.spawn_named("charlie", PingPongActor).unwrap();
rt.tick();
let removed = rt.unregister("charlie");
assert_eq!(removed, Some(charlie));
assert_eq!(rt.where_is("charlie"), None, "name freed by unregister");
rt.send_to(charlie, Ping { reply_to: *charlie_inbox.addr() }).unwrap();
rt.tick();
assert!(charlie_inbox.try_recv().is_some(), "actor still alive after name unregistered");
}
/// Actors resolve and register names from handlers using ctx.
#[test]
fn naming_from_actor_handlers() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<MyAddr>().unwrap();
// ctx.where_is from handler
let target = rt.spawn_named("target", PingPongActor).unwrap();
let looker = rt.spawn(NameLookupActor {
target_name: "target",
reply_to: *inbox.addr(),
}).unwrap();
rt.tick();
rt.send_to(looker, Ping { reply_to: ActorAddress::default() }).unwrap();
tick_n(&rt, 3);
assert_eq!(inbox.try_recv(), Some(MyAddr(target)), "ctx.where_is resolves");
// ctx.spawn_named from handler
let spawner = rt.spawn(NamedSpawnerActor { reply_to: *inbox.addr() }).unwrap();
rt.tick();
rt.send_to(spawner, Ping { reply_to: ActorAddress::default() }).unwrap();
tick_n(&rt, 3);
let child_addr = inbox.try_recv().expect("child address returned");
assert_eq!(rt.where_is("child"), Some(child_addr.0), "name registered from handler");
}
// ═══════════════════════════════════════════════════════════════════════════
// Groups / Pub-Sub
// ═══════════════════════════════════════════════════════════════════════════
/// Full groups lifecycle: join, publish broadcasts, leave stops delivery,
/// dead actor auto-removed, multi-group cleanup, empty group deleted,
/// join and publish from handlers.
#[test]
fn groups_pub_sub_lifecycle() {
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
// Join 3 actors, publish → all 3 get it
let a = rt.spawn(PingPongActor).unwrap();
let b = rt.spawn(PingPongActor).unwrap();
let c = rt.spawn(PingPongActor).unwrap();
rt.join_group(a, "workers");
rt.join_group(b, "workers");
rt.join_group(c, "workers");
rt.tick();
let count = rt.publish_to("workers", Ping { reply_to: *inbox.addr() });
assert_eq!(count, 3, "3 members, 3 messages sent");
rt.tick();
let mut pongs = 0;
while inbox.try_recv().is_some() { pongs += 1; }
assert_eq!(pongs, 3, "all 3 received");
// Leave stops delivery
rt.leave_group(c, "workers");
let count = rt.publish_to("workers", Ping { reply_to: *inbox.addr() });
assert_eq!(count, 2, "2 after leave");
rt.tick();
let mut pongs = 0;
while inbox.try_recv().is_some() { pongs += 1; }
assert_eq!(pongs, 2);
// Dead actor auto-removed
rt.stop_actor(b).unwrap();
rt.tick();
let count = rt.publish_to("workers", Ping { reply_to: *inbox.addr() });
assert_eq!(count, 1, "dead actor removed");
// Multi-group cleanup: actor in alpha/beta/gamma dies → all cleaned
let rt = std_runtime(RuntimeConfig::default());
let actor = rt.spawn(PingPongActor).unwrap();
rt.join_group(actor, "alpha");
rt.join_group(actor, "beta");
rt.join_group(actor, "gamma");
rt.tick();
rt.stop_actor(actor).unwrap();
rt.tick();
assert!(rt.group_members("alpha").is_empty());
assert!(rt.group_members("beta").is_empty());
assert!(rt.group_members("gamma").is_empty());
// Empty group auto-deleted
let rt = std_runtime(RuntimeConfig::default());
let actor = rt.spawn(PingPongActor).unwrap();
rt.join_group(actor, "temp");
assert!(rt.groups().contains(&"temp".to_string()));
rt.leave_group(actor, "temp");
assert!(!rt.groups().contains(&"temp".to_string()), "empty group removed");
// Empty group query
let rt = std_runtime(RuntimeConfig::default());
assert!(rt.group_members("nonexistent").is_empty());
// ctx.join_group from on_start
struct GroupJoiner;
impl ActorInterface for GroupJoiner {
type Incoming = Ping;
type Response = ();
fn on_start(&mut self, ctx: &Ctx) {
ctx.join_group("auto-joined");
}
fn handle(&mut self, _ctx: &Ctx, _msg: Ping) {}
}
let rt = std_runtime(RuntimeConfig::default());
let x = rt.spawn(GroupJoiner).unwrap();
let y = rt.spawn(GroupJoiner).unwrap();
rt.tick();
let members = rt.group_members("auto-joined");
assert_eq!(members.len(), 2);
assert!(members.contains(&x));
assert!(members.contains(&y));
// ctx.publish from handler
#[derive(Clone)]
struct BroadcastCmd { reply_to: ActorAddress }
struct Broadcaster;
impl ActorInterface for Broadcaster {
type Incoming = BroadcastCmd;
type Response = ();
fn on_start(&mut self, ctx: &Ctx) {
ctx.join_group("bcast");
}
fn handle(&mut self, ctx: &Ctx, msg: BroadcastCmd) {
ctx.publish("bcast", Ping { reply_to: msg.reply_to });
}
}
let rt = std_runtime(RuntimeConfig::default());
let inbox = rt.new_inbox::<Pong>().unwrap();
let p1 = rt.spawn(PingPongActor).unwrap();
let p2 = rt.spawn(PingPongActor).unwrap();
rt.join_group(p1, "bcast");
rt.join_group(p2, "bcast");
let broadcaster = rt.spawn(Broadcaster).unwrap();
rt.tick();
rt.send_to(broadcaster, BroadcastCmd { reply_to: *inbox.addr() }).unwrap();
tick_n(&rt, 3);
let mut pongs = 0;
while inbox.try_recv().is_some() { pongs += 1; }
assert!(pongs >= 2, "at least 2 PingPong members replied, got {pongs}");
}
// ═══════════════════════════════════════════════════════════════════════════
// Ask Pattern
// ═══════════════════════════════════════════════════════════════════════════
/// Ask pattern: basic ask, repeated asks track state, try_recv before/after
/// tick, dead actor times out.
#[test]
fn ask_pattern() {
let rt = std_runtime(RuntimeConfig::default());
// Basic ask
let actor = rt.spawn(PingPongActor).unwrap();
rt.tick();
let pong: Pong = rt.ask(actor, |reply_to| Ping { reply_to })
.unwrap().recv_ticking(&rt, 10).unwrap();
assert_eq!(pong, Pong);
// Repeated asks track state
let counter = rt.spawn(CounterActor { count: 0 }).unwrap();
rt.tick();
let c1: Count = rt.ask(counter, |reply_to| Increment { reply_to })
.unwrap().recv_ticking(&rt, 10).unwrap();
let c2: Count = rt.ask(counter, |reply_to| Increment { reply_to })
.unwrap().recv_ticking(&rt, 10).unwrap();
let c3: Count = rt.ask(counter, |reply_to| Increment { reply_to })
.unwrap().recv_ticking(&rt, 10).unwrap();
assert_eq!((c1, c2, c3), (Count(1), Count(2), Count(3)));
// try_recv: None before tick, Some after
let rt = std_runtime(RuntimeConfig::default());
let actor = rt.spawn(PingPongActor).unwrap();
rt.tick();
let ask = rt.ask::<Ping, Pong>(actor, |reply_to| Ping { reply_to }).unwrap();
assert!(ask.try_recv().is_none(), "no response before tick");
rt.tick();
assert_eq!(ask.try_recv(), Some(Pong));
// Dead actor → timeout
let rt = std_runtime(RuntimeConfig::default());
let actor = rt.spawn(PingPongActor).unwrap();
rt.tick();
rt.stop_actor(actor).unwrap();
rt.tick();
if let Ok(ask) = rt.ask::<Ping, Pong>(actor, |reply_to| Ping { reply_to }) {
assert!(ask.recv_ticking(&rt, 5).is_err(), "timeout with dead actor");
}
}
// ═══════════════════════════════════════════════════════════════════════════
// Supervision
// ═══════════════════════════════════════════════════════════════════════════
/// Restart policies: permanent always restarts, transient only on panic,
/// temporary never restarts, meltdown after max_restarts.
#[test]
fn supervision_restart_policies() {
// Permanent child panics → restarted
let rt = std_runtime(RuntimeConfig::default());
let counter = Arc::new(AtomicUsize::new(0));
let counter_c = counter.clone();
let inbox = rt.new_inbox::<Pong>().unwrap();
let sup = Supervisor::new(
SupervisorStrategy::OneForOne, 5,
vec![ChildSpec::new("worker", RestartPolicy::Permanent, move |ctx| {
ctx.spawn(PanicAfterN { trigger: 2, count: 0, counter: counter_c.clone() })
})],
);
let sup_addr = rt.spawn(sup).unwrap();
tick_n(&rt, 2);
let child = rt.stats().actors.iter()
.find(|(a, _)| *a != sup_addr).map(|(a, _)| *a).unwrap();
rt.send_to(child, Ping { reply_to: *inbox.addr() }).unwrap();
rt.tick();
assert_eq!(counter.load(Ordering::SeqCst), 1);
rt.send_to(child, Ping { reply_to: *inbox.addr() }).unwrap();
tick_n(&rt, 5); // panics, supervisor restarts
assert_eq!(rt.stats().workers[0].num_actors, 2, "supervisor + restarted child");
// Transient stops normally → NOT restarted
let rt = std_runtime(RuntimeConfig::default());
let sup = Supervisor::new(
SupervisorStrategy::OneForOne, 5,
vec![ChildSpec::new("worker", RestartPolicy::Transient, |ctx| {
ctx.spawn(StopsAfterFirst)
})],
);
let sup_addr = rt.spawn(sup).unwrap();
tick_n(&rt, 2);
let child = rt.stats().actors.iter()
.find(|(a, _)| *a != sup_addr).map(|(a, _)| *a).unwrap();
rt.send_to(child, Ping { reply_to: ActorAddress::default() }).unwrap();
tick_n(&rt, 4);
assert_eq!(rt.stats().workers[0].num_actors, 1, "transient+normal → no restart");
// Transient panics → restarted
let rt = std_runtime(RuntimeConfig::default());
let counter = Arc::new(AtomicUsize::new(0));
let counter_c = counter.clone();
let sup = Supervisor::new(
SupervisorStrategy::OneForOne, 5,
vec![ChildSpec::new("worker", RestartPolicy::Transient, move |ctx| {
ctx.spawn(PanicAfterN { trigger: 1, count: 0, counter: counter_c.clone() })
})],
);
let sup_addr = rt.spawn(sup).unwrap();
tick_n(&rt, 2);
let child = rt.stats().actors.iter()
.find(|(a, _)| *a != sup_addr).map(|(a, _)| *a).unwrap();
let inbox = rt.new_inbox::<Pong>().unwrap();
rt.send_to(child, Ping { reply_to: *inbox.addr() }).unwrap();
tick_n(&rt, 5);
assert_eq!(rt.stats().workers[0].num_actors, 2, "transient+panic → restarted");
// Temporary never restarts
let rt = std_runtime(RuntimeConfig::default());
let sup = Supervisor::new(
SupervisorStrategy::OneForOne, 5,
vec![ChildSpec::new("worker", RestartPolicy::Temporary, |ctx| ctx.spawn(PanicActor))],
);
let sup_addr = rt.spawn(sup).unwrap();
tick_n(&rt, 2);
let child = rt.stats().actors.iter()
.find(|(a, _)| *a != sup_addr).map(|(a, _)| *a).unwrap();
rt.send_to(child, PanicMsg).unwrap();
tick_n(&rt, 4);
assert_eq!(rt.stats().workers[0].num_actors, 1, "temporary → no restart");
// Meltdown: max_restarts=2, crash 3 times → supervisor stops
let rt = std_runtime(RuntimeConfig::default());
let counter = Arc::new(AtomicUsize::new(0));
let sup = Supervisor::new(
SupervisorStrategy::OneForOne, 2,
vec![ChildSpec::new("crasher", RestartPolicy::Permanent, {
let c = counter.clone();
move |ctx| ctx.spawn(PanicAfterN { trigger: 1, count: 0, counter: c.clone() })
})],
);
let sup_addr = rt.spawn(sup).unwrap();
tick_n(&rt, 2);
for _ in 0..3 {
if let Some((child, _)) = rt.stats().actors.iter()
.find(|(a, _)| *a != sup_addr)
{
let inbox = rt.new_inbox::<Pong>().unwrap();
let _ = rt.send_to(*child, Ping { reply_to: *inbox.addr() });
tick_n(&rt, 5);
}
}
let sup_alive = rt.stats().actors.iter().any(|(a, _)| *a == sup_addr);
assert!(!sup_alive, "supervisor stopped after exceeding max_restarts");
}
/// Strategies: OneForOne, OneForAll, RestForOne. Stopping supervisor kills children.
#[test]
fn supervision_strategies() {
// OneForOne: only failed child restarted
let rt = std_runtime(RuntimeConfig::default());
let counter_a = Arc::new(AtomicUsize::new(0));
let counter_b = Arc::new(AtomicUsize::new(0));
let sup = Supervisor::new(
SupervisorStrategy::OneForOne, 5,
vec![
ChildSpec::new("crasher", RestartPolicy::Permanent, {
let c = counter_a.clone();
move |ctx| ctx.spawn_named("ofo_a", PanicAfterN {
trigger: 1, count: 0, counter: c.clone(),
})
}),
ChildSpec::new("stable", RestartPolicy::Permanent, {
let c = counter_b.clone();
move |ctx| ctx.spawn_named("ofo_b", CountingPingActor { counter: c.clone() })
}),
],
);
rt.spawn(sup).unwrap();
tick_n(&rt, 2);
let child_a = rt.where_is("ofo_a").unwrap();
let child_b = rt.where_is("ofo_b").unwrap();
let inbox = rt.new_inbox::<Pong>().unwrap();
rt.send_to(child_a, Ping { reply_to: *inbox.addr() }).unwrap();
tick_n(&rt, 5);
let child_b_after = rt.where_is("ofo_b").unwrap();
assert_eq!(child_b, child_b_after, "child_b unchanged in OneForOne");
rt.send_to(child_b, Ping { reply_to: *inbox.addr() }).unwrap();
rt.tick();
assert!(counter_b.load(Ordering::SeqCst) >= 1, "child_b still processing");
// OneForAll: all children restarted
let rt = std_runtime(RuntimeConfig::default());
let sup = Supervisor::new(
SupervisorStrategy::OneForAll, 5,
vec![
ChildSpec::new("a", RestartPolicy::Permanent, {
let c = Arc::new(AtomicUsize::new(0));
move |ctx| ctx.spawn_named("ofa_a", PanicAfterN {
trigger: 1, count: 0, counter: c.clone(),
})
}),
ChildSpec::new("b", RestartPolicy::Permanent, {
let c = Arc::new(AtomicUsize::new(0));
move |ctx| ctx.spawn_named("ofa_b", CountingPingActor { counter: c.clone() })
}),
],
);
rt.spawn(sup).unwrap();
tick_n(&rt, 2);
let old_b = rt.where_is("ofa_b").unwrap();
let child_a = rt.where_is("ofa_a").unwrap();
let inbox = rt.new_inbox::<Pong>().unwrap();
rt.send_to(child_a, Ping { reply_to: *inbox.addr() }).unwrap();
tick_n(&rt, 8);
let new_b = rt.where_is("ofa_b").expect("ofa_b re-registered");
assert_ne!(old_b, new_b, "child_b restarted in OneForAll");
// RestForOne: failed child + later children restarted, earlier unaffected
let rt = std_runtime(RuntimeConfig::default());
let sup = Supervisor::new(
SupervisorStrategy::RestForOne, 5,
vec![
ChildSpec::new("a", RestartPolicy::Permanent, {
let c = Arc::new(AtomicUsize::new(0));
move |ctx| ctx.spawn_named("rfo_a", CountingPingActor { counter: c.clone() })
}),
ChildSpec::new("b", RestartPolicy::Permanent, {
let c = Arc::new(AtomicUsize::new(0));
move |ctx| ctx.spawn_named("rfo_b", PanicAfterN {
trigger: 1, count: 0, counter: c.clone(),
})
}),
ChildSpec::new("c", RestartPolicy::Permanent, {
let c = Arc::new(AtomicUsize::new(0));
move |ctx| ctx.spawn_named("rfo_c", CountingPingActor { counter: c.clone() })
}),
],
);
rt.spawn(sup).unwrap();
tick_n(&rt, 2);
let old_a = rt.where_is("rfo_a").unwrap();
let old_c = rt.where_is("rfo_c").unwrap();
let child_b = rt.where_is("rfo_b").unwrap();
let inbox = rt.new_inbox::<Pong>().unwrap();
rt.send_to(child_b, Ping { reply_to: *inbox.addr() }).unwrap();
tick_n(&rt, 8);
let new_a = rt.where_is("rfo_a").unwrap();
let new_c = rt.where_is("rfo_c").expect("rfo_c re-registered");
assert_eq!(old_a, new_a, "child_a unchanged in RestForOne");
assert_ne!(old_c, new_c, "child_c restarted in RestForOne");
// Stopping supervisor kills children
let rt = std_runtime(RuntimeConfig::default());
let sup = Supervisor::new(
SupervisorStrategy::OneForOne, 5,
vec![
ChildSpec::new("a", RestartPolicy::Permanent, |ctx| ctx.spawn(PingPongActor)),
ChildSpec::new("b", RestartPolicy::Permanent, |ctx| ctx.spawn(PingPongActor)),
],
);
let sup_addr = rt.spawn(sup).unwrap();
tick_n(&rt, 2);
assert_eq!(rt.stats().workers[0].num_actors, 3);
rt.stop_actor(sup_addr).unwrap();
tick_n(&rt, 5);
assert_eq!(rt.stats().workers[0].num_actors, 0, "stopping supervisor kills children");
}
/// handle_down dispatch and ctx.stop_actor from handler.
#[test]
fn handle_down_dispatch() {
// ctx.stop_actor from handler stops target
#[derive(Clone)]
struct StopCmd { target: ActorAddress }
struct Stopper;
impl ActorInterface for Stopper {
type Incoming = StopCmd;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: StopCmd) {
let _ = ctx.stop_actor(msg.target);
}
}
let rt = std_runtime(RuntimeConfig::default());
let target = rt.spawn(PingPongActor).unwrap();
let stopper = rt.spawn(Stopper).unwrap();
rt.tick();
rt.send_to(stopper, StopCmd { target }).unwrap();
tick_n(&rt, 4);
assert!(rt.send_to(target, Ping { reply_to: ActorAddress::default() }).is_err(),
"target stopped by ctx.stop_actor");
assert!(rt.send_to(stopper, StopCmd { target }).is_ok(), "stopper still alive");
}
// ═══════════════════════════════════════════════════════════════════════════
// Router
// ═══════════════════════════════════════════════════════════════════════════
/// Router distributes work: round-robin is even, broadcast hits all, random
/// uses multiple workers. Dead workers replaced. Stop router kills workers.
/// Meltdown after max restarts.
#[test]
fn router_work_distribution() {
// Round-robin: 3 workers, 6 msgs → 2 each
let rt = std_runtime(RuntimeConfig::default());
let collected = Arc::new(std::sync::Mutex::new(Vec::new()));
struct Collector(Arc<std::sync::Mutex<Vec<(ActorAddress, usize)>>>);
#[derive(Clone)]
struct Work(usize);
impl ActorInterface for Collector {
type Incoming = Work;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: Work) {
self.0.lock().unwrap().push((ctx.self_addr(), msg.0));
}
}
let c = collected.clone();
let router = Router::<Work>::new(
RoutingStrategy::RoundRobin, 3,
move |ctx| ctx.spawn(Collector(c.clone())), 10,
);
let router_addr = rt.spawn(router).unwrap();
rt.tick();
for i in 0..6 {
rt.send_to(router_addr, Work(i)).unwrap();
}
tick_n(&rt, 3);
let data = collected.lock().unwrap();
assert_eq!(data.len(), 6);
let mut per_worker = std::collections::HashMap::new();
for (addr, _) in data.iter() {
*per_worker.entry(*addr).or_insert(0usize) += 1;
}
assert_eq!(per_worker.len(), 3, "3 distinct workers");
for count in per_worker.values() {
assert_eq!(*count, 2, "each worker gets exactly 2");
}
// Broadcast: 5 msgs to 3 workers → 15 total
let rt = std_runtime(RuntimeConfig::default());
let total = Arc::new(AtomicUsize::new(0));
struct BCounter(Arc<AtomicUsize>);
#[derive(Clone)]
struct BPing;
impl ActorInterface for BCounter {
type Incoming = BPing;
type Response = ();
fn handle(&mut self, _ctx: &Ctx, _msg: BPing) {
self.0.fetch_add(1, Ordering::Relaxed);
}
}
let t = total.clone();
let router = Router::<BPing>::new(
RoutingStrategy::Broadcast, 3,
move |ctx| ctx.spawn(BCounter(t.clone())), 10,
);
let router_addr = rt.spawn(router).unwrap();
rt.tick();
for _ in 0..5 {
rt.send_to(router_addr, BPing).unwrap();
}
tick_n(&rt, 3);
assert_eq!(total.load(Ordering::Relaxed), 15, "5 broadcasts × 3 workers = 15");
// Random: 30 msgs → at least 2 workers used
let rt = std_runtime(RuntimeConfig::default());
let rcollected = Arc::new(std::sync::Mutex::new(Vec::new()));
struct RCollector(Arc<std::sync::Mutex<Vec<ActorAddress>>>);
#[derive(Clone)]
struct RWork;
impl ActorInterface for RCollector {
type Incoming = RWork;
type Response = ();
fn handle(&mut self, ctx: &Ctx, _msg: RWork) {
self.0.lock().unwrap().push(ctx.self_addr());
}
}
let c = rcollected.clone();
let router = Router::<RWork>::new(
RoutingStrategy::Random, 3,
move |ctx| ctx.spawn(RCollector(c.clone())), 10,
);
let router_addr = rt.spawn(router).unwrap();
rt.tick();
for _ in 0..30 {
rt.send_to(router_addr, RWork).unwrap();
}
tick_n(&rt, 3);
let data = rcollected.lock().unwrap();
let unique: std::collections::HashSet<_> = data.iter().collect();
assert!(unique.len() >= 2, "random uses at least 2 workers");
// Dead worker replaced
let rt = std_runtime(RuntimeConfig::default());
let spawn_count = Arc::new(AtomicUsize::new(0));
struct PanicOnFirst { first: bool }
#[derive(Clone)]
struct DWork;
impl ActorInterface for PanicOnFirst {
type Incoming = DWork;
type Response = ();
fn handle(&mut self, _ctx: &Ctx, _msg: DWork) {
if self.first { self.first = false; panic!("first message panic"); }
}
}
let sc = spawn_count.clone();
let router = Router::<DWork>::new(
RoutingStrategy::RoundRobin, 3,
move |ctx| { sc.fetch_add(1, Ordering::Relaxed); ctx.spawn(PanicOnFirst { first: sc.load(Ordering::Relaxed) == 1 }) },
10,
);
let router_addr = rt.spawn(router).unwrap();
rt.tick();
rt.send_to(router_addr, DWork).unwrap();
tick_n(&rt, 5);
assert!(spawn_count.load(Ordering::Relaxed) >= 4, "replacement spawned");
// Meltdown: max_restarts=2
let rt = std_runtime(RuntimeConfig::default());
struct AlwaysPanics;
#[derive(Clone)]
struct MWork;
impl ActorInterface for AlwaysPanics {
type Incoming = MWork;
type Response = ();
fn handle(&mut self, _ctx: &Ctx, _msg: MWork) { panic!("always"); }
}
let router = Router::<MWork>::new(
RoutingStrategy::RoundRobin, 1,
|ctx| ctx.spawn(AlwaysPanics), 2,
);
let router_addr = rt.spawn(router).unwrap();
rt.tick();
for _ in 0..3 {
rt.send_to(router_addr, MWork).unwrap();
tick_n(&rt, 5);
}
tick_n(&rt, 5);
assert_eq!(rt.stats().workers[0].num_actors, 0, "router melted down");
// Stop router kills workers
let rt = std_runtime(RuntimeConfig::default());
struct Dummy;
#[derive(Clone)]
struct SWork;
impl ActorInterface for Dummy {
type Incoming = SWork;
type Response = ();
fn handle(&mut self, _ctx: &Ctx, _msg: SWork) {}
}
let router = Router::<SWork>::new(
RoutingStrategy::RoundRobin, 3,
|ctx| ctx.spawn(Dummy), 10,
);
let router_addr = rt.spawn(router).unwrap();
rt.tick();
assert_eq!(rt.stats().workers[0].num_actors, 4);
rt.stop_actor(router_addr).unwrap();
tick_n(&rt, 5);
assert_eq!(rt.stats().workers[0].num_actors, 0, "stop router kills workers");
}

View file

@ -1,335 +0,0 @@
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use swactor::actor::{ActorAddress, ActorExited, ActorInterface, ExitReason};
use swactor::runtime::{Ctx, Runtime, RuntimeConfig};
use swactor_std::{CtxWatching, RuntimeWatching, StdExtension};
// ── Actors ──────────────────────────────────────────────────────────────────
/// An actor that panics when it receives PanicMsg.
struct PanicOnCommand;
#[derive(Clone)]
struct PanicMsg;
impl ActorInterface for PanicOnCommand {
type Incoming = PanicMsg;
type Response = ();
fn handle(&mut self, _ctx: &Ctx, _msg: PanicMsg) {
panic!("deliberate panic for test");
}
}
/// An actor that watches targets and counts exit notifications.
struct ExitWatcher {
exit_count: Arc<AtomicUsize>,
last_reason: Arc<std::sync::Mutex<Option<ExitReason>>>,
last_addr: Arc<std::sync::Mutex<Option<ActorAddress>>>,
}
#[derive(Clone)]
enum WatcherCmd {
WatchThis(ActorAddress),
UnwatchThis(ActorAddress),
}
impl ActorInterface for ExitWatcher {
type Incoming = WatcherCmd;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: WatcherCmd) {
match msg {
WatcherCmd::WatchThis(target) => {
ctx.watch(target);
}
WatcherCmd::UnwatchThis(target) => {
ctx.unwatch(target);
}
}
}
fn on_actor_exit(&mut self, _ctx: &Ctx, exited: ActorExited) {
self.exit_count.fetch_add(1, Ordering::SeqCst);
*self.last_reason.lock().unwrap() = Some(exited.reason);
*self.last_addr.lock().unwrap() = Some(exited.addr);
}
}
impl ExitWatcher {
fn new() -> (Self, WatcherState) {
let exit_count = Arc::new(AtomicUsize::new(0));
let last_reason = Arc::new(std::sync::Mutex::new(None));
let last_addr = Arc::new(std::sync::Mutex::new(None));
let state = WatcherState {
exit_count: exit_count.clone(),
last_reason: last_reason.clone(),
last_addr: last_addr.clone(),
};
(
ExitWatcher {
exit_count,
last_reason,
last_addr,
},
state,
)
}
}
/// Shared state for inspecting what ExitWatcher observed.
struct WatcherState {
exit_count: Arc<AtomicUsize>,
last_reason: Arc<std::sync::Mutex<Option<ExitReason>>>,
last_addr: Arc<std::sync::Mutex<Option<ActorAddress>>>,
}
impl WatcherState {
fn count(&self) -> usize {
self.exit_count.load(Ordering::SeqCst)
}
fn last_reason(&self) -> Option<ExitReason> {
self.last_reason.lock().unwrap().clone()
}
fn last_addr(&self) -> Option<ActorAddress> {
*self.last_addr.lock().unwrap()
}
}
/// A silent actor that does nothing (for targets that shouldn't panic).
struct Sleeper;
#[derive(Clone)]
struct Noop;
impl ActorInterface for Sleeper {
type Incoming = Noop;
type Response = ();
fn handle(&mut self, _ctx: &Ctx, _msg: Noop) {}
}
// ── Helper ──────────────────────────────────────────────────────────────────
fn tick_n(rt: &Runtime, n: usize) {
for _ in 0..n {
rt.tick();
}
}
fn watch_config() -> RuntimeConfig {
RuntimeConfig {
num_threads: 1,
..RuntimeConfig::default()
}
}
fn watch_runtime() -> Runtime {
Runtime::new(watch_config())
.with_extension(Arc::new(StdExtension::new()))
}
// ── Tests ───────────────────────────────────────────────────────────────────
/// Given a watcher and a target actor,
/// when the target panics,
/// then the watcher's on_actor_exit fires with ExitReason::Panicked.
#[test]
fn watch_receives_notification_on_panic() {
let rt = watch_runtime();
let (watcher_actor, state) = ExitWatcher::new();
let target = rt.spawn(PanicOnCommand).unwrap();
let watcher = rt.spawn(watcher_actor).unwrap();
// Tell watcher to watch the target
rt.send_to(watcher, WatcherCmd::WatchThis(target)).unwrap();
tick_n(&rt, 3);
// Kill the target
rt.send_to(target, PanicMsg).unwrap();
tick_n(&rt, 5);
assert_eq!(state.count(), 1, "watcher should have received exactly one ActorExited");
assert_eq!(state.last_reason(), Some(ExitReason::Panicked));
assert_eq!(state.last_addr(), Some(target));
}
/// Given a watcher that watches then unwatches a target,
/// when the target panics,
/// then the watcher receives NO notification.
#[test]
fn unwatch_prevents_notification() {
let rt = watch_runtime();
let (watcher_actor, state) = ExitWatcher::new();
let target = rt.spawn(PanicOnCommand).unwrap();
let watcher = rt.spawn(watcher_actor).unwrap();
// Watch
rt.send_to(watcher, WatcherCmd::WatchThis(target)).unwrap();
tick_n(&rt, 3);
// Unwatch
rt.send_to(watcher, WatcherCmd::UnwatchThis(target)).unwrap();
tick_n(&rt, 3);
// Kill target
rt.send_to(target, PanicMsg).unwrap();
tick_n(&rt, 5);
assert_eq!(state.count(), 0, "after unwatch, no notification should be delivered");
}
/// Given a watcher that dies before the target,
/// when the target subsequently panics,
/// then there is no panic or leak.
#[test]
fn watcher_dies_before_target_no_panic() {
let rt = watch_runtime();
let target = rt.spawn(PanicOnCommand).unwrap();
let target2 = rt.spawn(PanicOnCommand).unwrap();
// Watch via runtime-level API
rt.watch(target2, target);
tick_n(&rt, 3);
// Kill watcher (target2) first
rt.send_to(target2, PanicMsg).unwrap();
tick_n(&rt, 5);
// Kill target — should not crash
rt.send_to(target, PanicMsg).unwrap();
tick_n(&rt, 5);
// If we got here, no crash.
}
/// Given a watcher that calls watch() twice on the same target,
/// when the target panics,
/// then the watcher receives exactly one notification.
#[test]
fn idempotent_watch_delivers_one_notification() {
let rt = watch_runtime();
let (watcher_actor, state) = ExitWatcher::new();
let target = rt.spawn(PanicOnCommand).unwrap();
let watcher = rt.spawn(watcher_actor).unwrap();
// Watch twice
rt.send_to(watcher, WatcherCmd::WatchThis(target)).unwrap();
tick_n(&rt, 3);
rt.send_to(watcher, WatcherCmd::WatchThis(target)).unwrap();
tick_n(&rt, 3);
// Kill target
rt.send_to(target, PanicMsg).unwrap();
tick_n(&rt, 5);
assert_eq!(state.count(), 1, "double watch should produce exactly one notification");
}
/// Given multiple watchers on the same target,
/// when the target panics,
/// then all watchers receive the notification.
#[test]
fn multiple_watchers_all_notified() {
let rt = watch_runtime();
let (w1_actor, s1) = ExitWatcher::new();
let (w2_actor, s2) = ExitWatcher::new();
let (w3_actor, s3) = ExitWatcher::new();
let target = rt.spawn(PanicOnCommand).unwrap();
let w1 = rt.spawn(w1_actor).unwrap();
let w2 = rt.spawn(w2_actor).unwrap();
let w3 = rt.spawn(w3_actor).unwrap();
rt.send_to(w1, WatcherCmd::WatchThis(target)).unwrap();
rt.send_to(w2, WatcherCmd::WatchThis(target)).unwrap();
rt.send_to(w3, WatcherCmd::WatchThis(target)).unwrap();
tick_n(&rt, 3);
rt.send_to(target, PanicMsg).unwrap();
tick_n(&rt, 5);
assert_eq!(s1.count(), 1, "watcher 1 should be notified");
assert_eq!(s2.count(), 1, "watcher 2 should be notified");
assert_eq!(s3.count(), 1, "watcher 3 should be notified");
}
/// Self-watch doesn't crash the runtime.
#[test]
fn self_watch_does_not_crash() {
let rt = watch_runtime();
let (watcher_actor, _state) = ExitWatcher::new();
let actor = rt.spawn(watcher_actor).unwrap();
rt.send_to(actor, WatcherCmd::WatchThis(actor)).unwrap();
tick_n(&rt, 5);
// No crash = pass
}
/// Runtime-level watch (outside actor context) delivers notification.
#[test]
fn runtime_level_watch_delivers_notification() {
let rt = watch_runtime();
let (watcher_actor, state) = ExitWatcher::new();
let target = rt.spawn(PanicOnCommand).unwrap();
let watcher = rt.spawn(watcher_actor).unwrap();
tick_n(&rt, 2); // ensure both spawned
rt.watch(watcher, target);
rt.send_to(target, PanicMsg).unwrap();
tick_n(&rt, 5);
assert_eq!(state.count(), 1, "runtime-level watch should deliver notification");
assert_eq!(state.last_reason(), Some(ExitReason::Panicked));
}
/// Given a watcher watching target via on_actor_exit,
/// when target panics,
/// then the watcher can react by spawning a replacement (supervision pattern).
#[test]
fn watcher_can_react_to_death_by_spawning() {
let rt = watch_runtime();
let spawned = Arc::new(AtomicUsize::new(0));
struct Supervisor {
spawned_count: Arc<AtomicUsize>,
}
#[derive(Clone)]
enum SupervisorMsg {
WatchThis(ActorAddress),
}
impl ActorInterface for Supervisor {
type Incoming = SupervisorMsg;
type Response = ();
fn handle(&mut self, ctx: &Ctx, msg: SupervisorMsg) {
match msg {
SupervisorMsg::WatchThis(target) => ctx.watch(target),
}
}
fn on_actor_exit(&mut self, ctx: &Ctx, _exited: ActorExited) {
// React: spawn a replacement
let replacement = ctx.spawn(Sleeper).unwrap();
let _ = replacement;
self.spawned_count.fetch_add(1, Ordering::SeqCst);
}
}
let target = rt.spawn(PanicOnCommand).unwrap();
let sup = rt.spawn(Supervisor { spawned_count: spawned.clone() }).unwrap();
rt.send_to(sup, SupervisorMsg::WatchThis(target)).unwrap();
tick_n(&rt, 3);
rt.send_to(target, PanicMsg).unwrap();
tick_n(&rt, 5);
assert_eq!(spawned.load(Ordering::SeqCst), 1, "supervisor should have spawned a replacement");
}