zacheryasc
c1feddcab4
Stand up a runnable distribution stack on top of the core node logic. - distribution: add NodeDriver bridging DistributedNode to real TCP I/O (TcpTransport/TcpAcceptor), translating NodeActions to/from wire messages; refine swim probe timing and transport wiring. - node: new swactor-node binary (crates/node) hosting a real node over TCP. - tests/docker: multi-host LAN cluster harness (compose, run-lan-cluster.sh, cluster + lan_cluster integration tests) exercising the full SWIM path. - simulation: cluster_scenarios integration + distribution property coverage. - docs: reorganize into distribution/, runtime/, diagrams/, connectome/; add DOCKER_REALIZATION + SIMULATION_TESTING realization notes. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
140 lines
6.7 KiB
Markdown
140 lines
6.7 KiB
Markdown
# DistributedNode
|
|
|
|
`DistributedNode` is the top-level integration type that composes SWIM
|
|
membership, Kademlia routing, the actor directory, location cache, and
|
|
repair infrastructure into a single public API.
|
|
|
|
## Internal Composition
|
|
|
|
```
|
|
┌─ DistributedNode ────────────────────────────────────────────────────────┐
|
|
│ │
|
|
│ keypair: Keypair ← ed25519 identity + signing │
|
|
│ │
|
|
│ ┌── SWIM ─────────────────┐ ┌── Kademlia ────────────────────────────┐ │
|
|
│ │ SwimNode │ │ RoutingTable (256 k-buckets, K=20) │ │
|
|
│ │ ├ MemberList (CRDT) │ │ DirectoryShard (local entries) │ │
|
|
│ │ ├ SwimProbe (FSM) │ │ RepairQueue (death-triggered) │ │
|
|
│ │ └ DisseminationQueue │ │ RepublishTracker (periodic) │ │
|
|
│ └──────────────────────────┘ └────────────────────────────────────────┘ │
|
|
│ │
|
|
│ cache: LocationCache ← LRU: ActorAddress → NodeId │
|
|
│ tick_count: u64 ← monotonic clock │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Node Lifecycle
|
|
|
|
```
|
|
new(config)
|
|
│
|
|
v
|
|
Generate Keypair ──► NodeId = public key bytes
|
|
│
|
|
v
|
|
Initialize subsystems (SwimNode, RoutingTable, DirectoryShard, ...)
|
|
│
|
|
v
|
|
join(seeds) ──► SendJoinRequest to each seed
|
|
│
|
|
v
|
|
handle_join_response(members) ──► populate MemberList + RoutingTable
|
|
│
|
|
v
|
|
┌─────────────┐
|
|
│ tick() loop │ ──► drives SWIM probes, processes membership changes,
|
|
│ (caller) │ runs periodic republish, drains repair queue
|
|
└─────────────┘
|
|
│
|
|
v
|
|
leave() ──► disseminate Dead for self, graceful shutdown
|
|
```
|
|
|
|
See [distribution_minor_flows.svg](../diagrams/distribution_minor_flows.svg) for the
|
|
join handshake, dissemination piggybacking, and membership change cascade.
|
|
|
|
## Actor Registration
|
|
|
|
`register_actor(addr, generation)`:
|
|
1. Sign a `DirectoryEntry` with the node's keypair.
|
|
2. Store in local `DirectoryShard`.
|
|
3. Insert into `LocationCache` (self is the host).
|
|
4. Register with `RepublishTracker` for periodic re-STORE.
|
|
5. Return the signed entry — caller STOREs to `r`-closest nodes.
|
|
|
|
See [actor_resolution.svg](../diagrams/actor_resolution.svg) for the full datapath.
|
|
|
|
## Actor Resolution
|
|
|
|
`resolve_actor(addr)` implements a 3-tier lookup chain:
|
|
1. **LRU cache** — instant, O(1).
|
|
2. **Local directory shard** — checks entries this node holds.
|
|
3. **Kademlia lookup** — returns `NeedsLookup` with closest known nodes;
|
|
caller drives iterative FIND_VALUE + quorum read.
|
|
|
|
On delivery failure, `invalidate_cache(addr)` evicts stale entries.
|
|
|
|
## Membership Change Cascade
|
|
|
|
When SWIM detects a state change, `handle_membership_change(node_id, state)`
|
|
propagates effects through all subsystems:
|
|
|
|
| State | Actions |
|
|
|-------|---------|
|
|
| **Alive** | Insert/update in `RoutingTable` |
|
|
| **Suspect** | Keep in `RoutingTable` (could downprioritize) |
|
|
| **Dead** | Remove from `RoutingTable`, invalidate cache entries for that node, queue affected directory entries in `RepairQueue` |
|
|
|
|
This cascade ensures that a single SWIM death detection triggers routing
|
|
table cleanup, cache invalidation, and directory repair in one tick.
|
|
|
|
## NodeDriver — TCP Network Bridge
|
|
|
|
`NodeDriver` (`driver.rs`) bridges the pure state machine API with real TCP
|
|
networking. It owns a `DistributedNode` plus a `TcpTransport` (connection
|
|
pool) and `TcpAcceptor` (non-blocking listener).
|
|
|
|
```
|
|
┌─ NodeDriver ─────────────────────────────────────────────────┐
|
|
│ │
|
|
│ node: DistributedNode ← pure state machine │
|
|
│ transport: TcpTransport ← connection pool for outgoing │
|
|
│ acceptor: TcpAcceptor ← non-blocking listener │
|
|
│ streams: Vec<TcpStream> ← accepted connections │
|
|
│ │
|
|
│ tick() ──► node.tick() → map NodeAction → TCP send │
|
|
│ recv() ──► acceptor.try_recv() → dispatch → handler calls │
|
|
│ join() ──► node.join() → send JoinRequest via TCP │
|
|
│ │
|
|
└───────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
The caller runs a loop: `recv()` → `tick()` → sleep. The driver handles
|
|
all TCP I/O internally — the caller never touches sockets directly.
|
|
|
|
See [DOCKER_REALIZATION.md](../development_history/DOCKER_REALIZATION.md)
|
|
for implementation details of the driver, the node binary (`crates/node/`),
|
|
and the Docker cluster integration tests.
|
|
|
|
## Dashboard REST API
|
|
|
|
The runtime dashboard exposes `/api/distribution` (feature-gated with
|
|
`distribution`) which returns the `DistributionNodeSnapshot` as JSON.
|
|
This supplements the SSE stream (`/events`) with a synchronous polling
|
|
endpoint used by integration tests.
|
|
|
|
## Where Things Live
|
|
|
|
| Type | File | Role |
|
|
|------|------|------|
|
|
| `DistributedNode` | `node.rs` | Top-level integration facade |
|
|
| `DistributedNodeConfig` | `node.rs` | Node configuration |
|
|
| `ResolveResult` | `node.rs` | 3-tier resolution outcomes |
|
|
| `NodeDriver` | `driver.rs` | TCP ↔ NodeAction bridge |
|
|
| `LocationCache` | `cache.rs` | LRU actor→node cache |
|
|
| `Keypair` | `crypto.rs` | Ed25519 keypair + signing |
|
|
| `NodeId` | `types.rs` | 32-byte node identity |
|
|
| `MemberState` | `types.rs` | Alive / Suspect / Dead |
|
|
| `DirectoryEntry` | `types.rs` | Signed actor→node binding |
|
|
| `NodeRecord` | `types.rs` | Wire-format membership record |
|