From 0ca540b75001837002c7cf9e8803d531929bd614 Mon Sep 17 00:00:00 2001 From: Zachery Aaron Shores-Chmielewski Date: Sun, 15 Feb 2026 17:14:16 +0700 Subject: [PATCH 1/9] feat: distributed datastore primitives protocol Implements MVP for a swactor distributed datastore, addressing large files via content addressing. --- crates/datastore/PROTOCOL.md | 471 +++++++++++++++++++++++++++++++++++ 1 file changed, 471 insertions(+) create mode 100644 crates/datastore/PROTOCOL.md diff --git a/crates/datastore/PROTOCOL.md b/crates/datastore/PROTOCOL.md new file mode 100644 index 0000000..85dc919 --- /dev/null +++ b/crates/datastore/PROTOCOL.md @@ -0,0 +1,471 @@ +# Swactor Datastore Protocol Specification + +**Version:** 0.2.0 (MVP) +**Status:** Draft + +## 1. Overview + +The Swactor Datastore is a distributed personal file/blob storage protocol for small trusted clusters (laptop, phone, browser). It provides content-hash-first addressing with immutable content-addressed objects, replicated via a Kademlia-based metadata DHT. + +### Design Principles + +- **Content-hash-first addressing** — every object is identified by `blake3(blob_bytes)`. This is the primary key for all operations. +- **Immutable content-addressed objects** — content hashes are unique identifiers. There are no write conflicts by construction. +- **Names are metadata** — optional flat strings attached to objects, not keys. Multiple objects can share a name; distinguished by content hash. +- **Separation of data and metadata** — chunks are large opaque blobs; metadata is small, gossiped, and queryable. +- **Crash-safe** — fsync before acknowledge on all writes. +- **Actor-based** — three actor types coordinate via message passing within the swactor runtime. +- **Transport-agnostic** — protocol messages defined as `NetworkMessage` types; MVP uses iroh (QUIC + NAT hole-punch + encryption). +- **Pluggable storage** — `StorageBackend` trait abstracts I/O for filesystem (MVP), IndexedDB (browser), etc. + +## 2. Terminology + +| Term | Definition | +|------|-----------| +| **Object** | Content-addressed blob identified by `blake3(blob_bytes)`. May carry an optional human-readable name as metadata. | +| **Blob** | The raw byte content of an object. | +| **Chunk** | A fixed-size (1 MB default) slice of a blob, identified by its blake3 content hash. | +| **Manifest** | An ordered list of `ChunkRef`s describing how to reassemble an object from chunks. Stored under the object's content hash. | +| **ContentHash** | 32-byte blake3 digest. Primary identifier for blobs and DHT key. | +| **ObjectEntry** | Metadata record: content hash, optional name, owner node, tags. | +| **DHT overlay** | A Kademlia distributed hash table for object metadata, separate from the actor directory DHT. Keys are `blake3(blob_bytes)`. | +| **StorageBackend** | Trait abstracting chunk and manifest I/O for pluggable backends (filesystem, IndexedDB, etc.). | +| **Node** | A device running the swactor runtime with a datastore actor set (BlobStoreActor + MetadataActor). | + +## 3. Data Model + +### 3.1 ContentHash + +``` +ContentHash = blake3(data)[0..32] // 32 bytes +``` + +- **Hashing algorithm:** blake3 — 2-3x faster than sha256, tree-hashable (parallel hashing of large chunks), same 32-byte output. Supports streaming hashing for large blobs via `blake3::Hasher`. +- **Display:** first 8 bytes as hex + ellipsis (e.g. `a1b2c3d4e5f6a7b8…`). +- **XOR distance:** bitwise XOR of the 32-byte arrays, used for Kademlia routing in the metadata DHT. + +### 3.2 ObjectEntry + +``` +ObjectEntry { + content_hash: ContentHash, // blake3(entire_blob) — primary identifier + name: Option, // Optional flat string, not a path + node_id: NodeId, // Node that stores the object + tags: BTreeMap, // User-defined key-value tags + size_bytes: u64, // Total object size + created_at: u64, // Wall-clock creation time (informational) +} +``` + +No conflict resolution is needed — content hashes are unique identifiers. Storing the same blob twice is a no-op (same content hash). Different blobs always have different content hashes. + +### 3.3 ObjectManifest + +``` +ObjectManifest { + content_hash: ContentHash, // blake3(entire_blob) — NOT the hash of this manifest + chunks: Vec, // Ordered list of chunks + total_size: u64, // Total object size in bytes + chunk_size: u32, // Fixed chunk size used (e.g. 1MB) + content_type: Option, // MIME type +} + +ChunkRef { + hash: ContentHash, // Content hash of chunk data + offset: u64, // Byte offset in original object + size: u32, // Actual size (last chunk may be smaller) +} +``` + +The `content_hash` field is `blake3(entire_blob)`, computed via a streaming hasher alongside chunking. The manifest is stored and looked up using this content hash as the key. + +### 3.4 Storage Backend + +The `StorageBackend` trait abstracts all chunk and manifest I/O: + +```rust +pub trait StorageBackend: Send { + fn write_chunk(&mut self, hash: &ContentHash, data: &[u8]) -> Result<(), io::Error>; + fn read_chunk(&self, hash: &ContentHash) -> Result>, io::Error>; + fn delete_chunk(&mut self, hash: &ContentHash) -> Result<(), io::Error>; + fn has_chunk(&self, hash: &ContentHash) -> bool; + fn list_chunks(&self) -> Vec; + fn write_manifest(&mut self, manifest: &ObjectManifest) -> Result<(), io::Error>; + fn read_manifest(&self, content_hash: &ContentHash) -> Result, io::Error>; + fn delete_manifest(&mut self, content_hash: &ContentHash) -> Result<(), io::Error>; +} +``` + +#### MVP: FilesystemBackend + +Two-level directory sharding to avoid huge directories: + +``` +{storage_path}/ +├── chunks/ +│ └── {hex[0..2]}/ +│ └── {hex[2..4]}/ +│ └── {full_hex_hash} # Raw chunk bytes +└── manifests/ + └── {hex[0..2]}/ + └── {hex[2..4]}/ + └── {full_hex_hash} # JSON-serialized ObjectManifest +``` + +Example: chunk with hash `abcdef12...` is stored at `chunks/ab/cd/abcdef12...`. + +All writes are fsynced before acknowledging. + +## 4. Content Addressing + +### 4.1 Chunking Algorithm + +Fixed-size chunking (MVP): + +1. Read the input file in `chunk_size` byte blocks (default: 1,048,576 = 1 MB). +2. For each block, compute `ContentHash::of(block)`. +3. Store each chunk via the `StorageBackend`. +4. Build a `Vec` with sequential offsets. +5. Compute `content_hash = blake3(entire_blob)` using a streaming hasher fed alongside chunking. +6. Create the `ObjectManifest` with this `content_hash` and store it via the `StorageBackend` keyed by `content_hash`. + +The last chunk may be smaller than `chunk_size`. + +### 4.2 Reassembly + +1. Read the `ObjectManifest` (by its `content_hash`). +2. For each `ChunkRef` in order, read the chunk by `hash`. +3. Concatenate all chunk data to reconstruct the original blob. +4. Verify: `blake3(reassembled) == content_hash` (optional integrity check). + +## 5. Metadata DHT + +> **Status:** Types and routing table logic exist in the `distribution` crate. `MetadataActor` has a dissemination queue (`enqueue`/`take_pending`) and peer-to-peer replication via `SetPeers` + `DisseminateTick`. Verified in local multi-node simulation. Full Kademlia iterative lookup (FIND_VALUE with α-parallel queries) is not yet implemented — dissemination is epidemic/gossip-style. + +### 5.1 Overlay Design + +The metadata DHT is a **separate Kademlia overlay** from the actor directory. It stores `ObjectEntry` records keyed by `blake3(blob_bytes)` — the content hash of the entire blob. + +This separation ensures: +- Object metadata routing doesn't interfere with actor discovery. +- Different replication factors can be used (objects may be stored on fewer nodes). +- The DHT can be independently tuned for the metadata workload. + +### 5.2 Key Mapping + +``` +DHT key = blake3(blob_bytes) = entry.content_hash +``` + +### 5.3 Store Flow + +When storing object metadata: +1. Use `key = entry.content_hash`. +2. Find the `k` closest nodes to `key` in the metadata DHT routing table. +3. Send `StoreObjectRequest { entry }` to each of the `k` closest nodes. + +### 5.4 Lookup Flow + +When looking up object metadata: +1. Send `FindObjectRequest { content_hash }` to the `α` closest known nodes. +2. Each node responds with either `Found(ObjectEntry)` or `Closer(Vec<(NodeId, SocketAddr)>)`. +3. Continue querying closer nodes until convergence. + +No merge step is needed — content hashes are unique identifiers. + +## 6. Protocol Flows + +### 6.1 PUT — Store an Object + +``` +User MetadataActor BlobStoreActor + │ │ │ + │─── PutObject ────────────>│ │ + │ │ │ + │ │ (chunk the file, │ + │ │ stream blake3 hash) │ + │ │ │ + │ │─── WriteChunk ────────>│ + │ │<── ChunkStored ────────│ (repeat for each chunk) + │ │ │ + │ │─── WriteManifest ─────>│ + │ │<── ManifestStored ─────│ + │ │ │ + │ │ (create ObjectEntry, │ + │ │ store in local index,│ + │ │ enqueue for DHT │ + │ │ dissemination) │ + │ │ │ + │<── PutOk {content_hash} ─│ │ +``` + +### 6.2 GET — Retrieve an Object (Local) + +``` +User MetadataActor BlobStoreActor + │ │ │ + │─── GetObject ────────────>│ │ + │ {content_hash} │ │ + │ │ (lookup content_hash │ + │ │ in local index) │ + │ │ │ + │<── GetOk { entry, │ │ + │ manifest } ──────│ │ + │ │ + │ (for each chunk in manifest) │ + │───────────── ReadChunk ───────────────────────────>│ + │<────────────── ChunkOk ───────────────────────────│ + │ │ + │ (reassemble chunks into original file) │ +``` + +### 6.3 GET — Retrieve an Object (Remote) + +> **Status:** `TransferActor` state machine is functional and stores received chunks to the local `BlobStoreActor`. Chunks must be fed externally (via `ChunkReceived` messages). Automatic chunk pulling from remote nodes is not yet implemented — the test harness or a future network adapter plays the "pull" role. Verified in multi-node simulation. + +``` +User MetadataActor TransferActor Remote BlobStore + │ │ │ │ + │─ GetObject ──>│ │ │ + │ {content_hash}│ │ │ + │ │ (not in local │ │ + │ │ index; DHT │ │ + │ │ lookup) │ │ + │ │ │ │ + │ │─ StartDownload ───>│ │ + │ │ │ │ + │ │ │── GetChunkRequest ─>│ + │ │ │<─ GetChunkResponse ─│ + │ │ │ │ + │ │ │ (repeat for each │ + │ │ │ chunk) │ + │ │ │ │ + │ │<─ TransferComplete │ │ + │ │ │ │ + │<── GetOk ────│ │ (stops self) │ +``` + +### 6.4 DELETE — Remove an Object + +``` +User MetadataActor + │ │ + │─── DeleteObject ─────────>│ + │ {content_hash} │ + │ │ + │ │ (remove from local ) + │ │ (index, best-effort ) + │ │ (notify DHT peers ) + │ │ + │<── DeleteOk │ + │ {content_hash} ────────│ +``` + +Chunk data is **not** immediately deleted. Unreferenced chunks are cleaned up during GC sweeps (see Section 10). + +### 6.5 LIST — List Objects (Local) + +``` +User MetadataActor + │ │ + │─── ListLocal ────────────>│ + │ {name_filter} │ + │ │ + │ │ (filter local index ) + │ │ (by name substring ) + │ │ + │<── ListOk { entries } ───│ +``` + +### 6.6 LIST — List Objects (Swarm-Wide) + +> **Status:** `ListSwarm` currently delegates to `ListLocal` (returns local entries only). Fan-out to peer MetadataActors is not yet wired. Swarm-wide listing is verified in simulation by querying each node and merging results in the test harness. + +``` +User MetadataActor Remote MetadataActors + │ │ │ + │─ ListSwarm ──>│ │ + │ {name_filter} │ │ + │ │── ListObjectsRequest ─>│ (fan-out to all known + │ │<─ ListObjectsResponse ─│ alive nodes) + │ │ │ + │ │ (merge all results, │ + │ │ deduplicate by │ + │ │ content hash) │ + │ │ │ + │<── ListOk ───│ │ +``` + +## 7. Actor Architecture + +> **Status:** All three actor types are fully implemented and tested. `DatastoreNode` coordinator routes commands to internal actors. 83+ tests across 8 test files verify single-node operations. Multi-node dissemination and cross-node transfers verified in simulation. + +### 7.1 BlobStoreActor + +**Responsibility:** Chunk and manifest I/O via `StorageBackend` trait. + +- **State:** `Box` +- **Lifecycle:** Long-lived, one per node. +- **Guarantees:** Delegates to backend; filesystem backend fsyncs before acknowledging. + +**Message types:** `BlobStoreMsg` (see `messages.rs`) + +### 7.2 MetadataActor + +**Responsibility:** Object metadata index, DHT routing. + +- **State:** Local object index (`HashMap`), manifest cache, dissemination queue. +- **Lifecycle:** Long-lived, one per node. +- **Coordinates with:** BlobStoreActor (for manifest storage), remote MetadataActors (DHT operations). + +**Message types:** `MetadataMsg` (see `messages.rs`) + +### 7.3 TransferActor + +**Responsibility:** Downloading an object (all its chunks) from a remote node. + +- **State:** Manifest, pending/received chunk sets, retry counts. +- **Lifecycle:** Ephemeral — spawned per download, self-terminates on completion/failure/cancel. +- **Coordinates with:** Remote BlobStoreActor (chunk requests), local BlobStoreActor (chunk storage). + +**Message types:** `TransferMsg` (see `messages.rs`) + +## 8. Wire Protocol + +> **Status:** All message types are defined with `NetworkMessage` implementations and stable type tags. Serialization is JSON (serde). No transport integration yet — messages are passed directly via actor addresses in simulation. + +### 8.1 Message Types + +All inter-node messages implement `NetworkMessage` with a stable `type_tag()`: + +| Message | type_tag | Direction | +|---------|----------|-----------| +| `GetChunkRequest` | `swactor_datastore::GetChunkRequest` | requester → holder | +| `GetChunkResponse` | `swactor_datastore::GetChunkResponse` | holder → requester | +| `StoreObjectRequest` | `swactor_datastore::StoreObjectRequest` | writer → DHT nodes | +| `FindObjectRequest` | `swactor_datastore::FindObjectRequest` | reader → DHT nodes | +| `FindObjectResponse` | `swactor_datastore::FindObjectResponse` | DHT node → reader | +| `GetManifestRequest` | `swactor_datastore::GetManifestRequest` | requester → holder | +| `GetManifestResponse` | `swactor_datastore::GetManifestResponse` | holder → requester | +| `ListObjectsRequest` | `swactor_datastore::ListObjectsRequest` | requester → remote node | +| `ListObjectsResponse` | `swactor_datastore::ListObjectsResponse` | remote node → requester | + +`FindObjectRequest` contains a `content_hash` field (the `blake3(blob_bytes)` key). + +### 8.2 Serialization + +MVP: serde JSON for all messages. Binary format (bincode or msgpack) planned for later to reduce overhead, especially for `GetChunkResponse` which carries large payloads. + +### 8.3 Framing + +Messages are framed over iroh QUIC streams: +- Each request/response pair uses a single bidirectional stream. +- Message format: `[4-byte length (big-endian)][JSON payload]`. + +## 9. Naming + +Names are **optional flat strings** — human-readable labels attached to objects as metadata. + +- Names are not keys. The content hash is the only primary identifier. +- Multiple objects can share the same name. They are distinguished by content hash. +- Names are simple strings (e.g. `"vacation.jpg"`, `"backup-2024-01"`). No path hierarchy, no separators enforced. +- No conflict resolution is needed — different content always produces different content hashes. + +## 10. Garbage Collection + +> **Status:** Fully implemented. `MetadataActor::gc_tick()` builds a referenced chunk set from all local manifests and sends `GcUnreferenced` to `BlobStoreActor`. Verified with 6 GC-specific tests including deduplication safety, interval gating, and empty-store edge case. + +### 10.1 Entry Removal + +Deleting an object: +1. Remove the `ObjectEntry` from the local index. +2. Best-effort notify DHT peers to remove their replicas. +3. Remove the local manifest. + +### 10.2 Chunk Reference Counting + +Unreferenced chunk cleanup: + +1. Build a referenced set: union of all chunk hashes from all local manifest entries. +2. Send `BlobStoreMsg::GcUnreferenced { referenced }` to the BlobStoreActor. +3. BlobStoreActor diffs its chunk list against the referenced set and deletes unreferenced chunks. + +**Safety:** A chunk may be referenced by multiple objects (deduplication). Only delete when zero references remain. + +### 10.3 GC Schedule + +- `MetadataActor` runs `gc_tick()` every tick. Actual GC sweep happens every `gc_interval` ticks (default: 1000). +- Chunk GC is triggered less frequently (order of minutes) to avoid overhead. + +## 11. Failure Modes + +### 11.1 Node Offline + +- **Metadata persists** in the DHT (replicated to k-closest nodes). Lookups succeed as long as any replica is alive. +- **Chunk fetches fail** if the only copy is on the offline node. The TransferActor retries once, then reports failure. +- **Recovery:** When the node comes back, its metadata is re-disseminated (anti-entropy). + +### 11.2 Transfer Interrupted + +- **Partial state:** Some chunks may be written to the local BlobStoreActor before the transfer fails. +- **Cleanup:** Partially downloaded chunks are not harmful — they're content-addressed and may be useful for future downloads. Unreferenced chunks are cleaned up by GC. +- **Retry:** The user can retry the GET, and only missing chunks need to be fetched (future optimization). + +### 11.3 DHT Inconsistency + +- **Stale metadata:** A node may serve an outdated ObjectEntry. Anti-entropy dissemination ensures replicas converge. +- Content addressing eliminates write conflicts — storing the same content hash twice is idempotent. + +### 11.4 Disk Full + +- `StorageBackend::write_chunk` fails with an I/O error, which is propagated back to the requester as `DatastoreResponse::Error`. +- No partial writes — fsync ensures atomicity (filesystem backend). + +## 12. CLI Interface + +> **Status:** Command types defined in `src/cli.rs`. Parser, dispatcher, and `[[bin]]` target not yet implemented. Planned for a follow-up session. + +``` +swactor-store put [--name