Zachery Aaron Shores-Chmielewski
599678e7c9
Introduce a gossip-converged pooled datastore protocol layered on SWIM piggybacking, backed by a reusable gossip-channel abstraction, plus a Docker-free multi-process cluster test runner. - crates/distribution/src/gossip_channel.rs: add the GossipChannel trait (piggyback on SWIM messages) and a budget-limited DisseminationBuffer<T> that replaces the four duplicated Lambda*ceil(log2(n)) dissemination copies - crates/datastore/src/pool: add PoolDisseminator (CRDT state for membership/capacity/content-location/ACL with join/leave/announce) and PoolCoordinator (placement-aware CRUD actor delegating to the co-located DatastoreNode) - crates/shared-types/src/pool.rs: add shared pool protocol types (PoolId plus member/capacity/content-location/ACL entries and PoolConfig) consumed by both distribution and datastore - crates/dashboard/src/pool_html.rs: add a live pool dashboard page (membership, capacity, content locations) and add pool_tests integration coverage - xtask/src/sim_cluster.rs: add the sim-cluster runner that spawns N swactor nodes over iroh through a local relay server, reusing the docker cluster scenarios without Docker - crates/distribution/src/iroh_driver.rs: add relay-URL resolution (cache, then SWIM gossip, then home relay) with a 2s connect timeout to back the relay-based connections Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com> |
||
|---|---|---|
| .. | ||
| src | ||
| tests | ||
| Cargo.toml | ||
| README.md | ||
swactor-datastore
Distributed content-addressed datastore built on swactor. Objects are split into fixed-size chunks, identified by their blake3 hash, and replicated across a peer-to-peer network via epidemic gossip.
Building
Node binary (HTTP server + actor runtime):
cargo build -p swactor-datastore --features node
CLI client:
cargo build -p swactor-datastore --features cli
Both at once:
cargo build -p swactor-datastore --features node,cli
Node
Start a datastore node:
swactor-store-node
Flags
| Flag | Default | Description |
|---|---|---|
--port |
9091 |
HTTP API port |
--storage-path |
(in-memory) | Directory for persistent storage |
--dashboard-port |
(disabled) | Runtime dashboard port |
--chunk-size |
1048576 |
Chunk size in bytes (1 MB) |
--gc-interval |
1000 |
GC interval in ticks (~100s) |
--disseminate-interval |
50 |
Gossip interval in ticks (~5s) |
--config |
(none) | Path to a TOML config file |
Example with persistent storage and dashboard:
swactor-store-node --storage-path ./data --dashboard-port 9090
Config file
Create a store.toml and pass it with --config:
port = 9091
storage_path = "./my-data"
dashboard_port = 9090
chunk_size = 1048576
gc_interval = 1000
disseminate_interval = 50
CLI flags override config file values. Omitted fields use built-in defaults.
swactor-store-node --config store.toml --port 8080
CLI
The swactor-store command talks to a running node over HTTP.
Status
swactor-store status
Put
swactor-store put photo.jpg --name "vacation"
Get (metadata)
swactor-store get <hash>
Get (download)
swactor-store get <hash> --output photo.jpg
Delete
swactor-store delete <hash>
List (local)
swactor-store list
List (swarm-wide)
swactor-store list --all
Filter by name
swactor-store list --name vacation
Use --url to point at a different node:
swactor-store --url http://192.168.1.50:9091 list
Web UI
Visit http://<host>:<port>/ in a browser. The UI supports uploading, listing, downloading, inspecting, and deleting objects — works on desktop and mobile.
HTTP API
| Method | Path | Description |
|---|---|---|
GET |
/api/status |
Node identity |
POST |
/api/put?name=... |
Upload (body = raw bytes) |
GET |
/api/list |
List objects (?all=true for swarm) |
GET |
/api/get?hash=... |
Object metadata + manifest |
GET |
/api/data?hash=... |
Download reassembled binary |
POST |
/api/delete?hash=... |
Delete object |