A barebones peer-to-peer network for distributed AI workloads.
- Peer Discovery: Automatic peer discovery through bootstrap nodes
- Authenticated Handshake: Mutual ed25519 challenge-response — each peer proves possession of the key behind its PeerId before being admitted; spoofed identities are rejected
- Connection Management: Maintains persistent connections with automatic reconnection
- Exponential Backoff: Smart retry logic with configurable backoff parameters
- Persistent Identity: Node keys are saved to disk for identity persistence across restarts and used to authenticate the node to its peers
- JSON-RPC Server: Optional HTTP server for querying node state
- Keep-Alive: Heartbeat mechanism to keep connections alive
- GPU detection: GPU capability detection for use in workloads (currently supports Nvidia GPUs)
Ensure Rust is installed
Start a bootstrap node that other peers can connect to:
cargo run -- -l 127.0.0.1:5674Start additional nodes and connect them to the bootstrap node:
cargo run -- -b 127.0.0.1:5674 -l 127.0.0.1:2675You can specify multiple bootstrap nodes for resilience. If one is unavailable, the node will still join via the others:
cargo run -- -b 127.0.0.1:5674,127.0.0.1:5675 -l 127.0.0.1:2675Each node will:
- Attempt to connect to all specified bootstrap nodes concurrently
- Discover other peers in the network
- Maintain persistent connections with automatic reconnection
Enable the RPC server to query node information:
cargo run -- -l 127.0.0.1:2345 --rpc-addr 127.0.0.1:3330Usage: thala [OPTIONS]
Options:
-l, --listen-address <LISTEN_ADDRESS>
Address and port to listen on for peer connections
[default: 127.0.0.1:2345]
-b, --bootstrap-nodes <BOOTSTRAP_NODES>
Bootstrap node address(es) to connect to on startup (comma-separated)
-p, --peer-reconnection-interval <PEER_RECONNECTION_INTERVAL>
Time interval in seconds between reconnection attempts
[default: 30]
-x, --backoff-multiplier <BACKOFF_MULTIPLIER>
Multiplier for exponential backoff on failed reconnections
[default: 2.0]
-m, --max-backoff-interval <MAX_BACKOFF_INTERVAL>
Maximum backoff interval in seconds
[default: 2000]
-r, --reconnection-retries-cap <RECONNECTION_RETRIES_CAP>
Maximum number of reconnection attempts before giving up
[default: 13]
--claim-window <CLAIM_WINDOW>
Seconds a coordinator collects claims for a task before picking a worker
[default: 2]
-a, --models <MODELS>
Models this node can run (comma-separated)
[default: resnet50]
-q, --datasets <DATASETS>
Datasets this node has available (comma-separated)
[default: gpqa]
--rpc-addr <RPC_ADDR>
RPC server listening address (optional). Bind to loopback,
e.g. 127.0.0.1:3330, unless remote access is required
--rpc-token <RPC_TOKEN>
Bearer token required on every RPC call (optional)
[env: THALA_RPC_TOKEN]
-d, --data-dir <DATA_DIR>
Data directory for storing node identity and state
[default: ~/.thala]
-h, --help
Print help information
cargo run -- \
-b 127.0.0.1:2345,127.0.0.1:2350 \
-l 127.0.0.1:2346 \
-p 30 \
-m 300 \
-x 2.0 \
-r 10 \
--rpc-addr 127.0.0.1:3331 \
-d ./my-node-dataAny node can coordinate a task: the node a client submits it to becomes its
coordinator. A task moves Pending → Running → Completed | Failed.
- Submit. A client calls
broadcast_taskon a node. That node records the task asPendingand sends aTaskAnnouncementto each peer it is directly connected to. Announcements are not yet forwarded further. - Claim. A peer that receives the announcement ignores it if it has seen the
task before, or if the task expires less than 60 seconds from now. If the peer
can run the task, it sends a
TaskClaimback to the coordinator and keeps its ownPendingcopy. - Assign. The coordinator only accepts a claim if it comes from the claiming
peer itself, and if that peer's advertised capabilities match the task. The
first accepted claim opens a claim window (
--claim-window, 2s by default). When the window closes, the claim with the lowest estimated duration wins, and ties go to the earliest claim. The coordinator marks the taskRunningand sends aTaskAssignmentto the winner. Losing peers are not told; their copies stayPendinguntil the task expires. - Execute. The winner checks that the assignment came from the task's
coordinator, marks the task
Running, executes it, and sends aTaskResultback to the coordinator. - Complete. The coordinator only accepts a result from the assigned worker,
and only while the task is
Running. A success marks the taskCompleted; a failure marks itFailed. The coordinator stores the result with the task. - Collect. The client polls
task_status(orthala-client status <id>) on the coordinator to read the status, the assigned worker and the result.
Each node removes a task and everything it knows about it once the task's expiry time has passed, checking once a minute. Fetch results before then.
Execution is not implemented yet. Workers currently report every task as
Failedwith the errorno executor for <kind> tasks yet, so the protocol can be exercised end to end before real engines are wired in.
A node claims a task only if it can run it. Its capabilities are its detected
hardware plus the --models and --datasets it is started with:
| Task | Requires |
|---|---|
| Benchmark | model in --models and dataset in --datasets |
| Training | model in --models, dataset in --datasets, and total GPU memory of at least min_vram if set |
| Inference | model in --models, and total GPU memory of at least min_vram if set |
GPU memory is summed across all detected GPUs. A node with no detected GPU
cannot claim any task that sets min_vram.
For example, this node can claim benchmarks of llama-3-8b or resnet50 against
gpqa or mmlu:
cargo run -- -b 127.0.0.1:5674 -l 127.0.0.1:2675 \
--models llama-3-8b,resnet50 --datasets gpqa,mmluThe coordinator checks claims against the capabilities the claiming peer advertised during the handshake, so a peer cannot win a task it did not advertise support for.
When --rpc-addr is specified, a JSON-RPC server is started at the given address.
Bind it to loopback (127.0.0.1) unless you need remote access; see
RPC authentication below.
| Method | Description | Returns |
|---|---|---|
info |
Get node information | Node ID, listen address, peer counts |
peers |
Get all known peers | List of peer IDs and addresses |
connections |
Get active connections | List of currently connected peers |
capabilities |
Get a node's capabilities | cores, memory, gpus, models |
task_status |
Get a task this node coordinates, by task ID | Task with status, assigned worker and result, or null if unknown |
broadcast_task |
Announce a task to the network (state-mutating) | true if broadcast |
List active connections:
curl -X POST http://127.0.0.1:3330 \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $THALA_RPC_TOKEN" \
-d '{
"jsonrpc": "2.0",
"method": "connections",
"params": [],
"id": 3
}'Start the node with a token to require Authorization: Bearer <token> on every
method. Prefer the environment variable, since command-line flags are visible in
the process list:
export THALA_RPC_TOKEN=$(openssl rand -hex 32)
cargo run -- -l 127.0.0.1:2345 --rpc-addr 127.0.0.1:3330Tokens are compared in constant time. Calls without a valid token fail with
JSON-RPC error code -32001.
| Configuration | Read-only methods | broadcast_task |
|---|---|---|
| Token set | token required | token required |
No token, loopback --rpc-addr |
open | open |
No token, non-loopback --rpc-addr |
open (a warning is logged) | refused |
The token is sent in plaintext over HTTP. To reach a node remotely, put the RPC port behind TLS (a reverse proxy or SSH tunnel) rather than exposing it directly.
You can also interact with any thala node via the CLI client
cargo install --locked --path client --bin thala-clientPass the token with --token, or set THALA_RPC_TOKEN, when the node requires one.
Get node capabilities
thala-client --node-rpc http://127.0.0.1:3330 capabilitiesSubmit tasks to the network
Nodes claim tasks they are capable of: the model (and dataset, where applicable) must be in the node's --models/--datasets lists, and --min-vram-gb (optional) requires the node's aggregate GPU VRAM to meet the given amount.
thala-client --node-rpc http://127.0.0.1:3330 task benchmark --model resnet50 --dataset gpqa
thala-client --node-rpc http://127.0.0.1:3330 task inference --model resnet50 --min-vram-gb 16
thala-client --node-rpc http://127.0.0.1:3330 task training --model resnet50 --dataset gpqa --min-vram-gb 24Check a task's status
Query the node the task was submitted to, using the ID printed when it was created. The report shows the status (Pending, Running, Completed or Failed), the assigned worker, and the worker's result once it has reported one. Finished tasks are kept until they expire.
thala-client --node-rpc http://127.0.0.1:3330 status 00ee0e4a-388b-4102-b53d-b7791f27f252Peer connections are mutually authenticated at handshake time: each side signs a
challenge bound to the connection (fresh nonces from both peers, a role byte, and
the signer's identity claims), verified against the ed25519 public key inlined in
its claimed PeerId. A peer that cannot produce a valid signature is rejected before
it enters connections or known_peers.
Two limitations to be aware of:
- Gossiped peer info (
known_peersshared during the handshake) is hearsay until that peer is dialed and authenticates directly. Gossip never overwrites entries learned from an authenticated handshake. - Traffic is authenticated at handshake time but not encrypted — transport encryption (Noise) is a planned follow-up.
The node implements exponential backoff for reconnection attempts:
- Initial reconnection interval:
peer-reconnection-intervalseconds - After each failed attempt, the interval is multiplied by
backoff-multiplier - The interval is capped at
max-backoff-intervalseconds - After
reconnection-retries-capfailures, the peer is considered dead
Set RUST_LOG to control logging verbosity:
# Show all logs
RUST_LOG=trace cargo run
# Show only warnings and errors
RUST_LOG=warn cargo run
# Show info from thala, debug from other modules
RUST_LOG=thala=info,debug cargo run