Skip to content

Commit 6dfeccf

Browse files
committed
Add public shares design docs and remote-access server improvements
- Add SHARE-000..011 task specs and docs/design/shares.md defining the sd.app public sharing architecture and wire protocols - Rework INDEX-010 to scope ephemeral UUID reconciliation per-library instead of a single global map, since core can have multiple libraries loaded at once - Add sidecar serving and configurable bind host to apps/server, and wire the web platform's daemon status to the server's own origin - Add quick preview button to the file inspector, with a null-safe useOptionalExplorer for use outside the explorer context
1 parent 60369e9 commit 6dfeccf

21 files changed

Lines changed: 1031 additions & 67 deletions

.tasks/core/INDEX-010-bidirectional-uuid-reconciliation.md

Lines changed: 98 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ assignee: jamiepine
66
parent: INDEX-000
77
priority: Critical
88
tags: [indexing, ephemeral, persistent, uuid, foundation]
9-
last_updated: 2026-02-07
9+
last_updated: 2026-05-10
1010
related_tasks: [INDEX-001, FSYNC-003, FILE-006]
1111
---
1212

@@ -16,6 +16,8 @@ The ephemeral and persistent indexes currently share UUIDs in one direction only
1616

1717
This task makes the ephemeral index a true superset layer on top of the persistent index by reusing persistent UUIDs when they exist. This is the foundational primitive for file sync, smart copy, and path intersection operations.
1818

19+
The core can have multiple libraries loaded at the same time and does not have a global "active library". Reconciliation must therefore be library-scoped: the filesystem structure can stay shared in the global ephemeral index, but UUID identity must be resolved per library.
20+
1921
## Problem
2022

2123
- Volume indexing an already-persistent location generates new UUIDs, duplicating identity
@@ -35,44 +37,91 @@ Ephemeral Browse → Assign v4 UUIDs → [promote] → Persistent stores same UU
3537
### Target Flow (Bidirectional)
3638

3739
```
38-
Ephemeral Browse → Assign v4 UUIDs → [reconcile] → Check persistent index
39-
├── Match found → adopt persistent UUID
40-
└── No match → keep v4 UUID
40+
Library-scoped ephemeral browse → Assign temporary UUIDs for that library
41+
→ [reconcile] → Check that library's persistent index
42+
├── Match found → adopt persistent UUID
43+
└── No match → keep library-local v4 UUID
4144
```
4245

4346
### Design Constraints
4447

4548
1. **Do not slow down ephemeral discovery.** The ephemeral indexer must remain fast (~50K files/sec). No database queries during the filesystem walk.
4649
2. **Reconciliation is a separate pass.** After ephemeral discovery completes, run a background reconciliation against the persistent index for overlapping paths.
4750
3. **Lazy resolution as fallback.** If reconciliation hasn't run yet, UUID lookups can check the persistent index on demand.
48-
4. **Single EphemeralIndex instance.** The global `EphemeralIndexCache` holds one shared index. Reconciliation updates UUIDs in place.
51+
4. **Single shared filesystem index.** The global `EphemeralIndexCache` should keep one shared path and metadata structure for memory efficiency.
52+
5. **Library-scoped UUID overlay.** Reconciliation must not overwrite one global UUID per path. UUIDs are scoped by `(library_id, entry_id)` so two loaded libraries can map the same physical path to different persistent entry UUIDs.
53+
54+
### Library Scoping
55+
56+
The ephemeral cache is process-local and shared across all loaded libraries. Persistent databases are per-library. The same absolute path may exist in more than one loaded library, and each library can have different entry UUIDs, tags, metadata, sync state, and permissions.
57+
58+
The correct model is:
59+
60+
```text
61+
Shared ephemeral structure:
62+
path -> EntryId
63+
EntryId -> metadata
64+
EntryId -> content kind
65+
66+
Library identity overlay:
67+
library_id -> EntryId -> UUID
68+
```
69+
70+
This lets expensive filesystem discovery stay shared while keeping persistent identity correct for each loaded library.
4971

5072
## Implementation Steps
5173

52-
### 1. Add Persistent UUID Lookup to EphemeralIndex
74+
### 1. Add Library-Scoped UUID Storage to EphemeralIndex
5375

54-
Add a method that accepts pre-resolved UUIDs from an external source (the persistent DB) and patches them into the ephemeral index's `entry_uuids` map.
76+
Replace the single global `entry_uuids: HashMap<EntryId, Uuid>` with a library-scoped overlay. Existing call sites should pass the library ID when reading or assigning UUIDs.
5577

5678
```rust
5779
// core/src/ops/indexing/ephemeral/index.rs
5880

81+
pub type LibraryId = Uuid;
82+
83+
pub struct EphemeralIndex {
84+
// path and metadata fields stay shared
85+
entry_uuids_by_library: HashMap<LibraryId, HashMap<EntryId, Uuid>>,
86+
}
87+
5988
impl EphemeralIndex {
60-
/// Reconcile ephemeral UUIDs with persistent entries.
61-
/// For each path in the provided map, if a matching ephemeral entry exists,
62-
/// replace its UUID with the persistent one.
63-
/// Returns count of UUIDs reconciled.
89+
pub fn get_entry_uuid(&self, library_id: Uuid, path: &Path) -> Option<Uuid> {
90+
let entry_id = self.path_index.get(path)?;
91+
self.entry_uuids_by_library
92+
.get(&library_id)?
93+
.get(entry_id)
94+
.copied()
95+
}
96+
97+
pub fn get_or_assign_uuid(&mut self, library_id: Uuid, path: &Path) -> Uuid {
98+
let Some(&entry_id) = self.path_index.get(path) else {
99+
return Uuid::new_v4();
100+
};
101+
102+
let uuids = self.entry_uuids_by_library.entry(library_id).or_default();
103+
*uuids.entry(entry_id).or_insert_with(Uuid::new_v4)
104+
}
105+
64106
pub fn reconcile_persistent_uuids(
65107
&mut self,
108+
library_id: Uuid,
66109
persistent_uuids: &HashMap<PathBuf, Uuid>,
67-
) -> usize {
110+
) -> Vec<(PathBuf, Uuid)> {
111+
let uuids = self.entry_uuids_by_library.entry(library_id).or_default();
112+
let mut changed = Vec::new();
113+
68114
let mut count = 0;
69115
for (path, persistent_uuid) in persistent_uuids {
70116
if let Some(&entry_id) = self.path_index.get(path) {
71-
self.entry_uuids.insert(entry_id, *persistent_uuid);
72-
count += 1;
117+
let existing = uuids.insert(entry_id, *persistent_uuid);
118+
if existing != Some(*persistent_uuid) {
119+
changed.push((path.clone(), *persistent_uuid));
120+
}
73121
}
74122
}
75-
count
123+
124+
changed
76125
}
77126
}
78127
```
@@ -124,49 +173,28 @@ pub async fn extract_persistent_uuids_for_path(
124173

125174
For large persistent locations this query could return thousands of entries. Batch the path resolution and use the `directory_paths` cache (O(1) per directory) to keep it fast.
126175

127-
### 3. Reconciliation Pass on EphemeralIndexCache
176+
### 3. Library-Scoped Reconciliation Pass on EphemeralIndexCache
128177

129-
After ephemeral discovery completes for a path, check if any persistent locations overlap with the scanned path and run reconciliation.
178+
After ephemeral discovery completes for a path, reconcile against the library that requested the operation. Do not scan all loaded libraries and overwrite global UUIDs.
130179

131180
```rust
132181
// core/src/ops/indexing/ephemeral/cache.rs
133182

134183
impl EphemeralIndexCache {
135-
/// Run after ephemeral indexing completes for a path.
136-
/// Checks all libraries for persistent locations that overlap with the
137-
/// ephemeral path and reconciles UUIDs.
138184
pub async fn reconcile_with_persistent(
139185
&self,
186+
library_id: Uuid,
140187
scanned_path: &Path,
141-
libraries: &LibraryManager,
142-
) -> usize {
143-
let mut total = 0;
144-
145-
for library in libraries.list().await {
146-
let db = library.db();
147-
match extract_persistent_uuids_for_path(db, scanned_path).await {
148-
Ok(persistent_uuids) if !persistent_uuids.is_empty() => {
149-
let mut index = self.index.write().await;
150-
total += index.reconcile_persistent_uuids(&persistent_uuids);
151-
}
152-
Ok(_) => {} // No overlap with this library
153-
Err(e) => {
154-
tracing::warn!(
155-
"Failed to reconcile UUIDs for library {}: {}",
156-
library.id(), e
157-
);
158-
}
159-
}
160-
}
188+
db: &DatabaseConnection,
189+
) -> Result<Vec<(PathBuf, Uuid)>> {
190+
let persistent_uuids = extract_persistent_uuids_for_path(db, scanned_path).await?;
161191

162-
if total > 0 {
163-
tracing::info!(
164-
"Reconciled {} ephemeral UUIDs with persistent index for {}",
165-
total, scanned_path.display()
166-
);
192+
if persistent_uuids.is_empty() {
193+
return Ok(Vec::new());
167194
}
168195

169-
total
196+
let mut index = self.index.write().await;
197+
Ok(index.reconcile_persistent_uuids(library_id, &persistent_uuids))
170198
}
171199
}
172200
```
@@ -180,18 +208,19 @@ Wire reconciliation into the ephemeral indexing job completion path. The indexer
180208

181209
cache.mark_indexing_complete(&path);
182210

183-
// Reconcile with persistent index in background
211+
// Reconcile with this library's persistent index in the background.
184212
let cache_clone = cache.clone();
185-
let libraries = ctx.library().core_context().libraries().await;
213+
let library_id = ctx.library().id();
214+
let db = ctx.library().db().clone();
186215
let path_clone = path.clone();
187216
tokio::spawn(async move {
188-
cache_clone
189-
.reconcile_with_persistent(&path_clone, &libraries)
217+
let _ = cache_clone
218+
.reconcile_with_persistent(library_id, &path_clone, db.conn())
190219
.await;
191220
});
192221
```
193222

194-
Spawning as a background task keeps the indexing job fast. The UI shows ephemeral UUIDs immediately, then silently corrects them when reconciliation completes. Since the ephemeral index is the browsing layer, UUID changes propagate to the UI via the existing `ResourceChanged` event system.
223+
Spawning as a background task keeps the indexing job fast. The UI shows library-local ephemeral UUIDs immediately, then corrects them when reconciliation completes. Since UUIDs are library-scoped, another loaded library viewing the same path is unaffected.
195224

196225
### 5. Lazy Fallback: On-Demand UUID Resolution
197226

@@ -205,11 +234,12 @@ impl EphemeralIndex {
205234
/// Used when reconciliation hasn't completed yet.
206235
pub async fn get_or_resolve_uuid(
207236
&mut self,
237+
library_id: Uuid,
208238
path: &PathBuf,
209239
persistent_lookup: Option<&dyn PersistentUuidLookup>,
210240
) -> Option<Uuid> {
211241
// Fast path: already have a UUID (either generated or reconciled)
212-
if let Some(uuid) = self.get_entry_uuid(path) {
242+
if let Some(uuid) = self.get_entry_uuid(library_id, path) {
213243
return Some(uuid);
214244
}
215245

@@ -218,7 +248,10 @@ impl EphemeralIndex {
218248
if let Some(persistent_uuid) = lookup.lookup_uuid(path).await {
219249
// Cache for future access
220250
if let Some(&entry_id) = self.path_index.get(path) {
221-
self.entry_uuids.insert(entry_id, persistent_uuid);
251+
self.entry_uuids_by_library
252+
.entry(library_id)
253+
.or_default()
254+
.insert(entry_id, persistent_uuid);
222255
}
223256
return Some(persistent_uuid);
224257
}
@@ -240,7 +273,7 @@ pub trait PersistentUuidLookup: Send + Sync {
240273

241274
### 6. Emit Events on UUID Reconciliation
242275

243-
When a UUID changes from a temporary v4 to a persistent UUID, emit a `ResourceChanged` event so the frontend updates references.
276+
When a UUID changes from a temporary library-local v4 to a persistent UUID, emit a `ResourceChanged` event for that library/session so the frontend updates references.
244277

245278
```rust
246279
// In reconcile_persistent_uuids(), collect changed entries:
@@ -262,6 +295,7 @@ This is important because the frontend may have cached the temporary UUID in sel
262295
- `core/src/ops/indexing/ephemeral/cache.rs` - Add `reconcile_with_persistent()`
263296
- `core/src/ops/indexing/job.rs` - Wire reconciliation after ephemeral completion
264297
- `core/src/ops/indexing/mod.rs` - Add `reconciliation` module
298+
- Ephemeral query/search call sites - Pass `library_id` into UUID reads and lazy assignment
265299

266300
## Acceptance Criteria
267301

@@ -271,9 +305,11 @@ This is important because the frontend may have cached the temporary UUID in sel
271305
- [ ] Lazy fallback resolves persistent UUIDs on demand when reconciliation hasn't completed
272306
- [ ] ResourceChanged events emitted when ephemeral UUIDs are replaced with persistent ones
273307
- [ ] Tags and metadata attached to persistent entries are visible in ephemeral views after reconciliation
274-
- [ ] Multiple libraries with overlapping paths are handled (all checked)
308+
- [ ] Multiple loaded libraries with overlapping paths are handled via separate UUID overlays
309+
- [ ] Reconciliation for one library does not overwrite UUIDs returned for another loaded library
275310
- [ ] Paths with no persistent overlap are unaffected (keep v4 UUIDs)
276311
- [ ] Integration test: ephemeral index of persistent location produces same UUIDs
312+
- [ ] Integration test: two loaded libraries can reconcile the same path to different UUIDs
277313
- [ ] Integration test: volume index reconciles UUIDs for all persistent locations on volume
278314
- [ ] Performance: reconciliation of 100K entries completes in under 2 seconds
279315

@@ -291,9 +327,15 @@ Users expect ephemeral browsing to feel instant. Reconciliation involves databas
291327

292328
A persistent location at `/Users/james/Documents` overlaps with an ephemeral scan of `/Users/james` (the ephemeral path is a parent). The reconciliation needs to check both directions: persistent roots that are children of the scanned path, and persistent roots that are parents of the scanned path.
293329

330+
### Multiple Loaded Libraries
331+
332+
Core does not know an active library. It only knows loaded libraries and library-scoped operations. Directory listing, search, and volume indexing must pass the library ID from their operation context into ephemeral UUID access.
333+
334+
Do not reconcile against all loaded libraries into a single global `path -> uuid` map. That would make whichever library reconciles last win, causing the wrong tags and metadata to appear for other libraries.
335+
294336
### Memory Impact
295337

296-
The `entry_uuids` HashMap already exists in the ephemeral index. Reconciliation doesn't add new entries — it replaces v4 UUIDs with persistent ones. No additional memory overhead.
338+
The path tree, metadata, name cache, and content kind storage remain shared. Only UUID mappings become per-library. Memory overhead is proportional to the number of ephemeral entries that have been viewed or reconciled in each loaded library, not to the full filesystem metadata structure.
297339

298340
## Related Tasks
299341

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
---
2+
id: SHARE-000
3+
title: "Epic: Public Shares via sd.app"
4+
status: To Do
5+
assignee: jamiepine
6+
priority: High
7+
tags: [epic, sharing, cloud, networking]
8+
related_tasks: [CLOUD-000, CLOUD-002, SEC-007, NET-001]
9+
---
10+
11+
## Description
12+
13+
Public sharing of Spacedrive Spaces (and individual files/folders) via sd.app. A user creates a share from their local instance, gets a `sd.app/s/{token}` link, and visitors browse/download content in a web viewer. sd.app provides the Iroh relay, share registry, viewer SPA, and account management. Bytes never transit sd.app — the browser dials the user's core directly through our relay over QUIC.
14+
15+
## Architecture (decided)
16+
17+
- **Bytes path**: direct-dial. Browser → sd.app relay → user's core (QUIC). sd.app does not proxy file content.
18+
- **sd.app responsibilities**: Iroh relay infrastructure, share registry (token → node_id/relay_url), public viewer SPA, user accounts for share management.
19+
- **Share unit**: Spaces (primary), plus single files and arbitrary folder selections.
20+
- **Permissions**: read-only public, optional password, optional expiry.
21+
- **Repo layout**: sd.app cloud service lives in a separate repo; this epic covers core + desktop interface only.
22+
23+
## Sub-tasks
24+
25+
- `SHARE-001` — Design: share architecture & wire protocol spec
26+
- `SHARE-002` — Core: PublicShare schema, sync, migrations
27+
- `SHARE-003` — Core: share CRUD operations & RPC surface
28+
- `SHARE-004` — Core: custom relay configuration (sd.app)
29+
- `SHARE-005` — Core: guest access protocol over Iroh
30+
- `SHARE-006` — Core: file/folder share scope (beyond Spaces)
31+
- `SHARE-007` — Core: expiry & revocation enforcement
32+
- `SHARE-008` — Core: share registration with sd.app registry
33+
- `SHARE-009` — Interface: share creation UI
34+
- `SHARE-010` — Interface: share management view
35+
- `SHARE-011` — Core + Interface: sd.app account linking
36+
37+
## Related Work
38+
39+
- `CLOUD-000` — Cloud as a Peer (parent rationale)
40+
- `CLOUD-002` — Asynchronous Relay Server (re-scoped; see SHARE-004 note)
41+
- `SEC-007` — Per-Library Encryption Policies for Public Sharing
42+
- `NET-001` — Iroh P2P stack (foundation, already Done)
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
---
2+
id: SHARE-001
3+
title: "Design: Share Architecture & Wire Protocols"
4+
status: In Progress
5+
assignee: jamiepine
6+
parent: SHARE-000
7+
priority: High
8+
tags: [sharing, design, protocol, cloud]
9+
last_updated: 2026-05-25
10+
---
11+
12+
## Description
13+
14+
Produce the contracts between core, sd.app, and the browser viewer. This is the source-of-truth design that both repos build against.
15+
16+
## Scope
17+
18+
1. **Sequence diagrams** for: share creation, share resolution (visitor lands on link), file listing, file streaming, share revocation, share expiry.
19+
2. **Wire protocol: core ↔ sd.app registry** (HTTPS / signed requests)
20+
- `POST /api/shares``{token, node_id, relay_url, public_metadata, password_required, expires_at}`
21+
- `POST /api/shares/{token}/heartbeat` — refresh relay address
22+
- `DELETE /api/shares/{token}` — revoke
23+
- `GET /s/{token}` — public resolver returning dial info to the viewer
24+
- Authenticated via device-key signature; rate-limited per device
25+
3. **Wire protocol: browser viewer ↔ core** (over Iroh QUIC, new ALPN `sd/share/1`)
26+
- Capability handshake (token + optional password proof)
27+
- `list_contents(path)` — directory listing scoped to share root
28+
- `get_metadata(path)` — file metadata
29+
- `read_range(path, offset, length)` — byte range stream
30+
- All requests gated by share scope on the core side
31+
4. **Token format**: 128-bit CSPRNG → base32 (no padding, no ambiguous chars). URL shape: `https://sd.app/s/{token}`. Optional `#k={key}` fragment reserved for client-side decryption keys (never sent to server).
32+
5. **Password handling**: argon2id hash stored on core. Password proof = HMAC over server-issued challenge nonce; password never sent in cleartext.
33+
6. **Public metadata**: name, item count, optional cover image, password_required flag. Owner identity is NOT exposed by default.
34+
35+
## Deliverables
36+
37+
- `docs/design/shares.md` with diagrams and protocol specs
38+
- ALPN string reserved for guest protocol
39+
- API schema (typed Rust structs + JSON schema) shared between core and sd.app
40+
41+
## Acceptance Criteria
42+
43+
- [ ] Design document covers all six sequence diagrams listed above
44+
- [ ] Wire protocols are typed and machine-checkable on both sides
45+
- [ ] Token / URL / password protocols are specified with rationale
46+
- [ ] Design is approved before SHARE-002+ begins

0 commit comments

Comments
 (0)