feat(sw): parallel byte-range fan-out, streaming decrypt, Cache API persistence#1
Merged
Merged
Conversation
…ersistence The loader service worker fetched a resource's byte-range windows one at a time and buffered the full decrypted output before responding. For a large or multi-window resource this stacked N sequential RTTs and delayed the first byte to the page until every chunk had been decrypted. - Fetch: window 0 is still fetched alone (it is the only way to learn the resource's total length), but every remaining window is now fanned out concurrently (bounded to 6 in flight) instead of awaited one at a time, for both the cacheable GET content path and the POST JSON-RPC fallback. Verification is unchanged — the merkle inclusion proof still requires (and gets) the fully-reassembled ciphertext before anything is trusted. - Decrypt: chunk 0 is still decrypted eagerly, before any Response exists, preserving the exact prior fail-closed "wrong key/decoy -> 404" behavior. Chunks 1..N are now decrypted lazily as the Response's ReadableStream is pulled, so a large multi-chunk resource starts flowing to the browser before the last chunk is decrypted. Every chunk still goes through its own AEAD-authenticated decryptChunk() call. - Cache API: the verified dig-client WASM is persisted keyed by its pinned SHA-256, so a cache hit skips the fetch and lets the browser reuse its compiled-code cache instead of recompiling. A decrypted PINNED (concrete-root) resource is persisted keyed by store id + root + resource key, so a repeat load — even a fresh SW instance — skips the network round-trip and decrypt entirely. An unpinned "latest" read is never persisted this way, since that could serve stale content after the value changes. Adds a Node-native unit suite (test/, no dependencies) covering the new byte-range planner, the bounded parallel fan-out, the cache-key builders, and the streaming-decrypt assembly against a deterministic crypto stub — real AEAD/merkle correctness stays covered by digstore's dig-client-wasm Rust suite. Wires a new CI job to run it. Extends SPEC.md §9 with the normative behavior and bumps the minor version for the new capability.
MichaelTaylor3d
added a commit
that referenced
this pull request
Jul 10, 2026
…ersistence (#1) The loader service worker fetched a resource's byte-range windows one at a time and buffered the full decrypted output before responding. For a large or multi-window resource this stacked N sequential RTTs and delayed the first byte to the page until every chunk had been decrypted. - Fetch: window 0 is still fetched alone (it is the only way to learn the resource's total length), but every remaining window is now fanned out concurrently (bounded to 6 in flight) instead of awaited one at a time, for both the cacheable GET content path and the POST JSON-RPC fallback. Verification is unchanged — the merkle inclusion proof still requires (and gets) the fully-reassembled ciphertext before anything is trusted. - Decrypt: chunk 0 is still decrypted eagerly, before any Response exists, preserving the exact prior fail-closed "wrong key/decoy -> 404" behavior. Chunks 1..N are now decrypted lazily as the Response's ReadableStream is pulled, so a large multi-chunk resource starts flowing to the browser before the last chunk is decrypted. Every chunk still goes through its own AEAD-authenticated decryptChunk() call. - Cache API: the verified dig-client WASM is persisted keyed by its pinned SHA-256, so a cache hit skips the fetch and lets the browser reuse its compiled-code cache instead of recompiling. A decrypted PINNED (concrete-root) resource is persisted keyed by store id + root + resource key, so a repeat load — even a fresh SW instance — skips the network round-trip and decrypt entirely. An unpinned "latest" read is never persisted this way, since that could serve stale content after the value changes. Adds a Node-native unit suite (test/, no dependencies) covering the new byte-range planner, the bounded parallel fan-out, the cache-key builders, and the streaming-decrypt assembly against a deterministic crypto stub — real AEAD/merkle correctness stays covered by digstore's dig-client-wasm Rust suite. Wires a new CI job to run it. Extends SPEC.md §9 with the normative behavior and bumps the minor version for the new capability. Co-Authored-By: Claude <noreply@anthropic.com>
MichaelTaylor3d
added a commit
that referenced
this pull request
Jul 10, 2026
…ersistence (#1) The loader service worker fetched a resource's byte-range windows one at a time and buffered the full decrypted output before responding. For a large or multi-window resource this stacked N sequential RTTs and delayed the first byte to the page until every chunk had been decrypted. - Fetch: window 0 is still fetched alone (it is the only way to learn the resource's total length), but every remaining window is now fanned out concurrently (bounded to 6 in flight) instead of awaited one at a time, for both the cacheable GET content path and the POST JSON-RPC fallback. Verification is unchanged — the merkle inclusion proof still requires (and gets) the fully-reassembled ciphertext before anything is trusted. - Decrypt: chunk 0 is still decrypted eagerly, before any Response exists, preserving the exact prior fail-closed "wrong key/decoy -> 404" behavior. Chunks 1..N are now decrypted lazily as the Response's ReadableStream is pulled, so a large multi-chunk resource starts flowing to the browser before the last chunk is decrypted. Every chunk still goes through its own AEAD-authenticated decryptChunk() call. - Cache API: the verified dig-client WASM is persisted keyed by its pinned SHA-256, so a cache hit skips the fetch and lets the browser reuse its compiled-code cache instead of recompiling. A decrypted PINNED (concrete-root) resource is persisted keyed by store id + root + resource key, so a repeat load — even a fresh SW instance — skips the network round-trip and decrypt entirely. An unpinned "latest" read is never persisted this way, since that could serve stale content after the value changes. Adds a Node-native unit suite (test/, no dependencies) covering the new byte-range planner, the bounded parallel fan-out, the cache-key builders, and the streaming-decrypt assembly against a deterministic crypto stub — real AEAD/merkle correctness stays covered by digstore's dig-client-wasm Rust suite. Wires a new CI job to run it. Extends SPEC.md §9 with the normative behavior and bumps the minor version for the new capability. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the two performance items tracked in dig_ecosystem#4 (migrated TASKS.md #205d/#205e):
sw.jsfetched a resource's byte-rangewindows one at a time and only responded once every chunk had been decrypted into one buffer.
Window 0 is still fetched alone (it's the only way to learn the resource's total length), but
every remaining window — on both the cacheable GET content path and the POST JSON-RPC fallback —
is now fanned out concurrently (bounded to 6 in flight) instead of sequentially. Decrypt output is
now a
ReadableStream: chunk 0 is still decrypted eagerly (before anyResponseexists, so thewrong-key/decoy case still yields a clean 404 exactly as before), and chunks 1..N decrypt lazily as
the stream is pulled, so a large multi-chunk resource starts flowing to the browser before the last
chunk finishes decrypting.
by its pinned SHA-256 — a hit skips the fetch and lets the browser reuse the compiled-code cache it
automatically associates with a Cache Storage entry (
instantiateStreaming), instead ofrecompiling on every load. A decrypted PINNED (concrete-root) resource is persisted keyed by store
id + root + resource key, so a repeat load — even a fresh SW instance / new browser session — skips
the network round-trip and decrypt entirely. An unpinned
"latest"read is never persisted thisway, since that could serve stale content forever after the value changes — the whole point of the
chain-anchored-root pin.
Security invariants preserved, not weakened: the merkle inclusion proof still requires (and gets)
the fully-reassembled ciphertext before anything is trusted — its leaf is a hash of the whole
resource, so it cannot be verified incrementally regardless; only the network round-trips leading up
to it are now concurrent. Every chunk is still decrypted through its own AEAD-authenticated
decryptChunk()call — nothing is skipped. The wasm SHA-256 fail-closed gate is unchanged (hashverified at write-time; a same-origin, SW-private Cache Storage entry is trusted on read exactly like
an already-verified in-memory module would be). Full reasoning is in the doc comments in
assets/sw.jsand in
SPEC.md§9 (new).What changed
assets/sw.js— parallel window fan-out (planWindows/runParallel), streaming decrypt(
buildDecryptStream), WASM + decrypted-content Cache API persistence.test/(new, no dependencies) — Node-native (node --test) unit suite for the range planner,bounded parallel fan-out, cache-key builders, and decrypt-stream assembly, against a deterministic
crypto stub (
test/stub-dig-client.mjs); real AEAD/merkle correctness stays covered by digstore'sdig-client-wasmRust suite.test/load-sw.mjsdocuments howassets/sw.js(a browser-only ESmodule) is loaded under plain Node for these tests.
.github/workflows/ci.yml— newjs testjob running the suite (Node 22, no install step).SPEC.md— new §9 documenting the parallel-fetch/streaming-decrypt/persistent-cache behavior as anormative contract; §8 clarified to scope "byte-identical" to the external route/wire contract, not
sw.js's internal strategy.README.md/runbooks/local.md— mention the newtest/suite and local test command.Cargo.toml/Cargo.lock—0.1.0->0.2.0(minor: new capability, no behavior-breakingchange to any route, status code, or wire format).
Verification
node --test test/sw.test.mjs— 20 passing tests (range planner, parallel fan-out concurrencybound + error propagation, cache-key builders, single/multi-chunk/failing decrypt-stream assembly,
plus the pre-existing pure URN/header helpers now covered for the first time).
cargo test --all-targets— 29 passing (unaffected; no Rust source changed).cargo clippy --all-targets -- -D warnings/cargo clippy --bin bootstrap --features aws— clean.cargo fmt --all --check— clean.Test plan
node --test test/sw.test.mjsgreen locallycargo test --all-targetsgreen locallycargo clippy(lib +bootstrap --features aws) green locallycargo fmt --all --checkclean