Skip to content

feat(sw): parallel byte-range fan-out, streaming decrypt, Cache API persistence#1

Merged
MichaelTaylor3d merged 1 commit into
mainfrom
feat/sw-streaming-205
Jul 4, 2026
Merged

feat(sw): parallel byte-range fan-out, streaming decrypt, Cache API persistence#1
MichaelTaylor3d merged 1 commit into
mainfrom
feat/sw-streaming-205

Conversation

@MichaelTaylor3d

Copy link
Copy Markdown
Contributor

Summary

Closes the two performance items tracked in dig_ecosystem#4 (migrated TASKS.md #205d/#205e):

  • #205d — parallel range fan-out + streaming decrypt. sw.js fetched a resource's byte-range
    windows one at a time and only responded once every chunk had been decrypted into one buffer.
    Window 0 is still fetched alone (it's the only way to learn the resource's total length), but
    every remaining window — on both the cacheable GET content path and the POST JSON-RPC fallback —
    is now fanned out concurrently (bounded to 6 in flight) instead of sequentially. Decrypt output is
    now a ReadableStream: chunk 0 is still decrypted eagerly (before any Response exists, so the
    wrong-key/decoy case still yields a clean 404 exactly as before), and chunks 1..N decrypt lazily as
    the stream is pulled, so a large multi-chunk resource starts flowing to the browser before the last
    chunk finishes decrypting.
  • #205e — Cache API persistence. The verified dig-client WASM is persisted in the Cache API keyed
    by its pinned SHA-256 — a hit skips the fetch and lets the browser reuse the compiled-code cache it
    automatically associates with a Cache Storage entry (instantiateStreaming), instead of
    recompiling on every load. A decrypted PINNED (concrete-root) resource is persisted keyed by store
    id + root + resource key, so a repeat load — even a fresh SW instance / new browser session — skips
    the network round-trip and decrypt entirely. An unpinned "latest" read is never persisted this
    way, since that could serve stale content forever after the value changes — the whole point of the
    chain-anchored-root pin.

Security invariants preserved, not weakened: the merkle inclusion proof still requires (and gets)
the fully-reassembled ciphertext before anything is trusted — its leaf is a hash of the whole
resource, so it cannot be verified incrementally regardless; only the network round-trips leading up
to it are now concurrent. Every chunk is still decrypted through its own AEAD-authenticated
decryptChunk() call — nothing is skipped. The wasm SHA-256 fail-closed gate is unchanged (hash
verified at write-time; a same-origin, SW-private Cache Storage entry is trusted on read exactly like
an already-verified in-memory module would be). Full reasoning is in the doc comments in assets/sw.js
and in SPEC.md §9 (new).

What changed

  • assets/sw.js — parallel window fan-out (planWindows/runParallel), streaming decrypt
    (buildDecryptStream), WASM + decrypted-content Cache API persistence.
  • test/ (new, no dependencies) — Node-native (node --test) unit suite for the range planner,
    bounded parallel fan-out, cache-key builders, and decrypt-stream assembly, against a deterministic
    crypto stub (test/stub-dig-client.mjs); real AEAD/merkle correctness stays covered by digstore's
    dig-client-wasm Rust suite. test/load-sw.mjs documents how assets/sw.js (a browser-only ES
    module) is loaded under plain Node for these tests.
  • .github/workflows/ci.yml — new js test job running the suite (Node 22, no install step).
  • SPEC.md — new §9 documenting the parallel-fetch/streaming-decrypt/persistent-cache behavior as a
    normative contract; §8 clarified to scope "byte-identical" to the external route/wire contract, not
    sw.js's internal strategy.
  • README.md / runbooks/local.md — mention the new test/ suite and local test command.
  • Cargo.toml / Cargo.lock0.1.0 -> 0.2.0 (minor: new capability, no behavior-breaking
    change to any route, status code, or wire format).

Verification

  • node --test test/sw.test.mjs — 20 passing tests (range planner, parallel fan-out concurrency
    bound + error propagation, cache-key builders, single/multi-chunk/failing decrypt-stream assembly,
    plus the pre-existing pure URN/header helpers now covered for the first time).
  • cargo test --all-targets — 29 passing (unaffected; no Rust source changed).
  • cargo clippy --all-targets -- -D warnings / cargo clippy --bin bootstrap --features aws — clean.
  • cargo fmt --all --check — clean.

Test plan

  • node --test test/sw.test.mjs green locally
  • cargo test --all-targets green locally
  • cargo clippy (lib + bootstrap --features aws) green locally
  • cargo fmt --all --check clean
  • CI green on this PR (rust lint + test, js test, cargo-deny, terraform fmt + validate, commitlint, version-increment)

…ersistence

The loader service worker fetched a resource's byte-range windows one at a
time and buffered the full decrypted output before responding. For a large
or multi-window resource this stacked N sequential RTTs and delayed the
first byte to the page until every chunk had been decrypted.

- Fetch: window 0 is still fetched alone (it is the only way to learn the
  resource's total length), but every remaining window is now fanned out
  concurrently (bounded to 6 in flight) instead of awaited one at a time,
  for both the cacheable GET content path and the POST JSON-RPC fallback.
  Verification is unchanged — the merkle inclusion proof still requires
  (and gets) the fully-reassembled ciphertext before anything is trusted.
- Decrypt: chunk 0 is still decrypted eagerly, before any Response exists,
  preserving the exact prior fail-closed "wrong key/decoy -> 404" behavior.
  Chunks 1..N are now decrypted lazily as the Response's ReadableStream is
  pulled, so a large multi-chunk resource starts flowing to the browser
  before the last chunk is decrypted. Every chunk still goes through its
  own AEAD-authenticated decryptChunk() call.
- Cache API: the verified dig-client WASM is persisted keyed by its
  pinned SHA-256, so a cache hit skips the fetch and lets the browser
  reuse its compiled-code cache instead of recompiling. A decrypted PINNED
  (concrete-root) resource is persisted keyed by store id + root +
  resource key, so a repeat load — even a fresh SW instance — skips the
  network round-trip and decrypt entirely. An unpinned "latest" read is
  never persisted this way, since that could serve stale content after
  the value changes.

Adds a Node-native unit suite (test/, no dependencies) covering the new
byte-range planner, the bounded parallel fan-out, the cache-key builders,
and the streaming-decrypt assembly against a deterministic crypto stub —
real AEAD/merkle correctness stays covered by digstore's dig-client-wasm
Rust suite. Wires a new CI job to run it. Extends SPEC.md §9 with the
normative behavior and bumps the minor version for the new capability.
@MichaelTaylor3d MichaelTaylor3d merged commit 75b5a51 into main Jul 4, 2026
6 checks passed
@MichaelTaylor3d MichaelTaylor3d deleted the feat/sw-streaming-205 branch July 4, 2026 14:23
MichaelTaylor3d added a commit that referenced this pull request Jul 10, 2026
…ersistence (#1)

The loader service worker fetched a resource's byte-range windows one at a
time and buffered the full decrypted output before responding. For a large
or multi-window resource this stacked N sequential RTTs and delayed the
first byte to the page until every chunk had been decrypted.

- Fetch: window 0 is still fetched alone (it is the only way to learn the
  resource's total length), but every remaining window is now fanned out
  concurrently (bounded to 6 in flight) instead of awaited one at a time,
  for both the cacheable GET content path and the POST JSON-RPC fallback.
  Verification is unchanged — the merkle inclusion proof still requires
  (and gets) the fully-reassembled ciphertext before anything is trusted.
- Decrypt: chunk 0 is still decrypted eagerly, before any Response exists,
  preserving the exact prior fail-closed "wrong key/decoy -> 404" behavior.
  Chunks 1..N are now decrypted lazily as the Response's ReadableStream is
  pulled, so a large multi-chunk resource starts flowing to the browser
  before the last chunk is decrypted. Every chunk still goes through its
  own AEAD-authenticated decryptChunk() call.
- Cache API: the verified dig-client WASM is persisted keyed by its
  pinned SHA-256, so a cache hit skips the fetch and lets the browser
  reuse its compiled-code cache instead of recompiling. A decrypted PINNED
  (concrete-root) resource is persisted keyed by store id + root +
  resource key, so a repeat load — even a fresh SW instance — skips the
  network round-trip and decrypt entirely. An unpinned "latest" read is
  never persisted this way, since that could serve stale content after
  the value changes.

Adds a Node-native unit suite (test/, no dependencies) covering the new
byte-range planner, the bounded parallel fan-out, the cache-key builders,
and the streaming-decrypt assembly against a deterministic crypto stub —
real AEAD/merkle correctness stays covered by digstore's dig-client-wasm
Rust suite. Wires a new CI job to run it. Extends SPEC.md §9 with the
normative behavior and bumps the minor version for the new capability.

Co-Authored-By: Claude <noreply@anthropic.com>
MichaelTaylor3d added a commit that referenced this pull request Jul 10, 2026
…ersistence (#1)

The loader service worker fetched a resource's byte-range windows one at a
time and buffered the full decrypted output before responding. For a large
or multi-window resource this stacked N sequential RTTs and delayed the
first byte to the page until every chunk had been decrypted.

- Fetch: window 0 is still fetched alone (it is the only way to learn the
  resource's total length), but every remaining window is now fanned out
  concurrently (bounded to 6 in flight) instead of awaited one at a time,
  for both the cacheable GET content path and the POST JSON-RPC fallback.
  Verification is unchanged — the merkle inclusion proof still requires
  (and gets) the fully-reassembled ciphertext before anything is trusted.
- Decrypt: chunk 0 is still decrypted eagerly, before any Response exists,
  preserving the exact prior fail-closed "wrong key/decoy -> 404" behavior.
  Chunks 1..N are now decrypted lazily as the Response's ReadableStream is
  pulled, so a large multi-chunk resource starts flowing to the browser
  before the last chunk is decrypted. Every chunk still goes through its
  own AEAD-authenticated decryptChunk() call.
- Cache API: the verified dig-client WASM is persisted keyed by its
  pinned SHA-256, so a cache hit skips the fetch and lets the browser
  reuse its compiled-code cache instead of recompiling. A decrypted PINNED
  (concrete-root) resource is persisted keyed by store id + root +
  resource key, so a repeat load — even a fresh SW instance — skips the
  network round-trip and decrypt entirely. An unpinned "latest" read is
  never persisted this way, since that could serve stale content after
  the value changes.

Adds a Node-native unit suite (test/, no dependencies) covering the new
byte-range planner, the bounded parallel fan-out, the cache-key builders,
and the streaming-decrypt assembly against a deterministic crypto stub —
real AEAD/merkle correctness stays covered by digstore's dig-client-wasm
Rust suite. Wires a new CI job to run it. Extends SPEC.md §9 with the
normative behavior and bumps the minor version for the new capability.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant