Streaming/push-based chain sources: interest and coordination?

[rafaelturon]

Hey everyone,

I've been building a streaming Electrum chain source that uses
blockchain.scripthash.subscribe notifications (push model) instead of the
traditional full_scan()/sync() polling model. The project is at bdk-electrum-streaming-poc.

I'm opening this discussion to coordinate with the community before
investing further in a standalone bdk_electrum_stream crate.

Background: Issue #527 (2022, by @LLFourn) originally proposed real-time
streaming APIs for BDK. I'm aware of @evanlinjin's streaming experiment as well.
I'd love to understand how these relate to any current upstream plans.

What my PoC does:

• Hexagonal architecture: pure logic engine (no IO) + async Electrum adapter
  over raw TLS + orchestrator that bridges them
• Subscribes to all wallet script hashes via scripthash.subscribe
• Reacts to push notifications by fetching history, transactions, and block
  headers, then constructing Update structs with proper ConfirmationBlockTime
  anchors
• Produces matching balances compared to polling-based bdk_electrum

What I learned building it:

• The Update struct contract (anchors/seen_ats requirement) is the key
  interface between BDK and any custom chain source, but it's undocumented
  (filed as issue docs: Document TxUpdate temporal context requirements (supports Wizardsardine audit recommendation) #2133, reinforcing the Wizardsardine audit recommendation)
• The existing chain source crates handle temporal context internally, which
  means this knowledge is locked inside implementations rather than exposed
  as a documented API surface
• The change descriptor requirement (PR Make Wallet require a change descriptor #1390) is exactly right — I
  independently hit the same issue before discovering it was already fixed

Questions for the community:

1. Is there interest in a bdk_electrum_stream crate as an external chain
   source? If so, what API shape would be most useful — one-shot streaming
   sync, continuous background sync, or low-level engine access?

2. How does my work relate to @evanlinjin's streaming experiment? I want to
   complement, not duplicate.

3. Would a "Building Custom Chain Sources" guide in the Book of BDK be
   welcome? I've filed a separate issue on book-of-bdk for this: docs: Add "Building a Custom Chain Source" cookbook page book-of-bdk#142.

4. For the evicted_at / last_evicted timestamps introduced in PR Introduce evicted-at/last-evicted timestamps #1839 —
   should a streaming chain source also populate these? The Electrum protocol
   doesn't have an explicit "transaction evicted from mempool" notification,
   but absence from a get_history response could imply it.

Happy to share architecture details or discuss any of these points.

[tnull]

1. Is there interest in a bdk_electrum_stream crate as an external chain
   source? If so, what API shape would be most useful — one-shot streaming
   sync, continuous background sync, or low-level engine access?

Yes, we (LDK / LDK Node) would be very interested in using an async and streaming client as we ran into quite a few issues with the current blocking/polling client. Mostly, that's due to the blocking part of it which doesn't play nicely in an otherwise async environment. That said, to be really useful, we'd probably need to also adjust/reimplement the lightning-transaction-sync crate (see https://github.com/lightningdevkit/rust-lightning/tree/main/lightning-transaction-sync) based on a streaming model. That said, for us to fully switch to a streaming model we'd at least need to wait for Electrum protocol v1.7 which will add the ability to register for outpoint spend notifications.

[rafaelturon]

Great point on Electrum v1.7 and outpoint spend notifications — that's
a real constraint for Lightning use cases.

@evanlinjin pointed me to bitcoindevkit/electrum_streaming_client which
I wasn't aware of when I started. I'm going to study it closely — the
sans-IO architecture is very similar to what I built independently, so
the concepts should map well.

My PoC focuses on a different layer: turning streaming Electrum events
into correctly-formed bdk_wallet Updates (anchors, seen_ats, lookahead
management). If that orchestration layer is useful, I'd be happy to
help build the bridge between electrum_streaming_client and
lightning-transaction-sync as well.

[sueun-dev]

Thanks for writing this up so concretely. This looks like useful work, and I think there is real value in keeping the design layered.

My preference would be:

• a low-level streaming primitive that emits Updates or wallet-relevant deltas,
• plus an optional managed runner that handles reconnect, backfill, and reorg recovery.

That gives downstream users an escape hatch while still making the common case easy.

On evicted_at / last_evicted: I would be careful about populating those from absence alone. With Electrum, missing from a subsequent get_history response can mean eviction, but it can also reflect transient indexing lag, reconnect gaps, or reorg-related ambiguity. I would only set eviction metadata when you have positive evidence or a well-defined reconciliation rule; otherwise leaving it unset seems safer than asserting a false eviction.

A "Building Custom Chain Sources" guide also sounds worthwhile. The boundary around Update, anchors, and temporal semantics is exactly the kind of thing that is easy to get subtly wrong without written guidance.