Idea: pluggable event-sourced shared memory backend for long-horizon / multi-agent runs

[realtonyyoung]

Deer-Flow's persistent memory + multi-agent coordination is a great foundation for long-horizon agents. One pattern we keep seeing: as runs get longer and more agents participate, teams want memory that's durable, auditable, and replayable — not just a store that's read and overwritten.

We've been building exactly that at Kurrent. Capacitor is a vendor-agnostic shared memory layer for coding agents, backed by an immutable, queryable event log. Because every memory write is an append-only event, you get cross-agent handoff, full history, and the ability to replay or correlate what each agent knew and when — which is handy for debugging multi-agent runs.

Curious whether a pluggable memory backend interface is in scope here, or already on the roadmap. Check out capacitor.kurrent.io for more info. Happy to prototype an adapter if there's interest — not looking to push a dependency, just to see if the shapes line up.

[WillemJiang]

@realtonyyoung thanks for bringing it up. We have recently been planning the memory work and are open to integrating with the third-party memory system. A prototype could be helpful. One concern is that DeerFlow's backend was written in Python, so we may need to discuss the integration protocol first.

[realtonyyoung]

Thanks @WillemJiang — glad the timing lines up with the memory work you're planning.

On the integration protocol: I don't think we need to invent one — DeerFlow already has the right seam. memory.storage_class reflection-loads any MemoryStorage subclass (deerflow/agents/memory/storage.py), so the prototype I'll build is:

• KurrentdbMemoryStorage(MemoryStorage) — pure Python, using the official async gRPC client (kurrentdbclient on PyPI). Zero changes to DeerFlow core; config.yaml just points at the class. Shipped as an optional extra so the default install footprint is untouched.
• Event-sourced semantics: each memory update is appended as an immutable event to a per-user stream; current state is a fold over the stream. That's what buys durable/auditable memory writes, point-in-time replay ("what did the agent know at turn N"), and cross-agent handoffs with full history.
• A docker-compose overlay adding a single-node KurrentDB in dev mode, so you can evaluate with one command — no accounts or external services.
• Tests + docs per CONTRIBUTING (TDD, AGENTS.md/README updated in the same changeset).

I'll open a draft PR from a fork so you can react to things early. If the memory work you're planning has specific interface needs, just let me know

[realtonyyoung]

Hi @WillemJiang — the prototype is ready and we've been running it internally: #3910 (draft). Would love for you to take it for a spin.

On your integration-protocol question: it turned out DeerFlow already had the right seams, so there's no new protocol to discuss. Here's what the PR contains and why each piece matters from the maintainer's side:

1. Event-sourced memory backend (KurrentdbMemoryStorage) — the thing we originally discussed. Loads through your existing memory.storage_class reflection hook with zero core changes; misconfiguration falls back to FileMemoryStorage exactly like today. What you get: every memory update is an immutable event, so "what did the agent know at turn N" is a stream read, memory writes are auditable, and a transient failure can never silently clobber a user's memory (failed reads raise rather than masquerading as empty — the one deliberate divergence from the file backend, reasoning in the PR).

2. Event-sourced run-event backend (KurrentRunEventStore, run_events.backend: kurrent) — a fourth backend for your existing RunEventStore contract, same shape as the db/jsonl branches. What you get: per-thread message/trace history that's multi-process safe by construction (seq derives from stream revisions, id-correlated so concurrent writers can't misattribute), and run deletion as an appended redaction marker — an audit-friendly answer to regenerate flows. This is the only piece that touches core: ~10 lines (one Literal value + one factory branch), disclosed in the PR with an open question on whether you'd want config_version bumped.

3. Canonical cross-framework events — memory facts and conversation messages are additionally emitted in the kurrent-agents canonical schema (AgentMemory-*, AgentSession-* streams). Why you care: no lock-in in either direction — anything DeerFlow writes is readable by other agent frameworks speaking the schema, and DeerFlow could later read theirs. Best-effort by design: your streams stay authoritative; canonical emission failure can never affect a write.

4. Two demo layers showing the event log is a capability, not just storage: an example config entry for the KurrentDB MCP server so the agent can answer "which facts have you retained about me, and when?" from its own streams — and a worked server-side projection deriving observability (fact-retention over time) from events already written, no second telemetry pipeline. We also documented how Kurrent's agent skills would pair with your skills system — deliberately not bundled; we'd provide properly packaged versions if this moves forward.

Trying it is one command for the DB:

docker compose -f docker/docker-compose.kurrentdb.yaml up -d   # Admin UI on :2113
export KURRENTDB_CONNECTION_STRING="kurrentdb://localhost:2113?tls=false"
# config.yaml:  memory.storage_class: deerflow.community.kurrentdb.memory_storage.KurrentdbMemoryStorage
#               run_events.backend: kurrent
make dev

Chat for a minute, then open the Admin UI's Stream Browser and watch deerflow.memory, deerflow.runs, AgentMemory, and AgentSession fill up — every historical state one click away. Full walkthrough: backend/docs/KURRENTDB_MEMORY.md.

Engineering notes: 108 new unit tests (fake-client, no KurrentDB needed in CI), full suite 5301 passed / 0 failed, blocking-IO and boundary gates clean, dependencies behind an optional extra so the default install is untouched. Known prototype limitations are documented honestly in the walkthrough.

Feedback we'd value most: the stream-naming schemes, whether your planned memory work needs async or multi-agent sharing semantics from the interface, the config_version question, and whether community/ is the right home. Happy to reshape any of it.