A connection engine for philosophical texts.
This is not normal retrieval-augmented generation. Normal RAG answers a question by fetching the passages most similar to it — which surfaces the obvious, usually same-author neighbours. RhizomeDB does almost the opposite: it surfaces passages that are distant enough to be surprising yet resonant enough to be genuine — unexpected connections across authors and works, in the spirit of Deleuze & Guattari's rhizome (any point connectable to any other). A connection is only kept if it arises from the flow of the theory itself; forced links — those resting on shared vocabulary or a manufactured reading — are rejected.
The output is not an answer. It's an invitation to explore: a short essay that traces the lines of flight between passages, each one cited.
books/ ─► converted/*.md ─► chunk ─► embed (local, fastembed/ONNX) ─► index/
│
seed (theme | random passage | chunk id) ─────────────────────────────┘
│
▼ ① candidate generation — the rhizomatic geometry
│ cross-author only · skip the obvious nearest neighbours ·
│ keep the "resonance band" · MMR-diversify across books
▼ ② bridge-judging (Claude Opus 4.8)
│ genuine resonance, or forced? forced links are dropped
▼ ③ synthesis — weave survivors into a cited exploration
▼ ④ wander (optional) — follow a connection as the next seed
The geometry (steps ①) is the heart: similarity gives a thematic neighbourhood, but we skip the top matches (too obvious / near-duplicate), keep a band of related-but-distant passages, exclude the seed's own author, and use MMR to spread the picks across different books and conceptual angles. Then an LLM acts as the not-forced filter and the writer.
python3.12 -m venv .venv # needs Python ≥ 3.10 (PEP 604 unions at runtime);
.venv/bin/pip install -r requirements.txt # stock macOS python3 is 3.9 and won't importEmbeddings run fully locally (no API key, no network after the model downloads once). The judging + synthesis steps use the Claude API:
export ANTHROPIC_API_KEY=sk-ant-... # or `ant auth login`Without a key, explore still runs and prints the raw resonance band
(geometry only) — useful for seeing the mechanism.
.venv/bin/python -m rhizome.cli build # catalog -> chunk -> embedThis writes data/catalog.json, index/chunks.jsonl, and index/embeddings.npy.
Re-run after adding books to data/converted/ (drop new books in books/, run
scripts/convert_books.py, then build). Edit data/catalog.json to fix any
author/title for books the catalogue couldn't identify.
# seed from a theme or question
.venv/bin/python -m rhizome.cli explore --theme "the flight of the gods"
# seed from a random passage — "surprise me"
.venv/bin/python -m rhizome.cli explore --random
# seed from a specific passage
.venv/bin/python -m rhizome.cli explore --chunk being-and-truth#0042
# walk the rhizome: follow the strongest connection as the next seed
.venv/bin/python -m rhizome.cli wander --random --steps 3Useful flags: --candidates N (size of the band to judge), --seed N
(reproducible randomness).
One backend: a versioned FastAPI app (rhizome/api.py) that serves both the
reader-v2 anchoring spine and the whole exploration/panel surface — the live SSE
run (seed → resonance geometry → judging → synthesis), embedding compare, the
workflow source-view (pulled live via inspect.getsource, so it never drifts),
sessions, chat, and reading-rhythm behaviour. Everything above the transport
lives in rhizome/reader_service.py + rhizome/anchor.py; the routes are thin.
The UI is a Vite + React + TypeScript SPA in frontend/, talking to the API
under /api/v2. First augment an existing index with spine offsets and migrate
legacy quote annotations, then run the two dev servers:
.venv/bin/python -m rhizome.cli spine # one-time: add chunk offsets
.venv/bin/python -m rhizome.cli serve --reload # API on http://127.0.0.1:8010
# (docs at /docs)
cd frontend && npm install && npm run dev # UI on http://127.0.0.1:5174Tests are stdlib only on both sides — no pytest, no frontend test runner:
.venv/bin/python -m unittest discover -s tests
cd frontend && npm run typecheck && npm test # node --test, types stripped by Nodenpm run build emits frontend/dist/, which the API then serves directly at
/ (no second server in production). The retrieval backend stays on full
display, and quote resolution returns an orphan instead of guessing when a
match is weak or ambiguous — a wrong note is worse than a missing one.
It works in geometry-only mode without a key; set ANTHROPIC_API_KEY (or a
Gemini/Groq key) to light up judging, synthesis, chat, and the Plateau study map.
A book renders in its original format when its source file is present under
books/ (PDF via PDF.js, EPUB via epub.js — both vendored through npm,
no CDN), and always as Markdown off the spine otherwise. The format switch
lives in the reader bar; every renderer shares one selection toolbar, notes
rail, and resolver, so a highlight in any format anchors to the spine, records a
format-native locator (PDF {page, quads} · EPUB {cfi} · MD spine offsets),
and lands in the same annotations.jsonl.
The Spine toggle reveals the two-layer view (R6): the reading surface's right
rail becomes a navigable index of the book's chunks — the units the connection
engine actually reads — each with its id, character span, and preview. Click one
to jump to it in the native view; conversely /read/<book>?chunk=<id> opens the
book scrolled to that passage, so any chunk id anywhere is a shareable location.
A light/dark/auto theme toggle sits in the top bar. npm install copies PDF.js's
standard-font + cmap data into frontend/public/pdfjs/ (postinstall) so base-14
and non-Latin PDFs render locally.
The corpus source files live in R2 (books/ is gitignored). To exercise the
PDF/EPUB renderers on a fresh checkout without them, generate a couple of
faithful samples through the real pipeline:
PYTHONPATH=. .venv/bin/python scripts/make_sample_books.pyDrop a PDF / EPUB / MOBI onto the library (or POST /api/v2/books/upload). It
runs through the same convert → chunk → index pipeline (rhizome/ingest.py),
lands under books/uploads/ + data/converted/uploads/ (both gitignored,
runtime-only), and opens natively and immediately — a highlight anchors to
the spine right away. It joins the vector index (for explore) only when
embeddings are rebuilt; reading needs only the chunk index. MOBI is extracted to
EPUB (Calibre's ebook-convert if present, else the pure-Python mobi lib).
Annotations made elsewhere take on the same life as native ones — each importer
is a parser feeding the one quote resolver (rhizome/imports.py):
- Embedded PDF highlights —
Import ▾ → Embedded PDF highlightsreads the book's/Annots(Highlight/Underline/StrikeOut/Squiggly + popup notes), lifts the covered text and note, resolves against the spine, and repaints the mark at its original quads. Idempotent (re-import updates, never duplicates). - Markdown / Obsidian — paste notes;
==highlights==and>blockquotes (with an optional note line) each become a quote.
Every imported mark carries an origin tag (import-pdf / import-md, R12). A
quote that can't be anchored lands in the orphan queue rather than being
dropped (R11): pin it to a passage — candidates are suggested by word overlap —
or dismiss it.
The reader's Connections rail is not one retrieval geometry but eleven, behind
one contract (rhizome/engines/, see docs/ENGINES.md). Every engine answers
the same question — from this passage, what else in the corpus? — and returns
the same shape, so you can flip between them on a passage and feel the difference
between ordinary RAG and constellatory retrieval:
| key | what it does |
|---|---|
band |
today's resonance band (skip the obvious, keep the related-but-distant, MMR) — the reference |
spread |
the same band, chosen by set-level diversity (DPP / facility-location) instead of pairwise MMR |
plain |
nearest neighbours, no exclusions — ordinary RAG, the baseline |
lexical / hybrid |
BM25 keywords, and dense+BM25 reciprocal-rank fusion — the SOLID / lookup end |
echo |
echoes within the same book, far from where you are |
concept |
personalised PageRank over the concept graph — the graph as a retrieval path |
walk |
a line of flight: the strongest step into a new book, then from there, hop by hop |
marks |
your own highlights and notes as the index — what you once wrote, recalled |
structural |
a second, persisted "structural" vector per passage (HyDE at build time; needs an LLM key to build) |
fused |
union of the constellatory paths, re-ranked by relevance × surprise × provenance, then set-spread |
Every pick discloses its why, its surface similarity and its rank in the full
corpus sort ("#160 of 4110" — proof it was a mid-band resonance, not a top hit).
A calibrated noise floor makes every constellatory engine return nothing
when the corpus holds nothing that resonates with the seed (plain RAG deliberately
never does — that is the contrast). ?engine=<key> on a reader URL, the picker in
the rail, and Compare (every engine's top three for the same seed) expose them;
Connect on a text selection seeds them from a sentence instead of a passage.
.venv/bin/python -m rhizome.cli engines # what is ready
.venv/bin/python -m rhizome.cli connect --engine walk --chunk being-and-truth#0036
.venv/bin/python -m rhizome.cli compare-engines --chunk being-and-truth#0036
.venv/bin/python -m rhizome.cli eval-engines # the constellatory harnessThe harness (rhizome/eval_engines.py, served at /api/v2/engines/eval) scores
each engine on the PRD §6c proxies — mean corpus rank of its picks, intra-list
diversity, book spread, and the share of incoherent seeds it still answers.
All the geometry knobs live in rhizome/config.py:
| Knob | Meaning |
|---|---|
SKIP_TOP |
how many of the most-similar (obvious) matches to drop |
POOL |
size of the resonance band MMR draws from |
N_CANDIDATES |
how many connections to propose per seed |
MMR_LAMBDA |
< 0.5 favours diversity across books over closeness to seed |
MIN_SIM |
floor below which candidates are noise, not resonance |
EXCLUDE_SAME_AUTHOR |
never connect an author to themselves |
If connections feel too obvious, raise SKIP_TOP or lower MMR_LAMBDA. If they
feel like noise, raise MIN_SIM.
README.md this file
requirements.txt Python dependencies
books/ original PDFs / EPUBs / MOBIs (untouched, gitignored)
data/ corpus + metadata + human inputs
catalog.json author / title / year per book
converted/ Markdown, one file per book ← see scripts/convert_books.py
notes/ annotated reading notes (the human half of the loop)
eval/ in-domain embedding gold set
index/ generated: chunks.jsonl + embeddings.npy (gitignored)
build/ generated: self-contained HTML maps (gitignored)
scripts/ convert_books.py · upload_to_r2.py (one-off pipeline)
docs/ CHUNKING · ENGINES · FORMATS · OPERATING · ROADMAP · SCHEMA · VISION · PRDs · audits/
frontend/ Vite + React + TS SPA (the reader + panel UI)
tools/ chunkmap · conceptmap · docgraph · panel generators
rhizome/ the engine + the backend
config.py paths + tunable geometry
catalog.py corpus metadata
chunk.py / chunking.py Markdown → passages (+ multi-resolution levels) + spine offsets
embed.py local ONNX embeddings (multi-model)
concepts.py core-concept extraction (content lens)
store.py in-memory index + rhizomatic retrieval geometry
engines/ the retrieval mini-engines behind one contract (docs/ENGINES.md)
eval_engines.py the constellatory eval harness
llm.py Claude judging (structured) + synthesis
engine.py seed → candidates → judge → synthesize → wander
anchor.py durable quote → spine resolver (W3C selectors, orphan-safe)
workspace.py annotations / chats / sessions persistence
reader_service.py domain logic behind the HTTP boundary
api.py FastAPI app (the one backend)
cli.py command line