Skip to content

About

AI-powered second brain with agentic synthesis, hybrid RAG retrieval, and LLM-as-judge evaluation harnesses

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Distill

An AI-powered second brain that distills raw notes, voice memos, PDFs, web clips, and YouTube videos into a living, self-organizing knowledge wiki. Agentic multi-pass synthesis with a tool-calling loop, hybrid RAG chat over your sources, and an LLM-as-judge eval harness across four domains.

Distill — theme board view

The Problem

Knowledge is scattered across highlights, voice memos, articles, and documents. Traditional note-taking puts the burden on you to organize, connect, and maintain it. As volume grows, the system collapses under its own weight. Notes go stale, connections get missed, and retrieval becomes a search problem.

What It Does

Throw unstructured material into a collection and an AI agent synthesizes it into thematic wiki pages that evolve with every new source. Then chat with your knowledge base. Answers are grounded in your actual sources with a citation trail.

Capture: Five input mediums, one pipeline:

Medium How
Markdown Paste or file upload
Voice Real-time STT via Deepgram nova-3
Web clip Paste a URL, server extracts to Markdown
PDF Client-side text extraction
YouTube Transcript via Gemini API

Synthesize: An agentic tool-calling loop reads the existing wiki, receives a batch of new entries, and uses five tools (readTheme, createTheme, editSection, rewriteTheme, deleteTheme) to fold new material into the right theme pages. Every write is traced with provenance back to source entries.

Chat: A read-only answer agent navigates the wiki with the same tools, backed by hybrid vector + keyword retrieval for grounded, cited answers.

Ask agent — cross-corpus chat with grounded responses

Lint: An autonomous "dream cycle" agent reads the entire wiki on a nightly cron, identifying merges, splits, contradictions, and broken cross-links. Automated wiki gardening.

How It Works

flowchart TD
    subgraph CAPTURE["Capture (5 mediums)"]
        MD["Markdown"] & VC["Voice"] & WC["Web clip"] & PDF["PDF"] & YT["YouTube"]
    end

    CAPTURE --> IS["ingestSource\nimmutable raw entry + SHA-256 dedupe"]
    IS --> RE[("raw_entries\n(append-only)")]
    RE --> CHK["chunk + embed\npgvector 1536-d"]

    RE -->|token threshold crossed| SA

    subgraph SA["Synthesis Agent (tool-calling loop)"]
        direction LR
        IDX["Wiki index\n+ new batch"] --> TOOLS["readTheme · createTheme\neditSection · rewriteTheme\ndeleteTheme"]
        TOOLS --> TS["ThemeStore port"]
    end

    SA --> TN[("topic_notes\n(living wiki)")]
    SA --> PROV["note_revisions\nentry_note_links\n(provenance)"]

    RE --> RAG["Hybrid retrieval\nvector + keyword → RRF fusion"]
    TN --> ANS["Answer Agent\n(read-only, forced grounding)"]
    RAG --> ANS

    classDef agent fill:#eef2ff,stroke:#6366f1,color:#3730a3;
    classDef store fill:#f0fdf4,stroke:#16a34a,color:#15803d;
    classDef capture fill:#fefce8,stroke:#ca8a04,color:#854d0e;
    class SA,ANS agent;
    class RE,TN,PROV store;
    class CAPTURE,MD,VC,WC,PDF,YT capture;
Loading

Key Design Decisions

  • Port/adapter pattern: The synthesis agent operates through a ThemeStore interface and never sees the database. Production uses PgThemeStore (buffered atomic writes with revisions and provenance). Evals use MemoryThemeStore (same contract, in-memory). No mocking frameworks needed.

  • Immutable sources, derived everything: Raw entries are append-only. Topic notes, revisions, and provenance links are all derived state that can be replayed from source. Makes the system auditable and eval-friendly.

  • Batched, threshold-gated synthesis: Capture and synthesis are decoupled. ingestSource stores the entry immediately; a background trigger fires synthesis once accumulated tokens cross a threshold. planChunks splits large backlogs into digestible batches.

  • Forced grounding: The answer agent must call readTheme or searchRaw before it writes a word. This guarantees a citation trail and prevents hallucinated answers.

  • Evals over integration tests: LLM behavior can't be pinned by deterministic tests. Each agent has its own eval suite with LLM-as-judge scoring and calibrated rubrics. Pure logic gets traditional unit tests.

Evaluation Harnesses

Four domain-specific eval suites under evals/, each measuring a different aspect of the system. All share a common harness (evals/lib/): CLI tooling, run-directory persistence, a provider-agnostic LLM judge seam, and Langfuse trace export.

Domain What it measures Approach Fixtures included
Synthesis Prose quality + source grounding LLM judges on 4 criteria (synthesis depth, coherence, connectedness, density) + faithfulness judge + deterministic provenance checks. Run comparison across prompt/model variants. Harness code only. Raw entry corpus excluded.
Answer Chat response quality Question taxonomy (answerable, verbatim, provenance, confabulation-bait, off-wiki refusal). Separate quality + faithfulness judges. 12 model configs for cross-model comparison. Question sets included (questions.json) showing the taxonomy design.
RAG Retrieval accuracy P@k, R@k, MRR, nDCG@k over hand-labeled relevance judgments. Pre-registered hypotheses across config variants (vector-only, keyword-only, hybrid). Hand-labeled qrels included (qrels.jsonl). Documented sweep results in RESULTS.md.
Lint Reorg quality (WIP) Alignment judge (preservation vs. improvement), structural validation (broken links, orphans). Harness code only. Fixtures excluded (lint agent is still in development).

Stack

Layer Technology
Framework Next.js 16 (App Router)
Language TypeScript
Styling Tailwind CSS v4
Database Neon Postgres + Drizzle ORM
Vector search pgvector (1536-d, cosine distance)
LLM orchestration Vercel AI SDK + OpenRouter
Speech-to-text Deepgram nova-3
Auth Clerk (per-user isolation)
Hosting Vercel
Eval telemetry Langfuse

Local Development

cp .env.example .env.local
# Fill in: DATABASE_URL, OPENROUTER_API_KEY, DEEPGRAM_API_KEY,
#          NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY, CLERK_SECRET_KEY

npx drizzle-kit push          # Push schema to Neon
npm run rag:db-setup          # Create pgvector tables + indexes
npm run dev                   # Start dev server

License

MIT. See LICENSE.

About

AI-powered second brain with agentic synthesis, hybrid RAG retrieval, and LLM-as-judge evaluation harnesses

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages