v1.1.0 — quality & robustness pass from systematic testing

Conversion: OCR via tesseract stdin/stdout (fixes silent vision fallthrough under
sparse/sandboxed PATH), scanned-PDF OCR via pypdfium2, PATH self-heal for Desktop.
Digest: accumulative digestion + working reset; hard-split oversize chunks (no
silent loss); dedupe + low-value-chunk skip + reported chunk cap (fixes hang on
pathological corpora); memory-aware extraction concurrency. Quality: fixed
entity-resolution over-merge (embeddings now only confirm token-sharing merges),
nomic search_document/search_query prefixes for sharper recall. Lifecycle: idle
floor lowered to 5s so MTA_IDLE is honest. 11 regression tests, all green.

Author: GRU-953

.claude-plugin/marketplace.json

@@ -6,15 +6,15 @@
   },
   "metadata": {
     "description": "Local, token-free file digestion → knowledge-graph memory for Claude.",
-    "version": "1.0.1",
+    "version": "1.1.0",
     "pluginRoot": "."
   },
   "plugins": [
     {
       "name": "memorised-them-all",
       "source": "./",
       "description": "Convert any attachment to Markdown locally, then digest it into a token-free knowledge graph + exportable memory + offline mind map. 100% local & open-source; auto-installing; Apple-silicon first.",
-      "version": "1.0.1",
+      "version": "1.1.0",
       "license": "MIT",
       "homepage": "https://github.com/GRU-953/memorised-them-all",
       "keywords": ["memory", "knowledge-graph", "mind-map", "markitdown", "ocr",

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "memorised-them-all",
-  "version": "1.0.1",
+  "version": "1.1.0",
   "description": "Locally convert any attachment to Markdown and digest it into a token-free knowledge-graph memory for Claude — global synopsis, per-theme summaries, exportable Markdown notes, and an offline interactive mind map. 100% local, free & open-source, auto-installing, Apple-silicon first.",
   "author": {
     "name": "Aninda Sundar Howlader",

CHANGELOG.md

@@ -4,6 +4,42 @@ All notable changes to **Memorised them All** are documented here. This project
 adheres to [Semantic Versioning](https://semver.org/) and
 [Keep a Changelog](https://keepachangelog.com/).
 
+## [1.1.0] — 2026-06-01
+
+### Added
+- **Accumulative digestion**: digesting another folder into the same project now
+  *extends* the memory (rebuilt from the full converted corpus on disk) instead
+  of replacing it. `reset=True` clears the project first.
+- **Scanned-PDF OCR**: image-only PDFs are rasterised with pypdfium2 and OCR'd
+  page-by-page (previously a no-op).
+- **Parallel extraction**: knowledge extraction runs across a memory-aware thread
+  pool (`MTA_EXTRACT_WORKERS`, default auto: 1–3 by RAM) — the LLM calls are
+  I/O-bound, but too much concurrency thrashes a unified-memory Mac running a 7B
+  model, so the default scales with available memory.
+- **Workload guards**: identical chunks are de-duplicated, degenerate low-
+  information passages are skipped, and a reported cap (`MTA_MAX_CHUNKS`, default
+  1500) prevents pathological corpora from hanging — all surfaced in stats, never
+  silent.
+- **PATH self-heal**: the server/CLI prepend common Homebrew/system bin dirs so
+  `tesseract`/`ffmpeg`/`ollama` resolve under a host app's sparse PATH (Claude
+  Desktop).
+
+### Fixed
+- **Entity resolution no longer over-merges**: short proper nouns embed almost
+  identically with nomic-embed-text (e.g. two unrelated organisations can score
+  cosine ≈ 1.0), which previously collapsed distinct entities into a single node.
+  Embeddings now only *confirm* a merge that also shares tokens (fuzzy floor), and
+  the cosine threshold was raised. Distinct entities stay distinct.
+- **Better recall ranking**: prefix-aware embedding models (nomic) now receive
+  their `search_document:` / `search_query:` task prefixes, sharpening retrieval.
+- **Honest idle timeout**: the on-demand Ollama idle-stop floor was lowered from
+  30 s to 5 s so small `MTA_IDLE` values are respected (default remains 300 s).
+- **OCR robustness**: image OCR now pipes PNG bytes to `tesseract stdin stdout`
+  (PSM 1 auto-orientation) instead of using temp files, which failed under
+  sandboxed/sparse environments and silently fell through to vision captioning.
+- **No silent content loss**: an over-long unpunctuated passage is hard-split into
+  chunk-sized windows, so the extractor's input cap no longer drops the tail.
+
 ## [1.0.1] — 2026-06-01
 
 ### Fixed
@@ -45,5 +81,6 @@ The first public release.
 - **Distribution**: Claude Desktop `.mcpb`, Claude Code plugin/marketplace, PyPI
   package, and a Homebrew tap; CI and tagged releases with assets.
 
+[1.1.0]: https://github.com/GRU-953/memorised-them-all/releases/tag/v1.1.0
 [1.0.1]: https://github.com/GRU-953/memorised-them-all/releases/tag/v1.0.1
 [1.0.0]: https://github.com/GRU-953/memorised-them-all/releases/tag/v1.0.0

README.md

@@ -161,7 +161,7 @@ mta update            # pull the latest MarkItDown + dependencies
 
 | Tool | What it does | Returns |
 | --- | --- | --- |
-| `digest(paths, project?, reset?)` | convert + digest files/dirs/globs | counts, paths, graph stats |
+| `digest(paths, project?, reset?)` | convert + digest files/dirs/globs (**accumulates** into the project; `reset=true` starts fresh) | counts, paths, graph stats |
 | `recall(query, project?, k?)` | answer from memory | a small, citable slice |
 | `memory_overview(project?)` | synopsis + themes | compact overview |
 | `export_memory(dest, project?)` | export portable Markdown | files written |
@@ -187,6 +187,8 @@ settings (Desktop).
 | `MTA_WHISPER_MODEL` | `base` | on-device transcription model |
 | `MTA_IDLE` | `300` | seconds of idle before Ollama is stopped |
 | `MTA_WORKERS` | `0` (auto) | parallel conversion workers |
+| `MTA_EXTRACT_WORKERS` | `0` (auto) | parallel extraction workers (memory-aware: 1–3 by RAM) |
+| `MTA_MAX_CHUNKS` | `1500` | safety cap on chunks per digest (truncation is reported) |
 | `MTA_COMMUNITY_ALGO` | `auto` | `leiden` · `louvain` · `greedy` |
 | `MTA_AUTO_UPDATE` | `on` | auto-update MarkItDown & dependencies |
 | `MTA_NO_OLLAMA` | unset | hard offline switch (classical + hashing) |

manifest.json

@@ -2,7 +2,7 @@
   "manifest_version": "0.3",
   "name": "memorised-them-all",
   "display_name": "Memorised them All (local, token-free)",
-  "version": "1.0.1",
+  "version": "1.1.0",
   "description": "Convert any attachment to Markdown and digest it into a token-free knowledge-graph memory — locally.",
   "long_description": "Memorised them All turns a folder of documents into durable, **token-free** memory for Claude, entirely on your machine. It converts every attachment (PDF, Word, Excel, PowerPoint, HTML, EPub, images, audio, and more) to Markdown locally using Microsoft's MarkItDown (kept up to date from upstream automatically), Tesseract OCR, on-device Whisper, and a local Ollama vision model. It then runs an original local pipeline — semantic chunking, local embeddings, local-LLM (with a classical fallback) extraction of entities/relations/facts, embedding+fuzzy entity resolution, and community detection — to build a layered knowledge graph: a global synopsis, per-theme summaries, per-document Markdown notes, an exportable memory bundle, and an offline interactive mind map. Every tool returns only compact metadata or a tiny relevant slice, never document contents, so digesting and recalling whole folders costs ~0 Claude context tokens. Everything is free and open-source and runs locally; the Ollama model server starts on demand and stops after 5 minutes idle. Tuned for Apple M-series silicon (performance-core parallelism, GPU Whisper via MLX, unified-memory-aware concurrency).\n\nTools: digest, recall, memory_overview, export_memory, list_digestible, memory_status, open_mindmap.",
   "author": {

mta/__init__.py

@@ -3,5 +3,5 @@
 Clean-room implementation by GRU-953 (Aninda Sundar Howlader).
 """
 
-__version__ = "1.0.1"
+__version__ = "1.1.0"
 __author__ = "Aninda Sundar Howlader"

mta/cli.py

@@ -57,6 +57,8 @@ def main(argv: list[str] | None = None) -> int:
     sub.add_parser("serve", help="run the MCP server (stdio)")
 
     args = p.parse_args(argv)
+    from .core.platform import bootstrap_path
+    bootstrap_path()
     cfg = load_config().with_project(args.project)
 
     if args.cmd == "digest":

mta/core/config.py

@@ -53,6 +53,8 @@ class Config:
     community_algo: str = field(default_factory=lambda: _env("MTA_COMMUNITY_ALGO", "auto"))  # auto|leiden|louvain|greedy
     chunk_chars: int = field(default_factory=lambda: _env_int("MTA_CHUNK_CHARS", 1200))
     recall_k: int = field(default_factory=lambda: _env_int("MTA_RECALL_K", 8))
+    max_chunks: int = field(default_factory=lambda: _env_int("MTA_MAX_CHUNKS", 1500))
+    extract_workers: int = field(default_factory=lambda: _env_int("MTA_EXTRACT_WORKERS", 0))  # 0=auto
 
     # Lifecycle & maintenance.
     idle_seconds: int = field(default_factory=lambda: _env_int("MTA_IDLE", 300))

mta/core/convert.py

@@ -65,21 +65,76 @@ def _try_markitdown(path: Path, cfg: Config) -> tuple[str | None, str]:
         return None, f"markitdown-error:{type(e).__name__}"
 
 
-def _try_ocr(path: Path, cfg: Config) -> tuple[str | None, str]:
+def _tesseract_bin() -> str | None:
+    import shutil
+    from .platform import bootstrap_path
+    bootstrap_path()
+    return shutil.which("tesseract")
+
+
+def _ocr_image_bytes(png_bytes: bytes, cfg: Config) -> str | None:
+    """OCR via piping PNG bytes to `tesseract stdin stdout` — no temp files.
+
+    The temp-file path pytesseract uses by default breaks under sandboxed/sparse
+    environments; piping is robust everywhere and supports PSM 1 auto-orientation.
+    """
+    import subprocess
+    tess = _tesseract_bin()
+    if not tess:
+        return None
+    lang = cfg.ocr_lang or "eng"
     try:
-        import pytesseract
-        from PIL import Image
-    except Exception:
+        proc = subprocess.run([tess, "stdin", "stdout", "-l", lang, "--psm", "1"],
+                              input=png_bytes, capture_output=True, timeout=180)
+        return (proc.stdout.decode("utf-8", "replace").strip() or None)
+    except (OSError, subprocess.SubprocessError):
+        return None
+
+
+def _try_ocr(path: Path, cfg: Config) -> tuple[str | None, str]:
+    if not _tesseract_bin():
         return None, "ocr-missing"
     try:
+        import io
+
+        from PIL import Image
         with Image.open(path) as im:
-            text = pytesseract.image_to_string(im, lang=cfg.ocr_lang or "eng")
-        text = (text or "").strip()
-        return (text or None), "tesseract"
+            buf = io.BytesIO()
+            im.convert("RGB").save(buf, format="PNG")
+        text = _ocr_image_bytes(buf.getvalue(), cfg)
+        return text, "tesseract"
     except Exception as e:  # noqa: BLE001
         return None, f"ocr-error:{type(e).__name__}"
 
 
+def _ocr_pdf(path: Path, cfg: Config, max_pages: int = 50) -> tuple[str | None, str]:
+    """OCR a scanned/image-only PDF by rasterising pages with pypdfium2."""
+    if not _tesseract_bin():
+        return None, "ocr-missing"
+    try:
+        import io
+
+        import pypdfium2 as pdfium
+    except Exception:
+        return None, "pdf-ocr-missing"
+    try:
+        pdf = pdfium.PdfDocument(str(path))
+        pages = []
+        for i in range(min(len(pdf), max_pages)):
+            bitmap = pdf[i].render(scale=2.0)
+            pil = bitmap.to_pil()
+            buf = io.BytesIO()
+            pil.convert("RGB").save(buf, format="PNG")
+            txt = _ocr_image_bytes(buf.getvalue(), cfg)
+            if txt:
+                pages.append(txt)
+        pdf.close()
+        joined = "\n\n".join(pages).strip()
+        return (joined or None), "tesseract-pdf"
+    except Exception as e:  # noqa: BLE001
+        return None, f"pdf-ocr-error:{type(e).__name__}"
+
+
 def _try_vision(path: Path, cfg: Config, ollama: OllamaManager) -> tuple[str | None, str]:
     if cfg.vision_mode == "off":
         return None, "vision-off"
@@ -165,8 +220,8 @@ def convert_file(path: Path, out_dir: Path, cfg: Config,
         text, method = _native_text(path)
     elif ext in _MARKITDOWN_EXTS:
         text, method = _try_markitdown(path, cfg)
-        if not text and ext == ".pdf":      # scanned PDF → OCR
-            text, method = _try_ocr(path, cfg)
+        if not text and ext == ".pdf" and cfg.ocr_mode != "off":  # scanned PDF → OCR
+            text, method = _ocr_pdf(path, cfg)
     elif ext in _IMAGE_EXTS:
         if cfg.ocr_mode != "off":
             text, method = _try_ocr(path, cfg)

mta/core/digest.py

@@ -119,24 +119,58 @@ def digest(cfg: Config, paths: list[str], reset: bool = False,
     ollama = ollama or OllamaManager(cfg)
     t0 = time.time()
 
+    if reset:
+        _reset_project(cfg)
+        cfg.ensure_dirs()
+
     files = _expand(paths)
     if not files:
         return {"status": "no_input", "project": cfg.project,
                 "message": "No convertible files found.", "paths": paths}
 
     conv = _convert_all(files, cfg, ollama)
-    ok_md = [Path(c["output"]) for c in conv if c.get("status") == "ok" and c.get("output")]
+
+    # Accumulative: rebuild the graph from the FULL markdown corpus on disk, so
+    # digesting another folder into the same project extends the memory rather
+    # than replacing it. `reset=True` clears the corpus first (above).
+    all_md = sorted(cfg.markdown_dir.glob("*.md"))
 
     # Segment + extract.
     embedder = Embedder(cfg, ollama)
     all_chunks = []
-    for md in ok_md:
+    for md in all_md:
         all_chunks.extend(segment_file(md, cfg.chunk_chars))
 
+    # Dedupe identical chunks, drop degenerate low-information passages (repetitive
+    # filler whose boundary-shifted windows defeat exact dedupe), then cap the
+    # workload with explicit reporting (never silently truncate).
+    unique: dict[str, object] = {}
+    skipped_low_value = 0
+    for ch in all_chunks:
+        if _low_value(ch.text):
+            skipped_low_value += 1
+            continue
+        unique.setdefault(ch.text, ch)
+    unique_chunks = list(unique.values())
+    truncated = 0
+    if len(unique_chunks) > cfg.max_chunks:
+        truncated = len(unique_chunks) - cfg.max_chunks
+        unique_chunks = unique_chunks[:cfg.max_chunks]
+
+    # Extract across a modest, memory-aware pool — LLM calls are I/O-bound (HTTP
+    # to Ollama); too much concurrency thrashes a unified-memory Mac running a 7B
+    # model, so the default scales with RAM.
     extractions: list[tuple] = []
     mentions: list[dict] = []
-    for ch in all_chunks:
-        ex = extract_chunk(ch, cfg, ollama)
+    workers = max(1, min(_auto_extract_workers(cfg), len(unique_chunks))) if unique_chunks else 1
+    if workers > 1:
+        from concurrent.futures import ThreadPoolExecutor
+        with ThreadPoolExecutor(max_workers=workers) as tp:
+            results = list(tp.map(lambda c: (c, extract_chunk(c, cfg, ollama)),
+                                  unique_chunks))
+    else:
+        results = [(c, extract_chunk(c, cfg, ollama)) for c in unique_chunks]
+    for ch, ex in results:
         extractions.append((ch, ex))
         mentions.extend(ex.entities)
 
@@ -178,13 +212,14 @@ def digest(cfg: Config, paths: list[str], reset: bool = False,
         "edges": [{"source": u, "target": v, "weight": d["weight"],
                    "labels": sorted(d.get("labels", []))} for u, v, d in G.edges(data=True)],
         "communities": communities,
-        "documents": [{"name": c.get("source"), "output": c.get("output"),
-                       "status": c.get("status"), "method": c.get("method", ""),
-                       "chars": c.get("chars", 0)} for c in conv],
+        "documents": _documents(cfg, all_md, conv),
         "stats": {
             "files": len(files),
-            "converted": len(ok_md),
+            "converted": len(all_md),
             "chunks": len(all_chunks),
+            "unique_chunks": len(unique_chunks),
+            "chunks_truncated": truncated,
+            "chunks_skipped_low_value": skipped_low_value,
             "entities": G.number_of_nodes(),
             "relations": G.number_of_edges(),
             "communities": len(communities),
@@ -250,6 +285,78 @@ def _recall_units(graph_doc: dict) -> tuple[list[dict], list[str]]:
     return units, texts
 
 
+def _low_value(text: str) -> bool:
+    """True for degenerate, near-zero-information passages (repetitive filler).
+
+    Real prose has high lexical diversity; a window of one word repeated hundreds
+    of times does not and is not worth an LLM call.
+    """
+    words = text.split()
+    if len(words) < 40:
+        return False
+    uniq = len(set(w.lower() for w in words))
+    return (uniq / len(words)) < 0.12
+
+
+def _auto_extract_workers(cfg: Config) -> int:
+    if cfg.extract_workers > 0:
+        return cfg.extract_workers
+    from .platform import memory_gb
+    gb = memory_gb()
+    # Conservative on unified-memory Macs running a 7B extractor.
+    return 1 if gb < 16 else (2 if gb < 48 else 3)
+
+
+def _reset_project(cfg: Config) -> None:
+    """Wipe a project's converted corpus and derived memory (for reset=True)."""
+    import shutil
+    for path in (cfg.markdown_dir, cfg.memory_dir):
+        shutil.rmtree(path, ignore_errors=True)
+    for f in (cfg.graph_path, cfg.vectors_path,
+              cfg.vectors_path.with_suffix(".json"), cfg.memory_md, cfg.mindmap_html):
+        try:
+            f.unlink()
+        except OSError:
+            pass
+
+
+def _parse_md_header(md: Path) -> tuple[str, str]:
+    """Recover (source name, method) from the provenance comment we write."""
+    try:
+        first = md.read_text(encoding="utf-8", errors="replace").splitlines()[0]
+    except (OSError, IndexError):
+        first = ""
+    src, method = md.name[:-3] if md.name.endswith(".md") else md.name, ""
+    if first.startswith("<!-- source:"):
+        body = first.strip("<!-> ")
+        for part in body.split("·"):
+            part = part.strip()
+            if part.startswith("source:"):
+                src = part[len("source:"):].strip()
+            elif part.startswith("method:"):
+                method = part[len("method:"):].strip()
+    return src, method
+
+
+def _documents(cfg: Config, all_md: list[Path], conv: list[dict]) -> list[dict]:
+    """Document manifest across the FULL corpus, plus this call's non-ok files."""
+    docs = []
+    for md in all_md:
+        src, method = _parse_md_header(md)
+        try:
+            chars = max(0, len(md.read_text(encoding="utf-8", errors="replace")) - 60)
+        except OSError:
+            chars = 0
+        docs.append({"name": src, "output": str(md), "status": "ok",
+                     "method": method, "chars": chars})
+    for c in conv:  # surface files that failed/were unsupported this run
+        if c.get("status") != "ok":
+            docs.append({"name": Path(c.get("source", "")).name, "output": None,
+                         "status": c.get("status"), "method": c.get("method", ""),
+                         "chars": 0})
+    return docs
+
+
 def _conv_tally(conv: list[dict]) -> dict:
     tally: dict[str, int] = {}
     for c in conv: