Merge pull request #1 from GRU-953/v1.2-hardening

v1.2.0 — fast mode, multi-agent hardening, cross-platform

Author: GRU-953

.claude-plugin/marketplace.json

@@ -6,15 +6,15 @@
   },
   "metadata": {
     "description": "Local, token-free file digestion → knowledge-graph memory for Claude.",
-    "version": "1.1.0",
+    "version": "1.2.0",
     "pluginRoot": "."
   },
   "plugins": [
     {
       "name": "memorised-them-all",
       "source": "./",
       "description": "Convert any attachment to Markdown locally, then digest it into a token-free knowledge graph + exportable memory + offline mind map. 100% local & open-source; auto-installing; Apple-silicon first.",
-      "version": "1.1.0",
+      "version": "1.2.0",
       "license": "MIT",
       "homepage": "https://github.com/GRU-953/memorised-them-all",
       "keywords": ["memory", "knowledge-graph", "mind-map", "markitdown", "ocr",

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "memorised-them-all",
-  "version": "1.1.0",
+  "version": "1.2.0",
   "description": "Locally convert any attachment to Markdown and digest it into a token-free knowledge-graph memory for Claude — global synopsis, per-theme summaries, exportable Markdown notes, and an offline interactive mind map. 100% local, free & open-source, auto-installing, Apple-silicon first.",
   "author": {
     "name": "Aninda Sundar Howlader",

.github/workflows/ci.yml

@@ -12,10 +12,11 @@ permissions:
 
 jobs:
   test:
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
         python-version: ["3.10", "3.12"]
     env:
       # Fully offline: classical extraction + hashing embeddings. No models, no network.
@@ -33,7 +34,7 @@ jobs:
       - name: Install (package + test deps only — heavy converters are not needed offline)
         run: |
           python -m pip install --upgrade pip
-          pip install numpy networkx rapidfuzz "mcp>=1.2" pytest
+          pip install numpy networkx rapidfuzz "mcp>=1.2" psutil pytest
           pip install -e . --no-deps
 
       - name: Run offline smoke + e2e tests

CHANGELOG.md

@@ -4,6 +4,38 @@ All notable changes to **Memorised them All** are documented here. This project
 adheres to [Semantic Versioning](https://semver.org/) and
 [Keep a Changelog](https://keepachangelog.com/).
 
+## [1.2.0] — 2026-06-01
+
+Fast mode + a multi-agent evaluation/hardening pass (accuracy, reliability,
+token-safety, reusability, cross-platform) and security review.
+
+### Added
+- **Fast mode** (`MTA_FAST=on`, `mta digest --fast`, `fast=true` tool arg): skips
+  the LLM (classical extraction + deterministic summaries, keeps the embedding
+  model) for a fully **deterministic**, ~100× faster digest. The default path
+  stays on the LLM for maximum accuracy.
+- **Cross-platform support beyond Apple silicon**: psutil-based physical-core and
+  memory detection (correct pool sizing on Intel/Linux/Windows), platform-aware
+  PATH healing, portable mind-map opener, `psutil` process-tree teardown for the
+  idle-stop, CUDA Whisper on Linux/Windows GPUs, Linux package-manager install
+  paths (apt/dnf/pacman), broadened platform metadata, and a CI matrix across
+  Ubuntu/macOS/Windows × Python 3.10/3.12.
+- **Per-file size cap** (`MTA_MAX_FILE_MB`, default 200) — oversize files are
+  skipped before being read into memory (bounds OOM/decompression-bomb risk).
+
+### Fixed / Hardened (from agent reviews)
+- **Token-safety**: `recall(k=…)` is hard-clamped (≤50) so a caller can't pull the
+  whole graph's text into context; LLM fact strings are length-capped.
+- **Accuracy**: facts attach to entities by **word boundary** (no "Cat" inside
+  "Category"; CJK-aware), de-duplicated per chunk; stable community tiebreak.
+- **Reliability**: per-file isolation for the conversion process pool and
+  per-chunk isolation for threaded extraction (one failure or a mid-run model
+  death no longer aborts/redoes the whole digest); PDF OCR handle leak fixed;
+  updater throttle stamped before work to avoid concurrent pip races.
+- **Reusability**: `graph.json` stores basenames (no absolute-path leakage —
+  portable across machines); exports now include the vector store so recall works
+  from a copied bundle; `load_graph` rejects an incompatible future schema.
+
 ## [1.1.0] — 2026-06-01
 
 ### Added
@@ -81,6 +113,7 @@ The first public release.
 - **Distribution**: Claude Desktop `.mcpb`, Claude Code plugin/marketplace, PyPI
   package, and a Homebrew tap; CI and tagged releases with assets.
 
+[1.2.0]: https://github.com/GRU-953/memorised-them-all/releases/tag/v1.2.0
 [1.1.0]: https://github.com/GRU-953/memorised-them-all/releases/tag/v1.1.0
 [1.0.1]: https://github.com/GRU-953/memorised-them-all/releases/tag/v1.0.1
 [1.0.0]: https://github.com/GRU-953/memorised-them-all/releases/tag/v1.0.0

README.md

@@ -189,10 +189,18 @@ settings (Desktop).
 | `MTA_WORKERS` | `0` (auto) | parallel conversion workers |
 | `MTA_EXTRACT_WORKERS` | `0` (auto) | parallel extraction workers (memory-aware: 1–3 by RAM) |
 | `MTA_MAX_CHUNKS` | `1500` | safety cap on chunks per digest (truncation is reported) |
+| `MTA_MAX_FILE_MB` | `200` | skip files larger than this before reading (0 disables) |
 | `MTA_COMMUNITY_ALGO` | `auto` | `leiden` · `louvain` · `greedy` |
 | `MTA_AUTO_UPDATE` | `on` | auto-update MarkItDown & dependencies |
+| `MTA_FAST` | `off` | fast mode — skip the LLM (classical extraction, deterministic, keeps embeddings) |
 | `MTA_NO_OLLAMA` | unset | hard offline switch (classical + hashing) |
 
+> **Accuracy vs speed.** The default path uses the local LLM for the highest
+> extraction accuracy. **Fast mode** (`MTA_FAST=on`, `mta digest --fast`, or the
+> `fast=true` tool arg) skips the LLM for a fully **deterministic**, ~100× faster
+> digest that still builds the graph and keeps semantic recall — ideal for large
+> or frequently-updated corpora.
+
 ## Apple silicon first
 
 - Conversion fans out across **performance cores** (`hw.perflevel0.physicalcpu`),
@@ -227,6 +235,52 @@ and left alone. Only an instance *this tool* starts is stopped on idle.
 **Where are my files?** Under `MTA_HOME/projects/<project>/` — `graph.json`,
 `memory.md`, `memory/`, `mindmap.html`. `export_memory` copies them anywhere.
 
+## Modes & performance
+
+Two digest modes — the default favours **accuracy & consistency**, fast mode favours **speed & determinism**:
+
+| | Default (accurate) | Fast (`--fast` / `MTA_FAST=on`) |
+| --- | --- | --- |
+| Extraction | local LLM (qwen2.5) | classical (deterministic) |
+| Theme summaries | local LLM | deterministic fact-join |
+| Embeddings / recall | local (nomic) | local (nomic) |
+| Reproducible | per-model | **byte-identical across runs** |
+| Relative speed | baseline | **~100× faster** |
+| Best for | highest fidelity | large or frequently-refreshed corpora |
+
+Both are **token-free** and **fully local**. Digestion is incremental — pointing `digest` at another folder *extends* the same project; `reset=true` starts fresh. Degenerate/repetitive content is de-duplicated and a reported `MTA_MAX_CHUNKS` cap keeps even pathological inputs bounded.
+
+## Platform support
+
+Apple M-series is the primary, most-optimised target. Other platforms are supported with portable fallbacks:
+
+| Platform | Status | Notes |
+| --- | --- | --- |
+| macOS (Apple silicon) | ✅ optimised | performance-core pool, MLX GPU Whisper, unified-memory-aware |
+| macOS (Intel) | ✅ supported | physical-core sizing via psutil, CPU Whisper |
+| Linux | ✅ supported | apt/dnf/pacman install paths, CUDA Whisper if a GPU is present |
+| Windows | 🧪 experimental | `pip install memorised-them-all` + run `mta serve`; psutil process management & PATH healing |
+
+CI runs the offline test suite across **Ubuntu, macOS, and Windows** on Python 3.10 & 3.12.
+
+## Generated files & reuse
+
+Each project under `MTA_HOME/projects/<name>/` is self-contained and portable:
+
+| File | What it is |
+| --- | --- |
+| `graph.json` | source of truth — nodes, edges, communities, layered summaries, stats (`version`-stamped; stores basenames, no absolute paths) |
+| `memory.md` | compact, layered digest for reading / pasting |
+| `memory/<doc>.md` | one note per source document |
+| `mindmap.html` | offline interactive graph (Cytoscape inlined) |
+| `vectors.npz` + `vectors.json` | local embeddings for recall |
+
+A memory built once can be **copied to another machine** and reused read-only — recall and the mind map work with no rebuild. `export_memory` bundles all of the above (including the vector store) into a folder you choose.
+
+## Quality & testing
+
+This project is exercised hard: a multi-format corpus (Office, PDF, scanned PDF, OCR images, audio), **14 regression tests** (determinism, token-safety, fact attribution, accumulation, OCR, lifecycle, cross-platform), green CI on three OSes, and a multi-agent review pass covering accuracy, reliability, token-safety, reusability, cross-platform, and security. The token-free guarantee is enforced (recall slices are hard-capped) and the digest never returns document contents to the model.
+
 ## Acknowledgements
 
 Built on the shoulders of excellent open-source work — see

install.sh

@@ -55,15 +55,40 @@ fi
 # Install the package itself (provides the `mta` entry point).
 "$PYBIN" -m pip install --quiet -e "$DIR" >/dev/null 2>&1 || true
 
-# --- 3. System applications via Homebrew -----------------------------------
+# --- 3. System applications (Homebrew on macOS/Linux; native pkg mgrs on Linux) ----
+# A non-interactive sudo helper: only elevates if it won't block on a password.
+SUDO=""
+if [ "$(id -u 2>/dev/null)" != "0" ] && command -v sudo >/dev/null 2>&1 && sudo -n true 2>/dev/null; then
+  SUDO="sudo -n"
+fi
 if command -v brew >/dev/null 2>&1; then
   for app in ollama tesseract ffmpeg; do
     command -v "$app" >/dev/null 2>&1 || { log "brew install $app"; brew install "$app" || true; }
   done
   # Many OCR languages (incl. Bengali) + igraph C lib for python-igraph.
   brew list tesseract-lang >/dev/null 2>&1 || brew install tesseract-lang >/dev/null 2>&1 || true
+elif command -v apt-get >/dev/null 2>&1 && [ -n "$SUDO$( [ "$(id -u)" = 0 ] && echo root )" ]; then
+  log "Installing system apps via apt…"
+  $SUDO apt-get update -y >/dev/null 2>&1 || true
+  $SUDO apt-get install -y tesseract-ocr tesseract-ocr-all ffmpeg >/dev/null 2>&1 || true
+  if ! command -v ollama >/dev/null 2>&1; then
+    # Download then execute (not curl|sh) so partial/garbled output can't run.
+    _oll="$(mktemp)"; curl -fsSL https://ollama.com/install.sh -o "$_oll" \
+      && sh "$_oll" >/dev/null 2>&1; rm -f "$_oll" || true
+  fi
+elif command -v dnf >/dev/null 2>&1 && [ -n "$SUDO$( [ "$(id -u)" = 0 ] && echo root )" ]; then
+  log "Installing system apps via dnf…"
+  $SUDO dnf install -y tesseract ffmpeg >/dev/null 2>&1 || true
+  if ! command -v ollama >/dev/null 2>&1; then
+    # Download then execute (not curl|sh) so partial/garbled output can't run.
+    _oll="$(mktemp)"; curl -fsSL https://ollama.com/install.sh -o "$_oll" \
+      && sh "$_oll" >/dev/null 2>&1; rm -f "$_oll" || true
+  fi
+elif command -v pacman >/dev/null 2>&1 && [ -n "$SUDO$( [ "$(id -u)" = 0 ] && echo root )" ]; then
+  log "Installing system apps via pacman…"
+  $SUDO pacman -S --noconfirm tesseract tesseract-data-eng ffmpeg ollama >/dev/null 2>&1 || true
 else
-  log "Homebrew not found — install Ollama/Tesseract/ffmpeg manually for full features."
+  log "No usable package manager (or sudo unavailable) — install Ollama/Tesseract/ffmpeg manually for full features."
 fi
 
 # --- 4. Local models (background unless MTA_SKIP_MODELS=1) ------------------

manifest.json

@@ -2,7 +2,7 @@
   "manifest_version": "0.3",
   "name": "memorised-them-all",
   "display_name": "Memorised them All (local, token-free)",
-  "version": "1.1.0",
+  "version": "1.2.0",
   "description": "Convert any attachment to Markdown and digest it into a token-free knowledge-graph memory — locally.",
   "long_description": "Memorised them All turns a folder of documents into durable, **token-free** memory for Claude, entirely on your machine. It converts every attachment (PDF, Word, Excel, PowerPoint, HTML, EPub, images, audio, and more) to Markdown locally using Microsoft's MarkItDown (kept up to date from upstream automatically), Tesseract OCR, on-device Whisper, and a local Ollama vision model. It then runs an original local pipeline — semantic chunking, local embeddings, local-LLM (with a classical fallback) extraction of entities/relations/facts, embedding+fuzzy entity resolution, and community detection — to build a layered knowledge graph: a global synopsis, per-theme summaries, per-document Markdown notes, an exportable memory bundle, and an offline interactive mind map. Every tool returns only compact metadata or a tiny relevant slice, never document contents, so digesting and recalling whole folders costs ~0 Claude context tokens. Everything is free and open-source and runs locally; the Ollama model server starts on demand and stops after 5 minutes idle. Tuned for Apple M-series silicon (performance-core parallelism, GPU Whisper via MLX, unified-memory-aware concurrency).\n\nTools: digest, recall, memory_overview, export_memory, list_digestible, memory_status, open_mindmap.",
   "author": {
@@ -95,7 +95,7 @@
     }
   },
   "compatibility": {
-    "platforms": ["darwin"],
+    "platforms": ["darwin", "linux"],
     "runtimes": {"python": ">=3.10"}
   }
 }

mta/__init__.py

@@ -3,5 +3,5 @@
 Clean-room implementation by GRU-953 (Aninda Sundar Howlader).
 """
 
-__version__ = "1.1.0"
+__version__ = "1.2.0"
 __author__ = "Aninda Sundar Howlader"

mta/cli.py

@@ -14,8 +14,6 @@
 
 import argparse
 import json
-import subprocess
-import sys
 
 from .core import recall as recall_mod
 from .core import render, updater
@@ -36,6 +34,8 @@ def main(argv: list[str] | None = None) -> int:
     d = sub.add_parser("digest", help="convert + digest files/dirs/globs")
     d.add_argument("paths", nargs="+")
     d.add_argument("--reset", action="store_true")
+    d.add_argument("--fast", action="store_true",
+                   help="skip the LLM (classical extraction); faster, fully deterministic")
 
     r = sub.add_parser("recall", help="query the memory")
     r.add_argument("query")
@@ -62,7 +62,7 @@ def main(argv: list[str] | None = None) -> int:
     cfg = load_config().with_project(args.project)
 
     if args.cmd == "digest":
-        _print(run_digest(cfg, args.paths, reset=args.reset))
+        _print(run_digest(cfg, args.paths, reset=args.reset, fast=args.fast))
     elif args.cmd == "recall":
         _print(recall_mod.recall(cfg, args.query, k=args.k or None))
     elif args.cmd == "overview":
@@ -77,8 +77,8 @@ def main(argv: list[str] | None = None) -> int:
             _print({"status": "no_memory", "project": cfg.project})
         else:
             if args.open:
-                opener = "open" if sys.platform == "darwin" else "xdg-open"
-                subprocess.run([opener, str(cfg.mindmap_html)], check=False)
+                import webbrowser  # portable across macOS/Linux/Windows
+                webbrowser.open(cfg.mindmap_html.as_uri())
             _print({"status": "ok", "path": str(cfg.mindmap_html)})
     elif args.cmd == "update":
         _print(updater.run_check(cfg, force=args.force))

mta/core/config.py

@@ -50,10 +50,18 @@ class Config:
 
     # Extraction / digestion.
     extract_mode: str = field(default_factory=lambda: _env("MTA_EXTRACT", "auto"))  # auto|llm|classical
+    # Fast mode: skip the LLM entirely (classical extraction + deterministic
+    # summaries), keeping the small embedding model for recall. Opt-in; the
+    # default stays the accurate LLM path.
+    fast: bool = field(default_factory=lambda: _env("MTA_FAST", "off").lower()
+                       in ("on", "1", "true", "yes"))
     community_algo: str = field(default_factory=lambda: _env("MTA_COMMUNITY_ALGO", "auto"))  # auto|leiden|louvain|greedy
     chunk_chars: int = field(default_factory=lambda: _env_int("MTA_CHUNK_CHARS", 1200))
     recall_k: int = field(default_factory=lambda: _env_int("MTA_RECALL_K", 8))
     max_chunks: int = field(default_factory=lambda: _env_int("MTA_MAX_CHUNKS", 1500))
+    # Skip individual files larger than this (MB) before reading them into memory,
+    # bounding OOM/decompression-bomb risk. 0 disables the cap.
+    max_file_mb: int = field(default_factory=lambda: _env_int("MTA_MAX_FILE_MB", 200))
     extract_workers: int = field(default_factory=lambda: _env_int("MTA_EXTRACT_WORKERS", 0))  # 0=auto
 
     # Lifecycle & maintenance.
@@ -118,4 +126,6 @@ def ensure_dirs(self) -> None:
 
 def load() -> Config:
     cfg = Config()
+    if cfg.fast:
+        cfg.extract_mode = "classical"  # no LLM extraction or summaries
     return cfg