docs(consolidate-perf-001): Epic 5 — CHANGELOG + CLAUDE.md note + feature doc + post

Documents the dynamic_edges O(n²) finding + the circuit-breaker fix with the
live Tier-3 results (dynamic_edges 29min→34ms, post_clustering ~230×, watchdog
cycle 38min→~47s). CLAUDE.md note corrects the prior 'consolidation is expected
to be CPU-heavy / 38-min re-cluster' framing — it was dynamic_edges all along.
Records the Sprint-B targets (vector-index rewrite, incremental ForwardPass,
summaries_llm).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: rhenley1958

CHANGELOG.md

@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Sprint CONSOLIDATE-PERF-001 (Sprint A) — consolidation per-phase instrumentation + the dynamic_edges bottleneck** (2026-06-30). The real performance work the now-truthful `neo4j-cpu` alert (ALERT-TRUTH-001) pointed at. **Measure-first paid off immediately:** the new `mdemg_consolidation_phase_duration_seconds{space_id,phase}` instrumentation (shared `metrics.RecordConsolidationPhase`, wired into BOTH the service-level `RunConsolidation` watchdog driver and the `handleConsolidate` HTTP path, + per-pipeline-step `step:<name>` timing) found the ~38-min cost on the first cycle — and it wasn't the inferred ForwardPass. **`post_clustering` ran ~29 min and hit the 30-min timeout; the per-step breakdown pinned it to `dynamic_edges`** — `CreateDynamicEdges`' find-pairs query is a Cartesian product over all L≥minLayer nodes (`MATCH (a),(b) WHERE a.layer>=3 AND b.layer>=3`), which at scale (live: **8,705 L3+ nodes → ~75.8M pairs**) dominated the cycle for a paltry yield (272 ANALOGOUS_TO/BRIDGES edges total, `LIMIT 50`/run). **Fix (data-decided):** a node-count circuit-breaker — `CreateDynamicEdges` counts L≥minLayer nodes (via the existing `memorynode_layer_idx`) and SKIPS the O(n²) cross-join loudly above `DYNAMIC_EDGES_MAX_NODES` (default 2000); small graphs keep full behavior, large graphs skip until the Sprint-B vector-index rewrite. Also raised `CONSOLIDATE_TIMEOUT_MS` default 30→60 min (the manual path aborted mid-cycle at 30). The planned cheap win `findSimilarConcept` was **dropped** (measured 1.6s, negligible). **Live Tier-3: `dynamic_edges` 29 min → 34 ms; `post_clustering` ~1,760,000 ms → 7,674 ms (~230×); the watchdog cycle (the 2-3/day Neo4j-CPU driver) 38 min → ~47 s; identity preserved (L1=5130, L3=4634 stable); CPU calm (165 < the 500 alert).** The 19-alert Neo4j-CPU class is eliminated at the source. New config: `DYNAMIC_EDGES_MAX_NODES` (2000), `CONSOLIDATE_TIMEOUT_MS` default 60 min. Sprint-B targets (now visible): `summaries_llm` 120s (manual-path LLM), forward/backward ~27s (incremental rewrite), the `dynamic_edges` vector-index rewrite. Feature doc: `docs/features/consolidation-performance.md`. Sprint: `docs/development/consolidate-perf-001/`.
+
 - **Sprint ALERT-TRUTH-001 — alert & dashboard correctness** (2026-06-30). A Grafana review found 50 pending alerts while the substrate was healthy (RSIC 0.73–0.79, 0 breakers, ~0% real LLM error rate) — the channel was dominated by **miscalibrated rules and dashboard bugs firing on normal operation**. This is a correctness pass (no behavior-contract change) that makes the alerts and panels tell the truth. **(1) Alert-rule SQL hygiene:** the last 4 rules still on the `ORDER BY time DESC LIMIT 1` anti-pattern (`neo4j_high_cpu`, `neo4j_high_memory`, `low_cache_hit_ratio`, `jiminy_follow_rate_drop`) rewritten to windowed `AVG` + `COALESCE` so each returns one non-NULL row on an idle window (TSDB-CONSUME-001 contract; pin test `TestAllRules_NoLimitOneAntiPattern` gathers every rule group). **(2) Neo4j CPU calibration:** the fixed `80` was "% of ONE core" — on multi-core hardware normal consolidation runs 200–837%, so it fired on any real graph work (the 19-alert driver). Extracted to a config-driven `Neo4jCPURule(NEO4J_CPU_ALERT_THRESHOLD_PERCENT`, default **500**) over a 5-min windowed AVG; 500 is calibrated above the worst normal-operation 5-min windowed AVG (live 24h max 471; 0 windows >500). Distinct Service `neo4j-cpu` (was colliding with memory's `neo4j` under the NOSILENT-001 cooldown key). **Live-verified:** during active consolidation the windowed AVG was 447 (<500) → the rule stayed silent, while the old LIMIT-1 rule (latest 377 > 80) would have fired; **0 new CPU alerts in 70+ min**. **(3) LLM error-count floor:** `llm_error_rate_spike` adds `RSIC_LLM_ERROR_MIN_COUNT` (default 5) — the rate-only gate (>5% & >10 calls) re-fired HIGH on a couple of transient errors at low volume. (The remaining truthful firing is `retrieval.rerank_cross` at a real 9.6%/9-error rate — the rerank-cancellation root cause is a separate track.) **(4) Stale NLI bias eliminated:** `GetNLICalibrationReport()` now returns nil when the NLI sidecar isn't operational — a gated-off sidecar left a stale window producing a phantom mean-bias (live: 0.638 at 0 requests) that pinned a continuously-firing `nli_bias_alert`. Gating at the source covers all three consumers (both `j17_nli_mean_bias` gauge emitters + the RSIC calibration-drift insight). **Live-verified:** gauge 0 (0/10 readings non-zero) post-fix. **(5) Latency histogram buckets** extended 10s→120s (added 20/30/60/120) — the set topped out at le=10s, so LLM-backed paths (retrieval includes the rerank; ape.reflect/jiminy.synthesize run 10–76s) clamped the p95/p99 estimate at 10, a permanent red. **(6) Dashboard panel bugs:** overview `P95 Latency` red 0.25s→120s (aligns with the `ALERT_RETRIEVE_P95_MS` SLO); neo4j `Neo4j Memory` 512MiB/1GiB→24GiB/27GiB; `Null-Weight Abstraction Edges` unit percent→short + thresholds 60/85→50/100 (it's a count); `Conversation Coverage Ratio` unit percent→percentunit + inverted thresholds red<0.2/green≥0.4; j17 `Avg Comprehension` calc `mean`→`lastNotNull` (the 0/real flush-interleave dragged the mean to 0.43 vs real 0.858). New config: `NEO4J_CPU_ALERT_THRESHOLD_PERCENT` (500), `RSIC_LLM_ERROR_MIN_COUNT` (5). Build + lint clean; alert/ape/jiminy/metrics suites green. Feature doc: `docs/features/alert-dispatcher.md`. Sprint: `docs/development/alert-truth-001/`.
 
 - **Sprint FT-RECURSIVE-002 (Phase 6b) — the recursive-retrain actuator (shipped default-off; live cycle next)** (2026-06-28). Makes the no-op `trigger_training_pipeline` real — a supervised controller that runs a gated retrain cycle — but **dormant behind `FT_LOOP_ENABLED=false`** (nothing trains or mutates serving state until opt-in). **Ledger:** `ft_training_cycles` (V0002, first writer) — event-sourced state machine (`triggered→curating→training→gating→promote_pending→{promoted|failed|rolled_back}`), DB-backed single-flight + retrain-interval gate that survive restarts. **Trigger gate** (`internal/ftloop`): a cycle launches only on Ready ∧ fresh-fraction (`FT_LOOP_MIN_FRESH_FRACTION` 0.30) ∧ no cycle within `FT_LOOP_MIN_RETRAIN_INTERVAL_HOURS` (168) — **and the SF-2 fix: when disabled/open/within-interval the trigger is SUPPRESSED** (no alert), ending the per-reflection-cycle `rsic-trigger_training_pipeline` spam (live-verified: 0 dispatches post-restart). **Compute lease** (single-host lockfile, reclaimable on expiry so a crashed trainer can't wedge RSIC) **+ `OrchestrationPolicy.Quiesce`** (pauses new RSIC triggers during a retrain) **+ disk-floor preflight.** **Controller:** supervised, walks curate→train→gate as ctx-cancellable Python subprocesses, ledger + `ft-loop:<stage>` jobhealth per stage, FAIL→archived+one alert, lease-expiry/disk-floor→class-4 alert-and-halt, PASS→`promote_pending` (halts — promotion is operator-gated via `mdemg ft-loop promote --cycle-id [--reject]`). `[AMD-7]` `TRAINING_READINESS_THRESHOLD` (+per-task) replaces the hardcoded 500; `[AMD-1]` epoch cap passed to train_ft. New config: `FT_LOOP_ENABLED`, `FT_LOOP_{MIN_RETRAIN_INTERVAL_HOURS,MIN_FRESH_FRACTION,POLL_INTERVAL_SEC,LEASE_MAX_HOURS,LEASE_PATH,MIN_FREE_DISK_GB,PYTHON_BIN,MODEL_VERSION}`, `FT_LORA_EPOCHS_CAP`, `FT_EARLY_STOP_VAL_LOSS_FACTOR`, `TRAINING_READINESS_THRESHOLD(_OVERRIDES)`. Unit tests across ledger/gate/lease/quiesce/controller; config guard 756/756; lint 0. **Epic 6 (2026-06-29/30):** every pipeline stage validated **live against the real system** — export (`mdemg data export`) → curate (`paradigm_router`, 4660-row SFT split) → train (real 168 MB LoRA adapter; loss 3.28→2.41) → convert (`fuse --dequantize`→`convert_hf_to_gguf`→`llama-quantize Q5_K_M`, 11 GB candidate) → gate (`run_benchmark` real calls, no-zero-call) — and the controller **wired with those proven commands** (`controller_stages.go`, PR #492). The orchestration (lease/quiesce/ledger/FAIL) was validated live in "Option A" (a real cycle walked `triggered→curating→failed`, SF-2 held, lease released). **A real bug was caught + fixed along the way:** the base-model SHA drift-guard had rotted (E6-8, PR #489 — upstream `mlx-community/Qwen3-14B-4bit` re-published its config; would've broken any retrain) and the quiesce design was confirmed necessary (E6-12: training saturated the box). 14 findings ledgered (`epic6_issues.md`). Remaining: a single *enabled* tiny-subset integration drill (the heavy convert + slow contended-box train) + Phase 7 (auto-promote/canary). Still **default-off** behind `FT_LOOP_ENABLED`. 12 Epic-6 config knobs (`FT_LOOP_{WORK_DIR,BASE_MODEL,BASE_SHA,UAITS_SPEC,BENCHMARK_CONFIG,LORA_RANK,LORA_ALPHA,GATE_PORT,EXPORT_SINCE_DAYS,GATE_TASK_FILTER,GATE_MIN_SCORE}`). Feature doc: `docs/features/ft-recursive-loop.md`. Sprint: `docs/development/ft-recursive-002/` (incl. `epic6_issues.md`, `run_record.md`).

CLAUDE.md

@@ -330,6 +330,8 @@ Endpoints:
 
 - **Alert & dashboard correctness (Sprint ALERT-TRUTH-001 — 2026-06-30):** a Grafana review surfaced 50 pending alerts while the substrate was healthy — the channel was **miscalibration noise, not failure**. Closed the last of the LIMIT-1/threshold/measurement defects. ⚠️ **The 4 remaining `ORDER BY time DESC LIMIT 1` rules** (`neo4j_high_cpu`/`_memory`, `low_cache_hit_ratio`, `jiminy_follow_rate_drop`) → windowed `AVG`+`COALESCE` (the TSDB-CONSUME-001 contract; **pin `TestAllRules_NoLimitOneAntiPattern` now gathers EVERY rule group** + `TestAllRules_DistinctServicePerSeverity` for the NOSILENT-001 cooldown key — new rules are auto-covered). ⚠️ **`docker-stats CPU% is per-SINGLE-core`** — the fixed `neo4j_high_cpu` threshold `80` fired on any real multi-core graph work (normal consolidation 200–837%; the 19-alert driver; live 7d p50=1/p95=255 also flapped the single-sample read). Extracted to config-driven `alert.Neo4jCPURule(NEO4J_CPU_ALERT_THRESHOLD_PERCENT` default **500**, host-relative) over a 5-min windowed AVG; 500 calibrated above the worst NORMAL 5-min windowed AVG (live 24h max 471, 0 windows >500) so it fires only on worse-than-any-normal-consolidation load. Distinct Service `neo4j-cpu` (was colliding with memory's `neo4j`). **Reducing the consolidation cost itself is a separate concern — it is EXPECTED to be CPU-heavy (the full 22-phase re-cluster on 83k nodes runs ~38 min); HIDDEN-CHURN-003 made only the L0-orphan hidden step incremental.** ⚠️ **`GetNLICalibrationReport()` returns nil when `!nliScorer.IsOperational()`** — a gated-off NLI sidecar left a stale calibration window producing a phantom `j17_nli_mean_bias` (live 0.638 at 0 sidecar requests) that pinned a continuously-firing `nli_bias_alert`; gating at the source (vs each consumer) covers both gauge emitters (`self_assess.go`, `live_collectors.go`) AND the RSIC drift insight (`self_reflect.go`) in one place — extends the DH-004 `RecordNLIFallback` guard to the bias/alert path. `llm_error_rate_spike` gained `RSIC_LLM_ERROR_MIN_COUNT` (5) so sub-5-error transients don't re-fire HIGH (the rerank_cross 9.6%/9-error spike is a TRUE positive → the rerank-cancellation fix is a separate track). ⚠️ **Latency histogram buckets MUST cover the LLM path** — the set topped out at `le=10s` while retrieval (incl. cross-encoder rerank) + ape.reflect/jiminy.synthesize run 10–76s, so every p95/p99 panel clamped at 10 (permanent red); extended to 120s (added 20/30/60/120 in `metrics/registry.go`; high buckets stay percentile-neutral for fast histograms). Dashboard fixes: overview `P95 Latency` red 0.25s→120s (ALERT_RETRIEVE SLO), neo4j `Neo4j Memory` 512MiB/1GiB→24/27GiB, `Null-Weight Edges` percent→short, `Conversation Coverage` percent→percentunit (inverted), j17 `Avg Comprehension` calc `mean`→`lastNotNull`. **Live Tier-3:** during active consolidation windowed CPU AVG 447 (<500) → 0 new CPU alerts in 70+ min (old rule would fire on latest 377>80); NLI gauge 0/10 readings non-zero. Sprint: `docs/development/alert-truth-001/`.
 
+- **Consolidation per-phase instrumentation + the dynamic_edges bottleneck (Sprint CONSOLIDATE-PERF-001 Sprint A — 2026-06-30):** the real perf work behind the `neo4j-cpu` alert. ⚠️ **The ~38-min consolidation cost was NOT the re-cluster / ForwardPass (the prior CLAUDE.md guess) — it was `dynamic_edges`.** `mdemg_consolidation_phase_duration_seconds{space_id,phase}` (shared `metrics.RecordConsolidationPhase`, wired into BOTH `RunConsolidation` (watchdog driver) and `handleConsolidate` (manual) + per-pipeline-step `step:<name>` timing in `RunPhaseRange`) caught it on the first cycle: **`post_clustering` ran ~29 min and hit the timeout, all of it `CreateDynamicEdges`** — a Cartesian product `MATCH (a),(b) WHERE a.layer>=3 AND b.layer>=3` over every L≥minLayer node (live: **8,705 → ~75.8M pairs**) for 272 edges total. ⚠️ **Fix: node-count circuit-breaker** — `CreateDynamicEdges` counts L≥minLayer (`countNodesAtOrAboveLayer`, via `memorynode_layer_idx`) and SKIPS the cross-join loudly above `DYNAMIC_EDGES_MAX_NODES` (default 2000); the real O(n·logn) vector-index rewrite is Sprint B. `CONSOLIDATE_TIMEOUT_MS` default 30→60 min (manual path aborted mid-cycle). **Live Tier-3: `dynamic_edges` 29 min → 34 ms; `post_clustering` ~1.76M ms → 7.7s (~230×); the watchdog cycle (2-3/day, the CPU-saturation source) 38 min → ~47 s; identity preserved; CPU calm.** ⚠️ **When adding a consolidation phase, instrument it (`metrics.RecordConsolidationPhase`) and NEVER write a full Cartesian `MATCH (a),(b)` over a graph layer — it's O(n²); use the vector index or bound the seed set.** Now-visible next costs (Sprint B): `summaries_llm` 120s (manual-path serial LLM), forward/backward ~27s (the incremental-ForwardPass rewrite — `last_forward_pass` infra exists), the `dynamic_edges` vector-index rewrite. Feature doc: `docs/features/consolidation-performance.md`. Sprint: `docs/development/consolidate-perf-001/`.
+
 - **Recursive-retrain loop Phase 6a — manual-path observability (Sprint FT-RECURSIVE-001 — 2026-06-28):** the first *build* sprint of the recursive-retraining loop (design: `docs/development/ft-recursive-001/SPEC_recursive_retraining_loop.md`). Observability + hardening ONLY — **no actuator, no autonomy, no serving mutation** (those are 6b/7/9). Shipped: **`mdemg ft-loop report-stage --stage <capture|curate|train|benchmark|gate|promote> --status <success|failure>`** records each manual retrain stage to `scheduled_job_events` (`job_name=ft-loop:<stage>`) via the NOSILENT-001 CLI-job pattern, failure → high alert under the distinct `ft-loop` service (added `jobhealth.ReportWithService(service)` so ft-loop doesn't share the generic `scheduled-job` cooldown key). ⚠️ **The CLI reads TSDB config from the env** (load `.env` / export `TSDB_HOST_PORT` — a bare invocation silently no-ops the pool, like the other CLI jobs). SF-3: RSIC diagnostic alerts now use `rsic-<action>` services (were all `("rsic",Medium)` → mutual suppression — same cooldown-key class as NOSILENT-001/ORPHAN-ALERT-001). SF-7: `mdemg data status` surfaces per-gate not-ready reasons (`evaluateReadinessGates`, pure). SF-1: heartbeat gauge `mdemg_rsic_readiness_assessed` (set only on a *successful* readiness assessment) + `training_readiness_stale` rule (`FT_READINESS_STALENESS_MIN` 30) catch a silently-dormant loop. `[AMD-2]`: the promotion-gate eval is pinned (`docs/development/ft-recursive-001/augmented_eval_manifest.json` — recipe + SHA + spec_hashes + leak verdict). SF-6: export archives prune (`MDEMG_EXPORT_RETENTION_HOURS` 168). Feature doc: `docs/features/ft-recursive-loop.md`. Sprint: `docs/development/ft-recursive-001/`.
 
 - **Recursive-retrain loop Phase 6b — the actuator (Sprint FT-RECURSIVE-002 — 2026-06-28→30, SHIPPED default-off):** the no-op `trigger_training_pipeline` is now a real supervised controller (`internal/ftloop`), but **dormant behind `FT_LOOP_ENABLED=false`** — nothing trains/mutates serving until opt-in. ⚠️ **The `ft_training_cycles` ledger (event-sourced, V0002 first writer) is the single source of truth for "is a cycle running" — single-flight is a DB query (`tsdb.OpenCycle`), NOT the RSIC 300s in-memory cooldown (wrong tool at ~9h retrain scale).** **SF-2 fixed:** `executeTriggerTrainingPipeline` routes through the gate (`ftloop.Gate`), which SUPPRESSES the trigger alert when disabled/open-cycle/within-`FT_LOOP_MIN_RETRAIN_INTERVAL_HOURS` — the `rsic-trigger_training_pipeline` per-cycle spam is gone (live-verified 0 dispatches). On a trigger decision the gate OPENS a cycle; the controller consumes it **out-of-band via the ledger** (SPEC fork 7 — the action stays diagnostic; the mutating-action class is Phase 7). ⚠️ **Compute lease (`ftloop.AcquireLease`, single-host lockfile) is reclaimable on expiry (`FT_LOOP_LEASE_MAX_HOURS` 14) so a crashed trainer can't wedge RSIC forever; `OrchestrationPolicy.Quiesce(until)` pauses new RSIC triggers while a retrain holds the box.** Controller walks curate→train→gate as **ctx-cancellable subprocesses** (a client/shutdown cancels the trainer), ledger + the 6a `ft-loop:<stage>` jobhealth per stage, PASS→`promote_pending` (halts — operator confirms via `mdemg ft-loop promote`). ⚠️ **typed-nil gotcha:** a nil `*pgxpool.Pool` passed to `RecordCycleEvent` (interface param) is a non-nil interface — guard with a concrete `if pool == nil` at the call site. **Epic 6 (2026-06-29/30) wired the five pipeline stages** (`controller_stages.go`: export→curate→train→convert→gate), each command validated **live against the real system** (export `mdemg data export`; curate `paradigm_router` venv/cwd-`neural/`; train real 168 MB LoRA; convert `mlx_lm.fuse --dequantize`→`convert_hf_to_gguf`→`llama-quantize Q5_K_M`; gate side-port `llama-server`+`run_benchmark`, no-zero-call). ⚠️ **E6-8: the base-model SHA pin had rotted** (`cdc16756…`→`a54ec18f…` byte-identical to the production config; upstream re-published `config.json`) — it would have failed *any* retrain; re-pinned in `train_ft.py`/`stratified_split.py`/tests (PR #489). ⚠️ **E6-10/11 curation gotchas:** `paradigm_router` emits `val.jsonl` but `mlx_lm.lora` wants `valid.jsonl` (controller copies it); `train_ft` reads iters from the manifest `splits.train.rows`, not the file (subset runs must edit the manifest count). ⚠️ **E6-14:** the 4bit-fused MLX adapter can't `convert_hf_to_gguf` directly — `mlx_lm.fuse --dequantize` to bf16 first. **E6-12:** on-box training saturates the machine and degrades the production `llama-server` (confirms the quiesce design). Controller wired PR #492; 11 `FT_LOOP_*` Epic-6 pipeline config knobs. Remaining: a single *enabled* tiny-subset drill off-peak. Feature doc: `docs/features/ft-recursive-loop.md`. Sprint: `docs/development/ft-recursive-002/` (`epic6_issues.md`, `run_record.md`).