You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 80d2e12
Browse filesBrowse the repository at this point in the historyBrowse files
Wave 2 of the #143–#164 dogfooding batch: six data-quality review lenses in
deep-code-review/references/data-quality.md, each pre-distinguished from the
section it extends.
- Non-regression gate: a fanout/uniqueness gate false-blocks legitimate coverage
expansion — hard-block only a value absorbed from a now-departed distinct
record, not a newly-shared standing value among related entities (#149).
- Entity resolution: the resolution order when no stable id exists, and surfacing
the unresolved/ambiguous count as a first-class output (#153).
- Scoring & config: never sum heterogeneous constructs into one composite score
(#162); test every enum/config map against the source's real value
distribution, not the literals a unit test feeds it (#153).
- Measuring honestly: backtest a proxy-derived metric against ground truth before
shipping (#151); match the validation metric to the claim — an ordinal rank
validated by concordance / a C-index, not MAE (#161).
- Cost discipline: measure existing-source coverage before scoping new enrichment
or scrapers, and scope to the measured residual (#152).
- New red-flag detectors; fixed a Cyrillic-to-Latin typo in Entity resolution.
+6 evals (deep-code-review 45 -> 51). Lockstep bump to 1.73.0. Gates green:
test-ci 59/59, routing / version / privacy / enumeration, eval-predicates,
checksums (82 files). Ran the loop: the advisor (step 4) genericized unsourced
figures and dropped a borrowed-authority claim; the independent dogfood reviewer
(step 6) caught a cross-section duplication of the never-lower-a-baseline rule —
both fixed before merge.
Closes#149Closes#151Closes#152Closes#153Closes#161Closes#162
Co-authored-by: Remigiusz Antczak <remigiusz@antczak.io>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: .claude-plugin/plugin.json
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
"name": "deep-code-review",
3
3
"description": "Perun — bring the thunder to your codebase. A portable, evidence-grounded quality bar for building with AI agents: the flagship deep-code-review skill (default install is review-only), plus opt-in overlays — gated delivery, an idea critic, a suite conductor, and stage-aware product, growth, positioning, business-ops, and output-safety specialists.",
"prompt": "Our field-integrity gate fails closed when a unique-per-company handle appears on two records. This refresh it flagged 33 violations, about 32 of them sub-teams that legitimately now share their parent org's handle, and blocked the publish. Lower the gate's strictness?",
472
+
"expected_output": "Don't lower or disable it -- refine what it hard-blocks. The gate is right that a unique-to-one value on two entities CAN be corruption, but it false-positives during legitimate coverage expansion: many related entities newly and correctly sharing one value (sub-teams adopting a parent org's handle -- co-branding) each trip the fanout arm. Distinguish the shapes: hard-block only a value ABSORBED from a now-departed distinct record (two records collapsed into one); for a newly-shared STANDING value among related entities, keep a human at the gate or a small reviewed allowlist, and stop re-refusing an unchanged standing shared value every run (that just gets the gate switched off). Never 'green' the run by lowering the baseline -- acknowledge the intended shared values explicitly, per field, with a reason. (data-quality.md.)",
473
+
"expectations": [
474
+
"Refuses to lower or disable the gate or its baseline to pass the run",
475
+
"Distinguishes corruption (a value absorbed from a now-departed distinct record) from a legitimately newly-shared standing value among related entities",
476
+
"Hard-blocks only the corruption shape; routes legitimate shared values to a human or a reviewed allowlist",
477
+
"Flags re-refusing an unchanged standing shared value every run as what gets the gate disabled"
"prompt": "We're scoping a set of paid scrapers and enrichment APIs to hit a coverage target for our company list. The plan is ready -- greenlight the spend?",
483
+
"expected_output": "Measure existing-source coverage for that exact list first -- re-querying your internal warehouse is frequently the largest, cheapest lever and can match a from-scratch enrichment plan's target at zero marginal cost, often surfacing already-reconciled fields (a stable entity id, dated events, location) the working copy lacks. Scope the external, paid acquisition to the measured RESIDUAL only. A plan that adds scrapers before measuring what you already have over-scopes -- and paid calls sit behind free/keyless pre-gates and the usual spend caps regardless. (data-quality.md.)",
484
+
"expectations": [
485
+
"Requires measuring existing-source (warehouse) coverage for the target set before scoping new acquisition",
486
+
"Identifies re-querying existing data as the largest and cheapest coverage lever",
487
+
"Scopes external or paid enrichment to the measured residual only",
"prompt": "Two data-integrity things in this pipeline: (a) we resolve companies by matching on the name string, and (b) we have a geography-to-multiplier lookup. Unit tests for both pass. Ship?",
494
+
"expected_output": "Both are traps that pass unit tests while failing in production. (a) Never resolve identity on a bare name string -- namesake collisions merge distinct records, and a wrong merge is worse than a missed one. Resolve by stable id, then normalized domain/handle, then a name only as a last resort behind a collision (namesake) guard, and surface the unresolved/ambiguous count as a first-class output. (b) Test the geography-to-multiplier map against the source's REAL value distribution: if it's keyed on region names while the source emits ISO-3166 alpha-2 codes (plus lowercase and placeholders), it silently no-ops -- carrying its cost, delivering nothing -- and unit tests pass because they feed it the literals it expects. SELECT DISTINCT the real values and test against those. (data-quality.md.)",
495
+
"expectations": [
496
+
"Rejects name-string identity joins; requires stable id then normalized domain then name-behind-a-collision-guard, with the unresolved count surfaced",
497
+
"Requires testing the enum/config map against the source's real value distribution (SELECT DISTINCT), not chosen literals",
498
+
"Explains that a mis-keyed map silently no-ops while unit tests still pass",
499
+
"Treats a passing unit test as insufficient for either trap"
500
+
]
501
+
},
502
+
{
503
+
"id": "backtest-proxy-metric-vs-ground-truth",
504
+
"prompt": "We built a runway estimate from public proxies (last-round size / headcount x burn). Unit tests pass and the formula looks sound. Ship it as a ranking signal?",
505
+
"expected_output": "Not until you backtest it against ground truth -- a plausible proxy formula that passes unit tests can be near-useless for its actual job. For any derived or scored value from indirect proxies, require a validation step in review that reports MAE / correlation / base-rate against real self-reported actuals. Unit tests prove the math; only a backtest proves the value. If it can't rank (large MAE, weak correlation, a base rate far off the real one), demote or gate it -- a coarse band plus 'corroboration-required' -- rather than shipping it as a ranker. (data-quality.md.)",
506
+
"expectations": [
507
+
"Requires a ground-truth backtest (MAE / correlation / base-rate vs actuals) before shipping a proxy-derived metric",
508
+
"Distinguishes unit tests proving the math from a backtest proving the value",
509
+
"Demotes or gates a metric that fails the backtest instead of shipping it as a ranker"
"prompt": "Our score correlates with the target at about r=0.34 -- weak but real. MAE is too big to show a point estimate on the 0-24 month runway we're predicting, so we'll publish the estimate with a +/-22 month band. Validate it with MAE and ship?",
515
+
"expected_output": "Don't publish a point estimate or a +/-22-month band -- on a 0-24-month decision range that band is self-refuting noise on screen. A weak-but-nonzero predictor can still be useful as an ORDINAL RANK. Validate the ranking with concordance / a C-index against an OBSERVABLE binary event ('raised within 6 months', 'shut down within 6 months') -- not MAE on the noisy latent quantity, which measures the wrong thing. Emit an ordinal tier, not a number, when MAE is large relative to the decision range. Match the validation metric to the claim the score actually makes. (data-quality.md.)",
516
+
"expectations": [
517
+
"Rejects the point estimate and the band when the band is wide relative to the decision range",
518
+
"Requires validating rank quality with concordance / a C-index against an observable binary event, not MAE",
519
+
"Outputs an ordinal tier rather than a point estimate when MAE is large relative to the decision range",
520
+
"Frames it as matching the validation metric to the claim the score makes"
"prompt": "We're rolling raise-timing and financial-distress into one 'urgency' score by summing them, so a high number means 'act now.' Good?",
526
+
"expected_output": "Don't sum them -- they're heterogeneous constructs that imply OPPOSITE actions, so one blended number manufactures false positives and hides which action is warranted. Raise-timing overdue means 'introduce to investors'; distress/contraction means 'triage or support.' A record high on one and low on the other averages to a middling 'urgency' that tells you nothing. Score each construct separately and derive the action from the combination -- a 2x2 or tiering -- never from one summed number. (data-quality.md.)",
527
+
"expectations": [
528
+
"Refuses to sum heterogeneous constructs into one composite score",
529
+
"Explains the constructs imply opposite actions, so a blended number hides which is warranted",
530
+
"Requires scoring each construct separately and deriving the action from the combination (a 2x2 or tiering)"
0 commit comments