Skip to content

Commit c73a4e1

Browse files
theletterfclaude
andauthored
Replace skill-failure noop with manual-analysis fallback (#134)
Reverses the strict abort rule from #133 in the openings, applies-to, and style sweeps. Real-run evidence (elastic/docs-content runs 25377518194 + 25378183549, both openings) showed the hardened "two failures = noop" rule firing prematurely: - 13:09:34 ✗ skill(docs-page-opening-optimizer) Skill not found - 13:09:37 ✗ skill(docs-page-opening-optimizer) Skill not found - 13:09:41 ● noop "skill unavailable — confirmed after exact-form attempts" - 13:14:36 ● create_issue "shard 20/28 — 20 pages" ← agent eventually produced an issue, but suppressed because noop already fired The agent's tool serialization keeps logging the skill call as `skill(docs-X)` — without the `skill:` prefix — which always returns "Skill not found" from Copilot CLI. Whether the agent's underlying invocation was actually reformatted, or whether it's a log-rendering quirk, doesn't matter: my hardening committed the agent to noop after two of these "failures", before it had a chance to do useful work. Replace with frontmatter's softer pattern: try the skill once; if "Skill not found" comes back, fall back to manual analysis using bash + the agent's own judgment, and note the skill failure once in the issue body's Notes section. Only noop if even manual analysis produces no high-confidence findings. Applied uniformly to: - gh-aw-docs-openings-sweep.md (the sweep that's been noop'ing) - gh-aw-docs-applies-to-sweep.md - gh-aw-docs-style-sweep.md Frontmatter, coherence, staleness, typos already had softer or deterministic-only fallback paths and don't need this change. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent de00cb2 commit c73a4e1

6 files changed

Lines changed: 57 additions & 63 deletions

.github/workflows/gh-aw-docs-applies-to-sweep.lock.yml

Lines changed: 15 additions & 16 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.github/workflows/gh-aw-docs-applies-to-sweep.md

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -205,13 +205,12 @@ Invoke `skill(skill: docs-applies-to-tagging)` against `/tmp/gh-aw/sweep-data/sc
205205

206206
**Audit mode only — do not write any files.** The skill defaults to validation when not asked to fix; reinforce that intent in the call. This sweep produces an issue, not edits.
207207

208-
**Only treat the skill as unavailable after a confirmed exact-form failure.** Stochastic agent retries sometimes invoke the skill with a shortened form (e.g., `skill(docs-applies-to-tagging)` without the `skill:` prefix) which fails before the exact form is tried. Procedure:
208+
**If the skill returns "Skill not found", do not noop.** That error is ambiguous — it can mean the skill genuinely isn't installed, or that the invocation form was reformatted by the agent's tool serialization. Either way: fall back to manual analysis instead of aborting. Procedure:
209209

210-
1. Invoke with the exact form `skill(skill: docs-applies-to-tagging)`.
211-
2. If the result is "Skill not found", retry once more with the same exact form.
212-
3. Only after a second exact-form failure, abort the run by calling `noop` with `"docs-applies-to-tagging skill unavailable"` — there is no fallback heuristic worth filing as findings.
210+
1. Try invoking the skill with the exact form `skill(skill: docs-applies-to-tagging)`.
211+
2. If that fails or returns "Skill not found": fall back to **manual analysis**. Use `bash` to scan each in-scope file's frontmatter (the YAML block at the top), checking for missing/invalid `applies_to` keys against the documented allowed values. Produce findings as you would have from the skill output, and add a single line in the issue body's `Notes` section: "skill `docs-applies-to-tagging` did not produce output in this run; findings are agent-only."
213212

214-
A single first-attempt failure (especially with a non-exact form) is **not** sufficient evidence to noop.
213+
Only call `noop` if you cannot produce any high-confidence findings even from manual analysis.
215214

216215
## Step 2: Build the findings list
217216

.github/workflows/gh-aw-docs-openings-sweep.lock.yml

Lines changed: 15 additions & 16 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)