Skip to content

pr-review-toolkit: Agents hallucinate unsupported gh pr diff --stat flag #498

@bbarry

Description

@bbarry

Summary

When using /pr-review-toolkit:review-pr, agents frequently execute gh pr diff <number> --stat which fails because --stat is not a supported flag for gh pr diff.

Observed Behavior

During PR reviews, agents generate commands like:

gh pr diff 123 --stat 2>&1

This fails because gh pr diff only supports --color, --name-only, --patch, and --web. There is no --stat flag (confirmed on gh 2.86.0 and checked release notes through 2.87.2).

Root Cause

The commands/review-pr.md file instructs agents to identify changed files (line 33: git diff --name-only) but provides no guidance on how to get a diffstat summary. Agents infer they should use gh pr diff --stat by analogy with git diff --stat (which is valid), but the gh CLI does not support this flag.

This is an LLM hallucination — the flag looks plausible because git diff --stat works, so the model assumes the same flag exists on gh pr diff.

Suggested Fix

Add explicit guidance in commands/review-pr.md step 3 ("Identify Changed Files") specifying valid commands for getting diff statistics:

- Run `git diff --stat` for a diffstat summary (do NOT use `gh pr diff --stat` — the `--stat` flag is not supported by `gh pr diff`)
- Run `gh pr diff <number> --name-only` for changed file names only

Alternatively, a note in the agent files (agents/code-reviewer.md, etc.) reminding agents that gh pr diff only supports --color, --name-only, --patch, and --web would also help.

Environment

  • Claude Code version: 1.0.33
  • gh CLI version: 2.86.0 (also verified against 2.87.x release notes)
  • Plugin source: claude-plugins-official marketplace

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions