📚 Documentation Noob Test Report - Feb 12, 2026

[github-actions]

📚 Documentation Noob Test Report - February 12, 2026

Executive Summary

Date of test: February 12, 2026
Pages analyzed: 5+ key documentation pages (index, quick-start, creating-workflows, cli)
Overall impression as a new user: The documentation has great content and ambition, but several critical blocking issues prevent new users from getting started successfully. Most notably, 263 broken internal links and confusing navigation would leave a beginner frustrated and stuck.

---

🔴 Critical Issues Found (Block Getting Started)

1. BUILD FAILURE - 263 Broken Internal Links 🔴 BLOCKING

Severity: 🔴 BLOCKING - Documentation cannot be built with link validation enabled

What I found:

• The documentation build fails completely with 263 invalid links across 89 files
• Had to disable link validation entirely just to build and test the documentation
• This is the first thing a beginner contributor would encounter

Most commonly broken links:

• /gh-aw/setup/creating-workflows (18+ broken references)
• /gh-aw/reference/tools/
• /gh-aw/reference/safe-outputs/
• /gh-aw/reference/workflow-structure/
• /gh-aw/reference/permissions/
• /gh-aw/reference/network/
• /gh-aw/reference/threat-detection/
• /gh-aw/reference/triggers/
• /gh-aw/reference/safe-inputs/
• /gh-aw/troubleshooting/common-issues/

Impact:

• Users clicking links get 404 errors
• Cannot find critical reference documentation
• Frustrating "dead end" experience when trying to learn more
• Makes the docs feel incomplete and untrustworthy

Example from Quick Start:
The Quick Start guide references /gh-aw/troubleshooting/common-issues/ which doesn't exist, leaving stuck users with no help.

Recommendation:

• URGENT: Fix all 263 broken links as highest priority
• Run link validation in CI to prevent new broken links
• Consider a "Link Health" dashboard to track link status

2. Missing Prerequisites Context 🔴 BLOCKING for non-GitHub experts

What I found:
In the Quick Start guide, prerequisites list:

• "AI Account" - GitHub Copilot subscription, Anthropic Claude, or OpenAI Codex API key
• "GitHub Repository" - write access or higher
• "GitHub Actions enabled"

What's confusing:

1. No cost information: A beginner doesn't know:

   • How much does GitHub Copilot subscription cost?
   • Can I try this for free?
   • What's the cheapest option to get started?

2. "Write access or higher" - what does this mean?

   • Is "Maintainer" role enough?
   • Do I need to be an "Admin"?
   • What if I'm just a "Contributor"?

3. No GitHub Actions knowledge assumed:

   • The guide says "check Settings → Actions"
   • But what if Actions is disabled? How do I enable it?
   • What permissions do I need to enable Actions?

Impact:

• Beginners without GitHub Actions experience feel lost
• No clear path to "getting unstuck" if prerequisites aren't met
• Might give up before even starting

Recommendation:

• Add a "Cost & Free Tiers" section explaining pricing
• Link to GitHub's official permissions documentation
• Add a troubleshooting section for "What if Actions is disabled?"
• Consider a "Can I use this?" decision tree

3. Jargon Overload Without Definitions 🔴 BLOCKING

What I found:
The documentation uses technical terms without explaining them on first use:

From the homepage:

• "Frontmatter" - Used multiple times, never defined in context
• "Lock file" - What is a ".lock.yml file"? Why do I need it?
• "Safe outputs" - Mentioned frequently but not explained until later
• "Sandboxed execution" - What does this mean practically?
• "MCP servers" - What's MCP? (Model Context Protocol?)

From Quick Start:

• "Trigger a run" - How? From where? GitHub UI? Terminal?
• "Repository secrets" - Where do I add these? How?

Impact:

• Beginners feel overwhelmed by unfamiliar terminology
• No clear mental model of how the system works
• Users abandon the docs to search Google for definitions

Recommendation:

• Add a glossary link on first use of jargon (e.g., "frontmatter [?]")
• Create a "Key Concepts" section before Quick Start
• Use tooltips or inline definitions for technical terms
• Consider a "New to GitHub Actions?" primer section

4. Unclear File Structure Expectations 🟡 CONFUSING

What I found:
In "Creating Workflows" guide:

• Step 1: "Create the workflow file in .github/(workflow-name).md"

What's confusing:

1. Should this be .github/workflows/(workflow-name).md?
2. The Quick Start mentions .github/workflows/ but this says .github/
3. No explanation of why both .md and .lock.yml files exist
4. No visual diagram showing file structure

Impact:

• Users create files in wrong location
• Confusion about which files to edit vs. which are generated
• Risk of editing .lock.yml files directly (should be generated)

Recommendation:

• Add a file structure diagram showing:

  .github/
    workflows/
      my-workflow.md        ← Edit this
      my-workflow.lock.yml  ← Generated (don't edit)
  

• Clarify that .lock.yml is auto-generated
• Add warning: "⚠️ Never edit .lock.yml files manually"

---

🟡 Confusing Areas (Slow Down Learning)

5. "Add-Wizard" vs "Add" Command Confusion

What I found:
Quick Start uses gh aw add-wizard but other docs reference gh aw add

What's confusing:

• Are these the same command?
• When should I use add-wizard vs add?
• Is add-wizard deprecated or preferred?

Recommendation:

• Explain the difference between add and add-wizard in CLI docs
• Use one consistently in Quick Start (recommend add-wizard)
• Add alias note: "Tip: add-wizard is an interactive version of add"

6. Video Without Fallback Text

What I found:
Both homepage and Quick Start rely heavily on videos for key concepts.

What's confusing:

• What if the video doesn't load?
• What if I prefer reading over watching?
• No transcript or step-by-step text version

Recommendation:

• Always provide text fallback for video content
• Add video transcripts or step summaries
• Consider: "Prefer reading? See text version below ↓"

7. Secrets Setup Feels Like an Afterthought

What I found:
Quick Start says: "Set up required secrets" but doesn't show how.

What's confusing:

• Where exactly do I add secrets? (GitHub UI? CLI?)
• What should the secret names be exactly?
• Can I test if secrets are set correctly?
• What happens if I get the secret name wrong?

Impact:

• Users waste time trying to figure out secret setup
• Silent failures when secrets are misconfigured
• No way to debug secret issues

Recommendation:

• Add screenshot showing GitHub Secrets UI location
• Provide exact secret names: COPILOT_GITHUB_TOKEN, ANTHROPIC_API_KEY
• Add validation step: "Test your secret with: gh aw test-secrets"
• Link to troubleshooting for common secret errors

8. "Estimated Time" Doesn't Match Reality

What I found:
Quick Start says "⏱️ Estimated time: 10 minutes"

Reality for a beginner:

• Installing prerequisites: 10-15 min
• Setting up secrets: 5-10 min (if first time)
• Waiting for first workflow run: 2-3 min
• Debugging if something goes wrong: 15-60 min

Actual time: 30-60 minutes minimum for a complete beginner

Recommendation:

• Update to "⏱️ Estimated time: 30-60 minutes (first time)"
• Break down time estimates:
  • Prerequisites setup: 15 min
  • Workflow creation: 5 min
  • First run and verification: 10 min
  • Troubleshooting buffer: 15+ min

---

🟢 What Worked Well

Despite the issues, several things work great:

1. ✅ Clear structure: Introduction → Setup → Guides → Reference is logical
2. ✅ Good examples: The Daily Status Report example is concrete and useful
3. ✅ Multiple AI engines: Choice between Copilot, Claude, Codex is great
4. ✅ Visual design: Clean, professional layout (when links work)
5. ✅ Code blocks: Copy-paste ready commands with good syntax highlighting
6. ✅ Comprehensive reference: Extensive reference documentation (when not 404ing)

---

📊 Statistics

Build Status:

• ❌ Build fails with link validation enabled
• ✅ Build succeeds with validation disabled
• ⚠️ 263 broken links across 89 files

Pages Analyzed:

1. Homepage (index.mdx)
2. Quick Start (setup/quick-start.mdx)
3. Creating Workflows (setup/creating-workflows.mdx)
4. CLI Commands (setup/cli.md) - first 100 lines

Key Metrics:

• Broken links: 263
• Jargon without definitions: 10+
• Missing context sections: 5
• Video-only content: 2 critical sections

---

🎯 Recommendations by Priority

Immediate (This Week)

1. ⚠️ FIX BROKEN LINKS - All 263 of them
2. Add "Key Concepts" explainer before Quick Start
3. Update time estimates to be realistic
4. Add secrets setup screenshot and validation

Short-term (This Month)

1. Create glossary with inline links
2. Add file structure diagram
3. Add cost/pricing information
4. Provide video transcripts
5. Clarify add vs add-wizard commands

Long-term (This Quarter)

1. Build "Can I use this?" decision tree
2. Create "New to GitHub Actions?" primer
3. Add interactive troubleshooting guide
4. Build link health monitoring dashboard
5. Create beginner vs advanced tracks

---

💡 Quick Wins (Can Fix Today)

These would immediately help new users:

1. ✏️ Add this to Quick Start Prerequisites:

   **💰 Cost:** GitHub Copilot starts at $10/month for individuals. 
   Claude and Codex require API keys (pay-as-you-go).

2. ✏️ Add this warning to Creating Workflows:

   ⚠️ **Important:** Never edit `.lock.yml` files manually. 
   Always edit the `.md` file and run `gh aw compile`.

3. ✏️ Add this to homepage:

   **New to agentic workflows?** Start with our 
   [5-minute overview](./introduction/overview) or jump to 
   [Quick Start](./setup/quick-start).

4. 🔗 Fix the broken /gh-aw/troubleshooting/common-issues/ link

5. 📖 Add inline definitions:

   • "frontmatter (the YAML configuration at the top of the file)"
   • "lock file (auto-generated GitHub Actions workflow)"
   • "safe outputs (pre-approved write operations)"

---

🔧 Testing Limitations

Note: This test was conducted in a containerized environment where:

• Playwright browser automation could not connect to localhost
• Documentation was analyzed via source files and curl
• Visual screenshots could not be captured
• Interactive elements could not be tested

For complete testing, recommend:

• Manual testing on actual developer machines
• Screen recordings of first-time user experience
• Usability testing with 3-5 real beginners
• A/B testing different onboarding paths

---

Conclusion

GitHub Agentic Workflows has tremendous potential and excellent foundational content. However, the 263 broken links and missing context for beginners create significant barriers to entry.

The good news: Most issues are documentation problems, not product problems. Fixing links, adding definitions, and providing more context would dramatically improve the new user experience.

Key insight: The documentation is written for people who already understand GitHub Actions, AI agents, and DevOps concepts. For true beginners, more scaffolding and explanation is needed.

Bottom line: With focused effort on link fixing and beginner context, this could go from "confusing" to "excellent" relatively quickly.

---

Automated Report Generated by: Documentation Noob Testing Workflow
Repository: github/gh-aw
Workflow Run: #21934841923

---

Note: This was intended to be a discussion, but discussions could not be created due to permissions issues. This issue was created as a fallback.

AI generated by Documentation Noob Tester

• expires on Feb 19, 2026, 5:40 AM UTC