Skip to content

Add SnippetValidator tool to check code example introductions match referenced snippets #11720

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Aug 22, 2025

This PR introduces a new validation tool that automatically checks if code example introductions in XML documentation files accurately match the content of their referenced code snippets.

Problem

The .NET API documentation contains thousands of code examples with introductory text that describes what the code demonstrates. However, there's currently no automated way to verify that these introductions actually match the code being shown. This can lead to documentation inconsistencies where:

  • Introduction mentions LoadAsync but the snippet uses synchronous Load()
  • Documentation describes synchronous operations but the code shows async patterns
  • Introduction references methods or classes not actually demonstrated in the snippet

Solution

The SnippetValidator tool provides automated validation by:

  1. Scanning XML files for snippet references like :::code language="csharp" source="~/snippets/..." id="Snippet1":::
  2. Extracting introduction text from the lines preceding each snippet reference
  3. Parsing snippet files to extract the actual code within specific snippet boundaries (<snippet1> to </snippet1>)
  4. Analyzing mismatches using intelligent pattern matching to identify legitimate inconsistencies

Usage

The tool supports multiple operation modes:

# Quick demonstration on sample files
dotnet run -- --sample

# Test specific known files
dotnet run -- --test

# Full repository validation (18,000+ XML files)
dotnet run

Example Issues Found

The tool successfully identifies real documentation problems:

LoadAsync vs Load Method Mismatch:

  • File: System.Media/SoundPlayer.xml:348
  • Issue: Introduction mentions LoadAsync method but snippet uses synchronous Load() method

Async/Sync Operation Mismatch:

  • File: System.Media/SoundPlayer.xml:855
  • Issue: Introduction mentions synchronous PlaySync operations but snippet contains async loading code

Features

  • Smart filtering to minimize false positives while catching real issues
  • Detailed reporting with file names, line numbers, and specific mismatch descriptions
  • Scalable processing with progress indicators for large-scale validation
  • Multiple output formats suitable for CI/CD integration

This tool helps maintain documentation quality by automatically detecting when code example descriptions drift from their actual implementations.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@Copilot Copilot AI changed the title [WIP] Check if code example introductions match what's actually shown in the referenced code snippet. To do that, you'll have to look inside the reference code files. If an ID is referenced in the code file, then check that that specific snippet ID matches t... Add SnippetValidator tool to check code example introductions match referenced snippets Aug 22, 2025
@Copilot Copilot AI requested a review from gewarren August 22, 2025 00:21
Copilot finished work on behalf of gewarren August 22, 2025 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants