feat: Automate llms.txt Generation via GitHub Action

[canburaks]

Summary

This PR introduces an automated process using GitHub Actions to generate and maintain llms.txt and llms-full.txt based on the SDK's Markdown documentation.

This follows prior discussion on email regarding this feature. Evaluating the suggestion to utilize the existing Speakeasy generation process revealed that implementing it directly could introduce unnecessary complexity and maintenance overhead compared to a dedicated solution.

Description

This PR introduces an automated process for generating and updating llms.txt and llms-full.txt files based on the project's documentation. This ensures these files remain accurate and up-to-date with minimal manual intervention, improving the project's discoverability and usability for AI agents and language models.

Changes Introduced

1. Python Generation Script (.github/scripts/generate_llms_files.py):

   • Parses project metadata (name, description) from pyproject.toml.
   • Recursively scans a specified directory (e.g., docs/sdks) for Markdown (.md) files.
   • Generates llms.txt:
     • Includes project name (H1) and description (blockquote).
     • Adds a dynamically named section (e.g., ## Sdks Files) based on the source directory.
     • Lists links to discovered .md files.
     • Includes an optional link to llms-full.txt.
     • Generates absolute raw GitHub URLs using a provided --base-url or relative links otherwise.
   • Generates llms-full.txt:
     • Includes the base content from llms.txt.
     • Appends the full content of each discovered .md file under a ## Content: <filepath> heading.
     • Adjusts heading levels within the appended content (H1 -> H3, H2 -> H4, etc.) to maintain correct Markdown hierarchy.

2. GitHub Actions Workflow (.github/workflows/generate-llms-txt.yml):

   • Triggers: Automatically runs on pushes to main affecting docs/sdks/**, pyproject.toml, the script, or the workflow itself. Also supports manual triggers (workflow_dispatch).
   • Execution:
     • Sets up Python 3.10 and installs the toml dependency.
     • Constructs the correct base URL for raw GitHub file content (using github.ref to include refs/heads/...).
     • Executes the Python generation script, passing the source directory (docs/sdks) and the constructed base URL.
   • Auto-Commit: Uses stefanzweifel/git-auto-commit-action to automatically commit changes only to llms.txt and llms-full.txt directly to the branch, using the github-actions[bot] user.

Benefits

• Automation: Eliminates the need to manually update llms.txt files when documentation changes.
• Accuracy: Ensures generated files always reflect the current state of the documentation and project metadata.
• Consistency: Guarantees adherence to the llms.txt specification and correct link/heading formatting.
• Maintainability: Centralizes generation logic in a dedicated script.

How to Verify

• Pushing changes to files within docs/sdks/ on the main branch should trigger the action and result in an automatic commit updating llms.txt and llms-full.txt if necessary.
• The workflow can be manually triggered via the Actions tab on GitHub by appending the correct branch name to .github/workflows/generate-llms-txt.yml file under on -> push -> paths.

---

Happy to make any necessary changes based on feedback!