Add skill metadata validation with CI, and fix 65 invalid skill names - #223
Open
Chirag6722 wants to merge 2 commits into
Open
Add skill metadata validation with CI, and fix 65 invalid skill names#223Chirag6722 wants to merge 2 commits into
Chirag6722 wants to merge 2 commits into
Conversation
The catalog accepts community skills through PR review alone, with no script or CI job defining what a valid skill looks like. Codex discovers skills by reading the YAML frontmatter in SKILL.md, so a malformed name means the skill loads under the wrong identifier or not at all - and 65 skills had already drifted past review. - scripts/validate_skills.py: stdlib-only validator for SKILL.md frontmatter (present, name is a slug matching the directory, description non-empty and within 1024 chars) and README index consistency, with --json and --strict. - tests/test_validate_skills.py: pytest coverage per rule, plus a test asserting the real catalog satisfies its own rules. - .github/workflows/validate-skills.yml: run both on PRs and master. - Normalise 64 Title Case names under composio-skills/ to their directory slugs, and video-downloader (was youtube-downloader). - Index composio-skills/ in the README and document the local check. The 28 skill directories whose own names use underscores or a leading hyphen are reported as warnings rather than errors, since renaming a directory breaks the install path for anyone who already has the skill. Closes composio-community#222 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 12, 2026
A bulk import added 26 Composio toolkits twice, once under the raw toolkit id and once under a hyphenated slug (zoho_mail-automation alongside zoho-mail-automation), and nothing noticed. Flag directories in the same collection whose names differ only in separators. Reported as a warning rather than an error because those 26 duplicates are still in the tree; composio-community#227 removes them, after which this can be promoted to an error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Author
|
Pushed a follow-up commit adding a uniqueness rule to the validator: it flags directories in the same collection whose names differ only in separators ( It currently surfaces 26 such pairs, all from the bulk import in The rule is a warning rather than an error for now, so this PR stays green while those duplicates are still in the tree; once #228 lands it can be promoted to an error. Validator output on this branch: Tests are now at 23 passing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #222
Why
The catalog accepts community skills through PR review alone. There is no script defining what a valid skill is and no CI job running one, so metadata drift is invisible until a user installs a broken skill. Codex discovers a skill by reading the YAML frontmatter in
SKILL.md, so anamethat is not the directory's slug means the skill is registered under an identifier that does not match the folder users install — or fails to load.That drift has already happened: 65 skills ship an invalid
name:— 64 Title Case values undercomposio-skills/(name: Ahrefs Automation) andvideo-downloader/, which declaresname: youtube-downloader.What this PR adds
scripts/validate_skills.py— stdlib-only, Python 3.10+, runs in ~0.4s over all 880 skills:Errors (exit 1):
SKILL.mdmissing, or not starting with a YAML frontmatter blocknamemissing, empty, or not equal to the directory's lowercase hyphenated slugdescriptionmissing, empty, or over 1024 characters (descriptions are loaded into context every session)Warnings (reported, do not fail): the 28 directories whose own names use underscores or a leading hyphen (
zoho_mail-automation,-2chat-automation). Renaming those directories breaks the install path for anyone who already has the skill, so that is deliberately left as a follow-up decision for maintainers rather than folded into this PR.composio-skills/is treated as a collection whose children are the skills;scripts/andtests/are excluded.tests/test_validate_skills.py— 21 pytest cases building temporary skill trees, one per rule (valid tree, missingSKILL.md, missing frontmatter, each required field missing/empty, name/directory mismatch, Title Case name, non-slug directory downgraded to a warning, oversized vs. exactly-at-limit description, README desync in both directions, collection traversal, exit codes, JSON output, frontmatter parsing of quotes/block scalars/nested keys) plus a test asserting the real catalog satisfies its own rules..github/workflows/validate-skills.yml— runs the validator and the tests on every PR and on pushes tomaster.Fixes for the existing violations, so the repo lands green: 65
name:fields normalised to their directory slugs (a one-line change per file, nothing else touched),composio-skills/indexed in the README under a new App Integrations heading, and a "Validate before you open a PR" section added to Contributing.Verification
Before this PR the same command reported 66 errors.
🤖 Generated with Claude Code