Skip to content

Add skill metadata validation with CI, and fix 65 invalid skill names - #223

Open
Chirag6722 wants to merge 2 commits into
composio-community:masterfrom
Chirag6722:feature/skill-metadata-validation-ci
Open

Add skill metadata validation with CI, and fix 65 invalid skill names#223
Chirag6722 wants to merge 2 commits into
composio-community:masterfrom
Chirag6722:feature/skill-metadata-validation-ci

Conversation

@Chirag6722

Copy link
Copy Markdown

Closes #222

Why

The catalog accepts community skills through PR review alone. There is no script defining what a valid skill is and no CI job running one, so metadata drift is invisible until a user installs a broken skill. Codex discovers a skill by reading the YAML frontmatter in SKILL.md, so a name that is not the directory's slug means the skill is registered under an identifier that does not match the folder users install — or fails to load.

That drift has already happened: 65 skills ship an invalid name: — 64 Title Case values under composio-skills/ (name: Ahrefs Automation) and video-downloader/, which declares name: youtube-downloader.

What this PR adds

scripts/validate_skills.py — stdlib-only, Python 3.10+, runs in ~0.4s over all 880 skills:

python scripts/validate_skills.py          # human-readable report
python scripts/validate_skills.py --json   # machine-readable
python scripts/validate_skills.py --strict # warnings fail too

Errors (exit 1):

  • SKILL.md missing, or not starting with a YAML frontmatter block
  • name missing, empty, or not equal to the directory's lowercase hyphenated slug
  • description missing, empty, or over 1024 characters (descriptions are loaded into context every session)
  • a top-level skill directory not linked from the README index, or a README skill link pointing at a path that does not exist

Warnings (reported, do not fail): the 28 directories whose own names use underscores or a leading hyphen (zoho_mail-automation, -2chat-automation). Renaming those directories breaks the install path for anyone who already has the skill, so that is deliberately left as a follow-up decision for maintainers rather than folded into this PR.

composio-skills/ is treated as a collection whose children are the skills; scripts/ and tests/ are excluded.

tests/test_validate_skills.py — 21 pytest cases building temporary skill trees, one per rule (valid tree, missing SKILL.md, missing frontmatter, each required field missing/empty, name/directory mismatch, Title Case name, non-slug directory downgraded to a warning, oversized vs. exactly-at-limit description, README desync in both directions, collection traversal, exit codes, JSON output, frontmatter parsing of quotes/block scalars/nested keys) plus a test asserting the real catalog satisfies its own rules.

.github/workflows/validate-skills.yml — runs the validator and the tests on every PR and on pushes to master.

Fixes for the existing violations, so the repo lands green: 65 name: fields normalised to their directory slugs (a one-line change per file, nothing else touched), composio-skills/ indexed in the README under a new App Integrations heading, and a "Validate before you open a PR" section added to Contributing.

Verification

$ python scripts/validate_skills.py
OK: 880 skills validated, 28 warning(s), no errors.

$ python -m pytest tests -q
21 passed in 0.43s

Before this PR the same command reported 66 errors.

🤖 Generated with Claude Code

The catalog accepts community skills through PR review alone, with no
script or CI job defining what a valid skill looks like. Codex discovers
skills by reading the YAML frontmatter in SKILL.md, so a malformed name
means the skill loads under the wrong identifier or not at all - and 65
skills had already drifted past review.

- scripts/validate_skills.py: stdlib-only validator for SKILL.md
  frontmatter (present, name is a slug matching the directory,
  description non-empty and within 1024 chars) and README index
  consistency, with --json and --strict.
- tests/test_validate_skills.py: pytest coverage per rule, plus a test
  asserting the real catalog satisfies its own rules.
- .github/workflows/validate-skills.yml: run both on PRs and master.
- Normalise 64 Title Case names under composio-skills/ to their
  directory slugs, and video-downloader (was youtube-downloader).
- Index composio-skills/ in the README and document the local check.

The 28 skill directories whose own names use underscores or a leading
hyphen are reported as warnings rather than errors, since renaming a
directory breaks the install path for anyone who already has the skill.

Closes composio-community#222

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A bulk import added 26 Composio toolkits twice, once under the raw toolkit
id and once under a hyphenated slug (zoho_mail-automation alongside
zoho-mail-automation), and nothing noticed. Flag directories in the same
collection whose names differ only in separators.

Reported as a warning rather than an error because those 26 duplicates are
still in the tree; composio-community#227 removes them, after which this can be promoted to
an error.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Chirag6722

Copy link
Copy Markdown
Author

Pushed a follow-up commit adding a uniqueness rule to the validator: it flags directories in the same collection whose names differ only in separators (zoho_mail-automation vs zoho-mail-automation).

It currently surfaces 26 such pairs, all from the bulk import in 7404a6c — see #227 for the details, including three pairs whose two files give contradictory instructions about the same toolkit. #228 removes them.

The rule is a warning rather than an error for now, so this PR stays green while those duplicates are still in the tree; once #228 lands it can be promoted to an error. Validator output on this branch:

OK: 880 skills validated, 54 warning(s), no errors.

Tests are now at 23 passing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No automated validation for skill submissions: 65 skills ship invalid name: frontmatter

1 participant