Skip to content

ci: lint and test demos/ and train/ instead of excluding them - #209

Merged
ASuresh0524 merged 3 commits into
mainfrom
ci-cover-demos
Nov 17, 2026
Merged

ASuresh0524 merged 3 commits into
mainfrom
ci-cover-demos

Conversation

@ASuresh0524

Copy link
Copy Markdown
Collaborator

The hole

pyproject.toml has:

[tool.ruff]
exclude = ["tmp", "train", "demos", "docs", "skill"]

and ci.yml runs uvx ruff check . (which honors that exclude) plus uv run pytest tests/ -v. So everything under demos/ and train/ was covered by neither lint nor tests.

That makes a green badge misleading rather than merely incomplete. #175 merged on 4/4 green; those four checks covered its README edit and none of its code. Every vendor demo PR lands in exactly that blind spot — #191 is open in the same shape right now — and both fix(train) PRs (#205, #206) are in the other half of it.

Lint

The hidden debt was small — 2 errors in demos/, 53 in train/, 51 auto-fixable (41 unsorted-imports, 5 explicit-f-string-type-conversion, 3 for-loop-writes, 2 regex-flag-alias). Four needed a judgment call:

  • train/.cursor/rules/code-critic.md is a dead symlink → .claude/agents/code-critic.md, which isn't in the repo. Tracked since 82c5794 "Initial release of PixelRAG" and invisible for as long as train/ was excluded. It doesn't just lint badly — it makes ruff format fail to read the path, so un-excluding train/ without removing it breaks CI outright. Removed.
  • train_contrastors.py appended in a loop → extend.
  • clean_queries_simpleqa_style.py validates a model's JSON response, not a caller's argument, so ValueError is the right contract and callers already catch it. TRY004 suppressed on the raise with the reason, rather than changing the exception type.
  • demos/quickstart.ipynb used sorted(...)[0] for a minimum.

ruff check . and ruff format --check . now pass across 194 files, with nothing excluded but tmp/, docs/ and skill/.

Tests

Each demo is its own uv project depending on the repo root via pixelrag = { path = "../.." }, which is the mechanical reason pytest tests/ never reached them. The new demo-tests job discovers every demos/*/ that has tests and runs it in its own environment.

demos/parallel_search is the only one today and it's fully offline — it monkeypatches httpx.AsyncClient and returns synthetic responses, no network and no API keys:

3 passed in 19.11s        (40 packages installed in 1.24s)

so this is close to free. The discovery loop skips demos without tests and uses nullglob so an empty demos/ can't iterate a literal glob.

Verification

  • uvx ruff check . → all checks passed; uvx ruff format --check . → 194 files already formatted.
  • uv run --project demos/parallel_search --group dev pytest → 3 passed.
  • pytest tests/ reports 22 collection errors in my checkout (ModuleNotFoundError: pixelrag_embed) — I confirmed the identical 22 on unmodified origin/main, so that's this checkout missing uv sync --extra dev, not this branch.
  • Every non-import hunk in the diff is one of the four rules above; I audited the staged diff for anything else and there is nothing.

Not included: eval/pyproject.toml, eval/uv.lock, train/README.md and two lockfiles were already modified in my working copy from other work. They're unstaged, not reverted, so nothing was lost — but they're deliberately out of this PR.

🤖 Generated with Claude Code

`[tool.ruff] exclude` listed "demos" and "train", and CI runs
`pytest tests/` only, so everything under those two directories was
covered by neither lint nor tests. A green CI badge on a PR that only
touched demos/ therefore attested to nothing: #175's 4/4 covered its
README edit and none of its code.

Drop both from the exclude list. The hidden debt was small: 2 errors in
demos/ and 53 in train/, 51 of them auto-fixable (41 unsorted-imports,
5 explicit-f-string-type-conversion, 3 for-loop-writes, 2
regex-flag-alias). The four that needed a decision:

- train/.cursor/rules/code-critic.md is a dead symlink pointing at
  .claude/agents/code-critic.md, which is not in the repo. Tracked since
  the initial release and invisible while train/ was excluded; it makes
  `ruff format` fail to read the path outright. Removed.
- train_contrastors.py appended in a loop; now extends.
- clean_queries_simpleqa_style.py validates a model's JSON response, so
  ValueError is the right contract for callers already catching it.
  TRY004 suppressed on the raise with the reason.
- demos/quickstart.ipynb used sorted(...)[0] to take a minimum.

Add a `demo-tests` job. Each demo is its own uv project depending on the
repo root through a path source, which is why `pytest tests/` never
reached them. The job discovers every demos/*/ with tests and runs it in
its own environment; demos/parallel_search is the only one today, and it
is offline (monkeypatched httpx.AsyncClient, no keys) — 3 passed in 19s
with a 1.2s install, so this costs almost nothing.

`ruff check .` and `ruff format --check .` now pass across 194 files
with nothing excluded but tmp/, docs/ and skill/.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
web Ready Ready Preview Nov 17, 2026 8:10am UTC

Removing the ruff exclusions brings train/ into scope for `ruff format`,
which reformats the fenced python example in train/README.md. Applying it
here so `ruff format --check .` passes with the wider scope.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pyproject edit here had grown a second subject: it moved aiohttp into the
kiwix extra, added botocore to distributed, and deleted the root eval extra.
Those are #208's change, not this one's, and carrying them here broke CI in a
way that was easy to misread as flaky.

Dependency edits invalidate uv.lock, so `uv sync --extra dev` re-resolved from
scratch on every run instead of installing from the lock. That re-resolution
fails on Linux:

    No solution found ... (python_full_version >= '3.14' ... sys_platform == 'linux')
    faiss-gpu-cu12{sys_platform == 'linux'}>=1.13.2 has no Linux-compatible wheels

a latent gap in the gpu extra that main never hits, because main's lock matches
main's pyproject and is installed as-is. Locally it looked fine only because
macOS resolves that split differently.

pyproject is now one line -- the ruff exclude -- which is what the title says.
`uv sync` leaves uv.lock untouched, 308 passed / 2 skipped, ruff check and
format clean over the wider 194-file scope.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ASuresh0524
ASuresh0524 merged commit 3e5f297 into main Nov 17, 2026
7 checks passed
ASuresh0524 added a commit that referenced this pull request Nov 17, 2026
#209 removed `train` from the ruff exclude list, and #205/#206 added these
two test files while it was still excluded, so they had never been checked.
Together on main they fail CI:

    train/tests/test_loss_world_size.py:26:33: RUF100 Unused `noqa` directive
    2 files would be reformatted

Each PR was green on its own; the combination was what nobody ran. Fixed with
`ruff check --fix` and `ruff format` over train/tests/ only.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
ASuresh0524 added a commit to esteininger/PixelRAG that referenced this pull request Nov 17, 2026
StarTrail-org#209 removed `demos` from the ruff exclude list, so this file is now linted.
It was written while demos/ was excluded, so this is the goalposts moving,
not a defect in the PR:

    demos/mixpeek/web_search.py:26:8: F401 `json` imported but unused
    demos/mixpeek/web_search.py:322:11: F541 f-string without any placeholders

Both auto-fixed, plus a `ruff format` pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 96a1a837 Deployed Nov 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant