Skip to content

build: declare directly-imported deps, drop the dead root eval extra - #208

Merged
ASuresh0524 merged 1 commit into
StarTrail-org:mainfrom
dex0shubham:fix/declare-direct-deps
Nov 17, 2026
Merged

ASuresh0524 merged 1 commit into
StarTrail-org:mainfrom
dex0shubham:fix/declare-direct-deps

Conversation

@dex0shubham

Copy link
Copy Markdown
Contributor

Follow-up to #197. That PR removed core dependencies nothing imported; this one
comes from the opposite half of the same audit — every module-level third-party
import in the shipped packages (render/, embed/, index/, serve/, src/)
checked against what the extras actually declare.

Three imports were not declared where their importer lives. One of them breaks a
documented command.

1. aiohttp — a real break

serve/src/pixelrag_serve/zim_server_async.py:16 does from aiohttp import web
at module scope. aiohttp was declared only in the root eval extra, so the
invocation in that module's own docstring fails after
pip install 'pixelrag[kiwix]':

$ python -m pixelrag_serve.zim_server_async --zim wikipedia.zim --port 9454
  File ".../pixelrag_serve/zim_server_async.py", line 16, in <module>
    from aiohttp import web
ModuleNotFoundError: No module named 'aiohttp'

(reproduced on main in a clean --extra kiwix env; clean after this change.)

It belongs beside libzim in kiwix: that module imports both, and the sync
zim_server.py needs only libzim, so [kiwix] now covers both ZIM servers.

Whether a given install happened to work was down to luck — aiohttp can arrive
transitively through fsspec's optional http extra — which is exactly why it went
unnoticed.

2. botocore

index/src/pixelrag_index/distributed.py:24 imports botocore.exceptions
directly while only boto3 was declared. boto3 always pulls botocore in, so
nothing was broken here — but a direct import belongs in the declaration
rather than relying on a transitive one.

3. pandas (in eval/)

eval/lib/benchmarks.py:19 and eval/lib/simpleqa_data.py:19 import pandas
directly; it reached eval's venv only transitively via datasets
(eval/uv.lock:2171). Declared in eval/pyproject.toml, where those files live.

4. Dropping the root eval extra

  • nothing in the repo references pixelrag[eval] or --extra eval — no docs, CI,
    scripts or code;
  • it could not run the harness regardless: eval imports requests and tqdm at
    module scope and the extra listed neither;
  • eval/ is its own uv project (pixelrag-repro) with a complete dependency list,
    and eval/README.md documents uv sync --frozen from inside it;
  • it was the only place aiohttp was declared, which is how the gap in §1 hid.

So it is a stale duplicate — but removing an extra is user-facing, even one
that never worked. Flagging that explicitly rather than burying it: anyone with
pixelrag[eval] in a script would need to switch to the eval project, which is
the documented path and the only one that actually works. Say the word if you'd
rather keep it as an alias.

Verification

Per-extra, in fresh uv run --isolated environments:

check result
[kiwix] → import zim_server_async was ModuleNotFoundError on main, clean now
[distributed] → import index/distributed.py clean (as before)
full suite (--extra dev) 298 passed, 2 skipped

Extras left alone because their imports check out: playwright
(strategies/connection.py), pdf (pdf2image in backends/pdf.py), qdrant,
embed, serve, gpu. This PR changes no Python, so the suite count is main's.

Follow-up to StarTrail-org#197, from auditing every module-level third-party import in
the shipped packages against what the extras declare. Three of them were
not declared where their importer lives.

**aiohttp — a real break, not tidying.** zim_server_async.py does
`from aiohttp import web` at module scope, but aiohttp was declared only
in the root `eval` extra. So the documented invocation in its own
docstring (`python -m pixelrag_serve.zim_server_async --zim ... --port`)
failed after `pip install 'pixelrag[kiwix]'`:

    File ".../pixelrag_serve/zim_server_async.py", line 16, in <module>
        from aiohttp import web
    ModuleNotFoundError: No module named 'aiohttp'

It belongs with libzim in `kiwix`: that module imports both, and the sync
zim_server.py needs only libzim, so `[kiwix]` now covers both servers.

**botocore.** index/distributed.py imports botocore.exceptions directly
while only boto3 was declared. boto3 always pulls botocore in, so nothing
was broken — but a direct import belongs in the declaration.

**pandas (eval/).** eval/lib/benchmarks.py and simpleqa_data.py import
pandas directly; it reached eval's venv only transitively through
datasets. Declared in eval/pyproject.toml, where those files live.

**Dropped the root `eval` extra.** Nothing in the repo references
`pixelrag[eval]` or `--extra eval`, and it could not run the harness
anyway: eval imports requests and tqdm at module scope and the extra
listed neither. eval/ is its own uv project (pixelrag-repro) and
eval/README documents `uv sync --frozen` from inside it, so the extra was
a stale duplicate — and the only place aiohttp was declared, which is how
the gap above hid. It is a user-facing removal of an extra that never
worked.

Verified per extra in fresh `uv run --isolated` envs: `[kiwix]` now
imports zim_server_async cleanly (it raised ModuleNotFoundError on main),
`[distributed]` imports index/distributed.py, and the suite is 298 passed
/ 2 skipped. The other extras are backed by real imports and untouched —
playwright (strategies/connection.py), pdf (pdf2image in backends/pdf.py),
qdrant, embed, serve, gpu.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 6, 2026

Copy link
Copy Markdown

@dex0shubham is attempting to deploy a commit to the andylizf's projects Team on Vercel.

A member of the Team first needs to authorize it.

@ASuresh0524

Copy link
Copy Markdown
Collaborator

The aiohttp finding is a genuine break and well diagnosed — zim_server_async.py:16 imports it at module scope while it's declared only in the root eval extra, so the command in that module's own docstring fails after pip install 'pixelrag[kiwix]'. That part I'd take.

The lockfile changes need another look before this lands. The diff drops platform markers from a lot of entries, for example:

- { name = "cuda-pathfinder", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }
+ { name = "cuda-pathfinder" }

and the same for cuda-bindings, cuda-core, nvidia-cublas-cu12, nvidia-cusparse-cu12, nvidia-nvjitlink-cu12. Unmarked, those become resolvable on Windows, where they previously were not.

I checked whether this was just lock drift from a newer uv, and it isn't: running uv lock on main with uv 0.10.8 leaves the marker count unchanged. So something about how this lock was regenerated differs from what the repo currently produces — a different uv version or different resolution settings seems most likely.

Could you re-run uv lock with uv 0.10.8 on top of the pyproject.toml changes and push the result? That should shrink the diff to the dependency declarations the PR is actually about, and make the -565 lines reviewable. Splitting the aiohttp fix out so it can land on its own would also be fine by me.

ASuresh0524 added a commit that referenced this pull request Nov 17, 2026
The pyproject edit here had grown a second subject: it moved aiohttp into the
kiwix extra, added botocore to distributed, and deleted the root eval extra.
Those are #208's change, not this one's, and carrying them here broke CI in a
way that was easy to misread as flaky.

Dependency edits invalidate uv.lock, so `uv sync --extra dev` re-resolved from
scratch on every run instead of installing from the lock. That re-resolution
fails on Linux:

    No solution found ... (python_full_version >= '3.14' ... sys_platform == 'linux')
    faiss-gpu-cu12{sys_platform == 'linux'}>=1.13.2 has no Linux-compatible wheels

a latent gap in the gpu extra that main never hits, because main's lock matches
main's pyproject and is installed as-is. Locally it looked fine only because
macOS resolves that split differently.

pyproject is now one line -- the ruff exclude -- which is what the title says.
`uv sync` leaves uv.lock untouched, 308 passed / 2 skipped, ruff check and
format clean over the wider 194-file scope.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ASuresh0524

Copy link
Copy Markdown
Collaborator

Correcting my previous comment — I was wrong about the lockfile, and the PR is fine.

I said the dropped platform markers looked like they came from locking with a different uv version, and asked you to re-lock. I tested that the wrong way round: I checked whether uv lock changes the markers on main (it doesn't) and concluded the diff here must therefore be environmental. What I should have done is regenerate the lock from this branch's pyproject.toml.

Doing that now, on top of main:

$ uv lock
$ cmp uv.lock <the lock as committed here>
IDENTICAL

Byte-identical. So the marker removal is exactly what uv produces for this dependency set, not drift — and it follows from the change itself: dropping the root eval extra shrinks the resolution universe, and the sys_platform markers that existed to keep that extra satisfiable go with it. Nothing to re-lock. Sorry for sending you after a non-problem.

Verified the rest on the merge with current main:

  • uv sync --extra dev leaves uv.lock untouched — the lock and the manifest agree, which is the property that actually matters.
  • ruff check and ruff format --check clean.
  • uv run pytest tests/ — 308 passed, 2 skipped.

The aiohttp diagnosis stands on its own and is worth the PR by itself.

One related note, since it bit elsewhere: #209 had quietly grown a copy of these same dependency edits without a matching lock update, which made uv sync re-resolve on every CI run and fail on Linux (faiss-gpu-cu12 has no cp314 Linux wheel). I've stripped those from #209 so it's the ruff-scope change only. This PR is the right home for them, and it does them properly — with the lock.

@ASuresh0524
ASuresh0524 merged commit 46cd8c5 into StarTrail-org:main Nov 17, 2026
4 of 5 checks passed
ASuresh0524 added a commit that referenced this pull request Nov 17, 2026
* ci: lint and test demos/ and train/ instead of excluding them

`[tool.ruff] exclude` listed "demos" and "train", and CI runs
`pytest tests/` only, so everything under those two directories was
covered by neither lint nor tests. A green CI badge on a PR that only
touched demos/ therefore attested to nothing: #175's 4/4 covered its
README edit and none of its code.

Drop both from the exclude list. The hidden debt was small: 2 errors in
demos/ and 53 in train/, 51 of them auto-fixable (41 unsorted-imports,
5 explicit-f-string-type-conversion, 3 for-loop-writes, 2
regex-flag-alias). The four that needed a decision:

- train/.cursor/rules/code-critic.md is a dead symlink pointing at
  .claude/agents/code-critic.md, which is not in the repo. Tracked since
  the initial release and invisible while train/ was excluded; it makes
  `ruff format` fail to read the path outright. Removed.
- train_contrastors.py appended in a loop; now extends.
- clean_queries_simpleqa_style.py validates a model's JSON response, so
  ValueError is the right contract for callers already catching it.
  TRY004 suppressed on the raise with the reason.
- demos/quickstart.ipynb used sorted(...)[0] to take a minimum.

Add a `demo-tests` job. Each demo is its own uv project depending on the
repo root through a path source, which is why `pytest tests/` never
reached them. The job discovers every demos/*/ with tests and runs it in
its own environment; demos/parallel_search is the only one today, and it
is offline (monkeypatched httpx.AsyncClient, no keys) — 3 passed in 19s
with a 1.2s install, so this costs almost nothing.

`ruff check .` and `ruff format --check .` now pass across 194 files
with nothing excluded but tmp/, docs/ and skill/.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* style: format the python block in train/README.md

Removing the ruff exclusions brings train/ into scope for `ruff format`,
which reformats the fenced python example in train/README.md. Applying it
here so `ruff format --check .` passes with the wider scope.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* build: drop the dependency changes; this PR is the lint scope only

The pyproject edit here had grown a second subject: it moved aiohttp into the
kiwix extra, added botocore to distributed, and deleted the root eval extra.
Those are #208's change, not this one's, and carrying them here broke CI in a
way that was easy to misread as flaky.

Dependency edits invalidate uv.lock, so `uv sync --extra dev` re-resolved from
scratch on every run instead of installing from the lock. That re-resolution
fails on Linux:

    No solution found ... (python_full_version >= '3.14' ... sys_platform == 'linux')
    faiss-gpu-cu12{sys_platform == 'linux'}>=1.13.2 has no Linux-compatible wheels

a latent gap in the gpu extra that main never hits, because main's lock matches
main's pyproject and is installed as-is. Locally it looked fine only because
macOS resolves that split differently.

pyproject is now one line -- the ruff exclude -- which is what the title says.
`uv sync` leaves uv.lock untouched, 308 passed / 2 skipped, ruff check and
format clean over the wider 194-file scope.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants