internal(test-selection): fingerprint the collected tests and ask after collection - #138
Merged
Conversation
Merge Protections🟢 All 6 merge protections satisfied — ready to merge. Show 6 satisfied protections🟢 🤖 Continuous Integration
🟢 👀 Review Requirements
🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
🟢 🔎 Reviews
🟢 📕 PR description
🟢 🚦 Auto-queueWhen all merge protections are satisfied, this pull request will be queued automatically. |
…er collection Mergify can only serve a reduced rerun when the rerun collects the same tests its predecessor did, and pytest-mergify gave it no way to know: the plugin asked for a selection from `pytest_configure`, before it had collected anything, so the request described the run's coordinates and nothing about its contents. A rerun whose collection had changed underneath it — a renamed file, a `-k` a developer added, a test suite that grew — would have been served a subset computed for a collection that no longer exists. Fingerprint the collection instead, and move the request behind it. The recipe is SHA-256 of each identifier, the digests sorted, SHA-256 of their concatenation — sorted, so collection order does not matter, and concatenated rather than XORed, so a repeated identifier is not a collection that cancels itself out. It lives in `mergify-ci-core` rather than in six lines of `hashlib` here because the same recipe is owed to the Vitest and Playwright clients, which bind that same crate, and a drift between three implementations of it is silent and permanent: nothing matches, the full suite runs, and the feature stops doing anything without a single error. The request now happens in `pytest_collection_modifyitems`, `trylast` as before, so the fingerprint covers what survives the user's own filters; it travels as a `collection_fingerprint` query parameter, and is reported with the run's spans as the `test.collection.fingerprint` resource attribute so the engine can persist it for the next run to be matched against. Under xdist neither happens. The request cannot: the controller never collects, so the hook never fires there, and workers were already excluded — which is what it amounted to before, the filtering hook having never run on the controller either, except the controller used to print a "✂️ Test selection" block claiming a subset it had applied to nothing (MRGFY-8632). The fingerprint is withheld too, and that one is a correctness point rather than a consequence: a worker collects the whole suite and runs a share of it, so every worker of a run would report the same identity over partial results, and a worker that died before uploading would leave its siblings carrying a complete-looking, failure-free identity for a suite a fifth of which never ran. Vitest and Playwright are untouched: they pass no fingerprint and the parameter is omitted rather than sent empty. Related to MRGFY-8614 Change-Id: I5f2f95e0762d62e4161aaa57567a9093eb7f8067
AlexandreGaubert
force-pushed
the
devs/alexandregaubert/devs/alexandregaubert/pytest-collection-fingerprint/fingerprint-collected-tests-ask-collection--5f2f95e0
branch
from
September 1, 2026 08:59
7d7871d to
f988cb1
Compare
Contributor
Author
Revision history
|
sileht
approved these changes
Sep 1, 2026
jd
approved these changes
Sep 1, 2026
Merge Queue Status
|
Merge Queue Status
This pull request spent 34 seconds in the queue, including 5 seconds running CI. Required conditions to merge
|
mergify
Bot
deleted the
devs/alexandregaubert/devs/alexandregaubert/pytest-collection-fingerprint/fingerprint-collected-tests-ask-collection--5f2f95e0
branch
September 1, 2026 09:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mergify can only serve a reduced rerun when the rerun collects the same tests
its predecessor did, and pytest-mergify gave it no way to know: the plugin asked
for a selection from
pytest_configure, before it had collected anything, sothe request described the run's coordinates and nothing about its contents. A
rerun whose collection had changed underneath it — a renamed file, a
-kadeveloper added, a test suite that grew — would have been served a subset
computed for a collection that no longer exists.
Fingerprint the collection instead, and move the request behind it. The recipe
is SHA-256 of each identifier, the digests sorted, SHA-256 of their
concatenation — sorted, so collection order does not matter, and concatenated
rather than XORed, so a repeated identifier is not a collection that cancels
itself out. It lives in
mergify-ci-corerather than in six lines ofhashlibhere because the same recipe is owed to the Vitest and Playwright clients, which
bind that same crate, and a drift between three implementations of it is silent
and permanent: nothing matches, the full suite runs, and the feature stops doing
anything without a single error. The
request now happens in
pytest_collection_modifyitems,trylastas before, sothe fingerprint covers what survives the user's own filters; it travels as a
collection_fingerprintquery parameter, and is reported with the run's spansas the
test.collection.fingerprintresource attribute so the engine canpersist it for the next run to be matched against.
Under xdist neither happens. The request cannot: the controller never collects,
so the hook never fires there, and workers were already excluded — which is what
it amounted to before, the filtering hook having never run on the controller
either, except the controller used to print a "✂️ Test selection" block claiming
a subset it had applied to nothing (MRGFY-8632). The fingerprint is withheld too,
and that one is a correctness point rather than a consequence: a worker collects
the whole suite and runs a share of it, so every worker of a run would report the
same identity over partial results, and a worker that died before uploading would
leave its siblings carrying a complete-looking, failure-free identity for a suite
a fifth of which never ran.
Vitest and Playwright are untouched: they pass no fingerprint and the parameter
is omitted rather than sent empty.
Related to MRGFY-8614