20260730v3/manifest collision and coverage - #1497
Merged
LLMZhangYC merged 5 commits intoJul 31, 2026
Merged
Conversation
Two operators share a file name with an operator in another directory, so the
derived test names collided:
examples/normalization/rms_norm.py -> test_rms_norm.py
examples/pos_embedding/rms_norm.py -> test_rms_norm.py
examples/pos_embedding/rope.py -> test_rope.py
examples/xllm_kernels/rope.py -> test_rope.py
Pytest imports a test module under its base name unless the directory is a
package, so the second one of a pair fails to collect once both files exist:
ERROR xllm_kernels/test_rope.py - import file mismatch:
imported module 'test_rope' has this __file__ attribute:
examples/pos_embedding/test_rope.py
which is not the same as the test file we want to collect:
examples/xllm_kernels/test_rope.py
That surfaces as an error rather than a failure, so the run reports "95 passed,
1 error" and still exits 1, pointing at a contributor who wrote nothing wrong.
The directory the operator belongs to keeps the plain name; the other one takes
a suffix. rms_norm belongs to normalization and rope to pos_embedding, so the
copies under pos_embedding and xllm_kernels are the ones renamed. The manifest
still holds 140 entries, now with 140 distinct base names.
docs/coverage_guide.md documents `bash bench_test.sh --coverage` as the way to produce the report. Registering an operator excludes it from the collection in this script, and the Pytest run that covers it instead lives in ci_cd.yml, which that command never goes through: the operator contributes nothing, and neither does its test. Measured on 910B, examples/batch_gemm/batch_gemm.py alone executes 2897 lines of tilelang, 563 of which a full testing/python run never reaches. They sit in jit/adapter/cython/adapter.py, jit/kernel.py, cache/kernel_cache.py, engine/phase.py and jit/adapter/libgen.py, so what drops out is the path that compiles a kernel and loads it. Three operators are registered today and the remaining 137 still cover those lines; once they migrate the report loses them. The tests run here only when coverage is on, so the normal path is untouched and ci_cd.yml keeps running them. --forked is unconditional: each test loads an example module and allocates on the device, unlike the testing/python run below which drops it under C++ coverage to avoid concurrent .gcda writes. COVERAGE_FILE lands in coverage_data/, which the combine step already globs.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
👋 Hi! Thank you for contributing to the TileLang project. Please remember to run We appreciate you taking this step! Our team will review your contribution, and we look forward to your awesome work! 🚀 |
…d them Two defects in the previous commit, both found by running the documented command against a single directory. The block sat after the early return taken when the collection finds nothing, so `--coverage --dirs <dir>` on a directory whose only operator is migrated skipped the tests it was added to run: the operator is excluded by name, the count reaches zero, and the script returns before ever getting there. It now runs before that return, which also means the tests compile against a cold cache instead of one the examples have already filled. Coverage does not follow a fork, so --forked discarded most of what these tests execute. Over the eleven registered tests it records 2237 lines of tilelang with it against 3285 without, and the loss is concentrated in exactly the path the report is missing: jit/kernel.py 45 against 110, cache/kernel_cache.py 48 against 125, engine/phase.py 12 against 58, jit/adapter/libgen.py 27 against 70. The xdist workers still give each test file its own process, and the wall time is unchanged at 51 seconds. Running them serially is not an alternative: pytest errors out on collection. Verified on 910B over four invocations, scoped to a migrated and to an unmigrated directory with coverage on and off. The block runs once with coverage and not at all without it, and coverage_data/ receives .coverage_operator_tests for the combine step to pick up.
Coverage does not follow a fork, so this run cannot use --forked, and without it the caching allocator keeps everything the tests reserve. Measured over three of the sparse attention kernels the reserve climbs 6.34, 18.19 and 49.21 GiB against a 61 GiB device while the allocation returns to zero each time: the tensors do go out of scope, it is the process ending that returns the memory. With three registered operators the single call fit. Forty five do not, and the seven sparse attention kernels are adjacent in manifest order, so any batch large enough to hold several of them exceeds the device. Three per batch leaves room for the heaviest run. Each batch writes its own coverage file. pytest-cov merges its workers into COVERAGE_FILE when it exits, so a shared name would have left only the last batch behind, and the combine step already globs for the rest. The summary is accumulated for the same reason: there is now one line per batch rather than one for the run.
Three per call still ran the device out. The batch was the wrong bound: with xdist those three go to three workers and run at once, and of the sparse attention kernels one reserves 31 GiB of a 61 GiB device by itself, so any pair of the heavy ones overlapping is enough. Two of the seven failed that way. Sequentially within a worker has the same ceiling for a different reason, since without a fork the reserve is only returned when the process ends. One file per call with no workers is the only arrangement where neither applies. Measured over the batched attempt: sixteen calls, 3580 lines of tilelang once combined against the 3285 an unbatched run of eleven reached, and the compile path intact at jit/kernel.py 113 and cache/kernel_cache.py 125. What that run lacked was the memory bound, not the coverage.
Author
|
/re-test |
|
🔄 Re-running failed jobs Original workflow run: View details Only the failed jobs will be re-executed. |
Author
|
/re-test |
|
🔄 Re-running failed jobs Original workflow run: View details Only the failed jobs will be re-executed. |
Author
|
/re-test |
|
🔄 Re-running failed jobs Original workflow run: View details Only the failed jobs will be re-executed. |
aynewsd
pushed a commit
to aynewsd/tilelang-ascend
that referenced
this pull request
Jul 31, 2026
Since PR tile-ai#1497 fixed the issue that test files with identical names under different subdirectories in example could not be distinguished, we reintroduce the test file with naming conflict: example/pos_embedding/test_rope.py.
aynewsd
pushed a commit
to aynewsd/tilelang-ascend
that referenced
this pull request
Jul 31, 2026
Since PR tile-ai#1497 fixed the issue that test files with identical names under different subdirectories in example could not be distinguished, we reintroduce the test file with naming conflict: example/pos_embedding/test_rope.py.
aynewsd
pushed a commit
to aynewsd/tilelang-ascend
that referenced
this pull request
Jul 31, 2026
iiiii fidsdfthe chsdasfsdfsadfasdsdfasfanges in ci/operator_test_manifest.yaml(tile-ai#1497), the file names corresponding to the name has been modified.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.