TEST: deliberate regression to verify CodSpeed comparison (do not merge)#3080
Closed
Pierre-Sassoulas wants to merge 3 commits into
Closed
TEST: deliberate regression to verify CodSpeed comparison (do not merge)#3080Pierre-Sassoulas wants to merge 3 commits into
Pierre-Sassoulas wants to merge 3 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## codspeed-wizard-1774989768270 #3080 +/- ##
==============================================================
Coverage 93.56% 93.56%
==============================================================
Files 92 92
Lines 11345 11350 +5
==============================================================
+ Hits 10615 10620 +5
Misses 730 730
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Merging this PR will degrade performance by 40.2%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
|
Pierre-Sassoulas
changed the title
Pierre-Sassoulas
force-pushed
the
perf/bench-endtoend-primer
branch
from
31c248a to
a5d48a0
Compare
Pierre-Sassoulas
changed the title
Pierre-Sassoulas
force-pushed
the
codspeed-wizard-1774989768270
branch
from
9986de3 to
28184b1
Compare
Pierre-Sassoulas
force-pushed
the
perf/bench-endtoend-primer
branch
from
a5d48a0 to
a07e875
Compare
Pierre-Sassoulas
force-pushed
the
codspeed-wizard-1774989768270
branch
from
28184b1 to
f8520dd
Compare
Pierre-Sassoulas
force-pushed
the
perf/bench-endtoend-primer
branch
from
a07e875 to
3e25126
Compare
Pierre-Sassoulas
force-pushed
the
codspeed-wizard-1774989768270
branch
from
f8520dd to
2d391a6
Compare
Three benchmarks based on pylint's primer corpus, with SHAs pinned to match what pylint sees nightly: - test_bench_endtoend_parse_flask (rebuilder hot path, ~50 .py) - test_bench_endtoend_walk_infer_flask (pylint-shaped traversal) - test_bench_endtoend_walk_infer_black (medium scale, ~250 .py) The benchmarks call astroid's API directly (ast_from_file + nodes_of_class + safe_infer on Call/Attribute/Name) so they mirror what a pylint checker does without depending on pylint. The fixture uses 'git clone --no-checkout --filter=blob:none --no-tags --single-branch' plus 'sparse-checkout' so only the declared source subdirs are fetched. Workflow uses 'mode: simulation' so the CodSpeed dashboard provides per-function attribution; 'CodSpeedHQ/action' is allowlisted in repo Actions settings. conftest.py uses 'collect_ignore_glob' to skip the directory cleanly when pytest-codspeed is not installed locally. Drops the 19 microbenchmarks the CodSpeed wizard generated: at microsecond scale they showed >40 % StdDev on CI, too noisy for regression detection, and their coverage is subsumed by the end-to-end benches. Local smoke (one iter each, no --codspeed): 3 passed in 11.6s. Supersedes #3022. Refs #2014.
Pierre-Sassoulas
force-pushed
the
codspeed-wizard-1774989768270
branch
from
2d391a6 to
2ef6d50
Compare
Adds ~5-10 µs of busywork (sum(range(1000))) at the very top of every NodeNG.infer() call. Since infer() is called O(1M) times during a pylint-shaped traversal, this adds several seconds to each end-to-end benchmark. The microbenchmarks should also show a clear regression. Purpose: verify that CodSpeed CI reports the perf delta vs the baseline (the merged-ahead state of #3079) and renders it correctly on the PR. This is an experiment, not a real change. The branch will be discarded after the comparison is observed. Refs #2014 (the profile-first feasibility study).
Pierre-Sassoulas
force-pushed
the
perf/bench-endtoend-primer
branch
from
3e25126 to
19f5c5b
Compare
… MERGE Eagerly imports nine heavyweight stdlib modules (pydoc, multiprocessing, xmlrpc.server, xml.dom.minidom, unittest.mock, email.mime.multipart, wsgiref.simple_server, http.server, concurrent.futures) at the top of astroid/__init__.py. Local measurement: `import astroid` goes from ~76 ms to ~198 ms (+120 ms per cold start). The cold-lint bench (test_bench_endtoend_cold_lint) shells out `python -m pylint` per iteration and re-pays import cost every run, so it should be the loudest reporter; the in-process benches import once at module load and will only see this on the first iteration. Purpose: verify that CodSpeed's walltime workflow (cold-lint) reports the startup delta vs the baseline (#3079) and that it shows up distinctly from the in-process `infer()` regression added in 19f5c5b. This is an experiment, not a real change. The branch will be discarded after the comparison is observed. Refs #2014 (the profile-first feasibility study).
Pierre-Sassoulas
force-pushed
the
codspeed-wizard-1774989768270
branch
2 times, most recently
from
2fb0a52 to
7e482f9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR was created to verify that the CodSpeed CI pipeline added in #3079 catches, measures, and reports performance regressions on a real PR.