You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Enhance test runner to detect and prioritize incomplete test runs
This commit adds incomplete test detection to perl_test_runner.pl to help
identify high-impact fix opportunities where tests crash before completion.
## Problem
Previously, when a test file crashed (e.g., pat_rt_report.t planning 2514
tests but crashing at test 193), the runner would only report:
- status: 'fail'
- ok_count: 193
This didn't highlight that 2,321 tests were BLOCKED and never ran, making
it hard to prioritize high-impact fixes.
## Solution
1. **Detect incomplete runs:** Parse '1..N' plan and compare to actual tests run
2. **Calculate blocked tests:** incomplete_tests = planned - actual_tests_run
3. **Count as failures:** Add incomplete_tests to not_ok_count for visibility
4. **Mark status:** Set status='incomplete' for special handling
5. **Prioritized display:** New section shows top incomplete files by blocked count
## Features Added
### Enhanced parse_tap_output():
- Tracks planned_tests from '1..N' line
- Tracks actual_tests_run from ok/not ok count
- Calculates incomplete_tests (blocked tests that never ran)
- Detects 'Looks like you planned N tests but ran M' message
- Adds incomplete_tests to not_ok_count for pass rate visibility
- Sets status='incomplete' when tests are blocked
### New print_incomplete_opportunities():
- Shows "HIGH-PRIORITY OPPORTUNITIES" section
- Lists incomplete tests sorted by blocked test count (descending)
- Displays: file name, blocked count, actual/planned tests
- Shows first error for each incomplete test
- Calculates total blocked tests across all files
### Updated print_summary():
- Calls print_incomplete_opportunities() after main summary
- Shows incomplete test count in file summary
- Pass rate now reflects blocked tests as failures
## Example Output
```
TEST SUMMARY:
Total files: 9
Passed: 7
Failed: 0
Errors: 0
Timeouts: 0
Incomplete: 2
Total tests: 502
OK: 435
Not OK: 67 # Includes 62 blocked tests!
Pass rate: 86.7%
🎯 HIGH-PRIORITY OPPORTUNITIES (Incomplete Test Runs):
(Tests that crashed/failed before completion - fixing these unlocks many tests at once)
Total incomplete tests: 2 files, 62 blocked tests
Top 2 incomplete test files (by blocked test count):
1. lex.t Blocked: 61 tests (68/129 ran)
Error: (?{...}) code blocks in regex not implemented...
2. term.t Blocked: 1 tests (6/7 ran)
```
## Impact
This enhancement helps prioritize fixes by highlighting:
- Which test files have the most blocked tests
- What errors caused tests to stop running
- How many tests could be unlocked by fixing each issue
For example, the $^R fix unlocked 2,266 tests in pat_rt_report.t by fixing
a single crash at test 193. This feature makes such opportunities visible.
## Files Modified
- dev/tools/perl_test_runner.pl: Added incomplete detection and display
0 commit comments