The test runner over ctest (mitcad#790). It runs a build tree's ctests,
each as its own ctest --test-dir <build> -R ^name$ -FA .*, so that the
tests' commands, environments, timeouts and labels stay as ctest defines
them, and:
- measures every test of every run: wall time, CPU time (user and system) of the whole process tree, hence the mean busy cores, peak memory, and the kind of result (passed, skipped, failed, timeout, killed for memory, environment, not run);
- keeps the machine's profiles between runs: the last 5 runs of each test, its failure score (+10 for a failure, +2 for an environment failure, -1 for a pass, never below 0) and whether it is timing-sensitive (it failed under a load of 70 % or more after passing before; 5 passes under load clear the mark);
- learns from the same runs how many UI entries the machine runs at once without one disturbing another (below);
- either runs the tests at a fixed parallelism and only records
(
--measure, the default), or schedules them (--schedule).
It is built with the project, beside the build's other programs:
build\dev\mitcad-testrun.exe on Windows, build/dev/core/mitcad-testrun
on Linux and macOS (a member of the core workspace). Its scheduling,
profiles and parsers are tested by the ctest tools.testrun.
On Linux and macOS:
# Measuring at the parallelism of today's runs (records only):
build/dev/core/mitcad-testrun --build build/dev --measure -j 3
# Scheduling by memory and CPU:
build/dev/core/mitcad-testrun --build build/dev --schedule
# On bare metal, by the memory the machine has free (less a reserve):
build/dev/core/mitcad-testrun --build build/dev --schedule --memory-budget free --memory-reserve 6G
# What would run, in which order, with what predictions (runs nothing):
build/dev/core/mitcad-testrun --build build/dev --schedule --dry-run
# A selection, as ctest takes it:
build/dev/core/mitcad-testrun --build build/dev --measure -R '^core\.' -E corpusOn Windows (the VM), from a shell where ctest is on PATH (or with
--ctest <path to ctest.exe>); the UI ctests need the build configured
with MITCAD_MESA_DIR as before:
tools\dev-env\msvc.cmd build\dev\mitcad-testrun.exe --build build\dev --schedule--help lists every option. The run writes into <build>/testrun
(--out): logs/<test>.log (ctest's output of each test),
status.txt (a line per failure as it happens: kind, test, priority lane, log, note; then finished), report.txt (the end report) and
results.tsv (a row per test: result, start, wall, CPU, peak, limit,
prediction, lane). The exit status is 0 when every test passed or was
skipped, 1 otherwise, 2 for a usage or listing error.
- Windows: each ctest starts suspended, goes into a Job Object of its
own and then runs; every process it starts stays in the job (ctest's own
job for its test nests in it). The job gives the tree's total CPU time
and peak committed memory (
PeakJobMemoryUsed), holds the hard memory limit (JOB_OBJECT_LIMIT_JOB_MEMORY, reported through a completion port), and kills what is left of the tree when the test ends. - Linux: each ctest runs in a systemd scope of its own (
systemd-run --user --scope, cgroup v2) when the user's systemd is there (--no-cgroupturns it off): the scope's working set (memory.currentless the page cache,memory.stat'sactive_fileandinactive_file) is sampled while the test runs and its largest is the peak, itsmemory.eventsOOM kills mark a test killed for memory, andMemoryMaxis the hard limit. Notmemory.peak: it counts the cache of every file the test wrote or read, which the kernel reclaims under the limit (a fresh cargo test build showed a 16 GiB peak). The CPU time comes fromwait4's rusage of ctest and the children it waited for. Without a scope the peak is the largest single process's (ru_maxrss). The scopes go into the slice of the user's systemd that the runner itself runs in (--slice=, the last*.slicebelowuser@<uid>.servicein/proc/self/cgroup), not into the user's default slice: a runner started in a job slice with a memory ceiling, or in the slice of its own thattools/check-all.shmakes for each run, keeps its tests under that ceiling. The memory the budget is a share of is the lowest limit (memory.max, ormemory.highwhere lower) on the runner's cgroup path, from its own scope up, or the machine's memory; the memory available to a test is the machine'sMemAvailable, or less where a cgroup on that path is nearer its limit (its limit less its working set:memory.currentless the page cache ofmemory.stat), so that other jobs in a shared slice count too. - macOS:
libprocsamples the sum of resident memory of the test's process group and descendants. PID plus process start time identifies ownership, including observed detached/orphaned children; PID reuse cannot transfer it. Live memory greater than the prediction reserves that larger amount before another test starts. An unreadable sample keeps the prediction reserved. Shared pages may be counted more than once and short-lived children can end between samples: this is sampled accounting, not a kernel hard limit.wait4supplies CPU time and a largest-process peak fallback for such short processes. System load comes from the host's CPU ticks. Available memory is read withhost_statistics64: free plus inactive pages under normal memory pressure, only free pages under warning pressure, zero under critical pressure. Speculative and purgeable pages are not added again. Without a pressure reading only free pages count; a failed VM reading gives no additional headroom. This is a conservative estimate, not a whole-tree memory measurement. The report separately records the admission budget and the sampled memory of the running tests; neither is system-wide memory in use. No hard memory limits.
The system load for the CPU gate: GetSystemTimes (Windows), /proc/stat
(Linux), host_statistics (macOS), sampled every 0.5 s.
tools/test-macos.sh uses
--schedule --memory-budget free --corpus-children. The runner's
history-based ceiling (half the cores until enough tests have profiles,
then the cores) replaces a fixed test count. The available-memory reserve
defaults to the larger of 4 GiB and a tenth of physical memory; explicit
runner options override these defaults. Aqua tests still share
macos_desktop and the cross-checkout GUI lock, with at most one
timing-sensitive UI test below 70% CPU load. Five audited workflows in
each frontend instead own Aqua only for each ui_run_app lifetime: at
most two harnesses may prepare isolated fixtures or verify files beside
each other. Actual apps still never overlap. These labels do not make
AppKit, native input, screenshots or Metal headless. Set
MITCAD_MACOS_PARALLEL_HARNESSES=OFF at configure time to keep the entire
harness under its original CTest desktop lock.
Qt component workflows labelled macos_headless run on the offscreen
platform without viewport rendering or an OpenGL context. Their processes have
private settings, files and virtual screens. They use normal memory/CPU
admission rather than the UI cap or sensitive lane, including when an old
profile records desktop contention. Their assertions and test commands
remain the same with MITCAD_MACOS_HEADLESS_TESTS=OFF, which restores the
normal GUI for comparison. Actual Aqua input and viewport acceptance stay
in the desktop lane.
Cargo build fixtures keep a common cargo_build lock, their fixture
dependencies, a minimum 6 GiB prediction and at most two compiler jobs
(one when current free headroom requires it, or the caller requests one).
They may overlap unrelated tests. MITCAD_CARGO_MAX_JOBS=2 also caps
direct CTest execution. Timed Metal checks still run alone. The wrapper gives no UI
learning bound, so its contention-rerun mode is not enabled. --measure
still defaults to one test and an 85% budget.
Per machine (cores and memory differ), outside every build tree and
mirror so that fresh build folders and removed mirrors keep it:
MITCAD_TEST_PROFILES, else ~/.mitcad/test-profiles.json
(%USERPROFILE%\.mitcad\test-profiles.json on Windows), or --profiles.
Runs of several checkouts on the machine add to it one at a time (a lock
file beside it); --no-record leaves it alone.
{
"format": 1,
"machine": { "cores": 16, "memory": 34359738368 },
"tests": {
"core.model": {
"runs": [
{ "when": 1793000000, "result": "passed", "wall": 41.2, "cpu": 160.3,
"peak": 2147483648, "load": 0.62, "limit": 8589934592 }
],
"failure_score": 0.0,
"sensitive": false,
"loaded_passes": 0,
"memory_kills": 0
},
"app.ui-windows-pick": {
"runs": [
{ "when": 1793000100, "result": "failed", "wall": 31.0, "cpu": 20.1,
"peak": 402653184, "load": 0.55, "limit": 4294967296, "beside": 3 },
{ "when": 1793000900, "result": "passed", "wall": 28.4, "cpu": 19.0,
"peak": 398458880, "load": 0.21, "limit": 4294967296, "beside": 0 }
],
"failure_score": 1.0,
"sensitive": false,
"loaded_passes": 0,
"memory_kills": 0,
"ui_room": 1
}
},
"ui_caps": { "windows": { "cap": 3, "clean_runs": 1 } }
}beside (a UI entry's runs only), ui_room, room_passes and
ui_caps are left out where they are not set
(UI entries at once). A corpus test keeps its
files' last runs too, "files": { "<key>": { "wall": 41.3, "peak": 805306368, "when": 1793000000 } } (below), left out
for the other tests.
The prediction of a test is the 90th percentile of its last runs' wall
time, busy cores and peak memory, the memory times --margin (1.25). A
test with no history:
- borrows the median prediction of its siblings that have one: the tests
that require one of its fixtures or carry one of its labels (the crates'
tests share
cargo_tests_debug, for example); - else is guessed by its kind: UI tests (
app.ui-*, or labelledui) 1 GiB and 1 core, corpus tests (corpusin the name or a label) 2 GiB and 2 cores, the others 512 MiB and 1 core (--default-memory,--default-cores), all 60 s (--default-wall); ctest'sPROCESSORSraises the cores. The cargo test builds run one at a time (RESOURCE_LOCKcargo_buildon Linux and Windows, RUN_SERIAL on macOS).
It stays a test with no history until its own first run, which replaces
the guess. While fewer than half of the tests have a history, at most half
the cores' worth of tests run at once (as ctest -j <cores / 2>), so a
first run is not slower than ctest at that; with most tests known the cap
is the cores. -j sets the cap.
Whenever a test ends, and every 0.5 s:
- Memory is the hard limit: the running tests' predicted peaks plus
the candidate's stay within the budget, and the memory available to
the tests (read again at each decision; on Linux the cgroup headroom
above, see "Measuring per platform") stays above
--memory-floor(1.5 GiB). With nothing running any test starts, so that one larger than the budget still runs, alone. The budget (--memory-budget) is 85 % of a basis:MITCAD_TEST_MEMORYwhen set, else the larger of the physical memory (on Linux the lowest cgroup limit above the runner where lower) and, on Windows, the commit limit (physical memory plus the paging file); the basis is read again during the run and only grows. An absolute--memory-budget 24Gis used as given. A VM with Hyper-V dynamic memory reports only its current assignment as physical memory (6 or 18 GiB on different runs of the same VM): give it--memory-budgetas a size (e.g.24G) or setMITCAD_TEST_MEMORYto its maximum memory. On bare metal,--memory-budget free(mitcad#872;tools/check-all.shgives it there, not in a VM) is a budget from the free memory: at each decision it is the running tests' predicted memory plus the memory available less a reserve (--memory-reserve, default the larger of 4G and a tenth of the basis) less what the running tests are still predicted to grow into (each one's prediction less its scope's working set now: a test just started has not yet taken its memory, which the available memory does not show), at most 85 % of the basis (schedule::free_budget). So a candidate starts only while the memory available exceeds the reserve by its prediction and that growth: when another job (an agent's build) takes memory, fewer tests start, and more when it ends. The corpus tests get the reserve asMITCAD_CORPUS_RESERVEfor their children (below). The run's first lines and the end report give it (Memory budget from the free memory: ..., at the start and its range over the run); the report'sbudgetis its cap. - CPU is the soft limit: no test starts at or above
--cpu-gate(0.9) of the load; the load is the measured one or the running tests' predicted cores where higher (a test just started has not yet loaded the processors).-jcaps the tests at once (default: the cores, or half of them while fewer than half of the tests have a history). - Longest first: each test's rank is its predicted wall time plus the longest chain of tests waiting for it (DEPENDS, fixtures).
- Complementary pairing: among the tests ranked within 60 % of the longest, the one whose memory and cores leave the remaining memory and CPU in the most even shares (a memory-heavy CPU-light test when much memory and little CPU is left, and the other way round).
- The priority lane: tests with a failure score of at least
--lane-score(1) start first, highest score first; while lane tests wait, the others may use only 80 % of the budget (--lane-share) and one slot less, so the lane runs beside the long tests. With--changed FILEthe tests related to the change join the lane too (below). - Timing-sensitive tests start only below
--sensitive-load(0.7), at most--sensitive-max(2) at once;--sensitive 'app.ui-*'marks tests sensitive from the start. With a bound for the UI entries (--ui-maxorMITCAD_UI_JOBS), UI entries start only while fewer than the machine's learnt cap run, and only where neither they nor a running UI entry would have more UI entries beside them than their room (below); a UI entry that failed beside others runs again alone, after the others. Without a bound they start as other tests do. - Safety: each test's hard memory limit is the largest of 4 GiB,
--limit-factor(2) times the p90 of its history's peaks and that factor times its kind's guess, at most 95 % of the memory. RUN_SERIAL tests (they run alone) and tests whose history's largest peak is over 4 times its smallest (a cold build after warm ones) get no limit. A test killed for memory counts as an environment failure (+2), its profile keeps the run with the larger of its peak and the limit, and it reruns once alone without a limit (the report names the rerun); no test starts while the build folder's disk has less than--disk-floor(10 GiB) free, and the tests left are reported as not run for that reason.
Both modes keep exactly the properties ctest reports, nothing more:
DEPENDS and a fixture's setup tests order only the tests that name them,
a test whose fixture setup failed is not run, cleanups run after every
test of their fixture, a RESOURCE_LOCK keeps apart only the tests with the
same lock, and only a test with RUN_SERIAL set runs alone (as with ctest
-j, nothing else runs beside it). In this tree the cargo test builds,
core.cargo_build and core.cargo_build_release, share the lock
cargo_build on Linux and Windows and have RUN_SERIAL on macOS
(core/CMakeLists.txt); the Windows UI tests that use the system
clipboard, which all tests of one ssh logon share, hold the lock
system_clipboard (mitcad#870). macOS Aqua UI and render-surface tests
share macos_desktop, so they exclude each other while independent model,
geometry and offscreen unit tests may run beside them. The shell UI
harness also holds a per-user file lock across separate runs and checkouts.
RUN_SERIAL remains for the timed cli.perf_*, the macOS Cargo builds and
Metal scene acceptance, and the Metal surface tests whose parallel
execution has not been validated. A macOS Cargo build needs this isolation because
its memory profile records the largest child process rather than the
whole compiler process tree, and no hard memory limit is available.
--dry-run shows
each test's RUN_SERIAL and RESOURCE_LOCK. A test still running two minutes after its ctest TIMEOUT
(1500 s by default) is stopped and counted as a timeout.
Every failure is printed as it happens (marked priority lane for the
lane's tests) and appended to status.txt, so that a fixer can start
while the run goes on. The end report gives the total time; the means over
the run of the system's CPU load and memory in use (sampled every 0.5 s,
weighted by time), of the tests' busy cores, of the tests running at once
and of their predicted memory (the sums of each test's CPU time, wall
time and predicted memory times wall time, over the run's wall time); the
critical path, the chain of tests that ended the run, each with what held
it back (its dependency, a test that runs alone, a resource lock, or memory
and CPU that came free) and how long it waited; the longest tests, the
failures by kind and the tests whose profile changed much (wall time or
peak halved or doubled). In results.tsv the lane column is 1 for a
test that ran in the priority lane.
How many UI entries a machine runs at once without one disturbing another
(roadmap step 5: the Windows VM's session 0, Linux's Xvfb displays,
macOS) is learnt from the ordinary runs, not from a measuring run of its
own (src/concurrency.rs). A UI entry is a test of the UI kind
(app.ui-*, or labelled ui), or a UI test tools/check-all.sh runs
itself and hands over (--record-ui, below).
- What each run records: a UI entry's run keeps
beside, the most other UI entries that ran beside it. In--schedulewith a bound (see below), a UI entry that failed or timed out with others beside it runs once more alone, after the others (as check-all's own UI step does, mitcad#657); its result counts, the first attempt is named in the report (Rerun alone (failed beside other UI entries)), instatus.txt(ui) and in its note. - A contention failure: a UI entry failed beside
bothers and then passed with fewer beside it (alone). It adds +2 to the failure score, as other failures of the environment, not +10. - A test's room (
ui_roomin its profile): the other UI entries it may have beside it. A contention failure besidebhalves it tob / 2(0: no other UI entry beside it); 3 passes with the room full (at least that many beside it) grow it by one.--dry-runshows it (UI room N). - The machine's cap (
ui_caps, per platform,std::env::consts::OS): UI entries at once. Bounded by--ui-max, elseMITCAD_UI_JOBS(4 for the learning when neither is given), it starts at the bound (Linux withMITCAD_UI_JOBS=12: 12, as before), drops by one (at least 1) after a run with contention failures of two or more UI entries (CAP_DROP_TESTS, mitcad#872: one test's contention failure lowers only its room, which the scheduler and check-all keep; ui-pick and ui-sweep alone held Linux's 196 UI tests at 7 or 8 at once of the bound 12, since check-all did not keep the rooms), and grows by one after 2 clean runs that reached it (no UI entry failed beside others), never above the bound. A run with fewer UI entries at once, or with a UI failure that failed alone too, leaves it. The end report gives the run's line (UI entries at once: cap 6 -> 7; this run at most 6, ...). Measuring mode records and learns too, but neither reruns nor holds UI entries to the cap.
A machine opts in by giving the bound (--ui-max N or
MITCAD_UI_JOBS). Only then does the scheduler start a UI entry only
while the UI entries running are fewer than the cap and its room and every
running UI entry's room allow one more beside them (non-UI tests are not
counted), and rerun a UI entry alone after a failure beside others.
Without a bound the runs still record beside and learn the cap and the
rooms (within 4), but UI entries are scheduled as before: no cap, no
rooms, no rerun. With a bound the fixed MITCAD_UI_JOBS is no longer the
number at once but its ceiling: the runner starts there, goes below it
only after contention failures and back up as clean runs show, never
above it. --dry-run and the run's first
lines say which applies.
tools/check-all.sh runs the Linux UI tests (tools/ui-*-test.sh) itself,
not as ctests. For such a caller:
# The cap to run at (within the bound; MITCAD_UI_JOBS when --ui-max is not given):
build/dev/core/mitcad-testrun --ui-cap --ui-max "$MITCAD_UI_JOBS"
# Each UI entry's learnt room, a line each: name, the most other UI entries
# beside it (check-all starts such a test only while few enough run, and
# no other while it runs with its room full, as the scheduler does):
build/dev/core/mitcad-testrun --ui-rooms
# Its UI runs, a line each in the order they ended (both attempts of a
# test rerun alone): name, exit status, seconds, other UI entries beside it.
printf 'ui-pick\t1\t40\t11\nui-pick\t0\t38\t0\n' > ui-runs.tsv
build/dev/core/mitcad-testrun --record-ui ui-runs.tsv --ui-max "$MITCAD_UI_JOBS"--record-ui adds the runs to the profiles (exit status 0 passed, 124 and
137 timed out, 77 skipped, others failed), learns the cap from them and
prints what it learnt; neither needs --build.
A corpus test (corpus in its name or a corpus label) imports its
files in child processes, several at once (core/tests/parallel_runs.hpp:
test_f3d_import, test_brep_import, test_exchange, and
mitcad_run_parallel for the scripts' corpora, f3d.corpus_designs
among them, its imports with a debug build's own-start time, mitcad#847),
mitcad#832. rvt.corpus (its project and its link check, keyed by the
content ids they import), dwg.corpus (a drawing each) and 3dm.corpus (a model each, mitcad#1715) do so too
since mitcad#875; they run in tools/check-all.sh's ctest step, which
gives them --corpus-children as the corpus step does its tests:
- Each file's time and peak. For every corpus test the runner names
a table in
MITCAD_CORPUS_RECORD(<out>/corpus/<test>.runs.tsv), to which the test appends a line per child: the file's key (its path, or the label of amitcad_run_parallellist), seconds, peak bytes andokorfailed. The profiles keep each file's last run under the test'sfiles; a file not run for 60 days goes.--no-recordleaves them alone. - The longest files first. A test with files in its profile gets
their times in the table
MITCAD_CORPUS_FILESnames (<out>/corpus/<test>.files.tsv: key, seconds, peak bytes). Its children start by them, the longest first; a file without a time by its size at the known files' median seconds per byte, or among the first when it has no size. - Children from the profile (
--corpus-children, whichtools/check-all.shgives its corpus step unless--corpus-jobsis given). Each corpus test with files in its profile gets its children at once inMITCAD_CORPUS_JOBS: every such test starts with two (one with a single file), then a child at a time goes to the test that would end last (its work, the sum of its files' times, over its children, never below its longest file), as long as one more shortens it, all the children together within the machine's cores and their predicted memory (256 MiB per test and each child its test's largest file's peak times--margin) within the budget. So a long test gets more children and a short one fewer, and they end at about the same time. A child's memory limit,MITCAD_CORPUS_MEMORY, is--limit-factortimes its test's largest peak of a file (a limit on committed memory, which exceeds the resident peak), never below the caller'sMITCAD_CORPUS_MEMORY(1536M when unset). The test's own prediction and hard memory limit follow its children. The corpus tests without files in their profile (their first run, or a test that imports in one process) keep the caller's variables, and their predictions are kept out of the room the others share. The run's first lines name each planned test's children;--dry-runshows the variables. - A test that no longer imports in children (mitcad#872): a corpus
test that passes while its children recorded nothing, although its
profile has files, imports in one process now, and its files go from
the profile (
corpus::one_process), so that the next run predicts it by its own history.f3d.corpus_designschecked its designs one after another from 0.32.0 until mitcad#874 put them back on children; planned by its files it was predicted at its longest design's 314 s with two children's 7 GiB held for it, and took 498 s. - Children wait for memory (mitcad#872): with the budget from the
free memory the runner gives each corpus test
MITCAD_CORPUS_RESERVE(the reserve, unless the caller set it); a child beside others then starts only while the memory available (MemAvailable, or less where a cgroup above is nearer its limit, its working set without the page cache) exceeds the reserve by the child's expected peak (its file's last, fromMITCAD_CORPUS_FILES, else the limit per child), checked again every 0.5 s (parallel_runs.hpp,may_start).
--changed FILE (roadmap step 4) names a list of the changed files, a
path a line, relative to the source tree as git diff --name-only <base>
prints them (absolute paths in the tree are taken too; empty lines, lines
starting with # and paths outside the tree are left out). Each test
related to one of them gets a failure's raised place for this run: 10 is
added to its failure score (to order the lane; the profile keeps its own
score) and, in scheduling mode, it joins the priority lane whatever
--lane-score says, so that a change's own tests run and report first.
The run prints the changed files' count and the related tests.
A test's sources, from its ctest registration and the tree
(src/changed.rs):
- the files and folders of the source tree its command and its
ENVIRONMENTname (its script, a cli test's JSON, acmake -Pscript, a folder of tests), not those in the build folder; - the areas each of those files declares in
# check-all sources: <path>...lines, the header linestools/check-all.shreads for its UI tests (the UI ctests' scripts:tools/ui-*-test.py,ui-windows-*-test.ps1,ui-macos-*-test.sh,ui-scripts/*.mitcad-ui); - for a crate's tests (
cargo-tests.cmake -- run|parts|build <package>,cargo test --doc -p <package>) each package's folder, from the workspace's members (core/Cargo.toml; no globs); - for a test program CMake builds (the command's program, e.g.
test_sheet,mitcad-cli) its own target's sources, from CMake's file API reply in the build folder (.cmake/api/v1/reply, whichtools/check-all.shasks for; without it this part is left out).
The source and build trees come from the build's CMakeCache.txt. A
source covers a changed path that is that file or lies in that folder.
Only these direct sources count, not the crates and libraries they use in
turn, so that a change to a crate or library every test uses does not put
every test in the lane (the fingerprints of tools/ctest-fingerprints.py
decide which tests run at all; this only orders them).
A release gate runs every test on every platform (Linux tools/check-all.sh --full, the Windows and macOS full ctest), although the kernel's geometry
tests, about two fifths of all tests' CPU time, seldom have anything new to
check when geometry/ and the OCCT port have not changed since the last
release (mitcad#1822). So in a release gate's run (--release-passed DIR --release-tag X.Y.Z-rcN) a geometry* ctest is skipped when its
fingerprint is one with which it passed in an earlier release gate of the
same platform. No other test is skipped this way, nor a test that sets up
or cleans up a fixture.
The fingerprint (src/fingerprint.rs) is the logic of
tools/ctest-fingerprints.py (the development checks' ctests, the check
script) in Rust, so that it is
the same on Linux, Windows and macOS: a SHA-256 of what the test depends
on, found from the build itself:
- its registration in ctest's listing (command, properties), without the checkout's and the build folder's paths;
- the files and folders of the source tree its command and environment name;
- each program it runs or names (the test executable): the CMake targets it
is built from, through their dependencies, from CMake's file API in the
build folder (codemodel-v2): each target's folder (less the folders of
other CMake directories and Rust crates, and less the files tests name),
its listed sources, a Corrosion crate's Rust closure (
cargo metadata); the build files (the top-levelCMakeLists.txtandCMakePresets.json,cmake/,triplets/,third_party/with the OCCT port and its patches,vcpkg.json,rust-toolchain.toml); the compiler and its version (the file API's toolchains-v1) and the build's options (MITCAD_*switches and the build type ofCMakeCache.txt); - the fingerprints of the fixtures it requires.
Markdown documents count for no test, and the release's version is left
out of the files release.sh version sets for every release (the
top-level CMakeLists.txt, vcpkg.json, core/Cargo.toml,
core/Cargo.lock): otherwise no fingerprint would ever repeat. A different
compiler or compiler version, other options, another OCCT patch or any
changed source the test is built from runs it. Without the file API's
reply every fingerprint covers the whole tree (the release scripts ask for
it before they configure: .cmake/api/v1/query/codemodel-v2 and
toolchains-v1 in the build folder); without a fingerprint (ctest cannot
list the tests, no CMakeCache.txt) every geometry test runs.
The passes are kept per platform (std::env::consts::OS) in a folder that
outlives the build folders, beside the release's persistent cargo folder:
DIR/<platform>/<test>, a line per fingerprint, <fingerprint>\t<release>
(the release and RC in which it last passed), the last 20 of each test.
After the run the fingerprint of each geometry test that passed is kept
with this gate's release; the platforms' records never mix, also in a
shared folder.
The run's first lines and its report.txt say what was decided, after the
results line:
Release fingerprints: 120 of 165 geometry tests skipped (passed with the same fingerprint in an earlier release gate on windows), 45 run
Skipped geometry.sheet (last passed in 0.62.0-rc2)
...
Release fingerprints kept: 45 passes as 0.63.0-rc1 in C:/dev/release-passed/windows
or all 165 geometry tests run: first release with fingerprints on linux
when the platform has no record yet (the first release after this
landed), or all 165 geometry tests run (--all-geometry), which forces
every geometry test to run (its passes are kept as usual).
<out>/release-skipped.tsv lists the skipped tests (name, fingerprint,
the release they last passed in); the release scripts copy it into the
platform's release report. The skipped tests are not in results.tsv.
The time saved is read from the release timelines, not measured apart.
- Each test is its own ctest process; parallel ctests of one build tree
share its
Testing/Temporaryfiles (LastTest.log,CTestCostData.txt). The results come from each ctest's own output, but a platform that refuses the concurrent writes would show it as environment failures in the first runs. - Linux: the peak is the largest working set sampled every 0.1 s, so a
spike shorter than that is seen only through the largest process's
ru_maxrss; the last 0.1 s before the scope goes is not sampled. - Windows: the peak is committed memory, which can exceed the working set.
- Step 1 and 2 are here: measuring as today, and scheduling, first for the Windows VM.
- Step 3: Linux
tools/check-all.shruns its ctest and corpus steps through--schedule(ctest -j in a test slot when the runner is not built or does not finish), and so does the Linux release build, whose tests arecheck-all.sh --full; the check's slice holds the tests' scopes (the check script). macOS usestools/test-macos.shfor merge and release checks (one Aqua UI test at a time; initial take-up in serial measure mode). - Step 4: the tests related to the files a change touches join the
priority lane (
--changed, above);tools/check-all.shhands it the files of its diff. - Step 5: how many UI entries run at once without disturbing each other
(Windows session 0, Xvfb displays, macOS), learnt from the runs: a
machine cap per platform and each test's room
(above), applied where a machine gives the bound
(
--ui-maxorMITCAD_UI_JOBS); check-all's own Linux UI step can take--ui-capand hand its runs over with--record-ui. macOS's Aqua UI tests holdmacos_desktop, so they stay one at a time while non-UI tests can use the remaining resources.