-
Notifications
You must be signed in to change notification settings - Fork 5
Open
Labels
Description
I prefer to keep PR CI times reasonably brief, which is in tension with a desire for thorough tests and benchmarks. I try to target runtimes below 10 minutes and, since these benchmarks need to be run twice to evaluate both versions, this means we’d be limited to about 5 minutes of benchmarking to achieve.
In trying to balance these goals for benchmarking, I propose to add a separate benchmark script benchmarks/benchmark_extended.jl
that will provide more thorough coverage and can be run locally via benchpkg
with an explicit script location flag. Coverage in the standard CI script can then be a little shallower and wider.