Parallel cucumbers

[MarcelloPerathoner]

Issue

Fixes #7307 - CI runs up from 0:45h to 4:30h

Improvements

Parallel execution of cucumber tests tremendously improves running times.

Note: when using --parallel N make sure there are N contiguous free ports at the
configured port number (eg. at ports 5000--5000+N).

A lot more can be configured with profiles (in cucumber.mjs). Usage examples:

$ npx cucumber-js -p home -p ch -p mmap --format html:/tmp/testlog.html --parallel 8
$ npx cucumber-js -p github -p mld -p directly --parallel 8
$ npm test -- --parallel $JOBS --fail-fast

Github step summaries display the test results:

On github CI full logs, also containing the outputs from all binaries, are published to
https://reports.cucumber.io/.

Code refactored and simplified. ~150 LoC less.

Other Fixes

During development other problems were found and fixed:

• The datastore load method now works as expected: one and the same osrm-routed will
  be running in the background for the whole duration of the test. Previous behaviour: for
  each scenario an osrm-routed would be spawned, loaded with datastore, and then killed
  again.

• The data watchdog in osrm-routed now will emit the message: 'updated facade' right
  after the update (previous behaviour: right before the update), so that it can be used
  for process synchronization.

Breaking changes

If you run the tests using the npx cucumber-js ... calling method you must add either
-p home or -p github as the first profile, eg.:

$ npx cucumber-js -p home -p mld -p mmap --parallel 8

instead of:

$ OSRM_LOAD_METHOD=mmap npx cucumber-js -p mld

Old cached files are not cleaned automatically anymore because that interfered with
concurrency. There is no hook in cucumber that runs before the workers start.

We don't support changing the load method in mid-run anymore. (eg. "Given data is loaded
with datastore" no longer works). Reason: With the datastore load method an
osrm-routed will be constantly running in the background. No second osrm-routed can
open the same port again.

Retired Environment Variables

• OSRM_CONNECTION_RETRIES
• OSRM_CONNECTION_EXP_BACKOFF_COEF

Other changes

Removed one cache level. CH and MLD are processed and cached separately.

The outputs of osrm binaries are not written into lots of small files anymore. Instead
use the --format html:test/logs/testlog.html commandline argument to get one big HTML
file with all outputs embedded.

New configuration variables

New environment variables and/or cucumber worldParameters entries were introduced. The
whole configuration is done in cucumber.mjs.

Environment Variable	worldParameters	Defaults to
CUCUMBER_TIMEOUT	timeout	5000	Scenario timeout in ms.
CUCUMBER_HTTP_TIMEOUT	httpTimeout	2000	HTTP timeout in ms.
CUCUMBER_TEST_PATH	testPath	test	The test directory
CUCUMBER_PROFILES_PATH	profilesPath	profiles	The profiles directory
CUCUMBER_LOGS_PATH	logsPath	test/logs	The logs directory
CUCUMBER_CACHE_PATH	cachePath	test/cache	The cache directory
OSRM_BUILD_DIR	buildPath	build	Path to the binaries
OSRM_LOAD_METHOD	loadMethod	datastore	Data load method
OSRM_ALGORITHM	algorithm	ch	Routing algorithm
OSRM_IP	ip	127.0.0.1	IP Address
OSRM_PORT	port	5000	IP Port

New tags

Tag	A scenario thus tagged ...
@isolated	will not run while any other scenario is running in parallel
@with_(datastore|directly|mmap)	will be executed iff the load method matches
@no_(datastore|directly|mmap)	will be executed unless the load method matches
@with_(ci|mld)	will be executed iff the algorithm matches
@no_(ci|mld)	will be executed unless the algorithm matches

Tasklist

• CHANGELOG.md entry (How to write a changelog entry)
• TODO: clean up and remove some unrelated stuff
• update relevant Wiki pages
• review
• adjust for comments

Requirements / Relations

Link any requirements here. Other pull requests this PR is based on?

[afarber]

Thanks for working on speeding up the CI! The parallel cucumber approach looks promising.

One concern: could we keep the CI matrix changes out of this PR? Switching from macos-13 to macos-15-intel drops test coverage for users on Ventura/Sonoma, and adding clang-20 (which isn't released yet) seems unrelated to the parallelization work. These changes could be discussed separately if there's a reason to drop older platform support.

Also spotted a few bugs in osrm_loader.js - there are references to an undefined err variable in the error handlers (lines 45, 118, 132) that would crash if osrm-routed fails.

[MarcelloPerathoner]

The PR is still WIP. I will cleanup before releasing it.

BTW clang-21 is current. The reason to drop macos-13 is that github does not support it anymore.

[TheMarex]

Left-over?

[TheMarex]

Do you think the non-cucumber changes (datastore, routed, etc) can be broken out independently without messing with the tests? It is always quite the challenge to review large changes like this so I would appreciate the extra effort if possible. 🙏

[MarcelloPerathoner]

The shared memory changes could easily go elsewhere, as they are not the primary reason for sync loss between the test suite and routed. But the change in routed is critical (and trivial too).

The changes to the github workflow should go elsewhere. I'm just playing around to get windows to run the test suite too. See if any surprises come up.

[MarcelloPerathoner]

Ready for review. I have removed everything that is not strictly necessary. The shared memory fixes for macOS and Windows will go into another PR. Until that macOS tests with datastore will be flaky.

[afarber]

You have removed the setDefaultTimeout() method and that is why the CUCUMBER_TIMEOUT: 60000 from the conan-macos-arm64-release-node section is not passed anywhere and default timeout of 5000 is used causing the macOS arm64 job to fail

[MarcelloPerathoner]

The macOS builds fail randomly because of a race condition: see #7320 #7321. The build fails if osrm-datastore exits too early, not too late. Thus a longer timeout would not solve anything here.

And if osrm-routed dies for whatever reason, then a longer timeout would just slow down things because each scenario would take exactly timeout ms to fail.

OTOH the timeout was the reason the builds kept failing for such a long time: each time a build failed somebody just raised the timeout. Then, if the next CI run did by pure chance succeed, the problem was declared solved. Didn't you ever wonder how a totally absurd timeout of 60 seconds was reached?

I'm thinking of removing the timeout setting altogether. It induces people to fiddle with the parameters instead of finding the underlying cause. The default 5 sec is plenty.

[afarber]

The semicolon is not needed

[afarber]

Consider adding a timeout fallback:

  before(scenario) {
    this.current_scenario = scenario;
    this.semaphore = new Promise((resolve, reject) => {
      this.resolve = resolve;
      setTimeout(() => reject(new Error('Timed out waiting for facade update')), env.wp.timeout);
    });
    // ...
 }

Or use Promise.race() with the existing waitForConnection() timeout.

[MarcelloPerathoner]

In that case Cucumber will timeout. No need to complicate the code.