Add size-based pairwise dispatch to all_to_all

[iemAnshuman]

Proposed Changes

• Add a direct pairwise path to the basename-based asynchronous and synchronous all_to_all overloads.
• Add pairwise_threshold_arg, with a default of 4096 bytes, to select between routed and pairwise exchange.
• Cache the channel communicator across generations, using the generation number as the exchange tag.
• Add unit and distributed dispatch tests, and support --pairwise_threshold in benchmark_collectives.

Any background context you want to provide?

The basename-based implementation currently routes every contribution through one communicator site. This limits the number of messages sent by each site, but concentrates the exchange on one locality. The pairwise path sends each destination row directly to its peer. This removes the relay when the payload is large enough to offset the additional messages.

Pairwise exchange requires at least three sites and an explicit positive generation. Calls using the default generation remain on the routed path. Automatic dispatch uses sizeof(T) for trivially copyable, fixed-size row types. Dynamically sized rows remain routed because local size checks could cause different sites to select different algorithms. Callers that know the rows are uniform can pass pairwise_threshold_arg(0) at every site to request pairwise exchange.

The channel communicator is cached by normalized basename and site count. Reusing it across generations avoids repeated AGAS registration and peer discovery, while generation tags keep exchanges separate. The implementation preserves the existing argument validation and supports vector<bool> rows and rows that cannot be default-constructed or assigned.

The 16-node Rostam/Buran campaign covered 16, 32, 64, and 128 localities, with payloads from 4 B through 64 KiB per destination. All 960 executions completed without retries or failures. A 4 KiB threshold was the smallest sampled threshold for which every selected pairwise result beat the fastest non-pairwise HPX result in the same campaign.

Full measurements: 16-node pairwise all_to_all results

Verification

• Built the pairwise tests and benchmark_collectives.
• Passed the asynchronous, synchronous, and hierarchical all_to_all tests.
• Passed the pairwise helper and three-locality dispatch tests.
• Checked positive-threshold dispatch at the 1024-byte boundary with 255 and 256 int elements.
• Passed clang-format --dry-run --Werror and git diff --check.

Checklist

Not all points below apply to all pull requests.

• I have added a new feature and have added tests to go along with it.
• I have fixed a bug and have added a regression test.
• I have added a test using random numbers; I have made sure it uses a seed, and that random numbers generated are valid inputs for the tests.

[iemAnshuman]

Threshold selection for the 4 KiB default

I evaluated each sampled global threshold over the 32 cells in the 16-node sweep: four locality counts and eight payload sizes. Cells below the threshold keep the fastest non-pairwise result and contribute 1.0x to the overall geometric mean. Cells at or above the threshold use the measured pairwise result.

For each cell, the non-pairwise baseline is the fastest result among standard one-shot, standard reused, forced flat, and tree arities 2, 3, and 4.

Threshold	Pairwise cells	Wins	Regressions	Whole-matrix geomean
0, forced	32	16	16	1.76x
16 B	28	16	12	1.88x
64 B	24	16	8	2.00x
256 B	20	15	5	2.10x
1 KiB	16	14	2	2.15x
4 KiB	12	12	0	2.08x
16 KiB	8	8	0	1.76x
64 KiB	4	4	0	1.36x
Disabled	0	0	0	1.00x

The 1 KiB threshold has the highest aggregate geometric mean, but it introduces two regressions:

• At 16 localities and 1 KiB, pairwise takes 389 us, compared with 278.5 us for the best non-pairwise path: 0.72x.
• At 32 localities and 1 KiB, pairwise takes 724 us, compared with 614 us: 0.85x.

The 4 KiB threshold wins all 12 cells it selects and produces a 2.08x whole-matrix geometric mean, 3.2% below the maximum at 1 KiB. At the 4 KiB boundary, pairwise is 1.14x, 2.79x, 5.25x, and 13.82x faster at 16, 32, 64, and 128 localities, respectively.

This makes 4096 bytes the smallest no-regression threshold in this sweep. The crossover can change with the transport and topology, so callers can override the default.

Full measurements and methodology: 16-node pairwise all_to_all results

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c2fa0a42-9edf-4ecd-984e-ef31b06ae52a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[hkaiser]

@iemAnshuman Why did you close this PR? Has it been superseded with #7431?

[StellarBot]

Performance test report

HPX Performance

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR	PARALLEL_EXECUTOR	SCHEDULER_EXECUTOR
For Each	(=)	(=)	(=)

Info

Property	Before	After
HPX Datetime	2026-03-09T14:08:29+00:00	2026-08-03T19:45:38+00:00
HPX Commit	0eeca86	a99e5ce
Compiler	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8
Datetime	2026-03-09T09:15:24.034803-05:00	2026-08-03T18:32:17.384498-05:00
Envfile
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu

Comparison

BENCHMARK	NO-EXECUTOR
Future Overhead - Create Thread Hierarchical - Latch	+++

Info

Property	Before	After
HPX Datetime	2026-03-09T14:08:29+00:00	2026-08-03T19:45:38+00:00
HPX Commit	0eeca86	a99e5ce
Compiler	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8
Datetime	2026-03-09T09:17:15.638328-05:00	2026-08-03T18:33:53.265236-05:00
Envfile
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR	PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR	SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR
Stream Benchmark - Add	(=)	-	--
Stream Benchmark - Scale	(=)	--	-
Stream Benchmark - Triad	(=)	-	--
Stream Benchmark - Copy	(=)	+++	+++

Info

Property	Before	After
HPX Datetime	2026-03-09T18:50:37+00:00	2026-08-03T19:45:38+00:00
HPX Commit	ba89f5d	a99e5ce
Compiler	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8
Datetime	2026-03-09T17:49:10.837937-05:00	2026-08-03T18:34:13.491077-05:00
Envfile
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu

Explanation of Symbols

Symbol	MEANING
=	No performance change (confidence interval within ±1%)
(=)	Probably no performance change (confidence interval within ±2%)
(+)/(-)	Very small performance improvement/degradation (≤1%)
+/-	Small performance improvement/degradation (≤5%)
++/--	Large performance improvement/degradation (≤10%)
+++/---	Very large performance improvement/degradation (>10%)
?	Probably no change, but quite large uncertainty (confidence interval with ±5%)
??	Unclear result, very large uncertainty (±10%)
???	Something unexpected…