Rerouting aggregator

[scottwittenburg]

Add an option to perform rerouting during BP5 aggregation, as well as a mechanism to introduce simulated interference from other applicatoins.

Summary of changes

These are the major contributions of this PR.

Implement a rerouting aggregator

Implement rank rerouting in the BP5Writer, based on the paper "Can I/O Variability Be Reduced on QoS-Less HPC Storage Systems" (link).

Currently this rerouting aggregator only works if DataSizeBased aggregation, due to the flexibility provided by DSB to allow ranks to change subfiles between timesteps. However, there is no reason we couldn't extend this in the future so that EWS and EW could both benefit from rerouting.

Introduce simulated interference

Introduce an optional (compile-in) feature that provides a controlled "writer interference" mechanism. If the feature is enabled at configure time, the user can slow down writes to any subfile using environment variables at runtime. Individual environment variables up to the total number of subfiles can be specified at the start of a run, each one a dial causing ranks writing to the subfile to sleep for an amount of time proportional to the actual time they spent writing. For example, specifying "ADIOS2_WRITE_DELAY_0=0.5" in your environment will cause ranks writing to data.0 to sleep for half of their actual write time after writing. The total write time plus the delay/sleep time is then recorded in the profiling data.

This PR also fixes an issue with profiling DataSizeBased aggregation. When DSB was used over multiple timesteps, only a single transport section was included in the profiling.json. This change iterates over all the transports cached by a rank and includes a profiling section for each one. Additionally, the name of the subfile (data.x, md.x, etc) is included in each section.

Add a rerouting message class

Add a message class that serialize and deserialize itself into a vector of char buffer, as well as send and receive itself over MPI.

Add a test of the new message class that also exercises the newly added MPI features (probe and receive from anyone).

Support probing and receiving from any rank

To support the rerouting aggregator, add api and implementations for two MPI features that were previously missing:

1. MPI_Probe method allows checking if a message can be received
2. Receving messages from the special value MPI_ANY_SOURCE

Include testing rerouting in the BP5-only tests

Update a swath of BP5 tests to be repeated once more to exercise the new rerouting aggregation scheme.