Skip to content

Backports from master #4184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vicentebolea merged 12 commits into from
May 29, 2024
Merged

Backports from master #4184

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
99fc509
Fixes for FreeBSD, including upstream (#4138)
eisenhauer Apr 15, 2024
04ba058
Merge pull request #4142 from vicentebolea/fix-pugixml-dep
vicentebolea Apr 21, 2024
e766045
ReadModeFlatten (#4147)
eisenhauer Apr 24, 2024
0dea90b
Minor docs tweaks (#4153)
eisenhauer Apr 26, 2024
78137c5
Merge pull request #4161 from anagainaru/derived-performance
anagainaru May 11, 2024
ee179ef
Typo in the build command inside the documentation (#4166)
anagainaru May 17, 2024
83112ab
Add API call to reset memory selection (#4169)
franzpoeschel May 24, 2024
7e8248e
EVPath 2024-05-27 (45471c87) (#4181)
eisenhauer May 28, 2024
cc91ec3
Remove special ReadFlattenSteps Mode, switch to control by writer-sid…
eisenhauer May 11, 2024
f839acd
fix: make changes from #4144 abi compat
vicentebolea May 28, 2024
e77056e
fix: make changes from #4160 abi compat
vicentebolea May 28, 2024
260166d
Kill outdated reference to bindings.Mode.ReadFlattenSteps (#4185)
eisenhauer May 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions bindings/C/adios2/c/adios2_c_io.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,8 +329,8 @@ adios2_error adios2_remove_all_attributes(adios2_io *io);
* MPI Collective function as it calls MPI_Comm_dup
* @param io engine owner
* @param name unique engine identifier
* @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append, and
* adios2_mode_readRandomAccess
* @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append
* and adios2_mode_readRandomAccess
* @return success: handler, failure: NULL
*/
adios2_engine *adios2_open(adios2_io *io, const char *name, const adios2_mode mode);
Expand All @@ -341,7 +341,7 @@ adios2_engine *adios2_open(adios2_io *io, const char *name, const adios2_mode mo
* MPI Collective function as it calls MPI_Comm_dup
* @param io engine owner
* @param name unique engine identifier
* @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append, and
* @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append and
* adios2_mode_readRandomAccess
* @param comm communicator other than adios' handler comm. MPI only.
* @return success: handler, failure: NULL
Expand Down
2 changes: 1 addition & 1 deletion bindings/CXX11/adios2/cxx11/Variable.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ class Variable
* variable.Count() = {Ny,Nx}, then memoryCount = {Ny+2,Nx+2}
* </pre>
*/
void SetMemorySelection(const adios2::Box<adios2::Dims> &memorySelection);
void SetMemorySelection(const adios2::Box<adios2::Dims> &memorySelection = {{}, {}});

/**
* Sets a step selection modifying current startStep, countStep
Expand Down
37 changes: 36 additions & 1 deletion docs/user_guide/source/components/anatomy.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,10 @@ named `adios2::Mode::ReadRandomAccess`. `adios2::Mode::Read` mode allows data ac
current timestep. `ReadRandomAccess` can only be used with file engines and involves loading all the file metadata at
once. So it can be more memory intensive than `adios2::Mode::Read` mode, but allows reading data from any timestep using
`SetStepSelection()`. If you use `adios2::Mode::ReadRandomAccess` mode, be sure to allocate enough memory to hold
multiple steps of the variable content.
multiple steps of the variable content. Note that ADIOS streaming
engines (like SST, DataMan, etc.) do not support `ReadRandomAccess`
mode. Also newer file Engines like BP5 to not allow
`BeginStep/EndStep` calls in `ReadRandomAccess` mode.

.. code:: C++

Expand All @@ -134,3 +137,35 @@ multiple steps of the variable content.
| |--> IO goes out of scope
|
|--> ADIOS goes out of scope or adios2_finalize()

Previously we explored how to read using the input mode `adios2::Mode::Read`. Nonetheless, ADIOS has another input mode
named `adios2::Mode::ReadRandomAccess`. `adios2::Mode::Read` mode allows data access only timestep by timestep using
`BeginStep/EndStep`, but generally it is more memory efficient as ADIOS is only required to load metadata for the
current timestep. `ReadRandomAccess` can only be used with file engines and involves loading all the file metadata at
once. So it can be more memory intensive than `adios2::Mode::Read` mode, but allows reading data from any timestep using
`SetStepSelection()`. If you use `adios2::Mode::ReadRandomAccess` mode, be sure to allocate enough memory to hold
multiple steps of the variable content. Note that ADIOS streaming
engines (like SST, DataMan, etc.) do not support `ReadRandomAccess`
mode. Also newer file Engines like BP5 to not allow
`BeginStep/EndStep` calls in `ReadRandomAccess` mode.

.. code:: C++

ADIOS adios("config.xml", MPI_COMM_WORLD);
|
| IO io = adios.DeclareIO(...);
| |
| | Engine e = io.Open("InputFileName.bp", adios2::Mode::ReadRandomAccess);
| | |
| | | Variable var = io.InquireVariable(...)
| | | | var.SetStepSelection()
| | | | e.Get(var, datapointer);
| | | |
| | |
| | e.Close();
| |
| |--> IO goes out of scope
|
|--> ADIOS goes out of scope or adios2_finalize()


2 changes: 1 addition & 1 deletion docs/user_guide/source/components/io.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ A particular ``Engine`` type is set to the current ``IO`` component with the ``I
Engine polymorphism is handled internally by the ``IO`` class, which allows subclassing future derived ``Engine`` types without changing the basic API.

``Engine`` objects are created in various modes.
The available modes are ``adios2::Mode::Read``, ``adios2::Mode::Write``, ``adios2::Mode::Append``, ``adios2::Mode::Sync``, ``adios2::Mode::Deferred``, and ``adios2::Mode::Undefined``.
The available modes are ``adios2::Mode::Read``, ``adios2::Mode::ReadRandomAccess``, ``adios2::Mode::Write``, ``adios2::Mode::Append``, ``adios2::Mode::Sync``, ``adios2::Mode::Deferred``, and ``adios2::Mode::Undefined``.


.. code-block:: c++
Expand Down
10 changes: 10 additions & 0 deletions docs/user_guide/source/engines/bp5.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,14 @@ This engine allows the user to fine tune the buffering operations through the fo

#. **Threads**: Read side: Specify how many threads one process can use to speed up reading. The default value is *0*, to let the engine estimate the number of threads based on how many processes are running on the compute node and how many hardware threads are available on the compute node but it will use maximum 16 threads. Value *1* forces the engine to read everything within the main thread of the process. Other values specify the exact number of threads the engine can use. Although multithreaded reading works in a single *Get(adios2::Mode::Sync)* call if the read selection spans multiple data blocks in the file, the best parallelization is achieved by using deferred mode and reading everything in *PerformGets()/EndStep()*.

#. **FlattenSteps**: This is a writer-side parameter specifies that the
reader should interpret multiple writer-created timesteps as a
single timestep, essentially flattening all Put()s into a single step.

#. **IgnoreFlattenSteps**: This is a reader-side parameter that
tells the reader to ignore any FlattenSteps parameter supplied
to the writer.

============================== ===================== ===========================================================
**Key** **Value Format** **Default** and Examples
============================== ===================== ===========================================================
Expand All @@ -156,6 +164,8 @@ This engine allows the user to fine tune the buffering operations through the fo
StatsLevel integer, 0 or 1 **1**, 0
MaxOpenFilesAtOnce integer >= 0 **UINT_MAX**, 1024, 1
Threads integer >= 0 **0**, 1, 32
FlattenSteps boolean **off**, on, true, false
IgnoreFlattenSteps boolean **off**, on, true, false
============================== ===================== ===========================================================


Expand Down
5 changes: 3 additions & 2 deletions docs/user_guide/source/setting_up/source/cmake.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ To build ADIOS v2.x, clone the repository and invoke the canonical CMake build s

$ git clone https://github.com/ornladios/ADIOS2.git ADIOS2
$ mkdir adios2-build && cd adios2-build
$ cmake ../ADIOS2 cmake -DADIOS2_BUILD_EXAMPLES=ON
$ cmake ../ADIOS2 -DADIOS2_BUILD_EXAMPLES=ON
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
...
Expand Down Expand Up @@ -111,10 +111,11 @@ Optionally, run the tests (need to configure with ``-DBUILD_TESTING=ON`` cmake f
Total Test time (real) = 95.95 sec


And finally, use the standard invocation to install:
And finally, use the standard invocation to install (setting the install path beforehand):

.. code-block:: bash

$ cmake ../ADIOS2 -DCMAKE_INSTALL_PREFIX=/path/to/where/adios/will/be/installed
$ make install


Expand Down
3 changes: 2 additions & 1 deletion docs/user_guide/source/tutorials/helloWorld.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,14 +86,15 @@ Start editing the skeleton file `ADIOS2/examples/hello/helloWorld/hello-world_tu

adios2::Engine reader = io.Open("hello-world-cpp.bp", adios2::Mode::Read);
std::string greeting;
reader.BeginStep();
reader.Get(varGreeting, greeting);
reader.EndStep();
reader.Close();
return greeting;

.. note::

The ``BeginStep`` and ``EndStep`` calls are required when **reading** one step and multiple steps. We will see in
In Mode::Read, the ``BeginStep`` and ``EndStep`` calls are required when **reading** one step and multiple steps. We will see in
another tutorial how to read multiple steps. It's important to note that the ``BeginStep`` should be called **before**
all ``Inquire*`` / ``Available*`` function calls.

Expand Down
40 changes: 20 additions & 20 deletions source/adios2/common/ADIOSMacros.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
</pre>
*/
#define ADIOS2_FOREACH_ATTRIBUTE_PRIMITIVE_STDTYPE_1ARG(MACRO) \
MACRO(char) \
MACRO(int8_t) \
MACRO(int16_t) \
MACRO(int32_t) \
Expand All @@ -43,24 +44,24 @@
MACRO(uint64_t) \
MACRO(float) \
MACRO(double) \
MACRO(long double) \
MACRO(char)
MACRO(long double)

#define ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO) \
ADIOS2_FOREACH_ATTRIBUTE_PRIMITIVE_STDTYPE_1ARG(MACRO) \
MACRO(std::complex<float>) \
MACRO(std::complex<double>)

#define ADIOS2_FOREACH_ATTRIBUTE_STDTYPE_1ARG(MACRO) \
ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO) \
MACRO(std::string)
MACRO(std::string) \
ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO)

#define ADIOS2_FOREACH_STDTYPE_1ARG(MACRO) \
ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO) \
MACRO(std::string)
MACRO(std::string) \
ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO)

#define ADIOS2_FOREACH_TYPE_1ARG(MACRO) \
MACRO(std::string) \
MACRO(char) \
MACRO(signed char) \
MACRO(unsigned char) \
MACRO(short) \
Expand All @@ -75,10 +76,10 @@
MACRO(double) \
MACRO(long double) \
MACRO(std::complex<float>) \
MACRO(std::complex<double>) \
MACRO(char)
MACRO(std::complex<double>)

#define ADIOS2_FOREACH_PRIMITIVE_TYPE_1ARG(MACRO) \
MACRO(char) \
MACRO(signed char) \
MACRO(unsigned char) \
MACRO(short) \
Expand All @@ -93,18 +94,17 @@
MACRO(double) \
MACRO(long double) \
MACRO(std::complex<float>) \
MACRO(std::complex<double>) \
MACRO(char)
MACRO(std::complex<double>)

#define ADIOS2_FOREACH_COMPLEX_PRIMITIVE_TYPE_1ARG(MACRO) \
MACRO(float) \
MACRO(double) \
MACRO(long double)

#define ADIOS2_FOREACH_CHAR_TYPE_1ARG(MACRO) \
MACRO(char) \
MACRO(signed char) \
MACRO(unsigned char) \
MACRO(char)
MACRO(unsigned char)

#define ADIOS2_FOREACH_NUMERIC_TYPE_1ARG(MACRO) \
MACRO(short) \
Expand All @@ -123,6 +123,7 @@

#define ADIOS2_FOREACH_ATTRIBUTE_TYPE_1ARG(MACRO) \
MACRO(std::string) \
MACRO(char) \
MACRO(signed char) \
MACRO(unsigned char) \
MACRO(short) \
Expand All @@ -137,10 +138,10 @@
MACRO(double) \
MACRO(long double) \
MACRO(std::complex<float>) \
MACRO(std::complex<double>) \
MACRO(char)
MACRO(std::complex<double>)

#define ADIOS2_FOREACH_ATTRIBUTE_PRIMITIVE_TYPE_1ARG(MACRO) \
MACRO(char) \
MACRO(signed char) \
MACRO(unsigned char) \
MACRO(short) \
Expand All @@ -155,8 +156,7 @@
MACRO(double) \
MACRO(long double) \
MACRO(std::complex<float>) \
MACRO(std::complex<double>) \
MACRO(char)
MACRO(std::complex<double>)

/**
<pre>
Expand Down Expand Up @@ -185,6 +185,7 @@
#define ADIOS2_FOREACH_ATTRIBUTE_STDTYPE_2ARGS(MACRO) \
MACRO(std::string, string) \
MACRO(int8_t, int8) \
MACRO(char, char) \
MACRO(uint8_t, uint8) \
MACRO(int16_t, int16) \
MACRO(uint16_t, uint16) \
Expand All @@ -196,11 +197,11 @@
MACRO(double, double) \
MACRO(long double, ldouble) \
MACRO(std::complex<float>, cfloat) \
MACRO(std::complex<double>, cdouble) \
MACRO(char, char)
MACRO(std::complex<double>, cdouble)

#define ADIOS2_FOREACH_PRIMITVE_STDTYPE_2ARGS(MACRO) \
MACRO(int8_t, int8) \
MACRO(char, char) \
MACRO(uint8_t, uint8) \
MACRO(int16_t, int16) \
MACRO(uint16_t, uint16) \
Expand All @@ -212,8 +213,7 @@
MACRO(double, double) \
MACRO(long double, ldouble) \
MACRO(std::complex<float>, cfloat) \
MACRO(std::complex<double>, cdouble) \
MACRO(char, char)
MACRO(std::complex<double>, cdouble)

#define ADIOS2_FOREACH_MINMAX_STDTYPE_2ARGS(MACRO) \
MACRO(int8_t, int8) \
Expand Down
7 changes: 7 additions & 0 deletions source/adios2/core/VariableBase.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,13 @@ void VariableBase::SetMemorySelection(const Box<Dims> &memorySelection)
const Dims &memoryStart = memorySelection.first;
const Dims &memoryCount = memorySelection.second;

if (memoryStart.empty() && memoryCount.empty())
{
m_MemoryStart.clear();
m_MemoryCount.clear();
return;
}

if (m_SingleValue)
{
helper::Throw<std::invalid_argument>("Core", "VariableBase", "SetMemorySelection",
Expand Down
Loading