Catch the error of undefined pipe sections within the pipeflow

[dlohmeier]

Is your feature request related to a problem? Please describe.

When running the following code snippet:

from pandapipes.networks.simple_heat_transfer_networks import heat_transfer_delta
import numpy as np
net = heat_transfer_delta()
net.pipe.at[0, "sections"] = np.nan
pandapipes.pipeflow(net)

The following exception is raised:

/home/daniel/workspace/pack/.venv/lib/python3.12/site-packages/pandapipes/component_models/pipe_component.py:73: RuntimeWarning: invalid value encountered in cast
  return np.array(net[cls.table_name()].sections.values).astype(np.int32)
/home/daniel/workspace/pack/.venv/lib/python3.12/site-packages/pandapipes/component_models/pipe_component.py:73: RuntimeWarning: invalid value encountered in cast
  return np.array(net[cls.table_name()].sections.values).astype(np.int32)
Traceback (most recent call last):
  File "/home/daniel/workspace/pack/.venv/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3747, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-86-d69b2c49678a>", line 1, in <module>
    pandapipes.pipeflow(net)
  File "/home/daniel/workspace/pack/.venv/lib/python3.12/site-packages/pandapipes/pipeflow.py", line 70, in pipeflow
    initialize_pit(net)
  File "/home/daniel/workspace/pack/.venv/lib/python3.12/site-packages/pandapipes/pf/pipeflow_setup.py", line 388, in initialize_pit
    pit = create_empty_pit(net)
	  ^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/workspace/pack/.venv/lib/python3.12/site-packages/pandapipes/pf/pipeflow_setup.py", line 429, in create_empty_pit
    pit = {"node": np.empty((node_length, node_cols), dtype=np.float64),
	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
numpy._core._exceptions._ArrayMemoryError: Unable to allocate 288. GiB for an array with shape (2147483658, 18) and data type float64

Which is extremely confusing for a user who doesn't know where this might come from.

Describe the solution you'd like
This issue does not appear exactly pipeflow related and is hard to understand. The issue lies in the BranchWOInternals component:

@classmethod
def create_branch_lookups(cls, net, ft_lookups, table_lookup, idx_lookups, current_start, current_table, internals):
    """
    Function which creates branch lookups.

    :param net: The pandapipes network
    :type net: pandapipesNet
    :param ft_lookups:
    :type ft_lookups:
    :param table_lookup:
    :type table_lookup:
    :param idx_lookups:
    :type idx_lookups:
    :param current_start:
    :type current_start:
    :param current_table:
    :type current_table:
    :param internals:
    :type internals:
    :return:
    :rtype:
    """
    internal_branches = cls.get_internal_branch_number(net)
    internal_branches_num = int(np.sum(internal_branches))
    ...

The internal_branches_num becomes an extremely large number. I would add a check here and either throw a clear exception or replace all nans with 1 with at least a logger warning.

Describe alternatives you've considered
It could be left to the user or an external function to identify this problem, but this might lead to machines hanging up, just due to a small input error. I would prefer a clear exception over a silent warning with assumed value of 1, but either seems mandatory.

[dad616610]

I see 2 solutions to this:

1. As you described we raise an error with a clear message.
   But while we've improved on an error message, the user still needs to figure out where nans have crept in

2. We use some DataFrame validation code to ensure our DataFrames have the data we expect.
   We would place this validation code to places, where user creates net entries (create_* functions, functions for loading the net from disk) and right before we start solving the net (pipeflow function)

   This way we could pinpoint the user to the erroneous input more precisely (though, the preciseness isn't that different from the 1st option). But more importantly, we could validate the whole DataFrame, not just one field -- this would ensure that internally we're working with the data we expect. This might get us less astype calls, cleaner code, etc.

For the option 2 I found Pandera. It lets you write a schema for DataFrame and use it to validate a DataFrame. A quick tour: try pandera in jupiter notebook, quick start section in docs

An example of applying Pandera to this case

import pandera.pandas as pa
import pandas as pd
import numpy as np

df = pd.DataFrame({"sections": [0, 1, 2]}, dtype=np.uint32)
df.loc[2, "sections"] = np.nan

schema = pa.DataFrameSchema({"sections": pa.Column(np.uint32, pa.Check.ge(1))}, name="PipeSchema")

schema.validate(df, lazy=True)

And here's the output I get:

pandera.errors.SchemaErrors: {
    "SCHEMA": {
        "SERIES_CONTAINS_NULLS": [
            {
                "schema": "PipeSchema",
                "column": "sections",
                "check": "not_nullable",
                "error": "non-nullable series 'sections' contains null values:2   NaNName: sections, dtype: float64"
            }
        ],
        "WRONG_DATATYPE": [
            {
                "schema": "PipeSchema",
                "column": "sections",
                "check": "dtype('uint32')",
                "error": "expected series 'sections' to have type uint32, got float64"
            }
        ]
    },
    "DATA": {
        "DATAFRAME_CHECK": [
            {
                "schema": "PipeSchema",
                "column": "sections",
                "check": "greater_than_or_equal_to(1)",
                "error": "Column 'sections' failed element-wise validator number 0: greater_than_or_equal_to(1) failure cases: 0.0"
            }
        ]
    }
}

While being quite verbose, it does indicate 3 problems with the DataFrame:

• we have np.nan
• our dtype is float64, instead of uint32
• we have a value < 1 (because of pa.Check.ge(1))

Personally I'd go with the 2nd approach because we can validate our DataFrames more rigorously and at places where we see fit, not where we're forced to