Limited coverage in regression testing

[RaphaelGebhart]

Issue: Limited coverage in regression testing

Our default regression testing workflow evaluates states only. In case of the TFS this is primarily the mass flow rate.

With this selection simulation variables, e.g. the thermodynamic state, is often not sufficiently represented. This is particularly problematic for models in ThermofluidStream.Idealized, where the mass flow rate is frequently an input and not a simulation result. As a result, regression testing may fail to detect relevant behavioral changes.

Proposed way forward

Adopt a CSV-based comparison as the default regression testing workflow:

• Set up the csv-compare workflow
• Generate reference datasets
• Provide a concise, beginner-friendly how-to guide

Introduce dedicated states for selected variables (e.g. via Modelica.Blocks.Continuous.FirstOrder) to make them accessible for regression testing. After further discussion, we decided not to pursue this approach.

[CorentinLepais]

Since the regression testing tool is simply calling the model management library to do the testing, I do not know if it is easy to extend it to include additional variables. I think the check function from the model management library is comparing the states automatically and I do not know if you can add other variables. At least it does not seem to be the case when doing a regression test manually by calling the function Check from Dymola directly. Therefore I would go for the second option, by adding in the text layer filtered variables which are considered as important for the test model and force their selection as states by Dymola.

Another possibility could maybe to use the Test library from Dymola which seems to have tools for directly comparing signals and variables against reference files instead of just states but it might take a lot of time to create a new regression tool which works with this library. Similarly, we might find some open-source libraries for regression testing but it means taking time to work with a new tool.