General Issue: Verify template results

[StefanThoma]

Background Information

It would be ideal to have a way to verify whether the outputs of the templates have changed.

One idea would be to write unit tests, so that running the unit tests would already check that.
I think it could be done like this in a unit test:

1. Create new file using template
2. (Potentially) modify the new file to reduce the dataset size used
3. Source the file
4. Compare the newly created file to the one created last time using diffdf

Definition of Done

Either find a way of verifying the outputs of the templates, or to decide that this is not necessary.

[bms63]

I would be hesitant to make these into unit tests as they are very big files to run.

The current CI check for the templates is a separate workflow. I was thinking we would build off that one - pharmaverse/admiralci/.github/workflows/check-templates.yml@main

[StefanThoma]

I see your point.
The downside with a separate run is that you cannot run it locally, afaik.

[bms63]

What about how Edoardo approached that new check in the blog? He sources a file in the workflow - maybe something to consider? Then you could grab the file and run it

[StefanThoma]

Yeah that would not be a bad idea. Let me experiment a bit.

[bundfussr]

The downside with a separate run is that you cannot run it locally, afaik.

We could implement a function which runs a specified list of templates and compares the results with the references datasets (in pharmaverseadam?).

This function could then be used in the CI/CD workflow and also be called locally.

[StefanThoma]

The downside with a separate run is that you cannot run it locally, afaik.

We could implement a function which runs a specified list of templates and compares the results with the references datasets (in pharmaverseadam?).

This function could then be used in the CI/CD workflow and also be called locally.

That sounds like a good idea.

[bms63]

The downside with a separate run is that you cannot run it locally, afaik.

We could implement a function which runs a specified list of templates and compares the results with the references datasets (in pharmaverseadam?).
This function could then be used in the CI/CD workflow and also be called locally.

That sounds like a good idea.

So - pharmaverseadam got re-deployed last week so things might not line up that well with this method going forward. Feels like it is getting decoupled form admiral.

I think we should compare templates runs via stuff in main and stuff in a branch. Could this action just run if code in the templates is updated in the branch? I feel like that is possible ????