Unit Test `test_experiment_result` tolerance check fails in `main` and varies based on development setup/platform.

[danielbarcklow]

Unit Test test_experiment_result tolerance check fails in main and varies based on development setup/platform.

This issue can be recreated by removing the @pytest.mark.skip from test_experiment_result in tests/unittests/test_experiment.py. It was recreated locally and also in the github status checks.

ablation/tests/unittests/test_experiment.py

Line 127 in 882932a

def test_experiment_result(config_file):

The differences between values obtained from the test experiment varies from that which is found in test_result.csv. It was observed that the score columns only pass with tolerance 1e-1 or 1e-2. It was also observed that the test results vary across development setup/platform. Modifications are needed to reduce this variability and allow test_experiment_result to pass successfully with an appropriate tolerance.

[danielbarcklow]

This issue is still relevant.