Currently we have integration tests which creates a trainite project and then runs them end to end.
One problem I noticed is that the integration tests uses the same testing environment like dev dependencies installed for testing, the tests will pass even if the generated project contained the broken dependency.
One solution could be to isolate the creation and running of the project but then we have to install dependency again and again that's time consuming and results of them cant be cached.
Also integration testing involves actuually model training , even though small for atleast few to 1 epoch which is expensive for tests.
I am not sure if we should even have the integration testing whats your thoughts @vfdev-5 .
Currently we have integration tests which creates a trainite project and then runs them end to end.
One problem I noticed is that the integration tests uses the same testing environment like dev dependencies installed for testing, the tests will pass even if the generated project contained the broken dependency.
One solution could be to isolate the creation and running of the project but then we have to install dependency again and again that's time consuming and results of them cant be cached.
Also integration testing involves actuually model training , even though small for atleast few to 1 epoch which is expensive for tests.
I am not sure if we should even have the integration testing whats your thoughts @vfdev-5 .