Product test executions should be independent

[kokosing]

Currently product tests create a docker image which then is reused by subsequent tests. The next product tests execution will reuse previously created images disregarding that product (presto admin package) changed or not.

[kokosing]

CC @ebd2

[kokosing]

The work around this problem is to run below command before each product tests execution:

make clean-test-containers clean-test dist

[cawallin]

When you run make test, it runs clean-test-containers and clean-test before. The issue is when you run multiple times in intellij without cleaning.

[ebd2]

I'm not sure I understand the issue. By design, clean-test cleans up any images that are tagged teradatalabs/patest/*. When we run the tests, we need images in a couple basic configurations: bare, with presto-admin, and with presto. The first test that needs an image of a particular type creates it and tags it so that future tests don't incur the cost of creating the image from a bare image. This saves a lot of time installing prestoadmin and presto.

The clean-test target can only clean up images that don't have any running containers. If you're finding that you have to run clean-test-containers in order for clean-test to succeed, it means something has leaked a running container. Usually, as @cawallin said, that happens if you're debugging in PyCharm and stop a test instead of running it to completion so the cleanup can happen automatically.

This works well, except for the case where you're iterating on presto-admin itself; In that case, you'll hit the issue where images in teradatalabs/patest have stale copies of presto-admin installed. Even if we could solve this issue for make by teaching it about the dependency between pa and those two docker images (and we probably could, although when I originally set up the caching, getting an mtime out of docker in a parseable format was an issue), we still would solve the problem of running the tests through PyCharm, because it doesn't go through make.

Fixing this in docker_cluster.py might be possible if we could compare the mtime of the installer with that of the docker images that have pa installed. The problem is, nothing understands the dependencies between the individual .py files and the installer well enough to reliably decide whether or not to rebuild the installer, so when we build it, we build it unconditionally.

If we solve that problem (probably via make), we're still stuck with the issue that when we build the installer for the product tests, we do so in a docker container, probably to avoid the fun and excitement of dealing with all the fun of trying to build locally and dealing with the quirks of everybody's individual system python/venv situation.

I'll take a look at this this week since I keep waking up at 5:30 and I don't have anything to do until 10:30 at the hadoop summit...

[kokosing]

I encountered this problem running individual product test sequentially from Intellij. With each iteration I changed product (presto-admin).

Moreover, I see running containers leak when doing such tests, but this is another story.

Anyway, this issue has minor priority to me. Please do not investigate to much time into it, unless you have nothing better to do ;)