How long should task caches stay around?

[ostenbom]

Hi there!

We're using the task caches input to cache our node_modules when compiling JS, which is an amazing speed boost (~50%!). We're deploying our concourse with control-tower.

The cache directories don't seem to hang around for very long though, maybe a few hours at most? Our worker disk usage is very low - around 20%, so there is plenty of space these directories to lie around.

Can anything be done to make these hang around for longer? Is this configurable at any level?

I noticed that "volumes" are consistently being removed. Are these "volumes" also the cache directories?

Any help or advice on where to look is greatly appreciated!

[vito]

Caches are indeed stored in volumes, but so is everything else - so it's normal for volumes to be GCed pretty frequently, and that's not necessarily an indicator that your caches are being GCed. If you're curious, I did a bit of a write-up on volumes in this discussion.

Specifically, it mentions that...:

Task caches are garbage collected when the task step is removed from its build plan (identified by the task name), when its job is removed from the pipeline, or when its pipeline itself is removed. They are kept while the pipeline or job is paused.

So as long as the task hasn't been removed, its caches should stick around indefinitely. Note that caches are per-worker, so if you've warmed the cache by running a task on worker A but then the next build runs the task on worker B, it'll be a cache miss.

What behavior are you observing that makes it seem like they're going away?

[ostenbom]

I see, thanks for pointing me to that @vito, I haven't been removing/renaming tasks so I don't think that's it.

I'm observing our yarn install step taking an extra 3 minutes on runs every few hours (even though the packages have not changed). I figured the only thing that would cause that much extra time would be re-downloading a bunch of packages (eg. cache has been invalidated). We only have a single worker, which is an xlarge AWS EC2 instance, and as I mentioned before I don't thing disk space is the issue.

I should put some extra effort into debugging this it seems. I'll configure the pipeline to list the amount of files in the /node_modules directory on every run, that should give a better indication if it's actually a wiped cache directory that's causing the extra time, or me misunderstanding the intricacies of yarn.

[jamieklassen]

If you're running 6.0, you may be encountering a known bug: #5485 (comment) never mind, task caches never get streamed.

[ostenbom]

I have noticed that every time we set-pipeline, the task caches for sure disappear. Is that a bug? Should I try to make a reproduction and file an issue if so?

[vito]

@ostenbom Hm, that might be fixed by #5497 - there was indeed a bug where we wouldn't walk recursively through the plan: to find all the tasks with caches. So if you had any task steps nested, perhaps under an in_parallel: or other step, I believe we would reap their caches each time.

[ostenbom]

That fits our use case! Thanks