Feature request: cache job attachments between jobs

[crowecawcaw]

Describe the problem

Currently, job attachments are cached within a session and there's one session per job. If two tasks run on a worker and use the same job attachments, the attachments will only be downloaded once. If two different jobs use the same job attachments though, the job attachments must be downloaded twice. The downloads could be reused to optimize run times.

Proposed Solution

Make a job attachments cache on the worker that's associated with a queue instead of a session so that as long as jobs belong to the same queue and are within the same permissions boundary, they benefit from job attachment caching.

Example Use Cases

Submitting multiple jobs that use the same job attachment inputs. For the use case I have in mind, we're submitting jobs with large gen AI models that change rarely and are reused between jobs.

[leongdl]

This is an interesting case we have considered but there are some tricky cases.

1. While we can build a local cache of the file, the file must be first downloaded, and then a duplicate is copied to the session folder. Sessions can modify input files, which can pollute the cache.
2. The permissions of the cache folder must be secure, it cannot leak data that comes from different queues, so the cache permission design on the worker is important. [Because each worker can be from a fleet connected to >1 queue]

[komapa]

Second this requests. In out case, our S3 data-out cost is 1/3 of our rendering compute costs.
Another suggestion is to make the worker agent S3 download caching proxy friendly. If we set HTTP_PROXY or HTTPS_PROXY, can you use that with the S3 client to effectively go through the HTTP proxy and allow it to cache the files?