feat: unbloat, reduce image size

[MQ37]

These changes bring image size from ~1.4 GB to ~600 MB.

[matyascimbulka]

@MQ37 Could you give us links to the build as well as some runs (STANDBY and STANDALONE) using it?

[MQ37]

@MQ37 Could you give us links to the build as well as some runs (STANDBY and STANDALONE) using it?

Sure 👍

Build link: https://console.apify.com/actors/B2VM9FhWyxLEMb7tm/builds/1.0.25/log
Normal run: https://console.apify.com/view/runs/osLeK8ehOOosBGuot
Standby run: https://console.apify.com/view/runs/UQmDACw95ugfnTSDm

[matyascimbulka]

Wow, this is really cool. Reducing the image size by 1GB without reducing functionality is impressive. Thank you.

[metalwarrior665]

Have you checked if this makes sense to do for the Apify Docker base images?

[MQ37]

Have you checked if this makes sense to do for the Apify Docker base images?

Since it uses distroless image where we need to copy all the requirements I don't think it makes sense and is viable to make this a base image for Actors - they can use some specific libs or different versions.

[jirispilka]

Yeah, this is great. It will speed up start significantly (when image is not cached)

I think it could make sense to use it as a base image for certain use cases — or to include it in the templates as an example.

Assuming there isn’t some catch I’m not seeing right now? @metalwarrior665 can you think of any?
Or discuss this with someone from tooling?

We definitely need to test it properly before releasing it in the RAG Web Browser.

[metalwarrior665]

This should definitely go through tooling/platform review as the base Dockerfiles are cached/preloaded in some way.

Btw: When we dropped image size in Google Maps from 2 TB to 400 MB (browser -> cheerio), we saw no improvement in startup time. I think the caching just takes care of that. So I would measure what benefit we actually want - usually startup times and build times.

[MQ37]

just found out that when I tested I probably tested locally with the playwright which does not use the docker image and when copyting from builder image I forgot to copy the browser with it's requirements 🤦 When including the browser and all the libs we can realistically at maximum save ~300 MB which I don't know is worth the overhead (we would have to solve loading of the dynamically linked libraries). But we can look into the distroless images for other, simpler use cases for node only Actors.

[MQ37]

just found out that when I tested I probably tested locally with the playwright which does not use the docker image and when copyting from builder image I forgot to copy the browser with it's requirements 🤦 When including the browser and all the libs we can realistically at maximum save ~300 MB which I don't know is worth the overhead (we would have to solve loading of the dynamically linked libraries). But we can look into the distroless images for other, simpler use cases for node only Actors.

Taking this back, hacked it to work (its really hacky) the image is ~600 MB and playwright works 👍 More than 300 MB but way less than original 1.3 GB.

[MQ37]

Build: https://console.apify.com/actors/B2VM9FhWyxLEMb7tm/builds/1.0.37/log

Runs
Cheerio normal: https://console.apify.com/view/runs/ZgqKfUbfwrpJGrPlN
Playwright Firefox: https://console.apify.com/view/runs/HLqHFA2BmTWjVK5IO
Standby both: https://console.apify.com/view/runs/uHWmjNsJG1lI5A2gE

[metalwarrior665]

Cool! Could you still make a post in #product-dev-tools and present this achievement before we merge? We can take this Actor as PoC of the distroless approach but it should be approved by the Tooling team. There might be some assumptions we have about our base Dockerfiles

[metalwarrior665]

@MQ37 If you look at the first 2 log lines of your runs, the Docker pull takes 6+ seconds which is terrible. Usually it takes between 1-2 seconds. You can try a few more tests but I think this is cache miss and unless we make your distroless approach a new standard, we have to pause this.

[jirispilka]

Yes, It would mean that the image is not cached. If you ran it more often, then the time should improve.

Since the rag web browser is used 2-5k runs per day, we need to caution we non-standard changes.
Therefore, I'm holding with media blocking PR #54 , I need more time to properly tested it

[MQ37]

@MQ37 If you look at the first 2 log lines of your runs, the Docker pull takes 6+ seconds which is terrible. Usually it takes between 1-2 seconds. You can try a few more tests but I think this is cache miss and unless we make your distroless approach a new standard, we have to pause this.

Did a few more tests and the pull is way faster when I hit the cache. But I noticed the startup time, time from "Starting Docker container." to the printed system info in logs, is slower/unstable (higher std). Which is what we want to optimize for.

Original RAG Browser:
Pull to Start - Mean: 0.173s, Median: 0.179s, Std: 0.029s
Start to Sys - Mean: 1.607s, Median: 1.586s, Std: 0.177s

RAG Browser Distroless:
Pull to Start - Mean: 4.545s, Median: 4.028s, Std: 4.744s
Start to Sys - Mean: 2.123s, Median: 1.597s, Std: 0.981s

[metalwarrior665]

We need to optimize from Actor start to system info but these parts will depend on multiple teams.

Platform team has been working on optimizing the base Dockerfiles for years so you will definitely need to talk with them. The problem is a bit of a catch-22 as they will not want pre-cache images that are not the base for everybody. And even if we would change it for everyone, there will still be old versions and people not upgrading. But I'm just speculating, the best is to talk to them.

And then tooling team can probably influence how fast Node.js or Python process starts

[MQ37]

We need to optimize from Actor start to system info but these parts will depend on multiple teams.

Platform team has been working on optimizing the base Dockerfiles for years so you will definitely need to talk with them. The problem is a bit of a catch-22 as they will not want pre-cache images that are not the base for everybody. And even if we would change it for everyone, there will still be old versions and people not upgrading. But I'm just speculating, the best is to talk to them.

And then tooling team can probably influence how fast Node.js or Python process starts

Talked to @jirimoravcik, and the first thing is we need more test runs (thousands) to get statistically significant results. Jirka tried a few runs, and they seem okay to him - https://apify.slack.com/archives/CD0SF6KD4/p1743774919039049?thread_ts=1743622813.549859&cid=CD0SF6KD4. I already have this on my to-do list, so I will run the tests when I have time and report the results.