Tracking Issue: Concurrent Downloads

[reitermarkus]

Concurrent Downloads

• Implement MVP of concurrent downloads for brew fetch.

  Implemented in Implement concurrent downloads in brew fetch. #17756.

• Improve output logic.

  For simplicity, the output currently only uses at most terminal height - 1 lines due to a trailing newline.

  Implemented in Use full terminal height for concurrent output. #19194.

• Handle custom download strategies.

  These may not be possible to run concurrently so let's handle them differently.

• Implement graceful cancellation of downloads.

  Currently, cancelling downloads can only be done by killing the whole thread pool, i.e. the sledgehammer approach. Proper cancellation based on https://ruby-concurrency.github.io/concurrent-ruby/master/Concurrent/Cancellation.html should be implemented, making it possible to neatly show successful, failed and cancelled downloads.

• Replace direct puts output with different output formatters for serial/concurrent download output.

• Implement concurrent downloads for brew install, brew reinstall, brew upgrade etc. with a --concurrency flag.

• Implement global concurrent downloads with a public/documented/supported HOMEBREW_DOWNLOADS_CONCURRENCY=<int> opt-in variable.

• Limit concurrent connections per host to avoid overloading smaller (i.e. non-CDN) web servers.

• Enable concurrent downloads by default, i.e. change default value for --concurrency flag.

[MikeMcQuaid]

Thanks for the write-up @reitermarkus! Makes sense to me. Please focus on getting #17756 merged ASAP.

• In order to implement the following parts, custom download strategies need to be deprecated. Their API surface is too big since they depend on many private methods from their superclass, making changes here practically impossible without breaking things.

We can't/won't do this. Things as basic as "download a release from a private GitHub repository" require this. Instead we should expect to provide suboptimal/poor/no progress reporting, graceful cancellation for these strategies.

[MikeMcQuaid]

Thoughts so far:

• missing support for falling back to earlier bottle tags

$ brew fetch --retry --concurrency=10 boost qt
Fetching: boost, qt
Warning: Bottle for tag :arm64_sequoia is unavailable.
Warning: Bottle for tag :arm64_sequoia is unavailable.

• the above message should probably also warn which bottle it relates to
• we should limit the hosts we'll download in parallel from to an allowlist so we don't e.g. try to download the 10 files at once from a poor personal web server when building from source

[reitermarkus]

We can't/won't do this. Things as basic as "download a release from a private GitHub repository" require this.

If it's that basic, we should support it using an official download strategy. In any case, we don't want to maintain two different types of download strategies.

[MikeMcQuaid]

@reitermarkus We historically supported many more types of download strategies that we didn't use e.g. those for private resources. Problem is: when we don't actually use and rely on them ourselves, they end up bitrotting.

In general: even if we were to support them: I'd rather not break an existing, public (implicit or not) API for people for new functionality if we can just degrade to not support that functionality there instead.

[Bo98]

This will probably be easier to discuss with a demonstration of what exactly needs changing in download strategies that would be a breaking change.

[Bo98]

• Currently, cancelling downloads can only be done by killing the whole thread pool, i.e. the sledgehammer approach. Proper cancellation based on https://ruby-concurrency.github.io/concurrent-ruby/master/Concurrent/Cancellation.html should be implemented, making it possible to neatly show successful, failed and cancelled downloads.

Concurrent::Cancellation isn't stable yet so isn't actually available in concurrent-ruby. It's a WIP feature in a separate beta gem for now.

Worth noting that download strategies already need to support Ctrl+C Interrupt exceptions I think. So Thread#raise Interrupt shouldn't be surprising.

[MikeMcQuaid]

Thanks for updates @reitermarkus. Have made some edits but feel free to discuss any down here.

[reitermarkus]

We don't need HOMEBREW_NO_DOWNLOADS_CONCURRENCY since HOMEBREW_DOWNLOADS_CONCURRENCY=1 would do the same thing.

[MikeMcQuaid]

We don't need HOMEBREW_NO_DOWNLOADS_CONCURRENCY since HOMEBREW_DOWNLOADS_CONCURRENCY=1 would do the same thing.

Yeh, that makes sense. Wondering if we'll want to have a HOMEBREW_DOWNLOADS_CONCURRENCY=auto or something though to avoid requiring users to pick how many threads to use (as we pick it for them for e.g. make)

[reitermarkus]

Wondering if we'll want to have a HOMEBREW_DOWNLOADS_CONCURRENCY=auto or something

I think not specifying HOMEBREW_DOWNLOADS_CONCURRENCY or --concurrency would do that anyways once concurrent downloads are the default.

[MikeMcQuaid]

I think not specifying HOMEBREW_DOWNLOADS_CONCURRENCY or --concurrency would do that anyways once concurrent downloads are the default.

Yup, I guess it just seems like a nicer interface than requiring users to figure out and specify a sensible number here before its default. Not a blocker or anything, just a nice-to-have.