Why are brew update and brew upgrade so unreliable nowadays? Download errors and timeouts all the time for several weeks now.

[ChrisJ60]

Output of brew config

HOMEBREW_VERSION: 4.0.14
ORIGIN: https://github.com/Homebrew/brew
HEAD: ac0663ae5d2d4c76bb62363592d77dbad49ac30c
Last commit: 6 days ago
Core tap JSON: 22 Apr 07:06 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_EDITOR: /usr/bin/vi
HOMEBREW_MAKE_JOBS: 12
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: dodeca-core 64-bit arm_blizzard_avalanche
Clang: 14.0.3 build 1403
Git: 2.40.0 => /opt/homebrew/bin/git
Curl: 7.87.0 => /usr/bin/curl
macOS: 13.3.1-arm64
CLT: 14.3.0.22.14.1
Xcode: 14.3
Rosetta 2: false

Output of brew doctor

Your system is ready to brew.

Description of issue

I've been using HomeBrew for several years, mostly on Intel Macs but more recently (past year or so) also on Linux and ARM Macs. I have always found it easy to use and very reliable but these past few months things seem to have degraded a lot. I am now finding that 'brew update' and 'brew upgrade' are:

a) Very slow in general

b) Regularly fail (as in 90%+ of the time).

The failures are mainly:

Error: Failed to download https://formulae.brew.sh/api/formula.jws.json

or

Error: Failed to download https://formulae.brew.sh/api/cask.jws.json

But I do frequently often see errors with downloading items from https://ghcr.io and https://pkg-containers.githubusercontent.com

I'm sure someone will say that this is just problem with my Internet connection, but I really do not think so. M
I have a high grade connection which is monitored 24x7 for all kinds of issues. I do sometimes see transient issues (very infrequently) but otherwise the connection is rock solid and is in use 24x365. if there were some general issues then I would know about them.

I hope that someone somewhere can help figure out thsi issue as currently HomeBrew is almost unusable for me, which is a severe problem...

[SMillerDev]

Try downloading those files outside of brew using your 24x7 monitoring system to see if it might be an issue with your connection to GitHub. Additionally, since those domains are hosted by GitHub you might want to check https://www.githubstatus.com/ if you're having issues.

[ChrisJ60]

After further test I can confirm that the effect (or not) of '--compressed' was a red herring. However, after many more experiments and diagnostics I now know the cause of the problem. First, let me recap my setup and the issue.

Here at home I have 8 Macs of various kinds (5 are Intel, 2 are M1/M2 based). Most are running macOS Ventura 13.3.1 except for one (very) old MacBook still running Big Sur. I also have one Linux system (CentOS 8). My home network is fully dual-stack and most of my internal traffic runs over IPv6. My Internet connection is also dual-stack and ~55-60% of that traffic runs over IPv6. The network has been like this for some years. All of these systems have HomeBrew installed (I have used HomeBrew for many years) and are set to perform regular (weekly) automated 'brew update; brew upgrade' operations.

My Internet connection is a cable connection running at 1 Gbit/s with average speeds always > 850 Mbit/s. I have automated pro-active monitoring in place for my home network (both internal and external); regular IPv4 and IPv6 pings, HTTP/HTTPS access and file downloads, SSH access to cloud systems, regular speedtests etc. so I rapidly become aware of any connectivity issues or speed degradation.

For some time now (at least a few months I think) I have noticed that 'brew update' and 'brew upgrade' have become very, very slow (when they work at all) and most of the time (~90%) they fail. This is affecting all of my systems, so recently I started to investigate this.

The cause is that GitHub seems to have poor/problematic IPv6 connectivity. All modern OS and tools prefer IPv6 connectivity when it is available. It seems that github.io is available via IPv6; specifically homebrew.github.io resolves as follows:

homebrew.github.io
185.199.110.153
185.199.109.153
185.199.111.153
185.199.108.153
2606:50c0:8002::153
2606:50c0:8003::153
2606:50c0:8000::153
2606:50c0:8001::153

I don't know if IPv6 is something relatively new for github.io but I do know that it is the problem (for me at least). If I change my internal DSN such that the github.io domain only resolves to IPv4 addresses (thus preventing any of my internal systems from using IPv6 to connect) and after flushing the resolver cache on all systems then Hebrew is restored to its previous good performance and reliability. Switching back to allowing IPv6 (and flushing the resolver cache on all systems) again results in terrible performance and failures.

This is not a general IPv6 connectivity issue for my site since, as mentioned, a lot of my external traffic runs over IPv6 without any reliability or performance issues. The problem is specific to github.io. Here are IPv4 and IPv6 traceroutes just FYI.

traceroute to 185.199.110.153 (185.199.110.153), 64 hops max, 72 byte packets
1 router.lan.thejenkinsfamily.org.uk (10.0.200.1) 0.870 ms 0.341 ms 0.310 ms
2 10.112.32.129 (10.112.32.129) 12.007 ms 17.360 ms 17.838 ms
3 winn-core-2a-xe-830-0.network.virginmedia.net (62.253.122.149) 17.688 ms 11.454 ms 9.200 ms
4 * * *
5 eislou2-ic-4-ae0-0.network.virginmedia.net (62.254.59.130) 13.596 ms 12.071 ms 12.521 ms
6 157.52.127.8 (157.52.127.8) 14.037 ms 11.920 ms 12.310 ms
7 cdn-185-199-110-153.github.com (185.199.110.153) 11.564 ms 13.248 ms 10.399 ms

traceroute6 to 2606:50c0:8002::153 (2606:50c0:8002::153) from 2001:470:1f09:2df::40, 64 hops max, 16 byte packets
1 router.ext.thejenkinsfamily.org.uk 0.410 ms 0.305 ms 0.297 ms
2 tunnel238782.tunnel.tserv5.lon1.ipv6.he.net 15.265 ms 13.240 ms 12.907 ms
3 e0-19.core2.lon2.he.net 13.199 ms 18.694 ms 12.788 ms
4 * * *
5 * * *
6 port-channel2.core2.lon3.he.net 23.385 ms 13.568 ms 14.038 ms
7 * * *
8 2606:50c0:8002::153 21.238 ms 10.968 ms 10.229 ms

Nothing obviously problematic. Average IPv4 ping to home-brew.github.io is 12.9 ms while for IPv6 it is 19.8 ms (with show zero packet loss). So again nothing obviously bad there either.

I suspect that the issue is on the GitHub side. Luckily I have a workaround (force access to github.io to use IPv4 only) but you might want to take this up with GitHub since it may affect more people over time.

[SMillerDev]

I think it's a much better option if you report it to GitHub since I can't help debug a problem I'm not experiencing.

[ChrisJ60]

I have reported it. Just FYI, I also had to force githubusercontent.com to be IPv4 only as well to fully fix the issue.

[cbruegg]

@ChrisJ60 Did you ever find out more? I'm facing the same issue. About half of my IPv6 non-browser requests to GitHub fail, but only via my DSL connection. Via mobile data it's fine.

[ChrisJ60]

@cbruegg Sadly no. However, last year I changed my ISP so my IPv6 connectivity to the Internet changed from using a Hurricane Electric (tunnelbroker.net) 6in4 tunnel to native IPv6 and since then no more IPv6 issues with Github. So it seems to me it was possibly some issue related to the 6-in-4 tunnel. It did have a smaller than normal MTU (1480 versus 1500), but whether it was that or some routing issue, or ... I never did manage to pin down.