429 Too Many Requests permanently with SwarmCD

[vhaudiquet]

Hello,

I'm using SwarmCD on my home production docker VM now, and I've noticed some issues.

One that is kind of annoying is that I'm always really close to the docker pull rate limit, or even above it and can never use docker pull anymore.

I see a lot of

429 Too Many Requests
toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit

At first, I thought that it was normal because I had redeployed everything on a newer machine.
However, it has been multiple days now and it is still like that.

I'm not sure if SwarmCD is the problem or if docker swarm is the problem, as I was using compose before.
But something seems to be agressively re-pulling or at least asking the daemon to pull images constantly. I still have some containers using latest images, could it be that something is always trying to see if there is a more recent one ?

It seems I'm only running 45 containers at the moment, which is not gigantic and should definetely not trigger pull request limits I think.

root@docker-r740:~# docker ps | wc -l
45

Anyway, I'm not sure if the issue is really within SwarmCD, so I wanted to ask other users if they had similar experience :)

Thank you !

[viktor-fortuna]

hello.

I think, i found a reason that problem.
When you using only the tag in the image, it would constantly redeploy.
You need to specify it with @sha256:

like image: ".../nginx:latest@sha256:68dca76..."

I think this is Docker or Swarm behavior

[sanzoghenzo]

Hi, some related but random-ish facts:

• docker stack deploy (used by swarmcd) doesn't re-pull the images automatically, in fact there is an open issue since 2017 that asks to enable this behavior.
• PR Persist current revision to a SQL-lite db and only update the stack if the revision and stack content changed #110 should reduce the stack update frequency by swarm-cd
• I see that the unauthenticated rate limit is 100 pull per IPv4 address or IPv6 /64 subnet every 6 hours. You can register with a free account (and pass the auth to docker in the VM) to double that limit.
• Bear in mind that the address is the public one, so you need to count all the other machines that use docker, if any.
• If your ISP uses CGNAT and your "neighbor" runs an HomeLab too, you're pretty much out of luck - ask for a dedicated IP.
• I never reached the pull rate limit, even though I don't specify the sha. But I only have 40 containers and only 23 of them are from docker hub.

[tgaugry]

I also had the same problem - dependencies between services could cascade, and if a service couldn't reach a stable status, swarm will redo an http GET even if it has the image.
For a reason I am not certain of (but I assume it's the number of errors), swarm then starts to move containers between hosts, which add to the problem...

One "solution" I have is to use a cache for dockerhub. It doesn't prevent all "useless" calls to dockerhub, but it prevent most of them. I didn't get the issue since.

Another thing that could mitigate the issue is not triggering swarm-cd if there hasn't been new commits. Not sure it won't create a slew of other problems though (like, if you remove a container or a stack to reset/clean them, they won't get restarted until you do a commit...)

[vhaudiquet]

Hello, just answering to your different points @sanzoghenzo, on top of my head:

• This is weird, as the images I have with :latest are redeployed automatically by SwarmCD at least. My own website and tools are redeployed automatically as soon as I push a new image to my private container registry.
• That PR should solve my issue, and many others that I have (micro-latencies due to constant redeployment, same as in the PR). This is the way to go IMHO, and having that on-disk is kind of better than in-memory if you ask me, because it covers more use cases while also not being that heavy
• I don't want to login to docker on the server that runs SwarmCD, I'm keeping my account on my personal computer so that I can still use docker to pull images there :)
• There are other machines using docker, but not using it a lot, and the configuration has not changed since the deployment of SwarmCD.
• My ISP gives me a dedicated IP (CGNAT is really awful and I feel bad for people who have to suffer through that :( )
• I had never reached it before ! But it seems to be quite recent, right ? That limit thing from docker hub started at the beginning of the year or something like that IIRC

Basically, TLDR: I think PR #110 should fix this !

[sanzoghenzo]

I got another look at this, and indeed I was wrong on the image not being re-pulled.

The default behavior is to always "Query the registry to resolve image digest and supported platforms", so if an image is updated frequently with the same tag, this lead to frequent redeployment.

Using the switch --resolve-image changed would solve this particular aspect.
We could add a global configuration, that can be overridden per-stack, to enable/disable this behavior.

Just a note: #110 will solve your problem, but if you manually remove a service/stack, it won't be restored until you change and commit something in the compose file, so you may experience drifts from the desired configuration declared in your git repos.

[peterdutton]

Hi,

I also have this issue, where I'm running some containers on :latest and when their images get updated it fails with a no such image error in Docker.

How easy would it be to enable the use of the the --resolve-image changed switch in swarmcd?

Thanks

[sanzoghenzo]

How easy would it be to enable the use of the the --resolve-image changed switch in swarmcd?

I submitted the PR #144 to fix this, waiting for approval...

[peterdutton]

I submitted the PR #144 to fix this, waiting for approval...

Looks great - thanks @sanzoghenzo!