Git-Sync Containers Keep failing and restarting

[ahipp13]

Hi all,

Am writing this discussion to ask some help from the Airflow community. Currently we are having some weird behavior with the git sync container on the scheduler and triggerer pods and want to get some insights from people to see if they think this is an Airflow issue or something else.

To start, we are running Airflow 2.5.1, and are using Kubernetes and the Official Airflow Helm Chart 1.8.0, with an external postgres backend database. We are also using a git sync sidecar to store our DAGs, and to get Airflow to access this repository, we updated the _helpers.yaml file and values file to put in the username and password.

We currently have 3 environments running this Airflow setup (prod, test, dev). All the environments are basically the same except for they each do have their own DAG repository and have different DAGs in them.

What is happening is at the exact same time, everyday, for some reason the git-sync containers for the scheduler and triggerer go crazy and just start restarting out of no where:

And this exact same thing happens in all of our environments. It has these episodes at approximently 5:40 and 7:40 UTC time, and then the rest of the day it is perfectly fine.

Looking at the logs from the git-sync containers this is what it shows:

it shows a 429, which is the error code for too many requests.

At first I thought it was maybe the DAGs, so I ran all of them in each env at once to see if that would cause it. Nothing happened, and there were no restarts.

Then what I did was changed the wait time in our dev env from 5 seconds to 15 seconds and then checked it the next morning. It reduced the restarts, but 2 still did happen at that same time frame, 7:40 UTC:

So my question is, is this something wrong with the Airflow git sync, or is something that we do not know about accessing our repos at this time?
Really, I am just trying to determine if this is an Airflow issue, or if this is something outside of Airflow.

@potiuk or anybody else I would really appreciate some insight on this. If you have any questions or need more information please feel free to ask. Thank you!

[potiuk]

Error 429 is "too many requests" - https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429 - you seem to exceed the limits.

The "timing" of the issue indicate that there is something that uses the same git credentials around that time that overwhelms GIT with requests. It does not seem it comes from Airflow - because you would have this problem all the time, but very likely around that time you are running some massive GitHub sync or smth that exceed the limits.

Unless the problem is that you are starting many tasks around the same time and they are generating massive number of requests when they start quickly around the same time.

https://docs.github.com/en/rest/overview/resources-in-the-rest-api?apiVersion=2022-11-28#rate-limiting

Rate limits for GitHub are applied "per authenticated user" when the user is authenticated, so the solution to that is to have a separate user only used by Airflow and NOT to share the credentials with some other systems of yours (especially if they generate a lot of HTTP requests".

[ahipp13]

To get back on this, you were correct: it is our enterprise network, our proxy is sending too many unauthorized requests which makes the 429s. Interestingly, it is saying that our team as well as others is making unauthorized requests using this URL: “GITHUB_DAG_REPO.git/info/refs?service=git-upload-pack"

Now as you know the git sync runs this command: "git ls-remote -q GITHUB_DAG_REPO.git refs/heads/main" and in order for us to even get it to work we had to add a user and a PAT token. So why git-upload-pack is making unauthorized requests, I have no idea... I have responded to the GitHub support guy that made us aware of this, I will provide an update when he answers

[potiuk]

Very interesting indeed . I might also take a look knowing all the details. Maybe i can find something

[potiuk]

I think this is the same issue as here: kubernetes/git-sync#522 -> though it has been closed due to lack of reproducibility, but it seems rather similar.

[ahipp13]

Interesting read. After talking more with the GitHub vendor, it seems as if what I said before may not be the issue, and after the limit is hit we may have just been lumped in with everybody else. Which is why it looked like we were making unauthed requests even though we are not. At the EOD, the reason it is happening is because every team is lumped together on one IP, and there are some teams that are making tons of unauthed requests. I lowered our git sync wait time to 30s and now we only see 1 or 2 restarts if that, so that is what we are going with for now. I have been engaged with the our GitHub team so I am hoping they will find whoever is making all the requests or create multiple IPs so that this does not happen. But I am going to mark this as the answer since I myself cannot do anything else!

[potiuk]

Thanks, yeah. That might well be.