feat: add support for httpcaching of github requests (alpha)

[jd-hatzenbuhler]

This PR adds a LRU cache to the applicationset controller for github scm provider.
This uses a specific httpcache strategy described in here and recommanded in here

Fixes #11376

Deployed the fix on our production by building a custom image and we get a ~90% cache hit
see

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Title of the PR
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

[bunnyshell]

❗ Preview Environment deployment failed on Bunnyshell

See: Environment Details | Pipeline Logs

Available commands (reply to this comment):

• 🚀 /bns:deploy to redeploy the environment
• ❌ /bns:delete to remove the environment

[crenshaw-dev]

Do we need to be concerned with security implications? For example, might an AppSet be able to retrieve a response that happens to be available in the cache that it wouldn't be able to access if it had to go to GitHub to auth? Do we need the cache to be segmented by AppSet or secret or something like that?

[jd-hatzenbuhler]

@crenshaw-dev

Do we need to be concerned with security implications?

The caching library used here handle the segmentation internally
The library inject caculate the expected Etag value based on the request headers including Authorization.
So the request is made with the actual Authorization header and if the response is not 304 Not Modified, the response cached is not used (401, 403)

IMO the security questions to consider are more:

• on the library dependency, do we want to have an external dependency for that?
• if there is vulnerability in the ArgoCD that allow an attacker to dump the cache content, do we want to protect against that?

Do we need the cache to be segmented by AppSet or secret or something like that?

Could be an option but it will improve the performance more than the security the Etag recalculation will happen a bit less.

[ppapapetrou76]

can we also add metrics?

You can use lru.NewWithEvict to capture evictions and increment metrics.

[jd-hatzenbuhler]

can we also add metrics?

@ppapapetrou76 will add the metrics layer later once other remarks will be solved

[alexymantha]

The library inject caculate the expected Etag value based on the request headers including Authorization.

I'm a bit worried about this actually. The library is pretty clear that it does this by having reverse-engineered the Etag and this is undocumented behavior and is not supported by GitHub. And while it works fine, I'm not sure we should be depending on behaviors that could be changed by GitHub any day without notice.

I'm wondering if we should just use standard HTTP conditional requests despite the drawback of having more cache misses when the credentials change

[jd-hatzenbuhler]

I'm wondering if we should just use standard HTTP conditional requests despite the drawback of having more cache misses when the credentials change

@alexymantha for github app it's not useful at all since we recreate the token on each reconciliation loop
for token since it's more static it could at least be useful

On our side we are using github app and we do understand the risk if Github change the Etag behaviour.
At least we need another flag for github app cache with a big warning about it