Skip to content

Conflicts on update don't count as retry #98

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 25, 2025

Conversation

maruina
Copy link
Member

@maruina maruina commented Mar 17, 2025

What type of PR is this?

/kind bug

What this PR does / why we need it:

When updating the resource quota, e.checkAttributes calls e.checkQuotas up to 3 times. This is to prevent transient failures when updating the resource quota object via e.quotaAccessor.UpdateQuotaStatus. In case of an error, we call e.checkQuotas again and we decrement the remaining retries.

On busy clusters, we might exhaust the 3 retries before making a successful update, bubbling up the error to the client.

This PR is adding a check to avoid decrementing the retries when the error is because of a conflict. While this might increase the processing times because of the additional retries, there are two benefits:

  • doesn't bubble up the error to the user
  • we're still respecting the 10s timeout

If upstream doesn't like this approach, we might consider switching to a workqueue.TypedRateLimitingInterface so we can use a rate limiter before re-adding the object on the queue. This will required more code changes the requeue would happen at a higher level, so we need to be able to inspect the waiter to understand why it got denied.

Which issue(s) this PR fixes:

Fixes kubernetes#67761

Special notes for your reviewer:

Does this PR introduce a user-facing change?


Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@maruina maruina force-pushed the maruina/1.31.6-dd.2-rq branch from d5bd7ac to d821ad8 Compare March 19, 2025 13:50
@maruina maruina changed the title [WIP] Add retries when updating Resource Quota to prevent conflicts [WIP] Conflicts on update don't count as retry Mar 19, 2025
@maruina maruina force-pushed the maruina/1.31.6-dd.2-rq branch 2 times, most recently from 20bba71 to 0894cfe Compare March 20, 2025 09:47
@maruina
Copy link
Member Author

maruina commented Mar 20, 2025

@maruina maruina force-pushed the maruina/1.31.6-dd.2-rq branch from 0894cfe to d8b3b3b Compare March 20, 2025 15:04
@maruina maruina changed the base branch from release-1.31-dd-v1.31.6-dd to release-1.31-dd-v1.31.7-dd March 25, 2025 13:49
@maruina maruina force-pushed the maruina/1.31.6-dd.2-rq branch from d8b3b3b to 43653fd Compare March 25, 2025 13:50
@maruina maruina changed the title [WIP] Conflicts on update don't count as retry Conflicts on update don't count as retry Mar 25, 2025
@maruina maruina force-pushed the maruina/1.31.6-dd.2-rq branch from 43653fd to d819024 Compare March 25, 2025 14:56
@maruina maruina merged commit 8cc6dd7 into release-1.31-dd-v1.31.7-dd Mar 25, 2025
@maruina maruina deleted the maruina/1.31.6-dd.2-rq branch March 26, 2025 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants