fix(semaphore): ensure `holderKey`s carry all information needed. Fixes #8684 by isubasinghe · Pull Request #13553 · argoproj/argo-workflows

isubasinghe · 2024-09-03T11:38:37Z

Motivation

The Holder keys in the status field of a workflow do not carry enough information to work as expected.
This change ensures a single function (getHolderKey) is used for both acquiring and releasing keys.
This in turn deprecates(getResourceKey).

Modifications

Store the entire holderKey in the status. Simple but effective change to fix #8684.
Allows controllers to support V2 holder names even if the holder was first populated by a v1 holder name.

Verification

The unit tests were run with HOLDER_KEY_VERSION=v2.
NOTE: E2E tests HAVE NOT BEEN RUN WITH THIS ENV VAR.
Largely due to being unable to run the E2E tests in a reliable way on my machine.

isubasinghe

Generally looks alright, a few minor issues

agilgur5

In #8684 (comment), you said this would be a breaking change? Then this should have a ! in the title and the appropriate milestone.

Or are you making this backward compatible with the env var?

NOTE: E2E tests HAVE NOT BEEN RUN WITH THIS ENV VAR.
Largely due to being unable to run the E2E tests in a reliable way on my machine.

If this is intended to be required in the future, we should probably enable it during testing similar to #12413 (comment)

isubasinghe · 2024-09-04T22:15:44Z

In #8684 (comment), you said this would be a breaking change? Then this should have a ! in the title and the appropriate milestone.

Or are you making this backward compatible with the env var?

Yeah this should mean that you can upgrade workflows to use the new holder key name without any issue, no longer a breaking change.

NOTE: E2E tests HAVE NOT BEEN RUN WITH THIS ENV VAR.
Largely due to being unable to run the E2E tests in a reliable way on my machine.

If this is intended to be required in the future, we should probably enable it during testing similar to #12413 (comment)

Yeah that sounds good to me.

Joibel

At minimum we need duplicates of the tests that are V1 only as V2 tests.

My preference is to drop V1 completely, agreeing with @agilgur5

isubasinghe · 2024-09-16T09:17:55Z

legacy holding keys have been eliminated, now only v2 style keys exist. Meaning existing keys now test v2 directly.
We upgrade v1 keys to v2. There is a test for this upgrade process in place.

isubasinghe

~~@Joibel opinions?~~ wait please, adding more tests

isubasinghe · 2024-09-17T06:23:49Z

This PR is now ready, I believe I have addressed the edge cases. Unfortunately there is some ambiguity (as expected really) if a sync exist at both the template level and workflow level.

We just choose to go with workflow level in this case of ambiguity.

P.S. I hate this bug

Signed-off-by: isubasinghe <isitha@pipekit.io>

isubasinghe · 2024-09-18T01:19:16Z

/retest

isubasinghe · 2024-09-18T07:29:19Z

/retest

isubasinghe · 2024-09-18T07:29:48Z

Irrelevant tests seem to be broken since a rebase.

argoproj#8684 (argoproj#13553) Signed-off-by: isubasinghe <isitha@pipekit.io>

#8684 (#13553) Signed-off-by: isubasinghe <isitha@pipekit.io>

Fixes: argoproj#8684 Backports: argoproj#13553 Signed-off-by: isubasinghe <isitha@pipekit.io> Co-authored-by: Isitha Subasinghe <isitha@pipekit.io>

isubasinghe changed the title ~~fix: ensure Holding keys carry all information needed~~ fix: ensure Holding keys carry all information needed. Fixes #8684 Sep 4, 2024

isubasinghe marked this pull request as ready for review September 4, 2024 05:02

isubasinghe commented Sep 4, 2024

View reviewed changes

Comment thread pkg/apis/workflow/v1alpha1/workflow_types.go Outdated

Comment thread pkg/apis/workflow/v1alpha1/workflow_types.go Outdated

Comment thread workflow/controller/operator_concurrency_test.go Outdated

Comment thread workflow/sync/sync_manager.go Outdated

agilgur5 reviewed Sep 4, 2024

View reviewed changes

isubasinghe mentioned this pull request Sep 4, 2024

Workflow deadlocks on mutex in steps template if controller is restarted #8684

Closed

3 tasks

agilgur5 reviewed Sep 5, 2024

View reviewed changes

Comment thread pkg/apis/workflow/v1alpha1/workflow_types.go Outdated

Comment thread workflow/sync/mutex_test.go Outdated

Comment thread workflow/controller/operator_concurrency_test.go Outdated

Comment thread pkg/apis/workflow/v1alpha1/workflow_types.go Outdated

juliev0 requested a review from sarabala1979 September 6, 2024 16:06

isubasinghe requested a review from agilgur5 September 9, 2024 05:15

agilgur5 reviewed Sep 10, 2024

View reviewed changes

Comment thread docs/environment-variables.md Outdated

isubasinghe mentioned this pull request Sep 10, 2024

Release v3.4 patch releases discussion #11851

Closed

Joibel requested changes Sep 13, 2024

View reviewed changes

Comment thread docs/environment-variables.md Outdated

Comment thread workflow/sync/sync_manager.go Outdated

Comment thread pkg/apis/workflow/v1alpha1/workflow_types.go Outdated

Comment thread pkg/apis/workflow/v1alpha1/workflow_types.go Outdated

isubasinghe requested a review from Joibel September 16, 2024 09:16

isubasinghe commented Sep 16, 2024

View reviewed changes

isubasinghe marked this pull request as draft September 16, 2024 09:48

isubasinghe marked this pull request as ready for review September 17, 2024 06:19

isubasinghe commented Sep 17, 2024

View reviewed changes

Comment thread workflow/sync/sync_manager.go

isubasinghe commented Sep 17, 2024

View reviewed changes

Comment thread workflow/sync/sync_manager.go Outdated

Joibel requested changes Sep 17, 2024

View reviewed changes

isubasinghe added 9 commits September 18, 2024 10:22

fix: ensure Holding keys carry all information needed

74895c0

Signed-off-by: isubasinghe <isitha@pipekit.io>

fix: added tests for edge cases

bd53d0f

Signed-off-by: isubasinghe <isitha@pipekit.io>

fix: lint issue addressed

335342a

Signed-off-by: isubasinghe <isitha@pipekit.io>

fix: addressed comments

193320e

Signed-off-by: isubasinghe <isitha@pipekit.io>

fix: address feedback

7872148

Signed-off-by: isubasinghe <isitha@pipekit.io>

fix: replace os.Setenv with t.Setenv

37a251c

Signed-off-by: isubasinghe <isitha@pipekit.io>

fix: added godoc comment

16d01e3

Signed-off-by: isubasinghe <isitha@pipekit.io>

fix: remove v1 support

2b21394

Signed-off-by: isubasinghe <isitha@pipekit.io>

docs: reset envrionment variable docs

c56d7ac

Signed-off-by: isubasinghe <isitha@pipekit.io>

isubasinghe added 7 commits September 18, 2024 10:22

fix: removed unnecessary v1 holder key check

2fd00eb

Signed-off-by: isubasinghe <isitha@pipekit.io>

chore: general code cleanup for holding key changes

854907e

Signed-off-by: isubasinghe <isitha@pipekit.io>

chore: remove unnecessary import

2771425

Signed-off-by: isubasinghe <isitha@pipekit.io>

fix: add template level migration

3e7a825

Signed-off-by: isubasinghe <isitha@pipekit.io>

fix: check level first

94828e9

Signed-off-by: isubasinghe <isitha@pipekit.io>

fix: address feedback

6d4da2c

Signed-off-by: isubasinghe <isitha@pipekit.io>

fix: removed logging

e445525

Signed-off-by: isubasinghe <isitha@pipekit.io>

isubasinghe force-pushed the fix-deadlock-mutex branch from a48aeca to e445525 Compare September 18, 2024 00:28

isubasinghe requested a review from Joibel September 18, 2024 00:29

Joibel approved these changes Sep 18, 2024

View reviewed changes

Joibel merged commit 440f1cc into argoproj:main Sep 18, 2024

Joibel pushed a commit to pipekit/argo-workflows that referenced this pull request Sep 19, 2024

fix(semaphore): ensure holderKeys carry all information needed. Fixes

97981b0

argoproj#8684 (argoproj#13553) Signed-off-by: isubasinghe <isitha@pipekit.io>

Joibel mentioned this pull request Sep 20, 2024

Release v3.5 patch releases discussion #11997

Closed

Joibel pushed a commit that referenced this pull request Sep 20, 2024

fix(semaphore): ensure holderKeys carry all information needed. Fixes

a84edd9

#8684 (#13553) Signed-off-by: isubasinghe <isitha@pipekit.io>

isubasinghe mentioned this pull request Oct 8, 2024

docs: add mutex improvement proposal #10191

Closed

Joibel deleted the fix-deadlock-mutex branch November 7, 2024 10:41

isubasinghe mentioned this pull request Dec 16, 2024

The mutex is not released after the pod completes. #14002

Open

4 tasks

Uh oh!

Conversation

isubasinghe commented Sep 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verification

Uh oh!

isubasinghe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

agilgur5 left a comment

Choose a reason for hiding this comment

Uh oh!

isubasinghe commented Sep 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Joibel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

isubasinghe commented Sep 16, 2024

Uh oh!

isubasinghe left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isubasinghe commented Sep 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

isubasinghe commented Sep 18, 2024

Uh oh!

isubasinghe commented Sep 18, 2024

Uh oh!

isubasinghe commented Sep 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

isubasinghe commented Sep 3, 2024 •

edited

Loading

isubasinghe commented Sep 4, 2024 •

edited

Loading

isubasinghe left a comment •

edited

Loading

isubasinghe commented Sep 17, 2024 •

edited

Loading