Skip to content

fix: mark node failed if pod absent. Fixes #12993#13454

Merged
Joibel merged 7 commits into
argoproj:mainfrom
isubasinghe:fix-hung-workflows
Aug 15, 2024
Merged

fix: mark node failed if pod absent. Fixes #12993#13454
Joibel merged 7 commits into
argoproj:mainfrom
isubasinghe:fix-hung-workflows

Conversation

@isubasinghe

@isubasinghe isubasinghe commented Aug 12, 2024

Copy link
Copy Markdown
Member

Fixes #12993

Motivation

If the pod is absent and the node has not been completed, we should mark the node as failed after a timeout and mark the workflowtaskresult as completed.

Modifications

Verification

Signed-off-by: isubasinghe <isitha@pipekit.io>
Signed-off-by: isubasinghe <isitha@pipekit.io>
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe isubasinghe changed the title fix: mark node failed if pod absent fix: mark node failed if pod absent. Fixes #12993 Aug 12, 2024
@isubasinghe
isubasinghe marked this pull request as ready for review August 12, 2024 10:07
@isubasinghe
isubasinghe requested a review from Joibel August 12, 2024 10:07
Comment thread workflow/controller/taskresult.go Outdated
if foundPod {
woc.log.Debugf("Got pod %s with phase %s for task result %s and node id %s with label %s", pod.Name, pod.Status.Phase, resultName, result.Name, label)
woc.log.Debugf("The node phase was %s for node named %s", node.Phase, node.Name)
} else if !foundPod && !node.Completed() {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could technically hit this case where a pod exists but hasn't made it to the cache yet.

@isubasinghe

Copy link
Copy Markdown
Member Author

Also @Joibel note that I haven't been able to reproduce any of the open source issues.
So this hasn't been tested against #12993

@Joibel Joibel self-assigned this Aug 12, 2024
Comment thread workflow/controller/taskresult.go Outdated
Comment thread workflow/controller/taskresult.go Outdated
Comment thread workflow/controller/taskresult.go Outdated
Comment thread workflow/controller/taskresult.go Outdated
Comment thread workflow/controller/taskresult.go Outdated
Comment thread workflow/controller/taskresult.go Outdated
Comment thread workflow/controller/taskresult.go Outdated
Comment thread workflow/controller/taskresult.go Outdated
Comment thread workflow/controller/taskresult.go Outdated
Signed-off-by: isubasinghe <isitha@pipekit.io>
Signed-off-by: isubasinghe <isitha@pipekit.io>
Signed-off-by: isubasinghe <isitha@pipekit.io>
Signed-off-by: isubasinghe <isitha@pipekit.io>
@Joibel
Joibel merged commit 36b7a72 into argoproj:main Aug 15, 2024
Joibel pushed a commit to pipekit/argo-workflows that referenced this pull request Sep 19, 2024
…3454)

Signed-off-by: isubasinghe <isitha@pipekit.io>
Joibel pushed a commit that referenced this pull request Sep 20, 2024
Signed-off-by: isubasinghe <isitha@pipekit.io>
@Joibel
Joibel deleted the fix-hung-workflows branch October 25, 2024 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/controller Controller issues, panics

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v3.5.3: Workflow processing fails to complete due to incomplete WorkflowTaskResult from interrupted pod

3 participants