-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PipelineRun is reconciled thousands of times #8495
Comments
I was able to reproduce it using a simple pipeline run. In this case, it's not thousands of times but still quite a lot. Pipeline Run apiVersion: "tekton.dev/v1"
kind: "PipelineRun"
metadata:
generateName: "run-"
namespace: "default"
spec:
pipelineSpec:
tasks:
- name: "task-1"
taskSpec:
steps:
- args:
- "-c"
- |-
echo 'Start of task'
sleep 30s
echo 'Task completed'
command:
- "sh"
image: "alpine:3.7"
name: "simple" Controller Logs
|
In theory, reconciliation happens anytime there is an change on the objects watched. For example, a PipelineRun will be reconciled each time something changes on the TaskRun. Same goes with the TaskRun and pods. And there could be a lot of things happening. From creation of the Pod to the Pod running, there is several updates from kubernetes. Each might generate a TaskRun change (in status) and thus a reconciliation on the PipelineRun itself. And all this is not taking account any other controller that would apply some labels or annotations for example. From the log it is a bit difficult to see if it's expected or not here. |
Thanks for looking into this! I also noticed that the controller's CPU consumption is high (which makes sense when the controller is constantly reconciling resources?). Looking into our monitoring, this seems to be related to when we updated from v0.61.1 to v0.62.0. With v0.61.1 we had an average CPU consumption of around 150 mCores. Afterwards around 3.000 😮 |
... further looking into this, we also wanted to disable any timeout for pipeline runs and added |
Expected Behavior
Reconciliation only happens on change
Actual Behavior
Reconciliation happens thousands of times over and over again
Steps to Reproduce the Problem
I still need to investigate further what might cause the behavior and how to reproduce it reliably. Eventually, the pipeline run completes successfully. However, while it is in progress, I can see in the controller's logs that it is reconciled repeatedly.
I´ve attached an example of logs of the controller.
controller-logs.txt
Additional Info
Has anyone seen similar behavior?
Thanks,
Fabian
The text was updated successfully, but these errors were encountered: