KafkaConnector reconciliation: Why are some PUT requests executed ~10 minutes after a change?

[Daumel]

Hi,

I am observing a behavior in our Strimzi-managed Kafka Connect cluster that I cannot fully explain and I am hoping someone here might have an idea what could be causing it. The situation typically occurs when multiple KafkaConnector configurations are changed at the same time. For example, when we delete some connectors and add new ones in a single ArgoCD sync. Because Kafka Connect uses incremental cooperative rebalancing, I would expect these changes to be applied relatively quickly. However, that is not always what we observe.

Observed behavior
After triggering an ArgoCD sync:

• Some connectors are deleted almost immediately.
• Some tasks of newly created connectors also start quickly.
• However, a subset of the new connector tasks only starts almost exactly 10 minutes later.

When looking at the Kafka Connect worker logs, I observed the following:

• The Strimzi operator sends DELETE and PUT requests to the Kafka Connect REST API almost immediately, but not for all connectors that were added.
• For the remaining connectors, the PUT requests appear almost exactly 10 minutes after the sync.
• During those ~10 minutes, the worker logs show no REST calls (GET/PUT/DELETE) from the operator.

From the worker perspective it almost looks as if the operator stops interacting with the REST API for ~10 minutes, and then continues afterwards.

However, the Strimzi operator logs themselves still show continuous "Reconciled" messages, so the operator itself does not appear to be stuck.

Hypothesis
One idea I had (though I am not very familiar with the internal implementation of the operator):

• The operator sends DELETE / PUT requests for some connectors.
• This triggers a rebalance in the Kafka Connect workers.
• While the workers are performing the rebalance, they might not be able to respond to further REST API calls. They are too busy doing other operations and therefore not log failed API calls from the operator within the 10 minutes.
• The operator therefore cannot apply the remaining connector updates immediately.

However, this still does not explain why the delay is very consistently around 10 minutes.

Possibly related observations
I noticed a few additional things that might or might not be related:

• We run 3 Kafka Connect worker nodes. During syncs that modify multiple connectors, I see failed Kubernetes health checks in the events of one or two worker pods. This made me wonder if the workers are temporarily too busy (e.g. during rebalancing, that would support my hypothesis).
• I also see some (not many) "graceful stop of task failed" messages from JDBC connectors.

Question
Is there any mechanism in the Strimzi operator or Kafka Connect reconciliation logic that could explain this ~10 minute delay before some connector updates are applied? Or is there something in Kafka Connect rebalancing behavior that might cause the operator to postpone REST API calls? Any insights would be greatly appreciated 🙂

[scholzj]

I think you would need to share the full logs from the operator (ideally on DEBUG log level) and Connect and the custom resources. Without that, it would be just random guessing.

[katheris]

Hi @Daumel, it's worth noting that internally the operator does process the changes to the connectors one-by-one and after creating a connector will wait until the status endpoint returns with something other than 404 before continuing. If you wanted to get better logs I would suggest turning on DEBUG level logs for io.strimzi.operator.cluster.operator.assembly.AbstractConnectOperator, io.strimzi.operator.cluster.operator.assembly.KafkaConnectAssemblyOperator and io.strimzi.operator.cluster.operator.assembly.KafkaConnectApiImpl. Those should give a better picture of what the operator is doing.

[Daumel]

Hi, thanks for the replies. The issue has been resolved, it turned out that someone had set the fullReconciliationIntervalMs really high...