Prevent accidental resource deletion by checking deletionTimestamp

[leisma]

I came across this discussion which seems related: https://github.com/orgs/strimzi/discussions/9784. However, I think we went through an unexpected scenario and I would like to discuss it:

We reset our lab cluster, and afterwards the KafkaUser resources were temporarily not applied via GitOps. During that period, the Strimzi operator deleted the corresponding users from the Kafka cluster. From our perspective, there was no explicit deletion event for these resources, so the operator should not have deleted them. This makes the behavior potentially risky in cases where there might be transient issues such as etcd inconsistencies, API server glitches, or temporary reconciliation gaps.

From other Kubernetes operators I have worked with, a common pattern is to check the deletionTimestamp on resources to ensure that an actual deletion intent exists before performing destructive actions. In our case, such a check would likely have prevented the deletion and made the operator more resilient against temporary “missing resource” states.

It seems to me that applying a deletionTimestamp check (or a similar explicit deletion-intent check) could be beneficial for all resources managed by the operator.

What do you think about this approach?

[scholzj]

The operator pattern does not work based on deletion events. It just reconciles the desired and actual state:

• The desired state are the Kubernetes resources
• The actual state are the resources in Kafka

So when the resources are not there, it means in the desired state the resource should not exist. And the operator would update the actual state accordingly. Depending on the circumstances, it might find out about the deletion from the deletion event. But even if it misses the deletion event (e.g. when the User Operator was not running or when due to some error the deletion event was not received), it will sooner or later see that the resources are missing and delete them from Kafka. So this works as designed.

I'm not sure what exactly it means that resources were temporarily not applied via GitOps ... does it mean the resources were deleted I guess? In any case, I have not seen that Kubernetes would hide the resources due to some glitch, but it is of course possible that anything might happen. However, keep in mind this this is also a security risk when seen from the other hand. If we do not delete the Kafka resources ASAP when they are removed from the Kubernetes cluster, they might be actually used by bad actors etc.

[ppatierno]

What do you mean by "we reset our lab cluster"? Did you delete the Kafka custom resource corresponding a Kafka cluster? Did you uninstall the operator?

[leisma]

Hey @scholzj Thanks for the answer! @ppatierno Here a more detailed explanation what happens during the cluster reset and what that means in our setup:
Our setup:

• Lab cluster is periodically reset and reinstalled, meaning we uninstall Kubernetes from our VMs and completely reinstall it
• Strimzi User Operator is deployed via ArgoCD with auto-sync enabled
• KafkaUser resources are managed via ArgoCD with auto-sync disabled (intentional, to avoid accidental changes)

What happens after a cluster reset:

• The operator is deployed and starts reconciling
• The KafkaUser resources have not been re-applied yet (ArgoCD auto-sync is off for them)
• The operator sees no KafkaUser resources → treats this as desired state → deletes all users from Kafka
• When the KafkaUser resources are eventually re-synced, the users are recreated, but there was an unnecessary destructive action in between

The KafkaUser resources were never deleted, they simply hadn't been re-applied yet at the time the operator first ran and Kubernetes has never set a DeletionTimestamp on them. Checking DeletionTimestamp before triggering a delete would distinguish between:

• Intentional delete: kubectl delete kafkauser ... → Kubernetes sets DeletionTimestamp → operator deletes from Kafka
• Resource not yet present: no resource in the cache, no DeletionTimestamp → operator skips deletion and waits

This would not change the security model for the normal case. A resource that is explicitly deleted by a user or GitOps tooling will always have a DeletionTimestamp set by Kubernetes before it disappears from the cache. Bad actors removing access would still trigger deletion.

[scholzj]

The KafkaUser resources were never deleted, they simply hadn't been re-applied

Sorry, I still don't get this. If they were never deleted, then they:

• Either had to be there all the time, in which case there was no reason for the User Operator to delete anything.
• Or never were there in the first place. In which case it is absolutely valid and expected reaction to delete anything from Kafka not enabled by corresponding CR.

For a migration from another cluster, you have to make sure you setup the KafkaUser resources first before deploying the operator.

This would not change the security model for the normal case. A resource that is explicitly deleted by a user or GitOps tooling will always have a DeletionTimestamp set by Kubernetes before it disappears from the cache. Bad actors removing access would still trigger deletion.

Again, this is not possible. First it assumes 100% operator availability to always catch all deletions. While finalizers might help, they do not provide 100% guarantees. The User Operator also has to treat the custom resources as a single source of truth. That is important because it has to own the security details of the Kafka configuration. So the assumption that whatever exists in Kafka has to exist in the CRs is a key assumption and breaking it would make the user operator useless. Without that, anyone could add anything to the Kafka cluster - including bad actors - and the User Operator would be mostly useless.

[leisma]

Hey @scholzj, thanks for the detailed explanation.

You are right that in our scenario the resources were never present on the reinstalled cluster and therefore the operator correctly deletes the corresponding Kafka users. We can solve this on our end by ensuring the KafkaUser resources are always applied before the operator starts.

We think this is a question of which failure mode is more acceptable:

In a properly secured Kafka setup, direct broker access to create users should be restricted at the network and security level. If that assumption holds, a user that exists in Kafka without a corresponding KafkaUser CR is most likely a remnant of a disaster scenario rather than a bad actor. In such cases, e.g. when a user exists in Kafka that should be managed by a CR, a drift warning or error log would give operators a chance to react before a destructive action is taken, without breaking the single source of truth model.

From our point of view, silently deleting users in a disaster scenario carries more operational risk than briefly tolerating an undeclared user.

We understand this is a deliberate design decision and we respect that. We just wanted to share this perspective.

Thanks for taking the time to discuss this!