Operator Canary Upgrade support

[Manicben]

We are using Strimzi for the majority of our Kafka clusters atm and aside from some resourcing issues (we're running like 20 single broker Kafka clusters in 1 k8s cluster), it's been great. We'd like to raise a problem that we are about to face and see whether there would be any interest by the community for such a feature.

Problem Statement

Upgrading the Cluster Operator for a k8s cluster hosting many Kafka clusters can be a time consuming process. All Kafka clusters need to be upgraded to a Kafka version supported by the current CO version and the target CO version. This process can take a long time to complete. This gets compounded when the target Kafka version is a higher version (e.g. 2.8 to 3.3 involves multiple CO and Kafka upgrades). This also becomes harder if these clusters are being used (e.g. for demo purposes) and thus the users would push back on such upgrades.

Proposed Solution

We propose doing something similar to Istio's Canary Upgrades, where the CO can be deployed with a revision label, and discovers namespaces to watch via a matching revision label on the namespace. This will allow users to deploy multiple versions of the CO to the same cluster/namespace, and change which namespaces they watch via namespace labels. We believe this will also make gradual rollouts of new CO and new Kafka versions easier.

Alternatives Considered

We could run a job when creating/upgrading a Kafka cluster, which would modify the watched namespaces list on the CO deployment. Whilst this would technically work, it will cause the CO to roll out, making Kafka cluster upgrades longer, potentially blocking new cluster creation as well.

[scholzj]

I don't think we would want to support any namespace selectors. That would require watching per-namespace which doesn't scale and is inefficient. Why don't you use custom resource labels instead? They are already available.

[Manicben]

Thanks for the advice! I'm looking into this approach and I'm wondering if there's any gotchas around the CRDs. I can see that between 0.29.0 and 0.33.2 that there are some slight differences in the CRDs. Will CO 0.29.0 complain if the 0.33.2 CRDs are installed for example? Or should the lowest version's CRDs be used?

EDIT: Ignore me, as long as the changes are backwards compatible, the old CO version should work with the new CRD.

I understand that if the APIVersions were to change, then we wouldn't be able to use COs that don't support them. In that scenario we would just upgrade all clusters to compatible CO versions before upgrading the CRDs.

[scholzj]

Yes, the changes here should be backward compatible within the same API version and should work. Essentially, there might be new fields and old fields might be deprecated. But no fields are removed within the API version.

[Manicben]

Hey, I made a bad assumption when deploying 2 operator versions to the same k8s cluster. To isolate the two in case of any differences, I used kustomize to add a suffix to the name of every resource, including ClusterRoles and ClusterRoleBindings.

I have learned the hard way that some ClusterRoleBindings are created dynamically per Kafka cluster by the operator and these have a hardcoded reference to the strimzi-kafka-broker ClusterRole, which doesn't exist in my case, it exists as strimzi-kafka-broker-v0-29 for example.

One solution would be to allow us to specify what that roleref should be, but I'm also considering deploying the RBAC resources separately to the operator deployments. I would be using the RBAC resources supplied with the latest operator version deployed to the cluster. Question is, do you foresee any issues with this? Is it possible that changes in a future version may remove permissions that could break an older operator version?

---

For context, I am running Strimzi 0.29.0 and 0.33.2 in the same namespace on my cluster. The label feature is working perfectly as intended for choosing the operator to reconcile the Kafka CR and swaps over smoothly when edited. We upgraded from Kafka 3.0.1 to 3.2.0 to 3.3.2 seamlessly.

[scholzj]

I'm afraid that nobody is able to guarantee that things like this will work forever between any different Strimzi versions. In most cases, using the latest Cluster Roles should give you a good basis. But it is possible that in the future, you would for example need to mix manually the rights from the different cluster roles to run some specific operator versions etc. I don't think anyone can really guarantee things like this for future versions.

[Manicben]

Thanks for the quick response. I think splitting them out will also make them easier to review. We can just add a step in the upgrade process to diff the RBAC resources to check for any changes and see if any would break the oldest operator version we deploy.

Thanks once again for all this. Would a feature request for customising or adding a suffix to the roleref on the ClusterRoleBindings that the operator dynamically creates be something worth considering? Completely understand if not, we might be in the minority of users that are deploying multiple operator versions, but I would be happy to contribute.