rack.topologyKey VS pod affinity

[leunamnauj]

Hello Everyone:
I currently have a nodepool with an specific label to identify it, I'm trying to simply get my kafka and zk nodes not running in the same node plus getting them allocated in different "racks" by following the doc example (topology.kubernetes.io/zone as key).
The best I could get is to get them running on different nodes but not in different zones.

Any idea what Im doing wrong?
Thanks in advance

Truncated config

spec:
  kafka:
    rack:
      topologyKey: topology.kubernetes.io/zone
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - MYCLUSTER
                topologyKey: kubernetes.io/hostname
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: MYPOOL
                      operator: In
                      values:
                        - POOLNAME
        tolerations:
          - effect: NoSchedule
            key: MYPOOL
            operator: Equal
            value: POOLNAME

[scholzj]

If you want to have the pods in different zones, the nodes should be using topology.kubernetes.io/zone as the topologyKey as well? kubernetes.io/hostname will only make sure they are each on a different node.

Also ...

• If you are changing it for an existing cluster, be aware that things such as provisioned storage might mess with this
• If you have more than one pod per zone, Topology Spread Constraints might do this better than podAntiAffinity.
• Make sure the MYCLUSTER value (I assume this is a placeholder and not the actual value, so hard to say for me if it is set crrectly) is correct ;-).

[leunamnauj]

Thanks for your quick answer @scholzj
I've double checked the config, I'm always using "kubernetes.io/hostname" as topologykey and is still the same behaviour: Got pods allocated in the same zone, even in the same node.
Just to be sure I've recreated this testing cluster from the scratch and is still the same.

I've flipped all the stones and cannot find the mistake. If you have a working config to model after I'll really appreciate it

[scholzj]

Well, hostname is a hostname. So that will never give you across zone distribution on its own. If you have the pods scheduled even on the same hostname, I guess you have to take it step by step. The first step would be to go to the pods and check if the affinity from the Kafka CR got passed to the Pods. Whether it is there or isn't there should then show you where to look next.

[leunamnauj]

I've noticed that I missed a few details

Using CLUSTER-NAME-kafka for matchExpressions in podAntiAffinity (as shown in documentation) and of course moving topologykey to "kubernetes.io/zone"

Perhaps would be good idea adding this example to documentation because having pods allocated in different nodes but also in different zones is a pretty common case.

spec:
  kafka:
    rack:
      topologyKey: topology.kubernetes.io/zone
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - MYCLUSTER-kafka
                topologyKey: kubernetes.io/zone
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: MYPOOL
                      operator: In
                      values:
                        - POOLNAME
        tolerations:
          - effect: NoSchedule
            key: MYPOOL
            operator: Equal
            value: POOLNAME

[saraengineer]

Hello,

I tried to do the same thing with my amq streams kafka cluster but it doesn't work.
Is it because I don't have enough zones? how can i check?