Add security context

[MirahImage]

This closes #1910 and #1960

Adds a securityContext to the cluster operator deployment context.
Adds securityContext to RabbitMQ Pods, containers, and init containers.

[MirahImage]

Given the security context appears in 4 different places (once in the operator deployment itself, then in three places in the cluster statefulset), I suspect we may need to improve our documentation about overrides for OpenShift.

[Zerpet]

Docs regarding openshift will need some updating. I'm going to run this locally in CRC, but we likely need to override the RunAsUser bit in favour of the FSGroup

[MirahImage]

Let me know if you've figured out the necessary OpenShift overrides (or just push a commit to the branch).

[Zerpet]

I had to switch to something else on Friday. I'm looking into it now.

[Zerpet]

Indeed it doesn't like the RunAsUser: 999 nor the FSGroup: 0

STS condition

Warning  FailedCreate      8m27s (x16 over 11m)  statefulset-controller  create Pod rabbitmq-server-0 in StatefulSet rabbitmq-server failed error: pods "rabbitmq-server-0" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or servieaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [[]int64{0}: 0 is not an allowed group, provider restricted-v2: .initContainers[0].runAsUser: Invalid value: 999: must be in the ranges: [1000660000, 1000669999], provider restricted-v2: .containers[0].runAsUser: Invalid value: 999: must be in the ranges: [1000660000, 1000669999], provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "hostpath-provisioner": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]

The error describes the problem:

provider restricted-v2: 
  .spec.securityContext.fsGroup: Invalid value: [[]int64{0}: 0 is not an allowed group, 
provider restricted-v2: 
  .initContainers[0].runAsUser: Invalid value: 999: must be in the ranges: [1000660000, 1000669999], 
provider restricted-v2: 
  .containers[0].runAsUser: Invalid value: 999: must be in the ranges: [1000660000, 1000669999]

This can be workaround with the suggestion from our docs that basically removes the Pod security context. With the security contexts in the containers, it's enough to pass the "restricted" validation.

A potential improvement would be to set to nil, or not set at all, the problematic fields. It left unset, Openshift does the right thing and assigns a valid value. We could customise the behaviour for Openshift if an annotation e.g. rabbitmq.com/platform-is-openshift: some-value-doesnt-matter is present in RabbitmqCluster objtect.

I'll leave it to you to decide if you want to implement the annotation behaviour.