diff --git a/README.md b/README.md index 72e5d3b..e9f8ff5 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,7 @@ A collection of helpful charts for Piraeus and other projects. * [snapshot validation webhook](./charts/snapshot-validation-webhook) offers stricter validation of snapshot resources. * [linstor-scheduler](./charts/linstor-scheduler) offers smart scheduling for Pods using LINSTOR volumes. * [piraeus-ha-controller](./charts/piraeus-ha-controller) enabled faster fail-over of workloads when using Piraeus volumes. +* [linstor-affinity-controller](./charts/linstor-affinity-controller) ensures PV affinity matches the state of LINSTOR volumes. ### Contributing diff --git a/charts/linstor-affinity-controller/.helmignore b/charts/linstor-affinity-controller/.helmignore new file mode 100644 index 0000000..0e8a0eb --- /dev/null +++ b/charts/linstor-affinity-controller/.helmignore @@ -0,0 +1,23 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*.orig +*~ +# Various IDEs +.project +.idea/ +*.tmproj +.vscode/ diff --git a/charts/linstor-affinity-controller/Chart.yaml b/charts/linstor-affinity-controller/Chart.yaml new file mode 100644 index 0000000..d8ec00a --- /dev/null +++ b/charts/linstor-affinity-controller/Chart.yaml @@ -0,0 +1,18 @@ +apiVersion: v2 +name: linstor-affinity-controller +description: | + Deploys the LINSTOR Affinity Controller. It periodically checks the state of Piraeus/LINSTOR volumes compared to + PersistentVolumes (PV), and updates the PV Affinity if changes are detected. +type: application +icon: https://raw.githubusercontent.com/piraeusdatastore/piraeus/master/artwork/sandbox-artwork/icon/color.svg +maintainers: + - name: The Piraeus Maintainers + url: https://github.com/piraeusdatastore/ +keywords: + - storage +home: https://github.com/piraeusdatastore/helm-charts +sources: + - https://github.com/piraeusdatastore/linstor-affinity-controller + +version: 1.1.2 +appVersion: "v0.2.2" diff --git a/charts/linstor-affinity-controller/README.md b/charts/linstor-affinity-controller/README.md new file mode 100644 index 0000000..2a72e88 --- /dev/null +++ b/charts/linstor-affinity-controller/README.md @@ -0,0 +1,70 @@ +# LINSTOR Affinity Controller + +The LINSTOR Affinity Controller keeps the affinity of your volumes in sync between Kubernetes and LINSTOR. + +Affinity is used by Kubernetes to track on which node a specific resource can be accessed. For example, you can use +affinity to restrict access to a volume to a specific zone. While this is all supported by Piraeus and LINSTOR, and you +could tune your volumes to support almost any cluster topology, there was one important thing missing: updating affinity +after volume migration. + +After the initial PersistentVolume (PV) object in Kubernetes is created, it is not possible to alter the affinity +later[^1]. This becomes a problem if your volumes need to migrate, for example if using ephemeral infrastructure, where +nodes are created and discard on demand. Using a strict affinity setting could mean that your volume is not accessible +from where you want it to: the LINSTOR resource might be there, but Kubernetes will see the volume as only accessible on +some other nodes. So you had to specify a rather relaxed affinity setting for your volumes, at the cost of less optimal +workload placement. + +There is one other solution (or rather workaround): recreating your PersistentVolume whenever the backing LINSTOR +resource changed. This is where the LINSTOR Affinity Controller comes in: it automates these required steps, so that +using strict affinity just works. With strict affinity, the Kubernetes scheduler can place workloads on the same nodes +as the volumes they are using, benefiting from local data access for increased read performance. + +It also enables strict affinity settings should you use ephemeral infrastructure: even if you rotate out all nodes, +your PV affinity will always match the actual volume placement in LINSTOR. + +## Deployment + +The best way to deploy the LINSTOR Affinity Controller is by helm charm. If deployed to the same namespace +as [our operator](https://github.com/piraeusdatastore/piraeus-operator) this is quite simple: + +``` +helm repo add piraeus-charts https://piraeus.io/helm-charts/ +helm install linstor-affinity-controller piraeus-charts/linstor-affinity-controller +``` + +If deploying to a different namespace, ensure that `linstor.endpoint` and `linstor.clientSecret` are set appropriately. +For more information on the available options, see below. + +### Options + +The following options can be set on the chart: + +| Option | Usage | Default | +|-------------------------------|----------------------------------------------------------------------------------------------|---------------------------------------------------------------| +| `replicaCount` | Number of replicas to deploy. | `1` | +| `options.v` | Set verbosity for controller | `1` | +| `options.leader-election` | Enable leader election to coordinate betwen multiple replicas. | `true` | +| `options.reconcile-rate` | Set the reconcile rate, i.e. how often the cluster state will be checked and updated | `15s` | +| `options.resync-rate` | How often the controller will resync it's internal cache of Kubernetes resources | `15m` | +| `linstor.endpoint` | URL of the LINSTOR Controller API. | `""` (auto-detected when using Piraeus-Operator) | +| `linstor.clientSecret` | TLS secret to use to authenticate with the LINSTOR API | `""` (auto-detected when using Piraeus-Operator) | +| `image.repository` | Repository to pull the linstor-affinity-controller image from. | `quay.io/piraeusdatastore/linstor-affinity-controller` | +| `image.pullPolicy` | Pull policy to use. Possible values: `IfNotPresent`, `Always`, `Never` | `IfNotPresent` | +| `image.tag` | Override the tag to pull. If not given, defaults to charts `AppVersion`. | `""` | +| `resources` | Resources to request and limit on the container. | `{requests: {cpu: 50m, mem: 100Mi}}` | +| `securityContext` | Configure container security context. | `{capabilities: {drop: [ALL]}, readOnlyRootFilesystem: true}` | +| `podSecurityContext` | Security context to set on the pod. | `{runAsNonRoot: true, runAsUser: 1000}` | +| `imagePullSecrets` | Image pull secrets to add to the deployment. | `[]` | +| `podAnnotations` | Annotations to add to every pod in the deployment. | `{}` | +| `nodeSelector` | Node selector to add to a pod. | `{}` | +| `tolerations` | Tolerations to add to a pod. | `[]` | +| `affinity` | Affinity to set on a pod. | `{}` | +| `rbac.create` | Create the necessary roles and bindings for the controller. | `true` | +| `serviceAccount.create` | Create the service account resource | `true` | +| `serviceAccount.name` | Sets the name of the service account. If left empty, will use the release name as default | `""` | +| `podDisruptionBudget.enabled` | Enable creation of a pod disruption budget to protect the availability of the scheduler | `true` | +| `autoscaling.enabled` | Enable creation of a horizontal pod autoscaler to ensure availability in case of high usage` | `"false` | + +*** + +[^1]: That is not 100% true: you can _add_ affinity if it was previously unset, but once set, it can't be modified. diff --git a/charts/linstor-affinity-controller/templates/NOTES.txt b/charts/linstor-affinity-controller/templates/NOTES.txt new file mode 100644 index 0000000..5c0a538 --- /dev/null +++ b/charts/linstor-affinity-controller/templates/NOTES.txt @@ -0,0 +1,3 @@ +LINSTOR Affinity Controller deployed. + +Used LINSTOR URL: {{ include "linstor-affinity-controller.linstorEndpoint" .}} diff --git a/charts/linstor-affinity-controller/templates/_helpers.tpl b/charts/linstor-affinity-controller/templates/_helpers.tpl new file mode 100644 index 0000000..a46c2f6 --- /dev/null +++ b/charts/linstor-affinity-controller/templates/_helpers.tpl @@ -0,0 +1,134 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "linstor-affinity-controller.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "linstor-affinity-controller.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "linstor-affinity-controller.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "linstor-affinity-controller.labels" -}} +helm.sh/chart: {{ include "linstor-affinity-controller.chart" . }} +{{ include "linstor-affinity-controller.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "linstor-affinity-controller.selectorLabels" -}} +app.kubernetes.io/name: {{ include "linstor-affinity-controller.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Create the name of the service account to use +*/}} +{{- define "linstor-affinity-controller.serviceAccountName" -}} +{{- if .Values.serviceAccount.create }} +{{- default (include "linstor-affinity-controller.fullname" .) .Values.serviceAccount.name }} +{{- else }} +{{- default "default" .Values.serviceAccount.name }} +{{- end }} +{{- end }} + +{{/* +Find the linstor client secret containing TLS certificates +*/}} +{{- define "linstor-affinity-controller.linstorClientSecretName" -}} +{{- if .Values.linstor.clientSecret }} +{{- .Values.linstor.clientSecret }} +{{- else if .Capabilities.APIVersions.Has "piraeus.linbit.com/v1/LinstorController" }} +{{- $crs := (lookup "piraeus.linbit.com/v1" "LinstorController" .Release.Namespace "").items }} +{{- if $crs }} +{{- if eq (len $crs) 1 }} +{{- $item := index $crs 0 }} +{{- $item.spec.linstorHttpsClientSecret }} +{{- end }} +{{- end }} +{{- else if .Capabilities.APIVersions.Has "linstor.linbit.com/v1/LinstorController" }} +{{- $crs := (lookup "linstor.linbit.com/v1" "LinstorController" .Release.Namespace "").items }} +{{- if $crs }} +{{- if eq (len $crs) 1 }} +{{- $item := index $crs 0 }} +{{- $item.spec.linstorHttpsClientSecret }} +{{- end }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Find the linstor URL by operator resources +*/}} +{{- define "linstor-affinity-controller.linstorEndpointFromCRD" -}} +{{- if .Capabilities.APIVersions.Has "piraeus.linbit.com/v1/LinstorController" }} +{{- $crs := (lookup "piraeus.linbit.com/v1" "LinstorController" .Release.Namespace "").items }} +{{- if $crs }} +{{- if eq (len $crs) 1 }} +{{- $item := index $crs 0 }} +{{- if include "linstor-affinity-controller.linstorClientSecretName" . }} +{{- printf "https://%s.%s.svc:3371" $item.metadata.name $item.metadata.namespace }} +{{- else }} +{{- printf "http://%s.%s.svc:3370" $item.metadata.name $item.metadata.namespace }} +{{- end }} +{{- end }} +{{- end }} +{{- else if .Capabilities.APIVersions.Has "linstor.linbit.com/v1/LinstorController" }} +{{- $crs := (lookup "linstor.linbit.com/v1" "LinstorController" .Release.Namespace "").items }} +{{- if $crs }} +{{- if eq (len $crs) 1 }} +{{- $item := index $crs 0 }} +{{- if include "linstor-affinity-controller.linstorClientSecretName" . }} +{{- printf "https://%s.%s.svc:3371" $item.metadata.name $item.metadata.namespace }} +{{- else }} +{{- printf "http://%s.%s.svc:3370" $item.metadata.name $item.metadata.namespace }} +{{- end }} +{{- end }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Find the linstor URL either by override or cluster resources +*/}} +{{- define "linstor-affinity-controller.linstorEndpoint" -}} +{{- if .Values.linstor.endpoint }} +{{- .Values.linstor.endpoint }} +{{- else }} +{{- $piraeus := include "linstor-affinity-controller.linstorEndpointFromCRD" . }} +{{- if $piraeus }} +{{- $piraeus }} +{{- else }} +{{- fail "Please specify linstor.endpoint, no default URL could be determined" }} +{{- end }} +{{- end }} +{{- end }} diff --git a/charts/linstor-affinity-controller/templates/deployment.yaml b/charts/linstor-affinity-controller/templates/deployment.yaml new file mode 100644 index 0000000..9592181 --- /dev/null +++ b/charts/linstor-affinity-controller/templates/deployment.yaml @@ -0,0 +1,89 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "linstor-affinity-controller.fullname" . }} + labels: + {{- include "linstor-affinity-controller.labels" . | nindent 4 }} +spec: + {{- if not .Values.autoscaling.enabled }} + replicas: {{ .Values.replicaCount }} + {{- end }} + selector: + matchLabels: + {{- include "linstor-affinity-controller.selectorLabels" . | nindent 6 }} + template: + metadata: + {{- with .Values.podAnnotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + labels: + {{- include "linstor-affinity-controller.selectorLabels" . | nindent 8 }} + spec: + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + serviceAccountName: {{ include "linstor-affinity-controller.serviceAccountName" . }} + securityContext: + {{- toYaml .Values.podSecurityContext | nindent 8 }} + containers: + - name: {{ .Chart.Name }} + args: + - /linstor-affinity-controller + {{- range $opt, $val := .Values.options }} + - --{{ $opt | kebabcase }}={{ $val }} + {{- end }} + securityContext: + {{- toYaml .Values.securityContext | nindent 12 }} + image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} + resources: + {{- toYaml .Values.resources | nindent 12 }} + env: + - name: LEASE_LOCK_NAME + value: {{ include "linstor-affinity-controller.fullname" . }} + - name: LEASE_HOLDER_IDENTITY + valueFrom: + fieldRef: + fieldPath: metadata.name + apiVersion: v1 + - name: LS_CONTROLLERS + value: {{ include "linstor-affinity-controller.linstorEndpoint" . }} + {{- if include "linstor-affinity-controller.linstorClientSecretName" . }} + - name: LS_USER_CERTIFICATE + valueFrom: + secretKeyRef: + name: {{ include "linstor-affinity-controller.linstorClientSecretName" . }} + key: tls.crt + - name: LS_USER_KEY + valueFrom: + secretKeyRef: + name: {{ include "linstor-affinity-controller.linstorClientSecretName" . }} + key: tls.key + - name: LS_ROOT_CA + valueFrom: + secretKeyRef: + name: {{ include "linstor-affinity-controller.linstorClientSecretName" . }} + key: ca.crt + {{- end }} + readinessProbe: + httpGet: + port: 8000 + path: /readyz + livenessProbe: + httpGet: + port: 8000 + path: /healthz + {{- with .Values.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.affinity }} + affinity: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tolerations }} + tolerations: + {{- toYaml . | nindent 8 }} + {{- end }} diff --git a/charts/linstor-affinity-controller/templates/hpa.yaml b/charts/linstor-affinity-controller/templates/hpa.yaml new file mode 100644 index 0000000..5a70cb9 --- /dev/null +++ b/charts/linstor-affinity-controller/templates/hpa.yaml @@ -0,0 +1,32 @@ +{{- if .Values.autoscaling.enabled }} +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: {{ include "linstor-affinity-controller.fullname" . }} + labels: + {{- include "linstor-affinity-controller.labels" . | nindent 4 }} +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: {{ include "linstor-affinity-controller.fullname" . }} + minReplicas: {{ .Values.autoscaling.minReplicas }} + maxReplicas: {{ .Values.autoscaling.maxReplicas }} + metrics: + {{- if .Values.autoscaling.targetCPUUtilizationPercentage }} + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }} + {{- end }} + {{- if .Values.autoscaling.targetMemoryUtilizationPercentage }} + - type: Resource + resource: + name: memory + target: + type: Utilization + averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }} + {{- end }} +{{- end }} diff --git a/charts/linstor-affinity-controller/templates/poddisruptionbudget.yaml b/charts/linstor-affinity-controller/templates/poddisruptionbudget.yaml new file mode 100644 index 0000000..19283c4 --- /dev/null +++ b/charts/linstor-affinity-controller/templates/poddisruptionbudget.yaml @@ -0,0 +1,18 @@ +{{- if .Values.podDisruptionBudget.enabled -}} +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: {{ include "linstor-affinity-controller.fullname" . }} + labels: + {{- include "linstor-affinity-controller.labels" . | nindent 4 }} +spec: + selector: + matchLabels: + {{- include "linstor-affinity-controller.selectorLabels" . | nindent 6 }} + {{- if .Values.podDisruptionBudget.minAvailable }} + minAvailable: {{ .Values.podDisruptionBudget.minAvailable }} + {{- end }} + {{- if .Values.podDisruptionBudget.maxUnavailable }} + maxUnavailable: {{ .Values.podDisruptionBudget.maxUnavailable }} + {{- end }} +{{- end -}} diff --git a/charts/linstor-affinity-controller/templates/rbac.yaml b/charts/linstor-affinity-controller/templates/rbac.yaml new file mode 100644 index 0000000..8b0b8e9 --- /dev/null +++ b/charts/linstor-affinity-controller/templates/rbac.yaml @@ -0,0 +1,99 @@ +{{- if .Values.rbac.create }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "linstor-affinity-controller.serviceAccountName" . }} + labels: + {{- include "linstor-affinity-controller.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +rules: + - apiGroups: + - "" + resources: + - persistentvolumes + verbs: + - get + - list + - watch + - create + - update + - patch + - delete + - apiGroups: + - "storage.k8s.io" + resources: + - storageclasses + verbs: + - get + - apiGroups: + - events.k8s.io + resources: + - events + verbs: + - create + - patch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "linstor-affinity-controller.serviceAccountName" . }} + labels: + {{- include "linstor-affinity-controller.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "linstor-affinity-controller.serviceAccountName" . }} +subjects: + - kind: ServiceAccount + name: {{ include "linstor-affinity-controller.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- if get .Values.options "leader-election" }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ include "linstor-affinity-controller.serviceAccountName" . }} + labels: + {{- include "linstor-affinity-controller.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +rules: + - apiGroups: + - coordination.k8s.io + resources: + - leases + verbs: + - create + - get + - update +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ include "linstor-affinity-controller.serviceAccountName" . }} + labels: + {{- include "linstor-affinity-controller.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ include "linstor-affinity-controller.serviceAccountName" . }} +subjects: + - kind: ServiceAccount + name: {{ include "linstor-affinity-controller.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- end }} +{{- end }} diff --git a/charts/linstor-affinity-controller/templates/serviceaccount.yaml b/charts/linstor-affinity-controller/templates/serviceaccount.yaml new file mode 100644 index 0000000..29c8f69 --- /dev/null +++ b/charts/linstor-affinity-controller/templates/serviceaccount.yaml @@ -0,0 +1,12 @@ +{{- if .Values.serviceAccount.create -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "linstor-affinity-controller.serviceAccountName" . }} + labels: + {{- include "linstor-affinity-controller.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} diff --git a/charts/linstor-affinity-controller/values.yaml b/charts/linstor-affinity-controller/values.yaml new file mode 100644 index 0000000..972b764 --- /dev/null +++ b/charts/linstor-affinity-controller/values.yaml @@ -0,0 +1,68 @@ +replicaCount: 1 + +linstor: + endpoint: "" + clientSecret: "" + +image: + repository: quay.io/piraeusdatastore/linstor-affinity-controller + pullPolicy: IfNotPresent + # Overrides the image tag whose default is the chart appVersion. + tag: "" + +imagePullSecrets: [ ] +nameOverride: "" +fullnameOverride: "" + +options: + v: 1 + leader-election: true + #reconcile-rate: 15s + #resync-rate: 15m + +serviceAccount: + # Specifies whether a service account should be created + create: true + # Annotations to add to the service account + annotations: { } + # The name of the service account to use. + # If not set and create is true, a name is generated using the fullname template + name: "" + +rbac: + # Specifies whether RBAC resources should be created + create: true + +podAnnotations: { } + +podSecurityContext: + runAsNonRoot: true + runAsUser: 1000 + +securityContext: + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + +resources: + requests: + cpu: 50m + memory: 100Mi + +nodeSelector: { } + +tolerations: [] +affinity: { } + +podDisruptionBudget: + enabled: true + minAvailable: 1 + # maxUnavailable: 1 + +autoscaling: + enabled: false + minReplicas: 1 + maxReplicas: 3 + targetCPUUtilizationPercentage: 80 + # targetMemoryUtilizationPercentage: 80