feat: add VerticalPodAutoscaler to csi controller pod

deploy/example/vpa/README.md

@@ -0,0 +1,142 @@
+# Vertical pod autoscaler to Azure disk CSI controller pods
+## Prerequisites
+### Step 1: Install Vertical Pod Autoscaler (VPA)
+- **Recommend**: You can use the script located at `azuredisk-csi-driver/deploy/example/vpa/install-vpa.sh` to install the VPA. If you use this method, you can skip **Step 2**.
+- You can also refer to the github repo [vertical-pod-autoscaler](https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/README.md) for installation instructions.
+
+
+### Step 2: Adjust Admission Controller Webhooks in AKS
+In AKS, you need to add label `admissions.enforcer/disabled: true` to admission controller webhooks to impact kube-system AKS namespaces, refer to [Can admission controller webhooks impact kube-system and internal AKS namespaces?](https://learn.microsoft.com/en-us/azure/aks/faq#can-admission-controller-webhooks-impact-kube-system-and-internal-aks-namespaces-). This can be done by using the `--webhook-labels` flag in `vpa-admission-controller`, refer to [Running the admission-controller](https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/docs/components.md#:~:text=You%20can%20specify%20a%20comma%20separated%20list%20to%20set%20webhook%20labels%20with%20%2D%2Dwebhook%2Dlabels%2C%20example%20format%3A%20key1%3Avalue1%2Ckey2%3Avalue2.)
+
+If you do not use the recommended script to install VPA, you should manually add the `--webhook-labels=admissions.enforcer/disabled:true` flag in the vpa-admission-controller deployment.
+
+> edit vpa-admission-controller deployment
+```
+k edit deploy vpa-admission-controller -n kube-system
+```
+> add `- --webhook-labels=admissions.enforcer/disabled:true` in containers args
+```
+  template:
+    metadata:
+      creationTimestamp: null
+      labels:
+        app: vpa-admission-controller
+    spec:
+      containers:
+      - args:
+        - --v=4
+        - --stderrthreshold=info
+        - --reload-cert
+        - --webhook-labels=admissions.enforcer/disabled:true # add webhook-labels flags
+```
+
+> check vpa-admission-controller pod running 
+```
+k get po -n kube-system | grep vpa-admission-controller
+vpa-admission-controller-7fcb5c6b86-s69p8              1/1     Running   0               13m
+```
+
+## Create a VPA corresponding to CSI controller deployment
+> create a VPA for CSI controller
+```console
+kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/vpa/vertical-pod-autoscaler.yaml
+```
+> check the VPA config and current recommended resource requests, have 2 evictedPod events for example
+```
+kubectl get vpa -n kube-system
+NAME                       MODE   CPU   MEM        PROVIDED   AGE
+csi-azuredisk-controller   Auto   15m   43690666   True       8s
+
+kubectl describe vpa csi-azuredisk-controller -n kube-system
+Name:         csi-azuredisk-controller
+Namespace:    kube-system
+Labels:       <none>
+Annotations:  <none>
+API Version:  autoscaling.k8s.io/v1
+Kind:         VerticalPodAutoscaler
+Metadata:
+  Creation Timestamp:  2025-03-25T06:45:04Z
+  Generation:          1
+  Resource Version:    62488
+  UID:                 1675169f-6fd9-4c7c-b33e-fffe4f6fc57c
+Spec:
+  Resource Policy:
+    Container Policies:
+      Container Name:  *
+      Controlled Resources:
+        memory
+      Max Allowed:
+        Memory:  10Gi
+  Target Ref:
+    API Version:  apps/v1
+    Kind:         Deployment
+    Name:         csi-azuredisk-controller
+  Update Policy:
+    Update Mode:  Auto
+Status:
+  Conditions:
+    Last Transition Time:  2025-03-25T06:45:15Z
+    Status:                True
+    Type:                  RecommendationProvided
+  Recommendation:
+    Container Recommendations:
+      Container Name:  azuredisk
+      Lower Bound:
+        Memory:  43690666
+      Target:
+        Memory:  43690666
+      Uncapped Target:
+        Memory:  43690666
+      Upper Bound:
+        Memory:        10Gi
+      Container Name:  csi-attacher
+      Lower Bound:
+        Memory:  43690666
+      Target:
+        Memory:  43690666
+      Uncapped Target:
+        Memory:  43690666
+      Upper Bound:
+        Memory:        10Gi
+      Container Name:  csi-provisioner
+      Lower Bound:
+        Memory:  43690666
+      Target:
+        Memory:  43690666
+      Uncapped Target:
+        Memory:  43690666
+      Upper Bound:
+        Memory:        10Gi
+      Container Name:  csi-resizer
+      Lower Bound:
+        Memory:  43690666
+      Target:
+        Memory:  43690666
+      Uncapped Target:
+        Memory:  43690666
+      Upper Bound:
+        Memory:        10Gi
+      Container Name:  csi-snapshotter
+      Lower Bound:
+        Memory:  43690666
+      Target:
+        Memory:  43690666
+      Uncapped Target:
+        Memory:  43690666
+      Upper Bound:
+        Memory:        10Gi
+      Container Name:  liveness-probe
+      Lower Bound:
+        Memory:  43690666
+      Target:
+        Memory:  43690666
+      Uncapped Target:
+        Memory:  43690666
+      Upper Bound:
+        Memory:  10Gi
+Events:
+  Type    Reason      Age   From         Message
+  ----    ------      ----  ----         -------
+  Normal  EvictedPod  101s  vpa-updater  VPA Updater evicted Pod csi-azuredisk-controller-6658fb5fdc-d5mtr to apply resource recommendation.
+  Normal  EvictedPod  41s   vpa-updater  VPA Updater evicted Pod csi-azuredisk-controller-6658fb5fdc-hpdfk to apply resource recommendation.
+```

deploy/example/vpa/install-components/admission-controller-deployment.yaml

@@ -0,0 +1,61 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: vpa-admission-controller
+  namespace: kube-system
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: vpa-admission-controller
+  template:
+    metadata:
+      labels:
+        app: vpa-admission-controller
+    spec:
+      serviceAccountName: vpa-admission-controller
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65534  # nobody
+      containers:
+        - name: admission-controller
+          image: registry.k8s.io/autoscaling/vpa-admission-controller:1.3.0
+          imagePullPolicy: IfNotPresent
+          env:
+            - name: NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          args: ["--v=4", "--stderrthreshold=info", "--reload-cert", "--webhook-labels=admissions.enforcer/disabled:true"]
+          volumeMounts:
+            - name: tls-certs
+              mountPath: "/etc/tls-certs"
+              readOnly: true
+          resources:
+            limits:
+              cpu: 200m
+              memory: 500Mi
+            requests:
+              cpu: 50m
+              memory: 200Mi
+          ports:
+            - containerPort: 8000
+            - name: prometheus
+              containerPort: 8944
+      volumes:
+        - name: tls-certs
+          secret:
+            secretName: vpa-tls-certs
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: vpa-webhook
+  namespace: kube-system
+spec:
+  ports:
+    - port: 443
+      targetPort: 8000
+  selector:
+    app: vpa-admission-controller

deploy/example/vpa/install-components/delete-webhook.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+# Copyright 2018 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Unregisters the admission controller webhook.
+set -e
+
+echo "Unregistering VPA admission controller webhook"
+
+kubectl delete -n kube-system mutatingwebhookconfiguration.v1.admissionregistration.k8s.io vpa-webhook-config --ignore-not-found

deploy/example/vpa/install-components/gencerts.sh

@@ -0,0 +1,72 @@
+#!/bin/bash
+
+# Copyright 2018 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Generates the a CA cert, a server key, and a server cert signed by the CA.
+# reference:
+# https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/webhook/gencerts.sh
+set -o errexit
+set -o nounset
+set -o pipefail
+
+CN_BASE="vpa_webhook"
+TMP_DIR="/tmp/vpa-certs"
+
+echo "Generating certs for the VPA Admission Controller in ${TMP_DIR}."
+mkdir -p ${TMP_DIR}
+cat > ${TMP_DIR}/server.conf << EOF
+[req]
+req_extensions = v3_req
+distinguished_name = req_distinguished_name
+[req_distinguished_name]
+[ v3_req ]
+basicConstraints = CA:FALSE
+keyUsage = nonRepudiation, digitalSignature, keyEncipherment
+extendedKeyUsage = clientAuth, serverAuth
+subjectAltName = DNS:vpa-webhook.kube-system.svc
+EOF
+
+# Create a certificate authority
+openssl genrsa -out ${TMP_DIR}/caKey.pem 2048
+set +o errexit
+openssl req -x509 -new -nodes -key ${TMP_DIR}/caKey.pem -days 100000 -out ${TMP_DIR}/caCert.pem -subj "/CN=${CN_BASE}_ca" -addext "subjectAltName = DNS:${CN_BASE}_ca"
+if [[ $? -ne 0 ]]; then
+  echo "ERROR: Failed to create CA certificate for self-signing. If the error is \"unknown option -addext\", update your openssl version or deploy VPA from the vpa-release-0.8 branch."
+  exit 1
+fi
+set -o errexit
+
+# Create a server certificate
+openssl genrsa -out ${TMP_DIR}/serverKey.pem 2048
+# Note the CN is the DNS name of the service of the webhook.
+openssl req -new -key ${TMP_DIR}/serverKey.pem -out ${TMP_DIR}/server.csr -subj "/CN=vpa-webhook.kube-system.svc" -config ${TMP_DIR}/server.conf
+openssl x509 -req -in ${TMP_DIR}/server.csr -CA ${TMP_DIR}/caCert.pem -CAkey ${TMP_DIR}/caKey.pem -CAcreateserial -out ${TMP_DIR}/serverCert.pem -days 100000 -extensions SAN -extensions v3_req -extfile ${TMP_DIR}/server.conf
+
+echo "Uploading certs to the cluster."
+kubectl create secret --namespace=kube-system generic vpa-tls-certs --from-file=${TMP_DIR}/caKey.pem --from-file=${TMP_DIR}/caCert.pem --from-file=${TMP_DIR}/serverKey.pem --from-file=${TMP_DIR}/serverCert.pem
+
+if [ "${1:-unset}" = "e2e" ]; then
+  openssl genrsa -out ${TMP_DIR}/e2eCaKey.pem 2048
+  openssl req -x509 -new -nodes -key ${TMP_DIR}/e2eCaKey.pem -days 100000 -out ${TMP_DIR}/e2eCaCert.pem -subj "/CN=${CN_BASE}_e2e_ca" -addext "subjectAltName = DNS:${CN_BASE}_e2e_ca"
+  openssl genrsa -out ${TMP_DIR}/e2eKey.pem 2048
+  openssl req -new -key ${TMP_DIR}/e2eKey.pem -out ${TMP_DIR}/e2e.csr -subj "/CN=vpa-webhook.kube-system.svc" -config ${TMP_DIR}/server.conf
+  openssl x509 -req -in ${TMP_DIR}/e2e.csr -CA ${TMP_DIR}/e2eCaCert.pem -CAkey ${TMP_DIR}/e2eCaKey.pem -CAcreateserial -out ${TMP_DIR}/e2eCert.pem -days 100000 -extensions SAN -extensions v3_req -extfile ${TMP_DIR}/server.conf
+  echo "Uploading rotation e2e test certs to the cluster."
+  kubectl create secret --namespace=kube-system generic vpa-e2e-certs --from-file=${TMP_DIR}/e2eCaKey.pem --from-file=${TMP_DIR}/e2eCaCert.pem --from-file=${TMP_DIR}/e2eKey.pem --from-file=${TMP_DIR}/e2eCert.pem 
+fi
+
+# Clean up after we're done.
+echo "Deleting ${TMP_DIR}."
+rm -rf ${TMP_DIR}

deploy/example/vpa/install-components/recommender-deployment.yaml

@@ -0,0 +1,34 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: vpa-recommender
+  namespace: kube-system
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: vpa-recommender
+  template:
+    metadata:
+      labels:
+        app: vpa-recommender
+    spec:
+      serviceAccountName: vpa-recommender
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65534  # nobody
+      containers:
+        - name: recommender
+          image: registry.k8s.io/autoscaling/vpa-recommender:1.3.0
+          imagePullPolicy: IfNotPresent
+          resources:
+            limits:
+              cpu: 200m
+              memory: 1000Mi
+            requests:
+              cpu: 50m
+              memory: 500Mi
+          ports:
+            - name: prometheus
+              containerPort: 8942

deploy/example/vpa/install-components/rmcerts.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+# Copyright 2018 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Generates the a CA cert, a server key, and a server cert signed by the CA.
+# reference:
+# https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/webhook/gencerts.sh
+set -e
+
+echo "Deleting VPA Admission Controller certs."
+kubectl delete secret --namespace=kube-system vpa-tls-certs --ignore-not-found
+kubectl delete secret --namespace=kube-system vpa-e2e-certs --ignore-not-found