Add expand-and-modify test type to ebs-scale-test

[AndrewSirenko]

What type of PR is this?

/kind documentation

What is this PR about? / Why do we need it?

Add ebs-scale-test tool TEST_TYPE='expand-and-modify' to guard against regressions in how EBS CSI Driver handles concurrent volume modifications and/or expansions.

Creates $REPLICAS block volumes. Patches PVC capacity and VACName at rate of 5 PVCs per second. Ensures PVCs are expanded and modified before deleting them. Exercises ControllerExpandVolume & ControllerModifyVolume. Set MODIFY_ONLY or EXPAND_ONLY to 'true' to test solely volume modification/expansion.

Note: As of Feb 2025, EBS CSI Driver can handle high amounts of concurrent modifications expansions at a rate of 5 patches per second. However due to our current coalescing implementation and the current AWS EC2 ModifyVolume 6 hour cooldown, modifying AND expanding more than 1 volume per second may lead to errors.

How was this change tested?

❯ ./scale-test create; ./scale-test run; ./scale-test clean

See gist for full logs proof

Also tested that MODIFY_ONLY and EXPAND_ONLY env vars work.

Shortened version:

❯ export CLUSTER_TYPE='pre-allocated'
❯ export REPLICAS=10
❯ export TEST_TYPE=expand-and-modify
❯ ./scale-test create; ./scale-test run; ./scale-test clean
Cluster created...
Updating kubeconfig and restarting ebs-csi-controller Deployment
Updated context arn:aws:eks:us-west-2:208536326202:cluster/ebs-scale-pre-allocated in /Users/andsirey/.kube/config
deployment.apps/ebs-csi-controller restarted
Waiting for deployment "ebs-csi-controller" rollout to finish: 0 of 1 updated replicas are available...
deployment "ebs-csi-controller" successfully rolled out
Applying /Volumes/workplace/aws-ebs-csi-driver/hack/ebs-scale-test/helpers/scale-test/expand-and-modify-test/expand-and-modify.yaml. Exported to /tmp/ebs-scale-test/ebs-scale-pre-allocated-expand-and-modify-10-2025-02-06T20:01UTC/expand-and-modify.yaml
storageclass.storage.k8s.io/ebs-scale-test-expand-and-modify created
volumeattributesclass.storage.k8s.io/ebs-scale-test-expand-and-modify created
persistentvolumeclaim/ebs-claim-0 created
....
Wait for all PVCs to be bound
Patching PVCs with [{"op": "replace", "path": "/spec/volumeAttributesClassName", "value": "ebs-scale-test-expand-and-modify"},{"op": "replace", "path": "/spec/resources/requests/storage", "value": "2Gi"}]
persistentvolumeclaim/ebs-claim-0 patched
....
Waiting until volumes modified and/or expanded
0/10 volumes modified
10/10 volumes modified
All volumes modified
6/10 volumes expanded
10/10 volumes expanded
All volumes expanded
Deleting resources
storageclass.storage.k8s.io "ebs-scale-test-expand-and-modify" deleted
volumeattributesclass.storage.k8s.io "ebs-scale-test-expand-and-modify" deleted
persistentvolumeclaim "ebs-claim-0" deleted
...
persistentvolumeclaim "ebs-claim-9" deleted
Waiting for all PVs to be deleted
No resources found
No PVs exist in the cluster, proceeding...
Port-forwarding ebs-csi-controller containers
...
Deleting cluster
...
2025-02-06 15:13:52 [✔]  all cluster resources were deleted

Does this PR introduce a user-facing change?

Add volume expansion & modification scalability test type to `ebs-scale-test` tool

[github-actions]

Code Coverage Diff

This PR does not change the code coverage

[AndrewSirenko]

Decided to rely on Immediately bound PVCs instead of scaling statefulset (as we do in the scale-sts test).

Pros:

• Separation of concerns between scale-sts test and expand-and-modify test
• Order of magnitude shorter test: no waiting for all volumes to attach/detach + creating karpenter instances.
• Two way door. Changing this test to scale sts is trivial.

Cons:

• We do not excercise NodeExpandVolume at scale. (Though to me this sounds like a kubelet, AMI, resize2fs testing responsibility. We already have e2e tests)

Discussed this offline with @torredil, who lightly agreed with me.

[ElijahQuinones]

Will lgtm once other comments are addressed

persistentvolumeclaim/ebs-claim-2 patched
persistentvolumeclaim/ebs-claim-1 patched
persistentvolumeclaim/ebs-claim-0 patched
persistentvolumeclaim/ebs-claim-3 patched
persistentvolumeclaim/ebs-claim-4 patched
persistentvolumeclaim/ebs-claim-6 patched
persistentvolumeclaim/ebs-claim-5 patched
persistentvolumeclaim/ebs-claim-7 patched
persistentvolumeclaim/ebs-claim-8 patched
persistentvolumeclaim/ebs-claim-9 patched
Waiting until volumes modified and/or expanded
0/10 volumes modified
10/10 volumes modified
All volumes modified
8/10 volumes expanded
10/10 volumes expanded
All volumes expanded
Deleting resources

verified all resources where delete properly.

[ElijahQuinones]

/lgtm

[torredil]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: torredil

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [torredil]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment