EBS initiator always requests /dev/sdz on Nitro instances — only one EBS attach per node possible

[mateo-moon]

Summary

EbsInitiatorProvider can only ever attach one EBS volume per EC2 instance on modern (Nitro) instance types. The free-device-letter scan only recognizes sd*/xvd* kernel names, but on Nitro instances all EBS volumes appear as nvme*n1, so no letter is ever marked as used and every attach requests /dev/sdz. The first attach per instance succeeds; every further attach (while the first is still attached) fails forever with:

AmazonEC2Exception: Invalid value '/dev/sdz' for unixDevice.
Attachment point /dev/sdz is already in use (Status Code: 400; Error Code: InvalidParameterValue)

retried every device-manager cycle, never recovering.

Affected versions

All versions since the EBS feature landed (1.20.0) through current master — the relevant code is unchanged.

Mechanism

satellite/src/main/java/com/linbit/linstor/layer/storage/ebs/EbsInitiatorProvider.java:

1. findUnusedDevice() prunes the candidate set from local lsblk kernel names, stripping only the sd/xvd prefixes:

// cut of the prefix "sd" or "xvd" so we only have the last letter(s) left
if (id.startsWith("sd")) { id = id.substring(2); }
else if (id.startsWith("xvd")) { id = id.substring(3); }
availableLetters.remove(id);

On Nitro instances, attached EBS volumes are exposed by the NVMe driver as /dev/nvme<N>n1 (AWS docs on device naming), so nothing is ever removed from availableLetters.

2. The candidate letters are iterated descending from z:

for (char chr = 'z'; chr >= 'b'; chr--) // sda is usually reserved for root

so the scan deterministically returns z every time.

3. The attach then always requests the same attachment point:

client.attachVolume(new AttachVolumeRequest(ebsVlmId, ec2InstanceId, "/dev/sd" + deviceLettersForAttach));

EC2 tracks the requested device name (/dev/sdz) as the attachment point even though the guest sees an nvme device, so a second concurrent attach on the same instance collides server-side.

Reproduction

Any EC2 Nitro instance (e.g. m5/m6i/m6g/c5…) as an ebs_initiator node: create two LINSTOR resources backed by EBS and place both initiators on the same node. The second attachVolume fails with the error above and the device-manager retries indefinitely. (Observed live on m6g.xlarge/arm64, LINSTOR 1.30.4, in a Kubernetes/piraeus-operator v2 environment — but the bug is independent of Kubernetes.)

Since practically all current-generation EC2 instance types are Nitro-based, this limits the native EBS integration to one volume per node in most real deployments.

Suggested fix

In findUnusedDevice() (or connect()), additionally exclude device names already present in the instance's block-device mappings as reported by EC2 itself — DescribeInstances → instance.blockDeviceMappings[].deviceName — which is authoritative regardless of guest-side naming. Belt-and-braces: on InvalidParameterValue for the attachment point, retry with the next free letter instead of failing the cycle.

I'm happy to open a PR along those lines if this direction is acceptable.

Related

Found together with three other native-EBS issues while evaluating the integration: #505 (DUPLICATE_UNAME), #506 (no resource dispatch to EBS targets), #507 (snapshot rate-limit storm).