RFE: Per-volume encryption via SPDK crypto vbdev at the lvol layer

[AdityaHarindar]

Problem

Mayastor supports pool-level (lvolstore) encryption via SPDK's crypto vbdev. All PVCs on a given DiskPool share one key, which does not serve:

1. Distinct DEK per PVC for compliance controls that require per-data-owner key identifiers — PCI-DSS §3.5, HIPAA §164.312(a)(2)(iv), FedRAMP equivalents. Audit trails today distinguish pools, not tenants-within-pool.
2. Cryptographic erase per PVC. Right-to-erasure, dataset expiration, and offboarding workflows require O(1) provable data destruction by destroying one key. Pool-level encryption offers this only at pool granularity.

Threat model in scope: encryption at rest — disk theft, backup exfiltration, decommissioned-media leakage, and the compliance controls above.

Proposed Solution

Add per-volume encryption by stacking the existing SPDK crypto vbdev one level higher: above the lvol, below the NVMe-oF target. Each PVC carries its own DEK, sourced from a K8s Secret referenced by its StorageClass. External key stores (Vault, cloud KMS, HSM) are reached via the standard External Secrets Operator (ESO) bridge — Mayastor's interface is a K8s Secret reference, and the ecosystem handles the rest.

I intend to implement this across openebs/mayastor and openebs/mayastor-control-plane.

openebs/mayastor — data plane

• Extend ReplicaArgs with an optional encryption key, mirroring PoolArgs::with_encryption.
• Wrap the lvol via create_crypto_vbdev_on_base_bdev and publish the wrapping bdev over NVMe-oF.
• Persist a key-name reference (never key material) as an lvol xattr. On pool import, hold encrypted lvols in a keyed-out state until the control plane replays the key via a new SetReplicaKey RPC.
• Advertise a MayastorFeatures.per_replica_encryption capability bit and gate the code behind ENABLE_REPLICA_ENCRYPTION, mirroring the existing diskpool_encryption pattern.

openebs/mayastor-control-plane — control plane

• Add a VolumeEncryption { cipher, secret_ref } type to VolumeSpec (shape — additive vs. collapsed — is the open question below). Promote CreateReplica.encrypted from a bookkeeping-only field into a plumbed request property.
• Add a SecretReader in the core agent that resolves K8s Secret refs to plaintext DEKs held only in process memory.
• Thread the resolved key through every replica-creating path: primary create, rebuild, move, clone, snapshot restore.
• Extend the scheduler with an additive node.per_replica_encryption capability filter. The current pool-affinity filter and encryption_preference_soft() knob stay for pool-level users.
• Add a nexus child-alignment reconciliation pass so mixed encrypted / plaintext children in the same nexus do not fail alignment during rollout.

openebs/mayastor-extensions — CSI

• Template a per-PVC K8s Secret via StorageClass parameters (csi.storage.k8s.io/provisioner-secret-name and namespace).

The CSI node plugin and consumer-side nvme connect are unaffected. The crypto boundary sits below the target.

Non-goals

• Not tenant isolation from the Mayastor operator. The io-engine holds the plaintext DEK in memory while serving reads. True operator isolation requires consumer-side encryption or confidential computing — a separate feature.
• Not in-place key rotation. SPDK's crypto vbdev has no rekey primitive. v1 rotation is create-new + copy + swap.
• Ciphertext at the lvol layer, which precludes future lvol-layer compression or cross-volume dedup for encrypted volumes.

Rollout

Phased, feature-flagged at every layer. Opt-in for at least one release before default-on. Perf benchmark (AES-NI on/off, replica count sweep) lands with the data-plane PRs. Correctness tests through the existing lvs-eval harness; hardware-dependent code paths gated in CI.

Alternatives

• Pool-level encryption alone (status quo). Does not provide per-PVC key scope or per-PVC cryptographic erase.
• LUKS on the consumer (wrapping CSI or init-container, secrets-store-csi-driver pattern). Stronger — the platform never sees plaintext DEKs. Complementary, not competing; addresses an operator-isolation threat model this proposal explicitly does not.
• NVMe-oF TLS. Orthogonal; solves wire encryption only.
• Confidential-computing io-engine (SEV-SNP, TDX). Deferred; requires SPDK-side work and hardware availability.

Longhorn and Ceph have shipped similar per-volume approaches.

Additional context

Open question — VolumeSpec shape

Two options for expressing per-volume encryption on VolumeSpec:

• Additive: keep the existing encrypted: bool and add encryption: Option<VolumeEncryption> alongside. Backward-compatible; no etcd migration; introduces two fields with overlapping semantics.

• Collapsed: replace both with a single enum:

  EncryptionMode { None | PoolAffinity | PerVolume { cipher, secret_ref } }

  Single source of truth; requires an etcd schema migration and breaks any client that reads the boolean.

Which shape do you prefer to maintain?

[yugchaudhari]

Small clarification worth pinning down on the CSI side. The RFE says "template a per-PVC K8s Secret via StorageClass parameters (csi.storage.k8s.io/provisioner-secret-name and namespace)" which reads two different ways depending on whether the parameter values are static or templated. Explicitly committing to templating avoids a common trap where the SC ships hardcoded values and every PVC on that SC ends up sharing one key, defeating the per-PVC scope.

The external-provisioner supports templating those parameters against the PVC, the two useful forms are ${pvc.name} and ${pvc.annotations['<key>']}. Annotation-driven is the one that reads best as "the PVC chooses its own key":

kind: StorageClass
parameters:
  secure: "true"
  repl: "3"
  csi.storage.k8s.io/provisioner-secret-name: "${pvc.annotations['encryption.mayastor.io/secret-name']}"
  csi.storage.k8s.io/provisioner-secret-namespace: "${pvc.namespace}"

kind: PersistentVolumeClaim
metadata:
  annotations:
    encryption.mayastor.io/secret-name: my-per-pvc-key

The SC declares the mechanism, the PVC supplies the instance. Longhorn ships this exact pattern for its v2 encryption; Ceph-CSI also supports it.

[tiagolobocastro]

When we added encryption at rest we considered this way too, but ended up opting for the simpler pool encryption approach

I wasn't entirely happy with encrypted being a bool at the time tbh, so we'll have to add another field - we have to make it backward compatible I think, we have no plans for a breaking semver as of today.

IIRC we don't read secrets in the agent-core today, the io-engine reads them when importing the pools

Importing also becomes more complicated, since we won't have the keys until the agent-core provides them or the secret name.
May be worth considering keeping the encryption in the nexus bdev rather than the replicas?

May I suggest you raise an OEP for this?

[Abhinandan-Purkait]

I think having the vbdev on top of nexus would simplify things, as @tiagolobocastro mentioned. But then we would have to consider what happens if just the replica is shared and accessed that way.

The additive is a better option as that would be straightforward.

Although the migration is also not a bad idea. We already have some kind of etcd data migration at place on core-agent startup. So we can implement similar. i.e after upgrade the core-agent patches the VolumeSpec to the new format. Also, the clients are mostly in cluster or the kubectl plugin so, they are expected to be of the same version, so I think that should be fine. The newer storageclasses will not care about the bool: true even if its added. IMO this would be cleaner, unless there is some hard failure that I might have missed. But yeah, we can keep it simple by just the additive way for now.

Also, why need a storageclass param for identifying the annotation, can we not just add it to the PVC when creating the PVC and read from it. The annotation key can be made constant. If we tie the secret name with a storageclass the storageclass loses reusability. It would become a pain to create a sc for each volume with a specific key.

Also, I believe the since restores from snapshot of the volume will also have to share the same key, what would be better duplicating the k8s secret or always using the same secret?

[yugchaudhari]

@Abhinandan-Purkait fair point on SC reusability. If the annotation key is a Mayastor-owned constant, the SC stays generic and users only touch the PVC to opt in, that reads cleaner. The trade-off is that the CSI driver (or agent-core) needs to fetch the PVC via K8s API to read that annotation, which is a small dependency to add. But agent-core already reads Secrets from K8s for pool creation per OEP 3843, so extending it to also read the PVC annotation isn't a new capability, it's the same shape. Given we already have the K8s client wired up, I'd lean toward direct-annotation here.

On the snapshot-restore key question: sharing the same Secret is safer than duplicating. Duplication creates a key-versioning problem, if the source Secret's contents ever rotate, the clone's copy silently diverges and the snapshot data becomes unrecoverable via the clone's key. Sharing means one lifecycle owner and one place to look when something breaks. Costs: cross-namespace snapshot restore needs the Secret to be accessible from the restore namespace, which is a real operational constraint but a solvable one.

On @tiagolobocastro's suggestion to keep the crypto at the nexus bdev rather than per-replica, worth exploring but there's a caveat worth flagging. Nexus-level crypto simplifies the import path (nexus is built per publish, key is scoped to a single per-target open) at the cost of losing at-rest protection for any code path that touches a replica outside its nexus. Rebuild source (a replica read by another nexus during recovery), scrubbing / verification tooling, or direct-debug access via a raw replica handle would all see plaintext. If the threat model is "disk theft / decommissioned media," per-lvol keeps the guarantee end-to-end; nexus-level trades a small chunk of that guarantee for the import simplification. Probably worth calling this out explicitly in the OEP so reviewers can weigh it.

[tiagolobocastro]

But agent-core already reads Secrets from K8s for pool creation per OEP 3843, so extending it to also read the PVC annotation isn't a new capability, it's the same shape. Given we already have the K8s client wired up, I'd lean toward direct-annotation here.

I think that was not done that way in the end because there's no TLS on the gRPC at the moment (there's an in-progress work for TLS though)

cost of losing at-rest protection for any code path that touches a replica outside its nexus. Rebuild source (a replica read by another nexus during recovery), scrubbing / verification tooling, or direct-debug access via a raw replica handle would all see plaintext. If the threat model is "disk theft / decommissioned media," per-lvol keeps the guarantee end-to-end; nexus-level trades a small chunk of that guarantee for the import simplification. Probably worth calling this out explicitly in the OEP so reviewers can weigh it.

It doesn't loose the protection guarantee, anything under nexus would be encrypted, so the replica itself would have encrypted data?

[yugchaudhari]

It doesn't loose the protection guarantee, anything under nexus would be encrypted, so the replica itself would have encrypted data?

Yes yes, I mixed up at-rest with "which code paths see plaintext." The disk-layer guarantee holds because the nexus writes ciphertext down. The actual concern is that any code opening a replica outside its nexus (rebuild source, scrub, direct SPDK debug) would need to be crypto-aware to see plaintext, but per-lvol keeps the crypto inline in the bdev stack. That's an implementation-complexity thing, not a security thing.

[yugchaudhari]

But agent-core already reads Secrets from K8s for pool creation per OEP 3843, so extending it to also read the PVC annotation isn't a new capability, it's the same shape. Given we already have the K8s client wired up, I'd lean toward direct-annotation here.

I think that was not done that way in the end because there's no TLS on the gRPC at the moment (there's an in-progress work for TLS though)

I was reading the OEP wording ("control plane agent-core refers... the DEK from the Secret"); didn't check the codebase thoroughly and assumed that's how it landed. Without TLS on gRPC that would leak the DEK in transit, so io-engine reading directly makes sense. For per-volume the same method works, CSI driver (or io-engine) reads the PVC annotation to resolve the Secret name, the key never leaves io-engine.

[tiagolobocastro]

It doesn't loose the protection guarantee, anything under nexus would be encrypted, so the replica itself would have encrypted data?

Yes yes, I mixed up at-rest with "which code paths see plaintext." The disk-layer guarantee holds because the nexus writes ciphertext down. The actual concern is that any code opening a replica outside its nexus (rebuild source, scrub, direct SPDK debug) would need to be crypto-aware to see plaintext, but per-lvol keeps the crypto inline in the bdev stack. That's an implementation-complexity thing, not a security thing.

yeah, we'd have to weigh pros and cons of the approaches

[tiagolobocastro]

But agent-core already reads Secrets from K8s for pool creation per OEP 3843, so extending it to also read the PVC annotation isn't a new capability, it's the same shape. Given we already have the K8s client wired up, I'd lean toward direct-annotation here.

I think that was not done that way in the end because there's no TLS on the gRPC at the moment (there's an in-progress work for TLS though)

I was reading the OEP wording ("control plane agent-core refers... the DEK from the Secret"); didn't check the codebase thoroughly and assumed that's how it landed. Without TLS on gRPC that would leak the DEK in transit, so io-engine reading directly makes sense. For per-volume the same method works, CSI driver (or io-engine) reads the PVC annotation to resolve the Secret name, the key never leaves io-engine.

Well tbh the OEP should have been modified to account for the change in design after we found that hole :)

[abhilashshetty04]

@AdityaHarindar , @yugchaudhari , What if user opt-in for both Pool and volume encryption in the same SC? There may be performance penalty to decrypt/encrypt twice in IO path.

[yugchaudhari]

What if user opt-in for both Pool and volume encryption in the same SC? There may be performance penalty to decrypt/encrypt twice in IO path.

@abhilashshetty04 I can think of two options. Pool-level encryption is an SC parameter (placement policy), per-volume would be a PVC annotation (as per @Abhinandan-Purkait, this keeps the SC reusable). So we can introduce a validation check for "both opted in" at CSI CreateVolume time, which is the moment where CSI has both inputs in hand: SC parameters (from the external-provisioner) and PVC info (name, namespace, annotations). The check either rejects the combo with an actionable error, or gets marked as an explicit non-goal for v1 with the perf/security tradeoff documented.

cc @AdityaHarindar

[abhilashshetty04]

What if user opt-in for both Pool and volume encryption in the same SC? There may be performance penalty to decrypt/encrypt twice in IO path.

@abhilashshetty04 I can think of two options. Pool-level encryption is an SC parameter (placement policy), per-volume would be a PVC annotation (as per @Abhinandan-Purkait, this keeps the SC reusable). So we can introduce a validation check for "both opted in" at CSI CreateVolume time, which is the moment where CSI has both inputs in hand: SC parameters (from the external-provisioner) and PVC info (name, namespace, annotations). The check either rejects the combo with an actionable error, or gets marked as an explicit non-goal for v1 with the perf/security tradeoff documented.

cc @AdityaHarindar

If there is encryption at a replica layer below nexus there maybe no point in having pool level encryption also. plain-text is just between lvol -> lvs layer on the same io-engine I reckon.

Also, I wonder should we introduce VAC (Volume attribute class)? Implementing this with VAC would be a cleaner option. cc @tiagolobocastro, @Abhinandan-Purkait wdyt?

[tiagolobocastro]

I'm not sure I follow, how would VAC tackle the clash between per-pool and per-replica encryption?

[druhinabrol]

@abhilashshetty04 How is this related to #2033 ? I see that you have mentioned it there.