VA behavior when a pod is moved to another node

[Bash-mocart]

I have 2 nodes node A and node B

Pod-test that has a PVC mount is running in node A, so essentially it has a lets say VA-node-A

Correct me if my assumption is wrong

If I move a Pod-test away from node A to node B, this will create a new VA-node-B

So now I have 2 VAs, VA-node-A and VA-node-B, there will be a syncDetach on VA-node-A and syncAttach on VA-node-B

This sync operations call ControllerPublishVolume and ControllerUnpublishVolume, via grpc to the csi plugin

So in this scenario, the 2 controllers will try to reconcile their respective VAs, problem here is, both VAs, have the same storage, and the two controllers, will keep acting on the storage simultaneously.

If users of the external-attacher are not really careful this will make them easily introduce race conditions for the 2 controllers, one scenario is if their api, accepts link and unlink storages, and you need to keep polling status to know if storage has been linked or unlinked

Not an expert in the external attacher code, but do you think it makes sense to delay creating the VA-node-B till the VA-node-A has been unlinked successfully? in other words and essentially just, prevent the two sync operations from acting on the same storage at the same time ?

If we cant, them maybe makes sense to make this as explicit as possible for users, so they know and do the right thing, maybe in docs or somewhere else?

But i think this is just how the external attacher works, preventing this race condition is simply a responsibility of the csi plugin?

[kolluria]

@Bash-mocart I'm not an expert on the attacher mechanics but this is not a race condition per se because the attacher would be initiating two different requests(as you pointed out) with the storage backend. These are the possible scenarios that I can think of -

Scenario 1

1. VA-node-A detach gets invoked, completes successfully
2. VB-node-B attach gets called, completes successfully

Scenario 2

1. VA-node-A detach is invoked, VB-node-B attach is invoked before VA-node-A detach is successful
2. While VA-node-A detach is in progress, VB-node-B attach gets re-queued with exponential backoff
3. Eventually VB-node-B attach will succeed once VA-node-A detach is completed

Scenario 3

1. VB-node-B attach is invoked before VA-node-A detach detach
2. VB-node-B attach gets re-queued similar to scenario 2 while VA-node-A detach goes through
3. Eventually VB-node-B attach will succeed once VA-node-A detach is completed

PS: I'm making as assumption here that the storage provider's backend wouldn't run into any dead locks when parallel attach and detach are invoked on the same volume - as I don't see why a detach call for node A could be impacted by an attach call to node B.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale