Agent can't start even if only one KVM VM was using network attached primary storage and that storage becomes unavailable.

[Jayd603]

In a new 4.22 install I was able to only have local disk primary storage and network attached secondary storage on install. Which should make it possible to still boot machines if there is no primary network attached storage.

On a 4.21 cluster with both network attached primary storage and local disk primary storage, the Agents refused to start until network attached primary storage was available and healthy. Even if only a single VM is using that storage, preventing all other VMs from starting. Really only the machines on that missing storage should fail to start, and machines on local disk storage should always be able to boot.

Is anyone familiar with the details of this? Did 4.22 fix this? So If I use a mix of primary network attached storage and local disks the agents wont refuse to start all together if network attached primary storage is missing? At some point I will be able to test this myself , I just can't do it yet so I'm seeing if anyone has run into this and already knows the details.

Ty!

[DaanHoogland]

@Jayd603 , if the vm is already deployed (has run before) I would indeed expect it to start if no nas is available. If not at least a copy from secondary to primary should be possible, i.e. secondary should be available to the host.

do you have logging or stack traces available to have a look?

I do not know of any fix that would change the behaviour between 21 and 22. Maybe @slavkap, @harikrishna-patnala , @sureshanaparti or @rp- is aware?

[Jayd603]

@Jayd603 , if the vm is already deployed (has run before) I would indeed expect it to start if no nas is available. If not at least a copy from secondary to primary should be possible, i.e. secondary should be available to the host.

do you have logging or stack traces available to have a look?

I do not know of any fix that would change the behaviour between 21 and 22. Maybe @slavkap, @harikrishna-patnala , @sureshanaparti or @rp- is aware?

I don't have any further logs/details yet, this happened on a cluster that had lost power where most of the VMs were on local disk storage. The management server and all hosts were brought up fine but then agents refused to start until the primary (but barely used) NAS storage was brought up. This led me to realize the potential weakness. I think the management server had everything needed to bring up the instances that were on local disks but that wasn't possible because the agent's refused to start. On my most recent cluster I was able to install without any primary NAS storage at all, just local disks and secondary NFS storage. Now if I want to add some shared primary storage to the cluster, seems like it could prevent the agents from starting on all hosts even if I used it for storage on just one instance.

[slavkap]

@Jayd603, @DaanHoogland,
the behavior is the same with 4.22 (tested with NFS and StorPool). I've seen this with Ceph as a primary storage with older versions.
I think we need to have a discussion on the best way to resolve this issue. Simply fixing the connection with the agent won’t be enough—if the storage problem itself isn’t addressed, the user won’t notice the connectivity issue between the host and the storage (unless they look through the logs). Most likely, there will be a missing entry in the storage_pool_ref table in the database, which will cause problems for example when trying to start a virtual machine that uses a volume on the affected storage.

One possible solution could be to add an extra status to the storage that clearly indicates its connection to the host. For me this might be the most effective way to make the issue visible and prevent confusion. However, implementing this proposal would require significant changes to core functionalities, so it needs to be carefully evaluated.

[Jayd603]

@Jayd603, @DaanHoogland, the behavior is the same with 4.22 (tested with NFS and StorPool). I've seen this with Ceph as a primary storage with older versions. I think we need to have a discussion on the best way to resolve this issue. Simply fixing the connection with the agent won’t be enough—if the storage problem itself isn’t addressed, the user won’t notice the connectivity issue between the host and the storage (unless they look through the logs). Most likely, there will be a missing entry in the storage_pool_ref table in the database, which will cause problems for example when trying to start a virtual machine that uses a volume on the affected storage.

One possible solution could be to add an extra status to the storage that clearly indicates its connection to the host. For me this might be the most effective way to make the issue visible and prevent confusion. However, implementing this proposal would require significant changes to core functionalities, so it needs to be carefully evaluated.

I think first step should be to allow unaffected instances to start, even if logging/UI is broken for the instances on the shared primary NAS storage. I put in a request to allow quicker changing of the primary storage IP/endpoint in the UI - I'm wondering if simply changing it to any functional primary storage in place of the failed storage will at least allow agents to start, even if that primary storage has basically nothing on it. I would like to test that.

[Jayd603]

Feature request to edit storage IP in UI and with API is here: #12176