Kvm host go into alert and connecting state if the ceph storage pool is down

[kiranchavala]

problem

Kvm host go into alert and connecting state if the ceph storage pool is down

versions

ACS 4.22.1

The steps to reproduce the bug

1. Have a cloudstack env with nfs storage and a zone wide ceph storage pool

2. Disable the ceph storage pool from cloudstack

3. Shutdown down the ceph storage servers

4. Reconnect the kvm host

The kvm host go into alert and connecting state

Logs MGMT

2026-07-29 07:35:03,055 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-13:[ctx-923461b9]) (logid:9763922f) Notifying other nodes of to disconnect
2026-07-29 07:35:03,055 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-13:[ctx-923461b9]) (logid:9763922f) Failed to handle host connection: com.cloud.exception.ConnectionException: Unable to connect host Host {"id":4,"name":"Cloudstack-Kvm-Host2","type":"Routing","uuid":"d11d3454-5d60-4a2b-a53c-9f34c6cd4cd6"} to storage pool StoragePool {"id":10,"name":"ceph-lab-ec","poolType":"RBD","uuid":"15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f"} due to com.cloud.utils.exception.CloudRuntimeException: Unable to establish connection from storage head to storage pool StoragePool {"id":10,"name":"ceph-lab-ec","poolType":"RBD","uuid":"15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f"} due to Failed to create storage pool: Failed to create storage pool: 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
	at com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:179)
	at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:770)
	at com.cloud.agent.manager.AgentManagerImpl.connectHostAgent(AgentManagerImpl.java:1426)
	at com.cloud.agent.manager.AgentManagerImpl.sendReadyAndGetAttache(AgentManagerImpl.java:1388)
	at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1442)
	at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1538)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to establish connection from storage head to storage pool StoragePool {"id":10,"name":"ceph-lab-ec","poolType":"RBD","uuid":"15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f"} due to Failed to create storage pool: Failed to create storage pool: 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
	at org.apache.cloudstack.storage.datastore.provider.DefaultHostListener.hostConnect(DefaultHostListener.java:155)
	at org.apache.cloudstack.engine.subsystem.api.storage.HypervisorHostListener.hostConnect(HypervisorHostListener.java:29)
	at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1920)
	at com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:176)
	... 13 more

2026-07-29 07:35:03,055 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-13:[ctx-923461b9]) (logid:9763922f) Failed to send ready command: java.nio.channels.ClosedChannelException
	at com.cloud.utils.nio.Link.send(Link.java:319)
	at com.cloud.utils.nio.Link.send(Link.java:294)
	at com.cloud.utils.nio.Link.send(Link.java:290)
	at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1457)
	at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1538)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Kvm host agent logs

2026-07-29 07:28:38,112 WARN  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:84ffb064) Storage pool 803e29b5-b789-30cd-990c-7bb2fa499b21 was not found running in libvirt. Need to create it.
2026-07-29 07:28:38,112 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:84ffb064) Didn't find an existing storage pool 803e29b5-b789-30cd-990c-7bb2fa499b21 by UUID, checking for pools with duplicate paths
2026-07-29 07:29:08,596 ERROR [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:84ffb064) Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to connect to the RADOS monitor on: 192.168.55.245,: No such file or directory
2026-07-29 07:29:08,596 ERROR [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:84ffb064) Failed to create the RBD storage pool, cleaning up the libvirt secret
2026-07-29 07:29:08,653 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:84ffb064) Trying to fetch storage pool 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f from libvirt
2026-07-29 07:29:08,654 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:84ffb064) Asking libvirt to refresh storage pool 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
2026-07-29 07:29:38,657 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:84ffb064) Attempting to create storage pool 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f (RBD) in libvirt
2026-07-29 07:29:38,657 WARN  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:84ffb064) Storage pool 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f was not found running in libvirt. Need to create it.
2026-07-29 07:29:38,657 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:84ffb064) Didn't find an existing storage pool 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f by UUID, checking for pools with duplicate paths


2026-07-29 07:29:44,092 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Detected agent disconnect event, running through 0 disconnect hooks

2026-07-29 07:30:50,799 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Found existing defined storage pool b4b91040-eea6-4c03-ab8a-8bc21ab1d154, using it.
2026-07-29 07:30:50,799 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Trying to fetch storage pool b4b91040-eea6-4c03-ab8a-8bc21ab1d154 from libvirt
2026-07-29 07:30:51,811 INFO  [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Process agent startup answer, agent [id: 0, uuid: , name: ] connected to the server
2026-07-29 07:30:51,823 INFO  [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Startup Response Received: agent [id: 0, uuid: , name: ]
2026-07-29 07:30:52,107 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:7ee7ba43) Attempting to create storage pool e78e94d1-b759-3c4c-ab13-70e231006e0c (NetworkFilesystem) in libvirt
2026-07-29 07:30:52,108 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:7ee7ba43) Found existing defined storage pool e78e94d1-b759-3c4c-ab13-70e231006e0c, using it.
2026-07-29 07:30:52,110 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:7ee7ba43) Trying to fetch storage pool e78e94d1-b759-3c4c-ab13-70e231006e0c from libvirt
2026-07-29 07:30:52,165 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:7ee7ba43) Attempting to create storage pool e7409462-25e4-3184-9bee-33aa50363ccd (NetworkFilesystem) in libvirt
2026-07-29 07:30:52,165 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:7ee7ba43) Found existing defined storage pool e7409462-25e4-3184-9bee-33aa50363ccd, using it.
2026-07-29 07:30:52,167 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:7ee7ba43) Trying to fetch storage pool e7409462-25e4-3184-9bee-33aa50363ccd from libvirt
2026-07-29 07:30:52,261 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:7ee7ba43) Attempting to create storage pool 803e29b5-b789-30cd-990c-7bb2fa499b21 (RBD) in libvirt
2026-07-29 07:30:52,261 WARN  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:7ee7ba43) Storage pool 803e29b5-b789-30cd-990c-7bb2fa499b21 was not found running in libvirt. Need to create it.
2026-07-29 07:30:52,261 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:7ee7ba43) Didn't find an existing storage pool 803e29b5-b789-30cd-990c-7bb2fa499b21 by UUID, checking for pools with duplicate paths
2026-07-29 07:30:55,440 INFO  [cloud.agent.Agent] (Agent-Handler-1:[]) (logid:) Connected to the host: 192.168.55.217

Issue is present even if the cloudstack-agent is restarted

What to do about it?

The kvm host should be in up state even if the ceph storage pool is down as there is nfs storage available

[DaanHoogland]

@kiranchavala a storage pool not being available is an issue for any host, isn’t it? This would not depend on the type of host or the type of storage.

I think the question is, "how should a host behave when not all of it’s storage pools are available?”.
please correct me if I missed the issue.