Cloudstack Management unable to start properly

[j3cky]

HI, i have problem with my cloudstack server.suddenly unaccessible.
here some log i can find and it stuck at 2024-10-03 08:21:47,844 INFO [c.c.u.d.Merovingian2] (main:null) (logid:) Cleaning up locks for 345050719314

what does it mean?

2024-10-03 08:21:46,881 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Could not find module [bootstrap] context as they have no parents.
2024-10-03 08:21:46,881 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Trying to obtain module [system] context.
2024-10-03 08:21:46,881 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Trying to load module [system] context.
2024-10-03 08:21:46,882 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.19.1.1.jar!/META-INF/cloudstack/system/spring-core-system-context.xml]
2024-10-03 08:21:46,882 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.19.1.1.jar!/META-INF/cloudstack/system/spring-framework-config-system-context.xml]
2024-10-03 08:21:46,882 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.19.1.1.jar!/META-INF/cloudstack/system/spring-framework-db-system-context.xml]
2024-10-03 08:21:46,882 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.19.1.1.jar!/META-INF/cloudstack/system/spring-server-system-context.xml]
2024-10-03 08:21:46,882 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.19.1.1.jar!/META-INF/cloudstack/system/spring-engine-schema-system-checkers-context.xml]
2024-10-03 08:21:46,882 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.19.1.1.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml]
2024-10-03 08:21:46,883 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.19.1.1.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml]
2024-10-03 08:21:46,883 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.19.1.1.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml]
2024-10-03 08:21:46,883 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.19.1.1.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml]
2024-10-03 08:21:47,496 DEBUG [c.c.u.d.DbProperties] (main:null) (logid:) DB properties were already loaded
2024-10-03 08:21:47,497 INFO [c.c.u.d.T.Transaction] (main:null) (logid:) Is Data Base High Availiability enabled? Ans : false
2024-10-03 08:21:47,497 INFO [c.c.u.d.T.Transaction] (main:null) (logid:) Using the following URI to connect to cloud database [jdbc:mysql://10.120.25.11:3306/cloud?autoReconnect=true&prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC&scrollTolerantForwardOnly=true].
2024-10-03 08:21:47,502 DEBUG [c.c.u.d.DriverLoader] (main:null) (logid:) Successfully loaded DB driver com.mysql.cj.jdbc.Driver
2024-10-03 08:21:47,519 INFO [c.c.u.d.T.Transaction] (main:null) (logid:) Using the following URI to connect to usage database [jdbc:mysql://10.120.25.11:3306/cloud_usage?autoReconnect=true&serverTimezone=UTC&scrollTolerantForwardOnly=true].
2024-10-03 08:21:47,519 DEBUG [c.c.u.d.DriverLoader] (main:null) (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
2024-10-03 08:21:47,526 DEBUG [c.c.u.d.DriverLoader] (main:null) (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
2024-10-03 08:21:47,839 DEBUG [c.c.u.d.ConnectionConcierge] (main:null) (logid:) Registering a database connection for LockController1
2024-10-03 08:21:47,844 INFO [c.c.u.d.Merovingian2] (main:null) (logid:) Cleaning up locks for 345050719314

[j3cky]

Resolved after restart DB

[j3cky]

It happen again. i think it happen after upgrade to 4.19.1.1.
it naver happen before.

[DaanHoogland]

@j3cky , did restart of the DB work the second time as well?

You can try to debug this by

• installing a dummy site with the older version and
• then upgrade that one
• compare the DB schemes to see if anything is missing
  if you don't find anything try installing a fresh 4.19.1.1 and compare the DB scheme from that.