Title
v2beta1: operator panics in GetListenersServicePorts when default listeners are removed from EMQX runtime config
Body
## Bug Report
### Summary
The operator panics during reconcile when listener entries such as `listeners.ssl.default`, `listeners.ws.default`, or `listeners.wss.default` are removed from the EMQX runtime config.
The panic happens in `apis/apps/v2beta1/util.go` inside `GetListenersServicePorts()`, which assumes that each `listeners.<type>.<name>` entry is always a `hocon.Object` and performs an unsafe type assertion:
```go
obj := config.(hocon.Object)
When the runtime config returned by /api/v5/configs contains a string/scalar instead of an object for one of these listener entries, the operator crashes with:
panic: interface conversion: hocon.Value is hocon.String, not hocon.Object
Affected Versions
- emqx-operator:
2.2.25
- emqx-operator:
2.2.29
- EMQX:
5.8.4
I reproduced the same behavior on both operator versions above.
What I Expected
If default listeners such as 8883, 8083, and 8084 are removed or disabled, the operator should handle that gracefully and continue reconciling.
At minimum, GetListenersServicePorts() should skip invalid/non-object listener entries instead of panicking.
What Actually Happened
After removing default listeners from EMQX, the operator crashed during reconcile with the following stack trace:
panic: interface conversion: hocon.Value is hocon.String, not hocon.Object [recovered]
panic: interface conversion: hocon.Value is hocon.String, not hocon.Object
goroutine 290 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:116 +0x1da
panic({0x19e1200?, 0xc001bcf170?})
/usr/local/go/src/runtime/panic.go:792 +0x132
github.com/emqx/emqx-operator/apis/apps/v2beta1.GetListenersServicePorts({0xc002782600, 0x2567})
/workspace/apis/apps/v2beta1/util.go:268 +0x143e
github.com/emqx/emqx-operator/controllers/apps/v2beta1.generateListenerService(0xc000486c08, {0xc002782600?, 0x2567?})
/workspace/controllers/apps/v2beta1/add_svc.go:107 +0x19b
github.com/emqx/emqx-operator/controllers/apps/v2beta1.(*addSvc).reconcile(0xc00033c1a0, {0x1efd6a8, 0xc000ba7a10}, {{0x1f01f68?, 0xc000ba7a40?}, 0xc000ba7a10?}, 0xc000486c08, {0x1eff980, 0xc000e9a4c0})
/workspace/controllers/apps/v2beta1/add_svc.go:39 +0x385
github.com/emqx/emqx-operator/controllers/apps/v2beta1.(*EMQXReconciler).Reconcile(0xc00057aa50, {0x1efd6a8, 0xc000ba7a10}, {{{0xc00021c5a0?, 0x5?}, {0xc00021c598?, 0xc000963d10?}}})
/workspace/controllers/apps/v2beta1/emqx_controller.go:137 +0x7a4
How to Reproduce
-
Deploy EMQX with emqx-operator 2.2.29 (also reproducible on 2.2.25).
-
Enable listenersServiceTemplate.
-
Configure custom listeners such as:
-
Remove the default listeners (ssl.default, ws.default, wss.default) from the EMQX runtime configuration, or otherwise make them disappear/change shape in /api/v5/configs.
-
Wait for the operator to reconcile.
-
The operator crashes with the panic shown above.
Relevant Code Path
The operator does not read listener ports directly from the EMQX CR. Instead, it fetches runtime config from:
and then parses listeners from that returned HOCON string:
controllers/apps/v2beta1/add_svc.go
apis/apps/v2beta1/util.go
The problematic logic is here:
for name, config := range configs.(hocon.Object) {
obj := config.(hocon.Object)
cutConfig := hocon.Object{}
if v, ok := obj["enable"]; ok {
cutConfig["enable"] = v
}
if v, ok := obj["enabled"]; ok {
cutConfig["enabled"] = v
}
if v, ok := obj["bind"]; ok {
cutConfig["bind"] = v
}
...
}
This assumes every listener config is always a hocon.Object, which is not true in this case.
Suspected Root Cause
GetListenersServicePorts() performs an unsafe type assertion:
obj := config.(hocon.Object)
If config is a hocon.String (or any non-object value), the operator panics.
This is especially easy to trigger because the operator reads from /api/v5/configs (runtime config), not directly from the Kubernetes YAML / CR spec.
Suggested Fix
I think there are two issues here:
1. Avoid panic in GetListenersServicePorts()
Please add a safe type check before casting:
obj, ok := config.(hocon.Object)
if !ok {
continue
}
This would prevent the operator from crashing on unexpected listener shapes.
2. Reconsider fallback/default listener behavior in generateListenerService()
In controllers/apps/v2beta1/add_svc.go, when GetListenersServicePorts() returns no ports, the operator falls back to hardcoded defaults:
This behavior is surprising when the user intentionally removes or disables default listeners.
It may be better to:
- not inject fallback ports automatically, or
- make this behavior optional/configurable, or
- prefer
listenersServiceTemplate.spec.ports when explicitly provided.
Title
v2beta1: operator panics in GetListenersServicePorts when default listeners are removed from EMQX runtime configBody
When the runtime config returned by
/api/v5/configscontains a string/scalar instead of an object for one of these listener entries, the operator crashes with:Affected Versions
2.2.252.2.295.8.4I reproduced the same behavior on both operator versions above.
What I Expected
If default listeners such as
8883,8083, and8084are removed or disabled, the operator should handle that gracefully and continue reconciling.At minimum,
GetListenersServicePorts()should skip invalid/non-object listener entries instead of panicking.What Actually Happened
After removing default listeners from EMQX, the operator crashed during reconcile with the following stack trace:
How to Reproduce
Deploy EMQX with
emqx-operator2.2.29(also reproducible on2.2.25).Enable
listenersServiceTemplate.Configure custom listeners such as:
188328833883Remove the default listeners (
ssl.default,ws.default,wss.default) from the EMQX runtime configuration, or otherwise make them disappear/change shape in/api/v5/configs.Wait for the operator to reconcile.
The operator crashes with the panic shown above.
Relevant Code Path
The operator does not read listener ports directly from the EMQX CR. Instead, it fetches runtime config from:
and then parses listeners from that returned HOCON string:
controllers/apps/v2beta1/add_svc.goapis/apps/v2beta1/util.goThe problematic logic is here:
This assumes every listener config is always a
hocon.Object, which is not true in this case.Suspected Root Cause
GetListenersServicePorts()performs an unsafe type assertion:If
configis ahocon.String(or any non-object value), the operator panics.This is especially easy to trigger because the operator reads from
/api/v5/configs(runtime config), not directly from the Kubernetes YAML / CR spec.Suggested Fix
I think there are two issues here:
1. Avoid panic in
GetListenersServicePorts()Please add a safe type check before casting:
This would prevent the operator from crashing on unexpected listener shapes.
2. Reconsider fallback/default listener behavior in
generateListenerService()In
controllers/apps/v2beta1/add_svc.go, whenGetListenersServicePorts()returns no ports, the operator falls back to hardcoded defaults:1883888380838084This behavior is surprising when the user intentionally removes or disables default listeners.
It may be better to:
listenersServiceTemplate.spec.portswhen explicitly provided.