Skip to content

Operator panic in GetListenersServicePorts after removing default listeners #1182

@liuz0606

Description

@liuz0606

Title

v2beta1: operator panics in GetListenersServicePorts when default listeners are removed from EMQX runtime config

Body

## Bug Report

### Summary

The operator panics during reconcile when listener entries such as `listeners.ssl.default`, `listeners.ws.default`, or `listeners.wss.default` are removed from the EMQX runtime config.

The panic happens in `apis/apps/v2beta1/util.go` inside `GetListenersServicePorts()`, which assumes that each `listeners.<type>.<name>` entry is always a `hocon.Object` and performs an unsafe type assertion:

```go
obj := config.(hocon.Object)

When the runtime config returned by /api/v5/configs contains a string/scalar instead of an object for one of these listener entries, the operator crashes with:

panic: interface conversion: hocon.Value is hocon.String, not hocon.Object

Affected Versions

  • emqx-operator: 2.2.25
  • emqx-operator: 2.2.29
  • EMQX: 5.8.4

I reproduced the same behavior on both operator versions above.


What I Expected

If default listeners such as 8883, 8083, and 8084 are removed or disabled, the operator should handle that gracefully and continue reconciling.

At minimum, GetListenersServicePorts() should skip invalid/non-object listener entries instead of panicking.


What Actually Happened

After removing default listeners from EMQX, the operator crashed during reconcile with the following stack trace:

panic: interface conversion: hocon.Value is hocon.String, not hocon.Object [recovered]
    panic: interface conversion: hocon.Value is hocon.String, not hocon.Object

goroutine 290 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:116 +0x1da
panic({0x19e1200?, 0xc001bcf170?})
    /usr/local/go/src/runtime/panic.go:792 +0x132
github.com/emqx/emqx-operator/apis/apps/v2beta1.GetListenersServicePorts({0xc002782600, 0x2567})
    /workspace/apis/apps/v2beta1/util.go:268 +0x143e
github.com/emqx/emqx-operator/controllers/apps/v2beta1.generateListenerService(0xc000486c08, {0xc002782600?, 0x2567?})
    /workspace/controllers/apps/v2beta1/add_svc.go:107 +0x19b
github.com/emqx/emqx-operator/controllers/apps/v2beta1.(*addSvc).reconcile(0xc00033c1a0, {0x1efd6a8, 0xc000ba7a10}, {{0x1f01f68?, 0xc000ba7a40?}, 0xc000ba7a10?}, 0xc000486c08, {0x1eff980, 0xc000e9a4c0})
    /workspace/controllers/apps/v2beta1/add_svc.go:39 +0x385
github.com/emqx/emqx-operator/controllers/apps/v2beta1.(*EMQXReconciler).Reconcile(0xc00057aa50, {0x1efd6a8, 0xc000ba7a10}, {{{0xc00021c5a0?, 0x5?}, {0xc00021c598?, 0xc000963d10?}}})
    /workspace/controllers/apps/v2beta1/emqx_controller.go:137 +0x7a4

How to Reproduce

  1. Deploy EMQX with emqx-operator 2.2.29 (also reproducible on 2.2.25).

  2. Enable listenersServiceTemplate.

  3. Configure custom listeners such as:

    • 1883
    • 2883
    • 3883
  4. Remove the default listeners (ssl.default, ws.default, wss.default) from the EMQX runtime configuration, or otherwise make them disappear/change shape in /api/v5/configs.

  5. Wait for the operator to reconcile.

  6. The operator crashes with the panic shown above.


Relevant Code Path

The operator does not read listener ports directly from the EMQX CR. Instead, it fetches runtime config from:

/api/v5/configs

and then parses listeners from that returned HOCON string:

  • controllers/apps/v2beta1/add_svc.go
  • apis/apps/v2beta1/util.go

The problematic logic is here:

for name, config := range configs.(hocon.Object) {
    obj := config.(hocon.Object)
    cutConfig := hocon.Object{}
    if v, ok := obj["enable"]; ok {
        cutConfig["enable"] = v
    }
    if v, ok := obj["enabled"]; ok {
        cutConfig["enabled"] = v
    }
    if v, ok := obj["bind"]; ok {
        cutConfig["bind"] = v
    }
    ...
}

This assumes every listener config is always a hocon.Object, which is not true in this case.


Suspected Root Cause

GetListenersServicePorts() performs an unsafe type assertion:

obj := config.(hocon.Object)

If config is a hocon.String (or any non-object value), the operator panics.

This is especially easy to trigger because the operator reads from /api/v5/configs (runtime config), not directly from the Kubernetes YAML / CR spec.


Suggested Fix

I think there are two issues here:

1. Avoid panic in GetListenersServicePorts()

Please add a safe type check before casting:

obj, ok := config.(hocon.Object)
if !ok {
    continue
}

This would prevent the operator from crashing on unexpected listener shapes.

2. Reconsider fallback/default listener behavior in generateListenerService()

In controllers/apps/v2beta1/add_svc.go, when GetListenersServicePorts() returns no ports, the operator falls back to hardcoded defaults:

  • 1883
  • 8883
  • 8083
  • 8084

This behavior is surprising when the user intentionally removes or disables default listeners.

It may be better to:

  • not inject fallback ports automatically, or
  • make this behavior optional/configurable, or
  • prefer listenersServiceTemplate.spec.ports when explicitly provided.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions