KafkaConnect fails to start after upgrade from 0.45.0 to 0.46.0 - Missing required configuration "key.converter"

[artempol-dev]

KafkaConnect fails to start after upgrade from 0.45.0 to 0.46.0 - Missing required configuration "key.converter"

Bug Report

Strimzi version: 0.46.0 (upgraded from 0.45.0)
Kafka version: 3.9.0

Description

After upgrading Strimzi from version 0.45.0 to 0.46.0, KafkaConnect fails to start with the following error:

org.apache.kafka.common.config.ConfigException: Missing required configuration "key.converter" which has no default value.

The configuration that was working in 0.45.0 now fails in 0.46.0, suggesting a breaking change in default configurations or validation logic.

Error Log

2025-06-16 13:07:48,444 ERROR Stopping due to error (org.apache.kafka.connect.cli.AbstractConnectCli) [main]
org.apache.kafka.common.config.ConfigException: Missing required configuration "key.converter" which has no default value.
	at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:534)
	at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:524)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)
	at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:439)
	at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:593)
	at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:587)
	at org.apache.kafka.connect.cli.ConnectDistributed.createConfig(ConnectDistributed.java:107)
	at org.apache.kafka.connect.cli.ConnectDistributed.createConfig(ConnectDistributed.java:54)
	at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:123)
	at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:95)
	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:112)

My configuration

As you can see I have key converter property in my kafka connect configmap

    ##########

    # Provided configurations

    ##########

    offset.storage.topic=kafka-connector-cluster-test-offsets

    value.converter=org.apache.kafka.connect.json.JsonConverter

    config.storage.topic=kafka-connector-cluster-test-configs

    key.converter=org.apache.kafka.connect.json.JsonConverter

[scholzj]

You should start by sharing the full configuration (custom resources) and full logs.

[artempol-dev]

@scholzj
Hey! Here it is

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  annotations:
    meta.helm.sh/release-name: cdc
    meta.helm.sh/release-namespace: kafka2
    strimzi.io/use-connector-resources: 'true'
  creationTimestamp: '2025-06-06T15:32:37Z'
  generation: 1
  labels:
    app.kubernetes.io/component: kafka-connector-cluster
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kafka-connector
    app.kubernetes.io/version: 0.1.0
    helm.sh/chart: kafka-connector-0.1.0
  name: kafka-connector-cluster-cdc
  namespace: kafka2
  resourceVersion: '1750078091095023017'
  uid: 928d4ef3-3bfc-4cf0-a1f9-05b8d5b0a2ba
  selfLink: >-
    /apis/kafka.strimzi.io/v1beta2/namespaces/kafka2/kafkaconnects/kafka-connector-cluster-cdc
status:
  conditions:
    - lastTransitionTime: '2025-06-16T12:48:11.058928040Z'
      message: >-
        Exceeded timeout of 300000ms while waiting for Pods resource
        kafka-connector-cluster-cdc-connect-0 in namespace kafka2 to be ready
      reason: TimeoutException
      status: 'True'
      type: NotReady
  labelSelector: >-
    strimzi.io/cluster=kafka-connector-cluster-cdc,strimzi.io/name=kafka-connector-cluster-cdc-connect,strimzi.io/kind=KafkaConnect
  observedGeneration: 1
  replicas: 1
  url: http://kafka-connector-cluster-cdc-connect-api.kafka2.svc:8083
spec:
  bootstrapServers: kafka-cluster-cdc-kafka-bootstrap:9092
  config:
    cleanup.policy: compact
    config.providers: secrets
    config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider
    config.storage.replication.factor: -1
    config.storage.topic: kafka-connector-cluster-test-configs
    group.id: kafka-connector-cluster-test
    offset.storage.replication.factor: -1
    offset.storage.topic: kafka-connector-cluster-test-offsets
    status.storage.replication.factor: -1
    status.storage.topic: kafka-connector-cluster-test-status
  image: >-
    .../infra/kafka-debezium-image:strimzi-0.45.0-kafka-3.9.0-debezium-2.7.1.Final
  jvmOptions:
    '-Xms': 1g
    '-Xmx': 1g
  livenessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  logging:
    loggers:
      connect.root.logger.level: INFO
      log4j.logger.org.apache.kafka.connect.runtime.WorkerSinkTask: INFO
      log4j.logger.org.apache.kafka.connect.runtime.WorkerSourceTask: INFO
    type: inline
  metricsConfig:
    type: jmxPrometheusExporter
    valueFrom:
      configMapKeyRef:
        key: metrics-config.yml
        name: kafka-connector-cluster-cdc-connect-metrics
  readinessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  replicas: 1
  resources:
    limits:
      cpu: 1000m
      ephemeral-storage: 5G
      memory: 2048Mi
    requests:
      cpu: 50m
      ephemeral-storage: 1G
      memory: 2048Mi
  template:
    connectContainer:
      env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://tempo-distributor.observability.svc.cluster.local:4317
        - name: OTEL_SERVICE_NAME
          value: kafka-connector-service
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        privileged: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        runAsUser: 1001
    initContainer:
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        privileged: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        runAsUser: 1001
    pod:
      metadata:
        annotations:
          checksum/config: a58d7f50e8aad938da69c3a8a2dfe83a2cb43812da390de8d3669b78ae5d534d
          cluster-autoscaler.kubernetes.io/safe-to-evict: 'true'
      securityContext:
        fsGroup: 100
        fsGroupChangePolicy: OnRootMismatch
        runAsGroup: 100
        runAsNonRoot: true
        runAsUser: 1001
        seccompProfile:
          type: RuntimeDefault
  tracing:
    type: opentelemetry
  version: 3.9.0

And logs:

kafka-connector-cluster-c1dc-connect-0.log

[scholzj]

Please share the new logs then. From both the Cluster Operator as well as the Connect cluster

[artempol-dev]

Well, I just rebstart operator and it helped to start a cluster connect. But problem stayed in another format. Now connectors just dont work with this upgrade

Operator and cluster logs:
0.46.0-strimzi-cluster-operator-5886f5974c-hbqqq.log

0.46.0-kafka-connector-cluster-cdc-connect-0.log

And connectors status:

  connectorStatus:
    connector:
      state: FAILED
      trace: "org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.debezium.connector.postgresql.PostgresConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='3.9.0', encodedVersion=3.9.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='3.9.0', encodedVersion=3.9.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='3.9.0', encodedVersion=3.9.0, type=source, typeName='source', location='classpath'}\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.connectorClass(Plugins.java:321)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.newConnector(Plugins.java:292)\n\tat org.apache.kafka.connect.runtime.Worker.startConnector(Worker.java:314)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder.startConnector(DistributedHerder.java:2125)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$getConnectorStartingCallable$45(DistributedHerder.java:2131)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\n"
      worker_id: >-
        kafka-connector-cluster-cdc-connect-0.kafka-connector-cluster-cdc-connect.kafka2.svc:8083
    name: kafka-connector-cdc-fee-custody
    tasks:
      - id: 0
        state: FAILED
        trace: "org.apache.kafka.connect.errors.ConnectException: Failed to start task kafka-connector-cdc-fee-custody-0 since it is not a recognizable type (source or sink)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:2069)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$getTaskStartingCallable$40(DistributedHerder.java:2076)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\n"
        worker_id: >-
          kafka-connector-cluster-cdc-connect-0.kafka-connector-cluster-cdc-connect.kafka2.svc:8083

[artempol-dev]

Maybe 2.7.1 debezium version is too old for 0.46.0 strimzi version ? But I didnt see this in prerequisites

[scholzj]

Well, Debezium interfaces only with Kafka. Not with Strimzi. So as long as it worked with Kafka 3.9.0 in Strimzi 0.45.0 it should work with Kafka 3.9.0 in Strimzi 0.46.0. So I do not think that is the issue. From the log it seems like the plugin was not found in the image - are you sure it is there in the right place? Can you see the JARs when you exec into the Pod? How did you create the container image it is using?

[artempol-dev]

https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/2.7.1.Final/debezium-connector-postgres-2.7.1.Final-plugin.tar.gz

We just download this tar and build image with it
And basically nothing has changed , we used this image with 0.45.0 and 0.46.0

So I dont know what the difference

[maxmonterrey]

We are also facing the same issue after we upgraded to Kafka 0.47 from .0.45.

2025-09-15 14:14:44,407 INFO Added aliases 'PrincipalConnectorClientConfigOverridePolicy' and 'Principal' to plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2025-09-15 14:14:44,520 ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) [main]
org.apache.kafka.common.config.ConfigException: Missing required configuration "key.converter" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:493)
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:483)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:113)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:133)
at org.apache.kafka.connect.runtime.WorkerConfig.(WorkerConfig.java:366)
at org.apache.kafka.connect.runtime.distributed.DistributedConfig.(DistributedConfig.java:405)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)

KafkaConnect.yaml snipped

spec:
replicas: 3
version: 3.9.0
image: artifactory.xxxxxxx.com/docker_unix/kafka/kafkaconnect:v1.4
authentication:
type: scram-sha-512
username: negotiation-kafkaconnect-sa
passwordSecret:
secretName: negotiation-kafkaconnect-sa
password: password
bootstrapServers: xxxxxxxxxx-bootstrap:9093
tls:
trustedCertificates:

• secretName: xxxxxxx-wildcard
  certificate: xxxxxxxxxxxxx-com.cer
  config:
  group.id: kafka-connect-group
  key.converter: org.apache.kafka.connect.storage.StringConverter
  value.converter: org.apache.kafka.connect.storage.StringConverter
  offset.storage.topic: kafka-connect-offsets
  config.storage.topic: kafka-connect-configs
  status.storage.topic: kafka-connect-status
  config.storage.replication.factor: 3
  offset.storage.replication.factor: 3
  status.storage.replication.factor: 3
  ssl.endpoint.identification.algorithm: ""
  consumer.ssl.endpoint.identification.algorithm: ""
  producer.ssl.endpoint.identification.algorithm: ""

[scholzj]

Did you make sure you container image is built on the correct base image?

[maxmonterrey]

Hi Jakub, the issue is intermittent. It works in some other environments and not in others. Same Strimzi set up.

When I do a describe on the pod, I see that none of the config are getting parse even though it is in the yaml.

claimstracking-kafkaconnect-connect-0 0/1 CrashLoopBackOff 4 (90s ago) 4m3s
claimstracking-kafkaconnect-connect-1 1/1 Running 0 6d5h
claimstracking-kafkaconnect-connect-2 1/1 Running 0 6d4h

Working
Environment:
KAFKA_CONNECT_CONFIGURATION: offset.storage.topic=kafka-connect-offsets
value.converter=org.apache.kafka.connect.storage.StringConverter
config.storage.topic=kafka-connect-configs
key.converter=org.apache.kafka.connect.storage.StringConverter
group.id=kafka-connect-group
status.storage.topic=kafka-connect-status
config.storage.replication.factor=3
consumer.ssl.endpoint.identification.algorithm=
offset.storage.replication.factor=3
producer.ssl.endpoint.identification.algorithm=
ssl.endpoint.identification.algorithm=
status.storage.replication.factor=3

Not working one
Environment:
KAFKA_CONNECT_METRICS_ENABLED: true
STRIMZI_KAFKA_GC_LOG_ENABLED: false
STRIMZI_DYNAMIC_HEAP_PERCENTAGE: 75

This is only happening in Strimzi upgrade version 0.47 release

[scholzj]

Yes, because this is not how 0.47.0 passes the configuration. That is why you need the correct image. And if you get different results on different clusters, you might use different images on different clusters, have image cache issues, etc.

[maxmonterrey]

Thanks Jakub. Will update the image, but my concern is that the same image is deployed on other K8 clusters and it's running fine and if we would face the same issue with an updated image.

[scholzj]

Different operator version? Different (stale) image? I'm afraid you did not provided enough information for us to answer this. I can share only assumptions.

[maxmonterrey]

Jakub, my apologies
We are going through Strimzi upgrade of our 7 different k8 environments. We have upgraded 4 of them from Strimzi Operator 0.45 to 0.47.
We have many Kafka Clusters but only 3 that use KafkaConnect with MongoDB connector and the same image across all these environments.

K8 running 1.31.8 Version
Strimzi version 0.47

Please let us know what information you need to better assess this support request.

[scholzj]

Well, you cannot use the same image everywhere. The Connect image is always bound to a particular Strimzi version. So with Strimzi 0.47.0, you need to use image based on the same Strimzi version (0.47.0) and the correct Kafka version (which would be the latest - 4.0.0 - if no Kafka version is specified in .spec.version in KafkaConnect ... or the version set in .spec.version). With Strimzi 0.45.0, you need to use a different image built on Strimzi 0.45.0 and on its corresponding Kafka version.

[swapsCAPS]

Ran into the same while upgrading from 0.45.0 to 0.46.0. Have used strimzi.io/pause-reconciliation="true" to prevent crash. Sharing here for whoever might find this, but would also much appreciate discussion on this approach.

Original failing sequence similar to OP

Baseline

State: Operator 0.45.0, custom Connect image FROM 0.45.0-kafka-3.8.1
Result: Runs fine

Upgrade Kafka version

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
# <snip>
spec:
  version: 3.9.0
  image: "my-own-connect-image:0.45.0-kafka-3.9.0"
# <snip>

State: Operator 0.45.0, custom Connect image FROM 0.45.0-kafka-3.9.0 spec.version: 3.9.0
Result: Runs fine

Upgrade Strimzi Operator

helm upgrade strimzi-cluster-operator oci://quay.io/strimzi-helm/strimzi-kafka-operator  --version 0.46.0 -n kafka
kubectl apply -f strimzi-crds-0.46.0.yaml

State: Operator 0.46.0, custom Connect image FROM 0.45.0-kafka-3.9.0 spec.version: 3.9.0
Result: Connect gets automatically restarted when new operator pod starts running. Connect enters CrashLoopBackOff with Missing required configuration "key.converter" error

Updated successfully on Minikube using the following sequence

Disable reconciliation on all KafkaConnects

kubectl annotate -n kafka kafkaconnect connect-cluster strimzi.io/pause-reconciliation="true"

Download new CRDs

curl -LO https://github.com/strimzirstrimzi-kafka-operator/releases/download/0.46.0/strimzi-crds-0.46.0.yaml

Upgrade operator

helm upgrade strimzi-cluster-operator oci://quay.io/strimzi-helm/strimzi-kafka-operator  --version 0.46.0 -n kafka

Apply CRDs

kubectl apply -f strimzi-crds-0.46.0.yaml

Update spec.image in KafkaConnect spec

spec:
  image: "my-own-connect-image:0.46.0-kafka-3.9.0"

Apply the updated Connect cluster and re-enable reconciliation

kubectl apply -f playground/connect/minikube-setup/connect-cluster.yaml
kubectl annotate kafkaconnect connect-cluster -n kafka strimzi.io/pause-reconciliation-