[Bug]: Impossible to display Kafka JMX Mertrics from 0.41.0 examples

[avdev4j]

Bug Description

In a Strimzi 0.41.0 installation, it is not possible to display Kafka and Zookeeper JMX metrics when following this blog post: https://strimzi.io/blog/2019/10/14/improving-prometheus-metrics/.

The Grafana dashboard remains empty, even though Prometheus is scraping the metrics using a PodMonitor.

I tried multiple times to ensure that I was using the same version as in the GitHub examples and my installed version, but it always ended the same way: it was impossible to get these metrics.

Steps to reproduce

Enable the metrics in the resource:

    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml

Add the configmaps:

kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-metrics
  labels:
    app: strimzi
data:
  kafka-metrics-config.yml: |
    # See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
    lowercaseOutputName: true
    rules:
      # Special cases and very specific rules
      - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
        name: kafka_server_$1_$2
        type: GAUGE
        labels:
          clientId: "$3"
          topic: "$4"
          partition: "$5"
      - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
        name: kafka_server_$1_$2
        type: GAUGE
        labels:
          clientId: "$3"
          broker: "$4:$5"
      - pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
        name: kafka_server_$1_connections_tls_info
        type: GAUGE
        labels:
          cipher: "$2"
          protocol: "$3"
          listener: "$4"
          networkProcessor: "$5"
      - pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections
        name: kafka_server_$1_connections_software
        type: GAUGE
        labels:
          clientSoftwareName: "$2"
          clientSoftwareVersion: "$3"
          listener: "$4"
          networkProcessor: "$5"
      - pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):"
        name: kafka_server_$1_$4
        type: GAUGE
        labels:
          listener: "$2"
          networkProcessor: "$3"
      - pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
        name: kafka_server_$1_$4
        type: GAUGE
        labels:
          listener: "$2"
          networkProcessor: "$3"
      # Some percent metrics use MeanRate attribute
      # Ex) kafka.server<type=(KafkaRequestHandlerPool), name=(RequestHandlerAvgIdlePercent)><>MeanRate
      - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate
        name: kafka_$1_$2_$3_percent
        type: GAUGE
      # Generic gauges for percents
      - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
        name: kafka_$1_$2_$3_percent
        type: GAUGE
      - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
        name: kafka_$1_$2_$3_percent
        type: GAUGE
        labels:
          "$4": "$5"
      # Generic per-second counters with 0-2 key/value pairs
      - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_total
        type: COUNTER
        labels:
          "$4": "$5"
          "$6": "$7"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_total
        type: COUNTER
        labels:
          "$4": "$5"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
        name: kafka_$1_$2_$3_total
        type: COUNTER
      # Generic gauges with 0-2 key/value pairs
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
          "$6": "$7"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
        name: kafka_$1_$2_$3
        type: GAUGE
      # Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
      # Note that these are missing the '_sum' metric!
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_count
        type: COUNTER
        labels:
          "$4": "$5"
          "$6": "$7"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
          "$6": "$7"
          quantile: "0.$8"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_count
        type: COUNTER
        labels:
          "$4": "$5"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
          quantile: "0.$6"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
        name: kafka_$1_$2_$3_count
        type: COUNTER
      - pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          quantile: "0.$4"
    # KRaft mode: uncomment the following lines to export KRaft related metrics
    # KRaft overall related metrics
    # distinguish between always increasing COUNTER (total and max) and variable GAUGE (all others) metrics
    #- pattern: "kafka.server<type=raft-metrics><>(.+-total|.+-max):"
    #  name: kafka_server_raftmetrics_$1
    #  type: COUNTER
    #- pattern: "kafka.server<type=raft-metrics><>(.+):"
    #  name: kafka_server_raftmetrics_$1
    #  type: GAUGE
    # KRaft "low level" channels related metrics
    # distinguish between always increasing COUNTER (total and max) and variable GAUGE (all others) metrics
    #- pattern: "kafka.server<type=raft-channel-metrics><>(.+-total|.+-max):"
    #  name: kafka_server_raftchannelmetrics_$1
    #  type: COUNTER
    #- pattern: "kafka.server<type=raft-channel-metrics><>(.+):"
    #  name: kafka_server_raftchannelmetrics_$1
    #  type: GAUGE
    # Broker metrics related to fetching metadata topic records in KRaft mode
    #- pattern: "kafka.server<type=broker-metadata-metrics><>(.+):"
    #  name: kafka_server_brokermetadatametrics_$1
    #  type: GAUGE
  zookeeper-metrics-config.yml: |
    # See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
    lowercaseOutputName: true
    rules:
      # replicated Zookeeper
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+)><>(\\w+)"
        name: "zookeeper_$2"
        type: GAUGE
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)"
        name: "zookeeper_$3"
        type: GAUGE
        labels:
          replicaId: "$2"
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(Packets\\w+)"
        name: "zookeeper_$4"
        type: COUNTER
        labels:
          replicaId: "$2"
          memberType: "$3"
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(\\w+)"
        name: "zookeeper_$4"
        type: GAUGE
        labels:
          replicaId: "$2"
          memberType: "$3"
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+), name3=(\\w+)><>(\\w+)"
        name: "zookeeper_$4_$5"
        type: GAUGE
        labels:
          replicaId: "$2"
          memberType: "$3"

Add a podMonitor in the Prometheus (considering it does exist and works)

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
spec:
  namespaceSelector:
    matchNames:
      - kafka-namespace
  podMetricsEndpoints:
    - path: /metrics
      port: tcp-prometheus
      relabelings:
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(strimzi_io_.+)
          separator: ;
        - action: replace
          regex: (.*)
          separator: ;
          sourceLabels:
            - __meta_kubernetes_namespace
          targetLabel: namespace
        - action: replace
          regex: (.*)
          separator: ;
          sourceLabels:
            - __meta_kubernetes_pod_name
          targetLabel: kubernetes_pod_name
        - action: replace
          regex: (.*)
          separator: ;
          sourceLabels:
            - __meta_kubernetes_pod_node_name
          targetLabel: node_name
        - action: replace
          regex: (.*)
          separator: ;
          sourceLabels:
            - __meta_kubernetes_pod_host_ip
          targetLabel: node_ip
  selector:
    matchExpressions:
      - key: strimzi.io/kind
        operator: In
        values:
          - Kafka
          - KafkaConnect
          - KafkaMirrorMaker
          - KafkaMirrorMaker2

Ensure that Prometheus scrapping is working as expected (I have enabled Kafka Exporter in my cluster that's why I have 3 targets here).

In grafana, add the provided dashboard in the examples:

• https://github.com/strimzi/strimzi-kafka-operator/blob/0.41.0/examples/metrics/grafana-dashboards/strimzi-kafka.json
• https://github.com/strimzi/strimzi-kafka-operator/blob/0.41.0/examples/metrics/grafana-dashboards/strimzi-zookeeper.json

Expected behavior

The Dashboards should display the metrics captured by Prometheus.

Strimzi version

0.41.0

Kubernetes version

1.30.2

Installation method

Helm

Infrastructure

Managed OVH Kubernetes

Configuration files and logs

Here is the metrics-config.json from the Kafka configmap (btw, I can see a lot of empty value here such as clientId, topic, partition..., and I'm wondering if it could be the reason of the issue):

{
   "lowercaseOutputName":true,
   "rules":[
      {
         "pattern":"kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value",
         "name":"kafka_server__",
         "type":"GAUGE",
         "labels":{
            "clientId":"",
            "topic":"",
            "partition":""
         }
      },
      {
         "pattern":"kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value",
         "name":"kafka_server__",
         "type":"GAUGE",
         "labels":{
            "clientId":"",
            "broker":":"
         }
      },
      {
         "pattern":"kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections",
         "name":"kafka_server__connections_tls_info",
         "type":"GAUGE",
         "labels":{
            "cipher":"",
            "protocol":"",
            "listener":"",
            "networkProcessor":""
         }
      },
      {
         "pattern":"kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections",
         "name":"kafka_server__connections_software",
         "type":"GAUGE",
         "labels":{
            "clientSoftwareName":"",
            "clientSoftwareVersion":"",
            "listener":"",
            "networkProcessor":""
         }
      },
      {
         "pattern":"kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):",
         "name":"kafka_server__",
         "type":"GAUGE",
         "labels":{
            "listener":"",
            "networkProcessor":""
         }
      },
      {
         "pattern":"kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)",
         "name":"kafka_server__",
         "type":"GAUGE",
         "labels":{
            "listener":"",
            "networkProcessor":""
         }
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+)Percent\\w*><>MeanRate",
         "name":"kafka____percent",
         "type":"GAUGE"
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+)Percent\\w*><>Value",
         "name":"kafka____percent",
         "type":"GAUGE"
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+)Percent\\w*, (.+)=(.+)><>Value",
         "name":"kafka____percent",
         "type":"GAUGE",
         "labels":{
            "":""
         }
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+)PerSec\\w*, (.+)=(.+), (.+)=(.+)><>Count",
         "name":"kafka____total",
         "type":"COUNTER",
         "labels":{
            "":""
         }
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+)PerSec\\w*, (.+)=(.+)><>Count",
         "name":"kafka____total",
         "type":"COUNTER",
         "labels":{
            "":""
         }
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+)PerSec\\w*><>Count",
         "name":"kafka____total",
         "type":"COUNTER"
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value",
         "name":"kafka___",
         "type":"GAUGE",
         "labels":{
            "":""
         }
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value",
         "name":"kafka___",
         "type":"GAUGE",
         "labels":{
            "":""
         }
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+)><>Value",
         "name":"kafka___",
         "type":"GAUGE"
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count",
         "name":"kafka____count",
         "type":"COUNTER",
         "labels":{
            "":""
         }
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\\d+)thPercentile",
         "name":"kafka___",
         "type":"GAUGE",
         "labels":{
            "":"",
            "quantile":"0."
         }
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count",
         "name":"kafka____count",
         "type":"COUNTER",
         "labels":{
            "":""
         }
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\\d+)thPercentile",
         "name":"kafka___",
         "type":"GAUGE",
         "labels":{
            "":"",
            "quantile":"0."
         }
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+)><>Count",
         "name":"kafka____count",
         "type":"COUNTER"
      },
      {
         "pattern":"kafka.(\\w+)<type=(.+), name=(.+)><>(\\d+)thPercentile",
         "name":"kafka___",
         "type":"GAUGE",
         "labels":{
            "quantile":"0."
         }
      }
   ]
}

Here is a sample of metrics provided by kafka JMX exporter

# HELP jvm_memory_objects_pending_finalization The number of objects waiting in the finalizer queue.
# TYPE jvm_memory_objects_pending_finalization gauge
jvm_memory_objects_pending_finalization 0.0
# HELP jvm_memory_bytes_used Used bytes of a given JVM memory area.
# TYPE jvm_memory_bytes_used gauge
jvm_memory_bytes_used{area="heap",} 3.63893344E8
jvm_memory_bytes_used{area="nonheap",} 1.2289364E8
# HELP jvm_memory_bytes_committed Committed (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_committed gauge
jvm_memory_bytes_committed{area="heap",} 2.793406464E9
jvm_memory_bytes_committed{area="nonheap",} 1.2517376E8
# HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_max gauge
jvm_memory_bytes_max{area="heap",} 1.543503872E10
jvm_memory_bytes_max{area="nonheap",} -1.0
# HELP jvm_memory_bytes_init Initial bytes of a given JVM memory area.
# TYPE jvm_memory_bytes_init gauge
jvm_memory_bytes_init{area="heap",} 1.34217728E8
jvm_memory_bytes_init{area="nonheap",} 7667712.0
# HELP jvm_memory_pool_bytes_used Used bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_used gauge
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'",} 1646336.0
jvm_memory_pool_bytes_used{pool="Metaspace",} 6.8592992E7
jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'",} 2.6801536E7
jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 7637608.0
jvm_memory_pool_bytes_used{pool="G1 Eden Space",} 5.8720256E7
jvm_memory_pool_bytes_used{pool="G1 Old Gen",} 2.9067008E8
jvm_memory_pool_bytes_used{pool="G1 Survivor Space",} 1.4503008E7
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'",} 1.8215168E7
# HELP jvm_memory_pool_bytes_committed Committed bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_committed gauge
jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_pool_bytes_committed{pool="Metaspace",} 6.946816E7
jvm_memory_pool_bytes_committed{pool="CodeHeap 'profiled nmethods'",} 2.686976E7
jvm_memory_pool_bytes_committed{pool="Compressed Class Space",} 8060928.0
jvm_memory_pool_bytes_committed{pool="G1 Eden Space",} 2.60046848E8
jvm_memory_pool_bytes_committed{pool="G1 Old Gen",} 2.5165824E9
jvm_memory_pool_bytes_committed{pool="G1 Survivor Space",} 1.6777216E7
jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-profiled nmethods'",} 1.8219008E7
# HELP jvm_memory_pool_bytes_max Max bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_max gauge
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-nmethods'",} 5836800.0
jvm_memory_pool_bytes_max{pool="Metaspace",} -1.0
jvm_memory_pool_bytes_max{pool="CodeHeap 'profiled nmethods'",} 1.22908672E8
jvm_memory_pool_bytes_max{pool="Compressed Class Space",} 1.073741824E9
jvm_memory_pool_bytes_max{pool="G1 Eden Space",} -1.0
jvm_memory_pool_bytes_max{pool="G1 Old Gen",} 1.543503872E10
jvm_memory_pool_bytes_max{pool="G1 Survivor Space",} -1.0
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-profiled nmethods'",} 1.22912768E8
# HELP jvm_memory_pool_bytes_init Initial bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_init gauge
jvm_memory_pool_bytes_init{pool="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="Metaspace",} 0.0
jvm_memory_pool_bytes_init{pool="CodeHeap 'profiled nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="Compressed Class Space",} 0.0
jvm_memory_pool_bytes_init{pool="G1 Eden Space",} 1.6777216E7
jvm_memory_pool_bytes_init{pool="G1 Old Gen",} 1.17440512E8
jvm_memory_pool_bytes_init{pool="G1 Survivor Space",} 0.0
jvm_memory_pool_bytes_init{pool="CodeHeap 'non-profiled nmethods'",} 2555904.0
# HELP jvm_memory_pool_collection_used_bytes Used bytes after last collection of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_used_bytes gauge
jvm_memory_pool_collection_used_bytes{pool="G1 Eden Space",} 0.0
jvm_memory_pool_collection_used_bytes{pool="G1 Old Gen",} 2.53215232E8
jvm_memory_pool_collection_used_bytes{pool="G1 Survivor Space",} 1.4503008E7
# HELP jvm_memory_pool_collection_committed_bytes Committed after last collection bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_committed_bytes gauge
jvm_memory_pool_collection_committed_bytes{pool="G1 Eden Space",} 2.60046848E8
jvm_memory_pool_collection_committed_bytes{pool="G1 Old Gen",} 3.60710144E8
jvm_memory_pool_collection_committed_bytes{pool="G1 Survivor Space",} 1.6777216E7
# HELP jvm_memory_pool_collection_max_bytes Max bytes after last collection of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_max_bytes gauge
jvm_memory_pool_collection_max_bytes{pool="G1 Eden Space",} -1.0
jvm_memory_pool_collection_max_bytes{pool="G1 Old Gen",} 1.543503872E10
jvm_memory_pool_collection_max_bytes{pool="G1 Survivor Space",} -1.0
# HELP jvm_memory_pool_collection_init_bytes Initial after last collection bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_init_bytes gauge
jvm_memory_pool_collection_init_bytes{pool="G1 Eden Space",} 1.6777216E7
jvm_memory_pool_collection_init_bytes{pool="G1 Old Gen",} 1.17440512E8
jvm_memory_pool_collection_init_bytes{pool="G1 Survivor Space",} 0.0
# HELP jmx_exporter_build_info A metric with a constant '1' value labeled with the version of the JMX exporter.
# TYPE jmx_exporter_build_info gauge
jmx_exporter_build_info{version="0.20.0",name="jmx_prometheus_javaagent",} 1.0
# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 2985.3
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.726650776036E9
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 565.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1048576.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2.182158336E10
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.625847296E9
# HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_bytes gauge
jvm_buffer_pool_used_bytes{pool="mapped",} 1.908550636E9
jvm_buffer_pool_used_bytes{pool="direct",} 1.3916421E7
jvm_buffer_pool_used_bytes{pool="mapped - 'non-volatile memory'",} 0.0
# HELP jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool.
# TYPE jvm_buffer_pool_capacity_bytes gauge
jvm_buffer_pool_capacity_bytes{pool="mapped",} 1.908550636E9
jvm_buffer_pool_capacity_bytes{pool="direct",} 1.391642E7
jvm_buffer_pool_capacity_bytes{pool="mapped - 'non-volatile memory'",} 0.0
# HELP jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_buffers gauge
jvm_buffer_pool_used_buffers{pool="mapped",} 191.0
jvm_buffer_pool_used_buffers{pool="direct",} 34.0
jvm_buffer_pool_used_buffers{pool="mapped - 'non-volatile memory'",} 0.0
# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0
# HELP jvm_classes_currently_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_currently_loaded gauge
jvm_classes_currently_loaded 10945.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 11019.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 74.0
# HELP kafka_ Attribute exposed for management kafka.cluster:name=InSyncReplicasCount,type=Partition,attribute=Value
# TYPE kafka_ gauge
kafka_ 1.0
kafka_{quantile="0.",} 0.0
# HELP kafka_server_ The total number of new connections established kafka.server:name=null,type=socket-server-metrics,attribute=connection-creation-total
# TYPE kafka_server_ gauge
kafka_server_ 448.0
# HELP kafka_percent Attribute exposed for management kafka.network:name=IdlePercent,type=Processor,attribute=Value
# TYPE kafka_percent gauge
kafka_percent 1.0
# HELP kafka_total Attribute exposed for management kafka.network:name=ErrorsPerSec,type=RequestMetrics,attribute=Count
# TYPE kafka_total counter
kafka_total 22.0
# HELP kafka_server_connections_software The number of connections with this client and version. kafka.server:name=null,type=socket-server-metrics,attribute=connections
# TYPE kafka_server_connections_software gauge
kafka_server_connections_software 7.0
# HELP kafka_count_total Attribute exposed for management kafka.network:name=ResponseQueueTimeMs,type=RequestMetrics,attribute=Count
# TYPE kafka_count_total counter
kafka_count 636.0
# HELP kafka_server_connections_tls_info The number of connections with this SSL cipher and protocol. kafka.server:name=null,type=socket-server-metrics,attribute=connections
# TYPE kafka_server_connections_tls_info gauge
kafka_server_connections_tls_info 1.0
# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.
# TYPE jmx_scrape_duration_seconds gauge
jmx_scrape_duration_seconds 1.117229559
# HELP jmx_scrape_error Non-zero if this scrape failed.
# TYPE jmx_scrape_error gauge
jmx_scrape_error 0.0
# HELP jmx_scrape_cached_beans Number of beans with their matching rule cached
# TYPE jmx_scrape_cached_beans gauge
jmx_scrape_cached_beans 0.0
# HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
# TYPE jvm_gc_collection_seconds summary
jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 584.0
jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 10.743
jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0
jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0
# HELP jvm_info VM version info
# TYPE jvm_info gauge
jvm_info{runtime="OpenJDK Runtime Environment",vendor="Red Hat, Inc.",version="17.0.11+9-LTS",} 1.0
# HELP jvm_threads_current Current thread count of a JVM
# TYPE jvm_threads_current gauge
jvm_threads_current 92.0
# HELP jvm_threads_daemon Daemon thread count of a JVM
# TYPE jvm_threads_daemon gauge
jvm_threads_daemon 42.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
jvm_threads_peak 93.0
# HELP jvm_threads_started_total Started thread count of a JVM
# TYPE jvm_threads_started_total counter
jvm_threads_started_total 842.0
# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
# TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
# HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
# TYPE jvm_threads_deadlocked_monitor gauge
jvm_threads_deadlocked_monitor 0.0
# HELP jvm_threads_state Current count of threads by state
# TYPE jvm_threads_state gauge
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="TERMINATED",} 0.0
jvm_threads_state{state="RUNNABLE",} 34.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="WAITING",} 19.0
jvm_threads_state{state="TIMED_WAITING",} 39.0
jvm_threads_state{state="UNKNOWN",} 0.0
# HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_total counter
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 3.8116224E7
jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen",} 8.76209152E8
jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space",} 1.90136188912E11
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 2.0317952E7
jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space",} 9.93945648E8
jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 7668576.0
jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 6.8689288E7
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 1674112.0
# HELP jmx_config_reload_failure_created Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_created gauge
jmx_config_reload_failure_created 1.726650776241E9
# HELP jmx_config_reload_success_created Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_created gauge
jmx_config_reload_success_created 1.72665077624E9
# HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_created gauge
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.726650777031E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Old Gen",} 1.726650777063E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Eden Space",} 1.726650777063E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.726650777063E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Survivor Space",} 1.726650777063E9
jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.726650777063E9
jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.726650777063E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.726650777063E9

And the Zookeeper metrics.

# HELP jvm_memory_objects_pending_finalization The number of objects waiting in the finalizer queue.
# TYPE jvm_memory_objects_pending_finalization gauge
jvm_memory_objects_pending_finalization 0.0
# HELP jvm_memory_bytes_used Used bytes of a given JVM memory area.
# TYPE jvm_memory_bytes_used gauge
jvm_memory_bytes_used{area="heap",} 1.45770896E8
jvm_memory_bytes_used{area="nonheap",} 5.5107648E7
# HELP jvm_memory_bytes_committed Committed (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_committed gauge
jvm_memory_bytes_committed{area="heap",} 2.76824064E8
jvm_memory_bytes_committed{area="nonheap",} 5.7212928E7
# HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_max gauge
jvm_memory_bytes_max{area="heap",} 1.543503872E10
jvm_memory_bytes_max{area="nonheap",} -1.0
# HELP jvm_memory_bytes_init Initial bytes of a given JVM memory area.
# TYPE jvm_memory_bytes_init gauge
jvm_memory_bytes_init{area="heap",} 1.34217728E8
jvm_memory_bytes_init{area="nonheap",} 7667712.0
# HELP jvm_memory_pool_bytes_used Used bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_used gauge
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'",} 1341056.0
jvm_memory_pool_bytes_used{pool="Metaspace",} 2.7919184E7
jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'",} 1.5605248E7
jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 2902256.0
jvm_memory_pool_bytes_used{pool="G1 Eden Space",} 5.8720256E7
jvm_memory_pool_bytes_used{pool="G1 Old Gen",} 8.6597632E7
jvm_memory_pool_bytes_used{pool="G1 Survivor Space",} 453008.0
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'",} 7339904.0
# HELP jvm_memory_pool_bytes_committed Committed bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_committed gauge
jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_pool_bytes_committed{pool="Metaspace",} 2.8442624E7
jvm_memory_pool_bytes_committed{pool="CodeHeap 'profiled nmethods'",} 1.5663104E7
jvm_memory_pool_bytes_committed{pool="Compressed Class Space",} 3211264.0
jvm_memory_pool_bytes_committed{pool="G1 Eden Space",} 1.34217728E8
jvm_memory_pool_bytes_committed{pool="G1 Old Gen",} 1.34217728E8
jvm_memory_pool_bytes_committed{pool="G1 Survivor Space",} 8388608.0
jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-profiled nmethods'",} 7340032.0
# HELP jvm_memory_pool_bytes_max Max bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_max gauge
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-nmethods'",} 5836800.0
jvm_memory_pool_bytes_max{pool="Metaspace",} -1.0
jvm_memory_pool_bytes_max{pool="CodeHeap 'profiled nmethods'",} 1.22908672E8
jvm_memory_pool_bytes_max{pool="Compressed Class Space",} 1.073741824E9
jvm_memory_pool_bytes_max{pool="G1 Eden Space",} -1.0
jvm_memory_pool_bytes_max{pool="G1 Old Gen",} 1.543503872E10
jvm_memory_pool_bytes_max{pool="G1 Survivor Space",} -1.0
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-profiled nmethods'",} 1.22912768E8
# HELP jvm_memory_pool_bytes_init Initial bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_init gauge
jvm_memory_pool_bytes_init{pool="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="Metaspace",} 0.0
jvm_memory_pool_bytes_init{pool="CodeHeap 'profiled nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="Compressed Class Space",} 0.0
jvm_memory_pool_bytes_init{pool="G1 Eden Space",} 1.6777216E7
jvm_memory_pool_bytes_init{pool="G1 Old Gen",} 1.17440512E8
jvm_memory_pool_bytes_init{pool="G1 Survivor Space",} 0.0
jvm_memory_pool_bytes_init{pool="CodeHeap 'non-profiled nmethods'",} 2555904.0
# HELP jvm_memory_pool_collection_used_bytes Used bytes after last collection of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_used_bytes gauge
jvm_memory_pool_collection_used_bytes{pool="G1 Eden Space",} 0.0
jvm_memory_pool_collection_used_bytes{pool="G1 Old Gen",} 8.6088192E7
jvm_memory_pool_collection_used_bytes{pool="G1 Survivor Space",} 453008.0
# HELP jvm_memory_pool_collection_committed_bytes Committed after last collection bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_committed_bytes gauge
jvm_memory_pool_collection_committed_bytes{pool="G1 Eden Space",} 1.34217728E8
jvm_memory_pool_collection_committed_bytes{pool="G1 Old Gen",} 1.34217728E8
jvm_memory_pool_collection_committed_bytes{pool="G1 Survivor Space",} 8388608.0
# HELP jvm_memory_pool_collection_max_bytes Max bytes after last collection of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_max_bytes gauge
jvm_memory_pool_collection_max_bytes{pool="G1 Eden Space",} -1.0
jvm_memory_pool_collection_max_bytes{pool="G1 Old Gen",} 1.543503872E10
jvm_memory_pool_collection_max_bytes{pool="G1 Survivor Space",} -1.0
# HELP jvm_memory_pool_collection_init_bytes Initial after last collection bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_collection_init_bytes gauge
jvm_memory_pool_collection_init_bytes{pool="G1 Eden Space",} 1.6777216E7
jvm_memory_pool_collection_init_bytes{pool="G1 Old Gen",} 1.17440512E8
jvm_memory_pool_collection_init_bytes{pool="G1 Survivor Space",} 0.0
# HELP zookeeper_ org.apache.ZooKeeperService:name=null,type=null,attribute=ObserverElectionDelayMS
# TYPE zookeeper_ gauge
zookeeper_ 200.0
# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.
# TYPE jmx_scrape_duration_seconds gauge
jmx_scrape_duration_seconds 0.0191964
# HELP jmx_scrape_error Non-zero if this scrape failed.
# TYPE jmx_scrape_error gauge
jmx_scrape_error 0.0
# HELP jmx_scrape_cached_beans Number of beans with their matching rule cached
# TYPE jmx_scrape_cached_beans gauge
jmx_scrape_cached_beans 0.0
# HELP jmx_exporter_build_info A metric with a constant '1' value labeled with the version of the JMX exporter.
# TYPE jmx_exporter_build_info gauge
jmx_exporter_build_info{version="0.20.0",name="jmx_prometheus_javaagent",} 1.0
# HELP jvm_info VM version info
# TYPE jvm_info gauge
jvm_info{runtime="OpenJDK Runtime Environment",vendor="Red Hat, Inc.",version="17.0.11+9-LTS",} 1.0
# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0
# HELP jvm_classes_currently_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_currently_loaded gauge
jvm_classes_currently_loaded 5486.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 5525.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 39.0
# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
# HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
# TYPE jvm_gc_collection_seconds summary
jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 49.0
jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 0.33
jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0
jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0
# HELP jvm_threads_current Current thread count of a JVM
# TYPE jvm_threads_current gauge
jvm_threads_current 46.0
# HELP jvm_threads_daemon Daemon thread count of a JVM
# TYPE jvm_threads_daemon gauge
jvm_threads_daemon 17.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
jvm_threads_peak 46.0
# HELP jvm_threads_started_total Started thread count of a JVM
# TYPE jvm_threads_started_total counter
jvm_threads_started_total 47.0
# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
# TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
# HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
# TYPE jvm_threads_deadlocked_monitor gauge
jvm_threads_deadlocked_monitor 0.0
# HELP jvm_threads_state Current count of threads by state
# TYPE jvm_threads_state gauge
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="TERMINATED",} 0.0
jvm_threads_state{state="RUNNABLE",} 26.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="WAITING",} 12.0
jvm_threads_state{state="TIMED_WAITING",} 8.0
jvm_threads_state{state="UNKNOWN",} 0.0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 108.06
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.726650203751E9
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 247.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1048576.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.9863044096E10
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 4.54668288E8
# HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_bytes gauge
jvm_buffer_pool_used_bytes{pool="mapped",} 0.0
jvm_buffer_pool_used_bytes{pool="direct",} 3.3595393E7
jvm_buffer_pool_used_bytes{pool="mapped - 'non-volatile memory'",} 0.0
# HELP jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool.
# TYPE jvm_buffer_pool_capacity_bytes gauge
jvm_buffer_pool_capacity_bytes{pool="mapped",} 0.0
jvm_buffer_pool_capacity_bytes{pool="direct",} 3.3595392E7
jvm_buffer_pool_capacity_bytes{pool="mapped - 'non-volatile memory'",} 0.0
# HELP jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_buffers gauge
jvm_buffer_pool_used_buffers{pool="mapped",} 0.0
jvm_buffer_pool_used_buffers{pool="direct",} 12.0
jvm_buffer_pool_used_buffers{pool="mapped - 'non-volatile memory'",} 0.0
# HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_total counter
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 1.7526912E7
jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen",} 8.8045568E7
jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space",} 3.967811584E9
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 7357696.0
jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space",} 1.599796E7
jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 2922952.0
jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 2.7863936E7
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 1355136.0
# HELP jmx_config_reload_failure_created Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_created gauge
jmx_config_reload_failure_created 1.726650203951E9
# HELP jmx_config_reload_success_created Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_created gauge
jmx_config_reload_success_created 1.72665020395E9
# HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_created gauge
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.726650204263E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Old Gen",} 1.726650204286E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Eden Space",} 1.726650204286E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.726650204286E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Survivor Space",} 1.726650204286E9
jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.726650204286E9
jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.726650204286E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.726650204286E9

Additional context

Thanks for your time to check if it is a bug.

Feel free to ask for more details.

[scholzj]

It seems you are mixing completely different things:

• JMX versus Prometheus metrics
• Kafka metrics versus Kafka Exporter metrics

The blog post you linked to is from 2019, so I'm not sure it it is still valid. But it focuses only on Kafka Exporter metrics. Is that what does not work for you? Or is the Kafka metrics? How does your full configuration look like? What or where are the sample of metrics from? Is it from the Strimzi Pods (in which case the metrics work) or somewhere else?

[scholzj]

The blog post was written because we added the support for the Kafka Exporter at that time. So that is its focus.

[avdev4j]

@scholzj The problem is that the Dashboard provided by the examples repo is not able to display the kafka metrics. I'm wondering if the scrapping is applying the kafka-metrics-config.yml well. I'm not sure who is supposed to manage that, but I think the metrics json file injected should drive how the metrics have to be send to Prometheus, right ?

[scholzj]

In general, Strimzi provides only the HTTP endpoint in the pods with the metrics. The rest is always up to you because everyone uses different tools and even if you use Prometheus, how to scrape the metrics depends on how your Prometheus setup looks like. That might influence whether it gets the metrics or not but also some missing labels etc.

If you see the data in Prometheus but not in Grafana, there can be several reasons for it such as:

• Misconfigured Gradfana
• Missing labels on Prometheus

You would need to check those yourself as nobody else knows your setup.

Another thing to be careful about is that the Grafana Dashboards and the ConfigMaps for the metrics are not the same for every Strimzi and Kafka version. So if you use 0.41.0, make sure you have the dashboard and the metrics configuration from 0.41.0 and not from some other version.

[avdev4j]

Maybe I misunderstanding, but could we agree that the metrics config file (https://github.com/strimzi/strimzi-kafka-operator/blob/0.41.0/packaging/examples/metrics/kafka-metrics.yaml) is supposed to transform HTTP metrics to Prometheus metrics when they are scrapped?
And then, the provided Dashboard should be fine to use to display these metrics on Grafana (https://github.com/strimzi/strimzi-kafka-operator/blob/0.41.0/examples/metrics/grafana-dashboards/strimzi-kafka.json)

This is what I'm understanding when I check the repo, but finally there is something wrong there that could be my overall understanding, my actual config or something broken here (but I doubt about it since nobody raised that before).

Regarding the versions, I agree and I carefully choose the 0.41.0 tag in the repo.

Thanks

[scholzj]

No. The example uses the Prometheus JMX exporter to export the JMX metrics provided by Kafka as a Prometheus metrics export that you can scrape. That is what you could check with curl in the Kafka pod. Then you have:

• Get your Prometheus to scrape the metrics and store them
• Get your Grafana to get the metrics from Prometheus and show them

Strimzi does not control what happens to the metrics once they leave the metrics endpoint in the Kafka pods. That is up to your Prometheus and Grafana. And while we provide some examples of what you might do with them, they are just examples and you might need to adjust them to how your Prometheus and Grafana (or any of the competing tools you might use) are installed and configured. It is for example that your Prometheus setup treats some labels differently etc. You simply have to take it and debug what the problem is and then either adjust the JMX Exporter configuration, the Dashboard and the Prometheus configurations. But I do not know your environment and I'm neither expert on Prometheus or Grafana, so I cannot tell you what exactly is the problem, sorry.

[orbi-ci-bot]

@avdev4j Were you able to progress on this? I am stuck on this as well.

[avdev4j]

hi @orbi-ci-bot
Unfortunately, no.
For some reasons I needed to reinstall it all from scratch on a cluster, I was hoping this would helped me, but finally it did not.

I can see some "kafka_server_" metrics in prometheus, but it seems not to be the whole metrics we are supposed to have.

I suspect the configuration not to be well read by the JMX Prometheus exporter. I mean the one that is created in the configmap loaded in the Kafka FS from the yaml version pushed from my git repo.

I'll try to give another try then, and I'll be happy to share progresses here.

[orbi-ci-bot]

Thank you. I am stuck on this as well. My Pod is crashing a lot and I cannot find the reason why without the Metrics. I will try to install from scratch again on my Dev and see if I can get the metrics data.

[orbi-ci-bot]

https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/grafana-dashboards/strimzi-kafka-exporter.json

I got this dashboard working by enabling service monitor in this helm chart. The Pod monitor was not working for me and so did any custom service monitors. Not sure why I didn't do this sooner.
https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus-kafka-exporter/values.yaml

You can try relabeling the service monitor values and let me know if you have any success with it but I couldn't get the relabeling part working. Might need to spend more time on this.

As for the Kraft one. I can get some values if I hard code the namespace and cluster variables. Some queries/visualizations still don't work but this is good enough for me. Just wanted to monitor the Core metrics and JVM usage

[avdev4j]

it works now, I used to put env vars in my kafka descriptor file to replace some values on the fly (loaded in my CI as deployment context) and because we have such variables in kafka metrics config like "$1" "$2" they have been replaced by an empty that made the configuration totally unusable.