JMX Exporter not on HTTPS endpoint

[Jay-boo]

Unable to Access Metrics via HTTPS Using JMX Exporter on ZooKeeper Node

Description

I’m trying to configure the JMX Exporter to expose metrics over HTTPS for a ZooKeeper node. I’ve followed the documentation recommendations, but I can only access metrics over HTTP, not HTTPS.

Setup Details

• Java version: OpenJDK 11
• ZooKeeper version: Apache ZooKeeper 3.8.4
• JMX Exporter version: 1.0.1

Command Used to Start ZooKeeper

sudo JVMFLAGS='-javaagent:/opt/zookeeper/jmx_exporter/jmx_prometheus_javaagent-1.0.1.jar=9999:/opt/zookeeper/jmx_exporter/zookeeper.yaml \
-Djavax.net.ssl.keyStore=/etc/zookeeper/ssl/zookeeper.keystore.jks \
-Djavax.net.ssl.keyStorePassword=password \
-Djavax.net.ssl.trustStore=/etc/zookeeper/ssl/zookeeper.truststore.jks \
-Djavax.net.ssl.trustStorePassword=password' \
/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/zkServer.sh start

zookeeper.yaml Configuration

ssl: true
rules:
  # Replicated ZooKeeper
  - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+)><>(\\w+)"
    name: "zookeeper_$2"
    type: GAUGE
  - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)"
    name: "zookeeper_$3"
    type: GAUGE
    labels:
      replicaId: "$2"
  - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(Packets\\w+)"
    name: "zookeeper_$4"
    type: COUNTER
    labels:
      replicaId: "$2"
      memberType: "$3"
  - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(\\w+)"
    name: "zookeeper_$4"
    type: GAUGE
    labels:
      replicaId: "$2"
      memberType: "$3"
  - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+), name3=(\\w+)><>(\\w+)"
    name: "zookeeper_$4_$5"
    type: GAUGE
    labels:
      replicaId: "$2"
      memberType: "$3"
  # Standalone ZooKeeper
  - pattern: "org.apache.ZooKeeperService<name0=StandaloneServer_port(\\d+)><>(\\w+)"
    type: GAUGE
    name: "zookeeper_$2"
  - pattern: "org.apache.ZooKeeperService<name0=StandaloneServer_port(\\d+), name1=InMemoryDataTree><>(\\w+)"
    type: GAUGE
    name: "zookeeper_$2"

Issue

• I do not get any errors when starting the ZooKeeper process.
• I can access metrics over HTTP by curling http://localhost:9999/metrics.
• However, I cannot access metrics over HTTPS.
  I can see that the running process contains well the -Djavax arguments

root      282493  2.4  2.5 3649024 201712 pts/0  Sl   19:43   0:04 java -Dzookeeper.log.dir=/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../logs -Dzookeeper.log.file=zookeeper-root-server-ltssutkfkcm241.log -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../zookeeper-server/target/classes:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../build/classes:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../zookeeper-server/target/lib/*.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../build/lib/*.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/zookeeper-prometheus-metrics-3.8.4.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/zookeeper-jute-3.8.4.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/zookeeper-3.8.4.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/snappy-java-1.1.10.5.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/slf4j-api-1.7.30.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/simpleclient_servlet-0.9.0.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/simpleclient_hotspot-0.9.0.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/simpleclient_common-0.9.0.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/simpleclient-0.9.0.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/netty-transport-native-unix-common-4.1.105.Final.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/netty-transport-native-epoll-4.1.105.Final.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/netty-transport-classes-epoll-4.1.105.Final.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/netty-transport-4.1.105.Final.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/netty-resolver-4.1.105.Final.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/netty-handler-4.1.105.Final.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/netty-common-4.1.105.Final.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/netty-codec-4.1.105.Final.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/netty-buffer-4.1.105.Final.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/metrics-core-4.1.12.1.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/logback-core-1.2.13.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/logback-classic-1.2.13.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/jline-2.14.6.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/jetty-util-ajax-9.4.53.v20231009.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/jetty-util-9.4.53.v20231009.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/jetty-servlet-9.4.53.v20231009.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/jetty-server-9.4.53.v20231009.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/jetty-security-9.4.53.v20231009.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/jetty-io-9.4.53.v20231009.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/jetty-http-9.4.53.v20231009.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/jackson-databind-2.15.2.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/jackson-core-2.15.2.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/jackson-annotations-2.15.2.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/commons-io-2.11.0.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/commons-cli-1.5.0.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../lib/audience-annotations-0.12.0.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../zookeeper-*.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../conf: -Xmx1000m -javaagent:/opt/zookeeper/jmx_exporter/jmx_prometheus_javaagent-1.0.1.jar=9999:/opt/zookeeper/jmx_exporter/zookeeper.yaml -Djavax.net.ssl.keyStore=/etc/zookeeper/ssl/zookeeper.keystore.jks -Djavax.net.ssl.keyStorePassword=password-Djavax.net.ssl.trustStore=/etc/zookeeper/ssl/zookeeper.truststore.jks -Djavax.net.ssl.trustStorePassword=password -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/zookeeper/apache-zookeeper-3.8.4-bin/bin/../conf/zoo.cfg

Is there something I’m missing in my configuration or setup to make HTTPS work?

Any ideas or suggestions would be greatly appreciated.

[dhoard]

ssl: true is used when running the standalone exporter to indicate that you want to use RMI over SSL from the exporter to the application.

You will need to add configuration section for the HTTP server.

Something like

httpServer:
  ssl:
    certificate:
      alias: localhost
rules:
  - pattern: ".*"

Various configuration examples can be found in the integration test suite. Look at JavaAgent/exporter.yaml files.

https://github.com/prometheus/jmx_exporter/tree/main/integration_test_suite/integration_tests/src/test/resources/io/prometheus/jmx/test/http/ssl

[Jay-boo]

Thx it works using the following configuration pointing on the keystore 👍🏽

httpServer:
  ssl:
    keyStore:
      filename: path_to_zookeeper_keystore
      password: your_password
    certificate:
      alias: your_alias