Otel Java agent not exporting System.*, process.*, tomcat.*

[rapar8]

Describe the bug

I setup a project to export metrics to otel-collector via java agent but only certain metrics are being sent.
Not exporting:

process.*,  //Including CPU
system.*,    //Including CPU
tomcat.*

here is my below configuration

Dockerfile

WORKDIR /service-layer
RUN apk --no-cache add curl
RUN curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar --output otel-javaagent.jar
CMD ["java", "-javaagent:otel-javaagent.jar", "-jar","service-layer.jar"]

otel-java-agent-version: io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.7.0

Dependency included:

compile "io.opentelemetry:opentelemetry-exporter-otlp:*"
compile "io.opentelemetry.instrumentation:opentelemetry-runtime-metrics:1.26.0-alpha"
compile "io.opentelemetry:opentelemetry-sdk:*"
compile "io.opentelemetry:opentelemetry-sdk-metrics:*"
compile "io.opentelemetry:opentelemetry-exporter-otlp-metrics:1.14.0"
compile "io.opentelemetry:opentelemetry-api:1.41.0"
compile "io.opentelemetry.instrumentation:opentelemetry-log4j-appender-2.17:2.7.0-alpha"
compile "io.opentelemetry.instrumentation:opentelemetry-instrumentation-annotations:2.7.0"
compile "io.opentelemetry.semconv:opentelemetry-semconv:1.26.0-alpha"
compile "org.springframework.boot:spring-boot-starter-actuator:3.3.3"

Steps to reproduce

https://github.com/rapar8/opentelemetry-java-agent

Dcoker compose up

Expected behavior

it should export "process.", "system."

Actual behavior

only produce

http_server_active_requests
http_server_request_body_size_bytes_bucket
http_server_request_body_size_bytes_count
http_server_request_body_size_bytes_sum
http_server_request_duration_seconds_bucket
http_server_request_duration_seconds_count
http_server_request_duration_seconds_sum
http_server_response_body_size_bytes_bucket
http_server_response_body_size_bytes_count
http_server_response_body_size_bytes_sum
jvm_class_count
jvm_class_loaded_total
jvm_class_unloaded_total
jvm_cpu_count
jvm_cpu_time_seconds_total
jvm_gc_duration_seconds_bucket
jvm_gc_duration_seconds_count
jvm_gc_duration_seconds_sum
jvm_memory_committed_bytes
jvm_memory_limit_bytes
jvm_memory_used_after_last_gc_bytes
jvm_memory_used_bytes
jvm_thread_count
otlp_exporter_exported_total
otlp_exporter_seen_total
processedLogs_total
processedSpans_total
queueSize_ratio

Javaagent or library instrumentation version

2.7.0

Environment

• OTEL_EXPERIMENTAL_SDK_ENABLED=true
  - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:5555
  - OTEL_EXPORTER_OTLP_PROTOCOL=grpc
  - OTEL_IMR_EXPORT_INTERVAL=1000
  - OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=true
  - OTEL_INSTRUMENTATION_HTTP_CLIENT_EMIT_EXPERIMENTAL_TELEMETRY=true
  - OTEL_INSTRUMENTATION_HTTP_SERVER_EMIT_EXPERIMENTAL_TELEMETRY=true
  - OTEL_INSTRUMENTATION_JVM_METRICS_ENABLED=true
  - OTEL_INSTRUMENTATION_PROCESS_METRICS_ENABLED=true
  - OTEL_INSTRUMENTATION_RUNTIME_METRICS_ENABLED=true
  - OTEL_INSTRUMENTATION_SYSTEM_METRICS_ENABLED=true
  - OTEL_INSTRUMENTATION_TOMCAT_ENABLED=true
  - OTEL_JAVAAGENT_DEBUG=true
  - OTEL_JAVAAGENT_ENABLED=true
  - OTEL_JAVAAGENT_INSTRUMENTATION_RUNTIME_METRICS_ENABLED=true
  - OTEL_JAVAAGENT_LOGGING=application
  - OTEL_LOGS_EXPORTER=none
  - OTEL_LOG_LEVEL=trace
  - OTEL_METRICS_EXPORTER=logging
  - OTEL_METRIC_EXPORT_INTERVAL=1000
  - OTEL_RESOURCE_ATTRIBUTES="job=job,application=omsvclyr-d,service.name=service-layer,service.version=1.0,deployment.environment=development"
  - OTEL_RESOURCE_PROVIDERS_GCP_ENABLED=true
  - OTEL_SDK_ENABLED=true
  - OTEL_SERVICE_NAME=omsvclyr-d
  - OTEL_TRACES_EXPORTER=none
  - OTEL_TRACES_SAMPLER=always_on

Additional context

If I change the otel-java-agent version to 1.28.0. it produce "process.*" "system.*" metrics.

io.opentelemetry.javaagent.tooling.VersionLogger - Running on Java 17.0.12. JVM OpenJDK 64-Bit Server VM - Eclipse Adoptium - 17.0.12+7

[laurit]

Have a look at https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v2.0.0 For 2.x many metrics that were under process.runtime.jvm were moved to jvm. I'm not sure where the system and tomcat metrics came form, perhaps they are from micrometer/spring boot actuator? If so have a look at migration notes section in https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v2.1.0 If you decide to enable spring boot actuator instrumentation you may need to disable otel runtime-telemetry instrumentation to avoid duplicate metrics.

- OTEL_INSTRUMENTATION_JVM_METRICS_ENABLED=true
- OTEL_INSTRUMENTATION_PROCESS_METRICS_ENABLED=true
- OTEL_INSTRUMENTATION_RUNTIME_METRICS_ENABLED=true
- OTEL_INSTRUMENTATION_SYSTEM_METRICS_ENABLED=true

imo these flags don't exist

[rapar8]

Thanks for reply...

I am not using actuator/micrometer.

I am looking for metrics apart from JVM under

• System
• OS - process.*
• Container
• Tomcat

I have gone through doc, it says java-agent automatically support many metrics just by setting ENV variable.

Variable name you are asking comes from java-agent and doc, thought if we need to enable if not by default

Note: if I switch to old otel-java-agent.jar(1.28.0) I see all the other metrics.

[SylvainJuge]

For Tomcat, when capturing metrics with JMX Insight, you need to set OTEL_JMX_TARGET_SYSTEM=tomcat to enable them, see docs here for more details and other supported systems.

Also, the non stable JVM runtime metrics are only enabled when OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_EMIT_EXPERIMENTAL_TELEMETRY=true, see docs for more details.

[rapar8]

TOMCAT: Update
Below are the ENV fetch from deployed application

OTEL_EXPERIMENTAL_SDK_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:5555/
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_IMR_EXPORT_INTERVAL=1000
OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=true
OTEL_INSTRUMENTATION_COMMON_EXPERIMENTAL_VIEW_TELEMETRY_ENABLED=true
OTEL_INSTRUMENTATION_HTTP_CLIENT_EMIT_EXPERIMENTAL_TELEMETRY=true
OTEL_INSTRUMENTATION_HTTP_SERVER_EMIT_EXPERIMENTAL_TELEMETRY=true
OTEL_INSTRUMENTATION_JVM_METRICS_ENABLED=true
OTEL_INSTRUMENTATION_PROCESS_METRICS_ENABLED=true
OTEL_INSTRUMENTATION_RUNTIME_METRICS_ENABLED=true
OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_EMIT_EXPERIMENTAL_TELEMETRY=true
OTEL_INSTRUMENTATION_SYSTEM_METRICS_ENABLED=true
OTEL_INSTRUMENTATION_TOMCAT_ENABLED=true
OTEL_JAVAAGENT_DEBUG=true
OTEL_JAVAAGENT_ENABLED=true
OTEL_JAVAAGENT_INSTRUMENTATION_RUNTIME_METRICS_ENABLED=true
OTEL_JAVAAGENT_LOGGING=application
OTEL_JMX_TARGET_SYSTEM=tomcat                      <============
OTEL_LOG_LEVEL=trace
OTEL_LOGS_EXPORTER=none
OTEL_METRIC_EXPORT_INTERVAL=1000
OTEL_METRICS_EXPORTER=otlp
OTEL_PROPAGATORS=tracecontext
OTEL_RESOURCE_ATTRIBUTES="job=job,application=omsvclyr-d,service.name=service-layer,service.version=1.0,deployment.environment=development"
OTEL_RESOURCE_PROVIDERS_GCP_ENABLED=true
OTEL_SDK_ENABLED=true
OTEL_SERVICE_NAME=omsvclyr-d
OTEL_TRACES_EXPORTER=none
OTEL_TRACES_SAMPLER=always_on

May be you can correct me,

Still I could not find the metrics, here is my docker source code -
Execute:
Docker compose up

curl --location 'http://localhost:8080/test/test'
curl --location 'http://localhost:8080/test/env'
check the prometheus metrics in UI - http://localhost:9090/

[laurit]

To reiterate most likely the metrics you see with the 1.x version of the agent come from spring-boot-actuator/micrometer. 1.x version of the agent picks these metrics up and emits them otel metrics. The instrumentations that do this are disable by default in the 2.x version of the agent. Try using OTEL_INSTRUMENTATION_SPRING_BOOT_ACTUATOR_AUTOCONFIGURE_ENABLED=true and if that didn't help also add OTEL_INSTRUMENTATION_MICROMETER_ENABLED=true Note that when using micrometer metrics you may need to disable otel jvm metrics with OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_ENABLED=false to avoid duplicate metrics.
If you believe that the metrics that you see in 1.x don't come from actuator or micrometer you can always send the metrics to the collector with debug exporter (https://opentelemetry.io/docs/collector/troubleshooting/) and see what the instrumentation scope is. Or you could check if your telemetry backend shows it.

[github-actions]

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed automatically if there is no response from the author within 7 additional days from this comment.