OpenTelemetry java instrumentation (javaagent): TargetInvocationException

[dariusj1]

Hi!

I'm trying to set up OTEL auto-instrumentation using the k8s CRDs for our java application. It's a large framework certified by its vendor to run on java 8 max.

openjdk version "1.8.0_275"                                                                                                                                    
OpenJDK Runtime Environment (Zulu 8.50.0.51-CA-linux64) (build 1.8.0_275-b01)                                                                                  
OpenJDK 64-Bit Server VM (Zulu 8.50.0.51-CA-linux64) (build 25.275-b01, mixed mode) 

At first, the instrumentation caused some issues for the application to start up, because the JAVA_TOOL_OPTIONS was captured by java -version and that changed its output (I mean it, the vendor's startup scripts are very strict about java version). I remediated it by prefixing the version check with JAVA_TOOL_OPTIONS= ${JAVA_HOME}/bin/java -version.

Now with that put aside, another error crossed my path and prevented the app from starting up:

Picked up JAVA_TOOL_OPTIONS:  -javaagent:/otel-auto-instrumentation-java/javaagent.jar
Handler error
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:591)
        at org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:527)
        at org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:327)
        at java.util.logging.LogManager$3.run(LogManager.java:399)
        at java.util.logging.LogManager$3.run(LogManager.java:396)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.util.logging.LogManager.readPrimordialConfiguration(LogManager.java:396)
        at java.util.logging.LogManager.access$800(LogManager.java:145)
        at java.util.logging.LogManager$2.run(LogManager.java:345)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.util.logging.LogManager.ensureLogManagerInitialized(LogManager.java:338)
        at java.util.logging.LogManager.getLogManager(LogManager.java:378)
        at java.util.logging.Logger.getPlatformLogger(Logger.java:572)
        at java.util.logging.LoggingProxyImpl.getLogger(LoggingProxyImpl.java:41)
        at sun.util.logging.LoggingSupport.getLogger(LoggingSupport.java:100)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.<init>(PlatformLogger.java:602)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.<init>(PlatformLogger.java:597)
        at sun.util.logging.PlatformLogger.<init>(PlatformLogger.java:239)
        at sun.util.logging.PlatformLogger.getLogger(PlatformLogger.java:198)
        at sun.util.locale.provider.LocaleServiceProviderPool.config(LocaleServiceProviderPool.java:142)
        at sun.util.locale.provider.LocaleProviderAdapter.<clinit>(LocaleProviderAdapter.java:165)
        at java.util.Calendar.createCalendar(Calendar.java:1662)
        at java.util.Calendar.getInstance(Calendar.java:1655)
        at java.text.SimpleDateFormat.initializeCalendar(SimpleDateFormat.java:657)
        at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:601)
        at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:580)
        at io.opentelemetry.javaagent.slf4j.simple.SimpleLoggerConfiguration.init(SimpleLoggerConfiguration.java:99)
        at io.opentelemetry.javaagent.slf4j.simple.SimpleLogger.init(SimpleLogger.java:185)
        at io.opentelemetry.javaagent.slf4j.simple.SimpleLogger.lazyInit(SimpleLogger.java:179)
        at io.opentelemetry.javaagent.slf4j.simple.SimpleLoggerFactory.<init>(SimpleLoggerFactory.java:45)
        at io.opentelemetry.javaagent.slf4j.simple.SimpleServiceProvider.initialize(SimpleServiceProvider.java:45)
        at io.opentelemetry.javaagent.slf4j.LoggerFactory.bind(LoggerFactory.java:195)
        at io.opentelemetry.javaagent.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:182)
        at io.opentelemetry.javaagent.slf4j.LoggerFactory.getProvider(LoggerFactory.java:490)
        at io.opentelemetry.javaagent.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:476)
        at io.opentelemetry.javaagent.logging.simple.Slf4jSimpleLoggingCustomizer.init(Slf4jSimpleLoggingCustomizer.java:47)
        at io.opentelemetry.javaagent.tooling.AgentStarterImpl.start(AgentStarterImpl.java:95)
        at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:53)
        at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:47)
        at io.opentelemetry.javaagent.bootstrap.AgentInitializer.execute(AgentInitializer.java:64)
        at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:46)
        at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:57)
        at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:45)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
        at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.ExceptionInInitializerError
        at java.util.logging.StreamHandler.configure(StreamHandler.java:89)
        at java.util.logging.StreamHandler.<init>(StreamHandler.java:107)
        at java.util.logging.ConsoleHandler.<init>(ConsoleHandler.java:98)
        ... 53 more
Caused by: java.lang.NullPointerException
        at sun.util.locale.provider.LocaleProviderAdapter.findAdapter(LocaleProviderAdapter.java:282)
        at sun.util.locale.provider.LocaleProviderAdapter.getAdapter(LocaleProviderAdapter.java:254)
        at java.util.Calendar.createCalendar(Calendar.java:1662)
        at java.util.Calendar.getInstance(Calendar.java:1640)
        at java.util.Formatter$FormatSpecifier.printDateTime(Formatter.java:2826)
        at java.util.Formatter$FormatSpecifier.print(Formatter.java:2740)
        at java.util.Formatter.format(Formatter.java:2526)
        at java.util.Formatter.format(Formatter.java:2455)
        at java.lang.String.format(String.java:2940)
        at sun.util.logging.LoggingSupport.getSimpleFormat(LoggingSupport.java:174)
        at sun.util.logging.LoggingSupport.getSimpleFormat(LoggingSupport.java:153)
        at java.util.logging.SimpleFormatter.<clinit>(SimpleFormatter.java:62)
        ... 56 more
[otel.javaagent 2024-07-15 11:23:40:436 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.32.1

I see otel and slf4j frames in the stack trace. Somehow, this not only fails the OTEL to start up, but also prevents the application from completing its startup sequence. It doesn't crash, it just remains hanging.

The application in question is based on OSGI, IDK whether it's useful info.

How can I go around this?

[dariusj1]

Found it.

The fact that it's java 8 is important.

The thing is, my app is using -Djava.locale.providers=COMPAT. And in java8 this doesn't make much sense, as COMPAT is a compatibility mode, enabling j9+ to "understand" values used in j8 and older versions.

Removing this JVM option solved the exception (and the exception can't be debugged, because it occurs in static context)

ref.: https://docs.oracle.com/en%2Fjava%2Fjavase%2F11%2Fdocs%2Fapi%2F%2F/java.base/java/util/spi/LocaleServiceProvider.html