How to resolve "ClassCircularityError" when developing plugin for java/util/concurrent/ThreadPoolExecutor

[william-peng-cn]

Hello, my friends. When developing a plugin for java/util/concurrent/ThreadPoolExecutor I met "java.lang.ClassCircularityError". Please help me, thanks!

1、this is my ClassEnhancePluginDefine above:

public class ThreadPoolExecutorMethodInstrumentation extends ClassEnhancePluginDefine {
    private static final String THREAD_POOL_EXECUTOR_CLASS = "java.util.concurrent.ThreadPoolExecutor";

    @Override
    protected ClassMatch enhanceClass() {
        return byName(THREAD_POOL_EXECUTOR_CLASS);
    }

........

2、when apply my plugin, "ClassCircularityError" occurs when initializing:

INFO 2022-12-27 17:26:33.676 main PluginResourcesResolver : find skywalking plugin define in jar:file:E:\software\skywalking-agent\activations\apm-toolkit-trace-activation-8.12.0.jar!/skywalking-plugin.def 
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.util.ServiceConfigurationError: org.apache.skywalking.apm.agent.core.boot.BootService: Provider org.apache.skywalking.apm.agent.core.profile.ProfileTaskExecutionService could not be instantiated
	at java.util.ServiceLoader.fail(ServiceLoader.java:232)
	at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	at org.apache.skywalking.apm.agent.core.boot.ServiceManager.load(ServiceManager.java:144)
	at org.apache.skywalking.apm.agent.core.boot.ServiceManager.loadAllServices(ServiceManager.java:62)
	at org.apache.skywalking.apm.agent.core.boot.ServiceManager.boot(ServiceManager.java:42)
	at org.apache.skywalking.apm.agent.SkyWalkingAgent.premain(SkyWalkingAgent.java:137)
	... 6 more
Caused by: java.lang.ClassCircularityError: java/util/concurrent/ThreadPoolExecutor
	at java.util.concurrent.Executors.newSingleThreadScheduledExecutor(Executors.java:272)
	at org.apache.skywalking.apm.agent.core.profile.ProfileTaskExecutionService.<clinit>(ProfileTaskExecutionService.java:49)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	... 12 more
FATAL ERROR in native method: processing of -javaagent failed

Process finished with exit code 1

[wu-sheng]

The simple answer is you can't. java/util/concurrent/ThreadPoolExecutor should not be instrumented.

Because agent kernel service ProfileTaskExecutionService is using this class, this service is required when you call tracing APIs. This is called ClassCircularityError.
B's initialization is relying on A's instrumentation, but A's instrumentation codes are going to using B. B->A->B is a cycle.

[hxd123456]

May I ask when the next version is planned to be released approximately

[wu-sheng]

Not planned yet. Definitely not this month.

[zimu312500]

ClassCircularityError caused by ProfileTaskExecutionService used newSingleThreadScheduledExecutor， pr （https://github.com/apache/skywalking-java/pull/556） only resolved the logger module which used threadpool，so I don't know why these PR(556) resolved this problem？

[bert82503]

We encounter this issue in product env.

skywalking-agent-9.1.0
skywalking.apm.agent
apm-jdk-threadpool-plugin-9.1.0.jar

[bert82503]

Hi, brother. @wu-sheng
Other application encounter this issue, which occurs when the app has just finished start run ready for K8s Liveness Probe (存活探针). And the app doesn't recover until it restarts.
Thanks!

The environment

• skywalking-agent-9.1.0
• Dubbo Java 3.2.10
• jdk1.8.0_333

Shortly after the application was launched, this exception occurred.
应用启动后不久，出现了这个异常。

the full exception stack:

java.lang.ClassCircularityError: java/util/concurrent/ThreadLocalRandom
  at java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2263)
  at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
  at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
  at org.apache.dubbo.remoting.exchange.support.DefaultFuture.<init>(DefaultFuture.java:99)
  at org.apache.dubbo.remoting.exchange.support.DefaultFuture.newFuture(DefaultFuture.java:128)
  at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeChannel.request(HeaderExchangeChannel.java:153)
  at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeClient.request(HeaderExchangeClient.java:109)
  at org.apache.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.request(ReferenceCountExchangeClient.java:95)
  at org.apache.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:134)
  at org.apache.dubbo.rpc.protocol.AbstractInvoker.doInvokeAndReturn(AbstractInvoker.java:248)
  at org.apache.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:191)
  at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:71)
  at org.apache.dubbo.rpc.filter.RpcExceptionFilter.invoke(RpcExceptionFilter.java:40)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:197)
  at org.apache.dubbo.rpc.protocol.ReferenceCountInvokerWrapper.invoke(ReferenceCountInvokerWrapper.java:106)
  at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invokeWithContext(AbstractClusterInvoker.java:412)
  at org.apache.dubbo.rpc.cluster.support.FailfastClusterInvoker.doInvoke(FailfastClusterInvoker.java:48)
  at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:366)
  at org.apache.dubbo.rpc.cluster.router.RouterSnapshotFilter.invoke(RouterSnapshotFilter.java:46)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.monitor.support.MonitorFilter.$sw$original$invoke$1vamrv0(MonitorFilter.java:108)
  at org.apache.dubbo.monitor.support.MonitorFilter.$sw$original$invoke$1vamrv0$accessor$$sw$p4vq621(MonitorFilter.java)
  at org.apache.dubbo.monitor.support.MonitorFilter$$sw$auxiliary$8n9i9c3.call(Unknown Source)
  at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
  at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.cluster.filter.support.MetricsClusterFilter.invoke(MetricsClusterFilter.java:57)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:52)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.cluster.filter.support.ObservationSenderFilter.invoke(ObservationSenderFilter.java:62)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.spring.security.filter.ContextHolderParametersSelectedTransferFilter.invoke(ContextHolderParametersSelectedTransferFilter.java:40)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.metrics.filter.MetricsFilter.invoke(MetricsFilter.java:86)
  at org.apache.dubbo.rpc.cluster.filter.support.MetricsConsumerFilter.invoke(MetricsConsumerFilter.java:38)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.cluster.filter.support.ConsumerClassLoaderFilter.invoke(ConsumerClassLoaderFilter.java:40)

关键根因日志

java.lang.ClassCircularityError: java/util/concurrent/ThreadLocalRandom
  at java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2263)

ConcurrentHashMap.addCount(ConcurrentHashMap.java:2263)

ThreadLocalRandom.getProbe()

sun.misc.Unsafe#getInt(java.lang.Object, long)
native方法

The full detail please to see apache/dubbo#14578