Support across thread tracing for SOFA-RPC

[OrezzerO]

Now SOFA-PRC plugin only support in thread tracing. It does not support across thread tracing. InvokeCallback class is the key class which does the async callback work. This pr add the ability of across thread tracing by enhancing InvokeCallback class the way like org.apache.skywalking.apm.toolkit.activation.trace.CallableOrRunnableActivation do.

• Tests(including UT, IT, E2E) are added to verify the new feature.

• If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.

• Update the CHANGES log.

[wu-sheng]

Please explain your scenario. When this happens, and what was not working.

[wu-sheng]

I think you should add callback in https://github.com/apache/skywalking-java/tree/main/test/plugin/scenarios/sofarpc-scenario and verify the callback spans through the expected file.

[wu-sheng]

@OrezzerO Is #676 somehow affected this plugin? Is SOFA requiring JDK21? It seems not. Could you change your local JDK to JDK8 or 11 to finish this?

[OrezzerO]

@OrezzerO Is #676 somehow affected this plugin? Is SOFA requiring JDK21? It seems not. Could you change your local JDK to JDK8 or 11 to finish this?

I can change my jdk. I am not familiar with skywalking scenario test, need some time to finish this work.

[wu-sheng]

I think it is not hard to run. The basic workflow is, the test is using a demo app with a compiled latest Java agent up and running. Then there is a mock traffic generator to trigger traces to the mock collector. The last step is to write expected spans to verify.

[OrezzerO]

I have commit some code to verify the callback spans through the expected file. However I have some thing not very sure:

1. Is expectedData.yaml file strongly depends on the stop order of each span? I am using async invoke ,so the stop order is not fixed. It may affect CI success rate.
2. I am not sure if "my expect of cross thread tracing" is suitable for skywalking. Please review test/plugin/scenarios/sofarpc-scenario/config/expectedData.yaml carefully.

[wu-sheng]

/healthCheck should not be included in the file.

[OrezzerO]

Removed it.

[wu-sheng]

Is expectedData.yaml file strongly depends on the stop order of each span? I am using async invoke ,so the stop order is not fixed. It may affect CI success rate.

Considering this is a new segment, which part of the span order are you referring to?

[OrezzerO]

I mean main thread segment and callback thread segment. Which segment will stop first is not fixed.

I try to sleep in main thread to make main thread segment stop after callback thread segment. The test runs successfully. So, segment order in expectedData.yaml is ignored?

[wu-sheng]

Segment order is not a part of validation in the file. You don't need to worry about it.

[wu-sheng]

By reading this, the callback is calling another RPC?

[wu-sheng]

I think we could make the test a little simpler. Just end the chain from Thread/com.alipay.sofa.rpc.message.bolt.BoltInvokerCallback/onResponse. We don't have to do another RPC. So this one, and the other new entry span(new segment) is not necessary, too.

[OrezzerO]

Yes callback is calling another RPC. I will simplify it.

[wu-sheng]

There is a risk of using this field to propagate tracing context. The input(inherit InvokeCallback) is a object, which could be reused by many callback invocations.
If SOFA RPC indicates this callback service should never be shared with different RPCs? Especially if this callback could be statusless.

An alternative solution is, you could use a wrapper, a plugin level InvokeCallback implementation to wrap the original one. We could make sure, that is one time used, and add a field(not need to be dynamic field) to hold the context.

[OrezzerO]

In the context of SOFARPC, InvokeCallback is a wrapper. Users directly use the SofaResponseCallback class. Each SOFARPC callback request corresponds uniquely to an InvokeCallback.

However, in the context of SOFA BOLT (BOLT being a wrapper for Netty which SOFARPC relies on), InvokeCallback may be reused.

It's a good idea to create a plugin level wrapper, but I am afraid I will introduce an error because of cannot dealing with classloader correctly.

I plan to create a class InvokeCallbackWrapper in plugin code. How to make it loaded by business classloader?

public class InvokeCallbackWrapper implements InvokeCallback {

    private ContextSnapshot contextSnapshot;
    private InvokeCallback invokeCallback;

    public InvokeCallbackWrapper(InvokeCallback invokeCallback) {
        this.contextSnapshot = ContextManager.capture();
        this.invokeCallback = invokeCallback;

    }

    @Override
    public void onResponse(final Object o) {
        ContextManager.createLocalSpan("Thread/" + invokeCallback.getClass().getName() + "/onResponse");
        if (contextSnapshot != null) {
            ContextManager.continued(contextSnapshot);
        }
        try {
            invokeCallback.onResponse(o);
        } catch (Throwable t) {
            ContextManager.activeSpan().log(t);
            throw t;
        } finally {
            ContextManager.stopSpan();
            contextSnapshot = null;
        }

    }

    @Override
    public void onException(final Throwable throwable) {
        ContextManager.createLocalSpan("Thread/" + invokeCallback.getClass().getName() + "/onException");
        if (contextSnapshot != null) {
            ContextManager.continued(contextSnapshot);
        }
        if (throwable != null) {
            AbstractSpan abstractSpan = ContextManager.activeSpan();
            if (abstractSpan != null) {
                abstractSpan.log(throwable);
            }
        }
        try {
            invokeCallback.onException(throwable);
        } catch (Throwable t) {
            ContextManager.activeSpan().log(t);
            throw t;
        } finally {
            ContextManager.stopSpan();
            contextSnapshot = null;
        }
    }

    @Override
    public Executor getExecutor() {
        return invokeCallback.getExecutor();
    }
}

[wu-sheng]

I plan to create a class InvokeCallbackWrapper in plugin code. How to make it loaded by business classloader?

All plugin interceptor relative codes are loaded in your target class loader, otherwise, all existing logic will fail. I think in this case, users of SOFA would not be aware of the wrapper, as it is only running internally.

[wu-sheng]

Classloader doesn't need to be worried, AFAIK.

[wu-sheng]

These could be replaced by annotations.

[OrezzerO]

Fixed it.

[wu-sheng]

Generally, this PR is good, just some nits. Please fix them, then it is good to merge.

[wu-sheng]

I tried to apply the change, but it seems not work.

[OrezzerO]

I tried to apply the change, but it seems not work.

Missed import lombok.AccessLevel; statement; added it.