feat: Integration test for End to End tracing #3691
Conversation
product-auto-label
bot
added
size: m
manu2
changed the title
product-auto-label
bot
added
size: l
product-auto-label
bot
added
size: m
|
Looking through the details of the failure in Kokoro - Test: Integration with Directpath using the following target log Many of the tests seem to be facing "com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: The caller does not have permission" when on trace exporter. So the underlying issue regarding permission denied for trace export is not related to this change but is causing failure in the new test introduced. |
product-auto-label
bot
added
size: l
| traceConfigurationBuilder.setProjectId(options.getProjectId()).build()); | ||
|
|
||
| String serviceName = | ||
| "java-spanner-jdbc-integration-tests-" + ThreadLocalRandom.current().nextInt(); |
There was a problem hiding this comment.
| "java-spanner-jdbc-integration-tests-" + ThreadLocalRandom.current().nextInt(); | |
| "java-spanner-integration-tests-" + ThreadLocalRandom.current().nextInt(); |
|
|
||
| static { | ||
| SpannerOptionsHelper.resetActiveTracingFramework(); | ||
| SpannerOptions.enableOpenTelemetryMetrics(); |
There was a problem hiding this comment.
why do we need to enable metrics?
| </dependency> | ||
| <dependency> | ||
| <groupId>com.google.api.grpc</groupId> | ||
| <artifactId>proto-google-cloud-trace-v1</artifactId> |
There was a problem hiding this comment.
why are we adding this?
|
|
||
| @Before | ||
| public void initSelectValueQuery() { | ||
| selectValueQuery = "SELECT @p1 + @p1 "; |
There was a problem hiding this comment.
can you initialize selectValueQuery as a static variable?
private static final String selectValueQuery = "SELECT @p1 + @p1";
| .build(); | ||
| try (TraceServiceClient client = TraceServiceClient.create(settings)) { | ||
| // It can take a few seconds before the trace is visible. | ||
| Thread.sleep(5000L); |
There was a problem hiding this comment.
Is there any better way to know that trace is exported? I just wanted to make sure this is not flakky
There was a problem hiding this comment.
Do a forceflush(), instead of adding a sleep.
sdkTracerProvider.forceFlush();
| clientTrace.getSpansList().stream() | ||
| .anyMatch( | ||
| span -> | ||
| "CloudSpannerOperation.ExecuteStreamingQuery".equals(span.getName()))); |
There was a problem hiding this comment.
We are only asserting on Client Trace and not on server trace?
| .build(); | ||
| try (TraceServiceClient client = TraceServiceClient.create(settings)) { | ||
| // It can take a few seconds before the trace is visible. | ||
| Thread.sleep(5000L); |
There was a problem hiding this comment.
Do a forceflush(), instead of adding a sleep.
sdkTracerProvider.forceFlush();
| OpenTelemetrySdk.builder() | ||
| .setTracerProvider(sdkTracerProvider) | ||
| .setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance())) | ||
| .buildAndRegisterGlobal(); |
There was a problem hiding this comment.
Do not register the tracerProvider at global, instead inject OpenTelemerty object via spanner options.
If we register it globally, we might start seeing traces coming from other parallel integration tests as enableOpenTelemetryTraces is a static method.
| // The gRPC stub and connections are created during test env set up using SpannerOptions and | ||
| // are | ||
| // reused for executing statements. | ||
| options = spannerOptionsWithEndToEndTracing(options); |
There was a problem hiding this comment.
This configuration should move to GceTestEnvConfig class where the SpannerOptions is build.
We should add a systemproperty for "End to end tracing" and use it to initialize and set opentelemetry object with tracer provider.
Refer this to see how we are enabling Direct Path in test cases , based on system property.
There was a problem hiding this comment.
@sakthivelmanii I think if there is only going to be single test for "End to end Tracing", then we might create a new Spanner Object in the Test directly.
Or maybe extend IntegrationTestEnv class and use it.
WDYT?
There was a problem hiding this comment.
Agree. We should probably handle it through system property like how we handled in other places.
| boolean foundTrace = false; | ||
| for (int attempts = 0; attempts < 2; attempts++) { | ||
| try { | ||
| Trace clientTrace = |
There was a problem hiding this comment.
Create the traceClient in "before" block
|
@manu2 I don't understand the need for this Integration test. Do we want to verify server traces as I don't see that in the assert? |
With End to End tracing, customers are able to view server side traces. The new integration tests aims to verify this from a client side pov.
Note: Currently the InegrationTestEnv sets up default interceptor which in turns creates a default GlobalOpenTelementry sdk. This sdk is then injected to the gRPC connection builder and stud. So a new opentelemetry sdk has been created in IntegrationTestEnv that sets up the required config for End to End Testing and will be used by default for all integration tests.