TraceId can only be seen in my backend logs dashboard when I set the logging level as DEBUG in my spring boot application property.

[yatamraghu7]

Hi,

I am sending my logs from spring boot application to open telemetry Collector using dependency below and then from otel-collector using clickhouse I am trying to view the logs with qryn dashboard.

           <dependency>
		<groupId>io.opentelemetry.instrumentation</groupId>
		<artifactId>opentelemetry-spring-boot-starter</artifactId>
	</dependency>

<dependencyManagement>
	<dependencies>
		<dependency>
			<groupId>io.opentelemetry.instrumentation</groupId>
			<artifactId>opentelemetry-instrumentation-bom</artifactId>
			<version>2.6.0</version>
			<type>pom</type>
			<scope>import</scope>
		</dependency>
	</dependencies>
</dependencyManagement>

I can see in my backend logs dashboard which is qryn with my query Builder as logging level set to INFO only some plain logs without traceId and spanId. When i set the application log level to DEBUG and run my application and forward the logs I can see the traceId and spanId if I set the logging level as DEBUG in my qryn query but along with that I am getting huge number of irrelevant logs.

Can someone please help me how Can i sort this out and my code is present in below repository.
https://github.com/yatamraghu7/opentelemetry-poc

Thanks
Raghu Ram

[yatamraghu7]

Hi,

As per my statement above I can only see the traceID and spanID if I make the application logging level to DEBUG or else I could not find the trace information anywhere in qryn dashboard.
Here is the image reference below

Thanks,
Raghu Ram

[jeanbisutti]

If you can see the traces at the debug level, the problem may not be related to the OTel starter.

The problem may be that you have not enabled the traces in your collector config:

service:
  pipelines:
    traces:

See https://opentelemetry.io/docs/collector/configuration/

[yatamraghu7]

Hi @jeanbisutti

I did set the traces in my otel collector config file. Even though I am only able to see the traces in my qryn log dashboard when the application log level is DEBUG.

I have provided the screenshot as well in my above comment. You can refer my code here
https://github.com/yatamraghu7/opentelemetry-poc/blob/main/otel-config.yml

Thanks

[yatamraghu7]

Hi,

When I keep the logging level of sleuth to DEBUG and spring boot log level to INFO I can see the below information in my log backend

2024-08-01 11:40:46.532 UTC
Handled send of span [SdkSpan{traceId=00000000000000001111111111111111, spanId=fb2f51a3024397c0, parentSpanContext=ImmutableSpanContext{traceId=00000000000000001111111111111111, spanId=2222222222222222, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=true, valid=true}, name=HTTP POST, kind=SERVER, attributes=AttributesMap{data={http.path=/testSampleData, http.scheme=http, http.method=POST, http.request_content_length=32, http.status_code=200, net.transport=http, net.host.name=localhost, mvc.controller.class=TestController, http.route=/testSampleData, http.user_agent=curl/8.6.0, http.target=/testSampleData, mvc.controller.method=sampleTest}, capacity=128, totalAddedValues=12}, status=ImmutableStatusData{statusCode=UNSET, description=}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1722492646527886000, endEpochNanos=1722492646532540047}]

I am using my maven dependencies as below

                <dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-sleuth</artifactId>
			<version>3.1.3</version>
			<exclusions>
				<exclusion>
					<groupId>org.springframework.cloud</groupId>
					<artifactId>spring-cloud-sleuth-brave</artifactId>
				</exclusion>
			</exclusions>
		</dependency>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-sleuth-otel-autoconfigure</artifactId>
			<version>1.1.2</version>
		</dependency>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-sleuth-instrumentation</artifactId>
			<version>3.1.3</version>
		</dependency>

Can someone tell me why I can see two different span ID's and also the log is not relevant to the one that I can see in my spring boot console i.e. as below. Also it would be great If opentelemetry can support the spring cloud sleuth related logging pattern as it is so that We can export the logs as it is and can see the traceId in our log backend. Provided screenshot of missing traceId in my log backend as well. It would be great if someone can point where I am doing wrong.

Thanks
Raghu

[laurit]

Open telemetry and sleuth are competing telemetry solutions. It would be better ask sleuth questions from the sleuth community.

2024-08-01 11:40:46.532 UTC
Handled send of span [SdkSpan{traceId=00000000000000001111111111111111, spanId=fb2f51a3024397c0, parentSpanContext=ImmutableSpanContext{traceId=00000000000000001111111111111111, spanId=2222222222222222, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=true, valid=true}, name=HTTP POST, kind=SERVER, attributes=AttributesMap{data={http.path=/testSampleData, http.scheme=http, http.method=POST, http.request_content_length=32, http.status_code=200, net.transport=http, net.host.name=localhost, mvc.controller.class=TestController, http.route=/testSampleData, http.user_agent=curl/8.6.0, http.target=/testSampleData, mvc.controller.method=sampleTest}, capacity=128, totalAddedValues=12}, status=ImmutableStatusData{statusCode=UNSET, description=}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1722492646527886000, endEpochNanos=1722492646532540047}]

I think this was created by sleuth not otel instrumentation.

[yatamraghu7]

Hi @laurit and @marcingrzejszczak

I have changed my dependencies to below

                <dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-sleuth</artifactId>
			<version>3.1.3</version>
		</dependency>

Even after only using the sleuth dependency I can see that the traceID that was generated in my application logs is different than the one that i can see in my backend.

Can you please point out where the issue is here

Thanks

[yatamraghu7]

As there was no support provided here I am closing this issue for now.
THanks

[trask]

hi @yatamraghu7! since you are using sleuth you will need to seek support from that community

alternatively, you can use the OpenTelemetry Spring Boot Starter and we can support you in this community