OTEL: Exceptions not always shown human-readable in Kibana because of parsing result

[oldium]

Hi, this might be simply parsing issue or Kibana display issue, I do not know. We are using OpenTelemetry to send logs and traces via APM Server into Elastic, our version is 8.16.0.

apm-data/input/otlp/exceptions.go

Lines 90 to 95 in 1c7a133

	if err := setExceptionStacktrace(exceptionStacktrace, language, exceptionError.Exception); err != nil {
	// Couldn't parse stacktrace, set it as `error.stack_trace` instead.
	exceptionError.Exception.Stacktrace = nil
	exceptionError.Exception.Cause = nil
	exceptionError.StackTrace = exceptionStacktrace
	}

This code either fills Error.Exception.Stacktrace (object with various attributes) or Error.StackTrace (string). Kibana is unfortunately not able to display usable (human-readable) stack traces in Logs Explorer if they come parsed (see screenshots for comparison), according to code the Kibana has some code able to render parsed exceptions, but I have not seen it working neither in APM Errors, Traces nor in Logs Explorer.

For the unparsed message there would have to be a more tolerant parser, because the stack trace looks like the following:

com.google.api.client.googleapis.json.GoogleJsonResponseException: 501 Not Implemented
POST https://analytics.googleapis.com/upload/analytics/v3/management/accounts/54242820/webproperties/UA-54242820-11/customDataSources/De9KaW0jR8ufoMGGZv0epQ/uploads?uploadType=resumable
{
  "code": 501,
  "errors": [
    {
      "domain": "global",
      "message": "Operation is not implemented, or supported, or enabled.",
      "reason": "notImplemented"
    }
  ],
  "message": "Operation is not implemented, or supported, or enabled.",
  "status": "UNIMPLEMENTED"
}
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:583)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:506)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:616)
        <further lines>

Parsed error with no human-readable stack trace in Logs Explorer:

Unparsed error (there is a POST request logged as part of stack trace from Google library):

Anyway, we see no exceptions in APM Errors, though.

No stack trace in APM Errors display:

The trace document contains parsed exception:

Kibana is trying to detect exceptions even in the parsed state, but the check is actually useless and looks like a Kibana bug - the error.exception.stacktrace does not have any direct value for parsed exceptions, they are stored in parallel-structure sub-attributes error.exception.stacktrace.*:

https://github.com/elastic/kibana/blob/a41017e74e4f495c6892fbe023cff0045839bd9b/packages/kbn-discover-utils/src/utils/get_stack_trace_fields.ts#L18-L20

We used APM Java Agent until now and this always worked, so to me this looks more like an issue in conversion from OpenTelemetry messages into Elastic documents rather than Kibana display issue.

Is any more information needed? Should I create also Kibana issue? Feel free to suggest next steps.