Skip to content

Brave 4.9
Compare
Choose a tag to compare
@codefromthecrypt codefromthecrypt released this 17 Oct 16:54
· 967 commits to master since this release
4.9.1
e09b27b

Brave 4.9 adds Amazon X-Ray interop, extra field propagation and refined Kafka tracing

Extra Field propagation

We've had requests in the past to propagate extra fields, such as a request ID or experimental group flags.
For example, if you are in a Cloud Foundry environment, your edge request includes a "x-vcap-request-id" field. You can now use ExtraFieldPropagation to portably push arbitrary fields across your call graph.

// when you initialize the builder, define the extra field you want to propagate.
// fields here are added alongside trace identifiers in http or message headers.
tracingBuilder.propagationFactory(
  ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "x-vcap-request-id")
);

// You can also access these ad-hoc for tagging or log correlation
requestId = ExtraFieldPropagation.current("x-vcap-request-id");

Thanks to @jcchavezs for the help vetting this idea

Amazon X-Ray interop

Amazon X-Ray is a distributed tracing service that shares a lot in common with Zipkin. Brave 4.9 adds the ability to pass-through or actively participate in X-Ray services. Thanks very much to Abhishek Singh from Amazon for weekends of support on this and @cemo for early testing.

AWSPropagation

AWSPropagation switches header format from B3 to x-aws-trace-id, used by services like
ALB, API Gateway, and Lambda. Notably, this format uses a single field, not several. Using AWS propagation means you can read incoming traces that pass through Amazon infrastructure such as ALBs. It also means other Amazon services such as Lambda can join your traces, as can anything instrumented with Amazon's X-Ray SDK.

To switch to this header format, use Tracing.Builder.propagationFactory(AWSPropagation.FACTORY)

XRayUDPReporter

XRayUDPReporter converts spans from Zipkin v2 format to Amazon X-Ray format, and sends them via UDP messages. Sending to X-Ray is an alternative to normal Zipkin storage, and is advised if you are running in Amazon's cloud and using AWSPropagation. If you don't, traces in X-Ray and Zipkin will have gaps for spans sent to one and not the other.

To report spans to X-Ray, use Tracing.Builder.spanReporter(XRayUDPReporter.create())

Note XRayUDPReporter is a part of zipkin-aws and can be used by non-brave applications as well.

Lambda Tracing

You can use Brave to trace Java lambda functions. If you are, you'll need to extract to root trace ID differently, as the trace context ends up in env variables. If you are interested in more, please raise an issue!

Here's a snippet from a more complete lambda example:

  @Override
  public O handleRequest(I input, Context context) {
    /** Always start lambda functions from the root in the env */
    Span span = tracer.nextSpan(AWSPropagation.extractLambda())
        .name(context.getFunctionName())
        .start();

Pass through tracing

You may want to pass-through amazon trace headers as opposed to joining those traces. This is a hybrid way, where for example most of your data will be in Zipkin, yet brave won't break X-Ray traces either.

To enable pass-through, just add "x-amzn-trace-id" to extra fields

tracingBuilder.propagationFactory(
  ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "x-amzn-trace-id")
);

With this in place, you can tag the amazon formatted trace ID for correlation purposes like so:

// will look like 1-67891233-abcdef012345678912345678
String awsTraceId = AWSPropagation.currentTraceId();
if (awsTraceId != null) span.tag("aws.trace_id", awsTraceId);

Putting it all together!

If you want to send to X-Ray, here's the minimum setup in Java and Spring XML:

    return Tracing.newBuilder()
        .localServiceName("your_service_name")
        .propagationFactory(AWSPropagation.FACTORY)
        .spanReporter(XRayUDPReporter.create()).build()
  <bean id="tracing" class="brave.spring.beans.TracingFactoryBean">
    <property name="localServiceName" value="your_service_name"/>
    <property name="propagationFactory">
      <util:constant static-field="brave.propagation.aws.AWSPropagation.FACTORY"/>
    </property>
    <property name="spanReporter">
      <bean class="zipkin.reporter.xray_udp.XRayUDPReporter" factory-method="create"/>
    </property>
  </bean>

Refined Kafka tracing

Before, we created a "consumer span" for each message in a bulk poll. This can create a lot of spans, as for example the default max records per poll is 500. We now create one consumer span per poll/topic.

When you are ready to process a message, use KafkaTracing.nextSpan(record) to create a span.

Please check out the docs for more. Thanks very much to @ImFlog for the help on design and brainstorming this.

Other notes

  • To support a lot of the above, we made a new library hook: Tracer.nextSpan(extractedContext)
    • Use this if you are creating custom RPC or messaging instrumentation, as it automates some dancing
  • We dropped the JSR 305 @Nullable annotations as they interfere both with OSGi and Java 9
    • Pay attention to docs and our source-retention annotations, if you are writing custom code
Assets 2
Loading