Skip to content

Lambda functions OTEL cleanup #21462

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: develop
Choose a base branch
from
Draft

Lambda functions OTEL cleanup #21462

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 5 additions & 99 deletions ...on-monitoring/aws-lambda-monitoring/opentelemetry/lambda-opentelemetry-java.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,24 +4,15 @@ metaDescription: How to instrument Java applications on AWS Lambda with OpenTele
freshnessValidatedDate: never
---

<Callout
variant="tip"
title="PREVIEW"
>
AWS Lambda with OpenTelemetry for Java is still in development.

We have similar documentation for [.NET](/docs/serverless-function-monitoring/aws-lambda-monitoring/opentelemetry/lambda-opentelemetry-dotnet), but if you're using Python, Go, JavaScript, Ruby, or PHP for AWS Lambda with OpenTelemetry, you can use our Java or .NET documentation as a general guide to the setup. For additional information, see [AWS Distro for OpenTelemetry](https://aws-otel.github.io/docs/introduction).
</Callout>

This guide will cover how you can use [OpenTelemetry Lambda](https://github.com/open-telemetry/opentelemetry-lambda) to trace your Java Lambda functions using AWS's managed [OpenTelemetry Lambda Layers](https://aws-otel.github.io/docs/getting-started/lambda/lambda-java-auto-instr). OpenTelemetry makes it easy to instrument your functions, including automatic instrumentation for many popular libraries.
This guide will cover how you can use [OpenTelemetry Lambda](https://github.com/open-telemetry/opentelemetry-lambda) to trace your Java Lambda functions using AWS's managed [OpenTelemetry Lambda Layers](https://aws-otel.github.io/docs/getting-started/lambda/lambda-java-auto-instr). OpenTelemetry makes it easy to instrument your functions, including automatic instrumentation for many popular libraries. For additional information, see [AWS Distro for OpenTelemetry](https://aws-otel.github.io/docs/introduction).

## Prerequisites [#prerequisites]

This guide assumes you have the following:

* A New Relic account. If you don't have one, [create one for free](https://newrelic.com/signup).
* An AWS account. If you don't have one, [create one for free](https://aws.amazon.com/).
* A Java Lambda Function running under either the `java8.al2` or `java11` runtimes. If you don't have one yet, [create one now](https://docs.aws.amazon.com/lambda/latest/dg/lambda-java.html).
* Please refer to [AWS Distro for OpenTelemetry Lambda](https://aws-otel.github.io/docs/getting-started/lambda#getting-started-with-aws-lambda-layers) for the most up to date requirements.

### (Optional) Enable X-Ray [#xray]

Expand Down Expand Up @@ -52,69 +43,7 @@ yourFunctionHere:
This option requires that you have `AWSLambdaBasicExecutionRole` or an equivalent policy attached to your function.
</Callout>

## Step 1: Install the layer [#install]

AWS publishes a managed layer that includes the [OpenTelemetry Lambda Collector](https://github.com/open-telemetry/opentelemetry-lambda), the [OpenTelemetry Java SDK](https://github.com/open-telemetry/opentelemetry-java-instrumentation), and the [ADOT auto instrumentation agent](https://github.com/aws-observability/aws-otel-java-instrumentation).

To install it:

1. Open your function in the [Lambda Console](https://console.aws.amazon.com/lambda).
2. Under <DNT>**Layers**</DNT> in the <DNT>**Designer**</DNT> section, choose <DNT>**Add a layer**</DNT>.
3. Under <DNT>**Specify an ARN**</DNT>, paste one of the layer ARNs for your function's architecture form the list below. Replace `{region}`with your AWS region, such as `us-east-1`.
4. Choose <DNT>**Add**</DNT>.

* AMD64 / X86_64: `arn:aws:lambda:{region}:901920570463:layer:aws-otel-java-agent-amd64-ver-1-11-1:1`
* ARM64: `arn:aws:lambda:{region}:901920570463:layer:aws-otel-java-agent-arm64-ver-1-11-1:1`

For SAM or CloudFormation templates, add this to your function's properties:

```yaml
yourFunctionHere:
Type: AWS::Serverless::Function
Properties:
# ...
Layers:
# Use this if using x86_64 architecture
- !Sub arn:${AWS::Partition}:lambda:${AWS::Region}:901920570463:layer:aws-otel-java-agent-amd64-ver-1-11-1:1
# Use this if using arm64 architecture
- !Sub arn:${AWS::Partition}:lambda:${AWS::Region}:901920570463:layer:aws-otel-java-agent-arm64-ver-1-11-1:1
```

<Callout variant="important">
Refer to the [latest ARNs published by AWS](https://aws-otel.github.io/docs/getting-started/lambda/lambda-java-auto-instr) to verify the layer ARNs above are up to date.
</Callout>

## Step 2: Configure the layer [#configure]

To configure the layer we need to configure an [exec wrapper](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-modify.html#runtime-wrapper). The exec wrapper is a script that gets run during function initialization. In this case, the script configures OpenTelemetry.

1. Open your function in the [Lambda Console](https://console.aws.amazon.com/lambda).
2. Choose <DNT>**Configuration**</DNT> and then <DNT>**Environment variables**</DNT>.
3. Under <DNT>**Environment variables**</DNT>, choose <DNT>**Edit**</DNT>.
4. Choose <DNT>**Add environment variable**</DNT>.
5. For the <DNT>**Key**</DNT> set it to `AWS_LAMBDA_EXEC_WRAPPER` and set the <DNT>**Value**</DNT> to one of the following options (depends on your handler type). Then choose <DNT>**Save**</DNT>.

* `/opt/otel-handler`: for wrapping regular handlers (implementing `RequestHandler`)
* `/opt/otel-proxy-handler`: for wrapping regular handlers (implementing `RequestHandler`) proxied through API Gateway, enabling HTTP context propagation
* `/opt/otel-stream-handler`: for wrapping streaming handlers (implementing `RequestStreamHandler`), enabling HTTP context propagation for HTTP requests

For SAM or CloudFormation templates, add this to your function's properties:

```yaml
yourFunctionHere:
Type: AWS::Serverless::Function
Properties:
# ...
Environment:
Variables:
AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
```

<Callout variant="important">
Replace `/opt/otel-handler` if your function handler implements one of the other handler types.
</Callout>

## Step 3: Add New Relic environment variables [#env-variables]
## New Relic environment variables [#env-variables]

To send the OpenTelemetry data that this layer collects to New Relic, we need to configure the OpenTelemetry Lambda Collector that is packaged with the layer to export the telemetry it receives to the [New Relic OpenTelemetry Endpoint](/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/opentelemetry-quick-start). Before we do that, we first need to set some environment variables that it will depend upon.

Expand Down Expand Up @@ -148,7 +77,7 @@ yourFunctionHere:
Replace `your-license-key-here` with your New Relic <InlinePopover type="licenseKey"/>, and `otlp.nr-data.net:4317` with the endpoint appropriate for your New Relic region. See the list above.
</Callout>

## Step 4: Configure the Collector [#collector]
## Configure the Collector [#collector]

Now we will override the OpenTelemetry Lambda Collector's default configuration with one that exports telemetry to the New Relic OpenTelemetry endpoint. To do this we must include a `collector.yaml` config file with our function.

Expand Down Expand Up @@ -206,7 +135,7 @@ yourFunctionHere:
This assumes you bundled your `collector.yaml` in your function's root directory. If you bundled it somewhere else, replace `/var/task/collector.yaml` with the path to your `collector.yaml`.
</Callout>

## Step 5: View your data in the New Relic UI [view-data]
## View your data in the New Relic UI [#view-data]

First you will want to [invoke your Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/lambda-invocation.html) a few times to start generating telemetry. From there, head over to New Relic to find your [traces](https://one.newrelic.com/launcher/nr1-core.explorer?overlay=eyJuZXJkbGV0SWQiOiJkYXRhLWV4cGxvcmF0aW9uLnF1ZXJ5LWJ1aWxkZXIiLCJpbml0aWFsQWN0aXZlSW50ZXJmYWNlIjoibnJxbEVkaXRvciIsImluaXRpYWxOcnFsVmFsdWUiOiIiLCJpbml0aWFsUXVlcmllcyI6W3sibnJxbCI6IkZST00gU3BhbiBTRUxFQ1QgY291bnQoKikgd2hlcmUgbmV3cmVsaWMuc291cmNlPSclb3RscCUnIFRJTUVTRVJJRVMifV0sImluaXRpYWxDaGFydFNldHRpbmdzIjp7ImNoYXJ0VHlwZSI6IkNIQVJUX0xJTkUiLCJsaW1pdCI6NzU0MiwibGlua2VkRW50aXR5R3VpZCI6bnVsbCwibGlua2VkRGFzaGJvYXJkSWQiOm51bGwsInlTY2FsZSI6eyJzdGF0aWMiOmZhbHNlLCJkb21haW4iOltudWxsLG51bGxdfSwieVplcm8iOnRydWV9fQo=), [metrics](https://one.newrelic.com/launcher/nr1-core.explorer?overlay=eyJuZXJkbGV0SWQiOiJkYXRhLWV4cGxvcmF0aW9uLnF1ZXJ5LWJ1aWxkZXIiLCJpbml0aWFsQWN0aXZlSW50ZXJmYWNlIjoibnJxbEVkaXRvciIsImluaXRpYWxOcnFsVmFsdWUiOiIiLCJpbml0aWFsUXVlcmllcyI6W3sibnJxbCI6IkZST00gTWV0cmljIFNFTEVDVCBjb3VudCgqKSB3aGVyZSBuZXdyZWxpYy5zb3VyY2UgTElLRSAnJW90bHAlJyBUSU1FU0VSSUVTIn1dLCJpbml0aWFsQ2hhcnRTZXR0aW5ncyI6eyJjaGFydFR5cGUiOiJDSEFSVF9MSU5FIiwibGltaXQiOjc1NDIsImxpbmtlZEVudGl0eUd1aWQiOm51bGwsImxpbmtlZERhc2hib2FyZElkIjpudWxsLCJ5U2NhbGUiOnsic3RhdGljIjpmYWxzZSwiZG9tYWluIjpbbnVsbCxudWxsXX0sInlaZXJvIjp0cnVlfX0K), and [logs](https://one.newrelic.com/launcher/nr1-core.explorer?overlay=eyJuZXJkbGV0SWQiOiJkYXRhLWV4cGxvcmF0aW9uLnF1ZXJ5LWJ1aWxkZXIiLCJpbml0aWFsQWN0aXZlSW50ZXJmYWNlIjoibnJxbEVkaXRvciIsImluaXRpYWxOcnFsVmFsdWUiOiIiLCJpbml0aWFsUXVlcmllcyI6W3sibnJxbCI6IkZST00gTG9nIFNFTEVDVCBjb3VudCgqKSB3aGVyZSBuZXdyZWxpYy5zb3VyY2U9JyVvdGxwJScgVElNRVNFUklFUyJ9XSwiaW5pdGlhbENoYXJ0U2V0dGluZ3MiOnsiY2hhcnRUeXBlIjoiQ0hBUlRfTElORSIsImxpbWl0Ijo3NTQyLCJsaW5rZWRFbnRpdHlHdWlkIjpudWxsLCJsaW5rZWREYXNoYm9hcmRJZCI6bnVsbCwieVNjYWxlIjp7InN0YXRpYyI6ZmFsc2UsImRvbWFpbiI6W251bGwsbnVsbF19LCJ5WmVybyI6dHJ1ZX19Cg==).

Expand All @@ -220,29 +149,6 @@ time on AWS Lambda, and you will generally need to use this along with [provisio
Alternatively, you can use the following manual instrumentation method. By default, manual instrumentation requires fewer resources at function initialization. However, this
method requires a code change in most cases.

## Manual instrumentation [#manual-inst]

The manual instrumentation method is similar to the automatic instrumentation procedure in this document. The only difference is the Lambda Layer ARN you specify in [step 1](#install).

For manual instrumentation, instead of using the layer ARNs in [Step 2](#install), use one of the layer ARNs in the list below for your function's architecture:

* AMD64 / X86_64: `arn:aws:lambda:{region}:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-11-1:1`
* ARM64: `arn:aws:lambda:{region}:901920570463:layer:aws-otel-java-wrapper-arm64-ver-1-11-1:1`

Replace the `{region}` with your AWS region, such as `us-east-1`.

All other steps remain the same.

This alternative Lambda Layer will still wrap your Lambda function like the automatic method. It will also instrument the [AWS SDK](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/aws-sdk/aws-sdk-2.2/library) automatically. But all other libraries you use will require you to include
the library's OpenTelemetry instrumentation library from the [OpenTelemetry instrumentation repository](https://github.com/open-telemetry/opentelemetry-java-instrumentation)
in your function's dependencies and modify your code to initialize it.

You can see an [example with OKHttp on GitHub](https://github.com/open-telemetry/opentelemetry-lambda/tree/main/java#sample-applications).

<Callout variant="important">
Refer to the [latest ARNs published by AWS](https://aws-otel.github.io/docs/getting-started/lambda/lambda-java) to verify the layer ARNs above are up to date.
</Callout>

## Distributed Tracing [#distributed-tracing]

In some cases you may see fragmented distributed traces within New Relic with this configuration. This occurs when a trace starts or involves a service that is outside the ADOT context (for example, a managed AWS service). Spans from that service are created by X-Ray, not OpenTelemetry, and ADOT does not forward them to New Relic. Although the traces appear fragmented, they still provide complete insight into performance within the Lambda function as well as other services whose spans were forwarded to New Relic.
Expand Down