|
| 1 | +# Datadog Lambda Forwarder Changelog |
| 2 | + |
| 3 | +## v5.0.0 - BREAKING CHANGES |
| 4 | + |
| 5 | +### Overview |
| 6 | + |
| 7 | +Version 5.0.0 of the Datadog Lambda Forwarder introduces several breaking changes that remove deprecated features and improve log filtering behavior. This release introduces a new way to enrich your logs with tags that will reduce AWS Lambda related cost (S3, KMS and Lambda). |
| 8 | + |
| 9 | +### New Features |
| 10 | + |
| 11 | +#### 1. Backend Storage Tag Enrichment |
| 12 | + |
| 13 | +**Added:** |
| 14 | + |
| 15 | +- New `DD_ENRICH_S3_TAGS` / `DdEnrichS3Tags` parameter (default: `true`) |
| 16 | +- New `DD_ENRICH_CLOUDWATCH_TAGS` / `DdEnrichCloudwatchTags` parameter (default: `true`) |
| 17 | +- These instruct the Datadog backend to automatically enrich logs with resource tags **after ingestion** |
| 18 | +- New cloudwatch tags can appear on logs, check your Datadog log index configuration to ensure smooth transition. |
| 19 | + |
| 20 | +**Benefits:** |
| 21 | + |
| 22 | +- **Reduces forwarder cost** and execution time |
| 23 | +- Provides the same tag enrichment as `DdFetchS3Tags` and `DdFetchLogGroupTags` |
| 24 | +- Requires [Resource Collection](https://docs.datadoghq.com/integrations/amazon-web-services/#resource-collection) enabled in your AWS integration |
| 25 | + |
| 26 | +**Deprecation Notice:** |
| 27 | + |
| 28 | +- `DdFetchS3Tags` is now marked as **DEPRECATED** in favor of `DdEnrichS3Tags` |
| 29 | +- `DdFetchLogGroupTags` is now marked as **DEPRECATED** in favor of `DdEnrichCloudwatchTags` |
| 30 | +- `DD_FETCH_S3_TAGS` now defaults to `false` (previously `true`) |
| 31 | + |
| 32 | +--- |
| 33 | + |
| 34 | +### Breaking Changes |
| 35 | + |
| 36 | +#### 1. Changed Regex Matching Behavior for Log Filtering |
| 37 | + |
| 38 | +**What Changed:** |
| 39 | + |
| 40 | +- `IncludeAtMatch` / `INCLUDE_AT_MATCH` and `ExcludeAtMatch` / `EXCLUDE_AT_MATCH` regex patterns now match **only against the log message** itself |
| 41 | +- Previously, these patterns matched against the **entire JSON-formatted log** |
| 42 | + |
| 43 | +**Migration Required:** |
| 44 | + |
| 45 | +- **Review and update your filtering regex patterns** |
| 46 | +- If your patterns relied on matching against JSON structure or metadata fields, they will need to be rewritten |
| 47 | +- Example changes needed: |
| 48 | + - **Before (v4)**: `\"awsRegion\":\"us-east-1\"` (matched JSON with escaped quotes) |
| 49 | + - **After (v5)**: `"awsRegion":"us-east-1"` (matches the message content directly) |
| 50 | +- Patterns that matched the `message` field content should continue to work with minimal changes |
| 51 | + |
| 52 | +--- |
| 53 | + |
| 54 | +#### 2. Removed TCP Transport Support |
| 55 | + |
| 56 | +**What Changed:** |
| 57 | + |
| 58 | +- Removed the `DD_USE_TCP` / `DdUseTcp` environment variable and parameter |
| 59 | +- Deleted the TCP client implementation |
| 60 | +- All logs now **must** be sent via HTTP/HTTPS |
| 61 | + |
| 62 | +**Migration Required:** |
| 63 | + |
| 64 | +- Remove any configuration setting `DD_USE_TCP=true` or `DdUseTcp=true` |
| 65 | +- The forwarder will now exclusively use HTTP transport |
| 66 | +- If you were using TCP with custom ports (10516), these configurations will be ignored |
| 67 | +- The default HTTP endpoint is now `http-intake.logs.<DD_SITE>` on port 443 |
| 68 | + |
| 69 | +--- |
| 70 | + |
| 71 | +#### 3. Removed Deprecated PrivateLink Environment Variable |
| 72 | + |
| 73 | +**What Changed:** |
| 74 | + |
| 75 | +- Removed the `DD_USE_PRIVATE_LINK` / `DdUsePrivateLink` environment variable and parameter |
| 76 | + |
| 77 | +**Migration Required:** |
| 78 | + |
| 79 | +- Remove any configuration setting `DD_USE_PRIVATE_LINK=true` |
| 80 | +- **AWS PrivateLink is still fully supported**, but you must follow [PrivateLink documentation](https://docs.datadoghq.com/agent/guide/private-link/): |
| 81 | + 1. Set up VPC endpoints for `api`, `http-logs.intake`, and `trace.agent` as documented |
| 82 | + 2. Configure the forwarder with `DdUseVPC=true` |
| 83 | + 3. Set `VPCSecurityGroupIds` and `VPCSubnetIds` |
| 84 | + |
| 85 | +**Why This Changed:** |
| 86 | + |
| 87 | +- The variable was previously deprecated but not removed from past versions. |
| 88 | + |
| 89 | +--- |
| 90 | + |
| 91 | +### Upgrade Instructions |
| 92 | + |
| 93 | +Follow the usual [documentation](https://docs.datadoghq.com/logs/guide/forwarder/?tab=cloudformation#upgrade-to-a-new-version) about upgrading your Lambda Forwarder. |
| 94 | + |
| 95 | +#### Pre-Upgrade Checklist |
| 96 | + |
| 97 | +1. **Verify you're not using TCP transport:** |
| 98 | + |
| 99 | + ```bash |
| 100 | + aws lambda get-function-configuration --function-name "<YOUR_FORWARDER>" --query 'Environment.Variables.DD_USE_TCP' |
| 101 | + ``` |
| 102 | + |
| 103 | +2. **Verify you're not using the deprecated PrivateLink variable:** |
| 104 | +
|
| 105 | + ```bash |
| 106 | + aws lambda get-function-configuration --function-name "<YOUR_FORWARDER>" --query 'Environment.Variables.DD_USE_PRIVATE_LINK' |
| 107 | + ``` |
| 108 | +
|
| 109 | +3. **Review your log filtering patterns:** |
| 110 | + - If using `IncludeAtMatch` or `ExcludeAtMatch`, test your patterns against log messages only |
| 111 | + - Remove any JSON escaping (e.g., `\"` → `"`) |
| 112 | +
|
| 113 | +#### Testing |
| 114 | +
|
| 115 | +After upgrading: |
| 116 | +
|
| 117 | +1. Verify logs are being forwarded to Datadog |
| 118 | +2. Check that filtering rules still work as expected |
| 119 | +3. Confirm tag enrichment is working (check logs in Datadog Explorer) |
| 120 | +4. Monitor forwarder execution duration and errors in CloudWatch |
0 commit comments