[BUG]: Container ID misdetected on EKS Fargate since 2.24.0 (spans lose kube_*/version tags and hostname)

[shouki-s]

Tracer Version(s)

2.24.0 – 2.37.0 (reproduced on 2.37.0; introduced in 2.24.0 by #5028).

Ruby Version(s)

3.2.7

Relevent Library and Version(s)

No response

Bug Report

Since v2.24.0 (#5028, "Add origin detection"), on EKS on AWS Fargate the tracer reports the ECS task ID instead of the container's 64-hex containerd ID in the Datadog-Container-ID / Datadog-Entity-ID headers. The Agent cannot resolve the task ID to a container, so all agent-side enrichment is lost: spans lose kube_*, container_*, image_* and version tags, and arrive with an empty hostname (flagged issue_type:empty_hostname at intake). Traces themselves are still ingested.

On EKS Fargate, /proc/self/cgroup contains both cgroup v1 entries whose path ends with the real 64-hex container ID, and a cgroup v2 entry (0::) whose path ends at the task level:

12:pids:/ecs/0c7d9b9350cd4c7e97ddbf5becfd6f22/0c7d9b9350cd4c7e97ddbf5becfd6f22-3057940393/kubepods/podb820dc31-db91-45b0-b920-d04f63e6400f/b33dd7f9fe94f384574f5d69e73d9618920578c2b5b2ff820934b741c57382f4
(... 11 more v1 entries with the identical path ...)
0::/ecs/0c7d9b9350cd4c7e97ddbf5becfd6f22/0c7d9b9350cd4c7e97ddbf5becfd6f22-3057940393

Root cause in lib/datadog/core/environment/container.rb (since #5028):

1. Container.entry partitions the cgroup entries so that v2 entries are scanned first ("v2 entries are preferred over v1").
2. For the v2 entry, the last path segment is 0c7d9b9350cd4c7e97ddbf5becfd6f22-3057940393. CONTAINER_REGEX does not match it, so the fallback FARGATE_14_CONTAINER_REGEX ([0-9a-f]{32}-[0-9]{1,10}) matches — but this segment is the ECS task identifier, not the container ID.
3. The scan returns at the first match, so the v1 entries containing the correct 64-hex container ID are never examined.

The pre-2.24 implementation scanned the file in order and returned the correct 64-hex ID. Notably, dd-trace-py (get_container_info) and dd-trace-go (parseContainerID) still scan in file order and return the correct 64-hex ID for this exact input, even though they carry the same Fargate task regex — so since 2.24.0 the Ruby tracer diverges from the other Datadog tracers on identical input.

The inode fallback cannot compensate either: on EKS Fargate the process runs in the host cgroup namespace (/proc/self/ns/cgroup inode = 4026531835 = HOST_CGROUP_NAMESPACE_INODE), so no in-<inode> entity can be derived.

Observed values for the same pod environment:

Implementation	container_id
ddtrace 1.13.1 / datadog 2.23.0	b33dd7f9fe94f384574f5d69e73d9618920578c2b5b2ff820934b741c57382f4 (correct, resolvable by the Agent)
datadog 2.37.0	0c7d9b9350cd4c7e97ddbf5becfd6f22-3057940393 (ECS task ID, not resolvable)

Headers captured from a live pod running 2.37.0:

Datadog-Container-ID: 0c7d9b9350cd4c7e97ddbf5becfd6f22-3057940393
Datadog-Entity-ID:    ci-0c7d9b9350cd4c7e97ddbf5becfd6f22-3057940393

Suggested direction: scan all entries for CONTAINER_REGEX first and only fall back to FARGATE_14_CONTAINER_REGEX when no entry yields a container ID. This keeps ECS Fargate working (there, no 64-hex segment exists and the task-suffixed segment is the correct identifier) while restoring correct behavior on EKS Fargate, and matches the effective precedence of the Python/Go tracers.

Workaround we currently use: pinning gem 'datadog', '2.23.0'.

Reproduction Code

Feeds the real /proc/self/cgroup content captured on an EKS Fargate pod into the gem's own parser. Run with gem install datadog -v 2.37.0 && ruby repro.rb:

require "datadog"

CGROUP_LINE = "/ecs/0c7d9b9350cd4c7e97ddbf5becfd6f22/0c7d9b9350cd4c7e97ddbf5becfd6f22-3057940393" \
              "/kubepods/podb820dc31-db91-45b0-b920-d04f63e6400f" \
              "/b33dd7f9fe94f384574f5d69e73d9618920578c2b5b2ff820934b741c57382f4"

CGROUP_CONTENT = [
  "12:pids:#{CGROUP_LINE}",
  "11:misc:#{CGROUP_LINE}",
  "10:memory:#{CGROUP_LINE}",
  "9:cpu,cpuacct:#{CGROUP_LINE}",
  "8:net_cls,net_prio:#{CGROUP_LINE}",
  "7:cpuset:#{CGROUP_LINE}",
  "6:devices:#{CGROUP_LINE}",
  "5:hugetlb:#{CGROUP_LINE}",
  "4:perf_event:#{CGROUP_LINE}",
  "3:freezer:#{CGROUP_LINE}",
  "2:blkio:#{CGROUP_LINE}",
  "1:name=systemd:#{CGROUP_LINE}",
  "0::/ecs/0c7d9b9350cd4c7e97ddbf5becfd6f22/0c7d9b9350cd4c7e97ddbf5becfd6f22-3057940393",
].join("\n")

cgroup = Datadog::Core::Environment::Cgroup
entries = CGROUP_CONTENT.lines.map { |line| cgroup.parse(line.chomp) }
cgroup.define_singleton_method(:entries) { entries }

# Simulate the host cgroup namespace observed on EKS Fargate
# (/proc/self/ns/cgroup inode = 4026531835), though it does not
# affect this code path because a (wrong) container ID is found.
Datadog::Core::Environment::Container.instance_variable_set(:@running_on_host, true)

puts Datadog::Core::Environment::Container.container_id
# datadog >= 2.24.0 prints: 0c7d9b9350cd4c7e97ddbf5becfd6f22-3057940393  (ECS task ID — wrong)
# Expected (= ddtrace 1.x, datadog <= 2.23.0, dd-trace-py, dd-trace-go):
#                          b33dd7f9fe94f384574f5d69e73d9618920578c2b5b2ff820934b741c57382f4

Configuration Block

No response

Error Logs

No response

Operating System

Amazon EKS on AWS Fargate (agent runs as a per-pod sidecar; DD_EKS_FARGATE=true).

How does Datadog help you?

No response