Getting random string appended to timestamp and @timestamp fields

[rishabhToshniwal]

Bug Report

Describe the bug
We are shipping the logs(around 150GB/day) from Couchbase pod container scheduled in AWS EKS (1.29) to AWS Firehose streams which in turn ingesting the logs to AWS OpenSearch Domain. The fluentbit is configured as a a daemonset to pick logs from /var/log/containers on each node, parse them, modify them and output to firehose streams.

We have multiple Outputs configured for each type of logs which we are segregating using rewrite_tag filter based on the labels of the pods. One such segregation is for Couchbase logs.

However we see many of the Couchbase pod logs are getting rejected by OpenSearch domain as the timestamp or @timestamp field in the log has some random string appended to it, which looks like a value from some other field is getting appended.

e.g. "timestamp": "20.backing-svc.svc">>},\n {user,<<"@goxdcr">>},\n

I tried scanning the Couchbase container logs(kubectl logs) generated for a short duration using a script and couldn't find any occurrence of ill formed date format in timestamp or @timestamp field. The Couchbase pod itself also uses a fluentbit to put stdr logs to /var/log/containers. Then the fluentbit we configured as a daemonset in EKS picks those logs. We suspect the fluentbit configuration (at our end) is messing those logs and hence we see a large number of logs getting rejected due to timestamp or @timestamp field getting corrupt values.

Reference Log Reject by OpenSearch (Check timestamp field) :

log.txt

Error Thrown By OpenSearch:

failed to parse field [timestamp] of type [date] in document

Fluentbit Config Map

fluentbit-cm.txt

Lua Script

lua.txt

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:
  Below is an example of log from Couchbase, this is not the exact failed log

example_container_log.txt

• Steps to reproduce the problem:

Expected behavior
The logs should be well formatted with correct timestamp and @timestamp format.

Screenshots

Your Environment

• Version used: public.ecr.aws/aws-observability/aws-for-fluent-bit:2.32.4
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?): EKS (kubernetes 1.29), Containerd runtime
• Server type and version:
• Operating System and version: Linux 2
• Filters and plugins:

Additional context

[patrick-stephens]

The version you're using is maintained by AWS and is using an older unsupported OSS version (1.9). I would suggest testing against the OSS image which is fluent/fluent-bit:3.2.5 currently to confirm if it is already resolved and whether it is still occurring.

For Couchbase there is also a sidecar-implementation: https://github.com/couchbase/couchbase-fluent-bit