failed to write data into buffer by buffer overflow action=:throw_exception

[naveenrpp2017]

Describe the bug

• I have deployed fluentd as kubernetes pod to upload the kubernetes log to AWS OpenSearch.

• Initially for first few days it was working properly(sending log to AWS OpenSearch), after few days the latest kubernetes logs was not available in AWS OpenSearch.

• So when i checked the fluentd pod logs i was able to see the below error:
  failed to write data into buffer by buffer overflow action=:throw_exception

To Reproduce

create fluentd as a kubernetes pod and use the below fluentd.conf from configuration tab which will send logs to AWS OpenSearch

Expected behavior

logs needs to be sent continuously to AWS OpenSearch without buffer related error

Your Environment

- Fluentd version:1.16.5
- Operating system: Amazon Linux 2
- Kernel version: 5.10.210-201.852.amzn2.x86_64

Your Configuration

# Input plugin to tail container logs
<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  read_from_head true
  <parse>
    @type multi_format
    <pattern>
      format json
      time_key @timestamp
      time_format %Y-%m-%dT%H:%M:%S.%NZ
    </pattern>
    <pattern>
      format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
      time_format %Y-%m-%dT%H:%M:%S.%N%:z
    </pattern>
  </parse>
</source>

# Kubernetes metadata filter
<filter kubernetes.**>
  @type kubernetes_metadata
</filter>

# Output plugin to send logs to OpenSearch
<match kubernetes.**>
  @type opensearch
  include_tag_key true
  host "opensearch domain"
  port "443"
  scheme https
  ssl_verify true
  ssl_version TLSv1_2
  index_name services_log
  include_timestamp true
  tag_key @log_name
  time_key @timestamp
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  buffer_chunk_limit 2M
  buffer_queue_limit 32
  flush_interval 5s
  max_retry_wait 30
  disable_retry_limit
  num_threads 8
</match>

Your Error Log

2024-05-24 04:41:09 +0000 [warn]: #0 failed to write data into buffer by buffer overflow action=:throw_exception
2024-05-24 04:41:09 +0000 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/plugin/buffer.rb:330:in `write'" tag="kubernetes.var.log.containers.service.log"

Additional context

No response

[daipom]

failed to write data into buffer by buffer overflow action=:throw_exception

This is not a bug.
This warning occurs when the buffer becomes full.

The amount of logs and the transfer rate need to be adjusted.

Is it possible that the pace of transfer is not keeping up with the pace of log generation?

[naveenrpp2017]

This is not a bug.
This warning occurs when the buffer becomes full.

The amount of logs and the transfer rate need to be adjusted.

Is it possible that the pace of transfer is not keeping up with the pace of log generation?

---

@daipom, As i am new to this, can you help me more on this, is something i need to change in flunetd.conf.
can you please guide me what changes i should do in .conf file.

[daipom]

In your config, the total buffer limit size will be buffer_chunk_limit * buffer_queue_limit (2M * 32 = 64M).
buffer space has too many data means the buffer size has reached this limit and new data cannot be written.

You can increase this limit by adjusting buffer_chunk_limit or buffer_queue_limit.
(This is an old setting format for Fluentd v0 series. Although you can still use these options in v1 series, but you should use the v1 format if possible. Anyway, it has nothing to do with this issue.)

If there's a temporary increase in data generation and the transfer can't keep up temporarily, this should mitigate the issue.

If the transfers are not keeping up constantly, this will not be effective at all.
If so, you need to reduce the data or increase the transfer rate.
For example, the following approach would be used:

• Adjust containers' log settings and limit the logs.
• Adjust in_tail setting to limit files to be tailed.
• Use filter plugins such as filter_grep to filtering logs.
• Increase num_threads (flush_thread_count for v1 series format) if there are enough CPUs.
• Adjust retry settings not to retry flushing too much if retry occurs frequently.
• Investigate external factors such as slow transfers due to transmission speed issues on the internet.

Or, you can use overflow_action drop_oldest_chunk (For v0 series old format, you need to use buffer_queue_full_action drop_oldest_chunk) to discard old data so that new data can be transferred even when the buffer size reaches the limit.

[naveenrpp2017]

Thankyou so much, will check and implement as per your suggestions.

[naveenrpp2017]

@daipom, Once i have added buffer_queue_full_action drop_oldest_chunk i am getting below error,

2024-08-26 00:06:16 +0000 [warn]: #0 failed to write data into buffer by buffer overflow action=:drop_oldest_chunk
2024-08-26 00:06:16 +0000 [warn]: #0 dropping oldest chunk to make space after buffer overflow chunk_id="xxxxxx"
2024-08-26 00:06:26 +0000 [warn]: #0 failed to flush the buffer. retry_times=915 next_retry_time=2024-08-26 00:06:54 +0000 chunk="xxxx" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster ({:host=>\"vpc-xxyyyzzz.aws\", :port=>443, :scheme=>\"https\"}): Cannot get new connection from pool."

could you please help me on this?

[iamMrGaurav]

In My case, redeploy of fluent d and fluent bit configs works.