log: add system_config option to force stacktrace level

[daipom]

Which issue(s) this PR fixes:
None.

What this PR does / why we need it:

What this PR does

Add a new system_config option: forced_stacktrace_level.

• Option: system/log/forced_stacktrace_level
• Type: enum
  • none, trace, debug, info, warn, error, fatal
• Default: none

By default, the behavior of Fluentd does not change.
So, this does not affect existing users.

If you set this option and change the value, the log levels of stacktraces are forced to that value.

Stacktraces that do not meet the log level are discarded in advance.
After the log level is changed, it is judged again for logging or discarded finally.
For example, by default (log_level info), ...

• All trace/debug stacktraces are not logged regardless of this option.
• If you set forced_stacktrace_level debug, then all stacktraces are not logged.

This is because we should avoid logging logs that contradict log_level, and to avoid unexpectedly logging logs that are initially trace/debug level.

Setting example:

<system>
  <log>
    forced_stacktrace_level info
  </log>
</system>

why we need it

To make it easier to exclude stacktraces from monitoring Fluentd's own logs.
(You don't need this feature if using @FLUENT_LOG to transfer. For some reason, there are users who need to monitor Fluentd's log files directly. This feature is for those uses.)

For example, there could be a system that raises an alert when an error-level log is detected.
You may want to exclude some error-level logs from the detection.
However, if stacktraces are logged at error level, you need to exclude those stacktraces as well.
With the current specification, that is troublesome.

Ideally, it would be preferable to exclude all stacktraces from the detection.
It is sufficient to have access to stacktraces when investigating the details.

An easy way to improve the current situation is this feature.
If you are detecting error logs as in the previous example, you can easily exclude all stacktraces from the detection by forcing the stacktrace level to info with this feature.

Docs Changes:

• system-config: add option forced_stacktrace_level fluentd-docs-gitbook#589

Release Note:
System configuration: Add forced_stacktrace_level to force the log level of stacktraces.

Sample:

<system>
  <log>
    forced_stacktrace_level info
  </log>
</system>

<source>
  @type sample
  @id test
  tag test.fail
</source>

<match test.fail>
  @type null
  never_flush
</match>

Without forced_stacktrace_level info:

2025-06-23 18:16:45 +0900 [info]: #0 fluent/log.rb:371:info: fluentd worker is now running worker=0
2025-06-23 18:16:46 +0900 [warn]: #0 fluent/log.rb:392:warn: emit transaction failed: error_class=RuntimeError error="failed to flush" location="/home/daipom/work/fluentd/fluentd/lib/fluent/plugin/out_null.rb:54:in `process'" tag="test.fail"
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/out_null.rb:54:in `process'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/output.rb:865:in `emit_sync'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/event_router.rb:115:in `emit_stream'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/event_router.rb:106:in `emit'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `block in emit'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `times'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `emit'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:100:in `run'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

With forced_stacktrace_level info:

2025-06-23 18:13:59 +0900 [warn]: #0 fluent/log.rb:392:warn: emit transaction failed: error_class=RuntimeError error="failed to flush" location="/home/daipom/work/fluentd/fluentd/lib/fluent/plugin/out_null.rb:54:in `process'" tag="test.fail"
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/out_null.rb:54:in `process'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/output.rb:865:in `emit_sync'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/event_router.rb:115:in `emit_stream'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/event_router.rb:106:in `emit'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `block in emit'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `times'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `emit'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:100:in `run'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

[daipom]

The failure of CI is unrelated to this PR.
See

• HttpServer plugin helper can not work well with io-stream 0.8 on Windows and Ruby 3.3/3.4/HEAD #5009

[daipom]

Ready.

[daipom]

Note: This feature is for users who need to monitor Fluentd's log files directly.
You don't need this feature if using @FLUENT_LOG.

[kenhys]

LGTM.

[daipom]

Thanks!