Skip to content

Commit 72849b6

Browse files
committed
in_tail: add data reliability note
This commit adds a 'Data Reliability and Recovery' hint to the Tail input plugin documentation. It clarifies the behavior of the database offset mechanism during unexpected shutdowns (e.g., system crash, power loss). Specifically, it explains that while Fluent Bit guarantees at-least-once delivery, there is a possibility of slight offset lag and minimal data duplication upon recovery. This ensures users understand that no data is lost even in these scenarios. refs: fluent/fluent-bit#11269 Signed-off-by: jinyong.choi <[email protected]>
1 parent 7fb7ce2 commit 72849b6

File tree

1 file changed

+12
-0
lines changed

1 file changed

+12
-0
lines changed

pipeline/inputs/tail.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,18 @@ If no database file is present, positioning behavior depends on the value of `re
8989

9090
The database file essentially stores `inode=offset` so it should be unique per instance of the plugin, for example if you have two tail inputs then use two separate `db` files for each. That way each tail input can independently track its own state.
9191

92+
{% hint style="info" %}
93+
94+
**Data Reliability and Recovery**
95+
96+
During normal operations and graceful shutdowns (SIGTERM), Fluent Bit synchronizes file offsets with the database to ensure no data is lost.
97+
98+
In unexpected shutdowns (e.g., system crash, power loss, SIGKILL), Fluent Bit guarantees **at-least-once** delivery. The database offset may slightly lag behind the actual processed position if an **unexpected** shutdown occurs **after data is processed but before the new offset is committed** to the database.
99+
100+
Upon restart, Fluent Bit resumes from the last committed checkpoint. This ensures **no data is lost**, though it may result in the re-ingestion of a minimal amount of previously processed records (duplication).
101+
102+
{% endhint %}
103+
92104
{% hint style="info" %}
93105
The `unicode.encoding` parameter is dependent on the `simdutf` library, which is itself dependent on C++ version 11 or later. In environments that use earlier versions of C++, the `unicode.encoding` parameter will fail.
94106

0 commit comments

Comments
 (0)