Skip to content

Commit 4ea7b35

Browse files
authored
Separated Batch size block into reusable snippet (#70)
Fixes: #68 Unification of "Batch size" blocks with a snippet. Also replaced "Load balancing" with existing snippet.
2 parents ea5c7d8 + 4057d11 commit 4ea7b35

File tree

4 files changed

+23
-55
lines changed

4 files changed

+23
-55
lines changed
+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
## Batch size
2+
3+
The batch-bytes(), batch-lines(), and batch-timeout() options of the
4+
destination determine how many log messages syslog-ng OSE sends in a
5+
batch. The batch-lines() option determines the maximum number of
6+
messages syslog-ng OSE puts in a batch in. This can be limited based on
7+
size and time:
8+
9+
- syslog-ng OSE sends a batch every batch-timeout() milliseconds, even
10+
if the number of messages in the batch is less than batch-lines().
11+
That way the destination receives every message in a timely manner
12+
even if suddenly there are no more messages.
13+
14+
- syslog-ng OSE sends the batch if the total size of the messages in
15+
the batch reaches batch-bytes() bytes.
16+
17+
To increase the performance of the destination, increase the number of
18+
worker threads for the destination using the workers() option, or adjust
19+
the batch-bytes(), batch-lines(), batch-timeout() options.

doc/_admin-guide/070_Destinations/030_Elasticsearch-http/000_Batch_mode_and_load_balancing.md

+1-19
Original file line numberDiff line numberDiff line change
@@ -9,25 +9,7 @@ description: >-
99
the following section.
1010
---
1111

12-
## Batch size
13-
14-
The batch-bytes(), batch-lines(), and batch-timeout() options of the
15-
destination determine how many log messages syslog-ng OSE sends in a
16-
batch. The batch-lines() option determines the maximum number of
17-
messages syslog-ng OSE puts in a batch in. This can be limited based on
18-
size and time:
19-
20-
- syslog-ng OSE sends a batch every batch-timeout() milliseconds, even
21-
if the number of messages in the batch is less than batch-lines().
22-
That way the destination receives every message in a timely manner
23-
even if suddenly there are no more messages.
24-
25-
- syslog-ng OSE sends the batch if the total size of the messages in
26-
the batch reaches batch-bytes() bytes.
27-
28-
To increase the performance of the destination, increase the number of
29-
worker threads for the destination using the workers() option, or adjust
30-
the batch-bytes(), batch-lines(), batch-timeout() options.
12+
{% include doc/admin-guide/batch-size.md %}
3113

3214
### Example: HTTP batch mode
3315

doc/_admin-guide/070_Destinations/155_opensearch/000_batch-mode.md

+2-20
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,7 @@ description: >-
88
see the following section.
99
---
1010

11-
## Batch size
12-
13-
The `batch-bytes()`, `batch-lines()`, and `batch-timeout()` options specify how many log messages syslog-ng OSE sends in a batch. The `batch-lines()` option defines the maximum number of messages syslog-ng OSE includes in a batch in. This can be limited based on size and time:
14-
15-
* syslog-ng OSE sends a batch in time intervals defined by `batch-timeout()` in milliseconds, even if the number of messages in the batch is less than the amount defined in `batch-lines()`. This way the destination receives every message in a properly even if the messages cease.
16-
* syslog-ng OSE sends a batch if the total size of the messages in the batch reaches the amount specified in `batch-bytes()` in bytes.
17-
18-
To increase the performance of the destination, increase the number of worker threads for the destination using the workers() option, or adjust the batch-bytes(), batch-lines(), batch-timeout() options.
11+
{% include doc/admin-guide/batch-size.md %}
1912

2013
### Example: HTTP batch mode
2114

@@ -35,18 +28,7 @@ In the following example, a batch containing 100 messages, or a maximum of 512 k
3528

3629
## Load balancing between multiple indexers
3730

38-
In syslog-ng OSE version 3.19 and later versions, multiple URLs can be specified, for example, `url("site1" "site2")`. In this case, syslog-ng OSE sends log messages to the defined URLs in a load-balanced method. This means that syslog-ng OSE forwards each message to a single URL. For example, this can be used to send the messages to a set of ingestion nodes or indexers of a SIEM solution if a single node cannot handle the load. The order of the messages arriving to the servers can differ from the order syslog-ng OSE has received them. Use load-balancing only if a server can use the timestamp from the messages. If the server uses the timestamp when it receives the messages, the resulting order of the messages is incorrect.
39-
40-
![]({{ site.baseurl}}/assets/images/caution.png) **CAUTION:** If multiple URLs are set in the url() option, set the persist-name() option as well to avoid data loss.
41-
{: .notice--warning}
42-
43-
In syslog-ng OSE version 3.22 and later versions, any of the following formats can be used to specify multiple URLs:
44-
45-
```config
46-
url("server1", "server2", "server3"); # comma-separated strings
47-
url("server1" "server2" "server3"); # space-separated strings
48-
url("server1 server2 server3"); # space-separated within a single string
49-
```
31+
{% include doc/admin-guide/load-balancing.md %}
5032

5133
### Example: HTTP load balancing
5234

doc/_admin-guide/070_Destinations/210_Redis/000_Batch_mode_in_Redis.md

+1-16
Original file line numberDiff line numberDiff line change
@@ -6,22 +6,7 @@ description: >-
66
help of Redis's pipelining feature.
77
---
88

9-
## Batch size
10-
11-
The batch-lines(), batch-lines(), and batch-timeout() options of the
12-
destination determine how many log messages syslog-ng OSE sends in a
13-
batch. The batch-lines() option determines the maximum number of
14-
messages syslog-ng OSE puts in a batch in. This can be limited based on
15-
size and time:
16-
17-
syslog-ng OSE sends a batch every batch-timeout() milliseconds, even if
18-
the number of messages in the batch is less than batch-lines(). That way
19-
the destination receives every message in a timely manner even if
20-
suddenly there are no more messages.
21-
22-
To increase the performance of the destination, increase the number of
23-
worker threads for the destination using the workers() option, or adjust
24-
the batch-lines() and/or batch-timeout() options.
9+
{% include doc/admin-guide/batch-size.md %}
2510

2611
### Example: Redis batch mode
2712

0 commit comments

Comments
 (0)