Skip to content

Commit cd9d3a9

Browse files
GalaoaoaGalaoaoa
committed
add multi-config support in use case guide
1 parent bd973c7 commit cd9d3a9

File tree

3 files changed

+273
-0
lines changed

3 files changed

+273
-0
lines changed

‎README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
- [Use Case Guide](use_cases/)
88
- [Public Images](#public-images)
99
- [Using the stable tag](#using-the-stable-tag)
10+
- [Using the init tag](#using-the-init-tag)
1011
- [Using SSM to find available versions](#using-ssm-to-find-available-versions)
1112
- [Using SSM Parameters in CloudFormation Templates](#using-ssm-parameters-in-cloudFormation-templates)
1213
- [Using image tags](#using-image-tags)
@@ -67,6 +68,10 @@ A `stable` tag can be trusted that it is the latest version in which there are n
6768
* It has been out for at least 2 weeks or is a CVE patch with no Fluent Bit changes
6869
* No bugs have been reported in Fluent Bit which we expect will have high impact for AWS customers. This means bugs in the components that are most frequently used by AWS customers, such as the AWS outputs of the tail input
6970

71+
#### Using the init tag
72+
73+
The `init` tags indicate that an image contains init process and supports multi-config. Init tag is used in addition to our other tags, e.g. `aws-for-fluent-bit:init-latest` means this is a latest released image supports multi-config. For more information about the usage of multi-config please see our [use case guide](https://github.com/aws/aws-for-fluent-bit/blob/mainline/use_cases/init-process-for-fluent-bit/README.md) and [FireLens example](https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/multi-config-support).
74+
7075
#### Using SSM to find available versions
7176

7277
As of 2.0.0, there are SSM Public Parameters which allow you to see available versions. These parameters are available in every region that the image is available in. Any AWS account can query these parameters.

‎use_cases/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,9 @@ The goal of this guide is a central location for all tutorials on solving use ca
44

55
## Troubleshooting errors and issues
66

7+
## ECS FireLens New Features
8+
* [Source multiple configs from S3 or files](init-process-for-fluent-bit/README.md)
9+
710
## ECS FireLens Examples
811

912
All FireLens (side-car) examples for ECS are housed in their own repo: [aws-samples/amazon-ecs-firelens-examples](https://github.com/aws-samples/amazon-ecs-firelens-examples)

‎use_cases/init-process-for-fluent-bit/README.md‎

Lines changed: 265 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,265 @@
1+
# Init process for Fluent Bit on ECS, multi-config support
2+
3+
4+
5+
### What is init process and multi-config support?
6+
7+
Init process is the project we use to implement multi-config support. Multi-config support means you can use multiple config files conveniently to configure Fluent Bit on ECS. These config files can be your own config files that you upload to S3, or you can use the [built-in config files](https://github.com/aws/aws-for-fluent-bit/tree/mainline/ecs) we provide directly.
8+
9+
When you want to use the multi-config feature, please choose our Fluent Bit images with `init` tag like `aws-for-fluent-bit:init-latest` or `aws-for-fluent-bit:init-2.27.0` which contains the init process. You can find our images on [Public ECR](https://gallery.ecr.aws/aws-observability/aws-for-fluent-bit) and [Docker Hub](https://hub.docker.com/r/amazon/aws-for-fluent-bit/tags).
10+
11+
12+
13+
### Why did we create the init process?
14+
15+
Currently, if you want to set up additional configs for Fluent Bit running on ECS, you need to [build a custom image](https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/config-file-type-file). And you cannot easily set up a [PARSER] section for Fluent Bit because using a parser in Fluent Bit requires you to specify the path of the parser file under the [SERVICE] section. With the init process for Fluent Bit, these import statements are not needed, init process will automatically add them.
16+
17+
Init process is designed to solve above mentioned problems and improve user experience of using Fluent Bit on ECS. You only need to set the ARN or path of the config file in the ECS Task Definition, without telling us if it is a parser file or normal config file. Init process will process each config file, use [@INCLUDE](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file#config_include_file-1) keyword to add them to the main config file, and change the Fluent Bit command if the parser is used.
18+
19+
The init process also injects ECS Task Metadata into the Fluent Bit container as environment variables, so you can use them in the Fluent Bit config.
20+
21+
22+
23+
### How to use multi-config feature?
24+
25+
1. Use `aws-for-fluent-bit` images with `init` tag.
26+
27+
2. Specify config files as environment variable in the FireLens configuration Environment setting.
28+
29+
**Example:**
30+
31+
```
32+
"environment": [
33+
{
34+
"name": "aws_fluent_bit_init_s3_1",
35+
"value": "arn:aws:s3:::yourBucket/aaaaa.conf"
36+
},
37+
{
38+
"name": "aws_fluent_bit_init_s3_2",
39+
"value": "arn:aws:s3:::yourBucket/bbbbb.conf"
40+
},
41+
{
42+
"name": "aws_fluent_bit_init_file_1",
43+
"value": "/ecs/s3.conf"
44+
}
45+
]
46+
```
47+
48+
**We support two kinds of config files:**
49+
50+
* Config files which stored in your S3 bucket
51+
You need to set the name of the env var using prefix `aws_fluent_bit_init_s3_`, the number after this prefix cannot be repeated, and set the `ARN` of your config file as the value of the env var.
52+
53+
* Config files which from the image
54+
55+
* Use [our built-in config files](https://github.com/aws/aws-for-fluent-bit/tree/mainline/ecs)
56+
57+
You need to set the name of the env var using prefix `aws_fluent_bit_init_file_`, the number after this prefix cannot be repeated, and set the `Path` of your config file inside the image as the value of the env var.
58+
59+
* Build a custom image
60+
61+
You can build a custom image using image with `init` tag as the base and add your own configs. Set the name of the env var using prefix `aws_fluent_bit_init_file_`, the number after this prefix cannot be repeated, and set the `Path` of your own config file inside the image as the value of the env var. You can also specify a parser config directly without to do anything additional, just like the two uses above.
62+
63+
**Example:**
64+
65+
Dockerfile
66+
67+
```
68+
FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:init-latest
69+
ADD your-filter.conf /your-filter.conf
70+
ADD your-parser.conf /your-parser.conf
71+
ADD your-output.conf /your-output.conf
72+
```
73+
74+
FireLens configuration
75+
76+
```
77+
"environment": [
78+
{
79+
"name": "aws_fluent_bit_init_file_1",
80+
"value": "/your-filter.conf"
81+
},
82+
{
83+
"name": "aws_fluent_bit_init_file_2",
84+
"value": "/your-parser.conf"
85+
},
86+
{
87+
"name": "aws_fluent_bit_init_file_3",
88+
"value": "/your-output.conf"
89+
}
90+
]
91+
```
92+
93+
94+
95+
96+
3. Set up your ECS Task role
97+
98+
If you specify the config file from your S3 bucket, the init process will download it from your bucket. So please add the following permissions:
99+
100+
```
101+
{
102+
"Version": "2012-10-17",
103+
"Statement": [
104+
{
105+
"Effect": "Allow",
106+
"Action": [
107+
"s3:GetObject",
108+
"s3:GetBucketLocation"
109+
],
110+
"Resource": "*"
111+
}
112+
]
113+
}
114+
```
115+
116+
​ **Note:** [IAM roles for tasks](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) is different with [ECS task execution role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html).
117+
​ If you are not familiar with them, please check out more details.
118+
119+
120+
121+
### How to use ECS Task Metadata in Fluent Bit config?
122+
123+
The init process injects ECS Task Metadata into the Fluent Bit container as environment variables:
124+
125+
```
126+
AWS_REGION / ECS_LAUNCH_TYPE / ECS_CLUSTER / ECS_FAMILY
127+
ECS_TASK_ARN / ECS_TASK_ID / ECS_REVISION / ECS_TASK_DEFINITION
128+
```
129+
130+
You can use them as env vars directly in the Fluent Bit config.
131+
132+
**Example:**
133+
134+
```
135+
[OUTPUT]
136+
Name cloudwatch_logs
137+
Match *
138+
region ${AWS_REGION}
139+
```
140+
141+
142+
143+
### How init process works?
144+
145+
1. The init process will request [ECS Task Metadata](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4.html). Get valuable parts from the responses and set them using `export` as environment variables.
146+
147+
2. After you set the ARN or Path of the config files as environment variables in the ECS Task Definition, the init process will collect all config files which you specified (download files which come from S3 if needed).
148+
149+
3. The init process will create the main config file and use @INCLUDE keyword to add the config file FireLens generated into it.
150+
151+
4. The init process will process these config files one by one, use @INCLUDE keyword to add config files to the main config file. And the init process will check if each config file is a parser config. If it is a parser config, change the original Fluent Bit command, add -R to specify that parser. For more details, please see the example below.
152+
153+
5. The init process will finally invoke Fluent Bit with the modified main configuration file.
154+
155+
156+
157+
**Example:**
158+
159+
Suppose you specify config files as environment variable in the FireLens configuration as:
160+
161+
```
162+
"environment": [
163+
{
164+
"name": "aws_fluent_bit_init_s3_1",
165+
"value": "arn:aws:s3:::yourBucket/your-filter.conf"
166+
},
167+
{
168+
"name": "aws_fluent_bit_init_s3_2",
169+
"value": "arn:aws:s3:::yourBucket/your-parser.conf"
170+
},
171+
{
172+
"name": "aws_fluent_bit_init_s3_3",
173+
"value": "arn:aws:s3:::yourBucket/your-s3-output.conf"
174+
}
175+
]
176+
```
177+
178+
**your-filter.conf** (new config you created and init process downloaded from S3, path inside the image:`/init/fluent-bit-init-s3-files/your-filter.conf`)
179+
180+
```
181+
[FILTER]
182+
Name parser
183+
Match *
184+
Key_Name data
185+
Parser example
186+
```
187+
188+
**your-parser.conf** (new config you created and init process downloaded from S3, path inside the image:`/init/fluent-bit-init-s3-files/your-parser.conf`)
189+
190+
```
191+
[PARSER]
192+
Name example
193+
Format regex
194+
Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$
195+
```
196+
197+
**your-s3-output.conf** (new config you created and init process downloaded from S3, path inside the image:`/init/fluent-bit-init-s3-files/your-s3-output.conf`)
198+
199+
```
200+
[OUTPUT]
201+
Name s3
202+
Match *
203+
bucket your-result
204+
region ${AWS_REGION}
205+
total_file_size 1M
206+
upload_timeout 1m
207+
use_put_object On
208+
```
209+
210+
**fluent-bit.conf** ([original main config file FireLens generated](https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/), path inside the image: `/fluent-bit/etc/fluent-bit.conf`)
211+
212+
```
213+
[INPUT]
214+
Name forward
215+
unix_path /var/run/fluent.sock
216+
217+
.
218+
.
219+
.
220+
221+
222+
[OUTPUT]
223+
Name cloudwatch
224+
Match app-firelens*
225+
auto_create_group true
226+
log_group_name /aws/ecs/containerinsights/$(ecs_cluster)/application
227+
log_stream_name $(ecs_task_id)
228+
region us-west-2
229+
retry_limit 2
230+
```
231+
232+
233+
234+
**You don't need to set @INCLUDE or specify the path of the parser file under the [SERVICE] section, don't need to modify the Fluent Bit command, and don't need to tell us if there is a parser config.**
235+
236+
**The init process will download config files you specified in Task Definition from S3, and check if each config file is a parser config. If it's not a parser config, the init process will use @INCLUDE keyword to add this config to the main config file. If it's a parser config, the init process will change the original Fluent Bit command, add -R to specify that parser config.**
237+
238+
239+
240+
**After init process has processed:**
241+
242+
**fluent-bit-init.conf** (new main config file generated by init process, will used to invoke Fluent Bit, path inside the image:`/init/fluent-bit-init.conf`)
243+
244+
```
245+
@INCLUDE /fluent-bit/etc/fluent-bit.conf
246+
@INCLUDE /init/fluent-bit-init-s3-files/your-filter.conf
247+
@INCLUDE /init/fluent-bit-init-s3-files/your-s3-output.conf
248+
```
249+
250+
Original command to invoke Fluent Bit:
251+
252+
```
253+
/fluent-bit/bin/fluent-bit -e /fluent-bit/firehose.so
254+
-e /fluent-bit/cloudwatch.so -e /fluent-bit/kinesis.so
255+
-c /fluent-bit/etc/fluent-bit.conf
256+
```
257+
258+
Modified command to invoke Fluent Bit (change the main config file to `fluent-bit-init.conf`, add -R to specify the parser):
259+
260+
```
261+
/fluent-bit/bin/fluent-bit -e /fluent-bit/firehose.so
262+
-e /fluent-bit/cloudwatch.so -e /fluent-bit/kinesis.so
263+
-c /init/fluent-bit-init.conf
264+
-R /init/fluent-bit-init-s3-files/your-parser.conf
265+
```

0 commit comments

Comments
 (0)