[Q&A] is regex "capture until a character" supported?

[adelbordbari]

What is a problem?

in my regex pattern, I have few cases where I need to "capture everything until a character" (e.g., " or \n)
this works well in regex101 (where I test my patterns) but in fluentd, it does not. exactly any capture group that includes this rule, throws this error: [warn]: #0 pattern not matched data, while other captures are fine

here's the link of my regex pattern: https://regex101.com/r/1ptfv6/1
my question is, is this pattern supported? why doesn't it work in fluentd but it does in regex? what are my alternatives?

thank you.

Describe the configuration of Fluentd

<source>
        @type udp
        port xxx
        tag mikrotik.logs
        <parse>
                @type regexp
                expression /(?<service>dns|dhcp|firewall)[, ](?<type>packet|debug,packet|done|local)(\s+((question: (?<question>.*?(?=.:)))|(secs>
        </parse>
</source>

<filter mikrotik.logs>
        @type record_transformer
        <record>
                tag ${tag}
                timestamp ${time}
        </record>
</filter>

<match mikrotik.logs>
        @type stdout
        <format>
                @type json
        </format>
</match>

Describe the logs of Fluentd

{"service":"dns","type":"packet","udp_msg":"sending udp query to 192.168.40.2:53:","tag":"mikrotik.logs","timestamp":"2024-08-27 08:02:32 +0000"}
2024-08-27 08:02:32 +0000 [warn]: #0 pattern not matched data="dns,packet id:9855 rd:1 tc:0 aa:0 qr:0 ra:0 QUERY 'no error'"
{"service":"dns","type":"packet","question":"adliran.ir","tag":"mikrotik.logs","timestamp":"2024-08-27 08:02:32 +0000"}
{"service":"dns","type":"done","query_id":"777226","query_status":"failure","tag":"mikrotik.logs","timestamp":"2024-08-27 08:02:33 +0000"}
{"service":"dns","type":"done","query_id":"777227","query_status":"failure","tag":"mikrotik.logs","timestamp":"2024-08-27 08:02:34 +0000"}
2024-08-27 08:02:34 +0000 [warn]: #0 pattern not matched data="dhcp,debug,packet dhcp-client on ether4-slave-local sending discover with id 1016635723 to 255.255.255.255"
{"service":"dhcp","type":"debug,packet","secs":"6","tag":"mikrotik.logs","timestamp":"2024-08-27 08:02:34 +0000"}
2024-08-27 08:02:34 +0000 [warn]: #0 pattern not matched data="dhcp,debug,packet     flags = broadcast"
{"service":"dhcp","type":"debug,packet","ciaddr":"0.0.0.0","tag":"mikrotik.logs","timestamp":"2024-08-27 08:02:34 +0000"}
{"service":"dhcp","type":"debug,packet","chaddr":"D4:CA:6D:86:FA:84","tag":"mikrotik.logs","timestamp":"2024-08-27 08:02:34 +0000"}
2024-08-27 08:02:34 +0000 [warn]: #0 pattern not matched data="dhcp,debug,packet     Host-Name = \"dezhDHCP\""
{"service":"dhcp","type":"debug,packet","msg_type":"discover","tag":"mikrotik.logs","timestamp":"2024-08-27 08:02:34 +0000"}
2024-08-27 08:02:34 +0000 [warn]: #0 pattern not matched data="dhcp,debug,packet     Parameter-List = Subnet-Mask,Classless-Route,Router,Static-Route,Domain-Server,NTP-Server,CAPWAP-Server,Vendor-Specific"
2024-08-27 08:02:34 +0000 [warn]: #0 pattern not matched data="dhcp,debug,packet     Client-Id = 01-D4-CA-6D-86-FA-84"
{"service":"dns","type":"packet","udp_msg":"sending udp query to 192.168.40.2:53:","tag":"mikrotik.logs","timestamp":"2024-08-27 08:02:34 +0000"}
2024-08-27 08:02:34 +0000 [warn]: #0 pattern not matched data="dns,packet id:8540 rd:1 tc:0 aa:0 qr:0 ra:0 QUERY 'no error'"
{"service":"dns","type":"packet","question":"adliran.ir","tag":"mikrotik.logs","timestamp":"2024-08-27 08:02:34 +0000"}
{"service":"dns","type":"local","query_question":"query: #777229 sana.adliran.ir","dns_type":"A","tag":"mikrotik.logs","timestamp":"2024-08-27 08:02:34 +0000"}

Screenshot

for better readability

Environment

- Fluentd version: 1.16.2
- TD Agent version:
- Fluent Package version:
- Docker image (tag): fluent/fluentd:latest
- Operating system: ubuntu 20
- Kernel version: 5.15.0-117-generic