Fluent-bit hangs on large lines while using in_tail and has a large CPU usage

[AzureLeMoon]

Bug Report

Describe the bug

the bug seems to be with the in_tail plugin, when a large line is written to the file being ingested by fluent-bit it stops working or sending logs on any input whether in_tail or syslog or ... and i see very high ( maximum allowed basically) cpu usage
i have read the documentation and have tried any option related to memory usage, buffer , filesystem storage and so on but none of them seem to be fixing the issue.

what does fix the issue is removing all the db files (including -wal and -shm) related to this input and then restarting fluent-bit

docker stats :
e13912de3894 fluent-bit 100.29% 12.39MiB / 20MiB 61.95% 0B / 0B 455kB / 586kB 18

when using strace on the fluent-bit process here is the output:

root@e10c17f0-43fa-480c-9017-8183e666d6a0:~# strace -p 204478
strace: Process 204478 attached
restart_syscall(<... resuming interrupted read ...>) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffd89366ab0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffd89366ab0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffd89366ab0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffd89366ab0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffd89366ab0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffd89366ab0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffd89366ab0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffd89366ab0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffd89366ab0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffd89366ab0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffd89366ab0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffd89366ab0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, ^Cstrace: Process 204478 detached
 <detached ...>

i am running fluent-bit using docker and gathering logs from a log file using the following configurations

my inputs

pipeline:
  inputs:
    - name: syslog
      tag: syslog
      listen: 127.0.0.1
      parser: syslog-rfc3164
      storage.type: filesystem
      port: 5140
      mode: tcp 

    - name: tail
      tag: guest.agent
      path: /var/log/guest-agent/guest-agent-*.log
      DB: /fluent-bit/etc/db-files/guest-agent.log.db
      DB.locking: true
      DB.journal_mode: WAL
      Skip_Long_Lines: On
      Threaded: true
      buffer_chunk_size: 128k
      Buffer_Max_Size: 512k
      Mem_Buf_Limit: 3M
      Parser: guest_agent 

my output

pipeline:
  outputs:
    - name: forward
      match: '*'
      upstream: configs/upstream.conf

my upstream

[UPSTREAM]
    name       forward-balancing

[NODE]
    name       node-1
    host       fluentd-01.example.com
    port       24224
    Shared_Key    ${PASSWORD}
    Self_Hostname flb.local
    tls        on
    tls.verify off

[NODE]
    name       node-2
    host      fluentd-02.example.com
    port       24224
    Shared_Key    ${PASSWORD}
    Self_Hostname flb.local
    tls        on
    tls.verify off

[NODE]
    name       node-3
    host       fluentd-03.example.com
    port       24224
    Shared_Key    ${PASSWORD}
    Self_Hostname flb.local
    tls        on
    tls.verify off

my service

service:
  flush: 1
  parsers_file: configs/parsers-dbaas.conf
  parsers_file: parsers.conf
  parsers_file: parsers_extra.conf
  log_level: info
  storage.path: /fluent-bit/etc/db-files/
  storage.sync:              normal
  storage.checksum:          off
  storage.backlog.mem_limit: 5M
  storage.max_chunks_up: 2
  storage.metrics: On
  storage.delete_irrecoverable_chunks: On
  HTTP_Server:  On
  HTTP_Listen:  127.0.0.1
  HTTP_PORT:    2020

and the parser

[PARSER]
    Name guest_agent
    Format regex
    Regex ^\{"time":"(?<timestamp>[^"]+)",\s*"level":"(?<loglevel>[^"]+)",\s*"prefix":"(?<prefix>[^"]+)",\s*"file":"(?<file>[^"]+)",\s*"line":"(?<line>[^"]+)",\s*"message":"(?<message>(?:[^"{}]+|\"[^\"]*\"|\{[^\}]*\})*?)"\}$
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep Off
    Time_Key timestamp

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:

{"time":"2025-02-01T01:39:38.617749164Z","level":"-","prefix":"-","file":"upload_object.go","line":"59","message":"Couldn't upload file /opt/dbaas/mysql/binlog.005887 to dbaas-database-backups:/opt/dbaas/mysql/binlog.005887. Here's why: SerializationError: failed to unmarshal error message\n\tstatus code: 412, request id: , host id: \ncaused by: UnmarshalError: failed to unmarshal error message\n\t00000000  3c 21 44 4f 43 54 59 50  45 20 68 74 6d 6c 3e 3c  |<!DOCTYPE html><|\n00000010  68 74 6d 6c 20 6c 61 6e  67 3d 22 65 6e 22 3e 3c  |html lang=\"en\"><|\n00000020  68 65 61 64 3e 3c 6d 65  74 61 20 63 68 61 72 73  |head><meta chars|\n00000030  65 74 3d 22 55 54 46 2d  38 22 3e 3c 6d 65 74 61  |et=\"UTF-8\"><meta|\n00000040  20 6e 61 6d 65 3d 22 76  69 65 77 70 6f 72 74 22  | name=\"viewport\"|\n00000050  20 63 6f 6e 74 65 6e 74  3d 22 77 69 64 74 68 3d  | content=\"width=|\n00000060  64 65 76 69 63 65 2d 77  69 64 74 68 22 3e 3c 6c  |device-width\"><l|\n00000070  69 6e 6b 20 68 72 65 66  3d 22 68 74 74 70 73 3a  |ink href=\"https:|\n00000080  2f 2f 77 77 77 2e 61 72  76 61 6e 63 6c 6f 75 64  |//www.arvancloud|\n00000090  2e 69 72 2f 63 64 6e 2d  63 67 69 2f 61 73 73 65  |.ir/cdn-cgi/asse|\n000000a0  74 73 2f 63 73 73 2f 61  72 2d 32 2e 36 2e 30 2e  |ts/css/ar-2.6.0.|\n000000b0  63 73 73 22 20 72 65 6c  3d 22 73 74 79 6c 65 73  |css\" rel=\"styles|\n000000c0  68 65 65 74 22 3e 3c 6c  69 6e 6b 20 72 65 6c 3d  |heet\"><link rel=|\n000000d0  22 69 63 6f 6e 22 20 74  79 70 65 3d 22 69 6d 61  |\"icon\" type=\"ima|\n000000e0  67 65 2f 78 2d 69 63 6f  6e 22 20 68 72 65 66 3d  |ge/x-icon\" href=|\n000000f0  22 2f 2f 64 62 61 61 73  2d 64 61 74 61 62 61 73  |\"//dbaas-databas|\n00000100  65 2d 62 61 63 6b 75 70  73 2e 73 33 2e 69 72 2d  |e-backups.s3.ir-|\n00000110  74 68 72 2d 61 74 31 2e  61 72 76 61 6e 73 74 6f  |thr-at1.arvansto|\n00000120  72 61 67 65 2e 69 72 2f  66 61 76 69 63 6f 6e 2e  |rage.ir/favicon.|\n00000130  69 63 6f 22 3e 3c 74 69  74 6c 65 3e 3c 2f 74 69  |ico\"><title></ti|\n00000140  74 6c 65 3e 3c 2f 68 65  61 64 3e 3c 62 6f 64 79  |tle></head><body|\n00000150  3e 3c 73 65 63 74 69 6f  6e 20 63 6c 61 73 73 3d  |><section class=|\n00000160  22 65 72 72 6f 72 2d 73  65 63 74 69 6f 6e 20 65  |\"error-section e|\n00000170  72 72 6f 72 2d 73 65 63  74 69 6f 6e 2d 2d 6c 74  |rror-section--lt|\n00000180  72 20 65 72 72 6f 72 2d  73 65 63 74 69 6f 6e 2d  |r error-section-|\n00000190  2d 68 69 64 65 22 20 69  64 3d 22 65 6e 22 3e 3c  |-hide\" id=\"en\"><|\n000001a0  64 69 76 20 63 6c 61 73  73 3d 22 63 6f 6e 74 61  |div class=\"conta|\n000001b0  69 6e 65 72 22 3e 3c 64  69 76 20 63 6c 61 73 73  |iner\"><div class|\n000001c0  3d 22 65 72 72 6f 72 2d  73 65 63 74 69 6f 6e 5f  |=\"error-section_|\n000001d0  5f 77 72 61 70 70 65 72  22 3e 3c 64 69 76 20 63  |_wrapper\"><div c|\n000001e0  6c 61 73 73 3d 22 65 72  72 6f 72 2d 73 65 63 74  |lass=\"error-sect|\n000001f0  69 6f 6e 5f 5f 69 6e 66  6f 72 6d 61 74 69 6f 6e  |ion__information|\n00000200  22 3e 3c 68 31 20 63 6c  61 73 73 3d 22 65 72 72  |\"><h1 class=\"err|\n00000210  6f 72 2d 73 65 63 74 69  6f 6e 5f 5f 74 69 74 6c  |or-section__titl|\n00000220  65 22 3e 3c 73 70 61 6e  20 63 6c 61 73 73 3d 22  |e\"><span class=\"|\n00000230  65 72 72 6f 72 2d 73 65  63 74 69 6f 6e 5f 5f 73  |error-section__s|\n00000240  74 61 74 75 73 22 3e 45  72 72 6f 72 20 34 31 32  |tatus\">Error 412|\n00000250  3c 2f 73 70 61 6e 3e 3c  73 70 61 6e 20 63 6c 61  |</span><span cla|\n00000260  73 73 3d 22 65 72 72 6f  72 2d 73 65 63 74 69 6f  |ss=\"error-sectio|\n00000270  6e 5f 5f 73 70 6c 69 74  74 65 72 22 3e 20 26 6e  |n__splitter\"> &n|\n00000280  62 73 70 3b 7c 26 6e 62  73 70 3b 3c 2f 73 70 61  |bsp;|&nbsp;</spa|\n00000290  6e 3e 3c 73 70 61 6e 20  63 6c 61 73 73 3d 22 65  |n><span class=\"e|\n000002a0  72 72 6f 72 2d 73 65 63  74 69 6f 6e 5f 5f 72 65  |rror-section__re|\n000002b0  61 73 6f 6e 22 3e 50 72  65 63 6f 6e 64 69 74 69  |ason\">Preconditi|\n000002c0  6f 6e 20 46 61 69 6c 65  64 3c 2f 73 70 61 6e 3e  |on Failed</span>|\n000002d0  3c 2f 68 31 3e 3c 68 32  20 63 6c 61 73 73 3d 22  |</h1><h2 class=\"|\n000002e0  65 72 72 6f 72 2d 73 65  63 74 69 6f 6e 5f 5f 73  |error-section__s|\n000002f0  75 62 74 69 74 6c 65 22  3e 49 6e 76 61 6c 69 64  |ubtitle\">Invalid|\n00000300  20 55 52 4c 20 6f 72 20  74 68 65 20 77 65 62 73  | URL or the webs|\n00000310  69 74 65 20 69 73 20 6e  6f 74 20 70 72 6f 70 65  |ite is not prope|\n00000320  72 6c 79 20 63 6f 6e 66  69 67 75 72 65 64 21 3c  |rly configured!<|\n00000330  2f 68 32 3e 3c 64 69 76  20 63 6c 61 73 73 3d 22  |/h2><div class=\"|\n00000340  65 72 72 6f 72 2d 73 65  63 74 69 6f 6e 5f 5f 63  |error-section__c|\n00000350  6f 6e 74 65 6e 74 73 22  3e 3c 70 20 63 6c 61 73  |ontents\"><p clas|\n00000360  73 3d 22 65 72 72 6f 72  2d 73 65 63 74 69 6f 6e  |s=\"error-section|\n00000370  5f 5f 63 6f 6e 74 65 6e  74 22 3e 52 65 76 69 65  |__content\">Revie|\n00000380  77 20 74 68 65 20 55 52  4c 20 6f 72 20 63 6f 6e  |w the URL or con|\n00000390  74 61 63 74 20 3c 62 3e  64 62 61 61 73 2d 64 61  |tact <b>dbaas-da|\n000003a0  74 61 62 61 73 65 2d 62  61 63 6b 75 70 73 2e 73  |tabase-backups.s|\n000003b0  33 2e 69 72 2d 74 68 72  2d 61 74 31 2e 61 72 76  |3.ir-thr-at1.arv|\n000003c0  61 6e 73 74 6f 72 61 67  65 2e 69 72 3c 2f 62 3e  |anstorage.ir</b>|\n000003d0  20 73 75 70 70 6f 72 74  2e 3c 2f 70 3e 3c 2f 64  | support.</p></d|\n000003e0  69 76 3e 3c 2f 64 69 76  3e 3c 64 69 76 20 63 6c  |iv></div><div cl|\n000003f0  61 73 73 3d 22 65 72 72  6f 72 2d 73 65 63 74 69  |ass=\"error-secti|\n00000400  6f 6e 5f 5f 69 6d 61 67  65 22 3e 3c 73 76 67 20  |on__image\"><svg |\n00000410  77 69 64 74 68 3d 22 32  35 38 22 20 68 65 69 67  |width=\"258\" heig|\n00000420  68 74 3d 22 33 30 39 22  20 76 69 65 77 42 6f 78  |ht=\"309\" viewBox|\n00000430  3d 22 30 20 30 20 32 35  38 20 33 30 39 22 20 66  |=\"0 0 258 309\" f|\n00000440  69 6c 6c 3d 22 6e 6f 6e  65 22 20 78 6d 6c 6e 73  |ill=\"none\" xmlns|\n00000450  3d 22 68 74 74 70 3a 2f  2f 77 77 77 2e 77 33 2e  |=\"http://www.w3.|\n00000460  6f 72 67 2f 32 30 30 30  2f 73 76 67 22 3e 0a 3c  |org/2000/svg\">.<|\n00000470  72 65 63 74 20 78 3d 22  36 39 22 20 79 3d 22 33  |rect x=\"69\" y=\"3|\n00000480  37 2e 36 30 37 34 22 20  77 69 64 74 68 3d 22 31  |7.6074\" width=\"1|\n00000490  33 32 22 20 68 65 69 67  68 74 3d 22 31 37 35 22  |32\" height=\"175\"|\n000004a0  20 72 78 3d 22 36 22 20  66 69 6c 6c 3d 22 23 44  | rx=\"6\" fill=\"#D|\n000004b0  31 44 42 45 38 22 2f 3e  0a 3c 72 65 63 74 20 78  |1DBE8\"/>.<rect x|\n000004c0  3d 22 32 30 22 20 79 3d  22 37 35 2e 36 30 37 34  |=\"20\" y=\"75.6074|\n000004d0  22 20 77 69 64 74 68 3d  22 31 33 30 22 20 68 65  |\" width=\"130\" he|\n000004e0  69 67 68 74 3d 22 31 37  33 22 20 72 78 3d 22 35  |ight=\"173\" rx=\"5|\n000004f0  22 20 73 74 72 6f 6b 65  3d 22 23 42 32 43 32 44  |\" stroke=\"#B2C2D|\n00000500  36 22 20 73 74 72 6f 6b  65 2d 77 69 64 74 68 3d  |6\" stroke-width=|\n00000510  22 32 22 20 73 74 72 6f  6b 65 2d 64 61 73 68 61  |\"2\" stroke-dasha|\n00000520  72 72 61 79 3d 22 36 20  36 22 2f 3e 0a 3c 72 65  |rray=\"6 6\"/>.<re|\n00000530  63 74 20 78 3d 22 33 37  22 20 79 3d 22 35 37 2e  |ct x=\"37\" y=\"57.|\n00000540  36 30 37 34 22 20 77 69  64 74 68 3d 22 31 33 39  |6074\" width=\"139|\n00000550  22 20 68 65 69 67 68 74  3d 22 31 37 35 22 20 72  |\" height=\"175\" r|\n00000560  78 3d 22 36 22 20 66 69  6c 6c 3d 22 23 45 39 45  |x=\"6\" fill=\"#E9E|\n00000570  44 46 35 22 2f 3e 0a 3c  67 20 6f 70 61 63 69 74  |DF5\"/>.<g opacit|\n00000580  79 3d 22 30 2e 39 22 3e  0a 3c 6c 69 6e 65 20 78  |y=\"0.9\">.<line x|\n00000590  31 3d 22 36 35 2e 30 33  33 36 22 20 79 31 3d 22  |1=\"65.0336\" y1=\"|\n000005a0  37 33 2e 35 33 30 32 22  20 78 32 3d 22 35 32 2e  |73.5302\" x2=\"52.|\n000005b0  36 37 38 35 22 20 79 32  3d 22 38 35 2e 33 37 35  |6785\" y2=\"85.375|\n000005c0  22 20 73 74 72 6f 6b 65  3d 22 23 39 31 41 34 42  |\" stroke=\"#91A4B|\n000005d0  43 22 20 73 74 72 6f 6b  65 2d 77 69 64 74 68 3d  |C\" stroke-width=|\n000005e0  22 35 22 2f 3e 0a 3c 6c  69 6e 65 20 78 31 3d 22  |\"5\"/>.<line x1=\"|\n000005f0  36 35 2e 32 37 36 39 22  20 79 31 3d 22 38 35 2e  |65.2769\" y1=\"85.|\n00000600  36 34 31 32 22 20 78 32  3d 22 35 33 2e 34 33 32  |6412\" x2=\"53.432|\n00000610  31 22 20 79 32 3d 22 37  33 2e 32 38 36 31 22 20  |1\" y2=\"73.2861\" |\n00000620  73 74 72 6f 6b 65 3d 22  23 39 31 41 34 42 43 22  |stroke=\"#91A4BC\"|\n00000630  20 73 74 72 6f 6b 65 2d  77 69 64 74 68 3d 22 35  | stroke-width=\"5|\n00000640  22 2f 3e 0a 3c 2f 67 3e  0a 3c 67 20 6f 70 61 63  |\"/>.</g>.<g opac|\n00000650  69 74 79 3d 22 30 2e 39  22 3e 0a 3c 70 61 74 68  |ity=\"0.9\">.<path|\n00000660  20 64 3d 22 4d 39 33 2e  35 20 31 33 38 2e 36 30  | d=\"M93.5 138.60|\n00000670  37 43 39 33 2e 35 20 31  33 38 2e 36 30 37 20 39  |7C93.5 138.607 9|\n00000680  34 20 31 32 35 2e 31 30  37 20 31 30 36 2e 35 20  |4 125.107 106.5 |\n00000690  31 32 35 2e 31 30 37 43  31 31 39 20 31 32 35 2e  |125.107C119 125.|\n000006a0  31 30 37 20 31 31 39 2e  35 20 31 33 38 2e 36 30  |107 119.5 138.60|\n000006b0  37 20 31 31 39 2e 35 20  31 33 38 2e 36 30 37 22  |7 119.5 138.607\"|\n000006c0  20 73 74 72 6f 6b 65 3d  22 23 39 31 41 34 42 43  | stroke=\"#91A4BC|\n000006d0  22 20 73 74 72 6f 6b 65  2d 77 69 64 74 68 3d 22  |\" stroke-width=\"|\n000006e0  32 2e 35 22 2f 3e 0a 3c  63 69 72 63 6c 65 20 63  |2.5\"/>.<circle c|\n000006f0  78 3d 22 31 33 31 2e 35  22 20 63 79 3d 22 31 31  |x=\"131.5\" cy=\"11|\n00000700  37 2e 31 30 37 22 20 72  3d 22 34 2e 35 22 20 66  |7.107\" r=\"4.5\" f|\n00000710  69 6c 6c 3d 22 23 39 31  41 34 42 43 22 2f 3e 0a  |ill=\"#91A4BC\"/>.|\n00000720  3c 63 69 72 63 6c 65 20  63 78 3d 22 38 32 2e 35  |<circle cx=\"82.5|\n00000730  22 20 63 79 3d 22 31 31  37 2e 31 30 37 22 20 72  |\" cy=\"117.107\" r|\n00000740  3d 22 34 2e 35 22 20 66  69 6c 6c 3d 22 23 39 31  |=\"4.5\" fill=\"#91|\n00000750  41 34 42 43 22 2f 3e 0a  3c 2f 67 3e 0a 3c 6c 69  |A4BC\"/>.</g>.<li|\n00000760  6e 65 20 78 31 3d 22 35  36 2e 35 22 20 79 31 3d  |ne x1=\"56.5\" y1=|\n00000770  22 31 35 39 2e 31 30 37  22 20 78 32 3d 22 31 35  |\"159.107\" x2=\"15|\n00000780  36 2e 35 22 20 79 32 3d  22 31 35 39 2e 31 30 37  |6.5\" y2=\"159.107|\n00000790  22 20 73 74 72 6f 6b 65  3d 22 23 42 32 43 32 44  |\" stroke=\"#B2C2D|\n000007a0  36 22 20 73 74 72 6f 6b  65 2d 77 69 64 74 68 3d  |6\" stroke-width=|\n000007b0  22 35 22 20 73 74 72 6f  6b 65 2d 6c 69 6e 65 63  |\"5\" stroke-linec|\n000007c0  61 70 3d 22 72 6f 75 6e  64 22 2f 3e 0a 3c 6c 69  |ap=\"round\"/>.<li|\n000007d0  6e 65 20 78 31 3d 22 35  36 2e 35 22 20 79 31 3d  |ne x1=\"56.5\" y1=|\n000007e0  22 31 37 33 2e 31 30 37  22 20 78 32 3d 22 31 35  |\"173.107\" x2=\"15|\n000007f0  36 2e 35 22 20 79 32 3d  22 31 37 33 2e 31 30 37  |6.5\" y2=\"173.107|\n00000800  22 20 73 74 72 6f 6b 65  3d 22 23 42 32 43 32 44  |\" stroke=\"#B2C2D|\n00000810  36 22 20 73 74 72 6f 6b  65 2d 77 69 64 74 68 3d  |6\" stroke-width=|\n00000820  22 35 22 20 73 74 72 6f  6b 65 2d 6c 69 6e 65 63  |\"5\" stroke-linec|\n00000830  61 70 3d 22 72 6f 75 6e  64 22 2f 3e 0a 3c 6c 69  |ap=\"round\"/>.<li|\n00000840  6e 65 20 78 31 3d 22 35  36 2e 35 22 20 79 31 3d  |ne x1=\"56.5\" y1=|\n00000850  22 32 30 31 2e 31 30 37  22 20 78 32 3d 22 31 35  |\"201.107\" x2=\"15|\n00000860  36 2e 35 22 20 79 32 3d  22 32 30 31 2e 31 30 37  |6.5\" y2=\"201.107|\n00000870  22 20 73 74 72 6f 6b 65  3d 22 23 42 32 43 32 44  |\" stroke=\"#B2C2D|\n00000880  36 22 20 73 74 72 6f 6b  65 2d 77 69 64 74 68 3d  |6\" stroke-width=|\n00000890  22 35 22 20 73 74 72 6f  6b 65 2d 6c 69 6e 65 63  |\"5\" stroke-linec|\n000008a0  61 70 3d 22 72 6f 75 6e  64 22 2f 3e 0a 3c 6c 69  |ap=\"round\"/>.<li|\n000008b0  6e 65 20 78 31 3d 22 35  36 2e 35 22 20 79 31 3d  |ne x1=\"56.5\" y1=|\n000008c0  22 31 38 37 2e 31 30 37  22 20 78 32 3d 22 31 35  |\"187.107\" x2=\"15|\n000008d0  36 2e 35 22 20 79 32 3d  22 31 38 37 2e 31 30 37  |6.5\" y2=\"187.107|\n000008e0  22 20 73 74 72 6f 6b 65  3d 22 23 42 32 43 32 44  |\" stroke=\"#B2C2D|\n000008f0  36 22 20 73 74 72 6f 6b  65 2d 77 69 64 74 68 3d  |6\" stroke-width=|\n00000900  22 35 22 20 73 74 72 6f  6b 65 2d 6c 69 6e 65 63  |\"5\" stroke-linec|\n00000910  61 70 3d 22 72 6f 75 6e  64 22 2f 3e 0a 3c 6c 69  |ap=\"round\"/>.<li|\n00000920  6e 65 20 78 31 3d 22 35  36 2e 35 22 20 79 31 3d  |ne x1=\"56.5\" y1=|\n00000930  22 32 31 35 2e 31 30 37  22 20 78 32 3d 22 31 35  |\"215.107\" x2=\"15|\n00000940  36 2e 35 22 20 79 32 3d  22 32 31 35 2e 31 30 37  |6.5\" y2=\"215.107|\n00000950  22 20 73 74 72 6f 6b 65  3d 22 23 42 32 43 32 44  |\" stroke=\"#B2C2D|\n00000960  36 22 20 73 74 72 6f 6b  65 2d 77 69 64 74 68 3d  |6\" stroke-width=|\n00000970  22 35 22 20 73 74 72 6f  6b 65 2d 6c 69 6e 65 63  |\"5\" stroke-linec|\n00000980  61 70 3d 22 72 6f 75 6e  64 22 2f 3e 0a 3c 70 61  |ap=\"round\"/>.<pa|\n00000990  74 68 20 66 69 6c 6c 2d  72 75 6c 65 3d 22 65 76  |th fill-rule=\"ev|\n000009a0  65 6e 6f 64 64 22 20 63  6c 69 70 2d 72 75 6c 65  |enodd\" clip-rule|\n000009b0  3d 22 65 76 65 6e 6f 64  64 22 20 64 3d 22 4d 31  |=\"evenodd\" d=\"M1|\n000009c0  37 37 2e 33 36 36 20 31  39 32 43 31 37 37 2e 30  |77.366 192C177.0|\n000009d0  31 35 20 31 39 33 2e 34  38 32 20 31 37 36 2e 35  |15 193.482 176.5|\n000009e0  38 20 31 39 34 2e 37 36  37 20 31 37 36 2e 30 34  |8 194.767 176.04|\n000009f0  39 20 31 39 35 2e 35 35  33 43 31 37 35 2e 33 35  |9 195.553C175.35|\n00000a00  36 20 31 39 36 2e 35 37  39 20 31 37 33 2e 36 34  |6 196.579 173.64|\n00000a10  33 20 31 39 38 2e 33 38  36 20 31 37 31 2e 36 37  |3 198.386 171.67|\n00000a20  38 20 32 30 30 2e 34 35  39 43 31 36 37 2e 39 37  |8 200.459C167.97|\n00000a30  33 20 32 30 34 2e 33 36  38 20 31 36 33 2e 33 37  |3 204.368 163.37|\n00000a40  36 20 32 30 39 2e 32 31  38 20 31 36 33 2e 30 34  |6 209.218 163.04|\n00000a50  39 20 32 31 31 2e 35 34  43 31 36 32 2e 37 33 38  |9 211.54C162.738|\n00000a60  20 32 31 33 2e 37 35 33  20 31 36 31 2e 32 36 32  | 213.753 161.262|\n00000a70  20 32 31 38 2e 37 32 34  20 31 36 30 2e 30 37 33  | 218.724 160.073|\n00000a80  20 32 32 32 2e 37 33 43  31 35 39 2e 33 35 33 20  | 222.73C159.353 |\n00000a90  32 32 35 2e 31 35 34 20  31 35 38 2e 37 33 38 20  |225.154 158.738 |\n00000aa0  32 32 37 2e 32 32 36 20  31 35 38 2e 35 34 39 20  |227.226 158.549 |\n00000ab0  32 32 38 2e 31 31 39 43  31 35 38 2e 33 38 33 20  |228.119C158.383 |\n00000ac0  32 32 38 2e 39 30 38 20  31 35 38 2e 33 38 33 20  |228.908 158.383 |\n00000ad0  32 32 39 2e 33 30 33 20  31 35 38 2e 33 38 33 20  |229.303 158.383 |\n00000ae0  32 32 39 2e 36 31 43 31  35 38 2e 33 38 33 20 32  |229.61C158.383 2|\n00000af0  33 30 2e 32 32 34 20 31  35 38 2e 33 38 33 20 32  |30.224 158.383 2|\n00000b00  33 30 2e 34 38 37 20 31  35 37 2e 30 34 39 20 32  |30.487 157.049 2|\n00000b10  33 32 2e 38 35 35 43 31  35 36 2e 33 34 39 20 32  |32.855C156.349 2|\n00000b20  33 34 2e 30 39 39 20 31  35 36 2e 32 30 31 20 32  |34.099 156.201 2|\n00000b30  33 35 2e 33 34 33 20 31  35 36 2e 30 36 37 20 32  |35.343 156.067 2|\n00000b40  33 36 2e 34 36 43 31 35  36 2e 30 34 35 20 32 33  |36.46C156.045 23|\n00000b50  36 2e 36 34 33 20 31 35  36 2e 30 32 34 20 32 33  |6.643 156.024 23|\n00000b60  36 2e 38 32 34 20 31 35  36 20 32 33 37 48 31 37  |6.824 156 237H17|\n00000b70  37 2e 35 34 39 43 31 38  30 2e 38 36 33 20 32 33  |7.549C180.863 23|\n00000b80  37 20 31 38 33 2e 35 34  39 20 32 33 34 2e 33 31  |7 183.549 234.31|\n00000b90  34 20 31 38 33 2e 35 34  39 20 32 33 31 56 31 39  |4 183.549 231V19|\n00000ba0  32 48 31 37 37 2e 33 36  36 5a 22 20 66 69 6c 6c  |2H177.366Z\" fill|\n00000bb0  3d 22 23 46 34 46 41 46  46 22 2f 3e 0a 3c 63 69  |=\"#F4FAFF\"/>.<ci|\n00000bc0  72 63 6c 65 20 6f 70 61  63 69 74 79 3d 22 30 2e  |rcle opacity=\"0.|\n00000bd0  30 35 22 20 63 78 3d 22  31 38 37 22 20 63 79 3d  |05\" cx=\"187\" cy=|\n00000be0  22 32 32 36 22 20 72 3d  22 33 36 22 20 66 69 6c  |\"226\" r=\"36\" fil|\n00000bf0  6c 3d 22 23 30 35 30 35  30 35 22 2f 3e 0a 3c 70  |l=\"#050505\"/>.<p|\n00000c00  61 74 68 20 6f 70 61 63  69 74 79 3d 22 30 2e 35  |ath opacity=\"0.5|\n00000c10  22 20 66 69 6c 6c 2d 72  75 6c 65 3d 22 65 76 65  |\" fill-rule=\"eve|\n00000c20  6e 6f 64 64 22 20 63 6c  69 70 2d 72 75 6c 65 3d  |nodd\" clip-rule=|\n00000c30  22 65 76 65 6e 6f 64 64  22 20 64 3d 22 4d 32 31  |\"evenodd\" d=\"M21|\n00000c40  32 2e 38 36 20 32 31 35  2e 35 32 37 4c 31 37 32  |2.86 215.527L172|\n00000c50  2e 38 31 34 20 32 35 31  2e 34 33 43 31 37 31 2e  |.814 251.43C171.|\n00000c60  37 38 37 20 32 35 30 2e  35 37 20 31 37 30 2e 38  |787 250.57 170.8|\n00000c70  32 35 20 32 34 39 2e 36  33 31 20 31 36 39 2e 39  |25 249.631 169.9|\n00000c80  33 36 20 32 34 38 2e 36  31 38 4c 32 31 30 2e 32  |36 248.618L210.2|\n00000c90  35 34 20 32 31 32 2e 34  37 31 43 32 31 31 2e 31  |54 212.471C211.1|\n00000ca0  39 36 20 32 31 33 2e 34  31 36 20 32 31 32 2e 30  |96 213.416 212.0|\n00000cb0  36 38 20 32 31 34 2e 34  33 36 20 32 31 32 2e 38  |68 214.436 212.8|\n00000cc0  36 20 32 31 35 2e 35 32  37 5a 22 20 66 69 6c 6c  |6 215.527Z\" fill|\n00000cd0  3d 22 77 68 69 74 65 22  2f 3e 0a 3c 70 61 74 68  |=\"white\"/>.<path|\n00000ce0  20 6f 70 61 63 69 74 79  3d 22 30 2e 35 22 20 66  | opacity=\"0.5\" f|\n00000cf0  69 6c 6c 2d 72 75 6c 65  3d 22 65 76 65 6e 6f 64  |ill-rule=\"evenod|\n00000d00  64 22 20 63 6c 69 70 2d  72 75 6c 65 3d 22 65 76  |d\" clip-rule=\"ev|\n00000d10  65 6e 6f 64 64 22 20 64  3d 22 4d 32 31 36 2e 37  |enodd\" d=\"M216.7|\n00000d20  32 32 20 32 32 36 2e 31  31 34 43 32 31 36 2e 35  |22 226.114C216.5|\n00000d30  36 33 20 32 32 35 2e 34  30 37 20 32 31 36 2e 33  |63 225.407 216.3|\n00000d40  37 33 20 32 32 34 2e 37  30 32 20 32 31 36 2e 31  |73 224.702 216.1|\n00000d50  35 31 20 32 32 33 2e 39  39 38 43 32 31 35 2e 36  |51 223.998C215.6|\n00000d60  31 33 20 32 32 32 2e 32  38 37 20 32 31 34 2e 39  |13 222.287 214.9|\n00000d70  31 36 20 32 32 30 2e 36  36 39 20 32 31 34 2e 30  |16 220.669 214.0|\n00000d80  38 20 32 31 39 2e 31 35  37 4c 31 37 36 2e 34 36  |8 219.157L176.46|\n00000d90  31 20 32 35 33 2e 35 36  36 43 31 37 38 2e 35 32  |1 253.566C178.52|\n00000da0  34 20 32 35 34 2e 39 36  20 31 38 30 2e 37 37 38  |4 254.96 180.778|\n00000db0  20 32 35 36 2e 30 35 37  20 31 38 33 2e 31 35 31  | 256.057 183.151|\n00000dc0  20 32 35 36 2e 38 32 4c  32 31 36 2e 37 32 32 20  | 256.82L216.722 |\n00000dd0  32 32 36 2e 31 31 34 5a  22 20 66 69 6c 6c 3d 22  |226.114Z\" fill=\"|\n00000de0  77 68 69 74 65 22 2f 3e  0a 3c 70 61 74 68 20 64  |white\"/>.<path d|\n00000df0  3d 22 4d 32 30 38 2e 38  37 39 20 32 36 35 2e 30  |=\"M208.879 265.0|\n00000e00  34 39 4c 32 32 30 2e 31  39 33 20 32 35 35 2e 32  |49L220.193 255.2|\n00000e10  30 31 4c 32 35 33 2e 31  31 39 20 32 39 33 2e 30  |01L253.119 293.0|\n00000e20  33 43 32 35 35 2e 37 37  35 20 32 39 36 2e 30 38  |3C255.775 296.08|\n00000e30  31 20 32 35 35 2e 37 33  33 20 33 30 30 2e 34 36  |1 255.733 300.46|\n00000e40  36 20 32 35 33 2e 30 32  35 20 33 30 32 2e 38 32  |6 253.025 302.82|\n00000e50  33 4c 32 35 31 2e 35 31  37 20 33 30 34 2e 31 33  |3L251.517 304.13|\n00000e60  36 43 32 34 38 2e 38 30  39 20 33 30 36 2e 34 39  |6C248.809 306.49|\n00000e70  32 20 32 34 34 2e 34 36  20 33 30 35 2e 39 32 39  |2 244.46 305.929|\n00000e80  20 32 34 31 2e 38 30 34  20 33 30 32 2e 38 37 37  | 241.804 302.877|\n00000e90  4c 32 30 38 2e 38 37 39  20 32 36 35 2e 30 34 39  |L208.879 265.049|\n00000ea0  5a 22 20 66 69 6c 6c 3d  22 23 42 32 43 32 44 36  |Z\" fill=\"#B2C2D6|\n00000eb0  22 2f 3e 0a 3c 70 61 74  68 20 64 3d 22 4d 32 30  |\"/>.<path d=\"M20|\n00000ec0  32 2e 32 36 33 20 32 35  32 2e 38 37 38 4c 32 30  |2.263 252.878L20|\n00000ed0  39 2e 30 35 32 20 32 34  36 2e 39 36 39 4c 32 32  |9.052 246.969L22|\n00000ee0  30 2e 38 39 20 32 36 30  2e 35                    |0.89 260.5|\n\ncaused by: expected element type <Error> but have <html>\n"}

{"time":"2025-02-01T01:39:38.618926197Z","level":"ERROR","prefix":"-","file":"guest_start_incremental_backup.go","line":"42","message":"panic occurred: runtime error: slice bounds out of range [35:0]"}

• Steps to reproduce the problem:
  run fluent-bit using docker , with a 20mb memory limit and the given configurations
  then add the given log message or another large line similar to this one to the file being ingested using cat example.log >> /var/log/guest-agent/guest-agent-test.log

Expected behavior

given the fact that i have set Skip_Long_Lines to true i would expect the input to just skip this line and go on to the next line however the log output stops and fluent-bit starts having really high cpu usuage.

Screenshots

Your Environment

• Version used: 3.1.9
• Configuration: provided above
• Environment name and version (e.g. Kubernetes? What version?): docker engine v26.1.4
• Server type and version: OpenStack VM
• Operating System and version: ubuntu 20.04LTS
• Filters and plugins: forward plugin, tail input, tcp input

Additional context
although i understand that fixing the reason the log was generated and thus preventing such a long log message would fix the problem however it might happen again in some other context so i would like to troubleshoot the root of this issue

please do let me know if there is any additional information i can provide for this