fluentd/td-agent performance is degraded when worker number is 32

[jxin-sc]

What is a problem?

I have a fluend/td-agent running in Ubuntu22 VM and had below observation.

when VM CPU core is 8 and 16, the more is the number of td-agent workers, the better is the performance. But when VM CPU core is 32, 32 worker td-agent performs worse, actually, it performs best when worker is 20~24.

Is it expected? Thanks

Describe the configuration of Fluentd

<match session.**>
  @type copy
  @include /etc/td-agent/stellar_ext_server.conf
  <store>
    @type           bufferize_batch
    batch_size      100
    enable_resend   true
    <buffer>
       @type file
       path  /var/log/td-agent/session
       flush_thread_count 4
       flush_interval 4s
       total_limit_size 4294967296
       retry_max_interval 120
       disable_chunk_backup true
    </buffer>
    <config>
      type          http_aella
      endpoint_url  http://0.0.0.0:8888/aelladata/json
      http_method   post
      serializer    json
      compress_request identity
      #authentication  basic
      #username    aella
      #ssl_no_verify true
    </config>
  </store>
</match>

Describe the logs of Fluentd

<12>2019-12-11 01:45:04,669 sentinel - CEF:2|SentinelOne|Mgmt|Windows 10 Enterprise|rt=2019-12-11 01:44:59.503474|fileHash=cfc20af2eb778ebde876ff58f0e33b150f0e6406|filePath=\Device\HarddiskVolume4\Users\189518\Desktop\c-okiba\inj_org.exe|fileName=inj_org.exe|deviceAddress=52.194.224.241|deviceHostFqdn=apne1-1101-NFR.sentinelone.net|deviceHostName=apne1-1101-NFR.sentinelone.net|notificationScope=SITE|siteId=677161634913562187|siteName=Default site|accountId=677161634896784970|accountName=Tokyo Electron Device LTD.|vendor=SentinelOne|eventID=4003|eventDesc=New Suspicious threat detected - machine TED33437|eventSeverity=1|originatorName=TED33437|originatorVersion=3.4.3.48|sourceAgentLastActivityTimestamp=2019-12-11 01:44:59.495407|sourceAgentRegisterTimestamp=2019-10-09 04:33:55.257070|sourceNetworkState=connected|sourceOsRevision=16299|sourceOsType=windows|sourceAgentUuid=d18d81d3e8554331a398e3fbc9b5afee|sourceFqdn=TED33437.TELDEVICE|sourceThreatCount=31|sourceMgmtPrecievedAddress=211.125.53.165|sourceDnsDomain=TELDEVICE|sourceHostName=TED33437|sourceUserName=189518|sourceUserId=S-1-5-21-2460301981-1420676974-2449531052-17768|sourceAgentId=732886133197865592|sourceGroupId=729186307255830466|sourceGroupName=KazyLab|sourceIpAddresses=['192.168.6.200', 'fe80::6cbc:c973:7b8b:e1c2', '172.29.64.97', 'fe80::dcb:f4dd:9f78:6580', '169.254.173.26', 'fe80::c1bf:fb34:5565:ad1a']|sourceMacAddresses=['a8:13:74:95:8b:26', '90:61:ae:aa:67:7a', '02:00:4c:4f:4f:50']|threatClassification=Malware|threatClassificationSource=Static|threatDetectingEngine=windows.preExecutionSuspicious|threatClassifier=LOGIC|threatMitigationStatusLabel=suspicious|threatMitigationStatusID=3|threatCommandLineArguments=|threatID=778461979376499126|cat=MALWARE|activityID=778461979460385208|activityType=4003

Environment

- Fluentd version:
- TD Agent version:
1.1.1-0
- Fluent Package version:
- Docker image (tag):
- Operating system:
VERSION="22.04.4 LTS (Jammy Jellyfish)"
- Kernel version:
5.15.0-101-generic