Persistent logging in Skywalking via fluent-bit

[azibhassan]

Hi,

I am trying to implement persistent logging in Skywalking instance for k8 services which are running in Istio Mesh Environment. I wanted to ask few questions

• Is it possible to do persistent logging without setting up Kafka fetcher
• What would be the right format to send application logs. Looking at following example, its sending trace information
• What will be the destination on UI? Can application logs appear under Log Tab

Currently, I have fluent-bit instance setup with following configuration:

Lua script:

  fluent-bit-script.lua: |
    function skywalking_transform(tag, timestamp, record)
      local entry = {}
      local instance_name = record["labels"]["app.kubernetes.io/instance"]
      local namespace_name = record["namespace_name"]
      entry.service = instance_name .. "." .. namespace_name
      entry.serviceInstance = record["pod_name"]
      entry.layer = "GENERAL"
      entry.endpoint = record["labels"]["app.kubernetes.io/instance"]
      entry.body = {
        json = {
            time = timestamp,
            content = record["log"]
            }
      }
      return 1, timestamp, entry
    end

  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker,cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines off

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Kube_Tag_Prefix kube.var.log.containers.
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser Off
        K8S-Logging.Exclude Off
    [FILTER]
        Name grep
        Match kube.*
        Exclude $kubernetes['container_name'] ^istio-proxy$
    [FILTER]
        Name grep
        Match kube.*
        Regex $kubernetes['namespace_name'] ^galileo-uat$
    [FILTER]
        Name nest
        Match kube.*
        Operation lift
        Nested_under kubernetes
    [FILTER]
        Name modify
        Match kube.*
        Remove container_image
        Remove container_hash
        Remove docker_id
        Remove annotations
        Remove pod_id
        Remove host
        Remove time
        Remove _p
    [FILTER]
        Name lua
        Match kube.*
        Script /fluent-bit/scripts/fluent-bit-script.lua
        Call skywalking_transform
#################################################################
  outputs: | 
   [OUTPUT]
        Name http
        Match kube.*
        Host skywalking-oap.aks-istio-system
        Port 12800
        uri /v3/logs
        Format json
        Json_date_key time
        Json_date_format iso8601

Sample of output looks like:

{
	"date":1717117744.477046,
	"Tags":
	{
		"data":[{
			"key":"level","value":"INFO"
		}]},
		"service":"analytics.galileo-uat",
		"serviceInstance":"analytics-6bff67f5f7-vjr7r",
		"time":1717117744000,
		"layer":"GENERAL",
		"body":
		{
			"json":
			{
				"json":"2024-05-31T01:09:04.476910521Z stdout F [01:09:04 INF] notifying calculated value for H2 with computationType: null"
			}
		},
		"endpoint":"analytics"
}
[2024/05/30 03:51:50] [ info] [output:http:http.1] skywalking-oap.aks-istio-system:12800, HTTP status=200

I update my fluentbit configuration to send data to Skywalking OAP HTTP Port. According to Fluentbit logs, Logs are being sent but on OAP Server, I am not seeing any activity. OAP Server does not even throw any error. I have attached the logs output side by side in above picture.

Does the output format looks okay? Are these logs being rejected? There is no indication. Can I check via Database (PSQL) ?

Any suggestion would be helpful. If there is any existing example which can help, please do share

[wu-sheng]

You could follow their document, https://docs.fluentbit.io/manual/pipeline/outputs/skywalking.

SkyWalking protocol is here, https://github.com/apache/skywalking-data-collect-protocol/blob/743f33119dc5621ae98b596eb8b131dd443445c7/logging/Logging.proto. Fluent bit supported to send data via their HTTP endpoint.

[azibhassan]

Hi @wu-sheng, I updated the discussion with more details. As of now, Fluent bit is sending logs to OAP Server (HTTP PORT) but I don't see any activity of OAP Server side. Could you please take a look at the question again, I added some more details

[wu-sheng]

What will be the destination on UI? Can application logs appear under Log Tab

You need to determine the service name and layer. When name matched and layer=general, it would show up at the General Dashabord Log Tabs.

[azibhassan]

Thanks Wu-sheng. I will try your suggestion on my end. Please keep this discussion open if possible.