V8.7: Logback collects fewer logs than local files

[xiaogangfighting]

Spirngboot + logback integrates the skywalking log system. It is found that the number of logs collected by skywalking does not match the number of local logs, skywalking records much less, and oapserver / agent does not report error information。
thanks & best regards.

[wu-sheng]

I think this description is too abstract for us to response. It is better for you to use a simple codebase and verify which logs are missed.
We don't filter logs in default actually.

[xiaogangfighting]

The log is much less than it is

agent.conf:
plugin.toolkit.log.grpc.reporter.server_host=${SW_GRPC_LOG_SERVER_HOST:10.192.10.250} plugin.toolkit.log.grpc.reporter.server_port=${SW_GRPC_LOG_SERVER_PORT:51800} plugin.toolkit.log.grpc.reporter.max_message_size=${SW_GRPC_LOG_MAX_MESSAGE_SIZE:1048576000} plugin.toolkit.log.grpc.reporter.upstream_timeout=${SW_GRPC_LOG_GRPC_UPSTREAM_TIMEOUT:30}

[wu-sheng]

Comparing details, the page number is meaningless. We are even going to delete the whole page number, due to it is not accurate at all, especially for ElasticSearch.

[JerryGitHubAccount]

hi, i have done a test below with skywalking oap 10.0.1 ,agent 9.3.0 .
when log is bigger ,i can not find logs in skywalking ui.
can you give me any suggestions? thanks !

[JerryGitHubAccount]

@xiaogangfighting @wu-sheng
I have done a test, with spring boot 3.3.2+ logback
agent.conf
plugin.toolkit.log.grpc.reporter.server_host=${SW_GRPC_LOG_SERVER_HOST:127.0.0.1} plugin.toolkit.log.grpc.reporter.server_port=${SW_GRPC_LOG_SERVER_PORT:11800} plugin.toolkit.log.grpc.reporter.max_message_size=${SW_GRPC_LOG_MAX_MESSAGE_SIZE:52428800} plugin.toolkit.log.grpc.reporter.upstream_timeout=${SW_GRPC_LOG_GRPC_UPSTREAM_TIMEOUT:30}
oap application.yaml

receiver-sharing-server:   
  default:    
    # 修改 maxMessageSize: ${SW_RECEIVER_GRPC_MAX_MESSAGE_SIZE:0}
    maxMessageSize: ${SW_RECEIVER_GRPC_MAX_MESSAGE_SIZE:52428800}
configuration:    
  grpc:    
    #修改 maxInboundMessageSize: ${SW_DCS_MAX_INBOUND_MESSAGE_SIZE:4194304}
    maxInboundMessageSize: ${SW_DCS_MAX_INBOUND_MESSAGE_SIZE:52428800}
core:       
  default:        
    # 修改 maxMessageSize: ${SW_CORE_GRPC_MAX_MESSAGE_SIZE:0}    
    maxMessageSize: ${SW_CORE_GRPC_MAX_MESSAGE_SIZE:52428800}    

java demo

package com.skywalking.skywalking.test.controller;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
@RestController
@Slf4j
@RequestMapping("log")
public class LogController {
    @GetMapping()
    public String log() {

        String filePath = "D:\\Works\\IdeaProjects\\skywalking-test\\doc\\package.txt";

        StringBuilder contentBuilder = new StringBuilder();

        try (BufferedReader br = new BufferedReader(new FileReader(filePath))) {
            String sCurrentLine;

            while ((sCurrentLine = br.readLine()) != null) {
                contentBuilder.append(sCurrentLine).append("\n");
            }

            log.info(contentBuilder.toString());

        } catch (IOException e) {
            log.error("IOException: %s%n", e);
        }
        log.info("simple log info");
        return "success";
    }

}

my txt file is 588kb.

"simple log info" is shown , but txt info is missing in skywalking.

my question is how to config so that i can see big log in skywalking ui.

thanks!

[wu-sheng]

I believe that is the limitation of the network, such as grpc.

Why do you output so huge logs? That is not a product setup. We don't put the system designed in that way.

[JerryGitHubAccount]

thanks for your reply.

first of all, it's really not a goot product setup.

thanks for your suggestion, i can analyse the limitation of the network, such as grpc.

godd luck!