newlog: add filter, dedup and counter functions #4702

europaul · 2025-03-21T20:04:55Z

Newlog will have 3 ways to reduce the amount of logs:

filter: filter logs out based on the source code line that produced them
counter: count the number of logs produced by a specific source code line. Add that number to the first occurance of the log and remove the rest
deduplicator (for errors only): record the last X errors in a sliding window and remove duplicates

The benchmarking of the newlogd with the new features is in the dedup_test.go file. It shows that CPU and RAM usage increase by a factor of 3 when the features are enabled. So they can be disabled by setting the deduplication window size to 0 and not providing anything to the filter and counter functions.

Copilot

Pull Request Overview

This PR adds new features to reduce log volume in Newlog by implementing three log reduction mechanisms: filtering, deduplication, and counting.

Introduces a deduplication feature for error logs using a ring buffer.
Implements a log counter that appends occurrence counts to log entries.
Adds configurable log filtering and updates global configuration settings accordingly.

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pkg/newlog/cmd/dedup.go	Adds deduplication functions for real‐time log reduction
pkg/newlog/cmd/counter.go	Implements a counter to tag and suppress duplicate log entries
pkg/newlog/cmd/dedup_test.go	Provides tests for deduplication logic
pkg/newlog/cmd/filter.go	Introduces log filtering using configurable filename filters
pkg/newlog/cmd/newlogd.go	Updates main log processing to integrate the new deduplication and filtering logic
pkg/pillar/types/global.go	Adds new global settings keys for deduplication, log counting, and filtering
pkg/newlog/cmd/newlogd_test.go	Minor test update for gzip log parsing

pkg/newlog/cmd/dedup_test.go

pkg/newlog/cmd/counter.go

Copilot

Pull Request Overview

This pull request adds three new log reduction mechanisms—filtering, deduplication, and counting—to help reduce excessive logs. Key changes include adding deduplication functionality with a sliding window (pkg/newlog/cmd/dedup.go), implementing log counting (pkg/newlog/cmd/counter.go), introducing log filtering (pkg/newlog/cmd/filter.go), and integrating these features via configuration into the log compression routine (pkg/newlog/cmd/newlogd.go).

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/newlog/cmd/dedup.go	New deduplication logic with a sliding window for error log entries
pkg/newlog/cmd/counter.go	New log counting functionality to annotate log entries
pkg/newlog/cmd/dedup_test.go	Test cases for deduplication with an identified iteration bug
pkg/newlog/cmd/filter.go	New log filtering mechanism based on filename criteria
pkg/pillar/types/global.go	Added configuration items for deduplication and filtering settings
pkg/newlog/cmd/newlogd.go	Integrated log deduplication, counting, and filtering into file compression
pkg/newlog/cmd/newlogd_test.go	Test scaffolding for log compression and integration testing

pkg/newlog/cmd/dedup_test.go

europaul · 2025-03-24T15:15:44Z

Waiting on #4703 to be merged to import the newer version of pillar with the right global config parameters.

europaul · 2025-03-28T11:15:40Z

Yetus seems to be bailing out again because the PR includes too many updated vendor dependencies.

rene · 2025-03-28T14:30:02Z

@eriknordmark could you rebase this PR, go modules for pkg/newlog were updated by dependabot...

This commit updates the go version to 1.24 and packages eve-api and eve/pkg/pillar to latest versions. Signed-off-by: Paul Gaiduk <[email protected]>

Newlog will have 3 ways to reduce the amount of logs: - filter: filter logs out based on the source code line that produced them - counter: count the number of logs produced by a specific source code line. Add that number to the first occurance of the log and remove the rest - deduplicator (for errors only): record the last X errors in a sliding window and remove duplicates The benchmarking of the newlogd with the new features is in the dedup_test.go file. It shows that CPU and RAM usage increase by a factor of 3 when the features are enabled. So they can be disabled by setting the deduplication window size to 0 and not providing anything to the filter and counter functions. Signed-off-by: Paul Gaiduk <[email protected]>

europaul · 2025-03-28T14:49:43Z

@eriknordmark could you rebase this PR, go modules for pkg/newlog were updated by dependabot...

@rene done

eriknordmark

I see yetus/golangcilint is failing with the output
pkg/newlog/level=error msg="Running error: context loading failed: failed to load packages: failed to load with go/packages: err: exit status 1: stderr: go: downloading go1.24 (linux/amd64)\ngo: download go1.24 for linux/amd64: toolchain not available\n"
pkg/newlog/level=warning msg="Failed to discover go env: failed to run 'go env': exit status 1"

europaul · 2025-03-28T19:52:53Z

I see yetus/golangcilint is failing with the output pkg/newlog/level=error msg="Running error: context loading failed: failed to load packages: failed to load with go/packages: err: exit status 1: stderr: go: downloading go1.24 (linux/amd64)\ngo: download go1.24 for linux/amd64: toolchain not available\n" pkg/newlog/level=warning msg="Failed to discover go env: failed to run 'go env': exit status 1"

@eriknordmark ~~where can you see this? I cannot find the error~~

nvm, I found it in scan-results artefact

rene · 2025-03-29T09:37:35Z

pkg/newlog/cmd/newlogd.go

+			// done gzip conversion, get rid of the temp log file in collect directory
+			err = os.Remove(tmpLogfileInfo.tmpfile)
+			if err != nil {
+				log.Fatal("doMoveCompressFile: remove file failed", err)


Is it really needed to have this error as fatal?

I know you are moving the code, but maybe is time to change it...

rene · 2025-03-29T09:39:11Z

pkg/newlog/go.mod

@@ -1,14 +1,13 @@
 module github.com/lf-edge/eve/pkg/newlog

-go 1.23
-toolchain go1.24.1


Just a note, sources parser was updated here: 6c40145 so toolchain line is supported now.

rene · 2025-03-29T09:41:20Z

pkg/newlog/go.mod

@@ -1,14 +1,13 @@
 module github.com/lf-edge/eve/pkg/newlog


Please, fix typo on commit message: "...go verison" -> "...go version"

rene · 2025-03-29T09:42:59Z

pkg/newlog/go.mod

@@ -1,14 +1,13 @@
 module github.com/lf-edge/eve/pkg/newlog

-go 1.23
-toolchain go1.24.1
+go 1.24


I think it must be 1.24.1 due to security fixes....

rene · 2025-03-29T09:45:13Z

@europaul , LGTM, just left a few comments....

christoph-zededa · 2025-03-31T14:01:42Z

pkg/newlog/cmd/counter.go

+func addLogCount(logEntry *logs.LogEntry, filterMap map[string]int) bool {
+	if count, ok := filterMap[logEntry.Filename]; !ok {
+		return true
+	} else {


Suggested change

} else {

}

christoph-zededa · 2025-03-31T14:05:21Z

pkg/newlog/cmd/dedup.go

+
+		// If the file hasn't appeared in the last bufferSize logs, forward it.
+		if _, ok := seen[dedupField]; ok && logEntry.severity == "error" {
+			log.Tracef("Deduped log at %s because of the log at %s\n", logEntry.timestamp, seen[dedupField])


Isn't this slow? Even if trace logs are disabled it gets called quite often, doesn't it?

christoph-zededa · 2025-03-31T14:07:54Z

pkg/newlog/cmd/dedup_test.go

+	}
+}
+
+func BenchmarkDoMoveCompressFile(b *testing.B) {


Nice!

I guess we do not run it as a GH action, do we?

christoph-zededa · 2025-03-31T14:21:05Z

pkg/newlog/cmd/dedup.go

+
+// deduplicateLogs can be used to deduplicate logs on the fly reading from a channel
+// and writing to another channel
+func deduplicateLogs(in <-chan inputEntry, out chan<- inputEntry) {


Where is this actually used outside of tests?

christoph-zededa · 2025-03-31T14:42:37Z

pkg/newlog/cmd/dedup.go

+	}
+
+	// If the file hasn't appeared in the last bufferSize logs, forward it.
+	if _, ok := seen[dedupField]; ok && logEntry.Severity == "error" {


logEntry.Severity == "error"
Means that only error log messages are deduplicated? What is the reasoning behind it?

christoph-zededa · 2025-03-31T14:50:48Z

pkg/newlog/cmd/newlogd.go

+		log.Functionf("handleGlobalConfigModify: gonna count the logs from the following lines %v", filenamesToCount)
+
+		// parse a comma separated list of log filenames to filter
+		newFilenameFilter := make(map[string]any)


Suggested change

newFilenameFilter := make(map[string]any)

newFilenameFilter := make(map[string]struct{})

christoph-zededa · 2025-03-31T14:54:28Z

pkg/newlog/cmd/newlogd.go

+		if dirName == uploadDevDir {
+			if len(filenameFilter.Load().(map[string]any)) != 0 || len(logCounter) != 0 || dedupWindowSize.Load() != 0 {


Suggested change

if dirName == uploadDevDir {

if len(filenameFilter.Load().(map[string]any)) != 0 || len(logCounter) != 0 || dedupWindowSize.Load() != 0 {

if dirName == uploadDevDir &&

len(filenameFilter.Load().(map[string]any)) != 0 ||

len(logCounter) != 0 ||

dedupWindowSize.Load() != 0 {

christoph-zededa · 2025-03-31T14:55:38Z

pkg/newlog/cmd/newlogd.go

+					continue // we don't care about the error here
+				}
+				var useEntry bool
+				if useEntry = !filterOut(&logEntry); !useEntry {


Suggested change

if useEntry = !filterOut(&logEntry); !useEntry {

if useEntry = filterOut(&logEntry); useEntry {

or perhaps directly:

Suggested change

if useEntry = !filterOut(&logEntry); !useEntry {

if filterOut(&logEntry) {

europaul requested review from deitch and eriknordmark as code owners March 21, 2025 20:04

europaul requested a review from Copilot March 21, 2025 20:05

github-actions bot requested a review from naiming-zededa March 21, 2025 20:05

europaul mentioned this pull request Mar 21, 2025

WIP: deduplicate and filter logs #4689

Closed

Copilot AI reviewed Mar 21, 2025

View reviewed changes

pkg/newlog/cmd/dedup_test.go Outdated Show resolved Hide resolved

pkg/newlog/cmd/counter.go Outdated Show resolved Hide resolved

pkg/newlog/cmd/counter.go Outdated Show resolved Hide resolved

europaul force-pushed the log-filtering-and-dedup branch from c0462b9 to e947722 Compare March 21, 2025 20:07

europaul requested a review from Copilot March 21, 2025 20:08

Copilot AI reviewed Mar 21, 2025

View reviewed changes

pkg/newlog/cmd/dedup_test.go Outdated Show resolved Hide resolved

europaul mentioned this pull request Mar 21, 2025

Add log deduplication and filtering settings to global configuration #4703

Merged

europaul force-pushed the log-filtering-and-dedup branch 2 times, most recently from 8572d7f to a2357dc Compare March 21, 2025 20:16

europaul force-pushed the log-filtering-and-dedup branch from a2357dc to e2ff9b3 Compare March 28, 2025 09:56

europaul added 2 commits March 28, 2025 15:46

newlog: bump go verison and update vendor

dbaa708

This commit updates the go version to 1.24 and packages eve-api and eve/pkg/pillar to latest versions. Signed-off-by: Paul Gaiduk <[email protected]>

europaul force-pushed the log-filtering-and-dedup branch from e2ff9b3 to aac93d6 Compare March 28, 2025 14:46

eriknordmark reviewed Mar 28, 2025

View reviewed changes

rene reviewed Mar 29, 2025

View reviewed changes

christoph-zededa reviewed Mar 31, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

newlog: add filter, dedup and counter functions #4702

newlog: add filter, dedup and counter functions #4702

europaul commented Mar 21, 2025

Copilot AI left a comment

Copilot AI left a comment

europaul commented Mar 24, 2025

europaul commented Mar 28, 2025

rene commented Mar 28, 2025

europaul commented Mar 28, 2025

eriknordmark left a comment

europaul commented Mar 28, 2025 •

edited

Loading

rene Mar 29, 2025

rene Mar 29, 2025

rene Mar 29, 2025 •

edited

Loading

rene Mar 29, 2025

rene Mar 29, 2025

rene commented Mar 29, 2025

christoph-zededa Mar 31, 2025

christoph-zededa Mar 31, 2025

christoph-zededa Mar 31, 2025

christoph-zededa Mar 31, 2025

christoph-zededa Mar 31, 2025

christoph-zededa Mar 31, 2025

christoph-zededa Mar 31, 2025

christoph-zededa Mar 31, 2025

christoph-zededa Mar 31, 2025

	newFilenameFilter := make(map[string]any)
	newFilenameFilter := make(map[string]struct{})

		if dirName == uploadDevDir {
		if len(filenameFilter.Load().(map[string]any)) != 0 \|\| len(logCounter) != 0 \|\| dedupWindowSize.Load() != 0 {

	if useEntry = !filterOut(&logEntry); !useEntry {
	if useEntry = filterOut(&logEntry); useEntry {

	if useEntry = !filterOut(&logEntry); !useEntry {
	if filterOut(&logEntry) {

newlog: add filter, dedup and counter functions #4702

Are you sure you want to change the base?

newlog: add filter, dedup and counter functions #4702

Conversation

europaul commented Mar 21, 2025

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

europaul commented Mar 24, 2025

europaul commented Mar 28, 2025

rene commented Mar 28, 2025

europaul commented Mar 28, 2025

eriknordmark left a comment

Choose a reason for hiding this comment

europaul commented Mar 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rene Mar 29, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rene commented Mar 29, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

europaul commented Mar 28, 2025 •

edited

Loading

rene Mar 29, 2025 •

edited

Loading