Memory problems after issuing the reload --force command

[hosszugyula]

Hello!

I am currently struggling with a problem where, when I issue the caddy reload --config </path/to/config.json> --force command, the regex/aho-corasick processes that appear to belong to the waf handler double the memory usage. If I remove the waf handler from the config, the problem disappears completely.

This problem currently occurs when caddy is hosting a large number of domains. There is no problem with 10-15 domains, but on servers with 1000+ domains, caddy can consume up to 10-12Gb of RAM because of this.

At startup, even with 1000 domains, Caddy runs with 600-700 MB together with WAF. But after that, memory usage increases by 300-400 MB with each reload.

The memory usage of the first two processes increases significantly with each reload:

github.com/petar-dambovaliev/aho-corasick.(*iNFA).addSparseState
regexp/syntax.(*compiler).inst

(pprof) top
Showing nodes accounting for 762.99MB, 81.58% of 935.31MB total
Dropped 226 nodes (cum <= 4.68MB)
Showing top 10 nodes out of 104
flat flat% sum% cum cum%
260.13MB 27.81% 27.81% 260.13MB 27.81% github.com/petar-dambovaliev/aho-corasick.(*iNFA).addSparseState
115.22MB 12.32% 40.13% 115.22MB 12.32% regexp/syntax.(*compiler).inst
90.51MB 9.68% 49.81% 90.51MB 9.68% github.com/petar-dambovaliev/aho-corasick.(*transitions).setNextState
72.11MB 7.71% 57.52% 72.11MB 7.71% github.com/corazawaf/coraza/v3/internal/corazawaf.(*RuleGroup).Add
46.01MB 4.92% 62.44% 46.01MB 4.92% github.com/corazawaf/coraza/v3/internal/corazawaf.(*Rule).AddAction
45.89MB 4.91% 67.34% 46.39MB 4.96% github.com/petar-dambovaliev/aho-corasick.(*iDFABuilder).build
42.52MB 4.55% 71.89% 42.52MB 4.55% strings.(*Builder).WriteString
40.58MB 4.34% 76.23% 40.58MB 4.34% github.com/petar-dambovaliev/aho-corasick.newDense
31.01MB 3.32% 79.54% 31.01MB 3.32% github.com/corazawaf/coraza/v3/internal/corazawaf.NewRule
19MB 2.03% 81.58% 33MB 3.53% github.com/corazawaf/coraza/v3/experimental/plugins/macro.(*macro).compile

I need the force reload because I load the certificates manually into caddy, and when the certs are renewed, we issue a reload command.

[jptosso]

This is a known issue and it's addressed by using the memoize flag while compiling caddy with coraza
I hope we fix this in march, but it's an issue with Owasp crs memory footprint while creating duplicated rules

[hosszugyula]

Yes, I found an issue related to this #76, but I saw that a possible fix had already been merged into main, according to this, there is currently no workaround to solve this, right?

[jptosso]

Memoize is the fix. Unfortunately coraza waf 3 doesn't support closing the context, so it's not natively compatible with the memoize approach. we have been discussing this issue for a while and it implies releasing v4