ram usage on startup, profile python ?

[dsnz]

Hello,

I use netalertx in docker container,
all was fine for some days and ram usage was steady at 160 MB with some peeks at 185 MB so I put a docker limit at 256 MB

now the limit has been reached (I did not check every day the trend, it's holidays) and the container stopped,
when I restart it, it jumps very quickly in a few minutes to 250 MB usage and soon it dies by reaching the limit

the db is 50 MB in size (and shouldn't be all in RAM I guess ..), the largest log file is 23 MB (this shouldn't be loaded in RAM; also for info I have the log files in bind volumes on disk and not in tmpfs)

so I wonder what could be using the RAM in the python server (it's seen from ps that the python server is the main consumer of RAM)
and what can I do ?
also as a last resort .. how could I profile the python programs (I mean which functions) to find the memory hog ?

has anyone prior experience with RAM usage ?

[jokob-sk]

Hi @ingoratsdorf - not sure if you can help here? I think more ram use is expected based on data growth as we are caching some data for performance reasons and to reduce disk R/W.

I'd recommend trying to increase the upper limit to at least 300, 512 to be safe.

[jokob-sk]

Hi,

Thanks for digging into this.

The Plugins_History table is updated after each scan (I assume you set the scans once every 3 min). The entries are then fed into the updated API endpoint (e.g. used to display the Integrations -> Plugins data tables). BTW I have long-term plans to optimize this and do a GraphQL rewrite, but for that I need to update the DB schema - GraphQL doesn't like _ in column names.

but I understand that I must run frequently the DB cleanup and the Maintenance schedules in order to keep this table trimmed

That's my suggestion, to keep the DB size at reasonable levels, make sure the DBCLNP and MAINT plugins run regularly. You can also adjust the PLUGINS_KEEP_HIST to manage the minimum number of historical entries to keep.

you can run plugins also manually:

Or you can ssh in teh container and run it directly:

python3 /app/front/plugins/db_cleanup/script.py

Synology-NAS:/$ python3 /app/front/plugins/db_cleanup/script.py
10:44:56 [DBCLNP] In script
10:44:56 [DBCLNP] Upkeep Database: /data/db/app.db
10:44:57 [DBCLNP] REINDEX completed
10:44:57 [DBCLNP] Online_History: Delete all but keep latest 150 entries
10:44:57 [DBCLNP] Events: Delete all older than 60 days (DAYS_TO_KEEP_EVENTS setting)
10:44:57 [DBCLNP] SQL : DELETE FROM Events WHERE eve_DateTime <= date('now', '-60 day')
10:44:57 [DBCLNP] Plugins_History: Trim Plugins_History entries to less than 499 per Plugin (PLUGINS_KEEP_HIST setting)
10:44:57 [DBCLNP] Plugins_History: Trim Notifications entries to less than 1000
10:44:57 [DBCLNP] Trim AppEvents to less than 5000
10:44:57 [DBCLNP] Plugins_Objects: Delete all duplicates
10:44:58 [DBCLNP] WAL checkpoint executed to truncate file.
10:44:58 [DBCLNP] Shrink Database
10:44:59 [DBCLNP] Cleanup complete

[jokob-sk]

If you wanted to you could also mount the /tmp directory to a non-tmpfs location - this would then slow down the app, batter the HDD, but use less RAM.

[dsnz]

hello,
it's been 2 weeks since last update here , and in the meantime I tested further netalertx and I have more issues to add,
I consider the below marked 1-3 important and the 4-9 much less important or just informative

I will try to be short but accurate, if you think some issues should be opened in separate GitHub issues, then tell me

1. ***
   even with hourly db cleanup the plugins_history gets quickly new entries and in some cases has reached up to 30_000
   I inspected the table with SQLite cli and found that there are 2 types of entries

"new" and
"missing-in-last-scan" (!)

the "new" ones are the real ones
and when a "new" is entered there are also entered many "missing-in-last-scan" before it (before in the numeric id order)
these entries can be many hundred often and I saw one time 5500 such entries !
what's extra strange is that many have older created-changed dates (even many dates before)

are these normal entries and needed for operation or something is wrong with my settings ?

the good news is that with the hourly db cleanup and with usually 10-20k only entries max
the "[API] api_endpoint_class: Writing table_plugins_events.json" operation takes now 3-12 sec instead of 20-40 sec previously

2. ***
   initially I had many scanners and this together with the issue with long update time in 1. (Writing table_plugins_events.json) made the resulting schedule out of sync
   now I have only ARPSCAN, NBTSCAN (both at 3 minutes initially and hoping to get it down to 2 minutes) , Internet scan and Rogue DHCP scan (both at 5 minutes) , which are all very fast
   however I saw again that the schedule was never exact , or how to say .. cron-like..,
   and I think it's because of a main loop that does all scheduling and processing , which of course cannot do an accurate cron-like schedule
   but maybe this would not big issue except for the following new problem that surfaced and annoys a lot

it now happens that with these few and fast scanners that the
'Process Scan' runs twice in sequence without an intervening ARPSCAN in-between
but by some kind of processing the system assumes that all devices are down !! (ARPSCAN did not say the devices are up so they must be down !!)
so all day long I get frequently a notification listing 10-20 devices down including even the Router (!) and soon within 30-60 sec a new notification that all devices are up

is this a bug or have I done something wrong in settings ?

3. **
   I noticed that when a device is down (disconnects),
   the event time is not really the time the first scanner noticed it (the earliest time) but it's the time the 'Process Scan' runs and presumably makes a final event record and sends the notification (this description is my understanding, I don't really know the details of netalertx operation)
   the time difference can be up to 2 minutes
   I suspect that the same thing happens when a device connects again (ie. connection event time is later than true one) but I did not happen to check the logs for this until now

can you change the logic so that the correct event time is recorded ?

memory usage is steady around 200 M for the python server (with hourly db cleanup)
and about 250 M for nginx,php and other processes
so I stopped having always open the management web page and overall memory use went down to 350-380 M from about 480 M , still a bit high for my needs but more stable and workable
each icmp,arpscan plugin uses 50M (in fact I feel every plugin consumes 50 M , but I have not verified this exhaustively)
I think there is room for improvement but this will come after 1-3 are worked out first

5. *
   I often get notifications that the external-ip is down and then up,
   when the Internet scanner has not returned any error,
   it doesn't seem related to 2. above but I leave it for later research again after 1-3

clicking purge for the stdout.log file, gives message that file is not writable for /tmp/log/app_front.logFile ! (such a file does not exist and all files in /tmp/log are writable including /tmp/log itself)

7. *
   a MAINT setup for stdout.log is missing when there is one for app.log, but stdout.log is the one that's getting bigger and most affected by selecting debug logging

8. *
   a annoying thing is that,
   I had initially loaded the DIG plugin,
   later I unloaded it but still after saving any new settings
   I get to see the plugin in the schedule list on the top of the settings screen (!)
   and also see the full setting interface of DIG available and setup as before_name_updates,
   I have to set it as 'disabled' and save again a 2nd time to remove it entirely from netalertx screen

9. *
   I thought that ICMP and NBTSCAN would scan for new devices,
   it seems though that they just check that devices found by ARPSCAN are up,
   or am I missing something in the settings ?

[jokob-sk]

Hi, please break this down into separate github issues if you need me to address anything in more detail. Otherwise this will become messy. Glad the memory is in check now - I consider the original issue solved.

[dsnz]

done :)