Option to make memory enrichment tables persistent after Vector restarts

[marcosirianni]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

I'm using the new memory enrichment table feature (which is great btw) to enrich firewall logs with Wi-Fi controller logs. Wi-Fi controller logs bring information such as hostname, username and MAC address. The enrichment happens when source IP in firewall logs matches an IP in the following table:

enrichment_tables:
  wifi_ip_table:
    type: memory
    ttl: 86400
    flush_interval: 1
    inputs: [ "wifi_cache_ips" ]

"wifi_cache_ips" is a transform that populates the table or modifies entries with up-to-date information (i.e. when an IP is associated with new information about user, hostname, etc.)

As Vector gets restarted at midnight because the logs generated by File sinks get rotated, this memory table is wiped.

Attempted Solutions

No response

Proposal

I think a "survive_restarts" option in enrichment table configuration like the following:

enrichment_tables:
  wifi_ip_table:
    type: memory
    ttl: 86400
    flush_interval: 1
    survive_restarts: true
    inputs: [ "wifi_cache_ips" ]

Would be quite useful to avoid starting every day from zero IPs and avoid losing enrichment for IPs that are already assigned and won't appear again in Wi-Fi logs (as they're generated only when a new device gets an IP address).
The enrichment table could be saved in some kind of binary format when Vector gets a SIGTERM or similar and loaded every time Vector starts, if available.

References

No response

Version

vector 0.45.0 (x86_64-unknown-linux-gnu 063cabb 2025-02-24 14:52:02.810034614)