Skip to content

tools/filewatch: add VFS file access tracer with process ancestry - #5547

Open
ElCoyote27 wants to merge 5 commits into
iovisor:masterfrom
ElCoyote27:master
Open

tools/filewatch: add VFS file access tracer with process ancestry#5547
ElCoyote27 wants to merge 5 commits into
iovisor:masterfrom
ElCoyote27:master

Conversation

@ElCoyote27

@ElCoyote27 ElCoyote27 commented Aug 26, 2026

Copy link
Copy Markdown

Description

This PR aims to propose a new script : tools/filewatch

Why this approach

I wrote the filewatch tool (with the help of Cursor AI to do the heavy lifting) to observe files on production systems.
Sometimes, mysterious files pop up in directories, and it's difficult to trace their origin. Sometimes, short-lived files are created and immediately deleted in some shared directories (think /tmp, etc.), making it difficult to attribute ownership.

I am aware that the auditd filesystem, ftrace and bcc's opensnoop / fileslower tools exist but I wanted something to walk back the process to identify rogue processes/tools and that could be used by sysadmins without having to create a custom ad'hoc, script.

The most compelling examples are real-world "mystery solved" scenarios that sysadmins face daily. Here are three that I'd pick for the PR:

  1. "Who keeps modifying my config?" -- the classic sysadmin headache:
# ./filewatch /etc/resolv.conf
========================================================================
WRITE  03:15:01  file=resolv.conf  bytes=89  user=root
       path: /etc/resolv.conf
Process tree (caller first):
>>> [8842] NetworkManager
    cmdline: /usr/sbin/NetworkManager --no-daemon
    exe:     /usr/sbin/NetworkManager
      [1] systemd  (init)

This immediately answers "why does my resolv.conf keep reverting?" -- a question that with auditd would require setting up rules, parsing audit.log, and correlating ausearch output.

  1. Catching an automation tool mid-flight -- the ancestry is the value:
# ./filewatch -w /etc/hosts
========================================================================
WRITE  14:22:33  file=hosts  bytes=156  user=root
       path: /etc/hosts
Process tree (caller first):
>>> [51234] sed
    cmdline: sed -i s/oldhost/newhost/g /etc/hosts
    exe:     /usr/bin/sed
      [51230] bash
      cmdline: /bin/bash /tmp/ansible_xyz/command.sh
      exe:     /usr/bin/bash
        [51200] python3
        cmdline: /usr/bin/python3 /usr/bin/ansible-playbook site.yml
        exe:     /usr/bin/python3
          [4562] sshd
          cmdline: sshd: deploy@pts/2
          exe:     /usr/libexec/openssh/sshd-session
            [1] systemd  (init)

This shows the full chain: sed was called by a bash wrapper, spawned by Ansible, running over SSH as user deploy. No other tool gives you this in a single command.

  1. Catching short-lived processes -- the in-kernel cmdline capture:
# ./filewatch -cu /tmp/
========================================================================
CREATE 09:01:00  file=tmpXk9f3a  user=www-data
       path: /tmp/tmpXk9f3a
Process tree (caller first):
>>> [9921] php-fpm
    cmdline: php-fpm: pool www
    exe:     /usr/sbin/php-fpm
      [1203] php-fpm
      cmdline: php-fpm: master process (/etc/php-fpm.conf)
        [1] systemd  (init)
========================================================================
UNLINK 09:01:00  file=tmpXk9f3a  user=www-data
       path: /tmp/tmpXk9f3a
Process tree (caller first):
>>> [9921] php-fpm
    [...]

This catches temp file churn that would be invisible to lsof (file comes and goes in under a second) and that inotifywait would report without any process information.

And another one (who changed the sudoers global config without using sudo as his/her user?):

========================================================================
READ   09:43:38  file=sudoers  bytes=4096  user=root
       path: /etc/sudoers
Process tree (caller first):

>>> [307916] visudo
    exe:     /usr/sbin/visudo
    cwd:     /root
      [303669] bash
      cmdline: -bash
      exe:     /usr/bin/bash
        [303668] sudo
        cmdline: sudo -i
        exe:     /usr/bin/sudo
          [303492] sudo
          cmdline: sudo -i
          exe:     /usr/bin/sudo
            [302656] bash
            cmdline: -/bin/bash
            exe:     /usr/bin/bash
              [148341] screen
              cmdline: SCREEN -dRRR
              exe:     /usr/bin/screen
                [148321] screen
                cmdline: screen -dRRR
                exe:     /usr/bin/screen
                  [139545] bash
                  cmdline: -bash
                  exe:     /usr/bin/bash
                    [139417] sshd-session
                    cmdline: sshd-session: raistlin@pts/12
                    exe:     /usr/libexec/openssh/sshd-session
                      [135473] sshd-session
                      cmdline: sshd-session: raistlin [priv]  
                      exe:     /usr/libexec/openssh/sshd-session
                        [4514] sshd
                        cmdline: sshd: /usr/sbin/sshd -D -f /etc/ssh/sshd_config -oPidFile=/var/run/sshd.pid [listener] 0 of 10-100 startups
                        exe:     /usr/sbin/sshd
                          [1] systemd
                          cmdline: /usr/lib/systemd/systemd --switched-root --system --deserialize 31
                          exe:     /usr/lib/systemd/systemd  (init)

Happy to make changes and open to suggestions.


Checklist

  • Commit prefix matches changed area (e.g., tools/toolname:, libbpf-tools/toolname:, src/cc:, docs:, build:, tests/python:)
  • Commit body explains why this change is needed

For new tools only

  • Explains why this tool is needed and what existing tools cannot cover this use case
  • Includes at least one real production use case
  • Man page (man/man8/) with an OVERHEAD section
  • Example output file (*_example.txt)
  • README.md entry added
  • Smoke test added to tests/python/test_tools_smoke.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant