feat(commons): reclaim storage on a guest whose disk is already full

[kholisrag]

Host-agnostic — point it at any inventory in this tree.

ansible-playbook -i live/ansible/inventories/petruk-pve/petruk-pve0/pve-vms/cloudflared \
  live/ansible/playbooks/commons/storage/clean-storage.yaml

Why the first play uses raw and nothing else

This is the failure that motivated it, and it does not look like a disk problem:

TASK [Gathering Facts] *********************************************************
[ERROR]: Task failed: mkdir: cannot create directory
'/tmp/ansible-tmp-1785886323.066606-45077-165721258896029': No space left on device
fatal: [cloudflared_vm]: UNREACHABLE!

UNREACHABLE reads as SSH or connectivity. It is neither. Ansible copies every
module — including the implicit setup — into remote_tmp before it can run
it, and ansible.cfg sets that to /tmp. At 100% there is nowhere to put it, so
the run dies before task one and no ordinary playbook can help.

So play one is ansible.builtin.raw only, with gather_facts: false. raw runs
straight down the SSH channel and needs neither a temp directory nor Python. It
frees the journal and the apt cache — both pure deletes, which is what makes them
work at zero bytes free — and hands a usable filesystem to play two. If even that
is not enough it stops with the reason rather than repeating the UNREACHABLE.

What it removes

	Default	Notes
systemd journal	100 MiB / 14 days	systemd's own default is 10% of the filesystem
apt cache	emptied	every .deb is re-downloadable
orphaned packages	autoremove --purge	superseded kernels and their headers
rotated logs	older than 14 days	*.gz, *.xz, *.zst, *.1, … under /var/log
/tmp, /var/tmp	untouched 7 days	by atime, not mtime
docker	off	opt in per run

Tags: journal, apt, logs, tmp, docker, kernels. Every default is a
play var.

Two guards, both found by running it for real

apt-lock pre-check. ansible.builtin.apt waits lock_timeout then fails with
a message naming neither the holder nor its age — and it aborts the play, so the
log reclaim never reaches the host that needed it most. It now skips apt, prints
the holder with elapsed time, and continues. On cloudflared_vm the holder was an
apt-get -qq -y update from apt.systemd.daily wedged for 61 days.

Superseded-kernel release. autoremove only takes packages marked auto, so a
manually-marked kernel is invisible to the one tool meant to find it. This releases
them and lets apt's own retention rules decide, with the running and newest kernels
excluded — and asserted afterwards, because the cost of that filter being wrong is a
guest that does not come back from a reboot.

That second guard also records the trap that made this hard to measure:
dpkg-query -W lists rc packages — removed, config files remaining — beside
installed ones, each with a plausible Installed-Size. Counting those is how a
guest with two kernels reads as a guest with fourteen.

REBOOT PENDING

Reported when the running kernel is not the newest installed. autoremove keeps
both, so the pile only collapses on the reboot that makes them the same package —
the difference between "this playbook found nothing" and "this playbook cannot
help until you reboot".

Verified end to end

cloudflared_vm went from 100% used and UNREACHABLE to 77% used, 678 MiB
free, and a re-run is a clean no-op. ansible-lint passes at the production
profile; yamllint, typos and lychee clean.

The README also notes when cleaning is the wrong answer — if the reclaim is small
and the guest fills again next week, the disk is too small and the fix belongs in
the OpenTofu unit.