Skip to content

Commit 4220d3d

Browse files
authored
fix(updater): preserve Kamino disk headroom (#3458)
1 parent 4a9d9c1 commit 4220d3d

3 files changed

Lines changed: 163 additions & 1 deletion

File tree

‎home-manager/services/dotfiles-updater/README.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,20 @@ present, advances `~/.local/state/dotfiles-updater/installed-commit` (or the sam
1313
path under XDG_STATE_HOME). Failed installs and syncs are attempted on a later run.
1414
The marker records installation completion; it does not certify live service health.
1515

16+
On automated Kamino updates, a local Nix build starts only with at least 64 GiB
17+
free on the root filesystem. This reserve uses the 42 GiB needed to preserve
18+
one executor lane and adds margin based on roughly 20 GB of K4 Nix-store growth
19+
observed during an updater run. That observed growth does not bound every
20+
future build. Between 42 and 64 GiB, the updater can apply an activation
21+
closure already present in the local Nix store while disabling local Nix
22+
builds for that install. If the closure is missing, the updater leaves the
23+
installed marker unchanged and retries on the next timer. A trusted Nix builder
24+
can build this exact flake output at the target commit and copy its closure into
25+
the host store with `nix copy`; the updater verifies it with offline
26+
`nix path-info --recursive`. Otherwise, free root space or increase host capacity
27+
above 64 GiB so the timer can build locally. At less than 42 GiB free,
28+
automated updates defer without applying changes.
29+
1630
When the updater reports a blocked checkout, preserve the branch, tracked edits,
1731
and untracked files in their owning worktree or backup before restoring a clean
1832
deployment checkout. Do not reset or clean somebody else's work. Run deployment

‎home-manager/services/dotfiles-updater/update.sh‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,50 @@ fi
9191

9292
echo "Changes detected: ${CURRENT_COMMIT:0:8} -> ${REMOTE_COMMIT:0:8} (installed: ${INSTALLED_COMMIT:0:8})"
9393

94+
# A local Home Manager activation can add substantial space to the Nix store
95+
# while compiling large Rust packages. Kamino executor admission requires 40
96+
# GiB free plus 2 GiB per new lane. Reserve 64 GiB before an automated local
97+
# build, based on the 42 GiB one-lane floor and roughly 20 GB of K4 store growth
98+
# observed during its updater run, plus margin. That growth is not a per-build
99+
# maximum. An already installed activation closure can still be applied with
100+
# local Nix builds disabled.
101+
MIN_FREE_FOR_LOCAL_BUILD_KIB=$((64 * 1024 * 1024))
102+
MIN_EXECUTOR_RESERVE_KIB=$((42 * 1024 * 1024))
103+
if [ "${AUTOMATED_UPDATE:-}" = true ] && [[ "$HOST" =~ ^kamino[0-9]+$ ]]; then
104+
AVAILABLE_KIB=$(df -Pk / 2>/dev/null | awk 'NR == 2 { print $4 }')
105+
if [[ ! "$AVAILABLE_KIB" =~ ^[0-9]+$ ]]; then
106+
echo "Deferring automated Kamino install: root free space could not be read; installed marker is unchanged." >&2
107+
exit 0
108+
fi
109+
110+
if [ "$AVAILABLE_KIB" -lt "$MIN_EXECUTOR_RESERVE_KIB" ]; then
111+
AVAILABLE_GIB=$((AVAILABLE_KIB / 1024 / 1024))
112+
echo "Deferring automated Kamino install: ${AVAILABLE_GIB} GiB free is below the 42 GiB executor reserve; installed marker is unchanged." >&2
113+
exit 0
114+
fi
115+
116+
if [ "$AVAILABLE_KIB" -lt "$MIN_FREE_FOR_LOCAL_BUILD_KIB" ]; then
117+
ACTIVATION=".#homeConfigurations.${HOST}.activationPackage"
118+
if ! nix --offline --extra-experimental-features 'nix-command flakes' \
119+
path-info --recursive --impure --no-update-lock-file "$ACTIVATION" >/dev/null 2>&1; then
120+
AVAILABLE_GIB=$((AVAILABLE_KIB / 1024 / 1024))
121+
echo "Deferring automated Kamino install: ${AVAILABLE_GIB} GiB free and the activation closure is not fully in the local Nix store. Free more disk or run an operator-managed build; installed marker is unchanged." >&2
122+
exit 0
123+
fi
124+
125+
# The entire activation closure is already in the local store, so applying
126+
# it needs no compilation or cache download. Keep any other Nix operation
127+
# in this automated install from starting a local build at low headroom.
128+
NIX_CONFIG="${NIX_CONFIG:-}"
129+
if [ -n "$NIX_CONFIG" ]; then
130+
NIX_CONFIG+=$'\n'
131+
fi
132+
NIX_CONFIG+="max-jobs = 0"
133+
export NIX_CONFIG
134+
echo "Applying the store-resident Kamino activation with local Nix builds disabled (${AVAILABLE_KIB} KiB free)."
135+
fi
136+
fi
137+
94138
# Bootstrap's install.sh fetches again; install the revision already checked above.
95139
make install
96140

‎spec/dotfiles_updater_spec.sh‎

Lines changed: 105 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,11 +129,13 @@ setup() {
129129
export HOME="$WORK/home"
130130
export XDG_STATE_HOME="$WORK/state"
131131
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null
132+
export GIT_ALLOW_PROTOCOL=file GIT_TERMINAL_PROMPT=0
132133
export GIT_AUTHOR_NAME=t GIT_AUTHOR_EMAIL=t@t GIT_COMMITTER_NAME=t GIT_COMMITTER_EMAIL=t@t
133134
mkdir -p "$HOME" "$WORK/src" "$WORK/stub"
134135
# The EXIT trap starts timers; keep it off the host's systemd user manager.
135136
printf '#!/usr/bin/env bash\nexit 0\n' >"$WORK/stub/systemctl"
136-
chmod +x "$WORK/stub/systemctl"
137+
printf '#!/usr/bin/env bash\necho "Filesystem 1024-blocks Used Available Capacity Mounted on"\necho "/dev/sda1 400000000 300000000 70000000 80%% /"\n' >"$WORK/stub/df"
138+
chmod +x "$WORK/stub/systemctl" "$WORK/stub/df"
137139
export PATH="$WORK/stub:$PATH"
138140
git -C "$WORK/src" init -q -b main
139141
mkdir -p "$WORK/src/home-manager/activation"
@@ -314,6 +316,108 @@ The output should include 'Changes detected'
314316
The error should include 'Checkout changed during installation'
315317
The file "$XDG_STATE_HOME/dotfiles-updater/installed-commit" should not be exist
316318
End
319+
320+
Describe 'automated Kamino disk headroom'
321+
It 'defers a local activation build below the required headroom without advancing the marker'
322+
printf '#!/usr/bin/env bash\necho "Filesystem 1024-blocks Used Available Capacity Mounted on"\necho "/dev/sda1 400000000 350000000 55000000 88%% /"\n' >"$WORK/stub/df"
323+
cat >"$WORK/stub/nix" <<'NIX'
324+
#!/usr/bin/env bash
325+
expected='--offline --extra-experimental-features nix-command flakes path-info --recursive --impure --no-update-lock-file .#homeConfigurations.kamino4.activationPackage'
326+
if [[ "$*" != "$expected" ]]; then
327+
echo "unexpected nix arguments: $*" >&2
328+
exit 2
329+
fi
330+
exit 1
331+
NIX
332+
printf '#!/usr/bin/env bash\ntouch "$HOME/make-called"\n' >"$WORK/stub/make"
333+
chmod +x "$WORK/stub/df" "$WORK/stub/nix"
334+
chmod +x "$WORK/stub/make"
335+
cat >"$WORK/retry.sh" <<'RETRY'
336+
#!/usr/bin/env bash
337+
set -euo pipefail
338+
script_path=$1
339+
workspace=$2
340+
export PATH="$workspace/stub:$PATH"
341+
export AUTOMATED_UPDATE=true HOST=kamino4
342+
bash "$script_path" >"$workspace/deferred.log" 2>&1
343+
test ! -f "$XDG_STATE_HOME/dotfiles-updater/installed-commit"
344+
test ! -f "$HOME/make-called"
345+
expected_commit=$(git -C "$HOME/dotfiles" rev-parse HEAD)
346+
printf '#!/usr/bin/env bash\necho "Filesystem 1024-blocks Used Available Capacity Mounted on"\necho "/dev/sda1 400000000 300000000 70000000 80%% /"\n' >"$workspace/stub/df"
347+
bash "$script_path" >"$workspace/retried.log" 2>&1
348+
test "$(cat "$XDG_STATE_HOME/dotfiles-updater/installed-commit")" = "$expected_commit"
349+
cat "$workspace/deferred.log" "$workspace/retried.log"
350+
echo retried-same-commit
351+
RETRY
352+
chmod +x "$WORK/retry.sh"
353+
When run bash "$WORK/retry.sh" "$SCRIPT" "$WORK"
354+
The status should be success
355+
The output should include 'Deferring automated Kamino install'
356+
The output should include 'Changes detected'
357+
The output should include 'retried-same-commit'
358+
End
359+
360+
It 'defers below the executor reserve without querying or installing Nix paths'
361+
printf '#!/usr/bin/env bash\necho "Filesystem 1024-blocks Used Available Capacity Mounted on"\necho "/dev/sda1 400000000 360000000 40000000 90%% /"\n' >"$WORK/stub/df"
362+
printf '#!/usr/bin/env bash\ntouch "$HOME/nix-called"\nexit 0\n' >"$WORK/stub/nix"
363+
printf '#!/usr/bin/env bash\ntouch "$HOME/make-called"\n' >"$WORK/stub/make"
364+
chmod +x "$WORK/stub/df" "$WORK/stub/nix" "$WORK/stub/make"
365+
When run env PATH="$WORK/stub:$PATH" AUTOMATED_UPDATE=true HOST=kamino4 bash "$SCRIPT"
366+
The status should be success
367+
The output should include 'Changes detected'
368+
The error should include 'From '
369+
The error should include '38 GiB free is below the 42 GiB executor reserve'
370+
The file "$HOME/nix-called" should not be exist
371+
The file "$HOME/make-called" should not be exist
372+
The file "$XDG_STATE_HOME/dotfiles-updater/installed-commit" should not be exist
373+
End
374+
375+
It 'applies a store-resident activation with local Nix builds disabled below the build threshold'
376+
printf '#!/usr/bin/env bash\necho "Filesystem 1024-blocks Used Available Capacity Mounted on"\necho "/dev/sda1 400000000 350000000 55000000 88%% /"\n' >"$WORK/stub/df"
377+
cat >"$WORK/stub/nix" <<'NIX'
378+
#!/usr/bin/env bash
379+
expected='--offline --extra-experimental-features nix-command flakes path-info --recursive --impure --no-update-lock-file .#homeConfigurations.kamino4.activationPackage'
380+
if [[ "$*" != "$expected" ]]; then
381+
echo "unexpected nix arguments: $*" >&2
382+
exit 2
383+
fi
384+
NIX
385+
cat >"$WORK/stub/make" <<'MAKE'
386+
#!/usr/bin/env bash
387+
if [ "$1" = install ]; then
388+
last_config_line=${NIX_CONFIG##*$'\n'}
389+
if [[ "$NIX_CONFIG" != *"trusted-users = root"* || "$last_config_line" != "max-jobs = 0" ]]; then
390+
echo "expected cache-only build setting" >&2
391+
exit 1
392+
fi
393+
touch "$HOME/installed"
394+
fi
395+
MAKE
396+
chmod +x "$WORK/stub/df" "$WORK/stub/nix" "$WORK/stub/make"
397+
When run env PATH="$WORK/stub:$PATH" AUTOMATED_UPDATE=true HOST=kamino4 NIX_CONFIG="$(printf 'trusted-users = root\nmax-jobs = 8')" bash "$SCRIPT"
398+
The status should be success
399+
The output should include 'Changes detected'
400+
The output should include 'Applying the store-resident Kamino activation with local Nix builds disabled'
401+
The error should include 'From '
402+
The file "$HOME/installed" should be exist
403+
The file "$XDG_STATE_HOME/dotfiles-updater/installed-commit" should be exist
404+
End
405+
406+
It 'allows a manual updater run to build below the automated headroom threshold'
407+
printf '#!/usr/bin/env bash\necho "Filesystem 1024-blocks Used Available Capacity Mounted on"\necho "/dev/sda1 400000000 350000000 50000000 89%% /"\n' >"$WORK/stub/df"
408+
cat >"$WORK/stub/make" <<'MAKE'
409+
#!/usr/bin/env bash
410+
if [ "$1" = install ]; then touch "$HOME/installed"; fi
411+
MAKE
412+
chmod +x "$WORK/stub/df" "$WORK/stub/make"
413+
When run env PATH="$WORK/stub:$PATH" HOST=kamino4 bash "$SCRIPT"
414+
The status should be success
415+
The output should include 'Changes detected'
416+
The error should include 'From '
417+
The file "$HOME/installed" should be exist
418+
The file "$XDG_STATE_HOME/dotfiles-updater/installed-commit" should be exist
419+
End
420+
End
317421
End
318422

319423
Describe 'cliproxy service PATH'

0 commit comments

Comments
 (0)