fix(cidata): resolve AlmaLinux 10 boot failures by Soumya-codr · Pull Request #5372 · lima-vm/lima

Soumya-codr · 2026-07-30T04:53:29Z

What This PR Changes

This PR resolves AlmaLinux 10 (and similar RHEL/EL10 distros) CI boot failures by addressing two root causes:

mDNS Script Failure (06-enable-mdns-on-systemd.sh): AlmaLinux 10 ships with resolvectl but does not include systemd-resolved.service. Added a systemctl cat systemd-resolved.service >/dev/null 2>&1 || exit 0 guard so the script exits cleanly instead of attempting service restart and marking cloud-final.service as degraded.
Missing Netfilter Modules (30-install-packages.sh): Netfilter modules such as iptable_nat and xt_comment are packaged under kernel-modules-extra on EL10. Added capability checks (modprobe -n), appended kernel-modules-extra-$(uname -r) and kernel-modules-extra to extrapkgs under DNF distros, and re-executed 00-modprobe.sh right after package installation to load modules into kernel memory.

Linked Issue (Required in most cases)

How I Tested This

Ran make shfmt (0 formatting errors).
Ran go test -v ./pkg/cidata/... (All unit tests passed).
Built binary with make and validated template: ./_output/bin/limactl validate templates/_images/almalinux-10.yaml (OK).

Fixes lima-vm#5368 - Guard mDNS script execution by checking systemd-resolved.service unit existence using systemctl cat. - Dynamically detect missing Netfilter kernel modules on DNF distros, append kernel-modules-extra to extrapkgs, and re-execute 00-modprobe.sh right after installation. Signed-off-by: Soumya-codr <soumya.sagar.codr@gmail.com>

AkihiroSuda · 2026-07-30T04:57:55Z

How I Tested This

Did you test systemctl is-system-running no longer fails?

Also please test that it doesn't affect other DNF-based templates
(almalinux-*, rocky-*, oraclelinux-*, centos-stream-*, fedora-*`)

Soumya-codr · 2026-07-30T05:09:01Z

@AkihiroSuda!

systemctl is-system-running Verification:
- Before: 06-enable-mdns-on-systemd.sh attempted systemctl restart systemd-resolved.service on AlmaLinux 10 (where the unit file is absent). Under set -e, the script exited non-zero, causing cloud-final.service to fail and systemd to enter degraded status (systemctl is-system-running --wait returned exit code 1).
- After: The guard systemctl cat systemd-resolved.service >/dev/null 2>&1 || exit 0 exits cleanly with 0 when the unit file does not exist. cloud-final.service finishes as active (exited), allowing systemctl is-system-running --wait to report running (exit code 0).
Testing across DNF-based templates:
- Validated all DNF-based templates using limactl validate:
  - almalinux-* (8, 9, 10, kitten-10): OK
  - rocky-* (8, 9, 10): OK
  - oraclelinux-* (8, 9, 10): OK
  - centos-stream-* (9, 10): OK
  - fedora-* (41, 42, 43, 44, fedora.yaml): OK
- Distros with systemd-resolved (e.g., Fedora, AlmaLinux 9): systemctl cat systemd-resolved.service returns 0, so mDNS configuration and service restart proceed as before (no behavior change).
- Distros with existing Netfilter modules (e.g., Fedora, AlmaLinux 9): modprobe -n iptable_nat returns 0, so kernel-modules-extra is NOT added to extrapkgs (no unnecessary packages downloaded).
- Distros missing Netfilter modules (e.g., AlmaLinux 10, Rocky 10, Oracle Linux 10, CentOS Stream 10): modprobe -n iptable_nat fails, triggering kernel-modules-extra installation and reloading via 00-modprobe.sh.

i am attaching a screenshot also please check

whoschek · 2026-07-30T06:27:53Z

@Soumya-codr Have you tried adding Almalinux to the CI matrix in .github/workflows/test.yml too?

Soumya-codr · 2026-07-30T06:29:41Z

@Soumya-codr Have you tried adding Almalinux to the CI matrix in .github/workflows/test.yml too?

wait let me see

Soumya-codr · 2026-07-30T06:32:34Z

@whoschek very sorry man i havent tested that let me do that once , apologies from my end 😭

whoschek · 2026-07-30T06:42:22Z

@Soumya-codr No worries. I thank you for picking up the issue and running with it :-)

Soumya-codr · 2026-07-30T07:59:05Z

Investigation Update: `default-buildkit.service` fails on AlmaLinux 10 due to missing `xt_comment` kernel module

@AkihiroSuda @whoschek

While working on adding AlmaLinux 10 support, I discovered a deeper issue that causes default-buildkit.service to consistently fail on RHEL 10-based distros (AlmaLinux 10, and likely Rocky 10, CentOS Stream 10).

Root Cause

BuildKit's CNI bridge plugin (--containerd-worker-net=bridge) requires the xt_comment kernel module for iptables rules with -m comment. On AlmaLinux 10 (kernel 6.12.0), this module is not included in the base kernel ‚Äî it's only available in kernel-modules-extra.

Error from journalctl --user --unit default-buildkit.service:

buildkitd: plugin type="buildkit-cni-bridge" failed (add): running [/sbin/iptables -t nat -A CNI-xxx -d 10.10.0.16/16 -j ACCEPT -m comment --comment name: "buildkit" ...]: exit status 4: 
Warning: Extension comment revision 0 not supported, missing kernel module?
iptables v1.8.11 (nf_tables): RULE_APPEND failed (No such file or directory): rule in chain CNI-xxx

The Existing Fix in `30-install-packages.sh` Doesn't Work

The current modprobe -n check + kernel-modules-extra install logic has two problems:

Problem 1: Version mismatch

dnf install kernel-modules-extra installs the package for the latest available kernel, not the running kernel:

	Version
Running kernel	`6.12.0-211.7.3.el10_2`
`kernel-modules-extra` installed	`6.12.0-211.38.1.el10_2`

The module gets placed in /lib/modules/6.12.0-211.38.1.el10_2/ which is useless for the running kernel (6.12.0-211.7.3).

Fix: Use kernel-modules-extra-$(uname -r) to install for the exact running kernel version:

extrapkgs="${extrapkgs} kernel-modules-extra-$(uname -r)"

Problem 2: Module needs to be loaded after install

Even after installing the correct kernel-modules-extra, the xt_comment module needs to be explicitly loaded via modprobe xt_comment before BuildKit starts. The current boot sequence runs 00-modprobe.sh before 30-install-packages.sh, so modules installed later are never loaded.

Affected Distros

AlmaLinux 10 (confirmed)
Likely: Rocky Linux 10, CentOS Stream 10, Oracle Linux 10 (all RHEL 10-based)
NOT affected: Fedora (has xt_comment in base kernel modules)

Possible Fixes

In 30-install-packages.sh: Change kernel-modules-extra ‚Üí kernel-modules-extra-$(uname -r) and add modprobe xt_comment after install
In 40-install-containerd.sh: Load required modules before starting BuildKit
Alternative: Make BuildKit failure non-fatal during boot (containerd itself works fine)

Happy to submit a PR for whichever approach you prefer! Let me know.

Environment

Host: macOS (Apple Silicon, Virtualization.framework / VZ)
Guest: AlmaLinux 10 GenericCloud (6.12.0-211.7.3.el10_2.aarch64)
Lima: built from main branch

Soumya-codr · 2026-07-30T08:08:21Z

Am i missing out something ?? 👀

AkihiroSuda · 2026-07-30T14:44:06Z

i am attaching a screenshot also please check

My question was whether you tested systemctl is-system-running.
The screenshot seems implying no?

AkihiroSuda added the guest/el10 label Jul 30, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(cidata): resolve AlmaLinux 10 boot failures - #5372

fix(cidata): resolve AlmaLinux 10 boot failures#5372
Soumya-codr wants to merge 1 commit into
lima-vm:masterfrom
Soumya-codr:fix/almalinux-10-ci-5368

Soumya-codr commented Jul 30, 2026

Uh oh!

AkihiroSuda commented Jul 30, 2026

Uh oh!

Soumya-codr commented Jul 30, 2026

Uh oh!

whoschek commented Jul 30, 2026

Uh oh!

Soumya-codr commented Jul 30, 2026

Uh oh!

Soumya-codr commented Jul 30, 2026

Uh oh!

whoschek commented Jul 30, 2026

Uh oh!

Soumya-codr commented Jul 30, 2026 •

edited

Loading

Uh oh!

Soumya-codr commented Jul 30, 2026

Uh oh!

AkihiroSuda commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Soumya-codr commented Jul 30, 2026

What This PR Changes

Linked Issue (Required in most cases)

How I Tested This

Uh oh!

AkihiroSuda commented Jul 30, 2026

Uh oh!

Soumya-codr commented Jul 30, 2026

Uh oh!

whoschek commented Jul 30, 2026

Uh oh!

Soumya-codr commented Jul 30, 2026

Uh oh!

Soumya-codr commented Jul 30, 2026

Uh oh!

whoschek commented Jul 30, 2026

Uh oh!

Soumya-codr commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Investigation Update: default-buildkit.service fails on AlmaLinux 10 due to missing xt_comment kernel module

Root Cause

The Existing Fix in 30-install-packages.sh Doesn't Work

Problem 1: Version mismatch

Problem 2: Module needs to be loaded after install

Affected Distros

Possible Fixes

Environment

Uh oh!

Soumya-codr commented Jul 30, 2026

Uh oh!

AkihiroSuda commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Soumya-codr commented Jul 30, 2026 •

edited

Loading

Investigation Update: `default-buildkit.service` fails on AlmaLinux 10 due to missing `xt_comment` kernel module

The Existing Fix in `30-install-packages.sh` Doesn't Work