How to enforce host apparmor profiles on kind?

[mbana]

I would like to apply the /etc/apparmor.d/enroot that is found on the kind host to the worker nodes. I can see that from a worker node that /sys/module/apparmor/parameters/enabled is set to Y but I get an error indicating otherwise.

I do not want to build a new kind node.

I could try with kindest/node:v1.30.0@sha256:047357ac0cfea04663786a612ba1eaba9702bef25227a794b52890dd8bcd692e but I doubt this will make a difference?

host

$ cat /sys/module/apparmor/parameters/enabled
Y
$ systemctl status apparmor              
● apparmor.service - Load AppArmor profiles
     Loaded: loaded (/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled)
     Active: active (exited) since Fri 2024-08-09 15:21:30 UTC; 2h 38min ago
       Docs: man:apparmor(7)
             https://gitlab.com/apparmor/apparmor/wikis/home/
    Process: 683 ExecStart=/lib/apparmor/apparmor.systemd reload (code=exited, status=0/SUCCESS)
   Main PID: 683 (code=exited, status=0/SUCCESS)
        CPU: 216ms

Warning: some journal files were not opened due to insufficient permissions.
$ sudo cat /etc/apparmor.d/enroot
#include <tunables/global>

profile enroot flags=(attach_disconnected,mediate_deleted) {

  #include <abstractions/base>

  # added
  mount,
  pivot_root,

  # defaults
  network,
  capability,
  file,
  umount,

  deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)
  # deny write to files not in /proc/<number>/** or /proc/sys/**
  deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9/]*}/** w,
  deny @{PROC}/sys/[^k]** w,  # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
  deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,  # deny everything except shm* in /proc/sys/kernel/
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/kcore rwklx,

  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/** rwklx,
  deny /sys/kernel/security/** rwklx,

}

Worker Node(s)

$ docker exec -it kind-worker3 bash 
root@kind-worker3:/# systemctl status apprmor
Unit apprmor.service could not be found.
root@kind-worker3:/# cat /sys/module/apparmor/parameters/enabled
Y
root@kind-worker3:/# ls -lah /etc/apparmor.d/
ls: cannot access '/etc/apparmor.d/': No such file or directory

Error

$ kubectl get --namespace slurm pod/slurm-cpu-epyc-000-008 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    container.apparmor.security.beta.kubernetes.io/slurmd: localhost/enroot
...
  message: 'Cannot enforce AppArmor: AppArmor is not enabled on the host'
  phase: Pending
  qosClass: Burstable
  reason: AppArmor
  startTime: "2024-08-09T16:39:58Z"

Or from k9s:

│ Status:            Pending                                                                                                                                                                                    │
│ Reason:            AppArmor                                                                                                                                                                                   │
│ Message:           Cannot enforce AppArmor: AppArmor is not enabled on the host                                                                                                                               │
│ SeccompProfile:    Localhost                                                                                                                                                                                  │
│ LocalhostProfile:  profiles/enroot

Version

kind: kind v0.23.0 go1.22.3 linux/amd6
Node: kindest/node:v1.24.17@sha256:bad10f9b98d54586cba05a7eaa1b61c6b90bfc4ee174fdc43a7b75ca75c95e51

[stmcginnis]

Note, you have a typo here:

root@kind-worker3:/# systemctl status apprmor
Unit apprmor.service could not be found.

Just a hunch, but have you tried adding an extra mount to pull in the host's /etc/apparmor.d/ to the container?

[mbana]

Note, you have a typo here:

root@kind-worker3:/# systemctl status apprmor
Unit apprmor.service could not be found.

Just a hunch, but have you tried adding an extra mount to pull in the host's /etc/apparmor.d/ to the container?

Good catch! It's still not found though:

$ docker exec -it kind-worker3 bash 
root@kind-worker3:/# sudo systemctl status apparmor
bash: sudo: command not found
root@kind-worker3:/# systemctl status apparmor
Unit apparmor.service could not be found.

Are you suggesting that I do something like:

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
  - role: control-plane
    extraMounts:
      - hostPath: /etc/apparmor.d/
        containerPath: /etc/apparmor.d/

Thanks for the reply, BTW.

[stmcginnis]

Disclaimer: I have not done this!

Part of it may be needing to add the mount, but from what you've seen, that apparmor service is not installed into the base image. So maybe someone has a workaround to avoid it, but I think you may be better off building your own node image to include what you need here.

[mbana]

Thanks. Mounting did add the profiles, but I still the error message:

$ docker exec -it kind-worker3 bash 
root@kind-worker3:/# ls -lah /etc/apparmor.d/
total 140K
drwxr-xr-x 8 root root 4.0K Aug  9 15:18 .
drwxr-xr-x 1 root root 4.0K Aug  9 18:39 ..
drwxr-xr-x 2 root root 4.0K Apr  2 16:58 abi
drwxr-xr-x 4 root root  12K Apr  2 16:58 abstractions
drwxr-xr-x 2 root root 4.0K Apr 23  2022 disable
-rw------- 1 root root  941 Aug  9 15:19 enroot
drwxr-xr-x 2 root root 4.0K Aug  8 10:16 force-complain
-rw-r--r-- 1 root root  896 Nov  5  2021 lightdm-guest-session
drwxr-xr-x 2 root root 4.0K Aug  8 10:16 local
-rw-r--r-- 1 root root 1.4K Mar 10  2022 lsb_release
-rw-r--r-- 1 root root 1.2K Mar 10  2022 nvidia_modprobe
-rw-r--r-- 1 root root 3.5K Jan 31  2023 sbin.dhclient
drwxr-xr-x 5 root root 4.0K Apr  2 16:58 tunables
-rw-r--r-- 1 root root 1.5K Jun  7 17:52 ubuntu_pro_apt_news
-rw-r--r-- 1 root root 6.8K Jun  7 17:52 ubuntu_pro_esm_cache
-rw-r--r-- 1 root root 3.4K Mar 17  2022 usr.bin.man
-rw-r--r-- 1 root root 1.7K Feb  8  2024 usr.bin.tcpdump
-rw-r--r-- 1 root root  29K Jul 26 02:57 usr.lib.snapd.snap-confine.real
-rw-r--r-- 1 root root  677 Apr  6  2022 usr.sbin.cups-browsed
-rw-r--r-- 1 root root 6.0K Feb 14  2023 usr.sbin.cupsd
-rw-r--r-- 1 root root 2.4K Feb 16  2022 usr.sbin.ntpd
-rw-r--r-- 1 root root 1.6K Nov 16  2021 usr.sbin.rsyslogd
-rw-r--r-- 1 root root 1.4K Jun  7 14:54 usr.sbin.sssd

│ Status:            Pending                                                                                                                                                                                    │
│ Reason:            AppArmor                                                                                                                                                                                   │
│ Message:           Cannot enforce AppArmor: AppArmor is not enabled on the host                                                                                                                               │
│ SeccompProfile:    Localhost                                                                                                                                                                                  │
│ LocalhostProfile:  profiles/enroot 

I will have a look at building an image.

The only confusing thing is that /sys/module/apparmor/parameters/enabled returns Y yet one gets the error when they apply the apparmor annotation.

[BenTheElder]

We don't support this, apparmor doesn't mix well with nested containers, which to be fair are a hack that generally isn't expected to be supported by other tools.

https://kind.sigs.k8s.io/docs/user/known-issues/#apparmor

kind nodes ask docker to disable apparmor, because otherwise a lot of things would break, the host apparmor profiles would get applied to the nested kubernetes under the node containers and break things because the profiles are designed to run against a single non-nested container or only the host.

I would recommend developing apparmor roles only on a VM closer to your target cluster.

kind/pkg/cluster/internal/providers/nerdctl/provision.go

Line 197 in 00d659b

"--security-opt", "apparmor=unconfined", // also ignore apparmor