Skip to content

troubleshooting.adoc

Latest commit

 

History

History
295 lines (209 loc) · 11.2 KB

troubleshooting.adoc

File metadata and controls

295 lines (209 loc) · 11.2 KB
Raw

Troubleshooting

This document provides guidance for diagnosing issues with runtime-enforcer.

Enable verbose logs

To enable more verbose logs in each container via Helm values:

helm upgrade --install runtime-enforcer runtime-enforcer/runtime-enforcer \
  --namespace runtime-enforcer \
  --set agent.logLevel=debug \
  --set controller.logLevel=debug \
  --reuse-values

WorkloadPolicy status failures

Each WorkloadPolicy reports its current deployment status on all targeted nodes in its .status field. Run:

kubectl describe workloadpolicy <name> -n <namespace>

Look for the Status section. It shows the overall phase and a breakdown of nodes:

Status:
  Failed Nodes:         1
  Nodes With Issues:
    worker-3:
      Code:    PolicyFailed
      Message: failed to populate policy for wp default/allow-shell, container app: ...
  Observed Generation:  1
  Phase:                Failed
  Successful Nodes:     2
  Total Nodes:          3

The Phase field can be one of:

  • Ready - the policy is enforced on all targeted nodes

  • Failed - at least one node has a problem

  • Transitioning - some nodes are still switching enforcement mode

The node counters (SuccessfulNodes, FailedNodes, TransitioningNodes) always add up to TotalNodes.

Node issue codes

When a node has a problem, it appears in Nodes With Issues with a Code field. The codes are:

PodNotReady The controller cannot reach the agent pod on this node. The gRPC connection failed or the agent pod is not running. Check if the agent DaemonSet has a pod scheduled on that node and whether it is in Running state.

kubectl get pods -n runtime-enforcer --field-selector spec.nodeName=<node-name>
kubectl describe pod -n runtime-enforcer -l app.kubernetes.io/component=agent --field-selector spec.nodeName=<node-name>

If the agent pod is crash-looping, inspect its logs:

kubectl logs -n runtime-enforcer -l app.kubernetes.io/component=agent --field-selector spec.nodeName=<node-name>

MissingPolicy The agent pod is reachable but the policy is not loaded on it. This typically happens when the policy was just created and the agent on that node has not reconciled yet, or the pod carrying the policy label was not scheduled on that node. If this persists, check the agent logs for reconciliation errors.

PolicyFailed The agent received the policy but failed to apply it. The Message field contains the underlying error, usually a BPF map update failure or an internal resolver error. This means something went wrong inside the agent on that specific node. Check the agent logs on that node for more context:

kubectl logs -n runtime-enforcer -l app.kubernetes.io/component=agent --field-selector spec.nodeName=<node-name>

Look for log lines containing reconcile wp-policy followed by an error. The message in the WorkloadPolicy status is a copy of that error.

Controller-side diagnostics

If the Nodes With Issues section is empty but the phase is still Failed, the status sync controller itself may have encountered an error while collecting node data. The controller runs on a configurable interval and logs every sync attempt. Check its output:

kubectl logs -n runtime-enforcer -l app.kubernetes.io/component=controller

Look for lines from the WorkloadPolicyStatusSync logger. Errors at this level indicate problems listing WorkloadPolicies, connecting to agents, or updating the status subresource.

Quick overview with the kubectl plugin

The kubectl runtime-enforcer policy show protection command lists all workloads, their associated policies, enforcement mode, and current status phase in one table. This is useful when you have multiple policies and want to see which ones are not Ready.

kubectl runtime-enforcer policy show protection

Output:

WORKLOAD                   KIND          POLICY               MODE     STATUS
default/my-app             Deployment    allow-shell          Protect  Ready
default/legacy-app         Deployment    restrict-exec        Monitor  Failed
kube-system/coredns        Deployment    coredns-policy       Protect  Ready

Use the --all-namespaces flag to include policies from every namespace, and -o json for machine-readable output.

Debugger

The debugger is an optional Kubernetes Deployment that helps diagnose issues between the runtime-enforcer agents and the actual state of the Kubernetes cluster.

At the moment, it supports only a few basic operations listed below.

Features

Cache comparison

Each agent maintains an in-memory cache of the pods and containers it is aware of on its node. This cache is populated via the NRI (Node Resource Interface) plugin and is used to resolve container identities during policy enforcement.

If the agent cache drifts from the real cluster state, enforcement decisions could be applied incorrectly so it’s important to validate if the cache is in sync.

The debugger periodically:

  1. Queries every agent (via mTLS-secured gRPC) for its current pod/container cache.

  2. Lists all pods from the Kubernetes API.

  3. Compares the two views node-by-node and prints a diff to stdout.

If the caches are aligned you will see:

caches are aligned

If there is a discrepancy you will see a diff with the affected node and pods, followed by a full dump of the agent cache for that node.

NRI timeouts and required plugins

Runtime Enforcer relies on NRI (Node Resource Interface) integration provided by the container runtime.

Some environments may hit the NRI default request timeout (5s to register a plugin and 2s to handle an event), which can cause the NRI plugin to be detached from the container runtime. This can cause the agent to repeatedly retry connecting to the NRI plugin, resulting in the agent not coming up as expected.

When this happens, you would see runtime-enforcer agent crashing with the following information in its logs:

{"time":"2026-04-24T15:36:07.617127725Z","level":"INFO","msg":"ttrpc server closed 00-runtime-enforcer-agent : ttrpc: server closed","component":"agent","component":"nri-handler","component":"nri-stub"}

You will also see logs like the following in containerd’s logs, indicating that the configured timeout value was exceeded:

Apr 24 15:36:27 kind-control-plane containerd[114]: time="2026-04-24T15:36:27.664534740Z" level=info msg="connection to plugin \"00-runtime-enforcer-agent\" closed"
Apr 24 15:36:27 kind-control-plane containerd[114]: time="2026-04-24T15:36:27.664575617Z" level=info msg="failed to synchronize plugin: context deadline exceeded"
Apr 24 15:39:38 kind-control-plane containerd[114]: time="2026-04-24T15:39:38.782163498Z" level=error msg="closing plugin 00-runtime-enforcer-agent, failed to handle event 6: context deadline exceeded"

To mitigate this, you can increase the NRI timeout on the container runtime side by editing the container runtime configuration file.

containerd: increase NRI timeout

Edit /etc/containerd/config.toml and update the NRI plugin timeouts under [plugins."io.containerd.nri.v1.nri"]:

The example below changes the settings to 10s and 5s.

[plugins."io.containerd.nri.v1.nri"]
  # plugin_registration_timeout is the timeout for a plugin to register after connection.
  plugin_registration_timeout = "10s"
  # plugin_request_timeout is the timeout for a plugin to handle an event/request.
  plugin_request_timeout = "5s"

For details and additional NRI settings, see containerd NRI configuration documentation.

CRI-O: increase NRI timeout

Edit /etc/crio/crio.conf and update the crio.nri table:

The example below changes the setting to 10s and 5s.

[crio.nri]
  # Timeout for a plugin to register itself with NRI.
  nri_plugin_registration_timeout = "10s"
  # Timeout for a plugin to handle an NRI request.
  nri_plugin_request_timeout = "5s"

Required Plugins

Additionally, recent container runtimes provide an option to require a set of NRI plugins to always be present for container creation to succeed.

Follow the steps below to configure this on a per-workload basis. Check NRI docs for more information about the best option to suit your environment.

Prerequisites

The required plugins feature is available since containerd 2.2 and CRI-O 1.34.0.

Workloads

For workloads that you want to ensure have runtime-enforcer protection present, add the required-plugins.noderesource.dev annotation to its pod spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: opensuse-deployment
  labels:
    app: opensuse
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opensuse
  template:
    metadata:
      labels:
        app: opensuse
        security.rancher.io/policy: workloadpolicy-sample
      annotations:
        # This specifies "runtime-enforcer-agent" plugin to be required.
        required-plugins.noderesource.dev: '["runtime-enforcer-agent"]'
    spec:
      containers:
      - name: opensuse
        image: registry.opensuse.org/opensuse/bci/bci-ci:3
        stdin: true
        tty: true

When the required plugin is not available for any reason, the workload will fail to start with errors in Kubernetes events:

4s          Warning   Failed                    pod/opensuse-deployment-6655f5c7ff-xghnd    Error: failed to create containerd container: failed to get NRI adjustment for container: validator "00-default-validator" rejected container adjustment, reason: validation error: required plugin "runtime-enforcer-agent" not present

containerd: required_plugins

In /etc/containerd/config.toml, enable the default_validator:

[plugins."io.containerd.nri.v1.nri".default_validator]
  enable = true

CRI-O: nri_validator_required_plugins

In /etc/crio/crio.conf, configure the crio.nri.default_validator table to enable the default validator:

[crio.nri.default_validator]
  nri_enable_default_validator = true

Enabling the debugger

The debugger is disabled by default. Enable it via Helm values:

helm upgrade --install runtime-enforcer runtime-enforcer/runtime-enforcer \
  --namespace runtime-enforcer \
  --set debugger.enabled=true \
  --reuse-values
Note
 --reuse-values is important because allows you to enable/disable the debugger without restarting the agent.

If you want to customize how often the debugger compares the agent cache against the cluster state, you can set the debugger.interval value in your Helm command:

  --set debugger.interval=<interval> # e.g. 1m, 30s

Viewing debugger output

Once the debugger pod is running, inspect its logs:

kubectl logs -n runtime-enforcer -l app.kubernetes.io/component=debugger -f