Skip to content

container plugin: CRI event listener never re-subscribes after the runtime restarts #1499

Description

@skyf0l

Describe the bug

The CRI engine subscribes once to GetContainerEvents in Listen() (go-worker/pkg/container/cri.go). When containerd restarts the stream ends, the producer goroutine closes containerEventsCh, the consumer returns and closes outCh, and workerLoop in worker.go just drops the case (// Remove the stopped goroutine). There is no retry, so from then on the plugin never learns about new containers through the create/start hooks.

The fallback in fetcher.go does not save it either: it retries get() for 150 ms total, and once it gives up on a container id it never asks again. So every container created after the runtime restart stays <NA> for its whole life, not just short-lived ones. On my cluster stalwart-pg-1 has been running for a month and still logs container.name=<NA>.

How to reproduce it

  1. Falco 0.44.1 with container plugin 0.7.1, engines.cri enabled on a containerd socket.
  2. Restart containerd on the host (make sure the socket path is still reachable from Falco, for example by mounting its parent directory; with the current Helm chart the mount itself also breaks, see Container plugin sockets are mounted as files again, Falco loses containerd after a runtime restart (regression of #632) charts#1052).
  3. Start a new container and trigger any rule. container.name, container.image.repository, k8s.ns.name, k8s.pod.name are all <NA>. Containers created before the restart are still enriched.

Expected behaviour

Listen() should reconnect with a backoff when the event stream ends, and re-list running containers after reconnecting so nothing created during the gap is missed. It would also help if a failed fetch was not final, so a later lookup for the same container id gets another chance.

Additional context

Runtime restarts are normal on long-lived nodes (package upgrades with needrestart, containerd crashes, RKE2/k3s upgrades). Falco itself stays healthy and keeps emitting, so this failure is silent. The only signal I found is falcosecurity_falcosidekick_falco_events_total{k8s_ns_name=""} climbing for container-only rules.

Falco 0.44.1, container plugin 0.7.1, RKE2 v1.36.1 (containerd v2.2.3-k3s1), Helm chart 9.1.0. Same code on main as of today.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions