Networking failure in worker

[lenianiva]

I'm making a concourse module for NixOS. Currently the worker node fails to start, showing

concourse[579]: error: containerd garden server runner: failed to create CNI network opts: container MTU: route ip+net: netlinkrib: address family not supported by protocol

What could be causing this issue? I provided it with these variables:

CONCOURSE_RUNTIME=containerd
CONCOURSE_CONTAINERD_BIN=(path to containerd binary)

I also tried using guardian, and got the same error:

concourse[632]: {"timestamp":"2025-09-06T01:22:26.606249816Z","level":"error","source":"guardian","message":"guardian.failed-to-wire-networker","data":{"error":"route ip+net: netlinkrib: address family not supported by protocol"}}

Many thanks!

The system log around the message:

worker # [    7.317787] concourse[576]: {"timestamp":"2025-09-06T01:51:01.454401467Z","level":"info","source":"baggageclaim","message":"baggageclaim.using-driver","data":{"driver":"overlay"}}
worker # [    7.353874] concourse[576]: {"timestamp":"2025-09-06T01:51:01.491781357Z","level":"info","source":"baggageclaim","message":"baggageclaim.listening","data":{"addr":"127.0.0.1:7788"}}
server # [   26.089210] concourse[663]: {"timestamp":"2025-09-06T01:51:01.196155235Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.forwarded-tcpip","data":{"bind-addr":"0.0.0.0:7777","bound-port":39951,"command":"forward-worker","remote":"10.0.0.10:37102","session":"1.4.1"}}
server # [   26.095719] concourse[663]: {"timestamp":"2025-09-06T01:51:01.196272009Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.forwarded-tcpip","data":{"bind-addr":"0.0.0.0:7788","bound-port":36913,"command":"forward-worker","remote":"10.0.0.10:37102","session":"1.4.1"}}
server # [   26.101238] concourse[663]: {"timestamp":"2025-09-06T01:51:01.203196074Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.start","data":{"command":"forward-worker","remote":"10.0.0.10:37102","session":"1.4.1"}}
server # [   26.106340] concourse[663]: {"timestamp":"2025-09-06T01:51:01.203261725Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.register.start","data":{"command":"forward-worker","remote":"10.0.0.10:37102","session":"1.4.1.2","worker-address":"127.0.0.1:39951","worker-platform":"linux","worker-tags":"worker1"}}
worker # [    7.422602] concourse[576]: {"timestamp":"2025-09-06T01:51:01.560509645Z","level":"error","source":"worker","message":"worker.beacon-runner.beacon.forward-conn.failed-to-dial","data":{"addr":"127.0.0.1:7777","error":"dial tcp 127.0.0.1:7777: connect: connection refused","network":"tcp","session":"4.1.4"}}
worker # [    7.478288] concourse[620]: {"timestamp":"2025-09-06T01:51:01.615052331Z","level":"info","source":"guardian","message":"guardian.no-port-pool-state-to-recover-starting-clean","data":{}}
worker # [    7.482102] concourse[620]: {"timestamp":"2025-09-06T01:51:01.619965805Z","level":"error","source":"guardian","message":"guardian.failed-to-wire-networker","data":{"error":"route ip+net: netlinkrib: address family not supported by protocol"}}
worker # [    7.484877] concourse[620]: route ip+net: netlinkrib: address family not supported by protocol
worker # [    7.486173] concourse[620]: route ip+net: netlinkrib: address family not supported by protocol
worker # [    7.489707] concourse[576]: {"timestamp":"2025-09-06T01:51:01.626699063Z","level":"error","source":"worker","message":"worker.garden.gdn-runner.logging-runner-exited","data":{"error":"exit status 1","session":"1.2"}}
worker # [    7.492519] concourse[576]: {"timestamp":"2025-09-06T01:51:01.630449819Z","level":"error","source":"worker","message":"worker.garden-runner.logging-runner-exited","data":{"error":"Exit trace for group:\ngdn exited with error: exit status 1\n","session":"8"}}
worker # [    7.496066] concourse[576]: {"timestamp":"2025-09-06T01:51:01.633717832Z","level":"info","source":"worker","message":"worker.debug-runner.logging-runner-exited","data":{"session":"10"}}
worker # [    7.498080] concourse[576]: {"timestamp":"2025-09-06T01:51:01.633846899Z","level":"info","source":"worker","message":"worker.healthcheck-runner.logging-runner-exited","data":{"session":"11"}}
worker # [    7.500248] concourse[576]: {"timestamp":"2025-09-06T01:51:01.633942721Z","level":"info","source":"worker","message":"worker.container-sweeper.sweep-cancelled-by-signal","data":{"session":"6","signal":2}}
worker # [    7.502658] concourse[576]: {"timestamp":"2025-09-06T01:51:01.638182087Z","level":"info","source":"worker","message":"worker.container-sweeper.logging-runner-exited","data":{"session":"13"}}
worker # [    7.504879] concourse[576]: {"timestamp":"2025-09-06T01:51:01.640494392Z","level":"info","source":"worker","message":"worker.volume-sweeper.sweep-cancelled-by-signal","data":{"session":"7","signal":2}}
worker # [    7.507348] concourse[576]: {"timestamp":"2025-09-06T01:51:01.640525401Z","level":"info","source":"worker","message":"worker.volume-sweeper.logging-runner-exited","data":{"session":"14"}}
worker # [    7.509786] concourse[576]: {"timestamp":"2025-09-06T01:51:01.645207281Z","level":"info","source":"worker","message":"worker.beacon-runner.beacon.forward-conn.cancelled","data":{"addr":"127.0.0.1:7777","network":"tcp","session":"4.1.4"}}
worker # [    7.512521] concourse[576]: {"timestamp":"2025-09-06T01:51:01.647557580Z","level":"info","source":"worker","message":"worker.beacon-runner.beacon.run.context-done","data":{"command":"forward-worker","context-error":{},"session":"4.1.3"}}
server # [   26.206992] concourse[663]: {"timestamp":"2025-09-06T01:51:01.314379758Z","level":"error","source":"tsa","message":"tsa.connection.channel.command.register.failed-to-fetch-containers","data":{"command":"forward-worker","error":"Get \"http://127.0.0.1:39951/containers\": EOF","remote":"10.0.0.10:37102","session":"1.4.1.2"}}
worker # [    7.514907] concourse[576]: {"timestamp":"2025-09-06T01:51:01.647663180Z","level":"info","source":"worker","message":"worker.baggageclaim-runner.logging-runner-exited","data":{"session":"9"}}
worker # [    7.516954] concourse[576]: {"timestamp":"2025-09-06T01:51:01.650370228Z","level":"info","source":"worker","message":"worker.beacon-runner.beacon.run.signal-sent","data":{"command":"forward-worker","session":"4.1.3"}}
worker # [    7.519833] concourse[576]: {"timestamp":"2025-09-06T01:51:01.657801619Z","level":"error","source":"worker","message":"worker.beacon-runner.beacon.forward-conn.failed-to-dial","data":{"addr":"127.0.0.1:7788","error":"dial tcp 127.0.0.1:7788: connect: connection refused","network":"tcp","session":"4.1.5"}}
worker # [    7.523355] concourse[576]: {"timestamp":"2025-09-06T01:51:01.661123829Z","level":"info","source":"worker","message":"worker.beacon-runner.beacon.forward-conn.cancelled","data":{"addr":"127.0.0.1:7788","network":"tcp","session":"4.1.5"}}
server # [   26.220564] concourse[663]: {"timestamp":"2025-09-06T01:51:01.327685359Z","level":"error","source":"tsa","message":"tsa.connection.channel.command.register.failed-to-list-volumes","data":{"command":"forward-worker","error":"Get \"http://127.0.0.1:36913/volumes\": EOF","remote":"10.0.0.10:37102","session":"1.4.1.2"}}
server # [   26.226285] concourse[663]: {"timestamp":"2025-09-06T01:51:01.327746540Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.register.failed-to-reach-worker","data":{"baggageclaim-took":"13.232687ms","command":"forward-worker","garden-took":"111.200446ms","remote":"10.0.0.10:37102","session":"1.4.1.2"}}
server # [   26.232278] concourse[663]: {"timestamp":"2025-09-06T01:51:01.327774197Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.register.done","data":{"command":"forward-worker","remote":"10.0.0.10:37102","session":"1.4.1.2","worker-address":"127.0.0.1:39951","worker-platform":"linux","worker-tags":"worker1"}}
server # [   26.238327] concourse[663]: {"timestamp":"2025-09-06T01:51:01.327812750Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.done","data":{"command":"forward-worker","remote":"10.0.0.10:37102","session":"1.4.1"}}
worker # [    7.546915] concourse[576]: {"timestamp":"2025-09-06T01:51:01.684744150Z","level":"info","source":"worker","message":"worker.beacon-runner.beacon.exited","data":{"session":"4.1"}}
worker # [    7.548887] concourse[576]: {"timestamp":"2025-09-06T01:51:01.684827959Z","level":"info","source":"worker","message":"worker.beacon-runner.logging-runner-exited","data":{"session":"12"}}
server # [   26.242705] concourse[663]: {"timestamp":"2025-09-06T01:51:01.327838451Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.draining-forwarded-connections","data":{"command":"forward-worker","remote":"10.0.0.10:37102","session":"1.4.1"}}
worker # [    7.551044] concourse[576]: error: Exit trace for group:
worker # [    7.551927] concourse[576]: garden exited with error: Exit trace for group:
worker # [    7.552830] concourse[576]: gdn exited with error: exit status 1
worker # [    7.553596] concourse[576]: debug exited with nil
worker # [    7.555160] concourse[576]: healthcheck exited with nil
worker # [    7.555836] concourse[576]: container-sweeper exited with nil
server # [   26.247789] concourse[663]: {"timestamp":"2025-09-06T01:51:01.327952432Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.forward-process-exited","data":{"bind-addr":"0.0.0.0:7777","bound-port":39951,"command":"forward-worker","remote":"10.0.0.10:37102","session":"1.4.1"}}
worker # [    7.556584] concourse[576]: volume-sweeper exited with nil
worker # [    7.557406] concourse[576]: baggageclaim exited with nil
worker # [    7.558104] concourse[576]: beacon exited with nil
server # [   26.253207] concourse[663]: {"timestamp":"2025-09-06T01:51:01.345428397Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.forward-process-exited","data":{"bind-addr":"0.0.0.0:7788","bound-port":36913,"command":"forward-worker","remote":"10.0.0.10:37102","session":"1.4.1"}}

[lenianiva]

This was because of systemd sandboxing.

[taylorsilva]

Been going through my notifications and reading your posts in reverse order. Hope your concourse + nixos journey is going well! Feel free to jump on the Discord, can usually get a faster reply that way.

[lenianiva]

Been going through my notifications and reading your posts in reverse order. Hope your concourse + nixos journey is going well! Feel free to jump on the Discord, can usually get a faster reply that way.

All issues are solved now (i.e. can run a pipeline and get results). Thanks for the work on this application! The Concourse CI service on NixOS is ready to go and I'll write a tutorial about it.

[taylorsilva]

Looking forward to reading it! Please share it here in the show and tell section https://github.com/orgs/concourse/discussions/categories/show-and-tell