kind create cluster failed with mounted containerd config

[PoloDyBala]

---

Bug Report

What happened:

1. Configuration File (config.yaml):

   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   networking:
     ipFamily: ipv4
   nodes:
     - role: control-plane
       image: kindest/node:v1.23.4
       extraPortMappings:
         - containerPort: 65001
           hostPort: 65001
           protocol: TCP
       extraMounts:
         - hostPath: ./test/testdata/containerd/config.toml
           containerPath: /etc/containerd/config.toml
         - hostPath: /tmp/artifact
           containerPath: /tmp/artifact
         - hostPath: /dev/fuse
           containerPath: /dev/fuse

2. Cluster Creation Command:

   kind create cluster --name kind --config test/testdata/kind/config.yaml

   

   Google Document with More Information

3. Docker Information:

   docker info

   Output:

   Client: Docker Engine - Community
   Version:    27.2.0
   Context:    default
   Debug Mode: false
   Plugins:
     buildx: Docker Buildx (Docker Inc.)
       Version:  v0.16.2
       Path:     /usr/libexec/docker/cli-plugins/docker-buildx
     compose: Docker Compose (Docker Inc.)
       Version:  v2.29.2
       Path:     /usr/libexec/docker/cli-plugins/docker-compose
   
   Server:
     Containers: 0
     Running: 0
     Paused: 0
     Stopped: 0
     Images: 1
     Server Version: 27.2.0
     Storage Driver: overlay2
       Backing Filesystem: extfs
       Supports d_type: true
       Using metacopy: false
       Native Overlay Diff: true
       userxattr: false
     Logging Driver: json-file
     Cgroup Driver: systemd
     Cgroup Version: 2
     Plugins:
       Volume: local
       Network: bridge host ipvlan macvlan null overlay
       Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
     Swarm: inactive
     Runtimes: io.containerd.runc.v2 runc
     Default Runtime: runc
     Init Binary: docker-init
     containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
     runc version: v1.1.13-0-g58aa920
     init version: de40ad0
     Security Options:
       apparmor
       seccomp
         Profile: builtin
       cgroupns
     Kernel Version: 5.4.0-192-generic
     Operating System: Ubuntu 20.04.6 LTS
     OSType: linux
     Architecture: x86_64
     CPUs: 2
     Total Memory: 7.603GiB
     Name: iZf8z0l0a1ek5hrud8889tZ
     ID: d5657003-ef5f-4a70-a77b-fd363426c585
     Docker Root Dir: /var/lib/docker
     Debug Mode: false
     Username: polodybala
     Experimental: false
     Insecure Registries:
       127.0.0.0/8
     Registry Mirrors:
       https://docker.1panel.live/
     Live Restore Enabled: false
   
   WARNING: No swap limit support
   

4. Kind Version:

   kind version

   Output:

   kind v0.24.0 go1.18.3 linux/amd64
   

[stmcginnis]

/remove-kind bug
/kind support

You are running a very old version of kind. Please upgrade and try again. If things still fail with a more recent version, then proceed with more troubleshooting:

Please use the --retain argument to keep the failed cluster around, then kind export logs to get all logs. Somewhere in there it should have more details pointing to what actually failed. You can attached the compressed output file to this issue.

Some more helpful things here: https://kind.sigs.k8s.io/docs/user/known-issues/#troubleshooting-kind

I do notice this machine is running cgroupv1. It's highly recommended to switch over to cgroupv2.

[PoloDyBala]

backup.tar.gz

[PoloDyBala]

/remove-kind 错误 /kind 支持

您正在运行一个非常旧版本的 kind。请升级并重试。如果最新版本仍然失败，请继续进行更多故障排除：

请使用参数保留失败的集群，然后获取所有日志。在某个地方，它应该有更多细节指出实际失败的原因。您可以将压缩输出文件附加到此问题。--retain``kind export logs

这里有一些更有用的东西：https://kind.sigs.k8s.io/docs/user/known-issues/#troubleshooting-kind

我确实注意到这台机器正在运行 cgroupv1。强烈建议切换到 cgroupv2。

I have now upgraded the kind version and switched to cgroupv2. The issue still persists. The detailed log files have been uploaded. Thank you for your help.

[aojea]

  - hostPath: ./test/testdata/containerd/config.toml
   containerPath: /etc/containerd/config.toml

try first a kind create cluster without any config, you are modifying critical components configuration that may be the problem, we need to know if kind does not work at all or if it does not work with your config.

[PoloDyBala]

  - hostPath: ./test/testdata/containerd/config.toml
   containerPath: /etc/containerd/config.toml

首先尝试在没有任何配置的情况下，您正在修改可能是问题的关键组件配置，我们需要知道 kind 是否根本不有效，或者它是否不适用于您的配置。kind create cluster
Yes, the kind works without any configuration.

[PoloDyBala]

  - hostPath: ./test/testdata/containerd/config.toml
   containerPath: /etc/containerd/config.toml

首先尝试在没有任何配置的情况下，您正在修改可能是问题的关键组件配置，我们需要知道 kind 是否根本不有效，或者它是否不适用于您的配置。kind create cluster

and the containerd/config.toml is:

# explicitly use v2 config format
version = 2

[debug]
level = "debug"

[plugins."io.containerd.grpc.v1.cri".containerd]
# save disk space when using a single snapshotter
discard_unpacked_layers = false
# explicitly use default snapshotter so we can set it in entrypoint
snapshotter = "overlayfs"
# explicit default here, as we're configuring it below
default_runtime_name = "runc"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
# set default runtime handler to v2, which has a per-pod shim
runtime_type = "io.containerd.runc.v2"

# Setup a runtime with the magic name ("test-handler") used for Kubernetes
# runtime class tests ...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri"]
# use fixed sandbox image
sandbox_image = "registry.k8s.io/pause:3.5"
# allow hugepages controller to be missing
# see https://github.com/containerd/cri/pull/1501
tolerate_missing_hugepages_controller = true
# restrict_oom_score_adj needs to be true when running inside UserNS (rootless)
restrict_oom_score_adj = false

# Mirrors use d7y.io instead of docker.io.
# when using docker.io to generate an endpoint, https://index.docker.io will be returned by default.
# https://github.com/containerd/containerd/blob/main/pkg/cri/server/image_pull.go#L456
# https://github.com/containerd/containerd/blob/main/remotes/docker/resolver.go#L121
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."d7y.io"]
endpoint = ["http://127.0.0.1:65001"]

[PoloDyBala]

  - hostPath: ./test/testdata/containerd/config.toml
   containerPath: /etc/containerd/config.toml

try first a kind create cluster without any config, you are modifying critical components configuration that may be the problem, we need to know if kind does not work at all or if it does not work with your config.

[aojea]

so, it is your containerd config what is causing the problem 😄

[BenTheElder]

It looks like you're just trying to configure a mirror, for a different more future proof and reliable approach to that, see: https://kind.sigs.k8s.io/docs/user/local-registry/

[BenTheElder]

sandbox_image = "registry.k8s.io/pause:3.5"

this is probably the error, and why you should not overwrite the entire containerd config file, the pause version used depends on the node image, as it is pre-loaded into the node image to avoid needing to pull it.

[PoloDyBala]

sandbox_image = "registry.k8s.io/pause:3.5"

this is probably the error, and why you should not overwrite the entire containerd config file, the pause version used depends on the node image, as it is pre-loaded into the node image to avoid needing to pull it.

By commenting: sandbox_image="registry. k8s. io/page: 3.5", the above problem still occurred

[BenTheElder]

I recommend #3733 (comment) instead of overriding the entire file

[BenTheElder]

Overriding the entire containerd config is likely to break things again in the future. See the suggestion above.