Kubernetes - Nabla Containers with CRI-O "no such file or directory"

[joost14]

Hi,

I'm trying to setup runnc in combination with CRI-O and Kubernetes RuntimeClass, but I'm running into some issues.
I'm trying to setup nablacontainers with a nabla image, but when creating the container i get a error.

The image I used:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
      name: nginx
    spec:
      runtimeClassName: runnc
      containers:
       - name: nginx
         image: nablact/nabla-nginx-base
         imagePullPolicy: Always
         ports:
         - containerPort: 8080

And the error I am getting after starting the pod:

  Normal   Pulling    10s (x3 over 32s)  kubelet, k8sworker01  Pulling image "nablact/nabla-nginx-base"
  Normal   Pulled     7s (x3 over 25s)   kubelet, k8sworker01  Successfully pulled image "nablact/nabla-nginx-base"
  Warning  Failed     7s                 kubelet, k8sworker01  Error: container create failed: Unable to copy /var/run/containers/storage/overlay-containers/5cbde56556b9a8765c869a23ef880393c1de347822c50a7ba0301921dbbdb232/userdata/resolv.conf to /var/lib/containers/storage/overlay/0054dfe20729858a903a189026b60165e9e040cb80f1716864aae2b574760399/merged/etc/resolv.conf: open /var/lib/containers/storage/overlay/0054dfe20729858a903a189026b60165e9e040cb80f1716864aae2b574760399/merged/etc/resolv.conf: no such file or directory

Edit:
Same goes for node-express-nabla

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  labels:
    app: nabla
  name: nabla
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nabla
      name: nabla
    spec:
      runtimeClassName: runnc
      containers:
        - name: nabla
          image: nablact/node-express-nabla
          imagePullPolicy: Always
          ports:
          - containerPort: 8080

Error:

Normal   Pulling    8s (x3 over 25s)  kubelet, k8sworker01  Pulling image "nablact/node-express-nabla"
  Normal   Pulled     5s (x3 over 22s)  kubelet, k8sworker01  Successfully pulled image "nablact/node-express-nabla"
  Warning  Failed     5s                kubelet, k8sworker01  Error: container create failed: Unable to copy /var/run/containers/storage/overlay-containers/5f43472fb840f0cd4120a7f752b34065b6a5970a35dc5d707510e410fceb3f7f/userdata/resolv.conf to /var/lib/containers/storage/overlay/0c4ca112b3ebfa8ffcc158ae5f713aa8079042131f80ddeab74bc09d1742ec0a/merged/etc/resolv.conf: open /var/lib/containers/storage/overlay/0c4ca112b3ebfa8ffcc158ae5f713aa8079042131f80ddeab74bc09d1742ec0a/merged/etc/resolv.conf: no such file or directory

Could this be a storage driver issue or a DNS issue (seeing it can't find resolv.conf)?

[lumjjb]

Hi @joost14 , we have not tried running this with cri-o so you are the tester #1 :)

Just checking, what commit are you on? It sounds like it is related to:
https://github.com/nabla-containers/runnc/blob/master/llmodules/fs/iso_storage.go#L45

[InTheCloudDan]

that was my thought too, we should probably do a check on if it's running in Kubernetes. If it is, don't bake those files in but I'm not sure how Kubernetes handles those files in a pod.

[joost14]

HI @lumjjb and @InTheCloudDan, I'm on commit cda4aeb
I used go get github.com/nabla-containers/runnc and make to install runnc.

I don't know if those files have anything to do with Kubernetes..

[joost14]

HI @lumjjb ,

After updating to b1fe2d6 I get the following from kubectl describe pod:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  16s   default-scheduler  Successfully assigned default/nginx-6855cd74d7-zkvc9 to k8sworker01

kubectl logs gives me the following output:

Error from server (BadRequest): container "nginx" in pod "nginx-6855cd74d7-zkvc9" is waiting to start: ContainerCreating

[lumjjb]

Hi @joost14 , curious about the usecase you are looking at here? Could you share a bit about what you're trying with cri-o? We can chat on slack as well. We don't have a lot of expertise around cri-o, but we can chat through the different things to look out for with the runtimes. Do ping us on slack: http://slack.nabla-containers.com