Kubernetes ManageIQ - Postgresql not working

[ZGGrOydHRyl]

I'm getting the error mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied when trying to run postgresql.

Any idea how to fix this?

[ec2-user@ip-123 manageiq-operator]$ kubectl get pods
NAME                                 READY   STATUS                       RESTARTS     AGE
httpd-795f9ffd95-zjfld               0/1     CreateContainerConfigError                      0            77m
manageiq-operator-749867b9f9-nfwrq   1/1     Running                      0            77m
memcached-7ff46dd675-6sq4q           1/1     Running                      0            77m
postgresql-5db7cc5767-x2g7k          0/1     CrashLoopBackOff             1 (5s ago)   7s

[ec2-user@ip-123 manageiq-operator]$ kubectl logs postgresql-5db7cc5767-x2g7k
mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied

Also HTTPD container is not working. CreateContainerConfigError is the error as shown above. Any idea how to resolve both errors.

kubectl describe pod httpd-795f9ffd95-mn4x7 -n manageiq
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  16s                default-scheduler  Successfully assigned manageiq/httpd-795f9ffd95-mn4x7 to ip-192-168-12-189.eu-central-1.compute.internal
  Normal   Pulling    15s                kubelet            Pulling image "manageiq/httpd:2-el9"
  Normal   Pulled     12s                kubelet            Successfully pulled image "manageiq/httpd:2-el9" in 3.49s (3.49s including waiting)
  Warning  Failed     11s (x2 over 12s)  kubelet            Error: container has runAsNonRoot and image will run as root (pod: "httpd-795f9ffd95-mn4x7_manageiq(8834d45f-7e06-4f81-b7b5-b62e2dcc1b5e)", container: httpd)
  Normal   Pulled     11s                kubelet            Container image "manageiq/httpd:2-el9" already present on machine
[ec2-user@ip-123 ~]$

[Fryguy]

@ZGGrOydHRyl What flavor of Kubernetes are you running. We are seeing similar reports for people running on AKS and another on K3S as it seems some default policies have changed.

@bdunne Can you take a look here?

[ZGGrOydHRyl]

@Fryguy I'm using AWS EKS for kubernetes

[bdunne]

What type of storage are you using? And what are the permissions on the volume? It looks like we don't have write permission.

[ZGGrOydHRyl]

@bdunne I didn't create this PVC and SC. It was created by manageiq. I used the github repo and ran kubectl apply -f config/. BTW I had to create and attach some roles related to EBS CSI Driver. I don't know if something is wrong there or if the problem with manageIQ deployment.

aws iam create-role --role-name AmazonEKS_EBS_CSI_DriverRole --assume-role-policy-document file://"trust-policy.json"
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy --role-name AmazonEKS_EBS_CSI_DriverRole

eksctl create addon --name aws-ebs-csi-driver --cluster demo-eks-cluster --region eu-central-1 --service-account-role-arn arn:aws:iam::YOUR_ACCOUTN_ID:role/AmazonEKS_EBS_CSI_DriverRole --force

After running the above commands, the PVC started working but postgresql showed the error Permission denied

kubectl get sc -n manageiq

[ec2-user@ip-10-15-1-155 ~]$ kubectl get sc -n manageiq
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  2d1h

kubectl get pvc -n manageiq

[ec2-user@ip-10-15-1-155 ~]$ kubectl get pvc -n manageiq
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
postgresql   Bound    pvc-bfda17de-b1a1-4b8b-9b6c-3840d8e8e231   15Gi       RWO            gp2            <unset>                 2d1h

kubectl edit sc gp2 -n manageiq

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"creationTimestamp":"2024-06-30T11:05:17Z","name":"gp2","resourceVersion":"284","uid":"4c94d685-1416-4fe9-b547-381ef8e25fd2"},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2024-06-30T12:16:54Z"
  name: gp2
  resourceVersion: "13936"
  uid: e70990e2-4437-4c93-a361-03c78692535f
parameters:
  fsType: ext4
  type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

kubectl edit pvc postgresql -n manageiq

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
    volume.kubernetes.io/selected-node: ip-192-168-34-13.eu-central-1.compute.internal
    volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
  creationTimestamp: "2024-06-30T11:40:30Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: manageiq
    manageiq.org/backup: t
  name: postgresql
  namespace: manageiq
  ownerReferences:
  - apiVersion: manageiq.org/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ManageIQ
    name: manageiq-sample
    uid: 7a45955f-6ee4-49aa-874d-29afc5be9684
  resourceVersion: "16133"
  uid: bfda17de-b1a1-4b8b-9b6c-3840d8e8e231
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 15Gi
  storageClassName: gp2
  volumeMode: Filesystem
  volumeName: pvc-bfda17de-b1a1-4b8b-9b6c-3840d8e8e231
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 15Gi
  phase: Bound

[bdunne]

I'm not familiar with the EBS CSI Driver, but if you're not using the default StorageClass, you can specify the storage class that ManageIQ should use in the CR spec key storageClassName.

[ZGGrOydHRyl]

@Fryguy how can I edit manageiq deployment or scalefulsets? When ever I use kubectl edit deployment manageiq and save the changes, it gets reverted back to original state. I can't modify anything (sc, pvc, deployment, scalefulsets etc.,) in the manageiq namespace.

[bdunne]

The operator owns the deployments and other resources and may overwrite your changes. What needs to be manually changed?

[ZGGrOydHRyl]

@bdunne just want a way to modify whatever I need for testing purposes. Thanks for the tip, I disabled manageiq-operator deployment (replica count to 0) and then I was able to make changes .

kubectl -n manageiq scale deploy manageiq-operator --replicas=0

How can I make a change permanent when using operator? Does operator uses yaml which can be modified? How does operator work?

[ZGGrOydHRyl]

@Fryguy @bdunne any update? I'm still stuck

[bdunne]

chmod -R 777 /var/lib/pgsql/data would fix the permission problem.