trust_system_cert_pool CloudNativePG PostgreSQL ?

[albundy83]

Hello,

using CloudNativePG operator for my PostgreSQL databases in Kubernetes, I want to connect to PG using teleport.
I have created a simple database in namespace my-db-namespace with the following configuration:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
spec:
  affinity:
    podAntiAffinityType: preferred
  backup:
    ...
  bootstrap:
    initdb:
      dataChecksums: true
      database: my-db
      encoding: UTF8
      localeCType: C
      localeCollate: C
      owner: my-db
  enableSuperuserAccess: false
  imageName: ghcr.io/cloudnative-pg/postgresql:16.6
  instances: 1
  managed:
    roles: # Needed to connect to PG see https://mirakl.tech/mastering-kubernetes-and-database-administration-with-teleport-and-cloudnative-pg-a-step-by-step-f768f1c614de
      - comment: Teleport user with read-only access
        connectionLimit: 20
        ensure: present
        inherit: true
        login: true
        name: teleport_user_read
      - comment: Teleport user with read/write access
        connectionLimit: 20
        ensure: present
        inRoles:
          - my-db
        inherit: true
        login: true
        name: teleport_user_write
  monitoring:
    ...
  postgresGID: 26
  postgresUID: 26
  postgresql:
    parameters:
      ...
    pg_hba:
      - hostssl all /^teleport_user_.*$ 10.42.0.0/16 cert # Needed to allow cert authentication inside Kubernetes / RKE2
  primaryUpdateStrategy: unsupervised
  resources:
    ...
  storage:
    ....

On teleport server, according to doc Use a custom CA, I have created db-client.cas file using command:

tctl auth export --type=db-client --auth-server=teleport.youhou.fr:443 > db-client.cas

Then, I have appended db-client.cas content to Secret my-db-ca inside key ca.crt generated by CloudNativePG during Cluster creation.

Finally, I have deployed teleport kube agent inside the same namespace my-db-namespace with the following configuration:

helm upgrade --install teleport-kube-agent teleport-kube-agent \
  --repo https://charts.releases.teleport.dev \
  --namespace my-db-namespace \
  --version 17.1.2 \
  --set authToken=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
  --set proxyAddr=teleport.youhou.fr:443 \
  --set roles=db \
  --set labels.env=prod \
  --set labels.project=my-db-prod \
  --set labels.teleport_version=17.1.2 \
  --set databases[0].name=my-db \
  --set databases[0].protocol=postgres \
  --set databases[0].uri=my-db-rw:5432 \ # Service name inside Kubernetes
  --set databases[0].tls.mode=verify-full \
  --set databases[0].tls.ca_cert_file=/etc/teleport-tls-db/cnpg/ca.crt \
  --set databases[0].tls.trust_system_cert_pool=false \
  --set extraVolumes[0].name=cnpg \
  --set extraVolumes[0].secret.secretName=my-db-ca \ # With CloudNativePG, Certificate Authority is generated in Secret name <cluster_name>-ca
  --set extraVolumeMounts[0].name=cnpg \
  --set extraVolumeMounts[0].mountPath=/etc/teleport-tls-db/cnpg \
  --set extraVolumeMounts[0].readOnly=true \
  --set rbac.create=false # I already have a teleport kube agent inside this Kubernetes

And it works perfectly:

gbellongervais@localhost:~/work/my-db> tsh db connect my-db --db-user=teleport_user_write --db-name=my-db
psql (17.2, server 16.6 (Debian 16.6-1.pgdg110+1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, compression: off, ALPN: none)
Type "help" for help.

Now, the part I think I did not understand correctly, I hope I could use the feature trust_system_cert_pool.

According to this message, if I update the local system store with the content of Secret my-db-ca on my teleport server, I don't need to mount and use ca_cert_file parameter.

So I collect content of Secret my-db-ca, I add it to local Ubuntu 24.04 system store (in short file /etc/ssl/certs/ca-certificates.crt) where my teleport server (teleport.youhou.fr) is installed and I restart it.
I deploy again my teleport kube agent with the updated parameters:

helm upgrade --install teleport-kube-agent teleport-kube-agent \
  --repo https://charts.releases.teleport.dev \
  --namespace my-db-namespace \
  --version 17.1.2 \
  --set authToken=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
  --set proxyAddr=teleport.youhou.fr:443 \
  --set roles=db \
  --set labels.env=prod \
  --set labels.project=my-db-prod \
  --set labels.teleport_version=17.1.2 \
  --set databases[0].name=my-db \
  --set databases[0].protocol=postgres \
  --set databases[0].uri=my-db-rw:5432 \ # Service name inside Kubernetes
  --set databases[0].tls.mode=verify-full \
  --set databases[0].tls.trust_system_cert_pool=true \ # Now we set it to true
  --set rbac.create=false # I already have a teleport kube agent inside this Kubernetes

But it does not work, I have the following error:

gbellongervais@localhost:~/work/my-db> tsh db connect my-db --db-user=teleport_user_write --db-name=my-db
psql: error: connection to server at "localhost" (::1), port 43415 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
connection to server at "localhost" (127.0.0.1), port 43415 failed: Database service cannot validate database's certificate: failed to connect to `host=my-db-rw user=teleport_user_write database=my-db`: failed to write startup message (tls: failed to verify certificate: x509: certificate signed by unknown authority). Please verify if the correct CA bundle is used in the database config.
ERROR: exit status 2

Any idea ? Did I understand correctly the documentation ?
Thanks for reading me :)

[webvictim]

My understanding is that your teleport-kube-agent would need the my-db-ca added to the system cert pool, as that's where the Teleport db_service is running - inside the Kubernetes pod.

[albundy83]

Ah that's what I was afraid.
So, I think in Kubernetes, this feature is not very useful no ?
I hope I could load the database certificate directly in my teleport server, not in my teleport kube agent.

[benni-as]

@albundy83 We also want the CNPG with teleport connections and are following the same approach via the "Custom CA". However, we are now having trouble with the fact that the CNPG wants to renew the client certs for the pooler and the replication after a while, but then the clientCASecret no longer fits. How do you deal with this?

[albundy83]

Hello,
not sure to understand your needs but I don't use pooler.
Just remember that in teleport kube agent, I mount the main ca as show here:

--set extraVolumes[0].secret.secretName=my-db-ca \ # With CloudNativePG, Certificate Authority is generated in Secret name <cluster_name>-ca

And in this <cluster_name>-ca I have appended the teleport custom ca (output of command tctl auth export --type=db-client --auth-server=example.teleport.sh:443 > db-client.cas)

I don't know if it helps you ...

[benni-as]

Thanks @albundy83 for your explanations. Actually, we are not validating the cluster identity right now, as we trust the k8s network. Because of that we can skip mounting the clusters CA file into the teleport agent pod (with the tls.mode=insecure option).

But my question was actually about what you do when the CNPG operator wants to update the certificates. See https://cloudnative-pg.io/documentation/current/certificates/#operator-managed-mode

I would expect that by default after three months either the cluster replication or the teleport agent integration breaks. Have you already noticed this?

[albundy83]

I think when the certificates are updated, for database, it won't be a problem as certificates are hot reloaded.
For teleport, it is not able to hot reload certificate unfortunately.
But as you don't mount cluster PG ca.crt, I'm not sure to understand your problem .. :)