🐢 k8s-management

Bootstrap Kubernetes clusters with Cluster API, Talos and Proxmox.

This repo sets up a k3s management cluster, installs Cluster API and performs bootstrapping of Talos workload clusters on top of Proxmox infrastructure.

The workload cluster is bootstrapped by Flux using the k8s-infrastructure repository.

Alternative installation flavours

• k8s-bootstrap: Kubeadm based installation.
• k8s-bootstrap-talos: Talos based installation.

Management cluster

Install

Install the k3s management cluster:

sudo bash scripts/cluster.sh

Credentials for the management cluster will be available at the /etc/rancher/k3s/k3s.yaml file.

Bootstrap

To install Cluster API and bootstrap a workload cluster, set the GITHUB_TOKEN environment variable and run:

sudo \
GITHUB_TOKEN="$GITHUB_TOKEN" \
bash scripts/bootstrap.sh

Configure access to the workload cluster

To configure access to the workload cluster, set the WORKLOAD_CLUSTER environment variable and run:

WORKLOAD_CLUSTER="$WORKLOAD_CLUSTER" \
bash scripts/workload-cluster-credentials.sh

Credentials will be available in the kubeconfig and talosconfig files in the current directory. Set the following aliases to temporarily use the workload cluster:

alias k="kubectl --kubeconfig=kubeconfig"
alias t="talosctl --talosconfig=talosconfig"

Bootstrap workload cluster

To deploy the infrastructure components in the workload cluster, set the GITHUB_TOKEN environment variable and run following script to bootstrap the k8s-infrastructure repository using flux:

sudo \
GITHUB_TOKEN="$GITHUB_TOKEN" \
bash scripts/workload-cluster-bootstrap.sh

Upgrade

To upgrade the management cluster, set the K3S_VERSION environment variable to the desired version and run:

sudo \
K3S_VERSION="v1.33.1+k3s1" \
bash scripts/cluster.sh

Backup

To take an on-demand backup of the management cluster and push it to object storage, run:

sudo \
S3_ACCESS_KEY="<access-key>" \
S3_SECRET_KEY="<secret-key>" \
bash scripts/backup.sh

Restore

To restore a backup in a existing cluster, select the backup to restore with SNAPSHOT_NAME and and run:

sudo \
S3_ACCESS_KEY="<access-key>" \
S3_SECRET_KEY="<secret-key>" \
SNAPSHOT_NAME="on-demand-management-1749986019" \
bash scripts/restore.sh

Uninstall

To uninstall the k3s management cluster, run:

k3s-uninstall.sh

Workload cluster

Template immutability

ProxmoxMachineTemplate and TalosConfigTemplate resources are entirely immutable. Any change to their spec (disk size, RAM, CPU, strategic patches, etc.) will be rejected by the webhook:

ProxmoxMachineTemplate.infrastructure.cluster.x-k8s.io "compute-large" is invalid: spec: Forbidden: ProxmoxMachineTemplate is immutable

To update a template, use one of these approaches:

A) New name (safer, zero-downtime rollout):

1. Create a new template (e.g. compute-large-v2) with the updated spec
2. Update the infrastructureRef.name (for ProxmoxMachineTemplate) and/or bootstrap.configRef.name (for TalosConfigTemplate) in the affected MachineDeployment resources
3. CAPI will rollout new machines using the new template and delete the old ones
4. Delete the old template after the rollout completes

B) Same name (simpler, brief reconciliation gap):

1. Delete the old template from Git
2. Flux deletes it from the cluster
3. Add the new template with the same name
4. Flux creates it — no MachineDeployment changes needed

Approach B works because MachineDeployment references templates by name, not UID. The reconciliation gap is brief (seconds to minutes) and the MachineDeployment will pick up the new template automatically.

Reference

• Blog post from @qjoly.
• cluster-api-operator
• cluster-api
• cluster-api-ipam-provider-in-cluster
• cluster-api-provider-proxmox
• cluster-api-control-plane-provider-talos
• cluster-api-bootstrap-provider-talos