PV controller never labels a PV provisioned before the Tenant status lists its namespace (no requeue) — PVC bind then rejected forever

[antoinemichea]

Bug description

The capsule/persistentvolumes controller permanently fails to label a PersistentVolume with capsule.clastix.io/tenant when the PV is provisioned shortly after its namespace is created (before the namespace appears in the Tenant's .status.namespaces). Because the reconciler returns without requeueing in that case, and no further events are generated on the PV afterwards, the PV stays unlabeled forever.

The consequence is severe: the pvc.validating.projectcapsule.dev webhook then rejects the PVC↔PV bind indefinitely ("PersistentVolume pvc-… is missing the Tenant label (capsule.clastix.io/tenant), preventing a potential cross-tenant mount"), so the PVC stays Pending forever and the consuming pod is permanently unschedulable — while everything (quota, provisioner, StorageClass) looks healthy. This is easy to hit in practice with operators like CloudNativePG that create PVCs immediately after namespace creation.

Root cause

In internal/controllers/pv/controller.go (v0.13.9, identical on main today):

tnt, err := tenant.GetTenantNameByStatusNamespace(ctx, c.client, persistentVolume.Spec.ClaimRef.Namespace)
...
if tnt == "" {
    log.V(4).Info("skipping reconciliation, PV is claimed by a PVC not managed in a Tenant")
    return reconcile.Result{}, nil   // <-- no requeue
}

GetTenantNameByStatusNamespace (pkg/tenant/get_by.go) does a cached List with the .status.namespaces field index. That status is updated asynchronously by the Tenant reconciler. If the PV's CREATE event is processed before the new namespace shows up in the Tenant status (in the controller's cache), the lookup returns "" and the PV is treated as "not managed in a Tenant" — a permanent conclusion drawn from a transient state.

The only later events on the PV are its phase updates within the first couple of seconds (still inside the race window). After that, nothing re-triggers the reconciler until a full cache resync (~10h by default), and in our observations the PVs were still unlabeled 1h+ later (we labeled them manually to unblock the bind).

Observed timings (production cluster, tenant with ~90 namespaces)

The Tenant reconciler on a busy tenant (many namespaces, recurring optimistic-conflict retries on the ResourceQuotas + client-side throttling) takes ~30 s to reflect a new namespace in .status.namespaces. Measured on 2026-08-04 (Capsule v0.13.9, K8s 1.35, one incident per environment creation burst):

namespace created	PV provisioned	Δ	tenant label
15:40:21	15:40:39	18 s	❌ never applied
16:30:07	16:30:32	25 s	❌ never applied
13:49:33	13:50:07	34 s	✅
14:19:38	14:20:26	48 s	✅
15:55:59	15:56:47	48 s	✅

Every PV created < ~30 s after its namespace lost the race; every PV ≥ ~34 s was labeled fine. The larger the tenant (slower Tenant reconcile), the wider the window.

How to reproduce

1. A Tenant with enough namespaces that its reconcile takes a few tens of seconds (or any transient delay of the Tenant status update / controller cache).
2. Create a new namespace in the tenant and immediately create a PVC in it with any provisioner that binds fast (we reproduced with both Longhorn and local-path; a CloudNativePG Cluster does this naturally: the initdb PVC is created within seconds of the namespace).
3. If the PV is provisioned before the namespace appears in the Tenant .status.namespaces, the PV never gets capsule.clastix.io/tenant, and the PVC stays Pending forever.

Expected behavior

Either the reconciler retries until the tenant can be resolved, or it resolves the tenant from information that is available at namespace-creation time. The PV should eventually be labeled and the bind should succeed.

Logs

kube-controller-manager, looping every ~15 s per stuck PVC:

E0804 16:34:22.675611 1 pv_controller_base.go:266] "Could not sync volume" err="admission webhook \"pvc.validating.projectcapsule.dev\" denied the request: PersistentVolume pvc-0e374224-… is missing the Tenant label (capsule.clastix.io/tenant), preventing a potential cross-tenant mount" PVC="opencell-postman-collection-suite/postgres-1"

The Capsule controller logs nothing for these PVs (the skip is V(4)); its logs do show the Tenant reconcile pressure (client-side throttling, Operation cannot be fulfilled on resourcequotas conflict retries), which is what widens the race window.

Suggested fix — happy to contribute a PR

Two options (can be combined):

1. Fallback resolution via the namespace (preferred — removes the race instead of retrying it): when the status-index lookup returns empty, resolve the tenant from the claimRef namespace's ownerReferences — tenant.GetTenantNameByNamespace already exists in pkg/tenant/get_by.go and is populated at namespace creation, before any Tenant status update.
2. Requeue with backoff when tnt == "" but the claimRef namespace actually exists (e.g. return reconcile.Result{RequeueAfter: …}, nil), so a transient status/cache lag is retried instead of being treated as "not a tenant PV". A guard is needed so genuinely non-tenant PVs don't requeue forever — which is why option 1 alone (or as the primary path) seems cleaner.

I'm happy to submit a PR implementing option 1 (with option 2 as a safety net if you prefer), including a unit test that reproduces the race (PV reconciled before the Tenant status lists the namespace). Let me know which direction you'd prefer and I'll open it.

Additional context

• Capsule version: v0.13.9 (bug still present in internal/controllers/pv/controller.go on main as of 2026-08-04)
• Helm Chart version: capsule 0.13.9
• Kubernetes version: v1.35
• Workaround for stuck volumes: manually kubectl label pv <pv> capsule.clastix.io/tenant=<tenant> — the bind then succeeds on the next pv-controller retry (~15 s).