customResourceState registers the metric family and then serves nothing

[vde-dis]

Summary

With a Custom Resource State config, KSM v2.13.0 logs that it resolved the
plurals and added the metric family, and then serves zero series for it. No
error is emitted at any log level. The same config produced series exactly once,
and the only variable I could correlate it with was the Helm release name, which
should be irrelevant — so I am probably misreading my own reproduction and would
like help narrowing it.

The silence is the reason this is worth reporting: a config that is wrong in a
way KSM detects fails loudly, but this shape reports success at every step.

Environment

kube-state-metrics	v2.13.0
chart	prometheus-community/kube-state-metrics 5.25.1
Kubernetes	v1.35.3 (Talos)
Custom resources	Flux kustomize.toolkit.fluxcd.io/v1 Kustomization etc.

Config

Taken from fluxcd/flux2-monitoring-example
(monitoring/controllers/kube-prometheus-stack/kube-state-metrics-config.yaml),
used verbatim in the reproduction below.

What KSM logs — every time, including when it serves nothing

config.go:84  "Using custom resource plural" resource="kustomize.toolkit.fluxcd.io_v1_Kustomization" plural="kustomizations"
discovery.go:274 "discovery finished, cache updated"
custom_resource_metrics.go:79 "Custom resource state added metrics" familyNames=["gotk_resource_info"]

Reproduction

1. Install Flux; create one object so there is something to report:
   a Kustomization referencing a non-existent GitRepository (it goes
   Ready=False, which is what the config's ready label reads).
2. helm install kube-state-metrics prometheus-community/kube-state-metrics --version 5.25.1 -f <upstream file, kube-state-metrics: key unwrapped>
3. kubectl exec … -- wget -qO- localhost:8080/metrics | grep gotk_

Observed: one series, correct labels —
gotk_resource_info{customresource_kind="Kustomization",exported_namespace="flux-system",name="probe-broken",ready="False"} 1

4. Now install the SAME values under a different release name
   (kube-state-metrics-flux), everything else identical, same cluster, same
   Flux object.

Observed: zero series. Logs identical to step 3, including
familyNames=["gotk_resource_info"]. Checked at 35 s and again at 4 min 26 s.
RBAC verified with kubectl auth can-i list kustomizations.kustomize.toolkit.fluxcd.io --as=system:serviceaccount:<ns>:kube-state-metrics-flux
→ yes. Tried with and without collectors: [], and with
--custom-resource-state-only=true present in both.

What I already ruled out

• RBAC — the chart grants customresourcedefinitions list/watch whenever the
  feature is enabled, and auth can-i passes for the ServiceAccount.
• Config rejection — the plurals resolve and the family is registered.
• Label naming — exported_namespace (upstream's choice) and namespace
  behave the same.
• The --resources allowlist — tried unset, set to the chart default, and
  set to kustomizations, kustomizations.kustomize.toolkit.fluxcd.io and
  kustomize.toolkit.fluxcd.io/Kustomization.

The question

Is there state tied to the release name (ServiceAccount, ClusterRole binding
subject, the generated ConfigMap name) that the custom-resource store depends on
and that I am breaking? Or is a "family registered, zero series" outcome a known
shape with a cause I can look for?

Even if my reproduction is flawed, a log line saying the store produced no
objects would have saved two sessions here — related to #2903 and #2996, which
add logging for a missing CRD but not for this case.

[kubernetes-prow]

This issue is currently awaiting triage.

If kube-state-metrics contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.