What happened:
When a client sends an Accept header listing application/vnd.google.protobuf with the highest q-value alongside supported formats (application/openmetrics-text, text/plain), KSM responds with text/plain; version=0.0.4 instead of the next best supported format.
This is the standard Accept header sent by Prometheus scrapers:
Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited;q=0.6,
application/openmetrics-text;version=1.0.0;escaping=allow-utf-8;q=0.5,
application/openmetrics-text;version=0.0.1;q=0.4,
text/plain;version=1.0.0;escaping=allow-utf-8;q=0.3,
text/plain;version=0.0.4;q=0.2,
*/*;q=0.1
The response Content-Type is text/plain; version=0.0.4; charset=utf-8 — all OpenMetrics entries in the Accept header are silently ignored.
What you expected to happen:
KSM cannot serve protobuf, so it should skip it and select the next acceptable format it can serve. The response Content-Type should be:
application/openmetrics-text; version=1.0.0; charset=utf-8
This is the behavior mandated by https://datatracker.ietf.org/doc/html/rfc7231#section-5.3.2: the Accept header is a preference ordering, not a hard requirement. The server should select the highest-quality type it can produce and only fall back to a default if none
of the listed types are supported.
How to reproduce it (as minimally and precisely as possible):
Start kube-state-metrics (any standard deployment)
Then scrape /metrics with the Prometheus default Accept header:
curl -v \
-H 'Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited;q=0.6,application/openmetrics-text;version=1.0.0;escaping=allow-utf-8;q=0.5,application/openmetrics-text;version=0.0.1;q=0.4,text/plain;version=1.0.0;escaping=all
ow-utf-8;q=0.3,text/plain;version=0.0.4;q=0.2,/;q=0.1' \
http://localhost:8080/metrics 2>&1 | grep "Content-Type"
Actual: Content-Type: text/plain; version=0.0.4; charset=utf-8
Expected: Content-Type: application/openmetrics-text; version=1.0.0; charset=utf-8
Anything else we need to know?:
The root cause is in pkg/metricshandler/metrics_handler.go:
contentType := expfmt.NegotiateIncludingOpenMetrics(r.Header)
if contentType.FormatType() != expfmt.TypeOpenMetrics {
contentType = expfmt.NewFormat(expfmt.TypeTextPlain)
} expfmt.NegotiateIncludingOpenMetrics correctly returns FmtProtoDelim (the highest-quality match from the Accept header). The guard then discards it because KSM doesn't support protobuf — but instead of re-negotiating against only the formats KSM does support, it
unconditionally falls back to text/plain; version=0.0.4.
The practical impact is that any Prometheus scraper using the default Accept header will never receive OpenMetrics from KSM, even though both sides support it. This blocks OpenMetrics-only features from working correctly end-to-end.
Related: #2022
Environment:
- kube-state-metrics version:
- Kubernetes version (use
kubectl version):
- Cloud provider or hardware configuration:
- Other info:
What happened:
When a client sends an Accept header listing application/vnd.google.protobuf with the highest q-value alongside supported formats (application/openmetrics-text, text/plain), KSM responds with text/plain; version=0.0.4 instead of the next best supported format.
This is the standard Accept header sent by Prometheus scrapers:
The response Content-Type is text/plain; version=0.0.4; charset=utf-8 — all OpenMetrics entries in the Accept header are silently ignored.
What you expected to happen:
KSM cannot serve protobuf, so it should skip it and select the next acceptable format it can serve. The response Content-Type should be:
application/openmetrics-text; version=1.0.0; charset=utf-8
This is the behavior mandated by https://datatracker.ietf.org/doc/html/rfc7231#section-5.3.2: the Accept header is a preference ordering, not a hard requirement. The server should select the highest-quality type it can produce and only fall back to a default if none
of the listed types are supported.
How to reproduce it (as minimally and precisely as possible):
Start kube-state-metrics (any standard deployment)
Then scrape /metrics with the Prometheus default Accept header:
curl -v \
-H 'Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited;q=0.6,application/openmetrics-text;version=1.0.0;escaping=allow-utf-8;q=0.5,application/openmetrics-text;version=0.0.1;q=0.4,text/plain;version=1.0.0;escaping=all
ow-utf-8;q=0.3,text/plain;version=0.0.4;q=0.2,/;q=0.1' \
http://localhost:8080/metrics 2>&1 | grep "Content-Type"
Actual: Content-Type: text/plain; version=0.0.4; charset=utf-8
Expected: Content-Type: application/openmetrics-text; version=1.0.0; charset=utf-8
Anything else we need to know?:
The root cause is in pkg/metricshandler/metrics_handler.go:
expfmt.NegotiateIncludingOpenMetrics correctly returns FmtProtoDelim (the highest-quality match from the Accept header). The guard then discards it because KSM doesn't support protobuf — but instead of re-negotiating against only the formats KSM does support, it
unconditionally falls back to text/plain; version=0.0.4.
The practical impact is that any Prometheus scraper using the default Accept header will never receive OpenMetrics from KSM, even though both sides support it. This blocks OpenMetrics-only features from working correctly end-to-end.
Related: #2022
Environment:
kubectl version):