How to use label_values without full support of /api/v1/$label/values ?

[phandox]

Hi,

we deployed the frontend-proxy to access Managed Prometheus. In our dashboards we use variables such as label_values(kube_pod_info, cluster) but these queries doesn't work - they return 400 because Grafana uses /api/v1/$label/values endpoint - this is how the request look like:

GET /api/v1/label/cluster/values?match%5B%5D=kube_pod_info&start=1676550532&end=1676636932 HTTP/1.1

I understand from docs that this is not supported (https://cloud.google.com/stackdriver/docs/managed-prometheus/query#http-api-details) but I am not using the form label_values($label) but the recommended one label_values($metric, $label).

Is it possible to use /api/v1/series for getting just label values ? I am struggling with the query...

We use:

• Grafana v9.3.2
• gke.gcr.io/prometheus-engine/frontend:v0.5.0-gke.0

Thanks for help.

Edit: Relevant PR in Grafana - grafana/grafana#56510 . We specified the prometheus version in data source setting to 2.35.0 ("prometheusVersion": "2.35.0",)

[lyanco]

Does label_values work in older versions of Grafana?

I have a suspicion that Grafana recently changed the API call that label_values makes, breaking compatibility with GMP. If so, we'll have to implement a fix.

[phandox]

It started to work, when we didn't specify Prometheus version in when we add Prometheus Data Source. That correlates with change introduced in mentioned Grafana PR (grafana/grafana#56510) - as fallback mechanism it uses /series endpoint. I think all Prometheus DS with version v2.24+ forces the usage of /api/v1/$label/values endpoint - when version is not specified, it fallback to /api/v1/series

[lyanco]

Gotcha, thanks. Sounds like we need to gracefully handle this change in how label_values is translated.

While Monarch requires a metric name for all queries including autocomplete (which is why we can't support a naked labels() function), when using label_values() the constructed /values query does include the metric name in the match params. We should be able to do something like intelligently parse the vars out of /values call and return a valid result.

Thanks for the heads up! Will let you know when we fix it.