Reducing Costs

[AlHood77]

Trying to understand costs and high volume metrics. I have 2 fully managed prometheus instances (prod and nonprod). The prod has the majority of the exporters deployed and a number of pod monitors. The non-prod has a sole pod monitor deployed yet the costs per month are very similar. My highest volume metrics are not being scraped by any pod monitor and my highest volume namespace doesn't have a pod monitor. Is there something I'm missed when it comes to costs. Grateful for any clarification. Thanks.

[lyanco]

Try checking out our metrics diagnostics page to identify where metrics are coming from: https://console.cloud.google.com/monitoring/metrics-explorer/diagnostics

Documentation on it: https://cloud.google.com/stackdriver/docs/managed-prometheus/cost-controls#identify-cost-sources
https://cloud.google.com/monitoring/custom-metrics/diagnostics

I recommend sorting the bottom table by samples ingested!

[AlHood77]

Thanks for the links, especially the diagnostics page as wasn't aware of that and I found it extremely useful. Is there a way to to drop metrics that the managed service defines and writes, ie, those with the prefix 'prometheus.googleapis.com'. I can see in the docs about dropping metrics scraped via exporters but I'm less sure about other types of metrics.

[lyanco]

Glad to hear it's useful - we're going to add query stats to it and pull it up to the top nav so it's more discoverable.

At this time the only way to drop metrics is at source; e.g. modify the exporter or have a relabeling rule in your scrape configs that drops the metrics you don't want. Eventually we will have server-side filtering rules you can use to drop metrics globally before they're written, but that's a way off.

[AlHood77]

I had a play around and deployed a clusterPodMonitor to drop our highest ingested metric according to the table on the diagnostics page. All it seemed to do was change the status of the metric to a non-active state. The metric disappeared from the metric table which has its default set to active metrics. It becomes visible again when you turn off the active filter. However this metric was still the highest metric as per the 'billable samples ingested' dashboard. There was no change. I was expecting the number of billable ingested samples to drop. Have I misunderstood what the relabelling rules do in regards to dropping metrics?

[lyanco]

Metrics are set to inactive if there's no data seen in the last 25 hours.

Any chance you had the time picker on the top right set to a longer period than you had the metric turned off? In that case the table would show the data that was ingested during the period when the metric was still on.

[lyanco]

Ah, that's a container / cAdvisor metric. You can't filter those out with a PodMonitoring as they're not configured by one.

You can use a collection filter in your OperatorConfig to drop it: https://cloud.google.com/stackdriver/docs/managed-prometheus/exporters/kubelet-cadvisor

I recommend only using a negative match (as in the example) to filter out these specific metrics, otherwise you risk filtering out more metrics than you wanted to. And to only use this for container_* metrics...

[AlHood77]

I don't have my OperatorConfig configured to scrape these metrics. I do in my prod managed Service but not this one.

[AlHood77]

sorry i see that I do

[AlHood77]

I have clearly applied it to the wrong service

[AlHood77]

Thanks for your help - It's been a useful exercise for me in understanding all the different metrics, where they come from and how to filter them.