Automatic metric relabel to append _total to metric name

[sherabi]

Component(s)

prometheus receiver

What happened?

I am using the latest version of OTel with K8s. It seems that the prometheus receiver automatically appends _total to the metric name for metrics of type counter.
I wanted to keep my original metric name and it seems almost impossible to undo the automatic renaming of the metric. I tried setting trim_metric_suffixes: true but it doesn't seem to work.

I tried metric_relabel_configs which also didn't seem to work with the prometheus receiver

              metric_relabel_configs:
                - source_labels: [__name__]
                  regex: 'web_account_balance_fetch_results_total'
                  target_label: __name__
                  action: replace
                  replacement: 'web_account_balance_fetch_results'
                - source_labels: [__name__]
                  regex: 'web_account_balance_fetch_results'
                  action: keep
              metrics_path: /metrics
              scheme: http

I even tried the metricstransform processor which also did not work.

      metricstransform:
        transforms:
          - include: web_account_balance_fetch_results_total
            match_type: strict
            action: update
            new_name: web_account_balance_fetch_results

How can I keep the original metric name?

Collector version

0.118.0

Environment information

GKE K8S version 1.30

OpenTelemetry Collector configuration

config:
    receivers:
      prometheus:
        trim_metric_suffixes: true
        config:
          scrape_configs:
            - job_name: 'synthetics'
              static_configs:
                - targets: ['synthetics.default.svc:3000']
                  labels:
                    service: 'synthetics'
              metric_relabel_configs:
                - source_labels: [__name__]
                  regex: 'web_account_balance_fetch_results_total'
                  target_label: __name__
                  action: replace
                  replacement: 'web_account_balance_fetch_results'
                - source_labels: [__name__]
                  regex: 'web_account_balance_fetch_results'
                  action: keep
              metrics_path: /metrics
              scheme: http
    processors:
      k8sattributes:
        auth_type: serviceAccount
        passthrough: false
        extract:
          metadata:
            - k8s.pod.name
            - k8s.namespace.name
            - k8s.container.name
            - k8s.pod.uid
            - k8s.deployment.name
            - k8s.job.name
            - k8s.daemonset.name
            - k8s.statefulset.name
            - k8s.cronjob.name
            - k8s.node.name
            - container.image.name
            - container.image.tag
      attributes:
        actions:
          - action: insert
            key: node_id
            from_attribute: node
          - action: delete
            key: node
      # metricstransform:
      #   transforms:
      #     - include: web_account_balance_fetch_results_total
      #       match_type: strict
      #       action: update
      #       new_name: web_account_balance_fetch_results
    exporters:
      # debug:
      #   verbosity: detailed
      otlphttp/grafana:
        endpoint: "https://otlp-gateway-dev-us-central-0.grafana.net:443/otlp"
        headers:
          authorization: "Basic ${DEV_OTEL_GRAFANA_CLOUD_API_KEY}"
          instance-id: "xxxxx"
      debug:
        verbosity: detailed
    service:
      pipelines:
        metrics/dev:
          receivers:
            - prometheus
          processors:
            # - metricstransform
            - attributes
            - k8sattributes
          exporters:
            # - debug
            - otlphttp/grafana

Log output

No change to the metric name

Additional context

No response

[github-actions]

Pinging code owners for receiver/prometheus: @Aneurysm9 @dashpole. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[dashpole]

Thats very strange. The metricstransform processor should definitely work, and the metric_relabel_configs should also work. The prometheus receiver doesn't actually add any suffixes itself, but if you are scraping using OpenMetrics, some client libraries will automatically add the total suffix.

You verified that the metric name in the debug exporter still had the _total suffix, right?

[shezaan-hashgraph]

@dashpole

You verified that the metric name in the debug exporter still had the _total suffix, right?

There was no _total suffix that I could find in the debug exporter. Thats why I am confused. In the debug logs I only see the original name of my metric i.e. web_account_balance_fetch_results.

[dashpole]

I suspect the OTLP receiver for grafana is adding the suffix. That would make sense if they are using the Prometheus OTLP endpoint (which adds suffixes) in their agent or product.

[dashpole]

See https://github.com/prometheus/prometheus/blob/6ede9005066a9f8b6ace1f9b3e215c208e60c3bd/docs/configuration/configuration.md?plain=1#L185. All current options add suffixes. We are working on adding support for a suffix-less option.

[shezaan-hashgraph]

Thanks @dashpole that helps a lot. Since we have Grafana support, I will attempt to bring this to their notice. I find forcefully changing the metric name should be registered as a breaking change. It means changing alerts and dashboards that use that metric name.