Incorrect metric values after importing to Cloud Monitoring in GCP

[sachin-wickramasinghe]

Describe the bug

I used opentelemetry-collector-contrib:0.118.0 to collect metrics from the RabbitMQ Prometheus endpoint and export them to the Google Cloud Monitoring API. However, after importing, the actual values of some metrics do not match the exported values in Metrics Explorer in Cloud Monitoring.

Steps to reproduce

What did you expect to see?

Metric Name	Value
rabbitmq_connections	1
rabbitmq_connections_opened_total	1
rabbitmq_global_messages_acknowledged_total	0
rabbitmq_identity_info	1
rabbitmq_process_max_tcp_sockets	943.629k

What did you see instead?

Metric Name	Value
rabbitmq_connections	0
rabbitmq_connections_opened_total	0
rabbitmq_global_messages_acknowledged_total	0
rabbitmq_identity_info	1
rabbitmq_process_max_tcp_sockets	943.629k

What version did you use?

opentelemetry-collector-contrib:0.118.0

What config did you use?

otel-collector-config.yaml

apiVersion: v1
data:
  config.yaml: |
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'rabbitmq'
              scrape_interval: 5s
              static_configs:
                - targets: ['rabbitmq-prometheus.rabbitmq.svc.cluster.local:15692']
processors:
  resourcedetection:
    detectors: [env, gcp]
    timeout: 10s
    override: false

  filter/1:
    metrics:
      include:
        match_type: regexp
        metric_names:
          - "rabbitmq_process_max_tcp_sockets"
          - "rabbitmq_queue_consumers"
          - "rabbitmq_queue_messages"
          - "rabbitmq_queue_messages_ready"
          - "rabbitmq_queue_messages_unacked"
          - "rabbitmq_connections_opened_total"
          - "rabbitmq_queues"
          - "rabbitmq_build_info"
          - "rabbitmq_identity_info"
          - "rabbitmq_global_messages_received_total"
          - "rabbitmq_global_messages_acknowledged_total"
          - "rabbitmq_queue_messages_published_total"
          - "rabbitmq_connections"

exporters:
  googlecloud:
    log:
      default_log_name: opentelemetry.io/collector-exported-log

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [resourcedetection, filter/1]
      exporters: [googlecloud]

kind: ConfigMap

metadata:

name: otel-collector-config

namespace: opentelementry

Deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: opentelemetry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - args:
        - --config=/etc/otel-collector/config.yaml
        command:
        - ./otelcol-contrib
        image: otel/opentelemetry-collector-contrib:0.118.0
        imagePullPolicy: Always
        name: otel-collector
        volumeMounts:
        - mountPath: /etc/otel-collector
          name: config-volume
      volumes:
      - configMap:
          name: otel-collector-config
        name: config-volume
    

Environment

I used Google Kubernetes Engine for deploy collector with above mentioned docker image.

System Information

• OS: Google Kubernetes Engine (GKE)
• Kernel version: 6.1.100+
• OS image: Container-Optimized OS from Google
• Container runtime version: containerd://1.7.22
• Kubelet version: v1.30.5-gke.1443001
• Kube-proxy version: v1.30.5-gke.1443001

[github-actions]

Pinging code owners for receiver/prometheus: @Aneurysm9 @dashpole. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[github-actions]

Pinging code owners for exporter/googlecloud: @aabmass @dashpole @jsuereth @punya @psx95. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[dashpole]

I know this documentation is for google managed prometheus, but in this case it applies to the googlecloud exporter as well, and does a good job of explaining why that happens. In short, Google Cloud Monitoring requires start timestamps. The Prometheus ecosystem is in the process of adding start timestams, but most existing Prometheus endpoints don't include them today. To address this, the googlecloud exporter will drop the first point that doesn't have a start timestamp, and use that point's timestamp as the start timestamp for all future points. To ensure rates are correct, it "subtracts" the value of the initial point from all future points as well.

I'm working on factoring this logic out into its own processor to make it more transparent in the collector: #37186