Managed prometheus is blocked by the istio

[PawelHaracz]

Hey,
I enabled manages prometheus on my GKE cluster but I don’t gather any metrics from my pods. It’s worth to mention that my cluster uses istio service mesh, and it’s injected into every pod. Additionally, when I use Prometheus operator the metrics are scraped. Can someone of you have had a similar issue and know how to resolve it?

[lyanco]

Managed Prometheus doesn't scrape anything until you install a PodMonitor, which configures scraping. See https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed for details, and https://cloud.google.com/stackdriver/docs/managed-prometheus/exporters for some examples.

[PawelHaracz]

I forgot to mention earlier I already configured podMonitor and even service monitor, Attached some example configurations that I use:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name:  {{ .Chart.Name }}
spec:
  selector:
    matchLabels:
      app: {{ .Chart.Name }}
  endpoints:
  - port: 80
    interval: 30s

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: {{ .Chart.Name }}
spec:
  selector:
    matchLabels:
      app: {{ .Chart.Name }}
  endpoints:
  - targetPort: 80
    interval: 30s
    path:  "/metrics"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Chart.Name }}-deployment
  labels:
    app: {{ .Chart.Name }}
spec:
  selector:
    matchLabels:
      app: {{ .Chart.Name }}
  template:
    metadata:
      labels:
        app: {{ .Chart.Name }}
      annotations:
        sidecar.istio.io/proxyCPULimit: "2.0"
        sidecar.istio.io/proxyMemoryLimit: "1Gi"
        sidecar.istio.io/proxyCPU: "0.10"
        sidecar.istio.io/proxyMemory: "128Mi"
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "80"
        prometheus.io/scheme: http
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.containerRegistry }}/{{ .Chart.Name }}:{{ .Chart.Version }}"
          resources:
            limits:
              memory: {{ .Values.memory.limit }}
              cpu: {{ .Values.cpu.limit }}
            requests:
              memory: {{ .Values.memory.request }}
              cpu: {{ .Values.cpu.request }}
          envFrom:
            - configMapRef:
                name: {{ .Chart.Name }}-CM
          ports:
            - containerPort: 80
          livenessProbe:
            httpGet:
              path: /health/liveness
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 2
            timeoutSeconds: 2
          readinessProbe:
            httpGet:
              path: /health/readiness
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 3
            timeoutSeconds: 2

[lyanco]

I'm confused - GMP doesn't accept ServiceMonitors... are you configuring both prom operator and gmp with the same yaml?

[PawelHaracz]

As I mentioned at the beginning, I’m migrating the Prometheus self-hosted operator to managed Prometheus, and I have both Prometheus on the cluster. The self-hosted prometheus can scrap metrics, but GCP managed Prometheus can’t do it. That why I have podmonitor and service monitor

[pintohutch]

Can you try finding the collector pod that's co-located with your workload and running the commands mentioned here?

[pintohutch]

That snippet is concerned with scrapes of kube-state-metrics. Just replace that first command with the labels you're using for this workload to find the node it's running on.

[PawelHaracz]

Problem solved. The issue was that the istio injector replaced this port 80 to port 15020.

[pintohutch]

Ah thanks for the update @PawelHaracz. Can you elaborate here? How would Istio replace the metrics port?

[sreejesh-radhakrishnan-db]

@pintohutch hope you are still the SME on this area. We are having a similar issue where ISTIO is running on 15090 and scrape away all the metric. but my app exposing on 8084 is not getting through to GPM.
I followed your step above and I could see this error :

Get "http://240.13.206.101:8084/actuator/prometheus": read tcp xxx.xxx.xxx.xxx:37516->xxx.xxx.xx.xxx:8084: read: connection reset by peer

basically collector not able to connect to the Application POD.

any help please?

[pintohutch]

Hi @sreejesh-radhakrishnan-db - I'm going to move this discussion over to #966