Provision a basic PSQL dashboard for grafana

[reosarevok]

This is based on https://grafana.com/grafana/dashboards/14114-postgres-overview/ with an extra check for max query duration that seemed interesting, and is mostly intended as a proof of concept for provisioning dashboards. We can further improve the dashboard as needed.

[mwiencek]

This actually works quite nicely for me, though we could build on top of the metrics you added by creating a basic dashboard for them. That could also be used as an example for how to add additional dashboards to the repo. Here's one that includes a panel for the artist count:

{
  "__inputs": [
    {
      "name": "DS_PROMETHEUS",
      "label": "Prometheus",
      "description": "",
      "type": "datasource",
      "pluginId": "prometheus",
      "pluginName": "Prometheus"
    }
  ],
  "__elements": {},
  "__requires": [
    {
      "type": "grafana",
      "id": "grafana",
      "name": "Grafana",
      "version": "11.4.0"
    },
    {
      "type": "datasource",
      "id": "prometheus",
      "name": "Prometheus",
      "version": "1.0.0"
    },
    {
      "type": "panel",
      "id": "timeseries",
      "name": "Time series",
      "version": ""
    }
  ],
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": null,
  "links": [],
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "description": "Number of rows in the artist table.",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "displayName": "rows",
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "11.4.0",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "disableTextWrap": false,
          "editorMode": "builder",
          "expr": "artist_count",
          "fullMetaSearch": false,
          "includeNullMetadata": true,
          "legendFormat": "__auto",
          "range": true,
          "refId": "A",
          "useBackend": false
        }
      ],
      "title": "Artist Count",
      "type": "timeseries"
    }
  ],
  "schemaVersion": 40,
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "browser",
  "title": "Table Row Counts",
  "uid": "cea3e99siyr5sa",
  "version": 2,
  "weekStart": ""
}

• For comparing with Solr metrics like number of documents per Solr core, we probably only need a metric for each entity table corresponding to a Solr core?
• Currently these row count metrics are collected every 30s, which seems too frequent.

[reosarevok]

That seems like a good start. I expanded and provisioned the dashboard, adding all the counts that seem to be relevant as per https://github.com/metabrainz/sir/blob/e9e63641cd103c29a1aca456fb870d9f7d508774/sir/schema/__init__.py - and changed sql-exporter to only check stuff every 5 minutes.

[mwiencek]

We should probably align this with prometheus's scrape_interval too (and grafana's $__interval when fetching the data points). Since there's no point in producing duplicate data points for the same scraped value.

Edit: Pushed a commit to try and do that.

[mwiencek]

Now that I'm looking at the row counts dashboard again, I do lean towards defaulting to the time series view: it seems you can only view that by editing the panel or clicking "explore," which isn't very obvious, though I may be missing something. The main reason though is that the graph is more useful, because it shows you if the data is actually changing (which can be useful for determining if replication is working, for example).

[reosarevok]

Ok, I still think that's a niche case but I'm not going to be the one using this often, so sure, why not :) Changed that with an extra commit.

[yvanzo]

Thanks for having created these dashboards, and for the graph view.

The interval (5 min) might be too long for analyzing what is going on with replication. What do you think @mwiencek?

Also made a comment about service dependencies.

Anyway, feel free to make these checks and changes after merging if it makes it easier to move on with other pull requests.

[yvanzo]

That makes sql-exporter starts after these services, which is great.
Is sql-exporter robust enough to not stop if the database musicbrainz is not available yet?
Otherwise a custom condition or health check might be needed.

[reosarevok]

I tested this (as per @mwiencek's suggestion) by taking the services down, renaming the database, taking them back up, and then after a while renaming the database back to musicbrainz_db. Both errored while unable to connect, but eventually connected back and started showing data again, so they seem resilient enough.

[mwiencek]

The interval (5 min) might be too long for analyzing what is going on with replication. What do you think @mwiencek?

I thought it seemed reasonable for hourly replication, as 5m is already creating 12 duplicate data points per hour if replication is operating normally via cron.

I could see it being too long in a couple cases though:

• The mirror is behind and is applying several replication packets per 5m interval
• You're running LoadReplicationChanges by hand and have to wait 5m for the stats to update

[yvanzo]

for hourly replication

The main goal was to allow for debugging and profiling in development setup, where replication packets can be replayed much faster.

Allowing for monitoring actual mirrors would be great too, but possibly different dashboards then?

[yvanzo]

I merged SIR dev stuff into master, rebased the target branch monitoring on it, and rebased the source branch provision-psql-dashboard on it to resolve conflicts.

[mwiencek]

Will merge this as discussed yesterday so I can rebase my other PRs. If further changes are needed they can be made in the monitoring branch.