perf(metrics): tune histogram metric query #712
Conversation
Moves grouping, filtering and limits for the histogram query into the initial selection CTE. This reduces the memory usage required for query. Ref: HDX-1518
🦋 Changeset detectedLatest commit: 5c81adf The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
| END AS Point | ||
| FROM ${renderFrom({ from: { ...from, tableName: metricTables[MetricsDataType.Histogram] } })} HistogramTable | ||
| WHERE ${where} | ||
| GROUP BY ${concatChSql(',', [chSql`AttributesHash`, groupBy])} |
There was a problem hiding this comment.
we run into problems here if we try to group by granularity before calculating rate, usually when we start setting granularity to be something like looking back 14d and then we'll do something like 12h granularity, and with rates only having 12h granularity we run into huge errors with counter resets making the metric virtually useless.
In v1 I believe we've had success with doing our rate calculations first and then doing the percentile interpolation, I'm wondering what's the issue we're running into here, is it for some reason the rate calculations on arrays are really memory expensive?
Moves grouping, filtering and limits for the histogram query into the initial selection CTE. This reduces the memory usage required for query.
Ref: HDX-1518