You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
Existing OTEL Prometheus Receiver metricBuilder expects Prometheus scrape loop to send data-points organized and grouped by metricName. Currently, Prometheus doesn’t provide data-point organized and grouped by metric name for failed scrapes, and this causes non-deterministic generation of metrics by the metric builder for the failed scrapes.
Describe the solution you'd like
Currently, metricBuilder aggregates the metric data-points until new metricFamily is encountered. And upon encountering new metricFamily it converts the aggregated data-points into the metric.
This approach causes non-deterministic output upon encountering data-points that are in random order.
As per Prometheus, it is valid but rare for scrape loop to send metrics in random order even for successful scrapes.
One proposed solution is to modify the metricBuilder logic to buffer (aggregate) and sort the datapoints in AddDataPoint method and completely move the metric building logic to the commit logic.
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.
Describe the bug
Existing OTEL Prometheus Receiver
metricBuilder
expects Prometheus scrape loop to send data-points organized and grouped bymetricName
. Currently, Prometheus doesn’t provide data-point organized and grouped by metric name for failed scrapes, and this causes non-deterministic generation of metrics by the metric builder for the failed scrapes.Describe the solution you'd like
Currently,
metricBuilder
aggregates the metric data-points until newmetricFamily
is encountered. And upon encountering newmetricFamily
it converts the aggregated data-points into the metric.This approach causes non-deterministic output upon encountering data-points that are in random order.
As per Prometheus, it is valid but rare for scrape loop to send metrics in random order even for successful scrapes.
One proposed solution is to modify the
metricBuilder
logic to buffer (aggregate) and sort the datapoints inAddDataPoint
method and completely move the metric building logic to the commit logic.Additional context
Related to open-telemetry/wg-prometheus#57
Issues: #6000 #6087
cc: @alolita @Aneurysm9
The text was updated successfully, but these errors were encountered: