Airflow Issue with OpenTelemetry

[wolvery]

Hi!

We are experiencing an issue in Airflow 2.7.3 with the OpenTelemetry SDK. It seems that, while trying to send the metrics, we encounter this error when the OpenTelemetry Collectors are overwhelmed:

Transient error %s encountered while exporting metric batch, retrying in %ss.

This seems to impact the CPU consumption of the Scheduler, and when it happens with tasks, they get stuck.
Have someone experienced this?
Any suggestion to handle it?
I saw that the OpenTelemetry GRPC SDK for Python does not have this code to exponentially backoff the retry:
https://opentelemetry-python.readthedocs.io/en/latest/_modules/opentelemetry/exporter/otlp/proto/http/metric_exporter.html#OTLPMetricExporter