🐛 [BUG] - Unable to support concurrent requests with multiple Datadog backend

[bkamin29]

SLO Generator Version

2.*

Python Version

3.11

What happened?

I’ve encountered a limitation with the current implementation of the Datadog backend in slo-generator. Currently, it is not possible to declare multiple Datadog backends for concurrent SLO reporting. This limitation stems from the use of the datadog Python module, which relies on a globally shared configuration instead of independent client objects.

Issue Details

Since the Datadog backend implementation in slo-generator uses the datadog module directly, it lacks the ability to isolate multiple configurations or handle concurrent requests without shared session data or interference between clients. The global state maintained by the datadog module means that requests to separate Datadog instances (or with different API keys) overwrite each other’s configurations, preventing the independent handling of multiple Datadog backends.

Consequence

If concurrency is enabled—such as when running slo-generator in a cloud environment like Cloud Functions or in API mode with Cloud Run—the global configuration issue causes slo-generator to send SLO calculations to the incorrect Datadog organization. This can result in inaccurate reporting and potential data leaks between organizations.

Suggested Solution

A potential solution would be to refactor the Datadog backend in slo-generator to use the datadog-api-client library instead. This library enables the creation of isolated Datadog API clients and supports better concurrency control by avoiding global state. Each client instance can be configured independently, allowing for multiple Datadog backends within slo-generator to operate concurrently and without interference.

Steps to Reproduce

1. Declare multiple Datadog backends in slo-generator, each with different configurations.
2. Enable concurrency (e.g., deploy slo-generator as a Cloud Function or API with Cloud Run).
3. Observe that requests may be routed to the incorrect Datadog organization, leading to unexpected behavior.
4. Additionally, attempting to instantiate multiple client objects by printing each one (print(client)) will consistently return the same result, indicating that the instances are not isolated despite different configurations.

Expected Behavior

Each Datadog backend should be able to operate with its own independent configuration and handle concurrent requests without affecting other backends.

Additional Context

Refactoring to use datadog-api-client would improve flexibility and make it easier to implement robust multi-backend configurations with Datadog in slo-generator.

Code of Conduct

• I agree to follow this project's Code of Conduct