Adds a Dogstatsd Tag Filter

[chrisleavoy]

What does this PR do?

Adds a filtering capability to the dogstatsd server that removes or modifies undesirable tags.

Motivation

Many statsd clients out there add undesirable tags to their metrics that result in inaccurate histograms and financial burden. For example, some libraries add an extremely high cardinality tag called request_id. Some libraries add UUID's into tag values. Tags with empty string values also mess up the Datadog UI. The use cases for this are plentiful.

Additional Notes

No.

Describe your test plan

Run the agent with:

DD_DOGSTATSD_TAG_FILTERS='[":$", "^undesirable_tag:", ":undesirable_value"]'

[bits-bot]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Chris Leavoy seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[kayayarai]

no Documentation review needed

[remeh]

Hey Chris,

Thank you for this good PR.

That's indeed a solution for a few use-cases, especially since it's nearly free when not used (as you're just testing a nil pointer) 👌 .
I'm adding a couple labels and a milestone and we will look into this soon.

[chrisleavoy]

Anyone know if there a bot command or something to trigger a CI build run? I'd like to see the tests pass (works locally) and get docker image for the dogstatsd agent.

[remeh]

Could you eventually catch-up master? The AppVeyor issue should have been fixed in master, and that would trigger a CI run.

[olivielpeau]

Hi @chrisleavoy, apologies for not reviewing your PR in time for 7.24. We are reviewing similar use cases internally to understand how to best address them. I'll update this PR in the next few weeks accordingly

[alexb-img]

Hi @olivielpeau , a few questions on this PR feature
does this only cover dogstatsd only?
if not, could it be extended / modified to cover metrics from any source eg. openmetrics?

It would be useful to link this PR to issues in the repo for metric tag filtering, which include
#13588 and
#3130
The last one has 10 teams requesting it and 66 'likes' currently.
This may provide a motivation to progress the feature request / and/or discussion on it.
Many thanks!

[alexey-chumakov]

Are there any estimations when this feature can be released (or similar one)? It is a blocking issue for DataDog/Spark integration since by default DataDog includes even stageId/jobId into tags which leads to 10^5 custom metrics in real production environments.

[carlosroman]

Thank you for your pull request. While we appreciate your initiative, we're currently prioritizing caution and thoughtfulness in adding features to the Datadog agent to ensure optimal solutions for our users' needs.
As of now, we already have existing solutions to help customers safely remove tags from metrics while preserving mathematical accuracy of those metrics' queries.

We encourage you and our users to explore these existing options such as Metrics without Limits. For any further inquiries or specific needs, please reach out through our support channel (https://www.datadoghq.com/support/ ) or contact your Customer Success Manager (CSM).

[indrekj]

As of now, we already have existing solutions to help customers safely remove tags from metrics while preserving mathematical accuracy of those metrics' queries.

Could you elaborate which kind of problems this would cause? Maybe the PR author could address them. #3130 has around a hundred thumbs-up. It seems like a quite important feature for many of us.