Production - [Alerting] DotNetEng Status Failed Requests/Hour alert

[dotnet-eng-status]

💔 Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated.
To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 191.2

Go to rule

@dotnet/dnceng, @dotnet/prodconsvcs, please investigate

Automation information below, do not change

Grafana-Automated-Alert-Id-d2dd705a6c724ed68fcf6955561c06dd

[dotnet-eng-status]

💔 Metric state changed to alerting

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated.
To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

• failuresCount 191.2

Go to rule

[dotnet-eng-status]

💚 Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated.
To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[dotnet-eng-status]

💚 Metric state changed to ok

The number of failed DotNetEng Status requests per hour is above 20. This may indicate a systemic problem that needs to be investigated.
To intially investigate prod, run the following query in DotNetEng-Status-Prod, and to investigate staging, run the query in DotNetEng-Status-Staging:

union exceptions, traces
| project timestamp, operation_Name, customDimensions, message, problemId, details
| order by timestamp asc

Go to rule

[haruna99]

The number of failed DotNetEng Status requests per hour is below 20. Closing issue

[garath]

The number of failed DotNetEng Status requests per hour is below 20. Closing issue

Did the failed requests fall low because the overall request count also went low? Or perhaps there is a class of request that's failing that spiked in that window but continues to fail. It would be good to dig a bit to understand what was failing.

[haruna99]

The number of failed DotNetEng Status requests per hour is below 20. Closing issue

Did the failed requests fall low because the overall request count also went low? Or perhaps there is a class of request that's failing that spiked in that window but continues to fail. It would be good to dig a bit to understand what was failing.

I will conduct further investigation to better understand the cause of the failure.