[Microsoft.Extensions.Http.Resilience] Circuit-breakers and SelectPipelineByAuthority confusion

[natwallbank]

Hey there,

I recently raised an issue relating to the mechanism of being able to monitor circuit-breaker state and manual control, as I felt (and still feel) that the API doesn't really lend itself to being able to do this for a number of discrete dependencies that you want to monitor separately.

I worked around this by following the guidance and defining separate pipelines for each dependency, which allowed me to configure distinct instances of CircuitBreakerStateProvider and CircuitBreakerManualControl. That all works fine.

However, I've just hit another edge case, in that one of our dependencies has a dynamic base URL that we only resolve at runtime. It looks something like this:

https://some.downstreamservice.{cell}.com/

We query another service on behalf of the client, which tells us which cell value to use for a subsequent request.

This all works fine from an HttpClient point of view, but I wanted to be sure that the circuit breaker isn't shared across all instances, i.e. if the circuit is opened for https://some.downstreamservice.A.com/ then it should not also open for https://some.downstreamservice.B.com/.

I gather from the documentation that the answer to this is using SelectPipelineByAuthority.

Where I'm confused is: how can I query circuit-breaker state for each "instance"? I feel like there's a missing API to be able to inspect pipeline state, rather than supplying these instances as part of configuration?

Once again though, there's also a high chance that I'm missing the real answer - but I've struggled to find anything relevant in the docs.

Thanks, again, in advance!

[RussKie]

@dotnet/dotnet-extensions-resilience please triage

[iliar-turdushev]

@natwallbank

I gather from the documentation that the answer to this is using SelectPipelineByAuthority.

Correct, if you invoke SelectPipelineByAuthority on a resilience pipeline setup then you'll have a separate pipeline per each HTTP request authority.

Where I'm confused is: how can I query circuit-breaker state for each "instance"? I feel like there's a missing API to be able to inspect pipeline state, rather than supplying these instances as part of configuration?

I'm wondering what is the ultimate goal of "querying the state of each circuit-breaker".

[natwallbank]

Hey @iliar-turdushev - it just seems like a disconnect between the ability to monitor circuit breaker state and the reality of how fine-gained those circuits might be.

For my specific circumstance, we have detailed health checks that report the status of each of our dependencies. It would be really quite nice to be able to report the state of the associated circuit alongside that.

[iliar-turdushev]

Hey @iliar-turdushev - it just seems like a disconnect between the ability to monitor circuit breaker state and the reality of how fine-gained those circuits might be.

I don't know whether there is a way now to retrieve particular StateProvider or ManualControl. We need to investigate that.

For my specific circumstance, we have detailed health checks that report the status of each of our dependencies. It would be really quite nice to be able to report the state of the associated circuit alongside that.

Circuit breaker emits telemetry which produces metrics and logs. Take a look at those. They might be helpful for your monitoring purposes.