Support for Drain / Soft-Stop Semantics

[RedN4ks]

Description

As a:

• Application user/user of the configuration itself
• API user (application developer)
• SPI user (container or runtime developer)
• Specification implementer

I need to be able to:

Indicate that an application instance is still healthy but should temporarily stop receiving new incoming traffic, for example when it is under high CPU or memory load, while continuing to serve existing sessions or in-flight requests.

which enables me to:

Gracefully reduce traffic to stressed instances, improve overall availability, and avoid abrupt failures, without relying on non-standard health endpoints or load-balancer-specific workarounds.

[xstefank]

Thanks @RedN4ks, good idea. So what I'm thinking:

• We would need a new HealthCheckResponse.Status. This should clearly indicate "draining" schematic.
• Make the HTTP status code for "draining" status configurable.

Do I understand correctly, that the load balancer is purely expecting only the HTTP status code and the response is discarded? So we could still return the same JSON as we would for UP or DOWN?

[OndroMih]

As far as I know, Kubernetes expects 200 status code in the health probe, otherwise it would testart the pod.

What about using a HTTP response header to indicate more detailed status for load balancers?

[RedN4ks]

As far as I know, HAProxy supports this via http-check disable-on-404.
I am not sure about other load balancers or ingress controllers. However, even if they do not support this mechanism today, it is reasonable to expect that it could be implemented in the future, either based on the HTTP status code or by reading specific headers from the response, as @OndroMih suggested.

I therefore propose that we make this configurable, ideally supporting both approaches. It is true that if a user configures the MicroProfile health endpoint to return a status other than 200 in order to trigger drain mode, this may be interpreted as “kill the pod”. However, a more adapted configuration on the ingress side should be considered.

[NottyCode]

I thought that this is the scenario a readiness probe is intended for. The liveness probe will restart the container if it returns a non-200 response code, but the readiness probe should just stop http traffic being routed.

It is possible there is some nuance I am missing, but readiness probes seem to be a valid solution here.

[RedN4ks]

@NottyCode You’re right on the first part. For me, the only difference is this:

while continuing to serve existing sessions or in-flight requests.

Or does a failing readiness probe continue to route existing sessions to the same pod?

I may have been too brief in the issue description; I provided more details on the MicroProfile Google Group:
https://groups.google.com/g/microprofile/c/OqpoQrN-40s

[xstefank]

So maybe I also misunderstood. But do you mean that you want to continue consume "some" requests (i.e., the ones for the currently linked sessions) for the duration of the drain correct? And this is possible by giving your load-balancer 404.

So after a little bit of more thinking, I think we are talking about two different use-cases. Readiness has a clearly defined use-case as described above by @NottyCode. I think that you would like to have similar health check that would be intended for the load-balancer only and not directly called by k8s. In SR Health, we have a concept of several additional health groups (https://smallrye.io/docs/smallrye-health/3.0.1/health-groups.html and also https://smallrye.io/docs/smallrye-health/3.0.1/wellness.html) which I think might solve this? Of course, we would still need to have the option to customize the returned status code (if that's required).

[RedN4ks]

Exactly. What is lacking today is the ability to customize the response status code. Health groups can help separate the health-check endpoints consumed by Kubernetes from those consumed by the ingress, but even then we still need more control over the response headers (which I assume are not configurable) and over the response status code itself.

[xstefank]

I cannot think of why that couldn't be added. As long as UP is 2xx and DOWN is anything else we should be aligned with k8s. Headers can be added as they don't really mean anything to k8s.

[Emily-Jiang]

I am unclear why a pod needs to tell the infrastructure it is under load. Would the autoscaler be able to monitor all of the pods and adjust accordingly by scaling up or down. Are we trying to add something to replace the load balancer? I think it is better left to the infrastructure.

[RedN4ks]

I am unclear why a pod needs to tell the infrastructure it is under load.

@Emily-Jiang Consider a stateful web application storing user sessions (e.g., Jakarta HttpSession) with an ingress (HAProxy) that has session stickiness enabled. Normally, requests for existing sessions are routed to the pod where the session resides.

Now, suppose that pod is under load. If it declares itself as not ready via the standard readiness probe, Kubernetes will remove it from the Service endpoints, meaning it will stop receiving any requests, including requests for existing sessions.

If the pod is under pressure but can continue serving existing sessions, it could:

• Declare itself ready on the standard readiness endpoint consumed by Kubernetes.
• Return a custom status (e.g., 404) on another endpoint consumed by the ingress to signal temporary overload.

I can create a custom endpoint without using MP Health, but since I love using MP Health and sticking to standards, I wanted to discuss a potential evolution to to meet this need.

The goal is to that Kubernetes behavior remains consistent, while the ingress can still make traffic decisions for overloaded pods without breaking session stickiness.

Headers can be added as they don't really mean anything to k8s

@xstefank You mean it's possible to add headers to response generated by MP health?

I know that this evolution doesn’t directly target Kubernetes pod health management, but it does target Ingress Controllers. There is no standard yet in this area. Today, HAProxy is probably ahead of the others by offering this drain-mode optimization; maybe others will follow in a different way.

Ideally, this should be done via headers rather than status codes. That way, the same endpoint can be consumed by Kubernetes’ readiness/liveness probes while the Ingress Controller interprets the headers for traffic routing.

My proposal is the following:
Create a third state that can be mapped to a customizable status code, for example: [UP, DOWN, DRAIN].
This capability could be implemented either via Wellness or a new type of Health annotation, while leaving liveness and readiness probes unchanged.

[xstefank]

I mean that if k8s doesn't care about headers, there is no harm in adding it to our API. But third state in this regard would be too redundant.

If you need another endpoint for non-standard health checks, this is exactly why we introduced different health groups as I linked before in SR Health.