Summary
When nodes are removed from the cluster or no longer match label selectors, the operator immediately deletes pool members from the load balancer without first draining active connections. This can cause connection drops and service disruption for clients with active sessions.
Current Behavior
When a Kubernetes node is deleted or its labels change such that it no longer matches the ExternalLoadBalancer CR's selection criteria, the operator calls DeletePoolMember() directly, which issues an immediate DELETE request to the load balancer API. Any active connections to that pool member are terminated abruptly.
Example flow today:
- Node loses label or is deleted from cluster
- Operator reconciles and detects node should be removed from pool
DeletePoolMember() is called immediately- Active client connections are dropped
Desired Behavior
Implement a graceful drain period before removing pool members. The operator should:
- Disable the pool member first - This prevents new connections from being routed to the member while allowing existing connections to complete
- Wait for a configurable drain period - Allow time for active connections to finish gracefully
- Remove the pool member - Only after the drain period has elapsed
Example flow with graceful draining:
- Node loses label or is deleted from cluster
- Operator reconciles and detects node should be removed from pool
- Operator disables the pool member (e.g.,
session: user-disabled on F5)
- Operator waits for drain timeout (configurable, e.g., 30s default)
DeletePoolMember() is called after drain period
Proposed Implementation
CRD Changes
Add optional drain configuration to the ExternalLoadBalancer spec:
apiVersion: lb.lbconfig.carlosedp.com/v1
kind: ExternalLoadBalancer
metadata:
name: externalloadbalancer-infra-sample
spec:
vip: "192.168.1.45"
type: "infra"
ports:
- 80
- 443
# New drain configuration
drain:
enabled: true
timeoutSeconds: 30 # Default: 30, time to wait after disabling before removal
provider:
vendor: F5_BigIP
# ... rest of provider configBackend Provider Interface Changes
The Provider interface could be extended with an optional drain capability:
// Optional interface for providers that support graceful draining
type DrainableProvider interface {
Provider
// DisablePoolMember disables a member (stops new connections, allows existing to finish)
DisablePoolMember(m *lbv1.PoolMember, pool *lbv1.Pool) error
}Controller Logic Changes
The pool member removal logic would change from:
// Current: immediate deletion
err := backend.DeletePoolMember(member, pool)
To:
// New: graceful drain then delete
if drainEnabled {
if drainable, ok := backend.(DrainableProvider); ok {
// Disable the member first
err := drainable.DisablePoolMember(member, pool)
if err != nil {
return err
}
// Requeue to check again after drain timeout
return ctrl.Result{RequeueAfter: drainTimeout}, nil
}
}
// Delete after drain period (or immediately if drain not supported/enabled)
err := backend.DeletePoolMember(member, pool)Provider-Specific Implementation
F5 BigIP:
The go-bigip library already supports this via PoolMemberStatus():
func (p *F5Provider) DisablePoolMember(m *lbv1.PoolMember, pool *lbv1.Pool) error {
memberName := fmt.Sprintf("%s:%d", m.Node.Name, m.Port)
return p.client.PoolMemberStatus(pool.Name, memberName, "user-disabled")
}Citrix ADC (Netscaler):
The EditPoolMember() function already accepts a status parameter:
func (p *NetscalerProvider) DisablePoolMember(m *lbv1.PoolMember, pool *lbv1.Pool) error {
return p.EditPoolMember(m, pool, "disable")
}HAProxy:
HAProxy Dataplane API supports setting server state to drain or maint.
Use Cases
-
Production traffic during node maintenance - When cordoning/draining nodes for maintenance, active user sessions should complete gracefully rather than being dropped mid-request.
-
Rolling cluster upgrades - During OpenShift/Kubernetes upgrades, nodes are cycled. Graceful draining prevents users from experiencing connection resets.
-
Autoscaling - In environments with cluster autoscaling, nodes may be removed during scale-down events. Long-running connections (WebSockets, streaming, large file transfers) should be allowed to complete.
-
Label-based routing changes - When changing which nodes handle ingress traffic by updating labels, the transition should be seamless for end users.
Environment
- Operator version: v0.5.1
- Load Balancer: F5 BigIP 15.x (also applicable to Citrix ADC, HAProxy)
- Kubernetes: OpenShift 4.x
Additional Context
The underlying go-bigip library already has the capability to disable pool members via PoolMemberStatus(). Similarly, the Netscaler provider's EditPoolMember() already supports enable/disable status changes. The main work would be in the controller reconciliation logic to implement the two-phase removal process.
I'm happy to contribute a PR for this feature if the maintainers are interested and can provide guidance on the preferred implementation approach.
Related
Summary
When nodes are removed from the cluster or no longer match label selectors, the operator immediately deletes pool members from the load balancer without first draining active connections. This can cause connection drops and service disruption for clients with active sessions.
Current Behavior
When a Kubernetes node is deleted or its labels change such that it no longer matches the
ExternalLoadBalancerCR's selection criteria, the operator callsDeletePoolMember()directly, which issues an immediate DELETE request to the load balancer API. Any active connections to that pool member are terminated abruptly.Example flow today:
DeletePoolMember()is called immediatelyDesired Behavior
Implement a graceful drain period before removing pool members. The operator should:
Example flow with graceful draining:
session: user-disabledon F5)DeletePoolMember()is called after drain periodProposed Implementation
CRD Changes
Add optional drain configuration to the
ExternalLoadBalancerspec:Backend Provider Interface Changes
The
Providerinterface could be extended with an optional drain capability:Controller Logic Changes
The pool member removal logic would change from:
To:
Provider-Specific Implementation
F5 BigIP:
The go-bigip library already supports this via
PoolMemberStatus():Citrix ADC (Netscaler):
The
EditPoolMember()function already accepts a status parameter:HAProxy:
HAProxy Dataplane API supports setting server state to
drainormaint.Use Cases
Production traffic during node maintenance - When cordoning/draining nodes for maintenance, active user sessions should complete gracefully rather than being dropped mid-request.
Rolling cluster upgrades - During OpenShift/Kubernetes upgrades, nodes are cycled. Graceful draining prevents users from experiencing connection resets.
Autoscaling - In environments with cluster autoscaling, nodes may be removed during scale-down events. Long-running connections (WebSockets, streaming, large file transfers) should be allowed to complete.
Label-based routing changes - When changing which nodes handle ingress traffic by updating labels, the transition should be seamless for end users.
Environment
Additional Context
The underlying go-bigip library already has the capability to disable pool members via
PoolMemberStatus(). Similarly, the Netscaler provider'sEditPoolMember()already supports enable/disable status changes. The main work would be in the controller reconciliation logic to implement the two-phase removal process.I'm happy to contribute a PR for this feature if the maintainers are interested and can provide guidance on the preferred implementation approach.
Related
PoolMemberStatusfunction: https://pkg.go.dev/github.com/scottdware/go-bigip#BigIP.PoolMemberStatus