Adding GEP-3539: Gateway API to Expose Pods on Cluster-Internal IP Address (ClusterIP Gateway) #3608
Conversation
k8s-ci-robot
added
release-note-none
k8s-ci-robot
added
cncf-cla: no
|
Hi @ptrivedi. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/L
ptrivedi
force-pushed
the
gep-clusterip-gateway
branch
from
afc6467 to
835e6a3
Compare
…dress (ClusterIP Gateway) Signed-off-by: Pooja Trivedi [email protected]
ptrivedi
force-pushed
the
gep-clusterip-gateway
branch
from
835e6a3 to
6a061ca
Compare
k8s-ci-robot
added
cncf-cla: yes
|
Adding this comment here for tracking a few open items resulting from the comments on the google doc here: https://docs.google.com/document/d/1N-C-dBHfyfwkKufknwKTDLAw4AP2BnJlnmx0dB-cC4U/edit?tab=t.0
|
| potentially other resource kinds) directly to a Route via backendRef. | ||
|
|
||
| ```yaml | ||
| {% include 'standard/clusterip-gateway/tcproute-with-endpointselector.yaml' %} |
There was a problem hiding this comment.
these does not render. or does it suppose to be like this? see https://github.com/ptrivedi/gateway-api/blob/gep-clusterip-gateway/geps/gep-3539/index.md
There was a problem hiding this comment.
Yes, it is. Please see the deploy preview here: https://deploy-preview-3608--kubernetes-sigs-gateway-api.netlify.app/geps/gep-3539/
Also added this to the PR description:
Recommend reviewing deploy preview so examples are inlined: https://deploy-preview-3608--kubernetes-sigs-gateway-api.netlify.app/geps/gep-3539/
| is to have a GatewayClass corresponding to each type of service networking behavior that needs to be modeled | ||
| and supported. | ||
|
|
||
|  |
There was a problem hiding this comment.
image missing or incorrect file name?
There was a problem hiding this comment.
Was a missing image. Fixed. Thanks for catching
| | Feature | ServiceAPI options | Gateway API possibilities | | ||
| |---|---|---| | ||
| | sessionAffinity | ClientIP <br /> NoAffinity | Route level | ||
| | allocateLoadBalancerNodePorts | True <br /> False | Not supported for ClusterIP Gateway <br /> Supported for LoadBalancer Gateway | |
There was a problem hiding this comment.
we could say N/A for this approach, since you can create LB type without NodePort - sort of simplification.
There was a problem hiding this comment.
That might be clearer until further discussion on each of these.
k8s-ci-robot
added
cncf-cla: no
ptrivedi
force-pushed
the
gep-clusterip-gateway
branch
3 times, most recently
from
1e793b0 to
b5e81ee
Compare
k8s-ci-robot
added
cncf-cla: yes
* Fix missing image * Change GEP status to Memorandum * Make GEP navigable * Crop trailing whitespace from images Signed-off-by: Pooja [email protected]
ptrivedi
force-pushed
the
gep-clusterip-gateway
branch
from
b5e81ee to
e876ced
Compare
|
/assign @thockin |
| @@ -0,0 +1,25 @@ | |||
| kind: TCPRoute/CustomRoute | |||
There was a problem hiding this comment.
Is this syntax Foo/Bar for the example or is it somethign real? I don't think I have ever seen it and I don't know what it means
There was a problem hiding this comment.
From the context, it appears that this is covering the base of "I'm not sure what actual Kind we're talking about here".
@ptrivedi - if that's what you mean, I'd recommend leaving a comment next to it to explain and/or using an optional-selection notation like [TCPRoute|CustomRoute].
There was a problem hiding this comment.
It was meant to say "It could be a TCPRoute or another kind of L4 route, custom or not". Will modify to incorporate @youngnick's suggestion.
| - name: example-cluster-ip-gateway | ||
| rules: | ||
| config: | ||
| sessionAffinity: false |
There was a problem hiding this comment.
Not just the indent, this config section was meant to be under spec. Likely a copy/paste foobar, will fix.
To explain further, I had brought this up as an open question/discussion point during the community meeting presentation. For Service features like internalTrafficPolicy, sessionAffinity, etc. (and possibly other things in the future), does it make sense to have a RouteConfig section under RouteSpec? FWIW, for the internal implementation of ClusterIP Gateway we are doing for multi-network environments, we use a custom route where we had some debate around whether these parameters should go directly under RouteSpec or in a separate section under RouteSpec.
The table towards the end of the document lists Service API features and the possibility of translating those to Gateway API. But the discussion has evolved further since, and it may make sense to think holistically about Gateway API functionality enhancements rather than simply thinking in terms of Service API mapping/parity. For example, Gateway aiming at a more generic model of topology routing functionality than features like internalTrafficPolicy in Service API that are restrictive and somewhat confusing. We likely need an overall discussion around having something like 'trafficDistribution' (or whatever we decide to call it in GW world) and where it should live (Do we propose a Route extension? Or does it belong at Gateway level?). Same for some of the other features.
There was a problem hiding this comment.
There are parts of the service API that are deeply entangled with the implementation details. If we can untangle them, great, but they likely are not "generic" enough to apply to just any Gateway, or even to any L4 or L3 Gateway.
Let's look at the table from below:
sessionAffinity: seems generally applicable to any L3 or L4 GW
allocateLoadBalancerNodePorts: should be moot if NodePort is a different gateway than clusterIP or LB
externalIPs: a gross hack that perhaps we can abandon
externalTrafficPolicy: by definition does not apply to clusterIPs
internalTrafficPolicy: by definition ONLY applies to cluster IPs. It's not clear to me if we could have a generic L4Route.TrafficPolicy sort of thing. It makes sense for ClusterIP but not for a generic L4 or L3 route? I think? It feels "special", but I am not 100% sure
trafficDiustirbution: definitely matters for clusterIP, maybe is generic L4?
ipFamily: this is a request to the GW IP allocator, so it probably makes sense as a parameter to the Gateway? How do we express v4 vs v6 today?
publishNotReadyAddresses: This is related to endpoint selection which you are trying to factor out, but it IS used.
headless service: I feel like this is a different "kind" of Gateway. The fact that it was implemented as a variation of ClusterIP rather than a distinct value for Type is a historical thing.
externalName: Again, this is probably a different kind of gateway
type: obviated - the class of Gateway should capture the same concept space
I can't tell if I just argued for a generic L4 Route or for a special VirtualService route or something else?
There was a problem hiding this comment.
ipFamily: this is a request to the GW IP allocator, so it probably makes sense as a parameter to the Gateway? How do we express v4 vs v6 today?
We don't. When requesting a static address, you can request either type (by string conversion, not by structured field), and it's up to the implementation to figure out if it can fulfill the request, and also what addresses to give if you don't specify anything. Note that implementations are totally allowed to give out multiple addresses here if no static addresses are speciffied.
We do have a type field in the addresses config, so we can extend there as needed though.
We also have #3616 to make the value optional, so that you can request an dynamically-allocated address of a specific type. Combine those two together, and we could add v4 and v6 specific types that could be validated more clearly or have you request a specific address type only.
There was a problem hiding this comment.
Yeah, pre-allocated IPs matter a lot more for "north-south" than "east-west", but it seems inevitable that people want to express things like "IPv6 but you choose the address" or "dual-stack if possible, but single-stack is OK".
The Service ipFamilyPolicy covers that and I will be shocked if you don't need essentially the same gamut of config.
|
|
||
| ### EndpointSelector as Backend | ||
|
|
||
| A Route can forward traffic to the endpoints selected via selector rules defined in EndpointSelector. |
There was a problem hiding this comment.
FWIW, I can imagine a path toward maybe making this a regular core feature. I am sure that it would be tricky but I don't think it's impossible.
Eg.
Define a Service with selector foo=bar. That triggers us to create a PodSelector for foo=bar. That triggers the endpoints controller(s) to do their thing. Same as we do with IP.
There was a problem hiding this comment.
Interesting thought.
For starters at least, there seemed to be agreement on having a GEP for EndpointSelector as the next step.
There was a problem hiding this comment.
As always, Gateway proves something is a good idea, then core steals the spotlight.
| apiVersion: gateway.networking.k8s.io/v1 | ||
| kind: GatewayClass | ||
| metadata: | ||
| name: cluster-ip |
There was a problem hiding this comment.
Is this name "special" or can it be anything?
There was a problem hiding this comment.
It's intended that GatewayClass names can be any valid Kubernetes object name.
There was a problem hiding this comment.
See expanded question under https://github.com/kubernetes-sigs/gateway-api/pull/3608/files#r1964558745
| metadata: | ||
| name: cluster-ip | ||
| spec: | ||
| controllerName: "cluster-ip-controller" |
There was a problem hiding this comment.
Is this name "special" or can it be anything?
There was a problem hiding this comment.
The name can be anything but implementations must only reconcile GatewayClasses that has a controllerName that they expect. GatewayClass objects that do not match an implementation's controllerName must ignore that GatewayClass completely, and not update it at all (to prevent fighting on status).
Some implementations allow configuration of this string (for example, Contour allows it so that you can run multiple instances of Contour in a cluster).
There was a problem hiding this comment.
Is that the behavior we want here? In Service, its a single object with (many) multiple controllers consuming it. If I want my service exposed to the CNI, kube-proxy, service mesh, observability platform, ... do I need to make N Gateways?
There was a problem hiding this comment.
See expanded question under https://github.com/kubernetes-sigs/gateway-api/pull/3608/files#r1964558745
Agree with John's question, and I think it betrays a fundamental difference in perspective. I see this idea as "Services with a better API"
There was a problem hiding this comment.
Because we're using the same object that can be used in other contexts though (ie Gateway), we need a way to disambiguate, and the way we have is GatewayClass. I'd be happy to see proposals around alternatives to GatewayClass, but I haven't seen anything to date that handles the problem that implementations of Gateway API almost always need multiple-namespace access, and the only currently available thing we have that's bigger than a single namespace is cluster-wide.
| name: example-cluster-ip-gateway | ||
| spec: | ||
| addresses: | ||
| - 10.12.0.15 |
There was a problem hiding this comment.
How does kube-proxy (or Cilium or Antrea or ...) know which Gateways it should be capturing traffic for?
There was a problem hiding this comment.
Normally that's handled by the rollup of Gateway -> GatewayClass. Implementations own GatewayClasses that specify the correct string in GatewayClass spec.controllerName. All Gateways in that GatewayClass in that GatewayClass would need to be serviced by an implementation that can fulfill this request (that is, it both has the required functionality, and, in this case of requesting a static address, is actually able to assign that address). In the case that an implementation cannot fulfil this Gateway for some reason, it must be marked as not Accepted (by having an Accepted type condition in the Gateway's status with status: false).
There was a problem hiding this comment.
I can't tell if you are giving me a hard time or not :)
What I meant to ask is:
Service as a built-in API is (more or less) universally implemented by on-node agents (kube-proxy, cilium or antrea, ovn, etc). If we are trying to offer a form of ClusterIP Gateway which replaces part of the Service API, how does a user express "this is a cluster IP gateway" in a portable way such that all of the implementations know "this is for me"?
If each implementation has its own controllerName, and the GatewayClass can be named anything the cluster admin wants, how does our poor beleaguered app operator know what to put in their YAML?
Today they can say:
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
type: ClusterIP
selector:
foo: bar
ports:
- port: 8080
...and be confident that ANY cluster, regardless of which CNI, will allocate a virtual IP and route traffic.
I'd like to write a generic tool which does:
for each service S in `kubectl get svc -A` {
evaluate template with S to produce an equivalent Gateway
}
There was a problem hiding this comment.
Yeah, okay, I see the use case, but this is the problem with extensions v core - we left the flexibility there for implementations, (for good reason), and now we don't have a way to define a default GatewayClass at all, even for specific use cases.
I think that practically, a tool like you describe would need to know the gatewayclass it was targeting, and output Gateways based on that.
We could conceivably have a convention and pick a reserved name (like cni-clusterip or something), but we've been reluctant in the past to do that, preferring the increased specificity of requiring people to specify something (even though there is a friction cost to be paid there).
(And I wasn't trying to give you a hard time - I have details get pushed out of my head all the time, so wanted to make sure this hadn't happened here. 😄 But also, I wanted to help other readers understand too)
There was a problem hiding this comment.
I think that practically, a tool like you describe would need to know the gatewayclass it was targeting,
Hence my questions about "is this name special". One answer is "thou shalt use the name 'clusterip' and the 'clusterip' is the name thou shalt use", and just hope not to collide with users. Another answer is to define a sub-space of names that users can't currently use, or are exceedingly unlikely to be using e.g. k8s.io:clusterip. This is an appropriate place to ideate, right?
| {% include 'standard/clusterip-gateway/clusterip-gateway.yaml' %} | ||
| ``` | ||
|
|
||
| By default, IP address(es) from a pool specified by a CIDR block will be assigned unless a static IP is |
There was a problem hiding this comment.
I think the default path should be to allocate from the same ServiceCIDR resource. If you need an IP from a different resource you would do something different. Either a different class or a different allocator or something.
| in pods’ /etc/resolv.conf need to be programmed accordingly by kubelet. | ||
|
|
||
| ``` | ||
| <name of gateway>.<gateway-namespace>.gw.cluster.local |
There was a problem hiding this comment.
I think DNS is a fraught topic. We REALLY REALLY do not want to add more search paths, especially if they could cause ambiguous names. We could just lean on the "svc" space for this, since these are effectively services. We would need to define how to avoid collisions and I'd be lying if I said I had a great answer.
Maybe, like IPAddress, we extract ServiceName to new resource, and whomever gets there first wins? That sort of transaction doesn't work well for CRDs but I guess it could be async. Weird failure modes.
|
|
||
| | Feature | ServiceAPI options | Gateway API possibilities | | ||
| |---|---|---| | ||
| | sessionAffinity | ClientIP <br /> NoAffinity | Route level |
| | sessionAffinity | ClientIP <br /> NoAffinity | Route level | ||
| | allocateLoadBalancerNodePorts | True <br /> False | Not supported for ClusterIP Gateway <br /> Supported for LoadBalancer Gateway | | ||
| | externalIPs | List of externalIPs for service | Not supported? | | ||
| | externalTrafficPolicy | Local <br /> Cluster | Supported for LB Gateways only, Route level | |
There was a problem hiding this comment.
These are all interesting challenges which maybe need something more than a plain TCP Route?
There was a problem hiding this comment.
A more generic L4Route that combines TCP/UDPRoutes?
|
|
||
| When modeling ClusterIP service networking, the simplest recommendation might be to keep Gateway and Routes | ||
| within the same namespace. While cross namespace routing would work and allow for evolved functionality, | ||
| it may make supporting certain cases tricky. One specific example for this case is the pod DNS resolution |
There was a problem hiding this comment.
Given we are making a new DNS name, do we actually care to support this POD-IP DNS name?
There was a problem hiding this comment.
The pod-ip DNS name is for headless only: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#:~:text=Any%20Pods%20exposed,domain.example.
And that was before DNS specification. After DNS specification, the record is of the format
pod-hostname.service-name.my-namespace.svc.cluster-domain.example
Headless, in case of Gateway API would likely be expressed using a separate GatewayClass, in order to avoid conflating ClusterIP and Headless and provide clean separation of concerns. So, you are right, does not necessarily need to be supported for ClusterIP Gateway.
Headless and externalName that are more DNS specific warrant more discussion
There was a problem hiding this comment.
If we end up in a new DNS space, this whole proposal's value is diminished. IMO.
| Note that Gateway API allows flexibility and clear separation of concerns so that one would not need to | ||
| configure cluster-ip and node-port when configuring a load-balancer. | ||
|
|
||
| But for completeness, the case shown below demonstrates how load balancer functionality analogous to |
There was a problem hiding this comment.
All of this proposal makes sense as a logic way to solve "If you had to implement Service using Gateway API primitives, how would you do it".
What doesn't make sense to me is the why and the how this becomes something practically useful from a proposal to a thing in the real world.
The diagram below shows 1 object becoming 8. Do we expect users to actually create these 8 objects?
Which projects are expected to, and which are commited to, supporting these? Kube-proxy? Coredns? Various 3p CNIs (Cilium, calico, etc)? Service meshes? All gateway implementations?
| ## Goals | ||
|
|
||
| * Define Gateway API usage to accomplish ClusterIP Service style behavior | ||
| * Propose DNS layout and record format for ClusterIP Gateway |
There was a problem hiding this comment.
It doesn't seem like we have fleshed this out. Compared to https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ we have just 1-2 sentences with a lot of ambiguity here.
There was a problem hiding this comment.
That is correct, we haven't. DNS bit has been identified as a good candidate to be split out into its own GEP
There was a problem hiding this comment.
I've expressed elsewhere, but will say again - my ideal outcome is that users could 1-for-1 convert Services (maybe not "while running", but convert YAML) into this new form of Gateway and not change their clients at all. Ideally this new thing gets the same name as the equivalent Service.
| # GEP-3539: ClusterIP Gateway - Gateway API to Expose Pods on Cluster-Internal IP Address | ||
|
|
||
| * Issue: [#3539](https://github.com/kubernetes-sigs/gateway-api/issues/3539) | ||
| * Status: Memorandum |
There was a problem hiding this comment.
This should currently be Provisional, as it's the first iteration and we are still deciding on the approach here.
The Memorandum status is for registering general agreement about things, not for features that will require actual code changes to the Gateway API specification (which this definitely will).
This also needs to be changed in the corresponding metadata.yaml file - the YAML file is actually the canonical place for the status, this is just to remind everyone. I'll suggest the same change there.
| name: example-cluster-ip-gateway | ||
| spec: | ||
| addresses: | ||
| - 10.12.0.15 |
There was a problem hiding this comment.
Normally that's handled by the rollup of Gateway -> GatewayClass. Implementations own GatewayClasses that specify the correct string in GatewayClass spec.controllerName. All Gateways in that GatewayClass in that GatewayClass would need to be serviced by an implementation that can fulfill this request (that is, it both has the required functionality, and, in this case of requesting a static address, is actually able to assign that address). In the case that an implementation cannot fulfil this Gateway for some reason, it must be marked as not Accepted (by having an Accepted type condition in the Gateway's status with status: false).
| apiVersion: gateway.networking.k8s.io/v1 | ||
| kind: GatewayClass | ||
| metadata: | ||
| name: cluster-ip |
There was a problem hiding this comment.
It's intended that GatewayClass names can be any valid Kubernetes object name.
| metadata: | ||
| name: cluster-ip | ||
| spec: | ||
| controllerName: "cluster-ip-controller" |
There was a problem hiding this comment.
The name can be anything but implementations must only reconcile GatewayClasses that has a controllerName that they expect. GatewayClass objects that do not match an implementation's controllerName must ignore that GatewayClass completely, and not update it at all (to prevent fighting on status).
Some implementations allow configuration of this string (for example, Contour allows it so that you can run multiple instances of Contour in a cluster).
| @@ -0,0 +1,25 @@ | |||
| kind: TCPRoute/CustomRoute | |||
There was a problem hiding this comment.
From the context, it appears that this is covering the base of "I'm not sure what actual Kind we're talking about here".
@ptrivedi - if that's what you mean, I'd recommend leaving a comment next to it to explain and/or using an optional-selection notation like [TCPRoute|CustomRoute].
| (Gateway resource), implementation specifics and common configuration (GatewayClass | ||
| resource), and routing traffic to backends (Route resource). | ||
|
|
||
| ### Limitations of Service API |
There was a problem hiding this comment.
I think we need to be realistic here and acknowledge the benefits of the Service API from a user's POV - which I think we could summarize as that, for simple use cases, its very simple. It's only one object, as opposed to (at minimum) four in the simplest case here (GatewayClass, Gateway, Route, and EndpointSelector).
I completely agree that breaking Service apart for more advanced use cases is useful, but we should acknowledge the reason why it's stuck around for so long - the level of simplicity and flexibility it has allows folks to get started much more easily. Additionally, Service is a GA API that's not going anywhere, so we need to be very clear that we're not talking about deprecating or replacing Service with this. As with Gateway API north/south and Ingress, the GA core resource is going to stick around, but this proposal is about giving us a better base to look at adding features to rather than trying to fit them into the existing, overloaded Service construct.
Speaking from experience, putting a section outlining this into this document now will save a lot of discussion later.
There was a problem hiding this comment.
💯 agree, which is why, IMO, the best path forward here is making Service the higher level API that decomposes into these resources. Then you can chose to break the abstraction and manually configure the underlying resources (as you can do today by manually creating EndpointSlice!).
Otherwise we end up with a split universe indefinitely
There was a problem hiding this comment.
Whether literally core/v1.Service gets decomposed into these or some new type which is "familiar but better" or some embedding into Gateway itself, I think I agree. We need not fear layering.
Co-authored-by: Nick Young <[email protected]>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: ptrivedi The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Co-authored-by: Nick Young <[email protected]>
| {% include 'standard/clusterip-gateway/tcproute-with-endpointselector.yaml' %} | ||
| ``` | ||
|
|
||
| The EndpointSelector object is defined as follows. It allows the user to specify which endpoints |
There was a problem hiding this comment.
Would it make sense to have a config field so we can have implementation specific parameters?
For example, if I create a Layer 3/4 load balancer type of gateway, I would like to express how the traffic will be distributed (what algorithm is being used), the maximum number of endpoints and from where IP addresses should be selected (ResourceClaim (Multi-Network), Annotations (Multus)...).
| | externalIPs | List of externalIPs for service | Not supported? | | ||
| | externalTrafficPolicy | Local <br /> Cluster | Supported for LB Gateways only, Route level | | ||
| | internalTrafficPolicy | Local <br /> Cluster | Supported for ClusterIP Gateways only, Route level | | ||
| | ipFamily | IPv4 <br /> IPv6 | Route level | |
There was a problem hiding this comment.
If the IPs are specified only in the Gateway and not in the Routes, why would the ipFamily be at the Route Level?
Or should there be a Layer 3 and 4 Route that contains the Ports, Protocol and IPs (+ipFamily)?
Recommend reviewing deploy preview so examples are inlined: https://deploy-preview-3608--kubernetes-sigs-gateway-api.netlify.app/geps/gep-3539/
Signed-off-by: Pooja Trivedi [email protected]
What type of PR is this?
/kind gep
What this PR does / why we need it:
This defines via documentation how Gateway API can be used to accomplish ClusterIP Service behavior. It also proposes DNS record format for ClusterIP Gateway, proposes an EndpointSelector resource, and briefly touches upon Gateway API usage to define LoadBalancer and NodePort behaviors.
Which issue(s) this PR fixes:
Fixes #3539
Does this PR introduce a user-facing change?: