From 1fee9351cc5d73fe0b5656e102d2d9e6c6897860 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 21 Aug 2025 23:57:39 +0000 Subject: [PATCH 1/7] A102: xDS GrpcService Support --- A102-xds-grpc-service.md | 297 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 297 insertions(+) create mode 100644 A102-xds-grpc-service.md diff --git a/A102-xds-grpc-service.md b/A102-xds-grpc-service.md new file mode 100644 index 000000000..52cea7377 --- /dev/null +++ b/A102-xds-grpc-service.md @@ -0,0 +1,297 @@ +A102: xDS `GrpcService` Support +---- +* Author(s): @markdroth +* Approver: @ejona86, @dfawley +* Status: {Draft, In Review, Ready for Implementation, Implemented} +* Implemented in: +* Last updated: 2025-08-21 +* Discussion at: (filled after thread exists) + +## Abstract + +There are several features that require the xDS control plane to configure +gRPC to talk to a side-channel service, such as rate limiting ([A77]), +ExtAuthz ([A92]), and ExtProc ([A93]). This design specifies how the +control plane will configure the communication with these side-channel +services. It also addresses the relevant security implications. + +## Background + +The control plane configures communication with side-channel services +via the xDS [`GrpcService` +proto](https://github.com/envoyproxy/envoy/blob/7ebdf6da0a49240778fd6fed42670157fde371db/api/envoy/config/core/v3/grpc_service.proto#L29). +This message tells the data plane how to find the side-channel service +and what channel credentials and call credentials to use for that +communication. + +### Related Proposals: +* [gRFC A27: xDS-Based Global Load Balancing][A27] +* [gRFC A29: xDS-Based mTLS Security for gRPC Clients and Servers][A29] +* [gRFC A77: xDS Server-Side Rate Limiting][A77] (WIP) +* [gRFC A81: xDS Authority Rewriting][A81] +* [gRFC A92: xDS ExtAuthz Support][A92] (WIP) +* [gRFC A93: xDS ExtProc Support][A93] (WIP) +* [A97: xDS JWT Call Credentials][A97] + +[A27]: A27-xds-global-load-balancing.md +[A29]: A29-xds-tls-security.md +[A77]: https://github.com/grpc/proposal/pull/414 +[A81]: A81-xds-authority-rewriting.md +[A92]: https://github.com/grpc/proposal/pull/481 +[A93]: https://github.com/grpc/proposal/pull/484 +[A97]: A97-xds-jwt-call-creds.md + +## Proposal + +gRPC will support the `GrpcService` +message. In that message, gRPC will support only the +[`GoogleGrpc`](https://github.com/envoyproxy/envoy/blob/7ebdf6da0a49240778fd6fed42670157fde371db/api/envoy/config/core/v3/grpc_service.proto#L68) +target specifier, not +[`EnvoyGrpc`](https://github.com/envoyproxy/envoy/blob/7ebdf6da0a49240778fd6fed42670157fde371db/api/envoy/config/core/v3/grpc_service.proto#L33) +(see "Rationale" section below). + +### Security Considerations + +The control plane specifying the side-channel target and credentials +introduces a number of potential privilege-escalation attacks from a +compromised control plane. Here are some examples of such attacks: + +- Because the side-channel target name comes from the control plane + rather than being configured locally on the client, a compromised + control plane can tell the client to talk to an attacker-controlled + side-channel service. When used for functionality like ExtProc, this + would allow the control plane to get access to the contents of data + plane RPCs. + + - Note: Even if the client could enforce the use of a channel credential + type like TLS that verifies that the server's identity matches the + target name, that would not ameliorate this attack, because the + target name itself is coming from the control plane. + + - Note: Even if the client locally configured the target name of + the side-channel service but trusted the control plane to specify + the credential type, the control plane could specify + `InsecureCredentials`, and then it would just need to control the + client's name resolution in order to send the client to an + attacker-controlled side-channel service. + +- A compromised control plane could instruct gRPC to contact an + attacker-controlled side-channel service using a call credential that + sends an access token, which would leak that access token. + +There will be cases where it is acceptable to trust the control plane +to have that kind of privilege-escalation capability, and there will be +other cases where it is not. To differentiate between these two cases, we +will rely on the `trused_xds_server` server feature that was added to the +gRPC xDS bootstrap config in [A81]. + +When gRPC receives a `GrpcService` proto from an xDS server, it will +check to see if the `trusted_xds_server` server feature is present in +the bootstrap config for that xDS server. If so, then it will trust +the target name and credentials specified in the `GrpcService` proto. +If not, then we will provide a mechanism for it to obtain side-channel +configuration locally from the gRPC xDS bootstrap config. + +Specifically, we will add the following new top-level field to the +bootstrap config: + +```json5 +// The list of side-channel services allowed to be configured via xDS. +"allowed_grpc_services": { + // The key is fully-qualified target URI. + "dns:///xds.example.org:443": { + // List of channel creds. Client will stop at the first type it + // supports. This field is required and must contain at least one + // channel creds type that the client supports. + "channel_creds": [ + // The format is identical to xds_servers.channel_creds. + { + "type": , + // The "config" field is optional; it may be missing if the + // credential type does not require config parameters. + "config": + } + ] + // List of call creds. Optional. Client will apply all call creds + // types that it supports but will ignore any types that it does not + // support. + "call_creds": [ + { + "type": , + // The "config" field is optional; it may be missing if the + // credential type does not require config parameters. + "config": + } + ] + } +} +``` + +Note that the `channel_creds` and `call_creds` fields follow the same +format as the top-level `xds_servers` field. See [A27] and [A97] for +details. + +When gRPC receives a `GrpcService` proto from an untrusted control +plane, it will look up the target URI from the `GrpcService` proto in +the `allowed_grpc_services` map. If the specified target URI is not +present in the map, then the `GrpcService` proto will be considered +invalid, resulting in gRPC NACKing the xDS resource. If the specified +target URI *is* present in the map, then the `GrpcService` proto will be +considered valid, but gRPC will ignore the credential information from +the `GrpcService` proto; instead, it will use the channel credentials +and call credentials specified in the map. + +### Credential Configuration in `GrpcService` Proto + +In order to make channel and call credentials more pluggable, we are +introducing new extension points in `GrpcService`, as show in +https://github.com/envoyproxy/envoy/pull/40823. Specifically, this +introduces the following new fields: + +- `channel_credentials_plugin`: This provides an extension point to + specify channel credentials. Just like in the gRPC xDS bootstrap + format, in order to faciliate easier introduction of new credential + types, this field is structured as a list, and the client will iterate + over the list and stop at the first credential type that it supports. + If it does not find any supported credential type in the list, that is + a validation error, and the xDS resource will be NACKed. If it finds + a supported credential type but the config is invalid, then the xDS + resource will also be NACKed. The following extensions will be supported + in this field: + + - `envoy.extensions.grpc_service.channel_credentials.google_default.v3.GoogleDefaultCredentials` + - `envoy.extensions.grpc_service.channel_credentials.insecure.v3.InsecureCredentials` + - `envoy.extensions.grpc_service.channel_credentials.local.v3.LocalCredentials` + - `envoy.extensions.grpc_service.channel_credentials.tls.v3.TlsCredentials`: + In this message: + - `root_certificate_provider`: Required. References certificate + provider instances configured at the top level of the bootstrap + config. Validated the same way as in `CommonTlsContext` (see [A29]). + - `identity_certificate_provider`: Optional. References certificate + provider instances configured at the top level of the bootstrap + config. Validated the same way as in `CommonTlsContext` (see [A29]). + - `envoy.extensions.grpc_service.channel_credentials.xds.v3.XdsCredentials`: + In this message: + - `fallback_credentials`: Required. Specifies a channel credential + plugin to be used as fallback credentials. + +- `call_credentials_plugin`: This provides an extension point to specify + call credentials. Just like in the gRPC xDS bootstrap config format, + in order to faciliate easier introduction of new credential types, + this field is structured as a list, and the client will iterate over + the list adding all credential types that it supports, ignoring any + type that it does not support. Note that unlike channel credentials, + call credentials are optional, and there can be more than one, so the + client will need to iterate over the entire list, but it's valid if + none of the specified types are supported. If the client finds a + supported credential type but the config is invalid, then the xDS + resource will be NACKed. The following extensions will be supported + in this field: + + - `envoy.extensions.grpc_service.call_credentials.google_compute_engine.v3.GoogleComputeEngineCredentials` + +Note that this will require extending the channel credentials and call +credentials registries to support configuration via these protos, in +additional to the JSON formats that they already support for the gRPC +xDS bootstrap config. + +### `GrpcService` Proto Validation + +When validating a `GrpcService` proto, the following fields will be used: +- [`google_grpc`](https://github.com/envoyproxy/envoy/blob/7ebdf6da0a49240778fd6fed42670157fde371db/api/envoy/config/core/v3/grpc_service.proto#L303): + This field must be set. Inside of it: + - [`target_uri`](https://github.com/envoyproxy/envoy/blob/7ebdf6da0a49240778fd6fed42670157fde371db/api/envoy/config/core/v3/grpc_service.proto#L254): + Must be set to a valid gRPC target URI. The target URI must be + checked against the resolver registry during xDS resource + validation. + - `channel_credentials_plugin`: See above. + - `call_credentials_plugin`: See above. + - `channel_credentials`, `call_credentials`, + `credentials_factory_name`, and `config`: Ignored; we will use the + new credential plugin fields above instead. + - `stat_prefix`: Ignored; not relevant to gRPC. + - `per_stream_buffer_limit_bytes`: Ignored. We don't have a use-case + for this right now but could add it later if needed. + - `channel_args`: Ignored. Not supportable across languages in gRPC. +- [`timeout`](https://github.com/envoyproxy/envoy/blob/7ebdf6da0a49240778fd6fed42670157fde371db/api/envoy/config/core/v3/grpc_service.proto#L308): + If set, this will be used to set the deadline on RPCs sent to the + side-channel service. +- [`initial_metadata`](https://github.com/envoyproxy/envoy/blob/7ebdf6da0a49240778fd6fed42670157fde371db/api/envoy/config/core/v3/grpc_service.proto#L315): + If present, specifies headers to be added to RPCs sent to the side-channel + service. Inside of each entry: + - [`key`](https://github.com/envoyproxy/envoy/blob/7ebdf6da0a49240778fd6fed42670157fde371db/api/envoy/config/core/v3/base.proto#L404): + Value length must be in the range [1, 16384). Must be a valid + HTTP/2 header name. + - [`value`](https://github.com/envoyproxy/envoy/blob/7ebdf6da0a49240778fd6fed42670157fde371db/api/envoy/config/core/v3/base.proto#L415): + Specifies the header value. Must be shorter than 16384 bytes. Must + be a valid HTTP/2 header value. Not used if `key` ends in `-bin` + and `raw_value` is set. + - [`raw_value`](https://github.com/envoyproxy/envoy/blob/7ebdf6da0a49240778fd6fed42670157fde371db/api/envoy/config/core/v3/base.proto#L422): + Used only if `key` ends in `-bin`. Must be shorter than 16384 bytes. + Will be base64-encoded on the wire, unless the pure binary metadata + extension from [gRFC G1: True Binary + Metadata](G1-true-binary-metadata.md) is used. +- `envoy_grpc`: This field is not used. See "Rationale" section below + for details. +- `retry_policy`: This field is not used. If retries are needed, they + should be configured in the [service + config](https://github.com/grpc/grpc/blob/master/doc/service_config.md) + for the side-channel service, or by using xDS in the side-channel. + +### Temporary environment variable protection + +This gRFC does not describe a discrete feature; the functionality it +describes will be used only in the context of other features, which +will each have their own environment variable protection. Therefore, +no additional environment variable protection is needed here. + +Note that when implementing one of those other features, it will be +important for the appropriate environment variable guard to cover +reading the `allowed_grpc_services` field in the bootstrap config. + +## Rationale + +### `GoogleGrpc` vs. `EnvoyGrpc` + +Envoy supports both `GoogleGrpc` and `EnvoyGrpc` target specifiers. +The latter uses Envoy's own gRPC implementation, which is essentially +a small wrapper on top of its existing HTTP/2 functionality. Rather +than configuring the side-channel using a gRPC target URI, it specifies +the side-channel using the name of an xDS cluster, which must already be +part of the data plane's configuration. That approach does not make +sense in gRPC, for two reasons. + +First, even in Envoy, `EnvoyGrpc` makes sense only in cases where +the data plane's xDS configuration already includes a cluster for the +side-channel service; in any other case, `GoogleGrpc` would be used +instead. But unlike Envoy, gRPC is not a general-purpose proxy that +handles routing requests for multiple services in a single instance; +instead, each gRPC channel is created for one specific target (i.e., +one particular service) and generally contains only the configuration +for that service, which means that in practice its xDS configuration +never includes the xDS cluster for the side-channel service. + +Second, even if gRPC's xDS configuration did include the cluster for +the side-channel service, gRPC's architecture does not support sending +traffic to a specific cluster. Unlike Envoy, gRPC does not have a +distinct Cluster Manager that can be used to select a cluster to send +requests to, which means that it fundamentally doesn't make sense to +use an xDS cluster name to contact the side-channel service. + +### Security Concerns + +A more comprehensive approach to the security concerns would be to +provide a mechanism to cryptographically sign the xDS resources and have +the client verify the signature. This would ensure that a compromised +control plane would not be able to send arbitrary resources to clients; +instead, the attack would have to happen where the xDS resources are +constructed and signed, which could in principle be better protected. + +We are in favor of this approach, but it will require a lot more work, +so we are leaving it as a future improvement. + +## Implementation + +Will be implemented in C-core, Java, Go, and Node as part of either RLQS +([A77]), ExtAuthz ([A92]), or ExtProc ([A93]), whichever happens to be +implemented first in any given language. From 809d07f04bb9a992abfb8e1891d77de9040c1fa8 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 21 Aug 2025 23:59:49 +0000 Subject: [PATCH 2/7] fix typo --- A102-xds-grpc-service.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/A102-xds-grpc-service.md b/A102-xds-grpc-service.md index 52cea7377..06234086a 100644 --- a/A102-xds-grpc-service.md +++ b/A102-xds-grpc-service.md @@ -144,7 +144,7 @@ and call credentials specified in the map. ### Credential Configuration in `GrpcService` Proto In order to make channel and call credentials more pluggable, we are -introducing new extension points in `GrpcService`, as show in +introducing new extension points in `GrpcService`, as shown in https://github.com/envoyproxy/envoy/pull/40823. Specifically, this introduces the following new fields: From 2f598ae68fa243352db03770cc4f3a9e0a454ae1 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 22 Aug 2025 00:07:53 +0000 Subject: [PATCH 3/7] remove unnecessary comment --- A102-xds-grpc-service.md | 1 - 1 file changed, 1 deletion(-) diff --git a/A102-xds-grpc-service.md b/A102-xds-grpc-service.md index 06234086a..5aa06138c 100644 --- a/A102-xds-grpc-service.md +++ b/A102-xds-grpc-service.md @@ -104,7 +104,6 @@ bootstrap config: // supports. This field is required and must contain at least one // channel creds type that the client supports. "channel_creds": [ - // The format is identical to xds_servers.channel_creds. { "type": , // The "config" field is optional; it may be missing if the From 9fe9fc2957ad230e9669a10d0a157033e40c744a Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 22 Aug 2025 00:20:01 +0000 Subject: [PATCH 4/7] document validation for timeout field --- A102-xds-grpc-service.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/A102-xds-grpc-service.md b/A102-xds-grpc-service.md index 5aa06138c..e6e00e78b 100644 --- a/A102-xds-grpc-service.md +++ b/A102-xds-grpc-service.md @@ -214,7 +214,10 @@ When validating a `GrpcService` proto, the following fields will be used: - `channel_args`: Ignored. Not supportable across languages in gRPC. - [`timeout`](https://github.com/envoyproxy/envoy/blob/7ebdf6da0a49240778fd6fed42670157fde371db/api/envoy/config/core/v3/grpc_service.proto#L308): If set, this will be used to set the deadline on RPCs sent to the - side-channel service. + side-channel service. The value must obey the restrictions specified in + the [`google.protobuf.Duration` + documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), + and it must have a positive value. - [`initial_metadata`](https://github.com/envoyproxy/envoy/blob/7ebdf6da0a49240778fd6fed42670157fde371db/api/envoy/config/core/v3/grpc_service.proto#L315): If present, specifies headers to be added to RPCs sent to the side-channel service. Inside of each entry: From 1f13dc273895daf69615a29313843f5f11c5d21c Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 18 Sep 2025 19:08:43 +0000 Subject: [PATCH 5/7] support access token call creds instead of GCE creds --- A102-xds-grpc-service.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/A102-xds-grpc-service.md b/A102-xds-grpc-service.md index e6e00e78b..ede867411 100644 --- a/A102-xds-grpc-service.md +++ b/A102-xds-grpc-service.md @@ -4,7 +4,7 @@ A102: xDS `GrpcService` Support * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-08-21 +* Last updated: 2025-09-18 * Discussion at: (filled after thread exists) ## Abstract @@ -99,7 +99,7 @@ bootstrap config: // The list of side-channel services allowed to be configured via xDS. "allowed_grpc_services": { // The key is fully-qualified target URI. - "dns:///xds.example.org:443": { + "dns:///ratelimit.example.org:443": { // List of channel creds. Client will stop at the first type it // supports. This field is required and must contain at least one // channel creds type that the client supports. @@ -187,7 +187,13 @@ introduces the following new fields: resource will be NACKed. The following extensions will be supported in this field: - - `envoy.extensions.grpc_service.call_credentials.google_compute_engine.v3.GoogleComputeEngineCredentials` + - `envoy.extensions.grpc_service.call_credentials.access_token.v3.AccessTokenCredentials`: + In this message: + - `token`: Required. The access token. The token will be added as + an `authorization` header with header `Bearer ` (note trailing + space) followed by the value of this field. Note that the + token will not be sent on the wire unless the connection has + security level PRIVACY_AND_INTEGRITY. Note that this will require extending the channel credentials and call credentials registries to support configuration via these protos, in From b2837166fc48f2ffb87ddc0f124a904dae7cb17e Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 18 Sep 2025 22:57:14 +0000 Subject: [PATCH 6/7] add mailing list link --- A102-xds-grpc-service.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/A102-xds-grpc-service.md b/A102-xds-grpc-service.md index ede867411..d58a54028 100644 --- a/A102-xds-grpc-service.md +++ b/A102-xds-grpc-service.md @@ -5,7 +5,7 @@ A102: xDS `GrpcService` Support * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: * Last updated: 2025-09-18 -* Discussion at: (filled after thread exists) +* Discussion at: https://groups.google.com/g/grpc-io/c/3hguVpr8maE ## Abstract From f75b7cedde3a1e9cbf179f9203f74646d7257c39 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Wed, 24 Sep 2025 00:36:47 +0000 Subject: [PATCH 7/7] add sergiitk as co-author, since some of the text came from A77 --- A102-xds-grpc-service.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/A102-xds-grpc-service.md b/A102-xds-grpc-service.md index d58a54028..8222e526a 100644 --- a/A102-xds-grpc-service.md +++ b/A102-xds-grpc-service.md @@ -1,6 +1,6 @@ A102: xDS `GrpcService` Support ---- -* Author(s): @markdroth +* Author(s): @markdroth, @sergiitk * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: