Adding dual-stack support for Cluster API GCP Provider#1582
Conversation
k8s-ci-robot
added
kind/feature
|
/hold |
✅ Deploy Preview for kubernetes-sigs-cluster-api-gcp ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
k8s-ci-robot
added
cncf-cla: yes
|
Hi @barbacbd. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/L
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
salasberryfin
left a comment
salasberryfin
left a comment
There was a problem hiding this comment.
Thanks @barbacbd, I left a couple of comments.
barbacbd
force-pushed
the
dual_stack_support
branch
from
e85c011 to
db1abd6
Compare
|
Leaving this as a reference here. For
from the sdk: We can fill this out but it may not make any changes since we will not have ipv6 only vms |
salasberryfin
left a comment
salasberryfin
left a comment
There was a problem hiding this comment.
A couple of "nice to haves" that could be added here:
- E2E test case using a dual stack network configuration.
- A brief entry in the CAPG book that documents the new feature (this can be a follow-up PR).
k8s-ci-robot
added
size/XXL
| // InternalIpv6PrefixLength: The prefix length of the primary internal IPv6 range. | ||
| // +kubebuilder:validation:Minimum=0 | ||
| // +kubebuilder:validation:Maximum=128 | ||
| // +optional | ||
| InternalIpv6PrefixLength int `json:"internalIpv6PrefixLength,omitempty"` | ||
|
|
||
| // Ipv6Address: An IPv6 internal network address for this network interface. | ||
| // To use a static internal IP address, it must be unused and in the same | ||
| // region as the instance's zone. If not specified, Google Cloud will | ||
| // automatically assign an internal IPv6 address from the instance's subnetwork. | ||
| // +optional | ||
| Ipv6Address string `json:"ipv6Address,omitempty"` |
There was a problem hiding this comment.
If these are related, is it useful to have it as a combined CIDR string?
There was a problem hiding this comment.
These are related but I believe that they are independent. Specifying one does not mean you must specify the other.
barbacbd
force-pushed
the
dual_stack_support
branch
3 times, most recently
from
7e45964 to
f921678
Compare
|
/assign @justinsb |
barbacbd
force-pushed
the
dual_stack_support
branch
from
f921678 to
f3287da
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: barbacbd The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
k8s-ci-robot
added
needs-rebase
| }, | ||
| } | ||
|
|
||
| // For now we cannot assign the IPv6AccessConfigs. The bootstrap node is the only one on our side |
There was a problem hiding this comment.
@justinsb I would love to get your input on this area. The problem we are currently seeing is
- We need to set the ipv6 access to INTERNAL to mirror what is default for IPv4. This will also ensure we can have an internal Load Balancer.
- We set the
network.EnableUlaInternalIpv6 = trueto reflect this also and allow internal addresses
The problem is on our bootstrap node. This usually is given a public IP address and is handled in the lines above. But when this instance is created it is no longer given a public/global IPv6 address because the network was set to INTERNAL.
It looks like googles suggestion would be to add another subnet and set that to external then open the bootstrap node on that subnet. We would then create another AccessConfig to match. This seems like alot of work potentially adding more issues.
Any input would be helpful. Thank you!
k8s-ci-robot
added
size/XXL
barbacbd
force-pushed
the
dual_stack_support
branch
from
fab2e3c to
97ab30c
Compare
k8s-ci-robot
removed
the
needs-rebase
barbacbd
force-pushed
the
dual_stack_support
branch
from
97ab30c to
df500e9
Compare
api/v1beta/types.go:
Adding support for machines and simple network changes for IPV6 or dual stack work.
- InternalIpv6PrefixLength
- IPv6Address
- StackType
cloud/scope/machine.go:
Adding support for instances and networks to allow dual stack components.
- InstanceNetworkInterfaceSpec
- InternalIpv6PrefixLength
- Ipv6AccessConfigs
- ExternalIPv6
- ExternalIpv6PrefixLength
- Type (when present always set to DIRECT_IPV6)
- Name (always set to External IPv6)
- IPv6AccessType
- Ipv6Address
- StackType
- InstanceSpec
- PrivateIpv6GoogleAccess
- InstanceNetworkInterfaceAliasIPRangesSpec
** This did not change. The AliasIPs appear to only support IPv4 CIDR or single address format.
cloud/interfaces.go:
Expose a couple of new functions to get StackType and IPvAddress information from the cluster specs. These are only
getter functions for the Machine.go to access.
The test currently adds the following parameters: - internalIpv6PrefixLength: 64 - stackType: DualStack
- Added IPv6 APIServer Address (internal and external) for dual stack configurations - Added IPv6 Forwarding Rules (internal and external) for dual stack configurations - Added an address selection policy (IPv4 vs IPv6) 2. cloud/services/compute/loadbalancers/reconcile.go - Set the IPv6 API Server addresses when dual stack configuration is selected - Set the IPv6 Forwarding Rules when dual stack configuration is selected. - Set the ControlPlaneEndpoint to the IPv6 Address when IPV6 Primary is selected
1. IPv4Primary with IPv4Only stack type - Verifies IPv4 address is used (no IPv6 resources created)
2. IPv4Primary with DualStack type - Verifies IPv4 address is preferred even though both IPv4 and IPv6 are available
3. IPv6Primary with DualStack type - Verifies IPv6 address is preferred when configured
Key Implementation Details
- Mock Address Hook: Added an InsertHook to the mock addresses that automatically populates the Address field with actual IP addresses:
- IPv4 addresses get 192.0.2.1
- IPv6 addresses get 2001:db8::1
- Verification: The tests verify:
- The control plane endpoint host contains the correct IP address (IPv4 or IPv6)
- For DualStack configurations, both IPv4 and IPv6 resources are created
- For IPv4Only configurations, no IPv6 resources exist
Note: Tests written by Claude and editted by @barbacbd.
Located at: /Users/bbarbach/dev/k8s-cluster-api-provider-gcp/test/e2e/data/infrastructure-gcp/ This template creates a DualStack cluster with IPv6 as the primary address: - stackType: "DualStack" - addressPreferencePolicy: "IPv6Primary" 2. dual_stack_test.go Located at: /Users/bbarbach/dev/k8s-cluster-api-provider-gcp/test/e2e/ This comprehensive test file includes three test contexts: Test 1: DualStack with IPv4Primary (Default) - Uses flavor: ci-with-dual-stack - Verifies StackType is DualStack - Verifies AddressPreferencePolicy defaults to IPv4Primary - Validates both IPv4 and IPv6 addresses are allocated - Confirms control plane endpoint uses IPv4 address - Checks both IPv4 and IPv6 forwarding rules exist Test 2: DualStack with IPv6Primary - Uses flavor: ci-dual-stack-with-ipv6primary - Verifies StackType is DualStack - Verifies AddressPreferencePolicy is IPv6Primary - Validates both IPv4 and IPv6 addresses are allocated - Confirms control plane endpoint uses IPv6 address (key difference) - Checks both IPv4 and IPv6 forwarding rules exist Test 3: IPv4Only (Default Behavior) - Uses default flavor - Verifies StackType is IPv4Only or empty (defaults to IPv4Only) - Validates only IPv4 addresses are allocated - Confirms no IPv6 addresses or forwarding rules exist - Verifies control plane endpoint uses IPv4 address 3. Updated gcp-ci.yaml Added the new template to the e2e configuration so it can be used in tests. 4. Moved Dual stack tests from e2e_test.go to dual_stack_test.go
barbacbd
force-pushed
the
dual_stack_support
branch
from
df500e9 to
fbe8678
Compare
|
/test pull-cluster-api-provider-gcp-e2e-test |
|
/hold |
k8s-ci-robot
added
the
do-not-merge/hold
1. Detects instance group reference changes - Compares actual Group SelfLinks, not just counts 2. Handles recreated instance groups - If instance groups get recreated with new SelfLinks, the backend service will be updated 3. Prevents unnecessary updates - Only updates when backends actually change 4. Safe for empty backends - Protected by len(backendsvc.Backends) > 0 check 5. Works with dual-stack - Doesn't interfere with dual-stack forwarding rules
|
@damdo: GitHub didn't allow me to request PR reviews from the following users: sadasu, tthvo. Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs. |
Update test templates to include IPv6 pod/service CIDRs for complete dual stack
- When specified, the IPv6 Cidr block is set for each subnet (documented in the subnet spec). - When not specified, Google will assign the CIDR Block itself.
…uster nodes remain secure on private subnets!
The commented-out code couldn't work because all subnets were forced to use INTERNAL IPv6 (required for internal load balancers). Bootstrap nodes
with publicIP: true couldn't get public IPv6 addresses.
Solution
Implemented a dual subnet architecture where:
- Public subnets (externalIpv6: true) use EXTERNAL IPv6 with GUA ranges for bootstrap/bastion nodes
- Private subnets (externalIpv6: false) use INTERNAL IPv6 with ULA ranges for cluster nodes and load balancers
Changes Made
1. Added ExternalIpv6 field to SubnetSpec (defaults to false)
2. Updated subnet creation in cluster.go and managedcluster.go to set Ipv6AccessType based on this field
3. Extended ClusterGetter interface with Subnets() method
4. Fixed machine network interface logic to:
- Look up subnet configuration
- Enable IPv6 access configs only when both publicIP: true AND externalIpv6: true
- Removed the broken commented-out code
5. Regenerated all CRD manifests
Code Changes created by Claude, reviewed and suggested by @barbacbd.
Add the Dual Stack Templates to document Dual Stack Configuration support.
|
@barbacbd: The following tests failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
6 participants
api/v1beta/types.go:
Adding support for machines and simple network changes for IPV6 or dual stack work.
cloud/scope/machine.go:
Adding support for instances and networks to allow dual stack components.
InstanceNetworkInterfaceSpec
InstanceSpec
InstanceNetworkInterfaceAliasIPRangesSpec ** This did not change. The AliasIPs appear to only support IPv4 CIDR or single address format.
cloud/interfaces.go:
Expose a couple of new functions to get StackType and IPvAddress information from the cluster specs. These are only getter functions for the Machine.go to access.
What type of PR is this?
/kind feature
/kind api-change
What this PR does / why we need it:
This PR will serve as a focal point for adding dual stack support to CAPG.
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #
#1486
#478
Special notes for your reviewer:
Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.
TODOs:
Release note: