Simple discovery server #17818

justinsb · 2025-12-15T17:16:46Z

No description provided.

k8s-ci-robot · 2025-12-15T17:16:48Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

ameukam · 2025-12-15T19:19:50Z

discovery/cmd/discovery-server/main.go

+		MinVersion: tls.VersionTLS12,
+	}
+
+	server := &http.Server{


http3 ? I had white wine before

Sorry, I don't get it ... would you like to use http3? Have I opted in to http3 automatically? Do you want us to be the "guinea pig" for http3 in kube (if so, I'm game!)

I don't think there's anything special we need from http3. We do need the client certificate information. I was thinking we would probably end up deploying this directly behind an L4 load balancer, or (failing that) using ingress or gateway with SNI routing.

In terms of backends, right now I have this with a simple in-memory implementation. Honestly that's probably good enough to get started, as we will not be offering any guarantee as to retention of these objects.

But ... if we wanted to do better, I think we should put them into etcd because (1) we should be able to run etcd pretty cheaply and we don't have to worry about wracking up a huge GCS bill if someone figures out how to make us send queries to GCS etc and (2) it means that we can use etcd-operator, which would be good from the "all the wood behind one arrow" perspective

I was half-joking about using http3 for the discovery server but looks like the OIDC protocol is only compatible with HTTP 1.1.

I think we can support http3, but let's start with whatever go gives us out of the box (which I think is still http1 or http2)

I do think a controversial one would be to support DNS over HTTP, if you're feeling spicy :-)

justinsb · 2025-12-16T12:23:55Z

discovery/docs/kubeconfig.md

@@ -0,0 +1,82 @@
+# Using kubectl with Discovery Service
+
+Since the Discovery Service now emulates the Kubernetes API for `DiscoveryEndpoint` resources, you can use `kubectl` to interact with it.


I'm going to ask gemini to clean up these instructions / demo scripts.

Cleaned up!

justinsb · 2025-12-16T12:24:31Z

discovery/pkg/discovery/k8s_types.go

+}
+
+// DiscoveryEndpoint represents a registered client in the discovery service.
+type DiscoveryEndpoint struct {


I should move this to apis/discovery.kops.k8s.io/v1alpha1 for consistency (and probably change the version to v1alpha1)

justinsb · 2025-12-16T12:25:51Z

discovery/pkg/discovery/server.go

+			{
+				Name:         "discoveryendpoints",
+				SingularName: "discoveryendpoint",
+				Namespaced:   true,


I think this probably should be cluster scoped, although I guess we could use the namespace to indicate the cluster if we we wanted to allow multiple clusters to share the same CA (which isn't a terrible idea if someone is doing multicluster).

Do we consider RBAC in the equation ?

So this isn't technically kube-apiserver, and I haven't implemented RBAC.

Right now we have this: anyone that has any cert signed by a CA can read the objects for that CA's universe (defined by the hash of the CA public key). You can write an object that matches your own CN only.

I probably need to build out the client side here to better understand what we actually need, whether it's acceptable to have the same CA certificate etc. (e.g. maybe we should only let kubelet certificates register, or maybe we should only let control plane nodes register, or maybe we should create a dedicated CA only for discovery)

justinsb · 2025-12-16T12:27:16Z

discovery/go.mod

+
+require (
+	k8s.io/apimachinery v0.34.3
+	k8s.io/client-go v0.34.3


Technically client-go / apimachinery is only used by the clients / tests, so it might be nice to split them out. But to do that would require a separate go.mod, which is a bit of a pain.

ameukam · 2025-12-16T20:39:32Z

discovery/pkg/discovery/server.go

+	return s
+}
+
+func (s *Server) registerRoutes() {


so we never delete endpoints ?

Not currently, no. You're right - I should add a TTL. (Maybe 2 hours, and then we can have nodes register every hour?). It won't be a "hard" TTL - we reserve the rights to remove objects at any time (and I should add that to the README.md / GEMINI.md)

I should probably also add explicit deletion support.

justinsb · 2026-01-01T00:08:19Z

Behind a feature flag DiscoveryService
Server deployed (temporarily?) at https://discovery.kubedisco.com, using manifest in repo
Simple e2e test that verifies the behavior (but does not yet actually try using the issuer e.g. in the e2e test)
Data lives in-process, and is not garbage collected
nodeup on the control-plane registers JWKS data

Big TODOs, which I propose we do in follow on:

Deploy on k8s.io?
Implement etcd backend, including TTL
Implement re-registration (maybe by running nodeup as a periodic systemd job?, or maybe something more lightweight) so that we do not lose registration after TTL
Add kube e2e test to make sure this actually works!
Probably many other things

justinsb · 2026-01-01T03:00:49Z

Removing WIP; still work to be done but it's safely behind a feature flag

justinsb · 2026-01-01T14:34:41Z

/test pull-kops-kubernetes-e2e-ubuntu-gce-build

I think when we make this periodically re-register we also want to make a failure non-blocking (currently nodeup will fail if discovery.kubedisco.com is offline, which is obviously not what we want), but I don't think that is the problem here

hakman

LGTM, let's ship it! 😁

k8s-ci-robot · 2026-01-07T09:58:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2026-01-07T10:40:13Z

@justinsb: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kops-kubernetes-e2e-ubuntu-gce-build	`29cf465`	link	false	`/test pull-kops-kubernetes-e2e-ubuntu-gce-build`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

hakman · 2026-01-07T10:46:07Z

/test pull-kops-e2e-cni-cilium-etcd

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 15, 2025

k8s-ci-robot requested review from johngmyers and zetaab December 15, 2025 17:16

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 15, 2025

justinsb force-pushed the discovery branch 2 times, most recently from 8396a06 to eead354 Compare December 15, 2025 18:52

ameukam reviewed Dec 15, 2025

View reviewed changes

justinsb force-pushed the discovery branch from eead354 to 4e75f10 Compare December 16, 2025 12:23

justinsb commented Dec 16, 2025

View reviewed changes

justinsb force-pushed the discovery branch 4 times, most recently from 2136f14 to 294dd35 Compare December 16, 2025 18:00

ameukam reviewed Dec 16, 2025

View reviewed changes

justinsb force-pushed the discovery branch from 294dd35 to 44615e6 Compare December 26, 2025 17:19

k8s-ci-robot added area/api area/nodeup labels Dec 26, 2025

justinsb force-pushed the discovery branch 8 times, most recently from 6360fc0 to f7ff09b Compare December 30, 2025 22:58

justinsb force-pushed the discovery branch 4 times, most recently from 1d369ae to 8ee0bca Compare December 30, 2025 23:31

justinsb added 3 commits December 31, 2025 19:03

spike: discovery server

91463c1

discovery: add discovery service registration via nodeup

43fab44

discovery: add simple e2e for discovery service

29cf465

justinsb force-pushed the discovery branch from fe7573a to 29cf465 Compare January 1, 2026 00:03

justinsb changed the title ~~WIP: simple discovery server~~ Simple discovery server Jan 1, 2026

justinsb marked this pull request as ready for review January 1, 2026 02:59

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 1, 2026

k8s-ci-robot requested a review from olemarkus January 1, 2026 03:00

hakman approved these changes Jan 7, 2026

View reviewed changes

k8s-ci-robot assigned hakman Jan 7, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 7, 2026

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 7, 2026

k8s-ci-robot merged commit 82f4036 into kubernetes:master Jan 7, 2026
36 of 37 checks passed

k8s-ci-robot added this to the v1.36 milestone Jan 7, 2026

		@@ -0,0 +1,82 @@
		# Using kubectl with Discovery Service

		Since the Discovery Service now emulates the Kubernetes API for `DiscoveryEndpoint` resources, you can use `kubectl` to interact with it.

Simple discovery server #17818

Simple discovery server #17818

Conversation

justinsb commented Dec 15, 2025

Uh oh!

k8s-ci-robot commented Dec 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justinsb commented Jan 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

justinsb commented Jan 1, 2026

Uh oh!

justinsb commented Jan 1, 2026

Uh oh!

hakman left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jan 7, 2026

Uh oh!

k8s-ci-robot commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hakman commented Jan 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

justinsb commented Jan 1, 2026 •

edited

Loading

k8s-ci-robot commented Jan 7, 2026 •

edited

Loading