Simple discovery server

cmd/kops/create_cluster.go

@@ -223,6 +223,14 @@ func NewCmdCreateCluster(f *util.Factory, out io.Writer) *cobra.Command {
 		return nil, cobra.ShellCompDirectiveNoFileComp
 	})
 
+	if featureflag.DiscoveryService.Enabled() {
+		cmd.Flags().StringVar(&options.PublicDiscoveryServiceURL, "discovery-service", options.PublicDiscoveryServiceURL, "A URL to a server implementing public OIDC discovery. Enables IRSA in AWS.")
+		cmd.RegisterFlagCompletionFunc("discovery-service", func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
+			// TODO complete vfs paths
+			return nil, cobra.ShellCompDirectiveNoFileComp
+		})
+	}
+
 	var validClouds []string
 	{
 		allClouds := clouds.SupportedClouds()

discovery/GEMINI.md

@@ -0,0 +1,63 @@
+# Discovery Service Project
+
+## Overview
+This project implements a public discovery service designed for decentralized, secure peer discovery. The core innovation is the use of **Custom Certificate Authorities (CAs)** to define isolated "Universes". Clients register and discover peers within their own Universe, identified and secured purely by mTLS.
+
+The service emulates a Kubernetes API, allowing interaction via `kubectl`, including support for **Server-Side Apply**.
+
+## Key Concepts
+
+### 1. The "Universe"
+- A **Universe** is an isolated scope for peer discovery.
+- It is cryptographically defined by the **SHA256 hash of the Root CA's Public Key**.
+- Any client possessing a valid certificate signed by a specific CA belongs to that CA's Universe.
+- Different CAs = Different Universes. There is no crossover.
+
+### 2. Authentication & Authorization
+- **Mechanism**: Mutual TLS (mTLS).
+- **Client Identity**: Derived from the **Common Name (CN)** of the leaf certificate.
+- **Universe Context**: Derived from the **Root CA** presented in the TLS handshake.
+- **Requirement**: Clients **MUST** present the full certificate chain (Leaf + Root CA) during the handshake. The server does not maintain a pre-configured trust store for these custom CAs; it uses the presented chain to determine the scope.
+
+### 3. API Resources
+- **DiscoveryEndpoint** (`discovery.kops.k8s.io/v1alpha1`): Represents a peer in the discovery network. Can optionally hold OIDC configuration (Issuer URL, JWKS).
+- **Validation**: A client with CN `client1` can only Create/Update a `DiscoveryEndpoint` named `client1`.
+- **Apply Support**: The server supports `PATCH` requests to facilitate `kubectl apply --server-side`.
+
+### 4. OIDC Discovery
+The server acts as an OIDC Discovery Provider for the Universe.
+- **Public Endpoints**:
+  - `/.well-known/openid-configuration`: Returns the OIDC discovery document.
+  - `/openid/v1/jwks`: Returns the JSON Web Key Set (JWKS).
+- **Data Source**: These endpoints serve data uploaded by clients via the `DiscoveryEndpoint` resource.
+
+## Architecture
+
+### Project Structure
+- `cmd/discovery-server/`: Main entry point. Wires up the HTTP server with TLS configuration.
+- `pkg/discovery/`:
+  - `auth.go`: logic for inspecting TLS `PeerCertificates` to extract the Universe ID (CA hash) and Client ID.
+  - `store.go`: In-memory thread-safe storage (`MemoryStore`) mapping Universe IDs to lists of `DiscoveryEndpoint` objects.
+  - `server.go`: HTTP handlers implementing the K8s API emulation for `/apis/discovery.kops.k8s.io/v1alpha1`.
+  - `k8s_types.go`: Definitions of `DiscoveryEndpoint`, `DiscoveryEndpointList`, `TypeMeta`, `ObjectMeta` etc.
+
+### Data Model
+- **DiscoveryEndpoint**: The core resource. Contains `Spec.Addresses` and metadata.
+- **Universe**: Contains a map of `DiscoveryEndpoint` objects (keyed by name).
+- **Unified Types**: The API type `DiscoveryEndpoint` is used directly for in-memory storage, ensuring zero conversion overhead.
+
+## Security Model
+- **Trust Delegation**: The server delegates trust to the CA. If you hold the CA key, you control the Universe.
+- **Isolation**: The server ensures that a client presenting a cert chain for `CA_A` cannot read or write data to the Universe defined by `CA_B`.
+- **Ephemeral**: The current implementation uses in-memory storage. Data is lost on restart.
+
+## Building and Running
+
+### Build
+```bash
+go build ./cmd/discovery-server
+```
+
+### Run
+
+See docs/walkthrough.md for instructions on testing functionality.

discovery/README.md

@@ -0,0 +1,42 @@
+# Discovery Service
+
+A public discovery service using mTLS for authentication and "Universe" isolation, emulating a Kubernetes API.
+
+## Concept
+
+- **Universe**: Defined by the SHA256 Fingerprint of a Custom CA Certificate.
+- **Client**: Identified by a Client Certificate signed by that Custom CA.
+- **DiscoveryEndpoint**: The resource type representing a registered client.
+- **Isolation**: Clients can only see `DiscoveryEndpoint` objects signed by the same Custom CA.
+
+## Usage
+
+### Run Server
+
+```bash
+go run ./cmd/discovery-server --tls-cert server.crt --tls-key server.key --listen :8443
+```
+
+(You can generate a self-signed server certificate for testing, see the [walkthrough](docs/walkthrough.md) ).
+
+### Client Requirement
+
+Clients must authenticate using mTLS.
+**Important**: The client MUST provide the full certificate chain, including the Root CA, because the server does not have pre-configured trust stores for these custom universes.
+The server identifies the Universe from the SHA256 hash of the Root CA certificate found in the TLS chain.
+
+### Quick start
+
+See `docs/walkthrough.md` for detailed instructions.
+
+
+## OIDC Discovery
+
+The discovery server also serves OIDC discovery information publicly, allowing external systems (like AWS IAM) to discover the cluster's identity provider configuration.
+
+- `GET /<universe-id>/.well-known/openid-configuration`: Returns the OIDC discovery document.
+- `GET /<universe-id>/openid/v1/jwks`: Returns the JWKS.
+
+This information is populated by clients uploading `DiscoveryEndpoint` resources containing the `oidc` spec.
+
+## Building and Running

discovery/apis/discovery.kops.k8s.io/v1alpha1/types.go

@@ -0,0 +1,75 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1alpha1
+
+import (
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+)
+
+var DiscoveryEndpointGVR = schema.GroupVersionResource{
+	Group:    "discovery.kops.k8s.io",
+	Version:  "v1alpha1",
+	Resource: "discoveryendpoints",
+}
+
+var DiscoveryEndpointGVK = schema.GroupVersionKind{
+	Group:   "discovery.kops.k8s.io",
+	Version: "v1alpha1",
+	Kind:    "DiscoveryEndpoint",
+}
+
+// DiscoveryEndpoint represents a registered client in the discovery service.
+type DiscoveryEndpoint struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	Spec DiscoveryEndpointSpec `json:"spec,omitempty"`
+}
+
+// DiscoveryEndpointSpec corresponds to our internal Node data.
+type DiscoveryEndpointSpec struct {
+	Addresses []string  `json:"addresses,omitempty"`
+	LastSeen  string    `json:"lastSeen,omitempty"`
+	OIDC      *OIDCSpec `json:"oidc,omitempty"`
+}
+
+type OIDCSpec struct {
+	// IssuerURL string       `json:"issuerURL,omitempty"`
+	Keys []JSONWebKey `json:"keys,omitempty"`
+}
+
+type JSONWebKey struct {
+	Use       string `json:"use,omitempty"`
+	KeyType   string `json:"kty,omitempty"`
+	KeyID     string `json:"kid,omitempty"`
+	Algorithm string `json:"alg,omitempty"`
+	N         string `json:"n,omitempty"`
+	E         string `json:"e,omitempty"`
+	// Crv       string `json:"crv,omitempty"`
+	// X         string `json:"x,omitempty"`
+	// Y         string `json:"y,omitempty"`
+}
+
+// DiscoveryEndpointList is a list of DiscoveryEndpoint objects.
+type DiscoveryEndpointList struct {
+	metav1.TypeMeta `json:",inline"`
+	// Standard list metadata.
+	// We implement a minimal subset.
+	Metadata metav1.ListMeta     `json:"metadata,omitempty"`
+	Items    []DiscoveryEndpoint `json:"items"`
+}

discovery/apis/discovery.kops.k8s.io/v1alpha1/universeid.go

@@ -0,0 +1,29 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1alpha1
+
+import (
+	"crypto/sha256"
+	"crypto/x509"
+	"encoding/hex"
+)
+
+func ComputeUniverseIDFromCertificate(cert *x509.Certificate) string {
+	hash := sha256.Sum256(cert.RawSubjectPublicKeyInfo)
+	universeID := hex.EncodeToString(hash[:])
+	return universeID
+}

discovery/cmd/discovery-server/main.go

@@ -0,0 +1,74 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package main
+
+import (
+	"crypto/tls"
+	"flag"
+	"fmt"
+	"log"
+	"net/http"
+	"os"
+
+	"k8s.io/kops/discovery/pkg/discovery"
+)
+
+func main() {
+	certFile := os.Getenv("TLS_CERT")
+	flag.StringVar(&certFile, "tls-cert", certFile, "Path to server TLS certificate")
+
+	keyFile := os.Getenv("TLS_KEY")
+	flag.StringVar(&keyFile, "tls-key", keyFile, "Path to server TLS key")
+
+	addr := flag.String("listen", ":8443", "Address to listen on")
+	storageType := flag.String("storage", "memory", "Storage backend (memory, gcs)")
+	flag.Parse()
+
+	if certFile == "" || keyFile == "" {
+		fmt.Fprintf(os.Stderr, "Error: --tls-cert and --tls-key are required\n")
+		flag.Usage()
+		os.Exit(1)
+	}
+
+	var store discovery.Store
+
+	switch *storageType {
+	case "memory":
+		store = discovery.NewMemoryStore()
+	default:
+		log.Fatalf("Unknown storage type: %s", *storageType)
+	}
+
+	handler := discovery.NewServer(store)
+
+	tlsConfig := &tls.Config{
+		ClientAuth: tls.RequestClientCert,
+		// We do not set ClientCAs because we accept any CA and use it to define the universe.
+		MinVersion: tls.VersionTLS12,
+	}
+
+	server := &http.Server{
+		Addr:      *addr,
+		Handler:   handler,
+		TLSConfig: tlsConfig,
+	}
+
+	log.Printf("Discovery server listening on %s using %s storage", *addr, *storageType)
+	if err := server.ListenAndServeTLS(certFile, keyFile); err != nil {
+		log.Fatalf("Server failed: %v", err)
+	}
+}

discovery/dev/tasks/deploy-to-k8s

@@ -0,0 +1,42 @@
+#!/usr/bin/env bash
+
+# Copyright 2025 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+REPO_ROOT="$(git rev-parse --show-toplevel)"
+cd "${REPO_ROOT}/discovery"
+
+if [[ -z "${IMAGE_PREFIX:-}" ]]; then
+  IMAGE_PREFIX="${USER}/"
+fi
+
+IMAGE_TAG=$(date +%Y%m%d%H%M%S)
+
+# Build the discovery-server image
+VERSION=${IMAGE_TAG} GITSHA=$(git describe --always) KO_DOCKER_REPO="${IMAGE_PREFIX}discovery-server" go run github.com/google/[email protected] \
+  build --tags "${IMAGE_TAG}" --platform=linux/amd64,linux/arm64 --bare ./cmd/discovery-server/
+
+echo "Can install cert-manager with the following command:"
+echo "kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.2/cert-manager.yaml"
+
+kubectl create namespace discovery-k8s-io --dry-run=client -o yaml | kubectl apply -f -
+
+
+cat k8s/manifest.yaml | \
+sed "s|discovery-server:latest|${IMAGE_PREFIX}discovery-server:${IMAGE_TAG}|g" | \
+KUBECTL_APPLYSET=true kubectl apply -n discovery-k8s-io --prune --applyset=discovery-server -f -