SelfClusterAccessReview - Cluster discovery standard proposal.

[mjudeikis]

We been experimenting with mcp server in multi-cluster context, and found biggest limitation being ClusterDiscovery + access.

Idea we been brewing is to support generic cluster discovery API - SelfClusterAccessReview (SCAR for short), like:

type SelfClusterAccessReview struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`
    Spec   SelfClusterAccessReviewSpec   `json:"spec"`
    Status SelfClusterAccessReviewStatus `json:"status,omitempty"`
}

type SelfClusterAccessReviewStatus struct {
    // List of clusters the subject has at least 'view' access to
    Clusters []AccessEndpointSlice `json:"clusters"`
}

The idea is to have a standard api, MCP could call with user identity (bearer token or cert), especially when using an identity provider like Keycloak as a backing IDP and get a list of cluster user access and how to access them.

So one could implement a standard-looking API server, run it close to the MCP server, and integrate with any cluster provider.

[Cali0707]

@mjudeikis totally agree that understanding whether the user can access a target is a key issue. for the discovery problem, I'm curious if you have experimented with this tool (and if so, why it wouldn't work for the discovery problem):

kubernetes-mcp-server/pkg/toolsets/config/configuration.go

Lines 40 to 58 in 5a60cee

	{
	Tool: api.Tool{
	Name: "targets_list",
	Description: "List all available targets",
	InputSchema: &jsonschema.Schema{
	Type: "object",
	},
	Annotations: api.ToolAnnotations{
	Title: "Targets List",
	ReadOnlyHint: ptr.To(true),
	DestructiveHint: ptr.To(false),
	IdempotentHint: ptr.To(true),
	OpenWorldHint: ptr.To(false),
	},
	},
	ClusterAware: ptr.To(false),
	TargetListProvider: ptr.To(true),
	Handler: nil,
	},

@matzew and I were discussing if there was a way to make the list returned here be user access aware

[mjudeikis]

mcp --targets-provider-url=http://cluster.mcp.svc.cluster.local:8080 --require_oauth=true --authorization-url="https://auth.keycloak.example.com" --oauth-audience="bob"

As long scheme of api is pre-agreed - it should work?

[Cali0707]

cc @manusa @matzew any thoughts here?

[manusa]

We had an internal discussion about the topic.

It seems that this is a common problem to all multi-cluster providers.

However, having a dependency on an optional API server for this purpose is not great because some users might not have their cluster set up with these requirements.

One of the options @Cali0707 was considering is checking if the user has access to the cluster by performing a Self Subject Access Review (or similar -mn: maybe just trying to perform a standard API request-).
This could be centralized in a single place and all providers could benefit.

Paraphrasing his words, please correct anything I misinterpreted 😅

It's my impression that this solution might add some latency or overhead for each request (where we need to retrieve the list of clusters) and might not be optimal.

Anyway, it's clear that this is something to be solved, so it's definitely something we'll need to figure out.

[mjudeikis]

The problem with SAR requests with a cluster list is the scale of how many clusters you need to iterate over. So a solution which has a default option built in and a pluggable one where one can hook an external, scalable system would be desired.

We operate systems with thousands of clusters. So having a list of thousands of clusters to be returned to every user and SAR requests sent to each of them to validate, even if to go/no-go, is not an option. And AI was kinda hallucinating even with 10 clusters, which ones it has access to, and which ones don't.

Hence, the pluggable system where we can build an index based on the user's identity and just return a list of what the user has access to only. Think controller-runtime + index backing API endpoint.

It's my impression that this solution might add some latency or overhead for each request (where we need to retrieve the list of clusters) and might not be optimal.

I tried something like this. The entire system just collapsed on its own due to the number of lists and the SAR requests count. Hence the thread in the first place :)

[ntnn]

However, having a dependency on an optional API server for this purpose is not great because some users might not have their cluster set up with these requirements.

That is not an unsolvable problem. In multicluster-runtime we have the concpet of Providers. The idea was that we can implement an API server that just takes a provider implementation and answers SCAR requests based on this.
kube-mcp could use that server to manage local kubeconfigs and point at another dedicated instance if the users runs one themselves or works in an environment that provides one.

It's my impression that this solution might add some latency or overhead for each request (where we need to retrieve the list of clusters) and might not be optimal.

Not necessarily, depends on the implementation - if you have to go to each cluster and do a SAR that would take a while. But that's why an external implementation is important, so platforms can implement an efficient way to determine access that is tailored to their environment.

[Cali0707]

@mjudeikis would #843 allow you to implement the pattern that you want for kcp?

Ideally we wouldn't want to force something like a separate API server onto users of every provider, just users of specific providers that would guarantee that is deployed as well.

My thinking is this way you could use this approach in your kcp provider, but others (e.g. kubeconfig) would not need to

[mjudeikis]

I did a quick scan on the pr and I think yes. Basically, if I read it right, the cluster/targets list is dynamic now, and providers can implement their own way of doing this "per request" if that is right - yes, it should work.

[Cali0707]

yes that is the idea - the ctx passed into the GetTargets function will have the user's auth set on it, so you can do things like create a derived k8s client with the users RBAC, or extract that bearer token and user it somewhere else to check auth

[Cali0707]

FYI @mjudeikis this is now merged, so you should be able to follow up in the kcp provider with your user specific workspace list detection with whatever approach you think makes sense - sorry this took so long to sort out!

[mjudeikis]

No prob :) already testing it here https://github.com/faroshq/kedge/ :D or my AI bot is testing it :D Thanks!