fea: Registry API Data Source #2189

dmartinol · 2025-10-14T11:33:19Z

Fixes #2104

Changes

Added Registry API source handler with type: api
Added UT and IT
Updated docs

Postponed requirements

Management of upstream API and schema. Will track a separate issue for that

Signed-off-by: Daniele Martinoli <[email protected]>

codecov · 2025-10-14T11:58:21Z

Codecov Report

❌ Patch coverage is 83.51648% with 60 lines in your changes missing coverage. Please review.
✅ Project coverage is 53.85%. Comparing base (30fc0a1) to head (e8e60ce).

Files with missing lines	Patch %	Lines
cmd/thv-operator/pkg/sources/api_toolhive.go	79.80%	31 Missing and 10 partials ⚠️
cmd/thv-operator/pkg/sources/api.go	79.22%	14 Missing and 2 partials ⚠️
cmd/thv-operator/pkg/httpclient/client.go	91.17%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    stacklok/toolhive#2189      +/-   ##
==========================================
+ Coverage   53.45%   53.85%   +0.39%     
==========================================
  Files         229      234       +5     
  Lines       29375    29739     +364     
==========================================
+ Hits        15703    16015     +312     
- Misses      12542    12575      +33     
- Partials     1130     1149      +19

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Daniele Martinoli <[email protected]>

jhrozek · 2025-10-15T11:10:52Z

@claude provide a review. During the review, you must take into account the existing flows of the operator and how it synchronizes its data sources.

Copilot

Pull Request Overview

This PR adds support for fetching MCP registry data from API endpoints, enabling dynamic synchronization from remote registries via HTTP/HTTPS.

Key Changes:

Added API source handler with automatic format detection (ToolHive vs Upstream MCP Registry)
Implemented HTTP client for API operations with timeout and error handling
Added comprehensive unit and integration tests for API source functionality

Reviewed Changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`cmd/thv-operator/api/v1alpha1/mcpregistry_types.go`	Added `APISource` type and `RegistrySourceTypeAPI` constant to support API-based registry sources
`cmd/thv-operator/pkg/sources/api.go`	Main API source handler with format detection and delegation logic
`cmd/thv-operator/pkg/sources/api_toolhive.go`	ToolHive-specific API handler for fetching and converting registry data
`cmd/thv-operator/pkg/sources/api_upstream.go`	Upstream MCP Registry API handler (validation only, Phase 2 not implemented)
`cmd/thv-operator/pkg/sources/api_types.go`	Type aliases for ToolHive Registry API response types
`cmd/thv-operator/pkg/httpclient/client.go`	HTTP client implementation with context support and standard headers
`cmd/thv-operator/pkg/httpclient/types.go`	HTTPError type for structured error handling
`cmd/thv-operator/pkg/sources/factory.go`	Factory updated to create API source handlers
`deploy/charts/operator-crds/crds/toolhive.stacklok.dev_mcpregistries.yaml`	CRD updated with API source configuration
`docs/operator/crd-api.md`	Documentation for API source configuration
`cmd/thv-operator/REGISTRY.md`	User guide for API source configuration
`examples/operator/mcp-registries/*.yaml`	Example configurations for API source usage

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

cmd/thv-operator/api/v1alpha1/mcpregistry_types.go

docs/operator/crd-api.md

Copilot · 2025-10-15T11:11:05Z

deploy/charts/operator-crds/crds/toolhive.stacklok.dev_mcpregistries.yaml

+                          Phase 1 (ToolHive API):
+                            - /v1/registry/servers - List all servers (single response, no pagination)
+                            - /v1/registry/servers/{name} - Get specific server (future)
+                            - /v1/registry/info - Get registry metadata (future)
+                          Phase 2 (Upstream MCP Registry API):
+                            - /v0/servers - List all servers with pagination


The CRD documentation references /v1/registry/* paths for ToolHive API, but the implementation uses /v0/* paths. Update to match the actual implementation.

Suggested change

Phase 1 (ToolHive API):

- /v1/registry/servers - List all servers (single response, no pagination)

- /v1/registry/servers/{name} - Get specific server (future)

- /v1/registry/info - Get registry metadata (future)

Phase 2 (Upstream MCP Registry API):

- /v0/servers - List all servers with pagination

Supported API endpoints:

- /v0/servers - List all servers (with pagination)

generated code, will be committed as part of previous change

examples/operator/mcp-registries/mcpregistry-api-external.yaml

claude · 2025-10-15T11:11:07Z

Claude encountered an error —— View job

Failed with exit code 128

I'll analyze this and get back to you.

jhrozek

I think this is great. We need a new version to bump the CRDs. Most of the review comments are just comments and most of them can be addressed as follow ups so we can keep moving.

cmd/thv-operator/api/v1alpha1/mcpregistry_types.go

deploy/charts/operator-crds/README.md

deploy/charts/operator-crds/Chart.yaml

jhrozek · 2025-10-16T20:44:52Z

cmd/thv-operator/pkg/httpclient/client.go

+
+const (
+	// DefaultTimeout is the default timeout for HTTP requests
+	DefaultTimeout = 30 * time.Second


this seems...a lot :-)

jhrozek · 2025-10-16T20:45:43Z

cmd/thv-operator/pkg/httpclient/client.go

+	}
+
+	// Read response body
+	body, err := io.ReadAll(resp.Body)


should we use a LimitReader here to protect from too big responses that might DoS the client (Maybe 100MiB)

jhrozek · 2025-10-16T21:04:43Z

cmd/thv-operator/pkg/httpclient/client.go

+
+	// Set headers
+	req.Header.Set("User-Agent", UserAgent)
+	req.Header.Set("Accept", "application/json")


Do you think if there can be any issues if we later request the openapi.yaml (I know it's not implemented yet)

for server validation purposes, you mean?
have you looked at how we identify the type of server in this code? e.g. querying /info fot toolhive or the /openapi.yaml for upstream

jhrozek · 2025-10-16T21:06:57Z

cmd/thv-operator/pkg/sources/api_toolhive.go

+	baseURL := mcpRegistry.Spec.Source.API.Endpoint
+
+	// Remove trailing slash
+	if len(baseURL) > 0 && baseURL[len(baseURL)-1] == '/' {


TrimSuffix ?

cmd/thv-operator/pkg/sources/api_toolhive.go

jhrozek · 2025-10-16T21:09:39Z

cmd/thv-operator/pkg/sources/api_toolhive.go

+	}
+
+	// Fetch detailed information for each server
+	for _, serverSummary := range response.Servers {


I guess that since we won't keep the toolhive format around for more it doesn't make sense to parallelize this? Maybe worth adding a TODO?

jhrozek · 2025-10-16T21:35:00Z

cmd/thv-operator/pkg/httpclient/client.go

+	if timeout == 0 {
+		timeout = DefaultTimeout
+	}
+	return &DefaultClient{


Should the default client use TLS? There is a NewHttpClientBuilder in the codebase that could be used for this..

btw this could be a follow-up and I'll be happy to do it (I haven't contributed much to the registry effort lately..sorry..)

Signed-off-by: Daniele Martinoli <[email protected]>

dmartinol and others added 5 commits October 13, 2025 14:36

impl draft

9850663

Signed-off-by: Daniele Martinoli <[email protected]>

Registry API Source Handler

3bb4a84

Signed-off-by: Daniele Martinoli <[email protected]>

Merge branch 'stacklok:main' into registry_apisource

bbb6e0c

chart version bump

a3958b9

Signed-off-by: Daniele Martinoli <[email protected]>

Merge branch 'stacklok:main' into registry_apisource

61d24de

dmartinol and others added 2 commits October 14, 2025 14:56

increase test coverage

790debb

Signed-off-by: Daniele Martinoli <[email protected]>

Merge branch 'stacklok:main' into registry_apisource

b977f65

jhrozek requested a review from Copilot October 15, 2025 11:09

Copilot AI reviewed Oct 15, 2025

View reviewed changes

jhrozek reviewed Oct 16, 2025

View reviewed changes

dmartinol and others added 3 commits October 17, 2025 11:18

Merge branch 'stacklok:main' into registry_apisource

574e590

fixed api references to v0

ac62364

Signed-off-by: Daniele Martinoli <[email protected]>

chart version bump

e8e60ce

Signed-off-by: Daniele Martinoli <[email protected]>

fea: Registry API Data Source #2189

Are you sure you want to change the base?

fea: Registry API Data Source #2189

Conversation

dmartinol commented Oct 14, 2025

Changes

Postponed requirements

Uh oh!

codecov bot commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jhrozek commented Oct 15, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

claude bot commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jhrozek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Oct 14, 2025 •

edited

Loading

claude bot commented Oct 15, 2025 •

edited

Loading