[STORY] MCP route smoke test regression suite

[jsbattig]

Story: MCP Route Smoke Test Regression Suite

As a CIDX server maintainer
I want to have an automated smoke test suite that calls every MCP tool depending on app.py module-level globals via FastAPI TestClient
So that refactoring of app.py (like the v9.5.16 modularization) cannot silently break MCP tools with crash-level errors (NameError, AttributeError, ModuleNotFoundError)

Context and Motivation

During v9.5.16, app.py was modularized and critical MCP tools broke because module-level globals were missing. Specifically browse_directory, get_file_content, and list_files all crashed with "module 'code_indexer.server.app' has no attribute 'file_service'". These were only discovered through manual regression testing on staging -- there was no automated test catching this class of failure.

The MCP handler layer (handlers.py) accesses app.py globals via the app_module.X pattern (e.g., app_module.golden_repo_manager, app_module.file_service). If any of these globals are removed, renamed, or fail to initialize during app startup, the corresponding MCP tool crashes at runtime with a Python exception rather than a graceful error.

This story creates a regression suite that catches exactly this bug class: crash-level failures caused by missing or uninitialized app.py globals.

Implementation Status

• FastAPI TestClient fixture with in-process app instantiation
• Admin authentication fixture (login via /auth/login)
• Module attribute existence smoke test (all app_module globals)
• file_service tool smoke tests (list_files, get_file_content, browse_directory)
• golden_repo_manager tool smoke tests (search_code, list_repositories, discover_repositories, list_repo_categories, get_repository_status, check_hnsw_health, add_golden_repo, remove_golden_repo, refresh_golden_repo, change_golden_repo_branch, cidx_quick_reference)
• activated_repo_manager tool smoke tests (activate_repository, deactivate_repository, sync_repository, switch_branch, get_branches, get_all_repositories_status, manage_composite_repository)
• background_job_manager tool smoke tests (get_job_statistics, get_job_details)
• user_manager tool smoke tests (list_users, create_user, set_session_impersonation, list_api_keys, create_api_key, delete_api_key)
• semantic_query_manager / repository_listing_manager tool coverage (via search_code, get_repository_status, get_all_repositories_status -- already covered above)
• All tests passing in fast-automation.sh (seconds, not minutes)
• Code review approved
• E2E manual testing completed by Claude Code

Completion: 0/12 tasks complete (0%)

Algorithm

MCP Route Smoke Test Suite:

  FIXTURES:
    client = TestClient(app)  -- in-process, no external server
    admin_token = POST /auth/login {"username": "admin", "password": "admin"} -> access_token

  ATTRIBUTE EXISTENCE TEST:
    required_globals = ["golden_repo_manager", "activated_repo_manager", "user_manager",
                        "file_service", "background_job_manager", "semantic_query_manager",
                        "repository_listing_manager"]
    FOR EACH attr IN required_globals:
      ASSERT hasattr(app_module, attr) -- attribute exists on module
      ASSERT getattr(app_module, attr) is not None -- attribute was initialized

  MCP TOOL SMOKE TEST (per tool):
    INPUT: tool_name, minimal_arguments

    response = POST /mcp {
      "jsonrpc": "2.0", "id": 1,
      "method": "tools/call",
      "params": {"name": tool_name, "arguments": minimal_arguments}
    } WITH Authorization: Bearer admin_token

    ASSERT response.status_code == 200
    body = response.json()

    -- Extract response text for crash indicator scanning
    IF "result" IN body AND "content" IN body["result"]:
      text = body["result"]["content"][0]["text"]
    ELIF "error" IN body:
      text = str(body["error"])
    ELSE:
      text = str(body)

    CRASH_INDICATORS = ["NameError", "AttributeError", "ModuleNotFoundError",
                        "has no attribute", "is not defined"]
    FOR EACH indicator IN CRASH_INDICATORS:
      ASSERT indicator NOT IN text
      -- Functional errors like "repository not found" are ACCEPTABLE
      -- We are testing for crash-level errors only

  TOOL INVENTORY (grouped by app_module global):

    file_service tools:
      - list_files(repository_alias="nonexistent-repo")
      - get_file_content(repository_alias="nonexistent-repo", file_path="README.md")
      - browse_directory(repository_alias="nonexistent-repo", path="/")

    golden_repo_manager tools:
      - search_code(query_text="test", repository_alias="nonexistent-repo")
      - list_repositories()
      - discover_repositories()
      - list_repo_categories()
      - get_repository_status(repository_alias="nonexistent-repo")
      - check_hnsw_health()
      - add_golden_repo(url="https://example.com/test.git", alias="smoke-test")
      - remove_golden_repo(alias="nonexistent-repo")
      - refresh_golden_repo(alias="nonexistent-repo")
      - change_golden_repo_branch(alias="nonexistent-repo", branch="main")
      - cidx_quick_reference()

    activated_repo_manager tools:
      - activate_repository(repository_alias="nonexistent-repo")
      - deactivate_repository(repository_alias="nonexistent-repo")
      - sync_repository(repository_alias="nonexistent-repo")
      - switch_branch(repository_alias="nonexistent-repo", branch="main")
      - get_branches(repository_alias="nonexistent-repo")
      - get_all_repositories_status()
      - manage_composite_repository(action="list")

    background_job_manager tools:
      - get_job_statistics()
      - get_job_details(job_id="nonexistent-job-id")

    user_manager tools:
      - list_users()
      - create_user(username="smoke_test_user", password="Test123!", role="viewer")
      - set_session_impersonation(username="admin")
      - list_api_keys()
      - create_api_key(name="smoke-test-key")
      - delete_api_key(key_id="nonexistent-key-id")

Acceptance Criteria

Given the CIDX server app is instantiated via FastAPI TestClient
When the test suite checks all required app_module globals
Then every global (golden_repo_manager, activated_repo_manager, user_manager, file_service, background_job_manager, semantic_query_manager, repository_listing_manager) exists as a module attribute
And every global is not None after app initialization

Given an authenticated admin session via TestClient
When each MCP tool that accesses app_module.file_service is called with minimal arguments
Then the HTTP response status is 200
And the JSON-RPC response contains no NameError, AttributeError, or ModuleNotFoundError
And functional errors like "repository not found" are acceptable (not crash-level)

Given an authenticated admin session via TestClient
When each MCP tool that accesses app_module.golden_repo_manager is called with minimal arguments
Then the HTTP response status is 200
And the JSON-RPC response contains no crash-level Python exceptions

Given an authenticated admin session via TestClient
When each MCP tool that accesses app_module.activated_repo_manager is called with minimal arguments
Then the HTTP response status is 200
And the JSON-RPC response contains no crash-level Python exceptions

Given an authenticated admin session via TestClient
When each MCP tool that accesses app_module.background_job_manager is called with minimal arguments
Then the HTTP response status is 200
And the JSON-RPC response contains no crash-level Python exceptions

Given an authenticated admin session via TestClient
When each MCP tool that accesses app_module.user_manager is called with minimal arguments
Then the HTTP response status is 200
And the JSON-RPC response contains no crash-level Python exceptions

Given the smoke test suite is added to the standard test directory
When fast-automation.sh runs the full test suite
Then all smoke tests pass
And the smoke test suite completes in under 30 seconds
And no existing tests are broken by the addition

Given a future refactoring removes or renames an app_module global (simulating the v9.5.16 bug)
When the smoke test suite runs
Then at least one test fails immediately with a clear assertion message identifying the missing global
And the failure is caught before deployment to staging

Testing Requirements

• Unit tests: The smoke test file itself (tests/unit/server/mcp/test_mcp_route_smoke.py) contains all tests
• Attribute existence test: Validates all 7 app_module globals exist and are initialized (not None)
• Per-tool crash tests: One test function per MCP tool (~30 tools), each calling POST /mcp with minimal arguments and asserting no crash-level errors
• Crash indicator scanning: Each test scans response text for NameError, AttributeError, ModuleNotFoundError, "has no attribute", "is not defined"
• Performance: Entire suite must complete in under 30 seconds to remain in fast-automation.sh
• No external dependencies: Uses FastAPI TestClient (in-process), no running server needed
• No mocking of app_module globals: The entire point is to test that real globals are present and accessible -- mocking would defeat the purpose
• Manual testing: Run PYTHONPATH=src pytest tests/unit/server/mcp/test_mcp_route_smoke.py -v --tb=short and verify all tests pass, then run ./fast-automation.sh to confirm integration

Implementation Notes

Test File Location

tests/unit/server/mcp/test_mcp_route_smoke.py

Key Design Decisions

1. TestClient, not httpx/requests: FastAPI TestClient runs in-process. No server startup, no port conflicts, deterministic behavior. This is the standard FastAPI testing pattern.

2. Minimal arguments, not valid arguments: We pass arguments that look structurally correct (right field names, right types) but reference nonexistent resources. The goal is to get past argument validation into the handler code where app_module.X is accessed. A "repo not found" error proves the global was successfully accessed. A NameError proves it was not.

3. No parametrize for readability: Each tool gets its own test function (e.g., test_browse_directory_no_crash, test_list_files_no_crash). This makes failures immediately clear -- you see which specific tool crashed. Parametrize would show test_mcp_tool_no_crash[browse_directory] which is less grep-friendly in CI logs.

4. Crash indicators list: ["NameError", "AttributeError", "ModuleNotFoundError", "has no attribute", "is not defined"]. These are the exact Python exception types that appear in MCP error responses when a handler tries to access a missing global.

5. Admin authentication: Uses admin/admin (local dev credentials per CLAUDE.md). The auth fixture authenticates once per test session.

6. Todo tracking: The implementer MUST use the todo tool to create a checklist of every individual tool test BEFORE writing any code. Each test gets checked off as written.

Fixture Pattern

@pytest.fixture(scope="module")
def client():
    from code_indexer.server.app import app
    from fastapi.testclient import TestClient
    return TestClient(app)

@pytest.fixture(scope="module")
def admin_token(client):
    resp = client.post("/auth/login", json={"username": "admin", "password": "admin"})
    return resp.json()["access_token"]

Helper Pattern

CRASH_INDICATORS = ["NameError", "AttributeError", "ModuleNotFoundError",
                    "has no attribute", "is not defined"]

def assert_no_crash(resp, tool_name: str):
    """Assert MCP response has no crash-level errors."""
    assert resp.status_code == 200, f"{tool_name}: HTTP {resp.status_code}"
    body = resp.json()
    # Extract text from JSON-RPC response
    if "result" in body and "content" in body["result"]:
        text = body["result"]["content"][0].get("text", "")
    elif "error" in body:
        text = str(body["error"])
    else:
        text = str(body)
    for indicator in CRASH_INDICATORS:
        assert indicator not in text, (
            f"{tool_name} crashed with {indicator}: {text[:300]}"
        )

MCP Call Pattern

def mcp_call(client, admin_token, tool_name, arguments):
    return client.post("/mcp", json={
        "jsonrpc": "2.0", "id": 1,
        "method": "tools/call",
        "params": {"name": tool_name, "arguments": arguments}
    }, headers={"Authorization": f"Bearer {admin_token}"})

Complete Tool Inventory (30 tools across 6 globals)

#	app_module Global	MCP Tool Name	Minimal Test Arguments
1	file_service	list_files	{"repository_alias": "nonexistent-repo"}
2	file_service	get_file_content	{"repository_alias": "nonexistent-repo", "file_path": "README.md"}
3	file_service	browse_directory	{"repository_alias": "nonexistent-repo", "path": "/"}
4	golden_repo_manager	search_code	{"query_text": "test", "repository_alias": "nonexistent-repo"}
5	golden_repo_manager	list_repositories	{}
6	golden_repo_manager	discover_repositories	{}
7	golden_repo_manager	list_repo_categories	{}
8	golden_repo_manager	get_repository_status	{"repository_alias": "nonexistent-repo"}
9	golden_repo_manager	check_hnsw_health	{}
10	golden_repo_manager	add_golden_repo	{"url": "https://example.com/test.git", "alias": "smoke-test"}
11	golden_repo_manager	remove_golden_repo	{"alias": "nonexistent-repo"}
12	golden_repo_manager	refresh_golden_repo	{"alias": "nonexistent-repo"}
13	golden_repo_manager	change_golden_repo_branch	{"alias": "nonexistent-repo", "branch": "main"}
14	golden_repo_manager	cidx_quick_reference	{}
15	activated_repo_manager	activate_repository	{"repository_alias": "nonexistent-repo"}
16	activated_repo_manager	deactivate_repository	{"repository_alias": "nonexistent-repo"}
17	activated_repo_manager	sync_repository	{"repository_alias": "nonexistent-repo"}
18	activated_repo_manager	switch_branch	{"repository_alias": "nonexistent-repo", "branch": "main"}
19	activated_repo_manager	get_branches	{"repository_alias": "nonexistent-repo"}
20	activated_repo_manager	get_all_repositories_status	{}
21	activated_repo_manager	manage_composite_repository	{"action": "list"}
22	background_job_manager	get_job_statistics	{}
23	background_job_manager	get_job_details	{"job_id": "nonexistent-job-id"}
24	user_manager	list_users	{}
25	user_manager	create_user	{"username": "smoke_test_user", "password": "Test123!", "role": "viewer"}
26	user_manager	set_session_impersonation	{"username": "admin"}
27	user_manager	list_api_keys	{}
28	user_manager	create_api_key	{"name": "smoke-test-key"}
29	user_manager	delete_api_key	{"key_id": "nonexistent-key-id"}
30	(attribute test)	N/A	Validates all 7 globals exist and are not None

Definition of Done

• All acceptance criteria satisfied
• All 30 tool smoke tests + 1 attribute existence test passing
• Tests complete in under 30 seconds total
• fast-automation.sh passes with the new test file included
• No existing tests broken by the addition
• Code review approved (tdd-engineer + code-reviewer workflow)
• Manual end-to-end testing completed by Claude Code
• No lint/type errors (ruff, mypy clean)
• Test file follows existing conventions in tests/unit/server/mcp/

[jsbattig]

Story implementation complete

• 39 smoke tests (7 attribute + 32 per-tool), all passing in 5.73s
• Code review: rejected once (destructive side effects), all 6 findings fixed
• No destructive side effects: create_user uses existing username, create_api_key has cleanup
• Expanded crash indicators: ImportError, TypeError added
• Added: regex_search, get_repository_statistics, get_cached_content tests
• Exclusions documented in module docstring

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com