Skip to content

Commit 275f7bf

Browse files
cameroncookeclaude
andcommitted
fix(doctor): Improve representativeness and redaction controls
Align doctor output with current manifest and workflow architecture by removing legacy plugin reporting and replacing it with manifest-backed tool inventory summaries. Add default PII/path redaction to doctor output and introduce explicit opt-in non-redacted mode for both MCP doctor and the doctor CLI via nonRedacted/--non-redacted. Make dependency probing safer and more accurate by avoiding side effects in xcodemake checks and adding capability-level checks used by current tooling paths. Update generated tool docs and troubleshooting/bridge docs to reflect the new doctor behavior and options. Co-Authored-By: Claude <noreply@anthropic.com>
1 parent bfa9fef commit 275f7bf

File tree

9 files changed

+434
-89
lines changed

9 files changed

+434
-89
lines changed

docs/TOOLS-CLI.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,4 +189,4 @@ XcodeBuildMCP provides 73 canonical tools organized into 13 workflow groups.
189189

190190
---
191191

192-
*This documentation is automatically generated by `scripts/update-tools-docs.ts` from the tools manifest. Last updated: 2026-02-17T21:48:36.993Z UTC*
192+
*This documentation is automatically generated by `scripts/update-tools-docs.ts` from the tools manifest. Last updated: 2026-02-22T18:16:55.247Z UTC*

docs/TOOLS.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -205,4 +205,4 @@ This document lists MCP tool names as exposed to MCP clients. XcodeBuildMCP prov
205205

206206
---
207207

208-
*This documentation is automatically generated by `scripts/update-tools-docs.ts` from the tools manifest. Last updated: 2026-02-17T21:48:36.993Z UTC*
208+
*This documentation is automatically generated by `scripts/update-tools-docs.ts` from the tools manifest. Last updated: 2026-02-22T18:16:55.247Z UTC*

docs/TROUBLESHOOTING.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
- Check simulator/device availability and permissions.
88

99
## Doctor tool
10-
The doctor tool checks system configuration and reports on all dependencies required by XcodeBuildMCP.
10+
The doctor tool checks system configuration and reports dependency/capability status for key XcodeBuildMCP workflows and runtime features.
1111

1212
```bash
1313
npx --package xcodebuildmcp@latest xcodebuildmcp-doctor
@@ -16,7 +16,7 @@ npx --package xcodebuildmcp@latest xcodebuildmcp-doctor
1616
It reports on:
1717
- System and Node.js environment
1818
- Xcode installation and configuration
19-
- Required dependencies (xcodebuild, AXe, etc.)
19+
- Dependency and capability status (xcodebuild, AXe, debugger/backend requirements, etc.)
2020
- Environment variables affecting XcodeBuildMCP
2121
- Feature availability status
2222

docs/XCODE_IDE_MCPBRIDGE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ When `xcode-ide` is enabled and `mcpbridge` is available, XcodeBuildMCP exposes
2222
- `xcode_ide_list_tools`: Lists remote tools from Xcode's MCP server (name, description, schemas).
2323
- `xcode_ide_call_tool`: Calls a remote Xcode tool by name with a JSON argument payload.
2424

25-
These tools are stable and manifest-managed. They are shown only when `mcpbridge` is available.
25+
These tools are stable and manifest-managed. They are shown when the `xcode-ide` workflow is enabled in MCP runtime; calls will fail until `mcpbridge` is available.
2626

2727
CLI behavior is unchanged and continues to use dynamic `xcode_tools_*` proxy naming.
2828

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Investigation: Doctor Tool Representativeness Audit
2+
3+
## Summary
4+
`doctor` is partially representative, but it has material drift in dependency modeling, runtime/workflow accuracy, and messaging. The biggest issues are false-green/false-red states and one side-effecting check (`xcodemake`) inside a diagnostic command.
5+
6+
## Symptoms
7+
- Concern that `doctor` output no longer matches current manifest-driven tool/workflow architecture.
8+
- Concern that workflow/environment/dependency checks are incomplete or misleading.
9+
10+
## Investigation Log
11+
12+
### Phase 1 - Initial assessment
13+
**Hypothesis:** Doctor may have drifted from implementation after manifest/runtime refactors.
14+
**Findings:** Doctor report is assembled from mixed sources (static manifest summaries + runtime registry snapshots + ad-hoc binary checks).
15+
**Evidence:** `src/mcp/tools/doctor/doctor.ts:102-141`, `src/mcp/tools/doctor/lib/doctor.deps.ts:217-251`
16+
**Conclusion:** Confirmed; this mixed model introduces representativeness gaps.
17+
18+
### Phase 2 - Core doctor behavior audit
19+
**Hypothesis:** Doctor schema and output include stale/unused behavior.
20+
**Findings:** `enabled` exists in schema and tests but is not used by logic.
21+
**Evidence:** `src/mcp/tools/doctor/doctor.ts:24-26`, `src/mcp/tools/doctor/doctor.ts:102-137`, `src/mcp/tools/doctor/__tests__/doctor.test.ts:131-140`
22+
**Conclusion:** Confirmed stale surface area.
23+
24+
### Phase 3 - Dependency and feature checks
25+
**Hypothesis:** Dependency checks are not aligned with runtime behavior.
26+
**Findings:**
27+
- Doctor hardcodes `['axe','xcodemake','mise']` for dependencies.
28+
- Doctor calls `isXcodemakeAvailable()` which can download/install xcodemake (side effect).
29+
- Doctor reports UI automation as supported if AXe exists, but video capture requires AXe >= 1.1.0.
30+
**Evidence:**
31+
- `src/mcp/tools/doctor/doctor.ts:107-111`
32+
- `src/mcp/tools/doctor/doctor.ts:127-130`
33+
- `src/utils/xcodemake.ts:90-105`, `src/utils/xcodemake.ts:156-157`
34+
- `src/mcp/tools/doctor/doctor.ts:288-292`
35+
- `src/mcp/tools/simulator/record_sim_video.ts:81-84`
36+
**Conclusion:** Confirmed false-green/false-red and side-effect risk.
37+
38+
### Phase 4 - Workflow/runtime representativeness
39+
**Hypothesis:** Doctor's tool/workflow summaries do not match actual runtime exposure rules.
40+
**Findings:**
41+
- Plugin summary totals are derived from manifest workflow memberships, not runtime predicate-filtered exposure.
42+
- Runtime registration can be unavailable (doctor prints note) when registry wasn't initialized.
43+
- CLI doctor invokes `doctorLogic` directly without runtime bootstrap.
44+
**Evidence:**
45+
- `src/mcp/tools/doctor/lib/doctor.deps.ts:224-236`
46+
- `src/utils/tool-registry.ts:36-44`, `src/utils/tool-registry.ts:84-102`
47+
- `src/mcp/tools/doctor/doctor.ts:120-127`
48+
- `src/doctor-cli.ts:20-23`
49+
**Conclusion:** Confirmed reporting mismatch between manifest inventory and live exposure/registration.
50+
51+
### Phase 5 - Bridge and docs consistency
52+
**Hypothesis:** Xcode IDE bridge docs and doctor/workflow behavior are inconsistent.
53+
**Findings:**
54+
- Docs claim gateway tools are shown only when `mcpbridge` is available.
55+
- Tool manifests only require `mcpRuntimeOnly` (not bridge availability).
56+
- Runtime actually fails at call-time when `mcpbridge` is missing.
57+
**Evidence:**
58+
- `docs/XCODE_IDE_MCPBRIDGE.md:25`
59+
- `manifests/tools/xcode_ide_list_tools.yaml:7-8`, `manifests/tools/xcode_ide_call_tool.yaml:7-8`
60+
- `src/integrations/xcode-tools-bridge/tool-service.ts:145-149`
61+
**Conclusion:** Confirmed docs/runtime mismatch.
62+
63+
### Phase 6 - Environment and test coverage
64+
**Hypothesis:** Doctor env reporting and tests may not cover current config surface.
65+
**Findings:**
66+
- Doctor env list is partially hardcoded + `XCODEBUILDMCP_*` prefix scan.
67+
- Config also reads non-prefix env inputs (for example `AXE_PATH`, `XBMCP_LAUNCH_JSON_WAIT_MS`) not explicitly surfaced as first-class doctor checks.
68+
- Doctor tests mostly assert text existence/type instead of semantic correctness.
69+
**Evidence:**
70+
- `src/mcp/tools/doctor/lib/doctor.deps.ts:164-184`
71+
- `src/utils/config-store.ts:197-198`, `src/utils/config-store.ts:224`
72+
- `src/mcp/tools/doctor/__tests__/doctor.test.ts:152`, `:172`, `:191`, `:217`, `:283`
73+
**Conclusion:** Confirmed gaps in report fidelity and regression safety.
74+
75+
### Phase 7 - History check
76+
**Hypothesis:** Recent architecture/tool changes may have outpaced doctor updates.
77+
**Findings:**
78+
- Large manifest/runtime refactor landed (`907cfe3c`, Feb 4, 2026).
79+
- xcode-ide/gating changes landed (`9b182d46`, Feb 5, 2026; `898819b9`, Feb 15, 2026).
80+
- Doctor changed during this period but still retains mismatches above.
81+
**Evidence:** Git history via `git log/show` on doctor, manifests, and xcode-ide files.
82+
**Conclusion:** Drift is plausible and observable post-refactor.
83+
84+
## Root Cause
85+
Doctor currently blends multiple data models with different semantics:
86+
1. **Static manifest inventory** (workflow membership totals)
87+
2. **Live runtime registry snapshot** (only after registration runs)
88+
3. **Direct environment/binary probes** (including side-effecting xcodemake path)
89+
90+
Because these sources are not normalized into one explicit capability model, the output can be internally inconsistent and not fully representative of what tools will actually work in the current runtime/context.
91+
92+
## Eliminated Hypotheses
93+
- **"Doctor is totally disconnected from current manifests"**: eliminated. It does read manifests (`doctor.deps.ts:220-236`).
94+
- **"Doctor has no bridge visibility"**: eliminated. It does report bridge status and proxied count (`doctor.ts:329-340`).
95+
- **"No recent doctor maintenance"**: eliminated. Doctor-related files changed during Feb 2026 refactors.
96+
97+
## Recommendations
98+
1. **Make doctor read-only (P0)**
99+
- Replace `isXcodemakeAvailable()` usage in doctor with side-effect-free `isXcodemakeBinaryAvailable()`.
100+
- Keep auto-install behavior in build execution paths, not diagnostics.
101+
102+
2. **Separate inventory/exposure/registration views (P0)**
103+
- Report 3 explicit counts:
104+
- Manifest inventory
105+
- Exposed under current predicate context
106+
- Actually registered now
107+
- Avoid presenting manifest totals as runtime availability.
108+
109+
3. **Capability-based dependency checks (P1)**
110+
- Gate checks by enabled workflows + selected backend/config.
111+
- Add AXe version capability check for video capture (`>=1.1.0`).
112+
- Add explicit bridge dependency health to gateway-tool readiness section.
113+
114+
4. **Fix docs/runtime inconsistencies (P1)**
115+
- Correct `docs/XCODE_IDE_MCPBRIDGE.md` claim about visibility on `mcpbridge` availability, or add a predicate/system that enforces that claim.
116+
- Adjust `docs/TROUBLESHOOTING.md` wording (“all dependencies required”) to reflect scope-based checks.
117+
118+
5. **Improve env and workflow-management visibility (P2)**
119+
- Surface key non-prefix env vars that materially alter runtime behavior (for example `AXE_PATH`, `XBMCP_LAUNCH_JSON_WAIT_MS`).
120+
- Include current predicate context summary (runtime/debug/runningUnderXcode) so workflow results are explainable.
121+
122+
6. **Strengthen tests (P2)**
123+
- Add semantic assertions (specific sections/flags/warnings) instead of mostly `typeof text === 'string'`.
124+
- Add regression tests for false-green/false-red scenarios (AXe version mismatch, xcodemake unavailable but disabled, runtime registry missing).
125+
126+
## Preventive Measures
127+
- Define a typed internal "doctor capability model" (single source for checks and statuses) and render output from that model only.
128+
- Add contract tests that compare doctor output against manifest/runtime predicate expectations for known fixture configs.
129+
- Add CI check to fail if docs claims about tool visibility/dependencies contradict manifest predicates or runtime gating behavior.

src/doctor-cli.ts

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,16 +10,30 @@
1010
import { version } from './version.ts';
1111
import { doctorLogic } from './mcp/tools/doctor/doctor.ts';
1212
import { getDefaultCommandExecutor } from './utils/execution/index.ts';
13+
import { bootstrapRuntime } from './runtime/bootstrap-runtime.ts';
14+
15+
function shouldUseNonRedactedOutput(argv: string[]): boolean {
16+
return argv.includes('--non-redacted');
17+
}
1318

1419
async function runDoctor(): Promise<void> {
1520
try {
1621
// Using console.error to avoid linting issues as it's allowed by the project's linting rules
1722
console.error(`Running XcodeBuildMCP Doctor (v${version})...`);
1823
console.error('Collecting system information and checking dependencies...\n');
1924

25+
await bootstrapRuntime({ runtime: 'cli' });
26+
27+
const nonRedacted = shouldUseNonRedactedOutput(process.argv.slice(2));
28+
if (nonRedacted) {
29+
console.error(
30+
'Warning: --non-redacted is enabled; doctor output may include sensitive data.',
31+
);
32+
}
33+
2034
// Run the doctor tool logic directly with CLI flag enabled
2135
const executor = getDefaultCommandExecutor();
22-
const result = await doctorLogic({}, executor, true); // showAsciiLogo = true for CLI
36+
const result = await doctorLogic({ nonRedacted }, executor, true); // showAsciiLogo = true for CLI
2337

2438
// Output the doctor information
2539
if (result.content && result.content.length > 0) {

0 commit comments

Comments
 (0)