Skip to content

ab-testing-guide.md

Latest commit

 

History

History
218 lines (162 loc) · 5.08 KB

ab-testing-guide.md

File metadata and controls

218 lines (162 loc) · 5.08 KB
Raw

Claude Code Graph A/B Testing Guide

🎯 Objective Metrics for Social Media

Test Setup

# 1. Clone test repository
git clone https://github.com/vercel/next.js.git
cd next.js

# 2. Initialize graphs  
ccg init
ccg daemon start

# 3. Start A/B test session
ccg start

📊 Test Scenarios & Commands

Scenario 1: Find Authentication Files

Objective: Measure tool calls needed to find auth-related code

Normal Claude Code:

User: "Find all files related to authentication in this codebase"
Expected: 8-15 tool calls (broad Grep searches, multiple Read operations)

Graph-Enhanced Claude Code:

User: "Find all files related to authentication in this codebase"
Expected: 3-5 tool calls (graph analysis, targeted reads)

Measurement: Count tool calls in Claude's trace before getting complete answer

Scenario 2: Understanding Component Dependencies

Objective: Speed and accuracy of dependency discovery

Normal Claude Code:

User: "Show me what files depend on the Button component"
Expected: Multiple grep operations, manual file browsing

Graph-Enhanced Claude Code:

User: "Show me what files depend on the Button component"  
Expected: Single graph query with immediate relationships

Commands to verify:

  • /find-related packages/next/client/components/Button.tsx
  • /graph-overview (architectural context)

Scenario 3: Impact Analysis

Objective: Understanding change impact before making modifications

Normal Claude Code:

User: "What would break if I modify the routing system?"
Expected: Text searches, educated guessing

Graph-Enhanced Claude Code:

User: "What would break if I modify the routing system?"
Expected: Precise dependency tracing via graph relationships

Commands to verify:

  • /hot-paths (find critical code paths)
  • /cycles (identify circular dependencies)

Scenario 4: Architecture Overview

Objective: Speed of codebase understanding for new developers

Normal Claude Code:

User: "Explain the architecture of this project" 
Expected: 10+ file reads, manual exploration

Graph-Enhanced Claude Code:

User: "Explain the architecture of this project"
Expected: Instant overview via graph analysis

Commands to verify:

  • /graph-overview (instant architectural summary)
  • /graph-stats (quantitative metrics)

🏆 Success Metrics

Quantitative (Twitter-worthy)

  • Tool Calls Reduction: 60-80% fewer tool calls per task
  • Time to Answer: 3-5x faster for structural queries
  • Accuracy: 90%+ precision in file discovery vs 60% with text search
  • Context Efficiency: Graph provides architectural context immediately

Sample Tweet Metrics

🧠 Claude Code Graph Results:

Task: "Find authentication system files"
❌ Normal: 12 tool calls, 45 seconds  
✅ Graph: 3 tool calls, 8 seconds

Task: "What uses Component X?"
❌ Normal: 8 grep searches, manual filtering
✅ Graph: 1 relationship query, instant results

75% fewer tool calls, 80% faster 🚀

🎮 Interactive Testing Session

Live Demo Commands

# 1. Show system status
ccg status

# 2. Display architectural overview  
# (Run in Claude Code session)
/graph-overview

# 3. Find specific relationships
/find-related packages/next/client/router.ts

# 4. Show hot paths
/hot-paths --limit=5

# 5. Check for circular dependencies
/cycles

# 6. Performance comparison
time grep -r "useRouter" . --include="*.ts" --include="*.tsx" 
# vs
/find-related packages/next/client/use-router.ts

📈 Measurement Tools

Tool Call Counter Script

// Add to Claude session for counting
let toolCallCount = 0;
const originalTool = window.callTool;
window.callTool = function(...args) {
    toolCallCount++;
    console.log(`Tool call #${toolCallCount}:`, args[0]);
    return originalTool.apply(this, args);
};

// Reset counter
toolCallCount = 0;

Performance Timer

# Time graph operations
time ccg build

# Time daemon responsiveness
echo "test.js" > test.js && time ccg daemon status

🔄 A/B Test Protocol

  1. Control Group (Normal Claude Code):

    • Use standard Claude Code without graph system
    • Record tool calls and time for each task

  2. Test Group (Graph-Enhanced):

    • Use claude-code-graph with daemon running
    • Record same metrics for identical tasks

  3. Comparison Metrics:

    • Tool calls per task completion
    • Time to accurate answer
    • Precision of file discovery
    • Quality of architectural insights

  4. Document Results:

    • Screenshots of tool traces
    • Timing data
    • Accuracy comparisons
    • User experience notes

🎯 Ready-to-Tweet Results

Expected improvements:

  • 60-80% reduction in tool calls
  • 3-5x faster structural queries
  • 90%+ precision in file discovery
  • Instant architectural context vs manual exploration

The graph intelligence transforms Claude from a powerful text processor into a code architecture expert that understands your entire system structure! 🧠⚡