feat: add stepCount parameter for plan validation and progress tracking

[jerfowler]

Enhancement Issue: Plan Step Count Validation

Issue Type: Feature Request
Priority: High
Category: Validation, Performance
Related Issues: #56 (Bugs #1 and #3)

Problem Statement

The current MCP server architecture has fundamental issues with plan validation and progress tracking due to the lack of a single source of truth for plan structure. This causes:

1. Bug docs: add Git Feature Branch Workflow section to README #1: Progress percentage calculations returning incorrect values (0% or wrong percentages)
2. Bug feat: implement comprehensive GitHub issue workflow automation #3: Step number validation accepting out-of-bounds values
3. Performance overhead from repeated plan parsing
4. Inconsistent validation across different tools

Current Architecture Problems

1. Inconsistent Regex Patterns

Different tools use different regex patterns to parse checkboxes:

• submit-plan.ts: /^- \[ \] \*\*[^:]+\*\*:/gm (requires bold title with colon)
• report-progress.ts: /^- \[[ x]\] \*\*.*?\*\*/gm (requires bold, any content)
• track-task-progress.ts: /^(\s*)-\s*\[([ x])\]\s*(.+)$/ (flexible, no bold required)

2. Repeated Plan Parsing

Every operation that needs step count must:

1. Read the PLAN.md file
2. Parse the content with regex
3. Count checkboxes
4. Validate against that count

This happens in:

• report-progress.ts (lines 56-57, 71-72)
• track-task-progress.ts (lines 94-142)
• Any future tool needing step information

3. No Plan Metadata Storage

Plans are stored as raw markdown without metadata:

• No stored step count
• No validation timestamp
• No plan version tracking
• No structured data for quick access

Proposed Solution

Core Enhancement: Add Required stepCount Parameter

Transform the plan submission API to require explicit step count:

// Current API (vulnerable to mismatches)
submitPlan(config, {
  agent: string,
  content: string,
  taskId?: string
})

// Enhanced API (validated at submission)
submitPlan(config, {
  agent: string,
  content: string,
  stepCount: number,    // NEW: Required parameter
  taskId?: string
})

Validation Flow

1. At Plan Submission (Proactive Validation)

// In submit-plan.ts
function validatePlanWithStepCount(content: string, expectedStepCount: number) {
  const checkboxRegex = /^(\s*)-\s*\[([ x])\]\s+/gm;  // Standardized pattern
  const matches = content.match(checkboxRegex) || [];

  if (matches.length !== expectedStepCount) {
    throw new Error(
      `Step count mismatch: Expected ${expectedStepCount} steps, ` +
      `but plan contains ${matches.length} checkboxes`
    );
  }

  // Store metadata
  const metadata = {
    stepCount: expectedStepCount,
    createdAt: new Date().toISOString(),
    checkboxPattern: 'markdown',
    version: '2.0.0'
  };

  return metadata;
}

2. Plan Metadata Storage

Create PLAN.metadata.json alongside PLAN.md:

{
  "stepCount": 5,
  "createdAt": "2025-09-15T10:00:00Z",
  "agent": "qa-test-automation-engineer",
  "checkboxPattern": "markdown",
  "version": "2.0.0",
  "validation": {
    "checksumMatch": true,
    "lastValidated": "2025-09-15T10:00:00Z"
  }
}

3. At Progress Reporting (Use Stored Metadata)

// In report-progress.ts
async function validateStepNumber(step: number, taskDir: string) {
  // Read metadata instead of parsing plan
  const metadataPath = path.join(taskDir, 'PLAN.metadata.json');
  const metadata = await fs.readJson(metadataPath);

  if (step < 1 || step > metadata.stepCount) {
    throw new Error(
      `Invalid step number ${step}: Must be between 1 and ${metadata.stepCount}`
    );
  }
}

4. At Progress Tracking (Use Stored Metadata)

// In track-task-progress.ts
async function calculateProgress(taskDir: string) {
  const metadataPath = path.join(taskDir, 'PLAN.metadata.json');
  const metadata = await fs.readJson(metadataPath);

  const planPath = path.join(taskDir, 'PLAN.md');
  const planContent = await fs.readFile(planPath, 'utf8');

  // Parse only checked boxes, total is from metadata
  const checkedRegex = /^(\s*)-\s*\[x\]\s+/gm;
  const checkedCount = (planContent.match(checkedRegex) || []).length;

  return {
    totalSteps: metadata.stepCount,
    completedSteps: checkedCount,
    percentage: Math.round((checkedCount / metadata.stepCount) * 100)
  };
}

Implementation Plan

Phase 1: Foundation (Week 1)

1. Standardize Checkbox Regex Pattern

   • Create src/utils/plan-parser.ts with single regex
   • Export consistent parsing functions
   • Add comprehensive tests

2. Add Metadata Support

   • Create metadata types in src/types/plan-types.ts
   • Add metadata read/write utilities
   • Update TaskContextManager to handle metadata

Phase 2: API Enhancement (Week 1-2)

1. Update submit-plan Tool

   • Add stepCount parameter (optional initially)
   • Validate step count matches checkboxes
   • Create and store metadata file
   • Add deprecation warning for missing stepCount

2. Update report-progress Tool

   • Check for metadata file first
   • Fall back to parsing if no metadata (backward compat)
   • Use metadata.stepCount for validation

3. Update track-task-progress Tool

   • Use metadata for total steps
   • Parse only for checked count
   • Improve performance significantly

Phase 3: Testing & Migration (Week 2)

1. Comprehensive Testing

   • TDD tests for all new functionality
   • Integration tests for full workflow
   • Performance benchmarks

2. Migration Strategy

   • Auto-generate metadata for existing plans
   • Gradual rollout with feature flag
   • Documentation updates

Benefits

1. Correctness

• Bug docs: add Git Feature Branch Workflow section to README #1 Fixed: Accurate progress percentages using definitive step count
• Bug feat: implement comprehensive GitHub issue workflow automation #3 Enhanced: Robust step validation with stored bounds
• Consistency: Single source of truth for plan structure

2. Performance

• Current: ~100ms per validation (parse plan each time)
• Enhanced: <10ms per validation (read metadata)
• 90% reduction in validation overhead

3. Reliability

• Validation at creation: Catch errors early
• Immutable step count: Prevents drift over time
• Audit trail: Metadata tracks plan history

4. Maintainability

• Single regex pattern: Easier to update
• Centralized validation: One place to fix bugs
• Clear separation: Structure (metadata) vs content (markdown)

Breaking Changes & Migration

Breaking Changes

1. API signature change for submitPlan
2. New required parameter stepCount
3. Metadata file requirement for new plans

Migration Strategy

Phase 1: Soft Launch (v2.1.0)

// Make stepCount optional with warning
if (!args.stepCount) {
  console.warn('DEPRECATION: stepCount will be required in v3.0.0');
  const calculated = countCheckboxes(content);
  args.stepCount = calculated;
}

Phase 2: Enforcement (v3.0.0)

// Make stepCount required
if (!args.stepCount) {
  throw new Error('stepCount is required for plan submission');
}

Migration Script

# Script to add metadata to existing plans
npm run migrate:add-plan-metadata

Acceptance Criteria

Functional Requirements

• Plans with incorrect stepCount are rejected at submission
• Progress percentage calculations are always accurate
• Step validation prevents out-of-bounds updates
• Backward compatibility for existing plans
• Migration script successfully processes all existing plans

Performance Requirements

• Plan validation < 10ms with metadata
• No performance regression for existing operations
• Metadata file size < 1KB

Quality Requirements

• 95%+ test coverage maintained
• TypeScript strict mode compliance
• ESLint zero warnings
• Comprehensive documentation

Testing Strategy

Unit Tests

describe('Plan Step Count Validation', () => {
  describe('submitPlan with stepCount', () => {
    it('should accept valid stepCount matching checkboxes', async () => {
      const plan = '- [ ] Step 1\n- [ ] Step 2\n- [ ] Step 3';
      const result = await submitPlan(config, {
        agent: 'test-agent',
        content: plan,
        stepCount: 3  // Matches
      });
      expect(result.success).toBe(true);
    });

    it('should reject mismatched stepCount', async () => {
      const plan = '- [ ] Step 1\n- [ ] Step 2';
      await expect(submitPlan(config, {
        agent: 'test-agent',
        content: plan,
        stepCount: 5  // Mismatch!
      })).rejects.toThrow('Step count mismatch: Expected 5 steps, but plan contains 2 checkboxes');
    });
  });
});

Integration Tests

describe('Full Workflow with Step Count', () => {
  it('should track progress accurately using metadata', async () => {
    // Submit plan with stepCount
    await submitPlan(config, {
      agent: 'test-agent',
      content: '- [ ] Step 1\n- [ ] Step 2',
      stepCount: 2
    });

    // Report progress (uses metadata)
    await reportProgress(config, {
      agent: 'test-agent',
      updates: [{ step: 1, status: 'COMPLETE', description: 'Done' }]
    });

    // Track progress (uses metadata)
    const progress = await trackTaskProgress(config, {
      agent: 'test-agent'
    });

    expect(progress.percentage).toBe(50);  // 1 of 2 complete
  });
});

Risk Assessment

Low Risk

• Backward compatible with gradual migration
• Existing functionality preserved
• Feature flag for rollback if needed

Mitigations

• Comprehensive testing before release
• Phased rollout over 2 versions
• Clear migration documentation
• Monitoring for validation failures

Timeline

• Week 1: Foundation + API updates
• Week 2: Testing + Migration tools
• Week 3: Documentation + Release prep
• Total: 3 weeks to production

Related Issues

• Issue bug: Multiple critical bugs found in comprehensive MCP server testing #56: Critical MCP Server Bugs (source of Bug docs: add Git Feature Branch Workflow section to README #1 and Bug feat: implement comprehensive GitHub issue workflow automation #3)
• Issue feat: Enhanced Agent Context Reporting and Proactive Task Management System #51: Enhanced Agent Context (may need stepCount in context)

References

• PROTOCOL.md - Current API documentation
• Bug Report #56 - Detailed bug analysis
• CONTRIBUTING.md - Development workflow

---

Ready for GitHub Issue Creation

Use this content with the GitHub CLI:

gh issue create \
  --title "feat: add stepCount parameter for plan validation and progress tracking" \
  --body-file ./tmp/issue-56/ENHANCEMENT-ISSUE-STEP-COUNT.md \
  --label enhancement,priority:high,category:validation \
  --assignee @me

[github-actions]

🔗 Linked Pull Request

PR #65 has been created to address this issue.

PR Title: feat: implement Issue #60 - Add stepCount parameter for plan validation
Author: @jerfowler
Branch: issue-60-step-count-validation

View Pull Request →

[github-actions]

🔗 Linked Pull Request

PR #65 has been created to address this issue.

PR Title: feat: implement Issue #60 - Add stepCount parameter for plan validation
Author: @jerfowler
Branch: issue-60-step-count-validation

View Pull Request →