Skip to content

COMPREHENSIVE-VALIDATION-ADVICE.md

Latest commit

 

History

History
431 lines (357 loc) · 11.1 KB

COMPREHENSIVE-VALIDATION-ADVICE.md

File metadata and controls

431 lines (357 loc) · 11.1 KB
Raw

COMPREHENSIVE-VALIDATION-ADVICE.md

Best practices, optimization strategies, and troubleshooting guide for running the comprehensive validation suite.

Best Practices

1. Pre-Validation Checklist

Always verify before starting comprehensive validation:

# System resources check
df -h  # Ensure >5GB free disk space for logs
free -h  # Ensure >2GB free RAM for parallel tests

# Service readiness
docker ps  # Verify containers are running
lsof -i :8080  # Backend API
lsof -i :3001  # Frontend dev server

# Clean test environment
rm -rf validation-reports/temp_*  # Remove incomplete sessions
aws dynamodb scan --table-name quest-dev-test --select COUNT  # Check test data volume

2. Optimal Execution Strategy

Maximize efficiency while maintaining reliability:

Resource-Based Grouping

// Group tests by resource requirements
const executionGroups = {
  // Run first - minimal resources, critical for other tests
  infrastructure: [
    'validate-backend-health',
    'validate-frontend-backend-integration'
  ],

  // Run in parallel - independent database operations
  dataOperations: [
    'validate-data-persistence',
    'validate-animal-config-persistence'
  ],

  // Run sequentially - heavy browser usage
  uiValidations: [
    'validate-family-dialog',
    'validate-animal-config-fields',
    'validate-chat-dynamodb'
  ],

  // Run last - most comprehensive, longest duration
  comprehensive: [
    'validate-full-animal-config',
    'validate-family-management'
  ]
};

Parallel Execution Limits

# Determine safe parallel execution count
CPU_CORES=$(nproc)
SAFE_PARALLEL=$((CPU_CORES / 2))  # Use half of available cores

# For browser-based tests, limit to 2-3 concurrent
MAX_BROWSER_PARALLEL=3

3. Failure Recovery Strategy

Handle failures gracefully:

# Implement retry logic for flaky tests
retry_test() {
  local test_name=$1
  local max_attempts=3
  local attempt=1

  while [ $attempt -le $max_attempts ]; do
    if ".claude/commands/$test_name.md"; then
      echo "✅ $test_name passed on attempt $attempt"
      return 0
    fi
    echo "⚠️ $test_name failed on attempt $attempt"
    attempt=$((attempt + 1))
    sleep 5  # Wait before retry
  done

  echo "❌ $test_name failed after $max_attempts attempts"
  return 1
}

4. Performance Optimization

Speed up validation without sacrificing accuracy:

# Cache DynamoDB table descriptions
aws dynamodb list-tables --output json > /tmp/dynamo-tables.json

# Reuse authentication tokens
export AUTH_TOKEN=$(curl -X POST http://localhost:8080/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email": "test@cmz.org", "password": "testpass123"}' \
  | jq -r '.token')

# Skip redundant health checks
export SKIP_HEALTH_CHECK=true  # After initial verification

Common Pitfalls and Solutions

Issue 1: Test Interference

Problem: Tests modifying shared data cause failures Solution:

# Use test-specific namespaces
TEST_PREFIX="test_$(date +%s)_"

# Clean up after each test
trap 'cleanup_test_data' EXIT

cleanup_test_data() {
  aws dynamodb delete-item \
    --table-name quest-dev-conversations \
    --key "{\"conversationId\": {\"S\": \"${TEST_PREFIX}conv\"}}"
}

Issue 2: Resource Exhaustion

Problem: Too many parallel tests cause system slowdown Solution:

# Monitor system resources during execution
monitor_resources() {
  while true; do
    echo "$(date): CPU: $(top -bn1 | grep "Cpu(s)" | awk '{print $2}')% | MEM: $(free -m | awk 'NR==2{printf "%.1f%%", $3*100/$2}')" >> resource-usage.log
    sleep 10
  done
}

monitor_resources &
MONITOR_PID=$!
trap "kill $MONITOR_PID" EXIT

Issue 3: Incomplete Test Runs

Problem: Validation suite interrupted, leaving partial results Solution:

# Implement checkpoint system
save_checkpoint() {
  local test_name=$1
  local status=$2
  echo "{\"test\": \"$test_name\", \"status\": \"$status\", \"timestamp\": \"$(date -Iseconds)\"}" >> checkpoint.jsonl
}

resume_from_checkpoint() {
  if [ -f checkpoint.jsonl ]; then
    COMPLETED_TESTS=$(jq -r '.test' checkpoint.jsonl)
    echo "Resuming from checkpoint. Already completed: $COMPLETED_TESTS"
  fi
}

Issue 4: Flaky Browser Tests

Problem: Playwright tests fail due to timing issues Solution:

// Implement smart waits
async function smartWait(page, selector, options = {}) {
  const defaults = {
    timeout: 30000,
    retries: 3,
    retryDelay: 1000
  };

  const config = { ...defaults, ...options };

  for (let i = 0; i < config.retries; i++) {
    try {
      await page.waitForSelector(selector, { timeout: config.timeout });
      return true;
    } catch (error) {
      if (i === config.retries - 1) throw error;
      await page.waitForTimeout(config.retryDelay);
    }
  }
}

Issue 5: Report Generation Failures

Problem: Large test outputs cause report generation to fail Solution:

# Stream process large results
generate_report_streaming() {
  local results_file=$1

  # Process results in chunks
  split -l 100 $results_file /tmp/results_chunk_

  for chunk in /tmp/results_chunk_*; do
    process_chunk $chunk >> final_report.md
  done

  rm /tmp/results_chunk_*
}

Integration Guidelines

CI/CD Integration

# GitHub Actions example
name: Comprehensive Validation
on:
  schedule:
    - cron: '0 2 * * *'  # Run nightly
  workflow_dispatch:  # Manual trigger

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Setup environment
        run: |
          make build-api
          make run-api &
          cd frontend && npm install && npm run dev &
          sleep 30  # Wait for services

      - name: Run comprehensive validation
        run: |
          .claude/commands/comprehensive-validation.md --parallel

      - name: Upload reports
        uses: actions/upload-artifact@v2
        with:
          name: validation-reports
          path: validation-reports/

Slack/Teams Notifications

# Send summary to Slack
send_validation_summary() {
  local webhook_url=$SLACK_WEBHOOK_URL
  local passed=$1
  local failed=$2
  local total=$3

  curl -X POST $webhook_url \
    -H 'Content-Type: application/json' \
    -d "{
      \"text\": \"Validation Complete\",
      \"attachments\": [{
        \"color\": \"$([ $failed -eq 0 ] && echo 'good' || echo 'danger')\",
        \"fields\": [
          {\"title\": \"Total Tests\", \"value\": \"$total\", \"short\": true},
          {\"title\": \"Passed\", \"value\": \"$passed ✅\", \"short\": true},
          {\"title\": \"Failed\", \"value\": \"$failed ❌\", \"short\": true},
          {\"title\": \"Success Rate\", \"value\": \"$((passed * 100 / total))%\", \"short\": true}
        ]
      }]
    }"
}

Troubleshooting Procedures

Diagnostic Commands

# 1. Check which tests are hanging
ps aux | grep validate | grep -v grep

# 2. Find resource bottlenecks
iostat -x 1 10  # Disk I/O
netstat -an | grep ESTABLISHED | wc -l  # Network connections

# 3. Analyze test logs
find validation-reports -name "*.log" -exec grep -l ERROR {} \;

# 4. Check for port conflicts
lsof -i :8080,3001,5432,6379

# 5. Verify AWS connectivity
aws sts get-caller-identity
aws dynamodb describe-limits

Recovery Procedures

Kill Hanging Tests

# Find and kill hanging validation processes
pkill -f "validate-" || true
pkill -f "playwright" || true

# Clean up orphaned browser processes
pkill -f "chrome" || true
pkill -f "firefox" || true

Reset Test Environment

# Full environment reset
reset_test_environment() {
  echo "Stopping all services..."
  make stop-api
  pkill -f "npm run dev" || true

  echo "Cleaning test data..."
  aws dynamodb scan --table-name quest-dev-test \
    --filter-expression "begins_with(id, :prefix)" \
    --expression-attribute-values '{":prefix": {"S": "test_"}}' \
    --output json | jq -r '.Items[].id.S' | \
    xargs -I {} aws dynamodb delete-item \
      --table-name quest-dev-test \
      --key "{\"id\": {\"S\": \"{}\"}}"

  echo "Restarting services..."
  make run-api &
  cd frontend && npm run dev &
  sleep 30

  echo "Environment reset complete"
}

Advanced Usage Scenarios

1. Selective Validation

# Run only specific categories
./comprehensive-validation.sh --filter "animal,family"

# Exclude heavy tests
./comprehensive-validation.sh --exclude "full-animal-config"

# Run only failed tests from previous run
FAILED_TESTS=$(jq -r 'select(.status == "FAIL") | .test' last-run/results.jsonl)
for test in $FAILED_TESTS; do
  .claude/commands/$test.md
done

2. Performance Profiling

# Profile test execution
/usr/bin/time -v ./comprehensive-validation.sh 2>&1 | tee performance.log

# Analyze slowest tests
jq -s 'sort_by(.duration) | reverse | .[0:5]' results.jsonl

3. Comparison Testing

# Compare results across branches
run_comparison() {
  local branch1=$1
  local branch2=$2

  git checkout $branch1
  ./comprehensive-validation.sh
  mv validation-reports validation-reports-$branch1

  git checkout $branch2
  ./comprehensive-validation.sh
  mv validation-reports validation-reports-$branch2

  # Compare results
  diff -u \
    <(jq -s 'sort_by(.test)' validation-reports-$branch1/results.jsonl) \
    <(jq -s 'sort_by(.test)' validation-reports-$branch2/results.jsonl)
}

4. Load Testing Integration

# Run validation under load
start_load_generator() {
  artillery run load-test.yml &
  LOAD_PID=$!

  ./comprehensive-validation.sh --tag "under-load"

  kill $LOAD_PID
}

Performance Benchmarks

Expected Execution Times

Test Category Single Test Parallel (3 tests)
Infrastructure 5-10s 10-15s
Data Persistence 15-30s 20-35s
UI Validation 30-60s 60-90s
Comprehensive 2-5min N/A

Resource Usage

  • Memory: 2-4GB peak during parallel execution
  • CPU: 50-70% utilization with full parallelization
  • Disk I/O: 10-50MB of logs generated
  • Network: 100-500 API calls per full suite

Continuous Improvement

Metrics to Track

  1. Execution Time Trend: Monitor if tests are getting slower
  2. Flakiness Rate: Track tests that fail intermittently
  3. Resource Usage: Identify tests consuming excessive resources
  4. Failure Patterns: Analyze common failure causes

Optimization Opportunities

  1. Test Data Caching: Cache reference data between runs
  2. Smart Ordering: Run fast tests first for quick feedback
  3. Conditional Execution: Skip tests unaffected by changes
  4. Parallel Optimization: Continuously tune parallel execution groups

Recent Improvements

  • Added checkpoint/resume capability for interrupted runs
  • Implemented smart retry logic for flaky tests
  • Optimized parallel execution grouping
  • Added resource monitoring during execution
  • Improved error categorization in reports

Related Documentation

  • .claude/commands/comprehensive-validation.md - Main command
  • .claude/commands/validate-*.md - Individual validation commands
  • tests/README.md - Testing strategy documentation
  • CI/CD Pipeline Guide - Integration with automated testing