api-probe

Containerized API validation tool for post-deployment functional testing in CI/CD pipelines.

Features

✅ REST & GraphQL Support
✅ XML/SOAP Support (XPath)
✅ Multiple Execution Contexts (test different users/accounts)
✅ Variable Substitution in requests AND validations
✅ Output Variable Capture with expressions (len, has, empty)
✅ Conditional Probe Execution (ignore field with expressions)
✅ Conditional Validation (skip headers/body based on response)
✅ Parallel Group Execution (flat and staged with isolated variable scopes)
✅ Include Directive (!include) for external files
✅ Rich Validations (status, headers, body)
✅ Timeout, Retry, Debug support
✅ Response Time Validation
✅ Progress Reporting to stderr
✅ Silent Success / Verbose Failure
✅ Docker-First, CI/CD Native

Quick Start

Local Development

# One command - auto-setup and run
./run.sh examples/passing/simple.yaml

# Run all examples
chmod +x run-examples.sh
./run-examples.sh

Docker

# Build
docker build -t api-probe .

# Run
docker run --rm \
  -v $(pwd)/examples:/configs \
  api-probe /configs/passing/simple.yaml

Example

executions:
  - name: "Production User"
    vars:
      - ACCOUNT: "123456789"
      - API_KEY: "${PROD_API_KEY}"  # From environment
  
  - name: "Staging User"
    vars:
      - ACCOUNT: "987654321"
      - API_KEY: "${STAGING_API_KEY}"

probes:
  - name: "Login"
    type: rest
    endpoint: "${BASE_URL}/auth"
    method: POST
    headers:
      Content-Type: "application/json"
    body: !include includes/login-body.json  # External file
    validation:
      status: 200
      response_time: 1000  # Must respond within 1 second
      body:
        equals:
          account_id: "${ACCOUNT}"  # Variable substitution in validation!
    output:
      TOKEN: "access_token"
      HAS_PREMIUM: "has(body.premium)"  # Expression evaluation
  
  - name: "Get Profile"
    type: rest
    endpoint: "${BASE_URL}/profile"
    headers:
      Authorization: "Bearer ${TOKEN}"
    validation:
      status: 200
      body:
        present: ["id", "email"]
  
  - name: "Get Premium Features"
    type: rest
    endpoint: "${BASE_URL}/features"
    ignore: "!HAS_PREMIUM"  # Skip if not premium user
    validation:
      status: 200
      body:
        ignore: "empty(body.features)"  # Skip validation if empty
        present: ["features[0].name"]

Multiple Execution Contexts

Test different user accounts, regions, or environments in one run:

executions:
  - name: "US East User"
    vars:
      - CLIENT_ID: "client-123"
      - REGION: "us-east-1"
  
  - name: "EU West User"
    vars:
      - CLIENT_ID: "client-456"
      - REGION: "eu-west-1"

probes:
  - name: "API Test"
    endpoint: "https://api.example.com/${REGION}/data"
    headers:
      X-Client: "${CLIENT_ID}"
    validation:
      body:
        equals:
          region: "${REGION}"  # Validates against execution-specific value

Each execution runs independently with isolated variables.

See examples/passing/executions-block.yaml for details.

Conditional Execution

Skip probes or validation based on previous results:

probes:
  - name: "Get Offers"
    endpoint: "https://api.example.com/offers"
    output:
      OFFER_COUNT: "len(body.offers)"  # Capture count using expression
      HAS_PREMIUM: "has(body.premium)"
  
  - name: "Process Rich Offers"
    endpoint: "https://api.example.com/process"
    ignore: "OFFER_COUNT <= 2"  # Skip if not enough offers
    validation:
      status: 200
  
  - name: "Validate Premium Features"
    endpoint: "https://api.example.com/user"
    validation:
      status: 200
      body:
        ignore: "!HAS_PREMIUM"  # Skip body validation if not premium
        present:
          - "premium.tier"
          - "premium.benefits"

Expression Functions:

• len(VAR) - Get length of array/string/dict
• has(VAR) - Check if exists and not empty
• empty(VAR) - Check if empty or None

Operators: ==, !=, >, <, >=, <=, &&, ||, !

See SCHEMA_SPECIFICATION.md for details.

Parallel Groups

Groups support two modes: flat (all probes in parallel) and staged (stages in parallel, probes within each stage sequential).

Flat Group

All probes run at the same time:

probes:
  - name: "Sequential Test"
    endpoint: "https://api.example.com/test"
  
  - group:
      probes:
        - name: "Parallel Test 1"
          endpoint: "https://api.example.com/delay/2"
        - name: "Parallel Test 2"
          endpoint: "https://api.example.com/delay/2"
        - name: "Parallel Test 3"
          endpoint: "https://api.example.com/delay/2"
  # Group completes in ~2 seconds instead of 6 seconds

Staged Group

Stages run in parallel, but probes within each stage run sequentially. Each stage has an isolated variable scope — output captured in one stage does not leak to sibling stages:

probes:
  - group:
      name: "Multi-User Auth Flow"
      stages:
        - name: "User A"
          probes:
            - name: "Auth - User A"       # runs first
              type: rest
              endpoint: "${AUTH_URL}"
              output:
                TOKEN_A: "data.token"
            - name: "API Call - User A"   # runs after Auth - User A
              type: graphql               # skipped if TOKEN_A not set
              endpoint: "${BASE_URL}/graphql"
              headers:
                Authorization: "Bearer ${TOKEN_A}"

        - name: "User B"                  # runs in parallel with User A
          probes:
            - name: "Auth - User B"
              type: rest
              endpoint: "${AUTH_URL}"
              output:
                TOKEN_B: "data.token"     # isolated — not visible in User A stage
            - name: "API Call - User B"
              type: graphql
              endpoint: "${BASE_URL}/graphql"
              headers:
                Authorization: "Bearer ${TOKEN_B}"

See examples/passing/groups-parallel.yaml for details.

Include Directive

Keep large request bodies in separate files:

probes:
  - name: "Create User"
    type: rest
    endpoint: "https://api.example.com/users"
    method: POST
    headers:
      Content-Type: "application/json"
    body: !include includes/user-profile.json
    validation:
      status: 201

See examples/passing/include-directive.yaml for details.

Documentation

• GETTING_STARTED.md - Installation and basic usage
• SCHEMA_SPECIFICATION.md - Complete YAML syntax
• DOCKER.md - Container usage and CI/CD integration
• XML_SOAP_GUIDE.md - XPath expressions and SOAP testing
• INCLUDE_DIRECTIVE.md - YAML include directive usage
• PROMPT.md - Generate configs with AI

Examples

Passing Examples (Expected: ✓ Silent Success)

• simple.yaml - Basic REST API probes
• comprehensive.yaml - All validation keywords
• complex-validation.yaml - Advanced patterns
• graphql.yaml - GraphQL API testing
• xml-soap.yaml - XML/SOAP with XPath
• executions-block.yaml - Multiple execution contexts
• multi-context.yaml - Multi-user testing
• no-executions.yaml - Single run with env vars
• groups-parallel.yaml - Parallel group execution
• include-directive.yaml - External file includes
• advanced-features.yaml - JSONPath + parallel groups

Failing Examples (Expected: ✗ Verbose Errors)

• test-failures.yaml - Basic intentional failures
• validation-failures.yaml - All validator failures
• variable-validation-failures.yaml - Variable validation errors
• group-failures.yaml - Failures in parallel groups
• execution-names-in-reports.yaml - Execution names in failure output
• multiple-execution-failures.yaml - Different failures per execution
• auto-generated-names-failures.yaml - Auto-generated execution names

Running All Examples

# Set executable permission
chmod +x run-examples.sh

# Run all passing and failing examples
./run-examples.sh

This will run all 15 passing examples and 7 failing examples with proper environment variables.

CI/CD Integration

GitHub Actions

- name: API Tests
  run: |
    docker run --rm \
      -v ${{ github.workspace }}/configs:/configs \
      -e PROD_API_KEY="${{ secrets.PROD_API_KEY }}" \
      -e STAGING_API_KEY="${{ secrets.STAGING_API_KEY }}" \
      api-probe:latest /configs/tests.yaml

Concourse

- task: api-tests
  image: api-probe-image
  params:
    PROD_API_KEY: ((prod-api-key))
    STAGING_API_KEY: ((staging-api-key))
  run:
    path: api-probe
    args: ["/configs/tests.yaml"]

See DOCKER.md for more CI/CD examples.

Installation

Docker (Recommended)

docker build -t api-probe .
docker run --rm api-probe --help

Local Development

./run.sh examples/passing/simple.yaml

See GETTING_STARTED.md for details.

Status

Version: 2.5.0
Status: Production Ready

Version History

• v2.7.0

  • Added staged groups (stages:) — stages run in parallel, probes within each stage run sequentially
  • Each stage has an isolated variable scope (output does not leak between stages)
  • Retry and output capture warnings now printed atomically with their probe (no interleaving)
  • Output capture failure leaves variable unset — downstream probes are skipped rather than running with "None"
  • All-skipped runs now treated as failure for CI safety
  • Final report shows per-run probe counts in summary
  • Final report shows correct validation state (Passed, Passed (no validation), Passed (validation skipped))
  • Fixed in-group live output — removed duplicate probe name from result line

• v2.6.0

  • Added verify field for per-probe SSL/TLS certificate validation skip
  • Added !int, !bool, !float, !str type coercion tags for typed variable substitution
  • Unified output path convention — bare paths for body fields, headers. prefix for response headers
  • Fixed session isolation — fresh HTTP session per thread (no connection pool sharing)
  • Fixed debug output through print lock (no interleaving in concurrent executions)
  • Added response time display for every probe in report output
  • Added duplicate probe name detection (warning on validate)
  • Added retry config validation (max_attempts >= 1, delay >= 0)
  • Fixed skipped probe symbol consistency — ⊗ used in both live progress and final report

• v2.5.0

  • Added success reporting for probe execution
  • Executing probes in separate threads per execution block
  • Fixed the documentation in DOCKER.md

• v2.4.0

  • Added expression evaluation in output field (len, has, empty functions)
  • Added expression evaluation in ignore field (len, has, empty functions)
  • Added ignore field in validation headers and body sections
  • Expressions support operators: ==, !=, >, <, >=, <=, &&, ||, !
  • Expressions can access response data: status, body., headers.

• v2.3.0

  • Added ignore field for probes and groups (conditional execution)
  • Added name field for groups (with auto-generation)
  • Improved parallel group progress reporting with names

• v2.2.0

  • Added timeout field for request timeouts
  • Added retry configuration for automatic retries
  • Added debug flag for request/response logging
  • Added response_time validation for performance checks
  • Added status pattern matching (2xx, 3xx, 4xx, 5xx)
  • Added progress reporting to stderr
  • Fixed length validator for root-level arrays ($)

• v2.1.0

  • Added delay field for probes
  • Added length validator for arrays and strings

• v2.0.0

  • Added executions block
  • Added variable substitution in validation
  • Added parallel groups
  • Added XML/SOAP support
  • Added !include directive
  • Enhanced variable resolution

• v1.0.0

  • Initial release

License

See LICENSE