AI Conformance End-To-End (E2E) Tests

A comprehensive Kubernetes conformance testing framework that extends the standard Kubernetes e2e tests with AI-specific conformance tests. This project provides tools and test suites to validate AI workloads and infrastructure on Kubernetes clusters, supporting the CNCF AI Conformance self-certification process.

Overview

This project builds upon the Kubernetes e2e testing framework (v1.34.0) and adds AI-specific conformance tests to help organizations validate their Kubernetes clusters for AI workloads. The tests are designed to answer the CNCF AI Conformance self-certification questionnaire template.

Key Features

• Extended Kubernetes e2e Suite: Based on Kubernetes v1.34.0 e2e tests with AI conformance extensions
• Multiple Test Runners: Support for Ginkgo, Sonobuoy, and Hydrophone test runners
• Self-Certification Support: Generates results compatible with CNCF AI Conformance requirements
• Containerized Testing: Docker-based test execution with pre-built images
• Flexible Configuration: Configurable test focus, skip patterns, and execution parameters

Test Flags added by this project

Go test flags
  -ai.operator.chart string
    	chart name where to locate the requested chart
  -ai.operator.filename string
    	filename, directory, or URL to files to use to install the operator
  -ai.operator.namespace string
    	namespace scope for this request. If unspecified, a random namespace will be used
  -ai.operator.releaseName string
    	release name to create with this request. If unspecified, a random release name will be used
  -ai.operator.repo string
    	chart repository url where to locate the requested chart

Quick Start

Using Hydrophone

Run the AI conformance tests using Hydrophone:

hydrophone --conformance-image ghcr.io/carlory/ai-conformance:latest --focus="\[AIConformance\]" --skip="\[Disruptive\]|NoExecuteTaintManager" --output-dir ./results

Using Sonobuoy

Run the AI conformance tests using Sonobuoy:

sonobuoy run --plugin https://raw.githubusercontent.com/carlory/ai-conformance/master/sonobuoy-plugin.yaml

Check the status:

sonobuoy status

Retrieve results:

sonobuoy retrieve ./results

Using Make Commands

Run e2e tests with Ginkgo:

make test-e2e

Run tests with Sonobuoy:

# Generate the Sonobuoy plugin yaml file
make generate-plugin
# Run tests with Sonobuoy
make test-sonobuoy

Run tests with Hydrophone:

make test-hydrophone

Results and Reporting

Sonobuoy Results

(base) ➜  /tmp tree results
results
├── 202509260852_sonobuoy_3984fc5c-e24d-42c8-8c05-01545d84e6f8.tar.gz  ## generated by sonobuoy, including the following same files in the `plugins/ai-conformance/results/global` directory
├── e2e.log          ## generated by hydrophone
└── junit_01.xml     ## generated by hydrophone

1 directory, 3 files

Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests following the existing patterns
4. Ensure tests pass with make test-e2e
5. Submit a pull request

Related Projects

• Kubernetes - The base e2e testing framework
• Sonobuoy - Kubernetes conformance testing tool
• Hydrophone - Kubernetes e2e test runner
• CNCF AI Conformance - AI conformance specification