[WIP] mcp server for federated testing workflow

mcp_servers/integration_test_generator/README.md

@@ -0,0 +1,240 @@
+# Integration Test Generator MCP Server
+
+This MCP server helps generate OTel integration metric test files similar to `test_postgres_metrics.py` but for different integrations (Redis, MySQL, Kafka, Nginx, etc.).
+
+## Features
+
+- **Generate complete test files** with all three test classes:
+  - `Test_<Integration>MetricsCollection` - validates metrics received by collector
+  - `Test_BackendValidity` - validates metrics received by backend
+  - `Test_Smoke` - generates integration-specific activity and validates basic metrics
+
+- **Metric-based smoke test generation** (NEW!):
+  - Analyzes the metrics JSON file and generates specific operations for EACH metric
+  - Automatically skips metrics requiring replicas/multiple instances with explanatory comments
+  - Follows the detailed instructions from `prompt_template.py`
+  - Built-in generators for: Kafka, Redis, MySQL, PostgreSQL
+
+- **Uses shared utilities**:
+  - All tests use the shared `utils/otel_metrics_validator.py`
+
+- **Generates supporting files**:
+  - `__init__.py`
+  - Template for metrics JSON file
+
+## Installation
+
+### 1. Install MCP SDK
+
+```bash
+pip install mcp
+```
+
+### 2. Configure in Cursor/Claude Desktop
+
+Add to your MCP configuration file:
+
+**For Cursor** (`~/.cursor/mcp.json`):
+```json
+{
+  "mcpServers": {
+    "integration-test-generator": {
+      "command": "python3",
+      "args": [
+        "<PATH>/system-tests/mcp_servers/integration_test_generator/server.py"
+      ]
+    }
+  }
+}
+```
+
+**For Claude Desktop** (`~/Library/Application Support/Claude/claude_desktop_config.json`):
+```json
+{
+  "mcpServers": {
+    "integration-test-generator": {
+      "command": "python3",
+      "args": [
+        "<PATH>/system-tests/mcp_servers/integration_test_generator/server.py"
+      ]
+    }
+  }
+}
+```
+
+### 3. Restart Cursor/Claude Desktop
+
+## Usage
+
+### Basic Usage
+
+In Cursor or Claude Desktop, you can now use natural language to generate tests, i.e.:
+
+```
+Create a MySQL integration test, excluding the metrics: mysql.slow_queries, mysql.replication.delay
+```
+
+### Available Tools
+
+#### 1. `generate_integration_test`
+
+Generates a complete test file structure for an integration.
+
+**Parameters:**
+- `integration_name` (required): Name of the integration (e.g., "redis", "mysql")
+- `metrics_json_file` (required): Name of the metrics JSON file (e.g., "redis_metrics.json")
+- `sample_metrics` (optional): List of metric names to include in the metrics JSON template
+- `excluded_metrics` (optional): List of metrics to exclude
+- `feature_name` (optional): Feature name for decorator (defaults to `<integration>_receiver_metrics`)
+
+**Example:**
+```
+Generate an integration test for Redis with metrics file redis_metrics.json, sample metrics redis.commands.processed redis.keys.expired, and exclude redis.cluster.slots
+```
+
+**Note:** If you provide `sample_metrics`, the tool will generate a metrics JSON template with those metrics included. Otherwise, it will generate an empty template structure.
+
+#### 2. `list_supported_integrations`
+
+Lists all integrations with pre-configured smoke tests.
+
+**Example:**
+```
+What integrations are supported?
+```
+
+#### 3. `generate_metrics_json_template`
+
+Creates a template metrics JSON file structure.
+
+**Parameters:**
+- `integration_name` (required): Name of the integration
+- `sample_metrics` (required): List of sample metric names
+
+**Example:**
+```
+Generate a metrics JSON template for Redis with metrics: redis.commands.processed, redis.net.input, redis.keys.expired
+```
+
+## Workflow
+
+### Step 1: Generate the Test Files
+
+```
+Ex: Generate a MySQL integration test with metrics file mysql_metrics.json
+```
+
+The MCP server will provide:
+1. `test_mysql_metrics.py` - The main test file
+2. `__init__.py` - Package init file
+3. Directory structure instructions
+
+### Step 2: Create the Directory
+
+```bash
+mkdir -p tests/otel_redis_metrics_e2e
+```
+
+### Step 3: Create the Metrics JSON
+
+Create `tests/otel_<integration>_metrics_e2e/<integration>_metrics.json`:
+
+```json
+{
+  "postgresql.backends": {
+    "data_type": "Sum",
+    "description": "The number of backends."
+  },
+  "postgresql.bgwriter.buffers.allocated": {
+    "data_type": "Sum",
+    "description": "Number of buffers allocated."
+  },
+}
+```
+
+
+### Step 4: Customize the Smoke Test (Usually Not Needed!)
+
+The smoke test operations are **automatically generated** from your metrics JSON file! Each metric gets specific commands to generate it.
+
+For example, for Kafka:
+- `kafka.brokers` → `kafka-broker-api-versions` command
+- `kafka.messages` → Produce messages to topic
+- `kafka.partition.replicas` → Automatically skipped with comment (needs multiple brokers)
+
+**Only customize if:**
+- You're using an integration not yet supported (add to `metric_operations_generator.py`)
+- The auto-generated operations don't work for your specific setup
+
+```python
+def setup_main(self) -> None:
+    """When the container spins up, we need some activity."""
+    scenario: OtelCollectorScenario = context.scenario
+    container = scenario.redis_container
+
+    # Auto-generated operations - one for each metric!
+    r = container.exec_run("redis-cli SET test_key test_value")
+    logger.info(f"redis.net.output: {r.output}")
+    # ... more auto-generated operations
+```
+
+### Step 5: Add Feature to utils/_features.py
+
+If the feature doesn't exist, add it by following this [doc](https://github.com/DataDog/system-tests/blob/main/docs/edit/features.md).
+>Each new feature should be defined in _features.py. This consists of adding a feature in Feature Parity Dashboard, get the feature id and copying one of the already added features, changing the name and the feature id in the url, and the feature number.
+
+### Step 6: Format and Test
+
+```bash
+./format.sh
+./run.sh otel_collector  # or appropriate scenario
+```
+
+
+## Example Outputs
+
+### For MySQL
+
+The generator will create:
+- MySQL-specific database operations (CREATE DATABASE, CREATE TABLE, INSERT, SELECT)
+- Expected metrics for MySQL operations
+- Proper container reference (`mysql_container`)
+
+## Troubleshooting
+
+### MCP Server Not Showing Up
+
+1. Check the configuration file path is correct
+2. Ensure the Python path in configuration matches your system
+3. Restart Cursor/Claude Desktop after configuration changes
+4. Check logs: 
+   - Cursor: Developer Tools → Console
+   - Claude Desktop: Console logs
+
+### Import Errors
+
+Ensure MCP SDK is installed:
+```bash
+pip install mcp
+```
+
+### Permission Issues
+
+Make the server executable:
+```bash
+chmod +x /Users/quinna.halim/system-tests/mcp_servers/integration_test_generator/server.py
+```
+
+## Next Steps
+
+1. **Extend INTEGRATION_CONFIGS**: Add more pre-configured integrations
+2. **Custom Templates**: Create specialized templates for different test patterns
+3. **Metrics Discovery**: Add tools to discover metrics from OTel Collector configuration
+4. **Validation**: Add tools to validate generated tests against existing patterns
+
+## References
+
+- [MCP Documentation](https://modelcontextprotocol.io/)
+- [System Tests Documentation](../../docs/)
+- [OTel Metrics Testing Guide](../../docs/scenarios/otel_collector.md)
+

mcp_servers/integration_test_generator/constants.py

@@ -0,0 +1,71 @@
+from pathlib import Path
+
+METRIC_TYPES = {"sum", "gauge"}
+GH_BASE_API = "https://api.github.com/repos/open-telemetry/opentelemetry-collector-contrib"
+
+# Path to reference test files
+SYSTEM_TESTS_ROOT = Path(__file__).parent.parent.parent
+POSTGRES_TEST_PATH = SYSTEM_TESTS_ROOT / "tests/otel_postgres_metrics_e2e/test_postgres_metrics.py"
+MYSQL_TEST_PATH = SYSTEM_TESTS_ROOT / "tests/otel_mysql_metrics_e2e/test_otel_mysql_metrics.py"
+
+
+# Integration-specific configurations
+INTEGRATION_CONFIGS = {
+    "redis": {
+        "container_name": "redis_container",
+        "smoke_test_operations": [
+            'r = container.exec_run("redis-cli SET test_key test_value")',
+            "logger.info(r.output)",
+            'r = container.exec_run("redis-cli GET test_key")',
+            "logger.info(r.output)",
+            'r = container.exec_run("redis-cli INCR counter")',
+            "logger.info(r.output)",
+        ],
+        "expected_smoke_metrics": [
+            "redis.commands.processed",
+            "redis.keys.expired",
+            "redis.net.input",
+            "redis.net.output",
+        ],
+    },
+    "mysql": {
+        "container_name": "mysql_container",
+        "smoke_test_operations": [
+            "r = container.exec_run(\"mysql -u root -ppassword -e 'CREATE DATABASE IF NOT EXISTS test_db;'\")",
+            "r = container.exec_run(\"mysql -u root -ppassword test_db -e 'CREATE TABLE IF NOT EXISTS test_table (id INT PRIMARY KEY);'\")",
+            "r = container.exec_run(\"mysql -u root -ppassword test_db -e 'INSERT INTO test_table VALUES (1);'\")",
+            "logger.info(r.output)",
+            "r = container.exec_run(\"mysql -u root -ppassword test_db -e 'SELECT * FROM test_table;'\")",
+            "logger.info(r.output)",
+        ],
+        "expected_smoke_metrics": [
+            "mysql.operations",
+            "mysql.client.network.io",
+            "mysql.commands",
+        ],
+    },
+    "nginx": {
+        "container_name": "nginx_container",
+        "smoke_test_operations": [
+            'r = container.exec_run("curl -s http://localhost/status")',
+            "logger.info(r.output)",
+        ],
+        "expected_smoke_metrics": [
+            "nginx.requests",
+            "nginx.connections_accepted",
+            "nginx.connections_handled",
+        ],
+    },
+    "kafka": {
+        "container_name": "kafka_container",
+        "smoke_test_operations": [
+            'r = container.exec_run("kafka-topics --create --topic test-topic --bootstrap-server localhost:9092")',
+            "logger.info(r.output)",
+            'r = container.exec_run("kafka-console-producer --topic test-topic --bootstrap-server localhost:9092", stdin="test message")',
+        ],
+        "expected_smoke_metrics": [
+            "kafka.messages",
+            "kafka.brokers",
+        ],
+    },
+}

mcp_servers/integration_test_generator/formatters.py

@@ -0,0 +1,48 @@
+"""Helper functions for formatting template data."""
+
+
+def format_excluded_metrics(integration_name: str, excluded_metrics: list[str] | None) -> str:
+    """Format excluded metrics section.
+
+    Args:
+        integration_name: Name of the integration
+        excluded_metrics: List of metrics to exclude (optional)
+
+    Returns:
+        Formatted excluded metrics string
+    """
+    if excluded_metrics:
+        excluded_metrics_formatted = ",\n    ".join([f'"{m}"' for m in excluded_metrics])
+        return f"""
+# Exclude metrics that require specific setup or sustained activity
+_EXCLUDED_{integration_name.upper()}_METRICS = {{
+    {excluded_metrics_formatted}
+}}
+"""
+    else:
+        return f"_EXCLUDED_{integration_name.upper()}_METRICS: set[str] = set()"
+
+
+def format_smoke_operations(operations: list[str]) -> str:
+    """Format smoke test operations.
+
+    Args:
+        operations: List of operation strings
+
+    Returns:
+        Formatted operations string
+    """
+    return "\n        ".join(operations)
+
+
+def format_expected_metrics(metrics: list[str]) -> str:
+    """Format expected metrics list.
+
+    Args:
+        metrics: List of metric names
+
+    Returns:
+        Formatted metrics string
+    """
+    return ",\n            ".join([f'"{m}"' for m in metrics])
+