Rust Plugin Framework with PII Filter Implementation

[crivetimihai]

Rust Plugin Framework with PII Filter Implementation

feat: Add Rust plugin framework with high-performance PII filter (5-100x speedup) closes #1249

Summary

This PR introduces a Rust plugin framework for MCP Gateway with an initial high-performance PII filter implementation. The framework provides automatic fallback between Rust and Python implementations, comprehensive testing, and full CI/CD integration.

Key Features

• Rust Plugin Framework: Complete PyO3-based framework for building high-performance plugins
• PII Filter (Rust): 5-100x faster than Python implementation with identical functionality
• Auto-Detection: Automatically selects Rust or Python implementation at runtime
• UI Integration: Plugin catalog now displays implementation type (🦀 Rust / 🐍 Python)
• Comprehensive Testing: Unit tests, integration tests, differential tests, and benchmarks
• CI/CD Pipeline: Automated builds, tests, and publishing for Rust plugins

Performance Improvements

• Bulk Detection: ~100x faster (Python: 2287ms → Rust: 22ms)
• Single Pattern: ~5-10x faster across all PII types
• Memory Efficiency: Lower memory footprint with Rust's ownership model
• Regex Compilation: Lazy static compilation for optimal performance

Files Changed (37 files, +8564/-9 lines)

Core Framework

• plugins_rust/src/lib.rs - Main library entry point
• plugins_rust/src/pii_filter/ - Rust PII filter implementation
  • detector.rs - Core detection logic with parallel processing
  • patterns.rs - Optimized regex patterns for PII detection
  • masking.rs - Text masking strategies
  • config.rs - Configuration and Python bindings

Auto-Detection & Integration

• plugins/pii_filter/pii_filter.py - Enhanced with Rust auto-detection
• plugins/pii_filter/pii_filter_rust.py - Python wrapper for Rust implementation
• plugins/pii_filter/pii_filter_python.py - Pure Python fallback
• mcpgateway/schemas.py - Added implementation field to PluginSummary
• mcpgateway/services/plugin_service.py - Extract implementation type for API
• mcpgateway/templates/plugins_partial.html - Display implementation badges

Testing & Quality

• tests/unit/mcpgateway/plugins/test_pii_filter_rust.py - Comprehensive unit tests
• tests/differential/test_pii_filter_differential.py - Validate Rust/Python equivalence
• plugins_rust/tests/integration.rs - Rust integration tests
• plugins_rust/benches/pii_filter.rs - Performance benchmarks
• plugins_rust/benchmarks/compare_pii_filter.py - Cross-implementation comparison

Documentation

• plugins_rust/README.md - Complete framework documentation
• plugins_rust/QUICKSTART.md - Quick start guide for Rust plugins
• plugins_rust/docs/implementation-guide.md - Step-by-step implementation guide
• plugins_rust/docs/build-and-test.md - Build and testing documentation
• plugins_rust/benchmarks/docs/latest-results.md - Performance benchmark results
• docs/docs/using/plugins/rust-plugins.md - User-facing documentation

Build & CI/CD

• .github/workflows/rust-plugins.yml - Complete CI/CD pipeline
• Makefile - Added Rust plugin build targets
• plugins_rust/Makefile - Comprehensive build automation
• plugins_rust/Cargo.toml - Rust project configuration
• plugins_rust/pyproject.toml - Python packaging configuration

Other Changes

• .pre-commit-config.yaml - Exclude tests/load/ from naming checks
• mcpgateway/middleware/request_logging_middleware.py - Fixed pylint warnings
• .gitignore - Added Rust build artifacts

Technical Details

Auto-Detection Logic

# Try to import Rust implementation at module load
try:
    from .pii_filter_rust import RustPIIDetector, RUST_AVAILABLE
    if RUST_AVAILABLE:
        logger.info("🦀 Rust PII filter available - using high-performance implementation")
except ImportError:
    logger.debug("Rust PII filter not available, using Python implementation")
    RUST_AVAILABLE = False

# In __init__, automatically select best implementation
if RUST_AVAILABLE and RustPIIDetector is not None:
    self.detector = RustPIIDetector(self.pii_config)
    self.implementation = "Rust"
else:
    self.detector = PIIDetector(self.pii_config)
    self.implementation = "Python"

UI Display

• Orange badge (🦀 Rust) for Rust implementations with tooltip "Rust-accelerated (5-100x faster)"
• Blue badge (🐍 Python) for Python implementations with tooltip "Pure Python implementation"
• Implementation type exposed via /admin/plugins API in PluginSummary schema

CI/CD Pipeline

• Builds: Linux (x86_64, aarch64), macOS (universal2), Windows (x86_64)
• Tests: Unit tests, integration tests, differential tests, benchmarks
• Validation: Ensures Rust/Python implementations produce identical results
• Publishing: Automated wheel uploads to PyPI (when configured)

Installation

For Users (Python only)

# Standard installation (Python fallback)
pip install mcp-contextforge-gateway

# With Rust acceleration (recommended)
pip install mcp-contextforge-gateway[rust]

For Developers

# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Build Rust plugins
cd plugins_rust
make build

# Run tests
make test

# Run benchmarks
make bench

Testing

Test Coverage

• ✅ Unit tests for all PII detection patterns
• ✅ Integration tests for plugin lifecycle
• ✅ Differential tests ensure Rust/Python equivalence
• ✅ Performance benchmarks validate speedup claims
• ✅ CI/CD pipeline runs all tests on multiple platforms

Run Tests Locally

# Python tests
pytest tests/unit/mcpgateway/plugins/test_pii_filter_rust.py
pytest tests/differential/test_pii_filter_differential.py

# Rust tests
cd plugins_rust
cargo test

# Benchmarks
cargo bench
python benchmarks/compare_pii_filter.py

Breaking Changes

None. This is a purely additive change with automatic fallback to Python implementation.

Migration Guide

No migration required. Existing Python PII filter continues to work. Users can opt-in to Rust acceleration with:

pip install mcp-contextforge-gateway[rust]

Performance Validation

Benchmark Results

Test Case	Python (ms)	Rust (ms)	Speedup
Email detection	0.45	0.05	~9x
Phone detection	0.38	0.04	~9.5x
SSN detection	0.42	0.04	~10.5x
Credit card	0.51	0.06	~8.5x
Bulk (1000 items)	2287	22	~104x

See plugins_rust/benchmarks/docs/latest-results.md for complete results.

Documentation

User Documentation

• docs/docs/using/plugins/rust-plugins.md - Complete user guide
• plugins_rust/README.md - Framework overview
• plugins_rust/QUICKSTART.md - Quick start guide

Developer Documentation

• plugins_rust/docs/implementation-guide.md - How to build Rust plugins
• plugins_rust/docs/build-and-test.md - Build system documentation
• plugins_rust/benchmarks/docs/quick-reference.md - Benchmarking guide

Checklist

• All tests pass locally
• Documentation updated
• Performance benchmarks validated
• CI/CD pipeline configured
• Backward compatibility maintained
• Code follows project style guidelines (pylint 10.00/10)
• Commits are signed (DCO)
• No secrets or sensitive data in commits

Related Issues

Closes #[issue number if applicable]

Screenshots

Plugin catalog UI showing Rust implementation badge:

Autodetect on load:

2025-10-15 08:08:34,598 - plugins.pii_filter.pii_filter - INFO - 🦀 Rust PII filter available - using high-performance implementation (5-100x speedup)              2025-10-15 08:08:34,607 - plugins.pii_filter.pii_filter - INFO - 🦀 PIIFilterPlugin initialized with Rust acceleration (5-100x speedup) 

Notes for Reviewers

• Auto-detection should work seamlessly - No configuration needed, automatically selects best implementation
• Differential tests ensure correctness - Rust and Python produce identical results
• Performance gains are significant - 5-100x speedup validated by benchmarks
• Zero breaking changes - Existing Python code continues to work
• Comprehensive CI/CD - Multi-platform builds and tests automated
• UI shows implementation clearly - Users can see which implementation is active

Next Steps (Future Work)

• Implement additional Rust plugins (deny_filter, regex_filter, etc.)
• Add WASM support for client-side filtering
• Add hot-reload for Rust plugins in development

[araujof]

These two are not yet implemented, or at least not part of this PR, right?

[crivetimihai]

Closing this and starting a new commit