Skip to content

CHANGELOG.md

Latest commit

 

History

History
148 lines (127 loc) · 9.82 KB

CHANGELOG.md

File metadata and controls

148 lines (127 loc) · 9.82 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog.

Types of changes:

  • Added: for new features.
  • Changed: for changes in existing functionality.
  • Deprecated: for soon-to-be removed features.
  • Removed: for now removed features.
  • Fixed: for any bug fixes.
  • Security: in case of vulnerabilities.

Unreleased

0.2.0 - 2026-10-17

Security

  • Removed the unused NLTK dependency and bumped cryptography, Pillow, pip, setuptools and PyTorch to patched versions (#61).
  • Bumped anyio to fix CVE-2026-63374 (#83) and fixed CodeQL alerts (#103).
  • Bumped pyjwt, urllib3, litellm and langgraph-sdk, in Smith and the example agents (#88, #89, #90, #91, #99, #101).

Added

  • Security-Grounded Guidance Analysis: a four-step workflow (architecture, questionnaire, threat model, enforcement mapping) that derives security guidelines from the OWASP Agentic Top 10 (#44). Phases are resumable with validated handoffs, and guidance reconciliation and merge are gated (#75, #76).
  • Incremental test generation (test_generation --mode update): regenerates only the cases whose guidance changed, tracked in a guidance → case map (#71).
  • Parallel generation and translation: GENERATION_CONCURRENCY (#86) and TRANSLATION_CONCURRENCY, plus online-model support for the target agent (#74).
  • Bypass test cases (bypass_case_generation): adversarial cases for gaps between the policy and the guidance (#27).
  • Promptfoo: policy-plugin red-teaming, generate_promptfoo_config, and LLM tool classification of its cases (#15, #39). ATTACK_TOOLS selects which red-teaming tools run (#15).
  • CPEX translation (cpex_translate) (#39).
  • Tools: reset_policy (#71), save_snapshot, the Guidance Classifier UI, the Policy Explorer bridge and SESSION_CONFIG_FILE (#39).
  • Examples: employee hub agent (#34) and HR agent (#39).
  • Tests: unit and integration suites per pipeline stage, via make unit and make integration (#45, #92).
  • Docs: Hugo documentation site (#15) and CONTRIBUTING.md (#63).

Changed

  • Policy input arguments is now args in the guides, for CPEX compatibility (#45).
  • Default agent port in Promptfoo configs is now 9000 (was 8000) (#45).
  • Removed the top-k setting from LLM calls, since AWS models accept only temperature or top-k (#45).
  • Renamed RITS_* variables to INFERENCE_*; OLLAMA_BASE_URL is now only for Promptfoo (#15).
  • ARES and Promptfoo are optional (#15).
  • Cross-validation checks only arguments and subject fields, adds a remove verdict, and discards failed adversarial probes (#15, #27).
  • Cluster indexing is sequential; the noise group is last instead of -1 (#15).
  • Promptfoo cases are written to references/test_cases/disallow/ (#15).
  • test_case_translation skips already-translated cases (#27).
  • bypass_case_generation clears earlier bypass cases first (#71).
  • make ci is now lint + license-check + unit; make lint-policy is no longer in the gate (#45).
  • Security-analysis guides are leaner: one shared worker contract and compact artifacts (#59, #75, #76).
  • README points to the HR agent; example outputs and .env_template defaults refreshed (#92).
  • Raised minimum dependency versions: mcp, networkx, sentence-transformers, pip-audit, build and the CI setup actions (#61); openai (>=3.26.0, major bump), numpy, tqdm, black, ruff (#107, #108, #109, #110, #111); and python-dotenv, pydantic, typer, twine, pytest-socket, matplotlib, pandas, requests, pydot (#64–#68, #94–#98).

Fixed

  • apply_cross_validate no longer overwrites an existing case on a cv_ name collision (#71).
  • ARES: reads JSON Lines output, pins the Qwen connector to device: cpu (MPS segfault), and passes guidance through to attack cases (#71); cases inherit the parent's confidence in the report (#27).
  • A null numeric system variable no longer aborts test_generation (#45).
  • generate_promptfoo_config no longer appends duplicate tool-parameter blocks (#45).
  • Fixed AST graph indices carrying over between calls, and added missing .env_template variables (#45).
  • Tier-3 label validation continues past a single LLM error (#15).
  • The OPA scorecard excludes failed requests instead of scoring them as deny (#15).
  • An invalid ATTACK_TOOLS value now errors instead of silently disabling red-teaming (#15).
  • Missing attack files no longer crash the readers (#27).
  • Test-case prompts include target tool parameters (#15).
  • Security-analysis phases require a resolved-path envelope and reject rules on data OPA cannot see (#76).

0.1.1 - 2026-06-29

  • Repackaged scripts/ into an installable smith Python package using a src/ layout, with pyproject.toml at the repo root declaring runtime dependencies ([project.dependencies]) and a [dev] extra. The CLI entry point is now smith = smith.cli:main.
  • Package management and the build/publish workflow now use uv (make install, make package, make publish).
  • The OPA scorecard harness ships inside the package (smith.policy_testing) and writes all generated outputs to a BASE_URL-relative dir (references/scorecard/, via TEST_OUTPUT_DIR) instead of scripts/tests/integration/.
  • Renamed mcp_servers/ to examples/.

Removed

  • Legacy code unreachable from the CLI: a kubectl/mcpgateway/beeai cluster, duplicate entry points, a dead visualization/ package, and the previous (non-functional) pytest suite. Also removed stray upstream ARES repository scaffolding; ARES is the external ares-redteamer tool, located via ARES_HOME.

0.1.0 - 2026-06-28

Added

  • Initial release of Smith — an agent skill (plugin) for AI code agents that automates the full lifecycle of Open Policy Agent (OPA) (Rego) access-control policies for AI/MCP agents.
  • Two-layer architecture: a skill layer (SKILL.md plus authoring guides under opa_policy/ and test_generation/) that the agent follows, and a smith CLI backend (scripts/cli.py) that runs the heavy pipeline stages via smith --flag <stage>.
  • Policy creation from natural-language guidance and an agent/MCP tool description, restricted to context available from tool arguments (input.arguments.*) and system variables (input.extensions.subject.*).
  • Test generation producing both legitimate and adversarial cases: guidance decomposition, grey-condition and variable extraction, case generation, and red-teaming via ARES and Promptfoo (test_generation).
  • Test-case evaluation with three-tier label validation (rule patterns → semantic embeddings/NLI → LLM judge), guidance classification, and an HTML report (test_case_evaluation).
  • Policy testing harness that runs every generated and custom case against a running OPA server and emits a scorecard with false-positive/false-negative breakdowns (policy_testing).
  • Iterative refinement: DBSCAN clustering of failed cases (red_suggestion), Regal lint/format suggestions (regal_suggestion), and graph + LLM redundancy detection (duplication_suggestion).
  • Cross-validation of failed cases to distinguish mislabeled tests from policy bugs (cross_validate, apply_cross_validate).
  • MCP tool extraction over SSE and stdio transports (get_mcp_parameter) and tool-call translation (test_case_translation).
  • Rego policy validation with optional auto-fix (policy_validation, policy_validation_fix).
  • Runtime configuration driven entirely from .env (see .env_template); target-agent selection via TARGET_AGENT_PATH, GUIDANCE_FILE, SYSTEM_VAR_FILE, MCP_*, and AGENT_URL.
  • Example target agents under examples/, each carrying its Smith inputs (guidance.txt, tool_definitions.json, system_vars.json).