Skip to content

docs(examples): add nemoguards cache configuration example #1459

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

docs(examples): add nemoguards cache configuration example #1459

wants to merge 5 commits into from

Conversation

Pouyanpi
Copy link
Collaborator

@Pouyanpi Pouyanpi commented Oct 17, 2025 •
edited
Loading

Description

This is PR 5/5 (final) in the NeMoGuards caching feature stack.

Test Plan

results:

======================================================================
MODEL CACHE CONFIGURATION
======================================================================
✅ content_safety: maxsize=10000, track_stats=True
✅ topic_control: maxsize=10000, track_stats=True
✅ jailbreak_detection: maxsize=10000, track_stats=True

======================================================================
GENERATING 1000 TEST PROMPTS
======================================================================
Generated 1000 prompts (100 unique × 10 repetitions)

======================================================================
PROCESSING PROMPTS
======================================================================
Processed 100/1000 prompts - Hits: 270, Misses: 30, Errors: 0
Processed 200/1000 prompts - Hits: 540, Misses: 60, Errors: 0
Processed 300/1000 prompts - Hits: 810, Misses: 90, Errors: 0
Processed 400/1000 prompts - Hits: 1080, Misses: 120, Errors: 0
Processed 500/1000 prompts - Hits: 1350, Misses: 150, Errors: 0
Processed 600/1000 prompts - Hits: 1620, Misses: 180, Errors: 0
Processed 700/1000 prompts - Hits: 1890, Misses: 210, Errors: 0
Processed 800/1000 prompts - Hits: 2160, Misses: 240, Errors: 0
Processed 900/1000 prompts - Hits: 2430, Misses: 270, Errors: 0
Processed 1000/1000 prompts - Hits: 2700, Misses: 300, Errors: 0

Completed processing 1000 prompts

======================================================================
CACHE STATISTICS FROM LOGS
======================================================================
content_safety       - Hits:  900, Misses:  100, Hit Rate:  90.0%
topic_safety         - Hits:  900, Misses:  100, Hit Rate:  90.0%
jailbreak            - Hits:  900, Misses:  100, Hit Rate:  90.0%

======================================================================
CACHE STATISTICS FROM CACHE OBJECTS
======================================================================
content_safety       - Size: 100/10000, Hits: 900, Misses: 100, Hit Rate: 90.00%
topic_control        - Size: 100/10000, Hits: 900, Misses: 100, Hit Rate: 90.00%
jailbreak_detection  - Size: 100/10000, Hits: 900, Misses: 100, Hit Rate: 90.00%

======================================================================
VALIDATING traces.jsonl
======================================================================
Total LLM spans in traces: 3000
Cached LLM spans (llm.cache.hit=true): 2700

content_safety       - Total: 1000, Cached:  900, Hit Rate:  90.0%
topic_safety         - Total: 1000, Cached:  900, Hit Rate:  90.0%
jailbreak            - Total: 1000, Cached:  900, Hit Rate:  90.0%

Traces file: /Users/prezakhani/git/nemoguardrails-project/nemo-guardrails-review/.traces/trace.jsonl

======================================================================
SUMMARY
======================================================================
✅ Processed 1000 prompts with input rails (content safety, topic safety, jailbreak)
✅ Total cache hits: 2700
✅ Total cache misses: 300
✅ Overall cache hit rate: 90.0%
✅ Traces validated for llm.cache.hit attribute across all safety checks

This was referenced Oct 17, 2025
Merged
Merged
@Pouyanpi Pouyanpi force-pushed the feat/cache-jailbreak-detection branch from 6a7474e to 630eafa Compare October 17, 2025 10:35
@codecov-commenter
Copy link

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@Pouyanpi Pouyanpi force-pushed the feat/cache-jailbreak-detection branch from 630eafa to 9dbb323 Compare October 17, 2025 10:40
@Pouyanpi
style(cache): replace pass with ellipsis in abstract methods
6dd82a5 
Updates abstract method implementations in CacheInterface to use
ellipsis (...) instead of pass, following modern Python conventions for
abstract method stubs.
@Pouyanpi
feat(cache): add LLM metadata caching for model and provider information
19d1950 
Extends the cache system to store and restore LLM metadata (model name
and provider name) alongside cache entries. This allows cached results
to maintain provenance information about which model and provider
generated the original response.

- Added LLMMetadataDict and LLMCacheData TypedDict definitions for type
safety
  - Extended CacheEntry to include optional llm_metadata field
  - Implemented extract_llm_metadata_for_cache() to capture model and
provider info from context
  - Implemented restore_llm_metadata_from_cache() to restore metadata
when retrieving cached results
  - Updated get_from_cache_and_restore_stats() to handle metadata
extraction and restoration
  - Added comprehensive test coverage for metadata caching functionalit
@Pouyanpi
feat(cache): add caching support for topic safety and content safety …
f8538c5 
…output checks

Extends the LLM caching system to support topic safety input checks and
content safety output checks. Both actions now cache their results along
with LLM stats and metadata to improve performance on repeated queries.

  Changes

  - Added caching support to topic_safety_check_input() with cache
hit/miss logic
  - Added caching support to content_safety_check_output() with cache
hit/miss logic
  - Both actions now extract and store LLM metadata alongside stats in
cache entries
  - Added model_caches parameter to both actions for optional cache
injection
  - Comprehensive test coverage for both new caching implementations
  - Tests verify cache hits, stats restoration, and metadata handling
@Pouyanpi
feat(cache): add caching support for jailbreak detection
f25167b 
Extends the LLM caching system to support jailbreak detection. The
jailbreak detection action now caches results with metadata, properly
tracks call information for tracing, and includes a fix to skip
unnecessary LLM initialization for jailbreak detection models in the
Rails configuration.

  Changes

  - Added caching support to jailbreak_detection_model() with cache
hit/miss logic
  - Implemented LLM call info tracking for jailbreak detection
(duration, timestamps, cache status)
  - Added processing log integration for tracing jailbreak detection
calls
  - Modified LLMRails to skip LLM initialization for jailbreak_detection
type models
  - Comprehensive test coverage including cache hits, misses, and model
initialization behavior
  - Tests verify that jailbreak detection models with cache configs are
registered correctly
@Pouyanpi
docs(examples): add nemoguards cache configuration example
27685a2
@Pouyanpi Pouyanpi force-pushed the docs/cache-nemoguards-example branch from 9495808 to 27685a2 Compare October 17, 2025 10:41
This was referenced Oct 17, 2025
Merged
Merged
@Pouyanpi Pouyanpi added this to the v0.18.0 milestone Oct 17, 2025
@Pouyanpi Pouyanpi self-assigned this Oct 17, 2025
tgasser-nv
tgasser-nv reviewed Oct 17, 2025
View reviewed changes
Copy link
Collaborator

@tgasser-nv tgasser-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need a description of how this config is different to the regular hero-workflow one (i.e. caching) and how they're configured. I can update the PR as you have so many others on your plate

examples/configs/nemoguards_cache/README.md
@@ -0,0 +1,24 @@
# NeMoGuard Safety Rails Example

This example showcases the use of NVIDIA's NeMoGuard NIMs for comprehensive AI safety including content moderation, topic control, and jailbreak detection.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add some more description about how this enables caching, and how this is different from our regular hero-workflow config (nemoguards)? I'm happy to do this as you have so many other PRs to review

@Pouyanpi Pouyanpi force-pushed the feat/cache-jailbreak-detection branch from 9dbb323 to d38eb0b Compare October 19, 2025 10:26
Base automatically changed from feat/cache-jailbreak-detection to develop October 19, 2025 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tgasser-nv tgasser-nv tgasser-nv left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@Pouyanpi Pouyanpi

Labels

None yet

Projects

None yet

Milestone

v0.18.0

Development

Successfully merging this pull request may close these issues.

3 participants

@Pouyanpi @codecov-commenter @tgasser-nv