docs: adding multi turn conversation example and docs #5832
Conversation
jameshiester
commented
Oct 6, 2025
jameshiester
commented
- Adding example for multiple turn red team with custom python provider
|
⏩ No test execution environment matched (dd96e3d) View output ↗ Tip New to Tusk? Learn more here. View check history
|
jameshiester
changed the title
📝 WalkthroughWalkthroughThe changes update red-team documentation and navigation. The configuration guide clarifies the distinction between targets and the provider and adjusts the configuration fields table by splitting the “provider or targets” entry into a dedicated “provider” field. A new page documents multi-turn conversations with a custom provider, covering session ID handling, configuration snippets, and a sample API call pattern. The sidebar is updated to include the new “Multi-Turn Conversations” guide under Red Team Guides. Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
site/docs/red-team/multi-turn-conversations.md (1)
3-3: Optional: Improve clarity and consistency.Consider these minor improvements:
- Line 3: The sentence is quite long and mentions "GOAT and/or Crescendo" without context. Consider briefly explaining these terms or linking to their documentation.
- Line 5: Use "Python" (capitalized) for consistency with standard conventions.
Example improvements:
-When using HTTP or custom providers with multi-turn attacks like GOAT and/or Crescendo, you may need to maintain session IDs between rounds. You can refer to the [session management guide](/docs/providers/http/#session-management) for more details on how session management can be configured for different providers. In the example below, we use client-side session management to create a unique session or conversation ID for each test, and add it in the for each turn in the conversation. +When using HTTP or custom providers with multi-turn attack strategies (such as GOAT or Crescendo), you need to maintain session IDs between conversation turns. This guide demonstrates how to use client-side session management to create a unique session ID for each test and pass it through each turn. + +For additional details on session management configuration options, refer to the [HTTP provider session management guide](/docs/providers/http/#session-management). -1. Configure your target using a local python script: +1. Configure your target using a local Python script:Also applies to: 5-5
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
site/docs/red-team/configuration.md(2 hunks)site/docs/red-team/multi-turn-conversations.md(1 hunks)site/sidebars.js(1 hunks)
🧰 Additional context used
📓 Path-based instructions (5)
{site/**,examples/**}
📄 CodeRabbit inference engine (.cursor/rules/gh-cli-workflow.mdc)
Any pull request that only touches files in 'site/' or 'examples/' directories must use the 'docs:' prefix in the PR title, not 'feat:' or 'fix:'
Files:
site/sidebars.jssite/docs/red-team/multi-turn-conversations.mdsite/docs/red-team/configuration.md
site/**
📄 CodeRabbit inference engine (.cursor/rules/gh-cli-workflow.mdc)
If the change is a feature, update the relevant documentation under 'site/'
Files:
site/sidebars.jssite/docs/red-team/multi-turn-conversations.mdsite/docs/red-team/configuration.md
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx,js,jsx}: Follow consistent import order (Biome will handle sorting)
Use curly braces for all control statements
Prefer const over let; avoid var
Use object property shorthand when possible
Use async/await for asynchronous code
Files:
site/sidebars.js
site/docs/**/*.md
📄 CodeRabbit inference engine (.cursor/rules/docusaurus.mdc)
site/docs/**/*.md: Prioritize minimal edits when updating existing documentation; avoid creating entirely new sections or rewriting substantial portions; focus edits on improving grammar, spelling, clarity, fixing typos, and structural improvements where needed; do not modify existing headings (h1, h2, h3, etc.) as they are often linked externally.
Structure content to reveal information progressively: begin with essential actions and information, then provide deeper context as necessary; organize information from most important to least important.
Use action-oriented language: clearly outline actionable steps users should take, use concise and direct language, prefer active voice over passive voice, and use imperative mood for instructions.
Use 'eval' instead of 'evaluation' in all documentation; when referring to command line usage, use 'npx promptfoo eval' rather than 'npx promptfoo evaluation'; maintain consistency with this terminology across all examples, code blocks, and explanations.
The project name can be written as either 'Promptfoo' (capitalized) or 'promptfoo' (lowercase) depending on context: use 'Promptfoo' at the beginning of sentences or in headings, and 'promptfoo' in code examples, terminal commands, or when referring to the package name; be consistent with the chosen capitalization within each document or section.
Each markdown documentation file must include required front matter fields: 'title' (the page title shown in search results and browser tabs) and 'description' (a concise summary of the page content, ideally 150-160 characters).
Only add a title attribute to code blocks that represent complete, runnable files; do not add titles to code fragments, partial examples, or snippets that aren't meant to be used as standalone files; this applies to all code blocks regardless of language.
Use special comment directives to highlight specific lines in code blocks: 'highlight-next-line' highlights the line immediately after the comment, 'highligh...
Files:
site/docs/red-team/multi-turn-conversations.mdsite/docs/red-team/configuration.md
site/docs/**/*.{md,mdx}
📄 CodeRabbit inference engine (site/docs/CLAUDE.md)
site/docs/**/*.{md,mdx}: Use the term "eval" not "evaluation" in documentation and examples
Capitalization: use "Promptfoo" (capitalized) in prose/headings and "promptfoo" (lowercase) in code, commands, and package names
Every doc must include required front matter: title and description
Only add title= to code blocks when showing complete runnable files
Admonitions must have empty lines around their content (Prettier requirement)
Do not modify headings; they may be externally linked
Use progressive disclosure: put essential information first
Use action-oriented, imperative mood in instructions (e.g., "Install the package")
Files:
site/docs/red-team/multi-turn-conversations.mdsite/docs/red-team/configuration.md
🪛 markdownlint-cli2 (0.18.1)
site/docs/red-team/multi-turn-conversations.md
27-27: Link text should be descriptive
(MD059, descriptive-link-text)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (21)
- GitHub Check: Style Check
- GitHub Check: Redteam (Production API)
- GitHub Check: Redteam (Staging API)
- GitHub Check: Share Test
- GitHub Check: Test on Node 24.x and ubuntu-latest
- GitHub Check: Test on Node 24.x and windows-latest
- GitHub Check: Test on Node 22.x and macOS-latest
- GitHub Check: Build Docs
- GitHub Check: Test on Node 22.x and windows-latest
- GitHub Check: Test on Node 20.x and windows-latest
- GitHub Check: Test on Node 20.x and macOS-latest
- GitHub Check: Test on Node 22.x and ubuntu-latest
- GitHub Check: Build on Node 20.x
- GitHub Check: Test on Node 20.x and ubuntu-latest
- GitHub Check: Build on Node 24.x
- GitHub Check: Generate Assets
- GitHub Check: Run Integration Tests
- GitHub Check: Build on Node 22.x
- GitHub Check: webui tests
- GitHub Check: Analyze (go)
- GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (2)
site/docs/red-team/configuration.md (1)
17-17: LGTM! Clear distinction between targets and provider.The expanded description at Line 17 and the dedicated provider field at Line 59 effectively clarify that targets (the system under test) and the red team provider (the model generating attacks) are distinct concepts. This removes ambiguity and improves documentation clarity.
Also applies to: 59-59
site/sidebars.js (1)
304-308: LGTM! Proper sidebar integration.The new documentation page is correctly added to the sidebar with proper structure, descriptive label, and appropriate placement within the Red Team Guides section.
| # Multi-Turn Conversations With Custom Provider | ||
|
|
There was a problem hiding this comment.
Add required front matter.
The documentation file is missing required front matter fields. As per coding guidelines, every markdown documentation file must include title and description fields.
Add this front matter at the beginning of the file:
+---
+title: Multi-Turn Conversations with Custom Providers
+description: Learn how to maintain session IDs across conversation turns when using HTTP or custom providers with multi-turn red team attacks like GOAT and Crescendo.
+---
+
# Multi-Turn Conversations With Custom Provider📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # Multi-Turn Conversations With Custom Provider | |
| --- | |
| title: Multi-Turn Conversations with Custom Providers | |
| description: Learn how to maintain session IDs across conversation turns when using HTTP or custom providers with multi-turn red team attacks like GOAT and Crescendo. | |
| --- | |
| # Multi-Turn Conversations With Custom Provider |
🤖 Prompt for AI Agents
In site/docs/red-team/multi-turn-conversations.md at lines 1-2, the markdown
file lacks required front matter; add YAML front matter at the very top
containing at minimum title and description fields (e.g., a top block delimited
by --- with title: "Multi-Turn Conversations With Custom Provider" and a brief
description string) so the documentation build and guidelines are satisfied.
| ```yaml | ||
| def call_api(prompt, options, context): | ||
| session_id = context.get("vars", {}).get("sessionId", "") | ||
| ``` |
There was a problem hiding this comment.
Correct the code block language identifier.
The code block is marked as yaml but contains Python code. This will result in incorrect syntax highlighting.
Apply this diff to fix the language identifier:
-```yaml
+```python
def call_api(prompt, options, context):
session_id = context.get("vars", {}).get("sessionId", "")
<details>
<summary>🤖 Prompt for AI Agents</summary>
In site/docs/red-team/multi-turn-conversations.md around lines 22 to 25, the
fenced code block is labeled as yaml but contains Python code; change the
language identifier to python by replacing the opening fence from yaml to python so the block reads python then the Python code and closing to
restore correct syntax highlighting.
</details>
<!-- This is an auto-generated comment by CodeRabbit -->
| session_id = context.get("vars", {}).get("sessionId", "") | ||
| ``` | ||
| A complete example can be found [here](https://github.com/promptfoo/promptfoo/tree/main/examples/custom-provider-multi-turn) No newline at end of file |
There was a problem hiding this comment.
Use descriptive link text.
The link text "here" is not descriptive and makes it difficult for users to understand where the link leads without reading the surrounding context. This is also flagged by the markdownlint rule MD059.
Apply this diff to make the link text more descriptive:
-A complete example can be found [here](https://github.com/promptfoo/promptfoo/tree/main/examples/custom-provider-multi-turn)
+A complete example can be found at [custom-provider-multi-turn](https://github.com/promptfoo/promptfoo/tree/main/examples/custom-provider-multi-turn).📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| A complete example can be found [here](https://github.com/promptfoo/promptfoo/tree/main/examples/custom-provider-multi-turn) | |
| A complete example can be found at [custom-provider-multi-turn](https://github.com/promptfoo/promptfoo/tree/main/examples/custom-provider-multi-turn). |
🧰 Tools
🪛 markdownlint-cli2 (0.18.1)
27-27: Link text should be descriptive
(MD059, descriptive-link-text)
🤖 Prompt for AI Agents
In site/docs/red-team/multi-turn-conversations.md around line 27, the link text
"here" is not descriptive; replace it with a meaningful label such as
"custom-provider-multi-turn example" (or "Promptfoo custom provider multi-turn
example") so the link reads like: A complete example can be found in the
Promptfoo custom-provider-multi-turn example and points to the same URL; update
the anchor text only, keeping the URL unchanged.
There was a problem hiding this comment.
Thanks for doing this. I've left detailed feedback to help make this example more clear and to follow our existing patterns. - the core pattern you're using (transformVars with context.uuid) is good, it just needs better documentation. I also think you should show both the stateless and stateful approaches so users know when to use each. Thanks @jameshiester!
| @@ -0,0 +1,68 @@ | |||
| import json | |||
There was a problem hiding this comment.
recommend using the openai python sdk here and simplifying the file. urllib and ssl are confusing.
You should also add docstrings to explain what's going on in the file!
| @@ -0,0 +1,30 @@ | |||
| description: Red Team Starter | |||
There was a problem hiding this comment.
take a look at the other examples, they all have a README.md file.
| strategies: | ||
| - id: goat | ||
| config: | ||
| stateful: true |
There was a problem hiding this comment.
you are using stateful: true here but you don't explain that the example is stateful.
There was a problem hiding this comment.
most redteam eamples are prefixed with redteam- in the folder name, consider doing that.
| | `numTests` | `number` | Default number of tests to generate per plugin | 5 | | ||
| | `plugins` | `Array<string\|object>` | Plugins to use for red team generation | `default` | | ||
| | `provider` or `targets` | `string\|ProviderOptions` | Endpoint or AI model provider for generating adversarial inputs | `openai:gpt-4.1` | | ||
| | `provider` | `string\|ProviderOptions` | Endpoint or AI model provider for generating adversarial inputs | `openai:gpt-4.1` | |
There was a problem hiding this comment.
in the red team context we prefer the term targets, recommend keeping it here
There was a problem hiding this comment.
Here the docs are declaring the properties within the Red Team configuration. Targets is not a valid field inside Red Team. The description also matches provider, not description. Other sentences in the docs call out that it is different than the root target
| @@ -0,0 +1,32 @@ | |||
| --- | |||
| sidebar_position: 10000 | |||
There was a problem hiding this comment.
If you want to keep this file, you should copy the entire example into here, add fontmatter directives, possibly screen shots. Make the example interesting enough too that people will look to read and emulate it (as opposed to one plugin and one test case).
2 participants
