Skip to content

feat: Add instructions parameter in response object #3741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Oct 20, 2025
Merged

feat: Add instructions parameter in response object #3741

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f176e1a
add support for instructions parameter in response object
s-akhtar-baig Oct 14, 2025
dc26e67
Recordings update from CI (suite: base)
github-actions[bot] Oct 14, 2025
41c55fa
Add unit test to check instructions with previous response instructions
s-akhtar-baig Oct 14, 2025
70bfe62
Fix formatting issues
s-akhtar-baig Oct 15, 2025
f4556f1
Resolve merge conflicts
s-akhtar-baig Oct 15, 2025
70217cc
Resolve merge conflicts
s-akhtar-baig Oct 16, 2025
f7b6d7f
Handle review comments
s-akhtar-baig Oct 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/static/deprecated-llama-stack-spec.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9024,6 +9024,10 @@
"$ref": "#/components/schemas/OpenAIResponseUsage",
"description": "(Optional) Token usage information for the response"
},
"instructions": {
Copy link
Contributor

@ashwinb ashwinb Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

openai's docs say this can be a string or an array

Copy link
Contributor Author

@s-akhtar-baig s-akhtar-baig Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @ashwinb, for reviewing! I have updated the definition using openai/types/responses as my reference. Please take a look when you get a chance.

Copy link
Collaborator

@leseb leseb Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting that this shows a String only but the Python client is definitely the source of truth!

Copy link
Contributor

@jwm4 jwm4 Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, I tried this with the Python client talking directly to OpenAI (not Llama Stack):

r3 = open_ai_client.responses.create(
    model="gpt-4o",
    input="What is the capital of France?",
    instructions=["Always answer with rhyming poetry.", "Include the word 'cat' in your answer."]
)

What I got was:

BadRequestError: Error code: 400 - {'error': {'message': "Invalid type for 'instructions': expected a string, but got an array instead.", 'type': 'invalid_request_error', 'param': 'instructions', 'code': 'invalid_type'}}

So the OpenAI client does indeed accept the list and send it off to the server but then the server rejected it because the server API expects a string only. So I guess the API reference that @leseb links to above is correct.

Copy link
Contributor Author

@s-akhtar-baig s-akhtar-baig Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jwm4, that makes sense! The link that @leseb shared brings me to the create endpoint which accepts string, but the response object can store instructions as a string or an array.

Copy link
Contributor

@jwm4 jwm4 Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So is the conclusion that this PR is fine the way it is? (i.e., that Llama Stack should really just accept a string and not a list for instructions)

Copy link
Contributor Author

@s-akhtar-baig s-akhtar-baig Oct 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With these changes, we will be conforming to the OpenAI spec:

  • create response accepts instructions as a string
  • response object can store either a string or an array
  • instructions from previous response is not carried over

But, I am not sure whether that's what we want in Llama Stack or not. If we change create response to accept both a string and an array, then we won't be conforming to the spec and we don't have use-cases (that I can think of) to correctly implement logic that will handle array data type.

Copy link
Collaborator

@franciscojavierarceo franciscojavierarceo Oct 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

In [7]: from openai import OpenAI; client = OpenAI()

In [8]: openai.__version__
Out[8]: '1.107.0'

In [9]: client.responses.create.__annotations__
Out[9]:
{'background': 'Optional[bool] | NotGiven',
 'conversation': 'Optional[response_create_params.Conversation] | NotGiven',
 'include': 'Optional[List[ResponseIncludable]] | NotGiven',
 'input': 'Union[str, ResponseInputParam] | NotGiven',
 'instructions': 'Optional[str] | NotGiven',
 'max_output_tokens': 'Optional[int] | NotGiven',
 'max_tool_calls': 'Optional[int] | NotGiven',
 'metadata': 'Optional[Metadata] | NotGiven',
 'model': 'ResponsesModel | NotGiven',
 'parallel_tool_calls': 'Optional[bool] | NotGiven',
 'previous_response_id': 'Optional[str] | NotGiven',
 'prompt': 'Optional[ResponsePromptParam] | NotGiven',
 'prompt_cache_key': 'str | NotGiven',
 'reasoning': 'Optional[Reasoning] | NotGiven',
 'safety_identifier': 'str | NotGiven',
 'service_tier': "Optional[Literal['auto', 'default', 'flex', 'scale', 'priority']] | NotGiven",
 'store': 'Optional[bool] | NotGiven',
 'stream': 'Optional[Literal[False]] | Literal[True] | NotGiven',
 'stream_options': 'Optional[response_create_params.StreamOptions] | NotGiven',
 'temperature': 'Optional[float] | NotGiven',
 'text': 'ResponseTextConfigParam | NotGiven',
 'tool_choice': 'response_create_params.ToolChoice | NotGiven',
 'tools': 'Iterable[ToolParam] | NotGiven',
 'top_logprobs': 'Optional[int] | NotGiven',
 'top_p': 'Optional[float] | NotGiven',
 'truncation': "Optional[Literal['auto', 'disabled']] | NotGiven",
 'user': 'str | NotGiven',
 'extra_headers': 'Headers | None',
 'extra_query': 'Query | None',
 'extra_body': 'Body | None',
 'timeout': 'float | httpx.Timeout | None | NotGiven',
 'return': 'Response | Stream[ResponseStreamEvent]'}

Copy link
Collaborator

@franciscojavierarceo franciscojavierarceo Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah looks like the client (as mentioned by @s-akhtar-baig) only lists instructions as a str

"type": "string",
"description": "(Optional) System message inserted into the model's context"
},
"input": {
"type": "array",
"items": {
Expand Down Expand Up @@ -9901,6 +9905,10 @@
"usage": {
"$ref": "#/components/schemas/OpenAIResponseUsage",
"description": "(Optional) Token usage information for the response"
},
"instructions": {
"type": "string",
"description": "(Optional) System message inserted into the model's context"
}
},
"additionalProperties": false,
Expand Down
8 changes: 8 additions & 0 deletions docs/static/deprecated-llama-stack-spec.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6734,6 +6734,10 @@ components:
$ref: '#/components/schemas/OpenAIResponseUsage'
description: >-
(Optional) Token usage information for the response
instructions:
type: string
description: >-
(Optional) System message inserted into the model's context
input:
type: array
items:
Expand Down Expand Up @@ -7403,6 +7407,10 @@ components:
$ref: '#/components/schemas/OpenAIResponseUsage'
description: >-
(Optional) Token usage information for the response
instructions:
type: string
description: >-
(Optional) System message inserted into the model's context
additionalProperties: false
required:
- created_at
Expand Down
8 changes: 8 additions & 0 deletions docs/static/llama-stack-spec.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7600,6 +7600,10 @@
"$ref": "#/components/schemas/OpenAIResponseUsage",
"description": "(Optional) Token usage information for the response"
},
"instructions": {
"type": "string",
"description": "(Optional) System message inserted into the model's context"
},
"input": {
"type": "array",
"items": {
Expand Down Expand Up @@ -8148,6 +8152,10 @@
"usage": {
"$ref": "#/components/schemas/OpenAIResponseUsage",
"description": "(Optional) Token usage information for the response"
},
"instructions": {
"type": "string",
"description": "(Optional) System message inserted into the model's context"
}
},
"additionalProperties": false,
Expand Down
8 changes: 8 additions & 0 deletions docs/static/llama-stack-spec.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5815,6 +5815,10 @@ components:
$ref: '#/components/schemas/OpenAIResponseUsage'
description: >-
(Optional) Token usage information for the response
instructions:
type: string
description: >-
(Optional) System message inserted into the model's context
input:
type: array
items:
Expand Down Expand Up @@ -6218,6 +6222,10 @@ components:
$ref: '#/components/schemas/OpenAIResponseUsage'
description: >-
(Optional) Token usage information for the response
instructions:
type: string
description: >-
(Optional) System message inserted into the model's context
additionalProperties: false
required:
- created_at
Expand Down
8 changes: 8 additions & 0 deletions docs/static/stainless-llama-stack-spec.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9272,6 +9272,10 @@
"$ref": "#/components/schemas/OpenAIResponseUsage",
"description": "(Optional) Token usage information for the response"
},
"instructions": {
"type": "string",
"description": "(Optional) System message inserted into the model's context"
},
"input": {
"type": "array",
"items": {
Expand Down Expand Up @@ -9820,6 +9824,10 @@
"usage": {
"$ref": "#/components/schemas/OpenAIResponseUsage",
"description": "(Optional) Token usage information for the response"
},
"instructions": {
"type": "string",
"description": "(Optional) System message inserted into the model's context"
}
},
"additionalProperties": false,
Expand Down
8 changes: 8 additions & 0 deletions docs/static/stainless-llama-stack-spec.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7028,6 +7028,10 @@ components:
$ref: '#/components/schemas/OpenAIResponseUsage'
description: >-
(Optional) Token usage information for the response
instructions:
type: string
description: >-
(Optional) System message inserted into the model's context
input:
type: array
items:
Expand Down Expand Up @@ -7431,6 +7435,10 @@ components:
$ref: '#/components/schemas/OpenAIResponseUsage'
description: >-
(Optional) Token usage information for the response
instructions:
type: string
description: >-
(Optional) System message inserted into the model's context
additionalProperties: false
required:
- created_at
Expand Down
2 changes: 2 additions & 0 deletions llama_stack/apis/agents/openai_responses.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -545,6 +545,7 @@ class OpenAIResponseObject(BaseModel):
:param tools: (Optional) An array of tools the model may call while generating a response.
:param truncation: (Optional) Truncation strategy applied to the response
:param usage: (Optional) Token usage information for the response
:param instructions: (Optional) System message inserted into the model's context
"""

created_at: int
Expand All @@ -564,6 +565,7 @@ class OpenAIResponseObject(BaseModel):
tools: list[OpenAIResponseTool] | None = None
truncation: str | None = None
usage: OpenAIResponseUsage | None = None
instructions: str | None = None


@json_schema_type
Expand Down
1 change: 1 addition & 0 deletions llama_stack/providers/inline/agents/meta_reference/responses/openai_responses.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,7 @@ async def _create_streaming_response(
tool_executor=self.tool_executor,
safety_api=self.safety_api,
guardrail_ids=guardrail_ids,
instructions=instructions,
)

# Stream the response
Expand Down
4 changes: 4 additions & 0 deletions llama_stack/providers/inline/agents/meta_reference/responses/streaming.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ def __init__(
text: OpenAIResponseText,
max_infer_iters: int,
tool_executor, # Will be the tool execution logic from the main class
instructions: str,
safety_api,
guardrail_ids: list[str] | None = None,
):
Expand All @@ -133,6 +134,8 @@ def __init__(
self.accumulated_usage: OpenAIResponseUsage | None = None
# Track if we've sent a refusal response
self.violation_detected = False
# system message that is inserted into the model's context
self.instructions = instructions

async def _create_refusal_response(self, violation_message: str) -> OpenAIResponseObjectStream:
"""Create a refusal response to replace streaming content."""
Expand Down Expand Up @@ -176,6 +179,7 @@ def _snapshot_response(
tools=self.ctx.available_tools(),
error=error,
usage=self.accumulated_usage,
instructions=self.instructions,
)

async def create_response(self) -> AsyncIterator[OpenAIResponseObjectStream]:
Expand Down
3 changes: 2 additions & 1 deletion ...n/agents/recordings/00f8a71ccb939737ed72a289eede62998c6882727519858bbd5954307d10a673.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading