How to get structured output but use tools and retain messages

[ppetrick]

I have a rag chatbot, currently using usechat and streamtext, configured with openai.
Ideally, I'd like to format the output of my responses from openai, as I want to output not just text, but blocks of citations (with links), something like this:
export const rulesResponseSchema = z.object({ answer: z.string().describe('Answer to the user\'s question.'), citations: z.array(z.object({ pageNum: z.number().describe('Page number of the rule.'), citation: z.string().describe('Citation to the rule.'), })), });

I know there's streamObject and useObject, but streamObject doesn't let you use tools, and useObject doesn't manage the conversations the same way that useChat does. How might I go about achieving this? Or am I missing something obvious?

[lgrammel]

Streaming data might be a better fit for your use case: https://sdk.vercel.ai/docs/ai-sdk-ui/streaming-data

[lgrammel]

Would this be an option: https://sdk.vercel.ai/docs/ai-sdk-core/agents#structured-answers

[ppetrick]

Works great for the first question in the chat. But when I ask a second question, I get a
MessageConversionError [AI_MessageConversionError]: ToolInvocation must have a result
when callign convertToCoreMessages, which I gather is because the structured content isn't what the conversion function expects?

[lgrammel]

You can inject some fake tool call result (e.g. a string "processed" or something). It is required to ensure the conversation is consistent and can be processed by the LLM, which expects result parts for tool calls.

[ppetrick]

Appreciate the responses. Ended up using streaming data and nesting a generateobject in the onfinish of the streamtext to generate the citation object. Feels a bit clunky, but gets the job done.