AI Transport: Add a guide for token streaming using the OpenAI SDK #3024

lawrence-forooghian · 2025-12-12T18:20:40Z

Description

This guide provides a concrete example of how to implement the message-per-token pattern that Mike documented in #3014.

I initially got Claude to generate this but replaced a fair chunk of its output. I trusted that its prose is consistent with our tone of voice and AI Transport marketing position (whether mine is, I have no idea) and in general trusted its judgements about how to structure the document. I would definitely welcome opinions on all of the above, especially from those familiar with how we usually write docs.

I have tried to avoid repeating too much content from the message-per-token page and have in particular not tried to give an example of hydration since it seems like a provider-agnostic concept.

Checklist

Commits have been rebased.
Linting has been run against the changed file(s).
The PR adheres to the writing style guide and contribution guide.

coderabbitai · 2025-12-12T18:20:46Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch AIT-token-streaming-OpenAI-SDK

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

lawrence-forooghian · 2025-12-12T18:24:06Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+
+You should see publisher output similar to the following:
+
+```text


@GregHolmes is there some sort of collapsible component that I can use for this (like a <details> I guess)? I'd like to include this output but not force the the user to scroll through it all if just skimming.

lawrence-forooghian · 2025-12-12T18:25:28Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+This is only a representative example for a simple "text in, text out" use case, and may not reflect the exact sequence of events that you'll observe from the OpenAI API. It also does not handle response generation errors or refusals.
+</Aside>
+
+1. `{ type: 'response.created', response: { id: 'resp_abc123', … }, … }`: This gives us the response ID, which we'll include in all messages that we publish to Ably for this response. When we receive this event, we'll publish an Ably message named `start`.


@GregHolmes this list, and these JSON events, came out looking a bit cramped and ugly — any suggestions on what I could do to improve this?

I think having all the braces makes this look more complicated than it is and not helping the spacing. If you put it into a table, you could have a column with the heading type for the OpenAI messages and just put response.in_progress or response.output_item.added, then a column for any other interesting fields and a column with the Ably message mapping...?

Alternatively, keep the list format but just put the type name into the text and highlight important fields separately e.g.
3. response.output_item.added with item.type = 'reasoning': we'll ignore this event since we're only interested in messages

The more I think about this, the more I think we should skip this list. I love the technical detail, but it isn't reflected in the code (because we're handling only the specific events we need) and it doesn't aid understanding of Ably. So, I would suggest making the "Understanding Responses API events" section into a clear description of the flow of relevant events we get back from OpenAI, then have a table showing how we'll map those relevant messages to Ably events.

rainbowFi · 2025-12-16T16:01:36Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+import OpenAI from 'openai';
+
+// Initialize Ably Realtime client
+const realtime = new Ably.Realtime({ key: '{{API_KEY}}' });


The docs site is clever and fills this in with a demo API key, which is very convenient but I think means you need to explicitly tell people that this is their Ably API key in the accompanying comment (same for the subscriber snippet)

Do you have an example of where we do this? I looked through a few usages of {{API_KEY}} in the codebase and couldn't find examples of this practice.

rainbowFi · 2025-12-16T16:19:31Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+```javascript
+// Track responses by ID
+const responses = new Map();
+


I personally find it reassuring to see some output when I've started the listener script, just a quick print saying "Waiting for response..." or similar

Sure, have added a "Waiting to receive events from Ably"

rainbowFi · 2025-12-16T16:36:44Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+**Software requirements:**
+- Node.js 20 or higher
+
+We'll be using the OpenAPI SDK v4.x.


Nit - I wouldn't put this in pre-reqs, because you're going to cover installing it below. Instead, I'd mention the version inline right before the install snippet at line 43

Sure — I've added the following in the installing section (the only reason to mention the version number was to highlight that we aren't covering all possible OpenAI SDK versions):

<Aside data-type="note"> We're using version 4.x of the OpenAI SDK. Some details of interacting with the OpenAI SDK may diverge from those given here if you're using a different major version. </Aside>

rainbowFi · 2025-12-16T17:00:51Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+This is only a representative example for a simple "text in, text out" use case, and may not reflect the exact sequence of events that you'll observe from the OpenAI API. It also does not handle response generation errors or refusals.
+</Aside>
+
+1. `{ type: 'response.created', response: { id: 'resp_abc123', … }, … }`: This gives us the response ID, which we'll include in all messages that we publish to Ably for this response. When we receive this event, we'll publish an Ably message named `start`.


I think having all the braces makes this look more complicated than it is and not helping the spacing. If you put it into a table, you could have a column with the heading type for the OpenAI messages and just put response.in_progress or response.output_item.added, then a column for any other interesting fields and a column with the Ably message mapping...?

Alternatively, keep the list format but just put the type name into the text and highlight important fields separately e.g.
3. response.output_item.added with item.type = 'reasoning': we'll ignore this event since we're only interested in messages

rainbowFi · 2025-12-16T17:02:19Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+
+1. `{ type: 'response.created', response: { id: 'resp_abc123', … }, … }`: This gives us the response ID, which we'll include in all messages that we publish to Ably for this response. When we receive this event, we'll publish an Ably message named `start`.
+2. `{ type: 'response.in_progress', … }`: We'll ignore this event.
+3. `{ type: 'response.output_item.added', output_index: 0, item: { type: 'reasoning', … }, … }`: We'll ignore this event since its `item.type` is not `message`.


I think we're ignoring this event because we're not processing reasoning, we're only processing the response message. Either say this here or put it in some summary text at the top of the list

rainbowFi · 2025-12-16T17:08:45Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+
+## Understanding Responses API events <a id="understanding-events"/>
+
+OpenAI's Responses API streams results as a series of events when you set `stream: true`. The primary event type for text generation is `response.output_text.delta`, which contains incremental text content.


I think this is missing a sentence describing the overall response structure heirarchy (output array of content items, and one of those content items is the response, which is streamed as response.output_text.delta events). That will then make the list of events flow more naturally, because I'm expecting the heirarchy

rainbowFi · 2025-12-16T17:14:11Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+This is only a representative example for a simple "text in, text out" use case, and may not reflect the exact sequence of events that you'll observe from the OpenAI API. It also does not handle response generation errors or refusals.
+</Aside>
+
+1. `{ type: 'response.created', response: { id: 'resp_abc123', … }, … }`: This gives us the response ID, which we'll include in all messages that we publish to Ably for this response. When we receive this event, we'll publish an Ably message named `start`.


The more I think about this, the more I think we should skip this list. I love the technical detail, but it isn't reflected in the code (because we're handling only the specific events we need) and it doesn't aid understanding of Ably. So, I would suggest making the "Understanding Responses API events" section into a clear description of the flow of relevant events we get back from OpenAI, then have a table showing how we'll map those relevant messages to Ably events.

rainbowFi · 2025-12-16T17:16:49Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+
+**Key points**:
+
+- **Multiple concurrent responses are handled correctly**: The subscriber receives interleaved tokens for three concurrent AI responses, and correctly pieces together the three separate messages:


I never saw interleaved responses when I ran this script, any idea why? Luck, location, something else? It's not particularly important but I just want to make sure there isn't a change in the example code that is causing the behaviour

I was only seeing them intermittently when I first wrote it, and now I'm unable to get any at all, too! I think it would be good if users could observe this behaviour. One option would be to add small random delays before processing each event, what do you think?

(It's not ideal and distracts from the content of the guide)

Alternatively we could spin up two publisher instances at the same time: node publisher.mjs & node publisher.mjs — on testing this locally it seems to more reliably give interleaved events. But again it complicates the guide.

rainbowFi · 2025-12-16T17:33:26Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+
+### Publisher code
+
+Add the following code to `publisher.mjs`:


I think this would benefit from a quick summary of what the code is doing e.g. "The publisher needs to send the prompt to OpenAI and then process the response stream, publishing relevant events to the Ably channel". Same for the subscriber code snippet

Zak has some good examples in #3018

lawrence-forooghian · 2025-12-16T20:05:35Z

@rainbowFi I've updated the publisher to add an additional prompt ("Write a one-line poem about carrot cake"); missed this out of the original code but it was used when generating the responses shown here

This guide provides a concrete example of how to implement the message-per-token pattern that Mike documented (the one linked to in this guide). I initially got Claude to generate this but replaced a fair chunk of its output. I trusted that its prose is consistent with our tone of voice and AI Transport marketing position (whether mine is, I have no idea) and in general trusted its judgements about how to structure the document. I would definitely welcome opinions on all of the above, especially from those familiar with how we usually write docs. I have tried to avoid repeating too much content from the message-per-token page and have in particular not tried to give an example of hydration since it seems like a provider-agnostic concept.

lawrence-forooghian force-pushed the AIT-token-streaming-OpenAI-SDK branch from deae500 to 3d12fdf Compare December 12, 2025 18:21

lawrence-forooghian added the review-app Create a Heroku review app label Dec 12, 2025

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 12, 2025 18:22 Inactive

lawrence-forooghian commented Dec 12, 2025

View reviewed changes

lawrence-forooghian requested review from GregHolmes and mschristensen December 12, 2025 18:24

lawrence-forooghian commented Dec 12, 2025

View reviewed changes

lawrence-forooghian force-pushed the AIT-token-streaming-OpenAI-SDK branch from 3d12fdf to 9876218 Compare December 12, 2025 18:27

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 12, 2025 18:27 Inactive

lawrence-forooghian force-pushed the AIT-token-streaming-OpenAI-SDK branch from 9876218 to 281713a Compare December 12, 2025 18:33

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 12, 2025 18:33 Inactive

lawrence-forooghian marked this pull request as ready for review December 12, 2025 18:33

lawrence-forooghian force-pushed the AIT-token-streaming-OpenAI-SDK branch from 281713a to 4f188f3 Compare December 12, 2025 18:34

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 12, 2025 18:35 Inactive

mschristensen force-pushed the feature/AIT-51-token-streaming-granular-history branch 2 times, most recently from 0e663af to 52e32d8 Compare December 15, 2025 13:38

Base automatically changed from feature/AIT-51-token-streaming-granular-history to AIT-129-AIT-Docs-release-branch December 16, 2025 12:23

rainbowFi reviewed Dec 16, 2025

View reviewed changes

lawrence-forooghian force-pushed the AIT-token-streaming-OpenAI-SDK branch from 4f188f3 to cabebd0 Compare December 16, 2025 20:02

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 16, 2025 20:03 Inactive

lawrence-forooghian force-pushed the AIT-token-streaming-OpenAI-SDK branch from cabebd0 to f827802 Compare December 16, 2025 20:13

ably-ci deployed to ably-docs-ait-token-str-qkwgk2 December 16, 2025 20:14 View deployment


		You should see publisher output similar to the following:

		```text


		## Understanding Responses API events <a id="understanding-events"/>

		OpenAI's Responses API streams results as a series of events when you set `stream: true`. The primary event type for text generation is `response.output_text.delta`, which contains incremental text content.


		Key points:

		- Multiple concurrent responses are handled correctly: The subscriber receives interleaved tokens for three concurrent AI responses, and correctly pieces together the three separate messages:


		### Publisher code

		Add the following code to `publisher.mjs`:

AI Transport: Add a guide for token streaming using the OpenAI SDK #3024

Are you sure you want to change the base?

AI Transport: Add a guide for token streaming using the OpenAI SDK #3024

Conversation

lawrence-forooghian commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

coderabbitai bot commented Dec 12, 2025

Review skipped

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lawrence-forooghian commented Dec 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

lawrence-forooghian commented Dec 12, 2025 •

edited

Loading