feat(react): wire user-llm-text into usePipecatConversation - #225
Open
rahulsolanki001 wants to merge 2 commits into
Open
feat(react): wire user-llm-text into usePipecatConversation#225rahulsolanki001 wants to merge 2 commits into
rahulsolanki001 wants to merge 2 commits into
Conversation
mergeMessages was merging bot turns that were already finalized (final: true) into subsequent bot turns within the 30-second window, causing text from a completed turn to accumulate into the next turn's bubble. Add !lastMerged.final guard to the shouldMerge condition. RTVIMessageType.USER_LLM_TEXT had no corresponding RTVIEvent, no handler in PipecatClient.handleMessage, and no callback type. Add RTVIEvent.UserLlmText, UserLLMTextData type, onUserLlmText callback, and a USER_LLM_TEXT case in handleMessage so callers can react to user LLM text events. Conversation hook integration is intentionally left for a follow-up: wiring user-llm-text -> upsertUserTranscript unconditionally would duplicate user text in voice mode because user-transcription already finalizes a part before user-llm-text fires. Fixes pipecat-ai#210
In chat mode the server emits user-llm-text instead of user-transcription (typed text bypasses STT entirely), so user messages were never added to the conversation array. Handle RTVIEvent.UserLlmText in useConversationEventWiring with a guard that avoids duplicating text in voice mode: - If an in-progress user message already exists (voice mode — user-transcription already created it), skip upsertUserTranscript and only call finalizeLastMessage to mark the turn complete. - If no in-progress user message exists (chat mode), create one from the LLM text first, then finalize it. Depends on the RTVIEvent.UserLlmText addition in client-js. Closes pipecat-ai#210
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #211 — depends on it (branched from that fix, needs
RTVIEvent.UserLlmTextfrom client-js). Please merge #211 first.Problem
In chat mode the server emits
user-llm-textinstead ofuser-transcription(typed text bypasses STT entirely). BecauseusePipecatConversationhad no handler forRTVIEvent.UserLlmText, user messages were silently dropped and never appeared in the conversation array.Solution
Handle
RTVIEvent.UserLlmTextinuseConversationEventWiringwith a guard that avoids duplicating text in voice mode:user-transcriptionalready created an in-progress user message. We skipupsertUserTranscriptand only callfinalizeLastMessageto mark the turn complete.Changes
client-react/src/conversation/useConversationEventWiring.tsRTVIEvent.UserLlmTextwith voice/chat mode guardTest plan
usePipecatConversationuser-transcriptionmessage is finalized, not duplicatedUserTranscriptbehavior unchangedtsc --noEmit)