Skip to content

feat(react): wire user-llm-text into usePipecatConversation - #225

Open
rahulsolanki001 wants to merge 2 commits into
pipecat-ai:mainfrom
rahulsolanki001:fix/chat-mode-user-messages-in-conversation
Open

feat(react): wire user-llm-text into usePipecatConversation#225
rahulsolanki001 wants to merge 2 commits into
pipecat-ai:mainfrom
rahulsolanki001:fix/chat-mode-user-messages-in-conversation

Conversation

@rahulsolanki001

Copy link
Copy Markdown

Follow-up to #211 — depends on it (branched from that fix, needs RTVIEvent.UserLlmText from client-js). Please merge #211 first.

Problem

In chat mode the server emits user-llm-text instead of user-transcription (typed text bypasses STT entirely). Because usePipecatConversation had no handler for RTVIEvent.UserLlmText, user messages were silently dropped and never appeared in the conversation array.

Solution

Handle RTVIEvent.UserLlmText in useConversationEventWiring with a guard that avoids duplicating text in voice mode:

  • Voice modeuser-transcription already created an in-progress user message. We skip upsertUserTranscript and only call finalizeLastMessage to mark the turn complete.
  • Chat mode — No in-progress user message exists. We create one from the LLM text, then finalize it.
useRTVIClientEvent(
  RTVIEvent.UserLlmText,
  useAtomCallback(
    useCallback((get, set, data) => {
      const messages = get(messagesAtom);
      const lastUserIdx = findLastIndex(
        messages,
        (m: ConversationMessage) => m.role === "user"
      );

      if (lastUserIdx === -1 || messages[lastUserIdx].final) {
        upsertUserTranscript(get, set, data.text ?? "", true);
      }
      finalizeLastMessage(get, set, "user");
    }, [])
  )
);

Changes

File Change
client-react/src/conversation/useConversationEventWiring.ts Wire RTVIEvent.UserLlmText with voice/chat mode guard

Test plan

Rahul Solanki and others added 2 commits May 30, 2026 12:20
mergeMessages was merging bot turns that were already finalized
(final: true) into subsequent bot turns within the 30-second window,
causing text from a completed turn to accumulate into the next turn's
bubble. Add !lastMerged.final guard to the shouldMerge condition.

RTVIMessageType.USER_LLM_TEXT had no corresponding RTVIEvent, no
handler in PipecatClient.handleMessage, and no callback type. Add
RTVIEvent.UserLlmText, UserLLMTextData type, onUserLlmText callback,
and a USER_LLM_TEXT case in handleMessage so callers can react to user
LLM text events. Conversation hook integration is intentionally left
for a follow-up: wiring user-llm-text -> upsertUserTranscript
unconditionally would duplicate user text in voice mode because
user-transcription already finalizes a part before user-llm-text fires.

Fixes pipecat-ai#210
In chat mode the server emits user-llm-text instead of
user-transcription (typed text bypasses STT entirely), so user messages
were never added to the conversation array.

Handle RTVIEvent.UserLlmText in useConversationEventWiring with a guard
that avoids duplicating text in voice mode:

- If an in-progress user message already exists (voice mode —
  user-transcription already created it), skip upsertUserTranscript and
  only call finalizeLastMessage to mark the turn complete.
- If no in-progress user message exists (chat mode), create one from the
  LLM text first, then finalize it.

Depends on the RTVIEvent.UserLlmText addition in client-js.

Closes pipecat-ai#210
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant