inputAudioTranscription not working

[rimallo]

Dear community!
At some point after numerous updates audio transcribe stoped working.
Nor from respected @m1guelpf in the initial example (which was worked perfectly before)

try await conversation.whenConnected {
    try await conversation.updateSession { session in
        // update system prompt
        session.instructions = "You are a helpful assistant."

        // enable transcription of users' voice messages
        session.inputAudioTranscription = Session.InputAudioTranscription()

        // ...
    }
}

neither any modified variant like
session.inputAudioTranscription = Session.InputAudioTranscription.init(model: .gpt4oMini)
Please, someone help resolving the issue.

[rimallo]

App just receives 1-2 messages from user and stops responding. After commenting out any variation of line with session.session.inputAudioTranscription = ... inside updateSession - works perfectly but transcript: nil.

updateSession itself works perfectly - custom function(s), instructions, voice etc works like a charm, except of inputAudioTranscription.

[benvolioT]

The documentation for the transcription use case clearly states that when connecting for transcription, you must include the query parameter intent=transcription (e.g., the URL would look like wss://api.openai.com/v1/realtime?intent=transcription). However, in the SDK’s helper methods (the webSocket(authToken:model:) method), the URL is constructed with only the model query parameter. I don't see an intent parameter. I tried adding one alongside the model parameter, and the server rejected the connection request. I tried removing the model parameter, and that didn't seem to work either (although I forget now what the specific issue was).

This library's documentation emphasizes that transcription settings (like specifying the model, etc.) need to be applied to the session once connected. In the SDK, this is done by updating the session via updateSession. Maybe the issue is that the URL connection isn’t set for transcription (i.e. missing intent=transcription), so setting inputAudioTranscription is not working?

Or maybe the OpenAI documetation I linked to above is out of date?

I don't see any obvious answer at the moment. Does that spark any ideas from anyone else?

[arturonaredo]

As OpenAI states in their documenation

WebSockets are a broadly supported API for realtime data transfer, and a great choice for connecting to the OpenAI Realtime API in server-to-server applications. For browser and mobile clients, we recommend connecting via WebRTC.

Currently OpenAIRealtime class only supports web sockets and the implementation of webrtc is using wss, which is incorrect.

Using WebRTC the server transcription is correct:

WebRTC decoded server event: conversationItemInputAudioTranscriptionCompleted(OpenAIRealtime.ServerEvent.ConversationItemInputAudioTranscriptionCompletedEvent(eventId: "event_BIEzbMasdf9inJUri", itemId: "item_BIEasdf6kDcFlyIy", contentIndex: 0, transcript: "Ejemplo ejemplo ejemplo"))

There is a PR open https://github.com/m1guelpf/swift-realtime-openai/pull/26/files to fix webrtc implementation but...

GitHub

Added the WebRTC support. by jeffxtang · Pull Request #26 · m1guelpf/swift-realtime-openai

Added the Updated at the beginning. Tested with the iOS demo here.

[rimallo]

@m1guelpf please, could you review PR26 and let us go on with using InputAudioTranscription?

[rimallo]

@arturonaredo please, advice: after applying your commit as a patch how can I force the lib use WebRTC? It still uses WebSocket. Muchísimas gracias!

[arturonaredo]

Not my PR!

But I see that you need to specify a boolean in the method signature

public convenience init(authToken token: String, webRTC: Bool) async {

I have diverged too much now from this library given the number of issues I had with function calling, webrtc, transcription..

[rimallo]

@arturonaredo thank you for participation in solving issue I am facing. Tried to apply PR as a patch and after struggling I came up with running init with a call you mentioned above but WebRTC not starting and produces more errors. Nor my experience, neither coding AI helpers help to solve it :(

Media stream added.
Audio track received
ICE connection state changed to RTCIceConnectionState(rawValue: 2)
Data channel changed to RTCDataChannelState(rawValue: 1)

           AVAEInternal.h:71    required condition is false: [AVAudioNodeImpl.h:303:AVAE_CheckNodeHasEngine: (_engine != nil)]
*** Terminating app due to uncaught exception 'com.apple.coreaudio.avfaudio', reason: 'required condition is false: _engine != nil'

Feeling stuck and frustrated. I think RealTime API is useless without audio transcription working.

[sjwhitmore]

Running into this issue as well - did folks end up switching to the WebRTC fork? Or are there any other workarounds people would recommend?