Add ExternalAudioSource for pushing PCM frames from the application - #282
Draft
hiroshihorie wants to merge 7 commits into
Draft
Add ExternalAudioSource for pushing PCM frames from the application#282hiroshihorie wants to merge 7 commits into
hiroshihorie wants to merge 7 commits into
Conversation
…ed send streams Audio sources that deliver frames via AddSink (bypassing the ADM) mark themselves with is_external_source(). The flag propagates from AudioSourceInterface through LocalAudioSinkAdapter to the voice engine, which reconfigures the AudioSendStream so it never registers with AudioState. This prevents device-captured audio from being mixed into streams fed by an external source. Semantics ported from the rust-sdks external_audio_source.patch, with hardening beyond the original: - The flag is evaluated on every SetSource call, not only on first attach. On replaceTrack the sink adapter object stays the same while the underlying source type may change, so a latched flag would leave a device track silent (stream unregistered) or mix ADM audio into an external track (stream still registered). - The stream is cycled Stop -> reconfigure -> restart across a flag flip so AudioState registration is always added and removed under the same flag value. Without this, a flip while sending could leave a destroyed stream registered in AudioTransportImpl (use-after-free on the next device capture) or unregister a never-added stream. - StoreEncoderProperties also skips AudioState registration so a codec reconfigure mid-call cannot re-register an external stream. - The sink adapter flag is assigned unconditionally in AudioRtpSender::SetSend, so a sourceless replacement track resets it.
A LocalAudioSource subclass with a real sink list that applications push interleaved int16 PCM into, bypassing the AudioDeviceModule. Supports two modes: queue_size_ms == 0 delivers exact 10 ms frames synchronously on the calling thread; queue_size_ms > 0 buffers pushed audio and paces 10 ms deliveries on a dedicated task queue, feeding silence on underrun and flushing a sub-frame residual padded with silence so it never sits in the buffer adding latency. Back-pressure is signaled through an optional per-push completion callback that fires once the buffer drains below the notify threshold, or by polling BufferedDurationMs(). Design ported from the rust-sdks AudioTrackSource::InternalSource, with several deliberate divergences: - Format is validated on every push in both modes and invalid constructor arguments make Create() return nullptr, keeping the validation in one place for wrappers. - Consumed samples are tracked with a read offset and reclaimed in one amortized pass, instead of memmoving the remaining buffer on every 10 ms tick under the lock. - The pacer delivers from a snapshot outside the state lock so pushers do not wait on per-sink work, and RemoveSink fences on a dedicated delivery mutex so a removed sink can be destroyed safely as soon as the call returns. Synchronous mode delivers under the state lock, which is what serializes concurrent pushers. - Completion callbacks run outside the locks, so a handler may push the next frame or remove a sink without deadlocking. - The destructor stops the pacer before teardown and flushes a pending completion instead of stranding the caller. - BufferedDurationMs multiplies before dividing so rates like 44100 report exactly.
Covers both modes with a simulated clock: synchronous delivery and 10 ms enforcement, argument and format validation, buffered pacing and ordering, silence on underrun, sub-frame residual flushing, capacity rejection, completion-callback back-pressure (inline, deferred, single in-flight, re-entrant push from the handler), ClearBuffer, exact duration reporting for 44.1 kHz rates, and destruction while the pacer is running. A real-thread test verifies RemoveSink blocks until an in-flight delivery finishes and that a removed sink receives nothing afterwards.
…urce RTCExternalAudioSource subclasses RTCAudioSource so it works with the existing audioTrackWithSource:trackId:. Frames are pushed as raw interleaved int16 NSData, AVAudioPCMBuffer (int16/float32, interleaved or not), or CMSampleBuffer carrying linear PCM (e.g. ReplayKit app audio). The completion handler surfaces the buffered-mode back-pressure signal (a rejected push discards it, documented), and bufferedDurationMs supports polling instead. Created via RTCPeerConnectionFactory externalAudioSourceWithSampleRate:channels:queueSizeMs:, which relies on the native Create() for argument validation and returns nil on failure.
- Sort the external_audio_source entries in pc/BUILD.gn and sdk/BUILD.gn - Add the missing rtc_base:platform_thread dep for the unittest - Allow common_audio/include in sdk/objc DEPS (used by RTCExternalAudioSource.mm), matching the existing common_video/include rule - Drop an unused NSString+StdString.h import - Include external_source in AudioSendStream::Config::ToString so the flag that decides AudioState registration is visible in logs - Remove a dead audio_track() guard in AudioRtpSender::SetSend, guaranteed by the can_send_track() DCHECK above and contradicting its own comment - clang-format the unit test
- Make LocalAudioSinkAdapter::is_external_source_ a std::atomic<bool>. It is written on the signaling thread in AudioRtpSender::SetSend and read on the worker thread inside the same call's BlockingCall, so the ordering is already established, but the plain bool was racy by annotation next to the explicitly mutex-guarded sink_. Also fold the accessors into the existing access sections instead of reopening public: mid-class. - Decide whether PushFrame's on_complete fires inline before it is moved into on_complete_. absl::AnyInvocable documents the moved-from state as unspecified, so testing the callable after the move relied on a vendored implementation detail. - clang-format
BadLord1st
added a commit
to WolfSofware/webrtc-xcframework
that referenced
this pull request
Aug 27, 2026
Апстримный бинарник не содержит RTCExternalAudioSource (PR webrtc-sdk/webrtc#282 не влит), а без него звук демонстрации нельзя отдать отдельной дорожкой. Переставляем адрес и контрольную сумму на нашу сборку из WolfSofware/webrtc-build.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a pushable audio source to the fork so applications can publish audio (app or screen-share audio, files) as independent tracks without mixing it into the ADM microphone path.
Closes #210. Supersedes #258 (cc @cloudwebrtc).
What this adds
webrtc::ExternalAudioSource(pc/external_audio_source.h): aLocalAudioSourcesubclass the app pushes interleaved int16 PCM into. Synchronous mode delivers on the caller's thread. Buffered mode paces 10 ms frames from a dedicated task queue, feeds silence on underrun, and offers a single-slot completion callback for back-pressure.is_external_sourceplumbing:AudioSendStreamskips AudioState registration for externally fed streams, so ADM capture audio is not also pushed into them. The flag is re-evaluated on everySetSourceandSetSend, so replacing tracks in either direction restores the correct path.RTCExternalAudioSourceplus anRTCPeerConnectionFactorymethod, with NSData, AVAudioPCMBuffer, and CMSampleBuffer capture entry points.Review guide
The first commit (8bb153e) is the only fork-wide behavioral change: 7 files adding the flag and the AudioState registration guards. It is inert until something overrides the virtual, and nothing in-tree does until the second commit. The file set matches the
external_audio_source.patchthat rust-sdks currently applies at build time, which this supersedes so that patch can be deleted there.Fixes over that patch, found during implementation:
StoreEncoderPropertiesis a thirdAddSendingStreamcall site the patch missed. A codec reconfigure mid-call would have re-registered an external stream with AudioState.replaceTrack, which can leave a danglingAudioSender(use-after-free) or fireRemoveSendingStreamfor a never-registered stream.Testing
pc/external_audio_source_unittest.ccunder a simulated clock: pacing, underrun silence, capacity, back-pressure, destruction races, plus a real-thread RemoveSink fence test (10/10 under--gtest_repeat).AudioSendStream*(37),LocalAudioSource*andRtpSenderReceiver*(94),gn checkon the new and test targets.