🚨 Slim down `[audio]` to torchaudio + soundfile + soxr (drop librosa and phonemizer) by eustlb · Pull Request #50610 · huggingface/transformers

eustlb · 2026-11-11T11:23:52Z

Important

For the release notes (one entry covering #50408 and this PR):

🚨 transformers[audio] now installs only torchaudio, soundfile and soxr. It no longer installs librosa,
phonemizer, pyctcdecode or kenlm, shrinks from ~371 MiB to ~1.4 MiB on top of torch, and no longer downgrades numpy
to 1.x.

Who is affected: new environments (fresh venvs, Docker images, CI) built with pip install transformers[audio].
Upgrading an existing environment doesn't uninstall anything, so packages that are already installed keep working.

What breaks in such an environment:

If you…	You now get	Fix
`import librosa` in your own code	`ModuleNotFoundError`	`pip install librosa`
phonemize text with `Wav2Vec2PhonemeCTCTokenizer`, or with a VITS tokenizer that has `phonemize=True`	`ImportError`	`pip install phonemizer` + the `espeak-ng` system library
load `Wav2Vec2ProcessorWithLM`	`ImportError`	`pip install pyctcdecode kenlm` (deprecated, removed in v6)
run the ASR pipeline on a `*WithLM` checkpoint	no error: falls back to greedy CTC with a warning, so transcriptions can differ	same as above

What keeps working without them:

load_audio: same output as librosa.load.
The audio pipelines.
Every feature extractor: parakeet, cohere_asr and nemotron_asr_streaming no longer need librosa.
Loading and decoding with Wav2Vec2PhonemeCTCTokenizer: phonemizer is now only needed to phonemize text.

Deprecated: load_audio(..., backend="librosa"); use backend="soundfile" (same output).

What does this PR do?

Slims [audio] down to the audio core's dependencies: torchaudio, soundfile, soxr. librosa and phonemizer move
to [testing]. Stacked on #50408 (pyctcdecode / kenlm).

Assumption: `transformers[audio]` installs the audio core, nothing more

The audio core is the audio code shared across models and tasks:

audio_utils (load_audio, spectrograms, mel filters);
the audio pipelines;
the feature extractors and processors common to audio models.

[audio] should install exactly what the core needs, so that it works out of the box. Model-specific dependencies stay
opt-in behind @requires / requires_backends, and missing ones raise a clear ImportError. This is already how
[vision] works: it installs torchvision + Pillow, while opencv, scipy and pytesseract stay opt-in for the vision
models that need them.

[audio] currently breaks this assumption both ways:

Packages that aren't core:
- librosa: ~300 MiB (numba, llvmlite, scipy, scikit-learn), while the core only uses librosa.load
  (= soundfile + soxr) and librosa.filters.mel, both easy to do without it. Only pop2piano needs it beyond that.
- phonemizer: only used to phonemize text in two tokenizers, and it fails at runtime without the system
  espeak-ng, which pip can't install.
Core packages that aren't declared: soundfile and soxr, which the core does use, only arrive because librosa
depends on them.

With #50408: ~371 MiB → ~1.4 MiB on top of torch, install time 59 s → 3 s, and numpy is no longer downgraded to 1.x.

🚨 What breaks

Removed	What breaks	Severity
librosa	Only user code that imports librosa without declaring it. pop2piano still needs it, but it also needs essentia and pretty_midi, which were never in `[audio]`.	Medium: common in the wild, clear `ImportError`
phonemizer	Phonemizing text (`Wav2Vec2PhonemeCTCTokenizer`, VITS `phonemize=True`). It never worked from `[audio]` alone anyway (needs espeak-ng).	Low

Numbers change slightly in two places that loaded audio with librosa and now use load_audio's default auto backend: when torchcodec is installed, they decode and resample through FFmpeg instead of soundfile + soxr. Without torchcodec, the output is identical to before.

Serving (/v1/audio/transcriptions): m4a / mp4 / webm uploads now work when torchcodec is installed (they never worked before: librosa doesn't fall back to audioread for in-memory bytes).
NemotronH_Omni, when given a file path directly (processor(audio="clip.wav")): it now behaves like apply_chat_template for the same model, which already used auto.

Tests

bit-exact vs librosa: load_audio on 40 synthetic + 14 real formats; mel_filter_bank(dtype=np.float32) on 72 configs and the three extractors' default filters (librosa 1.0.0, numpy 2.5.4)
floor versions (soundfile==0.13.0, soxr==0.5.0) also bit-exact

Mel filters reproducer (not shipped as a test)

import itertools

import librosa
import numpy as np

from transformers import CohereAsrFeatureExtractor, NemotronAsrStreamingFeatureExtractor, ParakeetFeatureExtractor
from transformers.audio_utils import mel_filter_bank

# shared function: 6 rates x 4 n_fft x 3 n_mels
for sr, n_fft, n_mels in itertools.product(
    [8000, 16000, 22050, 24000, 44100, 48000], [256, 400, 512, 1024], [64, 80, 128]
):
    ours = mel_filter_bank(
        num_frequency_bins=n_fft // 2 + 1,
        num_mel_filters=n_mels,
        min_frequency=0.0,
        max_frequency=sr / 2,
        sampling_rate=sr,
        norm="slaney",
        mel_scale="slaney",
        dtype=np.float32,
    )
    ref = librosa.filters.mel(sr=sr, n_fft=n_fft, n_mels=n_mels, fmin=0.0, fmax=sr / 2, norm="slaney")
    assert ours.dtype == ref.dtype and np.array_equal(ours.T, ref), (sr, n_fft, n_mels)

# the three feature extractors, default configs
for cls in [ParakeetFeatureExtractor, NemotronAsrStreamingFeatureExtractor, CohereAsrFeatureExtractor]:
    fe = cls()
    ref = librosa.filters.mel(
        sr=fe.sampling_rate, n_fft=fe.n_fft, n_mels=fe.feature_size, fmin=0.0, fmax=fe.sampling_rate / 2, norm="slaney"
    )
    assert np.array_equal(fe.mel_filters.numpy(), ref), cls.__name__

print("all bit-exact")

HuggingFaceDocBuilderDev · 2026-11-11T11:34:51Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Pages changed in this PR:

eustlb

Self-review: notes on the main changes.

eustlb · 2026-11-11T16:22:37Z

    "scikit-learn",
    "scipy",
    "sentencepiece>=0.1.91,!=0.1.92",
+    "soundfile>=0.13.0",


first versions with wheels on every supported platform and Python (3.11–3.14), so nothing compiles from source

might be good to keep a comment exactly for that no? just super small on the side # ensures wheels are available

eustlb · 2026-11-11T16:22:37Z

        "parameterized",
        "psutil",
        "diffusers",  # Needed for VibeVoice TTS integration tests
+        "librosa",  # reference implementation in audio tests


librosa and phonemizer move to [testing], so [dev] and every CI image keep them.

eustlb · 2026-11-11T16:22:37Z


-    # torchcodec handles audio/video; librosa only plain audio. `backend` lets callers pin one.
+    # torchcodec handles audio/video; soundfile only plain audio. `backend` lets callers pin one.
+    if backend == "librosa":


backend="librosa" is kept as a deprecated alias of "soundfile". Its output is identical, so callers only have to rename one string.

yep thats good

eustlb · 2026-11-11T16:22:37Z


-    requires_backends(load_audio, ["librosa"])
-    return librosa.load(BytesIO(source) if isinstance(source, bytes) else source, sr=sampling_rate)[0]
+    # Same steps as `librosa.load`, so the output is identical: downmix by averaging, `soxr_hq` resampling, and


This replaces librosa.load, which is itself soundfile + soxr: same decode, mean downmix, soxr_hq resampling and fix_length. It's bit-exact with librosa.load on 40 synthetic cases (WAV/FLAC/OGG/MP3 × mono/stereo × 5 rates) and on the 14 non-torchcodec files of hf-internal-testing/all-audio-formats, including with the floor versions. auto falls back to it when torchcodec isn't usable.

implementation yes, but please move into a separate helper function

eustlb · 2026-11-11T16:22:37Z

    norm: str | None = None,
    mel_scale: str = "htk",
    triangularize_in_mel_space: bool = False,
+    dtype: str | np.dtype = np.float64,


Important for context ⚠️ The only difference between mel_filter_bank(norm="slaney", mel_scale="slaney") and librosa.filters.mel(norm="slaney") is that librosa rounds each band to float32 before the slaney normalization (the frequency grids are bit-identical). The default float64 leaves every existing caller unchanged; float32 is bit-exact with librosa (72 configs checked, reproducer in the PR description).

eustlb · 2026-11-11T16:22:37Z

-        mel_filters = librosa.filters.mel(
-            sr=sampling_rate, n_fft=n_fft, n_mels=feature_size, fmin=0.0, fmax=sampling_rate / 2, norm="slaney"
+        # float32 rounding before the slaney normalization: bit-exact with `librosa.filters.mel(..., norm="slaney")`
+        mel_filters = mel_filter_bank(


This is the call the old TODO had commented out. Parakeet, cohere_asr and nemotron_asr_streaming only used librosa for librosa.filters.mel, and they had @requires(..., "librosa"), so without this they'd stop working for plain [audio] users. The filters are bit-identical, so the features don't change. pop2piano keeps librosa: it also needs essentia and pretty_midi, which were never in [audio].

eustlb · 2026-11-11T16:22:37Z

-
-        if do_phonemize:
-            self.init_backend(self.phonemizer_lang)
+        # Created on first use so that loading the tokenizer and decoding work without phonemizer / espeak-ng


The backend used to be created in __init__, so even loading facebook/wav2vec2-xlsr-53-espeak-cv-ft for inference needed phonemizer and espeak-ng, although only encoding text uses it. With phonemizer out of [audio], it's now created on first phonemize(), so loading and decoding work without it.

eustlb · 2026-11-11T16:22:37Z

-
        sampling_rate = audio_processor.feature_extractor.sampling_rate
-        audio_array, _ = librosa.load(io.BytesIO(file_bytes), sr=sampling_rate, mono=True)
+        audio_array = load_audio(base64.b64encode(file_bytes).decode("ascii"), sampling_rate=sampling_rate)


This was librosa.load(BytesIO(...)). It now uses load_audio with the auto backend: torchcodec when usable, otherwise soundfile, which gives the same output as before.

not sure but wdyt about adding a kwarg that handles this case directly?

keeping it for another PR 👌

eustlb · 2026-11-11T16:22:37Z

        if "image" in self.model.input_modalities or "video" in self.model.input_modalities:
            requires_backends(self, "vision")
            requires_backends(self, "torchvision")
-        if "audio" in self.model.input_modalities:


This gate never called librosa. load_audio raises its own clear error when decoding needs soundfile.

eustlb · 2026-11-11T17:04:46Z

        for audio_item in audio:
            if isinstance(audio_item, str):
-                waveform = self._load_audio(audio_item, sampling_rate)
+                waveform = load_audio(audio_item, sampling_rate=sampling_rate)


Default auto backend on purpose. apply_chat_template already decodes this model's audio paths and URLs with load_audio(...) (auto), so a direct processor(audio="clip.wav") call now behaves the same way. It also restores m4a / mp4 / webm when torchcodec is installed (the old librosa.load(path) handled them through audioread. Without torchcodec the output is identical to the old librosa path.

ig it changes the integration tests slightly tho? or have you forced a backend there

integration test relies on .apply_chat_template path that already call load_audio directly, actually this loading in the processor call is strange, I'll have a look (but not so important tbh)

eustlb · 2026-11-11T17:04:46Z

    else:
        # frequencies of FFT bins in Hz
-        fft_freqs = np.linspace(0, sampling_rate // 2, num_frequency_bins)
+        fft_freqs = np.linspace(0, sampling_rate / 2, num_frequency_bins)


// 2 → / 2: the Nyquist frequency is sampling_rate / 2. For even rates (every model default) the two are the same value, so nothing changes. For odd rates (e.g. 11025) the grid used to stop half a Hz short; it's now correct, and bit-exact with librosa as well (reproducer in the description covers 8001 / 11025 / 16001 / 22051).

could we consider this as a separate fix tho? it's not related to the deps inherently

eustlb · 2026-11-11T17:04:46Z

 RUN uv pip install --no-cache-dir GitPython
 RUN uv pip install --no-cache-dir --upgrade 'torch<=2.15.0' 'torchaudio' 'torchvision' --index-url https://download.pytorch.org/whl/cpu
-RUN uv pip install --no-cache-dir "git+https://github.com/huggingface/transformers.git@${REF}#egg=transformers[quality,testing,torch-speech,vision]"
+RUN uv pip install --no-cache-dir "git+https://github.com/huggingface/transformers.git@${REF}#egg=transformers[quality,testing,vision]"


Drive-by: there is no torch-speech extra (removed from setup.py a long time ago) and uv pip ignores unknown extras, so the image doesn't change. It's dropped only because it still looks like an audio extra.

vasqu

Thanks a lot, some comments might not make sense as I saw things later but the core is good overall

I have a few smaller comments here and there

vasqu · 2026-11-11T18:24:35Z

Oh yea we have (had?) quite a lot of docs not using our load helpers, that also goes for vision. Mind to open an issue for the vision side to track to make this consistent

vasqu · 2026-11-11T18:25:42Z

question before reading the actual code: are we hard deprecating load_audio_librosa? Could we potentially keep an alias that warns and redirects to base load_audio?

yep we already have an alias for load_audio_librosa :)

vasqu · 2026-11-11T18:26:36Z


 ```bash
-pip install transformers datasets evaluate jiwer soundfile librosa torchcodec
+pip install transformers datasets evaluate jiwer soundfile torchcodec


not on you but might be nice to give options for each task at some point: so pip install transformers[audio-asr]

agree, keeping track of this (on my long list 😅)

vasqu · 2026-11-11T18:28:16Z

-
        sampling_rate = audio_processor.feature_extractor.sampling_rate
-        audio_array, _ = librosa.load(io.BytesIO(file_bytes), sr=sampling_rate, mono=True)
+        audio_array = load_audio(base64.b64encode(file_bytes).decode("ascii"), sampling_rate=sampling_rate)


not sure but wdyt about adding a kwarg that handles this case directly?

vasqu · 2026-11-11T18:29:22Z

+`apt install espeak-ng` on Debian/Ubuntu, `brew install espeak-ng` on macOS). Please note that you may need to restart
+your runtime after installation.


last sentence imo not necessary or am i heavily biased that this is obv? :D

vasqu · 2026-11-11T18:36:22Z

+            dtype=np.float32,
        )
-        self.mel_filters = torch.from_numpy(mel_filters).to(torch.float32)
+        self.mel_filters = torch.from_numpy(np.ascontiguousarray(mel_filters.T))


np.ascontiguousarray(mel_filters.T) could be directly done within mel_filter_bank no?

not really, since the output shape for it is already in the API it would be breaking :/

no worries wouldve been a nice to have but no biggie

vasqu · 2026-11-11T18:37:17Z

        for audio_item in audio:
            if isinstance(audio_item, str):
-                waveform = self._load_audio(audio_item, sampling_rate)
+                waveform = load_audio(audio_item, sampling_rate=sampling_rate)


ig it changes the integration tests slightly tho? or have you forced a backend there

vasqu · 2026-11-11T18:38:18Z

        if phonemizer_lang is not None:
            self.phonemizer_lang = phonemizer_lang
-            self.init_backend(phonemizer_lang)
+            self._phonemizer_backend = None


why is it reset here now?

backend now created lazily, on first call, so loading the tokenizer and decoding work without phonemizer / and espeak-ng.
Setting phonemizer_lang here drops the cached backend so the next rebuilds it for that language, the previous eager init_backend needed phonemizer + espeak-ng just to set language

vasqu · 2026-11-11T18:38:59Z

+@require_soundfile
+@require_soxr


I'd assume this combination to happen frequently, let's make anoth requires helper?

totally, added require_audio that mirrors require_vision for core loading, cleaner

vasqu · 2026-11-11T18:41:47Z

+                        except Exception:
+                            self.skipTest(f"libsndfile cannot encode {audio_format}")


is that really ok?

github-actions · 2026-11-11T22:12:49Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: cohere_asr, muse_spark, nemotron_asr_streaming, nemotron_h_omni, parakeet, qwen2_5_omni, qwen2_audio, vibevoice_acoustic_tokenizer, wav2vec2_phoneme

load_audio gets a "soundfile" backend (soundfile decode + soxr_hq resample + librosa's length fix) that is bit-exact with librosa.load; "auto" falls back to it when torchcodec is unavailable. backend="librosa" becomes a deprecated alias (FutureWarning). soxr is imported behind is_soxr_available instead of piggybacking on librosa. Serving, load_audio_as and nemotron_h_omni now go through load_audio; the any_to_any librosa gate is removed; docs and docstrings use load_audio instead of librosa.load. librosa leaves the `[audio]` extra, replaced by what the new path uses: soundfile>=0.13.0 and soxr>=0.5.0 (both previously pulled in only transitively through librosa). librosa moves to `[testing]` as the reference implementation in audio tests, so `[dev]` and CI keep it. Floors: soundfile>=0.13.0 (first Linux aarch64 wheels, bundled libsndfile with MP3) and soxr>=0.5.0 (first cp312-abi3 wheel, covers 3.12-3.14): every supported platform gets wheels, nothing compiles.

…thout librosa These three feature extractors only needed librosa for `librosa.filters.mel(norm="slaney")` (the `mel_filter_bank` TODO: it works in float64, librosa rounds each band to float32). Build the filters natively with librosa's rounding order in a `_slaney_mel_filters` method (parakeet, inherited by nemotron_asr_streaming through modular, and the same lines in cohere_asr, which already duplicated the librosa call), so they no longer require librosa and the slimmer `[audio]` doesn't depend on the AudioProcessor refactor. Bit-exact with librosa 0.11.0: new test compares all three classes' filters with `librosa.filters.mel` over sr in {8k, 16k, 44.1k}, n_fft in {256, 512, 1024}, n_mels in {64, 80, 128}, plus each default config.

`[audio]` now holds what the core audio path needs and nothing else: torchaudio (pipeline / torch-backend resampling), soundfile (decoding) and soxr (resampling for `load_audio` and the numpy backend). phonemizer moves to `[testing]` (so `[dev]` and CI are unchanged). (pyctcdecode / kenlm left with the CTC LM deprecation, librosa in the previous commit.) - `Wav2Vec2PhonemeCTCTokenizer` creates its phonemizer backend on first use, so loading and decoding (inference, ASR pipeline) need neither phonemizer nor espeak-ng; only phonemizing text does. The phonemizer error now also names the `espeak-ng` system dependency. - New `test-audio-extra` job in the nightly extras smoke test: a fresh environment with only `.[audio]`, running `utils/check_audio_extra.py` (WAV/FLAC/MP3 through `load_audio`, `auto` resolves to soundfile, moved-out packages absent, numpy >= 2). - Drop the non-existent `torch-speech` extra from the consistency image.

…float32)` The only difference between `mel_filter_bank(norm="slaney", mel_scale="slaney")` and `librosa.filters.mel(norm="slaney")` is that librosa rounds each band to float32 before the slaney normalization (the frequency grids are bit-identical). Add a `dtype` argument to `mel_filter_bank` (default float64, unchanged), cast the triangular filters to it before the normalization, and use it in parakeet, cohere_asr and nemotron_asr_streaming instead of their per-model `_slaney_mel_filters`. Filters stay bit-exact with librosa. The config-grid bit-exact test moves to `test_audio_utils.py` (6 rates x 4 n_fft x 3 n_mels); the per-model test checks each default config.

Out of scope for this PR (no equivalent exists for other modality extras); can come back as a follow-up covering all of them.

Bit-exactness with librosa is verified once (reproducer in the PR description); keep only a librosa-free test of `mel_filter_bank`'s `dtype` argument.

…er_bank`, soxr only to resample - NemotronH_Omni: direct file-path inputs go through `load_audio`'s default `auto` backend, like `apply_chat_template` already does for the same model, so m4a / mp4 / webm work again when torchcodec is installed. - `mel_filter_bank`: build the FFT grid up to `sampling_rate / 2` instead of `sampling_rate // 2`. No change for even rates; odd rates now get the right Nyquist frequency and are bit-exact with librosa too. - `load_audio_as`: require soxr only when `sampling_rate` is given.

eustlb added this pull request to stack #50618 November 11, 2026 14:46

eustlb commented Nov 11, 2026

View reviewed changes

eustlb marked this pull request as ready for review November 11, 2026 16:36

eustlb commented Nov 11, 2026

View reviewed changes

vasqu approved these changes Nov 11, 2026

View reviewed changes

eustlb force-pushed the drop-librosa-phonemizer branch from 063f9ce to 0a4eebb Compare November 11, 2026 22:11

Base automatically changed from deprecate-ctc-lm to main November 11, 2026 22:23

eustlb added 11 commits November 11, 2026 23:23

Drop the standalone [audio] smoke test job

22d9b90

Out of scope for this PR (no equivalent exists for other modality extras); can come back as a follow-up covering all of them.

Drop the librosa comparison tests for the slaney mel filters

4695350

Bit-exactness with librosa is verified once (reproducer in the PR description); keep only a librosa-free test of `mel_filter_bank`'s `dtype` argument.

Remove redundant mel filter comments

ccf3bc4

address review

48a6d83

keep bytes input for another PR

295ac0d

add require_audio

7297598

vasqu force-pushed the drop-librosa-phonemizer branch from 0a4eebb to 7297598 Compare November 11, 2026 22:23

		`apt install espeak-ng` on Debian/Ubuntu, `brew install espeak-ng` on macOS). Please note that you may need to restart
		your runtime after installation.

		except Exception:
		self.skipTest(f"libsndfile cannot encode {audio_format}")

		@require_soundfile
		@require_soxr

Conversation

eustlb commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Assumption: transformers[audio] installs the audio core, nothing more

🚨 What breaks

Tests

Uh oh!

HuggingFaceDocBuilderDev commented Nov 11, 2026

Uh oh!

eustlb left a comment

Choose a reason for hiding this comment

Uh oh!

eustlb Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eustlb Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eustlb Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eustlb commented Nov 11, 2026 •

edited

Loading

Assumption: `transformers[audio]` installs the audio core, nothing more

eustlb Nov 11, 2026 •

edited

Loading

eustlb Nov 11, 2026 •

edited

Loading

eustlb Nov 11, 2026 •

edited

Loading