Can we use parakeet models?

[nbCloud91]

Thanks for voxtype, I was looking exactly for something like this (there are other solutions but not as a single executable). I want to ask if we can use parakeet models?

[peteonrails]

It doesn't work with parakeet - that's a pretty big lift to make work. I haven't got this work planned. I have mentioned on the website a few alternatives you might want to try

[nbCloud91]

This is really out of scope for this thread. But are you open to packaging it for NixOS? Currently, I am using a personal custom derivation which pulls in the release versions (CPU and Vulkan) and wraps them in the correct env.

I also tried to make a flake.nix for this, the CPU version builds all right but the CUDA is a pain to compile and test, so I have left it for now.

[peteonrails]

I can give it a try. Definitely in favor of making it work on NixOS. Im focused on Arch and Omarchy right now, but should be able to do some more packaging work soon. If someone else brags this, happy to provide support or help.

[peteonrails]

@nbCloud91 I can build both binaries in a NixOS container using the flake.nix on #43 is this what you had in mind?

[nbCloud91]

Wow! That was fast, yeah I had something like that in mind.

[nbCloud91]

For distributing the flake outputs I would like to suggest cachix, they provide a generous free tier for open source projects.

[peteonrails]

For what it's worth, I started a Parakeet spike here: #41

The biggest challenge is that parakeet-rs uses a library that needs to be fixed upstream before I can package it and release it.

[nbCloud91]

Oh that is cool, I have subscribed to the draft. Thanks!

[peteonrails]

@nbCloud91 Until we have a solution for natively connecting to parakeet models, I have a workaround that would let you use a parakeet backend. If you're up for running this project locally to provide a OpenAI compatible interface to the Parakeet models, VoxType 0.4.11 has a remote whisper configuration that can consume it.

https://github.com/pnivek/Parakeet-ASR-FastAPI
or
https://github.com/Shadowfita/parakeet-tdt-0.6b-v2-fastapi

In my preliminary testing over a network connection, it was faster than my local GPU using Whisper models by quite a bit.

Backend	Avg Time	Speedup vs CPU
Parakeet (remote)	~0.28s	~57x
Local Vulkan GPU	~0.74s	~21x
whisper.cpp (remote)	~1.5s	~11x
Local CPU	~16s	1x

[nbCloud91]

Ohh I might test that out on a docker. I also notice a fork which works with parakeet-v3 which improves with multilingual support.
For daily use I will continue with whisper-turbo for now though.

I also saw the flake.nix and the home-manager modules, much appreciated!

[peteonrails]

I'll have a binary for you to try soon!

[peteonrails]

Oh wow - thanks for the pointer to the fork!

[peteonrails]

@nbCloud91 It's a first pass, but if you want to give it a try, it's a start. A little rough, but would be interested to hear if it runs on your GPU and whether it's fast. https://github.com/peteonrails/voxtype/releases/tag/v0.5.0-beta.1

[peteonrails]

Yep, the spacing is completely bonkers and it's my next thing to tackle. I will give it a try with the updated model and let you know what I find. I'm very encouraged that it runs on your machine without a core dump to be honest. That's huge!

Thanks for helping me test it out!

[nbCloud91]

I also see the flake.nix PR, I do not know if you missed the previous message on this, you can push the flake build artifacts to cachix for distribution.

[peteonrails]

Thanks! I'm not super familiar with Nix OS, so I am flying a little bit blind. I'll investigate cachix and a good distribution strategy over the weekend.

[nbCloud91]

Thanks, it is really just a nice-to-have thing rather than anything critical or urgent. Maybe with more people using and contributing, someone with greater familiarity with nix and cachix can help out.

[peteonrails]

I would definitely embrace the help!

[peteonrails]

@nbCloud91 New parakeet build with the TDT model is in prerelease if you want to try it out -- it is WAY better https://github.com/peteonrails/voxtype/releases/tag/v0.5.0-beta.2

[peteonrails]

Voxtype 0.5.0 will include Parakeet support and will support two models:

• parakeet-tdt-0.6b-v3 - 25 European languages, runs around 20x realtime on CPU without acceleration
• parakeet-ctc-0.6b - 80x realtime, but is a character based model that will require post processing

Most users who speak one of the languages covered by the TDT model will want to select it over Whisper, especially if you are doing CPU transcription. On my daily driver, the CPU based Parakeet build runs as fast as the GPU accelerated Whisper build on an AMD Radeon 7800XT.

Release date is not known yet, but likely within a week.

[nbCloud91]

I had downloaded the binary, but I needed to troubleshoot some issues with Tensorrt-fetch failing when building with nix. I finally got it built and patched for NixOS. But then I realized this binary will not run on my CPU:

═══════════════════════════════════════════════════════════════════
FATAL: Illegal CPU instruction (SIGILL)
═══════════════════════════════════════════════════════════════════

Your CPU doesn't support an instruction this binary requires.

This commonly happens when:
• Running in a VM that doesn't expose all host CPU features
• Using the AVX-512 binary on a CPU without AVX-512 support

Solutions:

1. If using voxtype-bin, switch to the AVX2 binary:
   sudo ln -sf /usr/lib/voxtype/voxtype-avx2 /usr/bin/voxtype

2. If running in a VM, enable CPU passthrough or use the AVX2 binary

3. Run 'voxtype setup check' to verify system compatibility

═══════════════════════════════════════════════════════════════════

[peteonrails]

Drat. I built the binary on a Zen5 processor. I'll do a new build on a more compatible CPU tonight and post an update

[peteonrails]

@nbCloud91 https://github.com/peteonrails/voxtype/releases/tag/v0.5.0-beta.4 I'm curious whether the AVX2 binary in the beta.4 release works for you. The Onyx runtime has AVX 512 instructions included in it, but it's should have runtime CPU detection to eliminate crashing. So if the AVX2 binary does crash, then I'll need to compile the Onyx runtime myself, which is a little bit of a lift because I don't know much about it.

[peteonrails]

I have announced Parakeet support, so I am going to close this discussion thread. Thanks for suggesting this feature! I would not have investigated and built it without you bringing it up.

[nbCloud91]

This is great, I have been busy with work so have not had the time to really check this. Thanks for making all the binaries available!