Whistle: Speech to Text in 16.9 MB

Interesting that this is here. I used whistle (and bunch of other things) to take ownership of my echo show. It now doesn't dial to Amazon at all - it does all processing locally with its own CPU and connects to my homeassistant for home automation. My initial setup involved qwen asr (1.7b model) running on rtx 5080. Compared to that, whistle was really bad (out of 170 messages, qwen recognized correctly 168, whistle - 70), but I adjusted whistle to work like jev - instead of free form transcription it recognizes only select set of templates (I trained tiny network with 10,000 generated utterances to translate whistle final state to probabilities within templates). The precision went up to 164/170 - almost matching qwen. By the way - I'm speaking with heavy accent. reply: This feels like we're going back to the past again with CMU's Sphinx4 in Java. It worked way better than I would ever have expected it to for being more than a decade old. It relied on the user defining a grammar of valid words and different flows through a standard format (Java Speech API Grammar Format). I wonder if we'll approach that again for these models just like how MCPs act like WSDLs in spirit....

I don't think the challenge with speech to text was size of the binary. In my experience the challenge is understanding my 84 year old Croatian father with a sagging mouth after a stroke, when he's trying to write his autobiography.I just setup Windows speech to text for him last week and it's great to see how he can write an entire page in 10 minutes, it would take him days using the keyboard.But every single sound he makes with his mouth ends up on the page too. reply: Sorry about your father. He needs a dictation model, not a general purpose speech-to-text model. They ignore umms and ahhs, change things like “an elephant, no a monkey, went up the tree” to “a monkey went up the tree,” support saying punctuation aloud sometimes, etc.Gemini team just released Gemini 3.5 Transcribe that’s supposed to be good at this; it’s available via api: https://blog.google/innovation-and-ai/models-and-research/ge...

What the demo does not do is show streaming output of transcribed text as we are speaking and recording (before we hit stop). That is an essential feature IMO for most general purpose live STT apps. reply: What do streaming implementations do when a bigger context reveals a different interpretation/parse? When I use Whisper in the terminal, I can see it going back and correcting itself. Are corrections off the table for a true streaming transcription?

How does this compare with Parakeet? I've been using that locally in my projects on an M-series macbook and it's been working great. It's fast and accurate enough for my use cases (meeting transcription, audio transcription for demo videos, etc.)This definitely seems lighter and faster. How does accuracy compare? reply: People keep praising Parakeet, but I've found it to be worse than Whisper Large. Yes, it is much, much faster and smaller, but accuracy matters a lot if you are to use dictation regularly and seriously.I ended up having AI optimize Whisper Large and create a plugin for TypeWhisper, and that's what I use (feeding the results through local Qwen 3.8 running under MTPLX).

Tried it on a random TV episode and it seems to get stuck sometimes where it just outputs "Thank you." as a default - at one point emitting that for 60s of dialogue (and no, the episode does not have someone repeating "Thank you." for 60s.) Happens several times during the transcription. reply: It's funny how so many models tend to generate "Thank you. Don't forget to subscribe" or "Thanks for watching" if there is silence. Shows you what they've been trained on :-)

I recently built a 3D printed ESP32 based transcription device[1] that offloads transcription to Parakeet running in the browser connected to the ESP32. I haven't done my own measurements of error rate but it'd be interesting to see how well this runs on the ESP32 itself.In my case I'm already using one core to run DSP for a beamforming mic array (which works amazingly well for noise cancellation!) so I don't have huge amounts of free processing though.[1] https://github.com/nlothian/earwing

Wow certainly in English this is incredibly accurate I tried to break it and it understood me perfectly!I know it's slightly off topic but surely it must be easy by now to train a spell checker that doesn't annoy the crap out of everyone using it (looking at you here Apple)!

Error rate is crazy high. And comparing to low models and omit a lot of other bigger open models.

Are there such STT models that allow ‘context’ to be provided? I’d love to be able to dictate for programming sessions. The problem is that STT models don’t handle jargon and acronyms well, especially if they’re specific to the project or company. Is there like a type of model you can provide a custom vocabulary to without needing training?

I initially read this as whistle to text, which would be way cooler.

Apple needs to incorporate this into iOS ASAP. This works much better than the speech recognition in iOS when you use technical terms. One of the most frustrating parts of iOS is speech-to-text in iMessage. For me no feature is more important in a phone.Try this example: My website uses ASP.NET technology and I am using .NET 10.0. Works perfectly in Whistle, but not in iOS.

Did anyone ever read The First Circle by Solzhenitsyn? It is amazing: in a Stalin gulag in the 1950's USSR, a bunch of Soviet engineers / techies are tasked with creating a speech recognition system. They show fake progress in order to keep from being jailed in worse ways. It is autobiographical, so good, but I read it so long ago and it convinced me how hard speech recognition would be, maybe even impossible!Highly recommended. I see there is a "new" (2009, I getting old) edition & translation - https://www.amazon.com/First-Circle-Aleksandr-I-Solzhenitsyn...

Let's say I want to build a hardware product now, voice-controlled, with voice feedback, so STT, LLM, and TTS. All local. What are the best libraries to do this now, say with 8GB of GPU memory available?

I really hope there's a TTS model in 16.9MBKokoro is good, but can't change the pitch easily.

Can't even be bothered find a human to write your model announcement article huh.

Might look into embedding Whistle into Whistle if it can make it 30x smaller (https://play.google.com/store/apps/details?id=com.blazingban...) - It's a shame there isn't as many languages supported though, I'm surprised at the amount of non-english downloads (I really shouldn't be, of course non-english speakers want dictation) of the app there is.

If you need speech2text try this, https://ccoreilly.github.io/vosk-browser/

Thanks for putting in the work. Only a matter of time before all of this runs locally at very high accuracy. And can handle complex vocabularies.

Can it run on any of the cheap uC / esp / stm ?

The Qwen STT model that's leading the open weight leaderboard right now is excellent. Parakeet v2 is blazing fast and accurate enough on English. I feel like STT gains from this point on will be marginal, especially given that you can do a quick LLM pass afterwards with a small model

Hm. I saw language=detect and tried some Japanese - which (given the actual list of supported languages) unsurprisingly turned into some mangled Spanish.Since it doesn't support Norwegian - I tried English - and it mis-transcribed "cleaning" for "training" - probably a failure due to context/training (Hello everyone, today we are going to do some cleaning).So, reasonable, but limited?

For something you could plausibly ship inside a webapp, it’s very good. I can definitely think of some cool uses for this.

Those are insane benchmarks at this size. Well done!

Sooo I haven't really been super impressed with the needle models before, but this is very impressive. It transcribed multiple sentences I gave it with complex timing and words and in such a small footprint, I'm super impressed. Excited to see what types of things can be built with something like this, the performance seems very good.

ZWhispr is where its at https://zwhispr.com/

Huh, this was really confusing. I already had an STT app called Whistle on my Android phone.

ZWhispr seems to be the best for those who care about accuracy as it uses three SOTA models.https://zwhispr.com/

This is actually a really great release. Congratulations team. I just tried few words. My Indian accent also was able to pick up.I'm gonna run it on my Linux Box.