Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

I'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality. I'm running 4-bit quants on an RTX Pro 6000 rented for approximately $1/hour and getting about 1.2 million tokens out and 40 million tokens in per hour with caching. The quality of 4-bit quant is good enough for difficult but well-scoped coding tasks. Here is the inference stack I am using: https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK reply: I just ran a set of benchmarks, ninfer-3090-qwen3.8-27b (so mix of Q4 and Q5) vs strata-qwen3.8-flash-next-iq3_xxs (so Q3):│------------------- │ Ninfer-3090 │ Strata│ Code generation │ 52/78 (66.7%) │ 70/78 (89.7%)│ Code completion │ 40/50 (80.0%) │ 44/50 (88.0%)│ Total------------- │ 92/128 (71.9%) │ 114/128 (89.1%)│ API failures------ │ 10 │ 5- Ninfer generation: ~122 min total.- Strata generation: ~142 min total.So strata is a little slower, but keep in mind that ninfer-3090 is very optimized for a Qwen 3.8....

I've just tested Strata on a simple 50 image vision benchmark. The task is to output the exact coordinates of a requested object. The result via Strata had a median error distance of 154.8 pixels, avg of 168.8. Running the exact same GGUF and vision adapter weights on llama.cpp gives me a median error of 46.5, avg 81.4.To put that into perspective, here are some more numbers from other models via llama.cpp:Median/AverageQwen 3.5 9B BF16: 46.5 / 193.3Qwen 3.6 35B Q4 K XL: 38.4 / 76.4Qwen 3.5 122B Q3 K M: 32.9 / 68.6The difference in vision performance is as large as the jump from a 9B model to a 35B model. All tests were performed at temp=0.I have done no further testing, as these results line up perfectly with my expectations. reply: How big are the images? Is 160px 1%, 10%…?

For what's worth, on high, 27b is better, but both models on high are too slow to be used (way too many output tokens).On low, 3.8 Flash Next is better AND considerably faster.My results[0][1], on a RTX 3090 + 128GB DDR4 RAM. I still have to see if I can optimize any more settings for either, but I think I will switch from 3.8 27b to 3.8 Flash Next for my local LLM uses.[0]: https://aibenchy.com/?q=qwen+3.8+27b%2C+qwen+3.8+next[1]: https://aibenchy.com/compare/qwen-qwen3-8-27b-low/qwen-qwen3... reply: These benchmarks seems to suggest that Flash Next performs best on Low thinking compared to the same model on higher thinking levels?And that Qwen 27b outperforms both when set to high?What quants are being compared here?

This is game changing. My R9700 32GB is now smarter and about 2x faster than using Qwen-3.8-27B. About 60 t/s when combined with my 96GB of DDR4. My motherboard limits me to PCIe Gen3 so that is likely a bottleneck.For the Nix inclined: https://github.com/cjdell/nixos-config/blob/main/hosts/zen3-...Even got it running on the iGPU of a GMKTec M6 Ryzen 6600H at reasonable speed (10 t/s). Fast enough to leave it with a prompt before I go to bed and wake up to a solution.

I tried it and it worked surprisingly well. On my machine (Nvidia 4090, 128GB DDR5, Ryzen 7950x3d) I'm getting 124 tokens per sec, thought to share it here.https://huggingface.co/Qwen/Qwen3.8-Flash-Next reply: Coder version with 30t/sec on a Ryzen 3600x with 48GB of RAM with a nvidia 3080.This is not a very fast desktop. Memory speed is around 2000mhz only. My SSD is some of the worst SSD I've seen and 3080 had its days of glory.I still have code, chromium, librewolf and many other programs running. I have video streams running while I also watch tv and many times youtube videos.I use it with the browser that has a great dashboard and with hermes agent and that it really makes this amazing.Only change I made is to set thinking to low.This is a coding model. Any other task, I still use Ornith 1.5 35B that throws 20t/sec and Laguna.XS-2.0.

I've been working on support for this model in ds4 on the RTX 6000 pro - it's been really great for my use cases. The ds4 q4 quant performs a lot better than other similar sizes that I've seen.Using the Q4 quant on an RTX 6000 Pro Workstation Edition at 450 watts: Code: prefill 1,251 tok/s decode 255.26 tok/s Prose: prefill 1,251 tok/s decode 198.78 tok/s Most important for me, I can run 4 concurrent streams at 400+ tok/s.https://github.com/fairfieldt/ds4 reply: DS4 is an odd model. I have it working on way too many GPUs and yet for many tasks Qwen 3.8 will do much better. It also tends to loop, which is super annoying.GLM5.3 runs on similar hardware and is much better so if you're going to burn cycles and brain power on this maybe look at GLM5.3 as a comparison as well?Other than that, when you're done with that card...

Dwarfstar already supports this, curious how it compares, but I use the q4 quant daily and it works really well.https://github.com/antirez/ds4/blob/main/docs/MODELS.md#qwen...

LLM threads the world over are spammed with Strata links, it remains to be seen how much of the breathless hype remains standing once the honeymoon period is over. I've tried it but so far I have not seen anything that overly impressed me in terms of accuracy, though the speed is definitely there. I'm sure there are applications for LLMs where the quality of the answers is less important but I don't have any of those. YMMV. reply: What does this inference engine do that others don't?

Qwen 3.8 Flash Next with mtplx on 128GB M5 Max is amazing, full context I'm getting around 72tok/s, it has replaced the frontier models for me.

Why isn't this type of expert caching in the native llama.cpp yet? Why do we need a separate codebase?

We're getting closer and closer to the day we can have an Opus-like model running locally. The dream!

More great work on local model but you’re still losing a lot. Down to 2 bit quantization and the coder model throws away half the MoE experts. In a world where anything is better than nothing, this is a net win. But we have a way to go still.

All headlines about LLM performance MUST have the quantization also mentioned in the headline.You know, so you're not wasting your time like in this post.

Q2 quantization. Not interested.

Currently I am running llama-cpp with `Qwen3.8-Flash-Next-UD-IQ3_XXS` on an old ryzen 8845HS with 96G of ram (and no dedicated graphics card) at 7tk/s and ~60tk/s filling, max ~120K context window.Surprisingly useful as long as you can leave it running a couple of hours at the very least.While huge models will still be better I think the general availability of RAM might be the downfall of AI companies.

I've been playing with this on a 3090 and it FLIES. Does a pretty good job too on the tasks I've thrown at it (php code base security audits).

There are so many AI generated inference engine for local models now, each of them are generally narrower but they are all faster than llama.cpp. Maybe llama.cpp needs to rethink their strategies...

I'm giving this a go, and so far, this is great. I have 1x RTX 4090 (24GB VRAM) and 128GB DDR4 RAM. I am seeing > 110 tokens/sec (3 token MTP). Using 60K/260k context currently. So far the results seem at least on par with my Qwen 3.8 q5 27B (~60 TPS). I realize ultimately tokens/sec don't really mean much if the quality sucks, but I am optimistic but still paying close attention to the results.Working with fast local models can be great. Fast prefill, and >100 TPS is quite quick.

I don't get it. It's file size is about 6 times larger than 27B model for the same quant, but the performance improvement is hardly 10% across all benchmarks, according the metrics on it's hf page. Why should one devote so much more hardware for so little benefit?

I wonder if a higher Qwen3.8-27b quant could beat or match these lower https://huggingface.co/IsValorum/Qwen3.8-35B-A3B-Distill-MLX...

100 tok/s on a 4090 could unlock entirely new use cases. What becomes possible when local inference is this fast?

- Tiny context size or hours to load it- Hard to benefit from thinking and preserve thinking given token cost.- Low quants reduce accuracy heavMTP draft can make it make the same mistakes all the time when calling tools, formatting output or following basic guidelines. Otherwise 2x-4x slower.- K/V quants probably quantized too make things less accurate.Useful would be combinations with:- Full context size so it can code and think a bit.- Draft MTP =92 GiB RAM.Source: I have low specs and tried them all for agentic use + coding.

My goto private setup, runs ~50t/sex on a dgx spark with sglang, nvfp4. Excellent model.

Thanks, trying it now. On my 3090 it seems to run at around 60tps, the IQ3_S variant. I am testing it now to see if it is better than Qwen 3.8 27b

I had this working with the FreeToken inference engine a month ago when they launched.https://github.com/FlashML-org/FreeToken

>> The model is a team of 24,576 small specialists ("experts")That's a neat number (576 is the square of 24). Ofcourse it must have come from 24 * 2^10.

Haven't had time to do much quality testing but the IQ2 model is running at 65 tps on a 9070xt/5900x. That is wild.

Anyone know how it compares to GLM 5.3 for real world use?