Nvidia Nemotron 3.5 Lightning and NeMo Switchyard

Coincidentally I've been playing with small (~30B) self-hostable models for coding tasks today -- specifically plugging them into Cloudflare OS (which I work on) and asking each to build a collaborative whiteboard.I'm finding that the Mixture-of-Experts (MoE) models (Qwen 3.6-35B, and Nemotron 3.5 Lightning) are, well, terrible at this. They just couldn't get the job done at all. Went way off the rails. They are really fast though!Whereas ~30B dense models (not MoE) are pretty decent. I tried Muse Glimmer, Gemma 4-31B, Qwen 3.6-27B, and Laguna XS[0]. They were all able to build a working collaborative whiteboard app, without any guidance (other than feeding back error logs to the model). I also asked each to then draw a monkey by calling the API of the whiteboard it has just built. Laguna drew random scribbles but the rest all managed to produce something monkey-like.(Frontier models in comparison will write the app in one shot with no errors at all.)Note that both Qwen 3.6 and Gemma 4 each have both MoE and dense variants. I find this very confusing, because e.g. ollama's model index typically only distinguishes variants by their size, but MoE vs.... reply:...

One major consequence of the ramapocalypse, I think, is an even higher focus on small efficient models. I personally believe that the multi-trillion parameter models are fundamentally missing things and the push to smaller, more efficient will drive evolutionary structural changes that will lead to future gains reply: I'd assume the closed weight models are all working on shrinking their parameter counts anyways. They too benefit from smaller models. It'd be foolish for these SOTA labs to not be working at reducing parameter counts.

> NeMo Switchyard, an open source library for smart routing> When deployed, NeMo Switchyard can intelligently direct each request to the most capable and suitable model for the jobHow do routers like this handle prompt caching when you send the second request?Sticky models per session? but then the second message of that session won't be sent to a suitable model, and will only be sent to the same model as previous one. reply: I've seen ones that are configurable to pick a trade off point between lower cost (cache stickiness) and routing performance (best model for that turn).But yeah I'm skeptical all this overhead is worth it.

They conveniently decided not to include the Qwen range of models in the Artificial Analysis graph, except the out-of-league Max variant. At least be brave and honest.

I would like to propose something:- problem: massive deluge of information because of AI- solution: human beings should adopt a minimalist style of communicating in writing.- e.g. this entire website page can be ten bullet points. reply: Why use more words when few do trick?

24 comments so far about Nemotron on this earlier submission: https://news.ycombinator.com/item?id=49257947

I love the wave of new small model releases. Pleasantly surprising that an NVIDIA model runs so well on Apple Silicon using MLX! I was using nemotron-3.5-lightning:30b-mlx with OpenCode on my old (cheap) Mac this morning and no bad experiences except for running slowly.

Nemotron 3.5 Lightning runs on how little GPU vram? Can q4 run on 16gb? reply: Not by the looks of it, but it got me thinking, currently in the middle of Level1Techs coverage on the model and switchyard and he mentions "how easy it is to customize it". Fully admitting that I haven't yet read the docs, my issue with that is "we can train LORAs for 35B as well, why use this (according to benchmarks) worse model for customization instead of a slightly bigger better one?"Assuming I eat my words after going through the docs and this is actually a more efficient model / loras adapt better, I don't see as much value in it as is, as a REAP of it (remove least-important experts, domain-locked tests show ~98% retained accuracy) to something like 20B-A3B (rouhgly matching gpt oss, which while a good model, is outdated knowledge-wise and not as good with tool in my xp).Having a 20B-A3B model at q4 that has a lora to be your local orchestrator (delegating coding to server/cloud models) and ci/cd runner does start sounding like an appealing proposition to me, as that would fit in 16gb vram easily (fitting many consumer gpus and 24gb macs).

Good luck getting it to run with NVFP4 on a DGX Spark, the very architecture Nvidia created that format for.

The new Meta 30B models seems A LOT better:https://aibenchy.com/compare/meta-muse-glimmer-30b-xhigh/nvi...