Dust: Pretraining Transformers Without Backpropagation

Every few years, a derivative-free neural network optimization algorithm gets some hype. I'd bet my life savings that none of them ever make an impact.Derivative-free optimization can be useful for genuinely discontinuous objectives [1], but common neural network objectives are smooth and/or Lipschitz.The gradient is useful. Instead of trying random directions and hoping that one of them is an improvement, it tells you where to go. The more parameters you have, the more useful it becomes.A strict complexity gap between gradient-based and derivative-free Lipschitz convex optimization has been suspected for decades and recently proved (using AI, [2]). Neural net optimization is nonconvex, but not radically different.IMO, a more promising direction is gradient-based optimizers specialized to the neural network structure, like Muon [3].[1] https://arxiv.org/abs/2202.00817[2] https://arxiv.org/abs/2607.13335[3] https://jeremybernste.in/writing/deriving-muon reply: Random selection seems to favor post traininghttps://arxiv.org/pdf/2603.12228

> There are many interesting open questions. The first is whether, and how, Dust can find better directions than backprop’s first-order gradientBoth algorithms are bound by the same Pareto frontier based on the Empirical Risk Minimisation Principle, so they’re already on the same trajectory. Interestingly backprop is limited by conditioning of the Hessian matrix in order to converge (differentiate correctly). So removing this limitation is actually a great step. I’m excited to see a comeback of evolutionary methods because they’re much more general, albeit costly and naive. We’re now very close to what can be described best as brute forcing the Pareto frontier out of our datasets. Not sure that’s what we want but I have no better ideas either. reply: Pareto frontier the Pareto frontier and then with a little Pareto frontier you might get to the Pareto frontier

It's computationally expensive and infeasible but what's the upside?Genuine question due to unfamiliarity with the subject.

Even though this is way more expensive than backprop, could a hybrid approach where you fine tune an existing checkpoint that's been backpropped unlock further gains? It would be cool to apply this to different stages and see if that affects the learning trajectory

Hmm I was more interested in this alternative to backprop:https://news.ycombinator.com/item?id=49701182

This is super yawn-worthy. Instead of backprop for the exact gradient you can run forward passes a thousand times with perturbed weights and get a Monte Carlo estimate of the gradient. Not very clever. Extremely NOT useful.But I guess the industry is littered with techniques for computing the same thing but vastly slower that some people find interesting. Homomorphic encryption. Zero knowledge proofs. Blockchain computing. Except in those cases there might be a legitimate reason to use it occasionally.

It sounds like this is less computationally efficient than backprop, but more easily parallelizable. Is that fair? reply: Not necessarily, backprop is highly parallelizable since it is just a bunch of matrix mults.Something like Dust skips the backward pass on backprop. But other techniques like Neural Predictive Coding can be completely asynchronous, each "weight" can fire independent of those far away from it. Innocenti, et. al have shown that NPC gradients converge to backprop within a certain "regime".The win with asynchronous techniques like NPC is that you do not need the extreme co-ordination that backprop requires and hence should be computationally much easier given the right device.Although at this point the industry has so much money in the forward-backward pass system that I doubt a backprop successor would win unless someone makes NPC hardware feasible and can prove scaling up to billions of params

Definitely more than meets the eye.

Dust's 243M model beating a 120x smaller one at most population sizes is the surprising part; bigger nets got more population-efficient, not less.