Compression is prediction

This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University.> Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine learning still belong together. Brains are the ultimate compression and communication systems. And the state-of-the-art algorithms for both data compression and error-correcting codes use the same tools as machine learning.Book (creative commons): https://www.inference.org.uk/mackay/itila/book.htmlLectures: https://m.youtube.com/playlist?list=PLruBu5BI5n4aFpG32iMbdWo... reply: I had a long ranting comment I deleted....

Grant Sanderson has an excellent video on the same topic [0]. It's part of a series that is ongoing.[0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98 reply: I wonder if the author of the article knew about the series, or do they both just independently came across this topic to talk about it.

Nope; there is a bit more nuance and the distinction is important.Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems. The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support! Eg: you observe a rare edge case in your training data and (lossy) compression could simply ignore it. But if you wanted generalization in that particular part of the space -- either because an adversary was testing you, or for design freedom where you choose to build in that specific corner -- then you don't just want data compression, but good prediction performance on a test distribution which peaks in that corner.Assuming that the training data distribution is exactly the distribution you will ever care for is implicitly doing a lot of the heavy lifting in the claim that compression = prediction, and I'm peeved at how much this statement is unthinkingly repeated like a manifesto.There is nothing natural about the training data distribution, especially if the data generation process is exploratory while the downstream usage will be ex...

I think a better headline would be something like: Compression is Abstraction and Decompression is Extrapolation.Many of the debates in the comments seem to come down to whether people believe prediction and extrapolation are synonymous.

There is Compression done by Prediction by partial matching [0]There is the Kolmogorov Complexity [1], Normalized Information Distance [2] and Normalized compression distance [3] that correlates those.Finally, there's the Pre-Big Bang Informational Compression and the Delayed Release of Antimatter [4]All big {rabbit/black} holes to lose some time, if you have any.[0] https://en.wikipedia.org/wiki/Prediction_by_partial_matching[1] https://en.wikipedia.org/wiki/Kolmogorov_complexity[2] https://homepages.cwi.nl/~paulv/papers/chapter08.pdf[3] https://en.wikipedia.org/wiki/Normalized_compression_distanc...[4] https://philarchive.org/rec/GREPBI

This perspective is a useful source of intuition against the “LLMs can’t have new ideas, they’re just next-token-predictors” style arguments. What if you shift your perspective to thinking of training as optimization over a vast parametrized family of compression algorithms? Well, it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process! reply: > it suddenly looks a lot more plausible that “new” “ideas” can emerge from that processThis is not intuitive to me. It seems like a "new idea" is something that (almost by definition) isn't in the training set. Can you elaborate a bit?Edit: but perhaps a good model could arise from training, which would be a good idea in the sense that parsimonious ideas are good scientific ideas.

This is a lot less surprising when you learn how non-LZ compressors work, that is, by modeling a probability distribution and using those probabilities to encode information in the minimum number of bits required to transmit the data. A less obvious conclusion is that LZ compressors do this to implicitly, the length of each symbol they could emit (literal or match, etc.) can be converted to the probability distribution the LZ compressor induces, since the number of bits to encode the symbol is related to its probability by the information content. reply: A common design in compressors is to use LZ as a first step, but to then represent the constant data and/or offset-length pairs from LZ using an entropy coder.Deflate (as used in gzip) uses a Huffman coder. LZMA (as used by xz) uses a predictive range coder. Zstandard can use either Huffman or FSE. Some high-speed compressors like LZ4 skip the entropy coding stage entirely at the expense of compression ratio.Bzip2 is an interesting aversion of this pattern - it uses the Burrows-Wheeler transform as a first pass instead of LZ. Unfortunately, this is one of the major reasons why it's so slow.

I stumbled across a connection between LLMs and compression when researching N-dim polytope emergence in neural networks. Toy Models of Superposition (Anthropic, 2022) suggests that gradient descent can independently discover efficient geometric packing arrangements for sparse features. LVQ compression uses regular lattice structures, including some based on 4D lattices.I found this interesting and wonder whether LLMs have a higher density ceiling, since training and inference don't rely on a fixed lattice and can instead learn their own representational geometry.

Ted Chiang made a similar point in his article "ChatGPT is a blurry JPEG of the web" a few years ago:https://www.newyorker.com/tech/annals-of-technology/chatgpt-...

Cool visuals and breakdown. I wrote something in early 2025 about how LLMs seem to be an emergent behavior of lossy compression, but did not have the knowledge or verbiage at the time to get this detailed. In retrospect my writing seems naive and I'm happy to have found this and the Google paper linked inside. To be a fly on the wall in some of the labs, man.Another thought that came from the same post is that, insofar as we see LLMs as human-style intelligence, they're more like stream of consciousness devices. Essentially incessant talking and buying enough time until you get to a usable answer. I think I associate some subset of intelligence with what you don't say, which is impossible with the SOC-style outputs, so this is something I think about a fair bit.What could maybe differentiate current gen models from next gen is the ability to call tools modeled within the layers themselves, not externally. I think as far as I understand it, model trainers expect the model to do this itself in a way we don't understand or control, like a version of the bitter lesson....

great talk by ilya sutskever on how this https://www.youtube.com/watch?v=AKMuA_TVz3A&t=2121s

The comparison can be carried on to another even crazier level: Evolution is compression. All the complexity of biology is executed at the highest possible efficiency.

This is simply wrong.Compression requires prediction.The better the prediction, the better the compression, whether you are measuring fidelity or result size.This doesn't mean that compression is prediction.

I was thinking about the same topic and the conclusion can be wrong. LLMs are compressors, but compressors are not LLMs. Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot.Specifically I was thinking about a way to inject knowledge into LLMs training by using statistical properties of text in such a way that you don't have to train the LLM to achieve some level of predictions. There are actually some papers that inject n-grams statistics as a part of the neural network weights.

Small world. I just did a podcast on this same topic, but coming at it from a different direction, ie. me and my neighbor trying to beat the hutter prize for compression.Hutter Prize being where you are paid if you can compress wikipedia small enough. LLMs do very well at that, if, big if, you ignore the cost of initial weights.A cool Claude Shannon story: Shannon wanted to measure how much information is actually contained in ordinary English text. His 1948 theory said such a number must exist, but he had no way to calculate it, because the patterns in English reach across dozens of letters and no equation or frequency table captures all of them at once. So instead of calculating it, he ran an experiment on a person. He took a passage from a novel that the subject had not read, and covered it with a card so only the text already guessed was visible. He asked the subject to name the first letter. If the guess was wrong, he asked again, and kept asking until the subject named the correct letter. He wrote down how many guesses it had taken, revealed the letter, and moved the card one position to the right. Then he repeated...

Compression is not prediction, it is recall. Can we make predictions based on compression? Absolutely. Is memory encoded into physical neurons technically compression? I would argue also yes.However, going from compression to prediction is a large jump that is unsubstantiated by this article and based on the claim that probabilistic recall is also prediction.Two perfect counterpoints to this are markets and weather patterns. One cannot predict future events based on past performance or behavior. Change is the only thing that's constant, and chaos/entropy is everywhere we look.For simple problems like programming, sure predictive recall works amazingly well, but let's not pretend LLMs are actually predicting something. This is exactly why LLMs suck at doing anything novel; they lack imagination and creativity.

Schmidhuber did it first:Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokeshttps://arxiv.org/abs/0812.4360

It's more or less obvious that the LLM is a lossy-compressed version of the training data; it reproduces sequences of tokens that are the sort of thing that could plausibly occur in the training data, and avoids sequences that are implausible.Because most of the training data has good grammar, the LLM is strongly trained on grammar; it will rarely predict ungrammatical gibberish. Even if there are grammar mistakes in the data, they are not systematic and so don't reinforce each other.

Something Ngrok is doing is working, because first they manage to get Sam Rose of samwho.com and now this? At this point I care more about their blog than their products

I always feel like people leave out the third case of the analogy: indexingThe article itself has decision trees for the compression explanation, which is also a lookup index.In each case you try to recognise (re)usable structure.Self-indexing succinct data-structures are a good example of the third side of the coin.So it's a trinity: compression, prediction, indexing

> compressors and LLMsWhy only LLMs? All statistical models are compressor. You can say "model" and "compressor" are synonyms.Article does not mention "embeddings" at all, even though it's commonly viewed as a compression method. Also "encoder" part on "auto-encoders".

This immediately reminded me of the Hutter Prize (http://prize.hutter1.net/) - a contest that has run since 2005(?) based on the premise that compression is closely related to intelligence.

Unrelated to the content: I was really pleased to see that this site defaults to the bare minimum for cookie consent. I reflexively clicked "Reject all" only to see that it was already the default, which threw me off.

I keep on seeing this claim, especially from popular creators such as 3Blue1Brown. How is this not borderline vacuous?I'm not a LessWrong^TM rationalist guy, but one really good thought experiment I always keep in the back of my mind from them is Solomonoff induction. AIT people take it as a framework to work with - it's pretty cool, I agree. But I (and some other people, such as certain AI execs at Amazon - according to my interpretation of their public interviews) think it just highlights the trap - given an arbitrarily powerful oracle, you can get compression down pat. Like, if you assume the source is generatable with a turing machine, and you write a function to brute force over all turing machines, then whoa, your compression works....

I see it that prediction is a form of compression.Say you have a computer file composed of two parts, the first represents the setup of an experiment, and the second is the data produced by the experiment.If you have a good theory relating to this type of experiment, then you can predict much of the second part of the file. So you only need to store the first part and possibly some corrections to the least significant bits of some of the parts of the second part of the file.Thus with good prediction, you can compress this type of file.

How about dictionary based compression as a counter example? Or the zig zag encoding scheme used in JPEG? I find it difficult to cast some of the things that effectively compress data as prediction.

i do agree with that point of view. I often referred to models as 'modern mp3s' storing a lossfull but lookalike version of information in order to counter that 'AI is totally new and not violating copyright by storing information in a magic fashion' argument.

Author's bio:> Annie Sexton is a Developer Educator at ngrok with a passion for nerd-sniping developers.