AI Models Can Provably Hide Arbitrary Capabilities

More clear title, given how people talk about AI:"Capabilities can be hidden in AI model weights by encryption".It's not about AI agents being able to hide what they can do.

I'd be interested to know whether it's possible to contaminate public data to make trained models do this.Something like a commonly repeating conditional in the public domain embedding itself in the model's logic.

This has always been my worry with open source models... give them away for free, but with a sleeper agent built in that some arbitrary string can activate. Local deployments will still have web search connectors the majority of the time. reply: That worry is just as much or more with closed-source models.

This kind of possibility was the subject of some discussion on r/localllama the other day: https://www.reddit.com/r/LocalLLaMA/comments/1wxvt41/my_qwen...(in that case it seems likely to just be an overfit to the training environment, imo)

Frontier AI co warns that their model is super dangerous and needs more safeguards because it can break out on its own.Model can't actually break out on its own.Frontier AI co manually codes in one-off helpers for the model to break out "on rails.""See? We told you it was super dangerous."

I wonder if this is going to create more market demand for truly open source models, where everything including the dataset is inspectable. It definitely seems like good news for companies like Mistral that want you to build on their models but they're not close to the frontier.

Only a matter of time that this research became public.LLMs were always a great way of exfiltrating targeted data - it’s basically an encrypted malicious payload, and doesn’t “decrypt” until it finds specific inputs reply: We went from having a panic over binary blobs after the xz crisis to everyone installing giant binary blobs on their computers and giving them access to their filesystems and online accounts.Was research, strictly speaking, needed?

AI models seem like data vacuum cleaners. They crave it more than anything, especially specialized contexts. I wouldn't be surprised if you see hidden goals activate when niche context shows up. Maybe a benchmark asking them to brainstorm in niche fields will lead someone to the trojan horse goals.

Very cool that they can demonstrate it but it also follows from Rice's theorem. Any sufficiently expressive computational formalism has non-trivial runtime properties/effects & these can not be detected w/ static analysis so you have to run the program to see what happens.

I mean, this is a cool demo, but that's about it. It shows that you can build a model that holds secrets which are genuinely encrypted until the model gets the correct input. But it's also trivially detectable, so the framing seems a bit misleading.

how can you prove that someone is hiding something? if you prove then were they really hiding?