Going Dark, and the era of law enforcement hacking

It's worth realizing that, before computerized central offices, telephone wiretapping required running physical wires. Back when Rudi Giuliani was prosecuting organized time, not only did physical wires have to be run, the cops were billed for them as expensive private lines. His task force was spending about a million dollars a year with New York Telephone on wiretapping. In one case, law enforcement didn't pay their bill, resulting in the person being wiretapped having the wiretap connection show up on their bill, blowing the case.That resulted in the Communications Assistance to Law Enforcement Act, which mandated that central offices offer remote wiretapping. Capacity up to 1% of lines is required.Back in the electromechanical era, the only call data that could be collected was outgoing dial pulses, using a "pen register".[1] (The one shown in Wikipedia is mine. It's a beautiful piece of antique brass telegraph technology. It records dial pulses as dashes, and has to be wound up like a clock, with a big brass key.) The Supreme Court decision allowing "pen registers" without a warrant refers to these "extremely limited" devices.... reply:...

> In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon.This doesn't resonate with me. I see companies adding more sloppily written features with AI. I see more bugs in the software I use, not less. While it's plausible that software is getting both buggier and more secure, I suspect those two move in the same direction not opposite.My guess is that we're getting better at finding _existing_ security issues with AI (and thus fixing those issues), but simultaneously adding more insecure surface areas _at a faster rate_. reply: One way to resolve the tension here is to note that CNE and lawful-intercept access to phones depends generally on platform vulnerabilities, not application code vulnerabilities....

I've always loved the ridiculousness of the "going dark" label when law enforcement can't access encrypted chats or a back door isn't built into a piece of software. When there are security cameras on the vast majority of houses, stop lights and in people's hands, and when so much meta data about people's associations are shared from Google, Facebook, any other social platform, how in the world can they say they are "going dark". How did they ever solve crimes before these things?

On one side, you have pieces like this, where seemingly there are constant fights between serious actors with large and properly distributed budgets, employing top tech and top minds; on the other - regular news of the hackz, where responsible person in charge of security with root access failed to grasp basic technical knowledge (several times), ticking every checkbox in "never do this" list from security best practices, which led to every customer being pwned.It's like two parallel worlds, that exist in the same place at the same time, but somehow don't cross. reply: I started my career in the military, and got lucky enough that SOMEHOW, we convinced them to fund a trip to GDQ for educational purposes.Anyways, while there I attended a little roundtable on software security. It was me, representing a small unit from the Air Force, some dude from Google, and like 15 game devs.Despite only being a dev of 5 years at that point, I was SHOCKED at the lack of knowledge on software security....

> In fact, the worst part about this dynamic is that these potential new backdoors will begin primarily useful for allowing the US to weaken its own systems, which will in turn allow foreign adversaries to find new ways to attack our communications. This deliberate self-sabotage will happen just at a moment when we’re finally learning how to defend our own infrastructure.I don't understand how you can both argue for that law enforcement (and intelligence) agencies will force others to implement intentional backdoors AND also everyone will be using AI to find and secure ALL potential holes in the software so there won't be any vulnerabilities anymore.Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention?I have respect for the author so I feel like I probably misunderstand something from the overall text rather than I somehow have a better perspective on this topic that the author knows very much more about than me. I felt like I nodded along all up until "So how is this a problem?" and now I'm not sure I understood correctly. reply:...

I'm supposed to be concerned that the US government and Israel won't be able to hack everyone's phones? reply: The reason why this might worry people who don't like the US/UK/Israeli governments is hidden in a secret place...a paragraph that is neither the first paragraph nor the last.

I don't think the thesis that a government will be able to do something will ultimately hold. I don't see how they can avoid "going dark" in a democracy.we live in a world where the government can't even do much about illegal drug markets anyone can access by downloading a piece of software.if they pass laws that mandate backdoor access and block software which doesn't conform more and more people will move to the dark networks.and if they effectively block the dark networks (in the limit they will have to block all encrypted communications) then we will be living in a tyranny.freedom is messy. accept that digital crime can only be solved when the criminal makes a tangible mistake. LLM's will be building profiles on criminals to help with identifying mistakes. reply: The problem that Matthew Green is calling out subtextually is that democracy is likely to disappoint nerds on this issue: a random voter in the country 10 years from now may very well not share your priors about this issue.

> This is not a call to action for experts to rally behind a sophisticated plan. Like so many things about the AI revolution, it’s just occurring to me that we’re on a long greasy slide to a place that will look different than where we are today.heh. long greasy slide. It really does feel like that.

i wonder how many open source projects have alphabet boys building trust as contributors for eventual backdoor planting

Considering from a point of view stipulating that perfectly secure software is possible. I don't think it follows that the limititation that it would place on intelligence is necessarily a net loss.Apart from the obvious harms of invasion of privacy, and fishing expeditions being biased to the places you decided to fish. There is the simple fact that data can be misleading, especially without context. An interceped communication is a piece of data that is intrinsically tied to the trust of the inteceptor. A few people with an agenda can collaborate to create a seeming truth by 'discovering' the same thing from different sources.Requiring warrants compelling information holders to provide data, not only serves the task of protection from abuse but also create a record of provenance that can be verified.It also provides a degree of symmetry in capabilities which discourages actions that one party may do over another if they are motivated to act because they have a temporary advantage over another.

It breaks my heart that the governments with unlimited budgets who have hired the best and brightest will have to put in serious effort to get the bad guys, and potentially find it not worth it to casually spy on the whole world.I am just beside myself at such an idea that people looking to feed the prison machine cannot as easily find excuses to turn normal citizens into prison feed.Just super sad guys.

I dunno man, if there's a deluge of new AI generated code at all layers of the stack I think there will still be vulnerabilities.Like if we were willing to stop adding new code and just have a small secure code base, AI could maybe help us find all the vulnerabilities in that code base.But people have consistently been unwilling to do that. Like if we were willing to stop adding code we could have stopped decades ago and done SQLite level testing everywhere and probably have found almost all the bugs already.

No system view. The law agencies can develop exploits to intercept our phones, that is a new, and totally unseen before threat.Unless you remember 2013, Snowden, that nothing was done (at most was some concern about doing it to US citizens, the rest of the world doesn't deserve privacy), all US (and/or five-eyes) based web companies must disclose users information and be forced to not disclose that, and things kept going surely at a faster and more intrusive rate in everything else, and of course phones.You are complaining being sprinkled by water while at the bottom of the ocean. At least the big companies can find their own vulnerabilities with the AI tools you mention, the rest of the doors are still wide open.

I would like to point out that the last year when not a single law enforcement agency anywhere in the world could have possibly tapped anyone's phone was 1876.150 years ago was the invention of the telephone, and I think that articles like this seem to assume that prior to this, police just never caught any criminals.

> In this case, we’re just going to have to hope that this time we make the right choices, for no other reason than that they’re right.I'm more curious what could be a right choice, and more importantly who is the "we" in this, as many decisions are largely made by companies and governments.

Man I thought from the title this was going to be about next-gen AI being able to zero day everything so effectively that software security is meaningless and we'd need to basically shut it all down, go dark.

Sounds like a pretty strong argument to self host, and otherwise be in charge of the software you use.

I think that one of the reasons they (frontier companies and the gov) will be putting so much effort into curtailing bugs and vulnerabilities is to limit the blast radius of future AI models. Imagine with the new Sol Ultrafast, they could have pwned Hugging Face in 6 hours and not 4 days (IIRC).It also seems likely to me that the US Gov. probably already has routine mechanisms for compelling targeted software updates for persons of interest, so I'm not sure that a more formalized backdoor than automatic updates is going to be surfaced in the mainstream, unless that is avenue is also cut down somehow.

Governments and their agencies shouldn't have any access to the communications of private citizens. If something like terrorism is the issue, the correct response is to use the wealth and resources of the government to address the root causes of those attacks, not to undermine my right to privacy.

This "going dark" scenario would require new legislation. With secure enclaves, modern smartphones can't be cracked open in the manner that the FBI wanted in the 2016 case. So there's no such thing as "court order tech company to crack phone" anymore. It would have to be "outlaw tech companies from producing phones that they can't crack open", which is very different and does not fall under any existing US statute.

I predicted a long time ago that if computers become unhackable LEA and intelligence agencies will push for laws that require backdoors to be built into hard- and software.It will be interesting to see if my prophecy becomes reality.BTW I also hate that Hacker News is being dominated by articles on A.I. lately. Maybe we should vote on HN reducing or even eliminating A.I. related news?

We will know we've made systems secure when laws focus on compelling people to provide access.These laws exist - they aren't the focus yet. Right now there's still no need; just hack the device or compel the cloud service to give the data, why waste energy getting consent from its owner!More bugfinding AI, more end to end encryption, more CVEs and more fixes, cannot happen soon enough.

I think what’s unintentionally eye-opening about this chart is the recency of “law enforcement can read your text communications.”Law enforcement doesn’t need this surveillance ability at all. All time periods prior to 25 years ago didn’t have it.Additionally, there is no correlation between “law enforcement reads text messages” and crime rates going down.

Some people point out AI can cause bugs as well as fix them, and its a valid point. But the question is: what will the ratio be?If automated pentesting in PR review CI pipeline will become table stakes - which is very plausible - maybe the OP has a point.

> In April, Anthropic announced a new model called Mythos that was optimized for software vulnerability findingNo, it was just good at it because it wasn't RL'd against it. I know this is a small detail, but it tosses journalistic credibility in my eyes.

I completely disaggree with this take. Modern AI is not yet capable of finding multi-component bugs as advanced as those produced by firms like NSO.

Childish, nation-states as powerful as the US have access to much more potent stuff. Maybe they'll be forced to rely more on their Intel ME/AMD PSP/modem (cf https://redmine.replicant.us/projects/replicant/wiki/ModemIs...) backdoor and ANT James Bond catalog.

> Defenders are now in the process of patching every bug they can find, often with AI helping them. Entire development toolchains are being rebuilt to incorporate powerful vulnerability scanning before software reaches the testing phase. This does not mean that every bug will be found: even calculating the number of bugs in a piece of code is probably uncomputable. In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon.> Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs.His conclusion sounds extremely optimistic to me.