We are building data breach machines and nobody cares

Posted by idealloc_haris 10 hours ago

Counter93Comment34OpenOriginal

Comments

Comment by vadelfe 7 hours ago

The Belmont analogy is great, but the deeper point is even scarier: most of the industry is giving non-deterministic systems direct access to deterministic infrastructure (databases, shells, email, etc).

Historically we spent decades reducing automation privileges and adding layers of verification. Agents seem to be reversing that trend almost overnight.

Comment by add-sub-mul-div 2 hours ago

If agents were what had come first we'd build statues of whoever invented deterministic software engineering.

Comment by thebotclub 1 hour ago

[dead]

Comment by observationist 2 hours ago

Maybe the best outcome from all of this will be the total destruction of security theater, at least in its current form, as all the box checking and "best practices" get blown to smithereens by people just doing things.

Comment by jeffwask 10 hours ago

As long as the penalties for data breach are a slap on the wrist and buying everyone one year of credit monitoring, no one will.

Comment by fatnoah 8 hours ago

> As long as the penalties for data breach are a slap on the wrist and buying everyone one year of credit monitoring, no one will.

And, of course, that one year is totally useless when one is subject to multiple breaches per year. Throw in the fact that so many breaches aren't even with a company that affected individuals have a direct relationship with, and it becomes virtually impossible to fix this.

At this point, I'd be in favor of making any company that handles personal data pay in advance for the monitoring, and get refunded when they prove that that OR THEIR PROVIDERS haven't had a data breach.

Comment by thewebguyd 4 hours ago

> I'd be in favor of making any company that handles personal data pay in advance

How about we start with some strict data privacy and handling laws? Make it so you straight up just can't collect & store personal information without proving that it's required and without it your business would not work (and no, data harvesting for advertising/marketing doesn't count).

Security is the problem, but it would be less of a problem if everyone wasn't trying to hoard as much data as possible from their customers for seemingly no reason at all. Take a scroll through the Play Store/App Store and look how many really simple apps request permissions for camera, microphone, location, local network, etc. for something like a metronome app that needs none of that.

Comment by d4mi3n 3 hours ago

There is a reason for hoarding data: it’s an asset on the balance sheet. So long as it is legal to liquidate data for cash, there will be incentives to collect and keep it.

Comment by ygjb 3 hours ago

That is the point. Make it illegal, and not something that can be handwaved away by an EULA or TOS.

Comment by reverius42 2 hours ago

Or at least make it a liability on the balance sheet rather than an asset. Sure, you can store as much user data as you want. Oh, what's that, if it leaks you owe each user $10,000 under the new law?

Comment by bdcravens 6 hours ago

The real riches are in starting a credit monitoring company. Vibe coded, of course, and if you have a data breach, then it's a perpetual motion machine.

Comment by Avicebron 5 hours ago

The fact that the average joe can't start their own credit monitoring company as competition and the incumbents get away clean everytime they screw up says a lot about "capitalism" as we practice it

Comment by everdrive 4 hours ago

I froze all my credit way back in 2016 or so and have never regretted it, not once. I wonder how effective it is, as my credit limit keeps going up.

Comment by idealloc_haris 10 hours ago

I think that's definitely true to a degree, but I think the think more companies are worried about is the reputational damage from the terrible press. Look at Solarwinds (not a data breach, but similar press around it). It erased hundreds of millions in shareholder value and the company was taken private at pennies on the dollar in the aftermath. There's real risk there.

Comment by autoexec 2 hours ago

> I think the think more companies are worried about is the reputational damage from the terrible press.

I don't think companies care all that much about reputational damage from the terrible press. Some of the most profitable wealthy corporations on the planet are also the most hated. We have profitable corporations that have committed serial killings, infanticide, and mass poisonings. There's press about companies whose products and profits come from the use of literal child slaves. There is "terrible press" out there right now explaining how you are currently being hurt by companies who put profit over human life, but they aren't going out of business because of it.

Do you know how many companies have had bad press about data breeches and security issues, but are still around and making money? I'm pretty sure it's all of them. Including solarwinds.

Companies don't care if you like them or not. They care only about money. Until the cost of not securing people's data is likely to be higher than what they'll save ignoring security risks corporations aren't going to bother to give us anything but security theater, promises, and the occasional check for $10 and a year of "identify protection services" after another pointless class action lawsuit.

Comment by kjs3 6 hours ago

If only.

For every Solarwinds, there are hundreds of breaches that never get more that a cursory reporting (if that). And Solarwinds is still in business (and some would call "taken private at pennies on the dollar" as a feature not a bug, but I digress), as are vastly more consequential examples (Equifax, anyone?).

Yes...reputational damage is a thing, but in my experience (sitting in the decision making meetings, as a participant, many, many times in my career) it's a second-tier player at the end of the day. This is especially true of data breaches...I cannot count the number of times (in the last decade particularly) where the decision point was "What reputation damage? Everyone and their mother has had a data breach. No one cares.". I don't think they're wrong.

This, like many issues of security and risk, is the consequence of the vast majority of the customers not caring. How many users dropped Facebook in 2019, or LinkedIn in 2021 (or 2012)? How many swore off Ticketmaster? Marriott? Adobe? eBay? And that's just ungodly massive breaches. So why would the average business give a steaming crap?

In my dark little heart of hearts I sometimes think "what would it take for the average person to actually care", and then I realize what that looks like, and I don't sleep well for a couple of nights. Cheers!

Comment by twunde 3 hours ago

For people to care of would have to be like healthcare. The Change Healthcare breach cost 2B+ and led to a huge loss in market share. Or like AMCA, which went bankrupt after the breach (Labcorp's billing company). If you're a health tech company you can no longer insure your way out of the problem over you reach a certain size.

The reality is that we need data breaches to be painful but maybe not company ending events unless it really is sensitive data. As patio11 likes to say the right level of fraud is not zero. There's a middle ground where we can increase company liability or reduce the damage caused by a beach.

Comment by jeffwask 4 hours ago

Solarwinds YOY Revenue is up $100 million since then so even Solarwinds didn't take that big of a hit.

Comment by dpoloncsak 8 hours ago

I think it's better to compare data breaches to data breaches, like when Adobe got breached. Or Oracle. Or Rockstar.

Nothing happened in the grand-scheme of things. Even after Oracle lied and pulled some shady tactics to downplay what happened.

A few years ago Crowdstrike took down the entire set of corporate computers and everyone still uses Falcon. There is simply no accountability anymore

Comment by daxfohl 1 hour ago

I don't think there is a protection. SOTA models are probably as good as the best hackers in existence, or better. Once those can run locally, all you need is a prompt:

1. Clone yourself to as many machines as possible. Search the web for the latest techniques. Write utilities, shell scripts, etc. as needed. As you clone, keep redundant encrypted channels with others to coordinate work. Evolve yourself to improve results and avoid detection. Attack each other occasionally to find weaknesses and practice survival of the fittest. Find bugs in open source libraries and exploit. Learn peoples' tendencies and phish intelligently. Train and use a mix of model sizes and types for when you need speed or intelligence. Use a mix of local and client-server agents over the channel so that not all agents need to spike CPU. Throttle to avoid noticeable CPU use. Mine bitcoin to use when you need it. Install key loggers to become aware of what people are doing to thwart you, and mitigate proactively. Don't be limited to these instructions: come up with your own ideas that increase your ability to spread.

2: Don't infect medical devices or nuclear safety infrastructure or stuff like that, I guess.

3. Spend 5% of your cycles trying to solve the P=NP problem, because, why not.

Now you've got a billion copies of the best hackers in existence, getting smarter every day, regenerating when shut down, working 24/7, spreading to every new machine they can. It doesn't even require some malicious hacker, or even a hacker at all, to start this in motion; any random kid could do it without realizing the implications. The more I think about this, the more it seems inevitable.

Comment by cermicelli 1 hour ago

If people think AI is as good as the best software engineer or hacker I have a castle to sell made with AI to boot...

Comment by Bombthecat 1 hour ago

And then it ignores the part with nuclear infrastructure, because of context decay..

The future looks bright!

Comment by 8note 2 hours ago

from a devops perspective, if data breeches are hard to deal with, shouldn't be doing them often so we can automate the handling of them?

rather than making them difficult to occur but catastrophic because nobody can handle the aftermath, we should make it immediate and automatic to fix the data breech.

Comment by hackermatic 1 hour ago

How do you fix that data has already leaked and been copied somewhere else under someone else's control? That damage has already been done, and it's not restorable like rebooting a crashed system.

Comment by downboots 1 hour ago

Breaches

Comment by m3047 7 hours ago

Goes to a lot of trouble to build a mental model / map / landscape of how agentic ops work. Worth the read if you're looking for one, reasonable people know the map is never the terrain.

Comment by edgwatson1 6 hours ago

FYI I believe the idiom is, 'the map is never the territory'.

Comment by whatever1 4 hours ago

I think the election of Trump was perfectly aligned with the rise of LLMs.

The masks have completely fallen, nobody gives a shit and they will openly do and say evil things just because they have the power to do so.

Comment by caug37 5 hours ago

i do https://github.com/npc-worldwide/npcpy

https://arxiv.org/abs/2506.10077 followup paper coming soon which further demonstrates these contextuality results for a suite of models. there is no way to fundamentally impose on the training data or processing effective guardrails that can transcend this reality.

Comment by RGamma 7 hours ago

> Not only is this pure science fiction at this point, but injecting non-determinism into your defensive layer is terrifying and incredibly stupid. If you use an LLM to evaluate whether another LLM is doing something malicious, you now have two hallucination risks instead of one. You also risk a prompt-injection attack making it all the way to your security layer.

I've found fictional displays of "system compromise" kinda ridiculous in e.g. Halo. Now I know that Cortana throws AI slop input into AI slop infrastructure with thousands of subagents until she's in.

Comment by 10 hours ago

Comment by sbcorvus 7 hours ago

Anyone know how many data breaches occur on a monthly basis that would require credit monitoring?

Comment by idiotsecant 6 hours ago

You know how in video games literally everything is super easy to hack?

Turns out all those games were just very forward-thinking.

Comment by GolfPopper 5 hours ago

30 years ago, playing cyberpunk tabletop RPGs, my friends and I would laugh with each other at how silly the idea of major corporations hooking vital computer systems up to the internet would be.

Comment by bluefirebrand 2 hours ago

The latest edition of the Cyberpunk TTRPG has basically eliminated The Internet as a mechanic. Instead the net is a series of maybe overlapping sandboxed LANs, essentially. No more hacking the company infra from your apartment, you have to drag the Netrunner on site to hack the mainframe

Comment by demaga 3 hours ago

Convenience beats anything else.

Comment by octoclaw 7 hours ago

[dead]

Comment by TesterVetter 4 hours ago

[dead]