Malware developers added nuclear and biological weapons text to to their spyware

Individuals obviously need not apply. But regular companies need not apply, either. Think "checkpoints and sign-out sheets that ensure that your own company will notice if some of this stuff disappears." Picture the sort of thing your mind might conjure if you've watched enough forensics protocol dramas and I say "evidence locker" and "tamper-evident seals" — except crossed with hazardous-materials handling policies.

The thing is, this whole chain-of-custody system can be pretty easily circumvented. I won't go deep into how (I'll just say: 1. there are principal-agent problems in academia, and 2. this system wasn't designed to handle sudden organizational bankruptcies well.) But the point is that a grey market for these precursor chemicals and specialized devices exists.

The main place that "false positive" events come from, that the state has to look into, is from people who manage to acquire precursor chemicals/devices without being part of any known chain of custody. (Which, note, doesn't mean that they did anything illegal per se. If it turns out they're just, say, a chemistry-education content creator, then the intelligence body just adds them to their knowledge graph and otherwise leaves them be. But they do have to do some interviewing to determine that first.)

To minimize the number of such events, the "knowledge" that is being truly suppressed here, isn't actually the knowledge of how to do the work; it's the knowledge of how to circumvent the chain-of-custody system. In other words: the logistics.

Information about "how to make a nuke" is general and evergreen; you can just absorb the lesson once and be good. So that info is just "out there", irrevocably. But information about "how to acquire the stuff to make a nuke" is both at least somewhat local to the country you're trying to do it from/in, and also changes all the time, as each state chases up and shuts down existing grey-market channels, and then new ones spring up to replace them. Thus, suppressing logistical knowledge is actually both useful and tractable. And so that's what states mostly go after.

(Mind you, the knowledge of "how to do the thing" does often end up roped into this knowledge-suppression scheme by overzealous downstream regulators who don't understand the load-bearing assumptions of the system they're working under.)

---

The worry states have about LLMs, I think, is that simply by scraping the web into a training dataset, they'll end up stumbling onto the right conversations (that sometimes do indeed happen anonymously in public) to end up with fresh + local chain-of-custody circumvention-logistics knowledge. (And it'd be very hard to "unpick" data like that from the training data.)

Or, even if they don't ingest the data at training time, they'll ingest "the places where that kind of info might end up", and thereby get so good at being "runtime demand-driven searching-and-scraping engines" for this type of thing that they'll be able to surface fresh sources of such info anyway — basically cranking the logistical-pipeline "reconnection speed" after state disruption of a supply channel down to near-zero.

Prohibiting the LLMs from speaking on this subject generally, prevents them specifically from enabling this specific fast-turnaround circumvention-logistics research use-case.

I make no claims as to how well adjusted I am, but I've at least survived 40-odd years of life since then.

Let the kids play.

Never let your age stop your curiosity.

But also learn from other's mistakes (and don't try to eat condensed milk when hanging head down)

I blame my dad though, he found the recipe online and printed it off at work to bring to me.

Fair enough, I misread your original comment.

The broader point stands that the limitation on creating nuclear weapons and reactors is not knowledge but materials. Even if he himself had a PhD in nuclear physics he still couldn't have built one in his backyard because he wouldn't be able to get the materials. A nuclear physicist can't build a reactor without materials anymore than a pilot can fly without an airplane.

There is an extremely narrow band of things that the AI shouldn't be answering, and that is generally immediately-actionable advice that allows someone to build something of harm to others. But even then, in an age where Tor, bittrent, i2p, abliterated local models, etc are freely available, let alone numerous books and online resources, is there even a point? Is it worth fully compromising the principles of free agency to an increasingly oppressed populace?

But instead of that we are handing the keys to regressive and repressive governments to order the suppression of any knowledge they deem inconvenient. I really doubt anyone is going to take a principled stance when the company's party minders threaten local staff with a rubber hose or incarceration.

I'm sure China et al are already doing this.

For the past 30-40 years humanity has received an incredible gift in these sand-powered thinking brainboxes. A gift that allows the common man to empower himself with a force multiplier towards his own success, and now access to superintelligence the likes of which few have ever seen. These can be tools to destroy the oppression that governs our lives from foolhardy, greedy, bootlicking control freaks. And here we are squandering it.

That sounds like what Claude would say unless he was really good at jailbreaking it, which would IMO imply he knew he was chasing after a bad idea.

It really isn't.

A pile of radioactive waste isn't a reactor. Marie Curie's notes are famously contaminated with radioactive materials but they aren't a reactor. This is about as close as the boy scout got.

The Oklo fossil reactor is unique because it happened to form in the right circumstances to produce a fission chain reaction, which does make it a reactor. Not every uranium mine is a reactor, in fact this is the only one known.

But I rather suspect there are improvements to be made in the realm that are a lot easier than building a uranium enrichment centrifuge hall under a mountain.

What's most worrying is, Russia showed that you can use carfentanyl / fentanyl for the very same purpose, and that kind of stuff is something you can get shipped by the kilos as "research chemicals" from China or make it yourself.

I'm imagining a comedy in the style of "The Office" in which the majority of the workers are agents of sabotage who are unaware that the majority of their coworkers are doing the same. How far fetched is it for the entire program to be a fake, with all the pomp and cost of a real program, but secretly existing only to string the leadership along with occasional dog and pony shows?

(30 seconds of googling.)

Or perhaps you meant Q clearance nuke stuff? That would be QUITE a bit harder to find and illegal to share. But it’s lack of availability is hardly a counterpoint to the comment you were replying to.

If the information is in the corpus then it's also in the public Internet and/or in books. The safeguards are there not because the model knows non-public information, but because it's a bad look for the model to dispense that information.

>they will tell you how to make a bomb with easily available items

Making a chemical explosive is trivial compared to making a nuclear weapon.