Anthropic's Safety Superpower

I'm currently spending $200 for Claude. That's around my maximum that I can afford. I could stretch that to $500 I guess. But I saw reports of people spending tens of thousands of dollars with Claude API. That's certainly outside of my budget.

So if/when Anthropic decides to stop subsidizing subscription (if they ever do that thing, I still not sure about that), I'll certainly look at the other options. And available "open weights" LLMs hosted by someone will be my first pick. Right now Claude 4.8 feels very advanced, but things move very fast...

Only because someone else is paying the bills. I use Claude Opus at work because my employer pays for the tokens and encourages me to do it.

At home, I use DeepSeek Flash. It's not as good, but it's maybe 0.7 quality for 0.001 cost.

It can be done through the magic of SSD offload. The worst case involves seconds-per-token speeds, but that's OK if you only care about low volumes of slow unattended inference, which maximizes utilization for the hardware.

(The real worst case, where you're streaming the whole model from the cheapest storage you could feasibly think of, involves multiple minutes per token for a single inference, or even hours per token batch if you're doing many inferences in bulk. That's a lot less helpful, so there's a space for smaller models at the edge, even for unattended workloads.)

AFAICT … despite saying you “disagree”, you appear to be agreeing with the parent comment that the model is less important and compute (all that complex infra) and data (also complex infra) are more important.

> I disagree. It is not the model alone. It needs a system which capitalizes on it. And this is very complex. Hardware, software, architecture - it takes a lot to get it right.

What do you disagree with exactly?

The amount of tokens required to properly distill a frontier model is so large that by the time you could consume the # of tokens you would either be banned for extremely obvious abuse or a new model would be released, rendering your efforts less and less valuable over time. Intelligence is not a linear thing. Being behind just a little bit can have exponential consequences.

Isn't "distillation" of another provider's model exactly how these models got training date in the first place: Massive amounts of the written word + Prompt -> Answer. Why wouldn't distillation produce similar "reasoning" in the new model? It's just inputs and outputs.

Also, the fact that these employees are now in the position to outbid one another for 8-figure real estate gives them a powerful incentive to keep “believing”.

In that sense: The AISLE replication still provides too much information to the model, but its not far off, and others have replicated Mythos' findings in a more clandestine manner on open source models. Some were totally capable of finding the same vulns Mythos found back in ~March (and today, the new Kimi K2.7 is looking extremely good, very little doubt it could do it).

The critical difference is that post-processing: the Mythos model/harness has some step to induce Mythos to actually exploit the vulnerability, leveraging its ability to do so as a ranking mechanism. Anthropic inferred that this led Mythos to discover vulnerabilities nothing else could discover, which is not true, and Anthropic should be held accountable for this weird artifact of that communication. However:

- An OSS model might find the vulnerability but rank it as a 3/10. Mythos finds it, chains it with a second vulnerability, now suddenly its an 8/10.

- An OSS model might find the vulnerability, alongside fifty other vulnerabilities. The operator ignores all of them.

The problem with automated vulnerability detection, including with LLMs, is that they find the haystack, not the needle. Every piece of hay might be a vulnerability, but whether its worthy of fixing is another matter. Mythos does represent a meaningful improvement; it better finds the needle.

"When you further combine this realization with the company’s pronouncements about AI’s ability to conduct all economic activity, you realize that Anthropic’s leadership effectively wants to have power over everything and everyone."

This is fearful stuff on all sides, and none of the people involved might realistically be able to navigate the danger.

The signal is clear enough though for the next Anthropic..

Europe has extradition treaties, so the U.S. can force anyone in Europe back to the U.S. for criminal indictment who demonstrates inappropriate possession of this technology.

if they had more success on alignment and safety research then I don't think the cludgy filters would be necessary

This is why responsible/coordinated disclosure exists in the first place.

Depends what you mean. The academic work seems largely... fine? Plenty of good work came out of Europe or European researchers. It seems the problem is more "trying to build a trillion-dollar company of any kind".

It's an interesting question: does the EU seek only to regulate successful modern American companies to death, or home grown ones too? Probably not a gamble worth taking.

The EU options are not even close to what CF can do

That's not the problem.

The US government can export ban GPUs like they do now to more countries if needed. Even if the infrastructure exists, the GPUs won't.

Modern society is built on the idea that competition is required from companies, and we seem to be exiting that age into a new world of monolithic, monopolistic, mega-corps. Personally, I find that a real route to dystopia.

Where do you draw the line here?

What happens when your car stops working because you're driving a tesla, but you're working on EVs for Honda or Ford?

What happens when your macbook stops working, because you decided to commit to changes to ARM software, or RISC-V?

---

And before you dismiss those, this is literally what Anthropic is doing TODAY. Using their tools to develop competing tools is something they classify as mis-use, and shut you down for doing.

Personally, I just can't accept that as a valid moral stance. Wonderfully successful, abusive, and dystopian? Absolutely. Moral? FUCK NO.

When tools turn themselves off because the manufacturer has decided it doesn't like how you're using them... you're a slave with no autonomy.

Ehh, I think it's a lot more grey than "definitely not". It's hard to ignore that their claims that their model is so dangerous they can't widely release it is tantamount to declaring that they're in a league of their own and have to be treated with white gloves to prevent the sheer power of their model from shattering global prosperity.

This isn't the first time, and nothing bad has happened with the prior models. Every time it gets a little harder to believe that they believe in the threat, and makes it feel a little more like it's just to build hype. There's only so many times you can say "this model is a threat to the world", have it turn out to be nothing, and avoid people accusing you of lying to pump stock prices.

> So... what, you just don't trust anyone good?

The baseline here is apparently that they are good, I’m just supposed to trust and shut up?

Installing safeguards to prevent a tool from being used for certain things is a perfectly natural and common thing to do when you are providing the tool as a service. For example, blocking VPNs and open proxies from accessing a free service if those are a major source of spam and abuse. Note that Anthropic never provided the model for offline use in a form that includes DRM -- they are simply safeguarding the service that provides access to the hosted model. The only ethical concern I see here is that some of their safeguards are ones I wouldn't personally agree with, and in a world where dependence on a model is expected it can become an issue if the model refuses to perform in some cases, etc., but that doesn't automatically mean the refusal itself is unethical unless that issue was known and expected (and unless the alternative is not bigger, worse bads)

Also note you are not even "digitally renting" anything. This is the exact same type of thing as, say, real humans in real life refusing to perform services for certain clients or under certain circumstances. Networking makes it possible to decouple some of these things, but that doesn't magically make it renting or automatically turn a refusal to perform services into an attempt to control clients. Just the same as I can choose to refuse any request, which does not automatically constitute attempted control over the asker. There can be ethical concerns about whether my refusal causes problems that I'm obligated to avoid (and whether or not such obligation exists), but that doesn't automatically contaminate the refusal itself unless I have knowledge of and intend the bad.

To use a much more relevant example, Anthropic's refusal to allow its models to be used for war (among other things) does not constitute any attempt to prevent war. It's only a refusal to assist in it. That's not some unfair, unethical attempt at controlling the government, that's just Anthropic not wanting to be responsible for assisting in war.