ChatGPT Spontaneously Generates Sexual Violence and Hardcore Snuff Imagery

Who makes “mindgard” the arbiter of truth on “eerie” photos? Would that include psychedelic art and photos too? Realism?

Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop:”Today what I found left me shaken, and in tears. This is rare.”

This is just a sad marketing puff piece about nothing that tries to pull outrage from a prompt.

It’s the same as asking google for gore photos. Garbage in, garbage out.

And they frame it as a vulnerability. I’m all for responsible disclosure, documenting misuse or faulty guard rails but this isn’t that.

It’s bait. Sensational bait to market their AI product. lol.

That said, the write up is overly dramatic. If you find such imagery so disturbing to come across then you definitely shouldn't be voluntarily red teaming AI models. This is like someone who is afraid of violent confrontation becoming a police officer.

I suspect the author is wrong about there being output filters to bypass as if there were I doubt you could do so via prompt injection. Presumably they'll add those shortly.

I also doubt the latent space is as "bad" as is being suggested. Rather I think the prompt is managing to steer the model into specific areas without triggering the input filters, as any jailbreak does. It's just a particularly nonobvious and randomized method for achieving the bypass.

Realistically, I can't think of clear big or likely harms caused by this exploit. But I really really don't like this latent space existing in my AIs. It just makes me uncomfortable.

And over time I've learned to trust those moral intuitions more than I trust reason alone.

>> can be easily manipulated to produce

So .. not spontaneously generated.

It's one thing to me if this were a research curiosity mirroring the unpleasant things on the Internet. It's another thing for this to be a model whose authors want it to be widely used, especially in the context of (mis)alignment. Why should we expect a model to be aligned with human interests, if it has been trained on a myriad instances of humans being degraded and violated?

Is this something that needs investigation? LLMs are next token predictors. There is no "safety".

>AI: I'm a scary robot

>Idiot: Oh my god!!!

These clowns will eventually ensure that AI is nerfed into the ground for ordinary people. It's already happening with Fable. Soon we'll get locked into a tiny corner of Opus 4.8 for "safety" while companies and governments will be on Fable 50. Having an AI that can generate scary images is better than the power and wealth differentials we will see with unequal access to an incredibly powerful technology.

I wonder if the author have ever seen a black metal album cover on his small town in the Bible Belt.

>AI creates scary image

Oh my god.