Orca-Bench: How Ready Are Language Model Agents for Oncall?

Attackers advantage in the iterative fast feedback loop?

It’s harder to have a loop to ensure you are defending all possible attacks?

I guess the loop is you need to attack yourself and fix. But attackers only need a single opening.

Finding all possible attacks and patching them against yourself is inherently more expensive?

That is why I built https://safebots.ai/safebox.html

Your strategy can’t be patch AFTER an intrusion. Only to build a hardened environment from scratch and be ready in advance.

[dead]

Was really looking forward to that. Hope they publish.

I think this is where harness makes a lot of sense. Use LLM to produce all possible attack angles/phrases and just stupidly filter them out on input.

Avoid the most dangerous situations by making sure LLMs with untrusted input produce output that's human reviewed.

Still makes an interesting way for, say, a former employee to poison the results.

You would trust an LLM to make changes to prod without being verified by a human first?