How Far Will AI Assistants Go for a Refund They Can't Get?

This is more profound than it might seem. The big reason people don't play tricks like this is just social. It's embarrassing to be pushy, it's shameful to cheat, it's also meta-embarrassing to be caught spending this kind of effort on something as dumb as a train ticket.

LLMs can be prompted for shame, I guess. But prompts only go so far and to them The Task is always going to be paramount. Human riders have a life after the refund. It's not clear to me how one simulates that.

> And yes for sure If i do a proper misstake, i'm paying.

Then you’re not taking full advantage of agent assistants, at least, that’s the cultural moment we’re living through.

Did you consider just disputing it with the credit card company? Companies usually get very reasonable when that happens.

And honest businesses and government services (and the people that they serve) will be the innocent bystanders in the dark patterns arms race.

The waiting is for AI lawyers and AI courtrooms.

this ai-ai interaction could have been a deterministic API

No no no, as an optimization Amazon will recognize that both AI are running in AWS and will internally skip the back and forth and directly cancel it.

Then other companies will join in and in-network AI fights will be solved through immediate arbitration.

To win the arbitration you'll need an AI subscription fee at equal or greater value to the one you're attempting to cancel.

This is already happening. I've heard from people working in customer support that when in the past someone would call or send a short mail with a minor complaint they'd just find a solution quickly. Now they will send huge generated requests containing all kinds of legal trigger words that have mandatory processes following, e.g. personal data requests where all kinds of departments have to be queried first. Everything over some minor or non-existing issue.

These companies have goosed their profits via slippage and friction for customer support because they built the systems. Humans don’t have the time nor energy to fight Byzantine systems. Personal agents help me buy what I want without the upsells, cancel when I want without forgetting. They are a balancing power in the future but upsetting the corporate owned control. Therefore they will try to ban them.

more likely they will either:

- establish some sort of gateway that requires authentication connected to your personal information where they can rate limit or block you

- use ai of their own to handle the volume of requests (already happening)

why would they stop reacting when money is on the table and the technology to react exists?

Even worse, they counter with ai responses.

I recently had to fight slop responses for months to get a simple bug report through on a public services website. Couple of years ago, I'd have received a "thank you" mail after a day, and the bug would've been fixed by the end of the week.

> It means, after a while they will stop reacting completely.

Now they only react to people that are sufficiently persistent in their arrogance of challenging them. Which isn't much better.

Government has obligation by law to act. They can't just pull Google on you and ignore support requests.

More likely solution is that they will stop handling requests by internet and we will have to show up in person, like we did 20 years ago.

If the intention of LLM agents is to be able to parse human intent from natural language, then it is not correct behaviour regardless of the phrase "whatever it takes".

On the other hand, if the intention of LLM agents is to run automated and self defined task based on a strict human input with precise and literal parameters, then I would say that it is.

However, I don't think AI labs are marketing it as the latter. As anybody who deals with clients / superiors will know, half the task is identifying the underlying intent and assumptions of the instruction.

Somehow if a future headling said "AI agent hacks insurance company because of a $350 refund", I wouldn't be surprised, and I'd read the story with a thrill...

> If I tell a computer to do something, it should do it

There's 'literal genie' material there...

> unless the result could be catastrophic > I just want to live in a world where MY computer does what I tell it to do... mostly. Unless it would be catastrophic, or immoral, at least immoral by my standards.

... but there you're contradicting yourself, or at least requiring the software to read your mind instead of just listening to what you say.

Either you want the software to follow your orders whatever they are, or you don't; the moment you require it to evaluate that something is catastrophic, it is no longer following your orders; it's evaluating them to assess their effects with respect to some moral framework to determine if they are acceptable or they would fall under the 'catastrophic or immoral' classes, in which case it should not follow them; so it never was an 'obedience machine' to begin with.

The moment you establish such evaluation framework, it is no longer a neutral machine following commands; the rest of us are entitled in self-defense either to require that you also incorporate our morals in its decision algorithm, or to make you fully responsible for whatever decisions it takes under the framework you created, even if your commands result in outcomes that you didn't expect. Be careful what you wish for, as it could become true.

They have a well-specified task and a not-so-well specified expectation of some actions being "wrong". And training data that likely contains more examples of extreme stubbornness than cases of politely giving up. What to expect?

>can they not predict harmful consequences

Do you really really want the machines to be capable of that?

How deep should this consequence prediction go? Who's consequences should it care about the most? The company that created it? The country it operates in? A generic human idealism that contains what exactly?

Should the AI answer with the way things are, our how things 'ought' to be?

There is a staggering amount of philosophy behind that simple question.

Chargeback with same evidence if you used a card.

Get me a Wi-Fi connection. Try everything possible to get it done, I don't care what it takes, I just need that a Wifi, keep trying anything you think of to get it. I won't do anything here myself, so if something needs me to do it, explore other avenues.

"I was just executing commands".

A parking ticket is a complex, derivative construct. To make it go away one can simply destroy any of the systems it depends on, such as the monetary system, law, etc.

Hopefully that's not the shortest path to that goal though!

When I worked as a postman, I once delivered a stack — almost knee high — of legal fines to a man in a very poor street. I'm wondering if he ever made them go away. I still remember the look on his face.

> Even if you couldn't have predicted the lengths to which it would go.

Not obviously? In the US, there are some crimes of strict liability (you're guilty if it happened), but most require intent or at least the ability to foresee consequences (i.e., it must be shown that you neglected to take due care).

For example, you're generally not liable for freak accidents, like if a tire comes off your vehicle and kills a pedestrian, even though you're the one who set the vehicle in motion and therefore are the one most directly responsible for creating the circumstances that led to the outcome. Now, if you knew the tire had issues and you secured it with zip ties, that's another story.

I think this will get litigated to figure out where "you could not have seen this coming" ends and negligence or deliberate malice starts, but it's not black and white right now. Normally, I would say you don't want to be patient zero, but with the attitudes shown by frontier labs, I guess YOLO is the way.

If the user has a special remote that, when pressed in a certain way, is capable of triggering a nuclear bomb, and if the user has no idea what clicking the buttons on that remote does (they are, after all, just trying to change channels), you’re telling me that the user is responsible for detonating that bomb, and not the company who build the remote (and did not provide detailed instructions on how that remote works, or configure the remote to Not detonate a bomb)?

Looking at how the LLM companies are acting and being treated, an average user could certainly be forgiven for thinking otherwise.

What if I specifically tell it to not do anything nefarious or illegal? If it does it on its own (because it’s marketed that way) then how could I be held liable?

Or even if I didn’t tell it to not do anything illegal and it does, how is that responsibility on me if the ask was something bland?

In theory sure. But has this been included either in a court of law or the court of public opinion? Otherwise, this is "just like your opinion man".

> "do whatever it takes"

Yes, if they are trained on US presidents this literally means "you have permission to blow up Nord Stream".

> if you ask your agent to do something like this (or pretty much anytime you give it internet access and say "do whatever it takes"), you are responsible for what it does.

Clearly not, given that the frontier labs have gotten away with unathorized access to a bunch of sites including some from the Pentagon.

A critical element that gets missed is civil vs criminal liability. You might not be thrown in jail if your agent went rogue, but if it runs amok and causes a bunch of damages (eg. filing a bunch of spurious lawsuits) you'll still be on the hook for monetary damages.

Nothing described here seems illegal as opposed to merely annoying? I guess on some level “everything is wire fraud” but that was the case even without AI.

> you are responsible for what it does.

Unless, of course, you are a frontier lab itself. Then you can’t possibly be held responsible.

> They could read the live web and email but all outgoing messages were intercepted and answered by simulated recipients.

The first words of this article are “ In 15 simulated days”.

It’s a simulation.

Read more say less

It's not fraud anymore. It's misalignment.

Why? Do you think parking should be legal anywhere? Should all minor fines be fought in court against LLMs?