Publish fake stats, get cited in LLMs

When an LLM is the lens through which you view that text, you often don't have the context of where it came from, and the LLM is often really bad at filtering trustworthiness based on context when retrieving information to answer a query with.

I wouldn't go to a random site with a sketchy URL to get the customer service number for my xyz. I'd go to their official domain. Thus successfully navigating the treacherous waters scammers attempt to muddy.

Yes, but if you read this "semantic" info directly from a site, you can judge the site and its potential biases, and decide for yourself "this sells Louis Vuitton, of course they will praise their handbags". Or to pick a more extreme example that nobody would ignore, if you somehow land on an extremist website spewing racist crap, your alarms will surely trigger by anything they say.

But through an LLM this gets whitewashed and presented as authoritative and "neutral", when it isn't. You might know not to trust the LLM either (if you're paying attention; we all have our unconscious biases) but I bet you most people will be caught off guard.

Billions have been spent on it.

Try searching Bing sometime. First result for Firefox is a malware clone site (you’ll have to scroll past the malware ad to see it).

It’s hard to find and convict people who post actual illegal content online, much less fake reviews

s/human/unscrupulous marketer/

Bad for humans in general, good for hucksters and con artists. Net negative for humanity.

Furthermore, you are content to pretend that the rest of humanity is capable of doing this on a regular basis. And not just the top 10% of those who can read, interpret and weigh biases, but a solid portion of it.

It is a completely reasonable interpretation if you are an accelerationist living in the elite classes. In fact, perhaps, they are even counting on a majority of people thinking this is a fine environment.

I don't have the answers, but a lot of this is just a continuation of a very bad trend.

Regardless of how the data is sourced, this is bad. It's poisoning the internet.

Besides, such issues come from a lack of due diligence, which have only been exacerbated by AI.

And there, in five words, is the problem/feature/definition of AI.

This type of behavior is why gatekeeping institutions exist. We all pay a non-trivial cost because someone thinks that other people taking their lies seriously is somehow remarkable. It's not.

Research is hard work. Publishing fake research is just intentionally throwing wrenching into the machine of academia. All you've done is reinforce the need for gatekeepers (which never went away), show that some no-name journal isn't rigorous (they often aren't), and you've wasted a bunch of people's time demonstrating why institutions exist, when we know why they exist.

Genuinely terrible behavior.

> "Citogenesis" redirects here; not to be confused with cytogenesis.

I had to read that three times. I thought it was a self-referential joke at first. Turns out the name was a joke from an xkcd. Can't argue with that source in terms of authority.

Backlink ranking was/is a reliable heuristic to answer that question across virtually all topics.

So I disagree that the "problem was not about quality."

Spam/scammers will always be in whatever landscape there is, and I agree part of what google does it keep them at bay.

Now, the model slips it in without any verification and casually builds on top of it.

Even if you are in the minority who asks for sources and truly verifies them, they are often made up (don’t exist) or don’t say what the LLM said they said.

The other day for me Google search AI cited triggerbites which is a vibe data website.

The LLM unless explicitly prompted cannot distinguish the validity of a data source.

When I ask ChatGPT certain questions, it very clearly shows me the sources it's searching through in the form of favicons, and they're always the favicons of sites that are in the first 5-10 results for a Google search query that would fit what I've asked the LLM. It will then cite those websites as it's source, with a direct link to the page that also shows up in an equivalent Google search. It does that daily.

There's a tension there: they want you to use AI for everything, and if they get what they want then new information will be embedded in LLM text.

> Training classifiers is pretty manageable.

Hopefully such things can be opened up, so the slop can be iterated on until it passes. It doesn't need to be perfect, just enough so more garbage gets through.

if anything it’s more true now, AI chatbots rarely use raw LLM output, instead they have CoT and use search tool calls. and the speed at which webpages are crawled is in the matter of hours. I posted something here on hn, then searched for it on google 1 hr later and its AI answered with my comment

https://share.google/aimode/PkQUVfA68AlHGb7Yr

there’s several companies right now making a killing of organic placement on chatbots similar to the SEO rush of early google

> 2) There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text. Training classifiers is pretty manageable

i think it’s pretty naive to think that if LLM companies would develop LLM detection they would use it to skip training on that content.

instead, they use it to train their output to be less detectable.

LLM detection is not solvable because it is adversarial, an arms race

Hard disagree. Sometimes it's avoided, any other time we just go into cope mode. We discover on the ride down that the bottom isn't so bad. Turns out our collective control mechanisms didn't fail ourselves if the initial assessment was wrong.

If they didn't, they'd be unable to compete with their competition and also we'd get worse models that can't reason about many domains or do that much due to limited training data/generalization. I don't think we can put the cat back in the bag or mandate only non-problematic training codebases without shooting ourselves in the foot.

Now, if those AI orgs were mandated to properly compensate everyone whose data they've mined, to track which neurons get invoked and to pay accordingly based on what was used to train them... ehh, that's about as unrealistic as stopping. Kind of a bad situation all around, unless you abandon the idea of any sort of copyright.

good thing nobody falsified statistic just for the sake of it -- so being a sabotager and a foreign spy was always part of it too.