Some thoughts about Anthropic's new cryptanalysis results

That is a good analogy. But the limitations of LLMs seem to be where they lack some information they tend to hallucinate/invent.

The rare benchmarks that measure "knowing what the model knows it doesn't know" show us there are only a couple models like Opus that are good in that field.

I think this is something that receives not enough attention from researchers.

> LLMs are like search engines on steroids

i like the analogy of a lossy compression algorithm. The LLM compresses all of the data it was trained on to answer the question it was asked.

Well put. I just think humans do this at a higher better level and not qualitatively different.

Great summarization of current capabilities of models, thank you

The “glorified autocomplete” framing isn’t to take literally. It’s a way to remove the mystic and whole anthropomorphization of AI. It’s saying they aren’t sentient or entities we are interacting with, even if that’s how the output presents itself. Instead they are “just” stochastic models

  > The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different.

Let's say I'm one of such people. How would you convince me that your comment is any different from the ones we've been getting since roughly GPT-3 claiming that AGI is finally here and that, while the previous models were bad, this time they'll live up to the hype?

For me it's more an expression of how much mileage you can get out of "glorified", how many tasks devolve down to "if you model language accurately enough, look what drops out" because it turns out that to model language you need to model how the world works. I don't use it to minimise the capabilities at all.

The models themselves are indeed glorified autocomplete in terms of what they actually do (with things like agentic coding harnesses being required as a wrapper around them to make that internal autocomplete something more useful). Many people use this fact to critique LLMs, but many other common instances of people pointing out LLMs' apparent lack of intelligence actually come from people not understanding that the model is a glorified autocomplete underneath whatever interface people access them through, and the interface isn't providing the underlying model all the information they assume it would, making it seem less intelligent than it actually is.

> I am not arguing that rote memorization is intelligence, or that we have achieved it, but does anyone know what AGI actually.. is?

I would argue that a core part of intelligence is being able to handle uncertainty. That + planning probably explains most of the evolutionary pressure for making our brains bigger. But if this is important to intelligence, chess bots in the 80s were more intelligent than their later counter-parts, which could simply remove uncertainty through rote memorization. Maybe the All-Knowing is a compete dud, no reasoning capabilities at all, just an extremely efficient, infinite lookup table of all facts.

Terms like intelligence and consciousness often just seem overloaded with meaning, and when discussing things concretely, we quickly switch to more specific terms like reasoning.

ARC-AGI has thought a fair deal about this and has a good definition here: https://arcprize.org/arc-agi#defining-agi:~:text=AGI%20is%20....

"The intelligence of a system is a measure of its skill-acquisition efficiency over a scope of tasks, with respect to priors, experience, and generalization difficulty."

Subtextually, when people discuss AGI as a threshold, they're talking about superintelligence, which I'd roughly define as reasoning (not just applying algorithmic search) at the speed with which computers do lookups and computations, with large-scale data set contexts.

Right now, models are doing well-defined knowledge work tasks, applying relatively well-worn patterns (but with the thoroughness of a computer, which yields interesting new results). AGI is us breaking the threshold of "doing stuff people already do to identify and solve problems".

It’s always been nebulous but that was fine because we were so incredibly far away from it. We never really planned to be close to it figuring out the edge. Some have it at human or beyond for all tasks, but then rarely touch on “one human” or “all humans”.

Personally having been in AI since before deep nets, systems have been incredibly narrow for decades.

Classifiers on images were battling with ten classes in 2010. Imagenet had 1k classes and people were getting half of the things wrong then and that was frankly amazing at the time.

And they only did images, only to known classes, only with very specific inputs.

Text classifiers only did a few classes usually and mostly threw all the words together.

The most advanced things I saw in the late 2000s were struggling so much to make general systems that the most general ones were still incredibly limited and bad at those things (we had a robot learning to play games that you showed it). Things like asking a thing for a book and having it parse the sentence, identify what was needed, that it didn’t know where it was but that was knowledge another human had and asking them - that was impressive yet also limited to very small sets of interactions.

The idea of a machine getting sarcasm, even if mostly built for it, was wild.

General meant capable of a broad range of tasks without retraining.

To me we have agi. It’s general, and it’s good enough to be useful.

I think it's just that we have hard time pinning down what is it that intelligence factor that isn't well-covered by LLMs. Just like we've had trouble with discerning human intelligence in the past.

It turns out that a savant with all the knowledge isn't "it".

> I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting.

I'd say LLMs have shown us the opposite problem: there were many different definitions whose differences we'd previously been able to ignore. We don't all agree even on a single letter of "A", "G", and "I".

And this is why it looks like a moving goalpost.

There's a specific singularity theory of AI that is very popular. Eliezer Yudkowsky helped popularize it among the Bay Area "rationalist" community, and it has this idea that AGI necessarily implies a self improving system that will quickly become a paperclip maximizer or other such dystopian or utopian world changing intelligence.

By that singularity definition, we're probably nowhere near AGI, but if we define it as something that is as good at text/information manipulation as the 50th percentile human? I think we're already there.

Were you trying to overoxidize all extant species?

It's the default when I pop open the Claude iPhone app, I usually don't bother to switch it.

If you're paying a flat monthly rate, unused tokens are wasted tokens.

because as OP of the article points out, it's not a guaranteed hit with these models no matter how capable. they are stochastic (but not parrots), humans are too (but in a different way), and so there is a chance that the answer is wrong. If I wanted the most accurate answer I'd be confident in believing, I'd ask the most advanced model.

Now obviously you can and should retort with hallucination and confabulation rates from external and Ant's own reports per model (pretty sure more advanced models are good at lying better, not less) instead of going with dumb "more expensive more accurate" mental model, but general principle stands for me still AFAIK.

it's not exactly rational I admit, but if I'm going to base my own work and reasoning from an LLM I'm going with the best available. This seems to trip up most normies because they are too lazy or too greedy to pay up for premium access and see for themselves why most of us are both awed and afraid. Generally, I'm too biased and too deep in ML/DL cargo cult (been in it since 2016) to know if the skepticism and disdain for such usage is warranted.

In general, I think the tools are broadly toxic in a Dune-sense of making me think less for myself, because just as any HN-poster knows coding and doing mundane low-level stuff is necessary the same way doing stretches is necessary before any workout. The process itself is what keeps your brain strong and its gradients from veering into overfitting. I'm not overly bullish on the whole reaching for the stars ending with these things. Paradoxically, you using them eventually hobbles both you and the model, because you become dumber and then you bottleneck their ability to self-direct (broadly true for next Mythos/GPT-7).

Sorry for a long rant, was just anticipating some things I'd have to say for myself.

i read his, too. his replies are indeed more directed, but also quite short, unstructured, and natural sounding. if i recall, maybe 1 or 2 of his prompts exceeded 50(ish) words.

in my head, the comparison is the multi-paragraph prompts (borderline essays) i would read in various communities on reddit and similar forums, that people (often self-proclaimed "prompt engineers") said were "required" to get good output. or some of the prompts ive read in various logs that are like a thousand words of setup.

even looking back at the first prompts i was sending when i started to use chatgpt were (in hindsight) crazy long and full of unnecessary guidance/caveats/"ignore xyz"/etc.

>Its quite likely they now found the counterexample with a more serious prompt, and then for virality re-tried a few times with meme-prompts

i am not sure why this is "quite likely". it'd be pretty silly to get a mathematical breakthrough and then hide it for an undisclosed amount of time to get a few more likes on a tweet, when the impressive part is the breakthrough.

not saying your theory is impossible, but i think the simple answer is that the model is just smarter than o1 and o3.

and, in any case, the model ended up getting the result with the meme prompt and "keep going", which was what i find fun. just like how the crypto results were from prompts of, more or less, "keep going", and that's pretty damn cool.

Not everything is a conspiracy