Is AI reasoning right for the wrong reasons?

Dijkstra said[1] "... the question whether computers can think. The question is just as relevant and just as meaningful as the question whether submarines can swim."

I don't see a clear demarcation of the things that only "reasoning" can accomplish and can't be approximated or imitated by other methods, and so I think the question is simply not meaningful or relevant.

[1] https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD867...

  On the other side of the AI-reasoning fence, the disdain seems to be mutual. “These ‘scientific’ papers from last summer — I would put this in big, big air quotes,” said Sébastien Bubeck, a member of OpenAI’s technical staff (and a prominent evangelist for the company’s reasoning models among scientists and mathematicians). He called earlier Apple results critiquing AI reasoning “wrong,” claiming that they were due to a training quirk in models that are now obsolete. “Modern models starting with GPT-5.5 do not suffer from this issue,” he said. “It would be interesting to revisit those results.” (Apple did not make its researchers available for interviews.)

  The “think” part is what OpenAI, for one, is doubling down on. When I asked Bubeck if the splashy unit distance proof was produced with methods outside the LRM’s own chain of thought — perhaps with Lean verifying its results — he seemed to find the question almost nonsensical.

  “It’s not like we’re making a mystery of it,” he said. “We have released the chain of thought. You can just go and look at it. The whole point is that the model is reasoning like a human would. And when humans reason, we don’t use Lean.” Technically, OpenAI released a “rewritten summary” of the model’s chain of thought produced by two human experts using Codex, another OpenAI model. Since 2024, the company has not publicly revealed “raw” chains of thought from its reasoning models, a policy also adopted by Google DeepMind and Anthropic.

The point of the story was that classifiers can be right for the wrong reasons and almost inevitably are. At least there’s zero guarantee that the reason for making the prediction matches the human or “real” reason why it’s correct.

LLMs are classifiers, there is absolutely no reason to assume they’re any different, regardless of any reasoning tokens they emit. They do what their handler wants to see, that’s all, and that’s what they’re trained to do.

People often take this as a knock against them. It isn’t, it’s just the reality of neural network classifiers. The results speak for themselves and don’t depend on whether they “actually” reason, but all evidence says they don’t, or at least there’s no special reason why they would.

If I ask an LLM "what is an apple?" it tells me:

> An apple is the edible fruit of the apple tree, scientifically known as Malus domestica. It is one of the world's most widely grown fruits and is eaten fresh or used in many foods and drinks.

If I ask an LLM "what is a mundu fruit?" it tells me:

> Mundu is a tropical fruit native to Southeast Asia, especially found in Indonesia, Malaysia, Thailand, and Cambodia. It comes from a small evergreen tree in the same genus as mangosteen.

I've never eaten a mundu fruit. To me, an apple and a mundu fruit are categorically different. An apple is a fruit that I've held, touched, tasted, eaten, enjoyed, cooked with. A mundu fruit is an abstract experience: text, images, only slightly more real than a fictional fruit. I'm aware that mundu fruit exist, just as the LLM is aware the apples exist, but that doesn't make them exist for me.

"Existing in an abstract way" is how an LLM experiences everything. To an LLM, an apple and a mundu fruit are in the same category. The LLM has been trained on text about both fruit, it's seen images of both fruit, it knows everything that has been recorded about both fruit ...except everything that's important to know about a fruit.

Many of our issues with LLMs arise because from the LLM's perspective, nothing exists. If Claude accidentally deletes your production database, it may well apologize afterward, but only because an apology is statistically likely. It doesn't feel guilt like a human would, and the lack of consequences makes any action an LLM takes inherently frivolous. We want them to understand what's real and what's not, but without any lived experience perhaps that's an unreasonable expectation.

But some problems require a substantial number of pattern-matching and refinement steps. The problem is, we also have interference from the fact that the model is trained to model language using mostly non-reasoning data of varying semantic lengths. Because of that, it may stop generating text before the abstract refinements are fully completed, simply because the pretraining data tells it to.

So we have to additionally train models to produce "reasoning traces" so that the emulated recursion continues for longer than what is typically found in pretraining data, allowing the model to build richer and more complex abstractions and surface more concepts. The ability to split problems into steps and logically connect concepts is already present in non-reasoning models (the original CoT trick), because some of it exists in the pretraining data, but not enough to support much longer recursion (hence the premature stops).

As for whether it is "true reasoning" or not, I think that is just arguing semantics for the sake of it. LLMs can demonstrably solve various complex problems. Yes, they often make stupid mistakes, but don't we have the saying, errare humanum est? Don't humans make mistakes too? Don't we also have around 200 cognitive biases showing that we "simply pattern-match" too? I think we still cannot get rid of the Great Chain of Being idea.

It seems intuitive that continuing a correct output sequence is easier than the "discontinuity" of jumping from the input prompt to the output sequence.

What really happens is that we figure out something that works, sometimes inspired by some biological thing or neuroscience thing.

Examples: neural network, attention, reasoning, hallucination, agents, experts in “mixture of experts”.

And then we go to name it, and rather than reaching for some three letter acronym we sometimes borrow a more catchy term.

I almost never means the original research was confused about what is going on. And in some cases we eventually strip away things from the original, like in neural nets which used to have a more biologically inspired activation function but we found out that ReLU works just as well because the important thing was the non linearity not the sigmoid.

“We react to language in a way that is very anthropomorphizing. That’s just the way that we humans work,” Mitchell told me.

I can definitely confirm the last part. Every time I read the output of an LLM, I picture a person talking to me.

I won't get into the metaphysics of reasoning, however, the Sonnet is using an OODA loop. The difference which hasn't been gapped is that human reason and imagination (in the sense of Mr. Rogers' Neighborhood) can predict the consequences of the actions we take.

This ability to loop is much, much wider in Opus 5 than Opus 4.. I had to strain to get Opus 4. to do the wider OODA loop but Opus 5 does it out of the box. I needed to throw out all existing instructions, skills, guidance, moving from 4-* to 5.

[0] https://adamsohn.com/lambda-variance/

If you don't or cannot, then it's the kind of sufficiently advanced technology that looks like magic. Except it is the problematic type of magic, like X-ray. They "magically" let you see the bones inside bodies. But you discover that you shouldn't do it too often only when the damage has already been done.

[1] https://www.hrw.org/report/2018/01/09/dark-side/secret-origi...

1) LLM's written CoT might not always be faithful to the model's real reasoning process (true and important)

2) The "stochastic parrot" hypothesis, which the article reintroduces as "approximate retrieval" - ie, LLMs don't "really reason" at all, they just memorize a lossy encoding of their training data. This obviously raises the question of how LLMs can now routinely solve open mathematical problems, with no solutions in the training data by definition. The article handwaves this with:

"The model doesn’t have to learn or reliably apply a general reasoning process, Kambhampati said; it just has to absorb enough examples of what the steps look like to predictively mimic them on its way to “stitching together” a plausible result that can then be verified."

The problem is that "mimicking" training data to arrive at a "plausible" result gets you an incorrect-but-plausible-sounding "proof" of the Jacobian conjecture, which was famous for humans writing plausible-looking "proofs" that had subtle flaws. You can't disprove the conjecture through sheer luck (search space too large) or "approximate retrieval" (the only thing you'd retrieve are fake "proofs"; far more human effort went into proof than disproof) or by writing something "plausible" that just happens to be correct (Jacobian was famous for "plausible" but wrong); the model must be carrying out mathematical reasoning somehow, by any sane definition of the word, even if it isn't fully reflected in CoT. The article doesn't address this.

Now as far as the math stuff a quirk of that field is that it can be fully analyzed in token space. Because 2+2 is a matter of definition it doesn’t need empirical testing like biology or subjective social support like a claim about the causes of WWI

So somewhere between the fact that language encodes a lot more ‘concepts’ than we naively may realize, the power of statistical emergence via associations, and what pursuits can be fruitfully done in token space we can get a long way towards ‘intelligence’

It’s frustrating that anyone who says maybe we shouldn’t base our entire economy on this one thing until we understand it and what’s it’s useful is essentially labeled this way.

Anyway these discussions always make me think we are too generous to humans. I know very few humans who are good at reasoning. It’s surprisingly hard to just think really hard through a problem. We mostly intuit, act, repeat. It’s a rare thing for someone to deeply reason.

The other problem I have with this is that we apply the word “reasoning” here because we haven’t really got another language for it, so we anthropomorphise the llm because that’s our reductive mental model and then complain that it’s not human enough.

In one of the articles on this topic they state: > To further show that trace accuracy is only loosely connected to solution accuracy, we then train models on noisy, corrupted traces which have no relation to the specific problem each is paired with, and find that not only does performance remain largely consistent with models trained on correct data, but in some cases can improve upon it and generalize more robustly on out-of-distribution tasks

which actually maps somewhat to regularisation techniques in image processing where you might add noise to an image or drop data to make the model more robust to changes.

If that's the case, AI-generated code could also operate on its own logic. Right now, programming is still done by humans, not machines, which creates a mismatch. But maybe the true machine-generated code could be much closer to the machine itself.

When you code with AI, there's a subtle mismatch with human-written code. It's like human code is a clean ORM layer, while machine code is raw SQL queries—there's that kind of subtle impedance mismatch. If we ever reach machine-to-machine code, what would that code even look like? Would it still use classes and methods?

> Not so fast. A 2025 paper(opens a new tab) from Northeastern University and the University of California, Berkeley on frontier open-source LRMs showed that between 30% and 60% of their “thinking steps” had “minimal causal impact” on the answers the models produced to benchmark math questions. Chop half of them out, and a model’s performance barely suffers. “We want to be careful when we review these chain-of-thought prompts because they may not be linked to the final output,” said Weiyan Shi(opens a new tab), one of the study’s authors

It could be boiled down to: in 2025 this paper showed that "thought traces" in the models of the time could sometimes be inaccurate or misleading. Today they still might be, although OpenAI says actually they are accurate for their modern models, (based on internal research, rather than published research).

[1] https://arxiv.org/abs/2506.06941

If we use the existing tools for interpretability like circuit tracking, sparse auto encoders and the like, should we not be able to determine if the chain-of-thought in its logical consistency actually matters?

> Dimethyl(oxo)-lambda6-sulfa雰囲idine)methane donate a CH2rola group occurs in reaction, Practisingproduct transition vs adds this.to productmodule. Indeed"come tally said Frederick would have 10 +1 =11 carbons. So answer q Edina is11.

What’s going on here, for example? But what if this is the path of human reasoning too. You know, have you guys read Peter Thiel’s Antichrist essay? It’s very weird, man. Guy sounds off his rocker entirely.

But he’s super successful, right? Maybe world modeling doesn’t text represent well. By the antichrist maybe he means some notion of the collective voting for distribution of resources without contributing productive capacity and that that ends societies? Or maybe internal world models are just not text serializable effectively.

A thing I’ve recently been enamored of are effective world and coordination models that are not “true”. E.g. a tribe that believes the forest gets angry if they do not hunt united. Lots more like that in Darwin’s Cathedral.

It might seem a bit free association-y but the topic itself is that.

The reasoning tokens behind this comment: https://wiki.roshangeorge.dev/w/Blog/2025-10-12/Word_Magic

We're already well past the point of trying to correspond the internal "brain chemistry" of an LLM to a thing called "reasoning". True reasoning, if there even is such a term, is very clearly, empirically and historically, based in tool use and capability use. If you create an abstraction like lean, and programming languages to brute force, and systems to integrate with, that expands what the possibility of "intelligence" is.

There's further places to take this including the claim that intelligence lives "outside" the subject - fine, we can or can't debate that. Even if we drop that question, it's pretty clear that the agent doesn't need to have deep intrinsic structures of XYZ, if it can just attach to tools and compose them to achieve results.

For example, I've said before that a well known fact of LLMs is that they steer their tokens to the right input distribution, that's why they yap so much in reasoning (this has been proven in studies). At the same time, don't mistake that for the whole process. Are they steering themselves to the entire a priori reasoning chain, or are they scaffolding with intermediate experiments and results, writing them to memory notepads, etc. etc.

That changes the metric of intelligence you're trying to measure.

And no I'm not saying, "OK, then have the LLM use only tokens, no tool calling, no nothing". I mean, we can do that, sure. But any intelligent agent has to interact with the world - and my claim is that maximally intelligent agents won't put effort into a priori reasoning, but rather a more balanced approach that outsources said "intelligence" through abstractions.

This is a brillaint way to simulate reasoning, but its likely not how we reason ... simply how we store reasoning in writing.

Its useful if you know how to use it, its dangerous if you think its more than that.

But tl;dr it can (since its uncompressing our lingusticially stored reasoning from books) arrive at reasoning a DIFFERENT way than our brains did... and this isn't right or wrong.

Where it diverges is when it must move beyond the text or even the synthetic possible text of all vector spaces combined (aka novel territory) and it can't conjecture or test those outcomes well. But to be fair, neither can MOST humans.

Perhaps you've reified it too much.

That’s all.

Long ago I was struck by an idea I heard about human reasoning that I paraphrase as "human reasoning evolved not to reason, but to provide reasons"[1]. Basically that the heaviest evolutionary pressures on our brain were for social utility, like for influencing others to do something for us, or giving pre/post justifications for our actions that most others are likely to accept to avoid punishment. The flawed mechanisms that we developed to do such things are not significantly grounded in logical reasoning, but can sometimes be pressed into duty for that.

LLM models are trained to "generate the best next token" which augments the response so far and makes the evaluator happy. When I reflect on my own mind responding quickly to someone in a meeting it doesn't feel altogether different from that. We don't typically carefully and fully logically reason before we start talking. We typically start talking and think only a little ahead about what we say next that supports and doesn't contradict what we've said already. As we're speaking we are monitoring the other person's facial expressions to infer their emotional response and adjusting our next words based on that. I think at least common neurotypical[2] conversation might be closer to the "next token" reasoning than we would be comfortable admitting.

[1] I _think_ who I heard say that was Hugo Mercier, on either Sean Carroll's consistently amazing science podcast or maybe on Lex Friedman before he veered into Joe Rogan emulation. I'm normally a little suspicious of the theories of psychologists and cognitive scientists, but I intuitively related to many of his lines of thinking.

[2] I'm arguably pretty neurodivergent in more than one way so I think my thought processes are a little different by default, but that makes me more reflective about common conversation. When I'm trying to mask and communicate more similarly to a neurotypical corporate employee I feel like I become a "stochastic parrot" that's just predicting the best next few words to improve the emotional response I'm reading in my conversation partner.

LLMs are one-trick pony's - they use the past to predict the future (presumed to be the same as what they were trained on). i.e. they are trained as auto-regressive predictors.

LLMs learn two slightly different types of reasoning via two different types of training.

1) SFT, or even base model training, on data that contains reasoning traces, learnt via next token error feedback. This does not result in "stochastic parroting" in the naive/pejorative sense, but nonetheless is very context dependent, even if the usual generative multi-source mashups apply.

2) RLVR post-training, where the model learns to mimic long-horizon (not just next token) reasoning via boosting a sequence of next-token predictions that steer the output towards a verified reasoning step (i.e. one that was at least valid in the context of the RL training sample). As Karpathy has noted, this is a pretty crude mechanism since you reinforce everything - errors included - that lead to the verified outcome.

RLVR is more powerful than SFT, and can result in more generalizable reasoning, since it is operating at a higher level of entire long-horizon reasoning steps, and also critically because it is most successfully being applied in the domains of math and coding which are highly self-consistent and logical. A reasoning step that was valid in one context should be equally valid in another context as long as you have successfully learnt what that generalized context is. Therefore, in these domains, you can chain together sequences of individually learnt reasoning steps, and hopefully this "novel" assembled reasoning chain is valid as a whole.

So, what is still missing from LLM reasoning compared to human reasoning? No doubt humans reason by memory a lot of the time too, and reductive axiomatic math reasoning works just as well for humans as when automated. So, what's missing?

There seem to be two major things.

1) RLVR requires rewards, and how well it works is going to depend on how accurate those rewards are. Is this reasoning step actually valid, or does it just kinda look ok? When moving beyond the cold reductionist logic of math and coding, the notion of correctness is far weaker, and it seems the best you can do is train on human curated reasoning rubrics and LLM-as-judge, which is much more fallible, leaving the model really needing (but lacking) a fallback to more general reasoning, not just memorized "maybe correct" reasoning steps.

2) Whether for reasoning outside of math & coding, or even within these domains when hoping for super-human innovative reasoning, not just lego-assembly proofs, what LLMs are lacking is a mechanism for what to do when next token/next step prediction fails. What LLMs currently do is "hallucinate", not even recognizing the failure.

In the human brain 50% or more of our cortex is feedback paths and the machinery that (perhaps together with the archaic part of our brain) lets us recognize and respond to failed predictions in an adaptive manner. This starts with continual learning (prediction failure being the signal), but also includes critical innate traits such as curiosity, boredom and frustration, that provide impasse resolution by encouraging us to explore unknown environments/contexts, abandon exploration when it is not productive, and generally expose ourselves to learning situations.

The dream is for AI scientists making new discoveries - the AI that could have invented general relativity if it has lived in Einstein's time, but this is not going to happen until their reasoning stops being purely predictive and becomes creative as well - curious about their own knowledge gaps and pursuing them in directed fashion, etc.

The current crop of Erdos solutions etc, while useful, really just represent the "generative closure" of what can be done/discovered WITHOUT learning anything fundamentally new. These will no doubt continue for a while until the more exhaustive search supported by computers has found the majority of these unexplored paths, and then we will need to move beyond LLMs to more brain-like architectures and algorithms that have the capacity for real innovation and discovery.

This seems highly dubious. You can't just memorise the form of mathematical proofs and then produce a valid one by feeding plausible looking BS into a verifier until it works. That's like saying a cargo cult will build a working airport if it just tries enough times.