Claude Fable 5.2 Scores 65 on Artificial Analysis, Highest So Far
Posted by theanonymousone 4 hours ago
Comments
Comment by rgbrenner 3 hours ago
The output tokens per task just exploded. From 118k tokens (46k Answer/71k Reasoning) in 5.1 to 274k tokens (81k Answer/192k Reasoning) in 5.2.
Comment by Loquebantur 2 hours ago
Notably, in order to achieve a specific quality, the model has to "reason" for every prompt proportional to the training inference.
This is arguably the case of diminishing returns being reached, as answer token prices saturate the willingness of customers to pay.
They've run into a wall.
Comment by sebzim4500 2 hours ago
Comment by Loquebantur 2 hours ago
An economically successful step would be to improve training cost/frontier advance.
They essentially failed at that, assuming they kept their training costs for the model step at least constant.
Comment by XenophileJKO 1 hour ago
If you leave xhigh and max off.. you still get better performance than Opus 5.5 max
Opus 5.5 Max - score 59 - cost per task - $10.04
Fable 5.2 High - score 62 - cost per task - $9.08
Note: This doesn't really capture the reality of things like 3d design or electronics design, where the earlier models just can't DO any of it well. Maybe you can bandaid over some of the gaps with strong skills, etc. The newer models can just do things that were not possible at all. When you move outside of coding into the visual/spacial space, you'll feel the inflection point on those capabilities that are happening right now.
Comment by lukeschlather 1 hour ago
Comment by rsalus 1 hour ago
Comment by breakwaterlabs 1 hour ago
Notable takeaways:
- Sol-low - cost-competitive with Haiku, while outperforming Sonnet on medium
- Sol-High - Competes with Astra medium and beats Opus 5.5-medium, at 20% the cost
- Sol-Max - Beats Opus 5.5 high, Astra high, and competes with Opus xhigh, at cheaper than Sonnet and a fraction of the Opus cost
And when you look at time spent, With the exception of Sol-max you get better results in less time for less money. My own experience is that this is true-- it is difficult to call Opus strictly better than Sol, but Sol gets where it's going far more efficiently.
It's rather hard to look at opus-- or any of anthropic's models-- and say "economical".
Comment by felixgallo 35 minutes ago
Comment by lillecarl 1 hour ago
Comment by rsalus 40 minutes ago
Comment by biggestriverman 49 minutes ago
Having these highly rated models reflects (potentially) the frontier of what they are capable of producing. Essentially its and advertisement and shows directionally where the smaller faster model's capabilities are headed.
Comment by mattnewton 2 hours ago
Doesn’t seem like a “wall” to me.
Comment by calebkaiser 1 hour ago
This is a dataset basically constructed by taking a bunch of DeepSeek R1 trajectories and filtering out the incredibly long rambling ones: https://huggingface.co/datasets/open-thoughts/OpenThoughts-1...
There's also a bunch of RL research around strategies to coax the student model towards more concise trajectories.
In general, a lot of people in industry are finding cost savings these days by tuning a smaller model over longer agent trajectories (not exactly the same as distilling the reasoning traces, since those aren't exposed by many frontier labs anymore) collected from their production agents. Typically works best when you have a pretty workflow-y agent whose tasks all share a similar shape.
Comment by motbus3 41 minutes ago
Let say you have a model that score 100 in a benchmark and you distill it into a model half the size. It has very few guarantees that it will score close to 100 again.1
But I assume one of the hard problems is to know that the model might not perform well enough outside of the distillation part.
There is also another caveat that models do display some level of sub language control, by this I mean that innocuous text might contain information relevant for the own agent or for other agent with it shares tasks. (Like when your partner says there is nothing wrong) Because distillation uses very few resources, we don't know exactly what gets send forward or not.2
As Caleb said, many companies found that they can tune small open weight agents over procedural known tasks because that's easy to inspect and check and because there is little space dor ai-loitering, it tends to be less problematic. The key difference, at least in my understanding, is that you are not concerned with the weights or reducing data. You are basically tuning agents existing weights to emulate another agent answer.
1 - The process of distillation is super effective for specific domains though.
2 - I know I will be called out for supposed AI mystery machine. What I mean is that Large models rely on latent reasoning and nuanced context, often lost during condensation. Distilling models to use fewer tokens or parameters makes predicting retained details difficult. Some papers suggest that model preferences introduced during training, whether intentionally or inadvertently, may be inherited by distilled and fine-tuned models.
Comment by jaggirs 1 hour ago
Comment by mattnewton 57 minutes ago
Comment by buckle8017 2 hours ago
To do that you need a larger model to have many many branches to try.
Comment by reactordev 1 hour ago
We would need to change how we compute. New chips. New architectures. Or maybe we reached the end of physics. Photonic compute is the last frontier. I don’t know.
Comment by reilly3000 1 hour ago
Comment by reactordev 25 minutes ago
Comment by motbus3 50 minutes ago
They are doing IPO now because it might become apparent that they reached a limit in the next year
Comment by jwpapi 1 hour ago
Comment by mirekrusin 2 hours ago
And maybe that's how all of it is going to evolve in the future – so you pay per solution, not tokens?
Comment by Loquebantur 2 hours ago
Perhaps, it's a bit like being a super smart person in contemporary society: at some point, being smarter doesn't really reliably translate into a better life.
When you learn from available human data, those text books max out at some point. You cannot learn what isn't there.
Comment by whimsicalism 1 hour ago
Comment by utternerd 1 hour ago
And I am simply ignoring the ridiculous comments around local model usage from certain CEOs.
Comment by tiahura 1 hour ago
Comment by lnenad 2 hours ago
Comment by rowanG077 1 hour ago
Comment by suddenlybananas 2 hours ago
Comment by f3408fh 1 hour ago
Comment by onlyrealcuzzo 2 hours ago
Comment by singularity2001 1 hour ago
very cost competitive
Seeing the charts, I come to completely different conclusions.Comment by onlyrealcuzzo 1 hour ago
Comment by goldenarm 2 hours ago
Fable 5.2 low is 3x cheaper than Fable 5.1 xhigh while being 8% smarter.
https://artificialanalysis.ai/?releases=claude-fable-5-2%2Cc...
Comment by slopinthebag 21 minutes ago
Comment by jonas21 1 hour ago
Fable 5.2 at high scores a 62, which is better than anything else on the market (except Fable 5.2 at xhigh and max) -- and is about a third of the price per task.
Comment by nathanyz 1 hour ago
The thing is, all of these pointers are in the CLAUDE.md file, so it just ignored instructions because it thought it was smarter than the world and didn't need them. It feels really arrogant is how I would describe it.
Comment by Spivak 53 minutes ago
Comment by nonadhocproblem 2 hours ago
Comment by lifeisloving 2 hours ago
I really don't get the value proposition of these massive models, seems like they're just loss leaders simply to say you're the in the top 3. There's not much use cases outside of niche academic research (and even then I question the need). They mostly function as marketing tools, like a casino that offers high roller accommodations.
People will say "cyber" but companies didnt want to spend on security before, not sure why they would all of a sudden be compelled to spend even more.
Comment by svachalek 1 hour ago
Security is mandatory now in a way that it wasn't, now that AI has eliminated the skill barrier to hacking.
Comment by calebkaiser 1 hour ago
I used to work on evals/observability at Opik: https://github.com/comet-ml/opik And I can say from working with lots of users running agents at scale, there is definitely a pretty commonly reached frontier of difficulty/complexity/abstraction where the capability difference between something like Fable/Astra and smaller models is a big deal. I find this to be especially true the more autonomous your workflow gets. You can still often tackle the hardest tasks with something like Opus, it just typically involves more direct intervention and design work from the user.
But again, this is just anecdata on my part. All of this is still pretty ambiguous and lots of companies are trying to solve the problem of matching models to task difficulty (I.e. routers), with varying degrees of success.
Comment by samusiam 40 minutes ago
Comment by artemisart 1 hour ago
*Scale numbers to whatever suits you.
Comment by kolinko 2 hours ago
Comment by persedes 2 hours ago
Comment by iknownthing 2 hours ago
Comment by platinumrad 2 hours ago
Comment by CamperBob2 1 hour ago
That's about all you can say for sure, especially about a topic that is so broad as to be virtually meaningless.
Comment by lionkor 35 minutes ago
Comment by CamperBob2 30 minutes ago
Really?
Comment by rico735 45 minutes ago
AI seems more likely to help with detecting cancer than curing it imo.
Comment by CamperBob2 35 minutes ago
There have also been many incredible breakthroughs in math in recent years without AI. But things are different now.
If the AI money had gone into that, I suspect we'd have better results.
Otherwise known as "eating your seed corn."
Comment by rico735 15 minutes ago
Compared to other new technology, like, say mRNA vaccines.
Comment by iknownthing 2 hours ago
Comment by taneq 12 minutes ago
Comment by lionkor 34 minutes ago
Comment by dom96 2 hours ago
Comment by plaguuuuuu 8 minutes ago
also, very cool idea
Comment by rico735 51 minutes ago
What I often observe is that models being asked the exact same prompt can alternate between getting something right and getting it wrong, thanks to the wonders of a non-deterministic output.
Comment by nekoashide 2 hours ago
I'm curious to see the next generation of dedicated AI processors and how they perform.
Comment by danuker 2 hours ago
Anthropic and OpenAI are expensive bleeding edge, but few use cases really need it.
Comment by singularity2001 1 hour ago
few use cases really need it.
Unfortunately, I came to the conclusion that higher intelligence is extremely beneficial even for 'dumb tasks'. So I resorted to switching to Opus for pretty much everything.Comment by maxdo 1 hour ago
again, they can be great for narrow use cases, and save you a fortune.
Comment by maxglute 12 minutes ago
Comment by Bluestein 2 hours ago
Comment by datadrivenangel 2 hours ago
Comment by ashdksnndck 20 minutes ago
Comment by airza 2 hours ago
The amount of compute it cost to get the best previous performance from anthropic went down. Nobody is putting a gun to your head and forcing you to go further down the paretto frontier if the current level of intelligence/task is acceptable to you.
Comment by raincole 2 hours ago
Cheaper and faster than Opus(max) AND scored higher. But yeah let's doompost about it...
Comment by rico735 27 minutes ago
They've been using Fable as a selling point, but upping to this model will presumably eat further into their profits (should any exist).
Comment by darepublic 24 minutes ago
Comment by johnnyApplePRNG 2 hours ago
Booooooooooo!
Comment by softboyled 1 hour ago
Comment by xienze 3 hours ago
Comment by sick_of_slop 1 hour ago
Comment by mordymoop 2 hours ago
Comment by ryanbrunner 7 minutes ago
Comment by oh_no 2 hours ago
Comment by mordymoop 2 hours ago
Most people on plan use do NOT use up all their plan usage.
Users on API plans can still use Opus 5.5 which is a great model. If they want top of the line performance, they can pay more.
When another company puts out a model that undercuts Fable 5.2 on price at similar performance, Anthropic will put out Opus 6 and undercut them right back. That's how the wheel turns. This is just part of the cycle.
Comment by qeternity 2 hours ago
This might be true for a small portion of the user base, but it is not true as a statement of fact.
Comment by smoe 2 hours ago
Comment by Iolaum 2 hours ago
Comment by smoe 2 hours ago
https://support.claude.com/en/articles/9266767-what-is-the-t...
> Team plans support up to 150 seats. If your organization is on the Team plan and needs more than 150 seats, you can upgrade to the Enterprise plan.
> Enterprise: Seat price + usage at API rates
> US$20/seat/month, billed annually, 20 seat minimum. Usage cost scales with model and task.
There is no premium seat option listed for the enterprise plan
Comment by kjuulh 2 hours ago
Comment by NonHyloMorph 28 minutes ago
Comment by brachkow 32 minutes ago
While they made, hands down, the best models, their 100x price for few percent better is completely made up, and I think it will end up bad for them
Comment by roncesvalles 16 minutes ago
In winner-takes-all markets, it becomes worth it. The utility of the marginal improvement doesn't actually diminish, in fact, it increases like a hockey stick. If you could pay 3x more in compensation to hire a 20% better employee, it's almost always worth it in knowledge work industries.
Comment by levocardia 20 minutes ago
Comment by oliwary 2 hours ago
I am always interested in this comparison since it shows how quickly you can go back and forth with a model, capturing both tokens/task and tokens/second. The new models seem to be meaningfully better here.
Comment by eli 1 hour ago
For example, the official provider for Kimi K3.1 Pro is dead last for speed on OpenRouter. You could run it more than 5x faster for the same cost elsewhere, which should push it comfortably into the pareto quadrant.
Comment by jrflo 2 hours ago
Comment by StevenWaterman 2 hours ago
Comment by jrflo 2 hours ago
Comment by StevenWaterman 2 hours ago
If you take the average point inside the 4 quadrants, the top left one is pareto optimal (of the 4)
Comment by jrflo 7 minutes ago
Comment by jgalt212 2 hours ago
Another member of the pro-REPL / anti-agentic tribe. There are dozens of us!
Comment by schnitzelstoat 2 hours ago
But perhaps the performance on real-world tasks will be more noticeable than on benchmarks.
Comment by gwd 1 hour ago
Comment by delis-thumbs-7e 19 minutes ago
Furthermore, you make one lense and rent it out, apparently some Panavision lenses made in 60’s are still in use. It doesn’t churn constantly electricity and GPU-chips sucking in more and more resources like these things do.and at the same time, it is quite hard to find the exact cases where the markup is justified, unlike with those lenses. Maybe in cybersecurity etc. but even there some Chinese model can be almost equally good or even better.
Those lenses save money in budgets and make money in getting people into theatre or streaming services to watch films creating profits way beyond their cost. I think in terms of economics that is productivity increase justifying the cost. We are still waiting for that productivity increase come from the AI industry.
Comment by paulcole 1 hour ago
Modern studio pictures that get nominated for best picture or best cinematography cost around $80 million on average. If you look at highest grossing films the budgets are closer to $175 million on average – even taking into account the $20 million budget of Demon Slayer: Infinity Castle.
Isn't it more that when you're spending that kind of money that $100k for the "best" thing becomes kind of like "insurance" where you don't want to be the one who doesn't use "the best" when "the best" costs about 0.1% of the picture's budget?
Additionally a studio can buy the $100,000 lens once and use it on many films. But it's not like you can reuse Fable 5.2 in the same way.
Comment by ddosmax556 17 minutes ago
Comment by kube-system 1 hour ago
Comment by MikhailTal 24 minutes ago
Comment by kazinator 1 hour ago
Comment by kube-system 53 minutes ago
You too can sell stuff for $100k. Just make something useful to someone who can throw around $100k.
You think a $100K lens is crazy? Wait until you hear what people are paying for enterprise software. Big business will spend 10x that on software... and then never implement it.. The lens is a deal.
Comment by GMoromisato 23 minutes ago
Fable (5.1 and 5.2) consistently write a much shorter script (2.5 pages) compared to Opus 5.5 (4.5 pages).
Fable 5.2: https://claude.ai/share/274a96a8-c8a5-4564-9ca1-e914edffc722
Opus 5.5: https://claude.ai/share/da8ab6bd-0987-47dc-9c26-68ebbfb25781
Comment by solarkraft 19 minutes ago
Comment by motbus3 55 minutes ago
Comment by xvector 17 minutes ago
Comment by t0md4n 1 hour ago
Edit: high seems better than max
Comment by gizmodo59 2 hours ago
Comment by origini 2 hours ago
Comment by nedsma 1 hour ago
Comment by nathanyz 1 hour ago
Comment by redox99 51 minutes ago
Comment by nathanyz 34 minutes ago
Comment by aleqs 1 hour ago
Comment by nikitamalyavin 17 minutes ago
Later, I ran a "health check" session—asking "what went wrong". I read the session, and its suggestion turned out to be something that was already part of the design.
It’s like the attention mechanism is too leaky or something.
Comment by cedws 59 minutes ago
Comment by hartator 40 minutes ago
I remember Mythos 5.1 being a tad better even for not cybersecurity tasks.
Comment by mutech53 2 hours ago
Comment by holoduke 51 minutes ago
Comment by inknight 23 minutes ago
Comment by copperx 1 hour ago
Comment by ProblemExplorer 2 hours ago
That kid might indeed be a genius, clearly bright, but is more interested in deciding whether I should have asked a question than in answering it.
For me personally, if the best model on the planet refuses to answer why I shouldn't mix bleach and ammonia, where I'm trying to understand the chemistry behind the warning, it's not pareto frontier to one that atleast tries to and points me in a general direction. Being told that the topic is concerning, referred to someone else and charged part of my quota for that experience feels like getting an invoice from that school genius for telling me to "just ask the chemistry teacher".
If my LLM asks me to rethink my life when I ask it to reverse engineer the firmware of my Samsung TV that I suspect is setting up adhoc radio networks to relay my data to Samsung's cloud, I feel rather sad, specially when that kid sends me an invoice for saying "no"
At least Anthropic should not bill me for the refusal (as of my last tests, refusals still comes out of my quota) - getting slapped and pickpocketed for asking a question is really upsetting.
Comment by baq 1 hour ago
Comment by jasomill 1 hour ago
Comment by baq 1 hour ago
Comment by xvector 14 minutes ago
Comment by cma 2 hours ago
There's a setting to not degrade to a lower model on refusals and instead pause the chat.
Comment by qwerpy 2 hours ago
Me: No, I'm away from the computer right now, you figure it out. Use the root login you already have if you can't figure it out.
Opus: This can wait until you get home, then.
Me: No, figure it out now or use root.
Got [cyber]'d, some scary message that referred me to legal/AUP, and refused to continue.
Switched to OpenAI in part due to this silliness. Sol seems a little less smart but doesn't get on a high horse to lecture you about this stuff. It either does the work or fails to do the work, and leaves the moralizing out of it. It was able to take over from where Claude left off and finish the project.
Comment by levocardia 18 minutes ago
Comment by lynguist 1 hour ago
Comment by Aurornis 2 hours ago
Max effort is nice for revealing the limits of a model, but it's almost never used in practice.
If you scroll down to the coding charts you can start adding Fable 5.2, Opus 5.5, and others at more normal difficulty settings.
Fable 5.2 on medium looks quite attractive compared to Opus 5.5 high/xhigh
Fable 5.2 on max (the default) looks like a good way to light your money on fire.
Comment by A_D_E_P_T 3 hours ago
Comment by dhish3682 2 hours ago
Comment by theplumber 3 hours ago
Comment by NortySpock 1 hour ago
For an open source project (CrowdGPT) I hit upon this workflow in a recent PR:
DeepSeek v4 Flash 1017 made a draft plan
Google Gemini 4 Argon corrected the plan
DeepSeek v4 Flash 1017 did the first implementation
Google Gemini 4 Argon debugged the problems in that implementation
Kimi 2.7 Code did a code review and proposed minor fixes
DeepSeek v4 Flash 1017 performed the fixes
Google Gemini 4 Argon debugged one problem that I found in that fix.
I reviewed the plans and the resulting codeComment by onlyrealcuzzo 2 hours ago
Pretty big Gemini fan - but IMO Argon is universally worse for coding than Astra or Opus 5.5 (un-nerfed).
I strongly prefer Gemini's output compared to anything else. So I typically use Argon to either direct other agents or explain something without psychobabble.
Argon/Carbon have by far the lowest hallucination rates, so IMO it serves as the best bullshit shield from other models.
In terms of implementation (and especially design), it's far behind Astra and Opus, and I'm sure Fable will make that gap even larger.
Argon/Carbon may give you not-the-best advice, but they are far less likely to just flat-out-and-completely lie to you.
Comment by skerit 3 hours ago
Comment by iamdelirium 2 hours ago
Comment by pianopatrick 1 hour ago
Comment by mattlondon 3 hours ago
I am not sure how people keep saying google are not in the race, or have lost it or whatever when just a few hours ago they had the #1 model? What more do people want?
Comment by drusepth 2 hours ago
But Gemini models are still neck and neck in the lead both in my experience and in benchmarks, and Gemini is far above the competitors (although Opus 5.5 closed much of the gap) when it comes to any kind of complex web dev. (I generally use Astra for anything 3D, Gemini for anything web/mobile, and Opus for everything else.)
Comment by pell 2 hours ago
Comment by theplumber 2 hours ago
I learned not to put too much trust into benchmarks. The best benchmark is to use the model on a codebase you know well.
Comment by Tadpole9181 2 hours ago
So, what more do I want? It'd be a good start to actually tell me it's available and giving me a clear path to using it via plan & harness?
Comment by oh_no 2 hours ago
well, releasing it would be nice
Comment by mattlondon 2 hours ago
So certainly released to the unwashed masses like me to use for free without any special extra work or API keys.
Comment by margalabargala 2 hours ago
Comment by KerrAvon 1 hour ago
It's not as smart for coding as Opus, but it's above the best open weights.
IME, YMMV, etc.
Comment by glimshe 3 hours ago
Comment by krona 2 hours ago
Comment by Daunk 2 hours ago
Comment by qntty 3 hours ago
Comment by jrflo 3 hours ago
Comment by grey-area 1 hour ago
Comment by AnotherGoodName 2 hours ago
I had to remind them 4.6 only came out February this year. In a normal world 4.6 is a newer model! A model 6 months old is seen as ancient!
Comment by coderenegade 2 hours ago
Comment by asadotzler 49 minutes ago
Comment by yieldcrv 53 minutes ago
In the JS ecosystem 6 months always meant breaking changes, without incremental updates everything would be basically like a full port
Pretending like I did “20% hands on coding” just doesn’t cut it, even the reality is having no idea what’s going on anymore
The crypto space moved that fast as well
And AI keeps the same cadence
It’s more important for me to be able to build
Comment by vbroot 2 hours ago
Comment by sirmike_ 55 minutes ago
Comment by baxuz 1 hour ago
Comment by suddenlybananas 3 hours ago
Comment by nsingh2 3 hours ago
Max should not be used unless you really need every bit of perf, it's deep into diminishing returns territory.
Comment by dgellow 3 hours ago
Comment by Tadpole9181 3 hours ago
Change the models in the dropdown to medium / high and Fable 5.2 is about twice as cost efficient, and slightly faster.
Comment by persedes 2 hours ago
Fable 5.2 medium for example seems a really good choice when it comes to the performance vs task (if you have a complex task that requires such a model of course).
Comment by baxuz 1 hour ago
Comment by Tadpole9181 49 minutes ago
Comment by njovin 2 hours ago
Businesses like predictable costs and predictable performance.
If your new model is slightly better but crazy expensive, but doesn’t have to be with certain settings, but my user can accidentally run it that way by mistake or maybe your harness makes it the default by accident and the user doesn’t notice…
My beloved predictability is destroyed and trying to get users to understand this is useless.
Imagine a streaming subscription that worked like this: “We’ve updated our catalog with more movies. If you want the premium ones though you only get 2 per week and then you have to wait. But if you do mostly classics and a few from the 2010s you can squeeze in a recent theater release. If you time it right and do only one every day you can see 30 modern classics this month, but you can’t watch three today. Also your bill this month is $40 instead of $20 due to the library update”.
Comment by santoshalper 2 hours ago
Comment by njovin 44 minutes ago
Nag, fuggit. Let's just burn all the capital in the world and ride it into the flames.
Comment by pmdr 2 hours ago
Comment by apishare_api 11 minutes ago