GPT-5.6 vs. Claude Fable 5 for Physical AI, which performs best?

Posted by mbauman 5 days ago

Counter100Comment21OpenOriginal

Comments

Comment by xnorswap 5 days ago

A really frustrating partial presentation, given an apparent lack of testing with a spread of efforts for each model.

Given that there's no reason to believe that Fable's xhigh is comparable to GPT-sol's xhigh, or Opus xhigh, for that matter, it would be far more useful to see the effort level where these tasks no longer achieved their goals.

Comment by mrr7337 5 days ago

These benchmarks are done with incorrectly and missing a lot of baselines. The article seems very vibe coded too.

Comment by draginol 4 days ago

So Fable "won" but it cost $124.76 for marginal performance benefits over the $22.56 5.6 Sol run.

Comment by JumpCrisscross 4 days ago

I'm finding a happy middle ground using Fable to proof work other models have done.

Anthropic's guardrails still make it garbage. But eventualy it will leak.

Comment by cft 4 days ago

I just spent 50% of my weekly Sol Ultra on a $100 plan to purge Claude worktrees on something that may have tripped Fable guardrails.

Comment by giwook 5 days ago

Please forgive my naivety, but are world models (once they are in a consumer-ready form) expected to outperform any currently existing LLM on these sorts of tasks (i.e. of the physical world)?

Comment by GuB-42 5 days ago

From what I have seen "world models" are more like video models. The idea is: from a video clip, predict the next frame, like LLMs predict the next word/token from a bit of text.

The idea is that these models should be able to internalize the laws of physics and properties of objects just like LLMs do with grammar rules. To make training more efficient, frames from game engines with hardcoded physics are used. The application is mostly robotics, including autonomous vehicles.

This is a bit different, the idea is to write a physics engine, it means numbers: like for an airplane model, calculate the climb rate. The application is mostly science and engineering.

Comment by lucrbvi 5 days ago

"world model" is, sadly, a new buzzword; most of the world models we are seeing are video generative models (like GPT-Image), they receive an action vector and previous frames and must generate the next frame. It's working of course.

The whole field of world models also have lighter approaches based on using latent representations as the target (and not pixel frames). LeWorldModel[0] and DINO-WM[1] are a good way to learn how these works.

Strangely, most of the startups in the space of world models are focused on video-generative world models.

A good application of world models is in robotics where an action policy generate a distribution of actions, you can then predict the future state of the world and use some reward policies to choose the "best" future state.

Comment by hartator 5 days ago

It's kind of interesting this is already out of data as it's missing Kimi 3 and Opus 5.

Comment by StefanKarpinski 4 days ago

Yeah, things are moving fast. Those benchmarks will be out soonish. Takes a while to run these things.

Comment by pohl 4 days ago

Seemed strange they would bench Terra and Luna but not Opus, Sonnet, and Haiku

Comment by StefanKarpinski 4 days ago

They were all benchmarked, but not included in the post to try to keep the amount of data from being excessive. (Results are also not surprising — Opus good, Sonnet struggles, Haiku fails.)

Comment by arisAlexis 5 days ago

Google with apptronic should have good models soon

Comment by esperent 4 days ago

I'm curious how good the terra/luna results are but there's not enough context here for me to judge 0.786 (terra) vs 0.814 (sol) vs 0.889 (fable).

How do the two lower models compare to e.g. gpt 5.5 or 5.4 or various opus models?

Comment by grim_io 5 days ago

I'd expect google to do well here, since they were historically strong at multimodal and physics.

Comment by Archit3ch 4 days ago

I wouldn't trust Gemini anywhere near my code.

However, it's pretty good at regurgitating textbook answers (classic Google...). This makes it an asset for circuit design. Try one of my real queries with Gemini Pro:

"Create a differential current splitter that splits a bias current Ib into a positive current Ipos = Ib/2 + ky*Vy and a negative current Ineg = Ib/2 - ky*Vy where Vy = Vy+ - Vy- is a differential voltage input Y."

Comment by jespinel 5 days ago

Nice! It is missing Codex in the agent harnesses comparison IMO.

Comment by effnorwood 4 days ago

Define "best" and "performs"

Comment by busssard 4 days ago

compare to opus-5 ?

Comment by gizmodo59 5 days ago

Yet another "benchmark to promote their own harness"

Comment by DwarvenEngineer 4 days ago

Honestly, I just hate the term "physical AI". They're robots. It's unfortunate that we had to adopt a term with the words AI in it, just to get investor's attention.

Comment by 4 days ago