AMD acquires Taalas to boost inference performance by etching models in silicon

The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I belive higher amounts of SRAM ? )

I don't think this works out from a cost/silicon perspective. Small models already run pretty well in software (since the weights fit in cache) and big models require silicon area proportional to the size of weights. On a mobile device putting a chip like this is competing directly in BOM and power against a whole lot more l3 cache, and the l3 cache makes everything faster

That's actually a really good point... There's currently zero incentive to buying more hardware, and that's one very good reason do have a new one.

From what I remember, these chips are not mobile size yet

Slightly besides your point, but it's interesting how many here naturally ponder about how the current winner could or "should" keep winning, instead of how another company could become a competitor by doing the more clever thing the incumbent isn't thinking about.

Apple is somewhere between fashion company and second rate tech company.

They could have 9 year old AI and still post profits.

Not sure if it's my pixel or android, but I made a randos jaw drop with what the crappy AI on android can do.

When are we getting android OpenClaw?

If you’re only running models for frontier capabilities, yeah. For tasks where current models are smart enough, running them 100x faster is the most impactful improvement you can make. Consider all the things you could use a model for, but don’t, because the latency is just a bit too high.

It depends on how quickly you can bake new architectures.

Text diffusion might be a disruptor here, but let me just say the most cutting edhe form of image diffusion (JiT and DiT) right now is just a big fat stack of alternating attention and MLP matmulls. Not theoretically hard to bake

I'd gladly pay for a Claude Opus 4.6 Thinking High in silicon and use it for 1-2 years. It's good enough for many coding tasks.

Depends on how much it costs the consumer. If I could buy a "cartridge" of Kimi K3 for 300 bucks I 100% would buy that shit asap. Even if it's "no good" after lets say 4 months still would be worth it IMO.

"seems like baking models into silicon is speed-running obsolescence"

Now maybe. When models are flying passenger aircraft, other prerogatives will assert themselves. When a 50TB ROM means you can impulse purchase a ChatGPT 6.3 xhigh that runs on batteries, yet more use cases will be apparent.

I could see this making sense when model development start to settle down ... it's going to settle down, right? ...

Not sure. You can fix the transistors but leave the connections between them open for flexibility, so you only need to change the manufacturing process for the upper masks for every new model.

Compute the cost of producing n of them devices, imagine a fair price based on that, and see if that local, blazing fast card* can be an asset that could be replaced periodically.

*(It's local: private files managing firm oriented. It's blazing fast: it can be placed into recursive, intensive local workflows.)

Which is exactly what companies and shareholders want to increase sales.

Look at it the other way: compared to the cost of training a model, the cost of making a custom ASIC is trivial.

obsolescence is the whole point. apple gets to sell a new phone very 6-12 months because of it.

i have written about this:

"For device makers

Packaging models with laptops and smartphones will let application access near free, low latency inference and potentially offer users a better experience with the option of preserving data on-device. This is viable under the condition that tasks that do require larger expert models that run in the cloud can be routed to external models. A side-effect of local models and what will let Apple cut upgrade cycles from ~4 years (?) down to 12-18 months is specialized hardware to run them. For almost a decade, smartphones have been trying to compete on better cameras. This coming decade will see them selling better GPUs, NPUs, ASICs and whatever other things they'll be calling the inference chips, to drive re-purchase. Every six months will see a better model on new hardware, which will enable better performance in certain applications."

https://try.works/role-model-the-case-for-a-model-routing-pr...

> cheaper in all ways,

Bitcoin mining doesn't have large memory requirements, but does have huge compute requirements. ASICs work great there because it's very straightforward to add some circuits for computing hashes. If you _also_ have to add many GB of memory, then suddenly ASICs will cost as much or more than comparable off-the-shelf hardware and they won't be faster unless you've also invested in huge memory bandwidth.

Not necessarily: it is relevant to Taalas only if it is a compute-in-memory architecture.

The Jalapeño mentioned («Anthropic is not alone in walking this path») in the article is still a classical Von Neumann architecture.

And Taalas' idea makes sense in a perspective of scale - producing a large number of cards; "for internal use" (a lower order of items) means a high production cost.

It is really fast and ... really hallucinates. I asked "Does the Wang corporation still exist? If not, what happened to it?" and it replied (in part):

"Yes, the Wang Corporation, the company that originally developed and marketed the Wang 2200 computer, still exists as a rebranded company under the name PPL (Precision Pencil and Label), but it has undergone significant changes and challenges over the years.

Here's a brief overview of what happened:

    Founding and Growth: The Wang Corporation was founded by An Wang in 1969."

In fact, Wang labs was founded in 1951. PPL seems to be a made up entity. But it did generate those "facts" in 0.033 seconds. If people value speed over accuracy then I can write an LLM that is 100x faster than chatjimmy.ai and make big bucks by responding one of N canned responses to any question.

Pretty incredible to see. It reminds me of when I first used the Groq chatbot, except in this case it's a full response instantly.

No, that's backwards. OpenAI are the ones investing in Cerebras. Part of the deal is that they can't sell to Anthropic.

OpenAI and Anthropic are both designing ASICs.

Depends what you mean by relevant. If you use AI primarily as a search/knowledge engine, it makes no sense. If it's your capable assistant that has a lot of general knowledge, can do tool calls, and has a big context window, very doable.

Indeed, for some kinds of applications involving secure/legal data etc. I can see the consistency of silicon winning out, because it combines performance with immutability and guardrails in hardware. Some chips have write-once PROMs to store password hashes and similar, you could do the same thing with prompt hashing to absolutely force or forbid certain behaviors. A model that can't be updated is also a model that can't be hacked.

People already buy new phones every year, this just creates even more reason to do so

Base model sure, but the stack will be hybrid. It’s still early days here. Too bad FPGAs have such large feature size.

One of these chips smart enough to take orders at a drive-thru would be relevant for a decade, minimum.

Not so if it's embedded in something smart enough for its intended purpose.

Think vision, spatial reasoning, speech synthesis, even some speech analysis. Think self-driving cars (and drones) that need 10x less power for the brain, and can think at 10x situation per second.

This is only true for people who are solely focused on performance. There is absolutely a market for acceptable performance combined with predictability.

If a model is good enough today, it's still gonna be good enough in a year. Except you'll be able to serve it 1/100 of the price. Or 100x the speed.

OTOH, people get a new iPhone every year and they are ok with it.

A model you can run for practically no cost is a new proposition. It is the CPUification of AI. Sure there be supercomputers but you PC will be pretty super too.

What would possibly tell you that?

Cerebras is literally the entire wafer, so it can't get bigger. So where is the jump from 30x to 100x coming from? Node improvements only yield like 10-20% gains these days...

This.

So, like the present

It failed on my usual test. But it failed really fast:

"A farmer has a wolf, a goat, and a cabbage. The wolf is imaginary and doesn't exist. He wants to cross the river, but the boat is only big enough to hold him and one of them. The farmer can't leave the wolf and the goat together, because the wolf will eat the goat. Similarly, he can't leave the goat and the cabbage together, because the goat will eat the cabbage. What is the smallest number of trips the farmer needs to make to get everything across the river?"

I read the paste, it got the etymology wrong, no? Schlong comes from shlang (snake), not shlemp (is this even a word? I don't speak Yiddish but couldn't find it on Google).

Oxford also claim that its first recorded use was from the 60s, not the 20s; https://www.oed.com/dictionary/schlong_n?tl=true

I didn't realize there was a SchlongBench™ (but of course there is). What's it test? (asking seriously)

I had the same reaction but then I showed it to my partner. She completely didn't get it, in her words "how can it be thinking of a good answer when it's that quick?"

I tried to explain but I fear were probably going to be adding artificial sleeps to these things to convince the masses it's doing something clever.

I feel like Ray Kroc in the McDonald's movie trying to figure out how his hamburger could possibly be done when he just ordered it

For those old enough to remember, this is like dial up internet to broadband. So fast it creates new markets

If you still want the experience, go and browse McMaster Carr. Wizards designed that website.

or LiveGrep fast search of the Linux kernel source code with regex support: https://livegrep.com/search/linux

its fast but try to get it to give you pi to 50 decimal places. it didnt go well for me.

It's running a very small, non-reasoning model at the moment. But more generally, almost all LLMs argue on the hardware/model they are/are on.

AIs don't intrinsically know anything about themselves so they often give wrong answers to such questions. This can be fixed by putting info in the system prompt but they may consider it a waste of tokens since most usage doesn't benefit from that information.

That proves it's conscious!

(/s!)

Back in the kazaa and limewire days, you'd sometimes try to get a movie / episode from a series, wait hours / days for it to download, and when it was done you had a ~50/50 chance to actually watch what you wanted or an old german porn movie :/

This is a good point. Opus does some silly shenanigans sometimes but then catches it later. It’s still an order of magnitude faster at getting to a working system than I am, for ones I don’t know.

It’s really a dream for setting up a homelab

There's a benchmark for this and a lot of models get negative scores because they're so unreliable: https://artificialanalysis.ai/evaluations/omniscience

Can you give some examples of these tasks that require an hour or more of reasoning?

Makes sense. Actually to expand, I believe this can make a lot of sense for industrial robots and such which have a more or less fixed job and latency matters more, so a well tested model may be more valuable than need to keep updating them

You don’t need this chip to do that. Computer vision has used machine learning for decades. The task you’re describing is pretty rudimentary and an off the shelf model with a control system would do it way cheaper.

subscription "ai-powered" dishwasher with personalized user ads, most of the chip dedicated to "personalized" not spots.

And the second hand market. I'd love to see this integrated into motherboards like RAM. Someone could have a motherboard with 4 sticks of different AI with various models. Swap, change and trade.

I’m hoping for SNES style cartridges

How does that work, as in re-etching of silicon? Any pointers to read?

It's not just about it "being smart enough". It's about there being actual user demand when it needs to compete with the shiny new model.

A 10 year old iPhone is probably good enough, but is there demand for it? In a vacuum a 10 year old iPhone is good, but why would you pick it if you can have a current one for a reasonable price?

Yeah I had to double check, o3 feels like it was ages ago. But GPT 5 came out Aug 7, so it's only one day off from my 1 year ago cutoff!

What you're describing is what Cerberas does.

Talaas is different, it's a true compute-in-memory architecture where the weights are stored in the connections between the transistors that perform the matrix multiply, rather than in seperate memory cells.

Most of the benefit comes from this architecture; hardwiring the weights into the silicon is just the easiest way to implement it. SRAM requires too many transistors, DRAM requires an incompatible manufacturing process, and exotic phase-change memories aren't readily available.

Taalas does not have cache so...

I agree that Groq with multilayer hybrid bonding could be a good idea.

And their team, if they treat them well.

FPGAs already configure themselves on boot.

I have no idea. It makes my comment a statement posted as a proxy for a question, a question you correctly pose explicitly.

If it can, then deployment in a sea of gates can make a chip viable across model generations as weights change, inside some scale factor.

If not, unless the part is under a pinout and address model which can scale on the bus, and can be easily replaced, it makes the entire dependency a replacement, not just this part. So embedded use has consequences.

Just looked this up, no longer true. Explore subagents inherit whatever model the parent is. And you can of course make other subagent configs.

Customer service has really degraded huh. 4 years ago they expected opus performance out of human call center agents

I guess losing some customers due to poor customer service is ok if the price of customer service is right.

There's soooo much stuff that such a model is still capable of doing in the pursuit of getting a better overall answer. Imagine a powerful research agent that blasts out dozens of the small, cheap models to fetch and summarize one page each. Then the beefy researcher model performs the final analysis.

The cartridge could be a small mac-mini type unit connected and powered over thunderbolt. If it included like an m5 or m7 with 64GB of memory and a PCIe5/6 4TB Nvme it would be amazeballs. Hopefully when the bubble corrects and hardware advances and prices reset something like that will become available.

Just even comparing compute from 10 years ago (Apple silicon vs Intel) and it's significant. 20 years it gets crazy. My first computer was an 8 bit 6502 with 64K RAM and a 128K floppy drive (I think, it's fuzzy). Everything amazing now will look quaint in due time.

i'm looking forward to Qwen3.8 27B launch to see how much models have peaked at a given size.

it might already be time to start burning the best small models onto hardware since it's possible they can't get much better at many tasks like knowledge recall due to the inherent information density limits for models at a given size.

> Imagine a cartridge with a physical model on it.

I can finally have my own Dixie flatline. Cool.

very interesting idea. i didnt think of that. i was just assuming youd have an additional one of these in your phone for actual lightning fast local inference

I wouldn't expect companies not sharing their weights today to be any more likely to share them if they're on hardware, this doesn't sufficiently hide weights from a local user.

The weights are very unlikely to be on the chip itself. That wouldn't work for SOTA models that are terabyte scale, even quantized. This is probably an accelerator for specific kernels in the model, but the weights are likely loaded from memory. The chip may have SRAM to store some of the weights temporarily during inference.

I don’t get why this is an issue? You can run Claude/OpenAI SOTA models through Amazon bedrock. These weights have to live somewhere to run on Bedrock.

One idea would be to use an open model.

You have a robot. You need it to be smarter. You buy a new model cartridge (probably a PCIE 9.x). Now you need some domain specific skills. You'd like it to be able to cook, and you'd like it to not dent your walls anymore. You buy 'improved spatial reasoning LORA' card and 'Gordon Ramsey's Chef ULTRA9000' card.

Now your robot can respond sarcastically when you ask for chicken nuggets. Again. It also doesn't dent your walls anymore.

SOTA American models are not. SOTA Chinese models are. From a physics aspect, closed source models cannot be too far from open source ones in terms of size. There’s only so much you can squeeze out a B100 style cluster even with fancy Dflash style diffusion model for the speculative model.

If we had deepseek v4 flash 0731 etched on a chip it would be more than capable enough and fast enough for so many people's needs, even hardcore engineer.

if they were still exponentially increasing, they wouldn't be preparing for an IPO. IPO is where companies go to die and founders escape.

What's interesting about this is that I as a user would find this useful, but I think the AI industry as a whole would find it an absolute goddamn disaster. Opus 5 is a very good tool, but it is not a human-replacement-level intelligence, which means the entire revenue stream the industry's built on - labor replacement - is not met by this, and the only slightly charitable read of the industry's finances is that they're gonna bootstrap their way to creating the labor replacement hypothesis by getting people to spend money on Opus/etc, whereas if the actual product is a 1/20th the cost Opus-on-a-chip, the entire business and financing model that's tying up $N Trillion dollars of investment money goes out the window.

Great for us, looks like a recession as far as the Market is concerned.

I wouldn't call it supremely fast but zippy and versatile, yes: https://shop.m5stack.com/products/ai-pyramid-computing-box-p...

They were specializing their forthcoming system on 4-bit FP - which I understand is a structural decision.

Bonsai Ternary (1.7bits/weight) is a compromise, compromise that has to make sense in the context - efficient when translated into transistors.

For fast Kimi K3? You're damn right I do

Yeah. I have 3 max20 plans.

Me? Probably not. A business or a hoster, sure. There'd probably end up being an aftermarket in used cartridges with slightly older but still good models on them.

> How tolerant are models today to a few broken weights.

Extremely! You can remove entire layers and the model will still work just fine, with barely perceptible capability losses.

I've cut/bypassed ~15% of total parameters out of Gemma 4 31B on a pod once. Still got perfectly coherent responses out of it. Certain layers are a lot more important than others, particularly early and late ones; but it's honestly astonishing how much can be cut out from the middle without destroying the model's coherence.

I didn't run any meaningful benchmarks, so I have no idea what the capability loss looks like exactly. But "produce coherent and sensible English in response to a wide variety of prompts" was definitely not among the things the model unlearned.

I wonder if you had a few percent of problems in the yield, if it would be functionally equivalent to the difference between a unsloth-published Q6 standard size GGUF vs. the nearly perfect precision of an unsloth Q8-K-XL. Or more like Q4 vs Q8 where a lot is lost.

Well, given the data above, roughly a 220b transistors chip for the HC1 tech.

Umm I have an extra 35, do you have layer 6?