Qualcomm Cloud AI 100 Ultra – 150W AI Accelerator Card with 128GB at 548GB/s

Comments

Comment by tokepi 1 day ago

If the specs seem outdated, it’s because this chip launched in 2024. It’s available at cloud providers including Cirrascale

Comment by jauntywundrkind 1 day ago

Thats relieving!

I'm starting to believe in Qualcomm! The Snapdragon X2 Elite Extreme is alas only available in a single system, with it's tri-channel on-package (!) memory good for 228 GB/s. But it's a beast. The core is just good, even in the mainstream form in other systems: some of the best single core performance you can get, shy of Apple.

And they are upstreaming their stuff much better! They hired a bunch of really good known FOSS developers and it seems to be working!

Now they are talking up High Bandwidth Compute as their next big thing. And I want to believe, even if it makes me nervous that they too intend to leave consumer computing completely in the dust. https://www.qualcomm.com/news/onq/2026/11/qualcomm-hbc-valid...

This is, as commented, at M4 Max levels of GB/s. It is not impressive. It has some uses! But if this was their modern opening salvo I would not be impressed.

Comment by redanddead 1 day ago

> I'm starting to believe in Qualcomm

Someone drop the YouTube link of their robot presentation

Comment by gleenn 1 day ago

The way it drops is priceless.

https://youtube.com/shorts/aZmt5E151Bk

Comment by 3eb7988a1663 1 day ago

Those guys were fast on the draw to cover the body with those sheets.

Comment by a012 1 day ago

He’s hired to follow the robot and put the sheet on it whenever it’s dropped down to the floor

Comment by wojciii 1 day ago

The way that manager looks like he thinks the robot can recover from collapse is the priceless part here.

Comment by Rohansi 1 day ago

> tri-channel on-package (!) memory good for 228 GB/s

Why is on-package memory impressive/better in this case? It's one channel more than your typical consumer x86 system (192-bit bus vs 128-bit) which is something but you could technically have quad channel with two LPCAMM2 sockets?

Comment by jauntywundrkind 1 day ago

As a computer engineer, I can barely state how excited I am for systems with high bandwidth where I don't have to route gobs and gobs of very high speed ram.

It's radically simpler and smaller a computer to build. That has huge upsides. For laptops maybe this isn't a huge win, but for tablets and smaller computers, this is such an incredible win. And for making it simpler to build computers, to expand the range of who can make systems? It's everything.

Comment by Rohansi 12 hours ago

Small devices are the best use case for on package memory, yes. But at the same time I don't know of any that go beyond a 128-bit bus. Most are 64-bit.

Comment by harkbox 1 day ago

[flagged]

Comment by rpcope1 1 day ago

Given the rush to dig MI50s and V100s out of the trash and press them into service, 2024 seems like future space alien technology.

Comment by eyegor 1 day ago

Holy cow the v100s go for like $1k on ebay now... They were selling for $100 or less a couple years ago.

Comment by thway15269037 1 day ago

What's the price?

Comment by simlevesque 1 day ago

According to their own website, they cost 1 cent.

Comment by wewewedxfgdf 1 day ago

Less than I expected for this level of GPU power.

It's a trend I like.

Comment by bigwheels 1 day ago

8,750 in stock at 1 cent, too. The more expensive 47 and 63 cent variants are all sold out.

Comment by thway15269037 1 day ago

Yeah, I clicked that too. But knowing qualcomm they obviously cost an arm and a leg. Still interesting how much.

Comment by bigwheels 1 day ago

My research indicates the price is $10-17K USD.

Comment by IgorPartola 1 day ago

I mean at ten bucks it’s not a bad deal but that’s quite a range.

Comment by 1 day ago

Comment by InsidiousWombat 1 day ago

that tracks, $0.01 for chip, and $9999.9 for the memory

Comment by 1 day ago

Comment by dingdingdang 1 day ago

Honestly, let me buy one, it's many times faster than the minipc I run local ai on so if the price is right I'm in.

Comment by martin1975 1 day ago

hopefully less than 5090?

Comment by wmf 1 day ago

Not a chance.

Comment by pearl_panel5 1 day ago

Curious how the memory bandwidth actually holds up under sustained inference loads compared to the spec sheet numbers.

Comment by swiftcoder 1 day ago

So it's an M4 Max Mac Studio, RAM-wise, with a GPU's worth of compute?

Comment by trvz 1 day ago

(I know it's you comparing it to the M4 Max, not them, but:) it's Qualcomm, so it'll be compared to the M4 Max, while being slower than a M2 Max.

Comment by zitterbewegung 1 day ago

Sort of interesting design It has half a terabyte of SRAM on the accelerator with a 128 GB of RAM replacing HBM with SRAM but keeping regular RAM.

Comment by 2001zhaozhao 1 day ago

half a gigabyte not terabyte lol

Comment by zrm 1 day ago

Yeah, this thing is unusual. It resembles a consumer GPU but using normal DRAM with more channels, so then you get more RAM but less memory bandwidth than GDDR/HBM, which is the same thing Apple does. And the TDP is pretty attractive for what it is.

But what is that amount of SRAM for? It simultaneously seems like a lot, because SRAM is so expensive, but still not very much, because what can you fit in 0.5GB? Is there a specific workload that needs something that looks like this?

Comment by tromp 1 day ago

0.5 GB of SRAM is exactly what the Cuckatoo32 [1] Proof-of-Work scheme needs for optimal solving efficiency. Solutions are 42-cycles in random bipartite graphs of 2^32 edges on 2^32 + 2^32 nodes, which can be found by repeatedly removing edges that have only one adjacent edge. Tracking which nodes have incident edges takes 2^32 (completely randomly accessed) bits which is exactly 0.5GB. Another 0.5 GB is needed to track alive edges but those can be accessed serially for which DRAM suffices.

[1] https://github.com/tromp/cuckoo

Comment by zamadatix 1 day ago

The Intel D270, Nvidia DGX Spark, and (somewhat) AMD Strix Halo also use the same approach, so it seems to be getting pretty common.

SRAM wise it doesn't seem too insane. About the same ratio of memory:SRAM as the 5090 (IIRC). Still more than I would have expected. I assume they found it helps process the current layer faster or something.

Comment by 0917236141 1 day ago

Seems like a reasonable general-purpose card.

Comment by WarOnPrivacy 1 day ago

Buy for the low, low price of - nope. Not for you.

    Cirrascale (8-card Ultra server, 128 GB each, 512 GB system RAM): 
    $4,699/mo (monthly term) or $3,759/mo (annual term)

    Cloud GPU rental (per-GPU): from $5.15/GPU/hr

ref: https://cloud-gpus.com/qualcomm-ai-100-ultra/

Comment by fancyfredbot 1 day ago

At $5 per hour these are absolutely terrible value. You can rent an H200 for that price and an H200 is much better in every way. Weird

Comment by teruakohatu 1 day ago

Not if the customers cannot rent a H200 (because they have been told not to rely on foreign gpus)

Comment by txdv 1 day ago

I guess they get their customers on scale, can you get hundreds of h200?

Comment by skavi 1 day ago

is this hexagon?

Comment by rsendv 1 day ago

Yeah, bunch of HTPs

Comment by g58892881 1 day ago

that's a slim card

Comment by aetherspawn 1 day ago

Ok so it’s significantly worse than Apple M6 and we don’t know if it’s cheaper or not

Edit: Ah sorry you’re right, the bandwidth is lower but the TOPS is a lot higher. I’m not sure how much effect you get from higher TOPS with such low bandwidth though.

Comment by zozbot234 1 day ago

> I’m not sure how much effect you get from higher TOPS with such low bandwidth though.

Raised arithmetic intensity. It would be quite good for LLM prefill, and decode would want to adapt by pushing parallelism higher (saving on memory bandwidth by reusing the same layer weights for multiple streams) or, if unable to do so (e.g. due to KV cache constraints) by resorting to speculative decoding (where the expensive 'validate' step handles tokens in parallel within a single stream). It does not make the platform inherently unviable. It would also do well at other tasks such as image/video generation, which is already quite compute bound.

Comment by dist-epoch 1 day ago

Apple M6 has 80-100 TOPS.

Qualcomm Cloud AI 100 Ultra has 800 TOPS, almost 10 times more.

Comment by grosswait 1 day ago

This could be very fast at running embedding models (if it supports the right modes)

Comment by ipsum2 1 day ago

Considering that memory bandwidth is the most important for inference for most neural networks, that matters more.

Comment by 1 day ago

Comment by dist-epoch 1 day ago

Modern inference stacks use different GPUs for decode versus prefill. This is a GPU suitable for prefill.

Comment by varispeed 1 day ago

Aren't these specs already outdated? Seems meh. I would think players would like to bring something new to the table instead of bringing what was okay two years ago or so.

Give me 1TB RAM and 5TB/s or something that exciting.

Comment by entrope 1 day ago

For most purposes, this seems reasonable? This looks like a single-height card, so fitting four of these in a workstation or server would give 512 GB RAM and 2 TB/s total bandwidth. That's pretty attractive for a lot of current LLMs, even if one wishes for 1 TB capacity at 5 TB/s bandwidth.

Comment by throwawayffffas 1 day ago

If you are going down this path, MI350P and RTX PRO 6000 Blackwell is what you would want. A lot more compute a lot more vram bandwidth.

Comment by zozbot234 1 day ago

> A lot more compute a lot more vram bandwidth.

But this gives you significantly more memory capacity for a compute/vram bandwidth tradeoff (i.e. arithmetic intensity) that's ultimately comparable to Blackwell, perhaps even more skewed towards compute. This seems good if you care about being able to store larger sparse models and more KV caches because that ultimately bottlenecks your ability to do inference.

Comment by entrope 1 day ago

Maybe. I don't know power for the MI350, but the RTX PRO 6000 is normally a 200-300W card, so you pay 2x the power for 3/4th the RAM and 3x the bandwidth. The performance frontier looks very different in terms of power to hold a mid-size model and throughout per user or across users. Price and inference stack would be big factors in what I would choose.

Comment by throwawayffffas 1 day ago

MI350P, H200 NVL and Nvidia RTX PRO 6000 are all 600W but compute wise all of them should be 3 to 4 times more performant than A100 Ultra according to their specifications.

https://www.servethehome.com/the-amd-instinct-mi350p-is-a-hb...

Comment by eyegor 1 day ago

600w is way off the vf curve though... Tiny bit of tuning and you'll get ~80-90% of the performance at ~150w nominal. At least that's what I got out of some pro 6000s when I spent 15 mins to underclock it.

Comment by varispeed 1 day ago

But there are already plenty of products for most purposes.

Comment by tempest_ 1 day ago

5 year old GPUs are still at crazy prices.

Plenty of businesses want in local inference and are priced out.

Comment by varispeed 1 day ago

You can't do quality inference on such a toy. It's a myth.

Comment by redanddead 1 day ago

A great opportunity for huawei

Nobody helps China like American corps helps China. Domestic anticompetitive behavior spurs demand for international competition

Comment by creato 1 day ago

What exactly do you propose they do? Do you think they are holding back supply? Should they sell at less than market price? That's just handing the profits to scalpers/middle men.

I'm sure NVIDIA, AMD, Intel, Qualcomm, etc. would sell more hardware if they could, but they can't.

Comment by varispeed 23 hours ago

They have no reason to sell more if they can gouge the existing supply

Comment by throwawayffffas 1 day ago

This is all the CPU companies playing catch up with Nvidia and AMD.

The thing you are describing is much closer to AMD and Nvidia latest generation server offerings.

See MI455X for the absolute cutting edge and MI350P for something you can use outside of a datacenter.

Comment by mciancia 1 day ago

8 of those cards give you 1TB of ram and 4.4TB/s

Comment by PhunkyPhil 1 day ago

It may be a semantic thing but I don't think bandwidth pooling works like that? Unless your blocks are distributed perfectly between cards there's going to be data passed between each card and from VRAM to compute. Correct me if I'm wrong though

Comment by msandford 1 day ago

I think the way it works is that the weights have to get streamed through the compute and the weights can be split very easily. With multiple cards you do have a coordination/communication phase after each layer of weights but that's usually much less significant than the compute phase.

Comment by throwawayffffas 1 day ago

It depends on how you slice the weights. You can split along layers which means each GPU waits for the previous one to finish or you can split the tensors across gpus.

Splitting the tensors is closer to n times the effective vram bandwidth but there are still overhead, sync waits, and reduce steps that eat away at the overall performance.

Comment by simoncion 1 day ago

> I would think players would like to bring something new to the table instead of bringing what was okay two years ago or so.

Speaking of... from [0]:

  Introducing Qualcomm Cloud AI 100 Ultra
  High performance and cost-optimized cloud AI inferencing for generative AI
  Written by
  Rashid Attar
  Nov 13, 2023

  Today we announce the Qualcomm Cloud AI 100 Ultra, a new member of our portfolio of cloud artificial intelligence (AI) Inference cards, purpose-built for generative AI and large language models (LLMs). ...

[0] <https://www.qualcomm.com/news/onq/2023/11/introducing-qualco...>