Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

There is just a stupendous amount of everything packed into a 35B size or larger model. For instance Qwen 3.6 35B A3B (Q8) can do a fairly decent job translating English to Arabic, but it can also generate python code with a reasonable layout and commenting.

I try to remember that as a mental model, an epub copy of a truly gargantuan sized 1000+ page novel such as the unabridged/2nd revision of Steven King's The Stand is about 800KB, and we're talking about a GGUF file that's 37GB in size or something like that.

That's kind of the problem, isn't it? How do you know which part of the model to put in memory? You have to make a per-parameter decision of whether or not it's worth it to have it in memory or whether the value should just be treated as zero. Then you have to "re-link" the layers of the model to the new positions of each of the weights. For billions of parameters, that's a lot of calculations. And it requires us to know what each parameter actually represents, which nobody does.

Always curious when someone will figure out how we can elide most of the data from an LLM (but retain the logical ability). I don't actually need an LLM to have a very big internal knowledge base to be useful, so long as it can invoke a search tool...

The A in 26B-A4B is the active weights.

The problem is that this is a per-token load/unload at best, not for the whole prompt.

The division happened until one of these can fit in a single GPU and they stopped scaling it down any more, because you can wire up 8 of them to do their share of the work.

Dense LLMs typically perform better, but slow down much more than MoE models when you try offloading layers.

It's still early days and we "just" don't really know how to do it well.

I think a rough analog is that it would be difficult to organize the rows of a hash table of everybody in a country by their geographic location.

At this point, people are just thrilled LLMs can even function as they do…

my 2c on King Charles vs 42.

OP suggests that the token rate of of their solution is ~5 per second. That's at least an order of magnitude slower than commercially available models.

The idea that people in 10+ frontier labs (OpenAI, Anthropic, Google, Alibaba, Z.ai, DeepSeek, xAI, Amazon, etc) in a trillion dollars industry are all dumb is frankly, hilarious.

    @@ -4,8 +4,7 @@ import PackageDescription
     let package = Package(
         name: "TurboFieldfare",
         platforms: [
    -        .macOS(.v26),
    -        .iOS(.v26),
    +        .macOS(.v15)
         ],
         products: [
             .library(name: "TurboFieldfare", targets: ["TurboFieldfare"]),

With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work

And common weights still use mmap for simplicity

So, I believe llama.cpp might run it under 2gb, but I assume it will be slower

What are your thoughts on this?

I believe it would be a perfect match!

Feel free to use any parts of my project or drop me a message. There’s my LinkedIn link at the end of the readme. Or I will drop you a message later!

And here.

In all fairness, maybe it's just that they let some post-2022 recipe blogs get into the training runs around ~4.6-4.8 time

They don't add anything of value, did the author use an LLM to fix his prose but no useless slop was added in the process: who cares ? Is the article useless slop: fine, downvote it to oblivion.

(1) For those not old enough to remember that wonderful practice please use your nearest LLM to find out or, you know, visit a library and do your own research.

Everyone's on high alert. Maybe writing will get better!

When using SSD streaming, the GPU is practically always waiting for the SSD to fetch the right expert, rather than the other way around. There is basically zero slack on the SSD side, so I'm not sure how "prefetching" is supposed to help. It would mostly hurt by fetching the wrong predicted experts, which already makes conventional MTP practically unhelpful for typical (not widely batched) SSD streamed inference.

There is a different set of experts at every layer, and each layer has a small router that decides which ones to use.

The router needs to look at the state produced by the experts below it.

Drafted tokens from the MTP head can be used to predict which experts the first layer will want, but not beyond that. To know what layer 10 experts needs, you have to run layers 1-9 which means loading their experts.

So, yes, instead of a next-token drafter like MTP, you'd want something trained to predict the expert activation across all layers at once.

> In the Blackmagic Disk Speed Test, the SSD in the M5 MacBook Pro achieved read speeds of up to 6,323 MB/s, compared to just 2,031 MB/s on the M4 MacBook Pro. It's not like the M4 is "slow" in a vacuum, but the M5 SSD is over three times faster, which is a great generation uplift.

https://www.tomshardware.com/laptops/macbooks/m5-macbook-pro...

If that's true, inference speed would be even lower if you have only 2GB total, including OS caches

It was 83ms read per token for M2 and 12ms on M5 pro. Total is 163ms/tok vs 30ms/tok for M5. So yeah, there is a faster read and faster gpu processing

https://github.com/danveloper/flash-moe https://github.com/JustVugg/colibri

With 64 GB of unified memory, you should be able to run a DeepSeek V4 Flash quantisation at 7–10 t/s, for example with: https://github.com/antirez/ds4 or https://github.com/steadfastgaze/MoEspresso (my engine).

The routed experts needed for the next tokens that are not already in memory need to be read from the SSD, so the speed becomes SSD reading bound and the larger the memory, the faster the inference.

Memory bandwidths (* = rumored):

  M1:       68 GB/s
  M2:       100 GB/s
  M2 pro:   200 GB/s
  M2 max:   400 GB/s
  M2 ultra: 800 GB/s
  M5:       153 GB/s
  M5 pro:   307 GB/s
  M5 max:   460 GB/s
  M6:       200 GB/s*
  M7:       240 GB/s*
  
  Nvidia 4090 1008 GB/s
  Nvidia H100 3.35 TB/s

It also gets very hot. If you’ve never heard the fans on Apple Silicon really spin up, it could surprise you. Makes the full GPU setup feel quiet by comparison.

I think after the hardware market calms down the ticket is going to be a light laptop with a second dedicated inference server on the network.

My friend tried it on an M4 MacBook Pro and got 25–27 tok/s

I saw a pretty cool project to run an llm on an esp32 device https://github.com/slvDev/esp32-ai

Just toss GBs of file structure: "AI, do your work baby!"

I for one break things down much smaller into very specific tasks involving very particular text. Maybe I'm overdoing it lol.

For me, an AI security review would still take hours or days, it would hardly be a 1-shot prompt like this.

I think I might understand your use case. You ssh the Mac and want something like `ollama run` with an interactive chat in terminal. Am I right?

There is already experimental OpenAI-compatible server in this repo:

``` swift build -c release --product TurboFieldfareServer .build/release/TurboFieldfareServer \ --model scratch/gemma4.gturbo ```

After that a small terminal client can run inside the same ssh session and talk to `/v1/chat/completions`

The client needs to keep a messages array, add each user message, send the full array with `stream:true`, print SSE chunks until `[DONE]`, then add the response back to the array. `/reset` can clear it

There is a python example in the server docs. (https://github.com/drumih/turbo-fieldfare/blob/main/docs/OPE...)

It is non-streaming, but can be used as a starting point.

The server is still experimental and I am fixing some problems currently. But you can try to vibecode a simple terminal client around it.

If not, create an issue on Github and describe desired behaviour

I think it will throttle quite soon, but I haven't tried runs longer than 30minutes with this engine.

However, there is no constant load on ssd or gpu. i/o and gpu work are alternating and there is a brief idle periods for each i/o and gpu during inference (because gpu waits for i/o and after that i/o waits for gpu)

The longest exact repeat we found was only two tokens. Coding tasks may have higher reuse if code related experts are selected repeatedly

I tried it on my wife's M1 MacBook Air 512GB and it gets 4–5 tok/s

Also, it must be easy to adjust for iPhones and iPads in theory

after that download 14gb of weights and enjoy offline inference (and a bit of Gemma4 intelligence) for your everyday tasks

multi turn chat is coming!

I measured this exact model with a 4k context on the mlx engine. It runs at 75 tok/s on my M5 Mac Pro and using 14 GB of RAM. For my engine the same model uses 2 GB of RAM and produces 31–35 tok/s.

The project is still experimental so performance may vary as it continues to improve. If you want to save around 12 GB of RAM for other tasks and you are ok with 35 tok/s (afaik it is roughly comparable to ChatGPT’s speed for basic responses) my engine may be a good fit.

If you need maximum speed and flexibility just use MLX

Windows PCs would require a completely different approach

Otherwise IMO it codes about as well as the Qwen MoE for PHP and SQL. It's a fully impressive model (though it is not as mindbendingly impressive as the 12B, which is outrageously good for its footprint)

https://sleepingrobots.com/dreams/stop-using-ollama/

afaik ollama relies on llama.cpp and mmap. mmap loads pages on demand and doesn't use the same explicit cache or parallel reads like my engine. Most likely ollama/llama.cpp will be way slower in this case

You are not really accessing the biggest frontier model every time, and you're not really doing an end-to-end LLM request on each prompt.

I would go so far to say frontier models have peaked and improvements from here come from clever (or very elaborate) harnessing. "LLLMHs" - Large Large Language Model Harnessing !

That said, in theory this same technique should be able to run it on a 64GB Macbook, probably at <1 tps.

Owen 3.6-35b-a3b was my initial idea, but I switched to Gemma because of its simpler architecture and kernels

One obvious thing is that the memory requirements for this are substantially smaller than DwarfStar-- which AFAIK can only start to be used at 64GB ram and upwards. Another obvious thing is that antirez is pretty obsessed with making sure that DwarfStar passes all of DeepSeek V4 Flash's generating tests (loosely). I suspect that is also true of DwarfStar's implementation of GLM5.2, but I don't use that.