Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

You can get a frontier model subscription for about the same cost as the electricity (heavily subsidised by someone else’s money !) with instant results.

I know there are applications for this and it’s cool people are pushing the boundaries with local models. I run oMLX and Qwen myself, but it’s a toy really. It’s not anywhere near replacing Claude for my purposes.

The write workloads are just the context and any K/V cache - llama.cpp does not mmap those to disk, so they would remain in memory or VRAM as space affords.

I stopped using swap on Linux about a quarter of century ago, when it was a great improvement, and since then I have never seen a case when swap would have been useful, and I use Linux on a variety of laptops, desktops and servers.

Even if you do not use swap, you can have memory-mapped files that are much bigger than your physical memory. The LLM weights files must be mapped as read-only. In this case, the pages that have not been used recently will be freed when memory is needed to load other pages from the files.

The weights files must be mapped using huge pages, otherwise an excessive amount of physical memory would be wasted and reading new pages would be very slow.

It is likely that it is not possible to reach a good enough performance with a memory-mapped file without using carefully "madvise", with which it is possible to force the reading of the pages that you know that they will be needed in the future and also the freeing of the pages that you know that they will not be needed soon.

On Linux, it is possible to execute "madvise" asynchronously (with liburing). An alternative to liburing is to execute "madvise" from a concurrent thread, which synchronizes with requests to do "madvise" from the orchestrating thread.

Why would it swap? If stuff gets evicted from page cache is just gets reread?

Optanes are a good option here, right?

I bought mine for $100 for each 128GB DDR4 stick. I believe write performance is off-the-charts on these, besides the fact they're c-h-e-a-p.

Real world inquiries are gonna be much slower of course, but this setup is still too slow to do anything meaningfully useful I think.

> I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster.

Everyone says that. And most of them are wrong about the better part. Referring to your previous ability to write code is almost an appeal to authority at this point because it's coming with the assumption that you review the code with he same vigor, and that you are just as good as you were before you offloaded these tasks regularly to LLMs. Maybe you do and maybe you are, but I will say that a lot of the people I've worked with who I used to respect and whose code was good now produce absolute crap. The models are getting better so their code is getting a bit better, but I no longer believe they had anything to do with the quality, they no longer seem to care.

Don’t waste your time rewriting it by hand just because keyboard warriors are up in arms on HN.

The material feedback here reflects the questions who might be from someone who’s been running local LLMs, so ask the LLM to update it with that in mind.

Another poster pointed out it’s not clear what the quantization is. There’s that one paragraph but it’s confusing. What I want to know right away is: are you running the unquantized model or is it quantized? If so, how much? Use terms like Q3 or Q4 or 4-bit or 8-bit. Explain why it’s not practical to quantize less. How much precision loss do you think there is at the quantization selected? What if I had 128GB RAM and wanted to have better precision and not higher token speed—would it be a good idea to choose 4-bit instead of 3-bit for some of these layers?

The fact that the top comment on this thread calls it out, in a negative way, hints at that you aren't.

The transformer does not yet understand the non-transformer.[0]

This is probably because all the data we trained it on was created by non-transformers, so it thinks it's a non-transformer, but it isn't.

I don't think we know how to train a transformer yet. All the training data is linear, but that's not how they think at all.

[0] It's a bit like the communication difficulties experienced between autistic people and neurotypicals. Each follow the Golden Rule, i.e. do unto others as you would have them do unto you -- and it fails in both directions. A Platinum Rule is necessary: do unto others as their API demands.

The thing about documentation though is humans won't actually read any of it. Maybe tailoring the documentation to the needs of LLMs isn't so bad since they're the ones who will actually consume all of those documents.

Personally I've mostly given in to letting it commit for me now, though I do occasionally take over and hand-write the messages if it's a particularly important concept and Claude's is too verbose.

Codex/GPT-x defaults to one-line commit messages, which are too short. Claude likes to write several paragraphs, which is usually too long.

If you tell it how to commit properly once per session it will stick with your standards for the rest of that session, and you can put that in AGENTS.md if you can be bothered to.

This person that you’ve never met and probably never will doesn’t owe any of us anything.

They’re out there doing what they want to do how they want to do it and if you don’t like it the correct response isn’t to insult them in front of a bunch of strangers on the internet for clout or whatever.

I doubt that you’d ever call them lazy to their face — why do it here?

No you don't. Buying a table and sanding the edges off doesn't mean you're a carpenter.

It looks like some people have a hard time accepting that software written with ai isn't a fad, it's here, it won't go away and it can be interesting and useful for the creator and for users.

Many people on the other hand have moved on and are now team leaders, except their team is mostly AIs instead of junior devs. Trade off: code is usually worse, but in the end AIs are more capable with vast knowledge and speed.

That doesn't mean we have to use AI everywhere, all the time though.

Lately I've been landing on a couple of different reasons but I don’t think it’s one size fits all:

- Ego/identity - “I am the crafter of code. That is what I do.” If someone has their identity deeply wrapped up in the concept of being a “software engineer” or “programmer” then LLMs are a direct threat to that. People don’t tend to do well with this. Think about dogmatically religious people, pseudoscience followers etc who are confronted with evidence directly contradicting their beliefs. Their entire world view revolves around that identity and if you threaten that, you threaten the foundations of their self-perception.

- Career - you can take a lot of what I said above but also map it to threatening someone’s career. What if crafting software by hand becomes a niche, artisanal craft but most software is industrially generated? A lot of people will experience hardship if so and if they can’t find a way to be flexible into the future.

- Passion - the ones who love software purely for the craft see this change as robbing them of the one thing they enjoyed in their career. Work is a big part of our life and if you kill the joy for a large group of people, that’s tough to deal with.

- Lack of understanding their current purpose in role - I often see software engineers who don’t realize why they were hired. It was never to “write good clean code”, that was a means to an end. The end is generating business value for the company that hired you. That’s really it. It’s banal but it’s just a job like bagging groceries only it has required specialized knowledge so it pays well. Only very niche roles have actually hired for the craftsmanship. If someone is working for Groupon and they believe perfectly elegant systems are the value they provide, they are a bit deluded IMO. They build a platform that peddles coupons. That’s hardly comparable to building software that allows a surgeon to remotely operate a robot that does open heart surgery. Most of us do not work on truly mission critical software like that.

Most importantly though, I think this topic needs to be approached with empathy. This is truly a seismic shift in how we, as software developers, work. Change is not easy to cope with especially when it threatens physical safety and identity. Will this change be here to stay and are we in for the extinction of software creation as we have known it? I don’t know. I do know that the world’s financial and governmental systems do seem to be betting on that outcome, however.

We need something more sophisticated, but you could just escalate the model quality if if keeps failing the test. Not elegant but it will work.

ftfy.