Ask HN: Is there any good open source model with reliable agentic capabilities?

Posted by baalimago 2 days ago

I don't want to send my data to third party vendors all the time. But from my experience, the LLMs needs to be quite beefy in order to understand tool-calling, especially at longer contexts (200k+).

Before I dive headlong into investigating this and spend money on a project doomed to fail, do anyone have experience with a local model which can handle this sort of workload? I intend to run it on decent gaming CPU with 64-128GB ram.

Comments

Comment by throwaway2027 1 day ago

CPU? Good luck.

Comment by baalimago 9 hours ago

WDYM? I don't want to train a model, only use inference. From what I know it must be much cheaper to buy "normal" ram + a decent CPU vs a GPU with similar amounts of vram.

The bottleneck of the inference is fitting a good enough model into memory. A 80B param model 8bit fp quantization equates to roughly ~90GB ram. So 2x64GB DDR4 sticks is probably the most price efficient solution. The questions is: Is there any model which is capable enough to consistently deal with an agentic workload?