Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

+1 on SID-1. we were definitely inspired by that paper

*MixedBread

https://www.mixedbread.com/

> Claude code does some of this by handing off the "explore" agent work to haiku.

That is not handing off to a specialized model, its just handing off to a lighter and interior model (compared to the parent model). That by itself can create issues like the lighter model not capturing all the data that the parent needs.

The idea is that we get specialized models that are better then general purpose models. But its rare for a specialized model to beat a strong general model.

There is a reason why we hear less about this idea of smaller expert models, because large strong models to the tasks just as good.

And if the tasks is repetitive to the point that specialization is useful, you can get into a situation that your better off having a program written for that reputative nature, then delegating to other models. And then have the main strong model, deal with the (semi)cleaned up data.

There are! Chroma has Context1, SID has SID-1, and you'd actually be surprised at how easy it is to post-train your own with pretty good pass@ recall@ ndcg@ etc.

There's also Hornet who have shared some interesting talks & blogs lately. I don't know that I'd exclusively use agents for retrieval the way Neon outlines here as well. I think distillation similar to what ZeroEntropy has done for bespoke retrieval & reranking with _some_ agent manipulation on top-k results works better (IME).

> Claude code does some of this by handing off the "explore" agent work to haiku

This is no longer necessarily true. As of 2.1.198 [0] (released July 1st): "The built-in Explore agent now inherits the main session’s model (capped at opus) instead of running on haiku"

[0] https://code.claude.com/docs/en/changelog#2-1-198

I feel like the future is people building applications with tightly integrated LLMs that work hand in hand with the application's own lifecycle and code.

I also didn't realize that people were using agentic harnesses for search, it's an interesting idea. If the context length is short enough it should be fairly cheap compared to running "normal" agentic coding workloads where you have O(100k) context length for doing almost anything.

IMO, purpose built or “adapted” models are The Next Big Thing. If I’m using a model to write Python code, for instance, I really want the 8B or 27B expert model for exactly that, which would also be runnable locally. I don’t care about the 1.8T model that can answer every query under the sun and that only runs in a remote data center.

Yep. It's a real shame that the labs are incentivized not to go in this direction. They all want to try and suck us into the cloud and take away full control and local processing, but there's far more opportunity by building small AI systems and tools into the harness itself to make the models more intelligent.

They naturally don't like this direction, because it draws the intelligence away from their systems, and onto the local machine, where idea moats cannot be protected and hidden, and costs can be dramatically cut. Imagine though, how powerful our harnesses could be if the best researchers were thinking about how to utilize the power of the gaming GPUs that most PC users have (or can get), to supplement the frontier model processing. Instead of trying to have the frontier model do everything, the frontier model can serve as the orchestrator over all of the smaller dedicated harness models. Right now my rtx4090 sits there unused for most of the day while I'm paying for inference in the cloud... It's such a waste of parallel intelligence bandwidth.

I'm not just talking about LLMs either, most people seem unaware that there are a plethora of dedicated AI models for all sorts of conceivable pipeline usecases, from all sorts of classification tasks all the way down to things like code duplication detection. Right now the LLMs completely suck at cleaning up code and architecture, and a big part of that is because the frontier LLM cannot fit your entire codebase + all of its long-chain reasoning into the context window. But using small local models and tools bypasses this problem because small fast models can iterate over an entire codebase quickly. A harness that creates a big model bundle + routing system + DAG-based memory/execution management over all of these has the potential to be incredibly powerful.

Even better, building a framework around this concept and having the frontier model dynamically and adaptively generate the ideal execution system for any given task/domain. We're working on coding today? Okay, here's a recipe we can use: ..., and it generates a local model pipeline execution system that it feeds all of your prompts through in real time by using pre-defined or shared recipe building blocks, etc... Lots of interesting possibilities.

There has been an over-obsession with frontier models and benchmarks. Most of the work will be done by task specific models. You don't put Phds on the factory floor.

Models keep on improving though, so doesn't fine tuning become an ongoing task with ongoing maintenance burden?

This is how Sakana's Fugu model works, achieving similar performance to Opus/Fable with a mix of open source & mainstream LLMs.

isn't this also the threat to frontier AI houses? As in they want you to expend tokens in their ecosystem, but the optimization at 100x is their profit?

this is exactly what leopold talks about in situational awareness

yes, and this is why we need model routing

> There is so much opportunity for purpose built models like this.

OpenAI etc could themselves do this, and maybe they already do? Where the public-facing interface delegates to multiple little goblins behinds the scenes

Electricity becoming a commodity doesn't necessarily make nuclear reactors become commodities. And as we can see, the publication of open-weight models doesn't really diffuse the know-how of training SoAT models as we expected. Even the major cloud providers are still unable to build competitive ones from scratch on their own. That's very different than the traditional FOSS ecosystem, where everyone can copy, learn, and evolve on one's own (like, Bitcoin -> altcoins).

On token pricing, I think it's very much bottlenecked by hardware (the aggregate of compute) rather than the number of competing models. Assuming that the ceiling of the token price is determined by the economic value a unit of compute can provide, then the less efficient ones would be priced out of the compute allocation. It's not consumers bidding up a limited number of different models, but more like tokens of different models bidding up the limited computing resource. Less-intelligent tokens (which are generated by weaker models) are crowded out by smarter tokens from the limited compute. My prediction is that we'll see a meaningful downward pressure on token prices only when the new batches of next-generation hardware get mass-deployed.

castform founder here. despite our bet on fine-tuned smaller open-source models, i'm still quite bullish on the big labs. i think scaled closed models will continue to dominate for more general purpose use-case like codegen, search, etc. but intelligence has lots of long-tail applications and i think for these longer-tail applications, finetuned custom models will rule

Small models aren't going to take people's jobs. Agents using large models might. That's how their numbers make sense.

Considering the state of the field ( RAG/retrieval/evaluation) I have 0 trust in it, even more if it's closed source with bullshit claim like that.

Everything is vibe sloped to death, and dead after a few months to a couple of years (and not hard to be 100 cheaper than GPT-5.6 sol ... DS is basically free and I guess already 100 times cheaper or more, and here another slope ).

I use detailed project files. It has data regarding the project and subtasks as well as task status. It doesn’t depend on agent context and it’s managed to keep the agent on track. Feature creep with the new models is a very real issue. Capturing principles and how to reconcile tasks helps too. Even today it brought up a source of truth issue it had detected. There were multiple authorities born out of a patch and it used that principle to highlight and resolve the problem.

<founder of castform here> tldr: we generated synthetic training questions from the gitlab product handbook.

totally agree that this larger corpus with harder to search information would be a good way to stress test - i'm sure we will encounter more interesting problems to solve. love to hear any suggestions of corpus to search against that is not just the public internet

the training run link is also a little buried but here, you can see the comparison against the various models and their exact traces: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

Yes, and it's only logical things move in this direction because there's massive hardware incentive to do so. If frontier models can be broken down into small models, networks of smaller-GPUs can be utilized. Right now the smaller GPUs are basically paper weights for frontier intelligence.

founder of castform here, we believe that as agent deployment moves from experimentation phase where cost doesn't matter as much to deployment (what's the margin of serving the request), there will be a rise in interest in optimized models.

It's a matter of cost. Did you see the 100x cheaper?

If you have a workload that is going to be very heavy, incurring a large training cost to make a cheaper model work well with the dataset will be dramatic cost reduction. Most large AI workloads can't afford, or truly need, the expense or capability of GPT 5.6 Sol when cheaper models can do.

Of course you could skip that and just use GPT-5.6 Sol everywhere instead. If you're running a fast food restaurant you could hire Michelin star chefs to make your burger and fries without further training. Or you could have a training program for teenagers, a sourcing program, etc. to scale up to your chain to still get consistent quality without needing that level of cost in each store, but replacing it with a centralized repeatable process.

(founder of castform here) the model you post-train should ideally learn general patterns & search strategies over your dataset that should transfer to new docs you add to the search corpus (unless its super out of distribution)

we actually have the test benchmark against luna but no deepseek flash (we haven't added DSFlash into our benchmarking model pipeline)

you can check out the full comparison against all the other models here: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

- founder of castform

I agree with this. Interestingly, I’ve been using Deepseek v4 Flash a lot these days but I definitely have to constrain it a lot with tests. Fortunately, I can have it write the tests. It’s extremely cost effective. Still trying to figure out whether the latest update last week that made it smarter actually translates into something I can see in the output. It’s not dumber, but it’s still an open question as to whether it’s “real world smarter.”

Anecdotally, it feels like Opus, Fable, and Sol "get distracted" when you use them for writing code. Great at reasoning and coordination but they will go off on a tangent and refactor half the code base. I only use them for reasoning (of course) and coordinating subagents.

Just an anecdote but thats why Deepseek v4 flash 0731 is my current favorite model. It's really not very "eager" and stays on the task at hand.

Have been feeling the same. There's a sweet spot that threads the needle between "too dumb to search the right thing / relay the correct results" and "too smart to just stop overthinking and just report the damn thing"

founder of castform here again - slightly unrelated to retrieval but on the topic that folks are discussing here, i was actually collecting benchmarking various coding traces for the purpose of training a model router and surprisingly, luna held up very well against sol and terra. it was able to solve close to >95% the that sol can handle at a fraction of the cost. have not benchmarked the OSS models yet but will add the popular ones to the list like Deepseek Flash and Kimi k3 to see how they fare.

will share the full results soon!

we actually have the test benchmark against luna too! it's just not in our title but you can see it in the first diagram below the title. luna does pretty well tbh but sol is just a tad bit better. but luna is way cheaper.

if you want to dive down into the various traces of the benchmark, you can check this out: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

- founder of castform

Any data or public links you can share? That surprises me

castform founder here. unfortunately, we are cloud-hosted at this point.

but some easy options on the open-source side include huggingface's trl & unsloth.

you can run our data-generation scripts here: https://github.com/castform-ai/benchmax and then hook it up trl/unsloth for training. should be able to do all of this on your own compute.

In OpenRouter there are Zero Retention options. And if you use a EU provider you can be somewhat sure that your privacy is given.

Other than that there is not really a difference to renting a GPU since the GPU provider can also just steal your data.

Local GPU(s) are always an option if you have the possibility. It is also not that difficult to run with stuff like “LocalAI”

cheaper in terms of token costs

Keeping track of any AI progress is becoming harder by the day, because there's ambiguity around common/clear/consistent benchmarks. Everything is constantly skewed into favourable directions.

(founder of castform here) - we didn't get to dive too deep into the dataset we were using for the retrieval in the blogpost for brevity, but we did link the training run (which shows the dataset) here: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

the page shows the exact trace of all the models we are comparing against and the aggregate scores

we generated the question & answer pair from gitlab product handbook (https://handbook.gitlab.com/) since the point is to show that you can generate training questions from raw data corpus (something a company already has today)

Usually in these cases you don't need to do much tuning to the retriever. So you just give it BM25 or somesuch.

I'm hesitant to say absolutely zero tuning, because there are cases where you do want to say, bias towards trustworthy results or recent results etc to help the model avoid wasting tokens. But probably not much beyond that.

You can also just create a param in the tool for the agent that selects for "recent" or "popular" or "trustworthy" in ranking.

castform founder here. it uses lakebases's native bm25 and vector search and fuses the results using rrf (https://medium.com/@devalshah1619/mathematical-intuition-beh...)

Have you tried Claude? 5.6 feels less verbose, and less messy to me.

You can just tell it not to be verbose...

it is definitely more verbose.

try adjusting model_verbosity. it defaults to verbose. and of course, use agents.md.

Deepseek flash (especially after the recent update) has to be one of the most slept-on models. Price-performance is ridiculous, and its available on a number of cheap coding subscriptions

> I realized that z.ai gives you access to deepseek 4 flash.

How? Can you give details?

(castform founder here) we should have made it more prominent on the blogpost but here's the github repo: https://github.com/castform-ai/benchmax/tree/main/examples/n...

castform founder here. i'm personally a little against techniques like self-consistency/majority voting during rl training because they tend to result in the model's output distribution "sharpening" a lot. this means the model will lose it's exploration ability and probably won't be able to explore/discover new solution strategies, which can be harmful for both rl training + generalization to unseen cases

(founder of castform here!) - having worked at FAANG / big tech, i totally get this. our example was on gitlab's open source company handbook but i think a real company's corpus is way more messy and has many sources of truth.

a few ideas i have yet to validate are: - prioritize recently updated docs when generating the training questions (assumption those docs are more correct than others) - actually including contradicting documents that talks about the exact same topic might be a good training example - ideally the model should surface all the relevant info it can find, and explain what it has found. (usually contradiction comes from the fact that the later document is the updated stance) - you could also mine high quality Q&A from public slack / communication channels where questions were asked and someone else in the team linked some docs / answer. those are strongly validated "ground truth" answers