13 Models and 4 Agents on SWE Tasks: Go, Java, Python, Rust, TS

Posted by ibragim_bad 3 days ago

Comments

Comment by sathish316 3 days ago

What does it mean when Fable 5 is 1st place and Opus 5 is 3rd place, while Claude code is 7th place? Which model and effort is used for Claude in 7th place, compared to 1st and 3rd?

Comment by spullara 3 days ago

They are all different problems for the different languages. I was hoping this was a benchmark that attempted to see which languages were more efficient to use with which models.

Comment by ducktective 3 days ago

Exactly. I was wondering what language would be better for heavy agentic workflow. Due to a recent blog post, Clojure seem to be the most terse [1]. But some people are suggesting static types would help in this workflow.

[1] : https://martinalderson.com/posts/which-programming-languages...

Comment by revetkn 3 days ago

I'm curious about this as well. I use Java quite a bit and I feel like my results are excellent, but colleagues who use other languages are less impressed. I wonder if this is because Java is more amenable to agentic workflows, or there is a large corpus of example code, or some other reason.

Comment by dia80 3 days ago

Why test Fable high effort vs Sol medium? Especially when Sol comes out 4-5x cheaper in their tests at those effort levels.

Comment by cbg0 3 days ago

I think you just answered your own question.

Edit: In DeepSWE Sol High scores the same as Fable High for ~1/3 of the cost.

Comment by stared 2 days ago

Call me biased, but if Grok 4.5 is above GPT-5.6 Sol, I don’t trust this benchmark.

Comment by guywithahat 2 days ago

Consistently impressed with the performance of Grok, given they started so much later than everyone else and, in some ways, are less well funded compared to OpenAI and Anthropic. I wonder how much of this is just luck, name recognition, or management style. Marc Andreesen likes to talk about how Elon companies have a unique engineering-heavy management structure, which contrasts the research heavy cultures of OpenAI and other labs, and it makes me wonder if that sort of thing could be behind their relative success.

Comment by pseudosavant 2 days ago

Distillation. It isn’t just Chinese models doing that to OpenAI and Anthropic.

Comment by goldenarm 3 days ago

Why are models better than agents, isn't it supposed to be the opposite? I don't understand the difference and what you are measuring.

Comment by joshka 2 days ago

Some agents have specific tools that the models have been trained to use. E.g. diff formats for editing that aren't the same as the standard unified diff format. Access to specific thread / subagent / etc. tooling or the base prompt can perhaps also impact how the tasks are completed.

Comment by abratabia 3 days ago

[flagged]

Comment by tcdent 3 days ago

What happened in July 2025 and January 2026? If that can't be explained in a concise way this entire dataset is vibes at best.

Comment by citizenpaul 3 days ago

I've had much better experiences with the "budget" tier models than this test suggests I should. I rarely even consider the higher models due to price/use. Perhaps I'm just more vigilant about spec'ing my prompts out before submitting them? Am I just doing too pleb of work? I'm not trying to write custom cuda kernels or hardware integration.

Comment by stpedgwdgfhgdd 3 days ago

Weird

Comment by rsyring 3 days ago

https://swe-rebench.com/about

> Potential data contamination: The SWE-bench dataset, comprising a collection of GitHub issues, has been publicly available since the end of 2023. As a result, models released after this date may have seen these exact issues or highly similar data during training. This raises the risk of inflated performance metrics and makes it harder to distinguish genuine generalization from memorization.