Terminal Bench 4 Is Saturated

Posted by nharada 2 days ago

Counter26Comment3OpenOriginal

Comments

Comment by Good4boothee 2 days ago

I think ambiguously worded questions have their place, but you have to give up automatic scoring on such tasks.

Comment by vintermann 2 days ago

You don't have to give it up entirely, you could grade on the interpretation they chose. Personally I'm a fan of models which seek clarification on critical ambiguities rather than barging ahead "autonomously" though.

But wow, they found a lot of issues with the benchmark.

Comment by scotty79 2 days ago

I really like the benchmarks where frontier models score each other's solutions to a problem.