Fancy harnesses make coding agents more expensive but not better, study finds

Earlier comparisons were measuring noise Previous studies report gaps of more than 20 points between harnesses, but those numbers almost always come from a single run, according to the authors. Pure reruns of the same setup already vary by up to six points on SWE-bench...