MIT and Sakana AI Slash Coding Agent Eval Costs With New SIFT Framework

MIT and Sakana AI researchers developed Recursive Self-Improvement via Fast Tree Search (SIFT) to reduce the cost of evaluating changes in coding agents. A benchmark run on the Polyglot coding evaluation reached 35.1% accuracy in under five hours, costing about $150 in API credits and 42 CPU hours. The framework uses an LLM-as-a-judge to make pairwise comparisons of agent versions at roughly 4.4 cents per call, rather than running full 50-task evaluations that cost $6. SIFT runs patch generation, judging, and benchmark evaluations asynchronously in parallel so the search tree keeps growing without waiting for tests to finish. Tests using Qwen3-Coder-30B and o3-mini showed better accuracy with less compute compared to previous methods like HGM and DGM. "The poor signal-to-cost trade-off of existing evaluation methods." — Authors