Snorkel's Senior SWE-Bench stumps top AI agents 75% of the time

Snorkel AI released Senior SWE-Bench, an open benchmark that grades coding agents against the expectations placed on senior engineers rather than junior ones. Instead of over-specified tickets, tasks come as terse, natural-language requests — the median instruction is about a third the length of SWE-Bench Pro’s — forcing agents to infer requirements, honor unstated codebase conventions, and produce solutions judged not just for passing tests but for ‘taste.’ Feature tasks are drawn from real multi-service PRs touching an average of 11 files, while bug tasks are sourced from fixes that demanded genuine runtime investigation (reading logs, profiling, reproducing failures). A validation agent writes behavioral tests that adapt to each submission, and scoring blends correctness with quality metrics derived from observed repo practices.

The results are humbling. The strongest model on the leaderboard, Claude Opus 4.8 running the Mini-SWE-Agent harness, solves only 24% of tasks. GPT-5.5 lands at 16%, with a long tail of models — GLM-5.2, Kimi K2.6, Gemini variants — falling into single digits. In other words, frontier agents fail to deliver senior-level, correct-and-tasteful work more than three-quarters of the time.

The benchmark’s design targets exactly where current agents are weakest: long-horizon problems requiring hundreds of steps, ambiguous specs, and cross-stack changes where the ‘right’ answer depends on judgment rather than a checklist. It reframes the evaluation gap in agentic coding — the hard part isn’t writing code to spec, it’s knowing what to build when nobody spells it out.