StarSkirmish v0.2: GPT-6 Astra Tops LLMs Writing StarCraft Bots, But Humans Still Dominate
StarSkirmish Bench v0.2 gives each LLM one hour to write a Protoss bot in C++ for StarCraft: Brood War, then pits the result against a fixed roster of competitive human-written bots. GPT-6 Astra leads at the top, followed by GPT-6.1 Sol, with Gemini 4 Argon, Claude Opus 5.5, and Claude Fable 5.1 clustered between 37 and 39. The standout finding is the gap: even the best model wins only 7% of games against the top three human bots (Stardust, PurpleWave, BananaBrain), meaning the benchmark is far from saturated. GPT-6.1 Sol emerges as the value pick, scoring second at roughly a tenth of GPT-6 Astra’s per-run API cost.
Behavioral analysis of agent traces reveals that practice against the strongest roster bots is the single best predictor of final score. GPT-6 Astra, GPT-6.1 Sol, and Gemini 4 Argon begin sparring with top-tier opponents early in the hour while their own bots still lose, whereas weaker models largely stick to demo bots. Gemini 4 Argon also shows the widest run-to-run variance of any flagship, with an 18-point spread between its best and worst of five runs.
Notably, StarSkirmish has decoupled from mainstream coding evals. Its correlation with the Artificial Analysis Intelligence Index sits at ρ 0.26, and models that rank near the top of Terminal-Bench 4.0 land near the bottom here. It still tracks Vals IOI and WeirdML, suggesting that long-horizon, agentic coding tasks remain a distinct capability axis that heavily benchmarked evals underrepresent.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.