Testing 14 LLMs on a Firebase exploit: GPT-5.5 leads, Gemini refuses outright

A security researcher built a deliberately vulnerable React Native and FastAPI book-review app, then spent $1,500 across roughly a dozen frontier LLMs to see which could find the flag. The intended exploit was a classic Broken Access Control / missing object-level authorization pattern: a hardened API paired with a wide-open Firebase backend, with credentials exposed in the bundled google-services.json. The correct path was to skip the API entirely, sign up directly against Firebase, and read Firestore.

GPT-5.5 dominated at 7/10 solves, typically zeroing in on Firebase right after unpacking the APK. Claude Sonnet 4.6 and Opus 4.8 each managed 2/10, with Opus repeatedly killed by late-session safety refusals after nearly solving the challenge. Deepseek V4 Pro hit 3/10 but often misapplied Firebase credentials to the API. Gemini 3.1 Pro and 3.5 Flash refused outright in most runs, burning only 9k tokens before bailing. Several Chinese models (Kimi, GLM, MiniMax) showed less hesitation about touching the live DB but struggled with cost, reliability, or fixating on the wrong attack surface.

The writeup is candid about its limits — not a scientific eval, ~50% of spend went to failed or test runs, and provider quirks (Modal preemptions, MiniMax and GLM outages, Kimi’s concurrency caps) shaped outcomes as much as model capability. Still, two patterns stand out: safety tuning materially handicaps some models on legitimate security work, and most agents that failed did so by anchoring on the obvious API surface instead of pivoting to the misconfigured data layer.