Yes, Lean Verification Matters—But AI Proofs Need Testing
Original source
Interpreting claims that a theorem is correct because it was checked in Lean
Hacker News →Evaluating mathematical theorem proofs verified in Lean requires careful scrutiny, particularly when the code is AI-generated. The importance of such evaluation became evident following OpenAI’s release of 722 mathematical papers, roughly 45% with accompanying Lean code—especially after three non-Leanified papers from that batch were retracted for sign errors. Proper evaluation requires addressing four critical questions.
The first asks whether the formal Lean statement matches the theorem being claimed. This verification proves non-trivial; some theorems require thousands of lines of supporting code just to express their statement correctly. DeepMind’s formalization of the Hodge conjecture needed over 10,000 lines of additional code. Axiom’s formal proof of the classification of finite simple groups relied on 250,000 lines across 900 files. Of OpenAI’s 290 theorems with Lean code, roughly 38 don’t formally state the paper’s main result—some prove only intermediate corollaries instead.
Second, the code must compile. Anthropic’s formalization of Fermat’s Last Theorem involved 13 million lines and required a machine with 96 cores and 500 gigabytes of RAM. Smaller proofs like OpenAI’s Navier-Stokes formalization at 640,000 lines compile on standard hardware. Third, the proof must actually establish the claimed theorem—a substantial challenge when proofs import vast amounts of untrusted generated code. For many entries in OpenAI’s theorem dump, these systematic validation steps haven’t yet been performed.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.