Stanford study: law professors prefer AI tutor answers to peer responses 75% of the time

A Stanford-led study had 16 law professors from U.S. institutions write answers to 40 contracts law questions, then blindly evaluate roughly 3,000 head-to-head comparisons between their peers’ answers and responses from large language models. AI won 75% of matchups. More telling for educators: professors flagged AI answers as pedagogically harmful only 3.5% of the time, versus 12% for human-written ones. Researchers controlled for length and structure to keep the comparison honest.

The result matters because legal reasoning was supposed to be a hard case for LLMs. Unlike math or coding benchmarks, contracts questions often have no single correct answer and reward judgment, synthesis, and the ability to defend a position against a plausible counterargument. The authors, from Stanford, Yale, NYU, and Chicago, frame the finding as evidence that LLMs can meet the professional standard lawyers use to judge each other’s work, at least at the tutoring level. Performance varied across systems tested, including Google’s NotebookLM and commercial tutoring tools.

Lead author Julian Nyarko is careful not to call this a green light for wholesale AI tutoring in law schools — open questions remain about hallucinations, student overreliance, and atrophy of analytical skill. But the data undercuts blanket skepticism and shifts the debate from whether AI can produce competent legal explanations to how schools should deploy it without undermining the training of future lawyers.