Study: AI advice cut accuracy to a third while doubling users' confidence

Researchers from three French and Italian universities ran an experiment designed to catch AI at its weakest: they asked people trivia about visual details from films, then offered advice from a model (Step 3.5 Flash) that reliably got those questions wrong. The point was to rule out sensible delegation to a trustworthy tool. When AI was available, participants’ willingness to say “I don’t know” collapsed from 44% to 3%, accuracy fell from 27% to 9%, and self-reported confidence jumped from 30% to 76%. Some people who would have answered correctly on their own consulted the model and adopted its wrong answer. Cash incentives nudged the numbers slightly — admissions of ignorance rose to 8%, accuracy to 16% — but never came close to the no-AI baseline.

The striking finding isn’t just that people trust incorrect AI output; it’s that the mere presence of an AI assistant erodes the habit of recognizing the limits of one’s own knowledge. Co-author Valerio Capraro frames the ability to say “I don’t know” as a core cognitive skill, and worries most about children who lean on these systems before developing critical-thinking habits of their own. The work echoes Wharton researchers’ notion of “cognitive surrender,” where users accept bad AI answers roughly 80% of the time while feeling more confident than peers working unaided.

The results land amid a broader design problem: consumer AI products are built to always produce an answer, never to abstain. Google’s shift from search links to confident AI-generated summaries drew a warning from Common Sense Media this week, which called the format an “unacceptable risk” for students. The pattern reinforces itself — tools that never hedge are training the people who use them to stop hedging too.