"Intelligent and likable": Fluent but wrong AI fools users, study finds

The study, described by co-author and associate professor Barbara Müller, suggests that when text is polished and natural-sounding, people automatically perceive intelligence and unconsciously stop checking for errors. Co-author and professor Tibor Bosse adds that large language models are trained to produce statistically plausible, readable text — not necessarily truth — and while most chatbots display a general disclaimer that outputs may be inaccurate, they rarely express uncertainty for each answer.

Bosse argues that if companies were required to have AI state its own doubt, that could help prevent people from accepting falsehoods as fact. The findings point to a gap between how convincing an AI sounds and how correct it actually is.