AI stories rated higher than human ones—until readers learn the truth

Researchers Sydney Sears and Deena Skolnick Weisberg, publishing in the journal Judgment and Decision Making, conducted a series of experiments to test how people judge AI-generated versus human-written creative work. In the first experiment, 1,682 participants each read one of six short stories, approximately 1,000 words in length. Three stories came from well-known literary magazines and short story collections. The other three were generated using ChatGPT 4.0, with prompts designed to match the theme, style, and narrative perspective of the human originals.

The experiment employed a deception design: half the participants were told their assigned story was human-written, and half were told it came from ChatGPT. Critically, this information was accurate for only half of each group—meaning some readers were deliberately misled about authorship. The results showed a clear preference for AI-generated work. ChatGPT stories received a mean quality score of 1.54 compared to 0.97 for human-written stories on a scale from minus 3 to plus 3. On immersion, the gap was 1.42 for AI stories versus 1.00 for human ones. Yet when participants learned a human wrote the story, ratings in both groups rose above the levels given when the story was attributed to AI, demonstrating that the label alone powerfully shaped judgment.

Participants' attitudes toward AI also colored their responses. Those with positive attitudes toward AI consistently gave higher ratings across the board. When they were also told the story came from ChatGPT, their scores climbed even further. Participants skeptical of AI showed the opposite pattern. The researchers also examined whether people could distinguish human from AI work when given a direct comparison. In two additional experiments involving 905 total participants, each person read both a human-written and an AI-generated story and then attempted to identify which was which. Even with a side-by-side comparison, participants performed no better than chance. However, self-reported experience with AI systems correlated positively with the ability to correctly identify stories' origins, whereas self-reported experience with fiction did not improve performance.

The authors propose that AI-generated texts succeed partly because they tend to be smoother, easier to read, and more emotionally upbeat than human-written texts—traits that make material easier to process. Literary fiction, by contrast, is often intentionally difficult, designed to push readers to work for meaning. A story can be high-quality in a literary sense but still not engaging, and vice versa. The short-story format may also advantage AI; maintaining narrative coherence over 1,000 words is a different task than sustaining it across hundreds of pages. Nonetheless, the researchers' conclusion is unambiguous: AI can generate creative works that people perceive as at least equal to human work, yet people do not believe AI is capable of doing so. All data and materials from the study are freely available on the Open Science Framework. A separate study from Stony Brook University and Columbia Law School published last October adds nuance: professional readers preferred human texts with simple prompts, but when models were specifically trained on individual authors' styles, the experts preferred the AI-generated texts eight times more often for style imitation and twice as often for writing quality.