A new study suggests people struggle to tell the difference between stories written by a human and those generated by AI, and even prefer AI-written text when they believe a human authored it.
Study Details and Results
Researchers at Villanova University, led by Dr. Deena Weisberg, published findings in Judgement and Decision Making. Participants read six story extracts: three from human authors and three generated by ChatGPT. Across multiple tests involving hundreds of participants, accuracy in identifying the human-written stories was often low. In one test, only 39.39% correctly identified the human story, while 40.8% identified the AI one. In another, 51.97% correctly spotted the AI-generated text.
AI Stories Rated Higher
When participants believed a human wrote the AI-generated stories, they rated them higher on quality and absorption. AI stories scored 1.54 for well-written and 1.42 for absorption, compared to human stories at 1.00 and 0.97. Dr. Weisberg noted, 'People value the authenticity of human-made art.' She added that people may assume a liked story was written by a human, even if it wasn't.
Implications and Perceptions
The study does not conclude AI is better at writing, but highlights growing difficulty in distinguishing AI text. Weisberg explained that AI models draw from vast training data to assemble stories word by word. The findings also reveal biases about AI capabilities. Similar studies have shown readers may prefer AI books, yet polls indicate distrust in AI's creativity.
