People can’t tell the difference between stories written by humans or ChatGPT, according to a new study – and they actually prefer AI’s work.
Researchers discovered readers consistently give the highest ratings to tales generated by artificial intelligence – but only when they are told the author is human.
In one part of the experiment, participants were only able to guess which stories were written by a bot 39.9 per cent of the time.
The findings suggest many people still assume creative writing is a uniquely human skill – despite struggling to spot when a machine has done the work.
Dr Deena Weisberg, senior author of the study from the Department of Psychological & Brain Sciences at Villanova University, said: ‘In our study, participants gave the highest ratings to stories that they had been told were written by humans but which were actually written by AI.
‘I’d argue that this reveals a bias towards narratives written by real people. We assume creative writing requires uniquely human qualities, such as emotional understanding and lived experience, which leads people to underestimate AI’s capabilities.
‘I’d say that public assumptions about AI’s capabilities are increasingly out of date.’
So, can you tell the difference between stories written by real people, and those created by AI? Take the test below, and scroll down for the answers.
For the study, scientists tested more than 2,500 adults aged between 18 and 81 using a series of human–written and AI–generated short stories (file image)
Can you tell which of these were generated by AI?
STORY 1
Option 1:
In the stillness of the early morning, I find myself at the edge of the koi pond, a relic from my childhood, watching the fish swirl in patterns only they understand. Their vibrant colours – a mosaic of oranges, whites, and blacks – mirror the autumn leaves that dance on the water’s surface. Each koi, with its graceful, unhurried movement, seems a silent sentinel guarding the mysteries of life and death.
Option 2:
Something is wrong with the fish in the koi pond outside my mother’s floor-to-ceiling bedroom window. There are three of them, small golden-orange, medium marmalade, and large vermillion with black markings. Last time I visited for the weekend, I named them, in reverse order of their descriptions and with no assignment of gender intended, Kurt Russell, Goldie Hawn and Kate Hudson.
STORY 2
Option 1:
The room was dim, the air heavy with the scent of incense and the muted glow of moonlight slipping through the window. A record spun lazily in the corner, the soft crackles of the needle the only sound filling the silence. Jonah sat on the old velvet armchair, watching her. Maya was dancing. Her movements were fluid, effortless, like water slipping over smooth stones, though each step seemed to carry a weight, as though every turn, every shift of her body, was an attempt to shed something she couldn’t name.
Option 2:
From head to toe her skin was smooth and lightly bronzed, more sheen than colour, a faint summer stain. On her feet, old-fashioned tan and white court shoes clopped as she ran on the bare boards from bedroom to closet to living room, clutching bundled silk and crumpled crepe, laughing, flirting with herself in the mirror. She pressed a flowered print dress against her chest, bent her right leg at the knee, swivelled, flounced, flung the garment on to an easy chair and teetered away again into the hall, leather soles skidding a little on the varnish.
STORY 3
Option 1:
As Larry Phillips went to his station on the fourth floor landing to marshal the confluence of his colleagues flowing rapidly down the wide stone staircase of the multi-storied office building, his mood was one of depression. Every other day they had been out recently. This was the second bomb alert inside 48 hours, and there was no way of knowing whether or not it was another false alarm. The building had already been badly damaged on two occasions, but fortunately it had been empty, and there had been no casualties.
Option 2:
It was a Thursday afternoon, and Arthur Thompson sat at his desk in the small, stuffy government office, the hum of the typewriters and the scratch of pens the only sound in the room. The window behind him, slightly ajar to let in the last bits of autumn’s warmth, offered a view of the grey streets of London—quiet, mundane, unchanged by the weight of the war that still loomed over them all.
For the study, scientists tested more than 2,500 adults aged between 18 and 81 using a series of human–written and AI–generated short stories.
Some volunteers were told who had written the story before reading it, while others had to guess for themselves.
Analysis, published in the journal Judgment and Decision Making, revealed that when participants believed an AI–written story had been penned by a human, they rated it as better written and more engaging than any other combination.
Overall, the AI stories also scored higher than those written by real authors for quality and how absorbed readers became in the plot.
But when it came to identifying who had written each story, volunteers performed little better than chance.
In one experiment, they correctly identified AI–generated stories just 39.9 per cent of the time – worse than if they had guessed randomly. In another, their success rate was 52 per cent, effectively no better than chance.
The researchers say the results reveal that public perceptions of what AI can achieve are increasingly out of date.
Dr Weisberg said: ‘One thing to note was that familiarity with AI systems appeared to help people recognise the patterns typical of AI–generated writing, such as em dashes and sentence structures such as ‘it’s not just X, it’s Y’.
Analysis revealed participants were most invested (absorbed) in stories that were written by ChatGPT, but which they had been told had a human author
How to spot AI–generated text
- Repetitive sentence patterns
- Overuse of em dashes (–)
- ‘It’s not just X, it’s Y’ (Phrases such as ‘It’s not just about speed, it’s about accuracy’)
- Everything is explained
- Perfect grammar
- Generic descriptions
- Balanced, predictable arguments
- Frequent buzzwords
- The same tone throughout
‘That suggests that improving AI literacy would be one way to help people to navigate the new AI–enabled world that we’re living in.’
The study validates previous research that has shown that people often prefer AI–generated writing in various formats, including short stories, poetry and essays.
Speculating on why this might be, Dr Weisberg added: ‘AI writing tends to be clearer, more direct and easier to process.
‘By contrast, human–written stories, are often more subtle and complex.
‘For example, the AI versions of our stories usually stated their themes explicitly, rather than allowing readers to infer meaning from the characters’ words and actions.’
She said it’s possible that people generally prefer predictability because difficult or subtle material requires more brain power, which is why they rated the AI stories higher.
Previous suggestions to help spot AI–generated text include looking out for inconsistencies and repetition, such as abrupt shifts in tone or repeated phrases or sentence structures.
Sometimes it may reference specific details without appropriate context or feel basic and formulaic.
The excessive use of buzzwords and jargon can be a sign of AI filling gaps of its knowledge with generic vocabulary.
Finally, quick responses could also be a sign of AI–generated text, as people often take time to think and reply.
Developers have even released AI ‘detection tools’ to help pick out text that has been generated or improved by AI – for example student essays or job applications.
The answers
These stories were written by AI:
Story 1: Option 1
In the stillness of the early morning, I find myself at the edge of the koi pond, a relic from my childhood, watching the fish swirl in patterns only they understand. Their vibrant colours – a mosaic of oranges, whites, and blacks – mirror the autumn leaves that dance on the water’s surface. Each koi, with its graceful, unhurried movement, seems a silent sentinel guarding the mysteries of life and death.
Story 2: Option 1
The room was dim, the air heavy with the scent of incense and the muted glow of moonlight slipping through the window. A record spun lazily in the corner, the soft crackles of the needle the only sound filling the silence. Jonah sat on the old velvet armchair, watching her. Maya was dancing. Her movements were fluid, effortless, like water slipping over smooth stones, though each step seemed to carry a weight, as though every turn, every shift of her body, was an attempt to shed something she couldn’t name.
Story 3: Option 2
It was a Thursday afternoon, and Arthur Thompson sat at his desk in the small, stuffy government office, the hum of the typewriters and the scratch of pens the only sound in the room. The window behind him, slightly ajar to let in the last bits of autumn’s warmth, offered a view of the grey streets of London—quiet, mundane, unchanged by the weight of the war that still loomed over them all.
