Researchers from Cardiff University and Ca' Foscari University of Venice have found that large language models (LLMs) struggle to understand puns, despite being able to identify their structure. The study, presented at the 2025 Conference on Empirical Methods in Natural Language Processing in Suzhou, China, suggests that AI's grasp of humour is fragile and often illusory.
The team tested LLMs on puns such as 'I used to be a comedian, but my life became a joke.' When the punchline was replaced with a non-pun word like 'chaotic', the models still perceived a pun. Similarly, in 'Long fairy tales have a tendency to dragon,' substituting 'dragon' with 'prolong' or a random word did not fool the AI into recognising the absence of wordplay.
Professor Jose Camacho Collados of Cardiff University said: 'In general, LLMs tend to memorise what they have learned in their training. As such, they catch existing puns well but that doesn't mean they truly understand them.' He added that the models could be consistently fooled by modifying puns to remove double meanings, leading them to invent justifications for why a sentence is a pun.
The study found that LLMs' success rate in distinguishing puns from non-puns dropped to 20% when faced with unfamiliar wordplay. In one example, the pun 'Old LLMs never die, they just lose their attention' was altered to replace 'attention' with 'ukulele', and the model still identified a pun based on a phonetic similarity to 'you-kill-LLM'. The researchers noted this creativity but stressed that the AI had not understood the joke.
The findings highlight limitations in using LLMs for tasks requiring humour, empathy or cultural nuance. The paper, titled 'Pun unintended: LLMs and the illusion of humor understanding', was presented earlier this month.
