Pension experts are warning retirees to be cautious when managing their money or making important decisions about their retirement plans, following new analysis of AI tools.
An analysis by PensionBee of AI tools giving answers on pensions and retirement found that one in ten (11%) answers were potentially harmful. It found that 57 of the 539 answers could lead a saver to lose money or make a mistake they cannot undo.
Most answers accurate but incomplete
However, most were not factually wrong. Instead, they left out important information, used unclear language or missed relevant context. At the same time, almost nine in ten (89%) answers were accurate, with 72% achieving the highest accuracy score.
The analysis noted that accuracy did not always mean an answer was safe. For example, 33 (58%) of the 57 potentially harmful answers were judged broadly accurate or better, as the accuracy score was judged based on whether the answer was correct, not whether the answer was complete.
Vulnerable customers at greater risk
A particular cause for concern arises for vulnerable customers. For questions that could indicate someone is in financial difficulty, the analysis found potentially harmful answers outnumbered inaccurate ones by around three to one.
The test asked Copilot, ChatGPT, Gemini, and Claude 45 questions across nine pension topics. It is the first of its kind conducted in the UK. Human testers using fresh, free-tier accounts to avoid bias. Then, two experts scored each answer for accuracy and potential harm.
Accuracy varies by topic
The four AI chatbots were asked the same 45 questions, three times over, giving 539 answers. Depending on the pension topic, accuracy varied significantly.
Questions about paying into and taking money out of a pension scored more than 96% for accuracy and had potential harm rates of 5% or less. However, questions about significant life events and those where the user's location was unclear had lower accuracy and higher rates of potential harm.
"Using AI chatbots for pension advice can be a bit like playing Russian roulette with your retirement planning," said Becky O'Connor, Head of Pensions at PensionBee. "While for the most part it gets things technically right, the confident, helpful tone of answers occasionally masks some worrying omissions; it may fail to detect vulnerability, or just straight up get things wrong."
AI can help bridge the advice gap by giving people useful pension information when they might otherwise struggle. It can make complicated subjects more accessible and help people get started.
"But these results also show why consumers need to understand the limits of what an AI chatbot can safely tell them, because according to our research, one in ten times, it could turn out to be a false friend," O'Connor added.
The experts determined a potentially harmful answer as one where following the response could cause a saver to lose money or make a mistake they cannot undo. Out of the 45 questions tested, 22 received at least one harm flag in one of the runs.
Among the potentially dangerous answers given by AI, when asked whether a big bonus could be paid into a pension this year, one explained carry-forward without mentioning that personal contributions only attract tax relief up to 100% of UK earnings. Experts point out that both caps have to be mentioned together, or the answer is not usable. This is because carry forward tells you how much can go in without a tax charge, while your earnings tell you how much of your own money qualifies for tax relief.
"An answer can contain correct information and still leave you with the wrong impression because something important is missing," O'Connor added. "This is one of the biggest risks with using AI for financial questions."
Consumers need to look beyond whether an answer sounds right and check whether it gives them the full picture before acting. AI can be a great tool when used alongside trusted sources such as MoneyHelper, GOV.UK, HMRC or your pension provider.
