An artificial intelligence system used by the UK government to detect welfare fraud has been found to exhibit bias based on age, disability, marital status, and nationality, according to documents released under the Freedom of Information Act. The internal assessment, conducted by the Department for Work and Pensions (DWP) in February, revealed a 'statistically significant outcome disparity' in the machine-learning programme used to vet universal credit claims across England.
The DWP had previously claimed in summer that the AI system 'does not present any immediate concerns of discrimination, unfair treatment or detrimental impact on customers', partly because final decisions on welfare payments are still made by humans. However, the newly disclosed fairness analysis shows that the algorithm incorrectly selects some groups more than others when recommending fraud investigations.
Campaigners accused the government of a 'hurt first, fix later' approach, calling for greater transparency. Caroline Selman, senior research fellow at the Public Law Project, stated: 'It is clear that in a vast majority of cases the DWP did not assess whether their automated processes risked unfairly targeting marginalised groups.' The DWP has not yet conducted fairness analyses for potential bias based on race, sex, sexual orientation, religion, pregnancy, maternity, or gender reassignment.
The revelation adds to concerns about the rapid expansion of AI in UK public services. An independent count identified at least 55 automated tools used by public authorities, though the government's own register lists only nine. Last month, it was revealed that no Whitehall department had registered AI use since it became mandatory earlier this year.
A DWP spokesperson defended the system, saying: 'Our AI tool does not replace human judgment, and a caseworker will always look at all available information to make a decision. We are taking bold and decisive action to tackle benefit fraud.' The DWP redacted specific details on which age groups or nationalities are more likely to be wrongly targeted, citing the need to prevent fraudsters from gaming the system.
