Welf(AI)re: A Racial and Gender Injustice Analysis of AI and Automated Decision-Making in UK Social Security: a new report by Glitch
Today, we are publishing our report, Welf(AI)re: A Racial and Gender Injustice Analysis of AI and Automated Decision-Making in UK Social Security, on the use of AI, algorithms and automated decision-making systems known to be used in the context of housing, Universal Credit or other benefits.
Taking a systematic review approach to offer a bird’s-eye view of the evidence, literature and debates on ADM systems in welfare, we argue that racism and discrimination is baked into the AI-ification of services, that the use of data and algorithms further embed and reinforce unfairness and injustice, and that algorithmic tools may lead to racialised people being denied access to housing and rejected from welfare claims.
As the Government accelerates public sector adoption of AI tools often on the basis of cost savings and efficiencies, we urge for the preservation of human rights, equal access, and racial and gender justice, which cannot — and should not — be so easily quantified in GBP. For Black women and Black gender-expansive people, and communities who are already subject to systematic oppression, automation is likely to reproduce logics of exclusion, surveillance and criminalisation along racial and classed lines.
Our research found:
The increased capture of welfare by AI and ADM systems
AI, algorithms or automation are likely in use in almost every stage of the Universal Credit process. Automated decision-making tools are governing access to welfare and housing — from calculating payments to assessing entitlement, and automated tools are being used to match and verify highly sensitive data within and between public bodies.
Opacity as a feature, not a bug
Public details on the AI systems used in the public sector remain opaque and inconsistently held. Government documentation is particularly unreliable; and even landmark transparency measures such as the Algorithmic Transparency Reporting Standard have failed to result in significant change. Uncovering the extent of AI systems in use has predominantly been the purview of civil society and media investigations, requiring extensive freedom of information requests: making ongoing public scrutiny and transparency to individuals affected by these systems incredibly challenging.
AI and algorithmic anti-governance
At present, there is no overarching or holistic legal framework pertaining to governing AI in the United Kingdom. Instead various areas of law and regulation speak to some particular uses of AI, regulating AI in practice through fragmented and dispersed, predominantly “technology-neutral” legislation and non-statutory guidance - guidance that is not compulsory to follow, and which is often confusing, unspecific or contradictory. The existing protections we do have are also severely at risk, in the midst of political pressure to reduce human rights, data privacy and equality protections.
The experiences of Black women and Black gender-expansive people are in the margins
We were met with omissions and silences when it came to the question of the impacts of AI systems on our focal population, Black women and Black gender-expansive people. This omission is not accidental, but rather reflects a systemic disinterest in examining how AI and algorithmic decision making systems might disproportionately target people already subject to systematic oppression. For Black women and Black gender-expansive people, this silence constitutes a form of institutional neglect — their experiences of disproportionate scrutiny, wrongful suspicion, or disrupted payments when accessing the welfare state — are, for the most part, not only unaddressed, but also unmeasured.
Hyperfocus on bias
Where potential impacts on Black women and Black gender-expansive people are discussed in literature about AI in welfare, this tends to be through the lens of overarching concerns about AI ‘bias’. This widespread acceptance of the potential for algorithms to facilitate ‘biased’ and discriminatory decision making is not a bad thing: these systems do have significant negative impacts on the people subject to them.
But a hyperfocus on bias as the principle threat to racial and gender justice in AI risks obscuring the histories of systemic inequality and oppression that explain why racialised communities, in particular, are overrepresented in particular datasets in the first place - and risks leading to more data collection and surveillance of Black communities in service of developing more ‘representative’ datasets. We should be cautious about seeing this acceptance of the existence of bias as an algorithmic justice ‘win’ and potentially foreclosing the possibility of questioning the legitimacy and wider usage of AI, and automated decision-making systems wholesale.
Reproducing Racist Logics
Automated decision-making facilitates wider racial and class oppression. The distribution of social housing since its inception has involved assessments of people on the basis of racialised and classed distinctions: with new technological assessments reproducing old forms of discrimination, which are more easily invisibilised when they are made digital or automation and AI systems are introduced. There are also ways that ‘race’ and racialisation can show up as proxies in data, even when protected characteristics themselves are not specifically analysed: such as where proxies that disproportionately affect Black communities are used to determine social housing distributions.
Interconnected data
By automating systems that are already deeply flawed, the opportunity for discrimination to become further engrained is maximised. The welfare system is itself mediated through multiple different interfaces, and so Black women and Black gender-expansive people may be affected by multiple applications of automated decision-making within the system. Data collected for one purpose (such as for welfare) may end up being shared with other public services (like policing) and vice versa. This phenomenon is an example of wealth, race and social privilege shielding certain populations from algorithmic injustice: the less you interact with the State, the fewer data points likely exist about you that are accessible by public bodies. Consequently, the State is less able to track you across society and use data for punitive AI systems, which in turn might be used to re-target you and your communities. Automated decision-making systems by their very nature create systems of unequal (dis)advantage for those whose lives do and do not rub up against them.
Instead of acquiescing to the dogma that AI adoption is inevitable, our report calls for us to imagine a different model for public services, as infrastructure which is not just built in pursuit of speed, scale, or efficiency, but meets the material needs of the people that depend on it.
You can read our full report here.