Computer Science › Section L · Artificial intelligence › Topic L.7

Considering the consequences of AI

AI can diagnose, translate and discover, and it can also discriminate, deceive and displace. Many of the questions it raises are still being argued over by researchers, lawmakers and the public. This topic sets out the main debates fairly, with tools for checking a system's effects instead of guessing.

Promise and peril

AI already helps doctors screen medical images, captions speech for deaf viewers, translates between languages and speeds scientific work: the 2024 Nobel Prize in Chemistry recognised computational protein design and AI-based prediction of protein structures, and that year's physics prize honoured foundational work on neural networks. The same power raises serious concerns, and whether a system helps or harms often depends on who builds it, who uses it and on whom. 'The big themes of computer science and its impact on society' introduced four lenses for ethical dilemmas; this topic applies them to AI.

Where bias comes from

A learned system copies patterns in its data, including unfair ones. Bias can enter through historical data that reflects past discrimination, unrepresentative samples, proxy variables that stand in for a protected trait, such as a postcode standing in for ethnicity, or a poorly chosen target, as in the hospital example of 'The big themes of computer science and its impact on society'. A 2018 study by Joy Buolamwini and Timnit Gebru found commercial systems that classified gender from faces made errors many times more often for darker-skinned women than for lighter-skinned men. A 2019 US government (NIST) evaluation found demographic differences in error rates in many face-recognition algorithms, varying widely between algorithms.

Worked example: auditing a screening tool

An AI tool screens job applicants. Group A: 200 applicants, 60 selected, of whom 56 were among 80 later judged qualified. Group B: 100 applicants, 18 selected, of whom 16 were among 40 qualified. 1. Selection rates: A is 60/200 = 30%, B is 18/100 = 18%. 2. Compare with the US 'four-fifths' guideline, used since 1978 to flag possible adverse impact: 18% ÷ 30% = 0.6, below 0.8, so the tool deserves scrutiny. 3. Qualified applicants missed: A loses 24 of 80, or 30%; B loses 24 of 40, or 60%. 4. Conclusion: B's qualified applicants are rejected twice as often. The figures do not prove illegal discrimination, but they show where to investigate.

When fairness definitions collide

Fairness can be measured in several ways: equal selection rates across groups, equal error rates, or scores that mean the same risk whatever a person's group (calibration). In 2016 the news organisation ProPublica reported that a widely used US risk-assessment tool for defendants wrongly labelled Black defendants who did not reoffend as high risk about twice as often as white defendants. The tool's developer replied that its scores were equally well calibrated for both groups. Researchers soon proved that, when groups' underlying rates differ, calibration and equal error rates generally cannot both hold. Choosing which fairness to prioritise is therefore a value judgement, not only a technical fix.

Work and the economy

Automation has always destroyed some jobs and created others: mechanised looms displaced hand weavers in the nineteenth century, while whole new industries later grew. Optimists argue AI will mainly automate tasks rather than whole jobs, raise productivity and free people for more rewarding work. Others argue that AI reaches cognitive and creative tasks once thought safe, that change may come faster than retraining, and that gains may flow mainly to those who own the technology, widening inequality. Economists disagree about the size and speed of these effects. Proposed responses include retraining, stronger safety nets and policies on sharing productivity gains.

Responsibility and human judgement

When an AI system causes harm, who is responsible: the developer, the organisation that deployed it, or the person who relied on it? Many networks are hard to interpret, which makes decisions hard to explain or contest, so researchers work on explainable AI and many guidelines call for human oversight of high-stakes decisions. But oversight can fail through automation bias, the tendency to accept a computer's output even when other evidence contradicts it. Joseph Weizenbaum argued in Computer Power and Human Reason (1976) that some decisions call for human wisdom and judgement and ought not to be handed to computers, even if computers could make them; others reply that careful, tested systems can be fairer than inconsistent human judges.

Truth, privacy and ownership

Generative AI produces realistic text, images, audio and video. That serves art and accessibility, but deepfakes can impersonate real people, spread false claims or harass, and they make genuine evidence easier to dismiss as fake. Responses include labelling AI-generated content and recording a file's origin, though labels can be stripped. Face recognition and data analysis also make large-scale surveillance cheap. Ownership is contested too: authors, artists and news organisations have sued AI developers for training on their work without permission, while developers argue such training is lawful; courts in several countries were still weighing these cases in the mid-2020s.

Law and governance

The European Union's AI Act, adopted in 2024, regulates by risk: a few uses, such as certain kinds of social scoring, are banned; high-risk uses, such as hiring, credit decisions and critical infrastructure, face duties of risk management, documentation and human oversight; chatbots and similar systems must tell users they are dealing with AI; and most other uses face few rules. Its obligations phase in over several years. As of 2025 the United States had no comprehensive federal AI law; it relied on existing laws, executive orders, state legislation and the voluntary NIST AI Risk Management Framework of 2023. International principles, such as the OECD's of 2019, aim at common ground.

Safety, control and the environment

Some researchers worry about future systems far more capable than today's. They argue that an advanced AI pursuing goals not aligned with human values could cause severe harm, and in 2023 many AI scientists and industry leaders signed a short public statement saying that reducing the risk of extinction from AI should be a global priority. Others reply that such scenarios are speculative and that the focus distracts from present harms such as bias, surveillance and concentrated corporate power. There is also a physical cost: training and running large models uses substantial electricity, and data centres use water for cooling, though estimates of the totals vary widely.

Worked example: weighing an AI-writing detector

A school considers using software that flags essays as AI-written. 1. Purpose: protect honest assessment. 2. Stakeholders: students, teachers, parents, the vendor. 3. Errors: a false positive accuses an honest student, a serious harm; a false negative misses some cheating. Detectors are known to make both kinds of error, and at least one study has reported that writing by non-native English speakers was flagged more often. 4. Lenses: consequence-based ethics weighs wrongful accusations against cheating deterred; duty-based ethics stresses fair treatment and a right to respond. 5. Decision: a defensible policy treats a flag as a reason to talk with the student, never as proof, and checks the error rates for different groups.

Common mistakes to avoid

First, assuming mathematics makes an algorithm neutral; it inherits its data and design choices. Second, believing that dropping a protected attribute removes bias, when proxies such as postcodes can carry it. Third, treating one fairness measure as the definition of fairness. Fourth, reading a four-fifths flag as proof of discrimination rather than a reason to investigate. Fifth, presenting contested predictions about jobs or catastrophic risk as settled fact, in either direction. Sixth, assuming a human in the loop guarantees safety, when automation bias can make oversight a formality.

Key takeaways

AI brings real benefits and real risks. Bias enters through historical data, unrepresentative samples, proxies and targets; audits compare selection and error rates across groups, and the four-fifths guideline flags disparities for review. Fairness measures can conflict, so choosing among them is a value judgement. The effect on jobs is debated. Responsibility, explainability and automation bias complicate oversight. Deepfakes, surveillance and copyright raise further disputes. The EU AI Act regulates by risk; other jurisdictions differ. Long-term safety and environmental costs are debated too.

Flashcards

Flashcard

Algorithmic bias

Systematically unfair outcomes produced by a system's data or design

Not removed by being mathematical.

Test your understanding

Question 1 of 5

A tool selects 40 of 100 applicants from group X and 15 of 50 from group Y. Under the four-fifths guideline, what is the result?

Question 2 of 5

Two groups have different underlying rates of the outcome a risk score predicts. What did researchers prove about such scores?

Question 3 of 5

A nurse accepts a system's 'low risk' rating for a patient despite worrying symptoms, because the system is usually right. What is this an example of?

Question 4 of 5

How does the EU AI Act, adopted in 2024, regulate artificial intelligence?

Question 5 of 5

A loan model is trained only on past applicants from a bank's city branches and is then used for rural applicants. What is the main fairness risk?

← Previous topicNext topic →