AI in health care is changing what we mean by risk

Artificial intelligence (AI) is changing how people access and receive health care – from patients using chatbots like ChatGPT to understand their symptoms to clinicians deploying AI tools to interpret medical imagery or produce clinical notes. 

While the potential benefits of AI are significant, its use is also challenging some of our assumptions about what counts as harm, and where responsibility lies for preventing it. To support the MHRA's National Commission into the Regulation of AI in Healthcare, for which the Foundation is a research partner, we undertook a rapid evidence review of academic and non-commercial literature on the potential harms posed by AI use in health care and how to categorise them.

In this blog, we share what we learned and its implications for thinking about and mitigating harm in health care. If regulation is to keep pace with the rapid development of AI, we need to consider our assumptions about what regulation is trying to protect against.

Familiar problems, amplified by AI

Our review found that many of the potential harms associated with AI are not new. Health care has long had to manage risks such as inaccurate information, diagnostic errors, inappropriate treatments and unequal access to care. But our review also identified sources of harm specific to AI, including hallucinations – where AI systems make up false information but present it as fact– and AI systems with embedded values that don’t align with health care aims. For all potential health care harms, AI can change the scale, speed and nature of the risks to patients and the health system. 

AI can also make existing harms harder to detect, prevent or attribute. An error by an individual can become a systematic problem when embedded in a widely used system, biases in training data can be reproduced across many decisions and authoritative-seeming outputs can influence patients or professionals even when they are wrong. Responsibility may also be distributed among developers, deployers and users.

Harm is broader than patient safety

Our review identified potential harms from AI at several levels. For individuals, these include physical or emotional harm from inaccurate advice, misdiagnosis and overdiagnosis, data privacy violations and reduced patient agency. At a community level, AI could contribute to unequal access, outcomes or distribution of resources. At a societal level, harms could include the amplification of existing biases and difficulties establishing accountability when things go wrong.

This suggests that a definition of harm centred only on direct physical injury may not be sufficient to understand the harms posed by AI in health care. The application of AI in health care thus requires an expanded definition of harm. This should include non-physical harms, such as emotional distress and impacts on wellbeing, as well as introducing new categories of harm, like systemic inequalities and existential risks.

This is important because what we choose to define as harm shapes what regulators look for, what evidence developers are expected to produce and which risks receive attention. 

The same AI can create different risks in different contexts

We found that the potential for harm depends not just on the technology but on who is using it, how and in what setting. A tool used by a trained professional within a system of clinical oversight presents different risks from a similar tool used directly by someone seeking advice about a serious health concern without professional input. Factors including a person's level of agency or vulnerability, and the capability of and support available to staff, can all influence the likelihood and consequences of harm.

This creates a difficult regulatory challenge. A narrow definition of harm risks overlooking important consequences of AI, but an overly broad one could dilute regulatory focus and make it harder to prioritise the risks that matter most. We need approaches that are sensitive to context and capable of responding to how AI is actually used in the real world.

We need a clearer understanding of what we’re trying to prevent

Our review found no single existing framework that fully captures the range of potential harms associated with AI in health care or provides the considerations regulators need to prioritise mitigating them. We also found that the evidence on the real-world impacts of AI remains limited.

This points to the need for a more systematic approach to rapidly testing and evaluating AI in health care. We need to learn not only whether existing categories of harm capture what happens in practice but also where they fall short and how new forms of harm might emerge as AI changes how care is delivered. This will enable a clearer shared understanding of what counts as harm and better evidence about how harms emerge in different settings and who may be disproportionately affected. 

Patients and the public should also be part of defining the risks that matter, how these should be evaluated and the expected protections. AI use does not simply add another set of risks for health care to manage. In some cases, it is changing the boundaries of what we understand risk and harm to be. If regulation is to keep pace, it will need to develop a sufficiently broad understanding of harm to avoid missing what matters, while remaining focused enough to act effectively.