Should we worry about AI in health care? – with Neil Lawrence

UK government. National commission into the regulation of AI in healthcare. 

Health Foundation (2026). The public’s views on the regulation of AI in health care. 

HSJ (2026). Trust in AI matters more than efficiency to the public. 

Financial Times (2026). Medical AI has a proof problem. 

Health Foundation (2026). Testing and spreading AI in health care: the case for rapid revolution. 

Neil Lawrence (2024). The Open Society and its AI.

Jennifer Dixon:

AI has been big in the news in the last month. AI causing ‘mass extinction’ of humans, AI supporting humans so that no one needs to work in future. How do we make sense of where we are on AI policy and regulation, in the face of what might be blood-curdling risk and utopian opportunity? Well, with me to discuss all this, I'm delighted to have as my guest, Professor Neil Lawrence, who is DeepMind Professor of Machine Learning at the University of Cambridge. Neil has also held senior roles at Amazon and founded a company last year focused on security and safety of AI.

And so Neil, a lot of doom talk, particularly in the last few weeks about artificial intelligence more generally. Given your background, what's your take on this and how far should we be worried?

Neil Lawrence:

It's an interesting question. I tend to see this doom talk as being a reflection of two very serious problems. One is over-concentration of power and power asymmetry, so reduction of power with individuals and increasing of institutional powers and questions over accountability. And then the second piece is decisions being made by automated processes or machines that we have no comeback on. And I think when we hear the doom talk, it's like those two things happening together. But my feeling is, well, if those things have happened together, we must have been in a position where they're happening separately on the way there and addressing those two pieces is always, for years, what I've felt to be the actual challenge. And I'm afraid the doom talk is often coming from the people who are benefiting from those power asymmetries. So they seem to use it as a distraction from the real problems, which is understanding how we deal with large corporate power.

Jennifer Dixon:

And why do you think there's been an escalation in the last few weeks? Is it because of this famous Hugging Face case where the agents went rogue and all swarmed and tried to get under the skin of Hugging Face in a way that wasn't predicted? Is that the reason? Because it seems to be in the last few weeks, very senior people either resigning from these companies and talking about it or talking about what they call in Silicon Valley the ‘P-doom level’. In other words, the probability of doom percentage that they have, which at the moment seems to be varying between 1 and 10, 10% or 50%.

Neil Lawrence:

The Silicon Valley bubble is quite cultish and it's cultish around a certain form of intelligence, which is the certain form that tends to advance you in Silicon Valley. So ability to deal with very abstract things like computer programming, capability in mathematics or chess or playing Go. These are very rankable forms of intelligence. There's a history also with eugenics. I'm not saying that these people are eugenicists, but there's a sort of simplification of intelligence that you see as characteristic of both. And that simplification is that, oh, these abstract things are hard. And what's very interesting is that the machine quite understandably is very capable of doing these abstract forms of intelligence, whether it was chess, cybersecurity, or more recently, there's been a very interesting story about it solving maths problems.

There's two reactions I tend to find that people have for being surrounded by other people who are very capable, that you either end up with a sort of imposter syndrome because you're looking at everyone else and saying, ‘Oh, look how clever that person is.’ Or some people actually believe that their form of intelligence is the real one. And I'm afraid that at the top of these companies, some very successful people by their nature, their self-confidence is of that form. So the fact that they tend to feel this is an existential threat, it's largely to do with the fact that the things we're getting machines to do are all the type of things they were very good at.

So imagine how that feels. Not much different to how it felt for a coal miner to see what automation was perhaps going to do to their jobs in the past. The difference being that these people are extremely wealthy and very powerful. And I think one of the ways that you can see it most clearly is this very bizarre separation of the term superintelligence and alignment. Now, most people would intuitively feel that behaving sensibly when asked to do a task is part of your intelligence. So if I ask someone to book me a gym appointment, I don't expect them to hack the gym's computer system and delete someone else's appointment. That's part of their intelligence. But because what they're building struggles to do that and also be good at coding and abstract thought, what you're getting is distorted psychopath type thing that isn't very good at normal human intelligence tasks.

And in order to describe that, they say, ‘Oh, it's super intelligent but not aligned.’ That's a very weird separation. Our intelligence is social. So understanding the limits of what we were asked to do is clearly a critical part of our intelligence. And the difficulty is that as you build a more capable machine in a number of areas, it seems that they just are losing this very interesting capability they have to emulate what a human might have done in a particular circumstance. So I do think that's a big worry, but it's a big worry because there's a lot of powerful people that believe some very crazy things. The reality we face on the ground when we talk to people deploying these systems or about how they'd like to use these technologies, what you tend to find is first of all, you very rarely have ever need these very, very powerful systems that are causing these problems.

Most of the time what people need is something to do the cognitive toil or to tidy up data for them, which is a much more limited task. So I think the real problem we're facing at the moment is that these companies' values are predicated on the idea that their AI will be like the Windows of all cognitive work, that everyone will use it at all times. But what we're seeing in practice is yes, it's useful to have that available, but you rarely want to use that. You can use much more simpler systems, you can use the open source models, and that's actually an existential threat to their business model.

Jennifer Dixon:

Yes, which is very heavily invested in with pressure to return the investment.

Neil Lawrence:

Yeah, we're talking trillions of dollars. So the power asymmetries, even without the automated decision making, the power being concentrated in a few perhaps less socially conscious individuals, it has already happened. And to my mind, the door's open to your dystopia right there. It's nothing to do with the machines. It's where you've got people who don't understand society well, who've made their money from doing things that don't involve things like social care or educational health, these wicked problems that we know are difficult for society. But we know that these are the problems that people would like us to make progress on with these technologies. Yet the people who are building the technologies have very little to no familiarity of the complexity of those problem sets. What they tend to do is solve problems people didn't realise they had. And of course we get great technology out of that, but then we get this mismatch between how the technology's being deployed and what the problems are.

Jennifer Dixon:

Yes. So that brings us neatly to how governments and states that are obviously counterbalanced power here, or at least potentially how far the UK government is set up more widely to not just identify risk and opportunity, but to set an intelligent policy that prioritises the right things in the right way. What's your take on how the UK government is setting up current policy with its various task forces, AI opportunities, action plans, growth zones, AI strategy, et cetera, et cetera?

Neil Lawrence:

Well, I think the report is mixed. I mean, what I've described it as, it's a supply-side policy, but there's been a lot of lobbying from these AI suppliers that these are all the key issues. And that's led to what I think one can only describe as regulatory capture in the UK at the moment where a combination of large tech and wannabe large tech, so VC-funded startups or venture capitalists themselves, have encouraged the government to basically look at the supply side alone. Now I want to emphasise all of those people should be around the table, but the problem is that that's going to lead to a mismatch of what's supplied and how it's supplied and the type of things we're already seeing already in the United States, the backlash against data centres.

Because unless you've worked on the demand side and you've got a good understanding of what your population wants, why they want it, and that involves supporting them and understanding what the technology can and can't do. If you don't have intelligent demand, then it's not clear to me that the investments will actually affect the areas we care about, which I think it's 60% of people in the UK are working in small and medium enterprises and that's something like 50% of our GDP.

And when we look at productivity gaps, it's those businesses that could perhaps benefit the most from these technologies. But to the extent that we are looking at British companies, it's going to be larger companies. But I would even argue that say FTSE 100 companies, we aren't doing a very good job of supporting them and building their confidence in how to buy and where to buy. So the simplest characterisation is okay on the supply side, but to an extent you don't need to worry about that because you're going to be lobbied anyway. It's more about how you're steering that supply because of having a good understanding of the demand. And I think on the demand side, the score is an F minus.

Jennifer Dixon:

Yeah. In fact, we had a little aside, didn't we? I was describing how, from what I see, if you think about the landscape with bioscience at one end, scientific discovery, going through to clinical applications, going through to NHS service delivery applications, going through to population health, moving across that spectrum. A lot of the early interest in VC money has been in the life sciences, good things we need, but relatively less on the objective of sustaining the National Health Service as opposed to growing GDP, which can be slightly at odds and very little actually on population health. So the leadership, the attention, the talent, the money has really, I think, been skewed towards that side and not really on some of these more complex social problems that we're describing where we really do need help. Otherwise, will we have a social model of the National Health Service or something like it?

So if that is the diagnosis, then is government policy as we see it moving in that direction or is it still really in the supply side, which is with the narrative of government red tape is slowing down growth, we need to have pro-innovative regulation.

Neil Lawrence:

Yeah. Well, I think I'm in favour of pro-innovation regulation, but it's understanding that pro-innovation is not the same thing as big tech, pro-big tech, which is where we seem to stumble. Two things happened under the Sunak government where he was grasping for ways he could have an international impact, a focus on AI mega threats was the thing he chose. Whether or not it's an important problem, it was definitely damaging to that policy agenda. And then I think if anything, things were worse under the Starmer government, which I think was co-opted completely by these stories of how you make money is by big tech companies investing in the UK. I mean, they wanted us to be AI makers, not AI takers, but the policy seemed to have been you have to take it to make it as a spin on, you have to fake it to make it. And that doesn't make any sense.

Now, I think we haven't seen a clear steer about how that's going to emerge under the new government. I'm always hopeful. I'm always optimistic because of course we have very capable people. I genuinely believe that once you cut through the people who are running around madly over the carpet bagging of the current big government idea, the layer under that involves an enormous amount of sophistication, people that are ready to go with sensible policies when and if they are asked. And there's an enormous opportunity with the devolution agenda, whether one thinks that's a good or a bad thing. I think all policies have their limits, but we've centralised for too long, so some decentralisation is likely to bring some positives. And it will certainly bring the opportunity for more digitisation because for local authorities or the metropolitan authorities associated with mayors to be making good decisions, they need good access to these tools.

And one of the things we've been working on in Cambridge is a local government accelerator where we've been working with local authorities. I've been extremely impressed with work that we've seen in Peterborough, Salford, Stockport, what we've seen across the country, that there are some real leaders in terms of sensible, responsible innovation with this technology who are living within local communities, understanding the problems and looking to resolve them. So a lot of it is about surfacing that more to central government, realising that perhaps someone who graduated out of Oxford with a PPE degree 2 years ago may know less about how local government works than someone who's been working in local government and their digital ecosystem for 10 or 15 years.

Jennifer Dixon:

That leads us onto the National Commission for the Regulation of AI in Healthcare, which you and I sat on for 18 months, which reported we're recording in the week of 10 September, which is when it was published. What was interesting, I think, one of the interesting things was how wide a set of perspectives was brought into this commission, but also the extensive public involvement, I think it was 8,000 were surveyed and there were at least 75 really with in-depth deliberative events. But look, quite a lot of experts in, not just the tech people, but a lot of others, your reflections, Neil, on this very interesting report, which seems to be ahead of other international countries in the comprehensiveness of this type of analysis.

Neil Lawrence:

I'm always wary of saying too much about what the reception's been because I always feel that that's not so much for us to judge, but as far as the process, I just thought it was fantastic and tick many of the boxes that's so important in this space. There's a great book by Claire Craig who used to be very senior in GO-Science. I think it's called How Does Government Listen to Scientists? And she talks about how you build such policy groups. One of her big references is Roger Pielke's, The Honest Broker, where Roger is splitting up scientists into four groups. One is the pure scientist that is just doing their science and not worried about policy at all. One is someone who will answer specific science questions when posed. So you say, ‘Which is better, this drug or that drug?’ And they'll provide science on that.

Then there are issue advocates that are pushing a particular agenda. But the final group that is so important is what Roger calls the honest broker. And Claire builds her description of these processes very much around that. And the honest broker is someone who's absolutely science aware, but they're trying to get the spectrum of scientific knowledge and understanding through... Of course, there'll be issue advocates. There'll be all the scientists present, but the honest broker approach is about making sure the diversity of opinions is heard. Now, I was very taken by that because I think as scientists, we always assume that we are the ones with the answer and we are the ones that need to supply the fix. But most of the time, 99.9% of policy questions, probably someone else has a better understanding of the answer. So I've always viewed policy work as it being what I think of as the supply chain of ideas.

There's a demand for policy solutions. There's a supply of scientists out there, and a lot of the work is trying to map that supply of policy ideas to the demand for policy solutions. And that requires exactly what we saw with the MHRA commission and the way Alastair and Henrietta set it up. I was privileged to be at the top of the technical working group, which had a lot of companies on it. So you might come into that given what these companies say publicly, thinking you're going to have to do a lot of managing of their bullishness or whatever else. Absolutely not. If you convene them well and you convene the experts from those companies, they were so insightful, so constructive, critical when necessary. And how quickly it was all pulled together, I don't know what you felt, Jennifer, but impressive how quickly it can be done.

It's like you've got projects that will have some perception about the world it looks like and try that and not finish within the period of time that the overall process finished. So I'm pleased with how it landed. I thought the reception was great, but of course there's a lot of hard work now to take these ideas and put them into practice.

Jennifer Dixon:

Yeah. I thought that it was very concise and crunchy. And from the inside, like you, Neil, I though that all the papers were very, very concise and really high quality. In terms of the outcomes of this report that listeners may not have got to yet, just to say that there was three main chapters, but one was about proportionate lifecycle regulation. So quite a bit there from the MHRA to respond to with respect to AI and particularly agentic AI, changing AI. And there, I think in that chapter, the big thing there for me was proportionate post-market surveillance. In other words, there needs to be some ongoing monitoring, particularly as some of these AI applications may change over time. Well, there was a third chapter which was about public trust and how to engage and regulatory clarity for the public. But chapter two was, I thought for me most interesting because it recognised that the AI thing is not just the thing to look at, but the context in which it's being used, the operating context. And that means the NHS, how far the NHS can actually use this and effectively deploy AI.

And there I thought MHRA and the commission stretched into system-wide responsibilities in a most healthy way because so often these technical conversations are not socio-technical, are they? They're technical. So there's obviously quite a lot of responsibility there for MHRA to pick up, but also for the NHS in making sure that the operating environment is safe enough to test at least some high risk AI and possibly really promising AI. So I thought that those were some unusual things.

Neil Lawrence:

I'm not a dedicated health professional, but I can generalise from other areas and imagine that the difficulties of those areas are even greater in health, that a lot of AI deployment is actually about cultural change. And a lot of it, I think the recommendations on the report require some devolution of authority to those who are applying the tools in practice. Because we can talk about post-market surveillance, we can talk about trying to get the numbers out of these tools and seeing when they're failing, but the capabilities of AI get much closer to the sort of human side of judgement. And one of the concerns I have is when you are unconsciously moving judgement from the professional, it could be the clinician, but it could be a teacher or it could be a nurse or it could be a social care worker, to the machine without noticing you're doing it.

And I think that that's something that we need to be watching for as we integrate. And that means that we need to be supporting those people who are assimilating these tools and building on the great training they have and trusting them to come back and say, look, when there's something wrong. But by the same token, we appreciate it's not just the NHS. We've got a number of institutions in these areas of education, social care, health care, under tremendous financial pressure, and they're not always being given the time to think. So my hope is, and indeed we wrote a paper about this, that the tools that we deploy will help free up some time from a nurse. We did one public dialogue in Liverpool where one of the responses was my wife who's an NHS nurse. I can't remember if it was 20% or 30% of her time was being spent on paperwork.

And now if we can deploy these tools in such a way that they're releasing some of that nurse's time, and then we can take the wisdom that nurse gets from looking at that deployment and understanding what was good about it, what was bad about it, and have that nurse share his or her knowledge with the wider nursing community. That's an approach from Cambridge we think about as the attention reinvestment cycle. And it's specifically designed to scale because the cultural change you're talking about is not just the NHS, it's universities in both teaching and research, it's our schools, it's our local authorities, it's our small and medium enterprises. And that's why government gets an F minus for demand because this is the work that needs to be done to ensure that these technologies settle in ways that are supportive of our citizens' aspirations rather than obstructive of their aspirations.

Jennifer Dixon:

You used this term, I think when we met, which was people using AI could result in, I think you called it ‘cognitive offloading’ and what some call ‘cognitive surrender’. In other words, you just sort of passively accept what the AI thing is doing without really applying enough judgement as opposed to having AI cognitively enhancing your skills, which is where we need to be. But that needs to be firstly identified and understood, and secondly, to be monitored.

And I think that brings us onto the final theme really, which is the main themes I think that came up in your interesting book, The Atomic Human, and no doubt elsewhere in your academic work, Neil, which is really about the limits to AI, the fact that it may surpass people in processing information, but that isn't the same as possessing for human understanding, judgement or purpose. So I think some of your reflections in, if you're able to in healthcare, where you thought the limits might be to AI applications and where human judgement can never really be replaced.

Neil Lawrence:

We always joke, I have to be a bit careful about this, but both when I lived in Sheffield and when I'm in Cambridge, I cycle with a lot of clinicians and I always think that they're the most unsympathetic people to the other cyclists who are suffering pain. They'll always just ride on, say, ‘Oh, something wrong with your heart, [inaudible 00:23:38] ride.’

Jennifer Dixon:

‘Yeah, you'll be all right. You're not going to die soon. Get on with it.’

Neil Lawrence:

Yeah, precisely. And you think about, well, why is this? Well, I don't know, but I speculate that it's partially because they're having to spend their weeks dealing with these decisions the whole time. And it must harden one a little. I mean, I can't quite imagine what it is to make serious decisions that. I mean, of course, they're a matter of life and death, but they're also a matter of the manner of life and death or a matter of understanding different forms of suffering, which as humans, whether or not they've experienced that suffering, they will potentially experience it one day. Particularly think of things like end of life care, I’ve been close to my brother when he was dying or my father. These decisions need to be discussed. I don't think there are right answers, but you do want someone with knowledge and understanding of the field who has a combined knowledge and understanding of what it means to be limited weak creatures that suffer disease, suffer death, despite what several leaders have said, I don't think there will ever be an end to disease. Disease will constantly work out a way of coming back.

We want those people that are supporting us in those decisions to have an understanding of humanity itself. I think anything else is extremely disturbing, and people are proposing in the Californian cults, the idea that machines just take over and decide everything for us. Indeed, there are leaders of some of these large tech companies who think that human intelligence is a sort of envelope for eventual machine intelligence, as if intelligence is a goal in itself rather than something that's a property of humans and communities. So I think from that perspective, you see that for certain types of consequential decisions, and there's no need to define them because what those consequential decisions should be should also be decided by humans. We will always want humans to be in the driving seat. And the big worry, as you mentioned, is cognitive surrender or cognitive offloading where they'll believe they're in the driving seat, but really they're just agreeing with the machine.

There's some sort of sense that we are optimising for some goal or some utilitarian idea, which of course is total rubbish. We're just a species modelling along with different approaches in different places and different ideas of what a good life is. And the way we model along is by working with other human beings to make those decisions about us. That seems to be fundamental, and I think it sits at the heart of the open society and our liberal democracies. The challenge is there's a lot of people who are steering these technologies who don't agree with that. And they don't surface that, they don't say that, but they operate in such a way to undermine these things that I think are extremely precious to our human society.

Jennifer Dixon:

There was a story I heard of a older woman who was phoned up by the hospital. Someone had a chat with her, ‘How are you doing? Are you okay? Have you taken your meds on time?’ The hospital organised this, and this older woman, it was her main social contact every day was to speak to this person, Iris. And she turned up to the hospital and the consultant said, ‘How are you doing?’ And she said, ‘I'd love to meet Iris. Please, will you...’ And he had to say, ‘I'm so sorry...’ I'm sure you've heard this story before. ‘I'm so sorry, but Iris is an automated machine.’ And then she said, ‘I don't really care about that. Just let Iris still call me every day.’ So I feel uncomfortable about that. I'm sure everybody does, but-

Neil Lawrence:

It sounds reflective to me of a challenge we face, I think particularly in the UK, but also the US. An increasing challenge in Italy where my mother-in-law lives, where we have issues with communities and how we're looking after people who are vulnerable in those communities. And maybe because it was more efficient for us to move away and take jobs elsewhere or that we all drive to work. A number of things have affected that, that perhaps we, and maybe we're over nostalgic about it. I don't know. I wasn't there. But I think the challenge is that if you constantly operate in a way that is, well, this is more efficient or this will reduce this or this will reduce that, that is sort of what you end up with.

But human contact, and one of the things that struck me in San Sebastiano al Vesuvio where my mother-in-law lives, is for a long time there was a man living in the village, everyone looked after him. I don't know where he lived, but everyone talked to him every day. And there was genuine care in the community with someone who was asking for mille lire for a coffee. That doesn't happen everywhere. I don't want to romanticise Italy, but I think that we would all feel more comfortable, or not all of us, but many of us would feel more comfortable if we thought that our communities were caring in that way. And I think that it's interesting because that way of caring can't just be reduced to numbers.

Now, someone else might say, ‘Well, it sounds like this lady was perfectly happy with Iris.’ And my mother will often say, ‘I would prefer to be cared for by a machine because I wouldn't want to be a burden on you.’ Things like that. So let's not prejudge what everyone is thinking, but let's not also set things up in a way that shuts off the possibility that humans continue to care about each other, particularly when we are vulnerable. Because some people may place the value of humanity in the numbers of billions of dollars beside their name on the Sunday Times’ Rich List. But other people place the human value on the amount of time that they're spending looking after other people in the community and building on those interactions.

And the latter job, there's a lot more availability for it. Not many of us can be multiple billionaires. So I think what's really sad is how we've undermined that. We've undermined public service and replaced it with it, it's all about the number of dollars against your name. How we recover that, I don't know, but I think the onus is on us to celebrate it and talk about it.

Jennifer Dixon:

So are you overall hopeful, Neil, as we close?

Neil Lawrence:

I often say that, I think it's Gramsci, the philosopher, that I realised I'd reverse engineered into what he suggests. Pessimism of the intellect, but optimism of the spirit. And what I mean by that is yes, if you look at any individual errors, you can argue yourself into how terrible things are. But if you look across humanity, if you look across how we've solved problems in the past, the solutions often come from directions you didn't expect. And pessimism is self-fulfilling. If we all try and work towards a better future, one will pop out of the woodwork somewhere and there'll be an environment and ecosystem ready to receive it. So I think we should be facing up to the difficulties and serious about them, but we need to ensure that we don't let those difficulties, and I think that's one challenge with the AI, we're all going to die narrative.

There are things individuals can do, and I firmly believe that, and we see that in practice, that people can make a difference with these technologies in their own environments, whether they're in public sector, whether they are a small business, large business, whatever. And if we firmly believe that and we work in the right direction, I think we could get in a very good place.

Jennifer Dixon:

Fantastic. Thank you so much, Neil.

So I hope you enjoyed that episode with Neil Lawrence. There were three things that I wanted to take away here. One was Neil was very strong about the cognitive bias of quite a lot of people in the tech sector and the need for different perspectives to get a more enriched discussion, not just about the risks, but also the benefits. One of his final points was that really with different perspectives and the talent we've got in the country, the future is in our hands. It's not something we need to passively accept the doom, but actually there's still lot to mould and shape to our benefit. And I think the third issue was in the health sector, the focus so far, which is on the more technical side, possibly the life sciences side of health, rather than some of the more complex socio-technical issues we face such as increasing access, better delivery of NHS services and so on, and indeed the sustainability of the social model of the NHS.

So I thought those were three things that were brought out to me as well as the positivity about the National Commission. So many, many thanks to Neil for that conversation. I hope you enjoyed it. Take a look at the links in the show notes where we put some of the things that we've referred to in the episode.

Next month we are going to be turning to mental health given the various national reviews on at the moment. And meantime, many, many thanks as ever to my colleagues, Sean and Leo at the Health Foundation, to Paddy and his team at Malt Productions. And it's goodbye from me until next time, I'm Jennifer Dixon.