Making rapid evaluation of AI in health care a reality

One of the key challenges facing the NHS’s digital transformation is the speed and cost of evaluating technology, and specifically AI. The Health Foundation has been exploring this issue and I recently joined a panel session chaired by our Chief Executive, Jennifer Dixon, as part of our Strategy 2030 launch. 

We’re all aware that our health service is struggling with workforce pressures, stubbornly low productivity and challenges in access to care. Technology, including AI, has significant potential to help. But fast and cheap ways of generating robust, real-world evidence are lacking. That means timely, evidence-informed decisions on which AI tools are worth adopting are much harder to make than they should be.

Our panel brought together some experts we’ve been talking to in recent months: Mary Dixon-Woods, Director of The Healthcare Improvement Studies (THIS) Institute; Will Monaghan, Group Chief Digital Information Officer of University Hospitals of Leicester and University Hospitals of Northamptonshire; and Haris Shuaib, Founder and CEO of Newtons Tree.

The challenge

Assessing the performance and effectiveness of AI tools is currently often based on testing the algorithmic model in simulated or computational environments, rather than directly in the real world. This, however, is not the full picture.  

Early-phase, real-world evaluation is essential, as Mary Dixon-Woods explained, because the effects of technology depend on many features of context and performance. These range from characteristics of patients and populations, local workflows, staff behaviour and acceptance through to organisational readiness, infrastructure and capabilities, as well as many other factors. 

But current approaches to real-world evaluation cannot keep pace with the rate that new AI tools are developing. Haris Shuaib illustrated the point by describing a trial of AI reading CT head scans to see if it could improve turnaround time for patients in emergency departments. The trial cost about £1.2m, took 4 years and the results were inconclusive. We do not have the time or resources to evaluate AI tools in this way. 

The opportunity

Clearly, we need to find new ways of conducting real-world evaluation. Will Monaghan described how his trust is doing just that, trialling 20–30 new AI tools at any given time over a 3–6 month period. With 90% of these ultimately deemed unsuitable for adoption, this validates the need to make evaluative practices business as usual, to find the 10% that are worthwhile.

We need to be able to make ‘good enough’ decisions quickly about whether a technology is promising, based on robust, rigorous and credible early phase evaluation, and learn what needs to happen to implement it. At the Health Foundation, we’ve been exploring ideas about how to build this capability across the health care system with NHS England/DHSC, NIHR, MHRA, CQC and others. Through these conversations, we’ve identified four elements that could help:

1. Establish a number of test-bed sites with the capability to set up and run rapid evaluations of priority or high-risk AI solutions quickly. Where possible, that will require front-loading what can be done once for every evaluation; for example, information governance approvals, approaches to IT integrations and contractual arrangements. Rapid evaluations at these sites would aim to be delivered in 3 months, assessing whether a solution is usable, acceptable to staff, clinicians and patients, and giving early signs of cost-effectiveness. During the evaluation period, the sites can also develop and test approaches for ongoing post-market surveillance or further evaluation.
2. Use a digital platform that supports online learning communities to allow test-bed sites to learn from each other, support rapid feedback cycles and share experiences of what works well and what doesn’t when it comes to using the AI solution. As well as gathering data for early-phase evaluation in test beds, the platform will engage communities at scale to support wider implementation where a solution shows sufficient promise. 
3. Develop a standard set of methods for rapid evaluation of AI that are trusted and seen as credible by clinicians, staff and patients, as well as system decision makers.
4. Build capability in providers and care settings to conduct their own evaluations using these standardised methods with tools, templates and reusable evaluation protocols, some of which can be developed in the test beds.

While a 3-month early-phase evaluation will not replace large-scale trials where needed, it could act to filter whether or not tools are worth the time and expense of a full evaluation, optimising interventions and identifying the conditions necessary for success. 

Piloting the rapid evaluation approach

Testing this proposed approach to rapid evaluation is important to understand whether and how it could work in practice. We are collaborating with THIS Institute and NHS England on a pilot rapid evaluation of AI-enabled automated retinal imaging analysis systems (ARIAS) in NHS England’s Diabetic Eye Screening programme.

Alongside this, we are continuing to explore how to speed up rapid evaluation of AI with stakeholders, and will continue to share our findings as we develop our work in this area.