‘Rules of thumb’ when assessing the quality of analysis for decision making

Imagine you are in an unfamiliar city. It’s lunchtime and you are hungry.

Your time and money are – as ever – limited. The options before you are not. Any number of cafés, sandwich shops, restaurants and street stalls are within easy reach. 

How do you decide what to do? What evidence do you use to navigate this choice?

The ideal option, of course, is personal recommendation: expert advice from a trusted friend who knows the local area and your preferences. 

But without this, what would you look for? The restaurant with the best online reviews? The cafe advertised in the tourist brochure? The one with the comfortable armchairs and familiar menu? What are your ‘rules of thumb’ for deciding?

The decisions faced by NHS leaders are less appetising. Strategic commissioners decide how to reduce demand for unplanned acute care. Hospital leaders examine ways of making significant savings. Programme designers choose whether to invest in promising-looking digital innovations, unsure of what might work. 

The choices are different, but the limitations on time and money are the same. And so is the need for reliable evidence and advice. 

There is an abundance of analyses out there and more are published every month. Although many are well thought through and robust, some sources are far less reliable than others. Some types of analysis might give misleading results. Decisions based on inappropriate or poor-quality analysis could waste precious resources or even cause harm.

Are there ways of quickly assessing analytical quality?

NHS decision makers do not typically have analytical training, and local analytical advice can be (too) hard to come by. So how can decision makers arrive at a first, and rough, judgement about likely analytical quality? Are there any ‘rules of thumb’ that might help? 

The Health Foundation commissioned the Strategy Unit, a specialist NHS analytical team, to run a short, exploratory project to investigate this topic. The work drew on interviews with 11 highly experienced analysts from the health and care sector, think tanks and universities. 

The questions were simple: 

• How do you – as an expert analyst – begin to assess quantitative analysis when you first come across it? What tests do you apply to spot work of higher or lower quality?
• Of the tests that you perform, could any be used by someone without specific analytical expertise? 

The interviews illustrated how skilled people with deep, technical knowledge begin to test the quality of an analysis. Reassuringly, this work found many common themes and several of the experts’ ‘tests’ could be applied by decision makers without analytical training. 

‘Rules of thumb’ for non-analysts to assess the quality of quantitative analysis

The resulting ‘rules of thumb’ have been set out in the Strategy Unit’s report as a series of questions and prompts for decision makers around four themes.

The first is: what are my starting beliefs?

Recognising that none of us begin from a neutral position – and that we are all prone to confirmation bias – encourages decision makers to ask themselves whether they want certain things to be true. If so, they might be more likely to trust the analysis that supports their prior beliefs – the equivalent of choosing the comfy cafe with the familiar menu. 

Another prompt is to use their own prior knowledge and experience: if an analysis claims that an intervention reduced readmissions by 30%, is that believable? Have any other known interventions reduced readmissions by so much?

Decision makers can also check how open and transparent the analysis is. Are the methods and data sources detailed? Are assumptions clear and plausible? Do the authors describe limitations? Have they shared any underlying models for technical examination?

Furthermore, who has undertaken and funded the research, and do the authors have a vested interest in the results?

The full set of these ‘rules of thumb’ is presented on a single page in the report, providing a simple, accessible tool for decision makers.

Ultimately, no guidance can replace years of analytical experience. By far the best rule of thumb for any decision maker is to seek trustworthy analytical advice. This could be through the integrated care board’s (ICB’s) chief analytical officer or its local intelligence function, for example. 

But these questions and prompts could prime decision makers to spot especially poor, but perhaps attractively laid out, analytical work. These may also provide a common language, helping decision makers and analysts to discuss what high-quality analysis means in their context.

The NHS faces some unenviable choices. Wisdom, judgement and high-quality analysis are needed. Our hope is that these ‘rules of thumb’ make some contribution to the process.

Fraser Battye is Head of Policy at the Strategy Unit.