1. Why have the Health Foundation and NHS England and NHS Improvement set up the Improvement Analytics Unit?
Over the last decade, a number of national transformation programmes have been established with the aim of improving quality and efficiency in health care, often through the provision of more integrated care. However, the best analytical approaches to evaluate the impact of complex changes such as these are demanding to apply and the required capability is not always available in the NHS.
The Improvement Analytics Unit aims to fill this gap by:
- providing robust evaluation of complex change to support the development of strong and effective health care systems
- identifying whether outcomes for patients have changed following introduction of a new initiative and to clarify, so far as possible, why
- helping to spread the use of data analytics in the NHS for the purposes of quality improvement, and
- strengthening the robustness of the body of evidence available to inform policy development.
The unit supports delivery of NHS England’s commitment to evaluating the impact of complex local initiatives to improve care and the operational efficiency of services in health and social care such as those being delivered as part of the sustainability and transformation partnerships (STPs) and Integrated Care Systems (ICSs) or as part of major national programmes.
2. Why have the Health Foundation and NHS England and NHS Improvement embarked on this partnership?
NHS England and the Health Foundation form a strong and complementary partnership. The Health Foundation provides technical and methodological skills as well as knowledge and a reputation for rigour. NHS England provides a mechanism for delivering change as well as access to data and networks within the NHS.
The Health Foundation is contributing to the Improvement Analytics Unit as part of its charitable mission to bring about better health and health care. One element of this is building and understanding the evidence base for what interventions and policies lead to improved health outcomes. The Health Foundation is also committed to improving analytic capability within the NHS.
3. How will you publish the evaluation reports produced by the Improvement Analytics Unit?
The Improvement Analytics Unit is committed to transparency regarding the analysis it undertakes. Evaluation reports will be published by the Health Foundation following presentation to the local initiative and national NHS England team and after a suitable period of reflection.
In addition, the Health Foundation will publish national and thematic reports which summarise the learning from the Improvement Analytics Unit.
4. What will happen to those initiatives where the analysis generated by the unit doesn’t show a positive impact?
The purpose of the evaluation reports is to generate learning and improvement at a local and national level. If the results demonstrated at a local level are not in line with expectations, those involved will understandably want to use the evidence alongside other local intelligence to assess whether any appropriate changes should be made in order to improve patient outcomes. However, some of the impact of the local initiatives may not be fully measured by the data used by the unit.
Combining analysis from the Improvement Analytics Unit with local intelligence helps to guide the development of improvement on the ground. We know such improvement is complex and takes time. However along the way, the analysis can be used by those delivering change to bring valuable insights and to identify whether they need to alter their approach (‘course correct’) to improve outcomes.
5. What metrics do you use to evaluate the local initiatives? What factors are you exploring?
The metrics used vary depending on the initiative, the outcomes that were expected and the evaluation question being analysed. However, in our studies we routinely evaluate the impact an initiative has had on emergency admissions, inpatient bed-days, potentially avoidable admissions, among other measures of secondary care use for a specific patient group.
The Improvement Analytics Unit uses a standard set of metric definitions across its studies to ensure consistency, facilitate comparison between studies and enable more rapid production of the analysis.
6. What data does the Improvement Analytics Unit analyse?
The Improvement Analytics Unit uses patient data that has been ‘pseudonymised’ (anonymised according to the Information Commissioner’s Office code of practice). The outcomes we evaluate are primarily derived from national secondary care datasets used by NHS England in its statutory and commissioning functions. Where necessary, we link this data with information provided by a local organisation to help identify those patients who received the intervention. In future, we plan to include outcome measures that are derived outside secondary hospital care. We also intend to include insight and qualitative information from those involved in implementing the local initiative to help understand what may be driving the results.
The Improvement Analytics Unit employs rigorous standards of information governance and data security. Find out more about how we use patient data
7. How does the analysis provided by the Improvement Analytics Unit differ from the existing dashboard and other local evaluations?
The Improvement Analytics Unit is an important part of NHS England’s approach to rapid cycle evaluation. It is part of a spectrum of complimentary evaluation tools available to assess improvement and support learning.
While there are different types of evaluation available to the NHS, robustly evaluating the impact of change often requires the use of a counterfactual control group. Our counterfactual analysis relies on making a comparison between a group of patients who have experienced an intervention with a carefully matched control group. This helps us to determine whether the patients experiencing the intervention have had different outcomes to similar patients who have not.
Our approach overcomes the problems inherent in simple ‘before and after’ studies, which can lead to misleading results.
No. The agreement between the two organisations is based on an equal partnership of sharing resources and there is no financial payment to the Health Foundation or NHS England. Staffing costs are funded jointly by both organisations for the duration of the partnership.
9. How do you identify local improvement initiatives to work with? Who is eligible?
The unit focuses on local initiatives that can demonstrate a distinct change in service delivery, usually cross-organisational, for the purpose of providing better quality or more efficient care for patients. See our previous publications for information on initiatives we have evaluated to date.
Sites are chosen based on criteria which enables the Improvement Analytics Unit to maximise the impact and learning for local initiatives and national policy makers. If you would be interested in working with the Improvement Analytics Unit, or have any questions, please email Arne Wolters.