Criteria for evaluating programme theory diagrams in quality improvement initiatives A structured method for appraisal

Funded by

The Health Foundation Logo

9 April 2018

Published journal: International Journal for Quality in Health Care

Abstract

Background

Despite criticisms that many quality improvement (QI) initiatives fail due to incomplete programme theory, there is no defined way to evaluate how programme theory has been articulated. The objective of this research was to develop, and assess the usability and reliability of scoring criteria to evaluate programme theory diagrams.

Methods

Criteria development was informed by published literature and QI experts. Inter-rater reliability was tested between two evaluators. About 63 programme theory diagrams (42 driver diagrams and 21 action–effect diagrams) were reviewed to establish whether the criteria could support comparative analysis of different approaches to constructing diagrams.

Results

Components of the scoring criteria include: assessment of overall aim, logical overview, clarity of components, cause–effect relationships, evidence and measurement. Independent reviewers had 78% inter-rater reliability. Scoring enabled direct comparison of different approaches to developing programme theory; action–effect diagrams were found to have had a statistically significant but moderate improvement in programme theory quality over driver diagrams; no significant differences were observed based on the setting in which driver diagrams were developed.

Conclusions

The scoring criteria summarise the necessary components of programme theory that are thought to contribute to successful QI projects. The viability of the scoring criteria for practical application was demonstrated. Future uses include assessment of individual programme theory diagrams and comparison of different approaches (e.g. methodological, teaching or other QI support) to produce programme theory. The criteria can be used as a tool to guide the production of better programme theory diagrams, and also highlights where additional support for QI teams could be needed.

Citation

Laurel Issen, Thomas Woodcock, Christopher McNicholas, Laura Lennox, Julie E Reed; Criteria for evaluating programme theory diagrams in quality improvement initiatives: a structured method for appraisal, International Journal for Quality in Health Care, Volume 30, Issue 7, 1 August 2018, Pages 508–513, https://doi.org/10.1093/intqhc/mzy063

You might also like...

Newsletter feature

Bringing an improvement mindset to COVID-19

Dominique Allwood is Assistant Director of Improvement at the Health Foundation, and was recently...

Blog

Digital and data-driven innovation: supporting it to thrive

Arne Wolters explores how digital and data-driven innovation can be used in ways that maximise the...

Blog

How the NHS can use anchor strategies to build a healthy and sustainable post-COVID-19 recovery

We look at how the NHS as an anchor institution has an essential role to play in recovering from the...

Kjell-bubble-diagramArtboard 101 copy

Get social

Over 700,000 patients served by GPs at high risk of COVID-19 could be left without access to face to face GP appoin… https://t.co/wZJJPCuRCn

Follow us on Twitter
Kjell-bubble-diagramArtboard 101

Work with us

We look for talented and passionate individuals as everyone at the Health Foundation has an important role to play.

View current vacancies
Artboard 101 copy 2

The Q community

Q is an initiative connecting people with improvement expertise across the UK.

Find out more