A comparison of synthetic control approaches for the evaluation of policy interventions using observational data Evaluating the impact of redesigning urgent and emergency care in Northumberland

Abstract

Objective

To compare the original synthetic control (OSC) method with alternative approaches (Generalised (GSC), Micro (MSC), and Bayesian (BSC) synthetic control methods) and re-evaluate the impact of a significant restructuring of urgent and emergency care in North-East England, which included the opening of the UK's first purpose-built specialist emergency care hospital.

Data sources

Simulations and data from Secondary Uses Service data, a single comprehensive repository for patient-level health care data in England.

Study design

Hospital use of individuals exposed and unexposed to the restructuring is compared. We estimate the impact using OSC, MSC, BSC and GSC applied at the general practice level. We contrast the estimation methods’ performance in a Monte Carlo simulation study.

Data collection/extraction methods

Hospital activity data from Secondary Uses Service for patients aged over 18 years registered at a general practice in England from April 2011 to March 2019.

Principal findings

None of the methods dominated across all simulation scenarios. GSC was generally preferred. In contrast to an earlier evaluation which used OSC, GSC reported a smaller impact of the opening of the hospital on Accident and Emergency (A&E) department (also known as emergency department or casualty) visits and no evidence for any impact on the proportion of A&E patients seen within 4 hours.

Conclusions

The simulation study highlights cases where the considered methods may lead to biased estimates in health policy evaluations. GSC was found to be the most reliable method of those considered. Considering more disaggregated data over a longer time span and applying GSC indicates that the specialist emergency care hospitals in Northumbria had less impact on A&E visits and waiting times than suggested by the original evaluation which applied OSC to more aggregated data.

This article is protected by copyright. All rights reserved.