Evaluating complex change: The role of formative evaluation

Traditionally, evaluation’s role in complex change has been primarily to inform judgement. Evaluation has often been employed towards the end of a programme to help determine whether the programme succeeded in achieving its goals and to inform decisions about whether the programme should be stopped, continued, modified or adopted more widely. This is known as ‘summative’ evaluation.

There is another approach, however, in which evaluation informs improvement as an integral part of implementation. This ‘formative’ evaluation provides the opportunity to act on findings while the programme is still developing. This method is particularly attractive when the changes being made are complex and the evidence base is evolving, with the potential to produce outcomes that were not anticipated at the outset.

By providing local teams with robust information on the impact of their initiatives to improve care, often at an early stage of implementation, the Improvement Analytics Unit aims to inform ongoing improvement efforts while they are still underway.

Early stage evaluation is challenging

Evaluating the early stages of an intervention brings challenges, as the Improvement Analytics Unit found recently for the Sutton Homes of Care vanguard - it often means that fewer people have experienced the change being evaluated. In Sutton, this meant that we had a relatively small sample size, analysing the outcomes for 297 residents who entered a care home during the study period. During the earlier stages of an intervention, initiatives often evolve, adapt and improve, with effects changing over time.

The enhanced support in Sutton was commissioned by the Sutton Homes of Care vanguard for all care homes within the CCG area. The support was organised around three priority areas (or ‘pillars’). These were:

• a series of measures aimed at improving integrated care, including the hospital transfer pathway (the ‘Red Bag’ scheme) and weekly health and wellbeing reviews
• ongoing education and development for care home staff, for example through e-learning on continence care, dementia care and person-centred thinking
• promotion of quality assurance and safety, for example through a joint intelligence group to share information among local health and care partners and a dashboard to benchmark care homes performance.

Initiatives under these pillars were implemented in different types of care home (nursing, residential, mental health and learning disabilities) at different times. We also conducted our evaluation using data from relatively early in the life of the vanguard. These factors may explain why we couldn’t be conclusive about whether the enhanced support led to lower or higher levels of hospital use for residents who moved into care homes in the Sutton CCG area between January 2016 and April 2017. Our findings do not necessarily mean that there was no effect, but rather that we were unable to detect conclusive evidence of one. We could not rule out the possibility that the enhanced support was unable to significantly reduce emergency admissions among the new Sutton residents in the study.

Gaining insight from multiple sources

Alongside another local evaluation, we uncovered insights which could provide prompts for further investigation and help highlight potential areas of improvement for Sutton CCG as they move forward with the implementation of changes introduced by Sutton Homes of Care.

There were some indications that the Sutton residents in our study might have experienced more emergency admissions than the control group, including more admissions that could be considered potentially avoidable. These findings were strongest when we analysed data for nursing home residents only (rather than analysing data for all nursing and residential home residents). There were more emergency admissions that were potentially avoidable among Sutton residents than among the control group, which might indicate that the enhanced support had an unintended effect for this group. It would then be possible to examine whether the difference observed in nursing homes suggests a need for a greater focus in this area.

Meanwhile, we also found that, across both types of care home, there were some indications of positive effects for residents that moved into a Sutton care home in the last eight months of the study. These residents went to A&E less often than controls, although this did not seem to result in fewer admissions. This could be a positive finding because conveyance to A&E can be a stressful and confusing experience for care home residents and their carers.

Despite the challenge of having largely inconclusive results, it is still possible to gain insight from this type of evaluation and to help increase the effectiveness of complex changes such as those implemented in Sutton. This kind of formative impact evaluation can inspire us to look again, ask questions and explore our understanding, potentially identifying the need for corrective action as part of a continual process of learning and improvement.

Martin Caunt is Programme Director for the Improvement Analytics Unit and Senior Analytical Lead at NHS England