Comment by Martin Roland, Director of the National Centre for Primary Care Research and Development
This commentary aims to do two things. First, and after a brief review of Christenson’s findings, I have provided some more recent references, mainly on the UK’s financial incentives. However, it should be noted that papers on pay for performance are being published every month in the US and UK, so this is a field where the available evidence is changing very rapidly. Second, I have contextualised some of the findings of the review in respect of current health policy in the UK.
Christenson’s review is exceptionally valuable in bringing together evidence from a wide range of sources on the impact of financial incentives on quality of care. Financial incentives are increasingly widely used as a method of improving quality of care, but those who look to this review for an answer on ‘what’s works?’ or ‘how to do it’ will be disappointed because the conclusions of the authors are very guarded on both these issues. There are a number of reasons for this caution.
The first is that the majority of evaluations of pay for performance (P4P) are observational studies where other things around were changing too. Often P4P was only one element in a whole series of interventions aimed at improving quality of care.
Second, there are very few experimental studies of P4P in which it is possible to disentangle the impact of individual elements of the incentives, eg the RCT of different levels of incentive quoted in Christenson’s review (Kouides et al 1998)
Third, the context of the P4P intervention may be critical – a point made repeatedly by Christenson and his colleagues. It is this dependency on context which makes it so difficult to draw generalisable conclusions.
Fourth, the impact of P4P on unincentivised aspects of quality is generally not reported, and unexpected negative consequences may therefore go unnoticed or unreported.
Nevertheless, there are important general conclusions which the authors are able to draw. These include:
- P4P is not a panacea. In fact many evaluations of P4P show small or negligible impacts of financial incentives. The context in which incentives are introduced may be very important, and detailed context needs to be reported in evaluations of P4P schemes
- Because of the limited evidence for impact of P4P as an isolated intervention, P4P should probably be regarded as one of a range of interventions that need to be considered as part of a package of quality improvement measures (eg including information technology, guidelines, education etc). This is consistent with reviews from the Cochrane collaboration which suggest there is ‘no magic bullet’ for quality improvement.
- The impact of P4P on unincentivised aspects of quality is generally not reported, and this should be part of the routine monitoring of P4P schemes in order to guard against unexpected or unintended consequences of financial incentives.
- Financial incentives to change other behaviours (e.g. reduce hospital utilisation by changing patterns of referral) could have a negative impact on quality. Although there is little evidence that this has occurred, this is largely because the data are not usually collected. Quality needs to be monitored in incentive schemes where quality could be adversely affected.
Recent publications
Papers were included in Christenson’s review up to summer 2006, and this section includes some additional references mainly from the UK published since then. This is a field which is changing very rapidly, and regular updating of any review of this area will be necessary to keep up to date, especially as P4P schemes are themselves evolving quite rapidly in terms of changing size of financial incentives, inclusion of more quality indicators, and the adoption of more sophisticated approaches such as risk adjustment (Rosenthal et al 2007).
Impact on health outcomes
A number of UK studies have looked for relationships between quality of care as evidenced by QOF scores and emergency admissions. No association was found for emergency admissions for a range of conditions included in the QOF (Downing et al 2007) admissions for coronary heart disease by Bottle et al (2007). However, Shohet found that admissions for epilepsy were less common in practices with high scores for epilepsy (Shohet et al 2007). The inconsistency of these relationships is perhaps not surprising, partly because not all indicators would be expected to have a direct or rapid impact on admissions (the epilepsy ones would), but also because of the strong influence of other factors such as socio-economic deprivation on emergency admissions (Giuffrida et al 1999, Bottle et al 2007).
Impact on health inequalities
Published studies continue to document improvements in quality of primary care in the UK (e.g. Khunti et al 2007) though, as with the quoted studies in Christenson’s review, all are observational, only some contain data from before the introduction of financial incentives, and none can adequately address the relative importance of the QOF incentives compared to other quality improvement activities (e.g. national guidelines etc).
A number of recent studies have also investigated the impact of the Quality and Outcomes Framework on health inequalities, reflecting concerns also in the US literature that socio-economically deprived and/or sicker populations may be selectively disadvantaged by P4P schemes (Casalino et al 2007). In the UK, the differences in QOF scores between practices in affluent and deprived areas is statistically significant, but relatively small compared to many other types of health disparity with a difference of only 30 QOF points between practices in the least and most deprived quintiles, though with larger relative differences for some indicators (Wright et al 2006, Ashworth et al 2007). However, these studies, like the one quoted in Christenson’s review (Doran et al 2006) use area-based data to characterise practices, and larger differences might be found if individual level data were available.
Impact on unincentivised aspects of care
Two recent studies have also been published which start to address the concern that financial incentives may need to neglect of non-incentivised conditions (McGlynn 2007). This concern does not appear to have been realised in two recently published studies, one from the US (Ganz et al 2007) and one from the UK (Steel et al 2007). However this type of study inevitably compares quality of care for aspects of care that can readily be measured. Much of the criticism of the Quality and Outcomes Framework in the UK relates to the potential loss of the caring aspects of the general practitioner’s work (Mangin and Toop 2007). The literature continues to provide a wide range of largely anecdotal views on the QOF, ranging from those who believe that good care is being appropriately rewarded and that patients will benefit, to those who believe that the ethos and values of general practice are being fundamentally undermined to the detriment of patient care.
Implications for the UK
What are the implications of this important review for the UK?
Size of financial incentives in primary care
There is no clear evidence on how big incentives should be, but my view, which is supported by extensive if only anecdotal evidence, is that the size of the financial incentives in the Quality and Outcomes Framework was too large. This is partly because care was already improving rapidly in certain areas (and so didn’t need additional payment), and partly because the Quality and Outcomes Framework may have had an inappropriately distorting effect on other aspects of care. Well known critics of the QOF blame the financial incentives for changes in the ethos and culture of general practice. Some of these criticisms are too superficial, not least because of the many other changes occurring simultaneously in medicine – e.g. a young generation of doctors who expect working lives to have clear boundaries in terms of both time and personal commitment. Nevertheless, my view is that the QOF should, if anything, gradually reduce as a proportion of overall practice income.
Pay for performance or pay for reporting?
Though not widely described as such, the Quality and Outcomes Framework is a ‘pay for reporting’ as well as a ‘pay for performance’ scheme. The results, down to individual indicator level are available for every practice in the country on the internet (www.ic.nhs.uk/qof ). Some behaviour by GPs does not appear to follow a rational economic model – e.g. the enthusiasm with which practices chase every last point, and practices are clearly also interested in their reputations – evidenced by the number of advertisements for GP posts that mention the practice’s high QOF score. This is consistent with Frølich’s model of behavioural change among clinicians in which there is a close link between financial and behavioural incentives (Frølich et al 2007).
There are numerous opportunities to exploit this. In particular, the UK is now moving towards more systematic administration of patient questionnaires as part of the contract in a way that will make it possible to make valid comparisons of patients’ assessments between practices. In my view there is insufficient evidence to recommend payment based on patient questionnaire scores (tried in only a few schemes in the US, and not yet widely reported on). However, this does represent an opportunity to investigate how the planned publication of practice scores might in itself be used as a lever for change. There is an important opportunity here for evaluation and, possibly, experimentation.
Incentive schemes in hospitals – who should be incentivised?
In the largest experiment with P4P in the UK (the GP contract), it is clear who is being incentivised. However, it is much less clear in some other innovations, especially in specialist care (e.g., the Premier Hospital interventions being tested in the North West of England).
As Christenson found, it is often not clear how incentives might be expected to operate in hospitals – is it the managers or clinicians who are being incentivised? Individuals or teams? Hospitals are much more complex organisations that general practices, and those trying out such schemes need to think very clearly about how incentives might reasonably be expected to operate. Who will notice that something they do is being incentivised? Will they expect a personal reward, or could financial incentives to a hospital still motivate individuals in some way? Christenson points out that, in some schemes, the person delivering care did not even know that incentives were operating. It is not sufficient to reward an NHS trust for performance without thinking through just how that incentive could reasonably be expected to change the behaviour of individuals.
Conclusion
For all its detailed faults, the Quality and Outcomes Framework is an interesting example of how it has been possible to introduce an infinitely flexible framework which has brought quality improvement high on the agenda for many primary care doctors in the UK. It was never intended or expected that the indicators would stay static. They can should develop and improve as it becomes possible to include new areas, or new evidence becomes available.
References
Ashworth M, Seed P, Armstrong A, Durbaba S, Jones R. The relationship between social deprivation and the quality of primary care: a national survey using indicators from the UK Quality and Outcomes Framework. British Journal of General Practice 2007; 57: 441-118
Bottle A, Gnani S, Saxena S, Aylin P, Mainous A, Majeed A. Association between quality of primary care and hospitalization for coronary heart disease in England: national cross sectional study. Journal of General and Internal Medicine 2007 Oct 9th e-print ahead of publication: doi: 10.1111/j.1475-6773.2007.00742.x
Casalino L, Elster A. Will pay for performance and quality reporting affect health care disparaities? Health Affairs 2007; 26: w405-w414. doi: 10.1377/hlthaff.26.3.w405
Downing A, Rudge G, Cheng Y, Tu Y, Keen J, Gilthorpe M. Do the government’s new Quality and Outcomes Framework (QOF) scores adequately measure primary care performance? A cross sectional survey of routine healthcare data. BMC Health Services Research 2007; 7: 166
Frølich A, Talavera J, Broadhead P, Adams Dudley R. A behavioural model of clinician responses to incentives to improve quality. Health Policy 2007; 80: 179-193.
Ganz D, Wenger N, Roth C et al. The effect of a quality improvement initiative on the quality of other aspects of health care. The law of unintended consequences? Medical Care 2007; 45: 8-18.
Giuffrida A, Gravelle H, Roland M. Measuring quality with routine data: avoiding confusion between performance indicators and health outcomes. British Medical Journal 1999; 319: 94-98
Gulliford M, Ashworth M, Robotham D, Mohiddin A. Achievement of metabolic targets for diabetes by English primary care practices under a new system of incentives. Diabetes Medicine 2007; 24: 505-511
Khunti K, Gadsby R, Millett C, Majeed A, Davies M Quality of diabetes care in the UK: comparison of published quality-of-care reports with results of the Quality and Outcomes Framework for Diabetes. Diabetic Medicine 2007; 24: 1436-41
Kouides R, Bennet N, Lewis B, Cappuccio J, Barker W, LaForce F. Performance based physician reimbursement and influenza immunisation rates in the elderly. American Journal of Preventive Medicine 1998; 14: 88-95.
Mangin D, Toop L. The Quality and Outcomes Framework: what have you done to yourselves? British Journal of General Practice 2007; 57: 435-7
McGlynn E. Intended and unintended consequences. What should we really worry about? Medical Care 2007; 45: 3-5
Millett C, Gray J, Saxena S, Netuveli G, Majeed A. Impact of a pay for performance incentive on support for smoking cessation and on smoking prevalence among people with diabetes. Canadian Medical Association Journal 2007; 176: 1705-1710.
Rosenthal M, Landon B, Howitt K, Song H, Epstein A. Climbing up the pay for performance learning curve: Where are the early adopters now? Health Affairs 2007; 26: 1674-1682
Shohet C, Yelloly J, Bingham P, Lyratzopoulos G. The association between the quality of epilepsy management in primary care, general practice population deprivation status and epilepsy related emergency hospitalisations. Seizure 2007; 16: 351-355.
Sigfrid LA, Turner C, Crook D, Ray S. Using the UK primary care Quality and Outcomes Framework to audit health care equity: preliminary data on diabetes management. Journal of Public Health 2006; 28: 221-225.
Steel N, Maisey S, Clark A, Fleetcroft R, Howe A. Quality of clinical primary care and targeted incentive payments: an observational study. British Journal of General Practice 2007; 57: 449-54
Tahrani A, McCarthy M, Godson J et al. Diabetes care and the new GMS contract: the evidence for a whole county. British Journal of General Practice 2007; 57: 483-485
Wright J, Martin D, Cockings S, Polack C. Overall Quality of Outcomes Framework scores lower in deprived areas. British Journal of General Practice 2005; 56: 277-279.
Download the report 'Financial incentives, healthcare providers and quality improvements' >