Measuring performance in the NHS: what really matters?

Encouraging the efficient use of NHS resources is vitally important for patients and doctors. Inefficiency means that money is wasted, but more importantly it means that the opportunity is lost to save lives, avert pain, and provide care to those in need. In a system as large and as complex as the NHS, however, actively promoting efficiency through the design and use of incentives is a difficult task.

Two recent attempts to grapple with efficiency were the implementation of the internal market (which relied on supply side competition within the NHS to drive down costs) and the combined monitoring and measurement of performance targets known as the purchaser efficiency index (which was a demand side management tool aimed at encouraging health authorities to provide an incentive for purchasing efficiently provided services).¹ Although the new NHS white paper has retained the split between purchasers and providers, the competitive thrust of the internal market has been abandoned.² And, although the purchaser efficiency index has also been jettisoned, the annual requirement for health authorities to make efficiency savings of around 3% a year has been retained.

The NHS white paper strongly criticised these mechanisms, arguing that the internal market increased administrative costs, fragmented decision making, distorted incentives, and promoted inequality.² The efficiency index—a ratio of changes in activity to changes in financial inputs—was criticised for being narrowly focused and providing perverse incentives.¹,³,⁴ What should replace these performance measures, and are the proposed solutions radical departures or simply tweaks at the periphery?

Summary points

• The national reference costs database and indices provide a first step in describing and understanding variations in costs between NHS trusts

• There are serious drawbacks to the quality and coverage of the data used to compile the unit costs

• Much of the variation in overall costs identified by the composite reference cost index is likely to be statistically insignificant with real differences occurring only between the trusts that have very high costs and those that have very low costs

• It is difficult to assess the scale of any perverse incentives introduced by these new measures without evidence that the reference costs are actually being used by health authorities and trusts

• There is no evidence that reference costs encourage more efficient production of health care than either the purchaser efficiency index or the internal market

Part of the government's response to the potential gaps in performance management has been to introduce a national assessment framework for performance.⁵ This framework identified six broad areas that could be monitored and compared across the NHS.

From November 1998, the performance framework included a national schedule and an index of reference costs which detail the unit costs for more than 500 surgical procedures across 249 NHS trusts. (These data are available from the Department of Health's website at www.doh.gov.uk/nhsexec/refcosts.htm.) The second edition of these data was issued in January 2000. The schedule and index have identified enormous variations in costs. For example, in 1997-8, the cost of a cataract removal with lens implant varied fivefold across the country, from £337 to £1659.

Although government ministers have made it clear that they do not welcome criticism from trusts either of the method used or the data underlying the unit costs, the performance of the NHS is too important to be left solely to the spin doctors.⁶ Nevertheless, there are questions to be asked of any performance measure⁷: are the measures and the data on which they are based accurate, and is the methodology sound? How are the performance measures to be interpreted, and how will the behaviour of the whole health system change in response to the performance information?

Methods

Information on the costs of performing various procedures in NHS trusts, a summary composite measure of trusts' overall costs, and the methodology used to compile these data were critically reviewed. All the data reported here, including associated reports, are available from the Department of Health's website (address given above).

The national schedule and index of reference costs

The 1998 national schedule of reference costs is a database of unit costs for 1997-8 drawn from 249 NHS trusts in England. The second edition covers slightly fewer trusts (due to mergers) but includes a greater proportion of trusts' costs and data on more services. Costs are based on healthcare resource groups (box). The databases allow unit costs to be compared on various levels: by healthcare resource group, specialty, hospital, or hospital type, and for inpatients, day cases, etc. All data reported here (including associated reports) are available from the Department of Health's website.

Healthcare resource groups

Healthcare resource groups were developed in conjunction with the royal colleges and are the British version of diagnosis related groups, which were developed in the United States. Like diagnosis related groups, the 573 healthcare resource groups are based on diagnostic codes from the international classification of diseases (as well as, sometimes, the age of the patient).

The healthcare resource groups have been organised according to the amount of healthcare resources a typical patient would “consume” in hospital. For example, category H17 is “soft tissue for other bone procedures for patients aged under 70 years without complications and comorbidities.” All patients given this classification on discharge would be expected to remain in hospital for a similar length of time and to use a similar amount of healthcare resources.

However, H17 includes both simple lipomas and some soft tissue sarcomas that are costly to treat and for which patients are seen at supraregional centres. Even with additional editing of the cases for which patients have long stays in hospital, this attempt to standardise the “inputs” (the patients) to reduce cost variations arising from different case mixes is not perfect.

In addition to the databases, there is also a unit cost index. The index is a simple number that is calculated by dividing a hospital's actual total costs (its own unit costs for healthcare resource groups × the number of healthcare resource groups) by its expected costs (the national average unit costs of healthcare resource groups × the number of healthcare resource groups). The index is adjusted to account for unavoidable differences in costs caused by trusts' different locations. Hospitals are then ranked according to their index value. Comparisons between the old purchaser efficiency index and the new reference cost index are shown in the box.

Comparing the new reference cost index with the old purchaser efficiency index

Reference cost index

Index for hospital j={[Σi(c_ij . A_ij)/(C_i . A_ij)]. 100}/MFF_j

Where c_ij is the unit cost of healthcare resource group i for hospital j;

A_ij is the number of finished consultant episodes for healthcare resource group i for hospital j;

C_i is the average unit cost for healthcare resource group i in England;

MFF_j is the market forces factor for hospital j.

Purchaser efficiency index

Index for health authority j=[Σ_i(W_i . ΔA_ij)]/ΔR_j

Where W_i are weights for activity i (inpatients, outpatients, ambulance journeys, etc) based on shares of national expenditure for these activities;

ΔA_ij is the change in the amount of activity for activity j for health authority j from one year to the next;

ΔR_j is the real change in allocations to health authority j from one year to the next.

The reference cost index and the purchaser efficiency index differ in a number of ways. Although both are designed to concentrate on the costs of healthcare activity, the purchaser efficiency index was aimed at health authorities and the reference cost index is aimed at trusts. The reference cost index is designed to allow for comparisons across trusts within a year whereas the purchaser efficiency index tended to preserve relative inefficiencies between health authorities, concentrating on changes in efficiency over time for individual authorities. The reference cost index covers a fairly narrow range of activity and uses healthcare resource groups and finished consultant episodes as its measures of these activities. The purchaser efficiency index was more broadly based and used a variety of measures (finished consultant episodes, ambulance journeys, outpatient attendances, etc) weighted together.

Technical issues

Data

The current sets of unit costs are somewhat limited in their coverage of hospitals (box). Community, mental health, and ambulance trusts have been excluded, as have outpatient costs. Also, the 1998 data exclude information from medical specialties.

In calculating how much each healthcare resource group costs, hospitals have to make decisions about how to allocate their shared costs, such as nursing and heating, to each group. Although there are methodologies for allocating such costs, inevitably there will be variations in practice. Moreover, the apparent exclusion of certain income streams, such as the service increment for teaching and money for research and development, will have distorted the true costs of providing care since such funds can act as hidden subsidies. Unless there are tightly controlled systems that set out the precise way in which costs are allocated, there is always a possibility of “playing the system.” For example, anecdotal evidence suggests that services available to fundholding general practitioners under the old internal market were priced competitively because some overhead and shared costs were shifted onto medical emergency services (which were paid for automatically by health authorities).

On their own, the schedule and index provide a rather narrow view of performance. Quality, acceptability, and the overall service provided to a local population also need to be considered (box). The difficulty is in deciding how to interpret such a broad range of performance factors: for example, what is an acceptable level of trade-off between high costs and high quality?

The wider picture

Making sense of the reference costs in the wider context of other performance measures and the alternative data available on costs at a local level can be difficult.

For example, in Birmingham the bulk of the orthopaedic service is provided by the fourth most expensive provider in England (the Royal Orthopaedic Hospital), according to the 1998 reference cost index. In Manchester the service is provided more typically by district general and university hospitals.

Data from local health authorities for 1997-8 show that while Birmingham spends £9.98 per head of population on elective orthopaedics, Manchester spends £5.16 (Manchester Health Authority finance department, personal communication, 1998). However, Birmingham purchases 478 finished consultant episodes per 100 000 population whereas Manchester purchases 128. So, even without adjustment for case mix, the cost per admission in Birmingham is just over half (£2087) that in Manchester (£4031).

Moreover, public health data suggest that providing 300-400 total hip replacements per 100 000 elderly people is reasonable.⁸ Performance data show that Birmingham purchases 316 total hip replacements per year (84% above the average rate for England), whereas Manchester is underproviding at 215 (22% below the average rate for England).⁹ This low provision is reflected in longer waiting lists and in the fact that some patients wait more than 18 months for surgery.

Birmingham may be more expensive than Manchester according to the reference cost index (although local data suggest a slightly different picture) but the overall provision of hip replacements is higher and both waiting lists and times are shorter.

Statistical significance

A matter which neither the 1998 nor 1999 documents on reference costs address directly is the extent to which variations between hospitals' unit costs can be attributed to uncertainties inherent in the data. As Goldstein and Thomas have noted in relation to surgical mortality, once uncertainty intervals are calculated for the point estimates it becomes impossible statistically to separate the vast majority of surgeons in a league table.¹⁰ Similarly, for the unit costs for healthcare resource groups—and in particular the unit costs index—confidence intervals need to be shown for every hospital to highlight the degree of uncertainty associated with the numbers. As with clinical performance indicators in the performance assessment framework, it is likely that much of the variation in unit costs will be insignificant.

Although the 1998 and 1999 reference costs vary in the way in which they are compiled, some indication of the uncertainties in the data is reflected in the movement of trusts up and down the rankings of the reference cost index. In fact, there is virtually no relation between the rankings on the two indices: some hospitals moved from near the bottom of the 1998 index to near the top of the 1999 index.

Disentangling the impact of case mix (despite the apparent uniformity imposed by the use of healthcare resource groups) from the effects of the way a hospital is run or the behaviour and experience of clinicians is also difficult. If there are accurate data on costs, the statistical technique known as multilevel modelling can help identify the reasons for variation in unit costs.¹¹

The lack of detail on the types of costs that make up each hospital's unit cost for healthcare resource groups restricts investigation of the possible causes of variation between hospitals.¹²

Market forces factor

Both indices make adjustments for some costs outside the control of trusts. The market forces factor is a weighting applied to account for regional variations in costs, similar to London weighting, for example. The impact of the factor is heavily skewed: 10 trusts in London have their crude costs reduced by between 20% and 33% as a result of the weighting but 171 in other parts of England have their costs increased by between 0.1% and 4.5%. Whether these adjustments adequately compensate for uncontrollable variation in costs is unclear. Hospitals in south east England may believe that they are not dealt with properly by the factor and hospitals in the provinces may see it as yet another subsidy for “fat cat” institutions in London.

SUE SHARPLES

Behavioural and interpretational issues

The NHS Executive suggests that the database of unit costs will be a powerful tool not only for the NHS but also for patients' organisations. However, there may be a tendency for patients' organisations to equate high cost with high quality while the NHS is working towards reducing costs (and perhaps quality). While health authorities and primary care groups will need to take a more global view of efficiency, it is hard to predict exactly how they will use these data in practice.

Just as difficult to judge is how trusts will behave in response to publication of the data. Although the possibility exists for perverse incentives to develop, the extent to which trusts' behaviour is pushed in undesirable ways will depend on the ways that other parts of the NHS use the reference costs. If the Department of Health uses unit costs as targets, for example, this may lead to undesirable distortions in the behaviour of hospitals, such as a tendency to concentrate on low cost cases or to cut costs without paying attention to the impact on quality; this is what happened with the purchaser efficiency index.

Conclusion

Before the new reference costs become too embedded as a tool of performance management, the NHS should remember and learn from its experiences with the efficiency index. Many of the problems with the efficiency index stemmed from a lack of initial research and testing.

The interrelation between cost and other performance indicators needs to be assessed by streetwise observers. On the Department of Health's website, the NHS Executive has stated that “a high position on the [unit cost] Index may not necessarily indicate inefficiency . . . [and] a low position on the Index should not be a cause for complacency.” The weight given to different performance indicators, such as cost, quality, and equity, and the trade-offs between these measures remain unresolved. The possibility that this information may change incentives in a perverse way, producing unexpected and undesirable outcomes, needs to be considered.¹³

The reference costs provide a starting point for those who commission health care to engage with providers about costs, but there is no compelling evidence that the reference costs will necessarily encourage more efficient production of health care than the internal market and the purchaser efficiency index did. Although research is being carried out by the Department of Health into the causes of the variation in costs between trusts, and more sophisticated indices are being constructed, it is possible that (as with many of the clinical indicators) observed variations will be statistically insignificant.¹⁴ If this is the case, then it may well be more appropriate to tackle the reasons for variation at the extremes of the cost distribution both for trusts and individual procedures rather than use the general comparative approach of the reference costs.