Skip to main content
The BMJ logoLink to The BMJ
. 1999 Jul 10;319(7202):94–98. doi: 10.1136/bmj.319.7202.94

Measuring quality of care with routine data: avoiding confusion between performance indicators and health outcomes

Antonio Giuffrida a, Hugh Gravelle a, Martin Roland b
PMCID: PMC28159  PMID: 10398635

Abstract

Objective

To investigate the impact of factors outside the control of primary care on performance indicators proposed as measures of the quality of primary care.

Design

Multiple regression analysis relating admission rates standardised for age and sex for asthma, diabetes, and epilepsy to socioeconomic population characteristics and to the supply of secondary care resources.

Setting

90 family health services authorities in England, 1989-90 to 1994-5.

Results

At health authority level socioeconomic characteristics, health status, and secondary care supply factors explained 45% of the variation in admission rates for asthma, 33% for diabetes, and 55% for epilepsy. When health authorities were ranked, only four of the 10 with the highest age-sex standardised admission rates for asthma in 1994-5 remained in the top 10 when allowance was made for socioeconomic characteristics, health status, and secondary care supply factors. There was also substantial year to year variation in the rates.

Conclusion

Health outcomes should relate to crude rates of adverse events in the population. These give the best indication of the size of a health problem. Performance indicators, however, should relate to those aspects of care which can be altered by the staff whose performance is being measured.

Key messages

  • The NHS executive has proposed that admission rates for asthma, diabetes, and epilepsy could be used at health authority level as indicators of the quality of primary care

  • There is considerable year to year variation in the ranking of health authorities by admission rates for these conditions, even when rates are aggregated. This makes it hard to interpret a single year’s data: a 3 year average would be more reliable

  • Morbidity, socioeconomic characteristics, and secondary care supply are important confounding factors that explain between a third and a half of the variation in admission rates across health authority areas

  • Performance indicators should relate to aspects of care that can be controlled by decision makers. Confounding factors have a clear impact on admission rates and must therefore be taken into account if such rates are to be used as performance indicators

Introduction

There is an increasing emphasis on measurement of performance in the NHS. In setting out its future policy on the NHS in the white paper The New NHS the government emphasised the need for a new performance framework to measure progress towards its objectives.1 A subsequent consultation paper proposed performance indicators for the comparison of the quality of health care provided in health authorities.2

We analysed admission rates for asthma, epilepsy, and diabetes, which are three of the indicators proposed for assessment of performance in primary care. Admission rates for chronic conditions have been used in other countries, principally the United States, as measures of access to primary care.310 The conditions chosen are those for which timely and effective primary care could be expected to reduce the risk of admission to hospital by preventing the onset of illness, controlling an acute episode of illness, or better long term management.

Previous research in the United Kingdom suggests that some characteristics of primary care that might be taken to reflect quality of practice are related to admission rates for chronic diseases. For example, lower admission rates for asthma have been found in practices whose prescribing patterns suggested better preventive care,11 and lower admission rates for diabetes were found in practices with better organised diabetic care.12 Griffiths et al found that higher admission rates for asthma in east London were found in small practices, in which, the authors said, it might have been more difficult to develop systems for identifying, reviewing, and educating patients.13

The interpretation of admissions for the chronic conditions used in the proposed indicator is not straightforward because of potentially confounding factors such as the socioeconomic characteristics of the population.710,14 Admission rates might also be affected by hospital policies. Durojaiye et al found admission rates for asthma in Nottingham increased sharply between 1975 and 1985, a period of time when there seemed to be improvements in primary care; this was attributed in part to changing admission policies.15 Casanova and Starfield, however, found that admission rates for children in Spain were not correlated with supply side or socioeconomic factors.16 More generally, Baker and Klein found that socioeconomic conditions explained considerable proportions of variation among family health services authorities in primary care outcomes such as cervical cytology rates.17

In the absence of direct measures of incidence and prevalence of disease, crude admission rates (table 1) can be used as a measure of the absolute magnitude of a health problem. Such rates should not, however, be used to monitor the performance of a geographical area when they are affected by factors which are outside its control and vary across areas, so that areas are affected differentially. Performance indicators should be measures of what the relevant decision makers can reasonably be held to account for.

Table 1.

Average crude rates of admission to hospital per 10 000 population*

Condition and year Mean (SD) Range
Asthma
1989-90 19.91 (6.04) 9.78-36.00
1990-1 19.62 (5.97) 6.40-35.40
1991-2 21.57 (5.83) 9.44-35.83
1992-3 20.95 (5.78) 9.94-36.47
1993-4 22.17 (5.69) 12.58-40.92
1994-5 20.27 (5.87) 3.24-34.73
Diabetes
1989-90 10.48 (2.92) 4.43-19.91
1990-1 10.89 (3.57) 4.74-21.41
1991-2 11.74 (4.55) 4.05-27.11
1992-3 12.46 (5.26) 4.86-30.55
1993-4 12.50 (5.54) 4.83-35.64
1994-5 13.00 (5.46) 2.19-31.50
Epilepsy
1989-90  7.57 (2.46) 3.13-14.38
1990-1  7.87 (2.52) 3.49-14.95
1991-2  8.40 (2.54) 4.08-17.24
1992-3  8.34 (2.33) 4.17-14.26
1993-4  8.27 (2.29) 3.82-16.98
1994-5  8.72 (2.75) 1.82-16.52
*

Source of numerators: hospital episodes statistics; source of denominators: Office for National Statistics. 

We have used the example of the admission rates for the three chronic diseases proposed as primary care performance indicators in the United Kingdom to illustrate the difference between health outcomes and performance indicators. We examine the extent to which demographic composition, socioeconomic factors, measures of population health, and secondary care characteristics influence admission rates and thus cloud any relations between admission rates and the quality of primary care.

Method

Data on hospital admission rates were obtained from the hospital episodes statistics unit at the Department of Health for each of the financial years 1989-90 to 1994-5. The data covered the 367 English local authorities and were aggregated to the level of family health services authority. The admission rate for an area was derived from the number of admissions (including readmissions) or day case episodes of residents, with asthma (ICD-10 (international classification of diseases, 10th revision) codes J45-J46), diabetes mellitus (ICD-10 codes E10-E14) and epilepsy (ICD-10 codes G40-G41) as the primary diagnosis. Rates of hospital admissions were measured per 10 000 residents. We used data from 1989-90 to 1994-5 because we also wanted to examine the magnitude of year to year variation on a consistent geographical basis before the introduction of new unified health authorities.

We used multiple regression analyses to determine how much of the variation in age and sex standardised admission rates for asthma, diabetes, and epilepsy could be explained by the socioeconomic characteristics of the areas, population health, and provision of secondary care.

Socioeconomic variables were derived from the 1991 census and included housing conditions, social class, unemployment, and, as an indicator of wealth, car ownership. We included two measures of population health from the 1991 census: the proportion of the working age population who reported being permanently sick and who reported limiting long term illness.

The variables used to estimate the supply of secondary care services were the number of hospital medical staff in general medicine per 10 000 population and a variable that reflect the number of beds per head of population weighted for distance from the hospital.18 Because of the large number of potential explanatory variables we selected the final set by stepwise regression, retaining only those variables that showed significant partial correlation with the admission rate. For example, we initially included the standardised mortality ratio as a health measure but found that it had no additional explanatory power when it was added to regressions already containing the other health measures. The variables used in the final regression analysis are shown in the table in the Appendix.

We calculated three predicted admission rates for each condition for each geographical area using the coefficients from the regression. The first was the rate predicted by using only the health variables. The second predicted rate used the health variables and the socioeconomic variables; and the third predicted rate used all the variables in the final set: health, socioeconomic factors, and supply of secondary care. The differences between the actual rate for an area and the predicted rates are measures of the possible effect of quality of primary care on admissions after allowance for possible confounding by health, socioeconomic characteristics, and supply of secondary care.

Results

Table 1 shows the clear variation across the 90 health authority areas in crude admission rates for the three conditions. Although there was stability in average rates between years, there was considerable instability between the ranked position of individual health authorities from year to year. For example, when health authorities were ranked in order of admission rate, the ranking of an individual health authority in 1994-5 compared with 1993-4 changed by 10 or more places in 46 (51%) areas for asthma, 28 (31%) areas for diabetes, and 36 (40%) areas for epilepsy. We found that combining these three rates, as proposed in the NHS executive’s consultation document, did not produce rankings that were notably more stable, with 28 (31%) authorities still changing rank order by more than 10 places and 10 (11%) by over 20 places.

The regression analyses show that a high proportion of the variance in age and sex standardised admission rates can be explained by socioeconomic and secondary care factors (table 2). Overall, these variable explained 45% of the variance in admission rates for asthma, 33% for diabetes, and 55% for epilepsy.

Table 2.

Multiple regression analysis of effect of socioeconomic and secondary care factors on variance in age standardised rates of admission for three conditions. Figures are regression coefficients (SE)

Variables Asthma Diabetes§ Epilepsy
Constant –28.39** (7.95) 3.70** (0.65) –7.40** (1.45)
Supply of secondary care:
 Hospital beds 2.97** (0.46) 0.97** (0.12) 0.77** (0.20)
 General physicians 1.60* (0.93) 0.38** (0.15) 0.68** (0.24)
Morbidity measures:
 Limiting long term illness 0.42** (0.17) NU NU
 Permanently sick NU 0.61** (0.09) 0.35** (0.12)
Socioeconomic characteristics:
 Population density 0.08** (0.03) –0.07** (0.03) –0.02** (0.01)
 Unemployment 1.71** (0.24) NU 0.17** (0.06)
 No central heating NU 0.13** (0.03) 0.05** (0.01)
 Crowded accommodation 0.51** (0.20) NU NU
 No car –0.92** (0.14) –0.47** (0.09) NU
 New Commonwealth –0.41** (0.11) NU –0.18** (0.03)
 Retired living alone 1.18** (0.30) NU 0.24** (0.05)
 Students –0.13** (0.05) –0.37** (0.18) NU
 Social class I and II NU –0.38** (0.15) NU
 Population mobility NU 0.36** (0.07) NU
Year dummies:
 1990-1 –0.45 (0.74) 0.01 (0.04) 0.28 (0.28)
 1991-2 1.37* (0.73) 0.06 (0.05) 0.74** (0.27)
 1992-3 0.82 (0.77) 0.10** (0.05) 0.74** (0.27)
 1993-4 2.08** (0.75) 0.10** (0.05) 0.67** (0.25)
 1994-5 0.68 (0.76) 0.13** (0.05) 1.37** (0.28)
R2 adjusted 0.45 0.33 0.55
**

P⩽0.01; **0.01<P⩽0.05; *0.05<P⩽0.1. 

White’s method19 used to adjust calculation of SE to allow for relation between variance in admission rates and size of health authority. 

Variables are defined and their souces identified in the Appendix

§

Regression estimated with logarithmic transformation as Mackinnon White and Davidson test20 indicated that this was more appropriate than linear model for this analysis. 

NU=variable not used in final regression analysis because of lack of significance. 

Table 3 uses the example of admission rates for asthma to show how the ranking of health authorities is affected when they are ranked by crude rates (column 1), by rates adjusted for age and sex (column 2), by rates adjusted for age, sex, socioeconomic factors, and limiting long term illness reported in the census (column 3), and finally by rates which are also adjusted for factors related to supply of secondary care (column 4). It is conventional to standardise for age and sex, but inspection of the table shows that other potentially confounding factors have at least as great an impact on the rankings. Eight out of the 10 areas with the highest crude admission rates are in the top 10 ranked by age and sex standardised rates, whereas only four out of the top 10 by age and sex standardised rates remain in the top 10 when allowance is made for morbidity, socioeconomic factors, and secondary care factors.

Table 3.

 Top 10 and bottom 10 family health services health authorities according to rates of admission to hospital for asthma, 1994-5, with changes in ranking compared with ranking by age and sex standardised rate shown in paraentheses

Rank Crude rate Directly standardised for age and sex Allowing for morbidity factors Allowing for morbidity and socioeconomic factors Allowing for morbidity*, socioeconomic factors, and supply of secondary care
Top 10 rates
1 Manchester (1) Kingston and Richmond Kingston and Richmond (0) Kingston and Richmond (0) Kingston and Richmond (0)
2 Rochdale (2) Manchester Rochdale (2) Rochdale (2) Rochdale (2)
3 Liverpool (0) Liverpool Manchester (1) Enfield and Haringey (9) Berkshire (24)
4 St Helens and Knowsley (1) Rochdale Liverpool (1) Bury (2) Doncaster (3)
5 Bury (1) St Helens and Knowsley Kensington, Chelsea, and Westminster (3) Sandwell (8) Coventry (5)
6 Sheffield (10) Bury Bury (0) Gateshead (12) Enfield and Haringey (6)
7 Oldham (2) Doncaster Enfield and Haringey (5) Coventry (3) Sefton (4)
8 Doncaster (1) Kensington, Chelsea, and Westminster St Helens and Knowsley (3) Bradford (9) Sandwell (5)
9 Coventry (1) Oldham Coventry (1) Manchester (7) Gateshead (9)
10 Sandwell (3) Coventry Oldham (1) Doncaster (3) Buckinghamshire (22)
Bottom 10 rates
81 Cornwall and Isles of Scilly (7) Kirklees Cornwall and Isles of Scilly (7) Northamptonshire (6) Ealing, Hammersmith, and Hounslow (5)
82 Avon (4) Hampshire Kirklees (1) Kent (5) Kirklees (1)
83 Kirklees (2) Northumberland Northumberland (0) Avon (5) Essex (3)
84 Essex (4) Somerset Somerset (0) Dudley (11) Camden and Islington (43)
85 Hampshire (3) North Yorkshire North Yorkshire (0) Essex (5) Dudley (12)
86 Northumberland (3) Barnet Barnet (0) Cornwall and Isles of Scilly (12) Barnet (0)
87 Somerset (3) Northamptonshire Northamptonshire (0) Somerset (3) Croydon (1)
88 North Yorkshire (3) Croydon Croydon (0) Croydon (0) Salford (43)
89 Derbyshire (0) Derbyshire Derbyshire (0) Derbyshire (0) Derbyshire (0)
90 Bedfordshire (0) Bedfordshire Bedfordshire (0) Bedfordshire (0) Bedfordshire (0)
*

Limiting long term illness reported in the 1991 census. 

Discussion

Our results highlight the fundamental difference between performance indicators and health outcomes. A simple count of adverse health events—such as deaths, admissions, or disability—is a measure of the burden of a health problem in a population—that is, the health outcome. However, a performance indicator designed to improve that outcome should relate only to those factors that are under the control of the staff to whom it is being applied.

Admission rates for conditions for which admission could be avoided by good primary care have been widely used in the United States. Their main application has been as a measure of access to primary care rather than of the quality of primary care. Nevertheless, there are data which suggest that good primary care should help to avoid admission for these conditions. What we have shown is that these admissions are substantially influenced by factors outside the control of the primary care team—that is, the characteristics of their population and the supply of secondary care resources. The admission rates should be adjusted for these factors before being used as measures of the quality of primary care. Even so, we do not know whether there are other important factors that we have not been able to include in our analyses. For example, there are no data available that would enable allowance to be made for the prevalence of these conditions in individual health authority areas or the admission policies of individual hospitals.

We used a particular type of performance indicator, but the lesson is more general and would apply to other suggested indicators.21 It is essential to test for confounding of indicators by factors outside the control of the decision makers whose performance is being monitored. Our results point to other potential problems of using admission rates as indicators of quality of care. The rates fluctuate greatly from year to year, showing the statistical instability of any relatively rare event.22 While this difficulty could be reduced by using a 3 year moving average for a health authority, it means that the indicators would be even more difficult to apply to individual practices, where greater year to year variation would be expected to occur because of the smaller population size. Marshall and Spiegelhalter have emphasised the importance of accompanying any performance indicators with measures of sampling variability.23

Any single performance indicator may be a misleading guide to the overall performance of an organisation as it covers only one dimension of that performance, and concentration on one aspect of care may produce perverse incentives to ignore other aspects of performance. If performance indicators are to be used, it is important that they cover the full range of outputs and inputs for the sector in question. While health outcome should be related to crude rates of adverse events in the population, performance indicators should relate only to those aspects of care that can be altered by the staff whose performance is being measured.

Appendix

Table.

Explanatory variables in final regression analyses

Variable Description Mean (SD) Range Source
Supply of secondary care:

 General physicians/10 000 Hospital medical staff in general medicine specialty group per 10 000 population. By regional health authority 2.83 (0.36) 2.23-3.64 HSI
 Hospital beds provision Distance weighted No of hospital beds per head of population 2.85 (0.75) 1.29-4.64 Carr-Hill et al18
Morbidity measures:
 Limiting long term illness Proportion of total population with limiting long term illness 13.28 (2.23) 8.87-18.48 Census
 Permanently sick Proportion of adult population permanently sick 4.13 (1.57) 2.01-7.98 Census
Socioeconomic characteristics of the population:
 Population density Population per hectare 17.80 (20.19) 0.61-100.76 ONS
 Unemployment Proportion of population claiming unemployment benefits 10.09 (3.52) 5.09-21.75 ONS
 No central heating Proportion of households lacking central heating 17.00 (8.16) 4.87-46.43 Census
 Crowded accommodation Proportion in households in crowded accommodation (>1 per room) 4.88 (2.68) 2.32-19.35 Census
 No car Proportion in households with no car 26.52 (9.81) 10.96-51.30 Census
 New Commonwealth Proportion of private households headed by person born in New Commonwealth or Pakistan 4.05 (4.85) 0.31-31.19 HSI
 Retired living alone Proportion of those of pensionable age living alone 34.11 (3.18) 29.87-48.51 Census
 Students Proportion of 17 year olds who are students 40.99 (7.32) 30.69-67.86 Census
 Social class I and II Percentage of households with head in class I or II 35.72 (7.98) 18.28-58.76 Census
 Population mobility Proportion of residents moving outside district in past year 3.84 (1.81) 1.76-12.74 Census
Year dummies:
 1990-1 Dummy variable; equals 1 in financial year 1990-1; 0 otherwise
 1991-2 Dummy variable; equals 1 in financial year 1991-2; 0 otherwise
 1992-3 Dummy variable; equals 1 in financial year 1992-3; 0 otherwise
 1993-4 Dummy variable; equals 1 in financial year 1993-4; 0 otherwise
 1994-5 Dummy variable; equals 1 in financial year 1994-5; 0 otherwise

Census: 1991 census of population. 

HSI: health service indicators database.24 

ONS: office for national statistics. 

Editorial by Jankowski

Footnotes

Funding: NPCRDC is funded by the Department of Health. The views expressed in this paper are those of the authors and not necessarily those of the Department of Health.

Competing interests: None declared.

References

  • 1.Secretary of State for Health. The new NHS. London: Stationery Office; 1997. (Cm 3807.) [Google Scholar]
  • 2.NHS Executive. The new NHS: a national framework for assessing performance. Leeds: NHSE; 1998. [Google Scholar]
  • 3.Arnold J, Zuvekas A. Using health outcomes to evaluate the primary care system: a manual. Washington: US Department of Health and Human Services, Public Health Service, Health Resources and Service Administration; 1989. [Google Scholar]
  • 4.Billings J, Hasselblad V. Use of small area analysis to assess the performance of the outpatient delivery system in New York city: a preliminary study. New York: Health System Agency of New York; 1989. [Google Scholar]
  • 5.Massachusetts Division of Health Care Finance and Policy. Improving primary care: using preventable hospitalization as an approach. Boston: Division of Health Care Finance and Policy; 1995. [Google Scholar]
  • 6.Lambrew JM, DeFriese GH, Carey TS, Ricketts TC, Biddle AK. The effects of having a regular doctor on access to primary care. Med Care. 1996;34:138–151. doi: 10.1097/00005650-199602000-00006. [DOI] [PubMed] [Google Scholar]
  • 7.Begley CE, Slater CE, Engel MJ, Reynolds TF. Avoidable hospitalizations and socio-economic status in Galveston County, Texas. J Community Health. 1994;19:377–387. doi: 10.1007/BF02260406. [DOI] [PubMed] [Google Scholar]
  • 8.Bindman AB, Grumbach K, Osmond D, Komaromy M, Vranizan K, Luire N, et al. Preventable hospitalizations and access to health care. JAMA. 1995;274:305–311. [PubMed] [Google Scholar]
  • 9.Billings J, Anderson GM, Newman LS. Recent findings on preventable hospitalizations. Health Affair (Millwood) 1996;15:239–249. doi: 10.1377/hlthaff.15.3.239. [DOI] [PubMed] [Google Scholar]
  • 10.Weissman JS, Gatsonis C, Epstein AM. Rates of avoidable hospitalization by insurance status in Massachusetts and Maryland. JAMA. 1992;268:2388–2394. [PubMed] [Google Scholar]
  • 11.Aveyard P. Monitoring the performance of general practices. J Eval Clin Pract. 1997;3:275–281. doi: 10.1046/j.1365-2753.1997.t01-1-00004.x. [DOI] [PubMed] [Google Scholar]
  • 12.Farmer A, Coulter A. Organization of care for diabetic patients in general practice: influence on hospital admissions. Br J Gen Pract. 1990;40:56–58. [PMC free article] [PubMed] [Google Scholar]
  • 13.Griffiths C, Sturdy P, Naish J, Omar R, Dolman S, Feeder G. Hospital admissions for asthma in east London: associations with characteristics of local general practices, prescribing, and population. BMJ. 1997;314:482–486. doi: 10.1136/bmj.314.7079.482. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Watts JP, Cowmen P, Lewis RA. The relationship between asthma admission rates, routes of admission, and socio-economic deprivation. Eur Resp J. 1996;9:2087–2093. doi: 10.1183/09031936.96.09102087. [DOI] [PubMed] [Google Scholar]
  • 15.Durojaiye LIA, Hutchison T, Madeley RJ. Improved primary care does not prevent the admission of children to hospital. Public Health. 1989;103:181–188. doi: 10.1016/s0033-3506(89)80073-3. [DOI] [PubMed] [Google Scholar]
  • 16.Casanova C, Starfield B. Hospitalizations of children and access to primary care: a cross-national comparison. Int J Health Services. 1995;25:283–294. doi: 10.2190/PCF7-ALX9-6CN3-7X9G. [DOI] [PubMed] [Google Scholar]
  • 17.Baker D, Klein R. Explaining outputs of primary health care: population and practice factors. BMJ. 1991;303:225–229. doi: 10.1136/bmj.303.6796.225. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Carr-Hill RA, Hardman G, Martin S, Peacock S, Sheldon TA, Smith P. A formula for distributing NHS revenues based on small area use of hospital beds. York: Centre for Health Economics, University of York; 1994. (Occasional Paper.) [Google Scholar]
  • 19.White H. A heteroskedasticity consistent covariance matrix and a direct test for heteroskedasticity. Econometrica. 1978;46:817–838. [Google Scholar]
  • 20.MacKinnon JG, White H, Davidson R. Tests for model specification in the presence of alternative hypotheses: some further results. J Econometrics. 1983;21:53–70. [Google Scholar]
  • 21.McColl A, Roderick P, Gabbay J, Smith H, Moore M. Performance indicators for primary care groups: an evidence based approach. BMJ. 1998;317:1354–1360. doi: 10.1136/bmj.317.7169.1354. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Moore AT, Roland MO. How much variation in referral rates among general practitioners is due to chance. BMJ. 1989;298:500–502. doi: 10.1136/bmj.298.6672.500. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Marshall EC, Spiegelhalter DJ. Reliability of league tables of in vitro fertilisation clinics: retrospective analysis of live birth rates. BMJ. 1998;316:1701–1704. doi: 10.1136/bmj.316.7146.1701. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.NHS Executive. Health services indicators handbook 1994. Leeds: NHSE; 1994. [Google Scholar]

Articles from BMJ : British Medical Journal are provided here courtesy of BMJ Publishing Group

RESOURCES