Unwarranted Variation in Pediatric Medical Care

The findings from studies of unwarranted variation in medical care rival the decoding of the human genome in the potential for improving patients' health and well-being. Small area analysis has revealed myriad ills in our health care system and at the same time has led the way in improvement efforts and reform. Although child health researchers in the United States were relatively slow in initiating analysis of medical care variation, there has been notable progress in the past 15 years, and some research has surpassed that of adult health care. In other areas, the study of pediatric variation substantially lags, suffering from an absence of conceptual frameworks and from incomplete data about pediatric health care resources and delivery. This deficiency has impeded the child health care community from fully engaging in the current national debates about health care system reform.

This article provides a survey on the concepts, methods, and applications of the study of unwarranted variation in health care with particular attention to children's medical services. Unwarranted variation is the variation in medical care due to differences in health system performance. Originally, analyses examining unwarranted variation were termed small area analysis in reference to their examination of variation of health care resources (eg, hospital beds, physicians) and utilization across empirically defined health care service areas. With improvements in data quality, recent efforts have been directed toward the measurement of variation across providers such as hospitals and clinicians. Regardless of the units that define the population or patient denominators, the study of health care variation faces similar challenges in methods and interpretation.

Origins of Small Area Analysis

Tonsillectomies in England and Wales from 1908 to 1938

Although the connection of place with health is an ancient concept, the idea that medical practice might also vary across locales, haphazardly and without apparent reason, was initially recognized in England in the first half of the twentieth century. In response to growing concerns about the “physical deterioration” of the public, Parliament included a clause in the 1907 Education Bill which provided for the universal medical inspection of children at entry to schools and periodically thereafter. Parliament initially deferred funding treatment, a policy reflecting worries about the costs, but included the vague language, “… the power to make such arrangements as may be sanctioned by the Board of Education for attending to the health and physical condition of children educated in Public Elementary Schools.”¹ For advocates of publicly funded treatment, this was a clever compromise. As the full picture of the ill health of school-aged children developed, the provision of pediatric medical services inevitably followed.

In the 1920 annual report of Chief Medical Officer of the Board of Education, the most common “defects” identified were dental disease, nose and throat disease, defective vision, uncleanliness, and malnutrition. About 4% of children were diagnosed with “adenoids and enlarged tonsils,” a figure that rose to 6% by 1931.^2,3 Treatment did indeed follow diagnosis. In 1923, the majority of local education authorities provided adenoid/tonsillectomies either through their own clinics or at local hospitals. The surge in diagnosis and surgical treatment was a surprise to the Chief Medical Officer and led to the establishment of the Committee on Enlarged Tonsils and Adenoids in 1924.⁴ The Committee issued cautions against the indiscriminate use of tonsillectomies and worked to improve surgical safety; however, no consensus could be reached on the function of the tonsils, the definition of tonsillar disease, or the indications for the procedure. In the absence of professional consensus within the Board of Education or the larger medical community, the total number of procedures grew to 84,000 by 1931, representing three quarters of the total procedures in English and Welsh school-aged children.³

Although no one knew what to do with the information, the Board of Education medical staff conducted exquisite epidemiologic analyses of the tonsillectomy epidemic that reached their full maturation in 1938 with the publication of “The Incidence of Tonsillectomy in School Children” in the Proceedings of the Royal Society of Medicine by J.A. Glover.⁵ This article was the first extant academic report using the study design we now call small area analysis and was nearly the only publication for 35 years.

Dr. Glover's article is essential reading for all students interested in unwarranted variation of pediatric medical care. He documents the remarkable rise of the use of operative treatment for enlarged tonsils in epidemiologic terms, calling it an “epidemic,” and notes the over tenfold variation in rates across local educational authorities. He also identified an intriguing difference in procedure rates between two communities with prominent medical schools, noting that the 1936 annual incidence in children aged 5 to 14 years was 0.3% in Cambridge and 3.1% in Oxford. In a prescience and forthright discussion, he argued that “A study of the geographic distribution in elementary school children discloses no correlation between…any other factor, such as overcrowding, poverty, bad housing, or climate. In fact it defies any explanation, save that of variation of medical opinion on the indications for operation.”⁵

The convergence of circumstances and ideas that led to these novel and provocative studies of pediatric tonsillectomies reveals a great deal about the methods and usefulness of small area analysis. First, the identification of unwarranted variation requires good population-based data. Education funding in England, including for the School Medical Service, was highly centralized, with money flowing from and data flowing to London. In an era when the tools of data tabulation were ledger books and pencils, the School Medical Service kept meticulous records that were hand tabulated into tables and figures that rival any of today. These records about the numbers of children, “defects,” and treatments could have been doomed to obscurity except for the second important condition—the close alignment of the Department of Health with its epidemiologists and the School Medical Service. Indeed, for many years they were both directed by Dr. George Newman, a renowned leader in British public health. The application of epidemiologic methods led to precise denominator and event definitions, stratification by student characteristics, and, ultimately, to calculation of incidence rates by place. For the first time, the methods used to measure and characterize diphtheria and rheumatic fever were applied to the study of medical care.

By using local educational authorities as units of analysis, Drs. Newman and Glover attributed tonsillectomy rates to the medical staff responsible for the delivery of tonsil-lectomies. The use of other political boundaries might have been easier but would have smoothed the variation and obscured the connection between procedure rates and the local medical staff.

John Wennberg and Medical Care in Vermont, Circa 1970

After Glover's 1938 article, few publications described regional variation in medical care,^6,7 and none were about children until John Wennberg and Alan Gittelsohn published their seminal report “Small Area Variation in Health Care Delivery” in the journal Science in 1973.⁸ Using population-based hospitalization data, Wennberg assigned Vermont towns to hospital service areas based on the patterns of travel for inpatient care. The geography of Vermont was particularly favorable to small area analysis, with scattered population centers that were principally reliant on a single hospital and its associated medical staff. The population of each hospital service area was linked to an identifiable supply of hospital beds and physicians. Despite the relatively homogenous population, the per capita number of hospital beds, hospital personnel, and physicians varied over 50% across hospital service areas. Hospital discharge rates for medical and surgical conditions varied more than twofold, and tonsillectomy rates varied more than tenfold. Wennberg could find little relation between greater utilization rates and need, such as would be indicated by higher mortality rates. Costs were higher in the areas with higher capacity (beds, physicians) and utilization.

Given today's recognition of the irrationalities in health care delivery, it may be difficult to appreciate that in the 1970s these findings were initially ignored, later attacked⁹ before being replicated,^10,11 and then widely embraced by clinicians, health system administrators, and policy makers. The most common criticism of small area analysis was that the variation in capacity and utilization was expected and reflected needed treatment for sicker populations.^12–14 Some attributed the variation to chance and imprecision in the rates, leading to the publication of several reports on statistical inference in measuring variation.^15,16 Other cited causes were patient preferences for more care or differences in malpractice environments.¹⁷

A notable testament to the seminal character of Wennberg's article in Science is its citation by over 750 other academic reports.¹⁸ Subsequent studies by Dr. Wennberg and his Dartmouth colleagues examined a large number of different types of medical care and populations. In addition to studies across regions, access to Medicare claims permitted examination of variation in the care of the elderly across specific hospitals. Cohort study designs were used in addition to cross-sectional designs, and ecologic studies were supplanted by increasingly sophisticated hierarchical models with the patient as the unit of analysis.¹⁹ With funding from the Robert Wood Johnson Foundation, Wennberg, Fisher, and colleagues²⁰ developed the Dartmouth Atlas of Health Care series as a dissemination tool directed toward nonacademic audiences such as health care administrators, health policy makers, and congressional staffers. In the past 35 years these and other studies of unwarranted variation have had an unparalleled influence on the practice of medicine, the organization of delivery systems, the financing of medical care, and federal health care policy.²¹

Pioneer Studies in Unwarranted Variation in Pediatric Medical Care

Studies of variation in children's health care took root as a distinct field of study in the 1980s. Although Wennberg studied pediatric hospitalizations in Vermont and Maine,^22,23 these analyses were primarily an outgrowth of adult medical care research. In 1982, Connell and colleagues²⁴ conducted a specific study of hospitalization rates in Medicaid children across small areas in Washington State and noted 18-fold differences in gastroenteritis and 15-fold differences in lower respiratory infections. Wissow and colleagues²⁵ also observed tenfold differences in asthma hospitalizations across small areas in Maryland. In 1989, Perrin and colleagues reported that Boston children were more than twice as likely (relative risk, 2.65; 95% CI, 2.53–2.78) to be hospitalized for medical conditions when compared with children residing in Rochester, New York. The relative risk for children in New Haven, Connecticut was 1.80 (95% CI, 1.68–1.93). This pattern persisted in specific diagnosis-related groups, including bronchitis and asthma, fractures and sprains, and gastroenteritis. For conditions in which the diagnosis was certain and hospitalization requisite, little variation was observed. The relative risk between Boston and Rochester for femur fractures was 1.0 (95% CI, 0.7–1.4), for appendectomy 1.1 (95% CI, 0.9–1.4), and for bacterial meningitis 1.3 (95%CI, 0.8–1.9). A lower degree of variation was noted for surgical hospitalizations. These three studies stimulated several studies of pediatric medical care using similar methods.

Methodological Considerations

Because analyses of variation are now associated with a diverse set of research questions including racial and socioeconomic disparities, it is helpful to discuss the methods needed to measure health system performance. Studies of unwarranted variation require methods that link either locations or providers of care with populations through data with a high degree of locational specificity. For studying common causes of hospitalization and physician care, the minimal location information are data with the zip codes of populations and providers. As important as counties and states might be for examining public policy driven variation, these geopolitical units are usually too large or are geographically discordant with health markets.²⁶ Zip code referenced utilization data permit classic small area analysis in which the records are used to define geographic representations of health care markets, such as hospital service areas^27,28 or primary care service areas.²⁹ These areas are then used in calculating population-based utilization rates. Counties can be appropriate for developing health markets for care that is highly regionalized, such as in defining neonatal intensive care regions.³⁰

Unfortunately, this research has been slowed by a scarcity of population-based utilization data. For example, the bare minimum data elements necessary for small area analysis of hospitalization rates are patient zip code, provider zip code, patient characteristics (eg, age, sex, race/ethnicity), diagnoses, and procedures. To date, no national data set with these fields is available to pediatric researchers. The Health Care Cost and Utilization Project (HCUP) Kids' Inpatient Database is a national sample of hospitalizations and does not provide the geographic information necessary to conduct small area analysis.³¹ Some states offer discharge records through HCUP with the necessary zip code information,³¹ although prohibitions against directly or indirectly identifying providers limit the analyses to analyzing overall variation phenomena, removing one of the most basic reasons for studying unwarranted variation, that is, to provide public accountability through open description of specific health system performance.

Medicare data, in comparison, provide (for non-HMO–enrolled beneficiaries) information about 100% of beneficiary hospitalizations and, for a 20% sample of beneficiaries, all physician and other provider claims. These claims are linkable to other utilization of a patient irrespective of Medicare data file or year and to a denominator file with beneficiary characteristics including date of death. Institutional providers' (eg, hospitals, ambulatory surgical centers) names and associated data are releasable to the public. The Centers for Medicare and Medicaid require that tabulations with less than 11 events for a patient or a clinician are suppressed, and neither patients nor clinicians can be publicly identified. The Medicare data sets allow for national analyses using cross-sectional and cohort study designs and hospital-specific analyses with public release of hospital rates.^20,32

Because pediatric studies have been shaped by data availability, most of the early research used hospitalization data to study variation in discharge rates, with more recent efforts reliant on registry data of participating practices or care units. Although the quality of the research conducted has generally been excellent, the research questions are severely limited in scope when compared with studies in adults and are usually reported without identification of pediatric facility names.

Interpreting Variation in Pediatric Medical Care

An expanded definition of unwarranted variation is that it is the variation in medical resources, utilization, and outcome that is due to differences in health system performance. Not all variation is unwarranted. Dozens of alternative reasons might explain why medical care utilization rates are different across places or providers. Even if the underlying probabilities of a health care event are the same, rates may have large standard errors, suggesting the variation is a chance phenomenon. More importantly, population characteristics differ, such as in median age. Differences may also occur in underlying health risk as reflected in obesity or infant mortality rates. Areas with children having higher health risks as measured directly or through surrogate metrics such as zip code median household income are expected to have higher physician and hospital utilization and to have more physicians and hospital beds to provide that care.

Child clinicians and researchers are particularly attuned to the socioeconomic determinants of children's health and health care. Although health status contributes to differences in pediatric hospitalization rates, it is not the sole or even the primary cause.^27,33–37 In fact, in the evaluation of health system performance, these population differences are not of primary interest and are considered cofounders. “Unwarranted” refers to the portion of the variation that is explained not by population difference but by the quality, appropriateness, and efficiency of health care. In small area analysis, health status is controlled through study design or statistical models.

Over the past two decades, the most unifying classification of medical care variation has been developed by Wennberg and colleagues. Variation of utilization is parsed into three parts—effective, preference-sensitive, and supply-sensitive care. Variation in health care capacity, such as hospital beds and physicians, is a fourth nonutilization category.

Variation in Effective Care

Variation in effective care reflects differences in technical quality. Usually, the “right” rate is known for a given population. Immunization rates are the obvious pediatric example. Deviation from near 100% reflects less than ideal health system performance. Pediatrics has been actively involved in developing effectiveness data and is a leader in promoting system change and improvement to achieve the right rate.

Although the absence of population-based data that are comparable to Medicare claims has limited some variation research, pediatric health service researchers have excelled in developing patient registries through practice networks. For example, the Vermont-Oxford Network collects primary data on infants with birth weights ranging from 401 to 1500 g cared for in 750 member neonatal intensive care units, representing about two thirds of all very low birth weight infants in the United States.³⁸ Data on processes of care and outcomes are tabulated and reported back to individual units for use in quality improvement efforts.³⁹ There are some disadvantages to this approach. The rates are not population based in the epidemiologic or small area analysis sense of reporting the experience of an entire population, and they cannot be used to study predictors of admission to a neonatal intensive care unit. Similar to most other practice-based registries, the unit data are not publicly available, and each unit's data are also blinded to other units. The advantage of this strategy is that units do not have to worry about the consequences of public reporting. Perhaps as a result, the Vermont-Oxford Network has a high level of unit participation and high quality data.

The Cystic Fibrosis Foundation's Patient Registry was developed along similar lines, but all cystic fibrosis centers in the foundation are required to participate. Although the registry also reports data about care processes and outcomes to each center, the Cystic Fibrosis Foundation has recently made seven center-specific measures publicly available.⁴⁰

Variation in Preference-based Care

Variation in preference-based care refers to differences in rates for diagnostic and therapeutic procedures where there is uncertainty in the outcome or when there is a complex balance of possible benefits and harm. In this type of variation, there is no single right rate. The right rate reflects the decisions of fully informed patients and families. It is expected that the care choice will differ across families and, in turn, across regions.

The original analyses that led to this concept were studies of benign prostatic hypertrophy in adult men,⁴¹ decidedly a nonpediatric problem. In the early 1980s, there was high regional variation in the use of transurethral prostatectomies, with the decisions driven by the local theories of benefit and harm held by urologists. The literature on outcomes was not complete, but to the extent that it existed, it appeared to be ignored by urologists. Today there is a full effectiveness literature on the ever widening available treatment choices, and formal decision aids⁴² are used to assist men in making a choice that is consistent with their own utilities (ie, values). Often, but not always, the introduction of decision aids reduces utilization rates. A list of available decision aids and their sources can be found at the Ottawa Health Research Web site (http://decisionaid.ohri.ca/index.html). These aids greatly differ in quality; only a relatively few are available for pediatric illness.

Preference-based care remains a fruitful area of pediatric research. Some of the pediatric diseases that would seem to lend themselves to preference-based care include management of middle ear disease, chronic sinusitis, allergies, acne, and enuresis. Practice guidelines are an unsatisfactory solution to the complexity of decision making, often mixing physician opinion with evidence. Naturally, the recommendations will differ depending on the membership of the committee authoring the guideline. Variation in use rates reflects local practice styles and, in some instances, the relative dominance of the pediatrician or pediatric surgeon in the child's care. Even if all of the outcomes and associated probabilities are known, an individual child's outcomes are likely to be maximized when the child or family can participate fully in evaluating the treatment choices and making the medical care decision.

Variation in Supply-sensitive Care

Supply-sensitive care refers to medical services for which use rates are sensitive to the local availability of health care resources such as beds and physicians. Although in some instances effective care may be constrained by the lack of resources, this category is principally concerned with the many types of medical care for which there is weak or no evidence of effectiveness, and for which the complexity of the decision making and the rich number of alternatives make efficacy and effectiveness studies difficult. These services are also termed discretionary. Generally, the right rate is the lowest rate with comparable outcomes.

Medical admissions are often considered supply-sensitive care.^43,44 Medical discharge rates vary 200% to 300% across hospital services areas and are strongly influenced by area bed supply.^27,45 Asthma hospitalizations are a useful pediatric example. A clinician caring for a child with an asthma exacerbation has the option of longer treatment and monitoring in the office or emergency department or hospitalization, and will decide after considering physiologic parameters, past medical history, social support available to the family, and parental preferences. Although this seems to be a rational judgment, clinical opinions will differ frequently, and the guidelines available for management of an acute ill asthmatic patient do not provide clear guidance.⁴⁶ Invisible to the clinician is the supply of beds and its subtle influence on local practice patterns. There is no evidence that hospital beds are explicitly rationed in lower supply areas or that patients are harmed by fewer hospital days.

In the 1990s, McConnochie and colleagues⁴⁷ studied the variation in pediatric discharge rates in New York State using counties and small areas in Monroe County as units of analysis. Discharge rates for lower respiratory illness varied over threefold and were inversely correlated with area socioeconomic indicators. These analyses were extended to a wider set of conditions with similar findings.^34,48 Capacity measures, such as bed supply, were not explicitly examined. Although these and other studies⁴⁹ confirm the known relationship between wealth and health, the studies suffer from possible ecologic biases and provide little insight into relative health system performance. Indeed, Gorton's study of pneumonia discharges in Pennsylvania children suggests differences in illness levels may be relatively unimportant.³⁷

Other studies of pediatric hospitalization variation have focused on the appropriateness of admissions⁵⁰ or the quality of ambulatory care.⁵¹ Differences in the quality of ambulatory care are thought to be an important cause of varying hospitalization rates, although the empiric evidence remains weak. When studies have examined the association of capacity and pediatric use, a strong association has been observed.^27,45 Perhaps both the quality of ambulatory care and capacity will turn out to be important explanations of unwarranted variation in pediatric hospitalization rates.

Other types of adult care that have been shown to be supply sensitive include ICU days, imaging studies, and physician visits.^52,53 Unfortunately, the lack of population-based pediatric data linked to provider and patient location has severely limited comparable studies.⁴⁵

Variation in Health Care Capacity

Measuring the variation in the number of child health physicians has proven to be easier than counting pediatric beds. Although the literature has been relatively silent on the former, several studies have shown marked variation in the per capita (eg, child or newborn) number of general pediatricians and pediatric subspecialists. Chang and Halfon⁵⁴ showed a lack of association between general pediatrician supply across states and indicators of child health needs. Mayer⁵⁵ observed a high degree of variation across Dartmouth Atlas regions for different pediatric subspecialists, and Goodman and colleagues⁵⁶ found little relation between the supply of neonatologists and regional differences in perinatal risk. None of these studies linked capacity to the utilization of pediatric services.

Future Directions

Since Glover's 1938 report, studies of unwarranted variation in pediatric medical care have taken root but have only flourished in studies of technical quality (ie, evidence-based care). The causes of differences in utilization rates and costs have been largely attributed to differences in patient need, although these conclusions are based upon a small set of studies with limited data. This view places the child health delivery system in a disadvantageous position as the debate about health system performance and reform is driven by considerations of the efficiency (ie, cost controlled by quality) of providers. Further research in unwarranted variation in pediatric care will require the use of commercial payer claims data sets that include the location or provider variables necessary to conduct small area analysis of hospital and physician care over large and diverse child populations.