Skip to main content
PMC home page
Wiley Open Access Collection logoLink to Wiley Open Access Collection
. 2019 May 28;26(3):687–696. doi: 10.1111/jep.13181

Unwarranted clinical variation in health care: Definitions and proposal of an analytic framework

Kim Sutherland 1, Jean‐Frederic Levesque 1,2,✉
PMCID: PMC7317701  PMID: 31136047

Abstract

Rationale, aims, and objectives

Unwarranted clinical variation is a topic of heightened interest in health care systems around the world. While there are many publications and reports on clinical variation, few studies are conceptually grounded in a theoretical model. This study describes the empirical foundations of the field and proposes an analytic framework.

Method

Structured construct mapping of published empirical studies which explicitly address unwarranted clinical variation.

Results

A total of 190 studies were classified in terms of three key dimensions: perspective (assessing variation across geographical areas or across providers); criteria for assessment (measuring absolute variation against a standard, or relative variation within a comparator group); and object of analysis (using process, structure/resource, or outcome metrics).

Conclusion

Consideration of the results of the mapping exercise—together with a review of adjustment, explanatory and stratification variables, and the factors associated with residual variation—informed the development of an analytic framework. This framework highlights the role that agency and motivation, evidence and judgement, and personal and organizational capacity play in clinical decision making and reveals key facets that distinguish warranted from unwarranted clinical variation. From a measurement perspective, it underlines the need for careful consideration of attribution, aggregation, models of care, and temporality in any assessment.

Keywords: framework, performance measurement, quality improvement, unwarranted clinical variation

1. INTRODUCTION

Unwarranted clinical variation is a topic that attracts significant attention in developed health care systems internationally.1, 2, 3, 4, 5, 6, 7 Interest in the topic is not new, however. Seminal papers by Guy,8 Codman,9 Glover,10 Wennberg and Gittelsohn,11 and Lewis12 all shaped the field of enquiry, highlighting variation in either service utilization or outcomes of health care. In the past 15 years, the work of the Dartmouth Institute has been instrumental in influencing measurement and reporting approaches in use around the world,13 catalysing the development of atlases of variation in multiple jurisdictions.1, 2, 4, 7

Clinical variation has been quantified across a wide range of acute and chronic care specialties, in primary care and hospital settings, and with regard to diagnosis, treatment, and prescribing practices.14, 15, 16, 17, 18, 19, 20 Variation has been found in almost all areas of health care where it has been looked for. A 2014 systematic review of medical practice variation in OECD countries found 836 published studies and detailed variation across regions, hospitals, and physician practices for almost every surgical field, condition, and procedure studied.21

However, despite this widespread and enduring interest in unwarranted clinical variation, the literature lacks strong conceptual frameworks to guide rigorous measurement and remediation efforts; and there are few typologies that systematically map the field.21, 22, 23 While the Dartmouth approach identifies three categories of care—namely, effective care (where variation implies some underuse of valid treatment), preference‐sensitive care (where variation implies more than one option of care is available and the exercising of patient choice), and supply‐sensitive care (where variation implies the volume of care provided is a reflection of capacity rather than patient need)—the distinction between what is warranted and unwarranted clinical variation remains poorly delineated.24, 25

This paper addresses this issue and has three main objectives. First, it seeks to describe and classify studies that explicitly refer to “unwarranted clinical variation” or “medical variation.” Second, it draws on these studies to inform the development of an analytic framework to identify factors associated with warranted or unwarranted clinical variation. Third, it discusses key issues to resolve if we are to advance the field of unwarranted clinical variation—in terms of both measurement and action to reduce it.

2. METHODS

2.1. Building a definition

In linguistic terms, variation is defined as “something that is slightly different from the usual form or arrangement.”26 Clinical refers the examination and treatment of patients—that is, focusing on patient‐provider interactions and including preventive, diagnostic, therapeutic, and supportive care. Combining these two terms, clinical variation refers to differences in health care services provided to patients that diverge from the “usual form or arrangement.” Unwarranted clinical variation goes beyond this; however, it is a values‐based concept that requires an informed judgement about the extent to which clinical variation is legitimate. It is primarily concerned with appropriateness of care—whether the right care is provided in the right way and in the right amount to address patients' needs and expectations. Accordingly, we define unwarranted clinical variation as “patient care that differs in ways that are not a direct and proportionate response to available evidence; or to the healthcare needs and informed choices of patients.”

From a theoretical perspective, this definition integrates two disparate schools of thought. First, it aligns with the positivism of evidence based medicine—which has been described as “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.”27 Second, it adopts an interpretive stance—acknowledging the importance of judgement, social context, and values in interpreting available evidence; different types of evidence and the context in which it is acted upon; and engaging with patients to make decisions in light of the available evidence. Interpretivists often align with critics of evidence‐based medicine who consider it to be simplistic and dogmatic—“synthesizing a certainty based on what is statistically probable, which, in the clinical setting, does not represent certainty at all”.28

2.2. Searching the literature

To assess the ways in which unwarranted clinical variation is characterized in research studies, a search of the PubMed database to December 2017 was conducted, using terms used to refer to “unwarranted clinical variation” including “medical practice variation”, and “clinical variation” (see Appendix A).

The search yielded 489 papers. Initial screening removed papers that were not relevant (eg, focused on genetic variation) leaving 315 for abstract review. Of these, 190 were empirical studies of unwarranted clinical variation and were included in a mapping process (listed in the supplementary file). The mapping comprised a seven‐stage process realized by the authors in an iterative manner where disagreements were assessed and resolved through discussion and deliberation:

  1. Define “a priori” key study characteristics: disease or patient group; setting (primary care, hospital, etc); unit of analysis (regions, hospitals, clinicians etc); metrics;

  2. Categorize included studies using a data extraction tool (deductive process);

  3. Identify emergent themes from studies (inductive process): statistical methods used; adjustment variables; residual/unexplained variation; scale of variation reported;

  4. Cluster themes and factors contributing to warranted and unwarranted variation;

  5. Synthesize and resynthesize concepts;

  6. Validate using reclassification of studies using the emergent criteria; and

  7. Visualize.

3. RESULTS

There are three key analytic dimensions in the UCV literature: (a) perspective (whether variation is measured across geographical areas or between providers); (b) criteria for assessment (whether measurement uses an “absolute” assessment, ie, variation from a predefined standard; or a “relative” assessment, ie, variation within a comparator group); (c) and object of analysis (whether assessment relies on process, structure/resource, or outcome metrics).

Of the 190 studies included in the review, 74% were focused on geographic variation; 90% focused on relative variation; and 73% were focused on processes of care (Figure 1). Looking at the various perspectives in combination, our initial mapping exercise identified 12 different combinations across the literature (Figure 2). The most frequently seen combination was geographic perspective/relative criteria/process metric—comprising 48% of the studies. The mapping and classification process highlighted measurement issues in each of three key dimensions in the UCV literature: perspective, criteria, and object of analysis.

Figure 1.

Figure 1

Open in a new tab

Summary of retrieved studies by analytic perspective

Figure 2.

Figure 2

Open in a new tab

Mapping of thematic combinations in the unwarranted clinical variation literature

3.1. Perspective: Geography or provider

Our review found that the variation literature is dominated by geographical‐area studies, typified by atlas publications.4, 13, 14 Geography‐based studies commonly enumerate utilization rates with differences described in terms of “x‐fold variation.” Atlases are of particular interest to policymakers, as they often reflect allocative issues and provide a broad range of indicators. There are however few attempts to explicitly distinguish warranted and unwarranted variation, or to assess levels of care, relative to patient needs, expectations, and preferences. In general, studies do not define an acceptable level of variation nor is there a way to consider multiple processes simultaneously—which limits the value of atlas‐based approaches in complex and multifaceted care pathways.

There are far fewer unwarranted clinical variation studies focused on differences across providers. Here, comparisons are made in terms of health care delivered to patients—by hospitals, units, teams, or individual professionals. More relevant to clinicians and managers, there are generally efforts made to cluster or stratify similar units for comparison and performance assessment purpose.

3.2. Criteria: Absolute or relative

When there are clear standards about best practice, it is possible to make absolute assessments of care and quantify levels of unwarranted variation. Interpretation is most straightforward when the standard is either to “always” or “never” provide a treatment or achieve an outcome in a clearly defined set of circumstances (for example, to always provide annual foot examinations to diabetic patients or never prescribe antibiotics for viral infections). Assessment is much more difficult when there is uncertainty or clinical equipoise29—that is, when there are two or more equally valid approaches to meet patient needs, and the best choice depends on how individuals (both clinicians and patients) value the risks and benefits of treatments. Absolute assessment is also difficult in situations of a dynamic and rapidly evolving evidence base where innovations are diffusing throughout a system and differences in clinicians' readiness to adopt innovations result in variation.

In contrast, relative assessment looks for variation between units and generally requires a range of data items and sophisticated analytical techniques to ensure that fair comparisons are made. Combinations of adjustment variables and factors assumed to be contributing to “residual” variation are highly heterogeneous—meaning there are few shared assumptions about what constitutes warranted and unwarranted variation.

3.3. Objects of analysis: Process, outcome, and resource‐based metrics

Only two studies included in our review used multiple metric types.30, 31 About three quarters of the studies used process metrics. These are direct measures of tests, treatments, and procedures provided to patients and reflect differences in clinical decisions. Process metrics are meaningful in terms of variation only when they are clearly linked to the evidence base and interpreted in the context of needs and expectations of patients.

This direct measurement of clinical decision making and care delivery is not always feasible however, nor is it always the most informative approach. Proxy measures or indirect measurement is often the most insightful or efficient way to gauge unwarranted clinical variation and includes resource and outcome metrics. Resource metrics focus on inputs and comparisons are made in terms of dollars or other common units that allow for assessment of different combinations of processes and models of care. Outcome metrics also provide indirect measurement of variation in clinical care and are perhaps the most salient of the metrics, focusing on the consequences of unwarranted clinical variation. Like resource measures, outcome measures can capture complex care pathways, bundles, and the multiple processes inherent within them—one outcome measure can be a reflection of dozens of discrete care processes.

3.4. Identifying themes that help delineate warranted and unwarranted clinical variation

The retrieved literature identified various factors that have been used to categorize variation as either warranted or unwarranted and hence key factors to consider in analyses. While a range of adjustment or stratification variables could be relevant for all variation analyses, the literature suggests the use of these covariates or stratifying approaches is predominantly used in the assessment of relative variation between organizations in resource, process, or outcome metrics. Our inductive process of identifying themes regarding what is considered to constitute unwarranted and warranted clinical variation revealed that within these three metric subgroups of process, outcome, and resource measures, there were clear types of adjustment variables, explanatory and stratification variables, and factors associated with residual variation (Table 1).

Table 1.

Adjustment variables, stratification, and residual variation factors

Metric Adjustment Variables, Legitimate Variation—Warranted No. of Study Strata, explanatory, and residual variables—Unwarranted No. of Study

Process (eg, treatments, surgical procedures, diagnostic tests, hospital admissions)

 Clinically relevant patient characteristics (eg, age, sex) 68 Organizational characteristics (eg, size, staffing, equipment) 24
Diagnostic uncertainty/complexity 2 Regional characteristics (eg,) rurality, deprivation) 17
Equivocal evidence 1 Non‐clinically relevant patient characteristics (eg, socio‐economic status 10
Clinician characteristics 8
Change over time 6
Clinician preferences and payment 4
Outcome (eg, mortality, readmission, unplanned hospitalization (ACSC), rapid decline, ED visit, failure to rescue, bacteraemia, lower extremity amputation) Clinically relevant patient characteristics 25 Countries/regions 9
Patient socioeconomic status 7
Time 4
Resources/staff availability 4
Access to appropriate care 3
Organization (bed availability) 2
Evidence‐based processes 2
Patient race/ethnicity 2
Patient behaviour 1
Cost of interventions 1
Length of stay 1
Resource (eg, cost, length of stay) Clinically relevant patient characteristics 7 Region 4
Alternative procedures 3
Alternative settings (eg, outpatient, hospital in the home, inpatient) 1
Compliance with best practice standards 1
Hospital characteristics/organization/staffing levels 3
Insurance status of patients 1
Sex of clinician 1
Race ethnicity of patient 1
Open in a new tab

4. DISCUSSION

Following synthesis of the literature, and a combination of deductive and inductive inquiry, an empirically derived analytic framework that delineates warranted and unwarranted variation emerged (Figure 3). The model highlights the important roles that patients' and clinicians' agency, scientific and clinical evidence, and personal and organizational capacity play in shaping variation—and how these factors should be considered in assessing if variation is warranted or unwarranted.

Figure 3.

Figure 3

Open in a new tab

Schematic of warranted and unwarranted clinical variation

4.1. The analytic framework—Warranted variation

In this analytic framework, “agency” encapsulates issues of motivations and for whom clinical decisions are made, focusing particularly on questions about whose needs and expectations drive clinical decisions.

From this agency perspective, clinical care must vary if it is to respond to patients' needs and expectations. Services should be tailored to patients' physical, social, and psychological requirements. Clinicians and clinical teams increasingly seek to provide care that is patient‐centred—eliciting patient preferences, supporting informed choice, and engaging patients in decisions about their care. Value and judgement are used to tailor clinical decisions and actions to the social and psychological needs of patients.32 This means that the best care for one patient will not be the best care for all patients.

Considering patients' legitimate expectations about care and consent to various options for care is therefore key to assessing clinical variation. Increasingly, with the advance in personalized medicine and shared decision making, clinical variation should be expected and will be warranted if it is based on unbiased discussions and informed consent.

“Evidence,” in our analytic framework, focuses on whether clinical decisions align and resonate with the extant knowledge base and considers questions about the basis on which decisions are made. Not all health problems have a unique clinical solution, and in the context of a paucity of evidence about the effectiveness of an intervention, homogeneity of clinical practice may provide a false reassurance and prevent the emergence of clinical innovations. Similarly, the gradual emergence and testing of innovations may create temporary clinical variation. Hence, an explicit and critical appraisal of the nature of the evidence base in any clinical variation assessment is crucial.

From this evidence perspective, variation can be warranted if following appraisal, evidence‐based recommendations are adapted in order to respond to salient contextual cues. Variation can also be warranted where there is uncertainty within the expert clinical community about a preferred test or treatment.33 Similarly, as the knowledge base about clinical care is constantly evolving, the concept of the “best” care is dynamic. The introduction of a new treatment, test, or model of care inevitably takes time to be adopted or implemented all across a health care system and the process of diffusion results in variation.34 In innovation terms, variation is warranted and is often a positive feature—bringing with it opportunities to compare ways to provide care—so that the best option can be adopted into routine practice.

“Personal and organisational capacity” focuses on whether clinicians are able to provide care in the way they seek and includes questions about how decisions are enabled and supported. These relate especially to when variation focuses on clinicians and the need to consider any organizational constraints they face. These issues are further explored in the following sections.

From within the capacity perspective, where there are differences in skill‐mix or types of resources available between different local areas or organizations, variation in care processes can be a reflection of adaptation. Clinicians provide services in different ways, using different models of care within different circumstances—and as long as patients achieve equivalent outcomes, this variation can be regarded as warranted. In instances where there are many acceptable and effective ways to care and cure, where there are no guidelines or where guidelines allow for multiple approaches, clinicians can capitalize on their particular skill sets to provide care. Where there are unanticipated complexities, such as suddenly deteriorating patients, clinicians act as expert problem solvers, responding in real time to developing emergencies that will differ from most other routine care—not unwarranted variation but an example of desirable and appropriate variation.

4.2. The analytic framework—Unwarranted variation

Our model also identifies six key categories of unwarranted clinical variation, organized across the same three perspectives of agency, evidence, and capacity. Considering variation in terms of agency, if decisions are made on the basis of non clinically relevant patient characteristics such as age, gender, race, or socio‐economic status or where insufficient information is provided to patients to support properly informed and shared decision making, variation can be considered to be unwarranted.

More starkly perhaps, variation is unwarranted where clinicians' preferences or financial needs take precedence over the evidence‐base or patient interests. The literature features discussions and empirical examples about variation shaped by providers' expectations (eg, scheduling of procedures at certain times for the sake of clinicians' convenience and overuse of certain procedures for financial benefit).35 This can result in tests, procedures, and treatments which have been shown to be ineffective, continuing to be provided to patients in ways that are wasteful of resources and place them at unnecessary risk.36 Variation is also unwarranted if it is a result of responsibility for patient‐care, particularly for complex multimorbid patients, being parsed and resulting in episodes of disjointed and incomplete care.37

From an evidence perspective, variation is deemed to be unwarranted when practice is clearly at odds with the available knowledge base.38, 39, 40 Unwarranted variation can also stem from “indication creep,” where the use of a procedure or treatment grows beyond the original patient group in which it was trialled and shown to be valuable.14 In an interesting twist, there are cases where a lack of variation can be unwarranted. An illustrative study by Tang et al19 measured variability in antipsychotic prescribing patterns among psychiatrists and found that less‐expert providers had more homogeneous prescribing behaviours with some physicians relying heavily on a small number of agents, where appropriately tailored care would elicit different treatment regimens. So variation is unwarranted if there is unjustified deviation from the evidence base—that is, evidence is applied in variable ways despite a lack of key contextual confounders, or evidence is applied in a way that is not responsive to context. This element of unwarranted variation resonates with debate about the evidence‐based medicine movement—which for some threatens to overemphasize the importance of general research in routine clinical practice, devaluing the role of clinical judgement.28

From a capacity perspective, variation is unwarranted if it is a result of differences in the level of training, competency, and technical proficiency of providers 41, 42 or of limitations in clinicians' ability to resolve uncertainty.43 Now, more than ever, the provision of reliable and resilient care to ensure patients' safety is seen as a minimum requirement for health care delivery systems and is not something that can vary according to where patients live or where they are treated.

Unwarranted clinical variation can also be a result of local delivery systems—resulting in some clinicians being unable to provide certain elements of care because of resource constraints. For example, variation in surgical waiting times and surgical outcomes can be a result of differences in resourcing across hospitals. Conversely, patterns of resourcing can also promote unnecessary activity, for example, where additional availability of resources results in greater propensity to treat or admit patients to hospital. This notion of “build it and they will come” underpins the concept of supplier‐induced demand.13

4.3. Acknowledging and tackling the complexity of measuring unwarranted clinical variation

Our mapping exercise showed that clinical variation studies predominantly focus on process metrics; however, their ability to determine the extent to which measured variation is unwarranted has, to date, been limited. This may be because we need a way to more reliably identify and quantify key factors in play. For example, if there is evidential equipoise, the measurement of variation of a single therapeutic option will be misleading; if there is substitution of services to respond to different contexts, or if there is variation that is a reflection of diffusing innovations or changing models of best practice, processes are not strong measures of unwarranted variation.

While outcome‐based analyses provide a means to overcome many of these issues, appropriate statistical adjustment is key. Underadjustment will lead to an overinterpretation of variation—suggesting that there are opportunities to improve care when in reality, much of the variation may be warranted. Overadjustment represents the corollary case—“adjusting away” the impact of factors that, if addressed, could reduce meaningful variation—masking the impact of modifiable factors that should be tackled to improve care.

The measurement of unwarranted clinical variation is vulnerable to a range of failures in analytic design. Limits of measurement to be acknowledged and mitigated include difficulties in interpreting variation in small units, the recognition of “normal variation” and distinguishing it from “special cause variation”44, and regression to the mean.45 While these concerns are well described in measurement and analytic literature, we need greater cognisance of their implications when interpreting and critically appraising measures of variation.

In addition to such well‐established analytic concerns, there were four key considerations that emerged from our mapping exercise—attribution, aggregation, models of care, and temporality—that are fundamental to enhancing our understanding of variation.

When comparing variation across individual clinicians or units, it is essential to distinguish between contextual factors that are outside direct control at that level and those that are tractable or amenable to change. This means, for example, that while geography‐based analyses can reveal disparities, the charge of unwarranted clinical variation cannot necessarily be applied to individual decision makers. Patients living in lower socio‐economic status (SES) areas are often reported to have worse outcomes in readmission or mortality than patients living in higher SES areas; however, a doctor at an inner city practice may be consistent in his or her decision making regardless of patients' SES. Context—either organizational or more broadly in terms of wider determinants of health—constrains clinicians' ability to provide care that delivers equal outcomes to all patients. Taking account of context does not diminish the unwarranted variation—it still exists—but it reflects a system or allocative issue rather than an issue with individual clinical decision makers.

The assessment of the unwarranted nature of clinical variation therefore requires a consideration of the nexus of control. In situations where organizational context is at the root of variation, and clinicians are constrained by structures, regulations, or intractable resource constraints, any resultant variation in the practice and decision making of those clinicians when compared with other clinicians in different organizational contexts can be considered to be reasonable rather than unwarranted. This does not mean that such variation is acceptable—it is clearly featured in the framework of unwarranted variation but as a reflection of policy and management, rather than clinical decisions.

Furthermore, inappropriate analytic choices regarding the level of analysis can confound assessment. Hospitals may appear to be similar to each other despite significant variation within hospitals.46 Geographical analyses are especially prone to aggregation issues capturing multiple providers in a single area. Overuse or underuse can be occurring simultaneously and be masked by aggregation.

In addition, variation in provision of care may be warranted if different modalities are used for care delivery (eg, yearly eye exams for diabetics may be provided by ophthalmologists or optometrists and rates of eye exams may vary between regions when looked by types of providers separately but not vary in terms of reception of eye exam by patients between regions). However, variation in provision of care may be unwarranted if such allocative decisions prevent some patients receiving the care they need and choose (eg, yearly exams for diabetics may vary if neither the ophthalmologist nor the optometrist options are available locally). This raises the interesting conceptual question of whether variation in access to care should be considered as unwarranted clinical variation. Fundamentally, unwarranted clinical variation focuses on appropriateness of care—measured directly by care process measures or indirectly through resource use and outcome measures. For access issues, we cannot generally attribute unmet patient needs to individual clinicians. However, from a system perspective, a lack of access to appropriate care can be considered to be unwarranted clinical variation.

Finally, the issue of temporality adds further complexity to the assessment of unwarranted clinical variation. Provision of care may vary if the research base is dynamic and innovations are emerging and being tested. The diffusion of innovations takes time—however, after sufficient time has been allowed for uptake and practice change, continuing variation becomes unwarranted. So time is also important as standards of care are continually changing and innovations constantly emerging. In circumstances where the evidence about current care is either equivocal or suggests poor effectiveness of care, variation may be warranted to ensure innovation can emerge, be evaluated, and eventually disseminated at scale.

4.4. Limitations

Our model identifies six key categories of unwarranted clinical variation. It does not however provide guidance about which category investigators of clinical variation should focus upon—should all categories be explored simultaneously? If not, which category to choose? While this undoubtedly is a limitation, prompting systematic consideration of all six categories, and deliberate choices about measurement approaches in light of those considerations would represent a significant step forward.

As has been acknowledged by other researchers,23, 24 despite a considerable and enduring interest in measurement of unwarranted clinical variation, there are few conceptually based frameworks available. This means that there was very little theoretical groundwork on which to build. This paper does not present a fully elucidated theory but represents a step forward in seeking to strengthen the conceptual underpinnings and contribute to academic discourse.

Therefore, there are many unanswered questions—such as when is evidence strong enough to require action? How should we resolve conflicting sources of evidence and patient preferences? How do we place value on different elements of the model? How do we reconcile population based science with personalized medicine? Notwithstanding these limitations, this paper aims to provide a critique of current literature around the concept of unwarranted clinical variation and represents a step towards the development of a rounded theoretically based conceptual model.

5. CONCLUSION

Identifying, quantifying, and reducing unwarranted clinical variation promises to deliver a range of benefits to health care systems and to individual patients—more reliable provision of indicated and evidence‐based care, reduction in wasteful or unnecessary care, improved safety of care, greater system efficiency, and better patient outcomes. It has the potential to move us beyond a naïve view that only patients' needs and provider preferences drive delivery of care.

We know that almost all studies that look for variation find it. This ubiquity means that we need to develop sophisticated ways to prioritize measurement efforts and to more clearly distinguish warranted from unwarranted variation. Realizing potential gains is far from straightforward however. While there has been a great deal of sustained interest in the notion of unwarranted clinical variation internationally, there are few conceptual frameworks to guide investigation, systematic measurement, and change management processes. The complexity of unwarranted clinical variation is legion: Availability of evidence and contextual factors both affect how we judge variation between providers; equipoise or equivocal evidence makes variation uninterpretable; and variation in resourcing can make variation unattributable to the individual clinician. Delineating and defining unwarranted clinical variation place a heavy burden on measurement efforts.

In addition, the current lack of clarity around how best to measure unwarranted clinical variation can result in an overemphasis on ranges reported in atlases and propels the field towards an ever increasing focus on adjusting for comorbidities and health factors, without discussing other potential sources of variation. The predominant measurement approach—one that focus on enumerating and reporting relative process measures—is for the most part limited in its capacity to guide efforts to reduce unwarranted variation and improve care. All providers could be performing poorly and little variation revealed; there could be a mix of overuse and underuse of appropriate care; measurement may not be able to capture contextual factors that shape our judgement about the level of appropriateness in variation.

However, many health care systems are developing more sophisticated approaches to assessing unwarranted clinical variation—in the United Kingdom, the Getting it Right First Time programme encapsulates peer‐led deep dives, tailored feedback, and support for implementation.47 More broadly, there is renewed interest in audit and feedback48—engagement of clinical decision makers in data analyses, fair comparisons, attribution and interpretation, and subsequent quality improvement. These efforts point to a way forward in a highly complex field.

The elements of warranted and unwarranted variation are interrelated and are highly sensitive to context. This means that it is difficult to measure quantitatively using administrative data and assessment requires more nuance and reflexivity, pointing us towards a mixed methods approach that is sensitive to uncertainty, social context, sense‐making, and scientific evidence.

Health care systems are dynamic and complex. Health care is unique in terms of the extent to which it is grounded in science but indelibly shaped by social context and values. It is of paramount importance that we start to look in a more informed and sophisticated way at variation in clinical care and patient outcomes in order to distinguish when variation is expected or desirable and when it is unwarranted. Only then can we start to focus on reducing the unwarranted and potentially harmful variation through quality and safety assurance processes and improvement and clinical innovation programs.

CONFLICT OF INTEREST

Both authors are full time employees of NSW Health. There are no external funding sources. There are no conflicts of interest to disclose.

Supporting information

Data S1 Mapped studies

Click here for additional data file. (55.7KB, docx)

ACKNOWLEDGEMENTS

This work was supported through a HARC fellowship from the Sax Institute, NSW. Preliminary findings were presented on a scientific poster at the World Hospital Congress, Brisbane Australia in October 2018. We thank our colleagues in the NSW Health Unwarranted Clinical Variation Taskforce for their insight and discussions.

APPENDIX A.

Schematic of search strategy for empirical studies of unwarranted clinical variation Inline graphic

Sutherland K, Levesque J‐F. Unwarranted clinical variation in health care: Definitions and proposal of an analytic framework. J Eval Clin Pract. 2020;26:687–696. 10.1111/jep.13181

REFERENCES

  • 1. Australian Commission for Quality and Safety in Health Care. The Third Australian Atlas of Healthcare Variation Sydney: ACQSHC, 2018. [Google Scholar]
  • 2. Organisation for Economic Cooperation and Development Geographic Variations in Health Care. Paris: OECD; 2014. [Google Scholar]
  • 3. Institute of Medicine Crossing the Quality Chasm. Washington DC: IOM; 2001. [Google Scholar]
  • 4. NHS Atlas of Variation, London, 2015.
  • 5. Love T, Ehrenberg N. Variation and improving services: analysing and interpreting variation. Health Quality and Safety Commission New Zealand, 2014.
  • 6. Mercier G, Georgescu V, Bousquet J. Geographic variation in potentially avoidable hospitalizations in France. Health Affairs (Project Hope). 2015;34(5):836‐843. [DOI] [PubMed] [Google Scholar]
  • 7. Bernal‐Delgado E, Garcia‐Armesto S, Peiro S. Atlas of variations in medical practice in Spain: the Spanish National Health Service under scrutiny. Health Policy (Amsterdam, Netherlands). 2014;114(1):15‐30. [DOI] [PubMed] [Google Scholar]
  • 8. Guy WA. On the mortality of London hospitals: and incidentally on the deaths in the prisons and public institutions of the Metropolis. Journal of the statistical Society of London. 1867;30(2 (Jun., 1867)):293‐322. [Google Scholar]
  • 9. Codman EA. Study in hospital efficiency: as demonstrated by the first case report of the first five years of a private hospital. Boston, MA: Thomas Todd Co; 1918. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10. Glover JA. The incidence of tonsillectomy in school children. Proc R Soc Med. 1938;31(10):1219‐1236. [PMC free article] [PubMed] [Google Scholar]
  • 11. Wennberg J, Gittelsohn A. Small area variations in health care delivery. Science. 1973;182(4117):1102‐1108. [DOI] [PubMed] [Google Scholar]
  • 12. Lewis C. Variations in the incidence of surgery. NEJM. 1969;281:880‐884. [DOI] [PubMed] [Google Scholar]
  • 13. Wennberg JE. Time to tackle unwarranted variations in practice. BMJ (Clinical Research Ed). 2011;342:d1513. [DOI] [PubMed] [Google Scholar]
  • 14. Australian Commission for Quality and Safety in Health Care. The Second Australian Atlas of Healthcare Variation Sydney: ACQSHC, 2017.
  • 15. McCulloch P, Nagendran M, Campbell WB, et al. Strategies to reduce variation in the use of surgery. Lancet. 2013;382(9898):1130‐1139. [DOI] [PubMed] [Google Scholar]
  • 16. Appleby J, Raleigh V, Frosini F, Bevan G, Gao H, Lyscom T. Variations in Healthcare: The Good, the Bad and the Inexplicable. London: King's Fund; 2011. [Google Scholar]
  • 17. Bailie RS, Si D, Connors CM, et al. Variation in quality of preventive care for well adults in indigenous community health centres in Australia. BMC Health Serv Res. 2011;11(1):139. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Sturm E, Piersma FE, Tanner MS, Socha P, Roberts EA, Shneider BL. Controversies and variation in diagnosing and treating children with Wilson disease: results of an international survey. J Pediatr Gastroenterol Nutr. 2016;63(1):82‐87. [DOI] [PubMed] [Google Scholar]
  • 19. Tang Y, Chang CC, Lave JR, Gellad WF, Huskamp HA, Donohue JM. Patient, physician and organizational influences on variation in antipsychotic prescribing behavior. J Ment Health Policy Econ. 2016;19(1):45‐59. [PMC free article] [PubMed] [Google Scholar]
  • 20. Davis P, Gribben B, Lay‐Yee R, Scott A. How much variation in clinical activity is there between general practitioners? A multi‐level analysis of decision‐making in primary care. J Health Serv Res Policy. 2002;7(4):202‐208. [DOI] [PubMed] [Google Scholar]
  • 21. Corallo AN, Croxford R, Goodman DC, Bryan EL, Srivastava D, Stukel TAA. Systematic review of medical practice variation in OECD countries. Health policy (Amsterdam, Netherlands). 2014;114(1):5‐14. [DOI] [PubMed] [Google Scholar]
  • 22. Harrison R, Manias E, Mears S, Heslop D, Hinchcliff R, Hay L. Addressing unwarranted clinical variation: a rapid review of current evidence. J Eval Clin Pract. 2019;25(1):53‐65. [DOI] [PubMed] [Google Scholar]
  • 23. Schang L, Morton A, DaSilva P, Bevan G. From data to decisions? Exploring how healthcare payers respond to the NHS Atlas of Variation in Healthcare in England. Health Policy (Amsterdam, Netherlands). 2014;114(1):79‐87. [DOI] [PubMed] [Google Scholar]
  • 24. Mercuri M, Gafni A. Medical practice variations: what the literature tells us (or does not) about what are warranted and unwarranted variations. J Eval Clin Pract. 2011;17(4):671‐677. [DOI] [PubMed] [Google Scholar]
  • 25. Peabody JW, Hauck LD. Physicians' variation in care: the practical balance of warranted versus unwarranted variation. Crit Care Med. 2017;45(12):e1297‐e1298. [DOI] [PubMed] [Google Scholar]
  • 26.Cambridge dictionary [online] https://dictionary.cambridge.org/dictionary/english/variation
  • 27. Sackett D, Rosenberg W, Gray J, Haynes B, Richardson W. Evidence‐based medicine: what it is and what it isn't. BMJ. 1996;312:71‐72. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28. Polychronis A, Miles A, Bentley P. Evidence‐based medicine: reference? Dogma? Neologism? New orthodoxy? J Eval Clin Prac. 1996;2:1‐3. [DOI] [PubMed] [Google Scholar]
  • 29. Cook D, Thompson JE, Habermann EB, et al. From ‘solution shop’ model to ‘focused factory’ in hospital surgery: increasing care value and predictability. Health Aff (Project Hope). 2014;33(5):746‐755. [DOI] [PubMed] [Google Scholar]
  • 30. Partington A, Chew DP, Ben‐Tovim D, Horsfall M, Hakendorf P, Karnon J. Screening for important unwarranted variation in clinical practice: a triple‐test of processes of care, costs and patient outcomes. Aust Health Rev. 2017;41(1):104‐110. [DOI] [PubMed] [Google Scholar]
  • 31. Krumholz H, Nuti SV, Downing NS, Normand SL, Wang Y. Mortality, hospitalizations, and expenditures for the Medicare population aged 65 years or older, 1999‐2013. JAMA. 2015;314(4):355‐365. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32. Mercuri M, Sherbino J, Sedran R, Frank J, Gafni A, Norman G. When guidelines don't guide: the effect of patient context in management decisions based on clinical practice guidelines. Acad Med. 2015;90:191‐196. [DOI] [PubMed] [Google Scholar]
  • 33. Freedman B. Equipoise and the ethics of clinical research. N Engl J Med. 1987;317(3):141‐145. [DOI] [PubMed] [Google Scholar]
  • 34. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581‐629. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35. Saini V, Garcia‐Armesto S, Klemperer D, et al. Drivers of poor medical care. Lancet. 2017;390(10090):178‐190. [DOI] [PubMed] [Google Scholar]
  • 36. Elshaug AG, Watt AM, Mundy L, Willis CD. Over 150 potentially low‐value health care practices: an Australian study. Med J Aust. 2012;197(10):556‐560. [DOI] [PubMed] [Google Scholar]
  • 37. Feufel MA. How to uncover sources of unwarranted practice variation: a case study in emergency medicine. Qual Health Res. 2018;28(9):1486‐1498. [DOI] [PubMed] [Google Scholar]
  • 38. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ (Clin Res ed). 1996;312(7023):71‐72. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39. Hunter DJ. Evidence‐based policy and practice: riding for a fall? J R Soc Med. 2003;96(4):194‐196. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40. Guyatt G, Voelker R. Everything you ever wanted to know about evidence‐based medicine. JAMA. 2015;313(18):1783‐1785. [DOI] [PubMed] [Google Scholar]
  • 41. Offerhaus PM, Geerts C, de Jonge A, Hukkelhoven CW, Twisk JW, Lagro‐Janssen AL. Variation in referrals to secondary obstetrician‐led care among primary midwifery care practices in the Netherlands: a nationwide cohort study. BMC Pregnancy Childbirth. 2015;15:42. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42. Mayer M, Naylor J, Harris I, et al. Evidence base and practice variation in acute care processes for knee and hip arthroplasty surgeries. PloS One. 2017;12(7):e0180090. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43. Lutfey KE, Link CL, Marceau LD, et al. Diagnostic certainty as a source of medical practice variation in coronary heart disease: results from a cross‐national experiment of clinical decision making. Med Decis Making. 2009;29(5):606‐618. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44. Lilford R, Mohammed MA, Spiegelhalter D, Thomson R. Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. Lancet. 2004;363(9415):1147‐1154. [DOI] [PubMed] [Google Scholar]
  • 45. Morton V, Torgerson D. Effect of regression to the mean on decision making in health. BMJ. 2003;326:1083‐1084. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46. Mercuri M, Birch S, Gafni A. Using small area variations to inform healthcare service planning: what do we “need” to know? J Eval Clin Pract. 2013;19:1054‐1059. [DOI] [PubMed] [Google Scholar]
  • 47. NHS Providers. The Getting It Right First Time Programme. 2018.
  • 48. Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;(6). [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Data S1 Mapped studies

Click here for additional data file. (55.7KB, docx)

Articles from Journal of Evaluation in Clinical Practice are provided here courtesy of Wiley

RESOURCES