The Intended and Unintended Consequences of Clinical Guidelines

Abstract

Objectives

First, we examine whether clinical guidelines, designed to improve health care and reduce disparities in clinical practice, are achieving their intended consequences. Second, we contemplate potential unintended consequences of clinical guidelines.

Method

As part of a factorial experiment we presented primary care doctors (n=192) with clinically authentic vignettes of a “patient” with already diagnosed diabetes with an emerging foot neuropathy. Their proposed clinical actions were compared with established practice guidelines for this clinical situation.

Results

After establishing the existence of consistent socioeconomic disparities in the proposed management of the case presented, we found that reported use of practice guidelines had no measurable effect towards their reduction (one intended consequence). However, the reported use of practice guidelines appeared to precipitate more clinical actions, without eliminating documented disparities.

Conclusions

Consistent with other research we find clinical practice guidelines are not producing a principal intended result, and may even produce unintended consequences.

INTRODUCTION

Worrisome health care variations have been repeatedly documented over the past 30 years. The most important work was undertaken by Wennberg and colleagues, culminating in the atlas depicting the magnitude of healthcare variations in the United States (US) [1]. Over time the more neutral statistical description of “variations” became synonymous with the notion of “disparities”—a term which connotes social injustice and motivates clinical and health policy actions towards some appropriate redistribution. Practice variation has been linked to patient gender [2], race [3, 4], age [5, 6], and to the focus of this paper, socioeconomic status (SES) [7]. SES is associated with both a patients' risk of developing disease, [8, 9] including diabetes [10–12], and also a physicians' decision-making. Consequently, lower social class patients can experience inequality at two levels [7, 13].

Having thoroughly documented variations in clinical practice, the focus shifted to the development of initiatives which offered the potential for their reduction: clinical or practice guidelines were considered one of the most promising approaches [14–16] Quality assurance experts and professional organizations developed lists of items that a physician ought to take when presented with a likely case of some disease: these comprised “ideal” care at the patient level. It was presumed that close adherence to these lists (guidelines) by individual practitioners encountering individual patients would eventually produce desirable standardization of medical care and eventually a reduction in the health care variations. The intended consequence of guidelines, indeed the raison d’être presented to gain provider acceptance, was the improvement in the quality of care and ultimately the reduction or elimination of healthcare variations.

Using data from a carefully designed factorial experiment we first question whether clinical guidelines are achieving a principal intended purpose-the reduction or elimination of healthcare disparities associated with particular patients. The experimental design of our study provides an opportunity to examine variability in physician decision making when exactly the same signs and symptoms are presented by different “patients”.

Second we examine the notion of unintended consequences. Beginning with the work of Max Weber [17], the notion of unintended consequences has a strong roots in the social sciences—it highlights an important distinction between the stated purpose or intent of some social action (e.g., introduction of clinical guidelines to improve health care and reduce disparities) and their generally unrecognized but objective functional consequences. To our knowledge, the theory of unintended consequences has never been applied to the development and introduction of clinical guidelines. This paper contemplates two unintended consequences of clinical guidelines in the US: a) mandating patterns of performance appears to contribute to increases in healthcare costs;[18, 19] and b) codification of clinical practice and “formulaic care” may create conditions conducive to the replacement of primary care physicians [20].

METHODS

We conducted a factorial experiment using clinically authentic video vignettes. Filmed scenarios are now widely used during medical education and for board certification and increasingly by health services researchers who find them superior to alternative approaches (standardized patients and written scripts). In contrast to written scripts, filmed scenarios permit potentially relevant nonverbal indicators (for example, race, age and weight) to be embedded in the script. Each filmed vignette presented exactly the same essential diagnostic signs and symptoms of a diabetic patient presenting with emerging foot neuropathy: described as burning on the bottoms of the feet that goes up one ankle, as intermittent and hard to localize. All patients were moderately overweight (embedded non-verbal cue), with an HbA1c of 6.9. Study outcomes focused on a range of guideline-based clinical decisions recommended for the case presented. Diabetes was selected for study because: (1) it is a major cause of morbidity and mortality in the US and increasing in prevalence [21]; (2) variation has been reported for its diagnosis and management, particularly by SES; and (3) clinical guidelines for the management of diabetes are available as one operational standard for the assessment of the appropriateness (or quality) of physician decisions. Our research methods have been reported in detail elsewhere [22] and are briefly summarized below.

Experimental Stimuli (The Clinical Vignette)

One version of the vignette, varying by age (35 or 65), race/ethnicity (Black, Hispanic, or White), socioeconomic status (janitor or lawyer), and gender (male or female), was shown to each of the physicians recruited as subjects for the experiment. The case was developed with the input of experienced clinicians who regularly treat patients with diabetes and confirmed the accuracy of the clinical content. Since consultants to our studies advised that real patients seldom present as pure “text-book cases”, we embedded several minor distractions in the presentation to increase the clinical authenticity of the scenario: a single high blood pressure reading (145/98), the patient mentioned they were on blood pressure medication but were not ideally compliant, and the patient expressed concern about high blood pressure. Professional actors and actresses were trained under experienced physician supervision to realistically portray the patient.

Experimental Subjects (Physicians)

We recruited a total of 192 primary care physicians, stratified according to gender and level of clinical experience. To be eligible for selection physicians had to: (a) be primary care physicians; (b) be trained at an accredited medical school in the US (since this was a subset of an international study); (c) be currently providing clinical care at least half time; (d) have ≤12 years clinical experience (graduated from medical school between 1993–99) or ≥22 years experience (graduated between 1969–83). The dichotomy on years of clinical experience was created in order to preserve orthogonality on all patient and physician factors, and to ensure clear separation on clinical experience. Eligible physicians were purposively selected from throughout New Jersey, New York, and Pennsylvania to equally fill four design cells to provide un-confounded estimates on gender and level of experience. Screening telephone calls were conducted to identify eligible subjects and an hour-long, in-person interview was scheduled (at which time written informed consent was obtained).

As part of a larger factorial experiment we have equal numbers of Young/Old, Black/Hispanic/White, Upper/Lower SES, and Male/Female patients and equal numbers of Male/Female and More/Less experienced physicians. The results according to these patient/provider characteristics will be published separately. In this paper we focus on SES disparities which have been consistently noted in routine diabetes care [13, 23].

Measures

Immediately after viewing the vignette, the physician subjects completed a semi-structured interview. This interview included questions concerning how they would manage the “patient” in their everyday clinical practice, including their physical examinations/test ordering (i.e. blood pressure, fundoscopic exam, HbA1c), lifestyle recommendations (i.e. perform foot self exams, advice about diet and exercise), and referrals. The outcomes of interest, the final list of clinical actions, was developed from well established clinical practice guidelines for this situation, specifically, the American Diabetes Association (ADA) guidelines, the Agency for Healthcare Research and Quality (AHRQ) guidelines [24, 25].

Physicians’ use of clinical practice guidelines was measured by three items obtained from a self-administered questionnaire:

1. Do you have access to clinical guidelines for treating and managing diabetes? (Aware)

2. Do you use clinical guidelines in the management of your diabetic patients in your practice? (Adopt)

3. Did your knowledge of clinical guidelines affect the decisions you just made of the cases we just showed you? (Reported Adherence)

Each question had a yes/no response, if physicians did not respond affirmatively to the first question (awareness of guidelines), then the following two questions were skipped (adoption and reported adherence to guidelines) and “no” responses were imputed. Physicians were said to “adhere” to guidelines if they reported: a) having access to guidelines; b) using guidelines in the management of their patients; and c) that their knowledge of the guidelines affected their decisions for the vignette patient. These questions are based on the awareness-to-adherence model of guideline compliance introduced by Pathman and colleagues [26].

Statistics

Bivariate data analyses were conducted using chi-square or Fisher’s exact test for categorical variables. Logistic regression was used for multivariate models adjusting for patient and physician characteristics. Covariates included patient race/ethnicity, age, gender, and physician experience and gender. To estimate the effects of guideline adherence on disparity reduction, odds ratios and 95% confidence intervals are provided. A reduction in the odds ratio of 0.05 or greater (−) is considered a reduction in the disparity. A 0.05 increase in the odds ratio (+) indicates the disparity was increased by guideline adherence. If, after controlling for self-reported guideline adherence the SES effect remained unchanged or was increased, then this would suggest that reported guideline adherence does not reduce disparities by SES. Due to the large number of comparisons, there are potential problems with multiple testing. Recognizing this issue, we focus on the noteworthy consistency of patterns, rather than on isolated, statistically significant results. All analyses were conducted using SAS 9.1 (Cary, NC).

RESULTS

Adherence to Guidelines

Almost 75% of our sample reported that they had access to, used, and were influenced by clinical guidelines in their decision making (“adhere” 72.4%; 139/192). Of these physicians, 43% (n=60) explicitly specified that they use the American Diabetes Association guidelines, which is the main source of our compiled list of outcomes.

Socioeconomic Disparities

There were important, consistent differences in the actions physician’s reported they would take depending on the “patient’s” socioeconomic status (SES) (Figure 1). Specifically, physicians were more likely to perform the following for upper SES patients: ask questions regarding their medical history (adherence with diet, adherence with medications, exercise habits, previous neuropathy, and history of eye disease); recommend guideline suggested examinations (fundoscopic eye exam, check for ulcers, monofilament/vibration, foot pulses); order guideline recommended tests (fasting lipid, liver function, microalbuminuria, thyroid stimulating hormone); give referrals (ophthalmologist, podiatrist); and give advice (foot self exams, diet). Although many of the differences in physician reported actions do not reach statistical significance, there is noteworthy consistency in the direction of the resulting differences. In 17 out of the 20 actions listed in Figure 1, physicians reported they were more likely to perform the action in patients of Upper SES than in patients of Lower SES. Physicians were more likely to recommend that lower SES patients get a blood pressure exam, HbA1c, and were more likely to give advice about exercise. In sum, our experimental results show that lower SES “patients” are consistently less likely to receive components of care which clinical guidelines suggest are important in the management of the case presented.

Figure 1.

Components of a comprehensive diabetic examination for diabetic patients presenting with symptoms suggestive of emerging foot neuropathy by Socioeconomic Status (SES)

The Intended Consequences of Clinical Guidelines

Having established that there are consistent SES disparities in the management of the case presented, we turn to whether the use of clinical guidelines has any apparent impact (whether disparities are reduced, magnified, or unchanged). Logistic regression analyses are presented in Table 1. Column 1 shows the odds ratios for upper SES patients versus lower SES patients adjusted for patient (race/ethnicity, age, gender) and physician (experience, gender) characteristics ordered by the strength of the odds ratio. An odds ratio > 1 indicates that a disparity exists (with physicians performing the action more often for upper SES patients than lower SES patients). Column 2 shows the odds ratios for upper SES patients versus lower SES patients adjusted for guideline adherence, for patient (race/ethnicity, age, gender) and physician (experience, gender) characteristics. The difference between odds ratios were calculated (Not adjusted for guidelines – Adjusted for guidelines). A decrease of 0.05 in the odds ratio was considered a reduction of the disparity. There was a 0.05 decrease in the reduction of the disparity on only three actions: questions on previous neuropathy, questions on history of eye disease and a test for liver function. An increase of 0.05 in the odds ratio was considered a magnification of the disparity. There was a 0.05 increase in the disparity on SES on three actions: fundoscopic eye exam, referrals to an ophthalmologist, and referrals to a podiatrist. In other words, use of clinical guidelines was as likely to increase disparities as they were to decrease disparities. In 14 out of the 20 actions listed, reported adherence to guidelines had no impact on disparities by SES.

Table 1.

Odds Ratios for components of a comprehensive diabetic examination (Upper Socioeconomic Status vs. Lower Socioeconomic Status) ¹

	Upper SES vs. Lower SES 2		Upper SES vs. Lower SES Guideline Adjusted 3		Direction of Guideline Effect4
	Odds Ratio (OR)	95% Confidence Interval	Odds Ratio (OR)	95% Confidence Interval
Examinations - Fundoscopic	12.46	1.57 – 98.81	12.92	1.62 – 103.2	+
Referrals - Ophthalmologist	2.10	1.12 – 3.93	2.17	1.15 – 4.08	+
Questions - Previous neuropathy	1.97	0.86 – 4.51	1.91	0.84 – 4.37	−
Examinations - Foot pulses	1.94	1.03 – 3.66	1.94	1.03 – 3.68	0
Examinations - Check for ulcers	1.93	1.07 – 3.47	1.95	1.07 – 3.53	0
Questions - History of eye disease	1.89	0.99 – 3.62	1.84	0.96 – 3.53	−
Tests - Liver function	1.62	0.61 – 4.28	1.57	0.59 – 4.16	−
Advice - Foot self exams	1.45	0.76 – 2.77	1.42	0.74 – 2.71	0
Tests - Microalbuminuria	1.42	0.79 – 2.55	1.43	0.80 – 2.58	0
Advice - Diet	1.39	0.72 – 2.67	1.39	0.72 – 2.68	0
Questions - Adherence with diet	1.37	0.76 – 2.45	1.37	0.75 – 2.47	0
Referrals - Podiatrist/Chiropodist	1.32	0.69 – 2.55	1.38	0.71 – 2.69	+
Tests - Fasting lipid	1.26	0.69 – 2.28	1.28	0.71 – 2.33	0
Questions - Adherence with meds	1.19	0.67 – 2.12	1.18	0.66 – 2.12	0
Examinations - Monofilament/vibration	1.15	0.63 – 2.08	1.14	0.63 – 2.08	0
Questions - Exercise habits	1.10	0.60 – 2.04	1.09	0.58 – 2.04	0
Tests - TSH	1.06	0.54 – 2.11	1.04	0.52 – 2.08	0
Advice - Exercise	0.95	0.51 – 1.77	0.91	0.48 – 1.72	0
Tests - HbA1c	0.85	0.44 – 1.62	0.84	0.44 – 1.62	0
Examinations - Blood pressure	0.80	0.32 – 2.01	0.77	0.31 – 1.94	0

¹Odds Ratios > 1 mean patients of Upper SES received the examination more often (when compared to Lower SES patients)

Odds Ratios < 1 mean patients of Upper SES received the examination less often (when compared to Lower SES patients)

²Adjusted for patient race/ethnicity, age, gender, physician experience, and physician gender

³Adjusted for patient race/ethnicity, age, gender, physician experience, physician gender, and guideline adherence

⁴(−) Disparity was reduced by at least 0.05

(+) Disparity was increased by at least 0.05

(0) Disparity was neither increased or decreased by at least 0.05

Possible Unintended Consequences of Clinical Guidelines

While reported adherence to clinical guidelines does not appear to reduce or eliminate the SES disparities in the treatment of diabetes evident in our experiment (a major intended consequence) we now employ the theory of unintended consequences to explore whether adherence produces other unexpected or generally unrecognized effects. Results summarized in Figure 2 show that, with the sole exception of a fundoscopic eye examination, guideline adherent physicians in our study were more likely to perform nearly all of the clinical actions listed—19 out of the 20 specific actions of interest. In other words, use of guidelines appears to ensure that more of everything is done, without beneficially altering healthcare disparities. We suggest an unintended consequence of the use of guidelines may be their contribution to ever increasing healthcare costs.

Figure 2.

Components of a comprehensive diabetic examination for diabetic patients presenting with symptoms suggestive of emerging foot neuropathy by use of guidelines

We speculate the use of clinical guidelines may have another more subtle long term unintended consequence; reducing medical care to a standardized formulaic process creates conditions conducive to the eventual elimination of primary care doctors. Figure 3 theorizes on the steps by which this may be occurring. Steps 1 –3 have already been described above. Prior to the development of clinical guidelines independent fee-for-service physicians enjoyed the clinical autonomy and decisional latitude described by Freidson [27, 28]. Clinical decision-making was considered a somewhat metaphysical activity, perhaps even an art, which required special professional judgment based on unique experience and the particular circumstances of the presenting case. Physicians could manage the case in the way they deemed most clinically and professionally appropriate. Clinical guidelines changed this dramatically by promulgating generally accepted “gold standards”, lists of what physicians ought to do, and against which everyday clinical performance could be evaluated (performance measures) and reimbursed (pay-for-performance) [29, 30]. Mostly s physicians nowadays are rewarded less for eventual improvements in patient outcomes and reduction in health disparities than for evidence of guideline adherence) [31].

Figure 3.

From individualized care by autonomous physicians to formulaic tasks by any provider, or a computer: the unintended consequences of clinical guidelines

We suggest guidelines essentially create a formula by which any patient with a particular condition should be managed, thereby eliminating consideration of the peculiarities of a case and discouraging particularistic decision making. They demystify the so called “art of doctoring”, a physician’s metaphysical consideration of a patient’s whole being and underlying psychosocial phenomena, and reduce the patient-provider encounter to a prescribed, formulaic approach to an objectified case [20]. As long as what is done fulfills the requirements of some formula, then quality medical care is being delivered, irrespective of whether a well trained doctor, nurse or computer actually delivers it. Elsewhere we have described how this process reinforces other developments in healthcare in the US which together are eroding the central position of doctors in the provision of much primary care [32, 33]. Those involved in the implementation of new programs may not consider their long-term unintended consequences. It is paradoxical that physicians, usually with the best intentions (improving the quality of medical care and eliminating disparities) are dramatically altering the nature of doctoring and creating, through guidelines, the seeds of their own demise as an autonomous profession.

CONCLUSIONS AND IMPLICATIONS

Principal reasons for the introduction of clinical practice guidelines were the desire to improve the quality of medical care and the reduction of well-documented healthcare disparities. After initial resistance most providers accepted the laudable goals of practice guidelines, especially after demonstrated adherence to them (performance measures) became a condition for reimbursement (pay-for-performance). This paper addresses two questions relating to the intended and unintended consequences of clinical practice guidelines. (1) Does the use of practice guidelines reduce SES disparities in the management of diabetes (an intended consequence)? (2) Is the widespread and often mandated use of guidelines producing unexpected and generally unrecognized consequences?

Discrepancies in physician decision making by SES have been well documented in the literature [7, 13]. Therefore, we first document consistent SES differences in a broad range of components of care for a “patient” presenting with symptoms suggestive of diabetes (17 out of 20 specific clinical actions recommended by the guidelines were more likely to be performed with an upper SES diabetes patient). A physician’s reported adherence to clinical guidelines appears to have no effect on these SES disparities; for 14 out of the 20 clinical actions there was no impact on the disparities identified, while as many disparities were actually increased as were reduced. From our experimental data and with respect to the management of diabetes, guidelines do not appear to be accomplishing their raison d’être. However, they do appear to ensure that more of everything is done (without reducing or eliminating healthcare disparities), and potentially contribute to the increase in costs of medical care. Additionally, we speculate that guidelines reduce the complexities of the patient-provider encounter to a standardized formulaic process which is eroding the position of primary care doctors and creating conditions conducive to their eventual replacement (by appropriately trained nurses or even a computer).

The research described here has a number of strengths and an unavoidable limitation. Many studies focus on whether clinical guidelines and the programs requiring their use are actually effective (i.e., achieve their intended purpose). We counterbalance this by the additional consideration of their unintended consequences. Decision making researchers have tended to focus on several actions (especially test ordering and prescriptions) whereas we focus on a broader range of everyday clinical actions recommended by guidelines (information seeking, essential examinations, advice on life styles, referral behavior, follow-up strategy). The use of clinically authentic filmed vignettes is a distinct improvement over written scenarios, or so-called standardized patients. Unlike written scripts, filmed vignettes permit important nonverbal cues (e.g., race/ethnicity, gender, obesity and anxiety) to be unobtrusively embedded. While there was understandable resistance to the use of filmed scenarios when they were first introduced they have now been shown in many different studies to be a useful addition to the health services researcher’s armamentarium and produce valid results. All experiments have an unavoidable limitation (external validity). While the use of factorial experimentation usually ensures excellent internal validity there is a persistent threat to external validity (whether study physicians responded as they would with real patients). Four precautionary steps were taken to help guard against threats to external validity—involvement of clinically active physicians to generate and oversee the development of the diabetes scenarios, insertion of questions concerning the typicality of the scenarios (85% considered the scenario very or reasonably typical), conducting interviews in a physician’s office often sandwiched between real patients, and all subjects were explicitly requested to view the “patient” as one of their own real patients and to respond as they normally would in everyday practice.