The Relative Contribution of Patient, Provider and Organizational Influences to the Appropriate Diagnosis and Management of Diabetes Mellitus

Abstract

Objective

To estimate the relative contribution of patient attributes, provider characteristics and organizational features of the physicians' workplace to the diagnosis and management of diabetes.

Research Design and Methods

In a factorial experimental design physicians (n=192) viewed clinically authentic vignettes of “patients” presenting with identical signs and symptoms. Physician subjects were primary care physicians stratified according to gender and level of experience. During an in-person interview scheduled between real patients, physicians were asked how they would diagnosis and manage the vignette ‘patients’ in clinical practice.

Results

This study considered the relative contribution of patient, physician and organizational factors. Taken together patient attributes explained only 4.4% of the variability in diabetes diagnosis. Physician factors explained only 2.0%. The vast majority of the explained variance in diabetes diagnosis was due to organizational factors (14.3%). Relative contributions combined (patient, provider, organizational factors) explained only 20% of the total variance.

Conclusion

Attempts to reduce health care variations usually focus on the education/activation of patients, or increased training of physicians. Our findings suggest that shifting quality improvement efforts to the area which contributes most to the creation and amplification of variations (organizational influences) may produce better results in terms of reduced variations in health care associated with diabetes

That there is considerable variation in the everyday clinical behavior of physicians (both diagnosis and clinical management) has been extensively documented over several decades.¹ The focus of research to explain this variability has shifted over time-from patient attributes, to provider characteristics, to organizational influences. We find it useful to conceive of three evolving generations of work: First generation studies showed how clinical decisions are influenced by who the patient is (e.g. their gender, race/ethnicity, age and socio-economic status); Second generation studies shifted to characteristics of the provider (e.g., medical specialty, age/clinical experience and gender, among other factors); Third generation work is now examining organizational and health system differences (e.g., type of ownership, practice size, availability of electronic support systems, practice culture and reimbursement schemes). Differences between physicians practicing in different national healthcare systems are an emerging area of work.² Together, these studies suggest that physician decision making is influenced as much by who the patient is, who the physician is, and the type of setting in which care is received, as it is by what the patient actually has (the signs and symptoms of disease).

Most of the work to date has examined these different influences on clinical decision making in isolation (i.e., they are de-contextualized), without regard to the likely simultaneous contribution of factors representing other generations of work. An effect that is attributed to, say, patient attributes (like age and race/ethnicity) may overlook other competing influences (e.g., physician characteristics and features of the practice setting). This is akin to a recognized problem in epidemiology whereby the influence of a solitary risk factor is highlighted (e.g., nutrition) with little regard for additional and possibly even more influential factors (e.g. exercise and smoking). The concept of relative contributions gives emphasis to the simultaneous influence of a range of variables, or levels of influence, to clinical decision making. From a statistical perspective we are essentially interested in the proportion of the total variance in physician behavior explained by all three generations of work combined, and then how much is explained by each generation separately, and within each generation, the explanatory contribution of separate factors. We hypothesize that patient attributes, physician characteristics and features of the practice setting contribute separately and equally to the diagnosis and management of diabetes.

The goal of this paper is to estimate the relative contribution of patient attributes, physician characteristics and organizational influences to: a) the appropriate diagnosis for a “patient” presenting with signs and symptoms strongly suggesting diabetes; and b) the appropriate management of an already diagnosed case of diabetes (with an emerging foot neuropathy). This work has both clinical and health policy implications—by identifying factors which influence the quality of care provided by primary care physicians (who continue to manage the vast majority of diabetes in the community) and by identifying specific areas where interventions designed to improve the quality of diabetes care are likely to be more effective.

Methods

We conducted a factorial experiment using two clinically authentic video-based scenarios: the first presented an undiagnosed “patient” with signs and symptoms strongly suggestive of diabetes (the physician's task was initial diagnosis); while a second “patient” presented with already diagnosed and sub-optimally controlled diabetes with an emerging peripheral neuropathy (the physician's task was clinical management). An ordered version of the filmed conditions presented on DVD (varying only the “patient's” age, race/ethnicity, gender, and SES) was shown to each of 192 licensed, internists, family physicians, or general practitioners practicing in New Jersey, New York (excluding the New York City metropolitan area, Long Island, and the extreme northern counties), or Pennsylvania. We stratified these experimental subjects according to gender and level of clinical experience and recruited eligible physicians until each cell was complete. As summarized in Table 1 the first “patient” with undiagnosed diabetes presented clear indicators of diabetes including increased fatigue, non-intentional weight loss over more than three months without diet or increased exercise, increased thirst, and frequent urination, particularly at night. The second “patient” with already diagnosed diabetes displayed symptoms suggestive of an emerging peripheral neuropathy, reported “burning in the feet up to the ankle” and an intermittent burning sensation described as something that “comes and goes.” Each case was developed with the input of expert clinicians who regularly treat patients with diabetes. Several minor distractions were embedded in the presentation, not to make the task of diagnosis more difficult, but to increase the clinical authenticity of the clinical scenario (See Table 1). Following development of the clinical vignettes, clinical experts confirmed the accuracy of the clinical content and the realism of the presentation.

Table 1. Symptoms (and distractions) Embedded in Two Clinical Scenarios: A case representing undiagnosed diabetes and a case of diabetes with peripheral neuropathy.

Undiagnosed Case Symptoms	Diagnosed Case Symptoms
Thirst Fatigue Frequent urination Non-intentional weight loss for more than 5 months without changing diet Not feeling well Overweight	Burning in feet Bottoms of feet and up one ankle Intermittent Hard to localize Overweight
Distractions	Distractions
Patient concern about heart disease High blood pressure (135/95) Drinking a lot of caffeine Hasn't been to doctor in several years	Patient concern about high blood pressure On blood pressure medication, but compliance less than ideal Single high blood pressure reading (145/98) HbA1c = 6.9

Note: Since patients seldom present as ‘textbook cases’ minor distractions were also embedded to increase the clinical authenticity of the clinical scenarios.

For the estimation of main effects, a total sample of 192 physicians gives 80 percent power to detect an absolute difference in means of 20.4 percent for two groups, and an absolute difference of 24 percent for three groups. For two-way interactions our scale provides 80 percent power to detect an effect size of .204 (for factors with two levels each) – .225 (for one factor with two levels and one factor with three levels). The effect size is a ratio of the variability of the hypothesized means divided by the variability of the observations. For two means with a difference Δ, standard deviation of subjects s, the effect size is Δ/2s.³ Immediately after viewing the selected video for the experiment, the experimental subjects (physicans) completed a semi-structured interview, which included questions concerning how they would manage the “patient” in their everyday clinical practice, including their most likely diagnoses (for the first case), their certainty with respect to the diagnosis (for the first case), test ordering, prescriptions, lifestyle recommendations, and what other information they might seek.

Experimental Stimuli (The Clinical Scenarios)

Professional actors and actresses were recruited in New York City and directed (under experienced physician supervision) to realistically portray a “patient” presenting to a primary care provider with the signs and symptoms of the two clinical scenarios. Twenty-four identical versions of the scenario were created, systematically varying the “patient's” age (35 vs. 65 years in order to obtain adequate separation between the younger and older patients), race/ethnicity (white vs. black vs. Hispanic), gender, and socioeconomic status (lower vs. higher social class – a janitor vs. a lawyer). In contrast to written scripts, filmed scenarios permit potentially relevant nonverbal indicators to be embedded in the script; in this study, the “patients” were male and female, moderately overweight and from different race/ethnic groups. Each video-based encounter simulated an initial interview with a primary care physician and was of 5-7 minutes in duration, reflecting the average length of a consultation (face time) with a primary care physician (not including a physical exam and measurements taken by the clinic staff).

Diabetes was selected for study because: (a) it is among the most common and costly problems presented by patients to primary care providers; (b) while variation in the quality of diabetes care has been reported few studies have focused specifically on clinical decision making for this disease; (c) the scenarios depicted are likely to elicit a range of possible diagnostic, therapeutic and life style actions; and (d) clinical guidelines for the management of diabetes provide an operational standard for assessment of the appropriateness of physician behavior. Scripts for the scenarios of interest were developed from tape-recorded role-playing sessions with experienced, clinically active physicians.

Experimental Subjects (Physicians)

To be eligible for selection, physicians had to: (a) be internists, family practitioners, or general practitioners; (b) have ≤12 years clinical experience (graduated between 1993-99) or ≥22 years experience (graduated between 1969-83) in order to ensure clear separation by level of experience; (c) be trained at an accredited medical school in the US; and (d) be currently providing clinical care at least half time. We employed an availability sample to equally fill four design cells (gender by level of experience). Screening telephone calls were conducted to identify eligible subjects and an hour-long, in-person interview was scheduled (at which time written informed consent was obtained). Each physician subject was provided a modest stipend ($100) to partially offset lost revenue and to tangibly acknowledge participation. Each subject signed and was provided a copy of informed consent, and all study procedures were approved by the NERI Institutional Review Board.

Measures

Immediately after viewing the vignette, physician subjects completed a semi-structured interview. This interview included questions concerning how they would diagnose (undiagnosed vignette) or manage the case (diagnosed vignette) in their everyday clinical practice. In the first vignette (undiagnosed) we focus solely on whether a diabetes diagnosis was given. In the second vignette we examine whether physicians performed at least two of the following foot exams: (1) a vibration or monofilament test for loss of sensation; (2) examination for ulcers; and (3) palpation of foot pulses for peripheral vascular disease. These are key examinations that the American Diabetes Association (ADA) guidelines, the Agency for Healthcare Research and Quality (AHRQ) guidelines, as well as an experienced consulting physician recommend this patient should receive (at a minimum).^{4, 5}

A variety of explanatory variables were considered, including: patient attributes (age, race/ethnicity, gender, and SES), physician characteristics (gender, level of experience, self-reported work stress, and perceived clinical and administrative autonomy), and organizational factors (solo or group practice, for-profit status, physician/non-physician ownership, practice culture, use of clinical guidelines, disease management tools, electronic resources, and time pressures imposed on the physician).

Practice culture was measured by a standardized self-administered questionnaire with 10 Likert-scaled items measuring 9 cultural dimensions. These items constitute an abbreviated version of the medical group practice instrument.⁶ Participants were asked to indicate the extent to which they agreed with the statements from 1=Not at all to 4=A great extent. Values were dichotomized into high (3,4) and low (1,2) on the practice culture scales. Since practice culture questions are only relevant to group physicians, an indicator variable for solo practitioners (1=solo; 0=group) accompanies any practice culture variables.

Physicians' use of clinical practice guidelines as measured by three items was obtained from a self-administered questionnaire (listed in Table 2). Each question had a yes/no response, if physicians did not respond affirmatively to the first question (knowledge of guidelines), then the following two questions were skipped (use of and adherence to guidelines) and “no” responses were imputed. Physicians were said to adhere to guidelines if they reported: a) having access to guidelines; b) using guidelines in the management of their patients; and c) that their knowledge of the guidelines affected their decisions for the vignette patient.

Table 2. The Relationship of Patient, Provider, and Organizational Factors to Initial Diagnosis of Diabetes and Management of an Emerging Foot Neuropathy.

	Undiagnosed Vignette Diabetes Diagnosis		Diagnosed Vignette Perform ≥ 2 exams
	OR1	p-value	OR1	p-value
Patient Factors
Age (65 vs. 35)	0.88	0.66	1.41	0.24
Socioeconomic Status (High vs. Low)	0.74	0.30	1.54	0.14
Race (Black vs. White)	2.94	0.004	0.72	0.37
Ethnicity (Hispanic vs. White)	1.66	0.16	0.82	0.59
Gender (Female vs. Male)	0.67	0.18	0.55	0.04
Physician Factors
Gender (Female vs. Male)	1.48	0.18	1.54	0.14
Experience (More vs. Less)	1.62	0.10	0.84	0.56
Physician Work Stress (Higher Values = More Stress)
Overall Stress	0.94	0.75	1.18	0.38
Stress (Effort scale)	0.91	0.61	1.17	0.39
Stress (Reward scale)	0.98	0.89	1.12	0.47
Perceived Autonomy
Clinical Autonomy	1.09	0.83	0.99	0.98
Administrative Autonomy	1.26	0.35	0.96	0.85
Organizational Factors
Practice Size
Solo Practice (vs. Large Group)	1.52	0.24	1.25	0.51
Small Group (vs. Large Group)	0.70	0.35	1.07	0.85
Physician Owned (vs. Non-Physician Owned)	0.83	0.54	1.53	0.15
For-Profit (vs. Non-Profit)	1.47	0.23	1.26	0.47
Practice Culture (High vs. Low)
Collegiality	1.00	0.91	2.18	0.04
Information Emphasis	1.26	0.56	1.79	0.15
Quality Emphasis	0.65	0.29	1.16	0.71
Management Style	1.91	0.08	1.76	0.13
Cohesiveness	1.27	0.59	2.72	0.03
Organizational Trust	3.07	0.004	2.07	0.06
Adaptive	1.61	0.19	0.89	0.75
Autonomy	1.92	0.10	0.45	0.05
Profit Maximization	0.80	0.57	2.01	0.08
Guidelines (Yes vs. No)
Access to Guidelines	0.86	0.69	2.00	0.06
Use guidelines in your practice	0.79	0.52	1.64	0.15
Guidelines affect decisions	1.20	0.59	1.76	0.08
Do you have any disease management tools in your practice?
Any tools	0.87	0.65	1.42	0.23
Audit/Feedback/Tracking reports	1.11	0.74	1.04	0.90
Email/Chat rooms/Bulletin boards	0.28	0.01	2.85	0.07
Specially trained nurse, educator, etc.	1.10	0.82	0.78	0.53
Electronic reminders	1.32	0.50	1.19	0.67
Portable devices for patients	0.69	0.31	0.84	0.64
Email communication with patients	0.77	0.47	1.24	0.56
Use electronic medical record (EMR)	0.79	0.45	1.02	0.96
(Time needed - Time allocated) for patient visits
New patient appointment 2	0.76	0.09	1.05	0.74
Routine Consultation 2	0.84	0.47	1.23	0.42
Complete physical examination 2	0.85	0.24	1.20	0.21

¹Odds Ratio (OR)

²OR is for a 10 minute difference between time needed and time allocated

Physicians' work stress was measured by a standardised self-administered questionnaire with 9 Likert-scaled items. These items represent short versions of the two scales ‘effort’ (4 out of 6 items) and ‘reward’ (5 out of 11 items) of the original effort-reward imbalance questionnaire.⁷ Participants were asked to indicate whether or not the item content corresponds to their own typical experience. If this was the case, they evaluated to what extent they usually felt distressed by this experience (5-point scale ranging from ‘no’, ‘yes, and I am not distressed’ to ‘yes, and I feel very distressed’).

Level of physicians' professional autonomy was assessed by a self-administered questionnaire containing 12 items that measure two crucial aspects of professional autonomy, clinical autonomy (7 items), and administrative autonomy (5 items) on a 4-point Likert scale.⁸ Administrative autonomy focuses primarily on issues related to office management and provision of services while clinical autonomy deals primarily with diagnosis and management decisions related to patient care.

Statistics

Analyses were conducted using logistic regression. Due to the large number of comparisons, there are potential problems with multiple testing. Recognizing this issue, we focus on the consistency of our findings, rather than on any isolated statistically significant result. A generalized coefficient of determination (R²) statistic⁹ was used to determine the relative contributions of organizational, physician, and patient factors in a logistic regression model. The R² can be interpreted as the proportion of variation in the dependent variable that can be explained by each independent variable in the model.¹⁰ Model building was conducted in three phases: 1) each variable was introduced to the model independently, without other covariates; 2) the variables were ranked by their influence, as determined by the R² found in the first step; and 3) coefficients were added to the final model by order of influence, as found in the second step, and their contribution to the R² was calculated. To avoid over-fitting the model all explanatory variables that explained less that 0.2% of the outcome were removed from the final model. However, design variables (physician gender and level of experience and patient race/ethnicity, SES, age, and gender) were left in the final model irrespective of their impact on the model. All analyses were conducted using SAS 9.1 (Cary, NC). Missing values were replaced with plausible values by multiple imputation techniques. Twenty-five multiple imputations were performed using all relevant variables. At most 5% of the data were missing for some physician or organizational characteristics.

Results

Undiagnosed Vignette

Primary care physicians selected for the study viewed a clinically authentic scenario of a patient presenting with symptoms strongly suggestive of diabetes (for example, tiredness, weight loss, frequent urination and thirst). Overall, 60.9% of physicians provided a diagnosis of diabetes. Major results of patient, physician and organizational factors are summarized in Table 2 (columns 1 & 2).

Among patient attributes, the most important factor associated with a diabetes diagnosis was race/ethnicity. 73.4% of the physicians diagnosed diabetes when the patient was black, 60.9% when Hispanic, and 48.4% when white (p=0.004). Black patients had nearly 3 times the odds of receiving a diabetes diagnosis over white patients (OR = 2.94). Hispanic patients also had higher odds of diagnosis over white patients (OR = 1.66). Figure 1 depicts these marked variations in diagnostic behavior depending on the patients' race/ethnicity. It should be recalled that while the “patients” differed on age, race/ethnicity, gender and SES, they all presented in the exact same way with identical signs and symptoms.

Figure 1. Black and Hispanic Patients are More Likely than White Patients to Be Diagnosed with Diabetes.

Note: All patients presented with identical signs and symptoms suggestive of type 2 diabetes.

With respect to physician characteristics, female physicians were slightly more likely to diagnose diabetes than their male counterparts (65.6% vs. 56.3%, OR=1.48, p=0.18). More experienced physicians were somewhat more likely to diagnosis diabetes than their less experienced counterparts (66.7% vs. 55.2%, OR=1.62, p=0.10). Other physician factors considered such as work stress and perceived administrative and clinical autonomy had no discernable effect on the diagnosis of diabetes.

We examined a variety of organizational factors that may influence the way physicians diagnose and manage diabetes. Practice size, ownership, and for-profit status did not have a noticeable impact on diabetes diagnosis (Table 2). The most important organizational factor contributing to a diabetes diagnosis was practice culture, and specifically organizational trust. Physicians who answered that they agree or strongly agree that their practice has a high degree of organizational trust were much more likely to diagnose diabetes (65.8%) than physicians who said that they disagree or strongly disagree (39.1%, OR=3.07, p=0.004). Likewise, physicians in practices that have a management style that can best be described as “consensus building” were slightly more likely to diagnosis diabetes (62.7%) than physicians in practices that do not value this style of management (45.7%, OR=1.91, p=0.08). Physicians in practices that value autonomy were somewhat more likely to diagnose diabetes (60.7%) than their less autonomous counterparts (44.7%, OR=1.92, p=0.10). Access to clinical guidelines and the availability of disease management tools had no impact on diabetes diagnosis, with the exception of using email, web based chat rooms or bulletin boards specifically about diabetes. The 19 physicians who responded that they use these tools were less likely to diagnose diabetes (31.6% vs. 63.7%, OR=0.28, p=0.01). However, this may be a statistical artifact due to the small number of physicians who use these resources. Finally, time needed for patient appointments vs. the time actually allocated for patient appointments were analyzed. Physicians who feel they need more time than they are allocated are less likely to diagnose diabetes (OR=0.76, p=0.09).

Using the relative contributions model-building approach described above we developed a logistic regression for the probability of giving a diabetes diagnosis. We constructed the model by including only variables with an R² of 2.0% or greater. We included the patient/physician design factors in the model regardless of their significance so as to not unduly influence the model towards organizational factors. Overall, we found that 20% of the variability in diabetes diagnosis could be explained by factors measured in this study (Figure 2). Of the patient attributes considered only race/ethnicity would have been retained in the model (R² = 2.9%). However, when all patient attributes are retained they together explain only 4.4% of the variability in diabetes diagnosis. Less variability was explained by physician factors (2.0%) and only the design factors of gender and level of experience were included in the model (again so as not to unduly influence the model towards organizational factors). Most of the explained variance in diabetes diagnosis was due to organizational factors (14.3%). Practice size, for-profit status, practice culture, use of clinical guidelines, several disease management tools, and time pressure variables were retained in the model. We found that practice culture contributed the greatest percentage of variability in the diagnosis of diabetes (7.6%) among all factors considered. Availability of resources and especially the use of electronic resources (email, internet, electronic medical records) contributed to the known variability as well (3.2%).

Figure 2. How much of the Variation in the Diagnosis of Diabetes is Explained by Organization, Physician, and Patient Factors.

Diagnosed Vignette

In the second vignette physicians viewed a scenario of a patient with diabetes presenting with symptoms of peripheral neuropathy (described as an “odd, burning feeling in the souls of feet that occasionally radiates to ankles and toward the hip”). We focus on the three essential components of a foot examination for diabetes based on clinical guidelines and expert consultation: a monofilament/vibration test for loss of sensation, a check of foot pulses for peripheral vascular disease, and a check for ulcers.⁴ Given the emphasis on an emerging peripheral neuropathy it is not surprising that 58.3% would perform at least two of essential components of a foot examination (vibration/monofilament, foot pulses and a check for ulcers). Major results of patient, physician and organizational factors are summarized in Table 2 (columns 3 & 4).

Among patient factors, gender and SES influenced receipt of appropriate components of a foot exam. Female patients were less likely to receive two or more of the foot exams described above (51.0%) than their male counterparts (65.6%, OR=0.55, p=0.04). Upper SES were more likely to receive two or more of these essential exams (63.5% vs. 53.1%, OR=1.54, p=0.14).

The most important physician factor was gender. Female physicians were slightly more likely to perform the appropriate foot exams (63.5%) than male physicians (53.1%, OR=1.54, p=0.14).

Consistent with our findings for the undiagnosed vignette several components of practice culture contributed to the appropriate management of diabetes with complications. In the diagnosed vignette collegiality (sharing clinical information), management style, cohesiveness (identifiable practice style and a strong sense of belonging to a group), organizational trust, and profit maximization all contributed positively to performing essential components of the foot exam (Table 2, columns 3 & 4). However, physicians in practices with a greater sense of autonomy were less likely to perform the foot exams (OR = 0.45, p=0.05).

In our relative contributions analysis we found that 21.1% of the variability in performing vital foot exams could be explained by factors measured in this study (Figure 3). Similar to our results on the undiagnosed vignette we found that most of the explained variance in diabetes diagnosis was due to organizational factors (17.6%). However, patient (2.2%) and physician (1.3%) factors contributed even less to foot exams than they did to diabetes diagnoses. Gender was the single most important patient (2.1%) and physician (0.7%) level factor. We again found that practice culture contributed the greatest percentage of explainable variability (9.4%). Similarly, the availability of electronic resources was found to contribute a large percentage (3.3%).

Figure 3. How Much of the Variation in the Management of Diabetes Related Foot Neuropathy is Explained by Organization, Physician, and Patient Factors.

Conclusions

Consistent with other reports we found variations in both the diagnosis and management of diabetes by primary care physicians.^11-14 Our experiment revealed the independent influence of race/ethnicity on the initial diagnosis of diabetes and socio-economic status on the management of a diabetes-related foot neuropathy. We examined the relative contributions of patient attributes, provider characteristics and organizational factors (as covariates) to the appropriate diagnosis and management of diabetes. These three influences together contributed just over 20 percent of the total variance explained for each clinical scenario. Patient and provider characteristics had little influence on initial diagnosis and management of diabetes, while between 14 and 16 percent of the total variance explained was contributed by features of the physician's workplace. Although we hypothesized that patient, physician, and organizational factors would contribute separately and equally to physicians' diagnosis and management of diabetes this was not confirmed: considerably more of the known variation was contributed by features of the practice setting. It should be noted that a clear majority (approaching 80 percent) of the total variance for both the initial diagnosis of diabetes and the management of a related foot neuropathy remains unexplained.

Most efforts to reduce widely-reported health care variations have either focused on patients (e.g. health education and patient activation) and especially providers (e.g. continuing medical education, implementation of practice guidelines and performance-based reimbursement) to improve the overall quality of care. Unfortunately these efforts have yielded generally disappointing results, which may be explained by findings from the experiment reported here—namely that organizational factors contribute considerably more to diabetes-related health care disparities than either patient or provider characteristics most of which are not amenable to change. Focusing on features of the physician's workplace setting rather than on physicians themselves, or even on patients, may provide a new and more promising approach to the reduction of variations in diabetes care.

The research described here has a number of strengths and several unavoidable limitations. First, our focus on patient attributes, physician characteristics and health system influences within a single study represents a somewhat new direction for research on clinical decision making. Earlier research has tended to focus on decision making in several key areas like diagnosis and test ordering, whereas our study embraces a broader set of possible actions (e.g., life style recommendations, prescriptions, referrals and follow-ups). Second, the use of an experimental approach (instead of multivariate analysis of observational data) produces un-confounded estimates in a cost efficient manner. No matter how sophisticated the analysis of observational data, results are unavoidably threatened with residual confounding. Third, use of clinically authentic video-based vignettes is a distinct improvement over written scenarios or standardized patients. Unlike written scenarios, video-based vignettes permit important nonverbal cues (e.g., obesity, race/ethnicity, anxiety) to be included. While there was understandable resistance to the use of video-based scenarios when they were first introduced, they have now been shown in many different studies to produce results that are both valid and useful.^15-17 While the use of factorial experimentation usually ensures excellent internal validity, a limitation of our research (and all experimental studies) concerns threats to external validity (whether physicians respond as they would with real patients). Four precautionary steps were taken to enhance external validity (use of clinically active clinicians to generate clinically authentic scenarios, insertion of questions concerning the typicality of the scenario, conducting interviews in the practice office often sandwiched between real patients, and the physician subjects were explicitly instructed to view the ‘patient’ as one of their own patients, and to respond as they normally would in everyday practice).