Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Jan 1.
Published in final edited form as: Stigma Health. 2020;5(1):94–103. doi: 10.1037/sah0000179

Testing active learning workshops for reducing implicit stereotyping of Hispanics by majority and minority group medical students

Jeff Stone 1, Gordon B Moskowitz 2, Colin A Zestcott 3, Katherine J Wolsiefer 4
PMCID: PMC7597671  NIHMSID: NIHMS1031406  PMID: 33134507

Abstract

The present research tested if having first year medical students complete active learning workshops would reduce their implicit stereotyping of Hispanics as medically noncompliant. The workshops were tested with 78-majority (White) group, 16-target minority (Hispanic, African-American and American-Indian) group, and 42-non-target minority (Asian-American and foreign born students from East Asia and Southeast Asia) group students in the 2018 and 2021 classes in the American Southwest. Prior to the workshops, students completed an implicit association test (IAT), and then participated in two workshops that covered the psychology of intergroup bias, the role of implicit bias in patient care, and activities for learning six strategies for controlling the implicit stereotyping of patients. The results showed that before the workshops, the level of implicit stereotyping of Hispanics was significant for the majority and non-target minority group students, but it was not significant for the target minority group students. After the workshops, target minority students again showed no bias, and implicit stereotyping was significantly lower for the majority group students, but not for the non-target minority students. The results suggest that the workshops may have been effective for majority group and target minority group students, but that more cultural tailoring of the materials and activities may be necessary to address implicit bias among some minority group medical students.

Keywords: Implicit bias, health disparities, Hispanics, medical students, medical education

A growing body of evidence shows that healthcare providers hold implicit bias-- automatically activated negative stereotypes and prejudice (see Maina et al., 2017). For example, studies find that healthcare providers have negative implicit associations for Hispanics (Blair et al., 2013; Bean et al., 2013), African Americans (Blair et al., 2013; Green et al., 2007; Hagiwara et al., 2013; Moskowitz et al., 2012), Lesbians and Gay men (Sabin et al., 2015), obese people (Sabin et al., 2012), and patients with lung cancer (Schiller et al., 2013). This raises the possibility that these subtle biases in how a healthcare provider thinks and feels toward stigmatized patients may contribute to disparities in care. Implicit biases have been linked to lower quality medical decision-making (Green et al., 2007; Sabin & Greewald, 2012), negative non-verbal communication (Penner et al., 2010, Cooper et al., 2012; Hagiwara et al., 2013), and negative perceptions of providers and the care they provide. In a recent study, Penner and colleagues (2016) reported that White oncologists high in implicit prejudice toward African American cancer patients spent less time with them, and engaged in less patient-centered dialogue, compared to oncologists with lower implicit prejudice. In turn, after interacting with a high implicit bias oncologist, African-American cancer patients recalled fewer details from the meeting and reported lower confidence in treatment recommendations, which was partially mediated by their negative perceptions of the oncologists’ style of communication. Together, the extant research suggests that implicit bias can have a negative influence on how providers interact with and provide care for minority patients.

Studies indicate that medical students harbor implicit prejudice and stereotyping toward minority patients when they enter medical school, and that their levels of implicit bias remain constant or increase over time (e.g., Chapman et al., 2013). For example, Rubineau and Kang (2012) found that the treatment of Black standardized patients became worse between the first and second years of medical school, and Phelan and colleagues (2015) reported that explicit bias toward obese patients increased during the four years of medical school. Van Ryn and colleagues (2015) attribute the expression of bias against African-American patients by medical students to several factors, including having heard negative comments from attending physicians or residents and having had unfavorable contact with African American physicians.

In order to limit the role that implicit bias plays in creating disparities for racial and ethnic minority patients, it is important to develop approaches to reducing implicit bias in medical students that are grounded in both theory and research on understanding the unique ways implicit bias can manifest when providers interact with patients. The approach in the current paper was two-pronged. We first exposed first year medical students to psychological theory and research on implicit bias as part of their training in cultural competence and minority health. We then also provided an active workshop that relied upon innovations from psychological research on the regulation of bias that were then implemented within, and tailored to, the specific intergroup interaction settings frequently encountered by medical providers. Whereas some studies suggest that educational workshops on the psychology of intergroup bias can reduce both explicit and implicit forms of bias (Devine et al., 2012; Rudman et al., 2001), the effectiveness of these workshops for reducing the biases held by medical students has yet to be fully tested (Chapman et al., 2018). None of these existing approaches combine education about bias with active learning techniques developed specifically to apply laboratory-tested forms of bias-control to the unique occupational demands of a profession, such as medicine.

Best practices for reducing implicit bias among medical students

Conceptually, many educational interventions specifically designed for changing implicit bias focus on making the target audience aware of their implicit biases (Chapman et al., 2013; Teal et al., 2010). Whereas awareness of bias may be necessary for motivating people to take action, it may not be sufficient to achieve bias reduction. Indeed, confronting people about bias can have the opposite effect of making people defensive and resistant to change, and motivate them to challenge the science on implicit bias (Howell et al., 2015). Whereas framing the awareness training in a way that will reduce defensiveness is critical, many scholars propose that providing instruction on how to control the use of group-based information, and the expression of bias during interaction, should enhance the audiences’ ability to change (Blair et al., 2011; Burgess et al., 2007; Teal et al., 2012; Schultz & Baker, 2017; Stone & Moskowitz, 2012; Zestcott et al., 2016).

Research suggests that one strategy to change implicit bias is to train people to negate negative associations that automatically come to mind when exposed to a stigmatized group member (see Chapman et al., 2018). However, Gawronski and Bodenhausen (2006) argue that negating a proposition is unlikely to deactivate the associations underlying it for the long term. Perhaps more importantly, a limitation to this approach in a healthcare context is that the process of activating beliefs or “stereotypes” about patient groups serves an important function in patient care. A common method for helping medical students learn to diagnose and treat disease is by reading clinical case studies that present information about the age, race/ethnicity, and gender of the patient, before presenting information about the patient’s history of present illness (e.g., “A 52-year old Hispanic male arrives in the ER suffering chest pains”). Categorization of the patient into one or more groups (e.g., “middle-aged Hispanic male) facilitates the use of epidemiology and relevant knowledge about the individual’s cultural background, which can be useful in diagnosing and treating the patient’s condition. Thus, categorizing patients based on age, race/ethnicity, gender and other characteristics, and activating medically relevant beliefs or “stereotypes” associated with the patient’s group memberships, is necessary early in the process of diagnosing and treating illness. Indeed, research suggests that the association between group membership and disease can become an automatic process among physicians (Moskowitz et al., 2012). Given that medical students need to categorize and activate medically relevant group- based beliefs (stereotypes) as they diagnose disease, it would not be desirable to counteract their implicit associations regarding patient demographics and the incidence of disease while attempting to negate undesirable cultural stereotypes that are also linked to the category.

Nevertheless, in order to treat the patient as an individual after they initially categorize by race, gender and age, medical students must learn to “switch” from perceiving the patient as a member of a group, to perceiving the patient as an individual. The errors that stereotyping can cause are more likely to occur when medical students fail to switch from category-based to piecemeal processing of a patient’s information (Fiske & Neuberg, 1990). A more effective approach is to help them learn how to make the shift from group-based to individuating information at the point when they need to interact with the patient as an individual. To achieve this goal, the present workshops were designed to instruct medical students in how to control their use of group-based information when they interact with stigmatized patients.

Instructing medical students in how to control their implicit stereotyping is consistent with current theory and research in social psychology on the activation and control of implicit processes (Gawronski & Bodenhausen, 2006; Gonsalkorale et al., 2009). According to Moskowitz (2010), people can learn to use both proactive and reactive strategies for controlling implicit stereotyping. Proactive strategies rely on the individual having a goal in place that would eliminate the need to recruit the cultural stereotype as part of their implicit cognition. A goal to negate the stereotype could accomplish this, but as noted above, this proactive strategy could interfere with the functional association of group membership and the incidence of disease. Other proactive strategies, such as focusing on the goal of using epidemiological evidence to provide the best possible diagnosis and treatment of disease, might promote the use of categories for diagnosing, but it is not clear they would accomplish this without the side effect of triggering unwanted cultural stereotypes.

Reactive strategies, in contrast, permit the implicit activation of stereotypes, but these strategies then focus on the control of how the information is used, including the switch to individuating the patient. According to Fiske and Neuberg (1990), after people initially categorize a person based on salient characteristics like race, gender or age, and activate stereotypes, given sufficient personal relevance (motivation) and information, they can then transition to individuation of a target person. Medical students can learn several strategies to motivate the switch from category-based to piecemeal processing during a clinical encounter.

First, students can learn to activate and act upon their egalitarian goals to provide fair and equitable treatment to all patients. The activation of egalitarian goals can inhibit the incompatible negative stereotypes and trigger instead responses that promote fairness (Moskowitz, 2010). Research suggests that most medical students already hold strong egalitarian values and goals (Burgess et al., 2007). Therefore, what they need to learn is how to activate their existing goals for fairness and equality of care when they interact with stigmatized patients.

Second, students can learn to act on their egalitarian goals by implementing bias reduction strategies. One strategy that has not received much attention in the literature is the deliberate attempt to alter nonverbal and paraverbal behavior during intergroup interaction. Research shows that implicit bias is often communicated through negative nonverbal behavior like poor eye contact, body posture, and creating physical distance (Cooper et al., 2012; Dovidio et al., 2002) and through paraverbal behavior such as verbal dominance and speed of speech (Hagiwara et al., 2013). If students learn about the relationship between implicit bias, nonverbal and paraverbal behavior, they may then be able to monitor and adjust these behaviors when they interact with patients.

Third, students can learn to switch the way they categorize patients by focusing on a shared, common identity (Gaertner et al., 1994). To inhibit the activation of other identities and their concomitant negative attitudes and stereotypes, students can seek common identity information by developing questions about social groups, interests, or activities that they have in common with their patients. Fourth, students can learn to acquire counter-stereotypic information about a patient (Dasgupta & Asgari, 2004). Research suggests that if providers acquire information that is counter-stereotypic of the specific negative beliefs they hold about stigmatized patients, they may be more likely to individuate the patient (e.g., Lai et al., 2014).

A fifth strategy for reducing the activation of implicit stereotypes is to attain the perspective of the stigmatized patient. Gaining the perspective of an out-group member attenuates stereotype activation, increases feelings of empathy and self-other overlap, and improves the ability to see the injustice in the group’s plight (Todd & Galinsky, 2014). Blatt and colleagues (2010) reported that when medical students were instructed to engage in perspective taking before interacting with African-American patients, the patients report higher satisfaction with the clinical encounter, and this effect occurred across medical schools, clinical disciplines, and among racially diverse students.

A recent paper by Lai and colleagues (2014) with online samples comparing 18 individual strategies for reducing implicit bias, including those described above, reported that exposure to counter-stereotypic and common-identity information (e.g., shifting group affiliations or boundaries) may be especially effective for reducing implicit bias when providers interact with stigmatized patients. Lai and colleagues also concluded that the successful use of any bias reduction requires active involvement or engagement when using the strategy. These findings are consistent with research suggesting that instructing providers in the use of multiple strategies can attenuate implicit bias both immediately after training, and in some cases, for several weeks after the training (Devine et al., 2012; Rudman et. al, 2001).

To improve the effectiveness of the training for medical students, the present workshops included several elements of effective diversity interventions (Moss-Racusin et al., 2014; Penner et al., 2014). Specifically, to reduce the defensiveness that can make people resistant to change, the medical students did not receive feedback on their IAT performances, they were provided with brief readings and lecture notes so they were familiar with the topic before attending each workshop, and the lecture noted the importance of using group-based information in diagnosing and treating disease. Another important aspect was the use of active learning exercises to engage students in the material and allow them to practice the control strategies. Finally, all of the implicit bias materials were carefully tailored to a medical context so that all of the examples, anecdotes, clinical scenarios and research findings were specific to the role of implicit prejudice and stereotyping in doctor-patient interactions. Active learning exercises applied the control techniques to the unique elements that emerge at various points in these interactions.

Who should benefit most from training in implicit bias?

Whereas there is evidence to suggest that medical students acquire stereotypes about various patient groups during their training in medical school (Bean et al., 2014; van Ryn et al., 2015), there may be important racial and ethnic differences in how students respond to implicit bias training. Non-Hispanic Whites constitute the majority ethnic and racial group in most medical schools, and given the emphasis on reducing disparities toward stigmatized patients, White students represent an important majority group audience for implicit bias reduction interventions (Chapman et al., 2018). Egalitarian goals are common among medical students, but implicit bias remains high (as it is for most White Americans), making the intervention especially useful for students who likely assume they are immune to bias due to these explicit egalitarian goals (e.g., Plant & Devine, 1998). However, if stereotypes toward patients are also held by minority medical and nursing students, then it is important to consider the effects of an implicit bias reduction curriculum on ethnic and racial minority students, and the possibility that not all ethnic and racial minority students hold implicit biases or respond the same way to learning to control implicit bias.

For example, students from groups that are commonly stereotyped as medically noncompliant, such as Hispanic, African-American and American Indians, may have personal knowledge about how their group is viewed by providers, and they may have direct personal experience with being treated as medically noncompliant by their providers (see Sabin et al., 2009). As a result, they may hold more positive associations between their stigmatized identity and compliance, or have chronic control mechanisms that inhibit the expression of negative associations between a stigmatized group and noncompliance. Thus, “Target” minority medical students may enter medical school with lower implicit associations between their groups and medical noncompliance, which may not be influenced by exposure to the workshops on implicit bias.

In comparison, other ethnic and racial minority medical students may show higher levels of implicit stereotyping of stigmatized patients (Sabin et al., 2009). For example, Asian immigrants are stereotyped as competent but cold (Dovidio & Fiske, 2012), and may have less experience being the target of stereotypes about noncompliance by healthcare providers. Similarly, foreign-born medical students from East Asia or Southeast Asia have less experience with being stereotyped as noncompliant in their home country of origin where they represent the numerical majority. They may, therefore, learn and retain negative stereotypes about other, commonly targeted, minority patients. Consequently, “Non-target” minority medical students may show levels of implicit stereotyping of Hispanics that are similar to White majority group students. For these non-targeted minority students exposure to the implicit bias reduction workshops should attenuate their bias.

To examine variability among minority medical students in response to the workshops, medical students who identified as Hispanic, African-American, or American Indian were classified as the “Target Minority Group,” and students who identified as Asian-American, Pacific Islander, East Asian, Southeast Asian, were classified as the “Non-target Minority Group” in the study. Students who identified as “Caucasian or White” were classified as majority group members in the analyses of the results.

The present research

Two studies tested the effects of training in implicit bias with first year medical students. The first study was run in the fall of 2014 and the second was run in the fall of 2017 at the same medical school located in the American Southwest. Whereas there were some variations in the content and delivery of the two workshops (see the supplement), the learning goals were identical. The intervention materials provided broad instruction on the topic of intergroup bias in healthcare, with a special emphasis on the causes and consequences of implicit stereotyping and strategies for controlling the automatic expression of stereotypes in patient care. Thus, the primary measure of the workshops’ effectiveness was pre-to-post changes in the implicit associations between Hispanics and stereotypes regarding medical noncompliance.

Methods

Participants.

Study 1: Of the 127 students in the first-year class for 2018, 46 male and 51 female medical students agreed to participate in the research (79%). 55 participants identified as a member of the majority group (White), 11 identified as a target minority group member (African American, Hispanic, or American Indian), and 31 identified as a non-target minority (e.g., East Asian, Southeast Asian, or as a member of another non-White, non-target group). As compensation, participants were provided an opportunity to receive a $40 gift certificate to www.target.com or www.amazon.com at the completion of the research.

Study 2: Of the 130 students in the first year class of 2021, 19 male and 21 female medical students participated (31%). 23 identified as a member of the majority group, five identified as a member of a target group, and 11 identified as a member of a non-target minority group. All received the same compensation as participants in study 1.1

Two participants were excluded for error rates on at least one IAT of over 49% (each responded in under 300ms on at least 57% of trials). The total sample across both studies was 78 majority group members, 16 target minority group members, and 42 non-target minority group members.

Procedures

A University Human Subjects Protection Committee approved all of the procedures described below. The implicit bias workshops were delivered in two sessions. Three days prior to the first session, participants received an email directing them to a website hosted by the Inquisit Millisecond software package (Inquisit 4.0.6, 2014). The first page of the website contained the informed consent form. After they provided consent, participants indicated their profession and created a unique ID in order for the researchers to identify participants across the pre and post workshop measures.

They then completed an Implicit Association Test (IAT) to measure their level of implicit stereotyping of Hispanics vs. Whites (without feedback), measures of resistance to the workshops, and a demographics questionnaire. Students were required to complete the measures at least a day before the first workshop began.

In both studies, three days after the second workshop, students received an email request to complete the same IAT measure of implicit stereotyping that they completed prior to the first workshop. They were allowed to complete the post-workshop IAT up to seven days after it became available. At the completion of the follow-up IAT, students received a written debriefing and compensation for their time.

Measures

Stereotype IAT.

Participants completed an IAT (Greenwald et al., 1998) to measure their level of implicit stereotyping before and after the workshops. For the IAT, participants categorized images of faces and words as quickly and accurately as possible using the ‘E’ and ‘I’ keys on a computer keyboard. The images, adapted from Bean et al. (2013), were pictures of three White American males and three Hispanic American males rated as similar in age and attractiveness. The categorization task was to indicate if a face was White or Hispanic. The word sets included words related to the concept of “non-adherent,” which for the IAT, was labeled “reluctant” (e.g., reluctant, doubting, hesitant, apathetic, resistant, lax) and words related to the concept of “compliant” (e.g., willing, cooperative, compliant, reliable, adherent, helpful). The categorization task was to indicate if a word was reluctant or compliant. A red ‘X’ appeared on the middle of the screen if a participant incorrectly categorized a face or word and remained until the participant corrected their response.

The IAT contained 5 blocks of trials: (1) 20 trials of a single-discrimination between the reluctant and compliant words, (2) 20 trials of a single-discrimination between Hispanic American and White American images, (3) 40 trials of a double-discrimination between either Hispanic American+reluctant/White American+compliant or Hispanic American+compliant/White American+reluctant, (4) 20 trials of a reversed keyed singlediscrimination between Hispanic American and White American images, and (5) 40 trials of a double-discrimination using the categorizations not used in the third block. The order in which compatible and incompatible categorization trials appeared in blocks 3 and 5 was counterbalanced.

External and internal motivation to respond without prejudice.

Participants completed items from the External and Internal Motivation to Respond Without Prejudice scale (Plant & Devine, 1998) with Hispanics as the target group, using a 7-point Likert scale of 1 (completely disagree) to 7 (completely agree).

Tendency to perspective take.

Participants completed the perspective taking items from the Interpersonal Reactivity Index (IRI; Davis, 1980) by rating the items on a Likert scale of 1 (does not describe me well) to 7 (describes me well).

Lay theories of racial bias.

To measure their lay theories of racial bias (Neel & Shapiro, 2012), participants indicated whether they believe that racial bias is malleable, their desire for feedback on racial bias, and adherence to a colorblind ideology using a Likert scale of 1 (not at all) to 7 (a great deal). Finally, using a similar scale, participants indicated the extent to which they prefer to address racial issues directly, indirectly, through overcompensation, or by escaping the situation.

Demographics.

Participants reported their age, gender, Hispanic ethnicity, and race.

The workshops

Students completed two workshops in both studies. In Study 1, Workshop I consisted of a 50-minute face-to-face lecture on the psychology of implicit bias that was delivered by the first author. The lecture in Study 2 was 90 minutes with a five-minute break at the midpoint. The learning goals and content for Workshop I are described in the supplemental materials.

For Workshop II, students completed active learning exercises designed to give them experience practicing five implicit bias reduction strategies as clinical skills. In Study 1, the second workshop was held three days after the first session; in Study 2, it was held 3 weeks after the first session. Also in Study 1, the same instructor taught two sections of approximately 60 students each. In Study 2, the same instructor led one section of approximately 65 students, while another instructor led the other group of 65 students. See the supplemental materials for a description of the learning goals and content for Workshop II.

Results

The data for this project are available on the Open Science Framework (https://doi.org/10.17605/OSF.IO/YRA6D). Each participant’s IAT response latencies were converted into a d-score using the improved scoring algorithm (Greenwald et al., 2003) after removing response latencies under 400ms or over 10,000ms. D-scores were calculated such that higher, positive values indicated a stronger association of Hispanic with non-compliant and White with compliant compared to the reverse. Descriptive statistics for all measures can be seen for each study in Tables 1 and 2.

Table 1.

Bivariate Correlations and Descriptive Statistics

1 2 3 4 5 6 7 8 9 10 11 12 13
1. Pre IAT -
2. Post IAT .37*** -
3. IAT Change −.58*** .58*** -
4. Age .01 .04 .04 -
5. IMCP .13 .13 −.01 −.06 -
6. EMCP .03 .04 .01 −.12 .03 -
7. PT .09 .11 .02 −.05 .44*** 0 -
8. Lay Theory −.06 .05 .10 −.05 −.29*** .26** −.15 -
9. CB Ideology .09 .12 .02 .20* -.29* .08 −.13 .27*** -
10. Direct .03 .01 −.02 −.02 .42*** .04 .41*** −.23** −.13 -
11. Indirect −.01 .14 .13 −.11 .46*** −.08 .58*** −.32*** −.08 .60*** -
12. Overcomp. −.02 .06 .07 .03 .16 .29*** 0 .08 .11 −.01 .04 -
13.Escape −.04 .09 .11 −.01 -.08 .12 −.22* .17 .17* −.21* −.10 .36*** -
M .21 .14 −.07 24.8 30.03 17.58 39.08 13.16 13.52 11.16 11.55 7.30 9.16
SD .39 .38 .44 3.78 5.27 7.02 6.65 5.84 5.00 2.50 2.23 2.55 4.72
Open in a new tab

Note. IMCP- Internal Motivation to Control Prejudice, EMCP- External Motivation to Control prejudice, PT- Perspective Taking, CB- Colorblind Ideology. Direct, Indirect, Overcompensation and Escape are all strategies for handling an uncomfortable interracial interaction.

*

p < .05

**

p < .01

***

p < .001

Table 2:

Pre and Post workshop IAT scores, and their differences, by Group Identity and Gender

Group Identity
Majority Non-Target Target Across Identity
(n = 77) (n = 42) (n = 16)
Pre Workshops1 M SD M SD M SD M SD
Male (n = 63) 0.43*** 0.29 0.23* 0.37 −0.19 0.55 0.26*** 0.39
Female (n = 72) 0.14* 0.46 0.18* 0.25 0.13 0.48 0.15* 0.39
Across Gender 0.29*** 0.40 0.21** 0.29 −0.03 0.51 0.16*** 0.39
Post Workshops
Male (n = 63) 0.25*** 0.35 0.32** 0.29 0.08 0.32 0.25*** 0.33
Female (n = 72) 0.01 0.46 0.15+ 0.31 0.03 0.29 0.06 0.40
Across Gender 0.13** 0.43 0.24*** 0.17 0.06 0.31 0.14** 0.38
Difference2
Male (n = 63) −0.18* 0.42 0.09 0.42 0.28+ 0.55 −0.02 0.45
Female (n = 72) −0.14+ 0.46 −0.03 −.36 −0.10 0.50 −.10+ 0.43
Across Gender −0.16** 0.44 0.03 0.38 0.09 0.54 −0.01 0.44
Open in a new tab

Note 1. All means were adjusted means derived from a model including participant group identity, participant gender, and study. The values in the bottom right corner are the average pre-workshop IAT d-score adjusting for participant group identity, participant gender and study. Significance markers refer to the test of the adjusted mean from 0. + p <. 10;* p < .05; **p < .01; ***p < .001.

Note 2. Difference scores were calculated post-workshop minus pre-workshop so that negative scores indicate a reduction in implicit bias after the workshop. Significance markers refer to the test of the adjusted mean (or mean difference score) from zero. + p <. 10;* p < .05; **p < .01; ***p < .001.

Pre-workshop implicit stereotyping.

To examine the level of implicit stereotypes among medical students before the workshops, pre-workshop IAT scores were analyzed for their significance from the zero point on the scale. In addition, participant group identity was entered into the model in the form of two orthogonal contrast-coded predictors. The first contrast compared non-target and majority group students (each coded as - 1) to target minority group participants (coded as 2) The second compared majority group students (coded as −1) to nontarget minorities (coded as +1). Since there were differences in the administration of the workshop across the two studies, study was also entered into the model as a contrast-coded predictor (Study 1 = ‒.5, Study 2 = .5)2. Finally, gender was entered as a contrast-coded predictor (female = ‒.5, male = .5) in these models since evidence suggests that implicit biases often differ based on gender (Nosek et al., 2007). Thus, the model examining pre-workshop IAT scores regressed IAT score on participant group identity, participant gender, study, and all possible interactions among these variables.

Across participant group identity and gender, there was no evidence of study differences in pre-workshop implicit stereotypes, b = 0.07, t(123) = 0.81, p = 0.42, ηp2= 0.01. Additionally, there was no evidence that study moderated any other effect, all p > 0.06; all ηp2 2 < 002. Table 2 presents the adjusted mean pre-workshop IAT scores by participant group identity and gender. An effect of participant group identity emerged such that, across study and participant gender, majority group participants and non-target minority participants demonstrated higher baseline implicit stereotypes than target minority group participants, b = ‒0.09, t(123) = −2.56, p = 0.01, ηp2 = 0.05. There was no evidence that white participants different from non-target minority participants in their pre-training IAT scores, b = ‒0.04, t(123) = ‒1.04, p = 0.30, ηp2 = 0.01. These data indicate that, while majority and non-target minorities’ IAT scores revealed that they associated the group Hispanic with non-compliance, target-minorities’ pre-workshop IAT scores were not significantly different from zero (see Table 2).

The main effect of group identity (white & non-target minorities vs. target minorities) was qualified by a group identity X gender interaction, b = −0.16, t(123) = 2.24, p = 0.03, ηp2 = 0.04. For female participants, there was no evidence that majority group participants differed from minority participants in their baseline implicit stereotypes, b = ‒0.002, t(123) = −0.07, p = 0.95, ηp2 < −001. However, male majority group participants’ implicit stereotypes were stronger than those of minority males, b = 0.21, t(123) = 2.44, p = 0.02, ηp2 = −05- Participant gender did not reliably moderate differences in pre-workshop IAT scores for target vs. non-target minorities, b = 0.19, t(123) = 1.52, p = 0.13,ηp2 = 0.02.

Post-workshop implicit stereotyping.

To examine whether the workshop reduced implicit stereotypes, the difference in IAT d-scores, post- minus pre-workshop, were regressed on participant group identity, gender, study number, and their interactions. Adjusted mean difference scores by participant group identity and gender are presented in Table 2. Again, there was no evidence of a main effect of study, b = 0.05, t(123) = 0.45, p = 0.65, ηp2 = ‒003, and study did not interact with any of the other predictor variables, all p > 0.14; all ηp2 < .02 Only an effect of group identity emerged from this model, indicating a larger reduction in implicit stereotypes for majority compared to non-target minority participants, b =0.10, t(123) = 1.98, p = 0.05, ηp2 = .03. Whereas the reduction in implicit stereotypes was significant only for majority group participants (see Table 2), non-target minority group students did not show a significant reduction in implicit stereotypes following the workshops.

Resistance to the workshops.

Finally, the analysis showed that none of the individual difference measures completed before the workshops were related to the pre-post workshop differences in implicit stereotyping, indicating that these individual differences in resistance to learning about bias did not moderate the effects of the workshops on implicit stereotyping. Table 2 shows the relationships among the moderator variables and IAT scores across race and gender3.

Discussion

The results of the research suggest that the present workshops on the psychology of implicit bias may have significantly reduced the automatic stereotyping of Hispanic patients. Majority (White) students showed significant pre-to-post intervention change, although their average d-score in the post-workshop IAT was significant from zero, indicating that they still held a significant association between Hispanics (Whites) with noncompliance (compliance). Non-target minority students also showed significant implicit stereotyping of Hispanics as noncompliant before and again after the workshops, but target minority students, in contrast, did not show a significant level of bias before or after the workshops. To our knowledge, this is the first study to show that subgroups of racial minority medical students responded differently to learning about the role of implicit bias in health care.

Consistent with the findings of Devine and colleagues (2012), and contrary to other recent studies (e.g., Gonzalez et al., 2014; Lai et al., 2016), majority group students showed lower implicit stereotyping of Hispanics after a three-day delay. Whereas it is difficult to pinpoint the precise reason for the longevity of the effect, the present set of workshops focused medical students on the concept of controlling their use of stereotypic, group-based information about patients, as opposed to changing the stereotypes or preventing their activation. To facilitate control of stereotypes, the workshops were carefully tailored for medical practice, designed to reduce defensiveness while making students aware of their biases, focused on provider skill training through active learning exercises, and delivered information about multiple strategies that the students could adopt when interacting the patients. These features are well suited to help highly egalitarian medical students achieve their goals for equitable patient care.

However, the unexpected finding that non-target minority students-- those who reported Asian American, Southeast or East Asian group identity-- did not show a significant reduction in implicit stereotyping of Hispanics has important implications for implicit bias training. We can only speculate about why the workshop may have been less effective for this group, One reason may have been that many of the examples and studies presented in the training focused on the biases of majority group (white) providers when interacting with target minority patients. As a result, the workshop materials and activities may have seemed less relevant to the non-target minority group students. Another possibility is that the workshops’ emphasis on health equity inadvertently induced resistance among medical students from a vertical collectivistic culture (e.g., India; see Triandis, 1996). Research indicates that people from vertical collectivistic cultures have less permeable boundaries between in-group and out-group members (see Heine, 2008) and they may be motivated to maintain existing power distances between in-and out-group members (Hoefstede, 2001). The emphasis on providing equitable care to stigmatized patients, and some of the strategies they practiced during the workshops, may have clashed with these cultural values and beliefs. For example, collectivist minority students may have more difficulty perceiving a common-identity with an out-group patient and research suggests that collectivist minority students may be less willing to take the perspective of out-group patients (Ma-Kellams & Blascovitch, 2012). It will be important to identify other cultural factors that promote resistance to the materials and exercises (e.g., time in the US; assimilation), and if necessary tailor the materials for cultural appropriateness, in order to make the workshops effective for all students,

Limitations and Future Directions

There are limitations to the research reported in this paper that may restrict the reliability and generalizability of the findings. First, the lack of a no-treatment control condition prevents strong conclusions about whether the workshop caused the changes in implicit stereotyping among majority group students. Second, participation by medical students in the first-year classes across the two studies was 53%, and although the sample in the first study was close to 80% of the entire class, the lower participation rate in the second study reduces the representativeness of the overall sample, both in terms of the medical student population, and in terms of the racial group differences observed in the present outcomes. Third, the findings were limited to implicit stereotypes about the medical noncompliance of Hispanics. Despite the prevalence of this stereotype in studies on implicit bias towards other groups, it remains to be seen if the pattern of responses to the workshops replicates on a different measure of implicit bias with a larger sample at a different medical school. Finally, future research should survey students about their impressions of the workshops to gauge their interest in the material, their recall of the information, and their perceptions of efficacy in using the strategies when they meet with patients.

One important question about the observed changes in implicit bias among majority group students concerns the degree to which they reflect control processes or another strategy for altering implicit bias responses, like faking on the IAT. Across four studies, Lai and colleagues (2014) found that instructing participants to fake their responses on the IAT significantly reduced their average IAT scores. However, the standard deviations increased from 27–40% relative to studies that used effective implicit bias reduction strategies highly similar to what our students learned (e.g., counter-stereotypic exemplars; common-identity). In the present studies, majority group members showed only a 5–7% increase in variability from the pre- to post- workshop IAT, suggesting that the present results do not appear to be driven by learning to fake responses on the IAT. Nevertheless, although the focus of the workshops was on controlling implicit stereotyping, there is no direct evidence that students successfully learned to control their stereotypes. Future research should measure directly or indirectly the mechanisms by which similar workshops reduce implicit bias on a measure like the IAT.

Another question concerns how long the reductions in implicit bias last. Lai and colleagues (2016) showed that no single strategy for reducing implicit bias lasted beyond a 24- hour period, suggesting that the present changes may only be temporary. However, their intervention strategies were brief and implemented in an online setting, a very different context than our two highly structured workshops. In addition, even if the reduction in implicit stereotyping lasts, the changes may not affect how medical students diagnose, treat and interact with Hispanic patients. A recent meta-analysis by Forscher and colleagues (2018) showed that whereas implicit bias workshops similar to the current approach can significantly reduce the level of implicit bias that participants show on the IAT, the changes do not correspond with changes in downstream behavior. It is vital to examine if the changes among majority group students translate into changes in how students interact with patients in a clinical context.

Conclusion

It is important to develop and test new strategies for addressing the ethnic and racial health disparities that stem from the implicit stereotypes and prejudices held by providers. The present active learning workshops may have been effective for reducing majority group students’ implicit stereotyping of Hispanics as noncompliant. Students learned strategies that take advantage of their existing goals to be egalitarian and promote asking specific questions during interactions with patients. These strategies may improve patient care when they 1) signal openness and shared identity, 2) establish positive nonverbal and paraverbal signals, and 3) disrupt the use of stereotypes as a guide during the clinical visit.

However, there were important differences in the level of implicit bias held by racial minority medical students, and important differences in how they responded to learning about the role of implicit bias in health care. Future research is needed to fully understand minority group differences in implicit bias, the consequences of the differences for patient care, and what educational approaches are effective for reducing the implicit biases held by some groups of medical students.

Supplementary Material

Supplemental Material
Supplemental Tables

Acknowledgements

This research was funded by the National Institute On Minority Health And Health Disparities (NIH #5R01MD005902–02 and #1R01MD008940–01) and by the Merck Foundation Alliance to Advance Patient-Centered Care.

Footnotes

1

The majority of students who did not participate made the decision before the workshops began and not as a result of participating in the workshops. Specifically, in Study 1, 30 students chose not to participate before exposure to the workshops, and 8 students did not complete the post-workshop IAT. In Study 2, 78 students chose not to participate before exposure to the workshops, and 12 students did not complete the post-workshop IAT.

2

There were no significant moderating effects of study based on this analysis, but pre-training IAT scores, poststudy IAT scores, and pre-post differences are presented separately for the two studies in Tables S1 and S2 in supplementary materials.

3

Tables S3–S8 present these correlations separately for each race and gender group.

Contributor Information

Jeff Stone, University of Arizona.

Gordon B. Moskowitz, Lehigh University

Colin A. Zestcott, SUNY Geneseo

Katherine J. Wolsiefer, University of Arizona

References

  1. Bean MG, Stone J, Moskowitz GB, Badger T, & Focella E (2013). Evidence of implicit stereotyping of Hispanic patients by nursing and medical students. Nursing Research, 6, 362–367. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bean MG, Focella E, Covarrubias R, Stone J, Moskowitz GB, & Badger T (2014). Evidence for nursing and medical students’ stereotypes of Hispanic and American Indian patients. Journal of Health Disparities Research and Practice, 7, 14–22. [PMC free article] [PubMed] [Google Scholar]
  3. Blair IV, Steiner JF, Fairclough DL, Hanratty R, Price DW, Hirsh HK, . . . Havranek EP. (2013). Clinicians’ implicit ethnic/racial bias and perceptions of care among Black and Latino patients. Annals of Family Medicine, 11, 43–52. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Blair IV, Steiner JF, Havranek EP (2011). Unconscious (implicit) bias and health disparities: where do we go from here? The Permanente Journal, 15, 71–78. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Blatt B, LeLacheur SF, Galinsky AD, Simmens SJ, & Greenberg L (2010). Does perspective-taking increase patient satisfaction in medical encounters? Academic Medicine, 85, 1445–52. [DOI] [PubMed] [Google Scholar]
  6. Burgess D, van Ryn M, Dovidio J, & Saha S (2007). Reducing racial bias among health care providers: lessons from social-cognitive psychology. Journal of General Internal Medicine, 22, 882–887. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Chapman EN, Kaatz A, & Carnes M (2013). Physicians and implicit bias: How doctors may unwittingly perpetuate health care disparities. Journal of General Internal Medicine, 28, 1504–1510. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Chapman MV, Hall WJ, Lee K, Colby R, Coyne-Beasley T, Day S, ... Payne K (2018). Making a difference in medical trainees’ attitudes toward Latino patients: A pilot study of an intervention to modify implicit and explicit attitudes. Social Science & Medicine, 199, 202–208. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Cooper LA, Roter DL, Carson KA, Beach MC, Sabin JA, Greenwald AG, & Inui TS (2012). The associations of clinicians’ implicit attitudes about race with medical visit communication and patient ratings of interpersonal care. American Journal of Public Health, 102, 979–987. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Dasgupta N, & Asgari S (2004). Seeing is believing: Exposure to counterstereotypic women leaders and its effect on the malleability of automatic gender stereotyping. Journal of Experimental Social Psychology, 40, 642–658. [Google Scholar]
  11. Davis MA (1980). A multidimentional approach to individual differences in empathy. JSAS Catalogue of Selected Documents in Psychology, 10, 85. [Google Scholar]
  12. Devine PG, Forscher PS, Austin AJ, & Cox WTL (2012). Long-term reduction in implicit race bias: A prejudice habit-breaking intervention. Journal of Experimental Social Psychology, 48, 1267–1278. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Dovidio JF, & Fiske ST (2012). Under the radar: How unexamined biases in decision-making processes in clinical interactions can contribute to health care disparities. American Journal of Public Health, 102, 945–952. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Dovidio JF, Kawakami K, & Gaertner SL (2002). Implicit and explicit prejudice and interracial interaction. Journal of Personality and Social Psychology, 82, 62–68. [DOI] [PubMed] [Google Scholar]
  15. Fiske S & Neuberg S (1990). A continuum of impression formation, from category-based to individuating processes: Influences of information and motivation on attention and interpretation. Advances in Experimental Social Psychology. 23, 1–74. [Google Scholar]
  16. Forscher PS, Lai CK, Axt J, Ebersole CR, Herman M, Devine PG, & Nosek BA (2018, July 13). A Meta-Analysis of Procedures to Change Implicit Measures. 10.31234/osf.io/dv8tu [DOI] [PMC free article] [PubMed]
  17. Gaertner SL, Rust MC, Dovidio JF, Bachman BA, & Anastasio PA (1994). The contact hypothesis: The role of common in-group identity on reducing intergroup bias. Small Group Research, 25, 224–249. [Google Scholar]
  18. Gawronski B, & Bodenhausen GV (2006). Associative and propositional processes in evaluation: An integrative review of implicit and explicit attitude change. Psychological Bulletin, 132, 692–731. [DOI] [PubMed] [Google Scholar]
  19. Gonsalkorale K, von Hipple W, Sherman JW, & Klauer KC (2009). Bias and regulation of bias in intergroup interactions: Implicit attitudes toward Muslims and interaction quality. Journal of Experimental Social Psychology, 45, 161–166. [Google Scholar]
  20. Gonzalez CM, Kim MY, & Marantz PR (2014). Implicit bias and its relation to health disparities: A teaching program and survey of medical students. Teaching and Learning in Medicine, 26, 64–71. [DOI] [PubMed] [Google Scholar]
  21. Green AR, Carney DR, Pallin DJ, Ngo LH, Raymond KL, Lezzoni LI, & Banaji MR (2007). Implicit bias among physicians and its prediction of thrombolysis decisions for black and white patients. Journal of General Internal Medicine, 22, 1231–1238. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Greenwald AG, McGhee DE, & Schwartz JLK (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464–1480. [DOI] [PubMed] [Google Scholar]
  23. Greenwald AG, Nosek BA, Banaji MR (2003). Understanding and using the Implicit Association Test: I. An improved scoring algorithm. Journal of Personality and Social Psychology, 85, 197–216. [DOI] [PubMed] [Google Scholar]
  24. Hagiwara N, Penner LA, Gonzalez R, Eggly S, Dovidio JF, Gaertner SL, West T, & Albrecht TL (2013). Racial attitudes, physician-patient talk time ratio, and adherence in racially discordant medical interactions. Social Science & Medicine, 87, 123–131. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Heine SJ (2008). Cultural psychology. New York: W. W. Norton. [Google Scholar]
  26. Hofstede G (2001). Culture’s consequences: Comparing values, behaviors, institutions, and organizations across nations (2nd ed.). Thousand Oaks, CA: Sage. [Google Scholar]
  27. Howell JL, Gaither SE, & Ratliff KA (2015). Caught in the middle: Defensive responses to IAT feedback among Whites, Blacks, and Biracial Black/Whites. Social Psychological and Personality Science, 6, 373–381. [Google Scholar]
  28. Inquisit 4.0.6 [Computer software]. (2014). Retrieved from https://www.millisecond.com
  29. Lai CK, Marini M, Lehr SA, Cerruti C, Shin JL, Joy-Gaba J,S, . . . Nosek BA. (2014). Reducing implicit racial preferences: I. A comparative investigation of 17 interventions. Journal of Experimental Psychology: General,143, 1765–1785. [DOI] [PubMed] [Google Scholar]
  30. Lai CK, Skinner AL, Cooley E, Murrar S, Brauer M, Devos T, . . . Nosek BA (2016). Reducing implicit racial preferences: II. Intervention effectiveness across time. Journal of Experimental Psychology: General, 145, 1001–1016. [DOI] [PubMed] [Google Scholar]
  31. Maina IW, Belton TD, Ginzberg S, Singh A, Johnson TJ. (2018). A decade of studying implicit racial/ethnic bias in health care providers using the implicit association test. Social Science and Medicine. 199, 219–229. [DOI] [PubMed] [Google Scholar]
  32. Ma-Kellams C, & Blascovich J (2012). Reading the emotions of friends vs. strangers: Culture and self-construal moderate empathie accuracy. Personality and Social Psychology Bulletin, 38, 933–945. [DOI] [PubMed] [Google Scholar]
  33. Moskowitz GB (2010). On the control over stereotype activation and stereotype inhibition. Social and Personality Psychology Compass, 4, 140–158. [Google Scholar]
  34. Moskowitz GB, Stone J, & Childs A (2012). Implicit stereotyping and medical decisions: Unconscious stereotype activation in practitioners’ thoughts about African Americans. American Journal of Public Health, 102, 996–1001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Moss-Racusin C, Toorn J, Dovidio J, Brescoll V, Graham M & Handelsman J (2014). Scientific diversity interventions. Science, 343 615–616. [DOI] [PubMed] [Google Scholar]
  36. Neel R, & Shapiro JR (2012). Is racial bias malleable? Whites’ lay theories of racial bias predict divergent strategies for interracial interactions. Journal of Personality and Social Psychology, 103, 101–120. [DOI] [PubMed] [Google Scholar]
  37. Nosek BA, Smyth FL, Hansen JJ, Devos T, Lindner NM, Ranganath KA, Smith CT, Olson KR, Chugh D, Greenwald AG, & Banaji MR (2007). Pervasiveness and correlates of implicit attitudes and stereotypes. European Review of Social Psychology, 18(1), 36–88. [Google Scholar]
  38. Penner LA, Dovidio JF, West TV, Gaertner SL, Albrecht TL, Dailey RK, & Markova T (2010). Aversive Racism and Medical Interactions with Black Patients: A Field Study. Journal of experimental social psychology, 46(2), 436–440. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Penner LA, Dovidio JF, Gonzalez R, Albrecht TL, Chapman R, Foster T, ... Eggly S (2016). The effects of oncologist implicit racial bias in racially discordant oncology interactions. Journal of Clinical Oncology, 34, 2874–2880. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Penner LA, Blair IV, Albrecht TL, & Dovidio JF (2014). Reducing racial health care disparities: A social psychological analysis. Policy Insights from the Behavioral and Brain Sciences, 1, 204–212. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Phelan SM, Dovidio JF, Puhl RM, Burgess DJ, Nelson DB, Yeazel MW, . . . van Ryn M (2014). Implicit and explicit weight bias in a national sample of 4,732 medical students: The medical student CHANGES study. Obesity, 22, 1201–1208. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Plant A, & Devine P (1998). Internal and external motivation to respond without prejudice. Journal of Personality and Social Psychology, 75, 811–832. [DOI] [PubMed] [Google Scholar]
  43. Rubineau B, & Kang Y (2012) Bias in White: A longitudinal natural experiment measuring changes in discrimination. Management Science, 58, 660–677. [Google Scholar]
  44. Rudman LA, Ashmore RD, & Gary ML (2001). “Unlearning” automatic biases: The malleability of implicit prejudice and stereotypes. Journal of Personality and Social Psychology, 81, 856–868. [DOI] [PubMed] [Google Scholar]
  45. Sabin DJA, Nosek DBA, Greenwald DAG, & Rivara DFP (2009). Physicians’ implicit and explicit attitudes about race by MD race, ethnicity, and gender. Journal of health care for the poor and underserved, 20(3), 896. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Sabin JA, & Greenwald AG (2012). The influence of implicit bias on treatment recommendations for 4 common pediatric conditions: Pain, urinary tract infection, attention deficit hyperactivity disorder, and asthma. American Journal of Public Health, 102, 988–995. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Sabin JA, Marini M, & Nosek BA (2012). Implicit and explicit anti-fat bias among a large sample of medical doctors by BMI, race/ethnicity and gender. PLoS ONE, 7 e48448. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Sabin JA, Riskind RG, & Nosek BA (2015). Health care providers’ implicit and explicit attitudes toward lesbian women and gay men. American Journal of Public Health, 105, 1831–1841. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Schiller JH, Bowden CJ, Mills J, Lang E, Dickson H, Hamann HA, & Sriram N (2013). Explicit and implicit attitudes toward lung cancer (LC) relative to breast cancer (BC). Journal of Clinical Oncology, 31, 8017. [Google Scholar]
  50. Schultz PL, & Baker J (2017). Teaching strategies to increase nursing student acceptance and management of unconscious bias. Journal of Nursing Education, 56, 692–696. [DOI] [PubMed] [Google Scholar]
  51. Stone J, & Moskowitz GB (2011). Nonconscious racial bias in medical decision-making: What can be done to reduce it? Medical Education, 45, 768–776. [DOI] [PubMed] [Google Scholar]
  52. Teal CR, Gill AC, Green AR & Crandall S (2012), Helping medical learners recognize and manage unconscious bias toward certain patient groups. Medical Education, 46: 80–88. [DOI] [PubMed] [Google Scholar]
  53. Teal CR, Shada RE, Gill AC, Thompson BM, Fruge E, Villarreal GB, & Haidet P (2010). When best intentions aren’t enough: Helping medical students develop strategies for managing bias about patients. Journal of General Internal Medicine, 25 (Suppl 2), 115–118. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Todd AR, & Galinsky AD (2014). Perspective-taking as a strategy for improving intergroup relations: Evidence, mechanisms, and qualifications. Social and Personality Psychology Compass, 8, 374–387. [Google Scholar]
  55. Triandis HC (1996). The psychological measurement of cultural syndromes. American Psychologist, 51, 407–415. [Google Scholar]
  56. van Ryn M, Hardeman R, Phelan SM, Dovidio JF, Herrin J, Burke SE, . . . Przedworski JM. (2015). Medical school experiences associated with change in implicit racial bias among 3547 students: A medical student CHANGES study report. Journal of General Internal Medicine, 30, 1748–1756. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Zestcott CA, Blair IV, & Stone J (2016). Examining the presence, consequences, and reduction of implicit bias in health care: A narrative review. Group Processes & Intergroup Relations, 19, 528–542. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Material
NIHMS1031406-supplement-Supplemental_Material.pdf (26.7KB, pdf)
Supplemental Tables
NIHMS1031406-supplement-Supplemental_Tables.pdf (161.5KB, pdf)

RESOURCES