Abstract
BACKGROUND
As patient populations become increasingly diverse, we need to be able to measure residents’ preparedness and skillfulness to provide cross-cultural care.
OBJECTIVE
To develop a measure that assesses residents’ perceived readiness and abilities to provide cross-cultural care.
DESIGN
Survey items were developed based on an extensive literature review, interviews with experts, and seven focus groups and ten individual interviews, as part of a larger national mailed survey effort of graduating residents in seven specialties. Reliability and weighted principal components analyses were performed with items that assessed perceived preparedness and skillfulness to provide cross-cultural care. Construct validity was assessed.
PARTICIPANTS
A total of 2,047 of 3,435 eligible residents participated (response rate = 60%).
MEASUREMENTS AND MAIN RESULTS
The final scale consisted of 18 items and 3 components (general cross-cultural preparedness, general cross-cultural skillfulness, and cross-cultural language preparedness and skillfulness), and yielded a Cronbach’s alpha = 0.92. Construct validity was supported; the scale total was inversely correlated with a measure of helplessness when providing care to patients of a different culture (p < 0.001).
CONCLUSIONS
We developed a three-component cross-cultural preparedness and skillfulness scale that was internally consistent and demonstrated construct validity. This measure can be used to evaluate residents’ perceived effectiveness of cross-cultural medical training programs and could be used in future work to validate residents’ self assessments with objective assessments.
KEY WORDS: cultural competency, measurement, medical education-graduate
BACKGROUND
As the medical field becomes increasingly complex, and patients become increasingly diverse, medical education has had to adapt and develop new strategies for preparing future physicians to address these challenges. One important shift is the emphasis on improving physicians’ capacity to provide effective care for patients from a wide range of sociocultural backgrounds. This has been driven in part by the recognition that minorities and other vulnerable populations suffer worse health outcomes and disparities in health care. In its 2002 report Unequal Treatment: Confronting Racial/Ethnic Disparities in Health Care, the Institute of Medicine brought national attention to this area and recommended cross-cultural education become integrated in the training of all current and future health professionals1.
Acknowledging the importance of cross-cultural education, the Accreditation Council for Graduate Medical Education (ACGME) developed specific training criteria to ensure that residents are capable of providing quality care to diverse populations2. Cross-cultural training is addressed under two of the six competencies—Professionalism and Interpersonal and Communication Skills. The Professionalism competency states that residents must learn: “sensitivity and responsiveness to a diverse patient population, including but not limited to diversity in gender, age, culture, race, religion, disabilities, and sexual orientation.” The Interpersonal and Communication Skills competency states that residents must learn to: “communicate effectively with patients, families, and the public, as appropriate, across a broad range of socioeconomic and cultural backgrounds.” The ACGME requires that the effectiveness of such educational interventions be measured and documented, but there is no such validated measure.
Therefore, we conducted a two-phase study in which we measured preparedness and skills of resident physicians to provide quality care to diverse patient populations. In the first phase, we qualitatively interviewed 68 residents across the US3. In the second phase, these qualitative results were used to identify 18 preparedness (ability to care for specific types of patients) and skillfulness (ability to perform selected tasks or services) items and inform the wording development of these items as part of a survey that was cognitively tested and then administered to residents from seven specialty areas in the US.
The purpose of this paper is to describe the psychometric properties of the 18-item measure that assessed residents’ perceptions of how ready they were and what abilities they had to provide cross-cultural care. Our goal is to establish the measure’s internal consistency and construct validation and to set the stage for future predictive validation work.
DESIGN
Survey Design and Administration
The design of the survey and sampling has been described previously4. Survey items were developed based on findings from an extensive literature review, interviews with experts, and a national qualitative interview formative phase3,4. We conducted seven focus groups and ten individual interviews with residents across the US. This process resulted in a survey that contained two a priori measures: (1) self-perceived preparedness to care for diverse patient populations (eight questions; five-item response categories; higher scores = greater preparedness) and (2) self-assessed skillfulness to perform core techniques in the delivery of cross-cultural care (ten questions; five-item response categories; higher score = greater level of skillfulness). The instrument was cognitively tested by the Center for Survey Research at the University of Massachusetts, Boston4.
We selected a stratified random sample of graduating residents in seven specialties (emergency medicine, family practice, internal medicine, obstetrics/gynecology, pediatrics, psychiatry, and general surgery) from 1 of 121 academic health centers, defined as medical schools and their closely affiliated or owned clinical facilities. The mailed survey was administered by the Center for Survey Research in the winter and spring of 2003–2004. This resulted in 3,500 residents at 149 hospitals, including 563 different programs ranging from 43 programs sampled in pediatrics to 113 programs sampled in general surgery. Completion and return of the survey constituted consent. The protocol was approved by both the Massachusetts General Hospital and UMass Boston Institutional review boards.
Measurements and Analyses
Statistical analyses were conducted using SPSS 16.0 statistical software. All pooled analyses were weighted to correct for non-response bias and for the probability of selection within stratum (physician specialty), and the weighted distributions of gender and race/ethnicity are nearly identical to figures reported from the ACGME5. Principal components analyses (PCA) were performed on the 18 preparedness and skillfulness items; varimax, an orthogonal rotation, was used to facilitate interpretation. Scree plot analysis and eigenvalue greater than 1 were used to identify three components that were retained. Items were retained based component loadings and impact on Cronbach’s alpha; the internal consistency of the measure was assessed using a Cronbach’s alpha. A total scale score was then created by summing these items.
Four questions from the survey4 were used to assess the measure’s validity. To assess the construct validity, an ANOVA with post-hoc analyses was conducted with the total scale score and a one-item, five-response question that assessed residents’ sense of helplessness when providing care to patients of a different culture. In addition, known groups criterion validation was conducted by analyzing the total scale score according to three resident characteristics: (1) being able to treat patients in a language other than English, (2) presence of a mentor who provided good cross-cultural care, and (3) formal evaluation of doctor-patient communication, specifically about the ability to handle cross-cultural issues.
MAIN RESULTS
Responses were obtained from 2,047 of 3,435 eligible residents (response rate = 60%). Characteristics of the sample are shown in Table 1.
Table 1.
Resident Characteristics
| Number of respondents (weighted %) | |
|---|---|
| All | 2,047 (100) |
| Female gender | 1,043 (49.4) |
| Race/ethnicity | – |
| White, Non-Hispanic | 1,265 (59.9) |
| Black, Non-Hispanic | 119 (6.5) |
| Asian/Pacific Islander | 404 (5.3) |
| Native American or Alaskan Native | 65 (23.9) |
| Hispanic | 115 (4.5) |
| Born in US | 1,443 (65.8) |
| Able to treat patients in language other than English | 1,122 (58.1) |
| Specialty | – |
| Emergency medicine | 299 (9.2) |
| Family medicine | 308 (9.1) |
| General surgery | 278 (8.3) |
| Internal medicine | 271 (40.3) |
| OB/GYN | 276 (7.9) |
| Pediatrics | 291 (15.4) |
| Psychiatry | 312 (9.4) |
All proposed scale items were retained6, possessed excellent internal reliability (alpha = 0.92), and were summed (scale mean = 61.0; SD = 10.4). Three components emerged that accounted for 58% of the variance: general cross-cultural preparedness, general cross-cultural skillfulness, and language preparedness and skillfulness, with respective component loadings ranging from 0.57 to 0.75, 0.54 to 0.71, and 0.67 to 0.68 (Table 2).
Table 2.
Principal Component Factor Analysis
| Cross-cultural Preparedness and Skillfulness Scale | |
|---|---|
| Component loadings | |
| Component 1: General cross-cultural preparedness | – |
| From cultures different from own | 0.67 |
| With health beliefs at odds with Western medicine | 0.75 |
| With a distrust of the US health system | 0.75 |
| Whose religious beliefs affect treatment | 0.75 |
| Who use alternative/complementary medicine | 0.71 |
| Racial/ethnic minority | 0.57 |
| Component 2: General cross-cultural skillfulness | – |
| Determining how a new patient wants to be addressed | 0.68 |
| Taking a social history | 0.66 |
| Assessing a patient’s understanding of the cause of illness | 0.70 |
| Identifying mistrust of the system or physician | 0.68 |
| Negotiating a treatment plan | 0.66 |
| Identifying ability to read and write English | 0.54 |
| Identifying religious beliefs that might affect care | 0.65 |
| Identifying cultural customs that might affect care | 0.68 |
| Identifying how patient makes decisions with family | 0.71 |
| Component 3: Cross-cultural language preparedness and skillfulness | – |
| With limited English proficiency | 0.68 |
| New immigrants | 0.67 |
| Working effectively through a medical interpreter | 0.68 |
Residents’ total scale scores were inversely related to their self-reported level of helplessness when providing care to patients of a different culture (R = −0.30; p < 0.001). The results were monotically decreasing. Mean total scale scores according to each level of helplessness were 65.3 (SD = 10.4) for “never,” 60.9 (SD = 9.5) for “rarely,” 57.4 (SD = 10.1) for “sometimes,” and 49.6 (SD = 12.0) for “often” (posthoc analyses = p < 0.05 for each helplessness level category). Residents who reported being able to treat patients in a language other than English scored significantly higher on the scale (61.9 vs. 58.7; p < 0.001). Residents who reported having a role model for cross-cultural care (67%) scored significantly higher compared to residents who did not have a role model (62.4 vs. 57.9; p < 0.001). Lastly, among the 90% of residents who reported being evaluated for doctor-patient communication during their residency, those who reported that “a lot” of attention was paid to their ability to handle cross-cultural issues had significantly higher scale scores (71.9) compared to those for whom “very little” (58.9) or “none” (57.0) was paid (p < 0.001).
CONCLUSION
Preparing residents to care for diverse patient populations will only increase in importance over time. While the ACGME has created specific requirements related to the need for formal training for effective cross-cultural patient interaction and treatment, to our knowledge there is no reliable and validated scale to do this.
Based on previous work in the field of cross-cultural care and our own formative work for this study3, we developed items to assess final year residents’ perceived preparedness and skillfulness to deliver cross-cultural care. Our results yielded an 18-item, 3-component scale that demonstrated excellent reliability (alpha = 0.92) and supported the scale’s construct validity.
The General Cross-cultural Preparedness subscale measures perceived preparedness for dealing with a variety of cultural encounters: generically, “cultures different from own” and, more specifically, those with different health and religious beliefs, distrust of the US health system, alternative/complementary medicine, and racial/ethnic minorities. These items reflect the broad definition of “culture,” thus considering the diversity of possible patient encounters. The General Cross-cultural Skillfulness subscale measures perceived skillfulness to provide care to diverse patient populations. It is comprehensive in that it covers common cross-cultural care patient encounters in which residents should be trained. The Cross-cultural Language Preparedness and Skillfulness subscale captures items primary to many cross-cultural care encounters: those involving language issues. These results were akin to our qualitative findings3, in that residents conceptualized language issues as a discrete aspect of delivering cross-cultural care and expressed that different skills were needed to overcome language barriers compared to other cultural barriers.
There are several limitations to this work. First, the generalizability of our findings might be limited by the fact that we sampled residents from only seven specialties, and the 60% response rate could introduce bias. Second, the study was not designed as a stand-alone test of the scale’s reliability and validity, and we had only one sample from which to test its psychometric properties. Third, the scale measures residents’ perceptions of their preparedness and skillfulness and was not validated by an objective measure; self-report responses may be affected by residents’ over- or underestimation of their preparedness and skillfulness or by a social desirability bias. Although many studies have shown that residents’ self-assessments are consistent with faculty evaluations7–10, others have not11,12. Nevertheless, resident self-assessment is often used to evaluate the effectiveness of skills-based curricula13–17.
This measure could be used for several purposes. Residency programs could use the scale to assess residents’ perceived preparedness and skillfulness to treat patients in cross-cultural encounters before and after cross-cultural care training curricula and at the beginning and completion of residency programs. The scale could also be used to compare residents’ self-assessments to observational testing of their abilities in simulated patient interactions or actual clinical care to assess whether residents’ perceptions of their delivery of cross-cultural care correlate with other observer ratings.
In conclusion, the need for formal training in cross-cultural care has been well established, but measurement tools to assess these training efforts have remained elusive. Our effort to quantify residents’ perceptions of their cross-cultural care is an initial step for future work in this area. As accrediting bodies such as the Joint Commission, the Liaison Committee on Medical Education, ACGME, and state medical boards continue to integrate cross-cultural training requirements into their criteria for physician competencies, the need to evaluate the effectiveness of such initiatives with the help of validated survey tools is evident. Use of a validated tool could help institutions identify educational priorities and assess ongoing needs in cross-cultural training.
Acknowledgements
This work was supported by grant 20021803 from The California Endowment and grant 20020727 from The Commonwealth Fund. Additional support for Dr. Park was provided through the American Cancer Society’s Mentored Research Scholar Award (Park) (MRSG-005-05-CPPB). We would like to thank Ms. Jennifer Pandiscio for her assistance with preparing this manuscript.
Conflict of Interest Drs. Betancourt and Green have received honoraria from Merck Speakers’ Bureau to speak about disparities and cross-cultural care. Dr. Park reports that Pfizer has supplied medication for a pilot smoking cessation trial. Dr. Chun and Dr. Weissman do not have any potential conflicts of interest to report.
References
- 1.Unequal treatment: Confronting racial and ethnic disparities in health care (free executive summary) [homepage on the Internet]. Available from: http://www.nap.edu/catalog/10260.html. Accessed 06/03/09.
- 2.Accreditation Council for Graduate Medical Education. Common program requirements: General competencies. [homepage on the Internet]. Available from: http://acgme.org/acWebsite/dutyhours/dh_dutyhoursCommonPR07012007.pdf. Accessed 06/03/09.
- 3.Park ER, Betancourt JR, Kim MK, Maina AW, Blumenthal D, Weissman JS. Mixed messages: Residents’ experiences learning cross-cultural care. Acad Med. 2005;80(9):874–80. [DOI] [PubMed]
- 4.Weissman JS, Betancourt J, Campbell EG, Park ER, Kim M, Clarridge B, et al. Resident physicians’ preparedness to provide cross-cultural care. JAMA. 2005;294(9):1058–67. [DOI] [PubMed]
- 5.AMA - FREIDA online [homepage on the Internet]. Available from: http://www.ama-assn.org/ama/pub/education-careers/graduate-medical-education/freida-online.shtml. Accessed 06/03/09.
- 6.Nunnally JC, Bernstein IH. Psychometric theory. New York: McGraw-Hill; 1994.
- 7.Fincher RM, Lewis LA, Kuske TT. Relationships of interns’ performances to their self-assessments of their preparedness for internship and to their academic performances in medical school. Acad Med. 1993;68(2 Suppl):S47–50. [DOI] [PubMed]
- 8.Roberts KB, Starr S, DeWitt TG. The University of Massachusetts Medical Center office-based continuity experience: Are we preparing pediatrics residents for primary care practice? Pediatrics. 1997;100(4):E2. [DOI] [PubMed]
- 9.Schubert A, Tetzlaff JE, Tan M, Ryckman JV, Mascha E. Consistency, inter-rater reliability, and validity of 441 consecutive mock oral examinations in anesthesiology: Implications for use as a tool for assessment of residents. Anesthesiology. 1999;91(1):288–98. [DOI] [PubMed]
- 10.Biernat K, Simpson D, Duthie E Jr, Bragg D, London R. Primary care residents self assessment skills in dementia. Adv Health Sci Educ Theory Pract. 2003;8(2):105–10. [DOI] [PubMed]
- 11.Wayne DB, Butter J, Siddall VJ, Fudala MJ, Wade LD, Feinglass J, et al. Graduating internal medicine residents’ self-assessment and performance of advanced cardiac life support skills. Med Teach. 2006;28(4):365–9. [DOI] [PubMed]
- 12.Davis JD. Comparison of faculty, peer, self, and nurse assessment of obstetrics and gynecology residents. Obstetrics and Gynecology. 2002;99(4):647–51. [DOI] [PubMed]
- 13.Moskovic CS, Guiton G, Chirra A, Nunez AE, Bigby J, Stahl C, et al. Impact of participation in a community-based intimate partner violence prevention program on medical students: A multi-center study. J Gen Intern Med. 2008;23(7):1043–7. [DOI] [PMC free article] [PubMed]
- 14.Chou B, Bowen CW, Handa VL. Evaluating the competency of gynecology residents in the operating room: Validation of a new assessment tool. Am J Obstet Gynecol. 2008;199(5):571.e1–571.e5. [DOI] [PubMed]
- 15.Han PK, Keranen LB, Lescisin DA, Arnold RM. The palliative care clinical evaluation exercise (CEX): An experience-based intervention for teaching end-of-life communication skills. Acad Med. 2005;80(7):669–76. [DOI] [PubMed]
- 16.Adams KE, O’Reilly M, Romm J, James K. Effect of Balint training on resident professionalism. Am J Obstet Gynecol. 2006;195(5):1431–7. [DOI] [PubMed]
- 17.Juarez JA, Marvel K, Brezinski KL, Glazner C, Towbin MM, Lawton S. Bridging the gap: A curriculum to teach residents cultural humility. Fam Med. 2006;38(2):97–102. [PubMed]