Abstract
Background and Objectives
Medical schools must integrate educational curricula that teach how to apply quality improvement principles to improve care for vulnerable populations. In this report, we describe the development, implementation, and evaluation of a combined quality improvement (QI) and health disparities curriculum for third-year family medicine clerkship students.
Methods
After conducting an educational needs assessment, we developed a health disparities curriculum focused on QI principles for the family medicine clerkship. From November 2019 through August 2021, third-year medical students (N=395) completed the curriculum. The curriculum was delivered in an asynchronous online format, followed by a small group collaboration project to design and present a QI intervention through process mapping. Students also completed an individual reflection assignment that focused on care for vulnerable populations. Pre- and post assessment questions were administered on Qualtrics, after review by the clerkship director, research faculty and staff, and content experts for content and item validity. We analyzed quantitative data using SPSS version 27 software and used paired t tests for pre/post comparisons.
Results
In total, 392 students completed the preassessment survey, 395 students completed the postassessment surveys, and 341 had matching study identifiers. Pre-to-post assessment survey evaluations showed statistically significant changes for nine out of nine QI knowledge questions (P<.001), knowledge regarding a community health needs assessment (P<.001), and knowledge about caring for vulnerable populations (homeless, veterans, immigrants/refugees; P<.001).
Conclusions
Preliminary evaluation of a combined QI and health disparities curriculum shows improvement in students’ self-reported knowledge of use of a community health needs assessment, QI principles, and care for vulnerable populations.
Introduction
Despite improvements in population health efforts, health disparities persist.1 Medical education must teach students how to recognize and address health disparities to work toward health equity. One way to achieve this is to train students to apply quality improvement (QI) principles to the care of vulnerable populations.
Health equity curricula should be integrated with clinical context.2 The report Health Professions Education: A Bridge to Quality identified the application of QI methodology to patient care as a required competency,3 while the Unequal Treatment report highlighted nationwide health disparities.4 The National Healthcare Quality and Disparities Report1 reinforced the importance of combining QI with disparities reduction efforts.
In a 2020 survey of 41 US medical schools, 100% of surveyed schools included curriculum on health disparities and 83% of schools taught QI; however, only 16 of the 41 schools required students to complete a QI project.5 We developed a curriculum for third-year family medicine clerkship students involving how to (1) navigate a community health needs assessment, (2) recognize and identify health disparities, and (3) apply QI principles to design an intervention that would benefit vulnerable populations. This paper will describe the development, implementation, and evaluation of this curriculum.
Methods
Curriculum Design and Implementation
Previously, students on the family medicine clerkship were introduced to a QI curriculum.6 To assess students’ knowledge of community-based medicine, we designed a needs assessment described in a previous study.7 We used this information to develop a health disparities curriculum with the following asynchronous online lessons: introduction to a community health needs assessment; and care for homeless, veteran, immigrant/refugee, and sexual and gender minority individuals. The community health needs assessment (CHNA) lesson contained information on evaluation of data sources, the process of how a CHNA is created, identification of priority needs, demographics, and action planning. The lessons on care for vulnerable populations included sections on demographics, historical context, health disparities, improving clinical care, and community resources. Students were asked to design a QI intervention to improve a health care process and to complete a reflection assignment that focused on improving care for a vulnerable population. Table 1 outlines the curriculum. From November 2019 through August 2021, third-year medical students (N=395) completed the curriculum during their 6-week family medicine clerkship.
Table 1.
Curriculum Overview
| Component | Description | Activity |
|---|---|---|
| Institute for Healthcare Improvement 102 module | How to improve with the Model for Improvement | Upload completed certificate |
| Plan-do-study-act video | Video created by faculty member explaining the basics of the Plan-Do-Study-Act Model, a process for change | N/A |
| Community health needs assessment (CHNA) | Teaches about community health needs assessment—evaluation of data sources, the process of how a CHNA is created, identification of priority needs, demographics, and action planning | Worksheet to navigate content from the 2019 University’s Hospital CHNA16 Community Commons17 worksheet (students create CHNA on location of choice) |
Health disparities in vulnerable populations
|
Provides an overview of the populations’ demographics, historical background, health disparities, ways to improve clinical care, and community resources | Quiz on contents of the lesson |
| Group Shark Tank-style presentation | Students pitch their QI intervention to practice leadership and faculty, who provide hypothetical seed funding for projects. Faculty evaluate the intervention and challenge students to consider aspects of implementation such as prioritization, feasibility, and cost | Develop a QI question and present a process map outlining an area of improvement |
| Reflection assignment | Students are asked to reflect what barriers the vulnerable population they learned about would face in the QI intervention, ways to screen for and address these barriers, and about the value of learning about health disparities in their education | Complete an individual reflection assignment |
Data Collection and Analysis
Pre- and postassessment questions with demographics were developed and reviewed by the clerkship director, research faculty, and staff, to ensure alignment with goals and objectives. Content experts also reviewed the questions to ensure validity. The questions asked about students’ self-assessed knowledge of QI principles, how to use a CNHA, and how to care for vulnerable populations. Both surveys were administered on Qualtrics, with pre- and post matching with response scales ranging from 1 (strongly disagree) to 5 (strongly agree). Students were not required to answer all questions, and surveys were anonymous. We analyzed quantitative data using SPSS version 27 software and used paired t tests for pre/post comparisons. We used linear regression to assess whether any demographic (age, gender) or background variables (being a first generation student vs not, raised in an urban environment vs not, or intention to participate in a loan forgiveness program vs not) were associated with changes in students’ knowledge. Thomas Jefferson University’s Institutional Review Board approved this study.
Results
In total, 392 students completed the preassessment survey, 395 students completed the postassessment surveys, and 341 had matching study identifiers (Table 2). There were no demographic differences between those who completed both surveys or only one survey, according to χ2 comparisons. Participants were primarily White (59.53%), female (51.03%), and under the age of 26 years (75.37%). Pre- to postassessment survey evaluations showed statistically significant changes for nine out of nine QI knowledge questions, including formulation of a QI question to address health disparities for a vulnerable population (P<.001), knowledge regarding a community health needs assessment (P<.001), and knowledge to care for vulnerable populations (homeless, veterans, immigrants/refugees; P<.001). Care for individuals who are sexual and gender minorities was not significant, due to the lower number of students (N=24) who had taken that lesson thus far (Table 3). The effect sizes of their self-assessed increased knowledge ranged from moderate (d=0.57) to very large (d=1.43). We used additional regression analyses to examine the factors of age, gender, being a first generation student, community size, or participation in a loan forgiveness program; none of these were significantly related to the knowledge change scores.
Table 2.
Sample Characteristics of Medical Student Participants (N=341) Compared to Graduates of Institution and Medical School Matriculants Nationally 2021–2022
| Demographics | Participants n (%) |
Institution n (%) |
National n (%)* |
|---|---|---|---|
| Age in Years | |||
| 21–23 | 30 (8.80) | ||
| 24–26 | 227 (65.57) | ||
| 27–29 | 64 (18.77) | ||
| 30–39 | 29 (8.50) | ||
| Gender | |||
| Men | 162 (47.51) | 125 (48.45) | 10057 (44.37) |
| Women | 174 (51.03) | 133 (51.55) | 12589 (55.55) |
| Nonbinary, genderqueer | 2 (0.59) | ||
| Choose not to answer | 3 (0.88) | ||
| Race | |||
| White | 203 (59.53) | 171 (66.28) | 11682 (51.54) |
| Black | 1 (0.29) | 7 (2.71) | 2562 (11.30) |
| Asian | 81 (23.75) | 64 (24.80) | 6994 (26.49) |
| Latinx | 14 (4.11) | 15 (5.81) | 2869 (12.66) |
| Other/Mixed Race | 32 (9.38) | 9 (3.49) | 884 (3.90) |
| First Generation | |||
| Yes | 34 (9.97) | ||
| No | 299 (87.68) | ||
| Don’t know | 1 (0.29) | ||
| Choose not to answer | 7 (2.05) | ||
| Population of Community of Origin | |||
| Rural area (<2,500) | 12 (3.52) | ||
| Town (2,500–50,000) | 130 (38.12) | ||
| Small city (50,001–100,000) | 54 (15.84) | ||
| City (100,001–500,000) | 36 (10.56) | ||
| Metropolitan area (500,001–1,000,000) | 21 (6.16) | ||
| Major metropolitan area (>1,000,000) | 51 (14.96) | ||
| Desire to Work With Underserved Populations | |||
| Yes | 67 (19.65) | ||
| No | 84 (24.63) | ||
| Unsure | 186 (54.55) | ||
| Desire to Work in Underserved Areas | |||
| Yes | 56 (16.42) | ||
| No | 88 (25.81) | ||
| Unsure | 193 (56.60) |
Source: AAMC Applicant Matriculant Data File and AAMC Student Records System as of November 17, 2021.
Note: In z-score proportion comparisons (P<.05), there were no gender differences in response between the sample and the gender composition of the institution or nationally. Our sample was overrepresentative of White and other/mixed-race students and underrepresentative of Black and Latinx students compared to students nationally. In our sample, Black students were underrepresented and other/mixed-race students were overrepresented compared to the institution. No proportional differences were found among Asian students.
Table 3.
Paired t-Test Comparisons of Pre/Post Self-assessed Knowledge of Quality Improvement, Needs Assessments, and Care for Vulnerable Populations (N=337)
| Pre M (SD) | Post M (SD) | t | P | d | |
|---|---|---|---|---|---|
| Quality Improvement and Needs Assessment | |||||
| I can explain the differences between research and QI. | 2.77 (.63) | 3.41 (.56) | −15.83 | <.001 | −0.86 |
| I can identify the ways that clinicians are involved in QI on a daily basis. | 2.69 (.65) | 3.37 (.58) | −16.7 | <.001 | −0.91 |
| I understand the role of systems changes in QI. | 2.78 (.65) | 3.40 (.57) | −15.38 | <.001 | −0.84 |
| I can describe the stages of a plan-do-study-act (PDSA) cycle. | 2.27 (.78) | 3.45 (.55) | −25.17 | <.001 | −1.37 |
| I can identify the key elements of an effective aim statement for a QI project. | 2.38 (.67) | 3.44 (.54) | −24.7 | <.001 | −1.35 |
| I can map the processes related to caring for patients. | 2.48 (.66) | 3.34 (.57) | −21.44 | <.001 | −1.31 |
| I can explain how to use change concepts to come up with good ideas to test. | 2.39 (.64) | 3.34 (.55) | −23.68 | <.001 | −1.43 |
| I can implement a QI project. | 2.42 (.65) | 3.28 (.57) | −21.92 | <.001 | −1.33 |
| I can formulate a quality improvement question that addresses the health disparities affecting a vulnerable population. | 2.70 (.61) | 3.32 (.51) | −16.87 | <.001 | −0.92 |
| I can identify the steps that are needed to conduct a community health needs assessment. | 2.30 (.66) | 3.32 (.56) | −24.73 | <.001 | −1.35 |
| I feel confident in navigating the sections of a community health needs assessment to find information pertinent to a specific population. | 2.28 (.66) | 3.33 (.56) | −25.38 | <.001 | −1.38 |
| Knowledge to Care for Vulnerable Populations | |||||
| Homeless (n=185) | 2.69 (.63) | 3.35 (.52) | −11.63 | <.001 | −0.86 |
| Veterans (n=62) | 2.53 (.76) | 3.37 (.55) | −7.42 | <.001 | −0.94 |
| Immigrants/refugees (n=93) | 2.67 (.70) | 3.43 (.58) | −8.17 | <.001 | −0.85 |
| Sexual and gender minorities (n=24) | 2.88 (.68) | 3.38 (.50) | −2.77 | 0.011 | −0.57 |
The threshold for significance was adjusted for multiple comparisons within each domain. All quality improvement and needs assessment items were below the adjusted threshold (P=.05/10=.005) and all the vulnerable populations items were below the adjusted threshold (P=.05/4=.0125). Students were able to select one of the four vulnerable populations curricula to complete, and a new lesson was created annually. Populations are presented in the order that they became available and the number of students who completed each curriculum is noted within the table. Effect sizes of .5-.79 or higher are moderate and .80 or larger are considered large effect sizes.
Process Maps
Figure 1 shows a sample process map created by a student group that focuses on improving transportation access for elderly/disabled patients. During the group presentations of their QI projects, students were asked to consider aspects such as cost, feasibility, prioritization, and stakeholder engagement. Particularly, students were challenged to think about how these interventions could be implemented, with a focus on the importance of a multidisciplinary care team. Other examples of student QI interventions included improving behavioral health linkage for individuals with mental illness, connecting diabetic patients to a registered dietician, improving interpreter services to ensure continuity during a visit, and connecting patients with a chronic disability with food delivery services.
Figure 1.
Sample Quality Improvement Process Map
This process map focuses on improving transportation access for elderly/disabled patients. Customized Community Transportation (CCT) is a shared ride program for individuals over 65 years or who have disabilities.
Discussion
Similar to community medicine curricula after which students report an improvement in perceived knowledge about health disparities,8–10 our students also reported a significant increase in their knowledge on use of a community health needs assessment, QI principles, and care for vulnerable populations. Importantly, after completion of this curriculum, they reported having the knowledge to create a QI question that addresses health disparities in a vulnerable population. A prior review found that approximately one-third of published QI interventions targeted improving care for vulnerable populations.11
During the COVID-19 pandemic, students reflected that they felt empowered to be advocates for vulnerable populations and to address health inequities, but they wanted to learn action-oriented tools and strategies as a next step.12 We asked our students to synthesize the information they learned in this curriculum to design a QI intervention and reflect on whether this intervention would need to change based on the vulnerable populations they chose to learn more about. Taking a step further, students had to consider barriers to care for these patients, and how they might create solutions to accommodate patients’ needs. A future goal is for our fourth-year students in the family medicine track to have the opportunity to implement their QI project in their final year. Regarding the design, implementation, and evaluation of their QI projects, students could be assessed on Entrustable Professional Activity 13: Identify System Failures and Contribute to a Culture of Improvement13; and/or the family medicine milestones of Systems-Based Practice 1: Patient Safety and Quality Improvement; Systems-Based Practice 3: Physician Role in Health Care Systems; or Systems-Based Practice 4: Advocacy.14
Due to the 6-week time frame of our clerkship, we only asked students to propose a QI question, but not implement it. Therefore, our pre- and postcurricular surveys have a major limitation in only assessing students’ self-report of knowledge, which is affected by response bias and assesses the lowest level on Kirkpatrick’s level of learning evaluation.15 A future goal would be for students to implement their intervention, so that we could objectively assess skill acquisition or competency attainment. Other limitations include our use of a single-institution sample that was overrepresentative of White and Other/Mixed-race students, and underrepresentative of Black and Latinx students compared to students nationally (Table 2).
Overall, this curriculum is an important first step in teaching medical students to recognize disparities, to design a quality improvement intervention with a focus on equity, and how to adjust an intervention to ensure vulnerable populations can benefit. While the focus of our program was on medical education, this curriculum could be adapted for other health professional students, residents, or faculty/community preceptors.
Acknowledgments
Financial Support and Disclaimer: This work was funded by the Health Resources and Services Administration (HRSA) of the US Department of Health and Human Services (HHS) as part of a Primary Care Medicine and Dentistry Clinician Educator Career Development Awards Program (grant number K02HP30821). The contents are those of the author(s) and do not necessarily represent the official views of, nor an endorsement, by HRSA, HHS, or the US Government.
References
- 1.National Healthcare Quality and Disparities Report. Agency for Healthcare Research and Quality; 2019. [Accessed September 29, 2022]. https://www.ahrq.gov/research/findings/nhqrdr/nhqdr19/index.html . [Google Scholar]
- 2. Landry AM. Integrating health equity content into health professions education. AMA J Ethics. 2021 Mar 1;23(3):E229–234. doi: 10.1001/amajethics.2021.229. [DOI] [PubMed] [Google Scholar]
- 3.Greiner AC, Knebel E, editors. Health Professions Education: A Bridge to Quality. Institute of Medicine (US) Committee on the Health Professions Education Summit. National Academies Press; 2003. [PubMed] [Google Scholar]
- 4.Smedley BD, Stith AY, Nelson AR, editors. Institute of Medicine (US) Committee on Understanding and Eliminating Racial and Ethnic Disparities in Health Care. Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care. National Academies Press; 2003. [PubMed] [Google Scholar]
- 5. Morse R, Smith A, Fitzgerald-Wolff S, Stoltzfus K. Population Health in the Medical School Curriculum: a Look Across the Country. Med Sci Educ. 2020;30(4):1487–1493. doi: 10.1007/s40670-020-01083-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Mills G, Kelly S, Crittendon DR, Cunningham A, Arenson C. Evaluation of a quality improvement experience for family medicine clerkship students. Fam Med. 2021;53(10):882–885. doi: 10.22454/FamMed.2021.702090. [DOI] [PubMed] [Google Scholar]
- 7. Smith RS, Silverio A, Casola AR, Kelly EL, de la Cruz MS. Third-year medical students’ self-perceived knowledge about health disparities and community medicine. PRiMER Peer-Rev Rep Med Educ Res. 2021;5:9. doi: 10.22454/PRiMER.2021.235605. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Awosogba T, Betancourt JR, Conyers FG, et al. Prioritizing health disparities in medical education to improve care. Ann N Y Acad Sci. 2013;1287(1):17–30. doi: 10.1111/nyas.12117. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Denizard-Thompson N, Palakshappa D, Vallevand A, et al. Association of a health equity curriculum with medical students’ knowledge of social determinants of health and confidence in working with underserved populations. JAMA Netw Open. 2021;4(3):e210297. doi: 10.1001/jamanetworkopen.2021.0297. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Bernstein R, Ruffalo L, Bower D. A multielement community medicine curriculum for the family medicine clerkship. MedEdPORTAL. 2016;12:10417. doi: 10.15766/mep_2374-8265.10417. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Rolnitsky A, Kirtsman M, Goldberg HR, Dunn M, Bell CM. The representation of vulnerable populations in quality improvement studies. Int J Qual Health Care. 2018;30(4):244–249. doi: 10.1093/intqhc/mzy016. [DOI] [PubMed] [Google Scholar]
- 12. Casola AR, Kelly EL, Smith K, Kelly S, de la Cruz M. Impact of the COVID-19 pandemic on medical students’ perceptions of healthcare for vulnerable populations. Fam Med. doi: 10.22454/FamMed.2022.940208. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Obeso V, Brown D, Aiyer M, editors. Toolkits for the 13 Core Entrustable Professional Activities for Entering Residency. Association of American Medical Colleges; 2017. [Google Scholar]
- 14.Hamstra SJ, Kenji Y, Shah H, et al. ACGME Milestones National Report. 2019. [Accessed April 19, 2021]. https://www.acgme.org/Portals/0/PDFs/Milestones/2019MilestonesNationalReportFinal.pdf?ver=2019-09-30-110837-587 .
- 15.Kirkpatrick DL. The Four Levels. 3rd ed. Berrett-Koehler Publishers; 2009. Evaluating Training Programs. [Google Scholar]
- 16.Southeastern Pennsylvania Community Health Needs Assessment. [Accessed June 2, 2020]. Updated 2019. https://www.jeffersonhealth.org/content/dam/health2021/documents/informational/community-health-needs-assessment-2019.pdf .
- 17.Community Commons. [Accessed May 11, 2020]. https://www.communitycommons.org .