Abstract
Background
There has been a steady decrease in the number of physician-scientists and a lack of diversity and inclusion of underrepresented minorities (URMs) in medicine.
Objective
To assess the research productivity, interest, and experience of medical students, including URMs, and resident and faculty mentors of the Kaiser Permanente Oakland Medical Center’s 8-week, intensive, mentored Summer Clinical Otolaryngology and Obstetrics/Gynecology Research (SCORE) Program for second-year medical students.
Methods
A database of SCORE Program research projects was generated from 2016, when the program was launched, through 2018. SCORE Program students and faculty completed a brief, mixed-methods, anonymous exit survey that captured respondents’ experiences, perceived program strengths, and opportunities for improvement. The number of peer-reviewed manuscripts produced were counted.
Results
A total of 16 SCORE Program students (50% female and 38% URMs) between 2016 and 2018 and 8 residents and 8 faculty members in 2018 completed a brief, mixed-methods, anonymous exit survey that captured respondents’ experiences, perceived program strengths, and opportunities for improvement. The medical students coauthored 12 published peer-reviewed articles, 25 abstracts or national posters, and 1 opinion editorial. According to the program’s annual exit survey, 87% of students reported an increased interest in pursuing research, 93% had an increased interest in their respective specialties, and 93% believed they had positive mentorship experiences. Similarly, faculty and residents enjoyed mentoring students, thought that students positively contributed to their projects, and would support the SCORE Program in the future.
Conclusion
Structured research and mentoring experiences, such as the SCORE Program, may encourage students, including URMs, to pursue research throughout their careers in addition to adding research to their curriculum vitae to strengthen their residency applications.
Keywords: medical education, medical student, mentor, physician-scientist, research, underrepresented in medicine
INTRODUCTION
The number of physician-scientists has been steadily decreasing for decades, with more physicians opting to pursue purely clinical careers.1 Physician-scientists, through their direct patient interactions and experience in clinical settings, are ideally positioned to identify clinically driven research topics and translate research findings into direct patient care. In addition, despite efforts to improve representation of minority persons in medicine, only 7% of full-time academic faculty within medical schools based in the US identify as an underrepresented minority (URM).2 Training in research and clinical care to a racially and ethnically and socioeconomically diverse trainee population will not only help physician-scientists deliver evidence-based, data-driven care to patients,3 helping to produce a population of physician-scientists capable of identifying a diversity of areas in need of further investigation, but also build a physician-scientist population that better reflects the increasingly diverse patient population.4
In response to the decreasing number of physician-scientists and given the small number of participants in medical and doctoral programs, the medical education system is unable to meet the physician-scientist pool demand.5 There has been a push to incorporate research opportunities into residency training settings. Although these programs train participants in research skills and critical thinking, they are often underfunded and occur too late in training to influence physicians to further pursue research.6,7 In addition, although efforts have been made to improve diversity in academic medicine, including the establishment of mentorship programs and the targeting of undergraduate (and high school) students, these research programs do not take into consideration the diversity of their trainees.8–10
In an effort to reach physicians earlier in their careers, several research programs now exist specifically to train medical students. Medical schools that have implemented mandatory research in the students’ third year found that a greater number of graduating students were interested in research if they had positive experiences with mentors or success in publication.11–13 There are also several research programs geared toward medical students in the summer between their first and second years of school. Students participating in these highly structured and supportive programs reported greater interest in research and were more likely to conduct research after medical school.12–14
Kaiser Permanente Oakland Medical Center initiated the 8-week Summer Clinical Otolaryngology and Obstetrics/Gynecology Research (SCORE) Program in 2016 to provide early clinical exposure and research training to second-year medical students. The SCORE Program is unique in that it attempts to provide early clinical research exposure as well as serve as a pipeline for URMs in medicine. The Association of American Medical Colleges defines URMs as “racial and ethnic minorities in a designated health profession discipline relative to the percentage of that racial or ethnic group in the total population.”15
The aim of this initial evaluation of the SCORE Program students and resident and faculty mentors was to identify the scholarly activity productivity of the program, assess the program’s impact on student interest in pursuing residencies and research in otolaryngology as well as obstetrics/gynecology (OB/GYN), identify areas for improvement, and track the number of applicants to the program. In addition, perceived resident and faculty benefits were assessed.
METHODS
The SCORE Program is an 8-week, intensive summer program at the Kaiser Permanente Oakland Medical Facility for 6 second-year medical students. In its first year (2016), the SCORE Program accepted 4 second-year medical students to participate in research in otolaryngology. In 2017, SCORE expanded to include 2 students in OB/GYN research. The program continued with 6 students in 2018. Invitations to apply online were sent to medical schools across the US in the winter via graduate medical offices, and participants were selected in the spring.
During the program, students were matched with faculty and resident mentors on the basis of student research interest. The students assisted with ongoing research projects at various stages, supporting resident and faculty scholarly activity efforts via literature searches, data collection, data analysis, medical record review, abstract and manuscript writing, and/or oral or poster presentation development and delivery. To provide a diverse range of experiences, students in OB/GYN were encouraged to participate in multiple projects. In otolaryngology, students were assigned to 1 research project and mentor.
Throughout the program, approximately 75% of student time was dedicated to participation in research, whereas the remainder was devoted to clinical observation and didactic learning. Students observed a variety of clinics and operations within their department to better understand different career paths available within the specialty. In addition, students attended resident didactics, grand rounds, and morbidity and mortality conferences. Faculty and resident mentors provided weekly guidance and feedback, and students attended a weekly lunchtime lecture series on research methods and professional development. Guest speakers covered research-related topics, including library database searching, patient safety and quality improvement, research methods, biostatistics, and epidemiology, as well as topics focused on professional development, including health disparities, introduction to integrated health care, and a resident panel designed to answer student questions about life as a resident.
At the end of the 8-week program, students gave an oral presentation to the key SCORE Program faculty, residents, and students on their primary research project. The students completed a brief mixed-methods anonymous exit survey before program departure (2016–2018, see Supplementary Material Sidebar: Kaiser Permanente SCORE Exit Survey for Students— Anonymous Survey, available at: www.thepermanentejournal.org/files/2020/19.153-Supp). In 2018, a survey was also created for key faculty and residents (see Supplementary Material Sidebar: Kaiser Permanente SCORE Exit Survey for Residents and Faculty, available at: www.thepermanentejournal.org/files/2020/19.153-Supp). The surveys contained a combination of multiple choice and open-ended questions that evaluated the SCORE Program experience and the program’s strengths and opportunities for improvement.
This research project was deemed exempt by the Kaiser Permanente Northern California Research Determination Office. Data analyses were conducted using Microsoft Excel (Microsoft Corp, Redmond, WA), or Stata software (StataCorp, College Station, TX). Quantitative data were aggregated and analyzed using descriptive statistics.
RESULTS
From 2016 to 2018, 16 students participated in the SCORE Program. In 2016, fewer than 20 students applied for 4 slots, more than 70 applied for 6 positions in 2017, and nearly 90 applied for 6 positions in 2018. More than half of the students selected were women (50%), and more than one-third were URMs (37.5%) (Table 1). Most students were from out-of-state medical schools (75%).
Table 1.
SCORE Program student demographics
| Year | URMs, No. (%) | |
|---|---|---|
| Otolaryngology | Obstetrics and gynecology | |
| 2016 | 1 (25) | NA |
| 2017 | 1 (25) | 2 (100) |
| 2018 | 1 (25) | 1 (50) |
NA = not applicable; SCORE = Summer Clinical Otolaryngology and Obstetrics/Gynecology Research; URM = underrepresented minority.
The SCORE Program students have thus far been coauthors on a total of 25 posters or presentations at national conferences, 12 publications in peer-reviewed journals, and 1 opinion editorial (Table 2). Thus far, 63% of past SCORE Program students have published articles (2016 cohort: 100%, 2017: 83%, and 2018: 16%) and 75% have been involved in presentations or abstracts. Of those who have published articles, 75% were first authors. The impact factors of the 12 publications ranged from 1.13 to 2.55 (median, 2.442).
Table 2.
SCORE Program student productivity
| Year | Otolaryngology | Obstetrics and gynecology | ||
|---|---|---|---|---|
| Publications | Presentations and abstracts | Publications | Presentations and abstracts | |
| 2016 | 4 | 4 | NA | NA |
| 2017 | 5 (2 pending) | 6 | 2 | 4 |
| 2018 | 1 | 3 | 0 | 8 |
| Total | 10 | 13 | 2 | 12 |
NA = not applicable; SCORE = Summer Clinical Otolaryngology and Obstetrics/Gynecology Research.
Fifteen of the students completed the exit survey between 2016 and 2018, and all 15 reported that they were more likely to pursue clinical research in the future because of participation in the program, believed that the SCORE Program had increased their interest in a particular specialty, and reported that their faculty mentor was available and provided good direction, teaching, and mentorship (Table 3). On the basis of open-ended questions, the students reported on the program’s strengths, including the diverse range of research and clinical experiences and the training it provided, opportunities to interact closely with residents, and valuable mentorship. Overwhelmingly, students reported on the excellent mentorship, shadowing opportunities in their respective specialty, and engaging research experiences (see Sidebar: SCORE Program student comments representing the range of responses summarizing their research experience and providing feedback from the exit survey).
Table 3.
Likert scale student evaluation of the SCORE Program
| Response | Mean (SD)a |
|---|---|
| The program improved my understanding of clinical research | 4.6 (0.91) |
| I was able to participate in clinical research project(s) in a meaningful and productive way | 4.5 (0.92) |
| The program increased my interest in the specialty | 4.6 (0.63) |
| I am more likely to pursue a residency in the specialty | 4.3 (0.62) |
| I am more likely to pursue clinical research in the future because of my participation in the program | 4.1 (0.8) |
| I had good understanding, expectations, and ownership of my research project | 4.1 (1.2) |
| My faculty mentor was available and provided good direction, teaching, and mentorship | 4.7 (1.1) |
| The weekly didactic lectures were informative and helpful | 3.9 (0.67) |
| I would recommend this program to other medical students | 4.7 (0.71) |
A score of 1 indicates strongly disagree; 5, strongly agree.
SCORE = Summer Clinical Otolaryngology and Obstetrics/Gynecology Research.
SCORE Program student comments representing the range of responses summarizing their research experience and providing feedback from the exit survey.
Student responses regarding what worked well (survey question: “describe 3 strengths of the program”):
Ability to work on different projects at different stages of completion gave me a diverse range of research experiences.”
Dr X was an amazing mentor who took the time to get to know me and guide me along our projects. She was attentive to my desire to learn…She also allowed me to have a level of independence and ownership of my projects.
It [the SCORE Program] balanced clinical experience with clinical research better than any other program I have seen
Everyone I shadowed was accommodating and I got to see a huge diversity of Ob/Gyn practices and experiences
I loved the balance between research and shadowing. My schedule allowed me to get significant work done for my project and great experience in clinic and the OR!
Student responses providing feedback and areas for growth (survey question: “describe 3 improvements that can be made to the program”):
Increase hospital-wide knowledge of the SCORE program.
I would have liked a little more information prior to beginning the program. Maybe a list of projects that are available or clinics to shadow would have been nice to orient myself better prior to getting here.”
I think a little more communication amongst staff would make the program more efficient.
I thought the bonding especially between the ENT and Ob-Gyn students wasn’t as strong as it could have been.
SCORE = Summer Clinical Otolaryngology and Obstetrics/Gynecology Research.
In 2018, an exit survey was administered to the attending faculty and residents. Eight of the 16 residents (50%), 6 of the 16 attending faculty members (37.5%), and both program managers completed the exit survey (Table 4). All faculty, resident, and manager respondents enjoyed supporting the SCORE Program and agreed that they had a good understanding of mentorship expectations and were able to effectively engage with students during their clinical and research experiences. Interestingly, 81% of respondents strongly believed that SCORE Program students were able to contribute to their research in a meaningful and productive way; the remaining 19% agreed or somewhat agreed with this statement. All residents and attending faculty who responded were interested in being involved in the program in the future and provided supportive comments for the program (see Sidebar: Faculty and resident feedback regarding SCORE Program).
Table 4.
SCORE Program faculty and resident Likert scale evaluation of the program
| Response | Mean (SD)* |
|---|---|
| I had a good understanding of the purpose, structure and mentorship expectations | 4.3 (0.79) |
| I felt that I was able to effectively engage with the SCORE students during their clinical, lecture and/or research experiences | 4.6 (0.63) |
| SCORE students were able to contribute to my research project(s) in a meaningful and productive way | 4.7 (0.70) |
| I enjoyed supporting the SCORE Program | 4.9 (0.34) |
| I would like to be involved with the SCORE Program | 4.9 (0.34) |
A score of 1 indicates strongly disagree; 5, strongly agree.
SCORE = Summer Clinical Otolaryngology and Obstetrics/Gynecology Research.\
Faculty and resident feedback regarding SCORE Program.
It was incredible to have the SCORE students involved in my project. They were able to take it to the next level by having the time to put into literature review, organization and manuscript writing.
They were invaluable in moving resident research forward.
Providing students with early exposure to clinical and research experiences and providing residents and faculty with opportunities for support and personnel to help move their own research projects forward. It is a win-win-win!
I truly hope we can continue to keep this program going and potentially expand it to other departments. Other faculty have expressed an interest in having score students in their subspecialties.
Fabulous program. I hope it continues to be funded.
SCORE = Summer Clinical Otolaryngology and Obstetrics/Gynecology Research.
CONCLUSION
The SCORE Program exposed second-year medical students, including URMs, to a structured clinical research program, providing medical students with clinical and research skills training and mentorship, while providing residents and attending faculty with support to advance their research endeavors. This dual support of student mentees and resident and faculty mentors resulted in numerous scholarly products. These findings complement the increasing body of research indicating that early exposure to mentored clinical research during medical school increases interest in, knowledge of, and exposure to specialty-specific research.12–14 Furthermore, the program’s focus on recruiting and mentoring URMs is in alignment with the nationwide goal of improving diversity and inclusion in medicine. Lastly, the curriculum vitae of the SCORE Program students was more robust on their completion of the program, which could strengthen their residency applications.
Throughout the first 3 years of the SCORE Program, students have reported having positive experiences. All student participants responded that they were more likely to pursue clinical research in the future. In addition, all students ended the 8-week program with an increased interest in their specialty as well as a greater understanding of clinical clerkships and residency. These findings corroborate those of prior studies,16,17 which have found that early specialty exposure, including through mentorship and experience with patients, increases interest in a particular specialty as well as clinical competence.
The applicant pool increased from fewer than 20 applicants in 2016 for 4 positions to nearly 90 applicants in 2018 for 6 positions as the program became more well known. SCORE Program students in the first 3 years of the program came from across the US, with most being outside of California. This finding indicates a desire for structured summer research programs from medical students across the country. Furthermore, interest has been expressed by other departments for program expansion but is limited at this time by funding.
Although the SCORE Program has received positive feedback from students, residents, and faculty, areas for improvement were documented. Although the attending faculty and residents enjoyed hosting students overall, some found it challenging to incorporate the students into a clinical setting. Many physicians and hospital staff not directly mentoring the students were unfamiliar with the SCORE Program despite programmatic emails that were sent.
The Kaiser Permanente SCORE Program is still in its nascent state; therefore, there has not been the opportunity for long-term student follow-up. Future evaluations of this program could investigate the long-term impact it has on prior participants’ careers and interest in research during and beyond residency. In addition, because prior work11 suggests that interest in research is the biggest indicator of engaging in future research efforts, there may be a bias in who applies to the program, which may be something measurable using a pre-SCORE Program survey designed to gauge students’ interest in research at baseline.
Although the long-term impact of the Kaiser Permanente SCORE Program has not been examined, past studies of similar programs have found that early research experiences, strong mentorship, and publication success are all indicators of participation in future research, especially if individuals have a preexisting interest in research.11–13 In addition, mentorship for URMs alone increases their competitiveness when applying to residency programs.17 On the basis of the follow-up surveys, each SCORE student expressed an increased interest in incorporating research into their future careers. We believe that the Kaiser Permanente SCORE Program and similar experiences that expose medical students to research early in their careers may help reverse the downward trend in the number of physician-scientists. Furthermore, this strategy to increase the proportion of URMs in medicine and research may contribute to improvements not only in diversity and inclusion efforts in medical practice but also in better serving our patient community.
Kaiser Permanente SCORE Exit Survey for Students—Anonymous
Kaiser Permanente SCORE Exit Survey for Residents and Faculty
Acknowledgments
Thank you to all the SCORE Program students who participated in the program, Kaiser Permanente Graduate Medical Education for financially supporting the program, and Antoinette Niblett, our administrator of the program.
Laura King, ELS, performed a primary copy edit.
Footnotes
Disclosure Statement
The author(s) have no conflicts of interest to disclose.
Authors’ Contributions
Kellie Corcoran, MD, MPH, participated in the acquisition and analysis of data, drafting, and submitting of the final manuscript. Miranda Weintraub, PhD, participated in drafting and submitting of the final manuscript. Isabella Silvestre participated in drafting and submission of the final manuscript. Reshma Varghese participated in drafting of the final manuscript. Eve Zaritsky, MD, and Jonathan Liang, MD, participated in study design, acquisition and analysis of data, and drafting of the final manuscript. All authors have given final approval to this manuscript.
References
- 1.Schafer AI. The vanishing physician-scientist? Transl Res. 2010 Jan;155(1):1–2. doi: 10.1016/j.trsl.2009.09.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Diversity in medical education: Facts & figures 2016. Washington, DC: Association of American Medical Colleges; 2016. [Google Scholar]
- 3.Harding CV, Akabas MH, Andersen OS. History and outcomes of 50 years of physician-scientist training in medical scientist training programs. Acad Med. 2017 Oct;92(10):1390–8. doi: 10.1097/ACM.0000000000001779. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Palepu A, Carr PL, Friedman RH, Ash AS, Moskowitz MA. Specialty choices, compensation, and career satisfaction of underrepresented minority faculty in academic medicine. Acad Med. 2000 Feb;75(2):157–60. doi: 10.1097/00001888-200002000-00014. [DOI] [PubMed] [Google Scholar]
- 5.Feldman AM. The National Institutes of Health Physician-Scientist Workforce Working Group report: A roadmap for preserving the physician-scientist. Clin Transl Sci. 2014 Aug;7(4):289–90. doi: 10.1111/cts.12209. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Ercan-Fang NG, Rockey DC, Dine CJ, Chaudhry S, Arayssi T. Resident research experiences in internal medicine residency programs—A nationwide survey. Am J Med. 2017 Dec;130(12):1470–1476.e3. doi: 10.1016/j.amjmed.2017.08.033. [DOI] [PubMed] [Google Scholar]
- 7.Villwock JA, Hamill CS, Nicholas BD, Ryan JT. Otolaryngology residency program research resources and scholarly productivity. Otolaryngol Head Neck Surg. 2017 Jun;156(6):1119–23. doi: 10.1177/0194599817704396. [DOI] [PubMed] [Google Scholar]
- 8.Mahoney MR, Wilson E, Odom KL, Flowers L, Adler SR. Minority faculty voices on diversity in academic medicine: Perspectives from one school. Acad Med. 2008 Aug;83(8):781–6. doi: 10.1097/ACM.0b013e31817ec002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Merchant JL, Omary MB. Underrepresentation of underrepresented minorities in academic medicine: The need to enhance the pipeline and the pipe. Gastroenterology. 2010 Jan;138(1):19–26.e1. doi: 10.1053/j.gastro.2009.11.017. [DOI] [PubMed] [Google Scholar]
- 10.Wells RG. Building a better pipeline: The case for undergraduates in gastrointestinal research. Gastroenterology. 2007 Sep;133(3):740–1. doi: 10.1053/j.gastro.2007.07.036. [DOI] [PubMed] [Google Scholar]
- 11.Weaver AN, McCaw TR, Fifolt M, Hites L, Lorenz RG. Impact of elective versus required medical school research experiences on career outcomes. J Investig Med. 2017 Jun;65(5):942–8. doi: 10.1136/jim-2016-000352. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Zier K, Wyatt C, Muller D. An innovative portfolio of research training programs for medical students. Immunol Res. 2012 Dec;54(1–3):286–91. doi: 10.1007/s12026-012-8310-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Chang Y, Ramnanan CJ. A review of literature on medical students and scholarly research: Experiences, attitudes, and outcomes. Acad Med. 2015 Aug;90(8):1162–73. doi: 10.1097/ACM.0000000000000702. [DOI] [PubMed] [Google Scholar]
- 14.Solomon SS, Tom SC, Pichert J, Wasserman D, Powers AC. Impact of medical student research in the development of physician-scientists. J Investig Med. 2003 May;51(3):149–56. doi: 10.1136/jim-51-03-17. [DOI] [PubMed] [Google Scholar]
- 15.Frequently asked questions and answers about the new underrepresented in medicine definition. Washington, DC: Association of American Medical Colleges; 2003. p. 1. [Google Scholar]
- 16.Compton MT, Frank E, Elon L, Carrera J. Changes in U.S. medical students’ specialty interests over the course of medical school. J Gen Intern Med. 2008 Jul;23(7):1095–100. doi: 10.1007/s11606-008-0579-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Yang Y, Li J, Wu X, et al. Factors influencing subspecialty choice among medical students: A systematic review and meta-analysis. BMJ Open. 2019 Mar 7;9(3):e022097. doi: 10.1136/bmjopen-2018-022097. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Ramanan RA, Taylor WC, Davis RB, Phillips RS. Mentoring matters. Mentoring and career preparation in internal medicine residency training. J Gen Intern Med. 2006 Apr;21(4):340–5. doi: 10.1111/j.1525-1497.2006.00346.x. [DOI] [PMC free article] [PubMed] [Google Scholar]