Abstract
Introduction: The ACGME requires that residents perform scholarly activities prior to graduation, but this is difficult to complete and challenging to support. We describe a residency research program, taking advantage of environmental change aligning resident and faculty goals, to become a contributor to departmental cultural change and research development.
Methods: A research program, Scholar Quest (SQ), was developed as a part of an Information Mastery program. The goal of SQ is for residents to gain understanding of scholarly activity through a mentor-directed experience in original research. This curriculum is facilitated by providing residents protected time for didactics, seed grants and statistical/staff support. We evaluated total scholarly activity and resident/faculty involvement before and after implementation (PRE-SQ; 2003–2005 and POST-SQ; 2007–2009).
Results: Scholarly activity was greater POST-SQ versus PRE-SQ (123 versus 27) (p<0.05) with an incidence rate ratio (IRR)=2.35. Resident and faculty involvement in scholarly activity also increased PRE-SQ to POST-SQ (22 to 98 residents; 10 to 39 faculty, p<0.05) with an IRR=2.87 and 2.69, respectively.
Conclusion: Implementation of a program using department environmental change promoting a resident longitudinal research curriculum yielded increased resident and faculty scholarly involvement, as well as an increase in total scholarly activity.
INTRODUCTION
The Accreditation Council for Graduate Medical Education (ACGME) program requirements for emergency medicine (EM) call for programs to support and document resident scholarly activity prior to graduation. This requirement may be fulfilled through performance of a number of different activities, including review papers, case reports, textbook chapters, non-publishable projects, and participation in a research project or implementation of original research.1–3
This requirement is often difficult to fulfill for residents, and studies have revealed significant variance in the quality and quantity of scholarly activity accomplished.3–7 While the majority of EM residents state that they plan to conduct original research during their residency, only a minority complete this goal.4 However, residents who do complete research are provided funding and are supported to present their research at scientific meetings; they are also more likely to choose a career in academic medicine8–11 For residents not headed toward an academic career, exposure to research experiences might increase their awareness and receptiveness to new clinical research findings, the need to practice evidence-based medicine, and the need for society to support healthcare research to address gaps in knowledge and healthcare disparities.12–14 In 2003, the Institute of Medicine's Clinical Research Roundtable recommended that a clinical research curriculum be fundamental to all residency training requirements as one strategy to improve the provision of healthcare in the future.14
In a recent report in JAMA, Rothberg evaluated the obstacles to conducting research during residency and found that the barriers are multifactorial.7 The barriers on the resident level include a lack of resident interest in research coupled with a lack of residency time dedicated to conducting research and developing resident research skills. Barriers on the faculty side include difficulty in finding appropriate mentorship for projects along with faculty time to assist with research skills and project development.7 Both of these aspects are linked through the common lack of research infrastructure and funding to support research activities.7 However, Rothberg does note that these barriers may be overcome through “an intentional approach that addresses specific barriers, beginning with a commitment to change the underlying culture of the institution to create an atmosphere of inquiry and the financial investment to build the necessary infrastructure.”7
In this program evaluation and descriptive review, we describe a single-department residency research program whose goal is to provide an original research experience integrated with their EM curriculum. Development of this program took advantage of environmental change in the department to completely rethink the curriculum, align resident, faculty and department goals, and thus become a major contributor to the cultural change in an established academic department of emergency medicine.
METHODS
Residency Research Pre-Intervention
The University of Arizona Department of Emergency Medicine (DEM) had a distinguished history of scholarly work by an experienced faculty but had not produced a large portfolio of funded grants. The Arizona Emergency Medicine Research Center (AEMRC), the research arm of the DEM, was established and had the potential to become more active. The department head, Dr. Harvey Meislin, made the decision in 2003 to promote a change in the research culture of the ED. This required changes across the entire research system. Realigning the resident research experience was an important component, as it had the potential to become a source of future faculty members with strong academic interests and experiences and would also provide a platform to recruit future residents and faculty with academic experience and career trajectories.
Prior to 2005, the University of Arizona/UMC (UA/UMC) EM residency program expected scholarly activity by their residents to complete the program. This scholarly activity took many forms, including case studies, textbook chapters, non-peer reviewed publications, review articles, abstracts, peer-reviewed publications, posters or oral presentations. During those years, there was no requirement for original research.
The residents' scholarly activities were supported by a small core set of faculty mentors through informal mentorship. Annually, the residents received 4 hours of formal training in basic research methodologies, statistical analysis and critical analysis of the medical literature. The experience was not supported by dedicated funding and was not explicitly aligned with faculty research interests.
Although the scholarly activity satisfied the ACGME requirements, it was noted by Dr. Meislin that it did not help foster a department environment dedicated to research. The new stated departmental vision included a cultural change focused on an alignment of faculty and resident research efforts. In this way, a strong mentor/mentee relationship between residents and faculty could be formed with advantages to both.
Early in the development process of this concept, he identified and assembled key individuals who could refine and implement the vision. Key individuals in the planning stages included the department head, residency program director, research director, and vice directors of the AEMRC. The concepts discussed required the integration of resident learning, faculty goals for professional development, and the projected long-term research focus areas of the department.
Scholar Quest Evolution
Planning
The development of Scholar Quest (SQ) started with an identification of the resident learning needs that could be obtained from a research program. The goals of the new program would fundamentally need to satisfy ACGME program requirements. On an individual level, resident needs vary significantly. Future careers can range from working in the private sector with no research expectations to entering traditional academic pathways. The fundamental goal of the program was to provide an original research experience that would benefit all residents by providing a mentor-directed experience, preparing residents for both academic and/or private industry pursuits, with the vision to improve the provision of healthcare in the future.
The planning committee believed that the curriculum should be integrated into other existing experiences to support the overall cultural change. The program was designed to combine didactic lectures in theory and methodology education, a team-based original hypothesis-driven proposal, direct faculty mentoring to guide residents through the experience, and finally the development of an original research end product for presentation/publication. This design was aligned with an existing curriculum of evidence-based medicine and a rigorous critical-appraisal journal club.
Support
During the planning phases, the planning committee completed an assessment of the needed resources to be allocated for program success. They divided the resources estimated into 3 key areas: research core staffing, faculty mentors, and possible research funding to conduct the projects. Staffing needs predicted included 0.25 FTE research office staff for assistance with IRB submissions and manuscript preparation, 0.25 FTE epidemiologist/biostatistician to assist with teaching methodology to residents and assisting with statistical evaluation for projects, and lastly 0.25 FTE for a program coordinator for overall curriculum direction and monitoring the progress of resident research.
Mentors. The committee deemed significant faculty mentoring involvement to be essential. To promote this, a new incentive structure for faculty involvement was developed, which rewarded faculty time in conference, conducting research with residents, and performing lectures. Faculty could earn this “citizenship incentive” ($4,000 annual) for their involvement and at the same time gain access to research collaborators/assistants to improve personal academic productivity.
Funding
Research funding for projects was determined to be an important aspect of the program to support the research effort of the residents and faculty. The committee determined this to be best supported through a competitive seed grant process where residents and/or faculty submit applications for grants of up to $5,000. Evaluation of seed grants quality and funding was done through the existing departmental research committee.
Resident research was also supported through funding for presentation of findings at regional and national conferences. Residents who have abstracts accepted at these meetings are given a travel stipend to go to the meeting and present their findings.
Scholar Quest Curriculum
SQ is a key part of a highly structured 3-year EM Information Mastery program (IM) (Figure). The 3 components include an evidence-based medicine curriculum, Critical-Appraisal Journal Club, and the SQ program. The overall goals for participants of the IM program are 3-fold: (i) to better use information derived from valid medical literature sources for patient care, education and research, (ii) to improve the skills of critically appraising medical literature relevant to EM, and (iii) to acquire an understanding of scholarly activity through a directed experience in original research
Figure.
The Information Mastery program is a novel, integrated curriculum developed in the University of Arizona Department of Emergency Medicine. It comprises three complementary tracks: Evidence-Based Medicine (EBM), Scholar Quest, and Journal Club. PG, post graduate
SQ didactic sessions and mentored team meetings are scheduled during regular weekly conference time to insure access for all residents. Total conference time set aside per year toward research activities was 14 hours, not including optional external time with faculty mentors on research development or conducting studies. Scheduled sessions included research methodology lectures, hypothesis generation, meeting with epidemiologists/biostatisticians, interaction with faculty mentors, development of Institutional Review Board and Institutional Animal Care and Use Committee applications and preparations of publishable material.
At the completion of the research, each resident is required to present their research findings prior to graduation at the Annual Resident Research Forum. The Annual Resident Research Forum occurs during protected didactic time. The Forum is a venue for residents to present their research to all the faculty and residents in the department.
Method of Program Evaluation
To ascertain the overall outcomes, we measured total completed scholarly activity along with resident and faculty involvement from 2003–2010, both pre-intervention (PRE-SQ) (2002–2004) and post-intervention (POST-SQ) (2007–2009). Scholarly activity can be fulfilled through performance of a number of different activities, including review papers, case reports, textbook chapters, non-publishable projects, and participation in a research project or implementation of original research.1–3
We did not include the program adoption period in analysis. Data were tabulated from a broad definition of scholarly activities that are typically accepted by residency programs and divided into categories including: peer-reviewed publications, poster presentations, published abstracts, oral presentations and non-peer reviewed material, including chapters, electric journals, and non-scholarly publications.
We collected data from information extracted from program files and de-identified in a collection spreadsheet. All residents who graduated during the study period were also contacted for updated CVs focusing on their research work during residency. We compared these and generated a master list. Overall, scholarly activity was identified in ~94% of residents who graduated from the residency program during the evaluation periods.
We performed statistical analysis using a Wilcoxon rank sum test comparing differences PRE-SQ and POST-SQ. This was followed by a negative binomial regression analysis for annual incidence rates and annual incidence rate ratios. Annual incidence rate is the ratio of the number of cases per year (i.e scholarly activity) to the time period of the program in question (i.e PRE-SQ). The annual incidence rates were measured to evaluate the number of residents and faculty involved over time and the total scholarly activity over the time period. We calculated the rate ratios to control for growth of the faculty and resident population and its effect on the annual incidence rates. Analysis was done using STATA version 11 software.
RESULTS
The SQ program was started in 2004 and was integrated into the curriculum and departmental culture over a 2-year period. During this time, there was also a coincident increase in the number of residents in the residency and recruited faculty members. From PRE-SQ to POST-SQ periods, the total resident numbers increased from 98 residents PRE-SQ to 152 total residents POST-SQ (p<0.05). The total faculty numbers similarly increased from 73 for the PRE-SQ period to 106 total faculty members during the POST-SQ period (p<0.05).
The total number of scholarly activities was tabulated PRE-SQ (2002–2004) and POST-SQ (2007–2009) (Table 1). Included in the tabulation are the different types of activities conducted along with the totals. There was an increase in the total number of scholarly activities in the PRE-SQ period versus the POST-SQ period of 27 to 123 (p<0.05).
Table 1.
Scholarly activity for residents pre-intervention and post- intervention with Scholar Quest.
During the study period, resident and faculty involvement also increased as noted in Table 2. Resident involvement significantly increased from PRE-SQ to POST-SQ (22 to 98 residents involved in scholarly activities) (p<0.05). Faculty involvement followed a similar trend increasing from 10 to 39 faculty involved in resident scholarly activities (p<0.05).
Table 2.
Resident and faculty involvement pre-intervention and post-intervention with Scholar Quest.
To further investigate this rate of change, and control for the growth of both the resident and faculty populations, we calculated annual incidence rates and ratios for total scholarly activities, resident involvement and faculty involvement (Table 3). Rate ratio for scholarly activities was 2.35 (CI 1.05 to 5.3) (p<0.05). Both resident involvement and faculty involvement demonstrated similar increased rate ratios of 2.87 (CI 1.79 to 4.60) (p<0.05) and 2.69 (CI 1.34 to 5.38) (p<0.05), respectively.
Table 3.
Annual Incidence rates and ratios for resident scholarly activities, resident involvement and faculty involvement before and after the Scholar Quest (SQ) program initiation.
DISCUSSION
The development of a comprehensive original resident research experience is a significant challenge.7 Other important but competing priorities during residency include learning core foundational medical knowledge concepts, gaining procedural competency and maintaining personal wellness. However, the benefits of having an experience like SQ stimulate residents to consider an academic career, increase their awareness and receptiveness to new clinical research findings, substantiate the need to practice evidence-based medicine, and demonstrate the need for society to support healthcare research to address gaps in knowledge and healthcare disparities.12–14
In this study, we demonstrate that the implementation of a longitudinal resident research curriculum with an original research experience at the core can be successful when aligned with faculty and departmental goals. The use of a global environmental change in the department towards research facilitated its success evidenced by a doubling of the annual rate of scholarly activities (annual incidence ratio=2.35) (p<0.05). Moreover, the growth of the departmental culture and link between residents and faculty was apparent in the doubling of the rate of both residents and faculty involvement in research over time (annual incidence ratios of 2.87 and 2.69, respectively) (p<0.05).
Following the implementation of SQ in the residency curriculum, there was a significant increase in resident and faculty involvement in research along with an increase in the total scholarly activity from the residents. The success of this program was not solely due to a new dedicated curriculum, but was accomplished by a complete dedication to change in the environment of the department to one that fosters and nurtures research at all levels of development.
The reason for the success of this program is probably multi-factorial. The contributors to the success of this program are noted on all levels of the infrastructure including: 1) the vision of the department head to impart a philosophical change; 2) dedication of the departmental leadership to implement this change across all faculty and resident programs; 3) integration of the SQ research program into a new Information Mastery Program; 4) dedicated didactic time for research methodology and hypothesis generation; 5) development of departmental infrastructure to support research studies including epidemiologist/biostatistician and program coordinators; 6) monetary support for resident research projects through grants and also funding presentations at regional and national meeting; and 7) securing faculty involvement through monetary incentives.
The concept of an original research experience and its integration in resident education is one that has been difficult to implement due to many hurdles. SQ presents one department's efforts to produce an original research experience for their residents through the alignment of faculty and resident goals. Similar outcomes could likely be achieved at other programs by introducing key ingredients outlined in this article.
LIMITATIONS
The primary limitation of this study is that this is a single-site evaluation of the development and implementation of a longitudinal residency research curriculum. This makes it difficult to determine whether this program is generalizable to other sites. However, the fundamental effect of changing the culture and environment of a department to enhance the residency research experience is an important and unique finding supporting the recommendations noted by Rothberg.7 This study demonstrates that it is feasible to implement departmental cultural change in the manner described. It is reasonable to expect that implementation of many of the key aspects of this program would achieve similar outcomes.
This study is also limited by being unable to determine which intervention was most responsible for the success of the program. The study design does not allow for the delineation of which specific factor is primarily responsible (i.e integration of the program into Information Mastery Curriculum, development of research infrastructure, monetary support, etc.) but instead presents the concept that a holistic cultural change on all levels of the infrastructure may be responsible for the positive outcome of this program.
Further, as a single-site program attempting to align residents and faculty, we may potentially constrain resident choice in research areas. Although the program is designed for residents to self-generate research questions and hypotheses with mentor guidance, residents may be drawn to choosing mentors who are in research areas that are well represented in the department. This may inadvertently constrain resident choices.
Lastly, a potential confounder is the type of faculty recruited and hired throughout the time period of this evaluation. This is a potential confounder since newly hired faculty could be research oriented, highly productive and affect the success of the program. With this in mind, we analyzed the data for all scholarly activity that was primarily mentored by newly recruited faculty during the POST-SQ period. Newly recruited faculty was defined as faculty hired following the PRE-SQ period. During this POST–SQ period, 93% of the scholarly activity was mentored by established faculty who were not newly hired.
CONCLUSION
Implementation of a program using departmental environmental change to promote a resident longitudinal research curriculum designed to facilitate resident involvement in original research yielded increased resident and faculty involvement. This was associated with increased total scholarly activity.
Footnotes
*p=0.05 for comparison of PRE-SQ (2002–2004) to POST-SQ (2007–2009).
*p<0.05 for comparison of PRE-SQ (2002–2004) to POST-SQ (2007–2009) for both resident and faculty involvement.
Ref, referent
Supervising Section Editor: Christian McClung, MD
Full text available through open access at http://escholarship.org/uc/uciem_westjem
Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. The authors disclosed none.
REFERENCES
- 1.Accreditation Council for Graduate Medical Education Program Requirements for Graduate Medical Education in Emergency Medicine. Accreditation Council for Graduate Medical Education Program Requirements for Graduate Medical Education in Emergency Medicine. 2011 Available at: http://www.acgme.org/acWebsite/downloads/RRC_progReq/110emergencymed07012007.pdf. Accessed 6/30, 2011. [Google Scholar]
- 2.Holmes JF, Sokolove PE, Panacek EA. Ten-year experience with an emergency medicine resident research project requirement. Acad Emerg Med. 2006;13(5):575–579. doi: 10.1197/j.aem.2005.12.016. [DOI] [PubMed] [Google Scholar]
- 3.Levitt MA, Terregino CA, Lopez BL et al. Factors affecting research directors' and residents' research experience and productivity in emergency medicine training programs. Acad Emerg Med. 1999;6(4):356–359. doi: 10.1111/j.1553-2712.1999.tb00403.x. [DOI] [PubMed] [Google Scholar]
- 4.Neacy K, Stern SA, Kim HM et al. Resident perception of academic skills training and impact on academic career choice. Acad Emerg Med. 2000;7(12):1408–1415. doi: 10.1111/j.1553-2712.2000.tb00499.x. [DOI] [PubMed] [Google Scholar]
- 5.Blanda M, Gerson LW, Dunn K. Emergency medicine resident research requirements and director characteristics. Acad Emerg Med. 1999;6(4):286–291. doi: 10.1111/j.1553-2712.1999.tb00390.x. [DOI] [PubMed] [Google Scholar]
- 6.Terregino CA, Levitt MA, Lopez BL et al. A national profile of resident research experience. Acad Emerg Med. 1999;6(4):351–356. doi: 10.1111/j.1553-2712.1999.tb00402.x. [DOI] [PubMed] [Google Scholar]
- 7.Rothberg MB. Overcoming the obstacles to research during residency: what does it take? JAMA. 2012;308(21):2191–2192. doi: 10.1001/jama.2012.14587. [DOI] [PubMed] [Google Scholar]
- 8.Stern SA, Kim HM, Neacy K et al. The impact of environmental factors on emergency medicine resident career choice. Acad Emerg Med. 1999;6(4):262–270. doi: 10.1111/j.1553-2712.1999.tb00387.x. [DOI] [PubMed] [Google Scholar]
- 9.Sanders AB, Fulginiti JV, Witzke DB et al. Characteristics influencing career decisions of academic and nonacademic emergency physicians. Ann Emerg Med. 1994;23(1):81–87. doi: 10.1016/s0196-0644(94)70013-3. [DOI] [PubMed] [Google Scholar]
- 10.Burkhardt J, Kowalenko T, Meurer W. Academic career selection in American emergency medicine residents. Acad Emerg Med. 2011;18(Suppl 2):S48–53. doi: 10.1111/j.1553-2712.2011.01181.x. [DOI] [PubMed] [Google Scholar]
- 11.Regan L, Stahmer S, Nyce A et al. Scholarly tracks in emergency medicine. Acad Emerg Med. 2010;17(Suppl 2):S87–94. doi: 10.1111/j.1553-2712.2010.00890.x. [DOI] [PubMed] [Google Scholar]
- 12.McGlynn EA, Asch SM, Adams J et al. The quality of health care delivered to adults in the United States. N Eng J Med. 2003;348(26):2635–2645. doi: 10.1056/NEJMsa022615. [DOI] [PubMed] [Google Scholar]
- 13.Dans AL, Dans LF, Guyatt GH et al. Users' guides to the medical literature: XIV. How to decide on the applicability of clinical trial results to your patient. Evidence-Based Medicine Working Group. JAMA. 1998;279(7):545–549. doi: 10.1001/jama.279.7.545. [DOI] [PubMed] [Google Scholar]
- 14.Murillo H, Reece EA, Snyderman R et al. Meeting the challenges facing clinical research: solutions proposed by leaders of medical specialty and clinical research societies. Acad Med. 2006;81(2):107–112. doi: 10.1097/00001888-200602000-00002. [DOI] [PubMed] [Google Scholar]