Abstract
Background
There is rising concern that fundamental scientific principles critical to lifelong learning and scientific literacy are not sufficiently addressed during residency.
Objective
We describe the development, implementation, and evaluation of a systematic review and meta-analysis course designed to improve residents' research literacy.
Intervention
We developed and implemented a novel, interactive, web-enhanced course for third-year psychiatry residents to provide the theoretical and methodological tools for conducting and reporting systematic reviews and meta-analyses. The course is based on Bloom's learning model, and established criteria for reporting systematic reviews and meta-analyses. Eight sequential learning objectives were linked to 8 well-specified assignments, with the objectives designed to build on one another and lead to the creation of a scientific manuscript.
Results
From 2010–2014, 54 third-year psychiatry residents (19 unique groups) successfully completed the course as part of a graduation requirement. The majority rated the course as being good or very good, and participants reported a statistically significant increase in their confidence to conduct systematic reviews (χ2 = 23.3, P < .05) and meta-analyses (Fisher exact test, P < .05). Estimated total dedicated resident and faculty time over a period of 36 weeks was 36 to 72 hours and 60 hours, respectively. Residents' academic productivity included 11 conference presentations and 4 peer-reviewed published manuscripts, with 2 residents who were awarded honors for their projects.
Conclusions
A formal training course in systematic reviews and meta-analyses offers a valuable learning experience, which enhances residents' research skills and academic productivity in a feasible and sustainable approach.
What was known and gap
Research literacy, including the ability to aggregate data from multiple studies, are physician skills not well addressed in residency.
What is new
A required course in conducting and reporting systematic reviews and meta-analyses for psychiatry residents that has been sustained over multiple years.
Limitations
Single specialty, single institution intervention may reduce generalizability.
Bottom line
A formal course in systematic reviews and meta-analyses for residents offered a valuable learning experience, and enhanced their research skills and academic productivity.
Introduction
Participation in scholarly activity is important to advance residents' critical evaluation of research, ensure care is evidence based, and address the critical shortage of physician researchers.1 To accomplish these goals, many programs have developed longitudinal research curricula2–7 and/or research tracks.8–10 Approaches described in the literature vary in the amount of devoted research time, ranging from protected research blocks11–13 to a dedicated research year.14 Yet, programs continue to struggle to successfully integrate practical research experience into residency training.15–17 This lack of integration may in part be due to several factors, including increased clinical service demands,18,19 limited funding dedicated for residents to conduct research,20,21 insufficient faculty mentors, and limited resident20,22,23 and faculty time.19,23 Although the majority of programs report having some infrastructure to provide a research training experience, many report low levels of research knowledge (eg, research design, manuscript writing, and grant writing).24 To bridge this gap, it is critical that residency programs provide research training experiences that are educationally meaningful, and sustainable within the existing training framework.
Teaching residents how to conduct systematic reviews and meta-analyses may provide a solution to some of these barriers. Systematic reviews and meta-analyses are efficient means of integrating existing information; they provide data for rational decision making25 and can be taught using established guidelines26 and previously vetted tools.27 These skills are relevant to lifelong learning, quality improvement, and evidence-based practice, as systematic reviews and meta-analyses provide succinct syntheses of critical biomedical information that can inform clinical practice and improve patient care. The benefits of systematic reviews as research activities in graduate medical education have been described,25,28 but to our knowledge, this is the first article describing the integration of a systematic review course in a residency program.
Methods
Intervention
The Department of Psychiatry at the University of Maryland School of Medicine developed and successfully implemented a hybrid course utilizing interactive face-to-face and online elements for third-year psychiatry residents that provides the theoretical and methodological tools required to conduct and report the results of systematic reviews and meta-analyses.
Course Development: Theoretical Framework
The course was informed by Benjamin Bloom's mastery for learning model in which residents are helped to master each learning task before moving to a more advanced learning task.29 This model requires well-defined learning objectives that are sequentially organized to fulfill a given learning objective. Eight sequential learning objectives regarding data collection were developed based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, which are an evidence-based minimum set of items for reporting on systematic reviews26 (table). These learning objectives were each linked to 8 well-specified assignments, which were reviewed using a structured instrument to provide constructive feedback. Learning objectives were specifically designed to link to the PRISMA guidelines, and naturally build on one another to lead to the creation of a scientific manuscript.
TABLE.
Content of Course Lectures With Learning Objectives Linked to Assignments Based on PRISMA Criteria
Consistent with the principles of adult learning (eg, andragogic learning),30 course design focused on problem-based assignments in teams. Each group chose a research question that became the focus of the sequential, problem-based assignments listed above. The course emphasized working in small groups as this is how “science gets done in the real world,” and it additionally offered an opportunity for self-directed learning. The instructor served as a facilitator, answering questions, providing information (in the format of a small series of problem-based, informational lectures), and ensuring projects were completed efficiently.
Course Content
The course provided 9 succinct, 30- to 45-minute lectures that occurred over a 9-month period. Lectures emphasized critical learning objectives that were nearly always linked to an assignment (table) and were supplemented by group-based, self-directed learning in the service of completing each assignment.
Web-Enhanced Format
A web-based educational service (Blackboard)31 was used to provide the online structure for the course. Blackboard has the ability to post all educational material, remind participants of required readings and due dates, create chat rooms for open discussion and dialogue, and collect course-related projects and outcome data. For the purpose of this course, the site was used to disseminate course specific information (downloading and uploading assignments).
Course Implementation and Time Requirements
Residents
All third-year psychiatry residents participated in the course as a graduation requirement. Residents were expected to complete class assignments, culminating in a scientifically formatted manuscript ready for peer review. Residents were also required to present their work at a departmental research day. Submitting manuscripts or abstracts for publication and conference participation was optional. Class assignments were developed to take each resident approximately 5 hours of outside class time over a period of 36 weeks, and residents estimated spending approximately 1 to 2 hours weekly.
Faculty
One faculty member served as the course instructor. In addition to in-course instruction, the instructor met with each resident group separately for out-of-class consultation for approximately 1 to 2 hours over the length of the course (with an estimated total faculty time of approximately 60 hours).
Course Assessment
Residents anonymously completed an evaluation form created by the authors about whether the course met learning objectives. This consisted of 38 questions that used a 5-point Likert scale and evaluated the utility of key course components, and included 3 open-ended questions for feedback about strengths and weaknesses of the course. This retrospective pre-post method preserves the anonymity of residents' responses and has been shown to be effective in program evaluation.32
The evaluation was deemed exempt by the University of Maryland School of Medicine Institutional Review Board.
Analysis of course evaluations consisted of univariate distributions for each variable, as well as comparisons of percentages using 2-tailed χ2 test or Fisher exact test (Stata version 12, StataCorp LP).
Results
From 2010–2014, 54 residents (consisting of 19 unique groups, ranging in size from 2 to 4 members) successfully completed the course. From the results of these evaluations, residents who took the course reported a statistically significant increase in their confidence to conduct a systematic review (79.6% [43 of 54] reported being “confident” or “very confident” postcourse versus 7.4% [4 of 54] before the course; χ2 = 23.3; P < .05) as well as a statistically significant increase in their confidence to conduct a meta-analysis (77.8% [42 of 54] reported being “confident” or “very confident” postcourse versus 5.6% [3 of 54] before the course; Fisher exact test; P < .05). The course received an overall evaluation as being “good” to “very good,” and the course was rated as being neither too easy nor too hard. The majority (85%, 46 of 54) reported they agreed or strongly agreed that they had enough time to complete all assignments. Key course components were rated favorably: 90% (49 of 54) reported that meeting with the course instructor was “very helpful” or “helpful,” and 76% (41 of 54) reported that completing the final scientific paper was “very helpful” or “helpful.” In particular, residents noted that the structured, stepwise content of the course was linked to success. We present selected written comments taken from the evaluations to illustrate this point:
“It was very helpful learning how to do a meta-analysis/systematic review. I also feel that it was extremely helpful learning how to assess the quality of research studies. Lastly, I truly appreciated the strict deadlines for each assignment.”
“Stepwise approach to completing, constant guidance through each step, knowledge and confidence gained from completing project.”
Additionally, some residents chose to submit abstracts to conferences or submit their manuscript for publication. For the 19 groups that completed the course, there have been 11 abstracts accepted at national scientific meetings. One poster presentation received the distinction of being the “Best Resident Poster” at 1 meeting. There have also been 4 published peer-reviewed manuscripts. One manuscript received the “Best Resident Paper” award from a state professional society.
Discussion
Our report is on the successful development, implementation, and evaluation of a research training course that fits well within the context of a residency training experience. Most residents liked the course, felt they had enough time to complete the course, and gained confidence in their ability to conduct a systematic review and meta-analysis. Many residents published peer-reviewed manuscripts, which was infrequent prior to this course implementation.
It is important to consider factors that facilitated the success of the course along with potential barriers to its implementation. The course had support from the chairman and the residency program director, including the provision of faculty time and funding to support course creation and instruction. As the course was significantly different from previous research activities, it was vetted by residents and faculty to garner support and champion the effort. Barriers noted during this process included residents' concern over the time required to complete course assignments. These concerns were overcome by incorporating 1 hour per week for “protected time” to work on course activities. To help promote the success of the course, we added an award for the best resident project and a formal end-of-the-course oral presentation. Presentations were used to showcase the results of the course and were considered highly successful. After the end of the first year of the course, residents provided feedback that having protected time was not necessary and this component was dropped.
Strengths of this research training course include (1) provision of valuable research experiences without requiring significant programmatic restructuring; (2) emphasis on group-based, self-directed learning; and (3) minimal faculty burden. The time required for residents to complete the course was approximately 36 to 72 hours. It is important to note that Rivera and colleagues20 found that internal medicine residents presenting their work at a conference spent a median of 200 hours on research abstracts, and 50 hours on clinical vignettes.
An important limitation of this project evaluation is the lack of validity evidence for the author-developed survey. Course evaluation could be strengthened by using existing validated tools and/or establishing survey validity by comparing the authors' survey with other evaluation instruments. In addition, the course was developed and applied in a single institution for residents in a single specialty, limiting generalizability. Finally, a retrospective pre-post method may introduce recall bias, even though it is commonly used in educational program evaluation.
Implementing this type of course in a residency training program requires careful consideration, as adding 1 to 2 hours per week of additional resident work may not be feasible in some programs.
Identifying a faculty member to lead the course also may prove challenging. Faculty who do not have a background in research methodology may not feel prepared to teach a course such as this. However, these challenges are surmountable with guidance and consultation. The authors are currently developing a course guide with all the necessary teaching materials to help facilitate dissemination. This course guide will be available by the authors and supplemented through ongoing consultation, using a train-the-trainer model to facilitate implementation.
Conclusion
Providing the framework, resources, and training necessary to conduct a systematic review and meta-analysis led to the attainment of academically relevant skills that can be applied in lifelong learning. These outcomes suggest that the course is educationally meaningful, sustainable, and impacts both residents' research training and the profession at large.
Footnotes
Seth Himelhoch, MD, MPH, is Associate Professor, Department of Psychiatry, and Interim Director, Division of Services Research, Department of Psychiatry, University of Maryland School of Medicine; Sarah Edwards, DO, is Assistant Professor, Department of Psychiatry, and Program Director of Child and Adolescent Psychiatry Fellowship, Division of Child and Adolescent Psychiatry, Department of Psychiatry, University of Maryland School of Medicine; Mark Ehrenreich, MD, is Assistant Professor and Chief of Medical Education, Department of Psychiatry, and Director of Residency Training, University of Maryland School/Sheppard Pratt, Department of Psychiatry, University of Maryland School of Medicine; and M. Philip Luber, MD, is Professor and Interim Chair, Department of Psychiatry, and Associate Dean for Graduate Medical Education, University of Texas Health Science Center at San Antonio.
Funding: The authors report no external funding source for this study.
Conflict of interest: The authors declare they have no competing interests.
The authors would like to thank the psychiatry residents who participated in the research course.
References
- 1.Abrams MT, Patchan K, Boat TF. Research Training in Psychiatry Residency: Strategies for Reform. Washington, DC: The National Academies Press; 2003. Committee on Incorporating Research into Psychiatry Residency Training. [PubMed] [Google Scholar]
- 2.Hebert RS, Levine RB, Smith CG, Wright SM. A systematic review of resident research curricula. Acad Med. 2003;78(1):61–68. doi: 10.1097/00001888-200301000-00012. [DOI] [PubMed] [Google Scholar]
- 3.Hamoda HM, Bauer MS, DeMaso DR, Sanders KM, Mezzacappa E. A competency-based model for research training during psychiatry residency. Harv Rev Psychiatry. 2011;19(2):78–85. doi: 10.3109/10673229.2011.565249. [DOI] [PubMed] [Google Scholar]
- 4.Kirchner JE, Owen RR, Nordquist CR, Clardy JA. Developing clinician scientists through integrated research training in psychiatry. Teach Learn Med. 1998;10(3):183–187. [Google Scholar]
- 5.Mezzacappa E, Hamoda HM, DeMaso DR. Promoting scholarship during child and adolescent psychiatry residency. Acad Psychiatry. 2012;36(6):443–447. doi: 10.1176/appi.ap.11110199. [DOI] [PubMed] [Google Scholar]
- 6.Carek PJ, Dickerson LM, Diaz VA, Steyer TE. Addressing the scholarly activity requirements for residents: one program's solution. J Grad Med Educ. 2011;3(3):379–382. doi: 10.4300/JGME-D-10-00201.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Roane DM, Inan E, Haeri S. Galynker II. Ensuring research competency in psychiatric residency training. Acad Psychiatry. 2009;33(3):215–220. doi: 10.1176/appi.ap.33.3.215. [DOI] [PubMed] [Google Scholar]
- 8.Kohlwes RJ, Shunk RL, Avins A, Garber J, Bent S, Shlipak MG. The PRIME curriculum. J Gen Intern Med. 2006;21(5):506–509. doi: 10.1111/j.1525-1497.2006.00438.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Bhat V, Leong K, Lee J, Voineskos D, Daskalakis ZJ, Lam RW, et al. Research-track programs for residents in psychiatry: a review of literature and a report of 3 Canadian experiences. Can J Psychiatry. 2014;59(5):268–275. doi: 10.1177/070674371405900506. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Tsai A, Ordoñez A, Reus V, Mathews C. Eleven-year outcomes from an integrated residency program to train research psychiatrists. Acad Med. 2013;88(7):983–988. doi: 10.1097/ACM.0b013e318294f95d. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Chan RK, Lockyer J, Hutchison C. Block to succeed: the Canadian orthopedic resident research experience. Can J Surg. 2009;52(3):187–195. [PMC free article] [PubMed] [Google Scholar]
- 12.Robbins L, Bostrom M, Marx R, Roberts T, Sculco TP. Restructuring the orthopedic resident research curriculum to increase scholarly activity. J Grad Med Educ. 2013;5(4):646–651. doi: 10.4300/JGME-D-12-00303.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Vinci RJ, Bauchner H, Finkelstein J, Newby PK, Muret-Wagstaff S, Lovejoy FH., Jr Research during pediatric residency training: outcome of a senior resident block rotation. Pediatrics. 2009;124(4):1126–1134. doi: 10.1542/peds.2008-3700. [DOI] [PubMed] [Google Scholar]
- 14.Bernstein J, Ahn J, Iannotti JP, Brighton CT. The required research rotation in residency: the University of Pennsylvania experience, 1978–1993. Clin Orthop Relat Res. 2006;449:95–99. doi: 10.1097/01.blo.0000224040.77215.ff. [DOI] [PubMed] [Google Scholar]
- 15.Levine RB, Hebert RA, Wright SM. Resident research and scholarly activity in medical residency training programs. J Gen Intern Med. 2005;20(2):155–159. doi: 10.1111/j.1525-1497.2005.40270.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Rothberg MB. Overcoming the obstacles to research during residency: what does it take? JAMA. 2010;308(21):2191–2192. doi: 10.1001/jama.2012.14587. [DOI] [PubMed] [Google Scholar]
- 17.Rothberg MB, Kleppel R, Friderici JL, Hinchey K. Implementing a resident research program to overcome barriers to resident research. Acad Med. 2014;89(8):1133–1139. doi: 10.1097/ACM.0000000000000281. [DOI] [PubMed] [Google Scholar]
- 18.Potti A, Mariani P, Saeed M, Smego RA., Jr Residents as researchers: expectations, requirements, and productivity. Am J Med. 2003;115(6):510–514. doi: 10.1016/j.amjmed.2003.08.017. [DOI] [PubMed] [Google Scholar]
- 19.Alguire PC, Anderson WA, Albrecht RR. Poland GA. Resident research in internal medicine training programs. Ann Intern Med. 1996;124(3):321–328. doi: 10.7326/0003-4819-124-3-199602010-00007. [DOI] [PubMed] [Google Scholar]
- 20.Rivera JA, Levine RB, Wright SM. Completing a scholarly project during residency training. J Gen Intern Med. 2005;20(4):366–369. doi: 10.1111/j.1525-1497.2005.04157.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Hamann KL, Fancher TL, Saint S, Henderson MC. Clinical research during internal medicine residency: a practical guide. Am J Med. 2006;119(3):277–283. doi: 10.1016/j.amjmed.2005.12.001. [DOI] [PubMed] [Google Scholar]
- 22.Gill S, Levin A, Djurdjev O, Yoshida EM. Obstacles to residents' conducting research and predictors of publication. Acad Med. 2001;76(5):477. doi: 10.1097/00001888-200105000-00021. [DOI] [PubMed] [Google Scholar]
- 23.Ullrich N, Botelho CA, Hibberd P, Bernstein HH. Research during pediatric residency: predictors and resident-determined influences. Acad Med. 2003;78(12):1253–1258. doi: 10.1097/00001888-200312000-00014. [DOI] [PubMed] [Google Scholar]
- 24.Abramson EL, Naifeh MM, Stevenson MD, Todd C, Henry ED, Chiu YC, et al. Research training among pediatric residency programs: a national assessment. Acad Med. 2014;89(12):1–7. doi: 10.1097/ACM.0000000000000404. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Lang TA. The value of systematic reviews as research activities in medical education. Acad Med. 2004;79(11):1067–1072. doi: 10.1097/00001888-200411000-00011. [DOI] [PubMed] [Google Scholar]
- 26.Moher D, Liberati A, Tezlaff J, Altman DG. for the PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. doi: 10.1371/journal.pmed.1000097. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.The Cochrane Collaboration. Cochrane Database of Systematic Reviews. 2015 http://community.cochrane.org/editorial-and-publishing-policy-resource/cochrane-database-systematic-reviews-cdsr. Accessed May 28. [Google Scholar]
- 28.Badgett RG, O'Keefe M, Henderson MC. Using systematic reviews in clinical education. Ann Intern Med. 1997;126(11):886–891. doi: 10.7326/0003-4819-126-11-199706010-00006. [DOI] [PubMed] [Google Scholar]
- 29.Bloom BS. Learning for Mastery. Instruction and Curriculum. 2. Vol. 1. Regional Education Laboratory for the Carolinas and Virginia, Topical Papers and Reprints, Number 1. Evaluation Comment; 1968. p. n2. [Google Scholar]
- 30.Knowles MS. Andragogy in Action: Applying Modern Principles of Adult Education. San Francisco, CA: Jossey Bass; 1984. [Google Scholar]
- 31.Blackboard Learning Essentials. 2015 http://www.blackboard.com/higher-education/learning-solutions/learning-essentials.aspx. Accessed January 19. [Google Scholar]
- 32.Bhanji F, Gottesman R, de Grave W, Steinert Y, Winer LR. The retrospective pre-post: a practical method to evaluate learning from an educational program. Acad Emerg Med. 2012;19(2):189–194. doi: 10.1111/j.1553-2712.2011.01270.x. [DOI] [PubMed] [Google Scholar]