Taking Journal Clubs off Autopilot: A Case Study of Teaching Literature Evaluation Skills to Preclinical MD/PhD Students

Abstract

Researchers designed learner-directed journal clubs to develop literature evaluation skills in preclinical students. Sessions balanced student-led discussion with structured objectives and faculty support. During the pilot with preclinical MD/PhD students, self-rated mastery improved over all 17 measured objectives. Six exercises have since been incorporated into the full medical school curriculum.

Introduction

A clinician workforce trained in the principles of medical research is necessary both to conduct research and to apply findings in daily clinical practice. Clinicians must be involved in the design, implementation, and dissemination of research so that the questions addressed are clinically relevant and results are meaningfully reported in the literature. All clinicians, not only clinician-scientists, must be informed consumers of medical literature in order to apply advances to evidence-based patient care.¹

Journal clubs are ubiquitous at all levels of medical training to help healthcare professionals become informed consumers of the literature. Ironically, only limited evidence exists to suggest that journal clubs effectively support evidence-based decision making. The literature on this topic is limited by a lack of high-level, outcomes-based reporting. Additionally, the heterogeneity journal club formats described in published studies makes their conclusions difficult to generalize.²

At the University of Cincinnati College of Medicine (UCCOM), the MD/PhD Medical Scientist Training Program (MSTP) holds a weekly one-hour journal club for its physician-scientist trainees. As of 2010 this journal club had historically poor attendance and participation. In the summer of 2010, an average of 9/32 (28%) preclinical MSTP students attended sessions which typically ended after 30–45 minutes despite being scheduled for an hour.

As the MSTP identified concerns with its journal club, a UCCOM self-study demonstrated that the medical curriculum needed improvement in clinical and translational research training. This topic was particularly important given that education on how such research “is conducted, evaluated, explained to patients, and applied to patient care” is an accreditation requirement for U.S. and Canadian medical schools.³ A journal club format was chosen to address these concerns due to the perceived parallels between preparing for journal club and the reality of self-directed literature interpretation in clinical practice. Enhancements to the MSTP journal club thus became a pilot for a journal club to be incorporated into the small group learning sessions for all preclinical students.

The authors, tasked with leading the MSTP journal club during this transformation, noted that the journal club format was somewhat automatically selected as the method to teach critical literature evaluation skills. The authors conducted both a literature search and an informal survey of MSTP students to assess the strengths and weaknesses of this learning modality. The results were surprisingly similar.

Student participants reported that the student-directed format and actual application of literature review skills in journal club was a welcome departure from lectures. However, students were concerned about the inconsistent quality of student-selected articles and an overall lack of direct relevance of the journal club to their preclinical medical training. Similarly, a Best Evidence in Medical Education review stated that the major strengths of journal clubs are the active, learner-directed components. However, unclear learning objectives often make journal clubs problematic.² Journal clubs with more structure appear to be more effective. Characteristics of successful journal clubs include: regularly scheduled meetings, mandatory attendance, clear long- and short- term goals, and adequate time for learners to read papers before the session.⁴

The authors concluded from literature review and student survey that, for a journal club to be successful, it must have learning components tailored to its target audience and a clearly defined purpose. Two major objectives for this innovation were chosen: (1) tailor the MSTP journal club to the unique needs and skills of preclinical students and (2) create a generalizable, team-based learning format to give students the skills necessary to be informed consumers of the clinical and translational research literature.

Methods

The curriculum, described in Table 1, consisted of ten weekly sessions scheduled for one hour each. With support from the institutional Center for Clinical and Translational Research and Training (CCTST), the authors identified topics, learning objectives, and faculty content experts for each session. Depending upon the topic and learning objectives, additional elements beyond literature review were incorporated into the discussion, such as discussion of ethical dilemmas and how research can be presented to patients.

Table 1.

Program curriculum.

Session topic	Learning Objectives	Session Components
What is clinical and translational research?	Define “translational research.” Distinguish translational research from basic and clinical research.	Guest speaker on the definition of translational research Student-led discussion of recent translational paper produced with support from the University of Cincinnati’s Clinical and Translational Science Award (CTSA)
Study design	Understand basic tenets of good study design.	Student-led analysis of the design of a widely criticized, retracted paper Clinician-supported discussion on how reading the paper before its retraction might have modified students’ clinical practices
Peer review	Understand the process of peer review. Define “impact factor.” Know how to determine a journal’s impact factor.	Individual student mock peer review of a de-identified paper Student-led discussion on the paper’s validity and significance, with later revelation to students that the paper had been retracted for methodological inconsistencies
Patient and recruitment bias	Discuss how recruitment bias may influence the results of a study.	Student-led discussion of a clinical paper whose comorbidity exclusion criteria lead to a study population significantly different than the actual patient population Clinician perspective on how this difference impacts the translation of research literature into their own clinical practice
Placebos and placebo effects	Understand the purpose of placebo controls. Discuss ethical considerations associated with placebo controls.	Student-led discussion of ethical issues surrounding placebo use in clinical trials and practice Student-led discussion on how “real” the placebo effect is and its impact on clinical practice
Clinical vs. statistical significance	Understand what it means for a result to be “statistically significant.” Distinguish statistical, biological, and clinical significance.	Student-led detailed analysis breaking down the statistics in a major clinical trial Statistician perspective on the utility of the “p<0,05” significance standard and what factors affect p values
Conflict of Interest (COI)	Clearly define “conflict of interest.” Discuss how conflict of interest may influence study design or interpretation.	Student review of a groundbreaking clinical trial discredited due to undisclosed author COI, with later revelation to students that the study’s results were recently replicated Student-led discussion of ethics issues surrounding COI and its impact on the delay or inappropriate adoption of new therapeutics
Cost effectiveness	Understand how the cost-effectiveness of a treatment or intervention is evaluated.	Student analysis of a cost-effectiveness study on cancer screening methods Student-led discussion on the clinical purpose of screening and the ethical issues surrounding cost-effectiveness as a metric for test utility
Literature Wars	Be able to evaluate the quality of a scientific paper. Be able to discuss a scientific paper with colleagues.	Student-led comparison of two multi-center trials, with different conclusions, of tPA treatment for patients with acute ischemic stroke Health sciences librarian perspective on literature search methods
Real World Validity	Clearly distinguish internal and external validity.	Evaluation of effectiveness and efficacy studies of the rotavirus vaccine Clinician-researcher perspective on how efficacy/effectiveness studies affect clinician adoption of vaccine recommendations

Two to four students signed up to lead each session based on personal interest and availability. Session leaders were provided with learning objectives and contact information for the session’s content expert. Leaders and content experts collaboratively identified an article relevant to the topic and appropriate for preclinical students. This article was provided to participants at least five days before each session.

Student leaders were responsible for a 10–15 minute introduction intended to give preclinical attendees the background necessary to participate in group discussion. This introduction typically covered the pre-defined learning objectives, definitions of relevant terms, and clinical information relevant to the article at hand. Afterwards, student leaders, assisted by the content expert, facilitated a group discussion of each table/figure and the article’s overall relevance to clinical practice.

The pilot program was evaluated through attendance numbers and anonymous, paired pre- and post- intervention surveys as well as solicited participant feedback. Survey questions included both close-ended, five-point Likert scale items and open-ended reflections on the educational program. Seventeen closed-ended questions evaluated students’ achievement of learning objectives, while eight closed-ended questions evaluated the journal club’s overall efficacy before and after intervention. While this activity was optional, fourteen MSTP students received a credit hour by attending at least 8/10 sessions and leading one.

Representative Session

The “Literature Wars” session aimed to develop students’ skills in clinical decision-making in the face of conflicting research evidence. Students critically compared two conflicting articles describing different randomized controlled trials of tissue plasminogen activator (t-PA) for the treatment of acute ischemic stroke.

The student-led introduction provided an overview of t-PA use in stroke. The group discussion then focused on a side-by-side comparison of figures and methodological differences in each article to understand the differing conclusions. Students next discussed their own clinical conclusions on the effectiveness of t-PA and how they would present these conclusions to a patient. Students then explored if they would follow this recommendation for themselves or a loved one in the same clinical situation, and the ethical implications of the disparity in patient vs. personal recommendation identified by several students.

Results

These sessions consistently exceeded the scheduled hour and were attended by 26/32 (81%) of preclinical MSTP students, a significant improvement from the previous year. Additionally, four preclinical medical students conducting summer research heard about the journal club through the CCTST and participated. An average of 17 preclinical students attended each session.

The pre-intervention survey was open to MSTP students who participated in the first or second session, and had a response rate of 76% (14/19). The post-intervention survey was open to all participating MSTP students and had a response rate of 62% (16/26). We received paired pre-and post-intervention surveys for 61% (11/18) of MSTP students who attended at least half of the sessions.

Students’ self-reported achievement of learning objectives improved for all 17 objectives assessed, shown in Figure 1. Space limitations prevent showing all results. To reduce bias, these six representative objectives were chosen before the authors viewed the results.

Figure 1.

Self-Reported Achievement of Learning Objectives. Self-rated knowledge scores for six representative learning objectives (n=16 students who attended at least 5/10 sessions.). Students showed significant improvement in three learning objectives by the paired Wilcoxon Rank Sum test (n = 11 students completing both pre- and post-surveys).

Students gave ratings of 4 or 5 (of 5) for all 11 questions regarding the educational utility of the new format. Student comments suggested an appreciation of the structure provided to this student-directed activity. Representative comments include:

• “There was a theme for each journal club session. As a student, this gave me a more concrete goal to what I should be learning.”

• “There was a clear topic each week & I liked the structure that provided, rather than let's-just-discuss-this-paper-&-see-what-we-come-up-with.”

• “Being honest, a lot of people (including myself) pick papers that feature science that isn't solid or are otherwise lacking…(because we don't have time or whatever) and that makes the journal clubs not as useful as they could be. Having good exemplar papers was definitely helpful.”

To our knowledge, during this summer activity students were not participating in additional journal clubs or other organized academic activities which might otherwise explain this improvement.

Discussion

This innovation sought to take a deliberate approach to the ubiquitous tool of journal club by tailoring it to the specific needs of preclinical medical students. Defined learning objectives and a mix of structured and learner-directed components contributed to the innovation’s success. The most encouraging indicator of this innovation’s success was the sustained, enthusiastic student participation in this optional pilot program. The voluntary participation of 30 preclinical students and the high feedback scores suggest achievement of our first objective to tailor the journal club to these students.

The second objective to design a generalizable team-based learning format for clinical and translational research education was fully achieved. After the pilot’s success, UCCOM partnered with student coordinators to develop six sessions for the medical school which were implemented in 2012–2013.

A limitation of this innovation, inherent in its pilot sample size, is the ability to evaluate its impact beyond student self-report. This problem is common to published reports on journal clubs.^2,4 It is possible that results are positively biased due to the self-selection of participants in this pilot program or students’ skewed perception of personal progress. These potential biases suggest that the survey results should be taken as broad indicators of this format’s usefulness in the preclinical setting rather than unqualified proof of its success.

As this program is scaled up to the full preclinical curriculum, evaluation will include direct assessment of students’ knowledge. Because preclinical students attending a required medical school activity have different needs than students self-selecting for interest in research, the exercises developed for the medical school have an even greater emphasis on patient care and explaining results to patients. One exercise in development, recently piloted, involves having students discuss popular media reports on medical research with non-medical friends and family before locating the original research and discussing it in journal club. Exercises involving student explanation of research to patients are also being developed, which will allow for external assessment of student attitudes and changes in practice.

Institutions considering similar educational programs may wish to pay attention to the careful balance between providing structure and context for preclinical learners while still allowing students to direct their own learning. Pre-defined learning objectives, a semi-structured format, suggestions of quality papers, and faculty support give students the tools to lead a successful small group. Such structured student-led groups allow students to learn and present new concepts, strengthen faculty and student connections, and develop student confidence in self-directed learning and presentation skills.⁵

This innovation is now an official part of the UCCOM physician-scientist training curriculum, with outgoing coordinators advising incoming coordinators to ensure sustainability. The authors look forward to further piloting innovations in clinical and translational research education for general preclinical training.

Biographies

REBECCA L. CURRIER is an MD/PhD student in the Medical Scientist Training Program, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA and Division of Epidemiology and Biostatistics, Department of Environmental Health, Cincinnati, OH, USA.

MARGUERITE REID SCHNEIDER is an MD/PhD student in the Medical Scientist Training Program, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA and Neuroscience Graduate Program, University of Cincinnati College of Medicine, Cincinnati, OH, USA.

JAMES E. HEUBI, MD, is co-Director of the Center for Clinical and Translational Science and Training, a collaborative effort of the University of Cincinnati and Cincinnati Children’s Hospital Medical Center, Cincinnati, OH, USA.