A Review of Journal Clubs in Postgraduate Medical Education

Abstract

OBJECTIVE

To review the goals, organization, and teaching methods of journal clubs, summarize elements of successful clubs, and evaluate their effect on reading habits, and effectiveness in meeting teaching goals. Examples of clubs that utilize principles of adult learning are reviewed.

DATA SOURCES

English language articles identified through a MEDLINE search (1966–1997) using the MeSH terms “internship” and “residency,” and text words “journal club” and “critical appraisal.”

STUDY SELECTION

Articles on learning goals and organization were included if they represented national or regional surveys with a response rate of 65% or greater. Articles that evaluated teaching effectiveness were included if they used a controlled, educational design, or if they exemplified important adult learning principles.

DATA EXTRACTION

Data were manually extracted from selected studies and reviews.

DATA SYNTHESIS

A major goal for most clubs is to teach critical appraisal skills. Clubs with high attendance and longevity are characterized by mandatory attendance, availability of food, and perceived importance by the program director. Residents who are taught critical appraisal report paying more attention to the methods and are more skeptical of the conclusions, and have increased knowledge of clinical epidemiology and biostatistics, but studies have failed to demonstrate that these residents read more, or read more critically. Reading guidelines may be useful for teaching critical appraisal skills, and may be associated with increased resident satisfaction.

CONCLUSIONS

Journal club formats are educationally diverse, can incorporate adult learning principles, and are an adaptable format for teaching the “new basic sciences.”

Possibly the earliest reference to a journal club is in the biography of Sir James Paget during the period of 1835–1854, in which it is noted that a group of students met in a room over a baker's shop near St. Bartholomew's Hospital to read journals or play cards.¹ However, it is Cushing's description of Osler's journal club in Montreal in 1875 that is usually credited as the first report of such an organization in North America.¹ Since these early descriptions, journal clubs have become ubiquitous in graduate medical education training programs. Initially, journal clubs served to help practitioners stay abreast of scientific developments,^1–4 but more recently they have been used as a vehicle to teach critical appraisal skills, research design, medical statistics, clinical decision theory, and clinical epidemiology.^5–8 In internal medicine, the desirability of journal clubs has been reinforced by publication of the most recent accreditation requirements for training programs that mandate resident participation in journal clubs and instruction in the critical assessment of the medical literature, clinical epidemiology, and medical statistics.⁹

Although most training programs sponsor journal clubs, they are not equally successful in meeting their educational goals, or maintaining resident interest. Many journal clubs begin with a flurry of activity, languish in 1 or 2 years, and are renewed again with a new generation of residents. Common reasons for discontinuing journal clubs appear to be lack of time, inadequate preparation, and lack of goals, interest, or participation.^{2, 4} Despite these problems, journal clubs continue to be found in most training programs, possibly due to their educational diversity, adaptability, and topical pertinence. Journal clubs can be student-centered, problem-oriented, and relevant to the learner's immediate experience or long-term goals; they can relate theory to practice, be interactive, and may be able to teach skills that enable lifelong learning.

Over the last 10 years, a number of publications have documented the goals, organization, and teaching methods of journal clubs and measured important learner or programmatic outcomes. A summary of this information may be helpful to those responsible for organizing a journal club or teaching clinical epidemiology, research methodology, or critical appraisal skills. The purpose of this article is to review the goals, organization, and teaching methods of journal clubs, summarize elements of successful journal clubs, and evaluate the effect of journal clubs on resident reading habits and their effectiveness in meeting their stated teaching goals. Finally, examples of innovative journal clubs that utilize important principles of adult learning theory are reviewed.

METHODS

English language articles were identified through a MEDLINE (1966–1997) search using MeSH headings “internship” and “residency,” and the text words “journal club” or “critical appraisal.” The references of these articles were reviewed for additional pertinent articles. Articles on learning goals and organizational structure were included if they represented national or regional surveys and had a response rate of 65% or more. Articles that evaluated the impact of journal clubs on resident reading habits, critical appraisal skills, and clinical epidemiology were included if they used a controlled educational design. Articles presenting innovations in journal club format were included if the approaches were unique or exemplified important adult learning techniques. Using this search strategy, 50 articles were identified and reviewed. Of this number, 28 were eliminated, 19 because they were not pertinent to the purpose of the study, and 9 because they did not meet any one of the three inclusion criteria.

This review is not a structured meta-analysis of survey results or educational trials because of the paucity of published trials that meet the typical criteria for such an endeavor. However, I sought all published information that might be related to the optimal performance of journal clubs as an educational experience, attempting to highlight strengths and weaknesses.

RESULTS

Stated Goals of Journal Clubs

Articulating the goals of a learning experience is an important pedagogical step.¹⁰ In a survey of internal medicine chief residents in New York City, the three most common goals for journal clubs were to teach critical appraisal skills (67%), to have an impact on clinical practice (59%), and to keep up with the current medical literature (56%).¹¹ Chief residents in North American pediatric training programs responded to a similar survey by selecting teaching critical appraisal skills and keeping up with the literature as equally important goals, accounting for 77% of the responses; if only one goal was selected by the respondents, teaching critical appraisal skills was considered most important.¹² In family practice training programs, program directors ranked keeping participants abreast of current literature, followed by improving critical appraisal skills, and allowing residents and the faculty to work together on a common project as their three most important goals, but the responses were not quantified.² In a national survey of chief residents in physical medicine and rehabilitation, 52% of the respondents ranked disseminating new information as the most important goal of a journal club, improving clinical practice was ranked second, and teaching critical appraisal skills was ranked a distant third with only 28% of the first-place votes.¹³ In a national survey of emergency medicine residency programs, Jouriles et al. reported that 42% of training programs did not have learning goals for their journal clubs.¹⁴ Although they did not directly ask program directors to list their goals, these authors reported that the three most common journal club formats were designed to keep the residents current (21%), to teach research design (21%), or to review selected topics (16.2%); teaching critical appraisal of the medical literature was not listed as an organizational format.

Prevalence and Organization of Journal Clubs

In a 1995 survey of program directors in the eastern United States, Sidorov found that 95% of internal medicine training programs had an active journal club,¹⁵ as compared with 79% in family practice programs reported a decade earlier.² Between 50% and 60% of internal medicine, family medicine, and pediatric programs mandate resident attendance to the journal club, with internal medicine and family practice programs reporting an estimated attendance rate of 60%.^{2, 15, 16} In 1995, 100% of 75 accredited programs in physical medicine and rehabilitation had at least one journal club, and attendance was mandatory in 80%.¹³

More than 80% of internal medicine, family practice, emergency medicine, and physical medicine journal clubs meet at least once monthly, usually during the working hours, typically reviewing between two and three articles per session.^{2, 13–15} The presence of designated leaders for the journal club was specifically mentioned in one family practice survey, in which they were reported present in 86% of programs.²

Based on a survey of chief residents in eastern internal medicine training programs, only original research articles were reviewed in 50% of the journal clubs, and formal teaching of clinical epidemiology and biostatistics was provided in 23% of the programs.¹⁵ In contrast, a 1996 national survey of internal medicine program directors reported formal teaching of clinical epidemiology and biostatistics in 69% of programs.¹⁷

Factors Associated with Successful Journal Clubs

Sidorov defined successful journal clubs as those with longevity (at least 2 years) and high levels of resident participation (at least 50% attendance).¹⁵ In his study, journal club longevity was associated with the regular provision of food, and the presentation of only original research articles (i.e., not clinical reviews or case reports). Interestingly, although the strategy of restricting journal club articles to original research was associated with longevity, it was also associated with the lowest resident attendance rates. High resident attendance rates were associated with smaller programs, mandatory attendance, independent from faculty journal clubs, and formal instruction in biostatistics and clinical epidemiology. Journal clubs with both longevity and high attendance were characterized by mandatory attendance, availability of food, and association with smaller training programs.

Heiligman and Wollitzer defined success in family practice journal clubs by the level of satisfaction of program directors with the educational experience.² Variables associated with high levels of satisfaction were the regular attendance by program faculty and community physicians, having a designated leader, and mandatory attendance. Jouriles and colleagues also defined success in emergency medicine journal clubs by an overall satisfaction score by program directors.¹⁴ The overall satisfaction score was highest when the journal club was held in the evening or at a faculty member's home. The format of the journal club was not associated with the overall satisfaction score.

In a survey of family practice journal clubs in the southeast, Van Derwood et al. assessed journal clubs by resident attendance and perceived educational value by the program director.¹⁶ Attendance rates were highest in clubs moderated by the faculty or a faculty-and-resident team as compared with clubs moderated by residents alone. There was a significant correlation between the attendance at journal clubs by the faculty and residents and the program director's perception of its educational value. In all cases, attendance was highest when program directors rated its educational value as “vital” and lowest when it was rated as having “no educational impact.” In contrast, Linzer demonstrated higher attendance rates in a journal club led by a chief resident, with invited subspecialist faculty, as compared with one moderated by a general internist.¹⁸ However, residents assigned to the faculty-led club were considerably more pleased with their educational experience.

Impact of Journal Clubs on Reading Habits, Knowledge, and Skills

A number of authors have measured the impact of journal clubs on reading habits, knowledge of clinical epidemiology and biostatistics, and the critical appraisal skills of its participants (Table 1) In defining critical appraisal skills, most authors have stated or implied that this entails the ability to assess the validity and applicability of published clinical evidence. As previously noted, Linzer and colleagues compared two formats for teaching critical reading skills to internal medicine residents; one journal club was led by a generalist faculty member, and the other by a chief resident with invited subspecialists.¹⁸ Both teams met weekly and reviewed articles selected for their clinical as well as methodologic interest.

Table 1.

Table Summary of Controlled Educational Trials to Teach Residents Clinical Epidemiology, Biostatistics, and Critical Appraisal Skills

At the end of the academic year, both teams were tested for their knowledge of clinical epidemiology and surveyed regarding their reading habits. Scores on the objective examination were identical for the two teams. The faculty-led team reported reading fewer articles, but there was a trend of reading them more completely. More members from the faculty-led team reported that the journal club experience changed their reading habits, primarily by making them more skeptical of the author's conclusions. Neither team reported that the journal club experience altered the ways in which they utilized the literature in their practice of medicine. Although this trial is important as one of the first attempts to measure the effects of a journal club experience on participants' reading habits and critical appraisal, it is limited by its lack of randomization, preintervention evaluations, and formal evaluation of critical appraisal skills.

In a subsequent study, Linzer and colleagues measured the effect of a journal club that emphasized epidemiologic methods and critical appraisal skills on internal medicine residents' reading habits, knowledge of epidemiology and biostatistics, and critical appraisal skills.¹⁹ First-year residents were randomly assigned to a journal club or a series of lectures on ambulatory medicine; the average resident attended five educational sessions. All participants received preintervention and postintervention evaluations, separated by 9.5 months. The number of articles read declined in both groups by the end of the intervention, and there was no change in the proportion of articles that were completely read. However, journal club residents reported a marked improvement in their ability to read critically, and to subsequently incorporate the information into their practice. Only the journal club members reported a change in their reading habits, becoming more skeptical, and paying more attention to methods. Journal club members, but not the control group, significantly improved their scores on the epidemiology and biostatistics test; however, neither group demonstrated an improvement in critical appraisal skills when formally tested. The strengths of this trial are its randomized, controlled design and the use of preintervention and postintervention evaluations. A weakness of the trial may have been the “dose” of the intervention; an average of five sessions over 9 months may have been insufficient to learn the skills needed for critical appraisal.

Kitchens and Pfeiffer developed a curriculum and tested its effectiveness in teaching critical appraisal skills and clinical epidemiology to internal medicine residents in a controlled trial.²⁰ Two groups of residents began the trial: group A received the critical appraisal/clinical epidemiology curriculum; group B, an ambulatory curriculum. After the intervention, each group was tested for its knowledge of clinical epidemiology, then the groups switched curricula and were retested after completing the second intervention. Group B showed a statistically significant, although modest, improvement in clinical epidemiology knowledge, and group A maintained knowledge over the course of the trial. Although the groups were not randomized, they did appear to be comparable as judged by the results of their in-service training examination. Despite the stated intent, the trial did not actually test critical appraisal skills.

Langkamp et al. evaluated the effect of a journal club intervention on pediatric resident acquisition of clinical epidemiology and biostatistical knowledge.²¹ The intervention group received 2 hours of didactic lectures on research design, clinical epidemiology, and biostatistics, followed by eight monthly journal club session in which two articles were presented by residents who emphasized the article's methodology and content. Both control and intervention groups received preintervention or postintervention evaluations, separated by 9 months. The groups did not differ in their preintervention or postintervention scores. This study was limited by lack of randomization and power, and critical appraisal skills were not tested.

Although it was not a trial of an educational intervention, a study by Stern and colleagues reported the development of a new instrument to measure critical appraisal skills.²² In their model, the abstract, introduction, and discussion of the test article were removed to encourage unbiased evaluation by the trainee. Eleven statements about the article based on the McMaster reading guidelines (see below), and specific for the type of research article being reviewed, were provided, and each statement was linked to a Likert type of response. A correct answer for each statement was based on the responses from three expert faculty who also reviewed the article; variability among the experts was low, with 98% of their responses within 1 point of the median response. The residents' ability to assess the article was measured as the deviation from the correct response and used to calculate a percentage correct score. This instrument has the potential to measure objectively whether critical reading skills of trainees improve with instruction, and it appears to have face and content validity; however, additional testing in other settings is recommended by the authors.

A number of authors have advocated checklists or reading guides as tools for teaching critical appraisal skills.^{6, 7, 23, 24} The checklist or reading guide may offer a number of advantages, including consistency of review, speed, and help to maintain alertness of the reviewer for potential methodologic flaws. Probably the most widely used reading guides have been published by the McMaster University clinical epidemiology faculty. These reading guides consist of a background explanatory article and a summary of the important points in the form of focused questions.^25–28 The McMaster reading guides have been used in a number of educational trials designed to evaluate the effectiveness of teaching critical appraisal skills and clinical epidemiology to both students and residents.^{19, 20, 22, 29} Similar reading guidelines published by Rigelman have been used in a controlled trial of teaching critical appraisal skills to medical students.³⁰ Alguire et al. published a packaged critical appraisal curriculum that included reading guides and other teaching materials.²⁴

The use of a reading checklist has been evaluated by Burstein et al. in an emergency medicine training program.³¹ A satisfaction survey was distributed to all residents before and after the introduction of a “structured review instrument” into the monthly journal club. Resident use of the structured review instrument was associated with higher overall satisfaction with the journal club format as well as a perceived improvement in clinical education, but not with an increase in resident or journal club leader workloads. Improvement in critical appraisal skills was not tested.

Journal Clubs and Adult Learning

Educational research has identified several unique characteristics of adult learners, and it may be that instruction that incorporates these principles will enhance the learning experience for residents. Some of the more important adult learning principles are relating the learning task to the immediate work experience or long-term goals, presenting the learning objectives in the form of actual situations or patient problems, use of problem-solving, use of multiple teaching formats, active learner involvement, and use of frequent, constructive feedback. The purpose of this section is to describe briefly examples of journal clubs that meet many of these criteria, and have promise as effective teaching strategies, but have not been formally evaluated.

Inui published one of the first descriptions of a journal club using adult learning principles.⁵ Each session is devoted to the review of a single paper chosen by the residents because it has been alluded to recently by faculty or other residents in the process of caring for patients. During the session, the presenting resident identifies the study's objectives and hypotheses, then the conclusion is summarized as well as its local application, that is, how the study is actually being used on the wards in the care of patients. The faculty leader then facilitates a discussion of the study by the residents, and an explicit attempt is made to determine whether the study's inference is valid or flawed by methodologic problems, and whether the local application of the study is appropriate. This format has the advantage of relating the learning to the residents' immediate experience, presents the learning in terms of an actual patient problem, and involves active learner participation.

A problem-based journal club has been described by Joorabchi.³² Two or three residents each select a clinical study for presentation. Excerpts from the main sections of the paper are prepared along with a set of open-ended questions by the faculty leader. Each member of journal club is asked to individually record answers to the questions. Following this, small groups of four or five residents are formed, and each group is asked to arrive at consensual answers. Once the small groups have reached their conclusions, a representative from each group presents their answers to the larger group. The questions focus on the skills that are needed for critical appraisal. For example, residents may be asked to review the study hypothesis and design an experiment that would test the hypothesis; to evaluate the methods in terms of validity, reliability, feasibility, relevance, and cost; to interpret the primary data and compare their interpretation with the author's; or to review the raw data and choose the most appropriate statistical test. This format illustrates an interesting use of multiple teaching formats, and seems to maximize learner participation. Unfortunately, an estimate of necessary faculty preparation time was not provided.

Kitching described a cardiology journal club with four distinct activities.⁸ The first step consists of a trainee describing a case with an unresolved clinical problem. The problem selected must coincide with the critical appraisal theme chosen for the journal club session. Next, the trainee outlines and defends the search strategy used to obtain relevant literature and shares any tactics that resulted in difficulty or success. Following this, the trainee reviews the article using selected reading guides, in an attempt to determine if the study is capable of answering the research question. Finally, the journal club group discusses the validity and applicability of the study's conclusions and whether the study has solved the original clinical problem. This model emphasizes learning that is relevant to the learners' immediate task, is problem-focused, involves problem solving, and includes the process of actually finding relevant literature, as well as its critical appraisal.

SUMMARY

An important, explicit educational goal of most journal clubs is to teach critical appraisal skills, and for most programs, this is rated higher than keeping up with the medical literature. A journal club is an integral part of most training programs, and is often the format used to teach clinical epidemiology and biostatistics. Success of journal clubs can be defined by many different parameters, but factors associated with high attendance and longevity include mandatory attendance, availability of food, and perceived educational value by the program director. The impact of a resident or faculty moderator on attendance rates is not clear, but educational satisfaction may be higher in clubs moderated by the faculty.

Residents exposed to critical appraisal techniques in a journal club report paying more attention to the methods and becoming more skeptical of the author's conclusions. However, the amount of resident reading and the degree to which the resident incorporates new information into patient care may not be influenced by a journal club experience. Well-designed educational trials have demonstrated that it is possible to improve basic knowledge in clinical epidemiology and biostatistics, but not critical appraisal skills, using a journal club format. It may be that teaching critical appraisal skills requires more time on task than was provided in these educational experiments. Critical appraisal skills have been successfully taught in other settings.²⁹

Reading guides or checklists may be useful tools for journal clubs. They can help a trainee organize an oral presentation of an article, remind the trainee of potential methodologic flaws, and may be associated with higher trainee satisfaction with the journal club. Building on the foundation developed by the McMaster group,^25–28 a second generation of users' guides have been published beginning in 1993 by The Evidence-Based Medicine Working Group in the Journal of the American Medical Association and are increasingly used as a tool to help teach critical appraisal skills.^33–50

It is clear from this literature review that an ideal journal club format does not exist. Rather, the format depends on the goals of the journal club, the parameters used to define success, and the available resources. Nevertheless, a few common steps can be suggested for those faculty charged with initiating or “overhauling” their existing journal clubs. First, clearly articulate the goals. This step will aid in the creation of learning objectives, teaching format, and evaluation strategies. Next, select the most appropriate format, keeping in mind that longevity and high attendance rates are associated with mandatory attendance, and the provision of food. It is likely that direct teaching involvement by the department chairman and other respected faculty members will increase resident participation. The club moderator, whether faculty or resident, needs to be skilled in facilitating small group activities. Faculty development sessions that teach small group leadership skills are frequently offered at national specialty meetings including the Society of General Internal Medicine and Society of Teachers in Family Medicine. Greater resident satisfaction with a journal club may be achieved by utilizing adult learning principles including relating the learning task to immediate patient care experiences, use of problem-solving techniques, multiple teaching formats, and active learner involvement. Finally, the use of a structured format to review the literature may improve the residents' understanding of the material, the efficiency of their presentations, and their satisfaction with the educational experience.

For example, if the major teaching goals for a journal club were to teach critical appraisal skills and to improve the care of patients, the journal club could be designed to answer one or more clinical questions that are currently active on the medical ward or ambulatory clinic. A small part of each journal club could be used to teach the basic elements of critical appraisal or clinical epidemiology to give the residents the necessary tools to succeed in this endeavor. The clinical question could be addressed by using evidence-based medicine techniques with the help of users' guides or other structured review formats. A group consensus could be reached, and compared with the actual care being rendered.

Journal club formats are educationally diverse, and can allow for the incorporation of many important principles of adult learning. Journal clubs remain a fertile area for educational research, and provide an adaptable format for the teaching of the “new basic sciences.”