Learning to Critically Appraise Rheumatic Disease Literature: Educational Opportunities During Training and into Practice

Summary

Rheumatologists must perform critical appraisal to provide optimal patient care. Currently, there is a dearth of validated and effective resources to teach and encourage critical appraisal in rheumatology. Conceptual frameworks of Retrieval Enhanced Learning, Self-Determination Theory, and Communities of Practice can inform design of educational approaches such as HSS CLASS-Rheum to promote development of critical appraisal skills and use of critical appraisal in practice. Engaging trainees and rheumatologists in activities that involve retrieval of critical appraisal knowledge has the potential to increase their learning and ability to use that material in the future. Acquisition of basic critical appraisal knowledge through retrieval practice can facilitate rheumatologists’ entry to critical appraisal Communities of Practice. Opportunities for trainees and rheumatologists to connect through Peer Instruction, journal clubs, and a variety of virtual forums can help expand formation of and access to critical appraisal Communities of Practice. Critical appraisal Communities of Practice can foster internalization of motivation for critical appraisal, and ultimately promote application of critical appraisal in patient care.

The Need for Critical Appraisal Education

There have been tremendous scientific advances in rheumatology over the past decades. New diagnostic tools and therapeutic approaches have been developed, allowing us to prognosticate, communicate with, and treat patients better than ever. For patients to benefit from these advances, physicians cannot rely solely on their medical school and postgraduate training; it is imperative that physicians continue to learn.¹ With the advent of online publications, there has been an explosion in the volume of literature, making it more important to be able to identify which are the valid, high-quality studies.² Despite this, negative studies remain less likely to be published^3,4; recognizing this type of publication bias as well as the limitations and potential biases of available data must be considered.⁵ Rheumatologists must parse the literature and choose which data to act on. For example, it is crucial to identify high-quality studies to best determine which biologic would be most effective for a specific patient with nonradiographic spondyloarthritis,⁶ or to evaluate the generalizability of published data when deciding whether a patient with gout and a history of ischemic chest pain should be given febuxostat.⁷ Rheumatologists designing research or serving in administrative roles that influence clinical guidelines and policies should apply critical appraisal as well. Although literature reviews can be key resources for answering questions in lieu of reviewing primary data, selective assembly, bias, lack of applicability, and/or absence of the most current data may present significant drawbacks. Clinicians must be adept at identifying appropriate sources, properly interpreting evidence, and accurately sharing information with patients to optimize care in an information-saturated age.

Recognizing that all rheumatologists need to answer patient-centered questions, the Accreditation Council for Graduate Medical Education (ACGME) considers the ability to critique specialized scientific literature effectively to be a key characteristic of rheumatology trainees who are ready for unsupervised practice.⁸

Critical appraisal, the “…application of rules of evidence … to assess the validity of the data, completeness of reporting, methods and procedures, conclusions, compliance with ethical standards, etc.,”⁹ requires a specific set of knowledge and skills that can feel inaccessible to many physicians. If one is not fluent in the vocabulary of epidemiology and biostatistics, it can be bewildering to try to interpret, critique, and apply information from the literature. It takes time and resources to acquire the knowledge and skills required to critically appraise the literature, and additional time to maintain them.

Equipping rheumatologists with critical appraisal knowledge and skills will improve patient care only if critical appraisal is applied in clinical practice. Time constraints, insufficient support, false beliefs about evidence-based medicine, and limited critical appraisal skills have been identified as the most common barriers to health care providers’ performance of evidence-based practice.^10,11 It is therefore crucial to integrate critical appraisal skills into a framework that supports their implementation.

The Need for Resources for Critical Appraisal Education in Rheumatology Training Programs

Critical appraisal education at the undergraduate and graduate training levels is incompletely described in the literature. Available evidence indicates this topic is generally included in these curricula, but there is a large degree of variability in how, when, and where critical appraisal is taught.^12–15 Basic epidemiology and biostatistics are included on US Medical Licensing Examinations,¹⁶ but the small number of questions on these topics means it is possible to pass these examinations without correctly answering these questions. Despite inclusion of critical appraisal topics in undergraduate medical education curricula and residency training programs, trainees often demonstrate limited critical appraisal skills at the start of their rheumatology training. Thus, training programs should be structured to designate time and direct trainees to build critical appraisal skills, but it has been unclear how these skills should be taught.

Journal clubs involving use of critical appraisal are common components of training programs, but outcome data on their effects on subsequent evidence-based practice are limited.¹⁷ To improve the impact of journal clubs on evidence-based practice, Alguire¹⁷ suggested incorporating and formally evaluating adult learning principles in trainee journal clubs. A Cochrane systematic review found that, although teaching critical appraisal to medical professionals may improve knowledge, there was a lack of high-quality studies evaluating changes in process of care or patient outcomes after critical appraisal teaching.¹⁸ The investigators recognized a need for rigorous randomized trials of educational programs for critical appraisal informed by appropriate learning theories.

Educational resources available through the American College of Rheumatology (ACR) support trainees’ achievement of core competencies and specific skills such as ultrasound, bone densitometry, and practice management.¹ Resources for helping trainees learn elements of critical appraisal, however, are lacking. The ACR/Association of Rheumatology Health Professionals Annual Scientific Meeting¹⁹ includes a few sessions about statistical approaches, but these are generally targeted at researchers. Guided poster tours direct attendees’ attention to high-impact data but are not intended to teach critical appraisal. The ACR Continuing Assessment Review and Evaluation (CARE) modules include a few questions on epidemiology and biostatistics each year, but this handful of questions are necessarily narrow in scope.

The ACR has acknowledged the lack of curricular resources dedicated to critical appraisal in rheumatology training programs and the importance of addressing it. To fill this resource gap, the ACR recently embarked on a new initiative called Rheum4Science, to provide Web-based modules on elements of basic and clinical science that will allow asynchronous learning with interactive elements.²⁰ Some Rheum4Science modules will help promote critical appraisal, as there is overlap in the knowledge and skills relevant to both performing clinical research and critically appraising research publications.

Retrieval Enhanced Learning, Self-Determination Theory, and Communities of Practice as Frameworks for Educational Approaches to Critical Appraisal

Conceptual frameworks in education can serve as helpful lenses through which to view a problem and then design educational solutions.^21,22 In considering the challenges to teaching and learning critical appraisal, conceptual frameworks of Retrieval Enhanced Learning, Self-Determination Theory (Motivation), and Communities of Practice can guide effective interventions (Fig. 1).

Fig. 1.

Model of the integration of Retrieval Enhanced Learning Theory (Retrieval Practice), Self-Determination Theory (Motivation), and Communities of Practice to promote learning and application of critical appraisal.

Retrieval Enhanced Learning (Retrieval Practice)

Educational research demonstrates that learning approaches that require active learning, encoding, processing, and retrieval of learning lead to stronger memory, long-term retention, construction of new knowledge, and the ability to flexibly use that knowledge in new and unfamiliar contexts.^23–28 This is in contrast to more prevalent but less effective transmissionist approaches to education that rely on experts “transmitting” knowledge to passive learners.²⁹

Empirical evidence has established that testing does not simply measure knowledge, it changes the memory of the test taker, thereby causing learning.²³ Recalling information from memory, referred to as “retrieval practice,” promotes deep learning, and can do so more effectively than reading, reviewing, and re-reading materials. Retrieval practice has been proven to promote long-term retention of content across a range of complexity.^26,30 Beyond simply improving recall of an item, retrieval practice also promotes further encoding and processing of information and subsequent transfer of learning to different contexts,²⁸ allowing the knowledge gained through testing to be used flexibly in future situations with new and unfamiliar material (eg, knowledge about echolocation in bats can be transferred and applied to sonar in submarines).²³ Recalling knowledge in one domain can facilitate recall of proximate or related knowledge, even if it is not part of the tested item.³¹

There are hundreds of ways to engage learners in retrieval practice. Multiple-choice questions and short-answer formats can both be effective.²⁶ One approach involves posing a question, allowing time for the learner to recall and formulate an answer, and then providing feedback on the correct answer. When accompanied by feedback, retrieval supports and strengthens subsequent storage in memory. Feedback is important because it allows the learner to correct any errors and confidently maintain correct responses. Effective feedback includes correct answer identification and explanatory feedback for why that answer is correct. Explanatory feedback is particularly important for transfer of learning to new contexts.³² The structure of the retrieval practice can enhance learning; including repetition of the question, providing time between repeated retrieval attempts (spacing), and varying the type of retrieval activities can enhance the effects.³³ When the structure includes retrieval with feedback, learners are better able to engage in self-appraisal, self-monitoring, and self-regulation of learning.^33,34 Such metacognitive abilities are linked to a range of benefits, including deeper problem-solving skills and the ability to apply and use that learning more effectively in future situations.

This understanding of how tests can promote learning can inform the design of educational formats to improve retention and subsequent transfer of critical appraisal knowledge and skills.

Self-Determination Theory (Motivation)

Considering how to effectively motivate learners to engage in development of critical appraisal skills (through retrieval practice or otherwise) and use critical appraisal in practice is key when designing educational interventions. Deci and Ryan’s Self-Determination Theory provides a helpful framework to do so.^35–38

In some circumstances, learner motivation is driven by the appeal of the activity itself, rather than by a consequence of the activity. This “intrinsic motivation” can be quite powerful; when possible, educational tasks should be designed to maximize intrinsic appeal. Perceived competence (self-efficacy, the belief one can perform a task successfully)^39,40 and autonomy (self-direction, choice) have been demonstrated to be powerful internal motivators. Instilling a sense of self-efficacy for critical appraisal, and recognition of the ability to direct one’s own learning through critical appraisal, have the potential to motivate use of critical appraisal in practice.

Because of required roles and responsibilities, most adults’ activities are not primarily intrinsically motivated. Self-Determination Theory categorizes all forms of motivation that are not strictly intrinsic as forms of “extrinsic motivation.” The extent to which extrinsic motivations are self-determined, or internalized, has significant range. At one end of the continuum, behaviors are performed to satisfy an external demand, whereas at the other end of the continuum, behaviors are performed voluntarily, to obtain an outcome one personally values. Internalization of values is associated with enhanced engagement, persistence, and sense of well-being. Supporting senses of relatedness (belonging or connectedness), competence, and autonomy can foster internalization of values; we see that people are more motivated to perform a behavior when

1. they sense that behavior is valued by others they want to feel connected to

2. they sense they possess the relevant skills to perform it successfully

3. they personally appreciate the worth of the behavior

This framework can inform design of educational programs to enhance interest in, valuing of, and self-efficacy for critical appraisal, thereby promoting use of critical appraisal in practice. One may first be exposed to an activity in response to an external prompt, but then appreciate it as intrinsically interesting or personally valuable, leading to internalization of the motivation.

Communities of Practice

The establishment of “Communities of Practice” can help cultivate learning of critical appraisal skills in current and future generations of rheumatologists and advance the use of critical appraisal in rheumatology.

Emphasizing the social and situated nature of learning, Lave and Wenger⁴¹ described the educational framework of Communities of Practice as groups of people who “share and develop an overlapping knowledge base, set of beliefs, values, history and experiences focused on a common practice and/or mutual enterprise”.⁴² Communities of Practice are defined by 3 key features:

1. a shared domain of knowledge

2. a community of individuals working within the domain

3. the development of tools, resources, and innovations that advance the domain

Individuals interested in joining a Community of Practice move from peripheral participation as novice members, toward full membership as they acquire knowledge, skills, and a sense of shared identity or membership with the group. An individual’s learning can be seen as a journey through a landscape of Communities of Practice, modulated by the individual’s interest and access.

Communities of Practice can be problematic, as they can be exclusionary; there can be barriers to access and historical power relationships can be reinforced.⁴³ There can be a sense of discomfort when one is at the margins of a Community of Practice, unsure of the rules and unfamiliar with the language.^22,44,45 Sustaining a productive Community of Practice is dependent on new generations of members joining. “Participation, or coparticipation, with fellow learners and more senior members of the community…deepens the sense of engagement.”⁴²

Learning and applying critical appraisal can be seen as occurring within Communities of Practice; the domain is critical appraisal, the community is the network of individuals who engage in critical appraisal together, and the practice is the curation, development, and use of tools and resources to help advance critical appraisal. Communities of Practice can provide an infrastructure for learning, instilling a sense of the value and supporting the development of critical appraisal. As critical appraisal knowledge, skills, and values, are acquired, rheumatologists can move toward full membership in these Communities. An important educational goal is to actively assist novice learners to join these Communities.

Designing Curricular Support for Critical Appraisal during Training: Hospital for Special Surgery Critical Literature Assessment Skills Support – Rheumatology (HSS CLASS-Rheum)

Application of the educational frameworks of Retrieval Enhanced Learning, Self-Determination Theory, and Communities of Practice can be seen in the design and implementation of a new learning tool called Hospital for Special Surgery Critical Literature Assessment Skills Support – Rheumatology (HSS CLASS-Rheum®).

Identification of Learning Objectives

To establish which study designs are particularly pertinent to rheumatologists, our group reviewed 30% of the clinical research articles (excluding opinion pieces, letters, and reviews) published in Annals of Rheumatic Disease, Arthritis and Rheumatology, and Arthritis Care and Research over a 9-month period. We selected these journals based on impact factor, their focus on original research, and broad coverage of general rheumatology.⁴⁶ Registries, cohorts, and case control studies were most prevalent, followed by systematic reviews and meta-analyses, case series, and randomized controlled trials (RCTs) (Fig. 2). Although RCTs are generally thought of as providing a higher level of evidence, the high frequency of cohorts and case control studies in the rheumatic disease literature indicates that rheumatologists in particular must have a thorough understanding of these study designs. We used this information to inform the development of a set of key learning objectives reflecting these specific needs, to prepare rheumatology trainees to critically appraise the clinical rheumatic disease literature.

Fig. 2.

Distribution of study designs identified in clinical research articles from rheumatic disease literature.

Application of Retrieval Enhanced Learning (Retrieval Practice)

Given the proven benefits of retrieval practice, we developed a question-based curricular tool using varied response types. Recognizing the diversity of programs in terms of availability and expertise of faculty, and the different ways trainees’ time is organized, we designed HSS CLASS-Rheum to optimize flexibility in implementation. We used a Web-based assessment system capable of supporting retrieval practice through various question types and modes of delivery. HSS CLASS-Rheum can be deployed in 3 ways:

1. Individual, Asynchronous

2. Individual, Synchronous

3. Team-based with Peer Instruction (Box 1)

Box 1. Hospital for Special Surgery Critical Literature Assessment Skills Support–Rheumatology (HSS CLASS-Rheum) modes for use.

Individual, Asynchronous

• Trainees individually complete HSS CLASS-Rheum modules on their own time

• Provides trainees with the most flexibility in choosing when to complete modules

Individual, Synchronous

• Trainees individually complete HSS CLASS-Rheum modules during a set time, generally when they are scheduled to meet together

• Can be followed by group discussion to further support learning

• Provides trainees with dedicated time to complete modules

Team-Based, with Peer Instruction

• Trainees are grouped into teams and go through 2 rounds of questioning

• First they answer the module individually, without peer input

• Then trainees discuss the questions with their teammates and submit a team answer for each question

• Provides fellows with an opportunity to engage in Peer Instruction, associated with increases in problem-solving skills, conceptual knowledge of complex subjects, and performance on standardized assessments

We constructed introductory materials in print and video format (Video 1), explaining the purpose and use of HSS CLASS-Rheum.

The HSS CLASS-Rheum questions are grouped in 2 Question Sets, each composed of 10 thematic modules, with 3 to 9 American Board of Internal Medicine/National Board of Medical Examiners (ABIM/NBME)-style⁴⁷ multiple choice or numeric short-answer questions per module (Box 2). The 2 Question Sets mirror each other, being composed of different questions addressing the same learning objectives (“isomorphic questions”). Use of isomorphic questions enhances learning by providing another retrieval opportunity in a new context.

Box 2. HSS CLASS-Rheum modules.

• Data and Distributions

• Case Series

• Cohort Studies

• Case Control Studies

• Randomized Controlled Trials

• Crossover Studies

• Non-Inferiority Trials

• Survival Analysis

• Test Characteristics and Instrument Performance

• Systematic Reviews and Meta-Analyses

Each of the 2 Question Sets is organized into these 10 modules. The 2 Question Sets address the same learning objectives with isomorphic questions.

When using HSS CLASS-Rheum, each trainee is instructed to read the question, consider their answer, and enter a response. HSS CLASS-Rheum provides immediate feedback to the trainee on whether the submitted answer is correct, explanatory feedback through an answer rationale stating what the correct answer is and why, and information about why the distractors are incorrect. Immediate feedback has been shown to improve recall, and is especially important for multiple-choice questions that can otherwise lead to learning incorrect information.²³ This feedback can stimulate the trainee to reflect on their understanding, identify knowledge gaps or misunderstandings, and address them.

Application of Self-Determination Theory

Program directors are able to modulate external motivation for their trainees’ completion of HSS CLASS-Rheum questions; it can be deemed a training program requirement (which would more strongly drive completion of assigned modules) or an optional resource for interested trainees (which may then only be used by trainees who already have sufficiently internalized motivation for learning critical appraisal).

Program directors have access to their trainees’ performance in HSS CLASS-Rheum. This allows program directors to recognize their trainees’ learning needs and monitor them longitudinally to determine whether these needs have been effectively addressed. Program directors can potentially establish external motivation by requiring trainees to repeat HSS CLASS-Rheum modules or whole Question Sets.

To help trainees appreciate the relevance and application of the content and to promote motivation to engage in the learning, all HSS CLASS-Rheum questions are framed as cases within the scope of rheumatology trainees’ experiences. Questions highlight situations in which critical appraisal informs best care for patients with rheumatic disease.

Application of Communities of Practice

HSS CLASS-Rheum can be used in formats and settings that involve and foster collaborative learning and application of critical appraisal in practice, in effect introducing trainees to benefits of critical appraisal Communities of Practice, and potentially helping such Communities to emerge and develop. Use of HSS CLASS-Rheum with Peer Instruction and journal clubs are 2 ways to support development of critical appraisal Communities of Practice using HSS CLASS-Rheum.

Peer Instruction^33,48–51 is a teaching method shown to promote learner engagement, conceptual understanding, and problem-solving skills, particularly in science.^33,52 Peer Instruction offers a way to deliver instruction that is directly aligned with the benefits of retrieval practice, including repeated retrieval attempts, immediate and explanatory feedback, timing and spacing of retrieval, and variable retrieval activities. This method involves a sequence of 7 steps (Box 3), including both individual thought and sharing of knowledge and ideas.⁵² Empirical evidence has demonstrated that when Peer Instruction is used, learning outcomes improve, including gains in conceptual understanding, academic performance, and positive feelings toward content.⁵³ The HSS CLASS-Rheum Team format is designed to support Peer Instruction techniques.

Box 3. Steps in Peer Instruction.

1. Question posed (1 minute)

2. Students given time to think (1 minute)

3. Students record individual answers [optional^a]

4. Students convince their neighbors—peer instruction (1–2 minutes)

5. Students record revised answers [optional]

6. Feedback to teacher: tally of answers

7. Explanation of correct answer (2+ minutes)

^a Mazur has emphasized that step 3 promotes learning and therefore should not be skipped.

The process of Peer Instruction has been outlined by Eric Mazur as 7 steps that can optimize learning outcomes.

Data from Mazur E. Peer Instruction: A User’s Manual. Upper Saddle River: Prentice Hall, Inc.; 1997.

Aligning HSS CLASS-Rheum module content with common study designs facilitates their use in conjunction with a journal club. Trainees can read a relevant article from the rheumatic disease literature and complete the HSS CLASS-Rheum module for that study design, either in advance of a small group discussion or at the beginning of a journal club session. Pairing HSS CLASS-Rheum with an article can highlight important methodologies in the article and incorporate additional retrieval of critical appraisal knowledge. Retrieval is first attempted when each individual trainee initially critically appraises the article before discussion. Secondary retrieval attempts occur while answering the HSS CLASS-Rheum questions. A third set of retrieval attempts can be prompted in the journal club discussion. Some examples of questions that can prompt learning through retrieval practice in journal club discussions are provided in Box 4.

Box 4. Examples of questions to promote learning through retrieval practice in journal club discussions.

• “What is the study design?”

• “Do the participants in this study resemble patients in your clinic?”

• “In what ways could this have biased the study?”

• “How concerned are you that a confounding factor might have affected these results?”

• “What conclusions would you draw from this study?”

• “How might you apply the results of this study in patient care, or in directing future research?”

Targeted questions in journal club discussions can promote further learning through repeated retrieval practice. Example queries from the creators of HSS CLASS-Rheum can be particularly beneficial when used in conjunction with appropriate online modules.

Experience with HSS CLASS-Rheum

HSS CLASS-Rheum was piloted in 6 adult and pediatric rheumatology training programs between January and June 2016.⁵⁴ Trainees participating in this pilot demonstrated sustained gains of critical appraisal knowledge and skills after participating in sequential HSS CLASS-Rheum Question Sets. Trainees attained higher mean percentage of answers correct on the second HSS CLASS-Rheum Question Set compared with the first (P = .04), with 14% of participants’ scores increasing by more than 20 percentage points⁵⁴; 88% of participating trainees reported HSS CLASS-Rheum addressed a gap in their rheumatology training. Participants considered HSS CLASS-Rheum useful or very useful for learning epidemiology and biostatistics. Although other experiences may have also contributed to learning during the intervening months, these data are consistent with meaningful learning over this interval with effective learning transfer.

Program directors involved in the pilot of HSS CLASS-Rheum also reported that the modules filled an unmet need in their programs, helping them better understand trainees’ epidemiology and statistics knowledge, or lack thereof, and providing detailed explanations of pertinent material. In programs that already had courses covering critical appraisal material, program directors praised the modules as a useful complement to solidify knowledge. Some participants suggested future alignment of HSS CLASS-Rheum questions with a structured epidemiology curriculum in which additional content is delivered before completing each HSS CLASS-Rheum module. Publications referenced in the HSS CLASS-Rheum materials or Rheum4Science modules could be used to serve this purpose.

Trainees felt that by framing questions in a rheumatologic context, HSS CLASS-Rheum enhanced their sense of the learning experience’s relevance. Trainees also liked using HSS CLASS-Rheum with Peer Instruction in the Team format, and/or in association with journal club discussions. Program directors cited the benefits of setting aside time to meet as a group and work on the modules. They felt these targeted discussions were enjoyable and informative, allowing trainees to clarify and address areas of uncertainty. Participants expressed the hope that future trainees from smaller programs might be able to connect to HSS CLASS-Rheum online in virtual journal clubs coordinated between medical centers. Using HSS CLASS-Rheum in these ways (with Peer Instruction, journal club discussions, or small group discussions of the HSS CLASS-Rheum modules themselves) participants can be seen as working collaboratively in emergent Communities of Practice.

Overall, HSS CLASS-Rheum provides an example of the way in which educational frameworks and validated educational practices could be leveraged to improve rheumatology trainees’ grasp and future application of critical appraisal.

The Need for Critical Appraisal Education After Training

With enhancements to critical appraisal teaching in training, future rheumatologists may enter the workforce with greater knowledge and skills in this area, as well as attitudes that support critical appraisal in practice. However, the current workforce lacks skills needed for critical appraisal.^55–57 This is a major barrier toward teaching critical appraisal skills to the next generation. Local faculty may not be or feel able to teach critical appraisal, and this may telegraph a hidden curriculum to trainees and others in the medical community that critical appraisal skills are not necessary and/or that critical appraisal skills are too difficult to learn.

Applying Educational Theories to Critical Appraisal Education After Training

Retrieval Enhanced Learning theory, Self-Determination Theory (Motivation), and Communities of Practice are useful lenses for understanding and designing critical appraisal education during and after training. After training, retrieval practice remains a useful approach to promote encoding and processing of information and subsequent transfer of learning to different contexts. Factors affecting motivation for learning critical appraisal, however, can shift after formal training is complete, and access to Communities of Practice can be different for faculty than for trainees.

Motivating Factors Related to Learning Critical Appraisal After Training

To ensure uptake of critical appraisal skills, physicians must be motivated to learn and maintain them. Institutional requirements, Continuing Medical Education (CME), and/or Maintenance of Certification (MOC) are potential external motivators after training, but as acknowledged by the Accreditation Council for Continuing Medical Education (ACCME), “top-down mandates for education are not only typically ineffective, but create cynicism that erodes clinicians’ trust and engagement in their continuing medical education.”⁵⁸

Retrieval Practice Opportunities for Learning Critical Appraisal After Training

Recognizing that “professional development is most effective when the clinician is engaging in it for a purpose and when the material is meaningful and relevant to her or his scope of practice; is presented by a trusted authority; engages learners actively; and includes feedback, reflection, and reinforcement,”⁵⁹ there has been an effort to tie CME credit to relevant activities. Providing opportunities for self-assessment and retrieval practice, multiple-choice questions are provided in conjunction with some journal articles for MOC and CME credit. Rather than focusing exclusively on direct recall of conclusions stated in the journal article, asking some questions addressing critical appraisal skills and their application to the articles would expand the scope of the educational activity to help physicians raise their awareness of gaps in their critical appraisal knowledge and skills, while emphasizing relevance and bolstering learning.

Communities of Practice for Learning Critical Appraisal After Training

To foster internalization of motivation, Communities of Practice are potentially vital.^60–63 Communities of Practice can encourage adoption of values, and provide opportunities for collaboration. Convening journal clubs on a regular schedule is one way to “develop institutional culture and habits of evidence appraisal,”⁶⁴ which can be seen as an example of a Community of Practice. Providing food and time for socializing can potentially increase attendance in these forums.⁶⁴ Enhanced social or academic status associated with demonstration of appraisal, particularly high-quality appraisal, are cited as potential positive incentives to participation in appraisal activities.⁶⁴

Access to Communities of Practice can be an issue; creating specific infrastructure is important to facilitate bringing people together into these Communities. Not everyone has a local forum; online technologies can create virtual spaces to expand opportunities for rheumatologists to connect with colleagues to critically appraise the literature and in doing so, develop and reinforce critical appraisal skills in online Communities of Practice.⁶⁵ A good example of such a virtual Community of Practice is #RheumJC,⁶⁰ an innovative, monthly journal club hosted on Twitter, which has allowed international participation in structured discussions of relevant rheumatology research.⁶¹ Using a mailing list and tweets, cofounders announce the date and time of the next meeting, as well as the paper for discussion, which is often available through open-access for a short time before the meeting.⁶¹ Using their personal twitter accounts, interested individuals can join 1 of 2 live, 1-hour feeds hosted by #RheumJC cofounders, sometimes with invited authors,⁵⁵ or post to the feed anytime over a 24-hour open period. As of December 2017, 646 individuals from 36 countries had joined #RheumJC discussions and the #RheumJC Twitter account had more than 2700 followers.⁶³ These virtual journal clubs have received positive feedback, with more than 85% of respondents reporting that they were satisfied or very satisfied with the discussions, and 37% indicating the information discussed had influenced their clinical practice.^62,63 However, it must also be acknowledged that virtual spaces are less comfortable for some and thus are not the definitive solution for all.

We are optimistic about the potential for a variety of Communities of Practice to instill the value of critical appraisal and promote performance of critical appraisal. External prompts or requirements can provide important exposure. Teaching fundamental critical appraisal knowledge and skills in rheumatology training may help novices over the threshold⁴⁴ and into these Communities. Analogously, properly designed CME activities could help practicing rheumatologists join critical appraisal Communities of Practice.

Observing and connecting with others who are performing critical appraisal, developing a sense of personal self-efficacy for critical appraisal, and internalizing its value are all key for use of critical appraisal in practice.

Key Points.

• Equipping and motivating clinicians to critically appraise the literature supports optimal patient care.

• Conceptual frameworks of Retrieval Enhanced Learning, Self-Determination Theory, and Communities of Practice can inform design of educational approaches to promote critical appraisal in practice.

• HSS CLASS-Rheum® is a learning tool that can be used to help rheumatologists learn skills for critical appraisal through retrieval practice.

• Combining retrieval practice with opportunities for connection through Peer Instruction, journal clubs, and other forums can support engagement and internalization of motivation, promoting persistence with critical appraisal in practice.