The Badges Program: A Self-Directed Learning Guide for Residents for Conducting Research and a Successful Peer-Reviewed Publication

Abstract

Introduction

The Badges Program is a self-directed supplement to a program's research curriculum. This step-by-step resource helps medical residents to understand the resources needed to conduct their own research project and fulfill the Accreditation Council for Graduate Medical Education requirements for scholarly activity.

Methods

The curriculum is completed over varying amounts of time depending on the intricacy of scholastic activities. Simple case reports may take as little as 1–2 weeks whereas months to years may be required if residents are completing more intricate and elaborate projects. Associated materials include a guide organized by topic followed by sequentially completed task assignments demonstrating the user has gained a basic understanding of the corresponding objective. Upon completing a task, the user obtains its educational badge, a virtual certificate of completion the program can then track with a simple checklist for progress. Several supplementary online articles review core concepts or provide examples and web-based tutorials for becoming proficient with using reference software. No faculty training is necessary. Residents go through the steps necessary to perform research so as to gain familiarity with the process.

Results

Qualitative feedback obtained by informally surveying residents who have completed the Badges Program has been very positive, with residents reporting that the guide was easy to follow and the knowledge gained will help tremendously with future projects.

Discussion

We hope that with feedback, this resource will be expanded and refined so every resident will be able to use the Badges Program regardless of previous skills, experience, and publications.

Educational Objectives

After identifying his or her goals and a research mentor, a resident will be guided through each of the following:

• 1.Pass a quiz (100% correct) to demonstrate understanding of the different levels of evidence in research.
• 2.Create a document containing deadlines for eligibility to present at appropriate corresponding subspecialty conferences.
• 3.Complete an institutional review board training module.
• 4.Create a fact sheet summarizing a literature review of at least 20 articles.
• 5.Compose a reference list.
• 6.Complete the CONSORT 2010 checklist for a randomized controlled trial.
• 7.List the top five journals in a field or specialty using impact factors.
• 8.Propose five relevant outcome measures of an intervention.
• 9.Compose a standard operating procedure.
• 10.Complete the outline for the study design.
• 11.Define statistical terms and perform a statistical analysis if appropriate.
• 12.Create an oral PowerPoint or poster presentation.
• 13.Identify at least two organizations that fund research in the specific field of study.

Introduction

Across all resident specialties and subspecialties, the Accreditation Council for Graduate Medical Education (ACGME) requires that an accredited program curriculum “must advance residents’ knowledge of the basic principles of research, including how research is conducted, evaluated, explained to patients, and applied to patient care,” and recommends that “residents should participate in scholarly activity.”¹ At this stage in their careers, residents are required to have a basic understanding of general research knowledge; they have passed at least two of three medical licensing exams, which requires that they have the ability to interpret an experimental or clinical investigation abstract at a basic level, including knowledge of biostatistics and different types of study designs. As postdoctoral graduates, it is not uncommon for residents to have previously participated in clinical or laboratory research. However, most residents have never organized their own research project from beginning to end. One of the goals of this project was to demystify quality research projects and address some of the barriers that often exist when residents either think about conducting research or contemplate incorporating evidence-based medicine recommendations into their clinical practice, especially in those institutions that do not have the resources to provide extensive mentorship in research. For trainees interested in enhancing their research knowledge, this guide could be used to establish and nurture the mentor-mentee relationship and increase research literacy throughout the completion of the project.

The Badges Program was created to spark interest in the resident community in initiating, developing, and conducting quality research more independently. The idea was generated when Dr. Danielle Perret Karimi and Dr. Justin Hata brainstormed with Dr. David Reinkensmeyer about novel ways to engage and educate physical medicine and rehabilitation (PM&R) residents regarding the basics and benefits of conducting quality research. Dr. Reinkensmeyer had an existing curriculum for his graduate biomedical engineering research residents that was modified with the help of Dr. Kentaro Onishi to make it more applicable to PM&R and medical-surgery residents. These clinicians and researchers, along with other department leaders in basic science and clinical research, worked in collaboration to organize a final curriculum with educational objectives that contained the key elements of conducting quality research. This final product was then vetted by the Research and Education Committees of the Association of Academic Physiatrists.

In 2013, Abramson, Bostwick, Green, and DiPace created a research curriculum for residents, which included a set of interactive workshops and an independent project.² Their interactive workshops are very similar to our curriculum, guiding the residents through gaining an understanding of basic research principles. Abramson et al., however, established a didactic-based curriculum designed for several small-group sessions. The Badges Program, on the other hand, is a self-directed learning module for residents. The residents need to have a mentor for general guidance. However, the majority of the course is independent. In 2015, there were several faculty-oriented guides submitted to MedEdPORTAL. Fenton et al. published the Research Mentoring Faculty Toolkit, which focuses more on enhancing mentoring skills and resources of faculty and research directors.³ Similarly, the focus of the publications by Green et al.⁴ and Lebeau⁵ is on providing a resource for faculty development in mentoring either residents or medical students, respectively.

Methods

The materials associated with this resource include a guide (Appendix A) that is organized by objective. This guide has several educational objectives including identifying the knowledge and skills required for conducting and presenting research and answering some of the most common questions that residents encounter when undertaking research and publication. The user sequentially completes task assignments that demonstrate the user has gained a basic understanding of the corresponding educational objective. Upon completion of each task, the user obtains a figurative badge. The receipt of a badge represents completion of that stage and educational objective. Each learning objective corresponds to a different badge module. The Badges Program is designed to be finished over a variable amount of time depending on the intricacy of the scholastic activity. A simple case report may take as little as 1–2 weeks, whereas months to years may be required if the resident is completing badges for more elaborate projects. Either way, it is highly recommended that the resident work to identify an experienced mentor for supervision and actively nurture the mentor-mentee relationship. Successful completion of the badges in this guide should help to familiarize residents with the basic principles of research and scholarly activity.

The badges are listed in a recommended order for completion, but residents may change the order based on the needs of their project and/or mentor input. It is likely that resident may need to work on some badges simultaneously (see the suggested order of badges for various study designs at the end of the guide). Also, the minimum number of badges needed to complete the curriculum can be preset by the program.

Appendix B includes answers to the quiz given to earn Badge 3.

Several optional supplementary resources, many of them web-based files, augment explanations included in the guide:

• 1.The Craft of Research ⁶ is a book that provides a basic overview of how to conduct research, from the process of selecting a topic and gathering sources to the process of writing results. We use it as a resource for the resident to spark self-reflection regarding the purpose of their research.
• 2.Levels of evidence⁷ (Appendix C): The Centre for Evidence-Based Medicine offers a table for systematizing how to categorize levels of evidence for different question types. This resource is used for demonstrating in detail the variety of levels of evidence possible when reviewing or designing a project. To demonstrate understanding of the levels of evidence, the resident is asked to pass a quiz located at the end of the curriculum. The quiz answers are given in Appendix B.
• 3.Online HIPAA research tutorial: The HIPPA web-based research tutorial⁸ reviews core concepts and requirements for accessing, collecting, creating and/or providing Protected (Personal) Health Information in the conduct of research involving human research participants.
• 4.EndNote,⁹ RefWorks,¹⁰ and Mendeley¹¹ online tutorials are three free or inexpensive web-based tutorials in reference software. A resident may optionally choose one to explore or become familiar with all three as they are slightly different. Two recommended EndNote videos on the linked site in the footnote are “EndNote Overview” and “How to Use EndNote in 7 Minutes.” For the RefWorks online tutorial, go to the website in the footnote and click on Click Here for Tutorial Basics to start the modules. Mendeley is free to download and use, and its tutorials can be found at the link in the footnote.
• 5.CONSORT Guidelines are a set of principles that describe proper design/conduct for a randomized controlled trial (RCT); the guidelines are used here for understanding how to analyze an RCT. The resident is instructed to choose an RCT and analyze it by completing the CONSORT 2010 Guidelines checklist.¹² The checklist reference is an interactive website; also given is a link to the peer-reviewed paper,¹³ published in the Annals of Internal Medicine, that was the basis for the website.
• 6.Patient Reported Outcomes Measurement System (PROMS):¹⁴ This site provides information on measurements validated and recommended by NIH. It is mentioned here to demonstrate the variety of validated outcome measures that can be used when choosing the right measures for a resident's own study. Once on the site, you are required to register (upper right-hand corner) as a new user before being granted access to outcome measures or instruments. To access the list of instruments (after logging in with your login and password), you will need to create a study. You can name it whatever you want since your primary interest is to see if the site has instruments you can include in the study as outcome measures. Once you have created your proposed study, click the tab entitled Instruments on the top of the screen. This will pull up all validated instruments, and then, you can click on the Add tab on the right side to start browsing the available instruments. For ease of use, we recommend sorting the instruments by making a selection, such as “Pain-Pain Intensity,” “Physical Function-Upper Extremity,” “Emotional Distress-Depression,” and so on, under the Domain tab on the page.

Results

Each resident at the University of California, Irvine (UCI), PM&R program is required to complete all of the badges to finish this curriculum. The Badges Program is reviewed annually at the incoming resident orientation and then distributed to the residents by paper copy and email.

Especially for those residents with less extensive backgrounds in research, this resource was irreplaceable as it allowed the resident to be more independent in the steps throughout completion, including the poster presentation. It is still important and necessary that the residents have a mentor onboard for giving specific advice and helping deal with obstacles. The Badges Program has the potential to enhance the user's knowledge and skill base when beginning clinical research or starting to incorporate evidence-based medicine into his or her own clinical practice.⁷

Ten residents at UCI have fully completed the Badges Program, and five are currently working to complete the project. Qualitative feedback from the residents at UCI was obtained by surveying those who have completed the curriculum. The University of Alabama at Birmingham (UAB) also implemented the Badges Program to introduce a structured model to help build research skills and knowledge in their residents. UAB administrators used a survey to collect data assessing the residents’ perceived usefulness of the Badges Program and the practicality of completing the badges.¹⁵ Twelve residents completed the program, and feedback was obtained from 10 of the 12, consisting of three third- and fourth-year residents each and four second-year residents. When assessing the need for a research curriculum, 50% of the residents reported they had never received formal research training, and 70% stated there was no current curriculum available. Additionally, 90% of the residents reported that they had sufficient time to pursue research.

The UCI residents were required to complete all of the badges during the 7-week research block; however, most residents did not start a new project for the Badges Program. For these residents, the curriculum was applied to an existing ongoing project, retrospectively applying the principles to complete the course. We found that the biggest barrier was time required for institutional review board (IRB) approval. The research curriculum at UCI was actually revised this year from a 7-week research-intensive rotation to several weeks spread throughout the years to accommodate for this barrier.

We do not have data from UAB as far as how many residents were able to complete and submit for publication. However, most residents at UCI were able to complete at least a case report or develop the outline and methods of a new project in the time provided. Three residents continue to work on their projects, pursuing publication after the completion of their research rotation. We found that the information guide was generally sufficient; however, as specific questions arise, it may be necessary to review outside sources or websites for additional information.

Surveys at each program asked the residents how useful the curriculum was and to describe barriers to completing the curriculum. The results from the two programs were very positive; residents reported that the guide was well organized, logical, and easy to follow. The guide incorporates tasks required for completion of the project into the tasks that are required to receive various badges: “It isn't busy work because we are completing the research project throughout the curriculum.” Several residents commented that this guide was a good starting reference and that they “learned many tools that will help with future projects” as well. The residents at UCI believe that the module for RCT analysis was particularly helpful when critiquing and preparing for various journal club presentations.

A common theme in the reported barriers of the curriculum regarded the time line of events and overall time management. Residents at UAB found it difficult to complete the curriculum while keeping up with their clinical duties. The residents at UCI reported barriers in completing the project in the 7 weeks assigned, often needing more time.

Discussion

The PM&R residents at UCI have implemented the Badges Program in their curriculum for 4 years (at the time of this publication). One of the objectives during the 7-week research rotation is to complete the Badges Program, ideally with a project of the resident's own. Completing the entire Badges Program more than fulfills the ACGME scholarly activity requirement, so it is important from the beginning of the project for the resident and his or her mentor to develop expectations and/or goals for the completed project. More specifically, it is essential for the resident to identify his or her specific goal(s): a review article, a case report, working to complete an established project, or starting his or her own level 1 (RCT) or level 2 (cohort) study. Completing the Badges Program, however, does not necessarily mean that the ACGME/residency research requirements have been fulfilled. We recommend using this curriculum to guide residents through project completion; however, its purpose is for residents to go through the steps necessary to perform research so as to at least have familiarity with the process.

Regarding time management, the residents at UCI have a 7-week research rotation to complete the project. However, they can choose to continue beyond this block of time if they have selected a project that requires more time. When the resident chooses to participate in an existing project, we have found that applying ahead of time to obtain IRB approvals works best so that the resident can have access to records and focus most of his or her time on data collection or analysis.

Depending on the goals of the resident, the specific program requirements, and rotation or block scheduling logistics, the Badges Program will need to be adapted accordingly. Based on the feedback from the residents at UAB, developing structured time lines with expectation deadlines would be beneficial for each program.

The Badges Program was initially developed for PM&R residents. Because of the success it has had in the PM&R program at UCI, it has been broadened to be generalized and applicable across all resident specialties. We have found that some residents enter the program with a significant background in research and thus find this resource rudimentary. At UCI, an exemption is made if a resident has already published a first-authored peer-reviewed article. Such residents are excused from the requirement of completing the Badges Program. However, they are still required to complete the standard program requirements for research. Our hope is that with feedback, this resource will be expanded and refined so that every resident will be able to use the Badges Program regardless of previous skills, experience, and publications.

Furthermore, although it is ultimately up to the resident to research and find an appropriate mentor, it may be useful to incorporate a list of faculty willing to participate, their respective research interests, and any ongoing projects. This may spark interest with the residents or serve to save time if the resident has a project in mind. Some of the feedback received from surveying the participating residents demonstrated the need for mentorship in a wide variety of fields.

Appendices

A. The Badges Program Curriculum.pdf

B. Levels of Evidence (Badge 3) Quiz Answers.pdf

C. Oxford 2011 Levels of Evidence.pdf

All appendices are peer reviewed as integral parts of the Original Publication.