INTRODUCTION
Medical education is experiencing rapid growth with an increasing number of publications and journals dedicated to education research.1,2 Several new journals and special education issues (including the CDEM/CORD supplement by the Western Journal of Emergency Medicine) have arisen in recent years to address this increasing interest. As clinician educators, it is important to produce and disseminate research both for promotion and development of a subject niche, as well as to disseminate findings for others to learn from novel and successful educational interventions.
However, the quality of existing medical education research has been variable.3,4 Studies have suggested this may be due to limited mentorship,5 as well as challenges including available time, funding, small sample sizes, ability to navigate the institutional review board process, and difficulty with defining relevant and measurable outcomes.6,7 This article discusses five common challenges to education scholarship and provides suggestions for overcoming them.
COMMON BARRIERS
1. Lack of Clarity in the Research Question
The first challenge is developing the research question. While this may seem like a relatively straightforward task, developing a clear and important research question evolves from an iterative process. This process generally begins with an educator’s interest in a topic and a broad research question. For example, consider the case of a program director who is interested in identifying factors related to resident burnout. This is a broad topic because the “factors” are not clearly defined nor is the hypothesis clarified. Nonetheless, this is enough information to conduct a literature review to begin to understand prior work in the area, identify where there is a gap in knowledge, identify the novel research question, and to provide a link between the research idea and a conceptual framework.
The conceptual framework is a vital component to developing a good research question, yet it is often overlooked in medical education studies. A review of published medical education studies found that 55% did not mention a conceptual framework.4 The conceptual framework serves as the foundation of the study that informs all aspects of the research design and should not be ignored. Frameworks relevant to medical education may be found in fields outside of medicine, especially education and psychology; so medical education scholars may want to extend their literature search outside of their medical specialty. In the example above, a thorough search of the literature would reveal that burnout studies are often framed within the context of the Multidimensional Theory of Burnout, a theory found primarily within the psychology literature.8 Further discussion of conceptual frameworks and how they can be used to develop medical education research projects can be found in the classic paper by Bordage.9
After conducting a thorough literature review and identifying a specific problem to address, medical educational researchers can use two mnemonic tools to further develop their research question. The first is the PICOT framework, which is used to transform a broad question into a specific one that includes all of the necessary components: Population, Intervention, Comparison, Outcomes, and Time frame. One study demonstrated that research reports that used the PICOT format were rated as having better overall quality than those that did not.10 Figure 1 includes an example of a structured question using the PICOT format (Figure 1).
Figure 1.
Example of a research question in PICOT formatting.
PICOT, population, intervention, comparison, outcomes, time frame.
The other valuable tool for designing a research question is the FINER criteria (Figure 2). As opposed to the PICOT framework, which helps to determine that all elements of a study question are present, the FINER criteria assess the quality and likelihood of success of a given research study. The authors recommend applying both sets of criteria to a given study question to ensure that the question is adequately refined, so as to maximize the success of each academic endeavor.
Figure 2.
FINER criteria for assessing study questions.
FINER, feasible, interesting, novel, ethical, relevant; IRB, institutional review board.
2. Inadequate Methodology to Assess the Study Question
Editors and reviewers desire to publish interpretations that are generalizable and accurate (i.e., supported by validity evidence). Despite recommendations that medical education research adhere to the same rigorous methodological standards as clinical research, medical education research often fails in this regard.11 One study of submissions to a major medical education journal found that the top reasons for rejection included inappropriate statistics, over-interpretation of the results, an inadequate research instrument, an insufficient problem statement, inadequate literature review and an insufficient data, while a sound problem statement and study design significantly increased the likelihood of publication.12
Moreover, medical education research also fails to report substantial validity evidence, which presents a challenge to publication.13,14 Many of these threats can be minimized by choosing an appropriate study design, standardizing study conditions, and collecting and reporting detailed information about study participants and procedures.
While the measurement instruments in clinical research are typically well validated (e.g., d-dimer, troponin), education research instruments are rarely as fortunate. Therefore, not only do education researchers need to define and collect meaningful outcomes for research, they also need to ensure the validity and reliability of their measurements. Many education studies focus on novel curricula, innovations, learner behaviors, or the exploration of education concepts or environments, for which previously established instruments are unavailable. If a new instrument needs to be created, or if using an instrument from another field (e.g., psychology, sociology, secondary education), the researcher is advised to first assess the validity of the instrument with respect to the intended measurement. In order to gather enough validity evidence to support the instrument, it is essential that the instrument be matched to the goals and objectives, piloted to ensure that it performs as expected, and compared to other similar measurements or available data.15 The mere act of gathering validity evidence for an instrument or measurement (i.e., showing that it measures what it states it is going to measure and that it accurately distinguishes the target outcome from other outcomes) can be an important research study.
Novice education researchers faced with multiple competing demands may attempt to capitalize upon existing work by converting an ongoing education project into a research study. For example, an educator may develop a new curriculum and then subsequently decide to assess it after it has been ongoing for several months. Unfortunately, these research attempts are often unsuccessful due to insufficient planning and inadequate methodology and outcomes. To have a methodologically sound and successful study it is vital to define appropriate outcome measures at the onset and select an appropriate study design that best allows the researcher to measure the desired outcomes with minimal threats to validity. The researcher should collect validity evidence to create the assessment instrument during the developmental phase to ensure that the instrument is appropriate for the study. Involving a statistician or experienced education researcher early in the process is extremely beneficial to help avoid fatal flaws and wasted effort.
3. Losing Momentum
While manuscript publication should be one of the ultimate goals, it is important to set stepwise, attainable, intermediate milestones and celebrate their accomplishment on the route to manuscript completion and publication. Examples of early milestones in the scholarly process include abstract submission, initial paper development, and local and national presentations. There are also several digital mediums to publish medical education innovations. One example is the Academic Life in Emergency Medicine IDEA (Ideas in Didactics and Educational Activities) series (https://www.aliem.com/category/non-clinical/idea-series/), which allows authors to showcase novel education interventions to the broader medical education community. Additionally, educators may publish curricula online in MedEdPORTAL or the Journal of Education & Teaching – Emergency Medicine. Moreover, many institutions host “work in progress” sessions to highlight and discuss ongoing studies.
Ensuring early and attainable wins helps to maintain momentum for projects.16 To be successful, researchers must actively plan and establish short-term goals and recognize the accomplishment of these goals and the specific team member contributions throughout the process.16 Additionally, without a clear outline, unified writing plan, and identification of clear short-term goals, efforts can easily lose all momentum and dissolve into a disorganized, inactive to-do list without an end product. It is valuable to have regularly scheduled meetings or conference calls to ensure that all members are on track, especially for multi-institutional projects.
Education researchers should also consider maximizing the return on a given project by considering additional opportunities for expansion of a given project.17 Often, it requires little additional effort to convert one project into several deliverables, such as an abstract, manuscript, presentation, and digital description of the innovation. Tables 1 and 2 provide a list of potential arenas for publication, as well as examples of different formats. For example, the start of a project might include the publication of a review article or perspective on that topic. This could also be converted into a didactic for residents or training session for faculty development. As the work progresses, you might consider a reflection or short thought piece. This approach can also help maintain momentum by assisting with the early wins described above. We would like to emphasize that researchers must be conscious to avoid self-plagiarizing or artificially separating out study components to create multiple publications from a study addressing a single concept (i.e., “salami slicing”).18
Table 1.
Publication venues for medical education scholarship.
|
Table 2.
Outlets for digital dissemination.
|
4. Lack of Follow Through
Once the study is completed, it is important to go beyond the abstract with the goal of publishing it in a peer-reviewed journal. Historically, only 25–50% of abstracts presented at emergency medicine (EM) and medical education conferences are subsequently published as manuscripts.19–24 Peer-reviewed publication is important because it increases dissemination of information and is a significant consideration in achieving promotion, tenure, and future grant funding.25,26 This is particularly important for medical education research because of the relatively smaller proportion of outcomes-based studies in this field compared with clinical research.3,4
To make this process easier, the authors recommend that the researcher begin manuscript preparation at the start of protocol development, filling in components as the project progresses. Often, the introduction, research hypothesis, and methods can be drafted before the study begins, as part of the institutional review board proposal. This early planning will make the remainder of the paper more manageable when the study is completed.
When working as a team there may be more accountability to complete the paper; but team authorship can also create conflicts in author order. We therefore recommend discussing criteria for authorship, and drafting a potential order-of-authorship list, prior to beginning a study. Authors should also be aware that most publications are not accepted on the first submission, and often they may require submission to multiple journals.27 Authors should not let a paper linger after the first rejection. Rather, they should read the review, make appropriate edits, and quickly re-submit to another journal. Authors should also be aware that different medical education journals have different foci, and publication will be more successful if they select journals that publish similar topics or types of articles in line with their particular manuscript.
It is not unusual for authors to hit a roadblock during the writing process. This can occur at any point throughout the process from beginning to end. It can be helpful to set specific goals prior to initiating the writing process. Each goal should have a specific deadline, which can help maintain momentum and accountability. Education researchers often have many competing demands; scheduling specific times on one’s calendar for writing, similar to other appointments, can ensure dedicated time away from distractions for the author to concentrate on writing the manuscript.28,29 In order to focus on the manuscript itself, authors should avoid checking emails and other distractions.30 It may also be valuable to include small breaks when the author feels his or her attention waning.
Another tip is to start small. Rather than attempt to draft everything at one time, which may seem overwhelming, authors should start with either the first paragraph or the methods section (which are typically easier to write) and then expand from there. Use the standard framework for the manuscript, incorporating journal-specific components as needed. Since most articles follow a general layout, it is much easier to fill in the paper piece-by-piece using the layout as a recipe than attempting to create one’s own format.30 Write the first draft spontaneously and uncritically allowing for editing after the draft is written.30 Attempting to edit while writing can interrupt concentration and flow. Finally, authors should have other people review their work whenever possible. This provides an external perspective and helps identify errors and confusing concepts that the investigators may have overlooked. It may be valuable to include non-physician researchers (e.g., PhD, EdD, PsychD) who can provide a highly valuable external interpretation, as many reviewers for medical education journals may not be physicians.
5. Lack of Expertise
For faculty who lack local experts with an education research background, getting started can be difficult. In these situations, it may be helpful to reach out to the clinical researchers within or outside of the department. Likewise, many academic systems have medical education researchers in other departments or in the school of medicine who may provide guidance. Another option is to join a project team from another institution. EM research is improved by multi-site collaboration, and working with a broader group may help develop skills.
There are formal options for research skill development. These include advanced degrees such as a Masters of Health Professions Education, institutional faculty development programs in education research, or the MERC (Medical Education Research Certificate) through the Association of American Medical Colleges. Often, the EM national meetings host workshops on research methodology. In addition, finding a virtual mentor in EM or another specialty might provide support for the educational scholarship one hopes to accomplish.
FUTURE DIRECTIONS
As medical education-focused researchers, we urge our specialty to consider future directions for creation and dissemination of our work. Just as we actively advocate for increased production, training and funding of EM clinical research, we must do the same for medical education research. First, we must start by growing the body of rigorously-conducted medical education trials published in high-quality journals. Second, we must take on the critical task of growing and promoting junior faculty who can expand our methodologic and content expertise. This step involves developing and promoting high-quality fellowships, which must include specific research training. We must encourage our mentees and colleagues to use state-of-the-art methods. This step may also consist of honest inquiry into continued barriers to methodologically-sound research studies.
Finally, to truly change the trajectory of medical education we must pursue outside funding. The sources of funding for medical education research are currently limited. As a specialty, we actively encourage government and foundation funding sources to dedicate resources to EM-relevant clinical questions. Similarly, building upon our existing contributions to medical education within and outside of EM, we must push for external groups to fund high-quality, multi-center studies of innovative educational methods. This direction should include the following: partnerships with researchers who are not primarily education focused; training of our colleagues in ways to bridge the funding divide; and possibly creation of novel funding sources (such as the Society for Academic Emergency Medicine’s medical education research grants).
CONCLUSION
Medical education research continues to grow within EM and it is imperative that educators produce and disseminate high-quality publications to continue to advance this field. This article discusses several challenges and strategies for overcoming barriers to publication, in order to assist the educator with producing quality education research. It is the hope of the authors that this will encourage educators to publish more research to disseminate findings with the ultimate goal of further improving education and patient care.
Footnotes
Section Editor: Ed Ullman, MD
Full text available through open access at http://escholarship.org/uc/uciem_westjem
Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.
REFERENCES
- 1.Norman G. Research in medical education: three decades of progress. BMJ. 2002;324(7353):1560–2. doi: 10.1136/bmj.324.7353.1560. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Branch WT, Kern DE. An emerging renaissance in medical education. J Gen Intern Med. 2004;19(5 Pt 2):606–9. doi: 10.1111/j.1525-1497.2004.45001.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Derossis AM, DaRosa DA, Dutta S, et al. A ten-year analysis of surgical education research. Am J Surg. 2000;180(1):58–61. doi: 10.1016/s0002-9610(00)00420-7. [DOI] [PubMed] [Google Scholar]
- 4.Cook DA, Beckman TJ, Bordage G. Quality of reporting of experimental studies in medical education: a systematic review. Med Educ. 2007;41(8):737–45. doi: 10.1111/j.1365-2923.2007.02777.x. [DOI] [PubMed] [Google Scholar]
- 5.Farley H, Casaletto J, Ankel F, et al. An assessment of the faculty development needs of junior clinical faculty in emergency medicine. Acad Emerg Med. 2008;15(7):664–8. doi: 10.1111/j.1553-2712.2008.00152.x. [DOI] [PubMed] [Google Scholar]
- 6.Albert M, Hodges B, Regehr G. Research in medical education: balancing service and science. Adv Health Sci Educ Theory Pract. 2007;12(1):103–15. doi: 10.1007/s10459-006-9026-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Dyrbye LN, Thomas MR, Papp KK, et al. Clinician educators’ experiences with institutional review boards: results of a national survey. Acad Med. 2008;83(6):590–5. doi: 10.1097/ACM.0b013e318172347a. [DOI] [PubMed] [Google Scholar]
- 8.Maslach C. A Multidimensional Theory of Burnout. In: Cooper CL, editor. Theories of Organizational Stress. 1st ed. New York: Oxford University Press; pp. 68–85. [Google Scholar]
- 9.Bordage G. Conceptual frameworks to illuminate and magnify. Med Educ. 2009;43(4):312–9. doi: 10.1111/j.1365-2923.2009.03295.x. [DOI] [PubMed] [Google Scholar]
- 10.Rios LP, Ye C, Thabane L. Association between framing of the research question using the PICOT format and reporting quality of randomized controlled trials. BMC Med Res Methodol. 2010;10:11. doi: 10.1186/1471-2288-10-11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Yarris LM, Deiorio NM. Education research: a primer for educators in emergency medicine. Acad Emerg Med. 2011;18(Suppl 2):S27–35. doi: 10.1111/j.1553-2712.2011.01189.x. [DOI] [PubMed] [Google Scholar]
- 12.Bordage G. Reasons reviewers reject and accept manuscripts: the strengths and weaknesses in medical education reports. Acad Med. 2001;76(9):889–96. doi: 10.1097/00001888-200109000-00010. [DOI] [PubMed] [Google Scholar]
- 13.Reed DA, Beckman TJ, Wright SM, et al. Predictive Validity evidence for medical education research study quality instrument scores: quality of submissions to JGIM’s Medical Education Special Issue. J Gen Intern Med. 2008;23(7):903–7. doi: 10.1007/s11606-008-0664-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Torre DM, Picho K. Threats to internal and external validity in health professions education research. Acad Med. 2016;91(12):e21. doi: 10.1097/ACM.0000000000001446. [DOI] [PubMed] [Google Scholar]
- 15.Messick S. Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. Am Psychol. 1995;50:741–9. [Google Scholar]
- 16.Kotter JP. Leading Change: Why transformation efforts fail. Harvard Business Review. 2007;85(1):2–12. [Google Scholar]
- 17.Gottlieb M, Chan TM, Sherbino J, et al. Multiple wins: embracing technology to increase efficiency and maximize efforts. AEM Education and Training. 2017 May 8; doi: 10.1002/aet2.10029. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Norman G. Data dredging, salami-slicing, and other successful strategies to ensure rejection – twelve tips on how to not get your paper published. Adv Health Sci Educ Theory Pract. 2014;19(1):1–5. doi: 10.1007/s10459-014-9494-8. [DOI] [PubMed] [Google Scholar]
- 19.Li SF, Umemoto T, Crosley P, et al. SAEM Abstracts to Articles: 1997 and 1999–2001. Acad Emerg Med. 2004;11(9):985–987. doi: 10.1197/j.aem.2004.04.006. [DOI] [PubMed] [Google Scholar]
- 20.Macmillan CD, Moore AK, Cook RJ, et al. Abstract-to-publication ratio for papers presented at scientific meetings: a quality marker for UK emergency medicine research. Emerg Med J. 2007;24(6):425–6. doi: 10.1136/emj.2007.046078. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Kalkan A, Kose O, Bilir O, et al. Publication rate of abstracts presented at the emergency medicine congresses held by the European Society for Emergency Medicine (EUSEM) in 2011 and 2012. Emerg Med J. 2015;32(9):728–32. doi: 10.1136/emermed-2014-204063. [DOI] [PubMed] [Google Scholar]
- 22.Papp KK, Baker EA, Dyrbye LN, et al. Analysis and publication rates of Clerkship Directors in Internal Medicine (CDIM) annual meeting abstracts 1995–2005. Teach Learn Med. 2011;23(4):342–6. doi: 10.1080/10401334.2011.611771. [DOI] [PubMed] [Google Scholar]
- 23.Walsh CM, Fung M, Ginsburg S. Publication of results of abstracts presented at medical education conferences. JAMA. 2013;310(21):2307–9. doi: 10.1001/jama.2013.281671. [DOI] [PubMed] [Google Scholar]
- 24.Sawatsky AP, Beckman TJ, Edakkanambeth Varayil J, et al. Association between study quality and publication rates of medical education abstracts presented at the Society of General Internal Medicine Annual Meeting. J Gen Intern Med. 2015;30(8):1172–7. doi: 10.1007/s11606-015-3269-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Atasoylu AA, Wright SM, Beasley BW, et al. Promotion criteria for clinician-educators. J Gen Intern Med. 2003;18(9):711–716. doi: 10.1046/j.1525-1497.2003.10425.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Proctor EK, Powell BJ, Baumann AA, et al. Writing implementation research grant proposals: ten key ingredients. Implement Sci. 2012;7:96. doi: 10.1186/1748-5908-7-96. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Garg N, Gottlieb M. Resident research tips: surviving and thriving within the peer review process. Ann Emerg Med. 2017 doi: 10.1016/j.annemergmed.2017.09.015. pii: S0196-0644(17)31664-5. [DOI] [PubMed] [Google Scholar]
- 28.Yarris LM, Juve AM, Artino AR, Jr, et al. Expertise, Time, Money, Mentoring, and Reward: Systematic Barriers That Limit Education Researcher Productivity – Proceedings from the AAMC GEA Workshop. J Grad Med Educ. 2014;6(3):430–6. doi: 10.4300/JGME-D-14-00340.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Jordan J, Coates WC, Clarke S, et al. Exploring scholarship and the emergency medicine educator: a workforce study. West J Emerg Med. 2017;18(1):163–8. doi: 10.5811/westjem.2016.10.32636. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Coverdale JH, Roberts LW, Balon R, et al. Writing for academia: getting your research into print: AMEE Guide No. 74. Med Teach. 2013;35(2):e926–34. doi: 10.3109/0142159X.2012.742494. [DOI] [PubMed] [Google Scholar]