Analysing the literature: A research primer for low- and middle-income countries

Abstract

Effective critical appraisal of medical research requires training and practice. Evidence-based medicine provides a framework for standardised review of manuscripts of nearly any research design. Online resources and communities exist to provide free access to electronic search engines and critical appraisal of emergency medicine and non-emergency medicine research. An emerging array of Free Online Open Access medical education (FOAMed) resources also provide opportunities to observe Evidence-based medicine critical appraisal in written or audio format and to actively participate as a learner. This chapter will highlight accessible resources that provide both methodological background and virtual mentoring for readers to develop EBM skills.

African relevance

• •This paper provides a background of evidence-based medicine with worldwide relevance.
• •It highlights access to free evidence-based medicine search engines.
• •It also highlights access to free evidence-based medicine critical appraisal blogs and podcasts.

The International Federation for Emergency Medicine global health research primer

This paper forms part 2 of a series of how to papers, commissioned by the International Federation for Emergency Medicine. It describes the process that underpins evidence-based medicine (EBM) as well as how to search, identify and analyse relevant literature to a research question. We have also included additional tips and pitfalls that are relevant to emergency medicine researchers.

Background

Research constitutes the foundation upon which the science of medicine rests. Serendipity and astute observation of patterns merging into cause-effect relationships undoubtedly catalysed historical medical advances in a handful of cases. However, the scientific method of deriving educated hypotheses followed by empirical experimental designs to probe the actuality of these ideas has yielded substantially more discovery and innovation. The quality of published medical research is highly variable and ranges from the extremely useful (and practice changing) to the extremely poor-quality study that, at worst, can lead to harmful clinical practice. Critical appraisal, defined as “the process of carefully and systematically examining research to judge its trustworthiness, its value and relevance in a particular context”, is a skill every physician requires for lifelong learning [1]. Indeed, “a 21^st Century clinician who cannot critically read a study is as unprepared as one who cannot take a blood pressure or examine the cardiovascular system.” [2] Emergency medicine research is still relatively underdeveloped due to the novelty of the specialty, but more importantly because of the relative lack of academic emergency physicians. Therefore, understanding efficient and effective methods to find and critically appraise medical research is essential for every (emergency) clinician worldwide. However, research is only one pillar of Evidence Based Medicine (EBM). Clinical expertise and patient priorities are equally important, as is interpreting published research within the context of one's local healthcare system context (Fig. 1).

Fig. 1.

The evidence based medicine triad.

Keeping up with the latest advances in diagnosis and treatment is a challenge we all face. Information is needed that is valid and relevant to our patients and practice setting and a great deal of research is either poorly done or irrelevant to our setting [3]. Furthermore, poor quality or inconclusive evidence often permeates into textbooks, guidelines, or clinical policies and then becomes established knowledge or clinical practice that becomes challenging to reverse or expunge [4]. Separating which articles are important to your clinical practice takes skills that many of us were either never taught or taught in limited capacity, yet they are vital to improving the care we provide for our patients.

Gordon Guyatt introduced the term Evidence Based Medicine (EBM) in 1992. Following the early 20th Century Flexner report in North America, medical education transformed into a highly structured format with a vision of a scientific core rather than authoritarian dictate. Indeed, the pace of research accelerated throughout the 20th Century as funding increased with a concomitant surge in the number of manuscripts and journals. The discipline of emergency medicine was a comparative latecomer to the research table given its development in the 1960 and 70's, but emergency medicine has consistently been a strong advocate of what is now known as EBM because much published research, which may be useful in other settings, is often not applicable to the unique environment of the emergency department.

EBM provided a template for clinicians to seek, find, appraise, and apply research at the bedside. The How To prototype for clinicians to incorporate EBM was the Journal of the American Medical Association's User's Guide to the Medical Literature, a series of peer-reviewed articles providing prototypical questions to analyse the validity, reproducibility, and real-world implementation of manuscript data into bedside practice [5]. For example, meaningful critical appraisal of prognostic research differs from that of interventional studies and the Users' Guide provides detailed methods to guide these efforts.

Specific training, mentoring and practice in EBM are essential. Refining critical appraisal aptitude is similar to technical proficiencies like inserting a central line or intubating a patient, which often require dozens of attempts to master under the guidance of a knowledgeable expert [6]. Unfortunately, developing EBM skills are often underemphasized in residency/specialist training though clear competencies have been delineated [7,8]. Developing these skills in areas of the world with little access to specialized training or mentoring in EBM can be a challenge. Online learning in low-middle income countries has shown to be an effective way to learn EBM [9] and strategies for developing online Journal Clubs have been described [10]. However, EBM principles for analysing medical literature do not differ around the world. Regardless of where you practice medicine, dedicating time to developing these skills is imperative [11]. This chapter will highlight accessible resources that provide both methodological background and virtual mentoring for readers to develop EBM skills.

The paradigm, process, and components of evidence based medicine

Analysing research through the prism of EBM mandates that clinicians understand two truths:

• 1.Evidence alone is never enough.
• 2.Not all evidence is equally valuable.

The first statement emphasizes the essential overlay of clinician expertise and patient values when applying research to individual patients at the bedside. Physicians, nurses and patient caregivers spend thousands of hours encountering unique presentations of disease, while learning to recognize patterns and subtypes that guide medical decision-making [12]. Patients around the world seek emergency care for different reasons with varying priorities. Applying high-quality research without exploring patient's values is not EBM and can yield harmful results even if the scientific rationale is rigorous [13]. Incorporation of patient's unique values is essential since a common criticism of EBM is “the individuality of patients tends to be de-valued, the focus of clinical practice is subtly shifted away from the care of individuals towards the care of populations, and the complex nature of clinical judgment is not fully appreciated” [14]. This EBM blind spot is precisely the rationale for the Society of Academic Emergency Medicine to host a Consensus Conference on Shared Decision Making in 2016. The peer-reviewed contents of that Consensus Conference outlining the science, funding opportunities, and research priorities are freely available at https://onlinelibrary.wiley.com/toc/15532712/2016/23/12.

The second statement alludes to a hierarchy of research evidence with systematic reviews of randomized controlled trials at the top, representing the least biased form of evidence, and unsystematic observations like case reports at the bottom (Fig. 2). In general, this evidence hierarchy implies that higher quality evidence is most likely to find the true effect size, whether the question is diagnostic, prognostic, or therapeutic. Systematic reviews are not a panacea for developing and maintaining critical appraisal skills because poorly conducted systematic reviews asking the wrong questions on the wrong populations with less important outcomes are increasingly prevalent [15].

Fig. 2.

The evidence based medicine hierarchy of evidence.

However, the reality is that for many important questions, only lower quality evidence currently exists to inform practice in many settings, and this can be a rich source of future research questions to be answered by clinical researchers in all types of emergency settings. Despite high quality evidence being ideal for research, for those interested in research in resource limited settings, this is not feasible. Answering questions with what is considered lower quality evidence, is still an important piece of building and advancing emergency medicine research. As an example, imagine that you are confronted with this scenario during an ED shift: A 14-year old high school student suffers a laceration to her hand while operating a saw at school. She has a 3 cm laceration over the thenar eminence with no obvious tendon or arterial injury. After reassuring the patient and her concerned parents, you begin to contemplate your wound repair and a colleague suggests a simple interrupted suture repair after copious irrigation with tap water. You wonder whether tap water is the appropriate irrigation fluid when you could also use sterile saline. Fig. 3 provides step-by-step examples of how to accomplish the EBM process outlined below.

Fig. 3.

An example of the evidence based medicine process.

Evidence based medicine step 1: Derive an answerable question using the PICO format: [16]

• P = patient population.

• I = intervention (therapy, diagnostic test, prognostic factor).

• C = comparator (intervention, test).

• O = outcomes of interest.

In formulating the PICO question, clinicians distinguish background questions (Example: what is a myocardial infarction?) from foreground questions (Example: do normoxic acute myocardial infarction patients benefit from supplemental oxygen?). The PICO question also helps clinicians to identify key search terms for the subsequent search.

Another term for the PICO question is the 4-part question (often seen in the United Kingdom emergency medicine literature), and more confusingly, it is sometimes referred to as a 3-part question (the ‘I' and ‘C' elements are combined into a single ‘intervention/comparator’ element). All of these are simply variants of the PICO format.

Evidence based medicine step 2: develop a search strategy

Freely available online search engines include PubMed (https://www.ncbi.nlm.nih.gov/pubmed/) and Translating Research Into Practice (TRIP) database (https://www.tripdatabase.com/). PubMed's advantages include video tutorials (https://learn.nlm.nih.gov/documentation/training-packets/T0042010P/) as well as multiple filters to stratify findings by year, author, design-type, ages included, and other important research considerations.

The TRIP database is a meta-engine searching various resources (PubMed, guidelines, and textbooks) before conveniently prioritizing findings by the evidence hierarchy depicted in Fig. 2. Since copyright restrictions limit access to many journals, Google Scholar (https://scholar.google.com/) is useful to find access to some manuscripts.

Evidence based medicine step 3: critical appraisal

The most time-consuming EBM step is critical appraisal of research. Best Evidence in Emergency Medicine provides appraisal forms for randomized controlled trials, systematic reviews, diagnostic research, guidelines, decision instruments, and prognostic studies (http://thesgem.com/2014/03/make-it-so-beem-appraisal-tools/). The structured approach to evaluating research points clinicians toward key questions to understand the risks of bias that can skew observed effect sizes across various study designs, as well as the likelihood of replicating published results in dissimilar settings.

Constructing PICO questions, conducting electronic search strategies then identifying research most likely to address the question, and then critically appraising these manuscripts requires time and mentorship. Educational leaders in emergency medicine therefore emphasize that clinicians develop secondary peer review resources that they trust [8]. Secondary peer review refers to synopses of original research by practicing clinicians.

An excellent introduction to this process can be found in the BETS (Best Evidence Topics) website (https://bestbets.org/) which was created nearly 20 years ago by researchers in Manchester to provide short, easy to digest PICO/4-part questions relevant to emergency medicine practice, initially in the United Kingdom, but with a global outlook. Many of these have been published in the Emergency Medicine Journal (amongst others) but there is a large repository of reports on the website now and anyone can contribute a topic report. There are clear instructions on how to undertake a good BET and feedback can be provided.

Over the last decade, various online pundits arose on podcasts and blogs worldwide. Some of the resources highlighted below incorporate the Users' Guide approach to critical appraisal. Others editorialize about new research without providing a structured appraisal. Disadvantages of reliance upon secondary peer review include the danger of authoritarian dictate and the atrophy of individual clinician's critical appraisal skills. However, with over 1 million new peer-reviewed papers being published every year, it is not possible for any emergency physician to remain current with the emerging literature unless a compromise approach is taken to at least incorporate important positive research findings into contemporary clinical practice in a timely manner. Methods to accurately and reliably identify practice-ready research and higher quality social media resources continue to evolve [17,18].

Tips on this topic

Form a journal club in your department, your hospital and your region – get journal club participants to appraise papers and present their findings to colleagues to stimulate discussion and influence practice.

Formally appraise clinical papers of interest and present your findings to a journal club for peer-group discussion at regular intervals (weekly or monthly).

Maintain a high degree of suspicion about published clinical research papers – do not assume that published papers are any better than those that never make it to publication.

Don't assume that just because a paper is published in the top journals in emergency medicine or high impact disciplines, it will be perfect: no paper is perfect, and you will find flaws and positive lessons in every paper that you appraise.

Before considering doing any research project, look at your research question and review the literature on the topic. Critically appraise the recent literature on the topic and see where the flaws and criticisms are, and design your research to minimize your study's chances of repeating those mistakes or flaws.

Pitfalls to avoid

Don't rely on other's critical appraisal of the evidence in isolation – your own appraisal will identify important local and regional factors that can make a huge difference to the implementation of apparently good evidence in your locale.

Don't just assume that a paper is worthless based on a harsh critical appraisal – what is not very useful for one clinical setting may be extremely useful for others, and vice-versa. For example, a head injury rule based on CT scanning will not have utility in a hospital or region with no CT scanner!

Don't get out of practice – the more you read and appraise papers critically, the better you will get at applying EBM for your patients' benefit.

Don't be afraid to be critical – the most common error is not making critical appraisal critical enough – every paper has flaws and biases. Your role is to identify as many of these as possible to see how the evidence fits in your setting, and to provide ideas for future potential research projects.

Authors' contribution

Authors contributed as follow to the conception or design of the work; the acquisition, analysis, or interpretation of data for the work; and drafting the work or revising it critically for important intellectual content: CRC contributed 75%; ELS 5%; and CAG and BH contributed 10% each. All authors approved the version to be published and agreed to be accountable for all aspects of the work.

Declaration of competing interest

CRC serves on the Society for Academic Emergency Medicine Board of Directors and American College of Emergency Physicians' Clinical Policy Committee. He chairs the Schwartz-Reisman Emergency Medicine Institute International Advisory Board. He is also faculty for Best Evidence in Emergency Medicine, Skeptics guide to Emergency Medicine, Sketchy EBM and Emergency Medicine Abstracts. The authors declared no further conflicts of interest.