Advancing biomedical and health informatics knowledge through reviews of existing research

Rigorous reviews that adhere to methodological standards can advance biomedical and health informatics knowledge by synthesizing research and assessing its quality, identifying knowledge gaps, and making recommendations for research, practice, or policy. Thus, reviews are an important manuscript type for Journal of the American Medical Informatics Association (JAMIA) and complement other types of research papers. Reviews can be characterized based on 4 methodological aspects: search strategy (formal or informal), appraisal of quality (present or absent), synthesis (narrative or quantitative), and analysis (eg, quantity, quality, themes, knowledge gaps, limitations, recommendations).¹ In its 25 years, JAMIA has published more than 150 reviews; the frequency of reviews has increased in recent years with increasing awareness of the role of high-quality reviews as a foundation for future research on a topic. This has included scoping reviews that provide a preliminary assessment of potential size and scope of available research literature, but do not include a formal appraisal of study quality.² Most recent JAMIA reviews are identified as systematic reviews that include a formal search strategy, appraisal of study quality, and a narrative³ or quantitative (ie, meta-analysis)⁴ synthesis of findings. JAMIA has also published critical reviews that synthesize the literature conceptually and offer key recommendations for the field.⁵

This issue of JAMIA has 5 systematic reviews including 1 from an American Medical Informatics Association Working Group.⁶ In this editorial, I highlight how these systematic reviews demonstrate their relevance to the JAMIA audience and meet best-practice standards for systematic reviews as delineated in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement⁷ and other sources.

RELEVANCE TO JAMIA READERS

As a starting place, the purpose of the review must be relevant to JAMIA readers. Reviews that do not address a central topic in biomedical and health informatics are not a fit for JAMIA. The purpose may be relatively narrow or broad in scope. For example, Koleck et al⁸ focused on use of natural language processing to process or analyze information related to a selected set of symptoms in electronic health record free-text narratives while other systematic reviews examined broader topics such as clinical pathways,⁹ speech recognition technology for clinician documentation,¹⁰ and evaluation approaches for visual analytic technologies in health.⁶

FORMAL SEARCH STRATEGY

Systematic reviews must be based on a formal search strategy. The search strategy should be included in the manuscript or as an online supplement. Use of an informationist who is expert in search strategies increases the likelihood that the search strategy is appropriate. An inadequate search strategy is a common reason for rejecting a review submitted to JAMIA. Moreover, it is important that multiple databases are searched and that the databases are a match for the review topic. For example, Neame et al⁹ searched 3 databases for their review on the effects of implementing health information technology (HIT)–supported clinical pathways, Martin et al’s¹¹ systematic review on the impact of mobile technology on teamwork and communication in hospitals was based on a search of 7 databases, and Blackley et al¹⁰ reported a search of 10 scientific and medical literature databases to find articles about clinician use of speech recognition for clinical documentation. The range of years searched should be based on a sound rationale for the review topic.

FLOW OF INFORMATION THROUGH THE PHASES OF THE SYSTEM REVIEW

The flow of information through the different phases of a systematic review should be depicted with a PRISMA flow diagram that maps out the number of records identified, included, and excluded, and the reasons for exclusions.⁷ In addition to the flow diagram, the narrative should include who (minimum of 2) involved in the flow process, the level of agreement between them, and how discrepancies were resolved. The manuscript should also provide details about how the process was managed. The methods varied in our 5 examples, ranging from author-developed spreadsheets⁶ and specialized software such as Covidence from the Cochrane Collaboration (https://community.cochrane.org/help/tools-and-software/covidence).⁸

QUALITY ASSESSMENT

By definition, systematic reviews include a method of quality assessment. However, those methods vary depending on the focus of the review and whether there are published formal standards for the types of studies in the review. For their systematic review on the impact of mobile technology on teamwork and communication in hospitals, Martin et al¹¹ applied multiple formal methods of quality assessment: (1) the National Institutes of Health Study Quality Assessment tool for each type of study¹² and (2) a set of criteria specific to mobile health studies.¹³ In contrast, Blackley et al¹⁰ characterized the quality of studies by their reporting of metrics related to speech recognition for clinical documents, including number of speakers, number of documents, and a variety of accuracy measures. Similarly, Koleck et al⁸ reported on the number of documents and a set of metrics suitable for evaluating an natural language processing algorithm or pipeline including sensitivity, specificity, precision, recall, F measure, kappa statistic, area under the receiver-operating characteristic curve, and C-statistic. In 1 instance,⁶ the actual purposes of the review were to discover what evaluation measures were used in the literature and to make recommendations related to quality assessments. I encourage authors to visit the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network website (https://www.equator-network.org/reporting-guidelines/) to retrieve reporting guidelines for a variety of study designs to assist with selection of quality assessment criteria.

SYNTHESIS

No systematic reviews in this issue reported a meta-analysis; 1 review⁹ had planned to do so, but the heterogeneity and quality of the studies in the review precluded such an analysis. Synthesis approaches were primarily tabular along with narrative text. However, 3 reviews contained graphical components, including a histogram of studies over time,¹⁰ a chord diagram showing the associations between symptoms and studies,⁸ and a heatmap displaying the relationships among PICOS (participants, interventions, comparisons, outcomes, and study design)⁷ components and setting and measurement classifications.⁶ Now that JAMIA is an online journal, there are no charges for color, so I encourage authors to include graphs when appropriate.

ANALYSIS

Analyses for systematic reviews typically focus on 4 areas: what is known, what remains unknown, uncertainty around findings, recommendations for future research, and recommendations for practice.¹ To address the known vs unknown, the systematic reviews in this issue used PICOS⁷ or a variation on it to analyze studies and identify knowledge gaps. Given the informatics focus of the reviews, interventions were typically classified by a set of characteristics suitable for the types of technologies studied. Examples include clinical pathways (eg, clinical decision support, clinical documentation, dashboards)⁹ and visualizations (eg, type of intervention [dashboard, mobile app, etc.], type of visualization [planar, volumetric, network, etc.], unit visualized, interactivity).⁶ Although all reviews included some type of quality assessment and a limitations sections, 2 reviews^9,11 explicitly addressed the uncertainty in a more formal manner based on the rigor of the study designs. All reviews identified areas for future research. In addition, Wu et al⁶ provided 4 specific recommendations for the conduct of evaluation studies for visual analytic technologies.

REGISTRATION OF REVIEW PROTOCOL

Only 2 systematic reviews^9,11 in this issue, both from the United Kingdom, were registered in PROSPERO, an international database of prospectively registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, in which there is a health-related outcome.¹⁴ Similar to other prospective study registration sites, such as www.ClinicalTrials.gov, PROSPERO aims to help avoid duplication and reduce reporting bias by enabling comparison of the completed review with what was planned in the protocol. While not required for JAMIA, registration is encouraged for these reasons.

CONCLUSION

Bottom line, JAMIA is a great publication venue for systematic reviews focused on biomedical and health informatics topics. I hope that these examples offer guidance for JAMIA authors, motivate American Medical Informatics Association Working Groups to follow the lead of the Visualization Working Group to conduct and submit a systematic review, and assure JAMIA readers that the journal is a source of high-quality systematic reviews to inform future informatics research, practice, and policy.