Abstract
Introduction
The application of artificial intelligence (AI) technologies as a diagnostic aid in healthcare is increasing. Benefits include applications to improve health systems, such as rapid and accurate interpretation of medical images. This may improve the performance of diagnostic, prognostic and management decisions. While a large amount of work has been undertaken discussing the role of AI little is understood regarding the performance of such applications in the clinical setting. This systematic review aims to critically appraise the diagnostic performance of AI algorithms to identify disease from cross-sectional radiological images of the abdominopelvic cavity, to identify current limitations and inform future research.
Methods and analysis
A systematic search will be conducted on Medline, EMBASE and the Cochrane Central Register of Controlled Trials to identify relevant studies. Primary studies where AI-based technologies have been used as a diagnostic aid in cross-sectional radiological images of the abdominopelvic cavity will be included. Diagnostic accuracy of AI models, including reported sensitivity, specificity, predictive values, likelihood ratios and the area under the receiver operating characteristic curve will be examined and compared with standard practice. Risk of bias of included studies will be assessed using the QUADAS-2 tool. Findings will be reported according to the Synthesis Without Meta-analysis guidelines.
Ethics and dissemination
No ethical approval is required as primary data will not be collected. The results will inform further research studies in this field. Findings will be disseminated at relevant conferences, on social media and published in a peer-reviewed journal.
PROSPERO registration number
CRD42021237249.
Keywords: diagnostic radiology, surgery, computed tomography, magnetic resonance imaging
Strengths and limitations of this study.
This will be the first systematic review to evaluate the diagnostic performance of artificial intelligence models using cross-sectional radiological images of the abdominopelvic cavity, identifying current limitations and evidence gaps, and thereby focusing future research efforts.
Robust methodology will be undertaken including duplicate screening, data extraction and of risk of bias assessment.
Findings may be limited by the inclusion of English language publications only.
Introduction
In an era of ‘Big Data’, rapid developments in artificial intelligence (AI)-based technologies in medicine offer great potential to transform healthcare and improve patient outcomes.1 The widespread adoption of digital data in healthcare has provided a vast amount of data to enable computer algorithms to extract relevant information and recognise complex patterns.2 This includes quantitative (eg, laboratory values) and qualitative (eg, text-based electronic health records) data, as well as audio-visual data obtained from recordings from medical devices (eg, electrocardiograms, digital dictation). In a recent review, AI technologies were summarised as having an impact at three levels; clinicians, health systems and patients.3 For clinicians, AI technologies can help interpret images more rapidly and accurately improving the performance of diagnostic, prognostic and management decisions.4 For health systems, AI applications can improve workflow (eg, administrative jobs such as scheduling of operating rooms and clinic appointments). For patients, AI technologies can provide an opportunity for individuals to process their own data to promote health (eg, a smartwatch algorithm to detect a heart arrhythmia and the patient seeking appropriate medical attention).3
Medical imaging is considered a valuable source of diagnostic, prognostic and surveillance information. It also provides a pivotal role in supporting clinicians to perform procedural tasks. Images, however, have traditionally been dependent on human interpretation and there is an increasingly limited number of interpreters.5 There has been a surge of research exploring how AI technologies can be applied to medical images to support clinicians and provide greater efficacy and efficiency in clinical care.6 One of the most promising clinical applications of AI has been in diagnostic imaging,7 particularly for radiological7–14 and endoscopic15 16 investigations. AI diagnostic models have been used to detect pulmonary nodules,12 liver lesions,8 pancreatic cancer,9 colorectal cancer15 and hip fractures.13 14 These advancements can be stratified by imaging modality (eg, ultrasound, radiography, CT, MRI) and anatomical region (eg, head and neck, thorax, abdomen and pelvis, upper and lower limbs),17 organ7 or specialty. However, most AI studies are currently a proof-of-concept, rather than a model deployed in the clinical setting to explore the potential benefit. Few prospective studies and randomised controlled trials (RCTs) evaluating the application of AI have been undertaken, and those which exist are at high risk of bias.18
Evidence from some clinical specialties, including neurosurgery19 and gastroenterology20 offer insight into some of the promises and pitfalls for AI technologies in healthcare. Pitfalls include AI algorithms that can be difficult or impossible to interpret (referred to as ‘black box’ techniques) and requiring large amounts of high-quality data which can be difficult to access, especially across institutions.19 A review of the diagnostic accuracy of AI in radiological imaging of the abdominopelvic cavity is lacking. This could benefit a variety of different surgical specialties which employ diagnostic imaging for the abdominopelvic cavity. This systematic review aims to summarise the current research and critically appraise the diagnostic performance of AI models to diagnose disease from cross-sectional radiological images of the abdominopelvic cavity which may warrant an ‘invasive procedure’21 for ‘therapeutic intent’. This will be compared with standard practice. The quality of this research will also be assessed, to identify current limitations and inform future research efforts.
Methods
Protocol and registration
This protocol has been developed in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) Protocols (‘online supplemental file 1’) guidelines.22
bmjopen-2021-054411supp002.pdf (150.4KB, pdf)
Eligibility criteria
Primary research studies will be considered for eligibility using the PIRT framework (participants, index test(s), reference standard and target condition).23
Participants
Adults with abdominopelvic cavity pathology diagnosed from cross-sectional radiological imaging confined to CT, MRI and positron emission tomography (PET). Studies reporting endoscopy as an imaging modality will not be included in this review, as several reviews have already explored the performance of AI in this area.16 24
Index test
Studies considering AI models as an intervention with the aim to provide a diagnosis.
Reference standard
Standard practice.
Target condition
Abdominopelvic cavity pathology which has had, or may warrant, an ‘invasive procedure’21 for ‘therapeutic intent’.
Exclusion criteria for the studies as follows
Secondary research studies (eg, editorials and systematic reviews), case reports and case series.
Absence of full text publications (eg, conference abstracts).
Non-English articles.
Animal studies.
Outcome
The primary outcome is to evaluate the diagnostic performance of AI models using cross-sectional radiological images of the abdominopelvic cavity. The diagnostic performance will be referred to as previously defined: ‘the ability of a test to discriminate between the target condition and health’.25 Diagnostic measures of accuracy will include reported sensitivity, specificity and the area under the receiver operating characteristic curve.
Information sources
An electronic search of OVID SP versions of Medline, EMBASE and the Cochrane Central Register of Controlled Trials will identify all potentially relevant studies published since 1 January 2012, using a predefined search strategy (online supplemental appendix S1). The cut-off from 1 January 2012 is to accommodate for the advancement in machine learning performance with the development of deep learning approaches, an approach previously adopted in the literature.5
bmjopen-2021-054411supp001.pdf (40.7KB, pdf)
Search strategy and study selection
The search syntax will be developed with guidance from an information specialist using free text and Medical Subject Headings (MeSH) related to ‘artificial intelligence’, ‘diagnostic imaging’ and the ‘abdominopelvic cavity’ (online supplemental appendix S1). Database search results will be imported into the EndNote reference management software and duplicates will be removed.
Assessment of study eligibility will be a two-stage process. First, titles and abstracts will be screened for inclusion by two independent reviewers. Any identified conflicts will be resolved through discussion, including with the wider study team if required. Final eligibility will be assessed by full-text review of potentially eligible studies by the same process. The screening will be facilitated by Rayyan software.26 Reference lists of included studies will also be assessed for study eligibility.
Data extraction and management
Eligible studies will undergo data extraction by two independent reviewers using a predesigned standardised proforma and data management software (REDCap V.9.5.23). A standardised form will be used, which will include the following categories:
Study characteristics: first author, journal, year of publication, country of origin, study design (eg, case control, RCTs) reporting of ethical approval, regulatory approval (eg, Medicines and Healthcare products Regulatory Agency) and patient and public involvement (PPI).
Patient characteristics: pathology studied and surgical specialty of pathology.
Input features: modality of radiological imaging (eg, CT, MRI and PET), AI model used, size of training model, comparator group used and size of this data set.
Outcomes: diagnostic measures of accuracy and method of validation.
Risk of bias
Risk of bias of the primary diagnostic accuracy studies will be assessed using the QUADAS-2 tool.27 This will be done independently by at least two authors of the study and disagreements resolved by the study team.
Data synthesis
Search results and study selection will be presented in accordance with the PRISMA guidelines.28 Due to the broad nature of the PIRT and anticipated high levels of heterogeneity for the primary outcome a meta-analysis of data is not planned. Findings will be presented as a descriptive summary and narrative synthesis and will be reported according to the Synthesis Without Meta-analysis guidelines.29 The narrative synthesis will focus on the primary outcome with studies grouped by the modality of cross-sectional radiological imaging, pathology studied and surgical subspecialty.
Patient and public involvement
As part of the wider programme of work (Bristol Biomedical Research Centre, National Institute for Health Research (NIHR) Bristol BRC), patients and the public were consulted on their views of AI being used to guide doctors to make decisions about treatment. PPI will be sought for the dissemination of this systematic review.
Ethics and dissemination
Ethical approval is not required for the systematic review, as no primary data is collected. The review will be disseminated at relevant conferences, on social media and published in a peer-review journal.
Supplementary Material
Acknowledgments
The authors would like to thank Ms Catherine Borwick, an Information Specialist from the University of Bristol, for her expertise in developing the search strategy. The authors are also grateful to Dr Christin Hoffmann for leading patient and public feedback sessions as part of the wider programme of this research.
Footnotes
Twitter: @CSR_Bris, @NatalieBlencowe
Correction notice: This article has been corrected since it first published. Funding statement has been updated.
Contributors: GEF and NSB conceived the idea for this systematic review. The search strategy was developed by all authors. GEF drafted the manuscript protocol (guarantor of review), and it was critically appraised and revised by NSB, CH, MPC, NJS and RCM. All authors approved the final manuscript before submission.
Funding: NSB is funded by an MRC Clinical Scientist Award (MR/S001751/1). This study was also supported by the NIHR Biomedical Research Centre at University Hospitals Bristol and Weston NHS Foundation Trust and the University of Bristol. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.
Competing interests: None declared.
Provenance and peer review: Not commissioned; externally peer reviewed.
Supplemental material: This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
Ethics statements
Patient consent for publication
Not applicable.
References
- 1.He J, Baxter SL, Xu J, et al. The practical implementation of artificial intelligence technologies in medicine. Nat Med 2019;25:30–6. 10.1038/s41591-018-0307-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Alloghani M, Al-Jumeily D, Aljaaf AJ. The application of artificial intelligence technology in healthcare: a systematic review. Commun Comput Inf Sci 2020;1174:248–61. 10.1007/978-3-030-38752-5_20 [DOI] [Google Scholar]
- 3.Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med 2019;25:44–56. 10.1038/s41591-018-0300-7 [DOI] [PubMed] [Google Scholar]
- 4.Ben-Israel D, Jacobs WB, Casha S, et al. The impact of machine learning on patient care: a systematic review. Artif Intell Med 2020;103:101785. 10.1016/j.artmed.2019.101785 [DOI] [PubMed] [Google Scholar]
- 5.Liu X, Faes L, Kale AU, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health 2019;1:e271–97. 10.1016/S2589-7500(19)30123-2 [DOI] [PubMed] [Google Scholar]
- 6.Hosny A, Parmar C, Quackenbush J, et al. Artificial intelligence in radiology. Nat Rev Cancer 2018;18:500–10. 10.1038/s41568-018-0016-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Lundervold AS, Lundervold A. An overview of deep learning in medical imaging focusing on MRI. Z Med Phys 2019;29:102–27. 10.1016/j.zemedi.2018.11.002 [DOI] [PubMed] [Google Scholar]
- 8.Azer SA. Deep learning with convolutional neural networks for identification of liver masses and hepatocellular carcinoma: a systematic review. World J Gastrointest Oncol 2019;11:1218–30. 10.4251/wjgo.v11.i12.1218 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Liu K-L, Wu T, Chen P-T, et al. Deep learning to distinguish pancreatic cancer tissue from non-cancerous pancreatic tissue: a retrospective study with cross-racial external validation. Lancet Digit Health 2020;2:e303–13. 10.1016/S2589-7500(20)30078-9 [DOI] [PubMed] [Google Scholar]
- 10.Lakshmanaprabu SK, Mohanty SN, Shankar K. Optimal deep learning model for classification of lung cancer on CT images. Futur Gener Comput Syst 2019;92:374–82 https://scholar.google.co.in/citations?view_op=view_citation&hl=en&user=PLWCTg4AAAAJ&citation_for_view=PLWCTg4AAAAJ:dshw04ExmUIC [Google Scholar]
- 11.Titano JJ, Badgeley M, Schefflein J, et al. Automated deep-neural-network surveillance of cranial images for acute neurologic events. Nat Med 2018;24:1337–41. 10.1038/s41591-018-0147-y [DOI] [PubMed] [Google Scholar]
- 12.Nam JG, Park S, Hwang EJ, et al. Development and validation of deep learning-based automatic detection algorithm for malignant pulmonary nodules on chest radiographs. Radiology 2019;290:218–28. 10.1148/radiol.2018180237 [DOI] [PubMed] [Google Scholar]
- 13.Gale W, Oakden-Rayner L, Carneiro G. Detecting hip fractures with radiologist-level performance using deep neural networks, 2017. Available: http://arxiv.org/abs/1711.06504
- 14.Lindsey R, Daluiski A, Chopra S, et al. Deep neural network improves fracture detection by clinicians. Proc Natl Acad Sci U S A 2018;115:11591–6. 10.1073/pnas.1806905115 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Zhou D, Tian F, Tian X, et al. Diagnostic evaluation of a deep learning model for optical diagnosis of colorectal cancer. Nat Commun 2020;11:1–9. 10.1038/s41467-020-16777-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Hassan C, Spadaccini M, Iannone A, et al. Performance of artificial intelligence in colonoscopy for adenoma and polyp detection: a systematic review and meta-analysis. Gastrointest Endosc 2021;93:77–85. 10.1016/j.gie.2020.06.059 [DOI] [PubMed] [Google Scholar]
- 17.Pesapane F, Codari M, Sardanelli F. Artificial intelligence in medical imaging: threat or opportunity? radiologists again at the forefront of innovation in medicine. Eur Radiol Exp 2018;2:35. 10.1186/s41747-018-0061-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Raffort J, Adam C, Carrier M, et al. Artificial intelligence in abdominal aortic aneurysm. J Vasc Surg 2020;72:321–33. 10.1016/j.jvs.2019.12.026 [DOI] [PubMed] [Google Scholar]
- 19.Senders JT, Arnaout O, Karhade AV, et al. Natural and artificial intelligence in neurosurgery: a systematic review. Neurosurgery 2018;83:181–92. 10.1093/neuros/nyx384 [DOI] [PubMed] [Google Scholar]
- 20.Yang YJ, Bang CS. Application of artificial intelligence in gastroenterology. World J Gastroenterol 2019;25:1666–83. 10.3748/wjg.v25.i14.1666 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Cousins S, Blencowe NS, Blazeby JM. What is an invasive procedure? a definition to inform study design, evidence synthesis and research tracking. BMJ Open 2019;9:e028576–20. 10.1136/bmjopen-2018-028576 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Moher D, Shamseer L, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (prisma-P) 2015 statement. Syst Rev 2015;4:1–9. 10.1186/2046-4053-4-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Wisniewski DC, Diagnostic C. Guide to the contents of a cochrane diagnostic test accuracy protocol, 2013: 1–15. [Google Scholar]
- 24.Lui TKL, Tsui VWM, Leung WK. Accuracy of artificial intelligence-assisted detection of upper Gi lesions: a systematic review and meta-analysis. Gastrointest Endosc 2020;92:821–30. 10.1016/j.gie.2020.06.034 [DOI] [PubMed] [Google Scholar]
- 25.Šimundić A-M. Measures of diagnostic accuracy: basic definitions. EJIFCC 2009;19:203–11. [PMC free article] [PubMed] [Google Scholar]
- 26.Ouzzani M, Hammady H, Fedorowicz Z, et al. Rayyan—a web and mobile APP for systematic reviews. Syst Rev 2016;5:1–10. 10.1186/s13643-016-0384-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Whiting PF, Rutjes AWS, Westwood ME, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011;155:29–36. 10.7326/0003-4819-155-8-201110180-00009 [DOI] [PubMed] [Google Scholar]
- 28.McInnes MDF, Moher D, Thombs BD, et al. Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies: the PRISMA-DTA statement. JAMA 2018;319:388–96. 10.1001/jama.2017.19163 [DOI] [PubMed] [Google Scholar]
- 29.Campbell M, McKenzie JE, Sowden A, et al. Synthesis without meta-analysis (swim) in systematic reviews: reporting guideline. BMJ 2020;368:l6890–6. 10.1136/bmj.l6890 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
bmjopen-2021-054411supp002.pdf (150.4KB, pdf)
bmjopen-2021-054411supp001.pdf (40.7KB, pdf)