Artificial intelligence for diagnostics in radiology practice: a rapid systematic scoping review

Summary

Background

The aim of this review was to evaluate evidence on the use of Artificial Intelligence (AI) to support diagnostics in radiology, including implementation, experiences, perceptions, quantitative, and cost outcomes.

Methods

We conducted a systematic scoping review (PROSPERO registration: CRD42024537518) and discussed emerging findings with relevant stakeholders (radiology staff, public members) using workshops. We searched four databases and the grey literature for articles published between 1st January 2020 and 31st January 2025. Articles were screened for eligibility (N = 8013), resulting in 140 included studies. Studies evaluated implementation (N = 7), perceptions (N = 74), experiences (N = 14), effectiveness (N = 53), and cost (N = 6).

Findings

Factors influencing AI adoption were identified, including the high technical demand, lack of guidance, training/knowledge, transparency, and expert engagement. Evidence demonstrated improvements in diagnostic accuracy and reductions in interpretation time. However, evidence was mixed regarding experiences of using AI, the risk of increasing false positives, and the wider impact of AI on workflow efficiency and cost-effectiveness.

Interpretation

The potential benefits of AI are evident, but there is a paucity of evidence in real-world settings, supporting cautiousness in how AI is perceived (e.g., as a complementary tool, not a solution). We outline wider implications for policy and practice and summarise evidence gaps.

Funding

This project is funded by the National Institute for Health and Care Research, Health and Social Care Delivery Research programme (Ref: NIHR156380). NJF and AIGR are supported by the National Institute for Health Research (NIHR) Central London Patient Safety Research Collaboration and NJF is an NIHR Senior Investigator. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

Research in context.

Evidence before this study

There is limited existing knowledge about how Artificial Intelligence (AI) is being implemented in radiology settings and the clinical implications of using these technologies. Existing reviews on the use of AI for diagnostics in radiology have not yet evaluated literature which integrates findings on: implementation; perceptions of staff, patients and the public; staff experiences; and impact, including effectiveness and cost.

Added value of this study

This review summarises current evidence on how AI is being implemented in diagnostic imaging, what staff, trainees, patients, and members of the public think about AI, experiences of using AI in practice, the quantitative impact, and cost. Furthermore, we build on existing work by discussing review findings with staff in radiology and members of the public who may experience AI-based care using workshops.

Implications of all the available evidence

Research suggests that current AI implementation is based on experimental learning rather than being informed by rigorous evidence. To understand how to best use AI in the complexities of radiology practice, we highlight the importance of evaluating how AI is implemented and used as a complementary tool in real-world settings. Future research should focus on key evidence gaps, including the process of implementation (including procurement), experiences of using AI in practice, long term cost-effectiveness, the risk of increasing false positives, and the impact on wider patient pathways and hospital systems.

Introduction

The use of Artificial Intelligence (AI) in healthcare is increasing globally due to its potential to address workforce shortages and rising healthcare demands.1, 2, 3 In radiology imaging, AI is being applied to assist with detecting abnormalities, enhance accuracy, reduce routine task time, and support clinical diagnoses.4, 5, 6 However, evidence of these potential impacts is inconsistent, making it challenging to draw conclusions.⁶ While there is excitement and optimism about the use of AI, there is limited research evaluating the effectiveness of AI in real-world healthcare settings, which goes beyond testing how AI could theoretically work.^2,6,7

Recent guidelines for clinical implementation highlight the importance of addressing evidence gaps, emphasising that a strong evidence base is essential for the continued use of AI in radiology.⁸ With interest in AI on the rise, it is crucial to understand how AI is being adopted globally for diagnostic purposes, evaluate the overarching effectiveness and costs of these technologies, and consider the experiences and perceptions of relevant stakeholders. Existing reviews have not yet evaluated literature integrating all of these topics. To address this gap, we conducted a rapid systematic scoping review and stakeholder workshops. Our review addresses the following research questions:

• 1.How have AI tools been implemented to support diagnostics in radiology?
• 2.How have AI tools supporting diagnostics in radiology been experienced and perceived?
• 3.What evidence exists on effectiveness and cost of AI tools to support diagnostics in radiology?
• 4.How has evidence on implementation, experiences, perceptions, quantitative outcomes, and cost of AI been measured, collected, and analysed?
• 5.What do stakeholders (staff and the public) think about the review findings?

Methods

Registration

This review was registered on PROSPERO (CRD42024537518).

Design

As part of a wider rapid evaluation of AI deployment for chest diagnostics in the English National Health Service (NHS) (part of the AI diagnostic fund, AIDF),⁹ we conducted a rapid systematic scoping review.¹⁰ We used the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement,¹¹ supplemented by two stakeholder workshops (one with radiology staff and one with members of the public) to discuss review findings. Scoping review methodology was chosen to address the varied review topics in a field of emerging evidence¹² and involves bringing together evidence and summarising the findings.¹³ In rapid scoping reviews, the scope/focus and processes are streamlined to enable a quicker evidence synthesis.¹³

Search strategy and selection criteria

Studies eligible for inclusion:

• •Focused on AI being used to support diagnostics in radiology (algorithmic use for image interpretation and decision-making, not image generation).
• •Empirical studies (covering implementation, experiences, perceptions, quantitative or cost outcomes). Quantitative studies needed to evaluate AI as a support tool for human decision-making, rather than being used in isolation; in line with current guidance that AI should be used with human supervision.^14,15
• •Published between 1st January 2020 and 31st January 2025, due to the rapid advancements of technology within healthcare during this period, and the publication of policy documents on the use of AI.¹⁵
• •Covered United Kingdom (UK)-based and international evidence.
• •Written in English.

See Appendix S1 for detailed inclusion and exclusion criteria.

Four databases were searched: Medline-Ovid [PubMed], PsycInfo, CINAHL Plus, and Web of Science Core Collection. Grey literature was identified through policy websites, topic-specific websites, and grey literature databases (e.g., Royal College of Radiologists, The Health Foundation, and Google Scholar). We also searched reference lists of review articles^2,16,17 and discussed identification of further papers with external stakeholders and clinical experts (TOR, FG). One researcher (RL) conducted the search and inputted records onto EndNote,¹⁸ followed by Rayyan,¹⁹ to remove duplicates.

The search strategy was developed iteratively using Ovid-Medline, with assistance from a UCL librarian (DM). Search terms were informed by previous research¹⁶ and guidance from clinical experts (TOR, FG). These terms covered areas such as AI, radiology imaging, clinical practice implementation, experiences and/or perceptions and quantitative and cost outcomes (Appendix S2).

Screening and study selection

Studies were screened in three phases: i) title, ii) abstracts/study summaries, and iii) full texts.¹⁰ One researcher (RL) screened all titles and abstracts. 10% of excluded papers were reviewed by one of three researchers (ED, EM, CSJ). Full texts were screened by one of four researchers (RL, ED, EM, CSJ), depending on the methodology of the paper. Disagreements were discussed within the team until a consensus was reached.

Data extraction and charting

A data extraction tool was developed. This included study characteristics, setting, implementation context, methods and design, and study findings pertaining to 1) implementation, 2) experiences, 3) perceptions, and 4) quantitative and cost outcomes (Appendix S3). One of four researchers (RL, ED, EM, CSJ) piloted the tool on one study relating to each of the different topics explored in the review. The data extraction tool was used to extract findings from all studies. Disagreements when developing and editing the tool were discussed within the team until a consensus was reached.

Synthesis of results

Narrative synthesis was used to analyse review findings.²⁰ We extracted all data relating to implementation, experiences, and perceptions, including illustrative quotes from qualitative studies. Data were coded line-by-line and findings grouped thematically. Quantitative evidence on effectiveness was synthesised by organising results by diagnostic accuracy, time and workflow, and change in clinical decision making. The narrative synthesis for cost outcomes was supported by abridged data extraction tables, including incremental costs, incremental cost-effectiveness ratio, cost savings, net present value, and quality-adjusted life years (QALYs). Due to the heterogeneity of the studies, a quantitative synthesis was not feasible.

Critical appraisal

The Mixed Methods Appraisal Tool²¹ was used to evaluate study quality. The tool was applied to quantitative effectiveness studies by two researchers (ED, CSJ), qualitative studies, mixed-methods, and quantitative survey studies by one researcher (RL). The quality of cost studies was evaluated (EM) using the Drummond checklist²² and the rating scale developed by Doran.²³

Stakeholder workshops

Two online stakeholder workshops were conducted. Workshops were held with members of the public and radiology staff working in an English healthcare setting (Appendix S4). Participants were recruited via public involvement in research channels, or via AIDF colleagues and clinical expert co-authors sharing the advert with radiology staff via email. Participants were selected based upon their experience in radiology and/or their experiences of diagnostic care, to ensure that a range of perspectives were included (Appendix S4). All participants were sent an information sheet and consent form ahead of the workshop. Preliminary findings were sent to participants ahead of the workshop and presented during the workshop (Appendix S5). Participants then discussed the findings. Workshops were audio-recorded and transcribed. Findings were analysed using thematic analysis²⁴ structured around review findings.

Ethics

Ethical approval for the workshops was obtained from the University College London Ethics Committee (27037/001). Written informed consent was obtained from all participants before taking part in the workshops. Ethical approval was not required for the review.

Role of funding source

The funders did not have a role in study design, data collection and analysis, writing of the manuscript or the decision to publish. The views and opinions expressed are those of the authors and do not necessarily reflect those of the NIHR or the Department of Health and Social Care.

PPIE co-authors (RM, JL, AH) were involved in study conceptualisation and design. They also co-designed study materials for the workshops, including the summary document sent to participants and presentation slides.

Results

Study selection and characteristics

8013 studies were identified, and 140 studies included (see Fig. 1), of which 7 studied implementation, 14 experiences, 74 perceptions, 53 quantitative impact and effectiveness, and 6 cost-effectiveness. Forty of the included studies were published in 2024–January 2025, indicating rapid growth in the AI research field (N = 13 published in 2020, N = 27 in 2021, N = 25 in 2022, N = 35 in 2023, N = 38 in 2024, and N = 2 in January 2025). Some studies covered multiple topics and have been included more than once in Table 1. Included studies were conducted in varied countries, covered different imaging modalities, and applied AI to varied patient pathways (Table 1).

Fig. 1.

Preferred reporting items for systematic reviews and meta-analyses (PRISMA) flow diagram.

Table 1.

Summary of study characteristics.

Study characteristics	Overall (N = 140)
Study focusa
Implementation	7
Experience	14
Perceptions	74
Effectiveness (quantitative studies)	53
Cost	6
Focus across multiple topicsa
Implementation, experiences, and perceptions	3
Experiences and perceptions	3
Implementation and perceptions	1
Effectiveness and cost effectiveness	2
Effectiveness and experiences	2

Study characteristics	Overall (N = 140)	Study focusa
		Implementation (N = 7)	Experiences (N = 14)	Perceptions (N = 74)	Effectiveness (quantitative) (N = 53)	Cost (N = 6)
Location
United Kingdom (UK)	17	4	2	13	2	1
China	16	0	1	3	13	0
United States of America (USA)	15	0	3	5	8	2
Korea	11	0	3	1	8	0
Germany	9	0	1	5	3	1
Saudi Arabia	9	0	0	9	0	0
Italy	7	0	1	4	3	0
The Netherlands	6	3	1	2	2	0
Australia	5	0	0	3	2	0
India	3	0	0	2	1	0
Japan	3	0	0	0	2	1
Spain	3	0	0	3	0	0
World-wide	3	0	0	3	0	0
Africa	2	0	0	2	0	0
Europe-wide	2	0	1	1	0	0
France	2	0	0	0	2	0
Nordic countries	2	0	0	2	0	0
Singapore	2	0	0	1	1	0
Switzerland	2	0	0	0	2	0
United Arab Emirates	2	0	0	2	0	0
Argentina	1	0	1	0	0	0
Australia & New Zealand	1	0	0	1	0	0
Austria	1	0	0	1	0	0
Canada	1	0	0	1	0	0
China and Germany	1	0	0	0	1	0
Egypt	1	0	0	0	1	0
Finland	1	0	0	0	0	1
Ghana	1	0	0	1	0	0
Ireland	1	0	0	1	0	0
Jordan	1	0	0	1	0	0
Malaysia	1	0	0	1	0	0
Malta	1	0	0	1	0	0
Middle East and India	1	0	0	1	0	0
Nigeria	1	0	0	1	0	0
Taiwan	1	0	0	0	1	0
Thailand	1	0	0	1	0	0
Unclear	1	0	0	1	0	0
Vietnam	1	0	0	0	1	0
Western Europe	1	0	0	1	0	0
Imaging modalities
Any (not specified—focused on AI more broadly)	67	3	3	65	0	0
X-ray	19	1	2	1	16	1
Computed tomography (CT)	18	1	2	2	16	0
Magnetic resonance Imaging (MRI)	10	1	1	2	7	1
Colonoscopy	7	0	1	1	5	2
Ultrasound	7	0	0	0	7	0
Radiographs	5	0	5	1	0	0
Mammography	2	0	0	0	2	0
MRI/MRI fusion biopsy	1	0	0	1	0	0
Computed tomography (CT) or magnetic resonance imaging (MRI)	1	0	0	1	0	0
Computed tomography angiography (CTA)	1	0	0	0	0	1
Images acquired during endoscopic procedures	1	0	0	0	0	1
X-ray, CT, DXA (bone density scan), and MRI	1	1	0	0	0	0
Patient pathways (including condition/s)
Any (not specified—focused on AI more broadly)	66	3	3	63	0	1
Chest/thoracic/lung	23	1	5	3	17	0
Colorectal	7	0	1	1	5	2
Prostate	7	1	1	3	4	0
Fractures	5	0	2	0	4	0
Stroke	5	0	0	0	4	1
Breast	4	0	0	0	4	0
Coronary artery disease	3	0	0	0	3	0
Pulmonary embolism	3	0	0	0	3	0
Thyroid	3	0	0	0	3	0
Dentistry	2	0	0	1	1	1
Ligament ruptures	1	0	0	0	1	0
Anaesthesia	1	0	0	0	1	0
Bone maturity	1	1	0	0	0	0
Covid-19	1	0	0	0	1	0
Lumbar spinal stenosis	1	0	0	0	1	0
Musculoskeletal	1	0	0	1	0	0
Pneumonia	1	0	1	0	0	0
Possible gastric neoplasm	1	0	0	0	0	1
Skin	1	0	0	1	0	0
Varying: Pulmonary embolism, intercranial haemorrhage, and acute cervical spine fractures	1	0	1	1	0	0
Varying: Cardiac, pulmonary, and musculoskeletal	1	0	0	0	1	0
Varying: chest/lung nodules, Covid, fractures, scoliosis, prostate, neuro/dementia	1	1	0	0	0	0
Participant group
Radiology staff		5	6	34	0	0
Wider clinical staff		1	4	4	0	0
Residents/students/trainees		0	2	10	0	0
Patients		0	0	11	0	0
Members of the public		0	0	2	0	0
Radiology staff, wider clinical staff, stakeholders, and patients		1	1	1	0	0
Radiology and wider clinical staff		0	1	0	0	0
Radiology staff and radiology students		0	0	7	0	0
Radiologists and computer scientists/industry and IT staff		0	0	3	0	0
Staff (radiology and wider) and members of the public		0	0	1	0	0
Staff (radiology and wider), patients, and members of the public (including some carers)		0	0	1	0	0
Diagnostic images and readers		0	0	0	53	0
Diagnostic images		0	0	0	0	6

^aThe total will not always add up to 140 as some papers had multiple areas of focus (as shown above), were conducted across countries, covered different imaging modalities and patient pathways and recruited multiple stakeholder groups.

Methods used for data collection and analysis

All implementation studies (N = 7) used qualitative methods (see Appendix S6). Experiences (N = 14) and perceptions (N = 74) were evaluated using quantitative, qualitative, and mixed methods. Of the quantitative studies that measured effectiveness of AI (N = 53), only 23 studies25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47 evaluated AI in a live pathway, of which 7 measured diagnostic accuracy. The remaining studies simulated typical workflow, often using a bespoke tool developed and validated as part of the study or a companion study. The effect of reader experience was the most common influencing factor measured to assess effect on findings (25/53).

The cost-related studies (N = 6) used Markov based models^37,38,48, 49, 50 and Monte Carlo forecasting methods⁵¹ to demonstrate the costs and health outcomes of AI-aided and non-AI strategies. Analyses were conducted from a societal perspective in the context of the origin country, using observational data and assumptions as well as parameters from randomised trials. Costs were estimated using the origin country's public and private reference costs.

The AI tools used in quantitative studies of effectiveness were either a pre-existing commercially available tool or a bespoke tool designed as part of the study or companion study (Appendix S6 and Supplementary File S1).

Measuring effectiveness and cost effectiveness

Quantitative measures used to evaluate effectiveness covered three categories: (1) Diagnostic accuracy, (2) effect on time and workflow, and (3) effect on clinical decision-making. General measures of diagnostic accuracy included sensitivity, specificity, area under the curve (AUC), positive predictive value (PPV), negative predictive value (NPV), and accuracy (proportion of correct results). Cost-effectiveness studies measured health outcomes including tooth retention time (in years) until carries detection for recipients of oral health care and quality-adjusted life years (QALYs). Other outcomes measured included incidence, mortality, scan time, and withdrawal time (Supplementary File S1).

Implementation

Out of the seven implementation studies, four explored the process of implementation, including the integration of AI into clinical practice and associated barriers and facilitators.52, 53, 54, 55 Three studies56, 57, 58 explored experiences of implementing AI in practice, with two identifying the main barriers and facilitators.^56,58 One study referenced procurement when discussing the high cost of implementing AI,⁵⁵ but no papers explored key processes preceding implementation (e.g., procurement).

Function of AI tools

AI was used as a tool to assist with diagnosis.52, 53, 54 One study reported 15 different applications being used in clinical practice (8 fully integrated and 7 in exploration or implementation phases) at one medical centre, including lung nodule detection, fracture assessment, measurement, and quantification.⁵⁴

The process of implementation

The process of implementation involved integrating AI into existing radiology systems used to store and transmit images and reports.^52,54,55 This enabled AI to run automatically in the background.⁵² One study described a holistic approach to implementing AI, where integration aligned social and technological aspects of clinical practice.⁵⁴ In this study, multiple AI algorithms were working in clinical workflows across one medical centre,⁵⁴ achieved by having one central workflow engine where data analysed by AI was sent to relevant data repositories.⁵⁴ Reasons for integrating AI into existing systems were centred around minimising workflow disruption, although some discussed teething problems (e.g., AI causing interruptions as staff adapted to viewing AI outputs).^54,55

Experiences of using AI

Fourteen studies explored staff and trainees' experiences of using AI (Table 1). In most studies, AI was viewed positively as a reliable and useable tool.^33,52,59, 60, 61, 62, 63, 64, 65 AI was mainly used as a second reader (a support tool/second pair of eyes for clinicians).^{52,55,60,62,63} However, there were mixed findings about using AI in practice. In some studies, staff felt AI helped to reduce reading times,^52,63,66 improve accuracy,^33,63,65,66 and efficiency.^58,65 Despite this, studies reported concerns about limited evidence,^55,62,66 the risk of false positives,^52,62 and reduced efficiency.³³ In the quantitative studies evaluating effectiveness, three studies measured reader confidence level in diagnostic decisions. Findings from these studies were mixed, some found improved self-reported confidence with the use of AI⁶⁷ but others found AI did not impact confidence.^34,67,68

Perceptions about using AI

Seventy-four studies explored perceptions (Table 1). AI was viewed favourably in radiology^52,55,65,69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87 as a diagnostic support tool.^{57,69,73,76,77,79,}88, 89, 90, 91, 92, 93, 94, 95 However, one paper highlighted the importance of balancing positivity towards AI in radiology with an element of scepticism.⁹⁶ Research indicated a consistent view that AI should not replace humans,^{52,55,60,65,70,80,83,84,86,89,90,}97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108 demonstrating the perceived importance of continued human input. For patients, members of the public and staff, trusting AI was often reliant on human oversight,^72,108, 109, 110, 111, 112, 113, 114, 115 transparency, and explainability.^58,73,114, 115, 116 Some radiology staff felt their profession is safeguarded by their autonomy,⁷⁴ as the use of AI alone is not viewed favourably⁹⁰ or legally.¹⁴ Although it was clear that AI should not replace the human element of diagnostics, studies reported that AI will likely change the radiology profession^{58,73,74,80,95,98,102,117} and there were some concerns about job security and reduced demand,^118,119 despite increased need for medical imaging.^1,2 Regarding how AI may change radiology, studies suggested that AI may upskill and enhance roles^52,58 but reported a risk of reduced autonomy.⁵⁸

Factors influencing AI adoption

Table 2 provides an overview of influencing factors. Barriers included the high technical demand of AI (N = 6), lack of knowledge and training (N = 20), absence of evidence-based guidance (N = 5), complex governance processes (N = 2), and how AI lacks empathy and human connection (N = 8). Facilitators included integrating AI into existing systems (N = 4), transparency (N = 13), willingness to learn (N = 21), and having AI champions/experts (N = 8). The associated risks were staff becoming over-reliant (due to developing algorithmic bias and/or blindness) (N = 5), AI being inaccurate/making errors (N = 6), deskilling staff (N = 5), the cost of AI (N = 11), and the ethical implications (N = 7). The benefits were improving reporting times (N = 7), improving diagnostic accuracy/identifying missed diagnoses (N = 12), improving workflow efficiency (N = 5), and helping with routine tasks (N = 10).

Table 2.

Overview of factors influencing AI adoption.

	Experienced/actual (either during implementation or clinical use)	Perceived (what people think when asked about AI)
Barriers	•High technical demand and integrating AI into existing infrastructures52,55, 56, 57, 58 •Absence of evidenced guidance53,54,56, 57, 58 •Differing expectations and knowledge53,54 •Uncertain and limited funding53,56 •Complex governance processes56,58	•Lack of knowledge and training about how AI can be used in radiology specifically71, 72, 73,78,79,82,85, 86, 87,92, 93, 94,99,116,120, 121, 122, 123, 124, 125 •Lack of organisational readiness and infrastructure to support AI81,88,89,101 •AI lacks human connection and empathy82,91,126, 127, 128, 129, 130, 131
Facilitators	•Establishing AI champions/experts who can foster a sense of leadership and openness53,54,56,57 •Integrating into existing systems/workflows (e.g., as staff are familiar with the systems)52, 53, 54,58 •Imaging and clinical implementation groups53,54 •Prioritising training and familiarisation55,56 •Openness among leaders53,57 •Importance of transparency55,57 •Willingness to learn about AI80,104	•Importance of transparency—achieved by prioritising staff training57,101,132 and facilitating trust by explaining the use of AI58,109,114, 115, 116,127,129 •Expert leadership and support58,96,121,132 •Willingness to learn about AI70,71,73,77,79,80,83,85,90,95,99, 100, 101, 102,104,105,116,120,125,133,134
Risks	•Staff relying on AI (as a result of staff developing algorithmic bias and/or blindness)52,55 •AI being inaccurate/making errors104 •AI causing disruption to workflow104 •High costs of AI57 •Ethical implications (e.g., uncertainties about who is responsible for AI, data privacy, safety etc.)52	•Staff relying on AI (as a result of staff developing algorithmic bias and/or blindness)52,55,87,90,93 •AI being inaccurate/making errors55,60,72,82,90 •De-skilling of staff55,75,78,128,135 •High costs of AI58,72,79,87,93,95,99,118,120,133 •Ethical implications (e.g., uncertainties about who is responsible for AI, data privacy, safety etc.)87,88,95,103,107,108
Benefits	•Improve accuracy52,65 •Improve reporting times52 •Reduce likelihood of follow-ups52,58 •Helping with routine tasks51,58,104 •Identifying missed diagnoses58,104 •Using AI for knowledge generation104	•Improve accuracy52,60,72,75,84,87,90,91,136 •Improve reporting times70,90,126, 127, 128,132 •Improving workflow efficiency84,91,99,128,132 •Helping with routine tasks60,75,79,82,87,96,128

Note: Where factors are experienced, we are referring to actual experiences of implementing or using AI in practice. Where factors are perceived, we are referring to what people think are the key influencing factors.

AI – Artificial Intelligence.

Quantitative evidence on impact and effectiveness

Of the 53 quantitative studies, 33 measured diagnostic accuracy of AI.^{26,28,31,34,37,40,44,67,}137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160 Across these studies, there was consistent evidence that AI reduces the number of incorrect positive or negative results, although some found improvements were more pronounced among less experienced staff, and any reported improvements were likely to be highly dependent on study design, implementation setting, and imaging modality.

Of the 25 studies that measured sensitivity, 19 reported improvements, 5 reported no change, and one study reported a reduction when AI was used as a diagnostic tool to assist readers (Table 3). However, some of the findings were specific to less experienced staff. Sixteen studies assessed sensitivity by reader experience level and nine reported improvements among less experienced staff only (Table 3). Of the 23 studies measuring specificity, findings were varied, with a smaller proportion reporting improvements: 13 studies reported improvements, 7 reported no change, and 3 a reduction in specificity. Ten of the 16 studies that measured the effect of reader experience on specificity reported improvements with AI assistance (Table 3).

Table 3.

Number of studies that measured diagnostic accuracy and influence of reader experience.

	Improved	No change	Reduction	Total
Sensitivity
Total papers that measured sensitivity	1926,31,37,137, 138, 139,141, 142, 143, 144, 145,149,150,152,153,155,156,158,159	528,35,147,148,157	167	25
Papers that assessed effect of reader experiencea
For all levels of experience	531,138,141,150,156	2147,157	0	7
For less experienced readers only	9b26,139,143, 144, 145,149,155,158,159	0	0	9
Specificity
Total papers that measured specificity	1431,67,137, 138, 139,141,144,149,151,153,156,158	728,35,145,150,155,157,159	326,142,148	24
Papers that assessed effect of reader experiencea
For all levels of experience	431,138,151,156	5145,150,155,157,159	126	10
For less experienced readers only	6139,141,143,144,149,158	0	0	6

^aWe have included three studies that measured the effect of clinical specialism and reader self-efficacy here as proxies for reader experience.

^bOne paper found improvements for less experienced staff doing more difficult tasks.

Evidence of improvement in time and workflow was variable, with improvements primarily seen in both reducing image interpretation time and time to report. Few studies reported on workflow impacts further along the pathway (Table 4). Although there were minor variations in definitions for interpretation time and time to report due to different study designs, interpretation time referred to time taken for a reader to interpret a single image, and time to report referred to time between image acquisition and documentation of the report. Of the 18 studies that measured image interpretation time, 13 found a reduction, while 5 reported an increase. There was also a lack of consistently observed benefits across different levels of reader experience (seniority or number of years in post) and suspicion of images (Table 4). Findings on time to report were more variable. Of the 10 studies that measured this outcome, 5 reported a reduction in reporting time, either for all images,^27,46,47 for critical and urgent cases,¹⁴² or less experienced readers.¹⁵⁵ However, this study¹⁵⁵ also found an increase in reporting time for non-urgent/normal images, as these images were deprioritised despite having shorter interpretation times with the aid of AI. Four studies found no change when using AI for all images or less experienced readers (Table 4).

Table 4.

Number of studies by workflow outcome measures and findings.

	Reduction	No change	Increase	Total
Interpretation time
Total papers that measured interpretation time	13ab34,36,40,45,47,139,142,145,150,152,153,155,160	228,138	5a39,47,142,147,161	18a
Papers that assessed effect of reader experience
For all levels of experience	4139,145,150,155	1138	1161	6
For more experienced staff only	0	0	1147	1
Papers that assessed effect of urgency of image findings
For non-urgent/normal images	1a142	1b36	0	2
For critical/urgent/more suspicious images	1b36	0	2a39,142	3
Reporting time
Total papers that measured reporting time	5c27,46,47,142,155	427,28,36,159	2c26,142	10c
Papers that assessed effect of reader experience/priority of images
For all levels of experience	0	0	126	1
For less experienced readers only	1155	0	0	1
Papers that assessed effect of urgency of image findings
For non-urgent/normal images	0	0	1c142	1
For critical/urgent/more suspicious images	1c142	0	0	1

^aTwo papers reported both a reduction and an increase in interpretation time depending on selected confounders, hence the reported total does not equal the sum of studies.

^bOne paper reported both a reduction and no change in interpretation time depending on selected confounders.

^cOne paper reported both a reduction and an increase in reporting time depending on selected confounders, hence the reported total does not equal the sum of studies.

Six studies focused specifically on the effectiveness of AI in emergency departments (ED).^{25,28,36,41,42,158} Readers in these studies were emergency department physicians and non-specialists in radiology. Two^28,36 found no difference in the length of stay, the rate of revisiting the ED within 30 days, nor communication times. However, one study²⁵ found shortened ED lengths of stay for patients with a confirmed diagnosis, whilst another found a small increase in median wait time to discharge.⁴¹ Two studies assessing the effect of AI on clinical decision-making found that changes to the final report, patient management, and image recommendations were more likely for more critical images, regardless of reader experience.³⁰ One study¹⁶² found that incorrect AI results can influence a radiologist to make wrong decisions (Table 4).

Evidence on cost-effectiveness

Findings from cost-effectiveness studies (N = 6) were mixed. One study demonstrated that although accuracy using AI was improved, the cost-effectiveness was not, as more invasive treatment approaches generated costs over the patient's lifetime and diminished possible effectiveness advantages.³⁷ Five papers demonstrated monetary benefits and cost-effectiveness of the AI-aided pathways.^38,48, 49, 50, 51

Feedback from stakeholder workshops

Both groups discussed how review findings reflected the NHS being in the early stages of using AI and learning through ongoing implementation. The complexity and varied implementation of AI were described as the ‘wild west’ by staff, with a lack of guidance and structure. However, when it comes to implementing AI in real-world settings, staff spoke on the importance of integrating AI into existing systems effectively, which causes minimal disruptions to workflow. Discussions suggested that AI would be challenging in clinical practice unless the existing context is considered. Thus, the integration of AI needs to be managed within the infrastructure of existing radiology systems.

Both groups were unsurprised about the mixed review findings in terms of experiences (e.g., some reporting AI had benefits and others reduced efficiency). The complexity of how AI is being used was noted (e.g., in different ways to achieve a range of objectives), making it difficult to provide a single answer about whether AI is beneficial. Therefore, the importance of returning to the purpose of using AI and how these technologies can help, was emphasised.

Both groups felt the review was relevant because it highlighted current evidence limitations, especially as AI is often viewed as a transformative solution. Stakeholders noted the importance of addressing evidence gaps (e.g., patient, carer and public experiences, health inequalities), and among public members, including patient voices as AI advances (example quotes in Appendix S7).

Quality appraisal of included studies

Qualitative (N = 15), quantitative survey, and mixed-methods studies (N = 68) met most of the criteria but there were some reports of sampling bias and limited description of qualitative analysis in mixed-methods studies (Supplementary File S2). The quantitative studies (N = 53) met most of the criteria for randomised or non-randomised studies, but many did not address potential confounders such as patient characteristics and co-morbidities. Since the findings depended on the interpretation of individual readers, the sample of readers in many studies was low, leading to risk of individual bias. Cost studies rated good (N = 6) (Supplementary File S3) but acknowledged factors that might impact the reliability of conclusions, including the identified range of the relevant costs, consequences for each alternative, the earliness in costs, country and facility-specific considerations and consequences identification and measurement.

Discussion

Our review highlights the paucity of research conducted in real-world settings. From the current evidence, we conclude that AI can have positive outcomes in relation to improving diagnostic accuracy and reducing interpretation times, which aligns with some early staff experiences and perceived benefits of using AI. Factors influencing implementation (e.g., high technical demand, lack of guidance, training and knowledge, transparency, and expert engagement) were also identified. However, we do not know enough about the system-wide impact of AI, the process of procurement to implementation, experiences of using AI and/or receiving AI-based care. Current and future implementation should consider if and how AI can address the needs of healthcare systems, the implementation context and educational training needs.

The limited number of studies conducted in real-world settings aligns with research gaps highlighted in evidence generation plans.⁸ Existing research⁶ and findings from stakeholder workshops suggest this is because services are in the early stages of implementing AI,⁵⁷ with further work emerging.⁶ The positive outcomes in relation to improving diagnostic accuracy and reducing interpretation times resonate with previous literature.^2,163,164 However, findings illustrated that improvements in diagnostic accuracy were more likely among less experienced staff² and there was evidence of AI overcalling negative findings; a risk reported in previous studies.¹⁶³ Furthermore, there was inconsistent evidence regarding experiences, how AI can improve workflow efficiency and whether these technologies are cost-effective, with few papers studying cost specifically. Variation in the quantitative findings were also likely to be dependent on study design, the imaging modality, and clinical application. From the current evidence, we cannot draw conclusions on how the potential benefits of AI may impact longer-term patient outcomes and the wider healthcare system (e.g., changes in volumes of patients for diagnostic/treatment services). Findings support previous research which shows AI has the potential to positively impact diagnostics in radiology.² In parallel, the evidence highlights caution in how AI is perceived (e.g., as a complementary tool which can help to navigate current demand, rather than a solution).¹⁶⁵

We extend previous work by reviewing literature on real-world implementation, demonstrating AI has been used to support diagnostics in a complementary role and not a replacement. This aligns with previous research,^15,166 user guidance,¹⁴ and stakeholder views,¹⁶⁷ which highlight the central role of clinicians in maintaining human continuity (ensuring that humans have oversight and AI is not used with complete autonomy). Although AI has the potential to positively impact radiology, the synthesised literature shows that continued human oversight and transparency about how AI makes decisions, are needed to foster a sense of trust when using AI for diagnostic purposes. These findings relate to ethical concerns often associated with AI, with existing evidence recommending that AI implementation should promote safety and transparency whilst reducing risk of harm.¹⁶⁸

Furthermore, although few papers have evaluated implementation, our findings highlight the complexity of integrating AI into existing healthcare systems, especially when organisations may not be ready to support such technical advancements,¹⁶⁶ with an absence of clear guidance.⁵⁶ Integrating AI will likely cause initial disruptions; our findings suggest that it is essential to evaluate the implementation context and ensure there is capacity to support the integration process.¹⁶⁶ Otherwise, as highlighted in our workshops, attempts to improve workflow efficiency by using AI may have the opposite effect. Variation in how AI is being used, complexity surrounding the ethical landscape and how AI can be used effectively alongside clinicians, emphasises a need for further evidence that can continue to inform clear implementation guidance and/or practice frameworks,^58,168 such as the recently developed European Union AI Act.¹⁶⁹

We advance previous research by synthesising staff experiences of using AI. Importantly, staff and trainees had limited experience using AI; consistent with survey papers where only a small proportion of participants had used AI in a clinical setting⁵⁹ and likely reflected limited clinical implementation. No studies explored patient, carer, or public experiences of AI-based care, although workshop discussions and current policy guidance³ highlight the importance of patient voice/engagement. Therefore, the evidence needs to include the experiences of groups whose acceptance and trust are important for the ongoing use of these technologies.^17,109

Although evidence shows potential value in using AI, our findings suggest that implementation and use are happening ahead of developing a robust evidence base.⁸ This was reflected further in stakeholder workshops, where implementation was described as a continuous learning process rather than being evidence informed. However, AI needs to be implemented to build an evidence base that explores real-world implementation. Therefore, to ensure future use can be evidence informed, there needs to be a careful balance between implementing AI safely and conducting robust evaluations, to enable learning from important technological advancements. For evidence users, our review highlights what is already known and what needs to considered moving forward, when interest continues to grow¹⁶⁵ and AI is used in other clinical areas.¹⁷⁰

For example, it is essential to be clear on the specific needs of healthcare systems (e.g., improving clinical outcomes and administrative efficiency), whether AI can effectively meet these needs over other solutions, and that these needs are communicated to AI developers, so that implementation is problem-driven rather than product-driven.¹⁷¹ Additionally, considerations including the healthcare pathway, country, and clinical conditions are needed, as there might be differences from setting-to-setting. Another factor is the population size as AI tools seem more effective in high-prevalence populations.³⁷ Furthermore, we highlight the need for tailored educational programmes with input from experts, that acknowledge current knowledge and the complexity of using AI across different clinical contexts.^2,172

Future research should evaluate the process of implementing AI into live clinical pathways or in shadow/testing mode,⁸ including pre-implementation processes and patient, carer and public experiences of AI-based care (e.g., experiences of the diagnostic pathway featuring demonstrations,^2,172 ethical factors like consent and transparency, patient safety, trust in AI and the impact of AI on empathy and human connection/relationships¹⁷). Such research may also evaluate how unsupervised AI (used without human supervision) is implemented, used, perceived, and experienced by different stakeholders. Although we did not include these papers, this could be an emerging focus as AI continues to be implemented globally and may be used with greater autonomy. Secondly, further research is needed on the effect of AI on patient outcomes, wider hospital systems (e.g., time to treatment, changes in volumes for other diagnostic or treatment services), diagnostic outcomes, and inequalities.⁸ Future research on cost-effectiveness of AI solutions in radiology is also needed.^8,37 Finally, understanding the long-term impact and sustainability of AI in clinical settings is essential.³⁷

The review had a broad and inclusive focus, supported by guidance from clinical experts (FG, TOR). Findings present a summary about how AI is implemented, used, and experienced globally, as well as current evidence on effectiveness and cost, which may be relevant for healthcare systems worldwide. However, it may be difficult to generalise findings across different health systems and only papers in English were included. As AI is a rapidly evolving field, we may not have captured all evidence and papers where AI was used autonomously were out of scope. Although we searched four databases and the grey literature, not all databases were used. There is also a risk of publication bias in AI research, as in other fields. Lastly, stakeholder workshops strengthened findings by illustrating implications, but only in the context of the English NHS.

To conclude, our review suggests potential value in using AI for diagnostics in radiology, mirrored in the ongoing interest in AI. However, to assist with safe and effective procurement, implementation, validation, and evaluation, research must be planned, commissioned, and used to address the current gaps in the evidence base. This will help to draw conclusions about how best to use AI as a complementary tool in the complexities of radiology practice.

Contributors

All authors contributed to the conceptualisation and development of the study.

Data curation: RL, ED, EM, CSJ.

Formal analysis: RL, ED, EM, CSJ.

Funding acquisition: CSJ, AIGR, NC, PLN, RM, SM, NJF.

Investigation: RL, ED, EM, CSJ, AIGR, HW.

Methodology: RL, ED, EM, CSJ, AIGR, HW, TOR, FG, NC, KH, SM, NJF.

Project administration: HE, PLN.

Resources: RL, ED, EM, CSJ, AIGR, HW, TOR, FG, HE, RM, JL, AH.

Supervision: HW, AIGR, CSJ, SM, NJF.

Verification of the underlying data: RL, ED, EM, CSJ, HW, AIGR.

Writing – original draft: RL, ED, EM.

TOR and FG provided clinical expertise and guidance. All authors contributed to the interpretation of findings, visualisation, revising, and finalising the paper. All authors have read and approved the final manuscript.

Data sharing statement

Data will be made available upon request made to the corresponding author.

Declaration of interests

AIGR is a trustee at Health Services Research UK. RM is Chair of the Board of Trustees of the Middlesex Association for the Blind; Vice-Chair on the Board of Trustees of the Research Institute for Disabled Consumers; Trustee on the Board of Thomas Pocklington Trust; Non-Executive Director on the Board of Evenbreak; Co-chair and Director on the Board of Shaping Our Lives. SM is currently (2022-) a member of the Small Business Research Initiative (SBRI) Healthcare panel. His post is funded in part by RAND Europe, a non-profit research organisation. SM is also Deputy Director of Applied Research Collaboration East of England (NIHR ARC EoE) at Cambridgeshire and Peterborough NHS Foundation Trust. NJF was a Non-Executive Director at Whittington Health NHS Trust until October 2024, a trustee at Health Services Research UK until 2022 and is a Non-Executive Director at Covid-19 Bereaved Families for Justice UK. TOR is part of the AXREM AI Special Focus Group, the British Institute of Radiology AI Special Interest Group, NHSE AI Deployment Fund Oversight Committee and Society of Radiographers AI Advisory Group. FG is a shareholder in Optellum Ltd, is a co-founder and Chairman of the RAIQC Ltd, was an advisor to NICE on the use of chest x-ray AI in the NHS and a committee member of the RCR Advisory group. All other authors report no declarations of interest.

Appendix ASupplementary data

Supplementary File S1

Overall review summary table.

Supplementary File S2

Quality assessment.

Supplementary File S3

Quality assessment (cost papers only).