Impact of human and artificial intelligence collaboration on workload reduction in medical image interpretation

Abstract

Clinicians face increasing workloads in medical imaging interpretation, and artificial intelligence (AI) offers potential relief. This meta-analysis evaluates the impact of human-AI collaboration on image interpretation workload. Four databases were searched for studies comparing reading time or quantity for image-based disease detection before and after AI integration. The Quality Assessment of Studies of Diagnostic Accuracy was modified to assess risk of bias. Workload reduction and relative diagnostic performance were pooled using random-effects model. Thirty-six studies were included. AI concurrent assistance reduced reading time by 27.20% (95% confidence interval, 18.22%–36.18%). The reading quantity decreased by 44.47% (40.68%–48.26%) and 61.72% (47.92%–75.52%) when AI served as the second reader and pre-screening, respectively. Overall relative sensitivity and specificity are 1.12 (1.09, 1.14) and 1.00 (1.00, 1.01), respectively. Despite these promising results, caution is warranted due to significant heterogeneity and uneven study quality.

Introduction

Imaging examinations are increasingly used for disease detection owing to their noninvasiveness and convenience, resulting in an escalating workload for clinicians. In Europe, double-reading of mammograms is recommended for breast cancer screening, with 100,000 women needing to be screened annually. However, the cancer detection rate is only ~6 per 1000 screened participants^1,2. The limited availability of clinicians, coupled with stringent time constraints, exacerbates this strain, potentially compromising the quality of patient care. Medical imaging, as a specialty, facilitates the integration of artificial intelligence (AI) owing to its established digital workflow and universal standards for image storage. Over 200 commercial AI products based on image detection have been approved by the Food and Drug Administration³. Integrating AI into clinical practice holds the promise of reducing the workload by reshaping the role of clinicians and streamlining clinical workflows, thereby freeing scarce experts to engage in more critical tasks.

Previous studies have predominantly focused on the comparative evaluation of standalone AI and human clinicians, with findings indicating equivalent diagnostic performance^4,5. However, the exclusive deployment of AI in isolation from human oversight raises significant ethical and legal concerns⁶. Human-AI collaboration, harnessing the strengths of both AI and clinicians, represents the most pertinent solution for current clinical scenarios. Numerous AI integration strategies lie on the spectrum of fully manual and automatic reviews. Examples include AI assisting clinicians in real-time image review⁷, serving as a second reader to validate findings alongside humans⁸, and acting as a pre-screening tool to identify low-risk cases that do not require human review⁹. These strategies have the potential to reduce clinicians’ workload, both in terms of time spent on image interpretation and the quantity of images reviewed.

In medical literature, optimistic findings regarding the accuracy of medical AI are often presented as assertions of its capability to enhance efficiency. This blending of concepts is inaccurate and misleading¹⁰. Although the workload lacks a precise definition, it encompasses at least two dimensions: image review time and quantity. To date, apart from one protocol¹¹, limited systematic reviews or meta-analyses have investigated the potential reduction in workload associated with the integration of AI into clinical workflows for image-based disease detection. Therefore, we aim to systematically reviewed the research status in this field and quantitatively analyzed the impact of different human-AI collaboration strategies on workload and diagnostic performance.

Results

Study selection

Figure 1 shows a flowchart of study selection. In total, 4302 studies were identified using electronic databases and manual searches. After removing duplicates, 3197 studies underwent title and abstract screening. A total of 134 full-text articles were assessed for eligibility, and 85 were included in the systematic review. Forty-nine studies were excluded because of insufficient information, leaving 36 studies for meta-analysis.

Fig. 1. Flowchart of study selection.

In total, 4302 studies were identified using electronic databases and manual searches. After removing duplicates, 3197 studies underwent title and abstract screening. A total of 134 full-text articles were assessed for eligibility, and 85 were included in the systematic review. Forty-nine studies were excluded because of insufficient information, leaving 36 studies for meta-analysis.

Study characteristics

Of the 85 studies included in the systematic review, 60 (70.6%) focused on reading time^7,12–70, while 25 (29.4%) focused on reading quantity^{2,8,9,71–92}. The number of studies showed an increasing trend over time, rising from 5 (5.9%) in 2019 to 30 (35.3%) in 2023. Most of the research data were sourced from Asia (n = 45), followed by Europe (n = 26), and the United States (n = 14). The studies covered various imaging modalities, with X-ray accounting for over 44.7%, followed by computed tomography (CT), magnetic resonance imaging, histopathology, endoscopy, digital breast tomosynthesis, ultrasound, and cytology. The most studied health conditions were breast cancer (n = 29), chest abnormalities (n = 16), and fractures (n = 11), followed by prostate cancer, cardiovascular and cerebrovascular diseases, small bowel disease, esophageal cancer, and gastric cancer. The study characteristics are summarized in Table 1 and Supplementary Table 1.

Table 1.

Characteristics of included studies

	Overall (n = 85)	Reading time (n = 60)	Reading quantity (n = 25)
Publication year
2019	5 (5.9)	2 (3.3)	3 (12.0)
2020	8 (9.4)	6 (10.0)	2 (8.0)
2021	16 (18.8)	13 (21.7)	3 (12.0)
2022	26 (30.6)	18 (30.0)	8 (32.0)
2023	30 (35.3)	21 (35.0)	9 (36.0)
Data source regiona
Asia	45 (52.9)	39 (65.0)	6 (24.0)
Europe	26 (30.6)	12 (20.0)	14 (56.0)
USA	14 (16.5)	9 (15.0)	4 (16.0)
Other	5 (5.9)	3 (5.0)	2 (8.0)
Imaging modalityb
X-rayc	38 (44.7)	23 (38.4)	15 (60.0)
CT	18 (21.2)	18 (30.0)	0 (0)
MRI	9 (10.6)	5 (8.3)	4 (16.0)
Histopathology	5 (5.9)	4 (6.7)	1 (4.0)
Endoscope	5 (5.9)	2 (3.3)	3 (12.0)
DBT	5 (5.9)	3 (5.0)	2 (8.0)
Ultrasound	4 (4.7)	4 (6.7)	0 (0.0)
Cytology	2 (2.4)	1 (1.7)	1 (4.0)
Health condition
Breast cancer	29 (34.1)	11 (18.4)	18 (72.0)
Chest abnormalities	16 (18.8)	14 (23.4)	2 (8.0)
Fracture	11 (12.9)	11 (18.3)	0 (0.0)
Prostate cancer	5 (5.9)	5 (8.3)	0 (0.0)
CVD	5 (5.9)	5 (8.3)	0 (0.0)
Small bowel disease	4 (4.7)	1 (1.7)	3 (12.0)
Esophageal cancer	3 (3.5)	2 (3.3)	1 (4.0)
Gastric cancer	2 (2.4)	2 (3.3)	0 (0.0)
Othersd	10 (11.8)	9 (15.0)	1 (4.0)
Role of AIe
As concurrent aid	60 (70.6)	60 (100.0)	0 (0.0)
As second reader	3 (3.5)	0 (0.0)	3 (12.0)
As pre-screening	23 (27.1)	0 (0.0)	23 (92.0)

Cells are frequencies (percentages).

CT computed tomography, MRI magnetic resonance imaging, DBT digital breast tomosynthesis, CVD cardiovascular and cerebrovascular disease.

^aThe total does not equal 85 because some studies are cross-regional.

^bThe total does not equal 85 because some studies include more than one image modality.

^cIncluding mammogram.

^d“Others” includes one each of bone metastases, brain metastases, cervical cytology abnormalities, fetal intracranial malformations, brain abnormalities, intussusception, lumbar spine stenosis, oral squamous cell carcinoma, soft tissue diseases, and thyroid nodules.

^eThe total does not equal 85 because one study includes AI both as second reader and as pre-screening.

For the 60 studies comparing reading times, X-ray (n = 23) and CT (n = 18) were the most commonly studied modalities, with chest abnormalities (n = 14) being the most frequently studied health condition. AI was exclusively utilized as a concurrent aid for image interpretation. For the 25 studies comparing reading quantity, X-ray (n = 15) took precedence as the most commonly examined modality, predominantly concerning breast cancer diagnosis (n = 18). AI served as a second reader in three studies and as a pre-screening tool in 23 studies.

Among the 36 studies included in the meta-analysis, 21 (58.3%) focused on reading time^{12,13,18,23,24,27,31,32,41,43,44,49,50,52,55,57,61,65–67,69}, including six on chest abnormalities, six on fractures, five on cancer, two on cardiovascular and cerebrovascular disease, one on fetal intracranial malformations, and one on intussusception. The other 15 studies (41.7%) focused on reading quantity^{8,9,74–81,83,86–88,92}, including nine on breast cancer, two on chest abnormalities, two on small bowel disease, one on cervical abnormalities, and one on esophageal cancer.

Workload reduction

As shown in Fig. 2 and Table 2, with the integration of AI, there was a notable reduction in the human workload for image interpretation, resulting in a time-saving of 27.20% (95% CI, 18.22%–36.18%) and a quantity-saving of 58.48% (95% CI, 46.83%−70.14%). Substantial heterogeneity was observed, with I² exceeding 99% (p < 0.001).

Fig. 2. Pooled human workload reduction after AI integration.

With the integration of AI, there was a notable reduction in reading time and reading quantity. a Pooled reading time reduction; b Pooled reading quantity reduction.

Table 2.

Human workload reduction and relative diagnostic performance after AI integration

	No. of study	Percentage workload reduction	Test for subgroup difference	Relative sensitivity	Test for subgroup difference	Relative specificity	Test for subgroup difference
Reading time reduction
Overall	21	27.20 (18.22, 36.18)		1.12 (1.08, 1.16)		1.00 (0.98, 1.01)
Image modalitya			0.750		0.215		0.068
X-ray	10	21.08 (11.11, 31.04)		1.13 (1.08, 1.19)		0.99 (0.97, 1.00)
CT	6	25.61 (17.79, 33.43)		1.12 (1.06, 1.19)		1.05 (0.99, 1.11)
Ultrasound	3	30.34 (-13.04, 73.71)		1.06 (1.00, 1.13)		1.07 (0.96, 1.18)
Health conditionb			0.932		0.562		0.005
Chest abnormalities	6	27.24 (15.53, 38.95)		1.12 (1.06, 1.20)		0.98 (0.96, 1.01)
Fracture	6	25.53 (16.44, 34.63)		1.18 (1.09, 1.28)		1.03 (1.00, 1.06)
Cancer	5	30.72 (1.77, 59.66)		1.12 (1.04, 1.21)		1.04 (0.97, 1.13)
Reader experiencec			0.425		<0.001		0.878
Junior	8	22.16 (8.74, 35.58)		1.24 (1.14, 1.35)		1.03 (0.99, 1.06)
Senior	8	16.67 (14.91, 18.41)		1.09 (1.05, 1.14)		1.03 (1.00, 1.06)
Reading quantity reduction
Overall	15	58.48 (46.83, 70.14)		1.11 (1.08, 1.15)		1.01 (1.00, 1.01)
Role of AId			0.018		<0.001		0.056
As second reader	3	44.47 (40.68, 48.26)		0.99 (0.96, 1.01)		1.00 (1.00, 1.00)
As pre-screening	13	61.72 (47.92, 75.52)		1.15 (1.10, 1.19)		1.01 (1.00, 1.01)

95% confidence interval in parentheses.

CT computed tomography.

^a2 studies, including 1 cytology and 1 histopathology, are not displayed in imaging modality subgroup analysis for reading time reduction due to limited number of studies for pooling.

^b4 studies, including 2 cardiovascular and cerebrovascular disease, 1 fetal intracranial malformation and 1 intussusception, are not displayed in health condition subgroup analysis for reading time reduction due to limited number of studies for pooling.

^cOnly 8 studies categorized readers as junior and senior.

^dThe total does not equal 15 because one study included AI both as second reader and as pre-screening.

The meta regression did not reveal any interaction between reading time reduction and the factors of imaging modality, health condition, or reader experience. Time reductions of 21.08% (95% CI, 11.11%–31.04%) and 25.61% (95% CI, 17.79%–33.43%) were observed in X-ray and CT scans, respectively. However, no significant time-saving was found in ultrasound (30.34% (95% CI, –13.04%–73.71%)). The time required for detecting chest abnormalities, fractures, and cancers decreased by 27.24% (95% CI, 15.53%–38.95%), 25.53% (95% CI, 16.44%–34.63%), and 30.72% (95% CI, 1.77%–59.66%), respectively. Only eight studies distinguished between the expertise levels of the readers, showing similar time decreases for juniors (22.16% (95% CI, 8.74%–35.58%)) and seniors (16.67% (95% CI, 14.91%–18.41%)).

After stratifying AI roles, AI as a pre-screening tool can save more reading quantity compared to being a second reader (61.72% (95% CI, 47.92%–75.52%) versus 44.47% (95% CI, 40.68%–48.26%), p = 0.018). Figure 3 shows a schematic diagram of the workload reduction for different AI roles.

Fig. 3. Schematic diagram of workload reduction for different AI integration strategies.

AI serving as a concurrent aid to humans reduced the reading time by 27.20%; when humans acted as the first reader and AI as the second reader with inconsistent arbitration, it reduced the reading quantity by 44.47%; when AI was used for pre-screening to identify relatively high-risk cases for human review, it reduced the reading quantity by 61.72%. Red boxes indicate instances where AI replaces readers or could have a direct influence on reader’s behavior. Yellow boxes denotes where AI could have an indirect effect on reader’s behavior. Green boxes indicate corresponding workload reduction after AI integration.

Relative diagnostic performance

As shown in Fig. 4 and presented in Table 2, relative sensitivity and specificity were 1.12 (95% CI, 1.09–1.14) and 1.00 (95% CI, 1.00–1.01), respectively. For studies evaluating time reduction, AI integration yielded an 12% increase (1.12; 95% CI, 1.09–1.16) in sensitivity, while specificity remained unchanged (1.00; 95% CI, 0.98–1.01). In studies assessing quantity reduction, AI integration resulted in a 11% increase (1.11; 95% CI, 1.08–1.15) in sensitivity and 1% increase (1.01; 95% CI, 1.00–1.01) in specificity.

Fig. 4. Pooled human relative diagnostic performance after AI integration.

With AI integration, sensitivity significantly increased (left), while specificity remained unchanged (right).

After stratifying the reader experience, juniors showed a greater improvement in sensitivity with AI integration compared to seniors (24% versus 9%, p < 0.001). The diagnostic performance after AI integration is non-inferior to that before AI integration across different modalities and conditions.

Before AI integration, the overall absolute sensitivity and specificity were 0.79 (95% CI, 0.76, 0.82) and 0.93 (95% CI, 0.89, 0.96), respectively. After AI integration, the overall absolute sensitivity and specificity were 0.88 (95% CI, 0.83, 0.92) and 0.95 (95% CI, 0.92, 0.97), respectively. Corresponding summary receiver operator characteristics (SROC) curves were provided in supplementary Fig. 1.

Quality assessment

A summary plot of the study quality, risk of bias, and concerns regarding applicability for each study are outlined in Supplementary Figs. 2 and 3. Twenty-nine studies were considered to have a high or unclear risk of bias for the participant selection domain, and eight studies did not clarify the qualifications of the participants. For the index test domain, 17 studies were at high risk of bias since they did not explain the functions of AI systems and user interfaces. In the reference standard domain, 22 studies faced a high or unclear risk of bias, mainly due to the absence of clear definitions and measures for reading time and quantity. Regarding flow and timing, 12 studies were deemed to have a high or unclear risk of bias, as the authors did not mention whether there was an appropriate interval between the index test and reference standard. For the applicability concern domain, 22 studies raised high or unclear concerns regarding participant selection. Twenty-four studies exhibited high concerns in the index test due to deviations from clinical practice. Additionally, five studies had high concerns in the reference standard domain. Supplementary Fig. 4 indicates potential publication bias for the outcomes of reading time reduction, as evidenced by asymmetry in the funnel plots and statistically significant p-values from Egger’s test (p < 0.001).

Discussion

Our results demonstrated that human-AI collaboration significantly reduced the workload in image-based disease detection without compromising diagnostic performance. Nearly one-fourth of the reading time was saved when aided by concurrent AI. The reading quantity decreased by 44.47% and 61.72% when AI served as the second reader and pre-screening, respectively. Additionally, the sensitivity increased by 12% and specificity remained unchanged, respectively, after the integration of AI. Despite this study presenting an optimistic view of AI, caution is warranted, as there is still a distance to clinical practice.

Our findings suggest that the concurrent use of AI with image interpretation reduces time across various imaging modalities and health conditions. However, caution is needed when extrapolating this result owing to significant heterogeneity and the fair quality of the included studies. Disease prevalence, AI algorithms, and reader characteristics are all potential contributors to this heterogeneity. Several studies have noted decreased reading time with lower AI scores but unchanged or increased time with higher AI scores, representing disease risk^7,41,47,55. This could be attributed to clinicians reporting more confidence in normal cases after referring to the AI results, thus enabling quicker decision-making. Conversely, when abnormalities are flagged by AI, clinicians may spend more time scrutinizing the validity of the AI assessment, irrespective of the displayed AI accuracy. Therefore, the proportion of abnormal cases in different studies is one source of heterogeneity. Moreover, differences in AI functionalities also impact interpretation time³². In cytological slide interpretation, AI can focus on and outline abnormal cells from thousands of cells, reducing the interpretation time from 4 min to less than 1 min³². However, in some AI implementations, such as displaying a heat map on an entire chest X-ray, the time-saving effect was not as apparent¹². Half of studies were at high risk of bias in the index test domain due to unclear explanations of AI system functions and user interfaces, potentially affecting AI performance evaluation. Similarly, unclear bias in the reference standard might lead to inconsistent measurements of reading time and quantity, affecting result reliability. Reader attitudes and acceptance toward AI also contribute to heterogeneity, as those embracing AI may experience smoother processes and save more time, while resistance to AI may disrupt the conventional interpretation workflow. Additionally, in most studies, readers examined a test set outside clinical practice, which is likely to result in the laboratory effect.

While AI concurrently aids in reducing reading time, clinicians are still required to review all images. However, when AI replaces the second reader for arbitration in cases of discrepancy with the first reader, it can reduce the image volume by 44.47%. Furthermore, utilizing AI as a pre-screening method to eliminate cases that do not require human review can decrease the interpretation quantity by 61.72%. As AI transitions from serving as a concurrent aid to a second reader and then to a pre-screening tool, it progressively assumes a higher degree of substitution for human involvement, leading to a more substantial reduction in workload. Our study revealed a broader range of conditions, including chest abnormalities, fractures, cancer, and others, when evaluating reading time. In contrast, research focusing on the decrease in reading quantity appears to be more concentrated on specific conditions, with 60% of studies focusing on breast cancer. In scenarios with high demand for screening but low disease prevalence, such as breast cancer screening, AI-driven solutions with a high degree of substitution are particularly well-suited. The world’s first randomized controlled trial on AI-supported breast cancer screening demonstrated that using AI as a second reader reduced image volume by 44.3% while maintaining a similar cancer detection rate². It would have taken one radiologist 4.6 months less to read 46,345 screening examinations in the AI integration group compared with 83,231 in the control group.

The stable diagnostic accuracy of AI systems is a prerequisite and foundation for further discussion on whether they can reduce the workload. However, despite showcasing immense promise in research, the seamless transition of AI into real clinical settings encounters significant challenges. Most studies have a high or unclear risk of bias for the participant selection, which could lead to insufficient representativeness of the sample and limit the generalizability of the results. Specifically, models underlying AI systems are often evaluated only within the context of their training settings. Even with external validation, it largely consists of retrospective studies, such as multi-reader and multi-case studies, and lacks a prospective design or validation across diverse clinical contexts^31,43,61. The performance of AI may degrade in the real world due to the diversity in healthcare systems, patient demographics, and clinical practice, raising concerns about its generalization and limiting its clinical usefulness^93,94. Enhancing transparency and rigor with the use of checklists represents a pivotal approach to verifying the proper implementation of AI models and ensuring adequate reproducibility. These checklists serve to standardize the reporting of crucial components pertaining to the construction of AI algorithms, including data sources, eligibility criteria, demographic characteristics, and imaging equipment utilized during data acquisition. Furthermore, post-deployment monitoring is crucial for refining AI models by regularly updating the training data and incorporating feedback from clinicians. This iterative process improves model performance and ensures that they meet the evolving needs of clinicians.

The successful use of AI depends on effective human-AI collaboration. Studies indicate that the benefits of AI assistance vary among different clinicians, with the least experienced clinicians gaining the most from AI-based support⁹⁵, which is consistent with our findings. However, many of the included studies failed to specify how readers were chosen and their level of experience. There is a possibility that less-experienced readers are intentionally selected to favor AI results, underrepresenting their real expertise in clinical settings. Another potential risk in human-AI collaboration is the human tendency to overly rely on AI systems, leading to automation bias. AI systems will inevitably make mistakes, including biased outcomes, “hallucinations,” and AI drift. Blinding following AI results can cause serious harm to patients. Readers, regardless of expertise, can be subject to automation bias, with inexperienced readers being more vulnerable to following incorrect suggestions from AI⁹⁶. Novices, overconfident in the capabilities of AI systems and lacking confidence in their own abilities, may experience short-term skill improvements. However, in the long run, this may hinder their development of independent diagnostic skills and critical thinking abilities, rendering them less effective without the help of AI.

Specific strategies to mitigate overreliance and automation bias should be considered to maximize the benefits of human-AI collaboration. An increasing number of studies have embraced the decision-referral approach to refine threshold selections^9,75,92. This approach typically involves both predictive and referral AI components. The referral AI utilizes the confidence score generated by the predictive AI to determine whether to apply the predictive AI model or to refer the given medical image to the standard clinical workflow for evaluation. It selectively makes decisions on a subset of cases with a high degree of accuracy while still ensuring the central role of humans in the decision-making process. Another strategy involves exploring the explainable AI (XAI) to dissect its “black box” nature and allow inferences about how a certain judgment by the system is generated. Various types of XAI methods have been reported to enhance transparency and provide insight into different medical specialities^97,98. Educating clinicians on the reasoning processes of AI systems will facilitate easier comprehension and enable them to make reasonable decisions. Additionally, instilling a sense of accountability in AI users for their own decisions can also aid in reducing automation bias.

The introduction of new technologies often entails systematic changes, and caution should be exercised regarding short-term efficiency gains that may result in long-term workload increases. Although the average reading time decreased from 180 s to 22 s per case with AI assistance when interpreting cervical cytology slides⁹⁹, only 400 slides can be scanned in 20 h because of the high requirements for image scanning to avoid image blur and information loss caused by overlapping cells. This additional time consumption also includes training clinicians on AI usage, adapting AI integration into clinical workflows, and regulating AI to foster a healthy ecosystem. Some optimists believe that once clinicians become familiar with an AI system, they can anticipate significant increases in efficiency⁹⁹. However, their willingness to acquire new skills and undertake new responsibilities influences whether the possible benefits of AI integration will be realized. Omitting the considerable human labor and time dedicated to the development and validation of AI systems also paints an overly optimistic picture regarding the total human effort required. Moreover, continuous input such as expert-annotated images will be indispensable even after AI systems are integrated into daily workflows to maintain system accuracy. The systematic effect can often only be assessed years after the introduction of AI with the help of sociologists, historians, and empirical observations of users’ daily experiences.

This study represents the first quantitative assessment of the impact of different AI integration strategies on workload reduction. However, several limitations of this study should be considered before extrapolating the conclusions. First, five researchers were involved in literature screening and data extraction, making it inevitable to introduce some subjectivity. However, multiple rigorous sessions were undertaken to ensure consistency among participants. Furthermore, some subgroups had a limited number of included studies, potentially undermining the stability of the pooled results, such as those involving ultrasound and AI as second readers. Many studies have treated reading time as a secondary outcome, reporting only point estimates without standard deviations or confidence intervals. Additionally, numerous studies have reported sensitivity without specificity, precluding their inclusion in the meta-analysis. Furthermore, the significant publication bias suggests that studies with non-significant or negative results may have been underreported, potentially leading to an overestimation of the actual effects of AI in this study. Advocating for the pre-registration of future studies can help reduce the selective reporting of significant results, enhancing research transparency. Additionally, future research should prioritize standardized reporting practices to foster comparability and synthesis across studies, thereby obtaining higher levels of evidence.

Our study examined only the reduction of workload during image-based disease detection. However, AI can also enhance clinical efficiency by performing tasks such as image acquisition and reconstruction, image segmentation, and the measurement of anatomical structures. Moreover, the rapid advancements in large language models (LLMs) hold the potential to revolutionize medical imaging¹⁰⁰. For instance, in the pre-imaging stages, LLMs can provide decision support, integrate clinical history, and assist in protocoling. LLMs can also improve reporting workflows in various ways, such as automatically generating report impressions by summarizing key findings, converting free text into structured reports, and proposing relevant differential diagnoses. These additional tasks underscore the broad spectrum of AI’s potential to enhance clinical efficiency beyond disease detection. Future research should evaluate the impact of AI on efficiency in these areas as well.

In conclusion, our systematic review and meta-analysis quantitatively elucidate the reduction in clinician workload and improvement in diagnostic performance through human-AI collaboration, providing a positive perspective on the integration of AI into clinical practice. However, prospective studies aligned with real clinical settings and further research into AI’s broader impact beyond disease detection tasks are essential for bridging the gap between research and practice and maximizing its potential to revolutionize medical imaging practices.

Methods

The study protocol was registered with PROSPERO (CRD42023443557) and was conducted and reported following the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) 2020 guidelines¹⁰¹. Please see Supplementary Note 1 for details.

Search strategy and eligibility criteria

We searched PubMed, Embase, Cochrane, and IEEE Xplore databases for studies published between January 2018 and December 2023 with no language restrictions. Medical Subject Heading (MeSH) terms and keywords related to AI, workload, and medical imaging were used to identify comparative studies assessing human workload before and after the introduction of AI. The full search strategies are listed in Supplementary Note 2. We also conducted manual searches of bibliographies, citations, and related articles from the included studies to identify additional relevant articles not captured in the initial searches.

The eligibility assessment involved two reviewers who independently screened the titles and abstracts of the search results, resolving discrepancies with a third reviewer. Studies were included if they compared the workload for detecting diseases from medical imaging with and without AI integration. The workload refers to either the time spent on image interpretation or the quantity of images reviewed. We excluded studies that: (1) used medical waveform data graphics; (2) did not focus on classification tasks; and (3) were based on animal or non-human samples. Additionally, case reports, review articles, editorials, letters, comments, conference abstracts, proceedings, and duplicates were excluded.

Data extraction

The primary outcome was the percentage of workload reduction, including relative reductions in reading time and quantity. It is defined as the difference between the pre-AI and post-AI workloads (time or quantity of image reviews), divided by the pre-AI workload. The secondary outcome evaluated was the relative diagnostic performance, defined as the post-AI diagnostic performance divided by the pre-AI diagnostic performance, consisting of relative sensitivity and specificity. Given the paired study design of original studies, relative sensitivity and specificity may be more valuable and informative than absolute sensitivity and specificity. For studies with multiple clinicians, we used the overall result if it was reported. If the overall result is not provided and only individual or group-specific results are available, we pooled them into a single effect, preventing overlapping information and inflated confidence from separate calculations^102,103. For diagnostic performance, we used the arithmetic mean of diagnostic contingency tables (true positive, true negative, false positive, false negative) from each individual reader as the pooled effects. For studies reporting multiple reading quantities at different AI thresholds or in different scenarios within the same patient group, we selected either the primary outcome reported by the original author or the more conservative result based on sensitivity analysis.

Two reviewers independently extracted data using standardized forms, encompassing study characteristics, patient characteristics, reader characteristics, AI characteristics, and outcomes of interest. Any disagreements were resolved by a third reviewer. The data were obtained through direct extraction or indirect calculation.

Quality assessment

We modified a risk assessment tool, the Quality Assessment of Studies of Diagnostic Accuracy-Revised (QUADAS-2)^104,105, by streamlining the criteria and incorporating elements relevant to our study aim. Our modified version assessed bias across four domains (participant selection, index test, reference standard, and flow and timing), similar to QUADAS-2, and evaluated applicability to the review question across the first three domains. Each domain included a tailored signaling question. The risk of bias and applicability in each domain were evaluated based on the responses to these questions. Supplementary Note 3 illustrates the adapted QUADAS-2. Two independent reviewers applied the modified tool to assess bias and applicability, resolving disagreements through discussion.

Data analysis

Studies reporting mean reading time and standard deviation (SD) or reading quantity and diagnostic contingency tables both before and after AI introduction were eligible for inclusion in the meta-analysis. We pooled the reading time reduction rate and reading quantity reduction rate along with 95% confidence intervals using random-effects model given the considerable heterogeneity between studies. Bivariate random-effects model analysis was used to pooled the relative and absolute sensitivity and specificity since these two indicators are often correlated. Summary receiver operating characteristics (SROC) curves were also plotted. I² statistics and Cochran’s Q test were used to assess the degree and statistical significance of study heterogeneity. Meta-regression was conducted based on imaging modality, health condition, reader experience, and the role of AI, provided that there were at least three studies in each subgroup. Subgroup interactions were tested using the chi-squared test. Egger test was used to assess publication bias in reading time and quantity reduction, and Deeks’ funnel plot asymmetry test was used to evaluate publication bias in diagnostic performance before and after AI integration. We used the meta package in R (version 4.1.0) for the analysis of workload reduction and the metadta command in STATA 18 for the analysis of relative diagnostic performance. Statistical significance was set at p < 0.05.

Author contributions

M.C. and P.X. contributed to the study design and conceptualization. M.C. and Z.Y. contributed to the literature search. M.C., Y.W., Q.W., J.S., and H.W. extracted data, conducted the analysis, and evaluated the quality of included studies. M.C. wrote the initial manuscript. P.X. and Y.Q. revised the manuscript. All authors approved the final version of the manuscript and take accountability for all aspects of the work.

Data availability

The datasets used during the current study are available from the corresponding author on reasonable request.

Code availability

The codes used in the analysis of this study will be made available from the corresponding author upon reasonable request.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41746-024-01328-w.