2023 Survey on User Experience of Artificial Intelligence Software in Radiology by the Korean Society of Radiology

Abstract

Objective

In Korea, radiology has been positioned towards the early adoption of artificial intelligence-based software as medical devices (AI-SaMDs); however, little is known about the current usage, implementation, and future needs of AI-SaMDs. We surveyed the current trends and expectations for AI-SaMDs among members of the Korean Society of Radiology (KSR).

Materials and Methods

An anonymous and voluntary online survey was open to all KSR members between April 17 and May 15, 2023. The survey was focused on the experiences of using AI-SaMDs, patterns of usage, levels of satisfaction, and expectations regarding the use of AI-SaMDs, including the roles of the industry, government, and KSR regarding the clinical use of AI-SaMDs.

Results

Among the 370 respondents (response rate: 7.7% [370/4792]; 340 board-certified radiologists; 210 from academic institutions), 60.3% (223/370) had experience using AI-SaMDs. The two most common use-case of AI-SaMDs among the respondents were lesion detection (82.1%, 183/223), lesion diagnosis/classification (55.2%, 123/223), with the target imaging modalities being plain radiography (62.3%, 139/223), CT (42.6%, 95/223), mammography (29.1%, 65/223), and MRI (28.7%, 64/223). Most users were satisfied with AI-SaMDs (67.6% [115/170, for improvement of patient management] to 85.1% [189/222, for performance]). Regarding the expansion of clinical applications, most respondents expressed a preference for AI-SaMDs to assist in detection/diagnosis (77.0%, 285/370) and to perform automated measurement/quantification (63.5%, 235/370). Most respondents indicated that future development of AI-SaMDs should focus on improving practice efficiency (81.9%, 303/370) and quality (71.4%, 264/370). Overall, 91.9% of the respondents (340/370) agreed that there is a need for education or guidelines driven by the KSR regarding the use of AI-SaMDs.

Conclusion

The penetration rate of AI-SaMDs in clinical practice and the corresponding satisfaction levels were high among members of the KSR. Most AI-SaMDs have been used for lesion detection, diagnosis, and classification. Most respondents requested KSR-driven education or guidelines on the use of AI-SaMDs.

INTRODUCTION

With the availability of abundant image data, artificial intelligence (AI) is rapidly being used in radiology. AI-based software as medical devices (AI-SaMDs) have become increasingly available, offering applications for various imaging modalities [1,2,3]. Several recent studies reported positive feedback from radiologists regarding the adoption of AI in clinical practice [4,5,6,7,8,9].

The AI market is growing rapidly in Korea [10]. In 2021, $2.76 billion was invested in privately owned Korean AI industries [11]. By the end of 2022, 149 AI-SaMDs were approved for clinical use in Korea [12]. Most of these AI-SaMDs (75.2%, 112 devices; Supplementary Tables 1, 2) were used for radiological examinations. Consequently, radiologists in Korea are in a favorable position to utilize AI-SaMDs in clinical practice. In this context, conducting user surveys on their experiences with AI-SaMDs, future research and development directions, and the roles of stakeholders in AI technologies is beneficial. A survey of Korean neuroradiologists conducted in 2022 reported that 58.9% and 72.6% of the respondents had used AI-SaMDs and are familiar with AI technology, respectively [13]. However, the degree of penetration of AI-SaMD in general radiological practice in Korea, and the experiences and preferences of all members of the radiological society regarding AI-SaMDs remain unknown.

We conducted a survey to assess the current clinical use, implementation, convenience, and expectations of future AI development among members of the Korean Society of Radiology (KSR). We also investigated the role that industries, governments, and the KSR need to play in advancing AI-augmented practices.

MATERIALS AND METHODS

Survey Development

In March 2023, members of the Radiology Imaging Network of Korea for Clinical Research (RINK-CR), a committee under the KSR, designed a survey to investigate the current status of experience with the use of AI-SaMDs among KSR members. After reviewing previous surveys for similar purposes, a three-member consensus panel from the RINK-CR developed the survey questionnaire (E.J.H., J.E.P., and C.M.P; attending radiologists in academic institutions) [4,5,6,7,8,9]. The questionnaire was designed to cover the 1) penetrance of AI-SaMD in clinical practice, 2) types of AI-SaMDs used in clinical practice, 3) level of satisfaction and preference for the use of AI-SaMDs, 4) future demand for AI-SaMD development, and 5) opinions on the role of stakeholders. The target time required to answer the questionnaire was 10 min to maximize participation. The final questionnaire, comprising 23 questions approved by all members of the RINK-CR, is summarized as follows. Supplementary items 1 (English translation) and 2 (original Korean version actually used for the survey).

Experience and Pattern of Practice

• • Length of professional career as a radiologist

• • Type of affiliated institution

• • Primary subspecialty field of practice

Experience in Using AI-SaMDs

• • Presence versus absence of experience in using AI-SaMD

Questions for Respondents with Experience in Using AI-SaMD

• • Number of AI-SaMDs used

• • Regulatory approval for the clinical use of AI-SaMD

• • Name of AI-SaMD

• • Use-case of AI-SaMD

• • Length of time using AI-SaMD

• • Target imaging modality of AI-SaMD

• • Motivation for the use of AI-SaMD

• • Degree of integration between AI-SaMD and the picture archiving and communication system (PACS)

• • Hurdles of integration between AI-SaMD and PACS

• • Level of satisfaction with AI-SaMD

  • ✓ Performance of AI-SaMD
  • ✓ Convenience of use
  • ✓ Quality improvement in radiologic practice
  • ✓ Improvement of practice efficiency
  • ✓ Quantitative image analysis
  • ✓ Improvement of patient management
  • ✓ Improvement of image quality
  • ✓ Utilization for research

• • Preference for continuous use of AI-SaMD

• • Reason for halting the use of AI-SaMD

Questions for Respondents without Experience Using AI-SaMD

• • Reason for not using AI-SaMD

• • Preference for future use of AI-SaMD

Opinions on the Roles of Stakeholders Regarding the Clinical Use of AI-SaMDs

• • Requirements for the use of AI-SaMDs in clinical practice

• • Demands for the expansion of AI-SaMD applications

• • Key considerations for AI-SaMDs development

• • Demand for education or guidelines from the KSR on the use of AI-SaMDs

• • Key components of the education or guidelines from the KSR on the use of AI-SaMDs

Survey Distribution

The online survey was conducted between April 17th and May 15th, 2023, using the Naver form survey tool (Naver, Seongnam, Korea). All members of the KSR, including trainee members under residency programs, were invited to participate in the survey via e-mail. Considering the exploratory nature of the survey, we did not specifically define a target number of respondents. Weekly reminders were sent via e-mail to increase the participation rate.

Analysis

The proportion of respondents with experience using AI-SaMDs was described in subgroups according to the length of their professional career, type of affiliated institution, and subspecialty field of practice. The levels of satisfaction with the use of AI-SaMDs were evaluated in a subgroup of respondents based on the length of their professional careers and the duration of AI-SaMDs usage. The demand for future expansion of AI-SaMD applications was evaluated in subgroups of respondents according to the length of their professional career, type of affiliated institution, subspecialty field of practice, and experience of using AI-SaMD.

RESULTS

Respondent Characteristics

Among the 4999 KSR members with available e-mail addresses, invitations to the survey were successfully sent to 4792 members. This resulted in 370 participants (response rate: 7.7%) participating in the survey. Of the respondents, 91.9% (340/370) were board-certified radiologists and 8.1% (30/370) were residency trainees. In total, 56.8% (210/370), 17.8% (66/370), and 10.5% (39/370) respondents were affiliated with tertiary, secondary, and tertiary non-training institutions, respectively (Table 1).

Table 1. Characteristics of the respondents and their experience of using AI-SaMDs.

Respondent characteristics		Number of respondents (%)	Number of respondents with experience of using AI-SaMD (row %)
All respondents		370 (100)	223 (60.3)
Length of professional career
	Residency trainees	30 (8.1)	18 (60.0)
	Less than 5 years of experience as a board-certified radiologist	88 (23.8)	56 (63.6)
	5–10 years of experience	88 (23.8)	53 (60.2)
	10–15 years of experience	50 (13.5)	24 (48.0)
	15 years or longer experience	114 (30.8)	72 (63.2)
	Type of affiliated institution
	Tertiary, training institution	210 (56.8)	141 (67.1)
	Secondary institution	66 (17.8)	26 (39.4)
	Tertiary, non-training institution	39 (10.5)	25 (64.1)
	Primary institution	28 (7.6)	13 (46.4)
	Military or public healthcare service	15 (4.1)	13 (86.7)
	Others	12 (3.3)	5 (41.7)
Field of practice*
	General radiology	130 (35.1)	64 (49.2)
	Musculoskeletal radiology	74 (20.0)	42 (56.8)
	Abdominal radiology	55 (14.9)	29 (52.7)
	Neuroradiology	48 (13.0)	38 (79.2)
	Breast radiology	41 (11.1)	31 (75.6)
	Chest radiology	39 (10.5)	32 (82.1)
	Cardiovascular radiology	22 (5.9)	18 (81.8)
	Interventional radiology	21 (5.7)	5 (23.8)
	Genitourinary radiology	16 (4.3)	9 (56.3)
	Pediatric radiology	8 (2.2)	7 (87.5)
	Others	5 (1.4)	0 (0)

^*Multiple responses allowed.

AI-SaMD = artificial intelligence-based software as medical device

Frequency of AI-SaMD Usage Experience

Overall, 223 respondents (60.3%) had used AI-SaMD. The usage experience was relatively higher in those affiliated with military or public healthcare services (86.7%, 13/15), training (67.1%, 141/210), or non-training (64.1%, 25/39) tertiary institutions, compared to those affiliated with primary (46.4%, 13/28) or secondary (39.4%, 26/66) institutions (Table 1).

Characteristics of AI-SaMDs and Patterns of Usage

Table 2 shows the patterns and characteristics of using AI-SaMDs among the 223 respondents with experience of usage. The most common function of AI-SaMDs was the detection of radiological findings (82.1%, 183/223), followed by lesion diagnosis and classification (55.2%, 123/223) (Fig. 1). Target examinations of AI-SaMDs are summarized in Supplementary Table 3. The most widely used AI-SaMDs were those for chest radiography (n = 152), followed by those for brain MRI (n = 71), chest CT (n = 63), and mammography (n = 59). Regarding the motivation for AI-SaMD use, 47.1% (105/223) of the respondents answered that they used AI-SaMD just because their institution purchased it. Regarding PACS integration, 153 (68.6%) respondents answered that all AI results were sent to the PACS. The most common reasons for limited integration with the PACS were technical difficulties with integration (43.9%, 29/66) and institutional policies (33.3%, 22/66).

Table 2. Experience of 223 respondents with experience of using AI-SaMD: characteristics of AI-SaMD and its usage.

AI-SaMD characteristics		Number of responses (%)
Numbers of used AI-SaMDs*
	One	81 (36.3)
	Two	67 (30.0)
	Three or more	75 (33.6)
Length of usage period*
	Less than 6 months	70 (31.4)
	6 months to 1 year	44 (19.7)
	1–2 years	54 (24.2)
	2–3 years	32 (14.3)
	3 years or longer	23 (10.3)
Regulatory approval*
	Ministry of Food and Drug Safety of Korea approval	183 (82.1)
	Research purpose use before approval	103 (46.2)
Function of AI-SaMD*
	Detection of radiological finding	183 (82.1)
	Lesion diagnosis and classification	123 (55.2)
	Automated measurement and quantification	75 (33.6)
	Improvement of image quality	33 (14.8)
	Reduction of image acquisition time	15 (6.7)
	Prioritization of image interpretation	10 (4.5)
Target image modality*
	Plain radiography	139 (62.3)
	CT	95 (42.6)
	Mammography	65 (29.1)
	MRI	64 (28.7)
	Ultrasonography	8 (3.6)
Motivation for AI-SaMD usage*
	Purchase by the institution	105 (47.1)
	To conduct research using AI-SaMD	93 (41.7)
	To improve the efficiency of practice	71 (31.8)
	To experience the clinical performance of AI-SaMD	71 (31.8)
	To improve the quality of practice	67 (30.0)
Integration with PACS
	All AI results sent to PACS	153 (68.6)
	Summaries of AI results sent to PACS; access to separate workstation required for full results	29 (13.0)
	AI results not sent to PACS	37 (16.6)
	No response	4 (1.8)

^*Multiple responses allowed.

AI-SaMD = artificial intelligence-based software as medical device, PACS = picture archiving and communication system

Fig. 1. Comparison between applications of currently used artificial intelligence-based software as medical devices and demand for future expansion of application. The percentages are illustrated on a spider diagram. The current usage and future expectations are shown in blue and purple, respectively. The numbers of currently used detection/diagnosis tools in Table 1 have been merged as lesion detection and diagnosis prediction tools.

Satisfaction and Preference for the Use of AI-SaMDs

Figure 2 shows the level of satisfaction among respondents with experience using AI-SaMDs. Satisfaction with the performance of the AI-SaMD and convenience of use were 85.1% (189/222) and 83.3% (184/221), respectively. Meanwhile, relatively lower satisfaction rates were observed for improvements in image quality (69.4%, 84/121) and patient management (67.6%). Respondents with more professional experience (≥ 15 years of experience as board-certified radiologists) exhibited a relatively lower satisfaction rate for performance (79.1%), convenience of use (72.2%), improvement in patient management (63.6%), and improvement in image quality (62.9%). Respondents with ≥ 1 year experience of using AI-SaMD exhibited higher satisfaction rates compared to those with < 1 year of experience for all items (Supplementary Table 4). Among the respondents who had experience using AI-SaMD, 71.7% (160/223) answered that they would continue using AI-SaMD. 24.2% (54/223) answered that they no longer used AI-SaMD and 4.0% (9/223) said they planned to stop using it. The most common reason for stopping was license expiration (57.1%, 36/63).

Fig. 2. Level of satisfaction among respondents with experience using artificial intelligence-based software as medical devices. n = number of total respondents for each item.

Among the 147 respondents without experience using AI-SaMD, 44.2% (65/147) answered that they were willing to use AI-SaMD if it was available, and 52.4% (77/147) answered that they would use it if performance or circumstances were improved. The most common reason for not using AI-SaMD was that the institution did not purchase an AI-SaMD (75.5%, 111/147) (Table 3).

Table 3. Preferences for the use of AI-SaMDs among the 147 respondents without experience of usage.

Responses		Number of responses (%)
Reasons for not using AI-SaMD*
	Not purchased by the institution	111 (75.5)
	Clinical ineffectiveness of AI-SaMD	34 (23.1)
	High cost of purchasing AI-SaMD	27 (18.4)
	Inconvenience of using AI-SaMD	12 (8.2)
	Cumbersome process of purchasing and installing AI-SaMD	10 (6.8)
	Negative effect of using AI-SaMD	10 (6.8)
Preference towards use of AI-SaMD
	Willing to use AI-SaMD currently	65 (44.2)
	Currently not, but willing to use AI-SaMD if performance or circumstance are improved	77 (52.4)
	Not willing to use AI-SaMD	4 (2.7)

^*Multiple responses allowed.

AI-SaMD = artificial intelligence-based software as medical device

Opinions on the Usage and Development of AI-SaMDs

Regarding the areas where the application of AI-SaMDs requires expansion, the most common response was assistance for detection and diagnosis (77.0%, 285/370), followed by quantitative image analysis (63.5%, 235/370), workload reduction through automated interpretation (45.9%, 170/370), and quality control through screening of interpretation errors (40.8%, 151/370) (Fig. 1). The demand for assistance in detection and diagnosis was relatively higher among respondents in primary (96.4%) and secondary institutions (83.3%), compared with those in training (75.2%) and non-training (66.7%) tertiary institutions. Respondents in the military or public healthcare services exhibited relatively higher demands for image quality improvement (46.7%), reduction in image acquisition time (40.0%), and prioritization of interpretation (60.0%). The demand for workload reduction through automated interpretation was relatively higher in respondents with longer professional careers (56.1% for ≥ 15 years of experience versus 36.4% for < 5 years of experience as a board-certified radiologist). General radiologists expressed a relatively higher demand for assistance in detection and diagnosis (84.6%) and a lower demand for automated measurement and quantification (60.8%) (Supplementary Table 5).

Regarding the focus on the future development of AI-SaMDs, most of the respondents indicated improvement in practice efficiency and workload reduction (81.9%, 303/370), improvement in practice quality (71.4%, 264/370), and offering novel information through quantitative image analysis (56.5%, 209/370). Relatively minor opinions included image quality improvement (23.2%, 86/370), improvement in patient management (14.1%, 52/370), and utilization for research (8.1%, 30/370).

Opinions on the Roles of Stakeholders Regarding the Clinical Use of AI-SaMDs

Figure 3 illustrates the role of each stakeholder in the AI usage environment. Regarding the role of manufacturers of AI-SaMDs, 53.8% (199/370) and 45.9% (170/370) of the respondents indicated an improvement in the convenience of use and performance of AI-SaMDs, respectively. Regarding the role of the government, 48.1% (178/370) and 45.9% (170/370) of the respondents commented on the necessity of reimbursement for using AI-SaMDs and clarification of legal liability issues, respectively. Regarding the role of the KSR, 91.9% of the respondents (340/370) agreed on the need for education or guidelines for the use of AI-SaMD.

Fig. 3. Role of the Korean Society of Radiology, Government, and AI industries in the AI usage environment. The radiologists are in the middle of the interchangeable connections. The role of each stakeholder is written based on the highest percentage. AI = artificial intelligence, PACS = picture archiving and communication system.

Regarding the content of education and guidelines, many respondents asked for recommendations on the use of AI-SaMDs in various application scenarios (62.4%, 231/370) and objective information on the performance of AI-SaMDs (61.1%, 226/370). Requests for information on the introduction of available AI-SaMDs (53.5%, 198/370) and practical experience in purchasing, installing, and using AI-SaMDs (50.3%, 186/370) were frequently requested.

DISCUSSION

Herein, we report the results of a survey conducted to assess the current usage patterns, satisfaction levels, and future development directions of AI-SaMDs among members of the KSR. Of the respondents, 60.3% had experience using AI-SaMDs and most were satisfied with AI-SaMDs. Additionally, we summarize the necessary actions undertaken by stakeholders for better clinical practice based on AI-SaMDs in the future.

Most respondents (60.3%) had used AI-SaMDs. The penetration of AI into clinical practice was relatively high among the responding KSR members, especially considering that a systematic review of 758 healthcare respondents from 39 countries exhibited AI penetration rates of 10%–30% [14]. Furthermore, the penetration rate was twice that of the 2020 survey by the American College of Radiology Data Science Institute, which reported a penetration rate of 30% [5]. The penetration rate for neuroradiologists (79.2%) increased considerably compared to that reported in a previous survey conducted in April 2022 (58.9%) [13]. This suggests that the adoption of AI-SaMD is spreading rapidly. The most widely used AI-SaMDs are those used for chest radiography, brain MRI, chest CT, and mammography (Supplementary Table 3), which can be partly explained by the available AI-SaMDs (Supplementary Table 1). The most common function of AI-SaMDs was the detection of radiological findings (82.1%), followed by lesion diagnosis and classification (55.2%). This differs from the use of AI-based volumetry by neuroradiologists as the most common application, which reflects that the use of AI-SaMDs depends on the imaging modality or subspecialty field of the users [13].

Most users reported using two or more AI-SaMDs (63.7%), most of which had received regulatory approval (82.1%), and nearly half were purchased by their institutions (47.1%). Most AI-SaMDs were integrated with PACS (68.6%); however, considering that the most common imaging modality was plain radiography (62.6%), caution should be exercised to avoid overgeneralizing the high prevalence of PACS integration. The user satisfaction with AI-SaMDs was high, not only with their performance but also with respect to their convenience of use. Most users wanted to continue using AI-SaMDs, and 44.2% of non-users expressed a desire to use AI-SaMDs in the future. This is in line with previous survey studies [4,5,6,7,8,15,16], where they found that non-users shared positive attitudes with regard to the future use of AI tools.

Currently, the primary clinical purposes of AI-SaMDs include detection, lesion diagnosis, classification, and quantification. Despite the current availability of AI-SaMDs, most respondents expected an expansion in the application of AI-SaMDs for detection/diagnosis assistance (77.0%) and quantification (63.5%), suggesting that the current direction of AI-SaMD development and commercialization might be in accordance with the expectations of users. It is noteworthy that > 40% of the respondents suggest that workload reduction through automated interpretation and quality control through AI-based error screening systems would prevail in the future, which are currently unavailable (Fig. 1). Most users were satisfied with the performance of AI-SaMDs (85.1%) and the convenience of use (83.3%). However, satisfaction with improvements in practice efficiency (73.7%), image quality (69.4%), and patient management (67.6%) were relatively low. These relatively low satisfaction levels may be related to the fact that AI-SaMDs designed for these use cases are relatively scarce in the market.

Most respondents indicated that the future development of AI-SaMDs should focus on the improvement of practice efficiency, workload reduction (81.9%), and the improvement of practice quality (71.4%). These top two suggestions are in line with a previous survey conducted by the European Society of Radiology in 2022, which found that the two main expected advantages of AI were a lower error rate (73.0% of respondents) and optimization of radiologists’ work (67.9%) [7]. Similar expectations were also suggested by French radiologists in 2019, with the highest expectations being the lowering of imaging-related medical errors (81.1%, 219/270), followed by the lowering of the interpretation time of each examination (74.4%, 201/270) and an increase in the time spent with patients (52.2%, 141/270) [17]. These findings are similar to the top two targets for further development of AI-SaMDs in a survey involving Korean neuroradiologists, where their most important expectations include reduction of time spent on repetitive tasks (91.8%), increased interpretation accuracy, and reduced errors (72.6%) [13].

It is important to discuss future directions in the development of AI-SaMD with three important stakeholders: industry, academic societies, and the government. 1) For industry, it is important to recognize that there is a disparity between actual use and the direction desired by users. The need for workload reduction and automated interpretation is expected to increase. With recent advances in AI technologies [18], the value of AI-SaMDs is expected to increase. 2) For academic societies, including the KSR, there is a need to provide objective and comprehensive information on use cases and performance at the societal level as this information is fragmented. This is related to the motivation for use because most purchases are administratively decided and made passively within institutions. Therefore, it is important for the KSR to provide guidelines to clarify the purpose, provide clinical use cases, and suggest appropriate purchases of AI-SaMDs for institutions. 3) Government-level support is necessary for reimbursement and legal issues. Most respondents without experience in using AI-SaMD were willing to use AI-SaMDs (44.2% under the current situation, and 52.4% if performance or circumstances are improved), and the reason why most respondents (75.5%) were not using AI-SaMD was that institutions did not purchase AI-SaMDs. Reimbursement for using AI-SaMDs is an important motivation for institutions to adopt AI-SaMDs and aid in widening AI-SaMDs’ distribution. As many respondents (45.9%) indicated, the clarification of legal liability issues associated with using AI-SaMDs was another important hurdle for clinical adoption, which corresponds with the findings of a previous survey that found ethical and legal issues and a lack of knowledge to be the most often mentioned hurdles to AI-SaMDs implementation [9]. A joint approach for clinical adoption among the three axes of the industry, KSR, and government is needed.

Our study had several limitations. First, only a small proportion (7.7%) of KSR members participated in the survey. This was higher than the response rate obtained by the European Society of Radiology survey in 2018 (2.5%) [7] and an international survey of 1041 radiologists in 2019 [8,9], but lower than that of subspecialty neuroradiologists in Korea (21.9%) [13] and emergency radiologists in the United States (11.8%) in 2022 [4]. Second, the questionnaire comprised a limited number of questions (23 questions) to increase the response rate. Therefore, in-depth investigation regarding the experiences and opinions of users may be limited. Considering the rapid evolution of AI technology and practical circumstances, we believe that a follow-up survey using a more detailed questionnaire is necessary. Third, it is likely that only a subset of members of the KSR interested in AI were motivated to respond to our survey, leading to a selection bias that limits the generalizability of the results to the radiology society. Fourth, we did not specifically define the scope of AI-SaMDs in this survey. Therefore, individual respondents may have arbitrarily judged whether they have used AI-SaMDs.

In conclusion, this society-wide survey within the KSR shows a high penetration of AI-SaMDs, with > 60% of respondents using them. Current trends in the use of AI-SaMDs include lesion detection, diagnosis/classification, and quantification, with most respondents finding them satisfactory. A joint approach across industries, radiologic societies, and governments is required to aid the integration of AI-SaMDs into diagnostic radiology. Most respondents requested KSR-driven education or guidelines on AI-SaMDs usage.

Availability of Data and Material

The datasets generated or analyzed during the study are available from the corresponding author on reasonable request.

Supplement

The Supplement is available with this article at https://doi.org/10.3348/kjr.2023.1246.