Exploring healthcare professionals’ perceptions of artificial intelligence: Piloting the Shinners Artificial Intelligence Perception tool

Abstract

Objective

There is an urgent need to prepare the healthcare workforce for the implementation of artificial intelligence (AI) into the healthcare setting. Insights into workforce perception of AI could identify potential challenges that an organisation may face when implementing this new technology. The aim of this study was to psychometrically evaluate and pilot the Shinners Artificial Intelligence Perception (SHAIP) questionnaire that is designed to explore healthcare professionals’ perceptions of AI. Instrument validation was achieved through a cross-sectional study of healthcare professionals (n = 252) from a regional health district in Australia.

Methods and Results

Exploratory factor analysis was conducted and analysis yielded a two-factor solution consisting of 10 items and explained 51.7% of the total variance. Factor one represented perceptions of ‘Professional impact of AI’ (α = .832) and Factor two represented ‘Preparedness for AI’ (α = .632). An analysis of variance indicated that ‘use of AI’ had a significant effect on healthcare professionals’ perceptions of both factors. ‘Discipline’ had a significant effect on Allied Health professionals’ perception of Factor one and low mean scale score across all disciplines suggests that all disciplines perceive that they are not prepared for AI.

Conclusions

The results of this study provide preliminary support for the SHAIP tool and a two-factor solution that measures healthcare professionals’ perceptions of AI. Further testing is needed to establish the reliability or re-modelling of Factor 2 and the overall performance of the SHAIP tool as a global instrument.

Introduction

Artificial intelligence (AI) is the next wave of digital technology innovation impacting the healthcare setting as part of the fourth industrial revolution ¹ or Health 4.0. ² In the last decade, the advancements of AI in healthcare have been widely published.^3,4 However there is limited research investigating the experiences or perceptions of the healthcare workforce. ⁵

In the past, technology implementation has provided considerable benefits to clinicians, but has also been shown to cause changes in cognitive structures, social interactions and clinical workflow efficiencies which creates unintended risk.^6–8 Qualitative studies show that healthcare professionals are enthusiastic to learn about AI and can see value in its capacity to augment their work.^9,10 However, as many healthcare professionals have had little exposure to AI, they are often concerned about the ethical implications of AI, the management of data, the disruption of the patient-physician relationship, and the development of professional knowledge.^4,9 Healthcare professionals primarily agree that AI could support the health and well-being of their patients, but suspect it will have organisational and professional impacts that they are not prepared for.^11,12 There are concerns that these perceptions could undermine the potential benefits of AI before they have been implemented, and at great cost to the organisation.

Tools that explore human-technology interaction tend to focus on the technology and serve the needs of the system designer during implementation, rather than identify the needs of the user. The Technology Acceptance Model (TAM) ¹³ is a well-known tool that has explored the relationship between a user's perception and behaviour towards technology, and their likelihood to adopt it. The Technology Readiness Index (TRI) ¹⁴ and more recently, the General Attitudes towards Artificial Intelligence Scale (GAAIS), ¹⁵ measure the characteristics of an individual, such as ‘innovativeness’, ‘optimism’, ‘discomfort’ and ‘insecurity’, that may impact technology acceptance. However, they do not explore the baseline perception that healthcare professionals hold before they have begun using AI technology. Moreover, the current tools do not identify healthcare professional's feelings of preparedness, or their ideas about its impact to their professional role.

In 2019, we designed the SHAIP questionnaire which, unlike other human-technology interaction tools, focused solely on the healthcare professional and their perception of AI. ¹⁶ The SHAIP questionnaire was the result of an Australian e-Delphi study, which gathered opinions of an interdisciplinary panel of experts in health and technology, geographically located in four Australian states. Five healthcare professionals from optometry, medicine, nursing and allied health, and three information technology experts from health technology, engineering and finance participated in the study. They brought a diverse range of experience in their disciplines, working as clinicians, business owners, academics and executive level managers. The SHAIP tool was underpinned by socio-technical systems theory^17–19 which acknowledges the complex relationships between the individual, the technology, and the workplace, and supports the belief that organisations need a human-centred understanding of workforce characteristics and perceptions when implementing new technology.^19–21 The e-Delphi design consolidated the diverse opinions of individual members of the panel into a single representative opinion. It culminated in an 11-item tool designed to capture healthcare professionals’ perceptions of AI in relation to their professional role, the extent to which they felt prepared for AI, and their views on its impact on clinical decision making and patient care. The aim of this study was to psychometrically evaluate and pilot the SHAIP questionnaire.

Methods

Procedures

A cross-sectional study was conducted in a regional health district in Australia. This district was chosen because it included primary, secondary and tertiary healthcare settings with a diverse range of healthcare and administrative professionals. Approximately 3000 employees who were over 18 years of age were invited to participate in the study. Job roles were classified as either ‘non-clinical roles’ such as administration, or ‘clinical roles’ like nursing/midwifery, medicine, allied health professionals, and other healthcare professionals. Allied health professions include speech pathology, podiatry, physiotherapy, social work and occupational therapy. A 5-point Likert scale: 1 = Strongly Disagree 5 = Strongly Agree was used to measure the participants’ perception of the 11 items.

The SHAIP questionnaire was uploaded into Qualtrics, ²² a web-based survey tool. An invitation to participate in the study was sent to the staff of a local health district in regional New South Wales, Australia, via an email which contained a link to the SHAIP questionnaire and a detailed information sheet. An announcement regarding the study was also posted on the district's intranet. The questionnaire advised potential participants that consent was implied by completing the survey. The study was conducted according to the National Statement on Ethical Conduct in Human Research (2007) ²³ and approved by the Southern Cross University Human Ethics Committee (HREC Register Number ECN-19-177).

Data analysis

Completed responses were imported into SPSS V.25 ²⁴ for analysis. Basic demographics were collected and participants were asked about their job description (including health employee, senior manager, educator), health discipline (including allied health, medicine, nursing/midwifery, other health and non-clinical) and current use of AI (yes, no or I don’t know). Exploratory factor analysis was conducted using principal axis factoring to understand how the SHAIP items group together to create a construct. The Chi square goodness of fit test, and Reproduced Correlation Matrix were evaluated to determine how well the factor represent the data. Cronbach's alpha was used to determine the reliability of each of the factors and Oblique Oblimin rotation which permit the factors to be correlated with one another, was used to interpret them. Independent variables (use of AI, discipline and job description) were compared with healthcare professionals’ measures of perception, using a one-way ANOVA.

Results

Descriptive analysis

A total of 252 healthcare professionals were recruited from August to November 2019. The 252 total responses provided more than 10 respondents per item and were therefore adequate for factor analysis (see Table 1).^11,25 More than half of the respondents were females (n = 159, 63.1%), aged between 41–60 years (n = 150, 59.6%). Participants represented ‘non-clinical roles’ (n = 24, 9.5%), and ‘clinical roles’, including nursing/midwifery (n = 141, 56%), medicine (n = 34, 13.5%), allied health professionals (n = 26, 10.3%), and other healthcare professionals (n = 27, 10.7%). Most participants were working as clinicians (n = 190, 75.4%) and held jobs that included health practitioner (n = 153, 60.7%), senior manager (n = 68, 27%) or educator (n = 31, 12.3%). Half of the respondents reported currently using AI (n = 126, 50%) (See Table 1).

Table 1.

Descriptive statistics.

Demographics n = 252		Total Sample
		n	%
Age		<20	.4
		21–30	9.1
		31–40	17.1
		41–50	28.6
		51–60	31.0
		61–70	12.7
		70 +	1.2
		Total	252
Gender	Male	90	35.7
	Female	159	63.1
	Not identified	3	1.2
Discipline	non-clinical staff	24	9.5
	nursing/midwifery	141	56.0
	medicine	34	13.5
	allied health	26	10.3
	other health	27	10.7
Clinician	Yes	190	75.4
	No	62	24.6
Job Description	Health practitioner	153	60.7
	Senior management	68	27.0
	Educator	31	12.3
Using AI	Yes	126	50.0
	No	105	41.7
	I don't know	21	8.3

Factor analysis and reliability

The Kaiser-Meyer-Olkin (KMO), a measure of sampling adequacy, was satisfactory at .804 and the result of the Chi Square for Bartlett's test (χ² = 758.567 and p < .001) suggested that patterned relationships can be found amongst the variables and the correlation matrix was satisfactory for factor analysis. The model was found to be a good fit with a Reproduced Correlation Matrix of 14%, and absolute values >.05. ¹¹ Three factors were initially extracted but the third factor had a low eigenvalue (1.1), and a scree plot suggested that a two-factor solution was preferable. Item 11 had a very low communality (.115), did not load on any factor, and was removed. A subsequent factor analysis was re-run, extracting two factors which are shown in Table 2. Each item only loaded on one factor and the correlation between the two factors was .34. On the basis of the items that loaded on each factor, Factor 1 was interpreted to represent healthcare professionals’ perceptions of the Professional impact of AI, including improvements to care and decision making, as well as direct changes to their role. Factor 2 was interpreted to represent healthcare professionals’ perceptions of their Preparedness for AI, including adequate training, knowledge of ethical frameworks and readiness for its introduction. The final two factors accounted for 51.74% of the total variance. A cut-off for statistical significance of the factor loadings of 0.5 was considered, ¹² however based on a sample size of 252, it was decided that factor loading would need to be at least .32 to be considered statistically meaningful. ²⁶ Item 10 fell into the lower range (.362) and also had a low communality (.266): ‘I believe that should AI technology make an error; full responsibility lies with the healthcare professional’ . This item appears to have a legal or ethical angle which although an important indication of whether a healthcare professional has prepared themselves for the outcome of an AI-driven decision in healthcare, is a weaker item.

Table 2.

Factor loading of items in the SHAIP tool.

AI-IQ Items	Factor 1 Perception of professional impact	Factor 2 Perception of Preparedness for AI	Mean	SD
Item 1. I believe that the use of AI in my specialty could improve the delivery of patient care	.765		3.38	1.039
Item 2. I believe that the use of AI in my specialty could improve clinical decision making	.770		3.44	1.026
Item 3. I believe that AI can improve population health outcomes	.702		3.66	.866
Item 4. I believe that AI will change my role as a healthcare professional in the future	.655		3.74	.970
Item 5. I believe that the introduction of AI will reduce financial cost associated with my role	.452		2.99	.890
Item 6. I believe that overall healthcare professionals are prepared for the introduction of AI technology		.579	2.32	1.031
Item 7. I believe that one day AI may take over part of my role as a healthcare professional	.428		3.00	1.097
Item 8. I believe that I have been adequately trained to use AI that is specific to my role.		.576	2.25	1.058
Item 9. I believe there is an ethical framework in place for the use of AI technology in my workplace		.672	2.66	.903
Item 10. I believe that should AI technology make an error; full responsibility lies with the healthcare professional		.362	2.79	1.138
Total variance %	35.821	15.925
Mean of mean Scale Score	3.37	2.67

The individual raw scores for items 1 to 6 and 7 to 10 for each of the 252 participants were averaged and the overall mean scale scores were obtained for Professional impact of AI (3.66) and Preparedness for AI (2.55). In a preliminary analysis we examined the Cronbach's alpha estimate of internal consistency of each factor. Support for the 2-factor model required an alpha score = 0.70 or above. ²⁷ The reliability estimate of Factor 1 Professional impact of AI was a Cronbach's alpha of .804 which suggests that items had very little variance specific to individual items. Factor 2 Preparedness for AI had a Cronbach's alpha score of .620 which is marginally less than our target.

Analysis of variance

A factorial between group analysis of variance (ANOVA) was used to investigate the effects of demographic variables such as age, use of AI, discipline and job description on perceptions of Professional impact of AI and Preparedness for AI.

Age:

There was no significant main effect of Age on perceptions of Professional impact of AI, F(6, 245 = .730, p = .636), or Preparedness for AI F(6, 245 = .739, p = .619). The majority of the healthcare professionals were over 40, however this is likely an accurate representation of the current healthcare professional demography.

Use of AI:

There was a significant main effect of healthcare professionals’ Use of AI (yes/no/I don’t know) on perception of Professional impact of AI, F(2, 249 = 10.45, P < .001). A Tukey's test of multiple comparisons (P < .05) showed that the mean scale score for ‘Yes’ was significantly higher than the mean scale score for the other two categories: ‘No’ and ‘I don’t know’ (see Table 3). There was also a significant main effect of Use of AI on perception of Preparedness for AI F(2, 249 = 8.78, P < .001). The Tukey test showed that the mean scale score for ‘Yes’ was significantly higher than the mean scale score for the ‘No’ category. These results suggest that healthcare professionals who use AI were unsure or agreed (mean scale score = 3.55) that AI will change their role as a healthcare professional and that on average (mean scale score = 2.82) disagree that they are prepared for its use in the delivery of care than those that are not using it.

Table 3.

SHAIP tool subscale and total summary scores [mean scale score (SD)].

Subscale Total (n = 252)	Professional impact of AI	p-value	Preparedness for AI	p-value
Age
<20 (n = 1)	4.5		2.25
21–30 (n = 23)	3.38 (1.04)		2.41 (0.82)
31–40 (n = 43)	3.46 (0.78)		2.48 (0.69)
41–50 (n = 72)	3.28 (0.65)		2.44 (0.77)
51–60 (n = 78)	3.36 (0.63)		2.52 (0.65)
61–70 (n = 32)	3.36 (0.57)		2.69 (0.62)
70 + (n = 3)	3.50 (0.44)		2.58 (0.52
Use of AI
Yes (n = 126)	3.55 (0.61)*	<.001	2.82 (0.66)*	<.001
No (n = 105)	3.21 (0.74)		2.53 (0.55)
I don’t know (n = 21)	3.01 (0.69)		2.50 (0.62)
Discipline
Non-clinical staff (n = 24)	3.25 (0.67)		2.92 (0.67)
Nursing/midwifery (n = 141)	3.36 (0.68)		2.69 (0.62)
Medicine (n = 34)	3.59 (0.68)		2.59 (0.55)
Allied Health (n = 26)	2.94 (0.82) *	<.002	2.38 (0.60)
Other Health (n = 27)	3.60 (0.55)		2.77 (0.64)
Job Description
Health Practitioner (n = 153)	3.30 (0.69)		2.70 (0.60)
Senior Management (n = 68)	3.40 (0.77)		2.70 (0.67)
Educator (n = 31)	3.56 (0.54)		2.52 (0.64)

*Indicates a p value of <0.05. This item is significantly different from the other items in its category.

Discipline:

There was a significant main effect of Discipline (nursing/midwifery/allied health/medicine/other health/non-clinical) on perceptions of Professional impact of AI, F(4, 247 = 4.41, P < .002). A Tukey's test of multiple comparisons (P < .05) showed that allied health mean scale score was significantly lower than all other disciplines (see Table 3). These results suggest that allied health professionals on average disagree or are unsure (mean scale score = 2.94) whether AI will impact their role compared to other healthcare professionals. Allied health professionals’ representation (n = 26, 10.3%) appears underpowered, compared to nurses/midwifery (n = 141, 56%), medicine (n = 34, 13.5%), and other healthcare professionals (n = 27, 10.7%). although may reflect the ratio of allied health professionals working in the acute care settings. There was no significant main effect of Discipline on perceptions of Preparedness for AI F(4, 247 = .552, p = .580). Low mean scale score across all disciplines suggests that all disciplines perceive that they are not prepared for AI.

Job description:

There was no significant main effect for Job Descriptions on perceptions of Professional impact of AI F(2, 249 = .953, p = .146) or Preparedness for AI F(2, 249 = 1.763, p = .040) (see Table 3). Mean scale scores across all disciplines were close to the neutral point of 3 (unsure) so a one-sample t-test was conducted. The t-test found that overall Professional impact of AI t(251) = 8.307, p < .001 and Factor 2 ‘Preparedness for AI’ t(251) = 8.750, p < .001, participants tend to be more likely to respond positively to items about the professional impact of AI on job description and tend to perceive themselves as less than prepared for AI than would be expected if they were responding as ’unsure’.

Discussion

Findings support the structure and reliability of a 10-item SHAIP questionnaire. Patterned relationships were found amongst the items (KMO = .804 and Chi Square for Bartlett's test (χ² = 758.567 and p < .001), and the correlation matrix was satisfactory for factor analysis. The model was found to be a good fit when Item 11 was removed due to a low communality (.115), because only a small proportion of its variance could be explained by the factors. The 10 items used in final analysis, loaded on to 2 factors: Factor 1 Professional impact of AI and Factor 2 Preparedness for AI. The reliability estimate of Factor 1 (α = .804) conformed to Cronbach's definition of equivalence ²⁸ and items had very little variance specific to individual items. Factor 2 Preparedness for AI (α = .620) did not meet this target suggesting it is multidimensional in nature and may require other items that have not been elucidated by the original e-Delphi study. A review of this factor and how it can incorporate more global views is recommended to improve its application.

The SHAIP tool is different than other human-technology interaction tools, in that it acts as a baseline measurement of healthcare professionals’ perception of AI in relation to its ‘impact to their professional role’, and their ‘preparedness’. The GAAIS tool, ¹⁵ published in 2020 after this study was conducted, is not specific to healthcare but found that ‘comfortableness’ was a strong predictor of attitude towards AI. The authors found that AI applications related to health were amongst the lowest rated for both ‘comfortableness’ and ‘perceived capability’, and that a focus on the complex domain of healthcare was needed. It is suggested in the literature that healthcare professionals and possibly healthcare organisations, are not clear on what AI is. ²⁹ That being true, if a healthcare professional does not understand what AI is or how it is applied to the delivery of care, they are not able to associate a negative or positive perception to these questions. The SHAIP tool is a conduit for organisations to begin a conversation about AI.

While we offer some conceptual insights into healthcare professionals’ perceptions of AI, some study limitations are acknowledged. First, the initial e-Delphi framing was conducted by a panel of Australian interdisciplinary experts in health and technology which may limit the tool's use as a global instrument. Secondly, while the study analysed 252 complete surveys, this only represents a small proportion of the population of healthcare professionals working in the regional NSW local health district. It is unclear if socio-economic factors such as the geographical location of this survey impacted the results and whether there may be a greater exposure to AI in metropolitan healthcare settings, which is a consideration for further work. The study is also at high risk of non-response bias, which may impact the generalisability of the findings.

Conceptual insights

There was no significant main effect of Age on perceptions of Professional impact of AI or Preparedness for AI, but the study found that ‘use of AI’ did influence the perception of both factors. Due to the neutral framing of the items, where a perception would ordinarily be expected to be highly divergent based on age, profession or experience, the tool instead measures the magnitude with which healthcare professionals perceive a statement to be true. Interestingly, healthcare professionals who used AI strongly agreed that it would impact their role, but also felt more prepared for its use than healthcare professionals who were not using AI. Technology use has been shown to improve an individual's perception of competence, allowing them to see the merits of the technology more easily.^30–33 However, without an understanding of how it will enhance performance and improve care delivery, technology use may fail to reveal its benefits and lead to misuse or abandonment by the workforce. ³⁴ Sheng et al. ³⁵ found that physicians who were given hands-on experience of new telemedicine technology before its implementation and were supported with training, were able to single out the potential value of the technology in respect to their role and expressed a great interest in learning more about it. This result echoes the call for inclusive design, ³⁶ workforce training and development, and an ethical framework for artificially intelligent healthcare systems to realise the potential value this technology could offer.

‘Discipline’ had a significant main effect on Allied healthcare professionals’ perceptions of Factor 1, strongly disagreeing that AI would have a professional impact on them, which differed from other health disciplines and warrants further exploration. The results showed that the study was heavily skewed in favour of nurses, and the representation of doctors as a key user of AI technology in the future was also low. The scope and versatility of AI is expected to impact all stakeholders in the healthcare workforce^6,37 and the implications for AI technology implementation at a personal and organisational level may be profound. In Australia, technologies such as telehealth have been met by resistance from some members of the allied health community. ⁷ An Australian government survey exploring allied health professionals’ preparedness for e-Health technology found that the perception of its benefits was mixed. There was inequity across the allied health network to accommodate the technology and a lack of understanding about its risks and benefits that resulted in isolated technology adoption. ⁸ The results of this pilot suggests that some healthcare professionals in Australia are unaware of the nature and extent of AI and its planned applications in healthcare. A coordinated effort will be required from professional bodies to include, educate and orient the workforce to this type of technology, and at the same time develop a framework that addresses the human-centred challenges and diverse needs of the healthcare community.^8,38,39 This framework should prioritise an inclusive design element to any new AI technology application, encourage an interdisciplinary approach to implementation and ensure that education is initiated early and is targeted to the needs of the discipline.

The promise that AI innovation will improve care delivery and reduce workload is attractive to organisations but there is doubt amongst healthcare organisations that the implementation of technology such as AI is viable, because of the investment that is required for infrastructure, training and policy development.^40–42 The socio-technical systems theory recommends that the individual, the organisation and the technology are given equal consideration before the introduction of new technology. ⁴³ To help organisations maximise the benefit of this new technology, it is important to explore the experiences and perceptions of the healthcare workforce, so that barriers to implementing the technology can be identified, and training and education can be targeted to their needs.^44,45

Conclusions

The results of this study provide preliminary support for the SHAIP tool and a two-factor solution that measures healthcare professionals’ perceptions of AI. Further testing will establish the reliability or re-modelling of Factor 2 specifically. The tool is designed to measure healthcare professionals’ perceptions of AI, as a baseline for organisations.