A Sociotechnical Systems Framework for the Application of Artificial Intelligence in Health Care Delivery

Abstract

In the coming years, artificial intelligence (AI) will pervade almost every aspect of the health care delivery system. AI has the potential to improve patient safety (e.g. diagnostic accuracy) as well as reduce the burden on clinicians (e.g. documentation-related workload); however, these benefits are yet to be realized. AI is only one element of a larger sociotechnical system that needs to be considered for effective AI application. In this paper, we describe the current challenges of integrating AI into clinical care and propose a sociotechnical systems (STS) approach for AI design and implementation. We demonstrate the importance of an STS approach through a case study on the design and implementation of a clinical decision support (CDS). In order for AI to reach its potential, the entire work system as well as clinical workflow must be systematically considered throughout the design of AI technology.

1. Introduction

AI will touch and change many aspects of the $4.2 trillion dollar health care industry. AI applications and methods in health care delivery rely on a range of technologies, such as machine learning, natural language processing and predictive analytics (Matheny et al., 2019). Sanders and colleagues (2019) described various applications of AI in health care delivery, such as clinical decision support technologies for ordering processes, speech-recognition software for clinical documentation, and algorithms for enhancing imaging or pathology review and diagnosis of cancer and eye diseases. A 2019 report by the National Academy of Medicine (NAM) highlights “The Hope, the Hype, the Promise, the Peril” of AI in health care (Matheny et al., 2019), and describes the challenges of implementing AI in actual clinical settings. This reflects an increasing recognition of the human and organizational dimensions of AI, and the potential lack of fit of AI within the sociotechnical system of health care delivery.

1.1. Potential benefits of AI in health care delivery

There are vast opportunities for AI to transform health care delivery. For instance, AI-based clinical decision support (CDS) technologies have been developed for the diagnosis of dermatological diseases (Pangti et al., 2021) and diabetic retinopathy (Ruamviboonsuk et al., 2019). AI-based CDS technologies have the potential to enhance the cognitive support provided to clinicians by creating more sensitive and specific alerts that automatically integrate most up-to-date clinical knowledge (Matheny et al., 2019). AI will also change the way patients, caregivers, and clinicians interact. For instance, AI-based-symptom checkers could provide information to patients about their symptoms, potentially improving the early detection of health problems and reducing delays in diagnosis. A survey of users of an online AI-assisted symptom checker, the Isabel Symptom Checker, shows that survey respondents found the AI technology useful to find information on their health problem (Meyer et al., 2020).

1.2. Challenges of AI in health care delivery

With all of the hope and hype of AI, there have also been numerous challenges with the implementation and use of AI, in particular its application to health care delivery as described in the 2019 NAM report (Matheny et al., 2019); Matheny et al. (Matheny et al., 2019) outline explainability and interpretability, oversight and regulation, workforce education (Garvey et al., 2021), problem prioritization, data quality and access, workflow integration, and clinician and patient engagement as challenges to the effective use of AI. For instance, in the aforementioned study on an AI-assisted symptom checker (Meyer et al., 2020), patients found mixed reactions when discussing the results with physicians who were not often interested in the results from the AI symptom checker. This highlights the potential challenges of integrating AI technologies in the patient-clinician workflow.

Arbabshirani et al. (2018) developed and implemented a machine learning algorithm to analyze CT scans for signs of intracranial hemorrhage. The algorithm demonstrated high accuracy (Area Under the Curve: 0.846), and when implemented, resulted in 96% faster diagnosis of 5 intracranial hemorrhage cases, while median time to diagnosis decreased from 512 to 19 minutes. The researchers stated that there was “negligible added burden of the algorithm on radiology worklist prioritization”; however, they did not specifically study the impact of the technology on radiologists’ workflow or gather feedback from front-line clinicians using the technology. The algorithm bypassed clinical workflow as the AI technology re-ordered scan reading without the knowledge of the radiologist. The fact that the AI technology bypassed clinical workflow does not necessarily mean that the technology did not have an impact or “added burden” on clinical workflow; further investigation is needed to understand the potential positive and negative impacts of the AI technology on clinical workflows. Rather than fully autonomous AI, the future of AI is more likely to contain systems that combine the strengths of humans and AI collaboratively working together (Shneiderman, 2020). For instance, Patel et al. (2019) demonstrated that a radiologist and AI working together resulted in greater diagnostic accuracy compared to either system working alone. However, little is known about how to optimally integrate AI as a collaborative partner in clinical workflow (Patel et al., 2019). As described in section 3, AI developers and implementers should carefully consider how to integrate AI technology in clinical workflow (Zimlichman & Bates, 2021).

1.2.1. AI workflow integration challenges – An example of diabetic eye disease diagnosis

A human-centered evaluation of an AI technology aimed at diagnosing diabetic eye disease in a hospital in Thailand provides further insights on these challenges, in particular those related to the integration of AI technology in clinical workflows and the broader sociotechnical system (Beede et al., 2020) . A team at Google Health developed a deep learning algorithm for assessing diabetic retinopathy. A retrospective study demonstrated high accuracy (both specificity and sensitivity) of the algorithm (Ruamviboonsuk et al., 2019). However, when the AI technology was implemented in the clinical environment, numerous challenges emerged.

Using multiple methods of observation and interviews, a group of researchers evaluated the implementation of the deep learning algorithm in several clinics in Thailand (Beede et al., 2020). A comparison of the clinical workflow before and after the implementation of the algorithm showed that patients followed the same general journey at the clinic, except for receiving an immediate determination for the potential need for a referral to a specialist. Several problems arose as the AI technology was used in the eye screening process. The deep learning algorithm needs very high-quality images of the eye; however, this was not consistently achieved as photos were sometimes taken in non-darkened environments or with malfunctioning cameras. This resulted in about 20% of images that did not meet the high-quality standards for the deep learning algorithm. In addition to the technical challenges, nurses experienced frustration as images that they felt were readable were actually rejected by the AI technology. This created additional work for nurses and frustration for both patients and nurses. Problems with internet connection hindered the clinical workflow as images would take long to upload to the cloud; this limited the use of the algorithm.

This case study of the implementation of a deep learning algorithm to screen patients for eye disease clearly demonstrates the importance of understanding the context or work system in which the technology is implemented. Contextual factors of importance in this case study include the environment (e.g. lighting), and other technologies (e.g. internet connection). Contextual (work system) factors are important to consider in the early phase of technology development. Coiera (2019) described three stages of the development of AI technologies:

• stage 1 – acquisition, cleaning and labeling of data;

• stage 2 – developing and testing the AI technology; and

• stage 3 – real-world implementation.

Coiera argued for focusing on the ‘last mile’ challenges or the issues that arise when the AI technology is implemented in the clinical environment (stage 3). So-called ‘last mile challenges’ are actually deeply related to decisions made in early stages of the technology development (stages 1 and 2); therefore, our effort should be expanded to consider sociotechnical issues at all stages of AI technology development. In this paper, we propose a sociotechnical systems framework that examines contextual factors of importance for AI applications in health care delivery.

1.3. Need for a sociotechnical systems approach to AI technology for health care delivery

The emphasis on developing innovative, reliable and accurate AI technologies needs to be balanced with a strong consideration for how the technology will actually be used in the clinical environment. This is the essence of the sociotechnical systems approach that we propose. In particular, we need to pay attention to the integration of the AI technology in the clinical environment. This requires a deep understanding of the context in which the technology will be used, which allows technology designers and implementers to anticipate problems of integration of the AI technology with the temporal workflow of clinical care.

Workflow integration has been identified as a major challenge to the application of AI in clinical settings by the 2019 NAM report (Matheny et al., 2019). The report highlights the need to “understand the technical, cognitive, social, and political factors in play and incentives impacting integration of AI into health care workflows” (page 11). Optimal workflow integration of AI in health care delivery needs to go beyond the integration of AI with the EHR technology and consider other work system elements (see our conceptual approach in figure 2). In order for AI to improve health care processes and patient outcomes, there is a need to design AI technologies to fit the workflow of users and clinical context; to do this, a sociotechnical systems (STS) approach is needed.

2. Sociotechnical systems approach

2.1. Sociotechnical systems and the work system model

A sociotechnical systems approach means acknowledging that the social and technical systems of work are interrelated and cannot be decoupled (Trist, 1981; Trist & Bamforth, 1951). The Work System model (Carayon, 2009; Smith & Carayon-Sainfort, 1989) is one sociotechnical model that depicts the 5 system elements: people, tasks, tools and technologies, the physical environment and organization (see figure 1). Workflow is the interaction of these work system elements unfolding over time (Carayon et al., 2012) as people, complete tasks, using tools and technologies, in a physical environment and organizational context.

Figure 1:

Model of the Work System (Carayon, 2009; Smith & Carayon-Sainfort, 1989)

2.2. AI as a part of the broader work system

AI is only one element of a much larger sociotechnical (work) system. When AI technology is implemented, it affects the entire work system including altering the interactions among work system elements; this is depicted in figure 2. These new system interactions affect workflow integration, defined as “The technology is seamlessly incorporated within the work system elements (i.e. people, tasks, other technologies/tools, physical environment, organization) and their interactions over time (i.e. process), specifically considering the temporal order in which work is accomplished and the point in time in which the technology will be used” (Salwei, 2020; Salwei et al., 2021).

Figure 2:

AI implementation into the work system (Adapted from Salwei et al. (2021))

If the technology is integrated in the workflow, positive outcomes (e.g. acceptance, use) will occur, whereas a lack of integrating the technology in the clinical workflow can lead to negative outcomes (e.g. frustration, increased workload). Therefore, AI must be designed to fit with the system elements to support and not hinder clinical workflow. Integrating AI in clinical workflow requires systematic consideration of the work system elements as well as the multiple dimensions and layers of workflow integration (Salwei, 2020; Salwei et al., 2021).

Current AI research mainly focuses on development of the AI technology with minimal focus on the other work system elements and the real environment of use; this has resulted in challenges when these technologies are implemented (Beede et al., 2020; Matheny et al., 2019; Meyer et al., 2020). Often, AI is developed thinking of ideal circumstances, without consideration of the real context in which the AI will be used. For instance, will the necessary data be available in the real environment? Will the data be “clean” enough for the AI algorithm to work? Will the lighting be adequate to take high-quality images to be analyzed? The work system where the technology will be used is essential to consider early on in the design of AI.

The temporal nature of work, i.e. workflow, is another essential facet to consider in the design and implementation of AI. Previous studies implementing health IT have faced challenges because of a lack of integration of the technology in clinical workflows (Khan et al., 2019; Lesselroth et al., 2011). However, the concept of workflow integration for AI is poorly understood and few studies developing AI have considered how the technology will fit within clinical workflows (Celi et al., 2019).

In addition to fit with the work system and workflow, AI presents unique challenges beyond those of other health IT. For instance, issues of human-technology handoffs and situation awareness (Sujan, 2021; Sujan et al., 2020) as well as autonomy and automation (Shneiderman, 2020) are all exacerbated by the use of AI technologies (Sujan et al., 2020). We need to leverage prior research on health IT, such as CDS, and automation (Benda et al., 2022) to inform the successful integration of AI in healthcare.

While AI has the ability to support and improve many health care processes, its success ultimately relies on patients and clinicians accepting these technologies and their recommendations (Celi et al., 2019). This is especially true in the context of clinical decision-making. When a technology and its recommendations are not transparently communicated, this may result in a lack of trust and reluctance to use the technology. The 2019 NAM report described AI explainability and interpretability as 1 of 7 practical challenges of advancing AI in health care delivery, stating “To promote integration of AI into health care workflows, consider what needs to be explained and approaches for ensuring understanding by all members of the health care team” (page 11) (Matheny et al., 2019). For AI technology to support clinical decision-making, the AI must be transparent to support trust in its recommendations.

In summary, we propose 3 key sociotechnical considerations for integrating AI in the healthcare delivery system:

1. Does the AI technology integrate within the work system, i.e. with all of the system elements, people, tasks, other tools/technologies, physical environment, and organizational context?

2. Does the AI technology integrate in the clinical workflow, i.e. the temporal nature of work?

3. Does the AI technology support the decision-making process, and promote transparency and clinician trust of the technology?

AI technology has the potential to support human decision making for diagnosing, treating and monitoring patients, and extend the capabilities of clinicians (Israni & Verghese, 2019; Shneiderman, 2020). This requires a human-centered approach where the technology is designed to augment and support the work of clinicians (Endsley, 2017; Shneiderman, 2020; Vasey et al., 2021). Below we describe a case study of a CDS implemented in the emergency department (ED) to emphasize the importance of an STS approach for AI development.

3. Case study: the PE Dx CDS

CDS is one type of health information technology that can be significantly impacted by AI (Petersen et al., 2021; Sanders et al., 2019; Shortliffe & Sepúlveda, 2018). According to the US Office of the National Coordinator for Health Information Technology (Office of the National Coordinator for Health Information Technology, 2021), “Clinical decision support (CDS) provides clinicians, staff, patients or other individuals with knowledge and person-specific information, intelligently filtered or presented at appropriate times, to enhance health and health care.”. AI technologies, such as machine learning and natural language processing, offer new capabilities and can potentially enhance CDS; but they need to be designed to fit the complex, time-constrained clinical environment (Shortliffe & Sepúlveda, 2018).

Here, we use a case study of a CDS, i.e., PE Dx, to illustrate the importance of our 3 STS considerations outlined above. Before diving into the case study, it is important to clarify and highlight the similarities and differences between the PE Dx and AI-based technologies. AI has broadly been defined as computer methods designed to mimic, emulate, and/or improve human reasoning capabilities and decision making (Scott et al., 2019; Sloane & Silva, 2020). AI can range from simple rule-based systems, which aim to replicate interpretation and decision making of experts (Sloane & Silva, 2020), to advanced machine learning and deep learning algorithms, which can identify (and learn) patterns in large datasets (Scott et al., 2019). PE Dx is on the former end of the spectrum as a computerized tool that calculates and communicates risk information, without the use of advanced statistical methods, in order to improve human decision making. Table 1 details the similarities and differences between the PE Dx and advanced forms of AI technologies. As much of AI in health care is used to support clinical decision making (Matheny et al., 2019; Sujan, 2021; Zimlichman & Bates, 2021), our CDS offers a good example to identify STS issues that may arise with the implementation of AI-based CDS.

Table 1:

Similarities and differences between PE Dx and AI technologies

Similarities of PE Dx and AI

Differences between PE Dx and AI

Both are used to communicate assessments of risk Both are used to support the diagnostic process Both are based on information from specific populations (e.g., a large dataset or a population of patients in a randomized controlled trial) Specific information on the populations used to inform the risk calculation is not provided Clinicians need to determine when the risk information applies/does not apply to their specific patient (or blindly trust the system recommendation) Both use automation to acquire information, e.g., AI method of natural language processing compared to PE Dx auto-populating numerical information

PE Dx does not use machine learning or advanced statistical methodologies PE Dx has higher visibility (e.g., the point values on specific criteria selected) on where the risk score came from (e.g., based on elevated heart rate and age). This is more similar to traditional logistical regression compared to potentially ambiguous machine learning risk predictions.

We designed the CDS to support pulmonary embolism (PE) diagnosis (Dx) in the ED (Hoonakker et al., 2019) as a part of a larger study on prevention and management of venous thromboembolism (VTE) (https://cqpi.wisc.edu/research/health-care-and-patient-safety-seips/vte-and-health-it/#project-home). The CDS, i.e. PE Dx, combined two existing risk scoring algorithms recommended by the American College of Physicians (Raja et al., 2015) to support diagnosis of PE: (1) the Wells’ score (Wells et al., 2001) and (2) the PERC rule (Kline et al., 2010). We integrated the PE Dx in the electronic health record (EHR). ED physicians use the PE Dx by entering patient specific information (e.g. previous history of PE) into the CDS, which then generates a risk score of the patient’s risk for PE. The PE Dx then provides a recommendation on the appropriate next step based on the patient’s risk (e.g. blood test to further rule out PE, CT scan), and supports the physician in taking the recommended action (e.g. placing an order) and documenting their diagnostic pathway decision. A multidisciplinary team developed the PE Dx following the human-centered design process (International Organization for Standardization, 2010). We systemically integrated human factors (HF) design principles (i.e., consistency, error prevention) in the PE Dx throughout the design process (Carayon et al., 2019).

The CDS demonstrated high usability and acceptance in an experimental setting (Carayon et al., 2019). We implemented the PE Dx in one ED at a large academic health system in December 2018. The implementation process followed macroergonomic principles for change management (Karsh, 2004; Smith & Carayon, 1995) including top management commitment, stakeholder participation, and learning and training. However, we found limited acceptance and use of the CDS when implemented, and identified several challenges of integrating the CDS into clinical workflow (Salwei, 2020). Here, we describe examples of how the CDS did, and did not, integrate into the sociotechnical (work) system based on interviews with 12 ED physicians conducted 10 months after the implementation of PE Dx; these examples shed light on the three key sociotechnical considerations for the design and implementation of AI in health care (see above). Additional information on the interview data collection methods can be found in Salwei (2021).

STS Consideration 1: Fit with work system elements

When we implemented the PE Dx, we identified a barrier relating to a misfit of the technology with the work system elements. The PE Dx was embedded in the EHR and was therefore designed to be used on a computer. Before our CDS was implemented, physicians would look up risk scores on their phone after consulting a patient. When we implemented the CDS, we found a barrier to physician workflow as physicians preferred to calculate a patient’s risk for pulmonary embolism (PE) as they walked between patient rooms. Because our CDS (and the EHR) could only be used on a computer, the physician could not use it while they moved through the physical environment. This example demonstrates the interactions between the tasks, tools/technologies, and physical environment; these system elements and their interactions are shown in figure 3.

Figure 3:

Lack of CDS integration in work system as physicians move through the physical environment

Alternatively, we found the PE Dx fit within the work system as it supported the workflow of attendings and residents working together to determine the appropriate diagnostic pathway for a patient suspected of having a PE. Several physicians explained that after talking with a patient in the ED, residents meet with attendings and the two physicians work together to make a decision on next steps for the patient. Rather than simply discussing the patient case and next steps, the PE Dx provided a structure to the conversation; residents and attendings viewed the PE Dx in the EHR together to systematically consider all the patient risk factors (e.g., prior DVT, age over 50). We found that our CDS helped to support these conversations and confirmed the clinicians’ gestalt or “unstructured and empirical clinical assessment” (Lucassen et al., 2011) on what test to order for the patient. Attending physicians explained that use of the CDS helped train residents on the important information and risk factors to pay attention to for suspected PE. Detailed in figure 4, this represents interactions between the work system elements of person, tools and technologies, tasks, and organization. We found our CDS fit within the interactions of these work system elements.

Figure 4:

CDS integration in work system for collaborative decision making between attending and resident physicians

Figures 3-6 illustrate the system interactions for each example. The 5 elements of the Work System Model (see Figure 1) are depicted in boxes. Within each box, we listed the specific elements at play. For example, in Figure 3, one relevant task is “seeing patients in the ED”. As in the original Work System Model (see Figure 1), the arrows between the boxes represent the interactions between the system elements; the arrows are bi-directional, as the elements interact simultaneously as people perform their work over time, i.e., workflow.

Figure 6:

Lack of CDS integration in work system for physician decision-making process

STS Consideration 2: fit with the temporal nature of work

We identified a barrier to using the PE Dx relating to the temporal flow of work or workflow. We designed the PE Dx to be located in a section of the EHR called the “ED Navigator”; physicians used the ED navigator for various tasks including checking the patient history and vitals, placing orders, and writing notes. We embedded the PE Dx within this sequence of tasks so that physicians would work their way down the ED Navigator as they reviewed the patient case, evaluated their risk of PE using the PE Dx, and then placed orders and wrote notes. After the implementation of the PE Dx, the EHR structure was updated, which made the ED Navigator less prominent in the ED workflow. After this upgrade, PE Dx was not embedded in physician workflow since physicians no longer went to the ED navigator to complete most of their EHR tasks. This example demonstrates the importance of a technology fit (or lack of fit) within clinician workflow.

Following the upgrade to the EHR, we found the PE Dx best fit physician workflow when physicians were working in the triage area of the ED, called “CareStart”. While in CareStart, physicians still regularly used the ED navigator, making the PE Dx better fit in their workflow. This relates to the tasks, tools/technologies, and organization (see figure 5).

Figure 5:

CDS integration in work system for triaging patients early in the ED workflow

STS Consideration 3: Transparency and trust

We identified barriers to using our CDS relating to transparency and trust. In the evaluation of PE Dx, two physicians indicated their preference for another CDS, called MDCalc, over PE Dx. Physicians preferred MDCalc because it provides evidence on the patient populations used to inform the risk scoring for PE and contains links to the literature supporting the CDS assessment of risk. Our PE Dx did not contain this information, which hindered physician trust in the CDS and its recommendation. This challenge highlights interactions between the person, tasks, tools/technologies, and the organization work system elements (figure 6).

In the design of PE Dx, we made the decision to show the point values for each of the criteria in the Wells’ score (Carayon et al., 2019; Hoonakker et al., 2019); for instance, if a patient had a heart rate greater than 100, this added 1.5 points to their overall risk score for PE, whereas if they had clinical signs of DVT, this added 3 points to their overall risk score. By adding the point values for each criterion, we enhanced the transparency in the technology and in the patient’s final risk score. In interviews with physicians about PE Dx, both resident and attending physicians stated that the point values listed on the criteria supported their decision making process (Salwei et al., 2021). This relates to the person, tasks, and tools/technologies elements.

Compared to traditional CDS systems, such as PE Dx, AI presents greater challenges regarding transparency and trust due to the inherent, ambiguous “black box” nature of the technology (Challen et al., 2019; Macrae 2019). Designers of AI technology need to integrate the principle of transparency in their AI-based recommendations in the context of the sociotechnical work system in order to support clinical decision-making. For instance, a “machine learning label”, which depicts key information on machine learning algorithms such as validation and performance metrics, the target population, and training data, may be one approach to ensure more easily accessible and interpretable information on AI algorithm design and limitations at the point of care (Sendak et al., 2020).

4. Recommendations

Our work demonstrates the necessity of considering STS issues early in the design of AI. After reflecting on the barriers identified above, we developed two high-level recommendations to improve future AI integration in sociotechnical systems.

4.1. Need for collaborative design of AI

The design of AI needs to follow a collaborative design process (Détienne, 2006) that includes a deep understanding of the sociotechnical (work) system where the technology will be used. As demonstrated by our case study as well as other studies (Beede et al., 2020; Okolo et al., 2021), ‘last mile challenges’ (Coiera, 2019) of implementing AI into clinical practice are actually deeply related to decisions made in early stages of the technology development, and therefore human factors considerations need to be addressed early in the design process. Usability of the AI technology is critical, but not sufficient; we need to consider the entire sociotechnical work system and the temporal nature of work (i.e. workflow) so that the AI technology provides cognitive support to clinicians. Sociotechnical considerations are particularly important to address early in the design process (Vasey et al., 2021); they should not be an afterthought as this is not a last mile problem.

Sociotechnical considerations (see Table 2) should be part of an iterative and continuous process of AI design (Sujan et al., 2019; Vasey et al., 2021). Human factors experts should be involved at the outset of AI development to support a collaborative design process and ensure the technology will integrate within the intended sociotechnical system. This human-centered design process should include an in-depth analysis of the needs and work system of the intended users to determine the true problem that needs solving with the AI technology (Melles et al., 2021). That is to say, just because an AI solution is possible does not mean an AI solution is needed. A truly collaborative human-centered design process is needed to ensure we are leveraging the advanced capabilities of AI to address the right problems in the right way in the context of the sociotechnical work system.

Table 2:

Proposed STS considerations and examples from PE Dx implementation

STS considerations	Description	Examples
1	Does the AI technology integrate within the work system, i.e. with all of the system elements, people, tasks, other tools/technologies, physical environment, and organizational context?	PE Dx cannot be used on the phone which hindered physician tasks within the physical environment
2	Does the AI technology integrate in the clinical workflow (i.e. the temporal nature of work) according to the four dimensions of (1) time, (2) flow, (3) scope of patient journey, and (4) level?	Location of PE Dx in the EHR did not fit within the sequence of physician tasks
3	Does the AI technology support the decision-making process, and promote transparency and clinician trust of the technology?	Displaying point values on PE Dx criteria supported transparency

Because sociotechnical (work) systems are complex and exhibit emergent properties, the effective integration of AI technology in the clinical workflow requires an iterative human-centered design approach of learning early and quickly (Carayon & Salwei, 2021). Salwei et al. (2021) outline 4 specific dimensions of workflow integration of health IT: (1) Time (the technology is integrated with the temporal nature of work, e.g., sequentially, discontinuously), (2) Flow (the technology is integrated in the flow of tasks, people, information, and other tools/technologies), (3) Scope of patient journey (the part of the process in which the technology is integrated within, e.g., the patient interaction with a clinician), and (4) Level (the technology is integrated into the different levels of workflow including individual, team, and organizational workflows). When implementing AI technologies, it is important to consider how the technology, will or will not, integrate in the workflow according to these 4 dimensions. Designers of AI can utilize the conceptual framework and checklist of workflow integration proposed by Salwei et al. (2021) to support consideration of the sociotechnical system in design.

4.2. Need to build a learning health system for continuous STS design

It is critical to implement a collaborative design process for developing AI that will integrate in the entire sociotechnical (work) system as well as the temporal workflow. However, it is inevitable that issues will occur when the technology is implemented that were not anticipated. This emergence of unanticipated issues is a key principle of sociotechnical systems (Wilson, 2014). Acknowledging that unanticipated problems will occur, we need to build a system in order to identify and learn from these challenges and adapt the technology as problems arise. A continuous design process enables identification and response to issues that arise from using the technology within the sociotechnical system (Carayon, 2006, 2019; Carayon et al., 2008). For instance, as described under STS consideration 2, the upgrade to the EHR following implementation of PE Dx significantly altered the functionality of the “ED Navigator” in the EHR system. This modification resulted in workflow disruptions and decreased use of our CDS. In a continuous design process, we would have earlier identified these workflow problems and could have made just-in-time modifications to improve fit of the CDS in clinician workflow.

Matheny et al. (2019) have proposed a developmental life cycle of AI applications, which emphasizes a cyclical and iterative effort to improve the integration of AI in the clinical workflow. They describe 7 steps in the development of AI: (1) identify or re-assess needs, (2) describe existing workflows, (3) define the desired target state, (4) acquire or develop AI system, (5) implement AI system in target setting, (6) monitor ongoing performance, and (7) maintain, update, or de-implement. Steps 6 and 7 are essential for continuous sociotechnical system design. Early in the development of AI technologies, designers need to consider how feedback loops will be integrated in the work system to support the monitoring of the technology in real use and to help identify problems as they arise (Carayon et al., 2006; Carayon et al., 2014). This is similar to the concept of the Learning Health System (Grossman et al., 2011).

The transparency and trust of AI systems should also be viewed as a part of the continuous design process. Despite expanding recognition of the need for transparent and explainable AI (Sendak et al., 2020; Sujan, 2021; Warden et al., 2019), little is known about how AI will be interpreted by users and how this interpretation will influence decision making in healthcare (Matheny et al., 2019). Additional research is needed to explore how we can help users understand and use information provided by AI to support their decision-making processes. This should include consideration of how users’ trust in a system may evolve over time.

One limitation of this study is that the CDS described did not include advanced machine learning algorithms similar to common forms of AI technology. However, as demonstrated in Table 1, there are numerous similarities between our CDS and advanced AI technologies; therefore, the PE Dx is a useful case study. As our CDS did not use advanced algorithms, we did not encounter some issues of AI integration such as bias; this is an important area for future research. AI is a “technology” within the work system model that interacts with all other work system components; future research can continue to refine the specific work system components and their characteristics that are relevant to AI. Future research should also explore additional methods and analytic tools that can be used to prospectively improve sociotechnical system fit of AI technologies. In our paper, we describe three key sociotechnical considerations, which can be further refined in future research. Future research could also further quantify and quality workflow changes with AI applications in health care delivery and relate those to acceptance and use of the technology.

5. Conclusion

In order to make the promises of AI in health care delivery a reality, a sociotechnical systems approach is essential. Using the work system model, we outline 3 key sociotechnical considerations in the design of AI and demonstrate these considerations through a case study of a clinical decision support. Our sociotechnical systems approach can support AI designers and implementers in fully considering the work system in the development of AI. AI designers and implementers should ask questions about how the AI technology fits or interacts with other system elements: the people or users of the technology, the tasks involved in using the technology, the linkages of the AI technology with other tools and technologies, the physical environment in which the AI technology will be used, and the organizational context. We also recommend collaborative and continuous design processes of AI in health care delivery; this will support the emergence of a learning health system.

Biography

Megan E. Salwei, PhD is a Research Assistant Professor in the Department of Anesthesiology and the Department of Biomedical Informatics at Vanderbilt University Medical Center. She received her PhD in Industrial and Systems Engineering from the University of Wisconsin-Madison. Pascale Carayon, PhD is Professor Emerita in the Department of Industrial and Systems Engineering at the University of Wisconsin-Madison. She is also the founder of the Wisconsin Institute for Healthcare Systems Engineering.