Integrating and Adopting AI in the Radiology Workflow: A Primer for Standards and Integrating the Healthcare Enterprise (IHE) Profiles

Abstract

The deployment of artificial intelligence (AI) solutions in radiology practice creates new demands on existing imaging workflow. Accommodating custom integrations creates a substantial operational and maintenance burden. These custom integrations also increase the likelihood of unanticipated problems. Standards-based interoperability facilitates AI integration with systems from different vendors into a single environment by enabling seamless exchange between information systems in the radiology workflow. Integrating the Healthcare Enterprise (IHE) is an initiative to improve how computer systems share information across health care domains, including radiology. IHE integrates existing standards—such as Digital Imaging and Communications in Medicine, Health Level Seven, and health care lexicons and ontologies (ie, LOINC, RadLex, SNOMED Clinical Terms)—by mapping data elements from one standard to another. IHE Radiology manages profiles (standards-based implementation guides) for departmental workflow and information sharing across care sites, including profiles for scaling AI processing traffic and integrating AI results. This review focuses on the need for standards-based interoperability to scale AI integration in radiology, including a brief review of recent IHE profiles that provide a framework for AI integration. This review also discusses challenges and additional considerations for AI integration, including technical, clinical, and policy perspectives.

© RSNA, 2024

Summary

Standards-based interoperability, with guidance from Integrating the Healthcare Enterprise profiles, helps scale artificial intelligence integration in radiology workflow.

Essentials

• ■ Standards-based interoperability facilitates artificial intelligence (AI) integration by enabling seamless exchange and use of information between systems in radiology workflow.

• ■ A multidisciplinary approach with coordinated implementation of relevant standards provides a sustainable means for managing AI traffic and integrating results from multiple simultaneously deployed models.

• ■ Effective AI integration requires consideration of both how AI solutions will interact with data and information systems along the imaging chain and how radiologists will interact with AI results.

Introduction

As radiology practices embrace an array of artificial intelligence (AI) solutions, they must understand and address the requirements for effective AI integration into the radiology workflow. AI promises substantial productivity gains by automating tedious and time-consuming tasks. However, AI solutions must seamlessly integrate into the existing radiology workflow to maximize these promised efficiency gains.

Ideally, multiple AI solutions would operate simultaneously on images from different modalities and non–pixel-based data, like text and measurements, from picture archiving and communication systems (PACS) or the electronic health record (EHR). However, variable adoption of standards, multiple AI result formats, and an increasing number of AI tools contribute to a more chaotic, disorganized reality. A large AI portfolio places greater demands on existing workflow infrastructures and information technology (IT) staff to accommodate multiple concurrent processes while simultaneously optimizing radiologist-AI interactions and ensuring the stability of applications already in the workflow.

Accordingly, this article reviews common end-user AI interactions along the radiology workflow, emphasizing a need for standards-based interoperability to scale AI integration and leverage the potential benefits of AI-augmented workflow. Additionally, there is a brief overview of recent Integrating the Healthcare Enterprise (IHE) profiles (standards-based implementation guides that provide a framework for AI integration) and challenges and considerations for AI integration from technical, clinical, and policy perspectives. The goal of this article is to provide a foundation for general radiologists to critically evaluate the current state of deployed AI systems (whether commercial or “in-house,” referring to an AI solution developed and maintained by a radiology department or practice), scrutinize commercial AI solutions for their practice, and function as effective champions of AI integration. This review frames steps for standards-based integration, bringing attention to key considerations that must be addressed in an AI deployment plan.

AI Touch Points in Medical Imaging

Since AI can potentially support almost every aspect of medical imaging, radiologists can expect AI to eventually “touch” (ie, be incorporated into) every device and system or the data that it processes. Figure 1 shows examples of AI along an image acquisition workflow during study ordering, preprocessing, acquisition, postprocessing, reporting, and storage (1).

Figure 1:

Diagram shows examples of artificial intelligence (AI) touch points along the imaging chain. AI may interact with systems involved with study ordering, preprocessing, acquisition, postprocessing, reporting, and storage. DICOM = Digital Imaging and Communications in Medicine, EHR = electronic health record, PACS = picture archiving and communication system, RIS = radiology information system.

The degree of radiologist interaction and oversight required by different AI solutions at these touch points varies. For example, AI can automate lung nodule or tumor identification and characterization in PACS, requiring radiologist review, modification, and acceptance or rejection of the AI-generated annotations (“human-in-the-loop” framework with the radiologist confirming the final output of AI-assisted tasks). Similarly, an AI-augmented EHR system may recommend appropriate examination protocols for approval or modification. Alternatively, AI that optimizes image acquisition or extracts patients’ history and laboratory test results using natural language processing may only require radiologist monitoring and feedback for improving performance over time (“human-on-the-loop” framework with radiologist oversight without immediate intervention).

Packaging AI Solutions

Delivering AI functions to end users depends on constraints such as performance, dataflow, and security considerations. With cloud computing and application programming interfaces, actual data processing by AI may happen locally or off-site. AI models may function throughout the imaging pipeline, either communicating with or embedded directly within radiology systems.

For example, a disease detection model may operate on a separate server that receives images from the PACS and returns results. Alternatively, an MRI acceleration or CT dose optimization model would likely run on the scanner, processing raw data as they are acquired (2–4). A radiology appointment scheduler may directly integrate with scheduling and order management in the EHR, analyzing upstream order messages and integrating scanner availability with other scheduled studies and expected examination durations, to optimally schedule the appointments.

When large numbers of AI models may be run on a given study, an AI orchestrator can help as a “coordination hub” or “traffic cop” to select which ones are run, collect the results, and log them in the local archive (2). It can also marshal local input data to the AI service. For example, an AI orchestrator could ingest images from the modality, de-identify them to maintain compliance with the Health Insurance Portability and Accountability Act, send data to appropriate AI servers outside the hospital network, receive AI results, and send those results to a PACS or a vendor-neutral archive, a centralized repository that stores imaging data in a standardized format.

Regardless of the packaging, effective AI integration requires a standards-based approach to ensure seamless exchange and use of information between systems.

Interoperability Standards

Interoperability refers to the ability of two or more systems or components to exchange information and use the exchanged information (5,6). Interoperability facilitates clinical integration of different commercial or in-house AI solutions in a single environment. Interoperability between systems relies on standards, meaning documents established and approved by consensus from a recognized body that provide rules, guidelines, or characteristics for activities or their results (5).

Interoperability is often overlooked. During early AI model development, resources are spent on creating proof-of-concept prototypes, not well-integrated systems. Radiology practices often purchase systems from multiple vendors and may also deploy multiple in-house solutions. While developing a custom integration with a selected partner may work well for a single use case, that approach does not scale for enterprise-level deployment of many models, commercial or in-house, operating at multiple sites. As shown in Figure 2, multiple custom integrations contribute to disordered workflow in viewing AI results, placing the burden on radiologists and IT staff to accommodate different integrations simultaneously. Eventually, mature segments of the market recognize the value of interoperability, with users requesting adherence to standards and vendors offering standards-compliant products. As an example, the Imaging AI in Practice demonstrations hosted by the RSNA educate radiologists and draw attention to standards that facilitate integration of AI solutions in practice (1). Lessons from these demonstrations underscore the need to stay current on standards as additional IHE profiles are released and understand the practical considerations to address as next steps for integrating AI solutions.

Figure 2:

Diagram shows the accommodation of multiple artificial intelligence (AI) solutions in the radiology workflow with standards-based interoperability. (A) Current radiology workflow may require custom integrations to simultaneously integrate multiple AI solutions, creating a disordered workflow with proprietary viewers, external applications, and limited automation. Custom integrations may require viewing AI results in proprietary viewers. (B) Alternatively, a single viewer may display pixel-based AI results, non–pixel-based AI results, and the source image. AI results may be represented by both Digital Imaging and Communications in Medicine (DICOM) secondary capture (SC) and DICOM structured report (SR) formats, which are further detailed in Table 1 and Figures 4 and 5. AI results may then be manually dictated in a report or populated automatically in a report template from both pixel and nonpixel results sent from a separate deployed AI solution. Ideally, vendor support for standards-based interoperability streamlines integration of multiple AI solutions with automated report population. AI results can still be shown on a separate AI viewer that integrates multiple solutions. Alternatively, AI results from several solutions can be integrated through the picture archiving and communication system (PACS) with an option to accept, modify, or reject AI results. In contrast to A, accepted or modified results can then be populated in the report template directly from the PACS without a need for interfacing with separate AI solutions. FHIR = Fast Healthcare Interoperability Resources.

Adopting Standard Codes

The use of standardized codes to represent concepts, such as anatomy, pathology, and diagnoses, is central to making information machine-readable, which in turn enables the automation of tasks like report template selection or recommendation for follow-up monitoring.

Converging on a common set of codes is critical to achieving interoperability across the enterprise. While practices may opt to use custom local codes, this requires considerable maintenance costs and limits automation and data exchange outside the practice.

Fortunately, there are well-managed coding systems to draw from (6).

The Reporting and Data Systems, or -RADS, created by the American College of Radiology, provide standardized terminology and reporting guidelines for various pathologies (eg, BI-RADS [breast imaging], Lung-RADS, TI-RADS [thyroid imaging]) (7).

SNOMED Clinical Terms is a comprehensive hierarchical clinical terminology system including diseases, procedures, medications, and anatomy (8).

LOINC is a system of universal codes and names for clinical and laboratory tests, measurements, observations, and procedures (9).

RadLex is curated by the RSNA and provides standardized vocabulary for describing imaging techniques, procedures, and pathologic findings (10,11).

Consistent use of common codes is also a cornerstone for big data collection and analysis projects.

IHE for Radiologists

IHE is an initiative established and supported by the RSNA and Healthcare Information and Management Systems Society to address persistent obstacles to the effective integration of informatics systems (eg, radiology information system, EHR, PACS, vendor-neutral archive, reporting software) in health care, including radiology.

Integration requires products (eg, EHR, PACS, and modalities) from different vendors to achieve a common understanding of the shared task and a coordinated implementation of relevant standards. The standards are often large and full of features, which makes them powerful and versatile, intended for developers to apply them to a wide variety of problems. However, there can be more than one way to apply the standards to a given problem, making it almost inevitable that vendor A and vendor B will have made different incompatible choices. Solving this requires a resource-intensive effort by multidisciplinary teams working across corporate boundaries, including motivated clinicians to inform and support such work.

IHE regularly collects proposals for problems amenable to IHE-style solutions (ie, reasonably well-understood use cases for which standards exist). After a specific use case has been selected and detailed, IHE focuses on two things.

First, IHE tailors existing standards by selecting components of standards, specifying object definitions, defining transactions between relevant systems, and constraining how the standard is applied to solve the use case. These details are published in a document called an IHE profile.

Second, IHE tests implementations. IHE organizes annual “Connectathon” events in North America, Europe, and Asia and creates detailed test plans, arranges for testing tools, and invites expert assessors. Vendors bring products and prototypes to test and debug in a collaborative, engineering-oriented environment.

For system vendors, an IHE profile provides an implementation guide, telling them how to play their role in a multivendor solution. It specifies many technical details, making it easier for company engineers supporting the product development and testing processes.

For hospitals, an IHE profile provides a template for a solution project and an effective way to communicate with suppliers in a request for proposal; by referencing the name of the profile document, everyone is aligned on expected capabilities and how they will interoperate.

One common pattern of IHE profiles is addressing how to create, store, and access a certain type of information (termed a “content profile”). Another common pattern coordinates workflows and dataflows for information handling (termed a “workflow profile”).

In the IHE profile life cycle (Fig 3), “trial implementation” provides an opportunity for vendors to implement the specification in products, test those products at Connectathons, and encourage early adopters to try using the profile in practice. “Final text” profiles are then kept stable to preserve vendor and user investments in existing implementations and deployments. Radiologists, as domain experts and critical users, are encouraged to actively participate throughout the life cycle and adoption of IHE profiles.

Figure 3:

Integrating the Healthcare Enterprise (IHE) profile life cycle and how to contribute as a radiologist. Simplified diagram of the life cycle of an IHE profile demonstrates key points where stakeholders, including radiologists serving as domain experts, may contribute. The process starts with an approved proposal (month 0) and culminates with the final text. Radiologists are encouraged to actively participate by submitting ideas, suggesting comments for the initial draft, trying pilot projects for trial implementations, and deploying final profiles.

Given the breadth and scale of addressing interoperability across the entire health care enterprise and the IHE target of completing most of the profile life cycle in less than 12 months, each IHE profile is scoped to a separable piece of useful integration. IHE coverage increases as the library of profiles expands, and it is expected that most products will implement an expanding list of profiles. Site integration projects tend to leverage one or more relevant profiles.

Summary of AI-related IHE Trial Implementation Profiles for Radiology

IHE Radiology, one of several IHE domains, specifically addresses medical imaging, managing profiles for departmental workflow and information sharing. Several recent profiles pertaining to AI for radiology are summarized herein: AI Results (AIR); AI Workflow for Imaging, or AIW-I; Prioritization of Worklists for Reporting; and Integrated Reporting Applications.

AI Results

Meaningful action and clinical use of AI results depends on integrating them into existing radiology workflow. For pixel-based AI results, this may involve displaying them in PACS next to the original images or providing more advanced viewing functions.

Left to run without oversight, some implementations store AI results in proprietary file formats or private Digital Imaging and Communications in Medicine (DICOM) attributes. Those varied formats necessitate proprietary viewers and unrealistic customization burdens. Displays would need to customize interfaces for hundreds of potential AI models.

The AIR profile addresses how pixel-based AI results are encoded, stored, retrieved, and displayed in an interoperable manner (12). AI results are recorded using a set of standards-based basic types, such as measurements, qualitative findings, locations, and segmentations. Those standard encodings can be stored and retrieved by existing archives and displays. Conformant displays are able to present these results with basic viewing requirements (eg, automated nodule measurements or overlay and label segmentations on the corresponding image of the finding) (12). Decision support systems, clinical databases, and report creators can also access and process the standard results.

A key element of the AIR profile requires understanding the distinction between graphical representations and machine-readable semantic representations. AIR requires applications to be able to provide clinical information in specific machine-readable formats. This involves using common data structures (eg, the RSNA/American College of Radiology Common Data Elements Project) and standard codes so that software can parse the information.

A DICOM secondary capture is a graphical representation, analogous to a standardized screen snapshot (13). Some first-to-market implementations export AI results using DICOM secondary capture—for example, a static view of one section of a CT scan with an embedded (“burned-in”) outline of a pulmonary nodule or intracranial hemorrhage with data present in private secondary capture attributes. DICOM secondary capture is the most rudimentary form of data transfer. Although it has almost ubiquitous support across PACS and viewers, there is no way to interact with the presented image (eg, scroll, window or level adjustment) or private attributes (proprietary information available only to a vendor) or integrate the data with other systems. As seen in Figure 4, users are presented with a static image with “burned-in” finding annotations but without machine-readable information. The limitations of this format are akin to a colleague requesting a curbside consultation by sending a smartphone photograph of an image on their screen rather than linking to the study in the PACS.

Figure 4:

Static artificial intelligence (AI) result viewing with Digital Imaging and Communications in Medicine (DICOM) secondary capture. The DICOM secondary capture image conveys a frozen view of the result screen from an algorithm. It provides some information to the radiologist (eg, the location of suspected lesions), but they cannot interact or modify the information, such as the position of the boxes. More importantly, none of the information is readable by the reporting system, databases, or other automation tools. This example is provided by the 2021 RSNA Imaging AI in Practice Demonstrations. Displayed “patient information” is simulated and does not represent real patient data.

A DICOM structured report is structured data containing coded entries to represent findings in a diagnostic report (14,15). A DICOM structured report object is machine-readable and can also be rendered for human use. The structured report contains codes that describe anatomic locations, measurements, or other pathologic findings. For example, lung nodules could be documented with estimated diameters, volumes, nodule type, and location, allowing reporting software to insert findings into appropriate fields in a radiologist’s report (14). A worklist manager AI model could use data from a DICOM structured report to reprioritize a study’s position on the worklist if emergent findings are present. As seen in Figure 5, a DICOM structured report will allow most AI results to be viewed in a unified viewer on the PACS.

Figure 5:

Dynamic artificial intelligence (AI) result viewing with Digital Imaging and Communications in Medicine (DICOM) structured reports. Results encoded as DICOM structured reports, shown here rendered on the associated image by a general display, allows radiologists to interact with AI results. Radiologists can see a list of indexed pulmonary nodules and accept (yellow arrow) or reject findings. Radiologists may adjust lesion calipers (blue lines highlighted by green arrows) before accepting. The radiology report can incorporate data for the accepted results from the DICOM structured report. Viewers may include additional features, such as three-dimensional reconstructions of findings (bottom right panel). This example is provided by the 2021 RSNA Imaging AI in Practice demonstrations. LungRADS = Lung Reporting and Data System.

Table 1 provides an overview of the differences between DICOM secondary capture and DICOM structured report. Table 2 provides examples of AIR-supported machine-readable formats.

Table 1:

Differences Between DICOM SC and DICOM SR

Table 2:

Examples of AIR-supported Machine-readable Formats

AIR encourages, but does not mandate, the use of standard code sets (eg, RadLex, SNOMED Clinical Terms) when encoding AI results to improve semantics and therefore interoperability. Interoperable results can facilitate testing of AI solutions before deployment. If users perform manual segmentations, those segmentations should be stored as DICOM segmentations or as contours in DICOM structured report using template identifier 1500, as mandating the same format makes it easier to compare and assess AI performance against the reference standard.

AI Workflow for Imaging

Imaging centers running a handful of AI models may be comfortable with setting forwarding rules in PACS and/or a DICOM router to deliver studies directly to an AI model and letting the model choose what processing to perform (eg, route all studies for lung screening CT to a lung nodule detector). However, as dozens, or even hundreds, of AI models are added, the traditional model of forwarding entire DICOM studies (or even series) to AI models will no longer be practical. The sheer amount of these forwarded studies will overwhelm a hospital or imaging center’s network, resulting in delays in model processing. Additionally, without specific rules, many of these studies may be unnecessarily forwarded.

Published in 2020, the AI Workflow for Imaging profile describes how AI processing traffic can be managed in a scalable way (16). AI Workflow for Imaging describes a platform-neutral mechanism for flexible decoupling of pertinent AI processing tasks, including (a) the business logic for which tasks to schedule, (b) the coordination to get those tasks performed, and (c) the dataflow to feed the performers and collect their output. Central to this profile is the “task manager,” which accepts and manages tasks from “task requestors.” Tasks requested may be driven by local policies (eg, “run lung and spinal screening on all outpatient chest CT scans”), order details (eg, “run intracranial hemorrhage detection on head CT scans ordered from the emergency department”), or ad hoc requests (eg, radiologist requests all available kidney assessments be run on the current examination). Examples of a task manager include an AI orchestrator, an AI marketplace, or a component of the PACS (14). As AI tasks are queued, the task manager makes them available, or assigns them, to appropriate “task performers” (ie, AI applications).

A simplified workflow is shown in Figure 6. When a new imaging study comes into the PACS, the AI orchestrator notifies the models of its availability, providing some basic information about the study (eg, modality, study description). The model can then either notify the AI orchestrator of its intention to process the study, thus “claiming” a task, or request additional information about the study to decide whether or not it wants to claim the task.

Figure 6:

Diagram shows simplified workflow representation for the Integrating the Healthcare Enterprise (IHE) Artificial Intelligence (AI) Workflow for Imaging profile. The picture archiving and communication system (PACS) functions as a task requestor, queuing an AI task. The AI orchestrator functions as a task manager, ensuring that task performers (AI models 1–3) are notified about new tasks. The appropriate AI model, in this case AI model 3, claims the requested task and processes the data. Results are sent back to the AI orchestrator, which routes them to the task requestor.

Prioritization of Worklists for Reporting

AI triage models should inform reading worklist prioritization by highlighting cases with unexpected and potentially serious findings. For example, an AI solution analyzing all chest radiographs as they are captured might flag one as “positive” for a pneumothorax. A worklist manager with access to this information would likely reprioritize that study toward the top of the worklist, decreasing the time for the radiologist to confirm the finding and appropriate care to be provided to the patient. However, study prioritization depends on a multitude of factors, such as study availability, study completeness, ordering department, chronicity, and service-level agreements like required turnaround time. Standalone AI solutions, such as those screening for emergent findings, cannot directly claim the top position on the worklist, as they are unaware of those other factors and the relative importance of other pending studies.

The Prioritization of Worklists for Reporting profile, recently released for trial implementation, defines standard transactions for AI models, the radiology information system, and other systems with relevant information to feed the worklist driver application (eg, PACS or EHR), which balances factors across a large number of studies on a worklist.

Integrated Reporting Applications

Applications collaborating on radiology reporting (reporting software, PACS, EHR, AI solution) should communicate in real time. Currently, data exchange between applications leverages proprietary integrations, achieving real-time integration at the expense of interoperability. For example, an AI solution measuring aortic diameters may send results to a proprietary interface or widget that then communicates with the reporting software to populate desired measurements in a report, potentially involving manual steps that further delay this process.

The Integrated Reporting Applications profile provides a framework for real-time information exchange using a standard called FHIRcast (Fast Healthcare Interoperability Resources) (17). Following this framework, applications synchronize to the context (the current patient, study, or report) and share content created by other applications. Integrated Reporting Applications would enable real-time exchange of the aortic diameter measurements generated by the AI solution in the previous example as the measurements are created or as they are adjusted during radiologist review of AI results.

“AI Interoperability in Imaging” White Paper

The IHE Radiology white paper “AI Interoperability in Imaging” provides a useful framework of concepts for imaging-related AI (18). This resource reviews the scope of AI in imaging to improve radiology workflow and characterizes the steps to develop, deploy, and integrate AI models. The white paper is also intended to serve as a foundation for future IHE work.

Mandating IHE Compliance

Implementing IHE profiles requires development and testing resources from product vendors. Such expenditures are justified and prioritized based on clear market demand. Users and customers are encouraged to discuss IHE profile adoption plans during dialogue with vendor partners and mandate support for IHE profiles in requests for proposals, communicating market demand to adopt these standards.

Considerations for AI Integration

Radiology workflow in a digital imaging environment features a complex set of interactions between modalities, the radiology information system, EHR, PACS, reporting software, external applications, and other components necessary for clinical imaging consumption and management. Effective AI integration requires consideration of how AI solutions will interact with data and information systems along the imaging chain and how radiologists will interact with AI results.

Technical Considerations

Integration planning needs to address technical considerations (Table 3) to incorporate new systems and handle changing demands on existing systems. Seamless integration of AI solutions requires conforming to interoperability standards that enable exchange of data between a practice’s EHR, radiology information system, PACS, worklist solution, and reporting software.

Table 3:

Technical Considerations for AI Workflow Integration

AI deployment and integration across the enterprise is a gradual process. Sites are unlikely to overhaul all their systems at once. The reality is that radiologists will practice in a mixed environment where new AI solutions are deployed alongside, and must interface with, existing software and network architecture. Project planning needs to consider standards and features to address such mixed-environment integrations. Such decisions should be made strategically so that AI solutions are handled the same way, rather than imposing many different methods on radiologists and other users.

Consider integration of a portfolio of AI solutions that screen for emergent pathology, like intracranial hemorrhage or pulmonary emboli, and reprioritize worklists to triage positive studies. AI solutions that use DICOM to exchange images and results will integrate more easily with a DICOM-compatible PACS. If the results are presented by means of a third-party application, the third-party application should integrate with existing software and comply with security measures. Accommodating study worklist reprioritization based on AI results may require modifications to an existing worklist to recognize studies flagged as positive by the AI solutions. In this context, purchasing standards-based solutions provides a means for both forward and backward compatibility, allowing existing systems to support standards-based functions in new systems and newly deployed solutions to communicate with existing systems (5).

Clinical Considerations

Planning also needs to consider the impact on clinical staff and their daily operations (Table 4). Effective AI integration conforms to a practice’s routine workflow with minimal disruption. AI results must be easily accessed without additional steps.

Table 4:

Clinical Considerations for AI Workflow Integration

There should be ongoing discussion with clinical champions regarding the clinical performance and feasibility of using deployed AI models over time. This includes discussion about pairing imaging studies with appropriate AI models (eg, CT brain images processed by the intracranial hemorrhage detection algorithm and musculoskeletal radiographs processed by the fracture detection algorithm while avoiding having CT brain images be processed by the pulmonary emboli detection algorithm). Study-model pairing can, and often should, go beyond the specific reason for the examination to leverage opportunistic imaging. For example, it can be valuable to run pulmonary emboli detection models on all contrast-enhanced CT chest images (in addition to dedicated CT angiography chest studies ordered to rule out pulmonary emboli), CT angiography aorta images including thoracic structures, and CT abdomen images including lower thoracic structures. Similarly, automated quantification of lumbar spine CT attenuation can allow for assessment of abnormal bone mineral density on CT abdomen and pelvis images ordered from the emergency department or for alternative indications.

Governance and Policy Considerations

Practice-level governance and policy considerations are detailed in Table 5. For example, before radiologist confirmation, results may be incorrect, and practices should discuss the consequences of archiving or making unconfirmed results accessible to other physicians or patients. To prevent concern or clinical action on incorrect results, practices might initially embargo AI results until confirmed by a radiologist and released to the patient’s treatment team. Alternatively, practices may require labels on unconfirmed AI results indicating results are provided directly from the AI solution without radiologist review.

Table 5:

Policy Considerations for AI Workflow Integration

Practices should also determine who interacts with AI results (eg, technologists, nonradiologists, or radiologists). For example, practices may designate trained US technologists to review and adjust AI results that characterize thyroid nodules, followed by radiologist review of the adjusted AI results (with further adjustments, if needed).

Policy decisions can lead to new technical considerations. For example, embargoing results (before issuing a final report to avoid condoning information blocking) might need a separate storage location (“application entity” title) on the PACS.

Some topics may span technical, clinical, and policy considerations listed in Tables 3–5. Table 6 provides example considerations for AI result storage.

Table 6:

Example Technical, Clinical, and Policy Considerations for AI Result Storage

Continuous integration of AI solutions into the existing workflow requires sound strategic planning that encompasses model selection, predeployment validation, deployment on existing infrastructure, and monitoring (19,20). Integration of AI must be led by a multidisciplinary governing body, either at the institutional or enterprise level, consisting of clinical leadership, hospital administration, data scientists, IT management, and radiologist (“end user”) representatives, including clinical champions for relevant modalities (19,20). The governing body reviews project proposals, maintains communication with vendors, and supervises the formal process from AI tool selection to validation and deployment (19,20). Model validation at the site or sites of future deployment includes simulation in routine radiology workflow to assess baseline model performance on local data, identify workflow barriers, and collect feedback from end users. The simulation process may entail initial model testing in a research, or “sandbox,” environment with de-identified real-time data and “pilot testing” in real-time workflow to determine the ability of the current radiology workflow to accommodate the new solutions (with attention to considerations detailed in Tables 3–5). Additional longitudinal considerations include reassessment of project needs and reallocation of resources, including the number of dedicated IT staff to support ongoing maintenance and response to user issues.

Processes for continuous monitoring, including designation of specific performance and clinical metrics, must be in place before AI deployment (20). AI monitoring can leverage business intelligence solutions that aggregate data for direct (eg, sensitivity, specificity, accuracy) and indirect (eg, operational key performance indicators: report turnaround time, radiologist-AI concordance rates, AI processing turnaround) performance metrics. Advanced monitoring may assess relevant patient outcomes, such as time to disease detection, time to treatment, or medical error rate associated with AI use. Commercial AI solutions may have vendor support for continuous monitoring, while in-house AI solutions may require local support to build dashboards for monitoring deployed tools. AI governance frameworks should specify monitoring paradigms that evaluate specific metrics over time and determine guidelines for model retraining or suspension in the context of model drift. One example strategy includes retraining models on data from specific time frames (eg, past 3 months, past 3 weeks) following detection of degraded performance (21). Securing support for a robust pipeline for AI monitoring is essential before deployment to prevent downstream errors and patient harm from undetected aberrant model performance.

AI Literacy and Change Management

Introducing new AI solutions can substantially alter practice paradigms, requiring radiologists to learn new workflows and solution nuances to leverage the full potential of those AI solutions. Effective training considers radiologists’ baseline familiarity and experience with AI and associated applications. That familiarity and experience will vary from user to user, and educational material must be prepared that spans that range of experience. Training programs may be customized for a practice or leverage general material provided by a vendor. A “sandbox” environment for users to practice using AI solutions and interacting with AI results before interaction in practice can build familiarity and experience. Deployment should be accompanied by a system for obtaining user feedback to detect issues warranting intervention.

Upskilling IT Staff

Deploying new solutions also requires upskilling IT staff to ensure robust operations. Staff need understanding of how the new technologies work, including the standards and codes that underlie them. Additionally, staff need education on each use case, with attention to the clinical problem being addressed. Creating opportunities for IT staff to visit the reading room or converse with radiologist champions is necessary to clearly convey the clinical context for each deployed AI solution. Continuing education will be needed until AI product evolution slows.

Conclusion

Standards-based interoperability is a prerequisite for seamless integration of artificial intelligence (AI) in radiology workflow. Though custom integration may be feasible for deploying a single AI model, relying on custom integrations is neither sustainable nor scalable. In contrast, a multidisciplinary approach with coordinated implementation of relevant standards provides a sustainable means for managing AI traffic and integrating results from multiple simultaneously deployed models. Recent Integrating the Healthcare Enterprise profiles pertaining to AI for radiology provide guidance for these endeavors. However, evolving paradigms with new techniques requires continued efforts to design, and possibly redesign, frameworks for AI integration that optimize implementation of relevant standards to address new use cases. Radiologists represent a crucial driver in this process, providing clinical expertise to create appropriate frameworks for AI integration and their translation to practice.