Artificial intelligence predictive analytics in heart failure: results of the pilot phase of a pragmatic randomized clinical trial

Abstract

Objectives

We conducted an implementation planning process during the pilot phase of a pragmatic trial, which tests an intervention guided by artificial intelligence (AI) analytics sourced from noninvasive monitoring data in heart failure patients (LINK-HF2).

Materials and methods

A mixed-method analysis was conducted at 2 pilot sites. Interviews were conducted with 12 of 27 enrolled patients and with 13 participating clinicians. iPARIHS constructs were used for interview construction to identify workflow, communication patterns, and clinician’s beliefs. Interviews were transcribed and analyzed using inductive coding protocols to identify key themes. Behavioral response data from the AI-generated notifications were collected.

Results

Clinicians responded to notifications within 24 hours in 95% of instances, with 26.7% resulting in clinical action. Four implementation themes emerged: (1) High anticipatory expectations for reliable patient communications, reduced patient burden, and less proactive provider monitoring. (2) The AI notifications required a differential and tailored balance of trust and action advice related to role. (3) Clinic experience with other home-based programs influenced utilization. (4) Responding to notifications involved significant effort, including electronic health record (EHR) review, patient contact, and consultation with other clinicians.

Discussion

Clinician’s use of AI data is a function of beliefs regarding the trustworthiness and usefulness of the data, the degree of autonomy in professional roles, and the cognitive effort involved.

Conclusion

The implementation planning analysis guided development of strategies that addressed communication technology, patient education, and EHR integration to reduce clinician and patient burden in the subsequent main randomized phase of the trial. Our results provide important insights into the unique implications of implementing AI analytics into clinical workflow.

Background and significance

Heart failure (HF) is a common medical condition with prevalence increasing with age.¹ HF is one of the most frequent discharge diagnoses in the United States.² The morbidity related to HF remains very high; the rate of readmission for HF is approximately 20% at 30 days and 30% at 90 days after discharge. Readmission in HF patients is related to adverse outcomes including increased mortality.^3,4 Reduction of HF readmissions is, therefore, a strategic goal for both civilian and Veterans Health Administration hospitals.

Clinicians and patients are typically unaware of the physiologic changes of early HF exacerbation. Once overt symptoms and signs of HF exacerbation are present, it usually is too late to prevent hospitalization.⁵ This may, in-part, explain why monitoring of parameters such as body weight or blood pressure has not resulted in reduction of hospitalizations in previous clinical investigations.^6,7 Importantly, detection of HF exacerbation at its early stages could provide an opportunity for therapeutic intervention that would, in turn, decrease the risk of hospital admission.⁵

In an earlier study, we tested a noninvasive remote patient monitoring system consisting of a wearable disposable multisensory patch (sensor) placed on the patient chest that recorded continuous physiological data. We showed that artificial intelligence (AI) methods based on machine learning analysis of the continuous data streams accurately predicted HF rehospitalization approximately 1 week before the hospitalization took place. In addition, we demonstrated high patient adherence rate with the use of the sensor.⁸ The next step would be to determine whether early, standardized intervention based on this AI predictive algorithm could result in improved outcomes.

Use of AI in healthcare is gaining more focus across academia and industry.^9–11 Despite rising popularity, however, few real-world results of integration of AI analytics into clinical care have been published. Several authors have suggested that the lack of information and guidance on implementation is a limiting factor.^12–14 In addition, systematic reviews on the use of AI in clinical care also suggest that implementation in clinical practice is now the most important barrier to AI adoption.^13,15,16 Investigations on the barriers and facilitators for effective implementation of AI tools are a current research imperative. Implementation planning research explores factors that might impact actual use in a real setting¹⁷ which can be incorporated into a real-world pragmatic trial of an intervention’s effectiveness.

Continuous wearable monitoring analytics predict heart failure decompensation (LINK-HF2) is a pragmatic trial consisting of 2 phases. The nonrandomized pilot phase that explores issues related to implementation and the main randomized phase that implements the intervention in a real-world setting (Figure 1). This report presents results of the implementation planning conducted during the pilot phase, which focused on provider and patient beliefs, workflow, possible barriers and facilitators, and organizational factors that could impact the fidelity of the clinical intervention.^17,18 Specific attention was paid to the various issues of implementing AI in clinical settings,^9,13 including a multi-level approach,¹⁹ tailoring strategies by context,^20–22 and incorporating internal formative evaluation metrics into the pragmatic trial itself.²³

Figure 1.

Study design of the LINK-HF2 pragmatic trial.

Our specific objectives were to: (1) Characterize the workflow, environmental constraints, and communication channels relevant to implementation planning of an ambulatory monitoring program at the clinic and team level; (2) characterize provider perceptions of using AI data generated from an ambulatory monitoring system relevant to implementation planning in a real clinical setting; and (3) design generalizable and measurable implementation metrics relevant to implementation planning in a real clinical setting for a future pragmatic trial intervention.

Materials and methods

Setting and patient population

This work was conducted at 2 VA sites located in different geographical areas, with different patient populations and clinicians. The study received Institutional Review Board approval and Research and Development Committee approval at all institutions. All subjects provided written informed consent for study participation. Both sites had cardiac care services. Table 1 describes the clinics and settings.

Table 1.

Description of pilot settings.

Setting	Patients (n)	Clinicians (n)	Description
Salt Lake City, UT VA Cardiac Care	17	9	Academic urban hospital with 160 HF inpatient visits/year. Cardiac care service sees 763 unique HF patients/year.
Gainesville, FL VA Cardiac Care	10	5	Academic urban hospital with 190 HF inpatient visits/year. Cardiac care service sees 717 unique HF patients per year.

Description of design

LINK-HF2 is a pragmatic controlled trial funded by the VA’s Health Services Research and Development Service (HSR&D). It consists of 2 phases, a nonrandomized implementation planning pilot phase and the main randomized phase (Figure 1).

In the nonrandomized pilot phase, all patients had the study sensor were monitored remotely and all notifications were transferred to the clinical team. The design of the pilot work was both qualitative (interviews and observations) and quantitative (behavioral data) using a cohort design that included patients discharged after HF hospitalization as potential participants.

In the subsequent main randomized phase, all patients will wear the sensor, but will be randomly assigned to a clinician-monitoring group, where the clinicians receive notifications from the AI analytics, versus a control group, where the AI notifications are not communicated to the clinical team.

Description of intervention and study procedures

Study participants are enrolled at the time of discharge from a HF hospitalization when they are trained on how to activate and apply the disposable sensor and pair it with a provided cell phone. Patients are expected to wear the sensor 24 hours a day for up to 90 days and are instructed to change the sensor when the battery is depleted. In addition, patients complete a short version of the Kansas City Cardiomyopathy Questionnaire (KCCQ-12) and a visual analog scale at the time of enrollment, at 30 days and at 90 days after enrollment. No constraints are placed on subjects’ activities.^24,25

Continuous data were automatically uploaded from the sensor to the cell phone and then to a cloud-based server for analytical processing (PhysIQ portal). The system includes a cloud-based analytics platform that implements a machine learning method called similarity-based modeling that builds a personalized model of physiology behavior based on the patient’s initial data.²⁶ Physiological data obtained include heart rate, respiratory rate, and activity level. Windowed evaluations of relative change from baseline in the behavior of the monitored physiological data are used to generate notifications that trigger the intervention workflow we studied (Appendix SA). When changes that correlate with impending HF exacerbation are identified, a clinical notification is generated, messaged to site coordinators by email and forwarded to clinical team through patient’s electronic health record (EHR). Clinicians follow a standardized response algorithm aimed on altering therapy in patients with suspected incipient HF decompensation (Figure 2).

Figure 2.

Algorithmic decision protocol for response to notifications.

Stakeholder needs assessment in the pilot phase

Clinicians

Clinicians were interviewed 1-on-1 via secure video technology using a semistructured approach which focused on workflow, organizational structure, communication processes and technology, and general attitudes toward the intervention (Appendix SB). The initial 13 interviews were audiorecorded and transcribed. In addition, 5 observations of clinicians (2 RNs, 1 PharmD at first site and 2 RNs at the second site) preparing for patient calls post notification were conducted to identify information needs, EHR use, team communication, and patterns of preparation. All members of the clinic staff in each setting agreed to be interviewed and observed. The team members ranged in age from 29 to 63 years, averaged 13.3 years of experience in the VA, and 21.4 years of clinical experience.

Patients

Twelve patients were interviewed by phone (due to COVID restrictions) using a structured interview form with specific questions (Appendix SC). The phone calls were not recorded, but patient answers were recorded manually. The patients ranged in age from 61 to 82 years were all male, and had used the monitoring system for 2 weeks on average. The interviewers had advanced training in human factors engineering and biomedical informatics.

Qualitative analysis

Promoting Action on Research Implementation in Health Services (i-PARIHS) was the specific theoretical framework for implementation used in this study and includes 3 broad areas of focus: (1) Innovation (characteristics of the intervention), (2) Recipients (beliefs, attitudes, experience of participants), and (3) Context (culture, leadership, and policies). Each focus has subheadings which are discussed in the Results section.²⁷Facilitation is viewed as the underlying mechanism of action and is defined as “the construct that activates implementation through assessing and responding to characteristics of the innovation and the recipients (both as individuals and in teams) within their contextual setting.”²⁷ The overall goal of the planning process is to identify key strategies and tools that facilitators could use to tailor the AI intervention appropriately, that can contribute to an i-PARIHS facilitation planning tool, and that can be measured during the RCT portion of the pragmatic trial.^28,29

A hybrid analytical approach was used to qualitatively analyze the interview transcripts. The process involved an initial open-coding, inductive review followed by deductive categorization using theoretical constructs.³⁰ The analysis was conducted in 3 stages. Individual members of the research team conducted an initial review of transcripts to identify concepts and constructs of interest using NVivo (@Lumivero, 2018). After group discussion and refinement of codes, the team conducted a theoretical recoding and reanalysis based on i-PARIHS factors. Subcategories were rereviewed by 2 investigators to ensure consistency. The extracted quotations were again reviewed by 3 investigators (1 MD and 2 PhD cognitive psychologists) in order to agree on overarching themes. Table 2 describes the i-PARIHS constructs.

Table 2.

i-PARIHS theoretical constructs and description.

Constructs	Description
Innovation	The degree to which the intervention “fits” the context, the amount of familiarity that clinicians have with components of the intervention, trialability, and perceived relative advantage.
Recipients	Attitudes, roles, power structures, communication networks, goals, time, resources, and perceived support. Includes local organizational structures and technology.
Context	Three-level construct; (1) local (clinic) culture, organizational leadership, evaluation and feedback processes, workflow metrics, and learning environment; (2) organizational priorities, absorptive capacity, experience with similar changes, learning networks; (3) national priorities, regulatory requirements, incentives, and policy drivers.
Penetration	The degree to which the intervention is integrated as a practice within a service setting and its subsystems. Similar to the concept of “breadth” of adoption.
Sustainability	Defined as the extent to which a newly implemented treatment is maintained or institutionalized within a service setting’s ongoing, stable operations. It could also emphasize the integration of a given program within an organization’s culture through policies and practices as well.

Rigor and reproducibility

We addressed qualitative coding rigor by referencing the Standards for Reporting Qualitative Research recommendations³¹ and addressing recommended constructs for establishing reliability and validity in qualitative research.³²

Behavioral and adoption data

Data collected from the PhysIQ portal covered the 90-day pilot study period and 27 enrolled patients. All notifications were tracked automatically and the specific information collected included: (1) The number of notifications per patient and the number per event; (2) Response time for the study coordinators to acknowledge; (3) Response time to clinician response; (4) Type of response; (5) Timing of response; (6) Review of effort, information needs, and patient interaction; (7) Communication patterns between clinicians; and (8) Patient response.

Designing implementation evaluation metrics

An important aim of the pilot work was to propose measurable implementation metrics to be used by future facilitators. We used the RE-AIM (Reach, Effectiveness, Adoption, Implementation and Maintenance) framework for this purpose. RE-AIM is a planning and evaluation model widely used in public health, clinical implementation studies and in informatics research consisting of 5 dimensions—reach, effectiveness, adoption, implementation, and maintenance.^23,33,34 RE-AIM has been updated to include more attention to contextual issues³⁵ and to include implementation planning.³⁶ Development of the metrics consisted of multiple and iterative discussions within the research team.

Results

The results are presented in 3 sections: (1) Stakeholder needs assessment findings which include the provider and patient interviews, (2) behavioral results which include provider responses to notifications, notification data, and the observation data, and (3) proposed final RE-AIM implementation metrics.

Clinician stakeholder needs assessment

The results reflect the content coding of the transcripts of provider and patient interviews using the i-PARIHS constructs. Table 3 outlines broader categories of identified stakeholder needs with example quotations.

Table 3.

Qualitative coding examples of i-PARIHS constructs from clinicians and staff.

Innovation	Example quotations
Degree of fit with existing practice and values Home monitoring programs are common in the VA. The RNs and the pharmacists have expanded scopes of practice. The clinics provide continuity care and patients are well known to the team and are discussed frequently both formally and informally by clinic staff.	“… you kind of get a sense of what’s more acute versus a chronic issue with these with the patients and a lot of these patients have been following for quite a few years…Yes, I can even look at them and know what their normal creatinine and runs are.”(MD) “You know, that study is maybe similar to what the Boston Scientific device has in terms of heart failure index … So it’s a device. .that can give us thoracic impedance, heart rate variability, changes in F1 and F3. And, what they do is they track those, and it’ll, and it’s kind of an average over like three days, and then it’ll give the score. And if that score goes above a certain level, then the electrophysiology team is notified.” (MD)
Perceived relative advantage and user understanding Clinicians and patients opined that this system will aid in keeping a closer watch on patients. The system differs from current home monitoring programs because it pro-actively notifies clinicians of events, rather than requiring regular data review, thereby relieving workload.	“It’s like a smart technology where it can use like some sort of like hemodynamic thing way to measure the heart rate, some other stuff, and then figure out, can we predict if this patient is going to have an exacerbation? So, we'll get alerted?” (NP) “I don’t think it’s gonna be.. that intrusive, and that’s the one problem I have with Cardiomems, you know, because if you know, they say, well, you need more compliance out of your patients, you got to pick a better patient. Well, I can’t fix human nature.” (MD) “…help us presort you know, the really actionable items from the non-actionable items.” (MD)
Clarity and usability Responding to the notification required reviewing EHR (CPRS) for prior notes and reviewing data on the remote monitoring portal. Both were perceived as time intensive and not user friendly. There were several complaints about the CPRS template for recording the data.	“It is hard to use their website as I don’t understand the timeline of the notification.” (during observation) (RN) “The note template in CPRS doesn’t have a place to put all of the important information.” (during observation) (RN) “When I don’t know the patient, I have to spend a long-time reading notes in CPRS and catching up to what the goals are.” (RN during observation) “It wouldn’t really be any different. I think for me, if they call in or if I call, I still run through the same process, in terms of asking them like questions about symptoms, questions about weight, blood pressure.” (RN)
Recipient	Quotations
Motivation, values, attitudes and beliefs Clinicians perceived that the system would be easy to learn and would have minimal time burden. The hope was that it would relieve the burden of constantly reviewing patient data, reminding patients to enter data, and would identify changes automatically and earlier. Determining a cause was observed in every call.	“You know, I think with these complex Heart Failure patients that are not monitored from home,…there’s quite a number of them.. that could be benefited by having some monitoring and interaction and realizing somebody really cares about them..”(MD) <If they gained weight after Thanksgiving> “And understanding like if they were like, Oh, I don’t really have any shortness of breath, but I five pounds heavier on the scale, then I’d be like, and I ate a bunch of garbage. And I’d be like, Okay, well don’t eat a bunch of garbage… If they’re having shortness of breath, then I something would have to change.” (NP) “I think this will be another tool that will give us hopefully, better pre-selected information on this something that needs to be acted upon.”
Skills and Knowledge Both clinics were staffed by well-trained staff with high level of skill in cardiology and a broad scope of independent practice. The workflow tends to be of small medication changes based on symptoms and tracking the response to those changes closely.	“In general, but pharmacists do have a scope of practice. So we can prescribe under the scope and make adjustments and things like that…But essentially, we optimize medications in between clinic visits.” (PharmD) “However, based on the results of the heart failure index, it suggested elevated preload, so I increased his diuretics for a couple of days, reassessed early. I think I increased it on a Thursday, I reassessed on a Monday. Again, he continues to feel fine. But because he felt fine, I didn't think that there was a need to continue with his diuretics going forward, because he's prone to hypotension. So I cut it back to his usual baseline.”
Collaboration and teamwork, existing networks, local power, and authority The team members in both clinics work closely together and their communication tools vary substantially, using either Teams, secure messaging, CPRS notes, phone calls, or verbal communication. There is a real possibility of notifications reaching someone who is not available—coverage has to be planned.	“So we have a clinical pharmacist.. who is just amazing. And he is able to do phone visits and video visits .especially when starting. .new drugs,.. SGLT2i and the spironolactone…. There’s only three of us.” (MD) “Complex heart failure patients are best cared for, by teams of people. And I’m part of that team…but I would say that I’m the main point person for that patient… When the communication is from home, it’s either me or the RNs communicating for me.” (MD) “Or we'll do it orally communicating on a daily basis. And we know who are our sick ones are,.. - Hey, I haven't heard anything from Mr. So and So lately? Where's he at?… hunt him down what's going on?” (NP)
Context	Quotations
Organizational structure, senior leadership, and institutional resources The VA centers cover large geographical areas and use technology to provide high quality care. Both clinics have significant relationships with academic centers and have research infrastructure.	“Yeah, we have a large we have a large catchment area of about 200 miles. .”(MD) “ So what is the Med Action Plan? –” . .a whole separate application…The University pays for… it creates a medication list that is very aesthetically pleasing. And it's easy for our patient population to follow.” (RN) “So here in <this VA> my CPRS is different than the CPRS down in Arizona. So, everything is connected through JLV.”
Internal and external mandates and policies The VA has many national programs and reporting rules. An example is 14-day tracking after discharge.	“It’s a self-consult consulting service. In other words, we get a list that’s generated of who’s been admitted. Look through that list. That’s heart failure… Dr. xx (Attending) and the fellow, the cardiology fellow will go take up manage to impatient with those patients, and then we’ll put them into clinic within you know, seven to 14 days after discharge.”(RN)
Culture The clinical culture in both clinics is based on long-term working relationships, a long history of supporting research projects, and commitment to QI.	“… if it’s a difficult adjustment, or like there’s a lot of moving pieces that need to be addressed, then I’ll usually consult with the pharmacist or the nurse practitioner or the attending. I don’t necessarily cosign them to the note unless they request to be or unless I need a response from them. Or if it’s something that’s very tenuous situation that that I want to make sure that I have their signature on there to back me up.” (RN)
Evaluation and feedback processes, and learning environment Neither site had a single list (registry) of HF patients. They use a variety of methods to follow patients.	“And then it's really kind of tricky, because CPRS does not have a good way for us to like run a list or do data mining or anything like that. So, we have to be really careful on these patients, not to lose them to follow up….” (RN)

Patient stakeholder experience interview results

In total, 27 patients were enrolled in the pilot phase of the study and 12 of the patients were interviewed. All interviewed patients reported satisfaction with the usability of the equipment and the effectiveness of clinical support. Eight patients reported some initial difficulty in getting used to the equipment but reported it was “easy” and did not interfere with their general quality of life, social activities, or movement. Qualitative responses noted that they “felt safer” knowing they were being monitored. Three commented on some negative feelings of “being measured all the time” or to always being reminded of “how fat I am.” Ten expressed beliefs that the remote monitoring system would minimize trips to the clinic and to the hospital, which many veterans reported experiencing as a burden.

Behavioral and adoption data

Over a 90-day study period, 27 patients were enrolled, all were White males, with a mean age of 72.6 ± 8.6 years. Of these, 15 (55.6%) had HF with reduced ejection fraction, and 12 (44.4%) had HF with preserved ejection fraction. A total of 135 notifications were generated, including 38 repeat notifications for the same event. Of the 27 patients, 17 had at least 1 notification generated (63%) resulting in a median of 5 notifications per patient. Of the notifications that were responded to, clinicians responded to > 95% within 24 hours. In 26.7% of the notifications, clinicians intensified the patient’s medication regimen, scheduled a follow-up clinic appointment or referred the patient to the emergency department. In 40.7% of the cases, clinicians monitored the patients over the following 72-hour period. For 32.6% of the notifications, clinicians did not achieve patient contact. In 28.2%, multiple notifications were generated during 72 hours following the initial notification; clinicians identified these as repeated notifications and did not pursue contact. In 4.4% of the notifications, contact efforts were unsuccessful (Table 4). Of the 27 patients, 7 were hospitalized and 2 died.

Table 4.

RE-AIM based tracking metrics.

Construct	Description/definition	Metrics	Data source
Reach - Patients - Clinicians	Who is the target patient population for the LINK intervention? How many are agreeing to enroll? How long do they stay in the program? Which clinicians are involved and what are their roles?	Patients: 27 consented Clinicians: SLC: 9 (5 MDs, 1 NP, 2 RNs, 1 PharmD) GVL: 5 (1 MD, 2 RNs, 2 ARNPs)	Research coordinator tracking
Implementation	How is the notification understood by clinicians and patients? How can the notification be integrated into the workflow and communication strategies? What is the length of time to respond to notifications? Is the documentation chain complete? What technologies are involved in documenting the notification, communicating the notification, and integrating the notification into practice?	Qualitative interviews. Median (IQR) response times to notification:	Team OpenClinicaa
		System response times Email received to notification acknowledgement by clinicians: 2.5 (0.2-4.2) h Acknowledgment by the clinicians to patient contact: 4.2 (2.2-8.4) h Email received to patient contact: 7.8 (4.2-31.1) h	VA CPRS
		Responses to notification (count): Medication titration: n = 19 Clinic visit: n = 14 ER visit: n = 3 Increased monitoring: n = 55 Repeated notifications: n = 38 Unable to contact: n = 6	OpenClinicaa Research coordinator
Effectiveness	Is the monitoring platform working? How well do the patches work? (# that fail, days of nonrecordings).	Mean study duration SLC: 75.1 days GVL: 74.0 days Mean duration with recorded data SLC: 61.0 days GVL: 56.5 days Time with recorded data SLC: 81.1% GVL: 74.4% Time with good signal quality SLC: 96.5% GVL: 98.8%	OpenClinicaa VA CPRS note
Adoption	Individual provider level	Qualitative data on satisfaction and reports of burden	Interviews at culmination of study
Adoption	Clinic level	Descriptive data on resources use, enrollment, and dropout rates	Interviews at culmination of study
Maintenance	Program level	Annual and monthly reports of patients’ enrollment and dropout rates, data on provider resources, and cost of program	VA CPRS annual and monthly reports

^aOpenClinica is the study database linked to the PhysIQ platform.

The clinicians’ behavioral responses varied. All clinicians used the same general approach to respond to notification: (1) Read the notification and determine when to call the patient; (2) review in the patient’s EHR medications, labs, and recent appointment notes; (3) call the patient and assess the patient’s clinical status, asking specifically about weight gain, blood pressure, dizziness, dyspnea, edema, and current medication profile. The order of the questions varied. Only in 1 of the 5 observations did the clinician review data on the remote monitoring portal. Clinicians waited to make calls depending on their time burden and whether or not they knew the patient. Some patients had several notifications in a row, which resulted in some clinicians responding the following day. Observed chart review lasted 3-15 min and calls lasted between 12 and 22 min. There was a variation between clinicians on how much they engaged the patients in problem-solving. The template used in the progress notes to document notification responses did not pull in clinical data, which created extra effort. The template did provide some decision support instructions.

Implementation implications: The sensitivity, frequency, and information display of the notifications was modified based on this initial experience. The major reasons were: (1) Notifications were sent out repeatedly for the same event, resulting in clinician notification fatigue and complaints; (2) the information provided to clinicians on the web-based portal was expanded in order to assist the clinician with interpretation and to encourage portal use; and (3) the decision options were expanded from the original 3 options to 4, in order to include an “Increased monitoring” (or “watch and wait”) option based on clinicians’ suggestions.

Thematic description and recommended strategies

The results of data collection across interviews, behavior, and system performance were integrated in order to design initial implementation strategies. The thematic results and their implementation implications are as follows:

Theme 1: Anticipatory expectations for reliable patient communications, reduced patient burden, and less proactive provider monitoring were high : Clinicians expected that the system would replace the need for patients to supply information proactively, making the information flow more reliable and less burdensome on patients. A noninvasive intervention was expected to minimize harm and maximize access. Observations of the early patient-provider calls indicated a desire for both the patient and the clinician to identify “causes” resulting in a notification.

Implementation implications: Both the patient and the clinician expected that if they receive a notification, a cause could be identified. More information could be included with the notification and the EHR could be programmed to “pull in” critical patient history that could be reviewed “at a glance” as part of a note template.

Theme 2: The degree of trust required for the notification is associated with both the action implications and the role. RNs need to trust that the MDs are in agreement regarding the action decisions and would “have their back.” In other words, RNs will revert to “script” and trust the real-life patient reports more than the notification if they are not sure of their support.

The other component of trust for RNs is that the system is accurately “catching” patients with problems, thereby minimizing their constant monitoring work.

However, MDs tended to assume that the notification would provide more actionable information that goes beyond symptoms, but is based on the AI findings. Without specific decision support, MDs might then also simply go back to relying on patient symptoms.

Implementation implications: The notification might be perceived as more trustworthy and therefore more actionable by RNs if clinic leadership explicitly embedded it into current scope of practice protocols for RNs. For MDs, the notification may require both more information from the portal regarding the basis of the notification as well as specific action recommendations.

Theme 3: Prior clinic experience with other remote monitoring local programs influenced utilization : Sites appear to have significant and varied experience in home monitoring and other complex monitoring programs that include remote service, telecommunication tools, and experienced staff. The result is variability in both the monitoring components and the information/communication tools used.

Implementation implications: Key program workflow components that need to be addressed include: (1) Identification of high-risk patients; (2) assignment of responsibility for tracking; (3) assignment of responsibility for making calls; (4) specifying data exchanged (EHR, patient, monitoring devices), (5) documentation of content in notes; and (6) definition of rules for who else is informed. Communication flow should be assessed before implementation and the tools need to be standardized across the clinic.

Theme 4: Notification response involved significant effort, including EHR review, patient contact, and consultation with other clinicians : The conversation usually involved more than talking with the patient, but also reviewing several other data sources in EHR as well as other provider notes and plans. This requires a mental model of the care network and it helps when clinicians know the patient.

Implementation implications: Efforts will be needed to minimize burden for clinicians, such as enrolling the patient in the continuity HF clinic so that they are familiar to staff (an action that was implemented), pulling in key information such as medications, lab values, etc. from the EHR into the monitoring note template to reduce the need for chart review, and possibly asking the patients to send in their vital signs and weight before the follow-up call if increased monitoring was selected in response to a notification.

Implementation tracking metrics

Designing RE-AIM constructs to be used during the randomized phase of the pragmatic trial was an important implementation strategy (Table 4). The identified metrics were recommended to be used by the implementation team to monitor specific strategies and to track progress.

Discussion

Summary of findings

This pilot implementation phase of the LINK-HF2 pragmatic trial identified important items for implementation planning of an ambulatory monitoring intervention that provides clinicians with an AI generated notification indicating high likelihood of incipient HF exacerbation. Behavioral tracking of clinician’s response to notifications showed adequate response times. Observation of the 2 early intervention sites showed varied workflow patterns, communication tools and barriers to integrating the process into workflow. The results of the implementation work were used to modify study procedures in the randomized phase of the pragmatic trial. These changes included mapping communication technologies and notification displays to different roles, enhancing information displays to improve explainability, reducing work burden by adding an “Increased Monitoring” response option and increasing provider and patient education. In addition, the results informed the development of implementation process metrics to be used during the randomized phase of the study.

Social structures and communication patterns related to AI decision support

Our findings highlighted the necessity to tailor processes related to AI facilitated decision support to the particular healthcare setting, since we identified notable differences even among just 2 clinics. Variations in clinic structure affected how patient care was managed, the team dynamics, collaboration, and the use of communication technologies. Crucial elements in the implementation of AI decision support include social influences and group dynamics, resonating with Shaw and Greenhalgh’s concept of technology adoption as a social process.^12,37–39 Effective implementation in clinics relies on establishing trust and interpreting AI notifications within their proper context.⁴⁰

The utility of AI notifications varies according to how they are integrated into the clinical expertise of different roles, such as pharmacists, nurses, and doctors, each adapting the information to their specific needs.^38,39,41 Implementation strategies need to accommodate these diverse applications. While clinicians generally anticipate AI to enhance workflow and decrease mental burden, actual responses to AI notifications highlighted a need for clearer actionable guidance.

The primary challenge in implementation is to strengthen the link between AI notifications and clear action steps. A lack of explicit direction raises the potential for automation bias and increased cognitive strain.⁴² Employing visual aids and automated documentation tools may be crucial in alleviating cognitive demands.^40,42,43

Clinician mental models of AI

Another major finding was the variety of cognitive interpretations and mental models clinicians employed in response to AI notifications. The notification itself came with limited information or decision rules. The clinicians tended to engage in habitual schematic responses when calling the patient. If the patient had no current symptoms, no change in vital signs, and no increase in weight, then the incentive to make a therapeutic change was lower, and the increased monitoring choice was a frequent response (this was especially true for nurses). However, since the AI analytics are likely to detect HF exacerbation before overt changes in signs and symptoms, a timely intervention in the absence of other clinical warning signs might be more effective but requires very explicit guidance.

Implementing AI decision support is unique, as the guidance provided through analytics is often viewed as a “black box.” The concept of “explainability” has emerged in previous studies as critical to physician trust and adoption, and a variety of recommendations on how to address it have been proposed.^44–48 Combi et al. described a model of explainability that includes interpretability—the user can predict what the AI recommendation would do, understandability—the user has an accurate mental model of how it works, usefulness—the recommendations are useful and guide action, and usability—the system is easy to use.⁴⁹

Further work is needed on design an interface to improve all aspects of AI decision support, including role-based targeted explainability, links to role-based specific actions, and careful attention to the cognitive load needed for interpretation.⁴⁵ In our study, clinicians had an option to access a clinician portal with granular data, but few did. The notification did not have guidance on which specific action should be taken, nor was it integrated with other EHR data, resulting in increased cognitive load. Similar barriers have also been described by others.⁴⁵ Further, the assessments of provider attitudes to AI-enabled decision support should go beyond determining whether the attitudes are positive or negative, and explore provider understanding of the AI data. Holzinger et al. suggest new human-AI interfaces are needed to enable clinicians to re-enact and retrace AI results and as such improve their level of understanding of the results.⁵⁰

Our study has certain limitations. We only used 2 sites in the pilot work, and the patient and clinician sample were relatively small. However, we interviewed all team members that interacted with the system and the sites were geographically diverse and varied in their work processes. Although the clinician participants received training in the use of the remote monitoring system, no evaluation of skills was conducted.

Conclusion

We used implementation results from a pilot phase of a study of AI-based analytics in HF to develop strategies to address communication technology, patient and clinician education and EHR integration. The resulting modifications will increase efficiency and reduce shareholder demand in the randomized phase of the study. Our results also provide important insights into a broader application of AI analytics into clinical workflow.

Author contributions

Study concept and design: C.W., K.S., C.S., M.P., N.L., K.S., B.B., T.H., R.S., C.P.K., I.T., L.B., J.M.B., J.S.; data collection and analysis: K.L., P.H.T.; recruitment of participants: H.S., T.C., C.K.; drafting of the manuscript: C.W., K.S., J.M.B., J.S.

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

This work was supported by Veterans Health Administration HSR&D Merit Review (1 I01 HX002922-01A2, Principal Investigator: Stehlik J).

Conflict of interest

J.S. serves as a consultant to Natera, Medtronic, and TransMedics and received research support from Natera and Merck. M.P., K.L., and P.-H.T. are employees of PhysIQ, the provider of the platform. The remaining authors have nothing to disclose.

Data availability

The data underlying this article are available in the article and its online supplementary material.