Clinical informatics accelerates health system adaptation to the COVID-19 pandemic: examples from Colorado

Abstract

Objective

Large health systems responding to the coronavirus disease 2019 (COVID-19) pandemic face a broad range of challenges; we describe 14 examples of innovative and effective informatics interventions.

Materials and Methods

A team of 30 physician and 17 nurse informaticists with an electronic health record (EHR) and associated informatics tools.

Results

To meet the demands posed by the influx of patients with COVID-19 into the health system, the team built solutions to accomplish the following goals: 1) train physicians and nurses quickly to manage a potential surge of hospital patients; 2) build and adjust interactive visual pathways to guide decisions; 3) scale up video visits and teach best-practice communication; 4) use tablets and remote monitors to improve in-hospital and posthospital patient connections; 5) allow hundreds of physicians to build rapid consensus; 6) improve the use of advance care planning; 7) keep clinicians aware of patients’ changing COVID-19 status; 8) connect nurses and families in new ways; 9) semi-automate Crisis Standards of Care; and 10) predict future hospitalizations.

Discussion

During the onset of the COVID-19 pandemic, the UCHealth Joint Informatics Group applied a strategy of “practical informatics” to rapidly translate critical leadership decisions into understandable guidance and effective tools for patient care.

Conclusion

Informatics-trained physicians and nurses drew upon their trusted relationships with multiple teams within the organization to create practical solutions for onboarding, clinical decision-making, telehealth, and predictive analytics.

INTRODUCTION

On March 5, 2020, the first case of coronavirus disease 2019 (COVID-19) was reported in the state of Colorado. Since then, as of May 17, 2020, the number of cases has ballooned to over 21 000, and more than 1100 deaths due to COVID-19 have been reported in the state.¹

Unlike the rapid spike of COVID-19 cases in New York City ,² the rise of cases in Colorado occurred at a slower pace, granting health care institutions some time in March 2020 to ready their physical spaces, workforces, and information systems for the anticipated surge of patients while rapidly implementing infection control solutions. Our 12-hospital integrated health system leveraged electronic health record (EHR) and informatics strategies to meet these challenges, as have others.^3–6

OBJECTIVE

This report describes 14 examples of how collaboration, empathy, design, and agile huddles quickly solved complex, undefined problems.

MATERIALS AND METHODS

UCHealth is an integrated health system comprised of 12 hospitals and >800 clinics throughout Colorado, Wyoming, and Nebraska, that cares for >1.9 million active patients. UCHealth operates a commercially available EHR (Epic, Verona, WI) used by more than 6000 clinicians and 25 000 staff.

In early March 2020, UCHealth invoked the Incident Command Structure (ICS) commonly used during disasters to organize a systematic response.^7,8 The informatics team, comprising 30 physician informaticists (PIs) and 17 nurse informaticists already well-known as change agents, partnered with ICS leaders to translate decisions into highly usable and understandable workflows in the EHR.

The informatics team reorganized to increase agility. We set up a recurring 30-minute virtual meeting called the Daily JIG (Joint Informatics Group) to facilitate rapid decision-making and coordination. With as-needed input from expert guests from different services (eg, infectious diseases, pulmonary medicine, palliative care, pharmacy, clinical lab), the Daily JIG developed EHR tools and workflows to guide care across the system.

The informatics team encountered fast-paced, chaotic, and emotionally charged challenges related to COVID-19. Rumors and outdated information had the potential to cause disagreements and ineffective care. However, the Daily JIG, powered by existing relationships across the health system, helped move accurate information up and down the organizational hierarchy, thereby facilitating rapid decision-making, increasing trust, and accelerating our ability to act. The informatics team leveraged the EHR and other information technology tools to disseminate rapid changes across the health system to meet 14 main clinical challenges (Table 1). These challenges arose in late March, April, and May 2020 with solutions designed and implemented in days to weeks. These solutions were permanent changes applied system wide to all clinicians and patients.

Table 1.

COVID-19 challenges addressed with informatics

Category	Topic
Onboarding	Rapidly onboard hundreds of nurses to work safely in our hospitals
	Train a large pool of clinicians to “stand by” for possible hospital service
Clinical decision-making	Rapidly update clinical decision guides for SARS CoV-2 testing and treatment
	Restrict prescriptions to avoid shortages, without affecting COVID-19 prescribing
Telehealth	Train thousands of clinicians to perform video visits within 2 weeks
	Teach best practices for video visit communication
	Improve postdischarge monitoring of patients with COVID-19
	Use tablets in the hospital to improve communication with patients and families
Collaboration	Bring together hundreds of physicians across health systems to solve problems
	Keep clinicians aware of patients’ changing COVID-19 status
	Improve the use of Advance Care Planning tools and documentation
	Reduce pandemic-related communication barriers with patients and families
Prediction	Decide who receives lifesaving therapy under Crisis Standards of Care
	Predict future COVID-19 hospitalization rates

Abbreviations: COVID-19: Coronavirus Disease 2019; EHR: electronic health record; SARS CoV-2: Severe Adult Respiratory Syndrome Coronavirus 2.

RESULTS

Onboarding

How might we rapidly onboard hundreds of nurses to work safely in our hospitals?

To prepare for a possible surge in hospitalizations, we needed to retrain hundreds of nurses on inpatient care processes. Inpatient nurses typically require 8–16 hours of EHR classroom training, which was not possible due to social distancing requirements. Moreover, nurses varied in their level of comfort with electronic tools from older “boomers” to postmillennial smartphone wizards.

We worked with an innovation partner (Amplifire, Boulder, CO) whose core technology is the online assessment of a learner’s “confidently held misinformation.” The learner is presented with essential patient care actions in the EHR. If highly proficient, one can complete the exercise in 55 minutes. If not, the tool reviews the “misinformation” or knowledge gap, educates, and reassesses until all elements are mastered, typically in just over 2 hours.

Over 240 nurses completed the online training and reported high levels of satisfaction: 75% “enjoyed this teaching tool,” and 72% agreed that “this tool helped me learn strong EHR skills.”

How might we train a large pool of clinicians to “stand by” for possible hospital service?

Nearly 1000 clinicians in our health system volunteered for hospital work. Most had no recent experience in hospital medicine or in using the EHR in an inpatient setting.

A team of 5 of clinicians and a PI created a comprehensive training guide with a series of embedded videos and linked resources (Supplementary Material 1).^9,10 Built for rapid, flexible, and on-demand use, the guide provided instruction not only on EHR skills, but also hospital-specific details, such as rounding, multidisciplinary interactions, and common inpatient conditions and complications. The guide was posted on our intranet, within the EHR, and also embedded in a “smart-link” that could be accessed by typing “.inpatienthelp” at any point in the EHR workflow.

Clinical decision-making

How might we rapidly update the clinical decision guide for SARS CoV-2 testing and treatment?

During March and April 2020, ICS recommendations changed frequently on diagnostic testing, treatment, and cohorting of patients. We needed a tool, such as a clinical pathway document, to provide up-to-the-minute Severe Adult Respiratory Syndrome Coronavirus 2 (SARS CoV-2) education to frontline clinicians. As static resources, however, clinical pathway documents frequently fall out of date and out of use.

Prior to the COVID-19 crisis, we had partnered with a third-party pathways vendor (AgileMD, San Francisco, CA) and had integrated a number of nonpandemic clinical pathways into the EHR. Further, we had PIs who were dually trained to be both EHR “physician builders” and creators of AgileMD clinical pathways. Physician-designed interactive pathways provided a quick visual understanding of a patient’s current status and immediate ordering of appropriate tests, treatments, and referrals (Supplementary Material 2–8).¹¹

Early on, SARS-CoV-2 nucleic acid amplification test (NAAT) capacity was limited; but as capacity increased, the criteria for testing (eg, symptomatic admissions, immunocompromised patients, asymptomatic preoperative patients) changed almost daily. Within minutes of leadership decisions, PIs translated new testing and treatment recommendations into EHR-based pathways. Treatment categories included prone positioning, clinical trials, convalescent plasma therapy, antivirals, anticoagulation, intubation checklists, septic shock, and hyperinflammatory response treatment (Supplementary Material 2, 3, 5). The PIs were well-positioned to understand clinical workflow and engage in iterative design to deliver high usability. Using the tools integrated into the EHR, we were able to make frequent adjustments to match changing recommendations. In the first 2 months, over 1600 clinicians used the pathway to guide testing, treatment, palliative care, and postdischarge follow-up for 6000 patients (Figure 1).

Figure 1.

Cumulative COVID-19 visual pathway use. Patient encounters with use of pathway (brown). Unique providers using pathway (purple) at UCHealth.

Internal statistics showed that hospitals in 1 region had a higher mortality for COVID-19 patients in March 2020 compared to other regions. Our pathway launched at end of March. April 2020 statistics showed a major reduction in mortality in that region, which is now comparable to the rest of the health system.

How might we restrict prescriptions to avoid medication shortages, without affecting COVID-19 prescribing?

Hydroxychloroquine, azithromycin, and albuterol had been widely popularized as treatments for COVID-19, and there was concern that overprescribing of these medications might cause local shortages. On the other hand, no one wanted to discourage prescribing these agents for approved conditions.

Pharmacy leaders brought this concern to our Daily JIG huddle, and, in a single meeting, agreed to the following plan. First, we required an “Indication of Use” for all outpatient hydroxychloroquine and azithromycin prescriptions¹² to discourage unapproved uses. Next, all United States Food and Drug Administration (US FDA)–approved indications were listed with checkboxes, with “COVID-19” absent, which forced prescribers to manually type an indication. Then we created an alert for the prescriber anytime these drugs were ordered, discouraging their use for patients other than those hospitalized with COVID-19 or who required ongoing treatment of a chronic illness (eg, rheumatoid arthritis, pneumonia). Finally, we provided guidance for albuterol prescriptions: for hospitalized patients with COVID-19, we recommended inhalers over nebulizers to reduce the aerosolization risk to our staff; for hospitalized patients without COVID-19, we recommended nebulizers over inhalers to preserve the supply of inhalers. During the following weeks, we saw a flattening of the prescription volume for hydroxychloroquine and azithromycin and only a brief, slight increase in albuterol prescribing.

Telehealth

How might we train thousands of clinicians to perform video visits within 2 weeks?

The Stay-At-Home order in Colorado was issued on March 26, 2020. At the time, UCHealth had video visit technology (Vidyo, Hackensack, NJ) integrated into the EHR, but very few clinicians and staff had used it. As the number of in-person clinic visits plummeted, the demand for video visits rose dramatically across the health system (Figure 2).¹³

Figure 2.

In-person and video visits per week, Jan–May 2020. In-person visits (red) and video visits (green) for the UCHealth system.

Our existing EHR Sprint optimization team, including informaticists, project managers, and trainers¹⁴ joined with the centralized telehealth team¹⁵ to support a Virtual Health Command Center. We deployed a live learning platform provided by ECHO, a local nonprofit company.^16,17 Telehealth and Sprint staff led training webinars hourly from 8 am to 5 pm each weekday for 2 weeks, educating 1500 clinical faculty and staff. In coordination with coding and billing teams, this group developed a library of video visit and work-from-home best practice documents posted centrally on the EHR education dashboards (Figure 3). Within 2 weeks, UCHealth went from 350 to over 20 000 video visits per week, a 5700% increase.

Figure 3.

EHR dashboard with tip sheets on video visits and work from home resources.

How might we teach best practices for video visit communication?

For years, our organization had taught patient experience and communication¹⁸ skills for in-person visits to our clinicians and staff. However, video visits are new to both clinicians and patients. We wished to apply our expertise to help clinicians optimize video visits with patients. Coordinating with the patient experience team, we set up online webinars attended by over 300 clinicians across different disciplines. These webinars taught the principles of nonverbal communication, reflective listening, and other strategies known to improve patient outcomes. The content was distilled into a 1-page tip sheet, “Improving the Patient Experience with Virtual Visits” (Supplementary Material 10), and the recorded webinar was posted on YouTube for dissemination.^19,20

How might we improve postdischarge monitoring of patients with COVID-19?

COVID-19 has a biphasic course in some patients,²¹ and hospital clinicians worried that the clinical status of discharged patients might worsen later at home. At the same time, facing a growing inpatient census, there was a need to discharge recovering patients whenever possible.

Our health system had an existing team of nurse care coordinators who would call and speak with patients after discharge to ensure a smooth transition from hospital to clinic care. However, with rising numbers of patients with COVID-19, we could see the need for postdischarge monitoring ballooning into hundreds or thousands of patients.

Our health system partnered with an external vendor (Masimo, Irvine, CA) to deploy a wearable device to transmit a patient’s respiratory rate, heart rate, and pulse oximetry. We placed wearables on “high risk” patients with COVID-19 (eg, those who were pregnant, older, or had significant chronic illness) at discharge. A single Virtual Health Center (VHC) nurse could simultaneously monitor up to 40 patients with wearables. The VHC nurse was in close communication with the VHC supervising physician, the hospital team, and the patient’s primary care team.

We have placed the wearable on 40 patients and have “graduated” 18 from the program after a monitoring period of 8 days. Several patients have needed new home oxygen orders, but none required rehospitalization. One patient was “rescued” from acute hypoxemia when the patient failed to recognize that the home oxygen tank was empty!

How might we use tablets in the hospital to improve communication with patients and families?

The informatics team redeployed a supply of tablets for each inpatient COVID-19 unit to enhance communication with patients and families, conserve personal protective equipment, and minimize staff exposure to infectious patients. Some examples:

• A tablet with a remote translator service was placed in every non–English-speaking patient's room.

• The palliative care team coordinated frequent video conferences with the patient, a translator, the primary team, and multiple family members. At times, there were 20 participants in a family meeting, including international participants.

• An in-room tablet allowed a single member of a team to perform a physical exam while the remainder of the team interacted remotely from the hallway.

• Birth registrars used tablets to direct mothers of newborns and their partners at home to complete birth certificate forms. Tablets were also used to connect families with their newborns.

• Psychology and rehabilitation patients used tablets and video visits to resume group therapy sessions.

• When many single-bed intensive care unit (ICU) rooms were converted to 2-bed rooms, the second bed’s ventilator and vital signs monitors could not be connected to the hospital network. Nurses initially posted an assistant in the doorway to the room to listen for alarms from the second bed. Subsequently, nurses placed a tablet in the room to capture live audio and video of the second bed’s ventilator and monitor displays for transmittal via a Team’s video conference to the nursing station. This way, multiple “second bed” patients could be monitored from the nurse’s station alongside the monitors for the other beds. This “Apollo 13”²² creative approach quickly solved a sticky technology problem (Figure 4).

Figure 4.

Tablet-to-tablet video overcomes ICU connectivity limitations. Inset shows iPad mounted on rolling stand to capture monitor data for Bed B. iPad at nurse’s station displays vital signs with audio and visual alarms from Bed B. ICU rooms are designed for only 1 bed, therefore there are no network jacks for data from the second bed, and nurses at the nursing station cannot see or hear alarms from second bed. Tablet-to-tablet connection over wi-fi cleverly overcomes this technical limitation and can monitor 4 or more rooms simultaneously.

Collaboration

How might we bring together hundreds of physicians across health systems to solve problems?

In the pre–COVID-19 era, a group of pulmonary and critical care physicians from competing health systems across the state of Colorado would periodically confer about difficult cases, usually in 1-to-1 phone consultations or small informal meetings. At the onset of the pandemic, these physicians faced a deluge of information and needed to organize the data, build consensus when standards of care did not exist, and design a plan of action based on rapidly evolving evidence. There was a growing and immediate need to connect the group in an effective and more inclusive way.

A pulmonary PI created a Team’s online site and invited colleagues throughout the region to aggregate knowledge. These frontline intensivists organized themselves into 25 discussion groups focused on topics including ventilator management, pharmacotherapy, e-ICU (video consults), and resuscitation procedures (Supplementary Material 9). Clinicians posted new ideas and experiences, and these discussions informed the creation and updating of a public website for community providers (Figure 5)²³ and ICU clinical pathways described above (Supplementary Material 4, 5, 6).

Figure 5.

COVID-19 clinical practice documents published to the internet. Document contents can be viewed here: https://www.uchealth.org/today/clinical-practice-documents/.

Over 300 pulmonary/critical care physicians participated. The community used this asynchronous sharing site to have parallel, organized discussions, build consensus, and implement recommended practices. Over 2 months, more than 1000 clinicians viewed reference documents over 7000 times, unifying best practices across at least 6 regional health systems.

How might we keep clinicians aware of patients’ changing COVID-19 status?

Initially, our organization relied on the standard “travel screen” question, “Have you traveled outside of the US?” to identify patients with possible exposure to SARS CoV-2. However, it soon became clear that better logic was needed to identify and track patients needing isolation. Therefore, the informatics team, advised by lead infectious disease physicians, created logic for 3 levels of COVID-19 chart alerts. These alerts are triggered automatically when medical assistants document patient symptoms (eg, fever, cough, shortness of breath) at clinic check-in or when specific orders, test results, or diagnoses are entered in the EHR (Table 2).

Table 2.

COVID-19 chart alerts to warn clinicians of infection risk

Trigger	Alert	Duration of alert
Possible symptoms24	CAUTION: Possible COVID-19. Place mask on patient and isolate immediately.	48 hours, auto-discontinue. Short duration in case of alternate diagnosis. Allows time for test result to return.
Possible diagnosis or SARS CoV-2 NAAT ordered	CAUTION: Possible COVID-19. Place mask on patient and isolate immediately.	15 days, auto-discontinue. Alert extends until discharge for patients hospitalized for >15 days. Alert flags newborns aged <15 days if mother had suspected or confirmed COVID-19.
Confirmed diagnosis or Positive SARS CoV-2 NAAT result	CAUTION: Confirmed COVID-19. Place mask on patient and isolate immediately.	15 days, auto-discontinue. Repeat positive test result renews the flag for an additional 15 days.
Negative SARS CoV-2 NAAT result	Cancels all flags.

Abbreviations: COVID-19, Coronavirus Disease 2019; NAAT, nucleic acid amplification test; SARS CoV-2, Severe Adult Respiratory Syndrome Coronavirus 2.

Furthermore, if a “COVID-19 Confirmed” diagnosis appears within 90 days of the index date, a new flag is NOT added and this is NOT counted as a new infection without the presence of a new positive SARS CoV-2 NAAT result, as this diagnosis code may be used by a clinician seeing a follow-up case after hospitalization. We incorporated SARS CoV-2 NAAT results from other regional labs or health systems, either manually or through an electronic interface.

We did not use the following information to trigger the alerts above: SARS CoV-2 antibody tests, patient-reported “Coronavirus concern,” diagnosis codes for “COVID-19 exposure” or “COVID-19 concern.”

How might we improve the use of advance care planning tools and documentation?

Most patients have not completed advance directives, despite years of work by informaticists and clinicians. During the pandemic, our organization’s ICS mandated that advance care planning be addressed with each patient at every inpatient, ambulatory, or emergency department encounter.

Patients already had a way to complete their Medical Durable Power of Attorney document online via the EHR’s patient portal.²⁵ However, there was no standard location in the EHR to document a discussion of advance care planning. These narrative discussions were commonly written in nonindexed ambulatory progress notes that were difficult to search. Inpatient EHR tools allowed a “code status” and highlighted advance directives in the patient banner, but this was inconsistently used.

We invited our palliative care colleagues to a Daily JIG meeting and discussed the challenges of making advance directives easier to document, track, and highlight for inpatient, emergency department, and ambulatory settings. We discussed the proposed solutions for each setting and created consensus for a family of solutions. Within 2 weeks, note templates were created with bookmarks that would result in a unified advance directive listing in a central chart location (Storyboard in Epic EHR) that was easily accessible by clinicians in all settings. Under normal circumstances, this process of problem-solving and implementation might have taken months.

How might we reduce pandemic-related communication barriers with patients and families?

Patients with COVID-19 are cohorted in hospital isolation units. In most cases, patients may not have visitors. This is good medical practice, but it is heartbreaking for families who cannot be present when needed most.

A group of nurses who were not able to work in the hospital (eg, because they were pregnant and could not risk exposure to SARS-CoV-2 or because operating rooms were idled) recognized an opportunity to help. The nurses used an instant messaging system within the EHR (Epic Secure Chat) to communicate with bedside nurses, physical therapists, chaplains, and social workers involved in the care of hospitalized patients with COVID-19. They viewed the patient’s chart from home, called into hospital rounds, and then updated the family on the patient’s status.

Calling themselves Nurse Communication Liaisons (NCLs), they facilitated communication for COVID-19 patients in medical/surgical units and intensive care units.²⁶ One NCL nurse noted, “I love secure chat because it means that the bedside nurse who is gowned and gloved doesn’t have to scrub out to answer a phone call; they can catch up with chat-messages when there’s a break in the action.” Another NCL nurse commented, “Normally when I’m at the bedside, I’m always trying to ‘wrap up the conversation’ with family; there are so many other things needing my attention. [At home] I can really feel good about being focused, connecting with family, and freeing up the bedside nurse to do their jobs.”

Prediction

How might we decide who receives lifesaving therapy under crisis standards of care?

Following the guidance set by a coalition committee formed by the State of Colorado, the informatics team partnered with an academic data science team to create a semi-automated scoring tool based partly on the Charlson Comorbidity Index²⁷ and the Sequential Organ Failure Assessment.²⁸ The resulting patient scores would be used by the health system’s governing ethics group if and when Crisis Standards of Care²⁹ were invoked at the state level and rationing of ventilators and other life-saving therapies became necessary. The construction of these tools, understandably, provoked strong emotions and ethics-oriented discussions.

An EHR-based form was constructed to aggregate data from the EHR (such as diagnoses, medications, and lab results) and facilitate the capture of new data needed for the calculations (Supplementary Material 11). The scores were inserted into a patient list display column in the EHR. We shared our efforts so that health systems in Colorado could follow a consistent process. In mid-April 2020, our health system required this form to be completed for each patient within 48 hours of admission.

How might we predict future COVID-19 hospitalization rates?

As Colorado relaxes “Stay-at-home” restrictions, there is a risk of future waves of hospitalizations. One way to prepare for this is to create a leading indicator. A leading indicator is akin to using seismometers that detect minor earthquakes to predict the onset of a major earthquake. Colorado currently does not have widespread community SARS CoV-2 infection screenings, the results of which could function as a leading indicator. Therefore, alternatives are urgently needed.

With the goal of predicting regional hospitalization rates, UCHealth data scientists and informaticists are examining trends in the EHR data including outpatient “chief complaints” (eg, fever, cough, shortness of breath), the new COVID-19 ICD-10 diagnosis codes, and the results of SARS-CoV-2 NAATs from internal and regional laboratories that are part of EHR-based health information exchanges.^30,31 Socioeconomic variables, such as population density, median income, age distribution, and density of long-term care facilities have been tested for their predictive value. These variables will be evaluated in relationship to patient residence and clinic ZIP codes in order to predict hospitalization rates by region. We are in the midst of tuning the predictive model.

DISCUSSION

During the onset of the COVID-19 pandemic, the UCHealth Joint Informatics Group, or JIG, applied a strategy of “practical informatics” to rapidly translate critical leadership decisions into understandable guidance and effective tools for patient care. Informatics-trained physicians and nurses drew upon their trusted relationships with multiple teams within the organization to create practical solutions for onboarding, clinical decision-making, telehealth, and predictive analytics.

The structure and function of the JIG was reminiscent of the US Task Force described in Team of Teams, in which General Stanley McChrystal³² described several keys to Task Force success against Al Qaeda. One key was to develop a “shared consciousness” by disclosing critical information every day across previously compartmentalized teams. Another key was to build personal trust between members of naturally segregated and secretive teams. Furthermore, the Task Force set strategic goals and encouraged distributed decision-making. Avoiding micromanagement, McChrystal empowered teams to quickly make local decisions that were in line with the Task Force strategy. Similarly, the Daily JIG coordination huddle created a version of “shared consciousness” and spawned numerous solutions. The informatics team had previously established good working relationships with health system clinical and administrative leaders across different specialties and services, which enabled rapid collaboration and distributed decision-making in every part of the organization.

Other groups have described early informatics team responses to the pandemic crisis.^3–6 This report emphasizes that trusted relationships enable informatics teams to perform well during crises. It complements previous findings that an effective informatics team and EHR-based interventions can be swiftly deployed to educate clinicians to act in accordance with up-to-the-minute diagnostic and treatment recommendations.

The main limitation of this report is that it describes tactics that were effective in 1 health system that may not apply elsewhere, given regional variations in the prevalence of COVID-19 and its impact on health care resources.

CONCLUSION

We are far from the end of the COVID-19 pandemic. We hope that our approach and examples of informatics solutions may be of potential use to other institutions in the time to come.

FUNDING

This work received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

AUTHOR CONTRIBUTIONS

All authors acknowledge that they: 1) made substantial contributions to the conception or design of the work, or the acquisition, analysis, or interpretation of the data for the work; 2) contributed substantially to the drafting and final approval of the version to be published; and 3) agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.