Making connections: nationwide implementation of video telehealth tablets to address access barriers in veterans

Abstract

Background

Video telehealth technology has the potential to enhance access for patients with clinical, social, and geographic barriers to care. We evaluated the implementation of a US Department of Veterans Affairs (VA) initiative to distribute tablets to high-need Veterans with access barriers.

Methods

In this mixed methods implementation study, we examined tablet adoption (ie, facility-level tablet distribution rates and patient-level tablet utilization rates) and reach (ie, sociodemographic and clinical characteristics of tablet recipients) between 5/1/16 and 9/30/17. Concurrently, we surveyed 68 facility telehealth coordinators to determine the most common implementation barriers and facilitators, and then conducted interviews with telehealth coordinators and regional leadership to identify strategies that facilitated tablet distribution and use.

Results

86 VA facilities spanning all 18 geographic regions, distributed tablets to 6 745 patients. Recipients had an average age of 56 years, 53% lived in rural areas, 75% had a diagnosed mental illness, and they had a mean (SD) of 5 (3) chronic conditions. Approximately 4 in 5 tablet recipients used the tablet during the evaluation period. In multivariate logistic regression, tablet recipients were more likely to use their tablets if they were older and had fewer chronic conditions. Implementation barriers included insufficient training, staffing shortages, and provider disinterest (described as barriers by 59%, 55%, and 33% of respondents, respectively). Site readiness assessments, local champions, licensure modifications, and use of mandates and incentives were identified as strategies that may influence widespread implementation of home-based video telehealth.

Conclusion

VA’s initiative to distribute video telehealth tablets to high-need patients appears to have successfully reached individuals with social and clinical access barriers. Implementation strategies that address staffing constraints and provider engagement may enhance the impact of such efforts.

BACKGROUND AND SIGNIFICANCE

Access to health care is an overarching priority for the Department of Veterans Affairs (VA), the largest integrated health care system in the United States. Not only do Veterans frequently have complex medical and mental health conditions requiring multispecialty care, but many experience impediments to VA clinical and social service use, including geographic and transportation difficulties, physical and mental health challenges, and socioeconomic stressors.¹

Geographic and travel barriers are potentially the greatest obstacle to VA access. Of the approximate 9 million enrolled Veterans receiving VA health care, nearly a third live in rural, highly rural, and insular island areas.² Over half of Veterans living in rural areas receive their primary care at larger urban VA facilities, making receipt of appropriate and timely care challenging for them.³ Transportation difficulties and costs can be deterrents and have been shown to influence Veterans’ decisions to attend VA facility appointments.⁴

Perceived stigma for seeking health care is also an issue for some Veterans, and has a negative impact on the timeliness of care, as well as adherence to treatment itself. For patients with mental health conditions, stigma is a particularly challenging obstacle to care.⁵ Geographic barriers can exacerbate this problem, and Veterans in rural areas are significantly less likely to receive mental health treatment, such as psychotherapy, than those living in urban settings.⁶

To address these access barriers, VA was an early adopter of technology to extend care into Veterans’ homes and communities. Telehealth is a cornerstone of enhanced access for Veterans and is associated with improved disease control, quality of life, and patient satisfaction across a range of conditions.⁷ In FY 2016, more than 702 000 Veterans received VA care through telehealth, with 39 000 same-day encounters via live video telehealth.⁸ However, until recently, video teleconferencing was limited to Veterans who could travel to community-based outpatient clinics to connect with providers at other facilities. While a partial solution, clinic-based video care is subject to staffing shortages and does not fully address distance-related challenges.

In 2016, VA’s Offices of Rural Health and Connected Care initiated an effort to expand video teleconferencing into the home through tablet-enabled secure video technology. The goal of this initiative was to enhance access for Veterans with geographic, clinical, and social barriers to in-person care. We conducted a mixed methods evaluation of this initiative in order to assess tablet distribution patterns, examine variation in tablet use by sociodemographic and clinical characteristics, and explore factors that facilitated and impeded the implementation process. This study represents one of the first implementation evaluations of a nationwide effort to enhance access for high-need patients through tablet-enabled video technology.

METHODS

Tablet distribution and recipients

In 2016, VA’s Office of Rural Health purchased 5000 tablets from BL Healthcare to distribute to Veterans with barriers to in-person access. The tablets have built-in WiFi or 4G mobile data connectivity and pre-paid access to a national wireless provider’s data network. Among the 5000 tablets, 3500 were limited to video communication, and 1500 permitted the attachment of a general exam camera and optional peripheral devices (ie, stethoscope, BP monitor, pulse oximeter, thermometer, or weight scale). The devices were preconfigured by the vendor to be compatible with VA OI&T requirements including encryption. They were loaded with videoconferencing software and VA mobile apps but did not allow for other connectivity outside of the VA environment (no outside internet access or phone calls outside of VA Video appts).

VA providers referred eligible patients using a consult template in the VA’s electronic health record. Eligibility criteria included: (1) Enrolled in VA Healthcare, (2) Does not own a device or owns a device with suboptimal bandwidth for a quality video session, (3) Physically and cognitively able to operate the technology (or has caregiver who can assist), (4) Barriers to access, such as (a) distance or geography, (b) Transportation issues, (c) Homebound or difficulty leaving home, or (d) Other (described by provider), and (5) Provider and patient give informed consent agreeing to utilize telehealth for care.

Once the patient was issued a device, he or she received telehealth services from their provider during scheduled “clinical video to home” visits. Providers were trained to use Cisco Jabber video technology to connect to the patient tablets. Patients swiped to accept the video call and initiate the video encounter. Tablet appointments could be used for any care that didn’t require physical contact, including mental health therapy and medication management, primary care, palliative care, and specialty care and rehabilitation for conditions such as spinal cord injury.

VA’s Telehealth Program is overseen by the Office of Connected Care (OCC). Regional telehealth leads are responsible for the execution of telehealth services within a defined geographic area and the oversight of all medical centers under their purview. Facility telehealth coordinators (FTCs) direct and plan the execution of telehealth programs for a medical center and its associated community-based outpatient clinics. They are responsible for developing initiatives, procedures, and outcomes measures related to telehealth, and managing the technicians handling the technical aspects of telehealth programs. To prepare for this initiative, OCC developed and distributed specific training materials for clinicians, telehealth service agreements, and patient-facing materials and support. Leadership from OCC had weekly conference calls with FTCs to troubleshoot tablet distribution efforts.

Assessment of tablet initiative adoption and reach

Tablet distribution data for May 1, 2016 to September 30, 2017 were obtained from VA’s Denver Acquisitions and Logistics Center, which provided weekly tablet shipment information with the names and shipment addresses of Veterans who received tablets and/or peripheral devices. These data were merged with tablet usage data from the tablet vendor (Iron Bow Technologies, Herndon, VA), and patient sociodemographic and clinical characteristics (ie, age, sex, race/ethnicity, marital status, rural/urban status, clinical diagnoses) from VA’s national Patient Care Database standard encounter file and patient distance from VA’s Planning System Support Group VAST data. Guided by the RE-AIM Framework,^9,10 we examined the adoption and reach of the tablet initiative. Facility-level adoption rates were assessed by calculating the number of tablets distributed by each facility. Patient-level adoption was assessed by examining the proportion of tablet recipients who used their tablets (defined as an outpatient clinical encounter with documented tablet use, or a tablet call >2.5 minutes), and by examining tablet utilization patterns (i.e., frequency of use and clinical services received via video telehealth). Similar adoption measures have been used in studies of patient-facing technologies such as secure messaging, telehealth, and video telehealth.^11–13 Reach (which refers to “the absolute number, proportion, and representativeness of individuals who participate in a given initiative”⁹) was evaluated by examining the sociodemographic and clinical characteristics of tablet recipients to determine whether tablets were distributed to Veterans with access barriers.

Evaluation of implementation barriers and challenges

Survey of facility telehealth coordinators

In February 2017, we conducted an online survey with VA facility telehealth coordinators (FTCs) and regional telehealth leads to learn more about perceptions of the implementation process and barriers to roll-out. We identified and contacted 108 FTCs using the VA’s telehealth email directory; 68 FTCs completed surveys (63% response rate); respondents represented 45 of the 70 sites participating in tablet distribution (64% site representation), and 17 of 18 VA geographic regions that distribute tablets (94% region representation). Survey respondents were queried about provider and patient perceptions of the tablet initiative, the ease of integrating the initiative with local facility strategic plans, the availability of resources to support implementation, and specific barriers to implementation (derived a priori from literature reviews and discussion with VA program office and clinical provider stakeholders). We asked respondents to indicate if these factors were very big, big, moderate, small, or not barriers to program roll-out, and examined patterns in reported barriers by site’s tertile of tablet distribution (20 or fewer tablets, 21–59 tablets, and 60 or more tablets).

Semi-structured interviews with telehealth staff and leadership

To understand the experiences of FTCs and telehealth leads, we conducted in-depth interviews guided by the Consolidated Framework for Implementation Research (CFIR).¹⁴ Interviews covered the following domains: Innovation Characteristics (eg, tablet capabilities and appropriateness to patient and provider needs), Organizational Characteristics (eg, facility culture and resources, identification of patient needs), External Forces (eg, policies and incentives), Implementation Processes (eg, traits of active vs inactive facilities, proper guidance and support regarding tablet education and troubleshooting), Characteristics of Implementers (eg, traits of site champions and providers who adopt tablets in their practice), and Sustainability Factors (eg, perceived resource needs to continue implementation, lessons learned).

We used a purposive recruitment strategy to recruit interviewees from facilities whose tablet distribution was in the lowest, middle, and highest tertile (ie, 20 or fewer tablets, 21–59 tablets, and 60 or more tablets, respectively) during the first 10 months of the initiative. These categories were balanced across the range of facilities distributing tablets. For each group, we identified three FTCs who had completed the survey; selection was based on thoroughness of survey responses and geographic diversity. For a holistic understanding of a site’s experience, we also interviewed the regional telehealth lead for each participating FTC’s region. After obtaining consent, we conducted and digitally recorded 20 telephone interviews (9 FTCs, 2 telehealth leads who had previously served as FTCs (analyzed as FTCs), and 9 additional telehealth leads). Interviews ranged in length from 30 to 60 minutes, and participants were not compensated for participation.

We engaged in directed content analysis of transcripts adapted from Hsieh et al. (2005).¹⁵ We first used a rapid qualitative analysis approach to identify implementation facilitators and barriers associated with CFIR domains and grouped these factors into broad themes with representative quotes. Then, two authors (EW, CS) used inductive coding to independently map quotes from each theme to specific implementation strategies as defined by the Expert Recommendations for Implementing Change (ERIC) project.¹⁶ Discrepancies were reviewed by a third researcher and resolved through group discussion.¹⁷ Next, two authors (LH and EW) reviewed all interviews to determine recurring themes and identified recommendations based on the implementation strategies described as most effective. Two authors (LH and EW) independently reviewed transcripts to identify illustrative quotes for each theme.

This evaluation was reviewed and designated as nonresearch by the supporting VHA program office, the local Institutional Review Board, and VA Research Administration.

RESULTS

Adoption

Between May 1, 2016 and September 30, 2017, 86 VA facilities spanning all 18 geographic regions, distributed tablets to 6745 Veterans. Figure 1 illustrates the distribution of tablets across United States cities. The majority of tablet recipients (6175; 91%) received a tablet that was limited to video capability; 695 (10%) received a tablet capable of clinical monitoring; some patients received more than one tablet in the period of investigation. Among the 433 patients who received a peripheral device, 88% received a stethoscope, 77% received a blood pressure monitor, 61% received a thermometer, 56% received a weight scale, and 53% received a pulse oximeter.

Figure 1.

Distribution of VA-issues tablets, by destination city.

In terms of patient adoption, based on clinical encounter and tablet vendor data, 81% of tablet recipients used their tablets during the evaluation period. Among the 5320 patients with Ironbow call data, 82% had evidence of a call ≥ 2.5 minutes (1029 (19%) used the tablet once, 1773 (33%) used it 2-5 times, and 1574 (30%) used the tablet more than five times). Based on clinical encounter data, tablets were used most frequently for mental health care (54%), spinal cord injury care (15%), home-based primary care (7%), therapy/rehabilitation (7%), primary care (4%), and palliative care/hospice (2%).

Reach

The mean (SD) age of tablet recipients was 56 (17); 36% were over the age of 65. The majority (84%) of tablet recipients were male, but women were overrepresented at 16% (compared to 6.5% of VA patients).¹⁸ Approximately half of the tablet recipients lived in a rural or highly-rural geographic region (48% and 5%, respectively). Tablet recipients had a mean (SD) of 5 (3) chronic conditions, and 76% had a diagnosed mental health condition. Table 1 presents additional sociodemographic and clinical characteristics of tablet recipients as well as the subset of tablet users. In multivariate logistic regression, tablet recipients were more likely to use their tablets if they were 45-64 years (AOR 1.37, P < .001) or ≥ 65 years (AOR 1.33, P = .002) (compared to < 45 years), and if they were married (AOR 1.24, P = .002); they were less likely to use their tablets if they had ≥ 7 chronic conditions (AOR 0.73, P = 0.001). There were no statistically significant usage patterns related to gender, race, rural status, mental health diagnosis, or driving distance to primary VA facility (Supplementary Appendix Table 1).

Table 1.

Characteristics of tablet recipients and tablet users

	All tablet recipients (N = 6,745)	Tablet usersc(n = 5,503)
	%	%
Age, mean (SD)	56 (17)	56 (16)
18–44	30	29
45–64c	35	36
65b,c	35	36
Male	84	84
Marriedc	54	55
Race/Ethnicity
White, Non-Hispanic	72	73
Black, Non-Hispanic	11	11
Hispanic	5	5
Other	5	5
Unknown	4	4
Geographic location
Highly rural	5	5
Rural	48	48
Urban	47	47
Distance to nearest VA (mi), mean (SD)
VA primary care	23 (23)	23 (23)
VA secondary carec	63 (53)	64 (54)
VA tertiary carec	116 (103)	117 (104)
Chronic conditions,a,c mean (SD)	5 (3)	5 (3)
Depression	57	57
Hypertension	50	50
Post-traumatic stress disorder	48	48
Low back pain	37	38c
Diabetes	26	27
Arthritis	18	18
Chronic Obs pulmonary disease	16	16
Ischemic heart disease	14	14
Alcohol abuse	12	11c
Any drug dependence/abuse	11	10c

Unless otherwise specified.

^aCount of chronic conditions from a total of 34 included in study; Supplementary Appendix Table 2 lists rates for all conditions assessed.

^bTablet use based on calls >2.5 minutes or stop code 179 during 12 months after tablet shipment.

^c P < .05 when comparing tablet users and non-users (Supplementary Appendix Table 1).

Implementation barriers and promising strategies

The majority of facility telehealth coordinators that completed the survey agreed that the tablet initiative aligned with their facility’s strategic plan (88%), and that patients and providers had responded well to the initiative (86% and 82%, respectively). Fewer (65%) reported that there were adequate resources to easily integrate the tablet initiative. The most common issues rated as a ‘moderate’ to ‘very big’ implementation barrier were staffing shortages (59%), training needs (55%), and lack of provider interest (33%). Fewer than a quarter of survey respondents cited lack of support from regional or facility leadership as barriers. Figure 2 presents the proportions of FTCs who reported each barrier, by degree of impact on program implementation.

Figure 2.

Barriers to implementation of VA’s nationwide tablet initiative (N = 68 facility teleheath coordinators).

Of the 68 survey responses, 54 came from sites that distributed tablets during the study window. Table 2 shows the percent of respondents who indicated that the barriers in question were moderate to very big barriers. While the sample size was small, there were some noticeable trends. Sites with lower distribution were more likely to report lack of support from regional leadership and low provider interest in the initiative. Sites that had larger volumes of distribution were more likely to report space constraints, lack of adequate staffing to support tablet distribution, and need for more facility-based equipment. Respondents from sites with all levels of distribution reported challenges related to adequate preparation and training, and competition for time and resources with other telehealth initiatives.

Table 2.

Barriers reported by FTCs from sites with low, medium, and high tablet distribution volume

	Tablet distribution volume
Barrier to tablet distribution	Low %	Medium %	High %
Competing telehealth initiatives	17	18	13
Lack of support from regional leadership	23	19	9
Lack of support from facility leadership	21	35	0
Lack of provider interest in the tablets	47	25	11
Staffing shortages	31	71	67
Need for facility-based equipment	19	20	26
Space constraints in facility	18	35	33
Training needs for the tablet initiative	53	53	44

Interviews with FTCs and telehealth leads echoed the survey findings and suggest that most implementation barriers were related to staffing and training needs, and clinician engagement. Interviewees identified a third barrier, the challenges with connectivity and tablet use, which wasn’t included in the survey questions. Several interviewees described strategies that facilities employed to address these barriers. Supplementary Appendix Table 3 presents the strategies that address each barrier, supporting quotes, and the percentage of interviewees who endorsed a barrier that this strategy would address.

FTCs across sites and geographic regions described challenges with connectivity and tablet usability as a major barrier to tablet distribution and use. Connectivity problems were noted for patients in certain rural areas with poor broadband penetration. For example, a facility in rural Texas found that patients in tin-roofed houses were unable to connect the tablets to Wi-fi. This issue limited tablet reach and required additional patient assessment beyond the standard template at tablet referral. Though many FTCs who cited these challenges desired centralized resources to troubleshoot technology-related barriers, sites that sucessfully overcame these technical difficulties often had to acquire local expertise due to unique location-based technical issues.

When respondents elaborated on staffing and training barriers, respondents cited under-staffed teams of telehealth providers, competing facility priorities, and inadequate education prior to program roll out. Many interviewees learned how to use the tablets through trial and error, and felt there were few resources for troubleshooting. Some teams requested a demo tablet for temporary use, but sites that did not generally had no firsthand knowledge of how the Veterans’ tablets worked. Interviewees also spoke of the desire to learn from experienced, successful sites, and also to share training materials and lessons learned. Sites often created their own eductional materials and consult templates to supplement what was available at the national level. Sites particularly successful in tablet distribution were proactive in reaching out to their networks for help, or contacted the tablet vendor directly to request assistance.

Provider engagement was influenced by familiarity with the technology, ease of scheduling and coordinating tablet use across services, leadership support, and advertising of the initiative to potentially interested providers and patients. Tablet scheduling and the telehealth management platform required specialized knowledge, which presented problems for some sites and limited provider participation. While most interviewees described their leadership as supportive, some commented that lack of enthusiasm for tablets among regional leadership affected implementation resources and provider engagement at the local level. One way in which leaders showed support was by encouraging providers to help implement the tablet program. One interviewee described how leadership involvement fueled provider buy-in, which resulted in a dramatic increase in tablet distribution rates. Other sites were able to boost provider engagement by negoatiating telework options for participating providers, and through local champions who advocated for the tablets and shared successes at meetings.

DISCUSSION

VA’s implementation of video telehealth tablets appears to have successfully reached many patients in the target population, including those in rural and geographically isolated locations and individuals with complex medical and mental health needs. Nevertheless, one in five tablet recipients did not use their tablets, and these patients were more likely to be younger, have more chronic conditions, and lack social support. We also identified a number of barriers that influenced the implementation of VA’s national telehealth tablet initiative.

The primary implementation barriers that surfaced in surveys and interviews related to technology, staffing, and provider interest in light of competing demands. While these barriers were described by multiple interviewees, some facility telehealth coordinators described success in overcoming certain barriers through specific implementation strategies, such as developing a protocol to check for known issues (such as cellular coverage), fostering a network with other sites to share materials and lessons learned, and rolling out tablets in a step-wise fashion to first target clinical services that are most interested. In addition, as illustrated in Table 2, support from facility and regional leadership appeared to be an important factor associated with high distribution rates; interviews suggest that this support was central to adequate resource allocation and to provider engagement.

When we map the primary barriers raised in surveys and interviews to established implementation strategies, a number of recommendations emerge to guide future implementation efforts (Table 3; see Supplementary Appendix Table 3 for additional implementation strategies and supporting quotes).¹⁶ The strategies range from centralized technical support and tablet tracking systems accessible by local sites, to identifying local champions that can share knowledge across sites and with providers.

Table 3.

Recommendations for health systems planning disseminated implementation of patient-facing technology

Implementation barriers	Implementation strategies that address barriersa	Recommendation	Potential impact
Technology challenges (eg, challenges regarding use of the tablet, including functionality and connectivity)	Centralize technical assistance	Develop a system that provides technical assistance focused on implementation issues for the particular program being implemented	Providing a central, accessible point of contact for assistance will create confidence in the new initiative, and facilitate uptake and expansion
Facility telehealth coordinator staffing and training (eg, staffing shortages, competing telehealth priorities, lack of telehealth staff buy-in or education, lack of coordination between sites)	Assess sites for readiness	Conduct a local needs assessment and beta testing to ensure adequate preparation for large-scale program rollout	Prioritizing an initial, timely local gap analysis will identify key stakeholders, available resources, and known barriers/limitations. This may be particularly important for a program involving new technology to avoid early challenges among clinicians or patients that may preclude a positive experience and subsequent expansion
	Capture and share local knowledge	Capture local knowledge from implementation sites on how implementers and clinicians made something work in their setting and then share it with other sites	Facilitating and promoting networking among sites will build on supportive working relationships and strengthen goals related to implementation of the innovation
	Mandate and incentivize change	Develop performance measures to incentivize clinicians to dedicate time towards learning and meaningfully adopting new technology and approaching virtual patient assessment and communication	Financial incentives or institution-wide commitment to telehealth may accelerate uptake of telehealth/new technology across sites and specialties, particularly for late or struggling adopters
Clinician engagement (eg, lack of provider interest, knowledge, buy-in)	Identify and prepare champions	Employ regional “train-the-trainer” programs of early adopters to promote telehealth and optimize use of technology via hands-on demonstration; build coalition of experts	Local champions serve as experts who offer focused, regionally relevant, program support. They are knowledgable about applications of the technology, may be better aware of local limitations and may more effectively influence cultural change and attitudes about a new modality of care
	Change licensure standards	Mitigate clinicians’ apprehension about state licensure issues with respect to care into the home or across state lines by legislation that clearly communicates national practice standards for telehealth	Maximize clinicians’ willingness to adopt telehealth by decreasing apprehension regarding medical-legal risks, resulting in greater opportunities for patients to engage in a video visit

^aStrategies drawn from Expert Recommendations for Implementing Change (ERIC).¹⁶

This evaluation is timely in light of the enactment of the VA MISSION Act of 2018 which extends legislative authority to clinicians working at any VA facility to offer care through telehealth, irrespective of clinician or patient location.¹⁹ The VA MISSION Act will potentially address capacity and access issues by allowing Veterans to receive care from clinicians (including scarce specialists in distant urban medical centers). The VA has committed to offering any patient interested in video telehealth the opportunity to receive this care though their personal smartphone, tablet or computer, or a VA-loaned tablet in primary care and mental health by 2020.²⁰

Expanding access to high-quality, affordable broadband in rural areas is a needed step forward in the effort to mitigate the digital divide. The Federal Communications Commission recently voted to establish a new $100 million “Connected Care Pilot Program” to support telehealth for low-income and rural Veterans.²¹ The VA is also establishing telehealth access points in rural areas with VA equipment, including community centers, hospitals, Veteran Service Organization posts, public libraries and clinics for Veterans without home connectivity or Veterans living in areas without broadband availability. This action will be critical to address access for Veterans with connectivity barriers observed in this early evaluation.

This evaluation was limited to the perspectives of facility telehealth coordinators and telehealth leads at a select number of VA facilities. In addition, limited data were avilable regarding the specific ways in which tablets were used. Additional work is underway to examine the patient and provider’s perspective, and to examine the impact of tablet use on health care utilization patterns, clinical outcomes, and patient experience.

In summary, VA’s telehealth tablet initiative illustrates the potential for integrated health systems and other highly networked organizations to address access barriers through widespread dissemination of patient-facing technology. While a number of challenges restricted optimal tablet distribution and use, many sites developed local strategies and identified potential centralized approaches that facilitated implementation. Given rapid advances in video telehealth technology and the increasing use of this technology outside of the healthcare sector, rigorous evaluations of the impact of video telehalth on access, utilization, and clinical outcomes are critical.

FUNDING STATEMENT

This work was supported by VA’s Office of Rural Health Enterprise Wide Initiative and the eHealth Partnered Evaluation Initiative (QUERI, PI Timothy Hogan, Bedford Mass). Dr. Zulman is supported by VA HSR&D Career Development Award (CDA 12-173).

COMPETING INTERESTS

John Peters and Leonie Heyworth, MD, are employees of the Veterans Affairs Office of Connected Care, which was an operations partner in this evaluation.

CONTRIBUTORSHIP STATEMENT

Dr. Zulman, Dr. Heyworth, and Mr. Peters conceived and designed the evaluation. Ms. Wong and Dr. Heyworth conducted all data collection. Ms. Wong and Ms. Slightam conducted data analyses and interpretation with Drs. Zulman and Heyworth. Drs. Zulman and Heyworth took the lead in writing the manuscript, with input from all authors. All authors provided critical feedback and approve of the version to be published.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.