Virtual Urgent Care Quality and Safety in the Time of Coronavirus

Abstract

Background

Telemedicine use rapidly increased during the COVID-19 pandemic. This study assessed quality aspects of rapid expansion of a virtual urgent care (VUC) telehealth system and the effects of a secondary telephonic screening initiative during the pandemic.

Methods

A retrospective cohort analysis was performed in a single health care network of VUC patients from March 1, 2020, through April 20, 2020. Researchers abstracted demographic data, comorbidities, VUC return visits, emergency department (ED) referrals and ED visits, dispositions, intubations, and deaths. The team also reviewed incomplete visits. For comparison, the study evaluated outcomes of non-admission dispositions from the ED: return visits with and without admission and deaths. We separately analyzed the effects of enhanced callback system targeting higher-risk patients with COVID-like illness during the last two weeks of the study period.

Results

A total of 18,278 unique adult patients completed 22,413 VUC visits. Separately, 718 patient-scheduled visits were incomplete; the majority were no-shows. The study found that 50.9% of all patients and 74.1% of patients aged 60 years or older had comorbidities. Of VUC visits, 6.8% had a subsequent VUC encounter within 72 hours; 1.8% had a subsequent ED visit. Of patients with enhanced follow-up, 4.3% were referred for ED evaluation. Mortality was 0.20% overall; 0.21% initially and 0.16% with enhanced follow-up (p = 0.59). Males and black patients were significantly overrepresented in decedents.

Conclusion

Appropriately deployed VUC services can provide a pragmatic strategy to care for large numbers of patients. Ongoing surveillance of operational, technical, and clinical factors is critical for patient quality and safety with this modality.

As health care delivery systems confronted coronavirus disease 2019 (COVID-19), telemedicine was encouraged as a “virtually perfect”¹ solution. The US Surgeon General explicitly advocated for telehealth “revolution.”² Telemedicine would permit structured assessments while maintaining social distance, mitigating supply-demand mismatch, and preserving the health care workforce.3., 4., 5. We previously suggested the need for flexible, adaptive, scalable care systems to assess, stabilize, and treat patients in future disasters.⁶ As New York City's COVID-19 epicenter precipitously expanded in March 2020 and presentations to our institution rose abruptly, a nascent virtual urgent care (VUC) service rapidly expanded to address patient care.

Telemedicine has existed for decades, with a successful history of being leveraged to treat both acute and chronic infectious disease, including respiratory illness.^{7 , 8} In many cases, telehealth has provided equivalent outcomes to in-person care.⁹ Telemedicine has also demonstrated an important role within urban health care, where a variety of barriers may impair access.¹⁰ However, several limitations have been recognized, particularly constraints on the physical examination.¹¹ Technological, administrative, and legislative infrastructure present additional barriers.^{1 , 12 , 13} With widespread staff redistribution, telehealth training modalities¹ in the crisis application of technology might insufficiently provide clinicians with technologic expertise and diagnostic and disposition comfort. Absent robust antecedent telemedicine experience, providers might struggle to translate in-person skills to a virtual medium, and precise referral or escalation guidelines⁴ might be nonexistent in the face of rapidly changing recommendations from public health entities. Minimal or nonexistent outpatient laboratory and radiographic diagnostics (due to patient preference, provider attempts to preserve social distancing, and site closures), which would be typically available during normal operations, could further impede management decisions. A lack of dedicated follow-up⁵ and consultation services might similarly complicate management. Last, telemedicine might not maintain its generally low-risk demographics, might fail to address distinct patient population needs, and might overlook social determinants of health and health disparities.^{13 , 14}

Technology may follow a “hype cycle” or curve. Following an “innovation trigger,” a period of “inflated expectations” gives rise to a subsequent “trough of disillusionment” and a “slope of enlightenment,” until achieving a “plateau of productivity.”¹⁵ However, inflated expectations of telemedicine could compromise patient quality and safety if applied to undeveloped or immature systems. In this investigation, we review quality and safety aspects of rapid deployment of a VUC telehealth system during a pandemic. These data are critical to appraise telehealth quality and safety issues in a disaster, and to design patient care processes for potential subsequent COVID-19 outbreaks. Furthermore, they may inform rapid telemedicine implementation for health care delivery in general.

Methods

Study Design

This was a single-network, retrospective cohort analysis of all patients who presented to VUC services from March 1, 2020, to April 20, 2020. Operational deployment, technology, geographic distribution, and patient satisfaction in the early pandemic phase are described in a separate manuscript.¹⁶

The study was approved by the Institutional Review Board of the New York University Grossman School of Medicine, Office of Science and Research. A waiver of consent and authorization was granted for this study. No funding was received.

Study Setting and Population

NYU Langone Health is an urban, multisite tertiary academic network of four acute care hospitals and more than 350 outpatient locations. More than 300,000 patients are treated annually across its five emergency department (ED) and immediate care spaces. These sites care for a cross-section of patients both new to the system and associated through prior ED visits or other care elements (for example, outpatient providers). VUC services, administered by the academic Department of Emergency Medicine, were extant since 2017. VUC was staffed entirely by emergency physicians prior to the study period. During the pandemic, allied health providers were also used. Visits were conducted via an online video telehealth platform, which was integrated into the electronic health record (EHR) (Epic Systems Corporation, Verona, Wisconsin). A custom augmented EHR application on mobile devices and computers allowed patients to schedule visits up to 48 hours in advance (and within 30 minutes of any open time slot). Patients’ location was limited only by access to a mobile device or a computer with Internet access. Patients completed a questionnaire of their medical history, medications, allergies, pharmacy, and reason for visit. Providers, working stand-alone shifts or portions of shifts, accessed the EHR through computers for health record review and documentation, and conducted the video visit via EHR mobile applications. Twenty-four-hour coverage was provided throughout the study period. In March, patients were screened prior to network ambulatory visits for symptoms consistent with viral upper respiratory tract infection, prior to the closure of all network ambulatory sites on March 23. If they screened positive for these symptoms, they were directed to use VUC instead of an in-person visit. In addition, direct e-mails regarding VUC availability to health system patients were used. A formal VUC quality and safety framework was not in existence prior to the study period.

Of note, a severe deficit and delay in testing capacity and explicit public health authority recommendations discouraged COVID-19 testing for both outpatients and ED discharges (for example, “DOHMH [New York City Department of Health and Mental Hygiene] strongly recommends against testing persons with mild illness who can be safely managed at home” [emphasis in original]).^{17 (p. 2)} This meant that for the vast majority of VUC patients, COVID-19 testing was never performed, even if warranted. For local context during the study period, given these significant constraints (that is, testing reserved for hospitalized patients), the rates of positive COVID tests in Bronx, Kings, Nassau, New York, Queens, Richmond, Suffolk, and Westchester counties were 48%, 48%, 43%, 39%, 50%, 40%, 41%, and 33%, respectively.¹⁸ Hospitals faced widespread shortages of personal protective equipment (PPE) throughout the study period.¹⁹ Mandatory public face coverings were not required until April 17, 2020.²⁰

All patients presenting to VUC from March 1, 2020, to April 20, 2020, were included. As outcomes can trail index visits, patients were followed at least 21 days after the index VUC presentation. For study purposes, we evaluated only adult patients aged 18 or greater at the time of the visit. To provide context for patients managed as outpatients, we evaluated ED patients presenting during the same time period who were not admitted. We undertook several post hoc analyses for comparison purposes, evaluating cohorts of patients using VUC in the prior year.

Data Collection

Emergency services, inpatient facilities, and outpatient practices throughout the network use a uniform, integrated EHR. Regional data in the Epic platform is available, with extant patient consent, in sections of the EHR (Care Everywhere) and the public health information exchange (Healthix, James, New York).

Data were extracted from the Epic Clarity database via Oracle SQL Developer 17.4.0.355 (Oracle Corporation, Redwood City, California). Tableau 2020.1.0 (Tableau Software, LLC, Seattle) was then used on the extracted data to input all of the necessary data fields, join the multiple queries, and remove duplicate rows.

To assess patient and provider data, systems effectiveness, and ability to provide care with and without admission to the ED,^{21 , 22} we identified the following variables from the cohort records: provider type, patient age, self-assigned gender, visit date, self-assigned race/ethnicity,²³ comorbidities, previous primary care visits (as an indirect marker of regular access to care), VUC return visits, ED referrals and spontaneous ED visits, ED dispositions, intubations, and deaths following VUC visits. All decedents’ charts for clinical presentation and hospital course, including medications, interventions, consults, and procedures, were manually reviewed by emergency medicine (EM) attendings with additional subspecialty training in critical care, informatics, pediatric EM, and toxicology.

Partway through the study period, based on departmental and institutional assessments of COVID-19 risk factors,²⁴ VUC services introduced a workflow on April 7 targeting patients with telehealth visits for COVID-19–like complaints. Because this operational change might affect results, we further evaluated data within these two periods: March 1 to April 6, 2020 (period 1), and April 7–20, 2020 (period 2). VUC visits in patients aged 60 years and older who had fever, cough, or gastrointestinal complaints or a VUC discharge diagnosis of advice for or suspected COVID-19 were identified. Patients meeting criteria were called within 36 to 48 hours by a licensed independent provider who performed a standardized screening. Patients were asked about illness duration ≥ 6 days, worsening symptoms from index VUC visit, presence of symptoms on ambulation or new neurologic symptoms including dizziness or falls, and presence of body mass index (BMI) ≥ 25. Providers were instructed to consider immediate referral for in-person ED examination in patients having affirmative answers to any question. We evaluated and manually reviewed all patients identified for follow-up to assess outcomes, including ED referral during follow-up screening call, hospital admission and course, presence or absence of ED referral on index VUC visit, mortality, and interval metrics.

Proposed telehealth performance measures include equivalent outcomes compared to face-to-face care.²¹ Although no direct comparators existed during the pandemic, when decision thresholds to present to the ED were multifactorial, to provide context for departments managing patients as outpatients, we evaluated similar outcomes of patients presenting for ED care who were discharged (not admitted) during the same time period. These included patients who were discharged, were discharged to a nursing facility, departed against medical advice, eloped, left without being seen before triage, and left without being seen after triage. We evaluated return visits, return visits with admission (admitted, expired, sent to operating room, sent to labor and delivery, transferred to a procedure area, and transferred to another facility), and deaths.

Poison control centers have provided telephonic triage, medical decision making, and follow-up of millions of emergent and nonurgent cases for decades, including disasters and events of public health significance.²⁵ Therefore, previous telephonic quality and safety measures such as risks posed by dropped calls were applied to VUC.²⁶ Separately, telehealth frameworks also suggest that these “dropped sessions”²¹ represent a continuous performance improvement benchmark, and that technical issues should be assessed.²² Visit failures (recorded as terminations in the EHR) are mutually exclusive categories—“no-show,” “criteria unqualified” (for example, an out-of-state patient prior to revised Centers for Medicare & Medicaid Services guidelines), and “connection issue” (technological failure). We further evaluated these categories as to whether patients completed check-in to VUC (that is, were apparently actively intending to be seen) after scheduling an appointment. Terminations after check-in could approximate potentially concerning dropped visits. We also assessed whether patients were subsequently seen in VUC after visit termination (recovered) as an additional assessment of the access to care domain.²²

Data Analysis

Descriptive analysis was performed for age, self-assigned gender, self-assigned race/ethnicity, VUC returns, ED presentations, subsequent intubations, and death. Comorbidities were abstracted and stratified by age. Data visualization, manipulation, and analysis were performed in Tableau.

We stratified revisits to VUC by 24-hour periods to coincide with return visit measures (“bounce-backs”) previously published in EM literature.27., 28., 29. These were evaluated in two ways. First, we determined outcomes on a per-visit basis. Any repeat visit by a patient must be staffed; therefore, this metric better reflects operational load requirements. We also evaluated revisits on a per-patient basis, as multiple visits by a small group of patients might skew results. We did not differentiate whether revisits were planned versus unplanned. We similarly stratified presentations to the ED, as informed by quality frameworks,²¹ by 24-hour periods on a per-visit and per-patient basis. We did not differentiate whether these represented progression of disease (after a reasonable trial of outpatient management) versus an errant decision.

We conducted several post hoc analyses. We compared age and comorbidities with VUC patients presenting from an identical time frame one year earlier (March 1, 2019, to April 20, 2019). We also compared age and comorbidities in VUC patients to those presenting to the ED who were discharged. In addition, we evaluated a common telemedicine quality measure of antibiotic prescription rates for diagnoses of sinusitis and upper respiratory infection.³⁰

Mortality rates for the periods of March 1, 2020, to April 6, 2020, and April 7–20, 2020, were compared using chi-square test with Yates correction for any 2 × 2 contingency tables.³¹ A p value of less than 0.05 was considered significant. Decedent self-assigned gender and race/ethnicity were similarly evaluated. The Wilcoxon rank-sum test was used to compare median ages and number of comorbidities between the cohorts.

Results

In 18,278 unique patients, there were 22,413 completed VUC visits from March 1, 2020, through April 20, 2020 (Figure 1 a). This represented a 33-fold increase in visits from the identical time period one year prior. There were 17,422 visits in from March 1 to April 6, 2020 (period 1), and 4,991 from April 7 to April 20, 2020 (period 2). The VUC median age was 40 years (interquartile range [IQR] = 32–53 years), which was slightly older than the median age of 39 years (IQR = 31–50 years, p = 0.006) in the VUC cohort from one year prior. The VUC median age was slightly younger than the age of ED discharged patients (43 years; IQR = 31–58 years, p < 0.001). There were 4,086 visits in patients aged 18 to 29 years, 6,701 visits in patients aged 30 to 39 years, 4,872 visits in patients aged 40 to 49 years, 3,700 visits in patients aged 50 to 59 years, 2,114 visits in patients aged 60 to 69 years, 708 visits in patients aged 70 to 79 years, 191 visits in patients aged 80 to 89 years, and 41 visits in patients aged 90 years and older. The overall ED referral rate was 6.1% on a per-visit basis.

Figure 1.

(a) Completed VUC visit volumes (blue), ED referrals for each day (yellow), and 72-hour VUC re-presentations (green) and ED presentations after a VUC visit with (red) and without admission (light blue) that were attributable to the index VUC visit date are presented. (b) VUC incomplete visits (“terminations”) are shown stratified by whether or not the patient checked in; “no-shows” (green), patient criteria unqualified (blue), technology/connection issue (red). VUC, virtual urgent care; ED, emergency department.

Of VUC patients, 76.4% had a primary care provider (PCP) listed in the EHR; 31.1% of these patients visited their listed PCP within 30 days prior to their VUC visit. Only 36 patients were new to the health care network (dating back seven years to the original EHR implementation). For 25 days during the study period, VUC visits exceeded the network's combined ED visits (see Supplementary Figure 1 in the appendix, available in online article). Mean daily VUC visit volume was 470 patients per 24-hour period (min 41, max 863) in period 1 and 357 patients per 24-hour period (min 232, max 563) in period 2. During the study period, visits were staffed by a physician (70.0%), physician assistant (29.3%), nurse practitioner (0.4%), or undetermined (0.3%). Overall, 86.1% of providers were VUC novices (that is, had never previously staffed a telemedicine encounter)—83.0% of physicians and all physician assistants and nurse practitioners.

Incomplete Visits

Apart from 22,413 completed VUC visits, an additional 718 visits were incomplete (3.1% of total VUC patient-scheduled visits). Of these, 450 were due to no-shows; 218 (48.4%) of the no-shows had initially completed initial registration and check-in. Twenty-four percent of incomplete visits ultimately completed a VUC visit within 1 day (including 37.2% of no-shows who had checked in). An April 7 spike corresponded with an approximately 30-minute period of platform instability with inability to video chat with patients, conversion to telephonic communication, and concurrent technical malfunction in which patients were not appearing as "arrived.” This resulted in a higher number of incomplete visits as “no-shows.” There were 137 total visits with technical issues, of which 121 (88.3%) had initially checked in, and 141 visits in which the patient was not qualified (134 had checked in). The temporal pattern of incomplete visits, which did not appear to mirror the overall VUC encounters pattern, is presented in Figure 1b.

Comorbidities

Comorbidities present in patients are presented in Table 1 . We found that 50.9% of patients had at least one preestablished comorbidity. This prevalence was greater than 74% for patients aged 60 years and older. We found a substantial portion of patients at risk due to elevated BMI. Comorbidities on a per-visit basis are provided in Supplementary Table 1. VUC patients had a median of 1 comorbidity (IQR = 0–2 comorbidities), which was the same as the VUC cohort one year prior (1; IQR = 0–2 comorbidities, p = 0.09). VUC patients had a higher median number of comorbidities than the ED discharged population (0; IQR = 0–2 comorbidities, p < 0.001). Similar results were obtained when assessed on a per-visit basis (data not shown).

Table 1.

Virtual Urgent Care Patient Comorbidities Present on a Per-Patient Basis

	Total	18–29 years	30–39 years	40–49 years	50–59 years	60–69 years	70–79 years	80–89 years	90 and older
	18,278*	3,361	5,362	3,878	3,067	1,777	625	170	38
Comorbidity, n (%)
Asthma	2,203 (12.1)	395 (11.8)	593 (11.1)	515 (13.3)	390 (12.7)	216 (12.2)	77 (12.3)	16 (9.4)	1 (2.6)
BMI > 25	6,826 (37.3)	711 (21.2)	1,682 (31.4)	1,643 (42.4)	1,509 (49.2)	899 (50.6)	293 (46.9)	79 (46.5)	10 (26.3)
CAD	451 (2.5)	0 (0.0)	11 (0.2)	39 (1.0)	114 (3.7)	154 (8.7)	90 (14.4)	36 (21.2)	7 (18.4)
CHF	88 (0.5)	1 (0.03)	7 (0.1)	9 (0.2)	16 (0.5)	20 (1.1)	16 (2.6)	14 (8.2)	5 (13.2)
Cirrhosis	25 (0.1)	0 (0.0)	1 (0.02)	2 (0.1)	11 (0.4)	8 (0.5)	3 (0.5)	0 (0.0)	0 (0.0)
CKD	230 (1.3)	10 (0.3)	27 (0.5)	24 (0.6)	55 (1.8)	47 (2.6)	49 (7.8)	16 (9.4)	2 (5.3)
COPD	225 (1.2)	1 (0.03)	10 (0.2)	24 (0.6)	54 (1.8)	75 (4.2)	45 (7.2)	14 (8.2)	2 (5.3)
Dialysis	29 (0.2)	0 (0.0)	3 (0.1)	6 (0.2)	9 (0.3)	5 (0.3)	6 (1.0)	0 (0.0)	0 (0.0)
DM	1,259 (6.9)	62 (1.8)	167 (3.1)	270 (7.0)	357 (11.6)	231 (13.0)	138 (22.1)	29 (17.1)	5 (13.2)
HLD	3,271 (17.9)	88 (2.6)	369 (6.9)	631 (16.3)	956 (31.2)	778 (43.8)	334 (53.4)	94 (55.3)	21 (55.3)
HTN	3,056 (16.7)	70 (2.1)	341 (6.4)	621 (16.0)	890 (29.0)	686 (38.6)	327 (52.3)	96 (56.5)	25 (65.8)
Immunosuppression	390 (2.1)	26 (0.8)	85 (1.6)	95 (2.4)	95 (3.1)	48 (2.7)	35 (5.6)	4 (2.4)	2 (5.3)
Malignancy	982 (5.4)	27 (0.8)	86 (1.6)	132 (3.4)	245 (8.0)	268 (15.1)	158 (25.3)	57 (33.5)	9 (23.7)
Any comorbidity	9,312 (50.9)	1,008 (30.0)	2,191 (40.9)	2,140 (55.2)	2,038 (66.4)	1,286 (72.4)	484 (77.4)	133 (78.2)	32 (84.2)

BMI, body mass index; CAD, coronary artery disease; CHF, congestive heart failure; CKD, chronic kidney disease; COPD, chronic obstructive pulmonary disease; Dialysis, dialysis dependent; DM, diabetes mellitus; HLD, hyperlipidemia; HTN, hypertension.

^⁎Number of unique adult patients.

Revisits

A total of 1,521 unique visits (6.8%) had at least one subsequent VUC encounter within 72 hours following their index visit (see Figure 1a for trend). ED revisit benchmarks typically use 72-hour intervals, but we were interested in trending VUC revisits for up to 1 week. The cumulative revisits up to 1 week, stratified by age, are presented in Figure 2 . We were unable to distinguish planned repeat VUC encounters from those spontaneously chosen by patients. The 18-to-29-year, 30-to-39-year, and 40-to-49-year age cohorts had the highest rates of VUC reutilization within 1 week as a percentage of their age cohort (Supplementary Figure 2). On a per-patient basis, 1,116 unique patients (6.1%) had at least one subsequent VUC encounter within 72 hours following their index visit. The cumulative revisits on a per-patient basis through 1 week, stratified by age, are presented in Supplementary Figure 3.

Figure 2.

Cumulative revisits to virtual urgent care are shown on a per-visit basis.

There were 405 unique visits (1.8%) with presentations to the ED within 72 hours following their index VUC visit (see Figure 1a for trend). Of these, 118 unique visits (0.5% overall) required admission (see Figure 1a for trend). The cumulative revisits for repeat ED presentations for up to 1 week on a per-visit basis, stratified by age, are presented in Figure 3 a. ED visits requiring admission are presented in Figure 3b. Similar ED presentations and admissions on per-patient basis are presented in Supplementary Figures 4a and 4b. Opposite the trends of VUC re-presentations, the 60-to-69-year, 70-to-79-year, and 80-to-89-year age cohorts had the highest rates of visits to the ED after VUC by 1 week as a percentage of their age cohort (Supplementary Figure 5).

Figure 3.

(a) Cumulative visits to the emergency department following a virtual urgent care (VUC) visit are shown on a per-visit basis. (b) Emergency department visits following a VUC visit that required admission are shown on a per-visit basis.

To provide context, we evaluated ED revisit rates for patients who had originally presented to the ED but were then discharged (21,618 visits in 19,321 total patients; 19,245 visits in 16,888 adult patients) over the same period. The ED cumulative revisit rate for discharged ED patients on a per-visit basis was 4.4% at 72 hours (7.0% in adult patients), with an ED revisit with admission rate of 1.5% (1.7% of adults). Values for ED re-presentation on a per-patient basis were 4.6% and 1.6% (5.9% and 1.9% in adult patients). ED revisit and admission percentages through 1 week are provided on a per-visit basis in Figure 4 and on a per-patient basis in Supplementary Figure 6. Following their VUC visit, 37 individuals (0.17%) required intubation during a subsequent ED presentation and hospitalization (12 of which were in the ED). There were no significant differences in intubation rates between the periods (p = 0.49).

Figure 4.

Return visits to the emergency department (total and visits with admission) following an initial ED visit are shown on a per-visit basis.

VUC Referrals and Decedents

Overall mortality was 0.20% using visits as a denominator, and 0.25% per unique patient. All deaths after VUC visits appeared to be related to COVID-19 illness. All VUC decedents in the entire cohort had comorbidities, with the exception of two patients whose only risk factor was age (66 years and 76 years, respectively). Despite comprising only 37.8% of the visit population, male decedents (30, 0.44%) exceeded female decedents (15, 0.13%) (p < 0.001). Self-assigned decedents’ race was as follows: American Indian or Alaska Native, 0; Asian, 1 (0.08%); black or African American, 12 (0.55%); Native Hawaiian or other Pacific Islander, 0; white, 23 (0.25%); other race, 7 (0.29%); unknown, 2 (0.13%); patient refused, 0; not recorded, 0. These differences were significant (p = 0.03). The EHR insufficiently reflected ethnicity (null values in 84.3%, and unknown in an additional 2.4%), and statistical analysis was not undertaken.

During the period from March 1 to April 6, there were 37 deaths (0.21%). Of these deceased patients, 14 had been referred directly to the ED at their VUC encounter (Figure 5 a); 17 of the remaining 23 would have met age callback criteria in period 2. The ED referral rate from initial VUC visit was 5.7% during this period. The median time from VUC to ED presentation in these 37 deceased patients was 4 days (IQR = 1–7). Twelve patients presented to the ED within 48 hours of their initial VUC visit; 9 were referred. Of 3 who were not referred, 2 had no pulmonary complaints at all, and 1 had no discernable pulmonary symptoms at the time of visit. No patient deaths occurred within 48 hours of the initial VUC visit (Figure 5b). The median time from VUC visit to death in period 1 was 13 days (IQR = 9–20).

Figure 5.

Shown here are (a) the time from index virtual urgent care (VUC) visit to emergency department presentation in decedents and (b) the time from index VUC visit to death in decedents. Both are stratified by referral patterns.

In period 2 there were 4,991 visits and 8 deaths (0.16%). Mortality rate did not differ between the periods (p = 0.59). Four decedents had been referred on their initial VUC visit. Three patients presented to the ED within 48 hours of their initial VUC visit (all referred). Two deaths occurred within 48 hours; both patients had been referred to the ED. The overall ED referral rate from initial VUC visits was 7.6% during this period, which was statistically different from the prior period (5.7%) (p < 0.001). Time to ED presentation and death in this period are provided in Figures 5a and b.

By comparison, there were 103 deaths in patients who were seen in network EDs and had been discharged (overall mortality 0.5% per visit and 0.6% per patient). ED dispositions were discharged in 86.4%, discharged to a nursing or group facility in 7.8%, departure against medical advice in 3.9%, eloped in 1.0%, and left without being seen after triage in 1.0%. The cause of death in these patients was confirmed COVID-19 illness in 87 (84.5%); probable COVID-19 illness in 5 (4.9%); other causes in 7 (6.8%) (bowel perforation, complications of malignancy [2], malignancy and chronic obstructive pulmonary disease [COPD], pulmonary embolism, drug overdose, and sepsis); and unknown/undetermined in 4 (3.9%).

Additional Follow-Up Measures

A total of 517 patients met criteria for COVID-19 telephonic screening follow-up at 36–48 hours. Of these patients, 366 (70.8%) were successfully reached for telephonic follow-up. At the follow-up call, 24 patients (4.6%) were hospitalized; 13 (54.2%) of these patients were referred during their index VUC visit. Among the 24 hospitalized patients, there was 1 mortality, who was one of those initially referred. Among the 11 remaining hospitalized patients not referred at the index VUC visit, there were no mortalities. At follow-up, 22 patients (4.3%) had a positive screening and were referred to the ED during the call. Of those referred during follow-up VUC screening, 7 patients (31.8%) were captured as having a return ED visit, and 5 (71.4%) of those who returned as instructed were admitted to the hospital. There were no mortalities among those with ED referral at 48-hour follow-up screening. Among 320 patients reached at 48-hour follow-up call who were not referred, 1 death occurred 13 days after the VUC visit. The decedent was referred at the index visit, but in conjunction with her family, persistently declined hospital care. Finally, among 151 patients (29.2%) who were not reached at VUC telephonic screening, there were 4 deaths. Of these 4 decedents, 1 had been immediately referred during the patient's index VUC visit and presented approximately 24 hours after the index encounter. One patient, comanaged with the oncology service, was referred by her oncologist at day 6 of illness. One patient presented to VUC for COPD, requesting a medication refill for chronic pulmonary disease, but denied active symptoms. The patient was not reached for a 48-hour follow-up screening and was noted to have expired at an outside health system 4 days after the index VUC visit. The final deceased patient requested only a work note at the index VUC visit and presented to the ED with COVID-19 complaints 4 days later. This patient expired 32 days after the VUC visit. Figure 6 summarizes these results.

Figure 6.

The outcomes captured during period 2, in which telephonic follow-up occurred in the virtual urgent care workflow for selected patient visits, are shown. ED, emergency department.

Last, in evaluating antibiotic use in a respiratory pandemic, VUC antibiotic prescription rates for patients given a diagnosis of sinusitis or upper respiratory infection were 12.1% (13.7% in period 1 and 6.5% in period 2).

Discussion

Contingency and crisis standards of care (CSC) are invoked when conventional care is no longer possible due to pervasive disaster and when demand surges overwhelm available resources.³² The National Academies implored health care systems to do everything possible to avoid CSC in the COVID-19 pandemic.³³ As shifts from conventional to contingency and crisis standards occur along a continuum, recognition and mitigation of surge must occur as early as possible. Telehealth carries significant value in the pandemic setting, including decreased exposures to patients, providers, and those in transit; ongoing monitoring; education and treatment recommendations; and patient reassurance. Patients who were fearful of or outright refused to seek in-person care no longer required a physical visit. VUC removed barriers for providers, who could work virtually themselves from homes and offices and flexibly navigate between ED and VUC settings according to demand.

The VUC platform sustained access to care for thousands of patients. This immense workload was highlighted by the VUC's excess of the network's ED visits for 25 days. Our single-network VUC exceeded the pre-COVID volume seen by a North Carolina statewide VUC center by 30 times.³⁴ Continuous evaluation of “failures” (for example, dropped visits) during the crisis improved situational awareness to devise solutions in a flexible platform (for example, bringing on additional providers with nonconventional and unfixed shifts to meet demand or convert to alternative means of reaching patients in the face of technological disruption). Brief, minor technological disruptions were apparent. There did not appear to be a coincident increase in no-shows after check-in during the periods of highest volume, which would have suggested patients becoming frustrated with provider delays. We have separately reported that patients welcomed telehealth as a health care delivery modality. They were satisfied with their VUC care (average score 4.4/5), would recommend it (4.6/5), and would use VUC services again (4.4/5).¹⁶

In the midst of any crisis, operational quality and safety cannot be ignored. “Virtual perfection”¹ pronouncements must be pragmatically assessed. Along with the operational strain of a surge in patient volume, the VUC cohort included many patients who had multiple comorbidities. This presented a risk of demographic shift (from “typical” urgent care cohorts to urgent and chronic care, compounded by COVID-19). COVID-19 thus presented a catch-22 to vulnerable groups. Telemedicine services were provided to avoid hospital referrals⁵—with the intent to minimize exposures—and many patients refused referral. However, patients with risk factors for disease complications were precisely the patients most likely to need a physical and diagnostic assessment in complex decision-making scenarios to determine suitability for ongoing outpatient management. More than half of the overall cohort and 74.1% of patients aged 60 years or older had comorbidities. Interestingly, younger patients were more likely to have a VUC revisit, while older patients, as expected, had more frequent ED presentations.

These data extend to the outpatient setting previous findings of higher adverse outcomes in COVID-19, specifically mortality, in males^{35 , 36} and those of self-assigned black or African American race.^{37 , 38} VUC return visit and mortality rates compared favorably with those patients managed as outpatients after presenting to the ED. Although the comparator group is imprecise, many of these ED patients similarly did not receive laboratory testing or radiographs, as these modalities were actively discouraged for patients not anticipated to require admission. Return visit causality may be difficult to determine, including anticipated and nonpreventable progression of disease, an unrelated second diagnosis, a missed diagnosis, inadequate or inappropriate therapy of a known diagnosis, unanticipated progression of disease, inappropriate discharge instructions, and inappropriate follow-up. However they represent an operational load that must be retriaged, evaluated, treated, and dispositioned.

The balance of responsibility of patients to seek follow-up versus the requirements of providers or health systems to actively follow them has been interminably debated, codified, regulated, and litigated.39., 40., 41., 42., 43. We found the statement “it is the patient's responsibility . . . to return as advised for ongoing assessments of health, illness, and treatment outcomes”⁴⁴ to be problematic in the face of pandemic. Many patients outright refused in-person assessment, follow-up, and ED referrals, despite explicit instructions. Follow-up calls have been part of ED safe discharge strategies for more than 30 years.^{45 , 46} A 70.8% successful contact rate compares favorably with previous reports,45., 46., 47. which is notable in crisis. Historically it has been difficult to demonstrate a definitive decrease in revisits and mortality, but multiple studies report referral and aftercare interventions as a direct result of follow-up calls.^{45 , 46 , 48} A 48-hour follow-up process targeting those aged 60 years and older to assess for length of illness, symptom progression, and associated risk factors correlated with a decreased mortality rate, although this did not achieve statistical significance. Although a number of epidemiological confounders affect mortality, we did find that a 48-hour follow-up process improved the safety of patients undergoing telehealth visits for suspected COVID-19–like illness, in that 4.3% of patients were recommended for in-person evaluation, of whom the majority were admitted. Opportunities for further refinement would include broadening enhanced callbacks to other high-risk groups. Planners, operational leadership, and payers should anticipate and support the full-time equivalents required to accomplish critical callback and follow-up functions performed by various provider levels.

Limitations

This was a descriptive study of a single-network health care entity expanding VUC services during pandemic response to a public health emergency. Volume diverged from normal operations at peak by more than 25 times baseline and was uncontrolled. Definitive COVID-19 testing was largely unavailable and actively discouraged by public health authorities. Call type and provider level (previously board-certified emergency physicians and then expanded to EM physician assistants and nurse practitioners) did not mirror normal operations. VUC patients are not directly comparable with ED patients; various complex factors contribute to decision thresholds to access these various care modalities. However, our data provide a pragmatic spectrum of patients presenting during the COVID-19 pandemic, which can help hospitals plan for future surges. We provide comorbidity data, so that our patient cohorts could be placed in context with other telehealth programs. Based on our manual reviews of decedents, comorbidities were incompletely recorded in EHR problem lists and past medical history fields. This would suggest that that our percentages provide only a lower bound for extant comorbidities. There is no Emergency Severity Index (ESI),⁴⁹ such as that used for triage in EDs, to apply to VUC care. Measures of severity for use in virtual settings might be considered for future development, particularly in disaster. While our results may not be generalizable to all institutions, health care delivery structures, or disasters, they are reflective of a large-scale surge response in an academic medical network. We used surrogate markers to assess quality and safety, against a backdrop with no commonly agreed-upon benchmarks in pandemic.

Return VUC visits and return ED visits were not differentiated as to whether or not they were planned or unplanned, whether they were for the same reason as the VUC visit, or whether they represented simple progression of disease after a reasonable trial of outpatient management. Follow-up of patients was limited by the nature of multiple, independent health care systems in New York and the lack of a readily assessable national health care system database, which precluded us from completely evaluating visits to other EDs. However, both direct callbacks to patients and regional care functions within the EHR were used to mitigate this. Last, antibiotic prescription rates must be interpreted with caution, as outpatient treatment was largely empiric, weighted toward overtreatment to avoid social contact required of confirmatory diagnostics, and evolving.

Conclusion

Appropriately deployed and rapidly scaled VUC services appear to provide a pragmatic strategy to care for large numbers of patients during a pandemic or other surge scenarios. With this health care delivery modality, ongoing surveillance of technical/operational factors, as well as clinical factors, is critical for patient quality experiences and safety. The implementation of a follow-up process for patients at risk for adverse outcomes supports care in the setting of empiric risk stratification. Future efforts should be aimed at deploying diagnostic modalities and evidence-based guidance to improve telehealth assessments of patients with the potential for clinical decline, particularly for COVID-19. As virtual visits expand in both clinical and geographic scope, greater integration of all health care system records will be important to ensure continuity of care.