A Mixed-Methods Assessment of Coronavirus Disease of 2019–Era Telehealth Acute Care Visits in the Medical Home

Abstract

Objectives

To compare acute care virtual visits with in-person visits with respect to equity of access, markers of quality and safety, and parent and provider experience, before and during the coronavirus disease 2019 pandemic.

Study design

We compared patient demographics, antimicrobial prescribing rates, emergency department (ED) use, and patient-experience scores for virtual visits and in-person care at 2 academic pediatric primary care practices using χ² testing and interrupted time series analyses. Parent and provider focus groups explored themes related to virtual visit experience and acceptability.

Results

We compared virtual acute care visits conducted in March 2020-February 2021 (n = 8868) with in-person acute care visits conducted in February 2019-March 2020 (n = 24 120) and March 2020-February 2021 (n = 6054). There were small differences in patient race/ethnicity across the different cohorts (P < .01). Virtual visits were associated with a 9.6% (−11.5%, −7.8%, P < .001) decrease in all antibiotic prescribing and a 13.2% (−22.1%, −4.4%, P < .01) decrease in antibiotic prescribing for acute respiratory tract infections. Unanticipated visits to the ED did not significantly differ among visit types. Patient experience scores were significantly greater (P < .05) for virtual acute care in overall rating of care and likelihood to recommend. Focus group themes included safety, distractibility, convenience, treatment, and technology. Providers were broadly accepting of virtual care while parental views were more mixed.

Conclusions

Telehealth acute care visits may not have negative effects on quality and safety, as measured by antimicrobial prescribing and unanticipated ED visit rates. Efforts to increase parental acceptance and avoid creating disparities in access to virtual care will be essential to continued success of telehealth acute care visits.

The coronavirus disease 2019 (COVID-19) pandemic has been associated with many changes in health care delivery workflows, including a significant acceleration of telehealth adoption (“virtual visits”). Although a rapid shift to telehealth visits was a necessary response to the pandemic, it has raised concerns regarding how to ensure an equivalent level of safety, effectiveness, efficiency, and equity in virtual visits These concerns are particularly relevant for same-day acute care (ie, “sick visits”), where there is increased potential for errors of omission or commission.

Before the pandemic, studies and clinicians raised concerns about the potential for inappropriate care in telehealth.^{1 , 2} Studies have shown increased antibiotic overuse in telehealth in nonmedical home settings, such as free-standing urgent care centers. However, less is known about how such concerns translate to telehealth providers in a child's medical home, especially during the pandemic.³ There are also concerns regarding the safety of virtual visits, given the lack of standardized frameworks to identify which types of patients or concerns necessitate an in-person visit to ensure safe and high-quality care. A final consideration with the shift toward telehealth has been equity of access. Parents and patients may hesitate to engage in telehealth acute care visits due to a variety of technological, cultural, and personal factors. With the move to telehealth, new digital disparities may emerge that reinforce existing or historic disparities in access to care.

The rapid shift to telehealth necessitated by state shutdown orders prevented broad-scale assessment of parent or provider perspectives regarding virtual same-day sick visits and precluded the development of tools to ameliorate concerns about safety, quality, and equity of virtual acute care visits. The objectives of this study were to compare the quality, safety, and equity of access and use associated with virtual acute care visits with the quality, safety, and equity of access and use associated with in-person acute care visits before and during the COVID-19 public health emergency; to explore initial parent and provider experiences with virtual acute care visits; and to begin to understand drivers of parent and provider satisfaction with visits to help plan continued use of telehealth during the COVID-19 pandemic and beyond.

Methods

This mixed-methods study evaluated virtual acute care visits at 2 academic primary care practices serving 22 000 patients (majority non-White and publicly insured) where telehealth was the primary mode for acute care delivery during the first 12 months of the pandemic. Before the pandemic, parents or patients would call and schedule in-person acute care visits, with limited telehealth options. Nurses would triage acute care requests only on parental request or by protocol in specific clinical scenarios cases (eg, fever in a neonate or parent report of respiratory distress) (Table I; available at www.jpeds.com). During the first year of the pandemic, as in-person acute care was more limited, all in-person acute care visits (except walk-ins) were triaged by a nurse for suitability before scheduling, but scheduling staff could schedule a virtual visit without nursing triage unless the reason for the visit met the prespecified clinical scenarios that required nursing triage or was a genitourinary concern that was deemed inappropriate for a virtual visit (Figure 1; available at www.jpeds.com).

Figure 1.

Algorithm for scheduling virtual visits. GU, genitourinary; VV, virtual visit.

Antimicrobial stewardship, unanticipated emergency department (ED) visits within 72 hours of the index acute care visit, and the demographics of patients presenting for visits were assessed across 3 cohorts: in-person same-day acute care visits during the 12 months before the Massachusetts state of emergency for COVID-19 (which began March 10, 2020) and in-person and virtual same-day acute care visits during the first 12 months of the pandemic. (To assess equally sized months for the interrupted time series analyses, we used 30-day months and so assessed visits from March 15, 2019, to March 10, 2020, as the prepandemic period and March 11, 2020, through March 5, 2021, as the first year of pandemic period.) We identified in-person same-day acute care visits (“in-person visits”) using visit–type signifiers in electronic health records, so as to exclude well child visits and prescheduled follow-up visits. We identified virtual same-day acute care visits (“virtual visits”) using same-day scheduling and problem-based Current Procedural Terminology codes, with visits with comprehensive physical examination codes excluded. All patients received their primary care at the 2 sites or our affiliated adolescent clinic.

To evaluate appropriateness of antimicrobial prescribing at virtual visits, we evaluated monthly antibiotic prescription rates. We assessed rates of all antibiotic prescribing and of antibiotics prescribed at visits billed for acute respiratory tract infections (ARTIs). ARTI visits were defined using an International Classification of Diseases, Tenth Edition, condition list from Kronman et al, which includes diagnoses such as acute suppurative otitis media, group A streptococcal pharyngitis, and viral upper respiratory infection. This list includes conditions for which antibiotics prescriptions may be appropriate at in-person visits but for which antibiotic prescriptions are likely inappropriate at virtual visits, given that they require further examination or testing.⁴

We defined rates of unanticipated ED visits at our affiliated children's hospital within 3 days following an acute care encounter as a proximate measure of virtual visit safety. We excluded acute care visits where a pre-ED arrival note was present within the electronic medical record indicating that the provider who had conducted the visit had sent the patient directly to the ED. This was designed to avoid overestimating the effect of telehealth visits, as children are often sent to the ED from acute care visits for minor procedures (eg, laceration repair) that cannot be done in our acute care space.

Tests of significance of changes were performed using interrupted time series analysis for monthly rates at in-person visits before the pandemic shutdown and in-person and virtual visits during the shutdown.⁵ We created dummy variables for pandemic shutdown and virtual visits. To permit different time trends for virtual and in-person post pandemic, an interaction term between time postpandemic and virtual visits was used. To evaluate equity of access and use, we used χ² testing to assess for significance of differences in patient/parent-provided race and ethnicity, insurance access, and preferred language across the 3 cohorts.

The experiences of parents whose children had virtual visits were collected through a review of patient experience scores, followed by focus groups of parents and providers. Patient-experience survey results were reviewed across the 3 cohorts. The specific domains assessed were parent overall rating of care, likelihood to recommend practice, concern showed, and explanation provided by the visit provider. To analyze experience scores, we used χ² testing to compare rates of “top box” scores across the 3 cohort groups, ie, comparing the proportion of parents who gave the highest possible survey response to each question. To further explore parental experience of virtual visits, we conducted virtual focus groups among parents whose children had attended virtual visits before the focus group. Eleven parents participated in the 2 groups. Provider experiences were examined through a single focus group with 6 board-certified pediatricians who had conducted virtual visits during the pandemic shutdown period. The interview guide questions are in the Appendix (available at www.jpeds.com). Because of the exploratory nature of this analysis, we limited the approach to identification of themes via review of the focus group transcripts and did not perform a full thematic analysis.

All quantitative analyses were performed in SAS, version 9.4 (SAS Institute). This study was deemed to be quality improvement and therefore exempt from institutional review board approval.

Results

Demographics of Patients Seen

There were 24 120 acute care visits in the 12 months before the pandemic state of emergency; 6054 in-person acute care visits and 8668 virtual acute care visits took place in the subsequent 12 months (Table I). The proportion of publicly insured patients and rates of limited English proficiency were similar across visit types. Overall, there were significant differences in race and ethnicity between those with in-person visits before the pandemic and those with in-person or virtual visits during the pandemic (P < .01), but the magnitude of the differences was small, with patients with telehealth visits slightly more likely to report themselves as multiracial or White and less likely to be Black or Hispanic/Latine. The proportion of patients with unknown race/ethnicity data was greater during the pandemic period (Table II ). The most common diagnoses at in-person visits in the year before the pandemic were viral ARTIs, unspecified fever, rash, otitis media, and pharyngitis, whereas at in-person visits in the subsequent year the most common diagnoses were fever, viral ARTI, rash, sexually transmitted infection screening or treatment, and pharyngitis. The most common diagnosis codes at virtual visits were COVID-19 exposures or need for testing, rash, viral ARTI, fever, and nonspecific cough.

Table II.

Demographic categories of patients accessing in-person and virtual visits

Demographic categories	In-person visits		Virtual visits
	Before pandemic	First 12 months of pandemic	First 12 months of pandemic
Payor
Public	74.4%	74.2%	75.3%
Public as secondary	7.5%	7.9%	7.1%
Commercial	16.4%	16.8%	16.7%
Unknown	1.8%	1.1%	0.9%
Race/ethnicity∗
Hispanic or Latino	39.4%	36.1%	38.8%
Black or African-American	25.6%	25.7%	23.5%
Multiracial	15.2%	15.5%	15.7%
Unknown, declined, blank, or nonclassifiable	11.8%	14.9%	14.4%
White	3.7%	3.8%	3.9%
Asian	2.3%	1.6%	1.8%
Other†	2.1%	2.5%	2.0%
Interpreter need
No interpreter needed	77.8%	78.2%	77.0%
Interpreter needed	22.1%	21.8%	22.9%
Unknown	0.2%	0.1%	0.1%

^∗P < .01 for group differences.

^†“Other” includes “Brazilian,” “Cabo Verdean,” and “Middle Eastern/North African.”

Antimicrobial Stewardship

Antibiotic prescribing was significantly less frequent at virtual visits than at in-person visits. Prepandemic, 13.3% of in-person visits resulted in an antibiotic prescription, compared with 11.7% of in-person visits and 3.4% of virtual visits during the pandemic. Antibiotic prescribing in virtual visits initially rose in the first 2 months after the shutdown but then decreased over time. For visits with ARTI diagnoses, the differences were more pronounced. In total, 27.6% of in-person visits with ARTI diagnoses prepandemic and 29.1% of in-person visits during the pandemic resulted in antibiotic prescriptions, compared with 9.7% of virtual visits. On interrupted time series analysis, virtual visits were associated with a significant decrease of 9.6% in all antibiotic use (11.5%, 7.8%, P < .001) whereas in-person visits postpandemic were associated with a significant increase of 3.7% in antibiotic use (1.6%, 5.8% P < .01) as compared with predicted prepandemic in-person visit antibiotic use. There was a small but significant decrease of in-person antibiotic prescribing over time after the pandemic shutdown, with a small and nonsignificant increased trend in antibiotic prescribing for virtual visits (Figure 2 , A). For ARTI visits, virtual visits were associated with a 13.2% decrease in antibiotic use (22.1%, 4.4%, P < .01) and the pandemic period was associated with a 13.7% increase in antibiotic use (4.0%, 23.3% P < .01). The trend over time was a decrease in ARTI antibiotic prescribing for both in-person and virtual visits, but this was not significant (Figure 2, B). For virtual visits, the most common diagnosis for antibiotic use was for cellulitis/abscess, with pharyngitis, acute otitis media, unspecified bacterial infection, and rash the next most common diagnoses.

Figure 2.

Interrupted time series analyses for A, all antibiotic use, B, antibiotic use for acute respiratory tract infection diagnoses, and C, unanticipated ED visits.

ED Use

Excluding visits in which patients were sent to the ED from an in-person acute care visit or were recommended to go to the ED after a virtual visit, 0.8% of virtual visits and 2.3% of visits prepandemic and during the pandemic resulted in ED visits within 3 days. On interrupted time series analysis, there was no significant difference in the rate of ED visits after index acute care visits associated with visit type, the pandemic period, or time after the pandemic related shutdown (Figure 2, C).

Parent Experience

Parents rated virtual visits significantly greater than in-person visits (both prepandemic and during the pandemic) on likelihood to recommend the practice or provider on patient experience surveys (P < .01 on χ² testing). In addition, parents rated virtual visits greater on including parents in decision-making, care for parents’ worries or concerns, explanations for treatments and diagnoses, and overall ratings, but these differences were not statistically significant (Figure 3 ).

Figure 3.

Virtual and in-person patient experience rating scores (percent selecting highest category).

Patient and Provider Perceptions of Virtual Visits

Eleven parents of children who had virtual acute care visits participated in 2 moderated focus group sessions (consisting of 6 and 5 parents, respectively) and 6 physicians who had experience in providing virtual visits participated in one moderated focus group session. The participants had all recently participated in virtual visits at the time of the focus groups, with parents having attended virtual visits within 2 months of the focus groups and providers having conducted virtual sessions within 1-2 weeks of the focus groups. Key themes included safety, distractibility, treatment decisions and communication, convenience, and technology use. Themes and example quotations are available in Table III .

Table III.

Key themes and example quotes from provider and patient experience of virtual visit focus groups

Themes	Example provider quotes	Example parent quotes
Safety of virtual visits	“There are many things you can make your way around virtually, but there are lines we have to remember…there is a risk of harm because we don't lay hands on the patient” “I do tons of checking in…certainly it's not scalable, but at the same time it's become one way…the question is how do you build a system for follow-up.” “I feel like as a provider I've done an about face—I'm pretty much willing to see anything [via telehealth) and even if my advice is going to immediately be, ‘nope you need to be seen in person,’ that's ok.” “Finding out who is in the room, who could be listening [to the visit with an adolescent], and a lot of time the confidential information does not come out…ability to do a HEADSS assessment is a nonstarter”	“I definitely feel like going into the hospital just seems safer than a virtual video…a mother while her kid is not feeling well, that's stressful. It's less stressful when you're in the hospital and there's millions of doctors around.” “I don't know what is inside his ears, they say press his belly, I don't know what I'm pressing for” “I'm not a doctor…I don't know what's going on inside of his body…I ask ‘could I please be in person’—for me it's ok, but for the kids it's different…I know what I'm feeling, but with my kids I don't know what they're feeling” “If your kid is 1 year or 2 year and has a fever of 101…virtual zoom, isn't really helpful, especially for a new mother…I just feel like when they're younger than 2 years old and have a temperature of 101 or 102, they should definitely be seen even during COVID-19, because they're little babies and you're not a doctor or have the knowledge what is going on”
Distractibility	“Because there are unclear expectations, how [telehealth visits are] supposed to go, what the patient is supposed to get out of it…so [the] provider is distracted by real life and patients perceive virtual visit as not the same, and not as serious, so they come into doing their housework” “I have a kindergartener at home who struggles when I am working at home... crying and knocking on the door... distracting when I am trying to do [see patients]”	“I have a hyperactive child that keeps showing the doctor the toy and everything in the house…gets very distracted” “Not having nurses coming in and out, not having any interruptions [in virtual visits], it's nice” “I think you have their undivided attention [in virtual] …They are in a quiet room so I think you have their undivided attention”
Convenience	“There's an informal nature to it, partially a function of the convenience, [the patient] can be in the back of an uber and if it's time for [the] visit [they] can take it, that convenience has led to informality.”	“Virtual visit time is always convenient…I love the time…that's the only thing I love about it” “I think it's good because …you don't have to go to the office. I have an 11-month-old and sometimes it's hard to get him ready to go to the doctor”
Treatment decisions/communication	“I think I'm doing less, which is wonderful…in the beginning the presumption was every ear pain was an acute otitis media…I'm thankful that now we're doing less of that so we can have people come in so we can swab their throats or look in their ears”	“Any breathing issue, things with eyes, they can do an initial virtual visit but then if there are still issues going on they [should make] an in-person appointment”
Technical concerns	“There are some visits when you're able to take a look at the patient you get some sense of how sick/not sick. When it's a phone visit, which happens quite often [due to video not working] you have much more limited data”	“Hope would find a way to see the patient because not everyone has an iPhone or a laptop or a camera where they can have a zoom call”

HEADSS, adolescent confidential psychosocial history.

Safety

Parents expressed some concerns about attending a virtual visit. One parent said “I definitely feel like going into the hospital just seems safer than a virtual video…a mother while her kid is not feeling well, that's stressful. It's less stressful when you're in the hospital and there's millions of doctors around.” Another parent, who had been asked to help examine her child during the virtual visit, said “I don't know what is inside his ears, they say press his belly, I don't know what I'm pressing for.” Parents expressed comfort with virtual visits as a triage mechanism to determine whether a child had to come in for an in-person visit. Parents also mentioned concerns for virtual visits for younger children, with one saying: “I just feel like when they're younger than 2 years old and have a temperature of 101 or 102, they should definitely be seen even during COVID-19, because they're little babies and you're not a doctor or have the knowledge what is going on.” Provider feedback reflected similar concerns about limitations of virtual visits, with one provider noting, “There are many things you can make your way around virtually, but there are lines we have to remember…there is a risk of harm because we don't lay hands on the patient.” But in general, providers expressed comfort with virtual visits for most conditions, especially if there was the ability to bring patients in for in-person visits in the medical home.

Distractibility

Some parents appreciated that providers seemed less distracted, with one parent saying: “They are in a quiet room, so I think you have their undivided attention,” and another saying, “not having nurses coming in and out, not having any interruptions, it's nice.” Although parents felt providers were less distracted during virtual visits, providers shared that they were distracted by events in their own homes and felt parents were distracted with other tasks during virtual visits, including ones that could be unsafe (eg, conducting the visit while driving). One provider reported: “Because there are unclear expectations, how [telehealth visits are] supposed to go, what the patient is supposed to get out of it…so [the] provider is distracted by real life and patients perceive virtual visits as not the same, and not as serious, so they come into [them] doing their housework.”

Treatment Decisions and Communication

As providers became more comfortable with virtual visits, many were increasingly comfortable doing less, eg, deferring antibiotic prescribing for otalgia until an in-person examination could be arranged. Similarly, some parents expressed comfort with virtual visits as a triage mechanism to determine whether a child had to come in for an in-person visit. For example, one parent stated: “Any breathing issue, things with eyes, they can do an initial virtual visit but then if there are still issues going on they [should make] an in-person appointment.”

Convenience

Parents expressed appreciation for the ease and convenience of virtual visits—even parents who were less accepting of virtual visits overall appreciated the convenience aspect, with one parent declaring: “virtual visit[s are] always convenient… that's the only thing I love about [them].” Providers similarly appreciated the convenience of the virtual visits, but noted that it came at a cost, noting that the visit can “feel like it is never done…because you're not leaving, it feels, it's a paradox, I can go get a tea and the flipside is that work never leaves…being on Zoom, having to reengage even more meticulously, it's fatiguing [and] hard, hard.”

Technology

Most parents felt comfortable with the technology used, although some parents expressed concern about access for those without smart phones or laptops with cameras enabled. Providers reported that technical challenges can limit the data they have to make clinical decisions.

Discussion

In a multimodal analysis of virtual same-day urgent problem-based visits provided in the medical home during the first year of the COVID-19 pandemic, we found several encouraging factors about the use of virtual visits when coupled with more robust triage for appropriateness. We did not see a clear pattern of disparities in access, with similar rates of virtual visits for families on public insurance and with limited English proficiency and small differences in access for patients of diverse racial and ethnic backgrounds. Antibiotic prescribing in virtual visits increased slightly early in the pandemic but improved over time, and a large proportion of antibiotics were for conditions that may be appropriate for virtual visits, such as cellulitis. Furthermore, we did not see evidence of increased ED use that could indicate missed or delayed diagnosis with virtual visits. Our study also showed that many parents were accepting of virtual visits for urgent concerns, especially for older children.

Our findings showed that telehealth visits conducted within the medical home, in general and particularly for ARTI, may be less prone to overuse of inappropriate antibiotic prescribing than direct-to-consumer telehealth visits. It may be that telehealth providers from the medical home, with a stronger ability to follow-up on an evolving clinical course and with a foundation of trust between parent and clinic, feel less pressure to prescribe antibiotics in a telehealth encounter than direct-to-consumer telehealth providers. Our provider focus group participants described requesting patients return to clinic for in-person visits in lieu of prescribing antibiotics at a telehealth visit, adding some evidence for that hypothesis. The inclusion of expanded nurse phone triage and limited algorithms for deciding which conditions that were more likely to need an in-person physical exam (such as genitourinary complaints) for appropriate antibiotic prescribing also played a role in maintaining antibiotic stewardship in virtual visits.

Antibiotic prescribing via telehealth visits was found to be likely guideline-concordant when done in the medical home, with an increase in guideline-concordant antibiotic use later in the pandemic.⁶ This was similar to the trend we found of decreased antibiotic prescribing in telehealth visits for ARTI visits (which by nature are nonguideline-concordant) over time. This was likely due to increased comfort with telehealth visits by providers and improved access to in-person care as severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) rates decreased over the summer.

Our findings were encouraging in terms of disparities in access to digital care, with no significant differences in language type or payor, along with only very slight differences in race/ethnicity of families who had virtual as compared with in-person care. Our state mandated payment parity for virtual care for public and private payors during the COVID-19 state of emergency, which ameliorated potential drivers of inequities. Maintaining telehealth payment parity, as the American Academy of Pediatrics has advocated, will be vital for maintaining equitable access.⁷

Our study also showed that many parents were accepting of virtual visits for urgent concerns, especially for older children. We saw no significant differences in the demographics of patients using telehealth and in-person care, suggesting some degree of equity in access to telehealth. Telehealth visits also had greater patient experience scores when compared with in-person visits, both before and during the pandemic. Although those findings are encouraging as telehealth becomes more widespread, our parent and provider focus groups do provide some grounds for caution. Some parents felt uncomfortable with telehealth visits for acute care, especially regarding the lack of a physical examination and requests that parents perform aspects of the physical (eg, palpation of the abdomen). Requiring telehealth visits for parents who do not want them risks decreasing trust and harming the patient experience. This is particularly true for members of historically medically marginalized communities, for whom building and maintaining trust is paramount. The provider focus group also raised some concerns about provider satisfaction in virtual visit provision. Providers raised concerns about work–life balance and the strain of being in front of digital devices during the entire clinical session. Atop the stressors of providing care during the pandemic, the additional stress regarding virtual visits and lack of in-person care could increase provider burnout risk, with the resulting negative effects on provider retention and patient care.

There are several limitations to our study. Thematic analysis on the focus groups was not performed, so subtle themes in parent or provider responses may not have been captured. Focus groups were conducted with English-speaking parents who had attended virtual visits and were comfortable with using Zoom, which may bias the results in favor of those who were most likely to support virtual visits. Due to limitations in our billing systems and video platform we could not assess the modality (ie, video-based vs audio-only) of virtual visits and we were unable to assess for disparities in rates of successful connection to virtual visits. Anecdotally, a not-insubstantial number of patients were unable to log onto the virtual visit system or had technical issues that required transition to phone calls, although the frequency of this concern was not something we measured directly within this study. As payors require more specific information about telehealth modality, obtaining these data will be simpler in future analysis. Other studies have found that non-White, limited English-proficient, and underinsured families were more likely to have audio-only visits.8, 9, 10, 11 We also did not assess missed care opportunity rates, as to which others have found an association with markers of socioeconomic disadvantage.¹² Furthermore, although we did not find significant differences in access metrics between in-person and virtual care, if in-person care were characterized by underlying inequities in access, then the lack of difference would indicate the mere perpetuation of existing disparities. More complex and greater acuity patients may have had been more likely to have had in-person visits, either due to parent choice or to nursing triage, biasing the results toward the null for number of ED visits after telehealth visits.

With regards to our interrupted time series analyses, there may have been have competing explanations for the results we found. For example, as the shift to virtual visits occurred at the same time as school and business closures, this resulted in a decreased rate of many infectious diseases in children,¹³ which may have affected antibiotic prescribing rates. In addition, there were 2 small additional efforts to decrease inappropriate antibiotic use during this period: a few nursing testing slots for testing for Group A streptococcus when a patient had a virtual visit for pharyngitis were launched in May 2020 and an existing otitis media care pathway was annotated to suggest not prescribing antibiotics at virtual visits in September 2020. With respect to the revisit metric, it is possible that patients with virtual visits may live further away from our center than patients who came for in-person visits, and therefore may have gone to other EDs, biasing the results toward the null. In addition, some patients who may have been dissatisfied with virtual visits may have chosen to seek care elsewhere, which would also have the effect of biasing toward the null. Lastly, we did not assess costs of care. Studies have shown that telehealth does not necessarily lead to reduction in total cost of care,¹⁴ which will be important to assess in pediatrics.

Overall, our multimodal study of telehealth for sick visits showed broad acceptance of telehealth for acute care visits, with some aspects of care being rated more highly in virtual format and with encouraging signs regarding the quality of care in telehealth visits. However, if acute care telehealth is to remain the “new normal” in pediatric medical home care delivery, the concerns about safety, equity, and parental comfort with virtual acute care visits must be measured and addressed. Balancing those concerns, along with uncertain payment structures for telehealth,¹⁵ with the convenience of telehealth visits will be a key challenge for the viability of telehealth in the medical home in the years to come.

Appendix A. Supplementary Data

Table I.

List of conditions requiring triage by nursing

Algorithm: symptom or situation	Action
•Active bleeding/cut that may need stitches •Allergic reactions/EpiPen questions •Asthma attack/wheezing/difficulty breathing •Behavior or abuse concern •Burn •Cast (soiled or wet) •Chest pain •Family planning requests •Fever in an infant <3 months (rectal temperature >38°C or 100.4°F) Fever in an infant 4-6 months > 38.5°C or 101°F •Foreign body in nose or ear •Head injury that involves loss of consciousness OR head injury in any child <2 years •Headache with neck pain •Inconsolable infant <6 months •Ingestions (foreign body or poisoning) •Lethargic/unresponsive •“Private” If there is a safety concern •Seizures •Testicular pain/injury •Unable to eat or drink for >8 hours	Requires triage by RN
Rainbow/KASA patients all of the above AND •A concern about equipment malfunction (such as gastrostomy tube/nasogastric not working) •Rainbow patient/KASA (any age) calling to book with chief complaint “fever” phone room asks “would you like to speak with a nurse?” (If a parent/guardian expresses great concern for their child and you are unsure because the concern is not listed in algorithm, reach out to RN for advice.)	Requires triage by RN
∗All other calls	Nonurgent; message in a pool [in the electronic health record]

Title	Primary care phone triage algorithm
File Name	Algorithm and numbers 1 pager.docx
Date Created	1/1/2017
Date Modified	06/18/2020

Rainbow/KASA, children with medical complexity enrolled in primary care care coordination programs; RN, registered nurse.