Standardized Neurodevelopmental Surveillance of High-risk Infants Using Telehealth: Implementation Study during COVID-19

Supplemental Digital Content is available in the text.

Abstract

Introduction:

Neurodevelopmental surveillance is critical for high-risk infants following neonatal intensive care discharge and is traditionally performed in-person. COVID-19 interruption of regular surveillance necessitated a rapid development of telehealth models for effective and standardized care.

Methods:

We used implementation science and lean methodologies to develop an effective telehealth neurodevelopmental surveillance program for high-risk infants. Interventions included reorganization of visit flow processes and a telehealth toolkit for standardized neurological and developmental assessments. We tested and improved our intervention through plan-do-study-act cycles, value-added analysis, and parent- and provider-satisfaction questionnaires. Process metrics (standard elements, subspecialty referrals, diagnostic tests, and prescriptions ordered) were compared in group-level analyses between telehealth patients (N = 97) March 16, 2020–July 1, 2020 and a matched in-person cohort at the same period the previous year. Run charts examined shifts in balancing measures (provider efficiency and missed visits) over 8 weeks before and after implementation.

Results:

Primary outcomes were visit completion (100%), patient parent satisfaction (>90% strongly agreed or agreed telehealth procedures were valuable and easy to use) and ability to accurately diagnose cerebral palsy (no statistical difference with comparison visits). Providers (N = 6) rated telehealth experiences favorably. Process metrics indicated no differences between telehealth and in-person visits (all P > 0.05). Following telehealth implementation, provider efficiency increased to near baseline (median 88.9% versus 91.7%) and median missed visits decreased to 0% from 20% (in-person).

Conclusions:

Implementation of telehealth for neurodevelopmental surveillance in a tertiary high-risk infant follow-up clinic successfully provided standardized and timely care during stay-at-home orders; broader telehealth applications may overcome access barriers in this field.

INTRODUCTION

Surveillance for neurodevelopmental risks is critical for high-risk infants after neonatal intensive care. Early, accurate detection of developmental concerns allows referral to needed diagnostic assessments, procedures, and specialists, all impacting developmental trajectories into childhood and adolescence. Rigorous use of standardized schedules with psychometrically sound assessments for disorders facilitates diagnoses and treatment referrals; it also creates entry points for participation in new evidence-based research interventions.¹ However, most neurodevelopmental surveillance is designed for in-person follow-up, with direct interactions between nurses, medical providers, therapists, patients, and their parents. When barriers to access occur (eg, COVID-19 pandemic), an adaptation of neurodevelopmental surveillance becomes essential.^2,3

In March 2020, follow-up programs for high-risk infants across the United States were closed to in-person visits.⁴ Vulnerable individuals (eg, those with disabilities) were also most likely to suffer from discontinuation of regular developmental services and isolation. Developing new models for delivering developmental surveillance was urgent, especially in a large tertiary care program serving as the catchment area for 4 states and >12 Level III neonatal intensive care units (NICUs).

A close partnership already existed at our institution between clinical/organizational entities and a large research program conducting interventional and observational studies in infants at high-risk for neurodevelopmental problems. The research program piloted early interventions for infants with cerebral palsy (CP) through telehealth due to geographic restrictions stemming from a catchment area >150,000 km².⁵ Researchers adapted therapist assessments and physician neurological examinations for videoconferencing, even temporarily delivering examination kits, technology, or internet connectivity to homes. The clinical and research teams hypothesized that through collaboration with organizational partners in the hospital’s neonatal service line and experience in leveraging implementation science principles, they could develop an effective telehealth neurodevelopmental surveillance program for the youngest high-risk patients. We believed the implementation’s success would be demonstrated through primary outcomes (visit completion, parent satisfaction, and accurate diagnosis of CP), process measures (standardized element completion rate, referral rates, procedures and prescriptions ordered), and balancing measures (provider efficiency and missed visits). Matched in-person visits during the same period in 2019 provided a comparison group.

METHODS

The Active Implementation Framework (Fig. 1)⁶ was utilized, with LEAN methodology⁷ for process and value-added analyses, and 2 plan-do-study-act cycles of improvement.

Fig. 1.

Active implementation framework for telehealth. PDSA, plan-do-study-act.

Setting

In 2019, the high-risk infant follow-up clinic at our tertiary care referral center had 3,972 developmental visits, seeing all infants at high-risk for delays (eg, prematurity, birth depression with hypoxic ischemic encephalopathy, other neurological insults, intrauterine growth restriction, and neonatal abstinence syndrome) and conditions prompting pediatrician concern.⁸ Visit schedule is preset, with every infant receiving 3–4, 9–12, 22–26, and 33–36 month visits with standardized assessments by examiners whose reliability is continuously verified. Infants requiring additional care are seen in an equally standardized fashion at necessary intervals.⁹

Team

The follow-up team operates within a hospital and clinical leadership matrix: a business manager leads operations with a nursing clinical lead and specialized nurses; a director leads the clinical team of physicians, nurse practitioners, and therapists, who partner with dietitians and social workers. Multidisciplinary visits are templated to provide excellence in care while maintaining optimal efficiency of space and template utilization. Both teams have personnel educated in quality improvement, implementation science, and operational management, ensuring rapid and continuous change cycles and robust change culture.^10,11

Planning Phase

Each assessment in the standard schedule had a strong evidence-base, and was evaluated for appropriateness and feasibility of conversion to telehealth (Fig. 2), with some already designed for video review (eg, Prechtl’s General Movements Assessment [GMA]).¹² Because the clinic had GMA-certified personnel, infant on-screen visualization during telehealth visits with appropriate placement and parental guidance was an easy conversion.

Fig. 2.

Schedule of standardized high-risk infant follow-up visits with assessments. Bayley, Bayley Scales of Infant and Toddler Development; CBCL, Child Behavior Checklist; ITSP, Infant Toddler Sensory Profile; PLS, Preschool Language Scales; PDMS, Peabody Developmental Motor Scales; TIMP, Test of Infant Motor Performance.

Conversion of motor assessments requiring complex positioning or prescribed toys was more challenging. Parents could not easily perform the Test of Infant Motor Performance¹³ in a standardized manner. Similarly, rigorous and rapid telehealth replication of the Bayley Scales of Infant and Toddler Development, Third Edition¹⁴ is difficult. Instead, the team chose the Developmental Assessment of Young Children, Second Edition (DAYC-2),¹⁵ which is designed for home administration. It includes child behavioral observation and parent report while allowing for guided examiner challenge of developmental milestones using common household objects. The DAYC-2 is validated¹⁶ to predict infant CP risk in low-resource settings.¹⁷ Because nonemergent procedures (eg, magnetic resonance imaging) were postponed, the situation presented many challenges common to low-resource or difficult-access environments. The research team developed and trialed a protocol for standardized DAYC-2 telehealth administration. A written script clarified all DAYC-2 items, and a filmed demonstration video complemented the training protocol.

Conversion of the standardized neurological exam to telehealth was more straightforward. All clinic personnel were highly experienced in administering the Hammersmith Infant Neurological Examination (HINE)⁹; 2 medical providers of telehealth are licensed trainers who conduct standardized workshops throughout North America to train high-risk infant providers. Researchers trialed and refined an administration protocol with video and written instructions: (1) to test the feasibility of a parent accurately performing the maneuvers and (2) to test examiners’ ability to score during teleconferencing. Figure 2 compares final standardized in-person and telehealth assessments.

Installation Phase

Initially, the clinic had no telehealth technical capabilities for patient interactions, registration, or shepherding through multidisciplinary providers. However, the NICU follow-up program electronic health system (EHS) was flexible (EPIC platform), and the hospital quickly added secure Zoom video conferencing capability. The research team had previously used Zoom due to its intuitive operational nature, high-quality video, and ability to pin patient views, allowing providers to maintain a child-focus. Operational leadership accelerated laptop acquisition (EPIC- and Zoom-equipped) and ensured rapid clinical personnel training. Initially, we planned telehealth visits in regular patient rooms with providers sequentially entering the room (eg, nurse, physician, and therapist) with the clinical setting as reassuring physical background. To allow remote provider participation, additional choices included a home-office or a hospital photograph as institution-provided virtual background. Nurses modified intake questions to include medical history, review of systems, and developmental screening. Scheduling priorities shifted to infants who might miss critical developmental windows for feeding, sensory, and motor concerns at 3–4 months corrected age (CA). Other priorities included addressing any pressing concerns identified in previous visits or during nurse telephone triage. The hospital provided a standard nursing script to ensure compliance with healthcare systems regulations and realistic parental expectations of telehealth visits. These elements and patient identification were documented in easy-to-use mandatory EHS phrases. Hospital informatics personnel provided a 1-hour basic introduction to EPIC-ZOOM; no institutional training specific to telehealth etiquette, practices, or challenges were yet available. Instead, the research team (who had performed telehealth sessions approximating standard care for past studies) shared experiences and solutions with the clinical team.

We trained all providers (medical and therapy) in new assessment elements: first reviewing protocols and videos, then attending an interactive session with a neurodevelopmental provider (a senior HINE trainer for North America) with over 10 years’ experience administering the DAYC in clinical and research settings. Additionally, HINE examiners observed the senior trainer twice and were observed for 2 patient sessions each. Research practitioners and therapists were available for 1 month to answer trainee questions.

Early Implementation Phase

The team tested 2 patients through the following initial visit flow: (1) nurse-delivered information on telehealth; (2) nursing assessment; (3) physician-guided parent-performed neurological examination; (4) parent-concern answers; (5) therapist functional evaluation with treatment recommendations; and (6) follow-up (Fig. 3). Providers followed in sequence, notified each other of findings, allowing subsequent providers to build upon previous assessments. The first cycle visits averaged 102 minutes.

Fig. 3.

Telehealth value added analysis for the 3- to 4-month standardized visit. A and B, First visit process flow was mapped in swim lanes corresponding to multidisciplinary provider types. Then, a value-added analysis was performed from the parent/patient dyad’s perspective. Value-Added process steps met the following criteria: (1) transformed care in a way that moved it closer to its final state; (2) the step was unique and did not represent rework to correct previous steps performed incorrectly; and (3) the parent cared that the step was performed to achieve a successful visit and be willing to pay for the step. If the step did not meet all criteria in the affirmative it was not a Value-Added step, but was classified as business value-added (failed to meet parent willingness to pay for the step, but must be completed to comply with regulations or meet a business requirement) or non–value-added (fails to meet either other category and is considered waste). C, After value-added analysis, process improvements resulted in a 90% decrease in waste to the parent/patient dyad.

A value-added analysis revealed the following (Fig. 3):

• 1-Preregistration in the EHS was challenging when performed during the visit, accounting for 10–20 minutes of wasted time and causing parent/provider frustration.

• 2-Nurse-to-physician communication required provider and nurse to either be co-located (defeating social distancing) or have an additional call for off-site providers.

• 3-Similar problems occurred transitioning from medical provider to therapist and visit wrap-up.

• 4-Parents had waiting times in front of blank screens during transitions.

Conversely, quality of care did not appear to suffer: parents were extremely satisfied with the visit, providers were able to assess referral and prescription needs, and follow-up appointments had identical criteria as for in-person assessments.

Visit process refinements developed based on value-added analysis (Fig. 3):

• 1-Parents preregistered in EHS with the nurse and hospital scheduling team assistance.

• 2-The medical provider participated in all three components of the visit, making a brief introduction at the first contact and visit flow clarification for parents.

• 3-Providers listened with microphone- and camera-muted to nursing intake, minimizing family distractions. This added provider time to the visit but eliminated signing-out between nurse and provider, and allowed providers to fill-in key documentation components during nursing assessment.

• 4-Therapists participated (muted and video-off) during physician neurological assessments and verbalized impression, allowing therapists to refine developmental history and intervention goals. Medical providers informed parents they would be listening throughout and rejoin after therapist assessments.

• 5-Medical providers completed documentation of examination, overall impression, and plan during therapist evaluation. The final plan occurred after therapist recommendations.

• 6-Providers addressed final parent questions, and confirmed recall for the next visit; after-visit summary completion and mailed immediately afterward.

Although changes nominally added time to physician and therapist schedules, they reduced parent waiting, removed the need for sign-out between providers, and allowed completion of documentation at the close of the visit, thus requiring no further allocation of provider time. After the initial phase, all providers were trained in revised procedures and had observed standard visits to answer questions and ensure reliability.

Full Implementation

For week 1, we scheduled 2 visits per half-day to assess patient flow and challenges inherent to a new system: 4 physicians, 1 occupational therapist, and 1 physical therapist conducted 13 visits in 3 days. Although no further assessment administration challenges were identified, registration processes remained problematic. After solutions devised by informatics support and clinic organizational leadership were implemented, we conducted a ramp-up stage with 3, then 4 visits per half-day per provider, or 8 visits per day (Fig. 4). New EHS forms with unique fields would have been optimal; however, short turnaround (<1 week) necessitated clinic personnel to create therapist-customized automatic phrases instead. Less stable internet connectivity for families living in remote areas was a challenge that required provider adaptation (shifting location and completing appropriate sections without visuals).

Fig. 4.

Balancing Measures. A, Missed visits represent the percentage of patients who did not show for their visits compared to all scheduled visits, with the goal being 0%. This excludes patients who called to cancel 48 hours in advance. B, Provider efficiency represents the percentage of templated patient-visit slots per provider and per clinic half-day with the goal number being 4.5 or 100% (equivalent to 9 high complexity visits per day). Efficiency is influenced positively by scheduling providers to maximum capacity and negatively by cancelations that occur within 48 hours and cannot be replaced by waitlisted patients.

Study Methods

Primary outcomes were visit completion, parent satisfaction on a telehealth questionnaire adapted from one previously published,¹⁸ and accurate high-risk for CP classification or CP diagnosis (per guidelines).¹⁷ There was a 4-week lag between survey start and telehealth implementation because no online resources existed to solicit parent feedback, necessitating phone calls. Calls occurred between 2 weeks and 1 month postvisit, with results entered into paper forms. Secondary outcomes were provider reports of visit effectiveness and delivery systems. Process metrics were completion of prescribed elements, referral to needed services, diagnostic procedures ordered, and prescriptions written. Telehealth patients were matched by CA at visit and primary referral diagnosis to patients during the corresponding period in the previous year. All referrals and practice guidelines were left unchanged from before COVID-19, presuming rapid return to typical function. Balancing measures were provider efficiency and missed visits measured by run charts in the 8 weeks before and after clinic shutdown.

Statistical analyses were group-level ANOVAs and Pearson’s Chi-squared comparisons. Due to the small number of comparisons and relatively large cohort size (N = 194), we did not control for multiple comparisons.

Ethical Considerations

Our Institutional IRB determined this quality improvement study did not require review or approval.

RESULTS

Primary Outcomes

All telehealth visits were completed despite occasional technical challenges. Of 68 parents surveyed, 43 (63.2%) were contacted by phone in the required time. The previous in-person survey completion rate was 9% (2019). The primary reason for low follow-up rate was the study’s nature, precluding research procedures, schedules, or methods to contact parents. No parent refused to answer once contacted. Parent satisfaction was high across questions; >90% of respondents strongly agreed or agreed that procedures were valuable and easy to use (Table 1). The number of accurate high-risk CP classifications and CP diagnoses, was not significantly different between the telehealth cohort and matched cohort (Table 2).

Table 1.

Primary Outcome: Parent Satisfaction with Telehealth Visits (N = 43)

	Strongly Agree	Agree	Neutral	Disagree	Strongly Disagree	NA
Zoom was easy to use	28 (65.1)	13 (30.2)	1 (2.3)	1 (2.3)	0 (0.0)	0 (0.0)
The videoconferencing sessions were helpful	28 (65.1)	13 (30.2)	1 (2.3)	1 (2.3)	0 (0.0)	0 (0.0)
The nurse asked about my concerns	31 (72.1)	11 (25.6)	0 (0.0)	1 (2.3)	0 (0.0)	0 (0.0)
The visit with the doctor/nurse practitioner was helpful to understand how my child is doing	26 (60.5)	16 (37.2)	0 (0.0)	1 (2.3)	0 (0.0)	0 (0.0)
The visit with the therapist was helpful to understand how my child is doing	17 (39.5)	12 (27.9)	8 (18.6)	1 (2.3)	0 (0.0)	5 (11.6)
I like the procedures used during the visit	20 (46.5)	16 (37.2)	7 (16.3)	0 (0.0)	0 (0.0)	0 (0.0)
I felt well supported by the telehealth team despite the distance	31 (72.1)	10 (23.3)	1 (2.3)	1 (2.3)	0 (0.0)	0 (0.0)
I would recommend the telehealth visit to other families	24 (55.8)	15 (34.9)	0 (0.0)	4 (9.3)	0 (0.0)	0 (0.0)
I am satisfied with the overall quality of the telehealth visit	26 (60.5)	14 (32.6)	1 (2.3)	2 (4.7)	0 (0.0)	0 (0.0)

Data are n (%), percentages may not add up to 100 because of rounding.

NA, not applicable.

Table 2.

Visit Comparison During COVID-19 and Corresponding 2019 Period

	Telehealth, N = 97	In Person, N = 97	P
Visit characteristics
Corrected age at visit, months, mean ± SD	7.2 ± 7.6	7.2 ± 7.4	0.962*
Prematurity, N (%)	55 (56.7)	55 (56.7)	>0.99†
HIE, N (%)	4 (4.1)	4 (4.1)	>0.99†
IUGR, N (%)	9 (9.3)	9 (9.3)	>0.99†
NAS, N (%)	20 (20.6)	20 (20.6)	>0.99†
Other, N (%)	9 (9.3)	9 (9.3)	>0.99†
Primary outcome
No. high-risk for CP classifications or CP diagnoses, N (%)	12 (12.4)	10 (10.3)	0.651†
Process metrics
Standard visit elements
HINE, performed/required (%)	77/83 (92.8)	79/86 (91.9)	0.824†
GMA, performed/required (%)	48/53 (90.6)	42/51 (82.4)	0.220†
Developmental assessments, performed/required (%)	73/77 (94.8)	85/89 (95.5)	0.833†
Number subspecialty referrals made	76	52	0.053*
Number diagnostic procedures ordered	6	12	0.245*
Number prescriptions written	10	5	0.181*

*P based on group-level ANOVA.

†P based on Pearson’s chi-square.

HIE, hypoxic ischemic encephalopathy; IUGR, intrauterine growth restriction; NAS, neonatal abstinence syndrome.

Secondary Outcome

Overall, telehealth providers had positive perceptions of ease of use and visit value (see Table, Supplemental Digital Content 1, http://links.lww.com/PQ9/A276). Provider responses indicated certain types of visits might be better suited to telehealth than others and that remaining focused throughout the visits was often tiring. Providers cited added challenge of reading social cues from parents, difficult when image quality or camera positioning was suboptimal. The stress of communication was heightened when needing to provide a challenging, high-risk for CP classification or new CP diagnosis. However, it was still possible in an honest, direct, and hopeful manner.¹

Process Metrics

Cohorts were well matched by visit CA and primary referral reason to high-risk follow-up. Telehealth visits (compared to in-person) had no differences in standard elements completed, subspecialty referrals made, diagnostic tests, and prescriptions ordered (Table 2). Elements not completed or performed via telehealth were due to child non-cooperation with a parent or low video/connection quality.

Balancing Measures

Median telehealth missed visit rate was 0% after implementation, compared to 20% in person in the eight weeks prior; median provider efficiency, while lower during ramp-up, quickly approached baseline (88.9%) (Fig. 4).

Barriers and Limitations

Due to safety and policy concerns, we could not determine if parent/provider satisfaction was high relative to a no-visit option or whether satisfaction would remain high given an in-person option. We prioritized patients who (1) would soon “age-out” of developmental windows or needed these visits critically and (2) had parents agreeable to telehealth. In the research program, additional measures overcame internet connectivity and video equipment challenges (eg, COVID-safe iPads and internet hotspot “pick-up”/“drop-off”). Although clinic personnel could not feasibly replicate this, these issues were less challenging during COVID as parents were often home-bound and telehealth participation voluntary. Our hospital had no prior telehealth procedures or resources on the necessary scale. Hospital academic and organizational partners later developed comprehensive courses, guidelines, weekly communications, video materials, and tip sheets to support providers and parents, consistent with best practices discussed below. Satisfaction surveys were automatically and immediately sent through MyChart. Finally, clinical assessments, including neuropsychological testing, are demonstrably reliable compared to face-to-face assessments.¹⁹ However, additional in-person evaluation or other diagnostic testing may be necessary for some children with more complex diagnostic profiles.²⁰

DISCUSSION

Although it has not been previously reported, implementation of synchronous in-home telehealth program for developmental assessment and monitoring of high-risk infants and children is feasible, as demonstrated in the current study. High parent- and provider-satisfaction were consistent with published reports for general telehealth visits^20–23 and feasibility, accuracy, and clinical utility estimations.^20,23,24 There are currently no comparable studies, as telehealth use in this specialized population has been limited.^3,20,25,26 Lack of direct comparison stems from the study population (majority of ex-NICU patients) and modality (in-home, synchronous telemedicine approach with standardized provider-guided, parent-performed neurological and developmental assessments). Asynchronous home video methods have allowed infant gross motor or development evaluation.^12,24 For young children with neurodevelopmental disorders (often autism spectrum disorders), studies of synchronous telehealth behavioral assessments and interventions^{20,23–25,27} highlighted decreased stress, higher likelihood of successful evaluation, better child cooperation in the home environment, improved access to geographically limited subspecialists, and reduction/elimination of barriers to visit attendance (special transportation of medical equipment).^22,23,27

As in other reports,²¹ study providers cited reduced social cues as challenging for emotional connections. Most providers are confident in their ability to express empathy, compassion, and build trust during in-person encounters; yet, many fear that telemedicine may compromise patient-provider relationships. However, providers and parents felt access to services outweighs this drawback,²¹ though it may require providers developing new skills and approaches.²⁸ Therefore, standardized educational training for trainees and current providers in “website manner,” engagement, and relationship-building over telehealth is critical.^2,29,30 Other improvements include physical and staffing changes to customize telehealth clinics, aligning camera placement to facilitate eye contact, and technical staff to troubleshoot.^21,28

As healthcare systems continue to build and refine telehealth, it is imperative to consider impacts on patients and families with differing socioeconomic capabilities and geographic or technological limitations.³¹ The “digital divide” describes gaps in access to information and technology created by limited access and utilization of technology.³² Adverse consequences of underlying personal and sociocultural barriers predominantly affect low-income, rural, disabled, racial and ethnic minorities, and elderly populations—further exacerbating existing health and healthcare disparities.^28,32 Strategies proposed to address challenges in the digital divide include expanding broadband access, accommodations for language, literacy, disability, and telehealth literacy training.^28,32,33 Telehealth equipment provided in community settings^20,27 can help circumvent common challenges (eg, home internet access, device availability). During COVID-19, safety concerns prevailed, making this approach less desirable, as it may be during influenza season. Despite obvious challenges, most studies continue to show high patient and provider satisfaction with telehealth visits.^2,21

Next Steps

Many processes enabling widespread telehealth utilization during the COVID-19 pandemic were facilitated by governmental agencies rapidly lifting provisions that limited telehealth services’ reimbursement and relaxed strict privacy rules for multiple telecommunication platforms.^4,34 Although prioritizing patient health and safety, continued support of telehealth services by the Centers for Medicare and Medicaid Services, private insurance providers, and other governing bodies will be necessary to navigate sustained reimbursement and acceptance of these methods.

Beyond COVID-19, neurodevelopmental surveillance of high-risk infants might include a hybrid model of telehealth and in-person visits, with algorithms for which patients and visit types are most appropriate for each modality, based on parent-stakeholder feedback, scientific evidence, and care excellence. Comparison of technology-based surveillance with the traditional in-person approach will likely necessitate well-designed comparative effectiveness studies. Long-term outcomes of telehealth will also need to be evaluated. Healthcare’s digital revolution and rapid establishment of telehealth programs could expand and improve the efficiency of care for patients and families, avoid travel and wait times, and provide vital health services. However, careful consideration must be given to ensure continued high quality, evidence-based patient care while focusing on strategies to improve systemic and structural health inequities for our most vulnerable patients.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.

ACKNOWLEDGMENTS

We would like to thank the entire EDC clinic team. Although they could not all return to work, their selfless dedication and caring for the most vulnerable patients drove this project. We wish to thank the NCH Neonatal Marketing Team for helping make all schedules and clinic information available at https://www.nationwidechildrens.org/specialties/nicu-follow-up-programs. We thank Melissa Moore-Clingenpeel for her ongoing support of BBOP projects.