Evaluating Neonatal Telehealth Programs Using the STEM Framework

Abstract

The COVID-19 pandemic has caused an explosive adoption of telehealth in pediatrics . However, there remains substantial variation in evaluation methods and measures of these programs despite introduction of measurement frameworks in the last five years. In addition, for neonatal health care, assessing a telehealth program must measure its benefits and costs for four stakeholder groups – patients, providers, healthcare system, and payers. Because of differences in their role within the health system, each group's calculation of telehealth's value may align or not with one another, depending on how it is being used. Therefore, a common mental model for determining value is critical in order to use telehealth in ways that produce win-win situations for most if not all four stakeholder groups. In this chapter, we present important principles and concepts from previously published frameworks to propose an approach to telehealth evaluation that can be used for perinatal health. Such a framework will then drive future development and implementation of telehealth programs to provide value for all relevant stakeholders in a perinatal health care system.

Models to assess telehealth programs

Several models for measuring telehealth have been published, most cited of which are from the National Quality Forum and World Health Organization. These frameworks focus predominantly on health care quality domains instead of health outcomes and have not been applied to perinatal health. An evaluation toolkit developed by Supporting Pediatric Research in Outcomes and Utilization of Telehealth (SPROUT) reorganizes these measure concepts into a health outcomes centric model. Specific information about each framework is discussed below.

National quality forum

The national quality forum's Telehealth Measurement Framework.¹ is a comprehensive review that identified existing measures and measurement concepts, organizing them into four domains (with subdomains): Access to care, Financial impact/cost, Experience, and Effectiveness. Access refers to the ability of patient, caregivers, and family members to receive care from the providing team and exchange relevant clinical information. Financial impact/cost effects are those affecting patient/family, care team, health system, payer and society. Experience refers to the usability and effect of telehealth on patient/family, care team member, and community and whether the care meets expectations. Effectiveness is measured at the system, clinical, operational and technical level in which health outcome is under the subdomain of clinical effectiveness. Across these domains, the NQF further defines 53 measure concepts in six key areas: travel, timeliness of care, actionable information, added value to provide evidence based best practices, patient empowerment and care coordination.

The NQF framework explains how to develop measures that predominately focus on evaluating telehealth's ability to deliver high quality healthcare. Importantly, it emphasizes the perspectives from four stakeholder groups (patient, care team, health system, payers) as well as the need to understand the impact of a telehealth program on the community. However, safety is included only as a patient experience and not as a health system factor. While health outcome is included in the clinical effectiveness section, it is not an essential part of evaluating any telehealth initiatives. In their appendices, the authors provide a comprehensive list of measure concepts that are mostly adult related, but nevertheless exemplifies how perinatal measures could potentially be derived.

World Health Organization

In 2016, the World Health Organization (WHO) with several collaborators, offered a measurement strategy that differentiates “monitoring” as measuring functionality, fidelity, stability, and quality of the telehealth system from:”evaluation” as measuring usability, feasibility, efficacy, effectiveness, and economic/financial effects of the telehealth system.² In addition, the WHO recommended that evaluators consider the technology's implementation stage (concept, prototype, pilot, demonstration, scale-up, integration/sustainability) when deciding on which measurement area(s) to focus on. During prototype and pilot stages of a new telemedicine program, assessment focuses on whether the system is:

• -Functional: meet technical specifications.
• -Feasible: works as intended in a given context.
• -Stable: have acceptable technical failure rates during normal and peak use.
• -Usable: can be used as intended by users.

As programs mature, it becomes relevant to assess whether users in the field can consistently accomplish the stated objectives (fidelity) and whether the intervention's quality level is able to yield the intended outcomes. At the scale-up/integration implementation stage, evaluators can study whether the system demonstrates measureable impact to processes and outcomes (efficacy), and how close is the user able to reach best or potentially better practice standards using the system in the field (effectiveness). Relative to these measures, quantifying cost and resource expenditures would also be important.³

While the WHO model has the advantage of offering programs an evaluation roadmap from inception to scale, like the NQF, it focuses mainly on telehealth use in the adult setting. Furthermore, it does not emphasize tracking health outcomes until later in the implementation cycle. We suggest that evaluators should clearly identify and articulate the clinical health outcomes potentially affected by telehealth at the prototype stage, even if these outcomes are measures that may take time to change or are dependent on other non-telehealth factors. This recommendation comes from the experience that system changes like telehealth implementation is costly and therefore, it is not enough to identify how healthcare delivery will be better, but also which healthcare outcomes we hope to improve.

A 2016 AHRQ systematic review illustrated the critical connections between telehealth interventions and clinical outcomes.⁴ Filtering over 1400 citations, the authors summarized 58 systematic reviews and reported the level of evidence on association between telehealth use and outcomes such as mortality, quality of life, and reductions in hospital admissions. Telehealth use included communication, counseling, and monitoring of chronic conditions such as cardiovascular and respiratory disease. However, in the area of maternal and child health, the authors concluded that while there could be enough primary studies to constitute some evidence (e.g. showing no benefit for home uterine monitoring), additional studies and systematic reviews are warranted.

Combining measurement frameworks to evaluate telehealth's impact in perinatal health

The American Academy of Pediatrics Section on Telehealth Care's SPROUT has combined the invaluable work of organizations described above with its member expertise into a toolkit called SPROUT Telehealth Evaluation and Measurement (STEM).⁵ STEM's four measurement domains: (1) health outcomes, (2) health delivery - quality and cost, (3) experience, and (4) program implementation and key performance indicators (KPIs) cover themes that are relevant to all four stakeholder groups in varying degrees (Fig. 1 ).

Fig. 1.

STEM - Health Outcomes centric Telehealth Evaluation Framework.

STEM's first domain, health outcomes, is arguably the most critical one because these measures represent the end goal of all efforts to deliver high quality healthcare – to make patients healthier.

This domain includes clinical measures of individual or populations, many of which are already collected in large neonatal data registries such as the Vermont Oxford Network (VON) and the Children's Hospitals Neonatal Database (CHND).^{24 , 25} The National Quality Forum and Center for Medicare and Medicaid Services endorses a few of these measures related to neonatal infection and perinatal complications.⁶ This domain also includes mental health measures such as anxiety, depression, and stress (i.e., Center for Epidemiologic Studies Depression Scale, Impact of event Scale – Revised, NICU Parental Stress Scale, Patient reported outcomes) as well as assessment of burnout in providers.^{7 ,} 26, 27 ^{– 28}

Metrics associated with the provision of healthcare services are in the second domain – the quality and cost of healthcare delivery. This domain includes most of the National Academy of Medicine's quality constructs (safety, timeliness, patient centeredness, effectiveness, equity).²⁹ plus cost/resource burden measures. Most of NQF's domains and subdomains map onto STEM's second domain. Examples include percent of pregnant mothers receiving timely prenatal care, percent of mothers who got education on breastfeeding, referral and completed visit rate to high-risk obstetricians when needed, access to mental health wellness programs during the perinatal and postpartum period, number of safety issues encountered per patient treated. Other examples look at the timely access to pediatric subspecialists and how to deliver best equitable practices via tele-consultation, tele-coaching and tele-training. Such compliance with “clinical pathways” has recently been trackable through monitoring of electronic order set usage.⁸ and HL-7 formated message exchanged in hospital information systems.⁹

It is important to measure costs in dollars and resource expenditures of a tele-resuscitation or teleconsultation program for both, the originating (location of patient) and remote (location of consultant(s) sites. The cost/savings impact of telehealth encounters includes miles spent or saved, cost incurred or avoided, and workdays and school days lost or gained for caregivers and providers. Assessing safety events can be tracked through the hospital's existing safety reporting systems and quality/safety departments.

Measures of equity and related social determinates of health are increasingly important, as COVID-19 has uncovered wide gaps in technology penetrance in underserved populations.¹⁰ Variables to track include caregiver's ethnicity, race, gender, language preference, payer mix, census-based markers like the social vulnerability index,³⁰ and social determinates. Understanding associations between disparities and health outcomes, delivery quality and cost is critical to ensuring that all patients can benefit from judicious implementation of telemedicine.¹¹

To make telemedicine systems more effective in delivering better care and health outcomes, implementers need to understand the provider and patient/family's experiences. STEM's third domain measures the individual experience and the logistical impact/changes these encounters have on their daily lives. Published assessment tools such as the Telehealth Usability Questionnaire, Patient Assessment on Communication in Telemedicine (PACT), TSUQ, and Net Promoter Score administered to NICU parents and providers can assess the usability of technology, satisfaction with the communication between providers and patient, and likelihood of recommendation.12, 13, 14 ^{– 15}

The Fourth domain encompasses Key Performance Measures that describe the operational aspects of the Telehealth program – number of video visits or tele-resuscitation sessions, number and type of technology issues, types of conditions addressed, number of patients enrolled, the size of the telehealth network and number of partnering institutions, operational costs and staffing expenditures. These measures are typically important towards the enterprise's overall strategic and budget; therefore, they can overlap with measures in other domains – e.g., cost effective analyses (domain 2) and KPI's (domain 4).

When assessing a particular telehealth program/initiative, telehealth evaluators are encouraged to identify 1,2 measures assignable to each STEM domain. While many measures of clinical outcomes and health delivery quality and cost offers objective data points and can be found in existing data sources, they should undergo statistical testing for reliability and validity. Likewise, surveys asking for individual opinions, experiences and preferences can yield rich subjective data but must be carefully distributed and worded to mitigate sampling and responder bias. Stakeholders can use data differently and people's perception and relative value of this information may alter their benefits to costs analysis. Understanding what different stakeholders perceive to be telehealth's benefits and cost can help implement telehealth more effectively.

To illustrate, Table 1 shows how to use the STEM toolkit table for two perinatal telemedicine interventions; (1) teleconsultation for newborn resuscitation in community hospitals and (2) post discharge video visits to patient homes. The intervention column describes each telehealth intervention. The data capture method is stated beneath each domain to highlight the importance of identifying reliable data sources early in the evaluation planning process. The domain one column defines the health outcomes belonging to each intervention - in our examples they are, respectively, first NICU admission temperature and average weight gain within six months after discharge. The domain 2 column states the health delivery quality/cost measures. For tele-resuscitation, adherence with a neonatal resuscitation practice pathway to manage airway emergencies (called MRSOPA.¹⁶) may be measured by video recording review. For post discharge video visits, healthcare utilization and safety catches may be monitored by the electronic medical record and locally used safety reporting systems. The domain three column describes the attitude and experience of patient, caregiver, provider, and other stakeholder towards the telemedicine process, including appointment scheduling, technology's usability, and satisfaction with the encounter. The last domain, program key performance indicators (KPI), describes summary statistics that are important to the hospital administration, such as encounter completion rates, incidence of technical issues, average cost to sustain the program, and benchmarks with other similar telemedicine programs. Once the team has defined the STEM dataset variables for the telemedicine intervention, they can assess equity by measuring and comparing the variables amongst different disparity cohorts. In the post discharge video visit example, weight gain, readmissions, and patient satisfaction may be compared between patient cohorts living in areas with high and low social vulnerability indices.¹⁷

Table 1.

Summary of NQF and WHO Domains and Subdomains.

		Domain/ Subdomain	Maturity Stage of Telehealth Program
NQF	WHO		Prototype	Pilot	Demonstration	Scale-up	Integration/Sustainability
	Monitoring	Functionality	√
		Stability	√
		Fidelity		√	√	√	√
	Evaluation	Feasibility	√
√		Efficacy		√	√	√	√
√		Effectiveness			√	√	√
√		Access to Care		√	√	√	√
√		Financial Impact / Cost		√	√	√	√
√		Usability and Experience	√	√	√	√	√

Applying stem to QI

The telemedicine implementer's dilemma is often deciding how best to integrate telemedicine into existing workflows in ways that would lead to better and measurable health outcomes and delivery quality. The driver diagram is a powerful quality improvement tool that links SMART aims (Specific, Measurable, Attainable, Relevant and Time-bounded) with interventions that can achieve those aims.¹⁸ Health outcomes are often measures that may not improve immediately whereas healthcare quality measures can be improved more quickly and therefore make good targets of SMART aims. STEM's domain two (Healthcare quality and cost) and three (Individual Experience) aligns nicely with what is typically measured in a QI project. In the tele-resuscitation example (Fig. 2 ), the health outcomes are neonatal morbidity and mortality rates while the SMART aim is “Achieve 95% concordance with neonatal resuscitation steps in community nurseries that are not staffed by neonatologists within 12 months.” This drivers diagram, best constructed by a stakeholder group composed of neonatal and obstetric clinicians, nurses, local physicians and respiratory therapists, identified its key drivers to be (1) resuscitation team having necessary skills and knowledge of best practice, (2) availability of expert consultants, and (3) teamwork. Note that up to this point, the SMART aim and drivers are not linked to telemedicine. The next steps are where the team identifies telemedicine interventions that could help accomplish the stated drivers and explain how each intervention benefits the baby being resuscitated. These benefits mapped back onto STEM domains/subdomains, completing the link from the main health outcome to interventions and STEM.

Fig. 2.

Drivers Diagram for Neonatal Teleresuscitation.

Assessment of telehealth value requires understanding of its stakeholders

The value equation can be summarized as benefits over costs where benefits are variables that add value when they increase, and costs are variables that lower value when they increase. Examples of “benefit” variables are measurements of quality, efficacy and safety in telemedicine care while examples of cost are resource usage and dollars spent delivering care.¹⁹

Differences in value perspective from each stakeholder type (patient and family, provider, health system, payor, and policymaker) could result in synergistic or oppositional levels of support for a telemedicine intervention. Sometimes patients and providers are placed in conflict with non-clinical stakeholders - a conflict that has shown itself in situations where payers believe that a treatment's costs outweigh its benefits, such as bone marrow transplantation for treatment-resistant breast cancer or coverage of antiviral treatment for hepatitis C, but other stakeholders such as patients and providers disagree. The ability for stakeholders to view and understand each other's value perspectives is needed to create a better health care delivery system.

Patient and family

To parents, high value healthcare not only includes better clinical health of their babies, but also seeing relief of their baby's pain,²⁰ effective communication from care teams to them and with each other,²¹ greater closeness and bonding with their baby,²² among others. These factors are counterbalanced by higher out of pocket healthcare expenditures, loss of work or school days, and medical harm. Often, parents do not consider their own wellbeing to be part of the “high value healthcare” of their child.

Provider

To perinatal providers, high value healthcare would include maternal and neonatal outcomes and the health of the caregivers like stress and anxiety. Helping caregivers cope with the psychological effects of having a baby in the NICU could help the child's long-term outcomes because higher levels of maternal stress have been associated with receptive language and adjustment problems at four years old.²³ Other high value factors to providers include the system's ability to help them deliver best and safer care, and higher reimbursement rates. In contrast, variables that lower healthcare value include waste (i.e., excessive waiting time, inefficiencies in process and workflows, defective equipment), avoidable readmissions and medical errors.

Health system

To health systems, a high value perinatal program typically shows improving neonatal outcome rates over time and comparable or better benchmarking with similar programs. Higher payer reimbursement rates are valuable to the health system and supports ancillary services like laboratory and diagnostic suites, other clinical services that often consult in the NICU like genetics and pulmonary as well as research and innovation. Variables that lower value are higher operational cost, waste, and medical errors. Whether avoidable readmissions are a bottom-line cost or benefit to health systems depends on whether their payer contracts impose penalties or not.

Payer

To payers, a high value perinatal program is typically one that delivers the best neonatal and maternal outcomes for its plan members at the lowest monetary cost. This rather cynical view has merit because it drives more efficient and effective evidence-based health care. The Center for Medicare and Medicaid Services and commercial payers are becoming proponents of value-based reimbursement models where providers are paid depending on patient outcomes rather than on volume of procedures completed. A result of this has been bundled payments that include payment for performance of quality measures such as postpartum visit rates, where health systems are responsible for cost management but still incentivized to adhere to best practices. To a degree, such strategies help align the value equation between payers, health systems/providers, and patients such that, for example, higher avoidable readmissions become a cost to all stakeholders. However, implementation will only be successful when such strategies are created and executed through collaboration with all stakeholders making their value equations transparent.

Lawmaker

Lawmakers are critical stakeholders who can enact laws and regulations that drive provision of high-quality healthcare. Their role in this system highlights the impact health care systems have on communities and society. Examples include those listed by the CDC's community health status indicators (i.e., no care in first trimester, infant mortality disparities). Lawmakers could be concerned with how health care provisions impact unemployment rates, and school attendance rates in the community.

Conclusion

In conclusion, telehealth is a health delivery tool offering opportunities to improve neonatal outcome and care delivery. A standard approach to evaluating neonatal telehealth programs would allow data to be aggregated across multiple health systems, making studies of rare conditions and comparisons of different locations and methods for delivering services via telehealth possible. STEM offers a construct to define and organize telehealth measures in terms of health outcomes, health delivery quality and costs, individual experiences, and program implementation and benchmarks. When evaluating neonatal telemedicine use, stakeholders and program directors should undertake efforts to identify actionable measures under each domain (Table 2 ).

Table 2.

STEM Measurement Domains applied to Perinatal Health.

	Teleconsultation for Newborn Resuscitation to community hospitals		Post Discharge Video Visits for babies with NG tube feeds
	Measure	Data Source	Measure	Data Source
Domain 1 Physical or Mental Health Outcomes	Hypothermia – # of 1st NICU admission Temperature < 36 °c (NQF)	EMR record	Weight Gain trends over 6 months post discharge	EMR record
Domain 2 Health Delivery Quality and Cost	Effectiveness - Compliance with Delivery Room Resuscitation Best Practice (MRSOPA)	Direct Observation, video recording review	Healthcare utilization - Readmissions prevented by video visits), NICU length of stay Safety – number of safety risks detected	EMR, safety reporting system
Domain 3 Patient/Provider Experience	Community hospital care team satisfaction with Tele-Resuscitation consultation	Telehealth Usability Questionnaire	Patient satisfaction with video visits and interaction with care team	Patient Assessment of Communication of Telehealth (PACT) questionnaire
Domain 4 Program KPIs	-# of video visits/month -Average # technical issues -Implementation cost Staffing needs	EMR, Issues tracking	-# of video visits/month -Average # technical issues -Implementation cost -Staffing needs	EMR, Issues tracking

Declaration of Competing Interest

The authors have no conflicts of interest to declare.