Strategies for Evaluating Telehealth

The ability for our health care system to adapt with extraordinary speed under crisis has never been more evident than now as we face the coronavirus disease 2019 (COVID-19) pandemic. Ambulatory clinics and primary care providers have seen shifts in patient demand for their services, and elective and scheduled care at hospitals has been reduced to mitigate the spread of infection. Almost overnight, the health care system has shifted toward providing care through telehealth platforms to avoid the catastrophic consequences of “doing business as usual,” making telehealth a leading modality of health care delivery.¹

WHY NOW?

The telehealth alternative to in-person health care expanded rapidly as new federal and state legislation passed and payers reimbursed telehealth more broadly. Jefferson Health, one of the nation’s early pioneers in telehealth, reported a 10-fold surge in demand, scheduling up to 600 telehealth visits a day. As with many hospitals across the nation, the Children’s Hospital of Philadelphia saw telehealth visits increase from 5 to 10 per day to >1500 per day. More than 800 providers at the Ann & Robert H. Lurie Children’s Hospital of Chicago were trained and privileged in a matter of weeks to expand telehealth services. Now is a critical opportunity to systematically evaluate telehealth care delivery, identify patient cohorts that can benefit, and explore ways to incorporate telehealth into patient care workflows. Using this knowledge, we will evolve our health care system to improve how care is delivered now and during crises.

Telehealth features that distinguish it from conventional face-to-face methods of care delivery include the heavy reliance on technology for encounters and the inability to “lay hands on” patients. Because telehealth usage surged during COVID-19 and patients and providers are becoming accustomed with the technology, outcome information is largely disorganized and dispersed among organizations. Although a few measurement standards exist to guide the assessment of telehealth’s impact on the care delivered, current literature lacks a unified approach to evaluate telehealth in pediatric health care delivery. The National Quality Forum (NQF) framework offers a comprehensive guide for developing telehealth measures under 4 domains (access to care, financial impact and/or cost, experience, and effectiveness) and 53 measurement concepts.² In the World Health Organization framework, similar measurement concepts are applied in an approach that considers the maturity stage of a telehealth program.³ For example, the focus of a pilot telehealth program may be feasibility, whereas the focus of a more mature program may be scalability and impact to the health system. In a technical brief from the Agency for Health Research and Quality (AHRQ), researchers describe available research on patient outcomes and telehealth use and therefore emphasize the criticality of including the impact on health outcomes in any telehealth evaluation.⁴

ANSWERING THE CALL FOR TELEHEALTH EVALUATION AND MEASUREMENT TOOLS

Invaluable work done by the above organizations inspired Supporting Pediatric Research on Outcomes and Utilization of Telehealth (SPROUT), a part of the Section on Telehealth Care of the American Academy of Pediatrics, to synergize the above work into a single framework that researchers can use to study telehealth’s impact on patients. In the SPROUT Telehealth Evaluation and Measurement (STEM) profile (Table 1),⁵ the NQF, World Health Organization, and AHRQ concepts are reorganized into 4 measurement domains: (1) health outcomes; (2) health delivery (quality and cost); (3) experience, and (4) program implementation and key performance indicators (KPIs). The STEM profile is meant to communicate telehealth’s value to 4 key stakeholder groups: patients, providers, health systems, and payers.

Table 1.

Four Measurement Domains of STEM

Domain	Example of Potential Measures	Details and Specifications for Examples
Health outcomes (measuring the individual or population level related to physiology, mental health, and quality of life; these measures may come from diagnostic tests and encounter records or from patient-reported outcomes)	Mortality	No. deaths from COVID-19 divided by the No. patients with positive test results for COVID-19
	BMI and percentile	Patient’s BMI over time, >95% (obesity)
	Type 1 diabetes blood glucose management: monitoring and treatment (HbA1c target [<7.5%], with individualized considerations)	Percentage of children and adolescents with type 1 diabetes whose most recent HbA1c level is <7.5% (American Diabetes Association guidelines)
Health delivery (quality and cost) (measuring timely access, effectiveness of care, costs, safety, and equity)	Access: children and young adults aged 12 mo to 19 y who had a visit with a primary care practitioner	An HEDIS measure: https://www.ncqa.org/hedis/measures/children-and-adolescents-access-to-primary-care-practitioners-cap/
	Access: follow-up care for children aged 6–12 y who were prescribed an ADHD medication	An HEDIS measure: https://www.ncqa.org/hedis/measures/follow-up-care-for-children-prescribed-adhd-medication/
	Effectiveness: preventive care guidelines	Compliance with preventive care guidelines (ie, counseling on nutrition and physical activity for children and adolescents done through either an in-person or a telemedicine visit)
	Effectiveness: vaccination rate	Vaccination coverage in specific pediatric age cohorts
	Safety: antibiotic prescription rate	Rate of children aged 3 mo to 18 y who were given a diagnosis of upper respiratory infection and were not dispensed an antibiotic prescription (HEDIS): https://www.ncqa.org/hedis/measures/appropriate-treatment-for-children-with-upper-respiratory-infection/
	Safety of telehealth visits: percentage of unanticipated ED visits	No. completed telehealth visits with unanticipated, subsequent ED visits divided by the No. completed telehealth visits
	Patient cost: average miles saved from not traveling to the provider’s office	Miles from home to clinic or hospital setting: tracking distance allows for calculations of time saved and dollars saved
	Equity: percentage of clinical encounters delivered via telehealth in communities with low and high incomes	No. telehealth visits divided by the No. clinic encounters (telehealth and in-person visits) among patients from different income levels
Individual experience (measuring the quality and characteristics of the telehealth encounter itself and its impact on the individual patient or provider in terms of workload burden, satisfaction, and issues experienced with technology and logistics)	Net Promotor Score	A common customer-experience metric used to measure customer loyalty to a company that provides a specific service for that customer, often an estimator of customer satisfaction
	Patient centeredness: responses to survey questions about patient centeredness	How much did the provider include you in the decision-making? How much did you feel that the provider heard all your concerns?
	Staff wellness and burnout	Examples tools: Maslach Burnout Inventory, Oldenburg Burnout Inventory, Physician Work Life Study single-item measure, Stanford Professional Fulfillment Index, and Well-Being Index
	Patient burden reduction: saved workdays or reduced No. missed school days	No. lost workdays or missed school days due to health appointments
	Usability	Did you have difficulties getting to your appointment (either traveling to the clinic or using the telehealth system)? Did you have any technical difficulties?
Program implementation and KPIs (measuring program performance, implementation process, and benchmarks with peer programs; measuring how the system adapted human resources, processes, and tools)	System changes	No. and type of changes to the system to accommodate the program (eg, hiring remote diabetes educators, purchasing devices, investing in technology support services, and creating telehealth platforms)
	KPIs	No. telehealth visits completed within timeliness of care targets set by the program

ADHD, attention-deficit/hyperactivity disorder; ED, emergency department; HEDIS, Healthcare Effectiveness Data and Information Set

Health Outcomes

Identifying health outcome targets early is equally important as identifying quality outcomes when implementing telehealth services. Health measures are available through various federal agencies, national committees, and academic societies, such as the NQF, which summarizes the work from >40 measure steward organizations, such as the AHRQ and the Centers for Medicare and Medicaid Services.⁶ Telehealth’s rapid adaptation creates an opportunity to compare outcome measures after telehealth and in-person visits among patients with chronic conditions, such as hemoglobin A1c (HbA1c) in children with type 1 diabetes, BMI percentile to monitor obesity, or psychometric questionnaires to assess anxiety disorder symptoms in children. Publicly available data on COVID-19 disease burden and policy changes in specific geographies⁷ can be used to compare endemic areas with and without telehealth.

Health Delivery (Quality and Cost)

Telehealth has the potential to impact the quality and cost of health care delivery, including areas of access, effectiveness, cost, safety, and equity. With “stay-at-home” mandates in many communities, telehealth can help increase access and minimize the need for in-person appointments. Such impact can be measured at the pandemic’s start in terms of the proportion of in-person appointments that were successfully converted to telehealth encounters. Conversely, access may be impacted negatively in populations without access to Internet services, computers, or smartphone technology necessary for telehealth visits. Effectiveness is known typically as the extent evidence-based care is delivered reliably and consistently to patients; this care is often described by clinical recommendations. Although the convenience of telehealth may afford providers more opportunities to improve patient compliance, providers must continue to follow recommended practices and not skip safeguards, as reported recently on the tendency to overprescribe antibiotics in treating upper respiratory infections when patients are seen via telemedicine.

Travel time and miles avoided by using telehealth can be translated to cost savings by using standard conversion formulas (eg, $0.57/mile as business travel for providers or $0.17/mile as medical purpose travel for patients). However, the recent telehealth surge has exposed an equity gap in families who cannot access telehealth because of a lack of technology resources.

Experience

The provider and patient experience with a telehealth encounter and the logistic impact on their daily lives are important considerations. The Telehealth Usability Questionnaire, Telemedicine Satisfaction and Usefulness Questionnaire, Patient Assessment of Communication during Telemedicine, and the Net Promotor Score are examples of assessment tools that can be used to assess satisfaction with the provider-patient communication, technology, and usefulness. The impact on provider’s workload and disruption of family routines (ie, work and school) are personal burdens that may affect appointment adherence and overall satisfaction. Other experiential factors like encounter duration, video and audio quality, and connectivity should also be tracked.

Program Implementation and KPIs

In a time of rapid health care system transformations, it is critical to monitor both the creation of new telehealth programs and how existing programs change. Future economic analyses will be possible if programs gather data on the costs of telehealth service expansion and delivery, such as time and resources spent to credential and prepare members of the care team, establish new billing procedures, and provide telehealth tools. As implementation science frameworks guide program execution, programs should track KPIs to self-assess and benchmark with peer institutions.

CONCLUSIONS

Telehealth offers valuable opportunities to improve the process of health care delivery that may translate to better health outcomes. Identifying and studying these opportunities require a measurement strategy that allows data to be aggregated across multiple health systems, making possible the study of rare conditions and comparisons of different locations and methods for delivering services via telehealth. The STEM profile offers a construct to define and organize telehealth measures in terms of health outcome, health delivery quality and cost, and individual experience, as well as emphasizes program implementation and benchmarks. Findings from rigorous telehealth program evaluation in these areas can be used to inform data-driven reimbursement and policy changes that encourage appropriate telehealth use, especially amid the explosion of telehealth services associated with the COVID-19 pandemic.

ABBREVIATIONS

AHRQ:

Agency for Health Research and Quality

COVID-19:

coronavirus disease 2019

HbA1c:

hemoglobin A1c

KPI:

key performance indicator

NQF:

National Quality Forum

STEM:

Supporting Pediatric Research on Outcomes and Utilization of Telehealth Evaluation and Measurement