The telehealth engagement measure: a proportions-based measure of care delivered via video and phone for research, clinical, and policy assessments

Abstract

Objectives

In the absence of standardized measures, researchers have struggled to define meaningful use of telehealth (care received via video or telephone). We evaluated a novel Telehealth Engagement Measure (TEM) that quantifies an individual’s telehealth use, including telephone (TEM-t) and video (TEM-v), as a percentage of total care received.

Materials and Methods

In a retrospective cohort in the Veterans Health Administration (VHA), we identified 716,373 Veterans with at least three primary care visits between May 1, 2022 and April 31, 2023. We calculated the predicted annual TEM, TEM-t, and TEM-v stratified by age, gender, race/ethnicity, rurality, drive time, and Charlson Comorbidity Index, adjusting for patient demographics.

Results

The mean unadjusted TEM was 21.7%, TEM-t was 15.1%, and TEM-v was 6.6%. The predicted annual TEM decreased with higher age until age 80 (15.1% (95% CI 14.7,15.5)). This was driven by reduced video care utilization, which peaked at 11.2% (95% CI 10.7,11.8) among Veterans aged 20 and declined after age 50 (6.5%, 95% CI 6.2, 6.8). A higher drive-time to VHA was associated with a higher TEM, while Veterans with higher comorbidity were associated with a higher TEM-t.

Discussion

The TEM reflects the distribution of a person’s care across different modalities The measure can be applied in research and clinical settings and can be leveraged to examine the longitudinal impact of telehealth on a host of outcomes.

Conclusion

The TEM is a novel, proportions-based measure that can be used to assess telehealth utilization at various levels in a health care system.

Introduction

At the height of the COVID-19 pandemic, 50–70% of outpatient care in the United States (US) was provided through telehealth (e.g., synchronous telephone and video-based care).¹ Telehealth now accounts for approximately one in threee health care visits.^1,2 Reports suggest that 50 million in-person visits per year could be converted to virtual visits if adoption were extended equally across populations.³ Telehealth’s broad and growing use indicates it will remain a major part of healthcare. Despite telehealth’s increased growth and use, measuring its impact on health outcomes, utilization, and care quality remains challenging due to the lack of a measure that can (a) track telehealth’s impact over time, (b) integrate other types of care with telehealth, and (c) can be easily applied in clinical and research settings.^4–7

Various measures, including binary measures, counts, and frequency of telehealth use have been used to assess telehealth; each with limitations. A binary measure (e.g., never/ever had a virtual visit) are useful for assessing access to telehealth,^6,8–11 or significant single visits (e.g., follow-up post hospitalization),^12,13 but they cannot quantify the impact of additional visits beyond the initial visit, and have become less useful as telehealth adoption increases.^8,10 Count measures quantify the number of visits in a specific time-period but hard to compare between groups or across varying time periods. Count measures are also hindered by the dubious nature of attributing long-term outcomes to an arbitrary number of visits. Rate-based measures (e.g., visits per unit time) address some issues with count measures by standardizing the number of visits relative to a unit of time.^14–16 Yet, rate-based measures can still be skewed by high utilizers or underlying need for care (e.g., mental health care is provided more frequently than primary care).^17,18

Altogether, the current measures used to quantifying telehealth are not equipped to address two important questions on the use of telehealth, (1)“How do we calculate the cumulative impact of telehealth on health outcomes?” and (2) “What is the ‘right mix’ of telehealth and in-person care for any given patient?” As clinicians, researchers, and policy makers study telehealth’s impact on health, a new measure is needed to support longitudinal evaluation of telehealth use in the context of overall care patterns, including its integration with in-person services.

Objective

To address this gap, we examined the merits of a novel Telehealth Engagement Measure (TEM), which quantifies an individual’s telehealth use as a percentage of all care received. Using a nationwide cohort from the Veterans Health Administration (VHA), the nation’s largest integrated health care system and one of the largest utilizers of telehealth services in the United States, we describe and examine the TEM’s utility.^19,20 We also discuss its applications in clinical and research settings, and as a means of helping health care systems study and optimize their telehealth utilization.

Methods

Study population

In a retrospective cohort study, we evaluated Veterans’ outpatient primary care use using VHA electronic health records when telehealth rates stabilized following the acute phases of the COVID-19 pandemic; between May 1, 2022 and April 31, 2023.²¹ To ensure our study population comprised Veterans who had enough primary care visits to have a mix of care modalities, we excluded Veterans with fewer than three primary care visits and restricted eligible primary care visits to evaluation and management visits–where comprehensive assessment and management plans are enacted. We also excluded individuals with missing home address (needed to define drive time to VHA and rurality of their residence) (3.2% of data), or Charlson Comorbidity Index (CCI) (N = 12). We also excluded Veterans with outlier data, including those older than 101 years of age (N = 117), those with drive times over five hours (N = 5) and Veterans with more than 20 visits (N = 109) (Supplementary File 1, available as Supplementary material is available at [JAMIA Open] online). Our final analytic population comprised 716,373 individuals.

Telehealth engagement measure

The Telehealth Engagement Measure (TEM) is a proportionality-based measure developed for this evaluation, that represents the relative quantity of synchronous telehealth as a percentage of all care received in primary care. It can be defined in three ways: all telehealth (TEM) (e.g., [number of video + telephone visits]/[number of video + telephone + in-person visits]), just video-based interactions (TEM-v) (e.g., [video]/[video + telephone + in-person]), or just telephone-based interactions (TEM-t) (e.g., [telephone]/[video + telephone + in-person]). The TEM-v may be of interest in health care systems where video-based care is the preferred modality for remotely delivered care. However, as telephone care remains a common way to receive remotely delivered care,¹ the TEM-t may also be of interest when evaluating health equity and equitable access to telehealth care.²²

We calculated Telehealth Engagement Measures (TEM) (reported as percentages) using primary care visits. We identified primary care evaluation and management visits provided by primary care providers (e.g., including physicians, nurses, but excluding social workers, mental health clinicians) using VHA Managerial Cost Accounting Stop Codes and Current Procedural Terminology (CPT) codes (Supplementary File 1, available as Supplementary material is available at [JAMIA Open] online).^23,24 Visits were defined as in-person, video-based, or telephone-based encounters using Stop Codes as previously described (Supplementary File 1, available as Supplementary material is available at [JAMIA Open] online).^10,15

Covariates

We assessed individual demographic and clinical characteristics as of May 1, 2022 using VHA’s electronic health care records.²¹ Race and ethnicity were based on the most frequently identification in patient health records. Missing data on race, ethnicity, and marital status were categorized separately. Gender was classified into three categories (man, woman, and gender diverse) using self-identified gender identity data when available,²⁵ ICD-10 codes for gender-affirming care (Supplementary File 1, available as Supplementary material is available at [JAMIA Open] online),²⁶ and birth sex. Urban and rural definitions were derived from US Census Bureau criteria.²⁷ Drive time was sourced from the VHA Planning Systems Support Group Geocoded Enrollee Files (PSSG) and were calculated using expected driving routes, traffic, and average driving conditions for each Veteran’s residential address to the closest VHA primary care facility.²⁸ While rurality and drivetime are related, they are poorly correlated (r = 0.48) as long drive times are found in both dense urban areas and very rural areas (Supplementary File 1, available as Supplementary material is available at [JAMIA Open] online). We also adjusted for VHA medical center as there is variation in the provision of telehealth across facilities due to differences in provider and facility leadership preferences and acceptance of telehealth. We incorporated information from VHA’s priority-based enrollment system, which categorizes individuals into eight hierarchical and non-mutually exclusive groups based on their military honors, military service, disability rating, income, and other factors.²⁷ Veterans with military honors are placed in the highest group (Group 1) regardless of their disability or income status which reduces the sensitivity of the low-income and disability categories (Supplementary File 1, available as Supplementary material is available at [JAMIA Open] online).²⁷ We included the Charlson Comorbidity Index calculated without age as a measure of comorbid status.^29–31

The TEM measure reflects the proportion of care by modality but not the total number of visits (e.g., a TEM of 50% could be five telehealth visits out of ten total visits, or only two telehealth visits out of four total visits). To address this, we calculated each individual’s number of visits (in-person, telephone, and video), categorized as 3, 4, 5, and 6+ visits.

Statistical analysis

We used fractional regression models^32,33 to calculate the Veteran-level adjusted annual TEM, TEM-v, and TEM-t (e.g., marginal effects) by six variables previously identified to be important predictors of telehealth use: age, gender, race/ethnicity, rurality of individual’s residence, drive time to nearest VHA primary care facility, and Charlson Comorbidity Index.^10,34–37 Each model (TEM, TEM-v, and TEM-t) was mutually adjusted for these patient characteristics as well as marital status, VHA medical center, and VHA enrollment priority group. Each model was also adjusted for the total number of visits and as a sensitivity analysis we present the marginal effects predicted at each category of total visits. We used restricted cubic splines to flexibly model age and drive time. We used robust standard errors in all models and report 95% confidence intervals (95% CI). All analyses were completed in Stata 18 (StataCorp, LLC) and were designated as non-research quality improvement by the U.S Department of Veterans Affairs (VA) Office of Connected Care.

Results

Among 716,373 individuals, the average age was 66 years old and the majority of the population was White (61.1%), identified as cisgender-men (89.2%), urban dwelling (68.1%), and on average lived within 20 min of a VHA primary care facility (Table 1). Approximately 16.2% of individuals had at least one primary care video visit and 35.9% had a telephone visit. Individuals had an average of 3.65 primary care visits during the year of follow-up, an average of 0.25 video visits in the year, and an average of 0.59 phone visits in the year. The mean unadjusted TEM was 21.7% (SD: 27.7, median: 0), the mean unadjusted TEM-t was 15.1% (SD: 23.7, median: 0), and the unadjusted mean TEM-v was 6.6% (SD: 17.1, median: 0), reflecting a right skewed distribution common with health service utilization data. Unadjusted means and standard deviations by demographics are presented in Table 1 and in Supplementary File 2, available as Supplementary material is available at [JAMIA Open] online.

Table 1.

Demographic and Clinical Characteristics by Telehealth Engagement Measure (TEM) among 716,373 Veterans with three or more primary care visits between May 1, 2022 and April 31, 2023 at the Veterans Health Administration.

	Total		TEM %		TEM-t %		TEM-v %
	N	Col %	Mean	SD	Mean	SD	Mean	SD
Total	716,373	100%	21.7	27.7	15.1	23.7	6.6	17.1
Age
18-44	66,790	9.3%	29.6	30.2	16.3	23.9	13.3	23.5
45-64	220,067	30.7%	23.5	27.9	14.7	22.8	8.8	19.4
65-74	238,831	33.3%	19.9	26.6	15.0	23.5	4.9	14.6
75-84	149,178	20.8%	18.7	26.5	14.7	23.8	4.0	13.3
85+	41,507	5.8%	20.7	29.9	16.9	27.5	3.9	13.5
Race/Ethnicity
American Indian or Alaska Native	4637	0.6%	23.0	27.8	17.3	24.3	5.8	16.1
Asian	7758	1.1%	28.7	30.3	18.5	26.0	10.1	20.5
Black or African American	155,906	21.8%	21.6	27.7	14.1	23.0	7.5	18.2
Hispanic or Latino	56,739	7.9%	24.7	28.5	17.0	24.4	7.7	18.4
More than one race	6128	0.9%	24.5	29.1	16.9	25.0	7.6	18.3
Native Hawaiian or Other Pacific Islander	6286	0.9%	26.6	29.7	18.7	26.0	7.8	18.7
Unknown	41,035	5.7%	24.8	29.2	17.0	24.9	7.8	18.6
White	437,884	61.1%	20.8	27.3	14.8	23.5	6.0	16.3
Gender
Gender Diverse	2096	0.3%	27.7	30.1	15.3	23.3	12.4	22.8
Men	638,914	89.2%	21.0	27.4	15.0	23.7	6.0	16.4
Women	75,363	10.5%	27.5	29.4	15.9	23.6	11.6	21.9
Rural/Urban Status
Urban	487,810	68.1%	22.5	28.0	15.1	23.6	7.4	18
Rural	204,794	28.6%	20.1	27.2	15.1	23.8	5.1	15.3
Highly Rural/Insular	23,769	3.3%	19.7	26.6	15.5	24.0	4.1	13.7
Marital Status
Married	387,768	54.1%	21.3	27.4	14.7	23.4	6.5	17.0
Separated/Divorced	34,750	4.9%	21.8	27.6	15.3	23.7	6.5	16.9
Single/Never Married	200,323	28.0%	24.1	28.9	15.9	24.3	8.1	19.0
Widowed	86,165	12.0%	19.2	27.5	14.7	24.5	4.4	14.2
Unknown/Missing	7367	1.0%	26.2	29.8	16.6	24.6	9.5	20.5
Drive Time to VHA Primary Care
<5 min	34,759	4.9%	20.4	27.1	14.5	23.4	5.9	16.2
≥5–10 min	154,907	21.6%	21.4	27.6	14.7	23.5	6.6	17.1
≥11–16 min	179,760	25.1%	22.0	27.8	14.8	23.5	7.2	17.7
≥17–24 min	159,542	22.3%	22.2	27.8	15.1	23.6	7.0	17.6
≥25–47 min	152,676	21.3%	21.5	27.7	15.4	23.9	6.0	16.5
≥48 min	34,729	4.8%	21.8	28.0	16.7	24.9	5.1	15.3
Charlson Comorbidity Index
0	174,061	24.3%	24.2	28.6	15.0	23.3	9.2	20.1
1	158,826	22.2%	21.8	27.5	14.9	23.2	6.9	17.5
2	132,551	18.5%	20.6	27.0	14.6	23.1	6.0	16.3
3	86,133	12.0%	20.1	26.9	14.9	23.6	5.2	15.0
4	99,720	13.9%	19.9	27.1	15.1	24.1	4.8	14.4
>5	65,082	9.1%	21.8	28.7	16.7	25.8	5.1	14.7
Total Number of Visits per Year
3 visits	452,274	63.1%	19.2	26.9	13.0	22.4	6.2	17.1
4 visits	160,956	22.5%	23.2	27.3	16.3	23.5	6.9	16.9
5 visits	58,598	8.2%	26.8	28.1	19.5	24.8	7.4	17.0
6+ visits	44,545	6.2%	34.6	31.3	30.0	29.3	8.7	18.6

Abbreviations: TEM: Telehealth Engagement Metric; TEM-t: Telehealth Engagement Metric- Telephone; TEM-v: Telehealth Engagement Metric- Video.

In adjusted models stratified by sociodemographic characteristics of interest, individuals consistently had a higher percentage of care delivered via telephone than video (Figure 1, Supplementary File 3, available as Supplementary material is available at [JAMIA Open] online). Notably, with higher age, the predicted mean TEM decreased up until age 80 where the mean TEM was 15.1% (95% CI 14.7, 15.5) (Figure 1). Above the age 80, the predicted mean TEM increased, reaching 17.1% (95% CI 16.5, 17.6) among individuals aged 95 years. This pattern was driven by a lower amount of video care, which had its highest values among those aged 20 (11.2% (95% CI 10.7, 11.8)) and continued to decline significantly after age 50 (TEM-v at age 55, 6.5% (95% CI 6.2, 6.8)). In comparison, the mean amount of care delivered via phone decreased marginally from ages 20 (TEM-t 16.7% (95% CI 16.1, 17.3)) and increased after age 80 (12.6% (95% CI 12.2, 12.9)), peaking at among those aged 95 (14.4% (95% CI 13.9, 14.9)). Among individuals with a Charlson Comorbidity Index (CCI) of one, the predicted annual TEM was 19.3% (95% CI 19.2, 19.5) (Figure 1/Supplementary File 3, available as Supplementary material is available at [JAMIA Open] online). The TEM increased slightly with higher CCI. Among individuals with a CCI of eight, the TEM was 21.4% (95% CI 20.9, 22.0) and the largest predicted TEM was 22.2% (95% CI 20.2, 24.2) among individuals with a CCI of 14.

Figure 1.

Predicted Annual Telehealth Engagement Metrics (TEM) in Primary Care for a Veteran by Age, Drive Time, and Charlson Comorbidity Index among 716,373 Veterans.

Each model (TEM, TEM-v, and TEM-t) was mutually adjusted for these patient characteristics as well as race, gender, rurality of home, marital status, VHA medical center, and VHA enrollment priority group.

Abbreviations: TEM: Telehealth Engagement Metric; TEM-t: Telehealth Engagement Metric- Telephone; TEM-v: Telehealth Engagement Metric- Video

A higher drive time from primary care VHA facility was associated with an increase of all TEM measures (Figure 1). Yet, individuals who lived in urban settings had the highest mean TEM at 19.6% (95% CI 19.5, 19.7), while rural dwelling individuals had the highest mean TEM-t at 13.3% (95% CI 13.2, 13.5%) (Figure 2, Supplementary File 4, available as Supplementary material is available at [JAMIA Open] online). Those who lived in highly rural/insular areas or rural areas had a lower TEM-v (5.5% (95% CI 5.2, 5.7) and 5.6% (95% CI 5.5, 5.7%) respectively), compared to those who lived in urban areas (6.4%, 95% CI 6.3, 6.4%).

Figure 2.

Predicted Annual Telehealth Engagement Metrics (TEM) in Primary Care for a Veteran by race/ethnicity, gender, rurality of home among 716,373 Veterans.

Each model (TEM, TEM-v, and TEM-t) was mutually adjusted for these patient characteristics as well as race, gender, rurality of home, marital status, VHA medical center, and VHA enrollment priority group.

Abbreviations: AI/AN: American Indian/Alaska Native; AA: African American; NH/PI: Native Hawaiian/Pacific Islander; TEM: Telehealth Engagement Metric; TEM-t: Telehealth Engagement Metric- Telephone; TEM-v: Telehealth Engagement Metric- Video

There were minor differences in adjusted mean TEM by race and ethnicity (Figure 2, Supplementary File 4, available as Supplementary material is available at [JAMIA Open] online). Individuals who were multiracial or White had the highest mean TEM values (19.6% (95% CI 19.0, 20.2) and 19.7% (95% CI (19.6, 19.8)) while those who were Asian had the lowest (18.5% (95% CI 18.0, 19.0). Men had the lowest mean TEM and TEM-v compared to female and gender diverse individuals.

In our sensitivity analyses, we calculated the predicted TEM, TEM-t, and TEM-v at varying numbers of health care visits in the year (Supplementary Files 5–8, available as Supplementary material is available at [JAMIA Open] online). The relationship between TEM, TEM-t, and TEM-v and the patient characteristics evaluated stayed the same, but predicted TEM values were always highest among Veterans with six or more visits in a year and lowest among those with 3 visits a year across all characteristics assessed.

Discussion

The Telehealth Engagement Measure (TEM) captures a person’s distribution of care across in-person, video, and telephone modalities. In this study, we demonstrate how the TEM can be applied to a large primary care population with high healthcare utilization and describe its variations across social and contextual demographics. We found a negative association between TEM and age (i.e., older individuals have lower proportions of telehealth use compared to younger individuals), a positive association with drive-time (i.e., more distant Veterans had higher amounts of telehealth than those living closer), and a higher TEM-t and similar TEM-v among individuals with greater comorbidity (i.e., patients with high comorbid burden use more telephone-based care, but similar amounts video-based care). As health care systems expand their use of telehealth, nuanced measures like TEM, when used in analyses to identify patterns of telehealth use that may be associated with differences in health quality and outcome data, can help to evaluate the quality, safety, and effectiveness of these modalities and to examine their longitudinal impact.

Comparison to other telehealth studies

Our results, which reflect patterns among patients regularly engaged in outpatient care, show that the TEM measure indicates higher proportion of telephone use than video, similar to prior studies of telehealth.^6,9,11,12,14 In line with prior studies, we found older or rural individuals engaged in less video care than younger or urban individuals. Females had a higher mean TEM than males; reflecting the general trend of higher utilization of video and telephone use among females.¹⁶ Higher drive times to a VHA site of primary care were associated with an higher TEM, TEM-t, and TEM-v. While studies evaluating drive time and video care use are limited, one study found longer drive times among individuals with predominantly telephone visits.¹¹ In another evaluation, Tisdale et al. found higher drive time was associated with a lower odds of video care use in cardiology services.³⁴ These differences may stem from a higher prevalence of individuals using VHA-paid community care for cardiology, while only 7.5% receive primary care from VHA-paid community care.³⁸ This pattern may also reflect the appropriateness of telehealth for cardiology care may require physical examinations, diagnostic procedures, or laboratory assessments that are less amenable to remote delivery.²³ Therefore, longer drive times may encourage more telehealth use for VHA primary care but may lead to increased reliance on non-VHA providers for specialty care.

We found a higher mean TEM among urban-dwelling individuals compared rural-dwelling individuals. This a common finding as the COVID-19 pandemic increased the availability of telehealth, particularly among urban individuals who are less impacted by the digital divide.^8,10,15 While VHA primary care is available in rural areas, rurality may reflect broadband accessibility rather than proximity to services–which drive time captures more accurately.

We also noted a slight increase in TEM with higher Charlson comorbidity index (CCI) from 19.7% among individuals with a CCI of zero to 22.0% among individuals with a CCI of 15+. This contrasts with prior studies reporting considerably stronger positive associations of video and phone utilization and higher comorbidity status.^15,39 This difference is likely due to the confounding effect of sicker patients having increased numbers of health care visits¹⁶ rather than the modality of the visit itself, which we accounted by adjusting for number of encounters.

Clinical applications of the telehealth engagement measure

The Telehealth Engagement Measure (TEM) has versatile applications in clinical settings. One advantage of the TEM is the ability to apply it across various levels of clinical care, including patient, clinic, and population-levels. At the patient-level, if health care providers were provided a patient’s regularly updated TEM, they would have a sense of how much of an individual’s care has been provided through telehealth versus in-person. If future studies find that a higher TEM in older populations is associated with poorer clinical outcomes, a provider or health care system could identify older individuals with high TEM who may benefit from in-person care. This patient-centered use of the measure would allow providers and health systems to tailor the use of telehealth to meet patients’ needs without sacrificing quality of care. Many factors including age, socioeconomic status, broadband availability, distance from clinic, perception of care quality, provider characteristics impact an individual’s ability, choice, and access to use telehealth.^{10,34–37,40,41} Yet, the ability to quantify and describe a patient’s past telehealth use could help health care systems identify individuals who may need extra resources or assistance to engage virtually. For example, during the pandemic, VHA facilitated access to federal connectivity programs and provided video-enabled tablets to select individuals who had known barriers to accessing telehealth.^42,43 Knowledge about a patient’s percentage of care delivered via telehealth (i.e., their TEM) could be useful in identifying individuals who might benefit from such programs, or in highlighting individuals enrolled in these programs who are potentially under-utilizing resources and might benefit from additional support (e.g., digital education).

Clinics could also use the TEM measures to help optimize clinic efficiency and flow. For example, clinics could audit the TEM at both the clinic and provider-levels and use this information to adjust the volume of telehealth and in-person visits. Variations at the provider-level could also offer insights into which providers may need more training or resources in offering telehealth. Similarly, health care systems could track telehealth engagement at a population-level (e.g., city, region, hospital system, etc.) and make similar adjustments in how care is provided. Assessing the TEM at the diagnosis group level (i.e, ICD-10) could also offer insight into how telehealth modalities are being used for diagnosis and treatment of different conditions.^23,44 Finally, the TEM can be easily presented as a percentage or as a ratio (33% = 1:2, 50%= 1:1), the latter of which gives the user a relativistic interpretation of the amount of telehealth used compared to in-person care, which may be more interpretable and applicable for front line providers.

Research applications of a telehealth engagement measure (TEM)

The TEM has several characteristics that make it valuable as a research tool. First, the TEM can apply to different time periods including cumulatively over a patient’s time within a health care system which allows investigators to assess the cumulative impact of telehealth, rather than the effect of a single visit. TEM can be calculated as a measure of total telehealth use, or for subcategories that encapsulate video-based telehealth (TEM-v) or telephone-based telehealth (TEM-t), which allows investigators to more specifically explore how different modalities of synchronous telehealth are impacting their outcomes of choice.

In addition to supporting descriptive assessments of telehealth engagement across key sociodemographic characteristics, as we have illustrated here, TEM can be used to explore more complex interactions. For instance, the TEM could be used to investigate how the amount of telehealth an individual received affects health outcomes in certain clinical conditions. One example of this would be to use TEM to test whether certain primary conditions such as mental health conditions and diabetes could be safely managed through telehealth, while others that require more hands-on care (e.g., heart failure, low back pain, etc.) might require more in-person care. Researchers could also use the TEM to examine patient and provider-centric outcomes such as the association of patient satisfaction or provider burnout rates with higher or lower TEM rates. Finally, the TEM could facilitate much-needed economic evaluations⁴⁵ such as budget impact analyses and cost-effectiveness analyses of telehealth. While larger, integrated health care systems like the VHA may be especially invested in research about the impact of telehealth,^19,46 the overall simplicity and ease of this utilization measure could facilitate its use in small health care settings as well.

Limitations of the telehealth engagement measure (TEM)

The TEM has several limitations. First, like any percentage measure, TEM is sensitive to the number of visits in the denominator. For a more continuous distribution, a large range of total visits is required. When there are a small number of visits, the possible distribution of these visits by modality (in-person, video, or telephone) is limited. For example, among patients with two visits, the TEM will take on one of three values: 0%, 50%, and 100%. This limited variation among patients with few visits may will limit analyses for which variation is needed to understand differences in in how care is provided. While there is no established standard threshold for the minimum number of visits needed to utilize the TEM measure, prior studies using a similar percentage measure for evaluating care fragmentation use a threshold of four.^47,48 The ideal population for evaluating the right mix of care using the TEM are patients who have enough care to have a mix of care modalities. Therefore, using a Telehealth Engagement Measure among individuals with few interactions with a health care system may not be a suitable application. For these populations with fewer visits, a binary metric of never/ever or categorical (only telehealth, only in-person, both telehealth and in-person) may be more appropriate.

In this evaluation, we restricted to a population of individuals who had at least three but no more than 20 primary care visits in a year. Therefore, our results reflect patterns among regularly engaged patients, who may have greater healthcare needs, more complex conditions, and may be key targets for improved care coordination. These results may not be generalizable to patient populations with less frequent care. In other applications of the TEM, the threshold for inclusion can be adjusted based on the population of interest. The fact that TEM has a wider distribution when patients have a minimum of three visits also influences the time period for which it should be calculated. In the work presented here, we calculated the TEM using 12 months of data. While smaller time periods can be used, smaller time windows may result in fewer visits, thus leading to instability in the measure. This may also impact the types of care a TEM measure can be applied to, as there may not be enough variability when visits are infrequent. Thus, the TEM may be more useful if it is calculated over longer periods of time (e.g., cumulative exposure) or used among populations with high care utilization. Moreover, because health care consumers often change where they receive their health care, calculating and using a TEM in situations where patients receive care from multiple settings or with high ‘churn’ (i.e., high patient turnover), may not be advisable. In our analysis, we focused solely on outpatient primary care within the VHA system. Our results do not capture telehealth use outside VHA for Veterans who are dually enrolled in Medicare, Medicaid, private insurance, or Veterans who receive care through VHA-paid community providers.

Conclusion

In this evaluation, we describe an application of the TEM among a cohort of over 700,000 individuals with high primary care utilization over a 12-month time period. We show how the percentage of telehealth, video care, and telephone care varies by six variables previously identified to be important predictors of telehealth use: age, gender, race/ethnicity, rurality, drive time, and patient comorbidities. While the Telehealth Engagement Measure (TEM) shows variation among these important predictors of telehealth use, we have not assessed if this variation is clinically significant. In future studies, the TEM can be evaluated with health outcome data to help clinicians, health care leaders, and researchers better understand the impact of telehealth on health, optimize the use of telehealth in the clinical setting, and help answer the question of ‘What is the right mix of telehealth and in-person care?’’.

As researchers begin to explore the effectiveness of telehealth compared to in-person care, the TEM may offer value as a possible exposure measure. Clinical and policy leaders could apply the TEM at a variety of levels to quantify “mix of care”. Given the need for rigorous evaluations of the quality, safety, and value of telehealth across diverse clinical scenarios, the Telehealth Engagement Measure could help health care systems further evaluate and optimize their use of telehealth technologies moving forward.

Author contributions

Jacqueline M Ferguson(Conceptualization, Data curation, Formal analysis, Funding acquisition, Methodology, Visualization, Writing—original draft, Writing—review & editing), Donna Zulman(Conceptualization, Funding acquisition, Writing—review & editing), Ashok Reddy(Conceptualization, Writing—original draft, Writing—review & editing), and James Van Campen(Conceptualization, Data curation, Writing—review & editing), Charlie Wray(Conceptualization, Methodology, Supervision, Writing—original draft, Writing—review & editing)

Ethical approval

This evaluation was designated as non-research quality improvement by the U.S Department of Veterans Affairs (VA) Office of Connected Care.

Funding

This study was supported by the Veterans Affairs Office of Connected Care. Dr. Ferguson received funding from the Office of Connected Care as Virtual Care Associate Investigator [COR-20-199] and from a Locally Initiated Project (LIP) Award from US Department of Veterans Affairs (VA) Palo Alto (LIP-22-JF-1). Dr. Wray is supported by a VA Health Services Research and Development Career Development Award [IK2HX003139-01A2].

Conflict of interest

Views expressed are those of the authors and the contents do not represent the views of the U.S. Department of Veterans Affairs (VA) or the United States Government. The authors have no conflicts of interest or disclosures to report.

Data sharing statement

Due to U.S. Department of Veterans Affairs (VA) regulations and our ethics agreements, the analytic data sets used for this study are not permitted to leave the VA firewall without a Data Use Agreement. This limitation is consistent with other studies based on VA data. However, VA data are made freely available to researchers with an approved VA study protocol. For more information, please visit https://www.virec.research.va.gov or contact the VA Information Resource Center at VIReC@va.gov