Key Points
Question
Do the outcomes of outpatient mental health (MH) care differ when delivered by video, phone, or in-person?
Findings
In this comparative effectiveness study including 813 699 participants from the Department of Veterans Affairs health system, receiving MH care via video was associated with lower expected rates of MH hospitalizations, MH emergency department visits, and suicide behaviors and higher rates of completed appointments compared with receiving care via phone or in-person.
Meaning
These findings suggest that receiving MH care via video was associated with certain clinical advantages compared with receiving care via phone or in-person; however, the study was unable to fully account for the video group being healthier and better resourced, and the magnitudes of the effect estimates were small, so results must be interpreted with caution.
This comparative effectiveness study assesses whether mental health outcomes differ among patients who receive care delivered via video, phone, and in-person.
Abstract
Importance
The use of tele–mental health (MH) care is widespread, with approximately half of all MH visits occurring remotely within the US Department of Veterans Affairs health system. However, little is known regarding the relative quality of video, phone, and in-person MH care.
Objective
To study the comparative effectiveness of MH care delivered via video, phone, and in-person.
Design, Setting, and Participants
This retrospective comparative effectiveness study used administrative data for all patients who completed at least 3 outpatient MH appointments during the assignment period from July 2021 to October 2022. Each participant was assigned to a MH modality cohort based on how they received most of their outpatient care in the assignment period: by video, phone, or in-person. Outcomes were assessed over a 1-year follow-up period from 2022 to 2023. Data were analyzed from July 2024 to July 2026.
Exposures
Receiving most outpatient MH care via video, phone, or in-person.
Main Outcomes and Measures
Outcomes of interest were MH hospitalizations, MH emergency department (ED) visits, suicide behaviors, and percentage of completed appointments. Inverse probability–weighted regression adjustment was used to obtain an average treatment effect (ATE).
Results
The cohort included 813 699 participants (672 833 [82.7%] male; 354 686 participants [43.6%] aged ≥60 years), including 305 189 participants (37.5%) who received most of their MH care in person, 343 543 participants (42.2%) who received most of their care via video appointments, and 164 967 participants (20.3%) who received most of their care via phone appointments. Overall, 3027 video group participants (0.9%), 6547 in-person group participants (2.1%), and 2584 phone group participants (1.6%) experienced an MH hospitalization; 5237 video group participants (1.5%), 8009 in-person group participants (2.6%), and 3713 phone group participants (2.3%) had an MH ED visit; and 3539 video group participants (1.0%), 3780 in-person group participants (1.2%), and 2086 phone group participants (1.3%) exhibited suicidal behaviors. Video group participants completed a mean (SD) of 71.1% (21.7%) of appointments, compared with 68.0% (22.2%) of appointments in the in-person group and 68.4% (23.2%) of appointments in the phone group. The expected probability of MH hospitalization was 0.005 (SE, 0.001) points lower if all patients had received mostly video care instead of phone and 0.005 (SE, <0.001) lower vs in-person care . The same pattern emerged for MH ED visits and suicide behaviors, with expected probabilities being lower for the video group compared to phone (MH ED visit: ATE, −0.006; SE, 0.001; suicidal behavior: ATE, −0.003; SE, <0.001) or in-person (MH ED visit: ATE, −0.005; SE, <0.001; P < .001; suicidal behavior: ATE, −0.001; SE, <0.001) groups. By contrast, the expected percentage of appointments completed was 4.1 (SE, 0.1) percentage points higher in the video group vs phone group and 3.6 (SE, 0.1) percentage points higher in the video group vs in-person group.
Conclusions and Relevance
In this comparative effectiveness study, receiving MH care via video was associated with improved clinical outcomes compared with receiving care via phone or in-person. Findings of possible advantages of video- over phone-based care could impact care modality decision-making if video is a feasible option. Video-based care also was associated with improved outcomes compared with in-person care. However, despite controlling for imbalanced groups, there is still potential confounding, and the magnitudes of the ATEs were small; therefore, results must be interpreted with caution.
Introduction
The use of tele–mental health (MH) care via synchronous video visits and audio-only phone is widespread. Approximately half of outpatient MH care within the US Department of Veterans Affairs (VA) occurs remotely,1 with similar rates in community settings.2 While tele-MH care utilization skyrocketed during COVID-19 to help prevent infection,3,4 its popularity has persisted in the postpandemic period, with patients and clinicians endorsing high satisfaction rates.5 Given the ubiquity of video and phone tele-MH care, it is important to understand the relative clinical effectiveness of these modalities, both compared with each other as well as with in-person care.
A strong body of literature comparing the quality of video and in-person MH care within and outside of the VA existed well before the pandemic. Rigorous randomized clinical trials (RCTs) and noninferiority trials have demonstrated video’s equivalence to in-person care in reducing symptoms of depression and posttraumatic stress disorder.6,7,8,9 Studies conducted during the pandemic have also demonstrated increased appointment completion for video visits compared with in-person visits.10,11 The convenience of video visits is thought to have contributed to this effect, given that video care removes many barriers to attendance, including transportation, distance, traffic, parking, weather, and needing to arrange for childcare, eldercare, or time away from work or school.
There is relatively little literature comparing the quality of video and phone care. However, studies have noted that the loss of nonverbal information during phone visits may have significant impacts on quality of care by limiting clinicians’ ability to assess patient functioning as well as by potentially weakening rapport and therapeutic alliance.5,12 Although a meta-analysis found both video and phone to be effective when treating veterans with depression and trauma, effect sizes were stronger for video.13 In addition, smoking cessation studies have demonstrated more favorable outcomes for video- vs phone-based treatment, including greater medication adherence and higher rates of treatment completion and continued abstinence.14,15,16 Furthermore, a study by Meshberg-Cohen et al17 found that veterans who primarily attended phone appointments during COVID-19 were more likely to present to the psychiatric emergency department (ED) vs those who mainly received video care. However, the study by Meshberg-Cohen et al17 used a relatively small sample size, incorporated few covariates, and was specific to substance use disorders.
To our knowledge, there has been no large-scale investigation of the comparative effectiveness of video, phone, and in-person MH care across a variety of diagnoses and key clinical outcomes within naturalistic settings. This study sought to fill this gap by conducting a national analysis of veterans who received most of their MH care by video, phone, or in-person. We examined differences among these groups in rates of MH hospitalization, ED visits, suicide behaviors, and appointment completion. We hypothesized that veterans receiving mostly video care would have lower rates of MH hospitalizations, ED visits, and suicide behaviors and higher rates of appointment completion compared with those receiving mostly phone care and that there would be no significant differences in outcomes between veterans receiving mostly video vs in-person care, with the exception of completed appointments, such that video would have higher attendance rates.
Methods
This comparative effectiveness study was determined by the VA Boston institutional review board to be exempt from review and informed consent because the study involved retrospective analysis of existing electronic medical record data, presented minimal risk to patient privacy, and obtaining individual consent from a large retrospective cohort would not be feasible. This study is reported following the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) reporting guidelines for comparative effectiveness research.
This retrospective comparative effectiveness study of VA administrative data included all patients who completed at least 3 VA outpatient MH appointments between July 2021 and October 2022 (the assignment period). We limited the sample to those with at least 3 visits to best capture individuals in active, ongoing MH treatment. We excluded patients from 8 VA facilities who had incomplete or nonrepresentative data (eMethods in Supplement 1). We assigned participants to 1 of 3 modality cohorts based on how they received more than 50% of their outpatient MH care in the assignment period: video, phone, or in-person. To assess the association of one’s dominant care modality with study outcomes, we excluded patients who had an exact 50/50 split between 2 modalities or no majority modality (eg, 33% each of video, phone, in-person). We then estimated differences in outcomes between modalities during a 1-year follow-up period (2022-2023). The primary outcome was probability of at least 1 MH hospitalization during the follow-up year; secondary outcomes were probability of at least 1 MH ED visit, probability of any suicide behaviors (eg, suicide attempt, overdose), and proportion of MH appointments completed (eMethods in Supplement 1).
To account for nonrandom assignment of participants to modalities, we used inverse probability–weighted regression adjustment (IPWRA) to obtain an average treatment effect (ATE). We estimated separate models for the primary outcome and each secondary outcome. IPWRA is doubly robust; only the treatment model or the outcome model needs to be correctly specified for estimates to be valid for causal inference.18
A common treatment assignment process was used for all 4 outcomes. The treatment assignment model was specified as a logistic model and included the following patient-level variables: age; sex; rurality (assessed via the Rural-Urban Commuting Area Codes19); service-connectedness disability level20; homelessness history; socioeconomic status (assessed via the Area Deprivation Index21); driving minutes to closest primary care site22; adequate broadband access; Elixhauser medical comorbidities23; binary indicators for each of anxiety disorder, bipolar disorder, depressive disorder, posttraumatic stress disorder, schizophrenia, and substance use disorder diagnoses; binary indicators for prior-year history of suicide behaviors, MH hospitalization, MH ED visit, and most appointments being medication management (vs psychotherapy); and number of MH appointments completed in the prior year. All measures of health care use were assessed in the year prior to the index year used for cohort assignment. The treatment assignment model also included several facility-level covariates: parent facility complexity group (low or medium vs high)24; counts of MH clinician full-time equivalents (FTEs), clinician support FTEs, and unique patients receiving MH care; percentage of patients living in rural and highly rural areas; and percentage of visits using video or phone telehealth (eMethods in Supplement 1). The effectiveness of the treatment assignment model was assessed informally by comparing balance statistics before and after weighting and by examining propensity score overlap plots.
Statistical Analysis
Binary outcome models were specified as logistic models, and the proportion of completed visits model used a fractional logistic model. Covariates included in the individual outcome models varied by outcome, but all started with a common set of patient and facility characteristics. We specified initial models for each outcome based on hypothesized associations between study variables and outcomes. Models were refined with a 50% sample of the data to remove covariates that were not significant for any of the modalities. We followed prespecified steps that investigated alternative ways of including variables in the model for parsimony and improvement in model fit when covariates that were hypothesized to be significant were not. For example, if specific MH diagnoses were not significant, we examined whether using an overall count of all MH diagnoses improved model fit using the bayesian information criterion and Akaike information criterion. When the Akaike information criterion and bayesian information criterion conflicted, we computed out-of-sample estimates and selected the model with the better performance. All analyses were conducted with Stata version 18.0 (StataCorp). The threshold for statistical significance was 2-sided P < .05. Data were analyzed from July 2024 to July 2026.
While the IPWRA method is doubly robust, the model still fails to correctly estimate the ATE if the treatment assignment and outcome models are both misspecified. Unmeasured confounding is a common cause of misspecification. Prior research has demonstrated that patients who receive most of their care by video tend to have less severe clinical presentations and to be of higher socioeconomic status.25 While we controlled for many variables to capture these differences, if any unmeasured confounding remained, findings of lower MH hospitalization rates in the video group may be due to this group being generally healthier and better resourced. Therefore, we planned a sensitivity analysis using negative control outcomes to examine whether MH modality was associated with likelihood of medical and surgical hospitalizations.26 With this approach, we assume that if there were unmeasured aspects of overall health or personal resources within the sample, then patients receiving video MH care would have lower rates of medical and surgical hospitalizations. If, however, the variables included in the IPWRA models adequately accounted for the video group being healthier and better resourced, then there should be no observed differences in rates of medical and surgical hospitalizations among the video, phone, and in-person groups, as we posited that the modality through which one receives their MH care would not impact their likelihood of a medical or surgical hospitalization.
Results
From an initial sample of 1 735 814, we excluded 562 855 patients for having fewer than 3 MH appointments in the assignment period. We then excluded 154 493 patients for having exactly 50% of 2 modalities or no majority of any modality. An additional 204 767 patients were excluded for having missing covariates or for being from 1 of the 8 excluded VA facilities. This led to a final sample of 813 699 participants (672 833 [82.7%] male; 354 686 participants [43.6%] aged ≥60 years), including 343 543 participants (42.2%) in the video group, 305 189 participants (37.5%) in the in-person group, and 164 967 participants (20.3%) in the phone group. Patient characteristics were significantly different across cohorts (Table 1). Compared with the in-person and phone groups, the video group was younger, had higher income, and lived in more urban areas, with a greater percentage of women and lower clinical severity (eg, lowest rates of prior MH hospitalization, schizophrenia and bipolar disorder diagnoses, and comorbid medical conditions) (Table 1). However, we achieved good balance for a diverse set of covariates in our treatment assignment model (eTable and eFigure in Supplement 1).
Table 1. Characteristics of Study Participants.
| Characteristic | Participants, No. (%) | P value | |||
|---|---|---|---|---|---|
| Total (N = 813 699) | Modality cohort | ||||
| In-person (n = 305 189 [37.5%]) | Video (n = 343 543 [42.2%]) | Phone (n = 164 967 [20.3%]) | |||
| Age, y | |||||
| 18-29 | 31 454 (3.9) | 7931 (2.6) | 18 565 (5.4) | 4958 (3.0) | <.001 |
| 30-39 | 135 962 (16.7) | 34 845 (11.4) | 78 410 (22.8) | 22 707 (13.8) | |
| 40-49 | 137 257 (16.9) | 38 455 (12.6) | 76 383 (22.2) | 22 419 (13.6) | |
| 50-59 | 154 340 (19.0) | 52 635 (17.2) | 74 165 (21.6) | 27 540 (16.7) | |
| 60-69 | 154 216 (19.0) | 67 398 (22.1) | 51 238 (14.9) | 35 580 (21.6) | |
| 70-79 | 175 323 (21.5) | 90 868 (29.8) | 40 183 (11.7) | 44 272 (26.8) | |
| ≥80 | 25 147 (3.1) | 13 057 (4.3) | 4599 (1.3) | 7491 (4.5) | |
| Sex | |||||
| Male | 672 833 (82.7) | 271 251 (88.9) | 260 629 (75.9) | 140 953 (85.4) | <.001 |
| Female | 140 866 (17.3) | 33 938 (11.1) | 82 914 (24.1) | 24 014 (14.6) | |
| Racea | |||||
| American Indian or Alaska Native | 10 136 (1.2) | 3750 (1.2) | 4259 (1.2) | 2127 (1.3) | <.001 |
| Asian | 12 132 (1.5) | 2707 (0.9) | 7243 (2.1) | 2182 (1.3) | |
| Black or African American | 184 801 (22.7) | 61 464 (20.1) | 89 613 (26.1) | 33 724 (20.4) | |
| Native Hawaiian or Other Pacific Islander | 9286 (1.1) | 2994 (1.0) | 4405 (1.3) | 1887 (1.1) | |
| White | 539 863 (66.3) | 215 653 (70.7) | 210 564 (61.3) | 113 646 (68.9) | |
| Unknown | 57 481 (7.1) | 18 621 (6.1) | 27 459 (8.0) | 11 401 (6.9) | |
| Ethnicitya | |||||
| Hispanic or Latino | 66 653 (8.2) | 20 419 (6.7) | 34 275 (10.0) | 11 959 (7.2) | <.001 |
| Not Hispanic or Latino | 695 719 (85.5) | 268 174 (87.9) | 283 153 (82.4) | 144 392 (87.5) | |
| Unknown | 51 327 (6.3) | 16 596 (5.4) | 26 115 (7.6) | 8616 (5.2) | |
| Marital status | |||||
| Never married | 152 741 (18.8) | 52 378 (17.2) | 70 363 (20.5) | 30 000 (18.2) | <.001 |
| Married | 410 151 (50.4) | 156 506 (51.3) | 174 395 (50.8) | 79 250 (48.0) | |
| Separated | 32 800 (4.0) | 12 004 (3.9) | 13 730 (4.0) | 7066 (4.3) | |
| Divorced | 185 137 (22.8) | 71 505 (23.4) | 72 223 (21.0) | 41 409 (25.1) | |
| Widowed | 18 145 (2.2) | 8673 (2.8) | 4615 (1.3) | 4857 (2.9) | |
| Unknown | 14 725 (1.8) | 4123 (1.4) | 8217 (2.4) | 2385 (1.4) | |
| History of housing instability | 63 449 (7.8) | 26 481 (8.7) | 22 186 (6.5) | 14 782 (9.0) | <.001 |
| Service connectedness | |||||
| <50% VA disability rating | 104 087 (12.8) | 42 006 (13.8) | 40 212 (11.7) | 21 869 (13.3) | <.001 |
| ≥50% VA disability rating | 582 312 (71.6) | 207 943 (68.1) | 264 045 (76.9) | 110 324 (66.9) | |
| Missing | 127 300 (15.6) | 55 240 (18.1) | 39 286 (11.4) | 32 774 (19.9) | |
| Rurality of residenceb | |||||
| Urban | 557 732 (68.5) | 197 244 (64.6) | 252 564 (73.5) | 107 924 (65.4) | <.001 |
| Rural | 248 844 (30.6) | 105 019 (34.4) | 88 634 (25.8) | 55 191 (33.5) | |
| Highly rural | 7123 (0.9) | 2926 (1.0) | 2345 (0.7) | 1852 (1.1) | |
| Socioeconomic statusc | |||||
| Least disadvantaged tercile | 195 644 (24.0) | 53 871 (17.7) | 103 166 (30.0) | 38 607 (23.4) | <.001 |
| Middle tercile | 334 793 (41.1) | 121 898 (39.9) | 146 313 (42.6) | 66 582 (40.4) | |
| Most disadvantaged tercile | 283 262 (34.8) | 129 420 (42.4) | 94 064 (27.4) | 59 778 (36.2) | |
| Drive time to closest primary care site, mean (SD), mind | 19 (15) | 19 (14) | 19 (14) | 20 (16) | <.001 |
| High speed internet accesse | 765 942 (94.1) | 283 835 (93.0) | 328 278 (95.6) | 153 829 (93.2) | <.001 |
| Prior MH service use | |||||
| >50% of Appointments with a nonmedical MH clinician | 346 625 (42.6) | 121 874 (39.9) | 179 261 (52.2) | 45 490 (27.6) | <.001 |
| >50% of Appointments with a medical MH clinician | 446 676 (54.9) | 175 079 (57.4) | 155 701 (45.3) | 115 896 (70.3) | <.001 |
| MH hospitalization in previous 2 y | 29 787 (3.7) | 17 306 (5.7) | 6624 (1.9) | 5857 (3.6) | <.001 |
| Medical or surgical hospitalization in previous 2 y | 35 280 (4.3) | 15 773 (5.2) | 10 608 (3.1) | 8899 (5.4) | <.001 |
| MH ED use in previous 2 y | 40 763 (5.0) | 20 284 (6.6) | 11 900 (3.5) | 8579 (5.2) | <.001 |
| Appointments completed, mean (SD), No. | 6.1 (9.5) | 5.5 (9.2) | 6.9 (11) | 5.2 (7.2) | <.001 |
| Suicide behaviors in previous 2 y | 23 457 (2.9) | 9246 (3.0) | 8720 (2.5) | 5491 (3.3) | <.001 |
| Measures of severity | |||||
| <3 MH conditions | 656 997 (80.7) | 244 774 (80.2) | 276 743 (80.6) | 135 480 (82.1) | <.001 |
| ≥3 MH conditions | 156 702 (19.3) | 60 415 (19.8) | 66 800 (19.4) | 29 487 (17.9) | |
| Prevalence of behavioral health conditions | |||||
| Anxiety disorder | 295 325 (36.3) | 104 500 (34.2) | 132 563 (38.6) | 58 262 (35.3) | <.001 |
| Bipolar disorder | 69 895 (8.6) | 29 318 (9.6) | 25 242 (7.3) | 15 335 (9.3) | <.001 |
| Depression | 385 043 (47.3) | 140 382 (46.0) | 169 766 (49.4) | 74 895 (45.4) | <.001 |
| PTSD | 425 912 (52.3) | 157 101 (51.5) | 188 277 (54.8) | 80 534 (48.8) | <.001 |
| Schizoaffective disorder | 35 172 (4.3) | 20 805 (6.8) | 6663 (1.9) | 7704 (4.7) | <.001 |
| Substance use disorder | 166 205 (20.4) | 69 241 (22.7) | 62 495 (18.2) | 34 469 (20.9) | <.001 |
| No. of medical comorbidities, mean (SD)f | 1.9 (1.8) | 2.1 (2) | 1.6 (1.6) | 1.9 (1.9) | <.001 |
Abbreviations: ED, emergency department; MH, mental health; PTSD, posttraumatic stress disorder.
Race and ethnicity were self-reported.
Derived using Rural-Urban Commuting Area definitions.19
Derived from the Area Deprivation Index, with highest socioeconomic tercile indicating 1-33; middle, 34-66; and lowest, 67-100.21
Obtained from the VA Planning Systems Support Group Geocoded Enrollee Files, which provide precise geographic locations.22
Number and percentage of patients living in a census block that had at least 1 broadband provider offering download speeds of ≥100 megabits per second and upload speeds of ≥20 megabits per second, the current Federal Communications Commission benchmark for adequate broadband internet access.
Calculated using the Elixhauser Comorbidity Index.23
Overall, 3027 participants (0.9%) in the video group, 6547 participants (2.1%) in the in-person group, and 2584 participants (1.6%) in the phone group experienced an MH hospitalization; 5237 participants (1.5%) in the video group, 8009 participants (2.6%) in the in-person group, and 3713 participants (2.3%) in the phone group had an MH ED visit; and 3539 participants (1.0%) in the video group, 3780 participants (1.2%) in the in-person group, and 2086 participants (1.3%) in the phone group had suicidal behaviors. Video group participants completed a mean (SD) of 71.1% (21.7%) of appointments, compared with 68.0% (22.2%) of appointments in the in-person group and 68.4% (23.2%) of appointments in the phone group. (Table 2). After accounting for nonrandom assignment of participants to different treatment modalities, we estimated an ATE of −0.005 (SE, 0.001) on the probability of MH hospitalization for the video vs phone group (P < .001) and an ATE of −0.005 (SE, <0.001) for the video vs in-person group (P < .001) (Table 3). This can be interpreted as meaning that the expected proportion of patients with an MH hospitalization is a small but statistically significant 0.005 points lower, equivalent to 5 fewer hospitalizations per 1000 patients if all patients had received mostly video care instead of either phone or in-person care. There was no difference in the expected population-averaged probability of MH hospitalization if patients receive care via phone vs in-person (ATE, <−0.001; SE, 0.001; P = .41). MH ED visits showed a similar pattern with no difference between phone and in-person care but small and significantly lower rates for video compared with phone (ATE, −0.006; SE, 0.001; P < .001) or in-person (ATE, −0.005; SE, <0.001; P < .001) care.
Table 2. Follow-Up Period Outcomes.
| Outcome | Participants, No. (%) | P value | |||
|---|---|---|---|---|---|
| Total (N = 813 699) | Treatment modality | ||||
| In-person (n = 305 189 [37.5%]) | Video (n = 343 543 [42.2%]) | Phone (n = 164 967 [20.3%]) | |||
| Any MH hospitalization | 12 158 (1.5) | 6547 (2.1) | 3027 (0.9) | 2584 (1.6) | <.001 |
| Any MH emergency department visits | 16 959 (2.1) | 8009 (2.6) | 5237 (1.5) | 3713 (2.3) | <.001 |
| Any suicide behaviors | 9405 (1.2) | 3780 (1.2) | 3539 (1.0) | 2086 (1.3) | <.001 |
| Appointments completed, mean (SD), % | 69.4 (22.3) | 68.0 (22.2) | 71.1 (21.7) | 68.4 (23.2) | <.001 |
Table 3. Differences in Expected Probability of MH Hospitalizations, Emergency Department Visits, and Suicide Behavior and Proportion of Completed MH Appointments During Follow-Up.
| Model | Difference in expected probability of each outcome | Difference in expected proportion of completed MH appointments | ||||||
|---|---|---|---|---|---|---|---|---|
| MH hospitalization | MH emergency department visit | Suicide behaviors | ||||||
| ATE (SE) | P value | ATE (SE) | P value | ATE (SE) | P value | Proportion (SE) | P value | |
| Difference in outcome | ||||||||
| Video vs in-person | −0.005 (<0.001) | <.001 | −0.005 (<0.001) | <.001 | −0.001 (<0.001) | <.001 | 0.036 (0.001) | <.001 |
| Video vs phone | −0.005 (0.001) | <.001 | −0.006 (0.001) | <.001 | −0.003 (<0.001) | <.001 | 0.041 (0.001) | <.001 |
| Phone vs in-person | −<0.001 (0.001) | .41 | 0.001 (0.001) | .08 | 0.001 (<0.001) | .002 | −0.005 (0.001) | <.001 |
| Expected for modality | ||||||||
| In-person | 0.017 (<0.001) | <.001 | 0.022 (<0.001) | <.001 | 0.012 (<0.001) | <.001 | 0.680 (0.001) | <.001 |
| Video | 0.011 (0.001) | <.001 | 0.017 (0.001) | <.001 | 0.009 (0.001) | <.001 | 0.716 (0.001) | <.001 |
| Phone | 0.016 (<0.001) | <.001 | 0.023 (<0.001) | <.001 | 0.013 (<0.001) | <.001 | 0.675 (0.001) | <.001 |
| No. | 813 699 | NA | 813 699 | NA | 813 699 | NA | 635 867 | NA |
Abbreviations: MH, mental health; NA, not applicable.
For suicide behaviors, expected probabilities were lower if the population received mostly video care compared with phone (ATE, −0.003; SE, <0.001) and in-person (ATE, −0.001; SE, <0.001) care. Probabilities were 0.001 (SE, <0.001) greater in the phone group compared with the in-person group. The percentage of appointments completed was 4.1 (SE, 0.1) percentage points greater for the video group vs phone and 3.6 (SE, 0.1) percentage points greater for the video group vs in-person. The in-person group had a small but significantly increased proportion of completed appointments compared with the phone group (ATE, 0.005; SE, 0.001).
Sensitivity analyses found that MH modality was also significantly associated with probability of medical or surgical hospitalization, with a similar pattern to what was found in our primary analyses, such that the video group had lower expected probabilities compared with the phone and in-person groups. The expected probability of medical or surgical hospitalization for the video group vs phone was 0.009 (SE, 0.001) points lower. Similarly, the expected probability of medical or surgical hospitalization was 0.006 (SE, 0.001) points lower in the video group vs in-person group, nearly equal to the difference for MH hospitalization. The expected probability of medical or surgical hospitalization was 0.003 (SE, 0.001) points higher in the phone group vs the in-person group.
Discussion
To our knowledge, this comparative effectiveness study represents the first national analysis of video, phone, and in-person MH care conducted in naturalistic conditions across a range of clinical disorders. Receiving mostly video care was associated with small but significantly lower rates of MH hospitalizations, ED visits, and suicide behaviors and higher rates of appointment completion compared with receiving mostly phone care. However, all results must be interpreted with caution due to evidence suggesting remaining unmeasured confounding. Findings align with prior research demonstrating increased clinical effectiveness of video vs phone in the treatment of depression, trauma, and substance use.13,14,15,16,17 The loss of nonverbal information during phone visits may inhibit MH clinicians’ ability to assess patient functioning and clinical severity, and both patients and clinicians have noted feelings of weakened therapeutic alliance when visits are conducted by phone.5,12 Our findings are the first to our knowledge to provide large-scale documentation of poorer outcomes associated with phone vs video MH care across multiple key clinical quality indicators within a naturalistic sample.
In addition, the video group had higher appointment completion rates than the in-person group. This finding aligns with prior work10,11 and suggests the possibility that the increased convenience of video visits may lead to measurable differences in appointment attendance. However, we were surprised to find that video care was associated with small but significant advantages vs in-person for MH hospitalizations, ED visits, and suicide behaviors. This finding differs from the results of prior rigorous RCTs that demonstrated comparable outcomes between video and in-person in reducing symptoms of depression and PTSD.6,7,8,9
Despite our efforts to address potential unmeasured confounding by controlling for a host of variables using doubly robust IPWRA methods, we still could not reject the hypothesis that our results could be explained by confounding factors. Receiving mostly video MH care was associated with lower rates of medical or surgical hospitalizations compared with receiving mostly phone or in-person care, although there was no hypothesized association between MH care modality and medical or surgical hospitalization rates. This finding suggests that elements of a patient’s overall clinical complexity and personal resources were not captured in our models, despite including multiple variables that intended to address this concern. Indeed, it is well established that patients receiving video care tend to be healthier and better resourced than those receiving phone and in-person care,25 which was the case in the current sample.
In addition, it is worth noting that 2 of our primary outcomes, MH hospitalizations and ED visits, are by nature in-person visits. Although we controlled for access-related variables, such as rurality and drive time to the nearest facility, it remains possible that patients who receive their outpatient care in-person would also be more likely to present in-person for higher levels of care due to ease of access. For example, a patient in acute distress during an in-person outpatient visit could be escorted directly to the ED. In contrast, patients who are treated remotely via video or phone may be less likely to engage with higher levels of in-person care at brick-and-mortar facilities due to greater access barriers. Thus, this element of convenience and proximity may in part explain the higher levels of hospitalizations and ED visits in the in-person group.
Our findings highlight a key challenge of comparative effectiveness analyses vs randomized designs. In naturalistic studies, groups are inherently imbalanced in ways that may be related to their outcomes; while statistical methods can help to reduce those imbalances, they may not be able to fully remove them. Therefore, while we found significantly improved clinical outcomes in the video group vs phone and in-person groups, we cannot be certain that these differences stem primarily from the modality through which care was delivered vs from other factors. Of note, we are also currently conducting secondary analyses to determine whether study outcomes may differ between key subgroups of interest; for example, those with higher vs lower clinical severity. Findings are forthcoming and will be submitted for publication on completion. In addition, the observed group differences were small. This was largely expected, as MH hospitalizations, ED visits, and suicide behaviors are low-incidence events, with 1.5%, 2.1%, and 1.2% of the total sample experiencing each of these outcomes in the follow-up period, respectively. Therefore, a change of 0.5 percentage points, while objectively small in magnitude, can be considered significant, given the low incidence rate of the outcome. Still, caution is needed when interpreting the clinical significance of findings since the primary outcomes of interest are rare events.
Strengths and Limitations
The current study has multiple notable strengths. It includes a national sample of more than 800 000 veterans who received care via video, phone, and in-person appointments between July 2021 and October 2022. This is an ideal period to study, as it represents a more stabilized period of the pandemic in which health care systems had both adapted to telehealth processes and had resumed provision of in-person care1; thus, there were large numbers of patients receiving video, phone, and in-person services, which allows for careful examination of between-group differences. This is also the first large-scale study to our knowledge to examine differences on multiple key clinical quality outcomes: MH hospitalizations, ED visits, suicide behaviors, and appointment attendance. Sophisticated statistical methods, including IPWRA, were used to attempt to address imbalances across groups, and the treatment assignment model performed well. Results may have important implications for clinical decision-making. Given our findings of potential benefits associated with video vs phone care, coupled with a growing body of literature demonstrating similar findings,13,14,15,16,17 MH clinicians could consider prioritizing video over phone if video is a feasible and preferred option for patients. It will also be helpful to provide patients with data, such as what was found in this study, to help them make an informed choice. However, given that observed group differences were small across all outcome variables, it may be important to retain the option of phone visits for those who do not have access to in-person or video care.12,27 This is particularly relevant when considering well-established findings of a digital divide, such that older, lower-income, and rural veterans are less likely to complete video visits due to lower levels of telehealth access and digital literacy.25,27,28 VA programs that provide internet-enabled tablets and telehealth support to veterans in need have helped to close this gap; initiatives like this will remain critical to ensure equitable access to high quality telehealth care.29
This study has some limitations. We established several exclusion criteria for the study sample to best address our research question: namely, is the dominant care modality of those in active MH treatment associated with clinical outcomes. Our final sample size was large, with more than 800 000 patients represented. However, it remains the case that veterans with fewer than 3 MH visits and no dominant care modality were not included in our analyses, which could impact generalizability of findings. The reasons for completing a given session by video, phone, or in-person are likely complex and multifaceted and may vary from session to session for each patient based on factors such as access, clinical severity, and both patient and clinician preference. Our decision to limit inclusion to patients with 1 dominant modality prevents examination of this nuance. Therefore, future analyses that treat both number of visits and care modality as continuous variables are warranted. In addition, the current sample included patients who received MH care via individual and group therapy, as well as patients receiving both psychotherapy and medication management. Telehealth group therapy may invite additional complications, including increased technological glitches and reports of worse therapeutic alliance,30 and modality preference and effectiveness may vary between psychotherapy and medication management visits due to differences in appointment length and the nature of the content being discussed.12 Thus, there is a need for future work that examines potential differences in outcomes based on both modality of care (telehealth vs in-person) as well as visit type (eg, individual psychotherapy, group therapy, medication management).
Conclusions
In this comparative effectiveness study, we found small but statistically significant differences in clinical outcomes associated with receiving mostly video MH care vs phone or in-person care, such that the expected proportions of MH hospitalizations, ED visits, and suicide behaviors were lower, and the probability of completing appointments was higher. Findings suggest that video care was associated with advantages over phone visits. While not directly assessed in this work, this could be due to improved assessment of patient functioning and development of a stronger therapeutic alliance during video visits. In addition, video users had higher appointment completion rates than those receiving in-person care, perhaps due to the increased convenience of receiving care remotely. However, findings must be interpreted with significant caution due to potential unmeasured confounding. Future work is needed to understand whether outcomes may differ based on specific clinical presentations (eg, serious mental illness) or appointment types (eg, individual psychotherapy, group therapy, medication management). Collectively, this body of research will help to ensure that patients receive high quality MH care via the modality that aligns best with both clinical need and individual preferences.
eMethods.
eTable. Balance statistics for cohorts after IPWRA model
eFigure. Overlap plots by modality
Data Sharing Statement
References
- 1.Ferguson JM, Wray CM, Van Campen J, Zulman DM. A new equilibrium for telemedicine: prevalence of in-person, video-based, and telephone-based care in the Veterans Health Administration, 2019-2023. Ann Intern Med. 2024;177(2):262-264. doi: 10.7326/M23-2644 [DOI] [PubMed] [Google Scholar]
- 2.Uscher-Pines L, Sousa JL, McCullough CM, Dong S, Kapinos KA. Telehealth visits in health centers serving low-income patients in California: final results from the Connected Care Accelerator Initiative (2022-2024). Rand Health Q. 2025;12(2):3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Connolly SL, Stolzmann KL, Heyworth L, Weaver KR, Bauer MS, Miller CJ. Rapid increase in telemental health within the Department of Veterans Affairs during the COVID-19 pandemic. Telemed J E Health. 2021;27(4):454-458. doi: 10.1089/tmj.2020.0233 [DOI] [PubMed] [Google Scholar]
- 4.Rosen CS, Morland LA, Glassman LH, et al. Virtual mental health care in the Veterans Health Administration’s immediate response to coronavirus disease-19. Am Psychol. 2021;76(1):26-38. doi: 10.1037/amp0000751 [DOI] [PubMed] [Google Scholar]
- 5.Connolly SL, Ferris SD, Azario RP, Miller CJ. Patient and provider attitudes toward video and phone telemental health care during the COVID-19 pandemic: a systematic review. Clin Psychol (New York). 2024;31(4):488-503. doi: 10.1037/cps0000226 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Morland LA, Mackintosh MA, Glassman LH, et al. Home-based delivery of variable length prolonged exposure therapy: a comparison of clinical efficacy between service modalities. Depress Anxiety. 2020;37(4):346-355. doi: 10.1002/da.22979 [DOI] [PubMed] [Google Scholar]
- 7.Acierno R, Gros DF, Ruggiero KJ, et al. Behavioral activation and therapeutic exposure for posttraumatic stress disorder: a noninferiority trial of treatment delivered in person versus home-based telehealth. Depress Anxiety. 2016;33(5):415-423. doi: 10.1002/da.22476 [DOI] [PubMed] [Google Scholar]
- 8.Egede LE, Acierno R, Knapp RG, et al. Psychotherapy for depression in older veterans via telemedicine: a randomised, open-label, non-inferiority trial. Lancet Psychiatry. 2015;2(8):693-701. doi: 10.1016/S2215-0366(15)00122-4 [DOI] [PubMed] [Google Scholar]
- 9.Morland LA, Mackintosh MA, Rosen CS, et al. Telemedicine versus in-person delivery of cognitive processing therapy for women with posttraumatic stress disorder: a randomized noninferiority trial. Depress Anxiety. 2015;32(11):811-820. doi: 10.1002/da.22397 [DOI] [PubMed] [Google Scholar]
- 10.Cummins MR, Tsalatsanis A, Chaphalkar C, et al. Telemedicine appointments are more likely to be completed than in-person healthcare appointments: a retrospective cohort study. JAMIA Open. 2024;7(3):ooae059. doi: 10.1093/jamiaopen/ooae059 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Greenup EP, Best D. Systematic review and meta-analysis of no show or non-attendance rates among telehealth and in-person models of care. BMC Health Serv Res. 2025;25(1):663. doi: 10.1186/s12913-025-12826-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Connolly SL, Adusumelli Y, Azario RP, Ferris SD, Hwang AR, Miller CJ. A qualitative evidence synthesis of patient and provider attitudes toward audio-only telemental health care. Telemed J E Health. 2025;31(1):3-17. doi: 10.1089/tmj.2024.0355 [DOI] [PubMed] [Google Scholar]
- 13.McClellan MJ, Osbaldiston R, Wu R, et al. The effectiveness of telepsychology with veterans: a meta-analysis of services delivered by videoconference and phone. Psychol Serv. 2022;19(2):294-304. doi: 10.1037/ser0000522 [DOI] [PubMed] [Google Scholar]
- 14.Liebmann EP, Preacher KJ, Richter KP, Cupertino AP, Catley D. Identifying pathways to quitting smoking via telemedicine-delivered care. Health Psychol. 2019;38(7):638-647. doi: 10.1037/hea0000740 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Richter KP, Shireman TI, Ellerbeck EF, et al. Comparative and cost effectiveness of telemedicine versus telephone counseling for smoking cessation. J Med Internet Res. 2015;17(5):e113. doi: 10.2196/jmir.3975 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Kim SS, Darwish S, Lee SA, Sprague C, DeMarco RF. A randomized controlled pilot trial of a smoking cessation intervention for US women living with HIV: telephone-based video call vs voice call. Int J Womens Health. 2018;10:545-555. doi: 10.2147/IJWH.S172669 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Meshberg-Cohen S, Farook M, Gross G, Levina V, DeViva J. Treatment utilization and modality preference among veterans receiving outpatient substance use disorder treatment during a pandemic. Am J Addict. 2023;32(1):32-39. doi: 10.1111/ajad.13347 [DOI] [PubMed] [Google Scholar]
- 18.Wooldridge JM. Econometric Analysis of Cross Section and Panel Data. MIT press; 2010. [Google Scholar]
- 19.Rural Health Research Center . Using RUCA data. Accessed April 3, 2026. http://depts.washington.edu/uwruca/ruca-uses.php
- 20.Maynard C, Nelson K. Compensation for veterans with service connected disabilities: current findings and future implications. J Disabil Policy Stud. 2020;31(1):57-62. doi: 10.1177/1044207319875070 [DOI] [Google Scholar]
- 21.University of Wisconsin School of Medicine and Public Health . Area Deprivation Index. Accessed April 3, 2026. https://www.neighborhoodatlas.medicine.wisc.edu/
- 22.Phibbs C, Cowgill E, Fan A. Guide to the PSSG Enrollee FileGuidebook. Health Economics Resource Center; 2013. [Google Scholar]
- 23.Moore BJ, White S, Washington R, Coenen N, Elixhauser A. Identifying increased risk of readmission and in-hospital mortality using hospital administrative data: the AHRQ Elixhauser Comorbidity Index. Med Care. 2017;55(7):698-705. doi: 10.1097/MLR.0000000000000735 [DOI] [PubMed] [Google Scholar]
- 24.National Academies of Sciences, Engineering, and Medicine; Division of Behavioral and Social Sciences and Education; Board on Human-Systems Integration; Division on Engineering and Physical Sciences; Board on Infrastructure and the Constructed Environment; Committee on Facilities Staffing Requirements for Veterans Health Administration . Nature of Veterans Health Administration facilities management (engineering) tasks and staffing. In: Facilities Staffing Requirements for the Veterans Health Administration-Resource Planning and Methodology for the Future. National Academies Press; 2019. [PubMed] [Google Scholar]
- 25.Connolly SL, Stolzmann KL, Heyworth L, et al. Patient and provider predictors of telemental health use prior to and during the COVID-19 pandemic within the Department of Veterans Affairs. Am Psychol. 2022;77(2):249-261. doi: 10.1037/amp0000895 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Lipsitch M, Tchetgen Tchetgen E, Cohen T. Negative controls: a tool for detecting confounding and bias in observational studies. Epidemiology. 2010;21(3):383-388. doi: 10.1097/EDE.0b013e3181d61eeb [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Connolly SL, Amspoker AB, Walder A, et al. Phone-only mental health care within the Department of Veterans Affairs: associations with rurality, age, sex, and clinical severity. J Rural Health. 2025;41(2):e70043. doi: 10.1111/jrh.70043 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Nearing KA, Dryden EM, Pimentel CB, et al. Can telemedicine reach rural, older veterans on the edge of or caught in the digital divide—unique considerations for two distinct populations. Cogent Gerontol. 2024;3(1):1-27. doi: 10.1080/28324897.2024.2336899 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Ferguson JM, Van Campen J, Slightam C, Greene L, Heyworth L, Zulman DM. Evaluation of the Veterans Health Administration’s digital divide consult for tablet distribution and telehealth adoption: cohort study. J Med Internet Res. 2024;26:e59089. doi: 10.2196/59089 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Sousa JL, Raja P, Rabinowitz M, et al. Patient experiences with group teletherapy for the treatment of mental illness: a qualitative study. Psychiatr Serv. 2025;76(2):157-162. doi: 10.1176/appi.ps.20240058 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eMethods.
eTable. Balance statistics for cohorts after IPWRA model
eFigure. Overlap plots by modality
Data Sharing Statement
