Effectiveness of Hybrid Psychiatric Visits Utilizing Patient-Reported Outcome Measures

Abstract

Objective:

There is little empirical support for the effectiveness of hybrid psychiatric care, defined as the delivery of care through a combination of phone, video, and in-person visits. This study investigates patient outcomes in hybrid psychiatric care compared to outpatient waitlist control groups measured with Patient-Reported Outcome Measures (PROMs).

Method:

Study patients were recruited from an adult psychiatry clinic waitlist where the most common primary diagnoses were unipolar depression, generalized anxiety disorder, and bipolar disorder. Patients (N = 148) were randomized into one of two waitlist control groups that completed PROMs once or monthly prior to treatment initiation. PROM scores for patients who received hybrid psychiatric treatment during a different time frame (N = 272) were compared to PROM scores of the waitlist groups. PROMs assessed symptoms of depression (PHQ-9), anxiety (GAD-7), and daily psychological functioning (BASE-6). Patient measures were summarized descriptively with means, medians, and standard deviations and then compared using the Kruskal-Wallis test, a non-parametric equivalent of the one-way ANOVA, and associated effect sizes were calculated.

Results:

The PROMs of patients who engaged in hybrid care indicated significant improvements in symptom severity compared to the waitlist groups, regardless of the number of PROMs completed while patients were on the waitlist. Between the hybrid care and waitlist groups, the effect size for the PHQ-9 (d = 0.66) was moderate; effect sizes were small for the GAD-7 (d = 0.46) and BASE-6 (d = 0.45).

Conclusions:

The findings demonstrate the clinical effectiveness of hybrid care and that PROMs are a method to assess this effectiveness.

Prior to the COVID-19 Public Health Emergency (PHE), telepsychiatry was utilized by a minority of psychiatrists. In a survey of members of the American Psychiatric Association about telemedicine use prior to and during the pandemic, 64% of psychiatrists reported that they saw no patients via telemedicine prior to the PHE (1). In contrast, by 2021, 81% of psychiatrists indicated they saw the majority (more than 75%) of their patients via telemedicine (1). This shift has potential benefits of increasing access to care, reducing patient no show rates, and providing a glimpse into patients’ environmental conditions (2–6). Notably, as the pandemic waned, psychiatry appointments settled into a mix of in-person, video and phone visits, otherwise known as hybrid care. Literature supports the effectiveness of telepsychiatry where all visits are delivered remotely, however our review of the literature located no studies which define hybrid care or examine its effectiveness.

The utilization of patient-reported outcome measures (PROMs) during in-person care allows for ongoing assessment of patient outcomes during treatment (7,8). Additionally, it provides feedback on the patient’s psychiatric symptoms to the patient and clinician (9,10). The use of PROMs to evaluate treatment effectiveness for in-person visits is empirically supported (7, 11–14). PROMs are also likely to be a useful tool to measure treatment effectiveness in the context of hybrid psychiatric care. Thus, a good platform to examine and build an evidence base for the effectiveness of hybrid psychiatric care would be a healthcare setting where PROMs were implemented before and during the transition to hybrid care.

An ambulatory adult psychiatry clinic in the Southeastern United States implemented PROMs prior to the start of COVID-19 and continues to utilize them as part of routine patient care (13). Before their first appointment, all new patients complete a bundle of PROMs including the PHQ-9, GAD-7, BASE-6, US AUDIT and DAST-10. Thereafter, patients receive monthly reminders to complete the PHQ-9, GAD-7, and BASE-6. In a previous study, our team analyzed results from PROMs to measure psychiatric symptoms before and after the outbreak of COVID-19 in the United States, when the use of telepsychiatry rose sharply (13). During the initial months of the COVID-19 pandemic, more than 90% of patients’ visits were via telepsychiatry, compared to fewer than 5% prior to that time. As the pandemic persisted, the delivery of psychiatric care began to shift to a mix of in person, phone, and video visits. Results from the COVID-19 study period indicate that patients who received care via telepsychiatry and who completed PROMs showed improvements in depression, anxiety and psychological functioning (13). However, concerns surfaced during that research analysis. One concern was the inability to compare outcomes in telepsychiatry for patients who completed PROMs to patients who did not receive active care. A second concern was the inability to speak to the effectiveness of hybrid care, which had begun to eclipse telepsychiatry as a modality of care delivery. This study uses PROMs as a measurement tool to investigate outpatient hybrid care outcomes compared to waitlist control groups. To our knowledge, this is the first study to report results on the clinical outcomes of hybrid psychiatric visits.

Method

Patients

This study compared three patient groups: two randomized patient waitlist groups and one retrospective hybrid treatment patient group (Consort Diagram in online supplement). Randomized study patients were 18 years of age or older on a waitlist for their initial session with a psychiatric clinician between April 1, 2022, and January 31, 2023. Informed consent was obtained via telehealth/eConsent prior to randomization. The local Institutional Review Board approved this study for both waitlist groups (IRB-21–1426).

A third group of patients (hybrid treatment N=272) served as a retrospective comparator for the two randomized waitlist groups. Inclusion criteria for the hybrid treatment group comprised patients ≥18 years of age who attended an initial visit where they completed the PROMs bundle and at least one follow-up PROM between November 1, 2021, and April 5, 2022, as part of usual care. Hybrid treatment was defined as psychiatric services received via a blend of in-person and telehealth (video or telephone) visits. During this time, upwards of 70% of visits were via telepsychiatry and fewer than 30% were in person. The hybrid group received primarily pharmacotherapy, though they could also engage in psychotherapy based on clinician recommendation and patient preference. The hybrid group served as a clinical control for this study and data was collected retrospectively. Therefore, they were exempt from the informed consent process per the IRB. The most common ICD-10 codes for patients in all groups did not differ and included major depression, generalized anxiety disorder, and bipolar disorder.

Measures

The following PROMs were utilized for tracking of psychiatric symptoms and general functioning:

Patient Health Questionnaire-9 (PHQ-9).

The PHQ-9 is a self-report measure containing items assessing depression symptoms and severity. The questionnaire asks patients about their symptoms during the past 2 weeks and includes 9 items representing DSM-5 criteria for major depressive disorder. Total scores on the PHQ-9 range from 0 to 27, with item response options ranging from 0 (not at all) to 3 (nearly every day). The PHQ-9 demonstrates good test-retest reliability (r = 0.84) and internal consistency (α = 0.89) (15).

Generalized Anxiety Disorder-7 (GAD-7).

The GAD-7 is a self-report questionnaire containing 7 items assessing the presence and severity of anxiety symptoms. This measure queries patients about their anxiety symptoms in the past 2 weeks using a 4-point likert scale ranging from 0 (not at all) to 3 (almost every day); higher scores indicate a greater severity of symptoms. Total scores range from 0 to 21. The GAD-7 has excellent test-retest reliability (r = 0.83) and internal consistency (α = 0.92) (16).

Brief Adjustment Scale-6 (BASE-6).

The BASE-6 is a measure which assesses general psychological adjustment and functioning over the past week via self-report on 6 items. Item response options range from 1 (not at all) to 7 (extremely) on this scale, with scores of 19 or higher representing clinically meaningful distress. Total scores range from 6 to 42. Non-clinical samples demonstrate a 0.77 test-retest reliability. Internal consistency for the BASE-6 ranges from good to excellent in clinical and nonclinical samples (α = 0.87–0.93)(17).

US Alcohol Use Disorders Identification Test (USAUDIT).

USAUDIT is a self-report screening measure used to identify subjects using alcohol in a problematic way over the past 12 months. It measures three domains of alcohol use: hazardous use, dependence symptoms, and harmful alcohol use. The total score ranges from 0 to 46 points, with higher scores suggesting a mild/moderate/severe level of alcohol use and dependence (18).

Drug Abuse Screening Test (DAST-10).

The DAST-10 is a 10-item self-report questionnaire that assesses potential involvement with drugs, excluding alcohol and tobacco, during the past 12 months. Subjects report problematic drug use with yes/no answers. Results provide a quantitative index of the extent of problems related to drug abuse (total score ranges from 0 to 10) (19).

Procedures

Patients on the referral waitlist were contacted by a licensed professional counselor (LPC) who scheduled intake appointments and introduced patients to the clinic’s measurement-based feedback system of PROM delivery. PROMs were assigned 24-hours before the LPC appointment for patient completion via email or text. The initial PROM bundle included the PHQ-9, GAD-7, BASE-6, US AUDIT and DAST-10. Follow up PROMs included the PHQ-9, GAD-7 and BASE-6. The USAUDIT and DAST-10 were not administered repeatedly and patients with primary or significant substance use disorders were referred to the addiction medicine service. Before scheduling, the LPC ensured that the patient had access to the internet and ability to independently complete and understand PROMs.

Intakes by the LPC were completed via telephone. The LPC assessed for suicidality by asking about suicidal ideation, identified risks and protective factors, offered emergency resources in the event suicidality worsened, and acted on emergency services if a patient expressed active suicidal thoughts. At the conclusion of the intake appointment, the opportunity for study enrollment was introduced to the patient. Once enrolled, the patient was randomized to the waitlist group or the waitlist + group. Patients in the waitlist group completed the initial PROM bundle during the LPC intake and none thereafter until entry into the clinic. Patients in the waitlist + group completed the initial PROM bundle during the LPC intake and monthly follow-up PROMs (PHQ-9, GAD-7, BASE-6) while they remained on the waitlist. The waitlist + group had a significantly longer wait time from the intake assessment to the initial appointment (47.5 vs. 35.1 days; t=2.86, df=136, p = 0.005).

The hybrid care group included patients who completed a new intake PROM bundle for their initial appointment and at least one follow-up PROM between November 1, 2021, and April 5, 2022. The average length of time in treatment for patients in the hybrid group was 50.0 ± 29.8 days. Approximately 58% of the patients in the hybrid care group completed one follow up PROM during the described timeframe, followed by 22% who completed two PROMs, and 12% who completed three PROMs. The remaining 8% of patients completed four or five PROMs. For all three study groups, the initial and last completed PROMs were used for comparison.

Analysis Plan

In the waitlist group, 48 of the 70 patients completed one PROM at entry into the study, and one upon entry into the treatment clinic. In the waitlist + group, 43 of the 78 patients completed PROMs monthly, with 19 completing one set of PROMs before their initial appointment in addition to PROMs completion on entry into the clinic, 20 completing two PROMs, and 4 completing three PROMs. Patient measures for all three groups (waitlist, waitlist +, and hybrid care) were first summarized descriptively with means and standard deviations. The Kruskal-Wallis test, a non-parametric equivalent of the one-way ANOVA, was conducted to compare the differences between the three groups. For the next analysis, the two waitlist groups (waitlist and waitlist +) were combined and compared to the hybrid care group using a one-way ANOVA to compare the mean differences between intake and final scores. An effect size of Cohen’s d was calculated for both analyses (20). P-values less than 0.05 were evaluated as statistically significant throughout. All analyses were conducted using SAS Enterprise Guide 8.3.

Results

Baseline PROM Data (Table 1)

Table 1.

Demographic characteristics and initial PROM scores

Variable	Waitlist (N=78)		Waitlist + (N=70)		Hybrid Care (N=272)
	N	%	N	%	N	%
Demographica
Sex
Female	54	69	52	74	198	73
Male	24	31	18	26	74	27
Race or ethnic group
Asian					6	2
Black or African American	12	15	5	7	12	4
Hispanic					2	1
More than one			3	4	11	4
Other					6	2
White or Caucasian	66	85	62	89	235	86
Age, y (M±SD)	36±12		37±13		38±15
PROM (M±SD)
PHQ-9 scoreb	14±6		14±6		14±6
GAD-7 scorec	12±5		13±6		12±6
BASE-6 scored	28±9		27±9		27±9
DAST-10 scoree	1±1		1±2		1±1
USAUADIT scoref
Female patients	4±4		4±4		4±7
Male patients	6±8		3±4		5±7

^aThere were no significant differences in sex (χ²=0.54, N=420, df=2, p=.764), race or ethnic group (χ²=0.49, N=420, df=2, p=.782), or age (F=1.32, df=2, 417, p=.268) across the groups.

^bScores on the PHQ-9 range from 0 to 27, with 0 to 4 indicating no to minimal depression symptoms, 5 to 9 mild symptoms, 10 to 14 moderate symptoms, 15 to 19 moderately severe symptoms, and 20 to 27 severe symptoms.

^cScores on the GAD-7 range from 0 to 21, with 0 to 4 indicating no to minimal anxiety symptoms, 5 to 9 mild symptoms, 10 to 14 moderate symptoms, and 15 to 21 severe symptoms.

^dScores on the BASE-6 range from 6 to 42, with ≥19 indicating significant distress.

^eThe DAST-10 is a 10-question tool in which 1 point is assigned for a response of “yes” and no points are assigned for a response of “no.” A score of >6 provides excellent sensitivity for identifying patients with SUD.

^fScores on the USAUDIT range from 0 to 40, with >7 indicating misuse of alcohol.

PHQ-9.

The waitlist (13.72 ± 6.28), waitlist + (13.80 ± 6.10), and hybrid care (13.88 ± 6.32) groups did not significantly differ at intake (F=0.02, df=2 and 417, p=0.980). Scores for all groups indicate that patients were experiencing a moderate level of depression symptom severity.

GAD-7.

Patient scores on their first completion of the GAD-7 did not significantly differ across groups (F=0.87, df=2 and 414, p=0.418). The waitlist (12.49 ± 5.36), waitlist + (12.90 ± 5.68), and hybrid care (11.95 ± 5.87) groups all reported experiencing moderate levels of anxiety symptoms.

BASE-6.

Comparisons of initial BASE-6 scores for the waitlist (27.59 ± 8.99), waitlist + (27.41 ± 9.16), and hybrid care (26.61 ± 9.43) groups yielded no significant difference (F=0.45, df=2 and 407, p=0.639). With a clinical cutoff of 19, the scores indicate a clinically significant level of distress across groups.

DAST-10 and USAUDIT.

There was no significant difference in mean scores on the DAST-10 (F=1.31, df=2 and 406, p=0.270) or USAUDIT (F=0.81, df=2 and 290, p=0.448 for female; F=0.74, df=2 and 111, p=0.482 for male) among the studied groups, with consideration to gender on the USAUDIT (female waitlist: 3.52 ± 3.74; male waitlist: 5.83 ± 7.52; female waitlist +: 3.54 ± 4.07; male waitlist +: 3.22 ± 3.56; female hybrid care: 4.47 ± 7.04; male hybrid care: 4.75 ± 7.19) given the different risk ranges for men versus women. Mean initial scores indicated low risk for alcohol and drug use (0.55 ± 1.05 for waitlist group; 0.90 ± 1.87 for waitlist + group; 0.60 ± 1.48 for hybrid care) in the study population at the time of entry into the study.

Changes in PROM Scores Over Time

Table 2 shows the differences in median PROM scores between the waitlist, waitlist + and hybrid care groups. While intake scores are similar across groups, the hybrid care group has the largest median differences on all three PROMs, with a decrease by 3.0 points on the PHQ-9, 2.0 points on the GAD-7, and 3.0 points on the BASE-6. A Kruskal-Wallis test was conducted to compare the differences between the three groups. Across all three measurements, median scores from intake to final differed significantly between groups. For the PHQ-9, the effect size (Cohen’s d = 0.55) was moderate (20). For the GAD-7 and BASE-6, effect sizes were small (Cohen’s d = 0.35 for both). Since there were similar differences between waitlist groups, these were combined for subsequent analyses.

Table 2.

Changes in the PHQ-9, GAD-7, and BASE-7 scores over time

PROM	N	Initial		Final		Change		p	d
		Med.	IQR	Med.	IQR	Med.	IQR
PHQ-9 scorea								<.001	0.55
Hybrid care	267	14.0	9.0–19.0	9.0	6.0–15.0	-3.0	-6.0–0.0
Waitlist +	43	13.0	9.0–19.0	13.0	9.0–18.0	0	-2.0–2.0
Waitlist	48	12.0	8.5–20.5	12.5	9.0–17.5	0	-3.0–2.0
GAD-7 scoreb								.003	0.35
Hybrid care	268	12.0	8.0–17.0	8.0	5.0–14.0	-2.0	-5.0–1.0
Waitlist +	43	13.0	8.0–16.0	13.0	7.0–18.0	0	-3.0–2.0
Waitlist	48	12.0	8.5–17.0	12.5	7.5–15.0	-1.0	-4.0–1.0
BASE-6 scorec								.002	0.35
Hybrid care	261	28.0	20.0–34.0	22.5	14.0–31.0	-3.0	-8.0–2.0
Waitlist +	43	27.0	20.0–34.0	28.0	21.0–36.0	0	-2.0–3.0
Waitlist	47	29.0	20.0–33.0	27.0	19.0–32.0	-2.0	-5.0–3.0

^aScores on the PHQ-9 range from 0 to 27, with 0 to 4 indicating no to minimal depression symptoms, 5 to 9 mild symptoms, 10 to 14 moderate symptoms, 15 to 19 moderately severe symptoms, and 20 to 27 severe symptoms.

^bScores on the GAD-7 range from 0 to 21, with 0 to 4 indicating no to minimal anxiety symptoms, 5 to 9 mild symptoms, 10 to 14 moderate symptoms, and 15 to 21 severe symptoms.

^cScores on the BASE-6 range from 6 to 42, with ≥19 indicating significant distress.

Table 3 shows the comparison between the combined waitlist groups and the hybrid care group. For all three measures, the hybrid care group had significantly lower scores at the final measurement compared to the combined waitlist groups (PHQ-9: t=−6.19, df=255.13, p = <.001; GAD-7: t=−4.25, df=244.82, p = <.001; BASE-6: t=−4.21, df=262.78, p = <.001). For the PHQ-9, the effect size (Cohen’s d = 0.66) was moderate (20). For the GAD-7 and BASE-6, effect sizes were small (Cohen’s d = 0.46 and Cohen’s d = 0.45, respectively).

Table 3.

Changes in the PHQ-9, GAD-7, and BASE-7 scores over time in the hybrid care group and in the waitlist and waitlist + groups combined

PROM	N	Initial	Final	Difference	p	d
		M±SD	M±SD	M±SD
PHQ-9 scorea					<.001	0.66
Hybrid care	267	13.84±6.33	10.51±6.12	-3.33±5.76
Waitlist and waitlist +	91	13.45±6.53	13.29±6.52	-0.16±3.54
GAD-7 scoreb					<.001	0.46
Hybrid care	268	11.96±5.88	9.41±5.93	-2.54±5.16
Waitlist and waitlist +	91	12.47±5.54	11.92±5.80	-0.55±3.30
BASE-6 scorec					<.001	0.45
Hybrid care	261	26.68±9.37	22.99±9.92	-3.69±8.68
Waitlist and waitlist +	90	26.83±9.18	26.36±9.16	-0.48±5.15

^aScores on the PHQ-9 range from 0 to 27, with 0 to 4 indicating no to minimal depression symptoms, 5 to 9 mild symptoms, 10 to 14 moderate symptoms, 15 to 19 moderately severe symptoms, and 20 to 27 severe symptoms.

^bScores on the GAD-7 range from 0 to 21, with 0 to 4 indicating no to minimal anxiety symptoms, 5 to 9 mild symptoms, 10 to 14 moderate symptoms, and 15 to 21 severe symptoms.

^cScores on the BASE-6 range from 6 to 42, with ≥19 indicating significant distress.

Discussion

The primary purpose of this study was to compare the effectiveness of hybrid psychiatric care to randomized waitlist control groups using patient-reported outcome measures (PROMs). Our main finding demonstrated that patients who engaged in hybrid care showed significant improvements in symptom severity and general functioning compared to the waitlist groups. The findings also demonstrate the clinical utility of incorporating PROMs into hybrid psychiatric practice and highlight the potential benefits of PROMs as a tool for assessing patient outcomes in a hybrid setting.

In light of these findings, it should be noted that the hybrid care group’s final measurements were conducted, on average, 50 days following treatment initiation, or at 7 weeks of treatment. This was not necessarily the end of their treatment course. Still, on average, patients in hybrid care experienced a meaningful reduction of depressive symptoms. These findings are consistent with previous research that supports the use of PROMs in mental health intervention settings (9, 21–22).

Patients in the combined waitlist group did not demonstrate improvement in PROM scores whether they completed one or multiple sets of PROMs. Therefore, our findings do not support the notion that PROM completion by patients on a waitlist impacts their symptom course. Our results support prior research findings which showed that screening alone for mental health symptoms is insufficient without follow up and treatment (23). In mental health clinics, the role of PROMs is to track symptoms over time and inform treatment decisions rather than serve as a diagnostic tool. The significant reduction in all PROMs scores in our study emphasizes the effectiveness of active treatment and outcomes measured by PROMs but does not explain the mechanisms that led to patient improvement. This raises questions of what role PROM feedback plays in active treatment.

Limitations and Future Directions

This study has several limitations. First, while a waitlist control group was compared to an active treatment group, there was no control group who underwent psychiatric treatment without PROMs. Future research might incorporate a hybrid treatment comparison group without PROMs to determine whether there are differences in outcomes for those who engage in PROMs and those who do not. Second, PROM scores were not shared with patients in the waitlist groups, nor were they given education about what the scores meant clinically. Providing waitlist patients with their scores and interpretations of scores could be important to the design of future studies in this area. Third, the hybrid care sample receiving active treatment was not randomized, nor was it drawn from the same time as the waitlist groups. This could have introduced selection bias into the study, though demographics and diagnoses were not statistically different between the groups. Fourth, the data was collected during the COVID-19 pandemic from a single adult clinic located in the Southeast United States, and patients were predominantly white and female, limiting the generalizability of results to other settings or populations. Fifth, the duration of the active treatment was brief, only 50 days on average. Studies examining longer duration of hybrid care should be performed. In addition, data regarding the number of telepsychiatry versus in person visits were not available at the individual level, making it impossible to determine what mix of in person and telepsychiatry individual patients engaged in. Hybrid care may exist in any range of proportions from 1–99% being telemedicine. Our hybrid treatment group took the form of more telepsychiatry visits than in-person visits. Research assessing various degrees of hybrid care would be helpful to determine whether there is a relationship between the proportion of hybrid mix and patient outcomes. Finally, effect sizes for reduction in PROMs in the active treatment group were lower than those reported in research from other countries and settings (24, 25). We hypothesize that since the same PROMs were sent to all patients, regardless of diagnosis, and since we did not exclude scores that indicated mild symptoms, this led to lower baseline scores. As such, it could have impacted effect sizes. More targeted use of PROMs by clinicians and pairing PROMs with specific diagnoses in data analyses could be useful in future studies.

Conclusion

This study’s findings are the first to offer information about patient outcomes in the context of hybrid psychiatric care. The results also demonstrate the importance of incorporating PROMs into hybrid care as a tool for ongoing evaluation and monitoring of psychiatric symptoms and general functioning. Future studies might address the identified limitations and investigate the generalizability of these findings by studying a variety of populations and clinic settings. Studies that follow patients who complete PROMs while receiving hybrid care over an extended treatment span could provide information about long-term benefits as well as potential challenges of this model of care. Future research can also investigate the mechanisms of change related to the use of PROMs by examining the factors that contribute to the positive outcomes.

Highlights.

• The COVID-19 pandemic accelerated the transition from in person to remote visits in psychiatry, a practice that is likely to persist and which has potential to enhance patient care by increasing access to care and allowing for flexibility in assessment.

• Patient-Reported Outcome Measures (PROMs) are critical tools that support outcome monitoring in patient care, and as such, can further inform hybrid care research.

• Patients receiving hybrid psychiatric care showed clinical improvement, in contrast to patients on a psychiatry waitlist who did not improve clinically.