The impact of increased telehealth use on the treatment of substance use disorder during the COVID-19 pandemic

Abstract

Background

The COVID-19 pandemic necessitated a shift from in-person substance use disorder (SUD) treatment to virtual telehealth (TH) visits, creating opportunities to assess the impact of virtual visits on SUD treatment.

Methods

This study utilized retrospective, de-identified, electronic health record (EHR) data from Oracle EHR Real-World Data to examine the impact of TH on SUD treatment. Patients with a qualifying SUD diagnosis from 141 U.S. health systems were included and divided into pre-TH (January 1, 2017 through January 1, 2019) and COVID (January 1, 2020 through January 1, 2022) cohorts. This study analyzed TH utilization, medications for SUD (MSUD) prescribing, drug-related events, and mental health crises, comparing patient outcomes where the treating clinician was a high TH user versus a low TH user in both pre-COVID and COVID periods.

Results

Patients visiting high TH clinicians had lower MSUD prescribing rates, yet a higher MSUD day's supply, and higher rates of TH outpatient visits than those visiting low TH providers, with both groups having an increase in TH visits during the COVID period. Patients with high TH clinicians had lower rates of SUD-related hospitalizations than those with low TH providers but similar rates of drug overdoses, relapses, injection-related infections, and mental health crises.

Conclusions

TH modalities showed increased SUD-related outpatient visits without increasing adverse outcomes, indicating its potential as a sustainable alternative to in-person care. This study highlights the need for further research on TH efficacy for SUD-specific populations and supports the continued integration of telehealth in SUD treatment post-pandemic.

Highlights

• •Telehealth use for SUD treatment surged during the COVID-19 pandemic.
• •High telehealth clinicians had lower SUD-related hospitalization rates.
• •Patients of high telehealth clinicians had longer MSUD prescription days' supply.
• •Telehealth maintained similar rates of adverse events as in-person care.
• •Results support telehealth as a sustainable option for post-pandemic SUD treatment.

1. Introduction

The United States has seen a concerning rise in drug-related mortality in recent years, with the total number of drug overdose deaths increasing from 70,630 in 2019 to 107,941 in 2022. (Drug Overdose Death Rates) The annual increase from 2019 to 2020 alone was particularly notable, with the drug overdose deaths jumping from 70,630 to 91,799, marking a 30 % increase of the pandemic's onset. (Drug Overdose Death Rates) To mitigate the dangers of substance use disorders (SUDs), traditional forms of treatment exist and comprise SUD-related services in an outpatient setting (e.g., substance use monitoring, behavioral counseling and therapy, evaluation and treatment for co-occurring disorders like anxiety and depression (Miller and Sharp)) as well as pharmacotherapy (Douaihy et al., 2013; Elias & Kleber, 2017). However, both have been severely underutilized by clinicians and patients due to financial, regulatory, geographic, availability, and attitudinal barriers among clinicians and patients (Farhoudian et al., 2022; Sharma et al., 2017; Williams et al., 2018). To add greater complexity to the already challenging treatment landscape for those with SUD, the COVID-19 pandemic necessitated a rapid transition of in-person SUD treatment services to virtual telehealth platforms to adhere to social distancing guidelines (Drake et al., 2020; Molfenter et al., 2021). This presented a unique challenge to health systems as relatively little telehealth had been used for SUD treatment prior to the COVID-19 onset and as the pandemic provided an opportunity to deliver SUD treatment to individuals who previously could not access it (Huskamp et al., 2022).

Before the pandemic, telehealth faced reimbursement challenges and regulatory restrictions, such as the Ryan Haight Act's in-person prescription requirement (Drake et al., 2020; Huskamp et al., 2022). Until 2017 there were no widely accepted telehealth reimbursement policies, contributing to low utilization along with barriers like limited patient access to technology, clinician training issues, and strong preference for in-person services (Trout et al., 2017; Uscher et al., 2020). The emergency status of the pandemic led to telehealth legislation that temporary suspended these restrictions and brought about a significant increase in telehealth utilization, benefiting patients with transportation issues and those in rural areas (Sugarman et al., 2021; Hammerslag et al., 2023; Gaziel-Yablowitz et al., 2021; Carlson et al., 2012; Anawade et al., 2024; Health Care Access). The unique circumstances of the pandemic also resulted in the relaxation of restrictions on telehealth clinicians' ability to prescribe pharmacotherapy for SUDs (Hammerslag et al., 2023). Telehealth services have been shown to increase healthcare accessibility, reduce costs, and address underserved populations effectively among patients with SUDs (Edinoff et al., 2022; King et al., 2014; Uscher et al., 2020; Vinci et al., 2022). They appear to offer a convenient and safe alternative to in-person care, potentially revolutionizing SUD treatment by overcoming geographical and logistical barriers (King et al., 2014; Molfenter et al., 2021). However, there is a lack of literature assessing telehealth efficacy with respect to large-scale SUD patient populations.

Current literature, focused on patients with opioid use disorder (OUD), suggests that clinical outcomes were similar among patients treated by clinicians in low-use versus high-use telehealth groups, based on the percentage of SUD-related outpatient visits that were conducted via telehealth (Hailu et al., 2023). When looking into patients with alcohol use disorder (AUD), similarities were also found between technology-delivered, cognitive-behavioral therapy (CBT) interventions for alcohol use (CBT Tech) and treatment as usual (TAU). Yet CBT Tech in addition to TAU showed significantly positive findings, compared to TAU-alone, and was stable over 12 months of follow-up (Kiluk et al., 2019). Alcohol use has been shown to be effectively reduced through digitally delivered CBT among patients with AUD across other studies as well (Hester et al., 2013; Kiluk et al., 2016, 2019; Sinadinovic et al., 2014; Zill et al., 2019). There are also implications that telehealth can be a transformative mode of delivery for pharmacotherapy; with one study finding that telehealth initiation is associated with better odds of 90-day retention with buprenorphine (Hammerslag et al., 2023). Yet, there still remains a lack of information surrounding the effectiveness of telehealth on a large population of patients with SUDs.

Utilizing large-scale, real-world, clinical data, this study aims to explore the integration of digital health interventions in SUD treatment pre- and post-pandemic, focusing on their association with treatment and health outcomes. While in-person treatments continue to be regarded as the highest standard of care, virtual services can offer a safe and effective alternative (Molfenter et al., 2021). Hence, evaluating the effectiveness of telehealth services for SUD patients is vital in understanding whether these services offer a sustainable alternative to in-person care in the post-pandemic world.

2. Methods

2.1. Data source

This study was conducted using retrospective, de-identified, EHR data from the Oracle EHR Real-World Data™ (OERWD). Oracle EHR Real-World Data is extracted from the EMR of hospitals in which Oracle has a data use agreement. Encounters may include pharmacy, clinical and microbiology laboratory, admission, and billing information from affiliated patient care locations. All admissions, medication orders and dispensing, laboratory orders and specimens are date and time stamped, providing a temporal relationship between treatment patters and clinical information. Oracle has established Health Insurance Portability and Accountability Act (HIPPA) -compliant operating policies to establish de-identification for Oracle EHR Real-World Data. As of September 2023 refresh, 141 health systems in the U.S. contribute to OERWD providing data for over 111 million patients leading to a total of approximately 1.9 billion encounters. OERWD is deidentified using the “Safe Harbor” method under HIPPA where all personally identifiable information is removed at the patient level, and where all date/timestamps fields are shifted. Health systems are coded with unique identifiers to mask their identities (Ehwerhemuepha et al., 2022).

2.2. Sample

Patients were firstly included in this study if they had a qualifying SUD diagnosis, defined by the International Classification of Diseases, Ninth/Tenth Revision, Clinical Modification (ICD-9/10-CM) or Systemized Nomenclature of Medicine-Clinical Terms (SNOMED CT) codes (Supplemental Table 1). The first inpatient or emergency encounter with a diagnosis code or the first of ≥ 2 encounters, of any type, with diagnosis codes was chosen as the first SUD date. Further, patients with an SUD diagnosis were required to have either an outpatient office encounter for SUD services (defined by Current Procedural Terminology [CPT] and Healthcare Common Procedure Coding System [HCPCS] codes, Supplemental Table 2), outpatient SUD-related encounter (defined by encounters of type “outpatient” with qualifying CPT, HCPCS, International Classification of Diseases, Ninth/Tenth Revision, Procedure Coding System [ICD-9/10-PCS] codes, Supplemental Table 3), or outpatient treatment with medications for substance use disorder (MSUD, defined by National Drug Codes [NDC], Multum MediSource Lexicon [MMSL], HCPCS, and ICD-9/10 PCS codes, Supplemental Table 4) within one year of first SUD diagnosis. The first of these occurrences was considered the index encounter. Next, as similarly designed by Hailu et al. (Hailu et al., 2023), clinicians served as the method of telehealth exposure assignment. Clinicians were limited to those in departments that were more likely to be office-based prescribers of medications for opioid use disorders (MOUD) (e.g., anesthesiology, internal medicine, nurse practitioners, obstetrics and gynecology (OB/GYN), primary care, psychiatry & neurology, rehabilitation) as well as having prescribed ≥ 1 MSUD (because this study considered all SUD diagnoses, all MSUD prescriptions, comprising MOUD, medications for alcohol use disorder (MAUD), and medications for tobacco use disorder [MTUD] were allowed for inclusion). Patients were required to have been seen by one of the qualifying clinicians for at least their index encounter. Patients were also required to be ≥ 12 years old at index encounter and have at least one year of follow-up post index encounter. Finally, patients were split into two time-window periods, in which patients with first SUD diagnoses and index encounters between January 1, 2017 and January 1, 2019 were included into the “Pre-COVID” group, and patients with first SUD diagnoses and index encounters between January 1, 2020 and January 1, 2022 were included into the “COVID” group. These two separate windows allowed for comparison of changes in treatment shifts occurring before and after the onset of the COVID-19 pandemic. As the study included patients with first SUD diagnoses into each time-window, the two groups were mutually exclusive.

2.3. Study design

This is a retrospective cohort study with cohorts analyzed in both the pre-COVID and COVID periods. All analyses are thus repeated for patients in both periods. Patients were allowed inclusion for two years to allow those last recruited to have one year of follow-up. Thus, cohort inclusion was ended at January 1, 2019 for the pre-COVID cohort to allow one year of follow-up for those lastly recruited, with final one-year follow-up ending at January 1, 2020. This also prevented the final follow-up period from overlapping with COVID-19, and possibly confounding outcomes. Similarly, cohort inclusion was ended at January 1, 2022 for the COVID cohort to allow one year of follow-up for those lastly recruited, with final one-year follow-up ending at January 1, 2023. Patients were followed from ≥ 1 day post index encounter and up to one year after for assessment of study outcomes while comparing between those in the exposed and those in the non-exposed group.

2.4. Outcomes

The primary outcomes of this study comprise patient-level indications for telehealth visits, MSUD prescribing, drug-related events, and mental health events. The rate of telehealth visits was defined as the count of telehealth SUD-related outpatient encounters (defined by CPT, HCPCS, and SNOMED codes in Supplemental Table 5 along with any outpatient encounter types of “encounter by computer link”, “telehealth consultation with patient”, “telephone consultation”, “telephone encounter”, “videotelephone encounter”) divided by all SUD-related outpatient encounters. MSUD prescribing was focused on prescriptions comprising MOUD, MAUD, and MTUD with NDC and MMSL codes listed in Supplemental Table 4. MSUD was defined as a) a binary (yes/no) indication of whether the patient was prescribed ≥ 1 MSUD b) the number of days’ supply (difference in prescribed stop date and prescribed start date) among those prescribed MSUD and c) the average monthly percentage change in prescriptions (average of monthly percentage changes of MSUD prescription counts) among those prescribed MSUD. Drug-related events (sum of events divided by the sum of total patients) were defined as a) the rate of SUD-related hospitalizations (SUD diagnosis encounters with a length of stay [LOS] ≥ 1 day) per 100 patients b) the rate of all-drug overdoses (ICD-9/10 and SNOMED codes in Supplemental Table 6; inclusive of alcohol intoxications and tobacco poisonings) per 1000 patients c) the rate of relapses (SUD-related events or all-drug overdoses occurring 30 days apart) per 1000 patients and d) the rate of injection-related infections (ICD-9/10 and SNOMED codes in Supplemental Table 7; conditions that are potentially injection-related infections) (LaRochelle et al., 2020) per 1000 patients. Mental health events were defined specifically as the rate (sum of events divided by the sum of total patients) of mental health related events, comprised of emergency/inpatient encounters with diagnoses for anxiety/depression or encounters of any type for suicide-attempt/intentional self-harm (ICD-9/10 and SNOMED codes in Supplemental Table 8) per 1000 patients.

2.5. Exposure

The exposure of interest in this study was a binary (yes/no) indication of high or low telehealth (TH) use status. Specifically, this status was determined by clinician and defined by whether the clinician either had ≥ 20 % of all their SUD-related outpatient encounters conducted via TH (high TH) or <20 % of all their outpatient encounters conducted via TH (low TH) calculated across all time periods (both pre-COVID and COVID). This cutoff was chosen based on the reported TH percentage among medium TH users during the COVID pandemic (Hailu et al., 2023). Patient assignment to exposure groups was determined based on the majority (>50 %) of a patient's encounters occurring with either high-TH or low-TH clinicians. Patients who had more than 50 % (i.e., majority) of their SUD-related outpatient encounters with high-TH clinicians were assigned to the high-TH group, and those who had more than 50 % of encounters with low-TH clinicians were assigned to the low-TH group. This approach ensured that patient assignment reflected consistent exposure to telehealth, rather than occasional telehealth encounters, thereby minimizing potential selection bias.This clinician-based assignment method was used instead of a direct patient-level classification to avoid bias due to clinicians selecting telehealth use based on individual patient characteristics. However, we conducted a sensitivity analysis where exposure was defined at the patient level (i.e., patients who had ≥20 % of their outpatient encounters via telehealth were considered high-TH), and results remained consistent.

Additional patient-level demographic measures, at index encounter, included the continuous age (in years), gender (female, male), race (non-Hispanic [NH]-American Indian or Alaskan Native [AI/AN], NH-Asian or Pacific Islander [API], NH-Black, NH-White, Hispanic, NH-Other, unknown), census region (Northeast, Midwest, South, West), rurality (rural, urban), metropolitan status (metropolitan, non-metropolitan), and insurance (private, Medicare, Medicaid, other government/miscellaneous, self-pay, unknown). Clinical measures included comorbidity (Charlson Comorbidity Index [CCI]: 0, 1–2, 3–4, ≥ 5; codes for qualifying conditions listed in Supplemental Table 9) (Charlson et al., 1987), count of index SUDs (1, 2, 3, ≥ 4), and history of mental health conditions (0, 1, 2, ≥ 3; codes for conditions in Supplemental Table 10). Clinician departments were also captured.

2.6. Statistical analysis

Patient and clinician characteristics were presented overall as well as stratified by TH status and compared with significance testing (Chi-squared tests for categorical variables and two-sample independent t-tests [assuming unequal variances] for continuous, normally distributed, variables). Patient-level outcomes were presented and compared by TH status as well as modeled with mixed-effects regressions. Negative binomial regression quantified the incidence rate ratio (IRR) of telehealth visits, drug-related events, and mental health events. Logistic regression quantified the odds ratio (OR) of MSUD prescription. Exponential regression quantified exponentiated beta-hats (${\mathit{e}}^{\hat{\mathit{\beta }}}$) for days’ supply of MSUD prescription. Linear regression quantified beta-hats ($\hat{\beta }$) for average monthly percentage change in MSUD prescriptions. Variability of estimates was captured with 95 % confidence intervals (CIs). To compare outcomes between the pre-COVID and COVID cohorts, we assessed the overlap of 95 % CIs for the respective estimates in each period. Non-overlapping confidence intervals indicated statistically significant differences between the periods. All regressions were fit with the primary exposure of TH status comparing patient-level outcomes for those visiting high TH clinicians compared to those visiting low TH clinicians. Models were also adjusted for additional measures, previously mentioned, and accounted for clinician-specific variability by clustering standard errors (SEs) by clinician. Diagnostics were investigated to ensure model goodness-of-fit. All hypothesis tests were two-sided with a significance level of 5 %. All analyses were conducted in R version 4.0.2 (R Foundation for Statistical Computing).

2.7. Sensitivity analyses

Sensitivity analyses were conducted to confirm robustness of findings. For the first sensitivity analysis, TH status was redefined based on the mean percentage of TH encounters (out of all outpatient encounters) among clinicians in the data. Thus, instead of a cut point of 20 %, TH status was defined as high for those with ≥ 4 % telehealth encounters and low for those less than 4 %. The second sensitivity analysis defined TH status by the patient rather than the clinician. Thus, patients who had ≥ 20 % telehealth encounters (of all their outpatient encounters) were defined as high TH and patients who had less than 20 % were defined as low TH. The last sensitivity analysis redefined the COVID inclusion period to be from September 30, 2020 to September 30, 2022 (with final year follow-up ending September 30, 2023 at data refresh). This was done to allow for a period of time in which policy changes due to the pandemic had sufficient time to take effect. All main analyses were repeated, and results were compared.

3. Results

3.1. Descriptive statistics

Table 1 provides the overall demographic and clinical characteristics of patients with SUDs, overall and by TH status, in both the pre-COVID and COVID periods. There were 38,448 patients in the pre-COVID period, of which 61.1 % (23,486) were classified as low TH use and 38.9 % (14,962) high TH use. There were 41,325 patients in the COVID period, of which 65.4 % (27,030) were low TH use and 34.6 % (14,295) were high TH use.

Table 1.

Patient (with SUDs) and clinician demographic and clinical characteristics^a (overall and by period/clinician telehealth status) within Oracle EHR Real-World Data (OERWD)-affiliated health systems.

	Pre-COVIDb				COVIDc
	Overall	Low Telehealthd	High Telehealthe	p-valuef	Overall	Low Telehealthd	High Telehealthe	p-valuef
Patient	38,448	23,486 (61.1g)	14,962 (38.9g)		41,325	27,030 (65.4g)	14,295 (34.6g)
Age (Years), Mean (SDh)	48.24 (14.07)	47.49 (14.65)	49.42 (13.02)	<0.001i	48.01 (14.34)	48.11 (14.67)	47.83 (13.69)	0.06i
Gender, nj (%k)				<0.001				<0.001
Female	19409 (50.5)	12656 (53.9)	6753 (45.1)		20759 (50.2)	14651 (54.2)	6108 (42.7)
Male	19039 (49.5)	10830 (46.1)	8209 (54.9)		20566 (49.8)	12379 (45.8)	8187 (57.3)
Race, n (%)				<0.001				<0.001
NH-AI/ANl	157 (0.4)	119 (0.5)	38 (0.3)		281 (0.7)	198 (0.7)	83 (0.6)
NH-APIm	800 (2.1)	309 (1.3)	491 (3.3)		671 (1.6)	227 (0.8)	444 (3.1)
NH-Black	4682 (12.2)	2117 (9.0)	2565 (17.1)		3505 (8.5)	1807 (6.7)	1698 (11.9)
NH-Other	3088 (8.0)	1049 (4.5)	2039 (13.6)		2645 (6.4)	767 (2.8)	1878 (13.1)
NH-White	20798 (54.1)	15782 (67.2)	5016 (33.5)		25522 (61.8)	20462 (75.7)	5060 (35.4)
Hispanic	8167 (21.2)	3534 (15.0)	4633 (31.0)		7583 (18.3)	2766 (10.2)	4817 (33.7)
Unknown	756 (2.0)	576 (2.5)	180 (1.2)		1118 (2.7)	803 (3.0)	315 (2.2)
Census Region, n (%)				<0.001				<0.001
Northeast	3669 (9.6)	3055 (13.0)	614 (4.1)		5139 (12.5)	4529 (16.8)	610 (4.3)
Midwest	10196 (26.5)	9202 (39.2)	994 (6.6)		12558 (30.4)	10808 (40.1)	1750 (12.2)
South	2798 (7.3)	2449 (10.4)	349 (2.3)		5062 (12.3)	4402 (16.3)	660 (4.6)
West	21749 (56.6)	8746 (37.3)	13003 (86.9)		18493 (44.8)	7219 (26.8)	11274 (78.9)
Rurality, n (%)				<0.001				<0.001
Rural	5976 (15.5)	4893 (20.8)	1083 (7.2)		9405 (22.8)	7752 (28.7)	1653 (11.6)
Urban	32472 (84.5)	18593 (79.2)	13879 (92.8)		31920 (77.2)	19278 (71.3)	12642 (88.4)
Metropolitan, n (%)				<0.001				<0.001
Metropolitan	32407 (84.3)	18534 (78.9)	13873 (92.7)		32017 (77.5)	19372 (71.7)	12645 (88.5)
Non-metropolitan	6041 (15.7)	4952 (21.1)	1089 (7.3)		9308 (22.5)	7658 (28.3)	1650 (11.5)
Insurance, n (%)				<0.001				<0.001
Private	7775 (20.2)	6276 (26.7)	1499 (10.0)		10709 (25.9)	8982 (33.2)	1727 (12.1)
Medicare	8889 (23.1)	6062 (25.8)	2827 (18.9)		8301 (20.1)	6227 (23.0)	2074 (14.5)
Medicaid	15112 (39.3)	7500 (31.9)	7612 (50.9)		15983 (38.7)	7806 (28.9)	8177 (57.2)
Other Govt/Misc	2957 (7.7)	1382 (5.9)	1575 (10.5)		2775 (6.7)	1450 (5.4)	1325 (9.3)
Self-Pay	3297 (8.6)	1850 (7.9)	1447 (9.7)		2965 (7.2)	1987 (7.4)	978 (6.8)
Unknown	418 (1.1)	416 (1.8)	2 (0.0)		592 (1.4)	578 (2.1)	14 (0.1)
Comorbidity (CCIn), n (%)				<0.001				<0.001
0	21854 (56.8)	13732 (58.5)	8122 (54.3)		25675 (62.1)	16536 (61.2)	9139 (63.9)
1-2	7908 (20.6)	4451 (19.0)	3457 (23.1)		7208 (17.4)	4550 (16.8)	2658 (18.6)
3-4	5627 (14.6)	3480 (14.8)	2147 (14.3)		5652 (13.7)	3983 (14.7)	1669 (11.7)
≥5	3059 (8.0)	1823 (7.8)	1236 (8.3)		2790 (6.8)	1961 (7.3)	829 (5.8)
History of mental health conditionso(Yes)
Anxiety	9501 (24.7)	6767 (28.8)	2734 (18.3)	<0.001	14171 (34.3)	10644 (39.4)	3527 (24.7)	<0.001
Depression	10973 (28.5)	7568 (32.2)	3405 (22.8)	<0.001	13679 (33.1)	10362 (38.3)	3317 (23.2)	<0.001
ADD/ADHDp	1060 (2.8)	806 (3.4)	254 (1.7)	<0.001	1755 (4.2)	1367 (5.1)	388 (2.7)	<0.001
Bipolar	2701 (7.0)	2034 (8.7)	667 (4.5)	<0.001	3309 (8.0)	2518 (9.3)	791 (5.5)	<0.001
Schizophrenia/Psychotic	658 (1.7)	433 (1.8)	225 (1.5)	0.01	622 (1.5)	370 (1.4)	252 (1.8)	0.002
PTSDq	1291 (3.4)	993 (4.2)	298 (2.0)	<0.001	2370 (5.7)	1896 (7.0)	474 (3.3)	<0.001
Other	1427 (3.7)	1073 (4.6)	354 (2.4)	<0.001	2604 (6.3)	1992 (7.4)	612 (4.3)	<0.001
Number of mental health conditions				<0.001				<0.001
0	21796 (56.7)	12038 (51.3)	9758 (65.2)		19985 (48.4)	11185 (41.4)	8800 (61.6)
1	9156 (23.8)	5907 (25.2)	3249 (21.7)		10311 (25.0)	7298 (27.0)	3013 (21.1)
2	5016 (13.0)	3614 (15.4)	1402 (9.4)		6884 (16.7)	5276 (19.5)	1608 (11.2)
≥3	2480 (6.5)	1927 (8.2)	553 (3.7)		4145 (10.0)	3271 (12.1)	874 (6.1)
Index SUDsr(Yes)
Opioids	7469 (19.4)	5920 (25.2)	1549 (10.4)	<0.001	8478 (20.5)	6619 (24.5)	1859 (13.0)	<0.001
Alcohol	8338 (21.7)	5041 (21.5)	3297 (22.0)	0.19	9812 (23.7)	6322 (23.4)	3490 (24.4)	0.02
Tobacco	30027 (78.1)	17843 (76.0)	12184 (81.4)	<0.001	30540 (73.9)	19781 (73.2)	10759 (75.3)	<0.001
Cannabis	2991 (7.8)	1998 (8.5)	993 (6.6)	<0.001	3775 (9.1)	2689 (9.9)	1086 (7.6)	<0.001
Sedatives	802 (2.1)	660 (2.8)	142 (0.9)	<0.001	921 (2.2)	729 (2.7)	192 (1.3)	<0.001
Stimulants	3170 (8.2)	2228 (9.5)	942 (6.3)	<0.001	3706 (9.0)	2561 (9.5)	1145 (8.0)	<0.001
Hallucinogens	77 (0.2)	53 (0.2)	24 (0.2)	0.20	96 (0.2)	73 (0.3)	23 (0.2)	0.04
Inhalants	30 (0.1)	21 (0.1)	9 (0.1)	0.42	17 (0.0)	9 (0.0)	8 (0.1)	0.41
Psychotropic medications	4566 (11.9)	2991 (12.7)	1575 (10.5)	<0.001	4782 (11.6)	3292 (12.2)	1490 (10.4)	<0.001
Other SUD	17 (0.0)	16 (0.1)	1 (0.0)	0.01	17 (0.0)	17 (0.1)	0 (0.0)	0.01
Number of index SUDs, n (%)				<0.001				<0.001
1	25740 (66.9)	14961 (63.7)	10779 (72.0)		27980 (67.7)	17631 (65.2)	10349 (72.4)
2	8514 (22.1)	5468 (23.3)	3046 (20.4)		8652 (20.9)	5881 (21.8)	2771 (19.4)
3	2724 (7.1)	1909 (8.1)	815 (5.4)		2837 (6.9)	2083 (7.7)	754 (5.3)
≥4	1470 (3.8)	1148 (4.9)	322 (2.2)		1856 (4.5)	1435 (5.3)	421 (2.9)
Clinicians	6887	5475	1412		8145	6533	1612
Department, n (%)				<0.001				<0.001
Anesthesiology	42 (0.6)	40 (0.7)	2 (0.1)		59 (0.7)	51 (0.8)	8 (0.5)
Internal Medicine	1757 (25.5)	1354 (24.7)	403 (28.5)		1899 (23.3)	1445 (22.1)	454 (28.2)
Nurse practitioners	1321 (19.2)	1035 (18.9)	286 (20.3)		1910 (23.5)	1553 (23.8)	357 (22.2)
OB/GYNt	219 (3.2)	196 (3.6)	23 (1.6)		213 (2.6)	192 (2.9)	21 (1.3)
Primary Care	2092 (30.4)	1608 (29.4)	484 (34.3)		2497 (30.7)	1955 (29.9)	542 (33.6)
Psychiatry & Neurology	813 (11.8)	693 (12.7)	120 (8.5)		812 (10.0)	673 (10.3)	139 (8.6)
Rehabilitation	66 (1.00)	60 (1.1)	6 (0.4)		67 (0.8)	60 (0.9)	7 (0.4)
Other	577 (8.4)	489 (8.9)	88 (6.2)		688 (8.5)	604 (9.3)	84 (5.2)

^aPatients with qualifying first SUD diagnosis, with outpatient office encounter/outpatient SUD-related encounter/outpatient MSUD treatment (first date of which is index encounter) within one year of SUD diagnosis, being seen by qualifying provider (of given departments and having prescribed ≥ 1 MAT), ≥ 12 years old at index encounter, and having at least year of follow-up included into cohort.

^bIndex encounters between January 1, 2017 and January 1, 2019

^cIndex encounters between January 1, 2020 and January 1, 2022

^dPatients with majority of qualifying encounters with clinician having <20 % telehealth of all outpatient encounters.

^ePatients with majority of qualifying encounters with clinician having ≥ 20 % telehealth of all outpatient encounters.

^fChi-squared test (unless otherwise noted).

^g% out of respective period total (38,448 for Pre-COVID and 41,325 for COVID).

^hStandard deviation.

ⁱTwo-sample independent t-test (assuming unequal variances).

^jCounts may not add up to total due to removal of small missing values from certain variables.

^kColumn %.

^lAmerican Indian or Alaskan Native.

^mAsian, Native Hawaiian or Pacific Islander.

ⁿCharlson comorbidity index.

^oAny condition occurring prior to index encounter.

^pAttention deficit disorder/attention deficit hyperactivity disorder.

^qPost traumatic stress disorder.

^rDiagnoses occurring at index encounter (patients can have more than one).

^sClinicians can appear in both periods.

^tObstetrics and gynecology.

In the pre-COVID cohort, patients had a mean (standard deviation; SD) age of 48.24 (14.07). Patients were 50.5 % (19,409) female, 54.1 % (20,798) NH-White, 56.6 % (21,749) from the Western US and 39.3 % (15,112) of patients had Medicaid. There were also 8 % of patients having a CCI ≥ 5 and 43.3 % had ≥ 1 mental condition. All characteristics were significantly different between TH groups with the exception of hallucinogen and inhalant use disorders. In the COVID cohort, patients had a mean (SD) age of 48.01 (14.34). Patients were 50.2 % (20,759) female, 61.8 % (25,522) NH-White, 44.8 % (18,493) from the Western US and 38.7 % (15,983) of patients had Medicaid. There were 6.8 % of patients having a CCI ≥ 5 and 51.6 % had ≥ 1 mental condition. All characteristics were again significantly different between TH groups with the exception of age and inhalant use disorder.

There were 6887 clinicians in the pre-COVID period and 8145 clinicians in the COVID period, while noting that clinicians could appear in both periods. The pre-COVID cohort was comprised of 5475 low TH clinicians and 1412 high TH clinicians. The COVID cohort contained 6533 low TH clinicians and 1612 high TH clinicians. Most clinicians in this study specialized in primary care (comprising 30.4 % of the pre-COVID cohort and 30.7 % of the high COVID cohort), internal medicine (25.5 % pre-COVID, 23.3 % COVID), and nurse practitioners (19.2 % pre-COVID, 23.5 % COVID). There were significant differences in clinician departments between TH groups for both pre and post COVID cohorts (Table 1).

3.2. Inferential statistics

Table 2 showcases patient outcomes by clinician TH status stratified by pandemic period.

Table 2.

Patient outcomes by clinician telehealth status (stratified by pandemic period).

	Pre-COVID (n = 38,448)				COVID (n = 41,325)
Outcomea	Low Telehealth (n = 23,486)	High Telehealth (n = 14,962)	Modeled differencesb		Low Telehealth (n = 27,030)	High Telehealth (n = 14,295)	Modeled differencesb
Outpatient visits			IRRk(95 % CIl)	amIRR (95 % CI)			IRR (95 % CI)	aIRR (95 % CI)
Telehealth, n (%)c	186 (0.5)	1354 (2.1)	3.72 (2.82, 4.92)	2.18 (1.63, 2.91)	3380 (15.2)	30,392 (46.3)	6.75 (5.66, 8.05)	2.43 (2.20, 2.69)
MSUDd prescribed			ORn(95 % CI)	aOR (95 % CI)			OR (95 % CI)	aOR (95 % CI)
≥1 prescription, n (%)e	9922 (42.2)	2860 (19.1)	0.32 (0.31, 0.34)	0.79 (0.74, 0.84)	14,180 (52.5)	3847 (26.9)	0.33 (0.32, 0.35)	0.85 (0.80, 0.90)
MSUD days supply			${\mathit{e}}^{\hat{\mathit{\beta }}}$o(95 % CI)	${\mathit{e}}_{\mathit{A}\mathit{D}\mathit{J}}^{\hat{\mathit{\beta }}}$(95 % CI)			${\mathit{e}}^{\hat{\mathit{\beta }}}$(95 % CI)	${\mathit{e}}_{\mathit{A}\mathit{D}\mathit{J}}^{\hat{\mathit{\beta }}}$(95 % CI)
Days' supplyf, median (IQRg)	87.50 (148.00)	102.00 (140.07)	1.15 (1.07, 1.24)	1.24 (1.15, 1.33)	87.75 (138.00)	91.00 (137.47)	1.00 (0.94, 1.07)	1.09 (1.02, 1.16)
MSUD % change			$\hat{\mathit{\beta }}$p(95 % CI)	${\hat{\mathit{\beta }}}_{\mathit{A}\mathit{D}\mathit{J}}$(95 % CI)			$\hat{\mathit{\beta }}$p(95 % CI)	${\hat{\mathit{\beta }}}_{\mathit{A}\mathit{D}\mathit{J}}$(95 % CI)
Average monthly % change in prescriptionsh, mean (SD)	8.58 (35.87)	6.75 (33.60)	−0.01 (−0.03, 0.01)	0.00 (−0.02, 0.02)	7.13 (30.75)	7.69 (30.09)	0.00 (−0.01, 0.02)	0.01 (−0.01, 0.03)
Drug-related events, n (IRi)			IRR (95 % CI)	aIRR (95 % CI)			IRR (95 % CI)	aIRR (95 % CI)
SUD-related hospitalizations (per 100)	49,253 (209.71)	24,901 (166.43)	0.79 (0.78, 0.81)	0.88 (0.86, 0.89)	68,323 (252.77)	28,984 (202.76)	0.80 (0.79, 0.81)	0.92 (0.90, 0.93)
All-drug overdoses (per 1000)	863 (36.75)	345 (23.06)	0.75 (0.55, 1.04)	1.00 (0.75, 1.34)	984 (36.40)	328 (22.95)	0.57 (0.44, 0.75)	0.74 (0.56, 1.02)
Relapsesj (per 1000)	15,873 (675.85)	9968 (666.22)	0.99 (0.96, 1.01)	1.00 (0.97, 1.02)	21,792 (806.22)	10,256 (717.45)	0.89 (0.87, 0.91)	1.02 (0.99, 1.04)
Injection-related infections (1,000)	2514 (107.04)	1559 (104.20)	0.97 (0.91, 1.04)	0.99 (0.92, 1.05)	2979 (110.21)	1303 (91.15)	0.83 (0.77, 0.88)	1.02 (0.95, 1.11)
Mental health events, n (IR)			IRR (95 % CI)	aIRR (95 % CI)			IRR (95 % CI)	aIRR (95 % CI)
Mental health crises (per 1000)	2882 (122.71)	1304 (87.15)	0.71 (0.67, 0.76)	1.04 (0.97, 1.11)	2822 (104.40)	915 (64.01)	0.37 (0.35, 0.40)	1.08 (0.98, 1.17)

^aOccurring ≥ 1 day and up to 365 days after index encounter (index encounter defined as first: outpatient office encounter/outpatient SUD-related encounter/outpatient MSUD treatment; within one year after a qualifying SUD diagnosis).

^bDifferential change in outcome for those with high telehealth clinician compared to those with low telehealth clinician.

^cCount of telehealth outpatient encounters, % out of total outpatient encounters (by group).

^dMedication for substance use disorder (medications for opioid use disorder [MOUD], medications for alcohol use disorder [MAUD], medications for tobacco use disorder [MTUD]).

^eCount of patients prescribed ≥ 1 MSUD, % out of total patients (by group).

^fDifference in prescribed stop date and prescribed start date, among those prescribed MSUD.

^gInterquartile range.

^hAverage of monthly % changes of count of MSUD prescriptions, among those prescribed MSUD.

ⁱIncidence rate, sum of events divided by sum of total patients (by group).

^jCount of SUD-related events or all-drug overdoses occurring 30 days apart.

^kIncidence rate ratio.

^lWald confidence interval.

^mAdjusted (“a”) for patient age, gender, race/ethnicity, census region, urban status, metropolitan status, insurance, CCI, index SUD count, history of mental health conditions count, and clinician department; model standard errors clustered by clinician (models for all outcomes).

ⁿOdds ratio.

^oExponentiated beta-hat.

^pBeta-hat.

3.3. Outpatient TH visits

In the pre-COVID period, there were more TH outpatient visits among patients seeing high TH clinicians compared to low TH clinicians (2.1 % vs. 0.5 %; aIRR [95 % CI]: 2.18 [1.63, 2.91]). In the COVID period, TH visits increased in both groups, with greater visits again among those seeing high TH clinicians (46.3 % vs. 15.2 %; aIRR [95 % CI]: 2.43 [2.20, 2.69]).

3.4. MSUD prescribing

Across both pre-COVID and COVID periods, those visiting low TH clinicians had more prescriptions of MSUD than those visiting high TH clinicians (pre-COVID: 42.2 % low TH vs. 19.1 % high TH; COVID: 52.5 % low TH vs. 26.9 % high TH) yet differences in likelihood of prescription between TH groups became less pronounced in the COVID period (pre-COVID high TH vs. low TH aOR [95 % CI]: 0.79 [0.74, 0.84]; COVID high TH vs. low TH aOR [95 % CI]: 0.85 [0.80, 0.90]). Across both pre-COVID and COVID periods, those visiting high TH clinicians had longer days’ supply of MSUD than those visiting low TH clinicians (pre-COVID median [interquartile range; IQR] high TH vs. low TH: 102.00 [140.07] vs. 87.50 [148.00]; COVID median [IQR] high TH vs. low TH: 91.00 [137.47] vs. 87.75 [138.00]) yet again differences between TH groups became less pronounced in the COVID period (pre-COVID high TH vs. low TH $e_{ADJ}^{\hat{\beta }}$ [95 % CI]: 1.24 [1.15, 1.33]; COVID high TH vs. low TH $e_{ADJ}^{\hat{\beta }}$ [95 % CI]: 1.09 [1.02, 1.16]). In both periods, no differences were observed in average monthly % change in MSUD prescriptions between high and low TH groups.

3.5. Drug-related and mental health events

In both the pre-COVID and COVID periods, those visiting high TH clinicians had lower rates of SUD-related hospitalizations than those visiting low TH clinicians (pre-COVID aIRR [95 % CI]: 0.88 [0.86, 0.89]; COVID aIRR [95 % CI]: 0.92 [0.90, 0.93]). Yet in both periods, no differences were observed in all-drug overdoses, relapses, injection-related infections, and mental health crises between high and low TH groups.

3.6. Sensitivity analyses

When redefining high TH status by clinicians having ≥ 4 % of all outpatient encounters conducted via telehealth, results agreed with those of the main analysis (Supplemental Table 11). Also, when redefining TH status by the patient rather than the clinician, results again agreed with those of the main analysis with the exception of no difference in SUD-related hospitalization between high and low TH use during COVID (aIRR [95 % CI]: 0.99 [0.98, 1.01]; Supplemental Table 12). Finally, when redefining the COVID inclusion period, results again agreed with the main analysis (Supplemental Table 13).

4. Discussion

In this retrospective study, we examined two cohorts of patients with SUDs, prior to COVID and after COVID onset, while comparing outcomes between patients visiting high and low TH clinicians. As expected, we found that patients visiting high TH clinicians had higher rates of TH outpatient visits than those visiting low TH clinicians, with both groups having an influx in their TH visits during the COVID period. Additionally, we found that patients visiting high TH clinicians had lower MSUD prescribing, yet a higher MSUD days’ supply, than patients visiting low TH clinicians, and these differences reduced (not significantly) in the COVID period. Finally, high TH patients had lower SUD-related hospitalization than low TH patients, and we found no differences in all-drug overdoses, relapses, injection-related infections, and mental health crises between low and high TH groups across both periods. To our knowledge, this is the first study to assess such outcomes among a general SUD population from a large-scale repository of US health systems data.

Those visiting high TH clinicians had higher rates of TH visits than those visiting low TH clinicians and, additionally, both TH groups dramatically increased their number of TH outpatient visits from pre-COVID to the COVID period. This is understandable as the social distancing guidelines set during the COVID period called for the rapid scale-up of telehealth outpatient services for patients with SUDs (Carla King et al., 2022; Hailu et al., 2023; Yang et al., 2020).

We also found that the number of MSUD prescriptions among patients with SUDs was higher among the low TH group during both pandemic periods. Similar results were found in studies involving MOUD prescriptions (Hailu et al., 2023; Huskamp et al., 2022). However, when comparing the COVID period to the pre-COVID period, the difference between the number of MSUD prescriptions between low TH and high TH groups shrunk, albeit non-significantly. This suggests a shift in prescribing practices, with high TH clinicians potentially adjusting their treatment approach to provide longer-duration prescriptions while maintaining patient access to MSUD. Regulatory adjustments during the COVID-19 pandemic, such as policies that allowed for extended prescription durations to minimize in-person visits, may have influenced this trend. Additionally, differences in treatment strategies between high and low TH clinicians, such as varying levels of comfort with remote prescribing, could explain this pattern. Future research should explore whether telehealth-prescribing clinicians systematically differ in their approach to MSUD management and how these differences impact patient outcomes over time.

The unprecedented circumstances of the COVID-19 pandemic called for clinicians to prescribe more MOUD via telehealth to fit the social distancing guidelines best while effectively treating their patients (Huskamp et al., 2022). A national survey conducted electronically via WebMD's online clinician panel during the fall of 2020 showcased that clinicians varied considerably in using telehealth services to initiate MOUD (Huskamp et al., 2022). The results suggested that approximately 25 % of clinicians used telehealth services for MOUD initiations while 40 % used exclusively in-person visits (Huskamp et al., 2022). The majority of clinicians (55.8 %) expressed hesitation in using telehealth for treating new OUD patients, however, clinicians with more OUD patients were less likely to express discomfort (Dickson-Gomez et al., 2022; Huskamp et al., 2022; Jones et al., 2021). Studies have suggested that OUD care delivered via telehealth produced nearly identical outcomes to in-person care and demonstrated higher rates of retention in buprenorphine treatment (Hailu et al., 2023; Hammerslag et al., 2023; Vakkalanka et al., 2022; Zheng et al., 2017). This finding could warrant the permanent removal of the Ryan Haight Act which mandates clinicians to conduct an in-person visit before prescribing OUD medication (Huskamp et al., 2022). Currently, the suspension of the Ryan Haight Act was a temporary policy change intended to expand OUD treatment to adapt to social distancing guidelines (Huskamp et al., 2022). However, there remains uncertainty around the permanence of this policy removal (Huskamp et al., 2022). Future studies focusing on SUD overall as well as exploring additional specific groups can help elucidate these findings and provide greater justification for such long-term policy adjustments.

One of the key findings in this study is that patients with high TH clinicians had lower rates of SUD-related hospitalizations than patients with low TH clinicians. However, in a sensitivity analysis we found no difference between TH groups. This main analysis finding, of lower SUD-related hospitalization for high TH clinicians, could be due to patients with high TH clinicians being in a lower-risk population than patients with low TH clinicians or high TH clinicians serving a lower-risk population than low TH clinicians. Additionally, it may also be related to the comparative ease of scheduling more telehealth visits per patient than face-to-face visits, as patients may be more willing to follow up more frequently, and clinicians might schedule closer follow-ups due to the perceived convenience of telehealth. This is particularly relevant during the pandemic, where masking and other inconveniences made in-person visits more challenging. The increased risk of low TH patients could also be due to the lack of digital literacy and technology to engage in TH visits and other sociodemographic factors that hinder access to effective care. (Siwicki; Vaidya)

In addition, there was no difference between low and high TH cohorts in rates of all-drug overdoses, relapses, injection-related infections, and mental health crises during both pre-COVID and COVID periods. Current literature reinforces this finding showing that telehealth produces comparable results to in-person services including buprenorphine MAT treatment (Hailu et al., 2023; Hammerslag et al., 2023; Zheng et al., 2017), adverse drug-related events (Hailu et al., 2023), rehabilitation (Shigekawa et al., 2018), mental health treatment (Shigekawa et al., 2018), methadone psychotherapy (King et al., 2009; King et al., 2014), and cognitive behavioral smoking cessation (Carlson et al., 2012). Another study implies that although patients with OUDs have improved treatment retention when prescribed OUD medication via telehealth, there was no change in the odds of opioid-related non-fatal overdose (Hammerslag et al., 2023). It is also critical to note that the overall number of SUD-related hospitalizations and relapses rose significantly between the pre-COVID and COVID periods. The unprecedented nature of the pandemic left patients with SUDs experiencing additional stress caused by financial insecurities, social isolation, health anxiety, and uncertainty (Mehtani et al., 2021). These added stressors could contribute to the high rates of drug-related events outside of patients’ low and high TH clinician treatment. The social distancing guidelines made populations with SUDs more susceptible to the consumption of substances in isolation which escalated the risk of relapses (Edinoff et al., 2022).

4.1. Policy implications

The findings of this study have important implications for future telehealth policies, particularly in addressing ongoing regulatory uncertainties. One key area of focus is the Ryan Haight Act, which mandates in-person consultations for controlled substance prescriptions. Our results suggest that telehealth can be an effective and safe mode of delivering SUD treatment without increasing adverse outcomes, indicating the potential for revising or permanently removing restrictions like the Ryan Haight Act to allow more flexibility in prescribing medications via telehealth.

Additionally, our study highlights the necessity for healthcare policies to address telehealth reimbursement and prescribing rules post-pandemic. During COVID-19, temporary policy adjustments facilitated increased access to telehealth services, but there is a need for permanent reforms that ensure equitable reimbursement for telehealth services to sustain its integration into SUD treatment. Such changes would provide healthcare clinicians with the necessary support to continue offering remote care, ultimately expanding access for all patients.

Furthermore, our findings underscore the importance of telehealth in improving access to healthcare in underserved populations, particularly in rural areas where transportation and clinician shortages are significant barriers. By prioritizing policies that enhance digital infrastructure and reduce technological barriers, policymakers can promote greater health equity, ensuring that telehealth is not only available but accessible to all patients, regardless of geographic location or socioeconomic status.

4.2. Limitations

This study utilizes retrospective observational data which restricts our ability to prove causation. With this study using EHR data, results are contingent on the codes entered into systems as well as our classification definitions of conditions. Since we are dealing with prescriptions, not the administration of MSUD, we cannot speak to if the drugs were used when looking at prescribing outcomes. Although our sample was large and covered much of the United States, results were not nationally representative because only participating health systems contributed to our sample. Thus, our sample was fully contingent on participating health systems and the patient populations that contributed to it. Further, we did not distinguish between different methods of telehealth outpatient visits (ex: audio-only and video-only) which may have altered the effectiveness of treatment and outcomes. Another limitation of this study is the potential impact of confounding variables, such as differences in digital literacy and socioeconomic status among patients, which could affect telehealth utilization and outcomes. Patients with lower digital literacy or limited access to technology may have been less able to engage with telehealth services, introducing disparities in care. Furthermore, socioeconomic factors, including the affordability of internet access and devices, could have influenced the accessibility of telehealth. These limitations in telehealth technology, such as internet access disparities, may have affected the results and should be considered when interpreting the findings. Due to the coming and going of patients within EHR systems, we focused on the first SUD date and following patients for only one year, which restricted the analysis of patients uniquely into one pandemic period, rather than capturing the same patients in both periods to better assess time point differences on the same patients. Trying to focus only on patients with data in both periods would have dramatically reduced our sample sizes, and with rare SUD groups, we aimed to maximize our statistical power.

While our study examined the impact of telehealth on a broad cohort of patients with SUDs, we recognize that different types of SUDs may have unique treatment needs and responses to telehealth interventions. For example, prior research on OUD has demonstrated that telehealth-delivered buprenorphine treatment is associated with improved retention rates (Hailu et al., 2023; Hammerslag et al., 2023), whereas studies on AUD suggest that telehealth-based cognitive-behavioral therapy can be as effective as in-person counseling (Kiluk et al., 2016; Kiluk et al., 2019). However, less is known about the telehealth impact on stimulant or cannabis use disorders. Additionally, severity of SUD—ranging from mild to severe—may influence engagement with telehealth services and subsequent treatment outcomes. Given these potential variations, future studies should explore whether telehealth differentially affects patients based on their primary SUD diagnosis and severity, which could help refine telehealth interventions to better meet patient needs.

Notwithstanding these limitations, our study provides an initial estimate of the association of low and high TH status on health outcomes among patients receiving treatment for all SUD types among a large US health system database. Additionally, our study compares outcomes between pandemic periods, to assess the impact of COVID-19 onset on such outcomes.

5. Conclusion

In conclusion, we found evidence that telehealth integration into SUD treatment was associated with increased telehealth SUD-related outpatient visits and no differences in adverse outcomes when compared to standard in-person treatment. In addition, the reduced differences in MSUD prescriptions between low and high TH clinicians to patients during COVID could suggest that telehealth prescriptions may become a beneficial alternative option for patients with SUDs. Our study results suggest that telehealth could expand SUD treatment access to vulnerable populations and increase the utilization of SUD treatment overall. Future studies should be conducted on the utilization of telehealth treatment by individual SUD subgroups and the specific outcomes of each.

CRediT authorship contribution statement

Fares Qeadan: Writing – review & editing, Supervision, Resources, Project administration, Methodology, Investigation, Conceptualization. Sydney Shimizu: Writing – review & editing, Writing – original draft, Investigation. Benjamin Tingey: Writing – review & editing, Writing – original draft, Visualization, Validation, Methodology, Investigation, Formal analysis, Data curation. Philip J. Kroth: Writing – review & editing, Investigation. Talar Markossian: Writing – review & editing, Investigation.

Ethical statement

The institutional review board at Loyola University Chicago determined this study to be exempt from the Institutional Review Board (IRB) review.

Data sharing statement

The datasets generated during and/or analyzed during the current study are not publicly available due to restrictions by Oracle Cerner, the owner of the data. Data could be accessed by signing a data sharing agreement with Oracle Cerner and covering any costs that may be involved (Contact Kendra Stillwell: kendra.stillwell@cernerenviza.com).

Funding/support

None.

Declaration of competing interests

Authors have nothing to declare.

Appendix A. Supplementary data

The following is the Supplementary data to this article:

Data availability

The authors do not have permission to share data.