Methodological Innovations to Advance Substance Use Disorder Research: Proceedings of a NIDA Workshop on Target Trial Emulation and Translational Testing of Digital Health Tools

INTRODUCTION

Substance use disorder (SUD) is a complex chronic condition that requires a multi-disciplinary approach to both research and treatment. To address this condition, clinical decision-making must be informed by unbiased and rigorous science. Randomized controlled trials (RCTs) are the gold standard for evaluating interventions for SUDs. RCTs and hybrid implementation/effectiveness trials can establish causal relationships between an intervention and an outcome by limiting confounding and biases, but they face limitations, including extended timelines for producing evidence, high costs¹, resource-intensivity², lack of external validity due to recruitment of highly select patient populations, and impact on clinician and patients’ behaviors (e.g., Hawthorne effect)³. RCTs can also face challenges in rapidly generating evidence for urgent public health crises like the opioid overdose epidemic.⁴ The landscape of SUD prevention and treatment research stands primed for methodologies that can complement robust findings from RCTs and implementation-effectiveness trials and potentially address limitations of traditional research approaches.

Novel methodologies include target trial emulation (TTE) and translational testing of digital health tools. TTE is a methodological framework used to estimate causal effect by mimicking the design and analysis of a hypothetical (‘target’) RCT, using pre-existing observational data (e.g., electronic health records, claims data, or registries).^5-7 TTE is not a substitute for randomized trials, but rather a way to translate real-world data (RWD) from observational studies to elucidate further real-world evidence. The application of the TTE approach has historically focused on emulating trials of simple settings with a time-fixed treatment^8-12. Recently, this approach has been recommended for more explicit generalization to time-varying treatments and for studies where there might be misalignment of eligibility, start of treatment and start of follow-up,^{13, 14} that explore complex conditions like SUD to inform public health policy & practice. For example, an RCT was conducted to explore the comparative effectiveness of buprenorphine-naloxone versus extended-release naltrexone on treatment continuation (“treatment interruption”) of opioid use disorder (OUD) after medically-managed withdrawal.¹⁵ In the RCT, both treatment arms had similar risk of treatment interruption at 24 weeks. An emulation of that same trial using observational administrative data from the Massachusetts Public Health Data Warehouse (e.g., health records, claims) specified the same eligibility, interventions, follow-up time, and outcomes in real-world practice. In the emulation, treatment interruption in the real-world was higher than in the RCT, and patients on buprenorphine-naltrexone had a lower risk of treatment interruption compared to those on extended-release naltrexone.¹⁶ TTE has more recently been used to complement findings related to health interventions like COVID-19 vaccines¹⁷, and opioid medication management¹⁸. More broadly, fields such as nephrology, general surgery, cancer, and cardiovascular research have demonstrated the utility of TTE^19-21.

A variety of datasets are ripe for use with TTE, including those from surveillance systems, national health surveys, treatment systems, and administrative (e.g., claims) and mortality data. These large data sets can enable the selection of specific cohorts of individuals, facilitate the comparison of multiple active treatments, support the identification and adjustment for co-founding variables (e.g., baseline covariates such as comorbidities, demographic characteristics) and allow for the tracking of data over long periods of time. Applying this novel approach to study substance use – and its treatment – is particularly timely and significant considering the ongoing opioid, stimulant, and overdose crises in the United States (US).

There has been a steady proliferation of research and clinical demand for digital health’s integration in care models, including advances in tools for clinical settings to glean new insights into patient outcomes²², machine learning-based clinical decision support (CDS) systems integrated into electronic health records (EHRs)^{23, 24}, and wearable sensor technologies enabling real-time monitoring of physiological and behavioral markers associated with SUD²⁵. Harnessing the vast reservoirs of observational data available, these digital health advancements have laid a foundation to enhance the rigor and applicability of SUD research²⁵. In summary, there remains much opportunity for SUD researchers to gain new insights to patient experience and real-world outcomes through methodologies like TTE and translational testing of digital health tools to maximize the impact of SUD prevention and treatment efforts.

Workshop Overview

The National Drug Abuse Treatment Clinical Trials Network (CTN)²⁶ was established in 1999 and convenes experts from the National Institute on Drug Abuse (NIDA), addictions research, and community-based service providers across the US to drive innovation and translation in SUD treatments into real-world practice to improve patient outcomes²⁷. Recognizing the importance of these two areas of innovative methodology and their applicability to SUD research, the NIDA CTN conceived a workshop entitled “Target Trial Emulation in Observational Research and Translational Testing of Advanced Digital Health Tools for SUD Prevention and Treatment.” This half-day virtual workshop was hosted by the CTN in September 2022 and researchers from across NIDA and the CTN and its partners were invited to attend. The goal was to introduce these approaches, discuss their strengths, limitations, and potential, consider feasible research questions and study priorities, and explore actionable strategies to address barriers and challenges for advancing SUD research. The workshop brought together experts in epidemiology, health services research, and digital health to examine how large-scale observational data can improve causal inference and accelerate translation in SUD research (Figure 1). Sessions introduced principles and applications of TTE using real-world healthcare data (“TTE Using Real-World Healthcare Data”), addressed the use of national surveillance systems and administrative data available for TTE in SUD research (“Learning SUD Evidence from Nationwide Data Sources”), and highlighted the development and implementation of digital health tools—including machine learning-based risk prediction and CDS systems—for SUD prevention and treatment (“Translational Testing of Clinical- and Community-based Digital Health Systems”). After each session, a robust discussion was moderated by experts to enhance NIDA’s priority focus on expanding utilization of these novel methodologies. Across discussions, participants emphasized the need to integrate rigorous causal methods with scalable digital health approaches to enhance the public health impact of SUD research.

Fig. 1.

The insights derived from each session and accompanying discussion remain highly relevant. While advances have been made and new trials funded—including a CTN-funded study applying TTE methods (CTN-0153: Effects of Semaglutide and Tirzepatide on Incidence and Outcomes of Stimulant Use Disorders and OUD in Real-world Populations: TTE Using Patient EHR)—significant opportunities for methodological and translational advancement remain. The objective of this article is to synthesize the workshop presentations and discussions, delineate persistent research gaps and priority areas, examine key methodological and translational advances that have occurred since the workshop, and articulate future research directions to strengthen the evidence base, accelerate translation to practice, and improve public health outcomes related to SUDs.

TTE Using Real-World Healthcare Data

The goal of this session within the workshop was to review the principles and procedures of conducting a TTE, to explain how to evaluate the quality of TTE, to explore the feasibility of using RWD to emulate RCTs, and to review comparative effectiveness emulated trials of medications for OUD from an individually linked data system in Massachusetts.

Dr. Sara Lodi introduced TTE as a research method that attempts to use observational data to mimic, or emulate, a RCT. First introduced in 1986,¹⁴ the goal of using the TTE approach is to improve the quality of observational studies and to estimate the causal effect of interventions when it is not feasible to conduct a RCT.¹³ When developing a TTE, the first step is to define the causal question of interest and to specify the protocol of the target trial and to define the causal question of interest by focusing on key protocol elements (e.g., eligibility criteria, treatment strategies, treatment assignment, outcome timeline, and analysis plan). Then, using observational data, the researcher emulates each of the key protocol elements as closely as possible. Importantly, researchers should highlight where emulation is not possible and the possible resultant biases inherent to that limitation. RWD must be available for the key exposures and outcomes which are intended to be studied – for instance, TTE of a trial that seeks to use a laboratory test as an outcome that is not routinely implemented in clinical practice cannot be emulated using EHR data. Similarly, target trials must be pragmatic, unmasked trials comparing treatment strategies under the usual condition and visit schedule with which they would be applied in the EHR if implemented under routine clinical care. An important misconception of TTE is that the framework helps mimic randomization and reduces the concern of endogeneity in observational studies. Rather, with TTE the researcher should state how they will attempt to address confounding, but regardless of approach unmeasured residual confounding will always be a limitation when using observational data. One test to evaluate the robustness of controlling for confounding is to use a negative control outcome and rerun the analysis with an outcome expected not to be affected by the intervention. If an association between the intervention and this negative control outcome is observed, it suggests residual confounding may be present. Another critical component of conducting a TTE is replicating outcome definitions; clinical outcomes in RCTs are well defined and systematically ascertained, whereas outcomes in observational data are often less clearly specified or incompletely captured unless linking to additional data sources. For example, cause of death in an RCT is well documented, but typically not available in EHR records (though chart reviews or natural language processing can sometimes identify them). Once the data are collected for the emulation, analysis methods are employed depending on the type of intervention of the target trial.

Like RCTs and hybrid implementation effectiveness trials, the TTE process can be imprecise. Alongside compromise and adaptation of the dataset (e.g., creating variables from RWD that align with the RCT), expert knowledge is key for a meaningful specification of the target trial and a successful emulation, particularly when examining complex conditions - including SUD. It is important to have multidisciplinary teams, including clinicians, data scientists, database experts, and statisticians, who collaborate on an emulation. TTE is likewise useful for translating causal effects of RCTs from RWD to real-world evidence, providing a more rigorous framework for interpreting observational data and improving the validity of conclusions drawn from these studies in different populations.

Dr. Rolf Groenwold provided an overview of TTE using healthcare data and described how to understand when the evidence derived through TTE is convincing. The definition of RWD can vary widely across organizations.²⁸ One way of thinking about RWD is to think of it on a spectrum: on the left side data are derived from the highly controlled, traditional RCTs.²⁸ Move across the spectrum into viewable data (e.g., health surveys, post-authorization efficacy studies, observational studies, registries, post-authorization safety studies, and social media) until you reach the right side, in which data are routinely collected in everyday clinical practice (e.g., EHR, claims databases, patient charts). If the information on the right side of the spectrum is available for researchers (e.g., health records or claims databases), this provides the basis for an observational study which could be designed and analyzed according to the TTE framework. These RWD have become increasingly important because they can be utilized in areas where traditional RCTs face limitations, such as rare diseases or personalized medicine. Additionally, RWD can provide insights into treatments as they occur in real-world settings, rather than the highly controlled environments of RCTs.^29-32 In pharmacotherapy, RWD often satisfies post approval regulatory and safety monitoring requirements and may accelerate the approval process when leveraged.³³ Finally, TTE is valuable for a wide range of objectives, from post-market safety, expanded indication, effectiveness, or efficacy studies.^{34, 35}

Dr. Marc LaRochelle presented on an emulated trial that compared the effectiveness of buprenorphine (BUP) and extended-release naltrexone (XR-NTX) formulations on retention and opioid overdose outcomes after detoxification, called the COMParative Effectiveness Emulated Trials of Medication Treatment for OUD (COMPEET MOUD) study³⁶). The study leverages the Massachusetts Public Health Data warehouse—a large, linked statewide database—to compare buprenorphine and naltrexone formulations and their impact on retention and fatal and non-fatal opioid overdose after opioid detoxification. One emulated trial in COMPEET MOUD compared RWD from this warehouse to NIDA CTN-0051: Extended-Release Naltrexone vs. Buprenorphine for Opioid Treatment (X:BOT)¹⁵. The primary outcome of X:BOT was relapse-free survival, and the RCT found BUP superior to XR-NTX in the intention-to-treat analysis. However, a per protocol subgroup analysis of those that were successfully inducted onto treatment (94% for BUP compared with 72% for XR-NTX) identified similar results by formulation, suggesting that XR-NTX induction hurdle may be an important driver of differences observed in the intention-to-treat analysis. Real-world MOUD retention rates are often lower in observational data than those observed in RCTs^{15, 37, 38}. Considerations for future applications of TTE on MOUD trials abounded from this emulation. One of the key challenges encountered when harmonizing the X:BOT and COMPEET MOUD emulation protocols was identifying the relevant study population, as the inclusion and exclusion criteria differed between the trials. For example, the X:BOT trial included individuals who volunteered and were randomized to receive BUP or XR-NTX after detoxification. In contrast, the COMPEET MOUD analysis included all individuals discharged from opioid detoxification, regardless of whether they eventually received MOUD. Additionally, harmonizing exclusion criteria between the two trials presented difficulties, such as accounting for medical comorbidities, psychiatric conditions, and prior treatments like methadone.

Key Highlights of the Session Discussion

Although the TTE framework itself is not new, its use has expanded rapidly with the widespread availability of large observational data sources (e.g., EHR) in recent years, particularly for examining the effectiveness of treatment for SUDs. Like all observational approaches, TTE is only as strong as the data on which it is built, and careful attention must be paid to study design and analytic methods to ensure that causal questions are well defined and that potential sources of bias, including confounding, are adequately addressed. Despite these limitations, TTE provides a powerful and practical alternative when RCTs are infeasible. For example, RWD-based emulations can be used to make causal inferences when time is of the essence (e.g., treatment of rare yet fatal diseases) or to enhance RCT findings to inform treatment protocols by focusing on emerging interventions (i.e., new MOUD formulations) or less common outcomes (e.g., overdose) that are more difficult to power in a RCT. Importantly, TTE as a framework can help researchers minimize bias where possible and identify those that remain as limitations. Overall, employing the TTE framework to garner real-world evidence can be a more timely and less expensive alternative to a traditional RCT.

Learning SUD Evidence from Nationwide Data Sources

This session of the workshop provided an overview of surveillance, treatment system, administrative and mortality data available for population-level studies on SUD prevention, treatment, and policy and explored limitations for emulation trial consideration.

In this session, Dr. Christopher Jones, Dr. Beth Han, and Dr. Julie Donohue reviewed substance use-related datasets and surveillance systems, including those focused on youth surveillance, treatment system, administrative (e.g., emergency department, hospital discharge, and claims), and mortality data available for epidemiological studies on SUD prevention, treatment, and policy and explored limitations of each as novel sources of data for emulation.

Youth Surveillance:

The NIH-funded Monitoring the Future Survey (MTF)³⁹ and the CDC’s Youth Risk Behavior Surveillance System (YRBSS)⁴⁰ are informative youth-focused data systems capturing drug and alcohol use and related attitudes among school-aged adolescents. Limitations include the surveys’ self-report and cross-sectional designs, increasing the risk of recall bias and social-desirability bias, and absence of data from students who are not in school, which may lead to underestimation of substance use. The National Survey on Drug Use and Health (NSDUH)^{41, 42} has been used to examine national trends in mental health and substance use in adolescents and young adults⁴³, but it too is limited for emulation by its cross-sectional and self-reported design. Additionally, NSDUH may underestimate the prevalence of substance use, use disorders, and mental health problems because NSDUH excludes people who are unhoused not living in shelters and institutionalized populations (e.g., carceral populations) who often have more behavioral health problems than the general population.^{42, 44, 45} Finally, the NSDUH survey’s overall weighted response rate has been decreasing over the past two decades (2002: 71.3%; 2013: 60.2%; 2017: 50.4%; 2019: 45.8%; 2021: 10.3%; 2022: 12.1%).⁴⁶

Treatment System Data:

While several surveillance systems provide information on substance use treatment admissions and services, none capture the full universe of treatment providers in the US. Proprietary data systems including the National Addictions Vigilance Intervention and Prevention Program (NAVIPPRO)⁴⁷ and the Researched Abuse, Diversion and Addiction-Related Surveillance (RADARS) system⁴⁸ capture information on individuals being screened for or entering select substance use programs based on convenience samples of treatment facilities, but are not representative of the population. The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Treatment Episode Data Set (TEDS) captures records on substance use treatment admissions to and discharges from state-licensed or certified substance use treatment facilities in the US that receive federal public funding,⁴⁹ but the data do not represent individuals or prevalence estimates, only captures a maximum of three substances that led to the treatment episode, and comparisons across states are discouraged due to underlying differences in which facilities report data in each state. NSDUH, though a health survey, provides some mental health and substance use service utilization, as well as risk factors, sociodemographic and contextual variables, and examines national trends in the prevalence and treatment of depression⁴³, prescription opioid use and use disorder⁵⁰, use of medications for alcohol use disorder⁵¹, and methamphetamine use, methamphetamine use disorder, and associated overdose deaths among adults in the US⁵²; however, NSDUH’s design, underestimation, and decreasing response rates limit its relatability to public health needs in a specific community.

Administrative Data:

Multiple proprietary and government-run administrative datasets and surveillance systems capture information on emergency department visits and hospitalizations, many of which rely on medical claims using ICD-10 CM coding. CDC’s Drug Overdose Surveillance and Epidemiology (DOSE) system includes emergency department (ED) syndromic surveillance data and ED and inpatient hospitalization discharge data to capture information on nonfatal overdoses.⁵³ DOSE and DOSE ED provide complementary information to understand nonfatal drug overdose trends in the US based on syndromic queries and discharge billing codes, which are subject to coding and documentation errors. In addition, DOSE does not capture data from all EDs and is thus not considered a source of prevalence or burden of nonfatal overdose in the US. Other data sources discussed included Nationwide Emergency Department Sample (NEDS)⁵⁴ and National Inpatient Sample (NIS).⁵⁵

Comprehensive claims data systems have been used in a wide range of SUD studies and may be single payer and nationally-representative in the case of Medicare, or multi-payer but from a limited geographic area. Several states have All-Payer Claims Data systems that capture all but large self-insured employer payers. Medicare and Medicaid data are available from the Centers for Medicare and Medicaid Services via ResDAC, which provides extensive documentation on how to analyze and interpret the data.⁵⁶ The newly available Transformed Medicaid Statistical Information System (T-MSIS) Analytic Files (TAF)⁵⁷ is an important data source for SUD research given the disproportionate role Medicaid plays in financing SUD treatment. Many commercial claims data aggregators license data to researchers (HCCI, Marketscan, IQVIA, Truveta, and others); some of these data include commercially insured populations only while others include both private and public payers. Advantages of healthcare-related administrative claims data discussed include that they capture the full continuum of care (i.e., inpatient, residential, outpatient, pharmacy), and include heterogenous treatment providers (e.g., small primary care practices, large behavioral health systems). Claims data are longitudinal, allowing researchers to follow individuals across treatment episodes, and often contain more granular geographic identifiers than survey data (e.g., ZIP code) to permit SUD research on economic or environmental factors that may influence outcomes or service delivery. Disadvantages discussed in the Workshop included: limited sensitivity and specificity of diagnosis codes in claims, leading to biased estimates of prevalence and treatment particularly for marginalized populations that experience barriers to health care access; limited measurement of patient-centered outcomes, particularly person-centered recovery outcomes; missing information on social support; and inability to follow people who lose coverage (e.g., individuals for whom Medicaid coverage is suspended or terminated due to incarceration for substance use-related reasons) or who pay cash for certain services. A limitation of healthcare-related administrative data systems discussed was their high cost, which constrains their widespread use.

Mortality Data:

CDC’s National Vital Statistics System (NVSS)⁵⁸ represents the official source of deaths in the US. NVSS data are based on death certificates using ICD-10 coding and reported by vital statistics offices across the US. In addition, CDC’s State Unintentional Drug Overdose Reporting System (SUDORS)⁵⁹ provides comprehensive information on the characteristics and circumstances surrounding drug overdose deaths to inform prevention and response efforts. Mortality data depend on accurate coding and documentation by medical examiners, coroners, and other death certifiers, the veracity of which can vary significantly. Presenters also noted that mortality data are also dependent on varying toxicology testing practices, which may influence which drugs are determined to be a contributing cause of death.

Highlights of the Session Discussion

A variety of data sources are available for advanced methodological research on SUD treatment and outcomes, each with strengths and limitations. The systematic exclusion of incarcerated individuals across both survey and administrative data systems is a crucial gap, particularly given the high prevalence of SUD among that population. The multi-payer nature of the US health system also introduces additional limitations to administrative data systems that struggle to follow people as they transition from one source of insurance coverage to another (or to none). Given the impact of SUD on employment outcomes, and therefore insurance status, this is a significant limitation. In recent years, emerging research methods have been applied to help address the limitations of survey data through linkages to administrative data. For example, a recent study combined NSDUH and T-MSIS data to help adjust for the underestimated prevalence of SUD based on NSDUH data alone⁶⁰. Researchers have also linked NSDUH data and national mortality files at the county level to estimate the rates of deaths by suicide among adults who attempt suicide in the US⁶¹. In addition, researchers have linked Medicare and CDC National Death Index Data to examine fatal overdose among Medicare beneficiaries in the context of COVID-19-related policy changes⁶².

Some local and state agencies have undertaken novel data linkages that may help to address the limitations of any single data system on its own to generate new insights on SUD. For example, in 2015, Massachusetts invested in the ‘Chapter 55’ data system that linked more than a dozen types of information at the individual-level, including all-payer claims, emergency department and hospital discharges, birth and death records, substance use treatment, cancer registry, toxicology reports, prescription drug monitoring program, emergency medical services, and corrections.⁶³ These data have been used to generate more accurate estimates of SUD prevalence in Massachusetts and to examine health outcomes for individuals leaving carceral settings.⁶³

Translational Testing of Clinical and Community-Based Digital Health Systems

Advances in digital health tools for clinical settings and analytics to gather new insights into individuals’ health status and functioning are in demand for both researchers and clinical providers. Digital health approaches can expand the reach of evidence-based treatments in real-world settings and provide rich observational data from individuals’ daily lives. The goal of this section of the workshop was to focus on innovative translational testing of clinical and community-based digital health systems as advanced methodologies for enhancing gold-standard SUD research.

Analytic tools, including machine learning tools, are increasingly used to predict SUD-related outcomes and inform public health responses. For example, advances in machine learning are being applied to predict area-level fatal overdoses and/or community overdose risk to inform localized public health and policy efforts in the US. Dr. Brandon Marshall presented on a forecasting model to predict which neighborhoods are at high risk for future overdose death. These approaches are distinct from individual-level prediction tools because they aim to provide an assessment of either acute (e.g., overdose ‘outbreaks’) and/or longer-term community overdose susceptibility related to socioeconomic and/or structural disadvantage⁶⁴. The PROVIDENT study, led by Dr. Marshall and his research team, aimed to determine whether neighborhood-level predictions about where overdoses are most likely to occur could inform the prioritization of harm reduction efforts (e.g., street outreach, naloxone distribution) in communities at highest risk. They developed a spatial machine learning model to identify neighborhood-level predictors of fatal overdose at a census block group level (i.e., small areal units that contains between 600 and 3,000 residents) in the state of Rhode Island⁶⁵. Dr. Marshall and his team also created guidelines and principles for the equitable distribution of resources based on race/ethnicity and urbanicity, and demonstrated that focusing efforts in the top 20^th percentile of census block groups could potentially avert up to 40% of all overdoses.⁶⁶ A randomized, population-based, community intervention trial is currently ongoing to determine whether neighborhood-level overdose risk predictions could reduce the incidence of fatal and non-fatal overdoses compared to overdose prevention strategies relying on more traditional surveillance data.⁶⁷

Dr. Wei-Hsuan Lo-Ciganic presented on the development and implementation of a machine learning based opioid overdose risk prediction tool in an EHR. While various interventions have been implemented to reduce unsafe opioid prescribing and patient risk of opioid overdose, current approaches often target patients at lower risk and miss 70%-90% of opioid overdoses and OUD diagnoses^68-87. Previously developed risk prediction tools employed traditional statistical techniques to identify single overdose risk factors rather than predicting an individual’s overall risk based on the interaction of multiple factors. In contrast, machine learning analytics can predict an individual’s overdose risk using the complex interaction of factors from administrative or demographic data^88-90. Efforts are underway to develop and evaluate a machine-learning opioid prediction and risk stratification e-platform (DEMONSTRATE) in EHR systems in Florida. DEMONSTRATE refined and validated prediction algorithms to identify patients at high risk for opioid overdose and OUD using OneFlorida Data Trust (a Patient-Centered Clinical Research Network (“PCORnet”))⁹¹ and University of Florida (UF) Health Integrated Data Repository. The algorithms derived from structured EHR data achieved good predictive performance (e.g., C-statistics >0.80), and the addition of natural language processing–derived social and behavioral determinants of health and indicators of problematic opioid use from unstructured clinical notes further improved model performance⁹². DEMONSTRATE subsequently designed and prototyped a machine-learning driven CDS tool for opioid overdose risk prediction, which is currently being evaluated in a pilot clinical trial embedded in the EHRs of thirteen UF Health primary care clinicis in Gainesville.⁹³ User feedback indicated that clear statements of the tool’s real-world validation, along with transparent explanations of how the algorithm and risk scores were generated and their limitations were essential for establishing end-user trust. One of the main limitations was the quality and accuracy of EHR data, including missing quantity and dosing frequency fields in medication order data, which necessitated tailored analytic approaches for RWD. DEMONSTRATE also addressed key implementation challenges, including minimizing racial bias and improving fairness of risk prediction, mitigating unintended consequences (e.g., legal and ethical issues, prescriber dismissing patients from their practice), securing engagement and buy-in among the information technology teams and end users, and establishing feasible processes for regularly updating the algorithm. Together, DEMONSTRATE’s algorithm and CDS development efforts support the generation of an targeted alert that can improve patient care by providing clinically actionable information to augment prescribers’ professional judgment, while reducing the number of non-relevant alerts compared to current practice.

Dr. Walid Gellad presented on the implementation and evaluation of a machine-learning based opioid overdose risk prediction tool in both community and healthcare settings. In response to the opioid crisis, health systems and other partners have used multiple interventions to mitigate unsafe opioid prescribing practices to reduce patient overdose risk. However, there are two key challenges with this approach: 1) Existing risk prediction tools are limited in scope and accuracy, leading to burdensome interventions targeting an overly broad population or missing a significant number of high-risk individuals entirely^{94, 95}, and 2) Even when tools can more accurately identify high-risk patients, effective strategies to change clinician behavior remain limited^{96, 97}. To address these challenges, Dr. Gellad and his team combined a machine-learning risk prediction tool with primary care clinician-targeted behavioral nudges and rigorously tested them through their Machine-Learning Prediction and Reducing Overdoses with EHR Nudges (mPROVEN) RCT.⁹⁸ The team developed machine learning algorithms using a variety of datasets to more accurately identify patients at elevated risk of opioid overdose^{88, 99}. Several ongoing and recently completed studies¹⁰⁰ are applying behavioral economics principles to opioid prescribing, specifically using nudges in the EHR to change clinician behavior. mPROVEN’s trial intervention targets primary care clinicians in a large hospital system and combines a non-interruptive storyboard alert for PCPs identifying patients as being at elevated risk for overdose and four best practice alerts targeted towards patients at elevated risk based on the prediction algorithm: 1) active choice for naloxone prescribing; 2) accountable justification for high opioid dosage; 3) accountable justification for new prescription opioid start; and 4) accountable justification for overlapping benzodiazepine and opioid prescriptions. With mPROVEN, Dr. Gellad and his team aim to reduce opioid overdose risk by joining the machine-learning risk prediction tool with EHR behavioral nudges to improve clinician prescribing behavior. This study can serve as a model for how risk prediction and behavioral nudges can be used hand-in-hand in a scalable fashion and tested in large scale RCTs, setting the stage for other applications in substance use disorders and beyond.

In a separate project, working closely with the Allegheny County Department of Human Services, which maintains one of the nation’s most comprehensive integrated data warehouses, Dr. Gellad and colleagues developed a machine-learning model that uses healthcare, human services, and criminal legal data to identify individuals at highest risk of overdose upon release from jail⁸⁹. The algorithm predicts opioid overdose death within 90 days of release and stratifies individuals into risk-score decile subgroups. Individuals involved with the criminal legal system are especially vulnerable to overdose. For example, nearly 1 in 5 Allegheny County, Pennsylvania residents who died from an opioid overdose between January 2016 and June 2020 had a booking in the County Jail in the year prior to death¹⁰¹. A pilot intervention being planned in collaboration with Allegheny County would connect individuals at high-risk for overdose identified through the algorithm with harm reduction resources and services through an experienced local clinician and will assess facilitators and challenges of implementing the risk prediction tool in practice. The pilot project also utilizes a community advisory group, made up of 10-15 clients from the Allegheny Health Network with lived experience of both substance use and incarceration at the Allegheny County jail. These lived experience advisors provide feedback to the research team on topics such as what the potential value of being introduced to the overdose prevention program while in jail, how researchers need to talk about this risk prediction project, and what its components should be.

Dr. Majid Afshar presented on the use of a near real-time CDS system augmented with natural language processing for screening hospitalized patients with unhealthy substance use. Advancements in digital health include the incorporation of predictive analytics in EHRs using such methods as clinical NLP with deep learning¹⁰². Challenges arise from the free-text format of clinical narratives, hindering CDS. Large-scale clinical NLP focuses on retrospective research¹⁰³ leaving a scarcity of evidence for open-source NLP pipelines in health operations and standardized care. In addiction medicine, NLP and deep learning models can aid in screening patients. To overcome challenges related to opioid use screening and prioritization in hospitalized patients, Dr. Afshar and his team trained and validated an open-source convolutional neural network model for opioid use screening using medical concepts embedded in EHR notes collected during routine care¹⁰⁴. Investigators created an interoperable pipeline to integrate external artificial intelligence (AI) tools with a major EHR vendor, enabling complex data analytics using full EHR text in an elastic cloud computing environment with real-time decision support¹⁰⁵. This research employed a collaborative and evidence-based approach involving implementation scientists. User-centered design, human factors engineering, and engagement with decision-makers facilitated AI technology integration within the EHR. Limitations included calibration drift (when a once well-calibrated model or measurement system becomes misaligned over time, so its predictions or scores no longer match real-world outcomes as accurately) and cost effectiveness, but ongoing assessments and updates may ensure the effectiveness of these NLP-driven CDS tools. The versatile pipeline can be adapted for other addiction medicine CDS tools, offering AI support to combat opioid use and enhance health outcomes.

Dr. Rebecca Rossom presented the challenges and successes of using a web-based CDS to improve the detection and treatment of OUD in primary care. Primary care is the most common contact point for healthcare. National evidence-based treatment guidelines for SUD screening are relatively complex and not frequently conducted within primary care settings.^{106, 107} To improve the identification and treatment of OUD in primary care, Dr. Rossom and her team sought to integrate a CDS for OUD (created by a CTN working group¹⁰⁸) into an EHR system.¹⁰⁹ This work built on previously successful implementation of CDS systems for the treatment of diabetes and cardiovascular disease. Unlike these cluster-randomized CDS studies, this small 6-month pilot study was a clinician-randomized study. Instead of alerting rooming staff to print materials for patients and clinicians to review, clinicians directly received alerts in the EHR for patients at risk of an OUD diagnosis or overdose. Patients were flagged as at-risk if they had a diagnosis of OUD or three or more opioid prescriptions in the last year.

During the 6-month pilot intervention, clinicians opened the OUD-CDS tool for only 5% of eligible primary care visits¹⁰⁹. Despite low use, intervention clinicians reported increased confidence in screening for OUD compared to control clinicians. Non-buprenorphine-waivered intervention clinicians reported increased confidence in diagnosing OUD compared to control clinicians. Overall, 75% of MOUD prescribing primary care providers (PCPs) and 62% of non-MOUD prescribing PCPs were moderately or very likely to recommend the OUD-CDS tool. Rates of buprenorphine initiation for patients with OUD increased by 40% for MOUD prescribing intervention clinicians, with smaller increases for non-MOUD prescribing intervention clinicians (10%) and control clinicians (3%). Clinician-reported barriers to OUD-CDS use included not seeing the alert in the EHR, lacking adequate time to address OUD, and confidence they could determine which patients were at risk for OUD without the tool. Pilot study findings reinforced the importance of having the EHR alerts viewed by rooming staff, who could then print and distribute one-page summaries of the CDS to patients and clinicians. This increased uptake and supported the value of training to highlight tool functionality and address OUD stigma. Dr. Rossom and her team are now implementing these learnings in a multi-site cluster-randomized clinical trial.¹¹⁰

Highlights of the Session Discussion

The application of digital health tools and analytics in community and healthcare settings has tremendous potential to reduce the rate of overdoses within a specific region. Nonetheless, the panelists identified several limitations that reduce the utility of these tools. As with TTE, these models are only as accurate as the source data. Additionally, successful adoption of the CDS and other forecasting tools require provider buy-in. Providers need to understand how to use the tools and perceive clear value (e.g., timesaving, ease of use). Without such buy-in, these tools are unlikely to be used in practice. Despite these challenges, digital health tools hold substantial promise to identify at-risk individuals and communities, provide the opportunity for earlier intervention, and guide the allocation of life-saving resources more efficiently. These steps can have a meaningful impact on individuals and communities, particularly in the dynamic landscape of the opioid crisis.

Overall Discussion

The CTN workshop, Target Trial Emulation in Observational Research and Translational Testing of Advanced Digital Health Tools for Substance Use Disorder Prevention and Treatment highlighted advancements and challenges in leveraging TTE frameworks and digital health tools to address the SUD and overdose crises in the US. The aim of this effort was to launch and enhance dialogue on how these advanced methodologies can complement or expand on the work within the CTN and within NIDA broadly. SUD is a complex condition, and treatment and prevention efforts must be informed by an approach to data that are collected using rigorous scientific methods. Lessons learned across the workshop included: 1) the importance of utilizing TTE to enhance learning about not only the most timely and significant needs, but also the most impactful clinical research that can be conducted and disseminated with a focus on SUD prevention and treatment; 2) the benefits and limitations of nationwide data sources that can be considered for use in emulation; and 3) the emerging promise of digital health tools’ unique ability to enhance our understanding of overdose risk for both individuals and communities in a way that neither traditional research nor nationwide datasets can. Collaboration between clinicians, data scientists, statisticians, and public health officials was widely recognized as vital to successfully employing these approaches to complex conditions like SUD. Further, though not mentioned in the workshop, collaboration with those who have lived experience remains a missing but critical component. Though scoping review procedures are underway to map the involvement and engagement of patients and the public in TTE frameworks since 2011,¹¹¹ community-based participation is increasing as standard practice across SUD research,^{112, 113} and the advanced methodologies presented in this workshop will only benefit from development with the target population instead of for it.

While the TTE framework holds great potential for extracting causal insights from observational data, limitations render it a supportive tool to the RCT instead of a replacement.¹¹⁴ Since the workshop, TTE methodologies for SUD research have been increasingly utilized. Emulations with observational data have found evidence of potential treatment associations that may have taken years to develop through an RCT, including promising findings for adults with Type 2 Diabetes (those taking GLP-1 receptor agonist medications had fewer SUD-related hospitalizations than those taking oral gliptins;¹¹⁵ Semaglutide may lower risk of opioid overdose versus other antidiabetic medications (including other GLP-1RAs)¹¹⁶), Veterans (reducing depressive symptoms may also reduce pain, anxiety, and illicit opioid use,¹¹⁷) and patients with diagnosed OUD (MOUD provided alongside comprehensive syringe service programs may reduce HIV and HCV infection;¹¹⁸ starting buprenorphine may not be associated with a reduction in risk of self-harm, but reduced hazard for opioid-involved poisoning;¹¹⁹ halting illicit opioid use may also halt use of cannabis and cocaine, and may be associated with less pain and anxiety;¹²⁰ and weekly urine drug testing may slightly reduce the risk of MOUD discontinuation in the first year, but at a cost¹²¹). Further emulations currently under exploration include methadone and buprenorphine take-home dosing schedules versus continued daily dosing for optimal retention,^{122, 123} alternative dosing strategies’ risk of treatment discontinuation and all-cause mortality,^124-126 the clinical effectiveness of slow-release oral morphine versus methadone and buprenorphine/naloxone on retention, all-cause mortality, and OD-related acute care visitsw,¹²⁷ and perinatal and neonatal health outcomes of methadone compared to buprenorphine/naloxone during pregnancy.¹²⁸ Development of a new reporting standard, the TrAnsparent ReportinG of observational studies (TARGET)¹²⁹, is also gaining traction in the field to promote comprehensive reporting of these trials. CTN-0153,¹³⁰ an NIH HEAL Initiative funded study, will leverage TTE on national EHR data to assess the effectiveness of semaglutide and tirzepatide for treating stimulant use disorder and OUD. Moving the field forward requires the collaboration of multidisciplinary teams – including members of the target population – to ensure causal questions are well defined and address potential bias, enhancing sources of RWD to better utilize common data elements, and planning for TTEs to complement pivotal RCTs whenever possible.

As new surveillance and public health databases emerge nationwide, and systems are built to house multiple sources of data for a given community, the ability to leverage these data (and particularly their integration across data sets) presents a novel opportunity for substance use research.

As the proliferation of digitally captured data—EHRs, wearable sensor outputs, and NLP of clinical notes—continues, the vision for expanding research agendas must evolve accordingly. With existing repositories of RWD growing and new sources emerging, researchers and affected communities must remain actively engaged in discussions and consider how these advanced methodological approaches can be leveraged to enhance SUD research. As we move forward, incorporating diverse and real-time data streams and fostering interdisciplinary collaborations will be crucial for enabling more dynamic and responsive prevention approaches as well as treatment strategies for patients with SUD.