Abstract
Background:
People with opioid use disorder (OUD) often have a co-occurring psychiatric disorder, which elevates the risk of morbidity and mortality. Promising evidence supports the use of collaborative care for treating people with OUD in primary care. Whether collaborative care interventions that treat both OUD and psychiatric disorders will result in better outcomes is presently unknown.
Methods:
The Whole Health Study is a 3-arm randomized controlled trial designed to test collaborative care treatment for OUD and the psychiatric disorders that commonly accompany OUD. Approximately 1,200 primary care patients aged ≥18 years with OUD and depression, anxiety, or PTSD will be randomized to one of three conditions: (1) Augmented Usual Care, which consists of a primary care physician (PCP) waivered to prescribe buprenorphine and an addiction psychiatrist to consult on medication-assisted treatment; (2) Collaborative Care, which consists of a waivered PCP, a mental health care manager trained in psychosocial treatments for OUD and psychiatric disorders, and an addiction psychiatrist who provides consultation for OUD and mental health; or (3) Collaborative Care Plus, which consists of all the elements of the Collaborative Care arm plus a Certified Recovery Specialist to help with treatment engagement and retention. Primary outcomes are six-month rates of opioid use and six-month rates of remission of co-occurring psychiatric disorders.
Discussion:
The Whole Health Study will investigate whether collaborative care models that address OUD and co-occurring depression, anxiety, or PTSD will result in better patient outcomes. The results will inform clinical care delivery during the current opioid crisis.
Keywords: clinical protocol, collaborative care, mental health disorder, opioid use disorder, medication-assisted treatment, primary care, randomized controlled trial
1. Introduction
The widespread availability of prescription opioids, heroin, and, more recently, fentanyl has spurred one of the most tragic and consequential public health crises in American history [1–3]. For two decades, there have been sharp increases in the prevalence of opioid use disorder (OUD) and associated emergency room visits, hospital stays, and overdose deaths [4]. By 2018, nearly 450,000 people had died from overdoses involving an opioid, with 47,000 deaths in 2018 alone [5]. Mental health disorders are common among the estimated 2 million people in the United States living with OUD. In 2017, the co-occurrence of any mental illness was 64% and serious mental illness was 27% in those with OUD, with few receiving treatment [6]. Of those with OUD and a mental disorder, just 25% of those with any mental illness and 30% of those with a serious mental illness received treatment for both mental health and substance use [6].
The emerging standard of care for OUD is medication-assisted treatment (MAT), which combines medication (methadone, buprenorphine, or naltrexone) with behavioral therapies [7]. MAT results in better outcomes than either medication or behavioral therapies alone [8]. While there is strong evidence that it can significantly reduce risk for overdose or death, more than half of patients receiving MAT drop out of treatment [9–12]. One possible reason for treatment dropout is that MAT does not directly address psychiatric symptoms that may interfere with treatment [13]. Whether interventions designed to treat OUD must directly address psychiatric comorbidity to improve retention and effectiveness is presently unknown. In this study, we will investigate whether collaborative care models in primary care that address both mental illness and OUD result in better outcomes than those that address OUD alone. Primary care physicians (PCPs) in the United States can prescribe buprenorphine, but rarely do so, perhaps in part because they need additional support for patient treatment, monitoring and coordination [14]. In collaborative models of care, the treatment team includes a PCP who prescribes buprenorphine, a care manager, and a behavioral health consultant such as a psychiatrist who advises the team regarding patients who present diagnostic challenges or who are not adhering, tolerating, or responding to treatment [15]. Although collaborative care improves access, quality and outcomes in primary care patients with common mental health conditions, including addiction [16], collaborative models have not yet been tested for patients with OUD and the co-occurring mental health disorders that commonly accompany OUD.
Since poor treatment retention is relatively common in collaborative care models of OUD [17], new components and implementation strategies may be needed to address comorbid psychiatric and psychosocial needs that frequently accompany OUD and contribute to opioid relapse and patient dropout. An understudied addition to collaborative care are peers, trained laypeople in recovery who provide emotional and instrumental support to patients in OUD treatment [18–20]. The purpose of our study is to test the effectiveness of a multi-component collaborative care intervention for OUD and co-occurring mental health conditions and to examine the relative contribution of each component.
2. Methods
2.1. Study overview
The Whole Health Study will test the effects of collaborative care on treating OUD and comorbid mental health disorders in primary care. The study has two specific aims:
Aim 1. Conduct a randomized controlled trial (RCT) to compare (1) Augmented Usual Care, which consists of a primary care physician (PCP) waivered to prescribe buprenorphine and an addiction psychiatrist to consult on medication-assisted treatment; (2) Collaborative Care, which consists of a waivered PCP, a mental health care manager trained to treat both OUD and mental disorders, and an addiction psychiatrist who provides consultation for OUD and mental health; and (3) Collaborative Care Plus, which consists of all the elements of the Collaborative Care arm plus a Certified Recovery Specialist (CRS). A CRS is an individual in recovery (a peer) who uses their experience and training to provide emotional support to patients with OUD, motivate behavior change, and help support patient engagement and retention in treatment [18,19]. Participants will be randomly assigned to one of three conditions throughout the study period, with comprehensive in-person follow-up assessments at three and six months, and brief assessments by phone at one, two, four and five months post baseline. We hypothesize (1) that relative to patients in AUC, patients in CC will show improvements in six-month rates of opioid use and remission of the co-occurring psychiatric disorders, and (2) that the CC+ group will show greater improvements than the CC group.
We designed the study with three arms rather than two to isolate the most parsimonious set of elements of integrated collaborative care to maximize outcomes for individuals with OUD and psychiatric disorders. Our starting point was AUC, which is the current standard of MAT care in the primary care practices at the University of Pennsylvania. Our CC model builds on this foundation to provide additional patient support and clinical coordination to address the co-occurring burdens of mental health disorder and OUD. The CRS component in our CC+ model adds yet another layer of patient support in recognition of the very low completion rates that are common in OUD treatment programs. The RCT will provide an efficient test of the overall effectiveness of the three interventions as well as the unique contributions of the constituent components.
Aim 2. We will assess the costs of implementing and delivering collaborative care in each study arm and the changes in total healthcare costs associated with participants’ receipt of the intervention.
2.2. Setting
Participants will be enrolled from University of Pennsylvania Health System (UPHS) primary care practices that have or are considering implementing MAT services.
2.3. Target population
Our target population is primary care patients aged 18 years or older with OUD and depression, an anxiety disorder (panic disorder, social anxiety disorder, obsessive-compulsive disorder, or generalized anxiety disorder), or post-traumatic stress disorder (PTSD).
2.3.1. Inclusion criteria
Inclusion criteria are broad to increase the generalizability of the findings. Participants must: (1) be 18 years or older; (2) meet DSM-V criteria for OUD, or have taken medication for OUD, in the last 12 months; (3) agree to receive or continue medication for OUD at the primary care site, (4) have depression, an anxiety disorder (panic disorder, social anxiety disorder, obsessive-compulsive disorder, or generalized anxiety disorder), or PTSD; (5) be able to communicate in English; and (6) be willing to give informed consent.
2.3.2. Exclusion criteria
Patients will be excluded who are acutely suicidal; these patients will be connected immediately with clinicians for assessment and triaged to appropriate services. Patients with unmanaged mania or psychosis, confirmed through a diagnostic interview by a consulting psychiatrist, also will be excluded as these patients require more intensive treatment than is available through the collaborative care model. We also will exclude patients who do not have a phone. Phone calls are one of our primary methods for patient data collection, especially during the coronavirus pandemic. Our studies of alcohol use disorder treatment find that 90% of potential participants have phones [20,21].
2.4. Recruitment
2.4.1. Recruitment of UPHS primary care practices
We will recruit up to 39 geographically varied UPHS primary care practices ranging from urban sites in Philadelphia in the east to rural sites in Lancaster, PA, in the west, through suburban sites in PA and Princeton, NJ, in the north. Practices that currently have or are considering integrated behavioral health services will be invited to participate based on diversity of patient composition, community characteristics, and practice size, along with input from the UPHS leadership.
2.4.2. Patient recruitment
The primary source of recruitment will be clinician-directed referral at the primary care site supplemented by reviews of the UPHS electronic health record (EHR). Patient appointments with their PCP may be scheduled by the patient independently or as a referral for buprenorphine treatment from an emergency room visit or hospital stay, an inpatient behavioral health stay, or outpatient behavioral health clinics. UPHS also oversees Philadelphia’s paramedic system. We will work with staff in these units and paramedic system to increase awareness of the study and to encourage them to refer patients to a participating primary care site. At the medical visit, the PCP will identify potential participants by clinical assessment and chart review and ask if they are willing to receive information about the study. Interested patients will be referred to the research coordinator for screening and study intake either in-person or by phone. Positive screens for mental disorder will be based on validated measures of depression, anxiety, or PTSD. Due to the ongoing COVID-19 pandemic, research staff who interact with patients or colleagues are required to complete a daily symptom/exposure check and a weekly saliva-based COVID test through the University of Pennsylvania. They must also adhere to the same guidelines for personal protective equipment (PPE) and infection control practices as all clinical staff in the health system.
Potential participants will be told that we are asking them to participate because they are 18 years of age or older and have been diagnosed with OUD. We also will tell them that their PCP has referred them to the study and, if they are willing, they will be seen for a consent discussion and baseline visit at the participating practice, at our research offices, or during COVID-19 by telephone. In accordance with COVID-19 restrictions, only the first part of the baseline visit can be conducted in-person, with the latter half exclusively conducted by phone or a secure interactive video platform to minimize time in an enclosed space. Additionally, when conducting the baseline visit at a participating practice, only the research coordinator and CRS will be present from the study team. Both patients and research staff must wear PPE and practice physical distancing during all in-person encounters. All our staff has been vaccinated.
2.5. Patient Consent
The patient consent form includes information regarding participants’ rights under the Health Insurance Portability and Accountability Act. The informed consent form will be read aloud to each participant and they will be provided ample time to ask questions and discuss the study with family or friends. After consent, the research coordinator will conduct a baseline interview to establish eligibility. Patients will be referred back to their provider to schedule care or for referral. A list of available community resources will be provided to the patient.
2.6. Pilot testing
We will pilot the intervention at two UPHS clinics to refine the recruitment and assessment process, to assess the fit with clinic flow, and to evaluate the logistics, burden and timing of assessments.
We will recruit 10 patients who meet our eligibility criteria. We will then administer the consent to make sure that the language is clear to participants and will collect the baseline information. We will assess clinic flow and how to integrate the study procedures with clinic procedures. If necessary, we will pilot at additional sites. To be clear, the purpose of the pilot is not to refine the intervention itself or to preliminarily test its effectiveness; rather it is to ensure that the processes associated with patient flow and treatment delivery are appropriately operationalized and consistent with the practices’ existing operations.
2.7. Randomization
We will randomize patients to one of three conditions using a randomization code in the proportions 1:1:1 to AUC, CC, or CC+.
2.8. Treatment arms
2.8.1. Augmented Usual Care
In addition to the waivered PCP, the UPHS practices have access to an addiction psychiatrist who provides telephonic consultation to the PCP on issues related to OUD pharmacotherapy. The first-line pharmacotherapy is buprenorphine-naloxone. The second line pharmacotherapy is extended-release injectable naltrexone. AUC patients are routinely referred to community behavioral health services through the Penn Integrated Care Program for their mental health care. In UPHS primary care practices, AUC is the standard of care and thus will serve as our reference comparator group. The research team will not provide support to the PCP or practice staff in the AUC arm.
2.8.2. Collaborative Care
The Collaborative Care (CC) condition includes elements that will provide a greater level of patient support in treatment than AUC by providing integrated mental health and OUD care. First, an addiction psychiatrist with collaborative care expertise will provide the waivered PCP with telephonic treatment consultation and supervision for both addiction and psychiatric pharmacotherapy and behavioral treatment. Second, each patient will have a care manager – typically, a licensed clinical social worker – trained in evidence-based interventions for individuals with OUD and psychiatric disorders.
Whole Health Study research coordinators will refer patients to the care manager for an initial intake session. After reviewing the patient’s psychiatric history, medical history, and medication and illicit drug use, the care manager will work with the PCP to recommend treatment according to the Whole Health Study treatment guidelines [22]. The schedule of patient visits with the care manager will be standardized in the CC arm. The first visit will take place at baseline, then at home or office for induction when in moderate opiate withdrawal, then twice a week for two weeks with telephone calls between visits, then weekly, and when stable once a month. There will be a final visit at six months. Participants will be reminded by telephone about upcoming appointments with the care manager.
Prior to the start of the study, the care managers will have received 30 hours of clinical training in the Collaborative Care model and the selected evidence-based psychotherapies. The care managers will be supervised weekly by two psychologists with expertise in evidence-based SUD psychotherapy in addition to separate regular supervision by a consulting psychiatrist. Initially, the care managers will provide all patients with four sessions of motivational enhancement therapy (MET) [23] for substance use disorders [24] and behavioral activation (BA) for depression [25] when clinically indicated, all of which have demonstrated effectiveness for treating psychiatric disorders in primary care [26–29]. The primary target of MET is treatment engagement both for mental disorders and OUD. For patients with ongoing symptoms, the following additional treatments will be offered: (1) cognitive behavioral therapy (CBT) sessions focused on substance use when that is the primary concern (recommended for those with Brief Addiction Monitor–Intensive Outpatient Program [BAM-IOP] items 6-7 endorsed after the initial four MET sessions); (2) behavioral activation for those with primary depression symptoms (PHQ-9 total score ≥ 10 or PHQ-9 Item 1 ≥2); or (3) brief CBT-informed sessions for those with anxiety or more mild depression or substance use symptoms (BAM-IOP items 4-5 endorsed, GAD-7 ≥ 5, PHQ-9 = 5-9) [30,31]. Patients without significant continued substance use who have depression characterized by anhedonia will in most cases receive BA following MET, rather than CBT. We note that the evidence in support of CBT for treating OUD in conjunction with buprenorphine, while generally positive, is still quite limited [32,33].
Individuals requiring specialty care, such as exposure treatment for PTSD and obsessive-compulsive disorder, will be referred for that treatment outside of the primary care practice. All sessions will be delivered individually by the care manager. The treatment team will use measurement-guided care and treat-to-target practices, using validated measures of substance use, depression and anxiety. There will not be a specified number of sessions but it is anticipated that most patients will be seen for psychotherapy every 1-2 weeks and for a duration of 6 months or less. Due to COVID-19, all sessions are being conducted via telehealth at present until it is safe to resume in person sessions. Finally, the CC personnel will be trained to assist with scheduling, reminders and referrals.
A noteworthy feature of our collaborative care treatment model is the pragmatic blend of on-site direct care with video/telephonic supervision by the two addiction psychologists and the consulting psychiatrist. This organization of professional services, which allows each clinician to practice at the “top of their license,” elevates the overall standard of care by taking full advantage of the finite resource of expert time to supply guidance on complicated medical problems to a much larger panel of patients.
2.8.3. Collaborative Care Plus
In addition to the collaborative care model described above, patients in the Collaborative Care Plus (CC+) condition will receive services from a Certified Recovery Specialist (CRS). A CRS is a person in recovery who shares similar experiences to study participants and who provides non-clinical assistance to support patient engagement in treatment [18,19]. The assistance can take a variety of forms. The CRS may work with patients to help coordinate information with their providers and to make sure patient needs are met. The CRS may take participants to their PCP appointments and any other appointments that they may have to help them engage and stay in care to remain healthy. They may provide education to help participants define and work on their recovery goals. The CRS also may identify and facilitate linkages to community resources, including communities of recovery; educational, vocational, social, cultural, and spiritual resources; mutual self-help groups; and professional services. In short, the CRS will provide emotional, informational and instrumental support, helping participants identify and overcome the barriers to full engagement in their recovery.
Peers have been found to improve outcomes for many chronic medical conditions, including asthma, diabetes, HIV, hypertension, obesity, and tobacco dependence [34–38], and the early evidence of peer effectiveness in supporting OUD treatment is promising. Peer-support interventions have been associated with increased MAT engagement [39], opioid abstinence [39,40], and use of outpatient primary care services [39]. Peers interventions also have been associated with reductions in OUD-related hospitalizations [39,41], emergency department visits [39,42], and criminal justice system involvement [43]. The Whole Health Study is the first to investigate the effectiveness of peer support within a collaborative care model of treatment for OUD and co-morbid psychiatric illness. We hypothesize that adding a peer specialist to collaborative care will increase patient engagement and retention in care, and thereby improve treatment outcomes.
CRS certification in Pennsylvania requires a minimum of 54 hours of instruction on addiction and recovery management, education and advocacy, and ethics and confidentiality as well as successful completion of the Pennsylvania Certification Board CRS exam. We will supplement the CRS certification program with training to cover the role of the CRS in the Whole Health Study, collaborative care and the treatment model, creating a recovery plan, electronic medical records, documentation and communication, building a therapeutic relationship, and harm reduction practices in primary care. We also have a prepared a detailed CRS manual as a reference tool.
We have instituted precautions for patient-CRS interactions during the COVID-19 pandemic. Some aspects of CRS support that would normally happen in-person will occur via telephone or text messaging (i.e., coordinating information with providers, providing education, and linking to community resources). However, CRSs are still present in the clinics in person or allowed to meet in the community as long as appropriate physical distancing precautions and PPE are in place according to current guidelines.
The Whole Health Study RCT protocol is displayed in Figure 1.
2.9. Participant compensation
Participants will receive escalating payments for the in-person interviews to encourage retention. We will pay $50 to those who are eligible for enrollment and complete the baseline. (We will pay $25 to patients who start the baseline interview and are found to be ineligible for the study.) Compensation for the 3 month and 6 month interviews will be $60 and $70, respectively. For each of the monthly phone visits (months 1, 2, 4, and 5), participants will receive $20. If a participant completes all seven assessments they will receive a total of $260. Participants will be paid by Greenphire ClinCards.
2.10. Measures
2.10.1. Primary outcomes
The primary outcome measures are any opioid use in each month over the six-month follow-up and six-month rates of co-occurring psychiatric disorder. OUD diagnosis will be determined at baseline using the Diagnostic and Statistical Manual of Mental Disorders, 5th edition (DSM-5) criteria. Opioid use rates will be based on self-report assessments supplemented by a urine drug screen. The self-report assessment will indicate whether opioid use has occurred during the 30 days prior to each assessment. Urine toxicology screening obtained at every office visit will determine the type and amount of illicit drugs a patient has taken. If participants drop out of treatment but remain in the study, the research coordinator will arrange to collect urine samples at the primary care site or a research site, based on participant preference. Our urine collection procedure has been updated due to COVID-19-restrictions and now includes patient symptom screening 24 hours prior to office visits, patient temperature checks at the office, physical distancing and donning of personal protective equipment by patients and research staff, and enhanced cleaning and communication (to ensure partial capacity) processes.
We chose a dichotomous measure of any opioid use in each month over the six-month follow-up, which combines self-report and urine drug screen, as a primary outcome for two reasons. First, it is highly consistent with the goal of pharmacotherapy for OUD, which is to replace illicit opioid use. Second, it allows for more rigorous combining of self-report and urine drug screens to obtain a valid assessment of opioid use.
Each month of the follow-up, participants will be classified as:
no opioid use if they report no use and provide an opioid negative urine drug screen,
opioid use if they report use or provide a urine drug screen positive for opioid use, or
missing.
Similarly, we chose six-month rates of co-occurring psychiatric disorder as a primary outcome. Participants’ psychiatric disorders will be classified for each three-month period as:
in remission from their primary psychiatric disorder (depression, anxiety, or PTSD) if they are below the clinical threshold on the screening instrument for their primary disorder (5 on the Patient Health Questionnaire-9 [PHQ-9] for depression [44]; 5 on the General Anxiety Disorder-7 scale [GAD-7] for anxiety [45]; and 30 on the PTSD Checklist for DSM-5 [PCL-5] for PTSD) [46].
not in remission if they are above the cut off; or
missing.
The remission categories are intended to measure change in symptoms in response to treatment. A reduction in symptoms does not imply or suggest a “cure” or “complete remission” of the underlying disorder.
2.10.2. Secondary outcomes
Four secondary outcomes also will be examined: OUD medication adherence assessed through patient medical records and self-report; treatment retention measured as the time to dropout from both OUD and mental health treatment using clinic records; patient’s use of other illicit drugs, measured with a combination of monthly urine drug screen and self-report; and mortality assessed through death records. The adherence and retention outcomes will provide supporting evidence of the mechanisms that underlie the effects of the intervention on opioid use and psychiatric disorder, and the other drug use and mortality data will be used for evaluating additional outcomes of clinical and societal importance.
2.10.3. Covariates
We will use a set of covariates measured at baseline to evaluate the success – i.e., covariate balance across groups – of randomization to the AUC, CC and CC+ treatment conditions and to characterize the patient sample. These will include sociodemographic characteristics, years of prior opioid use, route of administration (e.g. IV vs. oral), use of prescription versus street opioids, years of alcohol and other drug use including tobacco, MAT history and overdose history. These variables were chosen as they may represent indicators of more severe OUD or other potential confounding conditions that should be balanced between groups.
At baseline, a two-item screen for psychosis and a four-item screen for mania will be used to identify more severe psychiatric disorders. The Columbia Suicide Severity Rating Scale (C-SSRS) will be used to assess suicide risk. The scale will be administered at baseline and at each follow-up. We will obtain specific information about various conditions such as Hepatitis C and insomnia. Other medical comorbidities will be measured using the Charlson Comorbidity Index [47] calculated from International Classification of Diseases (ICD) codes in medical records. We will assess for social determinants of health such as housing stability, transportation, food insecurity, employment and legal issues. We are including social determinants because we think they may mediate the effect of the CRS on outcomes, as the CRS will address these issues in the CC+ arm to help improve outcomes.
We also will collect baseline data on the primary care practices and clinician characteristics using electronic medical records and survey data. We will survey the clinicians in all treatment arms to ascertain clinical volume, years of MAT experience, organization support for the treatment program, and facilitators and barriers to implementation.
2.11. Planned statistical analysis
2.11.1. Descriptive statistics
Descriptive statistics will include means and standard deviations for continuous variables (treatment group differences analyzed using t-tests) and percentages for categorical variables (treatment group differences analyzed using chi-square).
2.11.2. Analyses for Primary Outcomes
The primary hypotheses are that six-month rates of opioid use will be lower, and six-month rates of remission from psychiatric disorders will be higher, in the CC and CC+ groups than in the AUC group, and in the CC+ group than in the CC group. Each primary hypothesis will be tested by comparing the groups on their repeated binary outcomes for use/no-use and remission/non-remission described above. We will use mixed effects logistic regression models [48] to estimate and compare the log-odds of use, or of remission, for each group across the 6-month period. The fixed effects (explanatory variables) will include the binary stratification variables, and a three-level categorical variable for intervention group (AUC, CC, CC+), and a four-level categorical variable for period. The random effects will include a random intercept for site, and random intercepts for patients within sites. We anticipate that these random intercepts, together with an autocorrelation structure for the residual covariances, will adequately model the within-site and within-participant correlation structure; as a check, we will select an appropriate structure based on information-criteria comparisons and graphical comparisons of empirical and model-based autocorrelation functions. We will test for group-by-time interaction effects. If we find that the group-by-time interaction terms are not statistically significant, then we will remove them from the model, and estimate and test the main effect of group, pooling over period; otherwise, we will report the period-specific results. We will check model adequacy and fit using standardized residuals, influence diagnostics, and graphical displays. We will calculate and report estimated probabilities of use for the groups, with their associated standard errors and confidence intervals.
For the analyses of the primary hypotheses described above, premature discontinuation from treatment and intermittent missing measures of opioid use or psychopathology will lead to incomplete data. The mixed effects models described above can make use of all available data provided by subjects, but the inferences drawn from them can, in principle, be invalid if the missing data are non-ignorable, or Missing Not At Random (MNAR) [49]. Since ignorability cannot be fully checked from available data [50], we will perform further analyses to assess the sensitivity of results to this assumption using (1) pattern-mixture models including an explanatory variable describing the missing data pattern (e.g. completer versus non-completer, or time to dropout) and its interaction with group. A significant interaction term suggests that the treatment comparisons differ across missing data pattern, which is one type of violation of MAR; (2) selection models, in which the dropout process is modeled using logistic regression models, with model specifications accommodating MAR and certain non-MAR models [49].
2.11.3. Power analysis for primary outcomes
To estimate power, we used simulation studies based on the bindata and gee packages in R. We assumed a 20% loss to follow up for each group; based on prior opiate and psychopathology studies, we anticipate a within-subject correlation of about 0.3 across time.
Our randomized sample will have 1,185 participants in total, evenly divided among the three conditions (395 per group). For a comparison of the usual care group to the combined collaborative care groups, this sample size provides 80% power to detect a 1.17 relative risk at α = .05, a 1.19 relative risk at α = .025, and a 1.21 relative risk at α = .0125. We also propose to test differences between the two collaborative care conditions (CC versus CC+). With the proposed sample, we have 80% power to detect a 1.22 relative risk at α = .05, a 1.24 relative risk at α = .025, and a 1.26 relative risk at α = .0125.
2.11.4. Analyses of Secondary Outcomes
Each participant’s monthly adherence to their OUD medication and their use of illicit drugs, will be assessed monthly, to create repeated binary adherence measures; the groups will be compared on these measures using mixed effects logistic regression models similar to those used for the primary hypotheses. Treatment retention will be analyzed as a time-to-event outcome, using Cox proportional hazards models to compare the groups, using random effects (frailties) to account for possible correlations among participants from the same site. The groups will be compared on mortality using logistic regression models.
2.12. Cost evaluation of the three models
We will assess costs to primary care practices of implementing and delivering CC and CC+, and changes in total healthcare costs associated with the implementation. The cost evaluation will make use of existing clinical and administrative data from electronic medical records and health care claims to determine patients’ ED, hospitalization, outpatient, and other healthcare service use. We also will collect data based on patient self-report on these major types of health care services received outside of the primary care office.
Implementation of collaborative care (CC or CC+) has cost implications for the primary care practice, including both fixed and variable costs. Fixed costs arise from staff time attending trainings and receiving regular implementation assistance, practice investment in technologies (e.g., building an EHR interface to assist with and track CC activities) and office changes to support the workflow of the CC team (e.g., additional office space for CRSs in CC+). These implementation activities are pre-determined and protocolized; intervention documentation will be used to obtain information to estimate fixed implementation costs. Variable costs are costs that vary with the number of patients receiving CC and include the costs of care manager time, consulting behavioral health specialist time, and CRS time (in CC+) spent on CC-related activities. As is typical in CC programs, care managers and CRSs contact patients in person and telephonically; they also spend time outside of direct patient care coordinating, charting/documenting, scheduling and conducting outreach, attending team meetings, and completing other care management activities. We will collect data on care manager time to estimate incremental care manager time/costs in the two CC conditions relative to the AUC. The AUC model involves psychiatrist time for MAT consultation only. The CC conditions may involve additional psychiatrist time for treatment consultation and supervision for both addiction and mental health issues. Electronic Health Records will be used to determine provider time spent on direct patient contacts and provider-provider coordination, as well as care manager/CRS time in direct patient contacts vs. other care management activities.
Public or private payers would want to know if and to what extent CC leads to cost-offsets as a result of reduced total health care costs. We will examine the frequency and costs of emergency department (ED) visits and hospitalizations related to overdoses and relapses and for all-causes that may be reduced by integrating treatment for OUD and mental health conditions. We also will estimate total healthcare costs for patients in all three study conditions.
2.13. Research ethics
The study was approved by the Institutional Review Board of the University of Pennsylvania, Philadelphia. It will be conducted in full accordance with all applicable University of Pennsylvania Research Policies and Procedures, good clinical practice standards and all applicable Federal and state laws and regulations.
3.0. Safety, monitoring and adverse events
We will use established University of Pennsylvania procedures and infrastructure for data and safety monitoring. During the course of the study, safety and data quality monitoring will be performed on an ongoing basis by the Principal Investigators and the study staff, who will report any adverse events, breaches of confidentiality, and unanticipated or unexpected problems to the University of Pennsylvania Institutional Review Board (IRB). Oversight also will be provided by a NIMH Data Safety Monitoring Board.
All patients will have comprehensive psychiatric screening prior to randomization and at each counseling session and assessment where evaluations for adverse events will be routine. A member of the research team will be available at all times to answer questions and assess possible adverse events. Participants will be withdrawn from the study if they show severe deterioration or if determined clinically necessary for other reasons.
4.0. Discussion
The Whole Health Study is designed to serve as a model for primary care practices treating OUD in the context of other mental health disorders. Our goal is to refine and rigorously test a collaborative care model for primary care patients with OUD and psychiatric comorbidity as primary care providers continue to grapple with the current opioid crisis. An integrated approach to OUD and other coexisting mental health disorders facilitates treatment for the patients most at risk for poor outcomes. Collaborative care models that meaningfully engage patients, address overall mental health, and provide peer support may result in better treatment engagement, retention, and outcomes.
The proposed study contains at least three important innovations. First, it would be one of the largest tests of collaborative care and mental health conditions to date, and the first to explicitly address the needs of comorbid individuals. Second, it would be the first to rigorously test the added value of peer support. Third, it would leverage the RCT to assess cost differences associated with alternative models of collaborative care relative to usual care in this population.
We are adopting a comprehensive set of guidelines for the management of OUD and comorbid mental health disorders into an integrated care intervention and applying them in the context of the collaborative care model in primary care. Primary care is where most adults receive their medical care, thus helping to streamline the treatment for substance use and other mental health disorders leading to important clinical and policy implications. Our study will provide definitive evidence regarding the clinical value and comprehensive costs of a collaborative care model, with and without peer support services, compared with usual care in improving outcomes for individuals with OUD and psychiatric disorders.
The limitations of our study require comment. First, there is the possibility that certain patient characteristics will be associated with patient outcomes. Covariates will therefore be used to assess the success of our randomization and to characterize the sample. Second, our results will be obtained from patients who receive care at primary care sites in the greater Philadelphia metropolitan area including sites in New Jersey and Central Pennsylvania. To enhance generalizability, we will include practices that vary in size and in community characteristics. Third, all methods for assessing opioid use have limitations. Opioid use rates will be based on self-report but also supplemented by a urine drug screen. A rigorous combining of self-report and urine drug screens will allow us to obtain a more valid assessment of opioid use. Fourth, we will need to monitor attrition closely over the 6-month follow-up period. Our protocols have been designed to minimize missing data and obtain follow-up research assessments on all patients initially enrolled in the study. Our primary analyses will regard missing data as ignorable, but we will perform sensitivity analyses to assess whether our inference is dependent on assumptions for missing data. Fifth, patients in the usual care group will not have the same number of in-person contacts as those in the collaborative care groups, and we are therefore unable to control for the effects of attention. Finally, our healthcare cost outcomes will be limited by the lack of information in claims data for health service use that occurred outside a health care encounter or a health care system. To address this concern, we will supplement our healthcare cost measures by collecting patient self-report data on healthcare utilization within and outside of the Penn health system.
In conclusion, the Whole Health Study has been carefully designed to refine and rigorously test a collaborative care model for patients with OUD and psychiatric disorders in primary care. In providing strong evidence regarding the most effective and parsimonious set of elements of integrated collaborative care required to maximize positive outcomes in primary care settings, the study will inform public health practice, clinical care provision, and policy formulation in the context of the current opioid crisis.
Acknowledgments
Funding Source
This study is funded through a grant from the National Institute of Mental Health to the University of Pennsylvania. Grant Number: UF1MH121944
Abbreviations:
- AUC
augmented usual care
- BA
behavioral activation
- BAM-IOP
Brief Addiction Monitor–Intensive Outpatient Program
- CBT
cognitive behavioral therapy
- CC
collaborative care
- CC+
collaborative care plus
- CRS
Certified Recovery Specialist
- C-SSRS
Columbia Suicide Severity Rating Scale
- DSM-5
Diagnostic and Statistical Manual of Mental Disorders, 5th edition
- ED
emergency department
- EHR
electronic health record
- GAD-7
General Anxiety Disorder-7 scale
- ICD
International Classification of Diseases
- IRB
Institutional Review Board
- MAR
missing at random
- MNAR
missing not at random
- MAT
medication-assisted treatment
- MET
motivational enhancement therapy
- OUD
opioid use disorder
- PCP
primary care physician
- PC-PTSD
Primary Care PTSD Screen
- PCL-5
Posttraumatic Stress Disorder Checklist
- PHQ-9
Patient Health Questionnaire-9
- PTSD
post-traumatic stress disorder
- RCT
randomized controlled trial
- UPHS
University of Pennsylvania Health System
Footnotes
Publisher's Disclaimer: This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Conflict of Interest
All investigators will follow the University of Pennsylvania Policy on Conflicts of Interest Related to Research.
AUTHOR DECLARATION
- We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.
- We confirm that the manuscript has been read and approved by all named authors and that there are no other persons who satisfied the criteria for authorship but are not listed. We further confirm that the order of authors listed in the manuscript has been approved by all of us.
- We confirm that neither the entire paper nor any of its content has been submitted, published, or accepted by another journal. The paper will not be submitted elsewhere if accepted for publication in the Journal.
- We confirm that we have given due consideration to the protection of intellectual property associated with this work and that there are no impediments to publication, including the timing of publication, with respect to intellectual property. In so doing we confirm that we have followed the regulations of our institutions concerning intellectual property.
- We confirm that any aspect of the work covered in this manuscript that has involved either experimental animals or human patients has been conducted with the ethical approval of all relevant bodies and that such approvals are acknowledged within the manuscript.
- We understand that the Corresponding Author is the sole contact for the Editorial process (including Editorial Manager and direct communications with the office). He/she is responsible for communicating with the other authors about progress, submissions of revisions and final approval of proofs.
Rebecca Arden Harris | 12/7/20 |
David S. Mandell | 12/8/20 |
Kyle M. Kampman | 12/7/20 |
Yuhua Bao | 12/8/20 |
Kristen Campbell | 12/8/20 |
Zuleyha Cidav | 12/7/20 |
Donna M. Coviello | 12/8/20 |
Rachel French | 12/7/20 |
Cecilia Livesey | 12/8/20 |
Margaret Lowenstein | 12/7/20 |
Kevin G. Lynch | 12/8/20 |
James R. McKay | 12/7/20 |
David W. Oslin | 12/7/20 |
Courtney Benjamin Wolk | 12/7/20 |
Hillary R. Bogner | 12/8/20 |
References
- [1].Pergolizzi JV, LeQuang JA, Taylor R, et al. Going beyond prescription pain relievers to understand the opioid epidemic: the role of illicit fentanyl, new psychoactive substances, and street heroin. Postgraduate Medicine. 2018;130:1–8. 10.1080/00325481.2018.1407618 [DOI] [PubMed] [Google Scholar]
- [2].Centers for Disease Control and Prevention. 2019 Annual Surveillance Report of Drug-Related Risks and Outcomes – United States Surveillance Special Report. Centers for Disease Control and Prevention, U.S. Department of Health and Human Services. Published November 1, 2019. Accessed 2/1/21 at https://www.cdc.gov/drugoverdose/pdf/pubs/2019-cdc-drug-surveillancereport.pdf. [Google Scholar]
- [3].Case A, Deaton A. Deaths of Despair and the Future of Capitalism. Princeton University Press, Princeton, NJ, 2020. [Google Scholar]
- [4].National Academies of Sciences, Engineering, and Medicine; Health and Medicine Division; Board on Health Sciences Policy; Committee on Pain Management and Regulatory Strategies to Address Prescription Opioid Abuse; Phillips JK, Ford MA, Bonnie RJ, editors. Pain Management and the Opioid Epidemic: Balancing Societal and Individual Benefits and Risks of Prescription Opioid Use. Washington (DC): National Academies Press (US); 2017. July 13. 4, Trends in Opioid Use, Harms, and Treatment. Accessed 2/1/21 at: https://www.ncbi.nlm.nih.gov/books/NBK458661/ [PubMed] [Google Scholar]
- [5].Hedegaard H, Miniño AM, Warner M. Drug overdose deaths in the United States, 1999–2018. NCHS Data Brief, no 356. Hyattsville, MD: National Center for Health Statistics. 2020. Data table accessed 2/1/21 at https://www.cdc.gov/nchs/data/databriefs/db356_tables-508.pdf#1 [Google Scholar]
- [6].Jones CM and McCance-Katz EF. Co-occurring substance use and mental disorders among adults with opioid use disorder. Drug and Alcohol Dependence. 2019; 197:78–82. 10.1016/j.drugalcdep.2018.12.030 [DOI] [PubMed] [Google Scholar]
- [7].National Academies of Sciences, Engineering, and Medicine. 2018. Medication-Assisted Treatment for Opioid Use Disorder: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press, 10.17226/25322. [DOI] [PubMed] [Google Scholar]
- [8].Dutra L, Stathopoulou G, Basden SL, et al. A Meta-analytic review of psychosocial interventions for substance use disorders. American Journal of Psychiatry. 2008; 165(2): 179–87. doi: 10.1176/appi.ajp.2007.06111851. [DOI] [PubMed] [Google Scholar]
- [9].Olfson M, Zhang V, Schoenbaum M, King M. Buprenorphine treatment by primary care providers, psychiatrists, addiction specialists, and others. Health Aff (Millwood). 2020;39(6):984–992. 10.1377/hlthaff.2019.01622 [DOI] [PMC free article] [PubMed] [Google Scholar]
- [10].Samples H, Williams AR, Olfson M, et al. Risk factors for discontinuation of buprenorphine treatment for opioid use disorders in a multi-state sample of Medicaid enrollees. J Subst Abuse Treat. 2018;95:9–17. 10.1016/j.jsat.2018.09.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- [11].Olfson M, Zhang VS, Schoenbaum M, et al. Trends in buprenorphine treatment in the United States, 2009-2018. JAMA. 2020;323(3):276–277. doi: 10.1001/jama.2019.18913 [DOI] [PMC free article] [PubMed] [Google Scholar]
- [12].Stahler GJ, Mennis J. Treatment outcome disparities for opioid users: Are there racial and ethnic differences in treatment completion across large US metropolitan areas? Drug Alcohol Depend. 2018;190:170–178. 10.1016/j.drugalcdep.2018.06.006 [DOI] [PubMed] [Google Scholar]
- [13].Substance Abuse and Mental Health Services Administration (SAMHSA). Strategies for developing treatment programs for people with co-occurring substance abuse and mental disorders. SAMHSA Publication No. 3782. Rockville, MD: SAMHSA, 2003. Accessed 2/1/21 at https://files.eric.ed.gov/fulltext/ED473993.pdf [Google Scholar]
- [14].Hooker SA, Sherman MD, Lonergan-Cullum M, et al. Mental Health and Psychosocial Needs of Patients Being Treated for Opioid Use Disorder in a Primary Care Residency Clinic. J Prim Care Community Health. 2020. Jan-Dec; 11:2150132720932017. doi: 10.1177/2150132720932017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [15].Lin EH, Von Korff M, Peterson D, et al. Population targeting and durability of multimorbidity collaborative care management. Am J Manag Care. 2014;20(11):887–995. [PMC free article] [PubMed] [Google Scholar]
- [16].Goodrich DE, Kilbourne AM, Nord KM, et al. Mental health collaborative care and its role in primary care settings. Curr Psychiatry Rep. 2013;15(8):383. doi: 10.1007/s11920-013-0383-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [17].Lagisetty P, Klasa K, Bush C, et al. Primary care models for treating opioid use disorders: What actually works? A systematic review. PLoS One. 2017;12(10):e0186315. ttps://doi.org/10.1371/journal.pone.0186315 [DOI] [PMC free article] [PubMed] [Google Scholar]
- [18].Reif S, Braude L, Lyman DR, et al. Peer recovery support for individuals with substance use disorders: assessing the evidence. Psychiatr Serv. 2014;65(7):853–861. doi: 10.1176/appi.ps.201400047. [DOI] [PubMed] [Google Scholar]
- [19].Bassuk EL, Hanson J, Greene RN, et al. (2016). “Peer-delivered recovery support services for addictions in the United States: A systematic review.” Journal of Substance Abuse Treatment. 2016;63:1–9. doi: 10.1016/j.jsat.2016.01.003. [DOI] [PubMed] [Google Scholar]
- [20].Muench F, van Stolk-Cooke K, Kuerbis A, et al. A randomized controlled pilot trial of different mobile messaging interventions for problem drinking compared to weekly drink tracking. PLoS One. 2017;12(2):e0167900. 10.1371/journal.pone.0167900 [DOI] [PMC free article] [PubMed] [Google Scholar]
- [21].McKay J, Gustafson D, Ivey M, et al. Effects of automated smartphone mobile recovery support and telephone continuing care in the treatment of alcohol use disorder: study protocol for a randomized controlled trial. Trials. 2018;19(82). 10.1186/s13063-018-2466-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- [22].DePhilippis D, Gibbons MB, McKay JR, et al. Collaborative care for individuals with opioid use disorders and co-occurring mental health conditions: Psychotherapy manual for care managers. Penn Center for Mental Health; 2020. [Google Scholar]
- [23].Lundahl B, Moleni T, Burke B, et al. Motivational interviewing in medical care settings: a systematic review and meta-analysis of randomized controlled trials. Patient Education and Counseling. 2013;93(2):157–168. 10.1016/j.pec.2013.07.012 [DOI] [PubMed] [Google Scholar]
- [24].Mignogna J, Hundt NE, Kauth MR, et al. Implementing brief cognitive behavioral therapy in primary care: A pilot study. Translational behavioral medicine. 2014;4(2):175–183. 10.1007/s13142-013-0248-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- [25].Martell CR, Dimidjian S, & Lewinsohn PM (2010). Behavioral activation therapy. In Kazantzis N, Reinecke MA, & Freeman A (Eds.), Cognitive and behavioral theories in clinical practice (p. 193–217). Guilford Press. [Google Scholar]
- [26].Funderburk JS, Shepardson RL, Wray J, et al. Behavioral medicine interventions for adult primary care settings: A review. Fam Syst Health. 2018. September;36(3):368–399. doi: 10.1037/fsh0000333. [DOI] [PubMed] [Google Scholar]
- [27].Tuccero D, Railey K, Briggs M, et al. Behavioral Health in Prevention and Chronic Illness Management: Motivational Interviewing. Prim Care. 2016;43(2):191–202. doi: 10.1016/j.pop.2016.01.006. [DOI] [PubMed] [Google Scholar]
- [28].Cuijpers P, Quero S, Dowrick C, et al. Psychological Treatment of Depression in Primary Care: Recent Developments. Curr Psychiatry Rep. 2019;21(12):129. doi: 10.1007/s11920-019-1117-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [29].Linde K, Sigterman K, Kriston L, et al. Effectiveness of psychological treatments for depressive disorders in primary care: systematic review and meta-analysis. Ann Fam Med. 2015;13(1):56–68. doi: 10.1370/afm.1719. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [30].Cacciola JS, Alterman Al, DePhilippis D, et al. Development and initial evaluation of the brief addiction monitor (BAM). Journal of Substance Abuse Treatment. 2013;44(3):256–263. doi: 10.1001/archinte.166.10.1092 [DOI] [PMC free article] [PubMed] [Google Scholar]
- [31].Oslin DW, Klaus J, Ingram E, et al. Foundations for Integrated Care: Behavioral Health Solutions for Primary Care. Vol 1-6. Philadelphia, PA: Philadelphia VA Medical Center; 2013. [Google Scholar]
- [32].Dugosh K, Abraham A, Seymour B, et al. A systematic review on the use of psychosocial interventions in conjunction with medications for the treatment of opioid addiction. J Addict Med. 2016;10(2):93–101. doi: 10.1097/ADM.0000000000000193 [DOI] [PMC free article] [PubMed] [Google Scholar]
- [33].Gregory VL Jr, Ellis RJB. Cognitive-behavioral therapy and buprenorphine for opioid use disorder: A systematic review and meta-analysis of randomized controlled trials. Am J Drug Alcohol Abuse. 2020;46(5):520–530. 10.1080/00952990.2020.1780602 [DOI] [PubMed] [Google Scholar]
- [34].Kim K, Choi JS, Choi E, et al. Effects of Community-Based Health Worker Interventions to Improve Chronic Disease Management and Care Among Vulnerable Populations: A Systematic Review. Am J Public Health. 2016;106(4):e3–e28. 10.2105/AJPH.2015.302987 [DOI] [PMC free article] [PubMed] [Google Scholar]
- [35].Kangovi S, Mitra N, Turr L, et al. A randomized controlled trial of a community health worker intervention in a population of patients with multiple chronic diseases: Study design and protocol. Contemp Clin Trials. 2017;53:115–121. 10.1016/j.cct.2016.12.009 [DOI] [PMC free article] [PubMed] [Google Scholar]
- [36].Kangovi S, Mitra N, Grande D, et al. Community Health Worker Support for Disadvantaged Patients with Multiple Chronic Diseases: A Randomized Clinical Trial. Am J Public Health. 2017;107(10):1660–1667. 10.2105/AJPH.2017.303985 [DOI] [PMC free article] [PubMed] [Google Scholar]
- [37].Weeks MR, Convey M, Dickson-Gomez J, et al. Changing drug users’ risk environments: peer health advocates as multi-level community change agents. Am J Community Psychol. 2009;43(3–4):330–344. DOI 10.1007/s10464-009-9234-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- [38].Gilmore B, McAuliffe E Effectiveness of community health workers delivering preventive interventions for maternal and child health in low- and middle-income countries: a systematic review. BMC Public Health. 2013; 13:847. 10.1186/1471-2458-13-847 [DOI] [PMC free article] [PubMed] [Google Scholar]
- [39].Magidson JF, Regan S, Powell E, et al. Peer recovery coaches in general medical settings: Changes in utilization, treatment engagement, and opioid use. J Subst Abuse Treat. 2021;122:108248. 10.1016/j.jsat.2020.108248. [DOI] [PubMed] [Google Scholar]
- [40].Bernstein J, Bernstein E, Tassiopoulos K, et al. Brief motivational intervention at a clinic visit reduces cocaine and heroine use. Drug Alcohol Depend. 2005;77:49–59. 10.1016/j.drugalcdep.2004.07.006 [DOI] [PubMed] [Google Scholar]
- [41].Min S-Y, Whitecraft J, Rothbard AB, et al. Peer support for persons with co-occurring disorders and community tenure: A survival analysis. Psychiatr Rehabil J. Winter 2007;30:2017–213. 10.2975/30.3.2007.207.213 [DOI] [PubMed] [Google Scholar]
- [42].Wakeman SE, Rigotti NA, Chang Y, et al. Effect of integrating substance use disorder into primary care on inpatient and emergency department utilization. J Gen Intern Med. 2019;34:871–877. 10.1007/s11606-018-4807-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- [43].Rowe M, Bellamy C, Baranoski M, et al. A peer-support, group intervention to reduce substance use and criminality among persons with severe mental illness. Psychiatr Serv. 2007;58:955–961. doi: 10.1176/ps.2007.58.7.955. [DOI] [PubMed] [Google Scholar]
- [44].Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606–613. 10.1046/j.1525-1497.2001.016009606.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- [45].Spitzer RL, Kroenke K, Williams JB, et al. A brief measure for assessing generalized anxiety disorder: the GAD-7. Archives of internal medicine. 2006;166(10):1092–1097. doi: 10.1001/archinte.166.10.1092 [DOI] [PubMed] [Google Scholar]
- [46].Blanchard EB, Jones-Alexander J, Buckley TC, et al. Psychometric properties of the PTSD Checklist (PCL). Behaviour research and therapy. 1996;34(8):669–673. 10.1016/0005-7967(96)00033-2 [DOI] [PubMed] [Google Scholar]
- [47].Charlson ME, Pompei P, Ales KL, et al. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. Journal of chronic diseases. 1987;40(5):373–383. 10.1016/0021-9681(87)90171-8 [DOI] [PubMed] [Google Scholar]
- [48].Agresti A (2002). Categorical Data Analysis. Wiley, second edition. [Google Scholar]
- [49].Fitzmaurice F, Laird N, and Ware J. Applied Longitudinal Analysis, 2nd Edition, John Wiley & Sons, 2011. [Google Scholar]
- [50].Xie H, Gao W, Xing B, et al. Measuring the Impact of Nonignorable Missingness Using the R Package isni. Comput Methods Programs Biomed. 2018. October;164:207–220. doi: 10.1016/j.cmpb.2018.06.014. [DOI] [PMC free article] [PubMed] [Google Scholar]