Effectiveness of policy and risk targeting for opioid-related risk mitigation: a randomised programme evaluation with stepped-wedge design

Abstract

Introduction

There is an epidemic of opioid use related to adverse events and deaths in the USA. The rates of chronic pain, mental illness and substance use disorder are higher at the Veterans Health Administration (VHA) compared with the general US population. The 2016 Comprehensive Addiction and Recovery Act requires the VHA to improve opioid therapy strategies in treating patients and to ensure responsible prescribing practices. The Stratification Tool for Opioid Risk Mitigation (STORM) is a web-based dashboard that prioritises review of VHA patients receiving opioids based on their risk. The VHA Partnered Evidence-based Policy Resource Center is coordinating a multiyear evaluation of STORM and aspects of the VHA policy that mandate case review of patients identified by STORM as very high risk.

Methods and analysis

This stepped-wedge cluster randomised controlled trial will test two hypotheses: (1) VHA medical centres randomised to facilitation for not meeting the targeted case review rate will achieve lower opioid-related serious adverse events (SAEs), relative to facilities not randomised to facilitation and (2) Patients whose cases are required to be reviewed will have a lower rate of opioid-related SAEs compared with comparable risk patients whose cases are not required to be reviewed. Patients who receive an opioid prescription at VHA medical centres will be followed for a minimum of 3 months after their first opioid prescription. Follow-up will continue until the last day of the project or death. The data will be analysed using an intention-to-treat approach with patient-month-level Cox proportional hazards models for both interventions.

Ethics and dissemination

Evaluation of the randomised roll-out was approved by the VA Boston Healthcare System Institutional Review Board (IRB) and Research & Development Committees (Protocol # 3069). Findings will be published in peer-reviewed journals and presentations at national conference meetings.

Trial registration number

ISRCTN16012111.

Strengths and limitations of this study.

• Randomised programme evaluation reflects Veterans Health Administration (VHA) commitment to rapid and rigorous evaluation of government programmes, an ambition promoted by the Office of Management and Budget.

• The stepped-wedge design evaluates aspects of the VHA policy and a web-based dashboard to identify risk factors and risk mitigation strategies for patients with an opioid prescription.

• This study will only include VHA patients and exclude patients with opioid use disorder.

Introduction

Opioid overdose deaths reached 33 000 in 2015, an increase of about 16% from the prior year, and are the leading cause of injury death in the USA.¹ The supply of opioid prescriptions remains high in the USA, with nearly 250 million opioid prescriptions written in 2013 or about one prescription per American adult.² These statistics underlie the Centers for Disease Control and Prevention’s characterisation of opioid use related to adverse events and deaths as an epidemic in the USA. The epidemic is potentially more acute in the Veterans Health Administration (VHA) patient population, which has higher rates of chronic pain, mental illness and substance use disorder compared with the general US population.^3–5 In particular, the prevalence of opioid use disorder in the VHA is approximately seven times higher than it is in commercial health plans.⁶

The epidemic in the USA in general, and in the VHA population in particular, has captured the attention of policy-makers. For instance, the 2016 Comprehensive Addiction and Recovery Act (Pub.L.No. 114–198; CARA) outlines a coordinated effort to confront opioid misuse and overuse through prevention, treatment, recovery, law enforcement, criminal justice reform and overdose reversal. In particular, CARA requires the VHA to improve opioid therapy strategies in treating patients, and to ensure responsible prescribing practices (Subtitle A Sec 911).

The VHA Office of Mental Health and Suicide Prevention (OMHSP; formerly Office of Mental Health Operations) developed a tool that is responsive to the CARA requirement that VHA opioid prescribers review existing adverse event risk characteristics for each patient before prescribing. The Stratification Tool for Opioid Risk Mitigation (STORM) is a web-based dashboard that prospectively prioritises review of VHA patients receiving opioids based on their risk for overdose-related, accident-related or suicide-related events (collectively, serious adverse events (SAEs)). The risk prioritisation is determined by a predictive model based on the association of patient characteristics (eg, age, race, prior history of mental illness) and opioid prescription with opioid-related SAEs.⁷ Designed to be easily incorporated into clinical practice, VHA clinicians can use STORM to identify risk factors and risk mitigation strategies potentially relevant for each patient.

Although STORM has gone through validation and usability testing, more evidence is needed to guide its use. Validation and usability reviews indicate that the STORM dashboard is an acceptable and efficient method of reviewing patient-specific risk information.⁷ User feedback indicated high face-validity for the patients STORM identifies as high risk and appropriate for intensive monitoring. It also indicated that the STORM dashboard can reduce the time required to review risk factors, assist with monitoring and systematic use of risk mitigation strategies, and improve awareness of the care patients are receiving across providers and care settings. However, the impact of identifying patient risk through STORM on opioid-related SAEs has not been rigorously evaluated. In addition, it is unclear how to best convey the CARA mandates to providers and ensure case review of patients identified by STORM.

Therefore, the VHA Partnered Evidence-based Policy Resource Center (PEPReC) is coordinating a multiyear evaluation of STORM and aspects of the VHA policy that mandate case review of patients identified by STORM as very high risk. In the following sections, we describe the STORM dashboard and the design of a cluster randomised trial to evaluate the effect of an expanded risk threshold and variations in policy language on time to opioid-related SAEs. This timely, randomised evaluation of STORM reflects VHA’s commitment to rapid and rigorous evaluation of government programmes, an ambition promoted by the Office of Management and Budget.⁸

STORM dashboard and implementation

On any given day, approximately 400 000–500 000 VHA patients have active prescriptions for opioids. Patients’ information will be displayed in the STORM dashboard until their prescription expires. For credentialed users (including VHA prescribers), the dashboard automatically sorts patients at their medical centres in descending order of predicted SAE risk. In this evaluation, we focus on patients prescribed an opioid who are in the top 5% of risk scores. Predicted risk is a function of demographics, comorbidities, prior history of mental illness and substance use disorders, and opioid prescription data. The dashboard also provides clinicians with a list of evidence-based clinical recommendations for risk mitigation, such as drug screening tests, bowel regimens and treatment alternatives to opioid prescription. Once a clinician reviews a case, the dashboard records and saves case review notes and dates of review. The dashboard compactly displays a patient’s name, age and gender, patient risk-level classification (low, medium, high or very high risk), diagnoses and medications that are relevant to opioid risk, and risk mitigation strategies and non-pharmacological pain treatment recommendations (figure 1). In addition, to facilitate care coordination, recent and upcoming appointments and patient care provider names are listed.

Figure 1.

Mock-up of Stratification Tool for Opioid Risk Mitigation (STORM) dashboard.

In the near future, VHA Central Office will release a policy notice mandating that VHA clinicians conduct case reviews and identify appropriate risk mitigation approaches for patients with opioid prescriptions who are identified by STORM as having a very high risk of SAEs. Differences in the key messages for the treatment and control groups are displayed in table 1.

Table 1.

The policy notice content for the treatment and control group

Policy notice content	Treatment	Control
Metric	Denominator: Patients with an opioid prescription who are in the ‘very high—opioid patients’ risk category in STORM for at least 7 days in the last quarter. Numerator: Patients in the denominator with case review within the last four quarters.
Monitoring	The STORM implementation team will review completion rates at the end of each quarter and notify facility point(s) of contact of their completion rate.
Implementation	Facilities with scores at or above 97% on this metric are considered fully implemented. Lack of implementation will trigger technical assistance and action planning starting in FY18Q4.	Facilities are expected to achieve scores at or above 97% by the end of FY18Q3.
Oversight and facilitation	If the facility fails to meet the targeted rate for completing case reviews by the end of FY18Q3, the STORM implementation team will notify the facility point(s) of contact. The goal of reviewing these patients will be added to the facility’s existing improvement goals. The facility point(s) of contact must then report quarterly on progress towards executing an action plan to meet the metric.	None

STORM, Stratification Tool for Opioid Risk Mitigation.

Randomised programme evaluation of STORM

Despite the benefits of randomised controlled trials, US healthcare policies and programmes are rarely tested with randomised designs.⁹ As a result, there is little evidence-based guidance for writing effective policy notices. The US Government Accountability Office has identified limitations in VHA policy notices, including a lack of clearly articulated accountability.¹⁰ Improving this aspect of VHA policy notices is a high priority. Therefore, two versions of the policy notice have been prepared; half of the medical centres will receive a version that states that if fewer than 97% of their cases are reviewed, facilitation, which includes technical assistance and action planning, will be provided to help them increase their case review rate; the other half will receive a notice that only states that case reviews are mandated. Sites that are required to develop action plans must: (1) add the metric (ie, >97% review of very high-risk patients) to their existing improvement goals and (2) submit quarterly reports detailing progress towards executing an action plan to meet the metric. To our knowledge, no prior study has compared the effects of alternative accountability approaches in policy documents on uptake of delivery system innovations.

In addition to randomising medical centres to different versions of the policy notice, we will rigorously evaluate the effect of the STORM dashboard on patient outcomes. To do this, we will use a randomised stepped-wedge design (described in detail below) to create two cohorts of patients in a similar risk group, one for which case review is required (treatment) and another for which it is not (control). This risk group will be created by expanding the threshold for very high risk from 1% to 5%.

Hypotheses

We will test two hypotheses: (1) VHA medical centres randomised to facilitation for not meeting the targeted case review rate will achieve lower opioid-related SAEs, relative to facilities not randomised to facilitation. (2) Patients whose cases are required to be reviewed will have a lower rate of opioid-related SAEs compared with comparable risk patients whose cases are not required to be reviewed.

Methods

Intervention 1: effectiveness of VHA policy

According to the policy notice, VHA medical centres are required to review the cases of very high-risk patients. Half of facilities (randomly assigned) will be asked to complete an action plan and receive additional oversight and facilitation from OMHSP if at least 97% of cases are not reviewed (the policy treatment group). Facilities that fail to meet the targeted rate for completing case reviews of very high-risk patients will be tasked to review these patients and report quarterly to the OMHSP on progress towards executing an action plan to meet the metric. The other half of VHA medical centres will receive a version of the notice without any mention of action plans, oversight or facilitation (the policy control group).

Intervention 2: effectiveness of STORM

Within the policy treatment and control groups, separately, the definition of very high-risk patients will be altered over time in a stepped-wedge manner. This will allow us to evaluate the effect of being targeted for case review by STORM. For the first 8 months, all medical centres will be required to conduct case reviews for patients in the top 1% of risk for an SAE. At baseline, patients with risk scores between 1% and 5% are not displayed in STORM (control group). At month 9, half of the policy treatment and half of the policy control facilities will be randomly assigned to review patients identified as high risk under an expanded risk threshold (up to 5%). At month 15, all facilities will be required to review patients in the top 5% of risk. This stepped-wedge design creates a cohort of patients who have opioid prescriptions and are between the top 1% and top 5% of SAE risk. Half of these patients will have mandated case review (the STORM treatment group) and half will receive usual care (the STORM control group). Figure 2 presents the two interventions and timeline.

Figure 2.

Project timeline. STORM, Stratification Tool for Opioid Risk Mitigation; VHA, Veterans Health Administration.

Randomisation and blinding

Randomisation was conducted in two steps, using permuted block randomisation. Permuted block randomisation allowed us to create groups with an even number of facilities. First, the 140 VHA medical centres were split into two groups with 70 medical centres each in the policy treatment and policy control groups. Then, to apply the stepped-wedge design for analysis of STORM treatment versus control, the 70 medical centres in each group were split into two groups of 35 hospitals using permuted block randomisation. The STORM dashboard will label patients as ‘very high risk’ using the respective risk score cut-offs (top 1% and top 5%) at each VHA medical centre. The risk scores will not be displayed, and providers will be blinded to changes in the risk score threshold that defines ‘very high risk’.

Recruitment/eligibility criteria/participant timeline

Our analytical cohort will include approximately 100 000 VHA patients with an opioid prescription in the top 10% of risk scores. Patients are eligible for inclusion in the study cohort for the first 18 months of the study. If a patient has an active opioid prescription on the day the policy notice is released and has a risk score in the top 10% of risk, he or she will automatically enter the study. Other patients will enter the cohort on the date of their first prescription that exceeds the 10% risk threshold. Patients will be followed for a minimum of 3 months after they are first prescribed an opioid. Follow-up will continue until the last day of the project (30 September 2019) or date of death. Our primary analyses of the effect of policy language will focus on patients in the top 1% of risk, and our primary analyses of the effect of STORM will be focused on patients between the top 5% and top 1% of risk scores. Over the course of this study, we anticipate over 50 000 patients will have risk scores in the top 5% of risk.

Outcome measures and control variables

Our primary outcome of interest is opioid-related SAEs (eg, opioid overdose, accidental falls, and possible and confirmed suicide attempts, etc (see online supplementary A for International Classification of Diseases (ICD)-9 and ICD-10 codes)). The outcome measure is censored by death or end of study. The STORM risk score that a patient receives when he or she first enters the study (baseline risk score) will be used as a control variable, since it reflects the probability that the patient will have an opioid-related SAE outcome. The risk score also captures the risk associated with general demographic characteristics (ie, age, race, gender) and comorbidities (ie, prior and current history of disease). In addition, facility indicators and time in study (indicators for current and past months) will be used as control variables.

Although case review is mandated, it is unlikely that all providers will review all identified cases. In addition, the risk mitigation strategies suggested in the STORM dashboard are optional. If lower SAE rates in treatment facilities are achieved, this could be due to higher case review rates or to greater use of risk mitigation strategies. That is, the case review rate acts as a mediator of the relationship between risk identification and opioid-related SAEs. In addition, risk mitigation strategies are intended to reduce risk of adverse outcomes. Thus, the SAE rates should be lower if risk mitigation strategies are more frequently implemented. We will test whether facility-level rates of case review and patient-level risk mitigation strategies implemented are mediators of the primary outcome.

Data collection and management

STORM risk scores are calculated and updated on a daily basis. On a patient’s entry into the study cohort (ie, at the date of the first opioid prescription on or after the release of the policy notice), his or her risk score will be recorded. VHA has a centralised corporate data warehouse (CDW) where all patient data, including demographics, appointments, visits, diagnoses and prescriptions are stored. From these data, any opioid-related SAEs for study patients will be identified. The case review notes and case review date, along with risk mitigation strategies implemented for each patient, will also be collected in CDW.

Sample size/power calculation

Sample size was calculated using the data that informed the original STORM model.⁷ That dataset included 1 135 600 patients with an opioid prescription from VHA anytime in 2010. The sample size calculation for the effectiveness of policy was completed using a baseline SAE rate of 0.029 per person-month for the policy control group, 140 medical centres with an average of 2112 patient-months and an intraclass correlation coefficient (ICC) of 0.01 in Stata’s clustersampsi function.¹¹ For the effectiveness of STORM, Stata’s stepped-wedge function was used to account for the changes in medical centres and patients included in the treatment group over time.¹² To calculate sample size, we used an expected baseline SAE rate of 0.01 per person-month for the STORM control group with an average of 352 patients per medical centre in the 1%–5% risk group, and we assumed an ICC of 0.01.

Based on these baseline rates, with an alpha of 0.05, we can detect a difference between the policy treatment and control groups of 28% (ie, an SAE rate difference at least as large as 0.037 or at least as small as 0.021) with 80% power. The evaluation can also detect a difference of 15% between the STORM treatment and control groups with 80% power (ie, an SAE rate difference at least as large as 0.012 or at least as small as 0.009).

Statistical analysis

The data will be analysed using an intention-to-treat approach with patient-month-level Cox proportional hazards models for both interventions: (1) effectiveness of policy and (2) effectiveness of STORM. For the effectiveness of policy analysis, a patient-level time-to-event Cox proportional hazards model will be used to evaluate the difference between facilities with and without facilitation language in the notice, controlling for the different targeted risk group at different times and facility fixed effects. Similarly, for the effectiveness of STORM analysis, the primary outcome will be modelled with a patient-level time-to-event Cox proportional hazards model to evaluate the difference between the STORM treatment and control groups, controlling for difference in policy and facility fixed effects. We will estimate the effect of the intervention during a single month as well as the cumulative effect of the intervention. In order to account for diminishing returns of additional months of exposure to time in STORM, we estimate separate effects for 1 month before treatment, 2 months before treatment and so on.

Equations 1 and 2 represent the planned analyses for effectiveness of policy and effectiveness of STORM, respectively.

$${\displaystyle \begin{array}{rl}\text{(1)}& Outcom{e}_{itk}=& intercept+{\propto }_i+{\beta }_t+P_i\gamma +x_kɸ+{\epsilon }_{itk}Outcom{e}_{itk}=& intercept+{\propto }_i+{\beta }_t+P_i\gamma +R_{it}{\theta }_t\\ \text{(2)}& & +R_{it-1}{\theta }_{-1}+R_{it-2}{\theta }_{-2}+\dots +R_{it-n}{\theta }_{-n}& +x_kɸ+{\epsilon }_{itk}\end{array}}$$

In these equations, ${\displaystyle i}$ represents medical centres, ${\displaystyle t}$ is time points (ie, months), ${\displaystyle n}$ is months before month ${\displaystyle t}$, and ${\displaystyle k}$ is individuals. In addition, ${\displaystyle \propto }$ is a random medical centre effect, ${\displaystyle \beta }$ is a vector of fixed time effects, ${\displaystyle P}$ is a policy indicator (${\displaystyle P}$=1 if policy treatment medical centre, 0 if policy control medical centre), ${\displaystyle \gamma }$ is a fixed effect for policy, ${\displaystyle R}$ is a risk targeting indicator (1 if patient in the 1%-5% risk stratum in the treatment medical centre ${\displaystyle i}$ at time ${\displaystyle t}$, 0 otherwise), ${\displaystyle {\theta }_t}$ is a fixed risk targeting effect at time ${\displaystyle t}$, ${\displaystyle {\theta }_{-n}}$ represents lagged risk targeting effects from targeting at time ${\displaystyle t-n}$, ${\displaystyle x_k}$ represents baseline covariates, represents fixed effects for baseline covariates and ${\displaystyle \epsilon }$ is residual error. In equation 1, we are interested in estimates of ${\displaystyle \gamma }$, the policy treatment effect. In equation 2, we are interested in estimates of ${\displaystyle \theta }$, the risk targeting effect at month ${\displaystyle t}$, as well as in estimates of ${\displaystyle \theta +{\theta }_{-1}+{\theta }_{-n}}$, the cumulative effect of months of risk targeting experienced prior to and including month ${\displaystyle t}$.

A statistically significant difference between the two policy groups (γ) suggests that the threat of facilitation modifies VHA providers’ behaviour to increase surveillance on very high-risk opioid-prescribed patients and to apply SAE risk mitigation strategies. A statistically significant effect of the STORM treatment group (θ) indicates that when opioid-prescribed patients are required to be case reviewed, they are less likely to experience opioid-related SAEs.

Our intention-to-treat analysis assumes that patients’ risk scores are relatively stable over the study period, and that baseline risk closely approximates the risk level of a patient at the time their facility is randomised to target an expanded risk threshold. However, it is possible that risk scores may change over time for patients who receive long-term opioid therapy and/or frequent short-term opioid therapy. Risk scores at baseline may simultaneously predict prospective risk scores, likelihood of exposure to the intervention and risk of SAEs. In addition, a patient’s risk score at a given month may be affected by the version of policy or level of risk threshold in place at a facility in previous months. To account for this potential endogeneity, we plan to conduct a secondary survival analysis that treats the facility treatment indicator and interaction between baseline risk score and facility treatment as instrumental variables.

We will conduct a sensitivity analysis to evaluate the effect of facilitation during the study period on the policy treatment group. The time when each medical centre is notified that they failed to meet the targeted case review rate will be tracked and the cohort will be stratified into three groups: policy control, policy treatment with facilitation and policy treatment without facilitation. A statistically significant effect of treatment with facilitation compared with treatment without facilitation indicates facilitation may lead to greater reduction in patients’ risk of opioid-related SAEs. We also will evaluate whether the effect of the threat of but not actual facilitation is associated with reduced risk of opioid-related SAEs.

Patient and public involvement

Patients and public were not involved in the development, design, recruitment and randomisation of this study.

Ethics and dissemination

Randomised roll-out of STORM to medical centres is occurring as part of the OMHSP’s activities and does not require IRB approval. This trial has been registered at ISRCTN (http://www.isrctn.com/ISRCTN16012111). In addition, our partner at the VHA Center for Health Equity Research and Promotion is conducting a complementary evaluation to identify strategies used to implement STORM across the two policy groups as well as barriers and facilitators to STORM implementation (https://www.hsrd.research.va.gov/research/abstracts.cfm?Project_ID=2141704557).

PEPReC’s protocol has been presented at the 2017 AcademyHealth National Health Policy Conference and a VHA cyberseminar. We are submitting abstracts about this protocol and randomised programme evaluations to other national conferences. Once the study is completed, the following two papers will be prepared and submitted to peer-reviewed journals; (1) Reduction of opioid-related SAEs in VHA medical centres with and without facilitation and (2) Effect of identification of high-risk patients via the STORM dashboard on opioid-related SAEs. Beyond providing rigorous evidence of the impact of STORM on patient outcomes, this study will provide insight to OMHSP and VHA leadership about how to optimise the STORM dashboard to reduce SAEs among high-risk patients.