Maximizing Effectiveness Trials in PTSD and SUD Through Secondary Analysis: Benefits and Limitations Using the National Institute on Drug Abuse Clinical Trials Network “Women and Trauma” Study as a Case Example

Abstract

Recent federal legislation and a renewed focus on integrative care models underscore the need for economical, effective, and science-based behavioral health care treatment. As such, maximizing the impact and reach of treatment research is of great concern. Behavioral health issues, including the frequent co-occurrence of substance use disorders (SUD) and posttraumatic stress disorder (PTSD), are often complex, with a myriad of factors contributing to the success of interventions. Although treatment guides for comorbid SUD/PTSD exist, most patients continue to suffer symptoms following the prescribed treatment course. Further, the study of efficacious treatments has been hampered by methodological challenges (e.g., overreliance on “superiority” designs (i.e., designs structured to test whether or not one treatment statistically surpasses another in terms of effect sizes) and short term interventions). Secondary analyses of randomized controlled clinical trials offer potential benefits to enhance understanding of findings and increase the personalization of treatment. This paper offers a description of the limits of randomized controlled trials as related to SUD/PTSD populations, highlights the benefits and potential pitfalls of secondary analytic techniques, and uses a case example of one of the largest effectiveness trials of behavioral treatment for co-occurring SUD/PTSD conducted within the National Drug Abuse Treatment Clinical Trials Network (NIDA CTN) and producing 19 publications. The paper concludes with implications of this secondary analytic approach to improve addiction researchers’ ability to identify best practices for community-based treatment of these disorders. Innovative methods are needed to maximize the benefits of clinical studies and better support SUD/PTSD treatment options for both specialty and non-specialty healthcare settings. Moving forward, planning for and description of secondary analyses in randomized trials should be given equal consideration and care to the primary outcome analysis.

1. Introduction

The behavioral health treatment landscape is in the midst of rapid change. Implementation of the Patient Protection and Affordable Care Act (ACA) (US GPO, 2010), along with the Mental Health Parity and Addiction Equity Act (US GPO, 2008), re-focused attention on the integration of behavioral health into non-specialty settings, such as primary care. Integration of care aims to improve the accessibility and acceptability of behavioral health care, while managing the high costs associated with mental health and addiction problems. As national concerns regarding the growing social costs of mental health and addictions care are on the rise, so too are the questions regarding how to maximize the impact and reach of treatment research findings along the translational spectrum. The lag time between research trials and diffusion to community practice continues to be large despite public health urgency to provide effective targeted care. As the demand for personalized medicine increases (Hamburg & Collins, 2010; Hutchison, 2010; MacDonald-Wilson, Deegan, Hutchison, Parrotta, & Schuster, 2013), it is incumbent upon those studying clinical treatments for behavioral health problems to maximize the translation of research finding into practice.

Nearly two decades of research now clearly documents the wide range of consequences associated with violence and posttraumatic stress disorder (PTSD) in the lives of substance abusers (Debell et al., 2014; Torchalla, Nosen, Rostam, & Allen, 2012; van Dam, Vedel, Ehring, & Emmelkamp, 2012) which span many domains of functioning: biological, interpersonal, and psychiatric (McCauley, Killeen, Gros, Brady, & Back, 2012; Norman et al., 2012). For individuals with PTSD, the risk of having an alcohol or substance use disorder (SUD) is approximately six times greater than for those without PTSD (Creamer, Burgess, & McFarlane, 2001; Pietrzak, Goldstein, Southwick, & Grant, 2011). This comorbid population seeks treatment more often than individuals with SUDs without PTSD, yet the prognosis for treatment is frequently poor (Ouimette, Ahrens, Moos, & Finney, 1997). Additionally, compared to those with SUDs without comorbid PTSD, those with comorbid PTSD spend a greater number of hospital overnights for addiction treatment even when there are no differences in substance abuse severity (McCauley et al., 2012). Studies show that substance dependent individuals with additional psychopathology are less compliant with treatment, more likely to drop-out, have a higher suicide rate, and receive less support for achieving and maintaining sobriety (Greenfield et al., 2007), therefore the healthcare burden of such patients is high.

Outcomes from randomized controlled trials (RCTs) provide a benchmark for how efficacious a treatment can be in reducing symptoms and improving health outcomes. However, recent critiques (Kazdin, 2008; Wampold, Hollon, & Hill, 2011) point out the limits of the RCT design with respect to cost, length of time to completion, and relevance of questions that can be answered. Further, because behavioral health issues, including SUD and PTSD, are often complex and co-occurring, myriad factors may contribute to psychosocial/health outcomes. These factors can include, but are not limited to, separate disease-related risk factors, individual differences (i.e., sex, age, race/ethnicity), severity of one disorder reciprocally influencing the course of another disorder, co-occurring medical problems, and extent of social support systems. Although current efficacious treatments for co-occurring PTSD and SUD do exist, adoption and implementation are halting due to modest effect sizes (e.g., Hien, Cohen, & Campbell, 2009; Torchalla et al., 2012), limited generalizability of findings, and difficulty in adhering to the complexity and/or length of the treatment (Gielen, Krumeich, Havermans, Smeets, & Jansen, 2014; Lamb, Greenlick, & McCarty, 1998; McLellan, Carise, & Kleber, 2003).

Among SUD populations with co-occurring PTSD, findings across pharmacologic and behavioral trials have yielded many guides for treatment, however, over 50% of patients still have symptoms at the end of treatment and the study of efficacious treatments in both areas has been hampered by methodological challenges (Hien et al., 2009). Challenges include an overreliance on “superiority” designs rather than equivalence designs, delivery of a short term intervention (typically no more than 6–12 weeks) to patients with the most severe presentations of multiple comorbidities (often accompanied by co-occurring medical problems), and symptom-based outcomes among a group of patients where given the process of treatment, we would reasonably expect an initial worsening of symptoms in one domain (i.e., PTSD) when there is a reduction in substance use (which was serving a self medication function for the patient [for a full review of these issues see Hien et al., 2009]).

Towards the aim of highlighting the potential benefits of using secondary analyses to address questions related to effectiveness of trauma/PTSD and SUD comorbidity treatments, this paper will first explore some of the limits of the RCT, highlighting important considerations in interpretation and methods for use of secondary analyses to enhance understanding of findings from existing RCT datasets (Curran & Hussong, 2009). A case study, from a set of published papers by the present authors generated from the National Drug Abuse Treatment Clinical Trials Network (NIDA CTN) “Women and Trauma” Study, will then be used to highlight the benefits and potential limitations of applying secondary analytic techniques to a large-scale randomized effectiveness trial. Finally, we will address the implications of this approach to improve our ability to identify best practices for community-based treatment of these disorders. In light of the limitations associated with overreliance on RCT designs, innovative methods are needed to maximize the benefits of efficacy and effectiveness studies.

2. Randomized control trials: burnishing the gold standard

In the field of behavioral treatment research, RCTs are a methodological gold standard for establishing an empirical basis for the efficacy of an emerging psychotherapy treatment intervention. However, while an RCT can demonstrate if one treatment is more efficacious than another for treatment of a specific problem (or pre-specified outcome), it cannot distinguish why or how the therapeutic change occurs. Most often, the causal mechanisms of the treatment's efficacy remain unknown and researchers are left with an empirically-validated, yet unspecified, picture about which treatment to select given a specific client's presenting clinical problem. If the mechanisms of change remain unknown to researchers, clinicians then, are similarly faced with making decisions about treatment selection (or adoption) and implementation without fine-grained information about how the treatment produces desired outcome effects — and in many cases, for whom (i.e., subgroup effects).

While RCT designs remain the strongest experimental method for establishing treatment efficacy in relation to a comparison or control intervention, these outcomes give rise to another methodological problem: the balance between internal validity (i.e., experimental control) and external validity (i.e., generalizability). Due to stringent inclusion and exclusion criteria – typically deployed to limit the sample to a specific psychological disorder – the findings from an RCT clinical sample may not generalize well to heterogeneous populations in the community. Treatment providers in the community commonly see patients presenting with complex co-morbidities and other psychosocial needs. Thus, larger effectiveness trials, with more heterogeneous samples, such as those conducted within NIDA's CTN (www.drugabuse.gov/about-nida/organization/cctn/ctn), and more recently grants offered by the Patient-Centered Outcomes Research Institute (www.pcori.org/) for large comparative effectiveness trials, offer an important opportunity to address critical clinical questions regarding subgroup effects in RCT designs.

Large samples are more likely to yield statistically significant effect sizes, however, the central tendencies representing large sample data preclude the interpretation and applicability of an RCT finding to any one individual (Clay, 2012; Kazdin, 2008; Westen, Novotny, & Thompson-Brenner, 2004). Moreover, the feasibility constraints associated with the implementation of RCT designs, specifically as it relates to the treatment of co-morbid populations, typically tests a short-term behavioral solution – more often than not, a truncated, cognitive behavioral therapy (CBT) – on a complex problem often requiring a longer-term, or multi-modal treatment. RCTs focusing solely on short-term therapy and focused behavioral outcomes limit the utility of findings for both clinicians in the community and for difficult-to-treat, chronically mentally ill individuals (Hien et al., 2009), greatly impacting the translational capacity of these interventions.

These methodological and other challenges related to the realities of today's healthcare environment, highlight the need for researchers to expand upon and integrate alternative behavioral therapy research methods to further address and supplement the narrow utility associated with RCT findings (Kazdin, 2007; Onken, Carroll, Shoham, Cuthbert, & Riddle, 2014). The high cost and restricted capacity for inquiry of the RCT design underscore the need for statistical innovations that enable researchers to answer pre-planned questions about mediation and subgroup effects through secondary analyses, particularly if the secondary outcomes can be used to inform the design of subsequent, more targeted studies. To this end, the National Institutes of Health have recently issued a program announcement (PAR-13-080) that promotes the use of preexisting datasets to conduct innovative analyses using new or advanced statistical methods to explore research questions across subgroups (e.g., individuals, families, age groups/development, gender, communities, and population groups) (DHHS, 2013). As will be discussed below, however, limited understanding and problems of implementation of secondary analyses have slowed the benefits.

3. Secondary analytic methodology

Secondary analyses of randomized efficacy and effectiveness clinical trials can be critical alternative sources of empirical data, particularly in light of the many barriers and escalating costs for conducting such trials. Such analyses, ideally pre-planned, but even when not, can advance our understanding of treatment efficacy, implementation, and dissemination, and inform the design of future research. Secondary analyses allow the investigator to explore the potential proximal and distal risk factors and contextual factors that are necessary to understanding how treatments work, and for whom (e.g., Hien, Campbell, Ruglass, Hu, & Killeen, 2010). Moderators and mediators that help identify mechanisms of action, and key drivers that promote or interfere with treatment efficacy, can also be examined (e.g. Hien et al., 2012).

3.1. Methodological concerns related to secondary analysis

Secondary data analysis still produces skepticism in some behavioral treatment research circles (Castle, 2003; Smith, 2008; Windle, 2010). Some argue that utilizing the same data in multiple publications may lend exaggerated support to psychotherapies that may be unwarranted (Nigel, 2012); grant reviewers may question the inclusion of too many specified secondary outcomes in an application. There are multiple methodological disadvantages to using existing data in order to achieve secondary research objectives that – if not addressed – may engender suspicion from the scientific community of resultant findings and prevent clinical researchers from fully taking advantage of this methodology (Castle, 2003; Garmon Bibb, 2007; Hofferth, 2005; Rew, Koniak-Griffin, Lewis, Miles, & O'Sullivan, 2000; Smith, 2008). For example, one of the limitations of secondary data analysis is that the data reflect the views and questions of the original researcher and may not adequately fit the specific research question or purpose of the secondary investigator (Castle, 2003, Coyer & Gallo, 2005; Rew et al., 2000). The use of secondary data also limits the new investigator's control over the research design, including inclusion/exclusion criteria and measurement/collection of specific variables that may be better suited for his/her research questions.

Secondary analyses may also be limited by the sample size. Many secondary analyses entail subgroup analyses (e.g., by gender, race/ethnicity, severity, or other demographic or clinical characteristics) in order to identify and understand individual differences or variability in treatment processes and outcomes (Geller, Koch, Pellettieri, & Carnes, 2011). These analyses also require adjustments for baseline severity of the psychiatric or substance use disorder. However, depending on the size of the data set, these subgroup analyses may be underpowered to obtain precise estimates of the variables in question (Hofferth, 2005). Moreover, the conduct of multiple comparisons without adjustments in p-values may render interpretation of the significance of such findings questionable. This becomes particularly true with access to publicly available datasets that are not regulated by any single body. In order to ensure scientific rigor in secondary data analysis, researchers need to have a grounded conceptualization of the problem being examined, a theoretical framework that guides the delineation of the research questions, identify and operationalize theoretically congruent concepts, and apply novel and rigorous statistical analyses to answer the questions at hand (Castle, 2003; Coyer & Gallo, 2005; Smith, 2008). Specific subgroup analyses also ideally should be justified a priori and limited in size and scope. It is important to note, however, that a priori specification does not mitigate the multiple testing burden. Without a theoretical framework, secondary analyses may be seen as inappropriate data mining and may yield results that are fundamentally questionable in terms of advancing the science of a particular area of research (Hofferth, 2005). Researchers must tread cautiously so that they do not draw overreaching or erroneous conclusions about the nature of their secondary findings.

3.2. Conducting secondary analyses

Notwithstanding some limitations and the need for a careful approach, there is an enormous amount that can be gleaned from secondary analyses of large effectiveness trial data. Such analyses may include: measures of secondary outcomes, subgroup/moderation and mediation analyses. In the following section, we highlight the strengths and weaknesses inherent in each approach (Curran, 2009; Curran & Hussong, 2009).

3.2.1. Secondary outcomes of substance use, treatment retention, and other psychiatric symptoms

Secondary outcomes preferably should be specified a priori in the study protocol. In addiction research, abstinence has been considered the “gold standard” outcome of an addiction treatment's efficacy; however, it may be just as informative or relevant to patient and provider to consider reduction in drug or alcohol use as a successful treatment outcome. For example, in studies of cannabis, examining a decrease in use would be particularly useful in light of changing social attitudes and varying degrees of legalization, comparable to the moderation literature for alcohol. While much of the scientific and treatment communities would prefer a primary outcome of abstinence, especially if the patient population has a defined substance use disorder, treatments that impact the amount or frequency of use may be important with regards to patient functioning and the social and economic burden to society. For example, NIAAA has adopted a standard outcome measure as a reduction in the number of heavy drinking days as opposed to complete abstinence from alcohol (WHO, 2000, p.57). Definitions of treatment success, although often correlated, are important considerations in understanding if and how treatments work.

Further, in terms of more patient-centered outcomes, an examination of psychosocial and cognitive functioning could indicate an improved quality of life. Similarly, if an individual continues to use, but at a significantly lower frequency that produces far fewer contacts with the criminal justice system, they might feel that treatment had a positive effect on their lives without actually achieving abstinence. On the other hand, outcomes such as these can be difficult to demonstrate using an RCT design, and thus may not be appropriate to select as a primary outcome measure. Yet arguably these measures should be considered key secondary outcomes that capture particularly relevant, patient-centered responses to treatment.

3.2.2. Subgroup/moderator analyses

Another common type of secondary analysis is that of examining subgroups, in which it is of interest to evaluate whether particular participants respond differentially to treatment. Investigators are often interested in differences in treatment efficacy across several types of factors such as a) demographic characteristics, b) pre-treatment psychiatric symptomatology or c) treatment attendance (e.g., Hien, Cohen, Campbell, et al., 2009; Hien, Wells, Jiang, et al., 2009; Hien et al., 2012; Morgan-Lopez et al., 2014; Zlotnick, Johnson, & Najavits, 2009). The National Institutes of Health specifically require examination of the subgroup effects of gender, race, and ethnicity (Geller et al., 2011). With the development of the Patient-Centered Outcomes Research Institute (PCORI), much focus has recently turned both to considering which kinds of patients or subgroups are best served by a particular treatment. Indeed these kinds of analyses have come to be critical for comorbidity research, where the heterogeneity in the psychiatric population is very high (Hien, Campbell, Killeen, et al., 2010; Nunes et al., 2011) and, where frequently, the most severe patient groups tend to benefit from the targeted treatment, whereas the less severe patients may require less individualized or specialized interventions.

A primary concern with such moderator analyses is the difficulty in making causal inferences in cases where the treatment condition was randomized but the moderator(s) of interest was(were) not. In a conventional RCT, where only treatment is randomized, both the moderator and covariates will be balanced across the treatment conditions due to randomization. However, causal inferences regarding the moderator and the interaction between treatment condition and the moderator will be biased as a function of the covariates, even if the main treatment effect remains unbiased; in other words, randomizing serves to remove bias only for the main treatment effect (Dong, 2012). This bias may be evident, for example, in the literature on treatment by dosage interactions when dosage is self-selected. Treatment appears to work best in moderate dosages (Barkham et al., 1996, 2006; Hien et al., 2012), but there may be selection effects driving the findings on dosage–outcome relations because patients who drop out of treatment and patients who attend all sessions may have greater pre-treatment impairment than those who attend a moderate number of sessions (López-Goñi, Fernández-Montalvo, & Arteaga, 2012). Advances in causal modeling of moderation analyses under propensity scoring frameworks when at least one of the focal predictors is not randomized are emerging as a potential option for mitigating bias in treatment by moderator analyses (Dong, 2012). Still, there may be underlying variables that are driving the moderator findings (i.e., race may be proxy for other sociodemographic moderators), so caution must be exercised in the selection of potential candidates for subgroup analyses.

Propensity scoring methodology for treatment × moderator interactions draws from Imai and van Dyk's (2004) framework for converting the propensity to receive a continuously measured treatment (e.g., treatment dosage) into probability density values from two models: (a) a “numerator” model and (b) a “denominator” model. The numerator model would be a regression model with treatment dosage as the outcome and treatment condition as the only predictor of treatment dosage. From this model, the standardized residuals are converted into normal probability density function values (i.e., numerator probabilities). The denominator model adds covariate terms as predictors of dosage. The numerator probabilities are divided by the denominator probabilities to produce the propensity weights, which, when entered into the subsequent outcome analysis, produce unbiased estimates for the treatment effect, the moderator, and their interaction (Coffman, Caldwell, & Smith, 2012; Dong, 2012).

3.2.3. Mediator analyses

Other common secondary analyses, particularly in behavioral health, are mediator analyses. Addiction treatment researchers may take for granted that the strengths of the RCT design protect causal inferences for questions of mediation. However, this is not the case in traditionally-designed RCTs where only treatment condition assignment is random. Special types of selection biases can compromise causal inference as it relates to mediation analysis (Jo, 2008; Robins & Greenland, 1992). In RCT data, causal inferences can be made regarding the link between treatment and the mediator because the treatment was randomized; however, it has been suggested that even in conventional RCTs, causal inferences often cannot be made regarding the links between the mediator and the outcome because the mediator is not, and in some cases cannot be ethically, randomized (Jo, 2008; Robins & Greenland, 1992). By virtue of randomization, covariates are balanced across treatment groups and, in fact, baseline levels of the mediator and outcomes could also be balanced. The issue of selection bias emerges with regard to the relation between changes in the mediator and changes in the outcome. As noted above, the mediator cannot be treated as though it has a causal influence on outcome, because the covariates that were balanced across treatments due to randomization could still bias the relation between the mediator and outcome (Coffman & Zhong, 2012). The bias evident in the link between mediators and outcomes is demonstrated in the following example: if changes in PTSD served as a mediator of treatment effects on SUDs in a conventional mediation analysis, inferences regarding the link between PTSD and SUDs could not be made as though clients had been randomized to different levels of change over time in PTSD (which, of course, can neither practically nor ethically be done). We anticipate that recent advances in propensity scoring for mediation analysis (e.g., Coffman & Zhong, 2012), where weights are estimated by modeling the joint probability of being in the active treatment group and experiencing a specific level of the mediator, will mitigate bias in mediation analyses in addiction treatment studies. These weights work to create an “artificial RCT” where one could estimate causal effects as though both the treatment and the mediator were randomized. The weights for mediation would be calculated in a manner similar to the weights described in Section 3.2.2 for causal moderator effects (Coffman & Zhong, 2012).

3.2.4. Other biases to consider when conducting secondary data analyses

When performing pre-planned secondary analyses, it is important to implement the analyses exactly as stated in the protocol or statistical analysis plan. Analyses which deviate from the pre-specified methods nullify the safeguards and confidence produced through pre-planning. If it is necessary to deviate from the pre-planned analyses, then this should be clearly stated in any manuscript or presentation. Similarly, if new secondary outcomes of interest are identified, any presentation of those results should mention that they were not pre-planned and also justify their consideration. Regardless of whether the analyses were pre-specified or not, it is important to describe the number of analyses performed and the number that was statistically significant. Reporting bias is a major consideration when evaluating secondary analyses, and transparency is absolutely vital in addressing this concern. A useful tool may be to generate a table that summarizes every analysis performed, and the result. Some journals require provision of the protocol and statistical analysis plan so that readers can determine which analyses were pre-specified and which were ad hoc. They can also be used to determine whether any pre-planned analyses followed the actual spec-ified methods. This requirement for supplemental material forces transparency by all authors and in the end can elevate our understanding of research outcomes and help to prioritize research findings based on statistical and scientific rigor.

Another source of bias in reporting of secondary analyses is the plethora of analyses reported from investigators utilizing the same RCT data. As investigators proceed with more and more secondary analyses, they may develop, consciously or not, their own biases in the outcomes and analyses of interest. A bias may also be induced by exploratory follow-up of positive findings, for example, if an association is found with analysis of substance use on one assessment one is more likely to explore the relationship with substance use via another assessment or different timeframe. This may manifest itself in biased reporting of secondary analyses and skepticism by the research committee. As the number of analyses performed by the same research team on the same data set increases, the likelihood of perceived biased reporting also increases.

It is common to assume a parametric model when performing sample size calculations during grant development, however, the model may not be appropriate for the data that are actually collected during the course of a study. In this case, it is important to consider alternative models or covariates for the primary outcome measure but only as secondary analyses. For example, exploration of the effect of time can be useful particularly in the case of behavioral interventions where there is generally a lag time in treatment effect (Campbell et al., 2014). The analyses of additional covariates or variations from the assumed model should be specified a priori, however this is not always possible. For example, during the conduct of a trial, a potential confounder for the relationship between treatment and outcome may be identified in recent literature that should be accounted for in analyses. When examining the observed data it may become clear that the assumptions made during the trial regarding distribution of the outcome measure have been violated. In these cases, it is impossible to pre-specify the alternative models or additional covariates. It is common in the substance abuse field to observe a lack of compliance (or adherence) with the treatment regimen, and this can also differ across treatment arms. Statistical methods have been developed to adequately address such situations (e.g., Angrist, Imbens, & Rubin, 1996; Jo & Stuart, 2009; Little & Rubin, 2000), as adjusting for post-randomization factors is not trivial.

Summarizing the previous section, secondary analyses of RCTs can help propel the addiction treatment field forward by addressing critical questions on how and for whom behavioral treatment interventions work. Careful grant development and innovative analytic methods can help to address historical concerns related to the use and interpretation of secondary analyses. The next section helps to illuminate the contributions of secondary analyses by offering a case study of a large randomized controlled effectiveness trial of a behavioral intervention for women with substance use disorders (SUDs) and co-morbid posttraumatic stress disorder (PTSD). We selected this study to use as an example of how primary outcome findings may be maximized through a thoughtful set of a priori and post hoc secondary analyses, and to highlight some of the cautions that must be applied to interpretation of findings.

4. “Women & Trauma” case example

Since 2000, The National Institute on Drug Abuse (NIDA) has funded the Clinical Trials Network (CTN), a multi-site research network charged with the mandate to develop and implement research protocols in collaboration with community-based treatment programs (CTPs). An important part of the vision of the CTN is addressing the critical need to adopt new and effective treatments, using research as a vehicle for conveying knowledge and promoting dissemination. In the early phases of the CTN, involved providers clearly articulated that trauma was a ubiquitous issue among their female clients with SUD and asked that a protocol be developed that addressed the needs of women with co-morbid trauma and addiction. From this collaborative juncture, the CTN “Women and Trauma” Study (WTS) began.

The WTS study used a prospective, randomized, controlled, repeated measures design to assess the effectiveness of adding a trauma-focused group intervention to a platform of substance abuse treatment (Hien, Cohen, Campbell, et al., 2009; Hien, Wells, Jiang, et al., 2009). Participants were 353 drug dependent women seeking treatment for SUD(s) who met criteria for PTSD. Trained counselors from seven outpatient community substance abuse treatment programs participating in the CTN across six states conducted one of two group interventions: Seeking Safety (Najavits, 2002) or Women's Health Education (Miller, Pagan, & Tross, 1998, unpublished manual). Seeking Safety is a structured cognitive–behavioral treatment with both safety/trauma and substance use components integrated into each session (e.g., psychoeducation on substance use disorders and PTSD, skill-building to prevent drug use and manage PTSD symptoms, cognitive restructuring with attention to maladaptive thoughts linked to substance use and trauma symptoms). The Women's Health Education (WHE; Miller et al., 1998) control condition was adapted from a treatment for female partners of injection drug users and is a psychoeducational, manualized treatment focused on relevant health topics (e.g., pregnancy, nutrition, diabetes, HIV/sexually transmitted infections). The 90-min groups occurred twice weekly over a 6-week period (12 sessions total for both treatments).

The impact of treatment on pre-specified primary outcomes, including (1) drug and alcohol abstinence and (2) PTSD symptom severity, was assessed pre- and post-treatment and at 3-, 6-, and, 12-month post-treatment follow-ups. Secondary outcome variables that were identified in the protocol a priori included: (3) treatment retention; (4) secondary measures of substance use and PTSD outcome (parallel to the primary outcomes); (5) other psychiatric symptom severity; and (6) HIV-risk sexual behaviors. Also planned were analyses to address delivery of drug abuse treatment for women by examining various characteristics of the sites (CTPs) for their potential impact on the effectiveness of the intervention (moderator) as well as the effects of various baseline demographic and individual characteristics (i.e., subgroups/ moderators) on primary outcomes and on the retention of subjects in treatment. In addition, in an exploratory fashion, mediator analyses that examined attendance patterns and level of symptom severity (i.e., mediators) on treatment outcomes were planned.

To date, there have been 19 publications (see Table 1) that have emerged from this study. All but four of these publications were derived together with the lead investigators. One of the papers (Hien et al., 2009) details the primary outcome analyses that were pre-specified in the study protocol; the remaining 18 focus on secondary or exploratory analyses. Of these 18 publications, 8 examined secondary outcome measures, 8 examined subgroups/subtypes of participants, and 2 examined mediating factors. Notably, 13 papers were prespecified in the protocol along with strategies for addressing power in the protocol data analytic plan. The mediational analyses were a priori deemed as exploratory, to be considered as hypothesis generating analyses. A publication plan that was submitted to the CTN publications subcommittee (PSC) also detailed each planned paper. All manuscripts were reviewed and approved by the CTN PSC prior to submission for publication. Findings from the secondary analyses have provided significant evidence for the effectiveness of the trauma-focused treatment on both PTSD and SUD. Below, we will detail some of the findings and critiques based on our prior delineation of the kinds of analyses that can be done, what we can take away from them, and what the limitation of the approaches was in terms of implications for the analysis of the WTS.

Table 1.

NIDA Clinical Trials Network “Women & Trauma” Study: publications of primary, secondary, subgroup/moderator, and mediator outcomes.

Publication	Sample/methods	Main findings
Primary outcome
1. Hien et al., 2009 Multisite Randomized Trial of Behavioral Interventions for Women with Co-occurring PTSD and Substance Use Disorders	N = 353 women (ITT) RCT in 7 outpatient psychosocial clinics; 6-week TX (12 sessions) open enrollment groups (SS or WHE); participants with PTSD and SUD; 54.4% minority/non-White. GLM to assess the effect of treatment on four outcomes.	• SS, WHE groups ↓ PTSD symptom severity (PSS-SR, CAPS) during TX; • SS, WHE sustained PTSD improvements during FU but at a slower rate than during TX; • Neither SS nor WHE ↑ drug/alcohol abstinence or % days drug/alcohol use (SUI, UDS, Saliva) during TX or FU; • Participants with ≥ 6 TX sessions: SS superior to WHE on PTSD severity (CAPS) during FU (X2(1) = 3.13, p = .08) (did not reach p < .05 cut-off).
Secondary outcomes
2. Killeen et al., 2008 Adverse events in an integrated trauma-focused intervention for women in community substance abuse treatment	N = 353; comparison of study-related AEs and association between session attendance and study-related AEs using a log linear model.	• 49 women reported 83 new study-related AEs; 1 was severe. No difference between SS and WHE in the number of study-related AEs (28 [9.6%] 21 [7.2%] respectively). • No association between study-related AEs and length of time to complete TX (X2(1) = 0.88, p = .30).
3. Hien et al., 2010 The impact of trauma-focused group therapy upon HIV risk behaviors in the NIDA Clinical Trials Network “Women and Trauma” multi-site study	N = 346; samples excluded one site (n = 7). ZINB model used to test TX effect on the count of USO.	• SS participants with higher sexual risk (i.e., at least12 USO/month) reduced the number of USO over FU compared to WHE (OR = 0.74, p < .05).
4. Pinto et al., 2011 Retention in the National Institute on Drug Abuse Clinical Trials Network Women and Trauma Study: Implications for Post-trial Implementation	N = 346; samples excluded one site (n = 7). Log-linear regression model of retention and logistic regression of minimum TX exposure (≥6 sessions).	• Mean number of TX sessions did not differ by TX arm (SS = 7.5 [SD = 3.9]; WHE = 6.8 [SD = 3.9]). • ↑ rates of attendance associated with being older, greater education, and stronger therapeutic alliance; 12-step meeting attendance was associated with ↑ retention, but only among those with ≥ 6 sessions. • The site with highest retention provided child care and had the lowest average monthly intake.
5. Resko and Mendoza, 2012 Early Attrition From Treatment among Women with Co-occurring Substance Use Disorders and PTSD	N = 340; participants with complete data on all predictor and outcome variables; multivariate logistic regression on the outcome of early attrition.	• Early TX attrition associated with perceived need for psychological TX (OR = 3.42), a history of youth partner violence (OR = 2.73), stimulant (OR = 1.30), alcohol (OR = 0.82) and opioid (OR = 0.75) abuse. • Logistical barriers (e.g., transportation, having children) not associated with early attrition.
6. Ruglass et al., 2012 Helping alliance, retention, and treatment outcomes: A secondary analysis from the NIDA Clinical Trials Network Women and Trauma Study	N = 223; samples included participants who completed the HAQ at wk 2. GLM to test association between therapeutic alliance and retention, PTSD symptom severity, days of substance use.	• SS had ↑ alliance than WHE at week 2 (p = .01); difference was small (SS M = 5.33, WHE M = 5.15 on 6-point scale; findings at wk 6 similar but not significant (p = .08). • ↑ alliance at wk 2 was associated with ↓ PTSD severity post-TX (p < .001) for SS and WHE; this weakened over time. • Alliance at wk 2 was associated with # of TX sessions attended (p = .05) for SS and WHE. • Alliance was not associated with substance use (p = .59).
7. Cohen, Field, Campbell, & Hien, 2013 Intimate Partner Violence Outcomes in Women with PTSD and Substance Use: A Secondary Analysis of NIDA Clinical Trials Network “Women and Trauma” Multi-site Study	N = 288; samples included all participants with ≥ FU (1-wk, 3 M, 6 M, 12 M post-TX). Logistic regression used to test association between TX and relevant risk factors on IPV.	• Significant risk factors associated with IPV during FU were: living with someone who has an alcohol problem (OR = 3.2), higher total lifetime traumatic exposures (p = .05), and recent physical/sexual assault (p = .06). • Baseline abstinence associated with ↓ risk IPV at FU (OR = 0.33, p < .05). • TX arm not associated with IPV; interaction between TX and baseline abstinence: those abstinent and in SS were less likely (OR = 0.24) to experience IPV compared to non-abstinent SS and abstinent WHE.
8. McHugh et al., 2014 Changes in Sleep Disruption in the Treatment of Co-Occurring Posttraumatic Stress Disorder and Substance Use Disorders	N = 353; GLM of association between TX and nightmares and insomnia; GEE approach of association between sleep improvement and PTSD severity and substance use during FU.	• 87% of participants ≥ 1 clinical-level sleep symptom at the start of TX, 64% at end of TX. • TX by TX-week interaction (p = .03): ↑ nightmares among SS (45%) early in TX compared to WHE (40%); ↓ nightmares among SS (22%) compared to WHE (24%) later in TX. • Insomnia ↓ during TX (OR = 0.88) for both groups. • Improvement in sleep symptoms during TX associated with improved PTSD symptom severity over time (p < .001); improvement in sleep not associated with substance use (p = .81).
9. Ruglass et al., 2014 Associations between Post Traumatic Stress Symptoms, Stimulant Use, and Treatment Outcomes: A Secondary Analysis of NIDA's Women and Trauma Study	N = 141; samples were participants with stimulant use in the 30 days before study entry; GEE used to examine PTSD and stimulant use over time.	• Heavy baseline stimulant users (≥ 13 days in last 30) had ↑ PTSD severity compared to light users (p = .02). • SS and WHE ↓ PTSD severity during FU (scores ↓ 49% from baseline to post-TX; ↓ additional 9% over FU); no difference by group (p = .32). • Both groups ↓ stimulant use during (p = .29). • SS more effective than WHE in ↓ stimulant use at FU among heavy stimulant users and who showed improvements in PTSD symptoms during TX (3-way interaction: baseline stimulant use, TX arm, PTSD symptom improvement; p < .01). No difference by group among light stimulant users.
Subgroup/moderators
10. Hien et al., 2010 The role of alcohol misuse in PTSD outcomes for women in community treatment: A secondary analysis of NIDA's Women and Trauma Study	N = 353; alcohol misuse defined as (1) daily alcohol use or (2) ≥ 1 day of alcohol use to intoxication in the prior 30 days. GEE used to examine PTSD outcomes over time as a function of alcohol misuse.	• Among women with alcohol misuser, PTSD (PSS-SR) symptoms lower in SS during TX (p < .05) and FU (p < .05) compared to WHE. For those without alcohol misuse, PTSD symptoms improved for WHE vs SS during the first week of TX (p < .01), but ↓ more quickly for SS over the rest of TX; no differences post-TX or during FU. • During TX, alcohol misusers in SS with higher baseline hyper-arousal improved more quickly than those with lower hyper-arousal (p < .05).
11. Cohen et al., 2010 Survey of Eating Disorder Symptoms among Women in Treatment for Substance Abuse	N = 122; samples were those who completed EDE-Q (29% reported ≥ 1 binge eating episode in the last 28 days). GEE used to test PTSD and substance use outcomes as a function of binge eating status.	• Binge eating group had ↑ PTSD severity scores (CAPS) over FU (9.5 points greater); all participants showed ↓ in PTSD symptoms. • No binge group more likely to be abstinent during TX and FU (OR = 1.82, p = .03); abstinent rate of binge group was 45% lower. No difference between binge groups on # days of drug/alcohol use (p = .67).
12. Hien et al., 2012 Attendance and Substance Use Outcomes for the Seeking Safety Program: Sometimes Less Is More	N = 353; LCPMM used to estimate attendance patterns and test for TX effects within and across latent attendance patterns and group membership turnover on the outcomes of past 30 day alcohol and cocaine use.	• Similar TX attendance patterns across alcohol and cocaine use groups (TX arm was not predictive of attendance class membership): Completers never ↓ below 80% probability of attendance; Droppers never exceeded 41% probability; Titrators 50% to 80% probability of attendance thru 7th session. • Completers had ↓ in the probability of alcohol use from baseline to post-TX (p = .01), followed by non-significant ↑ in alcohol during FU (p = .77). No difference by TX arm. • Titrators had non-significant ↓ in alcohol use from baseline to post-TX (p = .14) and during FU (p = .53). Titrators in SS had ↓ rates of alcohol use during FU vs. WHE (p = .02). • Droppers had non-significant increases in alcohol from baseline to 1-wk post TX (p = .71) and during FU (p = .88). • Among all classes, non-significant ↑ in cocaine use over the study; no difference by TX arm.
13. Winhusen, Winstanley, Somoza, and Brigham, 2012 The Potential Impact of Recruitment Method on Sample Characteristics and Treatment Outcomes in a Psychosocial Trial for Women with Co-occurring Substance Use Disorder and PTSD	N = 106 (single site analysis); GEE used to explore TX effect and TX effect (on PTSD symptoms) as a function of recruitment source (advertising 66% vs clinic referral 34%).	• Advertising group had ↑ levels of drug use and PTSD severity; more likely to meet cocaine use disorder and full PTSD criteria vs clinic recruits. • SS was associated with ↓ PTSD severity during TX (p < .001) and FU (p < .001), but not rates of weekly drug use or abstinence during TX or FU (p's all ≥ .05). • TX effect sizes (SS v WHE) for PTSD symptom ↓ were greater for advertising group vs. clinic recruits. • No TX arm differences in drug use or abstinence for either recruitment group during TX; during FU, clinic recruits in SS more likely to report past week drug use.
14. Morgan-Lopez et al., 2013 Synergy between Seeking Safety and Twelve-Step Affiliation on substance use outcomes for women	N = 353; LCPMM used to model variation in SS by 12-step interaction effects on alcohol and cocaine use.	• Post-TX 12-step not associated with post-TX alcohol (p = .27) or cocaine use (p = .29). TX arm not associated with post-TX 12-step (p-.97). • SS participants who also sought post-TX 12-step had greatest ↓ in alcohol use rates over time (p = .002). • Reductions in cocaine use during TX (p = .049) and over FU (p = .11) were also observed, but no difference by TX arm (p > .20); no interactions with post-TX 12-step attendance (p = .93).
15. Lopez-Castro et al., 2014 Pathways to Change: Use Trajectories Following Trauma-Informed Treatment of Women with Co-Occurring Posttraumatic Stress Disorder and Substance Use Disorders	N = 353; latent growth mixture modeling used to examine trajectories of substance use (past 30 day alcohol, cocaine use) and to test for proximal and distal predictors.	• Three distinct trajectories of substance during FU: low risk/infrequent use group (50.5%); high risk/infrequent use group (26.6%); high risk/frequent use group (22.9%). • Improvement in PTSD severity associated with membership in the low risk/infrequent use group (p = .01); TX arm was not (p = .70). • Substance use, age and after-care were different by trajectories (p < .05).
16. Ruglass et al., 2014 Racial/Ethnic Match and Treatment Outcomes for Women with PTSD and Substance Use Disorders Receiving Community-Based Treatment	N = 224; samples were women who identified as a single race/ethnicity and attended at ≥ 1 TX session (n = 23 Latina patients excluded due to small numbers). GLM used to test association between client race/ethnicity, individual race/ethnicity match (client/provider), group race/ethnicity match, attendance and outcomes (PTSD severity, max drug use days in last 30).	• No difference by race/ethnicity (p = .16) and no associations between individual or group racial/ethnic match on session attendance (p > .05). • Three-way interaction (baseline PTSD severity, client race/ethnicity, and individual racial/ethnic match) on PTSD severity at FU [F (1,417) = 5.94, p = .02]. White clients with high baseline PTSD and individual racial/ethnic match showed ↓ PTSD severity at FU vs. those who did not match (p = .03). • Two-way interaction (baseline substance use and individual racial/ethnic match) on substance use during FU [F (2,418) = 6.07, p < 0.01]. Light users who were matched were less likely to use substances heavily (p = .07); no baseline users who were matched more likely to use substances heavily (p = .03). Heavy users showed no difference by therapist match on substance use during FU (p = .51).
17. Anderson & Najavits, 2014 Does Seeking Safety Reduce PTSD Symptoms in Women Receiving Physical Disability Compensation?	N = 353; disability was defined as receiving a pension for a physical disability (n = 20, 5.7%). GEE was used to test disability status as a moderator of TX on PTSD symptoms over FU.	• Those with a disability had ↑ severity on ASI medical composite, BSI somatization subscale and CGI depression subscale vs. no disability. • Three-way interaction (p = .03) (disability status, TX arm, time). Disability group had ↓ PTSD symptoms more in SS vs. WHE over FU (both reduced during TX). Gains made during TX were sustained in SS; WHE returned to near baseline. CAPS scores 12-mo post-TX were 18.2 for SS and 56.0 for WHE. • No disability group ↓ PTSD symptoms over FU regardless of TX arm.
Mediators
18. Hien et al., 2010 Do Treatment Improvements in PTSD Severity Affect Substance Use Outcomes? A Secondary Analysis From a Randomized Clinical Trial in NIDA's Clinical Trials Network	N = 353; continuous Markov model used to explore temporality of PTSD improvement and substance use for four defined response categories (non-response; PTSD response; substance use response; global response (both PTSD and substance use) during TX. GLM used to examine max days of use and ASI drug and alcohol composite scores during FU.	• Non-responders, substance use responders, or global responders maintained original response classification. • PTSD responders more likely to transition to global response during TX (2.80 times more likely vs. substance use responders to transition to global response within 1 week). • SS more effective than WHE on ↓ substance use (p = .02) and ↓ drug composite (p = .03), but only among heavy baseline substance users who achieved significant PTSD reduction.
19. Morgan-Lopez et al., 2013 Indirect effects of 12-session SS on substance use outcomes: Overall and attendance class-specific effects	N = 353; longitudinal mediation analysis, accounting for changes over time in group membership and context to explore PTSD outcomes.	• SS showed steeper ↓ in PTSD frequency and severity, which in turn showed significant impact on ↓ cocaine and alcohol use (e.g., 95% CI for mediated effect of frequency on alcohol use = –1.77, –.105). • This pattern was primarily significant among Completers (compared to Titraters and Droppers) and only emerged during TX, not during FU.

AE = Adverse Event; ASI = Addiction Severity Index; BSI = Brief Symptom Inventory; CAPS = Clinician Administered Posttraumatic Stress Interview; CGI = Clinical Global Impression rating scale; EDE-Q = Eating Disorder Examination Questionnaire; FU = Follow-up; GEE = Generalized Estimating Equations; GLM = Generalized Linear Model; HAQ = Helping Alliance Questionnaire II; IPV = Intimate Partner Violence; ITT = Intent-to-Treat; LCPMM = Latent class pattern mixture modeling; PSS-SR = Posttraumatic Stress Symptoms–Self-report; PTSD = Posttraumatic Stress Disorder; RCT = Randomized Controlled Trial; SS = Seeking Safety; SUD = Substance Use Disorder SUI = Substance Use Inventory; TX = treatment; UDS = Urine Drug Screen; USO = Unprotected Sexual Occasions; WHE = Women's Health Education; ZINB = Zero-inflated Negative Binomial

4.1. Secondary outcomes of baseline substance use, treatment retention, therapeutic alliance, and other psychological/psychosocial symptoms

Eight separate analyses (six pre-planned) examined secondary outcomes such as retention, early attrition, adverse outcomes, therapeutic alliance, and risky sexual behaviors. Findings as displayed in Table 1 revealed that the mean number of sessions attended did not differ by treatment arm (SS versus WHE); although greater rates of attendance were significantly associated with being older, greater education, and stronger therapeutic alliance (Pinto, Campbell, Hien, Yu, & Gorroochurn, 2011). Early treatment attrition was significantly associated with perceived need for psychological treatment, a history of youth partner violence, stimulant, alcohol, and opioid abuse (Resko & Mendoza, 2012). Of the 353 participants, 49 experienced new study-related adverse events but this was not significantly different by treatment group and was not associated with treatment attendance (Killeen et al., 2008). An examination of the therapeutic alliance revealed that SS had significantly higher alliance ratings than WHE at week 2, although the difference was small (Ruglass et al., 2012). Higher alliance at week 2 was associated with lower PTSD severity post treatment and number of treatment sessions attended for both SS and WHE (Ruglass et al., 2012). Two analyses also revealed the association of baseline level of substance use and changes in PTSD symptoms during the course of treatment on psychosocial outcomes. For example, Cohen et al. (2010) utilized logistic regression to examine the association between treatment group and relevant risk factors on interpersonal partner violence (IPV) and found a significant interaction between treatment and baseline abstinence. Specifically, participants who were abstinent at baseline and in SS were significantly less likely to experience IPV compared to nonabstinent SS and abstinent WHE participants. Moreover, Ruglass, Hien, Hu, and Campbell (2014) and Ruglass, Hien, Hu, Campbell,, Caldiera, et al. (2014) found that SS was significantly more effective than WHE in reducing stimulant use at follow-up among women who were heavy stimulant users at baseline and who showed improvements in PTSD symptoms during treatment (3-way interaction between baseline stimulant use, treatment arm, and PTSD symptom improvement over time). Another analysis examined changes in sleep disruption and revealed that prevalence of insomnia decreased during treatment for both treatment arms (McHugh et al., 2014). Improvement in sleep symptoms during treatment was significantly associated with better overall PTSD symptom severity over time; however, improvement in sleep was not associated with substance use. Finally, among those who were engaged in higher sexual risk behaviors, the trauma therapy (SS) significantly reduced the frequency of unprotected sex compared to the WHE treatment group (Hien et al., 2010a).

Taken together, these analyses highlight the importance of examining client characteristics such as baseline substance use, sleep problems, risky health behaviors, and IPV (common conditions among those with co-occurring conditions) as these may inform treatment selection and also be targeted for more positive treatment outcomes. Our analyses also revealed that treatment process variables such as session attendance, attrition, and therapeutic alliance are critically important to examine to enhance treatment outcomes. Our results offer suggestions for improving session attendance and attrition among women with co-occurring conditions such as building a strong therapeutic alliance and providing additional supports to reduce barriers and facilitate treatment engagement.

4.2. Subgroup/moderator analyses

Eight separate papers addressed moderator/subgroup samples. Three of the papers (Hien et al., 2012; Lopez-Castro et al., in press; Morgan-Lopez et al., 2013) used a latent growth pattern mixture modeling technique to identify unique client subgroups and their responses to treatment over time. Attendance patterns were important to account for in evaluating the efficacy of the treatments; although treatment completers had positive outcomes, those who “titrated” their treatment attendance appeared to have the best response to the trauma focused therapy (Hien et al., 2012; Morgan-Lopez et al., 2013). Another paper (Lopez-Castro et al., in press) identified three distinct patient types (i.e., women who were consistent improvers, those who relapsed but continued to improve steadily, and those who appeared to be chronic relapsers) in their substance use over time that could be characterized with age of onset of use, life stressors and severity of treatment history as risk factors.

Across the board, all papers reported significant findings that for participants with greater severity in both behavioral risk and symptoms, benefits were derived from the targeted specialty treatment. That is, for those with more severe PTSD, receiving the trauma therapy impacted their substance use significantly whereas for the less severe patients, the health education curriculum appeared to work equally well as the trauma therapy. Many of these analyses provided important indicators to guide treatment matching by patient characteristics. For example, Morgan-Lopez et al. (2013) suggested a need to consider matching patients who were not yet engaged in twelve step programs with a less intensive, non-trauma focused therapy. In contrast, those who also attended twelve step programming benefited more from the trauma focused intervention in reducing drinking and drug use. Ruglass, Hien, Hu, and Campbell (2014) and Ruglass, Hien, Hu, Campbell, Caldiera, et al. (2014) found complex and nuanced differences across racial/ethnic groups that suggest the importance of identifying race/ethnicity of the providers and the participants in treatment in order to case match to treatment.

Regarding some of the limitations of these subgroup/moderator focused papers, five were specifically planned a priori, but since the aims of two papers involved examining moderator effects, since the participants were not randomized to the subgroups examined (for example, three types of treatment attenders (completer, titrator and dropper) or those who did and did not attend AA post-study intervention, causal inferences cannot be fully drawn specifically with regards to the trauma intervention efficacy found for those groups in which the intervention was associated with positive outcomes. Still, the findings of these eight analyses did provide some important guides with regards to future research efforts and areas for planning a priori analyses.

4.3. Mediation analyses

The two papers focusing on mediators (Hien, Campbell, Killeen et al., 2010; Hien, Campbell, Ruglass et al., 2010; Hien, Jiang, Campbell, et al., 2010; Morgan-Lopez et al., 2013) had been specified a priori in the protocol as exploratory papers. These papers examined the impact of the comorbid symptom improvement testing several possible hypotheses regarding changing symptom domains (i.e., PTSD versus SUD) and the differential impact of treatment upon these two outcome domains. Both analyses found a directionality pointing to improvements in PTSD enhancing substance use outcomes and not the reverse. Additionally, both analyses did ascertain a differential treatment impact such that for those in the trauma groups who derived a significant benefit on their PTSD symptoms, substance use (alcohol and cocaine use symptoms) was significantly reduced. These findings provided support for developing and implementing already existing evidence based treatments that can target PTSD symptoms more directly in order to impact substance use outcomes.

4.4. Summary

As can be seen in the number of findings from the 19 manuscripts, there were a large number of analyses performed from this single moderately-sized trial. Due to this multiplicity and the lack of adjustments for the many comparisons, caution has been made when interpreting the results. In examining the published articles with the aim of trying to conduct a correction for multiple comparisons across secondary analyses, it became quickly clear that because not all exact p-values were provided in the publications (e.g., just noting the p-value was <0.05), no post hoc adjustment for multiple comparisons could be readily performed. In order to facilitate a reader's ability to put all analyses from a particular RCT in context, it is vital that published manuscripts include the actual p-values for significant and nonsignificant findings. Due to the lack of randomization of the mediator, these results have been used to generate hypotheses, rather than to make causal inferences. Nonetheless, while the many results must be interpreted conservatively, there does appear to be a convergence of evidence suggesting that Seeking Safety can improve different facets of symptomatology and quality of life among subpopulations of clients. The WTS is an example of a clinical trial with a plethora of prespecified secondary analyses, with no multiple comparisons adjustment, but also with no overreaching inferences or implications of causality in manuscript abstracts or discussion sections.

Overall, despite these important and illuminating findings, some questions remain on how to interpret these secondary analyses given the fact that the primary outcome analysis failed to detect differential benefits of the trauma therapy for the intent-to-treat sample on PTSD and abstinence outcomes. There is a tension between the “no differences” primary outcome findings and the secondary analyses which suggested (a) differential benefits favoring the trauma-focused treatment and (b) impact of treatment on substance use outcomes. Did the treatment work or did it not work? However, while this arguably may be a straightforward and valuable question to pursue, its focus is on a “one size fits all” model. Other equally relevant questions include: For whom did the treatment work? How did the treatment work? What areas of functioning were impacted and over what time frames? How does the way a person attend treatment influence outcomes? These additional inquiries address personalized treatment questions that are particularly relevant to treatment development and to clinical providers. Secondary analyses of the WTS study have helped to answer many of these more nuanced questions and gave clinicians direction for how to use trauma-focused addiction therapy.

5. Future directions

This section makes several suggestions for maximizing the impact of secondary data analyses for effectiveness trials. Innovative study designs and data analytic methods should be considered to further the field of addiction science, especially in terms of the complex presenting issues of these heterogeneous patient populations.

5.1. The “virtual” multisite clinical trial

Integrative data analyses are secondary data analytic designs including retrospective cohort studies and cross-sectional studies that pool data from other studies including naturalistic research settings, clinical trials, and practice-based research settings. These designs, which have most commonly been applied in developmental longitudinal research thus far can offer rich information that can be integrated to determine which items best capture outcomes of interest while remaining sensitive to detecting change due to an intervention (Curran & Hussong, 2009). Pooling data from observational and other clinical trials allows for increased sample size and analysis of heterogeneous diagnostic presentations depending on the research question of interest and for the phenomenon seen in various clinic settings. Subgroup analyses to examine the sensitivity of prevalence methods across demographic sub-populations (e.g., socio-demographic and baseline clinical characteristics) with larger and more heterogeneous samples would increase our ability to make inferences to treatment efficacy for a broad range of patients. Resources needed for such studies would be moderate relative to a well-powered RCT, but the benefits would assist many future treatment efforts. Data from multiple studies can be pooled and propensity score matching can be used to compare long-term outcomes of interventions.

5.2. MOST (multiphase optimization strategy)

MOST is a comprehensive framework inspired by engineering models that seeks to optimize the evaluation and dismantling of multi-component behavioral interventions. MOST uses RCTs as a component of intervention evaluation, but also uses data from developmental phases of intervention testing in the analyses. In an iterative framework, MOST analyses rely on experimental designs to answer questions efficiently, thereby maximizing intervention study results without increasing the demand for new resources (Collins et al., 2011). Analytic strategies such as these will provide effectiveness and implementation research more flexibility in approach.

5.3. Design issues and solutions

As we have discussed previously, there are several methods that can be used during the study design phase to ensure sound secondary analysis possibilities. One of the key criticisms of secondary analyses is multiple testing. To address this during trial design, one can specify a priori how the multiple comparisons will be handled. For example, if there are two secondary outcomes, which are anticipated to be crucial in assessing treatment efficacy/effectiveness, then a separate type I error can be assigned to those two outcomes. The method of Hochberg (1988) can be used to maintain that type I error rate while testing both outcomes. For example, if there are two outcomes and an overall type I error rate of 5% then the testing proceeds as follows: (i) compare both p-values to 0.05, if both are less, then both outcomes are statistically significant; if only one p-value is less than 0.05, then that outcome is statistically significant only if that p-value is less than 0.025 (0.05/2); otherwise neither is statistically significant. To circumvent the criticism of ad hoc data mining, one must pre-specify the key secondary outcomes in either the protocol or the statistical analysis plan. This will only be effective if there are a limited number of secondary outcomes and analyses pre-specified, as the larger the list the less useful pre-specification becomes. The outcome measures and analytic approaches selected for pre-specification should be clinically relevant (e.g., reflect patient-centered outcomes), or deal with potential statistical issues with the primary outcome (e.g., sensitivity analyses for missing data methodologies). They should also have a sound scientific basis with a clear rationale for consideration, without which pre-specification may not preclude a data dredging criticism. During the pre-specification there must be a balance between the need for exploring the data and maintaining the rigor of a randomized controlled trial.

When there are key secondary outcomes that are considered vital to assessing efficacy/effectiveness, one option to consider during the design phase is actually adjusting the sample size to ensure sufficient power for these outcomes in addition to the primary measure. A recent study of cannabis cessation was powered to test for treatment effectiveness on the odds of a cannabinoid-negative urine during the active treatment phase (primary outcome measure), as well as end-of-treatment abstinence (McClure et al., 2014). To ensure sufficient power for the latter outcome, the sample size was increased beyond that needed for 80% power to detect a treatment effect on the primary outcome measure. By ensuring enough power to detect an effect on a key secondary outcome measure, it allows greater confidence in a negative finding.

While drafting the protocol and/or statistical analysis plan it is also important to thoroughly discuss missing data. The different ways that missing data can impact the study results should be mentioned and the different methods for handling missing data should be identified a priori. Since it has been shown that study conclusions may differ based on the method of incorporation of missing data (e.g., Hien, Cohen, Campbell et al., 2009; Hien, Wells, Jiang, et al., 2009; Hien et al., 2012; Morgan-Lopez & Fals-Stewart, 2007), it is imperative to identify the method(s) of analyses before data lock to avoid the appearance of selection bias. Ultimately, ensuring a published review of all papers achieving publication from an existing effectiveness trial data set, similar to the one provided here, would help to maximize the impact of secondary data analyses for effectiveness trials.

6. Conclusion

Towards the aim of highlighting the benefits for treatment translation of carefully specified secondary analyses, this paper has reviewed the strengths and limitations of randomized controlled trials, specifically as related to treatment for co-occurring substance use disorders and PTSD. We present the rationale for enhancing individualized treatment questions for the field specifically through the use of secondary analysis to better address the complex presenting issues of clients (and arguably the more important questions) that relate to for whom and how interventions work. The breadth and clinical importance of secondary analyses are exemplified through the case example of the NIDA Clinical Trials Network “Women and Trauma” Study, the largest effectiveness trial to date examining treatment for co-occurring SUD and PTSD. This paper offers a number of possibilities for developing and conducting scientifically strong secondary analyses. Given the continuing gap between research and practice, appropriately executed secondary analytic studies are an important step in addressing questions that have real-world value to community clinicians. If researchers can successfully translate these findings into clinically usable information, it could improve clinicians' beliefs and perceptions of the relative advantage (over current or other practices) and compatibility (fit with end user needs) (Rogers, 2003) of interventions for specific patient populations. In turn, this is likely to broaden the adoption and implementation of evidence-based treatments. Moving forward, planning for and description of secondary analyses in large efficacy and effectiveness RCTs, which encumber extensive economic resources and time, should be given equal consideration and care to the primary outcome analysis (Benjamini & Hochberg, 1995; Chambless & Ollendick, 2001; Fairburn & Wilson, 2013; Kar et al., 2013; Tang et al., 2014; White, Horton, & Carpenter, 2011; Xi et al., 2013).