Long-acting injectable versus daily oral antipsychotic treatment trials in schizophrenia: pragmatic versus explanatory study designs

Abstract

Trial design characteristics related to the explanatory : pragmatic spectrum may contribute toward the inconsistent results reported in studies comparing long-acting injectable (LAI) versus daily oral antipsychotic (AP) treatments in schizophrenia. A novel approach examined the hypothesis that a more pragmatic design is important to show the advantages of LAI versus oral APs. A literature search identified comparative studies assessing the clinical efficacy/effectiveness of LAI versus oral APs in more than 100 schizophrenia patients, with 6-month or more duration/follow-up, and published between January 1993 and December 2013 (n=11). Each study’s design was rated using the six-domain ASPECT-R (A Study Pragmatic : Explanatory Characterization Tool-Rating). Nonparametric Wilcoxon rank-sum tests compared ratings of studies supporting (n=7) and not supporting (n=4) a LAI advantage. ASPECT-R total and domain scores were significantly higher (more pragmatic) in studies finding a LAI versus oral AP treatment advantage than those that did not. The rank order of this significance among domains was as follows: ‘participant compliance assessment’ (P=0.005), ‘medical practice setting/practitioner expertise’ (P=0.006), ‘intervention flexibility’ (P=0.007), ‘follow-up intensity/duration’ (P=0.009), ‘primary trial outcomes’ (P=0.012), and ‘participant eligibility’ (P=0.015). Findings support that more pragmatic, less explanatory design features are important to show advantages for LAI treatment. Explanatory studies may introduce features that obscure advantages related to adherence.

Introduction

Schizophrenia has remained a chronic and often severely impairing mental disorder despite the development of effective antipsychotic (AP) treatments. One of the reasons for relapses is nonadherence with prescribed treatment (Kane et al., 2013a). To improve treatment adherence and outcomes, long-acting injectable (LAI) formulations of APs have been developed. The potential benefit of treatment delivered as a LAI versus a daily orally administered AP agent lies in advantages associated with removing the need for daily medication administration and signaling the clinician when nonadherence occurs. Treatment discontinuations (Zipursky et al., 2014) and intermittent treatment (Sampson et al., 2013) have been associated with increased relapses. Treatment with LAIs should increase the likelihood of continuous effective exposure over extended periods. An increasing number of published studies have compared the effects of LAI and oral APs in patients with schizophrenia. On the basis of the association between nonadherence and relapse, these studies hypothesized an advantage for the LAI treatment. Although mirror-image studies, which arguably include more broadly representative patient populations, have reported advantages on the basis of this difference in modality (Kishimoto et al., 2013), randomized-controlled trials have frequently failed to show advantages (Fusar-Poli et al., 2013; Kirson et al., 2013; Kishimoto et al., 2014; Buckley et al., 2015).

Although highly controlled studies are the gold standard to address many clinical research questions, we believe that more pragmatic approaches are required to address questions associated with adherence. Pragmatic (often referred to as effectiveness) studies aim for a high degree of external validity, seeking to answer whether an intervention works under usual clinical or ‘real-world’ conditions. In contrast, explanatory (often referred to as efficacy) studies aim for a high degree of internal validity, exploring whether an intervention works under more constrained conditions. To achieve this goal, explanatory trials are conducted under highly controlled and well-defined treatment conditions, which are necessary to minimize ambiguity and address the primary questions for which this type of trial is designed. They typically include populations that do not reflect the full clinical population in which the intervention will be used. Design elements inherent to explanatory trials may obscure factors that drive the advantage of certain treatment approaches. This is particularly true for studies that address the common issue of nonadherence. For example, the clinical advantage of ensured longer exposure to therapeutic doses with long-acting formulations of AP medications compared with oral formulations may not be evident in an explanatory trial that strongly reinforces adherence. Other explanatory design features that may obscure differences that occur under real-world conditions may include the frequent use of extensive but burdensome healthcare assessments, exceptionally close follow-up and reconnection with the patient, and financial incentives for patient participation. In addition, selection bias may result from the enrollment of participants in clinical trials who tend to be more adherent to research procedures. Individuals with less severe illness and greater insight into their illness may also be more likely to adhere to their assigned treatment regimen (Kane et al., 2013b).

Understanding the inconsistent body of literature comparing LAI and daily oral APs has been the focus of several recent publications. Although an earlier meta-analysis found a significant benefit of LAI versus daily oral APs (Leucht et al., 2011), two larger and more recent meta-analyses of randomized-controlled trials concluded that there is no advantage for LAI formulations in preventing relapse and hospitalization (Fusar-Poli et al., 2013; Kishimoto et al., 2014). The focus of these analyses on controlled, randomized studies likely resulted in a bias toward inclusion of highly explanatory trials. Some authors note that their findings contrast with those of recent naturalistic mirror-image and cohort studies, and suggest that pragmatic trial designs be utilized in future research to be more reflective of actual clinical care received by patients with schizophrenia (Kane et al., 2013b; Kishimoto et al., 2014; Buckley et al., 2015). In particular, these authors expressed concern that patients undergoing intensive consent and assessment procedures may be more adherent and less severely ill than those encountered in everyday practice. Consequently, they suggest that using a LAI AP formulation in a naturalistic setting might confer additional benefit over the corresponding daily oral formulation (Kane et al., 2013a). Supporting this consideration, in randomized-controlled trials where adherence was formally assessed, no differences were observed in adherence between LAI and daily oral AP formulations (Leucht et al., 2011; Kishimoto et al., 2014).

Recently, a meta-analysis by Kirson et al. (2013) was published that included studies of varying designs (randomized-controlled, prospective observational, and retrospective observational trials). These authors reported significant advantages for LAI treatments studied in trials with observational designs, but not in those with randomized-controlled designs. These conclusions are supported in a recent meta-analysis by Kishimoto et al. (2013) with 25 mirror-image studies in which 22 showed significant advantages of the LAI versus daily oral AP treatment for preventing psychiatric hospitalization. However, the authors acknowledge that mirror-image studies can also be biased by the fact that treatment status is not blinded, thresholds for hospitalization can change over time, and that LAIs are always started after suboptimal outcomes on daily oral APs. They also note that reverse mirror-image studies (i.e. from LAI to oral formulation) are lacking.

None of these meta-analyses used a formalized measure of the explanatory or the pragmatic nature of specific trial design features. In practice, most trial designs are neither purely explanatory nor purely pragmatic. Instead, most lie along a continuum between these two extremes. The research reported here uses a novel approach for quantifying an individual study’s design along this continuum and examines the hypothesis that a more pragmatic design is important for showing advantages for LAI versus daily oral AP treatment.

Methods

Literature review

The objective of this review was to identify comparative studies of the clinical efficacy of LAI versus daily oral APs. Selection criteria included studies published from 1993 to 2013, whose duration was 6 months or longer, and that had enrolled at least 100 patients with a diagnosis of schizophrenia. The publication period reflects a time when clinical trial designs were likely to be better described and when treatment modalities that are reflective of current realities were studied. The requirement for a 6-month or longer duration of follow-up was imposed to provide an adequate period for observing potential differences between long-acting and daily oral AP treatments. The 100-patient enrollment criterion was incorporated to increase the likelihood that the study would be sufficiently powered to detect meaningful differences between treatments.

This literature review consisted of three components: (i) a search engine-based literature review; (ii) an examination of relevant review articles; and (iii) any other published studies known to the authors (Fig. 1). The literature search was performed using MEDLINE/PubMed. Search terms and criteria were as follows: (((Antipsychotic) AND schizophrenia) AND ((depot OR injection OR long-acting))) AND oral. Filters included clinical trial, human, English language, and publication dates of 1 January 1993 to 31 December 2013. The manual review of citations identified by MEDLINE/PubMed removed those that: (i) did not include both a LAI and an oral AP treatment arm; (ii) did not include a measure of clinical efficacy or effectiveness; (iii) represented findings from a pooled analysis (vs. a single study); (iv) had a duration of less than 6 months; (v) enrolled less than 100 participants; (vi) were not in English; and (vii) were a secondary publication of a previously included study (i.e. post-hoc subpopulation data). This literature search was then supplemented by an examination of references cited in relevant review articles and any other published studies known to the authors through December 2013.

Fig. 1.

Flow chart of identification, screening and eligibility, and inclusion of clinical trials. AP, antipsychotic; LAI, long-acting injectable.

ASPECT-R, the tool

‘A Study Pragmatic : Explanatory Characterization Tool-Rating’ or ASPECT-R ((c) 2014 Janssen Pharmaceuticals, Inc., Titusville, New Jersey, USA) is a tool informed by the PRECIS tool (Thorpe et al., 2009; Tosh et al., 2011) that characterizes the explanatory : pragmatic nature of a study’s design (L.D. Alphs and C.A. Bossie, 2015, submitted). ASPECT-R considers six study design domains important to the conduct of clinical trials along the explanatory : pragmatic spectrum: (i) participant eligibility criteria; (ii) intervention flexibility; (iii) medical practice setting/practitioner expertise; (iv) follow-up intensity/duration; (v) primary trial outcomes; and (vi) participant compliance assessment. Each domain is rated using a detailed anchored seven-point scale where 0=extremely explanatory; 1=very explanatory; 2=explanatory; 3=elements of both designs; 4=pragmatic; 5=very pragmatic; and 6=extremely pragmatic. Specific descriptive anchors for each of the seven ratings are provided for each of the six domains.

The interclass correlation of the ASPECT-R total score is 0.87 (C.A. Bossie, L.D. Alphs, D. Williamson, L. Mao, C. Kurut, the ASPECT-R Rater Team, 2015, submitted), which corresponds to an excellent inter-rater reliability (Cicchetti, 1994). The domains included in ASPECT-R are generally accepted trial design elements relevant for distinguishing pragmatic and explanatory trials, as evidenced by peer-reviewed publications (Thorpe et al., 2009; Tosh et al., 2011; Lurie and Morgan, 2013; Roche et al., 2013; Alphs et al., 2014; Sedgwick, 2014), which lend support for the face validity of ASPECT-R.

ASPECT-R ratings

Full references of the studies identified were used as the source information for rating the study designs with the ASPECT-R tool. Two of the authors (C.A.B. and L.D.A.) independently rated each of the studies identified by the literature review using the ASPECT-R and then compared their ratings. Differences in domain ratings were resolved through a consensus rating process. The basis of the consensus ratings for each domain for each study was documented.

Illustrating ASPECT-R ratings relative to study results

ASPECT-R consensus ratings for each study were plotted using radar graphs.

Statistical analysis

Studies were then categorized according to the outcome as reported in the original publication, yielding two groups: those showing an advantage for LAI over daily oral AP treatment and those not showing such an advantage. Total and domain ASPECT-R scores were compared across the two groups of studies using the nonparametric Wilcoxon rank-sum test to address the non-normal distribution of the scores. Data were analyzed in JMP5 (5.0.1, 1989–2003; SAS Institute Inc., Cary, North Carolina, USA). All tests were two-sided and α was set at 0.05. No adjustment was made for multiplicity.

Results

Citation review and selection

Using the literature search terms and criteria summarized above, a total of 126 citations were identified through the MEDLINE/PubMed literature search. Three additional citations were identified through manual review of the reference lists of four meta-analyses (Leucht et al., 2011; Fusar-Poli et al., 2013; Kirson et al., 2013; Kishimoto et al., 2014). An additional citation (Bitter et al., 2013) was identified through one author’s (C.A.B.) general knowledge of the literature. Thus, a total of 130 citations were identified (Fig. 1).

One author (C.A.B.) and another contributor (S.R. in acknowledgments) reviewed the titles, abstracts, and full publication of these articles for compliance with search criteria and appropriateness of filters. A total of 119 citations were excluded as they did not fulfill the criteria as described in Fig. 1. The remaining 11 study citations (N=21 159 participants) included: Zhu et al. (2008), Olivares et al. (2009), Gaebel et al. (2010), Kane et al. (2010), Macfadden et al. (2010), Tiihonen et al. (2011), Rosenheck et al. (2011), Grimaldi-Bensouda et al. (2012), Bitter et al. (2013) (methods described in Ascher-Svanum et al., 2006), Keks et al. (2007), and Tiihonen et al. (2006). Study design features and main findings for these 11 studies are summarized in Table 1. The 11 studies were placed into two groups: those that showed a difference between LAI and daily oral AP treatments [seven studies, 18 829 participants (range 297–9567)] and those that did not [four studies, studies, 2330 participants (range 349–1065)].

Table 1.

Study design features and main findings by outcome grouping

Consensus ratings

ASPECT-R ratings of the seven studies concluding a benefit of LAI versus daily oral APs are shown in Fig. 2. Ratings of the four studies concluding no LAI versus daily oral AP difference are shown in Fig. 3. Total ASPECT-R scores (maximum possible score=36) ranged from 18 to 36 in the former group of studies and from 9 to 13 in the latter group (Table 2).

Fig. 2.

ASPECT-R ratings for the seven studies that concluded an advantage of long-acting injectable versus oral daily antipsychotic treatment in patients with schizophrenia. ASPECT-R, A Study Pragmatic : Explanatory Characterization Tool-Rating.

Fig. 3.

ASPECT-R ratings for the four studies that concluded no advantage for a long-acting injectable versus oral daily antipsychotic treatment in patients with schizophrenia. ASPECT-R, A Study Pragmatic : Explanatory Characterization Tool-Rating.

Table 2.

ASPECT-R individual domain and total scores by study outcome and citation

In five of the seven studies concluding a benefit of LAI compared with daily oral AP treatment, all domains were rated as more pragmatic (i.e. ASPECT-R ratings of 4, 5, or 6; Tiihonen et al., 2006; Zhu et al., 2008; Olivares et al., 2009; Tiihonen et al., 2011; Bitter et al., 2013). In one study, most domains were rated as more pragmatic (Grimaldi-Bensouda et al., 2012). In one study, domains were variously characterized as more pragmatic or more explanatory (Gaebel et al., 2010).

In three of the four studies concluding no benefit for LAI compared with daily oral AP treatment, most domains were rated as more explanatory (i.e. ASPECT-R rating of 0, 1, or 2; Keks et al., 2007; Kane et al., 2010; Macfadden et al., 2010). In one study, domains were variously characterized as more pragmatic or more explanatory (Rosenheck et al., 2011).

The mean ASPECT-R total score and individual domain scores were significantly higher (more pragmatic) in the seven studies finding an advantage of LAIs over daily oral APs compared with the four studies that did not (Table 2). The rank order of greatest significant differences in the six domains between the two groups of studies was as follows: ‘participant compliance assessment’ (P=0.005), ‘medical practice setting/practitioner expertise’ (P=0.006), ‘intervention flexibility’ (P=0.007), ‘follow-up intensity/duration’ (P=0.009), ‘primary trial outcomes’ (P=0.012), and ‘participant eligibility’ (P=0.015).

Discussion

A novel quantitative approach was used to examine the hypothesis that a more pragmatic study design is important for showing the advantages of LAI over oral AP treatment for patients with schizophrenia who are frequently nonadherent, increasing the risk of relapse. Theoretical advantages of LAIs are associated with removing the need for daily adherence. Several meta-analytic approaches have been used to examine this question, with mixed conclusions (Leucht et al., 2011; Fusar-Poli et al., 2013; Kirson et al., 2013; Kishimoto et al., 2014). This report describes the application of a new tool, ASPECT-R (L.D. Alphs, C.A. Bossie, 2015, submitted; C.A. Bossie, L.D. Alphs, D. Williamson, L. Mao, C. Kurut, the ASPECT-R Rater Team, 2015, submitted), which quantifies the pragmatic : explanatory nature of a study’s design and explores the relevance of the result to treatment failure, including relapse, hospitalization, and treatment discontinuation. The findings presented here support a hypothesis that explanatory designs introduce features that obscure advantages related to medication treatment adherence, whereas pragmatic design features enable identification of these advantages for LAIs that would be expected in a naturalistic setting for patients who clinicians would select for this treatment. In fact, the range of ASPECT-R total scores for the two groups of studies did not overlap (Table 2).

On the basis of the expected advantage of LAI AP treatment, it was hypothesized that the ‘Participant Compliance Assessment’ domain would be the most differentiating between two groups of studies. Findings were consistent with this hypothesis (P=0.005), although the mean scores for all domains differed significantly between the two groups.

Several limitations of this work must be considered. Studies with conventional (typical) depot AP agents were not well represented (i.e. three studies: Tiihonen et al., 2006; Zhu et al., 2008; Tiihonen et al., 2011). Consequently, it is unclear to what degree findings would translate to work with conventional depot APs. Nevertheless, Kishimoto et al. (2014) have noted that studies of first-generation LAIs [fluphenazine (n=8) and haloperidol (n=1)] show a significant benefit for LAI over oral treatment. Second, only the consensus ratings of two authors (C.A.B., L.D.A.) who developed the ASPECT-R were used for this analysis. Consequently, ASPECT-R ratings found in this study may not be representative of ratings from individuals less familiar with the instrument. However, a recently completed inter-rater reliability assessment with novice, but trained raters found an interclass correlation of 0.87, which corresponds to an excellent inter-rater reliability (C.A. Bossie, L.D. Alphs, D. Williamson, L. Mao, C. Kurut, the ASPECT-R Rater Team, 2015, submitted). Finally, relevant information to fully establish ASPECT-R ratings may not have been fully documented in the primary reports used for this study. Lack of access to source documentation, such as trial protocols, may impact ASPECT-R scores and the ability to assess all domains accurately.

Criteria for our literature search included a 20-year publication date range (1 January 1993 to 31 December 2013). However, a recently published study (PROACTIVE; Buckley et al., 2015) is quite important and relevant to our research question and requires comment (Buckley et al., 2015). The authors state that their time to relapse or hospitalization study of patients with schizophrenia randomized to either a LAI (risperidone) or an oral AP incorporated both explanatory and pragmatic design features. As such, and similar to the findings of the four studies in this analysis that found no difference (Keks et al., 2007; Kane et al., 2010; Macfadden et al., 2010; Rosenheck et al., 2011), these investigators found no significant difference in either time to relapse or hospitalization, and add that their study design is similar to several of these earlier trials. Many of their study design characteristics leaned strongly toward a more explanatory trial, such as uniform and frequent monitoring (i.e. every 2-week office visits) and LAI informed consent treatment requirements that may have diluted the potential for those with documented nonadherence to enroll. In their discussion, the authors acknowledge that these explanatory study design characteristics may have resulted in the enrollment of patients who are more engaged in their care, with a reduced inclusion of participants with documented nonadherence. These types of patients are less likely to stop taking oral medication, making it more difficult to detect differences between the LAI and oral treatment.

In conclusion, this research adds to the previous literature by providing a novel and informative approach that quantifies the pragmatic : explanatory design of studies that compare LAI and oral APs for the treatment of schizophrenia. Previous meta-analytic approaches applied to these studies are based on study results without a detailed and quantitative reference to their specific design and methodological features. The use of ASPECT-R represents a very different approach by providing a structured quantification of specific design elements, without consideration of study results (L.D. Alphs, C.A. Bossie, 2015, submitted; C.A. Bossie, L.D. Alphs, D. Williamson, L. Mao, C. Kurut, the ASPECT-R Rater Team, 2015, submitted). These two distinct approaches to address the same question are complementary and provide more information than either approach alone. Although highly controlled studies remain the gold standard for evidence-based trial designs to answer most questions in medicine and psychiatry, pragmatic study design elements are arguably more valuable for addressing questions such as those related to real-world populations, practice, and outcomes, especially when the primary target is enhancing adherence. Their use can add to the generalizability of available evidence. Our findings suggest that pragmatic study characteristics are important in showing the expected advantage of LAI over daily oral AP treatment in schizophrenia.