Assay sensitivity and study features in neuropathic pain trials

Abstract

Objective:

Our objective was to identify patient, study, and site factors associated with assay sensitivity in placebo-controlled neuropathic pain trials.

Methods:

We examined the associations between study characteristics and standardized effect size (SES) in a database of 200 publicly available randomized clinical trials of pharmacologic treatments for neuropathic pain.

Results:

There was considerable heterogeneity in the SESs among the examined trials. Univariate meta-regression analyses indicated that larger SESs were significantly associated with trials that had 1) greater minimum baseline pain inclusion criteria, 2) greater mean subject age, 3) a larger percentage of Caucasian subjects, and 4) a smaller total number of subjects. In a multiple meta-regression analysis, the associations between SES and minimum baseline pain inclusion criterion and age remained significant.

Conclusions:

Our analyses have examined potentially modifiable correlates of study SES and shown that a minimum pain inclusion criterion of 40 or above on a 0 to 100 scale is associated with a larger SES. These data provide a foundation for investigating strategies to improve assay sensitivity and thereby decrease the likelihood of falsely negative outcomes in clinical trials of efficacious treatments for neuropathic pain.

Multiple efficacious medications have been identified for the treatment of patients with neuropathic pain.^1–4 However, these treatments have significant limitations. For example, in placebo-controlled randomized clinical trials (RCTs), no more than 60% of patients with neuropathic pain experience clinically important pain reductions.⁵ In addition to modest efficacy, there are appreciable side effects and safety risks. These limitations of existing medications for neuropathic pain provide a compelling impetus for the development of treatments with improved efficacy, safety, and tolerability.

A major challenge in developing improved treatments for neuropathic pain is that a number of recent neuropathic pain RCTs have found that the medications being evaluated did not significantly differ from placebo in conditions in which their efficacy was previously demonstrated and for which they had been approved by regulatory agencies.^6–8 Assuming that previous evidence of efficacy was valid and that patients and outcome measures in these studies were comparable, such results may reflect a failure of the RCTs themselves to demonstrate analgesic effects.

Failure to demonstrate statistically significant evidence of efficacy can reflect limited assay sensitivity, which has been defined as the ability of a clinical trial “to distinguish an effective treatment from a less effective or ineffective treatment.”⁹ Identifying patient, research design, and study site characteristics that are associated with assay sensitivity has the potential to facilitate the development of truly efficacious treatments.^10–12 The objective of our investigation was to identify factors associated with assay sensitivity in a large sample of publicly available neuropathic pain trials.

METHODS

Study inclusion and exclusion criteria.

The Analgesic, Anesthetic, and Addiction Clinical Trial Translations, Innovations, Opportunities, and Networks (ACTTION) public-private partnership with the US Food and Drug Administration (FDA)^13,14 has established comprehensive databases of acute and chronic pain RCTs to examine relationships between clinical trial characteristics and assay sensitivity¹⁵ and also to evaluate the responsiveness of different outcome measures.¹⁶ The neuropathic pain trials database, on which our analyses are based, contains all double-blinded, randomized, and placebo-controlled trials of oral, intranasal, topical, and transdermal pharmacologic treatments for neuropathic pain conditions identified through MEDLINE, the Cochrane Collaboration, and other relevant summaries of neuropathic trials^5,11,17 that were publicly available before November 1, 2010. Only studies that had at least 7 days of treatment, used a parallel group or crossover design, and were available in English were included, but RCTs reported only in abstract form were not (for additional information on the search strategy, see appendix e-1 on the Neurology® Web site at www.neurology.org).

Measures.

Variables collected from each trial included the following, when available: 1) eligibility criteria (e.g., type of neuropathic pain, minimum baseline pain intensity); 2) demographic and clinical data; 3) study factors (e.g., research design, duration of therapy, number of sites); 4) active treatments, dosage regimen (e.g., fixed or flexible), and concomitant and rescue analgesic use; and 5) for the pain outcome, baseline and endpoint mean values and either the respective SD or information from which this SD could be derived (e.g., standard error).

For studies that reported data for more than one pain outcome, the numerical rating scale (NRS) was selected as the primary outcome in preference to the visual analog scale (VAS) for 2 trials; and pain assessed for a daytime period was chosen as the primary outcome in preference to pain assessed for a nighttime period for 1 trial. Studies were grouped according to the inclusion criteria for type of neuropathic pain: postherpetic neuralgia (PHN); painful diabetic peripheral neuropathy (DPN); both PHN and DPN; other peripheral neuropathy; central neuropathic pain; and both peripheral and central neuropathic pain.

Data from primary and secondary endpoints using 0 to 10 NRS, 10-cm VAS, and other measures were transformed to a 0 to 100 scale. Although the NRS and VAS may have somewhat different psychometric properties, responses to these 2 types of measures of pain intensity are highly correlated and there is no evidence that their responsiveness to change and to treatment effects differs.

The standardized effect size (SES) was calculated as the ratio of the treatment effect and the within-group SD for both parallel-group and crossover studies. For parallel-group studies, the within-group SD can be calculated by pooling across the treatment groups. For crossover studies, the within-group SD can be calculated as the pooled SD across the placebo condition and the active treatment condition¹⁸ (additional information on SES computation^19–21 is provided in appendix e-1).

Statistical analysis.

Descriptive statistics were used to summarize the characteristics of the studies. Before formal statistical analysis, the relationship between the SES and each continuous study characteristic was examined descriptively to evaluate the linearity assumption. This assumption appeared to be reasonable for all continuous variables examined. Two studies that had an SES greater than 4 were deemed outliers and omitted from the formal analyses.

Acknowledging the expected heterogeneity among the different studies (including eligibility criteria, intervention studied, evaluation protocol, concomitant treatments, and other factors), the relationships between SES and the study characteristics were examined using mixed-effects meta-regression models that treated study as a random effect and each study characteristic as a fixed effect.^22,23 For studies with multiple active treatment arms, the SES values for each treatment arm are correlated because a common placebo group is the basis for comparison. To account for these correlations in the analysis, methods for robust variance estimation were used.²⁴

The inconsistency of the SES values across studies was described using the I² statistic, which is a measure of the proportion of observed variance in SES values that is reflective of true variation in SES. The value of I² falls between 0 and 1, with higher values indicating greater inconsistency among the studies with respect to SES. To examine the associations between study characteristics and SES, we first conducted univariate analyses in which each study characteristic was included alone in the mixed-effects model. The results are reported as regression coefficients for continuous variables and as group mean SES values (and group differences in these) for categorical variables, along with associated 95% confidence intervals (CIs) and p values. An analog R² measure is also reported²⁵; this index also falls between 0 and 1 and indicates the proportion of the variance in the true SES that is accounted for by its association with the study characteristic of interest.

Study characteristics that exhibited a possible association with SES (p < 0.20) in univariate analyses and other characteristics thought to be important based on the existing literature were selected for construction of a multiple meta-regression model. Because of limitations of existing meta-analysis software, exploratory model selection procedures were implemented using fixed-effects models and treating multiple SES values within a study as independent. Three model selection procedures (forward, backward, and stepwise) were applied and all selected the same study characteristics. These characteristics were then included in a mixed-effects, multiple meta-regression analysis.

Analyses focused on RCTs of efficacious treatments, which were considered those medications approved by regulatory agencies for neuropathic pain indications or considered first- or second-line in published treatment recommendations.^1–4 Given the difficulty of establishing which other treatments truly lack efficacy, supplementary analyses are presented in tables e-2 and e-3 for RCTs of all treatments irrespective of whether they are currently considered efficacious or not.

A funnel plot of the SES against the standard error of the SES, along with the Egger test for funnel plot asymmetry, was used to assess possible publication bias.²⁶

All data were analyzed using SAS statistical software, version 9.2 (SAS Institute, Inc., Cary, NC) and R functions provided by Hedges et al.²⁴ A 2-tailed significance level of 5% was used for all analyses.

RESULTS

Study characteristics.

As shown in figures 1 and 2, the ACTTION neuropathic pain trials database includes 200 RCTs comprising 284 treatment groups. It was possible to either calculate or impute an SD for 225 treatment groups, and after the exclusion of 2 outliers,^27,28 there were 106 treatment groups for efficacious medications and 117 treatment groups for medications with uncertain or unknown efficacy. The characteristics of the 106 treatment groups for efficacious medications are presented in table e-1, which shows that subjects in these groups had pain at baseline that on average was moderate in intensity (i.e., 60 on a 0–100 scale) and an average pain duration of approximately 4 years. Most of the treatment groups (72%) used an NRS, 11% used a VAS, and 17% used other pain measures. Parallel-group designs were nearly 3 times more common than crossover designs, and the mean duration of the treatment periods was more than 7 weeks. Approximately half of the RCTs involved painful DPN or another painful polyneuropathy. The test for symmetry of the funnel plot of the trials included in the analyses (figure 3) was significant (p < 0.01), indicating the possible presence of publication bias. Table e-2 presents characteristics of the 223 treatment groups for all medications, which were generally similar to those of the treatment groups from trials of efficacious medications.

Figure 1. PRISMA flowchart.

Figure 2. Flow diagram for neuropathic pain trials included in the analyses.

* Imputed SD as the sample median of SDs from the studies that used the same pain scale. CI = confidence interval; SES = standardized effect size.

Figure 3. Funnel plot.

Univariate analyses.

There was considerable heterogeneity among the SES values derived from the treatment groups from trials of the efficacious medications (I² = 65.0%; 95% CI = 57.0%–71.4%). Table 1 presents the associations between study characteristics and SES for the 106 treatment groups from these trials. Larger SESs were significantly associated with trials that had 1) minimum baseline pain inclusion criteria of 40 or 50 vs 30 on a 0 to 100 scale, 2) greater mean subject age, 3) a larger percentage of Caucasian subjects, and 4) a smaller total number of subjects. As suggested in previous studies,²⁹ trials with longer treatment periods tended to have smaller SESs, but the association was not significant (p = 0.11).

Table 1.

Univariate analyses of relationships between study characteristics and standardized effect size (n = 106 treatment groups)

There were no significant associations between SES and any of the other study characteristics—for example, research design (i.e., parallel group vs crossover), number of treatment arms, number of study sites, trial-quality rating,³⁰ and specific type of pain—several of which have been associated with assay sensitivity in previous research on various chronic pain or psychiatric conditions.^10–13 Although some of the relationships between SES and these study characteristics were in the hypothesized direction in the analyses of the trials of efficacious medications, they were also not significant when all trials were examined, except for a significant relationship between larger SES and shorter treatment-period duration (table e-3).

Multiple meta-regression analysis.

As would be expected, several of the study characteristics we examined were associated with each other (e.g., number of sites and number of subjects randomized). Accordingly, a multiple meta-regression analysis was performed that included minimum baseline pain, trial-quality rating, age, number of patients enrolled, number of study sites, research design, mean baseline pain, treatment-period duration, and type of pain. The percentage of Caucasian patients was not included because of the number of treatment groups missing this information (42 of 106).

Minimum baseline pain inclusion criterion and age were both statistically significantly associated with SES in the final model (table 2). The results indicated that baseline pain inclusion criteria of 40 or 50 vs 30 on a 0 to 100 scale and greater mean age were each associated with a larger SES. In addition, trials with more than one study site had a smaller mean SES than those with only one study site, although the difference was not statistically significant (p = 0.052). The I² statistic was 63.0% (95% CI = 54.3%–70.0%) for the 102 treatment groups included in this multiple meta-regression analysis of trials of efficacious medications.

Table 2.

Multiple regression analysis of relationships between study characteristics and standardized effect size (n = 102 treatment groups)

DISCUSSION

We examined the associations between the assay sensitivity of neuropathic pain RCTs, as measured by the SES, and subject, research design, and study site characteristics. Univariate analyses indicated that larger SESs were significantly associated with trials that had 1) greater minimum baseline pain inclusion criteria, 2) greater mean subject age, 3) a larger percentage of Caucasian subjects, and 4) a smaller total number of subjects. In a multivariate meta-regression analysis performed to identify the study characteristics most highly associated with assay sensitivity, the associations between SES and minimum baseline pain inclusion criterion and age remained significant, and there was a suggestion that the use of only one study site may be associated with a larger SES.

One of the most important implications of our results is that potentially modifiable correlates of assay sensitivity can be identified. Specifically, we found that minimum baseline pain inclusion criteria below 40 on a 0 to 100 scale were associated with a smaller SES. This result is consistent with an analysis of 3 clinical trials of painful DPN that suggested that medication vs placebo differences were greater in subjects with greater baseline pain intensity.³¹ Considered together, these data suggest that neuropathic pain RCTs should use a pain inclusion criterion of at least 40 on a 0 to 100 scale (alternatively 4 on a 0–10 scale) to decrease the likelihood that studies of efficacious treatments will have falsely negative outcomes. We also found that age was associated with assay sensitivity, with greater subject age being associated with larger effect sizes. This relationship was not predicted and is difficult to explain. It is possible that it reflects, at least in part, the generally older ages of patients with specific pain conditions—especially PHN—that might have greater assay sensitivity,¹⁵ although we did not find any evidence of an association between effect size and pain condition in our analyses. Despite the fact that it would be possible to modify demographic inclusion criteria such as age in future RCTs, doing so could markedly reduce the generalizability of the results.

In addition, we found some evidence that smaller trials and those using only a single site had larger SESs, although these relationships were either limited to the univariate analyses or were not significant in the multiple meta-regression analysis. It has been suggested that such “small study” and single-site effects in RCTs of osteoarthritis³² and other conditions³³ might reflect various sources of bias, including publication bias (i.e., small negative trials may be more likely to remain unpublished). It has also been acknowledged, however, that such effects might reflect more careful treatment implementation, greater study-site expertise, or the enrollment of patients who are more likely to respond to treatment.^32,33 It is difficult to evaluate these different explanations; nevertheless, the major limitations of large multicenter clinical trials and recent proposals for transforming the international clinical trial enterprise³⁴ could be considered in interpreting these data.

It has been argued that one strategy for increasing the assay sensitivity of analgesic trials would be to reduce the magnitude of the placebo response, and that strategies for decreasing placebo response rates should therefore be investigated.^{11–13,15,35,36} We have not examined whether the associations we found between assay sensitivity and study characteristics reflected differences in placebo group response, active treatment group response, or variability. However, in considering relationships between placebo group response and assay sensitivity, it is important to emphasize that efforts to reduce placebo group response rates may also be associated with decreased response to active treatment, so decreasing placebo responses by modifying study characteristics may not necessarily increase the assay sensitivity of a trial.

Important methodologic limitations of the present study must be acknowledged. First, the characteristics of the RCTs we examined varied greatly, including trials of different peripheral and central pain conditions, sensory phenotypes (e.g., presence of allodynia), medications, eligibility criteria (e.g., pain intensity, disease duration), and outcome measures. This heterogeneity was reflected in the reported I² values and was addressed by our use of mixed-effects models. A second limitation involves interpretation of the results of published RCTs when many studies are unpublished, and the funnel plot showed that such biases might be present in our data.²⁶ Negative trials, which would typically have smaller SESs, are those most likely to remain unpublished,³⁷ perhaps especially when there is a small sample size or the study has been sponsored by industry. Our conclusions about the contribution of study characteristics to assay sensitivity will need to be re-evaluated when more comprehensive databases of clinical trials become available.³⁸

In addition, the study characteristics we analyzed were based on those frequently reported for analgesic RCTs. There may be interactions between these characteristics but these were not formally evaluated. Furthermore, some characteristics may be surrogates for different factors; for example, race may reflect the predominant race in the region where the study was conducted, which might reflect other potentially important characteristics such as access to health care. Another limitation of our study involves the difficulty of adequately distinguishing medications that truly lack efficacy in the condition studied from trials of efficacious medications that have not demonstrated benefit because of poor assay sensitivity (or chance). We focused our analyses on medications that have been approved by a regulatory agency or considered first- or second-line in recent treatment guidelines, but the results remained generally similar when analyses of all medications were conducted (table e-3). This suggests that our findings were not substantially affected by the exclusion of any trials of medications in conditions for which efficacy has not yet been demonstrated but might ultimately be established.

An additional limitation of our analyses is that they are retrospective analyses of aggregated study-level data. Whenever possible, relationships between study characteristics and assay sensitivity should be confirmed with analyses of patient-level data, ideally conducted with prospectively specified hypotheses. Doing so not only addresses the potential for “ecological bias” that is associated with group-level analyses,³⁹ but also makes it possible to examine subject characteristics that cannot be evaluated in study-level data (for example, the within-subject variability in baseline daily pain reports).^12,40 Finally, the retrospective analyses we conducted cannot establish causal relationships between the demographic and clinical factors we examined and assay sensitivity. To determine causal effects of patient characteristics and study design factors on assay sensitivity, it will ultimately be necessary to prospectively test these relationships in RCTs designed for this purpose, when possible.

Despite these limitations, our results suggest that various subject, research design, and study site characteristics can be associated with the assay sensitivity of RCTs of medications for neuropathic pain. Improved understanding of such relationships has the potential to increase assay sensitivity and thereby decrease the likelihood of trial failure, reduce the number of subjects required for trials, and ultimately accelerate the development of improved treatments for neuropathic pain.

GLOSSARY

ACTTION

Analgesic, Anesthetic, and Addiction Clinical Trial Translations, Innovations, Opportunities, and Networks

CI

confidence interval

DPN

diabetic peripheral neuropathy

FDA

US Food and Drug Administration

NRS

numerical rating scale

PHN

postherpetic neuralgia

PRISMA

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

RCT

randomized clinical trial

SES

standardized effect size

VAS

visual analog scale

AUTHOR CONTRIBUTIONS

All authors contributed to study concept or design, data interpretation, and critical review and revision of the manuscript. Dr. Dworkin drafted the manuscript except for the statistical analysis section, and Drs. He and McDermott performed the statistical analyses and drafted the statistical analysis section of the manuscript.

STUDY FUNDING

Financial support for this project was provided by a research contract (HHSF 223201000078C) from the FDA to the University of Rochester, and by the ACTTION public-private partnership with the FDA, which has received research grants or other revenue from the FDA, various pharmaceutical companies, and other sources.

DISCLOSURE

The views expressed in this article are those of the authors and no official endorsement by the FDA or the pharmaceutical companies that provided unrestricted grants to support the activities of ACTTION should be inferred. R.H. Dworkin has received in the past 2 years research support from the FDA (HHSF 223201000078C; U01 FD004187-01) and the US NIH (1R01 AR059102-01A) and personal compensation for activities involving clinical trial research methods and/or reimbursement of travel expenses associated with these activities from Adynxx, Allergan, Analgesic Solutions, Anika, Astellas, Avanir, Axsome, Bayer, Biogen, Bioness, Bristol-Myers Squibb, Cardiome, Collegium, Depomed, Depuy, Eli Lilly, Epicept, Flexion, Genzyme, Glenmark, Inhibitex, Johnson & Johnson, Medicinova, Merck, MMS Holdings, NeurogesX, Ono, Pfizer, Phillips, Prolong, Q-Med, Regenesis, Sanofi-Aventis, Smith & Nephew, Spinifex, Takeda, Taris, Teva, Theravance, and Xenon. D.C. Turk has received research support from Endo, Johnson & Johnson, Philips Respironics, the US NIH (NIAMS R01AR059102; NIDA 1R01DA032776, 1R01DA026887), and the FDA (HHSF223201000078C; U01 FD004187), and consulting fees from Eli Lilly, Ortho-McNeill Janssen, Pfizer, Philips Respironics, and SK LifeScience in the past 2 years. S. Peirce-Sandner and H. He report no disclosures. M.P. McDermott has served on Scientific Advisory Boards for the ALS Association, Biogen, Isis, and Muscular Dystrophy Association; has received research support from the US Centers for Disease Control and Prevention, FDA, US NIH, and the American Dental Association, Boehringer Ingelheim, Endo, Forest, Medivation, Michael J. Fox Association, Muscular Dystrophy Association, NeuroSearch Sweden, Pfizer, and Spinal Muscular Atrophy Foundation; and has received consulting fees from Bioness, Impax, New York State Department of Health, Smith & Nephew, Synosia, and Teva. J.T. Farrar has served in unpaid roles with the American Pain Society, Institute of Medicine, and National Academy of Science; has received research support from US government grants (UL1-RR024133; U01-AG030644; R01-CA158243; VA-DOD Subcontract D45061; DOD subcontract BD514845; R01-AR064153); and a foundation grant from Altarum, Inc. that serves as a contractor for the NIH Center of Excellence in Pain Education; and a research grant from Depomed, Inc.; and has served as a consultant to AstraZeneca, Bristol-Myers Squibb, Covidien, Dara, Depomed, Dr. Reddy’s Laboratories, Forest, Janssen, Johnson & Johnson, Merck, Prolong, Teva, and United Biosource. N.P. Katz is the owner of Analgesic Solutions, which has multiple pharmaceutical industry clients. A.H. Lin and B.A. Rappaport report no disclosures. M.C. Rowbotham has received consulting income from Abbott, Allergan, Arcion, Bristol-Myers Squibb, Cardiome, Kyowa, Nektar, Xenon, and Zalicus; served on a Scientific Advisory Board for Adynxx, Galderma, and Sanofi; and holds stock options in Afferent Pharmaceuticals. Go to Neurology.org for full disclosures.