Uncommon use of common measures in sulforaphane trial

I write to comment on the sulforaphane trial in autism (1). Forty-four subjects were randomly assigned to sulforaphane or placebo in a 2:1 ratio. Four subjects withdrew without postrandomization measurements; thus, the report is based on 40 subjects. The abstract states that the study applied common outcome measures. This is true and not true. The primary outcome, Social Responsiveness Scale (SRS), was designed as a population screening measure of social impairment and repetitive behavior as a single trait. It was not developed as an outcome measure (2). Although it has been used as an outcome measure, it is inaccurate and misleading to claim that it is commonly used. The five subscales mentioned in the report are the product of expert consensus rather than factor analysis. It is not clear what these subscales actually measure, and the meaning of change on the 65-item total score is also unclear.

The Aberrant Behavior Checklist (ABC) is indeed a commonly used outcome measure. However, the use of the total score on the 58-item ABC is not common for good reason. The ABC includes five subscales. Factor analysis indicates that these subscales are statistically separate, with correlations ranging from 0.31 to 0.73 across subscales (3). Simply stated, very different behavioral profiles across individuals could result in the same total score. Thus, the use of the total score as an outcome measure provides little information on what actually improved. The more common approach is to select a subscale that is relevant to the treatment target as the primary outcome measure such as irritability (4) or hyperactivity (5). The report provides line graphs showing change scores on ABC subscales. Unfortunately, baseline scores are not provided, making it impossible to compare these results to other studies that have used the ABC. In addition, the differences on these subscales between active and placebo are modest.

The application of the Clinical Global Impression Improvement (CGI-I) scale in this study was also uncommon. In most clinical trials in the autism population, a single CGI-I is used to reflect overall change. In this study, 10 CGI-Is were used, and 3 were significant. On the overall CGI-I, no subjects in either treatment group were rated as much improved or very much improved. Here again, it will be difficult to compare these findings with past and future trials that use the more common approach to rating the CGI-I.

The study also included a 4-wk recovery phase. It is not clear from the report whether this study phase was placebo controlled. Placebo-controlled discontinuation is an accepted design to evaluate efficacy. Open discontinuation, however, is subject to bias. In addition, the sample was nearly reduced by half at the posttreatment assessment. Collectively, the uncommon application of outcome measurement and design ambiguities make it difficult to interpret these findings.