Effectiveness of Remune

Churdboonchart et al. (2) paint an impressive picture of the effects of Remune on CD4 cell count in HIV-infected subjects. However, as individuals who were closely involved in the study, we believe that the paper presents a misleading account of the study results and a distorted view of the beneficial effects of Remune in this population.

We became involved in this study during its planning stages as a result of a request from Thailand's AIDS Subcommittee for HIV Vaccine Trials, National Commission for the Prevelopment and Control of AIDS, to provide statistical expertise. We were integrally involved in the study design, contributed to the development of case report forms, provided data management training for the sites and for Dr. Churdboonchart's staff, set up the study randomization and held the blinded treatment codes during the study, and gave presentations to the study's Data and Safety Monitoring Board (DSMB) on the role of DSMBs and the planned interim analysis for this study. One of us (S.K.) served as a member of the DSMB. We developed an analysis plan for the final study data that was approved by Dr. Churdboonchart, conducted the interim analysis of the study and presented this to the DSMB, and prepared a final statistical report on the study results.

The prespecified primary analysis of CD4 cell count for this study used the summary statistic approach (3), in which a slope was computed for each subject by fitting a linear regression to his or her log-transformed CD4 measurements at weeks 0, 12, 24, 36, and 40 and where the resulting slopes were then compared between the Remune and placebo (incomplete Freund's adjuvant) groups using the van der Waerden nonparametric test (4). This prespecified primary analysis of CD4 cell count yielded a P value of 0.34, indicating no significant difference between the Remune and placebo groups.

There were also several prespecified secondary analyses of the CD4 endpoint, all based on computing a single summary statistic for each subject and then comparing the Remune and placebo groups using the van der Waerden test. These secondary analyses were based on using untransformed (as opposed to log-transformed) CD4 counts, two alternative metrics to the CD4 slope (change between baseline and week 40 and normalized area under the CD4 curve [AUC]), and an alternative method for calculating an individual CD4 count based on the “averaging” method described by Churdboonchart et al. (2). The analyses based on the averaging method were added as secondary analyses at the request of Dr. Churdboonchart at the completion of the study. Table 1 lists the results of the primary analysis of CD4 counts and the nine secondary analyses included in our final report. We have not adjusted any of these P values to control for the inflated false-positive rate that arises when multiple tests are conducted (American Statistical Association [ASA] ethical guidelines for statistical practice [http://www.amstat.org/profession/ethicalstatistics.html]).

TABLE 1.

Statistical analyses of CD4 count

Method for calculating CD4 count	Log transformation	Metric	Pa
1, original	Yes	Slope	0.34b
2, recalculation	Yes	Slope	0.36
3, original	Yes	Change by wk 40	0.34
4, recalculation	Yes	Change by wk 40	0.20
5, original	No	Change by wk 40	0.13
6, recalculation	No	Change by wk 40	0.07
7, original	Yes	AUC	0.11
8, recalculation	Yes	AUC	0.07
9, original	No	AUC	0.044
10, recalculation	No	AUC	0.024

^aNot adjusted for multiple comparisons. 

^bPrespecified primary analysis. 

Note that the primary analysis and seven of the nine secondary analyses of CD4 cell count fail to demonstrate a statistically significant difference between the Remune and placebo groups. Although some of the secondary analyses suggested a possible difference between the Remune and Placebo groups, the multiplicity of tests undertaken, as well as the fact that these were secondary analyses, argues against much emphasis being placed on them (ASA guidelines [see above]). Accordingly, the final statistical analysis of the study that we prepared for Dr. Churdboonchart and the AIDS Subcommittee noted that while some of the secondary analyses were suggestive of a possible association between Remune and CD4 count, the study data overall did not demonstrate a significant difference in CD4 between the Remune and Placebo groups.

In contrast to these results, Churdboonchart et al. present only a single analysis of CD4 count, corresponding to analysis 10 in Table 1 but using the Wilcoxon test (4) instead of the van de Waerden (the Wilcoxon and van der Waerden tests usually give similar results, and in this case the Wilcoxon test gives P = 0.03 as opposed to the value of 0.024 in Table 1). Churdboonchart et al. do not acknowledge that the analysis of CD4 count they present was not the prespecified primary analysis, do not report the prespecified primary analysis, and do not acknowledge that their reported analysis was just one—the most statistically significant one—of multiple secondary analyses of CD4 count.

The pitfalls associated with reporting only selective analyses in a scientific report are well-known (1; ASA guidelines [see above]). For example, the ASA ethical guidelines state:

Running multiple tests on the same data set at the same stage of an analysis increases the chances of obtaining at least one invalid result. Selecting the one “significant” result from a multiplicity of parallel tests poses a grave risk of an incorrect conclusion. Failure to disclose the full extent of tests and their results in such a case would be highly misleading.

In our opinion, the paper by Churdboonchart et al. gives a distorted account of the clinical trial by virtue of its incomplete and selective reporting of the CD4 cell count results.