Type 1 Error on Type 2 Inflammation: Circular Analysis in Asthma Clustering

Over the past decade and a half, the field of severe asthma research has begun to utilize a form of machine learning called cluster analysis to “learn" latent or hidden phenotypes present in large datasets of clinically relevant variables. The goal of these analyses is to understand the clinical presentation of the patient with moderate and/or severe asthma, with the hopes that treatment can then be personalized to the individual. The approach that has been largely used in the seminal papers of the Severe Asthma Research Program and Haldar and colleagues (1, 2), as well as in many other asthma-phenotyping papers that followed (3–8), has been to first identify clusters and then subsequently compare clusters using a classical statistical test for differences in means, between outcomes, rates, levels of biomarkers, and so forth. This represents, however, a form of a statistical error called circular analysis (Figure 1), also called “double dipping,” whereby a hypothesis is generated on the basis of a dataset and then the test for the hypothesis occurs on the same dataset (9). As Gao and colleagues have emphasized in reference to clustering (10), this leads to an extremely inflated Type 1 error rate and broadly invalidates the statistical inference.

Figure 1.

In circular analysis, data gathering precedes hypothesis generation.

Type 1 error refers to the probability of incorrectly rejecting a “true” null hypothesis. In this context, it is falsely concluding that two clusters differ in a certain variable when, in actuality, they do not. It is common practice to set the acceptable probability of Type 1 error a priori (also known as the α-value cutoff) to 0.05. In the case of a simple experiment (e.g., comparing the FEV₁ between a treatment and control group), interpreting P values is straightforward. A P value less than the cutoff for Type 1 error set a priori represents evidence of statistical significance. However, if we begin our analysis by grouping similar observations (keeping dissimilar observations apart) and subsequently define our hypotheses on the basis of these pregrouped data (e.g., Does the mean of one cluster differ from the mean in another?), the standard P value interpretation is no longer valid. Therefore, we must consider that any estimated P value in our cluster comparison is conditional on the process used to identify the clusters in the first place. In other words, the P value becomes the probability of detecting a difference in means as extreme or more extreme than the one observed completely by chance, given the manner chosen to first cluster the data. Therefore, if the same data are used to both generate and evaluate hypotheses without accounting for the double use of data, this represents circular analysis. Put another way, if the same variables used to derive clusters are subsequently compared using a classical statistical test for a difference in means, the resulting P value is invalid. Indeed, in circular analysis, classical statistical testing between clusters will find the means to be highly different and produce small P values, even if the difference is due to random sampling variation. Hence, P values will be invalid. It may be tempting to consider sample splitting as a solution, yet this does not address the underlying circular reasoning (10).

We considered eight influential papers on asthma clustering (1–8). In all eight papers, the circular analysis described here was used, at least in part. Not all of the hypothesis testing was double dipping, however. For instance, Denton and colleagues (6) had only 3/26 hypotheses that fell into the trap; the remaining hypotheses compared the clusters for variables that were not involved in the clustering process. This avoids the circular analysis trap. However, our review of the foundational studies of the Severe Asthma Research Program (1) and of Haldar and colleagues (2), when taken together, show that over 50% of the hypotheses were a result of circular reasoning. In total, in the eight papers considered, we found 150 of the 330 (45.5%) statistical tests to involve double dipping, excluding supplementary materials. Of these 45.5%, a mere 15/150 (10.0%) of the null hypotheses were accepted at the α = 0.05 level. In comparison, of the remaining 180 tests, 102 (43.3%) of the null hypotheses that did not involve double dipping were accepted. This suggests that there is more overlap in clusters than reported, and where that overlap occurs is unclear (e.g., Does a young allergic cluster truly differ from a young hypereosinophilic asthmatic cluster?). Although our analysis does not include every published clustering study, we feel that our study represents an ample sample to highlight the pervasiveness of this fallacy in efforts to phenotype asthma.

A reader may wonder about applying a Bonferroni correction or a similar multiple-testing P value correction procedure used to control for the type 1 error rate. More broadly, a natural question that arises is: How well-founded are the clusters identified that fell into the circular analysis trap? First, we do not dispute the methodology that first led to the identification of the clusters: Cluster analysis is a powerful form of machine learning that should still be instrumental for identifying phenotypes and endotypes of asthma. We also do not dispute the findings that did not arise from double dipping. Also, we note that some aspects of the clusters appear to be replicated across independent analyses. However, because the tests comparing the clusters are invalid, we cannot conclude on the basis of the analyses conducted that the individual clusters differ significantly at any variable; nor would a P value correction be of any use, as those P values obtained are invalid to begin with.

When data gathering precedes hypothesis generation, performing valid inference is considerably more challenging. Nonetheless, performing valid inference is essential for reproducible medical research. Reanalysis of the clustering results of prior publications using the new methods laid out by Gao and colleagues (10), and it is important to note, combined with external validation, may elucidate previously unknown overlaps between clusters. In doing so, the authors could be able to identify both more precise phenotypes of asthma and the variables that truly differ between clusters.