Errors in the Brown et al. critical reanalysis

Brown et al. (1) critique our previous report (2) and judge the results “no more than the product of chance.” We share Brown et al.’s interest in protecting the field against false claims and appreciate their desire to ground their evaluation in reanalysis of our data. However, we have discovered that Brown et al.’s reanalysis itself contains major statistical and factual errors that ultimately invalidate their conclusions [as do new data replicating (2)].

One major error involves the “bitmapping” analysis Brown et al. (1) purport shows inflated false-positive error rates for our approach. Their bitmapping does not involve random sampling of observations (the only valid method for gauging analytic error) (3, 4), but instead iteratively repartitions observed psychometric variables within a fixed dataset and computes statistical tests on each resulting pair of “pseudofactors.” An investigating analyst will find that, regardless of whether bitmapping is applied to our data and analytic approach—or to totally random data and benchmark analyses, such as the t test—it produces aberrant distributions of parameter estimates and P values that bear no resemblance to valid sampling distributions (3, 4). This is evident in Brown et al.’s figures S7–S11 (1), which show centrally constricted bow-tie distributions, asymmetry despite random input data (figures S8 and S11 in ref. 1), and bias (never passing through 0,0 in figures S8–S11 in ref. 1). Although Brown et al. attribute these aberrations to our estimator, the distortions actually stem from their own invalid bitmapping procedure (for which they provide no reference or mathematical justification). Consequently, none of their false-positive estimates are valid.

Brown et al. (1) also capitalized on chance in attempting to refactor the Mental Health Continuum-Short Form (MHC-SF) well-being scale in a sample too small to support reliable factor discovery or item reallocation (5). The MHC-SF is extensively validated and we used established scoring of hedonic and eudaimonic items (references in ref. 2). In replication data, that established two-factor scoring structure fit substantially better than did the two-factor structure derived by Brown et al. (1).

Many of Brown et al.’s (1) other claims are false as well [e.g., pooling gene-specific associations is actually commonplace and represents an elementary statistical sum of random variables (3), whereas their proposed averaging with acceptance of null hypothesis 0s guarantees bias (3, 4)]. We do agree with Brown et al.’s (1) Supporting Information assertion that mixed-effect modeling would be a reasonable alternative approach to address correlated residuals.

We recently replicated divergent RNA associations with eudaimonic and hedonic well-being scores in an independent sample of 122 healthy adults. Analyses yielded similar pooled association estimates; eudaimonic b = −0.021 log₂ RNA SD⁻¹ vs. −0.028 in ref. 2, hedonic b = +0.025 vs. +0.028 in ref. 2. If previous results were spurious and true sampling variability were really >twofold inflated, as Brown et al.’s (1) bitmapping purports, it is highly unlikely that either association would replicate so closely, and the probability that both would do so is ∼1 in 100 (3). So, although Brown et al. judge “the chances of a successful reproduction ... to be remote,” their conclusion is wrong both analytically and empirically (1). Analyses of new independent data using mixed-effect linear models continue to validate the previous observations (2).