Response to commentary on ‘CorrAdjust unveils biologically relevant transcriptomic correlations by efficiently eliminating hidden confounders’

In a commentary on the CorrAdjust method [1], which we recently introduced for automatically identifying and correcting hidden confounders in transcriptomic correlation analyses, Babaie et al. embarked on a comparison with COBRA [2], a method they developed that requires confounders to be explicitly defined at the start of the computation. Babaie et al.’s comparison juxtaposes the outputs of methods whose premises and required inputs differ by design, is based on a metric and analyses with critical limitations, and arrives at conclusions that are not supported by the data they presented.

An important step in computing abundance correlations among groups of RNAs, or groups of proteins, is identifying and removing hidden confounders. The goal of this correction is to enumerate more credible subsets of putative regulator:effector interactions, help identify important underlying events and the actors effecting them, and help determine a regulator’s role by restricting the gamut of its potential direct targets to only the network in which it participates. Because of the importance of adjusting abundance correlation, methods have been developed to tackle the problem when confounders are not known a priori, e.g. SVA [3] and CorrAdjust [1], as well as when the confounders are known a priori, e.g. linear regression [4], and COBRA [2].

In their commentary, Babaie et al. present a comparison between CorrAdjust and COBRA and raise several points that we address below.

(i) Babaie et al. begin by establishing an initial set of confounders, which they represent in the COBRA design matrix, and then embark on a comparative analysis. Under this setting, COBRA had a lower best-module precision score in 5 of the 11 tested tissues. After inspecting the resulting enriched terms, they added one more covariate, the Hardy scale, and repeated the analysis. Under the new setting, COBRA had a lower best-module precision score in 3 of the 11 tissues. In evaluating the results from the second iteration, the authors conjectured that if race, ethnicity, and batch were also included, COBRA’s performance would improve further. The asymmetric comparison between the two methods notwithstanding, this post hoc expansion of the list of confounders demonstrates that the relevant covariates may not be known a priori. Therefore, the authors’ assertion that subtracting components corresponding to known unwanted covariates leaves “[…] only the biologically relevant correlation structure […]” holds only after all important confounders have been measured and correctly modeled. But this is precisely the problem that methods that automatically identify latent confounders are meant to address.

Our original study was motivated by the observation that confounders can be latent, incompletely recorded, or difficult to model directly. We list here two characteristic examples that were also discussed in our original publication [1]. In The Cancer Genome Atlas (TCGA), it was discovered post hoc that the library-preparation plate correlates strongly with small RNA expression [5]. However, this potentially relevant information is not available for the GTEx and Geuvadis projects. Similarly, RNA integrity number has a nonlinear association with expression [6, 7], is available for only a subset of TCGA samples, and is unavailable for Geuvadis.

Additionally, we note that an approach based on manual selection of confounders can lead to unintended consequences. For example, two different groups could end up reporting different sets of cleaned-up correlations for the same collection of datasets because they chose different sets of confounders at the outset.

(ii) The central “quality over quantity” claim of Babaie et al. rests heavily on a tissue-specific metric that favors a single module, the one achieving the highest score, while dismissing the contents of all remaining modules. This “best module metric” is problematic for two reasons. First, it is an inappropriate choice for this comparative analysis given the substantially different gene partitioning that COBRA generates. As Fig. 2 of Babaie et al. shows, COBRA generally partitions the initial set of genes into modules that are (a) more numerous and (b) smaller in size compared to CorrAdjust. This partitioning provides more opportunities for COBRA to produce at least one module that best matches the authors’ tissue-specific keywords. Second, this single-module metric does not indicate which of the remaining COBRA-generated modules capture signal and which capture noise. These two points suggest it would have been more appropriate to evaluate the complete correlation output rather than the highest-scoring module alone. In our original benchmark, which ranked all gene pairs in the test samples and aggregated enrichment evidence across all eligible genes, COBRA produced global enrichment scores that were below the uncorrected baseline for most datasets we examined.

On a related note, the precision values reported in Fig. 3A of Babaie et al. are not accompanied by the information needed to assess their significance. Precision is computed as the fraction of enriched terms matching manually defined tissue-specific keywords, but neither the numerator nor the denominator is reported. Does a precision value of 100% correspond to, e.g. two tissue-specific terms among two enriched terms? Or, to 50 out of 50? Without this information, a module containing 5 out of 5 tissue-specific terms would “outperform” a module containing 40 tissue-specific terms among 50 enriched terms. No confidence intervals, permutation analysis, or statistical comparisons were provided either.

(iii) For the case of the “spleen” tissue, one of the cases where COBRA underperforms, Babaie et al. suggest the comparatively small sample size of this dataset as the reason, in combination with the sensitivity of regression-based methods. However, this explanation is not supported by the authors’ own analysis. In fact, the “spleen” dataset contains 241 samples, including 121 training samples, whereas the “small intestine terminal ileum dataset” contains only 188 samples, including 94 training samples, i.e. considerably fewer samples than the spleen. Nonetheless, COBRA manages to partition the smaller “small intestine” gene collection into multiple modules, whereas it assigns the spleen genes to essentially a single module (Fig. 2 of Babaie et al.). Thus, sample size alone cannot explain this result. Additionally, no downsampling experiment or other systematic analysis of module recovery as a function of sample size was provided to support the claim of sample-size dependence.

(iv) Babaie et al.’s basic assumption is that “physiological functions in each tissue are carried by specialized and distinct functional modules.” We agree that such modules are undoubtedly informative. However, focusing only on those is unnecessarily limiting and overlooks the well-established observation that the function of genes and pathways is inherently context-dependent. Development and cancer provide characteristic examples of pathways that, while omnipresent, lead to different outcomes when their genes interact with different partners in different contexts, a situation that would be reflected by differences in gene–gene expression correlations. For example, TGFβ and Notch signaling can exert either tumor-suppressive or oncogenic effects depending on cellular and molecular context [8, 9]. A benchmark based primarily on tissue-specific keywords may therefore penalize biologically meaningful context-dependent relationships involving broadly active pathways.

(v) Babaie et al. also claim that the use of reference collections by CorrAdjust introduces “a bias that promotes gene pairs” annotated to co-occur in known pathways or biological processes. This is a misleading characterization. CorrAdjust does not directly modify correlations to favor annotated pairs. Instead, it uses publicly available, vetted collections of reference pairs to guide the automated selection of latent factors towards maximizing the aggregate enrichment of those pairs among highly ranked correlations.

While a priori knowledge is required in both COBRA (i.e. covariates) and CorrAdjust (i.e. a vetted collection of reference pairs), we stress CorrAdjust’s flexibility in this regard. It can accept pathway annotations as well as arbitrary collections of reference pairs. In our original study, we illustrated this flexibility using Canonical Pathways and Gene Ontology for messenger RNA (mRNA)–mRNA correlations, experimentally validated and sequence-predicted microRNA (miRNA) targets for miRNA–mRNA correlations, and an independent transcription-factor target collection for validation.

Additionally, unlike CorrAdjust, COBRA requires covariate data for every observation (e.g. sample), making it subject to all of the confounder-related limitations and biases discussed in the earlier Point (i).

(vi) Babaie et al. state that they were unable to reproduce our TCGA and Geuvadis results and speculate that this was due to custom “pipelines for data processing and normalization for small RNA counts that were not disclosed.” However, they do not state what aspect of the results they failed to reproduce or what discrepancy they observed. Also, we note that they did not contact us for information or clarification. This unspecified inability to reproduce our findings provides no basis for questioning the validity of the results we included in our original report and, at best, suggests that their re-implementation differed from our extensively documented analysis. As a matter of fact, our original report described all essential processing steps in the main presentation and the accompanying Supplement. We stated the data sources and releases; software and reference versions; small RNA mapping tools; sample exclusions; expression thresholds; modality-specific normalization; winsorization steps; transformations; and training/test procedures. Lastly, we provided the complete lists of samples and genes in Supplementary Table S1.

(vii) In our original publication, we stated that it was unclear to us how well COBRA models would transfer between training and test samples. Babaie et al. also addressed this question by outlining two estimation modes and incorporating them into a new release of COBRA. This new functionality has not been peer-reviewed and, of course, does not invalidate the benchmark we conducted using the version of COBRA that was available at the time. Moreover, Fig. 4 of Babaie et al., which accompanies the description of these new modes, evaluates the reconstruction of the observed test-set covariance matrix rather than the accuracy of the covariate-corrected network. As such, it does not directly answer the transferability question we raised in our original report.

(viii) Lastly, we address the matter of longer runtime, which Babaie et al. claim puts CorrAdjust at a disadvantage. First, we note that neither CorrAdjust nor COBRA is intended for interactive use, as confounder correction typically needs to be performed only once for a given collection of datasets. Second, CorrAdjust’s longer runtime reflects not poor resource use but rather a methodological difference: by searching across possible latent factors and evaluating gene-pair enrichments, CorrAdjust solves a different, computationally more demanding problem than COBRA, which operates on covariates specified in advance.

Notes

Present address: ZS Associates, 201 Washington St, Floor 28, Boston 02108 MA, United States

Conflict of interest

None declared.

This paper is linked to: https://doi.org/10.1093/nar/gkag932.