The Power of Theory, Research Design, and Transdisciplinary Integration in Moving Psychopathology Forward

Abstract

While the past few decades have seen much work in psychopathology research that has yielded provocative insights, relatively little progress has been made in understanding the etiology of mental disorders. We contend that this is due to an overreliance on statistics and technology with insufficient attention to adequacy of experimental design, a lack of integration of data across various domains of research, and testing of theoretical models using relatively weak study designs. We provide a conceptual discussion of these issues and follow with a concrete demonstration of our proposed solution. Using two different disorders – depression and substance use – as examples, we illustrate how we can evaluate competing theories regarding their etiology by integrating information from various domains including latent variable models, neurobiology, and quasi-experimental data such as twin and adoption studies, rather than relying on any single methodology alone. More broadly, we discuss the extent to which such integrative thinking allows for inferences about the etiology of mental disorders, rather than focusing on descriptive correlates alone. Greater scientific insight will require stringent tests of competing theories and a deeper conceptual understanding of the advantages and pitfalls of methodologies and criteria we use in our studies.

Not everything that can be counted counts, and not everything that counts can be counted 

- (Cameron, 1963)

1. Introduction

Understanding the etiology and structure of mental disorders is the purpose of psychopathology research, and an especially timely issue presently with recent revisions to the Diagnostic and Statistical Manual of Mental Disorders (DSM), and initiatives such as the Research Domain Criteria (RDoC). Evidence from several different lines of work such as behavioral and molecular genetics, neuroimaging, family studies, etc. has been considered in both of these initiatives but transformative progress has been more difficult than anticipated (Kupfer & Regier, 2011). We contend what is particularly required is a framework and strategy for integrating the data from such diverse studies. Cronbach (1957) proposed that studies in psychology could be broadly divided into two streams – experimental and correlational. Such a strict division may not necessarily be the case in current times, and a range of studies exist that amalgamate these two streams of thought to differing degrees, such as quasi-experimental research (Rutter, 2007; Shadish, Cook, & Campbell, 2001). Indeed, this kind of paradigm – one that combines the causal significance of laboratory tests with the import of studies involving people as they function in the real world – is much needed in psychopathology research and should be advocated on a much grander scale.

The place to begin is with a guiding theory that informs how to evaluate evidence for or against a particular hypothesis. Theories may be derived from pre-existing notions of a particular disorder, or may arise out of the data after evaluating a certain amount of evidence for or against it (such as with exploratory data analysis; Tukey, 1980). The strategy we advocate here, however goes beyond that – here, we focus on the interface between research design, statistics, psychology, and neurobiology, and the extent of theoretical inference that can legitimately be drawn from them. Such an approach will require us to (a) move beyond an overreliance on advanced statistics and the use of new technology, especially in the absence of causally informative research design; and (b) integrate data in a meaningful way across methodologies using theoretically guided designs, and (c) test such theories using stringent study designs that allow us to make less ambiguous inferences regarding causality. In particular, the emphasis on the process of quantification and the use of advanced technology in some research occurs at the expense of theoretical or study design considerations, and lends an illusory quality of validity or deeper meaning to the results than warranted (Tavris, 2010) – a process that can easily lead to misguided conclusions and misallocation of research dollars.

Stated more concisely, to advance our understanding of the etiology of mental disorder, we as a field must employ, whenever possible, risky tests of causal theories. What is a risky test? The risky test, in our view, is or approximates an experimental design and narrows the number of interpretive possibilities regarding causality, such that the riskier the test, the stronger the inference one can draw about the theory being tested. The ultimate risky test provides the opportunity to falsify and reject a theory, but we recognize this is seldom if ever possible in our field (Meehl, 1990), in part due to the inherent limitations of true experiments in psychology, especially when dealing with the ethical and legal obligations involved in research with human subjects (Meehl, 1978). In the absence of a true experiment, there are however, many clever ways to devise tests to be as risky as possible. Thus, we will use the phrase “risky test” to refer to (quasi-) experimental designs that not only allow strong inference about the tested theory, but are also practically feasible. If we perform risky tests from a variety of independent study designs then, when the results converge on the same interpretation, we have especially compelling evidence for theory corroboration or falsification.

We turn now to a conceptual discussion about the utility of statistics in testing theories about psychopathology, the use of novel technology in understanding mental disorders, the relationship between these domains and, lastly, the possible inferences one can legitimately draw this research. Our point here is not to examine the philosophical bases underlying mental disorders or statistics, though we do occasionally cover such concerns as part of our discussion. Nor do we focus solely on statistical issues such as replication, or multiple testing, or even questionable research practices (for an excellent discussion on these and related topics, see November 2012 issue of Perspectives on Psychological Science Pashler & Wagenmakers, 2012). Rather, the crux of our paper is focused on how to simultaneously leverage all our available research tools, use them to test etiological theory, and thereby arrive at the causal insights necessary to create, for example, a psychiatric nosology grounded in etiology, rather than one centered on descriptive psychopathology. We follow this with a concrete demonstration of our proposed solutions using two different disorders – depression and substance abuse – and link them back to the conceptual issues we have highlighted.

2. Conceptual issues on the use of statistics and technology

2.1 Statistics have their limits

Given our strong focus on the scientific process and strategies for deriving meaning from the results it provides, a few working definitions are useful. On pain of simplicity, we adopt a Popperian philosophy of science where the scientific process is a cycle in which one outlines a theory (a set of hypotheses based on prior data), designs and runs an experiment to test it, evaluates the findings, and accepts, rejects, or modifies the theory based on the results, runs another experiment if necessary, and so on (Lakatos, 1978; Meehl, 1990; Popper, 1959). There are several philosophical considerations here as to what it means to “accept” or “reject” a theory, but at least this is the basic premise. One option would be to ignore the theory part and rely solely on statistics, technology, and data mining. However, though some amount of tinkering and exploratory analyses can be useful and important (Behrens, 1997; Tukey, 1977), blind reliance on such methods cannot be counted on to provide consistent fuel for scientific progress or to aid the development of a larger theory of mental illness, no matter how “big” the data are. We shall discuss some examples below that illustrate why. A second option is to rely exclusively on theory to guide all our research; however, such a strategy would be focused solely on justification of existing theories, rather than allowing data exploration to spark new ideas and raise new questions. Another choice is to develop and refine theories in conjunction with clever research designs and risky tests. This may seem like a painfully obvious point to most readers but, as will be seen in the following sections, what may be obvious is not always implemented in practice.

First though, we will tackle the issue of what it means to test a theory in general. In contemporary psychology research, statistics are often seen as very useful ways to evaluate some theory or hypothesis. Most common statistics and statistical models that we use are based on linear relationships among variables, such as a correlation. Regardless of the available data, whether they are personality questionnaires, clinical interviews, electroencephalographic (EEG) brain oscillations, MRI scans, or genetic polymorphisms, it remains true that a correlation, analysis of variance, regression, principal component analysis, factor analysis, structural equation model, or any other approach, evaluates the degree to which a set of variables behave in relation to each other. In other words, they give us an idea of the kinds of lines we can fit to the dataset at hand – straight, curved, going up, down, etc. By themselves, they give little to no information about why the variables go up and down together, just that they do. The additional catch here is that there are an infinite number of lines (literally) that can fit any dataset. While we use the line of “best” fit as a rule of thumb, and generally use straight lines in our statistics, there are no limits to lines we can use (Breiman, 2001b; McDonald, 2010; Meehl, 2002).

A justified response from many scientists to the issue raised above (i.e., choosing the best-fitting line) is to use the principle of parsimony to choose the best model. This may sound easy, but quantifying parsimony is not a trivial issue with clear guidelines. For example, Meehl (2002) identified no less than four common conceptions of parsimony, including curve fitting, economy of theoretical postulates, economy of theoretical concepts, and Ockham’s Razor, and these were described in the context of 11 total criteria that scientists use in appraising scientific theories. It is far from clear why one particular definition of parsimony is the most important when it comes to deciding upon any one particular theory. However, the consequences can be striking depending on the choice (e.g., deciding whether the latent structure of psychopathology is categorical or continuous based on parsimony as the simplest line fitting the data using fit indices such as the Akaike Information Criterion or Bayesian Information Criterion; cf. Grove & Vrieze, 2010).

Just as important as quantifying model fit is the interpretive step from selected model to scientific take-home message. Unless one has selected a true model, the very fact that individual models are selected and interpreted means that bias is incurred in the resulting parameter estimates. Drawing conclusions (e.g., about the etiology and structure of psychopathology) as if the selected model is the only model to be considered ignores the fact that any selected model, no matter the weight of evidence in its favor, may have been selected in error. Thankfully this issue, referred to in the literature as model selection uncertainty, is becoming increasingly recognized as a major issue in statistical inference (Sterba & Pek, 2012; Yuan & Yang, 2005). Related to this, model averaging approaches (Claeskens & Hjort, 2008), where perhaps thousands of models are fit to a dataset through resampling methods, show that using all of the models is often predictively superior to methods where only the best model is selected (Breiman, 1996, 2001a, 2001b; Yang, 2005; Yuan & Yang, 2005). These results are found both in simulation studies and real-world applications, such as making predictions about future events and waiting for those events to unfold. Termed ensemble learning or ensemble classifiers, the superior performance of these methods drive home forcefully the fact that our model selection procedures are probabilistic and error-prone, and that even the best model is inferior to extremely complex model averages (which seem very far from parsimonious). The drawback to ensemble classifiers is that it is difficult to draw simple and substantive scientific conclusions from them; on the other hand, the drawback to selecting a single best model is that it is overly simplistic, and investigators run the risk of overstating results and drawing unwarranted simple conclusions about complex phenomena. While simple models based on the notion of parsimony can certainly be useful, parsimony itself is not a universal law, just a heuristic. Indeed, anti-razors such as Hickam’s dictum (“Patients can have as many diseases as they damn well please”) have been fruitfully used in other fields such as medicine (Abramowitz et al., 2008). Thus, one must be careful when emphasizing simplicity or parsimony, especially when model fitting is conducted in observational datasets, a best model selected, and inferences made from that model about the etiological structure of a set of variables.

What is required, then, to adequately evaluate a scientific question? Simple models can help make sense of unruly data, but heavy reliance on statistics is a poor route to scientific insight. As everyone reading this knows, it takes a question, a theory, a design, an analysis, and a result. The difficulty comes when actually implementing any of these steps. The theory is most useful when it postulates a risky hypothesis, preferably including some etiological mechanism, although strong theories are often elusive in psychopathology research. The design must then provide a test of the etiological hypothesis which, for most interesting questions, requires an experimental or quasi-experimental design. Statistics is clearly crucial to the analysis step, with the correct statistical approach depending on the question, theory, and design. Finally, conclusions are something with which statisticians can assist, but depend just as strongly on the scientific adequacy of the experimental design to measure and control variables relevant to the etiological hypothesis. We address each of these issues below, starting with an important distinction between descriptive and etiological hypotheses.

2.2 Testing a theory rigorously requires a risky experimental design

The more fundamental point behind the issue raised in the section above is whether we can use our knowledge of statistics and modeling to obtain insight about the etiology of mental disorders versus a description of how the disorders relate to one another. This requires etiological theories, and the submission of those theories to risky tests (Meehl, 1978, 1990; Popper, 1959). As mentioned earlier, risky tests refer to practical research designs that narrow the number of interpretive possibilities for our results. When two variables correlate it is reasonable to infer that they somehow share space in a common causal framework (Cronbach & Meehl, 1955). Despite what we teach undergraduate students, correlation implies causation. The problem is that it is very difficult to make inferences about a causal framework from correlational data other than that a framework exists. What is more, deriving from some theory the prediction that two variables are causally related, then testing the theory by taking measurements and computing a correlation, is far from a risky test of theory; the test can easily be passed even if the causal theory is false. An overreliance on statistical modeling separated from causally-informative research design and plausible etiological theory will never produce insight that goes beyond simple conclusions that two variables somehow share etiology because they correlate (Rubin, 2008). Taleb (2010) had an excellent analogy for this where he noted that fitting models to data and inferring causality is like figuring out the shape of a melted ice cube from the puddle of water that it has left behind. There are any number of ice cubes with varying shapes and structures that can generate any particular puddle of water.

Obtaining etiological insight (knowing the ice cube) requires etiologically informative research designs. Correlational data, also called descriptive or narrative data, whether longitudinal or cross-sectional often provide little leverage to address etiological questions because the Fundamental Problem of Causal Inference remains (Holland, 1986). This is the notion that to make causal inferences we would ideally experimentally manipulate an independent variable. In a longitudinal sample this would amount to observing Subject X have an experience (e.g., use alcohol excessively during adolescence) and not have an experience (e.g., abstain from alcohol) simultaneously, or to ocurr in parallel universes, which is probably not possible. The practical approach is typically to randomly assign participants to each condition. If this were possible – which it often is not – then we could rigorously test causal hypotheses, such as whether excessive alcohol use causes brain-related changes in the mesolimbic dopamine system or the prefrontal cortex, as some have hypothesized (Goldstein & Volkow, 2002; Kalivas & Volkow, 2005). If we only have longitudinal but otherwise etiologically uninformative samples, then we cannot observe every individual in their real life where they are exposed to the variable of interest as well as counterfactual life where they are not. We can, with a little ingenuity however, approximate a true experiment and get some information about the counterfactual state of affairs (Rubin, 2001, 2007, 2008). Monozygotic twins discordant for some variable (e.g., alcohol use) give one approximation to an experiment, as they provide for each other a quasi-experimental control for genetic and shared environmental background (McGue, Osler, & Christensen, 2010). The Minnesota Study of Twins Reared Apart is a discordant twin design (discordant for rearing environment) that provided a strong test of the causal effect of rearing environment on a wide range of psychological outcomes (Bouchard, Lykken, Mcgue, Segal, & Tellegen, 1990). This study was especially powerful because prominent psychological theory, such as psychodynamic theory, made predictions about the importance of rearing environment on a wide range of outcomes. The use of twins (Galton, 1875), children of twins (D’Onofrio et al., 2003), adoption designs (Rutter, 1998), natural disasters (Kilpatrick et al., 2007), migration studies (Ravussin, Valencia, Esparza, Bennett, & Schulz, 1994), Mendelian randomization (Ebrahim & Smith, 2008; Smith, 2011), and other natural experiments (Rutter, 2007), provide powerful methods to conduct quasi-experimental research (Shadish, et al., 2001) in humans when ethical and other considerations often prohibit true experimentation (Meehl, 1978). This is not to say observational research is not valid – indeed, one needs to start at descriptive research to even form a theory of some sort. However, the inferences that can be drawn from purely observational work are open to multiple causal interpretations, and thus, less risky than counterfactual designs such as the ones listed above, especially when it comes to positing etiological theories about mental disorders. However, it is worth emphasizing that, just as there are no surefire statistical methods for establishing the etiology of any disorder, neither are there any surefire research designs.

2.3 Advanced technology and complex statistical models do not always provide more insight

Recent years have seen exciting advances in our ability to use increasingly complex and computationally demanding statistical methods (e.g., latent trait, class, and factor mixture models) and technology that probes deeper and deeper into the human body (e.g., EEG to MRI to molecular genetics) in an attempt to understand psychopathology. The issue here is not with the tools we use, but rather what can be a noncritical reliance on them, simply because they appear to be “advanced” in some way. For example, the explosion of structural equation modeling since LISREL (Jöreskog & Sörbom, 1986), and more recently Mplus (Muthén & Muthén, 1998-2010), allows researchers to very easily fit extremely complex models and interpret them, in some cases with nothing beyond a superficial understanding of the method. Conclusions from these models at times go far beyond what is warranted, such as that the latent entities exist in nature (Maraun, 1996; Meehl, 1992, 1993), or that a causal relationship is “suggested” because an arrow in the path diagram is asterisked at p <.0001. There is undoubtedly much to learn about psychological phenomenology from factor analyzing covariance matrices, or using other dimensionality reduction techniques (e.g. The Big Five (Digman, 1990), Internalizing/Externalizing (Achenbach, 1991; Krueger, 1999), g (Spearman, 1904)). What makes these models particularly useful is the fact that they summarize large amounts of data in a convenient way (e.g., all the personality descriptors in the English language can be condensed into three to five dimensions) and they give researchers a common parlance in which to discuss constructs. More broadly, such analytic techniques are particularly useful in inductive or exploratory contexts where the goal is dimension reduction to explore, organize, and summarize information. However, great caution is warranted before drawing conclusions, especially about etiology, on the basis of these studies alone. In other words, deducing that the Big Five Traits exist in reality or that Internalizing and Externalizing are the only way to organize mental disorders is a much riskier process.

This issue of overreliance on advanced models or technology is by no means restricted to statistical modeling. For example, in neurobiology, functional magnetic resonance imaging (fMRI) research is published frequently in prominent journals with inferences made about particular brain regions implicated or even causing certain disorders that appear to outstrip the research design in which the technology is used (e.g., research on the “neurobiological basis” of some disorder or emotion). Recent times, however, have seen a burgeoning critique of how these technologies are used to advance our field. For example, blood-oxygen-level-dependent (BOLD) contrasts are the primary variable of interest in fMRI analyses. Logothetis (2008), though, observed that inferring exactly what kind of activity a BOLD signal represents can be unclear –e.g., excitation vs inhibition, activity relative to what kind of baseline, etc. Likewise, in another widely discussed paper, Vul, Harris, Winkielman, and Pashler (2009) have pointed out that researchers can get artificially high correlations between brain activation and some personality measures by only using voxels that cross a certain threshold in their analyses. Another concern is that of sample size – while it is relatively easy to administer and analyze self-report data from thousands of subjects, this is not the case with neurobiological research for practical and cost-prohibitive reasons. Thus, most fMRI studies are restricted to smaller samples (e.g., less than 50), and draw conclusions about relative activity or inactivity in various brain regions based on such samples. These limitations can pose substantive problems for researchers in this area. For example, stuides (Gonzalez-Castillo et al., 2012; Thyreau et al., 2012) have found that when samples include large numbers of subjects or trials (> 500 or 1000), most regions of the brain show up as statistically significant in analyses. In fact, prior work has shown that the average statistical power of neuroscience studies is quite low (Button et al., 2013). Of course, the issue that is being highlighted in these studies – statistical versus biological significance – is not specific to fMRI (Farah, 2014). But the point remains that it is unclear as to what exactly is the required evidence for drawing appropriate conclusions about brain regions implicated in mental disorders. Similar problems are present in other methodologies as well. Electroencephalography (EEG), for instance, has fine temporal resoluation (MRI does not) but it is a two-dimensional representation of a three-dimensional brain, and there is no way currently to identify a single anatomical source for any observed waveform. Consequently, knowing that two groups of subjects differed on some event-related potential (ERP) is not very informative unless we have a strong theoretical notion about the significance of that ERP in some experimental context.

Genomics represents another major technological innovation. Advances in this field have made measurement of millions of genetic variants in humans easy and inexpensive. The completion of the Human Genome Project was a landmark event in the history of science, and has had massive implications for biology that have gone far beyond the investigation of genetic etiology of disease (Lander, 2011). We have gained a great deal of knowledge about the genetic architecture of disease, and we now know that existing candidate gene studies for common psychiatric disorders and psychological traits based on a priori hypotheses about gene function are unreliable (Collins, Kim, Sklar, O’Donovan, & Sullivan, 2012), and have returned largely false positive results. This is in part due to the small effect of common single nucleotide polymorphisms (H. L. Allen et al., 2010; Schizophrenia Working Group of the Psychiatric Genomics Consortium, 2014; Speliotes et al., 2010; Sullivan, Daly, & O’Donovan, 2012) and the underpowered samples used to test their effect on disease (L. E. Duncan & Keller, 2011; Vrieze, Iacono, & McGue, 2012). The best single example is the serotonin transporter, initially thought to show interesting gene by environment interactions in predisposing to depression (Caspi et al., 2003). Though based on a plausible theory – i.e., that the serotonin system is implicated in depression – after a decade of research the initial findings remain inconclusive (Karg, Burmeister, Shedden, & Sen, 2011; Munafo, Durrant, Lewis, & Flint, 2009; Risch, 2009). The point: A more technologically sophisticated tool has not necessarily advanced our understanding of mental illness – it may in fact derail it for a time.

Fortunately for genetics, we now have genome-wide association studies (GWAS) and widely-accepted conventions to avoid false-positive results, which are now providing insight into the failure of prior candidate gene and linkage studies of complex traits/diseases (Visscher, Brown, McCarthy, & Yang, 2012). GWAS is predicated on a specific theory of genetic etiology, the common-variant common-disease hypothesis (Pritchard & Cox, 2002), and continues to make advances in understanding the genetic architecture of psychiatric disease (Sullivan, et al., 2012) but has its own limitations. While GWAS has been very successful for non-psychiatric complex diseases, the single most important result for us can be characterized as a very risky test of a large number of candidate gene hypotheses. GWAS has told us that even our most promising candidate gene theories were little better than random guesses at genotype-phenotype associations. For example, even in samples traditionally considered large (e.g., N = 10,000), serotonin-related genes show no more of a signal for depression than other randomly selected regions in the genome. It is on this basis that many candidate gene theories in psychiatric genetics have been very strongly discorroborated. In fact, a recent laudable paper by Hart et al. (2013) that attempted to replicate the authors’ own prior candidate gene findings showed that none of their 12 prior significant results held up even when attempting to replicate and extend their sample size to just 200 additional subjects. Now, we must keep in mind that such tests are not complete. The candidate gene theories could be revised to state that the effects of the candidate genes are just much, much smaller than previously expected (e.g., DRD2 and Schizophrenia; Schizophrenia Working Group of the Psychiatric Genomics Consortium, 2014). In addition, other hypotheses about genetic etiology such as the rare-variant common-disease hypothesis (Pritchard, 2001) are now capable of being tested with new technologies such as exome (Fu et al., 2013; Nelson et al., 2012) and whole genome sequencing (Abecasis et al., 2012), and we can expect scientific findings to continue to grow in biology and psychiatric genetics. Indeed, it may turn out that serotonin-related genes do have large effects, but we were unable to measure those effects until now because the relevant variants are very rare and heretofore unmeasured. We must be careful however that these new advances in technology are applied with appropriate care and circumspection in psychology and psychiatry.

Leveraging new technologies is crucial to the advancement of science, and always has been. The problem arises when technology is used without considering research design or substantive scientific theory. Carol Tavris (Voss, 2012) has referred to this tendency as “technomyopia” – or the tendency to be dazzled by the technology at hand rather than evaluating if what it reveals is substantive. Others (Marcus, 2012; O’Connor, Rees, & Joffe, 2012) have also noted that this tendency can become even more extreme when mainstream media pick up on such articles and try to convey them to readers. Part of our responsibility as a field involves “de-hyping” our results and conveying them in the proper context and with appropriate caveats. As a general rule, statistical models of correlations do not, without some kind of experimental control, allow etiological insight beyond that obtained with measures of association. Likewise, merely increasing the resolution of our tools or our sample size does not guarantee the worth of a study. Even the largest sample combined with the most advanced data mining techniques and the most sophisticated technology is only mildly informative for understanding etiology without some kind of (quasi) experimental control and, something we take up later, a good scientific theory that can account for results.

2.4 Relying solely on any single methodology can lead to incorrect conclusions

Descriptive, experimental, quasi-experimental: these and other diverse designs have strengths and weaknesses. Integrating them into a more general framework for the study of mental disorders is a significant challenge. Symptoms for most disorders in the DSM-IV involve self- or other-report of overt behaviors/emotional states and interpersonal dysfunction, a mixture of which generally amount to clinically significant distress or impairment in some aspect of life. While a section on neurobiological correlates is included for most disorders, none of these is among the actual criteria that lead to diagnosis. The past few years have thus seen a strong push for uncovering biomarkers and molecular genetic components of disorders as well, especially with initiatives such as RDoC and BRAIN which are being heavily promoted by the U.S. National Institutes of Health. The problem here though, is how to combine (or perhaps prioritize) such criteria in relation to overt, self-report symptoms. An obvious question here is why should one place so much emphasis on integrating results from various methodologies?

One answer is we are all studying related phenomena, whether we hail from psychology, psychiatry, social work, or neuroscience, in part thanks to the common language provided by the DSM. Instead of relying on a single discipline or single methodology, a more appropriate strategy involves identifying concordance between results from different methodologies. Relying on the wrong source, or on only one source, can be misleading. Consider, for example, intelligence in the general population. Intelligence, as measured by IQ tests, is essentially continuously distributed in the general population. Despite apparent continuity and polygenic background (Davies et al., 2011), it is known that low IQ can have discrete causes such as phenylketonuria, Down syndrome, Prader-Willi syndrome, and brain damage, to name but a few. It is very easy to see here how prioritizing just one methodology – overt measurement – in understanding intelligence is misleading. On the one hand, this may seem like an obvious recommendation. But on the other, what occurs often in psychology in that results from one domain tend to lead those in another. For example, given the lack of neurobiological correlates for mental disorders, most research in this field has generally been unidirectional, from disorder or personality constructs to their underlying neurobiology or genetics, with an implicit assumption that this is the right way to go (e.g., genes for X disorder, or brain activation in relation to some personality trait). The problem is not solved merely by conducting research in the opposite direction – e.g., tracing genes and brain circuits, and buliding up larger constructs from them. Rather, what is required, is a true synthesis of information – a strategy that looks for parallels in findings from multiple domains. That is to say, hypotheses generated in one must be tested in the other. For example, if a disorder or personality trait is considered to be continuous (as several proposals for various DSM domains have suggested), then one must be able to find a similarly continuous biological process that corresponds to this trait; or if not, the way in which such a discontinuous or perhaps nonlinear process translates into a continuous trait. We have attempted to provide examples of this strategy in the two disorder sections provided below.

While there are scientific reasons for integrating across disciplines and methodologies, there are also more pragmatic ones. For example, given the somewhat divergent paths of the RDoC and the DSM-5, if research on disorders as delineated by the DSM-5 are not prioritized for federal funding, what implications would this have for future versions of the diagnostic manual? Additionally, while the RDoC appears to have developed its own constructs and strategies for understanding mental disorders, it has done so agnostic of the DSM (NIMH, 2011). Developing a research classification system that is not fully aligned with the DSM has the potential to create confusion in the field, and over time, a schism between researchers and clinicians that might slow the translation of translational research; for this reason, it is important to maintain crosswalks between the two systems. Ultimately, the goal would be to reach a multidisciplinary and multi-perspective understanding of mental disorders, and in this regard the NIMH has announced that its long-term goal with RDoC is to create a research literature that supports future revisions to current diagnostic systems (DSM, ICD) rather than creating a new and competing nosology.

Statistical models and technological advances will be necessary to advance the study of mental disorders, but appropriately harnessing these technologies will require users to have intimate knowledge of the advantages and pitfalls of each methodology, and what conclusions can legitimately be drawn from them. Understandably, it is not possible to list every caveat each time one writes the discussion section of an article, but far more caution is warranted in such instances, and authors (and editors!) should feel less compelled to convince their reader (or reviewer) that their study provides a positive or useful result.

In the following section, we will synthesize information from a variety of literatures to draw broader inferences about the etiology of depression and substance use, and simultaneously address in each example the conceptual issues above.

1. We start with depression, where the results of numerous statistical analyses on self-reported symptoms conclude that depression is the product of a single continuously distributed liability, but evidence from other domains is not entirely consistent with this conjecture. Depression provides an excellent example of how theory-driven analyses and integration of data across multiple sources provides much more information than any single source.

2. We follow with a discussion of the etiology of substance use disorders from the perspective of the gateway and disinhibitory theories. This section focuses on the advantage of causally informative research design and a guiding theory in making meaningful scientific progress.

The main thread connecting these examples is the idea that using a mix of theory and methodology that allows for risky tests is crucial to develop an etiological understanding of these disorders, and that scientific questions cannot be easily or satisfactorily answered using just statistics, or data from one methodology, or even just theory.

3. Understanding Mental Disorders Using Theory and Risky Tests

3.1 Etiology of Major Depression

Major depressive disorder is among the most commonly diagnosed mental disorders with significant personal and societal impact (Bromet et al., 2011; Kessler, 2012). Here we have selectively reviewed the depression literature, unwedded to a preconceived etiological theory, to illustrate how it is possible to transition from an exploratory and inductive knowledge-development phase to a more deductive theory-development phase. This was accomplished by identifying common themes and bridging different research domains to derive a plausible theory regarding the nature of depression that fits the array of findings. The resulting formulation can be further evaluated and refined in subsequent investigations using research designs that provide for risky tests.

Much debate exists on whether depression should be considered a dimensional or categorical condition (Coyne, 1994; Flett, Vredenburg, & Krames, 1997; Prisciandaro & Roberts, 2009) and various subtypes of depression have been proposed over the years (endogenous vs exogenous, bipolar vs unipolar, melancholic vs nonmelancholic, etc.). Given the wealth of literature in this realm, it is not feasible for any one paper to summarize results of all prior studies. Broadly, however, prominent studies and major reviews (e.g., Andrews et al., 2007; Flett, et al., 1997; Haslam, 2003; Haslam, Holland, & Kuppens, 2012; Kendler & Gardner, 1998; Solomon, Haaga, & Arnow, 2001) undertaken in this area suggest that depression is more likely dimensional in nature, though there does occasionally appear to be evidence for taxonicity (Flett, et al., 1997; Haslam, 2003). Other researchers, however, have argued a case for at least one subtype of depression. For example, Parker and colleagues (2010) have proposed that melancholia (which they define as being similar to endogenous depression) should be classified as a distinct disorder in the upcoming version of the DSM based on various lines of evidence including enhanced responsiveness to biological interventions such as ECT. Others have proposed a two dimensional system – one indexing chronicity, and the other severity (D. N. Klein, 2008). Nevertheless, despite these contentions, in the absence of clear-cut evidence for distinct categories, depression, for now, is still considered by many to be dimensional in nature (Haslam, et al., 2012; Kessler, 2002).

Prior work attempting to study the nature of depression has focused primarily on self-report or interview-based measures of subjective symptomatology (Flett, et al., 1997; Haslam, et al., 2012). However, some laboratory-based measures produce a different picture. Studies that use startle blink reflex and event-related potential (ERP) measures, in conjunction with data from family studies, suggest there may a distinct subtype of depression or at least, a non-linearity at the extreme end of the depression continuum, characterized typically by recurrence. Whether this is equivalent to melancholia or not is not entirely clear based on the information available. The two neurobiological responses – the startle blink reflex response and the error-related negativity (ERN; a brain potential that is thought to reflect error monitoring) – that we discuss in relation to depression were chosen since there is (a) a strong theoretical and experimental literature that document the conditions that give rise to them, (b) the neural circuitry underlying them has been well studied, and (c) they have been investigated in relation to depression.

Among psychophysiological indices, one of the most widely used measures in the study of emotion is the startle blink reflex. Prior work has shown that the modulation of the amplitude of the blink response is directly mediated by the amygdala, and that it is affected by internalizing states such as fear and anxiety (Davis, Walker, Miles, & Grillon, 2009). It has been frequently used to study emotional processing and reactivity in mood and anxiety disorders. Subjects in the general population startle greatest in the context of unpleasant stimuli (such as pictures of mutilated bodies, threatening animals, etc.), and least in the context of pleasant stimuli (images of cute animals and babies, or erotic pictures), both relative to neutral scenes (e.g., household objects and neutral faces), resulting in a linear pattern of response amplitudes (i.e., pleasant < neutral < unpleasant; cf. Vrana, Spence, & Lang, 1988). This increase in response in the context of unpleasant stimuli is referred to as the fear-potentiated startle (FPS) effect.

Several of these kinds of studies have found that patterns of startle blink reactivity in depressed individuals appear to differ from those in the general population. In studies that present startle probes in the context of pictures, those with depression do not show the expected linear pattern of responses. Instead they show more of a flattened pattern of responses with not much differentiation between pleasant and unpleasant responses, or even increased responses to pleasant pictures. Interestingly, this effect is often restricted to those who have experienced multiple episodes of depression (Forbes, Miller, Cohn, Fox, & Kovacs, 2005), are severely depressed (N. B. Allen, Trinder, & Brennan, 1999), or have high levels of anhedonia (Kaviani et al., 2004). Similarly, in experiments by Lang and colleagues, where subjects are startled in the context of imagined scenes rather than pictures, depressed individuals show a smaller FPS than non-depressed subjects (McTeague et al., 2010; McTeague et al., 2009; McTeague, Lang, Wangelin, Laplante, & Bradley, 2012). Those with a greater number of recurrent episodes also have a smaller FPS than those with just single episodes of depression or those without depression. In our lab (Vaidyanathan, Welo, Malone, Burwell, & Iacono, 2014), we tested this hypothesis directly by contrasting startle response patterns amongst those who had 2 or more episodes of lifetime depression, those with just 1 episode of depression, and those who were never depressed. As hypothesized, only subjects with multiple episodes of depression showed abnormal startle responses. Further, the greater the number of episodes of depression, the stronger the abnormal pattern of responses. Clearly, if a single neurobiological mechanism undergirds all cases of depression, this is not the pattern of results that one would expect.

While most startle studies have tended to focus on current depression, Forbes et al.’s (2005) study revealed that it was specifically lifetime diagnoses that were associated with a lack of startle modulation, rather than current depression. More interestingly, Dichter et al. (2004) found that though individuals with major depression reported improvements in depressive symptoms after twelve weeks of using an antidepressant (Bupropion), their pattern of flat startle reactivity remained the same both before and after taking it. These results suggest that this abnormal startle pattern might be more trait- than state-like. In this context, it is worth noting that meta-analyses (Barbui, Cipriani, Patel, Ayuso-Mateos, & van Ommeren, 2011; Fournier et al., 2010) have suggested that antidepressants are effective relative to placebo only for those who are severely depressed rather than mildly or moderately depressed. In sum, startle studies suggest that there is something qualitatively different about recurrent or chronic depression that leads to decreased emotion modulated startle.

Another psychophysiological index – the error-related negativity (ERN) – also suggests a similar conclusion. The ERN has been used to study the brain’s ability to detect errors or monitor one’s performance. It is typically observed as a negative deflection in an ERP soon after an individual makes an error in a speeded reaction-time task. Several researchers have posited that the ERN arises from the anterior cingulate cortex (ACC; Dehaene, Posner, & Tucker, 1994; Holroyd, Dien, & Coles, 1998) thought to be involved in self-regulation (E. K. Miller & Cohen, 2001). A variety of studies have examined the ERN in relation to both depression and anxiety, with a greater or more negative ERN evident in anxious individuals (Gehring, Himle, & Nisenson, 2000; Hajcak, McDonald, & Simons, 2003, 2004; Hajcak & Simons, 2002; Johannes et al., 2001; Luu, Collins, & Tucker, 2000; Ruchsow et al., 2005; Stern et al., 2010; Anna Weinberg, Olvet, & Hajcak, 2010). Anxiety and depression are highly comorbid, with many researchers seeing them as distinct expressions of an internalizing spectrum (Krueger, 1999) related to negative affect (Clark & Watson, 1991). Adopting this continuum model, one might expect the results for anxiety disorder to hold for major depression. Contrary to such expectations, however, findings regarding the ERN in major depression have been quite mixed.

While some studies find that individuals with depression appear to show, as with anxiety, a larger ERN (Chiu & Deldin, 2007; Holmes & Pizzagalli, 2008, 2010) than controls, others have reported that depressed patients and controls do not appear to differ significantly in ERN amplitude (Compton et al., 2008; Olvet, Klein, & Hajcak, 2010; Ruchsow et al., 2004; Ruchsow et al., 2006; Schrijvers et al., 2008; Schrijvers et al., 2009). However, Schrijvers et al (2008; 2009) noted that the subjects used in their studies (where no differences were evident between depressed individuals and controls) were more severely depressed than the ones in studies where significant differences were found. Likewise, factors such as anhedonia (Olvet, et al., 2010) and psychomotor retardation (Schrijvers, et al., 2008) (which could be considered indicators of severity) were associated with a reduced or smaller ERN in such studies as well, making subjects with higher levels of these imparing characteristics more comparable to controls, relative to their less severe counterparts who, paradoxically, had larger ERNs. Along similar lines, more Weinberg and colleagues (A. Weinberg, Klein, & Hajcak, 2012) reported that while subjects with generalized anxiety disorder (GAD) alone showed an increased ERN, those with comorbid GAD and depression did not compared to controls; one possibility is that the observed comorbidity is a proxy for the recurrent subtype of depression in their studies. As a whole then, these results suggest that there is a subtype of depression that does not appear to affect the amplitude of the ERN – again, not what would be expected if depression were truly a continuous construct that mapped smoothly on to neurobiology.

Moving on to other domains, one can again see this theme of recurrence carrying over. For example, unlike non-recurrent depression, recurrence appears to occur at equal rates in men and women (Coryell, Endicott, & Keller, 1991; Kessler, McGonagle, Swartz, Blazer, & Nelson, 1993; Kovacs, Obrosky, & Sherrill, 2003; Simpson, Nee, & Endicott, 1997). Dysthymia also appears to be frequently comorbid with recurrent depression (Barkow et al., 2003; Warner, Weissman, Fendrich, Wickramaratne, & Moreau, 1992). Moreover, Klein et al. (2011) in their review on personality traits and depression noted that while negative affect was common to both depression and dysthymia, low positive affect was more specifically associated with dysthymia. They also reported that first-degree relatives of patients with chronic forms of major depression had greater levels of depressive personality traits. Examining the relationship between the course of depression and personality from age 17 to 29, Wilson et al. (2014) found that low positive affect at age 17 predicted the subsequent development of depression, but only for those who went on to have a recurrent course. Burcusa and Iacono (2007) and other researchers (Wichers, Geschwind, van Os, & Peeters, 2010) have observed that though recurrence in depression is a common phenomenon, exactly what causes recurrence is unclear. Additionally, they note that though it is posited that experiencing an episode somehow leaves a scar such that it increases the likelihood of future episodes (i.e., sensitizes or causes a “kindling” effect), there is not much evidence for scarring in prior studies. Indeed, Monroe and Harkness (2005) and Stroud, Davila and Moyer (2008) have found that the first episode of depression is more likely to be preceded by stressful life events rather than recurrent episodes. While this was interpreted as supporting kindling effects or the scar hypothesis, one can also see how the same results could be used to bolster the idea that recurrence breeds true relative to single episodes. Sullivan, Neale, and Kendler (2000) found precisely this in a meta-analysis of five family, adoption, and twin studies where recurrence was the clearest predictor of familial aggregation of the disorder in all the studies they reviewed as compared to other criteria such as number of symptoms, comorbid disorders, early age of onset, and duration of episodes.

This body of results suggests that recurrent depression may be qualitatively different from single episode depression, perhaps with a different genetic etiology, though our prelimiary efforts in this arena suggested that this might not be the case (Wilson, Vaidyanathan, Miller, McGue, & Iacono, 2014). What risky test could we undertake to evaluate further this possibility? One example would be to examine indices such as the blink reflex and ERN in a prospective study of adolescent identical twins where one twin has depression (the rationale behind such a twin study is detailed in the next section). If the affected twin has abnormal startle or a normal ERN, we would predict that the unaffected cotwin would show the same psychophysiological profile. As the twins mature, we would expect the affected twin to develop recurrent depression. Should the cotwin develop depression, we would expect it to become recurrent as well. In contrast, if recurrent depression reflected some sort of scarring effect, then we would expect the deviant psychophysiology to appear only after recurrent depression was established, and we would expect the two members of the pair to match on deviant psychophysiology only after a point where both had developed recurrent depression.

In summary, though prior work using self-report and diagnostic data (Haslam, et al., 2012; Kendler & Gardner, 1998) favors the conclusion that depression may be a continuous, unitary disorder, consideration of findings from other domains suggest that recurrent depression may not lie on this continuum. Recurrence, as specified in the current DSM, is not included among the criteria used to diagnose a major depressive episode; it can only be used as a specifier. This is an important point, as much of the influential statistical modeling in support of continuum conceptualizations uses cross-sectional epidemiological datasets and life-time or last-year snapshots of symptomatology or diagnosis (e.g., Krueger, 1999; Slade & Watson, 2006; Vollebergh et al., 2001). Developmental context, including onset and recurrence, is ignored in such designs.

To summarize then, while self-report symptoms and statistical models provided a common framework in which to define, organize, and understand depression, integrating information from other domains helps us refine our theoretical notions of the disorder much better. In other words, conceptual issue #1 that we raised earlier comes into obvious play here – i.e., statistical fit and parsimony of models of self-report symptoms can only take a researcher so far. As mentioned earlier, note also that genetic studies of the serotonin transporter gene (Karg, et al., 2011; Munafo, et al., 2009; Risch, 2009) have not shed much light on the etiology of depression, though arguably such studies are more technologically advanced than a simple blink reflex or ERP component (i.e., conceptual issue #3). Theoretical notions of the disorder (e.g., melancholia as a biologically distinct category – conceptual issue #2), along with stronger tests of that theory by integrating information from other domains (i.e., conceptual issue #4) add potentially important nosologically-relevant information not obtainable from statistical analyses of self-report symptoms or diagnoses alone.

3.2 Development of Adolescent Alcohol, Nicotine, and Marijuana Use and Abuse

There are a number of theories on the development and maintenance of substance use disorders covering a variety of perspectives. For the sake of simplicity, we consider a few that provide a concrete example of the main point of this paper – how to leverage the use of different research designs and methods to evaluate evidence for and against theories of mental disorders. Narrowing this focus allows demonstration of how one can derive a statistical model from a scientific theory and, through a combination of proper research design and statistical analysis, obtain a risky test of the scientific theory.

Nicotine dependence, alcohol dependence, marijuana dependence, and other drug dependence disorders, all correlate moderately (e.g., > .50), leading to the valid conclusion that these disorders share portions of an etiological framework (Krueger et al., 2002). Furthermore, researchers using twin studies have found that these correlations are driven primarily by genetic components (Kendler, Jacobson, Prescott, & Neale, 2003; Kendler, Prescott, Myers, & Neale, 2003; Krueger, et al., 2002; Young, Stallings, Corley, Krauter, & Hewitt, 2000), although the correlational framework and the relative importance of genes and environment significantly changes from adolescence into adulthood (Vrieze, Hicks, Iacono, & McGue, 2012). While an improvement over correlational findings in unrelated individuals, standard twin-based analysis (e.g., models of heritability) provides little insight into the nature of genetic etiology and, improperly used or interpreted, can easily mis-estimate the relative contributions of genes and environment to substance use behaviors (Keller, Coventry, Heath, & Martin, 2005; Plomin, DeFries, & Loehlin, 1977; Scarr & McCartney, 1983).

That individuals who use one drug tend to use another is now well established. Adding to this the fact that adolescents tend to experiment with drugs in temporal order (nicotine -> alcohol -> marijuana -> harder drugs) has led to a prominent theory of adolescent substance use development, termed the “gateway theory” (Kandel, 1975; Kandel & Jessor, 2002). This theory holds that when someone tries a drug, like tobacco or alcohol, the experienced high prompts them to try more drugs, perhaps those that provide stronger highs (Vanyukov et al., 2012). An excellent theory, the question is how one might test this as a causal hypothesis? The fact that adolescents tend to use drugs in a temporal ordering is consistent with the causal hypothesis, but this observation does not provide a risky test. One could think of other reasons for the temporal ordering; for example, the order of use is the same as the order of how difficult it is for an adolescent to obtain the drug and use it without being caught. The gateway theory would also appear to hold that nicotine, alcohol, and marijuana would correlate in adults, but the fact that they do correlate is also not strong evidence for the hypothesis. If the gateway theory survives these tests (temporal ordering; adult correlations) then it is corroborated, but only weakly because these are not risky tests. That is, the confirmatory observations could easily be the result of different causal processes that the tests do not rule out.

The ideal test would be to conduct a randomized controlled trial where one group of randomly assigned adolescents was exposed to a gateway drug (like marijuana) and another group was not. Then, during follow-up, determine the extent to which adolescents from each group are using other, harder, drugs. Clearly unethical, but there are ways to approximate this experimental state of affairs using natural experiments (Rutter, 2007), and create riskier tests for our theories, including the gateway theory. For example, Keyes and colleagues (2008) tested the association between parental smoking and offspring smoking in a sample of families with adopted and biological offspring. Since adopted offspring are genetically unrelated to the parents, this study can be construed as a reasonable test of the direct environmental impact of parental smoking, even though difficulties remain (e.g., parental smoking is confounded with other parental behaviors). Results indicated that adoptive offspring raised by smokers were somewhat more likely to smoke, but were no more likely to use any other drugs, than adoptive offspring raised by non-smokers – clearly not what we would expect if the gateway hypothesis were true and smokers tend to progress to other drugs. What is more, unlike the adoptive offspring of smokers, the biological offspring of smokers were much more likely to use tobacco, alcohol, and other drugs than biological offspring of non-smokers, suggesting a general genetic vulnerability to substance use and abuse that is largely independent of smoking experiences.

Along the same lines, Irons, McGue, Iacono, and Oetting (2007) used a quasi-experimental genetic method called Mendelian randomization (Ebrahim & Smith, 2008; Smith, 2011) in a longitudinal sample of South Korean adoptees to test the gateway hypothesis of whether environmental exposure to alcohol causes individuals to use other drugs later in life. Some individuals of Asian descent carry deficient variants in the ALDH2 gene, which metabolizes acetaldehyde to acetic acid. Those with the deficient gene do not metabolize acetaldehyde, a byproduct of alcohol. As a result, acetaldehyde builds up as a toxin in the body and causes a variety of unpleasant reactions such as tachycardia, flushing, and nausea. The result is that having the deficient version of ALDH2 is very protective for alcoholism (odds ratios confidence intervals of .06 to .30; Luczak, Glatt, & Wall, 2006), and those that have the deficient version of the gene will be exposed to less alcohol (because they refrain from drinking it) than those with the functional version. Thus, variation in this gene is analogous to an independent variable in an experiment—it is a proxy for environmental exposure to alcohol that is essentially randomized, in that it is unrelated to other environmental or biological risk factors for alcohol or drug use, especially in a sample where the parents are Caucasian (i.e., no one has the deficient gene) and biologically unrelated (it’s an adoption study). Irons et al. (2007) found that those individuals protected from alcohol exposure (effect size for a drinking index was .40), by virtue of having a deficient ALDH2 gene, were no more or less likely to use other drugs. The result is once again inconsistent with the gateway hypothesis. Other work has found analogous results for nicotine exposure, although using different genes as proxies for nicotine exposure (Vrieze, McGue, & Iacono, 2012). From these few studies alone, one can see that the natural experiments allowed more direct testing of the gateway hypothesis. The gateway hypothesis makes a prediction about an environmental cause of substance use. Both the adoption study and the Mendelian randomization study allow a quasi-experimental control of environmental versus genetic influences, and for this reason the tests are far riskier than those that depend on correlational data found in cross-sectional or longitudinal designs.

Of course, the gateway hypothesis is not falsified by such studies; it can be modified to be consistent with these findings (i.e., theorists could develop ad hoc hypotheses to deal with negative findings – within reasonable limits, of course; Meehl, 1990). For example, perhaps if you get drunk just once, this may be sufficient to potentiate the gateway effect. In that case, using ALDH2 for a Mendelian randomization study will not work, as deficiencies in ALDH2 do not prevent individuals from getting drunk once, they just discourage them from using as much and as often as others. In regards to the Keyes et al. (2008) adoption study of parental smoking, maybe tobacco is not a gateway drug, in part because it provides no obviously euphoric high. Future research will have to expand on these findings for alcohol and tobacco to marijuana and other drugs. To date, however, the gateway hypothesis passes only the relatively equivocal tests and fails riskier tests, those that approximate experimental designs (although work continues; Levine et al., 2011). The take-home point is that statistical modeling of correlational, non-experimental data is still correlational—it does not answer questions of causality, regardless of how many paths are drawn, models are fit, or fit indices are satisfied. Convincing nosological systems are those that are based on corroborated etiological theories, preferably with identified causal agents such as genes, pathogens, or environmental stressors that are developmentally informed. Obtaining this knowledge very often requires (quasi-) experimental research, and valid measures of the phenotype, genotype, and environment. Technological advances can contribute greatly to the measurement issue, but do not by themselves provide causal insight without appropriate research design and experimental controls.

The gateway theory is far from falsified at this point and clearly more work is required, hopefully using increasingly savvy designs and risky tests. In fact, one of the great virtues of the gateway theory is that it makes causal predictions that can be rigorously tested. However, let us assume for the sake of discussion that it has been falsified; what alternative theory could integrate and explain existing findings about the development of adolescent substance use? A plausible alternative could be termed a ‘common liability’ model, which posits that a general vulnerability to substance experimentation/use/addiction is an important component of adolescent use development (Vanyukov, et al., 2012). Our group, and others, theorize that the common liability to addiction can be substantially ascribed to individual differences in adolescent behavioral disinhibition (Iacono, Malone, & McGue, 2008; Zucker, Heitzeg, & Nigg, 2011), a broad concept that includes impulse control, incentive salience, and executive function (like planfulness). Deficits in these areas are posited to put youths at risk for experimenting, using, and eventually forming addiction to, substances like nicotine, alcohol, marijuana and other drugs. The theory posits that disinhibition is a neurodevelopmental trait with a substantial genetic etiology, and that children with greater deficits are at higher risk for use of substances generally, as well as for impulsive and antisocial behavior. The manifestation is heterotypic, in that children with these same disinhibitory deficits are at greater risk for oppositional behavior, ADHD traits, and difficulties in behavioral control. As the children age, these genetically driven deficits translate into increased risk for substance use experimentation and eventual addiction. While the theory places significant emphasis on the common liability (behavioral disinhibition), it also posits important drug-specific liabilities such as drug reward pathway susceptibility, peer influences, or drug availability. Indeed, it appears that the role of common liabilities is more important in adolescence, though drug-specific liabilities become more critical in early adulthood, at least for common substances (Vrieze, McGue, Miller, & Iacono, 2012). However, the point here is simply that a good theory provides the framework from which risky tests can be devised and experiments conducted. Without the theory we are blindly mining data.

Discussion of the common liability model leads to an illustrative conundrum in addiction research: why do individuals with histories of substance use and dependence suffer from poorer cognitive, interpersonal, and occupational dysfunction, as well as differences in neuroanatomy and function, compared to those without substance use histories? One possibility is that this result can be explained by the common liability theory, which would hold that poorer outcomes in adulthood are actually a heterotypic function of premorbid deficits. That is, if you have the common-liability-related deficits in executive function and behavioral control in adolescence, those deficits will continue into adulthood, whether or not you abuse psychoactive substances. Substance use may very well exacerbate those deficits, but the degree to which it does is an open question. Another theory, which might be called a toxicity theory, predicts that the role of premorbid deficits is small, but that negative outcomes for individuals with substance dependence histories is a result of brain damage from the substances. Which theory finds support from the evidence is far from a settled issue, the discussion of which provides us an opportunity to illustrate our conceptual issues discussed above, especially the use of theory and technological advances such as neuroimaging to address causal questions (i.e., the cause of negative outcomes for individuals with substance dependence histories).

Neuroimaging has shown that administration of nearly all abused psychoactive substances activate dopaminergic pathways in the brain (Parvaz, Alia-Klein, Woicik, Volkow, & Goldstein, 2011) including the prefrontal cortex (Goldstein & Volkow, 2002). These findings are the result of controlled experimental designs involving randomized controlled trials of substance administration followed by neuroimaging, as well as studies of individuals who are administered a substance in the midst of a brain scan (e.g., Volkow et al., 1988). A vast amount of neuroimaging research has also demonstrated, through contrasted groups designs, that the brains of those who become addicted to substances (like alcohol or cocaine) are in many ways different from the brains of those who do not become addicted to substances (Goldstein & Volkow, 2002). These results might seem to favor the toxicity hypothesis because the contrasted groups are selected based on their use of a substance, but this is far from clear. A contrasted groups design might show, beyond any doubt, that the brains of cocaine addicts are different from non-addicts, but they provide little to no evidence about why the brains are different. It may be that cocaine addicts have relative deficits in executive function and behavioral control prior to substance use, and these deficits persist or worsen into adulthood. The real utility of the contrasted-groups findings has been to cause scientists to speculate about why the contrasted-groups differences exist. Speculation of this nature has been extremely fruitful in addiction research, as neuroimaging results have driven scientists to create a wealth of etiological scientific theories of drug addiction (e.g., Goldstein & Volkow, 2002; Kalivas & Volkow, 2005; Robinson & Berridge, 1993) involving predictions about specific brain regions and function. Experimental psychology like this is, for practical and ethical reasons, primarily carried out in infrahuman species such as mice, and results are extrapolated to humans (e.g., Robinson, Gorny, Mitton, & Kolb, 2001; Spear, 2000). What is not clear at present is if and to what extent animal results extend to humans. Similar neuroimaging studies of adolescent humans with alcohol use disorders versus controls have found, for example, decreased hippocampal volumes (De Bellis et al., 2000; Medina, Schweinsburg, Cohen-Zion, Nagel, & Tapert, 2007; Nagel, Barlett, Schweinsburg, & Tapert, 2005; Nagel, Schweinsburg, Phan, & Tapert, 2005), altered parietal cortex and cerebellar responses during working memory tasks, and decreased white matter density (McQueeny et al., 2009), suggesting the possibility of decreased memory function caused by alcohol exposure. The literature on brain differences in adults with alcohol use disorders is even more extensive and suggestive of a causal role of alcohol exposure (Buhler & Mann, 2011). The major limitation of this research exists in all contrasted-groups designs: They are not risky tests. The exposure to alcohol is confounded with all the correlates of alcohol exposure that exist in the general population. It is known from long-term longitudinal studies that those adolescents who go on to use and abuse alcohol express higher levels of pre-morbid antisocial behavior (Hayatbakhsh et al., 2008; King, Iacono, & McGue, 2004; Sartor, Lynskey, Heath, Jacob, & True, 2007), impulsivity (Caspi, Moffitt, Newman, & Silva, 1996), cognitive deficits (Moffitt, 1993; Molina & Pelham, 2003), adversity in their home (Najman et al., 1997) and, most important here, premorbid differences in brain function (Iacono, Carlson, Malone, & McGue, 2002a; Iacono & McGue, 2006). That is, the existence of differences between alcohol-exposed brains and alcohol-naive brains does not provide a risky test of the hypothesis that alcohol exposure causes the observed brain differences, regardless of how technologically sophisticated the brain scan is.

What is required to test the toxicity theory is a (quasi-) experimental test in humans? A good start is with the theory (alcohol toxicity) and the prediction we derive from it. If we predict that adolescent alcohol exposure causes a different adult brain perhaps the most straightforward and obvious design is a prospective, longitudinal, co-twin control study, where monozygotic (MZ) twins are assessed prior to onset of alcohol use (e.g., prior to age 11), during the period of alcohol use (e.g., during high school), and at some point in early adulthood (e.g., after age 23). By nature, each MZ twin provides an experimental control for the other, in that they have (nearly) identical genotypes, as well as shared environmental experiences including rearing environment, school, neighborhood, access to alcohol, etc. If alcohol neurotoxicity is responsible for deleterious outcomes, MZ twins discordant for alcohol exposure should presumably also show discordances in brain structure and function as well as differences in psychosocial function and outcome (IQ, income, occupational success, personality, psychopathology). Alternatively, if premorbid risk factors are responsible for these outcomes, the exposed and unexposed twins should resemble each other on these outcomes, including the brain scans, which may provide the mosts sensitive test of whether and how the twins’ brains were different.

While we use alcohol toxicity to provide a concrete example, note that our discussion is applicable to any theory about the environmental causes of psychiatric disorder. Further, while a discordant MZ twin design might be preferred, we can get rough approximations to this quasi-experimental ideal with ordinary siblings. While not everyone has an MZ twin, most people have siblings or other first-degree relatives, and this practical observation has been used to great advantage in the study of rare neuropsychiatric disorders like schizophrenia. Such family-based controls are beginning to be applied to test the toxicity theory for various substances, using neuroimaging no less (Ersche et al., 2012). The Ersche et al. (2012) study found no striking differences between stimulant abusing probands and their healthy siblings, but both groups differed from non-psychiatric controls, indicating that the observed brain anomalies in the stimulant addicts likely preceded their abuse. Studies like this provide a great start to more rigorously evaluating the toxicity hypothesis.

The guiding theories here are central. If the alcohol toxicity hypothesis fails to explain the observed data, then it is discorroborated (or falsified), and this would have enormous public health and policy implications. If alcohol-use-discordant MZ twins are roughly equal in their outcomes, then it would suggest that premorbid problems are responsible for a significant amount of the morbidity and mortality associated with alcohol use. Alcohol use by itself would be considered less harmful, and intervention would focus less on the alcohol use itself and more on the disinhibition, undercontrol, and executive deficits posited by the common liability model. Preventative services instituted to high-risk children and adolescents would receive higher priority than at present and, by targeting common liability processes, would reduce their risk of developing addiction to many drugs, not just alcohol.

In fact, research addressing questions such as these using an alternative methodology, in this case borrowing again from psychophysiology, already exists. In the substance use literature, an early report that examined the pre-adolescent children of alcoholic fathers marshaled evidence that reduction in P3 amplitude was a candidate endophenotype reflecting genetic risk for alcoholism (Begleiter, Porjesz, Bihari, & Kissin, 1984). P3 waves arise in the ERP when applying laboratory tasks that require attention to and recognition of rare stimuli embedded in a series of more common stimuli. We do not argue the endophenotype candidacy of the P3 here (Euser et al., 2012; Iacono & Malone, 2011; G. A. Miller & Rockstroh, 2013), but only note that it is a technologically refined measure of psychophysiological processes related to information processing efficiency that has been profitably employed to study alcoholism as well as a wide range of psychiatric conditions.

Endophenotypes, such as the P3, are typically promoted for their supposed assistance in genetic association studies, but they have yet to demonstrate their promise for gene finding (Malone, Vaidyanathan, et al., 2014; G. A. Miller & Rockstroh, 2013; Vrieze, Malone, Pankratz, et al., 2014; Vrieze, Malone, Vaidyanathan, et al., 2014). In fact, we recently completed a comprehensive evaluation of endophenotypes using psychophysiological and molecular genetic data from paradigms examining P3, the startle blink response, antisaccade eye tracking performance, skin conductance orienting and habituation, and resting electronencephalogram (EEG) frequency band power. Employing a sample of approximately 5000 people, we subjected these putative endophenotypes to a risky test (Iacono, Malone, Vaidyanathan, & Vrieze, 2014; Iacono, Vaidyanathan, Vrieze, & Malone, 2014; Malone, Burwell, et al., 2014; Malone, Vaidyanathan, et al., 2014; Vaidyanathan, Isen, et al., 2014; Vaidyanathan, Malone, et al., 2014; Vaidyanathan, Malone, Miller, McGue, & Iacono, 2014; Vrieze, Malone, Pankratz, et al., 2014; Vrieze, Malone, Vaidyanathan, et al., 2014). While most of the indices in our sample were heritable in conventional biometric twin models, in molecular genetic tests, whether we examined GWAS common SNPs, rare variants via exome-sequencing, or used whole genome sequencing, we found little credible evidence that any of the molecular variants was associated with any of the indices. Even when using relaxed statistical significance thresholds that were justified by specifically examining lists of candidate genes and SNPs from prior studies demonstrating empirical support or theoretical relevance for our candiates, we did not find any that proved statistically significant. Such results indicated that despite the a priori promise derived from combining two advanced technologies – psychophysiology and molecular genetics – it was still not possible to uncover meaningful insights about genetic variants of possible importance to psychiatric disorders. To the contrary, our findings suggest that endophenotypes are just as complex as psychiatric disorders, and endophenotype genetics are just as complicated as psychiatric genetics. A far greater advantage of endophenotypes, in our opinion, is that they provide new ways to test and refine theories. Research on P3, in particular, is relevant to the evaluation of the disinhibition, gateway, and neurotoxicity theories considered in this section.

As noted earlier, studies using self-report data in genetically informed designs have concluded that a common genetic liability for behavioral disinhibition shared across substance use, childhood disruptive, and antisocial disorders accounts for their covariance (Bornovalova, Hicks, Iacono, & McGue, 2010; Hicks, Krueger, Iacono, McGue, & Patrick, 2004; Krueger, et al., 2002). Work with the P3 endophenotype reinforces the validity of this conclusion because each of these disorders individually has been shown to be associated with reduced P3 amplitude (Iacono, Carlson, Malone, & McGue, 2002b) as have externalizing problem behaviors such as early use of cigarettes (Yoon, Iacono, Malone, & McGue, 2006) and early sexual experience (Iacono & McGue, 2006). Further evidence that the clinical and endophenotype research are converging on the same etiological interpretation derives from reports showing that the general liability for externalizing psychopathology shares covariance with P3 amplitude (Patrick et al., 2006), and this association reflects shared genetic risk (Hicks et al., 2007). In combination, these findings support the conclusion that there are biological factors that account for why the same individual often engages in undersocialized behavior in adolescence and subsequently develops polydrug dependence. They also argue against the gateway hypothesis because reduced P3 amplitude is manifest prior to the initiation of substance use, predicts the subsequent development of all classes of substance dependence, and is associated with externalizing problem behaviors (e.g., conduct disorder, ADHD) that both precede and forecast the eventual development of addiction (Iacono & Malone, 2011).

Studies of P3 amplitude have also shed light on the alcohol neurotoxicity hypothesis. If brain mechanisms involved in the generation of P3 are damaged by drinking, then twins discordant for drinking should show corresponding discordance in P3 amplitude. However, results from prior work contradict this; within pair analyses comparing twins who abuse alcohol to their cotwins who don’t, have shown that the P3 amplitudes of discordant twins are the same, clearly challenging the neurotoxicity hypothesis (Carlson, Iacono, & McGue, 2002; Carlson, Iacono, & McGue, 2004). Another way to evaluate this hypothesis is to determine whether the high heritability observed for P3 amplitude is moderated by a history of alcohol consumption, thus indicating that alcohol is an environmental toxin altering the genetic influence on brain maturation. A prospective twin study that examined how alcohol consumption through adolescence moderated heritability of P3 amplitude at age 18 failed to find any effect, again refuting the neurotoxicity hypothesis at least for this one brain measure (Perlman, Johnson, & Iacono, 2009).

P3 studies are not without limitations. P3 does not capture all aspects of brain function, so the fact that P3 is unaffected by alcohol consumption does not argue that alcohol is without long-term deleterious effects on brain development. Moreover, as P3 accounts for only 3-4% of the variance in externalizing, it provides only a small part of the picture regarding the neurogenetic basis of the common liability model. Further, P3 is a complex waveform reflecting the aggregate influence of multiple neural generators. Decomposing it into component aspects has the potential to uncover brain activity more tightly tied to externalizing with more specific underlying neurobiology (Gilmore, Malone, Bernat, & Iacono, 2009; Gilmore, Malone, & Iacono, 2010). Given the success of P3, research on other endophenotypes for externalizing is clearly warranted.

To conclude, work with P3 supports the common liability-disinhibition model over the gateway hypothesis. The convergence of these P3 findings with the conclusions drawn from studies using other methodological approaches reviewed earlier (e.g., adoption and Mendelian randomization designs) is consistent with the common liability model. The jury is still out, but we attempted to draw this conclusion on the basis of research designs that controlled experimental variables and created riskier tests, which in turn gave us leverage to draw some conclusions about causality. In addition, we have discussed the great utility of technological advances in addiction research, but with comments on their limitations. Technological advances are not scientific advances until a scientist designs a study that uses that technology to engineer a test (preferably risky) of an etiological theory.

4. Concluding Remarks

We have argued that advances in understanding mental disorders are being stymied. We have discussed a variety of reasons for this: (a) overreliance on statistics and technology with little theoretical guidance, (b) a lack of integration of information across various methodologies that researchers use to study psychopathology, and (c) underutilization of research designs that provide for risky tests. Our point here was not merely to critique prevailing approaches to scientific investigation, but to also provide a constructive solution to some of these issues. Thus, we examined how to better conceptualize two different disorders – depression and substance use – using information from various domains, while keeping in mind the limitations we had outlined.

The key issue we have attempted to tackle is how to better extract meaning or interpret results we get from studies. We have more technological and statistical power than ever to conduct larger and larger studies, with more data at a greater resolution than has ever been available. Between this and the easy availability of software and methods for analyzing almost any kind of data, we have had an explosion of new journals and published articles. And yet, the progress we have made in understanding mental disorders is relatively modest given the degree of human suffering and their societal cost. This should be a very clear sign that what we lack is not information, but rather how to deal with it. Information, in and of itself, is inherently meaningless. It’s the quality of the information that we have in relation to our question, and how we use it that matters. The crux of the problem appears to be overreliance in the power of statistical software and technology. Otis Dudley Duncan (1984) referred to this tendency as “statisticism” and derisively noted that this was the “….notion that computing is synonymous with doing research, the naïve faith that statistics is a complete or sufficient basis for scientific methodology, the superstition that statistical formulas exist for evaluating such things as the relative merits of different substantive theories or the “importance” of the causes of a “dependent variable”; and the delusion that decomposing the covariations of some arbitrary and haphazardly assembled collection of variables can somehow justify not only a “causal model” but also, praise a mark, a “measurement model.”

Given the size and scope of datasets available today, running a factor analysis of personality data for 10,000 people or comparing brain scans or EEG responses between two disorders is easy to accomplish – and will likely yield results that are statistically significant but scientifically ambiguous. It is very easy at that point to retrofit a rationale to the findings, or come up with an explanatory hypothesis, and reify the presence of latent factors or the biological bases of disorders. This approach is reminiscent of what Richard Feynman (1974) referred to as “cargo cult science”. While such a procedure can be made to appear scientific superficially, chances of it leading to useful findings, theoretically or practically, are not very high. It is of course possible that we could stumble into some result at some point if we continue to conduct research this way. Alternatively, we could follow a more elegant and efficient route to useful findings if we thought a bit more critically about our constructs, methodologies, study designs, and statistics before running the next study.