Incorporating Neurophysiological Measures into Clinical Assessments: Fundamental Challenges and a Strategy for Addressing Them

Christopher J Patrick; William G Iacono; Noah C Venables

doi:10.1037/pas0000713

. Author manuscript; available in PMC: 2020 Dec 1.

Published in final edited form as: Psychol Assess. 2019 Mar 21;31(12):1512–1529. doi: 10.1037/pas0000713

Incorporating Neurophysiological Measures into Clinical Assessments: Fundamental Challenges and a Strategy for Addressing Them

Christopher J Patrick ¹, William G Iacono ², Noah C Venables ³

PMCID: PMC6754804 NIHMSID: NIHMS1019584 PMID: 30896211

Abstract

Recent scientific initiatives have called for increased use of neurobiological variables in clinical and other applied assessments. However, the task of incorporating neural measures into psychological assessments entails significant methodological challenges that have not been effectively addressed to date. As result, neurophysiological measures remain underutilized in clinical and applied assessments, and formal procedures for integrating such measures with report-based measures are lacking. In this paper, we discuss major methodological issues that have impeded progress in this direction, and propose a systematic research strategy for integrating neurophysiological measures into psychological assessment protocols. The strategy we propose is an iterative psychoneurometric approach that provides a means to establish multi-method (MM) measurement models for core biobehavioral traits that influence functioning across diverse areas of life. We provide a detailed illustration of a MM model for one such trait, inhibitory control (inhibition-disinhibition), and highlight work being done to develop counterpart models for other biobehavioral traits (i.e., threat sensitivity, reward sensitivity, affiliative capacity). We discuss how these measurement models can be refined and extended through use of already existing datasets, and outline steps that can be taken to establish norms for MM assessments and optimize the feasibility of their use in everyday practice. We believe this model-oriented strategy can provide a viable pathway toward effective use of neurophysiological measures in routine clinical assessments.

Keywords: neurophysiology, biobehavioral, multi-method, psychoneurometric, disinhibition

Recent scientific initiatives have advocated for the integration of neurobiological variables into clinical and applied assessments. For example, the National Institute of Mental Health’s (NIMH) Research Domain Criteria framework (RDoC; Cuthbert & Kozak, 2016) was established to foster increased use of neuroscience methods and concepts in models of mental health and illness. Specifically, RDoC calls for investigation of constructs such as acute threat, reward valuation, and response inhibition across multiple levels of analysis (ranging from genes to brain circuitry to overt behavior) to advance biobehavioral understanding and assessment of mental health problems. Parallel efforts are being pursued by the National Institute on Alcohol Abuse and Alcoholism (NIAAA; e.g., Kwako, Momenan, Litten, Koob, & Goldman, 2016) and the National Institute on Drug Abuse (NIDA; e.g., Ramey, 2017). Along similar lines, a recent report of the National Research Council (NRC; 2015) highlighted the need for incorporating data from neurophysiological tests into assessments of human capabilities in other applied contexts (e.g., employment selection, military duty assignment).

The shared focus of these initiatives is on characterizing psychological attributes of individuals (e.g., traits, capacities, symptom dimensions, disorders) in terms of core cognitive, affective, regulatory, and social processes, quantified using measures from physiological/neural along with behavioral-performance modalities. One basis of interest in doing so is to capitalize on the unique advantages of non-report based measures (e.g., objectivity, resistance to response bias, distinct prognostic value; Balsis et al., 2018; Iacono, 1991; NRC, 2015) to enhance the validity of psychological assessments. Another is to better understand the nature and origins of psychological attributes and how to manage or modify them. The field’s interest in moving toward multi-method (MM) clinical assessments is reflected in growing efforts to collect neurophysiological and task-behavioral data along with report-based measures in large-scale consortium projects focusing on etiological and developmental aspects of psychopathology (e.g., Alexander et al., 2017; Kotov et al., 2017; Schumann et al., 2010; Van Essen et al., 2013; Volkow et al., 2018). These newer consortium projects extend earlier-initiated, single-site projects of this type (e.g., Begleiter et al., 1995; Iacono et al., 1999).

However, the task of incorporating neurophysiological variables into applied psychological assessments involves significant methodological challenges (see, e.g., Iacono, 1991; Lilienfeld & Treadway, 2016; Patrick & Hajcak, 2016) that have not been effectively addressed in the literature to date. As a function of this, neural-response measures remain underutilized in clinical and applied assessments, and formal procedures for integrating such measures with report-based measures are lacking. As one illustration of this, neural measures are not listed as formal diagnostic indicators for any conditions in the current Diagnostic and Statistical Manual of Mental Disorders (DSM-5; American Psychiatric Association, 2013), apart from certain sleep disorders. Even conditions termed “neurodevelopmental disorders” do not include neurophysiological measures as formal diagnostic indicators. Recognizing the need for progress in this direction, the open call for proposed revisions to the DSM-5 manual (American Psychiatric Association, 2017) highlights as a specific priority evidence-based proposals for inclusion of biological indicators (“biomarkers”) in protocols for diagnosis.

With the foregoing in mind, one major aim of the current paper is to discuss major methodological issues that need to be addressed in order to effectively incorporate neurophysiological measures into clinical and other applied assessments. Our other major aim is to discuss a systematic research strategy for addressing these issues that can move the field toward an integrated, MM approach to the assessment of basic human attributes important to psychological health and adaptive performance. We focus specifically on how neurophysiological measures can be incorporated into a probabilistic, inter-individual (i.e., nomothetic, as opposed to idiographic) system for assessment, as this represents the dominant paradigm for psychological assessment at this time. The examples we provide to illustrate our points are mainly drawn from clinical assessment work, which has been the primary focus of our own research efforts, but many of the critical points we raise and the suggestions we offer for improvement pertain to other types of applied assessments as well (e.g., educational and aptitude testing, personnel selection and assignment). Our focus is not on neurophysiological signals per se or on methods of quantifying and analyzing brain-modality variables, but on basic measurement issues that have been seriously neglected in seeking to utilize neurophysiological measures for clinical and other applied assessment purposes. Without confronting and addressing these core measurement issues, research efforts directed at improving neurophysiological signal quantification and formulating models of neural response measures are unlikely to result in effective MM assessment protocols.

In the first part of this paper, we discuss major methodological challenges that have hampered progress in identifying biological indicators of psychopathology and integrating neurophysiological measures into clinical-psychological assessments. In considering these challenges, it becomes clear that a conceptual framework and methodology for achieving this does not yet exist. Following this, we discuss how a systematic modeling-oriented approach focusing on core biobehavioral traits quantified through multiple modalities of measurement can help to address these major unresolved issues. We provide an illustration of this approach as applied to MM assessment of inhibitory control, a biobehavioral construct of major relevance to impulse control (externalizing) problems. We end by describing ways in which this model-oriented measurement strategy can address methodological and logistical challenges confronting efforts to develop effective MM assessments and discuss how such assessments might be implemented in clinical settings.

Methodological Issues in using Neurophysiological Measures in Psychological Assessments

The most widely used methods for quantifying brain response in contemporary human research are scalp electroencephalography (EEG) and functional magnetic resonance imaging (fMRI). Other methods include magnetoencephalography (MEG; Papanicolaou, 2009), positron emission tomography (PET; Valk, Bailey, Townsend, & Maisey, 2004), event-related optical signal (EROS) imaging (Gratton & Fabiani, 2011), and functional near-infrared spectroscopy (fNIRS; Hirth et al., 1996). Physiological measures recorded from the face, limbs, or body such as electrodermal, cardiac, and electromyographic activity, though often referred to as “peripheral measures”, involve neural influences and have also been used to infer brain processes.

Given limits of space and scope, the empirical examples we provide in this article are primarily from studies using scalp-EEG based measures – in particular, event-related potential (ERP) responses (Coles & Rugg, 1995; Luck, 2014) – with some secondary reference to fMRI research. For example, several of the examples we refer to involve variants of the P3 (or P300) response, a positive-going ERP component that occurs within a few hundred milliseconds following rare or salient stimuli within an ongoing task. Of importance, however, the methodological points we discuss apply to physiological measures of any type when used to assess psychological attributes (e.g., traits, symptom dimensions, disorders).

As a point of reference for discussing methodological issues in the next section, we note that the main strategy used to date in studies attempting to identify non-report based indicators of psychological attributes (e.g., brain and other physiological variables; performance measures from behavioral tasks) has been a simple bivariate-mapping approach. This approach, which has serious disadvantages, is illustrated in Figure 1. It utilizes a report-based measure of a psychological attribute (phenotype) as the criterion and seeks to establish non-report based variables as indicators (“markers”) of this criterion. The report-based criterion can be either a continuous measure (e.g., a personality trait score or symptom score) or a categorical designation (e.g., clinical diagnosis present versus absent). The report-based measure is treated unnecessarily as an indisputable, unmodifiable criterion – i.e., as the actual psychological characteristic (“attribute-in-truth”) to which non-report based indicators are to be referenced. Although it seems natural from a psychological standpoint to use report-based measures as “gold standard” criteria, this places constraints on efforts to identify non-report based indicators of attributes, and – as we discuss in latter sections of this article – there are benefits to be gained from taking a different approach.

Figure 1. — Depiction of conventional bivariate-mapping approach to identifying neurobiological and other non-report based indicators of psychological attributes (e.g., traits, capabilities, symptom dimensions, disorders). Here, the target attribute (“X”) is first operationalized using report-based measures (e.g., questionnaire score, informant rating, clinician rating), and then efforts are made to identify individual variables (“ys”) from non-report modalities that correlate with the report-assessed attribute. y_{neuropsych-test} = neuropsychological test variable; y_{other-behavioral} = other behavioral-performance variable (e.g., from visual-motor task, affective-cuing task, etc.); y_{other-physiological} = other physiological variable (e.g., electrodermal activation, cardiac response, facial muscle tension, etc.). The report-based measure of the attribute is treated as an *unmodifiable* criterion, and neurobiological and task-behavioral variables are viewed as candidate indicators (markers) of this criterion.

Average Task Effects versus Inter-Individual Variation in Task Effects

A basic methodological question to consider is the extent to which observed variance in scores on a lab-task neurophysiological measure can be presumed to reflect a psychological characteristic of interest. In contrast with report-based scale measures (e.g., questionnaires, rating instruments) designed to measure psychological attributes of individuals relative to others (e.g., intelligence, anxiousness, suggestibility), neurophysiological tasks have historically been developed for purposes of indexing psychological processes or states of interest (e.g., fear activation, inhibition of prepotent responding, perceptual recognition). In contrast with report-based measurement, where items tend to be selected to index variations in a target attribute across individuals, the emphasis in developing neurophysiological tasks tends to be on maximizing average task effects by optimizing condition-manipulations to reduce variability across individuals. Moreover, while extensive effort is routinely devoted to confirming that variations in psychometric scale scores reflect individual differences in the attribute of interest (e.g., by showing that they covary with conceptually-related criterion measures but not conceptually-unrelated measures), researchers do not commonly undertake systematic validation work to confirm that variations in a neurophysiological-task measure reflect individual differences in extent of engagement of the specific process the task is designed to measure—as opposed to variations in some other process or processes.

As an example, a neurophysiological task procedure that has been widely used to investigate reward function in studies of psychopathology is the fMRI monetary incentive delay (MID) task, in which brain activation is measured during anticipation and subsequent receipt of gain outcomes (i.e., monetary rewards) as compared to no-gain or loss outcomes. The use of this task in clinical studies is premised on the idea that differences in brain activation between patient and nonpatient groups, or across individuals within a sample, are indicative of variations in reward sensitivity or responsiveness. However, this premise has not been systematically tested – i.e., the validity of MID-task brain activation scores for indexing individual differences in responsiveness to reward (i.e., dispositional reward sensitivity) remains to be established. Doing so would require demonstration of reliable convergent relations with previously validated measures of reward sensitivity, and consistent divergent (discriminant) associations with well-established measures of conceptually distinct attributes (e.g., threat sensitivity).

Score Reliability of Individual Neurophysiological Indicators

Until quite recently, scores for task-derived neurophysiological indicators have not been routinely evaluated for reliability. Score reliability within and across test sessions (i.e., both internal consistency and test-retest stability) is essential in assessment of trait dispositions, which are conceptualized as stable attributes that differentiate individuals. Score reliability within a session (i.e., consistency across items or test trials) is also important for quantifying psychological states (i.e., context-related conditions of individuals, such as fear activation or dysphoric mood) – to establish that measured variation across individuals is systematic rather than random, and capable of relating to other measures of interest. Although reports of reliability estimates for neurophysiological measures have appeared more frequently in the literature in recent years (e.g., Hajcak, Meyer, & Kotov, 2017; Infantolino, Luking, Sauder, Curtin, & Hajcak, 2018; Meyer, Bress, & Hajcak Proudfit, 2014; Thigpen, Kappenman, & Keil, 2017; Vetter et al., 2017; Yancey, Venables, & Patrick, 2016), this practice remains far from commonplace in clinical studies utilizing such measures.

Criterion Referent for the Neurophysiological Indicator

In assessments performed for clinical or other applied purposes, the aim is generally to assess psychological attributes (e.g., depression, impulsiveness, leadership) that relate to behavioral outcomes of interest (e.g., treatment response, suicide potential, work performance). The assessment utility of neurophysiological variables is typically evaluated by testing for relations with conventional report-based measures of attributes. A crucial issue with this approach is how effective (reliable, valid, specific) the measure is that serves as a criterion referent for the brain-response indicator. For example, categorical psychiatric diagnoses continue to be used as referents for identifying neural indicators of presence of or susceptibility to mental illness. However, there are well-documented weaknesses with operationalizing mental health problems in this way that pose problems for identifying brain indicators – including inadequate reliability of some disorder designations, diagnostic complexity (e.g., presence of symptom subdimensions and/or diagnostic subtypes), and comorbidity (systematic covariation among different disorders).

In the case of psychopathology, alternatives to categorical diagnoses exist. One is to utilize symptom counts (i.e., number of criteria met for a given DSM-defined disorder) as clinical referents in neurophysiological “mapping” studies. However, a problem with this approach is that DSM diagnoses were formulated using an expert-consensus process, rather than through systematic measurement-oriented research. To address this issue, efforts have been made historically to develop empirically based systems for quantifying psychological conditions or symptoms (e.g., Achenbach & Edelbrock, 1978; Eysenck, 1967). Contemporary instruments of this type include the Inventory of Depression and Anxiety Symptoms (Watson et al., 2007) for internalizing problems, the Externalizing Spectrum inventory (Krueger, Markon, Patrick, Benning, & Kramer, 2007; Patrick Kramer, Krueger, & Markon, 2013a) for impulse control problems, and the Personality Inventory for DSM-5 (Krueger, Derringer, Markon, Watson, & Skodol, 2012) for personality pathology. Building on these efforts, a major consortium initiative was recently launched to establish an integrative dimensional system for all forms of psychopathology (Kotov et al., 2017).

Empirically based systems focusing on continuous-dimensional quantification of clinical problems that consider patterns of overlap (comorbidity) are preferable to discrete categorical diagnoses as referents for identifying neurophysiological indicators of psychopathology. However, a dimensional diagnostic system in itself cannot ensure progress toward effective use of brain measures in clinical assessments. Reasons are discussed in subsections that follow.

Method Variance

One major challenge to identifying dependable brain indicators of psychological attributes or conditions is that that the latter are typically operationalized using report-based measures (i.e., questionnaires, informant or interviewer ratings). The difference in modality of assessment (self-report versus neurophysiological) poses an obstacle to convergence due to method variance, that is, systematic variability in scores attributable to distinct influences operating within a particular measurement modality (Campbell & Fiske, 1959). Due to unique influences affecting test scores within a particular assessment modality, comparably reliable and valid measures of a construct will covary more strongly with one another when operationalized in the same as compared to differing modalities of assessment. For example, two self-report-based measures with established validity as indices of fearfulness will tend to exhibit stronger relations with one another than either will with an overt behavioral or physiological measure of comparable validity—that is, reliable measures of the same construct from the same modality can be expected to correlate strongly (.6 - .8 range), whereas measures of the same construct from different modalities can be expected to correlate only moderately (.3 – 5). Correlations are likely to be even lower (i.e., .1 - .3) for measures of only somewhat related constructs from different modalities.

Because task-derived neurophysiological indicators can be expected to index constructs only somewhat related to those assessed by report-based measures of an attribute (which focus on self-ascriptions of general proclivities), correlations between neurophysiological and report-based measures are likely to fall in the .1 - .3 range. Though correlations higher than this have been reported in small-sample studies, recent large-N studies of relations between brain-response variables and report-based phenotypes point to only modest effect sizes (e.g., Castellanos-Ryan et al., 2014; Hicks et al., 2017; Yancey, Venables, Hicks, & Patrick, 2013; Yancey et al., 2016).

Specificity of Neurophysiological Variables as Indicators of Target Attributes

Apart from how reliable particular neurophysiological variables are, and to what extent they contain method-specific variance, a further issue is how selectively they index a specific attribute of interest (e.g., reduced sensitivity to rewarding outcomes, as related to depression) as opposed to other attributes. That is: How much of the reliable variance in the neural measure reflects the attribute of interest? This is important to consider because variation across individuals in a task effect of interest (e.g., brain reactivity to a cue for reward, relative to a non-reward cue) can reflect participant characteristics separate from the main attribute the task is designed to measure (e.g., reward sensitivity). Examples in this case might include attributes such as anxiousness or distractibility that affect neural reactivity within the task separately from reward sensitivity.

Recent research by Perkins, Yancey, Drislane, Venables, Balsis, and Patrick (2017) illustrates how a single neurophysiological measure can contain variance related to separate psychological attributes. The focus of this work was a brain potential measure of reactivity to abrupt acoustic stimuli occurring within a picture-viewing task – the noise-probe P3. This ERP component is a variant of the well-known P3 response that occurs to rarely-occurring (“oddball”) stimuli within a target detection task, as evidenced by a correlation of ~.3 between the two (Perkins et al., 2017). However, the noise-probe P3 is distinct from the oddball P3 response in that it occurs to an unexpected, intense stimulus (i.e., noise burst) that is perceived as aversive by most subjects. As such, it has been characterized as reflecting, in part, a defensive vigilance reaction to the acoustic stimulus that evokes it (Drislane, Vaidyanathan, & Patrick, 2013). Perkins et al. (2017) reported evidence of opposing directional associations for two questionnaire- assessed trait variables, threat sensitivity and disinhibition, with amplitude of noise-probe P3 in a picture-viewing task. The two traits, which were uncorrelated with each other, accounted for separate portions of variance in this brain response measure (i.e., both evidenced significant beta coefficients when entered together as predictors of noise-probe P3 in a regression model), indicating that their associations were driven by different sets of participants within the study sample. In addition, it was found that probe-P3’s association with disinhibition was attributable to variance in common with the oddball-P3 response, whereas its relationship with threat sensitivity was not. The authors’ interpretation was that the P3 response to noise-probe stimuli contains a component of variance reflecting a disinhibition-related process indexed by other variants of P3 (e.g., a general impairment in elaborative post-processing of stimulus events; see Patrick & Bernat, 2009), and a separate component of variance reflecting a fear-related process not represented in other variants of P3 (e.g., heightened vigilance following the occurrence of an unexpected aversive event; see Drislane et al., 2013).

A different scenario pertaining to specificity is one in which a neurophysiological measure taps variations in a distinct process across individuals that relates to different psychological attributes. An example of this might be a neurophysiological measure of arousability, such as change in alpha-frequency EEG activity during performance of a cognitive task (Ray & Cole, 1985), that relates to different psychological attributes which include a common element of arousability (e.g., sociability and sensation seeking). In this case, the two attributes would be expected to overlap in their associations with this neurophysiological measure (i.e., if variations in a common brain process account for observed associations of each). Conceptually, this would indicate an element of similarity between the two attributes that is not evident in the modality of self-report.

Both of these scenarios highlight potential discontinuity between how attributes are represented in modalities of person-report and neurophysiological response. In the first, correlations of differing portions of variance in a single neurophysiological measure with two distinct trait variables indicate that different attribute-related processes are tapped by the same reactivity index. In turn, this encourages a shift toward viewing the neurophysiological measure as multidimensional rather than unitary – and considering methods by which differing portions of variance in the measure might be parsed to index different attributes (e.g., quantitative methods such as structural equation modeling or multidimensional item-response modeling; Balsis et al., 2018). In the second scenario, relations of a common component of variance in a single neurophysiological measure with two putatively distinct traits suggests a role for some shared brain process in each. This could serve as an impetus for reconfiguring the two traits into three individual difference dimensions – one reflecting their mutual association with neurophysiological arousability, and the other two reflecting separate (and perhaps more psychological-experiential) aspects of each.

Score Aggregation

Report-based assessments of psychological attributes typically involve aggregation of scores across different items of an inventory or scale. Aggregating across items operates to harmonize responses to thematically-related but non-identical indicators around a common dimension of variation representing the target attribute. For example, self-report items such as “I frequently attend parties” and “I seek out positions of leadership” are less clearly indicative of extraversion when considered alone than when combined together with other items pertaining to outgoingness, friendliness, assertiveness, and activity level. Combining across multiple item-indicators results in lesser weighting of portions of variance in individual item responses that are unrelated to the target attribute, whether systematic (e.g., indicative of other attributes) or unsystematic in nature (e.g., related to carelessness or misreading).

Target-attribute related variance in neurophysiological response measures will invariably be entangled with non-relevant components of variance, thereby necessitating some means of isolating the variance of interest. One approach to achieving this is to aggregate across different attribute-related neurophysiological measures. Aggregation results in lesser weighting of portions of variance in individual indicators that are unrelated to the target attribute, to the extent the indicators covary mainly due to their mutual relations with the attribute of interest (i.e., they are “locally independent”, in latent-variable modeling terms). Aggregation is particularly useful for distilling attribute-related variance when neurophysiological indicators are of different types, or come from separate task procedures – because in these cases less of the covariance among indicators will reflect non-attribute related influences they share.

An example of this comes from work on reduced P3 amplitude as an indicator of externalizing proneness (disinhibition). This association has been demonstrated most frequently for P3 response to rare target stimuli in visual oddball tasks; consistent with expectation, the magnitude of this association is modest (~.2; e.g., Hicks et al., 2007; Yancey et al., 2013), indicating that about 4.4% of the portion of variance in oddball-target P3 response that is reliable (i.e., ~ .9, based on split-half estimation; Perkins et al., 2017) relates to externalizing proneness. However, other variants of P3 covary with externalizing proneness at similar levels, including ones derived from separate tasks as well as ones measured in the same oddball task (Nelson, Patrick, & Bernat, 2011; Patrick et al., 2013b). Variants of P3 from different tasks correlate less highly with one another (.2 – .4) than variants measured in the same task (.6 or higher; Nelson et al., 2011; Venables, Foell, Yancey, Kane, Engle, & Patrick, 2018a), indicating that a greater proportion of the covariance among different-task P3s reflects variance related to externalizing proneness (e.g., Burwell, Malone, & Iacono, 2016).

As a demonstration of this, Nelson et al. (2011) showed that the externalizing-related variance in P3 measures from three separate tasks (an oddball task, a flanker task, and a choice-feedback task) could be effectively distilled using factor analysis. Whereas the three individual P3 measures correlated only modestly with one another (median r = .26) and with a criterion measure of externalizing proneness (i.e., scores on a shortened version of Krueger et al.’s. [2007] Externalizing Spectrum Inventory), scores on a factor reflecting their shared variance correlated above .4 with scores on the externalizing criterion. Moreover, a factor analysis incorporating scores on the externalizing criterion together with the P3 brain measures yielded a single common factor on which all variables loaded to similar marked degrees (.44 - .60). The latter result highlighted the possibility of quantifying externalizing proneness (disinhibition) as an individual difference dimension residing between self-report and neurophysiological assessment modalities. We discuss further extensions of this work in the next major section of this paper.

Indicators of Liability for versus Expression or Consequences of Mental Illness

Another important issue in evaluating neurophysiological indicators of mental health problems is whether they represent indicators of liability for such problems, or indicators of neuropsychological dysregulation arising during episodes of mental illness – or of dysfunction arising from persisting expression of the mental illness, or medications used to treat it. Our perspective on biobehavioral attributes such as weak inhibitory control and high threat sensitivity is that they confer liability for clinical problems of certain types – i.e., impulse control disorders and focal fear disorders, respectively (Venables et al., 2017; Yancey et al., 2013). From this perspective, neural indicators of these attributes are expected to predate the occurrence of problems in later life (e.g., Berman, Whipple, Fitch, & Noble, 1993; Iacono, Carlson, Malone, & McGue, 2002) and remain evident across periods of illness and symptom remission (e.g., Porjesz, Begleiter, & Garozzo, 1980). Considered in this way, investigation of these traits, their neural correlates, and how they interface with clinical problems at different points in life will be essential to an understanding of the role neurobiological systems and processes play in the development, expression, and maintenance of psychopathology.

A Measurement-Based Approach to Integrating Neural Measures into Applied Assessments

Given the methodological challenges inherent in relating neurophysiological measures to psychological attributes, and the lack of large-scale datasets containing such measures (particularly in the case of neuroimaging), limited basis exists at this time for aggregating neurophysiological measures into norm-referenced scores that can effectively inform clinical diagnosis and decision-making. In this section, we describe a systematic methodological strategy for facilitating efficient progress in this direction, utilizing already-existing data together with coordinated collection of new data.

Psychoneurometric research strategy

In previous published writings (e.g., Patrick et al., 2012, 2013b; Moser, Durbin, Patrick, & Schmidt, 2015; Venables et al., 2017, 2018a, b; Yancey et al., 2016), we have described a psychoneurometric research strategy for incorporating neurophysiological measures into assessments of core attributes that are relevant to clinical (e.g., Depue & Iacono, 1989) and other applied outcomes (e.g., National Research Council, 2015). The term “psychoneurometric’ (abbreviated henceforth as PNM) connotes that this research approach is directed toward quantifying human attributes through combined use of psychological report and neurophysiological response indicators – but can be extended to incorporate other non-report based indicators (e.g., performance measures from behavioral tasks). This approach, which is rooted in the classic idea of delineating constructs in terms of “nomological networks” (Cronbach & Meehl, 1955), focuses on biobehavioral trait constructs such as threat sensitivity, reward sensitivity, inhibitory control, and affiliative capacity, which can be conceptualized in both psychological-behavioral and neurobiological terms (Kozak & Cuthbert, 2016; Patrick & Hajcak, 2016); these hypothetical trait constructs serve as anchors for integrating measures from modalities of psychological report and neurophysiology, and potentially other non-report modalities, into trait assessments with power to predict criterion measures across these different measurement modalities (e.g., reported and interview rated symptoms; other brain-response variables; behavioral performance outcomes).

This measurement-based strategy focuses on trait-dispositional constructs because traits are by definition trans-situational, and show broad predictive power for clinical-psychological and performance outcomes when assessed using report-based measures (e.g., Kotov et al., 2017; Krueger & Tackett, 2003; Samuel & Widiger, 2008; Widiger, 2013). However, the PNM approach focuses on trait constructs of a particular type – i.e., biobehavioral traits, reflecting classes of behavior theorized to relate to distinct evolved-adaptive systems of the brain (Lang, 1994, 1995; Miller & Cohen, 2001). As an example, the biobehavioral construct of threat sensitivity can be conceptualized as proneness to react more or less strongly to acute aversive stimuli (Yancey et al., 2016), as a function of constitutional and environmental influences that affect detection and processing of such stimuli and the degree to which they prompt activation of the brain’s defensive motivational system (e.g., Fanselow, 1994; Lang, 1995). Defined this way, variations in this trait can be expected to contribute, on average and across people, to self-perceptions of fearfulness in relation to threats of various types encountered in everyday life, brain and bodily reactivity within different situations involving threat, and inclination to enter versus avoid threat contexts and overt-measureable behavior within such contexts. This conception of biobehavioral dispositions reflects a realist position (e.g., Boorsboom et al., 2004; Tellegen, 1991), in which traits are viewed as psychobiological networks or structures (Allport, 1937; Eysenck, 1967) encompassing internal representations of percepts, actions/reactions, and semantics (e.g., connotations, interpretations, perspectives, beliefs) that can affect measured variables in different modalities (Lang, 1979, 1994).

Figure 2 illustrates the PNM research strategy, as applied to self-report (psychometric scale) and candidate neurophysiological indicators of a biobehavioral trait of interest. As outlined in Table 1, the approach proceeds in a series of iterative steps that lead to progressive modification of the initial conceptualization of the target trait: First, efforts are made to identify reliable neurophysiological indicators, from psychologically-relevant tasks, of a target attribute assessed through self- or other-report. We propose the use of psychometric scale measures of attributes conceptualized in biobehavioral terms as initial referents for this “mapping” process because they are coherent and reliable as well as efficient and cost-effective. However, to be considered a viable index of a biobehavioral construct, the scale measure should be composed of conceptually relevant items or item-sets that have been shown to relate to an established neurophysiological indicator of the attribute. Examples include scale measures of trait disinhibition that relate to P3 brain response (e.g., Brislin et al., 2018a; Yancey et al., 2013; see next subsection) and scale measures of trait fear/fearlessness that relate to aversive startle potentiation (e.g., López et al., 2013; Yancey et al., 2016; see also Kramer et al., 2012). Once additional indicators have been identified, analyses can be undertaken to evaluate their covariance structure, in part to refine neurophysiological quantification of the target attribute, and also to clarify the functional meaning of the neural indicators themselves (e.g., by considering common versus unique processing demands of lab-tasks they derive from). This is followed by efforts to (a) update conceptualization of the target attribute to incorporate insights gained from the structural analysis of neurophysiological indicators, (b) modify the psychometric scale measure of the target construct to reflect the neurally-informed conceptualization, and (c) apply understanding of relevant neural processes to create new lab tasks expected to yield more robust brain indicators of the target attribute. This process continues iteratively to the point where an optimal set of physiological tasks/measures exists for operationalizing the targeted biobehavioral construct in a sufficiently precise and reliable manner.

Table 1.

Procedural Steps in the Psychoneurometric Research Approach

Step	Description of Procedure
1. Develop scale measure of attribute	Create a report-based psychometric measure of the target attribute, composed of conceptually relevant items or item-sets that relate to a known neurophysiological (e.g., brain response) indicator of the attribute
2. Identify further neural indicators	Confirm that scale measure correlates with the initial neuro-physiological referent, and identify additional neural indicatorsby testing for relations with neurophysiological variables from psychologically relevant tasks in new or already existing datasets
3. Examine how neural indicators covary	Test for covariance among established neural indicators, and combine together sets of indicators that covary to form one or more neurometric measures of the target attribute
4. Test hypotheses as to nature of neurometric measure(s) of attribute	Generate and test hypotheses regarding the psychological nature of the neurometric indicator set(s), by assessing for predicted relations with criterion variables from psychologically relevant tasks
5. Revise conceptualization and scale measurement of attribute	Use insights gained from step 4 to revise conceptualization of brain processes related to the target attribute, and revise the scale measure to better index psychological counterparts to these processes

Open in a new tab

Note. The above-described approach is iterative in nature – i.e., the last procedural step (#5) entails revising the scale measure developed at step 1, after which steps 2 through 5 are repeated, until a coherent set of neurometric tasks/measures exists for assessing the target attribute reliably and effectively. Through this back-and-forth process, the original psychological conceptualization of the attribute shifts progressively across successive iterations of these latter steps. Alhough the description here focuses on neurophysiological indicators, it can also be applied to performance measures from conceptually relevant behavioral tasks – either separately from, or in conjunction with, neurophysiological indicators (see, e.g., Figure 3).

The PNM research approach provides a means for developing a MM measurement model for a biobehavioral trait of interest, which in turn can serve as a basis for a MM clinical assessment system. The next subsection describes an empirical illustration of a MM measurement model that highlights how the PNM approach can be applied to performance measures from laboratory tasks as well as neural-response measures to establish a model encompassing indicators from self-report, neural-response, and behavioral-performance modalities. The final major section below describes steps by which a measurement model of this type could be translated into a clinical assessment system.

Empirical Illustration: Multi-Method Model of Inhibitory Control Capacity

As a well-developed illustration of the PNM research strategy, we describe a recently reported quantitative-measurement model for the biobehavioral trait construct of inhibitory control (inhibition/disinhibition) that encompasses multiple variables from the modalities of self-report (i.e., trait scales), neurophysiological reactivity (i.e., ERP brain measures), and behavioral response (i.e., indices of performance from response inhibition tasks). We review lines of research leading to the formulation of this model and describe how it relates to criterion variables from different measurement modalities. Then, in the subsections that follow, we discuss how models of this type can help to address the methodological issues highlighted in part 1 of this article, and provide a pathway toward a MM framework for conceptualizing and quantifying human attributes for clinical and other applied assessment purposes.

Our efforts to develop a MM model of inhibitory control capacity began with the recognition that P3 brain response operates as a robust, replicable indicator of proneness to disinhibitory (externalizing) problems of various types. This idea was first put forth by Iacono et al. (1999) based on findings from a large, multi-method twin research project – the Minnesota Twin Family Study (MTFS). Begleiter, Porjesz, Bihari, & Kissin (1984) had presented evidence that reduced P3 amplitude might operate as an indicator of liability for the development of alcohol problems, and subsequent work suggested that reduced P3 might also index liability to problems with other substances (e.g., Biggins, MacKay, Clark, & Fein, 1997; Branchey, Buydens-Branchey, & Horvath, 1993). In addition, some evidence emerged for a relationship between reduced P3 and antisocial behavior problems (Bauer, O’Connor, & Hesselbrock, 1994). However, Iacono et al. (1999) were the first to posit that reduced P3 amplitude might be indicative of a general liability to externalizing psychopathology. Iacono et al. (2002) provided compelling evidence for this using data for 502 male MTFS participants assessed for P3 response at age 17 and externalizing problems both at age 17 and age 20. They reported that reduced P3 amplitude at age 17 was associated with externalizing problems of various types in participants themselves, and also with a history of such problems in participants’ fathers or their first-degree relatives. Additionally, reduced P3 response amplitude at age 17 predicted the later development of alcohol or other substance use disorders at age 20 in participants who did not exhibit such disorders at age 17.

In connection with this, other research was undertaken to clarify the nature and etiological basis of the general liability toward externalizing problems, and to directly demonstrate a relationship between this general liability and reduced amplitude of P3 response. Krueger et al. (2002) presented evidence that most (~80%) of the variance in general externalizing proneness as defined by impulsive symptoms (conduct disorder adult antisocial behavior, alcohol dependence, drug dependence) and disinhibitory personality traits was attributable to additive genetic influences, with the remainder accounted for by nonshared environmental influences. Krueger et al. (2007) later extended this work by developing a multi-scale questionnaire inventory, the Externalizing Spectrum Inventory (ESI), to provide a means for quantifying individual differences in externalizing proneness more continuously and comprehensively.

Patrick et al. (2006) used data for 17-year old male twins from the MTFS project (N = 969) to demonstrate that scores on a symptom-based general externalizing factor showed a robust negative relationship with P3 amplitude that accounted for relations of each individual diagnostic variable with this ERP measure. Hicks et al. (2007) then used data for a somewhat expanded version of this sample (N = 1196) to demonstrate that the relationship between general externalizing proneness and P3 amplitude was accounted for mostly by common genetic influences. More recently, using older adult twins (M age = 29.4; N = 419), Yancey et al. (2013) reported a modest but robust correlation (r = −.18, p < .0005) between scores on a scale measure of the general factor of Krueger et al.’s (2007) ESI, and showed that (a) a single common factor accounted for genetic variance in scores on this scale and genetic variance in interview-assessed symptoms of externalizing disorders, and (b) this common genetic factor accounted for correlations of each with P3 response amplitude. Taken together, results from these studies demonstrate that reduced P3 amplitude operates as an indicator of genetic liability for externalizing problems as a whole, and that this genetic liability can be indexed effectively using a scale measure of externalizing proclivities (termed “trait disinhibition” by Yancey et al.).

Separately from this work on externalizing and P3 response, evidence emerged during the 1990s and 2000s for impaired behavioral performance on task measures of frontal-brain based executive function (EF) in individuals with impulse control disorders. Pihl, Peterson, and Finn (1990), for example, reviewed evidence for EF-task deficits in sons of male alcoholics and suggested that impaired EF performance might be indicative of underlying genetic liability for alcohol problems. Subsequently, Morgan and Lilienfeld (2000) reported meta-analytic evidence for EF-task deficits in relation to antisocial behavior across studies of both children and adults. Drawing on these lines of evidence, along with research on the structure of cognitive performance measures (e.g., Engle, 2002; Miyake et al., 2000) and evidence for a large heritable component to externalizing proneness, Young et al. (2009) used data for a twin sample (M age = 17.4; N = 584) to quantify EF capacity in terms of performance on three response inhibition tasks (antisaccade, stop-signal, Stroop) and examine its relationship with general externalizing proneness and the etiological basis of this association. They found a robust association (~ −.4) between EF capacity defined in this way and externalizing proneness quantified as the factor in common among three symptom variables and a measure of disinhibitory personality. In line with twin-study findings for P3 response, biometric analyses of the etiologic bases of the association between EF task-performance and externalizing proneness showed it to be attributable mostly to shared genetic influences.

Considering these two lines of work, on associations of externalizing proneness with P3 brain response and with EF-task performance, Patrick, Foell, Venables, and Worthy (2016) hypothesized that P3 response and EF performance might operate as overlapping indicators of heritable liability for externalizing problems. This hypothesis served as part of the basis for work undertaken by Venables et al. (2018a) to develop a MM measurement model for the construct of inhibitory control (inhibition-disinhibition), conceptualized as externalizing proneness. This work tested for interrelations among 12 indicators of inhibitory control – 4 each from modalities of self-report, neurophysiology, and behavioral performance – and used structural equation analysis to organize patterns of covariance within and across modalities into an omnibus model. The self-report indicators were four scale measures of trait disinhibition, one of them composed of items from the ESI that index its general externalizing factor (cf. Yancey et al., 2013), two others developed to harmonize with this ESI Disinhibition scale (consisting of items from a general personality inventory in one case [Brislin, Drislane, Smith, Edens, & Patrick, 2015], and a psychopathy inventory in the other [Hall et al., 2014]), and the fourth a well-validated measure of unsocialized-delinquent proclivities (Gough, 1960). The neurophysiological indicators were four variants of P3 from three different tasks (cf. Nelson et al., 2011): a version of the ‘rotated heads’ oddball task that has been used in many studies of externalizing psychopathology; a flanker discrimination task; and a pseudo-gambling task in which gain or loss feedback appeared following choices between two alternatives. The behavioral performance indicators were measures of performance from four separate cognitive control (EF) tasks: an antisaccade task calling for instructed inhibition of natural eye movements; a version of the Stroop color-naming task; a stop signal task requiring inhibition of responses following their initiation; and a variant of a go/no go task in which a frequently enacted response had to be withheld on certain trials.

The structural model, depicted schematically in Figure 3, accounted for covariance among the 12 indicators quite well, as indicated by good fit according to quantitative indices. The model is a higher-order (correlated factors) model in which variables from each modality of measurement (self-report, neurophysiology, behavioral response) define lower-order modality factors, which load in turn onto a general, higher-order factor representing covariance across the three measurement modalities. This demonstration of a broad factor accounting for covariance among these different sets of indicators is consistent with the view of inhibitory control as a dispositional characteristic that influences measureable responses in different modalities.

Certain aspects of the model have interesting implications and warrant specific mention. One is that the model, when specified without constraints on loading parameters (either for individual indicators on modality factors, or for modality factors on the general factor), evidenced higher loadings for the neurophysiological and behavioral-response modality factors (−.77 and −.60, respectively) on the general factor than for the self-report modality factor (.40). The implication is that the general factor of the unconstrained model reflects inter-individual variation in brain reactivity and behavioral performance more so than variation in self-ascribed proclivities. As a function of this, scores on the general factor correlated more highly with criterion measures from the modality of neurophysiology (i.e., a separate variant of P3) and behavioral response (i.e., other task-performance variables) than with criterion measures of externalizing problems (i.e., self-report measures of antisocial conduct, alcohol/drug abuse, and impulsive-erratic personality disorder symptoms). The implication is that the general factor of this model was “shifted away” from the modality of psychological self-description toward the modalities of in-task neurophysiological and behavioral response. Of note, scores on the general factor of the model were negligibly correlated with a scale measure of socially desirable responding, in contrast with scores on the self-report modality factor, which showed a robust negative correlation. The implication is that multi-domain assessments of attributes may be less susceptible to well-known response biases.

Another notable point pertains to the fit and criterion-correlates of an alternative version of the model reported by Venables et al. (2018a), in which the loadings of the three lower-order modality factors onto the higher-order general factor were constrained to be equal, rather than being allowed to freely vary. This model – which yielded a general factor with comparable representation of variation in brain reactivity, behavioral performance, and self-reported proclivities – showed similar effectiveness in accounting for covariance patterns as the unconstrained model, as evidenced by comparable fit. However, the general factor of this alternative model showed increased correlations with criterion measures of externalizing problems relative to the counterpart factor of the unconstrained model – due to enhanced representation of disinhibition-scale variance in the general factor of the constrained model. The implication is that different versions of the general inhibition-disinhibition factor can be specified (by constraining loadings of modality factors, or by including other types of indicators in the model and/or modeling the data differently) to fulfill different assessment aims. For example, the general factor of the unconstrained model is likely to be more effective for quantifying levels of inhibitory control in studies of neurophysiological correlates/ mechanisms. By contrast, the general factor of the constrained model is likely to be more effective for clinical diagnosis and decision-making.

A further point regarding the Venables et al. (2018a) model is that it does not encompass all aspects of this construct as discussed in the literatures on personality (e.g., Berg et al., 2015; Carver et al., 2011; Whiteside & Lynam, 2001) and psychobiology (e.g., Bari & Robbins, 2013; Schall, Palmeri, & Logan, 2017). Rather, the model represents inhibitory control in a highly specific way, as resistance versus susceptibility to impulse-control problems (externalizing proneness), using indicators from different modalities of measurement that were expected to converge based on various lines of published research. However, the PNM research strategy does allow for delineation of other facets of a biobehavioral trait – based on observed patterns of covariation indicating separable sets of neural (or behavioral) indicators (see Table 1, steps 3 – 5). Though limited in scope, the Venables et al. model of inhibitory control capacity – along with others like it, for biobehavioral constructs like threat sensitivity, reward sensitivity, and affiliative capacity – can provide a concrete basis for addressing methodological and logistical challenges confronting efforts to incorporate neurophysiological measures into applied assessments, as discussed next.

From Bench to Bedside: Addressing Challenges to Development and Clinical Implementation of Multi-Method Assessments

Methodological Challenges

In the first part of this article, we discussed a number of significant methodological challenges facing efforts to incorporate neurophysiological measures into clinical assessment protocols. The first of these was the problem of interpreting inter-individual variation in a task-derived neural measure in relation to group-average effects for the task. The model-oriented strategy we have described addresses this issue by treating individual neurophysiological variables as candidate indicators (items) in a multivariate, MM measurement framework. Rather than presuming that scores on a task-derived neural response variable reflect individual differences in engagement of the process the task was designed to invoke on average (e.g., fear, error-monitoring), the meaning of variations in the neural variable is deduced from its relations with other variables – from within the same modality (i.e., neurophysiology) as well as from other modalities (e.g., self-report, other-rating, task performance). As an example, available research indicates that oddball-P3 brain response arises from different neural sources and reflects different sorts of cognitive operations. However, the findings of Venables et al. (2018a) indicate that the inter-individual variation in P3 response that relates to externalizing proneness (trait disinhibition) substantially reflects a brain process related to EF, as evidenced by the strong loadings of both the P3 and task-EF modality factors on the general factor of the MM model.

The next issue we discussed was score reliability. In our model-oriented approach, the measurement consistency (e.g., split-half, test-retest) of individual neurophysiological variables is not the prime concern. Individual variables are treated as items rather than as tests, so it is their aggregate reliability and collective ability to effectively index an attribute of interest that matters, more than their individual score reliabilities. From this standpoint, a modestly reliable difference-score variable may prove useful as an indicator (i.e., informative, in item response theory terms) if a sufficient portion of the systematic variance it contains relates to the attribute of interest. As an example, Yancey et al. (2016) found that aversive startle potentiation operated as an effective indicator of dispositional threat sensitivity even though it showed only modest reliability unto itself (Spearman-Brown corrected reliability = .43, based on a split-half r of .27).

Another methodological point we discussed was the nature of the individual difference characteristics to be used as a referents for identifying neurophysiological indicators. Our proposed measurement strategy focuses on attributes of a particular kind – biobehavioral traits. In contrast with clinical conditions or standard personality traits, biobehavioral traits are conceptualized specifically in terms of brain systems theorized to mediate distinct classes of adaptive behavior. Traits conceptualized in this way serve as anchors for linking together indicators from different measurement modalities, including neurophysiology, and interfacing these indicators with outcomes of applied interest (e.g., clinical conditions; Patrick et al., 2013; Yancey et al., 2016). To ensure efficient progress, the target biobehavioral trait is quantified initially using a report-based scale designed to relate both to the outcome of interest (e.g., clinical condition, or group of affiliated conditions) and to one or more candidate neurophysiological indicators of that outcome. An example of this was our development of a trait disinhibition scale to quantify general externalizing proneness in a manner that would relate to P3 amplitude. As another example, our conceptualization of threat sensitivity (Yancey et al., 2016) emerged out of evidence for a relationship between fear disorders of different types and aversive startle potentiation (see Vaidyanthan, Patrick, & Cuthbert, 2009), and efforts to quantify fear/ fearlessness in terms of a trait scale that would relate to aversive startle potentiation (Kramer et al., 2012; Yancey et al., 2015). The trait scales developed to index these two biobehavioral constructs show robust associations, respectively, with impulse-control problems of various types (Nelson et al., 2016; Yancey et al., 2013) and focal fear disorders (Nelson et al., 2016; Yancey et al., 2016). However, as described in Table 1 and illustrated in Figure 2, report-based measures of a target attribute are regarded as effective starting points for MM measurement models rather than relying on an arbitrarily chosen criterion as the gold standard for embodiment of the trait; the way in which an attribute is quantified, and conceptualized, can shift as a function of what indicators are included in the model and how they are weighted for particular assessment purposes.

Further methodological issues that we highlighted included method variance, specificity of neurophysiological variables as indicators, and score aggregation. Our model-oriented strategy confronts the issue of method variance through use of multiple indicators of a target biobehavioral construct, from different modalities of measurement. Reliability and specificity of measurement are provided by aggregating indicators both within modalities (through specification of lower-order factors) and across modalities (through specification of a higher-order, cross-modality factor). Within-modality aggregation enhances reliability by concentrating variance that is shared among indicators of a particular type; cross-modality aggregation enhances specificity of measurement by bringing together common-attribute related variance and de-emphasizing components of variance that are unique to particular modalities.

In these ways, our proposed strategy provides a means for separating out and clarifying portions of variance in individual indicators unrelated to the target attribute, but systematic (reliable) and potentially indicative of other psychological attributes. As an example, Perkins et al. (2017) presented evidence that separate portions of variance in noise-probe P3 related to attributes of threat sensitivity and inhibitory control. As such, noise-probe P3 could be used as an indicator in MM assessments of both of these attributes – e.g., by allowing it to cross-load onto modality-level factors in structural models for each, or through use of multidimensional item-response analysis (see Balsis et al., 2018, for discussion of applications of quantitative modeling to MM assessment).

One other methodological issue we discussed was the question of whether a particular neural indicator, or set of interrelated neural indicators, indexes liability to a particular problem or set of problems, or of an emergent pathological condition (or its neural consequences across time). Specialized research designs, such as prospective-longitudinal investigations or studies of twin participants, are critical to addressing this and other related questions (e.g., What neural processes should be targeted for intervention, at what points during development?). However, such studies are costly and time-consuming to conduct, and it is highly inefficient to utilize new candidate neurophysiological indicators in designs of this type without a systematic strategy for evaluating their potential effectiveness in advance. Our proposed measurement approach can provide conceptual-empirical frameworks for evaluating new candidate indicators – in the form of models that organize relationships among previously-established indicators from different modalities around individual difference constructs of broad relevance to clinical problems. Promising new neural indicators can be evaluated for convergence with elements of the model (i.e., other neural indicators; modality-level and cross-modality factors) for the most conceptually relevant attribute in samples requiring less time and expense to collect, before including them in more costly specialized designs.

As to the question of whether a given indicator indexes liability for or presence/consequences of psychopathology, our model-oriented approach provides a means for addressing this question through use of specialized research designs. As an example, the traditional approach to evaluating whether a neurophysiological variable operates as a liability indicator is to test in a longitudinal design whether it predicts the later emergence of a target disorder in asymptomatic individuals assessed for brain response early in life. This approach is subject to the various methodological issues described above, which complicate the task of identifying robust, specific indicators of liability or psychopathology expression. As an alternative, our proposed model-oriented approach calls for evaluating whether a given neurophysiological variable operates as an effective indicator of a biobehavioral attribute (trait) that relates to a particular clinical problem or set of problems. Specialized (e.g., longitudinal, twin) research designs can then be directed toward investigating the origins, temporal course, and emerging clinical correlates of these biobehavioral traits – how they manifest, as function of genetic and environmental influences, at earlier versus later ages; what neurophysiological variables relate to each across age periods, and the etiologic basis of these relations; and which indicators of each trait at early points in life are most associated (as a function of shared genetic influences, in particular) with the subsequent emergence of certain clinical problems. Proceeding in this way, problems arising at later ages are treated as nodes in a cross-temporal, MM prediction network. Neurophysiological variables that relate to later problems because they operate as indicators of relevant attributes early in life can be considered liability indicators; others that show associations with clinical problems but not with relevant early attributes can be evaluated as indicators of active pathology or its neural consequences.

Apart from methodological issues, efforts to develop standardized assessments that include neurophysiological measures face significant logistical challenges. In the next section, we discuss challenges of this type and how our proposed model-oriented approach can help to navigate them.

Logistical Challenges

Our model-oriented approach to developing integrated, MM assessment protocols for clinically-relevant biobehavioral traits focuses on procedures that are routinely used in developing self-report scale measures (i.e., iterative data collection and analysis; progressive refinement of scales, and constituent items, through analysis of their internal associations and relations with conceptually relevant criterion measures). Work of this kind requires relatively large samples and successive rounds of data collection. In the case of self-report scales, these requirements can readily be met by testing individuals in large numbers, either through group administrations or online survey systems. However, at this point in time, group or online testing is generally not feasible for neuro-physiological assessments.¹ The question arises, then, as to how to address issues of logistics and feasibility in seeking to develop MM assessment protocols for biobehavioral trait constructs.

Two features of our model-oriented approach – latent variable specification and measurement harmonization – are particularly helpful for addressing challenges of this sort. The first of these features refers to the focus of the model-oriented approach on specifying latent variables (i.e., lower-order factors representing expressions of an attribute in different measurement modalities, and higher-order factors representing attributes across modalities) and delineating their associations, rather than on relating individual manifest indicators (e.g., specific neurophysiological variables) to individual measured outcomes (e.g., clinical diagnoses, or symptom count variables). The focus on latent variables greatly enhances investigative efficiency by coordinating efforts around omnibus factors – corresponding to core biobehavioral traits, and their expression in different modalities – rather than individual measures, and it fosters replicability by providing stable referents (i.e., model dimensions reflecting systematic covariance among already-established indicators) for evaluating new candidate indicators.

More specifically, we propose that efforts to integrate neurophysiological measures into applied assessments should focus at this time on developing MM measurement models for a small number of biobehavioral constructs with clear relevance to distinct domains of personality and clinical problems. These include: inhibitory control (inhibition-disinhibition), a construct of particular relevance to impulsive traits and externalizing problems; threat sensitivity, a biobehavioral construct pertinent to fear-related traits and focal fear disorders; reward sensitivity, a construct with specific relevance to positive emotional traits and dysphoric-depressive conditions; and affiliative capacity, a construct pertinent to empathic versus antagonistic traits and clinical conditions marked by asociality and callous-disregard. As an indication of their potential utility for linking neurophysiological variables with personality and psychopathology, these dispositional constructs correspond to key process constructs in four of five domains of the NIMH RDoC framework, and they have clear counterparts in prominent trait models of personality (e.g., McCrae & John, 1992; Tellegen & Waller, 2008) and in a recently proposed hierarchical-dimensional model of psychopathology (Kotov et al., 2017). In addition, progress has been made toward identifying neural indicators of trait-scale measures of each and demonstrating relationships in turn with distinct clinical conditions (for the construct of inhibitory control, see, e.g.: Patrick et al. [2013b] and Venables et al. [2018a]; for threat sensitivity, see, e.g.: Yancey et al. [2016]; for reward sensitivity, see, e.g.: Bowyer et al. [2018] and Hajcak Proudfit [2015]; for affiliative capacity, see, e.g.: Brislin et al. [2018b] and Marsh et al. [2008]).

A second key feature of our model-oriented approach, related to its latent-variable focus, is the platform it offers for use of existing large-scale datasets, or integrating data across different projects, by means of measurement harmonization. By harmonization, we refer not just to formatting and organizing common variables in a compatible manner across datasets (e.g., Doiron et al., 2013), but to the linking of datasets through non-identical measures shown to operate as interchangeable indicators of a common construct (Friedman, Kern, Hampson, & Duckworth, 2014; Patrick & Hajcak, 2016). As an example, 3 of the 4 self-report scale measures used by Venables et al. (2018a) as indicators in their model of inhibitory control evidenced loadings of .8 or higher on the self-report modality factor – indicating that these scales operate as interchangeable indicators of inhibitory control as expressed in the modality of self-report. The close convergence among these three scales is not happenstance: One of them, the ESI Disinhibition scale, was developed to index the ESI’s general externalizing proneness factor (Patrick et al., 2013a), and the other two consist of items selected from separate inventories to index externalizing proneness in a parallel manner (Brislin et al., 2015; Hall et al., 2014). (The fourth scale indicator, which was not developed specifically to harmonize with ESI externalizing proneness, loaded less strongly onto the scale-modality factor.)

The availability of these compatible self-report measures of inhibition-disinhibition reflects efforts we have devoted to developing harmonized scale measures of this construct along with two others – threat sensitivity (vs. boldness; Patrick & Drislane, 2015) and affiliativeness (vs. callousness; Brislin et al., 2018b) – using items from widely used inventories of normal and abnormal personality (see, e.g.: Brislin et al., 2015; Drislane et al., 2015, 2018; Hall et al., 2014; Sellbom, Drislane, Johnson, Goodwin, Phillips, & Patrick, 2016). The creation and validation of these harmonized scales allows for these biobehavioral constructs to be operationalized through self-report in already existing datasets that include scores for these personality inventories – including datasets for specialized (longitudinal, twin) projects such as the MTFS (Iacono et al., 1999), the Dunedin Multidisciplinary Health and Development Study (Silva, 1990), and the Swedish Twin study of Child and Adolescent Development (TCHAD; Lichtenstein, Tuvblad, Larsson, & Carlstrom, 2007). The ability to quantify these constructs in the MTFS dataset is especially relevant to the current discussion because this dataset includes physiological measures of various types (including electrocortical measures for all twin participants, and fMRI measures for a portion of them) along with questionnaire and interview-diagnostic data. However, data from large-scale etiologically informative projects that do not include neurophysiological measures, such as the TCHAD longitudinal-twin project, can also be of enormous value for clarifying the etiology and developmental course of biobehavioral traits assessed via self- or other-report (e.g., parent ratings), the nature and bases of their associations with clinical problems, and the extent to which trait measures are indicative of problem liability versus expression.

In addition, it is also possible to develop harmonized measures of these constructs in other specialized datasets using items from content-relevant scales that happen to be available. As an example, Brislin et al. (2018a) developed a scale measure of trait disinhibition (externalizing proneness) using items from questionnaires administered in the European IMAGEN project (Schumann et al., 2010), a large-scale, multi-site longitudinal study of substance abuse risk that includes neuroimaging (structural and functional MRI) data along with self-report, interview-diagnostic, and task-behavioral measures. These authors showed that this IMAGEN-Disinhibition scale predicted externalizing symptoms both concurrently and prospectively in the IMAGEN sample, and – using data from a separate American sample – showed that the scale covaried strongly (~.8) with the ESI Disinhibition scale and modestly (~ −.3) with P3 brain response to target stimuli in an oddball task.

The latter finding for the IMAGEN Disinhibition scale – its correlation with a known brain-response indicator of externalizing proneness – illustrates a critically important point about our proposed model-oriented approach: namely, that measures that harmonize well with established indicators in a structural model can be expected to show parallel relations with other indicators in the model. Given the IMAGEN-Disinhibition scale’s strong convergence with ESI Disinhibition, and previously reported relations for the latter with P3 amplitude (Venables et al., 2018a; Yancey et al., 2013), we expected that scores on the former would similarly predict P3 amplitude. Based on prior findings (e.g., Venables et al., 2018a; Young et al., 2009), we also expected that the IMAGEN Disinhibition scale would correlate negatively with performance on cognitive control (EF) tasks administered to IMAGEN participants – and we found evidence for this as well (Foell et al., 2016). However, our main goal in creating a harmonized scale measure for the IMAGEN project was not to replicate previously reported correlates of trait disinhibition, but rather to use this scale to identify neuroimaging correlates in this very large, ongoing longitudinal project. The inclusion of resting state MRI and diffusion tensor imaging scans in the IMAGEN protocol, along with structural scans and fMRI data from four tasks (see Castellanos-Ryan et al., 2014), provides rich opportunities for this.

Apart from the unprecedented size of the IMAGEN sample, which allows for internal cross-validation of detected effects using a split-half approach, access to performance data for cognitive tasks along with trait-scale data provides distinct advantages for identifying structural and functional MRI correlates of disinhibition. As noted in our earlier discussion of the Venables et al. (2018a) model for this trait, especially strong convergence was found between neurophysiological and performance modality factors in the unconstrained version of the model – i.e., these factors each loaded more strongly on the higher-order [cross-modality] factor than did the self-report modality factor. The implication is that scores on the cross-modality factor, which preferentially reflect covariance between brain-ERP and task-performance measures, can be approximated (i.e., estimated) better through combined use of scale-measures and task-performance measures than through scale measures alone. Evidence for this was reported by Venables et al. (i.e., scales scores, when averaged together with task- performance scores, predicted brain-ERP scores more effectively than scale scores alone).

Challenges to Use in Clinical Practice

Assuming that the foregoing challenges can be surmounted, and effective MM protocols can be developed for assessing clinically-relevant biobehavioral traits, two potential impediments exist to their routine use in clinical practice. One is test length (i.e., time required to collect data for a battery of tasks designed to yield non-report based indicators; see, e.g., Kwako et al., 2016; Venables et al., 2018) and the other is the need for specialized recording equipment and administration skills.

Regarding the issue of test length, a means for addressing this would be to utilize a reduced set of tasks and indicators for routine assessment purposes. For example, in cases where standard report-based test information points to the need for a MM assessment, clients could first be assessed using a partial set of tasks/indicators – and administered the remainder of the task battery only if scores from the partial task-set indicate a need for this. The ability to estimate scores from partial data represents a notable advantage of a model-based assessment approach. A broad illustration of this is provided by the increasing practice of computerized adaptive testing, in which items sampled from a large test pool are used to assess positions of individual examinees on a latent trait of interest – through reference to a measurement model for the full item set. As a more specific illustration, we found that scores on the general factor of the Venables et al. (2018a) model could be estimated with high precision (i.e., r ~ .8) using just 3 of the 12 indicators of the model (i.e., 1 from each measurement modality).

It is beyond the scope of this article to discuss in detail how the feasibility of MM biobehavioral assessments can be enhanced through use of score estimation, so we refer readers to other recent writings for further elaboration of this point (e.g., Balsis, Benge, Lowe, Geraci, & Doody, 2015; Balsis et al., 2018). However, in principle, it should be possible to utilize normative-sample datasets containing scores for a full array of self-report, electrocortical (EEG/ERP), neuroimaging (fMRI), and task-behavioral measures to estimate, with specifiable probability, the positions of new examinees along MM attribute dimensions through use of partial data for those examinees. Item-response theory (IRT) methods, which focus on quantification of individual differences in latent traits, could be used to evaluate the reliability of model-derived score estimates (for a relevant discussion, see Balsis et al., 2018).

Regarding the issue of specialized equipment and skills needed to perform MM biobehavioral assessments, our view is that this is more an issue of supply and demand than of feasibility. Traditional neuropsychological assessments require specialized (and increasingly, computerized; Bauer et al., 2012) task procedures and skills training and have been widely used in used in clinical setting for many years. Equipment for electrophysiological (including scalp EEG/ERP) measurement has become more compact, affordable, user-friendly, and tailored to specific uses. If a palpable demand arose from psychological service providers for a fully automated system to administer tasks and collect data for purposes of a MM assessment of the four above-noted biobehavioral traits (inhibitory control, threat sensitivity, reward sensitivity, affiliative capacity), established medical device manufacturers such as Medtronic Inc. or Compumedics Ltd. could readily fill it. Specialized technicians could be trained and certified to collect and score data using an automated system of this kind, in order to perform reliable assessments that are standardized across persons and test sites, and generate algorithm-based score interpretations that take into account age, gender, and other characteristics of the individual in evaluating test data against norms.

Table 2 presents a summary of specific procedural steps that could be taken to allow a MM biobehavioral assessment protocol, when developed to a sufficient degree, to be implemented in clinical settings.

Table 2.

Procedures for Clinical Implementation of Multi-Method Biobehavioral (MMB) Assessments

Procedure	Description
1. Develop research base and preliminary norms for MMB assessment system	Use PNM strategy (see Table 1) to develop research base for the MMB assessment system, and use preliminary norms from this research base to launch clinical implementation steps that follow
2. Make requisite equipment and training available	Establish a corporate provider for a computerized MMB assessment system (i.e., for standard task administration and data collection/scoring); develop a training protocol for MMB assessment personnel
3. Establish clinical norms for MMB assessment data	Collect normative data from clinic patient samples, including full array of MMB assessment measures along with variables that may moderate MMB scores (e.g., demographics) and/or predict their clinical utility (e.g., family and personal health history)
4. Develop actuarial criteria for gauging need for MMB assessment	Use normative database to develop report-based criteria for estimating, in cost-benefit terms, the need for MMB assessment in individual cases
5. Establish decision-tree for partial and full task- based MMB assessments	If results of initial report-based assessment data indicate need, refer client for partial-task MMB assessment; if partial-task results indicate further need, complete full MMB assessment
6. Further evaluate MMB measurement models and refine tasks/indicators	Use data from ongoing client assessments to further evaluate the clinical validity and utility of MMB measurement models for target traits, and further refine task-based assessment protocols for each
7. Commercial release of MMB assessment system	Establish clinical certification protocol for MMB assessment technicians; release system commercially for use in clinic settings

Open in a new tab

Note. PNM = psychoneurometric research approach outlined in Table 1 and depicted in Figure 2. Multi-method biobehavioral (MMB) assessment = applied product of the PNM research approach.

Conclusion

In an article for another special issue of this journal published over 15 years ago, Iacono (1991) noted that: “Psychophysiological methods have been widely used in psychopathology research, but they have yet to be exploited as assessment techniques.” While highlighting obstacles to progress at the time including a predominant practice of reporting physiological test data in group form, typically for small samples, and limited or uncertain replicability of reported results, he expressed optimism about prospects for developing psychophysiological assessment protocols with clinical utility. Our central point in the current paper is that brain and other physiological measures remain underutilized in assessments of psychopathology and that a systematic research strategy is needed for integrating neurophysiological measures into protocols for clinical and other applied assessments. The research strategy we propose is an iterative psychoneurometric approach that can be used to establish MM assessment models for core biobehavioral traits that influence functioning in diverse arenas of life. We provided a detailed illustration of a MM model for one such trait, inhibitory control (inhibition-disinhibition), and highlighted work now being done to develop counterpart models for other core traits (i.e., threat sensitivity, reward sensitivity, and affiliative capacity; Bowyer et al., 2018; Brislin et al., 2018; Yancey et al., 2016). We discussed how these models can be extended through use of already existing specialized datasets, and outlined steps that can be taken to establish norms for MM assessments, optimize the feasibility of their use, and implement them in everyday practice. We believe this proposed approach can provide a pathway toward effective use of neuro-physiological measures in clinical assessments within the near term as opposed to the indefinite future.

Public Significance Statement.

Neurobiological factors are important for understanding mental health problems, but neural measures remain under-used in clinical assessments. We discuss methodological issues contributing to this lack of use, and propose a systematic research strategy for integrating neurophysiological measures into psychological assessment protocols.

Acknowledgments

Work on this article was supported by grant W911NF-14-1-0018 from the US Army, and grants R01 DA036216, R37 DA005147, and T320A037183 from the National Institute on Drug Abuse. The content of this paper is solely the responsibility of the authors and does not necessarily represent the official views of the U.S. Government, Department of Defense, Department of the Army, Department of Veterans Affairs, or U.S. Recruiting Command.

Footnotes

This statement is qualified by the fact that some physiological reactions, such as pupil response or facial muscle change, can be assessed through video recording and offline analysis.

Contributor Information

Christopher J. Patrick, Florida State University

William G. Iacono, University of Minnesota

Noah C. Venables, University of Minnesota

References

Achenbach TM, & Edelbrock CS (1978). The classification of child psychopathology: A review and analysis of empirical efforts. Psychological Bulletin, 85, 1275–1301. [PubMed] [Google Scholar]
Alexander LM, Escalera J, Ai L, Andreotti C, Febre K, Mangone A, ..., & Milham MP (2017). An open resource for transdiagnostic research in pediatric mental health and learning disorders. Scientific Data, 4:170181. doi: 10.1038/sdata.2017.181. [DOI] [PMC free article] [PubMed] [Google Scholar]
Allport GW (1937). Personality: A psychological interpretation. New York: Holt. [Google Scholar]
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (DSM-5). Washington, DC: Author. [Google Scholar]
American Psychiatric Association. (2017, August 1). Submit proposals for making changes to DSM-5. See: https://www.psychiatry.org/psychiatrists/practice/dsm/submit-proposals
Balsis S, Benge J, Lowe D, Geraci L, & Doody RS (2015). How do scores on the ADAS-Cog, MMSE, and CDR-SOB correspond? Clinical Neuropsychologist, 29, 1002–1009. [DOI] [PubMed] [Google Scholar]
Balsis S, Choudhury BS, Geraci L, & Patrick CJ (2018). Alzheimer’s disease assessment: A review and illustrations focusing on item response theory techniques. Assessment, 25, 360–373. [DOI] [PubMed] [Google Scholar]
Bari A, & Robbins TW (2013). Inhibition and impulsivity: Behavioral and neural basis of response control. Progress in Neurobiology, 108, 44–79. [DOI] [PubMed] [Google Scholar]
Bauer LO, O’Connor S, & Hesselbrock VM (1994). Frontal P300 decrements in antisocial personality disorder. Alcoholism: Clinical and Experimental Research, 18, 1300–1305. [DOI] [PubMed] [Google Scholar]
Bauer RM, Iverson GL, Cernich AN, Binder LM, Ruff RM, & Naugle RI (2012). Computerized neuropsychological assessment devices: Joint position paper of the American Academy of Clinical Neuropsychology and the National Academy of Neuropsychology. Archives of Clinical Neuropsychology, 27, 362–373. [DOI] [PMC free article] [PubMed] [Google Scholar]
Begleiter H, Porjesz B, Bihari B, & Kissin B (1984). Event-related brain potentials in boys at risk for alcoholism. Science, 225, 1493–1496. [DOI] [PubMed] [Google Scholar]
Begleiter H, Reich T, Hesselbrock V, Poriesz B, Li TK, Schuckit MA, Edenberg HJ, & Rice JP (1995). The Collaborative Study on the Genetics of Alcoholism. Alcohol Health & Research World, 19(3), 228–236. [PMC free article] [PubMed] [Google Scholar]
Berg JM, Latzman RD, Bliwise NG, & Lilienfeld SO (2015). Parsing the heterogeneity of impulsivity: A metaanalytic review of the behavioral implications of the UPPS for psychopathology. Psychological Assessment, 27, 1129–1146. [DOI] [PubMed] [Google Scholar]
Berman SM, Whipple SC, Fitch RJ, & Noble EP (1993). P3 in young boys as a predictor of adolescent substance use. Alcohol, 10, 69–76. [DOI] [PubMed] [Google Scholar]
Biggins CA, MacKay S, Clark W, & Fein G (1997). Event-related potential evidence for frontal cortex effects of chronic cocaine dependence. Biological Psychiatry, 42, 472–485. [DOI] [PubMed] [Google Scholar]
Bogg T, & Finn PR (2010). A self‐regulatory model of behavioral disinhibition in late adolescence: Integrating personality traits, externalizing psychopathology, and cognitive capacity. Journal of Personality, 78(2), 441–470. [DOI] [PMC free article] [PubMed] [Google Scholar]
Borsboom D, Mellenbergh GJ, & van Heerden J (2004). The concept of validity. Psychological Review, 111(4), 1061–1071. [DOI] [PubMed] [Google Scholar]
Branchey MH, Buydens-Branchey L, & Horvath TB (1993). Event-related potentials in substance-abusing individuals after longterm abstinence. American Journal of Addictions, 2, 141–148. [Google Scholar]
Brislin SJ, Drislane LE, Smith ST, Edens JF, & Patrick CJ (2015). Development and validation of triarchic psychopathy scales from the Multidimensional Personality Questionnaire. Psychological Assessment, 27(3), 838–851. [DOI] [PMC free article] [PubMed] [Google Scholar]
Brislin SJ, Patrick CJ, Flor H, Nees F, Heinrich A, Drislane LE, Yancey JR, …, & Foell J (2018a). Extending the construct network of trait disinhibition to the neuroimaging domain: Validation of a bridging scale for use in the European IMAGEN project. Assessment. Advance online publication. doi: 10.1177/1073191118759748 [DOI] [PMC free article] [PubMed] [Google Scholar]
Brislin SJ, Yancey JR, Perkins ER, Palumbo IM, Drislane LE,…, & Patrick CJ (2018b). Callousness and affective face processing in adults: Behavioral and brain-potential indicators. Personality Disorders: Theory, Research, and Treatment, 9, 122–132. [DOI] [PMC free article] [PubMed] [Google Scholar]
Burwell SJ, Malone SM, & Iacono WG (2016). One-year developmental stability and covariance among oddball, novelty, go/no-go, and flanker event-related potentials in adolescence: A monozygotic twin study. Psychophysiology, 53, 991–1007. [DOI] [PMC free article] [PubMed] [Google Scholar]
Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, & Munafo MR (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376. [DOI] [PubMed] [Google Scholar]
Carver CS, Johnson SL, Joormann J, Kim Y, & Nam JY (2011). Serotonin transporter polymorphism interacts with childhood adversity to predict aspects of impulsivity. Psychological Science, 22, 589–595. [DOI] [PubMed] [Google Scholar]
Castellanos-Ryan N, Struve M, Whelan R, Banaschewski T, Barker GJ, Bokde AL, …, & the IMAGEN Consortium (2014). Neural and cognitive correlates of the common and specific variance across externalizing problems in young adolescence. American Journal of Psychiatry, 171(12), 1310–1319. [DOI] [PubMed] [Google Scholar]
Coles MGH, & Rugg MD (1995). Event-related potentials: An introduction In Rugg MD & Coles MGH (Eds.), Electrophysiology of mind: Event-related potentials and cognition (pp. 1–26). New York: Oxford University Press. [Google Scholar]
Cronbach LJ, & Meehl PE (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302. [DOI] [PubMed] [Google Scholar]
Depue RA, & Iacono WG (1989). Neurobehavioral aspects of affective disorders. Annual Review of Psychology, 40, 457–492. [DOI] [PubMed] [Google Scholar]
Doiron D, Burton P, Marcon Y, Gaye A, Wolffenbuttel BHR,…, & Fortier I (2013). Data harmonization and federated analysis of population-based studies: The BioSHaRE project. Emerging Themes in Epidemiology, 10: 12. doi: 10.1186/1742-7622-10-12 [DOI] [PMC free article] [PubMed] [Google Scholar]
Drislane LE, Brislin SJ, Kendler KS, Andershed H, Larsson H, & Patrick CJ (2015). A triarchic model analysis of the Youth Psychopathic Traits Inventory. Journal of Personality Disorders, 29(1), 15–41. [DOI] [PubMed] [Google Scholar]
Drislane LE, Jones S, Brislin SJ, & Patrick CJ (2018). Interfacing five-factor model and triarchic conceptualizations of psychopathy. Psychological Assessment, 30, 834–840. [DOI] [PubMed] [Google Scholar]
Drislane LE, Vaidyanathan U, & Patrick CJ (2013). Reduced cortical call to arms differentiates psychopathy from antisocial personality disorder. Psychological Medicine, 43, 825–836. [DOI] [PubMed] [Google Scholar]
Engle RW (2002). Working memory capacity as executive attention. Current Directions in Psychological Science, 11(1), 19–23. [Google Scholar]
Eysenck HJ (1967). The biological basis of personality. Springfield, IL: Charles C. Thomas. [Google Scholar]
Fanselow MS (1994). Neural organization of the defensive behavior system responsible for fear. Psychonomic Bulletin and Review, 1, 429–438. [DOI] [PubMed] [Google Scholar]
Foell J, Brislin SJ, Palumbo I, Perkins ER, & Patrick CJ (2016). Brain mechanisms of externalizing proneness: Results from functional MRI studies in adolescents and adults. Psychophysiology, 53, S15. [Google Scholar]
Friedman HS, Kern ML, Hampson SE, & Duckworth AL (2014). A new life-span approach to conscientiousness and health: Combining the pieces of the causal puzzle. Developmental Psychology, 50(5), 1377–1389. [DOI] [PMC free article] [PubMed] [Google Scholar]
Gough HG (1960). Theory and measurement of socialization. Journal of Consulting Psychology, 24, 23–40. [Google Scholar]
Gratton G, & Fabiani M (2001). Shedding light on brain function: the event-related optical signal. Trends in Cognitive Sciences, 5(8), 357–363. [DOI] [PubMed] [Google Scholar]
Hajcak G, Meyer A, & Kotov R (2017). Psychometrics and the neuroscience of individual differences: Internal consistency limits between-subjects effects. Journal of Abnormal Psychology, 126(6), 823–834. [DOI] [PubMed] [Google Scholar]
Hajcak Proudfit G (2015). The reward positivity: From basic research on reward to a biomarker for depression. Psychophysiology, 52(4), 449–459. [DOI] [PubMed] [Google Scholar]
Hall JR, Drislane LE, Patrick CJ, Morano M, Lilienfeld SO, & Poythress NG (2014). Development and validation of Triarchic construct scales from the psychopathic personality inventory. Psychological Assessment, 26(2), 447. [DOI] [PMC free article] [PubMed] [Google Scholar]
Hicks BM, Bernat EM, Malone SM, Iacono WG, Patrick CJ, Krueger RF, & McGue M (2007). Genes mediate the association between P3 amplitude and externalizing disorders. Psychophysiology, 44, 98–105. [DOI] [PMC free article] [PubMed] [Google Scholar]
Hirth C, Obrig H, Villringer K, Thiel A, Bernarding J, Muhlnickel W,…, & Villringer A (1996). Non-invasive functional mapping of the human motor cortex using near-infrared spectroscopy. Neuroreport, 7(12), 1977–1981. [DOI] [PubMed] [Google Scholar]
Iacono WG (1991). Psychophysiological assessment of psychopathology. Psychological Assessment, 3, 309–320. [Google Scholar]
Iacono WG, Carlson SR, Malone SM, & McGue M (2002). P3 event-related potential amplitude and risk for disinhibitory disorders in adolescent boys. Archives of General Psychiatry, 59, 750–757. [DOI] [PubMed] [Google Scholar]
Iacono WG, Carlson SR, Taylor J, Elkins IJ, & McGue M (1999). Behavioral disinhibition and the development of substance-use disorders: Findings from the Minnesota Twin Family Study. Development and Psychopathology, 11, 869–900. [DOI] [PubMed] [Google Scholar]
Ioannidis JPA (2011). Excess significance bias in the literature on brain volume abnormalities. Archives of General Psychiatry, 68, 773–780. [DOI] [PubMed] [Google Scholar]
Infantolino Z, Luking KR, Sauder CL, Curtin JJ, & Hajcak G (2018). Robust is not necessarily reliable: From within-subjects fMRI contrasts to between-subjects comparisons. NeuroImage, 173, 146–152. [DOI] [PMC free article] [PubMed] [Google Scholar]
Knutson B, Adams CA, Fong G, & Hommer D (2001). Anticipation of increasing monetary reward selectively recruits nucleus accumbens. Journal of Neuroscience, 21(16): RC159. [DOI] [PMC free article] [PubMed] [Google Scholar]
Knutson B, Westdorp A, Kaiser E, & Hommer D (2000). FMRI visualization of brain activity during a monetary incentive delay task. Neuroimage, 12(1), 20–27. [DOI] [PubMed] [Google Scholar]
Kotov R, Krueger RF, Watson D, Achenbach TM, Althoff RR,..., & Zimmerman M (2017). The Hierarchical Taxonomy Of Psychopathology (HiTOP): A dimensional alternative to traditional nosologies. Journal of Abnormal Psychology, 126, 454–477. [DOI] [PubMed] [Google Scholar]
Kozak MJ, & Cuthbert BN (2016). The NIMH research domain criteria initiative: Background, issues, and pragmatics. Psychophysiology, 53(3), 286–297. [DOI] [PubMed] [Google Scholar]
Kramer MD, Patrick CJ, Krueger RF, & Gasperi M (2012). Delineating physiological defensive reactivity in the domain of self-report: Phenotypic and etiologic structure of dispositional fear. Psychological Medicine, 42, 1305–1320. [DOI] [PubMed] [Google Scholar]
Krueger RF, Derringer J, Markon KE, Watson D, & Skodol AE (2012). Initial construction of a maladaptive personality trait model and inventory for DSM-5. Psychological Medicine, 42(9), 1879–1890. [DOI] [PMC free article] [PubMed] [Google Scholar]
Krueger RF, Hicks BM, Patrick CJ, Carlson SR, Iacono WG, & McGue M (2002). Etiologic connections among substance dependence, antisocial behavior and personality: Modeling the externalizing spectrum. Journal of Abnormal Psychology, 111, 411–424. [PubMed] [Google Scholar]
Krueger RF, Markon KE, Patrick CJ, Benning SD, & Kramer MD (2007). Linking antisocial behavior, substance use, and personality: An integrative quantitative model of the adult externalizing spectrum. Journal of Abnormal Psychology, 116(4), 645–666. [DOI] [PMC free article] [PubMed] [Google Scholar]
Krueger RF, & Tackett JL (2003). Personality and psychopathology: Working toward the bigger picture. Journal of Personality Disorders, 17(2), 109–128. [DOI] [PubMed] [Google Scholar]
Kwako LE, Momenan R, Litten RZ, Koob GF, & Goldman D (2016). Addictions neuroclinical assessment: a neuroscience-based framework for addictive disorders. Biological Psychiatry, 80(3), 179–189. [DOI] [PMC free article] [PubMed] [Google Scholar]
Lang P (1979). A bioinformational theory of emotional imagery. Psychophysiology, 16, 495–512. [DOI] [PubMed] [Google Scholar]
Lang PJ (1994). The motivational organization of emotion: Affect-reflex connections In Van Goozen S, Van de Poll NE, & Sergeant JA (Eds.), The emotions: Essays on emotion theory (pp. 61–93). Hillsdale, NJ: Erlbaum. [Google Scholar]
Lang PJ (1995). The emotion probe: Studies of motivation and attention. American Psychologist, 50, 372–385. [DOI] [PubMed] [Google Scholar]
Lichtenstein P, Tuvblad C, Larsson H, & Carlstrom E (2007). The Swedish Twin study of CHild and Adolescent Development: The TCHAD-study. Twin Research and Human Genetics, 10(1), 67–73. [DOI] [PubMed] [Google Scholar]
Lilienfeld SO, & Treadway MT (2016). Clashing Diagnostic Approaches: DSM-ICD versus RDoC. Annual Review of Clinical Psychology, 12, 435–463. [DOI] [PMC free article] [PubMed] [Google Scholar]
Luck SJ (2014). An introduction to the event-related potential technique (2nd ed.). Cambridge, MA: MIT Press. [Google Scholar]
Marsh AA, Finger EC, Mitchell DGV, Reid ME, Sims C, Kosson DS, … & Blair RJR (2008). Reduced amygdala response to fearful expressions in children and adolescents with callous-unemotional traits and disruptive behavior disorders. American Journal of Psychiatry, 165, 712–720. [DOI] [PubMed] [Google Scholar]
McCrae RR, & John OP (1992). An introduction to the five‐factor model and its applications. Journal of Personality, 60(2), 175–215. [DOI] [PubMed] [Google Scholar]
Meyer A, Bress J, & Hajcak Proudfit G (2014). Psychometric properties of the error-related negativity in children and adolescents. Psychophysiology, 51(7), 602–610. [DOI] [PubMed] [Google Scholar]
Miller EK, & Cohen JD (2001). An integrative theory of prefrontal cortex function. Annual Review of Neuroscience, 24, 167–202. [DOI] [PubMed] [Google Scholar]
Miyake A, Friedman NP, Emerson MJ, Witzki AH, Howerter A, & Wager T (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41, 49–100. [DOI] [PubMed] [Google Scholar]
Morgan AB, & Lilienfeld SO (2000). A meta-analytic review of the relation between antisocial behavior and neuropsychological measures of executive function. Clinical Psychology Review, 20, 113–136. [DOI] [PubMed] [Google Scholar]
Moser JS, Durbin CE, Patrick CJ, & Schmidt NB (2015). Combining neural and behavioral indicators in the assessment of internalizing psychopathology in children and adolescents. Journal of Clinical Child and Adolescent Psychology, 44, 329–340. [DOI] [PubMed] [Google Scholar]
National Research Council (2015). Measuring human capabilities: An agenda for basic research on the assessment of individual and group performance potential for military accession. Committee on Measuring Human Capabilities, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy of Sciences. [Google Scholar]
Nelson LD, Patrick CJ, & Bernat EM (2011). Operationalizing proneness to externalizing psychopathology as a multivariate psychophysiological phenotype. Psychophysiology, 48, 64–72. [DOI] [PMC free article] [PubMed] [Google Scholar]
Papanicolaou AC (2009). Clinical magnetoencephalography and magnetic source imaging. New York: Cambridge University Press. [Google Scholar]
Patrick CJ, & Bernat EM (2009). From markers to mechanisms: Using psychophysiological measures to elucidate basic processes underlying aggressive externalizing behavior In: Hodgins S, Viding E, & Plodowski A (Eds.), Persistent violent offenders: Neuroscience and rehabilitation (pp. 223–250). London: Oxford University Press. [Google Scholar]
Patrick CJ, Bernat E, Malone S, Iacono WG, Krueger RF, & McGue MK (2006). P300 amplitude as an indicator of externalizing in adolescent males. Psychophysiology, 43, 84–92. [DOI] [PMC free article] [PubMed] [Google Scholar]
Patrick CJ, & Drislane LE (2015). Triarchic model of psychopathy: Origins, operationalizations, and observed linkages with personality and general psychopathology. Journal of Personality, 83, 627–643. [DOI] [PubMed] [Google Scholar]
Patrick CJ, Durbin CE, & Moser JS (2012). Conceptualizing proneness to antisocial deviance in neurobehavioral terms. Development and Psychopathology, 24, 1047–1071. [DOI] [PubMed] [Google Scholar]
Patrick CJ, Foell J, Venables NC, & Worthy DA (2016). Substance use disorders as externalizing outcomes. In: Beauchaine TP & Hinshaw SP (Eds.), Oxford handbook of externalizing spectrum disorders (pp. 38–60). New York: Oxford University Press. [Google Scholar]
Patrick CJ, & Hajcak G (2016). RDoC: Translating promise into progress. Psychophysiology, 53, 415–424. [DOI] [PubMed] [Google Scholar]
Patrick CJ, Kramer MD, Krueger RF, & Markon KE (2013a). Optimizing efficiency of psychopathology assessment through quantitative modeling: Development of a brief form of the Externalizing Spectrum Inventory. Psychological Assessment, 25, 1332–1348. [DOI] [PMC free article] [PubMed] [Google Scholar]
Patrick CJ, Venables NC, Yancey JR, Hicks BM, Nelson LD, & Kramer MD (2013b). A construct-network approach to bridging diagnostic and physiological domains: Application to assessment of externalizing psychopathology. Journal of Abnormal Psychology, 122, 902–916. [DOI] [PMC free article] [PubMed] [Google Scholar]
Perkins ER, Yancey JR, Drislane LD, Venables NC, Balsis S, & Patrick CJ (2017). Methodological issues in the use of individual brain meaures to index trait liabilities: The example of noise-probe P3. International Journal of Psychophysiology, 111, 245–255. [DOI] [PMC free article] [PubMed] [Google Scholar]
Pihl RO, Peterson J, & Finn P (1990). Inherited predisposition to alcoholism: Characteristics of sons of male alcoholics. Journal of Abnormal Psychology, 99(3), 291–301. [DOI] [PubMed] [Google Scholar]
Porjesz B, Begleiter H, & Garozzo R (1980). Visual evoked potential correlates of information processing deficits in chronic alcoholics In Begleiter H (Ed.), Biological effects of alcohol (pp. 603–623). New York: Plenum Press. [DOI] [PubMed] [Google Scholar]
Ramey T (2017, April 21). NIDA phenotyping battery [Webinar]. In Clinical Trials Network Dissemination Library: https://ctndisseminationlibrary.wordpress.com/2017/04/14/ctn-webinar-nida-phenotyping-battery-april-21-1-2pm-edt/ [Google Scholar]
Ray WJ, & Cole HW (1985). EEG alpha activity reflects attentional demands, and beta activity reflects emotional and cognitive processes. Science, 228, 750–752. [DOI] [PubMed] [Google Scholar]
Samuel DB, & Widiger TA (2008). A meta-analytic review of the relationships between the five-factor model and DSM-IV-TR personality disorders: A facet level analysis. Clinical Psychology Review, 28(8), 1326–1342. [DOI] [PMC free article] [PubMed] [Google Scholar]
Schall JD, Palmeri TJ, & Logan GD (2017). Models of inhibitory control. Philosophical Transactions of the Royal Society B: Biological Sciences, 372(1718): 20160193. [DOI] [PMC free article] [PubMed] [Google Scholar]
Schumann G, Loth E, Banaschewski T, Barbot A, Barker G, Büchel C, ... & the IMAGEN Consortium. (2010). The IMAGEN study: Reinforcement-related behaviour in normal brain function and psychopathology. Molecular Psychiatry, 15(12), 1128–1139. [DOI] [PubMed] [Google Scholar]
Sellbom M, Drislane LE, Johnson AK, Goodwin BE, Phillips TR, & Patrick CJ (2016). Development and validation of MMPI-2-RF Scales for indexing triarchic psychopathy constructs. Assessment, 23(5), 527–543. [DOI] [PubMed] [Google Scholar]
Silva PA (1990). The Dunedin Multidisciplinary Health and Development Study: A fifteen year longitudinal study. Paediatric and Perinatal Epidemiology, 4, 96–127. [DOI] [PubMed] [Google Scholar]
Tellegen A (1991). Personality traits: Issues of definition, evidence, and assessment In Cicchetti D & Grove WM (Eds.), Thinking clearly about psychology: Essays in honor of Paul E. Meehl, Vol. 1. Matters of public interest; Vol. 2. Personality and psychopathology (pp. 10–35). Minneapolis, MN, US: University of Minnesota Press. [Google Scholar]
Tellegen A, & Waller N (2008). Exploring personality through test construction: Development of the Multidimensional Personality Questionnaire In Boyle GJ, Matthews G, & Saklofske DH (Eds.), Handbook of personality theory and testing: Personality measurement and assessment (Vol. 2, pp. 261–292). Thousand Oaks: Sage. [Google Scholar]
Thigpen NN, Kappenman ES, & Keil A (2017). Assessing the internal consistency of the event-related potential: An example analysis. Psychophysiology, 54(1), 123–138. [DOI] [PMC free article] [PubMed] [Google Scholar]
Vaidyanathan U, Patrick CJ, & Cuthbert B (2009). Linking dimensional models of internalizing psychopathology to neurobiological systems: Affect-modulated startle as an indicator of fear and distress disorders and affiliated traits. Psychological Bulletin, 135, 909–942. [DOI] [PMC free article] [PubMed] [Google Scholar]
Valk PE, Bailey DL, Townsend DW, & Maisey MN (Eds.) (2004). Positron emission tomography: Basic science and clinical practice. New York: Springer-Verlag. [Google Scholar]
Van Essen DC, Smith SM, Barch DM, Behrens TEJ, Yacoub E, Ugurbil K & the WU-Minn HCP Consortium. (2013). The WU-UMinn Human Connectome Project: An overview. Neuroimage, 80, 62–79. [DOI] [PMC free article] [PubMed] [Google Scholar]
Venables NC, Foell J, Yancey JR, Kane MJ, Engle RW, & Patrick CJ (2018a). Quantifying inhibitory control as externalizing proneness: A cross-domain model. Clinical Psychological Science, 6, 561–580. [Google Scholar]
Venables NC, Yancey J, Kramer M, Hicks BM, Krueger R, Iacono WG, Joiner T, & Patrick CJ (2018b). Psychoneurometric assessment of dispositional liabilities for suicidal behavior: Phenotypic and etiological associations. Psychological Medicine, 48, 463–472. [DOI] [PubMed] [Google Scholar]
Venables NC, Yancey JR, Kramer MD, Hicks BM, Nelson LD, Strickland CM, Krueger RF, Iacono WG, & Patrick CJ (2017). Evidence of a prominent genetic basis for associations between psychoneurometric traits and common mental disorders. International Journal of Psychophysiology, 115, 4–12. [DOI] [PMC free article] [PubMed] [Google Scholar]
Vetter NC, Steding J, Jurk S, Ripke S, Mennigen E, & Smolka MN (2017). Reliability in adolescent fMRI within two years – a comparison of three tasks. Scientific Reports, 7, 2287. [DOI] [PMC free article] [PubMed] [Google Scholar]
Volkow ND, Koob GF, Croyle RT, Bianchi DW, Gordon JA, Koroshetz WJ,…, & Weiss SRB (2018). The conception of the ABCD study: From substance use to a broad NIH collaboration. Developmental Cognitive Neuroscience, 32, 4–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
Watson D, O’Hara MW, Simms LJ, Kotov R, Chmielewski M, McDade-Montez EA, & Stuart S (2007). Development and validation of the Inventory of Depression and Anxiety Symptoms (IDAS). Psychological Assessment, 19, 253. [DOI] [PubMed] [Google Scholar]
Whiteside SP, & Lynam DR (2001). The five factor model and impulsivity: Using a structural model of personality to understand impulsivity. Personality and Individual Differences, 30, 669–689. [Google Scholar]
Widiger TA (2013). Personality and psychopathology. World Psychiatry, 10(2), 103–106. [DOI] [PMC free article] [PubMed] [Google Scholar]
Yancey JR, Vaidyanathan U, & Patrick CJ (2015). Aversive startle potentiation and fear pathology: Mediating role of threat sensitivity and moderating impact of depression. International Journal of Psychophysiology, 98, 262–269. [DOI] [PMC free article] [PubMed] [Google Scholar]
Yancey J, Venables N, Hicks B, & Patrick CJ (2013). Evidence for a heritable brain basis to deviance-promoting deficits in self-control. Journal of Criminal Justice, 41, 309–317. [DOI] [PMC free article] [PubMed] [Google Scholar]
Yancey JR, Venables NC, & Patrick CJ (2016). Psychoneurometric operationalization of threat sensitivity: Relations with clinical symptom and physiological response criteria. Psychophysiology, 53(3), 393–405. [DOI] [PMC free article] [PubMed] [Google Scholar]
Young SE, Friedman NP, Miyake A, Willcutt EG, Corley RP, Haberstick BC, & Hewitt JK (2009). Behavioral disinhibition: liability for externalizing spectrum disorders and its genetic and environmental relation to response inhibition across adolescence. Journal of Abnormal Psychology, 118, 117–130. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R1] Achenbach TM, & Edelbrock CS (1978). The classification of child psychopathology: A review and analysis of empirical efforts. Psychological Bulletin, 85, 1275–1301. [PubMed] [Google Scholar]

[R2] Alexander LM, Escalera J, Ai L, Andreotti C, Febre K, Mangone A, ..., & Milham MP (2017). An open resource for transdiagnostic research in pediatric mental health and learning disorders. Scientific Data, 4:170181. doi: 10.1038/sdata.2017.181. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R3] Allport GW (1937). Personality: A psychological interpretation. New York: Holt. [Google Scholar]

[R4] American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (DSM-5). Washington, DC: Author. [Google Scholar]

[R5] American Psychiatric Association. (2017, August 1). Submit proposals for making changes to DSM-5. See: https://www.psychiatry.org/psychiatrists/practice/dsm/submit-proposals

[R6] Balsis S, Benge J, Lowe D, Geraci L, & Doody RS (2015). How do scores on the ADAS-Cog, MMSE, and CDR-SOB correspond? Clinical Neuropsychologist, 29, 1002–1009. [DOI] [PubMed] [Google Scholar]

[R7] Balsis S, Choudhury BS, Geraci L, & Patrick CJ (2018). Alzheimer’s disease assessment: A review and illustrations focusing on item response theory techniques. Assessment, 25, 360–373. [DOI] [PubMed] [Google Scholar]

[R8] Bari A, & Robbins TW (2013). Inhibition and impulsivity: Behavioral and neural basis of response control. Progress in Neurobiology, 108, 44–79. [DOI] [PubMed] [Google Scholar]

[R9] Bauer LO, O’Connor S, & Hesselbrock VM (1994). Frontal P300 decrements in antisocial personality disorder. Alcoholism: Clinical and Experimental Research, 18, 1300–1305. [DOI] [PubMed] [Google Scholar]

[R10] Bauer RM, Iverson GL, Cernich AN, Binder LM, Ruff RM, & Naugle RI (2012). Computerized neuropsychological assessment devices: Joint position paper of the American Academy of Clinical Neuropsychology and the National Academy of Neuropsychology. Archives of Clinical Neuropsychology, 27, 362–373. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R11] Begleiter H, Porjesz B, Bihari B, & Kissin B (1984). Event-related brain potentials in boys at risk for alcoholism. Science, 225, 1493–1496. [DOI] [PubMed] [Google Scholar]

[R12] Begleiter H, Reich T, Hesselbrock V, Poriesz B, Li TK, Schuckit MA, Edenberg HJ, & Rice JP (1995). The Collaborative Study on the Genetics of Alcoholism. Alcohol Health & Research World, 19(3), 228–236. [PMC free article] [PubMed] [Google Scholar]

[R13] Berg JM, Latzman RD, Bliwise NG, & Lilienfeld SO (2015). Parsing the heterogeneity of impulsivity: A metaanalytic review of the behavioral implications of the UPPS for psychopathology. Psychological Assessment, 27, 1129–1146. [DOI] [PubMed] [Google Scholar]

[R14] Berman SM, Whipple SC, Fitch RJ, & Noble EP (1993). P3 in young boys as a predictor of adolescent substance use. Alcohol, 10, 69–76. [DOI] [PubMed] [Google Scholar]

[R15] Biggins CA, MacKay S, Clark W, & Fein G (1997). Event-related potential evidence for frontal cortex effects of chronic cocaine dependence. Biological Psychiatry, 42, 472–485. [DOI] [PubMed] [Google Scholar]

[R16] Bogg T, & Finn PR (2010). A self‐regulatory model of behavioral disinhibition in late adolescence: Integrating personality traits, externalizing psychopathology, and cognitive capacity. Journal of Personality, 78(2), 441–470. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R17] Borsboom D, Mellenbergh GJ, & van Heerden J (2004). The concept of validity. Psychological Review, 111(4), 1061–1071. [DOI] [PubMed] [Google Scholar]

[R18] Branchey MH, Buydens-Branchey L, & Horvath TB (1993). Event-related potentials in substance-abusing individuals after longterm abstinence. American Journal of Addictions, 2, 141–148. [Google Scholar]

[R19] Brislin SJ, Drislane LE, Smith ST, Edens JF, & Patrick CJ (2015). Development and validation of triarchic psychopathy scales from the Multidimensional Personality Questionnaire. Psychological Assessment, 27(3), 838–851. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R20] Brislin SJ, Patrick CJ, Flor H, Nees F, Heinrich A, Drislane LE, Yancey JR, …, & Foell J (2018a). Extending the construct network of trait disinhibition to the neuroimaging domain: Validation of a bridging scale for use in the European IMAGEN project. Assessment. Advance online publication. doi: 10.1177/1073191118759748 [DOI] [PMC free article] [PubMed] [Google Scholar]

[R21] Brislin SJ, Yancey JR, Perkins ER, Palumbo IM, Drislane LE,…, & Patrick CJ (2018b). Callousness and affective face processing in adults: Behavioral and brain-potential indicators. Personality Disorders: Theory, Research, and Treatment, 9, 122–132. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R22] Burwell SJ, Malone SM, & Iacono WG (2016). One-year developmental stability and covariance among oddball, novelty, go/no-go, and flanker event-related potentials in adolescence: A monozygotic twin study. Psychophysiology, 53, 991–1007. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R23] Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, & Munafo MR (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376. [DOI] [PubMed] [Google Scholar]

[R24] Carver CS, Johnson SL, Joormann J, Kim Y, & Nam JY (2011). Serotonin transporter polymorphism interacts with childhood adversity to predict aspects of impulsivity. Psychological Science, 22, 589–595. [DOI] [PubMed] [Google Scholar]

[R25] Castellanos-Ryan N, Struve M, Whelan R, Banaschewski T, Barker GJ, Bokde AL, …, & the IMAGEN Consortium (2014). Neural and cognitive correlates of the common and specific variance across externalizing problems in young adolescence. American Journal of Psychiatry, 171(12), 1310–1319. [DOI] [PubMed] [Google Scholar]

[R26] Coles MGH, & Rugg MD (1995). Event-related potentials: An introduction In Rugg MD & Coles MGH (Eds.), Electrophysiology of mind: Event-related potentials and cognition (pp. 1–26). New York: Oxford University Press. [Google Scholar]

[R27] Cronbach LJ, & Meehl PE (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302. [DOI] [PubMed] [Google Scholar]

[R28] Depue RA, & Iacono WG (1989). Neurobehavioral aspects of affective disorders. Annual Review of Psychology, 40, 457–492. [DOI] [PubMed] [Google Scholar]

[R29] Doiron D, Burton P, Marcon Y, Gaye A, Wolffenbuttel BHR,…, & Fortier I (2013). Data harmonization and federated analysis of population-based studies: The BioSHaRE project. Emerging Themes in Epidemiology, 10: 12. doi: 10.1186/1742-7622-10-12 [DOI] [PMC free article] [PubMed] [Google Scholar]

[R30] Drislane LE, Brislin SJ, Kendler KS, Andershed H, Larsson H, & Patrick CJ (2015). A triarchic model analysis of the Youth Psychopathic Traits Inventory. Journal of Personality Disorders, 29(1), 15–41. [DOI] [PubMed] [Google Scholar]

[R31] Drislane LE, Jones S, Brislin SJ, & Patrick CJ (2018). Interfacing five-factor model and triarchic conceptualizations of psychopathy. Psychological Assessment, 30, 834–840. [DOI] [PubMed] [Google Scholar]

[R32] Drislane LE, Vaidyanathan U, & Patrick CJ (2013). Reduced cortical call to arms differentiates psychopathy from antisocial personality disorder. Psychological Medicine, 43, 825–836. [DOI] [PubMed] [Google Scholar]

[R33] Engle RW (2002). Working memory capacity as executive attention. Current Directions in Psychological Science, 11(1), 19–23. [Google Scholar]

[R34] Eysenck HJ (1967). The biological basis of personality. Springfield, IL: Charles C. Thomas. [Google Scholar]

[R35] Fanselow MS (1994). Neural organization of the defensive behavior system responsible for fear. Psychonomic Bulletin and Review, 1, 429–438. [DOI] [PubMed] [Google Scholar]

[R36] Foell J, Brislin SJ, Palumbo I, Perkins ER, & Patrick CJ (2016). Brain mechanisms of externalizing proneness: Results from functional MRI studies in adolescents and adults. Psychophysiology, 53, S15. [Google Scholar]

[R37] Friedman HS, Kern ML, Hampson SE, & Duckworth AL (2014). A new life-span approach to conscientiousness and health: Combining the pieces of the causal puzzle. Developmental Psychology, 50(5), 1377–1389. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R38] Gough HG (1960). Theory and measurement of socialization. Journal of Consulting Psychology, 24, 23–40. [Google Scholar]

[R39] Gratton G, & Fabiani M (2001). Shedding light on brain function: the event-related optical signal. Trends in Cognitive Sciences, 5(8), 357–363. [DOI] [PubMed] [Google Scholar]

[R40] Hajcak G, Meyer A, & Kotov R (2017). Psychometrics and the neuroscience of individual differences: Internal consistency limits between-subjects effects. Journal of Abnormal Psychology, 126(6), 823–834. [DOI] [PubMed] [Google Scholar]

[R41] Hajcak Proudfit G (2015). The reward positivity: From basic research on reward to a biomarker for depression. Psychophysiology, 52(4), 449–459. [DOI] [PubMed] [Google Scholar]

[R42] Hall JR, Drislane LE, Patrick CJ, Morano M, Lilienfeld SO, & Poythress NG (2014). Development and validation of Triarchic construct scales from the psychopathic personality inventory. Psychological Assessment, 26(2), 447. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R43] Hicks BM, Bernat EM, Malone SM, Iacono WG, Patrick CJ, Krueger RF, & McGue M (2007). Genes mediate the association between P3 amplitude and externalizing disorders. Psychophysiology, 44, 98–105. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R44] Hirth C, Obrig H, Villringer K, Thiel A, Bernarding J, Muhlnickel W,…, & Villringer A (1996). Non-invasive functional mapping of the human motor cortex using near-infrared spectroscopy. Neuroreport, 7(12), 1977–1981. [DOI] [PubMed] [Google Scholar]

[R45] Iacono WG (1991). Psychophysiological assessment of psychopathology. Psychological Assessment, 3, 309–320. [Google Scholar]

[R46] Iacono WG, Carlson SR, Malone SM, & McGue M (2002). P3 event-related potential amplitude and risk for disinhibitory disorders in adolescent boys. Archives of General Psychiatry, 59, 750–757. [DOI] [PubMed] [Google Scholar]

[R47] Iacono WG, Carlson SR, Taylor J, Elkins IJ, & McGue M (1999). Behavioral disinhibition and the development of substance-use disorders: Findings from the Minnesota Twin Family Study. Development and Psychopathology, 11, 869–900. [DOI] [PubMed] [Google Scholar]

[R48] Ioannidis JPA (2011). Excess significance bias in the literature on brain volume abnormalities. Archives of General Psychiatry, 68, 773–780. [DOI] [PubMed] [Google Scholar]

[R49] Infantolino Z, Luking KR, Sauder CL, Curtin JJ, & Hajcak G (2018). Robust is not necessarily reliable: From within-subjects fMRI contrasts to between-subjects comparisons. NeuroImage, 173, 146–152. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R50] Knutson B, Adams CA, Fong G, & Hommer D (2001). Anticipation of increasing monetary reward selectively recruits nucleus accumbens. Journal of Neuroscience, 21(16): RC159. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R51] Knutson B, Westdorp A, Kaiser E, & Hommer D (2000). FMRI visualization of brain activity during a monetary incentive delay task. Neuroimage, 12(1), 20–27. [DOI] [PubMed] [Google Scholar]

[R52] Kotov R, Krueger RF, Watson D, Achenbach TM, Althoff RR,..., & Zimmerman M (2017). The Hierarchical Taxonomy Of Psychopathology (HiTOP): A dimensional alternative to traditional nosologies. Journal of Abnormal Psychology, 126, 454–477. [DOI] [PubMed] [Google Scholar]

[R53] Kozak MJ, & Cuthbert BN (2016). The NIMH research domain criteria initiative: Background, issues, and pragmatics. Psychophysiology, 53(3), 286–297. [DOI] [PubMed] [Google Scholar]

[R54] Kramer MD, Patrick CJ, Krueger RF, & Gasperi M (2012). Delineating physiological defensive reactivity in the domain of self-report: Phenotypic and etiologic structure of dispositional fear. Psychological Medicine, 42, 1305–1320. [DOI] [PubMed] [Google Scholar]

[R55] Krueger RF, Derringer J, Markon KE, Watson D, & Skodol AE (2012). Initial construction of a maladaptive personality trait model and inventory for DSM-5. Psychological Medicine, 42(9), 1879–1890. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R56] Krueger RF, Hicks BM, Patrick CJ, Carlson SR, Iacono WG, & McGue M (2002). Etiologic connections among substance dependence, antisocial behavior and personality: Modeling the externalizing spectrum. Journal of Abnormal Psychology, 111, 411–424. [PubMed] [Google Scholar]

[R57] Krueger RF, Markon KE, Patrick CJ, Benning SD, & Kramer MD (2007). Linking antisocial behavior, substance use, and personality: An integrative quantitative model of the adult externalizing spectrum. Journal of Abnormal Psychology, 116(4), 645–666. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R58] Krueger RF, & Tackett JL (2003). Personality and psychopathology: Working toward the bigger picture. Journal of Personality Disorders, 17(2), 109–128. [DOI] [PubMed] [Google Scholar]

[R59] Kwako LE, Momenan R, Litten RZ, Koob GF, & Goldman D (2016). Addictions neuroclinical assessment: a neuroscience-based framework for addictive disorders. Biological Psychiatry, 80(3), 179–189. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R60] Lang P (1979). A bioinformational theory of emotional imagery. Psychophysiology, 16, 495–512. [DOI] [PubMed] [Google Scholar]

[R61] Lang PJ (1994). The motivational organization of emotion: Affect-reflex connections In Van Goozen S, Van de Poll NE, & Sergeant JA (Eds.), The emotions: Essays on emotion theory (pp. 61–93). Hillsdale, NJ: Erlbaum. [Google Scholar]

[R62] Lang PJ (1995). The emotion probe: Studies of motivation and attention. American Psychologist, 50, 372–385. [DOI] [PubMed] [Google Scholar]

[R63] Lichtenstein P, Tuvblad C, Larsson H, & Carlstrom E (2007). The Swedish Twin study of CHild and Adolescent Development: The TCHAD-study. Twin Research and Human Genetics, 10(1), 67–73. [DOI] [PubMed] [Google Scholar]

[R64] Lilienfeld SO, & Treadway MT (2016). Clashing Diagnostic Approaches: DSM-ICD versus RDoC. Annual Review of Clinical Psychology, 12, 435–463. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R65] Luck SJ (2014). An introduction to the event-related potential technique (2nd ed.). Cambridge, MA: MIT Press. [Google Scholar]

[R66] Marsh AA, Finger EC, Mitchell DGV, Reid ME, Sims C, Kosson DS, … & Blair RJR (2008). Reduced amygdala response to fearful expressions in children and adolescents with callous-unemotional traits and disruptive behavior disorders. American Journal of Psychiatry, 165, 712–720. [DOI] [PubMed] [Google Scholar]

[R67] McCrae RR, & John OP (1992). An introduction to the five‐factor model and its applications. Journal of Personality, 60(2), 175–215. [DOI] [PubMed] [Google Scholar]

[R68] Meyer A, Bress J, & Hajcak Proudfit G (2014). Psychometric properties of the error-related negativity in children and adolescents. Psychophysiology, 51(7), 602–610. [DOI] [PubMed] [Google Scholar]

[R69] Miller EK, & Cohen JD (2001). An integrative theory of prefrontal cortex function. Annual Review of Neuroscience, 24, 167–202. [DOI] [PubMed] [Google Scholar]

[R70] Miyake A, Friedman NP, Emerson MJ, Witzki AH, Howerter A, & Wager T (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41, 49–100. [DOI] [PubMed] [Google Scholar]

[R71] Morgan AB, & Lilienfeld SO (2000). A meta-analytic review of the relation between antisocial behavior and neuropsychological measures of executive function. Clinical Psychology Review, 20, 113–136. [DOI] [PubMed] [Google Scholar]

[R72] Moser JS, Durbin CE, Patrick CJ, & Schmidt NB (2015). Combining neural and behavioral indicators in the assessment of internalizing psychopathology in children and adolescents. Journal of Clinical Child and Adolescent Psychology, 44, 329–340. [DOI] [PubMed] [Google Scholar]

[R73] National Research Council (2015). Measuring human capabilities: An agenda for basic research on the assessment of individual and group performance potential for military accession. Committee on Measuring Human Capabilities, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy of Sciences. [Google Scholar]

[R74] Nelson LD, Patrick CJ, & Bernat EM (2011). Operationalizing proneness to externalizing psychopathology as a multivariate psychophysiological phenotype. Psychophysiology, 48, 64–72. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R75] Papanicolaou AC (2009). Clinical magnetoencephalography and magnetic source imaging. New York: Cambridge University Press. [Google Scholar]

[R76] Patrick CJ, & Bernat EM (2009). From markers to mechanisms: Using psychophysiological measures to elucidate basic processes underlying aggressive externalizing behavior In: Hodgins S, Viding E, & Plodowski A (Eds.), Persistent violent offenders: Neuroscience and rehabilitation (pp. 223–250). London: Oxford University Press. [Google Scholar]

[R77] Patrick CJ, Bernat E, Malone S, Iacono WG, Krueger RF, & McGue MK (2006). P300 amplitude as an indicator of externalizing in adolescent males. Psychophysiology, 43, 84–92. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R78] Patrick CJ, & Drislane LE (2015). Triarchic model of psychopathy: Origins, operationalizations, and observed linkages with personality and general psychopathology. Journal of Personality, 83, 627–643. [DOI] [PubMed] [Google Scholar]

[R79] Patrick CJ, Durbin CE, & Moser JS (2012). Conceptualizing proneness to antisocial deviance in neurobehavioral terms. Development and Psychopathology, 24, 1047–1071. [DOI] [PubMed] [Google Scholar]

[R80] Patrick CJ, Foell J, Venables NC, & Worthy DA (2016). Substance use disorders as externalizing outcomes. In: Beauchaine TP & Hinshaw SP (Eds.), Oxford handbook of externalizing spectrum disorders (pp. 38–60). New York: Oxford University Press. [Google Scholar]

[R81] Patrick CJ, & Hajcak G (2016). RDoC: Translating promise into progress. Psychophysiology, 53, 415–424. [DOI] [PubMed] [Google Scholar]

[R82] Patrick CJ, Kramer MD, Krueger RF, & Markon KE (2013a). Optimizing efficiency of psychopathology assessment through quantitative modeling: Development of a brief form of the Externalizing Spectrum Inventory. Psychological Assessment, 25, 1332–1348. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R83] Patrick CJ, Venables NC, Yancey JR, Hicks BM, Nelson LD, & Kramer MD (2013b). A construct-network approach to bridging diagnostic and physiological domains: Application to assessment of externalizing psychopathology. Journal of Abnormal Psychology, 122, 902–916. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R84] Perkins ER, Yancey JR, Drislane LD, Venables NC, Balsis S, & Patrick CJ (2017). Methodological issues in the use of individual brain meaures to index trait liabilities: The example of noise-probe P3. International Journal of Psychophysiology, 111, 245–255. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R85] Pihl RO, Peterson J, & Finn P (1990). Inherited predisposition to alcoholism: Characteristics of sons of male alcoholics. Journal of Abnormal Psychology, 99(3), 291–301. [DOI] [PubMed] [Google Scholar]

[R86] Porjesz B, Begleiter H, & Garozzo R (1980). Visual evoked potential correlates of information processing deficits in chronic alcoholics In Begleiter H (Ed.), Biological effects of alcohol (pp. 603–623). New York: Plenum Press. [DOI] [PubMed] [Google Scholar]

[R87] Ramey T (2017, April 21). NIDA phenotyping battery [Webinar]. In Clinical Trials Network Dissemination Library: https://ctndisseminationlibrary.wordpress.com/2017/04/14/ctn-webinar-nida-phenotyping-battery-april-21-1-2pm-edt/ [Google Scholar]

[R88] Ray WJ, & Cole HW (1985). EEG alpha activity reflects attentional demands, and beta activity reflects emotional and cognitive processes. Science, 228, 750–752. [DOI] [PubMed] [Google Scholar]

[R89] Samuel DB, & Widiger TA (2008). A meta-analytic review of the relationships between the five-factor model and DSM-IV-TR personality disorders: A facet level analysis. Clinical Psychology Review, 28(8), 1326–1342. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R90] Schall JD, Palmeri TJ, & Logan GD (2017). Models of inhibitory control. Philosophical Transactions of the Royal Society B: Biological Sciences, 372(1718): 20160193. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R91] Schumann G, Loth E, Banaschewski T, Barbot A, Barker G, Büchel C, ... & the IMAGEN Consortium. (2010). The IMAGEN study: Reinforcement-related behaviour in normal brain function and psychopathology. Molecular Psychiatry, 15(12), 1128–1139. [DOI] [PubMed] [Google Scholar]

[R92] Sellbom M, Drislane LE, Johnson AK, Goodwin BE, Phillips TR, & Patrick CJ (2016). Development and validation of MMPI-2-RF Scales for indexing triarchic psychopathy constructs. Assessment, 23(5), 527–543. [DOI] [PubMed] [Google Scholar]

[R93] Silva PA (1990). The Dunedin Multidisciplinary Health and Development Study: A fifteen year longitudinal study. Paediatric and Perinatal Epidemiology, 4, 96–127. [DOI] [PubMed] [Google Scholar]

[R94] Tellegen A (1991). Personality traits: Issues of definition, evidence, and assessment In Cicchetti D & Grove WM (Eds.), Thinking clearly about psychology: Essays in honor of Paul E. Meehl, Vol. 1. Matters of public interest; Vol. 2. Personality and psychopathology (pp. 10–35). Minneapolis, MN, US: University of Minnesota Press. [Google Scholar]

[R95] Tellegen A, & Waller N (2008). Exploring personality through test construction: Development of the Multidimensional Personality Questionnaire In Boyle GJ, Matthews G, & Saklofske DH (Eds.), Handbook of personality theory and testing: Personality measurement and assessment (Vol. 2, pp. 261–292). Thousand Oaks: Sage. [Google Scholar]

[R96] Thigpen NN, Kappenman ES, & Keil A (2017). Assessing the internal consistency of the event-related potential: An example analysis. Psychophysiology, 54(1), 123–138. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R97] Vaidyanathan U, Patrick CJ, & Cuthbert B (2009). Linking dimensional models of internalizing psychopathology to neurobiological systems: Affect-modulated startle as an indicator of fear and distress disorders and affiliated traits. Psychological Bulletin, 135, 909–942. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R98] Valk PE, Bailey DL, Townsend DW, & Maisey MN (Eds.) (2004). Positron emission tomography: Basic science and clinical practice. New York: Springer-Verlag. [Google Scholar]

[R99] Van Essen DC, Smith SM, Barch DM, Behrens TEJ, Yacoub E, Ugurbil K & the WU-Minn HCP Consortium. (2013). The WU-UMinn Human Connectome Project: An overview. Neuroimage, 80, 62–79. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R100] Venables NC, Foell J, Yancey JR, Kane MJ, Engle RW, & Patrick CJ (2018a). Quantifying inhibitory control as externalizing proneness: A cross-domain model. Clinical Psychological Science, 6, 561–580. [Google Scholar]

[R101] Venables NC, Yancey J, Kramer M, Hicks BM, Krueger R, Iacono WG, Joiner T, & Patrick CJ (2018b). Psychoneurometric assessment of dispositional liabilities for suicidal behavior: Phenotypic and etiological associations. Psychological Medicine, 48, 463–472. [DOI] [PubMed] [Google Scholar]

[R102] Venables NC, Yancey JR, Kramer MD, Hicks BM, Nelson LD, Strickland CM, Krueger RF, Iacono WG, & Patrick CJ (2017). Evidence of a prominent genetic basis for associations between psychoneurometric traits and common mental disorders. International Journal of Psychophysiology, 115, 4–12. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R103] Vetter NC, Steding J, Jurk S, Ripke S, Mennigen E, & Smolka MN (2017). Reliability in adolescent fMRI within two years – a comparison of three tasks. Scientific Reports, 7, 2287. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R104] Volkow ND, Koob GF, Croyle RT, Bianchi DW, Gordon JA, Koroshetz WJ,…, & Weiss SRB (2018). The conception of the ABCD study: From substance use to a broad NIH collaboration. Developmental Cognitive Neuroscience, 32, 4–7. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R105] Watson D, O’Hara MW, Simms LJ, Kotov R, Chmielewski M, McDade-Montez EA, & Stuart S (2007). Development and validation of the Inventory of Depression and Anxiety Symptoms (IDAS). Psychological Assessment, 19, 253. [DOI] [PubMed] [Google Scholar]

[R106] Whiteside SP, & Lynam DR (2001). The five factor model and impulsivity: Using a structural model of personality to understand impulsivity. Personality and Individual Differences, 30, 669–689. [Google Scholar]

[R107] Widiger TA (2013). Personality and psychopathology. World Psychiatry, 10(2), 103–106. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R108] Yancey JR, Vaidyanathan U, & Patrick CJ (2015). Aversive startle potentiation and fear pathology: Mediating role of threat sensitivity and moderating impact of depression. International Journal of Psychophysiology, 98, 262–269. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R109] Yancey J, Venables N, Hicks B, & Patrick CJ (2013). Evidence for a heritable brain basis to deviance-promoting deficits in self-control. Journal of Criminal Justice, 41, 309–317. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R110] Yancey JR, Venables NC, & Patrick CJ (2016). Psychoneurometric operationalization of threat sensitivity: Relations with clinical symptom and physiological response criteria. Psychophysiology, 53(3), 393–405. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R111] Young SE, Friedman NP, Miyake A, Willcutt EG, Corley RP, Haberstick BC, & Hewitt JK (2009). Behavioral disinhibition: liability for externalizing spectrum disorders and its genetic and environmental relation to response inhibition across adolescence. Journal of Abnormal Psychology, 118, 117–130. [DOI] [PMC free article] [PubMed] [Google Scholar]

PERMALINK

Incorporating Neurophysiological Measures into Clinical Assessments: Fundamental Challenges and a Strategy for Addressing Them

Christopher J Patrick

William G Iacono

Noah C Venables

Abstract

Methodological Issues in using Neurophysiological Measures in Psychological Assessments

Figure 1.

Average Task Effects versus Inter-Individual Variation in Task Effects

Score Reliability of Individual Neurophysiological Indicators

Criterion Referent for the Neurophysiological Indicator

Method Variance

Specificity of Neurophysiological Variables as Indicators of Target Attributes

Score Aggregation

Indicators of Liability for versus Expression or Consequences of Mental Illness

A Measurement-Based Approach to Integrating Neural Measures into Applied Assessments

Psychoneurometric research strategy

Figure 2.

Table 1.

Empirical Illustration: Multi-Method Model of Inhibitory Control Capacity

Figure 3.

From Bench to Bedside: Addressing Challenges to Development and Clinical Implementation of Multi-Method Assessments

Methodological Challenges

Logistical Challenges

Challenges to Use in Clinical Practice

Table 2.

Conclusion

Public Significance Statement.

Acknowledgments

Footnotes

Contributor Information

References

ACTIONS

PERMALINK

RESOURCES

Cite

Add to Collections

PERMALINK

Incorporating Neurophysiological Measures into Clinical Assessments: Fundamental Challenges and a Strategy for Addressing Them

Christopher J Patrick

William G Iacono

Noah C Venables

Abstract

Methodological Issues in using Neurophysiological Measures in Psychological Assessments

Figure 1.

Average Task Effects versus Inter-Individual Variation in Task Effects

Score Reliability of Individual Neurophysiological Indicators

Criterion Referent for the Neurophysiological Indicator

Method Variance

Specificity of Neurophysiological Variables as Indicators of Target Attributes

Score Aggregation

Indicators of Liability for versus Expression or Consequences of Mental Illness

A Measurement-Based Approach to Integrating Neural Measures into Applied Assessments

Psychoneurometric research strategy

Figure 2.

Table 1.

Empirical Illustration: Multi-Method Model of Inhibitory Control Capacity

Figure 3.

From Bench to Bedside: Addressing Challenges to Development and Clinical Implementation of Multi-Method Assessments

Methodological Challenges

Logistical Challenges

Challenges to Use in Clinical Practice

Table 2.

Conclusion

Public Significance Statement.

Acknowledgments

Footnotes

Contributor Information

References

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases