Modeling Developmental Change: Contemporary Approaches to Key Methodological Challenges in Developmental Neuroimaging

Developmental cognitive neuroscience is a truly interdisciplinary field of research that has the potential to answer critical questions about neural plasticity and neural substrates of learning and behavior across cognitive, affective, and social domains of functioning. It therefore has the potential to not only help us understand trajectories and mechanisms of typical development, but also translate this knowledge to the prevention and treatment of emerging psychopathology and health-risking behaviors. However, to reach these goals our field must be able to model how these processes change within individuals across time. Given how central this methodological issue is to our endeavours, it is surprising that there has been relatively little attention paid to integrating neuroscientific methods with cutting edge statistical techniques for modelling longitudinal change, nor have there been published methodological guidelines on many relevant topics. The current special issue sets out to begin to address this lacuna.

Many techniques have been employed to examine the brain across development, including (but not limited to) familiar modalities like structural, functional, and diffusion magnetic resonance imaging (s/f/d MRI), as well as less widespread ones like functional near infrared spectroscopy (fNIRS) or magnetic resonance elastography (MRE). Each of these methods addressed in the special issue have unique strengths and limitations that shape recommendations for data acquisition and analysis, and provide different information about normative and atypical trajectories of brain development. Importantly, researchers are increasingly relying on longitudinal data sets to investigate change within individuals. However, longitudinal studies require special consideration in design, as well as data acquisition, processing, analysis, and interpretation. Despite increasing acknowledgement of methodological issues across modalities and study designs in the cognitive neurosciences, there are relatively limited guidelines available to provide best practices, particularly with developmental populations. This lack of consensus could be contributing to inconsistencies in the literature. For example, recent studies have found that differences in sample composition, quality control procedures, and data analytic approaches affect observed trajectories of brain development (Ducharme et al., 2016, LeWinn et al., 2017). Further, it is unclear how these factors affect associations between brain development and cognition or other behavior.

This special issue of Developmental Cognitive Neuroscience, “Methodological Challenges in Developmental Neuroimaging: Contemporary Approaches and Solutions,” presents papers that make headway in understanding and overcoming these methodological concerns, as well as shape strategic research priorities, and suggest guidelines that may serve as best practices for study design, data acquisition, analysis, and dissemination of findings.

1. Individual Differences, versus Developmental Processes, versus Phasic Responses

One of the enduring challenges in any attempt to characterize longitudinal change within individuals is to understand exactly what kind of change one is observing. This is especially complex with respect to developmental studies. There are a wide variety of processes that will influence not only the measurements taken of an individual at a specific time, but also the particular processes that might be responsible for an observed pattern of change across longitudinal data collection. For example, there are individual differences in the intercept, slope (and shape), and final outcome of developmental growth processes. Indeed, one of the most difficult problems is that it is often impossible to fully characterize a growth process until it is complete. For example, if one is observing two 16 year-olds who are both six feet tall, it is not clear whether this height represents the final outcome of their adolescent growth spurt, or whether it is an intermediate point on that trajectory. Only further longitudinal observations can resolve the issue. Another complexity is that there may be processes that contribute to change that are not developmental per se, such as when a person experiences an environmental exposure such as trauma, or an episode of mental or neurological illness. These processes may also contribute to change across time and it can be extremely difficult within some designs to disentangle these effects. This issue is especially complex in the study of high risk samples where developmental issues are often confounded with stage of illness issues during the emergence of disorders.

An additional notable complexity here derives from the fact that we often probe phasic response processes, such as those associated with fMRI activation tasks, in our studies. This requires us to understand how the developmental dynamics of baseline brain structure and function might determine differences in these phasic responses across time. Indeed, one manuscript in the special issue directly tackles the relationship between tonic and phasic aspects of brain development by characterizing how the brain departs from its baseline functional architecture during task-induced functional connectivity modulations (Chauvin et al., this issue). The authors propose a novel measure called “task potency,” which allows direct comparison between tasks by operationalizing sensitivity to task manipulations. Chauvin et al. show that their potency measure can demonstrate maturational changes in task-dependent functional co-activation over and above maturation in baseline connectivity.

Ultimately, we will not be able to make significant progress on these issues without strong methodology, and here a number of challenges are notable. For example, Herting and colleagues (this issue) take on the fundamental and yet tricky issue of test-retest reliability of fMRI tasks. This is widely and increasingly recognized as an existential issue for the field, and one that is especially critical for developmental science. The authors review the current state of test-retest reliability for child and adolescent fMRI studies, and provide important guidance on the way forward by highlighting ways to improve fMRI test-retest reliability in developmental cognitive neuroscience research, emphasizing the critical role of open platforms for longitudinal fMRI study designs, analyses, and reporting of results.

Finally, a key question that remains unresolved is whether it is ultimately useful to provide generalized normative growth curves when we are trying to understand individual development. There has recently been significant discussion of whether it is even possible to characterize normative patterns that capture meaningful information about individuals given that for many measures, within-individual variation is often significantly greater than between-individual variation (Fisher et al., 2018). One methodological advancement that appears to be critical to further understanding this issue is the collection of intensive longitudinal data, where measurements are repeated with high frequency within individuals (Bolger and Laurenceau, 2013).

2. Ecological and Developmental Validity

A methodological challenge that bedevils all of cognitive neuroscience, indeed all of experimental psychology, is that of ecological validity - do our experimental tasks actually probe the processes associated with the issue of ultimate interest? In a previous publication in the journal (Pfeifer and Allen, 2016) we have pointed out the importance of this issue for the future of the field, and suggested it should be a critical criterion on which studies are evaluated. For too long experimental designs have been justified on face validity criteria in the absence of actual empirical data showing that performance or neural responses associated with the experimental paradigm correlate with the outcome of interest (e.g., everyday decision making, mood, interpersonal functioning), or more broadly the psychological and neurobiological processes that are specifically relevant to these functional activities in daily life. Sherman and colleagues (this issue) address this important issue in the context of a set of questions that have been extensively studied in developmental cognitive neuroscience - functional brain responses that are putatively related to vulnerability to engage in risky decision-making. Their findings suggest that region of interest approaches may be particularly problematic in this regard, possibly because neural factors differentiating riskier teens are not localized in specific regions. They suggest that whole brain approaches may therefore provide more ecologically valid conclusions. The field requires similar systematic analyses of other key ecological outcomes, with associated methodological recommendation in order to address this critical challenge.

Relatedly, van den Bos and colleagues (this issue) argue for the advantages of employing existing computational models of cognition to bridge the gap between neurobiological mechanisms identified via traditional neuroimaging approaches, and the descriptive level of psychological processes. The authors propose that computational models will help us build more specific theories about development as well as identify the processes that produce behavioral change across development. Van den Bos et al. then demonstrate the utility of computational modeling for understanding development in the context of risk-taking, strategy selection, and reinforcement learning. For example, heuristic models of risk-taking tend to focus on reward sensitivity and are loosely defined, whereas computational models attribute differences in risk preferences to more specific mechanisms and allow a more accurate characterization of behavior. In each of these contexts, use of computational models has significant implications for imaging. Continuing with the example of risk-taking, because expected utility and expected value differ, the choice of which to enter as a predictor will affect how the model fits particular voxels; and if these predictors also vary across development, the result may be murky or even faulty inferences.

3. Integrating Contemporary Statistical Techniques into Neuroimaging

One major goal of this special issue was to facilitate understanding and application of advanced statistical techniques suitable for longitudinal neuroimaging analysis. Over time, varying procedures for modeling, handling missing data, and power calculations have been constructed across laboratories, sometimes with minimal attention to the ways similar issues have been tackled in non-imaging applications. Developmental cognitive neuroscience is currently in need of informed consensus on many of these issues. Although some of the topics in the papers in this special issue are not necessarily new to statistical longitudinal modelling, they are designed to summarize key concepts, suggest best practice guidelines, and perform a didactic role for neuroimaging researchers specifically, often by demonstrating the use of these statistical methods with real or simulated neuroimaging data. We believe that these advanced concepts in longitudinal modelling are critical for developmental neuroimaging researchers to understand at a deep level, in order for the field to develop robust research designs and analytical practices that result in replicable and interpretable data.

3.1. A Priori Theory and Design

Although historically it has probably been common for statistical analysis to occur largely after data has been acquired, planning ahead for optimally appropriate ways to model data should improve study design and measure selection. For example, one’s theoretical model of change during development will have important implications for the number of time points included in a study and the spacing of the observations. King and colleagues (this issue) provide simulations to show that by changing the follow up time in a longitudinal study by just one year, the estimates for the model can change considerably. The authors strongly urge us to design both the frequency and age boundaries of our assessments to reflect our theory and hypothesis of change (Pfeifer and Allen, 2016). This is salient in developmental research; how many of us often group the period of “adolescence” into a vague age range? The authors also remind us not to mark time points in our models as equal intervals if the time periods are not, in fact, equal, and they also illustrate the importance of choosing where to center the models with regards to age or time (i.e., where to place the intercept), which has implications for results and should, again, align with the a priori theoretical model. Similarly, knowledge from clinical developmental research can be tied to neurobiological theory to include appropriate timing in models, as Haller et al. (this issue) illustrate in the case of psychopathological outcomes such as social anxiety disorder.

Another design consideration for our longitudinal studies is to include a measure asking participants (if possible) the reason they may not have come in for a wave of an ongoing study. Matta and colleagues (this issue) review the differences between types of assumed missing data mechanisms and point out that if we have information that is related to both the missingness (i.e., lack of a scan at a certain wave) and the outcome of interest, we can include that data as a covariate in our analytical model. It may be prudent to attempt to gain this information from our participants in cohort studies.

Finally, regarding the a priori power calculation, which is often a requirement of grant applications (sometimes irrespective of its appropriateness to the aims of the study), Kievit and colleagues (this issue) provide a freely available script to simulate a dataset and compare potential statistical models. This powerful tool will provide developmental neuroimaging researchers with more accurate and robust study designs.

3.2. Analysis

There are several modeling strategies to assess developmental change, which are reviewed in King et al. (this issue). Developmental neuroimaging research, especially fMRI, has long suffered from a dearth of studies with more than two time points, making it difficult if not impossible to estimate complex models. Many studies have attempted to fit non-linear trajectories without having three or more time points. However, all is not lost for those studies with only two time points; Kievit et al. (this issue) focus their didactic paper entirely on latent change score modeling (LCSM) and give examples of this analysis using datasets with only two time points. This modelling strategy is also useful for brain-behavior research questions using cross-domain coupling to ask if, for example, changes in cognition depend on initial neural measures such as ROI volume, or if volume depends on initial cognitive measures, or both. However, it is probably of no surprise that answers to more complex questions about development require more than two time points of data, and Kievit et al. explore more advanced techniques possible with these richer datasets, including dual change score modeling and multigroup comparisons.

Haller and colleagues (this issue) provide an excellent illustration of how advanced statistical techniques for longitudinal modeling can be applied to a specific topic area: the development of social anxiety disorder. After reviewing the literature on SAD and its emergence, and arguing for the necessity of conducting longitudinal studies to understand this disorder, the authors step through various analytical options to consider during the design phase. This includes multilevel modeling, parametric versus nonparametric models, and differential equation models.

Another modelling complexity specific to longitudinal research involves decisions about how to treat missing data. Matta et al. (this issue) provide the first sensitivity analyses comparing available and complete data for longitudinal neuroimaging data to illustrate how parameter estimates can change and bias can be introduced if missing data mechanisms are not modelled correctly depending on the assumed missingness mechanism. For example, when exploring a longitudinal fMRI dataset, the authors showed that when using all available data, two additional clusters were identified in a task that were absent when only complete cases were analyzed (i.e.,including only participants who had all waves of data). This is a powerful illustration of how much choices about how to treat missing data in longitudinal neuroimaging studies matter.

Although many of these advanced statistical analysis, design, and modeling tools have been available and used in developmental science for quite some time, longitudinal fMRI research in particular has not benefited from them. This may be partially because of software issues related to multiple model comparisons in whole-brain voxel-wise analysis that are reviewed in Madhyastha et al. (this issue). They provide a useful table summarizing various types of statistical models in longitudinal research and what they are capable of. They also point out that none of the advanced SEM models can be used in any of the current fMRI software (FSL, SPM, and AFNI are reviewed and current longitudinal capabilities described). This is hardly surprising given that voxel-wise analysis in fMRI involves tens of thousands of separate analyses using the GLM framework. However, they also reveal a new, sophisticated software solution (“Neuropointillist”) that allows researchers to interface with R to conduct the types of complex multivariate analyses for voxel-wise modelling that are described elsewhere in this special issue. Of note, Neuropointillist can accept output (i.e., parameter estimates) from the first-level analysis of any other fMRI software. It is also currently the only software that can handle missing data when correlating neural and behavioral data at the whole-brain level without listwise deletion. While Matta et al. (this issue) describe strategies for dealing with subject drop out (i.e., missing a scan/wave of a study completely), until now there was no good solution for missing data at the voxel level (e.g., with movement artifact). Neuropointillist is a promising direction for more flexible processing and analysis in this regard. Although this only begins to address how best to conduct model selection and comparison with voxelwise modelling, it is a flexible and powerful tool for neuroimaging researchers wishing to assess more complex models of developmental change. However, as they point out, we need much more progress in programming technology due to the time and computing power necessary for testing each model for such a large number of voxels.

Furthermore, for missing data mechanisms, the next challenge is data that we consider or assume to be missing not at random (MNAR). Do we, as developmental cognitive neuroscientists, think that there could be a good reason that the probability of someone missing a scan is dependent on the data that went uncollected? In many cases, the answer may be yes, and if so, using all available data could result in biased estimates. But, as discussed above, we could consider asking participants why they cannot or chose not to come in for scans, and include this covariate that is presumably related to both the missingness of the data and the dependent variable in our analytical models. Furthermore, as was demonstrated in Matta et al. (this issue), more research could provide sensitivity estimates comparing complete case and available data analyses.

4. Guidelines and Diversity of Methods

Another goal of the special issue was to outline best practice guidelines for the processing and analysis of developmental neuroimaging data, especially for longitudinal study designs. There was significant diversity in the methods covered - sMRI, fMRI, dMRI, fNIRS, and even magnetic resonance elastography (MRE) - although some modalities were noticeably absent from the special issue (such as EEG and MEG). Perhaps experts in those fields will be motivated by this special issue to produce manuscripts to serve similar guiding functions. In developmental cognitive neuroscience, best practices may also vary to some degree by age group, and there was a significant effort in this special issue to address imaging of infants and very young children. It is notable that several of the papers included in the special issue provided tables, checklists, and tools to facilitate decision making at every stage of the research process. For example, King et al. (this issue) distilled their essential messages into a box for easy printing, framing, and hanging next to one’s workstation for quick reference. With so many statistical models and considerations to track, an easy reference guide provides increased utility for developmental cognitive neuroscientists just beginning their journey of longitudinal analysis of developmental change.

4.1. Reviews by Modality

Vijayakumar et al. (this issue) provide a checklist for researchers to use when reporting methodological detail in longitudinal structural brain imaging studies. This checklist was developed after the authors systematically reviewed the existing longitudinal studies of brain structure in developmental samples, and realized that many studies left out essential details needed for comparing results across studies, such as quality control or model selection procedures. It will serve as a valuable resource for developmental cognitive neuroscientists to use when writing and reviewing papers. Standardizing how we report our study design, methods and results will benefit future meta-analyses, systematic reviews, and help us understand how methodological differences could be related to discrepant findings. Vijayakumar et al. also focus extensively on statistical analysis of sMRI data, including analytic methods such as multilevel or spline modeling, considering trajectories and peaks, as well as model selection, among several other issues. This provides excellent concrete translation of many of the concepts in the more theoretical statistical manuscripts from the special issue, such as King et al. (this issue).

Telzer and colleagues (this issue) turn a critical eye to longitudinal fMRI. They provide a complementary overview to Madhyastha et al. (this issue), with respect to commonly used software for longitudinal fMRI, and many insights are shared between the two manuscripts - particularly the limitations in many of the widely available packages. However, one unique aspect of this paper is its focus on the particulars of tasks frequently employed in developmental fMRI studies, and key issues to consider if utilizing these tasks in a longitudinal design. It also provides an overview of recent longitudinal developmental fMRI studies, emphasizing accelerated longitudinal designs using a region-of-interest (ROI) approach, and exploring using neural trajectories to predict outcomes. Readers are then treated to detailed “behind the scenes” coverage of three longitudinal fMRI papers, one from each of the three laboratories represented amongst the authors of the collaborative manuscript. This illustrates the host of decisions made from task design to analysis.

Tamnes et al. (this issue) summarize dMRI approaches, including diffusion tensor imaging (DTI) and other less common but more advanced techniques and metrics. Examples of the latter include high angular resolution diffusion imaging (HARDI) and whole-brain probabilistic tractography of ‘fixels’ (populations of fibers within voxels), as well as diffusion kurtosis imaging (DKI) and neurite orientation dispersion and density imaging (NODDI). These more advanced approaches are promising, but currently difficult to apply with developing populations because of their increased acquisition times relative to DTI. The authors discuss various approaches to maximize image quality with the shortest durations possible, and various trade-offs that researchers must consider. They also compare ROI, voxel-based methods, tractography, and graph theory analysis. A running theme throughout the special issue mirrored in this manuscript is the absolutely essential nature of quality control in dMRI, and yet the failures of the field to converge on consensus or automated approaches to carry out QC. After providing readers with this detailed treatise on the method, Tamnes et al. (this issue) conclude with a systematic overview of longitudinal dMRI studies, as well as developmental effects seen through the lens of sex and individual differences, puberty, and atypical development. However, their review highlights the difficulties associated with ascribing changes in dMRI to specific underlying cellular and molecular events, which is a key challenge for the future.

Perhaps the least familiar method in the special issue received introductory coverage from Johnson & Telzer (this issue). They provide a primer on the technique of MRE, which assesses brain tissue stiffness and viscoelasticity by imaging the shear deformations resulting from light vibration of the head, and then reconstructing the underlying mechanical properties. Although MRE has been around for about a decade, it has almost exclusively been applied to imaging the adult brain, including effects of aging and neurodegenerative diseases. MRE is particularly sensitive to motion as it has an acquisition time ranging from 5-10 minutes, which may pose difficulties in more widespread application in younger children and those with psychological or developmental disorders. Nevertheless, we are interested to see the application of this technique grow.

4.2. Imaging the Early Years

Issard & Gervain (this issue) provide an overview of functional near infrared spectroscopy (fNIRS), and outline important technical and physiological considerations for its use, particularly with infants. They focus extensively on constraints of this modality presented by the considerable variability of the hemodynamic response within the literature. In particular, there are reports of inverted hemodynamic responses, as well as extended durations to peak hemodynamic responses. This obviously creates interpretive challenges. The authors review these effects separately by sensory or cognitive function, providing a kind of roadmap for infant researchers looking to use fNIRS. Canonical response functions are typically seen earlier in temporal than occipital or frontal cortices and follow a more linear developmental trajectory. The authors also identify ways in which the paradigms and stimuli themselves may influence the hemodynamic response, especially in frontal cortex. Stimulus complexity and familiarity, biological development with age, and experimental design may all influence neurovascular responses.

Another article tackling methodological issues in infant neuroimaging turns the spotlight on fMRI (both activation- and connectivity-based approaches). Cusack, McCuaig, and Linke (this issue) focus on challenges specific to comparing activation and connectivity across age groups. Given the dramatic growth in head size, shape, and gyrification, the authors note that inter-subject registration in studies comparing across age groups including infants is best when first registering to an age-specific template. Different hemodynamic responses again asserts itself as a potential concern, as well as physiological noise, brain chemistry, infant behaviors such as motion or sleep, and peripheral sensory changes that affect the way in which the same stimulus is perceived by infants versus other age groups. Cusack et al. (this issue) provide recommendations specific to each of these challenges.

Meanwhile, yet another article in this special issue addresses some of the concerns inherent to the historically sparse literature in young children under the age of six years (Van Phan et al., this issue). fMRI studies in children out of infancy but under age six have only recently begun to increase in number. This is in part due to development of improved methods to acquire high-quality imaging data (Greene et al., 2016, Greene et al., 2018). Now that high-quality data are being collected, Van Phan et al. (this issue) argue it is critical to turn our attention to ensuring the data processing techniques applied to these data adjust for the unique population. The authors examine by turn each processing step and outline child-specific considerations and recommendations. Like others in the special issue, they make an essential point that quantifying data quality should be validated and explored in particular across other MRI modalities. They also argue for using age-specific atlases, to improve segmentation and registration. 4D spatio-temporal atlases, which consist of a series of age-dependent averaged 3D atlases that summarize details of brain structures by age, may be an important new frontier in this regard, but are not easily implemented in current neuroimaging software packages. Additionally, multi-atlas based methods may also be a promising new avenue, using learning algorithms to select the best atlas for each participant.

5. Open Methods

Several of the papers presented within this special issue have made methodological tools available to facilitate the adoption of advanced data analytic techniques. For example, in addition to describing the theory behind and rationale for using latent change score modeling (LCSM), Kievit et al. (this issue) presented a practical tutorial for applying LCSM to longitudinally acquired MRI data with the open-source software R and Ωnyx. They also made sample data and scripts available for researchers to adapt to their own projects. In a display of high conscientiousness, the authors even created a graphical interactive web application to help researchers understand the impact of changing various model parameters (http://brandmaier.de/shiny/sample-apps/SimLCS_app/).

Many contributors to the special issue attended the Modeling Developmental Change workshop held in September 2017 in Portland, Oregon, USA, immediately prior to the annual meeting of Flux: The Society for Developmental Cognitive Neuroscience. The purpose of this workshop was to teach best practices for processing, analyzing, modeling, and interpreting longitudinal neuroimaging data in developing populations. In effect, this workshop presented the opportunity for researchers to apply many of the methods discussed in the special issue. All of the resources used in the workshop, including presentations, tutorials, tools, scripts, and example data, were made freely available to researchers through open science repositories (available here: https://osf.io/hym23/). Published guidelines and methods papers represent one approach to increase the output of robust and reproducible research in developmental cognitive neuroscience, and hands-on workshops and online tutorials represent a complementary approach to allow researchers to return to their projects with tools in-hand. The two approaches are necessary to give researchers the theoretical knowledge of when and why to apply certain analytic techniques, and practical knowledge of how to do so.

6. Conclusion

To close this introduction, we want to strongly encourage readers to keep in mind that the best practices outlined in the special issue are guides rather than constraints, and that our field benefits from the diversity of methods employed for understanding the developing brain just as we do from standardizing our practices. The breadth of manuscripts in the special issue illustrates how guidelines are not meant to constrain the field, but rather draw attention to statistical and methodological considerations when conducting longitudinal and other forms of developmental neuroimaging research. We hope this special issue can play a role in potentiating high-quality research in the field for years to come.