Dense sampling approaches for psychiatry research: Combining scanners and smartphones

Abstract

Together, data from brain scanners and smartphones have sufficient coverage of biology, psychology, and environment to articulate between-person differences in the interplay within and across biological, psychological, and environmental systems thought to underlie psychopathology. An important next step is to develop frameworks that combine these two modalities in ways that leverage their coverage across layers of human experience to have maximum impact on our understanding and treatment of psychopathology. We review literature published in the last three years highlighting how scanners and smartphones have been combined to date, outline and discuss the strengths and weaknesses of existing approaches, and sketch a network science framework heretofore underrepresented in work combining scanners and smartphones that can push forward our understanding of health and disease.

“Bit by bit, putting it together

Piece by piece, only way to make a work of art

Every moment makes a contribution

Every little detail plays a part…

Putting it together

That’s what counts!”

- Stephen Sondheim, Putting it Together

Humans are complex systems, with feelings, thoughts, and actions that are interconnected and that change over time (1–3). Changes that occur in these complex systems are the product of dynamic processes that span multiple levels of analysis: biological, psychological, and environmental. An essential goal of biological psychiatry is to understand how between-person differences in the interplay within and across these levels leads some people to experience chronic difficulties in adaptively changing their behavior to meet life’s changing demands. Two influential methodological approaches have been used to meet this goal. One approach uses brain scanners to primarily capture aspects of the biological and psychological layers of human systems, identifying neural correlates of deviations in cognition, affect, and behavior accompanying clinical disorders. Creative designs incorporate aspects of the environmental layer of human systems into this work with scanners, simulating social exclusion by using ball tossing games in which participants are excluded from play (4), exposing participants to aversive odors while in the scanning environment (5), and using complex media (film, television, podcasts) as stimuli (6), for example. However, situating these data within the sociocultural milieu of human experience to understand how the interplay between biological, psychological, and environmental layers of human experience produces clinical symptoms remains a challenge. A second approach, smartphone-based techniques, captures individuals’ current symptoms, as well as the psychosocial correlates of those experiences, in naturalistic environments (7). Work in this modality has characterized the interplay between psychological and environmental systems but, unlike work with scanners, does not tie these relations back to the biological level of analysis.

Together, scanners and smartphones have sufficient coverage of biology, psychology, and environment to articulate between-person differences in the interplay within and across biological, psychological, and environmental systems thought to underlie psychopathology. An important next step is to develop frameworks that combine these two modalities in ways that leverage their coverage across important layers of human experience to have maximum impact on our understanding and treatment of psychopathology. What might this combination of scanners and smartphones look like? By undertaking a systematic review of literature using data combining brain scanners (inclusive of brain imaging modalities and related methods to record neural processes) and smartphones (inclusive of experience-sampling and related ambulatory assessments delivered via smartphone or other portable digital technologies) published in the last three years, we identified existing approaches to combining smartphones and scanners in psychiatry research (see supplement for details of the systematic review). With these findings from the extant literature in hand, we 1) outline and discuss the strengths and weaknesses of existing approaches and 2) sketch a network science framework heretofore underrepresented in work combining scanners and smartphones that can capture the richness of the multiple interacting units across the biological, psychological, and environmental systems highlighted in theories of psychopathology.

Existing approaches to combining scanners and smartphones

Six approaches to combining scanners and smartphones were identified in the extant literature.

Bivariate associations.

By far the most common way of combining scanners and smartphones was by estimating bivariate associations between indices from scanners and smartphones either via correlation or regression approaches (Fig. 1A). This approach to combination was concerned with the extent to which in-scanner data could predict “real-world” behavior (see (8) for example). The inclusion of smartphone data, often collected as participants went about their daily lives, allowed an assessment of the extent to which scanner data, high in experimental control but low in ecological validity, predicted ecologically valid experiences. When links were observed between scanner and smartphone data, this was sometimes interpreted as evidence for identifying mechanistic insight into real-world behavior. For example, an observed association between the blood oxygen level response (BOLD) response in reward-related regions during reward anticipation and daily reports of motivation and pleasure was taken as evidence that differences observed in the scanner had explanatory power for the differences observed in daily life behavior (9). While such bivariate associations are suggestive, combining scanner and smartphone data in this way remains correlative and should be interpreted cautiously when attempting to make mechanistic rather than solely predictive claims. A less common approach predicted scanner data from smartphone data (10,11). These efforts highlighted the greater feasibility of intensively sampling smartphone data as compared to obtaining longitudinal scanner data and the implications of this feasibility for tracking the course of clinical outcomes. For example, keyboard dynamics emerged as reliable measures that distinguished multiple sclerosis patients from controls, with the potential to be valid surrogate markers for clinical disability in multiple sclerosis than less feasible but common neuroimaging assessments (10).

Figure 1.

Existing approaches to combining scanners and smartphones emerging from systematic review of the literature combining scanners and smartphones.

The smartphone data in work estimating bivariate associations between scanners and smartphones generally consisted of static indices: proportion of positive experiences with classmates (12), average subjective stress (13), and average momentary subclinical psychosis (14). Thus, their inclusion, as compared to less burdensome retrospective survey reports, increased ecological validity and reduced retrospective biases often introduced in questionnaires that ask participants to recall and aggregate information about longer periods of time (e.g., previous 30 days (15)). However, only a few studies made use of one of the features unique to smartphone data over traditional survey measures: the ability to capture dynamics. Through repeated assessment of participants as they go about their daily lives, smartphone data collection results in rich time series data that can capture moment-to-moment or day-to-day fluctuations in clinical symptoms (16) social experiences (17) and changes in the environment (18,19). Biological psychiatry has a keen understanding that it is not simply the presence or absence of symptoms that characterizes clinical disorders. Instead, the temporal characteristics of symptoms across time are key considerations. For example, affective lability (i.e., intense, frequent, and reactive shifts in affect) is commonly observed in borderline personality disorder (20) while a diagnostic marker of depression is sustained depressed mood nearly every day (21). Capturing this lability or lack of change requires the ability to intensively sample affect over time, a task for which smartphone-based approaches are exquisitely suited.

The use of dynamic indices from smartphone data, particularly of affective experiences, is beginning to emerge in studies combining scanners and smartphones with bivariate correlations or regression (22–24). Even less common in the reviewed papers than temporal smartphone features were temporal scanner features (see for exception (24)). Although it is common to aggregate BOLD data across the entire duration of a scanning session, the brain exhibits dynamics over many timescales, from the sub-second to the lifespan (25,26). Just as biological psychiatry recognizes the importance of dynamics in behavior and symptoms as important for the understanding of psychopathology, the time-varying organization of functional brain systems in depression and schizophrenia deviates from healthy controls, for example (27,28). Attention can be directed to brain dynamics by, for example, taking a sliding window approach, subdividing data from a neuroimaging scan into smaller windows of time and computing functional connectivity indices within each window (29). Alternative, model-based approaches are also possible, computing the dynamics that a brain is capable of given its network structure and assumptions of how activity travels along that structure (30).

Studying brain dynamics has advantages, capturing temporal information about functional connectivity that predicts psychiatric states and conditions, often above and beyond status functional connectivity (31,32). However, in parallel to these affordances to studying brain dynamics are several limitations. For instance, the field lacks consistent analytical approaches, resulting in inconsistent treatment of time and consideration of temporal ordering in analytical pipelines. Further, there is little consensus surrounding appropriate null models for evaluating aspects of time-varying functional connectivity. It also remains an open question what nonneural factors drive changes in resting time-varying functional connectivity (see (33) for a review on questions and controversies in the study of time-varying functional connectivity).

Bivariate change.

The second way of combining scanner and smartphone data overcomes some of the limitations of cross-sectional, bivariate combinations by collecting scanner and smartphone data at multiple assessment periods and calculating bivariate change (Fig. 1B). For example, change in keystroke dynamics were associated with changes in disease activity as assessed by scanners (34,35). The temporal precedence afforded by multiple timepoints of data provides stronger evidence that the bivariate association between brain and smartphone index represents a causal association. Perhaps surprisingly, given the key role for dynamics in psychiatric disorders, collection of even two timepoints of smartphone and scanner data was not common in the reviewed papers. By combining longitudinal scanners and smartphone data, researchers could determine how fluctuations in behavior in real-world contexts both influence and are influenced by changes in brain function. In the ideal case, these data would take the form of intensive sampling of both brain and behavior, facilitating an examination of how naturalistic day-to-day fluctuations in experience (e.g., fluctuations in positive mood) are associated with fluctuations in aspects of functional brain architecture (e.g., brain network flexibility (36)). Such intensive sampling (encompassing multiple laboratory visits) could prove prohibitive, especially when aiming to reduce burdens placed on individuals experiencing psychopathology. However, creative approaches can be used. One such example that emerged in the review captured cognitive performance and neural correlates associated with naturalistic fluctuations in daily stress (37). To reduce the number of laboratory visits necessary to capture within-person differences in stress, participants provided stress ratings three times per day for two weeks, allowing investigators to identify a high-stress and a low-stress day, during which participants were brought to the laboratory to undergo scanning.

Predictors of clinical outcomes.

A third approach combined data from scanners and smartphones by treating them as features that could independently predict clinically-relevant outcomes (Fig. 1C). For example, the average feeling of peer connectedness across a 10-day daily diary and the BOLD response to positive peer feedback during an in-scanner social incentive delay task were used as predictors of suicidal idea in a regression analysis (38). These efforts build on brain-as-predictor applications that show that neuroimaging indices are often predictive of health-relevant behaviors above and beyond self-reports, explaining previously unaccounted for variance in behavioral outcomes (39,40). In the examples of this approach in the reviewed manuscripts, smartphone data consisted of aggregated data across the data collection period. Thus, although these smartphone data are high on ecological validity, their predictive capacity could be improved by creating dynamic features from the intensive longitudinal data (41).

Brain as mediator.

A fourth approach treated data from scanners or smartphones as mediators, or explanatory variables in a causal chain (Fig. 1D). Two examples emerged. The first tested the extent to which gender’s association with a greater proportion of positive experiences with peers as assessed via smartphones was mediated by greater nucleus accumbens-precuneus functional connectivity (12). The second tested the extent to which negative affect inertia mediated the association between default mode system efficiency and depression (42). Although both cross-sectional mediation analyses lack the longitudinal data required to test the causal process unfolding over time that mediation analyses implicitly endorse (43), combining scanners and smartphones in this way revealed theoretical positions whereby between-person differences in brain organization are thought to be causally implicated in between-person differences in real-world behaviors associated with psychopathology.

Brain as moderator of temporal process.

A fifth approach made use of the temporal richness of experience-sampling data to examine dynamic processes and tested the extent to which the brain might moderate these processes (Fig. 1E). For example, one study assessed repetitive negative thinking and sadness 10 times per day over 4 consecutive days (44). These dense repeated measures, coupled with appropriate analytic techniques (see (45) for review), facilitated a focus on between-person differences in how moments of increased sadness at one moment led to increases in repetitive negative thinking at the next moment. Including functional connectivity indices associated with cognitive flexibility as moderators of this sadness to repetitive negative thinking link allowed tests for the role of large-scale functional brain network activity as a moderator of real-world, dynamic cognitive-affective processes.

Intervention tools.

A final approach treated scanners and smartphones as intervention tools (Fig. 1F). For example, one study delivered a daily compassion meditation intervention to participants via smartphone over a four-week period (46). By bookending this smartphone intervention with functional brain scans, this design facilitated testing of the extent to which changes induced by the smartphone intervention became codified in the brain. In an example where the scanner was used as the intervention tool, one study examined the ability for in-scanner neurofeedback to change the extent of affective instability, as assessed by smartphone before and after neurofeedback training in patients with borderline personality disorder (47).

Advantages and opportunities for advancement.

By reviewing existing approaches to combining scanners and smartphones, we find that the field of biological psychiatry is making use of several advantages that stem from the unique combination of scanners and smartphones to provide insight into clinical outcomes. These advantages include increasing the ecological validity of behaviors being predicted by neuroimaging assessments, leveraging the different facets of human functioning captured by scanners and smartphones to improve prediction of clinical outcomes, and providing insight into dynamic processes and their neural correlates. There remain exciting opportunities for combining scanners and smartphones, especially by focusing on the unique opportunities afforded by intensive repeated measures available through both scanners and smartphones in the field of psychiatry. One particularly difficult challenge revealed by the current review is the implicit assumption that fluctuations in behavior are more substantial than fluctuations in brain function and organization. This assumption can be seen in the intensive sampling of behavior from moment-to-moment and day-to-day in most smartphone studies reviewed as compared to the often static, within-scanner assessments. Presumably, fluctuations in behavior observed in smartphone assessments derive, at least partially, from fluctuations in brain function, necessitating methodologies capable of more directly matching fluctuations in brain to fluctuations in behavior than is currently represented in the literature combining scanners and smartphones. A key methodological development to overcome this conceptual limitation is leveraging emerging brain modalities that can now more easily be deployed outside the lab (e.g., portable eye-tracking, fNIRS, and mobile EEG (48–50)) and assess fluctuations in brain function concurrent with fluctuations in behavior.

One specific way forward we highlight in the rest of the manuscript is a dynamic network approach that more directly connects the combination of these data modalities with theoretical perspectives that highlight that humans are complex systems made of many interacting components within and across biological, psychological, and environmental systems.

Ways forward: Networks and bringing together facets of human experience

Network science has emerged as a framework with the potential to characterize the complex interactions occurring across biological, psychological, and environmental systems (51). Advances in neuroimaging (52,53) and the development of appropriate tools to describe and model the parts and pathways for communication of the brain (54) have resulted in the booming field of network neuroscience, a field mapping, recording, analyzing, and modeling the elements and interactions of neural systems (51). As the name suggests, network neuroscience relies on formal representations of the brain as a network to capture the parts (nodes) and interrelationships of these parts (edges). From neuroimaging data, one can construct a graph, a simple mathematical representation of a network composed of nodes representing system elements and edges representing element relations or interactions. Nodes are typically parcels of gray matter voxels, ranging from single voxels to larger clusters of voxels. Associations among nodes (edges) are established in several ways, taking the form of either structural or functional connectivity. Structural connectivity describes anatomical, or physical connections between nodes or neural elements (55). With magnetic resonance imaging data, anatomical connections usually refer to white-matter fiber tracts that physically link brain regions and are derived from applying tractography algorithms to diffusion images. Functional connectivity, by contrast, represents communication or coordination between nodes and edges are defined based on statistical similarities in the time series of nodes (56).

Seven studies in our review applied network neuroscience approaches to scanner data (14,24,42,44,57–59). The benefits of network science are not specific to data from scanners. Although no studies in our review used networks to analyze smartphone data, the network perspective so relevant for the brain can be extended to behavior, emotion, cognitive, and environmental systems more broadly. There is a burgeoning literature, for example, that takes a network perspective of mental disorders (60). This network perspective conceives of mental disorders as a complex system of mutually reinforcing and interacting symptoms (61,62). In these networks, symptoms make up nodes of the networks. Unlike structural brain networks, there are no physical edges connecting emotions, thoughts, or actions. Instead, the collection of multiple reports of the intensity of certain emotions, thoughts, and actions via smartphones facilitates the estimation of edges in a way analogous to the construction of edges in functional networks of the brain: inferring coordination or causal associations among symptoms across time by estimating correlations, partial correlations, or regression coefficients characterizing both time-lagged and contemporaneous associations among nodes (63).

Extending this network perspective to smartphone data in studies combining scanners and smartphones will align the analytic treatment of smartphone data with theoretical notions of humans as complex systems. It will also expand the feature space used in existing work using smartphone data to predict clinical outcomes (e.g., Fig. 1C). But perhaps the most exciting potentiality of constructing both brain and behavior networks in work using scanners and smartphones is the avenues that would open for new ways of combining scanner and smartphone data. Networks need not be limited to one level or layer of the complex biological, psychological, and environmental components implicated in psychopathology. Instead, multilayer network approaches allow the combination of scanner and smartphone networks (64–70). In multilayer networks, each layer constitutes a different network. For example, a network constructed from a different participant, patient group, experimental condition, time point, or data modality. A node can exist in all layers or in a subset of layers and may be linked throughout layers by an edge representing the node’s identity. Multiple types of edges can link nodes within and between layers to represent different types of associations between network elements. Multilayer network approaches have successfully been applied to a diverse range of fields, including neuroscience, ecology, public health, biology, and political science among others (64,71–74).

Recent multilayer applications provide insight into why they may be useful for connecting scanner and smartphone networks. A recent study built a network of networks in which the cognitive nodes were scores from multiple cognitive tasks, including matrix reasoning and digit recall, and neural nodes were region-based cortical volume of several brain regions and fractional anisotropy (proxy for white matter integrity) of several brain regions (75). Partial correlation networks were estimated such that the edges represented conditional dependencies among the cognitive and neural variables. The resulting network was a complex, multi-layer structure of interdependent facets of brain and behavior. There are several benefits to this multilayer approach. First, the ability to combine different layers of human systems within the same overall multilayer network allows the application of network statistics to be applied to one object. This directly addresses the inherent dependency between biological, psychological, and environmental layers of human systems. Second, multilayer frameworks open the ability to probe and predict how perturbations at one node in one network layer (e.g., brain) might impact another node in another network layer (e.g., behavior), helping to guide where it may be best to intervene. Placing brain and behavior in the same analytic paradigm also avoids a hierarchical prioritization of brain or behavior, as observed in mediational approaches which posited a more central role for neuroimaging facets as causes for clinical outcomes than behavior (e.g., Fig. 1D), which reflects a reductionist thinking that does not reflect the interdependent nature of biology, psychology, and environment (76). And, perhaps most importantly, it provides a framework to test for between-person differences in the interplay within and across biological, psychological, and social levels of analysis which may provide fruitful for understanding clinical disorders.

Although a variety of neuroimaging modalities were used in the studies we reviewed (i.e., functional connectivity, structural, resting state, and lesion), in principle graph theoretical network analysis can be used for any imaging modality. In the typical application of network analysis, brain regions are represented as nodes and the connections between brain regions are represented as edges. The nature of the edges differs across imaging modalities. For example, in functional networks, edges typically represent statistical similarities in the BOLD times series of nodes while, in structural networks, edges are often estimated by reconstructing the trajectory of axonal tracts using indices of the diffusion of water molecules within fibers (77,78). Despite each of these approaches resulting in a network, each modality captures a different spatial and temporal scale of the multilevel brain (25, 51) such that the choice of modality will be driven by the researcher’s specific question. Where needed for the question at hand, a multilayer perspective allows multiple network layers from different modalities, capturing aspects of brain network organization at various temporal and spatial levels, to be considered in-tandem, each modality making up a layer (e.g., 79,80).

Thus, the groundwork has been laid to combine multimodal data from scanners and smartphones into a multilayer human system network (see Figure 3) that is capable of integrating the many facets of human experience (25,81–83). Of course, incorporating multilayer networks into biological psychiatry work incorporating scanners and smartphones will not be without difficulty. An important challenge researchers are currently tackling (see (84) for example) is precisely how to best connect distinct layers with one another to form a multilayer structure, especially in a way that maintains the within-person temporal associations among facets of experience that can be estimated from smartphone time series data in a way that cannot be achieved using traditional, retrospective survey measures.

Despite the strengths of multilayer network approaches, namely the flexibility to integrate different types of high dimensional data, there are some cautions that warrant further discussion. On the side of feasibility, there is inherent difficulty in collecting the types of high dimensional data ripe for multiplex network analysis (e.g., electronic health records, connected wearable gadgets, brain scanners, and smartphones). As the number of available measures increases, the choices to examine similarities across the layers also increases, and these indices may largely be based on what researchers are interested in, which can have a large influence on the results. For example, estimating the correlations in a multilayer network requires estimating all the edges across layers and not just the correlations among emotions in the emotion layer (Figure 2). In this way, communities in multilayer networks can occur within and between layers (70), nodes can have relations (edges) within and across layers, and nodes can communicate with one another even if they do not have direct edges between them across layers (85). Some research questions may necessitate collapsing multiple layers into a single layer describing the clustering of patient health states (e.g., cardiovascular disease, affective disorders, cerebrovascular disease (86)) whereas other questions may seek to incorporate multiple layers to uncover the combined impacts of genetics and lifestyle factors on disease to build comorbidity clinical profiles (87). These example applications of multilayer networks highlight the critical consideration that the relevant layers need to be measured with enough samples to achieve statistical power, which may require extra effort especially when collecting multimodal data. Consequently, researchers may seek to maximize efficiency by performing many analyses on the data, making multiple correction and preregistration especially relevant when using multilayer networks. Confronting such challenges is inherent to multidimensional systems modeling.

Figure 2.

Example of a multilayer human system network, encoding information about psychology (symptom network), biology (brain network), and environment (social network), and the interlayer links between them. Figure adapted from (88). A multilayer network framework that incorporates possible mechanisms beyond symptoms may offer additional explanatory insight in biological psychiatry. For example, densely connected symptom networks is associated with greater vulnerability to develop psychopathology (89,90) than less-dense networks. Similarly, many psychiatric disorders share brain network alterations in functional connectivity (e.g., altered functional connectivity in the default mode network has been observed in Alzheimer disease, autism, schizophrenia, depression, and epilepsy (91,92). Considering the social network layer, living alone and away from family have been observed in alcohol dependence (93) and low and high depressive symptoms have been strongly correlated with such scores in friends and neighbors (94). A multilayer network approach that integrates psychological, biological, and environmental can offer insight into shared and distinct features in each layer across psychiatric disorders. In this way, a multilayer network framework can identify common targets for intervention, be it in symptom, brain, or social networks as well as offer insight into explanations for psychiatric comorbidity.

Conclusion

Our review of recently published literature combining scanners and smartphones indicates that the field of biological psychiatry is making use of several advantages that stem from the unique combination of scanners and smartphones to provide insight into clinical outcomes. There remains room to grow in the ways scanners and smartphones are combined that will more directly connect the combination of these data modalities with theoretical perspectives that highlight that humans are complex systems made of many interacting components within and across biological, psychological, and environmental systems. In particular, network perspectives, especially with reference to smartphone data, were not represented in the reviewed work, highlighting a key gap to be filled. We look forward to continued work with scanners and smartphones and the potential this work holds for characterizing how between-person differences in the interplay within and across biological, psychological, and environmental levels leads some people to experience chronic difficulties in adaptively changing their behavior to meet life’s changing demands.