Systematic Representative Design: A Reply to Commentaries

We were unsure how the field would respond to our article. After all, Systematic Representative Design (SRD), enabled by today’s technologies, is fundamentally calling for a new paradigm for psychological science. Although SRD leverages many of the strengths of past designs in a new synthesis, it challenges major assumptions and ways of doing business: In our experience, that is often neither easily understood nor responded to positively. On the other hand, it affords opportunities for leveraging, integrating, and advancing more traditional research in psychology. The format of Psychological Inquiry, with opportunities for deep commentary, affords a unique and delightful opportunity to hear from and respond to the commentaries from an array of leading psychologists, who themselves have and are pushing the boundaries of the field. We are grateful for the opportunity to both clarify our original ideas and integrate the invaluable feedback of commentators joining us on this issue of Psychological Inquiry. The following response is organized around what we as authors identify as major themes or issues that our commentators raised regarding our call for a systematic representative design.

Broadly, given our prior expectations, the commentaries were to our ears, generally extremely positive, and at the same time thought-provoking. It was exciting that so many researchers mostly “got it”. Some offered responses that gave us new opportunities to clarify misunderstandings that are apt to arise more broadly in the field. The other major theme (that had many sub-themes) was the feasibility of this approach. We address both below.

Systematic Representative Design: “Got” and Not Quite

We were thrilled in reading Crano’s (this issue) commentary. We are profoundly grateful for his ability to take what we admit is a complex position and to convey it so succinctly to our audience. As we note above, the rest of the commentators “got” the broad strokes of our argument. However, there were some misunderstandings. Here, we attempt to address them.

The main point of our target article is that it is now possible – given today’s technologies – to “build into” our experimental designs not only the capacity for causal inference but also to “build into” those designs generalizability to everyday life (GEL). By “built in” we mean achieving both of these capacities (causal inference and generalizability to everyday life (GEL)) in the design phase and not after the completion of the study (e.g., as in external validity: GEL is not a kind of external validity). This is achieved by first designing a special kind of control group, a default control group (DCG): This control group is designed to be more representative -- via extensive formative research -- of the sequences (and situations) of interest (SOI) leading up to the behavior of interest (BOI) for the population of interest (POI). The DCG is a control condition that remains constant in comparison to possible experimental conditions: We thank Granic and colleagues (Michela, van Rooij, Klumpers, van Peer, Roelofs, et al., this issue) because their commentary suggested we needed to clarify this point. Experimental groups, in systematic representative design, are composed starting from the DCG base and one or more changes are made to it — added/subtracted. In that way, the DCG is created first (not as an afterthought) and each experimental condition systematically takes away or adds factors to the DCG base to create each of the experimental conditions (e.g. E₁DCG₁, E₂DCG₁) that is (are) then compared to the base DCG₁ control condition, creating various systematically manipulated factors in the overall design.¹

Generalizability to Everyday Life (GEL): Close approximations?

Michela et al. (this issue) say that our goal is to “create a close approximation to ‘real-life’” (p. 2): Much depends, however, by what is meant by “close approximation to ‘real-life’ ”. On the one hand, we are trying to representatively sample from everyday life and capture the psychological gist of the critical cues in the SOI leading up to the BOI for the POI. On the other hand, as several commentators understood (e.g., Crano, this issue), we are not trying to capture “mundane realism.” Nor are we trying to capture in the virtual space every moment in hours, days, weeks, months or even years, as if in “real life”.² Rather, our goal is much more conceptually focused. The goal is to capture the gist of those sequences representative of the critical sequences leading up to the behaviors of interest (BOI). Within those sequences, the “gist” we wish to capture is the psychological gist and critical cues of the often structured aspects of the sequence and the decision points within that structure (and probabilistic behavioral options for the target population of interest). Specific physical cues may be (or may not be) part of capturing that psychological gist. A thirty minute interaction in “real life” may have a story structure such that only a few seconds here and there are actually critical to the evolving psychological gist of the story and critical choices. The content in between may be heavily “filler” or noise and might not be necessary given our research goals^3,4. The challenge for researchers is to do the formative research to ascertain what is critical and what is superfluous in the construction of meaning and human-agent interaction, given one’s research goals. In many regards we are also “crunching time” to its essential psychological “gist”: Much like a director does, we are taking the film splices that are critical to produce the construction of meaning of interest, reducing the unnecessary “noise”.⁵ In essence we could ask what is critical to the “gist,” so if I remove or add more at one point are we at a “goldilocks”⁶ moment for our population in being sufficiently but economically representative of the SOI and BOI to which we wish to generalize to everyday life. Virtual technology is enabling but we, as researchers, must do some heavy conceptual lifting here. We need to better conceptualize behavior that is frequent/probable in what settings. We need to better specify how that behavior is structured and unfolds in social interaction in everyday life. This is necessary to better instantiate more representative critical cues in these virtual worlds and make more representative settings/sequences/situations of interest (SOI) for target populations of interest (POI)⁷. Within the smallest units of social interaction, probably timing and cues are critical to the meaning of interest given one’s research questions: So again, this is a research question that one’s formative work needs to address.

GEL is Not Ecological Validity

An understandably common misconception is that Brunswikian-based GEL is equated with variants of “ecological validity”, consisting of verisimilitude and veridicality (Franzen & Wilhelm, 1996) in which, “Verisimilitude is the level of believability or the extent to which an experimental task approximates the features of everyday life” (Michela et al., this issue). But, verisimilitude does not tell us much, if anything about how representative those cues are. A simulation could have verisimilitude for an infrequent situation, which would lack representativeness. “Veridicality, in contrast, is the degree to which the performance of a participant in an experimental setup accurately predicts what that person would do in reality” (Michela et al.,this issue). Veridicality also doesn’t tell us whether the situation is representative of everyday life. It just says that for this specific situation it would predict what someone would do in a similar real life situation, regardless of whether the situations were representative.

To repeat, the capacity to have generalizability to everyday life (GEL) in the default control group (DCG) of SRD is “built into” the design based on sampling theory (i.e., the SOI and BOI for the POI are representatively sampled and instantiated into the virtual design). This is analogous to the fact that the capacity for causal inference is “built into” experimental design based on meeting requirements for internal validity (that are separate from the specific operationalizations employed). GEL is, therefore, more akin to Brunswik’s concept of representative design. The key here is the process (e.g., based upon sampling theory to systematically sample from defined populations of settings, sequences, situations and choices leading to behaviors of interest) that builds in the capacity for GEL; it can be evaluated by assessing those sampling processes, just as one would evaluate sampling adequacy in other contexts (e.g., polling samples for population inference). An environment judged to have verisimilitude might or might not have GEL depending upon one’s sampling procedures and the populations (e.g., SOI, BOI, POI) to which one wishes to generalize.

Veridicality may be more akin conceptually to virtual validities,⁸ that are coefficients that reflect contextual feature-response relationships in the past time-frame of reference (e.g., past 90 days), and virtual choices in similar contextual feature-response option situations. They diverge, however, in several ways. First, veridicality would seem to occur “after the fact” like external validity, whereas virtual validities are assessed in the formative design process and in summative measures with “real-life” participant measures typically prior to participant engagement in the virtual environment. Second, as part of the assessment process we focus, throughout, on specific target populations of individuals and situations of interest and behavioral options likely in each context (real, virtual) over a given period of time. For example, sexual behavior preferences (e.g., being a top or a bottom) for young (18–24 year old) men who have sex with men (YMSM) are potential behavioral choices that may occur with sufficient regularity in real life and virtual life, with sufficient stable between-person variability within this population, given precipitating context for the specific target population of interest (e.g., YMSM), that we would expect to see similar between-person coefficients between summaries of context-behavior patterns in real and virtual contexts. Ecological validity as used in common parlance today in psychology -- but not using Brunswik’s definition — strikes us as a rather “fuzzy” concept in that it often is not specifically tied to a numerical operationalization to measure it. Virtual validity on the other hand, can be assessed statistically⁹. We use it as a check on GEL (at least between-persons) because, given the complexity of not having a full sampling frame of the situations to which we’d like to generalize, a check would appear a prudent methodological choice (one indeed, Brunswik (1952) recommended decades ago).

Ecological Validities are a Delightful But Different Concept Too

For Brunswik, ecological validities (note the plural usage here), as we noted in the target manuscript (Miller et al., this issue) are often confused with representative design (Araújo et al., 2007). For Brunswik, ecological validities “refer to coefficients indicative of proximal environmental cue validity in predicting to the specified distal criterion (environment) state or policy (Araújo, Davids, & Passos, 2007).” One reason to struggle to unlearn what we thought we knew (about ecological validity) and to deeply learn what Brunswik’s concepts actually meant (at least for Brunswik) is that we could be highly rewarded by their potential utility in virtual environments. For example, we might expect that top-down processes will play particularly significant roles in social interaction, but proximal cues could as well, especially if they are incompatible with top-down predictions. Imagine we had highly representative virtual environments. Imagine within them, that we could tell when people were considering (or not) what cues (e.g., eye tracking) to use. Given that, for example, it seems likely that we could begin to see patterns of given participant’s proximal cue attention, use, (and weights) as predictive of “top down” and “bottom up” cue utilization (and possible model accommodation for subsequent similar situations) following prediction surprise.

Many Approaches Afford Causal Inference Including Within SRD

Do only experiments afford causal inference? Nezlek (this issue) seemed to think this is what we were arguing. We are not: This is a misunderstanding. Clearly experiments, however, are one important method (which we think could be even better) for doing so (see Miller et al., target article). Brewer and Crano (2014) carefully specify the arguments that indicate that experiments are not necessary for causal inference, although experiments have many key properties that make it easier for them to afford causal inference capacity. Crano (this issue) elaborates on this point and takes us on a delightful journey through what we thought as a field then and now. We maintain, as others have (e.g., Crano & Brewer, 2014), that causal inference capacity depends, in part, upon our ability to rule out alternative explanations, and there are certainly other methods and procedures that enable researchers to do so. This includes natural observation studies and quasi-experimental methods (see Crano, this issue). Rest assured, we value them all, and since other data with other methods is indeed foundational for SRD, we are certainly not advocating to replace them!

And, of course, as suggested in the beginning of the target article, Pearl (and others) offer inspiring mathematical possibilities here (see also Pearl & Bareinboim, 2014) for examining causality in correlational data.

Furthermore, humans broadly are concerned with causal inference in their moment-to-moment, everyday lives, over time. Having the capacity to understand how those everyday causal inferences are made and constrained is also of interest to psychologists. It seems plausible to us that within the DCG condition one could use individuals’ choices and behaviors within the game over time -- within-persons as well as between-persons -- to better understand individuals’ capacity to make causal inferences. The probability that real-life causal inferences (X → Y) are likely to be captured in the DCG in virtual environments will likely depend upon instantiating the representative gist of the sequences, and behavioral options, leading up to the behaviors of interest for the population of interest.

Processes Unfolding Naturally Over Time Matter for SRD

Nezlek (this issue) raises the important matter of studying behavior over time and appears to suggest that we do not consider this important. This is a misunderstanding. Not only do we think that studying behavior over time is important, we also believe it is important to do so using “intensive repeated measures designs” (e.g., Leger, Charles, & Almeida, 2018). In addition, we would argue we need to do more: We need to understand more about moment-by-moment interactions within-individuals¹⁰. Lab studies (e.g., between married couples; between parents and children, etc.) with or without experiments, can provide insight into moment-to-moment interactions between individuals (e.g., in a dyad). Observational methods, such as those involving sensor and other smartphone data possibilities are exciting tools (Asselbergs, Ruward, Eidys, Schrader, Sijbrandj, & Riper, 2016; Harari, Gosling, Wang, & Campbell, 2015; Harari, Muller, Stachl, Wang, Wang, Buhner, Rentrow, Campbll, & Gosling, 2019; Shifman, Stone, & Hufford, 2008) with great potential insight. Nonetheless, due to privacy, ethical, and feasibility concerns it is unlikely that we will be able to “observe” in real-time using sensor and other smartphone technology all the complex interpersonal and interactional behavior (e.g., content of conversations) that we might want to observe, depending upon our research questions. Longitudinal studies over time of different grain sizes (e.g., in the moment; over longer time frames such as every day, week, month, year) may all be useful in piecing together our understanding of emerging processes (assuming we clearly understand when critical events and behaviors of interest are apt to occur so it can be measured relative to other variables).

Virtual environments are potentially useful ways to integrate, instantiate, and observe these unfolding behaviors (and processes) over time in environments likely to be generalizable to everyday life. Formative and existing research (e.g., involving processes over time) provide extremely useful data sources for designing a virtual environment and for being better able to instantiate representative sequences in longer time frames (over many levels -- of “crunched time” in games). The processes modeled in DCG could be compared to similar processes previously observed by the same participants in their everyday lives -- providing another check on whether similar processes (as well as situation-behavior patterns) occur. Prior and concurrent data (from whatever source including longitudinal data sources) act as useful inputs and constraints on designing virtual instantiations: Can we capture the processes suggested in various sources of real-life data (e.g., longitudinal data sets; other observational data; self-reports) in instantiations in virtual environments? Where data is collected in real-life contexts and virtual contexts on the same participants we can do additional process checks (during formative and summative evaluations). Virtual environments and other technologies (e.g., eye tracking, fMRI) can help us instantiate all these “pieces” from our collective research efforts in a test-bed to better test our theories about these complex processes.

SRD Relies Upon Other Methods Including “Deep Description”

As might be apparent from the above points, SRD requires considerable formative research including the use of many conventional methods in psychology. Qualitative (as well as quantitative) research is needed. Not surprisingly, then, we resonated to several of the commentaries that mentioned the importance of other methods, including observational methods that allow researchers — in a detailed way — to understand their population and phenomena of interest (e.g., Nezlek, this issue; Barsalou, this issue). Indeed, SRD relies heavily on — and deeply values — the diversity and utility of our many research methods (e.g., observation, surveys, focus groups). These methods are essential in providing the descriptive base not only necessary for representative design in a given instance (Miller, Wang, Jeong, & Gillig, 2018) but “deep description” also has a long history of being foundational in science.

Importance of detailed “deep description” in science.

We agree with Barsalou’s (this issue) note that, “classic work in biology, chemistry, and physics... begins with observing and describing these phenomena thoroughly and carefully.” (p. 5). Careful and detailed observation of the phenomena and their dynamics over time, what we refer to as “deep description,” has been the basis of major cumulative breakthroughs across fields of science. For example, modern astronomy has its roots in the early systematic eclipse recordings (time, location), using methodology that could be replicated, that predicted future eclipses (Steele, 2000) and, within a few hundred years, enabled the first mathematically-based comprehensive lunar theory of movements of the sun and moon (Britton, 2007). Today, daily, systematic, replicable, worldwide measurements of CO₂ over more than 50 years provide one data source for testing climate science hypotheses and developing computational models to better predict and understand global warming (Egger & Carpi, 2008 detail a number of these examples of the value of systematic description for building a replicable, cumulative science). Of course, detailed description was key to Darwin’s theoretical development (Darwin, 1859).

In psychology, description historically has often played an important role. One example of the foundational value of deep description comes from work on the origins of attachment theory, clearly one of the most widespread and impactful theories in psychology in the last 50 years (Cassidy, Jones, and Shaver, 2013). Over many decades, there were extensive case studies, interviews, and observational studies by both Bowlby and Ainsworth (Bretherton, 1992). Often this involved starting with an applied question. For example, one of Bowlby’s early papers (1944) was entitled, “Forty-Four Juvenile Thieves: Their Characters and Home Lives.” In that work, Bowlby descriptively examined and compared childhood background (e.g., separation at an early age from mother (or loved foster mother), child’s relationship with mother-figure, parental reactions to the child, situations (and events) that might immediately precipitate acts of theft) for thieves versus control children. Ainsworth, following the observational work by Robertson, one of Bowlby’s students (Bowlby, Robertson & Rosenbluth, 1952), similarly collected extensive and detailed analysis of participants’ naturalistic behavior (Ainsworth, 1983) and autobiographical narratives (including for her dissertation) that laid the foundation for her later preference for narrative data collection methods – all of which provided further foundation for impactful breakthroughs in prediction, understanding and theory development, and subsequent clinical interventions (Bretherton, 1992). As the attachment example illustrates, our understanding of behavior and our ability to change it often starts with “deep description” and with systematic, reliable, methods of collecting, coding, structuring, and interpreting observations, detailed interviews, case studies, and other systematic methods (e.g., extensive narratives) that enable us to better describe phenomena and when, where, and the circumstances that lead up to various behavioral sequences for whom. For attachment theorists, this formative research enabled a choice of fruitful narrative challenge situations, including one implemented in a lab as the strange situation procedure (Ainsworth & Wittig, 1969). This procedure involves a mini-drama in which: (1) mother and infant explore a playroom with experimenter, (2) mother and baby are alone, (3) an unfamiliar woman joins them, (3) the mother momentarily leaves her infant with a stranger, (5) mother returns and stranger leaves, (6) mother leaves, the infant is left alone, (7) the stranger returns, and then (8) the mother returns and stranger leaves. At each point there are potentially different ways in which the infants may respond, with some of these differential patterns differentiating attachment style assessments in everyday life.

Systematic methods of “deep description”, and behavioral sampling, can provide the grounded foundation for our most impactful work. But, this important “description” step is often shallow or inadequate. Even the strange situation procedure has been criticized as highly artificial and not as representative of the situations encountered for the infant with important adult figures in real life as it could be (Lamb et al., 1985). A larger and more representative sampling of caregiver-infant interactions might provide greater precision, reliability, and correspondence and everyday generalizability from the lab simulation and the child’s patterns of caregiver-infant interactions. However, real world constraints (e.g., time, resources) often preclude this.

The larger point here, is that SRD requires this type of descriptive formative research. Deep description has often proved productive in science in leading to our most insightful and comprehensive theories. Not surprisingly, we are fans of such descriptive qualitative work: SRD would be impossible without it. Nonetheless, it is sadly often devalued and hard to publish. We would hope that SRD might make deep description a more fashionable approach to research in the years to come. Later we further elaborate how it is feasible for researchers to capture the representative gist using “deep description” formative research.

Feasibility: Too Difficult to Implement

A fairly consistent issue raised by our commentators was SRD’s feasibility. A number of the commentators suggested researchers may not have the time or resources (Crano, this issue), technical expertise (Barsalou, this issue), or even interest (Barsalou, this issue) to create SRD. Others pointed out (Michela et al. this issue) that SRD was not an “off-the-shelf” method. In other words, it may not be easy to implement broadly and generally by most researchers. First we discuss virtual environments’ caveats and where we are now -- including cost, time, resources, motivation and ethical considerations. Then, we discuss feasibility pertaining to: how human-like are these agents and how human-like can they be? Are temporal dynamics possible? How feasible are meta-theoretical “backbones” as simulations and instantiated drivers of agent in games? Is the level of formative research needed feasible? Are scalable interventions feasible?

Virtual Environments: Where are we now

Caveats.

As Michela et al. (this issue) note, there have historically been substantial perceptual system lags (e.g., over time; in bodily response) that can affect user experience in VR (e.g., motion sickness). As we noted in the target article (Miller et al.,this issue), researchers need to be aware of and test for some of these issues and make necessary adjustments to their research questions or approach. That said, motion sickness is a common travel phenomenon outside of VR (e.g., sea sickness), and has much to do with accommodations over time that users often can make, as well as the speed with which they are introduced to stimuli in VR (Benoit, Guerchouche, Petit, Chapoulie, Maneva, Chaurasia, Drattakis, and Robert, 2015). Newer technologies (e.g., AR, mixed reality) in which an app (e.g., on a smartphone) projects images into an actual physical space are also likely to reduce such experiences (see also Mozgai, Harthold, & Rizzo, this issue).

Another effect is the uncanny valley hypothesis (Mori, 2012) that suggests that more human-like that agents are, the more they are responded to positively -- up to a point. Then, highly human-like agents might be unsettling and responded to negatively, creating a “dip” in human reactions to agents. But, a systematic review of research testing uncanny valley effects suggest that “whereas all human-likeness manipulations do not automatically lead to the uncanny valley [in fact, relatively few studies do], positive uncanny valley findings have been reported in studies using perceptually mismatching stimuli” (brackets added, Kätsyri, Förger, Mäkäräinen, & Takala, 2015, 390). Furthermore, as movements became more human-like there was a linear relationship with positive affect. Indeed, in a separate review of virtual characters, deBorst and deGelder (2015), found that mismatches in multi-modal stimuli in characteristics of humans were problematic for “eeriness” but these authors argued that researchers could avoid such negative reactions by matching human-likeness on multimodal features (e.g., auditory, movement, and appearance). Indeed, as deBorst and DeGelder (2015) further note in their review, for avatar faces that are too dissimilar from human faces, the expressed emotions may evoke reduced responses in the observers, as expressed by lower intensity ratings and reduced brain activity. When emotional avatar faces look highly similar to human faces, they may evoke similar emotional responses as expressed by mimicking responses in the face and activation of emotion regulatory regions. However, differences in brain activity [e.g., fusiform area] may occur as a response to the physical differences between avatar and human stimuli. This may be caused by the experience people have with viewing and interpreting human faces” (brackets added).

Thus, although more research is needed, and researchers should always test their stimuli in formative development, as the integration of multi-sensory systems of virtual agents improves, many of these caveats have or will likely be addressed. Indeed, below we discuss some of these remarkable advances in more detail.

Current technological affordances.

Rizzo and his colleagues (Mozgai, Hartholt, & Rizzo, this issue) give us a whirlwind, up to the moment “birds eye view” of the many exciting developments in this domain — many of these from a clinical and psychiatric perspective. The advances in this space in the last 5 years, in particular, have been dramatic, including recent additions to virtual agents in VR. Furthermore, these advances to VR and also the increasing role of AR and other technologies may mitigate the concerns raised above (e.g., motion sickness), and lend themselves well to the theoretical framework offered with SRD (Mozgai et al., this issue).

This is afforded by tools like Smartbody (Thiebaux, Marsella, Marshall, & Kallmann, 2008; Shapiro, 2011), which we illustrate in Figure 1 below. In our experiments (see below) users can be seated while having elaborate animated “conversations” (Feng, Jeong, Krämer, Miller, Marsella, 2017; Jeong, Feng, Krämer, Miller, Marsella, 2017; Krämer, Sobieraj, Feng, Trubina, & Marsella, 2018). Indeed, one of the most unique contributions to work in VR is the meticulous design of virtual humans, using virtual agent simulation software such as Smartbody, and the virtual human gesture software, Cerebella (Marsella, Xu, Lhommet, Feng, Sherer et al., 2013; Lhommet & Marsella, 2014). The dynamic combination of these two software systems allows for realistic virtual humans, equipped with theoretically-grounded graphical physics. For example, Cerebella automatically generates virtual human physical cues (e.g., gestures) based on a given dialog script, parsing information about the agent’s mental state (e.g., emotion, attitude, etc.), communicative intent, and voice prosody. Cerebella’s output allows for gestures that are highly nuanced, including explicit physical gestures (e.g., pointing), emotional gestures (e.g., animated hands; facial expressions), complex gaze behavior, and even micro-expressions (e.g., eye saccades changes).

Figure 1.

Virtual human developed via Smartbody, with gestures generated by Cerebella.

To illustrate the exciting possibilities afforded by the above virtual human simulation tools, we highlight the work by some of the authors of this piece. Namely, Marsella and colleagues (Feng, et al., 2017; Jeong, et al., 2017; Krämer, et al., 2018) have demonstrated the use of VR in highly sophisticated social situations. For instance Jeong et al. (2017) examined approach-avoidance behaviors in head movements (z-axis) in VR interactions with virtual humans, which afford precise measurements of human kinematic movements within-interactions that have previously not been feasible. Such precision in VR affords both the representation (simulation) and perception (measurement) of subtle movements (aberrations, combinations, and sequences) that communicate nuances in human interaction that extend beyond those communicated by cruder categories of movements.

Cost, Time, and Resources.

The cost of VR has gone down dramatically over the last 10 years. As we go to press, the cost of the Oculus Rift headset, including the wireless version, is under $500 -- this is well within the reach of most researchers and labs. The development cost for stimuli used in one’s experiments varies. But, as Mozgai, Hartholt, & Rizzo (this issue) suggest, some toolkits are already available for users with no or minimal cost (e.g., the Virtual Human Toolkit produced by USC’s Institute for Creative Technologies, https://vhtoolkit.ict.usc.edu ). The Virtual Human Toolkit involves a number of continually updated modules to interface with a range of software for embodied conversational agents (ECA).The technology affords agents the capacity to exhibit, in agent-human interactions, sophisticated nonverbal and verbal behavior (using natural language software) and social perception in social interactions, based on extensive formative research on human natural interactions (e.g.,Thiebaux, Marsella, Marshall, & Kallmann, 2008; Feng, Leuski, Marsella, Casas, Kang, & Shapiro, 2015). Other researchers in this field (e.g., Niewiadomski, Bevacqua, Mincini, & Pelachaud, 2009) have also been expanding the capabilities of agents, such as Greta, an embodied conversational agent (ECA) system. We will expand on this discussion below. These ECA systems draw heavily on existing research in psychology and other social sciences, such as in communication (e.g., Gratch, Hartholt, Dehghani, & Marsella, 2013).

The cost for augmented reality design through a commercial group (they will do it for you) could be essentially the cost of a smartphone and the cost of app development (e.g., as little as $5000 or less, to as much as $300,000 depending upon the complexity of one’s design). Unity, the popular real-time multi-platform game platform that allows users to create 3D, 2D VR and AR visualizations for games (and also film, animation, and cinematics) can be used on smartphones, ipads, computers, etc.: It has a no cost (Personal) and low cost starter versions (e.g., $35 annually; Pro $125 annually) for those researchers and labs who want to explore this technology (https://unity.com).

Yes, it will take time to learn new tools. But, really, it takes time to learn how to create better stimuli, learn to use new statistical methods, learn how to use a computer instead of a typewriter, learn how to do neuroscience studies, and simply learn how to do a lot of things that are now a standard part of our “toolkit” for doing psychology. The question is: will the time required to learn this new tool well enough to conduct meaningful work be worth it? That is a question each researcher must ask themselves. On the other hand, working on teams (see below) may be a way to contribute to these sorts of larger efforts (e.g., with formative research) and become scaffolded into the use of these tools through those collaborative networks.

Motivation.

On the motivation front, it is the utility of these systems for achieving things we thought improbable that pushed some of us on this team to begin to invest time and money in learning new things and using new tools. If there is one thing that nearly universally motivates research psychologists it is that a given method can greatly enhance one’s effect sizes. In our experience, VR can dramatically do so. For example, we used a VR (Oculus Rift) system and software (e.g., Smartbody) to create different stimuli for participants (see below). The data from those experimental conditions was extraordinary: There was amazing precision in measuring humans’ approach and avoidance motivation movements (e.g., Jeong, Feng, Krämer, Miller, & Marsella, 2017). That is, there were distributions of responses within a condition of an experiment that had no overlap with distributions on these responses in the other conditions. As wouldn’t surprise our audience, that dramatically impacts one’s effect sizes. That has implications for other things of interest to all of us (e.g., replication, needed sample size).

Ethical considerations.

We thank Michela et al. (this issue) for detailing some of the ethical considerations that apply to VR (as well as, we would note, in most cases, interventions broadly, including virtual environments for interventions). They also refer to other recent reviews of ethical concerns pertaining to VR (e.g., Madary & Metzinger, 2016). Although many of these additional articles addressing ethical concerns seem aimed at consumers or game designers without or with low levels of training in the ethical issues that arise in human research (especially with vulnerable or at-risk populations and involving interventions to change behavior), readers and their graduate students embarking on a line of VR work may find them a very useful reference indeed. Ethical considerations -- as every psychologist knows -- are extremely important and central to our research during its initial development stages (and every stage thereafter). Most of the guidelines for ethical research with VR (Madary & Metzinger, 2016) indeed would have a very familiar “feel” to most of us, especially those of us who have conducted research in clinical and health domains. For example, there are recommendations for familiar areas of ethical considerations such as: “non-maleficence” (e.g., “no experiment should be conducted using virtual reality with the foreseeable consequence that it will cause involuntary suffering or serious or lasting harm to a subject” along with the companion beneficence principle of maximizing participant well-being. This also includes “a rational, evidence-based identification and minimization of risks (also those pertaining to a more distant future) ought to be part of research itself”). This of course is part of a basic General Principle in the codes for ethical conduct from the American Psychological Association (Madary & Metzinger, 2016). “Informed consent” is also certainly part of the ethical consideration landscape in most of our work, including perhaps especially interventions with at-risk populations, although it is suggested that one add language regarding risks that suggests that VR may have “lasting behavioral influences” on participants, and that these “risks may be presently unknown.” There are a number of other useful guidelines here for readers who are considering conducting VR research and interventions in particular.

One advantage of SRD using virtual environments is that it has some features to reduce ethical risks. First, the DCG is designed first (before the experimental group) and the DCG is designed to create a representative environment of the gist of the psychological settings, sequences, and situations (SOI) leading up to the behaviors of interest (BOI) for the target population of interest (POI). Note that development of the DCG is a long process with many checks for indicators of potential effects (intended and otherwise) along the way. With GEL that means (and we can check on our ability to produce GEL gradually in development of our environments), that we are creating a virtual world and experience that has minimal risk, that is we have evidence to suggest that it is no more risky (or likely to have adverse consequences) than the world users would experience in everyday life.

Second, basing the experimental conditions on the DCG base means we have a much more precise idea about what is different in the experimental condition compared to the DCG (and typical everyday conditions) that might be causing what effect, over what period of time. Indeed, standard randomized controlled trials (RCT)-- including those that members of our team have conducted for reducing risky sexual behavior in virtual environments -- involve relatively long-term assessments (e.g., 6 months of more) to examine the continuing effects of a treatment at regular intervals on the BOI. Furthermore, since these conditions, as is the case of those our teams have created, often include additional measures as mediators (including other observational, self-report, and bio-behavioral measures of various sorts) and include pre-measures and immediate post-measures (as well as measures at other longer term intervals), SRD can afford considerable insight into the mediational processes that might be involved in change of what sort. That measurement collection can also potentially signal early on in development (or ongoing research) potential unintended consequences or unexpected benefits of one’s intervention components that can be immediately addressed to mitigate harm and optimize benefit (i.e.,arrange for DCG participants to receive a successful intervention at the end of the study, consistent with what our teams have done in the past).

How “Human” are These Agents?

Above we point to the tremendous advances in virtual bodies, voices, emotional expression and so on and their multi-sensory integration. These advances further enable us to increasingly see social interactions (i.e., between a human and an agent) affecting behavior in ways that are increasingly similar to what we might see in normal human-human interactions. Vallacher and Nowak (this issue) raise a related, but different and important question about assessing dynamics within SRD so that the dynamics produced in agent-human interactions are like those we would encounter in natural human-human interactions. One of those questions we will consider here is how “human” or “human-like” are these agents. This matters as they note below:

However, the essence of many social psychological phenomena is feedback between the actor’s behavior and the behavior of others. Thus, the actor’s behavior promotes changes in his or social environment, which in turn provide a new stimulus situation for the actor, potentially causing a change in his or her behavior. In this sense, the social context is not static, but rather changes dynamically and is co-created by the participant and others representing elements of the social context. Of course, an immersive environment can incorporate the potential for other agents to react to the actor’s behavior in accordance with a predefined set of conditional if-then rules. It may also be possible for the reactions of other agents to be governed by artificial intelligence. Although this allows for studying dynamic patterns of interaction, this is not a true substitute for actual interaction and thus runs the risk of introducing artifacts into the results. Whether controlled by the rules introduced by the investigator or by artificial intelligence, the reactions of other agents may not correspond to the natural reactions of agents in real-world situations. Thus, one part of the feedback between person and social context is created, either by the researcher or by AI, which can result in misleading conclusions (p. 10–11).

First, let’s take the behaviors of intelligent agents we are describing in SRD in games (including ones that authors of this paper have produced using PsychSim).¹¹ These agents are “autonomous” in the sense that they are not responding in a fixed “tree-scripted” if-then sort of way. The agents’ “mental models” of the world, self, and others, among other things, are changing over time as the interaction proceeds. While the agents may not be as “human-like” as humans, agent action sequences with humans are currently “good enough” to consistently yield high correlations (e.g., .70 across four studies) between past 90 day similar situation-behavior sequences and virtual situation-behavior sequences when we aren’t trying to change behavior.

Why are these coefficients so high? A tremendous amount of formative research goes into the creation of these virtual environments, including the modeling of the agents. One question we thought about is this: What controls the humans in social interaction naturally? One of our answers involved various goal-based knowledge structures, including goals and beliefs. The trick in designing more “human-like” social interactions is: 1) incorporating a backbone set of assumptions based on formative research pertaining to those naturalistic human interactions so that the human agent interactions would be based on the same underpinning assumptions, and 2) using formative research to deeply understand the structural organization of situations that humans respond to (e.g., components of a pick up in a bar scenario) and building that into the agents’ models of the scripts in this situation. It also involves choosing agent parameters that align with the major factors that humans in similar situations within this target population use to make decisions. That is, we recognize that humans are approaching social interaction with a set of knowledge structures and we are leveraging that in developing those agents. It is of course an empirical question whether variations in programming agents (that depart from what our formative research suggests may be driving humans-- for example as sexual partners -- in these scenarios) would produce lower virtual validity coefficients and other similar checks on the relationship between virtual agent-human dynamics and real-life dynamics in human-human interactions. And, such empirical research is an intriguing idea! We also suspect that going back and forth in our testing of models could tell us a lot about the “social landscape” for naturalistic human interactions that has often been difficult to discern.

Second, as Valacher and Nowak (this issue) suggest, using the PsychSim backbone in a multiplayer game, for example, we can substitute human players (e.g., human player, human partner) for agents and examine the same unfolding dynamics to see if the version with a human and an agent (or all agents; or all humans) differ, how, and why in terms of our questions of interest over “crunched” multi-leveled games over time. Therefore, the concern they raise is something that can be examined directly in a few ways and is not an “unknowable.” Third, if agents and humans have the same possible choices within a game, and we use machine learning and motion capture technology to inform the design of the virtual agents directly, it is possible to “tweak” agent performance to be more like the distribution of human responses within the game. Then, this can be iteratively improved to produce more human-like agents, leveraging more human data. Fourth, it is possible to simulate the distribution of possible interactions a given human might have with a representative set of other human-like agents in similar contexts over time and assess how those simulations (and the variability in them) predict human future human interaction dynamics in similar contexts.

Examining Temporal Dynamics with SRD?

We concur with Nezlek (this issue) that time, as well as context, matter in understanding, predicting, and changing behavior. And, we agree with Vallacher and Nowak (this issue) that, “the field needs to go beyond [unidirectional] causality to embrace and investigate the dynamical properties of social processes, and to express the relation between psychological variables in mathematical models” (p. 12). We would argue that DCG indeed can provide one window on the types of dynamics of social interaction in the “moment to moment” and over “crunched” time. This might include windows for observing dynamics that are often so difficult to observe in naturalistic studies (in part because it is difficult to predict when the naturalistic conditions in individuals’ interactions (or processes that accumulate over time) will manifest in emergent behavior (e.g., switching from narcissistic grandiosity patterns to an onset of narcissistic vulnerability) in detectable ways. Certainly the temporal dynamics they describe can and do occur in human interactions. Are they possible in agent-agent and agent-human interactions?

Indeed, simulations of intelligent agents in some software (e.g., PsychSim) can involve complex mutual influences of agents (involving cognitions, behaviors, etc.) as agents interact with other agents and adjust their “mental models” of one another over time (that then can create vicious cycles and feedback-like loops) and as agents adjust and update their prediction “look ahead” models in interaction with one another. For example, in PsychSim agents could receive rumors (messages) from other agents that subsequently proved false, altering agents current and future models of social interaction that can alter current and future models of that specific other, for example as less trustworthy (and their models of the other’s model of them). Mutually influential behavior patterns can further precipitate more rapid vicious cycles of accelerating negative mental models (e.g., mistrust, disliking) of one another. Where additional actors are implicated beyond 2 agents (e.g., small group; something a multi-agent platform like PsychSim is capable of), a negative evaluation of one member of a social network may affect evaluations of other members of that network (positively or negatively) depending on their relations (and behavior vis a vis self and others) and a re-evaluation of one’s model of those others. Indeed, a manipulation in an experimental condition built on a DCG base might also implicate “temporal patterns of change” (p. 5) in the patterns of interactions among agents (with or without agents-in-the-loop) following some manipulations.

Vallacher and Nowak (this issue) note that qualitative changes in “a system’s temporal pattern [e.g., moving from walking to running] are associated with changes in the system’s control parameters” (p. 5). We find this interesting given the opportunities within these systems for measuring and incorporating time (e.g., time to start an interaction, time to move towards others, time to react, “crunched time” across levels of a game, etc.) to similarly offer opportunities to understand and model human social behavior (and mimicking it with agents). This might afford a window on interesting patterns of human behavioral dynamics. A shift from “stranger” to “familiar” may be relatively sudden: It may covary with reductions in prediction error in interacting with this other. Perhaps, that is why skills in finding “common ground” are so important (and common) in meeting others. As reduced prediction error occurs, gaps between speech acts over time may become reduced (see Wheatley, 2019). Here, one potential “control parameter” may be a level of interaction “certainty” in predicting what might happen next and how to respond to our partner. Increased social interaction prediction that reduces variability in responding across people (strangers, friends) may be more likely to the extent one has deep and adaptive scripts and communication skills/strategies for responding in that (and a broad array of) specific contexts. With a shift from a human (or agent’s) model of other as “stranger” to a more predictive model of other as “familiar” a type of qualitative change in communication patterns in other ways seems possible (e.g., move from more formal to more informal language use). Social knowledge (e.g., norms, cultural scripts, etc.) based on learning/experience may enhance prediction confidence in one’s “look ahead” functions and confidence that one’s predictions will be actualized. We think that’s a potentially very interesting dynamic to observe not only in human interactions, but, at least some of these dynamics, might be quite feasible in human-agent interactions. How well human-agent dynamics map onto human-human dynamics at any point of time in technology/conceptual development is then an important question. From our vantage point it’s an iterative and exciting research question and program of research that ultimately “scaffold us up” to an instantiated model of the human-human dynamics of interest.

Feasibility of Backbones as Simulations and for Instantiations

In developing DCG “model systems” computational models, as we suggest above and in the target article, can provide a theoretical backbone of assumptions. These backbones can be run first as simulations to examine the effects of these assumptions without instantiating them in a game. Then, promising computational backbones can be used to drive the interactions of agents (and humans substituted for agents) in instantiated virtual game environments. PsychSim (e.g., Marsella, Pynadath, & Read, 2004) was developed initially as a simulation tool for modeling human social behavior. Then, given its simulation properties and promise for simulating social interactions with intelligent agents with theory of mind, Marsella and his colleagues began to use PsychSim’s simulation capacities to drive social interactions in virtual environments instantiated as 3-D games for military applications, such as language and culture learning of soldiers in Iraq (Rizzo, Difede, Rothbaum, Reger, Spitalnick et al., 2010). Given its effectiveness in that role, PsychSim was adapted for other applications.

Examples of backbone simulations.

Some of the commentators (e.g. Barsalou, this issue) asked us to describe the intelligent agent software in more detail. We are happy to do so.

PsychSim.

PsychSim (e.g., Marsella, Pynadath, & Read (2004); Pynadath & Marsella (2005)) is designed to generate plausible human behavior by operationalizing psychological theories about human behavior as boundedly rational computations) Each agent employs bounded decision-theoretic planning, allowing for the consideration of decision tradeoffs. In addition, agents can factor their beliefs about other agents into their own decision-making. In other words, they can possess the capacity for theory of mind reasoning. Marsella and colleagues have demonstrated a wide range of human behavior and decision making biases in these decision-theoretic approaches, including emotion (Marsella & Gratch, 2010, Si, Marsella, & Pynadath, 2010), social influence (Marsella & Pynadath, 2005), motivated inference (Ito, Pynadath, & Marsella, 2010) and framing effects (Ito & Marsella, 2011).

The basic computational infrastructure of these agents is recursively nested POMDP (Partial Observable Markov Decision Problems) models. The model of an agent includes its beliefs about the state, as well as its beliefs about other agents in the world, the actions it can take, an observation function, its preferences or goals and reasoning mechanisms. The particular state features that are instantiated depend upon the specific application. The beliefs about another agent, a mental model of another agent, is a fully encapsulated model of that agent that allows an agent to simulate that agent, predict its actions, to anticipate how its own actions may be perceived and responded to by that agent. This mental model of another agent can be incorrect, simplified or stereotyped.

The agent’s actions use transition probability functions to model the dynamics of those actions, capturing potential causal uncertainty of actions on following states (see Pynadath & Marsella, 2005). The observation function models its belief update based on the agent’s observations. The preferences (goals) include absolute and relative specification of goals, including minimization, maximization or achievement of some state feature or even some agent’s belief. The agent’s reasoning mechanisms encompass reactive as well as deliberative behavior using “lookahead” or simulation and simplifying assumptions (such as stereotypical mental models of others or bounds on the simulation).

PsychSim can also model messages (source, recipients, subject, and content), as actions used by one agent to influence another agent (or human in the loop) and can refer to any of these agent components mentioned above. Indeed, the software can be used to create a range of social simulations pertaining to how individuals and groups of individuals influence one another, alter beliefs, change behavior in response to those interactions and messages and so forth. In instantiating psychological assumptions about influence, PsychSim relied on extensive bodies of psychological research on influence (e.g., Cialdini, 2001; Abelson et al., 1968; Petty & Cacioppo, 1986), crunching these influence factors into a small set of key factors (e.g., consistency, self-interest, speaker’s self-interest, and views of the message source (e.g., trust, likability, affinity) in ways that could be mathematized to create a computational model of influence (for additional details see Pynadath & Marsella, 2005).

Virtual Personality Model.

Read and colleagues have been developing alternative simulation backbones that we have referred to earlier. One of these is referred to as the Virtual Personality (VP) Model (Read et al., 2010; Read et al., 2017). These computational models can simulate “what if” human interaction assumptions, such as the relative importance of chronic goals, especially biologically-inspired goals and goal systems, for underpinning traits and social inferences, such as traits. Conceptualizing situations (as goal affordances) in a similar “language” to that used to conceptualize persons (both involving goal activations) enabled us to suggest that both could be combined in the moment, to represent current goal activation for virtual personalities (e.g., Read et al., 2010). And, Read and his colleagues have been further developing those computational virtual personality/motivation/decision-making models in incorporating assumptions of within-person changes (e.g., in satiation, depletion) in interoceptive states that creates additional within-person variability by feeding into current goal activations (Read et al., 2017). These assumptions are all based on available existing research (e.g., derived in part from a variety of data/evidence sources such as experiments, observational research, surveys, and so forth) that warrants initial assumptions and encourages consideration of testing the implications of such assumptions for emergent patterns of behavior. The idea is to keep building onto these computational models in principled ways while affirming their utility for, for example, addressing enigmas (Read et al., 2017). But, the VP model as a simulation tool and also as a tool that could be used in a game, is certainly a “work in progress”.

There are many ways that the model can be enriched in the years ahead. For example, there should be a mechanism in the computational model by which meaning constructions (e.g., plot units of who is doing what to whom where how when and why) are specified that could affect current goal activations (and behavior). We are partial to, at present, hierarchical models offered by approaches such as interactive activation models (McClelland & Rumelhart, 1981; Rumelhart & McClelland, 1982) because they afford both “top down” and “bottom up” conceptualizations of how humans may make meaning, moment-to-moment. Here we concur with Barsalou (this issue) that “top down” and “bottom up” approaches are, of course, both needed for a comprehensive model. At present, we are separately working on our social perception and meaning computational model (Read & Miller, 1998) in testing its assumptions and utility for this kind of “module” to be added into the VP model to fill that gap.¹²

Simulations and instantiations.

The backbone simulations we’ve described can be run independently of any game within which they are instantiated. Such simulations provide runnable systems of assumptions that can grapple with change (e.g., learning; and how priors influence what happens and how it is construed next). In short, in terms of quantum mechanisms and generalizability (Barsalou, this issue) we may be in a good position with such backbone simulations because while no two situations may be exactly the same (and may now mean something quite different given the experience of all concerned) and much more in the world may be in flux, the assumptions of the backbone mechanisms that produce these complexities may remain constant or predictably variant.

Here a generalizability issue could be, does the system of assumptions instantiated in the simulation generalize? That is, while what is observed for whom, when, and why may differ, the system of assumptions underlying prediction in a given instance may continue to be generalizable¹³. We concur with Barsalou (this issue) that this may be more likely when we have grounded our simulations and instantiated games in the first place with representative data drawn from a range of methods and constraints (including deep description, in the moment deep observation, surveys, etc.) and also with a deep read and review of the available relevant literatures in social science, neuroscience, cognitive science, communication, biology, and so forth. And, certainly we agree with Barsalou (this issue) that we should spend more time building a successful “natural science” (p. 19). PsychSim has been instantiated in many games already, and we plan to examine the ability of the VP model for such instantiations in games within the next few years. Backbone simulations as this discussion suggests make a variety of meta-theoretical assumptions about psychological mechanisms (e.g. goals, states, and how they operate in producing decisions and behavioral outcomes over time given agents current state in an ongoing scenario -- with other agents contributing to that current state -- given particular contexts/affordances). A great deal of formative research needs to be conducted to better understand “in the moment” how individuals vary in how they construe behavior in representative and changing settings, sequences, contexts/situations. Formative research, including deep description and observation are critical to SRD.

Formative, basic research is needed but doable.

In the target article, we laid out some of the approaches we have used in the past to generate the formative research needed to adequately create the “gist” of representative moment-by-moment virtual environments. Often that type of detail is missing in much of our observational and other basic research today, and it is critically important. From our vantage point the feasibility of SRD depends most on researchers collecting the detailed in the moment descriptive work over time. We welcome other approaches that get at this moment-to-moment or more granular detail by context, within-persons. Here, we were excited by Barsolou’s (this issue) discussion of the importance of context-sensitivity (and the need for better measurement and conceptualization of same) for the replication crisis and for studying the complexities of human information processing and behavior: We, of course, concur. Barsalou’s (this issue) provides a delightful description of “an alternative (and classic) strategy for ecological assessment and causal analysis” (p. 4): In important ways the guiding principle is not all that different from our own for conducting formative research. Consistent with our own approach, it insists on starting with rich deep observation and qualitative and quantitative descriptive research. We think such approaches are critical and wish more researchers conducted such “deep description” research. It is indeed critical for conceptualizing backbones and fleshing them out for instantiation in representative virtual worlds. These approaches are highly compatible, and we believe can support one another. SRD, in our view, depends upon deep formative research, such as the research illustrated by the approaches of Barsalou (see his brief review of this exciting line of work in this issue) in developing what they refer to as the Situated Assessment Method (SAM). Simulation backbones drawing on a larger theoretical framework with more implementable detail might afford a runnable simulation that may inspire additional observation, hypothesis testing, and so forth.

Consistent with Barsalou (this issue), some of our research teams have been working towards measuring personality in virtual environments (Miller et al., under contract; Miller et al., under review). The methods employed by Barsalou fit extremely well with our own efforts to do formative research (getting at the goals, plans, resources, beliefs, etc. that may underpin individual differences in responses in given situations) so that we can, in the end, use that detail to constrain, advance, and develop representative DCG games as alternative measurements to self-reports, including within-person measures of trait-like and state-like behavior. We too are considering instantiating a range of behaviors and decisions in games (and their relative timing with respect to other agents), bodily and eye movement, approach/avoid physics and so forth to infer things like relative goal current activations, indications of expectancy failure (predictive coding), and other social concepts: It looks likely that Barsalou’s research program and approach could aid our own efforts in this regard. What fun! Perhaps future collaborations are possible? Indeed, that’s part of the point. SRD isn’t an “alternative” method in the sense that it aims to displace other methods. It relies on other methods, including deep observation. If anything, it only encourages (and perhaps provides more incentive to) researchers who do observational work to conduct more fine grained, moment-to-moment research of the organism in its more natural ecology (e.g., with other social creatures). That would be terrific for collectively moving psychology forward so that the whole (including representative game instantations) is greater than the sum of its parts.

Along similar lines, we concur with the broad strokes of Barsalou’s (this issue) suggestions regarding the importance of constraining the situation conceptualization challenge task. We do so by focusing on the BOI and the sequences leading up to it that are critical for the POI and in construing these sequences in terms of the components (e.g., in scripts, and how that also constrains meaning and behavior). In line with our approach, and also consistent, we think, with Barsalou, we have conceptualized narratives hierarchically, so that one way to think about narratives is that it is “narrative all the way down” to the plot unit (capturing behavior and sequence in the moment) all the way up to chunked narratives that are more biologically common (e.g., some of the master narratives for most humans are probably socializing young to survive; finding one’s place in the group/society; finding and sustaining an appropriate mateship, and so forth). Larger more abstract narratives may have more communality across cultures, with culture playing a larger and larger role in some of the more specific chunks for how one achieves goals and how normative beliefs about doing so, for example, affect ongoing sequences of actions, thoughts, emotional reactions, etc..

SRD is Feasible for Scalable Interventions

As Crano (this issue) notes, “in today’s hyper-competitive climate for research funds, studies that do not involve some appreciable degree of practical application are at a serious disadvantage” (p. 4), and that “the practically obligatory requisite of near-term impact of applicable knowledge is becoming ever more pressing” (p. 4). We certainly concur. As we and Mozgai, Hartholt, and Rizzo (this issue) note, virtual environments have reached a critical point where their potential utility in and for effective interventions is apparent. If our basic methods, not only in experiments, but naturalistic observational, new technologies (e.g., smart phones, sensors, bio-indicators, etc.), surveys, qualitative and quantitative research, and so forth can be brought to bear -- both for development and for constraining SRD, SRD may provide an efficient “common ground” for integrating and affording many of our most cherished goals of psychology (e.g., to describe, understand, predict, change/control). In doing so, such applied science, drawing on a basic science foundation, can as Crano this issue notes, help us address many of our challenges today (e.g., reliability, relevance, generalizability, etc.). In doing so, we are in a better position to make it abundantly clear not only why psychology as a science is so valuable today but where we can imagine traveling together, with greater common purpose, towards an even greater good, in the years ahead.

It Takes a Village

The difficult part of SRD is not the creation of the experimental groups, per se. Training psychologists to create experimental groups after the development of a DCG would be relatively straight-forward. It is the creation of the DCG base upon which those experimental conditions are built that is especially challenging. Many of the tools used are now feasible ones to learn and use for much of our field. But, the process for developing a default control group (DCG) requires both back-end (simulation modeling for the meta-theoretical backbones) and front-end (implementation) skill-sets that are unique and specialized, but where these experts could scaffold other team science collaborators and members. Psychologists building DCG need to be trained to interface with experts, not only within psychology (and other social scientists) but also other specialists (e.g., cognitive scientists, computer scientists/engineers, neuroscientists, game designers, VR specialists, etc. as needed for a given research project). In addition, many social scientists today, including psychologists, as suggested above, could contribute greatly with observational and other basic research to the construction of DCG.

Villaging.

Perhaps we are reaching the stage where this is one way for our field to go -- at least in some research areas. Much formative research is needed for developing games, for example (also see below). A community site in which people seek collaborators (and expert advice) for team science might be ideal: Perhaps, funding for sustaining such a site might be possible. One could then team up (do the formative research) to be part of a team where other researchers have the technical skills for game/app design for an implementation and can help guide the formative research process in terms of what is needed. Researchers could pre-determine authorship on different papers, and assuming those authors carry through on those commitments, such a process could benefit all authors with multiple publications (and first authors on some of those papers). In the process, researchers scaffolded into a game design process might decide it is so exciting and productive (and much easier than they thought) that that they themselves wish to develop these skills (or have their students learn these skills).

Instantiated review.

The rich findings we have in the field from a range of methods of various types are potentially part of the world we (and our meta-backbones and theories) are trying to describe, predict, explain and change both at the “between persons” level and the “within persons” level. As in the so called “hard sciences” a theory (and meta-theory embedded as “backbones” in SRD) ideally tries to explain what we know. Having instantiated it (e.g., in this case in a game) we can run new simulations and research to generate new testable hypotheses, to predict, and ultimately to change behavior. An instantiated review, represented in assumptions and content built into games could draw on other related efforts. Those efforts include proposing to build social inference engines drawing on broad sources of data (e.g., research findings, digital trace data, etc.) and with the help of statisticians (e.g., Pearl & Bareinboim, 2014) and computer scientists, creating better “what if” new machine learning models for inferences about which contexts where, when, for whom are likely to produce what patterns (Goroff, Lewis, Scheel, Scherer, & Tucker, 2018). In many of the hard sciences, large efforts (e.g., human genome) require working more as a community towards a common set of goals.

Scaffolding one another forward.

Scaffolding in the form of workshops, faculty and graduate training opportunities and fellowships, and other ways to provide technical assistance are needed. In science, new methods often require exceptional efforts from the field and funding agencies. For example, modern neuroscience today was made possible through the founding¹⁴ of the Society for Neuroscience in 1969, which fundraised from membership dues as well as grants from PNAS and the Sloan Foundation (SfN, 2019). Indeed, we contend that it is imperative for our field to develop a dedicated and sustainable center (or group of centers) with adequate interest and funding committed to constructing default control groups (and other resources) based on the needs of various researchers.

What is possible as “a village” is extraordinary. But, it will require much more teamwork in common goal pursuit than perhaps we are used to in psychology. Collaborative roles are many but we will have to find ways in our culture to value and reward them. The opportunities are exceptional, many risks have been mitigated, and the time is ripe for large federal and other funders to scaffold us and partnering fields into a new paradigm to advance a 21st century psychological science.