Tracing Physical Behavior in Virtual Reality: A Narrative Review of Applications to Social Psychology

Abstract

Virtual reality (VR) offers unique benefits to social psychological research, including a high degree of experimental control alongside strong ecological validity, a capacity to manipulate any variable of interest, and an ability to trace the physical, nonverbal behavior of the user in a very fine-grained and automated manner. VR improves upon traditional behavioral measurement techniques (e.g., observation and coding) on several fronts as data collection is covert, continuous, passive, and occurs within a controlled context. The current review synthesizes extant methods for tracing physical behavior in VR, such as gaze tracking and interpersonal distance measurement, and describes how researchers have applied these methods to understand important phenomena within the context of social psychology. To date, primary areas of application have included the assessment of social approach and avoidance, social evaluation and bias, and engagement. The limitations of behavioral tracing methods in VR, as well as future directions for their continued application and extension, are discussed. This narrative review equips readers with a thorough understanding of behavioral tracing methods that can be implemented in VR, their benefits and drawbacks, the insight they may offer into social processes, and future avenues of work for applying emergent technologies to research questions in social psychology.

Introduction

Behavioral measures have long been a gold-standard in social psychology, particularly when collected in a covert and continuous manner. Despite the benefits of such measures, they have traditionally been difficult to collect and quantify, especially when it comes to constructs that are enacted subconsciously or are subject to social desirability effects. Examples of such constructs include racial prejudice and bias, prosociality, and aggressive tendencies (Carlo & Randall, 2002; Fazio, Jackson, Dunton, & Williams, 1995; Harris, 1997; Wittenbrink, Judd, & Park, 1997).

First, many social constructs are difficult to assess in a covert manner that ensures the participant is unaware of what’s being measured. A prime example is the Implicit Association Test (IAT), the most commonly used method for assessing individuals’ implicit biases toward a given social group (Fiedler, Messner, & Bluemke, 2006; Greenwald, Poehlman, Uhlmann, & Banaji, 2009). The fast-paced nature of the test has resulted in the assumption that social desirability effects can be ruled out. However, users soon become explicitly aware of the purpose of the test; indeed, substantial evidence suggests that the IAT may be susceptible to voluntary influence (Fiedler et al., 2006). Generally, self-report measures have been criticized over their susceptibility to social desirability effects, and show relatively low correlations with implicit measures of bias (Hofmann, Gawronski, Gschwendner, Le, & Schmitt, 2005). Second, researchers have historically struggled to measure social phenomena in a continuous, fine-grained manner that allows for the examination of temporal change while remaining covert. For example, as an alternative to self-report approaches, many researchers have utilized chair-distance measurement as a proxy for bias. For instance, Goff, Steele, and Davies (2008) conducted a series of studies in which participants were instructed to arrange three chairs – one for themselves, and two for their race-varied conversation partners – then distance between chairs was measured, where greater distance reflected greater bias. This method yields a rough, aggregate measurement collected at only one timepoint, which does not allow for the study of temporal trends over the course of a research scenario.

In light of these limitations, Blascovich and colleagues published a seminal article in 2002 (Blascovich et al., 2002) that detailed the ways in which virtual reality (VR) could provide substantial methodological benefits as a tool for conducting research in social psychology, especially with regard to the precise measurement of nonverbal, physical movement behavior over the course of an entire scenario. In recent years, a body of studies has utilized such behavioral tracing – the fine-grained, nearly continuous measurement of physical behavior – to assess psychological processes in VR.

The goal of the current review is to synthesize extant methods for behavioral tracing in VR, and to explore how researchers have applied these methods to understand important phenomena within the context of social psychology. Rather than a comprehensive or exhaustive review, here we select examples from a variety of literatures including human-computer interaction, neuroscience, medicine, behavioral economics, and psychology, thereby illustrating the ways in which tracing methods in VR can offer important insights into the behavioral manifestations of social processes.

Virtual Reality as a Research Tool in Social Psychology

The arguments put forth in the 2002 paper by Blascovich and colleagues have served as the theoretical foundation for a significant body of empirical work that has since been conducted with VR. First, the paper argues that social psychological research outside the context of VR has continually been subject to a trade-off between experimental control and ecological validity. The most controlled experiments – often conducted in sterile laboratory environments with stripped-down variables – result in a significantly less life-like context, thus limiting mundane realism and ecological validity. On the other hand, field experiments, though high in ecological validity, are subject to myriad extraneous variables, observer subjectivity, and the challenge of quantitatively coding observed phenomena. VR presents a viable experimental paradigm where neither control nor ecological validity need be sacrificed. Second, VR allows for the manipulation of any variable imaginable, whether or not such a variable could be manipulated in real-world circumstances. This approach also allows for almost perfect replicability of any study, as it is largely pre-programmed and encapsulated within a virtual environment that can be shared and distributed to other research teams.

Finally, and most relevant to the current review, VR affords the ability to measure behavior of the user in a very fine-grained manner, or in other words, to trace the behavior of the user. To assess behavior within a field experiment, the typical approach is to code elements of the social interaction of interest in real time or via recordings. Processing and coding of such interactions for verbal and especially nonverbal content are time-consuming and resource-intensive. In a virtual interaction, however, physical behavior is already collected by the VR system that runs the experiment. For example, to accurately update the user’s visual perspective, the system must know how the user’s head is turning and moving at 60-plus instances per second. Such data can be easily collected from the system and used to create continuous tracings of physical behavior throughout the virtual interaction. These data can be processed, visualized, and analyzed in a variety of ways, which can lend crucial insights into the psychological processes unfolding within a given scenario.

Virtual Reality and its Affordances

Since its creation, VR has been assigned a plethora of definitions, but its core characteristics have remained the same. Essentially, VR amounts to a digitally-created environment that is experienced in an immersive way using specialized equipment. Though VR is sometimes defined to include non-immersive interfaces (e.g., screen-based multiplayer online games), this review will focus on the immersive versions of the technology. This type of VR is exemplified by several hardware systems currently on the market, including the Oculus Rift and the HTC Vive. In each of these systems, the user’s head orientation is tracked such that point of view changes appropriately as the head rotates. Head position may also be tracked such that a user can walk around within the virtual space. VR systems may include both orientation tracking and position tracking, or orientation tracking only. In addition, other body parts, such as hands, can be tracked such that their movement is used to control elements of the VR environment. Head and hand tracking are most typical, but the use of additional trackers for other body parts is also possible. In addition, multiple users can be tracked simultaneously in a shared VR environment.

While it is important that we preface this discussion by outlining the technical aspects of VR, from a psychological perspective what is most crucial is the experiential aspect of this technology. From the earliest uses of virtual reality, researchers and developers have agreed that the psychological experience of “presence”, or “being there” within the virtual environment is the core of a VR experience (Slater & Wilbur, 1997). Presence has taken on many definitions, but central to all these is the notion that one feels as if he/she is existing within the virtual environment as opposed to the physical environment in which he/she is actually situated. In other words, the digital environment becomes reality. While this concept is difficult to convey in writing, many individuals are introduced to it via a simulation in which they are asked to cross a narrow wooden plank many stories above the virtual ground. People are often surprised to find themselves afraid to cross the plank, and this apprehension is often accompanied by measurable physiologic stress (Meehan, Insko, Whitton, & Brooks Jr, 2002). There are a host of variables that have been shown to either boost or reduce presence in VR environments, including levels of immersion in and interactivity with the virtual world (Cummings & Bailenson, 2016) that can be capitalized on to create increasingly compelling VR simulations for research.

Crucial for social psychological applications of VR is the sister concept of “social presence.” Essentially, social presence refers to the notion that VR users experience other human representations as social entities, such that the user can perceive the virtual human’s mental states (Biocca, 1997). Social presence is the element that gives VR much of its power as a tool for studying interpersonal interaction. Like presence, there are several variables that contribute to higher versus lower levels of social presence and social influence in VR, including those related to the realism and agency of the virtual interaction partner and the degree of automaticity in the task being performed (Blascovich, 2002; Oh, Bailenson, & Welch, 2018). As VR continues to come of age, alongside artificial intelligence and other technologies, the ability to generate virtual humans that engender high levels of social presence is expected to continually improve.

Behavioral Tracing in Virtual Reality

At present, the analysis of users’ headset-tracking data (its location in space and/or its orientation) is the most common source of behavioral tracing data within VR-based research. Despite the capability of VR systems to also track the location and rotation of the user’s hands or other body parts, in general, the movement of one’s head is more readily interpretable for social psychological scenarios. In this vein, it is crucial to note that the interpretation of a user’s head movement (i.e., the underlying construct or psychological phenomenon represented by that movement) is completely dependent upon the context of the VR environment. For example, some environments use visual gaze as a mechanism for selecting virtual objects in lieu of using hand controllers (Verhulst, Normand, Lombart, Sugimoto, & Moreau, 2018). In such a scenario, visual gaze behavior would be interpreted in a very different manner than it would be in a scenario that elicits more naturalistic movement, such as a virtual social interaction. Further, across varying social scenarios (e.g., in a classroom filled with students vs. a one-on-one conversation), visual gaze behavior likely represents a variety of diverse underlying constructs. This context dependency can make behavioral measures difficult to validate. However, as we will later discuss, there are several mappings of body movements to psychological and interpersonal constructs that are relatively common and frequently reported.

Orientation Tracking

Head orientation is represented by the metrics of yaw, pitch and roll, each of which corresponds to an axis in 3-dimensional space. Yaw is the movement of the head from side to side, like shaking one’s head “no.” Pitch is an up-down movement, like nodding one’s head “yes.” Roll is the third axis, represented by bringing one’s ear from shoulder to shoulder. The combination of these three metrics (yaw, pitch, roll) from some origin point (usually looking straight ahead) pinpoints the orientation or rotation of one’s head. Often, the orientation of one’s head is used as a proxy for gaze direction. Essentially, an invisible ray extends out from the participants’ eyes to the center of their field of vision, and whatever object lies there is assumed to be the object in view. There is evidence that supports this approach, as head orientation has successfully been utilized as a proxy for social eye contact (Pfeiffer, Vogeley, & Schilbach, 2013; Rubo & Gamer, 2018). The underlying assumption is that users are looking straight ahead within their VR display, moving their eyes only for small changes in focal point, but moving their heads for larger changes in gaze direction. For this reason, it can be important to have objects of interest lie a certain distance from one another within the virtual world so that a buffer can be established. Readers will note that this is similar conceptually to eye tracking, but less exact. There have been attempts to integrate eye tracking equipment within VR headsets for many years, and a handful of studies have reported on eye tracking data obtained within VR (some of which will be reviewed here). Eye tracking VR equipment is becoming more available and robust, but is not standard in VR hardware at present. Thus, the majority of extant studies utilize head orientation as a proxy for gaze direction.

Orientation data can be collected continuously, but this can result in an unwieldy amount of data, meaning that many researchers sample these data less often than they are available (e.g., a few times per second). Even then, the data need to be reduced into a usable metric for analysis. Thus, most studies will calculate the percentage of time a certain object or virtual person was in the center of the participant’s view throughout the virtual scenario or during a segment of the scenario (e.g., Persky, Ferrer, & Klein, 2016; Wieser, Pauli, Grosseibl, Molzow, & Mühlberger, 2010). Other metrics include a count of the number of times, or the length of time a particular object was looked at (e.g., Gillath, McCall, Shaver, & Blascovich, 2008). Other work has focused on the variability of head movement throughout a scenario, posited to be a proxy for room scanning or attention (Won, Perone, Friend, & Bailenson, 2016).

Rotation data from hand tracking or from other body parts is rarely used as a behavioral measure, but this is likely to change as VR becomes more popular in the research arena and a wider assortment of research-oriented VR environments are developed. Again, the context of the virtual world will dictate the most appropriate measures, data reduction approaches, and interpretation of behavioral patterns in the data.

Position Tracking

Position tracking represents the location of the user’s head or hands within a three-dimensional space. Points along the X, Y, and Z axes represent movement in the left/right, up/down, and forward/backward directions, respectively. A variety of metrics can be obtained from positional coordinates.

The most common application of position tracking data in VR is as a metric of proxemics, which is defined as the interpersonal distance between individuals – whether or not one of those individuals is virtual (McCall, 2015). Proxemics is an inherently social measure, and arose out of the study of concepts related to personal space bubbles and social distance (Hall et al., 1968). Prior work has shown that the proxemics patterns that individuals follow in VR match with patterns followed in reality, such as approaching a virtual person more closely when facing their back than when facing their front (Bailenson, Blascovich, Beall, & Loomis, 2001, 2003).

Proxemic behavior is highly related to cognitive and affective variables such as social attention and interpersonal evaluations, making it particularly useful for social psychological studies (McCall, 2015). Even within the area of proxemics, however, different research projects have operationalized interpersonal distance in different ways. Minimum distance at which an avatar was approached is a frequently used proxemics metric, wherein a single point representing the smallest face-to-face distance between participant and interaction partner is reported. Other researchers have reported average distance maintained between the participant and virtual interactant. Both of these can be calculated within multiple segments of an interaction, but most commonly they are aggregated over an entire scenario. Some research teams have approached proxemics measures with more complex assessments combining both position and orientation. For instance, proxemic imaging (McCall & Singer, 2015) simultaneously assesses both interpersonal distance and gaze direction of both interactants. Indeed, individuals tend to give interaction partners more personal space when they are also making eye contact (McCall & Singer, 2015). These more thorough proxemics assessments can offer more nuanced insight into interpersonal distance as it relates to psychological processes and interpersonal interaction.

Beyond proxemics, there are other ways that position tracking data are used to understand psychological process. For example, a handful of projects have examined participants’ walking paths in space and used various metrics to quantify characteristics of the path. As tracking equipment becomes more accessible and less expensive, we anticipate the ability to track more body parts will be increasingly in reach, and will allow for more complex models of body movement that can be mapped to nuanced activities and nonverbal behavior patterns (e.g., fidgeting, posture, open versus closed body language) for additional insights into human behavior and psychology.

Constructs Quantified by Behavioral Tracing in VR

There are a variety of conceptual, psychological targets that researchers have aimed to quantify via behavioral tracing in VR. Here, we will outline some of the most commonly assessed constructs and metrics by which they have been operationalized. These constructs include social approach and avoidance, evaluation of a social other, and engagement in a situation or task. It is important to note that validation of these measurement techniques – i.e., establishing that a behavioral measurement is indeed reflecting what it is believed to reflect – is fairly rare. However, as replication is achieved across studies utilizing similar metrics to reflect a given psychological phenomenon, evidence for validity increases.

Social Approach and Avoidance

The construct most commonly quantified by behavioral process measures in VR is social approach and/or avoidance. Here, we broadly define social approach and avoidance as the tendency of an individual to avoid or engage in social interaction with others. For example, this may be quantified by an individual’s willingness to engage in eye contact, or an individual’s tendency to stand at a socially appropriate distance from their interactant. Without VR, such behaviors are difficult to precisely measure. Interpersonal distance in social settings has often relied on the use of confederates, for example, asking participants to tell an approaching confederate to stop moving once they begin to feel uncomfortable (e.g., Deus & Jokic-Begic, 2006; Uzzell & Horne, 2006), or the manual coding of videotaped interactions between participants and confederates based on approximate observed distances (e.g., Jones & Aiello, 1973; Remland, Jones, & Brinkman, 1995). The built-in orientation and position tracking features of VR, in combination with its ability to elicit social presence, underlie its use for studying social approach and avoidance. A variety of studies have aimed to characterize this construct in both clinical and non-clinical populations.

Studies focused on clinical populations have served two purposes, the primary of which is to better understand social behavior among individuals with a given diagnosis. Through these studies the research community has also learned about the properties of and influences on approach and avoidance behavior in VR, and how it varies among individuals with known patterns of social deficits or differences.

Individuals with social anxiety are an obvious target population for the study of social approach and avoidance. Indeed, two studies have quantified social avoidance in this population via gaze tracking during one-on-one social interactions (Dechant, Trimpl, Wolff, Mühlberger, & Shiban, 2017; Wieser et al., 2010); both observed that high-socially-anxious participants spent less time looking at the virtual conversation partner’s face, and were more likely to avert their gaze to the surrounding environment. Comparable patterns were observed among individuals with autism spectrum disorder (ASD; Raj & Lahiri, 2016). In addition, following the delivery of a VR-based social interaction training, ASD-affected individuals displayed greater subsequent increases in eye contact (Bekele et al., 2016). Studies with similar design have also been conducted among participants experiencing schizophrenic and paranoid ideation; here, greater presentation of symptoms was predictive of greater interpersonal distance maintained from an avatar during a simulated conversation (Fornells-Ambrojo et al., 2016; Park et al., 2009). Additionally, participants with schizophrenia displayed less change in interpersonal distance in response to the apparent emotional state of the virtual interaction partner (angry versus happy), perhaps indicating a deficit in cognition and/or social awareness (Park et al., 2009). As such, social avoidance in VR – whether measured by eye-tracking or proxemic distance – behaves as expected when assessed among individuals with known tendencies toward such avoidance.

Current findings suggest that interpersonal distance and eye gaze may both function as viable proxies for social approach and avoidance in relevant VR simulations. However, it is important to note that these studies have all employed dyadic social scenarios. Researchers who have aimed to generalize these findings to non-dyadic group scenarios, such as virtual cafeterias, have observed mixed results (Brinkman et al., 2011; Geraets et al., 2018). Going forward, VR will allow for more frequent simulation of multi-interactant scenarios, which will likely shed new light on our understanding and ability to predict behavior in group-based social settings among populations with and without known clinical deficits.

Social approach and avoidance studies have likewise been conducted among non-clinical samples. In fact, among healthy populations, this is a psychological construct for which validation has been well established. Two studies have validated the use of proxemic distance for quantifying social approach and avoidance by demonstrating that observed patterns in these measurements are in line with expected norms. On the most basic level, Kolkmeier, Vroon, and Heylen (2016) demonstrated that participants were more likely to move away (i.e., lean back) from a virtual conversation partner if he/she leaned toward them. Similarly, Hasler and Friedman (2012) demonstrated that Asian dyads maintained significantly greater interpersonal distance during social interactions compared to European and mixed-culture dyads, a pattern consistent with Eastern/Western differences observed in the real world (Hasler & Friedman, 2012).

Beyond validation-oriented work, a variety of behavioral tracing measures have been applied to explore social approach and avoidance in healthy samples. To explore the manifestation of sub-clinical levels of social anxiety in physical behaviors, Won et al. (2016) tracked the head rotations of students in a virtual classroom, which was quantified in terms of rotation variability, or “scanning” of the room. In this sample, greater scanning behavior was associated with greater self-reported anxiety about the virtual social partners in the room (e.g., higher responses to “In the virtual classroom, I wondered what the other students thought of me”). In another study, Martarelli, Borter, Bryjova, Mast, and Munsch (2015) utilized a proxemics-based approach to study the effect of parental weight status on children’s social avoidance. They measured the minimum distance reached between child participants and avatar children in a VR playground, and found that children of overweight mothers maintained greater distance from the avatar children with whom they were instructed to interact. In another example, Gillath et al. (2008) aimed to quantify prosocial approach behavior via proxemics. Participants were confronted with a struggling “beggar” (vs. a control avatar) at a virtual bus stop. Here, dispositional compassion predicted length of time spent looking at and amount of time spent nearby this avatar.

This group of studies demonstrates that social approach and avoidance are well-reflected in physical behaviors measured within VR scenarios, both gaze and interpersonal distance. Unlike traditional methods for studying social approach and avoidance, VR enables assessment to be conducted automatically and continuously. Additionally, as some above-mentioned work has already begun to capitalize on, VR allows for the assessment of behavior within environments that – though ubiquitous in the real world – would be difficult to simulate experimentally. Thus, VR allows for the study of behavior in key contexts where social behaviors are most often enacted (e.g., classrooms, parties, bars and restaurants). Going forward, this will allow for evaluation of interpersonal behavior within these socially important environments in ways that are controlled and precise.

Evaluation of a Social Other

There is a long history of research that has measured nonverbal behavior as a proxy for individuals’ evaluation of a social other. Here, we define evaluation of a social other to reflect an individual’s inwardly-held attitudes, opinions, or biases about another individual. In these scenarios, nonverbal behavior is typically posited to function as an implicit measure of these evaluations. In some contexts, these behaviors may reflect bias toward an outgroup member or members. Before the use of VR, behaviors of interest were usually observed or recorded naturalistically or in a laboratory setting, then translated into quantitative codes, such as the degree to which a participant smiled, made eye contact, raised his or her eyebrows, nodded his or head, and so on (e.g., Hall, Coats, & LeBeau, 2005). Manual coding of such interactions is labor-intensive and often does not reveal fine-grained or temporal sequences of behavior. Thus, VR has been introduced as a tool for studying physical behaviors as implicit measures of social evaluation.

Bias in virtual social interactions is most often quantified by proxemic distance, and this approach boasts well-established construct validity. On the most basic level, among a Russian sample, Menshikova, Saveleva, and Zinchenko (2018) observed that minimum distance between a participant and a virtual interaction partner was significantly greater when the avatar appeared to be of ethnic minority status. Dotsch and Wigboldus (2008) furthermore observed that greater distance maintained from minority-group avatars (here Moroccan avatars in a Dutch study) was predicted by participants’ implicit prejudice toward that minority group, assessed by an IAT. Finally, demonstrating a direct link with overt biased behavior, McCall, Blascovich, Young, and Persky (2009) found that greater proxemic distance from a Black avatar during a social introduction was predictive of more aggressive shooting behavior toward that avatar in a subsequent game. In a departure from these proxemic approaches, Persky and Eccleston (2011) linked gaze behavior to social bias; in a simulated clinical encounter between medical students and virtual patients, medical students kept the virtual patient’s face in view for a significantly smaller proportion of the interaction if the patient appeared to be obese (versus lean). When students were given information to reduce blame for the patient’s overweight, these biased gaze patterns were diminished. Taken together, these studies provide substantial evidence that proxemic distance, and likely gaze patterns, function as implicit measures of bias within social scenarios, and can be sensitive to prejudice reduction efforts.

VR-based behavioral tracing has also been used to reflect constructs that fall under other domains of social evaluation. For example, McCall and Singer (2015) utilized proxemic imaging to assess participants’ perceived fairness of a social partner. Participants were introduced to avatars who, via a game simulation, were portrayed as either “fair” or “unfair” players. Participants generally came closer to “fair” players, but those who chose to punish the “unfair” players were likely to come closer to those players and either look directly at or turn their back on the avatar. Here, proxemic imaging allowed for a more nuanced analysis of implicit social evaluation behaviors. In another study, Persky et al. (2016) used interpersonal distance to measure patients’ perceived stigma in a clinical scenario. Increases in interpersonal distance (i.e., the patient leaning away from a virtual doctor) over the course of the interaction were linked to patients’ reports of negative interpersonal reactions to the doctor. Finally, among romantic couples, Kane, McCall, Collins, and Blascovich (2012) observed greater interpersonal distance maintained among participants who perceived that their spouse was inattentive during a stress-inducing joint task. Here, physical distancing behavior was interpreted to be reflective of insecure attachment dynamics as a result of perceived inattention from one’s partner.

Taken together, these studies demonstrate that across diverse contexts, greater interpersonal distance can function as an implicit measure of negative evaluation of a social other. There is also evidence to suggest that avoidant gaze behavior may function similarly. Most notably, measures collected in VR are entirely unobtrusive and collected outside the conscious awareness of the participant, in contrast to other implicit measures of bias like the IAT. In addition, these measures are extremely fine-grained and allow for examination of temporal change, again in contrast to real-world approaches for measuring approach behavior as a proxy for bias, such as chair-distance methods (e.g., Goff et al., 2008; Macrae, Bodenhausen, Milne, & Jetten, 1994). It is notable, however, that only some nonverbal behaviors are readily accessible within VR, and many of these have yet to be analyzed in depth, such as head nodding and tilting. New measures, such as facial expressions, are expected to be incorporated into VR interactions in the future, which may open up additional avenues for behavioral assessment (Li et al., 2015).

Engagement and Attention

A more heterogeneous group of constructs, which can be loosely grouped under the umbrella of ‘engagement and attention’, have also been quantified via tracing measures in VR. Without VR, abstract constructs like engagement are difficult to assess via behavioral observation; indeed, there are few readily adaptable approaches to quantify how focused and on-task a research participant may be, or the amount of effort he/she may be expending. Thus, most existing measures of attention and engagement rely on participants’ performance on the given task or self-reported experience throughout the scenario. In contrast to this, VR enables the tracing of physical indexes of attention and engagement, which allows for better understanding of the behavioral manifestations of such constructs. Although extant studies have not yet applied such measurement techniques to purely social scenarios, these constructs are part and parcel of human interaction, and may serve as a precedent for future application to more inherently social constructs.

Behavioral tracing has been applied within virtual classrooms in order to quantify attention and engagement with educational tasks, and generally, this approach appears to be useful. For example, in a classroom scenario developed for Attention Deficit Hyperactivity Disorder (ADHD) assessment (Rizzo et al., 2006), Mangalmurti and colleagues (under review) measured the quantity of students’ head rotations, where greater head rotation was posited to reflect greater inattention to an assigned task on the classroom chalkboard. Here, head rotations partially mediated the relationship between ADHD symptoms and focused attention, and head rotation increased over the course of the task, indicating loss of attention over time. Other work has assessed the influence of interpersonal variables on classroom attention in a non-clinical population; one study (Jeong, Feng, Krämer, Miller, & Marsella, 2017) quantified head movement – leaning toward or away from the virtual teacher – as a proxy for engagement, and found that participants tended to move closer to a same-gendered instructor, and further away from an opposite-gender instructor. The authors speculate that this pattern was reflective of an in-group/out-group phenomenon wherein participants felt less comfortable and thus less focused with an opposite-gendered (i.e., outgroup) instructor.

To measure engagement in a different paradigm, Yaremych and colleagues (under review) utilized position-tracking coordinates obtained from parents as they moved throughout a VR-based buffet to select food for their child. They quantified the data in terms of the convolutedness of parents’ walking paths, and found that a more convoluted path was predictive of a reduction in how guilty parents felt about their child’s diet. Thus, in a food decision-making context, parents’ walking behavior may function as an implicit measure of perceived effort, or engagement, throughout the task.

Finally, in a behavioral economics example, Gürerk, Bönsch, Kittsteiner, and Staffeldt (2019) employed a virtual work scenario (i.e., a conveyor belt carrying cubes to be inspected) to study the effects of a highly productive vs. low productive virtual coworker on participant engagement with the work task. Here, head position was quantified in terms of movement between cube inspections, with less movement taken to represent greater productivity. Additionally, hand rotation was incorporated as a proxy for careful inspection of cubes (i.e., cubes turned such that all sides were inspected). Both behavioral metrics indicated that the highly productive coworker elicited greater participant engagement – more productivity and careful inspection – but only among competitive participants. Thus, in this context, head and hand movement may function as a useful proxy for effort expenditure and task engagement.

To summarize, it appears that gaze direction, positional movement, and sometimes hand movement may be reflective of engagement and attention across diverse contexts. However, the measurement of this construct within inherently social scenarios in VR remains limited. In addition to the above-mentioned contexts, behavioral measures of engagement and attention have the potential to inform research related to social attention, joint attention, social perception, cooperation, and related phenomena. Future work should continue to employ behavioral tracing to advance understanding of social engagement and attention.

Discussion

Researchers have utilized behavioral tracing in VR to quantify a variety of psychological constructs, including social approach and avoidance, bias and evaluation of a social other, and engagement. These studies demonstrate the exceptional potential of VR for measuring nuanced patterns of behavior that are often enacted implicitly, within key social scenarios that are ubiquitous in the real world. Additionally, the same metric can serve as a proxy for a variety of different constructs depending on the context of the VR environment. Proxemic distance from a virtual interaction partner, for example, could represent implicit bias, prosocial tendencies, or perceived stigma, depending on the scenario in which it is measured.

Limitations

Despite offering unique insight into implicit aspects of social processes, the measures discussed herein have many important limitations. First and foremost, construct validation is less common than it should be. In other words, the majority of studies presented here have not empirically demonstrated that the metric at hand is measuring the construct it is assumed to be measuring. The most well-validated approach presented here is the use of proxemic distance as a proxy for interpersonal bias; multiple studies have directly linked greater interpersonal distance to implicit and overt bias toward the target social group (Dotsch & Wigboldus, 2008; McCall et al., 2009). Gaze tracking as a proxy for social eye contact also enjoys substantial replication among non-clinical and clinical samples (Bekele et al., 2016; Dechant et al., 2017; Raj & Lahiri, 2016; Wieser et al., 2010). Beyond these, several studies are one-time experiments conducted within a specific context, and construct validity is largely assumptive. Although these assumptions often appear logical, it is important that validation takes places before findings can be considered reliable. Researchers often create unique VR scenarios to address their particular research questions, meaning that scenarios are infrequently reused and motivation to validate for future application is low. However, as VR gains in popularity, sharing of research environments may increase the likelihood that researchers perform validation work and that nonverbal behavioral findings will be replicated within a given context.

Another significant limitation to extant process measures is their highly aggregated nature. When examining proxemic distance, for example, most studies aggregate interpersonal distance over an entire interaction down to a single metric: average distance maintained between interactants (e.g., Brinkman et al., 2011; Park et al., 2009) or minimum distance reached between interactants (e.g., Martarelli et al., 2015; McCall et al., 2009). Such dramatic data reduction has the potential to mask meaningful effects (McCall, 2015). The challenge lies in reducing data to a manageable and interpretable form, while avoiding such dramatic oversimplification that meaningful patterns are washed out. McCall and Singer (2015) have attempted to ameliorate this problem by creating the proxemic imaging technique, which presents an encouraging alternative to the one-shot measures that are most often used. Other groups, rather than aggregating over an entire interaction, have aggregated across several smaller temporal blocks, thus beginning to unmask the dynamic nature of traced behaviors. For example, Mangalmurti et al. (under review) averaged head-rotation frequency within five blocks over the course of a VR classroom task; though still at a somewhat aggregate level, this approach revealed interesting temporal patterns. Similarly, Persky et al. (2016) averaged interpersonal distance between doctor and patient within multiple blocks tied to conceptual elements of the interaction, and also uncovered temporal trends in nonverbal behavior. It appears that if the care is taken to examine behavioral tracing data temporally, meaningful results can surface. Researchers should continue to utilize and develop approaches to avoid oversimplification of the rich data that VR provides.

Relatedly, VR simultaneously collects multiple data streams from the user. Current hardware frequently collects translational and rotation movement of the head and two hands, resulting in 18 continuous measures over time, and almost 2 million data points in a 20-minute use period (Bailenson, 2018). The expectation is that many other measures will be added as VR technology progresses (e.g., eye tracking, facial expression recognition), and that dyadic and crowd-based approaches will multiply these data streams (Moussaïd, Schinazi, Kapadia, & Thrash, 2018). However, as they currently exist, tracing measures typically fail to integrate these multiple data sources into a cohesive picture of the user’s behavior as time unfolds. Steptoe and Steed (2012) have argued the importance of methods that allow for the integration and synchronization of multiple data sources over time, emphasizing that temporal sequence must be taken into account in order for causal and/or reciprocal effects to be elucidated. McCall (2015) has argued a similar point, encouraging the use of time-series analysis for the longitudinal examination of behavioral data. VR certainly supplies the data for us to accomplish this objective, but most extant process measures are too aggregated to allow for the integration of multiple data streams or the disentangling of temporal patterns.

Finally, it is important to mention general limitations associated with VR use. First, cybersickness (i.e., motion sickness) can be a common side effect of VR use. Cybersickness varies depending on the characteristics of the user and the VR environment (e.g., those that require more movement and where one’s viewpoint is decoupled from physical movement), and is expected to diminish as VR systems become more sophisticated (Bockelman & Lingum, 2017; Porcino, Clua, Trevisan, Vasconcelos, & Valente, 2017). However, it remains a limitation that a non-trivial portion of the population is unable to participate in some types of VR experiments at present. Additionally, typically VR experiences are best administered in smaller time periods than other media (Kennedy, Stanney, & Dunlap, 2000; Yuan, Mansouri, Pettey, Ahmed, & Khaderi, 2018). This may present challenges for researchers who wish to simulate longer social scenarios. Lastly, due to its new and often exploratory role in research, many studies conducted with VR have low sample sizes, and thus, low statistical power. As in any other research area, replication and adequate sample sizes will be important for detecting meaningful effects in VR data.

Future Directions

There are several promising avenues of future research for the application of behavioral tracing in VR. First, there remains significant potential to integrate the measurement capabilities of VR with other measurement techniques. An example of this is the marrying of VR technology with neuroimaging approaches. The ability to simulate realistic social scenarios while simultaneously gathering neurologic data will undoubtedly prove useful in identifying the neural underpinnings of social phenomena (Parsons, Gaggioli, & Riva, 2017); thus, VR will likely offer unique benefits to the field of neuroscience. Alternatively, future work may wish to further incorporate the use of physiological measurements during VR scenarios, which has been an active approach in the VR space for many years (Jonsson et al., 2010; Meehan et al., 2002; Persky & Blascovich, 2008; Wiederhold, 2005). Indeed, this allows the marriage of continuous body movement data to continuous physiological marker data, enabling a fuller picture of participants’ internal states throughout social scenarios.

Additionally, machine learning techniques show promise for extracting meaning from the copious behavioral data collected by VR systems. These techniques offer the potential to strike a balance between complexity and interpretability. Machine learning approaches yield data-driven and nuanced models, thus avoiding the oversimplification problems described above. Still, results can be mapped onto theoretically meaningful constructs such that important trends can be deduced. For example, McGinnis et al. (2019) utilized a supervised machine learning algorithm to elucidate features of children’s physical movement throughout a frightening task that were predictive of internalizing disorder diagnosis. A combination of feature engineering and model selection revealed that certain characteristics of children’s movement, during a conceptually meaningful temporal block of the task, were predictive of diagnostic status with sensitivity and specificity comparable to existing methods. Future work should continue to explore the utility of machine learning approaches for classification, characterization, and prediction based on physical behavior.

It is also important to discuss the potential applications of employing VR users’ behavior as a real-time input variable, rather than an output variable alone. VR affords the opportunity to trace users’ physical behavior, and in real time, alter the virtual scenario according to that behavior. For example, researchers have studied mimicry in VR, wherein the physical movement of a virtual human (e.g., head tilting) mirrors the user’s physical movement, following a short delay (Fornells-Ambrojo et al., 2016). Indeed, mimicry by digital humans elicits comparable effects as mimicry by real-world social interactants (Bailenson & Yee, 2005); participants whose virtual interactant engaged in mimicry rated the interactant as more persuasive and remained more engaged throughout the scenario. Thus, capitalizing on VR user’s physical movement as a real-time input variable could be used as a tool, for example, to study basic characteristics of nonverbal behavior in a controlled manner, or more practically, to increase social engagement.

Another future avenue involves the incorporation of additional haptic components (i.e., touch) into VR systems. Currently, most VR systems include some level of haptic feedback, though this is usually limited to subtle vibrations in the hand controllers. The addition of more nuanced and varied haptic feedback (e.g., the ability to feel the shape and weight of objects visible in VR) offers potential for simulated scenarios to become more realistic, and also to leverage the fact that VR is capable of tracking the movement of multiple body parts. For example, Francis et al. (2017) simulated variations on the classic trolley dilemma in VR, wherein they incorporated a weighted human figure to be pushed, thus rendering the simulation more realistic and more likely to elicit ecologically valid behaviors. Though it was not integrated into that study, assessment of the body movement through which the figure is pushed (e.g., hesitation, force, speed) may be telling and could be assessed in similar future studies. Little extant work examines body-tracking data (e.g., hands, feet) and quantifies it meaningfully. Future work will certainly benefit from incorporating haptic elements into VR simulations to achieve a greater degree of realism, and this will also open doors for examining additional behavioral patterns over time. In addition to haptic components, olfactory elements (i.e., scent) will be increasingly incorporated into future VR systems. The use of scent, both as a tool for increasing realism (Baus & Bouchard, 2017; Munyan, Neer, Beidel, & Jentsch, 2016) and as an experimental manipulation, will likely open up new lines of inquiry for social scientists.

Finally, researchers are beginning to explore use of behavioral tracing in VR to predict outcomes with respect to both individuals and dyads. Examples given above demonstrate how VR tracing data can be used to tease apart participants with and without certain medical conditions (e.g., ADHD, internalizing disorders) with relative accuracy. Other work has shown that the examination of simultaneous behaviors among pairs can reveal characteristics about those dyads that would otherwise be impossible to predict; for example, Won, Bailenson, Stathatos, and Dai (2014) were able to predict creativity outcomes of dyads with high accuracy using behavioral synchrony measures fed into a machine learning algorithm. This ability to predict specific outcomes, among both individuals and dyads/groups, will grow ever stronger as more behavioral data points are integrated into VR use. It is worth noting that, on the flip side of this powerful research potential, is the notion that VR tracking data collected in non-research contexts raises serious privacy concerns. Researchers have noted that individuals may be identifiable based on patterns in their VR tracking data, and that personal behaviors (e.g., gaze patterns) may reveal mental processes and conditions that VR users would wish to keep private from tech companies, advertisers, and related entities. While this is a major ethical issue that the VR community grapples with (Bailenson, 2018), from a research standpoint, it highlights the value inherent in the behavioral data that is part and parcel of VR use. It is therefore of utmost importance that the research community continues to employ robust frameworks for consent, privacy, and security in this arena.

In conclusion, current applications of behavioral tracing to social psychological research have yielded new insight into phenomena such as social approach and avoidance, social evaluation, and engagement. However, there is still much work to be done with regard to validation, de-aggregation, and temporal examination of such measures. Notably, in addition to VR there are many associated emergent technologies (e.g., motion capture, augmented reality) that will allow for the collection and analysis of behavioral tracing data in the real world. As VR and related technologies continue to expand as research tools, data analytic techniques and their contextual applications should continue to expand in tandem.