Social Media and Well-being: A Methodological Perspective

Abstract

Due to the methodological challenges inherent in studying social media use (SMU), as well as the methodological choices that have shaped research into the effects of SMU on well-being, clear conclusions regarding relationships between SMU and well-being remain elusive. We provide a review of five methodological developments poised to provide increased understanding in this domain: (1) increased use of longitudinal and experimental designs; (2) the adoption of behavioural (rather than self-report) measures of SMU; (3) focusing on more nuanced aspects of SMU; (4) embracing effect heterogeneity; and (5) the use of formal modelling and machine learning. We focus on how these advances stand to bring us closer to understanding relations between SMU and well-being, as well as the challenges associated with these developments.

1. Introduction

For almost as long as people have been using social media, researchers have tried to understand the effects of social media use (SMU) on well-being. Despite the body of literature resulting from these efforts, clear conclusions regarding relationships between SMU and well-being remain elusive [1–3]. This is primarily due to two methodological factors.

First, are various methodological challenges inherent in studying SMU. Each social media platform offers its own set of constantly evolving features and interaction practices, which are then coupled with rapidly changing subjective and cultural contexts in which use occurs. This makes SMU a ‘moving target’ [4]. Moreover, the term ‘social media’ refers to a range of distinct platforms that each provide a unique set of affordances and content. This makes social media a particularly vague object of interest. In addition to these challenges, it has typically been difficult to access accurate data on SMU. Because SMU refers to a nebulous and ever-expanding set of actions that is difficult to accurately measure, it has been particularly difficult to generate robust evidence for SMU–well-being effects.

Second, are the various methodological choices that have shaped research into the effects of SMU on well-being. Until recently, studies have primarily relied on cross-sectional, self-reported data on aggregate SMU, and on between-person, verbal models of relationships between SMU and well-being [1]. These methodological choices imply that most research on SMU–well-being effects (1) generally does not permit causal inferences, (2) relies on biased usage data, (3) focuses on aspects of usage that provide only limited explanatory power over well-being, (4) ignores effect heterogeneity, and (5) is based on vaguely specified theories. These choices have limited the extent to which we can understand whether, how, and for whom SMU affects well-being.

In this paper, we evaluate five methodological developments in SMU research and focus on how these advances can bring us closer to understanding SMU–well-being relations, whilst also reflecting on the challenges they present: (1) increased use of longitudinal and experimental designs; (2) the adoption of behavioural measures of SMU; (3) focusing on more nuanced aspects of SMU; (4) embracing effect heterogeneity; and (5) the use of formal modelling and machine learning methods.

2. Moving on from cross-sectional designs

Most research on SMU and well-being has relied on cross-sectional designs [1]. Cross-sectional methods have helped researchers to identify important relationships and between-subjects differences, but such designs do not indicate temporal precedence. As such, they cannot support causal conclusions about effects of SMU on well-being, and have precluded the examination of within-subjects differences and temporal dynamics. There is, therefore, growing recognition of the need for more experimental and longitudinal research in this domain [1,5,6]. While such designs are well-established, they have accounted for only a small proportion of research on SMU and well-being [1].

Experiments involving SMU can be characterised along three lines: location (laboratory vs. in situ), platform-type (mock-up vs. actual), and focus (specific aspects of use vs. all usage). Recent studies have, for instance, used experiments to study the effects of social media content or interaction characteristics on well-being using mockups embedded in static vignettes [7], dynamic mockups involving fake content and interactions [8], or instructions that specify how participants should use actual platforms [9] or when usage in laboratory sessions can occur [10]. In contrast to these more controlled designs, others have used experiments that involve abstention from SMU for a specified period. For these designs, while findings are generally inconclusive [11], a more fundamental concern is whether testing for the effect of the absence of usage is the same as testing for the effect of usage itself.

While recent growth in the use of experimental designs is promising [1,11], there is also a need for improvements in the quality of experiments [2]. In addition to shortcomings in manipulations and control conditions, sampling biases and compliance remain challenges for field-experiments involving SMU. Moreover, given the algorithmically driven, “hyper-interactive and hyper-personalized” nature of many platforms [4, p. 487], the feasibility of manipulations involving content exposure in situ on actual platforms is limited. Open-source tools to create mock-ups of social media platforms with fake content, users, and interactions can circumvent some concerns, but there remains a need for increased rigour in experimental designs.

Alongside increased use of experiments, more studies are using longitudinal methods to investigate the temporal dynamics of SMU–well-being relations. Studies that follow a panel of participants over a period of years have, however, generally produced mixed results [12]. Other studies have begun to use intensive longitudinal designs and adopt the experience sampling method (ESM) to measure SMU and well-being throughout the day. Both ESM and traditional longitudinal designs enable the disentangling of within-person from between-person effects. Additionally, because ESM approaches harness smartphones to administer questions throughout the day, they enable the investigation of SMU and well-being dynamics in natural contexts [13–15].

3. Using behavioural measures of SMU

Almost all research on SMU and well-being has relied on self-reports of SMU [1]. Recent evidence suggests that there is a substantial discrepancy between behavioural data collected from usage logs and self-reports of media use [16]. This incongruence has led some to question the validity of self-reports and to call for the adoption of behavioural measures of SMU to supplement self-reports [16].

Recent advances have produced an array of tools that allow researchers to assess SMU through device or platform features [17]. Some approaches leverage native services to collect measures of usage duration or frequency [18], others collect such data via custom applications [19] or browser plugins [20], and others rely on data donations from participants [21]. Alternatively, some researchers have leveraged application programming interfaces (APIs) to access data on users’ actions [22] or partnered with social media companies to access proprietary data [23] normally only accessible to internal researchers [24]. These methods allow for more direct assessments of SMU free from many perceptual biases often associated with self-reports [18]. Given the foundational role that measurement plays in the validity of study conclusions, these behavioural measures are likely to yield more robust findings about relations between SMU and well-being.

While behavioural data allow for more accurate measures of usage, a number of technical, ethical, and conceptual challenges remain. First, various technical concerns limit the trust that can be placed in behavioural data.Very little research has been conducted to calibrate these tracking methods and determine the accuracy with which SMU is recorded. Most tracking tools, whether developed by social media platforms or third parties, are proprietary and do not provide information about the methods used [25]. A particular concern is the possibility that usage may be unknowingly misclassified or mis-timed [25,26] by third party tools not developed for research purposes [cf. 19].

A second challenge pertains to the types of data that can be collected. Extant research leveraging behavioural data generally involves the high-level construct of time spent on social media, which likely provides limited explanatory power over well-being. Given the range of affordances and content comprising SMU, aggregate measures of usage duration or frequency alone do not provide sufficient nuance to properly understand how (let alone whether) SMU is associated with well-being [27]. Behavioural data that only provide a measure of usage duration are, therefore, an improved method for measuring a relatively limited construct [16]. A central concern is that logs are, by definition, crude approximations of the sorts of data that social media platforms themselves collect about users’ behaviour. These proprietary data, however, are almost always inaccessible to academic researchers. A more robust understanding of relationships between SMU and well-being requires more open data practices from social media platforms.

Finally, the collection of behavioural SMU data introduces a number of ethical challenges. SMU data can contain sensitive information that participants may not want shared. While some study designs can involve explicit consent, others (e.g., data collection via APIs) might not. Participants may be aware that their behaviour is tracked by the platform, but not aware that it can be monitored by researchers [28]. Irrespective of the source, researchers have a responsibility to conduct research in accordance with accepted ethical standards [28]. This concern is particularly relevant given calls for increased transparency and the open sharing of study data [29]. Although data can, in theory, be anonymised for sharing, deanonymization remains a concern, especially for marginalized populations who are frequently statistical minorities [30,31]. Many forms of behavioural data contain personal identifiers that can easily link an individual with other variables (e.g., mental health) that are considered sensitive under most regulations [28,30]. When working with SMU data collected via tracking methods researchers need to be conscious of these, and other, ethical concerns and, when sharing study data, ensure that standards for de-identification are met [30].

4. Focusing on more nuanced aspects of SMU

To better understand the existence of, and mechanisms driving, relationships between SMU and well-being we need to use more nuanced measures focusing on how rather than how much social media are used. Building on research that investigates general behavioural patterns and contexts surrounding SMU, subjective experiences and reactions to SMU, and the media ecologies in which SMU is situated, there is also a need to focus on specific content and types of SMU.

While most studies still draw on high-level measures of SMU duration or frequency [1], a growing proportion now focus on more nuanced aspects of SMU [4,32]. Many studies, for instance, distinguish between active use, which involves “targeted one-on-one exchanges” and passive use which involves “monitoring the online life of other users without engaging in direct exchanges with them” [33, p. 33]. Others investigate SMU through the lens of addiction, habits, or intensity (see [34]). However, some argue that these frames do not offer sufficient insight into the range of engagements that social media enable [35] and that we need measures that better account for the diversity of social media content, interactions, and user responses [36].

Many conceptualizations of SMU are platform-specific [37]. As a result, findings may not generalize as platforms launch or remove features [4,37]. A greater focus on use of specific social media elements [4] such as profiles, networks, streams of information, or messages will help to generate evidence that is robust against platform changes. These elements represent relatively stable components that tend to shape users’ schemas for SMU in general [4]. Focusing on elements, among other factors (e.g., content, individual characteristics, and contexts), will force a greater emphasis on the theoretical mechanisms driving proposed relations between SMU and well-being and avoid over-pathologizing everyday behaviour [38,39].

5. Embracing effect heterogeneity

Most research shows generally negligible effects of SMU on well-being [2,3]. Some argue that this may be due to the predominant use of nomothetic analyses that aggregate data across individuals to arrive at group-level estimates [13,15]. These types of analyses assume that SMU–well-being effects are relatively homogeneous across individuals. Recent work that has used an idiographic approach to investigate variation around average effects at an individual level suggests that this assumption may be ill-founded [13–15]. By analyzing intensive longitudinal data, which are necessary for these types of analyses, studies have found that the association between SMU and well-being varies substantially in both direction and magnitude between individuals [13–15,40].

While promising, there are two primary criticisms of current idiographic research on SMU–well-being relations. First, extant research has relied on high-level measures of SMU. Thus, future research should combine the idiographic approach with usage measures for specific social media elements and content to provide further insight into the complexities inherent to SMU and well-being [27].

A second limitation of idiographic analyses of SMU–well-being relations is their inability to support inferences from an individual to a targeted population [41]. Johannes et al. [41] argue that, to properly study the variation in average effects, rather than focusing on individual users, the field needs to develop principles to quantify, interpret, and explain variation in average effects, and identify moderators of SMU–well-being relations. These authors, accordingly, outline a principled approach to investigate SMU effect heterogeneity.

Irrespective of the approach—idiographic analyses or moderator analyses—research into SMU–well-being relations needs to acknowledge the variation around average relations and embrace heterogeneity as a core characteristic of SMU effects. Idiographic approaches can describe the expected variation from person to person around the average effect, help to identify who might be more susceptible to effects, and point to potential moderators of interest. But this is not sufficient on its own. We also need to identify moderators that can systematically explain effect heterogeneity in a way that can be generalized to targeted populations [41].

6. Application of formal modeling and machine learning techniques

A final methodological advancement in SMU and well-being research is the increased deployment of both formal modeling and machine learning-based analyses [42–44]. Although its use within SMU research is still in a nascent stage, formal modeling has already proven useful to better understand how reward learning mechanisms contribute to SMU and may underlie some forms of compulsive SMU [45]. Formal models have also served to elucidate the behavioral reinforcement mechanisms underlying polarization and moral outrage on social media [46], highlighting interactions between platform design and social/cognitive processes that serve to amplify the expression of negative emotion online [47]. Future work using formal modeling to investigate SMU and well-being stands to benefit from increased availability of tutorials and tools for developing formal models [48], and from greater integration between models of individual-level, and macro-level processes [49].

Machine learning analyses at the intersection of SMU and well-being are now widespread, often seeking to predict individual differences in well-being from social media data. Notable recent work has leveraged these analysis techniques to investigate how “follow” networks relate to mental health [50], and how dictionary and data-driven language processing methods can reveal differences in subjective well-being across geographic areas [51]. Although these machine learning approaches are promising, theoretical development in this area is still sparse, stymied by high variability in both pre-processing and analysis techniques, and in construct operationalization [52]. Future progress in this area is contingent on balancing predictive and explanatory power in model development [44], and increasing rigor in both validation and reporting standards.

7. Conclusions

Early research into SMU–well-being effects primarily relied on cross-sectional, self-reported data on aggregate SMU, and on between-person, verbally articulated models of the relationship between SMU and well-being. As we have highlighted, the last few years have seen a number of important methodological advances that will support the development of a more robust understanding of whether, how, and for whom SMU affects well-being. In most cases, however, these advances have received only limited adoption [1]. We are optimistic that a critical mass of researchers will come to use many (or all) of the methods we have highlighted to investigate SMU–well-being relations. Above all, given the diversity in data, skills, and perspectives needed to investigate SMU, alongside the adoption of the methods discussed in this review, and the appropriate use of open and transparent research practices [29,53], research into SMU and well-being requires more interdisciplinary and industry collaboration to progress.