Measurement confounds in study on social media usage and adolescent life satisfaction

Billions of people engage with social media. Even small causal links with mental health outcomes can have enormous consequences. Orben et al. (1) recently published one of the largest (and certainly the longest) investigations of this topic to date. Their study is commendable in many ways. Unfortunately, their primary measure of social media usage has so many problems that we believe that we, as scientists, should be skeptical if not entirely dismissive of their conclusions.

Social media usage was assessed with a single question asked yearly as part of a much larger survey. The question read, “How many hours do you spend chatting or interacting with friends through a social web-site or app like [Bebo, Facebook, Myspace] on a normal school day? (1 = None, 2 = Less than an hour, 3 = 1–3 hours, 4 = 4–6 hours, 5 = 7 or more hours).” Let us disregard for a moment whether the participants (adolescents, aged 10–15) could realistically be expected to answer this question accurately. Rather, we want to focus on 2 other issues that are potentially even more problematic.

First, this question omits the weekend, when adolescents have the most free time and spend the most time on social media. For many adolescents, school days are largely occupied by school, extracurricular activities, and homework. These activities limit the amount of time spent on social media (of course, this is not equally true for all adolescents, which makes it important to control, which the authors did not do). This may account for the surprisingly low social media usage observed in this study. The modal time appeared to be “none” and the mean “less than an hour.” Surveying weekend usage would have provided a much more thorough and accurate assessment of the amount of free time that adolescents choose to spend on social media.

Second, the question limits social media usage to “chatting or interacting with friends,” which introduces several confounds. There are many things that adolescents do on social media. Interacting with friends is arguably the least concerning of these activities. A child who spends hours per day on social media but never interacts with friends “never” uses social media according to this question. If there is a child whose mental health we are worried about, it is that child, but they are not counted in this study. Indeed, rather than assessing social media usage, we suggest that this question is at least as much a measure of extraversion, social network size, and other related variables that are likely to predict, if anything, better mental health.

To be clear, the authors were working with archival data and we are not suggesting that the problems with this question are their doing. Nevertheless, we think the problems are profound and render the results of the study highly confounded. Indeed, considering how social media usage was measured, we would be surprised if the authors found any correlations with mental health. The fact that they (mostly) did not is not surprising and can easily be attributed to measurement error.