Reply to Foster and Jackson: Open scientific practices are the way forward for social media effects research

Research investigating the effect of new technologies on adolescents is more often characterized by media hype than sound science. We therefore welcome Foster and Jackson’s (1) consideration of this research area’s measurement practices because we believe a critical mindset benefits academic, civic, and industry stakeholders. While the authors raised important questions, however, the conclusions drawn by their letter are left unsupported by the available scientific evidence.

First, Foster and Jackson (1) posit that our findings are confounded because adolescents reported only weekday social media use, observing the engagement was “surprisingly low” (1). The amount of usage reported in our data were, however, in line with the United Kingdom’s telecommunications regulator’s annual report (2). Measurements of weekday and weekend engagement with technologies are also highly correlated (e.g., r = 0.72) (3), especially when viewed in light of correlations between estimates acquired using different measurement methods (values of r ≤ 0.18) (4). Furthermore, evidence suggests that associations between weekday use and well-being are equivalent, or more negative, than those associations examining weekend use (4). There is therefore no evidence that weekday assessments substantially challenge our (5) conclusions.

Second, Foster and Jackson (1) raise concerns about the self-report social media engagement measure’s framing. This is a formattable measurement challenge, and we noted our own dissatisfaction with relying on questionnaires in our paper (5) as they “only partially reflect the objective time adolescents spend engaging with social media.” With that understood, it is incorrect to conclude that this measurement introduces confounds. Like all large-scale cohort studies, the measures used in the Understanding Society dataset are extensively tested, revised, and harmonized, using innovation panels, interviews, and surveys, to adapt them to ever-changing social environments (6). No self-report measurement is perfect compared with the ground truth (7, 8), but there exists no evidence that this particular instrument is any more or less reliable than those used in other large-scale datasets (4).

In conclusion, we (5) report well-evidenced inferences about the enduring effects of social media engagement by pairing a robust analytic approach with one of the “best-quality datasets informing vital research in this area today” (5, 6). We agree with Foster and Jackson (1) that measurement practices deserve scrutiny but strongly believe that any given concern is only valid insofar it is supported by data. Because this was not provided, the reply is best understood as an opportunity to reflect on the value of scientists sharing data, materials, and code underlying their research inferences. This is not standard practice for those who routinely make extreme claims about technology effects (9) but is necessary for “independent scientists, policymakers, and industry researchers [to] cooperate more closely” (5). Longitudinal, transparent, and reproducible work is the only way forward for scientists seeking to promote meaningful and actionable technology effects research.