Associations between Self-Reports and Device-Reports of Social Networking Site Use: An Application of the Truth and Bias Model

Abstract

People are generally poor reporters of time spent using digital technology. Advancing smartphone features, such as the iOS Screen Time application, allow researchers to obtain more objective measurements of digital technology use. Truth and Bias models were used to test how self-reported social networking site use aligns with device-reported use as recorded by the iOS Screen Time app (N=1585). This study explored use across four major platforms (Facebook, Instagram, Twitter, Snapchat) and examined how individual differences moderate biases in reports. Participants overestimated their use for all platforms at comparable levels. Moderation by individual differences was not consistent. These findings add to the growing call from researchers to utilize assessments other than self-reports in measuring digital technology use.

The sharp rise of social networking site (SNS) use (Perrin & Anderson, 2019) has triggered a flurry of research about the predictors, correlates, and outcomes of use. Most of this research has relied on self-report assessments of time spent online (Kaye et al., 2020). As self-reports are inaccurate and systematically biased (Deng et al., 2019; Ernala et al., 2020; Haenschen, 2020; Junco, 2013; Scharkow, 2016; Sewall et al., 2020; also see Orben & Przybylski, 2019), other modes of data collection are needed (Ellis, 2019; Scharkow, 2019). Recent smartphone developments, including applications such as iOS Screen Time (e.g., Davidson et al., 2020; Ellis et al., 2019; Ohme et al., 2020; Shaw et al., 2020; Sewall et al., 2020), can simplify the capture of more objective SNS data. This study aimed to replicate previous studies by applying Truth and Bias models to examine discrepancies between self-reports and device-reports of SNS use as recorded by the iOS Screen Time app. We also examined different SNS platforms and tested several individual characteristics as moderators of discrepancies between self-reports and device-reports.

Inaccuracies in Reporting SNS Use

Activities that are frequent and habitual are harder to remember correctly (Schwarz & Oyserman, 2001; also see Prior, 2009). One such activity is smartphone use (Oulasvirta et al., 2012), and, by extension, SNS app use. People tend to overestimate their overall SNS use (Deng et al., 2019; Sewall et al., 2020). Most previous studies have either aggregated SNS use across platforms or focused on a single platform, with two exceptions. One study found greater accurate recall for Twitter compared to Facebook (Junco, 2013), whereas the other found greater accuracy for SNSs such as Facebook compared to SNSs such as Twitter (Scharkow, 2016). These discrepancies, as well as the omission of the more popular Instagram and Snapchat platforms, highlight a need for additional research.

Testing each platform separately can better ascertain whether reporting accuracy varies by platform. Recall accuracy may depend on habitual checking and frequent use. For example, as Snapchat and Instagram are used more frequently than Facebook and Twitter among college students (Massey et al., 2021), they may be subject to greater inaccuracy. If platforms do notably differ in recall accuracy, then this has important methodological implications for associations with use. If Instagram and Snapchat are less accurately recalled than Facebook and Twitter, then self-reports of Instagram and Snapchat use may be less reflective of “real” use and associations with key outcomes may be less valid.

This study applied Truth and Bias models (West & Kenny, 2011) to examine discrepancies between self-reports and device-reports of use for each Facebook, Instagram, Twitter, and Snapchat. These models are advantageous as they allow researchers to quantify directional bias (the degree to which use is under- or over-estimated) and tracking accuracy (the degree of alignment between self-reports and device-reports). Assessing these components can yield key methodological insights. If there is evidence of directional bias in which individuals overestimate use, then claims of alarmingly high SNS use may be incorrect. If there is weak evidence of tracking accuracy, then associations between self-reports of SNS use and outcomes may be inaccurate, as self-reports of use may not be an acceptable proxy for actual use (also see Davidson et al., 2020).

Moderation by Individual Differences

Individual differences may also predict biases in reported SNS use. An examination of characteristics that may predict biases is needed as conclusions may change based on key factors (Scharkow, 2016; Sewall et al., 2020). For example, differences in relations between SNS and Internet use with age and gender are attenuated or change when objective reports are used (Scharkow, 2016). Thus, the validity of conclusions drawn from self-report measures may vary depending on characteristics of a recruited sample. This study tested age, gender, frequency of use, conscientiousness, and frequency of Screen Time checking.

Younger adults may be less accurate than older adults in recalling texting (Vanden Abeele, et al., 2013), Facebook use (Ernala et al., 2020; but see Haenschen, 2020), and Internet use (Scharkow, 2016). Discrepancies may be explained by younger adults more frequently engaging in digital behaviors (cf. Vanden Abeele et al., 2013). Likewise, heavier SNS users are less accurate than lighter users in recalling use (Ernala et al., 2020; Sewall et al., 2020); heavier mobile phone and internet users underestimate use whereas lighter users overestimate use (e.g., Scharkow, 2016; Vanden Abeele et al., 2013; but see Deng et al., 2019). Although there is some evidence that women may commit more estimation errors when recalling Facebook use (Ernala et al., 2020), the opposite was observed for Internet use (Scharkow, 2016).

Conscientiousness and frequency of iOS Screen Time app checking have yet to be assessed as moderators. Those who check their Screen Time app can obtain reports of how often they use their SNS apps, which may facilitate more accurate self-reports of use. Individuals higher in conscientiousness are careful, detail-oriented, and dependable (John & Srivastava, 1999), and may more carefully consider and recall their behavior when completing self-report assessments (Ottenstein & Lischetzke, 2020). Digital technology research often relies on college student convenience samples, which vary in conscientiousness (Corker et al., 2017). More conscientious students complete surveys earlier in the semester (Witt et al., 2011) and higher conscientiousness is associated with greater survey completion (Brüggen & Dholakia, 2010). If conscientiousness emerges as a significant moderator, then this suggests that researchers must consider the validity of self-reports of SNS use in the context of the conscientiousness of the recruited sample.

The Current Research

We expected that participants would overestimate their use of Facebook, Instagram, Twitter, and Snapchat (i.e., self-reports would be greater than the corresponding device-reports recorded by the iOS Screen Time app). These analyses replicate previous research (e.g., Sewall et al., 2020) and extend upon it by exploring if effect sizes notably varied by platform. We also expected that younger participants, heavier users (i.e., through a quadratic effect), less conscientious participants, and those who check their Screen Time app less frequently would be less accurate in their recall. Due to mixed results in previous research, we treated the moderation analyses by gender as exploratory.

Method

Participants

Student iPhone users (N=1585) were recruited from two universities: one large Midwestern university (n=826; 62% female; M_age=19.29, SD_age=1.20; 66% White, 5% Black/African-American, 20% Asian/Asian-American, 3% Hispanic/Latinx, 7% Multiracial/Other) and one large Southwestern university (n=759; 75% female; M_age=20.81, SD_age=4.13; 24% White, 6% Black/African-American, 44% Asian/Asian-American, 14% Hispanic/Latinx, 12% Multiracial/Other). Recruitment site was controlled for in analyses due to demographic differences (see supplement). For both sites, an online advertisement was posted about the study. Interested students were able to complete the online study for partial course credit. Although the study was open to both iPhone and Android users (additional n=371), we limited our analyses to those who had an iPhone due to our focus on the iOS Screen Time app.

Measures

SNS Use

Participants reported whether they used Facebook, Instagram, Twitter, and Snapchat. If they answered ‘yes’, they were directed to questions about their frequency of use for each platform. Using an open-ended response format, participants were asked for each platform, “Approximately how many minutes do you spend per day using the [platform] app on your mobile phone, overall?”. Close-ended questions were also asked (see supplement), but we focus analyses on the open-ended questions as the response format was in the same metric as the device-reported questions. To provide better context on participants’ reliance on using apps to access the different platforms, participants were also asked about using alternative methods to access the platforms (see supplement).

For iOS Screen Time reports, participants were provided instructions on how to access their Screen Time app on their iPhone. Participants were then asked, “Over the last 7 days, according to your iPhone, how many minutes do you spend on the [platform] app?”. At the time of data collection, the iOS Screen Time report readily provided this information for the past seven days. Responses were divided by seven to obtain an average daily assessment of app use. During the study period, the iOS Screen Time report was available to participants with iOS version 12 or higher. We did not limit recruitment to these individuals due to alternative aims of the study. Participants had the option to leave the device-report items blank if they could not report (1%), although we cannot say with confidence how many did not report because they lacked the proper software compared to not reporting for alternative reasons.

Screen Time Checking

Participants were asked if they regularly used their Screen Time app, and indicated Yes/No. If ‘Yes’ was selected, they were further asked, “Approximately how many times per week do you check your screen time usage?”, with an open-ended response. Responses were divided by seven to obtain an average daily assessment of Screen Time app checks.

Conscientiousness

Participants completed the Big Five Inventory, including the 9-item conscientiousness subscale (John & Srivastava, 1999). Responses were made on a 5-point Likert-type scale (1=Disagree Strongly, 5=Agree Strongly; α=.78).

Analysis Plan

Truth and Bias models (West & Kenny, 2011) examined patterns in participants’ bias and accuracy of self-reporting SNS use. The model posits that judgments of phenomena (e.g., self-reports of Facebook use) are driven by the truth (e.g., device-reports of Facebook use) and different forms of bias (e.g., a tendency to over- or under-estimate Facebook use). Prior to regressing the judgment variable on the truth variable, both the truth and judgment variables are grand-mean centered using the mean of the truth variable. The effect of the truth variable on the judgment variable consequently captures tracking accuracy (i.e., the rank-order consistency between the truth and the judgment; how well participants’ self-reports of use align with device-reports), and the intercept from this analysis captures directional bias (i.e., the mean-level difference between the truth variable and the judgment variable; negative scores indicate under-estimation and positive scores indicate over-estimation). Moderators of directional bias and tracking accuracy were added by including their corresponding first-order effect (testing if the degree of directional bias depends on the moderator) and their interaction with the truth variable (testing if the degree of tracking accuracy depends on the moderator). Additional analyses used polynomial regression to test whether heavier SNS users evince more bias.

Robustness of the results was assessed by also running analyses: a) correcting for heteroscedasticity as the assumption of homoscedasticity was violated in all analyses; b) restricting the model to subsets of observations in which neither the objective or self-reported amount of platform use was equal to zero; c) estimating the model via robust regression using an M-estimator; and d) estimating the model after removing influential observations (as defined by Cook’s distance and the guidelines of Chatterjee & Hadi, 1988). Interpretation of the results is limited to findings robust across analyses to ensure that study conclusions are not vulnerable to analytic decisions. Due to the large number of tests, we adopt a p-value of .01. Prior to running analyses, four highly improbable responses were removed from the data (SNS use exceeding 20 hours a day, checking Screen Time app 120 times a day). Analyses including these values did not change the substantive pattern of results unless otherwise noted. Data, analyses, and the supplement are available at: https://osf.io/t5bjx/

Results

Descriptive information is in the supplement (Tables S1-S3). For all platforms, participants tended to overestimate use (Figure S1).

Truth and Bias Models

Results for Truth and Bias models including age, gender, conscientiousness, and Screen Time checking as moderators are in Table 1 (also see Tables S4-S7). Although tracking accuracy was comparably strong for each platform (Facebook: β=0.46; Instagram: β=0.46; Twitter: β=0.50; Snapchat: β=0.43), participants nonetheless overestimated their Facebook use by 14 minutes, Instagram use by 28 minutes, Twitter use by 25 minutes, and Snapchat use by 27 minutes. When applying the more stringent p-value across sensitivity checks, there was no consistent moderation by individual differences for Facebook, Instagram¹, Twitter, or Snapchat². Quadratic analyses tested if estimates of use varied depending on frequency of device-recorded use (Table 2; Tables S8-S11). Findings were only consistent for Facebook³. Follow-up analyses examined directional bias and tracking accuracy at lower and higher levels of Facebook use (higher=1 SD above the mean; lower=.25 SD below the mean due to skewness). Directional bias was stronger for heavier Facebook users (b=43.10, p<.001) compared to lighter users (b=9.04, p<.001). Likewise, tracking accuracy was weaker for heavier Facebook users (b=0.71, p<.001) compared to lighter users (b=0.92, p<.001)⁴.

Table 1.

Results for Truth and Bias Models

	Model Type
	Facebook		Instagram		Twitter		Snapchat
Predictors	b	SE	b	SE	b	SE	b	SE
Average Bias and Accuracy
Avg Directional Bias	13.94**	1.41	27.50**	1.52	24.92**	1.63	27.11**	1.61
Avg Tracking Accuracy	0.53**	0.08	0.69**	0.05	0.83**	0.06	0.68**	0.05
Moderators of Directional Bias
Site of Data Collection	−1.05	1.30	0.68	1.45	−1.44	1.60	1.76	1.51
Gender	−0.90	1.40	−3.53* c, d	1.52	−3.70* c, d	1.62	−1.95	1.53
Age	0.37 c	0.36	0.06	0.64	−0.65	0.65	−1.96* a	0.83
Conscientiousness	−1.38	1.96	−6.26**b	2.17	−4.74 c, d	2.42	−2.30	2.23
Screen Time Check	2.17 c	4.09	−1.30	2.73	0.08	2.41	4.25	4.37
Moderators of Tracking Accuracy
Site of Data Collection	0.07 c	0.04	0.03	0.04	0.00 c, d	0.06	0.01	0.04
Gender	0.27** a, d	0.07	0.15**c, d	0.05	−0.03 c	0.06	0.05	0.04
Age	0.01 c	0.02	0.04	0.02	0.01	0.02	−0.01	0.03
Conscientiousness	0.17* a, c, d	0.07	−0.05	0.07	0.05	0.07	0.01	0.07
Screen Time Check	0.39 c	0.38	−0.01	0.11	0.08	0.17	0.23	0.21

Note. Site of Data Collection was controlled for in each analysis and effect-coded, such that −1 = the Southwestern University and 1 = the Midwestern University. Gender was effect-coded, such that −1 = women and 1 = men. All continuous predictors were grand-mean centered.

^“a”denotes that the significance of the coefficient changed once heteroscedasticity was corrected.

^“b”denotes that the significance of the coefficient changed when analyses were restricted to those participants who used or had reported that they had used SNSs.

^“c”denotes that the significance of the coefficient changed when a robust variant of regression analysis was used.

^“d”denotes that the significance of the coefficient changed when influential observations were removed from the analysis.

^*p < .05.

^**p < .01.

Table 2.

Results for Truth and Bias Models Testing Quadratic Effects for Frequency of Use

	Model Type
	Facebook		Instagram		Twitter		Snapchat
Predictors	b	SE	b	SE	b	SE	b	SE
Covariates
Site of Data Collection	−1.941 d	1.147	−0.346	1.362	−1.937	1.469	2.179 d	1.382
Bias and Acc Terms
Directional Bias	16.545**	1.177	29.765**	1.402	27.583**	1.524	31.212**	1.507
Linear - Track Acc	0.876**	0.070	0.706**	0.050	0.950**	0.070	0.858**	0.058
Squared – Track Acc	−0.002**	0.000	−0.001* b, d	0.000	−0.001* a, b, c	0.001	−0.002**d	0.000

Note. Acc=Accuracy. Site of Data Collection was effect-coded, such that −1 = the Southwestern University and 1 = the Midwestern University. Truth variables were grand-mean centered prior to computing the quadratic terms. Superscript of

^“a”denotes that the significance of the coefficient changed once heteroscedasticity was corrected. Superscript of

^“b”denotes that the significance of the coefficient changed when analyses were restricted to those participants who used or had reported that they had used SNSs. Superscript of

^“c”denotes that the significance of the coefficient changed when a robust variant of regression analysis was used. Superscript of

^“d”denotes that the significance of the coefficient changed when influential observations were removed from the analysis.

^*p < .05.

^**p < .01.

Supplemental Analyses

Due to the ongoing debate on the relation between SNS use and well-being (Meier & Reinecke, 2020), we tested how device-reported and self-reported SNS use relate to depressive symptoms, life satisfaction, and self-esteem (Table S16). Although greater self-reported use (most consistently Twitter use) was often observed to have small correlations with poorer well-being, these associations were not observed for device-reported use. Moreover, those higher in depressive symptoms overestimated their Instagram and Twitter use compared to those lower in depressive symptoms (Tables S17-S20).

Discussion

Although participants demonstrated strong tracking accuracy regarding their Facebook, Instagram, Twitter, and Snapchat use, they nonetheless overestimated their use for all platforms. For Instagram, Twitter, and Snapchat, these overestimations were nearly identical. In terms of sheer minutes, participants overestimated Facebook use to a smaller extent, perhaps because they used Facebook less frequently than the other three platforms. These results replicate previous findings that self-report estimates of SNS app use are discrepant with device-reported use (e.g., Deng et al., 2019; Ernala et al., 2020; Junco, 2013; Sewall et al., 2020); they also extend past work by establishing that inaccuracies are comparable across platforms. Smartphone use is habitual (Oulasvirta et al., 2012), and habitual behaviors are more difficult to estimate (Schwarz & Oyserman, 2001). SNS app use may be particularly challenging to recall due to frequent notifications and alerts, which may trigger constant checking.

Individual differences did not consistently moderate participants’ directional bias and tracking accuracy, with one exception: heavier Facebook users overestimated their use to a stronger extent than lighter users. These results contrast previous findings on internet and mobile phone use suggesting that heavier users underestimate digital technology use and lighter users overestimate use (e.g., Scharkow, 2016; Vanden Abeele et al., 2013), but are in line with other research indicating that those who use SNSs more frequently are less accurate in their use (Ernala et al., 2020; Sewall et al., 2020). Studies finding that heavy users underestimate their use focus on internet and general phone use instead of SNS use, and there may be differences in the accuracies of self-reporting by type of digital technology. Indeed, one study examining app use, including SNS app use, found that heavier users overestimated their overall use more than lighter users (Deng et al., 2019). Moreover, this finding further supports the possibility that habitual behavior (e.g., heavier use) is more difficult to accurately recall than less frequent behaviors (Schwarz & Oyserman, 2001).

Several limitations must be noted. First, participants recorded in the questionnaire their SNS use from the Screen Time app. Future research should more carefully verify these reports, perhaps with screenshots (e.g., Ohme et al., 2020). Second, participants came from two U.S. universities and the sample was disproportionately female, limiting generalizability. Third, due to the nature of the Screen Time app, we could only assess overall use, rather than specific activities. Activities may vary in their accuracy of being recalled (e.g., posting being more easily recalled than scrolling); future research should develop tools that objectively assess individuals’ engagement in passive and active activities. Fourth, as we utilized an iOS-specific app, Android users were excluded from analyses. Android users may differ from iPhone users (e.g., Android users may be more likely to be a racial or ethnic minority; Hussong et al., 2021), and alternative methods are needed to recruit both types of users.

Despite these limitations, this study confirms previous research establishing inaccuracies in reporting SNS use and extends these findings by showcasing how these inaccuracies exist for four major SNS platforms. Given growing technological advancements that allow for easy use of apps to record media use objectively, it is time for those who investigate digital technology to take advantage of these tools and enhance our scientific practices.