Abstract
Children’s screen time increased as a result of the COVID-19 pandemic. In the summer of 2021, we explored the association between high screen time over a period of one year since May 2020 and behavioural problems among children and adolescents. The data were derived from the French EpiCov cohort study, collected in spring 2020, autumn 2020, and spring 2021. Participants (N = 1089) responded to online or telephone interviews about one of their children aged 3 to 14 years. Screen time was categorized as high if the daily mean screen time exceeded recommendations at each collection time. The Strengths and Difficulties Questionnaire (SDQ) was completed by parents to identify internalizing (emotional or peer problems) and externalizing (conduct problems or hyperactivity/inattention) behaviours in their children. Among the 1,089 children, 561 (51.5%) were girls, the average age was 8.6 years (SD 3.7). Internalizing behaviours: High screen time was not associated with internalizing behaviours (OR [95% CI] 1.20 [0.90–1.59]) or emotional symptoms (1.00 [0.71–1.41]) while it was associated with peer problems (1.42 [1.04–1.95]). Externalizing behaviours: High screen time was associated with externalizing problems (1.63 [1.01–2.63]) and conduct problems (1.91 [1.15–3.22]) only among older children aged 11 to 14 years. No association with hyperactivity/inattention was found. In a French cohort, exploration of persistent high screen time in the first year of the pandemic and behaviour difficulties in Summer 2021 resulted in mixed findings according to behaviour’s type and children’s age. These mixed findings warrant further investigation into screen type and leisure/school screen use to enhance future pandemic responses appropriate for children.
Keywords: Screen time, Internalizing problems, Externalizing problems, Cohort study, COVID-19
Introduction
Since the beginning of 2020, to limit the spread of the SARS-CoV2 virus, health restrictions were intermittently put in place in many countries worldwide, including nationwide school closures, lockdowns, social distancing recommendations, and so on. For children, these measures resulted in more time spent at home, an unprecedented change in educational practices with massive or total use of online teaching, and an increase in the use of the Internet to maintain social activities via social networks or online games [1]. A meta-analysis of 32 studies reported an increase in screen time for 67% of children aged 0–17 years during this period, with the largest increase among 6–10-year-old (+ 1.4 h/day) followed by adolescents (+ 0.9 h/day, 11–17 year-old) and pre-schoolers (+ 0.6 h/day, 0–5-year-old) [2].
Before the COVID-19 pandemic, children's screen use was steadily increasing and was beginning earlier in life [3]. With respect to mental health, screen time can have detrimental effects by replacing activities such as sleep, physical activity and real-life social interactions [4, 5] or by increasing excitement through the fast pace and intensity of audio-visual effects [6, 7]. In 2012, a review of the literature found internet addiction to be associated with a higher risk of some psychiatric disorders. Yet, as for substance abuse and mental health, these associations were likely to be bidirectional [8]. In 2018, a Canadian longitudinal study found that higher screen time was associated with a higher risk of poorer development in children aged 5 years or less while the reverse association was not observed. Yet, the study could not control for screen use before the age of 2 or the type of screen use [9]. Regarding the latter, in 2019, another Canadian study assessing various screen types and depression in adolescent found that increased social media and television use were associated with a higher risk of depression, but not video games [10]. In 2022, a systematic review and meta-analysis more specifically explored the associations between screen use and internalizing and externalizing behaviours in children under 12 years of age [11]. Regarding externalizing behaviours, screen time was overall associated with a higher risk but the association weakened when studies quality increased, notably in recent or longitudinal studies controlling for baseline externalizing problems. Screen time was also more associated with aggressive behaviours than with attention-deficit hyperactivity disorder (ADHD). Of note, in recent longitudinal studies controlling for baseline externalizing behaviours, any screen time was associated with a higher risk [12] but inconsistent associations were found when accounting for the type of screen used [13]. Regarding internalizing behaviours, screen time was weekly associated with a higher risk. Studies quality did not seem to moderate the association while the number of informants assessing screen time and internalizing behaviours did. Of note, in recent longitudinal studies controlling for baseline internalizing behaviours, any screen time was associated with a higher risk of internalizing problems [12, 14], with the association likely being bidirectional[12] and depending on the type of screen used[13]. Overall, inconsistent evidence exists regarding the effects of screen time on children's mental health and behavioural problems [15, 16] with common limitations being the bi-directionality of the associations, the small effect size [17], the lack of data regarding the type of screen used (television, smartphone, computer …), the context of use (alone or with family, friends), the content (leisure, educational, social media, online games…), or the use of objective screen time measures.
Regarding internalizing behaviours (symptoms directed inwards to the individual and related to conditions such as depression and anxiety) and externalizing behaviours (symptoms directed outward from individuals, considered disruptive and associated with conditions such as attention deficit hyperactivity disorder–ADHD), a recent meta-analysis of 87 studies conducted before the COVID-19 pandemic reported a significant but small association with high levels of screen time among children aged 12 years or under [11]. Effect sizes were r = 0.07; 95% CI [0.05–0.08] for internalizing problems and r = 0.11; 95% CI [0.10–0.12] for externalizing problems. For the COVID-19 period, a meta-analysis found an increased risk of behavioural problems, anxiety and depression in the 11 cross-sectional studies exploring associations between screen time and mental health outcomes among children during the COVID-19 pandemic. Of the 86 independent outcomes examined, 29% were significant. Specifically, leisure screen time was significantly associated with state anxiety in both healthy weight (r = 0.28; p < 0·05) and overweight/obese children (r = 0.20; p < 0.001). Playing online games (r = 0.11; p < 0.05), internet browsing (r = 0.21; p < 0.01), TV use (r = 0.16; p < 0.01), and social media use (r = 0.23; p < 0.01) were all associated with anxiety and depression. Regarding behavioural problems, increasing overall screen time was associated with aggression (β = 0.12; 95% CI [0.04–0.19]), irritability (β = 0.12; 95% CI [0.06–0.19]), frustration (β = 0.13; 95% CI [0.06–0.19]), and frequency of temper tantrums (β = 0.10; 95% CI [0.03–0.17]), but not with being afraid or restless [2]. The effect sizes are small but seem to be more pronounced during the COVID-19 period. Further investigations are needed as the results are not consistent across studies. Moreover, whether the persistence of a high level of screen time after lockdown periods is associated with later mental health issues is yet to be explored.
The aim of this study was, therefore, to explore whether a persistent high screen time over one year was associated with a higher risk of internalizing and externalizing behavioural problems among children aged 3 to 14 years in the context of the COVID-19 pandemic in France.
Methods
Study population
In May 2020, 371,000 individuals aged 15 years and over were randomly selected from the French national tax database and invited to participate in the EpiCov cohort. As shown in Fig. 1, a total of 134,391 individuals (aged 15 years or older) participated in the self-administered computer-assisted-web or computer-assisted-telephone questionnaire at baseline (participation rate: 36.6%) and were invited for follow-up in Autumn 2020 (first follow-up, 107,759 participants) and in Summer 2021 (second follow-up, 85,074 participants). A sub-sample of 10% of the participants was offered an extended version of the questionnaire at baseline (14,371 respondents) and first follow-up (11,255 respondents), including questions about children.
Fig. 1.
EPICOV study timeline
Outcomes – Internalizing and externalizing problems
Children’s emotional and behavioural problems were ascertained using a parental report on the Strength and Difficulty Questionnaire (SDQ) at the second follow-up in Summer 2021. The SDQ screens for behaviour problems among children aged 3 to 16 years with good specificity and moderate sensitivity [18] and it has been validated in French [19]. Briefly, parents were asked to report their level of agreement with 25 items regarding their child’s behaviour, using a Likert scale ranging from “not true” to “certainly true” with corresponding scores from zero to two. The 25 items are divided into 5 subscales, each with scores ranging from zero to ten, screening for the following: emotional symptoms, conduct problems, hyperactivity/inattention, peer problems, and prosocial behaviour. In this study, we focused on internalizing behaviours (emotional symptoms and peer problems) and externalizing behaviours (conduct problems, hyperactivity/inattention) where higher scores indicate greater difficulties [20]. According to validated cut-offs for the SDQ French version, children at risk (yes, no) were identified by a score of four or more for emotional symptoms, three for conduct problems, six for hyperactivity/inattention, and three for peer problems [19]. Children were considered at risk (yes, no) for internalizing problems if they were at risk for emotional symptoms or peer problems, and for externalizing problems if they were at risk for conduct problems or hyperactivity/inattention.
Exposure – High screen time
The World Health Organization recommends avoiding screen use among children under two years of age and limiting screen time to one hour a day for children aged two to five years [21]. To the best of our knowledge, no international consensus exists for children older than five years, but some academic societies advise limiting screen time to two hours a day among school-aged children and adolescents [22]. Participating parents of the EpiCoV cohort at baseline and at each follow-up were asked to report the daily mean time their child had spent using screen media in the last seven days as follows: 0 min; less than 30 min; 30 min to less than one hour; 1 h to less than 2 h; 2 h to less than 4 h; 4 h to less than 6 h; 6 h to less than 8 h; 8 h to less than 10 h; 10 h or more; don’t know. Children were categorized as having a high screen time if their daily mean time was over one hour for the 3–5-year-old or over two hours for older children at baseline and the two follow-ups. Children who complied with the recommendations at least once were used as the reference category [22, 23]. This study focused on children aged 3 to 14 years, as parental assessment of screen time did not seem appropriate for adolescents over 14 who are more likely to possess their own smartphones. In addition, the type of screen use and the context of use change from a predominantly parent-supervised, television-oriented use in childhood, to an increasingly less parent-supervised and social-network-oriented use with the cell phone [24–26].
Covariates
The choice of covariates was based on the literature. In particular, child characteristics, parental socioeconomic level, and household composition are factors that are associated with internalizing and externalizing problems among children and are also associated with screen time [27].
Parental report of children’s characteristics
Children's characteristics collected were: age (years), gender (boy, girl), any pre-pandemic behavioural difficulties (some, none), sleep difficulties before lockdown (yes, no), time spent by an adult on an average day assisting with schoolwork (less than 30 min, 30 min to less than 2 h, 2 h or more, no school work to do), conditions for completing school work at home (easily isolated, difficult or impossible to isolate, no schoolwork to do). Pre-pandemic behavioural difficulties were assessed using the following questions, “Overall, do you feel that your child has difficulties in any of these domains: emotion, concentration, behaviour, or relationships with others?: No, yes minor, yes significant, yes serious” and “These difficulties have been present for…: less than 1 month, between 1 and 5 months, between 6 months and 1 year, in the first lockdown in March 2020, before the health crisis (March 2020)”.
Parental characteristics
Participating parental characteristics were: age (year), gender (man, woman), educational level (less than high school diploma, high school diploma, higher education diploma), perceived financial situation (comfortable/acceptable vs. it's barely enough/it's difficult/impossible without incurring debt), migration background (born in France with parents born in France, immigrant, descendant of immigrants), baseline tobacco and alcohol use (never, occasionally, regularly), and depression (no depression, mild depression, moderate to severe depression). Depression was assessed using the Patient Health Questionnaire–9 items (PHQ-9). In line with validated cut-offs, a score < 5 indicates no depression, a score between 5 and 9 indicates mild depression and a score ≥ 10 indicates moderate to severe depression [28].
Household characteristics
The household characteristics included: household composition (couple with children, single-parent family or complex household), alternating custody (yes, no), siblings (yes, no), access to an outdoor space at home (yes, no).
All covariates were ascertained at baseline except for children's pre-pandemic behavioural difficulties which were ascertained at the second follow-up.
Statistical analyses
Persistent high screen time and covariate distributions were described according to both internalizing and externalizing behaviours (Table 1). Covariates associated with either internalizing or externalizing problems with a p value ≤ 20% using chi-square tests were selected as candidates for the multivariable model. Multivariable logistic regression then explored the association of high screen time with both internalizing and externalizing behaviours, with a descending variable selection procedure (the child’s age and gender were forced into the models). At each step, the variable with the highest p value was removed from the statistical model if parameters of the remaining variables had a variation of 10% or less once the variable was removed. Interaction of age (3–5 years old, 6–10 years old, 11–14 years old) on one hand, and the child’s gender (boy, girl) on the other hand in the association of high screen time with internalizing and externalizing behaviours was explored. In case of significant interaction, the results were presented stratified. Sensitivity analyses explored the association of high screen time with each of the difficulty subscales (emotional symptoms, peer problems, conduct problems, hyperactivity/attention problems), adjusted for the same covariates retained in the multivariable models for internalizing and externalizing problems.
Table 1.
Characteristics of children and participating parents and bivariate analyses according to internalizing and externalizing problems, n = 1089
| Total | Internalizing problems | Externalizing problems | |||||
|---|---|---|---|---|---|---|---|
| No N = 706 | Yes N = 383 | p* | No N = 771 | Yes N = 318 | p* | ||
| Children | |||||||
| Internalizing problems | 383 (35.2) | ||||||
| Emotional symptoms | 228 (20.9) | ||||||
| Peer problems | 249 (22.9) | ||||||
| Externalizing problems | 318 (29.2) | ||||||
| Conduct problems | 254 (23.3) | ||||||
| Hyperactivity/inattention | 165 (15.2) | ||||||
| High screen time | 410 (37.6) | 253 (35.8) | 157 (41.0) | 0.094 | 280 (36.3) | 130 (40.9) | 0.158 |
| Age (years) | |||||||
| 3–5 | 309 (28.4) | 208 (29.5) | 101 (26.4) | 0.492 | 198 (25.7) | 111 (34.9) | 0.007 |
| 6–10 | 366 (33.6) | 237 (33.6) | 129 (33.7) | 265 (34.4) | 101 (31.8) | ||
| 11–14 | 414 (38.0) | 261 (37.0) | 153 (39.9) | 308 (39.9) | 106 (33.3) | ||
| Girl | 561 (51.5) | 355 (50.3) | 206 (53.8) | 0.269 | 407 (52.8) | 154 (48.4) | 0.190 |
| Pre-pandemic behavioural difficulties | 382 (35.1) | 192 (27.2) | 190 (49.6) | < 0.001 | 210 (27.2) | 172 (54.3) | < 0.001 |
| Sleep difficulties before lockdown | 284 (26.3) | 161 (23.0) | 123 (32.5) | < 0.001 | 181 (23.7) | 103 (32.9) | 0.002 |
| Help by an adult with schoolwork | |||||||
| < 30 min per day | 125 (11.5) | 91 (12.9) | 34 (8.9) | 0.019 | 91 (11.8) | 34 (10.8) | 0.272 |
| 30 min—< 2 h per day | 450 (41.5) | 289 (41.1) | 161 (42.3) | 318 (41.4) | 132 (41.8) | ||
| 2 h per day or more | 435 (40.1) | 267 (37.9) | 168 (44.1) | 314 (40.8) | 121 (38.3) | ||
| No schoolwork to do | 75 (6.9) | 57 (8.1) | 18 (4.7) | 46 (6.0) | 29 (9.2) | ||
| Conditions for school work at home | |||||||
| Easy access to a quiet place | 770 (70.7) | 508 (72.0) | 262 (68.4) | 0.060 | 577 (74.8) | 193 (60.7) | < 0.001 |
| Difficult or impossible to isolate | 230 (21.1) | 135 (19.1) | 95 (24.8) | 141 (18.3) | 89 (28.0) | ||
| No schoolwork to do | 89 (8.2) | 63 (8.9) | 26 (6.8) | 53 (6.9) | 36 (11.3) | ||
| Parents | |||||||
| Age (years) | |||||||
| < 35 | 223 (20.5) | 147 (20.8) | 76 (19.8) | 0.669 | 149 (19.3) | 74 (23.3) | 0.012 |
| 35–45 | 489 (44.9) | 310 (43.9) | 179 (46.7) | 334 (43.3) | 155 (48.7) | ||
| > 45 | 377 (34.6) | 249 (35.3) | 128 (33.4) | 288 (37.4) | 89 (28.0) | ||
| Mother | 675 (62.0) | 427 (60.5) | 248 (64.8) | 0.166 | 463 (60.1) | 212 (66.7) | 0.041 |
| Education level | |||||||
| Less than high school diploma | 202 (18.5) | 118 (16.7) | 84 (21.9) | 0.029 | 133 (17.3) | 69 (21.7) | 0.185 |
| High school diploma | 197 (18.1) | 121 (17.1) | 76 (19.8) | 138 (17.9) | 59 (18.6) | ||
| Higher education diploma | 690 (63.4) | 467 (66.1) | 223 (58.2) | 500 (64.9) | 190 (59.7) | ||
| Poor perceived financial situation | 478 (43.9) | 290 (41.1) | 188 (49.2) | 0.010 | 313 (40.6) | 165 (52.1) | < 0.001 |
| Migration background | |||||||
| Born in France with parents born in France | 883 (82.8) | 577 (83.5) | 306 (81.6) | 0.724 | 626 (82.9) | 257 (82.6) | 0.183 |
| Immigrant | 73 (6.8) | 46 (6.7) | 27 (7.2) | 57 (7.5) | 16 (5.1) | ||
| Descendant of immigrants | 110 (10.3) | 68 (9.8) | 42 (11.2) | 72 (9.5) | 38 (12.2) | ||
| Tobacco use | |||||||
| Never | 845 (77.6) | 561 (79.5) | 284 (74.2) | 0.021 | 607 (78.7) | 238 (74.8) | 0.119 |
| Occasionally | 43 (3.9) | 31 (4.4) | 12 (3.1) | 33 (4.3) | 10 (3.1) | ||
| Regularly | 201 (18.5) | 114 (16.1) | 87 (22.7) | 131 (17.0) | 70 (22.0) | ||
| Alcohol use | |||||||
| Never | 288 (26.4) | 192 (27.2) | 96 (25.1) | 0.719 | 198 (25.7) | 90 (28.3) | 0.671 |
| Occasionally | 693 (63.6) | 446 (63.2) | 247 (64.5) | 496 (64.3) | 197 (61.9) | ||
| Regularly | 108 (9.9) | 68 (9.6) | 40 (10.4) | 77 (10.0) | 31 (9.7) | ||
| Depressive symptoms (PHQ9**) | |||||||
| No (score < 5) | 694 (63.8) | 488 (69.1) | 206 (53.9) | < 0.001 | 526 (68.3) | 168 (52.8) | < 0.001 |
| Mild (5 ≤ score < 10) | 292 (26.8) | 167 (23.7) | 125 (32.7) | 187 (24.3) | 105 (33.0) | ||
| Moderate to severe (Score ≥ 10) | 102 (9.4) | 51 (7.2) | 51 (13.4) | 57 (7.4) | 45 (14.2) | ||
| Household | |||||||
| ingle parent or complex household | 168 (15.4) | 104 (14.7) | 64 (16.7) | 0.388 | 116 (15.0) | 52 (16.4) | 0.587 |
| Alternating custody | 101 (9.3) | 57 (8.1) | 44 (11.5) | 0.064 | 73 (9.5) | 28 (8.8) | 0.732 |
| Rank among siblings | |||||||
| 1st | 337 (30.9) | 207 (29.3) | 130 (33.9) | 0.077 | 235 (30.5) | 102 (32.1) | 0.288 |
| 2nd or more | 265 (24.3) | 186 (26.3) | 79 (20.6) | 180 (23.3) | 85 (26.7) | ||
| Only child | 487 (44.7) | 313 (44.3) | 174 (45.4) | 356 (46.2) | 131 (41.2) | ||
| Access to an outdoor space | 856 (78.6) | 557 (78.9) | 299 (78.1) | 0.751 | 616 (79.9) | 240 (75.5) | 0.105 |
*p value of the chi-square test
**PHQ9 patient health questionnaire-9
Children with missing data for screen time or any of the behavioural problems were excluded from the analyses. Missing data for covariates (up to 2.48%) were imputed using multiple imputations by chained equations with the MICE package on R software, assuming data was missing at random [29]. The analyses were performed in R version 4.1.0.
Results
Sample selection and characteristics
As shown in the flowchart (Fig. 2), at the second follow-up, 21,512 participants had at least one child aged 3–17 years. Among them, 1,432 were part of the subsample completing the extended questionnaire at baseline and first follow-up, and therefore had screen time assessments since baseline. Then, 320 children were excluded from analyses because they were aged 15–17 years and 23 children had missing data on outcomes or exposure. The analyses were therefore performed on 1,089 children aged 3–14 years who provided parental assessments of screen time since May 2020 and parental answers to the SDQ in Summer 2021.
Fig. 2.
Flow chart for sample selection from the respondents to the second follow-up of the EpiCoV cohort study
The sample characteristics are described in Table 1. Among the 1,089 children, 561 (51.5%) were girls, the average age was 8.6 years (SD 3.7) and 383 (35.2%) children were at risk for internalizing problems, 228 (20.9%) for emotional symptoms, 249 (22.9%) for peer problems, and 318 (29.2%) for externalizing problems, 254 (23.3%) for conduct problems and 165 (15.2%) for hyperactivity/inattention. Concerning screen time, 410 (37.6%) children had persistent high screen time. All covariates were candidates for multivariable analysis except current parental alcohol use and household composition.
Internalizing problems
Table 2 shows the non-adjusted and adjusted associations of high screen time with internalizing problems, emotional symptoms and peer problems. Adjustment covariates in the final adjusted models were child age, gender, pre-pandemic behavioural difficulties, sleep difficulties before lockdown, time spent by a family member helping with schoolwork, conditions for completing schoolwork at home, parental educational level and depressive symptoms. Neither age nor gender interacted in the association of screen time with internalizing behaviours (p > 0.9 and p = 0.2, respectively).
Table 2.
Non-adjusted and adjusted associations between high screen use over one year since May 2020 and internalizing problems, emotional symptoms and peer problems during summer 2021, in the EpiCov cohort study, n = 1,089
| Internalizing problems | Emotional symptoms | Peer problems | ||||
|---|---|---|---|---|---|---|
| ORc [95% CI] | ORa [95% CI] | ORc [95% CI] | ORa [95% CI] | ORc [95% CI] | ORa [95% CI] | |
| High screen time | ||||||
| No | 1 (Reference) | 1 (Reference) | 1 (Reference) | 1 (Reference) | 1 (Reference) | 1 (Reference) |
| Yes | 1.24 [0.96–1.61] | 1.20 [0.90–1.59] | 0.93 [0.69–1.27] | 1 [0.71–1.41] | 1.62 [1.22–2.16]* | 1.42 [1.04–1.95]* |
*Statistically significant (p < 0.05)
High screen time was not associated with internalizing problems (OR [95% CI] 1.20 [0.90–1.59]). Regarding subscales, high screen time was not associated with emotional symptoms (1.00 [0.71–1.41]) but was associated with peer problems (1.42 [1.04–1.95]).
ORc crude odds ratio
ORa adjusted odds ratio. Logistic regression model adjusted for child age, gender, pre-pandemic behavioural difficulties, sleep difficulties before lockdown, time spent by a family member helping with schoolwork, conditions for doing schoolwork at home, parental educational level and depressive symptoms
Externalizing problems
Table 3 shows the non-adjusted and adjusted associations of high screen time with externalizing problems, conduct problems and hyperactivity/inattention. Adjustment covariates in the final adjusted models were child gender, pre-pandemic behavioural difficulties, conditions for completing schoolwork at home, parental education, perceived financial status and depressive symptoms. The interaction of child age in the association of high screen time with externalising behaviours was significant (p = 0.019). The results are therefore presented by age group (3–5 years, 6–10 years and 11–14 years). The interaction with gender was not significant (p = 0.4).
Table 3.
Non-adjusted and adjusted associations between high screen use over one year since May 2020 and externalizing problems, conduct problems and symptoms of hyperactivity/inattention, during summer 2021, stratified by age, in the EpiCov cohort study, n = 1089
| Externalizing problems | Conduct problems | Symptoms of hyperactivity/inattention | ||||
|---|---|---|---|---|---|---|
| ORc [95% CI] | ORa [95% CI] | ORc [95% CI] | ORa [95% CI] | ORc [95% CI] | ORa [95% CI] | |
| 3–5 years of high screen time | ||||||
| No | 1 (Reference) | 1 (Reference) | 1 (Reference) | 1 (Reference) | 1 (Reference) | 1 (Reference) |
| Yes | 0.93 [0.57–1.52] | 0.94 [0.56–1.58] | 0.91 [0.54–1.53] | 0.95 [0.54–1.64] | 1.53 [0.86–2.72] | 1.61 [0.86–3.02] |
| 6–10 years of high screen time | ||||||
| No | 1 (Reference) | 1 (Reference) | 1 (Reference) | 1 (Reference) | 1 (Reference) | 1 (Reference) |
| Yes | 1.21 [0.71–2.06] | 1.11 [0.63–1.97] | 1.40 [0.80–2.47] | 1.36 [0.74–2.46] | 1.25 [0.66–2.35] | 0.98 [0.48–1.91] |
| 11–14 years of high screen time | ||||||
| No | 1 (Reference) | 1 (Reference) | 1 (Reference) | 1 (Reference) | 1 (Reference) | 1 (Reference) |
| Yes | 1.74 [1.11–2.73]* | 1.63 [1.01–2.63]* | 2.01 [1.24–3.32]* | 1.91 [1.15–3.22]* | 1.43 [0.77–2.72] | 1.26 [0.66–2.48] |
*Statistically significant (p < 0.05)
High screen time was associated with a higher risk of externalizing problems among children aged 11 to 14 (OR [95% CI]: 1.63 [1.01–2.63]), but not among 3 to 5-year-old or 6 to 10-year-old (0.94 [0.56–1.58] and 1.11 [0.63–1.97], respectively). Regarding subscales, the same pattern was found for conduct problems (0.95 [0.54–1.64] among 3 to 5-year-old; 1.36 [0.74–2.46] among 6 to 10-year-old; 1.91 [1.15–3.22] among 11 to 14-year-old), while no association was found for hyperactivity/inattention (1.61 [0.86–3.02] among 3 to 5-year-old; 0.98 [0.48–1.91] among 6 to 10-year-old; 1.26 [0.66–2.48] among 11 to 14-year-old).
ORc crude odds ratio
ORa adjusted odds ratio. Logistic regression model adjusted for child gender, pre-pandemic behavioural difficulties, conditions for doing schoolwork at home, parental education, perceived financial status and depressive symptoms
Discussion
Main findings
This study assessing the association of persistent high screen use in the first year of the COVID-19 pandemic and internalizing and externalizing behaviours of children aged 3 to 14 years old resulted in mixed findings. For internalizing behaviours, neither gender nor age were found to moderate the associations. Persistent high screen time was associated with a weak higher risk of peer problems (OR [95% CI]1.42 [1.04–1.95]), whereas no association was found for emotional symptoms (1.00 [0.71–1.41]) or internalizing behaviours as a whole (1.20 [0.90–1.59]). For externalizing behaviours, age was a significant moderator. In children aged 11 to 14 years, persistent high screen time was weakly associated with a higher risk of conduct problems (1.91 [1.15–3.22]) and externalizing behaviour as a whole (1.63 [1.01–2.63]). Yet, no associations were found in children aged 3 to 5 or 6 to 10 years, or with ADHD symptoms. Overall, correlation coefficients of associations of persistent high screen use with peer problems, externalizing behaviours and conduct problems were below 0.18. To our knowledge, this is the first study to longitudinally explore the association between screen time among 3 to 14-year-old children during the first 18 months of the COVID-19 pandemic and emotional and behavioural issues.
Regarding pre-pandemic studies, a secondary analysis of a randomized control trial of 709 Californian children aged 9 to 11 years in 2009–2011, assessed whether anhedonia in 5th grade (inability to experience pleasure from activities usually found enjoyable) mediated the association between screen time outside the school on a regular school day in 4th grade, including computer, television, video movies, and video games (both sit-down and active), and substance use in 6th grade, using structural equation modelling [14]. In the model, an increase in screen time was significantly correlated to an increase anhedonia (standardized Beta estimate of 0.161). Although emotional symptoms can be early signs of depression, one of the symptoms being anhedonia, persistent high screen time was not associated with emotional symptoms in the present study. Differences in the study dates, how screen time is measured (leisure versus any), who reports the measurement and the outcome considered could explain these differences. In the Born in Ireland study, parents reported their child’s daily screen time (any screen) and internalising and externalising behaviours using the SDQ at age 3, 5, 7 and 9-year-old [12]. Associations between screen time and internalizing behaviours seemed bidirectional between 3 and 5, and directional in a higher screen time higher internalising behaviour manner between age 5 and 7 and higher screen time lower internalising behaviour manner between age 7 and 9. For externalising behaviour, only a directional association between externalizing behaviour at age 3 and higher screen time at age 5 was found. Discrepancies might come from the fact that, in the present study, pre-pandemic behaviour problems were included as a covariate in adjusted models, partially accounting for the bidirectionality of the association. Moreover, in the Growing up in Ireland study, subscales were not separate, complicating the comparison with the present study. Finally, the research question was slightly different as the present study assessed the association between persistent high screen time over a year with internalizing and externalizing behaviours at the end of the year, while the Growing up in Ireland study explored high screen time at a given time and behaviours two years later. In studies assessing specific types of screens, video games seemed associated with hyperactivity, but not emotional symptoms [13], but the association was not found for television [30], or when using a clinical diagnosis of hyperactivity [31].
Regarding studies in the COVID-19 pandemic context, the meta-analysis exploring screen time in this specific context reported that overall screen time (only for specific uses like leisure screen time, online gaming, internet browsing, television and social media use) was not associated with anxiety among children aged 12 years on average while screen time was associated with a higher risk of depression in all studies [2]. In the present study, the association between screen time and emotional problems, a potential early sign of depression, was not observed. In addition to the fact that the present study did not explore anxiety and depression but early markers, one difference could be that all but one of the studies in this meta-analysis were cross-sectional. As highlighted in another meta-analysis [11] these associations tend to weaken when studies quality increase, with cross-sectional studies potentially overemphasizing the association. In our study, persistent high-screen use over a year was associated with peer problems. Loneliness, peer rejection, or relationship issues in real life could lead the child to prefer solitary but interactive activities available on the computer, or interaction with others possibly facilitated by the feeling of protection conferred by the screen [32, 33]. Reverse causation is also, of course, plausible, as in a vicious circle in which a high screen use could reinforce peer problems which can, in return, reinforce the child’s trends toward the use of screens. Further research is needed to understand these pathways, but our results suggest that, in the context of the COVID-19 pandemic, persistent high screen time over one year could be a more specific risk factor for peer problems than for emotional problems. From a developmental perspective, the association of long-term, high screen time with peer problems is worrisome, as childhood is a key developmental period where children start to rely increasingly on their peers rather than their family. Although screens were one of the easiest, if not the only, way to keep in touch with loved ones during lockdowns, face-to-face interaction with peers could still be necessary for childhood to foster harmonious development. Regarding internalizing problems, whether before or during the pandemic, results concerning the association with screen time are inconsistent [34]. No association was found in our study, but an indirect effect of high screen time via social isolation or sleep disturbances could exist [35].
A longitudinal study examining the association between screen time and behavioural problems in the setting of the pandemic found an association between increased TV or digital media use and conduct problems in 2–4-year-old (β = 0.22, 95% CI [0.10–0.35]; p < 0.001) and 4–17-year-old (β = 0.07, 95% CI [0.02–0.11]; p = 0.007). For hyperactivity/inattention, there was no association among 2 to 4-year-old and an association among 4- to 17-year-old for television or digital screen time (β = 0.07, 95% CI [0.006–0.14]; p = 0.04) [36]. In the present study screen time was weakly associated with conduct problems but only in 11–14 years old children and no association was found for hyperactivity. These discrepancies might be due to the lack of information regarding the type of screen used in our study as screen use is likely to change with age. The present study also explored screen time and behaviours separately in 3–5 years old, 6–10 and 11–14 years old children and excluded older adolescents. Although debated [37], a hypothesis behind the higher risk of conduct problems related to screen use is that exposure to inappropriate content, aggression and violence, could be associated with lower empathy, stronger pro-violence attitudes and adolescent aggression [38–40]. However other factors not included in the EpiCoV cohort study, such as family environment or beliefs about aggression, physical activity, the child's temperament, the parents' style, could play a mediating or moderating role in this association, and should be investigated further [41].
Strengths and limitations
Our study has some limitations that should be noted. First, as the EpiCoV cohort began in the spring of 2020, no pre-pandemic information was prospectively collected. In particular, estimation of the impact of the pandemic itself on children’s screen time and behaviour was not feasible as no pre-pandemic parental assessment of children’s screen time or behaviour were available. Nonetheless, pre-pandemic sleep difficulties and behavioural difficulties were collected retrospectively and accounted for in this study. Secondly, the screen time evaluated is a global screen time, without distinction on the type of screen used (TV, smartphone, PC, video games…), or the use made of it (active or passive use of screens), and without distinction between screen time allocated to leisure and to schoolwork. School screen time possibly does not have the same impact on children’s behaviour as leisure screen time, depending on the type of program viewed. Here, screen time was considered longitudinally, using all three-time points in the EpiCov study to identify children who maintained high screen time after the first lockdown and the return to school. However, behavioural problems were assessed at the third time point, which means concomitant with the last measurement of screen time. We, therefore, lack some perspective on the long-term effects of this overexposure on behavioural problems. Thirdly, this study used parental reports of screen time. The declarative nature of the information makes it less accurate than an objective measurement and could be inaccurate for older children, which is why we excluded children over 14 years of age. In addition, the cut-offs used for this study are based on expert recommendations for screen use, but further research should be conducted to empirically identify a time beyond which screen use becomes harmful, taking into account the type of use. The EpiCov study enabled a longitudinal assessment of screen time over a one-year period prior to the collection of behavioural problems, and it included a large number of socio-demographic variables that were screened as potential confounders in the present study. Consistent with current literature, the present study observed mixed findings with moderate effect sizes.
Conclusion
The context of the COVID-19 pandemic changed, at least temporarily, the daily lives of families and children. Screen exposure increased, notably during lockdown periods, to maintain social relationships and professional activity, and to occupy leisure time. This study found that maintaining screen time beyond recommendations for a year after the first lockdown in France was not associated with emotional symptoms or hyperactivity/inattention. In contrast, it was weakly associated with a higher risk of peer problems in children aged 3–14 years, and moderately with a higher risk of externalizing and conduct problems in 11–14 years old. Nonetheless, the health situation required these lifestyle changes, and the opportunity to use screens during these lockdown periods may also have had beneficial impacts on the population's physical and mental health. Further studies exploring how context and content of screen use are associated with children and adolescents mental health, with negative and positive indicators, are needed to better understand how to support them during future pandemics.
Acknowledgements
EpiCoV Study Group: Josiane Warszawski, Nathalie Bajos, Guillaume Bagein, François Beck, Emilie Counil, Florence Jusot, Nathalie Lydie, Claude Martin, Laurence Meyer, Philippe Raynaud, Alexandra Rouquette, Ariane Pailhé, Delphine Rahib, Patrick Sicard, Rémy Slama, Alexis Spire.
Author contributions
Arthur Descarpentry: conceptualisation, data curation, formal analysis, methodology, project administration, validation, visualisation, writing - original draft Maria Melchior: conceptualisation, methodology, supervision, validation, writing - review & editing Cédric Galera: methodology, validation, writing - review & editing Jean-Baptiste Hazo: writing - review & editing Bruno Falissard: writing - review & editing Josiane Warszawski: investigation, validation, writing - review & editing Camille Davisse-Paturet: conceptualisation, methodology, project administration, supervision, validation, writing - review & editing Alexandra Rouquette: conceptualisation, methodology, project administration, supervision, validation, writing - review & editing EpiCoV study group: investigation
Funding
The EpiCov study received institutional funding from Inserm (Institut National de la Santé et de la Recherche Médicale), the French Ministry for Research and its Department of Research, Studies, Evaluation and Statistics (Direction de la Recherche, des Etudes, de l’Evaluation et des Statistiques, Drees), the French Ministry for Health, and the Région Ile de France. CDP was supported by a grant from the Department of Research, Studies, Evaluation and Statistics (Direction de la Recherche, des Etudes, de l’Evaluation et des Statistiques, Drees) of the French Ministry for Research attributed to AR (grant number R21094LL).
Data availability
The non-aggregated individual data cannot be shared publicly because of European Regulation 2016/679. Nonetheless, these data can be made available after submission to approval of French Ethics and Regulatory Committee procedure (Comité du Secret Statistique, CESREES and CNIL). The access procedure is available on the Centre of Secured Access to Data website (https://www.casd.eu/).
Declarations
Conflict of interest
The authors declare that they have no conflicts of interest.
Ethics approval
The EpiCov study received approval from an ethics committee (Comité de Protection des Personnes Sud Méditerranée III 2020- A01191-38) and from France’s National Data Protection Agency (Commission Nationale Informatique et Libertés, CNIL, MLD/MFI/AR205138).
Footnotes
The collaborators names associated with The EpiCoV Study Group are given in acknowledgements sections.
Camille Davisse-Paturet and Alexandra Rouquette have Contributed equally to this work.
Contributor Information
Arthur Descarpentry, Email: arthur.descarpentry@inserm.fr.
The EpiCoV Study Group:
Josiane Warszawski, Nathalie Bajos, Guillaume Bagein, François Beck, Emilie Counil, Florence Jusot, Nathalie Lydie, Claude Martin, Laurence Meyer, Philippe Raynaud, Alexandra Rouquette, Ariane Pailhé, Delphine Rahib, Patrick Sicard, Rémy Slama, and Alexis Spire
Bibliography
- 1.Madigan S, Eirich R, Pador P, McArthur BA, Neville RD. Assessment of changes in child and adolescent screen time during the COVID-19 pandemic: a systematic review and meta-analysis. JAMA Pediatr. 2022;176:1188–1198. doi: 10.1001/jamapediatrics.2022.4116. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Trott M, Driscoll R, Iraldo E, Pardhan S. Changes and correlates of screen time in adults and children during the COVID-19 pandemic: a systematic review and meta-analysis. EClinicalMedicine. 2022 doi: 10.1016/j.eclinm.2022.101452. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Dumuid D. Screen time in early childhood. Lancet Child Adolesc Health. 2020;4:169–170. doi: 10.1016/S2352-4642(20)30005-5. [DOI] [PubMed] [Google Scholar]
- 4.Christakis DA, Gilkerson J, Richards JA, Zimmerman FJ, Garrison MM, Xu D, et al. Audible television and decreased adult words, infant vocalizations, and conversational turns: a population-based study. Arch Pediatr Adolesc Med. 2009;163:554–558. doi: 10.1001/archpediatrics.2009.61. [DOI] [PubMed] [Google Scholar]
- 5.Christakis DA. Interactive media use at younger than the age of 2 years: time to rethink the american academy of pediatrics guideline? JAMA Pediatr. 2014;168:399–400. doi: 10.1001/jamapediatrics.2013.5081. [DOI] [PubMed] [Google Scholar]
- 6.Linebarger DL, Barr R, Lapierre MA, Piotrowski JT. Associations between parenting, media use, cumulative risk, and children’s executive functioning. J Dev Behav Pediatr. 2014;35:367–377. doi: 10.1097/DBP.0000000000000069. [DOI] [PubMed] [Google Scholar]
- 7.Nikkelen SWC, Valkenburg PM, Huizinga M, Bushman BJ. Media use and ADHD-related behaviors in children and adolescents: a meta-analysis. Dev Psychol. 2014;50:2228–2241. doi: 10.1037/a0037318. [DOI] [PubMed] [Google Scholar]
- 8.Ko C-H, Yen J-Y, Yen C-F, Chen C-S, Chen C-C. The association between Internet addiction and psychiatric disorder: a review of the literature. Eur Psychiatry. 2012;27:1–8. doi: 10.1016/j.eurpsy.2010.04.011. [DOI] [PubMed] [Google Scholar]
- 9.Madigan S, Browne D, Racine N, Mori C, Tough S. Association between screen time and children’s performance on a developmental screening test. JAMA Pediatr. 2019;173:244–250. doi: 10.1001/jamapediatrics.2018.5056. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Boers E, Afzali MH, Newton N, Conrod P. Association of screen time and depression in adolescence. JAMA Pediatr. 2019;173:853–859. doi: 10.1001/jamapediatrics.2019.1759. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Eirich R, McArthur BA, Anhorn C, McGuinness C, Christakis DA, Madigan S. Association of screen time with internalizing and externalizing behavior problems in children 12 years or younger: a systematic review and meta-analysis. JAMA Psychiat. 2022;79:393–405. doi: 10.1001/jamapsychiatry.2022.0155. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Neville RD, McArthur BA, Eirich R, Lakes KD, Madigan S. Bidirectional associations between screen time and children’s externalizing and internalizing behaviors. J Child Psychol Psychiatry. 2021;62:1475–1484. doi: 10.1111/jcpp.13425. [DOI] [PubMed] [Google Scholar]
- 13.Stenseng F, Hygen BW, Wichstrøm L. Time spent gaming and psychiatric symptoms in childhood: cross-sectional associations and longitudinal effects. Eur Child Adolesc Psychiatry. 2020;29:839–847. doi: 10.1007/s00787-019-01398-2. [DOI] [PubMed] [Google Scholar]
- 14.Christodoulou G, Majmundar A, Chou C-P, Pentz MA. Anhedonia, screen time, and substance use in early adolescents: a longitudinal mediation analysis. J Adolesc. 2020;78:24–32. doi: 10.1016/j.adolescence.2019.11.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Browne D, Thompson DA, Madigan S. Digital media use in children: clinical vs scientific responsibilities. JAMA Pediatr. 2020;174:111–112. doi: 10.1001/jamapediatrics.2019.4559. [DOI] [PubMed] [Google Scholar]
- 16.Odgers CL, Jensen MR. Annual Research Review: Adolescent mental health in the digital age: facts, fears, and future directions. J Child Psychol Psychiatry. 2020;61:336–348. doi: 10.1111/jcpp.13190. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Zahedi S, Jaffer R, Iyer A. A systematic review of screen-time literature to inform educational policy and practice during COVID-19. Int J Edu Res Open. 2021;2:100094. doi: 10.1016/j.ijedro.2021.100094. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Goodman R. Psychometric properties of the strengths and difficulties questionnaire. J Am Acad Child Adolesc Psychiatry. 2001;40:1337–1345. doi: 10.1097/00004583-200111000-00015. [DOI] [PubMed] [Google Scholar]
- 19.Shojaei T, Wazana A, Pitrou I, Kovess V. The strengths and difficulties questionnaire: validation study in French school-aged children and cross-cultural comparisons. Soc Psychiatry Psychiatr Epidemiol. 2009;44:740–747. doi: 10.1007/s00127-008-0489-8. [DOI] [PubMed] [Google Scholar]
- 20.Goodman A, Lamping DL, Ploubidis GB. When to use broader internalising and externalising subscales instead of the hypothesised five subscales on the strengths and difficulties questionnaire (sdq): data from british parents, teachers and children. J Abnorm Child Psychol. 2010;38:1179–1191. doi: 10.1007/s10802-010-9434-x. [DOI] [PubMed] [Google Scholar]
- 21.World Health Organization . Guidelines on physical activity, sedentary behaviour and sleep for children under 5 years of age. World Health Organization; 2019. [PubMed] [Google Scholar]
- 22.Tremblay MS, Carson V, Chaput J-P, Connor Gorber S, Dinh T, Duggan M, et al. Canadian 24-hour movement guidelines for children and youth: an integration of physical activity, sedentary behaviour, and sleep. Appl Physiol Nutr Metab. 2016;41:S311–S327. doi: 10.1139/apnm-2016-0151. [DOI] [PubMed] [Google Scholar]
- 23.Media use in childhood: Evidence-based recommendations for caregivers. https://www.apa.org/pi/families/resources/newsletter/2019/05/media-use-childhood (accessed Aug 8 2022).
- 24.The Common Sense Census: Media Use by Tweens and Teens, 2019 Common Sense Media n.d. https://www.commonsensemedia.org/research/the-common-sense-census-media-use-by-tweens-and-teens-2019 (accessed Aug 8 2022).
- 25.The Common Sense Census: Media Use by Kids Age Zero to Eight, 2020 Common Sense Media n.d. https://www.commonsensemedia.org/research/the-common-sense-census-media-use-by-kids-age-zero-to-eight-2020 (accessed Aug 8 2022).
- 26.Ramirez ER, Norman GJ, Rosenberg DE, Kerr J, Saelens BE, Durant N, et al. Adolescent screen time and rules to limit screen time in the home. J Adolesc Health. 2011;48:379–385. doi: 10.1016/j.jadohealth.2010.07.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Atkin AJ, Sharp SJ, Corder K, van Sluijs EMF. International children’s accelerometry database (ICAD) collaborators. Prevalence and correlates of screen time in youth: an international perspective. Am J Prev Med. 2014;47:803–807. doi: 10.1016/j.amepre.2014.07.043. [DOI] [PubMed] [Google Scholar]
- 28.Kroenke K, Spitzer RL, Williams JBW. The PHQ-9. J Gen Intern Med. 2001;16:606–613. doi: 10.1046/j.1525-1497.2001.016009606.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.van Buuren S, Groothuis-Oudshoorn K. Mice: multivariate imputation by chained equations in R. J Stat Softw. 2011;45:1–67. doi: 10.18637/jss.v045.i03. [DOI] [Google Scholar]
- 30.Peralta GP, Forns J, García de la Hera M, González L, Guxens M, López-Vicente M, et al. Sleeping, TV, cognitively stimulating activities, physical activity, and attention-deficit hyperactivity disorder symptom incidence in children: a prospective study. J Dev Behav Pediatr. 2018;39:192–199. doi: 10.1097/DBP.0000000000000539. [DOI] [PubMed] [Google Scholar]
- 31.Levelink B, van der Vlegel M, Mommers M, Gubbels J, Dompeling E, Feron FJM, et al. The longitudinal relationship between screen time, sleep and a diagnosis of attention-deficit/hyperactivity disorder in childhood. J Atten Disord. 2021;25:2003–2013. doi: 10.1177/1087054720953897. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Bonetti L, Campbell MA, Gilmore L. “The Relationship of Loneliness and SocialAnxiety with Children’s and Adolescents’ Online Communication. Cyberpsychol Behav Soc Network. 2010;13(3):279–85. doi: 10.1089/cyber.2009.0215. [DOI] [PubMed] [Google Scholar]
- 33.Niiranen J, Kiviruusu O, Vornanen R, Saarenpää-Heikkilä O, Paavonen EJ. High-dose electronic media use in five-year-olds and its association with their psychosocial symptoms: a cohort study. BMJ Open. 2021;11:e040848. doi: 10.1136/bmjopen-2020-040848. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Tang S, Werner-Seidler A, Torok M, Mackinnon AJ, Christensen H. The relationship between screen time and mental health in young people: A systematic review of longitudinal studies. Clin Psychol Rev. 2021;86:102021. doi: 10.1016/j.cpr.2021.102021. [DOI] [PubMed] [Google Scholar]
- 35.Youth Screen Time and Behavioral Health Problems: The Role of Sleep Duration and Disturbances - PMC n.d. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4851593/ (Accessed Nov 14 2022). [DOI] [PMC free article] [PubMed]
- 36.Li X, Vanderloo LM, Keown-Stoneman CDG, Cost KT, Charach A, Maguire JL, et al. Screen use and mental health symptoms in canadian children and youth during the COVID-19 pandemic. JAMA Netw Open. 2021;4:e2140875. doi: 10.1001/jamanetworkopen.2021.40875. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Savage J, Yancey C. The effects of media violence exposure on criminal aggression: a meta-analysis. Crim Justice Behav. 2008;35:772–791. doi: 10.1177/0093854808316487. [DOI] [Google Scholar]
- 38.Huesmann LR. The impact of electronic media violence: scientific theory and research. J Adolesc Health. 2007;41:S6–13. doi: 10.1016/j.jadohealth.2007.09.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Tomopoulos S, Dreyer BP, Valdez P, Flynn V, Foley G, Berkule SB, et al. Media content and externalizing behaviors in Latino toddlers. Ambul Pediatr. 2007;7:232–238. doi: 10.1016/j.ambp.2007.02.004. [DOI] [PubMed] [Google Scholar]
- 40.Violence exposure in real-life, video games, television, movies, and the internet: is there desensitization? - PubMed n.d. https://pubmed-ncbi-nlm-nih-gov.ezproxy.universite-paris-saclay.fr/15013258/ (Accessed Dec 6 2022). [DOI] [PubMed]
- 41.Shao R, Wang Y. The relation of violent video games to adolescent aggression: an examination of moderated mediation effect. Front Psychol. 2019;10:384. doi: 10.3389/fpsyg.2019.00384. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The non-aggregated individual data cannot be shared publicly because of European Regulation 2016/679. Nonetheless, these data can be made available after submission to approval of French Ethics and Regulatory Committee procedure (Comité du Secret Statistique, CESREES and CNIL). The access procedure is available on the Centre of Secured Access to Data website (https://www.casd.eu/).