COVID RADAR app: Description and validation of population surveillance of symptoms and behavior in relation to COVID-19

Willian J van Dijk; Nicholas H Saadah; Mattijs E Numans; Jiska J Aardoom; Tobias N Bonten; Menno Brandjes; Michelle Brust; Saskia le Cessie; Niels H Chavannes; Rutger A Middelburg; Frits Rosendaal; Leo G Visser; Jessica Kiefte-de Jong

doi:10.1371/journal.pone.0253566

. 2021 Jun 30;16(6):e0253566. doi: 10.1371/journal.pone.0253566

COVID RADAR app: Description and validation of population surveillance of symptoms and behavior in relation to COVID-19

Willian J van Dijk ¹, Nicholas H Saadah ^1,², Mattijs E Numans ¹, Jiska J Aardoom ^1,², Tobias N Bonten ^1,², Menno Brandjes ³, Michelle Brust ¹, Saskia le Cessie ^4,⁵, Niels H Chavannes ^1,², Rutger A Middelburg ^1,⁴, Frits Rosendaal ⁴, Leo G Visser ⁶, Jessica Kiefte-de Jong ^1,^2,^*

Editor: Simone Lolli⁷

PMCID: PMC8244909 PMID: 34191828

Abstract

Background

Monitoring of symptoms and behavior may enable prediction of emerging COVID-19 hotspots. The COVID Radar smartphone app, active in the Netherlands, allows users to self-report symptoms, social distancing behaviors, and COVID-19 status daily. The objective of this study is to describe the validation of the COVID Radar.

Methods

COVID Radar users are asked to complete a daily questionnaire consisting of 20 questions assessing their symptoms, social distancing behavior, and COVID-19 status. We describe the internal and external validation of symptoms, behavior, and both user-reported COVID-19 status and state-reported COVID-19 case numbers.

Results

Since April 2nd, 2020, over 6 million observations from over 250,000 users have been collected using the COVID Radar app. Almost 2,000 users reported having tested positive for SARS-CoV-2. Amongst users testing positive for SARS-CoV-2, the proportion of observations reporting symptoms was higher than that of the cohort as a whole in the week prior to a positive SARS-CoV-2 test. Likewise, users who tested positive for SARS-CoV-2 showed above average risk social-distancing behavior. Per-capita user-reported SARS-CoV-2 positive tests closely matched government-reported per-capita case counts in provinces with high user engagement.

Discussion

The COVID Radar app allows voluntarily self-reporting of COVID-19 related symptoms and social distancing behaviors. Symptoms and risk behavior increase prior to a positive SARS-CoV-2 test, and user-reported case counts match closely with nationally-reported case counts in regions with high user engagement. These results suggest the COVID Radar may be a valid instrument for future surveillance and potential predictive analytics to identify emerging hotspots.

Introduction

The world is in the throes of the coronavirus-disease-2019 (COVID-19) pandemic with more than 100 million cases and over 2 million confirmed deaths worldwide as of December 2020 [1]. In the Netherlands, the first case of COVID-19 was diagnosed in February 2020 and since then over one million cases and 17,500 deaths have been confirmed [2]. To date more than 60,000 COVID-19 patients have been admitted to Dutch hospitals, with over 12,000 of these eventually admitted to intensive care [2]–this in a country with just over 1,000 intensive care beds [3]. The strategies of Test Trace and Isolate (TTI), and of measures intended to reduce social contact, have been widely adopted to “flatten the curve” [4, 5]. An important limitation of the TTI strategy is transmission of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) by COVID-19 carriers without symptoms. Given their lack of symptoms, they may not be tested and remain unidentified by the TTI process despite being a possible source of viral transmission [6]. Recent studies show that this subpopulation may account for as much as half of COVID-19 transmissions [6, 7].

An instrument to continuously monitor social-distancing behavior and symptoms in the population at a local level may support and improve the TTI process by decreasing the delay in identification of risk areas and populations. Research using voluntary symptom self-reporting apps performed in the United Kingdom, the United States of America, and Israel show promising results in the local prediction of COVID-19 using symptom-based tracking [8–10]. However, we find no apps using voluntary social-distancing behavior-reporting to track local COVID-19 hotspots.

During the first COVID-19 wave in the Netherlands, the Leiden University Medical Center (LUMC) and the tech company ORTEC developed and introduced the COVID Radar app. This questionnaire-based app allows individuals to anonymously report COVID-related symptoms and social-distancing behaviors on a regional and population level. The app provides users with direct feedback on, and peer comparison with, their reported social-distancing behavior and symptoms. Our theory is that tracking of symptom and social-distancing behavior data at a population level can be used to identify regions where more COVID-19 cases will subsequently occur, allowing (regional) policy makers and healthcare professionals to affect changes to regulations earlier, and thus more effectively.

In this first descriptive study, our aim is to observe the associations between self-reported symptoms, social-distancing behavior, and self-reported COVID-19 infection by the app’s users (i.e. criterion validity), and the associations between these variables and state-reported COVID-19 infections by the National Institute for Public Health and the Environment (i.e. external validation).

Methods

COVID radar app

The COVID radar app was released on the 2^nd of April 2020 following a short publicity campaign in the local and national media [11]. The app is free to download and allows for multiple user accounts from the same household on one smartphone. The app is not age-limited, meaning children are allowed to download and use the app. Over 85% of the households in COVID radar’s user population with minors under 18 years of age are linked to an adult smartphone. Upon first use of the app, users are asked to provide informed consent to share the following information with the research institution as stipulated by the conditions of the European General Data Protection Regulation. Users may opt out by either removing the app or by requesting the data manager to remove all data collected from that individual. Users are asked to register by entering the four digits of their postal code, gender (Male/Female/Other/Not Specified), age category (0–5, 6–11, 12–18, 19–29, ten-year increments from 30–80 and a category for 80+), and occupation (healthcare, education, catering industry, or other occupation with high risk of close contact). Following the initial setup, users are asked to report their symptoms and behavior daily via a questionnaire. A push-reminder is sent every-other day to users reminding them to do so. Fig 1 shows screenshots of the app and S1 Table shows a list of the questions users are asked.

Each observation is comprised of questions assessing symptoms, social-distancing behavior, whether or not the user has been exposed to an individual with COVID-19 in the past 2 weeks, and a user’s COVID-19 test history. The questions asked were periodically updated with the addition/removal dates of each question detailed in the online supplement. Via maps displayed within the app, users are presented with regional incidences of symptoms and personal feedback on their social-distancing behavior compared to regional and national means (Fig 1).

Data are transferred daily to a safe data environment within the Information Technology system of the LUMC (S2 Fig). Following importation of the daily data, we exclude observations from users who had requested to opt out, observations listing nonexistent postcodes, and double measurements within one user. Given users are asked if they have tested positive for SARS-CoV-2 within the past two weeks, we considered users SARS-CoV-2 positive/negative if they indicated a SARS-CoV-2 test result at least twice in the app, with the date of the first report used as day zero. More details on the development of the app, selection of the chosen questions, (external) data sources, and data cleaning is available in the supplement. Ethical approval was provided by the Medical Ethical Board of the LUMC (dossier number N20.067), which gave permission to refrain from obtaining consent from parents or guardians as data collection was anonymous. Only data on age category, profession and four digit postal code was collected rendering the data was untraceable to an individual.

Comparison of included/excluded observations

Following the data cleaning process detailed in the online supplement, we compared the available data in the excluded cohort with that of the included cohort. For each of the binary (symptom) variables collected by the app, we compared the proportion of excluded and included observations reporting this symptom. For each of the continuous social-distancing behavior variables, we compared the mean values for the included/excluded cohorts.

Descriptive statistics

To describe participant characteristics, we used histograms to explore age distributions of the app users, the number of times the app was used each day, and the number of times individual users used the app. We further compared age, gender, and profession for users ever having tested positive with those never having tested positive for SARS-CoV-2.

Validation testing

Given the eventual goal of the COVID Radar app is to predict emerging hotspots, we tested the expected associations between symptoms/behavior and SARS-CoV-2 test outcome. We used user-reported test results as our outcome measure for criterion validity testing and cases reported by the National Institute for Public Health and the Environment (RIVM) as our outcome measure for external validation [2].

Criterion validity

As a measure of criterion validity, we explored associations between the binary symptom variables (e.g., cough, sore throat, loss of smell/taste) and the continuous social-distancing behavior variables (e.g., number of house outside house, number of people within 1.5m) within the cohort of users ever reporting a SARS-CoV-2 test. For users within this ever-tested cohort, we used the date of the test as day 0 and observed the 21 days before and after the test. We calculated the daily mean or proportion for each variable for the entire user- cohort. We then calculated the difference between ever-positive or ever-negative users’ reported values and the mean values for the entire user-cohort on that day. By comparing data from the same days, we eliminated bias introduced by variations in time due to the various lock-down measures implemented during the observation window, as well as seasonal effects on symptoms. The mean values and 95% confidence intervals for these differences were then plotted to show how the ever-positive and ever-negative cohorts compared to the cohort as a whole with regard to these variables in the days surrounding a test. Given the formulation of the question (“Have you tested positive/negative for SARS-CoV-2 in the past two weeks”), the date of the test cannot be determined for those answering this question in the 14 days following the implementation of the question about testing in the app. Given this and the fact that this analysis involved looking at the 14 days prior to a test, users reporting a SARS-CoV-2 test in the 14 days following implementation of the question about testing were not included in this analysis.

External validation

As a measure of external validation, we compared per-capita user-reported COVID-19 status among the 12 Dutch provinces with per-capita rates as reported by RIVM over the course of the pandemic [2]. Within each province, we plotted 7-day backward looking moving averages of the daily proportion of users reporting each symptom variable alongside the daily nationally reported COVID-19 case counts and the weekly proportions of users reporting each symptom variable alongside the number of Rhinovirus cultures reported by Dutch laboratories [12]. We further plotted daily means and 7-day backward looking moving averages of each social-distancing behavior variables and qualitatively observed how well they reflect nationally applied lockdown-measures and holidays.

Sensitivity analyses

We repeated the above-described analyses for (a) the cohort of users using the app an above-median number of times during the observation period, (b) the cohort excluding healthcare professionals, and (c) the cohort excluding inhabitants of the province ’Zuid-Holland’, the home province of the LUMC where the app was created and users were most exposed to COVID Radar app-related media and advertisements. All statistical analyses were performed in STATA 16.1 (StataCorp, College Station, USA). STATA syntaxes for all analyses are provided in the online supplement.

Results

In the period 2 April, 2020 to 31 January, 2021 (305 days), the COVID Radar app was downloaded by 278,523 unique users who filled in the in-app questionnaire 6,202,606 times. A total of 102,445 (1.65%) observations were excluded (S4 Fig).

Comparison of included/excluded observations

The data for the 102,445 (1.65%) excluded observations were fairly representative of the included observations’ data in terms of symptoms and behavior. However, excluded observations were less often from a health professional and showed a slightly different age distribution (i.e. older age groups are over-represented in the excluded cohort) (see S2 Table).

Descriptive statistics

The age distribution of the app’s users showed a fairly consistent distribution of users 18–69 years old, and an under-representation of young (<18) and old (>70) users. Female users were overrepresented compared to national figures (See S5 Fig). The number of observations (questionnaires answered) per day dropped from over 100,000 in the first week of the app to a steady-state of around 10,000 observations per day during the course of the observation window (2 April, 2020 to 31 January, 2021) (See Fig 2).

The effects of the push reminder sent every-other day to all users is seen in the periodicity in the number of observations between even and odd days. The number of daily observations was highest in the province Zuid-Holland, the home province of the LUMC where the app was conceived and advertised (see Fig 3).

The number of observations per user ranged from 1 to 305 with a median value of 6 (25^th,75^th percentiles [p₂₅, p₇₅] = 2, 21) (See Fig 4).

Criterion validation

From a total of 278,523 unique users, 1,981 (0.71%) reported ever testing positive and 1214 (0.44%) negative for SARS-CoV-2. Ever-positive users were more likely to be women, older than 40 years of age, and healthcare professionals (Table 1).

Table 1. Comparison of users never vs. ever tested SARS-CoV-2 positive.

Number of days	305
Number of users	278.523
Number of observations	6.100.261
Observations per user, median and Interquartile range (IQR)	6		[2; 21]
	Never SARS-CoV-2 +		Ever SARS-CoV-2+
	N	(%)	N	(%)
Total	276.542	99,3%	1.981	0,71%
Gender (female)	152.515	55,2%	1.201	60,6%
Profession
Catering	259	0,1%	9	0,5%
Education	1.427	0,5%	77	3,9%
Healthcare	28.830	10,4%	372	18,8%
Other	16.510	6,0%	412	20,8%
Other_Contact	1.239	0,4%	35	1,8%
Missing	228.277	82,5%	1.076	54,3%
Age
0–5	9.033	3,3%	1	0,1%
06–11	11.273	4,1%	17	0,9%
12–18	17.635	6,4%	72	3,6%
19–29	34.309	12,4%	102	5,1%
30–39	39.305	14,2%	157	7,9%
40–49	41.887	15,1%	327	16,5%
50–59	54.718	19,8%	604	30,5%
60–69	45.993	16,6%	476	24,0%
70–79	20.292	7,3%	208	10,5%
80+	2.097	0,8%	17	0,9%

Open in a new tab

SARS-CoV-2: Severe Acute Respiratory Syndrome Coronavirus 2.

The proportion of users reporting the eight symptom variables increased beginning approximately 7 days prior to a positive test. This increase was smaller in the cohort of negative tested users (Fig 5a and 5b).

Fig 5 — a) Daily reported proportion of users reporting symptoms surrounding positive SARS-CoV-2 test. b) Daily reported proportion of users reporting symptoms surrounding negative SARS-CoV-2 test. Data with 95% confidence intervals in online supplement.

The continuous social-distancing behavior-based variables likewise showed above-mean values in this ever-positive cohort until approximately 7 days prior to a positive test, at which point they sharply decreased to remain below-mean in the week before and after a positive test. These fluctuations were not seen in users testing negative for SARS-CoV-2 (see Fig 6a and 6b).

Fig 6 — a) Mean difference between SARS-CoV-2 positive users and daily means for behavior variables in the days surrounding positive SARS-CoV-2 test. Red line: no difference. (example: on day 20 before test, on average a positive tested user reported one extra person within 1.5m (blue line) compared with the daily mean). B) Mean differences between SARS-CoV-2 negative users and daily means for behavior variables in te days surrounding negative SARS-CoV-2 test. Data with 95% confidence intervals in supplements.

External validation

As of early January 2021, almost one million cases of COVID-19 had been reported in the Netherlands by the National Institute for Public Health and the Environment (RIVM). The RIVM-reported daily case counts varied from 0 to over 13,000 cases per day. Positive SARS-CoV-2 tests reported in the COVID Radar app alongside the case count as reported by the RIVM for each province show that the association between these two is highest in provinces with a higher number of users, especially Zuid-Holland (Fig 7).

Fig 7 — (the number written beside the name of each province represents the mean number of weekly unique users per 100.000 inhabitants in that province during the study period).

Symptoms and social-distancing behavior varied over time, with both showing a clear temporal association with RIVM-reported case counts over time (Figs 8 and 9).

Plotting the RIVM-reported number of reported positive cultures of Rhinovirus alongside our symptom data suggests variables ‘fever’, ‘pain in the chest’ and ‘loss of smell’ are associated with COVID-19 case count while variables ‘coughing’ and ‘sore throat’ correlated more closely with Rhinovirus cultures (Fig 10).

The daily mean number of people within 1.5 meters declined sharply around the middle of September, reflecting the national lockdown measures introduced, and showed peaks during national holidays (Fig 11).

Fig 11 — December: holidays, followed by closing of non-essential shops.

The variable ‘number of visitors’ likewise showed peaks in the period around Christmas and New Year’s Eve (Fig 12).

Sensitivity analyses

These analyses were repeated using (a) only users reporting an above median number of observations (referred to as ‘faithful’ users), (b) only users outside the province Zuid-Holland, and (c) only non-healthcare professionals. Differences in the results for these three sensitivity analyses were minimal and none of the trends seen here were reversed (data shown in supplements).

Discussion

Since April 2020, the COVID Radar app has collected over 6 million user-provided questionnaires detailing COVID-related symptoms and social-distancing behaviors from over 275,000 unique users within the Netherlands. Symptom and behavior data were temporally associated with user-reported SARS-CoV-2 tests. A correlation between in-app reported case count and national-reported case counts was likewise seen, especially in provinces with high user-engagement. Social-distancing behavior variables showed the expected pattern in relation to national applied lockdown measures and holidays.

Criterion validity

Our qualitative (visual) association testing showed clear associations between both user-reported symptoms and user-reported social-distancing behavior, and user-reported SARS-CoV-2 test results. While not here quantified, some variables (e.g. ‘fever’, ‘pain in the chest’ and ‘loss of smell’) were more closely associated with case-count than others (e.g. ‘coughing’ and ‘sore throat’), which seemed as associated with Rhinovirus as with SARS-CoV-2. These associations are supported by prior research [13–16]. The pattern of social-distancing behaviors within the cohort of users who eventually report a positive SARS-CoV-2 test was particularly interesting. This cohort showed above-mean risk social-distancing behavior (e.g. more people within 1.5m, more visitors at home) between 20 and 10 days prior to a positive test (i.e. the period during which transmission likely occurred), at which point their social-distancing behavior quickly drops to a below-mean value as they became symptomatic and decided to be tested. The extent of above mean risk behavior was lower in users eventually testing negative.

External validity

Comparing COVID radar data to external data sources showed logical (temporal) associations in symptoms, social-distancing behavior, and test results. The strongest associations were observed in regions with high user-engagement. Given the symptoms tracked by the app are common both to SARS-CoV-2 and other respiratory tract infections, future efforts directed at prediction will need to correct for Rhinovirus and other viruses using viral surveillance data from Dutch laboratories. The extent and types of restrictions imposed on the Dutch population varied during the observation period and their effects were clearly visible in the social-distancing variables reported by users.

Comparison of excluded and included observations showed slight differences in age distribution but relative consistency in other variables. The small size of the excluded cohort minimized the risk of bias being introduced via this exclusion step. There was a large variance in the number of observations per user, with some users answering questionnaires daily while others filled in the app only once during the observation period. While it is reasonable to assume more faithful users may provide more accurate data, sensitivity analyses performed using data from users with an above-median number of app entries show no significant differences as compared to our primary analyses. The lack of a clear difference in the results when analyzing users of different engagement-levels suggests any bias introduced by differences in the reporting habits of these users was small.

There was an overrepresentation of users from the province Zuid-Holland in our data, due to Zuid-Holland being the home province of the LUMC, the hospital in charge of app design/analysis. This also likely explains the over-representation of health care professionals, to whom the app was thoroughly advertised within the environment of the LUMC. Despite this overrepresentation, our sensitivity analyses excluding Zuid-Holland users and healthcare professionals showed similar results, suggesting any bias introduced by their overrepresentation is minimal. COVID radar users were more often female and middle-aged. This was due to the overrepresentation of healthcare workers (who were more often female and mid-aged). However the sensitivity analysis excluding healthcare workers resulted in no different conclusions.

Noteworthy too is the fact that fully 30% of those users reporting a positive SARS-CoV-2 test reported no symptoms on the day of the positive test (data shown in S8 Fig). This is in line with the estimated number of COVID-19 carriers without symptoms, as reported by other studies [6, 7]. Our analysis likewise showed loss of smell and cough may continue for weeks following the positive SARS-CoV-2 test, as also confirmed in previous studies [17].

Limitations

All data in the app was self-reported and thus subjected to differences in personal interpretation of the questions. However, we do not expect differential misclassification as we see logical trends in symptoms and behavior on both individual and national levels. State-reported case counts were those reported by RIVM, whose data should include tests performed in private practices as they are required to be forwarded to RIVM. However, as there is no oversight for this process, the RIVM-reported case-counts likely represent under-estimates of the number of confirmed cases [18].

COVID radar additionally provided direct feedback to users on how their symptoms and behavior compared to their peers which likely has an effect on user behavior. This may bias the generalizability of COVID radar data, especially behavioral data. The effect of this feedback loop on users’ behavior would be expected to lead to an overly conservative estimate of the behavior of the population. Despite this, expected changes in reported behavior in the periods following national holidays and changes to social distancing policies are observed in COVID radar data. Additionally, altered behavior due to app-feedback would be expected to be observed in more loyal users of the app. Our sensitivity analysis on loyal users showed no significant difference in reported behavior. Given these realities, while we accept that app feedback altering user behavior has the potential to bias our results, we feel any bias introduced has been shown here to be small.

Testing capacity in the Netherlands was low during the developmental stage of the app and has increased during the study period. In the final months of 2020, testing was expanded to include those without symptoms. As a result, the prevalence of COVID-19 in the Netherlands could be underestimated. Because of this change in testing policy, the question regarding negative tests was implemented at a later date resulting in less data about negative tests compared to data on positive tests during a shorter period of time. Nonetheless, we were able to show that the association between symptoms and a negative test is less apparent than their associations with a positive test, suggesting our conclusions remain valid. Also, testing of the underaged (<12 years) was rare during the study period, resulting in a relatively old SARS-CoV-2 positive cohort in this study.

Future implications

Having validated the expected associations between symptoms, social-distancing behavior, and COVID case-count, our next steps will involve attempted prediction of emerging hotspots by combining symptom and social-distancing behavior data to quantify risk of COVID-19 cases. Such predictions could be used to help guide COVID-19 policy. Our study indicated the quality of the submitted data is best where user-engagement is high. Prediction-based goals will thus be aided by increasing user count. Regional predictions may additionally be improved through incorporation of data from general practitioners, more detailed demographic data, and mobility data using a machine learning based approach. Another possibility for further research is testing of associations between regional SARS-CoV-2 cases, symptoms, behavior and other regional data related, for example, to the physical environment.

Conclusion

The COVID Radar app successfully collects anonymous, user-reported data on COVID-19-related symptoms and social-distancing behavior. Initial validation showed symptoms and behavior reported within the app are correlated with in-app reporting of a SARS-CoV-2 test. The predictive potential of the COVID Radar is demonstrated as external validation showed in-app reported positive SARS-CoV-2 tests track well with state-reported case counts. Future research will focus on regional predictions using these data.

Supporting information

S1 Fig. Screenshot of the app.

(TIF)

Click here for additional data file.^{(500.4KB, tif)}

S2 Fig. Dataflow of data in COVID radar.

(TIF)

Click here for additional data file.^{(106.5KB, tif)}

S3 Fig. Overview of data cleaning.

(TIF)

Click here for additional data file.^{(116KB, tif)}

S4 Fig. Flowchart of Inclusion of data.

(TIF)

Click here for additional data file.^{(46.5KB, tif)}

S5 Fig. Age and gender distribution of app’s users compared to national data.

(TIF)

Click here for additional data file.^{(45.7KB, tif)}

S6 Fig. Mean difference between values of users reporting symptoms and daily proportion per day surrounding positive SARS-CoV-2 test (red line: No difference with daily proportion) with 95% confidence intervals.

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S7 Fig. Mean difference between values of users reporting behavior and daily means per day surrounding positive SARS-CoV-2 test (red line: No difference with daily mean) with 95% confidence intervals.

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S8 Fig. Proportion of users asymptomatic per day surrounding positive SARS-CoV-2 test.

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S9 Fig. Mean difference between number of symptoms reported by users reporting symptoms and daily mean number per day surrounding positive SARS-CoV-2 test (red line: No difference with daily proportion) with 95% confidence intervals.

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S10 Fig. As S6 Fig, but than for negative SARS-CoV-2 test.

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S11 Fig. As. S7 Fig, but than for negative SARS-CoV-2 test.

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S12 Fig. As S8 Fig, but than for negative SARS-CoV-2 test.

(TIF)

Click here for additional data file.^{(3.4MB, tif)}

S13 Fig. As S9 Fig, but than for negative SARS-CoV-2 test.

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S14 Fig. Daily mean number of hours out of house (with 7-day backward moving average in black).

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S15 Fig. Daily proportion of users working/going to school outside of house (with 7-day backward moving average in black).

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S16 Fig. Daily mean number of persons spoken with within 1.5 meters (with 7-day backward moving average in black).

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S1 File. Syntax importation data STATA.

(DO)

Click here for additional data file.^{(483B, do)}

S2 File. Syntax selection of exclusions STATA.

(DO)

Click here for additional data file.^{(1.2KB, do)}

S3 File. Syntax importation external data STATA.

(DO)

Click here for additional data file.^{(6.2KB, do)}

S4 File. Syntax cleaning data STATA.

(DO)

Click here for additional data file.^{(9.2KB, do)}

S5 File. Syntax analyzes data, generation figures and sensitivity analyzes.

(DO)

Click here for additional data file.^{(145KB, do)}

S6 File. Supplemental text overview of development of the app.

(DOCX)

Click here for additional data file.^{(21.5KB, docx)}

S7 File. Sensitivity analyses (S5–S16 Figs repeated without below median reporting users, healthcare professionals and inhabitants of the province Zuid-Holland).

(ZIP)

Click here for additional data file.^{(9.7MB, zip)}

S1 Table. Overview of questions asked in the app.

(XLSX)

Click here for additional data file.^{(88.2KB, xlsx)}

S2 Table. Comparison of excluded and included data.

(XLSX)

Click here for additional data file.^{(16KB, xlsx)}

Data Availability

Because of legal restrictions in both the agreement between app users and the LUMC (research institution) and in the European General Data Protection Regulation, the authors are not allowed to share the entire dataset online. The data contain potentially identifying information (location, age and gender) and sensitive patient information (for example SARS-CoV-2 test result). Aggregated data can be shared upon request with “medical government or academic research institutions” only. These data requests can be sent to the dedicated data manager I. De Jong (I.de_Jonge@lumc.nl) of the METC of Leiden University Medical Center (metc-ldd@lumc.nl).

Funding Statement

Funding for the project was obtained from ZonMW (the Netherlands Organization of Health Research and Development, grant numbers: 10430042010016, 10430022010001 and 10430032010011). The company, ORTEC, provided support in the form of the salary of author MeB. The funder provided support in the form of salaries for authors WJD and NS. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

1.Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time (vol 20, pg 533, 2020). Lancet Infect Dis. 2020;20(9):E215–E. [DOI] [PMC free article] [PubMed] [Google Scholar]
2.(RIVM) RvVeM. Actuele informatie over het nieuwe coronavirus COVID-19 2020 [cited 2021 01-03-2021]. https://data.rivm.nl/geonetwork/srv/dut/catalog.search#/search?resultType=details&any_OR__title=covid-19&isChild=’false’&fast=index&sortBy=relevance&from=1&to=20.
3.(NICE) NICE. Jaarboek 2018: Stichting NICE; 2019. 54 p.
4.Girum T, Lentiro K, Geremew M, Migora B, Shewamare S. Global strategies and effectiveness for COVID-19 prevention through contact tracing, screening, quarantine, and isolation: a systematic review. Trop Med Health. 2020;48:91. doi: 10.1186/s41182-020-00285-w [DOI] [PMC free article] [PubMed] [Google Scholar]
5.Rajan S, C JD, McKee M. What do countries need to do to implement effective ’find, test, trace, isolate and support’ systems? J R Soc Med. 2020;113(7):245–50. Epub 2020/07/15. doi: 10.1177/0141076820939395 . [DOI] [PMC free article] [PubMed] [Google Scholar]
6.Buitrago-Garcia D, Egli-Gany D, Counotte MJ, Hossmann S, Imeri H, Ipekci AM, et al. Occurrence and transmission potential of asymptomatic and presymptomatic SARS-CoV-2 infections: A living systematic review and meta-analysis. PLoS Med. 2020;17(9):e1003346–e. doi: 10.1371/journal.pmed.1003346 . [DOI] [PMC free article] [PubMed] [Google Scholar]
7.Johansson MA, Quandelacy TM, Kada S, Prasad PV, Steele M, Brooks JT, et al. SARS-CoV-2 Transmission From People Without COVID-19 Symptoms. JAMA Netw Open. 2021;4(1):e2035057. Epub 2021/01/08. doi: 10.1001/jamanetworkopen.2020.35057 . [DOI] [PMC free article] [PubMed] [Google Scholar]
8.Varsavsky T, Graham MS, Canas LS, Ganesh S, Pujol JC, Sudre CH, et al. Detecting COVID-19 infection hotspots in England using large-scale self-reported data from a mobile application: a prospective, observational study. medRxiv. 2020:2020.10.26.20219659. doi: 10.1016/S2468-2667(20)30269-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020;369:m1328. doi: 10.1136/bmj.m1328 [DOI] [PMC free article] [PubMed] [Google Scholar]
10.Menni C, Valdes AM, Freidin MB, Sudre CH, Nguyen LH, Drew DA, et al. Real-time tracking of self-reported symptoms to predict potential COVID-19. Nature Medicine. 2020;26(7):1037–40. doi: 10.1038/s41591-020-0916-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
11.(LUMC) LUMC. COVID Radar 2020 [2020-12-1]. https://www.lumc.nl/org/coronavirus/covid-radar/.
12.RIVM. Virologische Weekstaten Viral Surveillance https://www.rivm.nl/virologische-weekstaten/onderzoek:RIVM; 2020 [cited 2020 9-12-2020].
13.Gwaltney JM Jr., Hendley JO, Simon G, Jordan WS Jr. Rhinovirus Infections in an Industrial Population: II. Characteristics of Illness and Antibody Response. JAMA. 1967;202(6):494–500. doi: 10.1001/jama.1967.03130190100014 [DOI] [PubMed] [Google Scholar]
14.Pappas DE, Hendley JO, Hayden FG, Winther B. Symptom Profile of Common Colds in School-Aged Children. The Pediatric Infectious Disease Journal. 2008;27(1):8–11. doi: 10.1097/INF.0b013e31814847d9 [DOI] [PubMed] [Google Scholar]
15.Stokes EK, Zambrano LD, Anderson KN, Marder EP, Raz KM, Felix SEB, et al. Coronavirus disease 2019 case surveillance—United States, January 22–May 30, 2020. Morbidity and Mortality Weekly Report. 2020;69(24):759. doi: 10.15585/mmwr.mm6924e2 [DOI] [PMC free article] [PubMed] [Google Scholar]
16.Tong JY, Wong A, Zhu D, Fastenberg JH, Tham T. The prevalence of olfactory and gustatory dysfunction in COVID-19 patients: a systematic review and meta-analysis. Otolaryngology–Head and Neck Surgery. 2020;163(1):3–11. doi: 10.1177/0194599820926473 [DOI] [PubMed] [Google Scholar]
17.Lee Y, Min P, Lee S, Kim SW. Prevalence and Duration of Acute Loss of Smell or Taste in COVID-19 Patients. J Korean Med Sci. 2020;35(18):e174. Epub 2020/05/10. doi: 10.3346/jkms.2020.35.e174 . [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Eenvandaag. Ruim een derde van alle coronatests niet door GGD, maar particulier: ’We verliezen het zicht op het virus’ [website]. 2020 [cited 2021 2021-01-01]. https://eenvandaag.avrotros.nl/item/ruim-een-derde-van-alle-coronatests-niet-door-ggd-maar-particulier-we-verliezen-het-zicht-op-het/.

PLoS One. doi: 10.1371/journal.pone.0253566.r001

Decision Letter 0

Simone Lolli

13 Apr 2021

PONE-D-21-03787

COVID RADAR app: Description and Validation of Population surveillance of Symptoms and Behavior in relation to COVID-19.

PLOS ONE

Dear Dr. Kiefte,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please take into consideration all the reviewers comments/issues in editing the new version. Especially pay attention in acknowledging any other potential limitations and biases present in a self-selecting app-based population, and add some more formal analysis to support the claims being made.

Please submit your revised manuscript by May 28 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Simone Lolli

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that Figure 1 and Figure S1 in your submission contain map images which may be copyrighted.

All PLOS content is published under the Creative Commons Attribution License (CC BY 4.0), which means that the manuscript, images, and Supporting Information files will be freely available online, and any third party is permitted to access, download, copy, distribute, and use these materials in any way, even commercially, with proper attribution. For these reasons, we cannot publish previously copyrighted maps or satellite images created using proprietary data, such as Google software (Google Maps, Street View, and Earth). For more information, see our copyright guidelines: http://journals.plos.org/plosone/s/licenses-and-copyright.

We require you to either (a) present written permission from the copyright holder to publish these figures specifically under the CC BY 4.0 license, or (b) remove the figures from your submission:

a. You may seek permission from the original copyright holder of Figure 1 and Figure S1 to publish the content specifically under the CC BY 4.0 license.

We recommend that you contact the original copyright holder with the Content Permission Form (http://journals.plos.org/plosone/s/file?id=7c09/content-permission-form.pdf) and the following text:

“I request permission for the open-access journal PLOS ONE to publish XXX under the Creative Commons Attribution License (CCAL) CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). Please be aware that this license allows unrestricted use and distribution, even commercially, by third parties. Please reply and provide explicit written permission to publish XXX under a CC BY license and complete the attached form.”

Please upload the completed Content Permission Form or other proof of granted permissions as an "Other" file with your submission.

In the figure caption of the copyrighted figure, please include the following text: “Reprinted from [ref] under a CC BY license, with permission from [name of publisher], original copyright [original copyright year].”

b. If you are unable to obtain permission from the original copyright holder to publish these figures under the CC BY 4.0 license or if the copyright holder’s requirements are incompatible with the CC BY 4.0 license, please either i) remove the figure or ii) supply a replacement figure that complies with the CC BY 4.0 license. Please check copyright information on all replacement figures and update the figure caption with source information. If applicable, please specify in the figure caption text when a figure is similar but not identical to the original image and is therefore for illustrative purposes only.

The following resources for replacing copyrighted map figures may be helpful:

USGS National Map Viewer (public domain): http://viewer.nationalmap.gov/viewer/

The Gateway to Astronaut Photography of Earth (public domain): http://eol.jsc.nasa.gov/sseop/clickmap/

Maps at the CIA (public domain): https://www.cia.gov/library/publications/the-world-factbook/index.html and https://www.cia.gov/library/publications/cia-maps-publications/index.html

NASA Earth Observatory (public domain): http://earthobservatory.nasa.gov/

Landsat: http://landsat.visibleearth.nasa.gov/

USGS EROS (Earth Resources Observatory and Science (EROS) Center) (public domain): http://eros.usgs.gov/#

Natural Earth (public domain): http://www.naturalearthdata.com/

3. Please provide additional details regarding participant consent.

In the ethics statement in the Methods and online submission information, please ensure that you have specified (a) whether consent was suitably informed and (b) what type you obtained (for instance, written or verbal). If your study included minors under age 18, state whether you obtained consent from parents or guardians.

If the need for consent was waived by the ethics committee, please include this information.

4. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

5. Thank you for stating the following financial disclosure:

'The funding organization had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.'

At this time, please address the following queries:

Please clarify the sources of funding (financial or material support) for your study. List the grants or organizations that supported your study, including funding received from your institution.
State what role the funders took in the study. If the funders had no role in your study, please state: “The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”
If any authors received a salary from any of your funders, please state which authors and which funders.
If you did not receive any funding for this study, please state: “The authors received no specific funding for this work.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

6. Thank you for stating the following in your Competing Interests section:

'The Covid-Radar was developed by Ortec and is freely available in the app- and playstore. Data of the covid-radar was transferred to LUMC for independent analyses. Funding for the project was obtained from ZoNMW (the Netherlands Organization of Health Research and Development, grant number: 10430042010016, 10430022010001 and 10430032010011. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.'

a. Please complete your Competing Interests statement to state any Competing Interests. If you have no competing interests, please state "The authors have declared that no competing interests exist.", as detailed online in our guide for authors at http://journals.plos.org/plosone/s/submit-now

b. This information should be included in your cover letter; we will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

7. Thank you for stating the following in the Financial Disclosure section:

'The funding organization had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.'

We note that one or more of the authors are employed by a commercial company: Ortec B.V.

a. Please provide an amended Funding Statement declaring this commercial affiliation, as well as a statement regarding the Role of Funders in your study. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. You can update author roles in the Author Contributions section of the online submission form.

Please also include the following statement within your amended Funding Statement.

“The funder provided support in the form of salaries for authors [insert relevant initials], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.”

If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement.

b. Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc.

Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests) . If this adherence statement is not accurate and there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

c. Please include both an updated Funding Statement and Competing Interests Statement in your cover letter. We will change the online submission form on your behalf.

Additional Editor Comments:

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is interesting work. I've a number of commments and suggestions that I think could further improve the quality of this manuscript; including better presentation, more acknowledgement of the potential limitations and biases present in a self-selecting app-based population, and some more formal analysis to support the claims being made.

Abstract, line 63: should be have, not having. And I'm not entirely sure what you mean by "Symptom variables increased noticeably in the week" - do you mean reports of users experiencing symptoms increased?

Were there any limits on who could sign up to the app - e.g., were children allowed to? Were users encouraged to report on behalf of others on the multiple accounts, or were people asked to each fill out their own reports?

Line 126 - I'd prefer 'individual with Covid-19' to the more colloquial 'Corona patient'.

Giving user feedback in the app may alter their behaviour and make the app less representative of the population behaviours. Did you see any evidence of this? I think this merits a bit more dicussion that already present as it has the potentialy to significantly bias your app poulation, and undermine the ability of the Covid RADAR to detect wider population changes.

I think at times the authors have put too much into the supplementary material and it has prevented the manuscript from being self contained. For example, line 135 "Following the data cleaning process detailed in the online supplement..." It may be that the cleaning process is extensive and a full description of it best belongs in a supplement, but I think it would help readers if a brief overviewing of the process was included in the main text.

Many of the figures don't have labels for the y-axis, please add.

There are a huge number of figure. It's good to have included so much detail, but I think some of these could be condensed into single figures with multiple panels. I think this would make it easier for readers to engage with and understand your work; it is difficult to do so when frequently scrolling between the texts and figures. For example, figures 5 and 6 could be combined into one.

It is well documented that populations of participatory digital surveys tend to not be representative of the population - they tend to be more female, younger, more educated. It appears the Covid RADAR is no exception; it would be help if this was made explicitr by comparison of your demographic to the country's general population, and the implications of this discussed. There are also harder to quantify biases: signing up to the app suggests a greater interest in Covid-19 than the average person, which might mean e.g. they are more likely to take the rules seriously and stick to them. It is very difficult to quantify the effect of this but it should be acknowledged as a limitation.

It is suprising that the Covid positive population tended to be older than the negative population. Could you comment on this - does this finding agree with other data sources?

Figure 9: are you able to show number of tests conducted as well as number of positive tests; as it's difficult to see whehther the peaks are true rises in population Covid-19 or just due to increased testing. The comparison is qualitative, and should be supported by measuring the quantiative correleation between your app data and cases; this would also help you formally justify your claim that the correlation is higher in regions with greather app participation.

Your claims that "‘fever’, ‘pain in the chest’ and ‘loss of smell’ are associated with COVID-19 case count while variables ‘coughing’ and ‘sore throat’ correlated more

closely with Rhinovirus cultures" is again, completely based on qualitative comparison and would be strenghtened by quantitative measurements. These findings appear to me consistent with findings in the literature, which is encouraging, and I think this should be noted.

Reviewer #2: The submitted manuscript reports analysis from data obtained from COVID RADAR app available in The Netherlands during COVID-19 pandemic outbreak (2020-2021). Overall, the analysis makes sense and the results are coherent. I have only a minor issue to address. As the app is based on self-reporting by app users, it should be specified that a bias toward serious illness exists, as the hospitalized users into Intensive Care Unit are likely not reporting their symptoms. This is a limitation in the analysis because it would be have been interesting to infer which symptoms and which areas might be prone to serious illness, i.e. also try to assess if pollution is playing a role. Other minor comments regard the manuscript, where sometimes too colloquial words are used, e.g. Corona. Lastly, on plots, x and y axis labels should be added.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Mark Graham

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2021 Jun 30;16(6):e0253566. doi: 10.1371/journal.pone.0253566.r002

Author response to Decision Letter 0

3 Jun 2021

Journal Requirements:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming.

We have adjusted the style of the manuscript according to PLOS ONE’s style requirements.

2. We note that Figure 1 and Figure S1 in your submission contain map images which may be copyrighted.

Please upload the completed Content Permission Form or other proof of granted permissions as an "Other" file with your submission.

We uploaded the completed Content Permission Form and updated the figure caption.

3. Please provide additional details regarding participant consent.

It is possible for minors under age 18 to download and use the app. Local ethical committee gave permission to refrain from obtaining consent from parents or guardians because of the construction of the data collection as data collection was anonymous: only data on age category and four digit postal code was collected which was untraceable to an individual person individual.

Methods line 110-115: “Children are allowed to download and use the app. Over 85% of the households with minors under 18 years of age are linked to an adult smartphone. Upon first use of the app, users are asked to provide informed consent to share the following information with the research institution as stipulated by the conditions of the European General Data Protection Regulation. Users may opt out by either removing the app or by requesting the data manager to remove all data collected from that individual.”

Methods line 142-145: “, which gave permission to refrain from obtaining consent from parents or guardians because of the construction of the data collection as data collection was anonymous. As such, only data on age category and four digit postal code was collected which was untraceable to an individual person individual”

4. We note that you have indicated that data from this study are available upon request.

PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. In your revised cover letter, please address the following prompts:

a. If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

To the reviewer

Because of legal restrictions in both the agreement between app users and the LUMC (research institution) and in the European General Data Protection Regulation, we are not allowed to share the entire dataset online. Our data contain potentially identifying information (location, age and gender) and sensitive patient information (for example SARS-CoV-2 test result). Aggregated data can be shared upon request with “medical government or academic research institutions” only. These data requests can be sent to the dedicated data manager I. De Jong (I.de_Jonge@lumc.nl) of the METC of Leiden University Medical Center (metc-ldd@lumc.nl).

5. Thank you for stating the following financial disclosure:

'The funding organization had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. At this time, please address the following queries:

a. Please clarify the sources of funding (financial or material support) for your study. List the grants or organizations that supported your study, including funding received from your institution.

b. State what role the funders took in the study. If the funders had no role in your study, please state:

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

c. If any authors received a salary from any of your funders, please state which authors and which funders.

Salaries of first and second authors were funded by the ZonMW grants. ORTEC provided support in the form of the salary of author MeB. The other authors have declared that no competing interests exist.

6. Please complete your Competing Interests statement

a. to state any Competing Interests. If you have no competing interests, please state

"The authors have declared that no competing interests exist.",

ORTEC provided support in the form of the salary of author MeB. The other authors have declared that no competing interests exist.

7. Please provide an amended Funding Statement declaring the commercial affiliation ORTEC, as well as a statement regarding the Role of Funders in your study.

You can update author roles in the Author Contributions section of the online submission form.

“The funder provided support in the form of salaries for authors WJD and NS but did not play any additional role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are detailed in the ‘author contributions’ section. ORTEC provided support in the form of the salary of author MeB. This author contributed to the development and updates of the app. He also reviewed the methods section, specifically the technical and developmental details of the app. He did not play any additional role in the data analysis or presentation of results in this manuscript. Maintenance of the app and salary of the researchers was funded by ZonMw (previously mentioned).

"This does not alter our adherence to PLOS ONE policies on sharing data and materials. The restrictions on sharing of data are due to national legal regulations.”

Reactions to additional Editor Comments:

1. Reviewer #1: This is interesting work. I've a number of comments and suggestions that I think could further improve the quality of this manuscript; including better presentation, more acknowledgement of the potential limitations and biases present in a self-selecting app-based population, and some more formal analysis to support the claims being made.

Thank you for the acknowledgment of the relevance of this work. We are very grateful for the reviewer’s feedback.

2. Abstract, line 61: should be have, not having. And I'm not entirely sure what you mean by "Symptom variables increased noticeably in the week" - do you mean reports of users experiencing symptoms increased?

We reviewed the text of the manuscript by a native English speaker.

We changed this (line 55-57): Amongst users testing positive for SARS-CoV-2, the proportion of observations reporting symptoms was higher than that of the cohort as a whole in the week prior to a positive SARS-CoV-2 test.

3. Were there any limits on who could sign up to the app - e.g., were children allowed to? Were users encouraged to report on behalf of others on the multiple accounts, or were people asked to each fill out their own reports?

The only limit on signing up for the app was access to a smartphone. Children were allowed to download and use the app. Users had the option of reporting on behalf of others. Over half of the observations are from individual users reporting only about themselves.

We have added this to the manuscript (line 110): Children are allowed to download and use the app. Over 85% of the households with minors under 18 years of age are linked to an adult smartphone.

4. Line 123 - I'd prefer 'individual with Covid-19' to the more colloquial 'Corona patient'.

We added in line 128: “been exposed to an individual with COVID-19”

5. Giving user feedback in the app may alter their behaviour and make the app less representative of the population behaviours. Did you see any evidence of this? I think this merits a bit more discussion that already present as it has the potential to significantly bias your app population, and undermine the ability of the Covid RADAR to detect wider population changes.

We agree that COVID radar app users’ behavior may not be representative of the population as a whole due to feedback from the app leading to an altering of behavior. The effect of this feedback loop on users’ behavior would be expected to lead to an overly conservative estimate of the behavior of the population. Still logical changes in reported behavior in the periods following national holidays and changes to social distancing policies are seen in COVID radar data. The possibility of this bias is acceptable given the goal of our study is to demonstrate patterns, rather than quantify associations. Additionally, altered behavior due to app-feedback would be expected to be observed in more loyal users of the app. Our sensitivity analysis on loyal users showed no significant difference reported behavior. Given these realities, while we accept that app feedback altering user behavior has the potential to bias our results, we feel any bias introduced has been shown here to be small.

Added in discussion line 357-366: This may bias the generalizability of COVID radar data, especially behavioral data. The effect of this feedback loop on users’ behavior would be expected to lead to an overly conservative estimate of the behavior of the population. Despite this, expected changes in reported behavior in the periods following national holidays and changes to social distancing policies are observed in COVID radar data. Additionally, altered behavior due to app-feedback would be expected to be observed in more loyal users of the app. Our sensitivity analysis on loyal users showed no significant difference in reported behavior. Given these realities, while we accept that app feedback altering user behavior has the potential to bias our results, we feel any bias introduced has been shown here to be small.

6. I think at times the authors have put too much into the supplementary material and it has prevented the manuscript from being self contained. For example, line 135 "Following the data cleaning process detailed in the online supplement..." It may be that the cleaning process is extensive and a full description of it best belongs in a supplement, but I think it would help readers if a brief overviewing of the process was included in the main text.

We added a brief overview of the cleaning process of the data in the main text (line 134-139): Data were transferred daily to a safe data environment within the Information Technology system of the LUMC (supplemental Figure 2). Following importation of the daily data, we exclude observations from users who had requested to opt out, observations listing nonexistent postcodes, and double measurements within one user. We considered users COVID-19 positive/negative if they indicated a COVID-19 test result at least twice in the app, with the date of the first report used as day zero.

7. Many of the figures don't have labels for the y-axis, please add.

We have added Y-axis labels to all figures.

8. There are a huge number of figures. It's good to have included so much detail, but I think some of these could be condensed into single figures with multiple panels. I think this would make it easier for readers to engage with and understand your work; it is difficult to do so when frequently scrolling between the texts and figures. For example, figures 5 and 6 could be combined into one.

We grouped all supplementary material, resulting in several panels with 7-11 figures each. Figure 5 and 6 are combined into one panel (figure 5 a and b), just as Figure 7 and 8 (figure 6 a and b).

9. It is well documented that populations of participatory digital surveys tend to not be representative of the population - they tend to be more female, younger, more educated. It appears the Covid RADAR is no exception; it would be helpful if this was made explicit by comparison of your demographic to the country's general population, and the implications of this discussed. There are also harder to quantify biases: signing up to the app suggests a greater interest in Covid-19 than the average person, which might mean e.g. they are more likely to take the rules seriously and stick to them. It is very difficult to quantify the effect of this but it should be acknowledged as a limitation.

The remarks of the reviewer are correct. We added a figure in the supplements (S.5) displaying the demographics of the Netherlands and the COVID radar app. The educational level of the users of the COVID radar app is unknown. The several sensitivity analyses we performed suggest a small bias towards more engagement to social distancing rules in more loyal users. We added this to the discussion.

Added to Results line 214-215: Female users were overrepresented compared to the national figures.

Added to Discussion lines 339- 342: COVID radar users were more often female and middle-aged. This was due to the overrepresentation of healthcare workers (who were more often female and middle-aged). However the sensitivity analysis excluding healthcare workers showed no different conclusions.

10. It is surprising that the Covid positive population tended to be older than the negative population. Could you comment on this - does this finding agree with other data sources?

We assume the reviewer is commenting on table 1. In this table we compare users who report a positive COVID-19 test to users who don’t; (we do not report demographics from negative testing users!). The change of reporting a positive COVID-19 test is higher in more loyal app users, who are on average older. In addition, during the first two COVID-19 waves in the Netherlands, younger inhabitants (<12 years) often remained untested.

Added to Discussion line 374-376: Also, testing of underaged (<12 years) was rare during the study period. Resulting in a relative old SARS-CoV-2 positive cohort in this study.

11. Figure 7: are you able to show number of tests conducted as well as number of positive tests; as it's difficult to see whether the peaks are true rises in population Covid-19 or just due to increased testing. The comparison is qualitative, and should be supported by measuring the quantiative correleation between your app data and cases; this would also help you formally justify your claim that the correlation is higher in regions with greather app participation.

During the research period testing policy indeed varied. The number of tests in December was almost twice as high as in October, but the number of positive tests were almost the same. However, we would emphasize that app participation was independent of formal test practices in The Netherlands. In addition, the number of users did not change during change in test policy. Therefore we believe it will not influence the observation that correlations between app reported cases and true rises in population COVID-19 in those regions with greater app policy is affected by change in test policy.

12. Your claims that "‘fever’, ‘pain in the chest’ and ‘loss of smell’ are associated with COVID-19 case count while variables ‘coughing’ and ‘sore throat’ correlated more

The analyses of symptoms specific to Rhinovirus or COVID virus are indeed qualitative due to the data structure and in our analysis there is no possibility to perform more quantitative analyses on this topic. However, inspired by your remark we have added references to prior research on symptomaticity of Rhinovirus and COVID-19, indeed consistent with our findings.

See discussion lines 306: “These associations are consistent with those seen in prior research [13-16].”

This is a limitation in the analysis because it would be have been interesting to infer which symptoms and which areas might be prone to serious illness, i.e. also try to assess if pollution is playing a role. Other minor comments regard the manuscript, where sometimes too colloquial words are used, e.g. Corona. Lastly, on plots, x and y axis labels should be added.

We thank this reviewer for this feedback. We reviewed the manuscript for colloquial words and added labels to all axis. It is true that users who are admitted to the ICU are not able to report their symptoms. We did not mention the bias toward serious illness in the discussion because we believe this is outside the scope of this manuscript (which is ‘description of COVID radar data’) and outside the ultimate goal of this tool: to predict regional casecounts before positive COVID-19 tests are known. At the time patients are admitted to the ICU a result of a SARS-CoV-2 test is often known, given the deterioration of patients with COVID-19 is in the second week of symptomaticity (see Chen, et al. (2020). Pattern of Deterioration in Moderate Patients with COVID-19: A Multi-Center, Retrospective Cohort Study.). So this category of users is not relevant for the purpose of this study.

The remarks and suggestions for analyses of areas that might be prone to serious illness are not possible with COVID radar’s data because of no registration of individual data about ICU admission within the app, and flawed data about regional ICU admissions due to intensive inter ICU transportations of patients. However, analysis of regional differences of long term lasting COVID-19 symptomaticity (for example shortness of breath) and the regional distribution of them (enriched with data about air pollution) is possible with COVID radar’s data. This was outside this manuscript’s scope, but we have added this in suggestions for further research in the discussion (lines 386-388): Another possibility for further research is testing of associations between regional SARS-CoV-2 cases, symptoms, behavior and other regional data about for example related to the physical environment.

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file.^{(25.8KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0253566.r003

Decision Letter 1

Simone Lolli

9 Jun 2021

COVID RADAR app: Description and Validation of Population surveillance of Symptoms and Behavior in relation to COVID-19.

PONE-D-21-03787R1

Dear Dr. Kiefte,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Simone Lolli

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Reviewer #1: (No Response)

Reviewer #2: I am happy that the authors addressed all my previously raised issues. Now the manuscript is ready for publication after technical corrections, e.g. typos, low-res

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

PLoS One. doi: 10.1371/journal.pone.0253566.r004

Acceptance letter

Simone Lolli

21 Jun 2021

PONE-D-21-03787R1

COVID RADAR app: Description and Validation of Population surveillance of Symptoms and Behavior in relation to COVID-19

Dear Dr. Kiefte-de Jong:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Simone Lolli

Academic Editor

PLOS ONE

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

S1 Fig. Screenshot of the app.

(TIF)

Click here for additional data file.^{(500.4KB, tif)}

S2 Fig. Dataflow of data in COVID radar.

(TIF)

Click here for additional data file.^{(106.5KB, tif)}

S3 Fig. Overview of data cleaning.

(TIF)

Click here for additional data file.^{(116KB, tif)}

S4 Fig. Flowchart of Inclusion of data.

(TIF)

Click here for additional data file.^{(46.5KB, tif)}

S5 Fig. Age and gender distribution of app’s users compared to national data.

(TIF)

Click here for additional data file.^{(45.7KB, tif)}

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S8 Fig. Proportion of users asymptomatic per day surrounding positive SARS-CoV-2 test.

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S10 Fig. As S6 Fig, but than for negative SARS-CoV-2 test.

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S11 Fig. As. S7 Fig, but than for negative SARS-CoV-2 test.

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S12 Fig. As S8 Fig, but than for negative SARS-CoV-2 test.

(TIF)

Click here for additional data file.^{(3.4MB, tif)}

S13 Fig. As S9 Fig, but than for negative SARS-CoV-2 test.

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S14 Fig. Daily mean number of hours out of house (with 7-day backward moving average in black).

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S15 Fig. Daily proportion of users working/going to school outside of house (with 7-day backward moving average in black).

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S16 Fig. Daily mean number of persons spoken with within 1.5 meters (with 7-day backward moving average in black).

(TIF)

Click here for additional data file.^{(5.5MB, tif)}

S1 File. Syntax importation data STATA.

(DO)

Click here for additional data file.^{(483B, do)}

S2 File. Syntax selection of exclusions STATA.

(DO)

Click here for additional data file.^{(1.2KB, do)}

S3 File. Syntax importation external data STATA.

(DO)

Click here for additional data file.^{(6.2KB, do)}

S4 File. Syntax cleaning data STATA.

(DO)

Click here for additional data file.^{(9.2KB, do)}

S5 File. Syntax analyzes data, generation figures and sensitivity analyzes.

(DO)

Click here for additional data file.^{(145KB, do)}

S6 File. Supplemental text overview of development of the app.

(DOCX)

Click here for additional data file.^{(21.5KB, docx)}

S7 File. Sensitivity analyses (S5–S16 Figs repeated without below median reporting users, healthcare professionals and inhabitants of the province Zuid-Holland).

(ZIP)

Click here for additional data file.^{(9.7MB, zip)}

S1 Table. Overview of questions asked in the app.

(XLSX)

Click here for additional data file.^{(88.2KB, xlsx)}

S2 Table. Comparison of excluded and included data.

(XLSX)

Click here for additional data file.^{(16KB, xlsx)}

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file.^{(25.8KB, docx)}

Data Availability Statement

[pone.0253566.ref001] 1.Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time (vol 20, pg 533, 2020). Lancet Infect Dis. 2020;20(9):E215–E. [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0253566.ref002] 2.(RIVM) RvVeM. Actuele informatie over het nieuwe coronavirus COVID-19 2020 [cited 2021 01-03-2021]. https://data.rivm.nl/geonetwork/srv/dut/catalog.search#/search?resultType=details&any_OR__title=covid-19&isChild=’false’&fast=index&sortBy=relevance&from=1&to=20.

[pone.0253566.ref003] 3.(NICE) NICE. Jaarboek 2018: Stichting NICE; 2019. 54 p.

[pone.0253566.ref004] 4.Girum T, Lentiro K, Geremew M, Migora B, Shewamare S. Global strategies and effectiveness for COVID-19 prevention through contact tracing, screening, quarantine, and isolation: a systematic review. Trop Med Health. 2020;48:91. doi: 10.1186/s41182-020-00285-w [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0253566.ref005] 5.Rajan S, C JD, McKee M. What do countries need to do to implement effective ’find, test, trace, isolate and support’ systems? J R Soc Med. 2020;113(7):245–50. Epub 2020/07/15. doi: 10.1177/0141076820939395 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0253566.ref006] 6.Buitrago-Garcia D, Egli-Gany D, Counotte MJ, Hossmann S, Imeri H, Ipekci AM, et al. Occurrence and transmission potential of asymptomatic and presymptomatic SARS-CoV-2 infections: A living systematic review and meta-analysis. PLoS Med. 2020;17(9):e1003346–e. doi: 10.1371/journal.pmed.1003346 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0253566.ref007] 7.Johansson MA, Quandelacy TM, Kada S, Prasad PV, Steele M, Brooks JT, et al. SARS-CoV-2 Transmission From People Without COVID-19 Symptoms. JAMA Netw Open. 2021;4(1):e2035057. Epub 2021/01/08. doi: 10.1001/jamanetworkopen.2020.35057 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0253566.ref008] 8.Varsavsky T, Graham MS, Canas LS, Ganesh S, Pujol JC, Sudre CH, et al. Detecting COVID-19 infection hotspots in England using large-scale self-reported data from a mobile application: a prospective, observational study. medRxiv. 2020:2020.10.26.20219659. doi: 10.1016/S2468-2667(20)30269-3 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0253566.ref009] 9.Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020;369:m1328. doi: 10.1136/bmj.m1328 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0253566.ref010] 10.Menni C, Valdes AM, Freidin MB, Sudre CH, Nguyen LH, Drew DA, et al. Real-time tracking of self-reported symptoms to predict potential COVID-19. Nature Medicine. 2020;26(7):1037–40. doi: 10.1038/s41591-020-0916-2 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0253566.ref011] 11.(LUMC) LUMC. COVID Radar 2020 [2020-12-1]. https://www.lumc.nl/org/coronavirus/covid-radar/.

[pone.0253566.ref012] 12.RIVM. Virologische Weekstaten Viral Surveillance https://www.rivm.nl/virologische-weekstaten/onderzoek:RIVM; 2020 [cited 2020 9-12-2020].

[pone.0253566.ref013] 13.Gwaltney JM Jr., Hendley JO, Simon G, Jordan WS Jr. Rhinovirus Infections in an Industrial Population: II. Characteristics of Illness and Antibody Response. JAMA. 1967;202(6):494–500. doi: 10.1001/jama.1967.03130190100014 [DOI] [PubMed] [Google Scholar]

[pone.0253566.ref014] 14.Pappas DE, Hendley JO, Hayden FG, Winther B. Symptom Profile of Common Colds in School-Aged Children. The Pediatric Infectious Disease Journal. 2008;27(1):8–11. doi: 10.1097/INF.0b013e31814847d9 [DOI] [PubMed] [Google Scholar]

[pone.0253566.ref015] 15.Stokes EK, Zambrano LD, Anderson KN, Marder EP, Raz KM, Felix SEB, et al. Coronavirus disease 2019 case surveillance—United States, January 22–May 30, 2020. Morbidity and Mortality Weekly Report. 2020;69(24):759. doi: 10.15585/mmwr.mm6924e2 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0253566.ref016] 16.Tong JY, Wong A, Zhu D, Fastenberg JH, Tham T. The prevalence of olfactory and gustatory dysfunction in COVID-19 patients: a systematic review and meta-analysis. Otolaryngology–Head and Neck Surgery. 2020;163(1):3–11. doi: 10.1177/0194599820926473 [DOI] [PubMed] [Google Scholar]

[pone.0253566.ref017] 17.Lee Y, Min P, Lee S, Kim SW. Prevalence and Duration of Acute Loss of Smell or Taste in COVID-19 Patients. J Korean Med Sci. 2020;35(18):e174. Epub 2020/05/10. doi: 10.3346/jkms.2020.35.e174 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0253566.ref018] 18.Eenvandaag. Ruim een derde van alle coronatests niet door GGD, maar particulier: ’We verliezen het zicht op het virus’ [website]. 2020 [cited 2021 2021-01-01]. https://eenvandaag.avrotros.nl/item/ruim-een-derde-van-alle-coronatests-niet-door-ggd-maar-particulier-we-verliezen-het-zicht-op-het/.

PERMALINK

COVID RADAR app: Description and validation of population surveillance of symptoms and behavior in relation to COVID-19

Willian J van Dijk

Nicholas H Saadah

Mattijs E Numans

Jiska J Aardoom

Tobias N Bonten

Menno Brandjes

Michelle Brust

Saskia le Cessie

Niels H Chavannes

Rutger A Middelburg

Frits Rosendaal

Leo G Visser

Jessica Kiefte-de Jong

Roles

Abstract

Background

Methods

Results

Discussion

Introduction

Methods

COVID radar app

Fig 1. Screenshot of COVID radar app (Left: Questions. Right: Feedback) “Reprinted from “urlmaps.com” under the Creative Commons Attribution License, with permission from “i-mapping”, original copyright 2021”.

Comparison of included/excluded observations

Descriptive statistics

Validation testing

Criterion validity

External validation

Sensitivity analyses

Results

Comparison of included/excluded observations

Descriptive statistics

Fig 2. Number of observations per day.

Fig 3. Number of observations per capita per province in the Netherlands.

Fig 4. Distribution of number of observations per user.

Criterion validation

Table 1. Comparison of users never vs. ever tested SARS-CoV-2 positive.

Fig 5.

Fig 6.

External validation

Fig 7. Weekly proportion of SARS-CoV-2 positive users reported in the COVID Radar app and case count per capita as reported by the RIVM for each province.

Fig 8. Symptoms vs. time alongside number of nationally reported positive SARS-CoV-2 tests (percentage of users reporting cough, sore throat, GI (gastrointestinal) symptoms, and shortness of breath).

Fig 9. Symptoms vs. time alongside number of nationally reported positive SARS-CoV-2 tests (percentage of users reporting pain in chest, fever, loss of smell, swollen eyes).

Fig 10. Symptoms vs. time (weekly) alongside number of nationally reported positive cultures of Rhinovirus (percentage of users reporting symptoms).

Fig 11. Daily mean number of people within 1.5 meters (with 7-day backward-looking moving average in black) September: Closing of restaurants.

Fig 12. Daily mean number of visitors vs. time (with 7-day backwards moving average).

Sensitivity analyses

Discussion

Criterion validity

External validity

Limitations

Future implications

Conclusion

Supporting information

Data Availability

Funding Statement

References

Decision Letter 0

Simone Lolli

Roles

Author response to Decision Letter 0

Decision Letter 1

Simone Lolli

Roles

Acceptance letter

Simone Lolli

Roles

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases