Abstract
Objective
The novel coronavirus 2019 (COVID-19) pandemic has triggered public anxiety around the world. So far, the evidence suggests that prevention on a public scale is the most effective health measure for thwarting the progress of COVID-19. Another critical aspect of preventing COVID-19 is contact tracing. We aimed to investigate the effectiveness of digital contact tracing applications currently available in the context of the COVID-19 pandemic.
Methods
We undertook a systematic review and narrative synthesis of all literature relating to digital contact tracing applications in the context of COVID-19. We searched 3 major scientific databases. Only articles that were published in English and were available as full-text articles were selected for review. Data were extracted and narrative syntheses conducted.
Results
Five studies relating to COVID-19 were included in the review. Our results suggest that digitalized contact tracing methods can be beneficial for impeding the progress of COVID-19. Three key themes were generated from this systematic review. First, the critical mass of adoption of applications must be attained at the population level before the sensitivity and positive predictive value of the solution can be increased. Second, usability factors such as access, ease of use, and the elimination of barriers are essential in driving this uptake. Third, privacy must be ensured where possible as it is the single most significant barrier against achieving critical mass.
Conclusion
Contact tracing methods have proved to be beneficial for impeding the progress of COVID-19 as compared to older, more labour-intensive manual methods.
Keywords: COVID-19, Contact tracing, Pandemic
Highlights of the Study
The COVID-19 pandemic has caused major adjustments to the way the world deals with day-to-day activities.
Prevention on a public scale has proven to be the most effective health measure in thwarting the progress of this disease.
Contact tracing applications have assisted in early detection and public protection, impeding the progress of this virus.
Introduction
The novel coronavirus 2019 (COVID-19) pandemic has triggered public anxiety and mayhem around the world. Globally, COVID-19 has caused over 3 million deaths, with over 140 million confirmed cases as of today. In the USA alone, over 45,000 people have lost their lives, whereas in the UK, that number has crossed 100,000 [1]. Human-to-human transmission of the virus is facilitated by the expulsion of respiratory droplets whenever someone sneezes, coughs, or talks. These expelled droplets do not travel for more than 2 m but may remain suspended in the air for a short period of time [2]. The SARS-CoV-2 virion can also spread indirectly when humans come into contact with contaminated surfaces and self-contaminate their mucous membranes by touching their eyes, nose, and mouth. So far, the evidence suggests that prevention on a public scale is the most effective public health measure for hindering progress. Social distancing, which is defined as the maintenance of a distance of at least 2 metres from others and the avoidance of social congregations, has been shown to be the most efficacious public intervention [3]. In fact, studies have shown that even a 24-h delay in the implementation of social distancing can end in containment delays that exceed 48 h [4].
Another critical aspect of preventing COVID-19 is contact tracing. Digital contact tracing for the attempted control of the severe acute respiratory syndrome outbreak has existed since 2007 [5]. Contact tracing as a public health intervention has some key advantages − it can rapidly identify suspected cases before the emergence of severe or critical symptoms, and it can prevent the subsequent transmission of disease from the secondary cases if implemented efficiently. Currently, countries around the world have adopted different methods of contact tracing to help block the progression of this disease, with some countries attempting limited contact tracing, and others opting for a more comprehensive type of tracing (Fig. 1) [6]. The importance and effectiveness of contact tracing was definitively validated in England when a coding error that occurred in late September 2020 caused a total of 15,841 COVID-19 cases (around 20% of all cases) to have timely contact tracing. By chance, some areas of England were much more severely affected than others, which suggested that the random breakdown of contact tracing led to more illness and death. Conservative causal estimates imply that relative to cases that were initially missed by the contact tracing system, cases subject to proper contact tracing were associated with a reduction in subsequent new infections of 63% and a reduction in subsequent COVID-19-related deaths of 66% across the 6 weeks following the data glitch [7]. Therefore, we aimed to investigate the effectiveness of digital contact tracing applications currently available in the context of the COVID-19 pandemic.
Fig. 1.
Use of contact tracing applications around the world during the COVID-19 pandemic (source: Oxford COVID-19 government response tracker).
Methods
Literature Review
The systematic review search strategy was conducted using three different scientific databases, PubMed, EMBASE, and OVID Medline in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The following search terms were used − “COVID-19,” “SARS-CoV-2,” “coronavirus,” “corona virus,” “coronaviridae,” “betacoronavirus,” “covid19,” “covid 19,” “novel CoV,” “contact tracing,” “app*,” “contact notification,” and “effective*.” We also manually searched the references of all relevant studies to supplement our searches. Basic Boolean operators such as “AND” and “OR” were used to support the search approach in order to propagate productive and focused studies for the systematic review. The final search strategy was as follows: (COVID-19) OR (Sars-Cov-2) OR (coronavirus) OR (“corona virus”) OR (coronaviridae) OR (betacoronavirus) OR (COVID19) OR (“COVID 19”) OR (“novel CoV”) AND (contact tracing) AND (app*) AND (performance) OR (effective*).
Inclusion and Exclusion Criteria
After search results were combined between the three online scientific databases, duplicates were removed, and the remaining full-text articles were organized in an online reference management software. Titles and abstracts were reviewed and potentially eligible articles identified by one reviewer. Full-text review and agreement on final inclusion was undertaken by two reviewers, Full-text articles were analysed through a set of inclusion and exclusion criteria. The inclusion criterion was digital application for COVID-19 human contact tracing regardless of the technical specifications of the application. The exclusion criteria were no direct measurement of the effectiveness of performance and scoped to ethics or privacy policies. Only articles that were published in English and were available as full-text articles were selected for review, and the publication time of studies was limited to December 2019–March 2021. Only primary research articles were selected for review, and secondary research such as systematic reviews, meta-analyses, and case series were excluded from the online scientific databases. No age limit was set on the scientific databases, as children and adolescents are also susceptible, and contact tracing efforts do not discriminate between the young, adults and the elderly.
Measure of Effectiveness of Performance Definition
We defined effectiveness of performance as follows: studies that evaluated one of three outcomes of interest: case detection rates among contacts or at the community level (hypothesized to increase with contact tracing); overall forward transmission of disease as measured by the reproduction number (R; hypothesized to decrease), secondary attack rates (hypothesized to decrease), or similar measures (including cases and deaths prevented, reinfection rates among contacts, and treatment rates among contacts, where increased treatment rates were interpreted as a proxy for decreased forward transmission); and overall disease incidence (hypothesized to decrease). Other downstream disease outcomes, such as mortality, were considered beyond the scope of this review. We included studies evaluating the effects of provider-initiated contact tracing compared with the absence of contact tracing or with patient-initiated contact tracing.
The workflow related to the abstract/title screening, identification of studies, and full-text review of the studies for final inclusion and exclusion is presented in the PRISMA flowchart in Figure 2. A total of 5 studies were included in the review. The Critical Skills Appraisal Program checklists were used for the critical appraisal of the included studies.
Fig. 2.
PRISMA flowchart for data collection.
Results
Systematic Review
The first study was authored by Cheng and Hao [8] for patients in China and was a proposal of a tool which serves as a supplemental contact tracing approach to support the shortage of healthcare staff during COVID-19. The authors described a tool which can be deployed on the Internet or as a plugin for a smartphone application. Individuals with a confirmed diagnosis of COVID-19 can use the tool to provide their contact information as well as the contact information of their close interactions; this action causes the system to automatically send a push message to these contacts to inform them of their interaction status, what the status implies, and what actions should be undertaken by these individuals (e.g., self-quarantine).
The main advantage of this tool is that it enables the anonymity of personal identifiers; individuals with a confirmed COVID-19 diagnosis can input their contact information (e.g., email address) without revealing their names or national registration/identification numbers. The message received (i.e., pushed by the system) by the contacts of the infected individuals does not contain disease-related information but instead contains a security passcode for the contact to access the secure platform to read the encrypted message.
The second study was authored by Huang et al. [9] and was identified through other sources as opposed to having been identified by conducting the search strategy on the three online scientific databases. In this Singaporean study, researchers sought to compare a novel digital contact tracing application − TraceTogether, with a wearable, tag-based, and real-time locating system (RTLS) and to validate both methodologies against electronic health records (EHRs) at the national referral centre for COVID-19 screening. In this study, which was conducted over 10 days, a cross-sectional design was employed to validate the TraceTogether mobile application and the older wearable RTLS wrist-tag against the EHR. All healthcare staff and patients in the national referral centre for COVID-19 screening were issued RTLS tags for contact tracing; the wearing of these tags was mandated for entry into the screening centre. Physicians who were deployed to the screening centre over the 10-day study duration were instructed to install the novel TraceTogether application on their smartphones and activate the phone's Bluetooth function during their work periods. Physicians were also urged to encourage their patients who were being screened for COVID-19 to download and activate the same application.
The TraceTogether mobile application was developed by a Singapore government-linked company in response to the COVID-19 pandemic. The application exchanges Bluetooth signals with other users who are nearby, and the data are encrypted and stored locally on the mobile phone. Users who have a confirmed diagnosis of COVID-19 are asked to upload data stored on their smartphones to aid with digital contact tracing. After 3 weeks of storage, data are automatically eradicated. The application stores only the user's mobile phone number and the proximity and duration of the users' contacts. The RTLS was fitted into the screening centre in September 2019; the tag-based system detected signals whenever it passed a location exciter and sent signals to access points to determine the precise physical location of the tag.
The gold standard against which TraceTogether and the RTLS system were measured was the EHR system. This is because every patient who was screened for COVID-19 at the national screening centre had their clinical appointments logged into the EHR system. Patient contacts identified by TraceTogether were compared with those detected by the RTLS system. The sensitivity, specificity, positive/negative predictive values, and likelihood ratios of both technologies were compared. The RTLS system had a high sensitivity (95.3%) in detecting all patient contacts, whereas TraceTogether had a sensitivity of 6.5%. Furthermore, the RTLS tags had a high sensitivity (96.9%) and specificity (83.1%) as they could detect patient contacts apart from those between the patients and their attending physicians. In contrast, the TraceTogether only identified 2 such patients and therefore had a sensitivity of 0% and a specificity of 98.4%.
The third study was authored by Yasaka et al. [10] and was a proposal for a proof-of-concept smartphone mobile application which implemented an anonymized graph of interpersonal interactions. The data structure utilized by the researchers, termed as a transmission graph, consists of nodes that represent contact points between people and directed edges which represent transmission vectors between these contact points. The transmission graph essentially is a graphical representation of the network of interactions between individuals. The researchers developed a proof-of-concept smartphone application which utilizes this transmission graph data structure to facilitate peer-to-peer contact tracing. This enables users to create contact points in the transmission graph, check their own risk level, and also to report their own positive status. Similar to the study conducted by Cheng and Hao [8], this study lacked real-world data as it was a proposed innovative digital solution and instead presented findings from a simulation model.
The model used by the researchers was based on the Susceptible, Infected, and Recovered epidemiological model which is publicly accessible on GitHub. This model enabled researchers to compare simulations with and without the use of their proposed contact tracing application, as well as between various population adoption rates. The simulations revealed that higher adoption rates of the application could result in better control of the pandemic. Even a 25% adoption rate suppressed the infection curve, as compared to no adoption at all [10].
The fourth study was a Japanese one which evaluated a Bluetooth-based mobile application that leveraged a peer-to-peer contact tracing framework without compromising the privacy of users [11]. The app, COVID-19 Contact-Confirming Application automatically records close contact by using Bluetooth technology and comprises three elements. These are two mobile terminal applications for individuals (infected and exposed) and an infection information sharing system that is maintained by health authorities. The app utilizes a decentralized system architecture which tracks close contacts and matches exposure risk without divulging personal information. The app requires individuals to self-triage their risk of exposure, much like the solution proposed by Cheng and Hao [8]. The authors demonstrated that since the deployment of this solution in Japan, there has been a significant (20%) reduction in the Japanese population mobility which has arguably flattened the infectious curve. The authors indicated, however, that the solution requires a 90% participation rate in order to effectively control the spread of COVID-19 [11]. The main limitation of this solution is the voluntary nature of participation, as it could lead to non-compliance and latency in responses and detection.
The fifth study was conducted in Taiwan; it involved the use of a centralized contact tracing system which relied on self-reporting via automatic text messaging or the use of a web application [12]. The Taiwan Centers for Disease Control had developed a digital contact tracing tool called TRACE in 2017. TRACE was used to monitor and contact trace cases of other infectious diseases such as measles and rubella. The technology augments only a few aspects of the contact tracing process, such as contact list generation, and requires self-reporting from its users. The researchers sought to demonstrate the utility of the contact tracing management system to enhance traditional contact tracing measures. Of 8,051 close contacts, 487 were confirmed COVID-19 cases, while the median time from the last exposure to quarantine was 3 days. The implementation of the proposed solution (SMS and Web-app self-reporting) increased the percentage of health status updates from self-reporting from 22.5% to 61.5%, representing an almost 3-fold increase. This significantly attenuated the workload of public health officials as they had fewer telephone calls or home visits to make [12]. A summary of the findings of the studies included has been included in Table 1.
Table 1.
Summary of the studies included in this systematic review
| Author | Year | Country | Mode of operation | Relevant findings |
|---|---|---|---|---|
| Cheng and Hao [8] | 2020 | China | Via Internet or as plugin for smartphone app | Confirmed cases with COVID-19 can use this tool to provide contact information (either email addresses or mobile phone numbers) of close contacts; then, the system will automatically send a message to the contacts informing them of their contact status, what this status means, the actions that should follow (e.g., self-quarantine, respiratory hygiene/cough etiquette), and advice for receiving early care if they develop symptoms |
| Huang et al. [9] | 2020 | Singapore | Wearable tag-based RTLS | TraceTogether had a much lower sensitivity than RTLS tags for identifying patient contacts in a clinical setting. Although the tag-based RTLS performed well for contact tracing in a clinical setting, its implementation in the community would be more challenging than TraceTogether |
| Yasaka et al. [10] | 2020 | USA | Smartphone app for creating “checkpoints” | Proof-of-concept smartphone app that allows users to create “checkpoints” for contact tracing, check their risk level based on their past interactions, and anonymously self-report a positive status to their peer network. The simulation results suggest that higher adoption rates of such an app may result in a better controlled epidemic or pandemic outbreak |
| Nakatoma et al. [11] | 2020 | Japan | Bluetooth-based mobile contact-confirming app | The matching inference of exposure is performed locally, and individuals can self-triage their risk of exposure. It works better in load-balancing than centralized frameworks. The detection of close contact is rapid and effective, and it reduces the likelihood of cross-transmission and in-person contacts. The background running feature enhances the efficacy of the approach and reduces errors of operation and has shown a 20% reduction in population mobility since its deployment in Japan |
| Jian et al. [12] | 2020 | Taiwan | Automatic text messages and web app | Among the 8,051 close contacts of the 487 confirmed cases, the median elapsed time from last exposure to quarantine was 3 days. By implementing this approach of self-reporting, the percentage of health status updates from self-reporting increased from 22.5% to 61.5%. The high proportion of secondary cases detected via contact tracing (88%) might reduce the R0 to under one and minimize the impact of local transmission in the community |
Discussion
The results of this systematic review suggest that digitalized contact tracing methods (i.e., smartphone applications) can be considerably more beneficial for impeding the progress of the COVID-19 pandemic as compared to older methodologies. An analysis of the results reveals three themes for discussion.
First, the success of digital contact tracing applications depends significantly upon the uptake of the technology and the adherence to the spirit of the technology. In the study conducted by Huang et al. [9], the TraceTogether application severely underperformed as compared to the RTLS technology. This was likely secondary to poor compliance on the part of the patient population, as regards the downloading of the application and the correct use of the application. Indeed, the RTLS technology was mandated, as opposed to the TraceTogether technology which was encouraged or urged by physicians. In Australia, researchers focused on promoting the downloading and use of a similar contact tracing application − COVIDSafe.
Yasaka et al. [10] demonstrated that even with a 25% uptake of such a solution could result in a suppression of the infection curve. Huang et al. [9] describe the desirable population-level uptake rate as a “critical mass”; this critical mass of application adoption must be attained before the sensitivity and positive predictive value of similar solutions can be increased [13]. According to Huang et al. [9], three factors are essential for the adoption of similar solutions to reach critical mass − partial regulatory enforcement, effective and constant communication regarding the merits of the application, and a good comprehension of the enablers and barriers of application adoption. According to Jian et al. [12], successful case investigation and contact tracing depend significantly on the robustness and competency of the public health workforce. In their study, over 90% of the contact tracers involved were public health workers with experience.
Second, the uptake of such solutions, which has been established in the previous section as a pre-requisite for achieving critical mass, is dependent on various usability-related factors such as the principle of operation of the underlying technology, as well as access and ease of use. Unlike the TraceTogether solution, which relies on the smartphone application being downloaded and installed, the solution offered by Cheng and Hao [8] presents users with the option to access it via an Internet browser. The TraceTogether solution requires users to have their Bluetooth service switched on continuously; this can serve as a significant deterrent because of its energy-sapping nature. Furthermore, Bluetooth-based technology has other technical limitations that could hinder effectiveness, such as operating system cross-compatibility, accounting for physical barriers and others.
Third, the importance of ensuring privacy when designing such solutions cannot be overstated. As seen in the solution proposed by Cheng and Hao [8], as well as TraceTogether, the dissemination of contact information to governmental agencies to facilitate contact tracing is dependent on the individual who is infected. Unlike the RTLS system which is mandated, individuals who have a confirmed diagnosis of COVID-19 may choose not to divulge their contact information to a central server managed by the government. Privacy concerns regarding the storage of data and location tracking are likely to be the most significant barrier against achieving critical mass for these innovative solutions [13, 14]. Nakamoto et al. [11] emphasized the importance of privacy in their paper, which introduced the decentralized COVID-19 Contact-Confirming Application mobile application, especially in the context of the Japanese population, which was described as “privacy-sensitive.” Privacy concerns and the collective public trust in health and governmental authorities could vary from culture to culture and should be considered by application developers looking to digitalize contact tracing.
Citizens have legitimate concerns regarding the commercial exploitation of their personal health data. Indeed, the concept of anonymization has been put forth as a strategy to prevent the re-identification of the subject of personal health data that have been inadvertently leaked. However, evidence suggests that as data sets are amplified and rapidly proliferate in this age of big data, the ability to amalgamate multiple sets of data to identify a subject is not far-fetched [15].
As previously iterated, a significant proportion of the population must download and use a digital contact tracing application in order for it to be substantially effective in suppressing the infectious curve of COVID-19. Apart from ensuring the privacy and confidentiality of health-related data of users, developers of contact tracing applications must also strive to make the application as user-friendly as possible. The importance of usability testing in application development cannot be overstated. Usability as a concept refers to effectiveness, efficiency, and satisfaction. Effectiveness can be defined as the extent to which users achieve their tasks and goals, efficiency can be defined as the degree to which users utilize resources to achieve said goals and tasks, and satisfaction can be defined as the level of comfort users experience in achieving these goals [16]. Usability testing enables application developers to understand their users' needs to enhance the overall utility of the application in an endeavour to promote user satisfaction [17]. With the advent of COVID-19, there has been a flurry of smartphone applications and digital health applications which focus on telemedicine and remote medical consultations [18]. However, there is a lack of guidance or standards which dictate how best to develop applications which are suitable for users and practical for ease of use. As a healthcare application, COVID-19 contact tracing applications should be freely useable by individuals who are older, with poor literacy, and with disabilities. This is because healthcare applications should aim to be as inclusive as possible. Mobile phone technologies today are often displayed on complex devices with clunky user interfaces that require considerable dexterity and visual acuity for optimal usage [19].
One design principle which originated from the US Navy − KISS (keep it simple, stupid) could be useful for application developers in this regard. This usability principle advocates for simplicity as a key theme in design and has been successfully implemented in the development of enterprise-level electronic medical record software [20]. Developers of contact tracing application should also implement intensive usability testing sessions with as heterogeneous and diverse groups as possible. This is because unlike other health applications which have a niche target audience (e.g., pregnant women), the COVID-19 contact tracing application does not have a specific user “avatar.” Indeed, the COVID-19 contact tracing application seeks to capture most, if not all, users in a given population to optimize contact tracing and flatten the COVID-19 infectious curve as rapidly as possible. Hence, the development team of this application faces a unique challenge to design the application to be as user-friendly to as many different user profiles as possible (i.e., young and old, healthy and individuals with co-morbidities, able and disabled, and literate and illiterate). In order to capture as many users as possible, developers should also endeavour to make the application compatible with both Android and iOS devices.
Although the discussion so far has been centred around the digitalization of the COVID-19 contact tracing effort, consideration should also be given to the human elements which exist in the periphery of the application proper. For example, application developers should think about the service-level agreements that they have with their users and the escalation process for queries and troubleshooting. Users who have clarifications or doubts to discuss should be provided with the appropriate channels and avenues to do so. While some applications leverage helpdesks, which are typically outsourced to deal with these issues, this strategy may not be feasible for a COVID-19 contact tracing application for the following reason. As previously stated, a COVID-19 contact tracing application would attract a much larger and more diverse user base as compared to a niche health application. This would quickly overwhelm a helpdesk with the sheer volume of requests and calls. Hence, developers could consider leveraging a chatbot as a first-line service centre, followed by a helpdesk as a second-line service centre for requests which could not be addressed by a chatbot. Chatbot technology is feasible today; the evidence suggests that there are several commercial and open-source options available for the development of chatbots [21]. Simple rules-based chatbots could be developed to address straightforward queries regarding the COVID-19 contact tracing application (e.g., concerns about privacy and confidentiality and duration of data storage).
Limitations
Even though this study included studies which utilized real-world data to glean insights into the effectiveness of a contact tracing application, one major limitation of the systematic review is that only five studies were included. This is because the literature search strategy yielded just nine studies from three online scientific databases, five of which were excluded after full-text review (reasons specified in Fig. 2). One additional study which was included in the final count of five studies was sourced from screening the bibliographies of the excluded studies and searching the grey literature. With just five studies included, this systematic review is considered to be a small one. Furthermore, two of the five studies did not utilize real-world data and were either proposals with conceptual benefits or proof-of-concept models with simulated data. This reduces the generalizability of the insights gleaned from this review to the real-world environment. Another limitation lies in the fact that digital contact tracing applications depend heavily on the GPS accuracy of the device they are utilizing. It has been previously demonstrated that GPS-enabled smartphones are typically accurate to within a 4.9 m (16 ft.) radius under open sky [22]. However, their accuracy worsens near buildings, bridges, and trees. If this is true, then the contact tracing apps cannot tell if the distance between the two phones is 4.5 m or 2.0 m. If the contact tracing apps intend to detect close contacts within 2 m, then there will potentially be a lot of contacts that are falsely flagged as close contacts.
Nevertheless, this paper represents insights generated from various parts of the world − Singapore in Asia, China, and California in the USA. This is relevant, considering that COVID-19 is a global pandemic and that almost 75% of the global population has access to the Internet and mobile services.
Conclusions
Our results suggest that digitalized contact tracing methods can be beneficial for impeding the progress of COVID-19 as compared to older, more labour-intensive manual methods. Three key themes were generated from this systematic review. First, the critical mass of application adoption must be attained at the population level before the sensitivity and positive predictive value of the solution can be increased. Second, usability factors such as access, ease of use, and the elimination of barriers are essential in driving this uptake. Third, privacy must be ensured where possible as it is the single most significant barrier against achieving critical mass. The main limitation of this systematic review is the paucity of studies included for review; this is a function of the novelty of the research question. As more innovative and smart contact tracing applications are developed, a future systematic review on the same topic may yield more published real-world data for synthesis and critical analysis.
Statement of Ethics
All procedures performed in studies involving human participants were in accordance with the ethical standards of the Institutional and/or National Research Committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
Conflict of Interest Statement
The authors declare that they have no conflict of interest.
Funding Sources
None.
Author Contributions
Ahmad Nabeel: collecting data and writing of manuscript, Salman K. Al Sabah: idea conception and draft review, Hutan Ashrafian: supervisor and draft review.
Data Availability Statement
Supplementary data will be made available if requested.
Funding Statement
None.
References
- 1.John Hopkins University COVID-19 dashboard. 2021. Available from: https://coronavirus.jhu.edu/map.html.
- 2.van Doremalen N, Bushmaker T, Morris DH, Holbrook MG, Gamble A, Williamson BN, et al. Aerosol and surface stability of SARS-CoV-2 as compared with SARS-CoV-1. N Engl J Med. 2020;382((16)):1564–7. doi: 10.1056/NEJMc2004973. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Qian M, Jiang J. COVID-19 and social distancing. Z Gesundh Wiss. 2022;30((1)):259–61. doi: 10.1007/s10389-020-01321-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Du Z, Xu X, Wang L, Fox SJ, Cowling BJ, Galvani AP, et al. Effects of proactive social distancing on COVID-19 outbreaks in 58 cities, China. Emerg Infect Dis. 2020;26((9)):2267–9. doi: 10.3201/eid2609.201932. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Altuwaiyan T, Hadian M, Liang X, editors. 2018 IEEE International Conference on Communications (ICC) 20–24 May 2018. 2018. EPIC: efficient privacy-preserving contact tracing for infection detection. [Google Scholar]
- 6.Hale T, Angrist N, Goldszmidt R, Kira B, Petherick A, Phillips T, et al. A global panel database of pandemic policies (Oxford COVID-19 Government Response Tracker) Nat Hum Behav. 2021;5((4)):529–38. doi: 10.1038/s41562-021-01079-8. [DOI] [PubMed] [Google Scholar]
- 7.Fetzer T, Graeber T. Measuring the scientific effectiveness of contact tracing: evidence from a natural experiment. Proc Natl Acad Sci U S A. 2021;118((33)):e2100814118. doi: 10.1073/pnas.2100814118. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Cheng W, Hao C. Case-initiated COVID-19 contact tracing using anonymous notifications. JMIR Mhealth Uhealth. 2020;8((6)):e20369. doi: 10.2196/20369. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Huang Z, Guo H, Lee YM, Ho EC, Ang H, Chow A. Performance of digital contact tracing tools for COVID-19 response in Singapore: cross-sectional study. JMIR Mhealth Uhealth. 2020;8((10)):e23148. doi: 10.2196/23148. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Yasaka TM, Lehrich BM, Sahyouni R. Peer-to-peer contact tracing: development of a privacy-preserving smartphone app. JMIR Mhealth Uhealth. 2020;8((4)):e18936. doi: 10.2196/18936. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Nakamoto I, Jiang M, Zhang J, Zhuang W, Guo Y, Jin MH, et al. Evaluation of the design and implementation of a Peer-To-Peer COVID-19 contact tracing mobile app (COCOA) in Japan. JMIR Mhealth Uhealth. 2020;8((12)):e22098. doi: 10.2196/22098. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Jian SW, Cheng HY, Huang XT, Liu DP. Contact tracing with digital assistance in Taiwan's COVID-19 outbreak response. Int J Infect Dis. 2020;101:348–52. doi: 10.1016/j.ijid.2020.09.1483. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Cho H, Ippolito D, Yu YW. Contact tracing mobile apps for COVID-19: privacy considerations and related trade-offs. arXiv preprint arXiv:2003.11511. 2020 [Google Scholar]
- 14.Cohen IG, Gostin LO, Weitzner DJ. Digital smartphone tracking for COVID-19: public health and civil liberties in tension. JAMA. 2020;323((23)):2371–2. doi: 10.1001/jama.2020.8570. [DOI] [PubMed] [Google Scholar]
- 15.Price WN, 2nd., Cohen IG. Privacy in the age of medical big data. Nat Med. 2019;25((1)):37–43. doi: 10.1038/s41591-018-0272-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Brooke J. SUS: a retrospective. J Usability Stud. 2013;8:29–40. [Google Scholar]
- 17.Brock D, Kim S, Palmer O, Gallagher T, Holmboe E. Usability testing for the rest of us: the application of discount usability principles in the development of an online communications assessment application. Teach Learn Med. 2013;25((1)):89–96. doi: 10.1080/10401334.2012.741547. [DOI] [PubMed] [Google Scholar]
- 18.Mann DM, Chen J, Chunara R, Testa PA, Nov O. COVID-19 transforms health care through telemedicine: evidence from the field. J Am Med Inform Assoc. 2020;27((7)):1132–5. doi: 10.1093/jamia/ocaa072. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Patrick K, Griswold WG, Raab F, Intille SS. Health and the mobile phone. Am J Prev Med. 2008;35((2)):177–81. doi: 10.1016/j.amepre.2008.05.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Newman S. A KISS for “meaningful use” of EMR: keep it simple stupid. Mich Med. 2010;109((4)):6. [PubMed] [Google Scholar]
- 21.Adamopoulou E, Moussiades L. An overview of chatbot technology. Artif Intelligence Appl Innov. 2020;584:373–83. [Google Scholar]
- 22.GPS Accuracy: Official U.S. Government information about the Global Positioning System (GPS) and related topics Available from: https://www.gps.gov/systems/gps/performance/accuracy/
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Supplementary data will be made available if requested.


