Table 2.

Summary of articles included in review.

Study	Location of tool or tools	Health care system or systems integrated into	Research question or purpose	Direction to care: the tool directs to	Sample size, the number of users, or number of times accessed	Key findings
Azadnajafabad et al [27], 2021	Iran	Yes—Ministry of Health and Medical Education of Iran	To determine the effectiveness of a web-based self-screening platform to offer a population-wide strategy to control the massive influx to medical centers	COVID-19 testing centers, hospital, and medical centers, to be more cautious	310,000 users	Successful implantation and proven potency of such platform suggest more application of telehealth in public health disasters. Details of the platform in this study can be useful for further deployment of similar platforms.
Collado-Borrell et al [35], 2020	Worldwide: 114 applications that were used in several countries	None	To identify and analyze the characteristics of smartphone apps designed to address the COVID-19 pandemic	It varied based on the app.	114 apps	This study found that the greatest number of downloads of self-assessment apps were for those developed by governments, except for the WHOa app. The app with the highest number of downloads was developed by the Indian government, followed by the Polish and Colombian governments. The main purpose of the apps was to provide general information about the pandemic. Mobile apps can be used as a tool for patient communication and monitoring.
Denis et al [15], 2021	France	Yes—French National Health Care System	To assess whether daily reports of anosmia (lack of smell) predicted positive RT-PCRb tests results, daily EDc visits, daily conventional hospitalization, and daily ICUd admissions	ERe, primary care, stay home, or use the tool again if symptoms evolve; if severe symptoms, advises to contact a GPf or ED	13,000,343 questionnaires completed from March to November 2020	Peak daily reports of anosmia on the website predicted hospitalizations, ICU admissions, and positive RT-PCR tests. However, in the second wave of the pandemic, this did not hold true. The authors attribute the difference to the fact younger people were affected in the second wave. They conclude numbers with anosmia predict hospital demand for older adults. “Although this tool does not accurately anticipate an increase in the magnitude of hospitalization, it seems to accurately predict the reduction in the hospitalization rate.”
Denis et al [26], 2020	France	Yes—French National Health Care System	To determine if self-reported symptoms could help monitor outbreak dynamics in France	ER, primary care, stay home, or use the tool again if symptoms evolve; if severe symptoms, advises to contact a GP or ED	3,799,535 questionnaires completed	“This study suggests that self-reported symptoms of COVID-19 are correlated with COVID-19–related hospitalizations and that anosmia may be strongly associated with COVID-19.”
Dhakal et al [34], 2020	United States, India, Nepal, Bangladesh	None	They develop an app that takes verbal input to self-assess for COVID-19, then test it with users to study the app’s performance, its usability, and demands on the user’s mental capacity	On the basis of symptoms users are advised to call 911 and visit ER; stay home and contact medical personnel and take over the counter medications as needed	22 users	Users of the novel IVACSg app did not experience high workload to use the tool. Some users experienced frustration as they had to repeat information; the system did not manage all dialects equally well. This did sometimes lead to different results for the same information input.
Galmiche et al [13], 2020	France	Yes—French National Health Care System	To determine if a self-triage tool could reduce the burden on emergency call centers and help predict increasing burden on hospitals	ER, primary care, stay home, or use the tool again if symptoms evolve; if severe symptoms, advises to contact a GP or ED.	3,494,687 questionnaires completed	“The launch of the self-triage web application was followed by a nearly 10-fold increase in COVID-19–related hospitalizations with only a 23% increase in emergency calls, even though the number of completed questionnaires quickly surged, including questionnaires leading to a recommendation to call an emergency call center.” The authors note that they cannot conclude that the application lead to alleviation of demand on emergency call centers.
Hautz et al [14], 2021	Switzerland	Yes—Swiss Federal Office of Public Health	To implement a web-based triage tool targeted at the current pandemic, adapt the content and goals, and assess its effects	Obtain test, call health care provider	17,300 site visitors during the first 40 days	“During the first 40 days of the triage tool’s deployment, the site saw more than 17,300 visitors—69.8% indicated they would have contacted the health care system if the web-based test had not been available”
Heo et al [36], 2020	Developed in South Korea. Translated into 5 languages; available worldwide	No	This study aims to aid the public by developing a web-based app that helps patients decide when to seek medical care during a novel disease outbreak.	10 levels of risk assessed; the highest advised testing; other levels suggested strength of recommendation to test, down to “no evidence of need to test.”	83,640 users in 141 countries during March 2020	An expert-opinion–based algorithm and app for patient screening and guidance can be beneficial in a circumstance where there is insufficient information on a novel disease and medical resources are limited.
Jensen et al [28], 2020	Denmark	Yes—Copenhagen Emergency Medical Services	To track call volumes and track web-based COVID-19 self-assessment tools, and to examine the potential effect of these initiatives on reducing nonessential EMSh call volume and EMS queue time in the ongoing pandemic	Hotline for additional evaluation, self-quarantine and monitor symptoms, educational materials	24,883 users	“The web triage was widely used with more than 107,000 users from its launch. However, no effect on call volume is indicated or documented. Users were mainly younger adults.” “The web triage was limited in interaction, and as expected, not all symptoms were presented; consequently, some potentially infected persons could have been missed.” “The authors find that the web triage might run the risk of being too simple to be useful for some. Furthermore, without revisions, some citizens might not trust the answers owing to the simplicity and rigidity of the first version.”
Jaeger et al [6], 2011	Canada	Yes, integrated into a campus health care clinic website	To develop a tool to ease the burden of H1N1 influenza on a campus clinic by promoting self-care, generating medical notes, and identifying vulnerable students	The resulting screen described steps for self-care along with instructions as to when, where, and how to seek further medical help if needed	1432 users	Integrating the triage tool into a university or campus clinic showed that “real-time influenza surveillance data from a campus community can be achieved by student-initiated, web-based input. This process is invaluable in monitoring influenza activity on campus, providing timely health advice, decreasing unnecessary visits to the campus medical clinic, and assisting the local public health department in valuable surveillance activities.”
Jormanainen and Soininen [33], 2021	Finland	Yes—Finnish government	To describe use, users, and some performance aspects of the Finnish Omaolo COVID-19 web-based symptom self-assessment tool in Finland	Put into 3 major groups: no need for treatment, low or high priority for treatment	1,937,469 questionnaires completed	The Finnish Omaolo COVID-19 self-assessment tool classified users into 3 major groups: no need for treatment, low or high priority for treatment. In total, there were 1,937,469 responses with 220,535 categorized as high priority.
Judson et al [3], 2020	United States	Yes—University of California, San Francisco Health	To rapidly deploy a digital patient-facing self-triage and self-scheduling tool in a large academic health system to address the COVID-19 pandemic	Asymptomatic patients were asked about exposure history and provided relevant information. Symptomatic patients were triaged into 1 of 4 categories: emergent, urgent, nonurgent, or self-care, and then connected with the appropriate level of care via direct scheduling or telephone hotline.	Completed 1129 times by 950 unique patients in the first 16 days	The tool was designed to “have high sensitivity to detect emergency-level illness and high specificity when recommending self-care, both of which were greater than 85%. Despite designing the tool with this conservative approach, the most frequent triage disposition was self-care. Most of these patients did not make further contact with our health system during the subsequent 2 days. This tool may have therefore prevented hundreds of unnecessary encounters.”
Kellermann et al [10], 2010	United States	Yes—Centers for Disease Control and Prevention	To rapidly develop and deploy a digital tool that could help minimally trained health care workers, screen large numbers of patients with influenza-like illness. The purpose evolved to be to create a patient-facing self-triage and self-scheduling tool available on web	ED, contact GP, go to a walk-in clinic, stay home	2758 users retroactively assessed	Tool was implemented by several organizations, including the Centers for Disease Control and Prevention. Authors noted it is possible the tool gave some wrong advice with harm that is unknown. No adverse events owing to use of the tool was reported. Authors reported one estimate that 10,000 unneeded visits to EDs were avoided by users of the tool on one website. Prospective data are needed to understand the tool’s impact further.
Kouroubali et al [32], 2020	Greece	Yes—The Center for eHealth Applications and Services of the Foundation for Research and Technology-Hellas	The purpose of this study was to design a platform, dynamically adapted according to patient preferences and medical history, to support patient-centered information, management and reporting of symptoms related to COVID-19. The platform incorporates modules for citizens, health care providers, and public health authorities to support safety during the current crisis.	Personalized recommendations, communication, position tracing, and public health visualizations	Not reported	The developed platform (ICTi), Safe in COVID-19, offered a way “for citizens to track their symptoms over time, enhancing a sense of safety during isolation.” The platform showed high user adherence and that users did not need high technology literacy (useful for older adults).
Lai et al [19], 2020	United States	Yes—Mass General Brigham	To use an AIj tool to capture the initial broad screening categories of risk to determine whether the patient required additional consultation with a COVID-19 expert via the Mass General Brigham COVID-19 expert either via the COVID-19 hotline, via an on demand virtual consultation, or in person	Information on what to do if influenza, self-quarantine, asymptomatic, or symptomatic COVID-19. Also provides advice for pregnant women, children, and older people with risk factors.	40,000 questionnaires completed	Implementing a digital prehospital triage system (using AI with a chatbot) helped redirect patient flow and risk factor scoring and eliminating bottlenecks in health care triage. The chatbot was made specifically to Mass General Brigham, which is an academic or integrated health care system. The authors suggest AI as an underused aspect in triage, and through AI, patients will be able to access prompt, evidence-based advice, and direction to the most appropriate care setting.
Lunn et al [31], 2021	Ireland	None	An experimental study to test whether decision aids can support people on when to self-isolate	Self-isolation, call GP, restrict movements for 14 days	500 users	Decision trees or aids in general were shown to support self-isolation during COVID-19. “In all three stages, the interventions generated some statistically significant, positive outcomes. Overall, therefore, the study provides evidence that decision aids can be used to support self-isolation during the COVID-19 pandemic.”
Mansab et al [12], 2021	United States, Japan, Singapore, United Kingdom	Yes—US Centers for Disease Control and Prevention Coronavirus symptom checker; United Kingdom 111 COVID-19 Symptom Checker; Singapore COVID-19 Symptom Checker; Japan Stop COVID-19 Symptom Checker	Using 52 use cases, to compare how likely it is each tool recommends clinical assessment, to ascertain whether they differentiate mild from severe COVID-19 cases, and how well they detect time-sensitive COVID-19 mimickers, such as bacterial pneumonia and sepsis	Stay home, contact a public health preparedness clinic or a GP, go to ED. Stay home or contact medical center. Stay home, call a medical provider within 24 hours, go to ED. Stay home, call telephone triage, call telephone triage, and talk to a nurse, or go to ED.	52 case scenarios were developed and applied to each of the 4 tools.	The tools varied in ability to appropriately advise whether to stay home or go on for clinical advice or assessment, including whether to go to an ED. The United States and United Kingdom tools often advised to stay home when clinical assessment was warranted. All 4 tools failed to advise going to an ED for the case with a form of sepsis.
Morse et al [25], 2020	United States	Yes—Sutter Health System	To evaluate the user demographics and levels of triage acuity provided by a symptom checker chatbot deployed in partnership with a large integrated health system in the United States.	Chatbot directed to 1 of 8 levels of triage advice, which were grouped into 3 levels of acuity	26,646 questionnaires completed	The characteristics and recommendations of the Sutter Health AI symptom checker and chat box offered 8 levels of triage advice. Patient demographics, such as age and health literacy were shown to be important to consider when developing symptom checkers. “Over a 9-month period, we saw robust use, particularly from younger and female users. Just under half of the assessments were completed outside of typical physician office hours, suggesting that there is a significant number of low-acuity concerns for which tailored guidance is not easily accessible during off-hours”
Owoyemi et al [30], 2021	Nigeria	Yes—Nigeria Centre for Disease Control	To build a public-facing tool (Wellvis) and deploy through mobile devices for the surveillance of COVID-19 in Africa and possibly other continents	Direct to 1 of 3 levels: low risk (retake assessment after a few days, safety precautions, health information on COVID-19), medium risk (retake assessment after a few days, observe for indicative symptoms, report to Nigeria Centre for Disease Control) or high risk (self-isolate and immediately report to their respective local disease control agency)	Not reported	Mobile phone apps used for surveillance and reporting on infectious diseases showed the value of citizen participation and offering risk information and possible next steps. This 8-item triage tool showed to be useful for managing COVID-19 and the reporting of symptoms contributed to public health’s ability to understand how to relieve burden on health systems and for prevention and control.
Perlman et al [11], 2020	United States	Yes—K Health Inc	To describe the characteristics of people who use digital health tools to address COVID-19–related concerns; explore their self-reported symptoms and characterize the association of these symptoms with COVID-19; and characterize the recommendations provided by digital health tools	Social distancing, quarantine, isolation, or seeking immediate medical evaluation. Users were also informed if they were at increased risk for COVID-19 complications, and users with risk factors and symptoms were encouraged to consult a physician	71,619 users	After investigating 3 digital health tools on the K Health app to directly manage COVID-19–related concerns, the authors suggested that automated, data-driven digital health tools, as well as remote care provided by a human physician (rather than AI) can help provide health information and guidance during a pandemic. Potential benefits would be to reduce exposure and burden on health care system.
Runkle et al [24], 2021	United States	Yes—Buncombe County Health and Human Services	To assess a participatory surveillance system. The study seeks to examine whether participatory surveillance efforts can aid local health officials in predicting and understanding COVID-19 activity in the community	Call 911, stay home, connect with health care provider, get tested, self-monitor	1755 users	A public health COVID-19 self-checker was shown to be a low-cost and flexible strategy to collect surveillance data on local changes in COVID-19 symptoms and to be used to monitor the efficacy of public health responses. The tool also “provided a means for local health officials to understand how many people with COVID-19 symptoms were in contact with a health care provider, were tested, and frequently encountered barriers to accessing health care and testing resources”
Yu et al [29], 2020	China	None	To assess a smartphone COVID-19 self-triage app	Influenza symptoms: stay home and care for self; self-quarantine if exposed to COVID-19 disease; seek medical treatment if experience COVID-19 symptoms; specific instructions for special needs; connect with caregiver on web; schedule appointment at hospital; provide web-based information	Not reported	Developing a smartphone app which was a tiered tool for self-triage of COVID-19 symptoms was purported to be able to reduce burden on hospitals, provide further self-isolation instructions for users, and to help patients make hospital appointments on web or for virtual visits with health providers, such as with psychologists. The app showed to be a comprehensive tool that may help to reduce spread and panic. The authors suggest further implementation into the popular WeChat app would improve usability.

^aWHO: World Health Organization.

^bRT-PCR: reverse transcription polymerase chain reaction.

^cED: emergency department.

^dICU: intensive care unit.

^eER: emergency room.

^fGP: general practitioner.

^gIVACS: Intelligent Voice Assistant for Coronavirus Disease (COVID-19) Self-Assessment.

^hEMS: emergency medical service.

ⁱICT: information and communications technology.

^jAI: artificial intelligence.