Utility and usability of an automated COVID-19 symptom monitoring system (CoSMoS) in primary care during COVID-19 pandemic: A qualitative feasibility study

Abstract

Background

COVID-19 telemonitoring applications have been developed and used in primary care to monitor patients quarantined at home. There is a lack of evidence on the utility and usability of telemonitoring applications from end-users’ perspective.

Objectives

This study aimed to evaluate the feasibility of a COVID-19 symptom monitoring system (CoSMoS) by exploring its utility and usability with end-users.

Methods

This was a qualitative study using in-depth interviews. Patients with suspected COVID-19 infection who used CoSMoS Telegram bot to monitor their COVID-19 symptoms and doctors who conducted the telemonitoring via CoSMoS dashboard were recruited. Universal sampling was used in this study. We stopped the recruitment when data saturation was reached. Patients and doctors shared their experiences using CoSMoS, its utility and usability for COVID-19 symptoms monitoring. Data were coded and analysed using thematic analysis.

Results

A total of 11 patients and 4 doctors were recruited into this study. For utility, CoSMoS was useful in providing close monitoring and continuity of care, supporting patients’ decision making, ensuring adherence to reporting, and reducing healthcare workers’ burden during the pandemic. In terms of usability, patients expressed that CoSMoS was convenient and easy to use. The use of the existing social media application for symptom monitoring was acceptable for the patients. The content in the Telegram bot was easy to understand, although revision was needed to keep the content updated. Doctors preferred to integrate CoSMoS into the electronic medical record.

Conclusion

CoSMoS is feasible and useful to patients and doctors in providing remote monitoring and teleconsultation during the COVID-19 pandemic. The utility and usability evaluation enables the refinement of CoSMoS to be a patient-centred monitoring system.

1. Introduction

Telemonitoring has been used in outpatient settings to improve self-care in patients with chronic diseases [1]. Telemonitoring refers to the transmission of patients’ symptoms and physiological data directly to healthcare providers via automated electronic means, web-based or phone-based data entry [2]. During the COVID-19 pandemic, telemonitoring systems were deployed in several countries to monitor patients with suspected or confirmed COVID-19 infection [3], [4], [5]. Besides that, teleconsultation, which is defined as synchronous or asynchronous consultation using information and communication technology [6], was also incorporated into the COVID-19 monitoring system to assess patients who required medical consultations. Telemonitoring with teleconsultation has emerged as a valuable tool for remote monitoring and reducing healthcare resources and hospitalisation [7], [8].

COVID-19 telemonitoring applications can enhance patient safety at home while reducing the risk of transmission to healthcare workers [9]. However, patient engagement and sustainability were reported barriers to implementing COVID-19 monitoring systems [10], [11]. Normally, digital applications for non-communicable diseases will undergo beta-testing at the initial phase to evaluate the utility and usability and identify changes before implementation [12], [13]. Unfortunately, due to the urgency of a pandemic, digital COVID-19 monitoring apps were developed and launched hastily without evaluating the utility and usability of these apps. Digital applications have been proven effective in COVID-19 surveillance, but some were reported to have usability issues [14].

Utility is defined as the functionality of the application and how useful it is to the users, while usability is defined as how easy to use and pleasant are functions to the users [15]. Experts recommend that patients’ acceptance of telemonitoring technology should be assessed before widespread implementation [16], [17]. In addition, end-users involvement in beta testing improves the utility and usability of an app [17]. Hence, engaging patients and healthcare providers in the utility and usability evaluation process of a COVID-19 telemonitoring system is important before its widespread implementation during a pandemic situation.

As the evidence of COVID-19 evolves rapidly, recommendations and guidelines are frequently revised following the latest evidence and pandemic situation. This makes the development of a telemonitoring system challenging; therefore, a rapid assessment of the utility and usability during the development process is critical to ensure the system can be updated and enhanced quickly. Many studies have reported their development and outcomes of the COVID-19 telemonitoring system [4], [18], [19]. However, there was a lack of evidence on users’ experiences of using a COVID-19 telemonitoring system. Therefore, this study aimed to evaluate the feasibility of implementing CoSMoS by exploring its utility and usability with end-users, including patients and doctors, in a primary care setting.

2. Methods

2.1. Study design

A qualitative approach was used to evaluate in-depth the utility and usability of CoSMoS from patients’ and doctors’ perspectives. Research ethics approval was obtained from the Medical Research Ethics Committee of the University of Malaya Medical Center (MRECID No. 202043–8434).

2.2. Study setting, sampling and recruitment

This study was conducted at the outpatient primary clinic and emergency department of University Malaya Medical Center (UMMC), one of the COVID-19 centres in Malaysia. This study was conducted from April 24 to May 21, 2020. Universal sampling was used to recruit all the patients and doctors who have used CoSMoS during the study period. The inclusion criteria for patients were: diagnosed with suspected COVID-19 infection or close contact with a COVID-19 patient, and were attending the primary care clinic and emergency department during the study period. For healthcare providers, doctors who were involved in recruiting and monitoring the patients using CoSMoS were recruited. We stopped the recruitment when data saturation was reached during data collection and analysis [20], when no additional data was found in both patient and doctor interviews.

2.3. The CoSMoS system

The COVID-19 Symptom Monitoring System (CoSMoS) was developed to provide remote home monitoring for patients with suspected COVID-19 infection in Kuala Lumpur, Malaysia. During the study period (April-May 2020), the number of COVID-19 cases in Malaysia increased rapidly, with 7059 confirmed cases as of May 21, 2020 [21]. The number of patients with suspected COVID-19 infection or close contacts who required home monitoring was much higher than the number of positive COVID-19 patients.

CoSMoS consists of an automated Telegram bot for patients and a dashboard monitoring as well as teleconsultation service by doctors. The system and algorithm were developed based on users’ needs, local guidelines, and experts’ inputs. The CoSMoS development and algorithm were described in detail in another paper [22]. CoSMoS focuses on risk stratifying the patients according to their symptoms using an algorithm-based Telegram bot, provides close monitoring to patients using a dashboard and timely teleconsultation. Once enrolled in CoSMoS, patients will be asked to report their symptoms using an interactive Telegram bot daily for 14 days. A ‘ping’ will be sent at 9 am every day to remind the patients about submitting their symptoms. The algorithm-based Telegram bot stratifies the patients according to their symptoms, reassures them if they are stable, or alerts patients and doctors when the symptoms change or worsen. Patient’s data are also sent to a centralized CoSMoS dashboard at the hospital, where the doctors monitor each patient’s condition in real-time. Fig. 1 shows the CoSMoS Telegram bot for patients, while Fig. 2 shows the CoSMoS dashboard for doctors to monitor the patients.

Fig. 1.

The CoSMOS Telegram bot for patient use.

Fig. 2.

The CoSMoS dashboard for doctor use to monitor patients.

2.4. Data collection

After clinic consultations, patients were invited to enroll in this study and use CoSMoS to monitor their conditions at home. The patients were given a paper-based participant information sheet (PIS). Patients were guided on how to download the Telegram app and activate the CoSMoS bot. Once activated, patients were asked to select their preferred language, read the terms of use and provide consent. The recruiting doctors then guided the patients to enter their symptoms into the CoSMoS bot. Patients continued to use the CoSMoS for 14 days at home. Once the patients had completed the 14-day symptom monitoring using CoSMoS (at day 15), they were contacted by the researchers for interviews. The in-depth interviews (IDIs) were conducted via phone calls guided by a topic guide developed based on Neilsen’s model of usability and utility [23]. The participants were asked to give their opinion regarding the CoSMoS, recruitment process, functions and features of CoSMoS.

Doctors who used the CoSMoS monitoring dashboard and teleconsultation were recruited. Paper-based written consents were taken from the doctors who agreed to participate in this study. These doctors attended a two-hour training course before using the CoSMoS system. After using CoSMoS to monitor patients for 3–4 weeks, IDIs were conducted face-to-face.

2.5. Data analysis

After each interview, the researchers listed the issues and comments raised with the aid of the field notes. All audio recordings were transcribed verbatim and checked independently by two researchers for accuracy (HML, AA). Two researchers (HML, IGV) immersed themselves in the data and assigned open codes for each statement independently. The codes were then mapped onto Neilsen’s model of usability and utility to systematically categorise the findings under usability and utility [23]. Axial coding was done subsequently to create the sub-themes and themes within usability and utility. The research team then met to discuss the coding of the transcripts. Any discrepancy of the coding was resolved through discussion with the other research team members (AA, CJN, CHT, HH, WLN and AMNA) until a consensus was reached. Data triangulation from both patients’ and doctors’ codes was done to enhance the credibility of the study findings. Coding of the transcripts was performed using Atlas.ti v8.

3. Results

3.1. Participants demography

A total of 15 participants (11 patients and 4 doctors) participated in the beta-testing and in-depth interviews. All 11 patients completed the 14-day monitoring using CoSMoS. The details of the participants are shown in Table 1 .

Table 1.

Characteristics of participants recruited for in-depth interviews (N = 15).

ID	Age (years)	Ethnicity	Gender	Patient Status
Patient 1	29	Malay	Female	Close contact
Patient 2	37	Malay	Female	Suspected of COVID-19
Patient 3	34	Malay	Female	Close contact
Patient 4	31	Malay	Male	Close contact
Patient 5	36	Malay	Male	Suspected of COVID-19
Patient 6	37	Malay	Male	Suspected of COVID-19
Patient 7	63	Malay	Female	Suspected of COVID-19
Patient 8	35	Malay	Male	Close contact
Patient 9	46	Malay	Female	Suspected of COVID-19
Patient 10	60	Indian	Male	Suspected of COVID-19
Patient 11	49	Indian	Male	Suspected of COVID-19
Doctor 1	34	Chinese	Male	–
Doctor 2	37	Malay	Female	–
Doctor 3	37	Chinese	Female	–
Doctor 4	34	Indian	Female	–

3.2. Utility

Five themes emerged from both the patients and doctors regarding the utility of CoSMoS. Table 2 shows a summary of themes and quotes regarding the utility of CoSMoS.

Table 2.

Summary of the themes and quotes regarding the utility of CoSMoS, using Nielsen’s model of system acceptability (1993).

Themes	Representative quotes
Theme 1: Providing close monitoring for patients who were quarantined at home	…I think it’s a very good way. It’s kind of a personal touch,…we’re already stressed out when we get sick…with this (CoSMoS), you get a kind of reassured that everything is okay…Somebody is taking personal care for you, [Patient 10]
Theme 2: Providing continuity of care to patients	When they (patients) come here, they already know us (doctors), then we continue to monitor them at home. So I think with that rapport… it’s easier because of the trust, doctor and patient relationship is already established. [Doctor 2]
Theme 3: Providing timely decision support for patients and doctors	I can understand the message, so okay, I stay at home, or if there is any change (of symptoms), directly go to the hospital. [Patient 5] We can triage the patient from the CoSMoS system…those with mild symptom, maybe we can give advice by phone and patients don’t have to come. [Doctor 2]
Theme 4: Ensuring adherence to daily symptoms reporting	… the system always reminds me at 9o’clock every morning to write the symptoms, which was really good because sometimes I used to forget. [Patient 11]
Theme 5: Reducing the burden of health care workers during a pandemic	Also, we can reduce the burden of the doctors in the clinic…We also can protect our doctors, our healthcare workers. [Doctor 2]

3.2.1. Theme 1: Providing close monitoring for patients who were quarantined at home

Patients found that CoSMoS was useful in providing remote monitoring without having to visit a doctor. Patients felt better when doctors monitored them closely through the CoSMoS system. CoSMoS reassured patients by providing close monitoring. It reduced their uncertainty and emotional stress during the COVID-19 pandemic.

3.2.2. Theme 2: Providing continuity of care to patients

CoSMoS provided continuity of care to patients after being discharged from the primary care clinic. Doctors found that CoSMoS enabled them to build a good rapport with patients and developed a trustable patient-doctor relationship.

3.2.3. Theme 3: Providing timely decision support for patients and doctors

Patients commented that CoSMoS helped them decide whether to go to the hospital or stay at home when their symptoms changed. Patients found that the automated feedback given to them after entering their symptoms was useful in their decision-making. They felt that the advice given were appropriate to their condition and easy to understand.

Doctors expressed that CoSMoS helped to triage patients according to the severity of symptoms. Patients with mild symptoms continued with home monitoring while those with warning signs would be flagged out in the CoSMoS dashboard for doctors’ attention.

3.2.4. Theme 4: Ensuring adherence to daily symptoms reporting

Both patients and doctors found that the reminder system (a “ping” from the Telegram at 9 am, 12 pm, and 1 pm) helped ensure patients check their symptoms daily. Sometimes, patients did not receive the reminder because they have disabled the notification. Despite so, the CoSMoS doctor would call and remind the patients to check in their symptoms if they have not reported their symptoms by 2 pm.

3.2.5. Theme 5: Reducing the burden of health care workers during a pandemic

The doctors expressed that CoSMoS helped ease healthcare workers' burden by reducing unnecessary clinic visits and the risk of COVID-19 transmission to healthcare workers.

3.3. Usability

Table 3 shows the themes and quotes regarding the usability of CoSMoS.

Table 3.

Summary of the themes and quotes regarding the usability of CoSMoS, using Nielsen’s model of system acceptability (1993).

Usability	Themes	Subthemes	Quotes
Learnability (How easy is it for users to accomplish basic tasks the first time they encounter the design?)	Theme 1: Familiarity with existing apps smoothen implementation		Oh, the process is easy. Just search for ‘CoSMoS’ in the Telegram. Ah, it is not difficult at all. [Patient 5]
		Subtheme 1a: Unfamiliarity with app poses challenges in installation	WhatsApp’s is something that people are very familiar with…I never used Telegram, so I don’t know how is it before, so that’s why there’s a bit of hiccup at the beginning part and then you feel like, ah lah (oh man!), need to download app. [Patient 3]
		Subtheme 1b: Using different approaches to guide patients to install the app	Ah, so she (doctor) explained everything…step-by-step, download the Telegram, and we answer the questions through the Telegram…ah Okay. [Patient 8] With the leaflets given, and on-going consultation through the phone together step by step, I think, the patient did quite okay. [Doctor 3]
Efficiency (Once users have learned the design, how quickly can they perform tasks?)	Theme 2: User-centered and evidence-based content of home monitoring app	Sub-theme 2a: Use of simple questions with a user-friendly interface	…although they are standard questions, they are very helpful questions, simple, because they help me ensure that I’m answering correctly without forgetting. [Patient 11]
		Sub-theme 2b: Updating the content according to the latest evidence	I realized in CoSMoS, we don’t ask anything about anosmia, or hyposmia, change of smell [Patient 3] I feel maybe they (CoSMoS) can ask for more questions? Because when I went online I saw many more COVID symptoms.[Patient 11]
Memorability (When users return to the design after a period of not using it, how easily can they re-establish proficiency?)	No theme emerged under this component
Errors (How many errors do users make, how severe are these errors, and how easily can they recover from the errors?)	Theme 3: Need for integrating with hospital electronic medical record		We don’t have detailed information about that patient (in the CoSMoS Dashboard), so we have to open the iPesakit (the name of UMMC EMR system) to see. [Doctor 4]
		Sub-theme 2c: Mismatched needs between patients and doctors regarding free text feature	That is really necessary because, for me, I have stomach symptoms, stomach ache, gastritis. [Patient 5] He (patient) skipped the questions and then at the “Remark”, it was like five to six symptoms and the calling status will be “no need to call”. [Doctor 4]
		Sub-theme 2d: Redundant content in the Telegram bot	I do not understand how to look at the ‘blue’. If it turns blue, I feel it was very bad. If turns blue, the blood doesn’t flow. [Patient 4] If already bluish, patient won’t be like using the CoSMoS system…[Doctor 4]
Satisfaction (How pleasant is it to use the design?)		Sub-theme 2a: Use of simple questions with a user-friendly interface	…although they are standard questions, they are very helpful questions, simple, because they help me ensure that I’m answering correctly without forgetting. [Patient 11]

3.3.1. Theme 1: Familiarity with existing apps smoothen implementation

Overall, patients preferred to use their existing social media app for COVID-19 symptoms monitoring. Two subthemes emerged regarding the installation of Telegram and CoSMoS.

3.3.1.1. Subtheme 1a: Unfamiliarity with app poses challenges in installation

Current Telegram users had no problem installing CoSMoS. For new Telegram users, some struggled to install and navigate the Telegram. Patients preferred to use the existing app on their phones, such as WhatsApp, rather than installing a new app.

3.3.1.2. Subtheme 1b: Using different approaches to guide patients to install the app

Although patients faced some difficulties installing Telegram, they could complete the Telegram installation with guidance, such as a setup guide (visual guide) and step-by-step guidance via phone calls.

3.3.2. Theme 2: User-centered and evidence-based content of home monitoring app

Four subthemes emerged related to the content of CoSMoS.

3.3.2.1. Sub-theme 2a: Use of simple questions with a user-friendly interface

Patients commented that the questions asked in the CoSMoS bot are simple and easy to answer. Patients just clicked on the option which suits them the best. The clickable buttons improved CoSMoS’s usability greatly without having to type their symptoms.

3.3.2.2. Sub-theme 2b: Updating the content according to the latest evidence

Patients and doctors suggested adding more questions on COVID-19 symptoms, especially anosmia when more COVID-19 cases reported presenting with anosmia.

3.3.2.3. Sub-theme 2c: Mismatched needs between patients and doctors regarding free text feature

There were conflicting comments between the patients and doctors regarding the free text feature in CoSMoS. A patient commented that the free text feature was necessary. However, a doctor found that patients put all their symptoms under free text, which would not trigger the decision-making algorithm. Patients also put in non-COVID related symptoms into the free text.

3.3.2.4. Sub-theme 2d: Redundant content in the Telegram bot

Asking about cyanosis “Is your face or lips turning blue?” was one of the questions recommended by the World Health Organization (WHO) and local guidelines. However, patients had difficulty understanding the question asking about cyanosis. Doctors felt that the question asking about cyanosis was redundant.

3.3.3. Theme 3: Need for integrating with hospital electronic medical record

The doctors found that the CoSMoS dashboard was easy to use and navigate. The information in the CoSMoS dashboard was adequate to provide a remote monitoring service. However, there was a suggestion to integrate patient data from the electronic medical record (EMR) into the CoSMoS Dashboard to have more comprehensive patient data.

4. Discussion

The CoSMoS system, which includes an algorithm-driven chatbot application, a dashboard for monitoring and teleconsultation, was found to be feasible and useful by both patients and doctors for COVID-19 symptoms monitoring. During the COVID-19 pandemic, in-person primary care clinic visits were reduced to a minimum to avoid infection transmission. CoSMoS offers a virtual patient safety net as it rapidly detects symptoms deterioration and alerts the health care providers, as reported by another similar monitoring system [3]. Patients using CoSMoS valued the algorithm-based medical advice given to them via the Telegram bot as it provided reassurance to those who were clinically stable. The feedback messages in CoSMoS gave psychological support to patients. People often feel worried, anxious, and frustrated about COVID-19, especially when their symptoms worsened. [24]. Hence, the interactive feature of a telemonitoring system improves the doctor-patient relationship and provides a better communication channel for patients to express their concerns [25], [26].

The automated function of the CoSMoS dashboard risk-stratified patients according to their reported symptoms. Other COVID-19 monitoring systems have similar dashboard functions to provide synchronous monitoring of patients [18], [19]. Besides the technical features of a monitoring dashboard, close supervision by a doctor is important rather than merely relying on the automated decision support tool [7]. In CoSMoS, synchronous teleconsultation was available for patients with worsening symptoms. Synchronous teleconsultation was reported to be cost-effective in reducing unscheduled visits and travel costs in diabetes care [27]. Hence, incorporating the synchronous teleconsultation feature in a COVID-19 symptom monitoring system is crucial to reduce outpatient visits and minimize infection transmission.

For successful implementation, usability exploration is important to get end-users views and experience to improve the system further. In our study, building a telemonitoring system in a social media app is feasible and acceptable. Choosing an appropriate social media app requires considerations of its costs, the familiarity of the target population and the license application to use the existing social media apps [4]. Although studies report that facilitating condition has no relation to the mHealth adoption [28], [29], providing patients with manual and technical support improve the adoption of CoSMoS in our study.

Patients and doctors expressed the need to update the content in the CoSMoS bot because the evidence of COVID-19 evolves rapidly. This is different from developing a digital monitoring tool used for chronic diseases where parameters needed for monitoring were supported with robust clinical evidence such as heart failure [30] and chronic obstructive pulmonary disease [31]. CoSMoS development had to accommodate timely change requests with the emerging evidence of COVID-19. The development of a telemonitoring application for an emerging communicable disease should be agile and adaptable to the rapidly changing clinical evidence. Often, telemonitoring for chronic diseases focused on self-care, patient empowerment, medication adherence and behavioural change [32]; however, these components were often not emphasised in a telemonitoring system for COVID-19. With increasing COVID-19 survivors who require long-term monitoring of COVID-19 sequelae [33], the telemonitoring system may include features for rehabilitation and self-care.

This study found different needs from patients and doctors regarding the COVID-19 monitoring system. Patients and doctors have different opinions regarding the use of free text in the CoSMoS which allows patients to report other non-COVID symptoms; while patients found it very helpful. On the other hand, doctors were concerned that they could not support patients with non-COVID related health issues. Sometimes, doctors and patients have contrasting views on disease perspective and mobile health in self-care [34]. The difference in user needs could be solved with multiple rounds of discussion and feedback to reach a consensus [35]. This co-design approach with different users’ engagement is relevant and important in COVID-19 telemonitoring system to ensure successful implementation in the community [36].

The CoSMoS system was developed within a short period due to the urgent need, and it was designed as a system independent from the hospital electronic medical record (EMR). EMR poses significant barriers to infection prevention during the COVID-19 pandemic as most of EMRs are not linked to other health systems, and a lack of standardization in information management [37]. The gaps in the interoperability of different healthcare systems were accentuated during this COVID-19 pandemic. With good integration of telehealth and EMR, remote medical services can support the public during the pandemic [38].

5. Strengths and limitations

The strength of this study was the involvement of both patients and doctors. By exploring the views and experiences from both parties, richer data was captured and allowed for triangulation. The limitations of this study were that we did not interview those patients who refused to use CoSMoS. Their refusal may point to usability and utility issues that could not be explored. The IDIs were conducted via phone calls, which tends to be shorter than a face-to-face interview and lacked visual and nonverbal cues. The majority of patients in our study were in their mid-30 s and Malay ethnicity; this limits the exploration of technology acceptance from older age groups and different ethnicities. However, the demographic characteristics in our study were similar to the COVID-19 patient profile reported in Malaysia, with a median age of 34 years and more than 70% of Malay ethnicity [39].

6. Conclusions

An algorithm-based chatbot home monitoring application, coupled with a dashboard for monitoring and teleconsultation, is feasible to be implemented during the COVID-19 pandemic. The findings from this study are useful to refine the CoSMoS application for expanded use and be considered by researchers globally when designing their monitoring system for COVID-19 patients. Further studies are needed to evaluate the effectiveness of CoSMoS in improving the health outcomes of COVID-19 patients as well as in optimising the use of health care resources.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the Supplementary data to this article: