Initial Development of User-Based Quality Evaluation Questionnaire of Smartwatch Technology for Applying to Healthcare

Abstract

Background:

Smartwatches are a consumer wearable device offering a potential, practical, and affordable method to collect personal health data in healthy adults. For patients with chronic diseases, this would enable symptom monitoring and aid clinical decision making. Therefore, providing customized checklists to recommend smartwatches is beneficial. However, few studies have evaluated the practical functions of smartwatches and their influence on user acceptance. We aimed at developing a reliable tool to assess the quality of smartwatches from the users’ perspective.

Methods:

To develop the smartwatch rating scale (SWRS), we conducted a comprehensive literature review as well as reviewed relevant websites. The SWRS includes 22 items for the usability (usability, functionality, safety, material, and display) and five items for the acceptance and adoption domain (satisfaction and intention). We measured the scale’s internal consistency and inter-rater reliability by evaluating seven smartwatches.

Results:

The overall scale demonstrated an excellent level of internal consistency (Cronbach’s alpha = 0.91), with each subscale’s internal consistency above good level (0.74 ~ 0.92). Inter-rater reliability using intraclass correlation coefficients (ICC) was at good level (2-way random ICC = 0.82, 95% CI 0.09 – 0.97).

Conclusions:

The SWRS is reliable, which can meet the need for assessment of smartwatch technology for utilizing in personal healthcare. Accounting for users’ perspectives will help make the most of technology without impairing the human aspects of care, this study can help consumers choose a smartwatch based on their preferences and provide guidelines for developing user-friendly wearable devices aimed at health behavior changes.

Introduction

In recent years, the development of wearable technology is increasing (1,2). Their usage has also increased, and wearable technology now plays a pivotal role in the lives of users (3,4). The worldwide market for wearable devices grew 82.3% in the fourth quarter of 2019 according to International Data Corporation (5). Wearable technology for the wrist is at the center of the fast-expanding wearable market (6,7). Smartwatches that are the electronic watch and computer with smartphone applications installed.

Smartwatches are highly promoted in the information and communications technology industry for their multiple functions that interest users, such as notifications that are synchronized with smartphones and other applications; smartwatches can also offer continuous data monitoring functions, such as step-counting, heart rate tracking, energy consumption as well as physical activity monitoring, which can promote health (7–9). Recent studies have confirmed the positive effects of smartwatches on preventive healthcare and self-management for chronic diseases; they can also function as a tool offering real-time health information to healthcare professionals (10–13). For example, by automatic monitoring of smoking episodes using the smartwatch, motivated smokers were able to facilitate smoking reduction (14). Smartwatches providing real-time feedback also improved the quality of cardiopulmonary resuscitation performed by healthcare providers (15).

However, despite the usefulness and effectiveness, the effort to assess the user-based quality evaluation of smartwatch has lagged when compared to smartphones and tablets, and few studies have been conducted on the factors of smartwatch adoption (16,17). One of the main reasons for the low user acceptance of smartwatch could be the excessive number of choices available, given little accurate information or knowledge about the devices (18). Previous studies have reported that navigate through the large number of available brands, such as Fitbit, Xiaomi, Apple, Garmin, and Samsung, is a burden for consumers (1,19–21). With such difficulties, therefore, it is no surprise that consumers heavily take into account the brand image and price of the product rather than their functional features (4). In addition, some studies have found that perceived uselessness, perceived price, perceived novelty, and self-efficacy of smartwatches had an impact on consumer resistance to using them (22,23).

Meanwhile, other electronic devices, such as smartphones, tablets, or laptops, have huge amounts of information available to consumers (24). In the case of mobile phone applications, consumers can readily view comprehensive consumer star ratings and reviews (22,25). For example, the user version of the Mobile Application Rating Scale (MARS), a simple and objective quality evaluation tool for consumers, serves as a credible and reliable guideline for choosing the best health mobile apps available, but no such tool exists for smartwatches (26,27). Thus, there is need to develop a comprehensive rating scale that is parallel to the MARS for smartwatches. However, previous studies regarding smartphones have mostly been focused on the extension of the technology acceptance model or unified theory of acceptance and use of technology and studied their implication on the factors that are associated with usage perception of users (28,29). Such approaches may help understanding the motivation of purchasing a smartwatch but tell little about what practical functions potential users are seeking. Even though Dehghani’s (2016) work categorized the adoption factors a more detailed level, the study still does not discuss quality assessment for adopting smartwatches (30).

Thus, we aimed to develop a smartwatch rating scale based on user aspects and to test its reliability for applying to personal healthcare based on user preferences.

Materials and Methods

Development process outline

This was a methodological study for developing a smartwatch rating scale (SWRS). To create the rating scale, we reviewed a number of published articles about acceptance, adoption, and usability of smartwatch and websites about rating smartwatches. Based on the literature review, keywords for smartwatch rating scales were extracted. By using the extracted keywords, we initially developed the SWRS for assessing the user-based quality evaluation.

Smartwatch rating scale (SWRS) development Extracting keywords from literature review and related websites

We conducted a comprehensive literature search on the factors that influence smartwatch adoption, such as the price, aesthetics, and functionalities, in PubMed, ACM digital library, IEEE-Xplore, and Google scholar. The search keywords were “smartwatch” AND “criteria” OR “assess” OR “evaluate”. Employing our inclusion criteria, we included articles that were focused on the selection criteria of wearable technology.

A total of 56 articles were found. After initial review, we removed 48 articles that did not explore the adoption of smartwatch technology and were left with 8 articles. However, the articles provided broad categories, such as aesthetics and daily use. As an alternative method, we searched readily available websites that provide comprehensive reviews and comparisons of smartwatches with same keywords adding “comparison” OR “chart” for a more explicit criterion. Out of the search results, we selected the websites that provide an explicit and unified rating criterion. Of the smartwatch reviewing websites, 16 met the inclusion criteria. We extracted a total of 141 keywords about the criteria of smartwatch adoption considering the user aspect from 8 articles and 16 websites. The irrelevant keywords, duplicates, and keywords with low frequency (appearing less than twice) were removed; 48 keywords remained (Supplementary Fig. 1- Not published. Readers may contact the authors if needed).

Creating a draft of the SWRS

An expert panel with extensive experience in medical informatics and ubiquitous health categorized the 48 keywords into the usability domain. Then, each keyword was converted into a corresponding question, utilizing a 5-point Likert scale for a quantitative survey. The usability domain of the scale consisted of 22 items about smartwatch quality, including whether the smartwatch is comfortable for daily use, does it have the functions that consumers demand, is it safe and strong enough for daily use, is its material appropriate, and is the smartwatch’s display clear and readable. The 22 items of assessment criteria identified in previous research were extracted. There is no criteria or evidence driven framework for smartwatch quality evaluation, we used relevant questions of a reliable tool called MARS which is most popular scale that evaluate the quality of health-related apps (26). The usability domain was divided into usability (five items), functionality (six items), safety (three items), material (four items), and display (four items). A smartwatch could be considered a computer, but at the same time, its aesthetic features significantly influence consumer adoption (4). This implies that potential consumers take into consideration numerous subjective factors when choosing a smartwatch. A rating scale to measure the user-based quality of smartwatches should incorporate this subjective feature. Therefore, we added the acceptance and adoption domain which includes five items to evaluate the satisfaction and intention of user based on the Mobile Application Rating Scale (26). Finally, the SWRS includes two domains with 27 items.

Testing the internal consistency and inter-rater reliability of the final SWRS

To test the internal consistency and inter-rater reliability of the SWRS, we checked the market ranking data of current mainstream market-leading wrist-wearable devices (3). The seven smartwatches were selected for the following reasons. First, we selected the internationally renowned smart bracelets, according to market research data by NPD and Canalys (31, 32), Fitbit had the largest market share and also exhibited a good market performance. So, Fitbit Surge (Fitbit Inc) was chosen to represent the foreign-made smart bracelets. Second, the shipment of the Mi Band in the health tracking devices market has been second to that of Fitbit. Thus, the Mi Band (Mi, China) was chosen to represent the Chinese brand bracelets (33). Third, when choosing smart watches, given that the functions of smart watches and mobile phones were close and that mobile phones from Samsung and Apple were leading products in the market, Samsung Gear S (Samsung Inc) and Apple Watch (Apple Inc) were chosen to represent the smart watches (34). Then, we selected the seven smartwatches as representative; Fitbit Charge HR, Fitbit Blaze (Fitbit Inc., San Francisco, CA, USA), Samsung Gear-fit 2, Samsung Gear S2, Samsung Gear S3 (Samsung Electronics Co., Ltd, Suwon, South Korea), Apple Watch 2 (Apple Inc., Cupertino, CA, USA), Xiaomi Mi-Band 2 (Xiaomi Inc., Beijing, China) (Supplementary Table 1- Not published. Readers may contact the authors if needed). After using each device for a minimum of three days, two researchers completed the survey for quality evaluation.

The internal consistency of the SWRS was calculated using Cronbach’s alpha. This indicates the degree (correlations) to which items measuring the same general construct produce similar scores. Inter-rater reliability of the SWRS was determined by the intra-class correlation coefficient (ICC) reflecting the accuracy that the rating process was based on. We performed ICC with a two-way random effect, average measures model with absolute agreement that regards evaluators as random effects, because the developed scale will be used by several evaluators (35,36). Confidence intervals (CIs) for all ICCs were calculated to assess whether they differed from each other.

Results

Final version of the SWRS

As shown in Table 1, the final form of questionnaire for assessing user-based quality evaluation of smartwatch composed of usability (five questions), functionality (six questions), safety (three questions), material (four questions), and display (four questions) for the usability domain and five questions for the acceptance and adoption domain, with a total of 27 questions. The self-reported scale for the usability questions was rated on a 5-point Likert scale, from 1-point (“Very bad”) to 5-point (“Very good”). Additionally, responses to the acceptance and adoption domain were categorized into the total score, which was calculated by summing the 27 items. Scores ranged from 27 to 135, and higher scores indicated better overall quality of a smartwatch.

Table 1:

The final version of the Smartwatch Rating Scale

Items		Rating scale
Usability and acceptance domain
Usability 1. Is the size of the smartwatch appropriate/comfortable for daily use? 2. Is the weight of the smartwatch appropriate/comfortable for daily use? 3. Does the battery of the smartwatch last long enough? 4. Is the charging method of the smartwatch convenient? 5. Is the outer appearance of the smartwatch attractive?		Very bad	Bad	Moderate	Good	Very good
Functionality 6. Is it convenient to access the main functions of the smartwatch? 7. Is the gestural movement of the smartwatch (touch screen, scroll, etc.) smooth? 8. Does the smartwatch respond quickly enough? 9. Is the smartwatch compatible with user’s phone? 10. Is the memory of the smartwatch large enough? 11. Does the smartwatch accurately measure what it should be measuring (Steps, calorie consumption, sleep, etc.)?		Very bad	Bad	Moderate	Good	Very good
Safety 12. Does the smartwatch have water resistance? 13. Does the smartwatch have dust resistance? 14. Is the smartwatch safe from external shocks or scratch?		Very bad	Bad	Moderate	Good	Very good
Material 15. Is the material of the watch appropriate (in terms of design, texture, strength, etc.)? 16. Is the case material of the watch appropriate (in terms of design, texture, strength, etc.)? 17. Is the band material of the watch appropriate (in terms of design, texture, strength, etc.)? 18. Is the band lock of the smartwatch comfortable to use?		Very bad	Bad	Moderate	Good	Very good
Display 19. Is the size of the display screen appropriate for daily use? 20. Is the screen resolution appropriate? 21. Is the layout and design of the screen appropriate? 22. Is the readability of the smartwatch (under sun, in dark, etc.) appropriate?		Very bad	Bad	Moderate	Good	Very good
Adoption domain	Rating scale
1. Are you satisfied with the functions of the smartwatch? 2. Are you satisfied with the price of the smartwatch? 3. Do you intend to recommend the smartwatch to your family members/friends/acquaintances? 4. How many times do you intend to use the smartwatch over a week? 5. What is your overall rating for the smartwatch?	Very dissatisfied	No	Somewhat		Yes	Very satisfied
	Very dissatisfied	No	Somewhat		Yes	Very satisfied
	Not at all	Maybe	Possibly		Yes	Definitely
	Not at all	1–2 times	3–4 times		5–6 times	Everyday
	1	2	3		4	5

Internal consistency and inter-rater reliability of the SWRS

As shown in Table 2, we confirmed the internal consistency and inter-rater reliability using ICC. The overall score of SWRS for the selected seven smartwatches demonstrated an excellent level of internal consistency (Cronbach’s alpha = 0.91). Each subscale’s internal consistency examined was above good level (0.74 ~ 0.92). The SWRS ratings of the seven smartwatches had a significantly good inter-rater reliability by the CI (2-way random ICC = 0.82, 95% CI = 0.09 – 0.97), meaning that the SWRS had an acceptable inter-rater reliability (35,36). However, the ICC of safety and material subscales were not significant.

Table 2:

Internal consistency and inter-rater reliability of the usability and acceptance domain (N=7)

Item	Subscale	Mean	SD	Internal consistency	Inter-rater reliability (95% CI) / p value
	Usability
1	Size	3.71	0.95	0.91	0.96 (0.79 ~ 0.99)/ P = 0.001
2	Weight	4.14	0.90
3	Battery	3.14	0.69
4	Charging	3.71	0.76
5	Appearance	3.86	1.07
	Functionality
6	Accessibility	4.14	0.38	0.92	0.85 (0.04 ~ 0.98)/ P = 0.023
7	Gesture	4.59	0.76
8	Speed	4.14	0.69
9	Memory	3.43	0.79
10	Measurement accuracy	3.71	0.76
11	Phone compatibility	4.29	0.95
	Safety
12	Water resistance	3.00	0.82	0.76	0.56 (−1.57 ~ 0.93)/ P = 0.177
13	External shock	3.00	0.58
14	Dust resistance	3.43	0.54
	Material
15	Material	4.00	0.58	0.74	0.24 (−3.30 ~ 0.87)/ P = 0.375
16	Case	3.86	0.69
17	Strap	4.29	0.76
18	Lock	3.14	0.90
	Display
19	Screen size	3.57	0.98	0.90	0.92 (0.57 ~ 0.99)/ P = 0.003
20	Resolution	3.86	0.90
21	Layout	3.86	0.69
22	Readability	4.29	0.49
	Total	3.77	0.45	0.91	0.82 (0.09 ~ 0.97)/ P = 0.027

Note, Internal consistency: Cronbach’s alpha / Inter-rater reliability: Intraclass correlation coefficient (ICC)

Discussion

The SWRS developed in this study provides a function chart which could be used as a checklist when searching for a smartwatch which has a specific function. The SWRS consisted of 27 items to evaluate usability, functionality, safety, materials, comprising the usability domain, and to evaluate user’s satisfaction as the acceptance and adoption domain, which is thought to reflect the result of a previous study that revealed that users’ intention of a smartwatch is affected by its usefulness, enjoyment, cost, and satisfaction (18). In addition, user-based perspective of the SWRS may help consumers choose a smartwatch that has functions tailored to their needs and help doctors recommend a wearable device to their patients for personalized symptom monitoring to allow for better disease management. Smartwatches are used in various interventions because they can collect individual information of users in real-time and identify physical information according to the situation (2). It is essential to select the most appropriate smartwatch for each intervention, and the SWRS can be used as an obvious criterion to make this selection. Conducting usability testing on the smartwatch prior to intervention could also increase patients’ compliance (8).

Data reliability is a critical issue, especially when data are used to develop the tool or create a program (37). Inter-rater reliability was assessed for the quantitative measurements scored by both raters in the composite scorecard using weighted kappa with a range of zero (no agreement) to one (perfect agreement) (38). In the present study, ICC was used to examine whether there was inter-rater reliability. The standard of ICC is 0.0 to 0.20 for slight, 0.21 to 0.40 for fair, 0.41 to 0.60 for moderate, 0.61 to 0.80 for substantial, and 0.81 to 1.00 for almost perfect (39). Our findings showed that the total ICC for the SWRS has high reliability within and between the assessors. However, the safety and material among the five subscales of the usability domain did not have significance for ICC. Less reliable and accurate ratings may result when data are rated in the middle of the rating scale, are ambivalent, hard to rate, or rated with low confidence. Consequently, it is especially important to the SWRS to review those items, and to carefully consider item length and vocabulary usage. Furthermore, prior training of raters would be helpful as it would standardize the information provided in these components. Future research is required to determine how to best evaluate the safety and material of smartwatch technology. The issue of measurement reliability is an important and necessary first step in determining a tool’s utility (36,37). However, further research into other psychometric properties, like test-retest reliability, which would compare the entire assessment process over two time points and exhibit concurrent validity with advanced camera systems, is warranted.

One interesting finding is that the number of functions contained within a smartwatch did not necessarily mean higher objective quality. These results seem to be related to the lack of consumer’s knowledge of the smartwatch function or lack of interest in function operation. Therefore, it is thought that smartwatches that reflect the common needs of users, regardless of their age or experience in use of smartwatches, will be more acceptable to them.

There are some limitations. First, this study included keywords which were extracted from non-scholarly articles and multiple sources, which can be the most frequently appearing or newly emerging keywords. Accordingly, this approach can draw a selection bias. Second, only a few well-established brands were assessed to test the inter-rater reliability. Third, only two raters assessed the reliability for SWRS, thereby affecting the accuracy of the inter-rater reliability measured. Last over the course of time it took to complete this research, much has changed with technology, market share and the number of related studies has changed. The smartwatches can monitor heart rate and respiration rate with and provide important information regarding arrhythmia.

Conclusion

The SWRS is reliable, which can meet the need for assessment of smartwatch technology. Accordingly, the SWRS could provide evidence-based information to support the need for smartwatches among customers and healthcare providers. Moreover, the SWRS could be used to identify problems between smartwatch types and their intended user populations so that developers can make responsive design changes. We suggest that raters should be sufficiently and appropriated trained. Future research is required to determine the suitability and reliability of the SWRS using larger samples.

Journalism Ethics considerations

Ethical issues (Including plagiarism, informed consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc.) have been completely observed by the authors.