Development and Cross-cultural Validation of the Korean Version of SMArtphone’s uSability Heuristics (SMASH)

Abstract

Objectives

The purpose of this study was to develop and cross-culturally validate the Korean version of SMArtphone's uSability Heuristics (K-SMASH).

Methods

In the study, it was used the adaptation process consisted of five stages, namely, translation, synthesis, back translation, expert committee review, and pretesting. In the pretesting stage, a mobile application, using the prefinal K-SMASH, was evaluated for the severity of usability problems by three experts in computer science and informatics. Each participant completed the evaluation and was interviewed about their understanding, interpretation, and opinion of the cultural relevance of the prefinal K-SMASH. Next, we reviewed the differences in the experts’ opinions and the questionnaire results.

Results

Twelve SMASH items, words and sentences, were translated, back translated, and revised, considering the conceptual meaning in the context of the Korean culture, by experts in various fields, including a Korean linguist and a bilingual translator, through the first stage to the fourth stage. In the pretesting stage, the results showed no major differences among the severity ratings of participants. Furthermore, all participants answered that there were no critical discrepancies or inconsistencies with the cultural relevance of the prefinal K-SMASH.

Conclusions

The results of the study provide preliminary evidence that the modified K-SMASH can be used for heuristic evaluation, one of the usability tests, when developing applications in Korea.

I. Introduction

In the last few years, smartphone ownership rates have rapidly increased in many countries. A global median of 43% own a smartphone, and South Korea stands out as the country with the highest smartphone ownership rate at 88%, followed by the United States at 72% [1].

The rapid proliferation of smartphones is related to the increase of access to health information [2] and the rapid growth of mobile health (mHealth) services, which rely heavily on mobile applications (app) [3]. It is predicted that, in 2017, more than 1.7 billion people will have downloaded health apps for mHealth [4]. Extensive research interest [5] has been attracted to topics related to the effectiveness of mobile apps for purposes such as intervention and the support of decision making in clinical practice [6,7,8,9].

Usability is the extent to which a system, product, or service can be used by specified users to achieve specific goals with effectiveness, efficiency, and satisfaction [10]. Insufficient usability of a system is identified as a major obstacle in mHealth [11]. Thus, usability testing is necessary to ensure that a mobile application is practical, effective, and easy to use, especially from a user’s perspective [5].

Heuristic evaluation, introduced by Neilson [12], is one of the most widely used methods for usability testing. It has several advantages over other usability testing methods. Its implementation is easy, fast, and cheap, and three or five evaluators normally participate to examine usability [12,13]. However, traditional usability evaluation methods rely on structured task-centered measures and relatively predictable tasks, which may be more applicable to desktop applications than to mobile apps [5,14].

Bertini et al. [14] developed a set of eight usability heuristics for mobile apps, and it has been used in some studies related to the development of mobile apps in Korea [15]. However, its heuristics principles are focused on the findability of a mobile device rather than the evaluation of mobile apps [16], and some of principles are duplicated, which may lead to mistakes when evaluating usability [17]. Some studies have also conducted usability testing with tools that do not consider the features of mobile apps [18].

A set of SMArtphone’s uSability Heuristics (SMASH) focusing on the features of mobile apps was developed by Inostroza et al. [17]. The study showed SMASH to be an effective and reliable tool that can be utilized with a variety of smartphones [17]. It is expected to be useful for the evaluation of mobile apps developed in Korea; however, it would necessitate translation and cultural adaptation due to the different sociocultural context [19].

This study aimed to develop a Korean version of SMASH (K-SMASH), according to the guidelines for cross-cultural adaptation introduced by Beaton et al. [19], and to evaluate the validity of K-SMASH.

II. Case Description

1. Research Design, Process, and Results

This research was a methodological study, and permission was obtained from the developer to use and translate the original scale into Korean after approval was obtained from the Institutional Review Board of Hanyang University (IRB No. HY1-16-035-2). Details of the process are summarized in Table 1.

Table 1. Process for making the Korean version of the SMArtphone's uSability Heuristics (K-SMASH).

Stage I (Translation): Two translators conducted the forward translation into Korean independently of the original English instrument. Following the guidelines [19], two translators were selected who had different backgrounds; translator 1 was aware of the concept, ‘Usability and Heuristic’. The first translator tried to translate a more reliable equivalent form from the perspective of the measurement. Translator 2 (called a naïve translator), who was a professional (bilingual) translator who was unaware of these concepts being examined in the original instrument, performed a forward translation.

Stage II (Synthesis): The author reviewed each translation (T1 and T2) with the first translator and determined which parts had different meanings and words in comparison with the original instrument. We then discussed the selection of proper words in the context of meaning to reach a consensus. One common translation (T-12) was produced through the synthesis process. The second translator did not participate as a researcher in the meeting, but we could contact him by e-mail. However, there were no specific discrepancies or questions related to T2 during this stage.

Stage III (Back translation): Based on the guidelines of Beaton et al. [19], a professional translator who had no medical or usability background conducted a back translation (BT2), in which T-12 was translated back into the original language. Additionally, we did not provide any information on the original instrument to the translator to avoid information bias.

Stage IV (Expert committee review): The expert committee comprised six experts (Table 1). The experts were selected considering their experience related to translation work or their knowledge of the concepts being examined in SMASH. The review began via email, with an original developer verifying the differences between the back translation (BT1) and the original scale, and no major discrepancies were found. Then the expert committee reviewed and consolidated all versions to achieve equivalence between the source and the Korean version in terms of four aspects: semantic, idiomatic, experiential, and conceptual. As a result, of the 12 K-SMAHS principles, 11 except for K-SMASH 8 were revised in terms of the postposition, verb. Foreign words that were used as load words in Korea, such as the word ‘error’, were corrected into Korean expressions. Through this stage, we completed the prefinal K-SMASH.

Stage V (Pretesting): The prefinal K-SMASH was used to evaluate the mobile app, ‘Brake of My Mind, BoMM,’ which was developed for adolescent suicide attempters based on cognitive behavior therapy. It is recommended to have between 30 and 40 participants [19]; however, considering this scale is for heuristic evaluation, it was reviewed by a small group of experts who were experienced in usability testing, per the guideline introduced by Neilson [12]. Three experts participated (Table 2) and completed consent forms after being informed of the purpose of the study. The participants were provided the URL to download the app, which was available for Android phones, and they installed it on their own phones and performed a heuristic evaluation with the prefinal K-SMASH individually. Each heuristic was rated according to a 5-point rating scale from 0 to 4 for the severity of usability problems [12]. After the evaluation was completed, general characteristics were collected, and each participant was interviewed briefly by e-mail or telephone to verify his/her understanding and interpretation of each heuristic principle as well as the cultural relevance of the prefinal K-SMASH. Then, we revised and finalized the prefinal K-SMASH. All data were analyzed with SPSS 20.2.

Table 2. Severity scoring of usability and problems based on a set of SMArtphone's uSability Heurestics (SMASH) [14].

S: five point rating scale of severity (0 = I don't agree that this is a usability problem at all, 1 = Cosmetic problem only: does not need to be fixed unless extra time is available on project, 2 = Minor usability problem: fixing this should be given low priority, 3 = Major usability problem: important to fix, so should be given high priority, 4 = Usability catastrophe: imperative to fix this before product can be released), P: number of usability problems.

^aExperts: 1 computer scientist, 1 nursing professor whose major was nursing informatics, and 1 digital media designer. ^bAs the mean of usability problems, it excluded overlapping problems among experts.

2. Results of Stage V (Pretesting)

The mean career time of the experts was 18.0 ± 2.64 years. All participants had experience with developing software systems from 1 to 10 times and with a usability evaluation at least once.

The results showed that there were no major differences among the severity ratings of the participants (Table 2). Regarding usability problems, the three experts offered similar or identical opinions for almost all the heuristics. Additionally, in brief interviews, all the participants answered that there were no critical discrepancies or inconsistencies with the cultural relevance of the prefinal K-SMASH. Some of the comments, such as expert suggestions to change the place of subjects or conjunctions in sentences, were revised with a Korean linguist.

III. Discussion

This study developed and cross culturally validated K-SMASH through a (back) translation process that reflected the socio-cultural and linguistic characteristics of Korean, and included a heuristic evaluation by experts.

The results of the pretesting showed that there were no differences in the expert participants' interpretations of each questionnaire item. This indicates that K-SMASH has achieved cross-cultural adaptation, which means that the tool has secured its equivalence and content validity in the Korean context. However, the validity of a tool cannot be proved by measuring it once, but rather by ensuring its evidence continuously through various methods [19,20]. Also, this study had a limitation in that Also, this study had a limitation in that there was a small number of participants in the pretesting. Therefore, various evaluation methods will be needed continuously to secure the validity of K-SMASH.

A professional's opinion in the pretesting can be a ‘guess’ rather than an accurate understanding of an item's meaning. To minimize inaccuracy in this study, the author conducted a brief interview with each expert to find out how the expert determined the meaning of each item. Also, they were asked for their opinions on the clarity of expression, and specifically, whether any words or items were unclear in comparison with other evaluation tools which they had used before. Through this process, some measure of quality in terms of content validity was provided [19].

In the back translation stage, it is recommended that two translators check whether the translated version (T-12) accurately reflects the content of the original version [19]. In this study, one professional translator conducted back translation, which was one of its limitations. To make up for this weak point, the developer of the original instrument participated and verified the discrepancy of the back translation (BT1) in comparison with the original scale in stage IV. This process provides insight into the construction of the tool and clarifies any questions that might arise between the original scale and back translation [21].

As mobile apps become a key factor to promote patients' health and their engagement in the clinical area, it is essential to consider usability as part of app development [3]. Therefore, it is expected that K-SMASH could be used as an adapted cross-cultural guideline or as a principle of heuristic evaluation during the development of mobile apps in Korea.