Abstract
Objective
The Health Information Technology Usability Evaluation Scale (Health-ITUES) is a validated and reliable instrument to evaluate usability of information technology (IT) tools. In this study, we aimed to establish the optimal cut-point of the Health-ITUES to identify usability of IT tools.
Methods
Adult participants were recruited to a trial evaluating a mobile app for self-managing HIV. Participants completed the Health-ITUES at the 3- and 6-month follow-up. Health-ITUES is a 20-item questionnaire that assesses four subscales: impact, perceived usefulness, perceived ease of use, and user control. The total score ranged from 1–5 and a higher score indicates greater usability. App use was defined as the proportion of activities completed by participants in both study arms. The selection of an optimal cut-point involved a series of multiple linear regression models with 500 bootstrap replications to examine the relationship between the Health-ITUES total score and app use, controlling for potential covariates.
Results
We included 158 participants; mean age was 49.7 years (SD 10.3), 71% were African American/Black, and 72% were non-Hispanic. Mean Health-ITUES total scores at 3 and 6 months were 4.39 (SD 0.75) and 4.43 (SD 0.75), respectively. App use completed by participants from baseline to the 3-month follow-up visit was 0.61 (SD 0.36, range 0–1.72) and from 3-month to the 6-month follow-up visits was 0.51 (SD 0.37). Participants who reported greater Health-ITUES total score completed more activities [β=0.18, 95% Confidence Interval (CI) 0.10–0.27]. The optimal cut-point of 4.32 [95% Confidence Interval (CI): 4.25–4.56] yielded the lowest p-value to identify usability of IT tools.
Conclusions
In this study of adults with HIV, we identified an optimal cut-point of 4.32 on the Health-ITUES total score to define usability. Further studies are needed to validate this cut-point.
Keywords: mobile health, usability, information technology, Health-ITUES
1.0. INTRODUCTION
According to Pew Research Center, 85% of Americans own a smartphone, 53% own a tablet computer, and 77% own a desktop/laptop computer [1]. Given the ubiquity of these tools, it is not surprising that mobile health (mHealth) applications (apps) are increasingly being utilized for healthcare [2, 3]. MHealth apps enhance patient engagement, facilitate communication, and allow for remote data capture [4–7]. Examples of activities that can be facilitated by mHealth apps include physical activity, weight management, and symptom monitoring [4–7]. To promote uptake and adoption of mHealth apps in clinical practice, it is important to demonstrate usability, acceptability, feasibility, efficacy, and scalability [8, 9]. Usability can be defined as the effectiveness, efficiency, and satisfaction with which users achieve specified goals in a specific context of use [10]. Usability is related to a reduction in user burden and users are more likely to utilize apps that are usable [11].
In 2021, there are approximately 53,000 healthcare apps available [2]. Despite the explosion of mHealth apps, most apps are not tested in the research settings[12]. Usability is generally evaluated qualitatively (via interviews, e.g., heuristic evaluation, cognitive walkthrough, think-aloud technique) or/and quantitatively [13–15], and may also include performance-based usability testing with scenarios and end-users with usability metrics (e.g., success rate, time, satisfaction rate) [16]. Quantitative evaluation of mHealth apps often utilize established usability scales and together with other usability evaluation methods allow for more rigorous testing. Some of the usability scales were developed for general evaluation of information technology (IT) tools [e.g., System Usability Scale (SUS) [17], Post-Study System Usability Questionnaire (PSSUQ) [18], Health Information Technology Usability Evaluation Scale (Health-ITUES)] [19], while some are more specific to mHealth apps [e.g., mHealth app usability questionnaire (MAUQ)] [20], The SUS is the most commonly used, with a cut-point of >68 as being above average or usable [21]. The remaining scales do not have a cut-point, which can prove challenging when attempting to measure usability of health IT tools.
The Health-ITUES is a validated instrument derived from the Health Information Technology Usability Model (Health-ITUEM) [22]. The Health-ITUEM is a model that integrates multiple usability theories including the Technology Acceptance Model and the International Organization for Standardization (ISO) 9241–11 [23, 24]. Prior studies have shown factorial validity, internal consistency, constructive validity, and predictive validity of the Health-ITUES [22, 25]. We have previously demonstrated reliability and validity of the Health-ITUES in assessing usability of mHealth apps in community-dwelling adults [19]. We also translated and validated this in Korean language [26]. Currently, Health-ITUES does not have a cut-point to define usability and it is generally presented as a continuous score which may be challenging in its interpretation. The aim of this analysis was to establish a cut-point for usability for the Health-ITUES.
2.0. METHODS
2.1. Study design, population, and setting
Data for this analysis were derived from the WiseApp trial (NCT03205982) [27]. The WiseApp Trial is an ongoing two-arm randomized controlled trial that utilizes a self-management mobile app that contains real-time medication monitoring linked to an electronic pill bottle (Figure 1) [27]. The mobile app is also linked to a fitness tracker. Participants were 18 years of age and older with a diagnosis of human immunodeficiency virus (HIV), speak English or Spanish, take anti-retroviral therapy (ART), report adherence of 80% or less to ART as measured using the Visual Analogue Scale or have a viral load of over 400 copies/mL in the past 30 days, and own a smartphone device. Participants were recruited through community outreach. Recruitment occurred between July 2017 and April 2021. Participants provided informed consent prior to undergoing any study procedures. The study was approved by the Columbia University Medical Center’s Institutional Review Board.
Figure 1:
WiseApp utilizes a self-management mobile app that contains real-time medication monitoring linked to an electronic pill bottle
2.2. Health-ITUES
The Health-ITUES was collected from participants via a Qualtrics survey administered at the 3- and 6-month follow-up visits during the WiseApp Trial. The Health-ITUES consists of 20 items measuring four subscales [22]: 1) Impact, 2) Perceived usefulness, 3) Perceived ease of use, and 4) User control. Each item is rated on a 5-point Likert scale ranging from 5 (strongly agree) to 1 (strongly disagree). The total score was created as the average of the four domain scores; it ranged from 1–5 and a higher score reflecting greater usability.
Compared to our prior validation study, items in the Health-ITUES were customized in the following ways [19]: 1) The name of the app was changed to HealthStar; 2) Changes in wording to reflect the purpose of the app (e.g., “monitor and learn about my health”, “track my health”, “managing my health”; Supplemental Table 1).
2.3. Dependent Variable – App Use
Participants in both study arms were asked to complete a total of 234 activities (described below) within the 6-month study duration; 117 activities between baseline and the 3-month follow-up visit and 117 activities between the 3-month and 6-month follow-up visits. The dependent variable app use was defined as the proportion of activities completed by participants (verified by log files) out of 117 activities assigned within each follow-up interval. Participants randomized to the intervention arm were asked to take ART medications daily (monitored by a linked electronic pill bottle), view weekly HIV and ART adherence educational content (articles and videos), and complete weekly surveys or quizzes about general wellness topics. Participants in the intervention arm were also reminded by the app to take their daily medications. Participants randomized to the control arm were asked to walk 5,000 steps a day (monitored by a linked fitness tracker), view weekly educational content (articles and videos) about health and wellness not specific to HIV/ART, and complete weekly surveys or quizzes about general wellness topics. Wellness topics included exercise, fatigue, healthy habits, mental wellness, nutrition, self-care, social support, and sleep.
2.4. Covariates
Covariates were collected at the baseline visit. These included age, gender, race, ethnicity, education, annual income, relationship status, sexuality, whether they have children (yes vs. no), confidence in completing medical forms independently (extremely, quite a bit, somewhat, a little bit, not at all), and health literacy. Health literacy was measured using the Newest Vital Sign (NVS) and the short version of the Test of Functional Health Literacy in Adults (STOFHLA) [28, 29]. The Newest Vital Sign consists of 6 questions based on a nutrition fact label; each correct answer translates to a score of 1 and the total score ranges from 0–6 [28]. Missing response was considered 0. The STOFHLA assesses a patient’ ability to read passages using real materials from the health care setting [29]. It consists of 36 questions generating a total score of 0–36, which can be categorized into inadequate (0–16), marginal (17–22), and adequate functional health literacy (23–36) [29].
2.5. Statistical analyses
To characterize the study sample, descriptive statistics were used to evaluate the potential covariates, Health-ITUES total and subscale scores as well as dependent variable app use measured as proportion of activities completed. Correlations of Health-ITUES total and subscale scores with app use were assessed. We also assessed the bivariate relationship between each potential covariate and the dependent variable app use. Covariates with a p-value less than 0.20 were entered into the final multiple linear regression models to select the optimal cut-point for Health-ITUES total score [30]. To control the correlations among repeated measures, the generalized estimating equations (GEE) solution was used. We used a procedure adapted from Faraggi and Simon’s method to select an optimal cut-point for the Health-ITUES total to define the outcome app use [31]. The optimal cut-point was the one that minimized the p-value associated with a comparison of outcome (app use) between participants with values above and below the cut-point [32]. The selection of an optimal cut-point involved examination of different potential cut-points with an increment of 0.01 from minimal score to maximum score through assessing the relationship between the binary total score (above and equal to the cut-point vs. below the cut-point) and app use, controlling for potential covariates in a series of multiple linear regression models using GEE solutions with robust estimates for repeated outcomes. Stability of the cut-points was studied with 500 bootstrap replications. The recommended final optimal cut-point was the most frequent one across the 500 replications. Confidence intervals (CIs) of the optimal cut-point were based on the percentiles of the distribution. Participants from both arms were included in all analysis. All data analyses were conducted in SAS version 9.4 (SAS Institute, 2013) [33].
3.0. RESULTS
We included 158 participants who completed the 3-month follow-up visit. Table 1 shows the demographics and health literacy of the study participants. Mean age was 49.7 years (SD 10.3), 50% male, 71% African American/Black, 72% non-Hispanic, 45% had above high school education, 54% had <$10,000 annual income. Participants scored 1.5 on average on the NVS (SD 1.5, range 0–6.0) and 30.5 on the STOFHLA
Table 1:
Demographics of study participants
| Characteristics | N=158 | |
|---|---|---|
| Age, mean (SD) | 49.7 (10.3) | |
| Gender, N (%) | Male | 79 (50.0) |
| Female | 72 (45.6) | |
| Othera | 7 (4.4) | |
| Race, N (%) | African American/Black | 112 (70.9) |
| White | 11 (7.0) | |
| Unknown | 22 (13.9) | |
| Other | 13 (8.2) | |
| Ethnicity, N (%) | Non-Hispanic | 113 (71.5) |
| Hispanic | 45 (28.5) | |
| Education, N (%) | Below high school | 40 (25.3) |
| High school | 47 (29.8) | |
| Above high school | 71 (44.9) | |
| Annual income, N (%) | <$10,000 | 85 (53.8) |
| $10,000–$19,999 | 38 (24.1) | |
| ≥$20,000 | 16 (10.1) | |
| Don’t know | 19 (12.0) | |
| Relationship status, N (%) | In a relationship | 46 (29.1) |
| Not in a relationship | 110 (69.6) | |
| Unknown | 2 (1.3) | |
| Sexuality, N (%) | Homosexual | 37 (23.4) |
| Heterosexual | 92 (58.2) | |
| Bisexual | 20 (12.7) | |
| Other | 2 (1.3) | |
| Unknown | 7 (4.4) | |
| Children, N (%) | Yes | 87 (55.1) |
| No | 71 (44.9) | |
| Confidence in completion medical forms independently, N (%) | Extremely | 89 (56.3) |
| Quite a bit | 35 (22.2) | |
| Somewhat | 28 (17.7) | |
| A little bit | 4 (2.5) | |
| Not at all | 2 (1.3) | |
| Newest Vital Sign, mean (SD; range)b | 1.4 (1.4; 0–6.0) | |
| Test of Functional Health Literacy in Adults, mean (SD)c | 30.5 (6.8; 2.036.0) |
Included transgender male, transgender female, genderqueer, and unknown
Consists of 6 questions and the total score ranges from 0–6; a higher score indicates greater health literacy; 6 participants had missing data
Consists of 36 questions and the total score ranges 0–36; a higher score indicates greater functional health literacy
3.1. Health-ITUES and app use
Mean Health-ITUES total scores at 3 and 6 months were 4.39 (SD 0.75, N=158) and 4.43 (SD 0.75, N=139), respectively. Table 2 shows the Health-ITUES total and subscale scores. App use measured as proportion of activities completed by participants from baseline to the 3-month follow-up visit was 0.61 or 61% (SD 0.36, range 0–1.72; upper limit is above 1 or 100% because participants completed the activities more than required). App use from 3-month to the 6-month follow-up visits was 0.51 or 51% (SD 0.37, range 0–1.60). Table 3 shows the correlation between Health-ITUES scores and app use.
Table 2:
Health ITUES total and subscale scores and app use at the 3-month and 6-month follow-up visits
| 3-month follow-up, mean (SD) | 6-month follow-up, mean (SD) | Total, mean (SD) | |
|---|---|---|---|
| N=158 | N=139 | ||
| Health-ITUES | |||
| Impact | 4.49 (0.83) | 4.48 (0.78) | 4.49 (0.80) |
| Perceived usefulness | 4.46 (0.78) | 4.41 (0.76) | 4.44 (0.77) |
| Perceived ease of use | 4.49 (0.79) | 4.54 (0.73) | 4.51 (0.76) |
| User control | 4.11 (0.90) | 4.28 (0.86) | 4.19 (0.89) |
| Total score | 4.39 (0.75) | 4.43 (0.72) | 4.41 (0.73) |
| Dependent variable - App use | 0.61 (0.36) | 0.51 (0.37) | 0.56 (0.37) |
Abbreviation: Health-ITUES, Health Information Technology Usability Evaluation Scale
Table 3:
Correlation of Health-ITUES subscale and total scores with app use (N=316 observations from 158 participants)
| Health-ITUES | App use | ||||||
|---|---|---|---|---|---|---|---|
| Total score | Impact | Perceived usefulness | Perceived ease of use | User control | |||
| Health-ITUES | Total score | - | 0.92a | 0.95a | 0.91a | 0.86a | 0.20 (P<0.001) |
| Quality of work life | 0.92a | - | 0.91a | 0.78a | 0.66a | 0.21 (P<0.001) | |
| Perceived usefulness | 0.95a | 0.91a | - | 0.83a | 0.72a | 0.18 (P=0.002) | |
| Perceived ease of use | 0.91a | 0.78a | 0.83a | - | 0.71a | 0.20 (P<0.001) | |
| User control | 0.86a | 0.67a | 0.72a | 0.71a | - | 0.14 (P=0.02) | |
| App use | 0.20 (P<0.001) | 0.21 (P<0.001) | 0.18 (P=0.002) | 0.20 (P<0.001) | 0.14 (P=0.02 | - | |
P<0.0001
Abbreviation: Health-ITUES, Health Information Technology Usability Evaluation Scale
Female (vs. male) and higher STOFHLA scores were associated with greater app use (Table 4). Sexuality (other vs. homosexual) was associated with lower app use.
Table 4:
Results from simple linear regression assessing the bivariate relationships between each covariate and app use* (N=316 observations from 158 participants)
| Covariates | β | 95% Confidence Interval | P-value | |
|---|---|---|---|---|
| Age | 0.003 | −0.0006, 0.007 | 0.10 | |
| Gender | Male | Ref | - | - |
| Female | 0.10 | 0.02, 0.19 | 0.01 | |
| Othera | −0.16 | −0.36, 0.04 | 0.12 | |
| Race | African American/Black | Ref | - | - |
| White | 0.06 | −0.11. 0.22 | 0.51 | |
| Unknown | −0.007 | −0.13, 0.11 | 0.91 | |
| Other | −0.13 | −0.28, 0.02 | 0.09 | |
| Ethnicity | Non-Hispanic | Ref | - | - |
| Hispanic | −0.04 | −0.13, 0.05 | 0.41 | |
| Education | Below high school | −0.05 | −0.16, 0.06 | 0.36 |
| High school | Ref | - | - | |
| Above high school | −0.08 | −0.18, 0.01 | 0.09 | |
| Annual income | <$10,000 | Ref | - | - |
| $10,000–$19,999 | 0.06 | −0.04, 0.16 | 0.28 | |
| ≥$20,000 | 0.05 | −0.09, 0.19 | 0.48 | |
| Don’t know | 0.06 | −0.08, 0.19 | 0.41 | |
| Relationship status | In a relationship | Ref | - | - |
| Not in a relationship | −0.02 | −0.11, 0.07 | 0.70 | |
| Unknown | −0.42 | −0.78, −0.05 | 0.02 | |
| Sexuality | Homosexual | Ref | - | - |
| Heterosexual | 0.003 | −0.10, 0.10 | 0.95 | |
| Bisexual | −0.06 | −0.21, 0.08 | 0.37 | |
| Other | −0.29 | −0.50, −0.08 | 0.007 | |
| Unknown | −0.05 | −0.42, 0.32 | 0.78 | |
| Children | Yes | Ref | - | - |
| No | −0.003 | −0.09, 0.08 | 0.95 | |
| Confidence in completion medical forms independently | −0.011 | −0.05, 0.03 | 0.63 | |
| Newest Vital Sign | 0.028 | −0.004, 0.06 | 0.07 | |
| Test of Functional Health Literacy in Adults | 0.007 | 0.001, 0.01 | 0.02 |
Results from simple linear regression with dependent variable app use
Abbreviation: Health-ITUES, Health Information Technology Usability Evaluation Scale
3.2. Optimal cut-point selection
In the multiple linear regression models assessing the relationship between the binary Health-ITUES total score (above or equal to vs. below the cut-point) and the dependent variable app use, we controlled for age, gender, race, education, relationship status, sexuality, NVS, and STOFHLA based on results from bivariate analyses in Table 4. Table 5 presents the first 20 lowest p-values associated with the binary total score for the full original data. Participants who reported greater Health-ITUES total score completed more activities [β=0.18, 95% Confidence Interval (CI): 0.10–0.27]. The cut-point of 4.32 yielded the lowest p-value associated with the binary Health-ITUES total score and also produced the largest beta, which measured the difference in outcome between the dichotomized groups. Figure 2 depicts the distribution of the cut-points from the bootstrapping with 500 replicates. The mode of the optimal cut-point was also 4.32. Thus the final optimal cut-point was 4.32 with 95% CIs of 4.25–4.56 estimated from the 500 bootstrap replication.
Table 5:
Multiple linear regression assessing the relationship between the binary Health-ITUES total score and the dependent variable app use (N=316 observations from 158 participants)a
| Cut-point for creating the binary Health-ITUES total score | βb | 95% Confidence Interval | P-value |
|---|---|---|---|
| 4.32 | 0.18 | 0.11, 0.45 | 5.49 × 10−5 |
| 4.33 | 0.18 | 0.09, 0.42 | 7.87 × 10−5 |
| 4.31 | 0.18 | 0.05, 0.40 | 1.03 × 10−4 |
| 4.35 | 0.18 | 0.03, 0.36 | 1.46 × 10−4 |
| 4.34 | 0.18 | 0.02, 0.39 | 1.46 × 10−4 |
| 4.26 | 0.18 | 0.02, 0.58 | 1.69 × 10−4 |
| 4.28 | 0.18 | −0.01, 0.38 | 1.72 × 10−4 |
| 4.25 | 0.18 | −0.08, 0.56 | 1.85 × 10−4 |
| 4.24 | 0.18 | −0.08, 0.56 | 1.85 × 10−4 |
| 4.30 | 0.17 | −0.08, 0.56 | 1.92 × 10−4 |
| 4.29 | 0.17 | −0.07, 0.40 | 1.92 × 10−4 |
| 4.37 | 0.17 | −1.44, 0.29 | 1.98 × 10−4 |
| 4.36 | 0.17 | −1.44, 0.29 | 1.98 × 10−4 |
| 4.48 | 0.17 | −1.44, 0.29 | 2.12 × 10−4 |
| 4.27 | 0.18 | −0.09, 0.40 | 2.13 × 10−4 |
| 4.50 | 0.16 | −0.19, 0.85 | 2.67 × 10−4 |
| 4.49 | 0.16 | −0.17, 0.73 | 2.67 × 10−4 |
| 4.40 | 0.17 | −0.11, 0.41 | 2.80 × 10−4 |
| 4.39 | 0.17 | −0.19, 0.58 | 2.80 × 10−4 |
| 4.38 | 0.17 | −0.23, 0.59 | 2.87 × 10−4 |
Only results from the full original sample with the first 20 lowest p-values were presented; each multiple linear regression model with generalized estimating equations solution was adjusted for age, education, gender, race, relationship status, sexuality, newest vital sign, and short version of the Test of Functional Health Literacy in Adults based on bivariate relationships from Table 4.
β is the regression coefficient assessing the differences in the dependent variable (app use) between the participants with Health-ITUES total score above or equal to the cut-points and those with Health-ITUES total score below the cut-points.
Figure 2:
Distribution of optimal cut-points of the Health-ITUES total score associated with app use from the 500 bootstrapping replicates
4.0. DISCUSSION
The Health-ITUES is a validated instrument used in the evaluation of usability for health IT tools [19, 25]. The factorial validity, internal consistency, constructive validity, and predictive validity of the instrument have been established.[19, 22, 25] The Health-ITUES also has the benefit of allowing for item customization, thereby facilitating comparison across studies. In this study of adults with HIV, we identified an optimal cut-point of 4.32 on the Health-ITUES total score to identify if a health IT tool is usable.
Given the increasing number of mHealth apps and studies in the healthcare setting, it is important that usability is assessed in a systematic fashion with a pre-specified cut-point. The Health-ITUES was developed in 2010 and validated for use in mHealth studies in 2018 [19, 24]. It has been incorporated in several technology and mHealth studies in the assessment of usability although none provided a cut-point to define usability [34, 35]. Our study advanced the field by establishing a cut-point for defining usability using the Health-ITUES. A cut-point helps to determine whether a health IT tool is usable. Compared to a continuous Health-ITUES score, a binary Health-ITUES score created from the cut-point (above the cut-point versus below the cut-point) allows for easier interpretation.
It is important to note that this cut-point should be used concurrently with other usability evaluation methods such as performance-based testing with usability metrics, think-aloud techniques, heuristic evaluation, and cognitive walkthroughs [35–41]. In addition, the cut-point is based on the mean of the 20 items from four subscales. Therefore, attention should be paid to potential outlier scores/subscales as that could provide valuable information. There are also situations where users may select “agree” to all items and score below the cut-point. In this case, it is arguable that the users likely perceive the mHealth app as being usable. Therefore, decision on whether a mHealth app is usable should not wholly be based on meeting this cut-point on the Health-ITUES.
To the authors’ knowledge, among the available instruments that assess usability, only the SUS has a cut-point (a score of 68 or above on the SUS is considered usable or average) [42]. The SUS also has a percentile rank (i.e., an SUS score of 74 converts to a percentile rank of 70%) [42]. However, the methods for which the optimal cut-point was determined is unclear. There is also little guidance on how SUS items can be modified. Further studies should assess usability with multiple instruments to allow for comparison [17].
Our study has a few strengths. Instead of using an arbitrary cut-point such as percentile or mean, we identified the optimal cut-point as the one that minimized the p-value associated with a comparison of the outcome (app use) between participants with values above and below the cut-point. In addition, we used bootstrapping with 500 replications to assess the stability of the cut-point. Our study has several limitations. Our population was limited to adults living with HIV who own a smartphone device. We validated the Health-ITUES and established its cut-point in the same population. In addition, we acknowledge that usability varies by disease severity, individual characteristics, and digital/health literary. Therefore, further studies are needed to increase the generalizability of these findings in other study population. Due to the relatively small sample, we did not split the data into training and testing datasets for cross-validation. Instead, we used the bootstrapping method to assess the stability of results. Future work should consider validating the cut-point in a larger sample of adults.
In conclusion, our study serves as an important first step in establishing an optimal cut-point for defining usability using the Health-ITUES, specifically in adults with HIV. This is important given the proliferation of health IT tools and the dearth of rigorous usability instruments available for evaluating these tools. Future usability evaluations can be conducted with more rigor given the availability of a discrete cut-point for establishing usability.
Supplementary Material
Summary table.
What was already known
Health Information Technology Usability Evaluation Scale (Health-ITUES) is a validated and reliable instrument to evaluate usability of information technology (IT) tools
No prior studies have established a cut-point for usability for the Health-ITUES
What this study added to our knowledge
Our study serves as an important first step in establishing an optimal cut-point for defining usability using the Health-ITUES
This is important given the proliferation of health IT tools and the dearth of rigorous usability instruments available for evaluating these tools
Highlights.
Evaluation of mobile health (mHealth) apps using established usability scales alongside other usability evaluation methods allow for more rigorous testing
Health Information Technology Usability Evaluation Scale (Health-ITUES) is a validated and reliable instrument to evaluate usability of information technology (IT) tools
In this study of adults with HIV, we identified an optimal cut-point of 4.32 on the Health-ITUES total score to identify if a mHealth app is usable
This is important given the proliferation of health IT tools and the dearth of rigorous usability instruments
Acknowledgments
Funding: The work was supported by the National Cancer Institute at the National Institute of Health (NIH; R00CA237744 to KPL), the National Institute of Nursing Research at NIH (K24NR018621 to RS), Agency for Healthcare Research and Quality (AHRQ; R01HS025071 to RS), and the Wilmot Research Fellowship Award (to KPL). The content is solely the responsibility of the authors and does not necessarily represent the official views of the AHRQ and NIH.
Footnotes
Conflicts of interest: Dr. Loh has served as a consultant to Pfizer and Seattle Genetics, and has received honoraria from Pfizer. All other authors have no relevant conflicts of interest to report.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- [1].Mobile Fact Sheet. Pew Research Center: Internet, Science & Tech April 7, 2021. https://www.pewresearch.org/internet/fact-sheet/mobile/.
- [2].Statista Research Department, & 26, M. Healthcare apps available Google Play 2021. Statista. May 26, 2021. https://www.statista.com/statistics/779919/health-apps-available-google-play-worldwide/..
- [3].Franklin R (2019, March 19). 11 surprising mobile health statistics. Mobius MD. https://mobius.md/2019/03/20/11-mobile-health-statistics/#:%7E:text=2.,available%20in%20major%20app%20stores. [Google Scholar]
- [4].Schnall R, Porras T, Cho H, Jia H, Siegel K, Mohr DC, Bakken S, Olender S, Efficacy, Use, and Usability of the VIP-HANA App for Symptom Self-management in PLWH with HANA Conditions, AIDS Behav, 25 (2021) 1699–1710. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [5].Schnall R, Kuhns L, Pearson C, Bruce J, Batey DS, Radix A, Belkind U, Hidalgo MA, Hirshfield S, Ganzhorn S, Garofalo R, Preliminary Results from a Pragmatic Clinical Trial of MyPEEPS Mobile to Improve HIV Prevention Behaviors in Young Men, Stud Health Technol Inform, 270 (2020) 1365–1366. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [6].Loh KP, Sanapala C, Di Giovanni G, Klepin HD, Janelsins M, Schnall R, Culakova E, Vertino P, Susiarjo M, Mendler JH, Liesveld JL, Lin PJ, Dunne RF, Kleckner I, Mustian K, Mohile SG, Developing and adapting a mobile health exercise intervention for older patients with myeloid neoplasms: A qualitative study, J Geriatr Oncol, 12 (2021) 909–914. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [7].Loh KP, Ramsdale E, Culakova E, Mendler JH, Liesveld JL, O'Dwyer KM, McHugh C, Gilles M, Lloyd T, Goodman M, Klepin HD, Mustian KM, Schnall R, Mohile SG, Novel mHealth App to Deliver Geriatric Assessment-Driven Interventions for Older Adults With Cancer: Pilot Feasibility and Usability Study, JMIR Cancer, 4 (2018) e10296. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [8].Akdur G, Aydin MN, Akdur G, Adoption of Mobile Health Apps in Dietetic Practice: Case Study of Diyetkolik, JMIR Mhealth Uhealth, 8 (2020) e16911. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [9].Wang C, Qi H, Influencing Factors of Acceptance and Use Behavior of Mobile Health Application Users: Systematic Review, Healthcare (Basel), 9 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [10].Scolari C, The sense of the interface: Applying semiotics to HCI research, 2009 (2009) 1–27. [Google Scholar]
- [11].Crane D, Garnett C, Brown J, West R, Michie S, Factors Influencing Usability of a Smartphone App to Reduce Excessive Alcohol Consumption: Think Aloud and Interview Studies, Front Public Health, 5 (2017) 39. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [12].Furlow B mHealth apps may make chronic disease management easier. Clinical Advisor. November 2, 2012. https://www.clinicaladvisor.com/home/features/mhealth-apps-may-make-chronic-disease-management-easier/.
- [13].Davis R, Gardner J, Schnall R, A Review of Usability Evaluation Methods and Their Use for Testing eHealth HIV Interventions, Curr HIV/AIDS Rep, 17 (2020) 203–218. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [14].Jake-Schoffman DE, Silfee VJ, Waring ME, Boudreaux ED, Sadasivam RS, Mullen SP, Carey JL, Hayes RB, Ding EY, Bennett GG, Pagoto SL, Methods for Evaluating the Content, Usability, and Efficacy of Commercial Mobile Health Apps, JMIR Mhealth Uhealth, 5 (2017) e190. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [15].Thyvalikakath TP, Monaco V, Thambuganipalle H, Schleyer T, Comparative study of heuristic evaluation and usability testing methods, Stud Health Technol Inform, 143 (2009) 322–327. [PMC free article] [PubMed] [Google Scholar]
- [16].Bailey RW, Wolfson CA, Nall J, Koyani S, Performance-Based Usability Testing: Metrics That Have the Greatest Impact for Improving a System's Usability, HCI, 2009. [Google Scholar]
- [17].Brooke J SUS-A quick and dirty usability scale. Usability Eval Ind. 1996;189:4–7. [Google Scholar]
- [18].Lewis JR, IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use, International Journal of Human–Computer Interaction, 7 (1995) 57–78. [Google Scholar]
- [19].Schnall R, Cho H, Liu J, Health Information Technology Usability Evaluation Scale (Health-ITUES) for Usability Assessment of Mobile Health Technology: Validation Study, JMIR Mhealth Uhealth, 6 (2018) e4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [20].Zhou L, Bao J, Setiawan IMA, Saptono A, Parmanto B, The mHealth App Usability Questionnaire (MAUQ): Development and Validation Study, JMIR Mhealth Uhealth, 7 (2019) e11500. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [21].Sauro J, Lewis JR, Chapter 2 - Quantifying user research, in: Sauro J, Lewis JR (Eds.) Quantifying the User Experience (Second Edition), Morgan Kaufmann, Boston, 2016, pp. 9–18. [Google Scholar]
- [22].Yen PY, Wantland D, Bakken S, Development of a Customizable Health IT Usability Evaluation Scale, AMIA Annu Symp Proc, 2010 (2010) 917–921. [PMC free article] [PubMed] [Google Scholar]
- [23].Davis FD, Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS Quarterly, 13 (1989) 319–340. [Google Scholar]
- [24].Yen P-Y, Health information technology usability evaluation: methods, models, and measures, 2010. [Google Scholar]
- [25].Yen PY, Sousa KH, Bakken S, Examining construct and predictive validity of the Health-IT Usability Evaluation Scale: confirmatory factor analysis and structural equation modeling results, J Am Med Inform Assoc, 21 (2014) e241–248. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [26].Lee J, Schnall R. Validity and Reliability of the Korean Version of the Health-ITUES Scale for Usability Assessments of Mobile Health Technology. [In Review]. [Google Scholar]
- [27].Flynn G, Jia H, Reynolds NR, Mohr DC, Schnall R, Protocol of the randomized control trial: the WiseApp trial for improving health outcomes in PLWH (WiseApp), BMC Public Health, 20 (2020) 1775. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [28].Weiss BD, Mays MZ, Martz W, Castro KM, DeWalt DA, Pignone MP, Mockbee J, Hale FA, Quick assessment of literacy in primary care: the newest vital sign, Ann Fam Med, 3 (2005) 514–522. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [29].Parker RM, Baker DW, Williams MV, Nurss JR, The test of functional health literacy in adults: a new instrument for measuring patients' literacy skills, J Gen Intern Med, 10 (1995) 537–541. [DOI] [PubMed] [Google Scholar]
- [30].Model-Building Strategies and Methods for Logistic Regression, Applied Logistic Regression 2000, pp. 91–142. [Google Scholar]
- [31].Faraggi D, Simon R, A simulation study of cross-validation for selecting an optimal cutpoint in univariate survival analysis, Stat Med, 15 (1996) 2203–2213. [DOI] [PubMed] [Google Scholar]
- [32].Faraggi D, Simon R, A SIMULATION STUDY OF CROSS-VALIDATION FOR SELECTING AN OPTIMAL CUTPOINT IN UNIVARIATE SURVIVAL ANALYSIS, Statistics in Medicine, 15 (1996) 2203–2213. [DOI] [PubMed] [Google Scholar]
- [33].SAS. Statistical Analysis Software. Users’ Guide Statistics Version 9.4. SAS Institute Inc., Cary. 2013. [Google Scholar]
- [34].Kuhns LM, Hereth J, Garofalo R, Hidalgo M, Johnson AK, Schnall R, Reisner SL, Belzer M, Mimiaga MJ, A Uniquely Targeted, Mobile App-Based HIV Prevention Intervention for Young Transgender Women: Adaptation and Usability Study, J Med Internet Res, 23 (2021) e21839. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [35].Stonbraker S, Cho H, Hermosi G, Pichon A, Schnall R, Usability Testing of a mHealth App to Support Self-Management of HIV-Associated Non-AIDS Related Symptoms, Stud Health Technol Inform, 250 (2018) 106–110. [PMC free article] [PubMed] [Google Scholar]
- [36].Salz T, Schnall RB, McCabe MS, Oeffinger KC, Corcoran S, Vickers AJ, Salner AL, Dornelas E, Raghunathan NJ, Fortier E, McKiernan J, Finitsis DJ, Chimonas S, Baxi S, Incorporating Multiple Perspectives Into the Development of an Electronic Survivorship Platform for Head and Neck Cancer, JCO Clin Cancer Inform, 2 (2018) 1–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [37].Schnall R, Rojas M, Bakken S, Brown W, Carballo-Dieguez A, Carry M, Gelaude D, Mosley JP, Travers J, A user-centered model for designing consumer mobile health (mHealth) applications (apps), J Biomed Inform, 60 (2016) 243–251. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [38].Cho H, Powell D, Pichon A, Kuhns LM, Garofalo R, Schnall R, Eye-tracking retrospective think-aloud as a novel approach for a usability evaluation, Int J Med Inform, 129 (2019) 366–373. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [39].Schnall R, John RM, Carballo-Dieguez A, Do High-Risk Young Adults Use the HIV Self-Test Appropriately? Observations from a Think-Aloud Study, AIDS Behav, 20 (2016) 939–948. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [40].Cho H, Yen PY, Dowding D, Merrill JA, Schnall R, A multi-level usability evaluation of mobile health applications: A case study, J Biomed Inform, 86 (2018) 79–89. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [41].Beauchemin M, Gradilla M, Baik D, Cho H, Schnall R, A Multi-step Usability Evaluation of a Self-Management App to Support Medication Adherence in Persons Living with HIV, Int J Med Inform, 122 (2019) 37–44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [42].Sauro J Measuring Usability with the System Usability Scale (SUS). February 3, 2011. Available at: https://measuringu.com/sus/ Accessed on July 27, 2021. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.