Skip to main content
PMC home page
JMIR Human Factors logoLink to JMIR Human Factors
. 2021 Mar 17;8(1):e24846. doi: 10.2196/24846

Effects of User Characteristics on the Usability of a Home-Connected Medical Device (Smart Angel) for Ambulatory Monitoring: Usability Study

Noémie Chaniaud 1,✉,#, Olga Megalakaki 1, Sophie Capo 1, Emilie Loup-Escande 1
Editor: Andre Kushniruk
Reviewed by: Daphne Michelet, Dillon Chrimes
PMCID: PMC8080268  PMID: 33729161

Abstract

Background

The Smart Angel home medical device allows ambulatory surgery patients to monitor their own health by taking their blood pressure and oxygen levels and answering a health questionnaire from home. Currently, this device is a prototype in the design phase, and no usability evaluation has been performed. This preventive device must be usable by patients with different profiles; however, it is important to select patients carefully to ensure their safety when using the device. As such, it would be interesting to know how to select or exclude patients. However, the links between user characteristics and the usability of this home medical device remain unclear.

Objective

This study aims to better understand the links between certain characteristics of potential patients (ie, age, education, technophilia, and health literacy) and the usability (ie, effectiveness, efficiency, and satisfaction) of Smart Angel, as defined by the ISO 9241-11.

Methods

We conducted an experimental study involving 36 participants investigating the effects of 4 patient characteristics (ie, age, education, technophilia, and health literacy) on usability, measured in terms of effectiveness, efficiency, and satisfaction. A mixed methods approach (subjective vs objective) using a variety of standard instruments was adopted (direct observation, video analysis, and questionnaires). First, to help participants project themselves into the real use of the Smart Angel device, they watched a scenario in a video. Second, the participants completed a set of questionnaires to show the extent of their health literacy level (Newest Vital Sign [NVS] and the Health Literacy Survey [HLS]) and then operated Smart Angel devices. Efficiency (ie, handling time) and effectiveness (ie, number of handling errors) measures were collected by video analysis. Satisfaction measures were collected by a questionnaire (System Usability Scale [SUS]). The qualitative observational data were coded using inductive analysis by 2 independent researchers specialized in cognitive psychology and cognitive ergonomics.

Results

The results show a moderate and positive correlation between age and effectiveness (r=0.359; P=.03) and efficiency (r=0.357; P=.03). There is strong correlation between health literacy scored by the NVS and effectiveness (r=0.417; P=.01), efficiency (r=-0.38; P=.02), and satisfaction (r=0.45; P=.006). However, there is a weak correlation between technophilia and usability and no relationship between education level and usability.

Conclusions

Our results show that literacy level and age are 2 important factors to consider when selecting future users of the Smart Angel device to ensure patient safety. This study also serves as an example promoting mixed methodologies in assessments of medical device usability that cannot be performed under real-world conditions.

Keywords: user characteristics, health literacy, home medical devices, usability study, remote health, home health

Introduction

Background

Outpatient surgery has been on the rise in recent years. Performed operations are increasingly complex and dangerous for patients who have to manage their convalescence at home. The Smart Angel device is a home-connected medical device specifically designed to prevent postsurgical complications related to outpatient surgery. The purpose of this device is to facilitate the patient's return home by maintaining a link with the hospital. Upon returning home after an operation, the patient is required to use the device to send all their vitals 3 times a day for 1 week before returning the equipment to the hospital center. This postoperative follow-up may also enable patients to manage their convalescence better by avoiding all-too-frequent returns to emergency services or outpatient consultations [1].

Currently, this system is in an early design stage. Like any medical device, this tool must follow safety and quality standards [2] and usability standards [3] to meet the requirements of European Conformity (CE marking) for marketing. However, even today, the deployment of these connected medical devices is still hindered by their complexity of use, directly implying a lack of usability [4-6], thus impacting patient safety. With this in mind, Kortum and Peres commented, “A lack of usability may cost lives” [7].

Usability

Usability is defined by the ISO 9241-11 [3] as “the degree to which a product can be used, by identified users, to achieve defined goals in an effective, efficient, and satisfactory manner, within a specified context of use.” This concept, which is still discussed by the scientific community, has 3 distinct dimensions: (1) effectiveness: the accuracy and completeness with which users achieve certain objectives; (2) efficiency: the relationship between accuracy and the resources used to attain it; and (3) satisfaction: user comfort and a positive evaluation of user interaction. Defined by these 3 dimensions, usability is linked to its context of use, characterized by 4 components: the task, the environment, the resources, and the users [3].

Despite the use of methodologies that involve the user in the design process [8-10], usability problems persist. There are 2 arguments in the literature that may explain this finding: (1) the lack of a standardized framework and method in usability studies [11-15], and (2) a lack of knowledge of the impact of the use context [16,17] on usability, in particular, user characteristics.

User Characteristics

Age, Level of Education, Technophilia, and Health Literacy

Several researchers have recently investigated the relationship between user characteristics and the usability of connected devices in health care [11,18-22]. In particular, 4 user characteristics have been studied in the scientific literature: age [11,20,22-26], level of education [11,19,20], technophilia (ie, experience in information technology and previous experience with medical devices [11,23,27]), and health literacy [20,24,28,29]. In most studies, authors tend to agree on these interrelationships when investigating different devices. We detail these studies below.

Age

Many authors have examined the influence of age on the usability of connected devices in health care. Most of these authors concur on the influence of age on usability. For example, Georgsson and Staggers [11] investigated the usability of a diabetes management app running on a smartphone using the metrics of ISO 9241-11 (effectiveness, efficiency, and satisfaction). The authors found that the younger age group (30-49 years old) made fewer errors (ie, was more effective), was faster (ie, more efficient), and more satisfied (System Usability Survey [SUS] score of 88.33 vs 77.14) than the older group (50-69 years old). Sparkes et al [23] examined the usability of remote cardiac testing and found that the age of the participants impacted their ability to install the equipment. Younger subjects appeared to be more comfortable than older subjects. Jones and Caird [25] examined the use of a blood glucose meter and found that younger subjects had fewer difficulties and made fewer errors (ie, were more effective) than older subjects. Mykityshyn et al [26] also examined the use of a glucometer and found that young subjects were faster (ie, more efficient) than older subjects, regardless of the instruction format provided (written and drawn vs video). Van der Vaart et al [20] evaluated the usability of an application for monitoring the symptoms of 32 narcoleptics and found that usability (measured in terms of the number of tasks completed and problems encountered) was moderately and positively correlated with age and eHealth literacy level.

However, Liang et al [19] found no relationship between age and satisfaction as measured by the SUS score in their study on the evaluation of 7 health devices used by the general public (eg, connected watches), conducted with a sample of 388 participants. Similarly, Jensen et al [18] found no relationship between usability and the age of participants with respect to access and use of online health information. The authors explain that this result is probably due to the contrast in health literacy levels that would have taken precedence over the other variables.

Level of Education

The level of education is also a variable found in many usability assessments. However, to our knowledge, no studies have proven this link. Georgsson and Staggers [11], Liang et al [19], and Van der Vaart et al [20] have all found a lack of association between participants' level of education and usability (ie, effectiveness, efficiency, and satisfaction).

Technophilia

Differing results have been reported regarding the influence of technophilia—experience with information technologies (IT) and previous experience of medical devices—on usability. Georgsson and Staggers [11] found that those with more technology experience (what the authors call “IT/computer experience”) made fewer errors (ie, were more effective), were faster (ie, more efficient), and were more satisfied with the diabetes management application (+5 points for the SUS score). Conversely, Harte et al [27] conducted regression analyses between technology experience and SUS score and found no significant effect when evaluating a smartphone health app. Finally, Sparkes et al [23] showed that familiarity with the technologies seemed to have an influence on the correct installation of their device.

Health Literacy

Definition and Assessments

Health literacy is a user characteristic that can be expected to influence medical device usability [18,20,28,30]. Due to its multidimensionality, however, this characteristic is complex to define and difficult to assess. Sørensen et al [31] describe it as “an individual's knowledge, skills, motivation, and ability to identify, understand, evaluate, and use health information in decision-making in health care, disease prevention, and health promotion to maintain or improve lifelong quality of life.” However, this notion is often mentioned as a determinant to be considered in therapeutic education [32], prevention [33], therapeutic adherence, access to health information [18], and even recovery rate [32,34]. However, to our knowledge, no study has assessed the level of health literacy among the French population at the national level.

In terms of evaluation, health literacy is particularly difficult to measure for at least two reasons. The first reason concerns its multidimensional specificity [31]. The second reason is that health literacy is not related to socioeconomic criteria as might be intuitively assumed [35].

Currently, there are 2 main methods of measuring health literacy [36]: (1) questionnaire methods, by which an individual's abilities are assessed, and (2) self-reported methods, by which an individual's behaviors towards a health professional are directly observed. Currently, few tools exist in the French language compared to the 51 English-language instruments identified by Haun et al [37]. The most frequently used and cited instruments are part of questionnaire-based methods; they are the Test of Functional Health Literacy in Adults (TOFHLA) [38], the Rapid Estimate of Adult Literacy in Medicine (REALM) [39], the European Health Literacy Survey (HLS-EU) [31], and the Newest Vital Sign (NVS) [40]. However, these instruments have several limitations. Among these instruments, the REALM is more like a reading test than a comprehension test since participants are asked to read medical terms. The short version of the TOFHLA (ie, the Short Test of Functional Health Literacy in Adults [S-TOFHLA]), which assesses respondents' level of comprehension, seems more adapted to Swiss culture than to French culture [41] (indeed, direct reference is made to the Swiss health insurance system and the transmission of certain documents that do not apply to the French social security model). In addition, the validity of S-TOFHLA is currently the subject of some controversy due to inconsistencies in the interpretation of its component items [42]. Another instrument proposed in the literature, the NVS [40], shows a strong correlation (Cronbach α>.76) with the measurement of S-TOFHLA [43]. It also assesses some of the respondents' cognitive skills (reading, writing, comprehension, numeracy). Finally, the HLS-EU is based on the multidimensional literacy model of Sørensen et al [31]. This tool has identified important gaps in 8 European countries, as approximately 1 in 2 people reportedly have a problematic or inadequate level of health literacy [44].

Health Literacy and Usability

In the context of health technologies such as connected medical devices, which are increasingly becoming part of patient life, studies on the correlation between health literacy and usability are still rare or exploratory. Monkman and Kushniruk [21] propose an assessment of usability by considering health literacy through the design and validation of heuristic criteria. To do so, the authors adapted a set of existing guidelines for designing health-specific websites to make the content more understandable to users with a reliable level of health literacy. Using an electronic personal health record system, Czaja et al [28] were able to show that populations with low literacy levels had more difficulty using these tools. Kim and Xie [29] conducted a systematic review of articles examining the impact of low health literacy on the use of eHealth devices. Based on 74 studies, the authors conclude that the major barrier to accessing and using online health information for individuals with low literacy is strongly related to website usability. Jensen et al [18] found that participants with low levels of health literacy (as measured by REALM) used health technologies less. Those with low levels of health numeracy (as measured by TOFHLA) would have limited access to these technologies. This latter finding is consistent with those of Kaufman et al [24], who also concluded that low numeracy could be a barrier to using a telemedicine system. Chaniaud et al [30] showed that it is necessary to obtain a minimum level of prior health knowledge to use home medical devices. Finally, to our knowledge, no experimental studies have empirically characterized links between health literacy and usability in terms of efficiency, effectiveness, and satisfaction.

Study Objective

We have seen that the complexity of using medical devices resides essentially in usability problems [29], all the more so as they must be usable by patients with diverse profiles. In this sense, consideration of user characteristics, including age, education, technophilia, and health literacy, are important factors to consider in the design of a connected medical device such as Smart Angel for a patient's home. However, to our knowledge, no study involving all 4 of these characteristics has been conducted. Moreover, the relation between these characteristics and usability remains unexplored in the literature. Thus, the aim of this paper is to better understand the relationships between the 4 user characteristics of age, educational level, technophilia, and health literacy, and the usability (measured by effectiveness, efficiency, and satisfaction) of a connected medical device intended for a patient's home.

To do this, we formulated 4 hypotheses: (H1) older users will be less effective, efficient, and satisfied with the Smart Angel connected medical device than younger users [11,20,25,26]; (H2) users with a low level of technophilia (IT and medical device experience) will be less effective, efficient, and satisfied with the Smart Angel connected medical device than those with a high level of technophilia [11,23,27]; (H3) the level of education will not affect the effectiveness, efficiency, and satisfaction with the Smart Angel connected medical device [11,19,20]; and (H4) users with low levels of health literacy (as measured by NVS and HLS-EU scores) will be less effective, efficient, and satisfied with the Smart Angel connected medical device than those with high levels of health literacy [18,24].

Methods

Participants

We enrolled 36 participants for this study: 17 (47%) females and 19 (53%) males aged 20-64 (mean 40.75, SD 14.45) years. The inclusion criteria were that participants had to (1) have a 4G connection at home, (2) be under 70 years of age, (3) be eligible for outpatient surgery, and (4) not be at home alone. All participants were native French speakers and signed a consent form after being informed of the study's progress. The study was in line with the ethical recommendations of the Declaration of Helsinki. The participants were recruited on a voluntary basis, and no compensation was offered. Handover of the Smart Angel device took place at the participant's home or workplace.

Materials and Measurements

The materials for this study included (1) the Smart Angel device, (2) personas and their scenarios, and (3) questionnaires (ie, 2 questionnaires assessing the level of health literacy, namely, the NVS and the HLS-EU; a questionnaire relating to sociodemographic data; and a questionnaire assessing satisfaction, namely, the SUS).

The Smart Angel Device

The Smart Angel device is designed by Evolucare Technologies. It consists of a Samsung 9-inch tablet with the Smart Angel application and 2 connected devices, a wrist blood pressure monitor (iHealth BP7) for blood pressure measurement and an oximeter (iHealth Oximeter PO3) for oxygen saturation and pulse measurement, which are available for the general public with European certification (Figure 1). To use the Smart Angel device, it is necessary to access the Smart Angel application and perform a digital medical “appointment” from a tablet application.

Figure 1.

Figure 1

Open in a new tab

The Smart Angel components. Upper left: a pulse oximeter (iHealth Oximeter PO3); lower left: a wrist blood pressure monitor (iHealth BP7); right: a tablet with the Smart Angel application.

The patient is given step-by-step instructions for connecting to and taking measurements with the blood pressure monitor and the pulse oximeter. The procedures for using the blood pressure monitor and pulse oximeter were built into the application; they include text and images for each step of the operation. For the 2 connected devices, the participant must first have a correct body position to then connect the equipment, install it correctly on themselves, start the measurement, and then remove and switch off the equipment. A schematic representation of this procedure is shown in Figure 2.

Figure 2.

Figure 2

Open in a new tab

Schematic representation of the main steps in the use of the Smart Angel device.

Once the blood pressure or oxygenation measurement has been taken, the patient's health data are displayed on a colored gauge (from green to orange) according to the level of severity of the constant collected (Figure 3). Then, the user is presented with a questionnaire with various items related to general health, pain, sleep, and nausea. These items are presented either in simple-choice question format (eg, “How are you feeling today? Good, not good, not good at all”) or on a Likert scale (eg, “Rate your pain on a scale of 1 to 10”).

Figure 3.

Figure 3

Open in a new tab

Screenshot of the Smart Angel application (Evolucare Technologies). Left: a form presenting an overview of the subjective state of health; right: the procedure for using the monitor.

Personas and Scenarios

We constructed 5 personas and their scenarios based on statistical surveys of outpatient surgical procedure types in France [45] and observations made in the field [46]. Generally used in the design phase, the personas method draws on the theory of mind and the theory of stereotypes and can provoke certain emotional states [47]. The personas scenarios were presented to the participants as audiovisual cartoons. All scenarios were constructed in the same way. Only the type of operation and the cause of the operation changed, according to each persona. An example of a persona is presented in Multimedia Appendix 1.

Questionnaires

Measuring Health Literacy: Objective (NVS) and Subjective (HLS-EU) Assessments

Given the limited options of French-translated and validated health literacy questionnaires, we chose to use 2 health literacy questionnaires for a holistic view of this multidimensional skill: the NVS and the 16-item Health Literacy Survey (HLS-EU-Q16):

The French-translated [48] NVS [40] is a validated test assessing a patient’s ability to comprehend reading material and manipulate numbers (numeracy). Consequently, the NVS provides an objective assessment of health literacy level. Participants were asked to use an ice cream nutrition label to answer 6 questions (eg, “If I am allergic to peanuts, can I eat this ice cream?” Answer: “No, because the ice cream contains traces of peanut oil”). The total sum of the items (0-6 points) classified respondents into 3 categories: 0-1 point = inadequate health literacy; 2-3 points = problematic health literacy; 4-6 points = sufficient health literacy. The interitem reliability of the NVS in this study was good (Cronbach α=.883) [49].

The French-translated [50] HLS-EU-Q16 [31] is the short version of the HLS questionnaire. This version is composed of 16 items, 13 of which assess the 4 types of health literacy skills: the ability to access, understand, evaluate, and apply health information. Respondents were asked to rate their own ability to access information (eg, “Please rate, on a scale of very easy to very difficult, how easy is it for you to understand your doctor's or pharmacist's instructions on how to take your medication?”). Consequently, the HLS-EU-Q16 provides a subjective assessment of health literacy level. Answers are provided in 4 categories, on a 4-point Likert scale ranging from ”very easy“ to ”very difficult.“ To calculate the total score, the answers ”easy“ and ”very easy“ were assigned 1 point per item, and the answers ”difficult“ and ”very difficult“ were assigned 0 points per item. The total sum of the items (0-16 points) classified respondents into 3 categories: 0-8 points = inadequate health literacy; 9-12 points = problematic health literacy; 13-16 points = sufficient health literacy. The interitem reliability of the HSL-EU in this study was good (Cronbach α=.803) [49].

Sociodemographic Measurements (Age, Education Level, Technophilia, Etc)

This questionnaire includes the following personal details: age, gender, educational level, residential area, technophilia, and hospital experience. IT experience was measured by 2 items, adapted from Agarwal and Prasad [51], related to the participant’s use of and willingness to explore IT innovations (eg, “Which of these technologies do you use and how often?”). On a 5-point Likert scale, the possible answers ranged from “never” to “very often.”

Measuring Usability (ISO 9241-11:2018)

Measuring Effectiveness

Effectiveness was measured by counting the number of manipulation errors, such as not putting the blood pressure cuff in the correct position. With respect to the use of the monitor, 5 categories of errors were identified: the participant (1) did not position the monitor correctly, (2) incorrectly directed the monitor toward the palm of the hand, (3) did not position the forearm correctly, (4) moved during the measurement, or (5) did not connect the monitor's Bluetooth to the tablet. Regarding the use of the pulse oximeter, 4 categories of error were identified: the participant (1) did not position the oximeter the right way, (2) did not insert the finger as far as the sensor, (3) removed the oximeter too early during the measurement, or (4) did not connect the Bluetooth from the oximeter to the tablet. With the tablet, 1 type of error was observed: the participant did not enter the appointment in the application. A scoring grid was used to identify these manipulation errors. When the participant made several attempts, we recorded the cumulative number of errors.

Measuring Efficiency

Measuring efficiency was based on the manipulation duration times of the various device tools for 3 measurements: blood pressure monitor manipulation, pulse oximeter manipulation, and total manipulation of the device, including the complete appointment. These times were measured from the time participants first touched the device (monitor, pulse oximeter, or tablet) to the time they turned it off after taking the measurement.

Measuring Satisfaction

Satisfaction was measured using the SUS. This ”quick and dirty“ questionnaire [52] consists of 10 items with 5 response options on a Likert scale (ranging from ”strongly disagree“ to ”strongly agree“), which allows for a subjective assessment of usability [53]. We used an adapted and validated version [54], in which we replaced the term ”system“ with the term ”medical device.“ Scores were calculated according to the recommendations of Brooke [52] and ranged from 0 to 100. Lower scores indicate low usability.

Procedure

The average duration of this experiment was 45 minutes. The selected participants did not come out of ambulatory surgery. Participants were first invited to choose among 5 proposed personas to allow them to project themselves into the needs of future users of the Smart Angel device [55]. The persona chosen had to be consistent with at least the participant’s age, profession, and previous surgery. Then, the researcher demonstrated the use of the Smart Angel device to the participant for about 3 minutes, sharing information about the correct manipulation of the device (eg, ”The monitor should always be at heart level“). Participants were asked to complete 3 questionnaires: the sociodemographic data questionnaire, the HLS-EU-Q16, and the NVS. Then they were asked to operate the Smart Angel device by taking a blood pressure measurement followed by an oxygen saturation measurement, and finally, by completing the general health questionnaire. There was no time limit for this. The participants were filmed during the process. The researcher could only intervene in the event of a technical problem (eg, battery problem). Finally, after the experiment, the participant had to respond to the SUS.

Data Analysis

The videos were analyzed using BORIS (Behavioral Observation Research Interactive Software) [56], which collected data on effectiveness and efficiency. Results were analyzed using SPSS software (version 22; IBM Corp). Each user characteristic was systematically compared to usability components, including effectiveness, efficiency, and satisfaction. For the health literacy measurement, we first analyzed the HLS-EU-Q16 result and then the NVS result. Bivariate correlations, ANOVAs, and Student t tests were performed when the sample met the homoscedasticity criteria, while nonparametric tests (Kruskal-Wallis and Mann-Whitney) were performed when the sample did not meet these criteria.

Interjudge Reliability: Objective Measures of Effectiveness and Efficiency

We used intraclass correlation (ICC) to verify interjudge reliability for quantitative data [57]. A 33% double coding of the collected video data was performed. The mean ICC measurement for total manipulation time (efficiency) was 0.978, with a 95% confidence interval of 0.918 to 0.994 (F11,11=45.436; P<.001). The mean ICC measurement (efficiency) for manipulating the monitor was 0.988, with a 95% confidence interval of 0.954 to 0.997 (F11,11=81.635; P<.001). The mean ICC measurement (efficiency) for manipulating the pulse oximeter was 0.956, with a 95% confidence interval of 0.838 to 0.988 (F11,11=22.955; P<.001). The mean ICC measurement (efficiency) for manipulating the tablet was 0.906, with a 95% confidence interval 0.652 to 0.975 (F11,11=10.688; P<.001). The mean measure of the number of manipulation errors (effectiveness) was 0.952, with a 95% confidence interval of 0.842 and 0.985 (F11,11=20.789; P<.001).

Results

Effects of User Characteristics on Usability

The correlations between user characteristics and usability components (ie, effectiveness = number of manipulation errors; efficiency = manipulation time; satisfaction = SUS score) were systematically analyzed (Table 1).

Table 1.

Descriptive analyses of user characteristics, user experiences in health, medical devices, and technology (n=36).

Characteristics Value Average effectiveness, number of errors (SD) Average efficiency, manipulation time in seconds (SD) Average satisfaction, SUS score (SD)
Age in years, mean (SD)

 40.75 (14.45) N/Aa N/A N/A
Gender, n (%)

Male 19 (52.8) 1.21 (1.27) 362.09 (144.16) 87.24 (11.18)

Female 17 (47.2) 2.06 (1.25) 373.91 (126.3) 81.03 (11.73)
Education, n (%)

Secondary education

 5 (13.9)

 2.8 (1.64)

 337.96 (89.67) 77 (9.75)

Higher education, 1st cycle 11 (30.6)

 1.36 (1.1)

 334.99 (106.25) 87.05 (13.82)

Higher education, 2nd cycle 11 (30.6)

 1.64 (1.2)

 412.95 (198.6) 82.73 (13.34)
Higher education, 3rd cycle 9 (25) 1.22 (1.3) 368.77 (80.88) 86.94 (5.97)
Residential area, n (%)

Rural 6 (16.7) 1 (0.89)

 362.92 (76.48) 88.75 (6.85)

Semi-urban 5 (13.9)

 1.8 (2.05)

 339.11 (72.3) 87 (11.37)

Urban 25 (64.9) 1.72 (1.24) 374.52 (154.78) 82.7 (12.62)
Persona chosen, n (%)

Persona 1 8 (22.2) *b * *

Persona 2 8 (22.2) * * *

Persona 3 8 (22.2) * * *

Personal 4 4 (11.1) * * *

Persona 5 8 (22.2) * * *
Health care experience with operations, n (%)

Yes 32 (88.9)

 1.59 (1.21)

 376.18 (136.24)

 85,39 (11.72)

No 4 (11.1) 1.75 (2.22) 299.55 (106.9) 75.62 (8)
Health care experience with outpatient operations, n (%)

Yes 18 (50) 1.39 (1.33) 367.15 (142.09) 86.11 (11.8)

No 18 (50) 1.83 (1.29) 368.19 (130) 82,5 (11.66)
Health care experience with suffering from a chronic illness, n (%)

Yes 11 (30.6) 1.27 (1.35) 386.68 (184.62) 81.14 (15.26)

No 25 (69.4) 1.76 (1.3) 359.3 (108.77) 85.7 (9.8)
Medical device experience with taking blood pressure, n (%)

Yes 24 (66.7) 1.54 (1.32) 361.43 (131.26) 63.3 (11.22)

No 12 (33.3) 1.75 (1.36) 380.15 (145.06) 86.25 (12.9)
Medical device experience with blood oxygenation testing, n (%)

Yes 5 (13.9) 0.4 (0.55) 309.16 (70.87) 89 (8.02)

No 31 (86.1) 1.81(1.3) 377.11 (140.31) 83.56 (12.12)
Information technology experience with ease of use of tablet/computer/telephone, n (%)

Very comfortable 23 (63.9)

 1.35 (1.23)

 360.16 (120.21) 86.85 (11.24)

Relatively comfortable 11 (30.6)

 2.27 (1.42)

 401.34 (166.78)

 78.18 (11.78)

Moderately comfortable 2 (5.6)

 1 (0)

 268.81 (33.95) 88.75 (5.3)


Rather uncomfortable 0 (0)

 —c — —

Not at all comfortable 0 (0) —

 — —
Frequency of use of technology, n (%)

Very often (every day) 5 (13.9) 1 (1.22)

 314.42 (48.65)

 92.5 (6.85)

Often (several times a week) 12 (33.3)

 1.25 (1.36)

 364.46 (133.26)

 90.42 (6.47)

Rarely (from time to time) 17 (47.2)

 2 (1.27)

 364.08 (99.16) 79.26 (11.38)

Very rarely (occasionally) 1 (2.8)

 3 (—)

 856.33 (—)

 55 (—)

Never 1 (2.8) 1 (—) 244.8 (—) 85 (—)
Open in a new tab

aN/A: not applicable.

b*Highly correlated to the ages of the participants.

c— Not available.

Age

The age of the participants (mean 40.75, SD 14.45, range 20-64 years) is significantly correlated (positively and weakly) with the number of manipulation errors (effectiveness: r=0.359; P=.03) and manipulation time (efficiency: r=0.357; P=.03). On the other hand, there was no significant correlation between age and SUS score (satisfaction: r=-0.138; P=.42). In addition, it is important to note that age is not correlated with the literacy level of the HLS-EU-Q16 (r=0.013; P=.94) or the NVS (r=-0.013; P=.94; Figure 4).

Figure 4.

Figure 4

Open in a new tab

Schematic representation of correlations between age, measurements of participants' health literacy, and usability (ISO 9241-11). HLS-EU-Q16: 16-item European Health Literacy Scale; NVS: Newest Vital Sign. *P<0.5; **P<0.01.

Technophilia

The IT experience of participants had no impact on the number of manipulation errors (effectiveness: F5,30=1.229; P=.32) or manipulation time (efficiency: F5,30=1.39; P=.26). On the other hand, there was a significant correlation between IT experience and SUS score (satisfaction: χ(3)=8.671; P=.03).

Moreover, previous experience of using medical devices that allow users to take their own blood pressure did not influence the number of manipulation errors (effectiveness: t34=0.443; P=.66), the manipulation time (efficiency: t34=0.39; P=.55), or the SUS score (satisfaction: Mann-Whitney U=104; P=.19). Previous experience of using medical devices for taking oxygen levels had a significant effect on the number of manipulation errors (effectiveness: t34=2.359; P=.02; η2=0.14), but this effect was not significant on the manipulation time (efficiency: t34=1.052; P=.30) or the SUS score (satisfaction: t34=-0.965; P=.34).

Educational Level

Educational level had no impact on usability in terms of the number of manipulation errors (effectiveness: F3,32=1.889; P=.15), manipulation time (efficiency: F3,32=0.698; P=.56), and SUS score (satisfaction: F3,32=1.076; P=.37).

Health Literacy

Systematic analyses were performed comparing the level of literacy (HLS-EU-Q16 and NVS) with each of the components of usability (effectiveness, efficiency, and satisfaction, as per the ISO 9241-11, 2018); Table 2 presents a descriptive representation of the results of the 2 health literacy questionnaires.

Table 2.

Descriptive statistics of the 16-item European Health Literacy Survey (HLS-EU-Q16) and the New Vital Sign (NVS) questionnaires.

Statistic HLS-EU-Q16 NVS
Mean (SD), range 12.97 (2.952), 5-16 4.17 (2.223), 0-6
Inadequate health literacy, n (%) 3 (8.3) 6 (16.7)
Problematic health literacy, n (%) 9 (25) 7 (19.4)
Sufficient health literacy, n (%) 24 (66.7) 23 (63.9)
Open in a new tab
HLS-EU-Q16 Questionnaire Results

There was no significant correlation between the results of the HLS-EU-Q16 and usability, either in terms of the number of manipulation errors (effectiveness: r=0.34; P=.84), manipulation time (efficiency: r=-0.40; P=.82), or the SUS score (satisfaction: r=0.144; P=.40). After correlation analysis, participants were clustered according to the HLS-EU-Q16 measures (Table 2), following the recommendations of Sørensen et al [31]. No intergroup differences could be observed between the HLS-EU-Q16 results and usability (Table 3) in terms of the number of manipulation errors (effectiveness: F2.33=0.277; P=.76), manipulation time (efficiency: F2.33=0.015; P=.99), and the SUS score (satisfaction: F2.33=0.483; P=.62).

Table 3.

Analyses of the 16-item European Health Literacy Survey (HLS-EU-Q16) score according to usability (effectiveness, efficiency, and satisfaction; n=36).

HLS-EU-Q16 score classification group (n=36) Effectivenessa, average number of errors (SD) Efficiencyb, average manipulation time (SD) Satisfactionc, SUSd score (SD)
Inadequate health literacy (n=3) 1.67 (2.08) 373.26 (88.76) 83.33 (3.82)
Problematic health literacy (n=9) 1.89 (1.27) 373.35 (98.53) 81.11 (14.53)
Sufficient health literacy (n=24) 1.50 (1.28) 364.84 (152.73) 85.62(11.3)
Open in a new tab

aANOVA: F2,33=0.277; P=.76.

bANOVA: F2,33=0.015; P=.99.

cANOVA: F2,33=0.483; P=.62.

dSUS: System Usability Survey.

NVS Questionnaire Results

There was a significant mean-size correlation between the results of the French version of the NVS questionnaire and usability (Table 4) in terms of the number of manipulation errors (effectiveness: r=-0.417; P=.01), manipulation time (efficiency: r=-0.38; P=.02), and the SUS score (satisfaction: r=0.45; P=.006). In other words, the higher a participant's level of health literacy (measured using NVS), the fewer manipulation errors they made (ie, they are more effective), the faster they manipulate (ie, they are more efficient), and the higher their SUS score will be (ie, they will be more satisfied).

Table 4.

Analyses of the New Vital Sign (NVS) results according to usability (effectiveness, efficiency, and satisfaction; n=36).

NVS score classification group (n=36) Effectivenessa, average number of errors (SD) Efficiencyb, average manipulation time (SD) Satisfactionc, SUSd score (SD)
Inadequate health literacy
(n=6)		 2.67 (0.816) 463 (165.18) 77.08 (14.27)
Problematic health literacy (n=7) 1.71 (0.756) 387.72 (219.2) 80.71 (15.05)
Sufficient health literacy (n=23) 1.30 (1.43) 336.7 (75.79) 87.28 (9.07)
Open in a new tab

aKruskal-Wallis test: χ2=6.679; P=.035, where P<.05 is significant.

bKruskal-Wallis test: χ2=3.07; P=.21.

cANOVA: F2,33=2.392, P=.11; Kruskal-Wallis test: χ2=2.618, P=.27.

dSUS: System Usability Survey.

After analyzing the correlations, the participants were clustered according to the NVS measurements (Table 2), following recommendations [40]. No intergroup differences could be observed between NVS literacy and usability (Table 4) except for the number of errors (effectiveness: χ2=6.679; P=.04).

Further intergroup analysis (Figure 4) shows a significant effect between the inadequate-health-literacy and sufficient-health-literacy groups as a function of the number of manipulation errors (effectiveness: Mann-Whitney U=27; P=.02).

Discussion

Principal Findings

This study’s objective was to better understand the relationships between 4 user characteristics (age, education, technophilia, and health literacy) and usability [3] (defined here as effectiveness, efficiency, and satisfaction) with regard to the use of the Smart Angel device. To do this, sociodemographic data were collected, literacy levels were investigated using the HLS-EU-Q16 [31] and the NVS [40], and usability measures were performed (errors and manipulation time, and SUS questionnaire).

We made 4 hypotheses that age (H1), technophilia (H2), and health literacy (H4) would have an impact on usability, while education level (H3) would not. Our first hypothesis (H1) was that older users would be less effective, efficient, and satisfied with the device compared to younger users. We can partially validate this hypothesis. The results show that the younger the individuals are, the less likely they are to make manipulation errors (ie, they are more effective) and the faster they manipulate the device (ie, they are more efficient). On the other hand, we did not observe any difference between the age of the subjects and the SUS score (satisfaction). All these results are in line with previous research [19,20,25,26]. Indeed, younger users are more effective (eg, Jones and Caird's glucometer [25]) and efficient (eg, Mykityshyn et al's glucometer [26] and Van der Vaart et al's application for narcoleptics [20]) compared to older users, with a positive and medium correlation [20]. However, younger users are as satisfied (SUS score) with the device as older users, which is consistent with the findings of Liang et al [19] while at variance with those of Georgsson and Staggers [11].

Our second hypothesis (H2) focused on technophilia (experience of information technology and medical devices). The results provide partial validation of this hypothesis, as no correlation was observed between IT experience and usability in terms of effectiveness and efficiency. On the other hand, the technophile participants had a significantly better SUS score (satisfaction) than participants with a low level of technophilia. While these results are consistent with those of Harte et al [27], they contradict previous works [11,23]. We explain these results by a relatively homogeneous representation of IT experience as a function of the age of participants in our sample. We believe that these items [51] highlight the subjective representation of technology use (in relation to age) rather than actual performance in the use of hardware. It is possible that older people may feel that they can properly manipulate a tablet without using other features available in the tool. They would then consider themselves to be quite technophilic, as they would be effective in the day-to-day use of the technology. However, their real capacity to adapt to the technologies is unknown. For example, if an update were to be performed on one of the applications commonly used, it is possible that this would destabilize the manipulation carried out by these individuals.

We also observed a correlation between experience with medical devices and usability. However, previous experience in the use of a blood pressure monitor had no impact on usability. Conversely, previous experience in the use of a pulse oximeter had a significant effect on effectiveness. Participants who had previously manipulated a pulse oximeter made significantly fewer errors than those who had never manipulated a pulse oximeter. In contrast, previous experience using a pulse oximeter had no effect on efficiency and satisfaction. All subjects who reported previous use of a pulse oximeter also reported previous manipulation of a blood pressure monitor. This result suggests that prior use of a pulse oximeter in combination with a blood pressure monitor would facilitate manipulation of the Smart Angel device in terms of effectiveness. We believe that participants who are accustomed to using this type of complex device are accustomed to being involved in health issues, which may be evidence of strong patient involvement in their own health [58].

Our third hypothesis (H3) was concerned with the lack of correlation between education level and usability. The results supported our hypothesis, as no significant correlation was found between participants' level of education and usability in terms of effectiveness, efficiency, and satisfaction. These results are also consistent with previous works [11,19,20].

Finally, the fourth hypothesis (H4) postulated that health literacy influences usability (effectiveness, efficiency, and satisfaction). The HLS-EU-Q16 scores showed no effect on usability (Figure 4). In contrast, the NVS scores showed a significant effect on the number of manipulation errors (effectiveness), manipulation time (efficiency), and SUS score (satisfaction). This is consistent with the results of previous studies [18,28,29]. Significant and medium-sized correlations between the NVS score and each of the usability dimensions were observed (Figure 4). This suggests that the higher the literacy level of the participants, the fewer manipulation errors they make (ie, the more effective they are), the faster they are (ie, the more efficient they are), and the higher the SUS score will be (ie, the more satisfied they are). However, after clustering the participants as recommended [40], there is a significant correlation between NVS literacy level and the number of errors (effectiveness) but no correlation with the manipulation time (efficiency) and the SUS score (satisfaction). Participants with a sufficient literacy level made significantly fewer errors than those with inadequate or problematic literacy.

It is important to note that the HLS-EU and NVS results are contradictory and demonstrate the complexity of health literacy assessment. In addition, our results suggest that the HLS-EU questioning the participants’ subjective abilities to access health information and make decisions introduces a significant bias in the measurement of health literacy. Some participants may claim to have no difficulty using health information, but there is no verification that this is, in fact, the case. Conversely, the NVS instrument appears to be better suited to gathering information on subjects' cognitive abilities, as it is a test that collects information on participants' thought processes when reading a food label, thus providing a more objective assessment of health literacy.

Conclusions and Research Prospects

This study provides theoretical insight into the effects of user characteristics (eg, age, experience, education, and health literacy) through the use of personas with respect to usability (effectiveness, efficiency, and satisfaction, according to ISO 9241-11 [3]) in the case of the Smart Angel connected medical device. This study provides a methodological contribution insofar as it revealed the differences in data collection between the NVS and the HLS-EU-Q16, thus demonstrating the importance of continuing research in the field of health literacy measurement tools. In addition, these results allow us to better understand the importance of the impact of technophilia among older people with a sufficient level of health literacy for usability.

The results of this study suggest 4 research prospects. First, the relevance of the personas method in the prototype evaluation phase has never been proven. This method is classically used in the design phase by designers (ergonomists, designers, engineers, and even future users) but more rarely used in an evaluation framework. To validate this method in this new context of use in the evaluation phase, it would be necessary to reproduce this study by adding a control group (ie, a group for whom the personas are not presented). Secondly, the training carried out by the researcher could be adapted according to the literacy levels of the participants. Indeed, the main difficulty in the use of a medical device is understanding the procedures, and this cannot be achieved if there is insufficient upstream training [59]. Training should certainly be adapted to the ages and literacy levels of the participants. Demonstration by the researcher may be sufficient for groups with adequate levels of health literacy. Conversely, for groups with inadequate or problematic levels of health literacy, further instruction should be considered. Third, the choice of questionnaire is a crucial step in measuring health literacy. Indeed, we observed a significant disparity in results between the HLS-EU-Q16 and the NVS. As already discussed, these 2 questionnaires do not appear to assess the same dimensions of health literacy. Further work is needed to understand what exactly is being assessed by each of the health literacy questionnaires. We believe that it is better to evaluate this skill with objective assessments. In the same way, it would have been interesting to perform objective measurements of technophilia.

Finally, beyond health literacy, it would now be appropriate to measure the level of eHealth literacy [20]. Unfortunately, there is no valid questionnaire in French on this subject. Thus, more systematic translations and adaptations of these tools should be considered in future studies.

Currently, as a result of this study, the Smart Angel device is in clinical trials where usability tests continue to be carried out in in situ conditions.

Acknowledgments

This project was financially supported by the European Regional Development Fund (ERDF), Evolucare, and the Programme d'Investissements d'Avenir (PIA), which is one of 5 research and development projects structuring competitiveness of the CSPP.

Abbreviations

HLS-EU

European Health Literacy Survey

HLS-EU-Q16

16-item European Health Literacy Survey

ICC

intraclass correlation

IT

information technology

NVS

Newest Vital Sign

REALM

Rapid Estimate of Adult Literacy in Medicine

SUS

System Usability Survey

S-TOFHLA

Short Test of Functional Health Literacy in Adults

TOFHLA

Test of Functional Health Literacy in Adults

Appendix

Multimedia Appendix 1

Example of a persona.

Footnotes

Conflicts of Interest: None declared.

References

  • 1.Ghosh S, Sallam S. Patient satisfaction and postoperative demands on hospital and community services after day surgery. Br J Surg. 1994 Nov;81(11):1635–8. doi: 10.1002/bjs.1800811124. [DOI] [PubMed] [Google Scholar]
  • 2.International Organization for Standardization (ISO) Medical devices - Application of risk management to medical devices. 2019. [2019-03-03]. https://www.iso.org/standard/72704.html https://www.iso.org/obp/ui/#iso:std:72704:en.
  • 3.International Organization for Standardization (ISO) Ergonomics of human-system interaction - Part 11: Usability: Definitions concepts. 2018. [2018-03-03]. https://www.iso.org/standard/63500.html https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en.
  • 4.Carayon P, Hoonakker P. Human Factors and Usability for Health Information Technology: Old and New Challenges. Yearb Med Inform. 2019 Aug;28(1):71–77. doi: 10.1055/s-0039-1677907. http://www.thieme-connect.com/DOI/DOI?10.1055/s-0039-1677907. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Cifter AS. Blood Pressure Monitor Usability Problems Detected Through Human Factors Evaluation. Ergonomics in Design. 2017 May 22;25(3):11–19. doi: 10.1177/1064804617691397. [DOI] [Google Scholar]
  • 6.Greysen SR, Chin Garcia C, Sudore RL, Cenzer IS, Covinsky KE. Functional impairment and Internet use among older adults: implications for meaningful use of patient portals. JAMA Intern Med. 2014 Jul 01;174(7):1188–90. doi: 10.1001/jamainternmed.2014.1864. http://europepmc.org/abstract/MED/24839165. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Kortum P, Peres SC. Evaluation of Home Health Care Devices: Remote Usability Assessment. JMIR Hum Factors. 2015 Jun 05;2(1):e10. doi: 10.2196/humanfactors.4570. https://humanfactors.jmir.org/2015/1/e10/ [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.LeRouge C, Ma J, Sneha S, Tolle K. User profiles and personas in the design and development of consumer health technologies. Int J Med Inform. 2013 Nov;82(11):e251–68. doi: 10.1016/j.ijmedinf.2011.03.006. [DOI] [PubMed] [Google Scholar]
  • 9.Matthew-Maich N, Harris L, Ploeg J, Markle-Reid M, Valaitis R, Ibrahim S, Gafni A, Isaacs S. Designing, Implementing, and Evaluating Mobile Health Technologies for Managing Chronic Conditions in Older Adults: A Scoping Review. JMIR Mhealth Uhealth. 2016 Jun 09;4(2):e29. doi: 10.2196/mhealth.5127. https://mhealth.jmir.org/2016/2/e29/ [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Schnall R, Rojas M, Bakken S, Brown W, Carballo-Dieguez A, Carry M, Gelaude D, Mosley JP, Travers J. A user-centered model for designing consumer mobile health (mHealth) applications (apps) J Biomed Inform. 2016 Apr;60:243–51. doi: 10.1016/j.jbi.2016.02.002. https://linkinghub.elsevier.com/retrieve/pii/S1532-0464(16)00024-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Georgsson M, Staggers N. Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics. J Am Med Inform Assoc. 2016 Jan;23(1):5–11. doi: 10.1093/jamia/ocv099. http://europepmc.org/abstract/MED/26377990. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform. 2009 May;78(5):340–53. doi: 10.1016/j.ijmedinf.2008.10.002. [DOI] [PubMed] [Google Scholar]
  • 13.Nelson LA, Mulvaney SA, Johnson KB, Osborn CY. mHealth Intervention Elements and User Characteristics Determine Utility: A Mixed-Methods Analysis. Diabetes Technol Ther. 2017 Jan;19(1):9–17. doi: 10.1089/dia.2016.0294. http://europepmc.org/abstract/MED/28099052. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Peute LW, Driest KF, Marcilly R, Bras Da Costa Sabrina, Beuscart-Zephir M, Jaspers MWM. A framework for reporting on human factor/usability studies of health information technologies. Stud Health Technol Inform. 2013;194:54–60. [PubMed] [Google Scholar]
  • 15.Yen P, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc. 2012;19(3):413–22. doi: 10.1136/amiajnl-2010-000020. http://europepmc.org/abstract/MED/21828224. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Borsci S, Uchegbu I, Buckle P, Ni Z, Walne S, Hanna GB. Designing medical technology for resilience: integrating health economics and human factors approaches. Expert Rev Med Devices. 2018 Jan 20;15(1):15–26. doi: 10.1080/17434440.2018.1418661. [DOI] [PubMed] [Google Scholar]
  • 17.Grebin SZ, Echeveste MES, Magnago PF, Tanure RLZ, Pulgati FH. [Analytical strategy for assessment of usability of medical devices from the user's perspective: a study of hemodialysis patients] Cad Saude Publica. 2018 Aug 20;34(8):e00074417. doi: 10.1590/0102-311X00074417. https://www.scielo.br/scielo.php?script=sci_arttext&pid=S0102-311X2018000805001&lng=en&nrm=iso&tlng=en. [DOI] [PubMed] [Google Scholar]
  • 18.Jensen JD, King AJ, Davis LA, Guntzviller LM. Utilization of internet technology by low-income adults: the role of health literacy, health numeracy, and computer assistance. J Aging Health. 2010 Sep;22(6):804–26. doi: 10.1177/0898264310366161. [DOI] [PubMed] [Google Scholar]
  • 19.Liang J, Xian D, Liu X, Fu J, Zhang X, Tang B, Lei J. Usability Study of Mainstream Wearable Fitness Devices: Feature Analysis and System Usability Scale Evaluation. JMIR Mhealth Uhealth. 2018 Nov 08;6(11):e11066. doi: 10.2196/11066. https://mhealth.jmir.org/2018/11/e11066/ [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.van der Vaart R, van Driel D, Pronk K, Paulussen S, Te Boekhorst Selma, Rosmalen JGM, Evers AWM. The Role of Age, Education, and Digital Health Literacy in the Usability of Internet-Based Cognitive Behavioral Therapy for Chronic Pain: Mixed Methods Study. JMIR Form Res. 2019 Nov 21;3(4):e12883. doi: 10.2196/12883. https://formative.jmir.org/2019/4/e12883/ [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Monkman H, Kushniruk A. A health literacy and usability heuristic evaluation of a mobile consumer health application. Stud Health Technol Inform. 2013;192:724–8. [PubMed] [Google Scholar]
  • 22.Sonderegger A, Schmutz S, Sauer J. The influence of age in usability testing. Appl Ergon. 2016 Jan;52:291–300. doi: 10.1016/j.apergo.2015.06.012. [DOI] [PubMed] [Google Scholar]
  • 23.Sparkes J, Valaitis R, McKibbon A. A usability study of patients setting up a cardiac event loop recorder and BlackBerry gateway for remote monitoring at home. Telemed J E Health. 2012;18(6):484–90. doi: 10.1089/tmj.2011.0230. [DOI] [PubMed] [Google Scholar]
  • 24.Kaufman DR, Patel VL, Hilliman C, Morin PC, Pevzner J, Weinstock RS, Goland R, Shea S, Starren J. Usability in the real world: assessing medical information technologies in patients' homes. J Biomed Inform. 2003;36(1-2):45–60. doi: 10.1016/s1532-0464(03)00056-x. https://linkinghub.elsevier.com/retrieve/pii/S153204640300056X. [DOI] [PubMed] [Google Scholar]
  • 25.Jones J, Caird JK. The Usability of Blood Glucose Meters: Task Performance Differences Between Younger and Older Age Groups. Proceedings of the Human Factors and Ergonomics Society Annual Meeting; Proceedings of the Human Factors and Ergonomics Society Annual Meeting Internet; 2017 Sep 1; 61st International Annual Meeting, JW Marriott, Austin, Texas, USA. 2017. Sep 28, pp. 604–608. [DOI] [Google Scholar]
  • 26.Mykityshyn AL, Fisk AD, Rogers WA. Learning to use a home medical device: mediating age-related differences with training. Hum Factors. 2002;44(3):354–64. doi: 10.1518/0018720024497727. [DOI] [PubMed] [Google Scholar]
  • 27.Harte R, Hall T, Glynn L, Rodríguez-Molinero Alejandro, Scharf T, Quinlan LR, ÓLaighin Gearóid. Enhancing Home Health Mobile Phone App Usability Through General Smartphone Training: Usability and Learnability Case Study. JMIR Hum Factors. 2018 Apr 26;5(2):e18. doi: 10.2196/humanfactors.7718. https://humanfactors.jmir.org/2018/2/e18/ [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Czaja SJ, Zarcadoolas C, Vaughon WL, Lee CC, Rockoff ML, Levy J. The usability of electronic personal health record systems for an underserved adult population. Hum Factors. 2015 May 05;57(3):491–506. doi: 10.1177/0018720814549238. http://europepmc.org/abstract/MED/25875437. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Kim H, Xie B. Health literacy in the eHealth era: A systematic review of the literature. Patient Educ Couns. 2017 Jun;100(6):1073–1082. doi: 10.1016/j.pec.2017.01.015. [DOI] [PubMed] [Google Scholar]
  • 30.Chaniaud N, Métayer Natacha, Megalakaki O, Loup-Escande E. Effect of Prior Health Knowledge on the Usability of Two Home Medical Devices: Usability Study. JMIR Mhealth Uhealth. 2020 Sep 21;8(9):e17983. doi: 10.2196/17983. https://mhealth.jmir.org/2020/9/e17983/ [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Sørensen Kristine, Van den Broucke S, Fullam J, Doyle G, Pelikan J, Slonska Z, Brand H, (HLS-EU) Consortium Health Literacy Project European Health literacy and public health: a systematic review and integration of definitions and models. BMC Public Health. 2012 Jan 25;12(1):80. doi: 10.1186/1471-2458-12-80. https://bmcpublichealth.biomedcentral.com/articles/10.1186/1471-2458-12-80. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Williams MV, Baker DW, Parker RM, Nurss JR. Relationship of functional health literacy to patients' knowledge of their chronic disease. A study of patients with hypertension and diabetes. Arch Intern Med. 1998 Jan 26;158(2):166–72. doi: 10.1001/archinte.158.2.166. [DOI] [PubMed] [Google Scholar]
  • 33.Mõttus René, Johnson W, Murray C, Wolf MS, Starr JM, Deary IJ. Towards understanding the links between health literacy and physical health. Health Psychol. 2014 Feb;33(2):164–73. doi: 10.1037/a0031439. [DOI] [PubMed] [Google Scholar]
  • 34.Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011 Jul 19;155(2):97–107. doi: 10.7326/0003-4819-155-2-201107190-00005. [DOI] [PubMed] [Google Scholar]
  • 35.Duteille Franck. Compétences en santé déficientes : obstacle à une prise en charge optimale. Revue Médicale Suisse, 8. 2019 Apr;:1016–1021. doi: 10.1016/j.refrac.2019.04.003. https://www.revmed.ch/RMS/2012/RMS-340/Competences-en-sante-deficientes-obstacle-a-une-prise-en-charge-optimale. [DOI] [PubMed] [Google Scholar]
  • 36.Jordan JE, Osborne RH, Buchbinder R. Critical appraisal of health literacy indices revealed variable underlying constructs, narrow content and psychometric weaknesses. J Clin Epidemiol. 2011 Apr;64(4):366–79. doi: 10.1016/j.jclinepi.2010.04.005. [DOI] [PubMed] [Google Scholar]
  • 37.Haun JN, Valerio MA, McCormack LA, Sørensen Kristine, Paasche-Orlow MK. Health literacy measurement: an inventory and descriptive summary of 51 instruments. J Health Commun. 2014 Oct 14;19 Suppl 2(sup2):302–33. doi: 10.1080/10810730.2014.936571. [DOI] [PubMed] [Google Scholar]
  • 38.Parker Ruth. Health literacy: a challenge for American patients and their health care providers. Health Promot Int. 2000;15(4):277–283. doi: 10.1093/heapro/15.4.277. https://academic.oup.com/heapro/article/15/4/277/595941. [DOI] [Google Scholar]
  • 39.Murphy P, Davis T, Long S, Jackson R, Decker B. Rapid Estimate of Adult Literacy in Medicine (REALM): A Quick Reading Test for Patients. Journal of Reading. 1993;37(2):124–130. https://www.jstor.org/stable/40033408. [Google Scholar]
  • 40.Weiss BD, Mays MZ, Martz W, Castro KM, DeWalt DA, Pignone MP, Mockbee J, Hale FA. Quick assessment of literacy in primary care: the newest vital sign. Ann Fam Med. 2005;3(6):514–22. doi: 10.1370/afm.405. http://www.annfammed.org/cgi/pmidlookup?view=long&pmid=16338915. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Connor M, Mantwill S, Schulz PJ. Functional health literacy in Switzerland--validation of a German, Italian, and French health literacy test. Patient Educ Couns. 2013 Jan;90(1):12–7. doi: 10.1016/j.pec.2012.08.018. [DOI] [PubMed] [Google Scholar]
  • 42.Housten AJ, Lowenstein LM, Hoover DS, Leal VB, Kamath GR, Volk RJ. Limitations of the S-TOFHLA in measuring poor numeracy: a cross-sectional study. BMC Public Health. 2018 Mar 27;18(1):405. doi: 10.1186/s12889-018-5333-9. https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-018-5333-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Baker DW, Williams MV, Parker RM, Gazmararian JA, Nurss J. Development of a brief test to measure functional health literacy. Patient Educ Couns. 1999 Sep;38(1):33–42. doi: 10.1016/s0738-3991(98)00116-5. [DOI] [PubMed] [Google Scholar]
  • 44.Sørensen Kristine, Pelikan JM, Röthlin Florian, Ganahl K, Slonska Z, Doyle G, Fullam J, Kondilis B, Agrafiotis D, Uiters E, Falcon M, Mensing M, Tchamov K, van den Broucke Stephan, Brand H, HLS-EU Consortium Health literacy in Europe: comparative results of the European health literacy survey (HLS-EU) Eur J Public Health. 2015 Dec;25(6):1053–8. doi: 10.1093/eurpub/ckv043. http://europepmc.org/abstract/MED/25843827. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Agence Technique De L'Information sure L'Hospitalisation (ATIH) Etat des lieux 2015 sur l'activité de chirurgie ambulatoire. 2016. [2016-03-03]. https://www.atih.sante.fr/sites/default/files/public/content/1504/note_gdr_chirambu_2015.pdf https://www.atih.sante.fr/sites/default/files/public/content/1504/note_gdr_chirambu_2015.pdf.
  • 46.Loup-Escande E, Valentin A, Metayer N, Ricquebourd V, Bidard A, Damay P, Boisson C, Cuvillon P, Chaniaud N. Adaptation de la démarche ergonomique en phase amont de la conception de technologies e-santé?xemple du projet Smart Angel. EPIQUE; 9 au 12 juillet; Lyon. 2019. p. e. [Google Scholar]
  • 47.Lallemand C, Gronier G. Methodes de design UX: 30 méthodes fondamentales pour concevoir et évaluer les systèmes interactifs. Paris: Editions Eyrolles; 2018. Methodes de design UX: 30 méthodes fondamentales pour concevoir et évaluer les systèmes interactifs; p. 712. [Google Scholar]
  • 48.Mansfield ED, Wahba R, Gillis DE, Weiss BD, L'Abbé Mary. Canadian adaptation of the Newest Vital Sign©, a health literacy assessment tool. Public Health Nutr. 2018 Aug;21(11):2038–2045. doi: 10.1017/S1368980018000253. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Nunnaly J. Psychometric Theory. Chicago: McGraw Hill; 1978. [Google Scholar]
  • 50.Rouquette A, Nadot T, Labitrie P, Van den Broucke S, Mancini J, Rigal L, Ringa V. Validity and measurement invariance across sex, age, and education level of the French short versions of the European Health Literacy Survey Questionnaire. PLoS One. 2018 Dec 6;13(12):e0208091. doi: 10.1371/journal.pone.0208091. https://dx.plos.org/10.1371/journal.pone.0208091. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Agarwal R, Prasad J. A Conceptual and Operational Definition of Personal Innovativeness in the Domain of Information Technology. Information Systems Research. 1998 Jun;9(2):204–215. doi: 10.1287/isre.9.2.204. [DOI] [Google Scholar]
  • 52.Brooke J. SUS - A quick and dirty usability scale. In: Jordan P.W, Thomas B, McClelland I.L, Weerdmeester B, editors. Usability evaluation in industry. London: CRC Press; 1996. p. 252. [Google Scholar]
  • 53.Lewis J, Sauro J. The Factor Structure of the System Usability Scale. Human Centered Design Internet Springer, Berlin, Heidelberg; Interantion Conference on Human Centered Design; 2009; Berlin. 2009. pp. 94–103. [DOI] [Google Scholar]
  • 54.Bangor A, Kortum PT, Miller JT. An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction. 2008 Jul 30;24(6):574–594. doi: 10.1080/10447310802205776. [DOI] [Google Scholar]
  • 55.Bornet C, Brangier E. La méthode des personas: principes, intérêts et limites. Bulletin de psychologie Internet. 2013 May 3;524:115–134. doi: 10.3917/bupsy.524.0115. https://www.cairn.info/revue-bulletin-de-psychologie-2013-2-page-115.htm. [DOI] [Google Scholar]
  • 56.Friard O, Gamba M : a free, versatile open‐source event‐logging software for video/audio coding and live observations. Methods Ecol Evol. 2016 May 28;7(11):1325–1330. doi: 10.1111/2041-210x.12584. [DOI] [Google Scholar]
  • 57.Elie C, Colombet I. Methods of evaluation of reproducibility. Sang thrombose vaisseaux---Sang Thrombose Vaisseaux Internet 2011 Mar. 2011;23(3):138–145. doi: 10.1684/stv.2020.1138. [DOI] [Google Scholar]
  • 58.Greene J, Hibbard JH. Why does patient activation matter? An examination of the relationships between patient activation and health-related outcomes. J Gen Intern Med. 2012 May 30;27(5):520–6. doi: 10.1007/s11606-011-1931-2. http://europepmc.org/abstract/MED/22127797. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 59.Larsen T. Nurses' instruction of patients in the use of INR-monitors for self-management of cardio-vascular conditions: Missed instructional opportunities. Patient Educ Couns. 2017 Apr;100(4):673–681. doi: 10.1016/j.pec.2016.10.001. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Multimedia Appendix 1

Example of a persona.

humanfactors_v8i1e24846_app1.pdf (706.6KB, pdf)

Articles from JMIR Human Factors are provided here courtesy of JMIR Publications Inc.

RESOURCES