Abstract
Background
A growing number of positive airway pressure (PAP) device users will develop physical/sensory impairments such as arthritis. For these individuals, the usability of their PAP devices (eg, efficiency and satisfaction) may impact the frequency and safety of device usage. Questionnaires to assess PAP usability are unavailable; therefore, we developed the Usability of Sleep Apnea Equipment – Positive Airway Pressure (USE-PAP) questionnaire.
Methods
Questionnaire development included in-depth interviews to identify relevant content areas, a technical advisory panel to review/edit items, cognitive interviews to refine items, and a cross-sectional survey of Veterans Affairs sleep clinic patients assessing PAP device usability overall (one multi-item scale), usability of PAP components (multi-item scales for machine controls, mask/headgear, tubing, and humidifier), frequency of usability-related issues (one multi-item scale), PAP device characteristics, and demographics.
Results
After conducting 19 in-depth interviews, a panel meeting, and 10 cognitive interviews, we administered the survey to 100 PAP device users (67% ≥60 years; 90% male). The items assessing machine control usability received the least favorable ratings. Twenty percent of respondents reported difficulty getting equipment ready for use, and 33 percent had difficulty cleaning equipment. The six multi-item scales had excellent internal consistency reliability (alpha ≥0.84) and item–rest correlations (≥0.39).
Conclusions
This study provides initial support for the USE-PAP for measuring PAP device usability. Studies that include large samples are needed to further evaluate the psychometric properties of the USE-PAP. In addition, comparisons of USE-PAP responses with direct observations of PAP-related tasks and objectively measured PAP adherence are needed to fully evaluate the questionnaire.
Keywords: Obstructive sleep apnea, Aging, Adherence, Positive airway pressure therapy, Patient-reported measure
1. Introduction
An increasing number of patient-centered care efforts are underway to enable patients with sleep disorders to more easily report on the status of their physical, mental, and social health and humanity of their care (eg, dignity, waiting time) [1–4]. These efforts include patient surveys [5], which contain patient-reported measures that assess physical, mental, and social health (eg, Patient-Reported Outcomes Measurement Information System sleep measures [6–8]) and capture patients’ experiences accessing health care and communicating with providers (eg, Consumer Assessment of Healthcare Providers and Systems [9]). One aspect of care lacking patient-reported measures concerns patients’ experiences with their home medical devices [10], despite the growing use of these types of devices, particularly in the evaluation and treatment of sleep disorders (eg, portable home sleep-testing devices and positive airway pressure (PAP) devices).
The United States Food and Drug Administration (FDA) has been leading efforts to ensure that home medical device are usable and safe for the diverse group of patients who use them [11]. The FDA’s focus has been on optimizing “human factors,” a term that is familiar to manufacturers in the electronics industry who strive to optimize consumers’ experiences with computer and electronic devices (eg, personal computers and cell phones) [12]. Human factors principles have been well described and encompass issues such as learnability, memorability, efficiency, and satisfaction [13,14]. The Association for the Advancement of Medical Instrumentation (AAMI) has published human factors guidelines to help manufacturers optimize the design of medical devices for users [15]. However, manufacturers tend to prioritize the views of health professionals above those of patients when designing medical devices, which could lead to lack of attention to issues that are most salient to patients [16]. Although manufacturers perform human factors testing when developing devices, little data are available to assess whether patients encounter usability problems after the devices have been FDA-approved. Currently, when patients and prescribers of home medical devices want to provide feedback on how well human factors have been addressed, their options include reporting an adverse event through the FDA MedWatch website [17,18], speaking with manufacturers’ customer service centers, and/or discussing issues with sales and product representatives. To our knowledge, survey measures of how well home medical devices have addressed human factors are not available to most patients or to researchers interested in estimating human factors-related issues.
PAP devices for sleep-disordered breathing (SDB) are a group of commonly prescribed home medical devices, and the safety (eg, ability to keep the components clean) and acceptance of these devices are influenced by patients’ or caregivers’ ability to efficiently set up and maintain the devices. Measures are needed to document how well human factors have been addressed for users of PAP (ie, the usability of PAP devices). These measures would complement measures that focus on side effects, treatment adherence, and impact of PAP treatment on the health-related quality of life. In this article, we describe the development and evaluation of a questionnaire to assess human factors issues in PAP use, by assessing patients’ experiences with the steps necessary to assemble, operate, and maintain/clean their PAP devices and PAP components.
2. Methods
2.1. Procedures
The Usability of Sleep Apnea Equipment – Positive Airway Pressure (USE-PAP version 1.0) is a self-administered paper-and-pencil questionnaire that assesses patients’ experiences setting up and cleaning their PAP devices and collects information about equipment type and demographic information. We used a multi-stage approach [19] to develop the USE-PAP questionnaire:
In-depth interviews: To identify the types of unaddressed human factors issues faced by patients who use PAP equipment and concepts to be measured, we conducted in-depth interviews, a method that enables researchers to explore usability from the participants’ perspective and identify dimensions about PAP usability that may have not appeared in the literature. As reported previously [20], our participants were patients aged 60 years or older who self-reported physical and/or sensory impairment and disability related to PAP equipment use and health-care providers (sleep, geriatrics, physical medicine, and rehabilitation) who care for these patients. We focused on older patients for these interviews, because of the higher prevalence of disability and caregiver assistance in this age group and our goal of including participants who could provide detailed descriptions of PAP usability issues. We conducted interviews 19 participants.
Initial item development: This included the development of a set of closed-ended survey items adapted from existing questionnaires [21–26] and creation of de novo items. For the de novo items, we used information from the in-depth interviews and human factors engineering literature to inform the content of the items. Once we had an initial set of items, we obtained informal feedback from clinical staff experienced in caring for patients with SDB and iteratively revised the items.
Technical advisory panel to obtain input on the content and format of the questionnaire: The panel consisted of nine experts (sleep, geriatrics, physical medicine and rehabilitation, questionnaire development). Both prior to and during the meeting, panelists rated how well each item measures the targeted usability attribute, including: (1) effectiveness (ie, accuracy and completeness with which users achieve specified goals), (2) efficiency (ie, the resources expended in relation to the accuracy and completeness), (3) errors (ie, errors made during the use of the equipment and how easy it is to recover from them), (4) learnability (how quickly and easily users can begin to use equipment that is new to them), (5) memorability (the characteristics of equipment that allows the user to return to using the equipment after some period of not having used it), and (6) satisfaction (the degree to which the user finds the use of the product acceptable and the design of the equipment to be pleasant) [27]. Recommendations for revising or removing items were also discussed. We made changes to the questionnaire based upon suggestions made by the majority of panelists.
Cognitive interviews to refine items: After revising the questionnaire, we conducted 10 interviews with patients. Eligible patients (aged ≥50 years who self-reported at least one physical and/or sensory condition that affected the patient’s ability to handle their PAP equipment) were presented with the survey items (iteratively revised based upon each interview). Two research staff observed whether the participant answered the item readily, hesitated, refused, and/or asked for clarification. The research staff also documented suggestions for rephrasing the items and measured the length of time required to complete the items. The participants provided their overall impressions of the survey, described the extent to which the survey captured their experience setting up or cleaning their PAP equipment, and shared their opinion of the paper-based format. Examples of the comments made by participants are provided in Appendix.
Field test: To assess the performance of the questionnaire among adults aged 18 years or older who have been prescribed PAP therapy (ie, our target population for the questionnaire), we administered the closed-ended survey items to a sample of 100 participants recruited through the VA Greater Los Angeles our sleep center. The questionnaire and study recruitment materials were available in two out-patient sleep clinic waiting areas and the sleep center check-in areas from August through October 2013. Patients who were interested in participating reviewed the study information sheet, self-administered the questionnaire, and returned it by placing it in a closed box.
Tables 1 and 2 provide the items that were included in the questionnaire. Usability items within the questionnaire were grouped together based upon the component of the PAP device: (1) mask/headgear, (2) tubing, (3) humidifier, (4) controls, and (5) general (the PAP device considered as one unit). Based upon these groupings, we created five multi-item scales and a usability total score (see data analysis section below). In addition, information about the frequency of difficulties for each component and overall was collected, from which we created a sixth multi-item scale. Finally, we collected information about the type of PAP equipment issued to the patient and demographic information. No patient identifiers (ie, name, patient identification number, and contact information) were collected.
Table 1.
Field test sample characteristics (N = 100).
| Patient characteristic | Frequency (%) |
|---|---|
| Gender | |
| Male | 90 (95.7%) |
| Female | 4 (4.3%) |
| Age range | |
| <30 years | 1 (1.0%) |
| 30–39 years | 6 (6.2%) |
| 40–49 years | 8 (8.2%) |
| 50–59 years | 15 (15.5%) |
| 60–69 years | 56 (57.7%) |
| >70 years | 11 (11.3%) |
| How long have you been using your current machine? | |
| <1 year | 33 (34.0%) |
| 1–5 years | 45 (46.4%) |
| >5 years but <10 years | 15 (15.5%) |
| ≥10 years | 2 (2.1%) |
| Don’t know | 2 (2.1%) |
| Other than your current machine, have you ever used any other sleep apnea equipment at home (not temporary equipment used for testing purposes)? | |
| No | 75 (78.9%) |
| Yes | 20 (21.1%) |
| Does your machine have a humidifier? | |
| No | 8 (9.8%) |
| Yes | 74 (90.2%) |
| Does your machine start blowing air as soon as you put on your mask (or nasal pillows) even if you don’t touch any of the machine’s controls? | |
| No | 30 (31.9%) |
| Yes | 64 (68.1%) |
| What type of mask(s) or nasal pillows do you currently have? | |
| Nasal pillows (insert into nostrils) | 32 (34.0%)a |
| Nasal mask (fits over nose only) | 39 (41.5%)a |
| Full face mask (fits over nose and mouth) | 29 (30.8%)a |
| Don’t know | 2 (2.1%) |
| Do you know the brand and model name of your current machine? | |
| No/Don’t Know | 66 (71.7) |
| Yes | 26 (28.3) |
| Do you know the brand and model name of your current mask or nasal pillows? | |
| No/Don’t Know | 77 (80.2) |
| Yes | 19 (19.8) |
Some patients reported use of more than one type of mask.
Table 2.
Field Test: patient-reported usability ratings of their positive airway pressure equipment (N = 100).
| Item (or scale) | Frequency
|
Mean (SD) | Median (IQR) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Strongly agree | Slightly agree | Neither agree nor disagree | Slightly disagree | Strongly disagree | Doesn’t apply to me | Missing | |||
| General Usability | 85.2 (15.7) | 88.9 (77.8,97.2) | |||||||
| When I first got my current equipment, I easily learned how to get it ready for use. | 52 | 25 | 7 | 7 | 8 | Optionnotoffered | 1 | 76.8 (31.8) | 100 (75, 100) |
| I could remember how to get my equipment ready for use, even if | 65 | 20 | 6 | 3 | 4 | Optionnotoffered | 2 | 85.5 (25.6) | 100 (75, 100) |
| I did not use it for a month. | |||||||||
| I can successfully get my equipment ready for use without assistance. | 78 | 16 | 1 | 2 | 0 | Optionnotoffered | 3 | 93.8 (14.4) | 100 (100, 100) |
| I can quickly get my equipment ready for use. | 76 | 15 | 2 | 3 | 2 | Optionnotoffered | 2 | 90.8 (21.1) | 100 (100, 100) |
| I know when my equipment is working properly. | 57 | 16 | 11 | 7 | 3 | Optionnotoffered | 6 | 81.1 (28.1) | 100 (75, 100) |
| I can quickly remove my equipment from my body. | 79 | 13 | 1 | 1 | 3 | Optionnotoffered | 3 | 92.3 (20.5) | 100 (100, 100) |
| My equipment is easy to clean. | 50 | 31 | 8 | 5 | 3 | Optionnotoffered | 3 | 80.9 (25.5) | 100 (75, 100) |
| My equipment is convenient for traveling. | 40 | 22 | 12 | 12 | 11 | Optionnotoffered | 3 | 67.5 (35.2) | 75 (50, 100) |
| I would recommend this equipment to a friend who has sleep apnea. | 60 | 15 | 12 | 7 | 3 | Optionnotoffered | 3 | 81.4 (38.0) | 100 (75, 100) |
| Mask/Headgear | 75.6 (23.6) | 83.3 (41.7,91.7) | |||||||
| I can quickly put on my mask/headgear. | 62 | 24 | 3 | 5 | 4 | Optionnotoffered | 2 | 84.4 (26.2) | 100 (75, 100) |
| I can adjust my mask/headgear so it seals properly on my face. | 57 | 27 | 7 | 3 | 4 | Optionnotoffered | 2 | 83.3 (25.6) | 100 (75, 100) |
| I can perform activities that require use of my eyes (such as reading) while wearing my mask/headgear. | 29 | 25 | 19 | 16 | 10 | Optionnotoffered | 1 | 61.9 (33.4) | 75 (25, 100) |
| I can keep my mask/headgear in place during the night. | 38 | 25 | 12 | 8 | 16 | Optionnotoffered | 1 | 65.4 (36.7) | 75 (50, 100) |
| I can put on my mask/headgear if it becomes displaced during the night. | 53 | 26 | 13 | 2 | 4 | Optionnotoffered | 2 | 81.1 (25.9) | 100 (75, 100) |
| I am satisfied with what it takes to put on my mask/headgear. | 52 | 22 | 13 | 4 | 8 | Optionnotoffered | 1 | 76.8 (31.2) | 100 (50, 100) |
| Tubing | 90.9 (18.9) | 100 (91.7, 100) | |||||||
| I can connect the tubing so that the air flows from the machine to my mask. | 78 | 15 | 4 | 0 | 2 | Optionnotoffered | 1 | 92.2 (18.4) | 100 (100, 100) |
| I can quickly connect the tubing. | 74 | 17 | 5 | 1 | 2 | Optionnotoffered | 1 | 90.4 (20.1) | 100 (75, 100) |
| I am satisfied with what it takes to connect the tubing. | 73 | 19 | 3 | 2 | 2 | Optionnotoffered | 1 | 90.2 (20.5) | 100 (75, 100) |
| Humidifier | 87.5 (21.0) | 100 (80, 100) | |||||||
| I can operate my humidifier. | 63 | 10 | 6 | 5 | 3 | 4 | 9 | 85.9 (26.9) | 100 (75, 100) |
| I can quickly open the water chamber. | 71 | 10 | 1 | 5 | 2 | 4 | 7 | 90.2 (23.4) | 100 (100, 100) |
| I can fill the water chamber with minimal effort. | 74 | 8 | 2 | 2 | 3 | 4 | 7 | 91.6 (22.6) | 100 (100, 100) |
| I am satisfied with what it takes to clean the water chamber. | 54 | 20 | 6 | 3 | 5 | 5 | 7 | 82.7 (27.9) | 100 (75, 100) |
| I am satisfied with what it takes to prepare my humidifier for use. | 54 | 20 | 9 | 1 | 2 | 4 | 10 | 85.7 (22.5) | 100 (75, 100) |
| Machine controls | 63.0 (33.1) | 68.7 (43.7, 100) | |||||||
| I know how to adjust my machine’s controls. | 31 | 23 | 16 | 10 | 15 | 3 | 2 | 61.8 (36.1) | 75 (25, 100) |
| I can operate my machine’s controls. | 35 | 28 | 14 | 4 | 14 | 2 | 3 | 67.4 (34.8) | 75 (50, 100) |
| I can quickly adjust my machine’s controls when I need to. | 29 | 22 | 18 | 12 | 14 | 3 | 2 | 60.5 (35.5) | 75 (25, 100) |
| I am satisfied with what it takes to adjust my machine’s controls. | 30 | 16 | 29 | 7 | 14 | 2 | 2 | 60.7 (34.5) | 50 (50, 100) |
| Usability Total Score | 29 | 83.4 (15.9) | 88 (75, 97) | ||||||
SD = standard deviation, IQR = interquartile range, N/A = not available.
2.2. Data analysis
We examined frequency distributions of items to examine the central tendency and variability of the items. We also examined frequencies of inconsistent responses – namely, two participants (2% of sample) who indicated that they did not have a humidifier provided humidifier usability ratings and six participants (6%) provided responses to the item inquiring about frequency of humidifier problems. These responses were recoded to “doesn’t apply to me.” Next, scale scores were transformed linearly to have a 0–100 possible range. We then computed internal consistency reliability estimates (Cronbach’s alpha coefficient [28]) for the six multi-item scales. We estimated item–rest correlations (the correlation between an item and the scale that is formed by all other items). We estimated product–moment correlations among the six scales. Next, to assess whether it makes sense to combine the scales into a composite scale (ie, usability total score), we estimated the reliability of the composite scale (calculations based upon Mosier’s formula) [29]. Because the reliability of the composite scale was high, we computed a usability total score by averaging the transformed scale score means (scales listed in Table 2). Finally, we estimated the correlations of the scales with respondent age and length of equipment use, because we suspected that older participants and shorter length of equipment use would be correlated with less favorable usability ratings. Analyses were run using Stata 10.
2.3. Institutional review board
The study was approved by the VA Greater Los Angeles Health-care System (#2012–040547) and David Geffen School of Medicine at UCLA institutional review boards (#12–000605).
3. Results of field test
A total of 100 patients in the field test returned a questionnaire that was either complete or partially complete. Sample characteristics are presented in Table 1. Most of the respondents (71%) were aged 50–69 and were male (90%). One-third of the sample had used their PAP equipment for <1 year. Most patients were unable to tell us what brand and model PAP equipment they were using.
Table 2 describes the number of respondents who self-report PAP usability problems, grouped by scale (four component-specific usability scales (ie, mask/headgear, tubing, humidifier, and machine controls) and one general usability scale). The percentage of missing responses for most items was <5%. All items had a skewed distribution with the majority of participants selecting either “strongly” or “slightly” agree (ie, favorable usability ratings). The general usability items with the most favorable ratings were those assessing whether respondents get their equipment ready without assistance and can quickly remove the equipment from their bodies, while the least favorable general usability items were the items assessing learnability and convenience for traveling. For the component-specific items, the items assessing tubing usability received the most favorable ratings, whereas the items assessing machine control usability received less favorable ratings.
Table 3 describes how often the respondent experienced usability issues for each PAP component and for the PAP device overall in the past 30 days. The percentage of missing responses for the items in Table 2 was ≤6%. Item responses were skewed; however, all of the response options were selected at least once, except for the response for everyday difficulty preparing the water chamber for use. Between 10% and 19% of respondents selected “Doesn’t apply to me,” a response option that might be selected if, for example, participants had not used their PAP equipment in the past 30 days (ie, non-adherent patient, no humidifier).
Table 3.
Field Test: patient-reported ratings of frequency of positive airway pressure usability issues in the past 30 days (N = 100).
| Item | Frequency
|
|||||
|---|---|---|---|---|---|---|
| No days | Some days | Most days | Every day | Doesn’t apply to me | Missing | |
| I had difficulty putting on my mask/headgear. | 61 | 18 | 5 | 2 | 10 | 4 |
| I had difficulty adjusting the position of my mask/headgear. | 49 | 27 | 6 | 4 | 10 | 4 |
| I had difficulty connecting the tubing. | 76 | 7 | 2 | 1 | 10 | 4 |
| I had difficulty adjusting my machine’s controls. | 52 | 8 | 11 | 4 | 19 | 6 |
| I had difficulty preparing my water chamber for use. | 64 | 10 | 5 | 0 | 17 | 4 |
| I had difficulty replacing my machine’s filter. | 69 | 10 | 4 | 1 | 10 | 6 |
| I had difficulty getting my equipment ready for use. | 69 | 11 | 5 | 1 | 10 | 4 |
| I had difficulty cleaning my equipment. | 58 | 21 | 5 | 2 | 10 | 4 |
Table 4 provides the estimates of internal consistency reliability and the item–rest correlations for each usability item. Cronbach’s coefficient alpha values suggest excellent internal consistency (all scales had alpha ≥0.84). The item–rest correlations were ≥0.37, indicating that each item had a large correlation (according to Cohen’s effect size) with the rest of the items in the scale. We found significant product–moment correlations (r) among the general usability scale and other scales (correlations ranging from 0.48 to 0.74, p < 0.01) (see Table 5). The estimated reliability of the composite scale (usability total score) was 0.97. Respondent age was not significantly correlated with any of the scales (p > 0.27). A longer duration of equipment use was associated with less favorable machine control usability (−0.22, p = 0.04); length of equipment use was not correlated with any of the other scales (p > 0.13). The total usability score was not significantly correlated with age or duration of equipment use (p > 0.07).
Table 4.
Internal consistency reliability estimates (Cronbach’s coefficient alpha (α)) and item–rest correlations.
| Item | Item–rest correlationa |
|---|---|
| General Usability (Cronbach’s α = 0.84) | – |
| When I first got my current equipment, I easily learned how to get it ready for use. | 0.69 |
| I could remember how to get my equipment ready for use, even if I did not use it for a month. | 0.70 |
| I can successfully get my equipment ready for use without assistance. | 0.65 |
| I can quickly get my equipment ready for use. | 0.70 |
| I know when my equipment is working properly. | 0.46 |
| I can quickly remove my equipment from my body. | 0.39 |
| My equipment is easy to clean. | 0.59 |
| My equipment is convenient for traveling. | 0.60 |
| I would recommend this equipment to a friend who has sleep apnea. | 0.42 |
| Mask/Headgear (Cronbach’s α = 0.87) | – |
| I can quickly put on my mask/headgear. | 0.73 |
| I can adjust my mask/headgear so it seals properly on my face. | 0.76 |
| I can perform activities that require use of my eyes (such as reading) while wearing my mask/headgear. | 0.47 |
| I can keep my mask/headgear in place during the night. | 0.67 |
| I can put on my mask/headgear if it becomes displaced during the night. | 0.69 |
| I am satisfied with what it takes to put on my mask/headgear. | 0.73 |
| Tubing (Cronbach’s α = 0.96) | – |
| I can connect the tubing so that the air flows from the machine to my mask. | 0.88 |
| I can quickly connect the tubing. | 0.95 |
| I am satisfied with what it takes to connect the tubing. | 0.91 |
| Humidifier (Cronbach’s α = 0.91) | – |
| I can operate my humidifier. | 0.73 |
| I can quickly open the water chamber. | 0.83 |
| I can fill the water chamber with minimal effort. | 0.76 |
| I am satisfied with what it takes to clean the water chamber. | 0.62 |
| I am satisfied with what it takes to prepare my humidifier for use. | 0.86 |
| Machine controls (Cronbach’s α = 0.96) | – |
| I know how to adjust my machine’s controls. | 0.91 |
| I can operate my machine’s controls. | 0.89 |
| I can quickly adjust my machine’s controls when I need to. | 0.93 |
| I am satisfied with what it takes to adjust my machine’s controls. | 0.85 |
| Frequency of trouble (Cronbach’s α = 0.86) | – |
| I had difficulty putting on my mask/headgear. | 0.71 |
| I had difficulty adjusting the position of my mask/headgear. | 0.67 |
| I had difficulty connecting the tubing. | 0.58 |
| I had difficulty adjusting my machine’s controls. | 0.69 |
| I had difficulty preparing my water chamber for use. | 0.60 |
| I had difficulty replacing my machine’s filter. | 0.45 |
| I had difficulty getting my equipment ready for use. | 0.64 |
| I had difficulty cleaning my equipment. | 0.68 |
Correlation between an item and the scale that is formed by all other items in the scale.
Table 5.
Product–moment correlations (r).
| Scale | General usability | Mask/Headgear | Tubing | Humidifier | Machine controls | Frequency of trouble |
|---|---|---|---|---|---|---|
| General usability | 1.00 | – | – | – | – | – |
| Mask/Headgear | 0.68 | 1.00 | – | – | – | – |
| Tubing | 0.52 | 0.63 | 1.00 | – | – | – |
| Humidifier | 0.70 | 0.62 | 0.74 | 1.00 | – | – |
| Machine Controls | 0.63 | 0.48 | 0.31 | 0.52 | 1.00 | – |
| Frequency of trouble | 0.64 | 0.68 | 0.62 | 0.66 | 0.60 | 1.00 |
P < 0.01 for all correlations listed above.
4. Discussion
SDB is prevalent and is most commonly treated with PAP devices, which require patients to regularly set up and maintain their equipment. A paucity of studies have examined the prevalence of PAP usability issues [30,31], in part because of a lack of available instruments. The current study began a line of inquiry into patients’ experiences with this aspect of their PAP therapy. We developed and evaluated a questionnaire that collects patient-reported usability ratings of PAP equipment, using a combination of qualitative and quantitative methods.
The USE-PAP has favorable measurement properties and offers items that assess constructs that differ from most of the existing patient-reported measures targeting patients with SDB. The five scales in Table 2 are scored on a 0–100 possible range and a total score can be computed as the average of these scale scores. In addition, a sixth scale that measures frequency of PAP usability issues can be administered. If desired, a subset of the scales can be administered independently rather than all of the USE-PAP items. In our cognitive tests, participants felt that the questionnaire captured important elements of PAP usability. In our field testing, we found low levels of missing values, variability in the responses selected, and excellent reliability for the scales. The majority of items in the USE-PAP questionnaire focus on human factors/usability of PAP equipment, while other questionnaires focus on side effects to PAP treatment, self-efficacy for PAP therapy, and symptoms related to SDB treatment [32–38]. Only a few items in the USE-PAP questionnaire assess overall satisfaction with equipment, a construct that has been previously assessed by other researchers [36]. Surveying patients with the USE-PAP questionnaire (administering the entire instrument or individual scales) has the potential to uncover PAP-related issues that other questionnaires might not find.
Research on usability barriers is an important investment for the future. Our pilot survey identified a group of patients who have difficulty with their mask/headgear, adjusting their machines’ controls, and/or preparing their humidifier. SDB is a chronic condition that requires lifelong therapy with PAP and is associated with conditions such as diabetes and stroke that increase patients’ risk of PAP usability problems. The large number of adults with SDB who might not currently have PAP usability issues (ie, the patients who responded on their USE-PAP survey that that they did not have difficulty with their PAP equipment) will continue to require PAP therapy, even after many of them acquire comorbidities that increase their risk of usability problems. One of the first steps in launching research in this area is measuring the prevalence of usability issues and identifying risk factors for poor usability.
Our study has both strengths and limitations. We used mixed methods (qualitative and quantitative) to develop the USE-PAP questionnaire, which is recommended for developing closed-ended survey items [19]. One limitation is that the sampling frame for the in-depth interviews, cognitive interviews, and field test was obtained from a VA medical center and therefore, most of the participants were male. The field test was conducted in a sleep clinic setting, which may be enriched with patients with difficulties using their equipment. Another limitation is that our field test sample size was modest. We also noted some redundancy in the items for some of the scales (eg, tubing), but dropping an item would lead to lower reliability and therefore we believe is not a preferred strategy. Finally, although our field test included adults aged 18 years or older, our interview participants did not include adults <50 years, so it is possible that the USE-PAP questionnaire fails to measure usability constructs that are important to younger adults.
5. Conclusions
The USE-PAP questionnaire has the potential to identify patients who are having difficulty with their PAP equipment. It provides patients with an opportunity to voice their concerns about their PAP equipment in a structured manner. Additional studies are needed to evaluate the questionnaire in a larger sample, especially one with more gender diversity. Future studies should further assess the psychometric properties of the USE-PAP, its ability to discriminate among different equipment types, and its responsiveness to interventions aimed at improving usability. Studies that assess the relationship between usability, equipment types (eg, mask brand and model), and patient characteristics (eg, cognitive and physical disability) and outcomes (eg, PAP acceptance/adherence and patient safety) are also needed. Finally, comparisons of USE-PAP responses with direct observation of PAP-related tasks (usability task analyses) are needed to fully evaluate the questionnaire.
Acknowledgments
Funding sources
Funded by Veterans Administration Greater Los Angeles Geriatric Research, Education and Clinical Center, Department of Veterans Affairs Advanced Geriatrics Fellowship, American Federation for Aging Research (AFAR to CHF), American Sleep Medicine Foundation Physician Scientist Training Award 77-PA-12 (to CHF), the John A. Hartford Foundation (to CHF), and National Institute on Aging of the National Institutes of Health under Award Number K23AG045937 (to CHF)/The Beeson Career Development in Aging Research Award Program (supported by NIA, AFAR, the John A. Hart-ford Foundation, and the Atlantic Philanthropies). RDH was supported in part by grants from the NIA (P30-AG021684) and the NIMHD (2 P20-MD000182). JMD was supported by UCLA Claude Pepper Older Americans Independence Center (5P30AG028748) and National Center for Advancing Translational Sciences UCLA CTSI (UL1TR000124). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Portions of this work were presented at the 2014 Associated Professional Sleep Societies Meeting in Minneapolis, Minnesota, USA, and 2014 American Geriatrics Society Annual Meeting in Orlando, Florida, USA.
We thank the staff at VA Greater Los Angeles Healthcare Sleep Center for providing us the opportunity to recruit patients from the center. We thank the members of our technical advisory panel: David Ganz, MD, PhD; Diana Guth, BA, RRT, RCP; Nancy Harada, PhD, PT; Ron Hays, PhD; Arun Karlamangla, MD, PhD; Michael Littner, MD; Glenn Noble, RCP, RPFT; Sarah Tubbesing, MD; and Michelle Zeidler, MD, MS. We also thank Michael N. Mitchell, PhD, for his guidance in Stata programming.
Appendix
Cognitive Interview Domains and Examples of Responses
| Category | Example of response | Example of action taken by questionnaire developers |
|---|---|---|
| Instructions to participants and definitions of terms used in questionnaire | (version 1) Confused by the detailed description of what constitutes machine controls | Reviewed description of machine controls and removed excess detail |
| Individual item stems | (version 3) Stated that he does not interact with his machine controls, so he had trouble answering a question about the usability of machine controls | Added an item to identify patients whose machine turns on automatically |
| Individual item responses | (version 7) Participant indicated “not applicable” to an item asking whether he would recommend this equipment to a friend who has sleep apnea. | (version 7) Removed the “not applicable” response item from the items assessing magnitude of usability issues, because the response does not make sense |
| Overall impression | (version 2) Gives user the chance to inform (health-care staff) of what is happening (version 9) “Survey is very good” |
None |
| Did it capture your experiences as a user of sleep apnea equipment | Multiple respondents indicated that the questionnaire captured their experiences as users of sleep apnea equipment | None |
Footnotes
All work was completed at VA Greater Los Angeles Healthcare System, 16111 Plummer Street (11E), North Hills, CA 91343, USA.
Conflict of interest
No conflict of interest is declared for any of the manuscript’s authors. This study did not involve any off-label or investigational use.
The ICMJE Uniform Disclosure Form for Potential Conflicts of Interest associated with this article can be viewed by clicking on the following link: http://dx.doi.org/10.1016/j.sleep.2015.01.019.
References
- 1.Black N, Jenkinson C. Measuring patients’ experiences and outcomes. BMJ. 2009;339:b2495. doi: 10.1136/bmj.b2495. [DOI] [PubMed] [Google Scholar]
- 2.Epstein RM, Street RL., Jr The values and value of patient-centered care. Ann Fam Med. 2011;9(2):100–3. doi: 10.1370/afm.1239. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Edwards ST, Abrams MK, Baron RJ, et al. Structuring Payment to medical homes after the affordable care act. J Gen Intern Med. 2014;29:1410–13. doi: 10.1007/s11606-014-2848-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Selby JV, Lipstein SH. PCORI at 3 years – progress, lessons, and plans. N Engl J Med. 2014;370(7):592–5. doi: 10.1056/NEJMp1313061. [DOI] [PubMed] [Google Scholar]
- 5.Anhang PR, Elliott MN, Zaslavsky AM, et al. Examining the role of patient experience surveys in measuring health care quality. Med Care Res Rev. 2014;71(5):522–54. doi: 10.1177/1077558714541480. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Cella D, Riley W, Stone A, et al. The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. J Clin Epidemiol. 2010;63(11):1179–94. doi: 10.1016/j.jclinepi.2010.04.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Buysse DJ, Yu L, Moul DE, et al. Development and validation of patient-reported outcome measures for sleep disturbance and sleep-related impairments. Sleep. 2010;33(6):781–92. doi: 10.1093/sleep/33.6.781. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Drake CL, Hays RD, Morlock R, et al. Development and evaluation of a measure to assess restorative sleep. J Clin Sleep Med. 2014;10(7):733–41. doi: 10.5664/jcsm.3860. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Browne K, Roseman D, Shaller D, Edgman-Levitan S. Analysis & commentary. Measuring patient experience as a strategy for improving primary care. Health Aff (Millwood) 2010;29(5):921–5. doi: 10.1377/hlthaff.2010.0238. [DOI] [PubMed] [Google Scholar]
- 10.Hartford J. [accessed 11.20.12];Federal Cuts Could Impact Home Healthcare Device Market [Internet] 2012 < http://www.mddionline.com/article/federal-cuts-could-impact-home-healthcare-device-market>.
- 11.U.S. Food and Drug Administration. Draft Guidance for Industry and Food and Drug Administration Staff – Applying Human Factors and Usability Engineering to Optimize Medical Device Design. U S Food and Drug Administration; 2011. [accessed 08.05.12]. < http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm259748.htm>. [Google Scholar]
- 12.Carroll JM. Conceptualizing a possible discipline of human-computer interaction. Interact Comput. 2010;22:3–12. [Google Scholar]
- 13.Nielsen J. Usability engineering. San Francisco: Morgan Kaufmann Publishers Inc; 1993. [Google Scholar]
- 14.Frokjaer E, Hertzum M, Hornbaek K. Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? New York: ACM Press; 2000. pp. 345–52. [Google Scholar]
- 15.Association for the Advancement of Medical Instrumentation. Human factors engineering – design of medical devices. Arlington, VA: Association for the Advancement of Medical Instrumentation; 2010. Report No.: ANSI/AAMI HE75:2009. [Google Scholar]
- 16.Money AG, Barnett J, Kuljis J, Craven MP, Martin JL, Young T. The role of the user within the medical device design and development process: medical device manufacturers’ perspectives. BMC Med Inform Decis Mak. 2011;11:15. doi: 10.1186/1472-6947-11-15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Simone LK, Brumbaugh J, Ricketts C. Medical devices, the FDA, and the home healthcare clinician. Home Healthc Nurse. 2014;32(7):402–8. doi: 10.1097/NHH.0000000000000107. [DOI] [PubMed] [Google Scholar]
- 18.Kessler DA. Introducing MEDWatch. A new approach to reporting medication and device adverse effects and product problems. JAMA. 1993;269(21):2765–8. doi: 10.1001/jama.269.21.2765. [DOI] [PubMed] [Google Scholar]
- 19.Krause N. A comprehensive strategy for developing closed-ended survey items for use in studies of older adults. J Gerontol B Psychol Sci Soc Sci. 2002;57(5):S263–74. doi: 10.1093/geronb/57.5.s263. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Fung CH, Igodan U, Alessi C, et al. Human factors/usability barriers to home medical devices among individuals with disabling conditions: in-depth interviews with positive airway pressure device users. Disabil Health J. 2015;8(1):86–92. doi: 10.1016/j.dhjo.2014.06.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Kwahk J, Han SH. A methodology for evaluating the usability of audiovisual consumer electronic products. Appl Ergon. 2002;33(5):419–31. doi: 10.1016/s0003-6870(02)00034-0. [DOI] [PubMed] [Google Scholar]
- 22.Kirakowski J, Corbett M. SUMI: the software usability measurement inventory. Br J Educ Technol. 1993;24:210–12. [Google Scholar]
- 23.Brooke J. SUS: a quick and dirty usability scale. In: Jordan PW, Weerdmeester B, Thomas A, McLelland IL, editors. Usability evaluation in industry. London: Taylor and Francis; 1996. [Google Scholar]
- 24.Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum Comput Interact. 1995;7(1):57–78. [Google Scholar]
- 25.Lin HX, Choong YY, Salvendy G. A proposed index of usability: a method for comparing the relative usability of different software systems. Behav Inf Technol. 1997;16(4/5):267–78. [Google Scholar]
- 26.Chin JP, Diehl VA, Norman KL. Development of an instrument measuring user satisfaction of the human-computer interface. New York: Association for Computing Machinery; 1988. pp. 213–18. [Google Scholar]
- 27.Nielsen J. [accessed 05.22.12];Usability 101: Introduction to Usability [Internet] 2010 < http://www.useit.com/alertbox/20030825.html>.
- 28.Nunnally JC. Psychometric theory. 2. New York: McGraw-Hill; 1978. [Google Scholar]
- 29.Mosier C. On the reliability of a weighted composite. Psychometrika. 1943;8:161–8. [Google Scholar]
- 30.Hui DS, Choy DK, Li TS, et al. Determinants of continuous positive airway pressure compliance in a group of Chinese patients with obstructive sleep apnea. Chest. 2001;120(1):170–6. doi: 10.1378/chest.120.1.170. [DOI] [PubMed] [Google Scholar]
- 31.Fung CH, Martin JL, Jouldjian S, Alessi CA. The association between difficulty using positive airway pressure equipment and adherence to therapy: a pilot study. Sleep Breath. 2013;17(2):853–9. doi: 10.1007/s11325-012-0779-y. [DOI] [PubMed] [Google Scholar]
- 32.Brostrom A, Arestedt KF, Nilsen P, Stromberg A, Ulander M, Svanborg E. The side-effects to CPAP treatment inventory: the development and initial validation of a new tool for the measurement of side-effects to CPAP treatment. J Sleep Res. 2010;19(4):603–11. doi: 10.1111/j.1365-2869.2010.00825.x. [DOI] [PubMed] [Google Scholar]
- 33.Stepnowsky CJ, Jr, Marler MR, Ancoli-Israel S. Determinants of nasal CPAP compliance. Sleep Med. 2002;3(3):239–47. doi: 10.1016/s1389-9457(01)00162-9. [DOI] [PubMed] [Google Scholar]
- 34.Flemons WW, Reimer MA. Development of a disease-specific health-related quality of life questionnaire for sleep apnea. Am J Respir Crit Care Med. 1998;158(2):494–503. doi: 10.1164/ajrccm.158.2.9712036. [DOI] [PubMed] [Google Scholar]
- 35.Weaver TE, Maislin G, Dinges DF, et al. Self-efficacy in sleep apnea: instrument development and patient perceptions of obstructive sleep apnea risk, treatment benefit, and volition to use continuous positive airway pressure. Sleep. 2003;26(6):727–32. doi: 10.1093/sleep/26.6.727. [DOI] [PubMed] [Google Scholar]
- 36.Bachour A, Vitikainen P, Virkkula P, Maasilta P. CPAP interface: satisfaction and side effects. Sleep Breath. 2013;17(2):667–72. doi: 10.1007/s11325-012-0740-0. [DOI] [PubMed] [Google Scholar]
- 37.Kreivi HR, Maasilta P, Bachour A. Willingness score obtained after a short CPAP trial predicts CPAP use at 1 year. Sleep Breath. 2014;18(1):207–13. doi: 10.1007/s11325-013-0872-x. [DOI] [PubMed] [Google Scholar]
- 38.Moroni L, Neri M, Lucioni AM, Filipponi L, Bertolotti G. A new means of assessing the quality of life of patients with obstructive sleep apnea: the MOSAS questionnaire. Sleep Med. 2011;12(10):959–65. doi: 10.1016/j.sleep.2011.07.010. [DOI] [PubMed] [Google Scholar]