Summary
We developed an instrument to measure the satisfaction of lung transplant recipients with home monitoring. The survey comprised 15 items, each scored on a five-point Likert-type scale (from strongly disagree to strongly agree). Three additional free-text items enabled subjects to provide comments. The survey had a scoring range of 15–75. In a test group of 43 patients, the internal consistency (Cronbach’s alpha) was 0.93 overall for all questions. The intra-class correlation for scores from the same 27 patients approximately 2.5 months apart was 0.77 for the total score. The survey was used to evaluate subject satisfaction in a randomized controlled trial of a computerized algorithm for triaging lung transplant recipients. Surveys were mailed to 50 study subjects and were returned by 32 (64% return rate). Ninety percent of respondents were satisfied with the home monitoring programme and would recommend it to other patients.
Introduction
There are few patient satisfaction survey instruments for evaluating spirometery in telemonitoring. The Telemedicine Perception Questionnaire (TMPQ), the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and the Telemedicine Satisfaction Survey (TSQ) are examples of telemedicine satisfaction surveys that have been validated and used to study general telemedicine applications.1–3 However, they do not provide information about using spirometry for remote monitoring of lung transplant recipients. The reliability and validity of a questionnaire for lung transplant patients were reported by DeVito Dabbs et al., but the questionnaire focused on symptoms and their effect on the recipient’s activities rather than the use of the monitoring programme to track clinical status.4
We have therefore developed a Remote Monitoring Satisfaction Survey (RMSS). It was used to evaluate subject satisfaction in a randomized controlled trial of a computerized algorithm for triaging lung transplant recipients who might need direct clinical assessment and intervention.
Methods
The guidelines outlined by Demiris were followed in developing the survey instrument.5 Potential satisfaction questions were compiled from the literature and reviewed for inclusion. Items were adapted from the Department of Veteran’s Affairs Telemedicine Provider Satisfaction Survey Data Bank, the American Telemedicine Association Home Telehealth Satisfaction Item Bank, the Telemedicine Perception Questionnaire, the Patient Satisfaction Questionnaire PSQ-18 and from our research experience in using remote monitoring of lung function with lung transplant recipients.1,6–8
Items were selected to evaluate three elements of remote monitoring: the equipment, the programme personnel and the overall programme. The final version of the satisfaction survey comprised 15 items, each scored on a five-point Likert-type scale (from strongly disagree to strongly agree). Three additional free-text items enabled subjects to provide comments. These additional questions enquired about the participants’ reasons for utilizing home spirometry, for not using home spirometry and miscellaneous comments regarding the home telehealth programme. The survey instrument was written at a sixth grade reading level based on the Flesch-Kincaid Readability Index.9 The survey is available from the authors.
Validity and reliability
Validity was determined by the content, face and criterion validity. Content validity indicates how the items in the survey cover the range of topics thought to be addressed by the instrument. Content validity was based on the sources of the survey items and the judgement of the research team. Face validity indicates the degree to which the instrument appears to measure what it is intended to measure. Face validity was assessed by the research team and through informal discussions with study participants. Criterion validity relates survey scores to objective performance measures. Criterion validity was assessed by the correlation between actual home monitoring adherence and self-reported adherence.
Reliability was measured by internal consistency and repeatability. Internal consistency reflects the consistency of responses across all items and within each of the three subscales of the survey. Internal consistency was evaluated by Cronbach’s alpha. Repeatability was measured by the test-retest reliability. This was based on the intra-class correlation between two surveys sent to the same participants in the survey development group (see below) approximately 2.5 months apart.
Psychometric properties
To determine the psychometric properties of the RMSS, the instrument was sent to 50 patients participating in a clinical home spirometry monitoring programme. Survey instruments were mailed to patients’ home addresses along with stamped self-addressed envelopes for returning the completed surveys. Surveys were identified by study number with no further subject identification. Completed surveys were returned to the researchers and not to the patients’ clinical coordinators to preserve anonymity and minimize the potential for biased responses. A coded demographic information sheet was also included with the mailings to the survey development group to ascertain the number of years since their transplant (less than 1 year, 1–2 years, 3–4 years, or 5 or more years), frequency of doing home spirometry (daily/almost daily, several times a week, once a week, rarely or none), age (21–35 years, 36–50 years, 51–65 years, 66 years or older), gender and education (high school, beyond high school). The study was approved by the appropriate ethics committee and all participants provided informed consent.
Nurse triage study
Once the validity and reliability of the survey had been established, it was used to assess the satisfaction of participants in a remote monitoring nurse triage study. In this study, lung transplant recipients were randomized into control and intervention groups. Subjects in both arms of the study used an electronic spirometer diary device (TransViva Home Spirometer, Minneapolis, MN, USA). Satisfaction surveys were mailed to 50 of the 68 participants at the conclusion of the study. Demographic information was collected from study records for the nurse triage study subjects. The study was approved by the appropriate ethics committee.
Results
Demographic information from the survey development group and the nurse triage study group is summarised in Table 1. Many of the patients in the survey development group were in the clinical home spirometry programme for 5–10 years, while the subjects in the nurse triage study group participated for a maximum of four years. Thus there was a difference in time since transplant between subjects in the two groups. There was also a difference in age distribution between the survey development group (75% were >50 years old) and the nurse triage study group (86% were >50 years old).
Table 1.
Demographic information. The P values for time since transplant and age were derived from the Wilcoxon rank sum test; the others were based on Fisher’s exact test
| Survey development group n = 43 | Nurse triage study group n = 32 | P value | |
|---|---|---|---|
| Time since transplant (years) | <0.001 | ||
| < 1 | 1 | ||
| 1–2 | 6 | 15 | |
| 3–4 | 24 | 17 | |
| 5 or more | 12 | ||
| Frequency of home spirometry | |||
| Daily/almost daily | 7 | ||
| Several times a week | 15 | ||
| Once a week | 5 | ||
| Rarely/never | 8 | ||
| Age (years) | 0.32 | ||
| 21–35 | 4 | ||
| 36–50 | 7 | 5 | |
| 51–65 | 24 | 20 | |
| 66 or more | 8 | 7 | |
| Gender | 0.65 | ||
| Female | 20 | 13 | |
| Male | 23 | 19 | |
| Education | 0.79 | ||
| High School | 14 | 9 | |
| Beyond High School | 20 | 17 |
Validity and reliability
The survey demonstrated content, face and criterion validity based on the sources of the survey items, the judgement of the research team, and the high correlation (0.87) between home monitoring adherence from self-reports and clinical records.
For the survey development testing group, the internal consistency (Cronbach’s alpha) was 0.93 overall for all questions. It was 0.84 for the equipment/technical questions, 0.80 for the personnel/communication questions and 0.90 for the programme evaluation questions, see Table 2.
Table 2.
Item analysis for Cronbach’s alpha
| Question | Mean of total score if deleted | Variance of total score if deleted | Item-total correlation | Alpha if deleted |
|---|---|---|---|---|
| Spirometer concerns | ||||
| The spirometer is easy to use | 58.3 | 58.7 | 0.64 | 0.92 |
| The spirometer display is easy to read | 58.6 | 56.0 | 0.66 | 0.92 |
| The spirometer is reliable and has few technical problems | 58.6 | 56.4 | 0.68 | 0.92 |
| I received adequate training in using my home spirometer | 58.4 | 59.0 | 0.62 | 0.92 |
| The spirometer gives me accurate test results | 58.8 | 59.9 | 0.38 | 0.93 |
| If technical problems occur, the staff are quick to respond and fix the problems | 58.4 | 58.2 | 0.67 | 0.92 |
| The amount of time it takes to complete my daily home spirometry is acceptable | 58.7 | 57.5 | 0.57 | 0.92 |
| Communications/interactions | ||||
| The home monitoring staff are responsive to my questions and concerns | 58.3 | 57.0 | 0.72 | 0.92 |
| My doctors are interested in reviewing my home spirometry tests | 58.9 | 56.8 | 0.39 | 0.93 |
| I am satisfied with amount of communication I receive from the home monitoring staff | 58.5 | 55.0 | 0.75 | 0.92 |
| I am satisfied with the quality of my interactions with the home spirometry staff | 58.3 | 56.4 | 0.78 | 0.92 |
| Programme evaluation | ||||
| I am satisfied with the home spirometry monitoring programme | 58.5 | 56.1 | 0.83 | 0.92 |
| Doing home spirometry makes me feel more secure in detecting problems with my lungs | 58.6 | 55.5 | 0.75 | 0.92 |
| Home spirometry allows me to stay better connected to my health-care providers | 58.7 | 55.3 | 0.72 | 0.92 |
| I would recommend using home spirometry to other patients | 58.5 | 55.1 | 0.80 | 0.92 |
The intra-class correlation for scores from the same 27 patients approximately 2.5 months apart was 0.77 for the total score; it was 0.66 for the equipment subscale, 0.67 for the personnel subscale (excluding question 9 on physicians’ interest in home monitoring results) and 0.50 for the programme evaluation subscale. The intra-class correlation for question 9 was 0.57. An intra-class correlation greater than 0.75 is considered to represent excellent agreement beyond chance and values less than 0.40 represent poor agreement. Values between 0.40 and 0.75 are considered to represent fair to good agreement.10
Psychometric properties
The RMSS has a scoring range of 15–75. If all responses were one (strongly disagree) the score would be 15, and if all responses were five (strongly agree) the score would be 75. There were no significant correlations between survey scores and time since transplant, frequency of home spirometry, age and education in each group. Only gender showed a significant difference in satisfaction scores (P = 0.029), with males tending to score higher than females in the survey development group, reflecting greater satisfaction.
Nurse triage study
Surveys were returned by 32 subjects (64% return rate). Survey responses for individual survey items ranged from 79% to 100% in the agree/strongly agree categories for all items, with two exceptions (see Table 3). Only 61% of subjects replied that they believed that their doctor was interested in reviewing their home spirometry tests. Replies to the staff-focused communication questions ranged from 90% to 100%, showing overall high levels of satisfaction regarding participant communications with programme nurses. Only 65% of respondents reported that the spirometer was reliable; 16% thought the spirometer was not reliable. This is in approximate agreement with staff observations regarding the number of device related problem calls that were handled during the study. Ninety percent of respondents were satisfied with the home monitoring programme and would recommend it to other patients if it were available as part of post-transplant care.
Table 3.
Survey results for nurse triage study group. The values are the % of respondents in each category. There were 28–31 respondents
| Question | Strongly disagree | Disagree | Neutral | Agree | Strongly agree |
|---|---|---|---|---|---|
| Spirometer concerns | |||||
| The spirometer is easy to use | 3 | 7 | 3 | 29 | 58 |
| The spirometer display is easy to read | 3 | 7 | 36 | 55 | |
| The spirometer is reliable and has few technical problems | 3 | 13 | 16 | 42 | 26 |
| I received adequate training in using my home spirometer | 32 | 68 | |||
| The spirometer gives me accurate test results | 3 | 13 | 45 | 39 | |
| If technical problems occur, the staff are quick to respond and fix the problems | 3 | 17 | 33 | 47 | |
| The amount of time it takes to complete my daily home spirometry is acceptable | 7 | 3 | 39 | 52 | |
| Communications/interactions | |||||
| The home monitoring staff are responsive to my questions and concerns | 26 | 74 | |||
| My doctors are interested in reviewing my home spirometry tests | 4 | 7 | 29 | 25 | 36 |
| I am satisfied with amount of communication I receive from the home monitoring staff | 3 | 3 | 29 | 65 | |
| I am satisfied with the quality of my interactions with the home spirometry staff | 10 | 23 | 68 | ||
| Programme evaluation | |||||
| I am satisfied with the home spirometry monitoring programme | 7 | 4 | 45 | 45 | |
| Doing home spirometry makes me feel more secure in detecting problems with my lungs | 7 | 7 | 31 | 55 | |
| Home spirometry allows me to stay better connected to my health-care providers | 4 | 17 | 38 | 41 | |
| I would recommend using home spirometry to other patients | 4 | 7 | 24 | 66 |
Discussion
Several lung transplant programmes have investigated using home spirometry to provide early indications of potential clinical problems in their lung recipients.11–14 Most use home spirometry when scheduled clinic visits or patient self-report indicates the onset of a deteriorating condition. In the nurse triage clinical trial, only 61% of participants believed that their doctor was interested in reviewing their home spirometry tests. This may affect the future feasibility of home monitoring programmes. Patient perceived indifference of their health-care providers to their clinical information has the potential to adversely influence a patient’s adherence to the data collection and transmission processes on which home monitoring depends. This was in contrast to participants’ perceptions regarding interactions with the home spirometry staff, primarily the project research nurses. The study nurses were in regular contact with study participants, which was reflected in a high satisfaction rating for study related communications. This level of communication between nurses and patients may not occur as consistently in clinic settings. In a discussion on facilitators and barriers to adherence with a very similar home monitoring protocol, Sabati et al. reported that encouragement by the patients’ health-care professionals was an important facilitator of adherence.15
Only 65% of respondents rated their spirometer as reliable and some (16%) thought their spirometers were not reliable. The difficulties included actual spirometer component problems such as screen, power supply and data transmission failures, as well as telephone line and data centre problems not directly related to the spirometer. In most cases the problems were resolved during a telephone conversation with the participant, or by replacing the equipment. Despite rapid resolution, such inconveniences were generally viewed as a system failure. Sabati et al. reported that non-adherent subjects in their study noted that both slow replacement of supplies and device problems prompted them to stop using their units.15 Device failures may also undermine patients’ broader confidence in their clinical care.
Earlier estimates of patient satisfaction with telehealth (e.g. videoconferencing, virtual visits, and remote monitoring) were high, frequently in the 80–100% range.1,16,17 A variety of problems in telemonitoring programmes have the potential to lead to poorer satisfaction. Some of the major patient concerns with telehealth programmes and remote monitoring include privacy and confidentiality, lack of face-to-face contact and technical concerns.18,19 Studies have shown that low patient satisfaction with health-care services reduces adherence with medical advice, service utilization and the relationship between patient and provider.20,21 It is likely that patient satisfaction is important in contributing to the overall success of home monitoring for lung transplant recipients. The judicious use of satisfaction surveys in the course of home monitoring programmes may serve as a screening mechanism to indicate when problems are emerging.
Despite the positive satisfaction scores for telehealth reported in the literature, there are concerns about the overall generalizability of the findings.2,17,22,23 These concerns include study sample sizes, subject selection, psychometric testing of the satisfaction instruments and study duration. The nature of our study and the limited sample size in the nurse triage study also limits the generalizability of our findings. Potential bias in subject selection was not a problem since all transplant recipients who were able and willing were candidates for study participation. Many studies collect satisfaction data at the study onset and after a short study period, thus confounding the sense of satisfaction with the novelty of using a new device/system not yet available to the general public. This should not be a concern in the nurse triage study, where subjects used the monitoring system for more than two years before satisfaction was measured at each subject’s conclusion of study participation. This survey may also be useful in studies of patients with other respiratory problems, who may require regular frequent pulmonary monitoring. The availability of a new valid and reliable satisfaction instrument focused on patient-dependent remote monitoring applications may also be useful for assessing other remote monitoring devices.
Acknowledgments
The study was partly funded by NIH grant R01 NR009212.
References
- 1.Demiris G, Speedie S, Finkelstein S. A questionnaire for the assessment of patients’ impressions of the risks and benefits of home telecare. J Telemed Telecare. 2000;6:278–84. doi: 10.1258/1357633001935914. [DOI] [PubMed] [Google Scholar]
- 2.Bakken S, Grullon-Figueroa L, Izquierdo R, et al. Development, validation, and use of English and Spanish versions of the telemedicine satisfaction and usefulness questionnaire. J Am Med Inform Assoc. 2006;13:660–7. doi: 10.1197/jamia.M2146. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Yip MP, Chang AM, Chan J, MacKenzie AE. Development of the Telemedicine Satisfaction Questionnaire to evaluate patient satisfaction with telemedicine: a preliminary study. J Telemed Telecare. 2003;9:46–50. doi: 10.1258/135763303321159693. [DOI] [PubMed] [Google Scholar]
- 4.De Vito Dabbs A, Hoffman LA, Dauber JH, Zullo T, Iacono AT. Evaluating the reliability and validity of the Questionnaire for Lung Transplant Patients. Prog Transplant. 2002;12:191–8. doi: 10.1177/152692480201200307. [DOI] [PubMed] [Google Scholar]
- 5.Demiris G. Principles of survey development for telemedicine applications. J Telemed Telecare. 2006;12:111–5. doi: 10.1258/135763306776738549. [DOI] [PubMed] [Google Scholar]
- 6.Wakefield B, Lowery J, Kobb R, Clements E, Brady M. Development of an item bank for telehealth evaluation. Telemed J E Health. 2002;8:268. [Google Scholar]
- 7.Britton BP. First home telehealth clinical guidelines developed by the American Telemedicine Association. Home Healthc Nurse. 2003;21:703–6. doi: 10.1097/00004045-200310000-00014. [DOI] [PubMed] [Google Scholar]
- 8.Marshall GN, Hays RD. The Patient Satisfaction Questionnaire Short Form, PSQ 18. Santa Monica, CA: RAND Distribution Services; 1994. [Google Scholar]
- 9.Flesh R. The Art of Readable Writing. New York: Harper and Row; 1974. [Google Scholar]
- 10.Fleiss JL. Statistical Methods for Rates and Proportions. 2nd. New York: John Wiley and Sons; 1981. [Google Scholar]
- 11.Morlion B, Knoop C, Paiva M, Estenne M. Internet-based home monitoring of pulmonary function after lung transplantation. Am J Respir Crit Care Med. 2002;165:694–7. doi: 10.1164/ajrccm.165.5.2107059. [DOI] [PubMed] [Google Scholar]
- 12.Bjørtuft O, Johansen B, Boe J, Foerster A, Holter E, Geiran O. Daily home spirometry facilitates early detection of rejection in single lung transplant recipients with emphysema. Eur Respir J. 1993;6:705–8. [PubMed] [Google Scholar]
- 13.Finkelstein SM, Snyder M, Stibbe CE, et al. Staging of bronchiolitis obliterans syndrome using home spirometry. Chest. 1999;116:120–6. doi: 10.1378/chest.116.1.120. [DOI] [PubMed] [Google Scholar]
- 14.Kugler C, Fuehner T, Dierich M, et al. Effect of adherence to home spirometry on bronchiolitis obliterans and graft survival after lung transplantation. Transplantation. 2009;88:129–34. doi: 10.1097/TP.0b013e3181aad129. [DOI] [PubMed] [Google Scholar]
- 15.Sabati N, Snyder M, Edin-Stibbe C, Lindgren B, Finkelstein S. Facilitators and barriers to adherence with home monitoring using electronic spirometry. AACN Clin Issues. 2001;12:178–85. doi: 10.1097/00044067-200105000-00002. [DOI] [PubMed] [Google Scholar]
- 16.Gustke SS, Balch DC, West VL, Rogers LO. Patient satisfaction with telemedicine. Telemed J. 2000;6:5–13. [Google Scholar]
- 17.Williams TL, May CR, Esmail A. Limitations of patient satisfaction studies in telehealthcare: a systematic review of the literature. Telemed J E Health. 2001;7:293–316. doi: 10.1089/15305620152814700. [DOI] [PubMed] [Google Scholar]
- 18.Dick PT, Filler R, Pavan A. Participant satisfaction and comfort with multidisciplinary pediatric telemedicine consultations. J Pediatr Surg. 1999;34:137–41. doi: 10.1016/s0022-3468(99)90244-0. [DOI] [PubMed] [Google Scholar]
- 19.Whitten P, Collins B, Mair F. Nurse and patient reactions to a developmental home telecare system. J Telemed Telecare. 1998;4:152–60. doi: 10.1258/1357633981932136. [DOI] [PubMed] [Google Scholar]
- 20.Kahan B, Goodstadt M. The Interactive Domain Model of best practices in health promotion: developing and implementing a best practices approach to health promotion. Health Promot Pract. 2001;2:43–67. [Google Scholar]
- 21.Hjortdahl P, Laerum E. Continuity of care in general practice: effect on patient satisfaction. BMJ. 1992;304:1287–90. doi: 10.1136/bmj.304.6837.1287. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Whitten P, Love B. Patient and provider satisfaction with the use of telemedicine: overview and rationale for cautious enthusiasm. J Postgrad Med. 2005;51:294–300. [PubMed] [Google Scholar]
- 23.Mair F, Whitten P. Systematic review of studies of patient satisfaction with telemedicine. BMJ. 2000;320:1517–20. doi: 10.1136/bmj.320.7248.1517. [DOI] [PMC free article] [PubMed] [Google Scholar]