Abstract
Background
Questionnaires are commonly used tools in telemedicine services that can help to evaluate different aspects. Selecting the ideal questionnaire for this purpose may be challenging for researchers. This study aims to review which questionnaires are used to evaluate telemedicine services in the studies, which are most common, and what aspects of telemedicine evaluation do they capture.
Methods
The PubMed database was searched in August 2020 to retrieve articles. Data extracted from the final list of articles included author/year of publication, journal of publication, type of evaluation, and evaluation questionnaire. Data were analyzed using descriptive statistics.
Results
Fifty-three articles were included in this study. The questionnaire was used for evaluating the satisfaction (49%), usability (34%), acceptance (11.5%), and implementation (2%) of telemedicine services. Among telemedicine specific questionnaires, Telehealth Usability Questionnaire (TUQ) (19%), Telemedicine Satisfaction Questionnaire (TSQ) (13%), and Service User Technology Acceptability Questionnaire (SUTAQ) (5.5%), were respectively most frequently used in the collected articles. Other most used questionnaires generally used for evaluating the users’ satisfaction, usability, and acceptance of technology were Client Satisfaction Questionnaire (CSQ) (5.5%), Questionnaire for User Interaction Satisfaction (QUIS) (5.5%), System Usability Scale (SUS) (5.5%), Patient Satisfaction Questionnaire (PSQ) (5.5%), and Technology Acceptance Model (TAM) (3.5%) respectively.
Conclusion
Employing specifically designed questionnaires or designing a new questionnaire with fewer questions and more comprehensiveness in terms of the issues studied provides a better evaluation. Attention to user needs, end-user acceptance, and implementation processes, along with users' satisfaction and usability evaluation, may optimize telemedicine efforts in the future.
Keywords: Telemedicine, Evaluation, Questionnaire
Background
Telemedicine provides healthcare services when patients and healthcare providers are at different locations using Information and Communication Technologies (ICT). The core purpose of telemedicine is to improve the health of individuals and communities by exchanging useful information for various goals, such as preventing diseases, for diagnosis, monitoring, providing treatment, educating healthcare providers, and for conducting research [1, 2].
It is essential to identify the limitations, find approaches to overcome them, and create a reasonable structure to implement and use a telemedicine service successfully. Therefore, precise evaluations of telemedicine services are required [3]. Evaluation is the use of systematic and logical methods to assess the attributes of different aspects of the project, including its design, implementation, operation, and outcome [4]. The telemedicine evaluation process should be done independently and systematically on various features as the feasibility of the project, acceptance by participants, availability of service, technical capabilities of participants, clinical outcomes, user satisfaction, quality, and the cost–benefit of the offered service [3].
There are several methods to evaluate different aspects of a telemedicine project. In order to choose the right evaluation method, several issues should be considered: (1) Considering project goals; (2) determining the required amount of budget, energy, and time for evaluation; (3) selecting the appropriate method based on the predetermined criteria and metrics; (4) using understandable and easy evaluation methods for users; (5) being completely aware of the evaluation method used in the study; and (6) using validated methods for the evaluation process [5].
In the field of telemedicine, the most commonly used tools for evaluating user satisfaction are questionnaires and interviews [6]; and in order to assess the usability of telemedicine systems, usually questionnaires, interviews, observations, self-descriptive, and logging methods are used [7]. The clinical outcomes of telemedicine services are evaluated by means of biometric measurements, quality of life, and disease-specific tools that are all generally questionnaires [8].
Researchers use a variety of questionnaires for various purposes. Some of them are valid questionnaires specified for telemedicine whereas some others are more general. Due to the great variety of questionnaires, it is difficult for researchers to choose the best tool to evaluate their telemedicine services. Therefore, this review aims at listing the most commonly used questionnaires for evaluating telemedicine, which would in turn help researchers to select the most appropriate questionnaire, based on their objectives, to evaluate telemedicine services.
Methods
Database and date
This is a scoping review. PubMed database was searched in August 2020 to retrieve articles without date limitation. Approximately, 80–90% of studies conducted in Telemedicine were accessible on PubMed, which sufficed for our purpose; therefore, no other database was used [9].
Search strategy
The following combinations of keywords were used to find relevant articles in the Title/Abstract search field: (telemedicine) AND (evaluation OR assessment) AND (questionnaire).
Inclusion criteria
Original observational and interventional research articles in which a valid and referenced questionnaire was used to evaluate telemedicine services and systems were included in this study.
Exclusion criteria
Articles were excluded if they were review articles, non-English language, without full text, not specifically addressing telemedicine, did not have communication between patients and healthcare providers in mHealth, evaluated clinical outcomes, evaluated acceptance of telemedicine prior to implementation, did not include details about the questionnaires used, questionnaires that have not been validated, or combined questionnaires.
Article selection
First, all retrieved articles were assessed based on title and abstract by one researcher (SH). Next, the same researcher reviewed the full-text of the selected articles. When necessary, the second researcher (K.B), being a telemedicine professional, provided consult. Manual searching was also performed to find additional articles that had used specific evaluation questionnaires to investigate Telemedicine. Finally, a list of all included articles was prepared.
Data extraction
The following information was extracted from included articles: author, year of publication, journal of publication, evaluation type, and evaluation questionnaire.
Data analysis
Descriptive statistics (frequency and frequency percent) was used to analyze data.
Results
PubMed database found 214 articles. After removing review articles and those written in a language other than English, 208 articles remained whose titles and abstracts were screened. After excluding 95 articles, the full-text of the 113 remaining articles were reviewed. Finally, 53 articles were included in this study (Fig. 1), from which the required data was extracted (Table 1).
Table 1.
Author and year | Journal of publication | Evaluation type | Evaluation questionnaire |
---|---|---|---|
Layfield et al. [10] 2020 | Head & Neck | Usability | Telehealth Usability Questionnaire (TUQ) [18] |
Mostafa et al. [11] 2020 | Journal of Dermatological Treatment | Usability | Telehealth Usability Questionnaire (TUQ) [18] |
Whitehouse et al. [12] 2020 | Research in Gerontological Nursing | Usability | Telehealth Usability Questionnaire (TUQ) [18] |
Vaughan et al. [13] 2020 | Telemedicine and e-health | Usability | Telehealth Usability Questionnaire (TUQ) [18] |
Cheng et al. [14] 2020 | JAAOS Global Research & Reviews |
Satisfaction, Usability |
Telehealth Satisfaction Scale (TeSS) [63] and Telehealth Usability Questionnaire (TUQ) [18] |
Mehra et al. [48] 2020 | Journal of Medical Internet Research | Usability | Usefulness, Satisfaction, and Ease of use (USE) questionnaire [64] |
Lin et al. [20] 2020 | Journal of Medical Internet Research | Satisfaction | Modified Telemedicine Satisfaction Questionnaire (TSQ) [26] |
Christensen et al. [31] 2020 | Telemedicine and e-health | Satisfaction | Client Satisfaction Questionnaire 3 (CSQ 3) [65] |
Christensen et al. [30] 2020 | Telemedicine and e-health | Satisfaction | Client Satisfaction Questionnaire 8 (CSQ 8) [65] |
Leppert et al. [49] 2020 | Clinical research in cardiology | Acceptance | Florida Patient Acceptance Survey (FPAS) [66] |
McGloin et al. [41]2020 | Journal of Medical Internet Research | Satisfaction | Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) [42] |
Talal et al. [21] 2019 | Telemedicine and e-health | Satisfaction | Modified Telemedicine Satisfaction Questionnaire (TSQ)[26] |
Le et al. [22] 2019 | Digestive Diseases and Sciences | Satisfaction | Patient Satisfaction Questionnaire (PSQ) [67] and Telemedicine Satisfaction Questionnaire (TSQ)[26] |
Serwe et al. [15] 2018 | International Journal of Telerehabilitation | Usability | Telehealth Usability Questionnaire (TUQ) [18] |
van der Meij et al. [50] 2018 | Journal of Medical Internet Research | Process | Model of Linnan and Steckler [68] |
Safdari et al. [33] 2018 | Journal of diabetes and metabolic disorders | Usability | Questionnaire for User Interaction Satisfaction (QUIS) [69] |
Losiouk et al. [23] 2018 | Journal of Telemedicine and Telecare | Satisfaction | Modified Telemedicine Satisfaction Questionnaire (TSQ) [26] |
Host et al. [51] 2018 | Clinical & experimental optometry | Satisfaction | Modified patient satisfaction with videoconferencing for specialty consultation questionnaire [70] |
Hosseini et al. [34] 2018 | The open medical informatics journal | Usability | Questionnaire for User Interaction Satisfaction (QUIS) [69] |
Hatton et al. [43] 2018 | Journal of pharmacy practice | Satisfaction | Modified Patient Assessment of Communication during Telemedicine (PACT) questionnaire [44] |
D'Hooghe et al. [52] 2018 | Multiple sclerosis and related disorders | Satisfaction | Dutch version of Quebec User Evaluation of Satisfaction with Assistive Technology (D-QUEST 2.0) [71] |
Ammenwerth et al. [53] 2018 | JMIR cardio | Satisfaction | Delone and McLean Information System Success Model [72] |
Torbjørnsen et al. [27] 2018 | JMIR human factors | Acceptance | Service User Technology Acceptability Questionnaire (SUTAQ) [29] |
van Rosmalen-Nooijens et al. [62] 2017 | Journal of Medical Internet Research | Acceptance | Web Evaluation Questionnaire (WEQ) [73] |
Segura-Sampedro et al. [24] 2017 | Annals of medicine and surgery | Satisfaction | Telemedicine Satisfaction Questionnaire (TSQ) [26] |
Oliveira et al. [54] 2017 | JMIR Med Education | Satisfaction | Wang’s an e-learning satisfaction model [74] |
Agnisarman et al. [46] 2017 | Applied ergonomics | Usability, Acceptance | IBM Computer System Usability Questionnaire (CSUQ) [75] and the NASA Task Load Index test [76] and Modified Technology Acceptance Model (TAM) [77] |
Serwe et al. [16] 2017 | International Journal of Telerehabilitation | Usability | Telehealth Usability Questionnaire (TUQ) [18] |
Yu et al. [17] 2017 | Disability and Rehabilitation: Assistive Technology | Usability | Telehealth Usability Questionnaire (TUQ) [18] |
Parmanto et al. [18] 2016 | International Journal of Telerehabilitation | Usability | Telehealth Usability Questionnaire (TUQ) [18] |
Smaradottir et al. [36] 2016 | Journal of Telemedicine and Telecare | Usability | System Usability Scale (SUS) questionnaire [78] |
Fields et al. [32] 2016 | Sleep | Satisfaction | Client Satisfaction Questionnaire 8 (CSQ 8) [65] |
Dario et al. [28] 2016 | International journal of integrated care | Acceptance | Service User Technology Acceptability Questionnaire (SUTAQ) [29] |
Hirani et al. [29] 2016 | Journal of Telemedicine and Telecare | Acceptance | Service User Technology Acceptability Questionnaire (SUTAQ) [29] |
Alanzi et al. [35] 2016 | JMIR research protocols | Usability | Questionnaire for User Interaction Satisfaction (QUIS) [69] |
Roberts et al. [55] 2015 | The Australian journal of rural health | Satisfaction | The questionnaire based on a validated instrument used in teledermatology [79] |
Poulsen et al. [56] 2015 | International journal of rheumatic diseases | Satisfaction | Questionnaire used to evaluate a similar medical oncology telemedicine service [80] |
Ligons et al. [37] 2014 | International journal of medical informatics | Usability | System Usability Scale (SUS) questionnaire [78] |
Lacerda et al. [38] 2014 | Journal of biomedical informatics | Usability | System Usability Scale (SUS) questionnaire [78] |
Vélez et al. [57] 2014 | Journal of midwifery & women's health | Usability | Health-Information Technology Usability Survey (Health-ITUES) [81] |
Penteado et al. [58] 2014 | JMIR medical informatics | Satisfaction | Satisfaction with Amplification in Daily Life (SADL) [82] |
Kwon et al. [45] 2014 | Telemedicine and e-health | Usability | Modified Post Study System Usability Questionnaire (PSSUQ) [83] |
McFarland et al. [39] 2013 | Telemedicine and e-health | Satisfaction | Modified Ware et al. ‘s Patient Satisfaction Questionnaire (PSQ) [84] |
Parra et al. [47] 2012 | Interactive journal of medical research | Acceptance | Modified Technology Acceptance Model (TAM) [77] |
Schutte et al. [19] 2012 | International Journal of Telerehabilitation | Usability | The After- Scenario Questionnaire (ASQ) [85] and The Post-Study System Usability Questionnaire (PSSUQ) [83] and Telehealth Usability Questionnaire (TUQ) [18] |
Dechêne et al. [25] 2011 | International Journal of Telerehabilitation | Satisfaction | Telemedicine Satisfaction Questionnaire (TSQ) [26] |
Scalvini et al. [59] 2009 | Telemedicine and e-health | Satisfaction | NHS patient questionnaire [86] |
Agha et al. [44] 2009 | Telemedicine and e-health | Satisfaction | Patient Assessment of Communication during Telemedicine (PACT) questionnaire [44] |
Bakken et al. [42] 2006 | Journal of the American Medical Informatics Association | Satisfaction | Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) [42] |
Kim et al. [60] 2004 | Telemedicine and e-health | Satisfaction | Patient attitudes and satisfaction questionnaire [87] |
Eminovic et al. [61] 2004 | Journal of Medical Internet Research | Satisfaction | Modified Telemedicine Perception Questionnaire (TMPQ) [88] |
Yip et al. [26] 2003 | Journal of Telemedicine and Telecare | Satisfaction | Telemedicine Satisfaction Questionnaire (TSQ) [26] |
Wallace et al. [40] 2002 | BMC family practice | Satisfaction | Ware et al. ‘s Patient Satisfaction Questionnaire (PSQ) [84] |
Journal of publication
Nine articles were published in the Telemedicine and e-health Journal (17%), six in the Journal of Medical Internet Research (11.5%), four in the International Journal of Telerehabilitation (7.5%), four in the Journal of Telemedicine and Telecare (7.5%), and the remaining articles were published in other journals (n = 30, 56.5%).
Evaluation type
A questionnaire was used in 26 articles (49%) to evaluate patients’ or physicians’ satisfaction; in 18 articles to evaluate system usability (34%), in 6 articles to evaluate acceptance (11.5%), and in one article to evaluate the implementation process (2%). In 2 articles a questionnaire was used to evaluate the usability and the acceptance or satisfaction of individuals (3.5%).
Evaluation questionnaire
The final list of articles showed that the Telehealth Usability Questionnaire (TUQ) (n = 10, 19%) [10–19] and Telemedicine Satisfaction Questionnaire (TSQ) (n = 7, 13%) [20–26] were the most commonly used. Then, the Service User Technology Acceptability Questionnaire (SUTAQ) (n = 3, 5.5%) [27–29], the Client Satisfaction Questionnaire (CSQ) (n = 3, 5.5%) [30–32], the Questionnaire for User Interaction Satisfaction (QUIS) (n = 3, 5.5%) [33–35], the System Usability Scale (SUS) questionnaire (n = 3, 5.5%) [36–38], the Patient Satisfaction Questionnaire (PSQ) (n = 3, 5.5%) [22, 39, 40], the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) (n = 2, 3.5%) [41, 42], the Patient Assessment of Communication during Telemedicine (PACT) questionnaire (n = 2, 3.5%) [43, 44], the Post Study System Usability Questionnaire (PSSUQ) (n = 2, 3.5%) [19, 45] and the Technology Acceptance Model (TAM) (n = 2, 3.5%) [46, 47] were the most used questionnaires, respectively. The rest of the articles had used other questionnaires (n = 18, 34%) [14, 19, 46, 48–62] and 4 articles had used more than one questionnaire.
The most used questionnaires
Out of 59 used questionnaires in the studies, the most frequently (more than two times) used were TUQ, TSQ, SUTAQ, CSQ, QUIS, SUS, PSQ, TSUQ, PACT, PSSUQ, and TAM, which have been shown in Fig. 2.
Some information about these questionnaires are presented below:
Parmanto et al. in 2016, formally introduced TUQ. This questionnaire evaluates the usability of telehealth services. It has 21 items that are based on 6 criteria including usefulness (3 items), ease of use and learnability (3 items), interface quality (4 items), interaction quality (4 items), reliability (3 items), and satisfaction and future use (4 items) [18]. The TUQ was designed using existing telehealth and general usability questionnaires and includes all usability aspects.
The TSQ was developed by Yip et al. in 2003 for evaluating patient satisfaction in using telemedicine. This questionnaire has 14 items with no categories [26].
Hirani et al. presented SUTAQ in 2017, as a tool to judge the acceptability of telehealth services by investigating user opinions. This questionnaire consists of 22 items that is categorized in six sectors: benefits (9 items), privacy (4 items), personal care skill (3 items), substitution (3 items), and satisfaction (3 items) [29].
Bakken et al. in 2006, designed TSUQ. This questionnaire evaluates the satisfaction and usefulness criteria of video consultation and telephone monitoring. This instrument has 26 items in two satisfaction (21 items) and usefulness (5 items) sections [42].
PACT questionnaire was introduced by Agha et al. in 2009. This questionnaire measures the satisfaction of patients and physicians of their communication by asking 33 questions. These questions are asked about the quality of physician and patient communication (16 questions), clinical skills (9 questions), interpersonal skills (6 questions), and comfortable of visit (2 questions) sectors [44].
Attkisson et al. in 1982, firstly introduced CSQ. This questionnaire evaluates the general users’ satisfaction with different services. The original version of CSQ has 18 items; however, the questionnaire has other versions in which there are only 3 or 8 items [65].
QUIS was developed by Chin et al. in 1988. It is a usability testing instrument that has been designed to measure the satisfaction of users about their interaction with the computer interface. QUIS consists of 27 items in five sections. The first section measures the overall satisfaction, and the four others measure user satisfaction regarding screen, terminology and information, learning, and system capabilities aspects [69].
The SUS questionnaire was created by Brooke et al. in 1986 for testing the usability of electronic systems. This tool has ten items with no specific categories [78].
The PSQ was firstly designed by Ware et al. in 1983 for measuring the satisfaction of patients with using medical cares. The original version of this questionnaire has 80 items measured in seven categories, such as overall satisfaction, technical quality, interpersonal mode, communication, financial features, time spent with physician, availability, and convenience [84]. A shorter version of this questionnaire also exists which consists of 18 items [67].
PSSUQ was developed by the International Business Machines Corporation (IBM) to evaluate users’ satisfaction at the end of the study. This questionnaire has three versions. The last version consists of 16 questions with seven Likert scales. These questions are arranged is such way to evaluate matters in the three following sections: system usefulness, information quality, and interface quality [75, 83].
TAM is a model that shows how users use and accept a technology. A questionnaire that has been developed based on this model was presented by Davis in 1989. This questionnaire investigates 12 items that are generally concerned with two matters: the perceived usefulness and perceived ease of use sections with 12 items [77].
Discussion
The TUQ, TSQ, SUTAQ were the three most common telemedicine-specific questionnaires used in the retrieved articles, respectively. The four most used general questionnaires related to satisfaction and usability evaluation were the CSQ, QUIS, SUS, and PSQ.
In the below sections it will be discussed why each questionnaire is used more frequently. To do this we chose the most three used questionnaires which are designed specific for telemedicine and four general questionnaires that are used in telemedicine evaluation as well.
TUQ is usable for evaluating different types of telehealth systems such as sole target videoconferencing systems, computer and mobile-based systems, and collecting the opinions of both patients and physicians [18]. As TUQ is considerably comprehensive when comparing with other questionnaires such as QUIS and SUS, it is most frequently used for evaluating the usability of telemedicine systems. SUS questionnaire, like QUIS, is a general usability evaluation tool that is also used to evaluate telemedicine systems; yet, unlike QUIS it has no segmentation and it examines fewer items [78]. Also, ever since TUQ has been introduced, SUS has been seldom used.
The TSQ is a preliminary tool for measuring patient satisfaction with telemedicine, and it is used quite frequently. This might be due to the fact that TSQ covers various satisfaction factors such as the quality of care, quality of virtual visits, interpersonal interactions, and also it has fewer number of items [26].
Although TSUQ was also introduced many years ago, it has rarely been used for studies conducted in telemedicine, which may be due to two reasons: (1) it has been designed specifically for telemedicine services provided to diabetes patients, and (2) it investigates more items than other questionnaires especially, TSQ [42].
The CSQ was a generic most used questionnaire for evaluating users’ satisfaction with the telemedicine services. This may be due to the fact that CSQ measures the quality of diverse attributes such as the physical environment, procedure, assistance staff, type of service, treatment staff, amount or length of service, service quality, outcome, and general satisfaction with few items [65]. PSQ is a general questionnaire as well and it is designed to evaluate patient satisfaction. However, with presenting the TSQ, evaluators preferred to use TSQ as a specific questionnaire in assessing the users’ satisfaction of a telemedicine service. Evaluators may use different tools depending on their purpose of investigation, and the only thing that matters is the validity of the used tool.
The SUTAQ is the only questionnaire that specifically designed to gather patients’ opinions about the acceptability of the telehealth systems [29]. While there are various models for assessing technology acceptance, this instrument is used more in this field since it has been specifically designed for evaluating telehealth acceptance.
In terms of evaluation types for telemedicine services, users’ satisfaction, usability, acceptance, and implementation process are non-clinical aspects that have been evaluated by use of validated questionnaires. Approximately, a questionnaire had been used on half of the collected articles for evaluating the satisfaction of telemedicine users; and approximately in a third of the articles, the usability of telemedicine systems was evaluated.
Telemedicine acceptance [27–29, 46, 47, 49, 62] and its implementation process [50] were evaluated in only a few articles. A review study investigating the evaluation methods for telemedicine services in hospitals showed that telemedicine users' satisfaction more frequently evaluated than clinical and economical aspects, and the most commonly used method to evaluate satisfaction was questionnaire. Similar to our result, this study also showed that the development and implementation process of telemedicine had gained less attention [6].
Evaluating the implementation process could be an essential stage for the successful usage of telemedicine services due to showing obstacles and facilitators of implementation [50]. Also, considering the needs of the users and planning the process based on these needs may affect the successful implementation of telemedicine services and increase the rate of their acceptance. Therefore, we recommend that researchers pay more attention to this aspect in the evaluation of telemedicine services. Another study that reviewed the usability evaluation methods of eHealth services for patients who had HIV revealed that questionnaire was the most employed method for evaluation [89].
Based on the researchers’ knowledge, this is the first review study that has identified validated and the most used questionnaires in evaluating telemedicine services. Nevertheless, this study has its own limitations. Only PubMed database was used for searching and retrieving articles. In the search strategy, we used telemedicine as a MeSH term and evaluation and assessment keywords that may use in different grammatical style. Moreover, we restricted our search to the Title/Abstract field. These issues may cause some articles missed from our study. For this reason, we conducted a manual search using the name of telemedicine specific questionnaires in the PubMed database and added additional articles.
Conclusion
Many questionnaires were used to assess telemedicine services. Some of them were specifically designed to evaluate telemedicine services while others were more general. As the results of this study showed, telemedicine service evaluators should use questionnaires specifically designed for telemedicine to assess its various aspects. However, if an evaluator wants to design a questionnaire for evaluating a telemedicine service, it is better to pay attention to goal-based design, the number of questions, and comprehensiveness in terms of the issues studied.
Users' satisfaction with telemedicine services and the usability of the system have been two of the most frequently investigated issues in telemedicine when comparing with other existing issues in the field. Attention to user needs, end-user acceptance, and implementation processes may optimize telemedicine efforts in the future.
Acknowledgements
We would like to express our gratitude to the Institute for Futures Studies in Health of Kerman University of Medical Sciences for providing the research environment.
Abbreviations
- ICT
Information and communication technologies
- TUQ
Telehealth usability questionnaire
- TeSS
Telehealth satisfaction scale
- USE
Usefulness, satisfaction, and ease of use
- TSQ
Telemedicine satisfaction questionnaire
- CSQ
Client satisfaction questionnaire
- FPAS
Florida patient acceptance survey
- TSUQ
Telemedicine satisfaction and usefulness questionnaire
- PSQ
Patient satisfaction questionnaire
- QUIS
Questionnaire for user interaction satisfaction
- PACT
Patient assessment of communication during telemedicine
- D-QUEST
Dutch version of Quebec user evaluation of satisfaction with assistive technology
- SUTAQ
Service user technology acceptability questionnaire
- WEQ
Web evaluation questionnaire
- CSUQ
Computer system usability questionnaire
- TAM
Technology acceptance model
- SUS
System usability scale
- Health-ITUES
Health-information technology usability survey
- SADL
Satisfaction with amplification in daily life
- PSSUQ
Post study system usability questionnaire
- ASQ
The after-scenario questionnaire
- TMPQ
Telemedicine perception questionnaire
Authors' contributions
SH and KB both contributed in designing the study. The selection and evaluation of the articles and data extraction were done by SH, and validated by KB. All authors participated in drafting the manuscript and approving the final version of the manuscript. All authors read and approved the final manuscript.
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Availability of data and materials
Not applicable.
Ethics approval and consent to participate
Not applicable.
Consent to publish
Not applicable.
Competing interests
The Authors declare that there is no competing interests.
Footnotes
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.WHO. Telemedicine, Telehealth, and Health Information Technology, 2006 [cited 2020 August 14]. https://www.who.int/goe/policies/countries/usa_support_tele.pdf?ua=1.
- 2.WHO. Telemedicine 2010 [cited 2020 August 14]. https://www.who.int/goe/publications/goe_telemedicine_2010.pdf.
- 3.Pan American Health Organization. Defining evaluation indicators for telemedicine as a tool for reducing health inequities: Study and results of a community of practice 2016 [cited 2020 August 14]. https://iris.paho.org/handle/10665.2/28562.
- 4.Ammenwerth E, Brender J, Nykänen P, Prokosch H-U, Rigby M, Talmon J. Visions and strategies to improve evaluation of health information systems: reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Informat. 2004;73(6):479–491. doi: 10.1016/j.ijmedinf.2004.04.004. [DOI] [PubMed] [Google Scholar]
- 5.McNair JB. Handbook of evaluation methods for health informatics. Amsterdam: Elsevier; 2006. [Google Scholar]
- 6.AlDossary S, Martin-Khan MG, Bradford NK, Smith ACJI. A systematic review of the methodologies used to evaluate telemedicine service initiatives in hospital facilities. Int J Med Informat. 2017;97:171–194. doi: 10.1016/j.ijmedinf.2016.10.012. [DOI] [PubMed] [Google Scholar]
- 7.Klaassen B, van Beijnum BJ, Hermens HJ. Usability in telemedicine systems—a literature survey. Int J Med Inf. 2016;93:57–69. doi: 10.1016/j.ijmedinf.2016.06.004. [DOI] [PubMed] [Google Scholar]
- 8.Kidholm K, Clemensen J, Caffery LJ, Smith AC. The Model for Assessment of Telemedicine (MAST): a scoping review of empirical studies. J Telemed Telecare. 2017;23(9):803–813. doi: 10.1177/1357633X17721815. [DOI] [PubMed] [Google Scholar]
- 9.Bahaadinbeigy K, Yogesan K, Wootton R. MEDLINE versus EMBASE and CINAHL for telemedicine searches. Telemed J E-health. 2010;16(8):916–919. doi: 10.1089/tmj.2010.0046. [DOI] [PubMed] [Google Scholar]
- 10.Layfield E, Triantafillou V, Prasad A, Deng J, Shanti RM, Newman JG et al. Telemedicine for head and neck ambulatory visits during COVID‐19: evaluating usability and patient satisfaction. Head Neck. 2020:1–9. [DOI] [PMC free article] [PubMed]
- 11.Mostafa PI, Hegazy AA. Dermatological consultations in the COVID-19 era: Is teledermatology the key to social distancing? An Egyptian experience. J Dermatol Treat. 2020:1–6. [DOI] [PubMed]
- 12.Whitehouse CR, Long JA, Maloney LM, Daniels K, Horowitz DA, Bowles KH. Feasibility of diabetes self-management telehealth education for older adults during transitions in care. Res Gerontol Nurs. 2020;13(3):138–145. doi: 10.3928/19404921-20191210-03. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Vaughan EM, Naik AD, Lewis CM, Foreyt JP, Samson SL, Hyman DJ. Telemedicine training and support for community health workers: improving knowledge of diabetes. Telemed J E-health. 2020;26(2):244–250. doi: 10.1089/tmj.2018.0313. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Cheng O, Law N-H, Tulk J, Hunter M. Utilization of telemedicine in addressing musculoskeletal care gap in long-term care patients. JAAOS Global Res Rev. 2020;4(4). [DOI] [PMC free article] [PubMed]
- 15.Serwe KM. The provider’s experience of delivering an education-based wellness program via telehealth. Int J Telerehabil. 2018;10(2):73. doi: 10.5195/IJT.2018.6268. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Serwe KM, Hersch GI, Pancheri KJ. Feasibility of using telehealth to deliver the “Powerful Tools for Caregivers” program. Int J Telerehabil. 2017;9(1):15. doi: 10.5195/ijt.2017.6214. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Yu DX, Parmanto B, Dicianno BE, Watzlaf VJ, Seelman KD. Accessibility needs and challenges of a mHealth system for patients with dexterity impairments. Disabil Rehabil Assist Technol. 2017;12(1):56–64. doi: 10.3109/17483107.2015.1063171. [DOI] [PubMed] [Google Scholar]
- 18.Parmanto B, Lewis AN, Jr, Graham KM, Bertolet MH. Development of the telehealth usability questionnaire (TUQ) Int J Telerehabil. 2016;8(1):3. doi: 10.5195/ijt.2016.6196. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Schutte J, Gales S, Filippone A, Saptono A, Parmanto B, McCue M. Evaluation of a telerehabilitation system for community-based rehabilitation. Int J Telerehabil. 2012;4(1):15. doi: 10.5195/ijt.2012.6092. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Lin CH, Tseng WP, Wu JL, Tay J, Cheng MT, Ong HN, et al. A double triage and telemedicine protocol to optimize infection control in an emergency department in Taiwan during the COVID-19 pandemic: retrospective feasibility study. J Med Internet Res. 2020;22(6):e20586. doi: 10.2196/20586. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Talal AH, McLeod A, Andrews P, Nieves-McGrath H, Chen Y, Reynolds A, et al. Patient reaction to telemedicine for clinical management of hepatitis C virus integrated into an opioid treatment program. Telemed J E-Health. 2019;25(9):791–801. doi: 10.1089/tmj.2018.0161. [DOI] [PubMed] [Google Scholar]
- 22.Le LB, Rahal HK, Viramontes MR, Meneses KG, Dong TS, Saab S. Patient satisfaction and healthcare utilization using telemedicine in liver transplant recipients. Dig Dis Sci. 2019;64(5):1150–1157. doi: 10.1007/s10620-018-5397-5. [DOI] [PubMed] [Google Scholar]
- 23.Losiouk E, Lanzola G, Del Favero S, Boscari F, Messori M, Rabbone I, et al. Parental evaluation of a telemonitoring service for children with type 1 diabetes. J Telemed Telecare. 2018;24(3):230–237. doi: 10.1177/1357633X17695172. [DOI] [PubMed] [Google Scholar]
- 24.Segura-Sampedro JJ, Rivero-Belenchón I, Pino-Díaz V, Rodríguez Sánchez MC, Pareja-Ciuró F, Padillo-Ruiz J, et al. Feasibility and safety of surgical wound remote follow-up by smart phone in appendectomy: a pilot study. Ann Med Surgery. 2012;2017(21):58–62. doi: 10.1016/j.amsu.2017.07.040. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Dechêne L, Tousignant M, Boissy P, Macoir J, Héroux S, Hamel M, et al. Simulated in-home teletreatment for anomia. Int J Telerehabil. 2011;3(2):3. doi: 10.5195/IJT.2011.6075. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Yip MP, Chang AM, Chan J, MacKenzie AE. Development of the telemedicine satisfaction questionnaire to evaluate patient satisfaction with telemedicine: a preliminary study. J Telemed Telecare. 2003;9(1):46–50. doi: 10.1258/135763303321159693. [DOI] [PubMed] [Google Scholar]
- 27.Torbjørnsen A, Småstuen MC, Jenum AK, Årsand E, Ribu L. The service user technology acceptability questionnaire: psychometric evaluation of the norwegian version. JMIR Human Factors. 2018;5(4):e10255. doi: 10.2196/10255. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Dario C, Luisotto E, Dal Pozzo E, Mancin S, Aletras V, Newman S, et al. Assessment of patients' perception of telemedicine services using the service user technology acceptability questionnaire. Int J Integrated Care. 2016;16(2):13. doi: 10.5334/ijic.2219. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Hirani SP, Rixon L, Beynon M, Cartwright M, Cleanthous S, Selva A, et al. Quantifying beliefs regarding telehealth: development of the whole systems demonstrator service user technology acceptability questionnaire. J Telemed Telecare. 2017;23(4):460–469. doi: 10.1177/1357633X16649531. [DOI] [PubMed] [Google Scholar]
- 30.Christensen LF, Gildberg FA, Sibbersen C, Skjoeth MM, Nielsen CT, Hansen JP. Videoconferences and treatment of depression: satisfaction score correlated with number of sessions attended but not with age. Telemed J E-Health. 2020;26(7):898–904. doi: 10.1089/tmj.2019.0129. [DOI] [PubMed] [Google Scholar]
- 31.Christensen LF, Gildberg FA, Sibbersen C, Skjoeth MM, Nielsen CT, Hansen JP. Disagreement in satisfaction between patients and providers in the use of videoconferences by depressed adults. Telemed J E-health. 2020;26(5):614–620. doi: 10.1089/tmj.2019.0055. [DOI] [PubMed] [Google Scholar]
- 32.Fields BG, Behari PP, McCloskey S, True G, Richardson D, Thomasson A, et al. Remote ambulatory management of veterans with obstructive sleep apnea. Sleep. 2016;39(3):501–509. doi: 10.5665/sleep.5514. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Safdari R, Langarizadeh M, Ramezani A, Khodaveisi T, Nejad AF. Development of a store-and-forward telescreening system of diabetic retinopathy: lessons learned from Iran. J Diabetes Metabolic Disorders. 2018;17(1):31–36. doi: 10.1007/s40200-018-0335-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Hosseini F, Ayatollahi H, Salehi SH, Jafar K. Teleburn: designing a telemedicine application to improve burn treatment. Open Med Informat J. 2018;12:33–41. doi: 10.2174/1874431101812010033. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Alanzi T, Istepanian R, Philip N. Design and usability evaluation of social mobile diabetes management system in the Gulf Region. JMIR Res Protocols. 2016;5(3):e93. doi: 10.2196/resprot.4348. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Smaradottir B, Gerdes M, Martinez S, Fensli R. The EU-project United4Health: user-centred design of an information system for a Norwegian telemedicine service. J Telemed Telecare. 2016;22(7):422–429. doi: 10.1177/1357633X15615048. [DOI] [PubMed] [Google Scholar]
- 37.Ligons FM, Mello-Thoms C, Handler SM, Romagnoli KM, Hochheiser H. Assessing the impact of cognitive impairment on the usability of an electronic medication delivery unit in an assisted living population. Int J Med Informat. 2014;83(11):841–848. doi: 10.1016/j.ijmedinf.2014.07.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Lacerda TC, von Wangenheim CG, von Wangenheim A, Giuliano I. Does the use of structured reporting improve usability? A comparative evaluation of the usability of two approaches for findings reporting in a large-scale telecardiology context. J Biomed Inform. 2014;52:222–230. doi: 10.1016/j.jbi.2014.07.002. [DOI] [PubMed] [Google Scholar]
- 39.McFarland LV, Raugi GJ, Reiber GE. Primary care provider and imaging technician satisfaction with a teledermatology project in rural Veterans Health Administration clinics. Telemed J E-health. 2013;19(11):815–825. doi: 10.1089/tmj.2012.0327. [DOI] [PubMed] [Google Scholar]
- 40.Wallace P, Haines A, Harrison R, Barber JA, Thompson S, Roberts J, et al. Design and performance of a multi-centre randomised controlled trial and economic evaluation of joint tele-consultations [ISRCTN54264250] BMC Family Practice. 2002;3:1. doi: 10.1186/1471-2296-3-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.McGloin H, O'Connell D, Glacken M, Mc Sharry P, Healy D, Winters-O'Donnell L, et al. Patient empowerment using electronic telemonitoring with telephone support in the transition to insulin therapy in adults with type 2 diabetes: observational, pre-post, mixed methods study. J Med Internet Res. 2020;22(5):e16161. doi: 10.2196/16161. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Bakken S, Grullon-Figueroa L, Izquierdo R, Lee N-J, Morin P, Palmas W, et al. Development, validation, and use of english and spanish versions of the telemedicine satisfaction and usefulness questionnaire. J Am Med Inform Assoc. 2006;13(6):660. doi: 10.1197/jamia.M2146. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Hatton J, Chandra R, Lucius D, Ciuchta E. Patient satisfaction of pharmacist-provided care via clinical video teleconferencing. J Pharm Pract. 2018;31(5):429–433. doi: 10.1177/0897190017715561. [DOI] [PubMed] [Google Scholar]
- 44.Agha Z, Schapira RM, Laud PW, McNutt G, Roter DL. Patient satisfaction with physician-patient communication during telemedicine. Telemed J E-health. 2009;15(9):830–839. doi: 10.1089/tmj.2009.0030. [DOI] [PubMed] [Google Scholar]
- 45.Kwon S, Kim J, Kang S, Lee Y, Baek H, Park K. CardioGuard: a brassiere-based reliable ECG monitoring sensor system for supporting daily smartphone healthcare applications. Telemed J E-Health. 2014;20(12):1093–1102. doi: 10.1089/tmj.2014.0008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Agnisarman SO, Chalil Madathil K, Smith K, Ashok A, Welch B, McElligott JT. Lessons learned from the usability assessment of home-based telemedicine systems. Appl Ergonom. 2017;58:424–434. doi: 10.1016/j.apergo.2016.08.003. [DOI] [PubMed] [Google Scholar]
- 47.Parra C, Jódar-Sánchez F, Jiménez-Hernández MD, Vigil E, Palomino-García A, Moniche-Álvarez F, et al. Development, implementation, and evaluation of a telemedicine service for the treatment of acute stroke patients: TeleStroke. Interact J Med Res. 2012;1(2):e15. doi: 10.2196/ijmr.2163. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Mehra S, van den Helder J, Visser B, Engelbert RH, Weijs PJ, Kröse BJ. Evaluation of a blended physical activity intervention for older adults: mixed methods study. J Med Internet Res. 2020;22(7):e16380. doi: 10.2196/16380. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Leppert F, Siebermair J, Wesemann U, Martens E, Sattler SM, Scholz S, et al. The INFluence of remote monitoring on anxiety/depression, quality of life, and device acceptance in ICD patients: a prospective, randomized, controlled, single-center trial. Clin Res Cardiol. 2020. [DOI] [PMC free article] [PubMed]
- 50.van der Meij E, Huirne JA, Ten Cate AD, Stockmann HB, Scholten PC, Davids PH, et al. A perioperative ehealth program to enhance postoperative recovery after abdominal surgery: process evaluation of a randomized controlled trial. J Med Internet Res. 2018;20(1):e1. doi: 10.2196/jmir.8338. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Host BK, Turner AW, Muir J. Real-time teleophthalmology video consultation: an analysis of patient satisfaction in rural Western Australia. Clin Exper Optometry. 2018;101(1):129–134. doi: 10.1111/cxo.12535. [DOI] [PubMed] [Google Scholar]
- 52.D'Hooghe M, Van Gassen G, Kos D, Bouquiaux O, Cambron M, Decoo D, et al. Improving fatigue in multiple sclerosis by smartphone-supported energy management: the MS TeleCoach feasibility study. Multiple Sclerosis Related Disorders. 2018;22:90–96. doi: 10.1016/j.msard.2018.03.020. [DOI] [PubMed] [Google Scholar]
- 53.Ammenwerth E, Modre-Osprian R, Fetz B, Gstrein S, Krestan S, Dörler J, et al. HerzMobil, an integrated and collaborative telemonitoring-based disease management program for patients with heart failure: a feasibility study paving the way to routine care. JMIR Cardio. 2018;2(1):e11. doi: 10.2196/cardio.9936. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Oliveira AC, Mattos S, Coimbra M. Development and assessment of an e-learning course on pediatric cardiology basics. JMIR Med Educ. 2017;3(1):e10. doi: 10.2196/mededu.5434. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Roberts S, Spain B, Hicks C, London J, Tay S. Telemedicine in the Northern Territory: an assessment of patient perceptions in the preoperative anaesthetic clinic. Aust J Rural Health. 2015;23(3):136–141. doi: 10.1111/ajr.12140. [DOI] [PubMed] [Google Scholar]
- 56.Poulsen KA, Millen CM, Lakshman UI, Buttner PG, Roberts LJ. Satisfaction with rural rheumatology telemedicine service. Int J Rheumatic Diseases. 2015;18(3):304–314. doi: 10.1111/1756-185X.12491. [DOI] [PubMed] [Google Scholar]
- 57.Vélez O, Okyere PB, Kanter AS, Bakken S. A usability study of a mobile health application for rural Ghanaian midwives. J Midwifery Women's Health. 2014;59(2):184–191. doi: 10.1111/jmwh.12071. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Penteado SP, Bento RF, Battistella LR, Silva SM, Sooful P. Use of the satisfaction with amplification in daily life questionnaire to assess patient satisfaction following remote hearing aid adjustments (telefitting) JMIR Med Informat. 2014;2(2):e18. doi: 10.2196/medinform.2769. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Scalvini S, Tridico C, Glisenti F, Giordano A, Pirini S, Peduzzi P, et al. The SUMMA project: a feasibility study on telemedicine in selected Italian areas. Telemed J E-health. 2009;15(3):261–269. doi: 10.1089/tmj.2008.0109. [DOI] [PubMed] [Google Scholar]
- 60.Kim HM, Lowery JC, Hamill JB, Wilkins EG. Patient attitudes toward a web-based system for monitoring chronic wounds. Telemed J E-Health. 2004;10(Suppl 2):S26–34. [PubMed] [Google Scholar]
- 61.Eminovic N, Wyatt JC, Tarpey AM, Murray G, Ingrams GJ. First evaluation of the NHS direct online clinical enquiry service: a nurse-led web chat triage service for the public. J Med Internet Res. 2004;6(2):e17. doi: 10.2196/jmir.6.2.e17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 62.van Rosmalen-Nooijens K, Lo Fo Wong S, Prins J, Lagro-Janssen T. Young people, adult worries: randomized controlled trial and feasibility study of the internet-based self-support method "feel the vibe" for adolescents and young adults exposed to family violence. J Med Internet Res. 2017;19(6):e204. doi: 10.2196/jmir.6004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 63.Morgan DG, Kosteniuk J, Stewart N, O'Connell ME, Karunanayake C, Beever R. The telehealth satisfaction scale: reliability, validity, and satisfaction with telehealth in a rural memory clinic population. Telemed J E-health. 2014;20(11):997–1003. doi: 10.1089/tmj.2014.0002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 64.Lund AM. Measuring usability with the USE questionnaire. Usability Interface. 2001;8(2):3–6. [Google Scholar]
- 65.Attkisson CC, Zwick R. The client satisfaction questionnaire: psychometric properties and correlations with service utilization and psychotherapy outcome. Eval Program Plan. 1982;5(3):233–237. doi: 10.1016/0149-7189(82)90074-x. [DOI] [PubMed] [Google Scholar]
- 66.Burns JL, Serber ER, Keim S, Sears SF, et al. Measuring patient acceptance of implantable cardiac device therapy: initial psychometric investigation of the Florida Patient Acceptance Survey. J Cardiovasc Electrophysiol. 2005;16(4):384–390. doi: 10.1046/j.1540-8167.2005.40134.x. [DOI] [PubMed] [Google Scholar]
- 67.Marshall GN, Hays RD. The patient satisfaction questionnaire short-form (PSQ-18). RAND; 1994.
- 68.Linnan L, Steckler A. Process evaluation for public health interventions and research. 2002.
- 69.Chin J, Diehl V, Norman KL. Development of an Instrument Measuring User Satisfaction of the Human-Computer Interface. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1988:213–8.
- 70.Fatehi F, Martin-Khan M, Smith AC, Russell AW, Gray LCJDT. Therapeutics Patient satisfaction with video teleconsultation in a virtual diabetes outreach clinic. Diabetes Technol Ther. 2015;17(1):43–48. doi: 10.1089/dia.2014.0159. [DOI] [PubMed] [Google Scholar]
- 71.Wessel R. KH, De Witte L. D-Quest meetinstrument voor de tevredenheid over een hulpmiddel verstrekking. Hoensbroek. 2000.
- 72.Delone WH, McLean ERJJ. The DeLone and McLean model of information systems success: a ten-year update. J Manag Inf Syst. 2003;19(4):9–30. [Google Scholar]
- 73.Elling S, Lentz L, De Jong M, editors. Website evaluation questionnaire: development of a research-based tool for evaluating informational websites. In: International conference on electronic government; 2007. Springer.
- 74.Wang Y-SJI. Management. Assessment of learner satisfaction with asynchronous electronic learning systems. Inf Manag. 2003;41(1):75–86. [Google Scholar]
- 75.Lewis JR, JIJoHCI IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Human Comput Interact. 1995;7(1):57–78. [Google Scholar]
- 76.Hart SG, Staveland LE. Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv Psychol. 1988;52:139–183. [Google Scholar]
- 77.Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 1989;13(3):319–340. [Google Scholar]
- 78.Brooke J. SUS: A 'Quick and Dirty' Usability Scale. Milton Park: Taylor & Francis Group; 1986. [Google Scholar]
- 79.Loane M, Bloomer S, Corbett R, Eedy D, Gore H, Mathews C, et al. Patient satisfaction with realtime teledermatology in Northern Ireland. J Telemed Telecare. 1998;4(1):36–40. doi: 10.1258/1357633981931254. [DOI] [PubMed] [Google Scholar]
- 80.Sabesan S, Simcox K, Marr I. Medical oncology clinics through videoconferencing: an acceptable telehealth model for rural patients and health workers. Internal Med J. 2012;42(7):780–785. doi: 10.1111/j.1445-5994.2011.02537.x. [DOI] [PubMed] [Google Scholar]
- 81.Yen P-Y, Wantland D, Bakken S, editors. Development of a customizable health IT usability evaluation scale. AMIA Annual Symposium Proceedings; 2010: American Medical Informatics Association. [PMC free article] [PubMed]
- 82.Cox RM, Alexander GC. Measuring satisfaction with amplification in daily life: the SADL scale. Ear Hear. 1999;20(4):306–320. doi: 10.1097/00003446-199908000-00004. [DOI] [PubMed] [Google Scholar]
- 83.Sauro J, Lewis JR. Quantifying the User Experience: Practical Statistics for User Research. Amsterdam: Elsevier; 2012. [Google Scholar]
- 84.Ware JE, Jr, Snyder MK, Wright WR, Davies AR. Defining and measuring patient satisfaction with medical care. Evaluat Program Plan. 1983;6(3–4):247–263. doi: 10.1016/0149-7189(83)90005-8. [DOI] [PubMed] [Google Scholar]
- 85.Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Human Comput Interact. 1995;7(1):57–78. [Google Scholar]
- 86.Bertini G. Il learning audit nella sanità. Guida ai processi di miglioramento autogestiti: FrancoAngeli; 2003.
- 87.Brick JE, Bashshur RL, Brick JF, D'alessandri RMJTJ, Public knowledge, perception, and expressed choice of telemedicine in rural West Virginia. Telemed J. 1997;3(2):159–171. doi: 10.1089/tmj.1.1997.3.159. [DOI] [PubMed] [Google Scholar]
- 88.Demiris G, et al. A questionnaire for the assessment of patients' impressions of the risks and benefits of home telecare. J Telemed Telecare. 2000;6(5):278–284. doi: 10.1258/1357633001935914. [DOI] [PubMed] [Google Scholar]
- 89.Davis R, Gardner J, Schnall R. A review of usability evaluation methods and their use for testing eHealth HIV interventions. Current HIV/AIDS Reports. 2020:1–16. [DOI] [PMC free article] [PubMed]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Not applicable.