Abstract
Objective
The study was aimed to translate and adapt the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Patient Questionnaire (TPQ) into Turkish, thereafter to analyze the psychometric properties of both questionnaires.
Methods
A total of 149 multiple sclerosis (MS) patients were recruited in the study. For 4 years, all patients were supervised by a department clinician using telemedicine. Cronbach’s alpha coefficient was used to assess internal consistency. By evaluating the scores of 41 retested patients 1 week later, the test-retest reliability was determined using the intraclass correlation coefficient (ICC). Pearson’s correlation coefficient was used to assess the construct validity (r).
Results
A total of 149 patients (103 women, 46 men) with a mean age of 40.9±10.9 years were included in the study. MS duration of the patients was 9.15±6.24 years. Internal consistency of all items and the total score of the TSUQ were excellent (>0.80; ranged 0.971–0.974). On the other hand, the internal consistency of all items and total score of the TPQ was excellent, either (>0.80; ranged 0.878–0.890). The ICC of the TSUQ’s and TPQ’s total score was excellent (>0.80). The correlation between TSUQ and TPQ was strong (r=0.734, p<0.01). In addition, there was moderate relationship between the TSUQ and the Beck Depression Scale (BDS) (r=−0.363, p<0.01). On the other hand, there was low correlation between TPQ and BDS (r=−0.217, p<0.05).
Conclusion
The Turkish version of the TSUQ and the TPQ is valid and reliable in individuals with MS.
Keywords: Multiple sclerosis, Psychometrics, Telemedicine Patient Questionnaire, Telehealth Satisfaction and Usability Questionnaire, Turkish version
Introduction
Telehealth services are common in neurological assessment and rehabilitation, including stroke, multiple sclerosis (MS), Parkinson’s disease, and epilepsy [1–4]. MS patients need specific healthcare services due to different neurological deficits. Individuals with MS living in rural areas with low socioeconomic levels and mobility problems experience difficulties receiving neurological care. It has been reported that at least 30% of individuals with MS do not have access to healthcare. In addition, MS patients who receive neurological care have more opportunities to receive specific health services such as physiotherapy and occupational therapy [5].
Disease-modifying therapies and neurorehabilitation are essential in the effective management of MS and are provided by neurorehabilitation specialists with proper coordination [6]. Robb et al. compared telemedicine and face-to-face visits in individuals with MS. It has been stated that the telemedicine service provided in the virtual environment for individuals with MS is low cost and saves time. Moreover, it has been reported that it is feasible and preferable for the clinician and that individuals with MS have a high level of satisfaction [7]. In both telemedicine and face-to-face interviews, patient and caregiver satisfaction has been reported to be similar. Individuals suffering from neurological diseases can receive fast and accessible healthcare services with telemedicine service and reduce the cost of healthcare services [8].
The services provided within the scope of telemedicine service in individuals with MS have been reported as functional assessment, treatment follow-up, treatment compliance, cognitive assessment, cognitive rehabilitation, mindfulness, and telerehabilitation [7, 9–13]. In a systematic review on this subject, evidence about the positive effects of telerehabilitation on fatigue, function, and quality of life of individuals with MS has been presented [14]. Moreover, it was stated that, with telerehabilitation, people with MS increased their participation in exercise and their level of satisfaction was good [2].
In order to get the maximum benefit from telehealth technologies, the telehealth application system can be used effectively for clinicians and patients [15]. In addition to the quality and cost of telemedicine services, the acceptance of this service by individuals, its availability, and satisfaction level should be evaluated [16]. The availability of telemedicine services can be made by observation, questionnaire, interview, and different recording methods [17, 18]. Clinical results of telemedicine services can be measured with questionnaires, quality of life, and different tools specific to the disease [19].
In the literature, the expectations, satisfaction, and usability of individuals who receive telemedicine services are evaluated with different surveys: Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Patient Questionnaire (TPQ) [20, 21]. The psychometric properties of the Turkish versions of both questionnaires were not investigated. Besides, there is no other tool that shows the availability and satisfaction of Turkish-speaking individuals’ telehealth services. Thus, it is not possible to control the usability of telehealth applications and receive patient feedback in terms of satisfaction in a standard way. The present study aims to translate and adapt the TSUQ and the TPQ into Turkish, and to analyze the psychometric properties of both questionnaires.
Materials and methods
Translation and adaptation process
Permission was obtained from Bakken and Demiris, who developed the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Patient Questionnaire (TPQ), respectively. The cultural adaptation of both questionnaires was carried out by taking into account the internationally accepted translation procedures in order to carry out the language translation stages [22, 23]. During the advanced translation phase, the original English was translated into Turkish by four academicians, native speakers of TSUQ and TPQ, who have a good command of English. Expert committee members in the field identified and recorded linguistic and cultural problems. In the second phase, the expert committee members created correction notes of TSUQ and TPQ for translations and discussed. The English translations of both questionnaires were synthesized and finalized. In the third stage, both questionnaires were independently translated into Turkish by two native English translators. Original English versions of both questionnaires were rejected and compared with the English versions in the fourth step, regardless of whether the final Turkish versions were conceptually and linguistically correct. In the last stage, draft versions of both questionnaires were created for use in pilot testing. At the last stage, a pilot study was conducted to question whether the Turkish versions of TSUQ and TPQ are suitable for understandability. A pretest was conducted on a 5-point Likert-type scale with 30 Turkish-speaking elderly individuals. Understandability was excellent in the pilot study. Therefore, no additional changes have been made. The latest Turkish versions of TSUQ and TPQ have been created.
Sample size estimation
The sample size was determined using statistical suggestions and power analysis. Fayers and Machin proposed that in cross-cultural adaptation experiments, the sample size should be at least 100 and, preferably, seven times the number of questionnaires [24]. TSUQ and TPQ have 21 and 17 items, respectively. The psychometric study required a total of 147 participants due to the outnumbered TSUQ items. The minimum sample for the reproducibility was calculated by G* power 3 software (Heinrich-Heine-Universität, Düsseldorf, Germany) with an effect size of 0.40, a probability of error (α) = 0.05, and the power (1-ß) of 0.80. Thirty-four patients were required for the reproducibility. As a result, 149 MS patients were tested in the first round, and 41 patients were refilled the TSUQ and TPQ, 1 week later for retest reliability [25].
Study design
The study was conducted as a prospective cross-sectional design by “X” University Neurology Department on individuals with MS. Inclusion criteria of the study: Individuals with MS who are older than 18 years old and have been monitored with telemedicine for at least 4 years, individuals who have not had an MS attack in the last month, individuals with MS who can read and understand Turkish. The study’s exclusion criteria are individuals who have had an MS attack the last month. Telemedicine services (e.g., medication modifications, monitoring of symptoms, exercise management, follow-up of relapses) conducted to the patients were carried out through several operators or through web-based applications, phone calls, messaging, teleconference, within the needs and possibilities of the patients. Socio-demographic and physical characteristics of individuals with MS were recorded. Expanded Disability Status Scale (EDSS) of the patients was scored by the neurologist [26]. In the first evaluation, 149 individuals with MS completed by thoroughly answering all the items of the Turkish versions of TSUQ, TPQ, and Beck Depression Scale (BDS) [20, 21, 27]. Due to the psychological criterion of pleasure was questioned, convergent validity was investigated by comparing it to BDS. In addition, since the expectations and benefits of the patients from the telemedicine service were also questioned, it was thought that it would have positive effects on their psychological state. A retest evaluation was carried out for individuals with MS 1 week later. In the retest evaluation, 41 individuals with MS answered the Turkish versions of TSUQ and TPQ.
Telemedicine Satisfaction and Usefulness Questionnaire
TSUQ was developed by Bakken et al. in 2006. It consists of 21 items. With this questionnaire, the satisfaction levels and usability of individuals using telemedicine service (from the software or system from which they receive treatment or other therapy services) are evaluated. A 5-point Likert scale is used in TSUQ evaluation (1 = strongly disagree, 5 = strongly agree). The total score ranges from 21 to 105 [20].
Telemedicine Patient Questionnaire
TPQ was developed by Demiris et al. It consists of 17 items in total. A 5-point Likert-type scale is used for TPQ evaluation. It evaluates individuals’ telemedicine service, expectation, satisfaction, and usability. Items 2, 3, 7, 15, and 17 are scored by reversing. The total score is scored between 17 and 85 [21].
Beck Depression Scale
The Turkish version of the scale was proved for validity and reliability. BDS evaluates depression severity. BDS consists of 21 items. Each item is pointed between 0 and 3. It is assessed over the sum score. A higher score shows high depression. The highest score is 63 [27].
Statistical analysis
SPSS for Windows v25.0 was used for statistic interpretations in all of the analyses (SPSS Inc, Chicago, IL, USA). The quantitative data was assigned with a mean and a standard deviation. For the qualitative results, percentages were used. To detect whether the data in the sample had a homogeneous distribution, the Shapiro-Wilk test was used. The correlation coefficients were agreed with 95% confidence intervals.
Reliability
The Cronbach alpha coefficient was calculated to determine the TSUQ and TPQ’s internal consistency. TSUQ and TPQ, which have a higher alpha coefficient, suggest more consistent results. An alpha of ≥0.6 was deemed acceptable; ≥0.8 was ranked excellent consistency [28]. The two-way random-effect model single-measure reliability study of Shrout and Fleiss was chosen. Reproducibility was measured using the intraclass correlation coefficient (ICC, 95% confidence interval). ICC was calculated for the total score and items of the TSUQ and TPQ. An ICC ≥0.8 shows excellent test-retest reliability [29].
Validity
The construct validity of TSUQ and TPQ was tested by Pearson’s correlation coefficient (r). The total of the TSUQ, TPQ, and BDS was compared with each other. A high correlation coefficient was presumed in terms of convergent validity. In addition, low correlation was considered divergent validity. If the coefficient was greater than 0.5, it was considered strong; if it was between 0.5 and 0.35, it was considered medium; and if it was less than 0.35, it was considered low [30].
Results
A total of 149 patients (103 women, 46 men) with a mean age of 40.9±10.9 years were recruited in the study. MS duration of the patients was 9.15±6.24 years. Table 1 shows the absolute values of the patients. TSUQ and TPQ were excellently understandable in linguistic and cultural concepts in a pilot study conducted with 30 Turkish-speaking individuals. As a result, after the pretest, no additional changes were required (Table 2).
Table 1.
The characteristics of the patients
| n: 149 | Total |
|---|---|
| Age (years, mean±SD) | 40.9±10.9 |
| BMI (kg/m2) | 25.1±4.3 |
| Gender (n, %) | |
| Female | 103 (69.1) |
| Male | 40 (30.9) |
| MS duration (years, mean±SD) | 9.15±6.24 |
| Education | |
| Primary school (n, %) | 17 (11.4) |
| Middle school (n, %) | 14 (9.4) |
| High school (n, %) | 48 (32.2) |
| University (n, %) | 64 (43.0) |
| Postgraduate (n, %) | 6 (4.0) |
SD standard deviation, n number of patients, BMI body mass index, MS multiple sclerosis
Table 2.
The clinical assessment values of the patients
| n: 149 | Total (mean±SD) | Range |
|---|---|---|
| TSUQ | 85.2±21.0 | 21-105 |
| TPQ | 62.2±9.9 | 39-85 |
| BDS | 11.7±10.4 | 0-41 |
| EDSS | 2.0±1.7 | 0-7 |
SD standard deviation, n number of patients, BMI body mass index, MS multiple sclerosis, TSUQ Telemedicine Satisfaction and Usefulness Questionnaire, TPQ Telemedicine Patient Questionnaire, BDS Beck Depression Scale, EDSS Expanded Disability Status Scale
Reliability and validity
The study’s reliability findings are summarized in Tables 3 and 4. Cronbach’s alpha coefficient was determined for the TSUQ and TPQ items and overall ratings. Internal consistency of all items and the total score of the TSUQ were excellent (>0.80; ranged 0.971–0.974). On the other hand, the internal consistency of all items and total score of the TPQ was excellent, either (>0.80; ranged 0.878–0.890). The ICC of the TSUQ total score was excellent (>0.80). The ICC score of items for TSUQ was ranged from 0.581 to 0.889. The reproducibility of the total score for the TSUQ was excellent (>0.80). The test-retest reliability of the items of the TPQ was ranged from 0.214 to 0.873. The validity results are presented in Table 5. The correlation between TSUQ and TPQ was strong (r=0.734, p<0.01). In addition, there was moderate relationship between the TSUQ and the total score of the BDS (r=−0.363, p<0.01). On the other hand, there was low correlation between TPQ and BDS (r=−0.217, p<0.05).
Table 3.
Test-retest reliability and internal consistency for the TSUQ
| Test (Mean±SD) | Retest (Mean±SD) | ICC (95% CI) | α | |
|---|---|---|---|---|
| Item 1 | 4.28±1.10 | 4.02±1.15 | 0.878 (0.77–0.93) | 0.972 |
| Item 2 | 3.32±1.47 | 3.29±1.40 | 0.581 (0.21–0.77) | 0.974 |
| Item 3 | 3.71±1.40 | 3.73±1.28 | 0.625 (0.29–0.80) | 0.972 |
| Item 4 | 3.97±1.25 | 3.85±1.29 | 0.817 (0.65–0.90) | 0.972 |
| Item 5 | 4.14±1.21 | 3.95±1.28 | 0.788 (0.60–0.88) | 0.971 |
| Item 6 | 3.97±1.25 | 3.56±1.46 | 0.821 (0.66–0.90) | 0.972 |
| Item 7 | 3.80±1.33 | 3.56±1.44 | 0.889 (0.79–0.94) | 0.972 |
| Item 8 | 4.18±1.17 | 3.85±1.37 | 0.817 (0.65–0.90) | 0.971 |
| Item 9 | 4.18±1.15 | 3.97±1.27 | 0.721 (0.47–0.85) | 0.971 |
| Item 10 | 4.18±1.22 | 3.92±1.33 | 0.741 (0.51–0.86) | 0.971 |
| Item 11 | 3.85±1.35 | 3.29±1.20 | 0.719 (0.47–0.85) | 0.972 |
| Item 12 | 3.85±1.30 | 3.60±1.26 | 0.786 (0.59–0.88) | 0.972 |
| Item 13 | 4.28±1.18 | 4.31±1.01 | 0.738 (0.50–0.86) | 0.972 |
| Item 14 | 4.15±1.21 | 3.75±1.19 | 0.786 (0.59–0.88) | 0.972 |
| Item 15 | 4.02±1.26 | 3.78±1.42 | 0.727 (0.48–0.85) | 0.972 |
| Item 16 | 4.00±1.24 | 3.85±1.35 | 0.804 (0.63–0.89) | 0.972 |
| Item 17 | 4.21±1.18 | 3.97±1.33 | 0.763 (0.55–0.87) | 0.971 |
| Item 18 | 4.38±1.30 | 3.90±1.37 | 0.772 (0.57–0.87) | 0.971 |
| Item 19 | 4.36±1.13 | 4.39±0.97 | 0.618 (0.23–0.79) | 0.971 |
| Item 20 | 4.36±1.10 | 4.43±0.86 | 0.600 (0.24–0.78) | 0.971 |
| Item 21 | 4.33±1.11 | 4.21±1.06 | 0.665 (0.37–0.82) | 0.972 |
| Total score | 85.2±21.0 | 81.26±20.74 | 0.850 (0.71–0.92) | 0.973 |
n number of patients, ICC intraclass correlation coefficient, CI confidence interval, α Cronbach’s alpha, TSUQ Telemedicine Satisfaction and Usefulness Questionnaire
Table 4.
Test-retest reliability and internal consistency for the TPQ
| Test (Mean±SD) | Retest (Mean±SD) | ICC (95% CI) | α | |
|---|---|---|---|---|
| Item 1 | 3.56±0.96 | 3.51±0.97 | 0.513 (0.08–0.74) | 0.884 |
| Item 2 | 3.84±0.89 | 3.68±0.81 | 0.573 (0.19–0.77) | 0.886 |
| Item 3 | 4.08±1.05 | 3.75±1.42 | 0.632 (0.30–0.80) | 0.890 |
| Item 4 | 3.57±1.09 | 3.53±1.09 | 0.686 (0.41–0.83) | 0.879 |
| Item 5 | 3.42±1.02 | 3.82±0.86 | 0.566 (0.18–0.76) | 0.880 |
| Item 6 | 4.14±0.77 | 4.26±0.70 | 0.652 (0.34–0.81) | 0.882 |
| Item 7 | 3.40±1.23 | 3.21±1.40 | 0.214 (–0.47–0.58) | 0.890 |
| Item 8 | 3.89±0.84 | 3.87±1.00 | 0.813 (0.64–0.90) | 0.878 |
| Item 9 | 3.11±1.15 | 3.29±1.00 | 0.627 (0.30–0.80) | 0.888 |
| Item 10 | 3.73±0.93 | 3.80±0.98 | 0.751 (0.53–0.86) | 0.878 |
| Item 11 | 4.08±0.83 | 4.17±0.73 | 0.657 (0.35–0.81) | 0.879 |
| Item 12 | 3.78±0.99 | 3.82±1.18 | 0.784 (0.59–0.88) | 0.880 |
| Item 13 | 4.08±0.74 | 4.24±0.73 | 0.789 (0.60–0.88) | 0.884 |
| Item 14 | 4.04±0.77 | 4.07±0.87 | 0.634 (0.31–0.80) | 0.884 |
| Item 15 | 2.62±1.19 | 2.80±1.16 | 0.873 (0.76–0.93) | 0.882 |
| Item 16 | 3.88±0.79 | 3.90±0.94 | 0.755 (0.54–0.86) | 0.882 |
| Item 17 | 2.99±1.02 | 3.17±1.04 | 0.729 (0.49–0.85) | 0.886 |
| Total score | 62.2±9.9 | 62.9±11.3 | 0.876 (0.76–0.93) | 0.889 |
n number of patients, ICC intraclass correlation coefficient, CI confidence interval, α Cronbach’s alpha, TPQ Telemedicine Patient Questionnaire
Table 5.
Correlation between TSUQ, TPQ, and BDS
| n: 149 | r | p |
|---|---|---|
| TSUQ-TPQ | 0.734 | <0.01 |
| TSUQ-BDS | −0.363 | <0.01 |
| TPQ-BDS | −0.217 | <0.05 |
TSUQ Telemedicine Satisfaction and Usefulness Questionnaire, TPQ Telemedicine Patient Questionnaire, BDS Beck Depression Scale, r Pearson’s correlation coefficient, p significance level
Discussion
This research aimed to demonstrate the TSUQ’s and TPQ’s translation, cross-cultural adaptation, reliability, and validity in MS patients who were monitored with telemedicine by the neurology department for 4 years. As a result, the Turkish versions of the TSUQ and TPQ have been proven reliable and valid patient-reported outcome measures for testing Turkish-speaking, multiple sclerosis cases. To adapt the questionnaires to Turkish patients, a scientific, textual, and cultural adaptation procedure was followed [22, 23]. Psychometric properties of the TSUQ and TPQ were also successfully presented. Since there are no Turkish screening methods for telehealth providers, both questionnaires are a one-of-a-kind assessment tool for Turkish-speaking patients.
It is evident that the importance of telemedicine applications during the COVID-19 era further increased [31]. The importance of telemedicine in terms of providing care and increasing accessibility in patients with MS during the COVID-19 period has been emphasized in a recent study [32]. In addition, it has been stated that telemedicine has vital importance in pandemic situations due to possible limited staff, limited outpatient working hours, and increasing outpatient needs [33]. In this study, psychometric properties of TSUQ and TPQ were analyzed in individuals who had been undergoing a telerehabilitation program for 4 years, but more intensively for the last 6 months, by the neurology outpatient clinic. In this way, it is proposed to provide an analysis to use these patients reported outcome measures in future studies safely. TSUQ is one of the most popular questionnaires frequently used in telemedicine studies. English and Spanish versions of the TSUQ are available [20]. On the other hand, TPQ is available only in its original English version [21]. The Turkish version of both questionnaires could greatly lead to clinical practice and other case-control or intervention trials, given the dramatic rise in telemedicine and telerehabilitation in recent years and the role of remote health services in the COVID-19 process [34–36]. The consistency of the telemedicine program and level of satisfaction could be compiled and tracked in a more detailed and systematic way using the quantitative data derived from patient reviews. Both questionnaires found to be practical, easy to use, and thus can be completed in a limited period of time.
The phrase “telemedicine program” was changed as “method” in the TSUQ and TPQ translation adaptation processes. The translation committee anticipated this technological revolution, speculating that telemedicine programs would no longer be restricted to the “system.” Furthermore, the term “clinician” was favored over “healthcare professional” in TPQ. The translation committee introduced this modification because the “clinician” identified a possible telemedicine provider in all health sciences to improve conformity with the questionnaire’s interprofessional mode. The TSUQ and TPQ were found to be appropriate in terms of comprehensibility in a pilot analysis. As a result, no other changes to the pretest were needed. The internal consistency for the total score and item scores of the TSUQ and TPQ was excellent (α>0.80). TSUQ and TPQ are consistently assessing the usefulness and satisfaction of telemedicine systems, according to the findings. Besides, the items of both questionnaires are compatible within themselves. Only Demiris and colleagues analyzed the Cronbach alpha for TPQ. The results of the alpha are above 0.80 [21]. Their calculation results were conformed to our study (α> 0.80). Cronbach’s alpha scores were not presented in the TSUQ development study [20].
The test-retest reliability for the total scores of the TSUQ and TPQ was excellent (ICC>0.80). In addition, all items (except item 2) of the TSUQ have moderate to excellent reproducibility (>0.60). The ICC value of only the second items of TSUQ was less than 0.6. In this article, the answer to the question given to the sentence “My health is better than it was before I used the technology” is sought. In the answer given by the patients to this question, we emphasized the possibility of not understanding telemedicine service with the word technology. In addition, the ICC score in items 1, 2, 5, and 7 in TPQ was below 0.60. However, despite this, having the ICC score of the total score above 0.80 is sufficient for the questionnaire’s reproducibility. The ICC was not mentioned in the development work of both questionnaires. Our study is also unique in this respect.
Construct validity was analyzed by comparing the TSUQ, TPQ, and BDS scores. The correlation between TSUQ and TPQ was strong (r=0.734, p<0.01). In addition, there was moderate relationship between the TSUQ and the total score of the BDS (r=−0.363, p<0.01). On the other hand, there was low relationship between the TPQ and the total score of the BDS (r=−0.217, p<0.05). In particular, the authors used BDS mainly within the scope of divergent validity. In other words, we were predicting that there would be a low to moderate correlation of BDS with TSUQ and TPQ. According to the validation results of our study, both questionnaires were found to be valid.
It is also necessary to note the study’s limitations. The current research did not look at responsiveness analysis. Furthermore, the “System Usability Scale” was not used to determine validity [37]. The construct validity could be further clarified using a 10-item System Usability Questionnaire. However, it could be time consuming to ask the participants more questions. It can also obstruct the processing of specific data. Furthermore, in both surveys, testing and validating more psychometric properties in various case types would be more helpful to clinical practice.
Conclusions
Ultimately, the Turkish versions of the TSUQ and TPQ are determined to be reliable and valid for measuring usability and satisfaction in MS cases. Both questionnaires are accurate patient-reported outcome measures for patients with multiple sclerosis. Both questionnaires also would contribute significantly to both science and clinical practice, given that telemedicine is a trendy era due to the rapid development in healthcare technologies.
Data availability
All data generated or analyzed during this study are included in this published article.
Declarations
Ethical approval
The permission for the translation for the Turkish version of the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Patient Questionnaire (TPQ) was acquired from the developers of the original questionnaires. The study was carried out in accordance with the ethical principles and the Helsinki Declaration. Informed consent of the patients was obtained. The study protocol was approved by the ethics committee of Ege University (No: 21-3T/29).
Informed consent
Informed consent of the patients was obtained.
Conflict of interest
The authors declare no competing interests.
Footnotes
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Mutgi SA, Zha AM, Behrouz R. Emerging subspecialties in neurology: telestroke and teleneurology. Neurology. 2015;84(22):e191–e193. doi: 10.1212/WNL.0000000000001634. [DOI] [PubMed] [Google Scholar]
- 2.Amatya B, Galea MP, Kesselring J, Khan F. Effectiveness of telerehabilitation interventions in persons with multiple sclerosis: a systematic review. Mult Scler Relat Disord. 2015;4(4):358–369. doi: 10.1016/j.msard.2015.06.011. [DOI] [PubMed] [Google Scholar]
- 3.Veazie S, Bourne D, Peterson K, Anderson J (2019) Evidence brief: video telehealth for primary care and mental health services. Depart Vet Aff 1–35 [PubMed]
- 4.Pareyson D, Pantaleoni C, Eleopra R, De Filippis G, Moroni I, Freri E, Zibordi F, Bulgheroni S, Pagliano E, Sarti D (2021) Neuro-telehealth for fragile patients in a tertiary referral neurological institute during the COVID-19 pandemic in Milan, Lombardy. Neurol Sci 42(7):2637–2644 [DOI] [PMC free article] [PubMed]
- 5.Minden SL, Frankel D, Hadden L, Hoaglin DC. Access to health care for people with multiple sclerosis. Mult Scler J. 2007;13(4):547–558. doi: 10.1177/1352458506071306. [DOI] [PubMed] [Google Scholar]
- 6.Tornatore C, Phillips JT, Khan O, Miller AE, Hughes M. Consensus opinion of US neurologists on practice patterns in RIS, CIS, and RRMS: evolution of treatment practices. Neurol Clin Pract. 2016;6(4):329–338. doi: 10.1212/CPJ.0000000000000254. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Robb JF, Hyland MH, Goodman AD. Comparison of telemedicine versus in-person visits for persons with multiple sclerosis: a randomized crossover study of feasibility, cost, and satisfaction. Mult Scl Relat Disord. 2019;36:101258. doi: 10.1016/j.msard.2019.05.001. [DOI] [PubMed] [Google Scholar]
- 8.Hatcher-Martin JM, Adams JL, Anderson ER, Bove R, Burrus TM, Chehrenama M, O’Brien MD, Eliashiv DS, Erten-Lyons D, Giesser BS. Telemedicine in neurology: telemedicine work group of the American Academy of Neurology update. Neurology. 2020;94(1):30–38. doi: 10.1212/WNL.0000000000008708. [DOI] [PubMed] [Google Scholar]
- 9.Shaw MT, Best P, Frontario A, Charvet LE. Telerehabilitation benefits patients with multiple sclerosis in an urban setting. J Telemed Telecare. 2021;27(1):39–45. doi: 10.1177/1357633X19861830. [DOI] [PubMed] [Google Scholar]
- 10.Sesel A-L, Sharpe L, Beadnall HN, Barnett MH, Szabo M, Naismith SL. The evaluation of an online mindfulness program for people with multiple sclerosis: study protocol. BMC Neurol. 2019;19(1):1–8. doi: 10.1186/s12883-019-1356-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Healey K, Zabad R, Young L, Lindner A, Lenz N, Stewart R, Charlton M (2018) An initiative to improve care in the community: multiple sclerosis at home access (MAHA). Int J MS Care 21(3):101–12 [DOI] [PMC free article] [PubMed]
- 12.George MF, Holingue CB, Briggs FBS, Shao X, Bellesis KH, Whitmer RA, Schaefer C, Benedict RHB, Barcellos LF. Feasibility study for remote assessment of cognitive function in multiple sclerosis. J Neurol Neuromed. 2016;1(8):10. doi: 10.29245/2572.942X/2016/8.1084. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Fuchs TA, Ziccardi S, Dwyer MG, Charvet LE, Bartnik A, Campbell R, Escobar J, Hojnacki D, Kolb C, Oship D. Response heterogeneity to home-based restorative cognitive rehabilitation in multiple sclerosis: an exploratory study. Mult Scler Relat Disord. 2019;34:103–111. doi: 10.1016/j.msard.2019.06.026. [DOI] [PubMed] [Google Scholar]
- 14.Khan F, Amatya B, Kesselring J, Galea M (2015) Telerehabilitation for persons with multiple sclerosis. Cochrane Database Syst Rev CD010508 [DOI] [PMC free article] [PubMed]
- 15.Stipa G, Gabbrielli F, Rabbito C, Di Lazzaro V, Amantini A, Grippo A, Carrai R, Pasqui R, Barloscio D, Olivi D. The Italian technical/administrative recommendations for telemedicine in clinical neurophysiology. Neurol Sci. 2021;42(5):1923–1931. doi: 10.1007/s10072-020-04732-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Hajesmaeel-Gohari S, Bahaadinbeigy K. The most used questionnaires for evaluating telemedicine services. BMC Med Inform Decis Mak. 2021;21(1):1–11. doi: 10.1186/s12911-021-01407-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.AlDossary S, Martin-Khan MG, Bradford NK, Smith AC. A systematic review of the methodologies used to evaluate telemedicine service initiatives in hospital facilities. Int J Med Inform. 2017;97:171–194. doi: 10.1016/j.ijmedinf.2016.10.012. [DOI] [PubMed] [Google Scholar]
- 18.Klaassen B, van Beijnum BJF, Hermens HJ. Usability in telemedicine systems—a literature survey. Int J Med Inform. 2016;93:57–69. doi: 10.1016/j.ijmedinf.2016.06.004. [DOI] [PubMed] [Google Scholar]
- 19.Kidholm K, Clemensen J, Caffery LJ, Smith AC. The Model for Assessment of Telemedicine (MAST): a scoping review of empirical studies. J Telemed Telecare. 2017;23(9):803–813. doi: 10.1177/1357633X17721815. [DOI] [PubMed] [Google Scholar]
- 20.Bakken S, Grullon-Figueroa L, Izquierdo R, Lee N-J, Morin P, Palmas W, Teresi J, Weinstock RS, Shea S, Starren J. Development, validation, and use of English and Spanish versions of the telemedicine satisfaction and usefulness questionnaire. J Am Med Inform Assoc. 2006;13(6):660–667. doi: 10.1197/jamia.M2146. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Demiris G, Speedie S, Finkelstein S. A questionnaire for the assessment of patients’ impressions of the risks and benefits of home telecare. J Telemed Telecare. 2000;6(5):278–284. doi: 10.1258/1357633001935914. [DOI] [PubMed] [Google Scholar]
- 22.Guillemin F, Bombardier C, Beaton D. Cross-cultural adaptation of health-related quality of life measures: literature review and proposed guidelines. J Clin Epidemiol. 1993;46(12):1417–1432. doi: 10.1016/0895-4356(93)90142-N. [DOI] [PubMed] [Google Scholar]
- 23.Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000;25(24):3186–3191. doi: 10.1097/00007632-200012150-00014. [DOI] [PubMed] [Google Scholar]
- 24.Fayers PM, Machin D. Sample size: how many patients are necessary? British J Cancer. 1995;72(1):1–9. doi: 10.1038/bjc.1995.268. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Faul F, Erdfelder E, Buchner A, Lang A-G. Statistical power analyses using G* Power 3.1: tests for correlation and regression analyses. Behav Res Methods. 2009;41(4):1149–1160. doi: 10.3758/BRM.41.4.1149. [DOI] [PubMed] [Google Scholar]
- 26.Kurtzke JF. Rating neurologic impairment in multiple sclerosis: an expanded disability status scale (EDSS) Neurology. 1983;33(11):1444–1444. doi: 10.1212/WNL.33.11.1444. [DOI] [PubMed] [Google Scholar]
- 27.Kapci EG, Uslu R, Turkcapar H, Karaoglan A. Beck Depression Inventory II: evaluation of the psychometric properties and cut-off points in a Turkish adult population. Depress Anxiety. 2008;25(10):E104–E110. doi: 10.1002/da.20371. [DOI] [PubMed] [Google Scholar]
- 28.Schermelleh-Engel K, Moosbrugger H, Müller H. Evaluating the fit of structural equation models: tests of significance and descriptive goodness-of-fit measures. Methods Psychol Res Online. 2003;8(2):23–74. [Google Scholar]
- 29.Baumgartner TA, Chung H. Confidence limits for intraclass reliability coefficients. Meas Phys Educ Exerc Sci. 2001;5(3):179–188. doi: 10.1207/S15327841MPEE0503_4. [DOI] [Google Scholar]
- 30.Juniper EF, Guyatt GH, Jaeschke R (1995) How to develop and validate a new quality of life instrument: quality of life and pharmacoeconomics in clinical trials. Lippincott-Raven Publishers, Philadelphia
- 31.Portnoy J, Waller M, Elliott T. Telemedicine in the era of COVID-19. J Allergy Clin Immunol Pract. 2020;8(5):1489–1491. doi: 10.1016/j.jaip.2020.03.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Corea F, Ciotti S, Cometa A, De Carlo C, Martini G, Baratta S, Zampolini M. Telemedicine during the Coronavirus Disease (COVID-19) Pandemic: a multiple sclerosis (MS) outpatients service perspective. Neurol Int. 2021;13(1):25–31. doi: 10.3390/neurolint13010003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Garg A, Goyal S, Thati R, Thati N. Implementation of telemedicine in a tertiary hospital–based ambulatory practice in detroit during the COVID-19 pandemic: observational study. JMIR Publ Health Surveill. 2021;7(1):e21327. doi: 10.2196/21327. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Saleem SM, Pasquale LR, Sidoti PA, Tsai JC. Virtual ophthalmology: telemedicine in a COVID-19 era. Am J Ophthalmol. 2020;216:237–242. doi: 10.1016/j.ajo.2020.04.029. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Bahl S, Singh RP, Javaid M, Khan IH, Vaishya R, Suman R (2020) Telemedicine technologies for confronting COVID-19 pandemic: a review. J Ind Integr Manag 5(4):547–561
- 36.Hau YS, Kim JK, Hur J, Chang MC. How about actively using telemedicine during the COVID-19 pandemic? J Med Syst. 2020;44(6):1–2. doi: 10.1007/s10916-020-01580-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Brooke J (1996) SUS-A quick and dirty usability scale. Usability evaluation in industry 189(194):4–7
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
All data generated or analyzed during this study are included in this published article.