The Telehealth Satisfaction Scale (TeSS): Reliability, validity, and satisfaction with telehealth in a rural memory clinic population

Abstract

Introduction

Patient satisfaction is a key aspect of quality of care and can inform continuous quality improvement. Of the few studies that have reported on patient satisfaction with telehealth in programs aimed at individuals with memory problems, none has reported on the psychometric properties of the user satisfaction scales employed.

Methods

We evaluated the construct validity and internal consistency reliability of the Telehealth Satisfaction Scale (TeSS), a 10-item scale adapted for use in a Rural and Remote Memory Clinic (RRMC). The RRMC is a one-stop interprofessional clinic for rural and remote seniors with suspected dementia, located in a tertiary care hospital. Telehealth videoconferencing is used for pre-clinic assessment and for follow-up. Patients and caregivers completed the TeSS after each telehealth appointment. With data from 223 patients, exploratory factor analysis was conducted using the principal components analysis extraction method.

Results

The eigenvalue for the first factor (5.2) was greater than 1 and much larger than the second eigenvalue (.92), supporting a one-factor solution that was confirmed by the scree plot. The total variance explained by factor 1 was 52.1%. Factor loadings (range 0.54 – 0.84) were above recommended cutoffs. The TeSS items demonstrated high internal consistency reliability (Cronbach’s alpha = 0.90). Satisfaction scores on the TeSS items ranged from 3.43 to 3.72 on a 4-point Likert scale, indicating high satisfaction with telehealth.

Conclusions

The study findings demonstrate high user satisfaction with telehealth in a rural memory clinic, and sound psychometric properties of the TeSS in this population.

Introduction

Recent international reports have highlighted the growing global prevalence and incidence of dementia and the urgent need for strategies to improve the care and support of individuals with dementia and their informal caregivers.^1,2 A 2012 World Health Organization report Dementia: A Public Health Priority estimated a worldwide prevalence of 35.6 million, which is projected to double every 20 years.² In Canada, the report Rising Tide: The Impact of Dementia on Canadian Society³ forecasts a doubling of dementia prevalence in Canada by 2038. Rural areas are disproportionately affected because of the higher proportion of older adults⁴ and dementia care⁵ and formal services for dementia⁶ in rural and remote areas identified many rural-specific barriers, such as time, travel, cost, and lack of access to services and educational opportunities.

The quality, delivery, and efficiency of healthcare services are being transformed by telehealth technology. Telehealth, or telemedicine, refers to the use of telecommunications and information technologies to provide healthcare services across distances.⁷ Satisfaction with telemedicine is an important research focus because it is a critical aspect of quality of care and health outcomes⁸ and assessing acceptability of services is in line with the growing emphasis on patient reported outcomes.⁹ The majority of studies examining patient satisfaction with telehealth have been conducted using surveys. In a review of 93 studies of patient satisfaction with telehealth, the most frequently measured dimensions were professional-patient interaction, patient experience, and technical aspects.¹⁰ Reviews of patient satisfaction studies^{8, 10,11} have consistently identified significant methodological weaknesses in much of the research, including small sample sizes, convenience samples, absence of a definition of satisfaction, and lack of reliable and valid tools.

Several studies of patient satisfaction with telehealth in diabetes care^12,13,14 and general medical settings¹⁵ have reported satisfactory psychometric properties of questionnaires. Of the few studies that have reported on patient satisfaction with telehealth in programs aimed at individuals with memory problems,^16,17,18 none has reported on the psychometric properties of the user satisfaction scales used. For example, a study exploring the feasibility of telemedicine in a rural memory disorder clinic¹⁶ used a 19-item questionnaire to assess patient satisfaction in a sample of 99 patients. Responses to the 10 items with the highest average rating are reported, but no information about the scale’s reliability and validity is provided. Barton et al.¹⁷ gathered informal feedback on satisfaction with telehealth from 15 patients in a memory disorders clinic but no standardized measure was used. An evaluation of a telemedicine-delivered cognitive intervention for older persons with memory problems¹⁸ used an 11-item user satisfaction questionnaire, but the sample was small (n = 11) and no information about the scale items or psychometric properties was reported. Given the limited data on the reliability and validity of user satisfaction scales, particularly within the context of health care services for individuals with memory problems and their caregivers, the aim of the current study is to evaluate the psychometric properties of the Telehealth Satisfaction Scale (TeSS) in a memory clinic serving a rural and remote population, and to report on satisfaction with telehealth from the perspective of clinic patients and caregivers.

Materials and Methods

This research is part of a larger on-going research program involving the development and evaluation of a Rural and Remote Clinic (RRMC) that incorporates telehealth videoconferencing with a one-stop interdisciplinary assessment in a tertiary care centre. The goal of the clinic is to provide a comprehensive, streamlined assessment of dementia and to reduce repeated travel over long distances. One objective of the research was to determine the acceptability of telehealth for pre-clinic assessment and follow-up appointments. We have previously reported on provider and patient-caregiver satisfaction with the use of telehealth in the clinic,¹⁹ the reliability of cognitive assessments administered in-person and by telehealth,²⁰ and the development and evaluation of a telehealth videoconferenced support group for spouses of RRMC patients diagnosed with atypical early-onset dementias.²¹ Details about the RRMC development and evaluation²² and interdisciplinary approach²³ are reported elsewhere.

Setting

This research was conducted in the western Canadian Prairie province of Saskatchewan (population 1,072,082, area 651,035 km², population/km² 1.88).²⁴ In 2011, 39.1% of the population was classified as Rural and Small Town (population less than 10,000).²⁵ Saskatchewan census data from 2011 show a higher proportion of older adults over 65 years of age living in towns (21.3%), villages (20.1%), and recreational villages (25.8%), compared with cities (14.0%).²⁵ The RRMC is located on the University of Saskatchewan campus in the city of Saskatoon (population 246,300) which is the referral centre for the middle and northern areas of the province. Telehealth services in Saskatchewan are provided through a provincial network initiated in 1999 and now encompassing 229 devices in 144 communities. Telehealth Saskatchewan services are located in community-based hospitals or clinics, and at each site a trained health professional provides technical and clinical support. High-performance videoconferencing equipment is used at all sites. Real-time connections are made across a province-wide broadband Internet protocol network dedicated to health, executive government, and education-based connections. The 768 kilobits/s baseline speed, enhanced with quality of service technology to protect the high quality of video images and sound, provides secure priority queuing for telehealth applications.

Participants

The target population for the RRMC is noninstitutionalized individuals in Saskatchewan who are referred to the clinic because of concern about a memory disorder. Eligible patients must live more than 100 kms from the two major centres in Saskatchewan (rural location) or in one of the three northern health regions (remote location). Patients referred to the RRMC are first seen via telehealth from their local telehealth centre, located in a hospital or health centre. Local telehealth coordinators facilitate all sessions, including managing equipment and assisting with administration of testing materials. During a 30-minute preclinic assessment the clinic nurse and neuropsychologist interview the patient and family about the referring problem and provide information about the clinic visit. This information is used to tailor the subsequent full-day interdisciplinary assessment that is conducted a few weeks later in Saskatoon by a neurologist, neuropsychology team, physical therapist, nurse, and dietitian. To evaluate telehealth in the clinic, a single case design²⁶ was used, with patients randomly assigned to either telehealth or in-person appointment for their first follow-up assessment, and then alternating between the two modes for the 6-week, 12-week, and 6-month follow-ups.²² At one year and annually, the patient and family are seen face-to-face for follow-up by the full team. Immediately following the pre-clinic telehealth assessment and each follow-up telehealth appointment, patients and their caregivers complete the TeSS. The psychometric evaluation reported in this article used data from the pre-clinic assessment, to control for potential differences in satisfaction due to familiarity with telehealth over time. The study received ethical approval from the University of Saskatchewan behavioral ethics review board.

The TeSS

The 10-item TeSS was adapted from a 12-item instrument used in previous telehealth studies in Saskatchewan. The scale was initially used as part of a national study to evaluate the use of telehealth applications in imporving specialist access by residents of isolated northern First Nations communities,²⁷ and it was later used in a pediatric surgery clinic²⁸ and amputee clinic.²⁹ We replaced the term “specialist” in the original scale with “Memory Clinic Team”.

For the analysis reported in this article, two items were excluded from the orginial 12-item scale prior to conducting factor analysis. The first item, “How satisfied were you with the length of time to get this appointment?”, was excluded based on the rationale that specialist appointment wait times are a function of the health care system rather than the telehealth system. The second item, “How satisfied were you with your overall treatment experience at using telehealth?”, was excluded on the basis that it was an overall measure of satisfaction rather than a measure of satisfaction with a particular aspect of the telehealth experience.

Scale items are rated on a 4-point Likert scale (1 = poor, 2 = fair, 3 = good, and 4 = excellent). The total score on the 10-item TeSS can range from 10 to 40, with higher scores indicating higher satisfaction.

The original scale was described as being developed to assess “satisfaction with the telemedicine service and the quality of specialist care,”²⁹ although no psychometric evaluation was reported, and the theoretical framework for scale development was not described. Rather, satisfaction items appear to have been chosen based on face validity.³⁰ Our examination of the scale items suggested the scale’s original authors were attempting to measure the following two constructs: professional-patient interaction and system/technical factors. They did not, however, examine its actual construct validity.

Data Analysis

Psychometric analyses were carried out to examine the construct validity of the TeSS and its internal consistency reliability. Factor analysis allows for the evaluation of construct validity (the internal structure of a scale) by demonstrating the degree to which scale items reflect particular constructs within the scale.³⁰ In addition, the procedure may be used to determine the number of dimensions within a single scale. Given a unidimensional scale, a single score may be calculated, whereas a multidimensional scale may require scores to be calculated for each subscale.³⁰ We then used the principal components analysis extraction method followed by varimax rotation to conduct an exploratory factor analyses on the 10-item TeSS. Case mean imputation of missing data was used where fewer than 25% of the items were missing (i.e., two or fewer items); if more than two items in the 10-item scale were missing, the scale for that patient was excluded.

Data were analysed using SPSS software (version 20 for Macintosh; SPSS, Inc., Chicago, IL). Frequencies, mean, range, and standard deviation (SD) were used to describe the sociodemographic and clinical characteristics of patients, and mean scores were calculated for the 10 TeSS items. Support for the use of factor analysis was evaluated with the Kaiser-Meyer-Olkin measure and Bartlett’s Test of Sphericity. The Kaiser-Meyer-Olkin measure of sampling adequacy varies from 0 to 1 and indicates the magnitude of the difference between zero-order correlations and partial correlations between pairs of variables, with measures of 0.80 and higher indicating that variables share a common factor and factor analysis is warranted.³¹ The evaluation of the TeSS also included the percentage of variance accounted for by the scale, as well as internal consistency reliability as assessed by Cronbach’s alpha.³¹ Cronbach’s alpha values can range from zero to one, with high alpha values indicating that the scale as a whole is measuring just one attribute.³²

Results

Demographic and clinical data

Of the 318 patients seen in the RRMC study between March 2004 and July 2011 (Data Release 5, 2012), 65 patients had missing data (i.e., were not able to attend preclinic assessment, the questionnaire was not completed, or the questionnaire was not returned by the telehealth coordinator), and 26 patients completed preclinic assessment by means other than telehealth (e.g., by telephone or in-person) or were not required to complete a preclinic assessment before clinic day (i.e., clinic day appointments became available on short notice because of cancellations by other patients). Of the 227 patients or their family members who completed the TeSS, 223 completed 8 or more of the 10 TeSS items included in the factor analysis, and were included in the present study.

Table 1 presents the demographic and clinical characteristics of the 223 patients included in the present study. Most of the clinic patients were male (57.4%) and the average patient age was 71.5 years (range 41–91 years; SD = 11.0 years). The average education level of patients was below high school graduation at 10.8 years (range 3–20 years; SD = 3.3 years) and most patients (72.6%) spoke primarily English in their home. Significant co-morbidity was common in this sample, with 73.5% of patients reporting 5 or more chronic conditions. After assessment at the clinic, 71.3% of patients were diagnosed with Mild Cognitive Impairment, Alzheimer’s disease, or another related dementia. A majority (57.8%) of the patients lived in communities classified as no, weak, or moderate Metropolitan Influenced Zones (see the footnote of Table 1). The average distance to the RRMC site in Saskatoon was 260.9 km (range, 103–595 km; SD = 102.3 km). The one-way travel saved by telehealth assessment was substantial at 227.9 km (range, 34–594 km; SD = 106.5 km).

Table 1.

Characteristics of Rural and Remote Memory Clinic patients (n=223)

	N	%
Gender
Male	128	57.4
Female	95	42.6
Agea (Mean, range, SD)	71.5 (41–91, 11.0)
< 65	52	23.3
65–74	46	20.6
75–84	81	36.3
≥ 85	21	9.4
Missing	23	10.3
Marital Status
Married/Common-law	131	58.7
Widowed/separated/divorced	50	22.4
Never married	4	1.8
Missing	38	17.0
Education (Mean, range, SD)	10.8 (3–20, 3.3)
Less than high school graduation	100	44.8
High school graduation	19	8.5
Beyond high school graduation	65	29.1
Missing	39	17.5
Language spoken most often at time
English	162	72.6
Other than English	10	4.5
Missing	51	22.9
Place of residenceb
Strong MIZ	94	42.1
Moderate MIZ	36	16.1
Weak MIZ	71	31.8
No MIZ	22	9.9
Chronic conditions
No chronic conditions	2	0.9
1–2	21	9.4
3–4	50	22.4
5 or more chronic conditions	114	51.1
Missing	36	16.1
Diagnosisc
Alzheimer’s disease	85	38.1
Mild cognitive impairment	25	11.2
Related Dementiab	49	22.0
Dementia not diagnosed	40	17.9
Missing	24	10.8
Distance (km) to telehealth site (Mean, range, SD)	33.0 (1–150, 42.1)
Distance (km) to Memory Clinic site (Mean, range, SD)	260.9 (103–595, 102.3)
Distance one-way (km) saved by telehealth (Mean, range, SD)	227.9 (34–594, 106.5)

^aAt Clinic Day

^bIn strong Metropolitan Influenced Zone (MIZ) communities, 30% or more of the workforce commutes for work to a census metropolitan area (CMA) (≥100,000 population) or to a census agglomeration (CA) (≥10,000); moderate MIZ communities, 5% to 29% of the workforce commutes to a CMA/CA; weak MIZ, <5% commute to a CMA/CA; and no MIZ, none of the workforce commutes to a CMA/CA.³⁴

^cVascular dementia, Dementia with lewy bodies, Frontotemporal dementia, Dementia multiple etiologies, Vascular cognitive impairment, Parkinson’s Disease, Huntington’s disease, normal pressure hydrocephalus, dementia due to other general medical conditions, cognitive impairment not otherwise specified.

SD, standard deviation.

Satisfaction with Telehealth

The 10 items in the TeSS are shown in Table 2. Patients and caregivers expressed a high degree of satisfaction with their telehealth preclinic assessment. Average scores ranged from 3.43 (SD = 0.52) for the length of time with the RRMC team to 3.72 for the courtesy, respect, sensitivity, and friendliness of the RRMC team. The majority of scores on the scale items averaged higher than 3.5, indicating a rating of ‘good’ to ‘excellent’ satisfaction.

Table 2.

Responses to 10-item Telehealth Satisfaction Scale (n = 223)

Item no.	Statement	Excellent n (n%)	Good n (n%)	Poor/Fair n (n%)	Mean (SD)
	How satisfied were you with:
1	The voice quality of the equipment	121 (54.3)	92 (41.3)	10 (4.5)	3.50 (0.58)
2	The visual quality of the equipment	135 (60.5)	86 (38.6)	2 (0.6)	3.60 (0.51)
3	Your personal comfort in using the Telehealth system	107 (48.0)	108 (48.4)	8 (3.6)	3.44 (0.57)
4	The ease of getting to the telehealth department	109 (48.9)	105 (47.1)	9 (4.0)	3.45 (0.57)
5	The length of time with the Memory Clinic team	98 (43.9)	122 (54.7)	3 (1.3)	3.43 (0.52)
6	The explanation of your treatment by the Memory Clinic team	116 (52.0)	105 (47.1)	2 (0.9)	3.51 (0.52)
7	The thoroughness, carefulness and skillfulness of the Memory Clinic team	134 (60.1)	87 (39.0)	2 (0.9)	3.59 (0.51)
8	The courtesy, respect, sensitivity, and friendliness of the Memory Clinic team	161 (72.2)	61 (27.4)	1 (0.4)	3.72 (0.46)
9	How well your privacy was respected	150 (67.3)	72 (32.3)	1 (0.4)	3.67 (0.48)
10	How well the staff answered your questions about the equipment	141 (63.2)	80 (35.9)	2 (0.9)	3.62 (0.50)

Data are number (%).

SD, standard deviation.

Factor Analysis

The number of patients in this sample (n = 223) is greater than the recommended minimum of 100 to 200 respondents for principal components analysis.³¹ The Kaiser-Meyer-Olkin measure (0.90) indicated strong relationships between the variables. With data from 223 patients, factor analysis was conducted using the principal components analysis extraction method with varimax rotation on the 10-item TeSS (Table 3). Bartlett’s Test of Sphericity was significant (<0.001), providing support for the use of factor analysis as an appropriate procedure with these data.

Table 3.

Factor analysis of the 10-item Telehealth Satisfaction Scale (n=223)

Item no.	Statement	Factor 1 loadings
9	How well your privacy was respected	0.79
8	The courtesy, respect, sensitivity, and friendliness of the Memory Clinic team	0.78
7	The thoroughness, carefulness and skillfulness of the Memory Clinic team	0.77
6	The explanation of your treatment by the Memory Clinic team	0.76
10	How well the staff answered your questions about the equipment	0.75
2	The visual quality of the equipment	0.72
5	The length of time with the Memory Clinic team	0.71
1	The voice quality of the equipment	0.70
3	Your personal comfort in using the Telehealth system	0.68
4	The ease of getting to the telehealth department	0.54
% Variance		52.12
Eigenvalue		2.7
Cronbach’s α		0.90
Sum Mean (range, SD)		35.5 (28–40, 3.75)

Factor analysis was performed using the principal components analysis extraction method and varimax rotation.

SD, standard deviation.

Eigenvalues produced by the factor analysis procedure represent the amount of variance captured by each component or factor. In this study, the eigenvalues provided evidence for a one-factor solution, including the large difference between the eigenvalue for the first factor (5.2) and the second factor (.92), and the “eigenvalue greater than one” rule for determining the number of factors.³⁰ The scree plot confirmed a sharp drop in eigenvalues for subsequent factors, supporting the one-factor structure. Factor loadings for all items (range 0.54 – 0.84) were greater than the recommended cutoff of .20 to .35³¹, indicating that the scale items loaded onto only one factor and that factor structure was strong. The total variance explained by factor one was 52.1%. The lowest factor loadings were for items related to the ease of getting to the telehealth department (0.54) and personal comfort using the telehealth system (0.68). Overall, the scale items demonstrated high internal consistency reliability, as indicated by the Cronbach’s alpha score of 0.90.

Separate analyses were also conducted with two proposed five-item subscales indentified in the scale. The Cronbach’s alpha score was 0.78 for the system/technical items (1, 2, 3, 4, and 10) and 0.86 for the patient-provider interaction items (5, 6, 7, 8, and 9) suggesting that these subscales also have acceptable internal consistency, but details of these analyses are not reported because the 10-item scale was shown to be unidimensional.

Discussion

Advancing our knowledge about satisfaction with telehealth services requires standardized measures that have been tested for reliability and validity and that can be applied across fields, and assess multiple dimensions of satisfaction.⁸ In this study, the 10-item TeSS demonstrated strong factor structure and excellent internal consistency reliability in a population of rural patients and caregivers accessing services at a memory clinic. The psychometric evaluation provides evidence that the TeSS items are highly correlated and exhibit a unidimensional internal structure. Using attributes of primary healthcare identifitied by Lévesque et al.,³³ the clustering of all scale items into one factor suggest that the construct of satisfaction with telehealth includes dimensions of interpersonal communication (items 5, 6, and 10), respectfulness (items 8 and 9), whole person care (item 3), and technical quality of care (items 1, 2, and 7). The factor loading for item 4, regarding ease of getting to the telehealth department (0.54) was lower than for other items, but still well above recommended cut-offs.³¹ Compared to earlier studies examining satisfaction with telemedicine in memory clinic settings,^16–18 this study includes a larger sample size (n = 223 vs. 11, 15, and 99 respectively) and provides evidence of scale reliability and validity. Results from our previous analyses of telehealth in the RRMC provided additional evidence for the validity of the TeSS.¹⁹ We found that patients who had discontinued follow-up at six months had significantly lower TeSS scores than those who continued at six months. To further explore predictors of discontinuing follow-up we conducted a univariate logistic regression analysis of pre-clinic assessment TeSS scores and other study variables, in which lower satisfaction with telehealth (lower TeSS scores) was a significant predictor of discontinuing, in addition to longer distance to telehelath, older patient age, and lower caregiver burden. These earlier analyses provide support for the discriminant validity of the TeSS, and together with the current results support the use of the TeSS in future studies of interventions that involve team-based telehealth service delivery similar to the RRMC.

The findings from this study also indicate high satisfaction with telehealth by memory clinic patients and caregivers, providing support for the use of telehealth to increase access to specialized dementia services for those living in rural and remote settings. As reported in our earlier study of satisfaction with telehealth,¹⁹ patients and families reported reduced travel and stress, greater convenience, and the high quality of care provided via telehealth as reasons for their high satisfaction. These identified benefits of telehealth are not surprising given the high level of comorbidity in the current study (71.3% reporting 5 or more chronic conditions) which may make travel difficult, and the long distances to travel to the tertiary centre where the clinic is located (over 260 km one way).

The high satisfaction reported in the current study is consistent with the few existing studies examining satisfaction with telehealth among patients with memory problems. Azad et al.¹⁶ did not report overall satisfaction scores, but 92% of participants were satisfied with the session, and 95% would use videoconferencing again. The informal feedback collected by Barton et al.¹⁷ indicated patient and provider satisfaction with the neurological and cognitive assessment conducted via telehealth. Poon et al.¹⁸ evaluated a 12-session cognitive assessment and intervention program delivered by videoconferencing or face to face. An 11-item user satisfaction scale was administered to the videoconference group (n=11). The items were not described, but the authors reported that 90% were satisfied with the audio and video quality.

Reviews of satisfaction with telehealth have suggested that high satisfaction levels may represent ceiling effects due to self-selection of those with higher satisfaction and instrument design that limits variation, as well as participants’ reluctance to criticize a needed service.¹⁰ We have reported previous analyses of satisfaction with telehealth by RRMC patients and caregivers that included continuing patients and those who discontinued.¹⁹ The study identified four predictors of discontinuing follow-up, including low satisfaction with telehealth, indicating there is variation in the satisfaction data and that the scale has predictive value.

There are several limitations to the study. Because patients and families completed the TeSS together following each telehealth appointment, we are not able to assess differences in satisfaction between patients and caregivers. Findings regarding satisfaction with telehealth may not be generalizable to other settings that do not have the same well-managed, coordinated telehealth network as exists in Saskatchewan, where trained professional staff are present to provide support to users. When the RRMC was implemented in 2004, the telehealth equipment was not as user-friendly as is the current technology. We did not expect patients and families to manage the equipment independently, particularly in a memory clinic context, so this type of usability was not evaluated in the TeSS. In a more recent study evaluating a videoconferenced support group for spouses of RRMC patients with atypical and early-onset dementias, participants received training from telehealth support staff and subsequently navigated the equipment without on-site supervision.²¹ The data used for the psychometric analysis reported in this article were from the first telehealth encounter; thus the reported satisfaction could reflect a positive bias due to novelty.¹¹ In earlier research, we have examined satisfaction with telehealth over time and found no difference in satisfaction from preclinical to 6-month follow-up appointments among continuing patients, but low satisfaction was a predictor of discontinuing follow-up when both groups were included.¹⁹

Strengths of the current study include the large sample size relative to much of the existing research assessing the psychometric properties of telehealth satisfaction questionnaires, the relatively homogenous sample of memory clinic patients, and the long-term nature of the clinic intervention. Many satisfaction studies are conducted in artificial contexts with nonrepresentative patient samples,¹¹ whereas the current study involved remotely delivered telehealth appointments conducted through the provincial telehealth system. The study finding should therefore be more readily translatable to setting where telehealth is used in routine practice.

In conclusion, the psychometric analyses reported in this article provide evidence for the construct validity and internal consistency reliability of the TeSS in a rural memory clinic population. These data demonstrating the psychometric properties of the 10-item TeSS are important, and support use of the TeSS as a measure of satisfaction in future telehealth-based clinical research.