Test–retest reliability and criterion-related validity of Shih–Hsu test of attention between people with and without schizophrenia

Abstract

Introduction

The “Shih–Hsu Test of Attention” (SHTA) is an iPad-based attention assessment tool developed in recent years by occupational therapists and has acceptable criterion-related validity and high test–retest reliability in preliminary application. This research project explores the criterion-related validity and test–retest reliability of SHTA between people with and without schizophrenia.

Method

The participants were 76 adults with schizophrenia aged 20–64 years, and 66 adults without diagnosed mental illness aged 20–64 years were recruited in this study on a voluntary basis. Each participant was assessed twice. The participants completed both the SHTA and Chu’s Attention Test (CAT) in the first test and the SHTA after 3 weeks.

Findings

Analytical results indicate that the SHTA has satisfactory test–retest reliability (ICC = 0.67) and criterion-related validity (γ = 0.29, p < 0.05*) for adults with schizophrenia and has high test–retest reliability (ICC = 0.90) and criterion-related validity (γ = 0.25, p < 0.05*) for adults without diagnosed mental illness. The MDC% value for the subjects without diagnosed mental illness was 12.1%, indicating acceptable random measurement error.

Conclusion

Our preliminary findings show that the iPad-based attention assessment tool, SHTA, has satisfactory criterion-related validity and test–retest reliability, supporting the future application of SHTA as an attention assessment tool.

Introduction

Attention performance is an important assessment item for occupational therapy because it significantly affects daily life and work. Among healthy people, attention performance is important for everyday activities that require modern technologies such as mobile phones, computers, and machines (Lin et al., 2018; Nygård and Rosenberg, 2016) and has a major effect on workability in tasks such as the safe operation of machines and vehicles. Attention additionally influences working ability and employment performance, and poor attention ability can lead to safety problems in everyday life and in the workplace (Nygård and Rosenberg, 2016; Shih et al., 2015). The most common case for psychiatric rehabilitation is schizophrenia, and cognitive interventions such as attention training are very important for schizophrenia with delusional disorder (Díaz-Caneja et al., 2018). A patient’s performance in employment is another important issue in determining the prognosis of schizophrenia. Attention is a cognitive function and can enable patients to accomplish tasks more efficiently. The damaged cognitive function of people with schizophrenia, due to brain damage or illness, tends to lead indirectly to reduced attention span. A schizophrenicpatient has worse attention function than an average person, negatively influencing the patient’s adaptation to a community (Lahera et al., 2017). Thus, the promotion of attention assessment tools for people with or without a diagnosed mental illness is an important subject for occupational therapy.

Attention performance has several facets, namely, reaction time, focused attention, sustained attention, selective attention, alternating attention, and divided attention (Mirsky et al., 1991; Sohlberg and Mateer, 1987). A previous study of attention deficit of people with schizophrenia found that it mainly affects sustained attention and recommended further investigation into sustained attention of relation to people with schizophrenia (Hoonakker et al., 2017). Another report on the driving ability of patients with schizophrenia found that they tend to drive abnormally slow and have poor driver reaction time (Fuermaier et al., 2018). A study involving 98 outpatients with schizophrenia found that their performance in reaction time was strongly correlated with their performance in emotional cognitive function (Lahera et al., 2017). Another study evaluated the performance of schizophrenia patients in performing tasks requiring sustained attention using fMRI brain scans and found that patients with and without schizophrenia had different brain activity levels in the cingulate cortex and thalamus (Sepede et al., 2014). The above studies demonstrate that attention function of schizophrenic patients is worth investigation. Another study focused on 522 children of parents with schizophrenia, due to the likely presence of family history of the disease, and found that many of these children would start from an early age to show deficit problems in sustained attention (Burton et al., 2018). Another scholar indicated that people with schizophrenia tend to experience declining sustained attention with age (Hoonakker et al., 2017). Although human attention function has various facets, new research literature has identified attention deficit among schizophrenic patients from reaction times and sustained attention (Fuermaier et al., 2018; Hoonakker et al., 2017; Sepede et al., 2014). Accordingly, psychiatric occupational therapy should focus on the attention assessment methods for these two aspects.

iPad-based tools are now very popular due to the rising prevalence of iPad devices. Therefore, new iPad-based assessment tools need to be developed to consider convenience (Shih et al., 2020). The “Shih–Hsu Test of Attention” (SHTA) is an iPad-based attention assessment tool with auditory stimuli, developed by occupational therapists to measure reaction time and sustained attention. Unlike previous visual attention tests, which adopt mostly pen and paper as the medium interface, the auditory attention assessment tool developed for this research project adopts an iPad computer. The software app randomly plays several sounds, such as drum, zither, piano, violin, Chinese flute, and trumpet, in 10 min. All sounds have the same volume and pitch, but differ in timbre. Each sound appears randomly at intervals of 0.5 s, 1 s, and 1.5 s. Upon hearing each sound, a test taker is asked to press the corresponding button (Yes or No) on the iPad touch screen, illustrated in Figure 1. Scoring is based on the rate of correct answers during those 10 min. The highest score is 100 points and the lowest is 0 points. These rules were explained to the participants in their native language. The SHTA results demonstrate that this new iPad-based attention has acceptable criterion-related validity (γ = 0.400, p < 0.05*) and high test–retest reliability (ICC = 0.920, p < 0.01**) in healthy adults aged 65–85 years (Shih, in press), but the SHTA has not been validated for adults with schizophrenia. A useful assessment tool needs to have reliability and validity for people with and without diagnosed mental illness. Accordingly, the present investigation has the following aims:

1. to study the criterion-related validity of SHTA for schizophrenic adults and adults without diagnosed mental illness, and

2. to explore the test–retest reliability of SHTA for schizophrenic adults and healthy adults without diagnosed mental illness.

Figure 1.

The iPad touch screen of Shih–Hsu test of attention (SHTA).

Method

This investigation examined the criterion-related validity and test–retest reliability of the SHTA administered to adults with schizophrenia and adults without diagnosed mental illness.

Study design

The study was divided into two parts. The first part was to examine the criterion-related validity of the SHTA and analysis using the intra-class correlation coefficient (ICC) and minimal detectable change (MDC). The second part examined the test–retest reliability of the SHTA and analysis using Pearson correlation.

Ethical issues

Ethical approval for the study was granted by the Institutional Review Board (IRB). All survey and audit data were accumulated anonymously. Written consent was taken immediately prior to the test, and participants were reminded that it could be withdrawn at any time and that researchers had no financial interest in the study.

Participants

Schizophrenic participants were recruited by convenience sampling from two community-based psychiatric rehabilitation centers. Healthy participants, without any diagnosed mental illness, were recruited by convenience sampling from the Internet and poster recruitment in Taipei City and New Taipei City. People with hearing impairments were excluded from these two groups. The 142 voluntary participants who provided informed consent were enrolled. They were 76 adults with schizophrenia aged 20–64 years and 66 adults without diagnosed mental illness aged 20–64 years.

Measures

The SHTA

The SHTA is an iPad-based assessment tool developed by occupational therapists for response time and sustained attention with musical stimuli (Shih et al., 2020). The official test takes 10 min, preceded by 15 s of practice.

Chu’s Attention Test (CAT)

Chu’s Attention Test is a standard evaluation tool frequently adopted in occupational therapy in Chinese societies and has high test–retest reliability (0.837, p < 0.001) and validity (0.44, p < 0.01) for attention performance (Chu, 1997). It is a writing test, comprising more than 100 questions, each requiring the participants to view a series of scrambled codes, search for the “∗” symbols among these codes, count the number of “∗” symbols, and record it as the answer. The testing time is 10 min, and the final score is the “total number of answers” minus the “number of wrong answers” (Chu, 2001; Shih et al., 2015).

Procedure

1. The SHTA and CAT were run to test the 76 adults with schizophrenia and to study the association between the scores of the SHTA and those of CAT (criterion-related validity).

2. The SHTA and CAT were run to test the 66 adults without diagnosed mental illness and to study the association between the scores of the SHTA and CAT (criterion-related validity).

3. After 3 weeks, the SHTA was run to test the same 76 adults with schizophrenia. The test scores from the two SHTA were compared (test–retest reliability).

4. After 3 weeks, the SHTA was run to test the same 66 adults without diagnosed mental illness. The test scores from the two SHTA were compared (test–retest reliability).

Data analysis

All statistical analyses were conducted using SPSS20.0 (SPSS Japan, Tokyo, Japan) with p < 0.05 considered to be statistically significant. First, the \ICC was used to estimate the test–retest reliability by comparing the test scores from the two SHTA (test–retest reliability). The ICC was used to examine the test–retest reliability of the SHTA. The ICC was calculated by a random-effects, two-way analysis of variance. An ICC value ≥0.80 indicates high reliability, 0.60–0.79 indicates moderate reliability, and <0.59 indicates poor reliability (Burton et al., 2018).

The measurement consistency between two repeated assessments was visualized with Bland–Altman plots with 95% limits of agreement (LOA) (Martin Bland and Altman, 1986). The differences between each pair of assessments were plotted against the mean of each pair of assessments. The 95% LOA lies between the mean difference $\pm$ 1.96 SD of the difference. Narrower limits indicate higher stability. In addition, the plots were used to demonstrate whether heteroscedasticity was present, that is, whether the difference between repeated assessments generally increased or decreased as the mean of the repeated assessments increased (Bland and Altman, 1999). The heteroscedasticity was investigated by calculating Pearson’s r between the absolute value of difference and mean scores of repeated assessment. An r lower than 0.30 indicates no heteroscedasticity.

We calculated the MDC of the 95% confidence level (MDC₉₅) based on the standard error of measurement (SEM) for two populations (i.e., the patients with schizophrenia and the normal subjects). The MDC₉₅ can be used to define the minimal score difference that indicates a real change between test and retest assessments that is not due to random measurement error.

In addition, the MDC₉₅ was further expressed as a percentage (MDC%). The MDC% was calculated using the MDC divided by the mean of all testing scores for the sample. An MDC% of <30% was considered acceptable and <10% was considered excellent (Smidt et al., 2002).

To examine the practice effects of the SHTA in the patients with schizophrenia and the normal subjects, we analyzed the mean differences between test and retest assessments using both the paired t-test and effect size (Cohen’s d). The significance level was set at 0.05. Cohen’s d was calculated by dividing the mean score differences of test and retest assessments by the SD of the total scores of two assessments. In terms of Cohen’s criteria, d > 0.80 was considered to indicate a large practice effect; 0.50–0.79, medium; 0.20–0.49, small; and <0.20, negligible (Cohen, 1988).

Second, the correlation between the first SHTA scores and the CAT scores was investigated using the Pearson correlation coefficient to estimate the criterion-related validity.

Findings

Test–retest reliability of the SHTA in persons with schizophrenia (n = 76)

The ICC was determined to compare the test scores from the two SHTA. The ICC of the two tests was 0.67 (p < 0.05*), as shown in Table 1.

Table 1.

Parameters of test–retest reliability, random measurement error, and practice effect of the Shih–Hsu test of attention in patients with schizophrenia and normal subjects.

	Baseline	Re-assessment	Difference	ICC (95% CI)	SEM	MDC (MDC%)	p	Effect size
Mean (SD)	Mean (SD)	Mean (SD)	Mean (SD)	Mean (SD)	Mean (SD)
Schizophrenia (n = 76)	0.52 (0.21)	0.58 (0.23)	0.03 (0.18)	0.67 (0.52–0.77)	0.13	0.35 (62.5)	0.101	0.27
Normal subjects (n = 66)	0.91 (0.13)	0.93 (0.12)	0.01 (0.06)	0.90 (0.84–0.94)	0.04	0.11 (12.1)	0.464	0.16

Table 1 shows the values of the test–retest reliability and random measurement error of the SHTA in patients with schizophrenia and normal. In the patients with schizophrenia, the ICC value was 0.67, the MDC₉₅ was 0.35, and the MDC% was 62.5%. The Bland–Altman plot for the patients with schizophrenia (Figure 2) showed wide limits of the 95% LOA (from −0.31 to 0.38). The association between the means and absolute values of change in the SHTA was very weak (Pearson’s r = 0.12, p = 0.294). The mean difference between repeated assessments was 0.03.

Figure 2.

Bland–Altman plot of the difference of scores against the mean scores of the Shih–Hsu test of attention in patients with schizophrenia (n = 76). The solid line represents the mean of the differences (0.03). The two dashed lines define limits of agreement $\left(\overline{\text{d}}\pm 1.96\times \text{SD}=-0.31\sim 0.38\right)$ .

Criterion-related validity of the SHTA in persons with schizophrenia (n = 76)

The scores of the SHTA and CAT were compared by the Pearson correlation coefficient. The SHTA and CAT scores were significantly correlated (γ = 0.285, p < 0.05*), as shown in Table 2.

Table 2.

Comparison of the scores between the SHTA and CAT with chronic schizophrenia (n = 76) by using the Pearson correlation coefficient.

	Mean	SD	Pearson correlation	Sig
CAT	55.395	24.431	0.285**	0.000
SHTA	54.20	20.957

*p < 0.05 statistically significant difference; ** p < 0.01; *** p < 0.005.

Note: SHTA: Shih–Hsu test of attention; CAT: Chu’s attention test.

The SHTA of the normal subjects had high ICC (0.90) with small MDC (0.11). The MDC% was 12.1%. The LOA ranged from −0.10 to 0.12 (Figure 3). The association between the mean and absolute values of change in the SHTA was weak (Pearson’s r = 0.23, p = 0.062).

Figure 3.

Bland–Altman plot of the difference of scores against the mean scores of the Shih–Hsu test of attention in normal subjects (n = 66). The solid line represents the mean of the differences (−0.02). The two dashed lines define limits of agreement $\left(\overline{\text{d}}\pm 1.96\times \text{SD}=-0.10\sim 0.12\right)$ .

Test–retest reliability of the SHTA in persons without diagnosed mental illness (n = 66)

The ICC was calculated to compare the test scores from the two SHTA. The ICC of the two tests was 0.900 (p < 0.01**), as shown in Table 1.

Criterion-related validity of the SHTA in persons without diagnosed mental illness (n = 66)

The scores of the SHTA and CAT were compared by the Pearson correlation coefficient. Table 3 presents the SHTA and CAT scores, which were significantly correlated (γ = 0.253, p < 0.05*).

Table 3.

Comparison of the scores between the SHTA and CAT with normal people (n = 66) by using the Pearson correlation coefficient.

	Mean	SD	Pearson correlation	Sig
CAT	125.121	19.733	0.253*	0.041
SHTA	91.273	13.074

*p < 0.05 statistically significant difference; ** p < 0.01; *** p < 0.005.

Note: SHTA: Shih–Hsu test of attention; CAT: Chu’s attention test.

The analyses of the practice effect reveal that the mean scores of the SHTA in the patients and the adults without diagnosed mental illness showed no statistical improvement (p ≥ 0.101). The effect sizes of score change in the patients were larger than 0.20 (Cohen’s d = 0.27), but smaller than 0.20 in adults without diagnosed mental illness (Cohen’s d = 0.16).

Discussion

Attention assessment is a significant issue for occupational therapy, since attention performance significantly affects work and daily life.

This investigation performed a preliminary examination of the criterion-related validity in comparison with CAT, and the test–retest reliability, on 76 participants with schizophrenia and 66 participants without diagnosed mental illness.

Test–retest reliability is a reliability measure obtained by administering the same test twice over a period of time to a group of individuals. Test–retest is a significant evaluation tool, because it measures stability of results in clinical applications (Chu, 1997; Lin et al., 2018).

The ICC of the SHTA was 0.90 in the normal subjects and that the ICC was 0.67 in the patients with schizophrenia. These results indicate that the test–retest reliability of the SHTA was excellent in participants without diagnosed mental illness and good in patients with schizophrenia. The lower test–retest reliability in the patients than in the adults without diagnosed mental illness may due to the patients’ high ratio of within-subject variance (which can be represented by the SD² of the score difference between test and retest assessments). The patients’ high ratio of within-subject variance may have resulted from lack of familiarity with the use of iPad touch screens. Nevertheless, given that the ICC of the SHTA in participants without diagnosed mental illness was excellent, it may be suitable for repeated assessments. The test reliability of the SHTA in patients should be interpreted from another reliability examination, including random measure error.

The MDC₉₅ values were 0.35 and 0.11 for patients with schizophrenia and adults without diagnosed mental illness, respectively. These findings indicate that an adult’s SHTA change score was larger than 0.11, which can be interpreted as a real change rather than variation resulting from random measurement error. In contrast, the SHTA change score in the patients should be larger than 0.35 points to conclude that the patient has a real change. As mentioned above, the ratio of within-subject variance in patients was higher than that in adults without diagnosed mental illness. The higher ratio of within-subject variance may be explained by the random measurement error (i.e., MDC₉₅). Notably, the random measurement error was larger in patients with schizophrenia than adults without diagnosed mental illness, suggesting that patients tend to yield inconsistent responses to the stimuli and assessment scores. Our findings can help researchers and clinicians interpret the change scores for patients with schizophrenia and adults without diagnosed mental illness more conservatively considering random measurement errors.

The MDC% value for the normal subjects was 12.1%, indicating acceptable random measurement error. However, the MDC% value for the patients with schizophrenia was 62.5%, indicating substantial random measure error between test and retest assessments. The reason for the substantial MDC% in the patients may be because the MDC% was the MDC divided by the mean of all patients’ scores which appeared to be lower than those of the normal subjects (the mean of all patients’ scores = 0.52 vs. that of the normal subjects = 0.91). The substantial MDC% implies that it will be very difficult for a patient to have a change score beyond the MDC of the SHTA. Therefore, reducing the measurement error of the SHTA for patients with schizophrenia should be a critical issue to be addressed. Such an issue may be resolved by the following two methods. One is administering 2–3 trials of the test to obtain an average score of the SHTA, which may offset the random measurement error of each score of the trials. Another method is adding more practice before the formal test to reach patients’ best performance and reduce random measurement error. In this way, clinicians could obtain more reliable and stable SHTA results. The effectiveness of these two methods to reduce the random measurement error needs to be further validated.

Pearson’s r between the mean and absolute values of change in the SHTA was smaller than 0.30 both in the patients and the normal subjects, indicating no notable heteroscedasticity (e.g., the difference rises as the mean scores rise or fall). That is, as shown in the Bland–Altman plots, no obvious systematic trends were found. These findings imply that random measurement error (i.e., MDC) of SHTA for patients and adults without diagnosed mental illness can be considered as a fixed error across the entire range of possible scores.

The SHTA effect size in the subjects without diagnosed mental illness was 0.16, indicating that the practice effect of the SHTA reached a plateau in the subjects without diagnosed mental illness. In contrast, the effect size of the SHTA (0.27) in the patients was larger than 0.20, which indicates that the practice effect of the SHTA did not reach a plateau at retest assessments. Consequently, the clinicians may overestimate improvement or underestimate deterioration of attention in patients with schizophrenia. We suggest that patients may need more practice before formal testing to reach their current best performance, enabling the clinicians could confirm the real level of attention.

The criterion-related validity refers to the performance of association between two different tests (SHTA and CAT in this investigation). The two tests had a positive association rate in people with schizophrenia (0.29) and people without diagnosed mental illness (0.25), revealing that the SHTA had positive criterion-related validity. Chu’s Attention Test measures visual attention ability, while the SHTA measures auditory attention ability. Clinically, the visual and auditory attention tests should be studied separately, but each still has its own applications. The iPad-based assessment tool caters to modern people’s usage habits and can keep score quickly and therefore should be further adopted in the next generation.

This research project still has its limitations, as detailed below. Only 76 adults with schizophrenia and 66 healthy participants aged 20–64 years took the tests. SHTA was only compared with CAT scores. Future work will adopt other test tool scores for comparison. This research is based on convenience sampling, so its findings may not be generalized to population. In addition, schizophrenia has many types, and this research did not consider factors such as the types of schizophrenia, gender, and age.

Conclusion

The iPad is an increasingly popular device, and iPad touch screens should be utilized to develop modern assessment tools for occupational therapy. The “SHTA” is a novel iPad-based attention assessment tool with auditory stimuli, developed in recent years by occupational therapists. This investigation draws the following conclusions and suggestions:

1. The SHTA has satisfactory criterion-related validity and test–retest reliability for adults with and without schizophrenia.

2. In order to enhance the credibility of the reliability–validity analysis, the SHTA test uses a much larger sample size and is compared with other attention tests for reliability and validity. Additionally, the participants have 1–3 practice sessions before the formal test, according to their needs, to increase their familiarity with this iPad-based tools in the future.

3. The criterion-related validity and test–retest reliability of SHTA should be studied for different types of schizophrenia, gender, and age in the future.