Test-retest reliability of physiotherapists using the action research arm test in chronic stroke

Abstract

[Purpose] The aim of this study was to determine whether physiotherapists (PT) scores are consistent over time when using Action Research Arm Test (ARAT) to assess upper limb (UL) function on a videotaped chronic stroke patient. [Participants and Methods] Quantitative correlational study. A convenience-snowball sample of 20 international PT (mean age and experience=32 ± 6.8 and 7.55 ± 7.4 years) used ARAT to score chronic stroke patient’s UL function, observing a video at baseline and again ≈ 2 weeks later. Two sets of non-parametric ordinal data were assessed with Spearman’s (rho) and the alpha (a) value was set at 0.01. Line of equality, Bland-Altman plots and Wilcoxon signed rank test were also considered. [Results] Spearman’s rho was found ≈ 0.78 at a significance level of 0.00. ARAT was scored with a mean difference of 16.6 days and a mean change of 0.6 points was observed. Limits of agreement and coefficient of reproducibility were ± 2.3 and ± 2.6 respectively. The patient’s arm impairment was categorised as moderate and floor or ceiling effects were not detected. [Conclusion] The results suggest that ARAT is consistent, valid and should be used by PT in chronic stroke.

INTRODUCTION

Stroke is a leading cause of disability or death, consisting of progressive neurological signs of cerebral dysfunction that last 24 hours, with a prevalent cause being of vascular origin¹⁾. While stroke mortality in UK accounts for 71 people per 100,000, the incidence rate is 150/100,000^{2, 3)}, contributing to high costs to the NHS, with a total estimated cost of £9 billion annually for care⁴⁾. After stroke, modifications in cortical representation due to spontaneous recovery and neuroplasticity occur over time⁵⁾. Furthermore, 60% of survivors have impaired manual dexterity six months post-stroke and remain unable to perform arm-hand movements^6,7,8,9). Dexterity reflect peoples’ performance in activities of daily living (ADL) and the aim of stroke rehabilitation is regaining upper limb (UL) function^{6,7,8,9,10,11)}.

Determining efficacy of treatments accurately and in a reproducible manner is crucial, thus clinicians remain concerned about the selection criteria of the most repeatable scale and its psychometric properties^{12, 13)}. Furthermore, measuring UL’s impairment in chronic stroke is essential for rehabilitation^{12, 13)}, and several outcome measures (OM) have been discussed with regard to the level of disability and relevance to UL progress^{14,15,16,17,18,19,20,21,22,23,24)}. OM with good psychometric properties and feasible administration time conducted by appropriately trained staff are recommended for stroke by national clinical guidelines²⁵⁾; however, recommendations for tools or specific raters are not provided. The Action Research Arm Test (ARAT)^{14,15,16,17, 19, 20,21,22,23,24)} is one of the most commonly used tools to capture change in hemiplegia. ARAT measures the ability to handle and move qualitatively either large or small objects regarding dexterity and proximal strength for both UL. KNGF (royal Dutch society of physiotherapists) clinical guidelines²⁶⁾ recommend ARAT to be applied by physiotherapists (PT) for UL examination in chronic stroke; however, EBRSR (evidence-based review of stroke rehabilitation)²⁷⁾ and ASA (American stroke association)²⁸⁾ clinical guidelines do not clarify who can use it and report significant floor and ceiling effects. Prior studies have reported the reliability and validity of ARAT to be high^{14,15,16,17, 19, 21,22,23,24)}.

ARAT¹⁸⁾ is based on the upper extremity function test (UEFT)¹⁶⁾, which comprises four subscales: grasp, grip, pinch, and gross. Lyle¹⁶⁾ eliminated items that were inconsistent, reducing the time required to complete the test by 50% for patients with chronic hemiparesis (cortical damage) as a consequence of multiple brain injuries. ARAT’s test-retest reliability (+0.98) was calculated with the Pearson product-moment correlation coefficient (r); a statistical measure useful for interval/ratio data²⁹⁾, and characterised by overestimation of the results due to its inability to indicate the proportion of agreement obtained by chance^{12, 30)}. The disease was not specified by Lyle and statistics were misused. Wagenaar et al.²⁰⁾ enhanced the consistency of the questionnaire, setting a time limit of 1 minute per task, to clarify scores 2 (takes abnormally long) and 3 (normal performance). Another reliability study¹⁴⁾, stated that Lyle’s protocol would be inadvisable and impracticable to therapists with less than five-years-experience. Yet, test-retest reliability was not considered and inter-rater reliability (+0.98) was estimated by three therapists with five years of experience. The intra-class correlation coefficient (ICC) was analysed considering the final score of both hands that implies a deterioration of the results due to a positive estimation ipsilateral to the lesion²¹⁾. Platz et al.¹⁷⁾ developed a standardised protocol identifying the test-retest (+0.96) of ARAT using a mixed small sample with different neurological deficits. Lin et al.²⁴⁾ reported test-retest (+0.99) reliability of ARAT using ICC in chronic stroke, however, they do not provide information on this type of reliability essential for appraising the evidence. Van der Lee et al.²¹⁾ videotaped the sample measurements and utilised ICC, weighted kappas and Bland and Altman plots to report clinically significant results. Yozbatiran²³⁾ tested inter-rater (+0.99) and intra-rater (+0.99) reliability of the standardised protocol of ARAT in chronic stroke (n=12). However, this study set the chronic phase as 3 months after the onset, did not explain statistics, and changed the value for MCID (minimal clinically important difference) from −2.4 to +2.8 points. Above all, only three studies^{16, 17, 24)} established test-retest reliability of ARAT in chronic stroke, but none of them stated to whom the protocol is referred.

The use of a video tape for analysis can enhance the accuracy of the assessment and minimise bias^31,32,33,34). This project will use videotaped evidence of a chronic stroke patient performing ARAT, to determine whether PT’s ARAT scores are consistent over time. The following research question was formulated: Is ARAT consistent when the same physiotherapists use it over time to score upper limb function on a videotaped chronic stroke patient?

PARTICIPANTS AND METHODS

A convenience and snowball sampling of 20 PT (mean age=32 ± 6.8 years, mean experience=7.55 ± 7.4 years) scored a videotaped administration of ARAT performed by a chronic stroke patient. The COSMIN checklist³⁵⁾ provided guidance to the study design, ensuring that all the appropriate information required was reported to ease quality evaluation. The recruitment was via emails and posters. Previous studies^31,32,33) selected licenced PT with a minimum of one-year clinical experience to observe videotaped gait analysis and rate lower limb’s outcome measures in chronic stroke. Therefore, the present study included qualified PT with more than a year of clinical experience regardless of ethnicity that met one of the following criteria: (1) male/female; (2) staff-student members from Salford University; (3) volunteers; (4) specialist/senior or expert; and (5) PT with prior familiarisation with the tool. Prior to testing, information sheets and consent forms were given to the participants.

The measurements were performed in a familiar location and a minimum time interval of one week was defined sufficient regarding the retesting, as being performed by other studies^{17, 24, 31, 33)}, to minimise data recall from the first measurement and not cause maturation. Maturation refers to alternations within the indivisual factors that occur over time which may provide a non-consistent outcome to the later measurement³⁶⁾. The researcher(s) explained the scoring of ARAT to PT. The video was not paused during the observation to reflect a real time patient interaction. The participants (n=15) invited more PT (n=5) to increase the sample size using a non-probability technique of snowball sampling^{36, 37)}. “Intention to Treat” (ITT) was used for incomplete data³⁸⁾. The protocol received ethical approval from School Research Ethics, University of Salford, Salford, United Kingdom. The data collection occurred in May–June 2017.

The experimental (H1) and a null hypothesis (H0) are stated below:

• H1: If the same physiotherapists score the same videotaped performance of a chronic stroke patient twice using ARAT and their scores are similar, then results are due to consistency.

• H0: If the same physiotherapists score the same videotaped performance of a chronic stroke patient twice using ARAT and their scores are not related, then results are due to chance factors.

This is a 2-tailed hypothesis testing, since direction is not yet identified^{29, 37)}, and the level of statistical significance Alpha (a) value was set at 0.01. This study focuses on the stability of ARAT over time, so both direction and relationship among variables (time−ARAT scores) are being predicted naturally without manipulation^{29, 37)}. Due to the study’s correlational design, there are no independent or dependent variables²⁹⁾. Extraneous variables such as situational variables (noise, temperature, lighting degree, cleanliness, neatness, size of the room, convenience of the participants, an isolated room), personal variables (fatigue, mental health, bad/good mood, headaches and dizziness, intelligence and previous knowledge in neurology, anxiety, nerves, concertation, distraction, stress and bad visual acuity) and other variables (interpretation, balance in sound, duration of video, size of the screen quality of video and angle of the camera, surroundings, researcher’s attitude and personality) may have an impact on ARAT performance scores^{32, 37, 39, 40)}; however, not all of them were recorded during the study due to unforeseen circumstances. The ARAT subtests formulate scores in an ordinal set of data^16,17,18).

The correlational study was analysed with the Spearman rank order correlation coefficient (rho)²⁹⁾ and the sum scores of each questionnaire were used to investigate the reliability as proposed by previous research^{15, 27)}. Non-parametric correlation tests are appropriate for ordinal data^{15, 29)} and Spearman’s rank correlation coefficient measures the strength and direction of association between two variables (test, retest)²⁹⁾ using SPSS. The means and standard deviation (SD) were estimated from continuous variables (age, time, experience). However, a high correlation (rho) does not always represent good agreement between two repeated measurements³⁰⁾; thus, the line of equality was also utilised to illustrate visual agreement among the two methods (test against retest)^{31,37, 41)} and the Bland and Altman plot^{30, 41, 42)} that addresses the proportion of agreement between two clinical measurements (2 sets of data) and searches for possible outliers³⁰⁾. It must be noted that the mean difference (δ) is zero, since the same process has been used^{30, 41, 42)}. The Wilcoxon signed rank test was used for ordinal data to assess the mean rank scores (cases) and the statistical significance of differences between scores⁴³⁾, as in previous studies^{17, 19)}.

The video was displayed on two different screens. The time limit of 1 minute was set for the scores to be distinguishable²⁰⁾ and the mean difference between the two measurements was 16.6 days ± 4.9, ensuring independent administration of the tool. Lyle’s protocol¹⁶⁾ was assigned to be scored by PT and no specific instructions were given. In actual clinical practice, a brief review and familiarisation with an instrument prior to its use is mandatory³³⁾; this was achieved by giving adequate time for the PT to familiarise themselves with the protocol. After the first session, PT was advised not to discuss, utilise, or further investigate the questionnaire until their final observation.

RESULTS

From the 20 sets of first data, 19 were collected after the second measurement. None of the PT had prior experience scoring the ARAT. One of the participants dropped out of the study due to personal reasons. A statistical approach known as ITT analysis was found useful on this occasion, since it ignored withdrawals and non-compliance in this study³⁸⁾. The observed tasks were completed within the time limit. The international sample had a median of 5 years-experience as PT and a mean age of 32 years ± 6.8. Their demographics and origin are presented in Table 1 and Table 2 respectively. The reliability of the sum scores was calculated, and the value for the Spearman (rho) was found higher than 0.77 (Table 3), indicating good agreement between the measurements over time. The statistical significance p-value was determined as 0.00, which is small and less than the a-value, {p<a, (0.00<0.01)}, so the H0 was rejected and H1 is accepted.

Table 1. Demographics.

Items	Value
Age (years)	32 ± 6.81
Gender (male/female)	12/8
Years of experience (years)	7.55 ± 7.4
Test retest (days)	16.6 ± 4.9
Staff/students/other	3/14/3
Instrument calibration (times)	2
Duration of the video (min)	12
Size of the screen (inches)	14/120
Image quality (pixels)	1,920 × 1,080/1,280 × 720
During the study: bad visual acuity −anxiety	=3
During the study: ARAT knowledge −neuroscience	=4
Adverse events (headaches, dysphoria, dizziness)	None

SD: standard deviation.

Table 2. Ethnicity and potential threats to internal validity.

County	Number ofparticipants	Location of test	PT	Major factors that may have an impact　on ARAT performance scores
United Kingdom	3	Familiar	2 Staff	Attentiveness-instrumentation
Saudi Arabia	7	Familiar	Students	Fatigue −motivation −drop out
Nigeria	1	Familiar	Student	Visual acuity −fatigue
India	1	Familiar	Student
Brazil	4	2 not familiar	1 Staff	Motivation
Greece	4	1 not familiar	1 Student	Viewing distance

PT: physiotherapist; ARAT: action research arm test.

Table 3. Statistical Analysis (n=20).

1. Non-parametric correlations (Spearman’s rho)	Test	Retest	p value
	1.000	0.775	0.000*
2. Descriptive statistics (mean ± SD)	Test	Retest
	17.4 ± 1.9	16.8 ± 1.85
3. Wilcoxon Signed Rank (Ranks)		Number	Mean Rank	Sum of Ranks
	Negative ranks	7a	4.71	33
	Positive ranks	1b	3	3
	Ties	12c	-	-
	Total	20	-	-
	Z value(retest−test)	−2.15¥
	p value	0.031
4. Coefficients		Unstandardized		Standardized	t value	p value
		B	Std. error	Beta
	Constant	0.09	2.7	-	0.033	0.97
	Mean	0.03	0.16	0.045	0.19	0.85

*p<0.05; ^aretest<test; ^bretest>test; ^cretest=test; ^¥based on positive ranks.

Non-parametric correlations are shown in Table 3. Line of equality showed that the range of scores from 14.00 to 22.00. A non-parametric Wilcoxon signed rank test was conducted to evaluate whether a statistically significant difference existed between the mean ARAT scores of individuals that watched the video over time (H1). The value of mean scores on test (T1) was 17.4, while on retest (T2) decreased to 16.8, thus the mean change is 0.6 (small), suggesting that there are no statistically significant differences between them as shown in Table 3. Table 3 also indicates the ranks and there are seven cases where retest scores were lower than test (a), one case where retest was higher than test (b) and twelve which were the same (c). Table 3 shows a z-value of −2.154 associated with a p-value of 0.031, that is more than the a-value of 0.01 (p>a), so the H0 that there are no statistically significant differences between the scores of the sample is accepted. Coefficient of reproducibility (CR) is expressed as two times the difference of SD, CR=2 × SD and the 95% limits of agreement (LOA) are (δ=0) −1.96 SD and (δ=0) + 1.96 SD^{30, 41, 42)}. CR=2SD (d2-d1)³⁰⁾=(−1)² +1+(2)² +(1)² +(1)² +(4)² +(3)² +(1)² ÷ N=34÷20=1.7 →CR=2 ×(√1.7) →CR=±2.6. The LOA are upper line (green): 1.96SD (+2.31) and lower line (red): −1.96SD (−2.31). The plot indicates two outliers from the LOA. A linear regression procedure was administrated to detect if there is any proportional bias, since there is a trend of points being above the mean difference. As can be seen in Table 3, the t-value (t=0.190) is linked with a statistical significance of p=0.851, greater than the a-value (0.01), so again H0 (null hypothesis) is accepted. Therefore, there is no proportional bias and there is a certain level of agreement between measurements.

DISCUSSION

The study investigated test-retest reliability of PT using ARAT in chronic stroke. Yozbatiran’s standardised instructions²³⁾ did not correspond to the videotape assessment, so Lyle’s decision rules¹⁶⁾ in conjunction with the time limit of 1 minute²⁰⁾ were implemented. The correlation coefficient (rho) of ≈ 0.78 linked with a statistical significance of 0.00, shows that ARAT is a reliable, consistent, and reproducible OM when used by PT in chronic stroke. The line of equality showed a positive correlation considering that the results lie along a line. However, relative reliability may give misleading results and a positive correlation is not necessarily related to good agreement^{30, 37, 41, 42)}. A previous study¹⁶⁾ overestimated test-retest reliability, using Pearson correlation that applies to parametric data, without converting the non-parametric ARAT data. The statistically significant results contradict Hsieh’s findings¹⁴⁾, which suggested that therapists with less than five years of experience are not consistent using ARAT. Our results are relatively consistent with previous findings²⁴⁾; the mean change of 0.6 points associated with a median of 16 days in this study is consistent with the mean change of 0.3 points and a median of 15 days found by van der Lee⁴⁴⁾. CR is not mentioned in their study⁴⁴⁾, the mean difference (δ) is not considered 0 in plots, as usual for repeated measurements³⁰⁾, and LOA (−5.7, +6.2) were greater than the MCID. The Wilcoxon signed rank test was used to analyse descriptive statistics and mean rank scores. Lin²⁴⁾ targeted the examiners for ARAT without defining terms such as ‘specially trained PT’, thus Lin’s²⁴⁾ sample is not representative and their results cannot be generalised due to sampling error. To the best of our knowledge, none of the previous studies^{16,17,18, 21, 23, 24, 44)} considered Bland and Altman plots and CR to estimate ARAT’s test-retest reliability.

PT with multiple nationalities were involved and either a laptop or a projector was utilised to play the videotape in each case accordingly. Situational variables were controlled; measurements being performed late morning provided an adequate degree of lighting, while the rooms for PT were clean, quiet and ensured a comfortable temperature to avoid distraction and injuries⁴⁵⁾. The individuals were observed naturally and examined verbally for their health (headaches, injury, convenience) and well-being (stress, thoughts, feelings). Natural observation refers to the process when the researcher joins the group to obtain a deeper understanding of PT⁴⁶⁾. Fatigue and distance from the display are two of the personal variables not recorded that may have influenced the outcome of the study; 8 PT that scored the protocol twice were Muslims and measurements were conducted during Ramadan. Fasting is directly linked with fatigue and headaches⁴⁷⁾, however, none of the PT complained about headaches. Three PT (1 Greek, 2 Brazilian) that scored the protocol were not familiar with the environment. The mean difference was 16.6 ± 4.9 (days) when the measurements performed, with a range of days (7–24). More than one week was highlighted by previous studies³³⁾ to reduce the potential for memory effects. Although, standardisation regarding maturation of a sample has not been yet identified and it basically depends on the nature of research⁴⁸⁾. Most PT that participated in the study were students and staff members, so knowledge acquisition may comprise a factor of maturation, particularly in longer term changes⁴⁸⁾. However, the mean difference of 16.6 days among the two measurements was within the time period suggested by previous studies^{33, 49)}. Motivation, reward, and interest are three extraneous variables that may have influenced the scoring procedure and were not assessed during the study.

Considering potential threats to internal validity, instrumentation and history may influence the findings of a study⁴⁸⁾. On this occasion, instrumentation occurs when the device used alters over time, so instrumental bias may have affected the results. In total, 39 observations were performed (1 drop out): 16 observations in which the video was played through a 14-inch laptop and 18 observations via a 120-inch projector, matched for both test-retest. The device provided was changed from projector to laptop for 4 of the observations regarding retesting. While Neuman⁵⁰⁾ suggests that larger screens with a poor-quality image may provide a greater sense of presence, enjoyment, and intense responses, Lombard⁵¹⁾ gives more value to smaller displays with high resolution. Consequently, stability and balance within electronic devices were verified in this study and instrumentation was not a limiting factor.

Chanubol et al.¹⁹⁾ defined a classification scale to track the severity of arm impairment based on ARAT, with a score less than 10 determined severe. Using Chanubols’ classification system in conjunction with the range of scores (14−22) and the mean scores of 16.8 and 17.4 respectively, the present study categorises patient’s arm impairment as moderate. This was confirmed by Nijland et al.⁵²⁾, who compared the Wolf Motor Function Test (WMFT) with ARAT and recommended the protocol for patients with moderate hemiparesis, while van der Lee et al.²²⁾ stated that ARAT may not be sensitive enough to identify change in chronic stroke patients approaching severe or normal arm function. International guidelines²⁷⁾ and previous studies^{22, 24, 52)} demonstrated that ARAT has notable floor and ceiling effects when capturing changes in severe or mild stroke patients. Floor and ceiling effects of ARAT are calculated as 5% of the scale, and to be identified, 20% of the total scores need to be appointed above or below the LOA. Bland and Altman plot showed two outliers, so there are no significant floor and ceiling effects in this study and the patient is likely to have moderate impairment⁵²⁾, in agreement with Chanubol et al.¹⁹⁾, but if the patient had severe or near normal UL impairment, this study would have given robust evidence regarding floor and ceiling effects. This study did not confirm the patient’s non-affected side.

Interestingly, the resulting value of CR=± 2.6 points approaches numerically the LOA (± 2.3) in this study, confirming that these two values are statistically related to each other, as proposed by other studies^{53, 54)}. It must be noted that CR does not represent the MCID⁵⁵⁾, and clinicians remain subjective while estimating CR, since many studies proposed the equation differently^{30, 41, 42, 53)}. CR is a psychometric property of ARAT which reflects the instrument’s ability to capture a difference of ± 2.6 points, while MCID is the selected minimal value, previously set by clinicians to determine whether the outcome has been clinically significant due to intervention^{53, 55)}. The value of CR=± 2.6 in this study is less than the MCID=5.7 points of ARAT (10% of the scale)²¹⁾ (CR<MCID), verifying that ARAT is a valid outcome measure that accurately identifies real change. Consequently, responsiveness and reproducibility are directly connected to each other^{53, 54)}. Since a trend being above the mean difference was detected, a linear regression was completed to search for proportional bias. Bland and Altman plot identifies two outliers likely due to the variability of the measurement. One of the outliers had bad visual acuity (confounding variable). No proportional bias was found and the LOA (± 2.3) was less than the MCID of 5.7 points, so ARAT is capable of detecting clinically relevant changes.

However, the findings in this study must be interpreted with caution. Using a videotape for analysis automatically reduces measurement variability, eliminating the need for the patient to perform the ARAT multiple times that may cause fatigue³³⁾. Non-probability sampling is vulnerable to selection bias, the PT were not blinded to the video, but the allowed time interval was considered adequate for both data memorisation and sample maturation. Despite the robust statistical methods used, the small a-value (0.01) increased the confidence for significant results. Three researchers with previous experience scored the ARAT, which may lead to biased results, and captured only the affected side of the patient. Another potential limitation may be instrumentation. The results would have yielded a more precise estimation of reliability if the two outliers have been excluded. Assuming this sample with a mean age of 32 ± 6.81 years and a mean experience of 7.55 ± 7.4 years to be representative of the whole population, then the results can be generalised to a wider population. Despite these limitations, the findings of this study suggest that PT should use the ARAT for the examination of UL function in moderate chronic stroke.

The ARAT demonstrates good test-retest reliability using statistical analysis with Spearman’s rank order correlation coefficient, Bland and Altman plots and linear regression. These results illustrate good test-retest and are consistent. The included sample accurately reflects a larger population with a mean age of 32 ± 7 years, mean experience of 7.55 ± 7.4 years and six different nationalities. Therefore, the results are representative internationally, and PT should use and score the protocol. However, the reported results should be interpreted with caution due to the small sample size and the selected type of sampling.

Conflict of interest

The author declares that he has no competing interests.