Reliability and minimal detectable change for a smartphone-based motor-cognitive assessment: implications for concussion management

Abstract

Our purpose was to investigate the reliability and minimal detectable change characteristics of a smartphone-based assessment of single-task and dual-task gait and cognitive performance. Uninjured adolescent athletes (n=17; mean age=16.6, SD=1.3 years; 47% female) completed assessments initially and again four weeks later. We collected data via an automated smartphone-based application while participants completed a series of tasks under (1) single-task cognitive, (2) single-task gait, and (3) dual-task cognitive-gait conditions. The cognitive task was a series of continuous auditory Stroop cues. Average gait speed was consistent between testing sessions in single-task (0.98, SD=0.21 vs. 0.96, SD=0.19 m/s; p=0.60; r=0.89) and dual-task (0.92, SD=0.22 vs. 0.89, SD=0.22 m/s; p=0.37; r=0.88) conditions. Response accuracy was moderately consistent between assessments in single-task standing (82.3% accurate, SD=17.9% vs. 84.6% accurate, SD=20.1%; p=0.64; r=0.52) and dual-task gait (89.4% accurate, SD=15.9% vs. 85.8% accurate, SD=20.2%; p=0.23; r=0.81) conditions. Our results indicate automated motor-cognitive dual-task outcomes obtained within a smartphone-based assessment are consistent across a one-month period. Further research is required to understand how this assessment performs in the setting of sport-related concussion. Given the relative reliability of values obtained, a smartphone-based evaluation may be considered for use to evaluate changes across time among adolescents post-concussion.

Introduction

Gait evaluations can be a clinically useful approach across a broad spectrum of pathological conditions.¹ Dual-task gait, where an individual is asked to walk and complete a concurrent mental task, provides the ability to understand motor-cognitive functioning within the context of disease progression, recovery, or treatment responses.² Traditionally, gait analyses occur within a biomechanics laboratory, where detailed movement profiles can be derived. However, the feasibility of this approach is limited in many clinical settings due to time constraints, as well as technical and financial burdens. As a result, researchers and clinicians have sought translational methods that use existing principles to move toward everyday devices, so that dual-task assessments may be incorporated into routine clinical practice in a ubiquitous and widespread manner.³

Dual-task gait assessments appear to be particularly useful to understand recovery from sport-related concussion.⁴ Following a concussion, dual-task gait deficits continue to persist despite concussion symptom,⁵ single-task,⁶ static balance control,⁷ neurocognitive,⁸ and neurophysiological⁹ recovery. Furthermore, understanding cognitive function during movement, rather than while stationary, may be a better reflection of how an athlete will perform during sports.¹⁰ Based on previous work examining cognitive task complexity during a dual-task, the use of an auditory Stroop task (response to the word “high” or “low” presented in a high or low pitch) may be able to provide an optimal level of cognitive perturbation to accurately identify differences between individuals who have sustained a concussion and uninjured individuals.¹¹ In comparison to other commonly used cognitive tests within a dual-task paradigm after concussion such as spelling backwards or serial subtraction,^12–14 a continuous auditory Stroop task also provides objective measurement of response time, rate, and accuracy, while providing the ability to identify persistent post-concussion deficits beyond clinical recovery.⁸ Yet, no established methods have been developed to provide reliable and objective cognitive performance information, such as response time and accuracy, within a routine clinical dual-task examination.

Translating dual-task approaches from the laboratory to the clinic may result in improved recovery recognition, ultimately leading to better return-to-play procedures. Currently, the majority of clinicians do not use dual-task analyses when evaluating individuals with concussion,^15–17 likely due to the burden of traditional gait analysis approaches or ability to quantify cognitive function during movement, indicating a need for ubiquitous and accessible analysis methods. Determining test-retest reliability and minimal detectable change (MDC) for both motor and cognitive elements within a dual-task paradigm among healthy athletes will help clinicians interpret post-injury performance. For example, using the Balance Error Scoring System researchers have identified a MDC range of 7-9 errors represents a meaningful change across time.¹⁸ However, these values are not available for cognitive-motor dual-task tests. Dual-task deficits persist after a concussion and it is theorized that these may relate to risk of incurring additional injuries after returning to sports.¹⁹ Given that existing tests are not equipped to identify those at risk for a further injury,²⁰ there is a need to understand the reliability and clinical interpretation of values obtained within an objective, integrated, and automated dual-task paradigm to improve clinical utility. Therefore, the purpose of this study was to investigate the reliability and minimal detectable change characteristics of a valid²¹ smartphone-based assessment of single-task and dual-task gait and cognitive performance among uninjured youth athletes as an automated field-based medical device.

Methods

Study Design and Participants:

We conducted a repeated measures study of adolescent athletes who were uninjured at the time of enrollment. All participants were tested at two time points approximately one month apart. No participant reported any new injuries between the two testing sessions. The re-test timeframe was selected because it is the expected period in which most adolescents will report symptom resolution following a concussion,^22,23 facilitating comparisons with future studies among youth with concussion. In addition, we aligned with the timeline of previous studies evaluating test-retest reliability of other measurements for concussion evaluations.^24–26 Uninjured adolescent athletes were recruited for participation in the study using a network of high school athletic trainers who posted recruitment flyers in their athletic training facilities. Participants were excluded if they were <12 or ≥19 years of age, if they had sustained a concussion in the past year, if they had an uncorrected visual impairment, or if they were experiencing any injury that withheld them from organized sport participation. The University of Colorado Multiple institutional review board approved of the study protocol (Protocol #18-0181) prior to commencement (may need to blind prior to peer review). All participants and their parents/guardians (if <18 years of age) completed written informed consent/assent procedures.

Experimental Approach:

We collected gait and cognitive performance data using a smartphone-based gait analysis application while participants completed a series of tasks under different conditions, in the following order: (1) single-task cognitive condition, (2) single-task gait condition, and (3) dual-task cognitive-gait condition. This approach is similar to previously described experiments,^14,21 although we used a different cognitive task in the current study. Participants completed four trials per condition.

During the single-task cognitive condition, participants were instructed to remain still while standing comfortably with their feet together and their eyes open for 30 seconds. During the trial, they listened and responded to a series of auditory Stroop tasks played through headphones attached to a smartphone (Samsung Galaxy S8; Gait Analyzer application²⁷), affixed to the participants’ lower back with a running belt (K1 Sport Belt, Sporteer Kinetic, Scottsdale, AZ, USA).

As with previous studies,^11,28 a computer-recorded voice spoke the words “high” or “low” in a high pitch or low pitch. Participants were instructed to identify the pitch of the word, regardless of the actual word spoken, resulting in congruent (word-pitch matching) and incongruent (word-pitch mismatching) cues. An initial randomly selected cue was played at the beginning of the trial, and randomly selected cues continued to play one second after each response for the duration of each 30-second single-task cognitive trial.

During the single-task gait condition, participants walked at a self-selected pace along a pathway approximately 6 meters long toward an object placed on the floor (a blank piece of paper). They were instructed to walk toward the object at a self-selected and normal pace, walk around it, and return to the initial position, resulting in a total distance walked of approximately 12 meters. Before the start of each trial, a test administrator hit a button to begin recording tri-axial accelerometer and verbal response data using the smartphone application. Once the trial was complete, the test administrator hit a button on the smartphone to stop recording. Data were recorded while they walked, and stopped once they returned to the initial position and stopped walking.

During the dual-task gait condition, participants completed the same walking task as described in the single-task gait condition while simultaneously responding to the continuous auditory Stroop cues, as described above, played throughout the duration of each trial. Participants were not instructed to focus on the cognitive or the gait task specifically but were told to walk and verbally respond to the cues simultaneously.

Data Analysis:

We analyzed cognitive task performance as the mean across trials in single-task and dual-task conditions separately. Our primary cognitive outcomes of interest were response time and response accuracy. Following each auditory Stroop cue, the Android Speech Recognizer application programming interface (API) detected and identified the verbal responses provided by participants as either “high”, “low”, or unknown. The onset, or the smartphone’s system time in milliseconds, for each auditory cue (stimulus) and verbal response was also recorded. Subsequently, automated analysis within the Gait Analyzer application classified correct, incorrect, or unanswered responses on each trial. Overall cognitive accuracy was calculated as the total number of correct responses divided by the total number of responses in single-task and dual-task conditions, excluding those responses marked as unanswered (27% of all cues provided were unanswered, most commonly occurring at the end of the trial). We then reviewed each trial individually to verify the application classified each response correctly and corrected if misidentification occurred. Due to errors in the voice recognition software (e.g. recognizing the participant response “low” as “slow”, marking it incorrect), the automated analysis misclassified 20% and 23% of responses in the single-task and dual-task conditions, respectively. Only responses corrected manually by the research team were used in our analyses. This error has been rectified in subsequent phone application algorithms used for data collection. To examine response time, we calculated the difference between the stimulus onset and the verbal response onset.²⁹ We also examined the response rate within each trial, expressed as the number of responses per minute (60 x number of responses during a trial ÷ trial time). In addition, we calculated response accuracy and response times within congruent and incongruent auditory Stroop conditions separately.

As with past work, we calculated temporal distance gait metrics during steady-state gait.^14,21,30 All gait parameters were calculated online within the Android application in real-time, with all outcome measures saved and exported for further analysis. In prior work, we observed high to very high validity compared to a wearable sensor suite (Pearson r = 0.77-0.98) and high to very high test-retest reliability (ICC = 0.81-0.95) for single-task and dual-task spatio-temporal gait variables.²¹ These past studies relied on offline processing techniques, while the current study integrated the existing processing algorithms into the application itself, so analyses were performed in an automated, real-time fashion. Briefly, positive peaks in the filtered accelerometer’s anterior-posterior axis were identified as steps. The step length (SL) was computed based on the vertical displacement across each step cycle (h) and the participant’s leg length (l) using the relationship:

$$SL=2\ast \sqrt{2\ast h\ast l-h^2}.$$

As the step time (ST) was based on the time differences between identified steps, average gait speed (AGS) was therefore calculated as:

$$AGS={\sum }_i\frac{S{L}_i}{S{T}_i}.$$

Our primary outcome variable was average gait speed, given its utility in concussion-related investigations using motion analysis and inertial sensor systems.⁴ We also calculated cadence, step length, step time, and average trial distance as secondary outcomes in both single-task and dual-task conditions.

Statistical Analysis:

Data are presented as means (standard deviations) and number included (corresponding percentage) for continuous and categorical variables, respectively. To examine mean difference changes between the two assessments, we used paired samples t-tests. We interpreted a significant mean change between assessments at p = 0.002, given the large number of dependent variables included in our investigation (n=24 test-retest dependent variable comparisons). We then evaluated the linear correlation between assessments on each outcome variable using bivariate Pearson correlations. When interpreting the correlations, we used the following cut points: <0.39 (low), 0.4-0.59 (moderate), 0.60-0.79 (moderately high), and ≥0.80 (high).³¹ In addition, we assessed test-retest consistency using Cronbach’s alpha and intraclass correlation coefficient (ICC [2, 4]) values.

We calculated minimal detectable change (MDC) scores at the 90% confidence level, given the ability of this approach to asses performance changes or treatment impacts.³² We calculated MDCs as: ${MDC}_{90}=\left(standard error of the measurement\right)\times \left(1.65\right)\times \left(\sqrt{2}\right)$.^32,33 For further interpretation, we also calculated MDCs at the 80% confidence level, consistent with previous studies,³⁴ as: ${MDC}_{80}=\left(standard error of the measurement\right)\times \left(1.282\right)\times \left(\sqrt{2}\right)$. All statistical tests were 2-sided and performed using Stata version 15 (StataCorp., College Station, TX, USA).

Results

A total of 17 adolescents completed the multimodal cognitive-motor dual-task assessment protocol at both time points approximately one month apart. No participants were lost to follow-up. The second assessment took place approximately one month following the first assessment, and all participants were adolescents (age range = 13.8 – 18.5 years; Table 1). After adjusting our significance level for the large number of dependent variables comparisons, we did not observe significantly differences on any variable when comparing mean values between time points (Table 2).

Table 1.

Participant characteristics (n=17). Data are presented as mean (standard deviation) and number included (corresponding percentage).

Variable	Mean (sd) or n (%)
Sex (female)	8 (47%)
Age (years)	16.6 (1.3)
Time between tests (days)	27.5 (0.5)
Height (cm)	168.8 (10.3)
Mass (kg)	63.3 (12.5)
BMI (kg/m2)	22.1 (3.0)
Leg length (cm)	89.5 (6.9)
History of concussion	3 (18%)
Diagnosed ADHD	3 (18%)

Table 2.

Comparison of mean values obtained at each assessment, as well as the linear agreement (Pearson r) of each variable between assessments.

Variable	Assessment 1 Mean (SD)	Assessment 2 Mean (SD)	P value	Test-retest correlation (r)
Gait performance without cognitive task completion (single-task)
Average gait speed (m/s)	0.98 (0.21)	0.96 (0.19)	0.60	0.89
Cadence (steps/min)	105.2 (12.6)	102.1 (10.7)	0.40	0.43
Step length (m)	0.55 (0.10)	0.55 (0.09)	0.88	0.78
Step time (s)	0.58 (0.08)	0.60 (0.06)	0.52	0.85
Average trial distance (m)	12.2 (3.4)	13.5 (2.3)	0.03	0.72
Gait performance with cognitive task completion (dual-task)
Average gait speed (m/s)	0.92 (0.22)	0.89 (0.22)	0.37	0.88
Cadence (steps/min)	104.1 (11.5)	98.7 (13.1)	0.03	0.60
Step length (m)	0.52 (0.09)	0.52 (0.09)	0.34	0.77
Step time (s)	0.58 (0.07)	0.63 (0.09)	0.05	0.52
Average trial distance (m)	12.4 (3.6)	13.3 (2.6)	0.25	0.49
Cognitive performance while standing (single-task)
Response time (ms)	1396 (165)	1300 (132)	0.03	0.50
Response accuracy (% correct)	82.3% (17.9%)	84.6% (20.1%)	0.64	0.52
Response rate (responses/min)	12.0 (3.9)	11.3 (4.5)	0.41	0.64
Response time: congruent cues (ms)	1349 (167)	1231 (108)	0.02	0.23
Response accuracy: congruent cues (% correct)	91.9% (11.6%)	94.7% (12.3%)	0.40	0.45
Response time: incongruent cues (ms)	1425 (178)	1368 (175)	0.17	0.62
Response accuracy: incongruent cues (% correct)	70.6% (25.4%)	77.7% (28.4%)	0.30	0.56
Cognitive performance during gait (dual-task)
Response time (ms)	1281 (142)	1327 (118)	0.05	0.79
Response accuracy (% correct)	89.4% (15.9%)	85.8% (20.2%)	0.23	0.81
Response rate (responses/min)	13.9 (5.0)	13.6 (4.6)	0.83	0.50
Response time: congruent cues (ms)	1238 (153)	1279 (108)	0.11	0.76
Response accuracy: congruent cues (% correct)	94.8% (9.7%)	98.7% (4.2%)	0.18	0.18
Response time: incongruent cues (ms)	1361 (194)	1399 (185)	0.44	0.44
Response accuracy: incongruent cues (% correct)	84.4% (27.6%)	76.1% (32.7%)	0.18	0.69

For our primary outcome variables of interest, no significant mean differences were observed for average gait speed (Figure 1), response time (Figure 2A and 2D), response accuracy (Figure 2B and 2E), or response rate (Figure 2C and 2F) under single-task and dual-task conditions. High test-retest correlations were observed for single/dual-task average gait speed and dual-task response accuracy (Table 2). Moderately high test-retest correlations were observed for single-task/dual-task step length, single-task cognitive response rate, single-task incongruent response time, dual-task overall/congruent response time, dual-task response rate, and dual-task incongruent response accuracy (Table 2).

Figure 1.

Violin plots describing the gait primary outcome variable (average gait speed) under (A) single-task conditions and (B) dual-task conditions at each time point. Violin plots describe the distribution of measurements in each group, and data are presented as median (center dot, with corresponding number included) and interquartile range (box around the median). The shaded area represents the probability density of data at each level of the scale, smoothed using a kernel density estimator.

Figure 2.

Violin plots describing Stroop primary outcome variables (accuracy, response time, response rate) under single-task (A, B, C) and dual-task (D, E, F) conditions at each time point. Violin plots describe the distribution of measurements in each group, and data are presented as median (center dot, with corresponding number included) and interquartile range (box around the median). The shaded area represents the probability density of data at each level of the scale, smoothed using a kernel density estimator.

MDC₉₀ values (Table 3) indicated reasonable changes across time for average gait speed were 0.12 m/s (single-task) and 0.13 m/s (dual-task), and response time changes were 308 ms (standing) and 109 ms (dual-task gait).

Table 3.

Cronbach’s alpha, intraclass correlation (ICC), and minimal detectable change values (presented at 90% and 80% confidence levels) for the gait and cognitive variables of interest under single-task and dual-task conditions.

Variable	Cronbach’s alpha	ICC (95% CI)	Minimal Detectable Change: 90% confidence level	Minimal Detectable Change: 80% confidence level
Gait performance without cognitive task completion (single-task)
Average gait speed (m/s)	0.94	0.94 (0.83, 0.98)	0.12	0.09
Cadence (steps/min)	0.31	0.56 (0.12, 0.84)	22.5	17.5
Step length (m)	0.87	0.88 (0.65, 0.96)	0.08	0.06
Step time (s)	0.09	0.10 (0.00, 0.68)	0.16	0.12
Gait performance with cognitive task completion (dual-task)
Average gait speed (m/s)	0.94	0.94 (0.83, 0.98)	0.13	0.10
Cadence (steps/min)	0.74	0.70 (0.18, 0.89)	14.9	11.5
Step length (m)	0.87	0.87 (0.65, 0.95)	0.07	0.06
Step time (s)	0.66	0.61 (0.02, 0.86)	0.11	0.09
Cognitive performance while standing (single-task)
Response time (ms)	0.52	0.59 (0.00, 0.85)	307.5	238.9
Response rate (responses/min)	0.78	0.78 (0.40, 0.92)	4.32	3.36
Response accuracy (% correct)	0.68	0.70 (0.07, 0.90)	23.5%	18.2%
Cognitive performance during gait (dual-task)
Response time (ms)	0.87	0.85 (0.57, 0.95)	108.6	84.4
Response rate (responses/min)	0.67	0.68 (0.09, 0.89)	6.64	5.16
Response accuracy (% correct)	0.88	0.88 (0.67, 0.96)	14.4%	11.2%

Discussion

The results from our investigation indicate that several measures obtained from an automated cognitive and motor testing paradigm are stable across a one-month period. These results may also aid in interpretation of test-retest change across time in the context of post-concussion recovery monitoring, although future studies are required to understand how these measures change across time in an injured group of athletes, relative to this uninjured group. However, given the consistency of measurements obtained across one month, these data provide test characteristic information for an assessment that may have future clinical utility among adolescents. Finally, further work is needed to develop normative values so the absolute values can be compared, and the level of impairment can be determined in a post-injury setting.

The primary intention of this investigation was to provide clinical context for healthcare professionals who intend to incorporate automated and objective dual-task approaches into their evaluations. The feasibility of this approach is high, compared to other studies that have evaluated dual-task motor-cognitive function, given the equipment demands are based on the use of a commonly available device (i.e. a smartphone), rather than a motion analysis laboratory or inertial sensor suite.^35–37 This high feasibility due to the ease of use, readily available technology, and automation of important clinical variables (e.g. average gait speed⁴) may allow for widespread implementation of dual-task approaches, extending existing clinical practices.¹⁵ Given that single-task movement deficits can exist after concussion without any static balance deficits,³⁸ and that examining dual-task function can identify post-injury deficits beyond typically expected clinical resolution after concussion,⁴ implementation may lead to improved clinical decision-making. Furthermore, our MDC and reliability results are similar to that found among health young adults and adolescents measured under dual-task conditions with traditional motion analysis.³³ For example, our single-task and dual-task average walking speed Cronbach’s alpha (single-task= 0.94; dual-task= 0.94) and MDCs (single-task= 0.09 m/s; dual-task= 0.10 m/s) were similar to previous reports (Cronbach’s alpha = single-task: 0.94; dual-task: 0.96; MDC= single-task: 0.07 m/s; dual-task: 0.06 m/s). Not all variables demonstrated high reliability across testing sessions. Specifically, single-task cadence and single-task step time each provided poor reliability despite similar mean values at each test. Upon further inspection, this may have been due to the differing directionality for individual participants on this variable at each testing time point: 59% had a higher single-task step time value at the second test relative to the first, while 41% had a lower value. In light of this lack of consistency for single-task step time, further work to ensure it demonstrates appropriate reliability is needed to ensure utility for injured patients.

The calculation of MDC values for our primary outcome variables provides an approach for clinicians to use in interpreting performance of adolescent athletes across time. For example, when assessing an athlete over a one-month period of time, a dual-task average gait speed change > 0.13 m/s would reflect a meaningful biological change, particularly in the context of understanding treatment efficacy across time. Other studies have evaluated MDC values for balance and gait measures for concussed populations. Specifically, one study observed the inter-rater MDC value for a single-task tandem gait test completion time was 0.38 seconds among collegiate athletes with concussion,³⁸ while another reported the MDC using a portable forceplate to quantify sway velocity during the BESS was 0.59 degrees/second.³⁹ Our work builds upon these previous studies, and extends this type of approach to a clinically viable dual-task gait test battery. Although our study was not designed to understand the level of impairment following an injury, others have developed clinical cutoff values to help to distinguish between concussed and control participants, such as a time of 22 seconds being a relatively accurate (81% accuracy) cutoff for dual-task tandem gait completion time.¹² Future studies should continue to incorporate these approaches to dual-task gait evaluations in order to better place the variables obtained into an appropriate context and augment clinical decision-making.

The cognitive element within a dual-task paradigm has been largely overlooked within the concussion literature, as the majority of studies focus on quantifying the motor component of a motor-cognitive dual-task study.^4,37 Our work suggests that dual-task cognitive response accuracy during a continuous auditory Stroop task is relatively consistent across a month of testing. Interestingly, however, we observed more accurate and faster responses to the auditory Stroop task during dual-task relative to single-task conditions. While there may be multiple reasons for this observation, we suspect that it may relate to a practice effect as participants completed the single-task condition first. The value of including a set of cognitive outcomes within a dual-task study is that researchers and clinicians can directly evaluate the additive effect of a perturbation. Thus, quantification can occur regarding how cognitive performance is affected between static and dynamic conditions, as well as how motor performance is affected between conditions where there is and is not a cognitive stimulus present. These comparisons can facilitate the calculation of dual-task costs (cognitive-motor tradeoff during a task),⁴⁰ and provide insights into task prioritization and may help optimize treatment pathways.

There were several limitations to the use of the automated assessment. The mean trial distance was longer during the second assessment relative to the first assessment. The difference likely arises within the turning section of the task (at the end of the walkway during the trial) and how our selected algorithm calculates step length, as well as the manual nature of starting/stopping the test by the test administrator. Future studies should seek to improve this detection and provide more reliable estimates for distance travelled. Second, some of the cognitive responses (~20-23%) were misclassified as “incorrect” when in fact, the response was correct, and therefore further refinement of the application is necessary. All responses within the current investigation were verified manually and adjusted where appropriate to ensure accuracy of our data. However, this timely process reduces the ecological validity of this approach. Third, the response time changes were moderate, suggesting a practice effect within the single-task cognitive condition. The clinical significance of a 96 ms faster response time while standing at the second assessment compared to the first should warrant caution regarding changes attributed to treatment effects or injury recovery. Finally, our study sample was comprised of 17 uninjured adolescent athletes recruited from a single local high school district and therefore these results cannot be generalized to other age populations or geographic areas. Given this small sample, we could not examine other factors which may have affected gait performance, such as sex or gender.^41,42 The application is currently available on Android and IOS platforms to download. The current study used a single smartphone device for data collection purposes to ensure consistency. However, in order to improve generalizability across software platforms and smartphone models, future work should compare results and consistency across different devices.

In conclusion, our study indicates it is feasible to conduct an automated dual-task assessment among adolescent athletes using a single smartphone device. Although further studies are necessary to ensure reliability and consistency over time, clinicians may consider measuring dual-task function in order to augment their evaluation techniques with an objective battery of measurements that may help to understand the way an athlete will perform in more realistic, sport-like environments.