Measurement Precision and Efficiency of Computerized Adaptive Testing for the Activities-specific Balance Confidence Scale in People With Stroke

Abstract

Objective

Administrative burden often prevents clinical assessment of balance confidence in people with stroke. A computerized adaptive test (CAT) version of the Activities-specific Balance Confidence Scale (ABC CAT) can dramatically reduce this burden. The objective of this study was to test balance confidence measurement precision and efficiency in people with stroke with an ABC CAT.

Methods

We conducted a retrospective, cross-sectional, simulation study with data from 406 adults approximately 2 months post-stroke in the Locomotor-Experience Applied Post-Stroke trial. Item parameters for CAT calibration were estimated with the Rasch model using a random sample of participants (n = 203). Computer simulation was used with response data from the remaining 203 participants to evaluate the ABC CAT algorithm under varying stopping criteria. We compared estimated levels of balance confidence from each simulation to actual levels predicted from the Rasch model (Pearson correlations and mean standard error [SE]).

Results

Results from simulations with number of items as a stopping criterion strongly correlated with actual ABC scores (full item, r = 1, 12-item, r = 0.994; 8-item, r = 0.98; 4-item, r = 0.929). Mean SE increased with decreasing number of items administered (full item, SE = 0.31; 12 item, SE = 0.33; 8 item, SE = 0.38; 4 item, SE = 0.49). A precision-based stopping rule (mean SE = 0.5) also strongly correlated with actual ABC scores (r = 0.941) and optimized the relationship between number of items administrated with precision (mean number of items 4.37, range [4–9]).

Conclusion

An ABC CAT can determine accurate and precise measures of balance confidence in people with stroke with as few as 4 items. Individuals with lower balance confidence may require a greater number of items (up to 9) which could be attributed to the Locomotor-Experience Applied Post-Stroke trial excluding more functionally impaired persons.

Impact

Computerized adaptive testing can drastically reduce the ABC test’s administration time while maintaining accuracy and precision. This should greatly enhance clinical utility, facilitating adoption of clinical practice guidelines in stroke rehabilitation.

Lay Summary

If you have had a stroke, your physical therapist will likely test your balance confidence. A CAT version of the ABC scale can accurately identify balance with as few as 4 questions, which takes much less time.

Introduction

Measurement of balance confidence with the Activities-specific Balance Confidence Scale (ABC) is highly recommended by clinical practice guidelines in stroke rehabilitation.^1,2 These guidelines, supported by the American Physical Therapy Association’s Academy of Neurologic Physical Therapy, are the result of the ABC’s strong association with other International Classification of Functioning, Disability and Health activity and participation domain measures in persons post-stroke across all rehabilitation settings.^3–8 The ABC also has high test–retest reliability,^5,6 a lack of floor and ceiling effects,⁵ and can distinguish between fallers and non-fallers^4,9 in the post-stroke population. Yet, despite the mounting evidence and professional association recommendations, adoption of measurements like the ABC in clinical practice is often limited because clinicians perceive test administration time as a barrier.^10,11 Originally designed for clinical practice, the ABC scale is relatively short with 16 items. However, the large rating scale (0–100% balance confidence) per item increases response burden for patients and can cause test administration times to exceed the commonly reported 20 minutes.¹² Unfortunately, a test lasting longer than 20 minutes in a busy clinical setting where physical therapists are expected to complete multiple outcome measures quickly faces a substantial barrier to adoption.

Computerized adaptive tests (CAT) provide an alternative to fixed-length measurement tools that can dramatically reduce administrative burden.^13–15 Specifically, CATs can reduce test administration time by customizing the way items are presented for individual patients, allowing them to skip items that are too easy or difficult. CATs accomplish this customization by using a software algorithm designed to select questions tailored to an individual’s responses and estimating their overall score until a desired level of precision is reached.¹⁴ This improved measurement efficiency is why CAT use has become heavily advocated for in rehabilitation medicine.^16,17

However, for a measure to be clinically useful as a CAT, clinicians need to be able to compare scores between or within patients when individuals are provided different items or a different number of items. This is possible when items in a CAT from a measurement tool meet the requirements of item response theory and are sufficiently unidimensional (only measuring 1 common trait).¹⁶ Previous studies examining the ABC using Rasch analysis, an item response theory model, have consistently shown the scale to be unidimensional and in alignment with the requirements of item response theory,^18–22 thus supporting the creation of a CAT form for the ABC. Therefore, the purpose of this study was to evaluate an ABC CAT’s measurement precision and efficiency of balance confidence in people with stroke under various stopping criteria with real data simulation using retrospective patient responses.

Methods

Participants

Retrospective ABC response data were obtained for 406 individuals approximately 2 months post-stroke who participated in the Locomotor-Experience Applied Post-Stroke (LEAPS) trial.^23,24 The LEAPS trial was a phase 3, multisite, randomized control clinical trial that included people with stroke over the age of 18 years who could walk at least 10 feet with maximum assistance of 1 person, walked slower than 0.8 m/s, and were living in the community.²⁵ ABC responses used in this analysis were collected during the LEAPS baseline assessment, approximately 2 months post-stroke, and there were no missing data points. This simulation study was considered secondary research because all data were free of individual identifying information and did not require institutional review board approval under the revised Federal Policy for the Protection of Human Subjects (Revised Common Rule).²⁶ Summary demographic data for the cohort are presented in Table 1.

Table 1.

Participant Demographics^a

Demographic Characteristics	Totalsb
ABC scorec	45.06 (23.88)
Age	61.97 (12.76)
Sex
Male	54.93%
Female	45.07%
Race
American Indian	1.23%
Asian	13.3%
Black or African American	22.17%
White	57.64%
Native Hawaiian	4.68%
Hispanic or Latino	15.52%
Stroke type
Ischemic	80%
Hemorrhagic	19%
Uncertain	1%
Stroke location
Right hemisphere	48%
Left hemisphere	35%
Bilateral hemispheres	6%

^a N = 406. ABC = Activities-specific Balance Confidence Scale.

^b Continuous variables are presented as mean (SD). Categorical variables are presented as a percentage.

^c ABC score was obtained approximately 2 months post-stroke.

Examination of Item Response Theory Assumptions

Previous work from our laboratory evaluated the fit of the ABC scale to the Rasch Andrich Rating Scale Model using Winsteps (version 3.93.1; John Lincare/Winsteps.com, Beaverton, OR, USA) and Mplus software (version 7.4; Muthén & Muthén; Los Angeles, CA, USA) in people with stroke.²¹ Consistent with similar analyses in other diagnostic populations, the ABC scale was found to be sufficiently unidimensional, have negligible item local dependence, and adequately fit the Rasch model.^18–22 Additionally, the ABC scale aligned well with the Rasch model when the rating scale was collapsed from a 101-category (0%–100%) rating scale to a 5-category rating scale structure using the following conversion; no confidence = 0%–9%, low confidence = 10%–30%, moderate confidence = 31%–60%, high confidence = 61%–90%, and full confidence = 91%–100%.^20–22 The collapsed rating scale structure preserves the concepts of “no confidence” and “complete confidence” while adding 3 intermediate levels for individuals to select: low, moderate, and high confidence. This scale is simpler, reduces response burden, and aligns with the ability of the ABC scale to separate people with stroke into 5 distinct categorical groups or strata.²¹ All ABC scores in this study were converted to the 5-category rating scale prior to CAT calibration and simulations.

CAT Algorithm Development and Calibration

The CAT was designed to use item 7, “sweeping the floor,” as the first item. Item 7 was selected based on the assumption that the person has a mean balance confidence that matches the normal population (theta = 0). This assumption is based on previous work that showed the population mean for balance confidence in people with stroke has a normal distribution that closely matches the mean and distribution of the item bank (or item difficulty).²¹ Balance confidence and the resulting SE from an individual’s response to each presented item were estimated with the expected a posteriori method. Subsequent items were selected to administer using maximum Fisher information.

Item thresholds for calibrating the CAT were calculated in R (version 3.5.3²⁷; R Foundation for Statistical Computing; Vienna, Austria) with the ‘ltm’ package ²⁸ using the Andrich Rating Scale Model and a random sample of the LEAPS cohort (n = 203).

CAT Simulations

We ran 4 simulations of an ABC CAT, each with a specific stopping rule based on the number of items administered (4, 8, 12, and the full 16-item bank) using Firestar (version 1.5.1; Northwestern University; Chicago, IL, USA)²⁹ software. ABC response data for the simulation were taken from the remaining one-half of the LEAPS cohort (n = 203). Validity of this simulation is based on the item response theory (IRT) assumption that item responses are independent, or, practically speaking, that people will respond to items similarly regardless of the context or order of the items.³⁰ The performance of the ABC CAT was evaluated by comparing balance confidence scores from the ABC CAT simulation with actual balance confidence scores (based on the Rasch model and calculated from the full item bank) using Pearson’s correlation and mean SE. We used Pearson’s correlation as a measure of the ABC CAT’s accuracy and mean SE as a measure of precision.³¹ Pearson’s correlation was selected as the correlation method because participants’ responses were normally distributed, and we expected a linear relationship between CAT predicted and actual ABC scores for each participant. Results from these simulations were used to inform testing of a precision-based stopping rule (based on mean SE) to optimize the tradeoff between the number of items administered and precision.

Role of the Funding Source

The funding sources for this research played no role in the design, conduct, or reporting of this study and findings.

Results

The results of each ABC CAT simulation strongly correlated with actual measures of balance confidence. Pearson’s correlations and mean SE values for each stopping rule are presented in Table 2. Limiting the number of items administered to 4 resulted in a small reduction in accuracy (r = 0.929) compared with the full item bank (r = 1). Measurement precision slightly decreased as the number of items administered was reduced approximately 0.18 logit from the full item bank to 4 items.

Table 2.

CAT Simulation Comparisons^a

Stopping Rule	r	Mean SE
Full item bank	1	0.31
12-item CAT	0.994	0.33
8-item CAT	0.980	0.38
4-item CAT	0.929	0.49
Precision-based CAT (mean SE = 0.5)	0.941	0.48

^a r = Pearson’s correlation between the final estimated balance confidence measured from the ABC CAT with the actual balance confidence measured from the full scale. ABC CAT = Activities-specific Balance Confidence Scale Computerized Adaptive Test; mean SE = mean standard error associated with the final estimated balance confidence measurement from the ABC CAT.

It is expected that measurement precision will improve by administering more items, but this relationship typically has diminishing returns. Audit trails from simulations using the ABC CAT with a full item bank stopping criteria are presented to visually represent this finding. Figure 1 contains 3 audit trails representing individuals with balance confidence comparable with (A) the population mean, (B) above the population mean, and (C) below the mean. This figure shows how the ABC CAT is typically able to identify the individual’s balance confidence level within error using just 1 item and arrive at the actual balance confidence level within 3 to 5 items. The audit trails illustrate the diminishing returns for precision when more items are administered. In our sample, the greatest reduction in error often occurred during the first few items administered, and precision saw diminishing returns after 5 items.

Figure 1.

Example audit trails from simulated responses to the full item ABC CAT. Audit trails in this figure show the estimated balance confidence (theta) measurement scores from the full item ABC CAT and associated 95% CI (theta ±1.96 ^* SE) for an individual after they responded to each presented item. Audit trails are presented for a representative individual (A) near the sample mean, (B) above the sample mean, and (C) below the sample mean with respect to their actual balance confidence score, represented by a red dash line. The order of the items presented by the ABC CAT is listed above each audit trail. In all 3 cases, the individual’s balance confidence was correctly estimated between 3 and 5 items. The audit trails demonstrate the diminishing return in measurement precision (mean SE) associated with increasing the number of administrated items. In these cases, the magnitude of improved precision appears negligible after and individual responds to 5 items.

We evaluated a precision-based stopping rule selected to optimize test length and SE. Ideally, precision-based rule considers the diminishing return on measurement precision and limits the number of items to reach an acceptable level of error as efficiently as possible. We set the precision-based rule to a mean SE of 0.50. Thus, the ABC CAT would proceed administering items until the estimated SE for an individual’s balance confidence fell below 0.50. Results from this simulation returned balance confidence estimates that strongly correlated with actual measures of balance confidence (r = 0.941; Fig. 2). Mean test length was 4.37 items, with a range of 4–9 items administered. At least 4 items per participant were required to reach the 0.5-SE threshold for the precision-based CAT simulation. No participants reached the threshold with fewer items.

Figure 2.

CAT estimated balance confidence vs actual balance confidence. Plotted are the estimated balance confidence scores (theta) from the ABC CAT for each individual against their actual balance confidence score. Estimated thetas from the ABC CAT were highly correlated (r = 0.941) with actual balance confidence.

To understand the relationship between the numbers of items administered and balance confidence ability for our population, we plotted the number of items given by the CAT to each participant against their estimated score in Figure 3. Participants required 4 items when their balance confidence scores were closer to the population mean. Individuals at the extreme ends of the population required more items for the ABC CAT to estimate their balance confidence within a SE at or below 0.5. We found that the ABC CAT required more items for individuals with lower balance confidence than the population mean (9 items) compared with individuals with higher balance confidence (5 items).

Figure 3.

Number of items administered for each individual based on estimated balance confidence. This figure plots the number of items administered by the ABC CAT using the precision-based stopping rule for each individual based on their estimated balance confidence (theta). The mean balance confidence of the sample is anchored at 0. The number of items needed to reach a mean SE of 0.5 logits when estimating balance confidence increases as an individual’s balance confidence is further from the population mean.

Discussion

Our findings demonstrate that a CAT of the ABC scale is able to efficiently measure balance confidence in people with stroke with a high degree of accuracy and precision. Results from our simulations indicate that 12-, 8-, 6-, and 4-item versions of the ABC CAT provide accurate estimates of balance confidence compared with the full 16-item measure. We found a relatively small decrease in the association between estimated and measured balance confidence as the number of items administered by the CAT decreased without a significant loss in measurement precision. Although a trade-off between the numbers of items administered a precision is expected, our results show that the loss of precision was relatively small and did not exceed 0.5 logits.

We also examined the efficiency and accuracy of a precision-based stopping rule. Precision-based stopping rules allow the CAT to administer items to an individual until a threshold of acceptable error is reached, balancing a desired level of precision with test length. Results from our simulation indicate that the ABC CAT was able to obtain this balance because most individuals only needed 4 to 5 items to estimate balance confidence accurately (r = 0.941) with a mean SE of 0.5. Our results show that the ABC CAT required more items to reach the desired precision with individuals as they moved further from the mean balance confidence of the population. In general, this is expected because SE of measures increases as scores become extreme,³² often requiring more items. We saw this effect in our simulation where the required number of items was higher for individuals whose balance confidence deviated more or less from the population mean. Our results indicated that this effect was greater for individuals with lower balance confidence (up to 9 items) compared with those with more balance confidence (5 to 6 items). This finding suggests that the ABC CAT algorithm may not be as sensitive to individuals with low balance confidence compared with those with high balance confidence. One possible contributing factor for this disparity between the 2 extremes could be the sample population used to calibrate the CAT parameters. Including response data from more individuals with low balance confidence in the initial calibration may allow the ABC CAT to reach a desired precision with fewer items administered to these less confident individuals. Future work will need to explore this possibility.

The biggest advantage of the ABC CAT is the program’s ability to drastically reduce test administrative time and response burden for individuals. Individual response burden is reduced by employing the collapsed rating scale, limiting individuals to 5 options, and reducing the number of items administered. Most individuals taking the ABC CAT will only respond to 4 items, reducing administration time by a minimum of three-quarters. Additionally, administrative time may be reduced even more because the electronic platform for CATs can provide real-time measures for individuals with the option to interface with electronic medical record systems. These benefits would help reduce administrative time for clinicians spent on scoring and documenting individual results. Another advantage electronic testing creates is an opportunity for individuals to take the test from mobile or computer devices prior to arriving for a clinic visit. Clinicians could receive accurate measure of balance confidence prior to a scheduled treatment, freeing up time for in-person performance assessments or hands-on interventions.

It is important to understand that tools like the ABC scale can serve both measurement and clinical assessment functions. It is possible that only administering the minimum number of items for measurement could limit assessment by preventing clinicians from observing how patients perform on the other items.³³ Clinicians may find it useful to administer these additional items, but they should be mindful of the burden that comes from collecting more information than is needed for measurement. Understanding principles of item response theory may help clinicians distinguish between the possible burden and type of information provided by measurement for clinical decision-making. Item response theory assigns difficulty to items creating an item hierarchy, which should match the underlying construct or theory of what you are attempting to measure.³⁴ For example, the item hierarchy of the ABC scale progresses from items such as getting in or out of car, reaching for things at eye level, and walking around the house to items that are more difficult such as navigating stairs, escalators, and eventually walking on ice.^18–22 Item response theory is also based on a probabilistic relationship between the respondent’s ability and item difficulty.³⁴ This probabilistic relationship matches clinical expectation for balance confidence on the ABC scale. For example, we would expect someone who has limited confidence with walking around the house to have even less or no confidence navigating crowded environments with perturbations. This item hierarchy and probabilistic relationship between a respondent’s abilities and items would indicate that items where people have a 50% chance of passing the item (50/50 chance of no/low confidence vs high/full confidence) would match the respondent’s true balance confidence. The ABC CAT increases a clinician’s efficiency to reach the item that matches the individual’s ability level.

The item hierarchy and probabilistic relationship also means that respondents have a high probability of having high levels of confidence on easier items and lower levels of confidence on harder items. Once we know what item matches an individual’s true balance confidence, we can extrapolate out response patterns to the other items. These response patterns are what enhance a clinician’s ability make judgements about someone’s ability level or in this case balance confidence.¹⁵ Keyforms, or keyform recovery maps, are a visual representation of these response patterns. Typically, keyforms are used to convert patient responses immediately into a logit score,³⁵ but they also can be used to take a logit score and extrapolate a response pattern. Extrapolation of a response pattern on a keyform would show clinicians items that would be of increasing difficulty for the patient. This should help clinicians make personalized treatment decisions and set rehabilitation goals.³³ We recommend that ABC CAT software output these keyform responses to enhance clinical interpretability of scores. Clinical interpretability can also be achieved by enhancing keyforms with functional staging. Functional staging goes beyond response patterns and links clinically meaningful interpretations with logit scores to help clinicians understand phenotypic characteristics of a patient.³⁶ Functional staging of the ABC scale for older adults in an outpatient physical therapy setting shows an example of how logit scores can be linked to general characteristics of a patient that can be used for treatment design and goal setting.²² We expect that functional staging of the ABC in the post-stroke population would yield similar results given that the scale can differentiate 5 distinct strata based on previous reports.²¹

Limitations

This study relied on simulation of retrospective data, which provide the opportunity to approximate real-time CAT administration, to develop and test the ABC CAT. There is precedent for using simulated real patient data to develop and test CATs using a similar methodological approach to this study.^30,31,37,38 However, these simulation methods have been criticized for potentially overestimating CAT accuracy. This is thought to be the case because the simulation uses existing datasets that have responses to each item, and the computer assumes that answers to the CAT-produced items would be the same as those to a clinician administering the test.^31,37 Haley et al^31,37 found that the effect of overestimating accuracy was negligible by comparing simulation results with a cross-validation sample where participants were given the CAT and full item bank by a clinician. To help account for some of this bias in our simulations, we separated our sample into 2 groups using random selection with 1 group used for determining item thresholds to calibrate the CAT and the other to run the simulations. Future prospective studies should seek to employ methods used by Haley et al³⁷ and administer the ABC CAT along with the full item bank to eliminate bias when evaluating measurement accuracy and precision. This design will also allow for assessing the differences in administrative and response burden for the CAT and full item ABC scale in clinical settings.

Our sample size was acceptable for item response theory assumptions, but the inclusion criteria of the LEAPS trial may limit the generalizability of our findings because only people with stroke who were ambulatory and living in the community were included.²⁵ The LEAPS sample is more representative of individuals participating in outpatient stroke rehabilitation, suggesting the ABC CAT may be more appropriate in this setting. However, there is some evidence based on other studies examining the ABC scale with the Rasch model that the scale has very similar measurement characteristics in other populations, indicating that the scale may be “diagnostic free.” That is, the item difficulty calibrations stay stable across a wide variety of diagnostic groups.^18–22 This conflict requires future work to examine the ABC CAT’s performance across a broader spectrum of post-stroke functional ability and in other populations to determine the stability of balance confidence as a “diagnosis”-free construct.

The ABC CAT was able to accurately estimate balance confidence in people with stroke comparable to the full 16-item scale. Based on simulations of real patient response data, the ABC CAT was able to estimate balance confidence for most individuals with as few as 4 items. The ABC CAT can drastically reduce administration and response burden. We expect adoption of the ABC CAT will enhance clinical practice guideline use by physical therapists in stroke rehabilitation.

Author Contributions

Concept/idea/research design: B.A. Seamon, C.A. Velozo

Writing: B.A. Seamon, S.A. Kautz, C.A. Velozo

Data collection: B.A. Seamon

Data analysis: B.A. Seamon

Project management: B.A. Seamon

Fund Procurement: B.A. Seamon, S.A. Kautz

Consultation: S.A. Kautz, C.A. Velozo

Funding

Partial funding for this project was provided by the VA Office of Research and Development (ORD), with additional support from the VA/ORD Rehabilitation R&D Service (1I01RX001935 and 1IK6RX003075), support from the National Institutes of Health (NIH P20 GM109040), and the Promotion of Doctoral Studies Level I Scholarship from the Foundation for Physical Therapy Research. Data for the study were provided by the NIH National Institute of Neurological Disorders and Stroke from the Locomotor Experience Applied Post-stroke (LEAPS) trial (R01 NS050506). Any opinions expressed in this publication are those of the authors and do not necessarily reflect the view of the US Department of Veteran Affairs or the NIH.

Clinical Trial/Systematic Review Registration

The LEAPS trial was registered at ClinicalTrials.gov (no. NCT00243919).

Disclosures and Presentations

The authors completed the ICMJE Form for Disclosure of Potential Conflicts of Interest and reported no conflicts of interest.

This material was previously presented as a poster presentation at the Medical University of South Carolina Translational Research Day, Charleston, South Carolina, January 31, 2020.