Evaluation of the Responsiveness of the Anterior Cruciate Ligament Return to Sport After Injury (ACL-RSI) Scale

Abstract

Background:

The Anterior Cruciate Ligament Return to Sport After Injury (ACL-RSI) scale is a reliable and valid tool for evaluation of psychological readiness to return to sport after ACL injury, but its responsiveness to change has not been extensively evaluated.

Purpose:

To determine the responsiveness of the ACL-RSI scale.

Study Design:

Cohort study (diagnosis); Level of evidence, 2.

Methods:

The ACL-RSI scale and the knee confidence question from the Knee injury and Osteoarthritis Outcome Score–Quality of Life subscale was completed at 6 and 12 months after ACL reconstruction surgery. Responsiveness was assessed using distribution and anchor-based methods for the full- and short-form versions of the scale and subgroup analyzed for sex. From distribution statistics, the standardized response mean (SRM) and the smallest detectable change (SDC) were calculated. Using the anchor-based method, the minimally important change (MIC) that was associated with an improvement in knee confidence was determined using receiver operating characteristic analysis.

Results:

A total of 441 patients (257 men, 184 women; mean age of 25 years) were included in this study. An SRM of 0.7 was found for both versions, indicating a moderate level of responsiveness. The MIC was 13.4 points for the full-form version and 15.1 points for the short-form version. These values were larger than SDC values at the group level but not at the individual patient level. Responsiveness was similar between male and female patients.

Conclusion:

The ACL-RSI scale had sufficient responsiveness to investigate the efficacy of an intervention at a group level, but it may be more limited at an individual patient level.

The importance of considering psychological recovery alongside physical recovery after anterior cruciate ligament (ACL) injury has become well recognized.^1,39 An ACL injury has the potential to elicit a significant psychological response, which may continue long after the injury has occurred and have a negative impact on rehabilitation and return-to-sport outcomes.³⁶ From a psychological perspective, the return-to-sport phase after ACL reconstruction surgery has been shown to be particularly challenging for some athletes, as negative emotions can be heightened at this time and can even lead to an avoidance of return to play.^3,24 It has been estimated that up to half of athletes do not return to competitive sport after ACL reconstruction surgery.²

Published in 2008, the Anterior Cruciate Ligament Return to Sport After Injury (ACL-RSI) scale was developed as a tool to evaluate psychological readiness to return to sport after ACL injury, and it is currently the only scale specific to ACL injury.⁴¹ A short-form version is also available.⁴⁰ Extensive validation of the scale has been conducted, with scores predictive of return to sport,^4,18,27,40 self-rated sport performance,⁴² and further ACL injury.^22,23 However, the responsiveness of the scale has not been extensively evaluated.³⁵

Responsiveness refers to the ability of a scale to detect clinically important changes over time.¹³ Smallest detectable change (SDC; ie, the minimum change in score that exceeds measurement error) have been reported for the Swedish²⁰ and Dutch³⁴ translations of the ACL-RSI, which have ranged from 1.5 to 3 points for groups and 15 to 19 points for individual patients on a 100-point scale. Slagers et al³⁵ also reported a minimally important change (MIC; ie, the smallest change in score that is perceived as important by the patient) of 2.6 points for the ACL-RSI Dutch language version over a 2-month period in patients who were between 3 and 9 months after ACL reconstruction surgery. As psychological readiness changes throughout the rehabilitation period, it is relevant to know the responsiveness of the scale at other time intervals. Previous work has indicated that there may be sex-based differences in psychological readiness to return to sport after ACL reconstruction surgery^19,43; however, the former studies on responsiveness had relatively modest sample sizes (48 and 70 patients, respectively), which precluded any subgroup analysis for sex.^20,35

The purpose of this study was to examine the responsiveness of the ACL-RSI scale over a period that captured the start of sport-specific training through actual return to sport, to reflect a time frame where there should be sufficient opportunity to observe change in psychological readiness. Both full- and short-form versions of the scale were examined, and data were subgrouped by sex. It was hypothesized that responsiveness will not differ between the full- and short-form versions of the scale or differ by sex, but it was hypothesized to be greater at the group level compared with the individual level.

Methods

Participants

Participants were part of a larger longitudinal study of primary ACL reconstruction and were enrolled between December 2013 and June 2018. Patients who were scheduled for additional surgery to address a concomitant posterior cruciate ligament injury or specific treatment of an associated fracture were not included in this longitudinal study. For the current analysis, patients were eligible for inclusion if they had played sport (minimum 1-3 days per month) before ACL injury, had no prior contralateral ACL injury, and had attended a scheduled 6-month review anytime between 5 and 8 months after surgery. This time point was chosen because changes in psychological readiness to return to sport are expected at this phase of rehabilitation, when patients begin to perform sport-specific drills and training before being cleared for return to play. All patients had undergone arthroscopically assisted surgery with suspensory fixation used on the femoral side and interference screw fixation on the tibial side. All were provided with the same rehabilitation protocol and guidelines, which encouraged immediate full knee extension and the restoration of quadriceps function as soon as possible.⁵ Clearance to return to competitive sport was typically between 9 and 12 months postsurgery and was determined by the treating surgeon.⁵ Patients who had any further surgery or sustained further ACL injury during the study follow-up period were excluded. All study procedures were approved by hospital and university ethics committees. Parental consent was obtained for patients under 18 years, but the measures were completed by the patients themselves.

Measures

ACL-RSI Scale

This 12-item scale is designed to measure psychological readiness to return to sport after ACL injury or reconstruction surgery.⁴¹ It includes 3 domains: emotions, confidence, and risk appraisal. Scores for each item are summed and averaged for a total score between 0 and 100. Higher scores indicate greater psychological readiness. A short form of the scale has also been produced, which contains 6 of the original items.⁴⁰

Knee Confidence

The Knee injury and Osteoarthritis Outcome Score–Quality of Life (KOOS-QOL) question “How much are you troubled with lack of confidence in your knee” was used to assess change in overall knee confidence.³¹ The 5 response options for this question are extremely, severely, moderately, mildly, and not at all.

Procedures

Patients completed the ACL-RSI scale, the KOOS-QOL confidence question, and their return-to-sport status (not returned, returned to training, returned to a lower level of competition, returned to the same/higher level of competition) when they attended for a 6-month postoperative visit (assessment 1). Patients then completed the same set of questions, which also included a battery of other scales not used in the current analysis (ie, International Knee Documentation Committee subjective knee form, 12-Item Short Form Health Survey), when they returned for a 12-month postsurgical review (assessment 2). Data were only included in the study if patients completed assessment 2 between 4 and 8 months after assessment 1.

Data and Statistical Analysis

Responsiveness of the scale was evaluated using distribution- and anchor-based methods,^9,21 The distribution-based method characterizes the ability of the scale to measure change over a specified time frame and is based on the statistical characteristics of the sample.⁴⁵ In contrast, the anchor-based method uses an external criterion to determine the extent to which changes in the scale score relate to changes in a reference measure of health status. Using this method, an MIC value is calculated and reflects a change that is regarded by clinicians or patients as a meaningful and important change in clinical status.¹²

For the distribution-based method, the standardized response mean (SRM) was calculated as the ratio of the mean change between the 2 assessments to the standard deviation of the mean.²⁸ Values <0.5 are considered to indicate low responsiveness, values between 0.5 and 0.8 moderate responsiveness, and values >0.8 large responsiveness.^11,12 The SDC, which is defined as the smallest change in score that passes the threshold of error for the scale, was also calculated with the standard error of measurement (SEM) according to the formula $SEM=SD\times \sqrt{1}-ICC$, with SD based on ACL-RSI scores from assessment 1 and ICC the intraclass correlation coefficient (2-way random effects model for agreement) between the 2 assessments.²⁹ Based on the 95% CIs, the following formulas were used to determine the SDC at both individual and group levels: $SD{C}_{Individual}=1.96\times \sqrt{2}\times SEM$ and $SD{C}_{Group}=(1.96\times \sqrt{2}\times SEM)/\sqrt{n}$.¹⁰

To determine the MIC in ACL-RSI score that is associated with an improvement in knee confidence, an anchor-based method with a receiver operating characteristic (ROC) curve was used.⁴⁴ The KOOS-QOL knee confidence question was used as the anchor criterion to dichotomize patients (improved vs not changed). Patients who changed their categorical response to indicate that they were less troubled with a lack of confidence at assessment 2 compared with assessment 1 (eg, recorded a value of “moderately” instead of “severely”) were classified as having improved confidence. Patients who did not change their categorical response between assessments were classified as having unchanged confidence. Patients who changed their categorical response to indicate that they were more troubled with a lack of confidence at assessment 2 compared with assessment 1 were classified as having reduced (less) confidence and were not included in the MIC analysis.

An ROC curve was generated with change in ACL-RSI score as the dependent variable and change status (improved confidence vs no change in confidence) as the independent variable.¹¹ After calculating the ROC curve, the diagnostic sensitivity and specificity for each potential MIC value were calculated, and the largest Youden index value (sensitivity + specificity – 1) was used to determine the ideal MIC.⁴⁶ The area under the curve (AUC) of the ROC curve represented the probability that the measure will correctly discriminate between improved and unchanged patients, in which an AUC of 0.5 is considered random, an area of 0.7 to 0.8 is acceptable, and an area of 0.8 to 0.9 is excellent.³⁷ The agreement between the newly determined MIC for the ACL-RSI with change in knee confidence was determined using 2 × 2 frequency tables, and the relationship between the SDC and MIC values was visualized using number lines, similar to the reporting method of Slagers et al³⁵ for the Dutch version of the ACL-RSI. All analyses were repeated for the short-form version of the scale (scores for the short form were derived by selecting the relevant reduced set of items from the full scale) and separately for male and female patients. Statistical calculations were conducted using SPSS Statistics Version 25 (IBM).

Results

Of the 546 patients who were eligible for the study, 50 did not have follow-up data. There were 55 patients who had follow-up data but were excluded from the analysis due to having further arthroscopic treatment or having had their data collected outside the permitted time frame for this analysis (see Figure 1). This left 441 participants, whose demographic and surgical characteristics are summarized in Table 1. Two patients also had lateral extra-articular tenodesis procedures. No patient had collateral ligament damage. All patients who were younger than 18 years (n = 102) at surgery were skeletally mature.

Figure 1.

Flow diagram of participant selection. ACL, anterior cruciate ligament.

TABLE 1.

Participant Demographic Characteristics and Surgical Details^a

Characteristic	Value
Age at surgery (y)	24.6 ± 7.4 (14-47)
Sex
Male	257
Female	184
Preinjury Marx activity score	12.5 ± 3.8 (0-16)
Preinjury sport level
Level 1 (jumping, hard pivoting, cutting)	376
Level 2 (running, twisting, turning)	49
Level 3 (no running, twisting, or jumping)	16
Preinjury sport frequency
4 to 7 days per week	255
1 to 3 days per week	175
1 to 3 days per month	11
Medial meniscal tear	118
No treatment	22
Repair	49
Resect	53b
Lateral meniscal tear	154
No treatment	68
Repair	16
Resect	70
Chondral damagec	88
ICRS grade 2	49
ICRS grade 3	29
ICRS grade 4	10
Graft type (all autograft)
Hamstring	388
Patellar tendon	11
Quadriceps	42
Time after surgery, mo
Assessment 1	6.3 ± 0.5 (5-8)
Assessment 2	12.0 ± 0.5 (11-15)
Time between assessments	5.8 ± 0.7 (4-8)
Sport status at assessment 1
No sport	304
Training	109
Returned to lower level of competition	22
Returned to same level of competition	6

^aData are reported as mean ± SD (range) or No. ICRS, International Cartilage Regeneration & Joint Preservation Society.

^bFive in a previous surgery.

^cTreatment performed in 44% (39/88) of cases of chondral damage (all debrided with an arthroscopic shaver and 1 microfracture procedure).

At assessment 1, a mean of 6.3 months after surgery, only 28 patients (6%) had attempted to play competition sport and most had not returned to any sport. At assessment 2, a mean of 12 months after surgery, 287 patients (65%) had returned to some form of sport (40% competition, 25% training).

The mean time between assessments was 5.8 months. During this time, ACL-RSI scores increased and the SRM was of a moderate size (0.7) for both the full- and the short-form versions of the scale. The SDC_Individual was 26.6 points and SDC_Group was 1.3 points for the full-form version, and it was 28.8 points and 1.4 points, respectively, for the short-form version (Table 2). SDC values were similar between male and female patients.

TABLE 2.

Descriptive Statistics, SRM, SEM, and SDC Values for the ACL-RSI and ACL-RSI (SF)^a

	Assessment 1 Score (6 mo)	Assessment 2 Score (12 mo)	Assessment 2 – 1	ICC	SRM	SEM	SDCIndividual	SDCGroup
ACL-RSI
All	54.4 ± 22.0	66.8 ± 23.7	12.5 ± 18.1	0.81	0.69	9.6	26.6	1.3
Female	52.3 ± 21.8	64.2 ± 23.5	11.9 ± 16.6	0.85	0.72	8.4	23.4	1.7
Male	55.8 ± 22.1	68.7 ± 23.7	12.9 ± 19.2	0.72	0.67	10.1	28.1	1.8
ACL-RSI (SF)
All	49.4 ± 22.7	63.3 ± 24.6	13.9 ± 19.9	0.79	0.70	10.4	28.8	1.4
Female	47.3 ± 22.9	60.4 ± 24.0	13.1 ± 17.9	0.83	0.73	9.4	26.2	1.9
Male	50.9 ± 22.4	65.3 ± 24.9	14.4 ± 21.2	0.75	0.68	11.2	31.1	1.9

^aACL-RSI, Anterior Cruciate Ligament Return to Sport After Injury; ICC, intraclass correlation coefficient; SDC, smallest detectable change; SEM, standard error of measurement; SF, short form; SRM, standardized response mean.

According to the responses to the KOOS-QOL question, there were 180 patients (41%) who had improved knee confidence, 209 patients (47%) who had no change, and 52 patients (12%) who had less confidence between the 2 assessments. The mean change in the ACL-RSI score for patients who had improved confidence was 19.6 points for the full-form version and 21.3 points for the short-form version. The MIC that was associated with an improvement in knee confidence was 13.4 points for the full-form version and 15.1 points for the short-form version (Table 3). MIC for female patients was around 2 points higher than for male patients. Table 4 shows frequency tables visually demonstrating the association between these calculated MIC thresholds and improved knee confidence. The percentage of patients misclassified was 34% for the full-form version (AUC = 0.7) and 37% for the short-form version (AUC = 0.65). The relationship between SDC and MIC values for the ACL-RSI is visualized in Figure 2.

TABLE 3.

MIC Values for Improvement in Knee Confidence for ACL-RSI and ACL-RSI (SF)^a

	No Change in Knee Confidenceb	Improved Knee Confidenceb	MIC
ACL-RSI
All patients	9.8 ± 15.8	19.6 ± 16.3	13.4
Female	8.9 ± 16.4	18.9 ± 15.3	15.3
Male	10.5 ± 15.4	20.0 ± 16.9	13.4
ACL-RSI (SF)
All patients	11.3 ± 18.0	21.3 ± 18.2	15.1
Female	10.6 ± 17.7	20.1 ± 17.1	17
Male	12.0 ± 18.3	22.0 ± 18.8	15.1

^aACL-RSI, Anterior Cruciate Ligament Return to Sport After Injury; MIC, minimally important change score; SF, short form.

^bChange in response to Knee injury and Osteoarthritis Outcome Score—Quality of Life confidence question between assessments 1 and 2.

TABLE 4.

Frequency Table of Agreement Between MIC of ACL-RSI and Knee Confidence^a

	Improved Knee Confidence	No Change in Knee Confidence	Misclassified, %	P b
ACL-RSI				<.0001
>MIC of 13.4 points	114	67	34
<MIC of 13.4 points	66	142
ACL-RSI (SF)				<.0001
>MIC of 15.1 points	115	80	37
<MIC of 15.1 points	65	129

^aACL-RSI, Anterior Cruciate Ligament Return to Sport After Injury; MIC, minimally important change; SF, short form.

^bP <.0001 indicates a significant association between meeting MIC threshold and improved knee confidence.

Figure 2.

Visual relationship between the minimal important change (MIC) and the smallest detectable change (SDC) of the ACL-RSI at the (A) group and (B) individual level. ACL-RSI, Anterior Cruciate Ligament Return to Sport After Injury.

Discussion

The current study investigated the responsiveness of the ACL-RSI scale. The scale was shown to be moderately responsive over a 6-month average time frame, which began when patients had just commenced sport-specific training but most had not yet attempted to play, through to when most had returned to some form of sport participation. The MIC values for both the full and short forms of the scale were larger than the SDC values at the group level, but not at the individual level, indicating that the scales have good ability to detect clinically relevant changes at a group level but may be more limited at the individual level. This is perhaps not surprising, as psychological responses can vary considerably between patients.³⁹

In the current study, it was shown that, at the individual level, an ACL-RSI score change above 26.6 points can be considered to reflect both a statistically significant and a clinically important change. Changes in score between 13.4 and 26.6 points reflect changes that may be important to the patient but cannot be distinguished from measurement error. There were 94 patients (21%) who had change scores above SDC and MIC thresholds. This was substantially more than the 6 (9%) patients who were reported to have exceeded these thresholds in a previous study by Slagers et al³⁵ of the Dutch translation of the ACL-RSI administered 2 months apart. The greater ability of the current study to identify patients with statistically and clinically important improvements on the ACL-RSI is likely due to the longer time frame of assessment, which allowed these changes to occur and be detected. Indeed, the MIC reported by Slagers et al³⁵ was 2.6 points and markedly smaller than the MIC of 13.4 points reported currently. The SRM was also small (0.3) in Slagers et al³⁵ compared with moderate (0.7) in the current study. These results indicate that psychological readiness to return to sport improves over time, but the improvement is gradual, and the magnitude of the change is dependent on the time interval between assessments. This is important to consider if the goal is to evaluate the efficacy of an intervention.

Despite the moderate responsiveness of the ACL-RSI at the individual level, at the group level the scale was significantly more responsive, with SDC values of less than 2 points. This is consistent with previous Swedish²⁰ and Dutch³⁴ translations of the ACL-RSI and now extends to the original English version of the scale. From a practical perspective, the ACL-RSI can be readily used in cross-sectional studies to evaluate the effectiveness of interventions at the group level. The scale can also be used as a screening tool to identify patients at risk of not returning to sport, as the reliability and validity for this purpose have been established in numerous studies and across many language translations.^§ There is also initial evidence of the association between ACL-RSI scores and further ACL injury in young athletes,^22,23 so it may also be suitable as a screening tool for reinjury risk.

There were no substantial differences in responsiveness values between male and female patients, particularly at the group level of analysis. The SDC_Individual values were smaller for female patients by approximately 5 points, and MIC values were higher by approximately 2 points. Therefore, the scale appears to be slightly more responsive to female patients, which may be beneficial given that female patients have previously been shown to have a more negative outlook with regard to return to sport and that it has been suggested they may benefit more from psychologically based interventions⁴³. However, based on the current study, any differences would be slight and are unlikely to have clinical significance. Responsiveness of the ACL-RSI has not previously been reported separately for male and female patients; and as such, there are no prior data with which to compare the current findings. Similarly, the responsiveness of the short-form version of the scale was not markedly different from the full-form version. While the MIC was 1.7 points higher and the SDC_Individual was 2.2 points higher for the short form—indicating a small sacrifice in responsiveness for the brevity of the scale—the SRMs were equivalent between the full and short forms. Overall, in terms of responsiveness, the full and short forms should be able to be used interchangeably.

In the current study, the relatively large patient cohort allowed for robust estimates of measurement error to be made for the total sample as well as examining for sex-based differences. The 4- to 8-month time frame between assessments also allowed for meaningful changes in psychological readiness to return to sport to occur at relevant phases of the rehabilitation period. This extended the findings from previous studies, which had examined much shorter time frames. To our knowledge, this is the first report of the responsiveness of the ACL-RSI for the original English version of the scale and the first to calculate responsiveness for the short-form version.

Responsiveness was analyzed using both distribution-based and anchor-based methods so that statistically significant changes that exceed measurement error could be determined in addition to changes that are important to the patient. Many anchor-based methods have been proposed and there is yet to be consensus regarding the best method. The choice of the external criterion that forms the anchor varies greatly.³⁰ A global-rating-of-change question is a frequently used anchor, which requires the patient to rate one’s perceived change in health status between the first and second administration of the scale. However, this has been criticized for being susceptible to recall bias, and acquiescence bias may also occur.¹² In the current study, an alternative method was used in which patients rated their overall knee confidence using a validated question from the KOOS-QOL at both assessments. Therefore, they were not asked to recall change, and instead it was directly measured by a change in their response category to that question over time. An alternative external anchor could be the Patient Acceptable Symptom State, which has been increasingly used to evaluate successful outcomes after ACL reconstruction surgery.^6,15,26

A potential limitation is that the short-form scores were derived from patients who completed the full version. While this has also been the procedure for some of the previous reliability and validity analyses,⁴⁰ it is not guaranteed that patients would respond identically if only given the short form; further research is required to confirm this. Another potential perceived limitation is that the patient group was relatively homogeneous and only included patients who had undergone a first, primary ACL reconstruction and were active in sport before ACL injury. This was, however, a deliberate decision so that past experience of ACL injury and/or surgery did not influence patients’ responses and the concept of readiness to return to sport was relevant. Further work is required to confirm whether the responsiveness of the ACL-RSI scale is similar for patients who have undergone revision ACL reconstruction or have had ACL injuries to both knees. Most patients in this study also received a hamstring tendon autograft. While there is no reason to hypothesize that psychological readiness would differ between graft choices, this could also be more thoroughly explored in future research along with a comparison between autografts and allografts.

While not elite-level athletes, the current patient cohort was highly active before injury, with 85% participating in level 1 sports that involve jumping, hard pivoting, and cutting and 58% reporting that they participate in sport 4 to 7 days per week. The 6- to 12-month postoperative time frame was chosen because this is typically a time where patients make the transition to some form of sport participation (ie, begin sport training) and it has been shown that negative psychological emotions may reemerge once patients are given clearance to return to play.²⁵ However, other time intervals would also be of interest and can also be explored in future work.

Conclusion

In this study, both full- and short-form versions of the ACL-RSI scale had moderate responsiveness over the 6-month study time frame. The scale had sufficient responsiveness to investigate the efficacy of a psychological intervention at a group level, but it may be more limited at an individual level.