The Minimal Clinically Important Difference for the Mayo-Portland Adaptability Inventory (MPAI-4)

Abstract

Objectives

To determine the Minimal Clinically Important Difference (MCID) and Robust Clinically Important Difference (RCID) of the Mayo-Portland Adaptability Inventory (MPAI-4) as measures of response to intervention.

Methods

Retrospective analysis of existing data. Both distribution- and anchor-based methods were used to triangulate on the MCID and to identify a moderate, i.e., more robust, level of change (RCID) for the MPAI-4. These were further evaluated with respect to clinical provider ratings.

Participants

Data for individuals with acquired brain injury in rehabilitation programs throughout the U.S. in the OutcomeInfo Database (n=3087) with two MPAI-4 ratings.

Main Measures

MPAI-4, Supervision Rating Scale (SRS), Clinician Rating of Global Clinical Improvement (CRGCI).

Results

Initial analyses suggested 5 T-score points (5T) as the MCID and 9T as the RCID. 81–87% of clinical raters considered a 5T change and 99% considered a 9T change to indicate meaningful improvement.

Conclusions

5T represents the MCID for the MPAI-4 and 9T, the RCID. Both values are notably less than the Reliable Change Index (RCI). While the RCI indicates change with a high level of statistical confidence, it may be insensitive to change that is considered meaningful by providers and participants as indicated by the MCID.

An increasing emphasis on individualized medicine has turned the focus from studies documenting mean differences between treatment and control conditions toward trials that not only demonstrate treatment efficacy but also identify the characteristics of those who benefit from the treatment (responders). We have shown in previous studies^1,2 that the precision of frequently-used traumatic brain injury (TBI) outcome measures can be improved through Rasch analysis resulting in an interval equivalent measurement metric. We have applied Rasch analysis to MPAI-4 data and demonstrated the responsiveness of MPAI-4 metrics to acquired brain injury (ABI) rehabilitation treatment.^3–5 Identification of the Minimal Clinically Important Difference (MCID) for MPAI-4 is an essential next step to clearly identifying responders to a given intervention evaluated with this measure. Systematic reviews of comprehensive rehabilitation interventions have indicated that studies to date have been minimally informative, in part due to the lack of identification of MCID for outcome measures.⁶ In order to provide the field with a critical metric to assess efficacy and identify responders across a range of outcomes in future rehabilitation treatment research, this study was designed to identify the MCID for the MPAI-4.

The MCID, as the name implies, represents the smallest change on a measure that is reliably associated with a meaningful change in the patient’s clinical status, function, or quality of life. Using the MCID, efficacy of a new treatment can be assessed by the proportion of individuals who receive benefit (rather than by group means), providing a concrete and pragmatic answer to the question posed by most practicing clinicians: what is the probability that the patient in front of me will benefit from this treatment?

There is no universally accepted way to establish MCID, and many ways have been proposed.^7–10 One set of approaches to determine the MCID uses distribution-based analyses and calculates the standard error of measurement (SEM). Another set seeks to determine the MCID with reference to external anchors, that is, other indicators of clinical change. Current literature on MCID methodology recommends using both distribution-based and anchored methods to triangulate on the best value for the MCID for a given measure.⁹ In this study, we aimed to establish the MCID using such a multi-modal method with Rasch-calibrated MPAI-4 data. In addition, we sought to identify a change that indicated a moderate effect which we refer to as the Robust Clinically Important Difference (RCID).

Method

Participants

Data used in this study were obtained from individuals who had been rated on the MPAI-4 on two occasions with data recorded in the national OutcomeInfo Database. Participants were 3087 individuals (65% male) with a mean age of 46.56 years (SD = 14.41 yrs) who were an average of 586.78 days (SD = 1788.59 days) post-injury at the time of program admission. These individuals were injured at an average age of 44.19 yrs (SD = 15.57 yrs). All cases submitted to the database were identified by contributing providers as having a history of ABI (e.g., open or closed traumatic brain injury, stroke, infection, tumor, anoxia). However, detailed diagnostic information was not available on most cases. Our previous analyses comparing MPAI-4 responses between two major diagnostic categories (TBI vs. stroke) in post-inpatient brain injury rehabilitation samples revealed differential item functioning for some items; however, examination of differential test functioning showed very similar metrics between the two groups at the test level, i.e., test characteristic curves were virtually identical.⁴ This finding supports combining diagnostic groups in this and other types of analyses in which the focus is on evaluating measures of function rather than measures of diagnostic signs and symptoms.

These individuals participated in 1 of 3 types of programs: Intensive Residential Rehabilitation (n=205), Intensive Community-based/Outpatient Rehabilitation (n=2781), and Residential or Community-based Supported Living (n=101). MPAI-4 profiles were obtained on admission and discharge from intensive rehabilitation programs and on admission and at the first subsequent follow-up for those receiving supported living services. Analysis of MPAI-4 outcomes showed significant improvement for those receiving intensive rehabilitation and stable functioning for those in supported living from first to second assessments. More detailed program descriptions and complete results of these analyses are available in a prior report.⁵ Anchored estimations of the MCID were also conducted with reference to the Supervision Rating Scale (SRS) using a subsample with available SRS data (n = 2726).

OutcomeInfo Database

OutcomeInfo¹¹, a web-based database system developed through a NIH Small Business Technology Transfer (STTR) grant, was designed to facilitate monitoring of progress and outcomes in post-inpatient programs primarily with the Mayo-Portland Adaptability Inventory (MPAI-4) as well as additional measures specific to each provider. Demographic and injury-related information about participants are also collected. Participating provider organizations do so on a volunteer basis and pay a subscription fee for data management and reporting. Data are managed at a HIPAA-approved level of security. Each contributing organization has complete access to their own data. However, analyses, such as the one reported here that combine data across organizations are conducted with anonymity of both cases and organizations. The data used here represents participants in 23 facilities operated by 9 different organizations in 14 states in the northeast, southern, northwest and midwest U.S.

Expert Raters

In a final step, we evaluated hypothesized values of 5 T-score points (5T) and 9T (derived from distribution and anchored approaches) for the MCID and RCID through clinical provider ratings of case protocols illustrating change over the course of rehabilitation. From the larger sample of participants, we randomly selected 9 who had participated in intensive rehabilitation services and changed 5T on the MPAI-4 over the course of treatment. Three cases were selected at each of 3 levels of disability on the MPAI-4 admission assessment: (1) mild disability (MPAI-4 admission T-score < 40), (2) moderate disability (MPAI-4 admission T-score = 40–60), and (3) severe disability (MPAI-4 admission T-score > 60). We selected an additional 9 cases at each of these 3 initial disability levels who had participated in intensive rehabilitation and changed 9T over the course of treatment. Graphic displays illustrating change on each item over the course of treatment were made accessible to clinical raters over the internet. Cases were selected that showed overall positive change of either 5T or 9T and not all change on individual items was in a positive direction. The rating clinicians were not informed about the participant’s initial disability level or that the protocols represented a magnitude of 5T or 9T.

Thirty clinical providers were asked to rate functional change for each of these cases on a 7-point Clinician Rating of Global Clinical Improvement (CRGCI) scale. Clinicians were required to have at least 1 year experience using the MPAI-4 in a clinical setting and were recruited through the Pennsylvania Association for Rehabilitation Facilities and Rehab Without Walls. These providers included 8 speech/language pathologists, 6 clinical managers or directors, 5 physical therapists, 5 occupational therapists, 4 neuropsychologists, a psychologist, and a behavioral counselor. Raters had an average of 11.97 years (SD = 8.93) experience in ABI rehabilitation and 4.90 years (SD = 3.06) using the MPAI-4 and worked in facilities in 10 states in the eastern, western, southern, and midwestern U.S.

Measures

The MPAI-4¹² consists of 30 items selected to assess commonly-occurring limitations after BI. It is divided into three subscales: Ability Index, Adjustment Index, and Participation Index. Prior studies have demonstrated satisfactory internal consistency, construct validity^3,13,14 as well as concurrent¹⁵ and predictive validity^16–18 for the full measure and its indices. The MPAI-4 has been found to be responsive the effects of rehabilitation interventions.^15,19,20

The Supervision Rating Scale (SRS) is a single item measure of need for supervision in the participant’s living situation (or conversely, of independent living) that describes 13 levels ranging from 24-hour supervision with physical restraints to completely independent living with no supervision.²¹ Each level marks a significant change from the adjoining levels in personal independence and burden of care. These 13 levels can be collapsed into 5 broader levels of required supervision: (1) independent, (2) overnight supervision, (3) part-time supervision, (4) full-time indirect supervision, and (5) full-time direct supervision.

A Clinician Rating of Global Improvement Scale (CRGCI) Scale was constructed which asked raters to rate each case by answering two questions: (1) “If rehabilitation began within the first year after brain injury, how do you rate this individual’s functional status at discharge compared to status on admission”; and (2) “If rehabilitation began one year or more after brain injury, how do you rate this individual’s status at discharge compared to status on admission.” For each question, ratings were made on a 7 point scale: (1) much better, (2) moderately better, (3) a little better, (4) no change, (5) a little worse, (6) moderately worse, and (7) much worse.

Procedures

We identified and evaluated the MCID for the MPAI-4 total score using both distribution-based and anchored methods. Distribution-based methods included computation of standard deviation and standard error of measurement (SEM) and recommended functions of SEM. For example, ½ SD, 1 SEM, or 2 (or more precisely 1.96) SEM have been recommended as measures of MCID.^10,22 These values, as well as 2.77SEM (equivalent to the Reliable Change Index; RCI¹⁰), were computed both for the entire sample and for only those subjects who showed zero or positive change on the MPAI-4. Rasch-derived T-scores were used in all analyses. Procedures used in Rasch analyses and T-score development are described fully in the prior study.⁵

We examined the degree of change in the MPAI-4 Total score anchored to a change of one level on the standard 13-level SRS and to a change of one level on the 5-level SRS. Following identification of potential values for the MCID and RCID using these distribution-based and anchored methods, we tested these values by comparing the percentage of individuals who met or exceeded these levels after receiving intensive residential or community-based/outpatient rehabilitation services compared to individuals receiving supported living services. In a prior study,⁵ the intensive rehabilitation groups showed significant improvement of about 1 SD on the MPAI-4 over the course of treatment, whereas those in supported living services showed stable functioning from admission to follow-up assessment. We also compared differences on the SRS for groups that met criteria for our hypothesized MCID and RCID values to further test these hypotheses. As described previously, we ultimately evaluated hypothesized MCID and RCID values through therapist ratings on a CRCGI scale.

Results

Distribution-based indicators

Because T-scores were used in these analyses, the standard deviation for the full sample was 10 by definition. We computed the SEM and recommended functions of the SEM (Table 1) using the Person Reliability of .89 obtained from the previous Rasch analysis of this data set as the reliability indicator.⁵ This value is very similar to the value of .91 obtained in prior analyses with large data sets.^3–4 Values for these indicators were very similar for both the total sample and for the subset that showed zero or greater change on the MPAI-4 from first to second assessment.

Table 1.

Distribution-based MCID indicators

	Standard Deviation (SD)	½ SD	SEM	1.96 SEM	2.77 SEM (Reliable Change Index)
Total Sample (n=3087)	10.0	5.0	4.07	7.98	11.27
Sample with no or positive change (n=2864)	9.9	4.85	4.03	7.91	11.17

SRS Anchoring

To further triangulate on the appropriate value for the MCID, we identified the optimal change score on the MPAI-4 through ROC analysis that differentiated those who did not change from first to second assessment from those who changed 1 unit on the standard 13-level SRS. To be conservative in this evaluation, we included individuals who changed 1 unit but no more (n=284) versus those who showed zero change (n = 950) on the 13-level SRS. Optimal change was defined as the best balance between sensitivity and specificity, i.e., most equivalent correct identification of both groups. We conducted these analyses both for the full sample and the subsample that showed zero to positive change on the MPAI-4, that is, those with negative change on the MPAI-4 were not included. We performed similar analyses to determine the optimal cutpoint to differentiate those who advanced 1 unit on the 5-level SRS (n = 724) from those who showed zero change (n = 1166). Results are summarized in Table 2.

Table 2.

ROC analysis anchored to Standard (13-level) and 5-level SRS.

	Optimal Cutpoint	No change: % correct identification (n)	1 unit change: % correct identification (n)
Standard SRS (full sample)	7.5	55% (284)	61% (950)
Standard SRS (sample with zero/positive change on MPAI-4)	8.5	54% (268)	62% (858)
5-level SRS (full sample)	8.5	64% (724)	62% (1166)
5-level SRS (sample with zero/positive change on MPAI-4)	8.5	66% (703)	59% (1061)

The modest % correct classification reported in Table 2 probably represents multiple factors, including the restricted range of scores in the selected sample; divergence in the constructs represented by the two instruments, i.e., a broad range of functional abilities and activities for the MPAI-4 compared to current level of supervision for the SRS; and the relative coarseness of the ordinal metric represented by the SRS. With this in mind, we considered the results of this analysis and proceeded with additional anchored evaluations.

MCID evaluation between treatment conditions

Based on these distribution-based and anchored estimates of the MCID, we hypothesized that the truly minimal clinically important difference may be about ½ SD or 5T. The SEM of 4.07 was just below this value. We were more comfortable setting the MCID at the ½ SD value instead of 1 SEM since, as the minimally detectable difference, 1 SEM leaves little room for error resulting from varying reliability in implementation across settings. Furthermore, a ½ SD value has been endorsed as a reasonable estimate of the MCID for most measures.⁹ Anchored evaluation with the SRS suggested a higher value between 7.5 and 8.5. The 1.96 SEM was also in this same range, i.e., about 8. We hypothesized that this level of change might indicate a more robust clinically important difference (RCID). We reasoned that the value of the RCID should be at least 1 SEM greater than the MCID; therefore, we chose 9 T-score points (9T) for the hypothesized RCID. We then tested these two possible values for the MCID and RCID by examining how well they differentiated between two groups in our sample: (1) those who demonstrated significant change as a group from admission to discharge from intensive rehabilitation services, and (2) those who, as a group, showed stable performance as a result of involvement in supported living services. Both values differentiated between groups with the 5T value providing clearest differentiation (Table 3).

Table 3.

Comparison of 5T (MCID) and 9T (RCID) benchmarks in intensive rehabilitation vs. supportive living participants

	Intensive Rehabilitation	Supported Living
≥ 5 T score change Χ2 = 169.74, p < .001	72%	12%
≥ 9 T score change Χ2 = 97.60, p < .001	54%	4%

MCID and RCID compared to change on SRS

To further evaluate the values of 5T for the MCID and 9T for the RCID, we divided the sample with available SRS scores into 3 groups: Group 1 included those who did not achieve a MCID on the MPAI-4 (n=719), Group 2, those who achieved a MCID but not a RCID (n=495), and Group 3, those who achieved a RCID (n=1512). Change on the 13-level SRS was then compared among these 3 groups using the Kruskal-Wallis nonparametric test which indicated a significant difference among the groups (Χ²=507.625, p< .001). Further contrasts between groups using the Wilcoxon test indicated that Group 2 showed greater change on the SRS than Group 1 (Z=−7.543, p<.001) and that Group 3 changed more on the SRS than Group 2 (Z=−11.724, p< .001).

MCID evaluation compared to clinician ratings of Global Clinical Improvement

We further evaluated the 5T and 9T benchmarks through therapist ratings on a CRGCI scale. Table 4 shows percentages for each rating by change level and time since injury. Ratings tended to be more favorable when the case was considered more than 1 year post-injury than less than 1 year post-injury both for a 5T change (χ²=9.81, df=4, p<.05) and for a 9T change (χ² =16.20, df=3, p<.001). Ratings also were more favorable for those making a 9T change compared to a 5T change both when the case was considered less than 1 year post-injury (χ²=114.75, df=4, p<.001) and when the case was considered greater than 1 year post-injury (χ²=119.14, df=4, p < .001). For these tests, cells with very small numbers were combined.

Table 4.

Change ratings for hypothesized 5T and 9T change by time since injury

Rating	5T Change		9T Change
	<1 Yr Post-injury	>1 Yr Post-injury	<1 Yr Post-injury	>1 Yr Post-injury
Much better	7%	8%	22%	37%
Moderately better	16%	23%	46%	36%
A little better	63%	50%	31%	26%
No change	7%	9%	1%	1%
A little worse	6%	8%	0%	1%
Moderately worse	1%	1%	0%	0%
Much worse	0%	0%	0%	0%

As can be seen in Table 4, virtually all raters (99%) indicated that those making a 9T change were at least “a little better” regardless of time since injury. For cases considered less than 1 year post-injury, 87% of those making a 5T change were rated as at least “a little better” and, for those considered more than 1 year post-injury, 81% making a 5T change were rated as at least “a little better."

Discussion

We systematically worked through a series of distribution-based and anchored procedures to triangulate on the value of the MCID for the MPAI-4. Distribution-based analyses identified values of 1 SEM, 1.96 SEM and ½ SD that have been recommended in literature previously cited as viable candidates for the MCID. The initial anchored ROC analysis with reference to the SRS suggested a value close to 1.96SEM (i.e., ~8). However, additional anchored analyses using hypothesized values of 5T and 9T indicated that both these values significantly differentiated a group that benefited from intensive rehabilitation compared to a group that maintained function in supported living services. These hypothesized MCID and RCID values also differentiated on the SRS among groups that (1) did not achieve a MCID, (2) achieved a MCID but not a RCID, and (3) achieved a RCID on the MPAI-4. It may be that our initial anchored analysis of SRS data using ROC resulted in a slightly inflated value because, in fact, progress from one level to the next on the SRS indicates substantial, moderate change. Ultimately we evaluated our hypothesized values for the MCID and RCID through MPAI-4 change protocol review by experienced ABI rehabilitation providers. This expert panel nearly unanimously (99%) agreed that a change of 9T represents positive change with a substantial majority (81–87%) agreeing that a 5T represents at least a minimal positive change.

Our aims in this study were to identify a value that indicated the smallest change on the measure that is reliably associated with a meaningful clinical change, i.e., the MCID, and a value that indicates moderate, more robust change, i.e., the RCID. Consistent with these labels, a larger percentage of the experts indicated that the RCID value of 9T represented a change that was “moderately better” or “much better” than a change of 5T which was more frequently endorsed as indicating that the participant was a “little better.” (See Table 4)

In clinical practice or program evaluation, we recommend that these labels be applied at face value, that is, a MCID (i.e., 5T positive change) should be considered to represent a small but meaningful change in the rehabilitation participant’s functional status and a RCID (i.e., 9T positive change) should be considered to represent moderate, more robust improvement. These values should be applied to the total score for the MPAI-4. Inspection of index (i.e., subscale) scores may subsequently be informative to appreciate the extent to which the change involved improvements in abilities, adjustment, and/or participation. Rasch analysis yields a metric (logit) that has equal intervals between levels. These logit scores provide an interval equivalent measure unlike ordinal raw scores in which small changes at the ends of the continuum may represent larger functional gains compared to similar raw score changes in the middle of the continuum. To provide a more familiar metric, we converted the logit scores to T-scores. By using logit-based T-scores in our analyses, we were able to equate changes across the continuum of disability. For this reason, the MCID and RCID recommended here would be appropriately applied regardless of the participant’s initial level of disability.

The MPAI-4 was originally developed using mixed samples of individuals with ABI. As such, the MPAI-4 is a functional, not a medical diagnostic, measure and consequently we believe that the measure and recommended MCID and RCID can be applied to individuals with ABI regardless of the participant’s specific brain diagnosis. Furthermore, individuals in this sample were admitted to the rehabilitation or supported living programs based on functional evaluations and potential to benefit from participation in the specific program. Because selection for treatment was based on functional, not medical, evaluation, participants in a program are very likely as homogenous (or conversely, heterogenous) across medical diagnostic categories as within category. For example, in a prior study, we examined for potential differences between the two largest populations of ABI, stroke and TBI.⁴ Differences were found on some items between these two groups that reflected expected differences between more lateralized and more diffuse/frontal brain injury. However, any two participants in either diagnostic group will also differ in their item profiles depending on the specific brain regions affected by ABI, premorbid characteristics and other factors that create individual patterns of functional impairment. Most likely because item profile variability within medical diagnostic group is as great as between groups, on analysis of the overall measure, the test characteristic curves for the stroke and TBI groups were virtually identical. These considerations (i.e., medical diagnostic groups perform similarly on the overall measure; the MCID and RCID were based on the overall measure; and the original design was as a functional measure for use with mixed ABI rehabilitation participants) support the use of the MPAI-4 for functional evaluations and the assessment of change using the MCID and RCID recommended here regardless of the participant’s medical diagnosis. Given the large and geographically diverse nature of the sample, facilities, and settings, we believe that these findings would be applied appropriately in post-inpatient brain injury rehabilitation and supported living programs. However, these values may differ in other settings, e.g., inpatient rehabilitation, or a different measure may be more appropriate for the assessment of change.

Both values for the MCID and RCID are lower than the value of the RCI of about 11. Turner and colleagues¹⁰ have described the limitations of and warned against using distribution-based methods alone to determine the MCID, such as computation of the RCI. The RCI is an indicator of the degree of change that is statistically very unlikely to occur by chance. However, the RCI may not be sensitive to smaller changes that are meaningful to providers and participants. Our systematic analyses using multiple methods to determine the MCID suggest that, in the context of post-inpatient ABI rehabilitation, the RCI for the MPAI-4 represents a conservative indicator of clinically meaningful change which may result in under-identification of cases that have benefited from rehabilitation as well as underestimate the potency of rehabilitation interventions. The MCID, in contrast, represents a degree of change that is likely to be clinically meaningful but does not control for measurement error as rigorously as the RCI. Which then represents the “true” measure of change? We suggest that this would depend on the purpose of the evaluation. For example, in testing a new rehabilitation procedure, a researcher may elect to use the RCI of a relevant measure to determine whether the treatment group obtains a superior outcome compared to the control condition with a high degree of statistical certainty. However, in this same context, the researcher may also appropriately report the percent of participants achieving an MCID to provide clinicians with an indication of how likely it is that a given participant will obtain meaningful benefit from the procedure.

Limitations

While we used multiple methods to triangulate on the MCID and RCID for the MPAI-4, we did not include evaluation with reference to patient-reported outcomes (PRO). In this retrospective study, we were not able to determine a practical way to obtain the opinions of individuals participating in ABI rehabilitation programs about the degree of benefit that they felt they received from these services. The protocol reviews that we asked of therapists seemed too technical to be completed in a valid fashion by lay participants. The validity of PRO from individuals with ABI may also be questioned since these individuals may have difficulty assessing their status or progress because of cognitive impairments resulting from ABI. Nonetheless, our study would have benefited from obtaining the perspectives of individuals with ABI who had participated in post-inpatient rehabilitation services regarding the degree of change recorded on the MPAI-4 that is associated with their perception of a meaningful change in their functional status.

Conclusions

With reference to a large, national sample of individuals participating in post-inpatient brain injury rehabilitation services, we were able to triangulate on values for minimal (MCID) and moderate (RCID) change on the MPAI-4 through systematic application of distribution-based and anchored methods. Results of these analyses recommend a 5T change in total score on the MPAI-4 as the MCID and a 9T change as the RCID. Future research may profitably assess the degree to which these values also represent meaningful change from the perspectives of participants or their significant others. These values may be used in future controlled or observational trials to evaluate the efficacy and effectiveness of post-inpatient brain rehabilitation procedures as well as to identify responders in such trials. The identification of responders, i.e., participants who obtain a positive change of at least a MCID, will be important in future rehabilitation treatment studies to characterize those who benefit from a specific intervention. This is particularly relevant in treatment studies conducted from an individualized medicine perspective in which an intervention may not show a significant effect when applied to a heterogeneous sample of participants who, for example, share a medical diagnosis, but may be effective for a subgroup within the sample who share specific functional, demographic, genomic, or other identifiable characteristics.