A Standardized Approach to Calculating Clinically Significant Change in Hoarding Disorder Using the Saving Inventory-Revised

Abstract

This shorter communication explores the concept of clinically significant change in treatment outcome studies for hoarding disorder. We argue that cross-study comparisons have been hindered due to researchers using different formulations to assess individual change. As a result, we propose that researchers adopt a standardized approach to calculating rates of clinically significant change for the Saving Inventory-Revised (SI-R) based on Jacobson and Truax’s (1991) two-step method. Specifically, we recommend that individuals whose SI-R total scores have reduced by at least 20 points and whose post-treatment score is 38 or less be classified as recovered. Individuals whose total score decreases by 20 points or more, but whose post-treatment score remains above 38, should be classified as improved but not recovered. Individuals whose total score increases by 20 or more points should be classified as deteriorated. Any individual whose total score has changed by less than 20 points should be classified as not changed. By adopting these criteria, researchers will facilitate cross-study treatment outcome comparisons and aid in our understanding of the impact that hoarding treatment has on its recipients.

In the hoarding literature, treatment outcomes have been evaluated by examining the statistical significance of pre- to post-treatment and between-group differences, the magnitude of effect sizes for these differences, and the proportion of people who meet criteria for clinical significance. Statistically significant differences and effect sizes are limited in that they can lead to erroneous perceptions that meaningful changes have occurred, when in fact, some individuals have shown little to no change or have even deteriorated. To overcome these limitations, Jacobson and Truax (1991) recommended a two-step method for determining clinically significant change at the individual level. The first step is to establish whether the observed change from pre- to post-treatment is more than would be due to imprecise measurement (reliable change). The second step involves establishing whether a post-treatment score is clinically significant (i.e., no longer in the pathological range, within the normative range, or closer to the normative range than to the pathological range). Although many hoarding researchers have assessed for clinically significant change, they have done so in different ways, which has made cross-study comparisons difficult (Ayers et al., 2014; Ayers, Wetherell, Golshan, & Saxena, 2011; Chandler, Fogg, & Smith, 2019; Chou et al., 2019; Crone, Angel, Isemann, & Norberg, 2020; Frost, Pekareva-Kochergina, & Maxner, 2011; Frost, Ruby, & Shuer, 2012; Gilliam et al., 2011; Linkovski et al., 2018; Mathews et al., 2018; Moulding, Nedeljkovic, Kyrios, Osborne, & Mogan, 2017; Muroff, Steketee, Bratiotis, & Ross, 2012; O’Connor et al., 2018; Steketee, Frost, Tolin, Rasmussen, & Brown, 2010; Tolin, Frost, & Steketee, 2007; Worden, Bowe, & Tolin, 2017). As a result, it is unclear if newer treatments fare any better than their predecessors. In this paper, we standardize the calculation of clinically significant change for the most commonly used treatment outcome measure for hoarding disorder, the Saving Inventory-Revised (SI-R; Frost, Steketee, & Grisham, 2004), to encourage researchers to use the same cut points across studies for the instrument’s total and three subscales scores (acquiring, difficulty discarding, and clutter).

Firstly, in order to determine whether change on a measure from pre- to post-treatment is greater than the change due to the measurement error that we would expect over that time interval, Jacobson and Truax (1991) recommend establishing a reliable change index (RCI). The RCI is calculated using Formula 1:

$$\text{RCI}=\frac{\left(x_{post}-x_{pre}\right)}{S_{diff}}$$

In which,

$$S_{diff}=\sqrt{2{\left(S_E\right)}^2}=\sqrt{2}{S}_E\text{and}{S}_E=S{D}_{clinical}\sqrt{1-r_{xx}}$$

The S_diff is the standard error of the difference between the pre- and post-treatment scores (X_pre and X_post, respectively) and describes the spread of the distribution of scores that would be expected during that sampling timeframe if no actual change had occurred. It can be calculated from the standard error of measurement (S_E), which depends on the test-retest reliability of the outcome measure (r_xx) and standard deviation in the clinical group of interest (SD_clinical).

Relevant RCI calculations are illustrated below using data from Table 1. The clinical and nonclinical means and standard deviations in Table 1 were taken from a psychometric paper on the SI-R (Kellman-McFarlane et al., 2019), whereas reliability estimates were derived from waitlist control samples for hoarding treatment studies (Chasson et al., 2015; Chasson et al., 2020; Tolin et al., 2019). As an RCI greater than ±1.96 is larger than would be expected due to random fluctuation (p < 0.05), an SI-R total score needs to change by at least 20 points, an acquiring subscale score needs to change by 7 points, a discarding subscale score needs to change by 8 points, and a clutter subscale score needs to change by 9 points.

Table 1.

Data used to calculate clinically significant change cut-off scores for the Saving Inventory-Revised

Symbol	Definition	Total Score Value	Acquiring Score Value	Discarding Score Value	Clutter Score Value
M clinical a	Mean SI-R total score for hoarding samples	59.17	15.22	19.09	24.85
Mnonclinical a	Mean SI-R total score for nonhoarding samples	21.57	6.43	7.91	7.25
SD clinical a	Standard deviation for the hoarding samples	13.56	5.35	4.60	6.70
SD nonclinical a	Standard deviation for the nonhoarding samples	18.22	5.51	6.81	7.61
r xx b	Test-retest reliability of the SI-R	0.74	0.81	0.62	0.78

^aValue was taken from Kellman-McFarlane et al. (2019).

^bValue derived from three waitlist control samples (Chasson et al., 2015; Chasson et al., 2020; Tolin et al., 2019).

The average Time 1 SI-R total score was 56.51 (SD = 12.94), the average Time 2 SI-R total score was 51.40 (SD = 12.73), and the average duration in between these assessments was 13.18 weeks (SD = 4.57).

$\text{Total:}{S}_E=13.56\sqrt{1-0.74}$	$S_{diff}=\sqrt{2}\times 6.91$	$\text{RCI}=\frac{\left(x_{post}-x_{pre}\right)}{9.77}$
$\text{Acquiring:}{S}_E=5.35\sqrt{1-0.81}$	$S_{diff}=\sqrt{2}\times 2.33$	$\text{RCI}=\frac{\left(x_{post}-x_{pre}\right)}{3.30}$
$\text{Discarding:}{S}_E=4.60\sqrt{1-0.62}$	$S_{diff}=\sqrt{2}\times 2.84$	$\text{RCI}=\frac{\left(x_{post}-x_{pre}\right)}{4.02}$
$\text{Clutter:}{S}_E=6.70\sqrt{1-0.78}$	$S_{diff}=\sqrt{2}\times 3.14$	$\text{RCI}=\frac{\left(x_{post}-x_{pre}\right)}{4.44}$

Secondly, Jacobson and Truax (1991) provided three different methods for determining whether a post-treatment score is clinically significant, with choice of method to be determined by the data that researchers have available to them. When nonclinical normative data are unavailable but clinical normative data exist, researchers should use Criterion A, which determines if a person’s level of functioning is two standard deviations away from the clinical normative mean on a measure. When clinical normative data are unavailable but nonclinical normative data exist, researchers should use Criteria B, which determines if a person’s level of functioning is within two standard deviations of the nonclinical normative mean on a measure. When both clinical and non-clinical normative data are available for a measure, researchers should use Criterion C for deriving a clinically significant cut-off score. Criterion C is meant to establish that an individual’s functioning is closer to the normative mean of the nonclinical population (M_nonclinical) than the normative mean of the clinical population (M_clinical) as shown in Formula 2:

$$C=\frac{(S{D}_{clinical}{M}_{nonclinical})+\left(S{D}_{nonclinical}{M}_{clinical}\right)}{(S{D}_{clinical}+S{D}_{nonclinical})}$$

Using normative data from Table 1, which was obtained from Kellman-McFarlane et al. (2019), we calculated the following Criterion C cut-off scores for the SI-R using Formula 2:

Total:	$C=\frac{(13.56\times 21.57)+(18.22\times 59.17)}{(13.56+18.22)}$	$C=43.12$
Acquiring:	$C=\frac{(5.35\times 6.43)+(5.51\times 15.22)}{(5.35+5.51)}$	$C=10.89$
Discarding:	$C=\frac{(4.60\times 7.91)+(6.81\times 19.09)}{(4.60+6.81)}$	$C=14.58$
Clutter:	$C=\frac{(6.70\times 7.25)+(7.61\times 24.85)}{(6.70+7.61)}$	$C=16.61$

Accordingly, the halfway point between the clinical and nonclinical means for the SI-R total score is 43.12, which means that scores 42 and lower should be closer to the nonclinical population than the clinical population. However, using receiver operating characteristic analysis and the Youden’s J statistic, Kellman-McFarlane et al. (2019) determined that total scores of 39 and greater are likely to be associated with a diagnosis of hoarding disorder. Given this, it does not seem that scores between 39 and 42 indicate that a client is closer to the nonclinical population than to the clinical population. The half-way point for the discarding subscale also is above the clinical cutoff established by Kellman-McFarlane et al. (2019), whereas the half-way points for the acquiring and clutter subscales are at the clinical cutoffs. As such, we suggest that the criterion for clinical significance for SI-R scores be one point below the clinical cut-off scores established by Kellman-McFarlane et al. (i.e., total score = 38; acquiring subscale score = 10; discarding subscale score = 12; clutter = 16) to ensure that scores are closer to the non-clinical population than the clinical population. It should be noted that this recommendation is less stringent than following Jacobson and Truax’s (1991) Criterion A, which requires clinical significance scores be completely outside the pathological range.

Patients can be classified as recovered, improved but not recovered, not changed, or deteriorated using Jacobson and Truax’s two-step approach for determining clinically significant change. In view of that, individuals whose SI-R total scores have reduced by at least 20 points and whose post-treatment score is 38 or less should be classified as recovered. Individuals whose total score decreases by 20 points or more, but whose post-treatment score remains above 38, should be classified as improved but not recovered. Individuals whose total score increases by 20 or more points should be classified as deteriorated. Any individual whose total score has changed by less than 20 points should be classified as not changed. See Table 2 for a summary and subscale classifications.

Table 2.

Clinically significant change cut points for the Saving Inventory- Revised

Scale	Recovered	Improved but not Recovered	Deteriorated	Not Changed
Total	Score has reduced by at least 20 points and is 38 or below	Score has reduced by at least 20 points, but is 39 or higher	Score has increased by at least 20 points	Score has not changed by at least 20 points
Acquiring	Score has reduced by at least 7 points and is 10 or below	Score has reduced by at least 7 points, but is 11 or higher	Score has increased by at least 7 points	Score has not changed by at least 7 points
Discarding	Score has reduced by at least 8 points and is 12 or below	Score has reduced by at least 8 points, but is 13 or higher	Score has increased by at least 8 points	Score has not changed by at least 8 points
Clutter	Score has reduced by at least 9 points and is 16 or below	Score has reduced by at least 9 points, but is 17 or higher	Score has increased by at least 9 points	Score has not changed by at least 9 points

Note. We do not recommend looking at clinical significance on its own as some individuals may score below a cut-off prior to beginning treatment. Thus, both reliable change and clinical significance must be assessed in tandem.

Conclusion

To date, treatment outcome studies for hoarding disorder have reported different rates of clinically significant change. This could in part be due to using different normative data and discrepant applications of Jacobson and Truax’s (1991) two-step approach. This hinders our ability to assess whether new treatments are more effective (or less) than other treatments. As a result, we recommend that all future hoarding treatment outcome studies use the SI-R cut scores presented within this paper to report on clinically significant change. Doing so will enhance the comparability of study results, as every treatment will be evaluated using the same criteria. It will additionally emphasize how well treatment leads to recovery.

The test-retest reliability coefficients used in this manuscript were substantially lower than what Frost et al. (2004) reported in their psychometric paper for the SI-R, and hence, our recommended RCI is greater than that used in past research. Frost et al.’s test-retest correlation was derived using a two-to-four week interval when assessing 12 patients. Not only can findings from small samples be unreliable (Button et al., 2013), but the duration between tests can affect reliability estimates (Allen & Yen, 1979). As longer test-retest durations are associated with lower reliability estimates (Trochim, Donnelly, & Arora, 2016), using a test-retest coefficient from a duration shorter than that used in treatment outcomes studies will lead to an artificially low RCI. Consequently, it is important to calculate the RCI using a coefficient derived from a similar test-retest duration as was done in this manuscript.

Although our recommended RCI is higher than that used in prior studies, Ayers et al. (2014; 2011) have used an approach that leads to a similar result. They have defined clinically significant change as a 35% reduction on two hoarding severity outcome measures. This corresponds to a 21-point reduction in scores when a patient’s pre-treatment score is a 59—the average score of an untreated patient with hoarding disorder (Kellman-McFarlane et al., 2019). Using this approach, Ayers et al. (2011) found that as few as 25% of patients improve with treatment and that this improvement may not last.

We hope that our standardization of clinically significant change criteria will assist in ongoing efforts to evaluate and improve treatment for hoarding disorder. We recommend that our criteria for clinically significant change be used as part of a comprehensive assessment of treatment outcomes that also includes structured clinical interviews, objective assessments of clutter, and measures of safety, impairment, and quality of life. We also encourage researchers to validate our criteria by determining if patients who recover as a result of treatment report less risk and impairment in their ability to function within the home and greater enjoyment and satisfaction with life compared to patients who fall into the other three classifications.

Highlights.

• It is difficult to compare outcomes across treatment studies for hoarding disorder

• Definitions for clinically significant change have varied

• We propose standardized metrics for clinically significant change for the SI-R