Defining cognitive-behavior therapy response and remission in pediatric OCD: A signal detection analysis of the Children’s Yale Brown Obsessive Compulsive Scale

Abstract

The objective of the study was to examine the optimal Children’s Yale Brown Obsessive-Compulsive Scale (CY-BOCS) percent reduction and raw cutoffs for predicting cognitive-behavioral treatment (CBT) response among children and adolescents with obsessive-compulsive disorder (OCD). Children and adolescents with OCD (N=241) participating in the first step of the Nordic Long-term OCD treatment Study and receiving 14 weekly sessions of CBT in the form of exposure and response prevention. Evaluations were conducted pre- and post-treatment, included the CY-BOCS, Clinical Global Impressions – Severity/Improvement. The most efficient CY-BOCS cutoffs were 35% reduction for treatment response, 55% reduction for remission, and a CY-BOCS raw total score of 11 for treatment for remission. The results show a divergence from previous research on pediatric OCD with more conservative cutoffs (higher cutoff reduction for response and remission and lower raw score for remission). Further research on optimal cutoffs is needed.

Obsessive-compulsive disorder (OCD) is a chronic and disabling disorder [31, 32, 39] with a population prevalence rate of 1–2% [5, 30, 32, 50]. Studies have shown the efficacy of cognitive-behavior therapy (CBT) and serotonin reuptake inhibitors [14, 36]. Expert guidelines recommend CBT as the first line treatment for mild and moderate OCD, and combined medication and CBT for moderate to severe OCD [12] and unsuccessful CBT [12, 25]

The Children’s Yale-Brown Obsessive-Compulsive Scale (CY-BOCS) [34] is the gold standard, dimensional measure of symptom severity in pediatric OCD is the most commonly used treatment outcome measure [22]. It is a semi-structured interview of OCD symptomology with acceptable psychometric properties [24, 34, 41]. Other measures frequently used are the Clinical Global Impressions scales of Severity and Improvement (CGI-S/CGI-I) [1, 11] measuring the overall illness severity and treatment improvement, respectively.

Despite the existence of these instruments, few definitions for characterizing treatment response (no longer fully symptomatic but may continue to evidence some minimal symptoms, e.g. [45]) or remission (“no longer meets syndromal criteria” [7]) are consistently used or have empirical support. Consequently, it has been difficult to compare outcomes across studies and apply clinical results to clinical practice [24]. For instance, the operational definition of treatment response has ranged from 20% [28] to 50% [3, 35] symptom reduction on the CY-BOCS (See [45] for a thorough overview). Similarly, the operational definition of treatment remission has varied from a raw score of seven [38], eight [51], ten [27, 29, 46] and, 12 [48] on the post-treatment CY-BOCS total score.

Previous studies in OCD and anxiety [2, 23, 40, 45] suggest using the rating for improvement (CGI-I) and severity (CGI-S) conducted by the clinician as a benchmark. Typically signal detection analysis (receiver operational characteristics) [43] has been used to compare judgments of response and remission with symptom reduction on symptom measures (e.g., the CY-BOCS). Storch et al. [40] used this method by comparing the CGI-I/S with the CY-BOCS total score in 109 children and adolescents that were treated with CBT for 14 weeks. The most efficient CY-BOCS cut-off score was 25% reduction in treatment response with positive and negative predictive values of .96 and .86 respectively. Likewise, 45–50% reduction was the most effective cut-off for remission with positive and negative predictive values of .96 and .72 respectively. A CY-BOCS total raw score of 14 was the most effective cut-off for remission based on the CGI-I with positive and negative predictive value of .96 and .79 respectively. Among adults, 30% or 45% reduction on the Yale-Brown Obsessive-Compulsive Scale was the most efficient cutoff for response using the CGI-I and 40–55% reduction was optimal cutoff for remission using the CGI-S [23, 45]. Twelve was the most optimal raw cutoff score for remission [23].The aim of this study is to replicate Storch et al. [40] using a large and cross-cultural sample of pediatric OCD patients treated with CBT in community child and adolescent clinics in Denmark, Sweden, and Norway within the Nordic Long-term OCD Treatment Study (NordLOTS) [44]. NordLOTS offers a large and unique sample (N=269) consisting of well-characterized youth who were medication free prior to treatment initiation. More specifically this investigation examines whether the proposed cut-offs identified by Storch et al. is consistent with this this cross-cultural sample from the NordLOTS study.

Methods

Participants

Two hundred sixty-nine participants (138 females) with a primary OCD diagnosis were included in the NordLOTS step 1. The mean baseline CY-BOCS total score was 24.6 (SD =5.1), subscale scores for obsession 12.3 (SD=2.8) and compulsion 12.3 (SD =2.7), which is comparable to the average range for treatment seeking youth in OCD specialty centers [24]. Child age ranged from 7–17 years (M=12.8, SD=2.7) years. The sample was primarily of Scandinavian ethnicity as 97% (n=261) had one or both parents of Scandinavian origin. Almost half (40.5%, n=109) of the patients had one or more comorbid psychiatric disorder, with common comorbid conditions including anxiety disorders (19.3%, n=52), tic disorders (18.6%, n=49), and attention-deficit hyperactivity disorder (8.9%, n=24), depressive disorders (3.7%, n=10), and oppositional defiant disorder/conduct disorder (3.7%, n=10).

Patients were included in the study if they fulfilled the following criteria: (1) primary diagnosis of OCD in accordance with the criteria in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (DSM-IV) (American Psychiatric Association, 2000), (2) CY-BOCS entry score ≥16, (3) 7 to 17 years of age, and (4) patients with attention-deficit-hyperactivity disorder (ADHD) were eligible, after having been stabilized on medication for at least 3 months prior to entry. Exclusion criteria included: (1) the presence of other psychiatric disorders having a higher treatment priority (i.e., psychosis and severe depression), (2) any specific developmental disorder (i.e., autism spectrum disorders). However, a diagnosis of PDD NOS was allowed as long as OCD was judged to be the primary disorder based on the respective Clinical Global Impression-Severity (CGI) scores, (3) a previous failed trial of exposure-based CBT for OCD within 6 months of inclusion, (4) medication treatment with an SRI less than 6 months of inclusion, and (5) inadequate language proficiency by the patient or the parent. See Torp and colleagues [46] for a detailed description of the NordLOTS sample.

Procedure

The trial was approved by the Norwegian, Swedish and Danish Committees for Medical and Health Research Ethics and the Medical Products Agencies. The project was registered in Current Controlled Trials (www.controlled-trials.com ISRCTN66385119). Informed consent was provided by parent(s) or guardian(s) and informed assent obtained from children 11 years of age or older. A diagnosis of OCD was made prior to the initial assessment (see below). Assessment points were at baseline, weeks 7 and 14 (post treatment), assessed by independent evaluators (IEs) using the CY-BOCS.

After assessment at baseline participants received 14 weekly sessions of exposure-based CBT regime consisted of 75 minutes. Parents were expected to accompany their children to all sessions. The children were seen together with their parents in six of the fourteen sessions (sessions 1–3, 5, 11, and 14). In the remaining sessions, the child was treated individually for 45 minutes and then the parents were seen with or without the child for an additional 30 minutes. This extra time was added specifically to address issues regarding the parents’ involvement in therapy and their attitude and feelings about their child’s OCD symptoms. Treatment is detailed in Torp et al. [46, 47].

Measures

The Schedule for Affective Disorders and Schizophrenia for School-Age Children-Present and Lifetime (K-SADS-PL) [15] is a semi-structured diagnostic interview that assesses a variety of child psychopathology. The K-SADS-PL demonstrates favorable psychometric properties, with an excellent inter-rater reliability of 98% and a 1 to 5-week test-retest kappa of .80 for anxiety disorder diagnosis [15]. Convergent and divergent validity [19] and the inter-rater reliability [20] of the K-SADS-PL have been documented in a Nordic sample of adolescents. In addition, the K-SADS-PL has been used in previous OCD treatment trials [9, 49]. Symptoms can be classified as “not present”, “possible”, “in remission” or “certain”. In this study OCD diagnoses and comorbidity were based on symptoms classified as “certain” only. The K-SADS-PL was used for diagnostic assessment at baseline of Step 1. All interviews were conducted by experienced clinicians, trained by the NordLOTS research group. Tic disorders included Tourette’s disorder and chronic motor or vocal tic disorder as defined by the K-SADS-PL.

The CY-BOCS evaluates the severity of obsessions and compulsions, using ten items across five dimensions (time occupied by symptoms, interference, distress, resistance and degree of control over symptoms). The total severity score can range from 0 to 40. CY-BOCS total scores in the range of 14–24 are considered moderate, 25–30 moderate-severe and over 30 severe (based on a normative study of 815 treatment seeking youth [24]). The CY-BOCS shows reasonable reliability and validity [10, 33, 42]. In particular, high internal consistency for the total score .87 [33] and good to excellent inter-rater agreement [intra-class correlation coefficient ICC)] have been reported [33], .84, .91 and .68 for total score, obsessions and compulsions, respectively). In the NordLOTS sample, the inter-rater agreement (ICC) was .92, .94, and .87 for total score, obsessions and compulsions, respectively [46].

The Clinical Global Impression scales [13] are two single item clinician ratings that measure severity and improvement. The CGI-Severity scale (CGI-S) was used by the clinician to rate the global severity of symptom severity. Ratings range from 0 (no illness) to 6 (extremely severe). It correlates strongly with the CY-BOCS total score in pediatric OCD patients, and is widely used and has been shown to be treatment sensitive [1]. Consistent with Storch et al. [40] we used a score of 0 or 1 (no illness or mild illness) for remission. The CGI-Improvement scale (CGI–I) was used to assess overall clinical improvement based on symptoms observed and impairment reported using a seven-point scale ranging from 0 (very much worse) to 6 (very much improved). Consistent with Storch et al. [40] ratings of 5 (much improved) or 6 (very much improved) designated treatment response.

Statistical analysis

To evaluate the properties of test cutoffs, we used receiver operating characteristic (ROC) analyses that are originally based on signal detection theory [43]. These analyses focus on classifying the rate of true and false diagnostic judgments at various thresholds (cut-offs) in order to identify the sensitivity and specificity of a measure at various discrimination threshold. In particular, we evaluated a number of ROC parameters, including sensitivity, specificity, positive predictive value, negative predictive value, and efficiency. Sensitivity is defined as the proportion of true positives captured by the test cutoff, and specificity is defined as the proportion of true negatives that fall below the test cutoff. The positive predictive value is the probability that all patients who meet the gold standard also exceed the test cutoff, and negative predictive value is the probability that all patients who do not meet the gold standard do not exceed the cutoff. Efficiency (also called accuracy) is the probability that the cutoff diagnosis and the gold standard agree.

Consistent with analyses in previous studies [40, 45] we broke up the CY-BOCS percentage reduction into 5% interval cut-offs in order to examine the different discrimination accuracy compared to remission (CGI-S) and response (CGI-I). We also examined the properties of different CY-BOCS raw score cutoffs in predicting symptom remission (CGI-S). However, raw scores were not used to predict response.

We also used quality receiver operation characteristic methods (QROC) in order to correct for errors in the CGIs [17, 18]. QROC reports specific forms of weighted κ statistics to measure quality of specificity κ(0.0), quality of efficiency κ(0.5), and, quality of sensitivity κ(1.0). For each measure, a value of 0 indicates that one cannot discriminate the cut-off from chance while a value of 1.00 indicates that one can perfectly discriminate between patients that responded/remitted. As previous studies [40] the current study focused on maximizing efficiency κ(0.5) by selecting the most efficient cut-offs and minimizing false-positive and false-negative equally. We conducted the analyses using the ROC4 program (available at http://www.stanford.edu/~yesavage/ROC.html).

Results

Missing data

Data with both participants’ baseline and week 14 measures were included. No data were missing for the CGI and CY-BOCS at baseline. At week 14, 29 (10.8%) of the CY-BOCS were missing and 30 (11.1%) of the CGI ratings made by therapists. We compared whether completers and non-completers differed on all predictors [used in a previous study [47]] at baseline and week 7. No significant group differences were detected at the p<.05 level

Sample characteristics

The mean CY-BOCS at baseline was 24.60 (SD=6.56), and at week 14 11.40 (6.70). As reported in Torp et al. [46] the pre-post difference was statistically significant. The mean reduction from baseline to week 14 was 52.9% (SD=30.9%). The mean baseline CGI-S was 3.43 (SD=0.84) and the mean week 14 CGI-S was 1.78 (SD=1.19). The difference between baseline and week 14 was significant using a paired samples t-test (t(215)=17.86, p<.001). The mean reduction was 54.5% (SD=39.6%). The mean CGI-I at week 14 was 4.82 (SD=1.00). At week 14, 48.3% (n=130) met criteria for remission using the CGI-S. Similarly, 61.7% (n=166) were responders using the CGI-I.

Predicting Treatment Response with CY-BOCS Percent Reduction

Table 1 shows series of CY-BOCS percentage symptom reduction cutoffs used to predict treatment response using the dichotomized CGI-I, along with ROC statistics for each cutoff. We conducted ROC analyses for each 5% symptom reduction on the CY-BOCS. We found maximum efficiency (.61) at a cut-off of 35% with positive predictive value of .88 and negative predictive value of .74, indicating a false-positive rate of 12% and a false-negative rate of 26%.

Table 1.

Prediction of treatment response at varying CY-BOCS percentage reduction cutoffs.

Value	Sensitivity	Specificity	Efficiency	Κ(0)	Κ (0.5)	Κ (1)	Positive Predictive Valuec	Negative Predictive Valued
≥5	1.00	0.24	0.77	0.18	0.31	1.000	0.75	1.00
≥10	0.99	0.27	0.77	0.20	0.33	0.931	0.75	0.95
≥15	0.99	0.34	0.79	0.25	0.40	0.893	0.77	0.93
≥20	0.96	0.38	0.78	0.28	0.40	0.745	0.78	0.82
≥25	0.95	0.45	0.79	0.33	0.45	0.690	0.79	0.79
≥30	0.93	0.58	0.82	0.46	0.55	0.685	0.83	0.78
≥35*	0.89	0.72	0.83	0.60	0.61	0.618	0.88	0.74
≥40	0.85	0.76	0.82	0.63	0.59	0.554	0.89	0.69
≥45	0.78	0.82	0.80	0.71	0.60	0.463	0.91	0.63
≥50	0.73	0.84	0.76	0.71	0.50	0.392	0.91	0.58
≥55	0.68	0.88	0.74	0.76	0.47	0.344	0.93	0.55
≥60	0.60	0.89	0.69	0.76	0.41	0.277	0.93	0.50
≥60	0.49	0.89	0.62	0.71	0.30	0.190	0.91	0.44
≥70	0.42	0.91	0.57	0.71	0.25	0.148	0.91	0.41

^*Optimal cut-off value based on the Kappa (κ(0.5))

Predicting Remission with CY-BOCS percent Reduction

Table 2 shows a series of CY-BOCS percent reduction cut-offs and ROC analyses. The maximum quality of efficiency (.74) was found at 55% symptom reduction on the CY-BOCS, with positive and negative predictive values at .83 and .92 respectively corresponding to false positive rate of 17% and false negative rate of 8%.

Table 2.

Prediction of treatment remission (per CGI-S at varying CY-BOCS reduction cut-offs.

Value	Sensitivity	Specificity	Efficiency	Κ(0)	Κ (0.5)	Κ (1)	Positive Predictive Value	Negative Predictive Value
≥5	1.00	0.14	0.53	0.07	0.13	1.00	0.50	1.00
≥10	1.00	0.16	0.55	0.08	0.15	1.00	0.50	1.00
≥15	1.00	0.21	0.57	0.11	0.19	1.00	0.52	1.00
≥20	1.00	0.26	0.60	0.14	0.25	1.00	0.53	1.00
≥25	1.00	0.32	0.63	0.18	0.30	1.00	0.56	1.00
≥30	0.99	0.42	0.68	0.24	0.39	0.96	0.59	0.98
≥35	0.99	0.55	0.75	0.35	0.52	0.97	0.65	0.99
≥40	0.98	0.61	0.78	0.41	0.57	0.95	0.68	0.98
≥45	0.95	0.70	0.81	0.50	0.63	0.87	0.73	0.94
≥50	0.94	0.77	0.85	0.58	0.69	0.86	0.77	0.93
≥55*	0.91	0.84	0.87	0.68	0.74	0.82	0.83	0.92
≥60	0.82	0.86	0.84	0.69	0.68	0.67	0.83	0.85
≥60	0.69	0.89	0.80	0.71	0.59	0.51	0.84	0.77
≥70	0.64	0.95	0.80	0.83	0.60	0.47	0.91	0.75

^*Optimal cut-off value based on the Kappa (κ(0.5))

Comparison of Treatment Response and Remission

Figure 1 depicts the κ(0.5) over the series of percent reduction cutoffs on the CY-BOCS on predicting treatment response and remission. Response shows that the maximal κ(0.5) is at 35% reduction on the CY-BOCS while the maximal κ(0.5) for remission is at 55% reduction.

Figure 1.

Quality index of efficiency [κ(0.5)] for the predictive values of the CY-BOCS percent cutoffs corresponding to response and remission using the CGI-I and CGI-S.

Predicting remission with CY-BOCS total scores

Table 3. shows a series of CY-BOCS post-treatment raw cut-off scores with ROC analyses that we used to predict the dichotomized CGI-S symptom reduction (mild or no illness). Maximal quality of efficiency (.69) was CY-BOCS raw cut-off score of ≤11 with positive predictive value of .88 (corresponding to 12% false positive) and negative predictive value .81 (corresponding to 19% false negative).

Table 3.

Prediction of treatment remission (per CGI-S at varying CY-BOCS raw cut-off scores according to therapists’ and independent evaluator (IE) respectively.

Value	Sensitivity	Specificity	Efficiency	Κ(0)	Κ (0.5)	Κ (1)	Positive Predictive Value	Negative Predictive Value
≤5	0.99	0.45	0.74	0.30	0.45	0.93	0.68	0.96
≤6	0.97	0.51	0.76	0.35	0.50	0.88	0.70	0.93
≤7	0.96	0.56	0.78	0.39	0.53	0.86	0.72	0.92
≤8	0.95	0.60	0.79	0.43	0.57	0.85	0.74	0.92
≤9	0.94	0.72	0.84	0.56	0.67	0.83	0.80	0.91
≤10	0.87	0.80	0.84	0.64	0.67	0.70	0.84	0.84
≤11*	0.82	0.87	0.85	0.75	0.69	0.64	0.88	0.81
≤12	0.79	0.90	0.84	0.79	0.69	0.60	0.90	0.79
≤13	0.70	0.96	0.82	0.89	0.64	0.50	0.95	0.73
≤14	0.65	0.99	0.80	0.97	0.62	0.45	0.99	0.70
≤15	0.59	1.00	0.78	1.00	0.56	0.39	1.00	0.67
≤16	0.51	1.00	0.73	1.00	0.49	0.32	1.00	0.63
≤17	0.47	1.00	0.71	1.00	0.45	0.29	1.00	0.61
≤18	0.42	1.00	0.69	1.00	0.40	0.25	1.00	0.59
≤19	0.37	1.00	0.66	1.00	0.35	0.21	1.00	0.57
≤20	0.33	1.00	0.64	1.00	0.31	0.19	1.00	0.56

^*Optimal cut-off value based on the Kappa (κ(0.5))

Discussion

It is important to standardize treatment outcome criteria so as to facilitate comparisons across trials and establishment of treatment guidelines. Currently, the inconsistent definition of terms such as response or remission is the rule [21, 26, 45]. For instance, the range of cutoffs using the CY-BOCS percent reduction has varied from 20% to 50% in different trials. This variation substantially hinders the ability to determine the most favorable target for OCD symptom reduction across trials. When a criterion is too conservative, fewer true positives are identified (decreased sensitivity). Likewise, when a criterion is less conservative one captures a higher proportion of true positives (sensitivity is increased) but with less specificity. The aim of the current study was to replicate previous research [40] in a Scandinavian-majority sample by examining series of CY-BOCS percent reduction that correspond with treatment response and remission, and series of raw CY-BOCS scores that correspond to remission.

We found that 35% reduction on the CY-BOCS had optimal correspondence (i.e., sensitivity and specificity equally was optimal at that cutoff) with treatment response. The rate of false-positives (children classified as responders without substantial improvement) was 12%, and the rate of false negatives (failure to classify true responders) was 26%. These results are somewhat different that in Storch and colleagues [40] who found optimal cutoff at 25% symptom reduction. In addition, their false-positive (4%) and false-negative (14%) rates were lower indicating better classification. One explanation for the differences in the percent reduction threshold is that a higher symptom reduction may have corresponded to greater improvement in this Scandinavian sample compared to the sample in Storch and colleagues. The consequences of lowering the threshold corresponding to Storch and colleagues (25% reduction) is that one increases children classified as responders without substantial improvement from 12% to 21%. However, the false-negative rate decreases from 26% to 21%. Consequently, this produces a lower quality of efficiency.

With regards to remission, a 55% symptom reduction was identified as optimal with 17% false-positive rate and 8% false-negative rate. This is a similar cutoff as found by Storch and colleagues [40] that showed that 50% symptom reduction had optimal correspondence with remission with false-positive rate of 4% and false-negative rate of 28%. Based on our data 50% symptom reduction cutoff means that we increase the false-positive rate to 23% (increase in children classified as responders without substantial improvement) while decreasing the false-negative rate to 8% (reducing the failure to classify true responders). We found that the raw cutoff score of 11 was optimal correspondence with remission with false-positive rate of 12% and false-negative rate of 19%. This cutoff is lower than reported in Storch and colleagues. They found that the raw cutoff score of 14 had the maximum efficiency with 4% false-positives and 21% false-negatives. Based on our data the consequences of increasing the raw cutoff score of 14 would result in decreasing the quality of efficiency slightly (.62), the false-positive rate would decrease (1%) (less children classified as responders without substantial improvement). However, this also results in a high increase of false-negatives of 30% (increasing the failure to classify true responders).

Taken together, we found similar percentage symptom reduction cut-off as Storch and colleagues [40]. However the symptom reduction cutoff for response and the raw cutoff score for remission were more conservative than those found by Storch and colleagues. However, the quality of the efficiency was generally higher in Storch and colleagues (.77, .72, and .77 for CY-BOCS percent reduction vs. response, remission, and CY-BOCS raw score vs. remission respectively). In our study, the quality of efficiency was .61, .74, and .69 for CY-BOCS percent reduction vs. response, remission, and CY-BOCS raw score vs. remission respectively.

Not surprisingly, remission had a higher threshold than response consistent with the concepts of response and remission [7]. As response is characterized as an improvement in symptoms following treatment [e.g., 5–50% [6, 8, 16]] and some responders although they benefited from a treatment may still have an active disorder that needs further improvement. However, remission is a more stringent criteria that requires that patients improve to the degree of virtual absence of syndromal criteria [4, 7, 26].

Strengths and limitations

Within the strengths of this study (e.g., well-characterized sample, methodological rigor, multi-site nature), all patients were free from OCD medications and all underwent the same standardized 14 weekly sessions of CBT. In addition, treatment was conducted in regular community child and adolescent psychiatry clinics and few exclusionary criteria were applied, making the results applicable to clinic-referred youth.

However, the study has several limitations. For instance, our results for the correspondence between percent reduction and response and CY-BOCS raw score and remission are lower than reported in Storch and colleagues, possibly indicating lower performance for these cutoffs in our data. Whether this indicates real differences which may be attributed to factors such as cross-cultural differences or measurement errors of the translated CY-BOCS merits evaluation in subsequent studies. In addition, the CGI ratings were by raters doing the CY-BOCS and may have been influenced by the CY-BOCS.

Implications

Our findings have implications for research and practice. By establishing percent reduction and raw total score on the CY-BOCS that corresponds to response and remission clinicians and researchers alike can more easily assess the treatment progress of individual child against the standard outcomes obtained in clinical trials. Established cutoffs can assist in decision-making for instance by informing whether to switch or augment treatment in the absence of response, continue treatment in the presence of response without remission or provide boosters sessions rather than regular treatment in the presence of remission. For example, if a patient does not obtain 35% symptom reduction after 14 weeks of CBT the clinician might consider adding or switching to SSRI [37]. However, if the patient has obtained at least 35% symptom reduction but still has a high CY-BOCS total score (above 11) he should receive further CBT sessions until he reaches the threshold of remission.