Development and validation of a progress monitoring tool tailored for use in intensive eating disorder treatment

Abstract

OBJECTIVE:

Despite calls for routine use of progress and outcome monitoring in private and intensive treatment centers for eating disorders (EDs), existing measures have limited relevance to these supervised treatment settings. This study sought to develop and validate the Progress Monitoring Tool for Eating Disorders (PMED), a multidimensional measure for progress monitoring in the context of intensive ED treatment.

METHOD:

Thirty-seven items were generated by a team of content experts, clinicians, and administrative staff from the target treatment setting. Adolescent and adult females (N = 531) seeking residential ED treatment completed the items at admission as part of the clinic’s routine assessment battery; 83% were retained for repeat assessment at discharge. Exploratory factor analysis was conducted for preliminary measure development.

RESULTS:

Results yielded a five-factor, 26-item structure explaining 50% of total variance. Final construct domains included: Weight and Shape Concern, ED Behaviors and Urges, Emotion Avoidance, Adaptive Coping, and Relational Connection. The measure demonstrated adequate internal consistency, sensitivity to change during treatment, and convergence with validated assessment measures.

CONCLUSIONS:

Preliminary data support the PMED as a novel and valid multidimensional measure of treatment-relevant constructs. This measure may have utility in measuring treatment progress for patients receiving intensive treatment for EDs.

Eating disorders (EDs) affect approximately 30 million Americans annually (Hudson, Hiripi, Pope, & Kessler, 2007; Wade, Keski‐Rahkonen, & Hudson, 2011) and are among the most medically compromising and life-threatening psychological disorders (Harris & Barraclough, 1998; Smink, van Hoeken, & Hoek, 2013). Relative to those with other psychological disorders, patients with EDs are particularly prone to poor treatment response and early termination (Fassino, Pierò, Tomba, & Abbate-Daga, 2009; Posse & Nemeroff, 2012). Among those who complete outpatient treatment, more than one-third fail to achieve clinically significant improvement (Shapiro et al., 2007; Watson & Bulik, 2013). Patients treated in intensive settings, such as inpatient and residential programs, typically present with more severe and complex ED psychopathology (Delinsky et al., 2010; Twohig, Bluett, Torgesen, Lensegrav-Benson, & Quakenbush-Roberts, 2015), which is generally associated with poorer treatment outcome (Vall & Wade, 2015). Significant room exists to further improve ED patient outcomes.

One means by which to improves outcomes is through use of routine outcome monitoring or progress monitoring (PM) procedures (Lambert & Harmon, 2018; Shimokawa, Lambert, & Smart, 2010). This refers to regular, quantitative assessment of symptom change throughout treatment, including upon treatment initiation and termination (Overington & Ionita, 2012). Such practices allow for real-time detection of treatment nonresponse or risk for poor outcome, in addition individualization of evidence-based interventions depending on a patient’s response to current treatment (Boswell, Kraus, Miller, & Lambert, 2013; Gondek, Edbrooke-Childs, Fink, Deighton, & Wolpert, 2016). Aggregation of treatment response data across all patients in a treatment program can also identify patient subgroups who tend to respond well or less well to a given approach (Boswell, Kraus, Castonguay, & Youn, 2015). Robust evidence supports their utility in improving outcomes in general outpatient psychotherapy settings (Lambert & Harmon, 2018; Shimokawa et al., 2010).

In line with this evidence, recent regulatory policy and calls from academic scientists have increased pressure to monitor and report on patient outcomes in ED treatment, particularly in private-sector programs in the United States and those offering higher levels of care such as residential and day hospital treatment (Guarda, Wonderlich, Kaye, & Attia, 2018; The Joint Commission, 2016, 2018). Unfortunately, although outcome monitoring from pre- to post-treatment is common in residential and other intensive treatment settings (Brown et al., 2018; Frisch, Herzog, & Franko, 2006), it is unknown how these data are used to inform individualized care while patients undergo therapy for their EDs, and we are aware of few programs using PM systematically to individualize care and prevent treatment nonresponse.

Clinicians cite many barriers to use of PM in clinical practice, for example concern over the time burden associated with use (Hatfield & Ogles, 2007; Zimmerman & McGlinchey, 2008). In intensive ED treatment, an additional barrier is the dearth of ED-specific PM measures available. Most available PM measures assess overall well-being or general functioning in daily life, including occupational and social role functioning (Barkham et al., 2001; Boswell et al., 2015; Lambert et al., 1996). These constructs may have limited relevance to patient outcomes in the structured setting of acute inpatient or residential psychiatric treatment. In addition, patients in intensive day or residential treatment are often far removed from their usual activities of daily living (e.g., school, work, household chores, social and leisure activities, etc.). Therefore, engagement in activities associated with one’s social, work, and/or academic roles is limited, and change in role functioning is unlikely to occur during the acute treatment phase.

Further consideration must be made for assessment of patients with EDs in particular. First, the intrinsically reinforcing clinical features of certain disordered eating behaviors (e.g., dietary restriction) may promote poor insight into impairment caused by the disorder and could lead to inflated estimates of role functioning by patient report (Vitousek, Daly, & Heiser, 1991). Therefore, for this setting and population, other elements of functioning (e.g., emotional coping and/or treatment engagement) may be more appropriate to assess. Another consideration must be made for specific ED symptom metrics in this setting. Patients receiving residential or day hospital treatment are under frequent supervision and have limited ability to engage in ED behaviors such as binge eating, compensatory behaviors, and excessive exercise. Low reports of these symptoms may represent a force of circumstances rather than a reduction in psychopathology, thus limiting accuracy of symptom reports on existing measures such as the Eating Disorder Examination—Questionnaire (Fairburn, 2008) or the Eating Pathology Symptoms Inventory (Forbush et al., 2013) Further, these measures assess symptoms occurring in the past month, which may limit the ability to detect patterns of symptom change throughout treatment. Therefore, development of a measure that considers the context and briefer timeframe when assessing ED symptoms and emotional/social functioning in intensive treatment may support a more accurate assessment of treatment response.

Development of such measures may also increase the likelihood that clinicians in pragmatic ED treatment settings will use results from PM assessments to inform and individualize treatment for their patients. PM measures are most likely to improve care when clinicians review results promptly and use findings to guide clinical practice (de Jong, van Sluis, Nugter, Heiser, & Spinhoven, 2012). Further, prior qualitative research evidence suggests that clinicians are more likely to use PM measures that are easy to use and interpret, pose minimal time burden, and measure constructs they view as essential to patient progress (Bickman et al., 2016; Moltu et al., 2016). To optimize implementation and dissemination, a PM measure must not only be valid for assessment of ED symptoms in intensive treatment but must also be viewed as acceptable and useful for informing treatment among stakeholders.

In summary, the proposed measure sought to address challenges with existing measures of ED symptoms that limit their utility in monitoring patient progress during intensive treatment. Specifically, the measure was expected to add to the existing set of available ED assessment tools in that it would: (1) achieve brief assessment time, (2) assess context- and timeframe-appropriate ED symptoms in a supervised treatment setting, and (3) assess additional non-ED functioning variables (e.g., emotion tolerance), and (4) to maximize uptake in a routine treatment setting, incorporate essential stakeholder input regarding final constructs to assess.

Pragmatic Research and Treatment Setting

This project capitalized upon an existing research-practice partnership with a large, private network of ED treatment facilities in the United States. This program recently built infrastructure for patient outcome evaluation between admission and discharge for its newly implemented, evidence-based treatment approach—the Unified Treatment Model (UTM)—which is a transdiagnostic, emotion-focused, cognitive behavioral treatment for patients with severe EDs and comorbid psychopathology (Thompson-Brenner, Brooks, et al., 2018). After successful implementation of the UTM, the increased focus on patient assessment between admission and discharge also led to increased attention to the need for PM during treatment. In the absence of an existing PM measure with demonstrable utility in intensive ED treatment settings, the research-practice collaborative sought to develop one.

Importantly, measure development sought to align with the therapeutic environment in which the tool would eventually be used to assess patient progress. The rationale for and structure of residential treatment in this setting is described in detail elsewhere (Thompson-Brenner, Boswell, Espel-Huynh, Brooks, & Lowe, 2018; Thompson-Brenner, Brooks, et al., 2018), but a key premise is that emotion intolerance drives and maintains eating pathology and comorbid symptoms. Briefly, patients attended a highly structured schedule of group and individual therapeutic activities, including group and individual psychotherapy, nutrition, psychiatry, creative arts therapy, and occupational therapy. Across disciplines, interventions focused on enhancing motivation for recovery, increasing mindful awareness of and tolerance of emotional experience, and approaching previously avoided emotion-evoking stimuli via exposure (e.g., mirror or food exposures). Thus, inclusion of constructs related to emotional functioning was important to measure development.

Objectives

The objective of this study was to develop a PM measure for use in intensive ED treatment that aligned with stakeholder needs and also achieved adequate psychometric properties. The measure was developed in partnership with the target treatment program in which it would be used. This allowed for integration of stakeholder input into measure development and, consequently, increased potential for sustained use by the program after measure development was complete. This study aimed to establish a PM measure with the following characteristics: (1) transdiagnostic suitability for patients encountered in residential treatment for EDs; (2) sound psychometrics, including good internal consistency and convergent validity with established ED treatment outcome measures; and (3) sensitivity to change during treatment.

The 37-item measure that was developed jointly by researchers and clinical program leaders was hypothesized to have a seven-factor structure. Hypothesized factors addressed domains considered most relevant to ED treatment response under the UTM, including: Weight and Shape Concern, Motivation, Emotion Intolerance/Avoidance, ED Urges, ED Behaviors, Relational Connection, and use of Adaptive Coping skills. We hypothesized that the ED-related scales (Weight and Shape Concern, ED Urges, and ED Behaviors) would be positively associated with a validated measure of Global ED symptom severity (Fairburn, 2008), and that the Emotion Intolerance/Avoidance and Adaptive Coping scales would be positively and negatively associated with a validated measure of experiential avoidance (Gámez et al., 2014), respectively. The final measure was also anticipated to demonstrate adequate validity, internal consistency, and sensitivity to change.

Methods

Participants

Participants were adolescent and adult female patients (N = 531) presenting for routine care at one of the treatment network’s two residential facilities. Data were collected from participants admitted from February through December 2016. Of 618 patients admitted during this timeframe, 605 (97.9%) consented to have their de-identified data included in research. Of those, 37 (6.1%) cases were repeat admissions during the data collection period. Patients with a repeat admission had only their first-admission data included (561 unique cases). Thirty participants did not complete the admission assessment and therefore were not included in analyses. The final sample size was N = 531. An additional 88 participants (16.6% of 531) did not complete the discharge assessment and were excluded from analyses of sensitivity to change from admission to discharge.

Ages ranged 13–67 years (M = 25.28; SD = 11.08). Participants primarily identified as White (89.6%), with the remainder identifying as Multiracial (3.8%), Asian/Pacific Islander (2.3%), African American (1.3%), Native American (0.002%), and Other (2.6%).

Procedures

Item generation.

Items were generated by a multidisciplinary team of doctoral-level content experts in ED treatment and PM methodology, treatment program research staff, and clinical leadership. Domains of interest for assessment were generated first and included those theorized to be most relevant to treatment outcome in this setting. Items for each domain (Weight and Shape Concern, Motivation, Emotion Intolerance/Avoidance, ED Urges, ED Behaviors, Relational Connection, and use of Adaptive Coping skills) were then generated by the collaborative team. Item phrasing was refined after receiving input from clinical intervention staff and patients at both sites. See Table 1 for a complete, final item list. All items were rated based on the extent to which they applied to a patient during the past week, on a Likert-style scale from 1 (“Never”) to 5 (“Always”).

Table 1.

Proposed Items for the Progress Monitoring Tool for Eating Disorders, N = 531

Item Content Descriptor	Subscale	M (SD)	Skewness	Kurtosis
1. My mood was influenced by my body weight, shape or size.	WS	3.92 (1.18)	−1.05	0.27
2. I felt that I have been making progress in my treatment for my eating disorder.	M	2.82 (1.14)	0.07	−0.69
3. I didn’t want anyone to see what my body looks like.	WS	3.76 (1.32)	−0.85	−0.42
4. When I got emotionally upset, it scared me.	EA	3.13 (1.24)	−0.16	−0.92
5. I felt that my sense of self-worth was strongly influenced by my body shape or weight.	WS	3.93 (1.28)	−1.01	−0.15
6. I felt like I didn’t want to give up my eating disorder.	M	3.35 (1.24)	−0.43	−0.70
7. I tried to avoid feeling sad or anxious.	EA	3.55 (1.08)	−0.42	−0.42
8. My eating disorder made me feel like a better person.	M	2.70 (1.40)	0.19	−1.30
9. I tried to suppress feelings I don’t like by trying not to think about them.	EA	3.53 (1.14)	−0.51	−0.48
10. I couldn’t stand being in a bad mood.	EA	3.46 (1.14)	−0.34	−0.58
11. I felt willing to challenge myself to try new foods and new experiences.	M	2.92 (1.17)	0.05	−0.83
12. I felt that I had to avoid doing things that would make me feel fat.	WS	3.79 (1.29)	−0.92	−0.21
13. I believed that there is nothing worse than feeling emotional pain.	EA	3.30 (1.21)	−0.26	−0.78
14. I had to keep checking my body to make sure I hadn’t gained weight.	WS	3.56 (1.49)	−0.56	−1.15
15. I was willing to do anything to avoid negative feelings.	EA	3.21 (1.13)	−0.21	−0.65
16. My urges to engage in eating disorder behaviors have been so strong that I would have acted on them if I had the chance.	EDU	3.57 (1.34)	−0.62	−0.80
17. I have chosen to eat new foods that I used to avoid.	EDB*	2.94 (1.25)	0.00	−0.95
18. I felt understood by my peers during group.	RC	3.31 (1.11)	−0.39	−0.42
19. I was able to identify different parts of my emotions, and how they unfold over time.	AC	2.99 (1.13)	−0.03	−0.65
20. I had a strong urge to restrict.	EDU	3.66 (1.42)	−0.69	−0.90
21. I have pushed myself to follow my meal plan.	EDB*	3.80 (1.20)	−0.90	−0.02
22. I felt respected by staff in this program.	RC	4.08 (0.96)	−1.00	−0.68
23. I challenged myself to do “exposures”—seeking out or staying in situations that provoke strong feelings I would normally avoid.	AC	2.95 (1.25)	0.03	−0.92
24. After I ate my meals and snacks, I had a strong urge to get rid of the food.	EDU	3.00 (1.53)	−0.02	−1.43
25. When I felt an urge to engage in eating disorder behaviors, I stopped myself before acting.	EDB*	3.73 (1.19)	−0.57	−0.63
26. I felt a sense of belonging in the patient community.	RC	3.21 (1.20)	−0.13	−0.80
27. I tried to tolerate uncomfortable physical sensations.	AC	3.53 (1.03)	−0.60	0.08
28. I had strong urges to exercise in order to burn calories or influence my shape/weight.	EDU	3.04 (1.49)	−0.12	−1.40
29. I did small things to control my weight, like refusing certain foods or exercising when staff were not around.	EDB	1.97 (1.26)	1.01	−0.21
30. I’ve been open and honest when talking to therapists.	RC	4.44 (0.90)	−1.93	3.77
31. I have been working to be flexible in my thinking.	AC	3.81 (1.07)	−0.64	−0.26
32. I had a strong urge to binge.	EDU	1.92 (1.33)	1.21	0.10
33. I engaged in pro-eating disorder talk with others.	EDB	1.79 (1.13)	1.24	0.50
34. I felt good about the relationships I’ve been developing in the program.	RC	3.40 (1.08)	−0.34	−0.39
35. When I experienced difficult emotions, I was able to stay focused on the present moment.	AC	2.81 (1.01)	0.18	−0.34
36. I purposefully checked my body shape or size.	EDB	3.47 (1.42)	−0.50	−1.11
37. I felt accepted for who I am.	RC	2.92 (1.21)	0.05	−0.99

Note.

^*refers to item intended to load negatively onto its hypothesized factor.

WC=Weight and Shape Concern; M=Motivation; EA=Emotion Intolerance/Avoidance; EDU=Eating Disorder Urges, EDB=Eating Disorder Behaviors, RC=Relational Connection; AC=Adaptive Coping. The full range of the Likert-type scale (1–5) was used by participants for all items, thus item range is not listed individually by item in the table above.

Data collection.

Patients were asked to complete a routine battery of assessment measures as part of the admission assessment process at the facility. As part of routine clinical practice, this assessment battery was repeated approximately 24–72 hours prior to discharge. Participants who failed to complete the discharge assessment before leaving the facility were asked to complete the survey remotely via a secure web link sent by research staff at the treatment facility.

Data were drawn from a de-identified dataset provided by the facility for research purposes. All patients were required to complete these assessments as part of treatment. At admission, patients were asked through an informed consent process whether they would allow their de-identified responses to be used for research. Only those who consented to have their de-identified data included in data analyses for research purposes were included in the present analyses. Data were cleaned and processed by the treatment centers’ research staff and provided to the investigative team in a de-identified format. All research activities were approved by the Institutional Review Boards of Drexel University and the treatment center’s internal research oversight committee.

Measures

Patient characteristics.

Demographic characteristics, including age and race/ethnicity, were collected via self-report during the intake assessment. Clinical variables, including charted primary ED diagnosis (assigned by a patient’s treating psychiatrist), comorbid diagnoses, and length of stay, were obtained from patients’ medical charts.

In addition to the 37-item PMED, the following measures were administered as part of the routine assessment battery at admission and discharge:

ED symptoms.

The Eating Disorder Examination—Questionnaire Global scale (EDE-Q; Fairburn, 2008) was used to assess ED symptom severity during the past 28 days. The EDE-Q is a widely used, 28-item self-report measure that includes subscales for dietary restraint, eating concern, shape concern, and weight concern. Global scores are calculated as an average of subscale scores, with possible total scores ranging from 0 to 6. The EDE-Q has been shown to have good reliability (Berg, Peterson, Frazier, & Crow, 2012) and strong convergent validity with clinical interview assessment (Black & Wilson, 1996). Reliability for subscale scores in this sample were good to excellent (αs 0.79 – 0.94). Although alternative structures have been proposed since initial scale development (Allen, Byrne, Lampard, Watson, & Fursland, 2011; Grilo, Reas, Hopwood, & Crosby, 2015), the four-factor structure was used for the present study based on current recommendations for data collection and reporting in intensive treatment facilities in the United States (Attia, Marcus, Walsh, & Guarda, 2017).

Depressive symptoms.

The Center for Epidemiologic Studies—Depression scale (CESD; Radloff, 1977) is a 20-item self-report measure of depressive symptoms during the past week. Possible scores range from 0 to 60 (sample α = .92). The measure has demonstrated strong psychometric properties among adults (Radloff, 1977) and adequate reliability and validity in adolescents (Radloff, 1991).

Mindfulness.

The Southampton Mindfulness Scale (SMQ; Chadwick et al., 2008) assesses mindfulness, defined as attending to emotion in a present-focused and nonjudgmental manner (sixteen items, scores ranging from 0 to 96). The measure exhibits adequate convergent and divergent validity (Chadwick et al., 2008) and has good internal consistency within the present sample (α=.89). Mindfulness is a foundational skill taught in the UTM employed by the facility at which this study was carried out. It is considered a mechanism through which ED symptom improvement occurs under the UTM (Thompson-Brenner, Boswell, et al., 2018). For the SMQ, higher scores indicate greater mindful awareness and hence lesser psychological impairment. For all other measures, greater scores indicate greater psychopathology.

Experiential avoidance.

Experiential avoidance was measured using the Brief Experiential Avoidance Questionnaire (Gámez et al., 2014), a 15-item measure assessing behavioral and cognitive avoidance of negative affect, or repression or denial of one’s emotions. Total scores range from 15–90 (sample α = .86). The measure has excellent convergent and content validity and also demonstrates divergence from other measures of emotional functioning (Gámez et al., 2014). Reduction of experiential avoidance is a key behavioral target of the UTM treatment (Thompson-Brenner, Boswell, et al., 2018).

Data Analytic Strategy

Sample size considerations.

General guidelines for factor analysis recommend a minimum ratio of participants-to-variables (questionnaire/measure items) of 10:1, with diminishing yet incrementally improved accuracy at 20 participants per variable (Costello & Osborne, 2005). Others have also recommended at least a 2.85:1 variable-to-factor ratio for scales with moderate-to-high communalities among items (MacCallum, Widaman, Zhang, & Hong, 1999). The PMED version under consideration included 37 items that were hypothesized to represent seven underlying factors or symptom domains. A minimum sample size of 500 would therefore exceed the recommended 10:1 participant-variable ratio (13.5:1) and would yield a satisfactory 5.29:1 variable-to-factor ratio.

Data preparation.

Data were analyzed in R v. 3.5.0 (R Core Team, 2017). Prior to inferential analyses, data were screened to determine whether assumptions for factor analysis were met. Sampling adequacy was tested with the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy. KMO statistics range from 0 to 1, with higher scores indicating presence of a compact data distribution and increased likelihood of obtaining distinct and reliable factors (Kaiser & Rice, 1974). Presence of non-singularity or absence of extreme multicollinearity was confirmed with Bartlett’s test of sphericity. Assumptions of univariate and multivariate normality were tested using the Henze-Zirkler Test of Multivariate Normal Distribution (Henze & Zirkler, 1990) and Mardia’s test of Multivariate Skewness and Kurtosis (Mardia, 1970). Item response frequency distributions were also visually examined for each PMED item prior to analysis.

Item selection.

Factor analysis was conducted using the psych package (Revelle, 2017). Due to multivariate non-normality of the data distributions (see Results section), a principal axis factor extraction method was used (Costello & Osborne, 2005). Number of factors retained was determined via parallel analysis (Horn, 1965). Oblimin factor rotation was used to account for expected intercorrelations among factors. Poorly fitting items were removed based on low communality (≤ 0.30), low factor loadings (i.e., no factor loading ≥ 0.40), high cross-loadings (i.e., two factor loadings ≥ 0.40 or a difference in loadings ≤ 0.20). After dropping items, the factor analysis was repeated in an iterative manner (with parallel analysis repeated) until an adequate factor structure was obtained and no items met the drop criteria.

Validity testing.

To increase feasibility and reduce burden on patients and research staff, only measures that were previously established as components of the program’s existing routine outcome assessment battery were used to ascertain convergent and discriminant validity. Bivariate correlations were examined between factor/subscale scores and validated measures of interest, included the EDE-Q global score, CESD, BEAQ, and SMQ. At the time of data collection, no measures of motivation or relational functioning were being administered as part of the routine battery of assessments at the treatment facility. Face validity and internal consistency values were used to evaluate these constructs during analysis.

Overall sensitivity to change.

To evaluate the measure’s overall ability to assess change in symptoms during ED treatment, a two-tailed, dependent-samples t-test compared PMED total scores at admission and discharge. To determine whether change on the PMED reflected change in the target UTM treatment outcomes of interest, bivariate correlations were examined between raw change in the PMED total score and change in other outcome measures of interest (EDE-Q global, CESD, SMQ, and BEAQ). Probability values for significance tests of correlation values were corrected using the Holm-Bonferroni adjustment method (Holm, 1979).

Results

Clinical Sample Characteristics

Primary ED diagnoses were as follows: bulimia nervosa (27.7%), anorexia nervosa—restricting type (24.9%), anorexia nervosa—binge/purge type (13.3%), binge eating disorder (5.6%), avoidant-restrictive food intake disorder (1.7%), and other specified feeding or eating disorder (26.6%). The majority of patients (72.5%) received at least one co-occurring psychiatric disorder diagnosis, most commonly major depressive disorder (occurring in 67.6% of patients). The mode number of comorbid diagnoses was two. The average length of stay was 31.5 days (SD = 14.1; range 2–99).

Item Selection

Tests for data adequacy.

The number of participants included in the factor analysis (531) exceeded the a priori sample size target (500). The Kaiser-Meyer-Olkin statistic was 0.91, which is in the excellent range and indicated that the intercorrelations among variables were relatively compact and likely to lend well to factor analysis. Bartlett’s Test of Sphericity yielded a significant result (χ²(666) = 9301.68, p < .0001), providing evidence that the item correlation matrix was not an identity matrix and factor analysis was appropriate. Results from Mardia’s Tests of Skewness (statistic = 14095.90, p < .0001) and Kurtosis (statistic = 32.96, p < .0001) yielded significant results, as did Henze-Zirkler’s Test of Multivariate Normality (statistic = 1.01, p < .0001). Thus, the data were determined to be multivariate non-normal, and principal axis factor extraction was selected as an extraction method robust to non-normality (Costello & Osborne, 2005; Fabrigar, Wegener, MacCallum, & Strahan, 1999).

Factor analysis.

Adequate factor structure was achieved after four sequential factor analyses. In the first step, results from parallel analysis suggested that six factors were optimal. This was one less factor than the seven total that were hypothesized. For this initial step, two factor analyses were run with six and seven factors extracted, respectively. The seven-factor model included a greater number of items with low communalities, lower overall loading values across items, and a greater number of high cross-loadings. In line with the parallel analysis result, the six-factor model was selected as a better fit. In this six-factor model, three items (22, 32, and 33 in Table 1) had low communality values (0.13 – 0.29) and were dropped.

The remaining 34 items were again submitted to parallel analysis, and six-factor extraction was again determined to be optimal per parallel analysis. Five items were dropped for failure to load strongly onto any single factor (all < .40; items 8, 16, 19, 23, and 35), and two items were dropped for high cross-loadings on one or more factors (item 30: cross-loading difference of < .20 on two factors; item 36: two loadings > .40). In the third analysis, parallel analysis recommended extraction of five factors. One additional item was dropped for high cross-loading (item 14, two loadings > .40). All items had at least one loading ≥ 0.40. Communalities were all above 0.30.

The final, 26-item model yielded a five-factor solution and explained 50% of total variance. The five factors appeared to capture two primary domains of eating pathology (Weight and Shape Concern, ED Behaviors and urges), and additional dimensions of Emotion intolerance and Avoidance, Relational Connection, and use of Adaptive Coping strategies. Table 2 summarizes loadings for all items included in the final model. Factor correlations ranged from −.27 to .68, with the highest correlation between the Weight and Shape Concern and ED Behaviors/Urges subscales (see Table 3). A factor analysis was repeated on the final 26 items using data from discharge. A similar pattern of results emerged, though a detailed report of these findings is beyond the scope of this manuscript. Results are available upon request from the corresponding author.

Table 2.

Complete Loadings for the Progress Monitoring Tool for Eating Disorders, N = 531

Item Content Descriptor	M (SD)	Weight/ Shape Concern	ED Behaviors	Emotion Avoidance	Adaptive Coping	Relational
Weight/shape affect mood	3.92 (1.18)	.82	.08	.03	.04	−.03
Exposure discomfort	3.76 (1.32)	.69	.03	.05	−.02	−.03
Weight/shape overvaluation	3.93 (1.28)	.91	−.04	.02	−.01	.02
Fat avoidance	3.79 (1.29)	.61	.24	.06	−.01	.04
ED ego-syntonic	2.70 (1.40)	.12	.44	.10	−.09	.05
Restriction urges	3.66 (1.42)	.21	.60	.06	−.11	−.02
Compensatory behavior urges	3.00 (1.53)	.21	.47	.12	−.04	.00
Exercise urges	3.04 (1.49)	.18	.62	−.02	.13	.03
Subtle weight control	1.97 (1.26)	−.03	.53	−.01	−.29	.05
Fear of emotion	3.13 (1.24)	.14	−.06	.54	−.07	−.02
Anxiety & sadness avoidance	3.55 (1.08)	−.08	.07	.68	.05	−.03
Emotion suppression	3.53 (1.14)	−.05	.20	.63	.06	−.05
Negative affect intolerance	3.46 (1.14)	.10	−.11	.64	.04	.05
Emotional pain sensitivity	3.30 (1.21)	.05	−.15	.61	−.10	.08
Negative affect avoidance	3.21 (1.13)	−.01	.00	.75	.00	−.01
Treatment progress	2.82 (1.14)	−.15	.00	−.08	.54	.03
New foods & experiences	2.92 (1.17)	−.03	−.20	−.12	.56	.06
Reduce food avoidance	2.94 (1.25)	.01	−.12	−.13	.58	.07
Follow meal plan	3.80 (1.20)	.07	−.02	.02	.77	.03
Prevent ED behavior	3.73 (1.19)	.00	.07	.08	.65	−.09
Sensation tolerance	3.53 (1.03)	−.08	.15	.09	.57	.04
Flexible thinking	3.81 (1.07)	.00	−.07	.09	.49	.22
Accepted by peers	3.31 (1.11)	−.05	.15	−.01	.10	.60
Community belonging	3.21 (1.20)	.03	−.03	.01	−.04	.87
Positive relationships	3.40 (1.08)	.03	.01	.02	.02	.78
Accepted overall	2.92 (1.26)	−.30	.00	−.10	.12	.49
Internal consistency (α)		0.89	0.80	0.81	0.82	0.79

Table 3.

Factor Correlations for Final Model

Factor	Weight/ Shape Concern	ED Behaviors	Emotion Avoidance		Adaptive Coping
Weight/Shape Concern	--
ED Behaviors	.68**	--
Emotion Avoidance	.53**	.37**	--
Adaptive Coping	−.26**	−.27**	−.12*	--
Relational Connection	−.13*	.00	−.03	.47**

Note.

^*p < .01;

^**p < .001

Validity Testing

Convergent and discriminant validity.

PMED subscale scores were computed for each participant by computing the simple mean of responses to all items loading onto a given factor. Higher scores indicated greater symptom severity for the Weight/Shape Concern, ED Behaviors, and Emotion Avoidance subscales. In contrast, higher scores on the Relational Connection and Adaptive Coping scales were suggestive of a stronger sense of interpersonal connection and use of UTM-consistent emotional coping strategies, respectively. Thus, scores for these two subscales would be expected to be negatively correlated with the other three. Results from correlation analysis supported this hypothesis, in that the Relational and Coping scores were negatively associated with the other factors (rs ranged from −.34 to −.12; see Table 4).

Table 4.

Bivariate, Zero-Order Correlations among PMED Subscales, Total Scale, and Primary UTM Treatment Constructs (n = 443)

Variable	M (SD)	PMED Total (AD)	PMED Total (DC)	PMED Δ	W/S	EB	EA	AC	RC
PMED Total (AD)	79.93 (15.60)	--
PMED Total (DC)	61.02 (17.00)	.56**	--
PMED Δ	−18.91 (15.38)	−.40**	.54**	--
Weight/Shape Concerna	3.87 (1.08)	.75**	.39**	−.33**	--
ED-Behaviora	2.91 (1.02)	.75**	.50**	−.21**	.65**	--
Emotion Avoidancea	3.39 (0.81)	.61**	.24**	−.35**	.40**	.32	--
Adaptive Copinga	3.37 (0.78)	−.68**	−.41**	.24**	−.27**	−.34	−.15*	--
Relational Connectiona	3.23 (0.89)	−.51**	−.30**	.19**	−.19**	−.12	−.14	.44**	--
EDE-Q Global (AD)	3.98 (1.53)	.69**	.39**	−.28**	.86**	.68**	.38**	−.24**	−.14
EDE-Q Δ	−1.47 (1.25)	−.19**	.35**	.58**	−.39**	−.19**	−.14	−.05	−.04
CESD (AD)	37.69 (12.35)	.66**	.34**	−.29**	.56**	.44**	.57**	−.31**	−.31**
CESD Δ	−12.87 (12.57)	−.10	.47**	.61**	−.18*	.02	−.25**	−.06	.00
BEAQ (AD)	58.74 (12.90)	.51**	.23**	−.26**	.39**	.25**	.62**	−.24**	−.20**
BEAQ Δ	−10.31 (13.51)	−.04	.48**	.56**	−.09	.06	−.23**	−.09	−.04
SMQ (AD)	30.96 (16.59)	−.60**	−.34**	.23**	−.52**	−.40**	−.56**	.29**	.21**
SMQ Δ	+12.27 (17.55)	.01	−.42**	−.47**	.07	−.03	.17*	.11	.07

Notes.

^**p < .001;

^*p < .05; reflects Holm-Bonferroni correction for multiple comparisons.

AD=Admission; DC=Discharge; EDE-Q=Eating Disorder Examination Questionnaire; CESD=Center for Epidemiological Studies Depression scale; BEAQ=Brief Experiential Avoidance Questionnaire; SMQ=Southampton Mindfulness Questionnaire. All variables represent baseline measures obtained at admission except change scores (raw change from admission to discharge), which are represented by a Δ sign.

^aSubscale scores were obtained by averaging all item ratings for a given subscale, to allow for comparability across factors.

Results from bivariate Pearson correlations are reported in full in Table 4. Key results for convergent and divergent validity with target UTM and treatment outcome variables are summarized here. Weight/Shape Concern subscale scores were strongly and positively correlated with EDE-Q global scores, as were those for the ED Behaviors/Urges subscale. Numerically, scores on both of these factors were less strongly associated with CESD scores than with EDE-Q scores, and associations between that of the EDE-Q and CESD measures were similar in magnitude. Thus, results indicated that the two ED-specific factors demonstrated unique validity as measures of ED symptomatology constructs.

Emotion Avoidance scores were most strongly associated with BEAQ total scores, providing evidence of convergent validity with an existing validated measure of emotion avoidance. Interestingly, the association with the CESD was similar to that for the BEAQ but numerically lower for the EDE-Q. Adaptive Coping scores were modestly negatively associated with UTM target constructs of experiential avoidance and mindfulness. However, given that patients are expected to build these skills during the course of treatment (and may have limited insight into them at admission), correlations among Adaptive Coping and relevant convergent constructs of mindfulness and emotion avoidance were also examined at discharge (results available upon request of first author). Associations were stronger in magnitude upon treatment completion, thus Adaptive Coping appeared to show convergence with relevant constructs from the patient assessment battery.

As stated previously, there was no existing measure in the treatment program’s admission and discharge assessment battery that could be used to evaluate convergent or divergent validity for the Relational Connection factor. However, all items loading onto this factor had good face validity, in that they referred to patients’ feelings of acceptance, belonging, and understanding from other members of the treatment center community. Further, items were reviewed and approved of by the treatment program’s Clinical Advisory board, which includes two key experts in relational and dynamic processes: a physician-trained expert in Relational-Cultural Theory, and a doctoral-level clinical psychologist trained in psychodynamic, relational, and cognitive-behavioral approaches to treating EDs.

Sensitivity to change.

Prior to computing total PMED scores at admission and discharge, all Relational Connection and Adaptive Coping items were reverse-scored such that higher scores indicated more negative symptomatology (total PMED score ranged 26–130). Total scores were the sum of all item ratings after recalculating reverse-scored items. Results from a dependent samples t-test indicated significant reduction in scores from admission (M = 79.59; SD = 15.88) to discharge (M = 61.02; SD = 17.00) with a large effect size (t(442) = 25.88, p < .0001, d = 0.78, 95% CI_diff [17.47, 20.34]). As depicted in Table 4, change in PMED scores was positively associated with change in ED symptoms, depression, and experiential avoidance, and negatively associated with change in mindfulness, providing further support for the measure’s ability to detect change in constructs directly relevant to outcome among patients with EDs treated in residential care.

Discussion

This study aimed to evaluate the factor structure and psychometric properties of the PMED, a tool developed specifically for use with patients receiving emotion-focused, cognitive behavioral intervention in a residential ED treatment setting. Results from exploratory factor analysis supported a five-factor structure for the final, 26-item measure, which included the domains of Weight and Shape Concern, ED Behaviors and Urges, Emotion Avoidance, Adaptive Coping, and Relational Connection. One initially proposed factor (Motivation) was removed, and the Behaviors and Urges scales were combined. The PMED represents the first measure we are aware of designed to reliably and validly assess ED psychopathology, in addition to related emotional functioning and treatment engagement constructs, in the context of intensive ED treatment.

Overall, the final measure demonstrated satisfactory psychometric properties. Relative to the initial proposed measure and factorial structure, the final model retained a majority of the initial seven proposed constructs and accounted for 50% of total variance. Although many advocate for a minimum total variance explained of 60% (Hair, Anderson, Tatham, & Black, 2014), the 50% value is consistent with the mean of 52.03% variance observed in the social and behavioral sciences (Henson & Roberts, 2006). Further, the measure’s final factors demonstrated convergent validity with existing and validated measures of ED symptoms, depression, experiential avoidance, and mindfulness. Although no measures of relational connection were available to test the validity of the Relational Connection factor, face validity supported its inclusion. Inclusion of this factor is also supported by existing work in general psychotherapy PM development, given that most general PM measures include some measure of treatment alliance or interpersonal connection with others (e.g., Partners for Change Outcome Management System, Outcome Questionnaire, and Treatment Outcome Package; Boswell et al., 2015; Duncan, 2012; Lambert et al., 1996). Further, this construct was identified as a priority by treatment program stakeholders and therefore warrants inclusion for pragmatic measure use.

Notably, items that were originally generated for the hypothesized ED Behaviors symptom domain were either eliminated or loaded onto a single factor in combination with ED Urges. The final ED Behaviors and Urges subscale was predominantly represented by urge-related items versus those of overt behavior. One possible explanation for this result is that the target behaviors were less relevant in the structured residential treatment setting, where patients’ ready access to overt ED behaviors is more limited due to staff supervision and structural barriers. Alternatively, it is possible that the behaviors assessed in the dropped items (e.g., body checking) occur with more variable frequency in this patient population and therefore are not the most appropriate ED behaviors to assess. It is also striking that the binge urge item did not fit well and was dropped, but that the urge items for restriction and compensatory behaviors (i.e., “getting rid of food” and exercising) were retained. One possible explanation for this result is that patients initiate a regular pattern of eating (including foods with a range of caloric density) immediately upon admission to residential treatment. Acute nutrient depletion via food restriction is therefore eliminated relatively early on in treatment, thus eliminating one key driving factor for binge eating (Hetherington, Stoner, Andersen, & Rolls, 2000; Holmes, Fuller‐Tyszkiewicz, Skouteris, & Broadbent, 2014; Zunker et al., 2011). Thus, while patients may continue to experience urges to compensate for the food eaten, binge urges may be attenuated by a structured and regular eating pattern. Further research is required to test this conjecture. Regardless, the retained items for the ED Urges and Behaviors factor demonstrated strong psychometric properties and contributed to an overall valid subscale of the PMED.

Overall, results from this analysis supported adequate factor structure of the PMED in its refined and finalized 26-item format. Strengths include good sensitivity to change during the course of treatment, convergent validity with existing validated outcome measures that were already being used within the program, and excellent internal consistency within factors. Of particular pragmatic relevance, this measure also had approval and buy-in from key stakeholders in the setting in which it will be used for PM.

Limitations of this work must also be considered. In its current form, the measure is limited by some factor loadings and a total variance explained that are lower than would be ideal. In addition, logistic constraints limited the ability to test for convergent validity of the Relational Connection subscale with other validated measures. Given the pragmatic emphasis of the current project and the desire to develop a practically relevant and clinically useful tool for clinicians, this scale was included to promote stakeholder buy-in and subsequent use of the measure for PM in clinical practice. In the future, confirmatory factor analysis with newly collected data and additional convergence testing would strengthen confidence in the PMED’s psychometric properties.

In sum, the final measure represents an efficient and parsimonious means of assessing transdiagnostic ED symptoms among patients receiving residential ED treatment. Future research should explore sensitivity of the PMED to weekly change throughout treatment, given that PM is ideally conducted at weekly frequency. The brief length of this measure is ideal for frequent administration. Further investigation should also determine the extent to which change in the five constructs assessed by the PMED predict or mediate change in ED and comorbid symptom outcomes in the context of emotion-focused, cognitive behavioral treatment for EDs, both in the centers in which this measure was developed and elsewhere (e.g., lower levels of intensive treatment such as day hospital or other treatment programs utilizing similar intervention approaches). Additionally, future longitudinal research and randomized trials are needed to determine whether routine monitoring of patients’ progress with the PMED can be feasibly integrated into clinical practice in residential treatment, and whether such monitoring can facilitate improved outcomes, particularly for those patients at risk for poor outcome in intensive ED treatment. The full measure is available in the Supplemental Materials associated with this manuscript.

Highlights.

• This study sought to develop and validate the Progress Monitoring Tool for Eating Disorders, a multidimensional measure for progress monitoring in the context of intensive ED treatment.

• Exploratory factor analysis yielded a five-factor measure assessing Weight/Shape Concern, ED Behaviors/Urges, Emotion Avoidance, Adaptive Coping, and Relational Connection.

• The measure detected change in symptoms during treatment with internal consistency and validity.