Psychometric Properties of the Parenting Stress Index-Short Form (PSI-SF) in a High-Risk Sample of Mothers and their Infants

Abstract

The goal of the present study was to evaluate the psychometric properties of the English and Spanish versions of the Parenting Stress Index-Short Form (PSI-SF) with mothers of 12- to 15-month-old infants with elevated levels of behavior problems and from predominately Hispanic, low-income backgrounds. Mothers of 58 infants were assessed as part of a larger study examining a brief home-based intervention for infants with elevated behavior problems. Internal consistency was good for all three subscales (i.e., Parental Distress, Parent-Child Dysfunctional Interaction, and Difficult Child) and the Total Stress scale. Convergent validity of subscales was supported by correlations with measures of theoretically related constructs, including maternal depressive symptoms, maternal parenting practices, and infant behavior. Furthermore, examination of the optimal clinical cutoff by examining sensitivity and specificity suggested that for this high-risk sample lower percentile scores (73^rd – 77^th), relative to the published 85^th percentile cutoff, were sufficient for identifying mothers with clinically elevated depressive symptoms and infants with clinically elevated behavioral and emotional difficulties. The current results provide psychometric support for the PSI-SF as an effective and appropriate measure for use with high-risk families that have been underrepresented in previous research, including mothers of very young children with behavior problems, Hispanic and Spanish-speaking populations, and low-income families.

Stress has been defined as coping with challenges (Lazarus, 2000) and shown to play a critical role in parenting, especially among clinical populations (Crnic, Gaze, & Hoffman, 2005). For example, research has demonstrated a transactional relation between parenting stress and child behavior problems from 3 to 9 years (Neece, Green, & Baker, 2012). The transition from infancy to toddlerhood can be especially stressful as parents experience new and challenging child behaviors, and higher levels of parenting stress during infancy has been associated with a difficult infant temperament and higher levels of infant negative affectivity (Chang et al., 2004).

In addition to the negative impact of early parenting stress on infant outcomes, levels of parenting stress are higher among families from racial and ethnic minority backgrounds (Franco, Pottick, & Huang, 2010). Furthermore, higher levels of parenting stress among minority groups are largely due to higher poverty rates, especially for Hispanics who struggle with acculturation (Huston, McLoyd, & Garcia Coll, 1994). Given the higher levels of parenting stress among parents of children from minority and disadvantaged backgrounds and its negative impact in infancy, it is critical to have a reliable and valid measure of parenting stress with this population.

One of the most common measures of parenting stress is the Parenting Stress Index (PSI), a 120-item parent-report questionnaire. To reduce burden of the full-length PSI, a brief measure known as the PSI-Short Form (PSI-SF), which consists of 36 items from the full PSI, was developed. The newest full and short versions, the PSI-4 and PSI-SF-4, are highly correlated with the original PSI and PSI-SF (Abidin, 2012). The PSI-SF was developed based on several exploratory factor analyses of the full PSI (Solis & Abidin, 1991) and is comprised of three subscales, each consisting of the items that best loaded together, which include the Parental Distress (PD), Parent-Child Dysfunctional Interaction (PCDI) and Difficult Child (DC) subscales, as well as a Total Stress scale. All three subscales and the Total Stress scale are highly correlated between the PSI-SF and PSI-SF-4 ranging from .97–.99 (Abidin, 2012). According to the manual, scores at or above the 85^th percentile on the Total Stress scale are considered to be borderline clinically significant based on the norms of the full PSI (Abidin, 2012). However, the clinical utility of this cutoff has not been examined in the literature. Although the PSI-SF is commonly used in a variety of settings with infants, such as hospital discharge (Thomas, Renaud, & Depaul, 2004), only 5 studies have examined the psychometric properties of the PSI-SF with mothers of infants and toddlers. Furthermore, no studies to our knowledge have examined the psychometric properties of the PSI-SF in a high-risk sample of mothers of infants and toddlers with elevated levels of behavior problems.

In addition to the original development of the PSI-SF, only three of the five studies with infants and toddlers demonstrated the reliability of the PSI-SF; two used the English version (Haskett, Ahern, Ward, & Allaire, 2006; Whiteside-Mansell et al., 2007) and one used the Spanish version (Díaz-Herrero, López-Pina, Pérez-López, de la Nuez, & Martínez-Fuentes, 2013). High internal consistency was demonstrated for the three subscales and the Total Stress scale for the Spanish version with a sample of Europeans (Díaz-Herrero et al., 2013). However, the Spanish version has not been examined with Hispanic parents, who report higher rates of child behavior problems (Holtrop, McNeil Smith, & Scott, 2014) and parenting stress (Franco, Pottick, & Huang, 2010). Studies on the English version in Head Start samples demonstrated high internal consistency estimates with mostly White Children (Whiteside-Mansell et al., 2007) and high test-retest reliability estimates with mostly Black children (Haskett et al., 2006).

In terms of validity, high correlations have been reported between all PSI-SF subscales and child behavior problems (Haskett et al., 2006). Furthermore, research has documented positive correlations between the PD subscale and parenting behaviors, including negative parenting practices and emotional responsiveness (Haskett et al., 2006; Whiteside-Mansell et al., 2007), as well as maternal depression (Whiteside-Mansell et al., 2007). Research has also supported that mothers who report a higher external locus of control of their child’s behavior, report higher levels of parenting stress (Hassall, Rose, & McDonald, 2005). However, all previous studies examining the validity of the PSI-SF used the English version and included samples of predominantly White or Black families and not Hispanic families.

Despite the demonstrated psychometric properties of the English and Spanish versions of the PSI-SF with White and Black samples, no research study to our knowledge has examined the PSI-SF with a predominantly Hispanic sample. Research with Hispanic samples is particularly relevant in the United States, where the number of Hispanic families continues to rise (Johnson & Lichter, 2008). Given that Hispanic mothers report higher levels of parenting stress (Franco et al., 2010) and early childhood behavior problems (Theule, Wiener, Tannock, & Jenkins, 2010) compared to White mothers, it is important to ensure adequate measurement of parenting stress in infancy and with a predominately Hispanic sample.

Therefore, the purpose of the current study was to examine the reliability and validity of the PSI-SF with a high-risk sample of mothers and their infants. To examine reliability, we measured estimates of both internal consistency and test-retest reliability for the PSI-SF subscales and the Total Stress scale. To examine convergent validity, we measured the association between scores on each PSI-SF subscale with scores on measures of theoretically related constructs, including maternal depressive symptoms, maternal parenting practices, and infant behavior. Finally, we examined the optimal percentile cutoff for the PSI-SF Total Stress Score using the Receiver Operating Curve fitting (ROC) methodology, which examines the sensitivity and specificity of cutoff scores in predicting relevant outcomes. The criterion validity of the identified cutoff score was subsequently assessed by comparing mothers who scored above and below clinical cutoffs on measures of maternal and infant outcomes.

Method

Participants were 58 mothers and their 12- to 15-month-old infant with elevated levels of behavior problems who participated in a larger study examining a brief home-based parenting intervention. Families were recruited at a large pediatric primary care clinic in a predominately Hispanic community. On average, mothers were 29.88-years-old (SD = 5.28), and their infants (53% male) were 13.52-months-old (SD = 1.30). Most mothers reported Hispanic ethnicity (91.4 %) and an income below the poverty line (60.3%). Thirty-five mothers (60.3%) completed the assessment in Spanish (Table 1).

Table 1.

Sample Demographic Characteristics

Characteristic	M (SD)	N (%)
Child age (months)	13.52 (1.30)
Child sex (% male)		31 (53.4)
Child ethnicity (% Hispanic)		55 (94.8)
Child race (% minority)		57 (98.3)
Mother age (years)	29.88 (5.28)
Mother ethnicity (% Hispanic)		53 (91.4)
Mother race (% minority)		55 (94.8)
Mother marital status (% married)		40 (69.0)
Mother education (% graduated high school or less)		40 (69.1)
Poverty Status (% below poverty)		35 (60.3)

In order to screen into the study, infants had to be rated by their mother to be above the clinical cutoff (i.e., ≥ 75th percentile) on the Brief Infant-Toddler Social and Emotional Assessment (BITSEA) Problem scale (Briggs-Gowan, Carter, Irwin, Wachtel, & Cicchetti, 2004), a screening measure of social, emotional, and behavioral functioning. English-speaking mothers had to receive an estimated IQ score ≥ 70 on the two-subtest version of the Wechsler Abbreviated Scale of Intelligence (WASI; Wechsler, 1999), and Spanish-speaking mothers had to receive an average standard score ≥ 4 on the two- subtest version of the Escala de Inteligencia Wechsler Para Adultos (EIWA; Pons et al., 2008). Exclusion criteria included any infants with major sensory impairments (e.g. deafness, blindness) or current child protection services involvement, although no families were excluded based on these criteria.

During recruitment, we approached 315 families at the pediatric clinic and 146 primary caregivers (46.3%) agreed to participate. Primary caregivers were the infant’s mother in all cases. Sixty families screened into the intervention study and were randomized to an intervention (n = 31) or standard care group (n = 29). Two families screened in and were randomized to condition but did not complete the Time 1 (baseline) assessment and thus did not complete the PSI-SF yielding a total of 58 families for the current sample. Families in the intervention group received a brief home-based adaptation of Parent-Child Interaction Therapy (PCIT) for infants at-risk for behavior problems. Details of the intervention and main outcomes for the randomized controlled trial are presented elsewhere (Bagner et al., in press). Families in both groups participated in a Time 2 assessment an average of 71.87 (SD = 16.42) days after the Time 1 assessment.

Measures

Parenting Stress Index-Short Form (PSI-SF; Abidin, 1995)

The PSI-SF is a 36-item self-report questionnaire of parenting stress with three subscales (PD, PCDI, and DC) and a Total Stress scale. Scores above the 85^th percentile on the Total Stress scale are considered borderline clinically significant (Abidin, 2012), but this cutoff has not been empirically tested. As described previously, there is evidence to support reliability and validity, but there is limited support for these psychometric properties in a high-risk mother-infant sample.

Center for Epidemiologic Studies Depression Scale (CES-D; Radloff & Locke, 1986)

The CES-D is a widely used 20-item self-report questionnaire of depressive symptomatology. Internal consistency estimates have been found to be high in a variety of samples (Orme, Reis, & Herz, 1984; Thomas, Jones, Scarinci, Mehan, & Brantley, 2001), including the current sample (α = .90). A cutoff score of 16 is commonly suggested for detecting depression, and has been found to predict Major Depressive Disorder with a sensitivity of .95 and specificity of .70 (Thomas, Jones, Scarinci, Mehan, & Brantley, 2001). The CES-D was used to examine the convergent validity with the PSI-SF PD subscale.

Parental Locus of Control-Short Form (PLOC-SF; Rayfield, Eyberg, Bogg, & Roberts, 1995)

The PLOC-SF is a 25-item self-report questionnaire of parents’ beliefs regarding their ability to influence their child’s behavior. Higher scores indicate parents believe they have less control over their child’s behavior. Internal consistency rates have been acceptable (α = .79; Rayfield et al., 1995), including in the current sample (α = .71). The PLOC-SF was included in the current study to examine convergent validity with the PSI-SF PCDI subscale.

Infant-Toddler Social and Emotional Assessment (ITSEA; Carter & Briggs-Gowan, 2006)

The ITSEA is a parent-report questionnaire of social-emotional and behavioral problems in 12- to 36-month-olds and includes four broad domains (Externalizing, Internalizing, Dysregulation, and Competence). Strong reliability estimates of the ITSEA domains were demonstrated in a nationally-representative sample (Carter & Briggs-Gowan, 2006), and internal consistency estimates in the current sample were good for the Externalizing and Dysregulation domains (α = .80 and .78, respectively) and adequate for the Internalizing domain (α = .69). For these domains, a T score ≥ 65 is considered Of Concern, which is at or above the 90^th percentile (Carter & Briggs-Gowan, 2006). The Externalizing, Internalizing, and Dysregulation domains were included in the current study to examine convergent validity with the PSI-SF DC subscale.

Data Analysis

Prior to analysis, the data were evaluated for multivariate outliers, and no outliers were detected. Values for missing data (17%) were imputed using multiple imputation in SPSS 20. For internal consistency, Cronbach’s alpha coefficients were computed for each of the three PSI-SF subscales and the PSI-SF Total Stress scale at Time 1. Internal consistency estimates were examined with the total sample and separately for mothers who completed the assessment in English and Spanish. Test-retest reliability between Time 1 and 2 for the subscales and Total Stress scale were evaluated using Intraclass Correlation Coefficients (ICC) using a two-way mixed, absolute consistency model (McGraw & Wong, 1996). In order to control for group, linear regressions were also conducted to measure test-retest reliability between Time 1 and 2.

To examine convergent validity, we examined correlations between scores for each of the PSI-SF subscales with other measures of theoretically related constructs at Time 1. Specifically, we examined correlations between the PD subscale and CES-D; PCDI subscale and PLOC-SF; and DC subscale and ITSEA Externalizing, Internalizing, and Dysregulation domains. To identify the optimal percentile cutoff for the PSI-SF Total Stress scale, we conducted ROC analyses in conjunction with Youden’s J index. To demonstrate the utility of the identified cutoffs, criterion validity was assessed.

Results

Internal consistency estimates were adequate for the PSI-SF PD subscale (α = .75, .71, .79) in the full sample, as well as in the English and Spanish samples, respectively. Internal consistency estimates were good for the PSI-SF PCDI subscale (α= .85, .87, .83) and DC subscale (α= .82, .81, .84), and excellent for the Total Stress scale (α= .91, .92, .90) in the full sample, as well as in the English and Spanish samples, respectively. Test-retest reliability estimates were good for the PSI-SF Total Stress scale (ICC = .77, .78, .77) and the PSI-SF PD subscale (α= .82, .80, .84) in the full sample, as well as in the English and Spanish samples, respectively. Test-retest reliability estimates were adequate for the PCDI subscale (α= .61, .66, .58), and DC subscale (α= .66, .60, .70) in the full sample, as well as in the English and Spanish samples, respectively. To control for group, PSI-SF subscales and the Total Stress scale at Time 2 were regressed on the PSI-SF subscales and the Total Stress scale score at Time 1. Mothers’ scores on the PSI-SF PD subscale at Time 1 significantly predicted scores at Time 2, B = .77, t(45) = 7.15, p < .001, and explained a significant proportion of variance in Time 2 scores, R² = .57, F(2, 45) = 29.94, p < .001. Mothers’ scores on the PCDI subscale at Time 1 significantly predicted scores at Time 2, B = .38, t(45) = 3.36, p = .002, and explained a significant proportion of variance in Time 2 scores, R² = .21, F(2, 45) = 5.98, p = .005. Mothers’ scores on the DC subscale at Time 1 significantly predicted scores at Time 2, B = .46, t(45) = 3.75, p < .001, and explained a significant proportion of variance in Time 2 scores, R² = .25, F(2, 45) = 7.30, p = .002. Finally, mother’s scores on the Total Stress scale at Time 1 significantly predicted scores at Time 2, B = .57, t(45) = 5.44, p < .001, and explained a significant proportion of variance in Time 2 scores, R² = .40, F(2, 45) = 14.81, p < .001.

For convergent validity, mothers’ scores on the PSI-SF PD subscale were moderately correlated with mothers’ scores on the CES-D, r(58) = .53, p < .001. Similarly, mothers’ scores on the PCDI subscale were moderately correlated with mothers’ scores on the PLOC-SF, r(58) = .44, p = .001. Finally, mothers’ scores on the DC subscale were moderately correlated with infants’ scores on the ITSEA Externalizing, r(58) = .50, p = .01, Internalizing, r(58) = .38, p = .01, and Dysregulation, r(58) = .44, p = .01 domains.

Sensitivity, specificity, and Youden’s J values were calculated for mother-reported Total Stress percentile scores at Time 1. Classification below or above the clinical cutoff on the CES-D (i.e., raw score ≥ 16) and ITSEA Externalizing, Internalizing, and Dysregulation domains (i.e., > T score ≥ 65) were used as the diagnostic outcome. Table 2 illustrates the cutoff score that optimized both sensitivity and specificity as indexed by the best Youden’s J value, and the percentage of the sample that fell above the cutoff. Overall, the 72.5%ile best identified mothers with elevated levels of depressive symptoms and infant externalizing problems, and the 77.5%ile was identified as the optimal cutoff for infant internalizing and dysregulation problems. Relative to mothers below the PSI-SF cutoff for the CES-D (72.5%ile), mothers scoring at or above the cutoff were more likely to have increased depressive symptoms, t(56) = 4.99, p < .001. Relative to mothers below the PSI-SF cutoff for the ITSEA domain scores, mothers scoring at or above the 72.5%ile were more likely to have infants with increased externalizing, t(56) = 2.25, p < .05 and internalizing, t(56) = 4.24, p < .001, and dysregulation, t(56) = 3.42, p < .001, for mothers scoring at or above the 77.5%ile.

Table 2.

Summary of cutoff thresholds using Total Stress Percentile Scores (n = 58)

Outcome	Optimal Percentile Cutoff	Percentage Above Cutoff	Sensitivity	Specificity	Youden’s J
CES-D	72.5	41.1	.86	.58	.45
ITSEA Externalizing	72.5	41.4	.68	.54	.23
ITSEA Internalizing	77.5	50	.90	.58	.48
ITSEA Dysregulation	77.5	50	.78	.63	.41

Note. ITSEA = Infant Toddler Social Emotional Assessment, CES-D = Center for Epidemiological Studies Depression Scale

Discussion

The purpose of the current study was to examine the psychometric properties of the PSI-SF in mothers of infants between 12- and 15-months-old with elevated behavior problems from predominately Hispanic and low-income families. Consistent with previous studies of the PSI-SF in White and Black samples (Díaz-Herrero et al., 2013; McKelvey et al., 2008), internal consistency was good and test-retest reliability ranged from adequate to good in the English and Spanish versions. Support for convergent validity based on the relation between mothers’ scores on the PD subscale and the CESD suggests parental distress independent of the infant’s behavior is related to maternal depressive symptoms, which is consistent with previous research documenting the association between depression and life stress (Hammen, 2005). Additionally, support for convergent validity based on the relation between mothers’ scores on the PCDI subscale and the PLOC-SF indicates stress related to the parent-child relationship is related to the extent to which the mother perceives she has control over her infant’s behavior. Lastly, support for convergent validity based on the relation between mothers’ scores on the DC subscale and the ITSEA domains suggests stress related to the parent’s view of having a more difficult infant is related to infant behavior problems and dysregulation.

We also examined optimal cutoff scores for the PSI-SF Total Stress scale using ROC analyses in conjunction with Youden’s J to identify parent and child outcomes. For maternal outcomes, the 72.5%ile on the PSI-SF Total Stress scale demonstrated the best sensitivity and specificity for predicting maternal depressive symptoms. For child outcomes, the 72.5%ile and the 77.5%ile demonstrated the best sensitivity and specificity for the externalizing and the internalizing and dysregulation domains, respectively. The lower Youden’s J for the Externalizing domain may be due to the high number of infants in the sample (n = 42; 72%) with clinically elevated levels (T-score ≥ 65) of externalizing behaviors, thereby limiting variability. Furthermore, we demonstrated these cutoffs adequately predicted both maternal and child outcomes. These findings suggest the 85^th percentile cutoff may exclude mothers who are experiencing difficulties and may need services, especially among those from high-risk families. Although the cutoff scores are helpful in referring families for additional services, it is important to acknowledge that cutoff scores can be arbitrary (Blanton & Jaccard, 2006) and thus additional factors should be considered when referring families for services.

The current study has some limitations, and it is important to interpret the current results in light of these limitations. First, the sample size used in the current analyses was relatively small in relation to other studies examining the psychometric properties of the PSI-SF, thereby limiting power. Therefore, the data presented should be interpreted as preliminary but should encourage future research on the psychometric properties of the PSI-SF in larger samples with similar demographic compositions. Additionally, the sample examined was specific to infants 12 to 15 months of age with elevated behavior problems and may limit generalizability to infants from a wider age range, infants without behavior problems, or infants at high risk due to other problems (e.g., medical conditions). Thus, future research should examine the psychometric properties of the PSI-SF in these other infant populations. Another limitation is that data were collected only from mothers in the current study, which did not allow for examination of inter-rater reliability. Lastly, the current timeline for the test-retest interval was in the moderate range and may have been confounded by the fact that half of the group received an intervention. Although we controlled for intervention status in our regression analyses, future studies should examine test-retest reliability of the PSI-SF over different periods of time in a sample that did not receive an intervention in order to ensure stability across the scales.

Despite these limitations, the current study was the first to examine the psychometric properties of the PSI-SF in a sample of mothers of infants with behavior problems from predominately Hispanic and low-income backgrounds. Examining the reliability and validity of the PSI-SF is especially important given the increased levels of parenting stress reported by mothers of young children with behavior problems and from Hispanic backgrounds and the need for instruments that accurately assess their difficulties related to parenting. Overall, these findings provide initial support for the psychometric properties of the English and Spanish versions of the PSI-SF in a high-risk sample. Given the identified lower threshold for maternal depressive symptoms and infant behavior problems, it will be important for clinicians to consider a lower cutoff when examining parenting stress among high-risk parents and for future research to further examine the usefulness of these lower cutoffs in longitudinal studies.