The PSC-17: Subscale Scores, Reliability, and Factor Structure in a New National Sample

Abstract

BACKGROUND:

The Pediatric Symptom Checklist-17 (PSC-17) is a widely used, briefer version of the PSC-35, a parent-completed measure of children’s psychosocial functioning. Despite the extensive use of the PSC-17 over the past 15 years there has not been a large-scale replication of the original derivation study.

OBJECTIVE:

To examine the prevalence of positive screens, reliability, and factor structure of PSC-17 scores in a new national sample and compare them with the derivation sample.

METHODS:

Data were collected on 80 680 pediatric outpatients, ages 4 to 15 years, whose parents filled out the PSC-17 from 2006 to 2015 via the Child Health and Development Interactive System, an electronic system that presents and scores clinical measures.

RESULTS:

The rates of positive screening on the overall PSC-17 (11.6%) and on the internalizing (10.4%) and attention (9.1%) subscales were comparable to rates found in the original sample, although the rate of externalizing problems (10.2%) was lower than in the derivation study. Reliability was high (internal consistency 0.89; test–retest 0.85), and a confirmatory factor analysis provided support for the original 3-factor model.

CONCLUSIONS:

Fifteen years after the PSC-17 was derived in a large nationally representative outpatient pediatric sample, a new and larger national sample found rates of positive screening, reliability, and factor structure that were comparable. Findings from this study support the continued use of the PSC-17 clinically as a screening tool in pediatric settings and in research.

What’s Known on This Subject:

The Pediatric Symptom Checklist-17 (PSC-17) has been widely used clinically and in research for more than a decade yielding higher rates for detecting psychosocial dysfunction than clinical judgement and case rates comparable to the PSC-35, other psychosocial screens, and semistructured interviews.

What This Study Adds:

This study shows that in a new national sample, the prevalence of risk, reliability, and factor structure of the PSC-17 were comparable to those reported in the original derivation study, thus supporting its continued clinical and research use.

Mental health problems are common among children and adolescents, ∼13% of whom are estimated to have a problem that impairs functioning.^1–3 Research continues to show that only about half of these children are identified,^4–6 and only a fraction of them receive mental health services.⁷ Since studies have shown that brief assessment tools can improve identification rates of mental health problems in primary care settings,^8–10 national programs (Head Start),¹¹ professional organizations (American Academy of Pediatrics),¹² blue ribbon commissions (the National Academy of Science),⁷ and the Early and Periodic Screening Diagnosis and Treatment regulations of the US Medicaid programs² have consistently recommended routine psychosocial screening for youth, especially in the context of the pediatric medical home.

One of the most frequently recommended screening measures is the Pediatric Symptom Checklist (PSC).² The original version of the PSC has 35 items and was validated 3 decades ago.^1,13 The PSC has been translated into >2 dozen languages and is also available in a briefer version,³ 2 pictorial versions,¹⁴ and a youth self-report form.^15,16 The measure has been widely used in research and clinical settings and has been endorsed by organizations such as Bright Futures Mental Health and the Medicaid programs of several states.^17–20 The PSC is also available online through its Web site (http://www.massgeneral.org/psychiatry/services/psc_home.aspx), the Child Health and Development Interactive System (CHADIS), organizations that provide testing materials (CNS Vital Signs),²¹ and mental health advocacy groups (Mental Health America).²²

The briefer version of the PSC³ is broadly used, with >40 published studies.²³ These studies have shown that the PSC-17 yields higher detection rates than pediatricians relying on clinical judgment alone²⁴ and has risk rates comparable to those of the PSC-35,³ semistructured interviews (Schedule for Affective Disorders and Schizophrenia for School-Age Children–Present and Lifetime Version),²⁵ and longer questionnaire measures.² The PSC-17 was derived from the PSC-35 through an exploratory factor analysis conducted on data collected from the 1994 to 1999 Child Behavior Study (CBS), a nationally representative sample of >20 000 pediatric outpatients.³ In that study, the exploratory factor analysis suggested that it was possible to create a briefer version of the PSC with 17 of the original 35 items. These 17 items loaded onto 3 distinct factors, which became the 3 subscales of the PSC-17.

Several studies have replicated the PSC-17’s 3-factor structure by using confirmatory factor analysis (CFA) in moderate-sized samples,^26,27 and a number of studies^28–37 have reported on the prevalence of risk by using the overall and subscale scores. However, despite the wide use of the PSC-17 and the passage of >15 years, there have been no replication studies in a large national outpatient pediatric sample. The availability of such a sample made it possible to conduct the current replication study. We hypothesized that, with respect to the original PSC-17 derivation sample,³ the new sample would be comparable in reliability, factor structure, and in positive screening, despite the passage of ∼15 years.

Methods

Data for this study were obtained from the CHADIS (https://www.chadis.com), a Web based clinical process support system that includes >200 different previsit screens supporting comprehensive pediatric care and is available for a licensing fee to clinicians, with unlimited use for patients. Although CHADIS provides some specific decision support and postvisit engagement, no PSC-specific supports of that kind were available during the study period apart from automated scoring and content-related e-chapters. Automated administration of additional subscale-specific questionnaires (eg, positive PSC anxiety subscale triggering the Screen for Child Anxiety Related Disorders (SCARED) anxiety scale) is now available but was not during the study period. For the current study, deidentified data from all practices that had chosen to use the PSC were available from a central server. Because the analyses used only deidentified data, the study was approved as exempt by the Partners Healthcare institutional review board.

Sample

This study used all fully completed, nonduplicate PSC-17s from parents of patients ages 4 to 15 years filled out via CHADIS from the time this questionnaire was first made available (April 2005) until April 1, 2015. More than 85% of the cases reported in this study’s primary analytic sample were collected after 2009 (15 years after most of the original CBS PSC-17 data were collected in 1995). The current sample was collected from 325 pediatric offices representing all regions of the United States, similar to the CBS sample used in the original derivation study, except that the CBS sample was designed to be nationally representative of pediatric and family practices in the United States and Canada, whereas the current sample was limited to US practices that had purchased CHADIS licenses. CHADIS practices generally use default settings for well-child visits for patients of specified age groups. Most of the parents of children 4 to 15 years old in this sample were assigned the PSC-17 for well-child visits. However, it is important to note that CHADIS permits clinicians to auto-assign for behavioral visits and manually add screens at any visit. No demographic data other than gender and age were collected for individual cases; however, practice information suggested that about two-thirds of all practices were primarily suburban, with the remainder about equally split between urban and rural.

The full dataset began with 100 114 cases of patients 0 to 24 years of age. To compare this sample with the original PSC-17 derivation sample, we excluded patients aged 0 to 3 years and 16 to 24 years (n = 8328), leaving 91 786 youth 4 to 15 years of age. Because this study sought to investigate the use of the PSC-17 in a US primary care pediatric population, an additional 11 106 patients screened in the specialty practices of developmental behavioral pediatricians and in 6 practices outside the United States were excluded. The remaining 80 680 patients constituted the primary analytic sample.

For a subsample of 21 940 patients, there were ≥2 PSC-17s, and it was possible to assess longitudinal changes across varying intervals. We selected cases in which a parent had completed a second form 8 to 14 days after the first to assess the test–retest reliability of the PSC-17.

Measures

The PSC-17 total score is designed to evaluate a child’s overall psychosocial functioning. The measure also has subscales consisting of either 5 or 7 items that can be used to assess functioning in the areas of internalizing, attention, and externalizing problems. Parents are asked to rate each symptom with 0 = never, 1 = sometimes, and 2 = often. Weighted scores for the 17 items are summed to produce a total score ranging from 0 to 34. Higher scores indicate greater risk. Total scores are also recoded dichotomously, based on a validated cutoff score of ≥15^3,25 on the global scale to indicate overall mental health risk. Subscale total and categorical scores are calculated in the same manner. Scores of ≥7 indicate risk on both the attention and externalizing subscales, and scores of ≥5 indicate risk on the internalizing subscale.

Analytic Method

By using our primary analytic sample of 80 680 cases, we assessed the prevalence of risk at the first administration of the PSC-17. We used Cronbach’s α to evaluate the item reliability of the total PSC-17 and its 3 subscale scores, and we used the 84 cases whose parents completed a second PSC 8 to 14 days after the first to calculate the intraclass correlation (ICC) between total scores at the first and repeat administrations. We also evaluated the 3-factor model proposed by Gardner et al³ for a subset of the sample consisting of the first administration of the PSC-17 for children aged 4 through 15 years (n = 59 836) by using CFA. Preliminary tests and assumptions are specified in the Results section devoted to factor analysis.

Results

Sample Characteristics

As shown in Table 1, there were 80 680 patients in the 4- to 15-year-old primary analytic sample, and the rate of positive screening scores on the global scale was 11.6%. The PSC internalizing, attention, and externalizing problem subscales had positive screening rates of 10.4%, 9.1%, and 10.2%, respectively. Table 1 also shows the breakdown of the primary analytic sample by gender (51.4% were male) and age group. School-aged children (6–15 years old) made up most (74.2%) of the sample, and preschool-aged children (ages 4 and 5) accounted for the rest.

TABLE 1.

Positive Screening Rates and Demographic Breakdowns for Full Samples and Subsamples Based on Presence and Interval of Longitudinal Follow-up

	Full Time 1 Sample, % (N = 80 680)	Only 1 PSC Score, % (Time 1 Only) (N = 58 241)	≥2 PSC Scores, % (at Any Follow-up Interval) (N = 21 940)	1st Follow-up PSC-17 Score 8–14 d After Initial Screen, % (N = 84)
Risk on the PSC-17
Global scale	11.6 (9351)	12.0 (7006)	10.4 (2287)	27.4 (23)
Internalizing subscale	10.4 (8356)	10.9 (6335)	9.0 (1983)	21.4 (18)
Attention subscale	9.1 (7346)	9.4 (5458)	8.4 (1843)	16.4 (26)
Externalizing subscale	10.2 (8229)	10.3 (5998)	9.9 (2181)	12.6 (20)
Gender
Male	51.4 (41 452)	51.3 (29 854)	51.7 (11 353)	52.2 (83)
Female	48.6 (39 228)	48.7 (28 387)	48.3 (10 587)	47.8 (76)
Agea
Preschool aged	25.8 (20 844)	23.8 (13 858)	31.3 (6859)	17.6 (28)
School aged	74.2 (59 836)	76.2 (44 383)	68.7 (15 081)	82.4 (131)

^aChildren were included in the preschool age group if they were between the ages of 4 and 5 y. School-aged children were 6–15 y old.

Table 1 also provides information on the 58 241 subjects with a single PSC-17 and Time 1 information for all 21 940 cases with a follow-up PSC-17 at any time point. Similar data are presented on the Time 1 scores of all subjects who had a second PSC-17 and for the 84 subjects who had a follow-up PSC-17 8 to 14 days after the first, the time window we chose for our test–retest analyses.

Psychosocial Functioning and Risk Factor Groups

Table 2 presents data on the positive screening rates for the overall PSC-17 and its subscales from the current study along with comparative data from the original PSC-17 derivation study.³ Data from the current study are broken out separately for the preschool- and school-aged age groups, with the same age cutoffs that were used in the original derivation study. Table 2 shows the positive screening rates for Time 1 for all 80 680 patients ages 4 through 15 in our primary analytical sample and the positive screening rates for patients ages 4 through 15 in the original derivation sample.³ In contrast to the positive screening rate of 11.6% in the current sample, the overall PSC-17 positive screening rate reported in the original study was 15% (z = −12.61; P < .001). The prevalence rate for positive screens on the internalizing subscale in the current study is similar to that of the original derivation sample¹ (10.4% vs 10%, z = 1.60; P = .110). The difference in positive screening rates on the attention subscale in the 2 samples was small (9.1% current study vs 10% original study, z = −3.77; P < .001) but statistically significant. The rate of positive screening on the externalizing subscale in the current study (10.2%) was substantially lower than the rate of 17% reported in the original study (z = −25.94; P < .001).

TABLE 2.

Rates of Positive Screening in the Current and Original Derivation Samples

	Current Study Sample (N = 80 680)	Gardner et al 1999 (N = 18 045)a
Measure	PSC-17	PSC-17
Setting	National sample of US pediatric practices	National networks of US and Canadian pediatric and family practice offices
Age range	4–15 y	4–15 y
PSC case rate, %
Total PSC	11.6 (9351/80 680)	15
Internalizing subscale	10.4 (8356/80 680)	10
Attention subscale	9.1 (7346/80 680)	10
Externalizing subscale	10.2 (8229/80 680)	17
Gender, %
Male	14.1 (5831/41 452)	—
Female	9.0 (3520/39 228)	—

—, indicates that information is not available for gender distribution.

^aData obtained from the 1994–1999 CBS.

Although the original Gardner et al³ PSC-17 study did not give a breakdown of positive screening rates by gender, we present it here for the current sample. The last 2 rows of Table 2 show that boys were more likely to screen positive than girls (14.1% vs 9.0%; z = 22.59; P < .001, 95% confidence interval [CI], 0.047–0.055).

PSC-17 Reliability

In the current Time 1 sample of 80 680 children aged 4 to 15 (Table 2), Cronbach’s α for the overall PSC-17 was 0.87, and for the internalizing, attention, and externalizing subscales αs were 0.78, 0.82, and 0.80, respectively. In the original derivation study³ these figures were 0.89 for the overall scale and 0.79, 0.83, and 0.83 for internalizing, attention, and externalizing problem subscales.

Using the scores of 84 patients whose parents had completed a second PSC from 8 to 14 days after the first, the test–retest reliability of the PSC-17 in the current sample was assessed through ICC. The ICC between the overall PSC-17 scores at Time 1 and Time 2 (8–14 days later) was 0.85 (F[83, 83] = 12.46, P < .001). The ICCs between the Time 1 and Time 2 scores for the internalizing, attention, and externalizing subscales were 0.76 (F[83, 83] = 7.48, P < .001), 0.83 (F[83, 83] = 10.92, P < .001), and 0.82 (F[83, 83] = 10.08, P < .001), respectively.

Factor Structure of the PSC-17

All items loaded on exactly 1 factor, and all measurement error was presumed to be uncorrelated. The latent variables (internalizing, attention, and externalizing) were allowed to be correlated. As a result, the model was overidentified, with 116 df. There were no cases with missing data because of the online method used. The data were evaluated for multivariate outliers via the Blocked Adaptive Computationally efficient Outlier Nominators (BACON) algorithm.³⁸ An outlier was defined as a case whose multivariate distance from the median was greater than or equal to the 85th percentile of the χ² distribution. There were no outliers in the data. The data were also tested for normality and were found to be multivariate nonnormal and heavily skewed right.

Because of the multivariate nonnormality of the data and the ordinal, noncontinuous nature of the item data, the CFA model was specified via a polychoric correlation matrix and asymptotic covariance matrix and an unweighted least squares estimation method.^39,40The matrices were generated in PRELIS 9.2 and analyzed with LISREL 9.2⁴¹ and are available on request.

Goodness of fit was evaluated via the Satorra–Bentler scaled χ² statistic, standardized root mean square residual (SRMR), root mean square error of approximation (RMSEA) and its 90% CI, comparative fit index (CFI), and the Tucker–Lewis index (TLI). Multiple indices were used to evaluate different aspects of model fit (ie, absolute fit, fit adjusting for model parsimony, fit relative to a null model). Hu and Bentler⁴² suggested that an acceptable model fit is defined by the following criteria: χ² (P < .05), RMSEA (≤0.06, 90% CI LB ≤0.06), SRMR (≤0.08), CFI (≥0.95), and TLI (≥0.95).

As expected given the very large sample size, the χ² test was significant (χ² = 22 996.62, 116 df, P < .001), indicating that the model estimates do not exactly reproduce the sample variances and covariances. SRMR (0.064) provided a measure of absolute fit; RMSEA (0.104; 90% CI, 0.104–0.105) provided a measure of fit adjusting for model parsimony; CFI (0.89) and TLI (0.87) provided measures of comparative fit. Completely standardized parameter estimates from the solution are presented in Supplemental Table 3. All freely estimated unstandardized parameters were statistically significant (P < .001). Factor loading estimates revealed that the indicators were moderately to strongly related to their purported factors (range of R² 0.40–0.81), consistent with the position that the PSC-17 items are reliable indicators of the constructs of internalizing, attention, and externalizing. Estimates from the 3-factor solution indicate moderately strong relationships between the 3 dimensions (available in Supplemental Table 4).

Discussion

The current study provides evidence from a large national pediatric primary care sample that the rates of risk and reliability of the PSC-17 found in the current sample were comparable to those reported in the original derivation study collected about 15 years earlier and that the previously identified factor structure fit the current data reasonably well.

The greatest difference between the current and the original PSC-17 derivation samples was on the externalizing subscale, which had a positive screening rate of 10.2% in the current sample and 17% in the original sample. This difference could be a result of differences in sampling, secular trends, or other factors. Although the current dataset collected little demographic data on individual cases, information from the practices showed a very high percentage of suburban practices in the current sample, suggesting the possibility that the overall socioeconomic status of these subjects might be much higher than it was in the original PSC-17 sample.¹ As noted in previous studies with the PSC⁴³ and other measures,^9,13,44,45 the rate of positive screening, especially for externalizing problems, is usually higher in lower–socioeconomic status populations. The slightly lower rate of overall positive screening (11.6% currently vs 15% originally) is small and appears to result primarily from the much lower rate of externalizing problems in the current sample because the rates of internalizing and attention problems were almost identical. Because the derivation study used a nationally representative sample and the current study used a sample of convenience, we believe that the original norms may still provide the most accurate estimate of prevalence in the general outpatient pediatric population.

Internal consistency reliability in the current study (0.87 for the overall PSC-17) was similar to that reported in the derivation sample (0.89), and the test–retest reliability was high (0.89). In the CFA, with the exception of SRMR, all goodness of fit indices failed to meet the predefined criteria for acceptable fit. In suggesting these criteria, Hu and Bentler⁴² were careful to specify that measures of fit should be “close to” the suggested cutoff points, because all these measures are affected by study characteristics such as sample size, estimation method, normality of data, data type, and model complexity. It is beyond the scope of this article to thoroughly investigate the impact of these on the measures of fit. However, when they are considered together and in the context of current study characteristics, the 3-factor model appears to have adequate fit.^46–48 Overall, parameter estimates and correlations of items to factors and between factors are consistent with expectations.

Although the lack of case-specific socioeconomic data is a limitation of this study, other than the prevalence-of-risk findings, which have been shown to be sensitive to socioeconomic differences, the reliability and CFA findings reported in this study are statistically meaningful even in nonrepresentative samples.

The current study provides strong evidence supporting the continued use of the PSC-17 as a brief psychosocial screen for children and adolescents in primary care. In replicating the prevalence and reliability, and confirming the factor structure of the PSC-17 subscale scores, the current study makes an important contribution because both pediatric and mental health clinicians often report the usefulness of having subscale scores from the 3 different domains of the PSC-17. Beyond their clinical usefulness, the subscale scores have been shown to be a convenient metric for tracking treatment and quality assurance.^49–51

The usefulness of the PSC-17 subscale scores carries with it the need for caution, however. Risk ratings on the PSC-17 subscale scores, like those on any brief screening measure, are valid only as indicators to be considered during more thorough clinical or structured assessments. Relying on subscale scores to generate a diagnosis or treatment in the absence of additional data is not appropriate. Automated pass-through to more diagnostic questionnaires based on subscale scores on the PSC-17 recently available in the online system used in this study is a promising approach to facilitate additional evaluation.

Conclusions

This study replicated in a new, large national sample findings from a derivation sample collected about 15 years earlier and therefore supports the continued use of the PSC-17 as a measure of psychosocial functioning in pediatric primary care and research.

Glossary

CBS

Child Behavior Study

CFA

confirmatory factor analysis

CFI

comparative fit index

CHADIS

Child Health and Development Interactive System

CI

confidence interval

ICC

intraclass correlation

PSC

Pediatric Symptom Checklist

RMSEA

root mean square error of approximation

SRMR

standardized root mean square residual

TLI

Tucker–Lewis index