Development and Validation of the Knowledge of Effective Parenting Test (KEPT) in a Nationally Representative Sample

Abstract

We report on the development and psychometric properties of an instrument for the assessment of knowledge of effective parenting skills specific to conduct problems using an item response theory (IRT) framework. The initial item pool (36 items) for the Knowledge of Effective Parenting Test (KEPT) was administered online to a national sample (N = 1,570) selected to match the U.S. population on key demographic variables. Items with strong psychometric properties and without significant differential item functioning (DIF) by race/ethnicity were retained, resulting in a 21-item version of the KEPT with excellent reliability and validity. We also created a brief 10-item version of the KEPT to reduce respondent burden and to enhance its utility for repeated measurement in longitudinal and intervention research. We report norms and percentiles for both the 21-item version (KEPT-Full) and the 10-item version (KEPT-Brief).

Effective parenting is a critical component of healthy child development and effective approaches are associated with fewer behavioral problems, decreased levels of aggression, and improved social skills (Gryczowski, Jordan, & Mercer, 2010; Kaiser, McBurnett, & Pfiffner, 2011). Effective parenting strategies are also associated with academic achievement and enhanced school performance (Swanson, Valiente, Lemery-Chalfant, & O’Brien, 2011). Parenting is also associated with long-term adjustment, affecting children throughout adolescence and into adulthood (Aquilino & Supple, 2001; Morris, Silk, Steinberg, Myers, & Robinson, 2007). For example, children raised with consistent and effective parental practices experience higher levels of positive adjustment (fewer depressive symptoms, decreased risk for substance abuse, and higher self-efficacy) in young adulthood. Conversely, children raised with ineffective parenting strategies experience lower levels of self-esteem and self-efficacy, decreased life satisfaction, heightened irritability, and an increased risk for substance abuse problems into young adulthood (Aquilino & Supple, 2001; Ben-Ami & Baker, 2012; Morris et al., 2007).

Parenting Skills as Target Mechanisms for Treating Disruptive Behavior Disorders

Parent management training (PMT) skills are specific treatment targets common across evidence-based psychosocial treatments for childhood disruptive behavior disorders (Lindhiem, Higa, Trentacosta, Herschell, & Kolko, 2014). Such treatments typically include (a) teaching caregivers skills including the use of praise, rewards, consequences (e.g., removing privileges), and time-outs, and (b) teaching children skills including problem-solving, communication, and emotion labeling (Eyberg, Nelson, & Boggs, 2008). Substantial evidence points to skill deficits in these areas as pathogenic and maintenance factors for disruptive behavior disorders. As a result, skills in these domains are conceptualized as active agents of therapeutic change and are specifically targeted in treatments. Most research on treatment adherence has focused on how well clinicians follow treatment protocols, and less research has focused on the extent to which patients and families acquire knowledge of relevant skills and use them outside of treatment sessions (Huey, Henggeler, Brondino, & Pickrel, 2000; Nock & Ferriter, 2005; Waltz, Addis, Koerner, & Jacobson, 1993). Despite the conceptual importance of skills as target mechanisms of treatment, skill acquisition in these domains is challenging to assess.

The Challenges of Measuring Parenting

Parenting involves complex and multifaceted patterns of behavior that are difficult to measure (e.g., Eddy, 2017; Lindhiem & Shaffer, 2017). Quantifying effective parenting is challenging for many reasons, including contextual factors, cultural considerations, and limits on observable behavior. Because of the complexity involved in measuring and quantifying the various dimensions of parenting, it is not surprising that existing measures are limited in several important ways, including recall biases inherent in retrospective self-reports, the contrived nature of laboratory observations, and the reactivity of home observations creating social desirability effects. Existing measures of parenting skills also tend to lack adequate psychometric properties or established normative data (Hurley, Huscroft-D’Angelo, Trout, Griffith, & Epstein, 2014). There have also been numerous calls for measures of parenting that are culturally sensitive (Beidas et al., 2015; Jensen-Doss & Hawley, 2010; Locke & Prinz, 2002). Thus, measures of parenting are still needed that reduce social desirability bias, are culturally sensitive, and are sensitive to change (Locke & Prinz, 2002; Smith, 2011). Finally, if evidence-based assessments are to be adopted widely, they must be available at low cost (ideally free), brief (to minimize respondent burden and to enhance compliance), and validated across relevant populations including ethnic minority groups and low SES populations (Beidas et al., 2015; Jensen-Doss & Hawley, 2010).

The Current Study

To address these challenges and needs, we developed a brief online measure of parent management training (PMT) skills that has strong psychometric properties, including national norms, and is available at no cost. The Knowledge of Effective Parenting Test (KEPT) is a measure of parenting knowledge specific to conduct problems for parents/guardians of children aged 5–12 years, intended for researchers conducting trials of parenting interventions. Respondents answer multiple-choice questions with four options in response to video vignettes of both common and challenging parenting scenarios. The assessment covers content including age-appropriate expectations, effective communication, planned-ignoring, rewards, consequences, time-outs, attending to positive behavior, and praise. These domains were selected because of their empirical association with childhood externalizing behavior and their representation in the 16 psychosocial evidence-based treatments (EBTs) currently available for children and adolescents with disruptive behaviors (Eyberg et al., 2008). By developing a measure that uses a common elements approach (Chorpita, Daleiden, & Weisz, 2005; Garland, Hawley, Brookman-Frazee, & Hurlburt, 2008) and that is also “free, brief, and validated” (Beidas et al., 2015), our goal was to maximize the likelihood of successful dissemination and adoption by developmental researchers and developers of early interventions. Finally, the use of an item response theory (IRT) framework has allowed us to select items from our initial item pool that are the most informative with regard to knowledge of effective parenting and that are free from significant differential item functioning (DIF). The lack of DIF ensures a measure of knowledge of parenting skills appropriate for racially and ethnically diverse samples, without racial bias in the meaning of item scores. The availability of measures that function equivalently across diverse groups is increasingly essential, as the fastest growing population of racial and ethnic minorities in the United States is children under the age of 18. The U.S. Census Bureau projects that in the next two-to-three years slightly less than 50% of children will identify as non-Hispanic White (Vespa, Armstrong, & Medina, 2018).

Method

Overview

The project was approved by the Institutional Review Board at the University of Pittsburgh. The Knowledge of Effective Parenting Test (KEPT) was developed in several stages. In the first stage, an item pool of 36 multiple-choice questions was developed to measure skills typically taught during parent management training (PMT). In the second stage, all 36 items were administered online to a national sample (N = 1,570) of parents/guardians of children aged 5–12 years, along with several other measures used to establish convergent and divergent validity. In the third stage, IRT analyses were performed to select items with strong psychometric properties and without significant DIF by race/ethnicity for the final version of the KEPT. In this stage, we also created a brief 10-item version of the KEPT to enhance its utility as a research tool for both longitudinal and intervention research. In the fourth and final stage, we examined the reliability and validity of the full 21-item version (KEPT-Full) and brief 10-item version (KEPT-Brief). Measures of related parenting constructs as well as child and parent psychopathology were used to examine convergent and divergent validity. We also estimated national norms and percentiles.

Item Development (Stage 1)

Items were developed through an iterative process that began by reviewing the “common elements” literature (Chorpita et al., 2005; Garland et al., 2008) to identify parenting domains that are consistently represented in empirically based treatments for disruptive behavior problems. We started by reviewing the major treatments for disruptive behavior disorders including Parent Management Training – Oregon Model (PMTO; Bullard et al., 2010), Parenting Through Change (Martinez & Forgatch, 2001), Fast Track (Conduct Problems Prevention Research Group, 2004), Parent Child Interaction Therapy (PCIT; Funderburk & Eyberg, 2011), Incredible Years (Webster-Stratton & Reid, 2003), Problem-Solving Skills Training (Kazdin, 2003), Positive Parenting Program (Triple P; Bor, Sanders, & Markie-Dadds, 2002), and Keeping Foster Parents Trained and Supported (KEEP; Leathers, Spielfogel, McMeel, & Atkins, 2011). Using this approach, six core domains of parenting skills (praise, rewards/point system, attending and ignoring, commands/expectations, consequences, and time-outs) were identified (see Table 1).

Table 1.

Content for the Item Pool (36 items), KEPT-Full (21 items), and KEPT-Brief (10 items)

	Item Pool (36 items)	KEPT-Full (21 items)	KEPT-Brief (10 items)
Praise	3 items	2 items	1 item
Rewards / Point System	7 items	6 items	2 items
Attending and Ignoring	4 items	4 items	1 item
Commands / Expectations	6 items	3 items	3 items
Consequences	5 items	3 items	2 items
Time-outs	3 items	-	-
Two or more skills	8 items	3 items	1 item

Next, multiple-choice items were drafted to assess each of these six domains. A subset of the questions (items 15–36) was based on video vignettes depicting common parenting scenarios (e.g., arguing, interrupting, non-compliance). Eight of the vignettes were video-taped to create audio-visual clips that are imbedded in the online measure. Professional actors of ethnically diverse backgrounds were hired from the standardized patient program at the University of Pittsburgh Medical Center (UPMC) to perform the vignettes. Five child actors (2 boys; 3 girls) ranged in age from 7 to 12.

To ensure that the items had strong face validity and were relevant across the major treatments for disruptive behavior disorders (DBD), we used a modified Delphi technique to solicit feedback on the item pool from key DBD treatment experts. We identified a group of 20 researchers involved in the development of one or more of nine evidence-based protocols for the treatment or prevention of childhood behavior problems (PMTO, Parenting Through Change, Fast Track, PCIT, Incredible Years, Problem-Solving Skills Training, Triple P, and KEEP). These 20 experts were contacted via email to provide feedback on a subset of 14 items. Items were presented in an online survey format and survey links were emailed to the experts, who were asked to indicate whether they agreed or disagreed on the “correct or best” answer indicated for each item. If they disagreed, they were given the option to suggest how to improve the item. The experts also provided general feedback on the entire assessment. We sent four email requests at weekly intervals to each of the 20 experts, generating a response rate of 65% (13/20). The respondents included at least one expert from each of the nine evidence-based protocols for the treatment or prevention of childhood behavior problems. Only two items had less than 80% expert agreement on the “correct or best” answer, indicating good consensus. See Figure 1 for a sample screen shot and item from the KEPT.

Figure 1.

Sample screen shot and item from the Knowledge of Effective Parenting Test (KEPT).

Data Collection (Stage 2)

Procedure.

We contracted with the survey company YouGov to administer all 36 items (see Supplemental Material), along with additional measures of parenting, child behavior, and parent and child psychopathology, to a national sample of parents/guardians of children aged 5–12 years that was representative of the U.S. population on key demographic variables. YouGov is a large survey company that uses a panel of over 1.2 million U.S. residents to conduct online surveys and assessments. Participants on the YouGov panels have been recruited through web advertising, permission-based email campaigns, partner contacts, random digit dialing, and mail (based on voter registration). YouGov has also obtained a large amount of demographic information from the panelists in previous studies. Links to the survey were emailed to YouGov panelists. Recent studies have found that 84% of Americans use the internet (Perrin & Duggan, 2015), 79% have regular access to internet in the home (File & Ryan, 2014), and internet samples reasonably represent the overall population (Hays, Liu, & Kapteyn, 2015). Participants reached via these recruitment strategies have opted-in and agreed to be contacted to complete surveys for which they are relevant. The response rate to the emailed survey link was 52.8%. Of those who initiated the survey, 46.2% met eligibility criteria. Participants received $25 in compensation for completing the full set of questions.

Participants.

The final sample included 1,570 parents/guardians of children aged 5–12 from all 50 states and the District of Columbia. Key demographic variables are summarized in Table 2.

Table 2.

Participant Demographic Characteristics (N = 1,570)

Parent/Guardian Age	M = 41.5; SD = 10.6
Parent/Guardian Gender
Female	63.7%
Male	36.3%
Child Age	M = 8.6; SD = 2.3
Child Gender
Female	46.8%
Male	53.2%
Parent/Guardian Race
White	78.7%
Black or African American	13.9%
Asian	3.4%
American Indian or Alaska Native	2.6%
Native Hawaiian or Pacific Islander	.6%
Some other race	5.5%
Parent/Guardian Ethnicity
Hispanic, Latino, or Spanish Origin	16.2%
Parent/Guardian Born in the United States	89.7%
Parent/Guardian Education
High school or lower	25.6%
Some college or higher	74.4%
Marital Status (% married)	68.2%
Family Income
Less than $10,000	6.0%
$10,000 – $29,999	20.0%
$30,000 – $49,999	20.3%
$50,000 – $69,999	16.6%
$70,000 and above	37.2%
U.S. Region
Northeast	18.3%
Midwest	24.3%
South	37.0%
West	20.4%

Measures.

Several additional measures of related parenting constructs and child and parent psychopathology were administered to examine convergent and divergent validity of the KEPT-Full and KEPT-Brief. Constructs included child behavior problems, child depression, child anxiety, child trauma, parent discipline strategies, parent psychopathology (depression and anxiety), and parent knowledge of behavioral principles.

Alabama Parenting Questionnaire (APQ).

The APQ is a 42-item measure that assesses five dimensions of parenting: (1) positive involvement, (2) supervision and monitoring, (3) use of positive discipline techniques, (4) consistency in the use of such discipline, and (5) use of corporal punishment, using a 5-point scale ranging from 1 (never) to 5 (always). The internal consistency of the scale has been reported to be acceptable with alphas for the five domains ranging from .46 to .80. The measure has well-established construct validity (Shelton, Frick, & Wooten, 1996).

Knowledge of Behavior Principles (KOBP-10).

The KOBP-10 is an abbreviated, 10-item version of the Knowledge of Behavior Principles measure. Questions are presented in a multiple-choice format. Short forms of the KOBP have Cronbach’s alphas ranging from .66 to .89 and strong content validity (Furtkamp, Giffort, & Schiers, 1982; Sturmey, Newton, Milne, & Burdett, 1987).

Vanderbilt Assessment Scale-Parent Report (VAS-P).

The VAS-P is a 55-item parent-report screen for Attention-Deficit/Hyperactivity Disorder (ADHD), Oppositional Defiant Disorder (ODD), and Conduct Disorder (CD). It also includes seven items on internalizing symptoms and eight items on school performance and social functioning. Symptom items are rated using a 4-point scale (“never” to “very often”) and the performance items are rated on a 5-point scale from “problematic” to “above average.” The measure has Cronbach’s alphas ranging from .79 to .95 and strong evidence of construct validity (Wolraich et al., 2003).

Screen for Child Anxiety Related Disorders (SCARED).

The SCARED is a 41-item measure of general anxiety, separation anxiety, social phobia, and school phobia. Items are rated using a 3-point Likert scale. The internal consistency of the SCARED is good with Cronbach’s alphas ranging from .74 to .93. The measure has strong convergent and divergent validity (Birmaher et al., 1997).

Patient Heath Questionnaire depression scale (PHQ-8).

The PHQ-8 measures symptoms of depression using a 4-point scale from “not at all” to “nearly every day.” Total scores range from 0 to 24. The measure has a reported Cronbach’s alpha of .82 and evidence of construct validity (Pressler et al., 2011).

Generalized Anxiety Disorder Screener (GAD-7).

The GAD-7 is a 7-item measure of anxiety. Items are rated on a 4-point scale from “not at all” to “nearly every day.” The measure includes an item to assess the duration of anxiety symptoms. The measure has excellent internal consistency (Cronbach’s alpha = 0.92), good test-retest reliability (intraclass correlation = 0.83), and strong convergent validity with other measures of anxiety (Spitzer, Kroenke, Williams, & Löwe, 2006).

Item Selection / IRT Analyses (Stage 3)

We began by computing item-total correlations for each item. Items with item-total correlations above .30 were analyzed further to ensure unidimensionality of items before initiating the IRT analyses. Unidimensionality was tested by examining the ratio of first to second eigenvalues from an exploratory factor analysis (Hawes et al., 2014; Morizot, Ainsworth, & Reise, 2007). One common threshold used to support unidimensionality is a ratio of ≥ 3. Two additional goodness of fit indices were evaluated. The Very Simple Structure criterion (VSS) has been shown to identify more consistently the optimal number of factors to extract than maximum likelihood tests and Kaiser’s eigenvalue cut-off of 1.0. The VSS evaluation is a degradation of the factor solution, testing how well the factor matrix fits the correlation matrix. The maximum value achieved represents the ideal number of factors to extract. The Velicer Minimum Average Partial (MAP) test characterizes the squared, average partial correlations among the scale items after removing the effect of the factors.

IRT analyses were conducted using IRTPRO (Cai, Toit, & Thissen, 2011) and the R statistical environment (R Core Team, 2018). Local independence of items was tested by examining the correlations of residuals (these should be close to zero), after accounting for the latent trait score. We generated item characteristic curves (ICCs), item parameters, and indices of differential item functioning (DIF) for each item. We used a three-parameter logistic (3PL) IRT model to estimate the discrimination (α), difficultly (β), and guessing (γ) parameters for each item. The discrimination (α) parameter is the slope of the item characteristic curve (ICC) at β and measures how well the item discriminates between participants whose latent trait ability level (θ) is above the item’s β parameter and those whose latent trait ability level (θ) is below the item’s β parameter. Higher α parameters indicate better discrimination. The difficulty parameter (β) is defined as the latent trait level at which a participant has a .50 chance of answering the item correctly. The guessing (γ) parameter estimates the lower asymptote of the ICC and is appropriate when participants have a reasonable chance of guessing the correct answer. Because most of the items consist of multiple-choice questions with four options (and only one correct answer), we expected each γ parameter to be approximately .25.

Differential item functioning (DIF).

We conducted DIF analyses to determine whether any of the items had significantly different ICCs and model parameters across our largest racial/ethnic subgroups. DIF occurs when an item performs differently for individuals from distinct groups at equivalent levels of the latent trait (when group membership is not related to measurement conditions). Tests for DIF that produce null results suggest the item is invariant across groups. DIF was examined across racial/ethnic groups: Hispanic or Latino, Black (non-Hispanic), and White (non-Hispanic), the three largest racial/ethnic groups of children in the United States currently (Vespa et al., 2018). Given the lack of consensus on the optimal form of DIF detection, two procedures were used and compared to identify whether items displayed DIF with both methods. Logistic Regression methods of DIF detection are based on fitting logistic regression models with a criterion such as the total scale score (Model 1), group membership in comparison to a reference group (Model 2), and the interaction between scale score and group membership (Model 3). Then, χ² statistics for each model are compared to detect DIF and determine the size of the DIF effect. A logistic regression approach to DIF detection was implemented in the R environment using the Likelihood Ratio test (Thissen, Steinberg, & Wainer, 1993). Pseudo R² values were calculated to determine effect size, representing the gain in log-likelihood from the explanatory variables in the model. Another well-established method of DIF detection is the Mantel-Haenszel (MH) χ² (Holland & Thayer, 1988; Mantel & Haenszel, 1959). The MH method involves matching respondents on total scale score and using a series of contingency tables for each item to make pair-wise comparisons between group-membership categories based on item performance. The MH method was also employed in the R software package. Items that demonstrated DIF across the two methods or that displayed a large DIF effect size were eliminated from the final measure.

Item selection.

Results from the IRT analyses were used to select items for the final version of the KEPT as well as a brief (10-item) version of the KEPT. Criteria for selecting the 10 “best” items for the brief version were: 1) items with high discrimination (α) parameters, and 2) items without significant DIF by racial/ethnic categories. See Figure 2 for a flowchart depicting the item selection process.

Figure 2.

Item selection process for the KEPT.

Psychometric Characteristics and Norms (Stage 4)

Reliability and validity.

We computed internal consistency (Cronbach’s α) for the KEPT-Full (21 items) and KEPT-Brief (10 items). We also examined convergent and divergent validity by computing correlations with related constructs. We expected higher correlations with more closely related constructs (e.g., knowledge of behavior principles and discipline practices) than with more distinct constructs (e.g., parent psychopathology).

National norms and percentiles.

We estimated national norms (Ms, SDs, and percentiles) for the KEPT-Full (21 items) and KEPT-Brief (10 items) based on the full sample (N = 1,570). Norms were also estimated for the “clinical” population described above (parents/guardians of children who screened positive for ODD and CD on the Vanderbilt Assessment Scale-Parent Version).

Results

Preliminary Analyses

Item-total correlations for the initial 36 items ranged from .09 to .59 (M = .39; SD = .11). Seven items with correlations below .30 were removed prior to subsequent analyses. The dimensionality of the remaining 29 items was examined using exploratory factor analyses (EFA) with maximum-likelihood (ML) estimation. The ratio of the first eigenvalue (8.64) to the second eigenvalue (.87) was 9.92, far exceeding the recommended cut-off of 3.0 for documenting unidimensionality. A unidimensional structure was also supported by additional goodness-of-fit indices. The MAP minimum value of .01 was reached with the one-factor solution, and the VSS maximum value of .62 was also reached with a one-factor solution. Finally, the correlations of the residuals for each item were all less than .10, indicating sufficient local item independence.

IRT Analyses

Model fit.

Table 3 presents the item-fit statistics for the 3PL models. Item-fit statistics indicated that the 3PL models fit the data well, with large Comparative Fit Index (CFI) and Tucker Lewis Index (TLI) coefficients. Separate models were fit for respondents identifying with each of the three racial/ethnic groups, White non-Hispanic, Black non-Hispanic, and Hispanic, prior to the DIF tests, and these separate models also had excellent fit.

Table 3.

3PL Model Fit Indices

	3PL model
	RMSEA	TLI	CFI
KEPT-Full (21 items)
Total sample	.03	.97	.98
Race/ethnicity
White (non-Hispanic)	.04	.96	.97
Black (non-Hispanic)	.04	.91	.93
Hispanic	.02	.99	.99
KEPT-Brief (10 items)
Total sample	.03	.99	.99
Race/ethnicity
White (non-Hispanic)	.03	.99	.99
Black (non-Hispanic)	.00	1.00	1.00
Hispanic	.00	1.00	1.00

Note. 3PL = 3 parameter logistic model; RMSEA = Root mean square error of approximation; CFI = Comparative fit index; TLI = Tucker Lewis Index.

Differential item functioning (DIF).

Six items were removed for evidence of DIF across racial and/or ethnic categories. Items were removed if they showed significant DIF using multiple methods, large effect sizes for DIF, or both. All six items that were removed demonstrated DIF for non-Hispanic Black respondents compared to non-Hispanic White respondents. Four of the six items also demonstrated DIF between non-Hispanic Black respondents and Hispanic respondents.

Item selection.

After the DIF analyses, two additional items were removed for having discrimination parameters below 1.0 in the full sample. The remaining 21 items comprised the final KEPT measure. Ten items with the strongest psychometric properties (highest discrimination parameters) were selected for the KEPT-Brief. Table 4 presents factor loadings for the full and brief versions of the KEPT. Table 5 summarizes the item parameters for each of the items in the KEPT-Full and KEPT-Brief. All discrimination (α) parameters were above 1.0, ranging from 1.05 to 2.20 (M = 1.58; SD = .39). Difficulty (β) parameters ranged from −1.92 to 1.51 (M = −.30; SD = .69). Guessing (γ) parameters ranged from .09 to .29 (M = .17; SD = .06). See Table 1 for the item content for the KEPT-Full and KEPT-Brief. Figures 3 and 4 show the test information curves for the KEPT-Full and KEPT-Brief along with the standard errors of measurement. As would be expected, the KEPT-Full affords greater precision of measurement with standard errors ranging from .33 to .52 for typical levels of theta (−1.5 to 1.5) compared to .40 to .74 for the KEPT-Brief.

Table 4.

Factor Loadings Resulting from EFA

Item number	KEPT-full	KEPT-brief
2	0.50	--
6	0.63	0.60
7	0.64	--
8	0.59	0.60
9	0.62	--
10	0.37	--
12	0.71	0.70
13	0.65	--
14	0.45	--
16	0.63	0.63
17	0.52	--
18	0.57	0.60
20	0.57	0.59
25	0.60	0.59
28	0.66	0.65
29	0.66	0.66
30	0.51	--
31	0.49	--
32	0.48	--
33	0.72	0.69
34	0.48	--
Internal consistency (α)	α = .84	α = .76

Note. EFA=Exploratory factor analysis based on tetrachoric correlations. The seven initial items in the pool with low inter-item correlations were removed prior to computing the EFA.

Table 5.

3PL Item Parameters for the KEPT-Full (21 items) and KEPT-Brief (10 items)

	Discrimination Parameter		Difficulty Parameter		Guessing Parameter
Item	α	s.e.	β	s.e.	γ	s.e.
2	1.06	0.13	−0.65	0.24	0.16	0.09
6*	1.76	0.19	−1.92	0.18	0.18	0.09
7	1.57	0.17	−0.60	0.14	0.14	0.06
8*	1.61	0.22	−0.14	0.14	0.21	0.06
9	1.47	0.17	0.04	0.11	0.09	0.05
10	1.81	0.35	1.51	0.11	0.14	0.02
12*	2.10	0.23	0.25	0.06	0.09	0.03
13	1.67	0.17	−0.87	0.15	0.16	0.07
14	1.35	0.25	0.39	0.16	0.25	0.06
16*	1.61	0.17	−0.60	0.14	0.15	0.06
17	1.08	0.12	−0.68	0.21	0.13	0.08
18*	2.01	0.29	0.04	0.10	0.28	0.04
20*	1.99	0.30	0.18	0.10	0.25	0.04
25*	2.11	0.33	−0.06	0.11	0.29	0.05
28*	1.82	0.23	−0.60	0.15	0.20	0.07
29*	1.73	0.19	−0.29	0.11	0.13	0.05
30	1.06	0.12	−1.15	0.24	0.15	0.09
31	1.05	0.14	0.31	0.15	0.11	0.05
32	1.11	0.15	−0.25	0.20	0.19	0.07
33*	2.20	0.24	−0.42	0.08	0.14	0.05
34	1.05	0.13	−0.86	0.25	0.20	0.09

Note. The ten items marked with an asterisk are included in the KEPT-Brief.

Figure 3.

Total test information curve for the KEPT-Full (21 items).

Figure 4.

Total test information curve for the KEPT-Brief (10 items).

Reliability and Validity

Cronbach’s alphas were .84 for the KEPT-Full and .76 for KEPT-Brief. Table 6 summarizes correlations between the KEPT-Full (21 items) and KEPT-Brief (10 items) and other measures of child behavior and child and parent psychopathology. As expected, the largest correlations were with other measures of parenting. Correlations were largest between the KEPT-Full and corporal punishment (r = −.42), poor monitoring/supervision (r = −.44), and knowledge of behavioral principles (r = .44). In contrast, the correlations with the other two APQ scales (involvement and positive parenting) were low and non-significant. Within the domain of child behavior/psychopathology, the KEPT-Full had the highest correlation with conduct problems (r = −.31). The correlation with oppositional and defiant behaviors was small (r = −.02) and nonsignificant. Scores on the KEPT-Full were not correlated with measures of parent depression (r = .00) or parent anxiety (r = .00). These converging and diverging patterns of correlations support the validity of the KEPT as a measure of knowledge of parenting skills with particular relevance to conduct problems. For the KEPT-Full, children whose parents/guardians scored below the 25^th percentile were twice as likely (risk ratio = 2.0) to screen positive for ODD and more than eight times as likely (risk ratio = 8.2) to screen positive for CD compared to children whose parents/guardians scored above the 25^th percentile. For the KEPT-Brief, these risk ratios were 2.0 for ODD and 10.7 for CD.

Table 6.

Correlations Between the KEPT-Full, KEPT-Brief, and Measures of Child Behavior, Parenting, and Parent and Child Psychopathology

	α	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
Parenting
1. KEPT-Full (21 items)	.84	.94**	−.42**	−.44**	−.19**	−.02	.02	.44**	−.31**	−.02	−.05*	.00	.04	−.18**	.00	.00
2. KEPT-Brief (10 items)	.76	-	−.39**	−.41**	−.17**	−.03	.00	.43**	−.29**	−.03	−.06*	−.01	.03	−.18**	.00	.00
3. Corporal Punishment (APQ subscale)	.80		-	.59**	.44**	−.17**	−.14**	−.09**	.58**	.31**	.35**	.34**	.29**	.39**	.20**	.21**
4. Poor Monitoring (APQ subscale)	.85			-	.54**	−.19**	−.05	−.08**	.58**	.28**	.26**	.31**	.31**	.36**	.18**	.18**
5. Inconsistent Discipline (APQ subscale)	.79				-	−.21**	−.17**	−.02	.45**	.43**	.37**	.40**	.37**	.37**	.28**	.27**
6. Positive Parenting (APQ subscale)	.81					-	.66**	−.14**	−.23**	−.19**	−.13**	−.21**	−.18**	−.11**	−.08**	−.11**
7. Involvement (APQ subscale)	.79						-	−.08**	−.16**	−.24**	−.16**	−.25**	−.15**	−.10**	−.16**	−.15**
8. Knowledge of Behavior Principles (KOBP)	.60							-	.00	.12**	.11**	.10**	.16**	.03	.10**	.12**
Child Behavior / Psychopathology
9. CD Symptoms (VADPRS)	.96								-	.59**	.49**	.48**	.61**	.61**	.40**	.41**
10. ODD Symptoms (VADPRS)	.91									-	.72**	.69**	.60**	.51**	.44**	.47**
11. Hyperactivity / Impulsivity (VADPRS)	.91										-	.74**	.48**	.45**	.40**	.43**
12. Inattention (VADPRS)	.92											-	.53**	.46**	.43**	.43**
13. Internalizing Symptoms (VADPRS)	.90												-	.72**	.49**	.49**
14. Anxiety (SCARED)	.95													-	.52**	.55**
Parent Psychopathology
15. Depression (PHQ8)	.91														-	.79**
16. Anxiety (GAD7)	.93															-

^*p < .05.

^**p < .01. α = Cronbach’s alpha

National Norms and Percentiles

Table 7 summarizes national norms for the KEPT-Full and KEPT-Brief. Scores did not differ by gender of the parent/guardian for either the KEPT-Full, F(1,1568) = .01, p = .92, or the KEPT-Brief, F(1,1568) = .17, p = .68. Means and standard deviations are also reported for parents/guardians of children who screened positive for ODD (N = 192; 12.2% of the full sample) and CD (N = 82; 5.2% of the full sample). As expected, mean scores were higher for the general population than for parents/guardians of children who screened positive for ODD and CD. Differences were most pronounced for CD, for which the mean score on the KEPT-Full was more than a full standard deviation lower for parents/guardians of children who screened positive for CD compared to those who screened negative, F(1, 1568) = 108.31, p < .00001. This difference represents a very large effect, Cohen’s d = 1.14. For ODD, the difference was smaller but still significant. Parents/guardians of children who screened positive for ODD scored about one-third standard deviation lower (d = .36) on the KEPT-Full compared to those who screened negative, F(1, 1568) = 21.65, p < .00001. Percentiles for the KEPT-Full and KEPT-Brief are summarized in Table 8.

Table 7.

Means and Standard Deviations for the KEPT-Full and KEPT-Brief

	Full Sample (N = 1,570)	ODD positive screen (N = 192)	CD positive screen (N = 82)
KEPT-Full (21 items)	13.56 (4.67)	12.10 (5.38)	8.51 (4.47)
KEPT-Brief (10 items)	6.69 (2.57)	5.84 (2.82)	4.02 (2.28)

Table 8.

Percentiles for the KEPT-Full and KEPT-Brief

KEPT-Full		KEPT-Brief
Score	%ile	Score	%ile
0	0	0	0
1	0	1	0
2	0	2	2
3	1	3	7
4	1	4	14
5	3	5	23
6	6	6	33
7	9	7	45
8	13	8	56
9	17	9	68
10	22	10	83
11	27
12	33
13	39
14	46
15	52
16	59
17	68
18	75
19	84
20	91
21	97

Discussion

The KEPT is unique among parenting measures because it was designed specifically as an assessment of a potential treatment target for evidence-based psychosocial treatments from a common elements perspective. Content was selected by reviewing the common elements literature on disruptive behavior disorders (DBDs) treatments and soliciting feedback from domain experts on pilot test items. The final item pool was tested rigorously using factor analyses and models from item response theory (IRT) to ensure that the final measure assessed a single construct and only contained items with strong psychometric properties. Each item was tested for differential item functioning (DIF) to ensure that the measure was free from racial or ethnic biases for the major U.S. census categories (White, African American or Black, Hispanic). The result is a measure of knowledge of parenting skills (treatment targets common to evidence-based psychosocial treatments for DBDs) with excellent reliability and validity. Most importantly, scores on the KEPT differentiate between parents/guardians of children who screen positive versus negative for ODD and CD. The difference in scores is particularly large for CD. The mean score on the KEPT-Full was more than a full standard deviation lower for parents/guardians of children who screened positive for CD compared to those who screened negative, a very large effect (Cohen’s d = 1.14). The KEPT was also designed to reduce social desirability bias. Because the measure is a test of knowledge of parenting skills with empirically derived correct and incorrect answers, it is difficult for respondents to inflate their score.

We also developed a brief version of the KEPT (10 items) to enhance its clinical utility. Assessment measures are more likely to be adopted if they are free, brief, and validated (Beidas et al., 2015). In developing the KEPT, we ensured that the measure met these three key criteria. The KEPT is freely available from the developers by request, can be administered in just a few minutes, and has strong psychometric properties. We acknowledge that the KEPT-Full has stronger indices of reliability and validity than the KEPT-Brief, and we recommend using the full version if time allows.

The KEPT as a Measure of Target Engagement in Treatment Trials

The KEPT is well suited to measure knowledge of parenting skills relevant to conduct problems as a treatment target in the context of clinical trials because it was designed to eliminate the problem of direction of causality that confounds so many measures of parenting. Because many measures of parenting ask respondents about their own use of discipline strategies and parenting behaviors, it is challenging to establish whether changes in parenting behaviors are, in fact, contributing to the reduction of behavior problems or vice versa (Lindhiem, Shaffer, & Kolko, 2014; Shaffer, Lindhiem, & Kolko, 2016). It is equally plausible that a reduction in behavior problems is contributing to changes in parenting behaviors. Because the KEPT measures parenting knowledge, however, it is far less plausible that reductions in child behavior problems could boost a caregiver’s score on the KEPT.

Strengths and Limitations

This study has numerous strengths including the systematic development of an item pool from a common elements approach, the solicitation of feedback on test items from a large cross-section of content experts, administration of test items to a large nationally representative sample, rigorous analyses of item characteristics and DIF of every item using models from IRT, the development of both a long form and a short form, the creation of national norms and percentiles, and evidence of strong reliability and validity. Limitations include a small number of items for each domain and the reliance on parents/guardians as the only informants for child behavior. This limitation is somewhat mitigated by substantial evidence that parents/guardians are accurate reporters of children’s externalizing behaviors (e.g., De Los Reyes & Kazdin, 2005; Loeber, Green, Lahey, & Stouthamer-Loeber, 1989). If the correlations between the KEPT and other measures of parenting and child behavior were due largely to shared method variance, then all correlations would be expected to be positive and convergent. However, this was not the case, as correlations with parent psychopathology, for example, were zero. Rather, the patterns of converging and diverging correlations between the KEPT and the other measures support the validity of the KEPT as a distinctive measure of knowledge of parenting skills. Another limitation is that feasibility considerations constrained our ability to collect observational data on parenting behaviors. This is somewhat less problematic than it would otherwise be, given that the KEPT is a measure of parenting knowledge versus a self-report measure of actual parenting behaviors. Finally, we were surprised at the non-significant correlation between the KEPT and the ODD subscale on the Vanderbilt Assessment Scale. This might suggest that deficits in PMT knowledge are more of a risk-factor for CD than for ODD, and certainly requires further investigation.

Future Directions

The KEPT is intended as a measure of parenting knowledge specific to conduct problems for use in treatment outcome studies. Before it can be adopted, however, the KEPT needs additional research evaluating its construct validity, test-retest reliability, and sensitivity to intervention effects. A reliable and valid measure of PMT knowledge will allow researchers to test whether instances of treatment failure are due to failure of target engagement (i.e., failure to understand and acquire PMT skills). This is a critical question because many families fail to respond to treatment, typically ranging from 20–60% depending on sample characteristics (age, diagnosis, severity, comorbid conditions, etc.) and definitions of nonresponse. The ability to test whether instances of treatment failure are due to failure of target engagement will help to refine existing treatments and lead to better outcomes.

It is important to note that the KEPT is not intended as a standalone measure but as a supplement to measures of actual parenting behaviors. Skill knowledge and skill use are conceptually related and understanding the variables that influence the translation of knowledge into practice is an important next step. There is meta-analytic evidence, for example, that parenting-based interventions are more effective in teaching parents new skills (i.e., knowledge acquisition) than in changing actual parenting behaviors (Lindhiem, et al., 2014). Measures of actual skill use with strong psychometric properties are needed to investigate the distinct antecedents and correlates of behavior change in parenting as well as techniques to enhance the utilization of effective parenting strategies.

Finally, another next step will be to develop a computerized adaptive testing (CAT) version of the KEPT using the 21 items from the KEPT-Full as an item bank. This could enhance the utility of the KEPT for repeated measurements in the context of clinical trials and longitudinal studies. A CAT version of the KEPT would administer items until a predetermined standard error of measurement was reached. In general, CAT achieves adequate precision in measurement with fewer items than static tests (often with as few as 4 to 6 items). A CAT option would mean an even briefer measure than the static 10-item KEPT-Brief with comparable measurement precision.

Public Significance Statement:

The study describes a new online research tool that will allow for the remote assessment of parenting knowledge relevant to conduct problems. Results indicate strong psychometric properties with implications for parenting-based intervention trials and developmental research.