Measuring the development of inhibitory control: The challenge of heterotypic continuity

Abstract

Inhibitory control is thought to demonstrate heterotypic continuity, in other words, continuity in its purpose or function but changes in its behavioral manifestation over time. This creates major methodological challenges for studying the development of inhibitory control in childhood including construct validity, developmental appropriateness and sensitivity of measures, and longitudinal factorial invariance. We meta-analyzed 198 studies using measures of inhibitory control, a key aspect of self-regulation, to estimate age ranges of usefulness for each measure. The inhibitory control measures showed limited age ranges of usefulness owing to ceiling/floor effects. Tasks were useful, on average, for a developmental span of less than 3 years. This suggests that measuring inhibitory control over longer spans of development may require use of different measures at different time points, seeking to measure heterotypic continuity. We suggest ways to study the development of inhibitory control, with overlapping measurement in a structural equation modeling framework and tests of longitudinal factorial or measurement invariance. However, as valuable as this would be for the area, we also point out that establishing longitudinal factorial invariance is neither sufficient nor necessary for examining developmental change. Any study of developmental change should be guided by theory and construct validity, aiming toward a better empirical and theoretical approach to the selection and combination of measures.

Inhibitory control, “the ability to inhibit responses to irrelevant stimuli while pursuing a cognitively represented goal” (Carlson & Moses, 2001, p. 1033), is a key construct in the domain of self-regulation (Kochanska, Murray, Jacques, Koenig, & Vandegeest, 1996). Self-regulation is a broad construct encompassing physiological, attentional, cognitive, emotional, and behavioral regulatory processes that promote adaptive or goal-directed behavior (Berger, 2011; Calkins & Fox, 2002). Self-regulation and inhibitory control measures have been associated with many important adjustment outcomes, including school readiness (Blair, 2002), health (Moffitt et al., 2011), and psychopathology (Dale & Baumeister, 1999), including in longitudinal studies (e.g., Brocki, Nyberg, Thorell, & Bohlin, 2007). Studies of self-regulation in children often include batteries of behavioral tasks thought to assess executive functioning, executive control, inhibitory control, response inhibition, behavioral regulation, and effortful control, among others (Carlson, 2005; Carlson & Moses, 2001; Kochanska, Murray, & Harlan, 2000). Despite the different terms used for similar measures, the different self-regulation constructs primarily reflect differences in research tradition rather than construct differences (Zhou, Chen, & Main, 2012). It has been difficult to achieve conceptual clarity among the different self-regulation constructs (McClelland & Cameron, 2012). The present review focuses on issues in measurement of inhibitory control, a key component of self-regulation across many research traditions, in longitudinal research. It is important to consider how to measure the development of inhibitory control because of the widespread use of inhibitory control measures in early childhood and its relevance to the development of psychopathology.

Inhibitory control, also referred to as response inhibition, is one of the separable cognitive processes thought to comprise the construct of executive function. In the past, inhibitory control was thought to emerge only in middle to late childhood, corresponding with improvements in the child’s ability to execute complex, higher-order integrative tasks (Welsh, Friedman, & Spieker, 2006). However, response inhibition, in its most basic form, is present in the first year of life including the inhibition of neonatal reflexes and the inhibition of predominant behavioral reaching responses (Diamond, 1990), and more robustly in the preschool years. The vast majority of research on the development of inhibitory control often focuses on development from early childhood to adolescence (Durston et al., 2002; Williams, Ponesse, Schachar, Logan, & Tannock, 1999), with behavioral measures of inhibitory control demonstrating dramatic improvement over this time span (Welsh et al., 2006; Williams et al., 1999). The age at which performance on inhibitory control tasks reaches adult levels depends largely upon task complexity and difficulty, with performance on some basic tasks reaching adult levels in early childhood, while performance on other tasks, which require the integration of multiple executive functions, continues to improve until adolescence (for a review, see Garon, Bryson, & Smith, 2008).

Executive function has been defined as “goal-directed cognitive control of thought, action, and emotion” (Zelazo et al., 2013, p. 16). In adults, executive function is thought to be a multi-dimensional construct comprised of separable cognitive processes including response inhibition, working memory, and cognitive flexibility (Miyake et al., 2000). However, it is less clear whether such a factor structure characterizes executive function in early childhood, with some studies suggesting that executive function is best represented as a single, unitary construct in early childhood (Fuhs & Day, 2011; Shing, Lindenberger, Diamond, Li, & Davidson, 2010; Wiebe, Espy, & Charak, 2008; Wiebe et al., 2011; Willoughby, Blair, Wirth, & Greenberg, 2010; Willoughby, Wirth, & Blair, 2012) that likely fractures into a multi-dimensional construct in later childhood, with other studies suggesting that a multidimensional factor structure similar to adults is present in early childhood (Lee, Bull, & Ho, 2013; Lee et al., 2012; Miller, Giesbrecht, Müller, McInerney, & Kerns, 2012).

The behavioral manifestation of inhibitory control changes across development, consistent with the notion of heterotypic continuity. Heterotypic continuity is defined as the “continuity of an inferred genotypic attribute presumed to underlie diverse phenotypic behaviors” (Caspi, 2006, pp. 349–50) or as “the manifestation of the same underlying process through different behavioral presentations at different developmental periods” (Cicchetti & Rogosch, 2002, p. 13). In other words, the capacity for inhibitory control may appear different in toddlers compared to older children. For example, in toddlers, observed inhibitory control may include using words rather than physical aggression to get a toy. For older children, it may include the ability to inhibit an action based on an abstraction despite a concrete command (in the case of the game, “Simon Says”) or despite social pressure (as for example when a peer invites a child to throw rocks at a window).

Measures of inhibitory control assess the capacity to withhold or delay a prepotent response or the capacity to perform a subdominant, yet correct response in the face of a conflicting and often prepotent (i.e., probabilistically dominant or habitual), yet incorrect response (Carlson & Moses, 2001). How does this important dimension of self-regulation develop? We think useful answers to this question will come from longitudinal studies. Longitudinal assessment of inhibitory control presents many challenges. The behavioral manifestation of inhibitory control changes across development, and measures must be valid for the developmental time frame of interest. Behavioral tasks, however, are likely to be appropriate for a restricted age range. The present meta-analytic review (1) determines the age ranges of usefulness for inhibitory control measures in childhood, (2) describes conceptual and methodological challenges in the longitudinal assessment of inhibitory control, and (3) offers recommendations for researchers studying the development of inhibitory control, with a hypothetical example of the longitudinal assessment of inhibitory control. Although this review focuses on the assessment of inhibitory control, the issues and recommendations may be common to the longitudinal assessment of many other domains and methodologies involving developmental change.

Measuring the development of inhibitory control: empirical considerations of age ranges of usefulness for inhibitory control measures

We first consider the age ranges for which common inhibitory control measures are useful for detecting individual differences in typically-developing children, focusing on measures of inhibitory control or response inhibition. Inhibitory control has been measured by response inhibition tasks that involve the withholding or delay of a prepotent response (for a review, see Garon et al., 2008). Consistent with Garon et al. (2008), we differentiate between simple response inhibition and complex inhibitory control paradigms. Simple response inhibition paradigms have minimal working memory demands, in which a child is given the opportunity to delay or the rule to inhibit a prepotent response. Examples of simple response inhibition tasks include delay of gratification (i.e., “Delay” tasks; Carlson & Moses, 2001) and “Don’t” paradigms. By contrast, complex inhibitory control (i.e., “Conflict” tasks; Carlson & Moses, 2001) paradigms have greater working memory demands because children are instructed not only to inhibit a prepotent response, but also to respond in a certain way to a salient, conflicting response option (Carlson & Moses, 2001; Garon et al., 2008). Here we focus on complex inhibitory control tasks involving conflict rather than delay (Carlson & Moses, 2001) – specifically those tasks that require remembering a rule and responding according to the rule rather than to the competing, prepotent response (Garon et al., 2008). We chose to focus on complex inhibitory control rather than simple response inhibition tasks because (a) complex inhibitory control tasks tend to assess behavioral accuracy whereas simple response inhibition tasks often assess duration of delay, (b) we expected greater developmental growth up to 8 years of age in complex than simple tasks (thus avoiding ceiling effects), and (c) simple response inhibition tasks have a different motivational component than complex inhibitory control tasks and likely have different antecedents (Rochette & Bernier, 2014).

Examples of complex inhibitory control tasks involving conflict include: Baby Stroop, Bear/ Dragon, Grass/Snow, Green-Red Signs, Day/Night, Hand Game, Knock-Tap, Less is More, Reverse Categorization, Shape Stroop, Simon Says, Spatial Conflict, and Tower (Carlson & Moses, 2001; Garon et al., 2008; Murray & Kochanska, 2002). Although the emphasis of the present review is on behavioral tasks, we also considered a questionnaire measure, the Inhibitory Control subscale of the Children’s Behavior Questionnaire (CBQ; Rothbart, Ahadi, & Hershey, 1994),¹ because of its relevance to the construct and because of the importance of incorporating multiple methods of assessment to estimate the amount of method variance (Strauss & Smith, 2009). Researchers have argued that, compared to structured lab tasks, naturalistic measures may more accurately reflect a child’s adaptive executive functioning in real life (Barkley, 2012). There are a number of temperament questionnaires of analogous constructs (e.g., Behavior Rating Inventory of Executive Function; Gioia, Isquith, Guy, & Kenworthy, 2000). For a point of comparison to behavioral measures of inhibitory control, we included the CBQ because it is very widely used. The CBQ measure of temperamental inhibitory control reflects caregivers’ impressions of child behavior across many relevant situations over long periods of time, unlike the structured lab tasks. Thus, the CBQ may be more consistent with a naturalistic approach to measurement of inhibitory control, and may be an important complement to behavioral tasks. See Table 1 for a description of these inhibitory control measures along with variants of each task that did not fundamentally alter the nature of the task. The table is not an exhaustive list of inhibitory control measures, but rather a list of some of the most prominent inhibitory control measures identified by Garon et al. (2008).²

Table 1.

Inhibitory control measures (behavioral tasks and one questionnaire measure) with descriptions and developmentally-appropriate and -sensitive age ranges.

Measure	Source	Description	Dependent variable	Variants	Useful age range
Baby Stroop	Hughes and Ensor (2005)	Child says (spoon/cup belongs to) “mommy” when E shows small spoon or cup, but says “baby” when E shows large spoon or cup.	Number of correct responses		24–36 (26–36)
Bear/Dragon	Kochanska et al. (1996), adapted from Reed, Pien, and Rothbart (1984)	Child follows directions from the bear puppet, but ignores commands from the dragon puppet.	Number of trials child does not move in response to inhibition (dragon) trials	Bear/Alligator, Bear/Dog, Bear/Elephant, Bear/ Lizard, Bird/Alligator, Bird/Dragon, Bunny/ Dragon, Dog/Dragon, Hedgehog/Snake, Horse/ Cow, Monkey/Dragon, Panda/Lion, Pig/Bull, Pig/Wolf, Policeman/Princess, Simon/Oscar, Squirrel/Badger	25–54 (33–54)
Day/Night	Gerstadt, Hong, and Diamond (1994)	Child says “day” when E shows picture of moon, “night” to picture of sun.	Number of correct responses	Similar to Grass/Snow, but involves verbal response: Sun/Moon, Car/Boat, Table/Chair, Fork/Knife, Cat/Dog, Happy/Sad, Fat/Thin, Bird/ Dragon, Yes/No, Boy/Girl, Big/Little, Up/Down, Red/Blue, Car/Book, Black/White, Mommy/Me	32–71 (33–71)
Grass/Snow	Carlson and Moses (2001)	Child points to white square when E says “grass,” green square when E says “snow.”	Number of correct responses	Similar to Day/Night, but involves pointing: Sun/Moon, Car/Boat, Table/Chair, Fork/Knife, Red/Blue, Rain/Snow, Mommy/Me	28–69 (30–69)
Green-Red Signs	Kochanska et al. (1997)	Child raises same hand as E (left or right) when E holds up green sign, child raises opposite hand as E when E holds up red sign.	Number of correct responses on inhibition (red) trials		52–71 (none)
Hand Game	Hughes (1996, 1998b), based on Luria, Pribram, and Homskaya (1964)	Child makes opposite gesture as E (fist or pointed finger).	Number of correct responses		29–63 (31–63)
Inhibitory Control (CBQ)	Rothbart et al. (13 items; 1994), Putnam and Rothbart (6 items; 2006)	“Capacity to plan and to suppress inappropriate approach responses under instructions or in novel or uncertain situations.” (Rothbart et al., 2001, p. 1406)	e.g., “Can lower his/ her voice when asked to do so.” (Rothbart et al., 2001, p. 1406)	6 items, 13 items	24–96 (26–96); designed for 36–84
Knock-Tap	NEPSY: Korkman, Kirk, and Kemp (1998), and Klenberg, Korkman, and Lahti-Nuuttila (2001)	Child taps table with open palm when E knocks table with fist, knocks table with fist when E taps table with open palm.	Number of correct responses	Child responds with an opposite action to one action of the E and does not respond at all to the second action of the E	36–61 (40–61)
Less is More	Carlson et al. (2005)	Child chooses between small and large trays of treats, child receives the treat tray not pointed to.	Number of small tray selections	Jelly Beans, chocolate chips, rocks, dots, mouse/elephant, stickers, marbles, goldfish, cereal. The game has been played with the child by themselves, in competition with an experimenter, on a team with an experimenter, or with a stuffed animal (monkey, bear). It has also been played with mouse/elephant stimuli instead of trays.	23–75 (35–71)
Reverse Categorization	Carlson et al. (2004)	Child sorts objects into two categories using a sorting rule (e.g. sorting red blocks into a blue bucket and blue blocks into a red bucket) that is opposite a prepotent sorting rule (e.g. sorting the red blocks into the red bucket and sorting the blue blocks into the blue bucket).	Number of correct responses	Goldfish/Teddy Grahams, mommy/baby animals, large/small objects, green apples/ green bananas, green pigs/green fish	18–60 (20–60)
Shape Stroop	Kochanska et al. (1997)	Child points to pictures of small objects embedded within pictures of larger, different objects.	Number of correct responses	Fruit, shapes, animals, letters, numbers	20–43 (20–43)
Simon Says	Strommen (1973)	One E gives commands. Child performs action only when command is preceded by “Simon says.”	Number of trials child does not move in response to inhibition (non- Simon) trials	E performs action, child faces away from E	58–86 (58–86)
Spatial Conflict	Gerardi-Caulton (2000)	Target stimuli (associated with different response keys) are presented on the left or right side of a monitor, and the child responds by selecting the correct key. In some trials the target stimulus is on the same side (left or right) as the correct key (congruent); on conflict trials the target is on the opposite side from the correct key (incongruent).	Number of correct incongruent trials	Cat/Dog	24–36(24–36)
Tower	Kochanska et al. (1996)	Child takes turns placing blocks to build a tower with the E. E waits to take turn until the child signals to give E a turn.	Number of turns child yields to E		23–56 (24–56)

Note: E = experimenter. Table adapted from Garon et al. (2008). Ages in months. Age ranges in parentheses reflect age ranges with stronger empirical support (two or more studies).

In September 2014, we reviewed studies using the inhibitory control measures we selected (or similar variants; Table 1). Table 2 lists the studies we identified, the inhibitory control measures that each examined, and, if the study was excluded from the meta-analysis, the reason for exclusion. Table 3 lists which studies examined each measure. We conducted a meta-analysis of these previous studies to estimate the age range for which each measure is useful, based on the measures’ average percentage-of-maximum (POM) score at each age (Little, 2013). We calculated POM for each study and inhibitory control measure by first scaling the measure’s metric to have a meaningful zero point (i.e., subtracting the minimum possible score from the metric). Then, to create a proportion score, we divided the zero-scaled observed mean score by the zero-scaled maximum possible score (and multiplied by 100 to convert it to a percentage score). For studies that dichotomized task scores into pass/fail, we calculated POM as the percentage of cases that were given a score of “pass.”

Table 2.

Studies reviewed using the inhibitory control measures from Table 1 (or similar variants).

Numeric index	Study year	Study authors	Inhibitory control measures	Exclusion criterion
1	1973	Strommen	SI	0
2	1977	Camp	SI	0
3	1981	LaVoie, Anderson, Fraze, & Johnson	SI	0
4	1984	Reed, Pien, & Rothbart	BD	0
5	1993	Ahadi, Rothbart, & Ye	IC	0
6	1994	Gerstadt, Hong, & Diamond	DN	0
7	1994	Rothbart, Ahadi, & Hershey	IC	2
8	1996	Diamond & Taylor	DN	0
9	1996	Hughes	HG	0
10	1996	Kochanska, Murray, Jacques, Koenig, & Vandegeest	BD, IC, TO	0, 0, 2
11	1996	Korkman, Liikanen, & Fellman	KT	4
12	1997	Kochanska, Murray, & Coy	GR, IC, SH, SI	0, 0, 0, 0
13	1998a	Hughes	HG	0
14	1998b	Hughes	HG	0
15	1998	Hughes, Dunn, & White	HG	0
16	2000	Berger, Jones, Rothbart, & Posner	SC	0
17	2000	Cole & Mitchell	BD	0
18	2000	Gerardi-Caulton	IC, SC, TO	2, 0, 2
19	2000	Kochanska, Murray, & Harlan	IC, SH, TO	0, 0, 2
20	2001	Carlson & Moses	BD, DN, GS, IC, SC, TO	0, 0, 0, 2, 0, 0
21	2001	Diamond	DN	1
22	2001	Floyd & Kirby	BD, TO	0, 0
23	2001	González, Fuentes, Carranza, & Estévez	IC	0
24	2001	Klenberg, Korkman, & Lahti-Nuuttila	KT	0
25	2001	Kochanska, Coy, & Murray	BD, GR, IC, SI, TO	2, 2, 0, 2, 2
26	2001	Rothbart, Ahadi, Hershey, & Fisher	IC	0
27	2002	Alexander, Goodman, Schaaf, Edelstein, Quas, & Shaver	DN, IC	2, 2
28	2002	Carlson, Moses, & Breton	BD	0
29	2002	Diamond, Kirkham, & Amso	DN	0
30	2002	Lang & Perner	HG	2
31	2002	Lengua	IC, SI	4, 4
32	2002	Murray & Kochanska	BD, GR, SH, SI, TO	1, 1, 1, 1, 0
33	2002	Perner, Kain, & Barchfeld	KT	0
34	2002	Sonuga-Barke, Dalen, Daley, & Remington	BD	0
35	2003	Blair & Peters	DN	2
36	2003	Deák & Narasimham	DN	0
37	2003	Fahie & Symons	HG	4
38	2003	Hala, Hug, & Henderson	DN	0
39	2003	Jones, Rothbart, & Posner	BD, IC	0, 2
40	2003	Kochanska & Knaack	BD, GR, SH, TO	2, 2, 2, 2
41	2003	Lengua	IC, SI	4, 4
42	2003	Peskin & Ardino	DN	0
43	2003	Rothbart, Ellis, Rueda, & Posner	SC	0
44	2003	Walker	BD, DN, GS, TO	0, 0, 2, 0
45	2004	Aksan & Kochanska	BD	2
46	2004	Blair, Peters, & Granger	IC	2
47	2004	Bohm, Smedler, & Forssberg	HG, KT	2, 2
48	2004	Carlson, Mandell, & Williams	BD, GS, HG, IC, RC, SH, TO	0, 0, 0, 0, 0, 0, 2
49	2004	Carlson, Moses, & Claxton	BD	0
50	2004	Dalen, Sonuga-Barke, Hall, & Remington	BD	2
51	2004	Flynn, O’Malley, & Wood	HG	0
52	2004	Jahromi	DN, IC	0, 0
53	2004	Joseph & Tager-Flusberg	DN, KT	3, 3
54	2004	Simpson, Riggs, & Simon	DN, GS	0, 0
55	2004	Wolfe & Bell	DN, IC	0, 0
56	2005	Berwid, Curko Kera, Marks, Santra, Bender, & Halperin	DN	0
57	2005	Carlson	BD, DN, GS, HG, LM, RC, SC, SH, SI, TO	0, 0, 0, 0, 0, 0, 0, 0, 0, 0
58	2005	Carlson, Davis, & Leach	BD, GS, LM, TO	2, 2, 0, 2
59	2005	Hughes & Ensor	BS	2
60	2005	Joseph, McGrath, & Tager-Flusberg	DN, KT	4, 4
61	2005	Lengua & Kovacs	IC	4
62	2005	Monks, Smith, & Swettenham	DN	2
63	2005	Olson, Sameroff, Kerr, Lopez, & Wellman	IC, TO	0, 0
64	2005	Pears & Fisher	DN	2
65	2005	Roberts & Powell	DN	0
66	2005	Roebers & Schneider	BD, DN, GS, HG	2, 2, 2, 2
67	2005a	Simpson & Riggs	DN	0
68	2005b	Simpson & Riggs	DN	0
69	2005	Tager-Flusberg & Joseph	DN, KT	3, 3
70	2006	Chasiotis, Kiessling, Hofer, & Campos	BD, DN, HG	0, 0, 0
71	2006	Chasiotis, Kiessling, Winter, & Hofer	BD, DN, HG	0, 0, 0
72	2006	Cromer, Stevens, DePrince, & Pears	KT	2
73	2006	Dennis	IC	0
74	2006	Hoff Esbjørn, Hansen, Greisen, & Mortensen	KT	0
75	2006	Hughes & Ensor	BS	2
76	2006	Livesey, Keen, Rouse, & White	DN	0
77	2006	Melinder, Endestad, & Magnussen	DN	4
78	2006	Pellicano, Maybery, Durkin, & Maley	HG	4
79	2006	Putnam & Rothbart	IC	0
80	2006	Sabbagh, Moses, & Shiverick	BD	0
81	2006	Sabbagh, Xu, Carlson, Moses, & Lee	BD, DN, GS, TO	2, 2, 2, 2
82	2006	Simpson & Riggs	DN	0
83	2006	Thorell & Wåhlstedt	DN	0
84	2007a	Bell & Wolfe	DN	1
85	2007b	Bell & Wolfe	DN	0
86	2007	Bernstein, Atance, Meltzoff, & Loftus	BD, DN	0, 0
87	2007	Bibok	SH, RC	0, 0
88	2007	Blair & Razza	IC	2
89	2007	Brocki, Nyberg, Thorell, & Bohlin	DN, KT	2, 2
90	2007	Carlson & Wang	SI	0
91	2007	Domitrovich, Cortes, & Greenberg	DN	0
92	2007	Flynn	BD, HG	0, 0
93	2007	Hinnant & O’Brien	DN	0
94	2007	Hrabok, Kerns, & Müller	SC	0
95	2007	Hughes & Ensor	BS, DN, HG	0, 0, 0
96	2007	Kochanska, Aksan, Penney, & Doobay	BD, GR, GS, SH, SI, TO	2, 2, 2, 2, 2, 2
97	2007	Lengua, Honorado, & Bush	BD, DN, GS	0, 0, 0
98	2007	Lewis, Dozier, Ackerman, & Sepulveda-Kozakowski	DN	0
99	2007	Marlow, Hennessy, Bracewell, Wolke, & The EPICure Study Group	KT	0
100	2007	Pellicano	HG	0
101	2007	Simpson & Riggs	DN	0
102	2007	Smith-Donald, Raver, Hayes, & Richardson	TO	0
103	2007	Tardif, So, & Kaciroti	DN, HG	0, 0
104	2007	Visu-Petra, Benga, & Miclea	KT	0
105	2007	von Stauffenberg & Campbell	DN	0
106	2007a	Wolfe & Bell	DN, IC	1, 2
107	2007b	Wolfe & Bell	BD, DN, IC	0, 0, 0
108	2008	Albertson & Shore	BD, GS	2, 0
109	2008	Bolnick	BD, IC, KT, SI	0, 0, 0, 0
110	2008	Brooker	TO	0
111	2008	Carlson & Meltzoff	IC, SI	2, 0
112	2008	Holmboe, Pasco Fearon, Csibra, Tucker, & Johnson	SC	0
113	2008	Hughes & Ensor	BS, DN, HG	2, 2, 2
114	2008	Jennings, Sandberg, Kelley, Valdes, Yaggi, Abrew, & Macey-Kalcevic	TO	2
115	2008	Karpinski, Scullin, & Montgomery-Downs	DN, HG	0, 0
116	2008	Kloo, Perner, Kerschhuber, Dabernig, & Aichhorn	RC	0
117	2008	Kochanska, Aksan, Prisco, & Adams	BD, GR, GS, TO	2, 2, 2, 2
118	2008	Leerkes, Paradise, O’Brien, Calkins, & Lange	DN	0
119	2008	Lunkenheimer, Dishion, Shaw, Connell, Gardner, Wilson, & Skuban	IC	0
120	2008	Montgomery, Anderson, & Uhl	DN	0
121	2008	Oh & Lewis	DN, GS, HG, TO	0, 0, 0, 2
122	2008	Putnam, Rothbart, & Gartstein	IC	0
123	2008	Ryan & Nelson	DN	2
124	2008	Sinn, Bryan, & Wilson	KT	4
125	2008	Talwar & Lee	DN	0
126	2008	Wellman, Lopez-Duran, LaBounty, & Hamilton	BD	0
127	2009	Beck, Riggs, & Gorniak	BD, DN	0, 0
128	2009	Bibok, Carpendale, & Müller	RC, SH	2, 2
129	2009a	Hughes & Ensor	BS	2
130	2009b	Hughes & Ensor	DN	0
131	2009	Hughes, Ensor, Wilson, & Graham	DN	0
132	2009	Ito	BD, GS, LM	0, 0, 0
133	2009	Kegel, van der Kooy-Hofland, & Bus	DN	0
134	2009	Kochanska, Barry, Jimenez, Hollatz, & Woodard	GR, GS, SS	2, 2, 2
135	2009	Lackner	GS, HG, LM	0, 0, 0
136	2009	Rhoades, Greenberg, & Domitrovich	DN	0
137	2009	Sabbagh, Bowman, Evraire, & Ito	BD, GS, LM	0, 0, 0
138	2009	Siegal, Iozzi, & Surian	DN	0
139	2009	Simpson & Riggs	DN, GS	0, 0
140	2009	Tregay, Gilmour, & Charman	HG	0
141	2010	Allan	GS, LM	1, 0
142	2010	Bernier, Carlson, Bordeleau, & Carrier	BS, SH	0, 0
143	2010	Bernier, Carlson, & Whipple	BS, SH	0, 0
144	2010	Bigham	HG	0
145	2010	Cutuli, Herbers, Rinaldi, Masten, & Oberg	SI	0
146	2010	Cutuli, Wiik, Herbers, Gunnar, & Masten	SI	4
147	2010	Dennis, Hong, & Solomon	IC	0
148	2010	Eisenberg, Vidmar, Spinrad, Eggum, Edwards, Gaertner, & Kupfer	IC	0
149	2010	Hayden, Klein, Sheikh, Olino, Dougherty, Dyson, Durbin, & Singh	TO	2
150	2010	Huyder	SI	1
151	2010	Kesek	IC	2
152	2010	Kloo, Perner, & Giritzer	DN	0
153	2010	Kolnik	DN, HG, SI	0, 2, 2
154	2010	Lackner, Bowman, & Sabbagh	GS, LM	0, 0
155	2010	Lane, Wellman, Olson, LaBounty, & Kerr	GR, SH, SI	2, 0, 0
156	2010	Molfese, Molfese, Molfese, Rudasill, Armstrong, & Starkey	KT	0
157	2010	Nampijja, Apule, Lule, Akurut, Muhangi, Elliott, & Alcock	KT	0
158	2010	Obradović	SI
159	2010	Pellicano	HG	3
160	2010	Prado, Hartini, Rahmawati, Ismayani, Hidayati, Hikmah, Muadz, Apriatni, Ullman, Shankar, & Alcock	KT, TO	2, 2
161	2010	Qu, Audrey, Jun, & Qun	LM	0
162	2010	Rakoczy	BD	0
163	2010	Sabbagh, Hopkins, Benson, & Randall Flanagan	GS	0
164	2010	Sulik, Huerta, Zerr, Eisenberg, Spinrad, Valiente, Di Gunta, Pina, Eggum, Sallquist, Edwards, Kupfer, Lonigan, Phillips, Wilson, Clancy-Menchetti, Landry, Swank, Assel, & Taylor	BD, IC, KT	0, 0, 0
165	2010	Willoughby, Blair, Wirth, & Greenberg	DN, SC	1, 1
166	2011	Allan & Lonigan	GS, LM	0, 0
167	2011	Baker, Gjersoe, Sibielska-Woch, Leslie, & Hood	BD, RC	0, 0
168	2011	Beck, Schaefer, Pang, & Carlson	RC	2
169	2011	Blair, Granger, Willoughby, Mills-Koonce, Cox, Greenberg, Kivlighan, & Fortunato	SC	0
170	2011	Chang, Olson, Sameroff, & Sexton	IC, TO	1, 1
171	2011	Cutuli	SI	1
172	2011	Drayton, Turley-Ames, & Guajardo	BD, GS	0, 0
173	2011	Evans, Xu, & Lee	BD, DN, TO	0, 0, 0
174	2011	Fuhs & Day	DN	0
175	2011	Geng, Hu, Wang, & Chen	IC	0
176	2011	Gleason, Fox, Drury, Smyke, Egger, Nelson, Gregas, &Zeanah	BD	2
177	2011	Gusdorf, Karreman, van Aken, Deković, & van Tuijl	IC, SH, TO	0, 0, 0
178	2011	Herbers, Cutuli, Lafavor, Vrieze, Leibel, Obradović, & Masten	SI	1
179	2011	Hughes, Ensor, & Marks	BS, DN	0, 0
180	2011	Kegel	DN	0
181	2011	King	BD	0
182	2011	Lagattuta, Sayfan, & Monsour	DN	0
183	2011	Matte-Gagné & Bernier	BS, BD, DN, SH	2, 2, 0, 2
184	2011	Monette, Bigras, & Guay	DN, KT	0, 0
185	2011	Olson, Lopez-Duran, Lunkenheimer, Chang, & Sameroff	IC, TO	1, 1
186	2011	Olson, Tardif, Miller, Felt, Grabell, Kessler, Wang, Karasawa, & Hirabayashi	DN, GS, HG	0, 0, 0
187	2011	Phillips, Ruhl, Montague, Gasparovic, Caprihan, Ohls, Schrader, & Lowe	BD	2
188	2011	Poulin-Dubois, Blaye, Coutya, & Bialystok	RC, SH	0, 0
189	2011	Qu	LM	0
190	2011	Raver, Jones, Li-Grining, Zhai, Bub, & Pressler	SI	2
191	2011	Reck & Hund	BD, DN, IC	4, 4, 4
192	2011	Rhoades, Greenberg, Lanza, & Blair	SC	1
193	2011	Sulik	BD, KT	0, 0
194	2011	Thompson	BD, DN	0, 0
195	2011	Wellman, Lane, LaBounty, & Olson	GR, SH, SI	1, 1, 1
196	2011	White, McDermott, Degnan, Henderson, & Fox	DN, GS, IC	0, 0, 0
197	2011	Wiebe, Sheffield, Nelson, Clark, Chevalier, & Espy	SH	0
198	2012	Bellagamba, Laghi, Lonigro, & Pace	RC	1
199	2012	Bernier, Carlson, Deschênes, & Matte-Gagné	BS, BD, DN, SH	1, 2, 0, 0
200	2012	Calderon, Angeard, Moutier, Plumet, Jambaqué, & Bonnet	KT	0
201	2012	Carroll, Riggs, Apperly, Graham & Geoghegan	GS	0
202	2012	Conway & Stifter	DN	0
203	2012	Cuevas, Hubble, & Bell	GS, IC	0, 0
204	2012	Duncan	DN	0
205	2012	Duvall	BD, RC	0, 0
206	2012	Dyson, Olino, Durbin, Goldsmith, & Klein	TO	2
207	2012	Ford, Driscoll, Shum, & Macaulay	BD, DN, HG	0, 0, 0
208	2012	Hammond, Müller, Carpendale, Bibok, & Liebermann-Finestone	BD, HG, RC, SH	2, 1, 1, 0
209	2012	Hardaway, Wilson, Shaw, & Dishion	IC	0
210	2012	Huyder & Nilsen	SI	4
211	2012	Kim & Kochanska	TO	2
212	2012	Lackner, Sabbagh, Hallinan, Liu, & Holden	GS, HG, LM	1, 1, 1
213	2012	Lahat, Degnan, White, McDermott, Henderson, Lejuez, & Fox	DN, GS	0, 0
214	2012	Masten, Herbers, Desjardins, Cutuli, McCormick, Sapienza, Long, & Zelazo	SI	0
215	2012	Monette	DN, KT	1, 1
216	2012	Müller, Liebermann-Finestone, Carpendale, Hammond, & Bibok	BD, HG, RC, SH	2, 0, 0, 0
217	2012	Nilsen & Graham	DN, GS	0, 0
218	2012	Röthlisberger, Neuenschwander, Cimeli, Michel, & Roebers	GS, HG, SI	2, 2, 2
219	2012	Sage	DN, GS	0, 0
220	2012	Smith, Sheikh, Dyson, Olino, Laptook, Durbin, Hayden, Singh, & Klein	TO	2
221	2012	Steele, Karmiloff-Smith, Cornish, & Scerif	SC	0
222	2012	van der Kooy-Hofland, van der Kooy, Bus, van Ijzendoorn, & Bonsel	DN	0
223	2012	Viterbori, Gandolfi, & Usai	TO	0
224	2012	Willoughby, Wirth, & Blair	DN, SD	2, 2
225	2012	Zalewski, Lengua, Fisher, Trancik, Bush, & Meltzoff	BD, DN, GS	0, 0, 0
226	2013	Allan, Lonigan, & Wilson	GS, IC, LM	4, 4, 4
227	2013	Bellagamba, Laghi, Lonigro, Pace, & Longobardi	RC	0
228	2013	Benson, Sabbagh, Carlson, & Zelazo	BD, GS	0, 0
229	2013	Bernier, Beauchamp, Bouvette-Turcot, Carlson, & Carrier	BS, SH	1, 1
230	2013	Brown, Ackerman, & Moore	BD, DN	0, 0
231	2013	Carranza, González-Salinas, & Ato	IC	0
232	2013	Caughy, Mills, Owen, & Hurst	GS, SH	0, 0
233	2013	Choe, Olson, & Sameroff	GR, IC, SH, SI, TO	1, 1, 1, 1, 1
234	2013	Choe, Lane, Grabell, & Olson	TO	1
235	2013	Cipriano-Essel, Skowron, Stifter, & Teti	DN, SH	2, 2
236	2013	Clark, Sheffield, Wiebe, & Espy	SH	0
237	2013	Eisenberg, Edwards, Spinrad, Sallquist, Eggum, & Reiser	BD, GS, IC, KT	0, 0, 1, 0
238	2013	Esposito, Baker-Ward, & Mueller	DN	0
239	2013	Evans & Lee	RC, SH	0, 0
240	2013	Kim, Nordling, Yoon, Boldt, & Kochanska	DN, GS, GR, TO	0, 0, 0, 2
241	2013	Kraybill	BD, DN, SH	0, 0, 0
242	2013	Kraybill & Bell	BD, DN	2, 2
243	2013	Kryski, Dougherty, Dyson, Olino, Laptook, Klein, & Hayden	TO	2
244	2013	Lane, Wellman, Olson, Miller, Wang, & Tardif	DN, GS, HG	0, 0, 0
245	2013	Leve, DeGarmo, Bridgett, Neiderhiser, Shaw, Harold, Natsuaki, Reiss	SH	0
246	2013	Lunkenheimer, Albrecht, & Kemp	TO	0
247	2013	Macaulay & Ford	DN, GS, HG	2, 2, 2
248	2013	Moran, Lengua, & Zalewski	BD, DN	1, 1
249	2013	Orta, Corapci, Yagmurlu, & Aksan	BD, DN	2, 2
250	2013	Pauli-Pott, Dalir, Mingebach, Roller, & Becker	BD, DN, TO	0, 0, 0
251	2013	Pellicano	HG	3
252	2013	Riggs, Jolley, & Simpson	BD	0
253	2013	Smith, Blake, & Harris	BD, DN	4, 4
254	2013	Smith, Kryski, Sheikh, Singh, & Hayden	TO	2
255	2013	Sulik, Eisenberg, Silva, Spinrad, & Kupfer	BD, KT	1, 0
256	2013	Thompson, Lengua, Zalewski, & Moran	BD, DN	1, 1
257	2013	Watson & Bell	DN, HG, IC, LM	0, 0, 0, 0
258	2013	Wyss	RC	0
259	2014	Allan & Lonigan	BD, DN, GS, KT, SH	4, 4, 4, 4, 4
260	2014	Calderon, Jambaqué, Bonnet, & Angeard	DN, HG	0, 0
261	2014	Causey & Bjorklund	BD, DN	0, 0
262	2014	Chien	DN, SI	0, 0
263	2014	Choe, Olson, & Sameroff	IC, TO	1, 1
264	2014	Choe, Shaw, Brennan, Dishion, & Wilson	IC	0
265	2014	Cuevas & Bell	BD, DN	0, 0
266	2014	Cuevas, Deater-Deckard, Kim-Spoon, Wang, Morasch, & Bell	BD, DN	0, 0
267	2014	Cuevas, Deater-Deckard, Kim-Spoon, Watson, Morasch, & Bell	BD, DN	1, 1
268	2014	Cutuli, Herbers, Lafavor, Ahumada, Masten, & Oberg	SI	1
269	2014	Earhart & Roberts	DN, HG	0, 0
270	2014	Fizke, Barthel, Peters, & Rakoczy	BD, DN, GS	0, 0, 0
271	2014	Herbers, Cutuli, Monn, Narayan, & Masten	SI	1
272	2014	Herbers, Cutuli, Supkoff, Narayan, & Masten	SI	1
273	2014	Gandolfi, Viterbori, Traverso, & Usai	BD, DN, RC, TO	0, 0, 0, 0
274	2014	Lengua, Kiff, Moran, Zalewski, Thompson, Cortes, & Ruberry	BD, DN	0, 0
275	2014	Lerner & Lonigan	BD, DN, HG, KT	0, 0, 0, 0
276	2014	Marshall & Drew	BD, SI	0, 0
277	2014	Mayer, Abelson, & Lopez-Duran	GR, IC, SH, SI, TO	1, 1, 1, 1, 1
278	2014	McClelland, Cameron, Duncan, Bowles, Acock, Miao, & Pratt	DN, SI	0, 0
279	2014	Pauli-Pott, Roller, Heinzel-Gutenbrunner, Mingebach, Dalir, & Becker	BD, DN, TO	1, 1, 1
280	2014	Skowron, Cipriano-Essel, Gatzke-Kopp, Teti, & Ammerman	DN, SH	0, 0
281	2014	Utendale, Nuselovici, Saint-Pierre, Hubert, Chochol, & Hastings	DN	0
282	2014	van Tilborg, Segers, van Balkom, & Verhoeven	GS	0
283	2014	Verkerk	TO	0
284	2015	Cheie, Veraksa, Zinchenko, Gorovaya, & Visu-Petra	KT	0
285	2015	Di Norcia, Pecora, Bombi, Baumgartner, & Laghi	RC	0
286	2015	Lengua, Moran, Zalewski, Ruberry, Kiff, & Thompson	BD, DN	1, 1
287	2015	Petersen, Bates, & Staples	BD, GS, SH	0, 0, 0
288	2015	Wiebe, Clark, De Jong, Chevalier, Espy, & Wakschlag	SH	0

BD = Bear/Dragon, BS = Baby Stroop, DN = Day/Night, GR = Green-Red Signs, GS = Grass/Snow, HG = Hand Game, IC = Inhibitory Control subscale of CBQ, KT = Knock-Tap, LM = Less is More, RC = Reverse Categorization, SH = Shape Stroop, SI = Simon Says, SC = Spatial Conflict, TO = Tower.

Note: “Inhibitory Control Measures” column lists the inhibitory control measures from Table 1 that were examined by each study. “Exclusion Criterion” denotes the reason for exclusion (the exclusion criterion number), if the study was excluded from the meta-analysis for each of the respective inhibitory control measures (0 if the study was included). Study exclusion criteria were: 1 = having identical data to that of another included study, 2 = not reporting the information necessary to determine POM (after contacting authors), 3 = including only a clinical sample, and 4 = providing a single mean score for a sample of children whose ages spanned more than 3 years.

Table 3.

Studies examining each inhibitory control measure (or similar variants).

Measure	No. of included studies	No. of excluded studies	Numeric index of included studies (from Table 2) examining measure	Excluded studies
Baby Stroop	4	7	95, 142, 143, 179	59, 75, 113, 129, 183, 199, 229
Bear/Dragon	50	28	4, 10, 17, 20, 22, 28, 34, 39, 44, 48, 49, 57, 70, 71, 80, 86, 92, 97, 107, 109, 126, 127, 132, 137, 162, 164, 167, 172, 173, 181, 193, 194, 205, 207, 225, 228, 230, 237, 241, 250, 252, 261, 265, 266, 270, 273, 274, 275, 276, 287	25, 32, 40, 45, 50, 58, 66, 81, 96, 108, 117, 176, 183, 187, 191, 199, 208, 216, 242, 248, 249, 253, 255, 256, 259, 267, 279, 286
Grass/Snow	31	13	20, 48, 54, 57, 97, 108, 121, 132, 135, 137, 139, 155, 164, 167, 172, 186, 196, 201, 203, 213, 217, 219, 225, 228, 232, 237, 240, 244, 270, 282, 287	44, 58, 66, 81, 96, 117, 134, 141, 212, 218, 225, 247, 259
Green-Red Signs	2	10	12, 240	25, 32, 40, 96, 117, 134, 155, 195, 233, 277
Day/Night	85	31	6, 8, 20, 29, 36, 38, 42, 44, 52, 54, 55, 56, 57, 65, 67, 68, 70, 71, 76, 82, 83, 85, 86, 91, 93, 95, 97, 98, 101, 103, 105, 107, 115, 118, 120, 121, 125, 127, 130, 131, 133, 136, 138, 139, 152, 153, 173, 174, 179, 180, 182, 183, 184, 186, 194, 196, 199, 201, 204, 207, 213, 217, 219, 221, 225, 230, 238, 240, 241, 244, 250, 257, 260, 261, 262, 265, 266, 269, 270, 273, 274, 275, 278, 280, 281	21, 27, 35, 53, 60, 62, 64, 66, 69, 77, 81, 84, 89, 106, 113, 123, 165, 191, 215, 224, 235, 242, 247, 248, 249, 253, 256, 259, 261, 279, 286
Hand Game	26	13	9, 13, 14, 15, 48, 51, 57, 70, 71, 92, 95, 100, 103, 115, 121, 135, 140, 144, 186, 207, 216, 244, 257, 260, 269, 275	30, 37, 47, 66, 78, 113, 153, 159, 208, 212, 218, 247, 251
Inhibitory Control (CBQ)	28	21	5, 10, 12, 19, 23, 25, 26, 48, 52, 55, 63, 73, 79, 107, 109, 119, 122, 147, 148, 164, 174, 177, 196, 203, 209, 231, 257, 264	7, 18, 20, 27, 31, 39, 41, 46, 61, 88, 106, 111, 151, 170, 185, 191, 226, 233, 237, 263, 277
Knock-Tap	16	11	24, 33, 74, 99, 104, 109, 156, 157, 164, 184, 193, 200, 237, 255, 275, 284	11, 47, 53, 60, 69, 72, 89, 124, 160, 215, 259
Less is More	11	2	57, 58, 132, 135, 137, 141, 154, 161, 166, 189, 257	212, 226
Reverse Categorization	13	4	48, 57, 87, 116, 167, 188, 205, 216, 227, 239, 258, 273, 285	128, 168, 198, 208
Shape Stroop	22	11	12, 19, 48, 57, 87, 142, 143, 155, 177, 188, 197, 199, 208, 216, 232, 236, 239, 241, 245, 280, 287, 288	32, 40, 96, 128, 183, 195, 229, 233, 235, 259, 277
Simon Says	15	20	1, 2, 3, 12, 57, 90, 109, 111, 145, 155, 158, 214, 262, 276, 278	25, 31, 32, 41, 96, 134, 146, 150, 153, 171, 178, 190, 195, 210, 218, 233, 268, 271, 272, 277
Spatial Conflict	9	3	16, 18, 20, 43, 57, 94, 112, 169, 221	165, 192, 224
Tower	15	27	20, 22, 32, 44, 57, 63, 102, 110, 173, 177, 223, 246, 250, 273, 283	10, 18, 19, 25, 40, 48, 58, 81, 96, 114, 117, 121, 149, 160, 170, 185, 206, 211, 220, 233, 234, 240, 243, 254, 263, 277, 279

Note: See Table 2 for reasons why, for each measure, studies were excluded from the meta-analysis.

Inclusion criteria for studies in the meta-analysis included: (1) the study examined one or more of the inhibitory control measures from the list in Table 1 and (2) the sample included children younger than 8 years of age because inhibitory control develops most rapidly at younger ages (Diamond, 2002). Exclusion criteria included: (1) having identical data to (or fully subsumed within) that of another included paper, (2) not reporting POM or the information necessary to compute POM (mean performance and possible minimum and maximum) for unstandardized behavioral accuracy (not latency) on the inhibition trials of the individual task (as opposed to a composite of multiple measures or trial types), (3) including only a clinical sample, or (4) providing a single mean score for a sample of children whose ages spanned more than 3 years. We focused on scores on inhibition trials only (not go/ activation trials) in order to ensure the scores were on a conceptually similar metric relating to inhibitory control. We excluded clinical subsamples from the full study sample if all children in the subsample had a clinical disorder, had low birth weight, or were born prematurely; however, we included control subsamples from these studies. We excluded studies providing a single mean score for ages spanning more than 3 years in order to have fine-grained estimates of developmental progression and because inhibitory control tasks have limited age ranges of utility (McClelland & Cameron, 2012). To avoid a systematic missingness bias, there was one exception to exclusion criterion 2: if a study reported that the task showed a floor or ceiling effect but did not report the actual POM, we imputed a POM of 10% or 90%, respectively.

For one task, Tower, widely discrepant scoring procedures were used across studies (e.g., some used bonus points for self-correction or careful placement of blocks and negative points for knocking down the tower). To make the POM estimates for Tower conceptually comparable across studies, we excluded studies (exclusion criterion 2) that altered the basic scoring metric (number of trials the child yields to the experimenter) by adding or subtracting points for other behaviors. We describe these limitations of the Tower task in greater detail later.

To include as many studies as possible, we contacted authors of studies that did not report POM requesting the mean and possible minimum and maximum. We identified 288 studies that included on average 1.83 inhibitory control measures that we considered, yielding 528 study-by-measure combinations that met our inclusion criteria.³ Ninety (31%) studies did not report POM estimates or the descriptive statistics necessary – mean and possible minimum and maximum – to calculate POM estimates for one or more inhibitory control measures (158, 30% study-by-measure combinations). We contacted authors of these studies to obtain the necessary descriptive statistics to calculate POM. After authors’ responses to our requests, 61 (12%) study-by-measure combinations were excluded for identical data, 107 (20%) were excluded for lacking the necessary descriptive statistics to determine POM, 6 (1%) were excluded for including only a clinical sample, and 27 (5%) were excluded for providing a single average for a sample of children whose ages spanned more than 3 years. The meta-analysis included the remaining 198 (68%) studies and 327 (62%) study-by-measure combinations.

Because some studies provided different POM estimates for multiple subgroups (e.g., 3- versus 5-year-olds), we calculated average POM at each age for each subgroup separately. We calculated an average of the subgroups’ POM at each age, weighted by each subgroup’s sample size, with higher POM scores reflecting more optimal performance. We used the age ranges of each subgroup for determining the ages to which each subgroup’s POM score applied. If subgroup age ranges were not reported, age ranges were estimated from the interval including ±1 standard deviation of the subgroup’s mean age. Our estimates of the useful age ranges were intentionally conservative. We assume a task is useful at a particular age if the POM at that age is between 20% and 80% (to prevent floor and ceiling effects). The useful age range was selected as the age range during which the POM was greater than 20% and less than 80%, after which 20% POM was last surpassed and before which 80% was first reached. The inhibitory control measures and their useful age ranges are in Table 1. Based on the measures’ useful age ranges, we grouped the measures into early, middle, and late measures. The average POM at each age is depicted in Figs. 1, 2, and 3 for the early, middle, and late measures, respectively.

Fig. 1.

Children’s average percentage-of-maximum score on “early” inhibitory control measures by age. Gray lines represent age ranges with weaker empirical support (i.e., fewer than two studies; line is interpolated if there are no studies at a given age). Dashed gray horizontal lines at 20% and 80% reflect our low and high cutoffs, respectively, for defining the useful age range of each task.

Fig. 2.

Children’s average percentage-of-maximum score on “middle” inhibitory control measures by age. Gray lines represent age ranges with weaker empirical support (i.e., fewer than two studies; line is interpolated if there are no studies at a given age). Dashed gray horizontal lines at 20% and 80% reflect our low and high cutoffs, respectively, for defining the useful age range of each task.

Fig. 3.

Children’s average percentage-of-maximum score on “late” inhibitory control measures by age. Gray lines represent age ranges with weaker empirical support (i.e., fewer than two studies; line is interpolated if there are no studies at a given age). Dashed gray horizontal lines at 20% and 80% reflect our low and high cutoffs, respectively, for defining the useful age range of each task

Several patterns are notable. First, in general and as expected, performance on inhibitory control measures improved with age. Second, performance on some tasks reached plateaus at later ages (beginning in some cases around 4 years of age). Researchers should therefore be thoughtful about the inhibitory control tasks they use, especially at later ages (4–8 years), and might consider ways to avoid ceiling effects to make the tasks usable for a longer developmental span (discussed later). Third, unexpectedly, the Shape Stroop, Bear/Dragon, and Tower tasks showed a pronounced drop in performance from about 25 to 30 months of age. We are hesitant to infer that inhibitory control abilities declined with development over this span. Speculatively, the drop in performance could owe to differences in systematic missingness rates. Assuming children’s performance remains relatively stable over this span, decreases in systematic missingness with age (because of lower noncompliance, better verbal comprehension, etc.) could result in more of the least-skilled children completing the task at later ages, resulting in lower average inhibitory scores.

Fig. 4 depicts the useful age range for each measure. Examination of Fig. 4 suggests that the measures have fairly limited age ranges of utility. Whereas the Inhibitory Control questionnaire subscale was useful for 6 years, behavioral tasks were useful for 2.49 years, on average. However, this does not necessarily mean that measures may not prove useful outside the age ranges suggested ranges suggested ranges suggested – we emphasize that the suggested age ranges are based on the POM and age ranges reported in prior studies with normative populations. Some of the measures’ POM did not exceed our 20% threshold for floor effects and our 80% threshold for ceiling effects (e.g., Less is More). This indicates that relatively few studies have examined the extremes of low/high performance, and that collectively, studies have not examined the full range of the relevant ages for many inhibitory control measures. As a result, the actual useful age ranges may extend to younger or older ages than so far have been examined in prior studies. Another limitation is that POM may not fully account for whether a task is useful at a given age, especially at younger ages, because POM does not capture the amount of systematic missingness. Some children may not complete a task at younger ages because they lack a skill required by the task (or for another reason that might correlate with inhibitory control ability, such as noncompliance or poor receptive language skills), resulting in systematic missingness that would artificially inflate the mean score and POM. In these cases, however, the useful age ranges would be even more abbreviated than our estimates. Thus, the suggested age ranges are provisional. Future studies will refine the useful age ranges for inhibitory control and other self-regulation measures. We also recognize that the age ranges we are reporting are based on analysis of the measures’ sensitivity to detect individual differences in behavioral accuracy. There are ways to make measures relevant to a wider range of ages, such as by measuring reaction times, increasing rule difficulty, adding rule switches, and decreasing the amount of time a child has to respond.

Fig. 4.

Depiction of useful age ranges of inhibitory control measures based on age ranges from Table 1 and Fig. 1. Dotted lines represent age ranges with weaker empirical support (fewer than two studies). CBQ= Children’s Behavior Questionnaire

Comparison of task demands of inhibitory control measures

As a further differentiation of the inhibitory control tasks, we grouped the tasks into four a priori clusters based on the task’s demand: (a) perceptual inhibition, (b) performance inhibition, (c) association inhibition, and (d) motivational inhibition. Perceptual inhibition included tasks that involve the inhibition of perceptual information, such as Baby Stroop, Shape Stroop, and Spatial Conflict that require inhibition involving one of the following dimensions: size (Baby Stroop, Shape Stroop) and location (Spatial Conflict). Performance inhibition included tasks that use a cue (person or puppet) to indicate whether or not to inhibit a prepotent behavioral response, such as the activation/go trials versus inhibition/no-go trials of Bear/Dragon and Simon Says. Association inhibition included tasks similar to performance inhibition tasks but required inhibition of a prepotent response and the generation of a competing response on all trials, such as Day/Night, Grass/Snow, Green-Red Signs, Hand Game, Knock-Tap, and Reverse Categorization. Motivational (or “hot”) inhibitory control tasks included tasks with a stronger motivational component, such as Less is More (correct choices yield more winnings) and Tower (children tend to prefer to take more turns). Other researchers have distinguished perceptual inhibition (Martin-Rhee & Bialystok, 2008) and “cold” versus “hot” inhibition (Zelazo & Carlson, 2012). However, we know of no groups that have distinguished tasks conceptually based on whether inhibition is required on a subset of trials and the generation of a competing response on the other subset of trials (i.e., performance inhibition) or whether inhibition and the generation of a competing response are required on all trials (i.e., association inhibition). We see the distinction between performance and association inhibition as speculative but possibly useful. We recognize that all measures tap multiple aspects of inhibitory control (e.g., association inhibition tasks also require perceptual inhibition), so our categorization, though a useful early step, should not imply mutually exclusive clusters. Nevertheless, although grouping the tasks into clusters of common task demands is an oversimplification, doing so allows us to compare the developmental rates of different aspects of inhibitory control.

The task demands within and across categories of inhibitory control appear to differ in their useful age ranges and rates of growth. These can inform our understanding of the construct of inhibitory control and the task demands that elicit greater levels of inhibition. The perceptual inhibition tasks generally showed the youngest useful age range, suggesting that perceptual inhibition develops earlier than other aspects of inhibitory control. Among the perceptual inhibition tasks, tasks involving multiple sets of stimuli (e.g., fruit, banana, and apple in Shape Stroop) may be more difficult than tasks involving one set of stimuli (e.g., cup/spoon in Baby Stroop). Among the performance inhibition tasks, Simon Says was more challenging, i.e., was useful at later ages than Bear/Dragon. This may be because Simon Says typically involves a single cue source (the same person provides cues for both inhibition and activation trials), whereas Bear/Dragon involves two different cue sources (bear for activation trials and dragon for inhibition trials), consistent with findings of Marshall and Drew (2014) who experimentally tested variations of Simon Says with one versus two cue sources. In addition, the nature of the cue source may make Simon Says more difficult because inhibiting a response to an adult may be more difficult than inhibiting a response to an inanimate object, possibly because children are more eager to comply with adult requests.

Among association inhibition tasks, Day/Night and Grass/Snow are conceptually similar and require the inhibition of a pre-established prepotent response; in each, the two stimuli have a widely known association (e.g., day is the opposite of night). Providing construct validation of their conceptual similarity, Day/Night and Grass/Snow had similar useful age ranges. They were somewhat more challenging than other association inhibition tasks including Hand Game and Knock-Tap that used an arbitrary association (e.g., tap when the experimenter knocks) rather than a pre-established association (say “day” when the experimenter points to a picture of a moon). The motivational inhibitory control task, Less is More, showed a later useful age range than other task groupings, suggesting that motivational inhibitory control tasks may tap later-developing affective processes. This was not the case with the Tower task, however, which may reflect methodological limitations of the task (discussed later). In sum, more affective/motivational components, multiple sets of stimuli, a single cue source of information reversal, an adult cue source, and inhibiting a pre-established prepotent response appear to require greater levels of inhibitory control. Further, some aspects of inhibitory control such as perceptual inhibition may develop earlier than other aspects of inhibitory control such as performance and association inhibition, which may develop earlier than motivational inhibition.

The questionnaire measure of inhibitory control, from the CBQ, showed comparatively little sensitivity to mean-level developmental change (see Fig. 3). The Inhibitory Control subscale of the CBQ would be useful when one’s goal is to examine temperamental individual differences in inhibitory control rather than group-level developmental change in inhibitory control over time. Although studies used the Inhibitory Control subscale of the CBQ across 2–8 years of age (and it was useful across this span according to our definition), it was only designed for children aged 3–7 years. Thus, consistent with the heterotypic continuity of inhibitory control, we suggest using the CBQ only for 3- to 7-year-old children for whom it was designed.

Methodological limitations of prior studies

We also noted common limitations across a number of studies. First, many studies included a wide range of ages in the same subgroup. A wide age range in a subgroup might render study inferences less valid because most of the tasks appear to have a fairly limited age range of utility, and as we describe later, the same task may not have construct validity at different ages. Second, many study-by-measure combinations (N = 160, 31%) did not report the basic descriptive statistics necessary in the manuscript (mean and possible minimum and maximum) to determine POM for the inhibitory control measure. Many studies reported descriptive statistics only on the composite measures, rather than on the individual tasks. We suggest that future studies report unstandardized descriptive statistics on behavioral performance (mean, standard deviation, and possible and observed minimum and maximum) even when using a composite of multiple measures (e.g., as was done by Cuevas & Bell, 2014). These basic descriptives are necessary for understanding the metric of task scores and for understanding the sample’s performance on this metric. Third, many studies dichotomized task performance into pass/fail. While sometimes appreciating the simplicity of pass/fail, we advise reporting on a continuous metric as possible to retain data precision and have higher statistical power to detect developmental change and associations with other variables.

Fourth, we noted obscure and inconsistent scoring procedures across studies of the same task. For instance, in the Tower task, children are asked to take turns placing blocks to build a tower with the experimenter. The experimenter waits to take a turn until the child signals that he or she is giving the experimenter a turn, with the dependent variable often being the number of turns the child yields to the experimenter. The assumption is that the “optimal score” would be to yield half of the turns to the experimenter. Thus, for 20 turns, an optimal score would be the child yielding 10 turns to the experimenter. This optimal score (10) is not the maximum possible score (20), however, making the interpretation of the metric difficult. Moreover, some studies further complicate this metric by adding bonus points for self-correction and careful placement, and subtracting points for knocking over the tower. Combining all of these measures puts the final score on a metric with no clear optimal/ maximum score. While sometimes appreciating creative scoring procedures, for the sake of allowing better cross-study comparisons, we advise reporting clear scoring procedures with a meaningful range of possible values.

Implication of heterotypic continuity for measurement

In addition to empirical considerations for the measurement of inhibitory control, there are theoretical and psychometric considerations as well. Theoretical discussions of the development of self-regulation often invoke the concept of heterotypic continuity (Kopp, 1982; Raffaelli, Crockett, & Shen, 2005). Heterotypic continuity is defined as the “continuity of an inferred genotypic attribute presumed to underlie diverse phenotypic behaviors” (Caspi, 2006, pp. 349– 50) or as “the manifestation of the same underlying process through different behavioral presentations at different developmental periods” (Cicchetti & Rogosch, 2002, p. 13). In other words, inhibitory control and self-regulated behavior more generally, although continuous in its underlying purpose or function, may appear different in toddlers than it does in older children and adolescents. Self-regulation and inhibitory control change over development from early reliance on external sources for control to ability for internal, self-initiated forms of control (Berger, 2011; Calkins & Howse, 2004; Kopp, 1982). Acceptance of the notion of heterotypic continuity has two main implications for measurement.

The first implication is that the same measure may or may not reflect the same construct at different ages (Widaman, Ferrer, & Conger, 2010). Across broad ranges of development, a given measure may not be developmentally appropriate or construct-relevant at all ages. Similarly, even if a measure taps the same construct at different ages, it may not be sensitive enough to detect variability at each age (Carlson, 2005) – it may not be developmentally sensitive across different ages. If the underlying construct changes in its manifestation depending on the stage of development, scores on the same measure across time may reflect differences in the meaning of the measure, rather than real change in the construct.

For example, our meta-analysis indicated that the Bear/Dragon task is most useful (with stronger confidence) from 33 to 54 months of age, and may therefore have a different meaning outside that developmental window. Anecdotally, in testing children longitudinally on a variant of the Bear/ Dragon task (Bird/Alligator, methods described in detail elsewhere; Petersen, Bates, & Staples, 2015), we have observed that children at 30 months of age who inhibited responses on the no-go trials also tended to inhibit on the go trials (i.e., performance on go and no-go trials was inversely correlated: r[137] = −.67). However, this was not as strongly the case at 36 (r[133] = −.22) and 42 (r[127] = .12) months (Fisher r-to-z values significant at p < .001 level). Thus, there appears to be a developmental change in the meaning of the Bear/Dragon task, where performance on the no-go trials has a different meaning at 30 months than at later ages. If this is the case, then scores on the task at 30 months would not be conceptually comparable with scores at 42 months. This developmental shift may be consistent with our previous findings that the no-go score on the Bird/Alligator task was associated with another inhibitory control measure, Shape Stroop, at 42 months but not at 30 months (Petersen et al., 2015). We plan to examine whether no-go inhibition on the Bird/Alligator task reflects more motivational or affective inhibition at 30 months and more executive inhibition at later ages. Some researchers have tried to ensure that higher inhibition scores reflect better performance on both go and no-go trials by multiplying the go and no-go scores (Eisenberg et al., 2013). However, if the meaning of the no-go trials changes with development, the composite would change in meaning, too. In sum, to make inferences regarding developmental change, we must use equivalently-functioning and construct-valid measures across time.

The second implication is that describing the heterotypic continuity of inhibitory control may require using different measures across time (Widaman et al., 2010) because children are expected to demonstrate different behaviors at different ages for the same underlying construct of self-regulation (Bates & Novosad, 2005). When theoretically sound, it is preferable to maintain the same measures over time to ensure comparability (Willett, Singer, & Martin, 1998). However, there are also occasions when studies of development benefit from different measures over time (McArdle, Grimm, Hamagami, Bowles, & Meredith, 2009). Ideally, longitudinal assessment of inhibitory control should accommodate changes in the manifestation of inhibitory control by using different measures (when necessary) of the same construct over time, with careful consideration of developmental appropriateness, developmental sensitivity, and construct validity across the time frame of study. In the following, we consider these methodological aims in detail.

Developmental appropriateness and sensitivity

By developmentally appropriate, we refer to measures that are consistent with the capacity of most children of a particular developmental level and with our understanding of the construct at that point in development. A measure could be considered developmentally appropriate if its scores cluster around a value close to the center of the plausible range. If a task is not developmentally appropriate for a given age, a score on the task at that age is meaningless for describing development in terms of the construct of interest. By developmentally sensitive, we refer to measures that yield enough variability to be useful for assessing individual differences within age and to compare scores across ages. A measure could be considered developmentally sensitive if there are large inter-individual differences around the mean (and not ceiling or floor effects). Developmental appropriateness and sensitivity help ensure proper measurement of each child’s inhibitory control and adequate spread between individuals. It is common for behavioral tasks to be developmentally appropriate and sensitive at one age, but too easy for participants at later ages, resulting in ceiling effects. It is also likely that a given task would be too hard for younger children (although this is not represented in publications, as seen in Figs. 1–3). Ceiling and floor effects prevent the researcher from detecting meaningful variability in the construct and result in restricted range, which increases Type II error by reducing power to detect change and associations with other variables. One way to prevent ceiling and floor effects may be to use a more precise metric for the dependent variable (e.g., reaction time in milliseconds): a continuous interval or ratio metric as opposed to an ordinal or nominal metric. Even if no differences may be observed in behavioral accuracy because of ceiling effects, investigation of the same task with a dependent variable of a higher precision metric may detect developmental differences that make the task useful across a wider age range.

Developmental appropriateness and sensitivity depend, however, on the population of interest. For instance, the assessment of clinically significant deficits in inhibitory control in a high-risk sample should discriminate at lower ability levels (i.e., lower difficulty in item response theory; Krueger et al., 2004) rather than only at normative levels. Thus, clinical assessment may involve tasks that are mastered by most typically-developing children of that age and that would yield ceiling effects if used in a normative sample (Carlson, 2005).

Behavioral tasks tend to be developmentally appropriate and sensitive for restricted ages of children, as demonstrated earlier (see Fig. 4), so most inhibitory control tasks are not valid or useful across a wide age range. In order to cover a wider age range, researchers can use different behavioral tasks at different ages to ensure construct validity, developmental appropriateness, and developmental sensitivity for each age studied. Researchers have noted the importance of investigating heterotypic continuity with changes in measures (Schulenberg & Maslowsky, 2009; Widaman et al., 2010).

Changing measures across time

Using different tasks poses a problem from a longitudinal perspective. One goal in longitudinal studies is to describe continuity and change. Developmental inferences are strengthened by establishing measurement invariance⁴ (equivalent measures), ensuring that differences over time reflect change in the phenomenon of interest, not simply changes in measurement. As explained, inhibitory control changes in its manifestation with development and most behavioral tasks have restricted age windows of usefulness, so longitudinal assessment often requires different behavioral tasks for different ages. The use of different measures at different ages violates measurement invariance in the strict sense (same measures, same meaning) and calls into question the comparability of scores across time. Yet in order to maintain construct validity invariance, i.e., heterotypic continuity, theory requires that we accommodate measures to account for changes in the manifestation of a construct.

Potential for using the same measure across time

In light of the restricted time frame of usefulness in behavioral tasks, researchers may be inclined to implement other assessment techniques that would allow comparability across time. There are ways to make behavioral tasks relevant to a wider range of ages, such as by increasing rule difficulty, adding rule switches, and decreasing the amount of time a child has to respond (for a discussion, see Carlson, Faja, & Beck, 2015). However, as discussed earlier, the same task could lack construct validity invariance if it does not reflect the heterotypic continuity of the construct. Another possibility is the use of informant ratings, commonly completed by a parent, teacher, or observer. However, some of the same problems with longitudinal behavioral assessment are present with longitudinal informant ratings. Given that inhibitory control changes in its manifestation with development, in order for a questionnaire to have construct validity, its items – each of which reflects explicitly or implicitly a task or social context – would have to change with age (see, e.g., the widely used Rothbart temperament questionnaires and the Achenbach System of Empirically Based Assessment; Achenbach, 2009; Rothbart, Ahadi, Hershey, & Fisher, 2001). For more information on how to equate questionnaire measures with different item sets, the reader is referred to examples using integrative data analysis (Curran & Hussong, 2009; Curran et al., 2008), proportional scoring metrics (Petersen, Bates, Dodge, Lansford, & Pettit, 2015), and other scaling techniques (Kolen & Brennan, 2014). In practice, however, many investigators use questionnaires across multiple ages, including ages for which the questionnaire was not initially designed. Although the individuals’ scores may be on the same mathematical metric from year to year, for example, if they were z-scores, the metrics may not be comparable across ages because the questionnaire may not assess the same construct at each age due to changes in the manifestation of the construct (Widaman et al., 2010). The scores may not be on the same conceptual metric.

Another approach for longitudinal measures may be the use of measures with a higher precision metric, including reaction time and psychophysiological techniques, which purportedly place all individuals on the same metric for equivalent comparison within and across ages. One example is the use of event-related potentials (ERPs) as measured by electroencephalography. Many ERP components are present across wide age ranges, suggesting that amplitudes and latencies of ERP components can be compared across ages. However, there are important developmental changes in ERP components (Hoyniak, 2016), related, in part, to changes in brain size and skull thickness (Rueda, Posner, & Rothbart, 2005). Thus, there is question whether developmental changes in ERP components reflect actual changes in cognitive processes over development, and whether ERPs can truly be compared across ages. In addition to physical changes that pose challenges for longitudinal ERP comparisons, the same task may have different task demands (and elicit different cognitive processes) at different ages. Researchers have also questioned the utility of comparing neural activation across time as measured by functional magnetic resonance imaging, because age differences could reflect technical aspects of data acquisition rather than developmental change (Gaillard, Grandin, & Xu, 2001).

In the case of reaction time, there is also reason to be cautious. Anderson, Nettelbeck, and Barlow (1997) found that faster reaction times among 11- than 7-year-old children in a simple stimulus pairing task were related to differences in the speed of response selection rather than speed of information processing, suggesting that differences across time in reaction time may not be meaningful in terms of processing speed. Thus, measures with a higher precision metric such as reaction time and psychophysiological measures may not be on the same conceptual metric across time, and therefore may not be fully comparable across development. Although other measures and methods may be useful, they should supplement but not replace behavioral measures in the assessment of inhibitory control (see, e.g., the importance of behavioral measures when interpreting ERPs; Fox, Hane, & Pérez-Edgar, 2006).

Measuring the development of inhibitory control in a piecewise fashion

Given the complexities of measuring the development of inhibitory control over a long time frame, why not just measure the development of inhibitory control in a shorter, piecewise manner? First, most of the widely used measures of inhibitory control are developmentally appropriate or sensitive for less than 3 years (see Fig. 4), so relying on a fixed set of tasks would severely restrict our ability to see growth over a longer span. Second, heterotypic continuity is a developmental complexity that arises in many different domains. Seeking to understand and predict changes across important developmental periods and transitions is better than ignoring the phenotypic complexities associated with meaningful developmental change. Third, the ultimate goal of developmental psychology is to understand the whole trajectory of an individual’s life, and not just transitory outcomes at a particular point in life. Thus, research should strive to build a bridge that spans from childhood to adulthood (Rutter & Sroufe, 2000). The following recommendations seek to improve our ability to build a stable bridge spanning development.

One promising approach to changing measures over time is an overlapping measurement approach, similar to the approaches implemented by Patterson (1993) and Pettit, Keiley, Laird, Bates, and Dodge (2007). A staggered, overlapping measures approach is akin to accelerated longitudinal designs (also known as cohort-sequential or mixed longitudinal designs; Willett et al., 1998) except that the measures stagger and overlap, rather than the participant cohorts. The following section describes this approach in further detail.

Recommendations for longitudinal measurement designs

We have done longitudinal research, and we are aware of the limitations. So we are seeking to improve our work, and have increasingly focused on seven ways to improve longitudinal research on children’s inhibitory control and its implications. Listed in the order that one might approach them in designing research, they include (1) choose the construct of interest; (2) define the age range of interest and frequency of assessment; (3) pick appropriate measures for each age of assessment; (4) use multiple measures at each assessment; (5) use overlapping measures across time points; (6) use structural equation modeling (SEM) with measures as indicators of a latent construct at each age – and in doing so, consider (a) sample size, (b) metric of variables, and (c) scale setting; and (7) test longitudinal factorial invariance. Although some of the suggestions may not be relevant or necessary depending on the research question, they would serve to increase confidence that one is measuring the same construct in a comparable way over time, and therefore, confidence in the resulting inferences about development. Although the recommendations are provisional, they should be useful especially for developmental studies involving heterotypic continuity and changing measurement over time. The seven steps are described in further detail below.

(1) Choose the construct of interest

To decide what construct (e.g., inhibitory control) to investigate, it is important for the construct to be as unidimensional as possible because multi-dimensional constructs can make it unclear which dimension is driving an effect (Strauss & Smith, 2009). At the same time, the construct should maintain the theoretical essence and expanse of the construct, which means it cannot be too narrowly described. For instance, as described earlier, the construct of inhibitory control possibly includes facets such as perceptual, performance, association, and motivational inhibition. Although we could drill down further (e.g., facets of perceptual inhibition), our reading suggests there is theoretical support for examining the higher-order inhibitory control construct (Miyake et al., 2000).

(2) Define the age range and frequency of assessment

The second step is to decide what age range the longitudinal study will encompass, and at what frequency to conduct assessments. The age range and frequency of assessment are critical considerations because they set the boundaries within which one can observe developmental change. In order for development to be observed, it must encompass a construct-appropriate age range, and must be sampled at a high-enough frequency to detect change. The frequency of assessment should be as frequent as, if not more frequent than, the highest frequency of change you seek to observe (Little, Card, Preacher, & McConnell, 2009). In other words, if you theoretically expect and seek to detect observable behavioral change every six months, the frequency of assessment should be at least twice per year, if not more often. Of course, limitations of time, resources, and participant burden place restrictions on the frequency of assessment and number of measurement occasions, but the age range and frequency of assessment set the boundaries on the ability to make inferences regarding developmental change. Repeated measures may involve carryover effects, especially practice effects in the case of behavioral tasks (Hertzog & Nesselroade, 2003). Nevertheless, non-comparable measures are usually a greater threat to the validity of developmental inferences in a longitudinal study than are carryover effects (Willett et al., 1998).

(3) Pick appropriate measures for each age of assessment

The researcher should identify various measures with developmentally-appropriate and -sensitive age ranges that, as a collection, assess the whole age range of interest with the desired frequency of assessment. Moreover, each measure should have construct validity for the same construct (e.g., inhibitory control) at each age the measure is used. In addition, each measure chosen should be as unidimensional as possible. Having unidimensional tasks can be difficult because behavioral tasks typically tap into multiple complex cognitive processes (e.g., Busemeyer & Stout, 2002). For instance, many inhibitory control tasks measure multiple processes, including working memory (Wolfe & Bell, 2007b). One way to promote the unidimensionality of a construct as measured by tasks with multiple task demands may be in the SEM framework, as discussed later. In summary, it is essential to use measures that are relevant to the same construct and that are appropriate and sensitive for the different developmental periods in which they are used.

(4) Use multiple measures at each age

Having multiple measures (ideally three or more) at each age is ideal for establishing measurement invariance in SEM with different measures across time and for distinguishing true construct variance from measurement error. At least three indicators (measures) reflecting each latent construct at each age are typically necessary for a just-identified (as opposed to an under-identified) model, which gives a greater likelihood of having an identified and usable model (Little, Cunningham, Shahar, & Widaman, 2002). Collectively, the measures should have content validity and should span the relevant content of the construct at a given point in development, consistent with its heterotypic continuity. In the case of inhibitory control, based on findings from our metaanalysis, greater emphasis might be placed on perceptual inhibition earlier in development and on motivational inhibition later in development.

(5) Use overlapping measures across time points

In general, fewer changing measures and greater overlap of measures provide the SEM model with a greater likelihood of being identified (McArdle et al., 2009). Overlapping measures provide a more coherent latent structure with measures that serve as anchors across time to link measurement occasions. The more time points that measures overlap, the more likely one will be able to successfully link measurement occasions and to establish longitudinal factorial invariance (statistical support for the claim that the measures reflect the same construct at each assessment). This point is analogous to the recommendation of Little, Preacher, Selig, and Card (2007) to ensure that multiple different age cohorts describe a long developmental process by having overlap of at least two time points. For measuring a long span of inhibitory control development, we recommend that (a) each measure is present for at least two consecutive time points and (b) most time points have few changes in measures – one or fewer measurement substitutions at any time point, if possible (for examples using growth models with changing measurement, see the figure on p. 101 of Muthén & Muthén, 2010; fig. 1 in Pettit et al., 2007).

(6) Use SEM with measures as indicators of a latent construct at each age

SEM permits testing longitudinal factorial invariance and partials out measurement error in the estimation of latent constructs. In a reflective SEM model, the latent variable seeks to capture the construct of interest by reflecting the common variance (i.e., covariance) among several measures. The unique variance and measurement error of each of the individual measures are set aside in the form of residuals. Because of the importance of using measures reflecting unidimensional constructs (Strauss & Smith, 2009), SEM may be useful for dissociating construct variance from measurement error and from the contributions of other task demands (e.g., dissociating inhibitory control from working memory). Thus, applying SEM to behavioral tasks may decompose variance from the tasks into unique measurement variance and common latent variance that is a reflection of a construct of interest and more unidimensional. As a result, we would endorse Garon et al.’s (2008) recommendation to use SEM as a framework to create purer, more unidimensional measures of inhibitory control. Finally, because of the capacity of SEM to measure the same latent variable with different indicators across time, we would argue that latent growth curve modeling in SEM can promote the study of constructs whose manifestation changes over time (Pettit et al., 2007). The different measures must assess the same construct at each age in order for the latent factors to be compared in a meaningful way, and for inferences to be meaningful about developmental change. In other words, the measures should have construct validity invariance for the phenomenon of interest, as we describe later. Further considerations are important in using an SEM approach:

(a) Sample size

SEM with latent factors requires a sizeable participant sample, particularly for linking time points with changing measurement. According to Westland's (2010, 2012) heuristic, the minimum sample size for a measurement model in SEM depends on the ratio of indicators to latent variables. He estimates that a sample size of 400, 200, and 100 would be necessary for models with 2, 3, and 4 indicators per latent variable, respectively. However, the minimum sample size to have adequate power to detect an effect depends on the effect size. Estimates from Westland’s (2010, 2012) equation for the minimum sample size, based on the effect size of the association between two latent variables, are in Table 4. The minimum sample size necessary for prediction in an SEM model is determined by taking the greater of the two values from (a) the minimum sample size necessary for a measurement model (see above based on formula in Westland, 2010) and (b) Table 4 (see the value corresponding to the number of latent variables in the model and the smallest effect size to observe). Westland notes, however, that the required sample size may exceed these estimates because the minimum sample size depends on many other factors, such as missingness and multicollinearity.

Table 4.

Minimum sample sizes for adequate statistical power to detect associations of varying effect sizes with latent variables in SEM.

Number of latent variables	Minimum sample size
	|ρ| = .1	|ρ| = .2	|ρ| = .3
1	87	22	10
2	152	37	16
3	290	69	28
4	387	92	38
5	463	110	45
6	526	124	50
7	579	137	55
8	625	148	60
9	666	157	63
10	703	166	67

|ρ| = the smallest effect (Gini correlation coefficient) to detect on latent variables.

Note: Assuming an alpha-significance criterion of α = .05 and power = .80. See Westland (2010, 2012) for the equation for estimating minimum sample size when predicting other variables. The minimum sample size refers to the minimum sample size to detect an association between latent variables. The actual lower bound sample size necessary is the greater of the two values from (a) the minimum sample size necessary for a measurement model (see formula in Westland, 2010) and (b) Table 4 (here).

(b) Metric of variables

In addition to issues of sample size, another issue in SEM involves the metric of variables. When combining multiple behavioral tasks with a single latent variable, one issue that arises is whether to standardize the tasks to be on the same metric. In most applications, the indicators of a latent factor do not have to be on the same metric. The issue of metrics must be considered, however, when using different manifest indicators at each time point. When using different measures across time, it is preferable for scores on measures to share a common, comparable metric (e.g., have the same possible minimum and maximum) in order for indicators to be comparable within time and for latent variables to be comparable across time.

There are ways to reach the goal of having items on the same comparable mathematical metric, while still preserving a meaningful mean level (e.g., inhibitory control ability level) that is sensitive to change over time. For example, proportional scoring metrics can be used to compare scores across ages. One option is the previously mentioned (e.g., Fig. 1) percentage-of-maximum (POM) scoring method that takes an individual’s score and divides it by the total possible score (Little, 2013). POM rescaling essentially converts each individual’s score on a task into a proportion representing the percentage correct (or percentage of the measured phenomenon) on the task. As with any metric, the assumption is that a child with a given POM has the same ability as another child with the same POM, and this may not be the case when different tasks have different levels of difficulty. Because all proportions have the same possible range (0–1), they have greater comparability than the raw metric, and still allow the researcher to observe growth (mean-level change) over time. An advantage of POM rescaling is that, unlike standardization (e.g., age-normed z- or t-scores, which are not recommended for longitudinal data; King, 1986; Willett et al., 1998), it does not distort any of the fundamental statistics of the variable (McArdle, Hamagami, Meredith, & Bradway, 2000). In other words, POM rescaling does not shift the mean or variance relative to other tasks. On the other hand, because POM-rescaled variables maintain their fundamental statistics relative to other variables, different POM-rescaled variables may not be fully on the same mathematical metric. For instance, Grass/Snow appears to be a more challenging task than Shape Stroop (see Fig. 4). If a sample of 3-year-old children has a lower mean score (relative to its possible range) on Grass/Snow than Shape Stroop, POM scores for Grass/ Snow and Shape Stroop would not be on the same metric for purposes of defining inhibitory control. Nevertheless, rescaling variables to be on the same mathematical metric (e.g., z-scores) does not ensure comparability either. The only way to ensure variables are comparable is for them to be on identical metrics prior to rescaling (Willett et al., 1998), which may not always be feasible with behavioral measures.

Alternatively, one could use the greatest observed score as the denominator rather than the total possible score in POM rescaling (e.g., Velásquez, 2010), which could shift the POM to have greater comparability across measures, but it may obscure mean-level change, as well. Consider a hypothetical study where children are measured with 100 Day/Night trials at 3 and 4 years of age. The mean is 30 and 50 at ages 3 and 4 years, respectively, and the observed maximum is 60 and 100, respectively (whereas the possible maximum would be 100 at both ages). Dividing by the observed maximum to calculate POM would shift the mean POM at each age to be the same (50) even though there is a meaningful mean-level difference. In this case, dividing by the possible maximum would retain this mean-level difference, resulting in a POM of 30 and 50 at ages 3 and 4, respectively. Thus, if dividing scores on a measure by the greatest observed score, it is recommended to divide by the same value over time (e.g., the greatest observed score on a measure across all time points; Little, 2013).

A limitation of POM rescaling is that it results in a different metric from the observed metric, which can make interpretation difficult (Velásquez, 2010). When combining measures with different metrics in the context of changing measurement over time, however, it remains advantageous to have a “reasonably comparable scale” that affords greater potential for comparability across time (for examples of studies using POM rescaling in the context of growth curves, see McArdle et al., 2000, p. 60; Petersen et al., 2015). Despite its limitations, POM rescaling provides a way to make scores on separate tasks more comparable with a similar metric (proportion correct), yet still maintaining the ability to observe changes in means and variances over time. Thus, for equating changing measurement over time when measures have different metrics, we recommend the use of POM rescaling in most cases rather than standardization.

(c) Scale setting

There are several options for scale setting in SEM. Scale setting is important because it defines the metric of the latent variable and the resulting interpretability of model parameters for describing developmental change. We consider two common approaches: the marker-variable and reference-group methods (Little, Slegers, & Card, 2006). The marker-variable method is the most common scale-setting approach in SEM and is the default method in some software for SEM analyses. In the marker-variable method, the first (reference) indicator at each time of measurement is constrained to be equal with its loading fixed to 1 and its intercept fixed to 0 (Little et al., 2007). Because the metric of the latent factor is based on the metric of the arbitrarily chosen first indicator, however, its interpretation may be difficult (Little et al., 2007), particularly in the context of indicators with different scales.

An alternative approach to the marker-variable method is the reference-group method, in which the latent variable for one “reference” group (baseline group of interest) is standardized and other groups are scaled relative to the metric of the reference group. For a single group in the longitudinal context, the reference-group method uses a reference time: the latent variable at only T1 (or a different time of interest) is standardized by fixing its mean to 0 and its variance to 1, and all of the factor loadings and intercepts are freely estimated (Ferrer, Balluerka, & Widaman, 2008; Little et al., 2006). The advantage of the reference-group method is that changes in the means and variances over time are scaled relative to the metric of the latent factor at T1 (Ferrer et al., 2008), and are not age-normed or standardized at each age. Thus, although the metric itself may be difficult to interpret, it provides the researcher with the ability to observe growth in means or variances. For most cases dealing with changing measures over time, we recommend using the reference-group approach because it allows observable change in means and variances over time on a more comparable metric.

(7) Test longitudinal factorial invariance

Equivalence of measures across time can be tested in SEM measurement models. Tests of longitudinal factorial invariance help provide evidence for whether one is measuring the same construct on the same scale across time. Longitudinal factorial invariance is demonstrated when multiple measures reliably and validly assess a construct in the same or a similar way across time (or across groups; Curran & Hussong, 2009). Longitudinal factorial invariance tests the extent to which the structure of interrelations among a set of measures (i.e., the factor structure) is statistically the same or different across time. In order for the measures to reflect the same construct across time, it is expected that the measures share a similar factor structure across time. Thus, longitudinal factorial invariance in the situation where different but overlapping measures have been used has the potential to greatly inform our understanding of heterotypic continuity in development (Nesselroade & Estabrook, 2009). However, longitudinal factorial invariance is not sufficient by itself to claim that equivalent measures reflect the same underlying construct and, as a result, is no substitute for construct validity invariance (Knight & Zerr, 2010), as we describe later. We recommend testing longitudinal factorial invariance whether the same or different manifest variables are used, because evidence of failed longitudinal factorial invariance can be informative of developmental change in the phenomena of interest (Edwards & Wirth, 2009; Widaman et al., 2010).

Setting aside the concept of heterotypic continuity, traditionally, in order to make the case that the same construct is being measured across time, two findings are necessary: (1) The measures have construct validity for the same construct of interest at every age in which a particular measure is used (i.e., they are theoretically and empirically associated with the construct at every age they are implemented). (2) The measures show longitudinal factorial invariance (i.e., measurement invariance across time) – the relations of the measures to the construct do not differ across time, and the construct is measured on the same metric across time. Both theoretical (construct validity invariance) and empirical (longitudinal factorial invariance) approaches are necessary to ensure that the same construct is being measured across time.

If both longitudinal factorial invariance and construct validity are established, one can be more confident that differences across time reflect meaningful change and not simply differences in measurement. Longitudinal factorial invariance is generally tested in confirmatory factor analysis (CFA) using SEM (Widaman et al., 2010). Tests of measurement or factorial invariance are critical whenever making comparisons across groups or time periods, as in the case of longitudinal assessment as exemplified by Odgers et al. (2008). Because the focus of this paper is assessing the same construct with different measures, the remaining discussion is framed in terms of longitudinal factorial invariance in CFA that can be tested in these steps:

Steps to test longitudinal factorial invariance

(a) Set up the CFA

Set up a longitudinal CFA in SEM, and represent the construct of interest (e.g., inhibitory control) by a latent factor at each age that is composed of indicators for all of the measures, including behavioral tasks, questionnaires, etc. A longitudinal CFA requires at least two indicators for every latent factor (Ferrer et al., 2008; Widaman et al., 2010). One of the key advantages of SEM is that it allows correlated residuals. It is important to try correlating the residuals of within-measure indicators across time because they often have common measure-specific variances (Little et al., 2007). For instance, residuals for Shape Stroop should be allowed to correlate with each other across time because the Shape Stroop task has construct-irrelevant qualities that are unique from other inhibitory control tasks (i.e., error variance).

(b) Determine whether the measures show longitudinal factorial invariance

There are four types of longitudinal factorial invariance (see Table 5 for a definition and description of longitudinal constraints as applied to the baseline model in Fig. 5), ordered from weak to strong: (1) configural invariance, (2) weak factorial invariance, (3) strong factorial invariance, and (4) strict factorial invariance. There are specific considerations for testing longitudinal factorial invariance in CFA with changing measures. The changing measures across time can be considered a form of missingness, where all of the measures from any measurement occasion inform the structure of the construct at every time point, but with missing measures at various measurement occasions (McArdle et al., 2009). As a result, longitudinal constraints should only be made within measure (see, e.g., Pettit et al., 2007) because all measures are different indicators of the same construct with different patterns of missingness. For example, if Spatial Conflict is measured from ages 2 to 3 years, and Day/Night replaces it after 3 years of age (e.g., see Table 6), Spatial Conflict and Day/Night should not be constrained to have the same factor loadings, intercepts, or residual variances.

Table 5.

Longitudinal factorial invariance applied to example longitudinal study.

Type of invariance (weak to strong)	Description	Constraints in Fig. 5
Baseline model	The model (see Fig. 5) to which the four models below are compared using a χ2 difference test with as many df as there are additional constraints. Scale-setting for the baseline model: Standardize the latent factor at T1 (M = 0, SD = 1).	α1 = 0 σ211 = 1
1. Configural invariance	All of the above constraints, in addition to the same pattern of free versus fixed factor loadings over time: (1)Estimate the loading of the first indicator on the latent variable at T1, but fix the corresponding first loadings to be equal across time. (2)Estimate the first intercept, but fix the corresponding first intercepts to be equal across time. (3)Freely estimate the remaining parameters.	(1) λ11 = λ12 = λ13 (2) τ11 = τ12 = τ13
2. Weak factorial invariance	All of the above constraints, in addition to: Fix corresponding factor loadings across time.	λ21 = λ22 = λ23 λ41 = λ42 λ52 = λ53
3. Strong factorial invariance	All of the above constraints, in addition to: Fix corresponding intercepts across time.	τ21 = τ22 = τ23 τ41 = τ42 τ52 = τ53
4. Strict factorial invariance	All of the above constraints, in addition to: Fix corresponding indicator residual variances across time.	θ11 = θ12 = θ13 θ21 = θ22 = θ23 θ41 = θ42 θ52 = θ53

Note: Suggestions for scale-setting and testing longitudinal factorial invariance with changing measures adapted from Widaman et al. (2010). “Corresponding” refers to same indicator (i.e., measure) across time. In other words, longitudinal constraints of indicators are only within measure.

Fig. 5.

Example baseline longitudinal CFA model of inhibitory control adapted from Widaman et al. (2010) for use with changing measures over time. In the example model, the four measures at each time point (rectangles) serve as manifest variables (indicators) of the latent variable reflecting inhibitory control at each time point (ovals). The example baseline model is an abridged CFA with only three time points for pictorial simplicity. A full test of longitudinal factorial invariance would incorporate all nine measurement occasions from Table 6. See Table 5 for a list of the constraints added to the baseline model to test longitudinal factorial invariance. Constraints are only made within measure. For factor loadings (λ), residual variances (θ), and intercepts (τ), the first numerical digit refers to the measure (e.g., 2 from λ₂₃ = Grass/Snow) and the second digit refers to the occasion of measurement (e.g., 3 from λ₂₃ = Time 3: 42 months). For covariances (σ²), the first numerical digit refers to the occasion of measurement from one end of the arrow and the second digit refers to the occasion of measurement of the other arrow. The covariance of a variable with itself represents that variable’s variance. The triangle represents the estimation of means (intercepts), including the means of the latent factors (α) and the intercepts of the indicators (τ). “mo” = child’s age in months, BD = Bear/Dragon, DN = Day/Night, GS = Grass/Snow, RC = Reverse Categorization, SH = Shape Stroop, SC = Spatial Conflict.

Table 6.

Changing, yet developmentally-sensitive measures of inhibitory control in example longitudinal study.

Age (years)	Measure 1	Measure 2	Measure 3	Measure 4	Facets
2	Spatial Conflict	Shape Stroop	Reverse Categorization	–	Perceptual, association
2.5	Spatial Conflict	Shape Stroop	Reverse Categorization	Grass/Snow	Perceptual, association
3	Spatial Conflict	Bear/Dragon	Reverse Categorization	Grass/Snow	Perceptual, association, performance
3.5	Day/Night	Bear/Dragon	Reverse Categorization	Grass/Snow	Association, performance
4	Day/Night	Bear/Dragon	Inhibitory Control (CBQ)	Grass/Snow	Association, performance, temperament
4.5	Day/Night	Bear/Dragon	Inhibitory Control (CBQ)	Grass/Snow	Association, performance, temperament
5	Day/Night	Simon Says	Inhibitory Control (CBQ)	Grass/Snow	Association, performance, temperament
5.5	Day/Night	Simon Says	Inhibitory Control (CBQ)	Less is More	Association, performance, temperament, motivational
6	–	Simon Says	Inhibitory Control (CBQ)	Less is More	Association, performance, temperament, motivational

Note: Introduction of different, more age-appropriate measures in bold. All measures are conjoined by two or more consecutive time points. No more than one measure is altered between two adjacent time points. Moreover, it would be ideal for a novel task to measure the same facet of inhibitory control as the preceding one that it replaced, and for the facets measured to reflect the heterotypic continuity of the construct.

Starting with a baseline unconstrained CFA model (similar to the one in Fig. 5), one would then test a sequence of levels of longitudinal factorial invariance by successively adding longitudinal within-measure constraints to the model starting with configural invariance and testing up to strict factorial invariance according to Table 5 or until longitudinal factorial invariance fails. Because the additional constraints result in nested models, the model fit between the baseline model and the model with additional constraints can be compared with the difference in chi-square between the two models with as many degrees of freedom as there are additional constraints (Little et al., 2007). If the additional constraints result in a significant chi-square change (i.e., a significant worsening in model fit), the model with additional longitudinal constraints is considered to be a significantly worse fit for the data, and the model has failed that particular type of measurement invariance. Because the chi-square change statistic is sensitive to sample size, the recommended criteria for evaluating model fit for studies with larger samples are a change in the comparative fit index (ΔCFI) greater than .01 (Cheung & Rensvold, 2002).

If longitudinal factorial invariance holds for all of the types of measurement invariance (see Table 5), the empirical structure of relations suggests that the variables are comparable across time. If longitudinal factorial invariance fails, however, the empirical evidence suggests that the variables may not be comparable across time and that there is a good chance of comparing “apples” to “oranges” when examining change over time (Odgers et al., 2008). Although different researchers use different guidelines to determine measurement invariance, Little et al. (2007) suggested that researchers establish at least partial weak longitudinal factorial invariance to examine covariances between latent variables and partial strong longitudinal factorial invariance to examine mean-level change in latent variables (“partial” invariance refers to invariance with some indicators but failed invariance with others; Ferrer et al., 2008), whereas Widaman et al. (2010) suggested establishing at least strong factorial invariance. For example, to examine whether inhibitory control is related to the development of attention deficit hyperactivity disorder, longitudinal factorial invariance will be considered to have failed if not achieving at least weak factorial invariance (the same factor loadings across time) for some of the measures of inhibitory control to serve as anchors. For the goal of examining whether mean levels of inhibitory control change as children get older, longitudinal factorial invariance will be considered to have failed if at least strong factorial invariance (the same factor loadings and intercepts/means across time) is not achieved for some of the measures. In our example study of mean-level growth in inhibitory control (described later), we would need to establish the same factor loadings and intercepts for at least one of the four combinations of measures (see Table 6): (1) Spatial Conflict and Day/ Night, (2) Shape Stroop, Bear/Dragon, and Simon Says, (3) Reverse Categorization and the Inhibitory Control subscale from the CBQ, and (4) Grass/Snow and Less is More. The more combinations that demonstrate the same factor loadings and intercepts, the more confident we can be in measurement comparability.

(c) Make any necessary adjustments

If longitudinal factorial invariance fails, therefore resulting in the lack of a statistical basis to assert longitudinal equivalence of measures, the researcher has several options that should be guided by theoretical and empirical considerations. To reiterate, longitudinal factorial invariance alone is not sufficient evidence that one has measured the same construct over time; see construct validity below. Theoretically, there are cases in which one might expect changes in a construct’s manifestation over time. For example, self-regulation and inhibitory control change over development from early reliance on external sources for control to ability for internal, self-initiated forms of control. These changes in the construct’s manifestation necessitate changes in measurement, which may result in failed longitudinal factorial invariance.

Empirically, one should test when (at what ages, Odgers et al., 2008) and with what measures (Millsap, 2010) the assumption of factorial invariance failed. To determine where factorial invariance failed, one can test a piecewise longitudinal CFA, gradually including successive years or tasks until the measurement assumption fails. Alternatively, one can gradually relax parameter constraints for longitudinal factorial invariance to determine with which measures or at what ages longitudinal factorial invariance becomes established (and by extension, where it fails). It will then be up to the researcher to determine, ideally based on additional research, whether the failed longitudinal factorial invariance resulted from change in the manifestation of the construct, changes in the functioning of the measures, or some combination of the two (Knight & Zerr, 2010), because there is no statistical method for choosing between these alternatives. Unfortunately, there are few empirical guidelines for determining the severity of failed factorial invariance (i.e., whether it owes to changes in the functioning of a measure; Knight & Zerr, 2010), although some have proposed using effect sizes or other estimates (Millsap, 2010). If the factorial invariance violations show important differences in the meaning of measures, say at a few of the ages in a series of measures of a construct, it may be necessary to exclude the measures at those ages from the analysis. Alternatively, one might substitute other measures reflecting the latent construct at those ages, as long as the set of measures under consideration remains a representative selection for the construct (Knight & Zerr, 2010).

We know of a few studies that have examined longitudinal factorial invariance of inhibitory control tasks. Studies have typically examined longitudinal factorial invariance of inhibitory control tasks in the context of the broader construct of executive function. Willoughby et al. (2012) found strong longitudinal factorial invariance for variants of the Day/Night and the Spatial Conflict task in relation to a latent variable composite of six executive function tasks from 3 to 5 years of age. The authors found partial strong longitudinal factorial invariance for the executive function composite across the same time frame. Fuhs and Day (2011) found partial strict longitudinal factorial invariance across 43–63 months of age for a latent composite of three executive function tasks that included the Day/Night task. Hughes, Ensor, Wilson, and Graham (2009) found full strong longitudinal factorial invariance from 48 to 72 months of age using a latent composite of three executive function tasks that included the Day/Night task.

In sum, there is some evidence that one can establish partial longitudinal factorial invariance with a latent composite that includes inhibitory control tasks over a fairly limited age range, but limited evidence of full longitudinal factorial invariance. This is consistent with the notion (but not sufficient to demonstrate) that the same task may reflect the same homotypic construct over a restricted developmental span, and that additional different measures are necessary to capture its heterotypic continuity over time. The prevailing bias for publishing studies may be that they must demonstrate longitudinal factorial invariance; studies covering lengthy developmental spans, however, may fail traditional criteria for longitudinal factorial invariance and may be less likely to be published. More studies of inhibitory control are needed that test, and possibly fail, longitudinal factorial invariance in order to understand how inhibitory control changes in manifestation with development.

Construct validity invariance

As mentioned before, longitudinal factorial invariance is not sufficient for evaluating whether one is measuring the same construct over time. Construct validity invariance should also be considered. Widaman et al. (2010) described a series of studies on the construct of numerical facility. Whereas second graders used reconstructive, counting strategies for simple addition, complex addition, and subtraction problems, the college students used memory retrieval strategies. Thus, the construct of numerical facility changed in manifestation, resulting in changes in the meaning of the measures, despite having established strict factorial invariance. Thus, to demonstrate that measures at different time points reflect the same processes, evaluating the construct validity of measures at each age as reflections of the same construct of interest, termed construct validity invariance, is also important (Knight & Zerr, 2010).

Construct validity includes both content and criterion validity (Strauss & Smith, 2009). There are many ways to explore the construct validity of a set of measures for a given construct. First, the task selection should be based on theory – the tasks should reflect the same construct and the tasks should adequately sample the different facets of the construct (content and face validity). Second, despite theoretically expected heterotypic continuity in the long-term, as in the case of inhibitory control, there should be short-term test–retest reliability (cross-time consistency) of the measures across time. Third, the measures should show convergent validity with each other and discriminant validity with measures of distinct constructs according to Campbell and Fiske’s (1959) example of the multi-trait multi-method matrix (for a contemporary longitudinal example, see Grimm, Pianta, & Konold, 2009). The measures should show stronger associations with measures of the same construct and weaker associations with measures of different constructs. For example, measures of inhibitory control should be more strongly associated with each other than with measures of sustained attention (after discarding method variance). Fourth, the measures should show a similar factor structure across time (not necessarily an invariant structure in the case of heterotypic continuity). Fifth, the measures should have high internal consistency (communality or equivalence) as measured by Cronbach’s alpha (Cortina, 1993) or McDonald’s total omega (Revelle & Zinbarg, 2009). Sixth, the measures should be associated with measures of other constructs in theoretically expected ways (criterion validity) as part of its nomological network, particularly when examining heterotypic continuity (Rutter & Sroufe, 2000). Despite these ways of developing evidence for or against the validity of a set of measures for a given construct, construct validity is inherently indefinite and is a continual process, not a discrete outcome (Strauss & Smith, 2009).

The final decision: whether to test a growth model

Taking into account the above recommendations and considerations, the researcher now must decide whether the totality of theoretical and empirical evidence supports the goal to measure the same construct over time with comparable measures. If researchers find that the theoretical and empirical evidence supports the ability to draw developmental inferences from differences over time in the measurement model, they can now test a growth curve model (or other longitudinal model) with changing measures. Although we do not provide an empirical example of the proposed approach in the current paper, there are previous examples of successful and useful implementations of similar approaches, suggesting that the proposed approach is feasible and defensible. Approaches with changing measures have been successful with diverse constructs such as externalizing problems (e.g., Petersen et al., 2015), intelligence (e.g., McArdle & Grimm, 2011; McArdle et al., 2009), and parental monitoring (e.g., Pettit et al., 2007). For examples of longitudinal growth curve models with changing measures, see Pettit et al. (2007) and Owens and Shaw (2003), or with changing raters, see Odgers et al. (2008), or with changing measures and raters, see Petersen et al. (2015). Testing a growth curve model allows the researcher to observe how children’s inhibitory control develops over time, and how inhibitory control development predicts meaningful developmental outcomes or is predicted by important risk or protective factors. For discussions of different types of longitudinal models and when to use them, see Singer and Willett (2003), McArdle (2009), and Fontaine and Petersen (2016).

Applying the recommendations to a hypothetical example of a longitudinal study of inhibitory control

To illustrate how to measure the development of inhibitory control using the above recommendations, we provide a hypothetical example of longitudinal assessment. We focus on the development of inhibitory control from ages 2 to 6 years, because previous studies have shown extensive development of inhibitory control (operationalized with measures in Table 1) during this time frame (Carlson, 2005; Carlson, Davis, & Leach, 2005; Carlson, Mandell, & Williams, 2004; Carlson & Moses, 2001; Carlson, Moses, & Claxton, 2004; Gerardi-Caulton, 2000; Hughes & Ensor, 2007; Kochanska, Murray, & Coy, 1997; Kochanska et al., 1996, 2000), supported by the findings from our meta-analysis (see Figs. 1–3). Many prior studies of inhibitory control conducted assessments annually or less frequently (e.g., Carlson et al., 2005; Murray & Kochanska, 2002), which may be insensitive to some forms of meaningful developmental change. Because Carlson (2005) noted considerable improvement in inhibitory control with development in a cross-sectional study with 6-month age groups from ages 3 to 5 (their full sample was from ages 2 to 8), we decided, for purposes of the hypothetical example, to propose assessments every six months from ages 2 to 6 years.

After selecting our construct and choosing the age range and frequency of assessment, we chose developmentally-sensitive measures from Table 1 that met our goals, including construct validity at each age, multiple tasks at each age, multiple measurement methods, overlapping time points for measures, at least two consecutive time points for each measure, and no more than one measurement change at any time point. See Table 6 for our selection of measures at each age. Consistent with the heterotypic continuity of inhibitory control, we selected measures that reflect the different facets of inhibitory control that are “coming online” at different points in development. For instance, we selected more perceptual inhibition tasks among the measures at earlier ages and motivational inhibition tasks among the measures at later ages, consistent with findings from our meta-analysis that perceptual inhibition may develop before other facets of inhibitory control. The measures of inhibitory control, however, are not unidimensional and reflect other processes including working memory. There have been attempts to dissociate working memory from inhibitory control in behavioral tasks (e.g., Carlson, Moses, & Breton, 2002; Raaijmakers et al., 2008; Senn, Espy, & Kaufmann, 2004), with studies commonly finding working memory and inhibitory control as correlated but separate factors. Thus, researchers have argued that task demands in behavioral tasks of self-regulation are dissociable (Carlson, 2005). As a result, a plausible next step would be to combine the measures in an SEM framework to create more unidimensional measures of inhibitory control.

In SEM, we would test longitudinal factorial invariance in CFA. First, we would start with the baseline model (see Fig. 5 for an abridged model with only three time points). Second, we would test models with various longitudinal constraints in sequence (see Table 5) to determine where our factorial invariance holds and where it fails. Following determination of longitudinal factorial invariance, evaluation of violations, and any changes to establish factorial invariance while ensuring construct validity invariance, we would then turn our changing measures into a longitudinal growth curve model.

Limitations

Even if researchers follow the empirical recommendations in the present paper, there is no guarantee that the measurement model will pass tests of longitudinal factorial invariance or will reflect the same construct over time. Establishing longitudinal factorial invariance may be difficult with changing measurement over time, especially with common methodological limitations such as high levels of missingness (McArdle et al., 2009). In all likelihood, given adequate construct validity of a set of measures, using more measures over a longer consecutive period of time (albeit within developmentally-appropriate and -sensitive age ranges) with fewer changes will be more likely to provide a meaningful longitudinal model. Moreover, as discussed earlier, establishing longitudinal factorial invariance does not necessarily mean the same construct has been assessed over time, with or without changing measures. Thus, researchers should be cautious when conducting longitudinal assessments to ensure that their measures have construct validity for the construct of interest at each assessment occasion in which they are used and that the measures are developmentally appropriate and sensitive for the same time frame.

In the current paper, we proposed using an overlapping measurement model in SEM to measure the heterotypic continuity of inhibitory control. We also recognize that there are other possibly useful approaches for linking changing measurement over time (especially with dichotomous or polytomous items) and testing longitudinal factorial invariance, including integrative data analysis (Curran et al., 2008) and IRT (Kolen & Brennan, 2014; McArdle et al., 2009; Meade, Lautenschlager, & Hecht, 2005; Millsap, 2010). The developmental cascade model is another approach to longitudinal measurement where a construct changes in manifestation over time as a function of changing influences that lead to other risks (Cox, Mills-Koonce, Propper, & Gariépy, 2010; Dodge, Greenberg, Malone, & Conduct Problems Prevention Research Group, 2008; Dodge et al., 2009). A discussion of these alternative approaches is beyond the scope of the current paper. For a review of other techniques for dealing with growth modeling in the context of changing measurement, see McArdle et al. (2009).

Future directions

Future research should examine how much overlap is necessary for longitudinal comparability, with methods such as Monte Carlo analysis (Pettit et al., 2007). Moreover, future studies should examine failed longitudinal factorial invariance to establish guidelines for determining when failed factorial invariance reflects change in the manifestation of a construct versus change in the meaning of a measure. Future studies might also develop measures that maintain construct validity invariance across lengthy spans of development (for attempts, see Bauer & Zelazo, 2014; Carlson & Zelazo, 2014; Weintraub et al., 2013). Future work should consider how best to measure inhibitory control across multiple levels of analysis, including bio-psycho-social processes, in ways that are consistent with its heterotypic continuity. For instance, children’s inhibitory control and self-regulation develop through the internalization of rules in the context of relationships with parents, siblings, caregivers, and teachers. Thus, future work should consider how to account for environmental influences in the heterotypic development of inhibitory control. Future applications of our recommendations might examine when and how inhibitory control develops over time in young children, how growth in inhibitory control is influenced by other risk and protective factors, and how inhibitory control influences other outcomes. Broader applications of the proposed technique for longitudinal measurement with changing measures may be relevant for many other domains involving constructs that change in manifestation over time (e.g., intelligence, McArdle et al., 2009; externalizing problems, Petersen et al., 2015).

Implications

Inhibitory control and related processes are among the most widely studied constructs in early childhood. As researchers place greater emphasis on measuring inhibitory control using multiple measures, we need to understand how these measures function at different ages, and when they are most useful. Our work is an important step toward understanding the heterotypic continuity of inhibitory control and the age ranges during which measures of inhibitory control are most useful. Using measures in a longitudinal context and testing longitudinal factorial invariance will lead researchers to place greater emphasis on the developmental utility of measures and their construct validity. Stronger longitudinal assessment of inhibitory control will be important for making inferences about the development of inhibitory control. At the same time, no empirical approach can compensate for the necessity of an informed theoretical approach to the selection and structure of measures in a longitudinal measurement model. Unfortunately, there is no way to be entirely certain that one is measuring the same thing across time. Thus, it is imperative to “consider all theoretical and empirical evidence that can strengthen our confidence in whether we are assessing the same construct… in a psychometrically equivalent way” (Curran & Hussong, 2009, p. 93). The suggested approach may also allow for greater coherence across studies so that results about inhibitory control development can be interpreted in meaningful ways. A more theoretically- and empirically-informed approach to longitudinal assessment has the potential to vastly improve our understanding of the construct of inhibitory control from a developmental perspective and its relation to other important predictors and outcomes.

Summary

Current theory suggests that inhibitory control changes in its manifestation over time (heterotypic continuity). The findings from our meta-analysis advance our understanding of the heterotypic continuity of inhibitory control, with perceptual inhibition developing at earlier ages and motivational inhibition possibly developing at later ages. Further, measures of inhibitory control were generally useful for a developmental span of less than 3 years. As a result, in order to properly measure the development of inhibitory control, appropriate changes in measurement should accompany developmental changes in the manifestation of the construct. To deal with changing measures over time, we recommend a longitudinal approach to studying the development of inhibitory control that involves overlapping measurement in an SEM framework to test longitudinal factorial invariance. Although longitudinal factorial invariance should be tested, establishing measurement or factorial invariance is neither sufficient, as seen where construct invariance does not hold despite establishing factorial invariance, nor necessary, as seen in the case of heterotypic continuity, for examining developmental change, because any developmental approach should be guided by construct validity. In combination, a better empirical and theoretical approach to the selection and combination of measures will lead to improved developmental sensitivity of measurement, and will improve our ability to make meaningful developmental inferences.