Identifying Causal Estimands for Time-Varying Treatments Measured with Time-Varying (Age or Grade-Based) Instruments

Abstract

This commentary discusses causal estimands of same-age and same-grade comparisons for assessing grade-retention effects on student ability and performance. Using potential outcomes notation, we show that same-age and same-grade comparisons refer to different retention–promotion contrasts and therefore assess different causal questions. We also comment on deleting versus censoring records of students who dropped out of the study or do not belong to the treatment regimes under investigation. Whereas deleting entire student records potentially induces collider bias, censoring circumvents bias if censoring is ignorable given the observed pretreatment covariates.

We appreciate the opportunity to comment on the excellent introduction on time-varying treatments by Vandecandelaere, Vansteelandt, De Fraine, and Van Damme. We congratulate the authors for writing a very accessible introduction to marginal structural models for estimating the effects of static treatment regimes. In particular, how they explain issues with respect to time-varying treatments and confounding and the assumptions required for causally identifying the treatment effects is very well done.¹ We are convinced that future research on grade retention will be guided by these authors’ application of marginal structural models for estimating the effects of early grade retention on math achievement.

Identifying retention effects is particularly challenging since the test instruments for measuring student achievement vary over time; that is, the administered achievement tests depend on the time-varying retention decision. Thus, as Vandecandelaere et al. correctly point out, the choice of an appropriate comparison strategy, either a same-age or same-grade comparison, is of crucial importance. We go a step further and formalize the two comparison strategies using potential outcomes notation in order to define and discuss different causal estimands and their underlying treatment contrasts. We argue that the same-age and same-grade comparisons differ in their retention–promotion contrasts and, thus, assess different causal questions. Another issue we address in this commentary refers to handling records of students who dropped out (missing data) or do not belong to the treatment regimes under investigation (i.e., multiple retentions or retentions after a certain grade). Instead of deleting the entire student records, Vandecandelaere et al. carefully censor corresponding records and address resulting biases by using censoring weights. Here, we highlight what would have happened if Vandecandelaere et al. had deleted rather than censored the data.

Same-age and same-grade comparisons

The average causal effect (ACE) of grade retention on an achievement score Y can be defined in terms of potential outcomes as ACE = E(Y (1) − Y (0)) E(Y (1)) − E(Y (0)), that is, the difference in the expected potential retention and promotion outcomes that would be observed if all at-risk students were retained, E(Y(1)), instead of promoted, E(Y (0)). Let Z ∈ {0,1} be the retention indicator, then Y (Z = z) represents the potential retention (Z = 1) or potential promotion outcome (Z = 0). The expectation in the ACE definition is typically taken over the population of at-risk students. For simplicity of exposition, we consider a situation where students can get retained only once and only at the same single grade g* ∈ {0, …, G}, that is, at a specific grade g* between the kindergarten grade 0 and a final grade G. For example, the retention condition (Z = 1) may refer to retention in kindergarten (g* = 0), whereas the promotion condition (Z = 0) implies no kindergarten or later retention.

Although the ACE definition seems natural, the ACE has a meaningful interpretation only if both potential outcomes are measured with the same instrument (i.e., achievement test) at the same time. However, since achievement tests vary by grade, retention effects are not directly assessable like medication effects on blood pressure where the blood pressure monitor remains invariant to patients’ treatment and age. Thus, to explicitly highlight the grade dependence of potential outcomes, we use a grade index g. Since retained and promoted students take tests of the assessment grade g (for g > g*) at different ages, we also introduce an age index k. Thus, the potential outcomes, $Y_k^g\left(0\right)$ and $Y_k^g\left(1\right)$, refer to a specific assessment age and assessment grade. Using this notation, we now can discuss same-age and same-grade ACEs with respect to outcomes measured t years after the retention decision. For a given t > 0, intervention grade g*, and a kindergarten entrance age of 5 years, a student’s assessment age refers to k = 5 + g* + t. Consider retention at the kindergarten (g* = 0) and suppose we are interested in the retention effect 3 years after the retention decision (t = 3), then the assessment age is k = 5 + 0 + 3 =8 years. Under the promotion regime, students would be in the third grade (g* + t = 0 + 3 = 3); under the retention regime, they would be in the second grade (g* + t – 1 = 0 + 3 − 1 = 2). We use this example (g* = 0, k = 8, and t = 3) to illustrate the differences between the same-age and same-grade comparison but provide a more general and formal notation in Table 1.

Table 1.

Treatment conditions, measurements, and causal estimands of same-age and same-grade comparisons of repeating grade g*.

	Same-age comparison	Same-grade comparison
	Intervention
Type of intervention	actual implementable intervention	sudden hypothetical intervention
Control condition (promotion)	grade g* + 1 to g* + t curricula with same-age peers	grade g* + 1 to g* + t curricula with same-age peers
Treatment condition (retention)	grade g* to g* + t — 1 curricula 1 year later with 1-year-younger peers	1 year of aging; grade g* to g* + t curricula with 1-year-younger peers
Treatment contrast (retention vs. promotion)	(a) second exposure to grade g* curriculum with 1-year-younger peers; (b) grades g* +1 to g* + t — 1 one year later with 1-year-younger peers; (c) absence of grade g* + t curriculum	(a) 1 year of aging; (b) second exposure to grade g* curriculum with; (c) 1-year-younger peers 1-year-younger peers from grade g* +1 to g* +1
	Instruments and measurements
Treatment condition	grade g-1 test and scores	grade g test and scores
Control condition	grade g test and scores	grade g test and scores
	Causal estimands
Main estimand	$AC{E}_k^s=\left({\tilde{Y}}_k^{g-1}\left(1\right)-{\tilde{Y}}_k^g\left(0\right)\right)$	$AC{E}_g=E\left(Y_{k+1}^g\left(1\right)-Y_k^g\left(0\right)\right)$
Alternative estimands	$AC{E}_k^g=E\left(Y_k^g\left(1\right)-Y_k^g\left(0\right)\right)$ $AC{E}_k^{g-1}=E\left(Y_k^{g-1}\left(1\right)-Y_k^{g-1}\left(0\right)\right)$

Note. Grade g ∈ {0, …, G} refers to a grade between kindergarten (g = 0) and a final grade G (g = G). Students can get retained once at grade g* but at no other grade (0 < = g* < G). We are interested in the average causal effects (ACEs) t years after the retention decision. For a given t > 0, intervention grade g*, and a kindergarten entrance age of 5 years, a student’s assessment age is given by k = 5 + g* + t. Potential promotion and retention outcomes, $Y_k^g\left(0\right)$ and $Y_k^g\left(1\right)$, refer to student achievements assessed with the instrument of grade g at age k. Potential outcomes ${\tilde{Y}}_k^g\left(0\right)$and ${\tilde{Y}}_k^g\left(1\right)$ denote the vertically equated scores.

Same-age comparisons

If we are interested in assessing the effect of grade g* retention on achievement scores measured at age k, we could define the retention effect as $AC{E}_k=E\left(Y_k^{g-1}\left(1\right)-Y_k^g\left(0\right)\right)$, which is the expected difference in the potential retention outcome assessed at age k with the grade g-1 test and the potential promotion outcome assessed at the same age k but with the grade g test. Thus, 3 years after kindergarten, the retention effect $AC{E}_{k=8}=E\left(Y_8^2\left(1\right)-Y_8^3\left(0\right)\right)$ is the expected difference in the 8-year-old students’ potential retention and promotion outcomes assessed at the second and third grade, respectively.

The same-age ACE_k=8 relies on a treatment contrast where retention requires repeating the kindergarten curriculum and taking the kindergarten to second grade with peers that are 1 year younger than in the promotion condition. Promotion requires taking the first to the third grade with same-age peers. Thus, retention differs from promotion in (a) repeating kindergarten with 1-year-younger peers, (b) taking the first and second grade 1 year later with 1-year-younger peers, and (c) the absence of exposure to the third grade (first column of Table 1). This treatment contrast is well defined and of primary interest in evaluations of retention effects on students. If there were no ethical issues, students could actually be randomly assigned to the corresponding retention and promotion conditions. However, the ACE_k=8 definition reveals that the retention effect is completely confounded with differences in the second- and third-grade achievement tests.

To avoid a confounding due to instrumentation differences, we can define two alternative ACEs: $AC{E}_k^g=E\left(Y_k^g\left(1\right)-Y_k^g\left(0\right)\right)$ and $AC{E}_k^{g-1}=E\left(Y_k^{g-1}\left(1\right)-Y_k^g\left(0\right)\right)$, or with respect to our example, $AC{E}_{k=8}^{g=3}=E\left(Y_8^3\left(1\right)-Y_8^3\left(0\right)\right)$ and $AC{E}_{k=8}^{g=2}=E\left(Y_8^2\left(1\right)-Y_8^2\left(0\right)\right)$. That is, under both the retention and promotion condition, we assess the 8-year-old students with the same achievement test, either with the third-grade test or the second-grade test (in practice, this would require an additional testing of either retained or promoted students). However, both ACEs are presumably not very interesting: With respect to $AC{E}_{k=8}^{g=3}$, we would expect that the potential third-grade retention scores $\left(Y_8^3\left(1\right)\right)$ are lower than the third-grade promotion scores $\left(Y_8^3\left(0\right)\right)$, just because the third-grade test requires problem-solving skills taught in the third grade but not in the second grade. And for $AC{E}_{k=8}^{g=2}$, we might expect that the potential second-grade retention scores $\left(Y_8^2\left(1\right)\right)$ are at least as good as the second-grade promotion scores $\left(Y_8^2\left(0\right)\right)$, just because of the more recent exposure to problem sets of the second grade. Thus, even if retention would have no differential effect on students’ ability growth, we would very likely obtain a negative $AC{E}_{k=8}^{g=3}$ but likely a nonnegative $AC{E}_{k=8}^{g=2}$ Nonetheless, both ACEs are well-defined causal estimands for evaluating retained and promoted students’ performance on grade-specific problem sets at age k.

Since researchers using same-age comparisons are often not directly interested in grade-specific problem-solving skills of retained and promoted students but instead in students’ ability growth, they aim at assessing the retention effect on students’ latent ability (rather than the observed performance score). Since ability is not directly observed and standard achievement scores cannot directly be compared across grades, researchers use vertically equated scores (e.g., Hong & Raudenbush, 2006; Vandecandelaere et al. also use this strategy). With perfectly equated scores for grades g-1 and g assessed at ages $k,{\tilde{Y}}_k^{g-1}\left(z\right)$ and ${\tilde{Y}}_k^g\left(z\right)$, we can establish for each student the following two equalities:${\tilde{Y}}_k^{g-1}\left(0\right)={\tilde{Y}}_k^g\left(0\right)$ and ${\tilde{Y}}_k^{g-1}\left(1\right)={\tilde{Y}}_k^g\left(1\right)$. With respect to our example, the equalities imply that it does not matter whether we assess a promoted or retained student’s ability with the second- or third-grade test. After vertical equating, both instruments measure student ability equally well. If this is true, then the ACE with scaled scores, $AC{E}_k^s=E\left({\tilde{Y}}_k^{g-1}\left(1\right)-{\tilde{Y}}_k^g\left(0\right)\right)=E\left({\tilde{Y}}_k^{g-1}\left(1\right)-{\tilde{Y}}_k^{g-1}\left(0\right)\right)=E\left({\tilde{Y}}_k^g\left(1\right)-{\tilde{Y}}_k^g\left(0\right)\right)$, no longer depends on the choice of a grade-specific achievement test. An $AC{E}_{k=8}^s$ of zero would imply that kindergarten retention has the same effect as being promoted on the ability of 8-year-old students. But it does not imply that retention and promotion necessarily provide the same skills for solving problem sets of the second or third grade.

The interpretability of $AC{E}_k^s$ crucially depends on the characteristics of the vertically equated scores, particularly when (a) the scores of retained and promoted at-risk students come from different tails of the grade-specific distribution of achievement scores and (b) assessments involve not only tests of two neighboring grades but tests across three or more grades (in case of multiple retentions). If the equated scores do not meet the equalities stated in the preceding paragraph, then the meaning of $AC{E}_k^s$ becomes opaque and may lead to invalid inferences. For example, if the equated potential outcomes assessed with the second-grade test are systematically lower than the equated outcomes assessed with third-grade test,${\tilde{Y}}_8^2\left(0\right)<{\tilde{Y}}_8^3\left(0\right)$ or ${\tilde{Y}}_8^2\left(1\right)<{\tilde{Y}}_8^3\left(1\right)$, the estimated retention effect would be negative even if retention has no effect on ability growth. Moreover, since the curricula for the first to second grade are taught 1 year later in the retention condition than in the promotion condition, curricula changes from one year to the next (but also other grade-specific changes) affect the treatment contrast and the generalizability of results.

Same-grade comparisons

To avoid potentially unfair comparisons of retained and promoted students due to changes in curricula and imperfect equating procedures of grade-specific achievement scores, researchers also use same-grade comparisons for assessing the effects of grade retention (e.g., Moser, West, & Hughes, 2012; Wu, West, & Hughes, 2008). Using the same promotion regime and the respective potential promotion outcomes as in the same-age comparison, we can define the average causal retention effect for the same-grade comparison as $AC{E}_g=E\left(Y_{k+1}^g\left(1\right)-Y_k^g\left(0\right)\right)$, that is, the expected difference in the potential retention scores assessed with the grade g test at age k + 1 and the potential promotion scores assessed with the same grade g test but at age k. Suppose we want to assess the effect of kindergarten retention at the third grade, then $AC{E}_{g=3}=E\left(Y_9^3\left(1\right)-Y_8^3\left(0\right)\right)$ compares 9-year-old students in the retention condition to the very same but 1-year-younger students (8-year-old) in the promotion condition. The difference in age arises because promotion to the third grade takes an additional year under the retention condition.

It is harder in the case of ACE_g than in the case of ACE_k to conceive of a realistic intervention that actually produces the treatment contrast. If we define the promotion condition as taking the first to third grade with same-age peers, one can think of the retention condition as a sudden hypothetical intervention after the end of first kindergarten year that lets a 6-year-old student instantaneously age for 1 year and simultaneously experience the kindergarten curriculum with 5-year-old peers again. After the intervention, the now 7-year-old student attends the first to third grade just like in the promotion condition but with 1-year-younger peers (due to the sudden aging). An instantaneous intervention is necessary because the grade-specific conditions (curricula, peers, achievement tests, and other grade-specific factors) need to be identical in the retention and promotion schemes, otherwise the potential retention and promotion outcomes would differentially depend on historical events. In addition, all prekindergarten grades need to be taken at the same age in both conditions; otherwise, the potential outcomes would not refer to the same individual. Thus, the treatment contrast of the same-grade comparison is given by (a) 1 year of aging, (b) the second exposure to the kindergarten curriculum with 1-year-younger peers, and (c) taking the first to third grade with 1-year-younger peers—due to the retained students’ sudden aging (second column in Table 1).

Although this is an interesting retention–promotion contrast, it is hard to imagine how it could actually be established in a credible experiment. We would need to randomly retain 6-year-old at-risk students that just finished their kindergarten year and match them to 5-year-old students that just finished the last prekindergarten year (in order to “simulate” the aging). Matching has to guarantee that the 5- and 6-year-old students are comparable in all aspects except for age. Then, the retention effect captures the double exposure to kindergarten and 1 year of aging. Note that aging is an integral part of the retention intervention.

In this sense, ACE_g=3 is a legitimate estimand for assessing the performance or even ability of retained students on the third-grade test in comparison to the 1-year-younger promoted students Wu et al., 2008). However, ACE_g=3 is less appropriate for evaluating the retention effect that is exclusively due to the double dose of kindergarten (but not due to aging) because ACE_g=3 confounds the double-dose effect with the effect of 1 year of aging. Only if the age effect would be zero, ACE_g=3 can be interpreted as the double-dose effect. However, the absence of age effects seems implausible given that student abilities naturally grow from one year to the next—even without any exposure to a grade-specific curriculum.

Summary

We think that both the same-age and same-grade comparisons are informative and useful for practice. However, one needs to be aware of the respective treatment contrasts and the different assumptions required for identifying ACEs. Same-age comparisons using $AC{E}_k^s$ are better suited for assessing the double-dose effect of repeating a grade that is not confounded with the naturally occurring ability growth due to aging. However, if vertical equating does not produce valid and reliable scores with respect to the underlying ability, one needs to be very cautious in interpreting the $AC{E}_k^s$ estimates. Same-grade comparisons (ACE_g) are particularly appropriate for comparing the performance of retained and promoted students at a specific grade, particularly so when grade-specific performance builds the basis for future promotions, course or school placements, or admissions to other educational programs. The results of Vandecandelaere et al. essentially support what we can expect from the different comparisons. However, a more thorough discussion of whether the assumptions underlying the same-age and same-grade comparison are met would help in assessing the meaning and credibility of the estimated effects.

Collider bias due to deleting records based on (post)-outcome variables

In analyzing their data, Vandecandelaere et al. deleted records of early dropouts (Year 0 or Year 1) and censored records of students that dropped out in Year 2 or later (attrition) or were retained after Year 3 or more often than once. Vandecandelaere et al. handle these issues with great care. For the deleted early dropouts, they plausibly argue that this type of attrition is MCAR (missing completely at random). Attrition from Year 2 onward they addressed with censoring weights. Vandecandelaere et al. apply the same strategy to the deliberately censored data of students who do not belong to the four treatment regimes (i.e., students who got retained twice or got retained after Year 3). Censoring weights work very well provided MAR (missing at random) holds. It is important to realize that deleting instead of censoring the corresponding student records would have caused serious biases, even if one were to weight cases with respect to their baseline covariates in Year 0 and Year 1.

Here we briefly demonstrate the consequences of deleting student records according to (post)-outcome variables with a simple data-generating scenario with 3 years of schooling as depicted in Figure 1. The time-varying treatment variable T_t (t = 1, …, 3) represents the sequence of grades taken by a student over 3 years. If a student gets promoted in all 3 years, the treatment variables take on values T₁ = g1, T₂ = g2, and T₃ = g3, where g1 corresponds to grade 1, g2 to grade 2, and g3 to grade 3. For a student who gets retained only in grade g1, the treatment sequence is T₁ = g1, T₂ = g1, and T₃ = g2. Similar sequences can be constructed for all possible retention regimes. The outcome measures Y_t refer to vertically equated achievement scores measured at the end of year t and reflect the underlying student abilities A_t. The treatment status T_t—that is, whether a student is retained or promoted to the next grade in year t—depends only on the observed achievement score Y_t−1 and the treatment status T_t−1 of the previous year (but it does not depend on ability because no arrows directly connect A_t−1 and T_t). The short arrows pointing into all the variables indicate independent but unobserved factors or error terms.

Figure 1.

Data-generating graph. T_t = the treatment variables; Y_t = the achievement scores; A_t = the unobserved abilities (for t = 0, …, 3).

Assume that we want to estimate the effect of being retained in T₂ (i.e., repetition of grade g1) on the outcomes Y₂ and Y₃. From the graph in Figure 1, we directly see that the ACE on Y₂ (i.e., T₂ → A₂ →Y₂) is identified because conditioning on the observed pretest Y₁ and treatment T₁ blocks all three confounding paths T₂ ← Y₁ ← A₁ → A₂ → Y₂, T₂ ← T₁ → A₁ → A₂ → Y₂, and T₂ ←T₁ ← Y₀ ← A₀ → A₁ → A₂ → Y₂. Similarly, one effect of static treatment regimes (T₂, T₃) on Y₃ are sequentially identified (Pearl, 2009).

Now assume that we deleted the records of students with missing values in Y₂ and Y₃ (with nonresponse or attrition being MAR or NMAR) and the records of students who got retained in T₃ because they do not belong to the treatment regime of interest (i.e., we are only interested in evaluating the effect of being retained in year t = 2 without being retained in year t = 3).² Deleting entire student records implies that the ACE needs to be identified from the subsample of remaining students that we visualized by putting dashed boxes around the conditioning variables (Figure 2).³ Since the conditioning (post)-treatment variables Y₂, Y₃, and T₃ are collider variables but also descendants of other collider variables (A₁, A₂, A₃, Y₀, Y₁, Y₃, T₁, and T₂), deleting records induces a series of spurious relations between the unobserved abilities, retention statuses, and outcomes (Elwert & Winship, 2014). If the observed variables fail to block these spurious associations, collider bias distorts the effect estimates of interest.

Figure 2.

Unblockable spurious associations (dashed edges) that bias the effect of retention in T₂ on Y₂ and Y₃ due to deleting entire student records based on Y₂, Y₃, and T₃ (as indicated by the dashed boxes). T_t = the treatment variables; Y_t = the achievement scores; A_t = the unobserved abilities (for t = 0, …, 3).

The dashed edges in Figure 2 only show the bias-inducing spurious relations that cannot be blocked by sequentially conditioning on the observed achievement scores (Y₀, Y₁, and Y₃) or treatment variables (T₁ and T₂). Thus, the retention effects on Y₂ and Y₃ are no longer identified because all noncausal paths from T₂ to Y₂ (and Y₃) that contain dashed edges induce collider bias. These unblockable noncausal paths confound the causal relation between T₂ and Y₂ either directly (T₂ --- Y₂) or indirectly via the unobserved abilities (e.g., T₂ --- A₁ → A₂ → Y₂). The same holds for the sequential identification of the (T₂, T₃) regimes’ effect on Y₃. It is important to realize that the noncausal paths cannot be blocked even if the nonresponse process or retention in T₃ is exclusively determined by the observed variables (MAR). This is so because the collider bias operates predominantly via the unobserved abilities.

However, censoring instead of deleting the corresponding student records is able to circumvent these bias issues but only if nonresponse is MAR and censoring T₃-retained student records is ignorable given the observed pretreatment covariates, precensoring treatment, and outcome history (as pointed out by Vandecandelaere et al.).⁴ First, consider the effect of T₂ on Y₂. Censoring based on T₃ cannot cause any bias because no data are deleted for timepoints 1 and 2; thus, the complete data set is analyzed. Censoring based on Y₂ and Y₃ does not induce collider bias either because no entire student records are deleted—all data up to T₂ remain complete. However, the missingness in Y₂ causes nonresponse bias, but if the nonresponse mechanism can be assumed to be MAR, censoring weights adjust for the bias. Thus, the causal effect of T₂ on Y₂ is identified. Now consider the effect of T₂ on Y₃ where censoring in T₃ actually induces confounding bias (but no collider bias). Since censoring is ignorable conditional on the observed T₂ and Y₂ (at least in our simple data-generating model), sequential censoring weights are able to take care of the confounding bias.

However, if nonresponse is NMAR or censoring T₃-retained students is not ignorable (because not all treatment-outcome confounders are observed), censoring weights will fail to fully remove nonresponse and censoring bias. Vandecandelaere et al. have been fully aware of these issues; instead of deleting student records, they relied on MAR and ignorability assumptions for the censored and addressed the biases via time-varying censoring weights. Although the assumptions need to be justified on subject-matter knowledge, we believe that Vandecandelaere et al. set new standards for future research on grade retention.