Building Bridges Between Structural and Program Evaluation Approaches to Evaluating Policy

Abstract

This paper compares the structural approach to economic policy analysis with the program evaluation approach. It offers a third way to do policy analysis that combines the best features of both approaches. We illustrate the value of this alternative approach by making the implicit economics of LATE explicit, thereby extending the interpretability and range of policy questions that LATE can answer.

1 Introduction

Few topics in economics evoke more passion than discussions about the correct way to do empirical policy analysis.¹ These discussions are sometimes framed as conflicts between “structural” and “reduced form” approaches.² In current usage, “structural” is taken to mean parametric, explicitly formulated, empirical economic models.

“Reduced form” has multiple and sometimes conflicting meanings. One strand of the reduced form approach uses explicit economic models to motivate and interpret empirical analyses and approximates the economic models using simple econometric techniques. Harberger (1964), Shimer and Werning (2008), Chetty (2009), and Einav et al. (2009) are good examples of this approach. Chetty (2009) surveys a large literature in this tradition.

Another strand is the “program evaluation” approach surveyed in Imbens and Wooldridge (2009). This approach focuses on “effects” defined by experiments or surrogates for experiments as the objects of interest, and not the parameters of explicit economic models. It often leaves implicit the economics and policy relevance of the output from its procedures. This paper compares structural and program evaluation approaches—the contrast between the economic parameters featured in the structural approach and the “effects” featured in the program evaluation approach.

Explicit structural models facilitate policy analysis. However, after 60 years of experience with fitting explicit structural models on a variety of data sources, empirical economists have come to appreciate the practical difficulties that arise in identifying, and precisely estimating, the parameters of fully specified structural models that can answer a wide variety of policy questions. There have been many demonstrations of the sensitivity of estimates of structural models to assumptions about functional forms and distributions of unobservables. Liu (1960), Hendry (1980), Sims (1980), and Leamer (1983) gave early warnings about the fragility of standard econometric estimates of explicit economic models. Killingsworth and Heckman (1986) and Pencavel (1986) summarize structural estimates of the effects of taxes and wages on labor supply and report estimates from the literature that are sometimes absurd. Lewis (1986) reports the sensitivity of structural estimates of the causal effect of unionism on wages to the use of alternative methodologies and reports many estimates that are incredibly large. An influential paper by LaLonde (1986) is widely interpreted as having demonstrated that standard structural estimation methods applied to non-experimental data cannot duplicate the estimates obtained from a job training experiment.

These and other studies reported in the literature more than 20 years ago fueled the flight of many empirical economists from structural models, even though Heckman and Hotz (1989) cautioned that many applications of the structural approach by those comparing structural estimates with experimental estimates did not perform specification tests to see if the estimated structural models were concordant with the pre-program data. They show that when such tests are performed, the surviving structural models closely match the estimates produced from the experiment analyzed by LaLonde, findings duplicated for other experiments (see Todd and Wolpin, 2006, Attanasio, Meghir, and Santiago, 2009, Attanasio, Meghir, and Santiago, 2009, and the discussion in Keane, Todd, and Wolpin, 2010).

The perceived failures of the structural methods of the 1970s and 1980s produced two different methodological responses.³ One response—the “program evaluation” approach—was a retreat to statistics, away from the use of explicit economic models either in formulating economic policy questions or in suggesting frameworks for estimating models to answer such questions. The second response was development of a more robust version of the structural approach.

The program evaluation approach replaces the traditional paradigm of economic policy evaluation with the paradigm of the randomized controlled trial. In place of economic models of counterfactuals, practitioners of this approach embrace a statistical model of experiments due to Neyman (1923) and Cox (1958) that was popularized by Rubin (1974, 1978, 1986) and Holland (1986). In this approach, the parameters of interest are defined as summaries of the outputs of experimental interventions. This is more than just a metaphorical usage. Rubin and Holland argue that causal effects are defined only if an experiment can be performed.⁴ This conflation of the separate tasks of defining causality and identifying causal parameters from data is a signature feature of the program evaluation approach. It is the consequence of the absence of clearly formulated economic models. The probability limits of estimators, and not the parameters of well-defined economic models, are often used to define causal effects or policy effects.

The retreat to statistics in the program evaluation literature left a lot of economics behind. A big loss was the abandonment of economic choice theory. Important distinctions about ex ante and ex post outcomes and subjective and objective evaluations that are central to structural econometrics were forgotten.

The influence of the program evaluation approach is widespread. It is now commonplace for many empirical economists to use the language of “treatment” and “control” to describe the comparisons made in empirical policy studies.

The structural response to the perceived failures of the 1970s and 1980s structural models has focused on nonparametric identification and estimation of well-posed economic models within which to conduct policy analyses. This line of work preserves the goals of the Cowles Commission pioneers of the structural approach — to estimate models that can make forecasts for a range of widely different policies and criteria. It is more explicit than the program evaluation approach in articulating economic models. It embraces developments in dynamic economics, game theory, auction theory, and the theory of mechanism design. The richness of the theoretical models (in contrast to the intuitive “effects” promoted in the program evaluation literature) make the fruit of this approach more intellectually interesting. It also produces estimates that cumulate across studies.⁵

On the negative side of the ledger on structural estimation, the often complex computational methods that are required to implement this approach make it less transparent. Replication and sensitivity analyses are often more difficult in this approach than in the program evaluation approach. Economists advocating the program evaluation approach dismiss the structural approach as overly complex and not “credible,” focusing on the statistical and computational properties of estimators as the measure of the credibility of procedures (see, e.g., Angrist and Pischke, 2008, 2010).⁶

An important paper by Marschak (1953) suggests a middle ground between these two camps, and is a motivation for the present paper. Writing in the early 1950s during the first wave of structural econometrics, Marschak noted that for many problems of policy analysis, it is not necessary to identify or estimate fully specified structural models—the goal of structural analysis as conceived by the Cowles pioneers and successor generations of structural economists. Marschak’s Maxim suggests that economists should solve well-posed economic problems with minimal assumptions. All that is required to conduct many policy analyses or to answer many well-posed economic questions are policy invariant combinations of the structural parameters that are often much easier to identify than the individual parameters themselves and that do not require knowledge of individual structural parameters. This approach advocates transparency and empirical robustness as does the program evaluation approach, but it also focuses attention on answering clearly stated economic and policy questions.

This approach is often less computationally intensive and focuses on a more limited range of policy questions than the very large range of policy questions contemplated by the Cowles pioneers. The computationally less demanding models, more transparent sources of identifiability and the relative ease of performing replication and sensitivity analyses give credibility to this approach. At the same time, this approach improves on the program evaluation approach by producing estimates that have clear economic and policy relevance.

The plan of this paper is as follows. In the next section, I review the range of questions that arise in evaluating economic policies and how the program evaluation approach and the structural approach address them. I use the Roy model (1951) and its extensions as examples of widely-used and prototypical structural models. In the following section, I apply Marschak’s Maxim to the analysis of the Roy model. This produces an empirical approach that simplifies policy analysis for a certain class of policies. It links the Roy model to the Local Average Treatment (LATE) framework of Imbens and Angrist (1994). This approach facilitates the interpretation of what LATE and extends the range of questions LATE answers. The final section of the paper summarizes the argument.

This paper is not a position piece for or against any particular statistical methodology. It is about the interpretability of estimates and their policy relevance. It is an exploration of how to get the most out of the data using economics to define the questions of interest and statistics to help answer them.

2 The Structural Versus the Program Evaluation Approach to Evaluating Economic Policies

Policy analysis is all about identifying counterfactual states. Counterfactual policy states are possible outcomes in different hypothetical states of the world. An example is the set of outcomes of the same persons in different tax regimes. Causal comparisons entail contrasts between outcomes in alternative possible states holding factors other than the features of the policy being analyzed the same across the contrasts. Thus the person subject to a particular policy is the same as the person who is not, except for treatment status and, possibly, the outcome associated with treatment status.

The concept of causality at the individual level is based on the notion of controlled variation — variation in treatment holding other factors constant. This is Alfred Marshall’s (1890) ceteris paribus clause which has been the operational definition of causality in economics for over a century.⁷ It is distinct from other notions of causality sometimes used in economics that are based on prediction (e.g. Granger, 1969, and Sims, 1972).⁸

There are two distinct tasks in causal inference and policy analysis: (a) Defining counterfactuals and (b) Identifying causal models from data. Table 1 delineates the two distinct problems.

Table 1.

Two Distinct Tasks that Arise in the Analysis of Causal Models

Task	Description	Requirements
1	Defining the Set of Hypotheticals or Counterfactuals	A Well-specified Economic Theory
2	Identifying Causal Parameters from Data	Mathematical Analysis of Point or Set Identification Joined With Estimation and Testing Theory

The first task entails the application of economic models. Models are descriptions of hypothetical worlds obtained by varying — hypothetically — the factors determining outcomes. Models may be motivated by empirical evidence, and they may crystalize evidence. They are, however, abstract representations of the evidence with an internal logic of their own.

The second task is inference from data. It requires solving the identification problem including solving practical problems of inference from empirical samples.⁹ Economists sometimes differ over what constitutes admissible data for examining any policy question, what prior information should be used and how the prior information should be used. There are no sharp rules to settle these differences.

Part of the controversy surrounding the construction of policy counterfactuals for evaluating policies is a consequence of analysts being unclear about the two distinct tasks represented in Table 1 and sometimes confusing them. Particular methods of estimation (e.g., randomization, matching or instrumental variable estimation) have become associated with the definition of “causal parameters”, because issues of definition and identification are often conflated.

The structural econometric approach to policy evaluation separates these two tasks and emphasizes the role of models in defining hypotheticals and causal effects. Some statisticians reject the use of models in defining causality and seek an assumption-free approach to causal inference and policy analysis (see, e.g., Tukey, 1986).

Any estimator makes assumptions (often implicit) about the behavior of the agents being analyzed. For example, the ability of a randomized controlled trial to identify parameters of interest depends on assumptions about the behavior of the agents being subject to randomization.¹⁰ The structural approach is explicit about these assumptions. The program evaluation approach is often not. Some economists confuse the absence of explicit statements of assumptions with the absence of assumptions.¹¹

The “causal models” advocated in the program evaluation literature are motivated by the experiment as an ideal. They do not clearly specify the theoretical mechanisms determining the set of possible counterfactual outcomes, how hypothetical counterfactuals are realized or how hypothetical interventions are implemented except to compare “randomized” with “nonrandomized” interventions. They focus on outcomes, leaving the model for selecting outcomes and the preferences of agents over expected outcomes unspecified.

The emphasis on randomization or its surrogates, like matching or instrumental variables, rules out a variety of alternative channels of identification of policy effects from data. The emphasis on randomization has practical consequences leading to the conflation of Task 1 with Task 2 in Table 1. Since a randomized protocol is used to define the parameters of interest, this practice sometimes leads to the confusion that randomization is the only way — or at least the best way — to identify causal parameters from data.

The models in the program evaluation literature do not specify the sources of randomness generating variability among agents, i.e., they do not specify why otherwise observationally identical people make different choices. They do not distinguish what is in the agent’s information set from what is in the observing economist’s information set, although the distinction is fundamental in justifying the properties of any estimator for solving selection and evaluation problems. They do not allow for interpersonal interactions inside and outside of markets in determining outcomes that are at the heart of game theory, general equilibrium theory, and models of social interactions and contagion (see,e.g., Aguirregebaria and Mira, 2010; Brown and Matzkin, 1996; Brock and Durlauf, 2001; Durlauf and Young, 2001; Manski, 1993; Tamer, 2003).

The goal of the structural econometrics literature, like the goal of all science, is to understand the causal mechanisms producing effects so that one can use empirical versions of models to forecast the effects of interventions never previously experienced, to calculate a variety of policy counterfactuals and to use theory to guide choices of estimators to interpret evidence and to cumulate evidence across studies. These activities require models for understanding “causes of effects” in contrast to the program evaluation literature that focuses only on the “effects of causes” (Holland, 1986).

Before turning to a specific comparison of the two approaches, it is useful to review the variety of questions that arise in policy analysis. What are the economically interesting questions? The success or failure of any methodology hinges on how well it answers substantive policy questions. Thus, it is helpful to have a list in front of us to examine which questions are addressed or ignored by different approaches.

2.1 Policy Evaluation Problems and Criteria of Interest

Three broad classes of policy evaluation problems arise in economics. Policy evaluation problem one is:

P1 Evaluating the Impacts of Implemented Interventions on Outcomes Including Their Impacts on the Well-Being of the Treated and Society at Large

It is useful to distinguish objective or public outcomes that can in principle be measured by all external observers from “subjective” outcomes that are the evaluations of the agents experiencing treatment (e.g. patients) or the agents prescribing treatment (e.g., physicians). Objective outcomes are intrinsically ex post (“after the fact”) in nature. The literature on program evaluation focuses on ex post objective outcomes.

The structural literature studies both subjective and objective outcomes. Subjective outcomes can be ex ante (“anticipated”) or ex post. The outcome of a medical trial produces both a cure rate and the pain and suffering of the patient. Ex ante anticipated pain and suffering from a procedure may be different from ex post realized pain and suffering. A similar distinction arises in an analysis of the returns to schooling. Monetary returns are only part of total benefits which include important elements of psychic cost.¹² Ex ante evaluations of outcomes by agents may differ from their ex post evaluations. Thus in advance of going to school, students may have expectations about rewards and costs that differ from their realizations. Expected rewards govern responses to market incentives. The impacts in P1 include individual level or population level counterfactuals and their valuations. “Well-being” in P1 includes the valuations of the outcomes of interventions by the agents being analyzed or other parties (e.g., the parents of the agent or “society” at large). They may be ex ante or ex post, and both are of interest in evaluating policy. Regret and anticipation are important aspects of public approval of policies.

P1 is the problem of internal validity. It is the problem of identifying the impacts of an intervention conducted in a given environment. Solving just this problem can be a challenging task and good answers are valuable. However, most economic policy evaluation is conducted with an eye toward the future and toward informing decisions about new policies and applications of old policies to new environments.

There is a second problem frequently encountered in policy analysis.

P2 Forecasting the Impacts (Constructing Counterfactual States) of Interventions Implemented in One Environment in Other Environments, Including Impacts on Well-Being

This is the problem of external validity: taking a treatment parameter or a set of parameters identified in one environment to another environment.¹³

The most ambitious problem is forecasting the effects of a new policy, never previously experienced:

P3 Forecasting the Impacts of Interventions (Constructing Counterfactual States Associated with Interventions) Never Historically Experienced, Including Their Impacts on Well-Being

P3 is a problem that economic policy analysts have to solve daily. Structural econometrics addresses this question. The program evaluation approach does not.

2.2 A Prototypical Economic Model for Policy Evaluation

Abstract discussions of policy evaluation problems become very tedious very fast. To be specific and to keep the discussion focused, consider the following version of the Roy model, which is a useful framework for policy evaluation.¹⁴ The Roy model and its extensions undergird a huge literature in microeconometrics.¹⁵ It was applications of the Roy model that fueled the flight from structural econometrics in the 1980s.

The Roy model is a model of hypothetical outcomes. Economic theory defines the space of possible counterfactual outcomes. It also specifies agent decision rules. It executes Task 1 of Table 1.

Roy (1951) considered an economy where agents face two potential outcomes (Y₀, Y₁) with distribution F_Y0,Y1(y₀, y₁) where “0” refers to the no treatment state and “1” refers to the treated state and (y₀, y₁) are particular values of random variables (Y₀, Y₁). More generally, the set of potential outcomes is ${\left\{Y_s\right\}}_{s\in \mathcal{S}}$ where $\mathcal{S}$ is the set of indices of potential outcomes. In the Roy model $\mathcal{S}=\{0,1\}$. I focus on two outcome models to simplify the exposition. In the application of Gronau (1974) and Heckman (1974), Y₀ is the value of nonmarket time, and Y₁ is the value of market time. In Willis and Rosen (1979), Y₀ is the present value of high school earnings, and Y₁ is the present value of college earnings. In the application of Tunali (2000), Y₀ and Y₁ are incomes in two regions. The central question recognized in this literature is that analysts observe either Y₀ or Y₁, but not both, for any person. In the program evaluation literature, this is called the evaluation problem.

In addition to this problem, there is the selection problem. The values of Y₀ or Y₁ that are observed are not necessarily a random sample of the potential Y₀ or Y₁ distributions. In the original Roy model, an agent selects into sector 1 if Y₁ > Y₀. Let D be an indicator or dummy variable. Then

$$D=1(Y_1>Y_0),$$ (1)

where “1” is a function that is 1 if the condition in the argument holds and is 0 otherwise.

A variety of more general decision rules have been considered in the structural literature. A simple generalization of the Roy model adds cost C. This can be thought of as a cost of moving from “0” to “1”, e.g., tuition in the Willis-Rosen model, costs of work in the Gronau-Heckman model, or costs of migration in the Tunali model. The decision rule becomes

$$D=1(Y_1-Y_0-C>0).$$ (2)

This framework defines a set of counterfactual outcomes and costs (Y₀, Y₁, C) with distribution F_Y0,Y1,C(y₀, y₁, c) and a mechanism for selecting which element of (Y₀, Y₁) is observed for any person. The outcome observed for any person, Y , can be written as

$$Y=D{Y}_1+(1-D)Y_0.$$ (3)

This representation is the Quandt (1958, 1972) switching regression framework.

Agents may make their choices under imperfect information. Let $\mathcal{I}$ denote the agent’s information. In advance of participation, the agent may be uncertain about all components of (Y₀, Y₁, C). The expected benefit is $I_D=E(Y_1-Y_0-C\mid \mathcal{I})$. Then

$$D=1(I_D>0).$$ (4)

Moreover, the decision maker selecting “treatment” may be different than the person who experiences the outcomes (Y₀, Y₁). Thus parents may make schooling decisions for their children; doctors may make treatment decisions for patients. More generally, decisions to participate may entail joint approval of all parties.¹⁶

The ex post objective outcomes are (Y₀, Y₁). The ex ante outcomes are $E(Y_0\mid \mathcal{I})$ and $E(Y_1\mid \mathcal{I})$. The ex ante subjective evaluation is I_D. The ex post subjective evaluation is Y₁ − Y₀ − C. Agents may regret their choices because realizations may differ from anticipations.

The ex ante vs. ex post distinction is essential for understanding behavior. In environments of uncertainty, agent choices are made in terms of ex ante calculations. Yet the treatment effect literature largely reports ex post returns. For example, the recent literature on the returns to schooling reports ex post returns (Autor, Katz, and Kearney, 2005; Katz and Autor, 1999; Katz and Murphy, 1992). Yet it is the analysis of ex ante returns that is needed to understand why, over time, responses to increases in ex post returns to schooling have been so sluggish.¹⁷

Carneiro, Hansen, and Heckman (2001, 2003), Cunha, Heckman, and Navarro (2005, 2006) and Cunha and Heckman (2007) develop econometric methods for distinguishing ex ante from ex post evaluations of social programs. Abbring and Heckman (2007) provide an extensive survey of this literature.¹⁸

In the language of the program evaluation literature, Y₁ − Y₀ is the individual level treatment effect. It is also the Marshallian ceteris paribus causal effect. Because of the evaluation problem, it is generally impossible to identify individual level treatment effects (Task 2). Even if it were possible, Y₁ − Y₀ does not reveal the ex ante subjective evaluation I_D or the ex post assessment Y₁ − Y₀ − C.

Economic policies can operate through changing (Y₀, Y₁) or through changing C. Thus, in the college going example, policies may reduce tuition costs or reduce commuting costs (Card, 2001). They may tax future earnings. The structural approach considers policies affecting both returns and costs.¹⁹

2.3 Population Parameters of Interest

Because it is generally impossible to identify individual-level treatment effects, analysts typically seek to identify parameters defined at the population level. Conventional parameters include the Average Treatment effect (ATE = E(Y₁ − Y₀)), the effect of Treatment on The Treated (TT = E(Y₁ − Y₀ | D = 1)), or the effect of Treatment on the Untreated (TUT = E(Y₁ − Y₀ | D = 0)).

However, in addressing economic policy evaluation questions, a variety of other population level parameters are often more interesting. In positive political economy, the fraction of the population that perceives a benefit from treatment is of interest. This is called the voting criterion and is

$$\mathrm{Pr}(I_D>0)=\mathrm{Pr}(E(Y_1-Y_0-C\mid \mathcal{I})>0).$$

In gauging support for a policy in place, the percentage of the population that ex post perceives a benefit may also be of interest: Pr(Y₁ − Y₀ − C > 0).

More generally, for evaluation of the distribution of welfare, knowledge of the ex ante and ex post joint distributions of outcomes are of interest.²⁰ Because of the evaluation problem, it is very difficult to identify the joint distribution because we generally do not observe Y₀ and Y₁ together. This problem plagues all methodologies including social experiments.²¹

Determining marginal returns to a policy is a central goal of economic analysis. In the generalized Roy model, the margin is specified by people who are indifferent between “1” and “0”, i.e., those for whom I_D = 0. The mean effect of treatment for those at the margin of indifference is

$$E(Y_1-Y_0\mid I_D=0).$$

I discuss approaches for identifying this parameter in Section 3.

2.4 Treatment Effects Versus Policy Effects

Different policies can affect treatment choices and outcomes differently. Each of the population-level treatment effects discussed in the previous subsection can be defined for different policy regimes. Economists are often more interested in the effects of policies on outcomes than in conventional treatment effects.

To illustrate this point, consider the Policy Relevant Treatment Effect (Heckman and Vytlacil, 2001c) which extends the Average Treatment Effect by accounting for voluntary participation in programs. It is designed to address problems P2 and P3. Let “b” represent a baseline policy (“before”) and “a” represent a policy being evaluated (“after”). Let Y^a be the outcome under policy a, while Y^b is the outcome under the baseline. ($Y_0^a,Y_1^a,C^a$) and ($Y_0^b,Y_1^b,C^b$) are outcomes under the two policy regimes.

Policy invariance facilitates the job of answering problems P2 and P3. If some parameters are invariant to policy changes, they can be safely transported to different policy environments. Structural econometricians search for policy invariant “deep parameters” that can be used to forecast policy changes.²²

Under one commonly invoked form of policy invariance, policies keep the potential outcomes unchanged for each person: $Y_0^a=Y_0^b$, $Y_1^a=Y_1^b$, but affect costs (C^a ≠ C^b).²³ A tuition policy in the absence of general equilibrium effects is an example. Invariance of this type rules out social effects including peer effects and general equilibrium effects. Let D^a and D^b be the choice taken under each policy regime. Invoking invariance of potential outcomes, the observed outcomes under each policy regime are Y^a = Y₁D^a+Y₀(1−D^a) and Y^b = Y₁D^b+Y₀(1−D^b). The Policy Relevant Treatment Effect (PRTE) is

$$\mathrm{PRTE}=E(Y^a-Y^b).$$

This is the Benthamite comparison of aggregate outcomes under policies “a” and “b”.²⁴ PRTE extends ATE by recognizing that policies affect incentives to participate (C) but do not force people to participate. Only if C is very large under b and very small under a, so there is universal nonparticipation under b and universal participation under a, would ATE and PRTE be the same parameter.

2.5 The Structural Approach Versus the Program Evaluation Approach

The recent literature on program evaluation in economics draws on a model of counterfactuals and causality attributed to Donald Rubin by Paul Holland (1986).²⁵ It defines causality using experimental manipulations and thereby creates the impression in the minds of many followers of this approach that random assignment is the most convincing way to identify causal models.

Neyman and Rubin postulate counterfactuals ${\left\{Y_s\right\}}_{s\in \mathcal{S}}$.²⁶ They do not develop choice mechanisms that determine which outcome is selected or the subjective evaluations of treatments. There is no discussion of the mechanisms producing the outcomes studied or the relationship between outcomes and choice mechanisms.

Rubin (1986) invokes a portion of the traditional econometric invariance assumptions developed by Hurwicz (1962).²⁷ Since he does not develop models for choice or subjective evaluations, he does not consider the more general invariance conditions for both objective and subjective evaluations that are features of the structural literature.²⁸ The range of issues covered by the two approaches is given in Table 2.²⁹

Table 2.

Comparison of the Aspects of Evaluating Social Policies that are Covered by the Neyman-Rubin Approach and the Structural Approach

	Neyman-Rubin Frame- work	Structural Framework
Counterfactuals for objective outcomes (Y0, Y1)	Yes	Yes
Agent valuations of subjective outcomes (ID)	No (choice-mechanism implicit)	Yes
Models for the causes of potential outcomes	No	Yes
Ex ante versus ex post counterfactuals	No	Yes
Treatment assignment rules that recognize the vol- untary nature of participation	No	Yes
Social interactions, general equilibrium effects and contagion	No (assumed away)	Yes (modeled)
Internal validity (problem P1)	Yes	Yes
External validity (problem P2)	No	Yes
Forecasting effects of new policies (problem P3)	No	Yes
Distributional treatment effects	Noa	Yes (for the general case)
Analyze relationship between outcomes and choice equations	No (implicit)	Yes (explicit)

^aAn exception is the special case of common ranks of individuals across counterfactual states: “rank invariance.” See the discussion in Abbring and Heckman (2007).

The Neyman-Rubin model does not consider many issues discussed in structural econometrics. It is at best an incomplete introduction to some of the important issues in evaluating social policies.³⁰ The analysis in Rubin’s 1974 and 1978 papers is a dichotomy between randomization and non-randomization, and not an explicit treatment of particular selection mechanisms in the non-randomized case as is developed in the structural literature.

The statisticians who have had the greatest impact on the program evaluation literature in economics conflate the two tasks stated in Table 1. The discussion of Holland (1986) illustrates this point and the central role of the randomized controlled trial to the Holland-Rubin analysis. After explicating the “Rubin model,” Holland makes a very revealing claim: there can be no causal effect of gender on earnings because analysts cannot randomly assign gender. This statement confuses the act of defining a causal effect (a purely mental act performed within a model) with empirical difficulties in estimating it.³¹ The local average treatment effect “LATE” parameter of Imbens and Angrist (1994), discussed in Section 3, follows in this tradition and uses instrumental variables as surrogates for randomization. LATE is defined by an instrument and conflates tasks 1 and 2 of Table 1. In Section 3, I present a framework that defines the LATE parameter within the generalized Roy model discussed in Subsection 2.2 that separates issues of definition of parameters from issues of identification.

2.6 Identifying Policy Parameters

The structural approach to policy evaluation addresses policy evaluation questions P1-P3 by estimating models for Y₀, Y₁, and C in different economic environments. Commonly used specifications write

$$Y_1={\mu }_1\left(X\right)+U_1,Y_0={\mu }_0\left(X\right)+U_0,C={\mu }_C\left(Z\right)+U_C,$$ (5)

³² where (X, Z) are observed by the analyst, and U₀, U₁, U_C are unobserved. Economic theory specifies the ingredients in Z and X. In general, there is no “objective” way to choose these conditioning variables. Any argument for inclusion or exclusion of variables has to be made by an appeal to theory — implicit or explicit.

To simplify notation, I define Z to include all of X. Variables in Z not in X are instruments. Write $I_D=E(Y_1-Y_0-C\mid \mathcal{I})={\mu }_D\left(Z\right)-V$ where ${\mu }_D\left(Z\right)=E({\mu }_1\left(X\right)-{\mu }_0\left(X\right)-{\mu }_C\left(Z\right)\mid \mathcal{I})$ and $V=-E(U_1-U_0-U_C\mid \mathcal{I})$. In this notation, choice equation (3) can be expressed as

$$D=1({\mu }_D\left(Z\right)>V).$$ (6)

In the early literature that implemented this approach μ₀(X), μ₁(X), and μ_C(Z) were assumed to be linear in the parameters, and the unobservables were assumed to be normal and distributed independently of X and Z.

The caricature of the structural approach in the recent program evaluation literature is that linearity and normality are essential to this approach. In truth, the essential aspect of the structural approach is joint modeling of outcome and choice equations. Structural econometricians have developed nonparametric identification analyses for the Roy and generalized Roy models. See Heckman and Honoré (1990), Heckman (1990), Ahn and Powell (1993), Andrews and Schafgans (1998), and Das, Newey, and Vella (2003). The field has moved well beyond the parametric functional forms used in the early papers that were the targets of the 1980’s criticism. Traditional distributional and parametric assumptions are relaxed in the recent structural econometric literature. (See Ackerberg, Benkard, Berry, and Pakes, 2007; Athey and Haile, 2007; Matzkin, 1992, 1993, 1994, 2007, 2010a,b; Powell, 1994; Vella, 1998; Abbring and Heckman, 2007; and Keane et al., 2010; for reviews.)

As an illustration of the benefits of the structural approach for solving policy problem P3, consider the analysis of college choice by Cameron and Taber (2004). Suppose that one seeks to know the effects of increases in the expected gross returns to college $E((Y_1-Y_0)\mid \mathcal{I})$ on college choices. From equation (6), if one knows the effects of variations in tuition (C) on college choices, one can use the choice outcomes associated with variations in C to accurately predict the response to changes of equal magnitude (in opposite sign) in expected mean gross returns, even if returns to schooling have never varied in the past.

2.7 Marschak’s Maxim and the Relationship Between the Structural Economics Literature and the Program Evaluation Literature: A Synthesis

Structural models make explicit the preferences and constraints which govern individual decisions, the rules of interaction among agents in market and social settings, and the sources of variability across agents. These features facilitate finding answers to policy questions P1–P3. They are absent from the program evaluation literature.

At the same time, that literature makes fewer statistical assumptions in terms of independence, functional form, exclusion and distributional assumptions than the standard structural estimation literature in econometrics. This is an attractive feature of the program evaluation approach.³³ The greater simplicity of its estimation schemes fosters transparency, replicability, and sensitivity analyses.³⁴ While the structural economics literature has advanced greatly in recent years in terms of producing a robust version of its product, it is an unpleasant fact that fully-specified structural models are often harder to compute. It is more difficult to replicate the estimates from them and to test the sensitivity of estimates to assumptions.

The two approaches can be reconciled by noting that for many policy questions, it is not necessary to identify fully specified models to answer a range of policy questions. It is often sufficient to identify policy-invariant combinations of structural parameters. These combinations are often much easier to identify (i.e., require fewer and weaker assumptions), and do not require knowledge of the particular individual structural parameters that form the combination.

Marschak (1953) recognized that the answers to many policy evaluation questions do not require knowledge of all of the component parts of full structural models. I call this principle Marschak’s Maxim in honor of his insight. Consider estimating the marginal effect of policy expansions. The traditional structural approach identifies the component parts of E(Y₁ − Y₀ | I_D = 0) constructed from estimates of the parameters of equations (5) and (6) and assembles them to estimate the marginal effect of the policy expansion (see Björklund and Moffitt, 1987).

In the next section, I exposit an approach that is consistent with Marschak’s Maxim that directly identifies the combination of parameters that define E(Y₁ − Y₀ | I_D = 0) to solve policy problems, rather than identifying the component parts of the structural model and building it up from the components. Marschak’s Maxim is an application of Occam’s Razor to policy evaluation.³⁵

3 Using Economics to Interpret What LATE Estimates and to Make It Useful For Evaluating a Broad Range of Policies

This section presents an example of an approach to policy evaluation that implements Marschak’s Maxim in the context of LATE. It makes the implicit economics in LATE explicit and thereby expands the range of policy questions that LATE can address.

In the economic theory of policy evaluation, a comparison between marginal benefits and marginal costs determines the optimal size of social programs. For example, to evaluate the optimality of a policy that promotes expansion in college attendance, analysts need to estimate the return to college for the marginal student, and compare it to the marginal cost of the policy. This task requires that analysts identify marginal returns.

In the spirit of the program evaluation literature, in the following discussion, I ignore general equilibrium effects, and I do not emphasize the ex ante and ex post distinction. Both topics are addressed in many papers in the structural approach. To simplify the notation, I keep the conditioning variables X implicit unless it clarifies matters by making them explicit. I follow standard conventions and denote random variables by capital letters and their realizations by the corresponding lower case letters. Thus Z = z means that random variable Z takes the value z. Z is a vector with K components, Z = (Z₁, … , Z_K). z^j means a particular realization of Z, i.e., $z^j=(z_1^j,\dots ,z_K^j)$. I assume for simplicity that all means are finite.

The equation for ex post outcome Y as a function of participation status is

$$Y=\alpha +\beta D+\epsilon ,$$ (7)

where D is a dummy variable indicating participation in a program, β is the individual return to participation or treatment effect and ε is an error term that is unobserved by the analyst. Equation (7) is one representation of the Quandt switching regression model (3). In terms of counterfactual notation, α = μ₀, ε = U₀ and Y₀ = μ₀ + ε, and β = (Y₁ − Y₀) = μ₁ − μ₀ + U₁ − U₀.

Estimating marginal returns to a policy that changes D is a relatively simple task if the effect of the policy is the same for everyone (conditional on X). This is the case when U₁ − U₀ = 0, and ε = U₀ = U₁. In this case, the mean marginal and average returns are the same for all people with the same X.

The recent literature on policy evaluation allows for the possibility that β varies among people even after conditioning on X. Denoting the mean of β by $\stackrel{‒}{\beta }$, the outcome equation can be written as

$$Y=\alpha +\stackrel{‒}{\beta }D+\{\epsilon +(\beta -\stackrel{‒}{\beta })D\},$$ (8)

where $\stackrel{‒}{\beta }={\mu }_1-{\mu }_0$.³⁶ If β is uncorrelated with D, the only new econometric problem that arises in the analysis of (8) that is not present in the traditional analysis of (7) is that the error term is heteroscedastic. As in the case where β is a common parameter shared by everyone with the same X, the main econometric problem for inference about $\stackrel{‒}{\beta }$ is that D is correlated with ε.³⁷

β is statistically independent of D if, given X, agents cannot anticipate their ex post idiosyncratic gains from participation so $\beta -\stackrel{‒}{\beta }$ is independent of D because it is not in the agent’s information set $\mathcal{I}$. Another reason why β might be independent of D is that agents know $\beta -\stackrel{‒}{\beta }$, but do not act on it in choosing D. In both cases, mean marginal returns are the same as mean average returns. Under standard conditions, application of instrumental variables identifies $\stackrel{‒}{\beta }$.³⁸ One does not have to specify the model by which D is selected. All valid instruments identify $\stackrel{‒}{\beta }$.

The recent literature analyzes the less conventional case where agents know and make choices about D with at least partial knowledge of β = Y₁ − Y₀, and the agent knows more about Y₁ − Y₀ than what is in the observing economists’ conditioning set (X, Z). Instrumental variables (IV) do not in general estimate $\stackrel{‒}{\beta }$ and instrumental variables estimators using different instruments have different probability limits.³⁹ Structural selection models can estimate the distribution of β (and hence $\stackrel{‒}{\beta }$) and answer a range of the public policy evaluation questions discussed in Section 2 but under assumptions that are held to be “incredible” in the program evaluation literature.⁴⁰ Angrist and Pischke (2008, 2010) offer the Local Average Treatment Effect (LATE) as a “credible” alternative to structural methods.

Under the conditions reviewed in the next subsection, Imbens and Angrist (1994) show that instrumental variable estimators identify LATE, which measures the mean gross return to treatment for individuals induced into treatment by a change in an instrument. The LATE parameter is widely interpreted as estimating the mean return at the margin defined by manipulation of the instrument.

In general, LATE is not the same as $\stackrel{‒}{\beta }$, but it might be all that is needed to evaluate any particular policy. The key question is “what question does LATE answer?” Unfortunately, the people induced to go into state 1 (D = 1) by a change in any particular instrument need not be the same as the people induced to go to state 1 by policy changes other than those corresponding exactly to the variation in the instrument. A desired policy effect may not directly correspond to the variation captured by the IV. The people induced to change state by the instrument are not identified in LATE. Widely held intuitions about what IV identifies break down in this case since different instruments identify different parameters. Moreover, if there is a vector of instruments that generates choices and the components of the vector are intercorrelated, IV estimates using the components of Z as instruments, one at a time, do not, in general, identify the policy effect corresponding to varying that instrument, keeping all other instruments fixed, the ceteris paribus effect of the change in the instrument. Recent research that builds on and improves LATE shows how to use the generalized Roy model implicit in LATE to estimate the mean marginal returns to alternative ways of producing marginal expansions of programs when variation in the available instruments does not correspond exactly to the variation induced by proposed policies.⁴¹ This research also enables analysts to determine the people who are affected by changes in instruments. I first review LATE and then consider recent extensions of it.

3.1 LATE

LATE is defined by the variation of an instrument. The instrument in LATE plays the role of a randomized assignment. Indeed, randomized assignment is an instrument.⁴² Using the notation of Section 2, Y₀ and Y₁ are potential ex post outcomes. Instrument Z assumes values in $\mathcal{Z},z\in \mathcal{Z}$. D(z) is an indicator of hypothetical choice representing what choice the individual would have made had the individual’s Z been exogenously set to z. D(z) = 1 if the person chooses (is assigned to) 1. D(z) = 0, otherwise. One can think of the values of z as fixed by an experiment or by some other mechanism independent of (Y₀, Y₁). All policies are assumed to operate through their effects on Z. It is assumed that Z can be varied conditional on X.

Imbens and Angrist (1994) make three assumptions to define LATE. Their first assumption is an instrumental variables assumption formulated in terms of a model of counterfactuals:

$$(\mathrm{IA}-1)(Y_0,Y_1,\left\{D\left(z\right)\right\}z\in Z)\coprod Z\mid X$$

where “⫫” denotes independence, and A ⫫ B | X means A is independent of B conditional on X. $(Y_1,Y_0,{\left\{D\left(z\right)\right\}}_{z\in \mathcal{Z}})$ are random variables defined over the population. Assumption (IA-1) states that the values of potential outcomes and potential choices are independent of Z (conditioning on X).

Imbens and Angrist also assume a rank condition:

$$(\mathrm{IA}-2)\mathrm{Pr}(D=1\mid Z=z)\text{is a nontrivial function of z conditional on X}.$$

This says that the distribution of P(Z) = Pr(D = 1 | Z) is nondegenerate conditional on X.

To make IV identify a treatment effect, they invoke a monotonicity condition on the D(z) at the individual level.

$$\left(\mathrm{IA}-3\right)\text{For any two values of Z, say}Z=z^1\text{and}Z=z^2,\text{either}D\left(z^1\right)\ge D\left(z^2\right)\text{for all persons, or}D\left(z^1\right)\le D\left(z^2\right)\text{for all persons}.$$

This condition is a statement across people. z¹ and z² are two different values of vector Z. Fixing the instrument at two values z¹ and z² moves choices across people in the same direction (either in favor of 1 or against it). This condition does not require that for any other two values of Z, say z³ and z⁴, the direction of the inequalities on D(z³) and D(z⁴) have to be ordered in the same direction as they are for D(z¹) and D(z²). It only requires that the direction of the inequalities are the same across people. Thus for any person, D(z) need not be monotonic in z.⁴³

Under these conditions, Imbens and Angrist establish that for two distinct values of Z, z¹ and z², IV applied to (7) identifies

$$\mathrm{LATE}(z^2,z^1)=E(Y_1-Y_0\mid D\left(z^2\right)=1,D\left(z^1\right)=0),$$

if the change from z¹ to z² induces people into the program (D(z²) ≥ D(z¹)).⁴⁴ This is the mean return to participation in the program for people induced to switch treatment status by the change from z¹ to z².⁴⁵

LATE does not identify which people are induced to change their treatment status by the change in the instrument. It also leaves unanswered many of the policy questions discussed in Section 2. For example, if a proposed program changes the same components of vector Z as used to identify LATE but at different values of Z (say z⁴, z³), LATE(z²,z¹) does not identify LATE(z⁴, z³). If the policy operates on different components of Z than are used to identify LATE, one cannot safely use LATE to identify marginal returns to the policy. LATE answers a version of policy problem P1 for objective outcomes, but ignores P2 and P3. It does not, in general, identify treatment on the treated, ATE or the other parameters discussed in Section 2.

3.2 Making Explicit the Implicit Economics of LATE

In a fundamental paper, Vytlacil (2002) shows that the LATE model is equivalent to a nonparametric version of the generalized Roy model. The Imbens-Angrist conditions imply the generalized Roy model, and the generalized Roy model implies the LATE model. Vytlacil’s analysis is the basis for defining LATE abstractly within a well-posed economic model and separating the task of definition (Task 1 of Table 1) from the task of identification (Task 2 of Table 1). Vytlacil’s analysis clarifies the implicit economic assumptions of LATE, what features of the generalized Roy model LATE estimates, and what policy questions LATE addresses. It also extends the range of policy questions that LATE can answer.

By Vytlacil’s theorem, the Imbens-Angrist conditions imply (and are implied by) a continuous latent variable discrete choice model, which represents the individual’s decision to enroll in the program being studied. Recall that I_D (in equation (4)) is the net benefit to the individual of enrolling in the program. A person takes treatment D = 1 (e.g., goes to college) if I_D > 0; otherwise D = 0. Vytlacil shows that the treatment choice equation underlying LATE can be expressed in terms of observed (Z) and unobserved (V) variables that can be represented by equation (6): I_D = μ_D(Z) − V and D = 1 if I_D > 0; D = 0 otherwise, where V is a continuous random variable with distribution function F_V .⁴⁶ μ_D(Z) is defined in the discussion preceding equation (6). V may depend on U₀ and U₁ in a general way.⁴⁷

LATE assumes that (U₀, U₁, V) are independent of Z given X. This relaxes the independence assumption (between X and the unobservables) that was frequently maintained in the early structural literature. The counterfactual choice indicator is generated by choice equation (6): D(z) = 1(μ_D(z) > V). This representation makes explicit the implicit random variable (V) used to define D(z) in the analysis of Imbens and Angrist, and the independence between Z and V that is part of condition (IA-1).

The additive separability between μ_D(Z) and V in the latent index model (6) plays an essential role in LATE. Model (6) is far from the most general possible representation of choices. If choice responses to variations in Z are heterogeneous in a general way, the same change in Z could lead some persons toward and other persons away from participation in the program, and the separability between μ_D(Z) and V in (6) would break down. Another way to say this is that monotonicity condition (IA-3) would be violated.⁴⁸

To understand the economic model implicit in LATE, let P(z) denote the probability of taking treatment (e.g., attending college, D = 1) conditional on Z = z: P (z) ≡ Pr(D = 1|Z = z). From equation (6), P(z) = Pr(μ_D(z) > V) = F_V(μ_D(z)). P(z) is a monotonic transformation of the mean utility function μ_D(z) in discrete choice theory. P(z) is sometimes called the propensity score.

Define random variable U_D = F_V(V), which is uniformly distributed over the interval [0, 1] and thus the p^th quantile of U_D is p, i.e., the proportion of U_D that is p or lower. Different values of U_D correspond to different quantiles of V. We can rewrite (6) using F_V(μ_D(Z)) = P(Z) so that

$$D=1(P\left(Z\right)>U_D).$$ (9)

From the estimated propensity score, one can identify the ex ante net benefit I_D up to scale. Thus, one can determine for each value of Z = z, what proportion of people perceive that they will benefit from the program and the intensity of their benefit. Using the nonparametric identification analyses of Cosslett (1983), Manski (1988), Klein and Spady (1993), and Matzkin (1992, 1993, 1994, 2007), one can nonparametrically identify the distribution of V and the mean valuation μ_D(Z) (up to scale).⁴⁹ Thus, from agent choices, one can supplement the information in LATE and ascertain ex ante subjective evaluations.

As a consequence of Vytlacil’s theorem, the LATE assumptions imply the selection model representation (i.e., the generalized Roy model) and using the selection model representation, one can establish that E(Y | Z = z) = E(Y | P(Z) = P(z)). Under the LATE assumptions, Z enters the model only through its effect on P(Z). This property is called index sufficiency where P(Z) is the index. It is a central property of the LATE model.

As a consequence of Vytlacil’s theorem, one can define LATE(z², z¹) using the latent variable U_D and the values taken by P(Z) when Z = z¹ and Z = z². To do so, I use the property that the Z enter the model only through P(Z).

$$\mathrm{LATE}(z^2,z^1)=E(Y_1-Y_0\mid P\left(z^1\right)\le U_D\le P\left(z^2\right)).$$ (10)

This is the mean gross return to persons whose U_D ∈ [P(z¹), P(z²)].⁵⁰

The LATE parameter can be defined within the generalized Roy model, without reference to an instrument. Thus the LATE produced by economic theory can be expressed as

$$\mathrm{LATE}({\stackrel{‒}{u}}_D,{\underset{‒}{u}}_D)=E(Y_1-Y_0\mid {\underset{‒}{u}}_D\le U_D\le {\stackrel{‒}{u}}_D),$$ (11)

the mean gross return to persons whose $U_D\in [{\underset{‒}{u}}_D,{\stackrel{‒}{u}}_D]$. This is a theoretical construct (Task 1). Proceeding in this fashion, we separate Task 1 of Table 1 from Task 2. A choice of two values of Z (z¹ and z²) picks specific values of [$[{\underset{‒}{u}}_D,{\stackrel{‒}{u}}_D]$] that identify the model-generated LATE from data (say Pr(D = 1 | Z = z¹) = p₁ = u_D and $\mathrm{Pr}(D=1\mid Z=z^2)=p_2={\stackrel{‒}{u}}_D$). This is Task 2.

3.2.1 The Surplus From Treatment and the Marginal Treatment Effect

Using Vytlacil’s theorem, it is possible to understand more deeply what economic questions LATE answers. Toward that end, it is useful to introduce the Marginal Treatment Effect (MTE) and show how it can be used to unify the literature on treatment effects and to make explicit the economic content of LATE.

For P(Z) = p, the mean gross gain of moving from “0” to “1” for people with U_D less than or equal to p is

$$E(Y_1-Y_0\mid P\left(Z\right)\ge U_D,P\left(Z\right)=p)=E(Y_1-Y_0\mid p\ge U_D)=E(Y_1-Y_0\mid {\mu }_D\left(z\right)\ge V).$$ (12)

⁵¹ The first equality follows from the LATE assumption that (Y₀, Y₁) are independent of the instruments Z (IA-1) and hence any functions of Z. The second equality follows from the definition of the propensity score. The mean gross gain in the population (or gross surplus S(p)) that arises from participation in the program for people whose U_D is at or below p is the product of the gain to people whose U_D is at or below p and the proportion of people whose U_D is at or below p: E(Y₁ − Y₀ | p ≥ U_D)p = S(p).

Using Vytlacil’s theorem, we can move from the theory (Task 1 of Table 1) to the data (Task 2 of Table 1) to identify the gross surplus S(p). The mean of Y given P(Z) = p depends on the gross surplus:

$$\begin{array}{cc}\hfill E(Y\mid P\left(Z\right)=p)& =E(Y_0+1(p\ge U_D)(Y_1-Y_0))\hfill \\ \hfill & =E\left(Y_0\right)+\underset{S\left(p\right)}{\underbrace{E(Y_1-Y_0\mid p\ge U_D)p}}.\hfill \end{array}$$ (13)

We can identify the left-hand side of (13) for all values of p in the support of P(Z).⁵² This is Task 2 in Table 1. It is not necessary to impose functional forms to obtain this expression, and one can avoid one of the criticisms directed against 1980’s structural econometrics. The surplus can be defined for all values of p ∈ [0, 1] whether or not the model is identified.

If p is increased by a small change in z, some people near the margin of indifference who chose not to participate in the program would now choose to participate. Small variations in p identify the mean marginal gross return to a policy expansion that changes P(Z). Formally, the marginal increment in outcomes is

$$\frac{\partial E(Y\mid P\left(Z\right)=p)}{\partial p}=E(Y_1-Y_0\mid U_D=p)=\frac{\partial S\left(p\right)}{\partial p}.$$ (14)

⁵³ This is the mean marginal gross return to treatment for persons indifferent between participation in the program or not at mean scale utility level p = U_D, and it is also the marginal change in the gross surplus. The sample analogue of (14) is the local instrumental variable (LIV) estimator of Heckman and Vytlacil (1999, 2005).⁵⁴ Adopting a nonparametric approach to estimating E(Y | P(Z) = p) avoids extrapolation outside of the sample support of P(Z) and produces a data sensitive structural analysis.

A generalization of this parameter defined for other points of evaluation of u_D is the Marginal Treatment Effect (MTE):

$$\mathrm{MTE}\left(u_D\right)\equiv E(Y_1-Y_0\mid U_D=u_D).$$

This parameter is very useful in understanding how to go from IV estimates to policy effects and in interpreting the economics of LATE.⁵⁵ Recall that U_D is a uniform random variable in the interval [0, 1], so that MTE for different u_D values shows how the mean gross returns to the program vary with different quantiles of the unobserved component of the utility of participation, U_D.

Expression (13) can be simplified to

$$E(Y\mid P\left(Z\right)=p)=E\left(Y_0\right)+\underset{S\left(p\right)}{\underbrace{{\int }_0^p\mathrm{MTE}\left(u_D\right)d{u}_D}},$$ (15)

where $S\left(p\right)={\int }_0^p\mathrm{MTE}\left(u_D\right)d{u}_D\text{and}\frac{\partial S\left(p\right)}{{\partial }_p}=\mathrm{MTE}\left(p\right)$. Figure 1 plots E(Y | P(Z) = p) (Figure 1(a)) and its derivative (Figure 1(b)) using values derived from a model discussed in Heckman and Vytlacil (2005). In this analysis, E(Y | P(Z) = p) increases at a diminishing rate in p, so MTE(u_D) is decreasing in u_D, i.e., there are diminishing returns to the marginal entrants attracted into the program by increasing P(z).

Figure 1.

Plots of E(Y|P(Z) = p) and the MTE derived from E(Y|P(Z) = p)

Source: Heckman and Vytlacil (2005)

Notice from (14) that persons with larger values of P(z) identify the return for those with larger values of U_D, i.e., values of U_D that make persons less likely to participate in the program. This is so because marginal increases in P(z) at high levels of P(z) induce those individuals with high U_D values into treatment. This is a consequence of the economic choice model (9). Those with low values of U_D already participate in the program for low values of P(z) = p. A marginal increase in P(z) starting from a high value has no effect on the participation decision of those with low values of U_D. From LIV, it is possible to identify returns at all quantiles of U_D within the support of the distribution of P(Z) to determine which persons (identified by the quantile of the unobserved component of the desire to take the treatment, U_D) are induced to go into the treatment (D = 1) by a marginal change in P(z), i.e., analysts can define the margins of choice traced out by variations in different instruments as they shift P(z). This clarifies what empirical versions of LATE identify by showing that all instruments operate through P(z), and variations around different levels of P(z) identify different stretches of the MTE. I now develop this point.

3.2.2 The Fundamental Role of the Choice Probability in Understanding What Instrumental Variables Estimate When β Depends on D

For any two values of p, say p₁ and p₂, generated by two different values of Z, where p₂ > p₁,

$$\begin{array}{cc}\hfill S\left(p_2\right)-S\left(p_1\right)& =E(Y_1-Y_0\mid p_1\le U_D\le p_2)\mathrm{Pr}(p_1\le U_D\le p_2)\hfill \\ \hfill & =E(Y_1-Y_0\mid p_1\le U_D\le p_2)(p_2-p_1),\hfill \end{array}$$

where the last expression follows from the fact that Pr(p₁ ≤ U_D ≤ p₂) = p₂ − p₁. Thus,

$$S\left(p_2\right)-S\left(p_1\right)={\int }_{p_1}^{p_2}\mathrm{MTE}\left(u_D\right)d{u}_D.$$

This expression can be obtained directly from equation (15). From the definition of LATE,

$$\mathrm{LATE}(p_2,p_1)=\frac{{\int }_{p_1}^{p_2}\mathrm{MTE}\left(u_D\right)d{u}_D}{p_2-p_1}=\frac{S\left(p_2\right)-S\left(p_1\right)}{p_2-p_1}.$$ (16)

Thus LATE is the chord of the gross surplus function over the interval [p₁, p₂]. The model-generated LATE approximates MTE(u_D) over an interval. By the mean value theorem, LATE(p₂, p₁) = MTE(u_D(p₂, p₁)) where u_D(p₂, p₁) is a point of evaluation and u_D(p₂, p₁) ∈ [p₁, p₂]. The model-generated LATE can be identified if there are values of Z, say $\tilde{z}$ and $\tilde{\tilde{z}}$ such that $\mathrm{Pr}(D=1\mid Z=\tilde{z})=p_1$ and $\mathrm{Pr}(D=1\mid Z=\tilde{\tilde{z}})=p_2$. Under standard regularity conditions

$$\underset{p_2\to p_1}{\lim }\mathrm{LATE}(p_2,p_1)=\mathrm{MTE}\left(p_1\right).$$

If we partition the support of u_D into M discrete and exhaustive intervals

$$[u_{D,0},u_{D,1}),[u_{D,1},u_{D,2}),\dots ,[u_{D,M-1},u_{D,M}],$$

where u_D,0 = 0 and u_D,M = 1, we can define

$$E(Y\mid U_D\le u_{D,k})=E\left(Y_0\right)+\sum _{j=1}^k\mathrm{LATE}(u_{D,j},u_{D,j-1}){\eta }_j,$$

where η_j = u_D,j − u_D,j−1. Thus

$$E\left(Y\right)=E\left(Y_0\right)+\sum _{j=1}^M\mathrm{LATE}(u_{D,j},u_{D,j-1}){\eta }_j$$ (17)

which is the counterpart to expression (15) when p = 1. It shows how mean income can be represented as a sum of incremental gross surpluses above E(Y₀).

These expressions are derived from an underlying theoretical model. Whether or not the components can be identified from the data depends on the support of Pr(D = 1 | Z). If Pr(D = 1 | Z = z) assumes values at only a discrete set of support points, say p₁ < p₂ < ⋯ < p_L, we can only identify LATE in intervals with boundaries defined by u_D,ℓ = p_ℓ, ℓ = 1, … , L.

MTE(u_D) and the model-generated LATE (10) are structural parameters in the sense that changes in Z (conditional on X) do not affect MTE(u_D) or theoretical LATE. They are invariant with respect to all policy changes that operate through Z. Conditional on X, one can transport MTE and the derived theoretical LATEs across different policy environments and different data sets. These policy invariant parameters implement Marschak’s Maxim since they are defined for combinations of the parameters of the generalized Roy model. Instead of separately estimating the components of the selection model presented in Section 2.6, one can identify an interpretable marginal gross benefit function by using the derivative of E(Y | P(Z) = p).

This deeper understanding of LATE facilitates its use in answering out-of-sample policy question P3 for policies that operate through changing Z. Thus if one computes a LATE for any two pairs of values Z = z¹, and Z = z², with associated probabilities Pr(D = 1 | Z = z¹) = P(z¹) = p₁ and Pr(D = 1 | Z = z²) = P(z²) = p₂, one can use it to evaluate any other pair of policies $\tilde{z}$ and $\tilde{\tilde{z}}$ such that

$$\mathrm{Pr}(D=1\mid Z=z^1)=\mathrm{Pr}(D=1\mid Z=\tilde{z})=p_1$$

and

$$\mathrm{Pr}(D=1\mid Z=z^2)=\mathrm{Pr}(D=1\mid Z=\tilde{\tilde{z}})=p_2.$$

Thus, one can use an empirical LATE determined for one set of instrument configurations to identify outcomes for other sets of instrument configurations that produce the same p₁ and p₂, i.e., one can compare any policy described by $\tilde{z}\in \{z\mid P\left(z\right)=p_1\}$ with any policy $\tilde{\tilde{z}}\in \{z\mid P\left(z\right)=p_2\}$ and not just the policies associated with z¹ and z² that identify the sample LATE. This is a powerful result and enables analysts to solve policy evaluation question P3 to evaluate new policies never previously implemented if they can be cast in terms of variations in P(Z) over the empirical support on Z.⁵⁶

Variation in different components of Z produce variation in P(Z). Analysts can aggregate the variation in different components of Z into the induced variation in P(Z) to trace out MTE(u_D) over more of the support of u_D than would be possible using variation in any particular component of Z. The structural approach enables analysts to determine what stretches of the MTE different instruments identify and to determine the margin of U_D identified by the variation in an instrument.

Figure 2 reproduces the MTE displayed in Figure 1(b) on a different scale. Consider values of instruments that are associated with P(z) = p₂ and P(z) = p₁ They identify the MTE at a value of u_D in the interval u_D ∈ [p₁, p₂], as depicted in the graph. This is LATE(p₂, p₁). If there is continuous variation in Z it could be used to trace out the entire interval of MTE for P(Z) ∈ [p₁, p₂], using LIV. Independent of any instrument, we can define the LATE and MTE using the underlying economic model. In this fashion, we separate the task of definition of parameters from the task of identifying them.

Figure 2.

MTE as a function of u_D: What sections of the MTE different values of the instruments and different instruments approximate.

Instruments associated with higher values of P(Z), [p₃, p₄], identify the LATE in a different stretch of the MTE associated with higher values of u_D. Thus different instruments can identify different parameters. Continuous instruments can identify entire stretches of the MTE while discrete instruments define the MTE at discrete points of the support (i.e., the LATE associated with the interval defined by the values assumed by P(Z)). As a consequence of Vytlacil’s Theorem, one can identify the intervals of u_D implicit in using LATEs formed using different instruments.

If the MTE does not depend on u_D, E(Y | P(Z) = p) = E(Y₀) + (μ₁ − μ₀)p, and all instruments identify the same parameter: $\stackrel{‒}{\beta }={\mu }_1-{\mu }_0$. In this case, MTE is a flat line parallel to the u_D axis. This is the case traditionally assumed in the analysis of instrumental variables.

A test of whether MTE(u_D) depends on u_D, or a test of nonlinearity of E(Y | P(Z) = p) in p, is a test of the whether different instruments estimate the same parameter.⁵⁷ The LATE model and its extensions overturn the logic of the Durbin (1954)–Wu (1973)–Hausman (1978) test for overidentification. Variability among the estimates from IV estimators based on different instruments may have nothing to do with the validity of any particular instrument, but may just depend on what stretch of the MTE they approximate.

3.3 All Treatment Effects Are Weighted Averages of the MTE

Using the economics implicit in LATE unifies and interprets the literature on treatment effects. All of the conventional treatment effects featured in the program evaluation literature can be written as weighted averages of the MTE or the structural LATEs where the weights can be estimated from the data over the sample support of P(Z). Thus for treatment effect e,

$$\text{Treatment Effect}\left(e\right)={\int }_0^1\mathrm{MTE}\left(u_D\right)h^e\left(u_D\right)d{u}_D,$$ (18)

where h^e(u_D) is a weighting function. For ATE, h^e(u_D) = 1 and $\mathrm{ATE}={\int }_0^1\mathrm{MTE}\left(u_D\right)d{u}_D$. Using the linearity of the integral, one can always break (18) into a sum of components over the intervals [u_D,0, u_D,1), [u_D,1, u_D,2), …, [u_D,M−1, u_D,M] to obtain

$$\text{Treatment effect}\left(e\right)=\sum _{j=1}^M{\int }_{u_{D,j-1}}^{u_{D,j}}\mathrm{MTE}\left(u_D\right)h^e\left(u_D\right)d{u}_D,$$

where u_D,j > u_D,j−1 > u_D,j−2 > ⋯. By the mean value theorem,⁵⁸ we may express each of the integrals in the sum as

$${\int }_{u_{D,j-1}}^{u_{D,j}}\mathrm{MTE}\left(u_D\right)h^e\left(u_D\right)d{u}_D=h^e\left(u_{j,j-1}^{\ast }\right){\int }_{u_{D,j-1}}^{u_{D,j}}\mathrm{MTE}\left(u_D\right)d{u}_D$$

for some $u_{j,j-1}^{\ast }\in [u_{D,j-1},u_{D,j}]$. From (16) we obtain

$$h^e\left(u_{j,j-1}^{\ast }\right){\int }_{u_{D,j-1}}^{u_{D,j}}\mathrm{MTE}\left(u_D\right)d{u}_D=h^e\left(u_{j,j-1}^{\ast }\right)(u_{D,j}-u_{D,j-1})\mathrm{LATE}(u_{D,j},u_{D,j-1}).$$

Thus we may write expression (18) as

$$\text{Treatment Effect}\left(e\right)=\sum _{j=1}^M\mathrm{LATE}(u_{D,j},u_{D,j-1}){\tilde{h}}^e\left(u_{j,j-1}^{\ast }\right),$$

where ${\tilde{h}}^e\left(u_{j,j-1}^{\ast }\right)=h^e\left(u_{j,j-1}^{\ast }\right)(u_{D,j}-u_{D,j-1})$. In the special case of ATE, $h_e\left(u_{j,j-1}^{\ast }\right)=1$ and

$$\mathrm{ATE}=\sum _{j=1}^M\mathrm{LATE}(u_{D,j},u_{D,j-1}){\eta }_j,$$

where η_j = u_D,j − u_D,j−1.

The Policy Relevant Treatment Effect (PRTE), defined in Section 2, is

$$\begin{array}{cc}\hfill & E(Y\mid \text{Alternative Policy}a)-E(Y\mid \text{Baseline Policy}b)\hfill \\ \hfill & E\left(Y^a\right)-E\left(Y^b\right)\hfill \\ \hfill & {\int }_0^1\mathrm{MTE}\left(u_D\right)h^{\mathrm{PRTE}}\left(u_D\right)d{u}_D,\hfill \end{array}$$

where $h^{\mathrm{PRTE}}\left(u_D\right)=F_P^b\left(u_D\right)-F_P^a\left(u_D\right)$, and $F_P^b$ is the distribution of P(Z) under policy b, and $F_P^a$ is the distribution of P(Z) under policy a.⁵⁹ Using the mean value theorem, one can generate a counterpart expression in terms of LATEs.

The PRTE weights MTE(u_D) by the change in the distribution of the probabilities of participation at different values of u_D.⁶⁰ Thus for a typical MTE as graphed in Figure 1(a), if a policy shifts the distribution of participants toward low u_D values, it generates a positive PRTE, since MTE(u_D) is higher for low u_D values than it is for high u_D values.

Notice that the same MTE(u_D) can be used to evaluate the impacts of a variety of different policies. MTE(u_D) is a structural function since it is invariant across policies that affect the distribution of the P(Z) but not the distribution of the potential ex post outcomes. We can evaluate the effect of new policies never previously experienced if we can characterize the distributions of P(Z) for those policies.

Table 3 displays the weights of the MTE that produce the traditional treatment parameters. All of the weights can be estimated from the distribution P(Z).⁶¹ There are corresponding expressions for the case of discrete support for P(Z) that can be obtained using the mean value theorem. The weights integrate to 1. When β is independent of D, MTE(u_D) does not depend on $u_D(={\mu }_1-{\mu }_0=\stackrel{‒}{\beta })$, so all treatment parameters equal $\stackrel{‒}{\beta }$.

Table 3.

MTE Weights For Different Treatment Parameter and IVs. (F_p is the distribution of P. f_P is its density.)

Treatment Parameter Weights	$h^{ATE}\left(u_D\right)=1$
	$h^{TT}\left(u_D\right)=\left[{\int }_{u_D}^1{f}_P\left(p\right)dp\right]\frac{1}{E\left(P\right)}$
	$h^{TT}\left(u_D\right)=\left[{\int }_0^{u_D}{f}_P\left(p\right)dp\right]\frac{1}{E(1-P)}$
	$h^{PRTE}\left(u_D\right)=F_P^b\left(u_D\right)-F_P^a\left(u_D\right)$
IV Weights	$h^{IV}\left(u_D\right)=\left[{\int }_{u_D}^1(p-E\left(P\right))f_P\left(p\right)dp\right]\frac{1}{Var\left(P\right)}$ for P(Z) as an instrument
	$h^{IV}\left(u_D\right)=\frac{{\int }_{u_D}^1{\int }_{-\infty }^{\infty }(j-E\left(J\right))f_{J,P}(j,p)djdp}{\mathrm{Cov}(J,P)}$ for a general instrument J(Z)*, a function of Z

^*f_J,P(j, p) is the joint density of J and P. For derivations of these weights, see Heckman and Vytlacil (1999, 2005, 2007b).

Figure 3 plots an MTE taken from the analysis of Heckman and Vytlacil (2005) and the weights for MTE associated with ATE, TT, and TUT for a case where β is not independent of D. ATE weights u_D evenly. TT oversamples low values of u_D (associated with persons more likely to participate in the program). TUT oversamples high u_D. In this example, because MTE(u_D) is decreasing in u_D, TT > ATE > TUT.

Figure 3.

MTE and the weights for the marginal treatment effect for different parameters for the model graphed in Figure 1

Source: Heckman and Vytlacil (2005)

3.4 What Does Conventional IV estimate?

In most empirical studies, more than the two values of Z are used to construct IV estimates. For this case, Imbens and Angrist (1994) use weights developed by Yitzhaki (1989) to express IV as a weighted average of component LATEs defined for different values of the instruments. The weights used by Imbens-Angrist are positive for each component LATE for the special instrument they consider (P(Z) or some monotonic function of P(Z)). Since MTE(u_D) may change sign over the interval u_D ∈ [0, 1], the IV may be negative even if some portion of the MTE(u_D) is positive. For general instruments that are not monotonic functions of P(Z), the output of IV is even more ambiguous. The IV weights can be negative over regions of u_D ∈ [0, 1]. Thus an IV based on general instruments may have a sign opposite to the true causal effect as defined by the MTE. Even if each component of LATE is positive, in the general case IV can be negative. Negative components of MTE(u_D) weighted by negative weights can generate a positive IV.

This analysis is constructive because the weights can be identified from the data. Analysts can ascertain whether or not the weights are negative and over what regions of u_D. In this subsection I analyze the case where P(Z) is the instrument and the weights are positive. I analyze the general case in section 3.6 below.

To understand what IV identifies, consider a linear regression approximation of E (Y | P(Z) = p):

$$E^{\ast }(Y\mid P\left(Z\right)=p)=a+bp,$$

⁶² where

$$b=\frac{\mathrm{Cov}(Y,P\left(Z\right))}{\mathrm{Var}\left(P\left(Z\right)\right)}=\frac{\mathrm{Cov}(E(Y\mid P\left(Z\right)),P\left(Z\right))}{\mathrm{Var}\left(p\left(Z\right)\right)}.$$

b is the same as the IV estimate of “the Effect” of D on Y using P(Z) as an instrument since Cov(P(Z), D) = Var(P(Z)).⁶³

Using condition (IA-1) (in particular that P(Z) is independent of Y₀), and expression (15), we obtain

$$b=\frac{\mathrm{Cov}(Y,P\left(Z\right))}{\mathrm{Var}\left(P\left(Z\right)\right)}=\frac{\mathrm{Cov}(S\left(P\left(Z\right)\right),P\left(Z\right))}{\mathrm{Var}\left(P\left(Z\right)\right)}=\frac{\mathrm{Cov}\left({\int }_0^{P\left(Z\right)}\mathrm{MTE}\left(u_D\right)d{u}_D,P\left(Z\right)\right)}{\mathrm{Var}\left(P\left(Z\right)\right)}.$$ (19)

Note that when MTE(u_D) is constant in $u_D(\mathrm{MTE}\left(u_D\right)={\mu }_1-{\mu }_0=\stackrel{‒}{\beta })$ so that β is independent of D, the numerator of the preceding expression simplifies to

$$\mathrm{Cov}\left({\int }_0^{P\left(Z\right)}\mathrm{MTE}\left(u_D\right)d{u}_D,P\left(Z\right)\right)=\mathrm{Cov}(\stackrel{‒}{\beta }P\left(Z\right),P\left(Z\right))=\stackrel{‒}{\beta }\mathrm{Var}\left(P\left(Z\right)\right)$$

so $b={\mu }_1-{\mu }_0=\stackrel{‒}{\beta }$. This is the traditional result for IV. In this case, the marginal surplus is the same as the average surplus for all values of p. Expression (19) arises because D depends on β (=Y₁−Y₀), something assumed away in traditional applications of IV. As a consequence, in general, the marginal surplus is not the average surplus.

An explicit expression for the numerator of (19) is

$$\mathrm{Cov}(Y,P\left(Z\right))={\int }_0^1\left[{\int }_0^p\mathrm{MTE}\left(u_D\right)d{u}_D\right](p-E\left(P\right))f_P\left(p\right)dp.$$

Reversing the order of the integration of the terms on the right-hand side and respecting the requirement that 0 < u_D < p < 1, we obtain

$$\begin{array}{cc}\hfill b=\frac{\mathrm{Cov}(Y,P\left(Z\right))}{\mathrm{Var}\left(P\left(Z\right)\right)}& =\frac{{\int }_0^1\mathrm{MTE}\left(u_D\right)\left[{\int }_{u_D}^1(p-E\left(P\right))f_P\left(p\right)dp\right]d{u}_D}{\mathrm{Var}\left(P\left(Z\right)\right)}\hfill \\ \hfill & ={\int }_0^1\mathrm{MTE}\left(u_D\right)h_{P\left(Z\right)}^{IV}\left(u_D\right)d{u}_D\hfill \end{array}$$

where

$$h_{P\left(Z\right)}^{\mathrm{IV}}\left(u_D\right)=\frac{{\int }_{u_D}^1(p-E\left(P\right))f_P\left(p\right)dp}{\mathrm{Var}\left(P\left(Z\right)\right)}.$$

^64,65 An alternative expression for the weight is as the mean of left truncated P(Z):

$$h_{P\left(Z\right)}^{\mathrm{IV}}\left(u_D\right)=\frac{E(P\left(Z\right)-E\left(P\left(Z\right)\right)\mid P\left(Z\right)>u_D)\mathrm{Pr}(P\left(Z\right)>u_D)}{\mathrm{Var}\left(P\left(Z\right)\right)}$$

which shows that the weight on the MTE(u_D) is non-negative for all u_D.

The weights can be estimated from the sample distribution of P(Z). The weights for P(Z) as an instrument have a distinctive profile. It is readily verified that they are non-negative, reach a peak at the mean of the distribution of P(Z), and are zero at the extremes u_D = 0 and u_D = 1. The weights integrate to 1.⁶⁶ Figure 4 plots the IV weights and the MTE from a study by Heckman and Vytlacil (2005). Comparing the IV weights with the weights for different treatment effects enables analysts to determine how closely IV approximates any particular mean treatment effect.

Figure 4.

The MTE and IV weights as a function of u_D.

Source: Heckman and Vytlacil (2005).

For discrete valued instruments mapped into P(z¹) = p₁ < P(z²) = p₂ < ⋯ < P(z^L) = p_L,

$$\mathrm{IV}=\sum _{\ell =1}^{L-1}\mathrm{LATE}(p\ell +1,p\ell ){\lambda }_{\ell }$$

where ${\lambda }_{\ell }=\frac{1}{\mathrm{Var}\left(P\left(Z\right)\right)}{\sum }_{t>\ell }^L(p_t-E\left(P\right))f_P\left(p_t\right)$ and f_P(p_t) is the probability that P(Z) = p_t. For a proof, see Heckman, Urzua, and Vytlacil (2006) or Appendix B.

3.5 The Problem of Limited Support

Before turning to the analysis of general instruments, I consider the problem of limited support for P(Z) for the special instrument P(Z) used by Imbens and Angrist. Analysis of this simple case establishes principles that apply to more general instruments. While the various treatment parameters can be defined from the generalized Roy model, they may not necessarily be identified from the data. Both the nonparametric structural approach and the nonparametric program evaluation approach avoid the problem of extrapolating estimates outside the support of the data.⁶⁷ The early literature in structural econometrics imposed strong functional forms (typically linearity) to obtain estimates.⁶⁸ P(Z) may not be identified over the full unit interval. Thus the lowest sample value of P(Z) may exceed zero and the largest value of P(Z) may be less than 1.

In addition, P(Z) may only assume discrete values. This limits the identifiability of MTE. In this case, only LATE over intervals of u_D ∈ [0, 1] can be identified from the values of P(Z) = P(z) associated with the discrete instruments.⁶⁹

One approach to this problem developed by Manski (1990, 1995, 2003) is to produce bounds on the treatment effects. Heckman and Vytlacil (1999, 2000, 2001a,b, 2007b) develop specific bounds for the generalized Roy model that underlies the LATE model. The bounds developed in the literature are for conventional treatment effects and not for policy effects.

Carneiro, Heckman, and Vytlacil (2010) consider an alternative approach based on marginal policy changes. Many proposed policy changes are incremental in nature, and a marginal version of the PRTE is all that is required to answer questions of economic interest. When some instruments are continuous, it is possible under the conditions in their paper to identify a marginal version of PRTE (MPRTE). MPRTE is in the form of representation (18) where the weights can be identified from the data and the support requirements are more limited than the conditions required to identify PRTE for large changes in policies. Their paper presents a derivation of the weights for classes of policy expansions.⁷⁰ Application of these data sensitive nonparametric approaches enables analysts to avoid one source of instability of the estimates of policy effects that plagued 1980s econometrics.

3.6 More General Instruments

Typically, economists use a variety of instruments one at a time and not just P(Z) or some function of P(Z), or some function of P(Z), as an instrument and compare the resulting estimates (see, e.g, Card, 1999, 2001). When there is selection on the basis of gross gains so that the marginal gross surplus is not the same as the average gross surplus, different instruments identify different parameters. IV is a weighted average of MTEs where the weights integrate to 1 and can be estimated from sample data. However, in the case of general instruments, the weights can be negative over stretches of u_D.

Consider using the first component of Z, Z₁, as an instrument for D in equation (7). Suppose that Z contains two or more elements (Z = (Z₁, … , Z_K), K ≥ 2). The economics implicit in LATE informs us that Z determines the distribution of Y through P(Z). Any correlation between Y and Z₁ arises from the statistical dependence between Z₁ and P(Z) operating to determine Y .

The IV estimator based on Z₁ is

$$I{V}_{Z_1}=\frac{\mathrm{Cov}(Y,Z_1)}{\mathrm{Cov}(D,Z_1)}=\frac{\mathrm{Cov}(E(Y\mid Z_1),Z_1)}{\mathrm{Cov}(D,Z_1)}.$$

Note, however, that choices (and hence Y) are generated by the full vector of Z operating through P(Z). The analyst may only use Z₁ as an instrument but the underlying economic model informs us that the full vector of Z determines observed Y . Conditioning only on Z₁ leaves uncontrolled the influence of the other elements of Z on Y . This is a new phenomenon in IV that would not be present if D did not depend on β(= Y₁ − Y₀). An IV based on Z₁ identifies an effect of Z₁ on Y as it operates directly through Z₁ (Z₁ changing P(Z₁, … , Z_K)) holding other elements in Z constant and indirectly through the effect of Z₁ as it covaries with (Z₂, … , Z_K), and how those variables affect Y through their effect on P(Z).

A linear regression analogy helps to fix ideas. Suppose that outcome Q can be expressed as a linear function of W = (W₁, … , W_L), an L-dimensional regressor:

$$Q=\sum _{\ell =1}^L{\varphi }_{\ell }{W}_{\ell }+\epsilon ,$$

where E(ε | W) = 0. If we regress Q only on W₁, we obtain in the limit the standard omitted variable result that the estimated “effect” of W₁ on Q is

$$\frac{\mathrm{Cov}(Q,W_1)}{\mathrm{Var}\left(W_1\right)}={\varphi }_1+\sum _{\ell =2}^L{\varphi }_{\ell }\frac{\mathrm{Cov}(W_{\ell },W_1)}{\mathrm{Var}\left(W_1\right)},$$ (20)

where φ₁ is the ceteris paribus direct effect of W₁ on Q and the summation captures the rest of the effect (the effect on Q of W₁ operating through covariation between W₁ and the other values W_ℓ, ℓ ≠ 1). An analogous problem arises in using one instrument at a time to identify “the Effect” of Z₁.

Thus if the analyst does not condition on the other elements of Z in using Z₁ as an instrument, the margin identified by variations of Z₁ does not in general correspond to variations arising solely from variations in Z₁, holding the other instruments constant. The margin of choice implicitly defined by the variation in Z₁ is difficult to interpret and depends on the parameters of the generalized Roy model generating outcomes as well as on the sample dependence between instrument Z₁ and P(Z). Thus an IV based on Z₁ mixes causal effects with sample dependence effects among the correlated regressors.⁷¹ In a study of college going, if Z₁ and Z₂ are tuition and distance to college, respectively, the instrument Z₁ identifies the direct effect of variation in tuition on college attendance and the effect of distance to college on college attendance as distance covaries with tuition in the sample used by the analyst. This is not the ceteris paribus effect of a variation in tuition. It does not correspond to the answer needed to predict the effects of a policy that operates solely through an effect on tuition. In models in which D depends on β, the traditional instrumental variable argument that analysts do not need a model for D and can ignore other possible determinants of D other than the instrument being used, breaks down. To interpret which margin is identified by different instruments requires that the analyst specify and account for all of the Z that form P(Z). Since different economists may disagree on the contents of Z, different economists using Z₁ on the same data will obtain the same point estimate but may disagree about the interpretation of the margin identified by variation in Z₁.

To establish these points, note that as a consequence of Vytlacil’s theorem, Z enters the distribution of Y only through P(Z). Thus the conditional distribution of Y given Z₁ = z₁ operates through the effect of Z₁ as it affects P(Z). That is a key insight from Vytlacil’s theorem. Thus

$$E(Y\mid Z_1=z_1)={\int }_1^{0}E(Y\mid P\left(Z\right)=p)g_{P\left(Z\right)\mid Z_1}(p,z_1)dp$$

where g_P(Z)|Z1 (p, z₁) is the conditional density of P(Z) given Z₁ = z₁.⁷² Putting all of these ingredients together, and using (15), we obtain

$$E(Y\mid Z_1=z_1)=E\left(Y_0\right)+{\int }_0^{1}S\left(p\right)g_{P\left(Z\right)\mid Z_1}(p,z_1)dp=E\left(Y_0\right)+{\int }_0^1\underset{S\left(p\right)}{\underbrace{\left[{\int }_0^p\mathrm{MTE}\left(u_D\right)d{u}_D\right]}}{g}_{P\left(Z\right)\mid Z_1}(p,z_1)dp.$$

Using this expression to compute $\frac{\mathrm{Cov}(Y,Z_1)}{\mathrm{Cov}(D,Z_1)}$, we obtain

$$\begin{array}{cc}\hfill I{V}_{Z_1}& =\frac{{\int }_{-\infty }^{\infty }(z_1-E\left(Z_1\right)){\int }_0^{1}S\left(p\right)g_{P\left(Z\right)\mid Z_1}(p,z_1)dpd{z}_1}{\mathrm{Cov}(Z_1,D)}\hfill \\ \hfill & =\frac{{\int }_{-\infty }^{\infty }(z_1-E\left(Z_1\right)){\int }_0^1\left[{\int }_0^p\mathrm{MTE}\left(u_D\right)d{u}_D\right]g_{P\left(Z\right)\mid Z_1}(p,z_1)dpd{z}_1}{\mathrm{Cov}(Z_1,D)}.\hfill \end{array}$$

This expression integrates the argument in the numerator with respect to u_D, p, and z₁ in that order. Reversing the order of integration to integrate with respect to p, z₁, and u_D in that order, we obtain

$$I{V}_{Z_1}={\int }_0^1\mathrm{MTE}\left(u_D\right)h_{Z_1}^{IV}\left(u_D\right)d{u}_D$$

where

$$h_{Z_1}^{IV}\left(u_D\right)=\frac{{\int }_{-\infty }^{\infty }(z_1-E\left(Z_1\right)){\int }_{u_D}^1{g}_{P\left(Z\right)\mid Z_1}(p,z_1)dpd{z}_1}{\mathrm{Cov}(Z_1,D)}.$$

The weight integrates to 1 but can be negative over stretches of u_D.⁷³ At the extremes (u_D = 0, 1), the weights are zero.

An illuminating way to represent this weight is

$$h_{Z_1}^{IV}\left(u_D\right)=\frac{E(Z_1-E\left(Z_1\right)\mid P\left(Z\right)>u_D)\mathrm{Pr}(P\left(Z\right)>u_D)}{\mathrm{Cov}(Z_1,D)}$$

As u_D is increased, the censored (by the condition P(Z) > u_D) mean of (Z₁ − E(Z₁)) may switch sign, and hence the weights may be negative over certain ranges. Thus the IV estimator may have a sign opposite to the true causal effect (defined by the MTE).

Figure 6 illustrates this possibility for the distribution of the data Z = (Z₁, Z₂) shown in Figure 5, where Z is continuously distributed. The support of the data only permits identification of P(Z) over the interval [0.1, 0.9]. Thus none of the conventional treatment parameters is identified. From LIV, we can identify the MTE over the interval [0.1, 0.9]. We can also identify the weights over this interval. For values of u_D > 0.65, the weights are negative in this example. Thus it is possible that the IV based on Z₁ can be negative even if the MTE is everywhere positive. Table 4, taken from Heckman, Urzua, and Vytlacil (2006) shows how three different distributions of Z for the same underlying policy-invariant model with the same ATE can produce very different IV estimates.

Figure 6.

MTE and IV weights for a general instrument Z₁, a component of Z = (Z₁, Z₂).

Figure 5.

Joint density of instruments Z = (Z₁, Z₂)

Table 4.

IV estimator for three different distributions of Z but the same generalized Roy model.

Data Distribution	IV	ATE
1	0.434	0.2
2	0.078	0.2
3	−2.261	0.2

Source: Heckman, Urzua, and Vytlacil (2006, Table 3).

This analysis elucidates the benefits and limitations of the method of randomized controlled experiments. Experiments that manipulate Z₁ independently of other components of Z isolate the effects of Z₁ on outcomes in comparison with the effects obtained by sample variation in Z₁ correlated with other components of Z. Neither set of variations may identify the returns to any given policy unless the experimentally induced variation corresponds exactly to the variation induced by the policy. Economists can use experimental variation to identify the MTE. The features of a proposed policy are described by its effects on the PRTE weights as it affects the distribution of P(Z). Proceeding in this way, one can use experiments to address a range of questions beyond the effects directly identified by the experiment.

Using the implicit economic theory underlying LATE, economists can do better than just report an IV estimate. We can be data sensitive but not at the mercy of the data. We can determine the MTE (or LATEs) over the identified regions of u_D in the empirical P(Z). We can also determine the weights over the empirical support of P(Z) to determine whether they are negative or positive. We can bound estimates of the unidentified parameters. (See Heckman and Vytlacil, 1999, 2001a,b, 2007b.) We can construct the effects of policy changes for new policies that stay within the support of P(Z) (see Carneiro, Heckman, and Vytlacil, 2010).

3.7 Policy Effects, Treatment Effects, and IV

A main lesson of this paper is that policy effects are not generally the same as treatment effects and, in general, neither are produced from IV estimators. Since randomized assignments of components of Z are instruments, this analysis also applies to the output of randomized experiments. The economic approach to policy evaluation formulates policy questions using well-defined economic models. It then uses whatever statistical tools it takes to answer those questions. Policy questions and not statistical methods drive analyses. Well-posed economic models are scarce in the program evaluation approach. Thus in contrast to the structural approach, it features statistical methods over economic content. “Credibility” in the program evaluation literature is assessed by statistical properties of estimators and not economic content or policy relevance.

We can do better than hoping that an instrument or an estimator answers policy problems. By recovering economic primitives, we can distinguish the objects various estimators identify from the objects needed to address policy problems and can address those problems. Constructing the PRTE is an example of this approach.⁷⁴

Figure 7, taken from an analysis of the returns to attending college by Carneiro, Heckman, and Vytlacil (2009), plots the estimated weights for MTE from a marginal change in policy that proportionally expands the probability of attending college for everyone. The figure also plots the estimated MTE and the IV weight using P(Z) as an instrument. The IV weights and the policy weights are very different. The policy weights oversample high values of u_D compared to the IV weights. Since the MTE is declining in u_D, this translates into an IV estimate of .095 compared to a marginal policy effect of .015.⁷⁵ The IV estimate would suggest a substantial mean marginal gross return. The true marginal policy effect is much lower. Since the MTE can be estimated (or approximated) from the data and the policy weights constructed from the data, one can produce more accurate policy forecasts using the economics of the model.

Figure 7.

MTE and Weights for IV and MPRTE in the Carneiro-Heckman-Vytlacil (2009) Analysis of the Wage Returns to College.

Source: Carneiro, Heckman, and Vytlacil (2009).

Notes: The scale of the y-axis is the scale of the MTE, not the scale of the weights, which are scaled to fit the picture. The IV is P(Z).

3.8 Multiple Choices

Imbens and Angrist analyze a two choice model. Heckman, Urzua, and Vytlacil (2006, 2010) and Heckman and Vytlacil (2007b) extend their analysis to an ordered choice model and to general unordered choice models.⁷⁶

In the special case where the analyst seeks to estimate the mean return to those induced into a choice state by a change in an instrument compared to their next best option, the LATE framework remains useful (see Heckman, Urzua, and Vytlacil, 2006, 2010; Heckman and Vytlacil, 2007b). If, however, one is interested in identifying the mean returns to any pair of outcomes, unaided IV will not do the job. Structural methods are required.

In general unordered choice models, agents attracted into a state by a change in an instrument come from many origin states, so there are many margins of choice. Structural models can identify the gains arising from choices at these separate margins. This is a difficult task for IV without invoking structural assumptions. Structural models can also identify the fraction of persons induced into a state coming from each origin state. IV alone cannot. See Heckman and Urzua (2010).

4 Conclusions

This paper compares the structural approach to empirical policy analysis with the program evaluation approach. It offers a third way to do policy analysis that combines the best features of both approaches. This paper does not endorse or attack any particular statistical methodology. Economists are fortunate to have a rich menu of estimation methods from which to choose.

This paper advocates placing the economic and policy questions being addressed front and center. Economic theory helps to sharpen statements of policy questions. Modern advances in statistics can make the theory useful in addressing these questions. A better approach is to use the economics to frame the questions and the statistics to help address them.

Both the program evaluation approach and the structural approach have desirable features. Program evaluation approaches are generally computationally simpler than structural approaches, and it is often easier to conduct sensitivity and replication analyses with them. Identification of program effects is often more transparent than identification of structural parameters. At the same time, the economic questions answered and the policy relevance of the treatment effects featured in the program evaluation approach are often very unclear. Structural approaches produce more interpretable parameters that are better suited to conduct counterfactual policy analyses.

The third way advocated in this essay is to use Marschak’s Maxim to identify the policy relevant combinations of structural parameters that answer well-posed policy and economic questions. This approach often simplifies the burden of computation, facilitates replication and sensitivity analyses, and makes identification more transparent. At the same time, application of this approach forces analysts to clearly state the goals of the policy analysis — something many economists (structural or program evaluation) have difficulty doing. That discipline is an added bonus of this approach.

I have illustrated this approach by using the economics implicit in LATE to interpret the margins of choice identified by instrument variation and to extend the range of questions LATE can answer. This analysis is a prototype of the value of a closer integration of theory and robust statistical methods to evaluate public policy.

Appendices

A Derivation of the Weights for PRTE

Preliminary Remarks

Recall that, if for two random variables J and K for 0 ≤ j ≤ 1 and 0 ≤ k ≤ 1, with density f_J,K(j, k)

$$\int \int \mid f_{J,K}(j,k)\mid djdk<\infty ,$$

the value of the integral for the region 0 < j < k < 1

$${\int }_0^1{\int }_0^k{f}_{J,K}(j,k)djdk={\int }_0^1{\int }_j^1{f}_{J,K}(j,k)dkdj.$$

Derivation

We can write

$$\begin{array}{cc}\hfill & E(Y\mid \text{Baseline}b)={\int }_0^{1}E(Y\mid P\left(Z\right)=p)f_P^{b}dp\hfill \\ \hfill & =E[1(P\left(Z\right)\ge U_D)Y_1+1(P\left(Z\right)<U_D)Y_0]\hfill \\ \hfill & ={\int }_0^1{\int }_0^p{\int }_{-\infty }^{\infty }{y}_1{f}_{Y_1,Y_D}(y_1,u_D)d{y}_{1}d{u}_D{f}_P^b\left(p\right)dp\hfill \\ \hfill & +{\int }_0^1{\int }_p^1{\int }_{-\infty }^{\infty }{y}_0{f}_{Y_0,U_D}(y_0,u_D)d{y}_{0}d{u}_D{f}_P^b\left(p\right)dp\hfill \end{array}$$

because P(Z) ⫫ U_D, Y₁, Y₀ | X.

Thus

$$\begin{array}{cc}\hfill & E(Y\mid \text{Baseline}b)\hfill \\ \hfill & ={\int }_0^1{\int }_0^{p}E(Y_1\mid U_D=u_D)d{u}_D{f}_P^b\left(p\right)dp+{\int }_0^1{\int }_p^{1}E(Y_0\mid U_D=u_D)d{u}_D{f}_P^b\left(p\right)dp.\hfill \end{array}$$

Interchanging the limits of each integral

$$\begin{array}{cc}\hfill E(Y\mid \text{Baseline})& =\hfill \\ \hfill & ={\int }_0^{1}E(Y_1\mid U_D=u_D)\left[{\int }_{u_D}^1{F}_P^b\left(p\right)dp\right]d{u}_D+{\int }_0^{1}E(Y_0\mid U_D=u_D)\left[{\int }_0^{u_D}{F}_P^b\left(p\right)dp\right]d{u}_D\hfill \\ \hfill & ={\int }_0^{1}E(Y_1\mid U_D=u_D)\left[1-F_P^b\left(u_D\right)\right]d{u}_D+{\int }_0^{1}E(Y_0\mid U_D=u_D)F_P^b\left(u_D\right)d{u}_D.\hfill \end{array}$$

By a parallel argument for

$$\begin{array}{cc}\hfill & E(Y\mid \text{Alternative Policy}a)\hfill \\ \hfill & ={\int }_0^{1}E(Y_1\mid U_D=u_D)[1-F_P^a\left(u_D\right)]d{u}_D+{\int }_0^{1}E(Y_0\mid U_D=u_D)F_P^a\left(u_D\right)d{u}_D.\hfill \end{array}$$

Subtracting the first expression from the second expression we obtain the expression in the text

$$\mathrm{PRTE}={\int }_0^{1}E(Y_1-Y_0\mid U_D=u_D)(F_P^b\left(u_D\right)-F_P^a\left(u_D\right))d{u}_D.$$

B IV For Discrete Instruments

Suppose that the support of the distribution of P(Z) contains a finite number of values p₁ < p₂ < ⋯ < p_K. The support of the instrument Z₁ is also discrete, taking I distinct values. E(Z₁|P(Z) > u_D) is constant in u_D for u_D within any (p_ℓ, p_ℓ+1) interval, and Pr(P(Z) > u_D) is constant in u_D for u_D within any (p_ℓ, p_ℓ+1) interval. Let λ_ℓ denote the weight on the LATE for the interval (p_ℓ, p_ℓ+1).

Under monotonicity condition (IA-3),

$$\begin{array}{cc}\hfill {\mathrm{IV}}_{Z1}& ={\int }_{K-1}E(Y_1-Y_0\mid U_D=u_D)h_{Z_1}^{\mathrm{IV}}\left(u_D\right)d{u}_D\hfill \\ \hfill & =\underset{\ell =1}{\overset{K-1}{\Sigma }}{\lambda }_{\ell }{\int }_{p_{\ell }}^{p_{\ell +1}}E(Y_1-Y_0\mid U_D=u_D)\times \frac{1}{(p_{\ell +1}-p_{\ell })}d{u}_D\hfill \\ \hfill & =\underset{\ell =1}{\overset{K-1}{\Sigma }}{\Delta }^{\text{LATE}}(p_{\ell },p_{\ell +1}){\lambda }_{\ell }.\hfill \end{array}$$

Let $z_1^i$ be the i^th smallest value of the support of Z₁:

$$\begin{array}{cc}\hfill {\lambda }_{\ell }& =(p_{\ell +1}-p_{\ell })\times \hfill \\ \hfill & =\frac{\underset{i=1}{\overset{I}{\Sigma }}(z_1^i-E\left(Z_1\right))\underset{t>\ell }{\overset{K}{\Sigma }}\left(f(z_1^i,p_t)\right)}{\mathrm{Cov}(Z_1,D)}.\hfill \end{array}$$ (21)