Unobserved time effects confound the identification of climate change impacts

Abstract

A recent study by Feng et al. [Feng S, Krueger A, Oppenheimer M (2010) Proc Natl Acad Sci USA 107:14257–14262] in PNAS reported statistical evidence of a weather-driven causal effect of crop yields on human migration from Mexico to the United States. We show that this conclusion is based on a different statistical model than the one stated in the paper. When we correct for this mistake, there is no evidence of a causal link.

The work by Feng et al. (1) exploits variation in crop yields, weather, and migration abroad across Mexico’s states for two periods (1995–2000 and 2000–2005) using linear regression methods. Equation 1 in ref. 1 states that the statistical model regresses the percent of the population in state i at the beginning of period t that migrated abroad during the period on crop yields (corn or corn plus wheat), state fixed effects, and a dummy variable for the second period (time effect). This strategy flexibly controls for unobserved time-invariant confounders at the state level (state fixed effects) and unobserved confounders affecting all states in the same period (the time effect). The model also uses weather variables as instrumental variables for yields to control for potential endogeneity. Theoretically, this approach is statistically valid for determining the causal effect of crop yield on emigration.

This comment checks the analysis in the work by Feng et al. (1). It provides a replication of the models using the original data as provided in the work by Feng et al. (1) (Datasets S1 and S2). Based on that replication, it determines that the work by Feng et al. (1) did not, in fact, include the time effect in equation 1 in ref. 1. We show that appropriately controlling for the time effect results in a finding of no significant effect of crop yield on emigration. This finding invalidates the finding that weather-driven yields affected Mexican migration abroad in the statistical sense as concluded in the paper, at least for the sample of states and time periods analyzed in the work by Feng et al. (1).

Results

The FKO PNAS panel for yield statistic: corn in Table 1 displays our regression results for a model with corn yield that excludes the time effect. The results replicate almost exactly the results in the work by Feng et al. (1), which according to that paper, were for a model that included the time effect. The very minor discrepancies are likely because the data in Datasets S1 and S2 were rounded for display. We also note that the R² reported in the work by Feng et al. (1) is most likely the overall R² instead of the adjusted R² as stated. As reported in the work by Feng et al. (1), results in Table 1, columns 3–5, indicate that corn yield has a highly significant, negative effect on migration.

Table 1.

Regression results from the work by Feng et al. (1) without proper inclusion of the time effect (unshaded) and regression results from regressions including the time effect (shaded) for corn and corn plus wheat

	(1) Pooled OLS	(2) RE	(3) FE	(4) FE-TSLS	(5) FE-LIML
Yield statistic: corn
FKO PNAS
Log of crop yields	0.00541	0.00341	−0.117*	−0.211*	−0.224*
SE	0.00803	0.00845	0.0259	0.0358	0.0415
Overall R2	0.0114	0.0114	0.6787	0.5914	0.5638
Corrected model
Log of crop yields	0.00829	0.00746	−0.0122	−0.0645	−0.0911
SE	0.00643	0.00854	0.0355	0.0417	0.0637
Overall R2	0.3427	0.3425	0.8586	0.8416	0.8200
Yield statistic: corn plus wheat
FKO PNAS
Log of crop yields	0.00583	0.00455	−0.113*	−0.183*	−0.214*
SE	0.00792	0.00824	0.0310	0.0354	0.0510
Overall R2	0.0138	0.00138	0.6503	0.6089	0.5638
Corrected model
Log of crop yields	0.00819	0.00768	−0.00578	−0.0369	−0.0453
SE	0.00630	0.00834	0.0349	0.0277	0.0342
Overall R2	0.3430	0.3429	0.8578	0.8522	0.8488

As in the work by Feng et al. (1), results are shown for five models: pooled ordinary least squares (OLS), random effects (RE), fixed effects (FE), two-stage least squares with fixed effects (FE-TSLS), and limited information maximum likelihood with fixed effects (FE-LIML), where random and fixed effects refer to Mexican states. Results shown for each model are the regression coefficient on crop yield, the robust SEs for the coefficient, and the R². FKO PNAS, the work by Feng et al. (1).

*Significance at P < 0.01.

The corrected model for yield statistic: corn in Table 1 is for a model that includes the time effect. The corrected results show that, when the time effect is included, none of the coefficients on corn yield are statistically different from zero, even at the 10% level. The overall fit improves significantly, and the point estimates, although still negative, are significantly smaller in magnitude.

Yield statistic: corn plus wheat in Table 1 shows the results for corn plus wheat. The consequences of including the time effect are the same. The coefficient on yield is no longer statistically different from zero, and the point estimate is substantially closer to zero.

Discussion

After the model in the work by Feng et al. (1) is correctly estimated using their data, the statistical evidence suggesting a causal relationship between weather-driven crop yields and emigration from Mexico to the United States disappears. The statistical evidence in the work by Feng et al. (1) is based on an incompletely controlled before-and-after comparison of emigration rates and crop yields across states. Any omitted factor positively (negatively) correlated with yields and negatively (positively) correlated with emigration rates over time affecting all states confounds this estimated effect. The results in the work by Feng et al. (1), therefore, cannot be given a causal interpretation, because a variety of factors changed during this period. According to the work by Feng et al. (1), these factors included “the effect of NAFTA, the Peso crisis, and changes in US border controls such as increased border enforcement after 2001” (ref. 1, p. 14258); regarding the second factor, after 1994, “the Peso depreciated considerably against the US dollar, doubling the real wage rate earned by emigrants” (1). The work by Feng et al. (1) also points to the “reform of the land tenure system and the opening of Mexico’s economy through liberalized trade and deregulation of markets” (1), which further impoverished small farmers and rural landholders. These factors would confound a pure before/after comparison, and therefore, they make it absolutely essential to control for them through time effects. Results from correct estimation of the model suggest that, for the sample used in the work by Feng et al. (1), these factors, and not weather, were responsible for the change in emigration rates between the two periods.

Data and Methods

Data.

Ref. 1 has information on the data.

Regression Analysis.

Ref. 1 has information on the regression analysis.