Matching and mechanisms in protected area and poverty alleviation research

Excitement among social scientists about the discovery of randomized controlled trials has been tempered by the recognition that experimental research and related designs may be infeasible, prohibitively expensive, or even unsuitable for an enormous range of questions of interest to social science and policy (1–3). Recourse to matching-based statistical approaches can enable more transparent causal inference with observational data. The recent upsurge in environmental science writings that use matching techniques borrows from a long and continuing history of such use in medicine, public health, and economics (4–6) and should be welcomed for demonstrating the utility of another important tool in the search for improved estimation of causal effects of environmental interventions. Ferraro and Hanauer (7), leaders in the application of matching-based techniques to identify impacts of protected areas (PAs), present a fresh innovation for environmental social scientists by supplementing the matching-based approach to estimate the effects of protected areas on poverty in Costa Rica with an analysis of three causal mechanisms that may connect protected areas to observed poverty effects. They found that only ecotourism reduced poverty. Changes in forest cover and infrastructure turned out not to have significant effects. My comment examines issues related to data, theory, and policy relevance that pertain to many recent matching-based studies of the effects of protected areas (7–9).

Data

Previous matching-based studies that have identified positive effects of protected areas on poverty have tended to rely on arguments about the rigor of their methods rather than on the identification and testing of specific causal mechanisms to assert causal effects (8, 9). A typical limitation is the lack of data needed to test alternative explanations or to estimate the causal mechanisms that link the explanans to the explanandum. Data on potential alternative explanations were likely not available, but Ferraro and Hanauer (7) make a plausible effort through thought experiments to rule out other explanations of the effects of protected areas on poverty. Were data available, they could simply test empirically whether alternative explanations should be ruled out. However, the proxy they use for ecotourism effects on poverty—park entrances—is still quite distant as a causal mechanism connecting poverty alleviation to protected areas.

Indeed, Ferraro and Hanauer agree that the ecotourism finding can and should be decomposed further when they observe that tourism likely reduced poverty through market

Analysis of the kind represented in Ferraro and Hanauer is a model for other scholars of renewable resource governance outcomes.

channels. In the absence of data on these channels—e.g., emergence of economic opportunities related to PA establishment, labor market deepening because of PAs, improvements in asset holdings among households that rely on PAs, or other unidentified processes—the existence of market channels as the next step in the causal chain remains untested.

The issue of data gaps is characteristic of many matching studies of PAs, although additional global datasets on variables that likely influence PA outcomes and location are available than used in matching-based statistical analyses of PAs (e.g., the datasets available at globe.umbc.edu). Usually, datasets on which matching studies are based contain large number of observations on relatively few factors. Missing from these datasets and others available publicly is fine scale spatial information about institutional development and density, measures of spatial variations in the implementation and impacts of agricultural, conservation, infrastructure, and social policies, and direct measures of poverty.

Theory Development

If lack of data is one obstacle that hinders broader application of matching-based methods for impact evaluation, at least an equal obstacle is the lack of a well-developed theory of PA creation and impacts. Contrast theoretical development related to the creation, management, and impacts of PAs with the situation for the management and impacts of common-pool resources (10, 11). In the latter case, there is a plethora of theory, particularly relative to the limited availability of spatial datasets on the commons, and the limited application of sophisticated quantitative analytical techniques of the kind represented in the Ferraro and Hanauer paper.

In the absence of compelling theory, the only recourse analysts have is to test for causal effects piecemeal and to look for possible mechanisms to explain estimated effects based on context-specific knowledge of the PA, country, or region. Fully elaborated theories are indeed a long distance off. However, the alternative is not just to carry out empirical analyses of the effects of environmental interventions case by case, intervention by intervention, country by country, region by region, or even globally. Improved understanding, even in the short run, requires theory development to occur in step with data collection and more rigorous analyses.

For example, what is the underlying theory of PA creation and location in existing studies (including those that do not use a matching-based approach)? Although a fully elaborated theory does not exist, selection of PA locations, in addition to factors cited in and controlled for in matching-based analyses, is likely also a function of the “ecotourism potential” of candidate locations. Without inclusion of controls for such potential—e.g., for charismatic megafauna, magnificent vistas, threatened species, and heritage values, to mention a few possibilities—existing analyses may be overestimating the poverty alleviation effects of PAs (but likely not of ecotourism). Current datasets do not include information on tourism potential, and this factor goes unexplored in most studies of protected area effects. (I am grateful to Christoph Nolte at the University of Michigan for this point.)

The point is not about this specific control, obviously. It is, more generally, about the limited effort at theory development in favor of sophisticated empirical analyses in matching-based analyses. Without directed attempts at theory development occurring in concert with analyses of data, should we conclude that in countries where ecotourism is not a major phenomenon, the statistically identified effects of protected areas on poverty are an example of spurious correlation? Would such doubts be especially apposite where additional data do not exist to permit tests of alternative explanations and causal mechanisms?

Policy Relevance

Important as the specific findings of the analysis are for debates over the effects of protected areas on poverty (12–14), equally important are the showcased methods. Ferraro and Hanauer (7) hint at this in calling on the “[conservation community] to build the evidence base on a policy-by-policy and country-by-country (or region-by-region) basis,” and in suggesting that armed with a better knowledge of how environmental interventions affect poverty, policy makers can design programs that foster mechanisms to alleviate poverty. Persuaded as I am of the importance of their analysis and paper, these suggestions raise larger issues related to what one might call the “limits of policy relevance” of research outputs.

Limits of policy relevance refers to the often expressed hope in impact evaluations that the findings of the research will lead to policy change and will have an effect on decision making. Improved knowledge can mostly be taken as a necessary condition for better policies; however, and this is surely a truism, it is never sufficient. The role of changing scientific understanding in prompting policy change is a vigorous research domain in itself, but better science alone is seldom enough.

However, the limits of policy relevance is more than simply a matter of whether actions of decision makers are influenced by individual studies or even by many studies. By definition, assertions of policy relevance of a study target prospective choices by decision makers. Impact evaluations, in contrast, are about past policies and interventions. Potential policy relevance of studies of impacts thus rests on assumptions about the stability of socioeconomic, demographic, macroinstitutional, political, technological, and climatic contexts and the continuity of future rates of change for these contextual features that may or may not be borne out. That does not mean one should not do more rigorous analyses of the impacts of environmental interventions. Analysis of the kind represented in Ferraro and Hanauer (7) is a model for other scholars of renewable resource governance outcomes and a strong argument in favor of why it is surely better to do better science. However, it just might mean that the reward of better science, like that of virtue, is itself. And that is no bad thing!