Hallmarks of science missing from North American wildlife management

Fundamental components of science are often lacking in U.S. state and Canadian provincial hunt management systems.

Abstract

Resource management agencies commonly defend controversial policy by claiming adherence to science-based approaches. For example, proponents and practitioners of the “North American Model of Wildlife Conservation,” which guides hunting policy across much of the United States and Canada, assert that science plays a central role in shaping policy. However, what that means is rarely defined. We propose a framework that identifies four fundamental hallmarks of science relevant to natural resource management (measurable objectives, evidence, transparency, and independent review) and test for their presence in hunt management plans created by 62 U.S. state and Canadian provincial and territorial agencies across 667 management systems (species-jurisdictions). We found that most (60%) systems contained fewer than half of the indicator criteria assessed, with more criteria detected in systems that were peer-reviewed, that pertained to “big game,” and in jurisdictions at increasing latitudes. These results raise doubt about the purported scientific basis of hunt management across the United States and Canada. Our framework provides guidance for adopting a science-based approach to safeguard not only wildlife but also agencies from potential social, legal, and political conflict.

INTRODUCTION

Governments often assert, and society often assumes, that science comprises the foundation of natural resource management. In the United States and Canada, for example, state and provincial agencies responsible for managing bird and mammal hunting [the primary focus of wildlife management in the area; (1, 2)] commonly state that they adhere to the North American Model of Wildlife Conservation. This model is predicated on seven tenets, one of which states that “science is the proper tool to discharge policy” (1–3).

Despite widespread use and endorsement of the North American Model, however, its proponents and practitioners rarely articulate what science-based management entails. In addition, no one has comprehensively assessed whether assumptions or claims of science-based management are supported. That knowledge gap is troubling given the “science-based” justifications for policy decisions commonly offered by agencies, the substantial public investments that support these agencies, and the considerable influence that hunting can have on otherwise self-regulating wildlife populations. In many taxa, adult mortality from hunting exceeds mortality from all other predators combined (4).

To address this gap, we identified four fundamental, interrelated hallmarks expected of science-based natural resource management (measurable objectives, evidence, transparency, and independent review) and 11 specific criteria as indicators of those hallmarks (for example, “Is the technique for setting hunting quotas explained?” for our transparency hallmark; Table 1; Materials and Methods; Supplementary Text). We assessed how many of these criteria appeared in 667 management systems, using hunt management plans (produced by agencies to describe how hunted species are managed) for 27 species (or groups of species) across 62 states, provinces, and territories in the United States and Canada (database S1). We limited our assessments to management under the jurisdiction of U.S. states and Canadian territories and provinces, which comprise most of the hunted areas in both countries.

Table 1. Hallmarks and their indicator criteria.

Assessed across 667 management systems (species-jurisdictions) produced by 62 wildlife agencies across Canada and the United States.

Hallmarks	Indicator criteria
Measurable objectives	Provide measurable objectives
Evidence	Report quantitative information about populations
	Report uncertainty in population parameter estimates
	Estimate realized hunting rates
Transparency	Explain technique for setting hunting quotas
	Explain how population parameters are estimated
	Explain how realized hunting rates are estimated
	Provide publicly available management information
	Respond to public inquiry
Independent review	Subject management plans to any review
	Subject management plans to external review

RESULTS AND DISCUSSION

Our results provide limited support for the assumption that wildlife management in North America is guided by science. Most management systems lacked indications of the basic elements of a scientific approach to management. Although some systems contained many of the 11 assessed criteria [for example, 5 systems (of 667; 0.7%) contained 10 (91% of total) criteria, and 66 systems (10%) contained at least 8 criteria (73% of total)], most systems (402 of 667; 60%) contained fewer than half (that is, 5 or less). The average number of criteria present per system was 4.6 (42%; range, 0 to 10; fig. S1). Moreover, 54% (6) of the examined criteria were found in fewer than 50% of systems (Fig. 1).

Fig. 1. Percent of management systems across Canadian provinces/territories and U.S. states (N = 667 plans) in which indicator criteria for hallmarks of scientific management (measurable objectives, evidence, transparency, and independent review) were present.

Presence of hallmarks

Measurable objectives

Measurable objectives were detected in only 26% of management systems (Fig. 1). Without a benchmark against which to measure performance, neither agencies nor the public that entrusts them to manage wildlife can assess the efficacy of management or the associated value of their investments in this public service. Moreover, a lack of objectives might lead to management procedures (surveys, data collection, and use of technical equipment) becoming the focus of agency activities, instead of contributing to strategic approaches and interventions designed to accomplish well-defined goals (5, 6).

Evidence

Indications of evidence were often absent from management systems. Whereas data on estimated hunting rates were present in 79% of systems, quantitative information about populations (for example, abundances and trends) was present in only half (52%; Fig. 1). Fewer (15%) provided measures of uncertainty in population estimates. These results might warrant concern because of the importance of evidence in determining reliable baselines, assessing population dynamics and management outcomes (7, 8), and buffering against inherent uncertainties (9) in management.

Transparency

Whereas 89% of management systems had some publicly available information (for example, any related documents on agency websites), and 76% explained how realized hunting rates were estimated, only 55% of systems described how population parameters (trends and abundances) were estimated, and only 11% of systems described how hunting quotas were set (Fig. 1). Regarding accessibility of agencies to the public, we received responses to our email inquiries in less than half of the cases (44%; Fig. 1). Finally, the moderate inter-observer variation in our assessments (mean of ~12% across all criteria; Materials and Methods and table S1) might, in part, be explained by the lack of clarity—itself an important component of transparency—in management plans.

Deficits in transparency reduce opportunities for external scrutiny and the associated constructive criticism that could inform improvements in management. In a review of global fisheries, degree of transparency had a strong positive association with measures of sustainability, even outweighing the effect of jurisdictions’ wealth (10).

Independent review

Only 9% of management systems reported any form of review. Fewer (6%) involved external review (Fig. 1). This deviates substantially from scientific processes, where external scrutiny is a core requirement, addressing potential issues with impartiality, rigor, and intelligibility (11). Independent biologists with experience in wildlife management might be well positioned to evaluate the scientific basis of agency policy. The involvement of external qualified parties might also allow access to an expanded knowledge base, facilitating transmission of best practices across agencies and learning that crosses institutional or taxon-focused divides. As in the sciences, independent review of hunt management plans could help benefit not only the rigor of those plans but also the trust and interest of the public for whom management is ostensibly conducted (7).

Associations with presence of hallmarks

We found positive associations between the number of criteria present in a management system and the system pertaining to big game species, being independently reviewed, and with increasing latitude of jurisdictions within each country (Fig. 2). Critics of the North American Model have previously noted that management agencies tend to focus disproportionately on taxa most valued by hunters and pay little attention to those that are not (1, 12). A similar mechanism might explain why more criteria were found in hunt management systems for big game, given their significance to hunters (13). The positive association with independent review, itself a criterion rarely found across systems (Fig. 1), might explain, in part, the general deficiency of other hallmarks. We do not know what drives the positive association with latitude, but suggest that it might be an interaction of taxonomic, political, and societal variation with latitude that warrants further investigation. We did not find strong differences between countries [that is, Canada versus United States (Fig. 2), although systems in the United States had more criteria compared to those in Canada when the two “independent review” criteria were included as part of the response (fig. S2), suggesting that this hallmark was more common in U.S. systems]. We did not find associations between the number of criteria present in a management system and longitude, or human population size of its jurisdiction; the origin (native versus non-native) of the focal taxon; or whether large carnivores were present in the jurisdiction (Fig. 2). We similarly did not find an association between the number of criteria present and whether agencies responded to our inquiries about a given system (fig. S3). The latter finding increases our confidence that the verification of our results was not subject to strong self-selection bias, in which respondents to our inquiries were motivated by awareness that they had or had not met the criteria.

Fig. 2. Effect of management system characteristics on number of criteria present.

Number of criteria out of 9, with both independent review hallmark criteria excluded as part of the response (see fig. S2). Coefficients shown are odds ratios from a multilevel model, with thick and thin bars representing 50 and 95% confidence intervals, respectively, and plotted on a log scale.

Broader implications

Our findings suggest that the assumed scientific basis of wildlife management across much of the United States and Canada might warrant reconsideration. More broadly, our results highlight the importance of disclosing the relative contribution of science compared with other considerations [for example, political and economic; (5, 8)] in management decision-making and disclosing the limitations, uncertainties, and related risks inherent in the data and approaches used (5, 9, 14). We do not suggest that science alone should shape management decisions (15). Social dimensions of management [as articulated in community-based management (16), including, for example, Indigenous practice (17)] and ethics (12) can and should play prominent roles, but in a transparent manner (18, 19). In addition, we note that, in many cases, a scientific foundation to management might be infeasible due to constraints (for example, financial and logistical). Moreover, decision-making might be necessary when reliable evidence is limited or lacking (although considerable caution must be exercised in such circumstances). Hence, we do not suggest that a scientific basis is required for all management systems or decisions, but that it ought to be present when it is expected by the public or claimed by agencies.

We offer the proposed hallmarks here to encourage the development of a common understanding and expectation of what scientific management of wildlife and, more broadly, natural resources entails. We also illustrate a framework for testing for the presence of these hallmarks at a large scale. We note that the results we described would be subject to change had criteria been chosen or weighted differently. Future research might improve upon the framework, for example, by identifying additional criteria. Moreover, the interdependence of these hallmarks warrants caution in interpretation. Management systems with low observed transparency might score poorly for other hallmarks: For example, if not described publicly, we would not have been able to detect whether agencies have measurable objectives for a given species, subject its management plans to review, or estimate its population parameters.

Comparisons among other wildlife and environmental management approaches within and beyond North America, following the framework described here, might yield additional insights. For example, research might examine whether hallmarks of science are more likely to be identified in species managed at different levels of governments (for example, national or international versus state or province) or subject to additional policies or protections (for example, endangered species protections and transboundary agreements). Similarly, insight might be gleaned from assessing associations between a scientific basis and other aspects of governance such as leadership and social capital, which have been identified as strong predictors of success in fisheries management (16). Finally, deeper examinations into the rigor of individual systems might prove useful. For example, future research could assess whether scientific information (for example, demographic data) is incorporated into components of a hunt management plan (for example, population growth models), whether it is used correctly, and whether it flows logically to management prescriptions.

We suspect that agencies might respond to our external audit and associated evaluations (database S1) with (i) disagreement or criticism of our framework, (ii) more judicious defense of policy invoking science-based claims, and/or (iii) steps to build and maintain a more scientific approach. The hallmarks provided here are not exhaustive but provide a foundation for building a science-based approach and for agencies and others to assess organizational change through time. In particular, we suggest that implementation of external review would be important for facilitating the adoption of other aspects of a scientific approach. For example, deficiencies in use of evidence, measureable objectives, and transparency might be easier to detect—and avenues for improvement more likely to emerge—if management approaches were subjected to independent review. Examples from other management regimes have shown how such scrutiny could be integrated into management, including reviews by external organizations (10), by editors and reviewers at academic journals (20), by scientific branches of agencies operating independently from management (for example, the Canadian Science Advisory Secretariat that reviews Canadian fisheries management) (21), and by independent committees (for example, the approach to endangered species assessments in Canada) (22).

While acknowledging constraints faced by agencies, many of the aspects (for example, transparently describing approaches already used) of our framework would not be expensive to implement. Other hallmarks might require greater financial investment (for example, reliable estimates of population dynamics). However, we speculate that some of the greatest barriers to implementing these hallmarks might be political. Others have cautioned that “agency capture” (that is, undue influence on agency decision-making by special interest groups, such as hunters), and traditions or social pressures within management agencies, might shape policy and management in an unacknowledged fashion (1, 23, 24). Management approaches founded on the hallmarks of science suggested here could increase transparency regarding the relative roles of all factors, scientific and otherwise, that necessarily affect management decisions. This might safeguard not only the rigor of management itself but also agencies from accusations of malfeasance and associated social, legal, and political conflict (10).

MATERIALS AND METHODS

We evaluated the presence of scientific hallmarks across 62 provincial, territorial, and state jurisdictions in wildlife management plans for 27 species (or species groups, for example, “upland birds”; database S1). One informed non-specialist (research technician, M. van Roy) searched agency websites for all available management-relevant information, including online sources, wildlife management plans, and other available documents (here collectively referred to as “management plans” or simply “plans”; N = 667) for all hunted species (or groups of species). Whereas plans varied in content, the description from the Oregon [State] Furbearer Program Report (2011, p. 4) was representative: “The purpose of this report is to provide information not only to trappers and hunters, but to all interested in furbearer management in Oregon. This report contains harvest management information, current and recent research and management projects, and monitoring efforts throughout the state, and primarily with ODFW partners.” Whereas most information we found online might have been intended for a lay audience less interested in technical information, we included and scored all available documents, including any additional documents sent to us by agencies.

To identify appropriate hallmarks, we reviewed a broad literature spanning the process and theory of science to applied management approaches and identified four recurrent interdependent components that together create a comprehensive approach to science: measurable objectives, evidence, transparency, and independent review. Within each management system, we searched for the presence of 11 criteria, each in the form of a basic question related to a hallmark (for example, “Is the technique for setting quotas explained?” for the transparency hallmark; Table 1; Supplementary Text). We scored generously to avoid overstating any apparent deficiencies detected. For example, if a management system covered multiple populations and/or species and we found support for a particular criterion in any population/species, then we scored it as present for all populations/species. Similarly, we tested only for the presence of criteria, not their rigor (for example, “Do they estimate realized hunting rates?” instead of “Do they reliably estimate realized hunting rates?”). Finally, we erred toward a low threshold in assessing criteria: For example, the external review criterion was scored as present for any form of external scrutiny (including public input, despite this being a considerably weaker requirement than the anonymous, binding review expected in most sciences).

We evaluated the quality of our data using two approaches. First, we provided agencies the opportunity to identify errors (or forward any relevant documents we might have missed) by emailing them our assessments (see “emails to agencies” below). After 24 months, we received 272 of 624 potential responses (that is, regarding management systems for which we were able to find an agency email address, of the 667 total). Of these, agencies indicated that no changes were required in 15% (n = 42 systems), indicated that they were unwilling or unable to review 18% (n = 48), and provided suggested changes for 8% (n = 22), resulting in an average of 1.1 criteria revised per identified system. We also assessed inter-observer agreement by providing two other observers nonoverlapping random subsets of 5% of management systems (n = 28 each) to rescore independently. We compared how often rescores matched original assessments across criteria and found agreement 88% of the time on average (range, 75 to 96; table S1). Two criteria were excluded from this inter-observer assessment: “Provide publicly available management information,” because only cases with available documents were rescored, and “Respond to public inquiry,” because this was not a scored criterion but was instead a measure of whether our emails to agencies received a response.

We used a multilevel modeling approach to test for associations between the number of criteria present in a given management system and whether the managed taxa were (i) “big game” or (ii) native; the jurisdiction’s (iii) country (United States versus Canada), (iv) latitude, (v) longitude, and (vi) human population; (vii) whether large carnivores [grizzly bears (Ursus arctos), cougars (Puma concolor), or wolves (Canis lupus)] existed anywhere in the jurisdiction; and (viii) whether a system was independently reviewed. We used the glmmADMB (25) package in R (26) to run the model, with a Poisson distribution and with deviations to intercepts varying randomly with jurisdiction (U.S. state or Canadian province). We centered all predictors (subtracted the mean from each observation) and scaled (divided by 2 SDs) all continuous predictors (latitude, longitude, and size of human population) using the arm package (27, 28). Because we included independent review (specifically the “subject management plans to any review” criterion) as a predictor in this model, both independent review criteria were excluded from the response (which did not qualitatively affect other coefficient estimates; see Fig. 2 and fig. S2). We replicated the model above but with agency response included as a predictor (and omitted from the response) to test whether systems for which we received an agency response were representative (fig. S3).

Determining suite of taxa hunted in each jurisdiction

We reviewed hunting synopses to determine whether there were species hunted in a given jurisdiction for which we lacked management plans. Hunting synopses, typically released annually, inform hunters of the rules and regulations regarding all hunted species. In cases where hunting synopses indicated that a species was hunted in a given jurisdiction, but management plans were not found on the website, we sent emails to relevant agency contacts whose contact information was found in management plans or on agency website. In cases where no relevant email addresses were found, we emailed any other available agency contacts that we found online, requesting them to suggest appropriate contacts (see “request for management plan letter”).

Cases excluded

We excluded polar bears, waterfowl, and migratory birds (except in 14 systems where doves were mixed in with other nonmigratory taxa), because their management differs from most terrestrial species by being governed through a mix of federal and state/province/territory-level laws and regulations. We similarly excluded assessment of hunting on parcels of federal and tribal lands. We excluded the Canadian province of Québec from our analyses because management plans were not available in English.

Species were categorized as native or non-native, with mixed-species systems designated based on the majority of species therein (for example, we classified 27 management systems that mostly included not only native bird species but also non-native pheasants as native). We did not exclude non-native species because, despite perhaps being guided by different objectives (for example, reducing or eliminating populations) than management of native species, the same science-based hallmarks might be expected in either case.

Emails to agencies

We sent emails to all agencies with available email addresses (n = 624 of 667 management systems; 94%). These emails were sent by our primary scorer on behalf of the lead author (K.A.A.; see the next section). In all cases, we requested corrections if applicable, contact information of agency contacts that we had not found, and any publicly available documents that we had not identified.

These emails served not only to evaluate the quality of our data but also as a proxy for how responsive agencies are to public inquiry using a common (as evidenced by email addresses being available for most plans) communication channel. Hence, they provided another method to assess agency transparency.

Of the 624 cases (management systems) where agency contacts were found and emails were sent, we received responses regarding 272 (44% of total; no responses were received for the remaining 56%).

Of the 272 systems with a response, agencies:

(i) indicated that no changes were needed in 42 (15%);

(ii) indicated that they were unwilling or unable to review 48 (18%);

(iii) answered emails but did not provide relevant information on assessments (for example, provided general statements about management without new documents or specific changes to scoring, but did not specifically state that they were unwilling to participate) in 130 (48%);

(iv) provided feedback, but only for draft criteria that were not analyzed here in 2 (0.01%);

(v) suggested changes in 22 (8%), providing corroborating evidence for most individual scorings, leading us to revise 23 among them (average of ~1 criterion per system), although revisions were not made for changes to 8 individual scorings that were provided without any corroborating evidence (average of 0.4 criteria per system), 1 that was corroborated only by internal, not publicly available documents (average of 0.05 criteria per system), and 2 whose submitted evidence did not support the suggested change (average of 0.1 criteria per system). Evaluation of documents provided as part of the response for these 22 systems resulted in additional (unrequested) changes to an additional 7 individual scorings (average of 0.3 criteria per system; all such scores revised from “absent” to “present,” as new documents provided evidence of previously undetected criteria);

(vi) suggested no changes in scorings of analyzed criteria, but provided additional documents that triggered reevaluation of 28 (10%) systems, resulting in changes to 6 (16 individual scorings among them, or an average of 2.7 criteria per system; all such scores revised from absent to present as new documents provided evidence of previously undetected criteria).

Template communications with agencies

Letter requesting management plans

Dear [Agency Representative]

I am writing you today about research we’re doing with the Applied Conservation Science lab at the University of Victoria and the Reynolds Lab at Simon Fraser University. We are collecting information on approaches to wildlife management across North America, with a special focus on hunt management. We have been amassing publicly-available documents that describe management objectives, approaches used for setting hunt quotas, descriptions of population monitoring, and descriptions of how science and other sources of knowledge contribute to decision-making. For each jurisdiction and taxon we have used publicly-available documents to characterize hunt management based on a set of categorical criteria.

We have not been able to find any information about [Species Name] in [Jurisdiction]. If relevant information is available, would you mind please forwarding the relevant documents, and an appropriate link if they are already available on a website? Alternatively, if there is someone else we should ask for these documents, could you please forward their contact information?

It would be very helpful if we could please receive this information, by October 23rd, 2014, which is when we will analyze the data.

Thank you very much for your help.

Yours sincerely,

Kyle Artelle

Letter requesting review of assessments

Dear [agency representative],

I am writing you today about research we’re doing with the Applied Conservation Science lab at the University of Victoria and the Reynolds Lab at Simon Fraser University. We are collecting information on approaches to wildlife management across North America, with a special focus on hunt management. We have been amassing publicly-available documents that describe management objectives, approaches used for setting hunt quotas, descriptions of population monitoring, and descriptions of how science and other sources of knowledge contribute to decision-making. For each jurisdiction and taxon we have used publicly-available documents to characterize hunt management based on a set of categorical criteria.

I have attached our categorization of [Species Name] management in [Jurisdiction], with reference to the publicly-available documents used in our assessments. We were wondering if you could please let us know if there are any errors in how we have categorized this management. If there are documents that describe management differently than we have categorized here, could you please reference the relevant sections and documents so we can re-assess the information?

Alternatively, if there is someone else we should ask for these documents, could you please forward their contact information?

It would be very helpful if we could receive this information, by October 23rd, 2014, please, which is when we will analyze the data.

Thank you very much for your help.

Yours sincerely,

Kyle Artelle

SUPPLEMENTARY MATERIALS

Supplementary material for this article is available at http://advances.sciencemag.org/cgi/content/full/4/3/eaao0167/DC1

Supplementary Text

fig. S1. Number of criteria (out of possible 11) present in wildlife management plans across Canadian provinces/territories and U.S. states (N = 667).

fig. S2. Effect of management characteristics on number of criteria present.

fig. S3. Effect of management characteristics on number of criteria present.

table S1. Inter-observer agreement.

database S1. Assessment data (assessments of all available management documents, for example, online resources, wildlife management plans, or other available documents).

metadata S1. Metadata about all management documents scored in this analysis (that is, those referenced in database S1; including URLs, where possible).

References (29–44)