Expert credibility in climate change

Abstract

Although preliminary estimates from published literature and expert surveys suggest striking agreement among climate scientists on the tenets of anthropogenic climate change (ACC), the American public expresses substantial doubt about both the anthropogenic cause and the level of scientific agreement underpinning ACC. A broad analysis of the climate scientist community itself, the distribution of credibility of dissenting researchers relative to agreeing researchers, and the level of agreement among top climate experts has not been conducted and would inform future ACC discussions. Here, we use an extensive dataset of 1,372 climate researchers and their publication and citation data to show that (i) 97–98% of the climate researchers most actively publishing in the field surveyed here support the tenets of ACC outlined by the Intergovernmental Panel on Climate Change, and (ii) the relative climate expertise and scientific prominence of the researchers unconvinced of ACC are substantially below that of the convinced researchers.

Preliminary reviews of scientific literature and surveys of climate scientists indicate striking agreement with the primary conclusions of the Intergovernmental Panel on Climate Change (IPCC): anthropogenic greenhouse gases have been responsible for “most” of the “unequivocal” warming of the Earth's average global temperature over the second half of the 20th century (1–3). Nonetheless, substantial and growing public doubt remains about the anthropogenic cause and scientific agreement about the role of anthropogenic greenhouse gases in climate change (4, 5). A vocal minority of researchers and other critics contest the conclusions of the mainstream scientific assessment, frequently citing large numbers of scientists whom they believe support their claims (6–8). This group, often termed climate change skeptics, contrarians, or deniers, has received large amounts of media attention and wields significant influence in the societal debate about climate change impacts and policy (7, 9–14).

An extensive literature examines what constitutes expertise or credibility in technical and policy-relevant scientific research (15). Though our aim is not to expand upon that literature here, we wish to draw upon several important observations from this literature in examining expert credibility in climate change. First, though the degree of contextual, political, epistemological, and cultural influences in determining who counts as an expert and who is credible remains debated, many scholars acknowledge the need to identify credible experts and account for expert opinion in technical (e.g., science-based) decision-making (15–19). Furthermore, delineating expertise and the relative credibility of claims is critical, especially in areas where it may be difficult for the majority of decision-makers and the lay public to evaluate the full complexities of a technical issue (12, 15). Ultimately, however, societal decisions regarding response to ACC must necessarily include input from many diverse and nonexpert stakeholders.

Because the timeline of decision-making is often more rapid than scientific consensus, examining the landscape of expert opinion can greatly inform such decision-making (15, 19). Here, we examine a metric of climate-specific expertise and a metric of overall scientific prominence as two dimensions of expert credibility in two groups of researchers. We provide a broad assessment of the relative credibility of researchers convinced by the evidence (CE) of ACC and those unconvinced by the evidence (UE) of ACC. Our consideration of UE researchers differs from previous work on climate change skeptics and contrarians in that we primarily focus on researchers that have published extensively in the climate field, although we consider all skeptics/contrarians that have signed prominent statements concerning ACC (6–8). Such expert analysis can illuminate public and policy discussions about ACC and the extent of consensus in the expert scientific community.

We compiled a database of 1,372 climate researchers based on authorship of scientific assessment reports and membership on multisignatory statements about ACC (SI Materials and Methods). We tallied the number of climate-relevant publications authored or coauthored by each researcher (defined here as expertise) and counted the number of citations for each of the researcher's four highest-cited papers (defined here as prominence) using Google Scholar. We then imposed an a priori criterion that a researcher must have authored a minimum of 20 climate publications to be considered a climate researcher, thus reducing the database to 908 researchers. Varying this minimum publication cutoff did not materially alter results (Materials and Methods).

We ranked researchers based on the total number of climate publications authored. Though our compiled researcher list is not comprehensive nor designed to be representative of the entire climate science community, we have drawn researchers from the most high-profile reports and public statements about ACC. Therefore, we have likely compiled the strongest and most credentialed researchers in CE and UE groups. Citation and publication analyses must be treated with caution in inferring scientific credibility, but we suggest that our methods and our expertise and prominence criteria provide conservative, robust, and relevant indicators of relative credibility of CE and UE groups of climate researchers (Materials and Methods).

Results and Discussion

The UE group comprises only 2% of the top 50 climate researchers as ranked by expertise (number of climate publications), 3% of researchers of the top 100, and 2.5% of the top 200, excluding researchers present in both groups (Materials and Methods). This result closely agrees with expert surveys, indicating that ≈97% of self-identified actively publishing climate scientists agree with the tenets of ACC (2). Furthermore, this finding complements direct polling of the climate researcher community, which yields qualitative and self-reported researcher expertise (2). Our findings capture the added dimension of the distribution of researcher expertise, quantify agreement among the highest expertise climate researchers, and provide an independent assessment of level of scientific consensus concerning ACC. In addition to the striking difference in number of expert researchers between CE and UE groups, the distribution of expertise of the UE group is far below that of the CE group (Fig. 1). Mean expertise of the UE group was around half (60 publications) that of the CE group (119 publications; Mann–Whitney U test: W = 57,020; P < 10⁻¹⁴), as was median expertise (UE = 34 publications; CE = 84 publications). Furthermore, researchers with fewer than 20 climate publications comprise ≈80% the UE group, as opposed to less than 10% of the CE group. This indicates that the bulk of UE researchers on the most prominent multisignatory statements about climate change have not published extensively in the peer-reviewed climate literature.

Fig. 1.

Distribution of the number of researchers (n = 908) in convinced by the evidence (CE) of anthropogenic climate change and unconvinced by the evidence (UE) categories with a given number of total climate publications. Tick marks indicate the center of right-inclusive categories (e.g., 20–50, 51–100, 101–150, etc.).

We examined a subsample of the 50 most-published (highest-expertise) researchers from each group. Such subsampling facilitates comparison of relative expertise between groups (normalizing differences between absolute numbers). This method reveals large differences in relative expertise between CE and UE groups (Fig. 2). Though the top-published researchers in the CE group have an average of 408 climate publications (median = 344), the top UE researchers average only 89 publications (median = 68; Mann–Whitney U test: W = 2,455; P < 10⁻¹⁵). Thus, this suggests that not all experts are equal, and top CE researchers have much stronger expertise in climate science than those in the top UE group.

Fig. 2.

Distribution of the number of the top 50 most-published researchers from CE and UE categories with a given number of total climate publications. Tick marks indicate the center of right-inclusive categories (e.g., 20–50, 51–100, 101–150, etc.).

Finally, our prominence criterion provides an independent and approximate estimate of the relative scientific significance of CE and UE publications. Citation analysis complements publication analysis because it can, in general terms, capture the quality and impact of a researcher's contribution—a critical component to overall scientific credibility—as opposed to measuring a researcher's involvement in a field, or expertise (Materials and Methods). The citation analysis conducted here further complements the publication analysis because it does not examine solely climate-relevant publications and thus captures highly prominent researchers who may not be directly involved with the climate field.

We examined the top four most-cited papers for each CE and UE researcher with 20 or more climate publications and found immense disparity in scientific prominence between CE and UE communities (Mann–Whitney U test: W = 50,710; P < 10⁻⁶; Fig. 3). CE researchers’ top papers were cited an average of 172 times, compared with 105 times for UE researchers. Because a single, highly cited paper does not establish a highly credible reputation but might instead reflect the controversial nature of that paper (often called the single-paper effect), we also considered the average the citation count of the second through fourth most-highly cited papers of each researcher. Results were robust when only these papers were considered (CE mean: 133; UE mean: 84; Mann–Whitney U test: W = 50,492; P < 10⁻⁶). Results were robust when all 1,372 researchers, including those with fewer than 20 climate publications, were considered (CE mean: 126; UE mean: 59; Mann–Whitney U test: W = 3.5 × 10⁵; P < 10⁻¹⁵). Number of citations is an imperfect but useful benchmark for a group's scientific prominence (Materials and Methods), and we show here that even considering all (e.g., climate and nonclimate) publications, the UE researcher group has substantially lower prominence than the CE group.

Fig. 3.

Distribution of the number of researchers (n = 908) in CE and UE categories with a given number times cited for each researcher's average of the first through fourth most-cited papers. Tick marks indicate the center of right-inclusive categories (e.g., 0–50, 51–100, 101–150, etc.), stepped by increments of 50 until 1,000 citations, and 500 thereafter.

We provide a large-scale quantitative assessment of the relative level of agreement, expertise, and prominence in the climate researcher community. We show that the expertise and prominence, two integral components of overall expert credibility, of climate researchers convinced by the evidence of ACC vastly overshadows that of the climate change skeptics and contrarians. This divide is even starker when considering the top researchers in each group. Despite media tendencies to present both sides in ACC debates (9), which can contribute to continued public misunderstanding regarding ACC (7, 11, 12, 14), not all climate researchers are equal in scientific credibility and expertise in the climate system. This extensive analysis of the mainstream versus skeptical/contrarian researchers suggests a strong role for considering expert credibility in the relative weight of and attention to these groups of researchers in future discussions in media, policy, and public forums regarding anthropogenic climate change.

Materials and Methods

We compiled a database of 1,372 climate researchers and classified each researcher into two categories: convinced by the evidence (CE) for anthropogenic climate change (ACC) or unconvinced by the evidence (UE) for ACC. We defined CE researchers as those who signed statements broadly agreeing with or directly endorsing the primary tenets of the IPCC Fourth Assessment Report that it is “very likely” that anthropogenic greenhouse gases have been responsible for “most” of the “unequivocal” warming of the Earth's average global temperature in the second half of the 20th century (3). We compiled these CE researchers comprehensively from the lists of IPCC AR4 Working Group I Contributors and four prominent scientific statements endorsing the IPCC (n = 903; SI Materials and Methods). We defined UE researchers as those who have signed statements strongly dissenting from the views of the IPCC. We compiled UE names comprehensively from 12 of the most prominent statements criticizing the IPCC conclusions (n = 472; SI Materials and Methods). Only three researchers were members of both the CE and UE groups (due to their presence on both CE and UE lists) and remained in the dataset, except in calculations of the top 50, 100, and 200 researchers’ group membership.

Between December 2008 and July 2009, we collected the number of climate-relevant publications for all 1,372 researchers from Google Scholar (search terms: “author:fi-lastname climate”), as well as the number of times cited for each researcher's four top-cited articles in any field (search term “climate” removed). Overall number of publications was not used because it was not possible to provide accurate publication counts in all cases because of similarly named researchers. We verified, however, author identity for the four top-cited papers by each author.

To examine only researchers with demonstrated climate expertise, we imposed a 20 climate-publications minimum to be considered a climate researcher, bringing the list to 908 researchers (N_CE = 817; N_UE = 93). Our dataset is not comprehensive of the climate community and therefore does not infer absolute numbers or proportions of all CE versus all UE researchers. We acknowledge that there are other possible and valid approaches to quantifying the level of agreement and relative credibility in the climate science community, including alternate climate researcher cutoffs, publication databases, and search terms to determine climate-relevant publications. However, we provide a useful, conservative, and reasonable approach whose qualitative results are not likely to be affected by the above assumptions. We conducted the above analyses with a climate researcher cutoff of a minimum of 10 and 40 publications, which yielded very little change in the qualitative or strong statistically significant differences between CE and UE groups. Researcher publication and citation counts in Earth Sciences have been found to be largely similar between Google Scholar and other peer-review-only citation indices such as ISI Web of Science (20). Indeed, using Google Scholar provides a more conservative estimate of expertise (e.g., higher levels of publications and more experts considered) because it archives a greater breadth of sources than other citation indices. Our climate-relevant search term does not, understandably, capture all relevant publications and exclude all nonrelevant publications in the detection and attribution of ACC, but we suggest that its generality provides a conservative estimate of expertise (i.e., higher numbers of experts) that should not differentially favor either group.

Publication and citation analyses are not perfect indicators of researcher credibility, but they have been widely used in the natural sciences for comparing research productivity, quality, and prominence (21–24). Furthermore, these methods tend to correlate highly with other estimates of research quality, expertise, and prominence (21–26). These standard publication and citation metrics are often used in many academic fields to inform decisions regarding hiring and tenure. Though these methods explicitly estimate credibility to other academics, which might not directly translate to credibility in broader discourse, polls suggest that about 70% of the American public generally trust scientists’ opinions on the environment, making this assessment broadly relevant (27). Criticisms of the two methods center around issues of self-citation, additionality of multiple authors, clique citation, and age demographic (e.g., age distribution where older researchers can accrue more publications and citations) differences between groups (21–26, 28, 29). All of these criticisms are expected to have the least influence at high levels of aggregation (e.g., an entire field) and high levels of citations, both of which are analyzed here (21–23, 25, 28, 29).

Regarding the influence of citation patterns, we acknowledge that it is difficult to quantify potential biases of self-citation or clique citation in the analysis presented here. However, citation analysis research suggests that the potential of these patterns to influence results is likely to decline as sample size of researchers, possible cliques, and papers analyzed for citations considered increases (22, 25–28). By selecting an expansive sample of 1,372 researchers and focusing our analysis only on the researchers’ four most-cited papers, we have designed our study to minimize the potential influence of these patterns. Furthermore, we have no a priori basis for assuming any citation (e.g., self-citation rates) or demographic differences (e.g., age effect on publications or citations) between CE and UE groups. Preliminary evidence suggests these differences would likely favor the UE group. From the ∼60% of researchers where year of PhD was available, mean year of receiving a PhD for UE researchers was 1977, versus 1987 for CE researchers, implying that UE researchers should have on average more publications due to an age effect alone. Therefore, these methods are likely to provide a reasonable estimate of the preeminent researchers in each group and are useful in comparing the relative expertise and prominence between CE and UE groups.

Ultimately, of course, scientific confidence is earned by the winnowing process of peer review and replication of studies over time. In the meanwhile, given the immediacy attendant to the state of debate over perception of climate science, we must seek estimates while confidence builds. Based on the arguments presented here, we believe our findings capture the differential climate science credentials of the two groups.