Using bibliometrics to demonstrate the value of library journal collections

Abstract

Although cited reference studies are common in the library and information science literature, they are rarely performed in non-academic institutions or in the atmospheric and oceanic sciences. In this paper, we analyze over 400,000 cited references made by authors affiliated with the National Oceanic and Atmospheric Administration between 2009 and 2013. Our results suggest that these methods can be applied to research libraries in a variety of institutions, that the results of analyses performed at one institution may not be applicable to other institutions, and that cited reference analyses should be periodically updated to reflect changes in authors’ referencing behavior.

Introduction

As scientific journal subscription prices rise at rates above the Consumer Price Index and the Higher Education Price Index, and as library subscription budgets mostly remain flat or increase more slowly than journal prices, academic and research libraries face increasing pressure from their institutions to justify their journal subscription budgets¹. At the same time, the rise of big data and analytics has resulted in renewed interest in evidence-based decision-making on the part of institutional leaders. This interest means that anecdotal evidence of the value of library journal collections no longer carries the weight it once did, prompting libraries to provide quantitative evidence of value to support their claims.

Many academic libraries have responded to these trends by analyzing the use of their collections through electronic usage statistics or through the cited references of their institution’s faculty and students². Although government research libraries perform a similar function as academic libraries in that they support the information needs of institutional researchers, published collection analyses performed in government library settings are extremely rare. In this paper, we analyze the cited references made by intramural researchers at the National Oceanic and Atmospheric Administration (NOAA) to demonstrate the value of the NOAA libraries system to NOAA scientists and to demonstrate that various techniques developed in the academic library setting can be applied at other types of research libraries and vice versa. We also compare our results to the few citation analysis studies that have been performed in the atmospheric and oceanic sciences.

The NOAA libraries system consists of 28 partially to completely autonomous libraries located across the United States. These libraries differ greatly in terms of size, reporting structures, and funding sources, ranging from unstaffed reading rooms containing a few hundred items for in-person use, to one- and two-person libraries of hundreds to thousands of items serving NOAA scientists at specific laboratories and science centers, to the NOAA Central Library, with 14 full-time staff and a collection of over 400,000 items. Whereas most of these libraries only provide services to the specific NOAA laboratory or science center that provides their funding, the NOAA Central and Regional Libraries—located in Silver Spring, Miami, and Seattle—provide services to NOAA staff across the agency. The services provided by these libraries are similar to those provided by academic libraries to their faculties: access to collections, reference and literature searching, custom information support, and so forth. The NOAA libraries are working with the newly established NOAA Library Advisory Committee to identify areas in which cooperation and coordination between the libraries would enhance access to, and development of, library collections and services. Demonstrating the value of library journal collections to the entire agency is one of those areas.

Literature review

Our method of demonstrating that value through the analysis of intramural research citations is well-established in the library and information science literature. Local referencing patterns have been used to inform library collection development since at least the early 1970s³. Although much of this literature focuses on the references made by undergraduate and graduate students, and therefore is not directly relevant to a government agency like NOAA, a number of studies have been performed that analyze faculty publications in various scientific areas. McCain and Bobick and Davis analyzed references by faculty and students in biology, Tsay and Rethlefsen studied references in biomedicine and public health, Choinski analyzed references in pharmacy, Ortega analyzed references in chemistry, LaBonte analyzed references in nanoscience, and Stephens, et al. analyzed references in aerospace engineering⁴. Other authors analyzed cited references across multiple subject areas or across their entire institutions⁵.

Hoffmann and Doucette reviewed 34 cited reference analysis studies published from 2005 to 2010 to assess the transparency and rigor of their methodologies⁶. They find that most of these studies do not provide sufficient detail for their results to be replicated by others. They also found that many of the decisions made by the authors of these studies were not adequately explained by a clear and defined rationale. Hoffman and Doucette recommend that future cited reference studies not only provide sufficient detail and explanation about their methods, but also present their results in ways that facilitate comparison across studies.

Despite the frequency with which citation analysis methods have been used, Rethelfsen is the only citation analysis we are aware of that was performed in a government library setting⁷. Citation analyses of faculty publications in the marine, fisheries, and atmospheric sciences are similarly rare. Walcott analyzed over 5,000 cited references by faculty at the Marine and Sciences Research Center at the State University of New York at Stony Brook to identify the most frequently referenced journals for inclusion in the information center’s collection⁸. Ibeun identified core fisheries journals for acquisition by Nigerian fisheries libraries through a targeted survey of Nigerian fisheries scientists and found that the distribution of recommended titles conformed to Bradford’s law of scattering⁹. More recently, Kimball, et al. analyzed over 5,000 cited references of faculty publications from the Department of Atmospheric Sciences at Texas A&M University to identify the sources, formats, and ages of the publications referenced¹⁰.

Finally, a few studies suggest that there are positive and significant correlations between local cited reference patterns and other measures of journal use. Blecic compared in-house use, circulation, and citation by faculty at the University of Illinois at Chicago’s Library of the Health Sciences and found positive and significant correlations between the measures, although the strength of the correlations, ranging from 0.591 to 0.776, was not particularly high¹¹. McDonald analyzed journals owned by the California Institute of Technology to compare local print use, online use, local publication counts, and local references and found that online use was a significant variable in predicting local cited reference patterns¹². Knowlton, et al. found that local citation and usage statistics are highly associated with faculty valuations for titles which faculty use frequently and value highly, but are not highly associated with titles which faculty use infrequently or rate as lower value¹³. These studies suggest that local cited reference statistics can be used to identify titles of high value to local faculty.

Methods

References cited by NOAA-authored articles were retrieved from the Web of Science, Science Citation Index Expanded database (WoS) during March 2014. We used the following search string to identify articles by NOAA-affiliated authors published each year:

AD=(NOAA OR "nat* ocean* atmos* adm*" OR "nat* mar* fish* serv*" OR NMFS) AND PY=2009

We specifically included the National Marine Fisheries Service (NMFS) in our search string because we know from prior experience of identifying publications by NOAA authors that authors affiliated with NMFS laboratories and offices are less likely to include NOAA in their stated affiliations than NOAA authors affiliated with other branches of the agency. Although our search string does not retrieve publications by NOAA authors who listed their NOAA laboratory or division but did not include any reference to NOAA or NMFS in their stated affiliation, we feel that, given the large number of articles and references the string did retrieve, it is unlikely that excluding such publications has significantly affected our results. In addition, the more restrictive search string has the advantage of increasing the precision of the articles retrieved, meaning that our analyses are unlikely to be affected by false matches.

We then added the results of each search to our “Marked List,” selected the Authors, ISSN, Title, Cited References, Times Cited, Accession Number, and Source fields for export, and exported all of the records to “Other File Formats.” We then copied the resulting files into a single text file for each year and then saved it as an .isi file for later import into the Science of Science (Sci2) Tool¹⁴.

To analyze the cited references by year and by journal, we converted the text file for a single year into a new .csv file and opened the .csv file with Excel, which parsed the cited references into columns for further analysis. We discarded all of the columns except the year and journal, sorted the references by publication year, and discarded the extraneous rows as well as a small number of incorrectly formatted references.

We then performed some rudimentary data cleaning on the titles of referenced journals to increase the accuracy of our final counts. We standardized variant forms of the journal title Eos, Transactions of the American Geophysical Union, since this standardization is not performed automatically by WoS. We also modified the cited references to two journals that had changed titles, so that all references to these journals refer to their current titles. That is, we changed references to the Journal of Applied Meteorology into references to the journal’s current title, the Journal of Applied Meteorology and Climatology, and changed references to Estuaries to become references to Estuaries and Coasts. References to single journals that have subsequently split into multiple parts, such as the Journal of Geophysical Research and Deep-Sea Research, were allowed to stand as referenced, since it was impossible to assign these citations to a specific part of the current journal.

Finally, we used various functions in Excel to calculate the number and percentage of references per year and per journal. We then repeated this process for each additional year.

Next, we created journal co-citation networks for each of the five years in our analysis. Journal co-citation networks are an elaboration of document co-citation networks in which nodes represent journals and edges, or connections between nodes, represent journal co-citations¹⁵. In such networks, journals are connected if at least one current article referenced previous articles published in both journals in the same current article. Journal co-citation networks have been frequently used to map the intellectual structure of scientific research in general or within certain disciplines and used to identify journals for collection development¹⁶.

To create a journal co-citation network for a single year, we loaded the original .isi file for that year into Sci2, which converted it into a .csv file, opened the new .csv file, and copied the “Cited References” column into a new Excel file. We then parsed the cited references using the “Text to Columns” feature in Excel, removed all of the citation information except the journal name, and saved the file as a .csv. We then opened the .csv file in Notepad++, replaced all of the commas (,) with pipes (|), removed the excess pipes, and then saved the file as a .csv file.

We then loaded the new .csv file into Science of Science Tool and created a co-occurrence network on the modified ‘Cited References’ column, using a custom function file to count the number of articles that cited each journal. Once the network had been created, we removed edges, or journal co-citations, with a weight of less than 500 to focus the network on the strongest co-citation relations between journals, extracted only the largest connected component of the network, and visualized the resulting network using Gephi¹⁷. We then repeated this process for the remaining years.

Results

We identified 8,737 articles published by NOAA-affiliated authors from 2009 to 2013. These articles contained a total of 402,126 cited references, which formed the basis of our analysis. Figure 1 shows the total number of articles, cited references, and average number of cited references per article per year in our document set. Both the total number of articles published per year and the number of cited references per year increased substantially during this time period, with the number of articles increasing from 1,532 in 2009 to 1,981 in 2013 (a 29% increase) and the number of references increasing from 66,753 in 2009 to 94,244 in 2013 (a 41% increase). The average number of references per paper, however, grew at a more modest pace, increasing from 43.57 in 2009 to 47.53 in 2013 (a 9% increase).

Figure 1.

Total numbers of articles, references per article, and cited references per year for NOAA-authored articles from 2008 to 2013.

Next, we analyzed the age of publications referenced by NOAA authors. Figure 2 plots the percentage of cited references in each year by the age of the publication referenced at the time the reference was made. That is, approximately 7% of the cited references made in 2013 were to articles 1 year old. The plotted curves are remarkably consistent over the 5 years analyzed, with references to publications peaking at 2 — 3 years after their original publication date, decaying at a relatively linear rate between years 4 and 15, and gradually flattening out after year 20. Figure 3 shows the cumulative distribution of all cited references by approximate quartile. The distribution shows that the citation half-life of NOAA cited references is approximately 8 years, with 50% of the cited references being to publications 0 — 7 years old, 25% to publications 8 — 15 years old, and so on. This half-life is normal for publications in the NOAA-related disciplines, since the 2012 edition of Journal Citation Reports gives citation half-lives of between 7.7 years for Meteorology and Atmospheric Sciences and 9.4 years for Oceanography.

Figure 2.

Percentage of cited references in each year per year of age of the publication referenced.

Figure 3.

Age distribution, in years, of all the cited references in our analysis.

Next, we analyzed the journals referenced by NOAA authors. Figure 4 plots the cumulative percentage of cited references per journal for the most frequently referenced journals in each year. Even more than with the distributions of references per year, the distribution curves per journal are remarkably consistent across years. It is interesting to note that 15% of the over 400,000 cited references in our analysis were to articles published in just 5 journals, while 25% of the references were to 10 journals, and 50% of the references were to 60 journals.

Figure 4.

Cumulative percentage of cited references per journal for the 75 most frequently referenced journals per year.

These distributions are consistent with those predicted by Bradford’s law of scattering, which suggests that the vast majority of the references in any publication set are to a relatively small number of “core” journals, and the remaining references are more widely dispersed among a larger number of peripheral journals. This suggests that the NOAA Central Library can provide for the majority of the references made by NOAA authors with a relatively small collection of core journals. A collection of just 60 journals would provide for 50% of the cited references made during 2009-2013, and a collection of 200 journals would provide for 66% of all cited references during this time period.

We next analyzed the number of references per journal per year to determine if the individual journals that make up this core in one year remained the same in subsequent years. That is, we attempted to determine if the journals that make up this core are fixed, or if they change from year to year. Figure 5 charts the number of cited references per journal per year for the 50 most frequently referenced journals in 2013. We focused on journals frequently referenced in 2013 both to see if journals that are currently important to NOAA authors were also important in previous years, and to identify journals that may have gained importance over this time period.

Figure 5.

Number of cited references per journal per year for the 50 most frequently referenced journals in 2013. Note the logarithmic scale of the y-axis.

Figure 5 suggests that, for the most part, those journals most frequently referenced in 2013 were also frequently referenced in all of the previous years in our analysis. Although some journals changed position from year to year, the 12 most frequently cited journals in 2013 were also the 12 most frequently cited journals in each of the previous years. We list these journals in Table 1. The remainder of the journals shown in Figure 5 remained highly referenced from year to year, although the yearly variability in reference counts increased as the total number of references decreased, suggesting that this second tier of journals is more susceptible to fluctuations in research focus by NOAA authors.

Table 1.

The most frequently referenced journals by NOAA authors from 2009 to 2013.

Journal	References	% of Total
Journal of Geophysical Research – Atmospheres	19,128	4.76
Geophysical Research Letters	13,383	3.33
Journal of Climate	12,339	3.07
Monthly Weather Review	9,679	2.41
Journal of the Atmospheric Sciences	8,121	2.02
Science	7,988	1.99
Atmospheric Chemistry and Physics	7,738	1.92
Marine Ecology Progress Series	6,968	1.73
Canadian Journal of Fisheries and Aquatic Sciences	5,981	1.49
Bulletin of the American Meteorological Society	5,732	1.43
Nature	5,732	1.43
Journal of Geophysical Research – Oceans	5,547	1.38

Finally, we created journal co-citation networks for each of the five years in our analysis to determine whether these core journals were referenced by all NOAA authors, or if there were differences in what journals were considered to be core by different NOAA authors. A visualization of the network derived from articles published in 2013 is given in Figure 6.

Figure 6.

Journal co-citation network derived from the cited references of NOAA-authored articles published in 2013. Font size indicates the number of NOAA articles that referenced articles in each journal. Line size and darkness indicates higher numbers of co-citations between the connected journals. To focus the visualization on the strongest journal relationships, connections of less than 500 co-citations were removed and only the largest connected component is shown.

As with the distributions of references per year and journal, the overall structure of the network remained fairly constant over the five years, with journals related to weather, climate, and the atmospheric sciences clustered on the left, and journals related to marine biology and ecology clustered on the right. These two clusters of journals are connected by interdisciplinary and oceanography journals, although the interdisciplinary journals tend to be more frequently referenced in combination with the atmospheric science journals than with the marine biology journals. Within the atmospheric sciences cluster, there also seems to be a loose divide between basic journals covering the physical composition and processes of the atmosphere and applied journals dealing with weather and climate. A similar divide seems to occur in the marine biology cluster between basic journals on marine biology and ecology and applied journals on fisheries and fishery policy. Finally, there seems to be relatively few connections between these two clusters of journals—that is, few NOAA articles reference both marine biology and atmospheric science journals—suggesting that NOAA authors tend to work along disciplinary lines.

Discussion

We find that access to substantial numbers of scientific publications is necessary for NOAA authors to produce new articles. Our analysis suggests that access to between 40 and 50 publications is necessary for an author to produce a single new article. Our results also suggest that NOAA authors required access to over 90,000 publications in 2013 alone to produce nearly 2,000 original articles. Of course, it is likely that some of these references are duplicates, since some NOAA authors undoubtedly published multiple articles during this time period that cite the same prior publications. Despite this, however, our results do indicate that NOAA authors require access to tens of thousands of publications every year to produce their original research. Since NOAA requires its employees to publish as proof of their scientific activity, our results suggest that NOAA thereby requires access to scientific information to support the production of these articles. Furthermore, since our results indicate the number of publications, references per publication, and cited references have all increased over the past five years, it is logical to conclude that this need has grown over time, and it seems likely that it will continue to grow in the near future.

The emphasis on recent publications in NOAA authors’ referencing patterns—references peak after 2 years and nearly a quarter of all references are to publications less than four years old—is not surprising. Nearly 50 years ago, de Solla Price found that the majority of references from current papers are to other relatively recent papers, so this aspect of NOAA authors’ referencing behavior is to be expected¹⁸. Perhaps more surprising is the relative importance of older publications in addition to the emphasis on current publications. Nearly a quarter of all the cited references in our analysis, or over 90,000 cited references over these five years, are to publications 16 or more years old. Taken together, these findings suggest that NOAA authors not only require access to the latest scientific publications, but also need continuing access to those publications for decades.

A similar trend in the ages of cited references was also reported by Kimball, et al. for atmospheric science faculty publications at Texas A&M¹⁹. They too found that although the cited references at their institution emphasized recent publications, there were also numerous references to publications over 16 years old. Walcott did not analyze the ages of the references analyzed, so our results could not be compared with hers²⁰. The general pattern identified in our analysis and in that of Kimball et al has also been observed by numerous cited reference analyses in other disciplines, although the age of the peak and the rate of decay vary between disciplines²¹.

Our analysis of the journals referenced by NOAA authors suggested that although the identity of the journals most highly referenced by NOAA authors remained fairly constant over time, the number of references to two journals in particular increased by over 500% over this time period. These journals are Nature Geoscience and Plos One. References to Nature Geoscience increased from 45 references in 2009 to 312 references in 2013, a 593% increase. Since the first issue of Nature Geoscience was published in January 2008, it is perhaps unsurprising that references to the journal would increase as increasing numbers of articles were published in the journal, but it is interesting to note the speed at which this new journal established itself as a major source of information for NOAA authors. References to Plos One, on the other hand, increased 1,729% over this time period, from 24 references in 2009 to 439 references in 2013. The reasons for this remarkable increase are not clear. Certainly the total number of articles published in Plos One increased dramatically during this time period—from over 4,000 articles in 2009 to over 31,000 articles in 2013—so, as with Nature Geoscience, the increasing availability of articles to reference may have driven some of this change. But mere increases in the number of articles published in both journals cannot entirely explain the increases in NOAA references, because articles published in the journals had to be brought to the attention of NOAA authors, be sufficiently relevant to their work, and be of sufficient quality for them to be referenced. Additional research seems necessary to fully understand the reasons for these increases and to determine if such changes are reflected in referencing patterns at other institutions.

Smaller increases in the number of references to other journals may signal shifts in the research focus of NOAA authors. Increases in the number of references to journals such as Climatic Change (a 214% increase), Global Change Biology (a 113% increase), and Climate Dynamics (a 150% increase) over this time period may signal a shift toward research on climate change and its effects on terrestrial and marine ecosystems. Other notable increases were to journals such as Atmospheric Chemistry and Physics (a 115% increase), IEEE Transactions on Geoscience and Remote Sensing (a 112% increase), Fisheries Research (a 101% increase), and the Journal of Hydrometeorology (a 106% increase). Our results suggest that analyzing the rate of references to various journals over time could be used to identify shifts in the research focus of authors at academic institutions as well.

Taken together, our analyses suggest that there is a set of 12 core journals for NOAA authors that remained constant throughout this time period and were consistently referenced more often than any other journals in our analysis. This identification of a core set of journals in local cited references was also reported by Walcott, Ibeun, Kimball et al, and by studies in a range of other disciplines²². Beneath this core set of journals were a large number of more peripheral journals whose identity and number of references were more fluid, although the magnitude of references to each journal remained relatively constant over the five years. Nevertheless, increases in references to certain journals suggest that the identity of the journals in this core is not firmly fixed and that journals that were peripheral in the past may become core in the future. This suggests the need to repeat cited reference analyses over time to identify such changes not only at NOAA, but at other institutions as well.

Our analysis also suggests that there are different sets of core journals for different communities of NOAA authors. The central journals for atmospheric science authors include four American Meteorology Society journals (Journal of Climate, Monthly Weather Review, Bulletin of the American Meteorological Society, and Journal of the Atmospheric Sciences) and three American Geophysical Union journals (Geophysical Research Letters, Journal of Geophysical Research—Atmospheres, and Journal of Geophysical Research—Oceans). The central journals for marine biology authors, on the other hand, are Canadian Journal of Fisheries and Aquatic Sciences and Marine Ecology Progress Series. Only a few journals like Nature and Science are referenced by both communities of authors.

Many of the core journals that we identified were also listed in previous studies. The core journals for NOAA’s atmospheric science authors are similar to those reported by Kimball et al.²³. The high usage of the American Meteorology Society journals and the American Geophysical Union journals, as well as other journals like Atmospheric Environment, the Quarterly Journal of the Royal Meteorological Society, and Atmospheric Chemistry and Physics, by NOAA authors is also reflected in their study. There are, however, differences in emphasis between their results and ours, with NOAA authors using more journals related to weather forecasting and Texas A&M authors using more journals related to optics. Our list of core journals for fisheries science is also similar to, but not exactly the same as, the list of fisheries journals identified by Ibeun, with his list emphasizing aquaculture journals and our list emphasizing marine ecology²⁴. Our and Ibeun’s lists are also similar to, but slightly different from, that of Walcott, which has more titles related to geochemistry and geology²⁵. These results suggest that although there may be broad agreement between institutions on the most highly used journals in a particular discipline, the actual journals most often used may differ between institutions depending on local interests and priorities.

In addition, all three lists differ, in some cases substantially, from those generated by the 2012 edition of Journal Citation Reports for the Meteorology & Atmospheric Sciences and Fisheries subject categories according to total cites, Journal Impact Factor, and 5-Year Journal Impact Factor. This suggests that the journals most often referenced by an entire discipline do not necessarily align with those most often referenced at a particular institution. This calls into question the appropriateness of using aggregate referencing statistics like the Journal Impact Factor to make collection development decisions at a particular institution, although more data are needed to examine potential differences at other institutions and in other disciplines.

Finally, although this analysis was performed in a government library, the similarities between our results and those performed at academic libraries suggest that this general method can be applied at a range of research libraries at academic, governmental, and corporate institutions. The journal co-citation method in particular could be applied in an academic library to identify the degree of overlap between the resources used by different faculty departments within the institution. It could also be used to identify potential differences between the resources used by faculty and those used by graduate students in their theses and dissertations. Our method of tracking references per journal per year could also be used in academic libraries to identify shifts in research focus at their institutions. Finally, the amounts of total references and references by age could be used to justify academic library journal collections, set thresholds for journal subscription vs. interlibrary loan, and identify less-used journals for potential cancellation.

Conclusion

Our analysis of the references cited by NOAA-affiliated authors in their articles published between 2009 and 2013 produced a number of potentially useful results for use within our agency. We found that NOAA authors require access to tens of thousands of scientific publications every year to produce their original research. Although the majority of these publications were published during the past 8 years, NOAA authors required access to substantial numbers of older publications as well. A small number of core journals published the majority of the publications referenced by NOAA authors and, for the most part, these core journals remained highly referenced throughout the years in our analysis. However, we found substantial differences in the journals referenced by authors from different disciplines, with relatively few journals referenced by authors across the agency. These findings largely support and confirm the results of cited reference studies performed in similar disciplines at academic institutions.

Our analysis also produced several results of broader interest to the library community. Our results demonstrate that the technique of analyzing cited references by institutional scientists to gain insights for collection development and justification purposes can be successfully applied to research-oriented libraries both inside and outside the academic environment. Our analysis also suggests that there may be differences in the journal referencing patterns of authors in the same discipline at different institutions, suggesting that the findings from an analysis performed at one institution may not be applicable to authors at other institutions. This also calls into question the use of general citation indicators like the Journal Impact Factor for collection development decisions, since the priorities and referencing patterns of an entire discipline may not be reflected in those of a particular institution. Finally, our analysis suggests that authors’ referencing patterns can change somewhat over time, suggesting that cited reference analyses performed at a particular institution should be updated periodically to account for such changes.