Into the gray: a modified approach to citation analysis to better understand research impact

Introduction

Academic authors and funders often want to know the “impact” of their publications, and this impact is generally judged by how and where the paper is cited in other academic works. This limited appraisal has been expanded in recent years as many are beginning to argue that nonacademic publishing venues should be included in assessing the impact of academic publications. This is an issue of particular concern with the growing emphasis on “knowledge translation” from the scientific literature to policy and practice applications 1–3 and to sources other than the traditional peer-reviewed and indexed venues, in other words, translation into the “gray literature” 4.

In this comment and opinion piece, the authors describe the process of developing and applying a “modified citation analysis” that builds on existing methods of examining a research paper's impact in two key ways: (1) by deliberately including gray literature in the citation analysis search process, and (2) by including quantitative and qualitative methods of analysis to gain a better understanding of how a research paper was used. By broadening the search and deepening the level of analysis, we suggest this new approach can better assess the impact of a given research paper—both within and outside of traditional peer-reviewed venues. We begin with a review of gray literature and then describe current methods for analyzing the impact of a research paper. Finally, we use a specific example to describe our new approach, highlight its potential for evolving the field of citation and impact analysis, and discuss future refinements and evaluation.

Gray literature

Gray literature is an important aspect of scientific evidence as it “produc[es] and distribut[es] the seeds of new knowledge” 5. Current definitions of gray literature are overly broad and lack clear lines of distinction between different producers, users, and dissemination techniques (Table 1). A commonly cited definition is: “Information produced on all levels of government, academia, business and industry in electronic and print formats not controlled by commercial publishing; i.e., where publishing is not the primary activity of the producing body” 6. Another definition proposed by the International Journal on Grey Literature is: “the information and resources that do not categorically fall into what is available via standard, traditional or commercial publishing channels” 7. Even these well-respected definitions provide only a vague understanding of what is classified as gray literature.

Table 1.

Differences between traditional publishing and gray literature

The distribution of gray literature in a multitude of mediums by individual scholars and scientists, research institutes, and community and nonprofit organizations has become widespread since the advent of online repositories and other open access venues 8–10. Organizations produce policy briefs, issue briefs, or technical reports on specific content areas (e.g., health, environment, and poverty) meant to inform decision making that may not enter formal publication venues. This can be attributed to a bias toward publishing larger studies with specific results (i.e., positive, novel, and generalizable), the considerable time lag between research production and publication 11, or the production of gray literature for internal organization purposes.

Gray literature as a key dissemination strategy.

Gray literature remains underused and undervalued. However, its importance and relevance is becoming apparent in areas such as policy development and research 12–14. In the medical field, the well-regarded Cochrane systematic evidence review process requires the inclusion of any relevant gray literature as part of the evidence base 15.

The exclusion of gray literature can skew the results of research syntheses that may have important ramifications for the accumulation and dissemination of scientific knowledge. For example, McAuley et al. 16 and Hopewell et al. 4 argue that the exclusion of gray literature can influence the results of meta-analyses. In one example, by extending a search to include literature outside “mainstream sources,” an estimated 29.2% more sources were found 17. Gray literature is also important in many fields of research in both understanding the theoretical underpinnings of findings 18 and identifying research gaps that produce research questions 19. This is especially true in health research, where findings published in the gray literature often inform the development of clinical trials 16.

Gray literature is extremely valuable in providing timely communication on complex issues—often using simple, actionable, and comprehensive language—making it a key resource for stakeholders outside of academia 20, 21. These works often include important information on context, policy decisions, and public interest that are of particular value to decision makers 22. It is apparent that while the amount and influence of gray literature is dramatically increasing, quality assurance, search methods, and impact measures for gray literature have not kept pace.

Quality of gray literature.

The sheer volume and accessibility of gray literature also raises the issue of the quality of these resources 23. With the general absence of peer review, the only way to ensure the quality of gray literature is through critical assessment. Although the producers of gray literature may be experienced and knowledgeable in their field, there is essentially no formal quality control. There have been suggestions to improve the quality of the more “traditional” types of gray literature such as technical and evaluation reports, strategic plans, policy briefs, and others. An example is to include standard information in documents such as: (1) the process of creating the product, (2) the review process (if any), and (3) any conflicts of interest 24. The Grey Literature International Steering Committee provides guidelines on what scientific and technical reports should look like. They should include a peer-review process, ethical considerations, and publishing and editorial recommendations 25. However, given the unregulated space that gray literature inhabits, it is unrealistic to expect all who create it to conform to such standards, leaving the end user responsible for assessing its quality.

Access to gray literature.

Searching for gray literature is challenging and can be time consuming. There is currently no accepted strategy or protocol for doing so. While some gray literature becomes incorporated in the published literature and vice versa, much gray literature remains “fugitive” 26 and difficult to capture. Since gray literature falls under the broad definition of “materials not published in a traditional way,” it can be found in numerous places and requires many different search strategies to collect it. Although some of the commercially available databases, such as Web of Science and Scopus, contain gray literature (e.g., dissertations, conference proceedings), this is not their primary function, and they likely capture only a fraction of existing material. Google Scholar is a good adjunct to these but similarly does not have a primary focus on gray literature. As its search algorithms are unknown, it is difficult to understand and assess what has been included and excluded from searching Google Scholar.

Another challenge with gray literature is the lack of bibliographic control or classification 27. Unlike scholarly works such as journal articles, most gray literature is not indexed in databases. Although some databases do include gray literature, poor or inconsistent choice of titles or keywords can make such documents difficult to source, catalog, and store 28. The “scattered” and disorganized nature of gray literature presents particular difficulty for those wishing to incorporate it into a systematic and comprehensive search strategy, as is required for many types of literature review.

Current methods for analyzing citation impact

Several methods of analyzing individual study citations are currently used to determine impact in terms of penetration into the broader academic literature. These include methods such as citation analysis, co-citation analysis, bibliographic coupling, and a co-citation–bibliographic coupling hybrid. In general, the purpose of these types of analysis is to track citations of publications and, in some methods, group similar papers together. Although these methods of impact analysis are widely used, they fail to capture gray literature, and all are based solely on counts or quantitative comparisons of citations. That is, none delve deeper to explore how (as opposed to simply how often) a research paper has been used.

A relatively new method of citation analysis is the Becker Medical Library Model for Assessment of Research (Becker Model), which analyzes research impact through indicators such as “contribution to the knowledgebase” and “change in practice” by tracking diffusion of research output and activities 29. The Becker Model suggests locating gray literature materials such as non-peer-reviewed journal articles, trade publications, and supplemental materials but does not assist in locating these items, and it still relies solely on quantitative counts of research outputs, albeit of more types. (For more information on this approach, visit https://becker.wustl.edu/impact-assessment.)

Modified citation analysis.

We were interested in tracking the impact of a specific research article published in a high-impact, peer-reviewed journal 30. This was the first randomized controlled trial (RCT) of universal screening in health care settings for exposure to intimate partner violence (IPV). We sought to examine how this new evidence was taken up in scholarly literature, as well as in broader policy, practice, and advocacy. (For a description of the findings of that analysis, see Wathen et al. 31.)

After reviewing current methods for citation analysis (as above), we determined that none of these could effectively track the influence of a single paper through both commercially published and gray literature. For this reason, we developed a “modified citation analysis” method that would capture both traditional and gray literature and allow both quantitative and qualitative interpretive analysis on extracted data. Such an approach was important because (1) we knew that the content of the paper was of interest and contentious in the academic realm and beyond; therefore, limiting our search to only commercially published sources would not provide an accurate portrayal of how the findings were used, and (2) we were interested in how the paper was interpreted and used; therefore, citation counts alone would be insufficient.

We searched the scholarly and academic and gray literature for sources with high potential to influence policy, practice, or further research. Since we were primarily interested in the various uses of the study findings, we did not assess source quality or perform any formal results synthesis of the included materials. Preliminary scans of the literature helped determine the most effective subject headings and keywords or phrases to use for these searches. These terms were then applied using the appropriate database or other online search tools to retrieve articles, documents, media reports, or other items that specifically cited or discussed the MacMillan et al. paper 30. All retrieved results that met inclusion criteria were stored in a reference management program (Figure 1).

Figure 1.

Search results flow diagram

Google Scholar and Google Scholar Update also index books.

The gray literature was searched using multiple resources, such as multidisciplinary databases (Scopus), websites (MedlinePlus), and point-of-care tools (MDConsult, UpToDate). Major health care associations and professional organizations likely to include related content were identified, and their websites were individually searched. News media reports were searched by using news databases, including Factiva, Lexis Nexis, Google News, and Proquest Canadian Major Dailies.

“Traditional” citation analysis statistics were computed using SPSS 20.0. These included the number of sources citing the MacMillan et al. paper 30 (by search method) and the number of MacMillan et al. 30 citations per source. Data analysis took place in three steps according to the specific research questions guiding the analysis: (1) analyzing content of text extractions, (2) categorizing sources regarding their position on IPV screening, and (3) coding sources regarding how they defined IPV screening. Table 2 illustrates our coding process.

Table 2.

Methods details: coding process, search results and source citations

Adapting this approach.

This method is theoretically adaptable to other contexts where tracing the uptake and interpretation of new knowledge would benefit from more than quantitative citation counts and inter-linkages. The gray literature search strategy would need to be specifically tailored to the area of interest and could involve more extensive searching depending on the discipline and topic area. For this reason, we recommend that experts in the field be involved in the process and that the searches be undertaken by trained information professionals.

In addition to a tailored search strategy, the quantitative and qualitative coding of the text citing the original document will also depend on the specific questions being addressed. Some codes may apply to any paper; for example, anyone might code whether the paper is cited for a methodological or theoretical point, or might examine whether the citation appears in the “Introduction,” “Method,” or “Discussion” sections. Most codes, however, will be highly specific to the goals of the analysis.

Discussion

Our modified citation analysis has several advantages over more mainstream methods of citation analysis. By expanding our search strategy, we were able to find more citing sources. Specifically, using search techniques designed to track citations in the commercially published academic literature (e.g., Google Scholar's “cited by” tool and Web of Science), we found eighty sources citing the MacMillan et al. paper 30. The gray literature searches (e.g., Google, news databases, and association websites) found an additional twenty-nine sources. Second, by moving beyond basic citation counts and coding how the paper was cited, we were able to provide a richer understanding of its impact on subsequent research, as well as policy and practice guidance regarding screening for intimate partner violence. It also provided a way to assess the evolution of the main debates in this area (for example, confusion caused by varying definitions of “screening”).

This method requires additional resources. The much broader range of potential search venues demands more time and expertise. Delving into gray literature is a challenging task and requires planning and coordination, including consideration-specific inclusion/exclusion searching. Unlike database searching, common nomenclatures rarely exist for searching diverse gray literature sources; therefore, the concept of consistency in search terms across sources is difficult to achieve. It is important to document all search strategies used in each source to attempt reproducibility in the ever-changing nature of online resources. Similarly, the addition of a qualitative analysis component necessitates multiple researchers to undertake coding, analysis, and interpretation that is rigorous and adheres to qualitative methodology standards 32. More research using this method is required to assess its validity and utility across different applications.

Conclusion

The modified citation analysis approach appears well suited to report how new research evidence is taken up, interpreted, and used in published material, a quality that is valuable to those attempting to assess the effectiveness of knowledge translation strategies 3, 33. We believe it is a useful extension of traditional citation analysis methods and addresses some of the gaps in the proposed Becker Model 29. Since there are no validated methods for conducting this method of citation identification, we created a new search and analysis strategy, building on the growing importance and acknowledgment of searching gray literature 34, 35.

This new approach adds to the ongoing discussion regarding the inclusion and analysis of gray literature resources in searching for different types of “evidence” to include in both academic research, practice, and policy contexts. We hope that it encourages a more holistic approach to assessing “impact” of new research evidence.

Shannon L. Sibbald, PhD (corresponding author), ssibbald@uwo.ca, Faculty of Health Sciences and Schulich Interfaculty Program in Public Health; Jennifer C. D. MacGregor, PhD, jmacgre9@uwo.ca, Faculty of Health Sciences and Faculty of Information & Media Studies; Marisa Surmacz, MLIS, marisa.surmacz@uwo.ca, Western Libraries; C. Nadine Wathen, PhD, nwathen@uwo.ca, Faculty of Information & Media Studies; Western University, 1151 Richmond Street, London, ON Canada

Biography

Shannon L. Sibbald, PhD (corresponding author), ssibbald@uwo.ca, Faculty of Health Sciences and Schulich Interfaculty Program in Public Health; Jennifer C. D. MacGregor, PhD, jmacgre9@uwo.ca, Faculty of Health Sciences and Faculty of Information & Media Studies; Marisa Surmacz, MLIS, marisa.surmacz@uwo.ca, Western Libraries; C. Nadine Wathen, PhD, nwathen@uwo.ca, Faculty of Information & Media Studies; Western University, 1151 Richmond Street, London, ON Canada