An exploratory analysis of PubMed's free full-text limit on citation retrieval for clinical questions

Abstract

Objective:

The research sought to determine (1) how use of the PubMed free full-text (FFT) limit affects citation retrieval and (2) how use of the FFT limit impacts the types of articles and levels of evidence retrieved.

Methods:

Four clinical questions based on a research agenda for physical therapy were searched in PubMed both with and without the use of the FFT limit. Retrieved citations were examined for relevancy to each question. Abstracts of relevant citations were reviewed to determine the types of articles and levels of evidence. Descriptive analysis was used to compare the total number of citations, number of relevant citations, types of articles, and levels of evidence both with and without the use of the FFT limit.

Results:

Across all 4 questions, the FFT limit reduced the number of citations to 11.1% of the total number of citations retrieved without the FFT limit. Additionally, high-quality evidence such as systematic reviews and randomized controlled trials were missed when the FFT limit was used.

Conclusions:

Health sciences librarians play a key role in educating users about the potential impact the FFT limit has on the number of citations, types of articles, and levels of evidence retrieved.

Highlights

• When searching 4 clinical questions in PubMed, use of the free full-text (FFT) limit resulted in approximately a 90% reduction in retrieved citations.

• When searching four clinical questions in PubMed, high-quality evidence such as systematic reviews and randomized controlled trials were missed when the FFT limit was used.

Implications

• Health sciences librarians can educate clinicians, researchers, and consumers about the impact of the FFT limit on the number of citations, types of articles, and levels of evidence.

• Providing cautionary statements in online search systems that use FFT limits, such as PubMed, about the effect FFT limits will have on searches may raise user awareness of the limitations in search results.

Introduction

Clinicians in the health professions encounter patient care questions daily for which they need to find answers [1,2]. Since the mid 1990s, there has been a strong push across health care disciplines to utilize evidence-based clinical practice (EBCP) [3]. Practitioners of EBCP pose a clinical question, search for and appraise the evidence, and use their clinical judgment in conjunction with the patient's values and circumstances when considering how or if to use the evidence [4,5].

Research on medical information-seeking behavior shows that clinicians increasingly rely on Internet-based or electronic resources to find evidence for patient care decisions [6,7]. The most frequently cited reasons for preferring electronic formats are ease of use, convenience, and remote access [8]. De Groote and Dorsch have reported that databases providing direct, seamless links to full-text articles were more heavily used than other resources at their institution [9]. Wentz has suggested that a “Full Text On the Net” (FUTON) bias exists, meaning that individuals may limit their research to journals that are available in full text and disregard relevant studies that cannot be accessed in full text [10]. In a study to determine the effect of full-text availability on the impact factor of journals, Murali and colleagues have predicted that the tendency to pick the “low hanging fruit of convenience” will increase the chances that a FUTON article will be read and cited [11]. They further state that “visibility and easy availability to the user may determine adoption of ‘available evidence’ as ‘current best evidence’ in health care” [11].

The National Library of Medicine's PubMed database is widely used by clinicians and researchers to retrieve biomedical journal literature. Since 2003, PubMed citations with links to subscription or fee-based full text and free full text (FFT) have been searchable using subset field tags [12], and since March 2006, the full text and FFT limit options have been prominently displayed on the PubMed Limits page [13]. In May 2007, 47.8% of PubMed citations were reported to have links to full text [14], and as of February 2008, 5,944 journals offered full-text access through LinkOut [15]. The FFT limit restricts search retrieval to citations for which online full text of an article is freely available. In May 2007, 10.3% of PubMed citations were reported to have FFT links [14], and as of February 2008, 976 LinkOut journals had FFT access [16].

To the authors' knowledge, no one has examined the effect of the FFT limit on a PubMed search arising from a clinical question. Using four clinical questions, this study explored how the use of the FFT limit affects the number of retrieved citations and how the use of the FFT limit affects the types of articles and levels of evidence retrieved.

Methods

The authors generated clinical questions by reviewing the Clinical Research Agenda for Physical Therapy for questions reported to be of importance for physical therapists [17]. These questions were potentially representative of multiple practice settings and were identified by the Agenda as being possible to address within five years [17]. Therefore, findings from research related to the questions reported in the Agenda could be available in 2007 when the authors' searches were performed.

Using consensus of the three authors, four specific, searchable therapy questions were developed. The four clinical questions were:

• Do resistive exercises reduce the risk of fracture in postmenopausal women?

• Do strengthening exercises improve gross motor function in adolescents with cerebral palsy?

• Does manipulation reduce pain in adults with acute low back pain?

• Does physical therapy improve mobility of adults post-stroke during their inpatient rehabilitation?

The authors constructed PubMed search strategies modeled on the findings of Herskovic and colleagues who analyzed a single day's query log from PubMed [18]. Herskovic and colleagues reported that the Boolean operator “AND” was the most commonly used operator, although it was only used in approximately 11% of the searches. They also found that advanced features such as Medical Subject Headings (MeSH) were seldom used and that the median number of search terms was 3. As a result of Herskovic and colleagues' findings, the authors used the Boolean operator “AND” to connect search terms, used the Boolean “OR” only to connect variations of the same terms, and did not use Clinical Queries or MeSH terms, as the authors considered these search options to be advanced features [18]. Rather the searches were performed using the automatic term mapping feature in PubMed [19].

For each question, the authors used the patient, intervention, comparison, outcome (PICO) format to identify search terms that were entered in the query box on the opening PubMed screen [20]. The Details tab was used to examine how the search terms were mapped. Searches were modified until the authors reached consensus on appropriate search terms. Because abstracts would be used to determine the types of articles and levels of evidence for relevant citations, the Abstracts and English limits were selected. Table 1 illustrates the automatic term mapping for each question. Each search was performed with and without the FFT limit, and the searches, retrieved citations, and abstracts were saved in My NCBI. All retrieved citations and abstracts were saved as text files.

Table 1.

Clinical questions automatic term mapping results^*

After the searches were performed for each question, retrieved citations were reviewed for relevancy. Citations were considered relevant if they broadly met the patient, intervention, and outcome aspects of the PICO statement for each question. When it was unclear from the abstract if the PICO elements were met, consensus of the three authors was used to determine relevancy.

The abstracts of all relevant citations were reviewed to determine types of articles and levels of evidence. Based on the work of Guyatt and colleagues, the authors defined ten types of articles [21] from the set of relevant citations:

• N of 1 randomized controlled trial (RCT)

• systematic review of RCTs

• single RCT

• systematic review of RCTs and observational studies

• systematic review of observational studies

• single observational study

• case series or study

• narrative review

• qualitative study

• practice guideline or position statement

The authors then modified the work of Guyatt and colleagues to arrange eight of the ten types of articles into a hierarchy of evidence: N of 1 RCT, systematic review of RCTs, single RCT, systematic review of RCTs and observational studies, systematic review of observational studies, single observational study, case series or study, and narrative review [21]. Qualitative studies were not included in the hierarchy of evidence because some authors have argued that qualitative studies do not belong in a typical scheme of levels of evidence [22,23]. Additionally, practice guidelines and position statements were not included in the hierarchy of evidence because the level of evidence in these types of articles varies depending on the information used to derive the guideline or statement (i.e., systematic reviews vs. expert opinion) [24]. Prior to searching the four questions, the methods were pilot-tested using the clinical question, “What is the effect of exercise on functional recovery after anterior cruciate ligament injury in adults?” The pilot question was derived from the Agenda, and no modifications to the methods were made after the pilot test[17].

Results

The number of retrieved and relevant citations varied across questions and was influenced by use of the FFT limit (Table 2). Across all questions, fewer citations to studies representing different levels of evidence were retrieved when using the FFT limit (Table 3). When the authors compared the search result with and without the FFT limit across all four questions, one of the ten citations to systematic reviews of RCTs were retrieved; one of the twenty-eight citations to single RCTs were retrieved; one of the six citations to systematic reviews of RCTs and observational studies were retrieved; ten of the seventy-one citations to single observational studies were retrieved; one of the six citations to case series or studies were retrieved; and twelve of the ninety-two citations to narrative reviews were retrieved. Across the four questions, four relevant citations were not representative of the defined types of articles and were not classified.

Table 2.

Number of retrieved citations and number of relevant citations without and with use of the free full-text (FFT) limit when searching the clinical question

Table 3.

Types of articles and levels of evidence of retrieved citations without and with use of the FFT limit when searching the clinical question

Discussion

As expected, the authors found that use of the FFT limit decreased the total number of citations retrieved for the clinical questions examined. Regardless of relevancy and across all 4 questions, the FFT limit reduced the number of citations to 11.1% of the total number of citations retrievable without the FFT limit. This finding was consistent with the LinkOut report that 10.3% of PubMed citations had FFT links [14]. The percentage of FFT will likely increase in the future; a large proportion of this increase will be due to initiatives such as the National Institutes of Health public access policy [25] and archiving of back issues for the PubMed Central repository [26]. Access to current studies may continue to be limited however because of embargo periods [26].

Across all four questions, high-quality evidence such as systematic reviews and RCTs were missed when the FFT limit was used. For example, when searching without the FFT limit, ten systematic reviews of RCTs were retrieved; only one was retrieved when the FFT limit was used. Likewise when searching without the FFT limit, twenty-eight RCTs were retrieved and only one was retrieved when the FFT limit was used. Consistently missing high-quality evidence when searching clinical questions is problematic because it undermines the process of EBCP [5,27].

Limitations

Limitations of this study were that only four questions were searched and the subject domain of those questions was rehabilitation. The authors, however, believe the clinical questions searched are of interest to professionals in a variety of health care disciplines including nursing, occupational therapy, and medicine. Additionally, though applying the FFT limit reduced the number of studies rated at the top of the hierarchy of levels of evidence, these articles were not specifically assessed to determine if they provided an answer to the clinical question. Furthermore, while the authors do not anticipate that a study using clinical questions from other subject domains would have markedly different overall findings, the effect of the FFT limit in additional subject areas may merit exploration.

Two additional limitations of this study are related to the methods used to develop search strategies. The authors did not observe actual PubMed users and did not use advanced features such as Clinical Queries and MeSH terms. Rather, the authors used the findings of Herskovic and colleagues, which were based on over 2 million queries issued by over 600,000 different users, to guide their search strategy development [18]. Therefore, the authors believe their search strategies reasonably represented how a user might search PubMed.

Implications

Health sciences librarians can use these data to educate users that the FFT limit could greatly reduce the number of citations and levels of evidence retrieved. In this study, the authors found approximately a 90% reduction in retrieved citations when the FFT limit was used. For health care consumers, researchers in developing countries, and other communities that may not have resources to obtain subscription-based journals, use of the FFT limit provides a way to access a portion of the biomedical literature. While the amount of freely available full text is currently not optimal, initiatives that seek to increase the amount of research that is freely available are encouraging. However, limitations in the use of the FFT filter are important to understand so that health sciences librarians can inform clinicians, scientists, and others about how to locate needed information.

Because some users may select the FFT limit as a convenient way to find citations to online articles and because use of the FFT limit can greatly decrease the number of retrieved citations, providing users with cautionary statements in search systems that use FFT limits, such as PubMed, can help raise searchers' awareness of the limitations of their use. A similar cautionary statement was posted on the PubMed Clinical Queries page in previous years [28]. Finally, health sciences librarians in their contacts with publishers can advocate on behalf of clinicians, researchers, and consumers for depositing more high-level evidence in freely available repositories such as PubMed Central [29].

Conclusions

For the four clinical questions searched, the use of FFT limit greatly reduced the number of retrieved citations and impacted the types of articles and levels of evidence retrieved. Health sciences librarians have a unique and critical role in the EBCP process. This role includes assisting users with constructing effective search strategies for questions and applying appropriate limits to their search results. Librarians can educate health care consumers, scientists, and clinicians about the effects that the FFT limit may have on their information retrieval and the ways it ultimately may affect their health care and clinical decision making.