Development and validation of search filters to find articles on palliative care in bibliographic databases

Abstract

Background:

Healthcare professionals and researchers in the field of palliative care often have difficulties finding relevant articles in online databases. Standardized search filters may help improve the efficiency and quality of such searches, but prior developed filters showed only moderate performance.

Aim:

To develop and validate a specific search filter and a sensitive search filter for the field of palliative care.

Design:

We used a novel, objective method for search filter development. First, we created a gold standard set. This set was split into three groups: term identification, filter development, and filter validation set. After creating the filters in PubMed, we translated the filters into search filters for Ovid MEDLINE, Embase, CINAHL, PsychINFO, and Cochrane Library. We calculated specificity, sensitivity and precision of both filters.

Results:

The specific filter had a specificity of 97.4%, a sensitivity of 93.7%, and a precision of 45%. The sensitive filter had a sensitivity of 99.6%, a specificity of 92.5%, and a precision of 5%.

Conclusion:

Our search filters can support literature searches in the field of palliative care. Our specific filter retrieves 93.7% of relevant articles, while 45% of the retrieved articles are relevant. This filter can be used to find answers to questions when time is limited. Our sensitive filter finds 99.6% of all relevant articles and may, for instance, help conducting systematic reviews. Both filters perform better than prior developed search filters in the field of palliative care.

What is already known about the topic?

• Palliative care is a relatively young, growing, and multidisciplinary area of expertise; relevant papers are published in palliative care journals, general medical journals, and discipline-specific journals.

• Systematically searching and finding relevant literature in the field of palliative care is complex.

• Many palliative care professionals inefficiently search the literature.

What this paper adds?

• Using a novel methodological approach, we developed and validated a sensitive search filter and a specific search filter for five frequently used databases.

• The specific filter had a specificity of 97.4%, a sensitivity of 93.7%, and a precision of 45%; the sensitive filter had a sensitivity of 99.6%, a specificity of 92.5%, and a precision of 5%.

• Both filters perform better than prior developed search filters in the field of palliative care.

Implications for practice, theory, or policy

• Our search filters can support literature searches in the field of palliative care.

• Our specific filter can be used to find answers to, for instance, clinical questions, when time is limited.

• Our sensitive filter may, for instance, help conducting systematic reviews.

Introduction

Search filters are developed for more efficient and effective searching of the literature. They are typically created by identifying and combining search terms to retrieve scientific publications with a common feature.¹ Filters can be expert informed, research based, or a combination.¹ Information about the methods of filter development, along with validation, is important to enable potential users to judge whether the filter is relevant and reliable.^1,2 Over the last two decades, research methods have been increasingly used to develop and test search filters, to make them more robust and reliable.²

In the field of palliative care, little attention has been paid to search filter development. Systematically searching and finding relevant literature in this field is rather complex. Palliative care is a relatively young and growing area of expertise that is multidisciplinary by its nature. Relevant papers are likely to be published in palliative care journals, general medical journals, and discipline-specific journals (such as oncology and geriatrics). Previous research has shown that many palliative care clinicians search PubMed ineffectively, predominantly because they do not use the right search terms or incorrectly narrow their search.³ Search filters can help in the process of identification of relevant publications. A search filter with high specificity is designed to find almost only relevant articles and can, for instance, be used in clinical practice to find answers to clinical questions when time is limited. A search filter with high sensitivity can be used in research, for instance, for systematic reviews, to make sure no relevant articles will be missed. Prior developed filters showed poor performance.^4,5 We developed and validated an improved specific search filter and an improved sensitive search filter for the field of palliative care, applicable in frequently used databases.

Methods

Definition of palliative care

To define articles as concerning palliative care during the development of the search filters, we used the palliative care definition from the World Health Organization, because this is a broad and worldwide accepted definition of palliative care:

Palliative care is an approach that improves the quality of life of patients and their families facing the problem associated with life-threatening illness, through the prevention and relief of suffering by means of early identification and impeccable assessment and treatment of pain and other problems, physical, psychosocial and spiritual.⁶

Creating sensitive and specific filters

In the development of the filter, we have followed a method that was developed by one of the authors (W.M.B.), and which has been used to create other filters before (Figure 1).⁷ The method was based on the quality criteria of the filter appraisal tool as developed by Glanville et al.²

Figure 1.

Flowchart of the development and validation of search filters to find articles on palliative care in bibliographic databases.

Creating a gold standard set

This step was conducted by the librarian who co-authored the manuscript (W.M.B.). Potentially relevant journals were identified by searching the NLM catalog of journals indexed in MEDLINE on PubMed with a predefined set of MeSH terms related to our topic: (“Terminal Care”(mh) OR “Palliative Care”(mh) OR “Hospice Care”(mh)). This resulted in eight journals, five of these (J Palliat Med, J Pain Symptom Manage, Palliat Med, Am J Hosp Palliat Care, and Int J Palliat Nurs) were randomly selected. A random set of 5 years was chosen between 2000 and 2014, and all articles published in these journals in these specific years were downloaded. We included any article that was relevant to end of life, regardless of study type. From these, we selected articles that were indexed in MEDLINE and that had an abstract. Of the 1842 retrieved articles, a random sample of 750 articles was exported into EndNote.

To identify potentially non-relevant articles, nine arbitrary journals were selected from the NLM catalog (Genet Couns, Health Promot Int, Healthc Inform, Hum Gene Ther, J Int Bioethique, J Vasc Nurs, Niger J Med, Oftalmologia, and Phys Med Rehabil Clin N Am), and articles were selected according to the same conditions as described above. Of the 1664 articles found in that set, 750 random articles were downloaded in EndNote. We removed three articles with a similar title, resulting in a gold standard set of 1497 unique articles.

All 1497 articles were anonymized in EndNote, leaving only titles, abstracts, and keywords. Two independent reviewers assessed whether each article was relevant to the topic of palliative care. Any discrepancies were solved first by discussion with the original two reviewers, but if no consensus was reached, a third reviewer was consulted to solve the conflict. For each article, a verdict was reached: positive if an article was relevant to the topic of palliative care or negative if that was not the case.

Creating a new search filter

For the actual filter development, a method was used, which has been used in another filter development project before.⁷ The gold standard set was split into three subsets: a term identification set (20% of all positive articles and 20% of all negative articles), a filter development set (40%), and a filter validation set (40%). Using PubReMiner,⁸ the most frequently occurring terms (MeSH terms, and words in title or abstract) were identified in both the positive and the negative articles of the term identification set. Each of those terms was combined with different field codes. It was tested how many of the positive and negative articles were retrieved by that term. The text analysis software AntConc⁹ was used to combine the most frequently present words in the positive set into phrases, and the frequencies of those phrases were also added to an Excel file. In Excel, χ² was calculated for all terms to determine the significance of the difference in relative frequencies of the terms between the positive and the negative articles. Terms were considered potential filter terms if they were present in more than 5% of all positives, were more common in the positives than in the negatives, and if the difference was statistically significant (p < 0.05).

All candidate terms were scrutinized by experts in palliative care (J.A.C.R.; E.C.T.G.; A.v.d.H.; W.H.O.), judging whether a search term was highly or potentially relevant to the topic. The highest scoring search terms were combined into a specific filter by W.M.B., and all potentially relevant search terms were combined into a sensitive filter, which was optimized using the filter development set. Manually, the filters were cleaned, removing redundant search terms which were covered by another more broad term if this did not result in the loss of a relevant article.

The filters that were thus created were tested by W.M.B. on the validation set. Specificity was defined as the percentage of negative articles not retrieved in the total number of negative articles. Sensitivity was defined as the percentage of positive articles retrieved compared to the total number of positive articles. Precision was defined as the percentage of relevant articles in the total number of articles retrieved. This could not be directly calculated from the gold standard set, because this set contained more relevant articles than would be the case in a random set of articles from PubMed. Therefore, the precision estimate using the gold standard set would be too optimistic. To assess the precision of the filters, we, therefore, searched PubMed with the filters, downloading the first 100 articles for each filter. Two reviewers scored the results on relevance.

Finally, we compared the performance of the filters with the performance of two other available palliative care search filters that we also tested on our gold standard.^4,5 Our specific filter was compared to the earlier published Master search, and our sensitive filter was compared to their adapted master search.

After creating the filter in PubMed, we translated the filters for use in other databases and interfaces (Ovid MEDLINE, Embase, and PsychINFO), embase.com, Cochrane library, and EBSCOhost (CINAHL).

Results

The gold standard set included 630 positive and 867 negative articles. The 332 candidate terms to be included in a search filter consisted of 44 MeSH terms, 238 text words, and 50 text phrases with various field combinations. Of these candidate terms, 21 were deemed relevant by all reviewers (J.A.C.R.; E.C.T.G.; A.v.d.H.; W.H.O.), and 10 relevant or potentially relevant. The filters that were developed are presented in Table 1.

Table 1.

Palliative care search filters for different databases and interfaces.

	Specific filter	Sensitive filter
PubMed	(“Terminal Care”(mh) OR bereave* OR hospice*(tw) OR “advanced cancer”(tiab) OR “end of life” OR “terminally ill”(tw) OR palliative*(tiab) OR “Palliative Care”(mh))	(“Terminal Care”(mh) OR caregiver*(tw) OR bereave* OR inpatient(tiab) OR “attitude to death”(tw) OR “end of life” OR hospice* OR “terminally ill”(tw) OR palliative*(tw) OR “Advance Care” OR palliat OR advanced OR (morphine AND cancer) OR “cancer pain”)
Ovid MEDLINE/Embase/PsychINFO	(exp Terminal Care/ OR bereave$.af. OR hospice$.mp. OR advanced cancer.tw. OR end of life.af. OR terminally ill.mp. OR palliative$.tw. OR Palliative Care/ OR exp palliative therapy/)	(exp Terminal Care/ OR caregiver$.mp. OR bereave$ OR inpatient.tw. OR attitude to death.mp. OR end of life.af. OR hospice$ OR terminally ill.mp. OR palliative$.mp. OR Advance Care.af. OR palliat.af. OR advanced.af. OR (morphine.af. AND cancer.af.) OR cancer pain.af.)
Embase.com	(“Terminal Care”/exp OR bereave* OR hospice*:de,it,lnk,ab,ti OR “advanced cancer”:ab,ti OR “end of life” OR “terminally ill”:de,it,lnk,ab,ti OR palliative*:ab,ti OR “palliative therapy”/exp)	(“Terminal Care”/exp OR caregiver*:de,it,lnk,ab,ti OR bereave*: de,it,lnk,ab,ti OR inpatient: ab,ti OR “attitude to death”:de,it,lnk,ab,ti OR “end of life” OR hospice*:de,it,lnk,ab,ti OR “terminally ill”:de,it,lnk,ab,ti OR palliative*:de,it,lnk,ab,ti OR “Advance Care” OR palliat OR advanced OR (morphine AND cancer) OR “cancer pain”)
Cochrane Library	((mh “Terminal Care”) OR bereave* OR hospice*:ti,ab,kw,pt OR “advanced cancer”:ab,ti OR “end of life” OR “terminally ill”:ti,ab,kw,pt OR palliative*:ab,ti OR (mh “palliative care”))	((mh “Terminal Care”) OR caregiver*:ti,ab,kw,pt OR bereave*:ti,ab,kw,pt OR inpatient: ab,ti OR “attitude to death”:ti,ab,kw,pt OR “end of life” OR hospice*:ti,ab,kw,pt OR “terminally ill”:ti,ab,kw,pt OR palliative*:ti,ab,kw,pt OR “Advance Care” OR palliat OR advanced OR (morphine AND cancer) OR “cancer pain”)
CINAHL EBSCOhost	(mh Terminal Care+ OR bereave* OR hospice* OR “advanced cancer” OR “end of life” OR “terminally ill” OR palliative* OR mh Palliative Care OR mh palliative therapy+)	(mh Terminal Care+ OR caregiver* OR bereave* OR inpatient OR “attitude to death” OR “end of life” OR hospice* OR “terminally ill” OR palliative* OR “Advance Care” OR palliat OR advanced OR (morphine AND cancer) OR “cancer pain”)

For easier copying and pasting of the search strings, the filters (along with field codes) are provided as plain text in Supplemental Table 1.

Table 2 shows that the specific filter had a specificity of 97.4%, with a sensitivity of 93.7% and precision of 45%. For the sensitive filter, the sensitivity was 99.6%, with a specificity of 92.5% and precision of 5%. Both the specific and the sensitive filters performed better than the previously published palliative care search,^2,3 as they retrieved more than twice as many articles than our filters with similar precision and less optimal sensitivity and specificity.

Table 2.

Performance of our filter and comparison with other filters in the field of palliative care.

	Specific filter this study	Sensitive filter this study	Master search (Sladek et al.4)	Adapted master search (Sladek et al.5)
No. of hits PubMed	130,000	680,000	271,000	2,047,000
Specificity	97.4%	92.5%	97.4%	80.5%
Sensitivity	93.7%	99.6%	90.9%	95.2%
Precision	45%	5%	45%	5%

Discussion

Main findings

We have created search filters for articles related to palliative care that perform well enough to find questions to clinical questions when time is limited (45% of the articles retrieved by the specific filter are relevant, while 93.7% of relevant articles are retrieved) and for systematic reviews (the sensitive filter retrieves 99.6% of all relevant articles, with a precision of 5%). We found that, when we searched with the filter alone in PubMed, the filters by Sladek et al.^4,5 retrieved more than twice as many results as our filters, with an equal percentage of relevant articles. Within our gold standard validation set, the filters had an equal or higher specificity and higher sensitivity. Hence, our filters significantly reduce the number of articles that researchers and physicians need to read, without having to fear they miss important articles.

The sensitive filter contains some terms that might be considered too general in a search. For instance, the term “inpatient” might cause too much noise. In our gold standard, this resulted in the retrieval of some relevant papers that might otherwise have been missed, but it is up to the end-users to decide which terms they ultimately want to include in their searches.

What this study adds

We hope that our search filters can provide some standardization in the broad area of expertise of palliative care, and support researchers and clinicians in efficiently and effectively supporting and providing evidence-based care.