How to identify existing literature on patients' knowledge, views, and values: the development of a validated search filter

Abstract

Objective

The research sought to establish validated search filters for retrieval of studies on patients' knowledge and values.

Setting

Two nonprofit organizations in the Netherlands were studied for guideline development.

Methods

An existing filter was adapted to three bibliographic interfaces. After defining the scope, a reference database was built for the development of the new filters. The performance of the new filters was validated in different disease categories.

Results

Sensitivity, specificity, precision, negative predictive value, and accuracy were (%): 90.5/98/77/99.2/97 (MEDLINE-Ovid), 90.1/98.8/79.3/99.5/98.3 (PubMed), and 93.1/98.4/81.8/99.6/98.7 (Embase).

Conclusions

The filters provide pragmatic tools for searching for patients' issues. Further optimization and validation is recommended.

An evidence-based clinical practice guideline is preferably based on three sources of knowledge: (1) best available evidence from clinical epidemiological studies, such as evidence on effects and side-effects of interventions; (2) expertise and experience-based knowledge of health care professionals; and (3) experiential knowledge and collective preferences of patients [1, 2]. Despite the growing awareness of the importance of patient and public involvement in guideline development, systematic and explicit consideration of empirical data and research evidence on collective patients' experiences, preferences, and values in clinical practice guideline development seem limited [3].

Currently, the emphasis in patient and public involvement appears to lie more on physical presence of patients or patient representatives in guideline working groups or in focus groups in the exploratory phase of the guideline development process. Failure to consider already existing empirical data and research evidence on experiences, preferences, and values of groups of patients may be due to the lack of understanding of or agreement on the conceptualization and measurement of these concepts.

There may also be a practical barrier in how to identify the existing research evidence on patient perspectives. Consulting with the most important patient groups and other stakeholders could yield important reports or publications on experiential knowledge and collective preferences. This method might, however, render incomplete or biased information. Searching relevant databases will retrieve studies on a more systematic and objective basis.

Our objective was to facilitate the retrieval of possibly relevant studies on patient participation and patient perspectives, in the context of guideline development, by developing a standardized set of search terms, a so-called “search filter.” To support the use of the search filter in different settings, the authors adapted the filter to the specific syntax of the three most commonly used medical databases or interfaces: MEDLINE with the PubMed and the Ovid interfaces, and Embase.

An existing search filter on patient perspectives, the Scottish Intercollegiate Guidelines Network (SIGN) turned out to have a high sensitivity and a rather low specificity (Appendix A, online only). As a result, the SIGN search filter yielded too much “noise” to be practical in our process of guideline development. Nevertheless, the SIGN definition of the scope of patient issues in guidelines appeared to be a useful starting point for developing a more specific filter. The central concepts that they defined were: experiences (both with living with a particular condition or health problem or with health care in general), information needs, fulfilled and unfulfilled needs, preferences, participation in decision making, and satisfaction with received care.

METHODS

Construction of the filters

The initial search terms for the filters were derived from the SIGN filter for patient issues, from the book User Involvement in Health Care [4], and from an analysis of Medical Subject Headings (MeSH) terms and free text words of known relevant articles.

Building a database consisting of a set of relevant records that served as a gold standard proved quite a challenge. First, we looked for references on patient issues in guidelines that SIGN developed, our assumption being that these would be references that were proved relevant for patient issues in the context of guideline development, which indeed turned out to be the case. Because references were scarce, they were supplemented with references in a recent article on patient participation [5], references in a recent guideline on diabetes self-management education [6], and references derived from a Cochrane review on methods of consumer involvement in developing health care policy and research, clinical practice guidelines, and patient information material [7]. Eventually, this resulted in a database consisting of 176 relevant references. These were the “gold standard” that an adequate filter should find.

The MeSH terms attached to all references in this database were analyzed with two open source applications [8, 9]. The references that were not retrieved by the filter were analyzed manually for potentially relevant missing MeSH terms and additional free-text concepts.

Based on both analyses, the filter was adjusted so that, finally, all the references of the gold standard set of references were retrieved (Appendix B, online only).

Validation of the filters

The filters were validated by testing them in practice for three subjects in primary care (the PubMed filter), three subjects in secondary care (the Ovid filter), and two subjects in primary or secondary care (the Embase filter) (Figure 1). Every subject, with the exception of cardiac pacemakers and implantable defibrillators in terminal care, was combined with the generic search filter and limited by publication date to get a reasonable number of included and excluded references. The subjects were chosen arbitrarily; they emerged in the guideline development process at that moment in our organizations (Dutch College of General Practitioners and Knowledge Institute).

Figure 1.

Disease categories

For each subject, all the references that were retrieved with the filters were assessed by different independent reviewers (recruited from the same organizations as the developers of the search filter) to determine whether they were relevant to the subject of patient preferences and values, and the defined scope of the filter. The same reviewers also assessed whether the references that were not retrieved by the filter were rightfully excluded. Finally, we discussed the results with each of the reviewers.

For each subject, we determined the true positives, the false positives, the true negatives, and the false negatives, from which we computed the sensitivity, specificity, precision, and accuracy using open source software [10] (Figure 2).

Figure 2.

General 2x2 table

Precision=a/(a+b), sensitivity=a/(a+c), specificity=d/(b+d), accuracy=(a+d)/(a+b+c+d).

RESULTS

As expected, the sensitivity of the original SIGN filter is very high (more than 98%), but the precision, that is the positive predictive value, is quite low (20%), meaning that of every 100 articles 80 will be not relevant for the subject of patient issues (Table 1). For search filters identifying randomized controlled trials and systematic reviews, a positive predictive value of 20% seems very reasonable [11], but for our purpose (i.e., identifying patient issues for guideline development), this was considered suboptimal. Our new filters appear to be more precise (around 80%), which means that of every 100 articles, only 20 will not be relevant, depending on the “prevalence” of articles on patient issues on the subject at hand. When articles on patient issues regarding a specific subject are scarce, the number of false positives tends to increase. The trade-off of the higher precision is the lower sensitivity of 90%.

Table 1.

Results of validation

DISCUSSION

The search filter showed a satisfactory precision, which could make it an efficient tool in the guideline development process, even when budgets are limited. The trade-off for higher specificity is that the sensitivity is lower compared to the original SIGN filter, but compared to other specific filters, the performance is above average [11]. Of course, other specific filters on patient values and perspectives might serve the same purpose.

Limitations

Our study has some limitations. The selection of the studies from the hits found in PubMed was done by only one independent reviewer per subject. The reviewer might have misinterpreted articles that should have been included. We believe that this potential for bias is very small because each reviewer was an expert on the subject at hand. The robustness of the set of known relevant records as a gold standard might be challenged, because this type of evidence is difficult to find. It can, for instance, be hidden in HTA publications, such as discrete choice experiments, and in the gray literature. Therefore, the building of this database should be considered an ongoing process. The fact that a lot of relevant literature cannot be found in the regular medical bibliographic databases implies that checking the references from relevant papers and asking experts in the field should be a necessary addition to the literature searching process.

Currently, there still is no consensus on what the central concepts regarding patient perspectives and values exactly are, let alone a framework or taxonomy [12]. We expect that the search filter will evolve when this language becomes clearer in the future. Gaining consensus on the definition of patients' perspectives is ongoing work that will hopefully be reflected in the terminology of the search filters in the years to come.

We believe that the new filters are sufficiently validated and pragmatic to use when developing guidelines. One just has to combine the terms for a complaint, illness, condition, or key question with one of the filters to retrieve the relevant articles on the patients' experiential knowledge, views, and values regarding the guideline topic. Nevertheless, we recommend further optimization of the filters in subsequent validation studies.

Practice implications

The results of the literature searches based on specific subjects combined with the patient filters can be used:

• ▪when initially exploring the guideline,
• ▪for setting the agenda for relevant problems and issues that should be covered with the guideline,
• ▪when formulating and prioritizing key questions,
• ▪when defining and prioritizing outcomes on desirable effects and on undesirable side-effects,
• ▪when describing the burden of treatment and other health care or self-management interventions,
• ▪when formulating recommendations,
• ▪when developing input for focus group discussions,
• ▪when developing accompanying patient education materials, and
• ▪for obtaining initial content as input for patient wiki [13].

Furthermore, the results can be used by a patient involvement officer, clinicians, patient representatives, epidemiologists, the guideline development group, and outside the field of guideline development as a pragmatic tool to identify patient issues on specific topics, for instance, when independent materials or interventions for patient education, counseling, or empowerment are being prepared.

Biography

Monique Wessels, MSc (corresponding author), m.wessels@kennisinstituut.nl, Knowledge Institute of Medical Specialists, Mercatorlaan 1200, Utrecht, The Netherlands; Lian Hielkema, BA, l.hielkema@nhg.org, Dutch College of General Practitioners (NHG), Utrecht, The Netherlands; Trudy van der Weijden, MD, PhD, trudy.vanderweijden@maastrichtuniversity.nl, Department of Family Medicine, School for Public Health and Primary Care (CAPHRI), Maastricht University, Maastricht, The Netherlands