Failure or success of electronic search strategies to identify adverse effects data

INTRODUCTION

One of the key difficulties in conducting specific searches for adverse effects stems from the absence of adverse effects terms in the title, abstract, and indexing of relevant articles. In 2001, Derry et al. evaluated articles that reported adverse effects data from clinical trials and found that about 23% of such articles had no adverse effects terms (either generic or specific) in the title, abstract, or indexing of records in MEDLINE or EMBASE [1]. Hence, electronic searches based on specific adverse effect–related terms could miss nearly a quarter of relevant papers. Current guidance on systematic reviews of adverse effects emphasizes the use of nonspecific searches (without relying on adverse effects terms) as well as the need to check full-text versions of retrieved articles to ensure a complete search [2]. Unfortunately, implementation of such guidance is onerous and time consuming compared to running more specific adverse effects–based searches.

However, methodological developments in the past decade might have changed the situation since Derry et al.'s study [1]. In 2003, ten new recommendations about the reporting of harms were added to the Consolidated Standards of Reporting Trials (CONSORT), including a recommendation that if the study collected data on harms, the title or abstract should state this [3]. Additionally, the Cochrane Adverse Effects Methods Group <http://www.aemg.cochrane.org>, which was formed in 2007, as well as many other authors, called for improved reporting of adverse effects [4–32]. However, it is unclear if these developments have had a meaningful impact on the prevalence of adverse effects terms in the title, abstract, or indexing of relevant adverse effects papers.

This study collected a large, diverse, and contemporaneous cohort of articles with adverse effects data so that the presence or absence of specific adverse effects terms could be assessed [1].

METHODS

Selection of cohort of studies

The first step involved collecting a cohort of papers that were known to report data on the frequency of adverse effects but that had not been retrieved using specific adverse effects search terms.

Identification of systematic reviews of adverse effects through hand searching of the Database of Abstracts of Reviews of Effects

The primary studies were identified from papers included in systematic reviews of adverse effects that were indexed as full abstracts in the Database of Abstracts of Reviews of Effects (DARE), one of the databases available from the Centre for Reviews and Dissemination website <http://www.crd.york.ac.uk/crdweb/>. A systematic review was considered eligible for inclusion if it was published between 2007 and 2010 and:

1. Adverse drug effects were the primary outcome.

2. Specific adverse effects search terms or specified/named adverse effects had not been used by the review authors to locate articles for the review. This created a cohort in which relevant articles had not already been chosen because of presence of specific adverse effects terms.

3. The search included either hand searching or reference checking in addition to database searches (this is consistent with the study by Derry et al. 2001 [1]).

Selection of relevant primary studies with adverse effects data

The included references in each systematic review were checked for papers published in English after the year 2000, so that a contemporaneous study cohort could be obtained. Non-English-language papers were excluded because of the difficulty of obtaining valid matches for adverse effects terms in different languages. Full-text articles were checked to confirm the presence of adverse effects data that had been used in the systematic review. The papers were then de-duplicated to remove copies of papers that had been included in more than one systematic review.

Analysis

Availability in selected databases

The first stage of the analysis checked whether each paper was listed and indexed in MEDLINE, EMBASE, or Science Citation Index (SCI). MEDLINE and EMBASE are the most commonly searched databases in systematic reviews of adverse effects [33, 34] and were used in the study by Derry et al. [1]. SCI is also commonly used [33, 34], and a systematic review of thiazodinediones-related fractures (by the authors but not yet published) suggested that this database contained more relevant references than other databases, including MEDLINE and EMBASE.

Adverse effects terms in the database records

For each database, the available papers were sorted according to the following criteria:

1. The authors mentioned terms synonymous with “adverse effects” in the title or abstract (thus potentially enabling the paper to be found in an electronic search). Examples of synonyms are “adverse events,” “side effects,” “tolerated,” and “unwanted effects.” This is in line with terms accepted by Derry et al. 2001 [1].

2. The authors mentioned terms describing specific relevant adverse effects (such as “headache” or “cancer”) in the title or abstract. The terms were considered relevant based on the adverse effects included in the systematic review. For example, for a systematic review on cancer as an adverse effect, only cancer-related terms were considered relevant. This part of the analysis was only conducted on included studies from reviews for a specific, named adverse effect.

3. The papers had been indexed in MEDLINE or EMBASE (using subject headings or subheadings) or allocated keywords in SCI with relevant terms for concepts related to adverse effects (thus potentially enabling the paper to be found in an electronic search). Relevant indexing terms were those that could be considered synonymous with “adverse effects” and would have been accepted by Derry et al. 2001 [1]. Examples of included indexing terms were “drug toxicity/” and “side effects/” and examples of included subheadings were “adverse effects (ae)” or “adverse drug reaction (ae).”

4. The papers had been indexed in MEDLINE or EMBASE or allocated keywords in SCI terms describing a specific adverse effect or adverse effects. This part of the analysis was only conducted on included studies from reviews where the focus was on specific, named adverse effects, and only indexing terms relating to these particular adverse effects qualified. For example, in a systematic review that assessed risk of cancer arising as an adverse drug reaction, only cancer-related indexing terms qualified.

RESULTS

Systematic reviews

Twenty-six systematic reviews met the inclusion criteria [35–60]. Half were concerned with the safety profile for an intervention [35, 36, 41, 44–49, 52–54, 57], whereas the other half were limited to a named, specific adverse effect with an intervention [37–40, 42, 43, 50, 51, 55, 56, 58–60].

Primary studies

Two hundred forty-two papers were eligible for use in the analysis. The majority of the papers were randomized controlled trials (RCTs) (89%, 216/242); however, there were also 13 case series, 5 chart reviews, 3 case reports, 3 cohort studies, 1 non-RCT study, and 1 uncontrolled study. Two hundred thirty-one of the 242 references were indexed in Ovid MEDLINE, 222 in EMBASE, and 238 in SCI.

Adverse effects terms

Searching with generic or specific, named adverse effects terms anywhere in the title, abstract, or indexing would have identified 89% of all relevant references in EMBASE, 80% in MEDLINE, and 70% in SCI (Table 1). Generic adverse effects terms in the title and abstract in any of the 3 databases, generic and specific indexing terms or subheadings for adverse effects in EMBASE, and adverse effects subheadings in MEDLINE retrieved the highest proportion of records.

Table 1.

Number of articles with adverse effects terms in the title, abstract, or indexing of electronic records in MEDLINE, EMBASE, or Science Citation Index (n = 242)

Comparison with Derry et al. 2001

Overall, a combined search using terms in the title, abstract, or indexing in both MEDLINE and EMBASE would have failed to retrieve 19 (all of which were RCTs) of the 233 papers (8%) across all 26 systematic reviews. This is much lower than the 23% of papers that would have been missed with a similar search approach in Derry et al. 2001 [1].

DISCUSSION

It is reassuring to note that the prevalence of adverse effects terms in the title, abstract, or indexing (in MEDLINE and EMBASE for articles that are known to contain adverse effects data) has increased compared to previous findings [1]. In the past, specific searches for studies that contained adverse effects data have been hindered by the frequent absence of adverse effects terms. However, this study's findings indicate that reviewers can cautiously choose to use more focused search filters or specific adverse effects terms, rather than face the arduous task of broad nonspecific searches, followed by evaluation of full-text articles.

The frequency of specific adverse effects indexing terms and generic adverse effects indexing terms was much higher in EMBASE compared to MEDLINE or SCI and reflects the general practice in EMBASE of assigning more indexing terms to records. There was a lack of relevant keywords for adverse effects in the records from SCI. This is not surprising given that these terms are assigned either by the author when the paper is submitted for publication or from words or phrases that frequently appear in the titles of the references cited by the paper. This means that the adverse effect would need to be a key aspect of the paper or in the titles of references for that paper to appear as keywords. In addition, not all records in SCI contain keywords. For example, meeting abstracts tend not to include any keywords as the authors of abstracts generally do not have to provide keywords and abstracts generally do not include references. MEDLINE and EMBASE, on the other hand, have controlled vocabularies, and indexing is a manual process based on the full text of the article.

CONCLUSIONS

Adverse effects terms are increasingly prevalent in the title, abstract, and indexing terms of articles that contain adverse effects data in MEDLINE and EMBASE.

Source of funding

This research was undertaken by Su Golder as part of a Medical Research Council (MRC) fellowship. The views expressed in this presentation are those of the authors and not necessarily those of the MRC.