Refining Boolean queries to identify relevant studies for systematic review updates

Abstract

Objective

Systematic reviews are important in health care but are expensive to produce and maintain. The authors explore the use of automated transformations of Boolean queries to improve the identification of relevant studies for updates to systematic reviews.

Materials and Methods

A set of query transformations, including operator substitution, query expansion, and query reduction, were used to iteratively modify the Boolean query used for the original systematic review. The most effective transformation at each stage is identified using information about the studies included and excluded from the original review. A dataset consisting of 22 systematic reviews was used for evaluation. Updated queries were evaluated using the included and excluded studies from the updated version of the review. Recall and precision were used as evaluation measures.

Results

The updated queries were more effective than the ones used for the original review, in terms of both precision and recall. The overall number of documents retrieved was reduced by more than half, while the number of relevant documents found increased by 10.3%.

Conclusions

Identification of relevant studies for updates to systematic reviews can be carried out more effectively by using information about the included and excluded studies from the original review to produce improved Boolean queries. These updated queries reduce the overall number of documents retrieved while also increasing the number of relevant documents identified, thereby representing a considerable reduction in effort required by systematic reviewers.

INTRODUCTION

Systematic reviews identify, assess, and synthesize the evidence available to answer complex research questions. They are essential in health care where the volume of evidence in scientific research publications is vast and cannot feasibly be identified or analyzed by individual clinicians or decision makers. However, the process of creating a systematic review is time-consuming and expensive, with a single review often requiring up to a year’s effort by expert reviewers^1,2 and costing up to quarter of a million U.S. dollars.³ The pace of scientific publication in medicine and related fields also means that evidence bases are constantly changing and review conclusions can quickly become out of date.⁴ In fact, it has been estimated that 7% of systematic reviews in the medical field are already out of date by the time of publication, with almost a quarter (23%) 2 years after they have appeared.⁵ Reliance on review conclusions based on out-of-date evidence increases the risk of recommendations for practice that are optimal and potentially harmful to patients. With over 8000 reviews being produced per year,⁶ keeping them up to date represents a formidable challenge. There is therefore a need to develop methods to support the update process of systematic reviews to reduce workload required from researchers and ensure the reviews are consistent with current evidence.⁷

The problem of identifying relevant evidence is a significant part of the effort required to produce and update reviews.² Evidence is typically identified via a 2-stage process. First, a search is carried out over databases of scientific publications to identify any studies of potential relevance to the question being addressed in the review. In the second phase, the studies are screened, normally by 2 reviewers, and relevance assessment judgments are applied to each document within the retrieved set.

The majority of studies included in a systematic review are normally identified by carrying out searches over databases of scientific publications. These searches generally rely on complex Boolean queries written by domain experts. The need to ensure that relevant evidence is not missed means that the searches have been developed to optimize recall rather than precision (which is typically as low as 1%-2%). Queries often return large numbers of studies and require significant effort during the screening stage to identify the studies that should be included in the review.

The Boolean queries used for systematic reviews are usually formatted using OVID or PubMed search syntax and employ advanced operators in addition to the standard logical operators (AND, OR, and NOT). Figure 1 shows an example Boolean query for review CD005025, “Reminder packaging for improving adherence to self-administered long-term medications.”⁸ This OVID format query consists of 52 lines (clauses). Lines are numbered so they can be referenced. For example, line 15 combines the results of all previous lines.

Figure 1.

Example portion of Boolean query for review CD005025.⁸

The process of evidence identification is similar for new reviews and review updates, although for updates it is generally only carried out over studies that have appeared since the search for the original review was conducted. This study proposes an approach to improving the identification of relevant studies for a systematic review update by adapting the Boolean query using information produced during the screening stage of the original review. An iterative algorithm is proposed to generate query variants by applying a set of transformations. These are assessed using information about which studies were included in the original review and the most effective chosen to update the query.

There has been significant interest in the development of techniques to support the identification of studies for inclusion in systematic reviews by applying text mining techniques, for example.^9–16 The vast majority of this work has focused on the identification of studies for new reviews, and only a few articles explored the problem of identifying relevant studies for review updates.^9,17,18 Most of those studies evaluated their approaches using simulations of the update process, that is, by choosing a cutoff point (eg, 3 years before the review was published) and assuming that all studies published before this point were included in the original review while those published afterwards were added during an update.¹⁸ An exception is work that used update information for 9 drug therapy systematic reviews.¹⁹

Scells and Zuccon²⁰ introduced an approach to improving the Boolean queries used for study identification in systematic reviews. The query used in the review was iteratively altered by applying a set of transformations such as replacing logical operators and field restrictions. They found that the modified queries generated by this approach improved on those used in the original review. The best modified queries were identified using classifiers and learning to rank methods. Their approaches produced queries with higher precision and F-measure scores than the original query but not improved recall (which was not their goal). Their method was used to demonstrate that it was possible to improve the Boolean query used for the original review, rather than to develop queries for review updates.

Previous work on the refinement and generation of Boolean queries for other types of professional search, such as prior art search, is also relevant to the work described here. Kim et al²¹ proposed a Boolean query suggestion technique in which a decision tree was learned from pseudo-relevant documents then used to generate queries. Graf et al²² developed a method for automatically generating queries for prior art search by analyzing the distribution of terms among topic-relevant documents. Harris et al²³ presented an interactive Boolean search system that helps the user to create a Boolean search query. The interactive system suggests semantically similar search terms to the user.

MATERIALS AND METHODS

The aim of the work proposed in this article is to improve the performance of Boolean queries used to retrieve relevant studies for updates of systematic reviews. The approach is based on the query transformation technique described by Scells and Zuccon²⁰ and extended in 2 important ways. First, it is applied to the problem of generating queries for review updates and makes use of information about which studies were included and excluded from the original review to guide the query modification. Second, it extends the set of query transformations introduced by Scells and Zuccon and demonstrates that the new transformation lead to the generation of more effective queries.

The proposed approach starts with the Boolean query used for the original review. A set of transformed queries is generated by applying a range of transformations (eg, operator substitution, query expansion, and query reduction) to the original query. Each transformed query is then evaluated using the relevance judgments produced for the original review and the best transformation selected. The process is then repeated by applying the transformations to the newly selected query and evaluating the transformed queries produced. The process continues until the best transformed query is no better than the query from the previous iteration (ie, the query cannot be improved using this process). The approach is outlined in algorithm 1.

In this approach, relevance judgments from the original review (ie, information about included and excluded studies) are used to select the transformed query at each iteration. This information is readily available because the studies returned by the Boolean query are manually screened and include and exclude decisions reported. However, because our aim is to develop improved queries that can be used to support review updating, evaluation is carried out using the studies that have been identified for the review update (see Experiments), and the approach does not have access to this information.

The individual steps of the approach are now described in further detail.

Step 1: Boolean query transformation

The first step of the algorithm applies a set of query transformations to generate new queries from the current one. Three types of transformation are applied.

1. Operator Substitution. This transformation replaces one query operator with another. For example, disjunction with conjunction [eg, (blind$ or mask$).ti. → (blind$ and mask$).ti.] or altering a restriction field [eg, (blind$ or mask$).ti. → (blind$ or mask$).ti,ab.]. (In the second example .ti,ab. indicates that both the title and abstract are searched for the terms rather than just the title.)

   A set of useful operator substitution transformations were identified during preliminary experiments: .tw.→.ti., .tw.→.ti,ab., .ti,ab.→.ti., .ti,ab.→.tw., .ti.→.tw., .ti.→.ti,ab., .ab.→.ti,ab., .ab.→.ti., .sh.→exp *, and→or and or→and. Some of these transformations were used in previous work²⁰: logical operator replacement (and→or and or→and) and 4 field restrictions (.ti,ab.→.ti., .ti.→.ti,ab., .ab.→.ti,ab. and .ab.→.ti.). The remaining transformations were developed for this study. Additional transformation types were also explored but not found to improve performance, including 3 field restriction transformations: .af.→.ti,ab., .af.→ .ti. and .af.→ .tw.

2. Query Expansion. This transformation adds new elements to the query and has not been applied in previous work on query transformation. Terms that tend to occur in included studies are identified using the log-likelihood statistic and added to the query.²⁴ The set of studies retrieved for the original review is partitioned into relevant and irrelevant subsets based on the relevance judgments of the original review and the score for each term computed as:

   $$\mathit{Log}\mathit{Liklehood}=2\mathrm{ }\times \mathrm{ }\left(O_{\mathit{rel}}\times \log \frac{O_{\mathit{rel}}}{E_{\mathit{rel}}}+O_{\mathit{irrel}}\times \log \frac{O_{\mathit{irrel}}}{E_{\mathit{irrel}}}\mathrm{ }\right)$$ (1)

   where $O_{\mathit{rel}}$ and $O_{\mathit{irrel}}$ are the observed frequency of the term in different subsets of the collection (eg, relevant and irrelevant studies), respectively. $E_{\mathit{rel}}$ and $E_{\mathit{irrel}}$are the term’s expected frequencies, respectively, calculated as:

   $$E_{\mathit{rel}}=\mathrm{ }{N}_{\mathit{rel}}\mathrm{ }\times \mathrm{ }\frac{O_{\mathit{rel}}\mathrm{ }+\mathrm{ }{O}_{\mathit{irrel}}\mathrm{ }}{N_{\mathit{rel}}+\mathrm{ }{N}_{\mathit{irrel}}},\mathrm{ }{E}_{\mathit{irrel}}=\mathrm{ }{N}_{\mathit{irrel}}\mathrm{ }\times \mathrm{ }\frac{O_{\mathit{rel}}\mathrm{ }+\mathrm{ }{O}_{\mathit{irrel}}\mathrm{ }}{N_{\mathit{rel}}+\mathrm{ }{N}_{\mathit{irrel}}}$$ (2)

   where $N_{\mathit{rel}}\mathrm{ }$and $N_{\mathit{irrel}}\mathrm{ }$represent the size of the corpus (ie, relevant and irrelevant studies). Terms are assigned high log-likelihood scores when their observed frequency is (much) higher than the expected frequency within a corpus. Log-likelihood scores are used to identify the 5 terms that are most closely associated with the relevant studies. These terms are then used to form a set of transformations in which the terms are added to a query clause using the logical or operator and .tw. as the restriction field (The full list of transformations that can be added to each clause is add 1^st term, add 2^nd term, add 3^rd term, add 4^th term, add 1^st and 2^nd terms, add 1^st to 3^rd terms, add 1^st to 4^th terms and add all 5 terms.).For example, the terms packaging, blister, pack, calendar, and medication are identified as the top 5 terms that identify relevant studies for the review CD005025.⁸ Transformations include adding the first term alone (Reminder Systems/→ Reminder Systems/ or packaging.tw.), adding the second term alone (Reminder Systems/→ Reminder Systems/ or blister.tw.), adding the top 2 terms (Reminder Systems/→ Reminder Systems/ or packaging.tw. or blister.tw.), adding the top 3 terms (Reminder Systems/→ Reminder Systems/ or packaging.tw. or blister.tw. or pack.tw.), and so on.

3. Query Reduction: The final transformation method simply deletes a clause from the Boolean query. For example, removal of the second clause

4. Reminder Systems/        1. Reminder Systems/

5. exp Patient Compliance/  →  2. Treatment Refusal/

6. Treatment Refusal/

The transformed queries produced during each iteration differ from the query selected during the previous iteration by a single clause. A total of 21 transformation types are used, leading to up to 21 × c transformed queries being produced during each iteration (where c is the number of clauses in the query selected during the previous iteration). However, this is an upper bound value because not all transformations types applicable to all clauses. For example, the operator substitution .tw.→.ti,ab. cannot be applied to a clause that does not contain the .tw. restriction field.

Step 2: Boolean query selection

The set of transformed queries generated during step 1 are evaluated by assessing them against the relevance judgments produced for the original review. Each transformed query is run against Medline and the list of studies it retrieves is returned. The query is then assessed using the following objective function which favors improvements in recall over improvements in precision:

$$f\left(\widehat{q}\right)\mathrm{ }=\mathrm{ }\mathit{recall}\left(\widehat{q}|R_{\mathit{orig}}\right)\mathrm{ }\times \mathrm{ }100\mathrm{ }+\mathrm{ }\mathit{precision}\left(\widehat{q}\right|R_{\mathit{orig}})$$ (3)

where $\widehat{q}$ is the transformed query and R_orig the relevance judgments from the original review. Recall and precision are calculated as follows:

$$\mathit{recall}\left(\widehat{q}|R_{\mathit{orig}}\right)\mathrm{ }=\mathrm{ }\frac{\mathit{Number}\mathrm{ }of\mathrm{ }\mathit{relevant}\mathrm{ }\mathit{studies}\mathrm{ }in\mathrm{ }{R}_{\mathit{orig}}\mathrm{ }\mathit{retrived}\mathrm{ }by\mathrm{ }\widehat{q}}{\mathit{Total}\mathrm{ }\mathit{number}\mathrm{ }of\mathrm{ }\mathit{relevant}\mathrm{ }\mathit{studies}\mathrm{ }in\mathrm{ }{R}_{\mathit{orig}}}$$ (4)

 

$$\mathit{precision}\left(\widehat{q}|R_{\mathit{orig}}\right)\mathrm{ }=\mathrm{ }\frac{\mathit{Number}\mathrm{ }of\mathrm{ }\mathit{relevant}\mathrm{ }\mathit{studies}\mathrm{ }in\mathrm{ }{R}_{\mathit{orig}}\mathrm{ }\mathit{retrived}\mathrm{ }by\mathrm{ }\widehat{q}}{\mathit{Total}\mathrm{ }\mathit{number}\mathrm{ }o\mathrm{f}\mathrm{ }\mathit{studies}\mathrm{ }\mathit{retrived}\mathrm{ }by\mathrm{ }\widehat{q}}$$ (5)

As can be seen from the equation 3, the objective function always assigns a higher score to a query that produces an improvement in recall to one that improves precision. This is due to the nature of the search problem in systematic reviews in which high recall is important because the goal is to identify all potentially relevant studies. However, retrieving a large number of irrelevant studies increases the screening effort required by reviewers, and it is therefore beneficial to ensure that the precision of queries is as high as possible. It would be straightforward to adapt the approach proposed here to favor a different balance of recall and precision by using a different objective function.

The transformed query that produces the highest score is then chosen for the next iteration. If there are multiple queries with the same highest score, then one is chosen at random. If there is no difference between performance of the highest-scoring query and the query from the previous iteration, then the algorithm stops.

Experiments

Experiments were carried out using a publicly available dataset containing 22 reviews from the Cochrane Library of Systematic Reviews.²⁵ All reviews are intervention type, the most frequently occurring in Cochrane, which evaluate the efficacy of a healthcare intervention for a specific disease.

The dataset includes information about original and updated versions of the reviews including the review title, Boolean query (in OVID format) and set of included PMIDs. For each review, the majority of the included PMIDs are identified using the Boolean query, but additional studies are often identified using alternative techniques such as hand-searching key journals and examination of the references lists of included studies. The gold standard dataset used in this article includes all the relevant studies that are available on PubMed, regardless of whether they were identified using the Boolean query or by other methods. Note that this means that the query used for the review may not achieve full recall because it may not retrieve all studies included in the review or an update.

PMIDs included after abstract level screening were used for the experiments because the goal of this work is to develop queries that are applied to databases of scientific abstracts, such as PubMed, and only very few studies are included after content level screening for some reviews. Between 1 and 46 studies are included in the updated reviews after abstract-level screening. Further information about the dataset can be found in the supplementary material (see Supplementary Tables 1 and 2).

Approaches are evaluated using the set of studies included in the update as a gold standard list of relevant studies. It is worth noting that this information is not available to the proposed approach, which only makes use of information about the studies considered for inclusion in the original review.

Approach 1: Proposed method

The first approach that was applied is the method proposed in this article (see Materials and Methods). Transformed queries are run against Medline using the Entrez package from biopython.org to retrieve studies for the updated version of the review. Publication dates are used to identify studies published since the previous version of the review. To run the queries against Medline, the OVID format Boolean queries are converted to a single-line PubMed format query. This is carried out automatically using a Python script created specifically for this purpose.

Approach 2: Baseline

A baseline approach was implemented which used the Boolean query from the original review to retrieve studies for updated review without any transformation. The original Boolean query is run against Medline and the set of the studies that match the query retrieved. The aim of this approach is to assess performance when the query developed for the original review is re-used for the update, which is common practice within the systematic review community.

Approach 3: Oracle

An oracle approach was also implemented that is similar to the proposed method (see Materials and Methods), with the exception that performance of the transformed query is assessed using the relevance judgments for the updated review ${(R}_{\mathit{update}}),\mathrm{ }$rather than for the original (ie, using the objective function):

$$f\left(\widehat{q}\right)\mathrm{ }=\mathrm{ }\mathit{recall}\left(\widehat{q}|R_{\mathit{update}}\right)\mathrm{ }\times \mathrm{ }100\mathrm{ }+\mathrm{ }\mathit{precision}\left(\widehat{q}\right|R_{\mathit{update}})$$ (6)

The oracle approach represents an unrealistic scenario because it has access to the relevance judgments for the updated review. However, it provides context for the results of the proposed method by placing an upper bound on the results that are possible by transforming queries for review updates.

Approach 4: Restricted transformation types

The final approach is a version of the proposed method (approach 1) in which only a single transformation type is applied. Three versions of this approach were explored (one for each transformation type): (1) operator substitution, (2) query expansion, and (3) query reduction.

RESULTS

Results are shown in Table 1. Recall and precision scores are shown for each method, both for each review individually and averaged across all reviews. Averages are weighted by the number of abstracts in each review to place more weight on reviews in which there are larger numbers of abstracts to be screened. The iteration of the algorithm that produced the final query is also shown for the proposed method, oracle, and restricted transformation type approaches. This information is not included for the baseline method, which is simply the unmodified query from the original review.

Table 1.

Recall and precision results for each review in the update dataset

									Restricted Transformations
	Baseline		Proposed Method			Oracle			Operator Substitution			Query Expansion			Query Reduction
Review	Recall	Precision	Recall	Precision	Iteration	Recall	Precision	Iteration	Recall	Precision	Iteration	Recall	Precision	Iteration	Recall	Precision	Iteration
CD000155	0.3333	0.0182	0.3333	0.0206a	8	0.3333	0.0667a	11	0.3333	0.0012	13	0.3333	0.0145	3	0.3333	0.0007	4
CD000160	1.0000	0.0005	1.0000	0.0108a	15	1.0000	0.0047a	12	1.0000	0.0008a	4	1.0000	0.0003	2	1.0000	0.0034a	13
CD000523	1.0000	0.0526	1.0000	0.0909a	8	1.0000	1.0000a	5	1.0000	0.0233	8	1.0000	0.0500	2	1.0000	0.0238	8
CD001298	0.0000	0.0000	0.5882a	0.0010a	17	0.5882a	0.0154a	2	0.5882a	0.0003a	13	0.5882a	0.0006a	3	0.2353a	0.0054a	20
CD001552	1.0000	0.0021	1.0000	0.0043a	12	1.0000	0.0444a	10	0.5000	0.0152a	11	1.0000	0.0021	1	1.0000	0.0039a	11
CD002064	1.0000	0.0833	1.0000	1.0000a	9	1.0000	0.5000a	5	1.0000	0.1429a	6	1.0000	0.0909a	2	0.0000	0.0000	9
CD004069	0.8889	0.0068	0.8889	0.0068	1	1.0000a	0.0089a	5	0.8889	0.0068	1	0.8889	0.0068	1	0.8889	0.0068	1
CD004214	0.0000	0.0000	0.5000a	0.0010a	2	0.5000a	0.0010a	2	0.5000a	0.0004a	3	0.5000a	0.0010a	2	0.0000	0.0000	1
CD004241	0.6000	0.0116	0.6000	0.0022	20	0.6000	0.1765a	6	0.6000	0.0005	15	0.6000	0.0052	4	0.6000	0.0002	12
CD004479	0.7500	0.0189	0.7500	0.0013	2	0.7500	0.0211a	2	0.7500	0.0149	3	0.7500	0.0001	2	0.7500	0.0013	2
CD005025	0.4130	0.0139	0.6304a	0.0017	18	0.7391a	0.0008	12	0.3913	0.0018	9	0.5652a	0.0014	4	0.2609	0.0028	16
CD005055	0.6667	0.0033	0.6667	0.0102a	7	1.0000a	0.0083a	6	0.6667	0.0114a	5	0.6667	0.0016	2	1.0000a	0.0001	4
CD005083	0.2222	0.0160	0.5556a	0.0025	3	0.5556a	0.0403a	11	0.2222	0.0160	1	0.5556a	0.0098	2	0.5556a	0.0025	3
CD005128	0.5556	0.0007	0.5556	0.0066a	7	0.5556	0.0066a	6	0.5556	0.0036a	7	0.5556	0.0007	1	0.5556	0.0066a	7
CD005426	0.0000	0.0000	0.0000	0.0000	11	0.0000	0.0000	1	0.0000	0.0000	1	0.0000	0.0000	2	0.0000	0.0000	1
CD006839	0.6667	0.0204	0.6667	0.2000a	9	1.0000a	0.3333a	11	0.6667	0.1000a	7	0.6667	0.0204	1	0.6667	0.1818a	8
CD006902	0.5000	0.0365	0.8000a	0.0014	4	0.8000a	0.0014	4	0.4000	0.0019	9	0.8000a	0.0014	2	0.6000a	0.0074	6
CD007020	0.2500	0.0156	0.2500	0.0192a	3	0.2500	0.0233a	4	0.2500	0.0083	3	0.2500	0.0147	2	0.2500	0.0052	2
CD007428	0.6667	0.0270	0.5556	0.0568a	11	0.7778a	0.0538a	7	0.6667	0.0435a	5	0.6667	0.0221	2	0.7778a	0.0104	4
CD008392	1.0000	0.0014	1.0000	0.0135a	10	1.0000	0.0159a	12	1.0000	0.0065a	7	1.0000	0.0013	2	1.0000	0.0136a	10
CD010089	0.5000	0.0004	0.7500a	0.0021a	5	0.7500a	0.0021a	5	0.5000	0.0008a	7	0.7500a	0.0005a	2	0.7500a	0.0009a	4
CD010847	0.6667	0.0755	0.8333a	0.0007	11	1.0000a	0.0003	8	0.6667	0.1081a	5	0.6667	0.0755	1	0.6667	0.0154	3
Weighted average	0.566a	0.003	0.669a	0.007a	9	0.691a	0.012a	7	0.571a	0.004a	7	0.666a	0.002a	2	0.641a	0.005a	7

^aResult improved when comparing with the baseline.

Considering average performance, the proposed method produces queries that improve on those used for the original review (baseline) both in terms of recall and precision. The increase in recall (10.3%) represents a marked increase in the number of relevant studies that are identified for review updates. Although the precision of the queries produced by the proposed method is still low (0.7%), this is more than double the precision obtained using the original queries, thereby halving the set of studies that need to be considered during the expensive manual screening process. Performance of the oracle method demonstrates the challenge of developing high-precision queries while also maintaining recall. Results of the restricted transformations approach indicate that using only one type of transformation generally produces queries that are more effective than the original query, but the improvement is much smaller than other approaches, indicating the importance of using different types of query transformation.

More generally, using queries produced by the proposed method leads to increased recall for 7 of the 22 reviews and the same recall for another 14. However, recall was reduced for a single review (CD007428), from 0.667 to 0.556. There were 9 relevant abstracts for this review, and this change represented a single document being missed. Precision increased for 13 reviews without any reduction in recall.

Restricting transformations to a single type leads to a reduction in performance compared with using all types. Operator substitution transformations have the lowest performance with average recall and precision only slightly higher than the baseline. Query expansion transformations are able to achieve almost as high recall as when all 3 transformation types are combined, although at the expense of precision. Perhaps surprisingly applying only the simple query reduction transformations is more effective than applying operator substitution transformations, leading to improvements in both precision and recall. However, recall drops for more reviews when only a single transformation type is used compared with all types: 2 for query reduction and 3 for operator substitution.

Figures 2 and 3 show the weighted average recall and precision scores for each iteration. The figures show the maximum number of iterations applied by each method (eg, 12 for the oracle), although note that the number of iterations applied to an individual review may be lower (eg, see Table 1). Overall, improvements in recall (compared with the baseline) appear to be generated during the first iteration, while subsequent iterations help to improve precision. The effect is particularly pronounced for the oracle but can still be observed for other methods.

Figure 2.

Weighted average recall scores for the various approaches for each iteration. Baseline approach included for comparison.

Figure 3.

Weighted average precision scores for the various approaches for each iteration. Baseline approach included for comparison.

DISCUSSION

Table 2 shows an analysis of the transformation types used by the various approaches. The table indicates the number of times each transformation was selected to generate the modified query. On the one hand, the transformation type applied most frequently by the proposed method and oracle was remove line (query reduction). The frequent use of this transformation may be explained by the fact that removing lines from queries makes them less restrictive and the objective function used to score queries prefers ones that maximize recall (ie, are less restrictive). On the other hand, the transformation types used most frequently by the operator substitution approach were or→and, .tw.→.ti. and .ti,ab.→.ti. All of these transformations lead to more restrictive queries, thereby increasing the possibility of missing relevant studies. This is reflected by the low recall achieved using this transformation type (see Table 1).

Table 2.

Analysis of transformation types used in the proposed method, oracle, and restricted transformations approaches

Transformation Category	Transformation Type	Proposed Method	Oracle	Restricted Transformations
				Operator Substitution	Query Expansion	Query Reduction
Operator substitution	.tw. → .ti.	6	8	42	—	—
	.tw. → .ti,ab.	2	2	1	—	—
	.ab. → .ti.	0	0	0	—	—
	.ab. → .ti,ab.	0	0	0	—	—
	.ti,ab. → .ti.	16	19	36	—	—
	.ti,ab. → .tw.	0	1	0	—	—
	.ti. → .tw.	0	0	3	—	—
	.ti. → .ti,ab.	0	0	1	—	—
	and → or	2	1	11	—	—
	or → and	1	2	46	—	—
	.sh. → exp*	0	1	0	—	—
Query expansion	1st top term	5	1	—	14	—
	2nd top term	3	1	—	7	—
	3rd top term	5	5	—	6	—
	4th top term	3	0	—	10	—
	5th top term	2	3	—	1	—
	1st and 2nd top terms	1	0	—	0	—
	1st to 3rd top terms	0	0	—	2	—
	1st to 4th top terms	0	0	—	0	—
	1st to 5th top terms	0	0	—	0	—
Query reduction	remove line	146	102	—	—	146
Total		192	146	140	40	146

The numbers represent how many times each transformation has been used through all iterations.

The original Boolean query is returned when the approach is unable to identify a transformation that improves performance. This happens for 1 review when the proposed method and oracle approaches are used, 3 for the operator substitution and query reduction transformation types, and 5 for the query expansion transformation type.

Figure 4 shows an example of a baseline Boolean query used for an original review and the transformed query produced by the proposed method (approach 1). For this review, the algorithm ran for 9 iterations with 2 types selected: operator substitution (use .ti. restriction for clauses 4, 8, and 16) and query reduction (removal of clauses 1, 2, 3, 5, and 7). The transformed query improved precision by 92% without any reduction in recall.

Figure 4.

Example of the original Boolean query (left) and the transformed Boolean query after 9 iterations by our proposed model (right), with highlighted lines representing the clauses transformed by the model (review CD002064).²⁶

In summary, the results of this study indicate that Boolean query transformations can improve the retrieval performance for the review update in term of recall and precision. The proposed model can produce queries that retrieve more relevant studies and reduce the workload required by researchers by half.

CONCLUSION

This study proposed a method to automatically refine Boolean queries for the study selection stage of systematic review updates. The proposed approach generates a set of transformed queries using 3 methods: operator substitution, query expansion, and query reduction. The best query is then selected using an objective function that considers both recall and precision. The method improves the original query both in terms of recall and precision. It produces queries that are able to identify relevant studies that would not be retrieved using the query from the original review.

Results demonstrated that information available from the original review, particularly the relevance judgments, can be used to produce queries that are more effective than the ones used for the original review. The method proposed here has the potential to assist researchers conducting updates of systematic reviews by supporting them to produce queries that both identify more relevant studies and reduce the number of studies that need to be screened, thereby reducing the workload required to ensure that reviews remain up to date.

The experiment described here was carried out on one type of systematic review (ie, intervention reviews) because suitable datasets are not available for other review types. In the future, it would be interesting to explore performance of the approaches described here to other types of review. Other areas of potential interest include the exploration of additional transformation types (eg, based on word embeddings or Unified Medical Language System) and alternative objective functions.

FUNDING

AA was funded by the Royal Embassy of Saudi Arabia, Cultural Bureau in London.

AUTHOR CONTRIBUTIONS

AA and MS conceived the study. AA carried out the experiments. Both authors contributed to and approved the final manuscript.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None declared.