Study within a review (SWAR)

Systematic reviews and other evidence syntheses bring together information from multiple studies to help inform policy and practice decisions. They use systematic methods to identify, select, appraise (where appropriate), and analyze a body of evidence and report their findings with the aim of minimizing bias and helping people make well‐informed decisions. Alongside the enormous increase in the number of systematic reviews, ¹ , ² recent decades have witnessed prodigious growth in the evidence base to inform decisions on how we plan, do and share the findings of systematic reviews. ³ For example, methodological papers and consensus statements on the development and validation of search strategies, assessment of the risk of bias, synthesis of complex interventions, qualitative evidence synthesis, etc., regularly appear in peer‐reviewed journals. ⁴ , ⁵ , ⁶ , ⁷

This evolving evidence base on how we plan, do, and share the findings of reviews is important, relevant and helpful. However, many evidence uncertainties exist and greater efforts, ideally in a coordinated manner, are needed to address them across the systematic review process. Examples include optimal approaches to searching and citation screening, ⁸ the effectiveness of machine learning in supporting screening and/or data extraction, ⁹ and the effectiveness of different modes of sharing the findings of reviews with different audiences. ¹⁰ In addition, developments in systematic review approaches need high‐quality evidence to inform decisions about how they are done. For example, living systematic reviews, which seek to reduce the time between the availability of the findings of primary research studies and their synthesis, need better evidence to inform decisions around team processes, managing workloads, integration of pathways from searching and screening through data extraction to updating analyses. ¹¹ These uncertainties will persist unless the evaluation of alternative approaches to review processes becomes commonplace. A resource‐efficient way of conducting these evaluations is to undertake a Study Within A Review (SWAR). A SWAR is a discrete study analogous to the Study Within A Trial (SWAT), which aims to strengthen the evidence base for trial methods. ¹² A SWAR and SWAT registry was established by the All‐Ireland Hub for Trials Methodology Research (now the Northern Ireland Methodology Hub) in collaboration with the Medical Research Council's Network of Hubs in the UK and the Global Health Network. ¹³ The registries are available for researchers across the globe to register outlines for their SWAR and SWAT at no charge and are free to view (go.qub.ac.uk/SWAT‐SWAR).

This paper aims to describe SWARs, highlight their importance, and discuss key considerations for conducting a SWAR. This will help systematic reviewers, funders, those mining the evidence base, and methodologists to plan, conduct, or support studies to improve the evidence base of systematic review and evidence synthesis methods. Researchers generally make the decision to conduct a SWAR: we hope this paper will convince more researchers to include one in their reviews.

1. WHAT IS A STUDY WITHIN A REVIEW (SWAR)?

A SWAR is a research study that can help provide evidence to inform decisions about how we plan, do and share the findings of future reviews. It addresses methodological uncertainty and is usually embedded within a systematic review or another evidence synthesis to evaluate the effectiveness of alternative ways of delivering or organizing a particular review process. Guided by the methodological uncertainty they seek to address, a research team can select from diverse study approaches, including randomized and nonrandomized comparisons, mixed‐methods, or qualitative study types, to design a SWAR to explore the implications and consequences of such methodological decisions.

Some SWARs can be conducted within a single systematic review, whereas others might answer questions that require embedding in multiple systematic reviews. SWARs can be undertaken in several reviews simultaneously or over time, building the evidence temporally. Alternatively, SWARs may explore qualitative considerations, such as the utility of the final review product or the quality of the interpretation associated with systematic review processes such as screening, data extraction, critical appraisal, synthesis, and sharing of findings. Given that much of the data required to answer SWAR research questions is either generated during a review or can be collected with little effort, SWARs offer a potentially cost‐effective approach to improving review processes.

SWARs comparing alternative ways of delivering or organizing a particular review process must consider not only the effectiveness of the alternate ways of planning, conducting, analyzing, and reporting systematic reviews but also the impact these processes may have on the review findings and author's conclusions.

2. WHY DO WE NEED TO DO SWARS?

Systematic reviews must be conducted with methodological rigor and transparency because their results affect people's care and, therefore, their lives. Reviews also need to be seen to be right. This is not only because they provide the justification for a given treatment decision but also because they are often used in regulatory and guidance processes. They, therefore, need to be seen to have followed a justifiable and auditable process for legal reasons. It is also important that the processes by which reviews are planned, conducted, analyzed, and reported are informed by up‐to‐date research evidence rather than by convention. This maximizes the likelihood that review processes are efficient, thereby minimizing waste of time and other resources. Researchers should aim for comparable rigor to that with which reviews seek to answer questions about health and social care when seeking to inform decisions on how we “do” reviews. Investigating systematic review methods and exploring alternative methods (also known as research on research) have a key role in improving the methods of conducting systematic reviews and evidence synthesis. SWARs can help to achieve this.

Although first proposed in 2012, ¹⁴ relatively few SWARs have been conducted or reported. As of September 2022, the SWAR Repository (SWAR Store | The Northern Ireland Network for Trials Methodology Research (qub.ac.uk)) lists 13 SWARs as registered with some published findings. ¹⁵ , ¹⁶ , ¹⁷ This makes it likely that many decisions about how reviewers do reviews have not been informed by good‐quality research evidence. We know, however, that studies that seek to answer methodological uncertainties in how we conduct reviews are not typically conceptualized as SWARs. Framing these studies as SWARs, including in their registration and reporting, would help for two reasons:

1. In keeping better track of completed and ongoing studies. This would minimize the risk of unnecessary duplication of research by failing to be cumulative in our knowledge generation—one of the things that systematic reviews aim to avoid

2. Better formalization of SWARs may contribute to higher standards in how they are conducted and reported.

3. KEY FEATURES OF A SWAR

• Aims to generate evidence to address uncertainties in the processes of how we plan, do, and share the findings of systematic reviews

• Selects from diverse study designs appropriate to the methodological uncertainty the SWAR is trying to answer

• Either is embedded within a single review or across multiple reviews to make any necessary comparison

• Should not compromise the objective, methods, integrity, outcome, and dissemination of the host review or reviews

• Is accompanied by its own discrete protocol. We encourage researchers to register protocols in the SWAR repository (SWAR Store | The Northern Ireland Network for Trials Methodology Research (qub.ac.uk))

• Informs the method, design, and implementation of future systematic reviews and evidence synthesis. Where appropriate, the outcome of a SWAR may inform decisions within its host systematic review(s)

A SWAR need not be prohibitively expensive. Indeed, a SWAR is likely more resource‐efficient than a separate discrete study. However, seed funding could be allocated to cover researcher time, which is always needed and may help to ensure that SWARs are seen as an important and justifiable use of time against competing demands. The findings of the SWAR should be made available as an open‐access research report.

In many cases, ethical approval is not required, given the absence of directly recruited research participants. However, where a SWAR research team plans to engage researchers, academicians, methodologists, end‐users, or the public, and collect information from them, ethical approval may be required. In addition, most syntheses are subject to time constraints imposed by review commissioners who need up‐to‐date evidence promptly. This can create tensions between those proposing and doing SWARs and those commissioning the host reviews. However, in practice, most SWARs are unlikely to affect the time taken to complete a review and potentially may reduce it. Indeed, the evidence that SWARs seek to generate is targeted at increasing review efficiency. Therefore, the SWAR agenda will likely help meet decision makers need for high‐quality evidence quickly.

4. EXAMPLES OF SWARS

One example of a SWAR is the randomized trial of two different methods of screening citations for inclusion of papers ¹⁸ in a systematic review of the effects of pulmonary rehabilitation in people with chronic obstructive pulmonary disease (COPD). ¹⁹ The authors randomized a database of 1072 citations to be screened by the same two reviewers using either title screening followed by the screening of title and abstract (two‐stage approach) or screening of the title and abstract simultaneously (one‐stage approach). The findings suggest that screening using a method in which titles and abstracts are presented simultaneously is less time‐consuming than a two‐stage process of presenting the title followed by the abstract.

Another study, ¹⁰ although not conceptualized as a SWAR, compared two different approaches for contacting authors to request additional data for a systematic review investigating whether central adjudication of the primary outcome in stroke trials impacts the main trial primary analysis. ¹⁶ Participants were randomized to either a short email with a protocol of the systematic review attached (“Short”) (n = 45) or a longer email containing detailed information but without the protocol attached (“Long”) (n = 40). Results indicated no difference between the two approaches in terms of the author response to the requested review information.

A similar SWAR in a review of hospital volume‐outcome relationships in total knee arthroplasty ²⁰ found no difference in author response rates to personalized requests for study information between email text or email attachment with self‐developed, personalized, personalized data request forms. The study authors also found that designing and prefilling data request forms resulted in at least 50% more reviewer time per author contacted.

We are not yet aware of examples of embedding a SWAR addressing the same question within multiple host reviews simultaneously. However, a simultaneous SWAT evaluation design proved successful for trial retention, being demonstrated as both feasible and efficient. ²¹ Such an approach appears suitable for SWARs. For example, a SWAR trial looking at single versus dual screening of citations for inclusion in a review could be embedded as a trial in multiple host reviews simultaneously, the SWAR results collated, synthesized, and reported allowing a more rapid increase of the evidence base.

5. IDENTIFYING PRIORITIES FOR SWARS: PRIORITY III

Key to the development of the SWAR agenda is the identification of priorities for research, and this need has been addressed by Priority III. Priority III was an Evidence Synthesis Ireland project that identified the top 10 unanswered questions on how we plan, do, and share the findings of rapid reviews. ²² Rapid reviews were defined as “A rapid review is a type of evidence synthesis that brings together and summarises information from lots of different research studies to produce evidence for people such as the public, researchers, policymakers and funders in a systematic, resource‐efficient manner. This is done by speeding up the ways we plan, do and/or share the results of conventional structured (systematic) reviews by simplifying or omitting a variety of methods that should be clearly defined by the authors.” ²²

Priority III involved extensive collaboration with patients, public, reviewers, researchers, clinicians, policymakers, and funders to do something similar for rapid reviews. The study used a modified James Lind Alliance (JLA) priority‐setting partnership (PSP). A typical JLA PSP identifies and prioritizes unanswered questions about health and social care that the public, carers, and professionals jointly agree are the most important. The prioritized questions offer a rich source of unanswered methodological questions on rapid reviews, of which many could be answered within a SWAR. Many of the top unanswered questions at https://evidencesynthesisireland.ie/priority‐iii/ could be best explored within systematic reviews.

6. MANAGING SWARS: SWAR PROGRAMS

Evidence Synthesis Ireland (ESI) is an all‐Ireland initiative funded by the Health Research Board (HRB) in the Republic of Ireland and the HSC R&D Division of the Public Health Agency in Northern Ireland. ESI aims to make evidence syntheses better planned, conducted, and reported, more useable for decision makers and within health care policy and clinical practice decision‐making across the island of Ireland and beyond. One of the initiatives of ESI is the ESI SWAR programme. This programme aims to support research that evaluates alternative ways of delivering or organizing a particular review process or explores factors associated with that process. Another example is the Strategy for Patient Oriented Research (SPOR) Evidence Alliance, which tests different approaches to automating reviews commissioned by decision makers. ²³ The UK's National Institute for Health and Care Research (NIHR) also supports SWARs in its Health Technology Assessment Programme. ²⁴ An example of a review conducted during the COVID‐19 pandemic for the World Health Organization was published in the BMJ Open where screening using the CAL(R) tool is currently being compared with screening by humans only through a SWAR. ²⁵ The ESI SWAR program and other such programs can potentially reduce research waste and improve the usefulness of systematic reviews. More information on ESI's SWAR program is available http://www.evidencesynthesisireland.ie/SWAR.

7. REGISTERING SWARS: THE SWAR STORE

The Northern Ireland Methodology Hub hosts the SWAR repository. The repository provides a centralized library of these methodology studies and a place where people can log their proposed study and deposit their findings, which may contribute to future meta‐analyses of the individual SWARs.

The list of registered and ongoing SWARs is available at SWAR Store | The Northern Ireland Network for Trials Methodology Research (qub.ac.uk). A new SWAR can be registered using a simple application form designated for the registration (SWAR Application | The Northern Ireland Network for Trials Methodology Research (qub.ac.uk)) and we encourage prospective registration of SWARs to promote transparency and avoid unnecessary duplication of effort.

8. DISSEMINATION OF SWARS

SWAR findings will, ideally, be published as a stand‐alone paper. If the SWAR was conducted within a host review, its findings could, where necessary and appropriate, be published within a section of the host review. In this case, its presence should be noted in the abstract with a clear indication of what was tested in the SWAR and the use of terminology such as “Study Within a Review (SWAR)” in the abstract to maximize its retrieval from searches of citation databases.

Where a SWAR addressing the same question is conducted within multiple host reviews simultaneously, the results from each SWAR can be synthesized and reported simultaneously, within one publication. Regardless, we encourage the publication of SWAR findings in peer‐reviewed, open‐access journals so that their findings can better inform decisions on how we “do” systematic reviews. Linking SWAR findings with their associated protocols in the SWAR Repository Store should also be explored. We encourage the dissemination of findings of SWARs through diverse channels, including open‐access peer review papers, webinars, podcasts, conference presentations, and blogs.

Devane D, Burke NN, Treweek S, et al. Study within a review (SWAR). J Evid Based Med. 2022;15:328–332. 10.1111/jebm.12505