Version Changes
Revised. Amendments from Version 2
We have made the following edits from version 2 to version 3:
We updated the affiliations of co-authors LAK, REK, IM, and NH
We updated the search from July 2020 to April 2021, our sample now includes 76 LSRs (270 living publications) instead of 32 LSRs (108 living publications). Hence, we updated the methods sections of the abstract and main-text.
We updated the survey findings in table 1 (characteristics of the living publications), table 2 (reporting on study flow), and figure 1 (Summary of the four tailored PRISMA flow diagram approaches) with the analysis of the 76 LSRs (270 living publications). Hence, we updated our results sections of the abstract and main-text.
Our conclusions did not change
Our suggestions of approaches to documenting and reporting LSR study flow did not change.
We referenced under implications for practice a web-based App that can facilitate producing flow diagrams for LSRs using the four approaches suggested.
We discussed under implications for research our plans and in developing extension to the PRISMA 2020 statement for LSRs.
We did some copy-editing.
Abstract
Background: While the PRISMA flow diagram is widely used for reporting standard systematic reviews (SRs), it was not designed for capturing the results of continual searches for studies in living systematic reviews (LSRs). The objectives of this study are (1) to assess how published LSRs report on the flow of studies through the different phases of the review for the different updates; (2) to propose an approach to reporting on that flow.
Methods: For objective 1, we identified all LSRs published up to April 2021. We abstracted information regarding their general characteristics and how they reported on search results. For objective 2, we based our proposal for tailored PRISMA approaches on the findings from objective 1, as well as on our experience with conducting Cochrane LSRs.
Results: We identified 279 living publications relating to 76 LSRs. Of the 279 publications, 11% were protocols, 23% were base versions (i.e., the first version), 50% were partial updates (i.e., does not include all typical sections of an SR), and 16% were full updates (i.e., includes all typical sections of an SR). We identified six ways to reporting the study flow: base separately, each update separately (38%); numbers not reported (32%); latest update separately, all previous versions combined (20%); base separately, all updates combined (7%); latest update version only (3%); all versions combined (0%). We propose recording in detail the results of the searches to keep track of all identified records. For structuring the flow diagram, we propose using one of four approaches.
Conclusion: We identified six ways for reporting the study flow through the different phases of the review for the different update versions. We propose to document in detail the study flow for the different search updates and select one of our four tailored PRISMA diagram approaches to present that study flow.
Keywords: PRISMA statement, living systematic review, update, research methodology research reporting, flow chart, systematic review reporting standards, evidence synthesis, research transparency, research replication
Introduction
During the coronavirus disease 2019 (COVID-19) pandemic, health research has proliferated exponentially 1 . Systematic reviews are essential to synthesize the evidence and inform policy and practice. Given the pace of research publication, those reviews need to be kept up to date. Living systematic reviews (LSRs) are an emerging type of systematic review that involves the continual search of the literature and incorporation of relevant new evidence, soon after it becomes available 2 . While many evidence synthesis groups are engaged in conducting LSRs or living network meta-analyses, others have developed living databases or living maps, including resources specific for COVID-19 literature 3– 17 .
An essential component of systematic reviews is to keep track of and report the number of records captured while searching the scientific literature and details of the selection process 18 . The PRISMA statement recommends the use of the PRISMA flow diagram to depict the flow of studies through the different phases of the systematic review 19 . While the PRISMA flow diagram is a widely used tool for reporting original systematic reviews, it was not designed to capture the results of continual searches typically used in LSRs. Hence, it’s unclear how authors of LSRs address the issue of presenting results of these continual searches.
The objectives of this study were (1) to assess how published LSRs report on the flow of studies through the different phases of the review for the different updates; and (2) to propose an approach to documenting and reporting on the flow of studies through the different phases of a LSR, for the different updates.
Methods
For objective 1, we collected relevant data as part of a larger methodological survey aiming to assess the methods of conduct and reporting of LSRs. We have described the details of that study in a previously published protocol 20 . Briefly, we identified all living reviews published up to April 2021 available from the following electronic databases: Medline, EMBASE and the Cochrane library (see extended data 21 of Khamis et al. 20 for the search strategy). An eligible living review was either (1) a protocol for an LSR, (2) a base version of an LSR, (3) a full update version of an LSR, (4) a partial update version of an LSR, or (5) a combination of any of these (e.g., one living review may constitute of a protocol, a base version, and a full update version; another living review may constitute of only a Box 1 the definition of each type of living reviews.
Box 1. Definition of the different publication types of living reviews.
• LSR protocol: the protocol that describes the planned methods of the living review
• Base version: the first version of the review that follows a living approach
• Full update version: a subsequent version of the review that includes all the typical sections of a systematic review, including an introduction, methods, and results sections. Such a version could stand-alone in terms of content.
• Partial update version: a subsequent version of the review that does not include all the typical sections of a systematic review, but instead refers to a previous version for complementary information. Such a version could not stand-alone in terms of content.
For the current study, we abstracted information about the following features of LSRs:
-
General characteristics:
Publication type, i.e., protocol, base version, full update version, partial update version.
Whether published in the Cochrane library or elsewhere.
Field (e.g., clinical, public health)
Whether COVID-19 related or not
Whether the base version of the living review conducted as a rapid review or not
-
Reporting on study flow
-
Method used to report on the study flow (including the search results and the results of the selection process):
-
▪
Narrative format and/or flow diagram.
-
▪
Whether the results of the base and update searches are reported separately or not.
-
▪
Type of flow diagram, if applicable (e.g., PRISMA).
-
For objective 2, we base our proposal for tailored PRISMA 2020 flow diagram approaches on the findings from objective 1, on our experience conducting Cochrane LSRs, and our methodological work on designing and reporting living evidence. Since 2017, our group has been responsible for the first series of three Cochrane LSRs, all of which address anticoagulation in patients with cancer 22– 24 . We conducted the base search in February 2016. Since then, we have been updating the search on a monthly basis. Through this experience, we have been able to apply and refine the guidance for conducting LSRs endorsed by the living evidence network group 25 . Specifically, we explored solutions for the reporting of the study flow that would address different scenarios. Our goal was not to be prescriptive and narrow, but rather to cover all possible resulting flows by reviewing the LSRs we identified based on objective 1. Two authors developed a draft of the tailored approaches to presenting the study flow, and then circulated to the author team for review and suggestions for improvement.
Data handling and analysis
We used REDCap to collect and manage the data abstraction process. All data were exported from REDCap and analyzed using Stata v. 13 26, 27 .
Results
Survey findings
Our search identified a total of 279 living publications relating to 76 LSRs. Table 1 shows their general characteristics. Of the 279 living publications, 11% were protocols, 23% were base versions, 50% were partial updates, and 16% were full updates. The median number of living publications per LSR was 2 (Interquartile range 1–4). Of the 76 living reviews, 22% were published in the Cochrane library, 63% were related to COVID-19, and 25% had a base version published as a rapid review. The majority were related to clinical topics (70%).
Table 1. General characteristics of the 279 included living publications related to 76 living reviews.
| N | n (%) | |
|---|---|---|
| Publication type | 279 living publications | |
| • Protocol | 31 (11.1) | |
| • Base version | 64 (22.9) | |
| • Partial update version | 138 (49.5) | |
| • Full update version | 46 (16.5) | |
| Living publications per LSR (Median (IQR)) | 76 LSRs | 2 (1 – 4) |
| Cochrane LSR | 76 LSRs | 17 (22.4) |
| Field | 76 LSRs | |
| • Clinical | 53 (69.7) | |
| • Public health | 20 (26.3) | |
| • Health system and policy | 3 (4.0) | |
| COVID-19-related | 76 LSRs | 48 (63.2) |
| Base version published as rapid review | 64 base versions | 16 (25.0) |
Abbreviations: LSR: living systematic review; IQR: interquartile range
Table 2 shows the results for the reporting on the study flow. Most base versions and full updates used a flow diagram to report on the search results (96% and 93% respectively), whereas only one partial update presented a flow diagram. In addition, none of the 279 living publications reported in their methods section how they plan to report on the study flow.
Table 2. Reporting on study flow.
| N | n (%) | |
|---|---|---|
| Inclusion of a flow diagram in a | ||
| • Base version | 64 base versions | 62 (96.9) |
| • Partial update version | 138 partial updates | 1 (0.7) |
| • Full update version | 46 full updates | 43 (93.5) |
|
Approach to reporting on study flow for
different versions |
184 update versions | |
| Base separately; each update separately | 39 (21.2) | |
| Base separately; all updates combined | 20 (10.9) | |
| Latest update separately; all previous versions combined (including the base) | 24 (13.0) | |
| All versions combined | 12 (6.5) | |
| Latest update version only | 47 (25.5) | |
| Numbers not reported | 42 (22.8) |
a When a flow diagram is not reported, the authors reported on the search results in a narrative format.
Among the 184 update versions ( Figure 1):
Figure 1. Summary of the four tailored PRISMA flow diagram approaches to present that study to presenting on the study flow for the different search updates.
21% reported the search results for the base version and for each update version separately.
11% reported on the search results for the base version separately and for all update versions combined
13% reported the search results for the latest update version separately and for all previous versions combined (including the base).
6% reported the search results for all the different versions combined.
26% reported the search results for the latest update version only.
23% did not report the search results at all (e.g., ‘new studies identified and integrated’ without specifying the number).
Proposed tailored PRISMA flow diagram approaches
Using the approach described in the methods section, we developed four approaches that allow authors to document and report the study flow for the different review update versions of an LSR.
1. Documenting LSR study flow
Authors should record in detail the results of the searches to keep track of all identified records. We propose using a spreadsheet for one LSR at a time. The format we present consists of tabs for each of the respective search sources: bibliographic databases (e.g. MEDLINE, EMBASE, Cochrane databases); conference proceedings; ongoing studies as captured in clinicaltrials.gov and WHO International Clinical Trials Registry Platform (ICTRP); other tabs as needed, and a final ‘cumulative’ tab.
We show in Figure 2 a snapshot of the ‘cumulative’ tab of the spreadsheet that keeps track of all records. It shows the study flow for a hypothetical example for an LSR published first in January 2020 (i.e., base version) and updated on a monthly basis up to August 2020. Each row corresponds to a different update version. The columns present the following information for each update (columns B to E): the number of records received, deduplicated, included at title and abstract screening, and included at full-text screening (i.e., newly included reports). Additional columns (F to I) present the distribution of the newly included reports as relating to either: (1) new studies, (2) previously included studies, (3) ongoing (unpublished) studies, or (4) preprints.
Figure 2. Snapshot of the ‘cumulative’ tab of the spreadsheet that keep track of all identified records.
After manually entering the information in the first five tabs (corresponding to the different search sources) the total is automatically computed in the ‘cumulative’ tab.
2. Reporting LSR study flow
The proposed spreadsheet can act as a basis for a tailored PRISMA flow diagram for LSRs. For structuring the flow diagram for LSR, one can select one out of four tailored PRISMA 2020 flow diagram approaches:
Approach 1: presenting the search results of the different versions separately (i.e., base and each update separately) ( Figure 3).
Approach 2: presenting the search results for the different versions combined (i.e., including base and all update versions) ( Figure 4).
Approach 3: presenting the search results for the base version separately, and the results of all update version combined ( Figure 5).
Approach 4: presenting the results of the latest update version separately, and the results of all previous versions (including the base) combined ( Figure 6).
Figure 3. Approach 1: presenting the search results of the different versions separately (i.e., base and each update separately).
Figure 4. Approach 2: presenting the search results for the different versions combined (i.e., including base and all update versions).
Figure 5. Approach 3: presenting the search results for the base version separately, and the results of all update version combined.
Figure 6. Approach 4: presenting the results of the latest update version separately, and the results of all previous versions (including the base) combined.
In our Cochrane reviews, we applied the second proposal where we present the results for the different searches combined.
Discussion
Summary
This study found that authors of LSRs are not consistent in reporting on the flow of studies through the different phases of the review for the different update versions. Thus, we propose to document in detail the study flow for the different search updates and select one of four tailored PRISMA 2020 flow diagram approaches to present that study flow.
Strengths and limitations
To our knowledge, this is the first methodological survey that assesses how LSR authors report on the flow of studies through the different phases of the review for the different update versions of LSRs. In addition, the research expertise on our team covers both living approach and regular updating of traditional SR. We believe that our assessment forms a vital baseline and allows us to propose best practices for visualization options to improve consistency whilst the production of LSRs is still at a relatively early stage. Indeed, this survey is part of a larger methodological survey aiming to assess the methods of conduct and reporting of LSRs 20 , that would allow us to update our findings in the future.
Interpretation of findings
Authors tend to produce more partial updates of LSRs rather than continually updating the full systematic review. This might seem like a pragmatic approach particularly for a rapidly growing research field and when methods do not seem to change from one update to another. The heterogeneity observed in the ways LSR authors report on the study flow is likely to be explained by the lack of clear guidance on how to do so.
Implications for practice
We built our proposal on the PRISMA 2020 flow diagram and provide four approaches to tailor the needs for continual searchers used in LSR. The fourth approach is the closest to the current PRISMA 2020 flow diagram as it presents the results of the latest update version separately and the results of all previous versions (including the base) combined.
In addition, we proposed three other different approaches to provide options to LSR authors and publishing journals. Authors should choose one or the other approach based on the number of new citations, presentation preferences, and the impression of what provides the greatest transparency in reporting. Whatever approach one decides to follow, for transparency purposes, the systematic reviewers should ideally archive previous versions of the flow diagram (e.g., in an appendix). One major challenge will be to accommodate a large number of updates in the same diagram; some approaches would work better than others in that case. Also, advanced information technology solutions may allow fitting a large number of updates. A web-based prototype is available that allows readers to explore different reporting options across these four approaches: an R package ( https://github.com/nealhaddaway/livingPRISMAflow) and web-based ShinyApp ( https://estech.shinyapps.io/livingprismaflow/) were developed that allow users to enter their own data (e.g., from the spreadsheet suggested above) to produce a bespoke flow diagram according to their desired approach or to create their own interactive diagram that allows readers to toggle between different versions of the same data 28 .
Advanced information technology can also be utilized to simplify updating and tracking the change in all LSR sections including the PRISMA diagram. It would be optimal to develop the base version in a certain platform where all SR and LSR sections are reported as units (i.e., title, authors, background, objectives, inclusion criteria, effect estimate for outcome x). With each update and for every unit, the author has the luxury to keep the same text (if no change has occurred) or edit (if change has occurred). Each unit can be updated in a differential speed based on certain criteria. The edits could be highlighted to visualize the change. For a certain section, one would easily have access to the entries in the previous versions and possibly visualize a trend across the different versions (i.e., cross-sectional view for that specific item). For example, dynamic documents can be developed using ‘R markdown’, a document preparation system, where static text can be combined with in-line code and ‘code chunks’ that produce instantly updatable documents given a modified input 29 .
Implications for future research
This study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs (please see registration form on EQUATOR network website: Equator Network. PRISMA for LSR – Extension of PRISMA 2020 for living systematic reviews. 2021; Accessed from https://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-systematic-reviews/#LSR]. This project will pilot the proposed approaches for documenting the study flow and for structuring the living flow diagram. In addition, qualitative studies would be helpful to explore: (1) the feasibility and acceptability by LSR authors, publishers, and users towards the proposal; and (2) what the end-users would like to see in an LSR update.
Conclusions
LSR authors are not consistent in reporting the flow of studies through the different phases of the review for the different update versions. We propose to document in detail the study flow for the different search updates. Authors can select one of our four tailored PRISMA 2020 flow diagram approaches to present that study flow until detailed guidance will become available. Improving the reporting of study flow in LSR methodology is essential for incorporating living evidence when developing living guidance, particularly in the context of an urgent response 30, 31 .
Data availability
Underlying data
All data underlying the results are available as part of the article and no additional source data are required.
Acknowledgment
We would like to acknowledge Dr. Mathew Page for his revision for the manuscript.
Funding Statement
The author(s) declared that no grants were involved in supporting this work.
[version 3; peer review: 2 approved]
References
- 1. Nasrallah AA, Farran SH, Nasrallah ZA, et al. : A large number of COVID-19 interventional clinical trials were registered soon after the pandemic onset: a descriptive analysis. J Clin Epidemiol. 2020;125:170–178. 10.1016/j.jclinepi.2020.06.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Elliott JH, Turner T, Clavisi O, et al. : Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 2014;11(2):e1001603. 10.1371/journal.pmed.1001603 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. John A, Eyles E, Webb RT, et al. : The impact of the COVID-19 pandemic on self-harm and suicidal behaviour: update of living systematic review [version 1; peer review: 1 approved, 2 approved with reservations]. F1000Res. 2020;9:1097. 10.12688/f1000research.25522.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Currie G, Macleod MR, Sena E, et al. : Protocol for a “living” evidence summary of primary research related to Covid-19.2020. 10.17605/OSF.IO/Q5C2V [DOI] [Google Scholar]
- 5. Geisler BP, Zahabi L, Lang AE, et al. : Repurposing Existing Medications for Coronavirus Disease 2019: Protocol for a Rapid and Living Systematic Review. medRxiv. 2020. 10.1101/2020.05.21.20109074 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Juul S, Nielsen N, Bentzer P, et al. : Interventions for treatment of COVID-19: a protocol for a living systematic review with network meta-analysis including individual patient data (The LIVING Project). Syst Rev. 2020;9(1):108. 10.1186/s13643-020-01371-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Maguire BJ, Guérin PJ: A living systematic review protocol for COVID-19 clinical trial registrations [version 1; peer review: 2 approved]. Wellcome Open Res. 2020;5:60. 10.12688/wellcomeopenres.15821.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Maguire BJ, McLean ARD, Rashan S, et al. : Baseline results of a living systematic review for COVID-19 clinical trial registrations [version 1; peer review: 2 approved]. Wellcome Open Res. 2020;5:116. 10.12688/wellcomeopenres.15933.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Schünemann HJ, Khabsa J, Solo K, et al. : Ventilation techniques and risk for transmission of coronavirus disease, including COVID-19: a living systematic review of multiple streams of evidence. Ann Intern Med. 2020;173(3):204–216. 10.7326/M20-2306 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Ortiz-Muñoz LE, Ferrer BM, Duarte-Anselmi G, et al. : Protocol of a Living systematic review: Gloves for the prevention of COVID-19 in healthy population.2020. 10.31219/osf.io/qykh2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Siemieniuk RA, Bartoszko JJ, Ge L, et al. : Drug treatments for covid-19: living systematic review and network meta-analysis. BMJ. 2020;370:m2980. 10.1136/bmj.m2980 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Rada G, Verdugo-Paiva F, Ávila C, et al. : Evidence synthesis relevant to COVID-19: a protocol for multiple systematic reviews and overviews of systematic reviews. Medwave. 2020;20(3):e7868. 10.5867/medwave.2020.03.7867 [DOI] [PubMed] [Google Scholar]
- 13. Cochrane: Coronavirus (COVID-19) - Cochrane resources and news.The Cochrane Collaboration.2019. Reference Source [Google Scholar]
- 14. World Health Organization: Global research on coronavirus disease (COVID-19).2020. Reference Source [Google Scholar]
- 15. Living Overview of Evidence (LOVE): Coronavirus disease (COVID‑19). Epistemonikos foundation.2020. Reference Source [Google Scholar]
- 16. EPPI center: COVID-19: a living systematic map of the evidence.2020. Reference Source [Google Scholar]
- 17. Boutron I, et al. : Interventions for preventing and treating COVID-19: protocol for a living mapping of research and a living systematic review. Syst Rev. 2020;9:108.32386514 [Google Scholar]
- 18. Millard T, Synnot A, Elliott J, et al. : Feasibility and acceptability of living systematic reviews: results from a mixed-methods evaluation. Syst Rev. 2019;8(1):325. 10.1186/s13643-019-1248-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Page MJ, McKenzie J, Bossuyt P, et al. : The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.2020. 10.31222/osf.io/v7gm2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Khamis AM, Kahale LA, Pardo-Hernandez H, et al. : Methods of conduct and reporting of living systematic reviews: a protocol for a living methodological survey [version 2; peer review: 2 approved]. F1000Res. 2019;8:221. 10.12688/f1000research.18005.2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Khamis AM, Kahale LA, Pardo-Hernandez HJ, et al. : Search strategies. figshare. Journal contribution.2019. 10.6084/m9.figshare.7688036.v1 [DOI] [Google Scholar]
- 22. Akl EA, Kahale LA, Hakoum MB, et al. : Parenteral anticoagulation in ambulatory patients with cancer. Cochrane Database Syst Rev. 2017;9(9): CD006652. 10.1002/14651858.CD006652.pub5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Kahale LA, Hakoum MB, Tsolakian IG, et al. : Anticoagulation for the long-term treatment of venous thromboembolism in people with cancer. Cochrane Database Syst Rev. 2018;6(6): CD006650. 10.1002/14651858.CD006650.pub5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Kahale LA, Hakoum MB, Tsolakian IG, et al. : Oral anticoagulation in people with cancer who have no therapeutic or prophylactic indication for anticoagulation. Cochrane Database Syst Rev. 2017;12(12): CD006466. 10.1002/14651858.CD006466.pub6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Brooker J, Synnot A, McDonald S, et al. : Guidance for the production and publication of Cochrane living systematic reviews: Cochrane reviews in living mode. Cochrane Collaboration. 2019. Reference Source [Google Scholar]
- 26. StataCorp: Stata Statistical Software: Release 13.College Station, Editor. StataCorp LP: TX.2013. [Google Scholar]
- 27. Harris PA, Taylor R, Thielke R, et al. : Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81. 10.1016/j.jbi.2008.08.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Haddaway N: SRflowdiagram: flow charts for systematic reviews and maps.2020. [Google Scholar]
- 29. Haddaway NR: DynamicSMapResults: template for designing updatable systematic map results text using R Markdown (Version 0.0.1). Zenodo. 2020. 10.5281/zenodo.4401173 [DOI] [Google Scholar]
- 30. Akl EA, Morgan RL, Rooney AA, et al. : Developing trustworthy recommendations as part of an urgent response (1-2 weeks): a GRADE concept paper. J Clin Epidemiol. 2021;129:1–11. 10.1016/j.jclinepi.2020.09.037 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Schünemann HJ, Santesso N, Vist GE, et al. : Using GRADE in situations of emergencies and urgencies: certainty in evidence and recommendations matters during the COVID-19 pandemic, now more than ever and no matter what. J Clin Epidemiol. 2020;127:202–207. 10.1016/j.jclinepi.2020.05.030 [DOI] [PMC free article] [PubMed] [Google Scholar]