Abstract
For researchers, policymakers, and practitioners facing a new field, undertaking a systematic review can typically present a challenge due to the enormous number of relevant papers. A scoping review is a method suggested for addressing this dilemma; however, scoping reviews present their own challenges. This paper introduces the “scoping meta‐review” (SMR) for expanding current methodologies and is based on our experiences in mapping the field of consumer engagement in healthcare. During this process, we developed the novel SMR method. An SMR combines aspects of a scoping review and a meta‐review to establish an evidence‐based map of a field. Similar to a scoping review, an SMR offers a practical and flexible methodology. However, unlike in a traditional scoping review, only systematic reviews are included. Stages of the SMR include: undertaking a preliminary nonsystematic review; building a search strategy; interrogating academic literature databases; classifying and excluding studies based on titles and abstracts; saving the refined database of references; revising the search strategy; selecting and reviewing the full text papers; and thematically analyzing the selected texts and writing the report. The main benefit of an SMR is to map a new field based on high‐level evidence provided by systematic reviews.
Keywords: scoping meta‐review (SMR), scoping review, meta‐review, systematic review, literature review, methodology
Background
Despite the astonishing advances in electronic databases and search facilities, researchers are increasingly experiencing challenges when mapping a particular field of interest due to rapid growth in the amount of data available combined with limited time to complete the task. The condition can be especially challenging for those scholars who have to navigate an emerging interdisciplinary topic, in which there might be inadequate direct studies in relation to the specific topic, but a large body of evidence in related fields. For example, if a new field is emerging based on a combination of a particular field of medicine, such as orthopedics, and a specific area in engineering, such as biomechanics, this new interdisciplinary field might lack direct evidence, despite numerous relevant papers in each contributing field being accessible. In such cases, undertaking a systematic review can be challenging and time‐consuming, and may prove too difficult for researchers with limited resources.
One way to address this challenge is to undertake a meta‐review. A meta‐review or “overview of reviews” is a particular form of systematic review that includes only systematic reviews, and is suitable whenever relevant systematic reviews are available.1 Computerized therapy for depression, for example, is an emerging topic and various systematic reviews have explored the related literature. Subsequently, a meta‐review undertaken in this field explored the available systematic reviews.2 While the stages for conducting a meta‐review are not materially different from completing other systematic reviews, this model examines systematic reviews rather than primary studies.1 Limiting the review to systematic reviews can reduce the workload in some cases; however, it is not unusual for a broad topic, particularly in an interdisciplinary area, to face vast number of systematic reviews. Therefore, undertaking a meta‐review can also be challenging due to the requirement to include all available systematic reviews and to undertake rigorous appraisal of each.
A scoping review is an emerging literature review methodology aiming to address the challenges emerging with systematic reviews.3, 4 Various definitions and purposes are suggested for a scoping review.4 For example, it can be used as a transparent method to map the literature and to address broad research questions on a topic.3 By mapping the literature, we intended to provide background information necessary for our upcoming projects;5 identifying concepts, gaps, strategies and barriers; and exploring the relationship between concepts, ideas, and problems in the field.6 Therefore, mapping the literature provides a broad view of the material available on a topic, rather than the more focused picture normally provided by a systematic review. Arksey and O'Malley have identified five stages for a scoping review:3 recognizing the research questions; identifying related papers and studies; selecting studies and papers; charting the data; and integrating, summarizing and reporting the results. However, those undertaking a scoping review can encounter a number of challenges and limitations. The process of decision making for inclusion and exclusion of papers, for example, is often not clear and there can be concerns over the quality of the evidence produced by this method.4
Methods
This paper is based on our experiences in mapping the field of consumer and community engagement (CCE) in healthcare. During this process, we developed and refined a novel method of reviewing the literature: the scoping meta‐review (SMR). While the SMR combines and incorporates the benefits of two methodologies—scoping reviews and meta‐reviews—it has unique features that distinguish it from each. The differences between a scoping review and a systematic review, such as a meta‐review, are summarized in Table 1 . In this paper, we introduce the SMR, and suggest when and how to use it. We also discuss the benefits and limitations of this method.
Table 1.
Differences between a systematic review, a scoping review, and a scoping meta‐review (SMR)
| A systematic review, such as a meta‐review | A scoping review | A scoping meta‐review (SMR) | |
|---|---|---|---|
| Aim of the study | To address very specific research questions | To deal with broader topics | To deal with broader topics |
| Types of included studies | Limited to particular types of studies (meta‐reviews are limited to systematic reviews) | Might include different types of studies | Limited to systematic reviews |
| Quality appraisal of the included studies | Quality appraisal is done and results are presented based on the weight and quality of evidence | Does not evaluate quality of the included studies | Quality appraisal is done to explore the methodology of studies and excluding nonsystematic reviews |
| Analysis and reporting the results | Synthesizes evidence and aggregates findings | Undertakes thematic construction and creates a narrative to summarize the results | Undertakes thematic construction and creates a narrative to summarize the results |
| Study selection | Study selection is based on the inclusion and exclusion criteria that are set at the beginning of the study | Criteria are developed gradually while familiarity with literature is increased | Study selection is based on the inclusion and exclusion criteria that are set at the beginning of the study, but are open to revision |
| Comprehensive | Attempts to include all relevant materials | Attempts to be as comprehensive as possible, but also consider cost and time limitation; e.g., there might be a time deadline for inclusion of studies | Attempts to be as comprehensive as possible, but also consider practical limitations and may exclude older systematic reviews |
What is an SMR?
An SMR is a scoping evaluation of systematic reviews that incorporates the benefits of both scoping reviews and meta‐reviews. Similar to a scoping review, an SMR provides researchers with a flexible model by which to map a field and a way to summarize relevant research activities and results. Similar to a meta‐review, an SMR enables researchers to rely on the useful summaries and high‐quality evidence provided by systematic reviews. However, an SMR also has unique features. During a scoping review, researchers do not appraise the quality of the included papers and are open to include any types of papers.3, 4 During an SMR, however, only systematic reviews are included and the methodology of the included studies is appraised. This difference introduces several benefits of SMRs. First, the selection process in an SMR is guided and papers which are not systematic reviews are excluded; unlike in a scoping review where the process of decision making for inclusion and exclusion of papers is often not clear.4 Second, in cases where the topic under review is broad, limiting the search to systematic reviews can reduce the volume of papers for examination to a manageable size.
An SMR differs from a meta‐review in other ways. A meta‐review is a systematic overview of reviews, in which researchers attempt to include all available systematic reviews and to undertake rigorous appraisal on each included systematic review. In this aspect, however, an SMR is more similar to a scoping review—although rigorous attempts are made to include all relevant citations, it is still possible to make pragmatic decisions.3 The primary aim is to report findings of the included papers, and rigorous quality appraisal is not essential other than in relation to identifying systematic reviews and excluding other papers.
When to undertake an SMR
To undertake an SMR, there are two prerequisites; a desire to map a particular field of interest; and the availability of relevant systematic reviews. An SMR is helpful when a broad investigation of a topic is warranted, such as interdisciplinary studies. Since an SMR is relying only on systematic reviews, undertaking an SMR can be possible only if an adequate number of systematic reviews on a topic area is available.
How to undertake an SMR
Undertaking an SMR has stages similar to a scoping review as compared with conducting a systematic review. However, some specific stages are suggested for an SMR:
Undertaking a preliminary nonsystematic review
Building a search strategy
Interrogating academic literature databases
Classifying and excluding studies based on titles and abstracts
Saving the refined database of references
Revising the search strategy
Selecting and reviewing the full text papers
Thematically analyzing the selected texts and writing the report
The stages we undertook in completing an SMR in CCE are illustrated in Figure 1 . In September 2011, we designed a project to systematically review the literature on CCE in healthcare, and to assess the best available evidence in this area.7 The eight steps are presented and discussed in the following paragraphs.
Preliminary review
Figure 1.
Illustrating the stages for undertaking an SMR on a project on CCE in healthcare.7
We did not begin with focused research questions, our intention being to map the field and identify factors related to CCE such as strategies, barriers, and methods. We started with a nonsystematic preliminary review to gain a sense of the literature. This review and search indicated that the field was very diverse and complex. The complexity of the interrelated topics in CCE is illustrated by Figure 2 . We found that various overlapping terms such as consumer engagement, patient participation, and public involvement are used to describe a wide range of activities. Therefore, we concluded that before undertaking a systematic review, we needed to adopt a scoping review methodology to map the field. In addition, we noted that a considerable number of available papers presented lower levels of evidence, such as opinion pieces and letters to editors. This posed a challenge to mapping the field based on high levels of evidence. In addition, due to the diversity of the field to be explored, and the associated volume of publications, we needed a way to limit the size of citations we had to explore. Therefore, we developed the SMR method.
Figure 2.
Mind map of the topics related to the CCE in healthcare.7 This illustration was manually produced at the fourth stage of our SMR on consumer and community engagement in healthcare, and was helpful in making sense of this complex field.
Undertaking a preliminary review also helped us to explore the availability of systematic reviews related to the field to ascertain whether an SMR was feasible. In addition, through the preliminary review we identified key words and further developed the search strategy. The preliminary review can inform the choice of most suitable literature review method to meet the objective.
-
2.
Building a search strategy
Constructing the search strategy included addressing the following elements: defining aims; identifying research questions; selecting databases; compiling a list of key search terms; and establishing inclusion and exclusion criteria. Our broad aim was to conduct an SMR on CCE in healthcare in order to map the field and identify the gaps in current knowledge. We also acknowledged that undertaking a more focused systematic review was not feasible, and should be postponed until the conclusion of the SMR. We intended to address the following research questions:
In what areas of healthcare has CCE been studied so far?
What tools and methods are used to enhance CCE in healthcare?
What barriers exist for CCE in healthcare?
What roles could patients be expected to adopt during CCE?
Based on our preliminary review, we produced a comprehensive list of nine medical subject headings and 47 phrases (Tables 2 and 3 ). We also elected to search the following electronic databases: Pub med central; Embase; EBM reviews; CINAHL; APAPsycNET; and Scopus. Our inclusion criteria were that studies were related to CCE and had a systematic review methodology. We included studies on all healthcare clients and all health problems; and excluded nonsystematic review papers, opinions, books, chapters, discussions, and letters. We agreed to have no geographical restriction and, initially, no time limit, but restricted the search to English publications. In order to ensure transparency of our SMR, we presented the complete search strategy in the related publications.7, 8, 9, 10
Table 2.
Search terms relating to CCE
| SEARCH TERMS |
|---|
| Citizen deliberation, citizen engagement, citizen involvement, citizen participation, citizen representative |
| Client engagement, client involvement, client participation, client representative |
| Community collaboration, community engagement, community input, community involvement, community led, community participation, community representative |
| Consumer advocacy, consumer driven, consumer engagement, consumer generated, consumer groups, consumer involvement, consumer network, consumer participation |
| Lay control, lay involvement, lay member, lay network, lay participation, lay perception, lay perspective, lay representative, lay voice |
| Patient driven, patient engagement, patient involvement, patient led, patient participation, patient perspective, patient representative |
| User contribution, user controlled, user involvement, user led, user oriented, user participation, user representative |
Table 3.
Search terms relating to CCE—Medical Subject Headings
| Medical Subject Headings |
|---|
| Community–institutional relation, community networks, community role |
| Consumer networks, consumer participation |
| Health information networks |
| Participation: healthcare decisions |
| Patient involvement, patient participation |
-
3.
Interrogating academic literature databases
We searched the six electronic databases and, despite limiting the search to English publications and review papers, identified 7,034 references (after removing duplicates). We downloaded these references into a database using a reference manager software tool (Endnote X5). This large number of records illustrates the usefulness of the SMR: although evaluation of this number of records was time‐consuming and difficult, it was still possible. Had the search not been limited to reviews, the number of papers identified would have been unmanageable.
-
4.
Classifying and excluding studies based on titles and abstracts
After excluding 4,875 papers by examining titles and abstracts relevant to the topic and methodology, we built a database of 2,159 citations. This stage provided a good opportunity for classification of the titles, which helped map the field and produce a mind map (Figure 2 ). Therefore, as we explored the citations by title and abstract, we classified them into emerging groups. Although this process was time‐consuming, as demonstrated in the next stage, the benefits outweighed the efforts.
-
5.
Saving the refined database of references
The database was saved, for subsequent work in related topics of interest. After completion of the initial SMR, for example, this database was reinterrogated during an investigation into the literature for methods for evaluation and measurement of CCE. We conducted a meta‐review on evaluation methods, and obtained data by searching our saved database on CCE by relevant key words such as “evaluation,” “measurement,” “outcome,” and “tool.” Similarly, we used the database to undertake a meta‐review exploring the relationship of CCE and patient safety.8 Producing the database of references was therefore cost‐effective and contributed to future research outputs.
-
6.
Revising the search strategy
During the process of conducting a scoping review, study selection criteria can be developed iteratively.3 For an SMR, we suggest developing a search strategy at the beginning of the study, based on the preliminary review, and then revising it in the light of the search results. To illustrate, at this stage of our example SMR, we had unearthed 2,159 citations for interrogation. Managing this number of papers was impractical, so we revised our search strategy. As each systematic review examines published studies we limited our study to include only systematic reviews published in the previous 2 years. Thus we indirectly accessed studies published earlier than the 2‐year cutoff period as they were reviewed in the systematic reviews that we included. Depending on the number of the identified papers, other SMRs could include all identified citations, or they could opt to select the most up‐to‐date reviews, as we did.
-
7.
Selecting and reviewing the full text papers
After applying the new time‐based criterion, we excluded 1,993 papers published before 2010. There were 166 full text papers remaining. If we were undertaking a meta‐review, a comprehensive critical appraisal of 166 papers would be required. However, for our SMR we appraised only the methodology to exclude papers that were not systematic reviews (N = 49), leaving 117 systematic reviews for analysis. It is notable that in an SMR study it is necessary to have predetermined criteria to identify papers whose methodology is rigorous enough to be considered as a systematic review. Review papers that fail to satisfy the criteria, for example, do not report their review methodology or their methodology is not sufficiently rigorous will be excluded from the study. We selected our criteria based on an appraisal tool developed by the Public Health Resource Unit, England.11
-
8.
Thematically analyzing the selected texts and writing the report
Through examination of the results and findings of the 117 systematic reviews, we were able to provide a snapshot of the field. Based on the findings of the included studies, we were able to develop our research questions, such as: in which areas of healthcare CCE has been studied; which tools and methods are used; and what are the barriers to CCE? We recognized the complexity that exists in this field. This SMR enabled us to identify important factors that one should take into consideration when implementing CCE, and to produce a model to guide future implementation of interventions. This model is introduced in the report of our SMR on CCE, and is available online;7 in addition, we have presented our data in various conference presentations8, 9, 10 and we are preparing three papers based on this work.
Discussion
What are the benefits of SMR?
Since it relies on systematic reviews only, an SMR can present a robust and reliable picture of the field, when compared to a scoping review. A particular advantage of undertaking an SMR is that, by limiting the search to “reviews,” a more manageable corpus of records can be examined. In addition, during an SMR it is possible to select only the most recent publications, as the findings of relevant earlier studies will be incorporated within the systematic reviews that comprise these texts.
Collecting all of the systematic reviews available within a proposed field, albeit only recent publications selected for a particular SMR, provides a useful comprehensive database for additional and more specific searches of the field. Such a database is a collection of systematic reviews that are broadly related to a field and can be readily searched for more specific studies.
What are the limitations of SMRs?
In an SMR, nonsystematic papers are excluded, and this can include recent studies that are not yet reflected in systematic reviews. A further limitation is that nonacademic works are not included. This is also a limitation of systematic reviews and meta‐reviews. However, it is notable that during a scoping review it is possible to include nonacademic and nonpeer reviewed literature. In mitigation, we suggest using the preliminary literature review stage to map nonacademic works typically found in the grey literature. This overview of nonacademic activities can be matched and compared with the “academic picture” that is depicted by the overview of systematic reviews. A final limitation is that, although SMRs provide the most reliable evidence on a field, they are dependent on the availability of systematic reviews in the area under investigation.
Conclusion
The main benefit of an SMR is to map a new field based on high‐level evidence provided by systematic reviews. Perhaps in the past, undertaking an SMR was infeasible; however, due to the availability of systematic reviews and ease of access to electronic databases, this is now possible, and in many cases, may be thought desirable. Comparison and synthesis of findings of systematic reviews is a rewarding process, uncovering valuable information in the literature. In presenting this method, we hope that other researchers can utilize and benefit from SMR in their own fields. We also welcome contributions to its further development.
Conflict of Interest
The authors declare that they have no competing interests.
Acknowledgments
The method of scoping meta‐review was developed during the Community Engagement Research Project (CERP), which was funded by the Agency for Clinical Innovation (ACI, Sydney, Australia). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
References
- 1. Higgins J, Green S, Churchill R, Cumpston M, Chandler J. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011] Available from www.cochrane‐handbook.org. The Cochrane Collaboration. 2011. [Google Scholar]
- 2. Sarrami Foroushani P, Schneider J, Assareh N. Meta‐review of the effectiveness of computerised CBT in treating depression. BMC Psychiatry. 2011; 11(1): 131. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005; 8(1): 19–32. [Google Scholar]
- 4. Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010; 5: 69. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Schloman B. Mapping the literature of allied health: project overview. Bull Med Libr Assoc. 1997; 85(3): 271. [PMC free article] [PubMed] [Google Scholar]
- 6. Ding Y, Chowdhury GG, Foo S. Bibliometric cartography of information retrieval research by using co‐word analysis. Inf Process Manage. 2001; 37(6): 817–842. [Google Scholar]
- 7. Sarrami Foroushani P, Travaglia JF, Eikli M, Braithwaite J. Consumer and community engagement: A Review of the Literature Sydney: University of New South Wales, Centre for Clinical Governance Research, Australian Institute of Health Innovation 2012. [Google Scholar]
- 8. Sarrami Foroushani P, Travaglia J, Debono D, Braithwaite J. A new framework for evaluating consumer and community engagement in health care systems: results of a meta‐review. Paper presented at: ISQua 30th International Conference: Quality and Safety in Population Health and Healthcare; 13–16 October, 2013; Edinburgh, Scotland. [Google Scholar]
- 9. Sarrami Foroushani P, Travaglia JF, Eikli M, Braithwaite J. What does it mean to engage consumers and communities in health care? A scoping meta‐review. Network to Network – The Inaugural Australasian Clinical Networks Conference; November, 2012; Sydney, Australia.
- 10. Sarrami Foroushani P, Travaglia JF, Eikli M, Braithwaite J. Improving patient safety via consumer and community engagement: a systematic review. The 10th Australasian Conference on Safety and Quality in Health Care; 3–5 September, 2012; Cairns, Australia. [Google Scholar]
- 11. THE PUBLIC HEALTH RESOURCE UNIT. Critical Appraisal Skills Programme Tools. OXFORD: available at http://www.casp‐uk.net/#!casp‐tools‐checklists/c18f8 accessed June 10, 2011.