Rapid reviews methods series: guidance on rapid qualitative evidence synthesis

Abstract

This paper forms part of a series of methodological guidance from the Cochrane Rapid Reviews Methods Group and addresses rapid qualitative evidence syntheses (QESs), which use modified systematic, transparent and reproducible methodsu to accelerate the synthesis of qualitative evidence when faced with resource constraints. This guidance covers the review process as it relates to synthesis of qualitative research. ‘Rapid’ or ‘resource-constrained’ QES require use of templates and targeted knowledge user involvement. Clear definition of perspectives and decisions on indirect evidence, sampling and use of existing QES help in targeting eligibility criteria. Involvement of an information specialist, especially in prioritising databases, targeting grey literature and planning supplemental searches, can prove invaluable. Use of templates and frameworks in study selection and data extraction can be accompanied by quality assurance procedures targeting areas of likely weakness. Current Cochrane guidance informs selection of tools for quality assessment and of synthesis method. Thematic and framework synthesis facilitate efficient synthesis of large numbers of studies or plentiful data. Finally, judicious use of Grading of Recommendations Assessment, Development and Evaluation approach for assessing the Confidence of Evidence from Reviews of Qualitative research assessments and of software as appropriate help to achieve a timely and useful review product.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Rapid Qualitative Evidence Synthesis (QES) is a relatively recent innovation in evidence synthesis and few published examples currently exists.

• Guidance for authoring a rapid QES is scattered and requires compilation and summary.

WHAT THIS STUDY ADDS

• This paper represents the first attempt to compile current guidance, illustrated by the experience of several international review teams.

• We identify features of rapid QES methods that could be accelerated or abbreviated and where methods resemble those for conventional QESs.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• This paper offers guidance for researchers when conducting a rapid QES and informs commissioners of research and policy-makers what to expect when commissioning such a review.

Introduction

This paper forms part of a series from the Cochrane Rapid Reviews Methods Group providing methodological guidance for rapid reviews. While other papers in the series^1–4 focus on generic considerations, we aim to provide in-depth recommendations specific to a resource-constrained (or rapid) qualitative evidence synthesis (rQES).⁵ This paper is accompanied by recommended resources (online supplemental appendix A) and an elaboration with practical considerations (online supplemental appendix B).

The role of qualitative evidence in decision-making is increasingly recognised.⁶ This, in turn, has led to appreciation of the value of qualitative evidence syntheses (QESs) that summarise findings across multiple contexts.⁷ Recognition of the need for such syntheses to be available at the time most useful to decision-making has, in turn, driven demand for rapid qualitative evidence syntheses.⁸ The breadth of potential rQES mirrors the versatility of QES in general (from focused questions to broad overviews) and outputs range from descriptive thematic maps through to theory-informed syntheses (see table 1).

Table 1.

Glossary of important terms (alphabetically)

Term	Definition
Framework synthesis	Applies primary research methods to map and interpret patterns of meaning (or ‘themes’) from multiple qualitative studies against a prespecified framework or theory
Mega-aggregation	Overview that applies methods that are more commonly used to aggregate (group together) findings from multiple primary qualitative studies to produce a synthesis of multiple qualitative evidence syntheses
Mega-ethnography	Overview that applies methods that are more commonly used to construct interpretations of findings (a line of argument) from multiple primary qualitative studies to produce a synthesis of multiple qualitative evidence syntheses
Meta-ethnography	Interpretive approach that synthesises qualitative research to ‘translate’ qualitative studies into one another and then bring the separate parts together to form a whole line of argument that is greater than the sum of its parts
Qualitative evidence synthesis (QES)	Umbrella term for process by which reviewers systematically review and synthesise data from individual qualitative studies on a shared topic of interest to create new understanding by comparing and analysing concepts and findings
Rapid qualitative evidence synthesis (rQES)	Synthesis product that employs accelerated and/or streamlined methods, to synthesise primary qualitative research data so that completion of the review output occurs earlier than with a typical qualitative evidence synthesis (QES)
Reflexivity	Ongoing, mutually shaping interaction between the researcher/reviewer and the research whereby primary research authors consider their relationship with the research context/participants and reviewers reflect on their relationship to the review topic and the studies under consideration
Thematic synthesis	Applies primary research methods to identify, analyse and interpret patterns of meaning (or ‘themes’) across multiple qualitative studies

As with other resource-constrained reviews, no one size fits all. A team should start by specifying the phenomenon of interest, the review question,⁹ the perspectives to be included⁹ and the sample to be determined and selected.¹⁰ Subsequently, the team must finalise the appropriate choice of synthesis.¹¹ Above all, the review team should consider the intended knowledge users,³ including requirements of the funder.

An rQES team, in particular, cannot afford any extra time or resource requirements that might arise from either a misunderstanding of the review question, an unclear picture of user requirements or an inappropriate choice of methods. The team seeks to align the review question and the requirements of the knowledge user with available time and resources. They also need to ensure that the choice of data and choice of synthesis are appropriate to the intended ‘knowledge claims’ (epistemology) made by the rQES.¹¹ This involves the team asking ‘what types of data are meaningful for this review question?’, ‘what types of data are trustworthy?’ and ‘is the favoured synthesis method appropriate for this type of data?’.¹² This paper aims to help rQES teams to choose methods that best fit their project while understanding the limitations of those choices. Our recommendations derive from current QES guidance,⁵ evidence on modified QES methods,^{8 13} and practical experience.^{14 15}

This paper presents an overview of considerations and recommendations as described in table 2. Supplemental materials including additional resources details of our recommendations and practical examples are provided in online supplemental appendices A and B.

Table 2.

Recommendations for resource-constrained qualitative evidence synthesis (rQES)

Recommendation number	Item
	Setting the review question and topic refinement
R1	Ensure involvement of knowledge users, even when the QES is abbreviated or accelerated; especially when setting the review question and refining the topic, to ensure key perspectives are included
R2	Use templates to fast-track writing of a protocol. The protocol should always be publicly available and should be registered if the rQES timescales permit
	Setting eligibility criteria
	Together with knowledge users
R3	Clearly define the included perspectives. A rapid QES (rQES) may need to limit the number of perspectives, with a focus on those most important for decision-making
R4	Define if ‘indirect evidence’ is to be used in the absence of direct evidence. An rQES may focus on direct evidence, except when only indirect evidence is available
R5	Consider privileging rich qualitative studies; consider a stepwise approach to inclusion of qualitative data and explore the possibility of sampling
R6	Consider including multiple QES within a mega--synthesis
	Searching
R7	Involve an information specialist (eg, librarian) in prioritising sources and search methods
R8	Consider limiting database searching to two or three multidisciplinary databases and, if resources allow, searches of one or two specialised (subject or regional) databases
R9	Even when resources are limited, consider factoring in time for peer review of at least one search strategy
R10	Selectively target appropriate types of grey literature and supplemental searches, including citation chaining, especially for diffuse topics
	Study selection
	Title and abstract screening/full-text screening
R11	Use pre-prepared, pretested templates to limit the scale of piloting, calibration and testing
R12	Target and prioritise identified risks of either over-zealous inclusion or over-exclusion specific to each rQES
R13	Focus quality control procedures on specific threats (eg, use additional reviewers and report percentages for double screening)
	Data extraction
R14	Use a single reviewer to extract data using a piloted template, with a second reviewer for checking, or code data directly from full-text articles, again with checking. Limit data extraction to minimal essential items. Consider re-using data extracted from primary studies included in previous QESs
	Assessment of methodological limitations
R15	In the absence of validated risk of bias tools for qualitative studies, choose a tool according to CQIMG guidance together with expediency
R16	Use a single reviewer to assess methodological limitations, with verification of judgements (and support statements) by a second reviewer
	Synthesis
R17	Favour descriptive thematic synthesis or framework synthesis, except when theory generation (meta-ethnography or analytical thematic synthesis) is a priority
R18	Consider whether a conceptual model, theory or framework offers a rapid way to organise/code/interpret/present findings
R19	Target GRADE-CERQual assessments at findings most critical to decision-making. Additional reviewers could verify all, or a sample of, assessments. Consider reusing GRADE-CERQual assessments if findings are relevant and of demonstrable high quality
	Additional considerations
R20	Use review management software or qualitative analysis management software to streamline the process

CQIMG, Cochrane Qualitative and Implementation Methods Group; GRADE-CERQual, Grading of Recommendations Assessment, Development and Evaluation approach for assessing the Confidence of Evidence from Reviews of Qualitative research; QES, qualitative evidence synthesis.

Setting the review question and topic refinement

Rapid reviews summarise information from multiple research studies to produce evidence for ‘the public, researchers, policymakers and funders in a systematic, resource-efficient manner’.¹⁶ Involvement of knowledge users is critical.³ Given time constraints, individual knowledge users could be asked only to feedback on very specific decisions and tasks or on selective sections of the protocol. Specifically, whenever a QES is abbreviated or accelerated, a team should ensure that the review question is agreed by a minimum number of knowledge users with expertise or experience that reflects all the important review perspectives and with authority to approve the final version^{2 5 11} (table 2, item R1).

Involvement of topic experts can ensure that the rQES is responsive to need.^{14 17} One Cochrane rQES saved considerable time by agreeing the review topic within a single meeting and one-phase iteration.⁹ Decisions on topics to be omitted are also informed by a knowledge of existing QESs.¹⁷

An information specialist can help to manage the quantity and quality of available evidence by setting conceptual boundaries and logistic limits. A structured question format, such as Setting-Perspective-Interest, phenomenon of-Comparison-Evaluation or Population-Interest, phenomenon of-Context helps in communicating the scope and, subsequently, in operationalising study selection.^{9 18}

Scoping (of review parameters) and mapping (of key types of evidence and likely richness of data) helps when planning the review.^{5 19} The option to choose purposive sampling over comprehensive sampling approaches, as offered by standard QES, may be particularly helpful in the context of a rapid QES.⁸ Once a team knows the approximate number and distribution of studies, perhaps mapping them against country, age, ethnicity, etc), they can decide whether or not to use purposive sampling.¹² An rQES for the WHO combined purposive with variation sampling. Sampling in two stages started by reducing the initial number of studies to a more manageable sampling frame and then sampling approximately a third of the remaining studies from within the sampling frame.²⁰

Sampling may target richer studies and/or privilege diversity.^{8 21} A rich qualitative study typically illustrates findings with verbatim extracts from transcripts from interviews or textual responses from questionnaires. Rich studies are often found in specialist qualitative research or social science journals. In contrast, less rich studies may itemise themes with an occasional indicative text extract and tend to summarise findings. In clinical or biomedical journals less rich findings may be placed within a single table or box.

No rule exists on an optimal number of studies; too many studies makes it challenging to ‘maintain insight’,²² too few does not sustain rigorous analysis.²³ Guidance on sampling is available from the forthcoming Cochrane-Campbell QES Handbook.

A review team can use templates to fast-track writing of a protocol. The protocol should always be publicly available (table 2, item R2).^{24 25} Formal registration may require that the team has not commenced data extraction but should be considered if it does not compromise the rQES timeframe. Time pressures may require that methods are left suitably flexible to allow well-justified changes to be made as a detailed picture of the studies and data emerge.²⁶ The first Cochrane rQES drew heavily on text from a joint protocol/review template previously produced within Cochrane.²⁴

Setting eligibility criteria

An rQES team may need to limit the number of perspectives, focusing on those most important for decision-making^{5 9 27} (table 2, item R3). Beyond the patients/clients each additional perspective (eg, family members, health professionals, other professionals, etc) multiplies the additional effort involved.

A rapid QES may require strict date and setting restrictions¹⁷ and language restrictions that accommodate the specific requirements of the review. Specifically, the team should consider whether changes in context over time or substantive differences between geographical regions could be used to justify a narrower date range or a limited coverage of countries and/or languages. The team should also decide if ‘indirect evidence’ is to substitute for the absence of direct evidence. An rQES typically focuses on direct evidence, except when only indirect evidence is available²⁸ (table 2, item R4). Decisions on relevance are challenging—precautions for swine influenza may inform precautions for bird influenza.²⁸ A smoking ban may operate similarly to seat belt legislation, etc. A review team should identify where such shared mechanisms might operate.²⁸ An rQES team must also decide whether to use frameworks or models to focus the review. Theories may be unearthed within the topic search or be already known to team members, fro example, Theory of Planned Behaviour.²⁹

Options for managing the quantity and quality of studies and data emerge during the scoping (see above). In summary, the review team should consider privileging rich qualitative studies²; consider a stepwise approach to inclusion of qualitative data and explore the possibility of sampling (table 2, item R5). For example, where data is plentiful an rQES may be limited to qualitative research and/or to mixed methods studies. Where data is less plentiful then surveys or other qualitative data sources may need to be included. Where plentiful reviews already exist, a team may decide to conduct a review of reviews⁵ by including multiple QES within a mega-synthesis^{28 29} (table 2, item R6).

Searching

Searching for QES merits its own guidance,^{21–23 30} this section reinforces important considerations from guidance specific to qualitative research. Generic guidance for rapid reviews in this series broadly applies to rapid QESs.¹

In addition to journal articles, by far the most plentiful source, qualitative research is found in book chapters, theses and in published and unpublished reports.²¹ Searches to support an rQES can (a) limit the number of databases searched, deliberately selecting databases from diverse disciplines, (b) use abbreviated study filters to retrieve qualitative designs and (c) employ high yield complementary methods (eg, reference checking, citation searching and Related Articles features). An information specialist (eg, librarian) should be involved in prioritising sources and search methods (table 2, item R7).^{11 14}

According to empirical evidence optimal database combinations include Scopus plus CINAHL or Scopus plus ProQuest Dissertations and Theses Global (two-database combinations) and Scopus plus CINAHL plus ProQuest Dissertations and Theses Global (three-database combination) with both choices retrieving between 89% and 92% of relevant studies.³⁰

If resources allow, searches should include one or two specialised databases (table 2, item R8) from different disciplines or contexts²¹ (eg, social science databases, specialist discipline databases or regional or institutional repositories). Even when resources are limited, the information specialist should factor in time for peer review of at least one search strategy (table 2, item R9).³¹ Searches for ‘grey literature’ should selectively target appropriate types of grey literature (such as theses or process evaluations) and supplemental searches, including citation chaining or Related Articles features (table 2, item R10).³² The first Cochrane rQES reported that searching reference lists of key papers yielded an extra 30 candidate papers for review. However, the team documented exclusion of grey literature as a limitation of their review.¹⁵

Study selection

Consistency in study selection is achieved by using templates, by gaining a shared team understanding of the audience and purpose, and by ongoing communication within, and beyond, the team.^{2 33} Individuals may work in parallel on the same task, as in the first Cochrane rQES, or follow a ‘segmented’ approach where each reviewer is allocated a different task.¹⁴ The use of machine learning in the specific context of rQES remains experimental. However, the possibility of developing qualitative study classifiers comparable to those for randomised controlled trials offers an achievable aspiration.³⁴

Title and abstract screening

The entire screening team should use pre-prepared, pretested title and abstract templates to limit the scale of piloting, calibration and testing (table 2, item R11).^{1 14} The first Cochrane rQES team double-screened titles and abstracts within Covidence review software.¹⁴ Disagreements were resolved with reference to a third reviewer achieving a shared understanding of the eligibility criteria and enhancing familiarity with target studies and insight from data.¹⁴ The team should target and prioritise identified risks of either over-zealous inclusion or over-exclusion specific to each rQES (table 2, item R12).¹⁴ The team should maximise opportunities to capture divergent views and perspectives within study findings.³⁵

Full-text screening

Full-text screening similarly benefits from using a pre-prepared pretested standardised template where possible^{1 14} (table 2, item R11). If a single reviewer undertakes full-text screening,⁸ the team should identify likely risks to trustworthiness of findings and focus quality control procedures (eg, use of additional reviewers and percentages for double screening) on specific threats¹⁴ (table 2, item R13). The Cochrane rQES team opted for double screening to assist their immersion within the topic.¹⁴

Data extraction

Data extraction of descriptive/contextual data may be facilitated by review management software (eg, EPPI-Reviewer) or home-made approaches using Google Forms, or other survey software.³⁶ Where extraction of qualitative findings requires line-by-line coding with multiple iterations of the data then a qualitative data management analysis package, such as QSR NVivo, reaps dividends.³⁶ The team must decide if, collectively, they favour extracting data to a template or coding direct within an electronic version of an article.

Quality control must be fit for purpose but not excessive. Published examples typically use a single reviewer for data extraction⁸ with use of two independent reviewers being the exception. The team could limit data extraction to minimal essential items. They may also consider re-using descriptive details and findings previously extracted within previous well-conducted QES (table 2, item R14). A pre-existing framework, where readily identified, may help to structure the data extraction template.^{15 37} The same framework may be used to present the findings. Some organisations may specify a preferred framework, such as an evidence-to-decision-making framework.³⁸

Assessment of methodological limitations

The QES community assess ‘methodological limitations’ rather than use ‘risk of bias’ terminology. An rQES team should pick an approach appropriate to their specific review. For example, a thematic map may not require assessment of individual studies—a brief statement of the generic limitations of the set of studies may be sufficient. However, for any synthesis that underpins practice recommendations³⁹ assessment of included studies is integral to the credibility of findings. In any decision-making context that involves recommendations or guidelines, an assessment of methodological limitations is mandatory.^{40 41}

Each review team should work with knowledge users to determine a review-specific approach to quality assessment.²⁷ While ‘traffic lights’, similar to the outputs from the Cochrane Risk of Bias tool, may facilitate rapid interpretation, accompanying textual notes are invaluable in highlighting specific areas for concern. In particular, the rQES team should demonstrate that they are aware (a) that research designs for qualitative research seek to elicit divergent views, rather than control for variation; (b) that, for qualitative research, the selection of the sample is far more informative than the size of the sample; and (c) that researchers from primary research, and equally reviewers for the qualitative synthesis, need to be thoughtful and reflexive about their possible influences on interpretation of either the primary data or the synthesised findings.

Selection of checklist

Numerous scales and checklists exist for assessing the quality of qualitative studies. In the absence of validated risk of bias tools for qualitative studies, the team should choose a tool according to Cochrane Qualitative and Implementation Methods Group (CQIMG) guidance together with expediency (according to ease of use, prior familiarity, etc) (table 2, item R15).⁴¹ In comparison to the Critical Appraisal Skills Programme checklist which was never designed for use in synthesis,⁴² the Cochrane qualitative tool is similarly easy to use and was designed for QES use. Work is underway to identify an assessment process that is compatible with QESs that support decision-making.⁴¹ For now the choice of a checklist remains determined by interim Cochrane guidance and, beyond this, by personal preference and experience. For an rQES a team could use a single reviewer to assess methodological limitations, with verification of judgements (and support statements) by a second reviewer (table 2, item R16).

Synthesis

The CQIMG endorses three types of synthesis; thematic synthesis, framework synthesis and meta-ethnography (box 1).^{43 44} Rapid QES favour descriptive thematic synthesis⁴⁵ or framework synthesis,^{46 47} except when theory generation (meta-ethnography^{48 49} or analytical thematic synthesis) is a priority (table 2, item R17).

Box 1. Choosing a method for rapid qualitative synthesis.

Thematic synthesis: first choice method for rQES.⁴⁵ For example, in their rapid QES Crooks and colleagues⁴⁴ used a thematic synthesis to understand the experiences of both academic and lived experience coresearchers within palliative and end of life research.⁴⁵

Framework synthesis: alternative where a suitable framework can be speedily identified.⁴⁶ For example, Bright and colleagues⁴⁶ considered ‘best-fit framework synthesis’ as appropriate for mapping study findings to an ‘a priori framework of dimensions measured by prenatal maternal anxiety tools’ within their ‘streamlined and time-limited evidence review’.⁴⁷

Less commonly, an adapted meta-ethnographical approach was used for an implementation model of social distancing where supportive data (29 studies) was plentiful.⁴⁸ However, this QES demonstrates several features that subsequently challenge its original identification as ‘rapid’.⁴⁹

Abbrevations: QES, qualitative evidence synthesis; rQES, resource-constrained qualitative evidence synthesis.

The team should consider whether a conceptual model, theory or framework offers a rapid way for organising, coding, interpreting and presenting findings (table 2, item R18). If the extracted data appears rich enough to sustain further interpretation, data from a thematic or framework synthesis can subsequently be explored within a subsequent meta-ethnography.⁴³ However, this requires a team with substantial interpretative expertise.¹¹

Assessments of confidence in the evidence⁴ are central to any rQES that seeks to support decision-making and the QES-specific Grading of Recommendations Assessment, Development and Evaluation approach for assessing the Confidence of Evidence from Reviews of Qualitative research (GRADE-CERQual) approach is designed to assess confidence in qualitative evidence.⁵⁰ This can be performed by a single reviewer, confirmed by a second reviewer.²⁶ Additional reviewers could verify all, or a sample of, assessments. For a rapid assessment a team must prioritise findings, using objective criteria; a WHO rQES focused only on the three ‘highly synthesised findings’.²⁰ The team could consider reusing GRADE-CERQual assessments from published QESs if findings are relevant and of demonstrable high quality (table 2, item R19).⁵⁰ No rapid approach to full application of GRADE-CERQual currently exists.

Reporting and record management

Little is written on optimal use of technology.⁸ A rapid review is not a good time to learn review management software or qualitative analysis management software. Using such software for all general QES processes (table 2, item R20), and then harnessing these skills and tools when specifically under resource pressures, is a sounder strategy. Good file labelling and folder management and a ‘develop once, re-use multi-times’ approach facilitates resource savings.

Reporting requirements include the meta-ethnography reporting guidance (eMERGe)⁵¹ and the Enhancing transparency in reporting the synthesis of qualitative research (ENTREQ) statement.⁵² An rQES should describe limitations and their implications for confidence in the evidence even more thoroughly than a regular QES; detailing the consequences of fast-tracking, streamlining or of omitting processes all together.⁸ Time spent documenting reflexivity is similarly important.²⁷ If QES methodology is to remain credible rapid approaches must be applied with insight and documented with circumspection.^{53 54} (56)

Data availability statement

No data are available. Not applicable. All data is from published articles.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

Not applicable.