Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Jan 1.
Published in final edited form as: J Adv Nurs. 2012 May 17;69(1):194–204. doi: 10.1111/j.1365-2648.2012.06037.x

Literature Search Strategies for Conducting Knowledge-building and Theory-generating Qualitative Systematic Reviews: Discussion Paper

Deborah Finfgeld-Connett 1,✉, E Diane Johnson 2
PMCID: PMC3424349  NIHMSID: NIHMS373598  PMID: 22591030

Abstract

Aim

This paper is a report of literature search strategies for the purpose of conducting knowledge-building and theory-generating qualitative systematic reviews.

Background

Qualitative systematic reviews lie on a continuum from knowledge-building and theory-generating to aggregating and summarizing. Different types of literature searches are needed to optimally support these dissimilar reviews.

Data Sources

Articles published between 1989 - Autumn 2011. These documents were identified using a hermeneutic approach and multiple literature search strategies.

Discussion

Redundancy is not the sole measure of validity when conducting knowledge-building and theory-generating systematic reviews. When conducting these types of reviews, literature searches should be consistent with the goal of fully explicating concepts and the interrelationships among them. To accomplish this objective, a berry picking approach is recommended along with strategies for overcoming barriers to finding qualitative research reports.

Implications

To enhance integrity of knowledge-building and theory-generating systematic reviews, reviewers are urged to make literature search processes as transparent as possible, despite their complexity. This includes fully explaining and rationalizing what databases were used and how they were searched. It also means describing how literature tracking was conducted and grey literature was searched. In the end, the decision to cease searching also needs to be fully explained and rationalized.

Conclusion

Predetermined linear search strategies are unlikely to generate search results that are adequate for purposes of conducting knowledge-building and theory-generating qualitative systematic reviews. Instead, it is recommended that iterative search strategies take shape as reviews evolve.

Keywords: Qualitative systematic review, literature search, sample, theory development, knowledge development, validity, nurse

Introduction

Qualitative research is an established form of inquiry and findings from this type of research complement those generated from quantitative approaches. To date, however, many qualitative research findings remain isolated and unavailable for use in practice and policy formation since they have not been rigorously and systematically reviewed (Finfgeld 2003, Finfgeld-Connett 2010b). As qualitative research findings are incorporated into substantive bodies of knowledge through systematic review, questions that pertain to patient behaviors and perspectives will be more fully answered (Booth 2010b, Gorecki et al. 2010). Unfortunately, an impediment to this occurring is a lack of clarity regarding how data collection should be carried out (Booth 2006, Booth 2010a).

Background

Qualitative systematic review methods are relatively new and they continue to evolve. A conservative count of these methods has the number at seven (Hannes & Lockwood 2011), which does not take into consideration mixed-method approaches (Grant & Booth 2009). Systematic review methods that are considered qualitative include meta-aggregation, meta-ethnography, critical interpretive synthesis, thematic synthesis, meta-study, content analysis, grounded theory and others (Grant & Booth 2009, Hannes & Lockwood 2011, Ring et al. 2011). The purposes of these qualitative approaches range from summation and aggregation of findings to knowledge development and theory generation (Hannes & Lockwood 2011, Ring et al. 2011). Among the latter are meta-synthesis methods that are based on the traditions of meta-ethnography (Noblit & Hare 1988) and grounded theory (e.g., Finfgeld-Connett 2010a).

Unlike summative and aggregative approaches to qualitative systematic review where data analysis tends to be linear and the goal is declarative statements or directives for action (e.g., Sandelowski & Barroso 2007), knowledge-building and theory-generating approaches lend themselves more to iterative data analysis and the goal is concept or theory development (e.g., Finfgeld-Connett 2010a). Using knowledge-building and theory-generating approaches, implications for practice and policy formation may be inferred from resultant concepts or theories instead of being directly extracted from summed or aggregated data (Hannes & Lockwood 2011). Although the line between summative/aggregative approaches and knowledge-building/theory-generating approaches may not always be clear cut (Sandelowski et al. 2007), the latter is the focus of this article.

Since the overall field of qualitative systematic review is still evolving, questions remain regarding optimal research approaches. In particular, uncertainty surrounds the objectives of the data collection process as well as the most optimal methods for searching the literature (Booth 2010a). In the case of qualitative systematic reviews, existing research reports comprise the sample and the findings in those reports constitute the data for analysis (Finfgeld 2003). The specific question under consideration here is, how can literature search methods be optimized to maximize trustworthiness (i.e., validity) of knowledge-building and theory-generating qualitative systematic reviews? This is in contrast to optimal search strategies that are recommended when aggregation or summarization of qualitative findings is the primary objective.

Data Sources

A hermeneutic approach (Boell & Cecez-Kecmanovic 2010) was used to assemble information for this discussion. Based on this perspective, literature searches are seen as open-ended iterative processes where the topic or research question of interest is honed over time as the nature of the evidence becomes more apparent. Typically, a small number of relevant documents are initially identified for review and searching proceeds from these essential reports.

The literature search for this project began by examining key articles (e.g., Barroso et al. 2003, Flemming & Briggs 2007) that the first author (DFC) had acquired over the course of conducting several qualitative systematic reviews. Based on these foundational articles, related documents were subsequently located using the electronic citation and author tracking functions that are available in the electronic database Scopus (2011), an Elsevier search tool.

In addition, a query was conducted using Scopus (2011) and key concepts such as qualitative review, literature search and method(s) for the years 2000 through mid May 2011. This search resulted in 572 citations, which were reviewed for topical relevance. In the end, a majority of these references were excluded because they were not directly related to optimal literature search strategies in the context of qualitative systematic reviews. Due to recent changes in the areas of qualitative systematic reviews and information technology, documents that were published prior to 2000 were automatically excluded from consideration with the exception of a few seminal classics (e.g., Lincoln & Guba 1985, Noblit & Hare 1988, Bates 1989).

Scopus (2011) was selected for use in conjunction with this literature search because international journals from the health and medical sciences are comprehensively indexed in this system. It was also the preferred database because it includes references from the computing and information sciences. Other attractive features were the electronic citation tracking and reference checking features.

To ensure that essential information was reviewed for purposes of developing an unbiased discussion, a second database was used to search the literature in October 2011. Of the databases available at a research intensive health sciences university, Cumulative Index to Nursing and Allied Health Literature (CINAHL, 2011) was determined to be the most appropriate. CINAHL, which is owned by EBSCO, includes references from nursing and allied health literature. The allied health fields that are covered include health information management, health sciences librarianship and information science. CINAHL features subject headings for qualitative research designs as well as for literature searching and systematic reviews. CINAHL also follows an indexing convention whereby major subject headings are used to denote articles about a study design while minor subject headings are used primarily to denote reports of studies that use a given design.

The second author (EDJ), a professional reference librarian, developed three strategies (see Table 1) to search the literature using CINAHL, but no additional articles were identified that substantively added to this discussion. The lack of additional documents is thought to relate to the fact that up until recently, the topic under discussion (i.e., searching for qualitative research reports to conduct knowledge-building and theory-generating qualitative systematic reviews) was not a part of the ‘dominant discourse’ (Mackay 2007, p. 235) in the health related disciplines. In the end, literature searching continued until references were redundantly identified and no new ones were apparent.

Table 1.

CINAHL Search Strategies and Results: 2000-October 2011, English Language

Search Strategies Items (prior to merging and removing duplicates)
Systematic review, literature review or meta-analysis as major concepts AND qualitative subject headings or textwords 118
Systematic review as a major concept AND databases/literature searching/information retrieval subject heading explosions 286
Systematic review, literature review or meta-analysis or narrow textwords for qualitative systematic reviews (e.g. meta- synthesis) AND literature searching or information retrieval subject heading explosions with attached subheadings: methods, evaluation, standards 89
Open in a new tab

Data Extraction and Analysis

Key ideas and research findings were retrieved from documents that were primarily published from 2002 – 2011. These data were extracted from each document and placed into an electronically-generated table. Information in the table was reflectively studied, annotated and subsequently organized into a detailed outline. In correspondence with the emergent outline, descriptive memos were written and revised in an iterative manner until ideas and arguments were fully articulated and supported by the literature.

Discussion

Background Considerations

Prior to conducting a knowledge-building or theory-generating qualitative systematic review, it is important to reflect on the overall nature of qualitative investigations. Generally speaking, singular qualitative investigations are largely driven by research questions that relate to context. As such, each qualitative study is carefully sculpted throughout the processes of problem identification, data collection and analysis by the contextual attributes of the topic under investigation. Included among these attributes are factors associated with gender, culture, class, geography and so forth. In the end, the intended result is qualitative findings that are firmly grounded in the perceptions and circumstances of the research participants. It is these context-rich findings that are used to conduct knowledge-building and theory-generating qualitative systematic reviews (Corbin & Strauss 2008).

In turn, the job of qualitative systematic reviewers is to identify a topic and an accompanying body of research reports that justify and sustain a review. This means that there must be enough topically-specific and contextually-rich reports of qualitative research to fully support analysis, interpretation and synthesis across studies. In some instances, there may simply be too few studies to sustain rigorous analysis. In other circumstances, there might be several topically-relevant reports available, but they may lack enough thick description to fully develop concepts and the interrelationships among them. In either case, the threat to trustworthiness is thin unsaturated findings (Finfgeld 2003, Corbin & Strauss 2008).

Knowledge-building and theory-generating qualitative systematic reviewers may also find themselves in the uncomfortable position of having an unwieldy number of research reports to efficiently and effectively analyze using qualitative methods. This problem is likely to occur when research topics and questions are too broad and/or poorly defined. In this situation, resulting findings are apt to be relatively meaningless (Noblit & Hare 1988) and a primary threat to trustworthiness is lack of fittingness (Lincoln & Guba 1985, Morse & Singleton 2001, Corbin & Strauss 2008).

In summary, there appear to be two main threats to trustworthiness when data are being collected for purposes of conducting knowledge-building or theory-generating qualitative systematic reviews. The first involves an unwieldy collection of study reports that are not well linked by expertly-crafted research topics and questions. In this situation, the solution may be to re-craft the original research topic and questions and hone the sample so that it aligns accordingly. Conversely, the second threat relates to having too few studies or too few descriptively rich studies to render saturated findings. The remainder of this article and the associated key points that appear in Table 2 relate primarily to the latter situation.

Table 2.

Key Points and Basis for Inference

Key Point Basis for Inference
Exhaustive searches are recommended when conducting summative and aggregative systematic reviews Booth 2006, 2010a, Boell & Cecez- Kecmanovic 2010, Hannes & Lockwood 2011
Expansive searches are recommended when conducting knowledge-building and theory-generating systematic reviews Boell & Cecez-Kecmanovic 2010, Hannes & Lockwood 2011
Searching the literature for the purpose of conducting a knowledge-building and theory-generating systematic review involves an iterative approach Mackay 2007, Boell & Cecez-Kecmanovic 2010, Hannes & Lockwood 2011
Barriers to searching for qualitative research reports include:

  • Bias toward quantitative research and publication of resultant reports

  • Non-optimal indexing of qualitative studies and lack of informative manuscript titles and abstracts
Methods for overcoming barriers to searching for qualitative research reports include:

  • Berry picking strategies

    • Electronic keyword searches

    • Electronic citation searches

    • Combing reference lists of key articles

    • Manually searching key journals

    • Searching grey literature

  • Optimized search strategies (i.e., hedges, filters)
Literature searches should be made as transparent as possible without jeopardizing the creativity and complexity of the process
A key consideration for ending a knowledge-building/theory-generating review is whether small amounts of conflicting information are likely to substantially change the findings
Open in a new tab

Expansive Versus Exhaustive Literature Searches

To avoid the potential problem of having too few descriptively-rich studies, knowledge-building and theory-generating systematic reviewers conduct expansive searches of the literature. The term expansive is purposefully used to differentiate the types of searches that are carried out in conjunction with knowledge-building and theory-generating qualitative systematic reviews versus exhaustive searches that are typically conducted in association with summative and aggregative qualitative systematic reviews (e.g., meta-aggregation) (Booth 2006, Hannes & Lockwood 2011).

When conducting exhaustive searches, topics and research questions are generally established a priori and ideally, all relevant research reports are subsequently identified and secured in a predetermined, methodical and progressive manner (Booth 2010a, Boell & Cecez-Kecmanovic 2010, Hannes & Lockwood 2011). Linear flowcharts (e.g., PRISMA Flow Diagram [Moher et al. 2009]) that illustrate the number of documents initially identified, excluded and retained are frequently used to illustrate these types of unidirectional search processes (Bates 1989, Yoshii et al. 2009).

Conversely, when knowledge-building and theory-generating qualitative systematic reviews are conducted, such prima facie thinking is less common and it may even inhibit scholarly creativity (Boell & Cecez-Kecmanovic 2010, Hannes & Lockwood 2011). From the outset, the focus, breadth and depth of a knowledge-building or theory-generating qualitative systematic review is less well defined and such reviews are thought to evolve in an iterative manner (Barroso et al. 2003). In these instances, the purpose of conducting an expansive literature search is to sculpt contextually-consistent and rich research topics, questions, samples and databases to promote the emergence of meaningful and saturated research findings (Boell & Cecez-Kecmanovic 2010).

Searching the Literature

Given the emergent nature of knowledge-building and theory-generating qualitative systematic reviews, it is not surprising that searching the literature is a dynamic process that requires vigilant oversight and intermittent recalibration. In the case of these types of reviews, curiosity is piqued, niggling questions emerge, and literature searches are initiated based on information at hand and personal insights (Barroso et al. 2003, Boell & Cecez-Kecmanovic 2010).

Bates (1989, 2007) described a general approach for searching the literature that is consistent with expansive searching. Accordingly, browsing ensues as the researcher glimpses, samples, examines and acquires or abandons information. Mere browsing might intermittently cease and more focused searching may result as phenomena of interest emerge, populations of relevance become evident, contextual elements take shape and formative research questions are identified. As research questions cohere, key search concepts are identified and foundational research reports begin to form the core database for a systematic review.

As suggested earlier, the search process advocated by Bates lacks the defined linearity of more classic literature searches (Bates 1989, Hider 2006, Boell & Cecez-Kecmanovic 2010, Hannes & Lockwood 2011). Instead, it is a circuitous problem solving process that involves alternating forms of inductive and deductive reasoning, synchronous searching, selecting and discarding, and finally, problem formation, fine-tuning and confirmation (Bates 1989, 2007, Barroso et al. 2003, Hider 2006, Johnson 2009, Boell & Cecez-Kecmanovic 2010, Hannes & Lockwood 2011).

Challenges of Searching for Qualitative Research Reports

Publication bias

Just as a biased sample is likely to lead to biased results in terms of any singular study, a biased collection of studies is also likely to lead to biased systematic review findings (Wilson 2009). Several challenges exist in terms of searching the literature and assembling an unbiased database for knowledge-building and theory-generating systematic reviews. In the qualitative and quantitative research paradigms, publication practices differ, which renders differences in the types of biases that qualitative and quantitative systematic reviewers must work to minimize (Wilson 2009, Finfgeld-Connett 2010b).

In general, researchers who conduct quantitative studies hope for significant positive results. When this is not the outcome, chances diminish that their findings will appear in highly accessible and prestigious journals. This publication bias is explained by several interconnected factors, among them are author disappointment and/or disinterest in negative findings, funding agency publication preferences and editor and/or reviewer biases. The end result is that positive quantitative findings tend to be more readily accessible for systematic review than negative ones (Wilson 2009, Booth 2010a, Song et al. 2010).

In addition to determining whether an intervention is effective or not, qualitative researchers are more broadly interested in knowing about contextual factors that surround a situation. Due in part to this altered focus, qualitative findings are not generally considered to be positive or negative, the stigma of non-significant findings is greatly diminished, and these types of associated publication biases tend to be eliminated (Finfgeld-Connett 2010b).

This does not mean that publication biases do not exist in relationship to qualitative research. On the contrary, a bias of potentially greater proportions may threaten searches for qualitative research reports. This threat relates to the fact that, in some circles, qualitative research is perceived to be of lesser quality and value than quantitative research (Booth 2010a-b, Shuval et al. 2011). For this reason, qualitative studies may be less frequently conducted, submitted for publication and/or published in high quality and easily accessible journals. The result is that raw data (i.e., research findings) that are needed to conduct a qualitative systematic review may not be readily available.

Evidence of this potential publication bias can be inferred from the results of several research investigations. In 2004, it was reported that only 4 of the top 20 high impact health care journals published qualitative studies (McKibbon & Gadd 2004). On the heels of these findings, investigators reported that only one influential general medicine journal published an average of 15 qualitative research reports between 2000 and 2004 (Yamazaki et al. 2009). At about the same time, it was reported that among a total of 2,574 reports of research that were published in ten high-impact nursing journals, 51% (n = 1,323) were classified as quantitative compared with 37% (n = 956) that were categorized as qualitative. The remainder were mixed-methods studies 2% (n = 57) or reports of secondary data analyses (39%, n = 238) (Mantzoukas 2009). Most recently, higher publication rates of qualitative research have been associated with journals where qualitative research was discussed in the author guidelines. The implication being that some journals do not encourage submission of qualitative research reports and this is reflected in the numbers that are published (Shuval et al. 2011).

Admittedly, these findings do not account for the actual numbers of qualitative versus quantitative research studies that have been conducted or the resultant number of manuscripts that have been submitted for peer-review. They also are not directly indicative of the proportions of quantitative or qualitative manuscripts that have been accepted or rejected. They do, however, support the inference that there appears to be a tendency to conduct and publish more quantitative than qualitative research. In addition, although biases relating to the publication of qualitative research reports appear to be diminishing (Yamazaki et al. 2009, Booth 2010a, Shuval et al. 2011), concerns persist due to the current emphasis on evidence-based practice and prevailing hierarchies of evidence that favor quantitative research (Yamazaki et al. 2009, Booth 2010a).

Indexing barriers

Another unique challenge that all qualitative systematic reviewers face is that qualitative research reports can be difficult to identify using electronic indexing systems. This is because standard indexing terms for locating reports of qualitative research do not exist in the same way that they do for quantitative reports. This problem is made worse by the fact that authors of qualitative and mixed-method research reports appear to be particularly remiss about providing good descriptions of their research methods in the titles and abstracts of documents (Evans 2002, Shaw et al. 2004, Wong et al. 2004, McKibbon et al. 2006, Flemming & Briggs 2007, Gorecki et al. 2010). To overcome these types of indexing barriers and publication biases, qualitative systematic reviewers have adopted some adaptive literature search strategies.

Methods for Overcoming Barriers to Searching for Qualitative Literature

Berry picking approach

Conducting literature searches for some qualitative systematic reviews has been likened to berry picking because the strategies that are used to locate relevant research reports resemble those that are used to search for wild berries (Barroso et al. 2003). Berry pickers typically amble about looking for patches of ripe fruit. Once the choicest berries have been harvested from one locale, searchers tend to circuitously wander until they find other nearby sources. Sometimes, just a few well-cultivated patches yield an abundant number of edible berries. When they do not, the circuitous searching process is broadened to include previously unexplored locales (Bates 1989, 2007).

Personal experience suggests (e.g., Finfgeld-Connett & Johnson 2011) that searching for qualitative research reports for the purpose of conducting a knowledge-building or theory-generating systematic review is similar to collecting wild berries. The process cannot be definitively mapped from the outset, nor is it easily reconstructed at the conclusion. The route involves many twists and turns and to-and-fro movements that defy simple cartography. By the time the researcher has come to the end of the search, it may be difficult to remember exactly how the journey started or what route was taken (Bates 1989, 2007). This is particularly true when other responsibilities compete for a researcher’s time and data gathering intermittently occurs over the course of many months.

The challenge of searching for qualitative research reports using electronic database keywords has already been discussed. Thus, when the practical use of these databases has been exhausted, other search strategies are recommended. Alternatives include electronically searching for articles where key citations appear, combing the reference lists of relevant articles and/or manually searching journals that are likely to publish pertinent qualitative research reports (Bates 1989, Wilson 2009).

It is argued that ignoring grey literature, such as dissertations/theses, government reports, monographs and books, on the basis that it may be of lesser quality is empirically and logically invalid (Wilson 2009). In fact, these types of documents may be particularly rich sources of qualitative data since page limits are not generally imposed. Also, although lengthy report formats are relatively uncommon in the health sciences, they tend to be the norm in disciplines such as anthropology (Noblit & Hare 1988), where context-rich data are likely to be found.

Optimized search strategies

Due to the challenges involved in identifying qualitative research reports using electronic databases, health informatics experts have conducted research to identify optimized search strategies for locating such documents. Based on these results, it appears that simple search strategies may be as effective as more complex ones for identifying reports of qualitative research. For example, in a study of patients’ perceptions of living with a leg ulcer, researchers determined that three broad-based terms (i.e., qualitative, findings and interviews) were as effective as more complex search strategies for the purpose of identifying relevant qualitative research reports (Flemming & Briggs 2007).

Based on these and other similar research results (e.g., Gorecki et al. 2010), health information experts have developed optimal search strategies, commonly referred to as hedges or filters, to help non-expert searchers identify reports of qualitative research. In this instance, the word filter is actually a misnomer, since these search strategies do not filter out qualitative research reports. Instead, they help to identify potentially relevant documents (Grant 2004). These optimal search strategies, which are comprised of terms such as interview, qualitative, themes and experience, are built into some electronic databases (e.g., CINAHL, EMBASE, MEDLINE) and can be selected as a search option (McKibbon et al. 2006).

Qualitative systematic reviewers are urged to make well-calculated decisions about whether to use expert-designed optimal search strategies to search the literature. Researchers may opt to use such strategies at the outset of a literature search, but if their results are sparse and data saturation appears unlikely, they may need to try other approaches. It might be necessary to search for topically-relevant documents (e.g., self-management of food allergies) and in turn, manually select reports of qualitative research from this initial set. Based on personal experience, this task can be time consuming since it may involve manually scanning thousands of document titles and abstracts and when necessary, entire articles (e.g., Finfgeld-Connett 2010a).

Transparency of the Literature Search

Knowledge-building and theory-generating qualitative systematic reviews are a form of research and as such, transparency is a necessary part of the research process (Wilson 2009, Yoshii et al. 2009). Scholars indicate that literature search strategies should be made available so that they can be replicated (e.g., McKibbon 2006), but researchers suggest that this may be more the ideal than a practical reality (Yoshii et al. 2009).

The process of searching for qualitative research reports can be messy and those ah-hah moments that irrevocably change the course of a literature search are difficult to immediately record or to retrospectively retrace. Even the most experienced searcher can easily forget when, why, or how a keyword was selected, excluded, or added back into a search strategy. Moreover, decisions like this may be revisited many times in the context of multiple databases. On the whole, personal experience (e.g., Finfgeld-Connett 2010a) suggests that assiduously maintaining a search strategy log can be creatively stifling if not virtually impossible.

Even when an accurate record (i.e., audit trail) of a data collection process exists, publishing exact search strategies may have limited utility. This relates to the fact that database indexing and algorithms change over time and subsequent researchers may have difficulty determining what was actually searched. In the case PubMed, the difference between user-entered search strategies and algorithmic transformations can be observed by selecting the details option. When authors feel compelled to share their PubMed search strategies, they are urged to publish the transformed algorithms rather than the versions that they typed into the database (Niederstadt & Droste 2010).

Ending a Literature Search

In the case of knowledge-building and theory-generating qualitative systematic reviews, ‘the more the better’ is not necessarily the most methodologically sound perspective. On the contrary, in the qualitative research paradigm, validity (i.e., trustworthiness) is not solely based on redundancy (Finfgeld-Connett 2010b) and there is no inherent value in continuing a literature search unless it enriches the sample in the context of the review (Booth 2010a). In contrast, more is, indeed, the objective when conducting aggregative and summative systematic reviews, and in this respect, they are associated more closely with the quantitative rather than the qualitative paradigm (Hannes & Lockwood 2011).

In relationship to knowledge-building and theory-generating systematic reviews, more is better only when it helps to fully explicate a concept, substantiate an interconnection between or among concepts (Corbin & Strauss 2008), or build a line of argument (Noblit & Hare 1988). Simply more of the same does not necessarily help to achieve these objectives. In fact, collecting more of the same may merely escalate the cost of a study, clutter the database and obfuscate important inferences (Chapman et al. 2009, Booth 2010a, Finfgeld-Connett 2010b). Concepts and the interrelationships among them can only be more fully explicated based on data that adds depth, breadth, meaning and understanding to a phenomenon (Corbin & Strauss 2008).

When deciding whether searching should cease, researchers are urged to consider whether unidentified studies are likely to substantively change the findings. More specifically, in the presence of apparent data saturation, would one or two pieces of conflicting information change the overall findings, or would they merely add to the inevitable plea for additional primary research to answer emergent questions (Barroso et al. 2003, Booth 2006, 2010a, Boell & Cecez-Kecmanovic 2010)?

Regardless of the type of systematic review that is conducted, ending the literature search is a judgment call, since it is impossible to know what data have not been captured (Barroso et al. 2003, Booth 2006). That said, every attempt should be made to insure that decisions to cease searching are well rationalized and substantiated in the context of each unique review (Booth 2010a). In the case of knowledge-building and theory-generating systematic reviews, the key criterion is saturation of concepts and the full explication of interrelationships among them (Noblit & Hare 1988, Corbin & Strauss 2008).

When deciding whether to end a literature search it is also important to acknowledge that no single systematic review will inform in perpetuity (Yoshii et al. 2009) and ‘lack of certainty must be seen as the first source of knowledge’ (Sturmberg 2011, p. 507). New studies will be conducted and findings from these studies will need to be examined in the context of existing knowledge (Boell & Cecez-Kecmanovic et al. 2010). Over time, it is also likely that more than one systematic review will be conducted in relationship to a single topic. When this occurs, scholars will inevitably feel compelled to consider the value of a systematic review of systematic reviews.

Implications for Nursing

In light of the current discussion, several implications for nurses who are interested in conducting knowledge-building or theory-generating systematic reviews are outlined below. To remain consistent with the aims of these types of reviews and to promote transparency in the data collection and reporting processes, nurses are urged to:

  1. Provide accurate accounts of their literature search processes, which tend to be iterative and circuitous versus linear.

  2. Identify which electronic databases were searched and why. Information about database coverage plus user tutorials can usually be located in each database website.

  3. Describe how grounding concepts (versus keywords) emerged and were selected for use across databases. Explain why some concepts (versus keywords) were excluded.

  4. Explain how barriers to searching for qualitative research reports were overcome (e.g., hedges, filters, manually identifying qualitative research reports from all topically-relevant reports).

  5. Describe which grey literature resources were and were not searched and why.

  6. Explain whether or not authors, references and citations were systematically tracked and how this was done (e.g., manually or electronically).

  7. Describe and rationalize when and how a decision to stop searching the literature was made.

This is not a comprehensive list of all of the information that needs to be documented when developing reports of knowledge-building or theory-generating qualitative systematic reviews. Instead, it is a list that is meant to serve as a reminder of information that is not always clearly and accurately presented. This is despite its importance for understanding the knowledge-building and theory-generating systematic review process.

Given the emergent nature of qualitative systematic review methods and the paucity of information that is currently available relating to searching the literature, the ideas that are presented in this article are nascent. To date, knowledge-building and theory-generating systematic reviewers have only begun to recognize and document literature searching methods that are grounded in actuated strategies. Validation of these methods is pending and so is the process of revising and fine-tuning them.

Conclusion

When conducting knowledge-building and theory-generating qualitative systematic reviews, the goal is to fully explicate concepts and the relationships among them. To accomplish this goal, qualitative systematic reviewers are urged to systematically search the literature in an expansive manner, allowing search strategies to emerge as the research investigation takes shape. This means carefully calibrating and re-calibrating search strategies to ensure that data collection efforts yield more than narrowly redundant data (i.e., research findings).

Researchers are urged to consider whether challenges involved in searching for qualitative research reports can be diminished by using empirically-based optimal search strategies. To enhance transparency of the complex literature search process, researchers are also encouraged to provide thorough and frank explanations of their search strategies. Finally, a full description of the decision to discontinue searching is recommended.

At the risk of sounding overly pedantic, it should be emphasized that expansive versus exhaustive literature searching is not an easy way out or a way to cut corners. On the contrary, collecting data to build knowledge and generate theory is an intellectually challenging and time-intensive activity that involves sophisticated problem-solving and decision-making. Conceptual saturation and full explication of interrelationships among concepts can only be accomplished based on insightful and rigorous searching of the literature.

Summary Statement.

What is already known about this topic

  • Methods for conducting qualitative systematic reviews continue to evolve.

  • Researchers are uncertain about what type of literature search to carry out when conducting knowledge-building and theory-generating qualitative systematic reviews.

What this paper adds

  • When conducting knowledge-building and theory-generating qualitative systematic reviews, the research objectives are to fully explicate concepts and the interrelationships among them.

  • Expansive versus exhaustive literature searches are consistent with the objectives of knowledge-building and theory-generating qualitative systematic reviews.

  • When conducting knowledge-building and theory-generating systematic reviews, searching for qualitative research reports involves a non-linear iterative approach, which is difficult to document in traditional ways.

Implications for nursing

  • Challenges involved in identifying qualitative research reports can be overcome by using a variety of strategies to search the literature.

  • Transparent accounts of literature search processes are recommended, including clear rationale for concluding a search.

Acknowledgments

Funding: The project described was supported by Grant Number R21DA024749 from the National Institute on Drug Abuse. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Footnotes

Conflict of Interest: No conflict of interest has been declared by the authors.

Author Contributions:

All authors meet at least one of the following criteria (recommended by the ICMJE*) and have agreed on the final version:

  1. substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data;
  2. drafting the article or revising it critically for important intellectual content.

* http://www.icmje.org/ethical_1author.html

Contributor Information

Deborah Finfgeld-Connett, Email: finfgeldD@missouri.edu, Associate Professor, S321Sinclair School of Nursing, University of Missouri, Columbia, Missouri, USA, 65211, Telephone: 573-884-7255, Fax: 573-449-6632.

E. Diane Johnson, Assistant Director, Information Services and Resources, J. Otto Lottes Health Sciences Library, University of Missouri.

References

  1. Barroso J, Gollop CJ, Sandelowski M, Meynell J, Pearce PF, Collins LJ. The challenges of searching for and retrieving qualitative studies. Western Journal of Nursing Research. 2003;25:153–178. doi: 10.1177/0193945902250034. [DOI] [PubMed] [Google Scholar]
  2. Bates MJ. The design of browsing and berrypicking techniques for the online search interface. Online Review. 1989;13:407–424. Retrieved from http://comminfo.rutgers.edu/~tefko/Courses/612/Articles/Bates.pdf on June 9, 2011. [Google Scholar]
  3. Bates MJ. What is browsing—really? A model drawing from behavioural science research. 2007;12(4):paper 330. Retrieved from http://informationr.net/ir/12-4/paper330.html on June 9, 2011. [Google Scholar]
  4. Boell SK, Cecez-Kecmanovic D. Literature reviews and the hermeneutic circle. Australian Academic & Research Libraries. 2010;41:129–144. [Google Scholar]
  5. Booth A. The number needed to retrieve: a practically useful measure of information retrieval? Health Information and Libraries Journal. 2006;23:229–232. doi: 10.1111/j.1471-1842.2006.00663.x. [DOI] [PubMed] [Google Scholar]
  6. Booth A. How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. International Journal of Technology Assessment in Health Care. 2010a;26:431–435. doi: 10.1017/S0266462310000966. [DOI] [PubMed] [Google Scholar]
  7. Booth A. On hierarchies, malarkeys and anarchies of evidence. Health Information and Libraries Journal. 2010b;27:84–88. doi: 10.1111/j.1471-1842.2010.00879.x. [DOI] [PubMed] [Google Scholar]
  8. Chapman AL, Morgan LC, Gartlehner G. Semi-automating the manual literature search for systematic reviews increases efficiency. Health Information and Libraries Journal. 2009;27:22–27. doi: 10.1111/j.1471-1842.2009.00865.x. [DOI] [PubMed] [Google Scholar]
  9. Cumulative Index to Nursing and Health Literature. Subject headings and subject coverage. 2011 Retrieved from http://www.ebscohost.com/cinahl/default.php?id=8.
  10. Corbin J, Strauss A. Basics of qualitative research: Techniques and procedures for developing grounded theory 3e. Sage; Los Angeles, USA: 2008. [Google Scholar]
  11. Evans D. Database searches for qualitative research. Journal of the Medical Library Association. 2002;90:290–293. [PMC free article] [PubMed] [Google Scholar]
  12. Finfgeld DL. Meta-synthesis: the state of the art—so far. Qualitative Health Research. 2003;13:893–904. doi: 10.1177/1049732303253462. [DOI] [PubMed] [Google Scholar]
  13. Finfgeld-Connett D. Becoming homeless, being homeless and resolving homelessness among women. Issues in Mental Health Nursing. 2010a;31:461–469. doi: 10.3109/01612840903586404. [DOI] [PubMed] [Google Scholar]
  14. Finfgeld-Connett D. Generalizability and transferability of meta-synthesis research findings. Journal of Advanced Nursing. 2010b;66:246–254. doi: 10.1111/j.1365-2648.2009.05250.x. [DOI] [PubMed] [Google Scholar]
  15. Finfgeld-Connett D, Johnson ED. Substance abuse treatment for women who are under correctional supervision in the community: A systematic review of qualitative findings. Issues in Mental Health Nursing. 2011;32:640–648. doi: 10.3109/01612840.2011.584363. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Flemming K, Briggs M. Electronic searching to locate qualitative research: evaluation of three strategies. Journal of Advanced Nursing. 2007;57:95–100. doi: 10.1111/j.1365-2648.2006.04083.x. [DOI] [PubMed] [Google Scholar]
  17. Gorecki CA, Brown JM, Briggs M, Nixon J. Evaluation of five search strategies in retrieving qualitative patient-reported electronic data on the impact of pressure ulcers on quality of life. Journal of Advanced Nursing. 2010;66:645–652. doi: 10.1111/j.1365-2648.2009.05192.x. [DOI] [PubMed] [Google Scholar]
  18. Grant MJ. How does your searching grow? A survey of search preferences and the use of optimal search strategies in the identification of qualitative research. Health Information and Libraries Journal. 2004;21:21–32. doi: 10.1111/j.1471-1842.2004.00483.x. [DOI] [PubMed] [Google Scholar]
  19. Grant MJ, Booth A. A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal. 2009;26:91–108. doi: 10.1111/j.1471-1842.2009.00848.x. [DOI] [PubMed] [Google Scholar]
  20. Hannes K, Lockwood C. Pragmatism as the philosophical foundation for the Joanna Briggs meta-aggregative approach to qualitative evidence synthesis. Journal of Advanced Nursing. 2011;67:1632–1642. doi: 10.1111/j.1365-2648.2011.05636.x. [DOI] [PubMed] [Google Scholar]
  21. Hider P. Search goal revision in models of information retrieval. Journal of Information Science. 2006;32:352–361. [Google Scholar]
  22. Johnson JD. An impressionistic mapping of information behavior with special attention to contexts, rationality and ignorance. Information Processing and Management. 2009;45:593–604. [Google Scholar]
  23. Lincoln YS, Guba EG. Naturalistic Inquiry. Sage; Newbury Park, CA, USA: 1985. [Google Scholar]
  24. Mackay G. Is there a need to differentiate between qualitative and quantitative searching strategies for literature reviewing? Qualitative Social Work. 2007;6:231–241. [Google Scholar]
  25. Mantzoukas S. The research evidence published in high impact nursing journals between 2000 and 2006: a quantitative content analysis. International Journal of Nursing Studies. 2009;46:479–489. doi: 10.1016/j.ijnurstu.2008.12.016. [DOI] [PubMed] [Google Scholar]
  26. McKibbon KA. Systematic reviews and librarians. Library Trends. 2006;55:202–215. [Google Scholar]
  27. McKibbon KA, Gadd CS. A quantitative analysis of qualitative studies in clinical journals for the 2000 publishing year. BMC Medical Informatics and Decision Making. 2004;4:11. doi: 10.1186/1472-6947-4-11. Retrieved from http://www.biomedcentral.com/content/pdf/1472-6947-4-11.pdf on November 2 2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. McKibbon KA, Wilczynski NL, Haynes RB. Developing optimal search strategies for retrieving qualitative studies in PsycINFO. Evaluation & The Health Professions. 2006;29:440–454. doi: 10.1177/0163278706293400. [DOI] [PubMed] [Google Scholar]
  29. Moher D, Liberati A, Tetzlaff J, Altman DG The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009;6(6):e1000097. doi: 10.1371/journal.pmed1000097. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Morse JM, Singleton J. Exploring the technical aspects of ‘fit’ in qualitative research. Qualitative Health Research. 2001;11:841–847. doi: 10.1177/104973201129119424. [DOI] [PubMed] [Google Scholar]
  31. Niederstad C, Droste S. Reporting and presenting information retrieval processes: the need for optimizing common practice in health technology assessment. International Journal of Technology Assessment in Health Care. 2010;26:450–457. doi: 10.1017/S0266462310001066. [DOI] [PubMed] [Google Scholar]
  32. Noblit GW, Hare RD. Meta-ethnography: Synthesizing qualitative studies. Sage; Newbury Park, USA: 1988. [Google Scholar]
  33. Ring N, Ritchie K, Mandava L, Jepson R. A guide to synthesizing qualitative research for researchers undertaking health technology assessments and systematic reviews. NHS Quality Improvement Scotland. 2011 Retrieved from: http://www.healthcareimprovementscotland.org/his/idoc.ashx?docid=04df262c-ec20-40e1-aef7-d2aa3ad03c06&version=-1 on November 8 2011.
  34. Sandelowski M, Barroso J. Handbook for synthesizing qualitative research. Springer; New York, NY, USA: 2007. [Google Scholar]
  35. Sandelowski M, Barroso J, Voils CI. Using qualitative metasummary to synthesize qualitative and quantitative descriptive findings. Research in Nursing & Health. 2007;30:99–111. doi: 10.1002/nur.20176. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Scopus. About Scopus. 2011 Retrieved from http://www.info.sciverse.com/scopus/about on June 8, 2011.
  37. Shaw RL, Booth A, Sutton AJ, Miller T, Smith JA, Young B, Jones DR, Dixon-Woods M. Finding qualitative research: an evaluation of search strategies. BMC Medical Research Methodology. 2004;4:5. doi: 10.1186/1471-2288-4-5. Retrieved from http://www.biomedcentral.com/content/pdf/1471-2288-4-5.pdf on June 9, 2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Shuval K, Harker K, Roudsari B, Groce NE, Mills B, Siddiqi Z, Shachak A. Is qualitative research second class science? A quantitative longitudinal examination of qualitative research in medical journals. PLoS ONE. 2011;6(2):e16937. doi: 10.1371/journal.pone.0016937. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, Hing C, Kwok CS, Pang C, Harvey I. Dissemination and publication of research findings: an updated review of related biases. Health Technology Assessment. 2010;14:1–220. doi: 10.3310/hta14080. Retrieved from http://www.hta.ac.uk/fullmono/mon1408.pdf on April 4, 2011. [DOI] [PubMed] [Google Scholar]
  40. Sturmberg JP. The illusion of certainty — deluded perception? Journal of Evaluation in Clinical Practice. 2011;17:507–510. doi: 10.1111/j.1365-2753.2011.01667.x. [DOI] [PubMed] [Google Scholar]
  41. Wilson DB. Missing a critical piece of the pie: simple document search strategies inadequate for systematic reviews. Journal of Experimental Criminology. 2009;5:429–440. [Google Scholar]
  42. Wong SS-L, Wilczynski NL, Haynes RB. Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE. Studies in Health Technology and Informatics. 2004;107(Pt 1):311–316. [PubMed] [Google Scholar]
  43. Yamazaki H, Slingsby BT, Takahashi M, Hayashi Y, Sugimori H, Nakayama T. Characteristics of qualitative studies in influential journals of general medicine: a critical review. BioScience Trends. 2009;3:202–209. [PubMed] [Google Scholar]
  44. Yoshii A, Plaut DA, McGraw K, Aanderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association. 2009;97:21–29. doi: 10.3163/1536-5050.97.1.004. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES