Beyond exploratory: a tailored framework for designing and assessing qualitative health research

Abstract

The objective of this commentary is to develop a framework for assessing the rigour of qualitative approaches that identifies and distinguishes between the diverse objectives of qualitative health research, guided by a narrative review of the published literature on qualitative guidelines and standards from peer-reviewed journals and national funding organisations that support health services research, patient-centered outcomes research and other applied health research fields. In this framework, we identify and distinguish three objectives of qualitative studies in applied health research: exploratory, descriptive and comparative. For each objective, we propose methodological standards that may be used to assess and improve rigour across all study phases—from design to reporting. Similar to hierarchies of quality of evidence within quantitative studies, we argue that standards for qualitative rigour differ, appropriately, for studies with different objectives and should be evaluated as such. Distinguishing between different objectives of qualitative health research improves the ability to appreciate variation in qualitative studies and to develop appropriate evaluations of the rigour and success of qualitative studies in meeting their stated objectives. Researchers, funders and journal editors should consider how further developing and adopting the framework for assessing qualitative rigour outlined here may advance the rigour and potential impact of this important mode of inquiry.

Article summary.

• Qualitative research in health services research (HSR) and allied fields has maintained steady, yet unsettled, interest and value over recent decades.

• Qualitative methods are epistemologically and theoretically diverse, which is a strength. However, it also means that investigators do not necessarily approach qualitative research using a unified set of evidentiary rules. As such, assessing rigour and quality across studies can be challenging.

• To help address these challenges, we propose a framework for assessing the rigour of qualitative approaches that identifies and distinguishes between three diverse study objectives. For each type of study, we propose preliminary methodological considerations to help improve rigour across all study phases. As is the case for quantitative studies, we argue that standards for qualitative rigourdiffer, appropriately, for different kinds of studies.

• The objective of this commentary is not to resolve all potential conflicts between philosophical assumptions of different qualitative approaches, but rather help to advance a broader and richer understanding of qualitative rigour in relationship to other evidence hierarchies.

In recent decades, the role of qualitative research in health services research (HSR) and allied fields has maintained steady, yet unsettled, interest and value. Evidence of steady interest includes publication of qualitative reviews and guidelines by leading journals including Health Services Research,^{1 2} Medical Care Research and Review ^3–5 and BMJ,^{6 7} and by funders including the Robert Wood Johnson Foundation,⁸ National Institutes of Health^{9 10} and National Science Foundation.^{11 12} In fields such as Patient-Centered Outcomes Research (PCOR) and implementation science, qualitative research has been embraced with particular enthusiasm for its ability to capture, advance and address questions meaningful to patients, clinicians and other healthcare system stakeholders.^{2 13} The majority (82%) of inaugural projects awarded by the Patient-Centered Outcomes Research Institute (PCORI) incorporated qualitative research methods.¹³ More recently, reflective of the continued prevalence of these approaches in the field, PCORI incorporated qualitative methods into their methodological standards.

Yet, despite this sustained interest, the status of qualitative health research remains unsettled, as illustrated by the BMJ’s changing engagement with the method. After championing qualitative methods in 2008,^{7 14–17} BMJ editors in 2016 noted that they tended to assign low priority to qualitative studies because such studies are ‘usually exploratory by their very nature’.¹⁸ This statement came in response to an open letter from scholars arguing that BMJ should adopt formal policies and training for editorial staff on what distinguishes ‘good from poor qualitative research’ rather than de-emphasising the method in toto.¹⁹ In sum, despite sustained interest from the HSR community, the status of qualitative research remains contested. This status reflects debate over the purpose of qualitative research—is it a valuable tool to advance the field or a low-priority exercise in exploration? —and an ongoing desire for guidance on how best to distinguish high-quality from low-quality qualitative research.

Assessing rigour and quality in qualitative research is challenging because qualitative methods are epistemologically diverse.^20–22 This diversity is a strength because it allows for the theoretical and methodological flexibility necessary to fully understand a specific topic from multiple perspectives.¹⁶ However, it also means that investigators do not necessarily approach qualitative research using a unified set of evidentiary rules.²² Thus, scholars may measure the quality of studies using different or even incompatible yardsticks.

The challenge of diverse epistemologies has become more acute as qualitative health research has expanded beyond its historical roots in phenomenological or grounded theory studies. Contemporary researchers may use qualitative data and methods to improve the descriptive accuracy of health-related phenomena that have already been characterised by exploratory work or are difficult to capture using other approaches.²³ Researchers also use larger scale, comparative qualitative studies in ways that resemble quantitative efforts to identify explanatory pathways.²⁴ Therefore, assessing the rigour of a specific qualitative study benefits from first identifying the analytic goals and objectives of the study—that is, identifying which yardstick investigators themselves have adopted—and then using this yardstick to examine how the study measures up.

To address these challenges, we propose a tailored framework for designing and informing assessments of different types of qualitative health research common within HSR. The framework recognises that qualitative investigators have different objectives and yardsticks in mind when undertaking studies and that rigour should be assessed accordingly. We distinguish three central types of qualitative study objectives common in applied health research: exploratory, descriptive and comparative. For each objective, we propose preliminary methodological considerations to help improve rigour across all study phases—from design to reporting. As is the case for quantitative studies, we argue that standards for qualitative rigour differ, appropriately, for different kinds of studies. The objective of this commentary is not to resolve all potential conflicts between philosophical assumptions of different qualitative approaches, but rather help to advance a broader and richer understanding of qualitative rigour in relationship to other evidence hierarchies. The proposed framework offers a nuanced set of categories by which to conduct and recognise high-quality qualitative research. The framework also supports efforts to shift debates over the value of qualitative research to discussions on how we can promote rigour across different types of valuable qualitative studies, and underscore how qualitative methods can advance clinical and applied health research.

Designing a tailored framework: methods and results

Our framework is based on a team-based review of published guidelines and standards discussing the scientific conduct of qualitative health research. Guided by expert consensus and a targeted literature scan, we identified and reviewed 17 peer-reviewed articles and expert reports published by journals widely read by the HSR community and by major funders or sponsors of qualitative health research (1–12, 21, 33–36). In contrast to previous reviews,²⁵ we did not seek to synthesise these guidelines. Rather we drew on them to develop a conceptual framework for designing and informing formal assessments of rigorous qualitative research.

Range of approaches in qualitative research

Qualitative research incorporates a range of methods including in-depth interviews, focus groups, ethnography and many others.²⁶ Even within a single method, accepted approaches and standards for rigour vary depending on disciplinary and theoretical orientations. Correspondingly, qualitative research cannot be defined by a single theoretical approach or data collection procedure. Rather many, often debated, approaches exist with distinct implications for appropriate standards for data collection, analysis and interpretation.

On one end of the spectrum, qualitative researchers guided by realism subscribe to the assumption that rigorous scientific research can provide an accurate and objective representation of reality, and that objectivity should be a primary goal of all scientific inquiries, including qualitative research.²⁷ These qualitative researchers generally consider standards such as validity, reliability, reproducibility and generalisability as similarly legitimate yardsticks for qualitative research as they are in quantitative research.²⁸ On the other end of the spectrum, relativist philosophical approaches to qualitative research typically argue that all research is inherently subjective and/or political,²⁹ and some relativists criticise the scientific approach specifically because it claims to be objective.^{30 31} Much of applied qualitative health research falls somewhere between the two ends of the spectrum. For example, Mays and Pope consider themselves ‘subtle realists’.⁶ They acknowledge that all research involves subjectivity and includes political dimensions, but they also contend that qualitative research should, nevertheless, be assessed by a similar set of quality criteria as quantitative studies. Although we recognise the value strictly relativist approaches provide, the framework and design considerations we propose are largely guided by a realist (or subtle realist) orientation. However, in addition to resonating with those who operate under similar orientations, we hope this framework will serve to advance discussions of how best to communicate and assess qualitative research using different theoretical and epistemological standpoints.

Tailored framework for qualitative health research

Given the diversity of approaches, a foundational step to improving the assessment of rigour in qualitative research is to abandon the attempt to develop a single standard for the best practices regardless of study orientation and objective. Instead, standards must begin with an assessment of epistemological assumptions and corresponding study objectives, an approach that is similar to standards for quantitative PCOR research³² and mixed-methods research.³³ In this vein, we identified and defined three general types of study objectives broadly used in applied qualitative health research (see figure 1). These three types reflect differences in primary study objectives and existing knowledge within a topic area.

Figure 1.

Three broad types of qualitative health research.

In table 1, we provide preliminary distinctions on how exploratory, descriptive and comparative studies compare across a range of standards and guidelines that have been proposed for qualitative research (see table 1). Regardless of study type, researchers should report study details in clear, comprehensive ways, using standardised reporting guidelines whenever possible.^{34 35}

Table 1.

Framework for designing different types of applied qualitative health research and developing evaluative instruments to assess their rigour

	Exploratory studies	Descriptive studies	Comparative studies
Epistemological framework	All studies should identify the epistemological framework under which the study and/or the investigators are guided.
State of evidence	Little to no data exist on the specific topic.	Exploratory data on the topic exist.	Exploratory and descriptive data on the topic exist.
Research aims	Define aims in broad, exploratory questions guided by the theoretical framework. A priori hypotheses are unnecessary and inappropriate.	Define aims based on existing knowledge and/or theoretical framework. A priori hypotheses may be useful, but not needed.	Define aims based on existing knowledge and/or theoretical framework. A priori hypotheses are recommended.
Sampling strategy	Appropriate to use a single, homogeneous sample. Convenience, purposeful or theoretical sampling is appropriate.	It may be appropriate to use a single, homogeneous sample if little is known about a specific subgroup or site. Purposeful or theoretical sampling is appropriate.	Include a diverse sample that supports comparison between groups. May consider integrating probability-based sampling stratified by groups of interest. Convenience sampling is inappropriate.
Data collection	Document interview or focus group data using audio recording and transcribe data verbatim, whenever possible. Any qualitative or ethnographical data that cannot be audio recorded should be collected using a systematic field note process.
Instrument	Develop an unstructured or semistructured guide based on aims. Adapt as new themes emerge.	Develop semistructured guide based on the aims and existing knowledge. Avoid changing key domains of interest; however, adding new themes is likely appropriate.
Data analysis	Develop clear analytic steps, guided by a theoretical or conceptual framework.
Coding	Inductive, iterative coding is appropriate. Consider developing a coding dictionary and using independent coders to code data.	A mix of deductive coding based on aims, and inductive, iterative coding to explore new themes is appropriate. Develop and systematically apply a coding dictionary. Use independent coders to code data, if possible.	A primarily deductive coding approach based on aims is appropriate. Develop and systematically apply a coding dictionary. Use independent coders to code data and assess intercoder reliability. Consider using data triangulation and negative case review to improve reliability.
Researcher reflexivity	Consider and declare potential biases of researchers.	Consider and declare potential biases of researchers. Consider ways to mitigate biases in study design.	Consider and declare potential biases of researchers. Identify ways to address and/or avoid strong biases.
Reporting results	Include clear details on study aims, sampling data collection and analysis. Consider using standardised reporting guidelines, such as the Consolidated Criteria for Reporting Qualitative Research (COREQ) or Standards for Reporting Qualitative Research (SRQR).
Level of evidence produced	Evidence of phenomena within a specific sample. Findings do not establish wider significance or prevalence of phenomena.	Evidence of previously known phenomena in different setting or group. Findings support the wider significance of phenomena.	Evidence of the wider significance and possible prevalence of defined phenomena within the bounds of the study populations or settings.

Compared with descriptive or comparative studies, exploratory studies approach the topic of study primarily in an inductive fashion to investigate the areas of potential research interest that remain mostly or wholly unexamined by the scientific community. Investigators undertaking exploratory studies typically have few expectations for what they might find, and their research design and approach may shift dramatically as they learn more about the phenomena of interest. An example of an exploratory study is a study that uses convenience sampling and unstructured interviews to explore what patients think about a new treatment in a single healthcare setting.

At the opposite end of this spectrum, investigators conducting comparative studies aim to use a primarily deductive approach designed to compare and document how well-defined qualitative phenomena are represented in different settings or populations. The qualitative methods employed in a comparative study are typically defined in advance, sampling should be systematic and structured by aims, and investigators enter the field with hypothesised ideas of what findings they may uncover and how to interpret those findings in light of previous research. An example of a comparative study is a multisite ethnography that seeks to compare how patient-provider communication varies by location, and uses random sampling of patient-provider interactions to collect data.

Descriptive studies occupy a middle position, building on previously conducted exploratory work so researchers will be able to proceed with more-focused inquiry. This should include well-defined procedures including sampling protocols and analytical plans, and investigators should usually articulate expected findings prior to beginning the study. However, as researchers investigate phenomena in new settings or patient populations, it is reasonable to expect descriptive studies to generate surprises. Thus, descriptive studies also feature inductive elements to detect unexpected findings, and must be flexible enough in design to accommodate shifts in research focus and methods based on empirical findings. An example of a descriptive study is a longitudinal study of patients with ovarian cancer that employs semistructured interviews and directed content analysis to examine decision-making across patients in a novel setting.

Discussion

Our review identified a number of qualitative standards and guidelines that have been published. The conceptual framework we present here draws on those extant guidelines through the recognition that qualitative health research includes studies of diverse theoretical and epistemological orientations, each of which has distinct understandings of scientific quality and rigour. Given this intellectual diversity, it is inappropriate to use a single yardstick for all qualitative research. Rather, assessments of qualitative quality must begin with an assessment of a study’s theoretical orientations and research objectives to ensure that rigour is assessed on a study’s own terms. This framework and suggested approaches may help to advance evaluations of qualitative rigour that acknowledge and differentiate between the studies that report exploratory, descriptive or comparative study objectives.

Existing standards for conducting health research and grading evidence, such as Grading of Recommendations Assessment, Development and Evaluation (GRADE),³⁶ do not capture the diversity of qualitative studies—often designating all qualitative studies as providing weak levels of evidence. PCORI’s own methodological standards have been largely silent regarding qualitative methods until recently,³² leaving applicants without clear direction on how to conduct rigorous qualitative research. Incorporation of tailored qualitative standards could help to clarify and improve the rigour of proposal design, review and completion. The establishment and integration of such standards could also guide journal editors in developing transparent standards for deciding priorities for publication. For example, editors may decide against publication of exploratory or descriptive studies, but prioritise well-executed comparative studies that advance the field in ways quantitative studies could not.

In addition to these immediate applications, implementing standards that incorporate the diversity of objectives within applied qualitative research has the potential to address broader challenges facing qualitative health research. These include: (1) the need to educate broader audiences about the many goals of qualitative research, including but not limited to exploration; (2) the need to create rigorous standards for conducting and reporting various types of qualitative studies to help audiences, editors and funders evaluate studies on their own merits and (3) the challenges of publishing qualitative research in prestigious and high-impact journals that will reach a wide range of practitioners, researchers and lay audiences. We contend that these challenges can be reframed as opportunities to advance the science of qualitative research, and its potential for improving outcomes for patients, providers and communities.