Eight ways to get a grip on intercoder reliability using qualitative-based measures

Abstract

The use of quantitative intercoder reliability measures in the analysis of qualitative research data has often generated acrimonious debates among researchers who view quantitative and qualitative research methodologies as incompatible due to their unique ontological and epistemological traditions. While these measures are invaluable in many contexts, critics point out that the use of such measures in qualitative analysis represents an attempt to import standards derived for positivist research. Guided by extant research and our experience in qualitative research, we argue that it is possible to develop a qualitative-based measure of intercoder reliability that is compatible with the interpretivist epistemological paradigm of qualitative research. We present eight qualitative research process-based guidelines for evaluating and reporting intercoder reliability in qualitative research and anticipate that these recommendations will particularly guide beginning researchers in the coding and analysis processes of qualitative data analysis.

Introduction

The debate

The use of quantitative intercoder reliability (ICR) measures, such as the kappa statistic, weighted kappa statistic, and binomial intraclass correlation coefficients (ICC), in the analysis of qualitative research data has often generated acrimonious debates among researchers who view quantitative and qualitative research methodologies as incompatible due to their unique ontological and epistemological traditions.^1-3 Braun and Clarke,¹ for example, assert that reliability is not an appropriate criterion for judging qualitative work and that quantitative measures of ICR are epistemologically problematic. ICR has been defined as a numerical measure of the agreement between different coders regarding how the same data should be coded.³ ICR can help provide confidence that systematic efforts were made to ensure the final qualitative data analytic framework is a credible and accurate representation of the data.³

ICR measures are used to assess the rigor and transparency of the coding frame and its application to the data.^4-7 A high ICR may be used to assure the research team and audience that the coding frame is sufficiently well specified to allow for its communicability across persons.^5,8,9 Performing an ICR assessment also ensures that multiple researchers can understand and contribute to the analytic process and provides confidence that the analysis transcends the imagination of a single individual. ICR assessment further ensures that the patterns in the latent content is fairly robust to the degree that if readers were to code the same qualitative text, they would make the same judgments or produce the same results.¹⁰ ICR fosters reflexivity and can serve as a badge of trustworthiness³ to the extent that some journal editors and reviewers now request or require a measure of ICR before agreeing to publish qualitative studies.¹¹ Taken together, ICR might improve the systematicity, communicability, and transparency of the coding process and promote reflexivity and dialogue within research teams. ³ Other critics note, however, that the use of ICR in qualitative analysis represents an attempt to import standards derived for positivist research^12,13 and that its use could mask the fact that a rigorous, in-depth qualitative analysis was not undertaken.

A major pitfall surrounding the use of quantitative ICR measures in qualitative research is that such use may create the incorrect assumption that somehow quantitative ICR measures do not essentially contradict the interpretative agenda of qualitative research^1,14-16 which requires the researcher to see the research field as composed of multiple perspectival realities that are intrinsically constituted by an individual’s social context and personal history.¹⁷ As O’Conner and Joffe³ note, the role of the qualitative researcher is not to reveal universal objective facts but to apply their theoretical expertise to interpret and communicate the diversity of perspectives on a given topic. Despite this inherent pitfall, some qualitative researchers often resort to quantitative based ICR measures or use their own methods that may not be well grounded in the literature. Also, in the absence of clear or adequate guidelines, some authors hesitate to engage in ICR assessments. We present eight process-based guidelines on ways to get a grip on intercoder reliability using qualitative-based measures. This paper is intended for use by researchers across the continuum and is particularly valuable for beginning researchers.

An alternative measure of ICR

We argue that it is possible to develop a robust measure of ICR that is unique and compatible with the interpretivist epistemological paradigm of qualitative research. This paradigm is premised on relativist ontology and subjectivist epistemology and assumes that reality as we know it is constructed intersubjectively through the meanings and understandings developed socially and experientially and that we cannot separate ourselves from what we know.¹⁸ This measure need not be statistical or quantitative. It can be descriptive and must be able to qualitatively characterize the extent to which independent coders agree or disagree on codes produced from interview, focus group, visual, and textual data. This approach must emphasize the need to achieve consistency between coders rather than mere quantification of the extent of agreement between coders and encourages reflexivity and authenticity throughout the qualitive analysis process. This alternative view of ensuring consistency is echoed by many qualitative researchers who argue that coding and identification of themes by independent researchers could be followed by a group discussion of overlaps and divergences¹⁹ without necessarily quantifying the degree of consensus achieved between the coders.³ In the rest of the commentary, we present and discuss a set of guidelines for evaluating and reporting ICR in qualitative data analysis based on prior research and the authors’ own experiences in the application of qualitative and quantitative research methods.^3,20-22 These guidelines are intended to be used in conjunction with other guidelines including those described elsewhere in the lietrature.²³ We have several years of diverse experience in mixed research methodology including coding and analyzing interviews, focus groups, and textual data, as well as narrative responses from survey data.

Ways to get a grip on evaluating and reporting ICR

Guided by extant research and our experience in qualitative research, we recommend eight ways to get a grip on evaluating and reporting ICR in qualitative research with the goal of achieving consistency in the coding process. These are summarized in Table 1.

Table 1.

Ways to get a grip on Intercoder Reliability

Aspects of Intercoder Reliability	Present		Justification (If ‘no’ selected)
There was a minimum of two coders.	Yes	No
At least one coder was more removed from data collection (to address bias).	Yes	No
At least one coder had expertise and previous experience with coding qualitative data.	Yes	No
If there were multiple participant groups, a minimum of two researchers (coders) coded transcripts from each participant group.	Yes	No
The coders used the same framework for analysis (e.g., inductive, deductive, abductive).	Yes	No
Coders focused on shared meaning of *codes through dialogue and consensus.	Yes	No
Another coder with expertise in qualitative methods was consulted to resolve outstanding conflicts.	Yes	No
Coder consensus resulted in a codebook** that was applied when coding the remaining transcripts.	Yes	No

* The code names do not have to be identical, but the meaning of the codes must be the same.

**In inductive and abductive analyses, coding can be an iterative process; therefore, new codes may be added to the codebook until code saturation is reached.

1. We suggest that at least two researchers must code the data, except in situations where the goal of the coding is to assess the extent of intracoder (within a single coder) reliability, wherein emphasis is placed on the extent of consistency with respect to how the same person codes data at multiple time points.^20-21 As Conner and Joffe³ describe, if the same person returns to the data at another time, it is possible to assess the extent of consistency in the coding process, thereby promoting researcher reflexivity.⁵

2. To ensure transparency and minimize bias, we recommend that at least one of the coders in the research team must be external to or removed from the data collection process in such a way that this external coder may view and code the data from a fresh perspective.

3. We recommend that at least one of the coders have expertise and previous experience with coding qualitative data to ensure that the coding and development of themes are done in a rigorous and robust manner, thereby increasing the consensus among coders.

4. Steps must be taken to ensure that use of novice coders (together with experienced coders) does not produce unreasonable discrepancies in coding and development of themes.

5. We also suggest that if a project includes multiple participant groups, a minimum of two researchers should code transcripts from each participant group.^3,20

6. We highly recommend that the coders use the same framework for analysis to ensure that basic concepts or themes developed within the analysis are consistent with the theoretical framework guiding the research.

7. Accordingly, we suggest that coders should focus on shared meaning of codes through a dialogue and consensus processes. However, where discrepancies in codes and themes emerge, we recommend that another coder with expertise in qualitative methods is consulted to resolve such observed discrepancies.

8. We recommend that the resulting codebook (based on consensus reached from selected transcripts) should be used to code the remaining transcripts. In inductive and abductive analyses, coding can be an iterative process; therefore, we suggest that new codes may be added to the codebook until a reasonable code saturation is reached.²⁴ The researchers could therefore schedule regular team meetings to discuss and achieve consensus on the newly added codes. We recommend that researchers should try to use as many criteria as often as possible to increase the rigor, trustworthiness, authenticity, and meaningfulness of qualitative research. However, if the researcher is unable to use all criteria, they should reflect and justify why they were unable to apply all the criteria.

Conclusion

We note that while there are valid reasons for incorporating quantitative-based measures of ICR into qualitative research, it is possible to develop a qualitative-based measure of ICR that is unique and compatible with the interpretivist epistemological paradigm of qualitative research. Drawing on prior research and research experience, we note further that this alternative measure does not need to be statistical in nature, however it must be able to characterize the extent to which independent researchers agree or disagree on codes produced from qualitative data and encourage reflexivity and authenticity throughout the qualitive analysis process. We anticipate that the recommendations presented here will guide researchers across the continuum, particularly beginning researchers in assessing the degree to which quality of process in ICR was met for qualitative data analysis.

Conflicts of Interest

The Authors declare no conflicts of interest.