Systematic debriefing after qualitative encounters: an essential analysis step in applied qualitative research

Abstract

Conversations regarding qualitative research and qualitative data analysis in global public health programming often emphasize the product of data collection (audio recordings, transcripts, codebooks and codes), while paying relatively less attention to the process of data collection. In qualitative research, however, the data collector’s skills determine the quality of the data, so understanding data collectors’ strengths and weaknesses as data are being collected allows researchers to enhance both the ability of data collectors and the utility of the data. This paper defines and discusses a process for systematic debriefings. Debriefings entail thorough, goal-oriented discussion of data immediately after it is collected. Debriefings take different forms and fulfill slightly different purposes as data collection progresses. Drawing from examples in our health systems research in Tanzania and Sierra Leone, we elucidate how debriefings have allowed us to: enhance the skills of data collectors; gain immediate insights into the content of data; correct course amid unforeseen changes and challenges in the local context; strengthen the quality and trustworthiness of data in real time; and quickly share emerging data with stakeholders in programmatic, policy and academic spheres. We hope this article provides guidance and stimulates discussion on approaches to qualitative data collection and mechanisms to further outline and refine debriefings in qualitative research.

Summary box.

• The quality of data in qualitative global health research is stronger when researchers engage local interviewers in a systematic debriefing process.

• Understanding interviewers’ strengths and weaknesses in real time allows researchers to enhance interviewer skills and thus data quality.

• Through systematic debriefing, researchers can identify and address gaps in the data; capture nuances and other non-verbal information; enhance intellectual partnership within teams; triangulate data; and build theory.

• Drawing from our research experiences in Tanzania and Sierra Leone, this paper outlines the process and value of debriefings.

Introduction

A major goal of qualitative research, as applied in global public health, is to give voice to those whose lives are affected by health policies and programs, but whose ability to be heard by those in power and to effectively change health systems and structures is limited. The archetypal means through which such data are collected is the interview, a one-on-one encounter where a researcher and a respondent ‘are talking and asking questions of one another’ usually with the help of an interview guide.^1–3 During the 1–2 hours that a typical interview lasts, a researcher must juggle competing demands: maintaining the interest and openness of the respondent, listening for responses that merit further probing, capturing unspoken cues and gestures, all while simultaneously ensuring that the data collected are relevant to the health issue of interest.^{1 4}

In the academic ideal, qualitative research is undertaken by those who possess sociocultural understanding of the study context, and are formally trained in qualitative theory, methods and analysis. In global public health research this ideal is often not feasible. Instead, research teams commonly engage locally based qualitative data collection teams who possess essential knowledge in terms of context and language, but who lack formal qualitative training. These teams are routinely trained by an external research lead, an individual who possesses a graduate-level education in public health and qualitative research, but whose contextual or linguistic knowledge is insufficient.

In the ‘local team with outside technical support’ model of qualitative research—henceforth referred to as applied qualitative research or AQR—technical support takes different forms across five overarching phases in the research. See table 1 for a breakdown of tasks for the data collection team and research lead, respectively, during the preparation and execution of data collection, transcription and translation of files, data analysis and study write-up.

Table 1.

Organization of qualitative research and qualitative data collection teams

Phase of research	Roles for local research team	Roles for technical expert/qualitative research manager
1. Preparation for data collection	Identifying and gathering information on study sites and topic of interest Introducing study to leaders of study communities Attending training Practicing interviews Practicing use of transcription software	Situating the study and site within the context of existing literature Acquiring permission letters and introducing the study and research purpose to national-level gatekeepers Leading training on qualitative research including types of methods, interviewing techniques (probing etc.), management of qualitative data, research ethics, transcription software Overseeing and critiquing practice interviews Refining tools after pilot
2. Data collection	Conducting interviews, focus groups and observations Taking notes during data collection, audiotaping, uploading audio files routinely Memoing and sharing reflexive notes Conveying findings during routine debriefings Identifying new lines of inquiry Triangulating findings with fellow team members Rehearsing presentation of preliminary findings	Observing data collection (ensuring privacy, consent, troubleshooting logistical and scientific challenges) Collecting data (when language permits) Writing field notes Leading debriefing sessions (including taking notes during debriefings) Developing and refining theories via debriefing sessions Comparing findings with existing literature Guiding teams in terms of new lines of inquiry Conveying emerging findings to principal investigator(s) (PI) and broader research team Responding to PI and study team suggestions on how to refine or further explore lines of inquiry Uploading and organizing data files Developing slide deck of preliminary findings
3. Transcription and translation	Transcribing and translating audio files	Organizing data files—aligning audio files with cover sheets Assigning transcriptions Assigning translations Providing feedback on quality of transcripts and translations (with support of bilingual research assistants)
4. Analysis	Completing summary questionnaires on each interview Coding Memo writing	Designing summary questionnaires Designing coding framework Providing feedback on coding and memo writing Coding (when language permits)
5. Write-up	Writing up findings into a report	Writing report that responds to program/donor/ministry question Identifying findings that speak to broader debates in the literature Publishing findings in peer-reviewed journals

This practice paper focuses on the data collection phase of the table with an emphasis on debriefings, the process where a research lead interviews data collectors soon after a data collection activity. We view debriefings as a necessary element in qualitative research, particularly as the field comes to embrace styles beyond the conventional academic ideal (one person enacting the research process from conception to publication). Furthermore, while we observe a general consensus regarding the rationale and process for undertaking many points in the five-step table (table 1), we see that debriefings receive relatively little attention in the literature and in trainings on qualitative research for public health. When debriefings are mentioned in publications, including our own,^{5 6} there is minimal insight into what the debriefing entailed, how it was conducted or how debriefings fundamentally informed the data collection or analysis process. Finally, given that in AQR, much of the data are collected by individuals who have limited training in qualitative research or are more familiar with quantitative survey administration, we urge researchers to more thoughtfully consider discussions on process (ensuring that the data are collected in an iterative ⁱ fashion,⁷ that the data set is responding to the research question and that the skills of data collectors are strengthened in real time) rather than on outputs (transcription and coding). We now outline what debriefings are, how their purpose shifts in the process of data collection and how debriefings have amplified trustworthiness in our own research.^{8 9}

What are debriefings?

Debriefings are a discrete moment in the qualitative data collection process where a research manager sits with a data collector (or data collection team) to discuss the tenor, flow and resulting findings from a recently undertaken data collection activity. Ideally conducted after the close of a day’s data collection, debriefings are an essential supplement to qualitative methods such as focus groups, interviews or observations. ^{ii 10} During debriefings, the research lead takes copious notes. These notes then serve as one component of the full qualitative data set, and methods used to analyze transcripts and observational memos are also applied to debriefing notes.

Debriefings spark immediate reflection on emerging findings; they force data collectors to think through the data that have emerged and to better position findings relative to data collected by fellow data collectors either that same day or to date. Debriefings allow research teams to identify gaps in the data collected and to redirect course—whether refining a line of inquiry, reconceptualizing a research question, opting when or whether to seek out alternative perspectives (such as negative or disconfirming cases), or adding or eliminating a respondent group or research method. Debriefings are the best protection against an unfortunate scenario where, long after the close of data collection, transcripts reveal that the research team did not pursue essential lines of inquiry, or worse, that the data collected will not be able to respond to research aims. For examples of debriefing templates, see online supplementary appendices 1 and 2.

How should debriefings be done?

Debriefings serve a different purpose as the process of data collection unfolds.

At the outset of data collection, debriefings are one-on-one (the lead researcher interviews the interviewer) and largely procedural in content. In our studies, these early debriefings have been used to learn from the interviewer what could be done to improve the process of data collection. We ask the interviewer questions such as: Is it feasible to find and interview respondents in a private setting (or is the community trailing after the interviewer-respondent pair to listen in on the interview)? Did recording devices work (let’s have a quick listen and upload the recording)? Are the consent forms understandable and did you have them signed or fingerprinted (let’s put them in this waterproof folder)? Are the instruments too short or too long? Do we have concerns about respondents growing tired or bored in an interview and if so, what do you think we should do about this informant fatigue? The earliest debriefings also allow the research lead to gauge the interviewers’ strengths and weaknesses as both interviewers and qualitative researchers. Did the interviewer appear interested, observant and engaged in the data collection activity? Did they probe on valuable lines of inquiry? Did they feel capable of shifting the interview back on track if it digressed in a manner that was not informing the research question? Did they capture non-verbal cues? What would they like from the research lead in terms of troubleshooting through a difficult process? Early debriefings are a means to ensure that the messages conveyed during trainings whether procedural (getting consent) or scientific (probing) are gelling among the data collection team, and to refine or reinforce these messages if they are not.

Once roughly a quarter of the data collection process is completed, we typically conduct a refresher training on the craft of high-quality interviewing.^{1 4 11} This presentation often involves distributing interviewing tips and tricks (see box 1: Interview tips, a refresher), and asking data collectors to share with the group one challenge they have encountered in the preceding interviews and how they overcame that challenge.

Box 1. Interview tips,* a refresher.

Adhere to ethical principles

• Ask for consent.

• Ensure privacy throughout the interview or focus group discussion.

• Convey in your actions and your questions that you respect the respondent’s autonomy.

Remove any mystery about the recorder/recording device

• Put the recorder within reach of the respondent.

• Tell the respondent they can turn it off at will and show them how to do this.

• Assure the respondent that only researchers will listen to the recording.

Use all senses to capture details

• Recognize pauses long and short.

• Capture what is spoken and unspoken (gestures, glances, fidgeting, fear, smiles, sincerity, pride).

• Note the smells, sights, the ‘texture’ of the interview.

Keep a conversation comfortable

• Start simple.

  • Ask uncomplicated, unintrusive questions.

• Be prepared and open to responding to questions about who you are and why you are there.

• Avoid double-barrelled questions.

• Know when to pause.

• Give time for responses.

• Refer back to comments or phrases made by the respondent in the course of the interview.

Follow the golden rules of great interviews

• Avoid the temptation to interrupt the respondent.

• Use open-ended questions and probes.

  • Don’t attach your interpretation to a response.

• Ask "remarkable questions in an unremarkable tone".*

• Do not judge—not with your voice, body or face.

Avoid scientific jargon

• Words and phrases like ‘plural health systems’, ‘structural violence’ and ‘stigma’ "sap the power and beauty of plain language".*

Be reflexive, be conscious of your role in this endeavor

• Memo how you, as the human being you are, shaped this interview.

End every interview with this question: ‘Is there anything I should have asked you that I did not ask you?’

• If the respondent offers some suggestions—ask those questions!

As data collection progresses, the nature of the debriefing usually shifts. Procedural questions become less necessary as processes have become routinized. There is also less one-on-one engagement between the research lead and individual interviewers in favor of a debriefing session, which resembles a focus group (with the research lead serving as both a moderator and notetaker). During the debriefing session, each interviewer provides a 2–4 min summary of their interviews or focus groups, with a special interest in describing key points or new findings from their interview. Conversations regarding triangulation (comparing and contrasting findings across data collectors or data collection methods) and topic saturation (the point when similar ideas and insights are heard again and again) typically begin to emerge in this phase as interviewers are encouraged to jump in when they could contextualize, confirm or dispute a piece of information based on their own interview. For both the research lead—and the data collection teams—this is among the most enjoyable and enlightening periods of the research process. Group debriefings prompt new ways of looking at an issue, help the interviewers gauge whether a follow-up interview is necessary, force research teams to think through how to reframe old questions or create new questions for subsequent interviews, and serve as reminders to the team that an interview is a short window into a person’s life during which contextualization of experiences occurs. The debriefing often sparks vibrant conversations among data collection teams about social desirability bias, thoughts on power, autonomy and decision-making within households and communities, and the role of the interviewer and research teams generally in terms of advocacy, human rights and social responsibility. Along with conversations on triangulation, the research team typically begins to discuss reflexivity, questioning how their social standing, personal experiences and inherent biases affect the nature of the interviews.

In the final phase of the debriefing process, the research lead is almost wholly removed from the process. The team nominates one of the data collectors to serve as moderator and another data collector to serve as a notetaker. The language of the group debriefing often switches from English (or other official language in the country) in favor of the local language (or the preferred language of the data collection team and the language of the interviews). During this phase, the research lead begins to build theories, devise an outline of preliminary findings and draw up a list of key local phrasings (emic terms) that may be valuable when presenting the research to stakeholders. In our studies, at this phase of data collection, we begin to develop a slide deck that will be later presented to principal investigators and others on the conclusion of data collection.

The final debriefing occurs on the day after the conclusion of data collection. During this session, the research lead presents the slides of preliminary findings to data collectors. Slides are edited based on feedback, and data collectors are invited to practice and then present portions of the presentation to an audience of academic, ministerial or programmatic peers.

Debriefings in Tanzania

In Tanzania, debriefings informed a fundamental shift in how the research team conceptualized the research question, ‘How do women and their spouses/support networks make decisions regarding where to seek care throughout the maternal care continuum?’¹² Conducted in 2011, our team initially sought to test a hypothesis that care seeking for childbirth was largely determined by factors such as cost, risk, distance and intrahousehold negotiation. Following debriefings after the earliest interviews, it became apparent that the main issues driving women and their communities away from facilities centered on issues of disrespectful maternity care by providers toward patients. Thinking that these earliest interviews represented an outlier, the data collection continued to rely on the initial data collection instrument, which did not emphasize patient–provider relationships and made no mention of disrespectful or abusive care. As data collection progressed, however, themes related to disrespect continued to emerge. The research lead presented the findings to the study’s principal investigators, who confirmed that this line of inquiry warranted pursuit. The research lead then began a literature review to identify studies that emphasized disrespectful care, and—together with the study team—modified the tools. Had the data collectors not been in regular contact with one another and the research lead, they may have disregarded or downplayed findings related to disrespect, they may have been unsure how to probe about disrespect and abuse, or they may have felt hesitant to undertake a line of inquiry that was not outlined in the tool (and may spark politically contentious debates). Along with allowing the research team to recognize and then triangulate findings related to abuse, debriefings also allowed for immediate comparisons of how male-female pairs describe their role throughout care seeking for childbirth. This immediate comparison (and the incongruences that emerged when comparing accounts across husband-wife pairs) not only generated animated discussions within the team, but also identified another new line of probing (related to births before arrival), and guided decisions regarding whether and when to conduct follow-up interviews.¹³

Debriefings in Sierra Leone

In Sierra Leone, debriefings strengthened our study by enhancing the research team’s reflexivity, ability to build rapport and approach to sampling. Data were collected in 2010 with the aim of examining how families understood and manage childhood illnesses.¹⁴ In the course of the earliest debriefings, it became apparent that both researchers and respondents were weary of the data collection endeavor. Data collectors said they were shaken following discussions about child illness and child death with respondents who represented the poorest of the poor. Debriefings presented an opportunity for the team to talk through their anxiety and devise coping strategies collectively. Data collectors also described challenges of building trust with respondents, given strained relations in the wake of the country’s civil war. Several respondents were frightened by the audio recorder, concerned that their voice may be shared with a much wider audience (or used to inflict harm on them or their families). Many community members were also bothered by the presence of outsiders (the data collection team) in their communities; expressing incredulity that outsiders would travel long distances to ask about child health. The data collection team used debriefings to reconsider how to best present the team and explain the purpose of the research to community leaders (in a manner that would ensure all involved that this was a peaceful endeavor, that there was no ill will or underhanded intention of the data collection team toward the community). In terms of qualitative methods specifically, debriefings served as an opportunity to reiterate messages conveyed in the data collector training. Many members of the study team had more experience with quantitative rather than qualitative data collection, so there was a tendency at the outset of data collection to use interview guides as surveys—asking questions in exactly the manner they were written with no probing. Debriefings allowed the research lead to reiterate the open nature of interviewing, and provided a forum for data collectors with more qualitative experience to demonstrate how probing is best done. Finally, debriefings allowed the team to identify respondent types whose insights could inform the research question, but who were not initially a focal group for the study (first wives, mammy queens (female leaders), spiritual healers and traditional birth attendants).¹⁵ These individuals were not initially identified as key informants, but their essential role in deciding whether and when to take a child to a health facility emerged in the earliest interviews and compelled the team to change course in favor of including these individuals.

Conclusion

Systematic debriefings are a necessary complement to more conventional qualitative approaches. Debriefings make it possible to enhance the adaptable, thoughtful and empathetic-yet-questioning nature of qualitative research among data collection teams (thereby improving both the quality of data collected and the capacity of those collecting the data), to correct course in the event of unknowable changes, insights or challenges in a given context, and to quickly share emerging data with stakeholders in programmatic, policy and academic spheres. We have outlined herein a series of steps to conduct debriefings and demonstrated how we have used debriefings in studies across two contexts. We hope this article sparks interest and debate in the literature in terms of how debriefings could be used to improve the quality of qualitative data.