Embedding Big Qual and Team Science into Qualitative Research: Lessons from a Large-scale, Cross-site Research Study

Abstract

Background:

A major part of the HEALing Communities Study (HCS), launched in 2019 to address the growing opioid epidemic, is evaluating the study’s intervention implementation process through an implementation science (IS) approach. One component of the IS approach involves teams with more than 20 researchers collaborating across four research sites to conduct in-depth qualitative interviews with over 300 participants at four time points. After completion of the first two rounds of data collection, we reflect upon our qualitative data collection and analysis approach. We aim to share our lessons learned about designing and applying qualitative methods within an implementation science framework.

Methods:

The HCS evaluation is based on the RE-AIM/PRISM framework and incorporates interviews at four timepoints. At each timepoint, the core qualitative team of the Intervention Work Group drafts an interview guide based on the framework and insights from previous round(s) of data collection. Researchers then conduct interviews with key informants and coalition members within their respective states. Data analysis involves drafting, iteratively refining, and finalizing a codebook in a cross-site and within-site consensus processes. Interview transcripts are then individually coded by researchers within their respective states.

Results:

Successes in the evaluation process includes having structured procedures for communication, data collection, and analysis, all of which are critical for ensuring consistent data collection and for achieving consensus during data analysis. Challenges include recognizing and accommodating the diversity of training and knowledge between researchers, and establishing reliable ways to securely store, manage, and share the large volumes of data.

Conclusion:

Qualitative methods using a team science approach have been limited in their application in large, multi-site randomized controlled trials of health interventions. Our experience provides practical guidance for future studies with large, experientially and disciplinarily diverse teams, and teams seeking to incorporate qualitative or mixed-methods components for their evaluations.

PROJECT OVERVIEW AND CONTEXT

The opioid epidemic remains a serious and growing public health problem, contributing to significant mortality and morbidity (Center for Disease Control and Prevention, 2021), and imposing an economic burden of greater than $1 trillion (Florence et al., 2021). In response, the National Institutes of Health and the Substance Abuse and Mental Health Services Administration launched the HEALing Communities Study (HCS) in 2019 to test the effectiveness of the “Communities that HEAL” (CTH) intervention at reducing opioid-related overdose deaths in 67 highly impacted communities across four states: Kentucky, Massachusetts, New York, and Ohio (Chandler et al., 2020; The HEALing Communities Study Consortium, 2020). The CTH intervention uses a community-engagement approach to build diverse coalitions in each community to expand access to and use of evidence-based overdose prevention practices across healthcare, behavioral health, criminal justice, and other community-based settings.

In addition to testing the effectiveness of the CTH intervention, the HCS is also evaluating the CTH implementation process using a mixed-methods implementation science approach. As part of this approach, researchers from each research site (i.e., state) conduct semi-structured qualitative interviews with key informants to understand the important domains of the internal community context and external policy and systems context that can affect CTH implementation (Drainoni et al., 2022). As findings from various components of the HCS and the CTH implementation evaluation are beginning to emerge (Drainoni et al., 2022; Walker et al., 2022), we undertook the opportunity to reflect on our qualitative data collection and analysis approach to share lessons learned for designing and applying qualitative methods within an implementation science framework to improve understanding of how qualitative methods using a team science approach can be applied in large, multi-site evaluations.

OVERVIEW OF STUDY DESIGN

HCS is a multi-site, wait-listed, community-level cluster randomized trial with 34 communities randomized to Wave 1 and 33 communities randomized to Wave 2 (The HEALing Communities Study Consortium, 2020). Key informants (i.e., local/county health officials, criminal justice officials, medical providers, social service providers, etc., who are community coalition members of local stakeholders in substance use issues) from all HCS communities are recruited to participate in semi-structured qualitative interviews at four timepoints: baseline (i.e., prior to CTH implementation for Wave 1; contextual for Wave 2); follow-up 1 (i.e., midway through CTH implementation for Wave 1; contextual for Wave 2); follow-up 2 (i.e., end of CTH implementation for Wave 1; prior to CTH implementation for Wave 2); and follow-up 3 (i.e., 1 year after CTH implementation has been completed for Wave 1, midway through CTH implementation for Wave 2). To date, we have completed 382 baseline interviews with 389 participants, and 304 follow-up 1 interviews with 310 participants.

The qualitative implementation evaluation component proceeds in a cyclical manner through each interview timepoint (Figure 1). We have completed this process for two timepoints (i.e., baseline and follow-up 1) and will continue using this process for the remaining two rounds of data collection. First, an interview guide is drafted based on the RE-AIM/PRISM evaluation framework (Glasgow et al., 2019) and insights from previous rounds of data collection. Then, researchers and research staff from each site conduct interviews with key informants from their respective states. Data analysis begins with the drafting of a codebook, also grounded in the PRISM/RE-AIM framework, that is iteratively refined in each round of data collection through a cross-site consensus process that involves weekly meetings of the site qualitative leads to resolve difficult coding decisions; this is followed by a within-site consensus process. Once the codebook is finalized, interview transcripts from each research site are divided amongst the coding team from the respective state for individual coding. Emerging themes and changes to the codebook are documented in detail in a log, and the information is used for interview guide and codebook development in subsequent interview rounds. The baseline (i.e., prior to CTH implementation) interview guide as well as the interview and data analysis procedures have been published in detail elsewhere (Drainoni et al., 2022; Knudsen et al., 2020; Walker et al., 2022).

Figure 1.

Qualitative process for each wave of data collection.

PRACTICAL LESSONS LEARNED

Embedded within our qualitative evaluation of the CTH intervention are the notions of Team Science and Big Qual, both of which confer particular benefits and challenges in qualitative research. Team science refers to the leveraging of cross-disciplinary expertise to address complex scientific issues (Salas et al., 2018). By bringing together experts from different fields, team science blurs disciplinary boundaries and fosters a more inclusive and collaborative analytic process that can often enhance understandings of phenomena in ways that cannot be achieved with experts from a singular scientific field (Hall et al., 2012; Stokols et al., 2008). However, team science can create challenges for the qualitative research process, especially with building consensus and maintaining efficiency (Skillman et al., 2019; Vindrola-Padros & Johnson, 2020). These matters are further complicated when the qualitative research is Big Qual, or has data from at least 100 participants (Brower et al., 2019). While the large volume of data in Big Qual is beneficial for theory-building, innovation, and generalizability, the data collection and analysis processes are often time and resource intensive (Brower et al., 2019). And similar to team science, there are concerns with achieving consensus and ensuring data trustworthiness and validity in Big Qual (Hossain & Scott-Villiers, 2019).

The challenges posed by team science and Big Qual to the qualitative research process are especially salient in the CTH implementation evaluation, which not only contains data from several hundred participants, but also involves large teams of interviewers and coders (e.g., at baseline, a team of 28 interviewers and 25 coders) who are collaborating across four distinct research sites. Further, the CTH implementation evaluation also involves collecting data longitudinally and in different study arms, adding to the complexity (‘bigness’) of the dataset. Below, we highlight some of our successes and challenges, and discuss what we have learned from our experience so far with conducting a large-scale, multi-site qualitative implementation science study.

Researcher Collaboration and Skilled Leadership

Effective collaboration amongst all members of a research team is critical for ensuring consistent data collection and for achieving consensus during data analysis, both of which contribute to increase the dependability, credibility, and trustworthiness of our findings (Wisdom et al., 2012). Therefore, it was important to identify point persons at each research site and establish a process for collaborating early on. Given that researchers were geographically dispersed, coupled with the emergence of the COVID-19 pandemic, in-person collaboration was not possible. As such, we adapted to using virtual platforms such as Zoom for meetings and developed clear procedures for keeping track of group decisions and for organizing research documents. Furthermore, two experienced qualitative researchers from each site formed the cross-site qualitative analysis core (QAC), and they were charged with liaising between the QAC and the other interviewers and coders at their respective sites. This structure has helped facilitate efficient group decision making.

In addition to the QAC, each research site also designated a senior researcher as their site’s lead. Site leads varied with respect to their levels of qualitative expertise, but each shared a health services orientation to research and had experience with project management, design, and implementation of qualitative research. As a result, beyond facilitating the data collection and analysis process, the site leads were also critical in helping the interview and coding teams develop and flourish as they progressed through different stages of the implementation evaluation (Tuckman & Jensen, 2010). Furthermore, leads establish the group norms of their sites and provide coaching that help interview and coding teams perform at their expected levels.

Structured Processes for Communication

Another key success to our qualitative evaluation so far has been the use of structured processes for communication. At baseline, the QAC developed standard operating procedures to provide clear instructions on agreed-upon processes to all researcher team members. The QAC also developed procedures for sharing updates on data collection and analysis promptly and widely. Specifically, weekly meetings are held and facilitated by a senior qualitative researcher designated as the cross-site lead. To maximize the efficiency of these weekly meetings, the QAC came up with a process where even though all researchers involved in the qualitative implementation evaluation are expected and encouraged to attend these meetings, comments and feedbacks should be submitted in writing ahead of time to be presented at the meetings by a designated person from each site. This process facilitated discussion of changes and additions to the codebook, including voting when necessary to reach cross-site consensus. All disagreements could then be resolved through discussion with a tiebreaking vote by the cross-site lead. This process informed real-time updates to the written coding log - another structured process that has been critical in our research thus far to keep track of progress, decisions, and changes that have been made at any step of the data collection and analysis process. For example, during the data analysis process at baseline and first follow-up, issues with coding and coding discrepancies are added to the log (see Table 1 for an excerpt of log). These issues can then be brought up for discussion during either cross-site or within-site group meetings, and the date and outcome of the discussions can be added to the log, along with any requisite codebook changes. The cross-site teams utilized a password protected cloud-based drive (i.e., Box) to maintain confidentiality and allow for real-time access to the most current codebook and log.

Table 1:

Excerpt of Log of Coding Discrepancies and Codebook Updates

Description of Coding Issue/Comment	Identified by (Research Site)	Discussed by Qual Subgroup on (Date)	Description of Consensus Resolution	Codebook definition update (Date)
Determine whether housing broadly, and sober homes specifically are part of ‘health services environment’.	8 early coders	3/31/2020	Sober homes specifically are part of ‘health services environment’ and were added in the codebook definition. No decisions made for housing in general–will likely be a context-specific decision.	4/1/2020
Potential for significant double coding of ‘health services environment’ and ‘community risks’ since the latter includes references to overprescribing and other health services issues.	8 early coders	3/31/2020	Remove overprescribing and any other healthcare systems references from ‘community risks’ definition. Clarify that ‘community risks’ includes concrete services like transportation.	4/1/2020
‘Other coalitions’ prohibits coding for other agencies and organizations that exist in the ‘external context’	8 early coders	3/31/2020	Re-label ‘other coalitions’ as ‘other coalitions/ agencies/ organizations’	4/1/2020
‘Internal Context’ only refers to coalition members. MA and NY did not have formed coalitions at baseline and interviewed key stakeholders who may or may not end up as part of the coalition. These individuals neither fit into the internal or external contexts in the original codebook.	8 early coders	3/31/2020	Re-label ‘HCS community coalition member characteristics’ as ‘coalition member or key stakeholder characteristics.’ Remove ‘HCS’ from other labels. State in Internal Context parent definition that this group of codes applies to HCS coalitions and key stakeholders.	4/1/2020
Member roles are described in the definitions for ‘HCS community coalition characteristics’ and ‘HCS community coalition member characteristics. Unclear which to code member roles to.	8 early coders	3/31/2020	Restructure coding relationship so that newly renamed ‘coalition member or key stakeholder characteristics’ is a grandchild of ‘HCS Community Coalition Characteristics.’	4/1/2020

Structured Data Collection and Analytic Approaches

An interesting distinction between our data collection efforts and those of other qualitative research project is the amount of structure in both our interview guide and our interview process. While many interviewers on our implementation evaluation teams had prior experience conducting semi-structured interviews, it became apparent early on that our qualitative data collection process required a more well-defined interview guide than interviewers have used in the past. By incorporating specific question probes that interviewers were required to ask, the more directed interview guide helps to improve consistency throughout data collection while still allowing for flexibility in the interviews.

Also of note is our use of a deductive-dominant approach (Hsieh & Shannon, 2005) to data analysis for this study rather than a more traditional grounded theory approach (e.g., Glaser & Strauss, 1999), which might have been more common in some researchers’ prior work and in smaller studies. To support this shift, researchers met regularly to ensure shared understanding of the RE-AIM/PRISM framework and its alignment with the implementation evaluation interview guide that was used to focus coding discussions. Having a common understanding of this framework prior to data collection, and again before data analysis is critical to maintaining consistency in each round of coding, and to help coders reach consensus in the coding process given prior agreement about definitions and the codes themselves. For instance, in the consensus process as the codebook was refined, the QAC used the framework to guide decisions about the hierarchy of codes as parent, child or grandchild codes. Finally, the deductive-dominant approach also allowed for consideration of emergent new codes (e.g., ‘stories’ to capture anecdotal stories told by participants), yet kept the generation of these inductive codes minimal to remain focused on explicating the theoretical framework. This process was also important in guiding the expansion of inclusion and exclusion criteria as coding progressed (see Table 2 for Structure of the Codebook and Figure 2 for Coding Tree).

Table 2:

Structure of the Codebook

Sample Code	Definition, Inclusion Criteria, Exclusion Criteria, Potential for Double Coding
Health Services Environment [Child Code]	Definition: Statements about the availability, access to, quality of, evidence for, or need for health services related to substance use, mental health, primary care, recovery support services (e.g., recovery cafes, peer supports) or other treatment/prevention/harm reduction services. Sober/recovery homes and shelters are included as part of the health services environment. Inclusion criteria: • _Include statements about the availability, absence, existence or access to health services not related to or funded by HCS. • _Include statements about health services that are needed or need to be expanded. • _Include statements about the observed or intended results of health services provided in the community. • _Include statements about law enforcement related health initiatives (e.g., post-overdose outreach). Exclusion Criteria: • _Exclude health services funded through coalition’s existing (pre-HCS) resources and instead code those to Other Community Coalition Initiatives or Community Coalition Resources as appropriate. • _Exclude comments about the coverage of benefits in an insurance plan (e.g., Medicaid, commercial) and instead code those statements to Policy. Potential for Double Code: • _Passages that describe a lack of transportation services prohibiting individuals from accessing healthcare should be double-coded to Community Risks and Health Services Environment.
Public Perceptions of Health Services [Grandchild Code of Health Services Environment]	Definition: Statements about the broader community’s perception of access to, availability/existence of, need for or quality of health services including substance use, mental health, primary care, and recovery services. Inclusion Criteria: • _Include statements about the broader community’s perception of the availability, access to, or quality of health services. • _Include statements about the broader community’s lack of awareness of existing health services. • _Include statements about perceptions of health service quality, groups’ (i.e., different populations within the community) general preference or experience with health services. Exclusion Criteria: • _Exclude statements about the general existence of health services and instead code those to Health Services Environment. • _Exclude statements about an interviewee’s perceptions about health services and instead code those to Health Services Environment or Individual Coalition Member/ Key Stakeholder Personal Attitudes as appropriate.

Figure 2.

Coding tree follow-up 1 data collection.

Diversity Amongst Research Teams

One of the challenges that emerged during data analysis of baseline and the first follow-up interviews was related to the diversity in experience, training, and knowledge amongst our team of coders. Qualitative data collection and analysis trainings were held prior to the start of each round of data analysis to ensure a consistent approach to coding across all 25 coders. Qualitative trainings were used to review the goals of qualitative research, types of qualitative data collection, interviewing techniques, deductive coding and theme generation approaches, as well as training in the NVivo software that would be used for all project coding and analyses. Trainings provided the coding team with foundational and applied knowledge about the HCS study. However, these trainings could not eliminate differences in “insider knowledge” about HCS communities. For example, coders who were involved in other components in the HCS were more aware about the CTH intervention strategies chosen by a specific community, leading them to code transcripts differently than coders who did not have this information. To overcome these differences, the QAC had discussions on the epistemological stance of our coding and decided that coders should in general take a constructive approach to coding rather than an objectivist one (Chamberlain, 2014). In practice, this position meant that coders were advised to derive their interpretations and meanings based on their reading of the transcripts which involved a methodological tradeoff. While we gained consistency in our coding process, we may have lost additional insights that could have been gained from an objectivist approach. As small group secondary analysis focused on targeted codes would proceed after the completion of baseline coding and this secondary analysis would use a combination of deductive and inductive coding, we believed this would be the opportunity to consider these more nuanced insights.

Data Management and Data Sharing

To facilitate cross-site collaboration, it is important to establish reliable ways to securely store and manage the large volumes of data that the sites were collecting and analyzing. For HCS, RTI International serves as the data coordinating center (DCC). The DCC is responsible for data management and statistical support. After completing baseline coding, the original workflow for sharing of the qualitative data in HCS involved sites sending a single NVivo 12 (the coding software that the QAC selected for all qualitative analysis in HCS, as it was the tool most familiar and used across sites) file including primary coding of all transcripts for each round of interviews. The DCC then merge these files, and investigators could request code reports for analyses for manuscripts. The DCC would then supply these code reports as exported MS Word documents.

Two key issues merged with the original workflow. The first issue concerned sharing of personally identifiable information (PII) in the transcripts. The original workflow required sites to de-identify the transcripts prior to sharing with the DCC. However, sites were concerned that de-identifying the transcripts would diminish interpretability and requested to be able to send identifiable transcripts to the DCC. This change required the DCC to establish a revised data use agreement with the sites.

The second issue pertained to the code reports provided back to the investigators. The MS Word file exports of code reports do not retain the file structure that NVivo applies nor allow for examination of code overlap, making them unsortable by transcript identifiers and inefficient for in-depth analysis of the primary coded data. The DCC’s motivation for this approach was to preserve the data integrity by not sharing the full NVivo file, and to be better able to track use of the data. Moving forward, the QAC and the DCC have decided on several process improvements, including using cases to classify the data along key analytic variables (e.g., site, coalition role, study intervention assignment), and providing investigators with a limited NVivo data file that maintains the file structure and more readily supports secondary analysis to explore relevant code overlaps.

Saturation

Beyond the practical lessons described above, issues emerged around the concept of saturation that merits discussion. Saturation is a core concept in qualitative research (Glaser & Strauss, 1999), and refers to the idea that additional data collection will not produce new findings and is typically used to guide justification for ceasing data collection. For HCS, this concept was problematic to apply as our sampling strategy needs to achieve representation both within and across communities, while the working within the resource constraints of the study. These issues are common to all types of qualitative studies, yet there are additional needs for consistency in a multi-site study in terms of methods and timelines. Further, given our interest in the implementation of the CTH, we purposefully sampled key informants with heterogenous roles in their communities – raising questions about what level of saturation should be identified at (e.g., the role, the community, the site, the study intervention). Ultimately, our sampling approach was determined a priori and was designed to achieve broader study goals of understanding the context of implementation of the CTH (Sim et al., 2018). The strength of this approach is that it is well-oriented toward thematic saturation and code identification as well as framework-driven deductive analysis (Hennink et al., 2017; Saunders et al., 2018). Yet, these strengths must be weighed against the limitations of this approach; namely, that our data have more breadth than depth in any single community and are less ideal for developing theory. This limitation was acceptable for HCS given that the study’s goal was to test and explicate the RE-AIM/PRISM model rather than develop theory and is also aligned with our deductive-dominant coding approach. The impact of this limitation is also minimized in HCS due to the multiple rounds of data collection that can strengthen code refinement, as well as the additional data collection efforts that are part of HCS (e.g., surveys, fidelity reporting, case notes) that can help us triangulate our findings and develop in-depth case studies at the community level.

Challenges

Significant challenges arose in the IS qualitative evaluation related to its being one part of the larger HCS study. Many researchers on the larger HCS are experienced quantitative scholars with a more limited understanding of qualitative evaluation, which led to the need to advocate for sufficient resources to support the qualitative research activities. Additionally, competing goals and requirements that have taxed staff who are working on different aspects of the HCS may have drawn attention from the qualitative data collection and analysis process. Further, given the duration of the study, staff turnover occurred, and new interviewers and coders had to be added to the team throughout the process. While the complexity of the study was challenging to understand for new staff entering at different phases of the project, this challenge was mitigated by the established processes and trainings.

Considerations for Future Research

Given that the HCS is a multi-year study with multiple rounds of data collection, we have been able to incorporate what we learned during the baseline interviews and analysis process into our subsequent rounds of data collection and analysis. One particularly important consideration was the need to consider the timeframe for both our data collection and analysis processes. Due to the time required to obtain ethics approval and the need to quickly start the intervention in Wave 1 communities, baseline data collection occurred from late November 2019 through early January 2020. This resulted in interview schedules becoming compressed given limited interviewee availability over the holidays. In addition to scheduling interviews during a different part of the year for the next wave, our improved understanding of the time required to ensure consistency of coding across sites also led us to schedule additional time for the primary coding process for these new data.

An additional consideration for future research is related to analytic techniques for the qualitative data. So far, our teams have completed primary coding of 686 interviews across two rounds of data collection and anticipate this number will nearly double by completion over the next two rounds of data collection. Primary coding of each round of interviews, from codebook development to final coding, requires teams of 20-25 researchers approximately six to nine months to complete. While the teams have successfully sub-coded some themes for manuscript development, this process is resource intensive, and increasingly complex given the incorporation of study design elements (i.e., the start of the CTH intervention). In short, the HCS study has a Big Qual problem – how can we analyze the data into trustworthy, interpretable, and meaningful components? This challenge is an emergent one for qualitative research, as to our knowledge, there are limited qualitative studies of this scale. A critical component of the success of the HCS qualitative work is the level and duration of funding support for the study from the National Institute of Drug Abuse (NIDA). This level of funding, sustained for 5 years, was achieved by a large investment in a multi-site, multi-method study by NIDA, and allowed the sites to develop a robust process that facilitated collection and analysis of the Big Qual dataset. This situation has been extraordinary, and while qualitative work is becoming a more prominent feature of large-scale funded projects, the resources required to collect and analyze Big Qual datasets remain an obstacle.

Current qualitative methods may need to progress in order to lower the costs involved in managing datasets of this size and to work toward a more efficient and cognitively accessible process. Alternative approaches to traditional qualitative analysis, such as the breadth-and-depth approach (Davidson et al., 2019; Edwards et al., 2021), rapid analysis (Gale et al., 2019; Vindrola-Padros & Johnson, 2020), the matrix approach (Averill, 2002), qualitative comparative analysis (McAlearney et al., 2016), natural language processing (Abram et al., 2020; Crowston et al., 2012; Leeson et al., 2019), all offer potentially innovative methodological tool sets. However, a thorough understanding of the tradeoffs of these approaches is lacking, and deserves attention in future research to enable discovery using Big Qual.

Qualitative methods using a team science approach have been limited in their application in large, multisite randomized controlled trials of health interventions (Lewin et al., 2009; Mannell & Davis, 2019). While this paper reports the results of the experience so far of a large, multi-disciplinary, multi-site team, we are limited in that our focus is on a single study. We are hopeful that the perspectives we provide can inform future large-scale qualitative data collection and analyses projects that advance implementation science across settings. Incorporating qualitative methods is essential to understand intervention uptake and maintenance, particularly for etiologically complex phenomena such as the opioid epidemic. Our experience provides practical guidance for future multisite studies with large and experientially and disciplinarily diverse teams seeking to incorporate qualitative or mixed-methods components.