Factor selection in configurational comparative analysis – guidance for implementation researchers

Abstract

Background

Configurational comparative methods (CCMs) are designed for the systematic comparison of case data to identify necessary and sufficient conditions for an outcome. The complex interplay of contextual factors, implementation strategies, and intervention components makes CCMs particularly suitable for implementation studies. Factor selection is a stage in CCMs during which researchers determine a limited set of variables of interest for analysis. Guidance that helps implementation researchers to select a manageable set of factors from a plethora of potentially interesting components is warranted but lacking, especially, for pre-data collection. We provide such guidance with an emphasis on (1) intentional data collection, and (2) broad engagement of interest-holders across different stages of factor selection.

Methods

Our suggested approach consists of four stages: (1) factor identification, (2) factor prioritization, (3) factor determination, and (4) factor operationalization. Throughout these stages, we highlight opportunities for engaging interest-holders.

Results

We illustrate all stages using experience from a multinational hybrid implementation-effectiveness trial in the field of infection prevention. First, we conducted a systematic review to identify potentially relevant factors. Second, we applied the Nominal Group Technique in meetings with interest-holders to prioritize previously identified factors. Third, we conducted two project workshops to decide on a final set of factors. Fourth, we operationalized factors into measurable units and assessed them using qualitative and quantitative methods.

Conclusions

We present a methodology and applied case example for pre-data collection factor selection for CCMs. The proposed method comprises four stages, each requiring implementation researchers to balance scientific rigor with pragmatism, while ensuring meaningful engagement of interest-holders. This guidance closes a gap in existing recommendations, which have primarily focused on factor selection during data analysis. Making CCMs more accessible and increasing their use should remain a central priority in implementation science.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s43058-026-00981-4.

Contribution of Literature.

• With this factor selection guidance, we offer concrete methodological support to implementation scientists interested in using Configurational Comparative Methods in their work.

• We provide a roadmap for factor selection prior to data collection, thereby closing a substantial gap and complementing existing guidance for factor selection after data collection.

• We contribute to the development of Configurational Comparative Methods in Implementation Science by exemplifying a systematic factor selection process and reflecting on challenges specific to the field.

Background

Implementation scientists agree that implementation is context-dependent [1–3]. Compared to traditional effectiveness studies, in which context is “simply tolerated” ([4], p. 3), implementation studies seek to understand and influence context [4]. This involves investigating multiple factors, such as contextual determinants or implementation strategies, and examining how they contribute to and/or affect study outcomes.

In such an investigation, estimating factors’ contribution to an outcome is complicated. In any empirical method, only a limited number of independent factors can be used to analyze an outcome [5], increasing the risk that a resulting model (e.g., from regression analysis) might not do justice to the complexity of an implementation context. While this limitation is not unknown to Configurational Comparative Methods (CCMs), they have nevertheless been described as particularly well suited to navigate the complexity encountered in real-world implementation settings. They are case-based methods, built for systematic comparison of factors that vary across cases (e.g., organizations), resulting in Boolean models [6, 7].

For example, Yakovchenko et al. [8] compared CCM-generated findings with those from a traditional regression analysis conducted on the same dataset [9] to identify implementation strategies that explain Hepatitis C virus treatment uptake. Rogal et al. [9] had reported the use of traditional methods and identified 28 out of 73 implementation strategies as independently significantly and positively associated with treatment uptake. Using CCMs, Yakovchenko et al. [8] identified five combinations of 10 out of 73 implementation strategies that collectively explained treatment uptake. The authors conclude that the results of such CCM analyses, revealing which combinations of discrete implementation strategies were critical to observed outcomes, may be useful when planning and conducting implementation processes, as recommending fewer implementation strategies is more practical, efficient, and cost-effective [8].

Characterization of CCMs

Configurational comparative methods are designed to investigate factors and their complex interplay in real-world settings. In CCM language, factors are the units analyzed in relation to an outcome, like variables in traditional analysis methods. Factors take on specific values, representing conditions, combinations of which constitute configurations. Translated to implementation terminology, factors of interest to CCM analysts could be implementation determinants for a specific implementation outcome. Their manifestation – as barriers or facilitators – represents the conditions, and a combination of several barriers and facilitators constitutes a configuration. Similarly, a combination (configuration) of multiple distinct implementation strategies (conditions) could contribute to implementation or patient outcomes of interest (e.g., 8, 10].

When using CCMs, case data – derived from individuals, organizations, or systems – are compared to identify conditions that are necessary and sufficient for an outcome. A necessary condition must be present to produce an outcome. When a condition is sufficient, it always generates the outcome, but alternative conditions may also produce it [7, 11]. To model configurations across different cases, CCMs infer causal patterns of an outcome in line with regularity theories of causation by drawing on Boolean principles [6]. Accordingly, the two complexity dimensions are conjunctivity and disjunctivity. Conjunctivity describes how two or more conditions must be combined for an outcome to occur (or not occur), expressed with the Boolean operator AND (*). Disjunctivity explains that multiple alternative pathways of conditions can lead to the same outcome (or its absence), expressed with the Boolean operator OR (+).

Two methods dominate the CCMs family: Qualitative Comparative Analysis (QCA, 11–13], and the more recently developed Coincidence Analysis (CNA, 14]. They share mathematical foundations, are suited for equal data types, and build on the Boolean principles explained above. However, they differ in their model-building algorithms, potentially yielding different findings [15]. As different advantages have been reported for CNA [6] we chose it as the central method for this work.

CCMs in implementation research

CCMs emerged in the 1980s as a method for data analysis in the political and social sciences and have since found their way into health services research, including implementation science studies [6, 11]. A recent example is a study exploring feasibility determinants for delivering an adapted version of trauma-focused cognitive behavioral therapy in Western Kenya [16]. The researchers found that moderate and high feasibility levels were associated with implementation climate and leadership, and a strong clinical supervisor-supervisee relationship. The authors conclude that CNA helped contextualize study findings by identifying a distinct set of determinants linked to perceived trauma-focused cognitive behavioral therapy feasibility across contexts. This may subsequently inform more refined implementation strategy selection in similar contexts.

Investigating implementation determinants, their interactions, and ways to navigate this complex interplay through implementation strategies has long been a key interest for implementation scientists. While well-suited for CCMs, factor selection, i.e., deciding which factors to integrate into CCM analyses, remains a crucial methodical challenge for which little guidance exists. Healthcare contexts contain a multitude of determinants, among others represented in the 48 constructs of the Consolidated Framework for Implementation Research (CFIR, [17]). Similarly, various implementation strategies are used in most healthcare implementation settings, sometimes exceeding the 73 strategies already contained in the Expert Recommendations for Implementing Change (ERIC) compilation [18]. This complexity makes factor selection particularly difficult.

Various CCM scholars [16, 19] agree on the persisting factor selection challenge, even though this gap was described in landmark CCM literature published two decades ago [5]. Factor selection is a purposeful decision process, as only a limited number of factors can be analyzed. This is due to the limited diversity problem [20], which arises as the number of distinct combinations of conditions increases exponentially with the addition of factors to the analysis. The greater the number of logically possible combinations of conditions, the lower the proportion of those combinations observed in reality. Hence, the data set will have limited diversity when the number of possible combinations of conditions is high, due to the inclusion of many factors. To mitigate this issue, limiting the factors to those essential is recommended.

A prominent gap in current guidance exists for targeted factor selection prior to data collection. This increases the risk of collecting excessive data, much of which may go unused in analyses – a well-known and costly challenge in clinical trials [21]. The implementation literature provides many theories and frameworks usable to build hypotheses that guide data collection, including in CCM implementation studies (e.g., 8]. However, many of these frameworks (e.g., 17, 18] are complex, and researchers will need to pre-select factors from the frameworks resulting in their partial use. While this is common practice, merely considering scientific literature may not help avoid excessive data collection, as there is no common approach on which framework or theoretical components may be selected. With no such guidance, excessive data collection is not addressed with the current practice. To prevent valuable data from being underutilized, decisions about factor selection should not first be made in CCM analysis, but already when planning data collection. Ideally, these decisions reflect various interest-holder perspectives [22].

We propose an approach to CCM factor selection prior to data collection (i.e., not reflecting decisions made during the analytic process) and illustrate it with a multinational trial in infection prevention and control (IPC). Where applicable, we use the Standards for Reporting Implementation Studies [23].

Methodology

Our approach consists of four stages: (1) initial factor identification, (2) factor prioritization, (3) factor determination, and (4) factor operationalization and measurement (Fig. 1).

Fig. 1.

Four stages of factor selection

Stage 1: Initial factor identification

The objective of stage one is to get a broad overview of factors that may be relevant for the analysis. In this stage, it is essential to avoid excluding any factors purely due to anticipated concerns about feasibility of measuring the factor. Different approaches can be taken to identify factors, including reviewing existing evidence in the respective field (e.g., systematic reviews); using theories, models, and frameworks, such as the CFIR; or defining study hypotheses. Further, qualitative methods such as focus groups or collective brainstorming can be used to capture this initial, typically broad range of factors. For example, participants may build hypotheses and theorize about why a certain outcome occurs. This process could be framed with thought-provoking questions (e.g., how has early intervention adoption been explained?), and the discussion be founded on expertise and experience drawn from within and beyond the field.

Stage 2: Factor prioritization

Stage two builds on the results of the initial factor selection and foresees a first reduction in the number of factors identified through stage one. All factors deemed important at this stage should be maintained, even if the factor set is still too broad for CCM analyses after completing stage two. Factor prioritization will likely be a dynamic group process in which integrating different perspectives will help elaborate on the work of stage one. Various criteria can serve as guiding principles and should be defined depending on research needs. These may include the perceived urgency to understand factors’ influence, anticipated benefits of working with factors, factor representation in dominant theories, and evidence indicating factors’ importance (i.e., when the scientific literature demonstrates robust factor effects). Individuals involved in factor prioritization may include experts knowledgeable of their theoretical basis and current evidence, those with specific implementation skills or experience, or professionals with decision-making authority in the implementation contexts studied. Hypotheses built during prior stages could be evaluated using different prioritization methods, including the Delphi consensus [24], Nominal Group Technique (NGT, 25, 26], card sorting [27], dot voting [28] of factors, traffic-light scoring [29], or focus group discussions [30]. This list is not exhaustive and the choice of prioritization methods should align with the research context and need, and the individuals involved.

Stage 3: Factor determination

The focus of stage three is selecting a final set of factors meaningful for the planned research, considering criteria such as study hypotheses, prevailing theories, perceived need, anticipated factor importance, or extant evidence.

Furthermore, pragmatic criteria likely influence this stage. Researchers must assess the feasibility of measuring each factor. Considering the administrative and logistic demands of collecting factor data, and whether this is feasible and ethical for those involved, is essential when deciding which factors to include. Questions that can help evaluate feasibility are:

• How will the researchers access the data source?

• Who will collect the data?

• How burdensome (time, budget, etc.) will data collection be for those collecting and/or providing data?

Importantly, CCM data requirements should be considered at this stage. The degree to which factors will vary across cases may guide factor decision-making. Finding patterns in the data that allow analysts to identify necessary/sufficient conditions distinguishing cases with from cases without the outcome requires case variation. If a factor is expected to be present in most cases (e.g., 90%), it will not contribute to the understanding of necessary/sufficient conditions for an outcome, as there will likely be no pattern in that factor contributing to this differentiation. Hence, factors not expected to vary across cases may not be included in CCMs [20].

The choice of methods for reaching a final factor decision may depend on the number of factors remaining to be reduced after completing the two prior stages. As a rule of thumb, the literature suggests integrating four to seven factors for a medium-sized sample consisting of 10–40 cases [20]. As previously, the methods to conclude stage three may involve a group process integrating various perspectives, this time, involving individuals with a realistic understanding of local practice in the field under investigation and of circumstances for data collection. Further, knowledge about available options for operationalizing and measuring factors is crucial. When balancing different perspectives, the pragmatic criteria outlined above should guide the group’s decision-making. While the methods applicable for stage two may also be used here, further techniques may include using a pro and contra list to display advantages and disadvantages of including a factor, followed by a discussion to determine if disadvantages can be navigated within the project.

Stage 4: Factor operationalization and measurement

After factors have been determined, researchers must operationalize factors and decide how to measure them. Operationalization entails translating abstract constructs (the factors) into measurable units and can vary in complexity. Objective factors, such as individuals’ socio-demographics, or structural information about an organization, are typically easier to operationalize and collect, e.g., through electronic surveys or administrative databases. In contrast, more complex factors, such as team dynamics or organizational culture, may require a deeper examination of options for operationalizing and measuring them. This process entails outlining alternatives for defining factors and for measuring them quantitatively or qualitatively.

This methodical choice is again characterized by a range of complexities.

For quantitative factor data collection, the broader literature and databases of implementation measures can help identify tools previously used in relevant settings [31–33] and guide decisions regarding tool appropriateness and feasibility. Key questions to consider include:

• Has the instrument been used in similar settings?

• What are the instrument’s psychometric properties?

• Is the instrument suitable for use in healthcare settings (i.e., pragmatic), or does its application require specific resources (e.g., knowledge, time, staff)?

• Is the instrument freely available, or does it come at a cost?

• Is the instrument available in the required languages, or will it need to be translated first?

However, while, for example, validated self-report instruments for implementation leadership [34] and perceived sustainability [35] exist, their use is not straightforward, as respondents may have difficulty understanding the questions or be subject to social desirability bias [36].

Data requirements are also central in stage four. If qualitative factors are chosen, researchers must keep in mind that the CCM requirement for factor data to be assessed across all cases necessitates agreeing on a uniform approach to qualitative inquiry a-priori. Furthermore, while qualitative methods may allow for in-depth examination of factors, social desirability may be amplified by the role of the bystander (e.g., interviewer) in qualitative approaches [37]. Moreover, qualitative methods are typically more resource-demanding, limiting their usability when assessing many factors or a larger sample.

Hence, operationalizing factors and preparing them for measurement is a balancing act, where considerations of data quality, data requirements, appropriateness of measures, and availability of project resources must be weighed against one another. This process ultimately informs the factor determination on how to work with a selected factor. Notably, this balancing act may necessitate revisiting earlier stages to either broaden the factor set again or revise previous prioritizations. The iterative factor selection process is visualized in Fig. 1.

Case example: the REVERSE study

We applied the factor selection process in the REVERSE (pREVention and management tools for rEducing antibiotic Resistance in high prevalence Settings) study [38]. Here, we provide a brief study description, followed by a detailed outline of the four factor selection stages applied in REVERSE in the results section.

The complete REVERSE implementation evaluation methodology is reported elsewhere [39]. REVERSE was registered with the “International Standard Randomised Controlled Trial Number” (ISRCTN, Nr. 12956554).

Study background

Designed as a cluster-randomized stepped-wedge hybrid type two effectiveness-implementation trial, REVERSE aims to reduce healthcare-associated infections in 24 hospitals across Greece, Italy, Romania, and Spain. Aligned with this primary aim, the trial integrates the implementation of two clinical practice bundles: Infection prevention and control (IPC) and antibiotic stewardship. Both bundles contain different elements requiring behavior change among various hospital staff members, potentially complicating their implementation [1, 3].

The practice bundle of interest to our CNA is REVERSE IPC, as detailed in Table 1. We use CNA to investigate the role of organizational readiness for change (ORC) in implementing REVERSE IPC. ORC is defined as the degree to which hospital staff are behaviorally and psychologically prepared to implement REVERSE IPC [40]. Our key interest is to explore if and under which conditions ORC is necessary and sufficient for REVERSE IPC implementation.

Table 1.

Bundle elements of REVERSE IPC, adapted from the REVERSE study protocol [39]

REVERSE Infection prevention and control bundle element	Description
Hand hygiene	Standards for hand hygiene, based on WHO standards “5 moments for hand hygiene” [41]
Contact precautions	Separation of patients; limiting patient transport/movement; use of personal protective equipment; use and cleaning of patient care equipment; enhanced cleaning of patient rooms
Isolation and cohorting	Appropriate placement of MDRO-infected patients; use of dedicated staff to exclusively manage isolated/cohorted patients
Active surveillance	Regular, targeted screening of patients for MDROs on admission and at pre-defined intervals, e.g., weekly
Environmental hygiene	Development and use of protocols for monitoring and auditing cleaning practices in patient areas
Outbreak management	Development and use of protocols and team processes to detect, investigate and manage MDRO outbreaks

Note: WHO = World Health Organization; MDRO = Multidrug-resistant organism

Results

Below, we present how the four stages of the CNA factor selection approach were applied in REVERSE. We also describe our interest-holder engagement at each stage.

Stage 1: Initial factor identification in REVERSE

To create an initial overview of factors relevant to understanding if and under which conditions ORC is necessary and sufficient to implement REVERSE IPC, we conducted a systematic literature review (SLR) of the health care literature [42]. The objective was to identify factors that have been investigated in combination with ORC (as correlates or predictors), or as mediators and/or moderators of the relationship between ORC and implementation. Conducting an SLR is resource-intensive and may not always be feasible. More pragmatic methods include scoping the literature, mapping relevant theories, brainstorming, and conducting interviews or focus groups with key informants (e.g., practitioners, end-users, researchers), making this stage adaptable across different study contexts.

We chose the SLR for two reasons. ORC is a highly debated construct, with previous work criticizing inconsistent ORC conceptualization, definition and measurement in past research [43, 44], leading to uncertainty about its importance [45]. Through the SLR, we anticipated finding the broadest range of alternative constructs previously investigated in relation to ORC in health care. Additionally, we had limited familiarity with ORC studies in IPC and aimed to include any relevant studies. An SLR was most likely to accomplish this.

Forty-seven studies met our inclusion criteria. We mapped the factors extracted from these studies to the constructs of the Theory of Organizational Readiness for Change (TORC, [40]) and the CFIR [17]. Based on the TORC, Shea et al. [46] developed the pragmatic 12-item measure Organizational Readiness for Implementing Change. This measure was selected to assess hospitals’ ORC in REVERSE, both for its pragmatic and psychometric advantages [47]. To maintain consistency across ORC conceptualization and measurement in REVERSE, the TORC lent itself as a central theory. Integrating the CFIR into the TORC allowed broadening the set of potential factors across various levels. Simultaneously, since the CFIR is one of the most applied implementation frameworks in health care studies [17], we sought to maximize overlap in terminology and concepts across included studies, potentially facilitating synthesis.

The SLR resulted in a list of 58 factors (see Supplementary Material 1). Factors synthesized from included studies were mainly related to individuals targeted by an intervention, the intervention itself, and resource availability. The full SLR results are reported elsewhere [42].

Stage 2: Factor prioritization in REVERSE

To reduce and prioritize our original pool of factors, we collaborated with the National Focal Points (NFPs) of REVERSE. NFPs are country-based teams of infectious diseases experts who operate as liaisons between the different REVERSE research groups and hospitals and are accessible for both sides as clinical and administrative resources.

To leverage NFPs’ knowledge and systematically prioritize factors, we conducted country-based, modified nominal group technique (NGT) meetings [26]. We chose the NGT for two reasons. Its structured format enables prioritization of relevant issues to different constituents. Simultaneously, the NGT has been reported to be less prone to dominance by single attendees compared to methods like focus groups, by ensuring that all voices are equally considered when reaching a decision among multiple interest-holders [25].

An NGT process consists of silent idea generation, sharing and discussing these ideas, and prioritizing a subset of ideas through a ranking procedure. Our modifications included using an online instead of an in-person format, and a virtual alternative (Microsoft PowerPoint) to a flipchart for notetaking (Table 2). To ensure sufficient factor reduction, we also adapted the ranking procedure and limited the number of ideas to be prioritized to five. Finally, we conducted a single – rather than two – ranking rounds [26].

Table 2.

Nominal group technique procedure used with REVERSE National Focal Points

Step	Description
1. Introduction	• One researcher (L.Ca.) presented the rationale behind the meeting, including an introduction to ORC1, IPC2 implementation, and the meeting structure. • Nominal question: When you think about the REVERSE IPC bundle, what would be factors that you consider crucial for implementation success?
2. Silent notetaking of preferred factors	• A table with all 58 factors identified through the systematic review (stage 1) was shared, alongside an example or explanation of each factor. • Participants were instructed to silently write down ideas about which factors seem important for a successful implementation of REVERSE IPC practices. • No discussions were held at this stage; if questions were raised, participants unmuted and asked, or they used the chat function.
3. Round robin – each participant is given equal time and opportunity to share ideas	• Each participant shared their views of one selected factor that they deemed important. • No discussions were held until all views and ideas were presented. • Reported factors were written down in screen share mode on a blank slide, visible for all participants.
4. Clarification and discussion	• After all factors had been gathered in the round robin, attendees reviewed the whole list to assess whether: o Any factors were no longer deemed relevant and should be excluded post round robin o Any factors were missing (this did not necessarily need to be one of the 58 initially presented factors)
5. Factor ranking	• After clarification and discussion, each participant had time to individually select their top five factors. These were then shared in the plenary. • Participants allocated a total of 100 points to their top five factors, resulting in a list of five favorized factors including a priority ranking (unless all five factors were given an equal amount of 20 points). • Each factor’s priority points were then summed up across the four NGT3 meetings.

¹ORC: Organizational Readiness for Change; ²IPC: Infection Prevention and Control; ³NGT: Nominal Group Technique

Two REVERSE researchers conducted NGT meetings in June and July 2023 online via Zoom (Version 05.14.10, 48], with one researcher leading the conversation and the second taking notes. Four meetings, one per country, were held to ensure that factor selection reflected country-specific conditions. Between one and three participants attended NGT meetings, with one meeting having a single, one meeting two, and the remaining two meetings three participants. It is commonly recommended not to include more than 10 members per NGT [49].

All NGT meetings were held in English. At least one participant per NGT meeting was highly proficient in English, while others had good communication skills. The more proficient speaker helped others with rare translation challenges. Each meeting was structured as follows:

1. Introduction.

2. Silent notetaking of preferred factors.

3. Round robin – each participant is given equal opportunity to share ideas.

4. Clarification and discussion.

5. Factor ranking.

The NGT steps are detailed in Table 2.

Across the four NGT meetings, 20 out of 58 factors identified through the SLR were prioritized (see Supplementary Material 1). Factors of highest priority, i.e., scoring 50 or more points during the ranking, were: communication, leadership support, knowledge, resource availability, organizational culture, sustainability, level of barriers, and use of implementation strategies. The priority ranking of the different factors is shown in the Supplementary Material 1.

Stage 3: Factor determination in REVERSE

If the number of factors to be integrated into a CCM is sufficiently reduced through stage two, further steps may be redundant. In our case, the number of factors prioritized through stage two exceeded the currently recommended maximum number to integrate into a CCM with our sample size of 24 cases. Therefore, we conducted two 90-minute workshops with the REVERSE implementation research team to determine which of the 20 prioritized factors were of greatest interest and evidence, and highest feasibility to explore as potential difference-makers for REVERSE IPC implementation.

In these workshops, we first discussed the factors prioritized in NGT meetings (stage two), considering if any factors identified through the SLR (stage one) went missing in stage two, despite strong arguments in favor of their inclusion. As displayed in the Supplement, we decided to integrate implementation leadership given its widely recognized importance for implementation [50, 51], whereas the NFPs prioritized leadership attendance and support.

Further, we continued prioritizing factors from stage two, identifying candidate factors for clear exclusion. For example, we decided not to consider resource availability in REVERSE, because we anticipated challenges in defining it comprehensively, applicable to all 24 hospitals, as well as in collecting the corresponding data. These discussions also involved identifying clear priority factors for inclusion in the REVERSE-related CNA work, such as the complexity of the change. The 24 REVERSE hospitals were encouraged to select IPC Standard Operating Procedures (SOP, Table 1) according to local needs and resources. The IPC SOPs varied in complexity, and therefore this factor was foreseen to vary between hospitals. Complexity was also realistically and practically measurable, as the IPC SOPs were documented regularly within standard REVERSE data collection, which may also guide the final stages of factor selection.

The two workshops resulted in the selection of five factors for CNA. Next to ORC, the factor selection process led to the inclusion of:

• Complexity of the change.

• Implementation leadership.

• Perceived/anticipated sustainability of the change.

• Teamwork.

• Use of implementation strategies.

The factor reduction from stage one through stage three is visualized in the Supplement.

Stage 4: Factor operationalization and measurement

After selecting the final factors to analyze, the work on operationalizing them and preparing their measurement began. We operationalized the identified factors both quantitatively and qualitatively.

The Clinical Sustainability Assessment Tool [35] and Implementation Leadership Scale [34] are two examples of quantitative measures that we selected for operationalization. Both validated tools required translation into multiple languages, an additional effort to consider in planning measure administration. No ready-made tool was available to operationalize complexity of the change. Thus, we created a quantitative tool to rate the complexity of each IPC SOP. Two members of the clinical REVERSE IPC team rated each SOP according to eight pre-specified complexity criteria and discussed their ratings with two senior IPC experts to finalize each SOP’s complexity rating. These complexity scores were then summed up for each hospital, depending on which SOPs were selected for IPC implementation.

Use of implementation strategies was operationalized qualitatively, using the REVERSE Implementation Tool (RIT, 52]. The RIT was part of regular data collection and served as a tracking tool for hospitals to plan and document their implementation, including implementation strategies based on identified barriers. For each hospital, we assessed, in duplicate, whether selected implementation strategies matched the identified barriers.

Finally, we also decided to remove a prioritized factor. While teamwork was among the factors relevant to both the research team and NFPs, we failed to satisfactorily operationalize this factor for our infection control context and link it to an existing, sufficiently pragmatic measure for use in REVERSE, among others, due to insufficient resources available for using qualitative methods. Therefore, we decided not to include teamwork in REVERSE.

Once we operationalized and measured all factors, we followed the steps outlined by Whitaker et al. [6] to conduct CNA. The results are reported separately [53].

Interest-holder engagement

Using CCMs provides opportunities for interest-holder engagement [16]. This also applies to CCM factor selection, as outlined in the following. Our reflections may be complemented with more general guidance on interest-holder engagement in healthcare research [54, 55].

In the first stage, we decided against interest-holder engagement in validating our SLR methodology and findings. This was due to our use of broad inclusion criteria, leading to the eligibility of a wide variety of healthcare studies and, consequently, to a comprehensive initial pool of factors. We perceived interest-holder engagement as not needed at this stage. Narrow inclusion criteria may require interest-holder input on whether the initial factor pool is sufficiently broad, diverse, and relevant for the scope of the study. Regardless of the method for generating an initial factor pool, various individuals can be involved in deciding on next steps.

For stage two, we conducted NGT meetings with the four REVERSE NFP groups. In operating at the interface between REVERSE research teams and participating hospitals, they, experts in IPC and knowledgeable about the REVERSE hospitals in their respective countries, were familiar with the project and well-suited informants. Involving local subject matter experts who also understand research-related challenges, e.g., the need for scientific rigor, or adherence to timelines, is advantageous in the prioritization stage, as balancing these two realities is essential to factor prioritization and will inevitably influence decision-making. In our case, the already well-established relationship with most NFP representatives was a further argument for their involvement, allowing to conduct the NGT process in a comfortable, collegial atmosphere. We also knew from this collaboration that language would only be a minor barrier. This is a central consideration when prioritizing factors across multiple cultures and languages.

Stage three entailed factor determination in the REVERSE implementation research team. We again decided against further interest-holder engagement, both due to limited resources and project timelines that made it important to progress to stage four to initiate data collection in due time. Otherwise, it would have been possible to work toward further validation of the finally selected factors across NFP teams, to, e.g., receive input on how these factors are perceived in different countries. Furthermore, other interest-holder perspectives could be considered in this stage, e.g., other researchers involved in the project or experts in a given field.

In stage four, factor operationalization and measurement, we consulted with REVERSE project partners and external experts. For instance, we collaboratively created the complexity rating tool with members of the REVERSE IPC work package, and we contacted experienced teamwork researchers for input on possible measures to use.

Discussion

The limited guidance available for factor selection prior to CCM data collection in implementation research motivated this methodological work. We present a four-stage model of prospective CCM factor selection and illustrate its application in a multinational hybrid trial. Researchers using this guidance should note that, although the stage-based approach suggests linearity, realities of using the guidance may differ, since stages may overlap, require revisiting, or be skipped. For example, if a range of identified factors exists, initial factor identification may not be necessary, and the process could begin with prioritization. Hence, it serves as an overarching roadmap supporting implementation researchers aiming to include CCMs into their work. The intensity of activities in each stage may be adjusted based on resource availability.

While the approach outlined allowed us to distill a large pool of diverse yet interrelated factors into a manageable set of five, we also encountered challenges in moving through the four stages. Beyond the lack of guidance that sparked this work, these centered on the intercultural nature of REVERSE, and the lack of ready-made, high-quality measures for assessing factor data.

During NGT meetings, NFP groups from the four REVERSE countries prioritized different factors (see Supplement), potentially indicating that their perceived relevance for IPC implementation varies by national context, or that factor meaning and interpretation depend on culturally determined understandings [56, 57]. Nevertheless, the final factor set must be measured uniformly across cases, because CCMs, as case-based approaches, usually necessitate complete data. Consequently, administering identical measures across cultures complicates the interpretation of comparative research findings. When samples are small or medium-sized, researchers may use case knowledge to interpret results [58]. In our CNA, a basic understanding of hospital operations and culture, gained through site visits and project-related collaboration, supported a country-specific interpretation of study findings.

Another challenge affecting factor selection was the paucity of psychometrically robust measures. This challenge, noted in previous research [32, 33], is particularly important to REVERSE, since most existing measures originated in the United States, potentially representing values and norms that fundamentally differ from those characterizing Eastern and Southern Europe. This further reinforces the interpretative difficulties described above.

We sought to address this challenge by collaborating closely with NFPs and by translating validated tools, if available, using systematic forward-backward-translation that involved translators with cultural familiarity [39]. Nevertheless, previous calls for the development of stronger and more culturally adapted measures [57, 59] remain current, among others to facilitate study comparability and synthesis [36, 60], including those using CCMs.

A further central difficulty arose from the final factor determination in stage three, involving research team discussions that required thorough anticipation of challenges that could emerge during analysis. This assessment would have benefited from specific guidance for judging factor suitability, as our process is not designed to indicate the quality of the factors selected to explain the outcome studied. While, to the best of our knowledge, such guidance is lacking for the pre data collection phase, general recommendations for factor refinement exist and may complement the process outlined here. As these recommendations are rooted in the analytic stage, their application may also reflect the quality of pre data collection factor selection (i.e., whether there were undetected factors contributing to the outcome). This guidance, developed for targeted factor selection post data collection, and going beyond drawing on existing evidence and theory, is three-fold. The factor set can be limited to minimally sufficient conditions (msc) utilizing the msc function of the “cna” package in R Studio [8, 61, 62], which uncovers factor dependencies. It helps understand which factors may be carried into CNA due to their causal relevance, thus serving as an indicator of the appropriateness of the included factor set relative to the outcome studied. When conducting CNA, researchers may revisit the initial factor set if results are ambiguous, highlighting the iterative nature of CCMs. Berg-Schlosser and De Meur [20] describe a stepwise factor reduction by finding factor dependencies using discriminant analysis or confirmatory factor analysis. These pre-analytical results might inform subsequent factor selection. Alternatively, multiple factors may be logically combined, for example, by merging two different measures of implementation leadership into a single score for analysis [20]. The analysis may indicate that included factors do not adequately explain the outcome, i.e., if the results are ambiguous or difficult to interpret. In such cases, factors may be added post hoc. If more data is needed, planning data collection to ensure data sources remain accessible post analysis is advisable. We could not collect data post analysis, but doing so could have improved our understanding of certain factors [53].

Despite these challenges, this process facilitated early discussion and consideration of various factors. This opened the door to factors that were not initially on our radar as they were not part of prevailing theories, such as complexity of the change. This factor helped shape our work, and its importance was a central finding [53]. Most likely, complexity would not have been studied without this process, as the TORC does not list complexity as a main contributor to ORC or its outcomes [40].

Finally, factor selection is a central pillar of CCM work potentially underreported due to space limitations. Factor selection guidance can help structure CCM reporting and thereby improve transparency and replicability of reported findings. Further refining and adapting this guidance, especially for use in implementation research, would be a valuable contribution, offering an opportunity to engage scholars from different fields in advancing CCMs, which continue to gain popularity in implementation science.

Conclusions

With the methodology presented here, we propose implementation-specific guidance for CCM factor selection prior to data collection and lay the groundwork for further developing guidance in applying CCMs in implementation research. Factor selection is presented in four stages along with interest-holder engagement opportunities. Our recommendations may encourage thoughtful planning of data collection, ensuring that it will be analyzed and ultimately impact developments in science and public health. Thereby, we foster context-relevant investigations and meaningful use of resources – central pillars of implementation science and practice. Overall, the illustrated approach is to be considered as a roadmap open to user-testing, refinement, and feedback.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviations

ABS

Antibiotic Stewardship

CCM

Configurational Comparative Method

CFIR

Consolidated Framework for Implementation Research

CNA

Coincidence Analysis

ERIC

Expert Recommendations for Implementing Change

IPC

Infection Prevention and Control

MSC

Minimally Sufficient Condition

NFP

National Focal Points

NGT

Nominal Group Technique

ORC

Organizational Readiness for Change

QCA

Qualitative Comparative Analysis

REVERSE

pREVention and management tools for rEducing antibiotic Resistance in high prevalence Settings

SLR

Systematic Literature Review

SOP

Standard Operating Procedure

TORC

Theory of Organizational Readiness for Change

Author contributions

LCa, BA, and LCl conceptualized the methodology presented in this article. LCa and KB held the Nominal Group Technique meetings. LCa and BA drafted and finalized the manuscript. All authors provided their critical feedback to the manuscript.

Funding

All authors of this manuscript have either received funding from or are funded by the EU-project REVERSE. REVERSE has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 965265. The content of this article reflects only the authors’ views, and the European Commission is not responsible for any use that may be made of the information it contains.

Data availability

Not applicable.

Declarations

Ethics approval and consent to participate

Not applicable for the methodology reported here. The REVERSE study has received ethical approval from the Canton of Zurich (AO_2021-00078), and from the 24 hospitals’ ethics committees.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.