Implementation science research priorities for Universal Health Coverage: methodological lessons from the design and implementation of a multicountry modified Delphi study

Abstract

Delphi studies are rapidly gaining prominence in global health research. However, researchers’ modifications to the Delphi method are often not well-described or justified, limiting opportunities to systematically learn from these studies when the methods are applied to other topics and settings. This paper aims to describe an approach to implementing a modified Delphi study and reflect on the research process in the context of a multicountry study of implementation science research priorities to advance Universal Health Coverage (UHC). We review trends in the use of the modified Delphi method in global health research, outline our three-phased modified Delphi approach, and share reflections on five decision points for implementing the study: (I) identifying and recruiting participants for the expert panel, (II) addressing participant attrition between rounds, (III) justifying the most appropriate cutoff points, (IV) incorporating new items raised by participants in open-ended survey sections, and (V) ensuring maximum variation in perspective in the panel of experts. Insights from this work foster greater understanding of the underlying assumptions for, and interpretation of, ‘modified’ in modified Delphi studies. This study will encourage critical dialogue about points of methodological contention in Delphi methodology and thus are relevant for scaling the use of modified Delphi studies in public health, including global health research.

Key messages.

• Methodological gap: Delphi studies are used increasingly in global health research; nearly half of the articles included in systematic reviews of health research Delphi studies report using a ‘modified’ approach. The ‘modified’ concept is often elusive, or not well-justified, limiting opportunities for scaling lessons and methods.

• Methodological developments: We reflect on and draw out lessons from applying a modified Delphi approach in the context of a multicountry study to gain expert consensus on implementation science research priorities to advance Universal Health Coverage.

Introduction

The Delphi methodology was developed in the 1950s to aid decision-making through a consensus-building process (Barrett and Heale 2020). The approach engages experts with experience related to an issue where there is insufficient or contradictory information (i.e. uncertainty) for more informed decision-making. Delphi studies typically respond to one of three research question types: predictive (e.g. probabilities and likelihood of future developments), normative (e.g. desirability of future responses or actions), and instrumental (e.g. options for ‘shaping’ the future) (Gordon 2009).

The number of Delphi studies has increased in the fields of health, health systems, and global health research over the last 20 years. A systematic review of the application of the method found that nearly half of the included health research Delphi studies reported using a ‘modified’ approach (Boulkedid et al. 2011, Spranger and Niederberger 2023). The ‘classic Delphi’ method—characterized by (i) the anonymization of ‘expert’ participant responses, (ii) repeated and iterative questioning with controlled feedback, and (iii) the reintroduction of aggregated group responses back to the participant panel with the possibility for participants to respond in subsequent rounds (Dalkey 1969, Niederberger and Renn 2023)—is typically modified by researchers to increase flexibility, reduce time consumption, and facilitate real-time discussions (Keeney et al. 2006, Barrett and Heale 2020). Modifications have also been enabled by advances in technology (Niederberger and Renn 2023). Studies show that the modified Delphi method can be more effective than the classic method because it is more participatory and collaborative (Gustafson et al. 1973, Niederberger and Spranger 2020). However, researchers’ modifications to the classic approach are often not well-described or justified (Boulkedid et al. 2011, Fletcher and Marchildon 2014, Jünger et al. 2017, Taze et al. 2022), obscuring opportunities to replicate and systematically learn from these studies in other topics and settings.

In this article, we aim to reflect on and draw out lessons from applying a modified Delphi approach in the context of a multicountry study to gain expert consensus on implementation science research priorities to advance Universal Health Coverage (UHC). The research results are reported separately (Namyalo et al. 2025). Here, we describe the use of modified Delphi methods in global health research, outline our application of a three-phased modified Delphi approach, and share reflections on five decision points and challenges we faced while implementing the study.

The use of the ‘modified’ Delphi approach in global health research: findings from a bibliographic review

While the Delphi method is implemented in many fields (Barrett and Heale 2020), it has recently gained prominence in global health research. A rapid bibliographic search of the Scopus database demonstrates a steady increase in the number of Delphi research papers in the global health research literature over the last 24 years (Fig. 1).

Figure 1.

Publication trends since the year 2000 for global health research papers using the Delphi methodology. The rapid search was conducted in Scopus in May 2024; details of the search strategy to create this figure are outlined in the Supplementary data.

Delphi studies are uniquely relevant in health research in situations where standard research pathways may be logistically difficult (Nasa et al. 2021) and can be a useful tool for global collaboration and representation, e.g. in terms of priority setting or in the development of global health guidelines. Recent studies reveal trends in the use of the Delphi methodology in health sciences, although formal reviews of its application in global health research are lacking. Based on a review of 12 systematic reviews, Delphi processes conducted in the health sciences field usually involve two online rounds and one final round of in-person engagement (Niederberger and Spranger 2020). ‘Consensus’ in Delphi studies in the healthcare field is typically understood as the majority percentage of participants agreeing on a standardized item (Niederberger and Spranger 2020). The panel typically includes a wider representation of stakeholders beyond academia, including patients and caregivers (Khodyakov et al. 2020). However, studies lack consistent and transparent reporting of modifications. In response to this gap, the case below illustrates how our team reflects on and draws out lessons by applying a modified Delphi approach in the context of a multicountry study to gain expert consensus on implementation science research priorities to advance UHC.

Applying the approach: identifying priority research areas to advance UHC through implementation science

In line with the 2030 Agenda for Sustainable Development, UHC is a global priority; however, many countries are still far from achieving their UHC targets. Parallel to this need, there are numerous calls to increase the quality and quantity of implementation science. Implementation science is the scientific inquiry of the implementation of interventions (i.e. programmes, policies, and practices). It seeks to understand what interventions work, for whom, under what contextual circumstances, and whether these interventions are scalable in equitable ways (Peters et al. 2013, Di Ruggiero and Edwards 2018). Yet, only 5% of global health studies in the peer-reviewed literature harness implementation science, and even fewer focus on how implementation science can support progress on UHC (Ridde 2016). Implementation science can bridge this knowledge gap by producing robust evidence in real-world settings to inform priority interventions for UHC. An accelerated focus on implementation science can generate much needed evidence on scaling equitable and sustainable health system strengthening interventions that advance UHC.

The case we use to illustrate our application of the modified Delphi method aims to consolidate expert consensus and inform a research agenda for UHC using implementation science. We employed a three-round modified Delphi study design to identify and prioritize research gaps. The initial set of 64 research gaps was informed by two scoping reviews conducted by our team (Bhatia et al. 2022, Yanful et al. 2023) and supplemented by 10 papers that we identified through a search of Medline and CINAHL databases to ensure the most up-to-date information. We extracted and categorized UHC research gaps that could be addressed using implementation science. Expert panellists were purposively selected across geographic regions and from five categories: global health systems researchers, research funders or donors, global institution representatives, national and subnational health system policymakers and programme implementers (e.g. Ministries of Health and Finance policymakers and National Agencies), and civil society/nongovernmental organizations (e.g. UNICEF, Women in Global Health) to maximize representation of perspectives.

In Round I, we invited 272 experts to participate in an online survey and achieved a response rate of 34.9% (n = 95 participants). Research gaps ‘passed’ to Round II, if at least 75% of the participants considered the proposed research gap to be extremely or moderately important on a five-point Likert scale (score of 4 or 5). In Round II, we re-engaged the 95 participants to further consensus on the passed items and to identify new research gaps using open-ended responses from participants in Round I. The aggregated averages and participants’ previous responses were provided to them in this round. In Round II, the cutoff point to ‘pass’ to the next round was 85% agreement; the result was a streamlined set of 42 implementation science research priorities for progressing UHC across four topic areas (75.2% participant response rate).

In Round III, we engaged the same panel of participants in virtual workshops to discuss and select the top 20 research gaps from each topic, using a weighted ranking analysis. This synchronous meeting time also allowed participants to give feedback on the results in real time, which allows ‘disagreement, controversy, and conflict’ to arise and be discussed openly (Jünger 2023). The top 20 research gaps were shared by e-mail with the full participant panel (n = 95) for a final 5-min ranking activity to identify the ‘top 10’ (39.0% response rate). The detailed methods, results, and discussion of the top 10 final implementation science research gaps for advancing UHC are reported elsewhere (Namyalo et al. 2025).

Methodological reflections

Below we reflect on five decision points and challenges that we faced while implementing the modified Delphi method.

Participant recruitment

The quality of a Delphi study hinges on the experts involved. Identification and recruitment of participants is a known challenge (Steinmüller 2023). The purposive participant recruitment for our modified Delphi study was extremely time-intensive, and we found it challenging to strike a balance of participation across the five ‘expert’ categories (i.e. academics, funders, implementers, etc.) and geographic representation desired given the breadth of actors ultimately shaping UHC priorities. A major difference between one-time surveys and the multi-round Delphi surveys is that the expert panel is not randomly selected and therefore does not claim statistical representation (Okoli and Pawlowski 2004). We systematically employed three strategies to recruit participants: (I) purposively contacting corresponding authors of the research articles that informed the development of the Delphi survey items, (II) using convenience sampling from networks at the University of Toronto and University of Melbourne, and (III) using snowball recruitment to reach beyond these networks (Steinmüller 2023).

We selected participants to reflect heterogeneity across several domains, including their work organization type (e.g. funding organization and civil society), the multiple disciplines relevant to our study (e.g. UHC and implementation science), and geographic representation across the six WHO regions.

We found the application of multiple sampling methods (in our case, purposive, convenience, and snowball sampling) to be necessary to confirm the participant sample, specifically given a global scope, and recommend the use of multiple recruitment strategies to fellow modified Delphi researchers.

Participant attrition between rounds

The Delphi methodology relies on gaining consensus over multiple rounds, and participant attrition between rounds is another common challenge faced by Delphi researchers (Belton et al. 2019). We preemptively addressed high drop-off rates by allowing long response windows for each survey round, sending multiple reminders to nonresponders, holding the virtual workshop (Round III) at three separate times to account for time zone differences for our global panel, and adding a final 5-minute ranking survey to account for the lower attendance than anticipated in the virtual workshops. Revisiting the expert panel repeatedly for a modified Delphi study requires a delicate balance in having sufficient engagement to ensure that the study results are meaningful, while not reaching out so frequently as to cause ‘sample fatigue’ (Schmidt 1997, Hasson and Keeney 2011). This is true both in terms of the number of rounds employed and for repeated communications.

While we set out to conduct three rounds (two anonymous surveys, followed by one real-time virtual workshop), we added a very brief follow-up survey to the third round to aggregate the findings from the three workshops and to increase participation in arriving at the ‘top 10’ research gaps. We contacted the panel for a final input with hesitation, knowing that we would likely reach sample fatigue and did not want to overtax our panel.

Setting predefined points for survey termination and establishing efficient channels of communication to reduce respondent attrition are important modifications to establish a priori (Trevelyan and Robinson 2015). The research team should also establish expectations for contribution and time commitment for invited panellists upfront (Hasson and Keeney 2011).

Justifying appropriate cutoff points

Although ‘consensus’ in health research Delphi studies is typically understood as the majority percentage of participants agreeing on a standardized item (Niederberger and Spranger 2020), there is no universally agreed-upon cutoff point to determine that consensus (Hasson and Keeney 2011). Studies range from 51% to 100% (Barrett and Heale 2020), with some authors questioning the value of percentage measures instead of opting for the stability or consistency of a response across rounds to determine consensus (Hasson and Keeney 2011). This leaves substantial room for researcher interpretation to determine and justify the cutoff points—hence a modified approach.

For Round I of our study, we set a predetermined cutoff of 75% agreement (‘moderately’ or ‘extremely’ important on the five-point Likert scale) for items to be moved into Round II, justifying this based on previous studies (Barrios et al. 2021). However, in Round II, we found that a 75% agreement cutoff was not sufficiently restrictive. Firstly, this cutoff only eliminated 2 research gaps from the 58 posed to participants (a 56-point research agenda is not necessarily a helpful contribution), and secondly, some of the items in the second round were newly raised by participants and required more stringent vetting by the panel. Our team therefore internally compared how the range of research gaps changed when applying 75%, 80%, 85%, and 90% cutoff points (resulting in eliminating 2, 5, 16, and 34 items, respectively), with specific attention to how the cutoffs affected the newly included gaps following Round I. We determined that an 85% cutoff point allowed us to move forward with reaching consensus, without too restrictively diminishing the number of research gaps after the first round.

Given the inconsistency in guidance, our recommendation is for researchers to be as transparent as possible about the cutoff points they employed at each round, to clarify whether those decisions were predetermined, and to justify the decisions the team made in arriving at them.

Addressing new issues raised by participants

The quality of the evidence Delphi studies produce depends on the inputs available to the experts (e.g. systematic reviews and personal experience) and the methods used to ascertain consensus (Jorm 2015). In our study, we anticipated and grappled with striking a balance between the evidence-informed research gaps we presented to the participants in Round I and the suggested gaps that the expert panel identified as missing from this list which we incorporated in Round II.

Panellists in our study provided open-ended responses during the Round I survey, which we thematically analysed to add 15 new items to the Round II Delphi survey. In other words, some of the items presented in Round II were drawn from our published scoping reviews and passed the first round of vetting, and some resulted from our expert panel’s hard-won experience. There is little guidance in the literature for what should be done with data points in Delphi surveys which have received a different number of ‘rounds’ of vetting by the expert panel. We accounted for this by designating the items raised by participants as ‘NEW’ within the Round II survey so that participants were aware that these had arisen from open-ended responses from the panel in Round I (and that they had not seen them before). The new items could not be accompanied by the aggregate score from the previous round or the participants’ previous ranking. Besides providing these clarifications to participants, we treated all items equally in the analysis following Round II.

In grappling with this tension, we agree with Jünger (2023) that the epistemological approach of the team plays a critical role in how Delphi studies in the health sciences are designed, interpreted, and reported. Our team’s philosophy led us to keep the present evidence-based and expert-introduced items together (clearly designating which was which), allowing our panellists to weigh items as they saw fit.

Ensuring maximum variation in perspectives in the panel of experts

‘Experts’ in Delphi studies are understood as people with specialized knowledge, whether that is “operational, experiential, functional, or contextual knowledge” (Spranger and Niederberger 2023), but the concept of ‘expert’ is contested terminology in the Delphi methodology literature (Jünger 2023). Some Delphi researchers avoid using the term and instead opt to clarify the knowledge requirements of their panellists (Trevelyan and Robinson 2015). Other studies invite panellists to self-assess their competence (e.g. how certain they are in their assessment of the material), which can introduce issues around self-assurance or over-confidence (Steinmüller 2023). Health-related disciplines tend to include a range of stakeholders in Delphi studies, including “affected parties, such as patients” in the expert panel (Spranger and Niederberger 2023).

We defined our expert panel using five categories of UHC research users and developers (described previously); however, we were unable to achieve even representation, with academic/research institutions most heavily represented in our sample (61%) and research funders least represented (3%) (full demographic results are given in Namyalo et al. 2025). The reliance on authorship to demonstrate ‘expertise’ could itself be perceived as problematic, as authorship imbalances in terms of gender and high-income country versus low- or middle-income country representation are rampant in global health publications (Abimbola 2019, Merriman et al. 2021). We also sampled for representation across geographic regions; here, using the modified Delphi method through virtual asynchronous approaches (and offering the synchronous workshop at three different times to account for time differences) allowed us to engage a range of stakeholders globally and maintain a geographically diverse panel through the three rounds.

Ensuring maximum variation in perspective and attention to power dynamics is especially relevant in global health studies using modified Delphi approaches, where representation of worldviews and knowledge systems outside of dominant ones enables more representative study findings (Carter et al. 2023).

Conclusion

This paper aims to elucidate the modified Delphi approach in the context of a global health study focused on implementation science for UHC. We describe an approach to making transparent the elements of the study that constituted a modification, share five methodological reflections on what we learned throughout the process, and offer recommendations for improving the use of the modified Delphi method in future global health studies (Box 1).

Box 1. Overview of recommendations for global health researchers conducting modified Delphi studies.

Our recommendations include the following:

1. Employing a variety of sampling methods;

2. Establishing clear communication pathways and time commitment expectations with panellists;

3. Transparent reporting of ‘consensus’ definitions and cutoff point decisions;

4. Disclosing the team’s epistemological approach to the method; and

5. Paying careful attention to power dynamics in modified Delphi research.

The ways in which modifications affect the overall results and robustness of Delphi studies, for better or for worse, remain largely unexplored (Spranger and Niederberger 2023), although some studies have compared the resulting findings from different Delphi variants [e.g. real-time Delphi versus classic Delphi (Gnatzy et al. 2011)]. One step toward addressing this is to have more thorough and consistent reporting of modifications made to the classic Delphi approach, e.g. through the Recommendations for Conducting and Reporting Delphi Studies (CREDES) (Jünger et al. 2017). Our recommendations are not meant to add to these rigorously developed reporting guidelines, but instead offer lessons that demonstrate why and where transparency of modifications when reporting Delphi studies is needed, thereby echoing the CREDES guidelines.

The Delphi methodology has been called ‘reminiscent of a methodological chameleon’, as the approach is flexible and modifications are frequently made depending on the ‘theoretical presuppositions, epistemological interest, and objective’ of the study at hand (Jünger 2023). This adaptability underscores the method’s strength and utility in accommodating a wide range of research needs and ensuring robust, context-specific outcomes. As researchers, we should also ask ourselves whether consensus and convergence are always appropriate for the research question at hand (Jünger 2023), as divergence and disagreement are critical to growth and expansive thinking in global health research and priority setting.

Supplementary data

Supplementary data is available at Health Policy and Planning online.

Author contributions

Ophelia Michaelides (Conception or design of the work, Data analysis and interpretation, Critical revision of the article, Final approval of the submitted version), Beverley M. Essue (Conception or design of the work, Data analysis and interpretation, Critical revision of the article, Final approval of the submitted version), Sumit Kane (Conception or design of the work, Data analysis and interpretation, Critical revision of the article, Final approval of the submitted version), Erica Di Ruggiero (Conception or design of the work, Data analysis and interpretation, Critical revision of the article, Final approval of the submitted version), Breanna K. Wodnik (Data collection, Data analysis and interpretation, Drafting the article, Final approval of the submitted version), and Prossy Kiddu Namyalo (Data collection, Data analysis and interpretation, Drafting the article, Final approval of the submitted version).

Reflexivity statement

Our research team is based at two institutions in high-income countries, but we bring lived experience from Canada, the USA, Australia, India, and Uganda. Our diverse perspectives and backgrounds informed the multicountry study we draw from for this methodological musing article from design to analysis, particularly in terms of participant recruitment and ensuring maximum variation in perspective for our Delphi panel of experts, which are two important points we ‘muse’ upon. The Delphi study design we draw upon was primarily developed by S.K., O.M., B.M.E., and E.D.R. and co-led by PhD candidates B.K.W. (from the USA) and P.K.N. (from Uganda), giving the students an opportunity to lead the research and writing process and introducing new perspectives. Our small team has met regularly throughout the course of both the research and ‘musing’ processes to share ideas in an environment respectful of nuanced perspectives, differing experience, and individual skillsets. Our team ethos and mission aligns with the Health Policy and Planning journal aims to reduce inequities in global health research.

Ethical approval

The study used in this methodological musing article was approved by the University of Toronto Office of Research Ethics (Protocol Reference # 00043390).

Conflict of interest

Although the coauthor B.M.E. is a section editor of the journal, there was no involvement with the peer review process for this article.

Funding

This work was funded by the Canadian Institutes of Health Research (202302PCS-498832-ICS-CEAA-12911) and the Centre for Global Health, Dalla Lana School of Public Health, University of Toronto.

Data availability

The deidentified data underlying this article will be shared on reasonable request to the corresponding author.