Skip to main content
PMC home page
International Journal of Environmental Research and Public Health logoLink to International Journal of Environmental Research and Public Health
. 2022 Mar 31;19(7):4141. doi: 10.3390/ijerph19074141

Expert Consensus on a Proposed Study Framework to Explore Factors Influencing Plasmodium knowlesi Malaria Preventive Behavior: A Modified Delphi Method Protocol

Nurul Athirah Naserrudin 1,2,3, Rozita Hod 1, Mohammad Saffree Jeffree 2,4, Kamruddin Ahmed 2,5, Mohd Rohaizat Hassan 1,*
Editor: Italo Francesco Angelillo
PMCID: PMC8998657  PMID: 35409824

Abstract

The increasing incidence of P. knowlesi malaria infection among humans is a public health threat. This zoonotic disease is challenging to eliminate owing to the presence of animal reservoirs. Understanding the factors such as the community’s belief, social context, drivers, and barriers can provide insights into malaria preventive behavior. It is crucial to improve the current preventive measures. This study aims to achieve consensus among malaria experts based on evidence from literature reviews and experts’ opinions on possible factors influencing malaria preventive behavior among communities exposed to P. knowlesi malaria infection. A modified Delphi study protocol was developed to gather experts’ consensus on the study framework to explore the factors influencing preventive behavior among communities exposed to P. knowlesi malaria infection. The framework is adapted from the ideation model, and it is integrated with other relevant theories and extensive literature reviews. We will use the modified Delphi protocol to reach a consensus. The experts will respond to each questionnaire item and a related open-ended questionnaire. Consensus is predetermined at more than 70% agreement on the items. We will use descriptive statistics and thematic analysis to analyze the data. All experts will remain anonymous to maintain the characteristics of a traditional Delphi study.

Keywords: Plasmodium knowlesi, zoonotic malaria, Delphi study, modified Delphi study, decision making, expert consensus, consensus, preventive behavior

1. Introduction

The Delphi method originated in the early 1960s in a study developed by the Rand Corporation and sponsored by the United States Air Force, where it was first established as “Project Delphi”, anonymously [1]. The Delphi method gained popularity in the mid-1990s in military and non-military applications. The technique was described as valid for decision-making processes [2,3,4,5,6,7,8,9,10,11,12,13,14,15] and gradually applied in the healthcare sector [2,7,9,11,14,15]. Since then, it has been used as a technique to evaluate concepts and develop study frameworks [7,13,14], research justification, and theoretical processes [5].

The Delphi technique has been described as a qualitative data analysis technique [12,13]. However, each Delphi study requires rigorous measures—using both qualitative and quantitative inputs [10,11]. For the Delphi method to support a constructivist inquiry, the positivist criteria (objectivity, validity, and reliability) and trustworthiness (confirmability, credibility, transferability, and dependability) used in the qualitative research need to be applied [9,10]. However, various Delphi techniques have been used to define group consensus, expert selection, the number of rounds, and reporting on the method and results [4,5,8,9,10,11,12,13,14,15]. Therefore, researchers need to define “consensus” and the “cut-off value” if an agreement rate is used [8]. Although there are numerous challenges and questions raised when using this technique, it is essential for achieving consensus on issues where none existed previously [5].

A consensus is a general agreement or a unanimity of opinions of a predetermined group of experts [1,4,9]. Experts are knowledgeable, competent, and representative of the field of inquiry [13,14,15,16]. Such a consensus is considered most reliable because it depends on the controlled feedback of experts via a series of questionnaires [3,5,6,10]. The cut-off value, which defined as the value from which a consensus was made, can vary between 20 and 100% in health sciences literature [17]. A threshold of more than 60% was found in the majority of studies [17]. Some studies in the field of medicine define a consensus outcome when more than 70% of the participants are in agreement with the items presented in a questionnaire [13,14]. However, items that do not meet this criterion are either revised or discarded following a re-rating in a subsequent round of the study [13,14].

A Delphi method has some notable characteristics [2,5,6,7,8,9,10], such as the anonymity of the participants and controlled feedback [5,6,7,8,9,10]. However, since then, the traditional Delphi method has been modified by, for instance, using a structured questionnaire in the initial round [7,11] and including interviews or focus group discussions on supplementing the responses [10]. Participant anonymity is a primary characteristic and an advantage of the Delphi method, as it can reduce the effects of dominant individuals—a common concern when using group-based processes [2,9,11]. The issue of confidentiality is facilitated by the geographic dispersion of the participants along with the use of electronic communication [4,10,15].

Controlled feedback in the Delphi process reduces the effect of “noise”, which is described as communication that occurs in a group process that can distort the data, such as deals within the group or a focus on individual interests rather than group problem solving [4,8,15]. Additionally, controlled feedback allows participants to rethink prior answers and make changes based on the information and results provided [3,4,9,13,14,15,16]. It consists of a well-organized summary of prior iterations, which are intentionally distributed among the participants to generate additional insights and clarify the information obtained in the previous rounds. Through multiple iterations, participants are expected to become more oriented toward problem-solving, offer more insightful opinions, and minimize the effects of noise [4,15]. The use of various statistical techniques further reduces the potential of group pressure for conformity [4,8,10]. Specifically, statistical analyses can ensure that opinions generated by each participant in a Delphi study are well represented in the final iteration because there may still be a significant divergence in individual opinions at the end of the round [1].

This study aims to achieve consensus among malaria experts based on empirical evidence from literature reviews and the experts’ opinions on possible factors influencing malaria preventive behavior among communities exposed to P. knowlesi malaria infection.

2. Methods and Analysis

2.1. Study Aim

We address the following research question: What factors should be included in the study framework to guide the exploration of factors contributing to zoonotic malaria preventive behavior in communities exposed to P. knowlesi malaria? The results of the study are expected to help form a study framework to guide future P. knowlesi studies in exploring the factors influencing the malaria preventive behavior among exposed community to this zoonotic infection and strategize future zoonotic malaria control measures. We employ the Delphi method to achieve this goal and reach a consensus on items that, according to the participants, who are malaria experts, can facilitate exploration of the possible contributing factors.

2.2. Procedure

This study employed a pragmatic approach [2,8,15,17]. Previous studies have used the traditional and modified Delphi approach to reach a consensus on their study framework and health management procedures [7,14,16,18,19]. The research methods were established by the co-authors, who are experts in the fields of epidemiology, infectious disease, and social science. The author and co-authors had virtual meetings and email communication, in which an agreement was reached regarding the inclusion and exclusion criteria for the study’s participants. The study framework was discussed to integrate different models and design the draft questionnaire. Once the questionnaire is validated, participants will be recruited for the pilot study, and another group will be recruited for the definite Delphi study. Descriptive analysis will be conducted using Statistical Package for Social Science (SPSS) for the Windows version 26 (SPSS Inc., Chicago, IL, USA) [20]. The thematic analysis method by Braun and Clarke will be used for the open-ended response [21].

Internet technologies, such as email and Google forms, will facilitate the Delphi procedure and aid the study [4,7,9,11,12,13,15,22]. Moreover, Google Drive (Google LLC, Mountain View, CA, USA) will be used to store the research proposal, study-related documents, malaria literature, and all the related information in further rounds between the researchers and participants. Inputs from panelists were stored on online data storage drives [7]. Furthermore, it is more practical to use electronic and online modes for the Delphi procedure to eliminate the need for travel, allow for the recruitment of geographically dispersed experts, communicate with the experts more conveniently, and increase efficiency in data collection [4,7,9,11,12,13,15,22]. Based on a review paper by Niederberger and Spranger [17], as there is no standard technique for Delphi study, the Standard for Conducting and Reporting Delphi Studies (CREDES) will not be considered.

The number of rounds is determined a priori to avoid false consensus and participant fatigue, which can result in a premature agreement to end the process [9,10]. The estimated completion period for the Delphi study is five months, with each round lasting four weeks. After the initial invitation, a reminder will be sent, approximately every seven days [13], to track the responses [10,13,15]. Scholars generally recommended two to three rounds [2,3,8,10,13,14,17]. Accordingly, we opted for three rounds, as this is considered optimal [14,15,23]. The criteria for terminating the Delphi process and achieving consensus were predetermined as greater than 70% agreement on the questionnaire items [17].

We will employ a structured questionnaire based on the possible factors impacting malaria preventive behavior. These will be used as items in the proposed study framework. The questionnaire will include items for the theories, comprising of knowledge, beliefs, environmental barriers and drivers, and attitudes that contribute to P. knowlesi malaria preventive behavior. These factors were identified based on the literature and systematic reviews undertaken by the co-authors and presented at a recent conference. We assume that human behavior is influenced by both internal factors, such as the background and attitude, and external factors, such as environmental drivers and barriers. Thus, the proposed conceptual framework is adapted according to the ideation model of communication and behavior change [24], the explanatory model of Arthur Kleinman [25], and Murdock’s model of theories of illness causation [26]. The ideation model has been used to guide the evaluation of health communication intervention and strategic design of a health program [24].

The initial challenge in developing the questionnaire is defining the questions to obtain accurate and valuable feedback from experts. The questions have to be concise and direct, have a logical connection, and encompass the required information without being too time-consuming for the respondents [4,5,8]. We have created three headings (or domains) based on the three models, under which all related items in each model were included. On the basis of this initial items pool, we formulated the Round 1 questionnaire. We will use structured questions, for example: “Should this item be included to explore the factors influencing malaria preventive behavior in communities exposed to P. knowlesi infection?”. The participants will be given the opportunity in Round 1 to add items based on their opinions to generate more comprehensive items for the study framework. Each survey is estimated to take 20–40 min to complete. Participation in iterative rounds allows participants to review their own and collective answers before finally submitting their responses [4,10,14,16,17]. The participants will answer the questionnaire on a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree) [3,16,18]. The ratings will be grouped into three levels: low score (1–2), average score (3), and high score (4–5) [18,27]. In addition to quantitative responses, participants will also be allowed to provide comments or justify their choices on the items (qualitative data) [3,4,5,9,12,15,16,27]. They will be asked to provide recommendations or suggestions regarding any additions or deletions to the list of proposed items in the questionnaire. A consensus is defined as greater than 70% [17] of the participants agreeing (answer “3”), strongly agreeing (answers “4” or “5”), or disagreeing or strongly disagreeing (answers “1” or “2”) with a given item [13,14]. This will allow the ordinal data to be described using percentages of responses in each category. In addition, median will also be measured to achieve the consensus on the items. The employment of a median to measure the central tendency from ordinal data has been used by some researchers to reflects the convergence of opinion because the distribution of ratings is generally skewed [4]. A higher median suggests a greater degree of consensus [4].

2.3. First Step: The Problem Statement

The first step is to explain the research problem and define the questionnaire to be sent to the experts [3,9,10,21] (Figure 1). The threat of P. knowlesi malaria to humans has received significant attention in the World Malaria Report [28], in discussions among malaria experts in the Evidence Review Group on P. knowlesi malaria [29], and from the World Health Organization’s (WHO) Malaria Policy Advisory Group (MPAG) [30]. In the last few decades, the increasing number of P. knowlesi malaria cases in human has become a public health threat [29,30,31,32]. Its widespread geographical distribution in South East Asia [33], the disease burden presented by various disease outcomes from asymptomatic infections (that can only be detected by using more sensitive and specific molecular screenings), and severe disease and reported fatalities strongly indicate the need for future studies to improve our understanding of P. knowlesi malaria [29,30,34,35]. This is further compounded by the fact that it is a complex disease that is challenging to control [28,29,30]. Exposure to infection is highly significant among agricultural workers, males, adults, and individuals working near forests [29,30,31,32]. The modeling of malaria is arduous as the Plasmodium parasite co-evolves with the vertebrate host to survive [36]. The present study focuses on human behavior as the affected host. Understanding human behavior and social aspects in malaria control could help sustain the health program [37]. Human behavior is complex, as it encompasses daily life and is influenced by various factors [37]. Given the need to explore the social context, attitude, and other contributing factors that can influence human behavior, it is essential to build a study framework to explore these possible factors of malaria preventive behavior. The framework could be a theoretical lens to facilitate social and behavior change programs toward a sustainable malaria control program.

Figure 1.

Figure 1

Open in a new tab

The steps in the modified Delphi study for evaluating the proposed study framework.

Based on an extensive review of the literature on malaria, a study framework was designed as a theoretical guide for future studies. Two systematic reviews were conducted to identify the vulnerability factors of P. knowlesi malaria infection among human subjects (registered under PROSPERO: CRD42021243374). The first systematic review identified the background factors influencing the risk for P. knowlesi malaria, including being male, adult, plantation workers, living near the forest, and history of traveling to the forest. In contrast, the second systematic review identified the human behavior, perspectives, and activities at the risk of exposure to P. knowlesi malaria. This systematic review was reported as an oral presentation in the recent International Conference of Public Health 2021 (https://drive.google.com/file/d/1Sn5lUw-NJbh-BA-S5JO-E9mSkIMQJgnU/view), accessed on 15 August 2021. It has also been registered under Prospero: CRD42021251323.

We propose a metatheory to answer the research question. We propose the use of the ideation model to identify the factors that predict behavior [24,38,39]. It is an evidence-based predictive model for behavior change [24,38,39]. Behavior is influenced by multiple social and psychological factors and environmental conditions that facilitate behavior [24,38,39]. Ideation factors are grouped into three categories: cognitive, emotional, and social. These elements combine to affect behavior, similar to how multiple risk factors affect the probability of contracting a disease [24]. When the ideation is used in programs, researchers measure the factors that apply to a person and then combine them to create an ideation score that predicts how likely a person is to adopt a behavior [24]. The higher it applies to someone, the greater their likelihood of adopting a healthy behavior [24]. When ideational factors are aggregated in this way, they are highly predictive of health behaviors [24,38,39]. We integrate Murdock’s model of local illness belief in our framework, as it allows for a deeper exploration of community beliefs related to illness, caused either by supernatural or natural causes [26]. The emic of disease etiology, symptoms, and treatment differs from medical professionals and is conceptualized by the explanatory model [25]. The proposed study framework is shown in Figure 2.

Figure 2.

Figure 2

Open in a new tab

The proposed study framework.

2.4. Second Step: The Selection of Experts

In our study, the recruitment of experts commenced in 8 August 2021 after the ethics board approved the study. In our Delphi study, we identified experts after an extensive literature review and peer-supervisory recommendations [21,27]. We used purposive and snowball sampling [13] to recruit academia and the public sector [7,8,10]. There is no consensus in the literature on the optimal number of participants in a Delphi study. Twelve or more participants is effective, while less than six participants are generally considered insufficient [9]. Thus, we opt for a sample size of 12, mainly as the Delphi study depends on group dynamics to achieve consensus [13].

The participants must meet the following inclusion criteria: (1) individuals who have published one or more studies on malaria in peer-reviewed journals, (2) individuals who have more than five years’ experience in malaria studies or management, and (3) individuals who are currently working or practicing in any research institution or have an administrative post in an organization. The sample will include individuals from different professions, such as researchers, lecturers, professors, directors, and the heads of departments of different organizations worldwide. Candidates who do not indicate an interest in participating in the modified Delphi study after receiving two emails within a two-week period will be excluded from the study. Potential participants will be contacted via email, informing them about the study, and will be given an overview of the modified Delphi study, its objectives, phases, and their commitment to reach consensus. Each expert will be sensitized to the objectives and methodology of the study by sharing the research proposal and providing assurance of anonymity. Furthermore, upon receiving the experts’ consent to participate in the study, a formal e-consent will be obtained to complete the recruitment process. All data will be stored in a password-encrypted computer in a locked office, following standard guidelines. The data will be destroyed after five years, as per standard guidelines [13].

2.5. Third Step: Round 1 of the Study

2.5.1. Questionnaire Validation

Before Round 1 commences, we conducted a questionnaire validation that included a content validation process and was face-validated by six English-speaking participants to ensure the questionnaire’s quality [40] (see Supplementary Questionnaire S1 for questionnaire draft). The questionnaire validation allows further modification, clarification, and augmentation to fit the study’s objectives [40] (see Text S1 for participants’ responses and opinions on the draft). After completing the questionnaire validation phase, a pilot study will be conducted. A pilot study is relatively uncommon in published Delphi studies [41]; however, we aim to ensure that our study has the necessary methodological rigor to produce high-quality research.

2.5.2. Pilot Study

A pilot study helps ensure questionnaire clarity before it is distributed to the Delphi participants by allowing comments from respondents and improvements based on their comments [41]. Thus, feedback from the pilot study is essential for suggestions regarding the retention or omission of items [41]. As suggested by Clibbens et al. [41], we propose adopting an approach of purposive and convenience sampling for the pilot study. A sample is purposive when the volunteers meet some of the predefined inclusion criteria of the complete study. Convenience sampling is employed when each volunteer is already in contact with the researcher(s) through a university. Seven people with experience in conducting malaria studies will be recruited for this pilot study. By using such a method for the pilot study, the participants may have a stronger motivation to respond. The output of the pilot study will be evaluated by the research team before conducting Round 1 of the modified Delphi study [42] (see Supplementary Questionnaire S2 for the pilot study questionnaire).

2.5.3. Round 1

In a traditional Delphi study, Round 1 begins with an open-ended questionnaire. In the modified Delphi study, we propose the use of a literature review, a structured questionnaire, and open-ended questions to justify responses and share opinions from the malaria experts (participants) [3,4,5,9,12,15,16]. In Round 1, data on participant demographics will be collected, including gender, location of current work, current job position, and the number of years in the field of malaria research. The experts will also be allowed to provide input on other items that should be included in the study framework. The responses collected in Round 1 will be analyzed individually and collectively. By using descriptive statistics and thematic analysis, the responses will be shared iteratively in the following rounds [2,5,9,10]. Only items that reach a consensus of >70% agreement with a median of 4 to 5 will be kept for Round 2.

2.6. Fourth Step: Round 2 of the Study

Only participants who complete their responses in Round 1 will be invited to participate in Round 2. In Round 2, each Delphi participant will receive a questionnaire by email and be asked to review the responses provided in Round 1, as summarized by the researcher; participants will also be able to revise their initial responses [5,13,14]. The results will include the overall score for each item using a measure of central tendency, median, and the percentages of each item. The open-ended questionnaire will be analyzed using thematic analysis and shared as a supplementary file to ensure data trustworthiness and the credibility of the study [12]. Hence, the results of Round 2 could be lengthy and detailed [8,12,13,14]. All items that do not reach a consensual median score of at least 3 will be discarded in Round 2 [13,14]. There will be no option for a free-text response (open-ended questions) in Round 2. A consensus will be reached, and the items will be evaluated based on the group’s collective responses. The responses that reach an agreement of >70% [17] with a median of 4 to 5 will be retained [21].

2.7. Final Step: Round 3

In Round 3, the consensus stage, each participant will receive an individualized and collective response comprising only items with an agreement of greater than 70% [17] and a median of 4 to 5 [21]. Three Delphi rounds—considered optimal to reach a consensus—were predetermined to enable an adequate reflection on the group’s responses [14,15,23]. The consensus is the “general agreement of a substantial majority” of the Delphi participants [27] of the items proposed by the researcher and the experts to be included in the study framework.

2.8. Current Status

The questionnaire validation process commenced on 20 August 2021. The draft questionnaire was distributed to a different subgroup of malaria experts. Upon completion of the questionnaire validation, the questionnaire were distributed to pilot study participants. Subsequently, the Round 1 Delphi will be conducted upon the completion of pilot study analysis with a different subgroup of malaria experts, different from the participants for the questionnaire validation. A paper reporting this protocol will be submitted for a conference presentation, and once the original results are available, the study will be submitted for publication in 2022.

2.9. Patient and Public Involvement

There is no patient or public involvement in this modified Delphi study.

3. Discussion

The modified Delphi study will produce a framework that will act as a theoretical lens for future studies related to malaria preventive behavior in communities exposed to P. knowlesi malaria infection. There are currently known various contributing factors in a community that could impact the malaria preventive behaviors such as the emotion, attitude, and environmental challenges [24].

The adaptive framework of the ideation model is evidence-based and has been used in family planning [39], HIV/AIDS intervention, and during Ebola outbreaks [38]. However, it is yet to be adapted in zoonotic malaria programs. There is a positive relationship between high levels of ideation and behavior change; thus, it is a valid model to integrate into our study. The model can identify individual community factors relating to preventive measures that should be addressed in a strategic communication program [24,38,39]. It can also be used to evaluate interventions and verify the effectiveness of a strategy. In this way, the model can help tighten the links between strategic planning and program design in zoonotic malaria control. The implementation can result in more effective and sustainable communication activities in the targeted community and ultimately improve health outcomes. A comprehensive framework for the prevention and control of vector-borne diseases such as malaria can provide guidance on planning and coordinate strategic, efficient, and multisectoral collaboration and the roles and responsibilities of different stakeholders. It can help governments understand how nongovernmental stakeholders and communities can be engaged in the process of disease control [42]. The study’s findings will be disseminated in peer-review publications, conference presentations, and propagated among policymakers.

4. Conclusions

This modified Delphi method protocol presents an overall picture of how the study will be conducted. The application described its detailed strategies to address the research questions on possible factors influencing malaria preventive behavior among communities exposed to P. knowlesi malaria infection—from the opinions of malaria experts. Ensuring the consensus among these experts can provide novel contributions for a more effective strategy to control the disease in providing guidance to future exploratory study in the communities.

Acknowledgments

We are grateful to Azmawati Mohammed Nawi and Halim Ismail from the Department of Community Health, Universiti Kebangsaan Malaysia, for sharing their wisdom with us in the initial development of this research project. We also thank. Douglas Storey and April Monroe from the John Hopkins Center for Communication Programs for providing expertise that immensely helped our research. Both have answered our numerous questions with unfailing patience. Their generosity has improved this study in innumerable ways. Any errors are our own and should not tarnish the reputation of these esteemed professionals.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/ijerph19074141/s1, Supplementary Questionnaire S1: The questionnaire draft, Supplementary Questionnaire S2: The pilot study questionnaire, Supplementary Text S1: Participants’ response and opinion on the draft.

Click here for additional data file. (375.9KB, zip)

Author Contributions

N.A.N. was the principal investigator responsible for the study conceptualization, design of the study protocol, and writing the original draft. M.R.H. was the coordinating investigator, project administration and assisted with the funding acquisition. N.A.N. and M.R.H. were responsible for the study conceptualization and methodology. M.S.J., R.H., K.A. and M.R.H. contributed equally to the content of the study protocol, with critical intellectual revisions and supervision of the study. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by Universiti Kebangsaan Malaysia grant number FF-2021-462.

Institutional Review Board Statement

The study was conducted according to the approval by Ministry of Health Medical Research Ethics Committee (NMRR ID-21-01980-JEH (IIR) and the National University of Malaysia (UKM) Centre for Research and Instrument Management (FF-2021-462). The rights, integrity, and confidentiality of the participants will be protected.

Informed Consent Statement

Electronic informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Data available on request from the corresponding author.

Conflicts of Interest

There is no conflict of interest.

Footnotes

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  • 1.Dalkey N. An experimental study of group opinion. Futures. 1969;1:408–426. doi: 10.1016/S0016-3287(69)80025-X. [DOI] [Google Scholar]
  • 2.Powell C. The Delphi technique: Myths and realities. J. Adv. Nurs. 2003;41:376–382. doi: 10.1046/j.1365-2648.2003.02537.x. [DOI] [PubMed] [Google Scholar]
  • 3.Habibi A., Sarafrazi A., Izadyar S. Delphi technique theoretical framework in qualitative research. Int. J. Eng. Sci. 2014;3:8–13. [Google Scholar]
  • 4.Hsu C.-C., Sandford B.A. The Delphi technique: Making sense of consensus. Pract. Assess. Res. Eval. 2007;12:1–8. doi: 10.7275/pdz9-th90. [DOI] [Google Scholar]
  • 5.Keeney S., Hasson F., McKenna H. Consulting the oracle: Ten lessons from using the Delphi technique in nursing research. J. Adv. Nurs. 2006;53:205–212. doi: 10.1111/j.1365-2648.2006.03716.x. [DOI] [PubMed] [Google Scholar]
  • 6.Okoli C., Pawlowski S.D. The Delphi method as a research tool: An example, design considerations and applications. Inf. Manag. 2004;42:15–29. doi: 10.1016/j.im.2003.11.002. [DOI] [Google Scholar]
  • 7.Rajhans V., Rege S., Memon U., Shinde A. Adopting a modified Delphi technique for revisiting the curriculum: A useful approach during the COVID-19 pandemic. Qual. Res. J. 2020;20:373–382. doi: 10.1108/QRJ-05-2020-0043. [DOI] [Google Scholar]
  • 8.Mahajan V., Linstone H.A., Turoff M. The Delphi Method: Techniques and Applications. J. Mark. Res. 2002;13:317. doi: 10.2307/3150755. [DOI] [Google Scholar]
  • 9.Humphrey-Murto S., Varpio L., Gonsalves C., Wood T.J. Using consensus group methods such as Delphi and Nominal Group in medical education research. Med. Teach. 2017;39:14–19. doi: 10.1080/0142159X.2017.1245856. [DOI] [PubMed] [Google Scholar]
  • 10.Hasson F., Keeney S. Enhancing rigour in the Delphi technique research. Technol. Forecast. Soc. Chang. 2011;78:1695–1704. doi: 10.1016/j.techfore.2011.04.005. [DOI] [Google Scholar]
  • 11.Boulkedid R., Abdoul H., Loustau M., Sibony O., Alberti C. Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review. PLoS ONE. 2011;6:e20476. doi: 10.1371/journal.pone.0020476. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Sekayi D., Kennedy A. Qualitative Delphi Method: A Four Round Process with a Worked Example. Qual. Rep. 2017;22:2755–2763. doi: 10.46743/2160-3715/2017.2974. [DOI] [Google Scholar]
  • 13.Slade S.C., Dionne C.E., Underwood M., Buchbinder R. Standardised method for reporting exercise programmes: Protocol for a modified Delphi study. BMJ Open. 2014;4:e006682. doi: 10.1136/bmjopen-2014-006682. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Vogel C., Zwolinsky S., Griffiths C., Hobbs M., Henderson E., Wilkins E. A Delphi study to build consensus on the definition and use of big data in obesity research. Int. J. Obes. 2019;43:2573–2586. doi: 10.1038/s41366-018-0313-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Fink-Hafner D., Dagen T., Doušak M., Novak M. Delphi method: Strengths and weaknesses. [(accessed on 30 July 2021)];Adv. Methodol. Stat. 2019 16:1–19. doi: 10.51936/fcfm6982. Available online: https://www.researchgate.net/publication/337570516. [DOI] [Google Scholar]
  • 16.Gallotta B., Garza-Reyes J.A., Anosike A. Using the Delphi method to verify a framework to implement sustainability ini-tiatives; Proceedings of the International Conference on Industrial Engineering and Operations Management; Bandung, Indonesia. 6–8 March 2018; pp. 231–4115. [Google Scholar]
  • 17.Niederberger M., Spranger J. Delphi Technique in Health Sciences: A Map. Front. Public Health. 2020;8:457. doi: 10.3389/fpubh.2020.00457. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Yam C.H.K., Wong E.L.Y., Cheung A.W.L., Chan F.W.K., Wong F.Y.Y., Yeoh E.-K. Framework and components for effective discharge planning system: A delphi methodology. BMC Health Serv. Res. 2012;12:396. doi: 10.1186/1472-6963-12-396. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Culley J.M. Use of a Computer-Mediated Delphi Process to Validate a Mass Casualty Conceptual Model. Comput. Inform. Nurs. 2011;29:272–279. doi: 10.1097/NCN.0b013e3181fc3e59. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.SPSS Statistics. IBM. [(accessed on 27 August 2021)]. Available online: https://www.ibm.com/products/spss-statistics.
  • 21.Al Juffali L.A., Knapp P., Al-Aqeel S., Watson M.C. Medication safety problems priorities in community pharmacy in Saudi Arabia: A multi-stakeholder Delphi study using the human factors framework. BMJ Open. 2019;9:e032419. doi: 10.1136/bmjopen-2019-032419. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Cole Z.D., Donohoe H.M., Stellefson M.L. Internet-Based Delphi Research: Case Based Discussion. Environ. Manag. 2013;51:511–523. doi: 10.1007/s00267-012-0005-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Lange T., Kopkow C., Lützner J., Günther K.-P., Gravius S., Scharf H.-P., Stöve J., Wagner R., Schmitt J. Comparison of different rating scales for the use in Delphi studies: Different scales lead to different consensus and show different test-retest reliability. BMC Med Res. Methodol. 2020;20:1–11. doi: 10.1186/s12874-020-0912-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Health Communication Capacity Collaborative . Ideation—An HC3 Research Primer: Ideational Model of Communication. IBM; Armonk, NY, USA: 2006. [(accessed on 27 August 2021)]. Published Online First. Available online: http://www.healthcommcapacity.org/wp-content/uploads/2015/02/Ideation.pdf. [Google Scholar]
  • 25.Kleinman A. Concepts and a model for the comparison of medical systems as cultural systems. Soc. Sci. Med. Part B Med Anthr. 1978;12:85–93. doi: 10.1016/0160-7987(78)90014-5. [DOI] [PubMed] [Google Scholar]
  • 26.Murdock G.P., Wilson S.F., Frederick V. World Distribution of Theories of Illness. Ethnology. 1978;17:449. doi: 10.2307/3773194. [DOI] [Google Scholar]
  • 27.Verhagen A.P., de Vet H.C., de Bie R.A., Kessels A.G., Boers M., Bouter L.M., Knipschild P.G. The Delphi List: A Criteria List for Quality Assessment of Randomized Clinical Trials for Conducting Systematic Reviews Developed by Delphi Consensus. J. Clin. Epidemiol. 1998;51:1235–1241. doi: 10.1016/S0895-4356(98)00131-0. [DOI] [PubMed] [Google Scholar]
  • 28.World Health Organization . World Malaria Report: 20 Years of Global Progress and Challenges. World Health Organization; Geneva, Switzerland: 2020. Licence: CC BY-NC-SA 3.0 IGO. [Google Scholar]
  • 29.Abeyasinghe R. Outcomes from the Evidence Review Group on Plasmodium Knowlesi. 2017. [(accessed on 30 August 2020)]. Available online: https://www.who.int/malaria/mpac/mpac-mar2017-plasmodium-knowlesi-presentation.pdf.
  • 30.World Health Organization (WHO), WHO Malaria Policy Advisory Group (MPAG) Meeting. 2021. [(accessed on 27 August 2021)]. Available online: https://www.who.int/groups/malaria-policy-advisory-group.
  • 31.Cooper D.J., Rajahram G.S., William T., Jelip J., Mohammad R., Benedict J., A Alaza D., Malacova E., Yeo T.W., Grigg M.J., et al. Plasmodium knowlesi Malaria in Sabah, Malaysia, 2015–2017: Ongoing Increase in Incidence Despite Near-elimination of the Human-only Plasmodium Species. Clin. Infect. Dis. 2020;70:361–367. doi: 10.1093/cid/ciz237. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Grigg M.J., Cox J., William T., Jelip J., Fornace K.M., Brock P.M., von Seidlein L., Barber B.E., Anstey N.M., Yeo T.W., et al. Individual-level factors associated with the risk of acquiring human Plasmodium knowlesi malaria in Malaysia: A case-control study. Lancet Planet. Health. 2017;1:e97–e104. doi: 10.1016/S2542-5196(17)30031-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Shearer F.M., Huang Z., Weiss D.J., Wiebe A., Gibson H., Battle K., Pigott D.M., Brady O.J., Putaporntip C., Jongwutiwes S., et al. Estimating Geographical Variation in the Risk of Zoonotic Plasmodium knowlesi Infection in Countries Eliminating Malaria. PLOS Negl. Trop. Dis. 2016;10:e0004915. doi: 10.1371/journal.pntd.0004915. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Eede P.V.D., Van H.N., Van Overmeir C., Vythilingam I., Duc T.N., Hung L.X., Manh H.N., Annette E., D’Alessandro U., Erhart A. Human Plasmodium knowlesi infections in young children in central Vietnam. Malar. J. 2009;8:249. doi: 10.1186/1475-2875-8-249. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Fornace K.M., Nuin N.A., Betson M., Grigg M.J., William T., Anstey N.M., Yeo T.W., Cox J., Ying L.T., Drakeley C.J. Asymptomatic and Submicroscopic Carriage ofPlasmodium knowlesiMalaria in Household and Community Members of Clinical Cases in Sabah, Malaysia. J. Infect. Dis. 2016;213:784–787. doi: 10.1093/infdis/jiv475. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Loy D.E., Plenderleith L.J., Sundararaman S.A., Liu W., Gruszczyk J., Chen Y.-J., Trimboli S., Learn G.H., MacLean O.A., Morgan A.L.K., et al. Evolutionary history of human Plasmodium vivax revealed by genome-wide analyses of related ape parasites. Proc. Natl. Acad. Sci. USA. 2018;115:E8450–E8459. doi: 10.1073/pnas.1810053115. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Barber B.E., Rajahram G.S., Grigg M.J., William T., Anstey N.M. World Malaria Report: Time to acknowledge Plasmodium knowlesi malaria. Malar. J. 2017;16:135. doi: 10.1186/s12936-017-1787-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Figueroa M.E. A Theory-Based Socioecological Model of Communication and Behavior for the Containment of the Ebola Epidemic in Liberia. J. Health Commun. 2017;22:5–9. doi: 10.1080/10810730.2016.1231725. [DOI] [PubMed] [Google Scholar]
  • 39.Krenn S., Cobb L., Babalola S., Odeku M., Kusemiju B. Using behavior change communication to lead a comprehensive family planning program: The Nigerian Urban Reproductive Health Initiative. Glob. Health Sci. Pract. 2014;2:427–443. doi: 10.9745/GHSP-D-14-00009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Haynes S.N., Richard D.C.S., Kubany E.S. Content validity in psychological assessment: A functional approach to concepts and methods. Psychol. Assess. 1995;7:238–247. doi: 10.1037/1040-3590.7.3.238. [DOI] [Google Scholar]
  • 41.Clibbens N., Walters S., Baird W. Delphi research: Issues raised by a pilot study. Nurse Res. 2012;19:37–44. doi: 10.7748/nr2012.01.19.2.37.c8907. [DOI] [PubMed] [Google Scholar]
  • 42.World Health Organization Multisectoral Approach to the Prevention and Control of Vector-Borne Diseases. 2020. [(accessed on 20 August 2021)]. Available online: https://apps.who.int/iris/handle/10665/331861.

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Click here for additional data file. (375.9KB, zip)

Data Availability Statement

Data available on request from the corresponding author.

Articles from International Journal of Environmental Research and Public Health are provided here courtesy of Multidisciplinary Digital Publishing Institute (MDPI)

RESOURCES