Developing an evaluation framework for public health environmental surveillance: Protocol for an international, multidisciplinary Delphi consensus study

Douglas G Manuel; Carol Bennett; Emma Brown; David L Buckeridge; Yoni Freedhoff; Sarah Funnell; Farah Ishtiaq; Matthew J Wade; David Moher; on behalf of the PHES-EF Executive Group

doi:10.1371/journal.pone.0310342

. 2025 May 27;20(5):e0310342. doi: 10.1371/journal.pone.0310342

Developing an evaluation framework for public health environmental surveillance: Protocol for an international, multidisciplinary Delphi consensus study

Douglas G Manuel ^1,^2,^3,^4,^*, Carol Bennett ^1,², Emma Brown ¹, David L Buckeridge ⁵, Yoni Freedhoff ², Sarah Funnell ⁶, Farah Ishtiaq ⁷, Matthew J Wade ^8,⁹, David Moher ^3,¹⁰; on behalf of the PHES-EF Executive Group^¶

Editor: Angela Mendes Freitas¹¹

PMCID: PMC12111604 PMID: 40424249

Abstract

Introduction

Public health environmental surveillance has evolved, especially during the coronavirus pandemic, with wastewater-based surveillance being a prominent example. As surveillance methods expand, it is important to have a robust evaluation of surveillance systems. This consensus study will develop an evaluation framework for public health environmental surveillance, informed by the expanding practice of wastewater-based surveillance during the pandemic.

Methods

The public health environmental surveillance evaluation framework will be developed in five steps. In Step 1, a multinational and multidisciplinary executive group will be formed to guide the framework development process. In Step 2, candidate items will be generated by conducting relevant scoping reviews and consultation with the study executive group. In Step 3, an international electronic Delphi will be conducted over two rounds to develop consensus on items for the framework. In Step 4, the executive group will reconvene to finalize the evaluation framework, discuss standout items, and determine the dissemination strategies. Lastly, Step 5 will focus on disseminating the evaluation framework to all parties involved with or affected by wastewater-based surveillance using traditional and public-oriented methods.

Discussion

The Delphi consensus study will provide multidisciplinary and multinational consensus for the evaluation framework, by providing a set of minimum criteria required for the evaluation of public health environmental surveillance systems. The evaluation framework is intended to support the sustainability of environmental surveillance and improve its implementation, reliability, credibility, and value.

Introduction

Public health relies on effective surveillance systems to support informed decision-making and timely interventions [1]. Wastewater-based surveillance, a form of environmental-based surveillance, has historically informed public health responses to threats like typhoid, polio, and illicit drug use [2–6]. More recently, the coronavirus pandemic (Covid-19) marked a turning point, expanding wastewater-based monitoring and surveillance to an unprecedented scale and underscoring its potential for wider public health applications [7]. Since then, wastewater-based surveillance, has supported public health practice for controlling monkeypox [8], influenza [9,10], antimicrobial resistance [11,12], and other pathogens and health risks beyond infectious diseases [13,14]. It is now used globally across diverse settings [15], from metropolitan to rural areas [16,17]. Cited advantages include its ability to track disease transmission independently of clinical testing availability or health-seeking behaviour, offering a non-invasive, resource-efficient surveillance method [18,19].

Current frameworks for evaluating surveillance performance require reassessment and updating. Established guidelines, such as the U.S. Centers for Disease Control and Prevention (CDC) 2001 framework, were developed before widespread environmental-based surveillance and do not adequately address unique challenges like environmental measurement variability or the need for broad, multidisciplinary collaboration [5,20,21]. Meanwhile, the pandemic prompted shifts in surveillance practices—such as increased public engagement and open-access data—and encouraged consensus-driven guidance through international, multidisciplinary collaboration. Yet no systematic review has examined how these developments influence surveillance evaluation, and there is no rigorous consensus on measuring the performance of wastewater-based surveillance and other forms of environmental-based surveillance.

This study aims to develop a comprehensive public health evaluation framework. Wastewater-based surveillance will be an example; however, it will apply to broader public health environmental surveillance systems, including air and surface monitoring. Surveillance targets considered include infectious agents and health risks beyond SARS-CoV-2.

Study objectives

This project will use a multidisciplinary and multinational consensus approach to develop an evaluation framework for public health environmental surveillance (Public Health Environmental Surveillance Evaluation Framework, PHES-EF). See https://PHES-EF.org. An evaluation framework provides structured guidance to facilitate a systematic approach to program evaluation [22–24]. See below for the study’s definition of an evaluation framework.

Specific objectives for the framework development process are as follows: (1) engage a multidisciplinary executive group of international experts to inform the development of the evaluation framework; (2) review the literature to identify current guidance on surveillance evaluation items; (3) prioritize items for an evaluation framework for public health environmental surveillance using an international electronic-Delphi (e-Delphi) process; (4) conduct a consensus meeting, which engages experts across disciplines, to create an evaluation framework for public health environmental surveillance based on feedback from the international e-Delphi process; and, (5) develop a comprehensive dissemination plan for the evaluation framework.

Definitions

We define key terms as follows:

Environmental-based surveillance (environmental surveillance): Surveillance of substances or organisms in air, water, soil, and living organisms to detect changes or threats that may affect human health, ecosystems, or both [25].
Evaluation framework: A structured system or schema to assess a program systematically [1,22,24]. An evaluation framework presents a clear list of items or elements that should appear in an evaluation, such as constructs or concepts. It provides an organisation or structure for the list of items, defines how the list and organisation were developed, and potentially includes details about measuring the evaluation items. Importantly, an evaluation framework facilitates informed decision-making by providing a comprehensive understanding of program effectiveness and areas for improvement.
Public health surveillance: The ongoing systematic collection, analysis, interpretation and dissemination of health data for the planning, implementation and evaluation of public health interventions and strategies to protect and improve the health of populations [26].
Wastewater: Water that has been in contact with people (e.g., for washing) or used for cleansing and sanitation (e.g., for flushing away faecal matter), and is discharged via sewers or other sanitation systems [6].
Wastewater-based surveillance (wastewater surveillance): A public health approach that examines specific substances or organisms in wastewater (sewage). Wastewater surveillance can be referred to as wastewater-based epidemiology; however, the latter can be specific to using wastewater to study and analyse the distribution, patterns, and determinants of health and disease conditions in defined populations [26].

Materials and methods

Study design

This protocol was developed a priori. The development of PHES-EF is guided by recommendations for Delphi techniques in the health sciences [27]. Fig 1 provides an overview of the study flow.

Step 1: Establish the executive group

A multinational and multidisciplinary executive group will oversee the development of the evaluation framework. The group will bring expertise from various fields, including public health, infectious disease, epidemiology, environmental and physical sciences, mathematical sciences, social sciences, communication, knowledge translation and exchange, knowledge users, and an engaged public.

The composition of the study executive group, which includes content expert and knowledge user members, will be identified by examining relevant author lists, internet-based searching, and the study working group’s professional networks. In contrast, the engaged public member of the executive group will be identified through an open call for expressions of interest. The S1 Table details the composition of the study executive group.

The executive group will be responsible for providing feedback on the framework development process (e.g., framework organization, completeness, additions, redundancy), supporting and advising working groups in their area of expertise, connecting the study to other related projects, recruiting e-Delphi panellists, participating in the consensus meeting, disseminating the evaluation framework, advising and/or participating in post-publication activities, and promoting the adoption of the evaluation framework among their professional networks. Co-chairs will also be responsible for decision-making when the executive group cannot reach a consensus, leading executive group activities, and the overall framework development process.

Step 2: Generate candidate evaluation items

We will systematically compile a list of e-Delphi candidate items through scoping reviews, content analysis and extraction, and executive group consultation.

Scoping reviews

Two scoping reviews will be performed to provide a comprehensive view of public health environmental surveillance system evaluations. The first review will identify frameworks (e.g., sources which guide the planning, management, or conduct of processes) and criteria (e.g., principles on which a judgment may be based) for evaluating public health or environmental surveillance systems. The second review will explore insights from the Covid-19 pandemic, particularly focusing on the principles of equity, trust, open science, community engagement, and One Health in the context of wastewater surveillance [28–30]. Detailed protocols for the scoping reviews are available on OSF (https://doi.org/10.17605/OSF.IO/GSKF6; https://doi.org/10.17605/OSF.IO/C9A3Q). The completed scoping reviews will be reported following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping reviews (PRISMA-ScR) [31]. See S1 and S2 Appendices for associated PRISMA-ScR checklists.

Content analysis and extraction

We will conduct a content analysis on items identified during the data extraction phase for both scoping reviews. The content analyses will identify and group criteria for evaluating public health and environmental surveillance systems. In the first step, one team member will group all the extracted items into initial domains and themes. In the second step, two researchers will independently compare the category system of the first researcher to agree on the final category system consensually. In the third step, all criteria will be reviewed again to merge or modify duplicate concepts and create a parsimonious framework for the initial list of e-Delphi candidate items.

Executive group consultation

The executive group will review the initial list of e-Delphi candidate items and discuss the initial structure and presentation of Round 1 items. Their review will focus on creating an evidence-informed public health environmental surveillance framework centred around SARS-CoV-2 wastewater surveillance, and incorporating lessons learned from the pandemic (e.g., equity, open science, community engagement, and One Health). Additional items will be added if the executive group determines that key evaluation areas have not been addressed.

Step 3: International e-Delphi survey

A Delphi technique will be used to develop an internationally accepted evaluation framework for public health environmental surveillance. The Delphi method is an iterative multi-round approach that uses a series of sequential surveys, interspersed by controlled feedback, to elicit consensus among a group of individuals while maintaining anonymity [32,33]. An e-Delphi method will be used to overcome geographic barriers and allow us to engage panellists internationally across various time zones. Delphi surveys relating to health sciences have typically been conducted for 2–3 rounds [27,34,35]. Therefore, a two-round e-Delphi survey and a subsequent consensus meeting will be conducted to balance feasibility with rigour. Each round will be active for approximately 4–6 weeks to allow panellists sufficient time to participate in the survey while balancing study timeline constraints [36].

Recruitment of panellists

We will recruit a multinational, multidisciplinary panel of wastewater surveillance experts, knowledge users, and engaged members of the public to complete the e-Delphi survey. Content experts are adult individuals with knowledge in their respective subdiscipline groups (see S2 Table). Panellists will be selected to capture the multiple perspectives of those that influence the design, implementation, evaluation, use, and reporting of wastewater surveillance activities, including the following subdisciplines groups: public health, infectious disease, epidemiology; environmental and physical sciences; mathematical sciences; social sciences; and communication, knowledge translation and exchange. Targeted recruitment will also be conducted to identify knowledge users and engaged members of the public. We will use purposive sampling to identify potentially eligible panellists by examining relevant author lists from the scoping reviews, professional networks, and internet-based searching, among other potential methods. Potential panellists may also be recruited using social media, a study website or calls for expressions of interest during relevant meetings, presentations, or correspondence. Snowball sampling may also be employed by executive group members and potential panellists to maximize recruitment.

Potentially eligible panellists will receive a recruitment email from the study working group, executive group members, or other potential panellists. A call for panellists to share the e-Delphi with other potentially eligible panellists may be included in recruitment materials and the e-Delphi survey. Inclusion criteria for e-Delphi panellists are outlined in Table 1.

Table 1. Eligibility criteria for e-Delphi survey panellists.

Discipline group	Discipline subgroup	Inclusion criteria
Content experts	Public health, infectious disease, epidemiology Environmental and physical sciences Mathematical sciences Social sciences Communication, knowledge translation and exchange	An adult^a who is proficient in English and has a graduate degree in one of the listed specializations^b, or ≥ 3 years of professional experience, or ≥ 2 peer-reviewed publications relating to wastewater surveillance.
Knowledge users		An adult^a who is proficient in English and who is a professional who does not have specialized training or qualifications in wastewater surveillance, but who uses surveillance data to inform policy and action in their workplace.
Engaged public		An adult^a who is proficient in English and has relevant lived experience.

Open in a new tab

Definitions: (1) Professional experience – Paid employment or professional practice in a listed specialization (current or former); (2) Graduate or professional degree – Master’s or Doctor of Philosophy (PhD) relating to a listed discipline subgroup.

a Adult: ≥ 18 years of age

b See S2 Table for specializations associated with each discipline.

We will conduct an all-rounds e-Delphi invitation approach, meaning panellists will be invited to every round regardless of whether they participated in the previous round [37]. Previous Delphi studies have commonly ranged from 20 to 30 panellists; however, there is no agreed upon standard [35,38]. Studies have also shown that the number of panellists and their diversity can impact the quality of outcomes [38]. Therefore, assuming only half of registered panellists will participate in at least one round, we will aim to recruit at least 50 panellists. Preferably, with at least 8–10 people per discipline subgroup. The study working group will monitor the distribution of registered panellists based on their demographic information and will try to distribute appropriately across discipline subgroups and other demographic markers.

Registration to complete the e-Delphi will open approximately four weeks before the first round, and e-Delphi round surveys will be active for roughly 4–6 weeks. The tentative start and end dates for recruitment are 08/09/2025 and 19/12/2025, respectively. Recruitment or retention efforts will occur throughout each e-Delphi round and the four weeks of targeted recruitment before Round 1. Email reminders to complete the survey will be sent twice per e-Delphi round. Panellists may also receive up to two reminders to complete the screening questionnaire during the targeted recruitment period. The time required to complete each e-Delphi round survey will be approximately one hour. The estimated time commitment will be provided to all potential panellists during recruitment and before starting the consensus process.

Panellist withdrawal or termination

Panellists will not be able to withdraw from the study due to the safeguards in place to maintain anonymity; however, they may choose to stop actively participating in the study at any time.

Panellist compensation

Content experts and knowledge users who complete all rounds of the e-Delphi process will receive a $100 CAD pre-paid Visa card in compensation for the time required to complete the e-Delphi process. Engaged members of the public who complete all rounds will be entered to win one of five $100 CAD pre-paid Visa cards. Remuneration has been shown to encourage participation and retention from diverse groups, thereby minimizing bias, and motivating thoughtful, high-quality responses, thereby improving data quality [39].

Procedure

We will conduct a two-round e-Delphi survey to generate consensus on evaluation criteria (see Fig 2). The full survey will be pre-tested and validated before administration. A pilot test of the survey will be conducted with up to five test panellists. Test panellists may include members of the executive group or other potential panellists matching our recruitment criteria. The pilot test will focus on assessing the clarity and relevance of background information provided for candidate items. The survey will be revised as necessary following feedback from pilot testing. Summaries of Round 1 will be compiled for the subsequent round.

We will use the Surveylet Delphi platform to administer the survey [40]. The platform facilitates interactive dialogue between panellists by allowing them to justify their responses and comment on the responses of others. Panellists can also save their progress and answer survey questions across multiple sessions. Custom survey pathways will be generated for each panellist discipline subgroup (i.e., panellists from different discipline subgroups will be shown a different collection of candidate items). Within each custom discipline subgroup survey pathway, panellists can skip items or self-declare that they are not qualified to assess certain candidate items. To ensure the independent evaluation of items, the order of candidate items may be randomized for each panellist.

Individuals will complete the screening questionnaire separate from the e-Delphi platform using LimeSurvey [41]. The study working group will review eligibility, and if eligible to participate in the study, individuals will receive an individualized link for anonymous participation in the e-Delphi survey. Implied consent will be obtained electronically for all panellists using the Surveylet Delphi platform before participation in the survey. Panellists who are eligible for compensation or wish to receive acknowledgement for their contributions will be able to do so by clicking a link at the end of each survey round. The link will bring them to a separate LimeSurvey form that will not be linked to their survey responses, allowing for e-Delphi responses to remain anonymous.

Defining consensus

Candidate items will be assessed on the following four rating properties: (1) relevance and practical utility; (2) scientific rigor, validity and reliability; (3) feasibility, adaptability and resource implications; and (4) equity, inclusiveness, and mitigation of bias.

A 7-point Likert scale (1 = highly irrelevant to 7 = highly relevant) will be used to rate each property for all candidate items. A summary score for each candidate item will be created by calculating the median of the four property ratings. Panellist summary scores will then be categorized as excluded item (irrelevant: 1–2), further discussion (equivocal: 3–5), or core item (relevant: 6–7). Consensus for each item is defined as ≥ 70% of the panellist votes falling within the same category (1–2, 3–5, or 6–7).

The approach to use a 7-point fully-labeled Likert scale – a higher number of categories compared to many studies – is informed by the consideration that panellists are professionals and engaged members of the public with good cognitive skills; therefore, more categories and labels will be more discriminating and reproducible [42–45]. The ≥ 70% consensus cut-off – a lower cut-off compared to many studies – is informed by the rapidly evolving nature of environmental surveillance for public health and the wide range of disciplines invited to the panel; therefore, a high level of agreement may not occur [42–45].

The rating properties were synthesized from previous performance evaluation frameworks from public health and health systems [1,24,46–51]. Equity is a property that was not previously identified but was added following an expert group discussion of potential properties and the study’s goals and objectives. See S3 Appendix for further details on the rating properties for e-Delphi candidate item scoring.

Stopping rules

When consensus is reached for a given item, no subsequent ranking rounds for that item will be performed. Consensus items that are considered equivocal, in addition to remaining non-consensual items, will be further deliberated during the consensus meeting.

Round 1

Panellists will be invited to rate their level of agreement with candidate items generated from scoping review results and consultation with the study executive group. Free-text boxes will be included for panellists to provide feedback or identify additional candidate items to be included in the next e-Delphi survey round.

Round 2

Regardless of whether they participated in the previous round, panellists will be invited to participate in Round 2 of the e-Delphi survey. All panellists will be invited to rate new items and re-rate previous items that did not reach consensus. When re-rating their level of agreement, panellists will be presented with their previous round scores alongside the aggregate group results. Anonymous feedback from Round 1 will also be compiled and presented during Round 2. Any newly suggested items during Round 2 will be deliberated during the consensus meeting.

Summary panellist characteristics

At the start of the e-Delphi survey, panellists will provide basic demographic details to ascertain their discipline subgroup, ensuring they receive the correct list of candidate items. To maintain anonymity, this demographic data will not contain identifiable information, and only aggregate responses of 5 or more panellists will be analyzed. We will report aggregate panel characteristics based on gender, discipline group or subgroup (depending on the number of panellists), level and type of experience, country income level, and World Bank work regions.

Quantitative analysis

We will report response rates, agreement levels, consensus, and stability across rounds using median, range, and IQR for each criterion item. The proportion of panellists who skipped items or selected “not qualified to respond” will be reported. Fisher’s exact test will be used to evaluate differences in item ratings by panellist characteristics: primary panellist disciplines (see S2 Table), country income level (high-income versus low-and-middle income), and World Bank world regions (see S3 Table). Fisher’s exact test was chosen for its suitability in analyzing small sample sizes and categorical data, as it does not rely on the assumption of large sample sizes required by other tests, such as the chi-squared test. This makes it a good choice for comparing item ratings across subgroups, particularly given the potential variability in panellist numbers across disciplines, income levels, and regions.

Qualitative analysis

Content analyses will be conducted on panellists’ feedback to extract common themes and insights. A summary will describe the evolution of each criterion across rounds, including reasons for item modification or elimination.

Grouping of subgroups

Subgroups with fewer than 5 panellists will be combined with similar groups based on panellist numbers and group likeness. Decisions on combined subgroups will be made prior to analyses of the items.

Exclusion of data

All panellists that participate in at least one round will have their responses included in analyses unless the study working group determines that a panellist’s response warrants being excluded from analyses for reasons that are newly developed or not previously recognized.

Step 4: Consensus meeting

In preparation for the consensus meeting, the study working group will categorize the remaining candidate items for consideration into a preliminary evaluation framework. This will include merging and/or modifying the remaining items based on gathered feedback from the e-Delphi rounds.

We will invite all members of the executive group to participate in the consensus meeting. An in-person or hybrid (in-person and online) consensus meeting may be held using the Surveylet Delphi survey platform after the results of the e-Delphi have been compiled and analyzed. If in a hybrid format, Microsoft Teams will be used for messaging and videoconferencing. Video recording and transcription services may also be used to aid notetaking. The primary objective of the meeting will be to achieve expert consensus on the final list of items for PHES-EF, through review and discussion of salient items. This process will be guided by the empirical evidence that was identified during the scoping reviews, and the opinions gathered during the e-Delphi process. The secondary objective will be to discuss publication and dissemination strategies for the final evaluation framework.

Steps to produce the final list of items are as follows: (i) present the results of the e-Delphi exercise (name, rationale, and score of each item); (ii) discuss the rationale and relevance for including the items in the framework; and (iii) vote on equivocal and non-consensual items. PHES-EF will be developed based on the final list of items that receive consensus during the executive group consensus meeting.

Step 5: Dissemination of evaluation framework

The dissemination of this evaluation framework will start with the publication of the PHES-EF protocol. The development of the evaluation framework will be reported in a statement document that will include the rationale and a brief description of the meeting, and the panellists involved. While the publication strategy will be finalized based on the consensus meeting discussion, the preliminary approach will be to post to OSF as a preprint, before publication in an open-access peer-reviewed journal. During this time, we will actively seek feedback from experts, knowledge users, and the public within our networks and via social media. PHES-EF will be made available using an open license (CC-BY-SA-4.0 license).

Post-publication, we will aim to keep the framework dynamic and up to date using a website, Discourse server, or other social media platforms to elicit ongoing feedback. Post-publication strategies may also include the creation of guides, toolkits, or ontologies to enhance the practical applicability of the evaluation framework and facilitate its adoption for policy and decision-making. Active dissemination approaches will include presenting at relevant scientific conferences, holding webinars, and conducting workshops. Additional dissemination practices will be determined during the consensus meeting.

Equity, diversity, and inclusion (EDI)

Incorporating EDI into scientific research enhances creativity and innovation, and improves the quality of results by embracing diverse perspectives and minimizing biases. It also helps promote fairness and equity by aiming to ensure that research outcomes serve various communities and global needs.

“Equity is defined as the removal of systemic barriers and biases enabling all individuals to have equal opportunity to access and benefit from the program” [52]. To achieve this, members of the research team are committed to developing a strong understanding of the systemic barriers faced by individuals from underrepresented groups. “Diversity is defined as differences in race, colour, place of origin, religion, immigrant and newcomer status, ethnic origin, ability, sex, sexual orientation, gender identity, gender expression and age” [52]. Panellists will be asked to provide characteristic information (e.g., country of residence, gender, primary discipline) to assess and help ensure (through more targeted recruitment) a large diversity in perspectives. “Inclusion is defined as the practice of ensuring that all individuals are valued and respected for their contributions and are equally supported” [52]. To help ensure that all research team members are integrated and supported, we will have a code of conduct statement in the executive group Terms of Reference.

Where feasible, and with sufficient statistical power, e-Delphi results will be further analyzed by panellist demographic information, such as by distinguishing respondents from high-, middle-, and low-income settings, as well as different world regions. This will allow for a nuanced understanding of how these contexts may have influenced the development of the framework, and which candidate items may be most valued in these areas.

All study working group members will complete EDI-related Canadian Institutes of Health Research (CIHR) training [53], and the First Nations Principles of OCAP^© (Ownership, Control, Access, and Possession) training for research [54]. The executive group will receive an overview of EDI and OCAP^© and will be encouraged and supported to complete such training. Furthermore, staff trained in EDI and OCAP^© principles will review the framework from an EDI perspective to ensure appropriate language.

Knowledge user and public involvement

The international e-Delphi survey will collect responses from content experts, knowledge users, and engaged members of the public to ensure that all parties involved with or affected by wastewater surveillance are represented in the consensus-building approach. Apart from what has already been described, more detailed recruitment or public engagement methods may be informed by the executive group's engaged member of the public. Additionally, to ensure accurate and transparent reporting of knowledge user and public involvement throughout the study, we will refer to the Guidance for Reporting Involvement of Patients and the Public (GRIPP2) checklist [55]. We will document the methods used to engage knowledge users and members of the public, report the impacts and outcomes of their engagement, and report on lessons learned from the experience.

Acknowledgments

E-Delphi participation will not satisfy authorship criteria; however, panellists will have the option to receive individual acknowledgement for their contributions to the framework, given they provide consent to be named. Individuals who participate in the consensus meeting will have the opportunity to be named as authors.

Research data management

The project will follow an open-science approach and make available the data, code, and materials for all project stages, including the search strategy, search findings, initial curation of evaluation measures, summary of executive group discussions, e-Delphi processes, and agreement procedures. The management of research data will follow a data management and sharing plan.

All personal data will be managed using LimeSurvey. LimeSurvey is a free, open-source, online survey tool. Personal data will only be collected from individuals during the screening questionnaire, and from panellists who wish to receive compensation and/or individual acknowledgement for their contributions to the evaluation framework. LimeSurvey data management practices are overseen by The Ottawa Hospital (TOH) Research Institute Methods Centre that protects privacy and confidentiality while assisting authorized users in collecting and managing survey data. It has been installed on a TOH server in the TOH Demilitarized Zone and all data collected is stored in a TOH database within the TOH firewall. Because LimeSurvey is installed outside of the TOH firewall in the TOH Demilitarized Zone, surveys can be responded to from anywhere using an internet connection.

Calibrum’s Surveylet Delphi platform will be used to manage e-Delphi responses. It may also be used for the consensus meeting. The e-Delphi will only collect anonymous data using a modified Delphi approach; meanwhile, the consensus meeting may collect identifiable data using a real-time Delphi approach. Surveylet complies with the EU-U.S. Privacy Shield Framework and Swiss-U.S. Privacy Shield Framework regarding the collection, use, and retention of personal information. The Surveylet platform also allows its users to be General Data Protection Regulation (GDPR) compliant. Calibrum’s servers are protected by high-end firewall systems and vulnerability scans are conducted regularly. Calibrum uses Transport Layer Security (TLS) encryption (also known as HTTPS) for all transmitted data. Surveys are also protected with passwords and HTTP referrer checking. All data are encrypted at rest and hosted by SSAE-16 SOC II certified third-party data centres. Data on deprecated hard drives are destroyed by U.S. DOD methods and delivered to a third-party data destruction service.

Study record retention

Any data exported from either survey company will be stored securely in a password protected location within TOH’s SharePoint. All study records will be retained securely for 10 years following study completion, then destroyed in accordance with Ottawa Hospital Research Institute (OHRI) requirements.

Ethical considerations

This study has received ethical approval (20230428-01H) from the Ottawa Health Science Network Research Ethics Board (OHSN-REB). Implied consent will be obtained from all panellists prior to participating in the e-Delphi survey. Information about the survey and consent practices will be provided on the first page of the survey. Panellists will be informed that by providing their demographic information and proceeding to the next page, they are providing their implied consent to participate in the study.

Only anonymous panellist responses will be used for analyses. All panellist data and responses will be password protected and only REB-approved study working group members from the Ottawa Hospital will have access to the data.

Discussion

This paper outlines the protocol for a two-round e-Delphi study that will be used to develop a multinational, multidisciplinary evaluation framework for public health environmental surveillance. Developing an evaluation framework will contribute to a better understanding of the ideal scope and use cases for wastewater surveillance and improve the reliability and credibility of surveillance data for public health officials. An evaluation framework will also help promote the sustainability of wastewater surveillance systems by providing an effective performance measurement tool.

We strove to design the PHES-EF protocol using a rigorous consensus approach. The development of the PHES-EF study protocol was informed by ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi [38]. Surveillance evaluation frameworks are uncommonly developed using a consensus approach. PHES-EF will be developed for environmental surveillance and informed by wastewater surveillance during the pandemic. However, associated scoping reviews and prior surveillance evaluation frameworks are related largely to any public health surveillance system [56]. Therefore, PHES-EF may be applicable for evaluating public health surveillance beyond environmental systems.

Limitations

Although the Delphi technique uses a systematic process to generate consensus among a panel of experts, the method is not without its limitations. One of the primary concerns when using this technique is the ability to implement a truly representative expert panel. Despite our plan to use various recruitment methods to enroll panellists from all discipline subgroups with varying demographic characteristics, consensus results may ultimately be influenced by the final panel composition [57].

Another potential limitation in using a modified Delphi approach is the potential influence of controlled feedback on the convergence of panellist opinions. Although providing controlled feedback between survey rounds aids in generating consensus, this approach may bias consensus results by influencing individual panellists’ opinions [58].

Furthermore, due to feasibility and resource limitations, only sources that are available in English will be included in associated scoping reviews. Additionally, only individuals who are proficient in English will be eligible to participate in the e-Delphi survey and consensus meeting. Therefore, the findings of this study may not reflect relevant knowledge that is not available in English.

Supporting information

S1 Table. Executive group composition.

(PDF)

pone.0310342.s001.pdf^{(91.7KB, pdf)}

S2 Table. Executive group and e-Delphi panellist disciplines.

The list of specializations or related concepts is not extensive.

(PDF)

pone.0310342.s002.pdf^{(86.1KB, pdf)}

S3 Table. World region classifications.

Note: The regional classifications are used by the World Bank for organizing economic data, development projects, and policy analysis. These classifications are not strictly geographical but also consider economic and developmental similarities and relationships.

(PDF)

pone.0310342.s003.pdf^{(82.1KB, pdf)}

S1 Appendix. PRISMA-ScR checklist for scoping review #1.

Current Evaluation Practices for Public Health and Environmental Surveillance Systems: A Scoping Review Protocol.

(PDF)

pone.0310342.s004.pdf^{(104.8KB, pdf)}

S2 Appendix. PRISMA-ScR checklist for scoping review #2.

Lessons Learned from Wastewater-Based Surveillance During the COVID-19 Pandemic: A Scoping Review Protocol.

(PDF)

pone.0310342.s005.pdf^{(104.9KB, pdf)}

S3 Appendix. Rating properties for e-Delphi candidate item scoring.

(PDF)

pone.0310342.s006.pdf^{(194.8KB, pdf)}

S4 Appendix. Draft e-Delphi panellist screening questionnaire.

(PDF)

pone.0310342.s007.pdf^{(184.6KB, pdf)}

Acknowledgments

We thank Mary Jessome (PhD candidate, University of British Columbia) for her contributions to the study as a past executive group member with expertise in social sciences. We would also like to thank Peter Farrell (Research Librarian, University of Ottawa) for developing the associated scoping review search strategies for this study.

PHES-EF Executive Group (ordered alphabetically according to surname except for the first two authors):

Douglas G. Manuel* (0000-0003-0912-0845), David L. Buckeridge* (0000-0003-1817-5047), Yoni Freedhoff (0000-0001-9391-3455), Sarah Funnell, Bernd M. Gawlik (0000-0002-4406-5362), Farah Ishtiaq (0000-0002-6762-7014), Amy E. Kirby, Kerrigan McCarthy (0000-0001-8958-9795), David Moher (0000-0003-2434-4206), Beate Sander (0000-0003-2128-9133), Jeremy Veillard, Matthew J. Wade (0000-0001-9824-7121)

* Co-chairs of PHES-EF Executive Group

Data Availability

No datasets were generated or analysed during the current study. All relevant data from this study will be made available upon study completion.

Funding Statement

This project is funded by the Canadian Institutes of Health Research-funded network, CoVaRR-Net (Coronavirus Variants Rapid Response Network. FRN: 175622), and Health Canada (through the Safe Restart Agreement Contribution Program. Arrangement #: 2223-HQ-000098). The Canadian Institutes of Health Research and Health Canada have not been involved in the design or conduct of the study and the views expressed herein do not necessarily represent the views of either funding organization.

References

1.Woodward G, Manuel DG, Goel V. Developing a balanced scorecard for public health. Toronto: Institute for Clinical Evaluative Sciences; 2004. [Google Scholar]
2.McClary-Gutierrez JS, Mattioli MC, Marcenac P, Silverman AI, Boehm AB, Bibby K, et al. SARS-CoV-2 Wastewater Surveillance for Public Health Action. Emerg Infect Dis. 2021;27(9):1–8. doi: 10.3201/eid2709.210753 [DOI] [PMC free article] [PubMed] [Google Scholar]
3.Diamond MB, Keshaviah A, Bento AI, Conroy-Ben O, Driver EM, Ensor KB, et al. Wastewater surveillance of pathogens can inform public health responses. Nat Med. 2022;28(10):1992–5. doi: 10.1038/s41591-022-01940-x [DOI] [PubMed] [Google Scholar]
4.Zuccato E, Chiabrando C, Castiglioni S, Bagnati R, Fanelli R. Estimating community drug abuse by wastewater analysis. Environ Health Perspect. 2008;116(8):1027–32. doi: 10.1289/ehp.11022 [DOI] [PMC free article] [PubMed] [Google Scholar]
5.Hrudey SE, Bischel HN, Charrois J, Chik AHS, Conant B, Delatolla R, et al. Wastewater Surveillance for SARS-CoV-2 RNA in Canada. FACETS. 2022;7:1493–597. doi: 10.1139/facets-2022-0148 [DOI] [Google Scholar]
6.World Health Organization. Environmental surveillance for SARS-COV-2 to complement public health surveillance. World Health Organization; 2022. Apr. [Google Scholar]
7.Manuel DG, Delatolla R, Fisman DN, Fuzzen M, Graber T, Katz GM, et al. The Role of Wastewater Testing for SARS-CoV-2 Surveillance. Ontario COVID-19 Science Advisory Table; 2021. Aug. doi: 10.47326/ocsat.2021.02.40.1.0 [DOI] [Google Scholar]
8.Tiwari A, Adhikari S, Kaya D, Islam MA, Malla B, Sherchan SP, et al. Monkeypox outbreak: Wastewater and environmental surveillance perspective. Sci Total Environ. 2023;856(Pt 2):159166. doi: 10.1016/j.scitotenv.2022.159166 [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Wolfe MK, Duong D, Bakker KM, Ammerman M, Mortenson L, Hughes B, et al. Wastewater-Based Detection of Two Influenza Outbreaks. Environ Sci Technol Lett. 2022;9(8):687–92. doi: 10.1021/acs.estlett.2c00350 [DOI] [Google Scholar]
10.Sims N, Kasprzyk-Hordern B. Future perspectives of wastewater-based epidemiology: Monitoring infectious disease spread and resistance to the community level. Environ Int. 2020;139:105689. doi: 10.1016/j.envint.2020.105689 [DOI] [PMC free article] [PubMed] [Google Scholar]
11.Huijbers PMC, Flach C-F, Larsson DGJ. A conceptual framework for the environmental surveillance of antibiotics and antibiotic resistance. Environ Int. 2019;130:104880. doi: 10.1016/j.envint.2019.05.074 [DOI] [PubMed] [Google Scholar]
12.Tiwari A, Kurittu P, Al-Mustapha AI, Heljanko V, Johansson V, Thakali O, et al. Wastewater surveillance of antibiotic-resistant bacterial pathogens: A systematic review. Front Microbiol. 2022;13:977106. doi: 10.3389/fmicb.2022.977106 [DOI] [PMC free article] [PubMed] [Google Scholar]
13.Yang D, Zheng Q, Thai PK, Ahmed F, O’Brien JW, Mueller JF, et al. A nationwide wastewater-based assessment of metformin consumption across Australia. Environ Int. 2022;165:107282. doi: 10.1016/j.envint.2022.107282 [DOI] [PubMed] [Google Scholar]
14.Escolà Casas M, Schröter NS, Zammit I, Castaño-Trias M, Rodriguez-Mozaz S, Gago-Ferrero P, et al. Showcasing the potential of wastewater-based epidemiology to track pharmaceuticals consumption in cities: Comparison against prescription data collected at fine spatial resolution. Environ Int. 2021;150:106404. doi: 10.1016/j.envint.2021.106404 [DOI] [PubMed] [Google Scholar]
15.Woytus A. What’s in the wastewater? In: St. Louis [Internet]. 22 Sep 2021. [cited 1 Jun 2023]. Available from: https://www.stlmag.com/api/content/3f60d498-1b26-11ec-a8ea-1244d5f7c7c6/ [Google Scholar]
16.D’Aoust PM, Towhid ST, Mercier É, Hegazy N, Tian X, Bhatnagar K, et al. COVID-19 wastewater surveillance in rural communities: Comparison of lagoon and pumping station samples. Sci Total Environ. 2021;801:149618. doi: 10.1016/j.scitotenv.2021.149618 [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Shah S, Gwee SXW, Ng JQX, Lau N, Koh J, Pang J. Wastewater surveillance to infer COVID-19 transmission: A systematic review. Sci Total Environ. 2022;804:150060. doi: 10.1016/j.scitotenv.2021.150060 [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Liu P, Ibaraki M, VanTassell J, Geith K, Cavallo M, Kann R, et al. A sensitive, simple, and low-cost method for COVID-19 wastewater surveillance at an institutional level. Sci Total Environ. 2022;807(Pt 3):151047. doi: 10.1016/j.scitotenv.2021.151047 [DOI] [PMC free article] [PubMed] [Google Scholar]
19.Larsen DA, Wigginton KR. Tracking COVID-19 with wastewater. Nat Biotechnol. 2020;38(10):1151–3. doi: 10.1038/s41587-020-0690-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
20.Gretchen Vogel. Signals from the sewer—Measuring virus levels in wastewater can help track the pandemic. But how useful is that? Science. 9 Mar. 2022375. Available from: https://www.science.org/content/article/pandemic-signals-sewer-what-virus-levels-wastewater-tell-us [Google Scholar]
21.Manuel D, Amadei CA, Campbell JR, Brault J, Zierler A, Veillard J. Strengthening Public Health Surveillance Through Wastewater Testing: An Essential Investment for the COVID-19 Pandemic and Future Health Threats. World Bank Group. The World Bank; 2022. p. 1–67. [Google Scholar]
22.Fynn JF, Hardeman W, Milton K, Jones AP. A scoping review of evaluation frameworks and their applicability to real-world physical activity and dietary change programme evaluation. BMC Public Health. 2020;20(1):1000. doi: 10.1186/s12889-020-09062-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
23.Nilsen P. Making sense of implementation theories, models, and frameworks. Implement Sci. 2020;30:53–79. [DOI] [PMC free article] [PubMed] [Google Scholar]
24.Calba C, Goutard FL, Hoinville L, Hendrikx P, Lindberg A, Saegerman C, et al. Surveillance systems evaluation: a systematic review of the existing approaches. BMC Public Health. 2015;15:448. doi: 10.1186/s12889-015-1791-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Andrews JR, Yu AT, Saha S, Shakya J, Aiemjoy K, Horng L, et al. Environmental Surveillance as a Tool for Identifying High-risk Settings for Typhoid Transmission. Clin Infect Dis. 2020;71(Suppl 2):S71–8. doi: 10.1093/cid/ciaa513 [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Porta M. A Dictionary of Epidemiology. A Dictionary of Epidemiology. Oxford University Press; 2016. Available from: https://www.oxfordreference.com/display/10.1093/acref/9780199976720.001.0001/acref-9780199976720 [Google Scholar]
27.Spranger J, Homberg A, Sonnberger M, Niederberger M. Reporting guidelines for delphi techniques in health sciences: a methodological review. Z Evid Fortbild Qual Gesundh Wesen. 2023;172:1–11. [DOI] [PubMed] [Google Scholar]
28.Editorial. How epidemiology has shaped the COVID pandemic. Nature. 2021;589(7843):491–2. doi: 10.1038/d41586-021-00183-z [DOI] [PubMed] [Google Scholar]
29.Devine D, Gaskell J, Jennings W, Stoker G. Trust and the Coronavirus Pandemic: What are the Consequences of and for Trust? An Early Review of the Literature. Polit Stud Rev. 2021;19(2):274–85. doi: 10.1177/1478929920948684 [DOI] [PMC free article] [PubMed] [Google Scholar]
30.World Health Organization. Defining collaborative surveillance: a core concept for strengthening the global architecture for health emergency preparedness, response, and resilience (HEPR). Geneva: World Health Organization; 2023. Available from: https://iris.who.int/handle/10665/367927 [Google Scholar]
31.Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169(7):467–73. doi: 10.7326/M18-0850 [DOI] [PubMed] [Google Scholar]
32.Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: How to decide its appropriateness. World J Methodol. 2021;11(4):116–29. doi: 10.5662/wjm.v11.i4.116 [DOI] [PMC free article] [PubMed] [Google Scholar]
33.Shang Z. Use of Delphi in health sciences research: A narrative review. Medicine (Baltimore). 2023;102(7):e32829. doi: 10.1097/MD.0000000000032829 [DOI] [PMC free article] [PubMed] [Google Scholar]
34.Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017;31(8):684–706. doi: 10.1177/0269216317690685 [DOI] [PubMed] [Google Scholar]
35.Niederberger M, Spranger J. Delphi Technique in Health Sciences: A Map. Front Public Health. 2020;8:457. doi: 10.3389/fpubh.2020.00457 [DOI] [PMC free article] [PubMed] [Google Scholar]
36.Schmalz U, Spinler S, Ringbeck J. Lessons Learned from a Two-Round Delphi-based Scenario Study. MethodsX. 2020;8:101179. doi: 10.1016/j.mex.2020.101179 [DOI] [PMC free article] [PubMed] [Google Scholar]
37.Boel A, Navarro-Compán V, Landewé R, van der Heijde D. Two different invitation approaches for consecutive rounds of a Delphi survey led to comparable final outcome. J Clin Epidemiol. 2021;129:31–9. doi: 10.1016/j.jclinepi.2020.09.034 [DOI] [PubMed] [Google Scholar]
38.Gattrell WT, Logullo P, van Zuuren EJ, Price A, Hughes EL, Blazey P, et al. ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi. PLoS Med. 2024;21(1):e1004326. doi: 10.1371/journal.pmed.1004326 [DOI] [PMC free article] [PubMed] [Google Scholar]
39.Permuth-Wey J, Borenstein AR. Financial remuneration for clinical and behavioral research participation: ethical and practical considerations. Ann Epidemiol. 2009;19(4):280–5. doi: 10.1016/j.annepidem.2009.01.004 [DOI] [PubMed] [Google Scholar]
40.Surveylet. United States: Calibrum. Available from: https://calibrum.com [Google Scholar]
41.LimeSurvey: An open source survey tool. Hamburg, Germany: LimeSurvey GmbH; Available from: https://www.limesurvey.org [Google Scholar]
42.Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401–9. doi: 10.1016/j.jclinepi.2013.12.002 [DOI] [PubMed] [Google Scholar]
43.Belton I, MacDonald A, Wright G, Hamlin I. Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process. Technological Forecasting and Social Change. 2019;147:72–82. doi: 10.1016/j.techfore.2019.07.002 [DOI] [Google Scholar]
44.Weijters B, Cabooter E, Schillewaert N. The effect of rating scale format on response styles: The number of response categories and response category labels. International Journal of Research in Marketing. 2010;27(3):236–47. doi: 10.1016/j.ijresmar.2010.02.004 [DOI] [Google Scholar]
45.Lange T, Kopkow C, Lützner J, Günther K-P, Gravius S, Scharf H-P, et al. Comparison of different rating scales for the use in Delphi studies: different scales lead to different consensus and show different test-retest reliability. BMC Med Res Methodol. 2020;20(1):28. doi: 10.1186/s12874-020-0912-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
46.Locke EA, Latham GP. A theory of goal setting & task performance. Prentice-Hall, Inc. 1990. [Google Scholar]
47.Veillard J, Champagne F, Klazinga N, Kazandjian V, Arah OA, Guisset A-L. A performance assessment framework for hospitals: the WHO regional office for Europe PATH project. Int J Qual Health Care. 2005;17(6):487–96. doi: 10.1093/intqhc/mzi072 [DOI] [PubMed] [Google Scholar]
48.Kaplan RS, Norton DP. Balanced scorecard. Springer; 2007. [Google Scholar]
49.Mitchell RJ, Williamson AM, O’Connor R. The development of an evaluation framework for injury surveillance systems. BMC Public Health. 2009;9:260. doi: 10.1186/1471-2458-9-260 [DOI] [PMC free article] [PubMed] [Google Scholar]
50.Nothacker M, Stokes T, Shaw B, Lindsay P, Sipilä R, Follmann M, et al. Reporting standards for guideline-based performance measures. Implement Sci. 2016;11:6. doi: 10.1186/s13012-015-0369-z [DOI] [PMC free article] [PubMed] [Google Scholar]
51.Groseclose SL, German RR, Nsubuga P. Evaluating Public Health Surveillance. Principles & Practice of Public Health Surveillance. Oxford University Press. 2010. p. 166–97. doi: 10.1093/acprof:oso/9780195372922.003.0008 [DOI] [Google Scholar]
52.Government of Canada. Best Practices in Equity, Diversity and Inclusion in Research. In: New Frontiers in Research Fund [Internet]. 22 Jun 2021. Available from: https://www.sshrc-crsh.gc.ca/funding-financement/nfrf-fnfr/edi-eng.aspx [Google Scholar]
53.Government of Canada. Introduction to GBA Plus. 31 Mar 2021. [cited 1 Jun 2023]. Available from: https://women-gender-equality.canada.ca/gbaplus-course-cours-acsplus/eng/mod00/mod00_01_01.html [Google Scholar]
54.The First Nations Information Governance Centre. The First Nations Principles of OCAP®. In: The First Nations Information Governance Centre [Internet]. 1 Jun 2023. [cited 1 Jun 2023]. Available from: https://fnigc.ca/ocap-training/ [Google Scholar]
55.Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ. 2017;358:j3453. doi: 10.1136/bmj.j3453 [DOI] [PMC free article] [PubMed] [Google Scholar]
56.Yang L, Weston C, Cude C, Kincl L. Evaluating Oregon’s occupational public health surveillance system based on the CDC updated guidelines. Am J Ind Med. 2020;63(8):713–25. doi: 10.1002/ajim.23139 [DOI] [PMC free article] [PubMed] [Google Scholar]
57.Winkler J, Moser R. Biases in future-oriented Delphi studies: A cognitive perspective. Technological Forecasting and Social Change. 2016;105:63–76. doi: 10.1016/j.techfore.2016.01.021 [DOI] [Google Scholar]
58.Goodman CM. The Delphi technique: a critique. J Adv Nurs. 1987;12(6):729–34. doi: 10.1111/j.1365-2648.1987.tb01376.x [DOI] [PubMed] [Google Scholar]

PLoS One. 2025 May 27;20(5):e0310342. doi: 10.1371/journal.pone.0310342.r001

Author response to Decision Letter 0

9 Sep 2024

PLoS One. doi: 10.1371/journal.pone.0310342.r002

Decision Letter 0

Angela Mendes Freitas

30 Dec 2024

PONE-D-24-33538Developing an evaluation framework for public health environmental surveillance: Protocol for an international, multidisciplinary e-Delphi studyPLOS ONE

Dear Dr. Manuel,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

The reviewers have reviewed the work you have done and recognize the contribution of the research to the field. However, they have also provided constructive comments and suggestions to improve the clarity and overall quality of the manuscript, ensuring it meets the standards for publication. Please pay particular attention to the methodology section, ensuring detailed and clear information is provided regarding the selection of panelists, the statistical methods employed, and the criteria used for reaching consensus.

We kindly request that you address these comments in a revised version of your manuscript.

==============================

Please submit your revised manuscript by Feb 13 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols .

We look forward to receiving your revised manuscript.

Kind regards,

Angela Mendes Freitas

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. When completing the data availability statement of the submission form, you indicated that you will make your data available on acceptance. We strongly recommend all authors decide on a data sharing plan before acceptance, as the process can be lengthy and hold up publication timelines. Please note that, though access restrictions are acceptable now, your entire data will need to be made freely accessible if your manuscript is accepted for publication. This policy applies to all data except where public deposition would breach compliance with the protocol approved by your research ethics board. If you are unable to adhere to our open data policy, please kindly revise your statement to explain your reasoning and we will seek the editor's input on an exemption. Please be assured that, once you have provided your new statement, the assessment of your exemption will not hold up the peer review process.

3. One of the noted authors is a group or consortium [PHES-EF Executive Group]. In addition to naming the author group, please list the individual authors and affiliations within this group in the acknowledgments section of your manuscript. Please also indicate clearly a lead author for this group along with a contact email address.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Does the manuscript provide a valid rationale for the proposed study, with clearly identified and justified research questions?

The research question outlined is expected to address a valid academic problem or topic and contribute to the base of knowledge in the field.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Is the protocol technically sound and planned in a manner that will lead to a meaningful outcome and allow testing the stated hypotheses?

The manuscript should describe the methods in sufficient detail to prevent undisclosed flexibility in the experimental procedure or analysis pipeline, including sufficient outcome-neutral conditions (e.g. necessary controls, absence of floor or ceiling effects) to test the proposed hypotheses and a statistical power analysis where applicable. As there may be aspects of the methodology and analysis which can only be refined once the work is undertaken, authors should outline potential assumptions and explicitly describe what aspects of the proposed analyses, if any, are exploratory.

Reviewer #1: Partly

Reviewer #2: Yes

**********

3. Is the methodology feasible and described in sufficient detail to allow the work to be replicable?

Descriptions of methods and materials in the protocol should be reported in sufficient detail for another researcher to reproduce all experiments and analyses. The protocol should describe the appropriate controls, sample size calculations, and replication needed to ensure that the data are robust and reproducible.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors described where all data underlying the findings will be made available when the study is complete?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception, at the time of publication. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above and, if applicable, provide comments about issues authors must address before this protocol can be accepted for publication. You may also include additional comments for the author, including concerns about research or publication ethics.

You may also provide optional suggestions and comments to authors that they might find helpful in planning their study.

(Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Overall, this study protocol is well-prepared and addresses an important issue in public health surveillance. The authors’ structured approach, use of the Delphi method, and comprehensive dissemination plan indicate a strong foundation for a valuable contribution to public health. Nevertheless, to improve the study, the following items can be reviewed and revised:

• The background and rationale could be strengthened by adding a section specifically addressing the limitations of current evaluation methods and how this framework seeks to address those gaps.

• The article could include references to recent advancements or alternative applications of wastewater-based surveillance (e.g., monitoring antimicrobial resistance, drug residues).

• Consider a pilot round to identify potential issues with the survey questions or consensus-building platform (Surveylet Delphi).

• Additional details on post-publication use and integration of the framework in policy-making and public health systems would enhance practical applicability.

• How this framework might adapt to different socio-economic and geographic contexts should be discussed.

Almost, it could be say that if any other researcher were to conduct this study, they would likely use the same approach. As a result, I believe there is a lack of specific innovation or creativity in this protocol. It is recommended to consider modifications to the research methodology or interventions to enhance the protocol's appeal and impact, thereby creating more pronounced distinctions from similar studies.

Reviewer #2: The methods and protocol sections are comprehensive and well-structured. This reflects a good approach to the Delphi process. I think there is an opportunity to provide additional detail to better support clarity and detail reproducibility. For instance, specifying the scoring scale used by panelists and clearly defining the criteria for achieving consensus would strengthen the methodology. The protocol emphasizes EDI, but the selection process for panelists could be better explained to outline how diversity in expertise and demographics will be ensured, and importantly how it might shape/improve the outcomes. While the quantitative analysis is detailed, further explanation of how median, range, and IQR will be used to define consensus and stability is needed. Also, elaborating on why Fisher’s exact test was chosen and its threshold for significance would provide a stronger rationale for the approach. I like to see stats choices well defined.

The ethical considerations part is strong, though additional information on how informed consent will be managed during hybrid meetings will strengthen the methodology. Are there any fallback plans for managing (potential) panelist dropouts (it is a common challenge for a Delphi), particularly in subgroups? This should be noted.

Grammar and Style

The grammar and style of the paper are clear but could benefit from a careful edit and re-read. Several sentences, particularly in the "Study Record Retention" and "Research Data Management" sections, are lengthy and could be restructured into shorter, more readable forms. Redundancy in describing data encryption and ethical compliance could be streamlined to avoid repetition. And, yes I know this is stylistic, but use of active voice would improve the paper -- especially for professional audiences. I also suggest adding really basic introductory sentences before diving into complex details in sections (like EDI) to improve the overall flow and applicability of content.

A Few Suggestions

To strengthen the draft, maybe provide templates or examples for the demographic questionnaire and controlled feedback forms. These would enhance transparency. Plans for dissemination are good, particularly the integration of a website and social media platforms. Perhaps including strategies for keeping the framework dynamic and updated post-publication would be helpful? Are there strategies for participant fatigue? Such as incentives or even streamlined surveys. Recognizing the potential influence of controlled feedback on panelist opinions and detailing mitigation strategies to minimize bias could improve outcomes.

Overall, the protocol has a good methodology and commitment to inclusivity, but it would benefit from more precise descriptions of statistical methods, panelist selection processes, and importantly an outline of contingency plans. Refining language for consistency, reducing redundancy, and breaking down complex sentences will enhance the clarity and accessibility. It is an applied work, so writing for such audiences is important.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy .

Reviewer #1: Yes: Hojatolah Gharaee

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step.

PLoS One. 2025 May 27;20(5):e0310342. doi: 10.1371/journal.pone.0310342.r003

Author response to Decision Letter 1

13 Feb 2025

Review Comments to the Author

1. The background and rationale could be strengthened by adding a section specifically addressing the limitations of current evaluation methods and how this framework seeks to address those gaps.

Thank you for providing these insights. We agree with you, and we have introduced a paragraph to address this suggestion.

2. The article could include references to recent advancements or alternative applications of wastewater-based surveillance (e.g., monitoring antimicrobial resistance, drug residues).

We agree with your assessment and have highlighted the role WBS for antimicrobial resistance, with additional statements indicating WBS broad uses.

Consider a pilot round to identify potential issues with the survey questions or consensus-building platform (Surveylet Delphi).

We have incorporated your suggestion in the Procedure section. We will pilot test the survey for up to five panellists, composed of executive group members and other potential panellists matching our recruitment criteria. The pilot test will focus on assessing the clarity and relevance of background information provided for candidate items. The survey will be revised as necessary following feedback from pilot testing.

3. Additional details on post-publication use and integration of the framework in policy-making and public health systems would enhance practical applicability.

We have revised the “Step 5: Dissemination of evaluation framework” section to include further details about our post-publication knowledge translation strategies.

4. How this framework might adapt to different socio-economic and geographic contexts should be discussed.

Thank you for highlighting the importance of an evaluation framework’s ability to adapt to different socio-economic and geographic contexts. Our Executive Group emphasized the importance of this perspective, and we feel that this is a strength in our methodology, however we now realize that it was under-developed in our manuscript. We have added a description of how we will report e-Delphi findings relating to equity in the “Equity, Diversity, and Inclusion (EDI)” section of the manuscript. We have also included the rating properties for candidate items in the “Defining consensus” section of the manuscript.

Candidate items will be evaluated across four rating properties: (1) Relevance and practical utility; (2) scientific rigor, validity and reliability; (3) feasibility, adaptability and resource implications; and (4) equity, inclusiveness, and mitigation of bias. These rating properties were chosen to ensure that the framework addresses the needs of varied settings. From this, we will assess the level of consensus on these candidate items, specifically highlighting candidate items rated as strong or weak in terms of their equity implications. Of note, after reviewing the literature, our study is the first study that we are aware of that uses equity as a specific rating property.

Where feasible, and with sufficient statistical power, we will further analyze the data by subgroup, distinguishing respondents from high-, middle-, and low-income settings, as well as different world regions. This will allow for a nuanced understanding of how these contexts influence the development of the framework and which candidate items may be most valued in these areas.

5. Almost, it could be say that if any other researcher were to conduct this study, they would likely use the same approach. As a result, I believe there is a lack of specific innovation or creativity in this protocol. It is recommended to consider modifications to the research methodology or interventions to enhance the protocol's appeal and impact, thereby creating more pronounced distinctions from similar studies.

While we appreciate the perspective regarding the potential similarity of approaches among researchers, we respectfully suggest that the strength of this protocol lies in its emphasis on rigour and reproducibility. Using a multinational, multidisciplinary e-Delphi process ensures an evidence-based, comprehensive and collaborative approach. This methodology prioritizes consensus-building among diverse experts. We note that rigorous scoping reviews and consensus studies have not been performed for surveillance evaluation frameworks.

The concept of international consensus for public health policy and decision-making was elevated during the Covid-19 pandemic [1–3]. Therefore, our innovation is in applying lessons learned from consensus-building during the pandemic to this space. Given, that wastewater-based surveillance is multidisciplinary and currently evolving, we feel that is would benefit from a rigorous consensus-based exercise.

Works Cited:

1. Lazarus JV, Romero D, Kopka CJ, Karim SA, Abu-Raddad LJ, Almeida G, et al. A multinational Delphi consensus to end the COVID-19 public health threat. Nature. 2022;611: 332–345. doi:10.1038/s41586-022-05398-2

2. Hillmer MP, Feng P, McLaughlin JR, Murty VK, Sander B, Greenberg A, et al. Ontario’s COVID-19 Modelling Consensus Table: mobilizing scientific expertise to support pandemic response. Can J Public Health. 2021;112: 799–806. doi:10.17269/s41997-021-00559-8

3. Balestracci B, La Regina M, Di Sessa D, Mucci N, Angelone FD, D’Ecclesia A, et al. Patient safety implications of wearing a face mask for prevention in the era of COVID-19 pandemic: a systematic review and consensus recommendations. Intern Emerg Med. 2023;18: 275–296. doi:10.1007/s11739-022-03083-w

Reviewer #2: The methods and protocol sections are comprehensive and well-structured. This reflects a good approach to the Delphi process.

6. I think there is an opportunity to provide additional detail to better support clarity and detail reproducibility. For instance, specifying the scoring scale used by panelists and clearly defining the criteria for achieving consensus would strengthen the methodology.

Thank you. We agree and we have revised to better explain the consensus rating process. We have already incorporated the details of the fully labelled 7-point Likert scale to be used by panellists in Fig 2. However, we have now included the rating properties that will be used to score each candidate item. We have incorporated your feedback in the “Defining consensus” section of the manuscript and are including a new supporting information file as part of the revision titled “S4 Appendix. Rating properties for e-Delphi candidate item scoring.”

7. The protocol emphasizes EDI, but the selection process for panelists could be better explained to outline how diversity in expertise and demographics will be ensured, and importantly how it might shape/improve the outcomes.

Thank you for your feedback. The study working group will monitor the distribution of panellists based on demographic information submitted during active recruitment. Based on the submitted information, the study working group will conduct targeted recruitment of certain cohorts (e.g., discipline subgroup, world region, etc.) if it appears that they are currently underrepresented. Targeted recruitment of specific cohorts will be conducted to ensure a diversity in expertise and demographics among panellists. This information has been incorporated in the “Recruitment of panellists” and “Equity, diversity, and inclusion (EDI)” sections of the manuscript.

8. While the quantitative analysis is detailed, further explanation of how median, range, and IQR will be used to define consensus and stability is needed. Also, elaborating on why Fisher’s exact test was chosen and its threshold for significance would provide a stronger rationale for the approach. I like to see stats choices well defined.

Fisher’s exact test was chosen for its suitability in analyzing small sample sizes and categorical data, as it does not rely on the assumption of large sample sizes required by other tests, such as the chi-squared test. This makes it a good choice for comparing item ratings across subgroups, particularly given the potential variability in panellist numbers across disciplines, income levels, and regions.

Regarding the threshold for significance, we will use a standard alpha level of 0.05, unless multiple comparisons necessitate an adjustment, such as using the Bonferroni correction. The median, range, and IQR will not be used to define consensus but rather to report stability and the differences between groups. They will only be used for descriptive secondary analyses of e-Delphi results. Including this rationale in the manuscript will provide greater transparency and clarity, and we appreciate your suggestion to ensure our statistical choices are well-defined and justified.

9. The ethical considerations part is strong, though additional information on how informed consent will be managed during hybrid meetings will strengthen the methodology.

We have modified the “Step 4: Consensus meeting” section of the manuscript.

Study executive group members will attend the consensus meeting. Therefore, an informed consent process will not be required, as executive group members are co-authors on this study. The consensus meeting will be guided by a Terms of Reference (ToR) document that executive group members agreed upon at the start of the study process. Chatham House Rule will apply to all executive group meetings, including the consensus meeting. The rule will encourage the free expression of ideas without attributing them to an individual or institution.

10. Are there any fallback plans for managing (potential) panelist dropouts (it is a common challenge for a Delphi), particularly in subgroups? This should be noted.

Thank you for highlighting the importance of having a mitigation plan. Our fallback plans for managing potential panellist dropouts include recruitment and retention strategies. We aim to conduct robust recruitment approximately four weeks before the start of the e-Delphi survey. We will also continue recruiting throughout each e-Delphi survey round (approximately 4-6 weeks). We are allowing new panellists to participate in the second round to mitigate our reliance on panellist from the first round. This means panellists can participate in the first or second round only, or both, depending on their preference. However, to help retention efforts, we only offer compensation to panellists who complete all rounds of the e-Delphi survey.

As noted in our response to question #7, the study working group will monitor the distribution of panellists based on demographic information submitted during active recruitment. Based on the submitted information, the study working group will conduct targeted recruitment of certain cohorts (e.g., discipline subgroup, world region, etc.) if it appears that they are currently underrepresented. Targeted recruitment will occur to help mitigate against uneven panellist dropout across cohorts, to ensure a diversity in expertise and demographics among panellists.

11. The grammar and style of the paper are clear but could benefit from a careful edit and re-read. Several sentences, particularly in the "Study Record Retention" and "Research Data Management" sections, are lengthy and could be restructured into shorter, more readable forms. Redundancy in describing data encryption and ethical compliance could be streamlined to avoid repetition. And, yes I know this is stylistic, but use of active voice would improve the paper -- especially for professional audiences. I also suggest adding really basic introductory sentences before diving into complex details in sections (like EDI) to improve the overall flow and applicability of content.

Thank you for your helpful feedback. We have revised and streamlined the "Study Record Retention" and "Research Data Management" sections. We have also added two introductory sentences to the beginning of the “Equity, diversity, and inclusion (EDI)” section. We have reviewed the entire manuscript to improve the overall flow and applicability of the content.

12. To strengthen the draft, maybe provide templates or examples for the demographic questionnaire and controlled feedback forms. These would enhance transparency.

Thank you for your thoughtful suggestion. We have included two new supporting information files as part of the revision titled: “S4 Appendix. Rating properties for e-Delphi candidate item scoring” and “S7 Appendix. Draft e-Delphi panellist screening questionnaire.”

13. Plans for dissemination are good, particularly the integration of a website and social media platforms. Perhaps including strategies for keeping the framework dynamic and updated post-publication would be helpful?

Thank you for your valuable feedback. We have revised the “Step 5: Dissemination of evaluation framework” section to include further details about our post-publication knowledge translation strategies.

Post-publication, we will aim to keep the framework dynamic and up to date using a website, Discourse server, or other social media platforms to elicit ongoing feedback. We are planning on creating a living review from the scoping studies. Those studies have entailed generating an ontology to synthesize concepts. That ontology will be finalized based on the consensus exercise and published.

Post-publication strategies may also include creating guides and toolkits to enhance the practical applicability of the evaluation framework and facilitate its adoption for policy and decision-making.

14. Are there strategies for participant fatigue? Such as incentives or even streamlined surveys. Recognizing the potential influence of controlled feedback on panelist opinions and detailing mitigation strategies to minimize bias could improve outcomes.

Thank you for highlighting the importance of having a mitigation plan. To help retention efforts, we will only compensate panellists who complete all rounds of the e-Delphi survey. We will administer streamlined survey pathways depending on the panellist’s expertise. For example, members of the engaged public will have the option to skip ascertainment performance measurements to allow them to focus on attributes such as communication and partnerships.

Our fallback plans for managing potential panellist dropouts include recruitment and retention strategies. We aim to conduct robust recruitment approximately four weeks before the start of the e-Delphi survey. We will also continue recruiting throughout each e-Delphi survey round (approximately 4-6 weeks). We are allowing new panellists to participate in the second round to mitigate our reliance on panellist dropouts. This means panellists can participate in the first or second round only, or both, depending on their preference.

15. Overall, the protocol has a good methodology and commitment to inclusivity, but it would benefit from more precise descriptions of statistical methods, panelist selection processes, and importantly an outline of contingency

Attachment

Submitted filename: Response to Reviewers.pdf

pone.0310342.s009.pdf^{(285.1KB, pdf)}

PLoS One. doi: 10.1371/journal.pone.0310342.r004

Decision Letter 1

Angela Mendes Freitas

22 Apr 2025

Developing an evaluation framework for public health environmental surveillance: Protocol for an international, multidisciplinary Delphi study

PONE-D-24-33538R1

Dear Dr. Manuel,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Angela Mendes Freitas

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Does the manuscript provide a valid rationale for the proposed study, with clearly identified and justified research questions?

The research question outlined is expected to address a valid academic problem or topic and contribute to the base of knowledge in the field.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Is the protocol technically sound and planned in a manner that will lead to a meaningful outcome and allow testing the stated hypotheses?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Is the methodology feasible and described in sufficient detail to allow the work to be replicable?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors described where all data underlying the findings will be made available when the study is complete?

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

You may also provide optional suggestions and comments to authors that they might find helpful in planning their study.

(Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I have carefully reviewed the authors' responses to my initial comments and the corresponding revisions made to the manuscript. I appreciate the thorough and thoughtful manner in which the authors have addressed all the raised concerns.

Reviewer #2: The authors have addressed the reviewer comments and suggestion. I look forward to seeing how the project is implemented. It has relevance to other Delhi approaches and will be a helpful; model in other fields.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy .

Reviewer #1: No

Reviewer #2: No

**********

PLoS One. doi: 10.1371/journal.pone.0310342.r005

Acceptance letter

Angela Mendes Freitas

PONE-D-24-33538R1

PLOS ONE

Dear Dr. Manuel,

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team.

At this stage, our production department will prepare your paper for publication. This includes ensuring the following:

* All references, tables, and figures are properly cited

* All relevant supporting information is included in the manuscript submission,

* There are no issues that prevent the paper from being properly typeset

You will receive further instructions from the production team, including instructions on how to review your proof when it is ready. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few days to review your paper and let you know the next and final steps.

Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

If we can help with anything else, please email us at customercare@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Angela Mendes Freitas

Academic Editor

PLOS ONE

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

S1 Table. Executive group composition.

(PDF)

pone.0310342.s001.pdf^{(91.7KB, pdf)}

S2 Table. Executive group and e-Delphi panellist disciplines.

The list of specializations or related concepts is not extensive.

(PDF)

pone.0310342.s002.pdf^{(86.1KB, pdf)}

S3 Table. World region classifications.

(PDF)

pone.0310342.s003.pdf^{(82.1KB, pdf)}

S1 Appendix. PRISMA-ScR checklist for scoping review #1.

Current Evaluation Practices for Public Health and Environmental Surveillance Systems: A Scoping Review Protocol.

(PDF)

pone.0310342.s004.pdf^{(104.8KB, pdf)}

S2 Appendix. PRISMA-ScR checklist for scoping review #2.

Lessons Learned from Wastewater-Based Surveillance During the COVID-19 Pandemic: A Scoping Review Protocol.

(PDF)

pone.0310342.s005.pdf^{(104.9KB, pdf)}

S3 Appendix. Rating properties for e-Delphi candidate item scoring.

(PDF)

pone.0310342.s006.pdf^{(194.8KB, pdf)}

S4 Appendix. Draft e-Delphi panellist screening questionnaire.

(PDF)

pone.0310342.s007.pdf^{(184.6KB, pdf)}

Attachment

Submitted filename: Response to Reviewers.pdf

pone.0310342.s009.pdf^{(285.1KB, pdf)}

Data Availability Statement

No datasets were generated or analysed during the current study. All relevant data from this study will be made available upon study completion.

[pone.0310342.ref001] 1.Woodward G, Manuel DG, Goel V. Developing a balanced scorecard for public health. Toronto: Institute for Clinical Evaluative Sciences; 2004. [Google Scholar]

[pone.0310342.ref002] 2.McClary-Gutierrez JS, Mattioli MC, Marcenac P, Silverman AI, Boehm AB, Bibby K, et al. SARS-CoV-2 Wastewater Surveillance for Public Health Action. Emerg Infect Dis. 2021;27(9):1–8. doi: 10.3201/eid2709.210753 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref003] 3.Diamond MB, Keshaviah A, Bento AI, Conroy-Ben O, Driver EM, Ensor KB, et al. Wastewater surveillance of pathogens can inform public health responses. Nat Med. 2022;28(10):1992–5. doi: 10.1038/s41591-022-01940-x [DOI] [PubMed] [Google Scholar]

[pone.0310342.ref004] 4.Zuccato E, Chiabrando C, Castiglioni S, Bagnati R, Fanelli R. Estimating community drug abuse by wastewater analysis. Environ Health Perspect. 2008;116(8):1027–32. doi: 10.1289/ehp.11022 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref005] 5.Hrudey SE, Bischel HN, Charrois J, Chik AHS, Conant B, Delatolla R, et al. Wastewater Surveillance for SARS-CoV-2 RNA in Canada. FACETS. 2022;7:1493–597. doi: 10.1139/facets-2022-0148 [DOI] [Google Scholar]

[pone.0310342.ref006] 6.World Health Organization. Environmental surveillance for SARS-COV-2 to complement public health surveillance. World Health Organization; 2022. Apr. [Google Scholar]

[pone.0310342.ref007] 7.Manuel DG, Delatolla R, Fisman DN, Fuzzen M, Graber T, Katz GM, et al. The Role of Wastewater Testing for SARS-CoV-2 Surveillance. Ontario COVID-19 Science Advisory Table; 2021. Aug. doi: 10.47326/ocsat.2021.02.40.1.0 [DOI] [Google Scholar]

[pone.0310342.ref008] 8.Tiwari A, Adhikari S, Kaya D, Islam MA, Malla B, Sherchan SP, et al. Monkeypox outbreak: Wastewater and environmental surveillance perspective. Sci Total Environ. 2023;856(Pt 2):159166. doi: 10.1016/j.scitotenv.2022.159166 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref009] 9.Wolfe MK, Duong D, Bakker KM, Ammerman M, Mortenson L, Hughes B, et al. Wastewater-Based Detection of Two Influenza Outbreaks. Environ Sci Technol Lett. 2022;9(8):687–92. doi: 10.1021/acs.estlett.2c00350 [DOI] [Google Scholar]

[pone.0310342.ref010] 10.Sims N, Kasprzyk-Hordern B. Future perspectives of wastewater-based epidemiology: Monitoring infectious disease spread and resistance to the community level. Environ Int. 2020;139:105689. doi: 10.1016/j.envint.2020.105689 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref011] 11.Huijbers PMC, Flach C-F, Larsson DGJ. A conceptual framework for the environmental surveillance of antibiotics and antibiotic resistance. Environ Int. 2019;130:104880. doi: 10.1016/j.envint.2019.05.074 [DOI] [PubMed] [Google Scholar]

[pone.0310342.ref012] 12.Tiwari A, Kurittu P, Al-Mustapha AI, Heljanko V, Johansson V, Thakali O, et al. Wastewater surveillance of antibiotic-resistant bacterial pathogens: A systematic review. Front Microbiol. 2022;13:977106. doi: 10.3389/fmicb.2022.977106 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref013] 13.Yang D, Zheng Q, Thai PK, Ahmed F, O’Brien JW, Mueller JF, et al. A nationwide wastewater-based assessment of metformin consumption across Australia. Environ Int. 2022;165:107282. doi: 10.1016/j.envint.2022.107282 [DOI] [PubMed] [Google Scholar]

[pone.0310342.ref014] 14.Escolà Casas M, Schröter NS, Zammit I, Castaño-Trias M, Rodriguez-Mozaz S, Gago-Ferrero P, et al. Showcasing the potential of wastewater-based epidemiology to track pharmaceuticals consumption in cities: Comparison against prescription data collected at fine spatial resolution. Environ Int. 2021;150:106404. doi: 10.1016/j.envint.2021.106404 [DOI] [PubMed] [Google Scholar]

[pone.0310342.ref015] 15.Woytus A. What’s in the wastewater? In: St. Louis [Internet]. 22 Sep 2021. [cited 1 Jun 2023]. Available from: https://www.stlmag.com/api/content/3f60d498-1b26-11ec-a8ea-1244d5f7c7c6/ [Google Scholar]

[pone.0310342.ref016] 16.D’Aoust PM, Towhid ST, Mercier É, Hegazy N, Tian X, Bhatnagar K, et al. COVID-19 wastewater surveillance in rural communities: Comparison of lagoon and pumping station samples. Sci Total Environ. 2021;801:149618. doi: 10.1016/j.scitotenv.2021.149618 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref017] 17.Shah S, Gwee SXW, Ng JQX, Lau N, Koh J, Pang J. Wastewater surveillance to infer COVID-19 transmission: A systematic review. Sci Total Environ. 2022;804:150060. doi: 10.1016/j.scitotenv.2021.150060 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref018] 18.Liu P, Ibaraki M, VanTassell J, Geith K, Cavallo M, Kann R, et al. A sensitive, simple, and low-cost method for COVID-19 wastewater surveillance at an institutional level. Sci Total Environ. 2022;807(Pt 3):151047. doi: 10.1016/j.scitotenv.2021.151047 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref019] 19.Larsen DA, Wigginton KR. Tracking COVID-19 with wastewater. Nat Biotechnol. 2020;38(10):1151–3. doi: 10.1038/s41587-020-0690-1 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref020] 20.Gretchen Vogel. Signals from the sewer—Measuring virus levels in wastewater can help track the pandemic. But how useful is that? Science. 9 Mar. 2022375. Available from: https://www.science.org/content/article/pandemic-signals-sewer-what-virus-levels-wastewater-tell-us [Google Scholar]

[pone.0310342.ref021] 21.Manuel D, Amadei CA, Campbell JR, Brault J, Zierler A, Veillard J. Strengthening Public Health Surveillance Through Wastewater Testing: An Essential Investment for the COVID-19 Pandemic and Future Health Threats. World Bank Group. The World Bank; 2022. p. 1–67. [Google Scholar]

[pone.0310342.ref022] 22.Fynn JF, Hardeman W, Milton K, Jones AP. A scoping review of evaluation frameworks and their applicability to real-world physical activity and dietary change programme evaluation. BMC Public Health. 2020;20(1):1000. doi: 10.1186/s12889-020-09062-0 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref023] 23.Nilsen P. Making sense of implementation theories, models, and frameworks. Implement Sci. 2020;30:53–79. [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref024] 24.Calba C, Goutard FL, Hoinville L, Hendrikx P, Lindberg A, Saegerman C, et al. Surveillance systems evaluation: a systematic review of the existing approaches. BMC Public Health. 2015;15:448. doi: 10.1186/s12889-015-1791-5 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref025] 25.Andrews JR, Yu AT, Saha S, Shakya J, Aiemjoy K, Horng L, et al. Environmental Surveillance as a Tool for Identifying High-risk Settings for Typhoid Transmission. Clin Infect Dis. 2020;71(Suppl 2):S71–8. doi: 10.1093/cid/ciaa513 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref026] 26.Porta M. A Dictionary of Epidemiology. A Dictionary of Epidemiology. Oxford University Press; 2016. Available from: https://www.oxfordreference.com/display/10.1093/acref/9780199976720.001.0001/acref-9780199976720 [Google Scholar]

[pone.0310342.ref027] 27.Spranger J, Homberg A, Sonnberger M, Niederberger M. Reporting guidelines for delphi techniques in health sciences: a methodological review. Z Evid Fortbild Qual Gesundh Wesen. 2023;172:1–11. [DOI] [PubMed] [Google Scholar]

[pone.0310342.ref028] 28.Editorial. How epidemiology has shaped the COVID pandemic. Nature. 2021;589(7843):491–2. doi: 10.1038/d41586-021-00183-z [DOI] [PubMed] [Google Scholar]

[pone.0310342.ref029] 29.Devine D, Gaskell J, Jennings W, Stoker G. Trust and the Coronavirus Pandemic: What are the Consequences of and for Trust? An Early Review of the Literature. Polit Stud Rev. 2021;19(2):274–85. doi: 10.1177/1478929920948684 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref030] 30.World Health Organization. Defining collaborative surveillance: a core concept for strengthening the global architecture for health emergency preparedness, response, and resilience (HEPR). Geneva: World Health Organization; 2023. Available from: https://iris.who.int/handle/10665/367927 [Google Scholar]

[pone.0310342.ref031] 31.Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169(7):467–73. doi: 10.7326/M18-0850 [DOI] [PubMed] [Google Scholar]

[pone.0310342.ref032] 32.Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: How to decide its appropriateness. World J Methodol. 2021;11(4):116–29. doi: 10.5662/wjm.v11.i4.116 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref033] 33.Shang Z. Use of Delphi in health sciences research: A narrative review. Medicine (Baltimore). 2023;102(7):e32829. doi: 10.1097/MD.0000000000032829 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref034] 34.Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017;31(8):684–706. doi: 10.1177/0269216317690685 [DOI] [PubMed] [Google Scholar]

[pone.0310342.ref035] 35.Niederberger M, Spranger J. Delphi Technique in Health Sciences: A Map. Front Public Health. 2020;8:457. doi: 10.3389/fpubh.2020.00457 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref036] 36.Schmalz U, Spinler S, Ringbeck J. Lessons Learned from a Two-Round Delphi-based Scenario Study. MethodsX. 2020;8:101179. doi: 10.1016/j.mex.2020.101179 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref037] 37.Boel A, Navarro-Compán V, Landewé R, van der Heijde D. Two different invitation approaches for consecutive rounds of a Delphi survey led to comparable final outcome. J Clin Epidemiol. 2021;129:31–9. doi: 10.1016/j.jclinepi.2020.09.034 [DOI] [PubMed] [Google Scholar]

[pone.0310342.ref038] 38.Gattrell WT, Logullo P, van Zuuren EJ, Price A, Hughes EL, Blazey P, et al. ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi. PLoS Med. 2024;21(1):e1004326. doi: 10.1371/journal.pmed.1004326 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref039] 39.Permuth-Wey J, Borenstein AR. Financial remuneration for clinical and behavioral research participation: ethical and practical considerations. Ann Epidemiol. 2009;19(4):280–5. doi: 10.1016/j.annepidem.2009.01.004 [DOI] [PubMed] [Google Scholar]

[pone.0310342.ref040] 40.Surveylet. United States: Calibrum. Available from: https://calibrum.com [Google Scholar]

[pone.0310342.ref041] 41.LimeSurvey: An open source survey tool. Hamburg, Germany: LimeSurvey GmbH; Available from: https://www.limesurvey.org [Google Scholar]

[pone.0310342.ref042] 42.Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401–9. doi: 10.1016/j.jclinepi.2013.12.002 [DOI] [PubMed] [Google Scholar]

[pone.0310342.ref043] 43.Belton I, MacDonald A, Wright G, Hamlin I. Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process. Technological Forecasting and Social Change. 2019;147:72–82. doi: 10.1016/j.techfore.2019.07.002 [DOI] [Google Scholar]

[pone.0310342.ref044] 44.Weijters B, Cabooter E, Schillewaert N. The effect of rating scale format on response styles: The number of response categories and response category labels. International Journal of Research in Marketing. 2010;27(3):236–47. doi: 10.1016/j.ijresmar.2010.02.004 [DOI] [Google Scholar]

[pone.0310342.ref045] 45.Lange T, Kopkow C, Lützner J, Günther K-P, Gravius S, Scharf H-P, et al. Comparison of different rating scales for the use in Delphi studies: different scales lead to different consensus and show different test-retest reliability. BMC Med Res Methodol. 2020;20(1):28. doi: 10.1186/s12874-020-0912-8 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref046] 46.Locke EA, Latham GP. A theory of goal setting & task performance. Prentice-Hall, Inc. 1990. [Google Scholar]

[pone.0310342.ref047] 47.Veillard J, Champagne F, Klazinga N, Kazandjian V, Arah OA, Guisset A-L. A performance assessment framework for hospitals: the WHO regional office for Europe PATH project. Int J Qual Health Care. 2005;17(6):487–96. doi: 10.1093/intqhc/mzi072 [DOI] [PubMed] [Google Scholar]

[pone.0310342.ref048] 48.Kaplan RS, Norton DP. Balanced scorecard. Springer; 2007. [Google Scholar]

[pone.0310342.ref049] 49.Mitchell RJ, Williamson AM, O’Connor R. The development of an evaluation framework for injury surveillance systems. BMC Public Health. 2009;9:260. doi: 10.1186/1471-2458-9-260 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref050] 50.Nothacker M, Stokes T, Shaw B, Lindsay P, Sipilä R, Follmann M, et al. Reporting standards for guideline-based performance measures. Implement Sci. 2016;11:6. doi: 10.1186/s13012-015-0369-z [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref051] 51.Groseclose SL, German RR, Nsubuga P. Evaluating Public Health Surveillance. Principles & Practice of Public Health Surveillance. Oxford University Press. 2010. p. 166–97. doi: 10.1093/acprof:oso/9780195372922.003.0008 [DOI] [Google Scholar]

[pone.0310342.ref052] 52.Government of Canada. Best Practices in Equity, Diversity and Inclusion in Research. In: New Frontiers in Research Fund [Internet]. 22 Jun 2021. Available from: https://www.sshrc-crsh.gc.ca/funding-financement/nfrf-fnfr/edi-eng.aspx [Google Scholar]

[pone.0310342.ref053] 53.Government of Canada. Introduction to GBA Plus. 31 Mar 2021. [cited 1 Jun 2023]. Available from: https://women-gender-equality.canada.ca/gbaplus-course-cours-acsplus/eng/mod00/mod00_01_01.html [Google Scholar]

[pone.0310342.ref054] 54.The First Nations Information Governance Centre. The First Nations Principles of OCAP®. In: The First Nations Information Governance Centre [Internet]. 1 Jun 2023. [cited 1 Jun 2023]. Available from: https://fnigc.ca/ocap-training/ [Google Scholar]

[pone.0310342.ref055] 55.Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ. 2017;358:j3453. doi: 10.1136/bmj.j3453 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref056] 56.Yang L, Weston C, Cude C, Kincl L. Evaluating Oregon’s occupational public health surveillance system based on the CDC updated guidelines. Am J Ind Med. 2020;63(8):713–25. doi: 10.1002/ajim.23139 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0310342.ref057] 57.Winkler J, Moser R. Biases in future-oriented Delphi studies: A cognitive perspective. Technological Forecasting and Social Change. 2016;105:63–76. doi: 10.1016/j.techfore.2016.01.021 [DOI] [Google Scholar]

[pone.0310342.ref058] 58.Goodman CM. The Delphi technique: a critique. J Adv Nurs. 1987;12(6):729–34. doi: 10.1111/j.1365-2648.1987.tb01376.x [DOI] [PubMed] [Google Scholar]

PERMALINK

Developing an evaluation framework for public health environmental surveillance: Protocol for an international, multidisciplinary Delphi consensus study

Douglas G Manuel

Carol Bennett

Emma Brown

David L Buckeridge

Yoni Freedhoff

Sarah Funnell

Farah Ishtiaq

Matthew J Wade

David Moher

Roles

Abstract

Introduction

Methods

Discussion

Introduction

Study objectives

Definitions

Materials and methods

Study design

Fig 1. Overview of the development process for PHES-EF.

Step 1: Establish the executive group

Step 2: Generate candidate evaluation items

Scoping reviews

Content analysis and extraction

Executive group consultation

Step 3: International e-Delphi survey

Recruitment of panellists

Table 1. Eligibility criteria for e-Delphi survey panellists.

Panellist withdrawal or termination

Panellist compensation

Procedure

Fig 2. E-Delphi and consensus meeting methodology, methods to achieve consensus on core items for PHES-EF.

Defining consensus

Stopping rules

Round 1

Round 2

Summary panellist characteristics

Quantitative analysis

Qualitative analysis

Grouping of subgroups

Exclusion of data

Step 4: Consensus meeting

Step 5: Dissemination of evaluation framework

Equity, diversity, and inclusion (EDI)

Knowledge user and public involvement

Acknowledgments

Research data management

Study record retention

Ethical considerations

Discussion

Limitations

Supporting information

Acknowledgments

Data Availability

Funding Statement

References

Author response to Decision Letter 0

Decision Letter 0

Angela Mendes Freitas

Roles

Author response to Decision Letter 1

Decision Letter 1

Angela Mendes Freitas

Roles

Acceptance letter

Angela Mendes Freitas

Roles

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases