Protocol for a modified Delphi study of ethical oversight of data science health research (DSHR)

Abstract

Introduction

The use of data science for health research produces complex ethical, legal and social challenges that traditional ethical oversight mechanisms struggle to address. In Nigeria, the current ethical guidelines were not designed for these challenges which include pervasive data environments, consent for secondary data use, algorithmic decision-making and bias, privacy risks, involvement of commercial entities, data colonisation, inequitable benefit-sharing and commercial data holdings. To address these gaps, we developed a draft guideline incorporating principles like trust, veracity, global justice and alternative ethical approval mechanisms. Here, we describe the protocol for a study aimed at validating the guideline through stakeholder consensus on the content, feasibility and acceptability of this subcode for national implementation.

Methods and analysis

We describe the use of a modified e-Delphi approach to iteratively synthesize expert opinions about ethical oversight for data science health research (DSHR) led by a multidisciplinary working group from the Bridging Gaps in the ELSI of Data Science Health Research in Nigeria (BridgELSI) team. We will invite 65 experts, including health researchers, ethics committee members, data scientists, health policymakers, funders and key opinion leaders in Nigeria to participate. Participants will rate 13 core principles, including global justice, algorithmic bias, data governance and related governance provisions on importance, desirability for inclusion in national guidelines, feasibility and confidence in implementation, using 5-point Likert scales, with optional free-text comments. We will summarise responses using descriptive statistics, assess consensus and polarity using pre-specified thresholds for the mean and IQR, and iteratively refine statements between rounds using qualitative content analysis of comments.

Ethics and dissemination

Ethical approval was obtained from the Nigerian National Health Research Ethics Committee and the University of Maryland IRB, and participants will provide informed consent. Results will be shared with the expert panel and national regulators and disseminated via publications and conferences.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• The study engages a broad range of local stakeholders using a rigorous Delphi method to ensure that the guidelines are informed by diverse perspectives and tailored to the Nigerian context.

• The iterative Delphi design (anonymous feedback and multiple rounds) minimises dominant influence and geographical barriers for dispersed participants.

• As the resulting guidelines reflect Nigerian expert opinion, they may not generalise to other settings. The Delphi process also relies on sustained participant engagement, which could limit the breadth of viewpoints captured if significant number of participants disengage.

Introduction

The use of data science for health research is a novel and rapidly growing field derived from several disciplines that have coalesced around the use of novel computational algorithms and high-performance computing to collect, store, manipulate and analyse huge amounts of health and related data (big data) to generate results/reports and novel insights.^{1 2} Given the rapid and transformative growth of data science health research (DSHR) technologies and applications, it is necessary to develop ethical oversight mechanisms that ensure the protection of human and animal participants in DSHR.^{1 3} However, developing ethical oversight guidelines and infrastructure typically lags behind the development of new research methods.^{4 5} Whereas this delay is well known, the speed and scope of change being wrought by data science applications have significantly and substantially increased the lag to a greater extent than is usual.⁵

Data science in general and its applications in health research in particular raise major and new ethical, legal and social implications (ELSI) that traditional ethical oversight mechanisms are inadequate for.^{4 6} The traditional mechanisms struggle to address issues such as informed consent in secondary data use, algorithmic bias, privacy risks from re-identification and commercial exploitation.^{4 7} For instance, while anonymisation and de-identification are cornerstones of participant protection in health research, DSHR’s ability to re-identify individuals through data linkage undermines these safeguards.^{8 9} Similarly, algorithmic bias—stemming from unrepresentative training data or skewed researcher perspectives—can perpetuate health inequities, particularly for under-represented populations like those in Africa.^{10 11} The deep involvement of commercial entities further complicates DSHR, because multinational corporations dominate data ownership, raising concerns about data colonisation and inequitable benefit-sharing.^{12 13} These challenges necessitate robust ethical governance frameworks tailored to DSHR unique demands.

In Nigeria, a large and typical low- and middle-income country (LMIC), the National Health Research Ethics Committee (NHREC) established in 2006 oversees research through the National Code for Health Research Ethics (NCHRE).¹⁴ While the NCHRE provides foundational guidance, its provisions are inadequate for the complexities of DSHR, such as pervasive data environments, cross-border collaborations and algorithmic accountability.^{1 15} To address this gap in ethical oversight of DSHR, we developed a subcode of the NCHRE, incorporating principles like trust, veracity, global justice and alternative ethical approval mechanisms (eg, broad consent, exemptions for low-risk secondary data use).^{16 17}

To validate this subcode, we, members of the NIH-funded Bridging Gaps in the ELSI of Data Science Health Research in Nigeria (BridgELSI) team, are conducting a modified Delphi study to iteratively synthesise expert opinions about the ethical oversight of DSHR in a typical LMIC like Nigeria.¹⁸ The Delphi method, developed by the RAND Corporation, and appropriately modified, is an ideal research approach for eliciting and reconciling diverse stakeholder perspectives through structured consensus-building processes.^{19 20} Our objective is to build consensus among representative stakeholders on the practicality and acceptability of the subcode.²¹ By refining the subcode, we aim to balance DSHR’s transformative potential with equitable protections for participants and communities.

Methods and analysis

While there are many variants of the Delphi method, we will deploy the approach that is used to generate consensus of opinions from a group of experts by requiring them to respond to a series of questions interspersed with controlled feedback.^{22 23} This design has a wide range of usefulness and applicability across many disciplines and settings, including medicine, technology and the social sciences.24,26 Some of the main characteristics of this approach include anonymity of participants (experts), multiple iterations, consensus and controlled feedback. In this study, we will use a virtual Delphi method where participants respond to questions in a survey that is hosted in an electronic database (figure 1).

Figure 1. Procedure for achieving consensus on the contents of a subcode for ethical oversight of data science health research in Nigeria.

Working group

The BridgELSI team conducting this study is a multidisciplinary research group with members from Nigeria and the USA. Members have expertise in clinical medicine (CAA, TO, SNA), public health (AA, CAA, SNA), law (SC, OCM, SA), sociology (AJ), psychology (OA), philosophy (PI), data science (CAA, SNA) and health research (AA, SNA, CAA). We will constitute a three-person working group (WG) consisting of CAA (team lead), AA and PI, who will implement and provide overall oversight for the study. Members of this working group have the requisite expertise in quantitative, qualitative, mixed, and Delphi methods. The WG will develop the data collection tool, review the tool and conduct content and face validity of the tool to ensure accuracy, comprehensiveness, clarity of wording and appropriateness of structure. The WG will conduct statistical analysis, review study results and provide feedback to panellists after each round.²⁷ The working group will meet regularly throughout the study and will be responsible for identifying and responding to issues that arise, reviewing study conduct, disseminating findings and providing regular updates to the central project group.²⁸ Members of the WG will report their progress every 2 weeks to other members of the BridgELSI investigators team, receive feedback, suggestions and make modifications as required.

Panel selection and recruitment

We will identify potential participants from the list of researchers participating in the NIH-funded Harnessing Data Science for Health in Africa (DS-I Africa) initiative, members of ethics committees in their institutions, researchers and data scientists who have publications on data science health research in Nigeria identified through a search of PubMed database, research funders/sponsors, policy makers on research and research ethics employed by the Nigerian government, key opinion leaders including leaders of consumer organisations and advocacy groups through personal contacts and referrals. We will categorise participants into four groups based on:

1. Health researchers who are using data science research methods for their projects.

2. Members of Health Research Ethics Committees that have reviewed data science health research proposals.

3. Data scientists who are conducting health research.

4. Policy makers, funders and key opinion leaders, including leaders of consumer organisations and advocacy groups, in data science health research.

Other criteria are that participants must be at least 18 years of age, must be residents of Nigeria or conduct research that directly affects Nigerians. We will use multiple strategies, including phone calls, personal contacts and emails to invite participants. In the invitation, we will provide a short background of the study and the planned Delphi process. We will follow-up with reminders for those who do not respond to the original email invitations or fail to return a signed consent form. We will motivate participation by sharing publications from our group and others on the need to develop ethical oversight for data science health research. Participants will be compensated with US$100 for their time, which will be paid in instalments throughout the study duration.

Panel size

There is no agreement on the appropriate size for a typical Delphi study panel, but there should be sufficient diversity to ensure representation by age, sex and perspectives.²⁰ The number of participants must be large enough to be able to calculate and analyse statistics properly.²² For transdisciplinary research, some researchers have proposed between 30 and 50 participants as a minimum panel size.^{22 29 30} For our study, we will invite 65 stakeholders in data science health research. Ultimately, the size of a Delphi panel is determined by factors such as the nature of the issue under consideration, logistics and availability of experts and resources.31,33

Drafting the statements and development of the questionnaire

Drafting the statements and development of the questionnaire represent key components of the modified Delphi approach. The questions should be clear, unambiguous and unbiased and the response options should be relevant to the questions asked and exhaustive.³⁴ Open-ended questions are ideal when the range of possible responses cannot be predetermined, particularly when it is necessary to identify knowledge gaps. Conversely, closed questions are appropriate when only a limited number of responses are expected, and the dimensions of responses are standardised. Most Delphi approaches use closed questions so data can be statistically analysed after each round. For our study, we will use closed-ended questions with a 5-point Likert scale responses to enable statistical comparison across rounds. We will supplement this with optional comment fields to allow the provision of qualitative feedback, suggest amendments to questions or provide justification for responses— all of which will be systematically reviewed and incorporated into subsequent iterations of the Delphi process.²² After each round, we will review and analyse the comments and incorporate the feedback into the next round.

We used two approaches to generate our questionnaire items. An explorative method involving key informant interviews with Nigerian stakeholders (researchers, ethicists, policymakers) was used to identify locally relevant principles and challenges, and we conducted a comprehensive literature review of existing governance frameworks and guidelines (including the Nigeria constitution, National Health Act, National Code of Health Research Ethics (NCHRE) and Nigeria’s Data Protection Act), and data science health research ethics publications.^{35 36} This hybrid approach enables us to avoid relying solely on published literature because this may constrain expert input.³⁶ It also enables us to ensure local relevance and alignment with international practices.

From this process, we identified 13 core principles for ethical oversight of data science health research. These are Trust, Veracity, Global Justice, FAIR (Findable, Accessible, Interoperable, Reusable) Principles, Confidentiality and Privacy Concerns, Alternative Modalities for Ethical Approval, Weaponisation of Data, Commercialisation, Data Security Breaches, Inferential Risk, Post-Approval Monitoring, Predatory Inclusion and Additional Classes of Risks. For each principle, a multidisciplinary working group of three experts (from bioethics, law, public health and data science) drafted clear, concise statements. Panellists will evaluate these statements across four parameters using 5-point Likert scales. We reviewed the questionnaires for clarity and bias and provided each principle with a contextual preamble explaining its relevance. We will conduct pilot testing with 5–10 participants who will not be part of the Delphi panel to assess face validity and technical functionality of the questionnaire.

Description of Delphi technique

We will employ a modified e-Delphi approach to accommodate the geographical dispersion of our panellists. The e-Delphi method offers several advantages over traditional Delphi techniques, including enhanced accessibility that allows participants to complete surveys at their convenience, regardless of location, thereby improving engagement rates.³⁷ The electronic format enables save-and-resume functionality, reducing partial responses while maintaining data integrity.³⁸ Real-time response tracking and automated reminders facilitate efficient data collection and help maintain the study timeline.³⁹ The approach preserves participant anonymity, minimising peer pressure and dominant respondent effects while encouraging independent judgement.³⁰ The digital platform minimises logistical challenges and costs associated with postal surveys or in-person meetings, while supporting the complex ranking of multi-dimensional statements required in our study.³⁷ For implementation, we will distribute surveys electronically with unique access links, send weekly automated reminders to non-responders and analyse data while ensuring accuracy throughout the consensus-building process.

We will send email invitations to all participants, inviting them to visit the study’s online Research Electronic Data Capture forms (REDCap) survey page. The online landing page will contain information about the study’s purpose, expected rounds of surveys (potentially three), risk, voluntariness and ability to withdraw from participation at any time without adverse consequences to all participants. We will collect age, sex, expertise and institutional affiliation of participants. Subsequent pages on the survey instrument will present statements on ethical oversight of data science health research, Likert response scales, and space for optional open-ended comments. Non-respondents will receive up to two reminders per round (7–10 days apart). All data will be stored on password-protected REDCap servers accessed via password-protected laptops with encrypted drives.

During the first round of the survey, we will aim to establish baseline consensus or divergence among experts regarding the prioritisation of the ethical considerations in our survey, providing a foundation for subsequent rounds where responses are refined through iterative group feedback.^{30 39} Given the e-Delphi format, we will regularly monitor response submissions to ensure data accuracy and completion. Participants will have an 8-week turnaround time to submit their responses, during which we will send frequent reminders (eg, weekly or biweekly) to non-responders to maximise participation rates.³⁷ On closing the first round, we will analyse the data to assess the level of agreement between the panellists. The results of this analysis will then be distributed to all panellists, allowing them to compare their individual responses with the aggregated group findings. Participants will also be informed that they will have the opportunity to revise their rankings in the subsequent round(s) based on this feedback, which is a standard practice in Delphi methodology aimed at fostering structured convergence toward consensus.

After the analysis of the data from the first round, the working group will conduct a detailed review of the statements that were significantly discordant and analyse comments included by the panellists in their responses to see if there are other concerns regarding the statements. We will incorporate the findings from the analysis of the comments into the creation of revised statements which will be implemented in future rounds. We will also gather additional information about the discordant statements before presenting them to the experts for reranking. The need for additional rounds depends on the responses/findings from the analysis of data from each round and the level of consensus attained in the preceding round. In subsequent rounds, we will re-present reviewed statements to the experts for reranking similar to the procedure in the first round. We aim to keep the number of rounds to three to reduce participants’ fatigue and avoid ‘forced consensus’.

Management of non-responders

Participant attrition is a known challenge with Delphi methods, with attrition between rounds ranging from 0% to 92%.^{40 41} We will implement several measures to reduce participants’ non-response. We plan to limit the number of rounds, and we may reduce the number of items in the surveys to reduce participant fatigue.⁴² We will also grant delayed responders additional, but not excessive time between surveys.⁴² During the surveys, we will use personalised and regular reminders, build partnership and ownership of the study, provide support materials and show appreciation for participation.⁴² High attrition, progressive attrition between rounds and systematic differences between those who stay and those who withdraw can introduce bias and affect the validity of consensus results.^{42 43} We will set a response threshold at 75%, and we will document the attrition rate, compare responses of responders and dropouts to evaluate attrition bias, and analyse whether those with particular responses are more likely to drop out.⁴⁴ We will conduct in-depth interviews with non-responders to identify reasons for non-response and use the outcome to modify our study procedures as required.^{40 45} We will not replace non-responders in any round with new participants but will keep inviting them to subsequent rounds. Non-responders who return in subsequent rounds will receive the prior round’s feedback before participating in the new rating. If attrition is substantial, we will conduct sensitivity analyses to examine whether results differ with the inclusion of participants with incomplete data or when weighting responses based on completion patterns.^{40 45 46} Nevertheless, research comparing responders and non-responders in Delphi studies has yielded mixed results and attrition may not always introduce systematic bias.46,48

Data collection

We will collect data during the iterative rounds of surveys administered to the panellists until consensus is reached. To ensure inclusiveness and flexibility, we will employ two response formats: (a) electronic survey forms using REDCap and (b) soft-copy questionnaires (PDF/Word).⁴⁹ Panellists will receive an email with both options, allowing them to choose their preferred method. Participants opting for the soft copy will return it via email, and the research team will enter the responses into the REDCap database to maintain a centralised dataset. We selected REDCap because of its secure HIPAA-compliant infrastructure, audit trails, strong data security, institutional integration, compliance with data protection regulations and specialised features for multi-round Delphi studies, such as longitudinal survey management and data validation.^{37 49} Another advantage of the data validation is to prevent missing or submission of incomplete data.

Data analysis plan and determination of consensus

To analyse data for this study, we will rank the responses to the individual statements on consensus, polarity and support. Panellists will use a 5-point Likert scale to rate each statement on four categories, namely relative importance of the principle for ethical oversight of data science health research, desirability of inclusion of the principle in national guidelines, feasibility of implementation of the principle by ethics committees and confidence in ethics committees’ ability to implement the principles with 1 indicating least, and 5 indicating most important, feasible, desirable and confident response.^{34 50}

To determine the degree of consensus and strength of polarity of the responses, we will compute the mean, median, and IQR of the Likert ratings and use the recommended thresholds in the literature for consensus, high, moderate and low; and for polarity; strong, weak and none.51,53 We will develop a codebook to analyse the comments from each round using content analyses. The analyses of the Likert scales and the content analyses of comments will be used to determine “whether the group supported, opposed, or were ambivalent towards an option, whether the group was split or whether a clear picture of support did not emerge” for the statements in the draft subcode.^{52 54} Statements will be considered validated if the combined ratings and qualitative responses indicate high consensus, low to no polarity and at least weak support.⁵² We will publish statements on which consensus was not achieved and highlight these as areas for further research and discourse in the field of ethical oversight of DSHR.

The panellists would be given a summary of their ratings and responses from previous rounds for comparison with the aggregate results from the group. Statements with low consensus, high polarity or strong-to-weak opposition will be retained for re-presentation and review in the subsequent round. All descriptive statistical analyses will be conducted using STATA (College Station, Texas, USA) and we will use Microsoft Excel (Microsoft 365 V.16.97.2) for the qualitative coding of the comments.

Patient and public involvement

It was not appropriate or possible to involve patients or the public in the design or conception of our research.

Ethics and dissemination

We will share the results of our work first with the panelists who contribute to the study. Next, we will write a report and present it to the NHREC, HRECS, other research ethics regulatory agencies, research sponsors, consumer organisations, advocacy groups and the DS-I Africa Consortium. We will also engage the regulatory stakeholders on the implementation of our findings for ethical oversight of DSHR in Nigeria and Africa. We will disseminate our findings through scientific publications and presentations at scientific conferences in Nigeria, at DS-I Africa consortium meetings, and internationally.

This is the first study that aims to develop guidelines for oversight for DSHR by engaging broad categories of stakeholders using a modified Delphi approach. Hence, the study holds immense potential for arriving at consensus-driven guidelines for the governance of DSHR. We recommend this approach for rapid response to emerging ethical oversight challenges in DSHR and other novel technologies.

Ethical consideration

This study has been approved by the National Health Research Ethics Committee (NHREC number: NHREC/01/01/2007-24/12/2024D) and the University of Maryland IRB (HP-00102012).