Identifying and counteracting fraudulent responses in online recruitment for health research: a scoping review

Abstract

Objectives

This study aimed to describe how health researchers identify and counteract fraudulent responses when recruiting participants online.

Design

Scoping review.

Eligibility criteria

Peer-reviewed studies published in English; studies that report on the online recruitment of participants for health research; and studies that specifically describe methodologies or strategies to detect and address fraudulent responses during the online recruitment of research participants.

Sources of evidence

Nine databases, including Medline, Informit, AMED, CINAHL, Embase, Cochrane CENTRAL, IEEE Xplore, Scopus and Web of Science, were searched from inception to April 2024.

Charting methods

Two authors independently screened and selected each study and performed data extraction, following the Joanna Briggs Institute’s methodological guidance for scoping reviews and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews guidelines. A predefined framework guided the evaluation of fraud identification and mitigation strategies within the studies included. This framework, adapted from a participatory mapping study that identified indicators of fraudulent survey responses, allowed for systematic assessment and comparison of the effectiveness of various antifraud strategies across studies.

Results

23 studies were included. 18 studies (78%) reported encountering fraudulent responses. Among the studies reviewed, the proportion of participants excluded for fraudulent or suspicious responses ranged from as low as 3% to as high as 94%. Survey completion time was used in six studies (26%) to identify fraud, with completion times under 5 min flagged as suspicious. 12 studies (52%) focused on non-confirming responses, identifying implausible text patterns through specific questions, consistency checks and open-ended questions. Four studies examined temporal events, such as unusual survey completion times. Seven studies (30%) reported on geographical incongruity, using IP address verification and location screening. Incentives were reported in 17 studies (73%), with higher incentives often increasing fraudulent responses. Mitigation strategies included using in-built survey features like Completely Automated Public Turing test to tell Computers and Humans Apart (34%), manual verification (21%) and video checks (8%). Most studies recommended multiple detection methods to maintain data integrity.

Conclusion

There is insufficient evaluation of strategies to mitigate fraud in online health research, which hinders the ability to offer evidence-based guidance to researchers on their effectiveness. Researchers should employ a combination of strategies to counteract fraudulent responses when recruiting online to optimise data integrity.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Online recruitment for health research is increasingly popular but susceptible to fraudulent responses.

• Fraudulent survey responses can compromise the integrity of health research data and result in incorrect study conclusions.

WHAT THIS STUDY ADDS

• This scoping review revealed the various strategies used to identify and counteract fraudulent responses in online health research recruitment.

• By synthesising these data, identifying strategies and exploring potential solutions, this review seeks to inform researchers about the extent of the issue and strategies to counteract fraudulent responses, thereby enhancing the quality and credibility of online research.

• In the absence of a single effective strategy, researchers should use a combination of strategies to counteract fraudulent responses.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• The findings underscore the urgent need for both comprehensive evaluation and subsequent enhancement of antifraud strategies in online health research.

Introduction

In the digital age, the landscape of health research recruitment has been transformed by online platforms such as social media (eg, Facebook, X/Twitter, Instagram, Snapchat and TikTok).¹ These tools have revolutionised how researchers connect with potential study participants, offering unprecedented opportunities to reach diverse and widespread populations. Online recruitment methods streamline the participant enrolment process² and allow researchers to gather more representative samples^{3 4} cost-effectively.³

The expansive reach and targeted capabilities of social media platforms also facilitate access to a wide range of individuals while providing a perceived sense of anonymity.⁵ Using these platforms for recruitment, such as through paid advertisements and targeting of specific populations, enables researchers to connect with a broad audience without direct, in-person interaction. This sense of anonymity may help individuals respond more candidly.⁶ However, it can also reduce accountability, potentially leading to lower engagement with the study materials, careless responses and even fraud.⁶

While online recruitment offers numerous advantages for health research, it is susceptible to various biases. Selection bias, whereby specific characteristics such as age, education or gender are over-represented or under-represented, is a persistent threat to data integrity.⁷ Increasingly, the risk of fraudulent responses presents a significant challenge. Fraudulent responses occur when individuals or automated bots provide false information or misrepresent themselves to qualify for and participate in studies, including intentional duplicate entries and inaccurate data. These fraudulent activities can compromise the integrity and validity of the research data. If detected, researchers may need to increase sample sizes to exclude invalid responses⁸; however, if undetected, fraudulent responses can lead to incorrect study conclusions. To mitigate these risks, researchers must implement rigorous data validation techniques and employ advanced methods to identify and exclude fraudulent responses. Nonetheless, there is limited evidence guiding researchers on how to effectively identify and counteract these fraudulent responses.

This scoping review aims to describe how researchers identify fraudulent responses in health research when recruiting online and describe processes to counteract fraudulent responses.

Methods

Data sources and search

This scoping review adheres to the framework established by Arksey & O’Malley⁹ and later recommendations by Levac et al.¹⁰ Furthermore, this review follows the methodological framework set out by the Joanna Briggs Institute for scoping reviews and is conducted and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews.¹¹ The protocol was prospectively registered on the Open Science Framework (https://osf.io/fpdsb/).

Identification of the research question

The primary research question was ‘How do health researchers identify and counteract fraudulent responses when recruiting research participants online?’.

Population, concept and context

The population of interest in this scoping review includes participants recruited for health research through online platforms. The concept under study is the prevalence and characteristics of fraudulent responses, which refer to instances where individuals provide false or misleading information during the recruitment process. By focusing on these elements, the review aims to provide valuable insights that can inform future strategies to identify and counteract these issues. Counteracting involves implementing measures to diminish or eliminate the impact of fraudulent responses, thereby enhancing the integrity and reliability of online health research recruitment.¹²

Search strategy

Nine research databases were systematically searched: Medline via Ovid, Informit, AMED (Allied and Complementary Medicine Database), CINAHL (Cumulative Index of Nursing and Allied Health Literature), Embase, Cochrane Central Register of Controlled Trials (CENTRAL), IEEE Xplore (Institute of Electric and Electronic Engineers), Scopus and Web of Science. Peer-reviewed articles, not limited to any design, published in English until April 2024, were eligible for inclusion. Keywords included “recruitment”, “participant recruitment”, “surveys and questionnaires”, “online survey”, “online platforms” and “social media”. The full electronic search strategy is available in online supplemental file 1.

Inclusion and exclusion criteria

The inclusion criteria for this review were as follows: peer-reviewed studies published in English, irrespective of the study design or year of publication; studies that report on the online recruitment of participants for health research; and studies that specifically describe methodologies or strategies to detect and address fraudulent responses during the online recruitment of research participants. Health research included participants with specific health-relevant behaviours (eg, smoking) or populations with a health condition (eg, COVID-19). Given that qualitative studies necessitate participant interviews, which were presumed to inherently mitigate the risk of fraud, these studies were excluded from the analysis.

Fraudulent response in online surveys was defined as survey entries characterised by intentional deceit or invalid participation, such as submitting multiple entries to gain additional incentives, using automated systems (bots) to generate responses or providing false information to receive rewards.¹³

Study selection and data extraction

All the studies identified in the search strategy were exported into Covidence systematic review software. Duplicates were removed in Covidence.¹⁴

A group of seven investigators (AP, AB-A, AT, JA, RRa, RRo and EO’H) independently conducted the title and abstract screening to identify relevant studies. Subsequently, full texts were retrieved and assessed for eligibility. Pairs of investigators (JC, JA, RRa, AB-A or RRo) independently extracted data using a customised data extraction form developed specifically for this review. Descriptive summaries were provided for the general characteristics of the papers and the studies. Any disagreements on the extracted data were resolved through arbitration by a third investigator (EO’H).

The extracted data included information on (1) study details including the author(s), year of publication, study design (eg, randomised controlled trial, observational study), the specific topic or research question being investigated, the sample size and any demographic characteristics of the sample (eg, age, gender, ethnicity and socioeconomic status). (2) Methods used for participant recruitment/advertising: We documented approaches employed by the studies to recruit or advertise to participants. This included the platforms used (ie, social media, online forums or dedicated research recruitment websites), the type of recruitment tools (eg, advertisements, emails and targeted posts) and any specific strategies or protocols followed to attract and engage potential participants (ie, incentives). (3) Strategies to identify fraudulent responses: methods and criteria used by the studies to detect fraudulent or suspicious participant responses. We recorded whether the studies used techniques such as completion time analysis, consistency checks, IP address tracking or other indicators to flag potentially fraudulent data. (4) Strategies to mitigate fraudulent responses: any strategies implemented to prevent or minimise the occurrence of fraudulent responses.

We obtained and verified data from investigators by contacting the corresponding authors of the included studies via email when required. If the authors did not respond, we sent a reminder email after 2 weeks to follow up on the request.

Data analysis

The sample was descriptively analysed to provide a comprehensive overview. The number of fraudulent and valid responses relative to the total responses recorded by the authors of each study was reported when available. Identifying and counteracting fraud are often intertwined, with the identity of fraud immediately leading to the exclusion of that participant and therefore counteracting fraud. In this review, strategies used to identify fraud were described using a predefined framework.¹² The framework was developed by researchers in response to their experience encountering fraud in two web participatory mapping experiments conducted in 2021. The study involved an online participatory mapping platform where participants were asked to map landscape values and perceived risks, alongside completing a traditional survey questionnaire. The authors compared results and from these developed five indicators to detect fraudulent survey responses, including completion time, non-conforming responses, temporal events, unexpected language and spatial incongruity. Using this framework facilitated the systematic assessment and comparison of the effectiveness of different strategies across studies.

Results

The search identified 2289 records. After duplicates were removed, 1826 records underwent title and abstract screening. Then, full-text article screening was performed on 34 articles retrieved. Finally, a total of 23 articles were included. The process of selecting studies is outlined in figure 1.

Figure 1. Flow diagram of the study selection process.

*Not online recruitment for health research and not related topics. **The list of excluded full-text articles assessed for eligibility can be found in online supplemental file 2.

Of the 23 studies, 18 were quantitative and 5 were qualitative in design. Most studies were published within the last 5 years (22/23, 96%) and from the USA (19/23, 83%). When reported, most used the online survey platform Qualtrics (9/16, 56%) and recruited participants from social media (ie, Facebook, X/Twitter) (11/19, 57%).

The included articles most commonly investigated specific health behaviours or populations, including opioid users,¹⁵ alcohol, tobacco, and other drugs,16,19 cancer survivors,²⁰ COVID-19 patients,20,23 and people living with HIV.24,26

All included studies used multiple methods for detecting fraud. 18 studies (78%) provided data on the number of participants excluded due to fraudulent or suspicious online survey participation relative to the total sample recruited. A summary of the characteristics of the included studies can be found in table 1. More details can be found in online supplemental file 3.

Table 1. Summary of characteristics of the included studies.

Author, year	Incentives offered for study completion	Total sample screened	Participants excluded for fraudulent or suspicious responses	Total valid entries	Types of fraudulent or suspicious data
1. Ballard et al24 2019	Yes	414	161 (38%)	253	Ineligible geolocation; invalid phone number; mismatching names; unusual email addresses.
2. Bonett22, 2024	Yes	7950	4207 (52%)	3743	Ineligible geolocation; invalid residential address; repeated email address; invalid recruitment URL.
3. Campbell et al25 2022	Yes	739	413 (55%)	326	Geolocation data; repeated responses suggesting bots.
4. Chandler et al34 2020	NR	NR	NR	NR	Poor-quality responses, where participants may not pay sufficient attention to survey questions.
5. Fernandez-Lynch,35 2019	Yes	2275	500 (21%)	1775	Participants misreported their eligibility based on varying payment incentives and eligibility criteria for participation.
6. Glazer et al31 2021	Yes	1766	125 (7%)	1641	Potential or confirmed discrepancy in provided information on interest form.
7. Guest et al26 2021	Yes	13 931	493 (3%)	3253	Ineligible geolocation; incompletion of open-ended questions.
8. Godinho et al33 2020	Yes	860	276 (32%)	506	Incomplete addresses; attempted to enrol into the study more than once; fake addresses identified were created by one individual or group.
9. Habib and Jha16 2021	Yes	16	NR	NR	Multiple entries.
10. Hammond,17 2022	Yes	1503	324 (21%)	1179	Respondent replaced their face recording with a stock photo or video from the internet; used a private VPN to change the IP address.
11. Hohn et al32 2022	NR	45	0	45	Strategies such as multiple questions, several surveys were entered by respondents but not completed, survey time stamps and locations were employed.
12. Kramer et al27 2014	Yes	600	293 (48%)	307	Illogical responses to related paired items; shorter than expected time to complete the survey and inconsistently reported personal information such as names.
13. Levi et al28 2022	Yes	482	182 (37%)	300	Geolocation did not match the self-reported address; geolocation was located outside of the recruiting counties; and the name could not be verified in our partner databases; same IP address.
14. Loebenberg et al18 2023*	Yes	5562	1161 (20%)	5602	Invalid contact numbers where participants were not known; postcode did not match the first line of the address; the same landline number provided by multiple respondents with different geographical postal addresses.
15. Pozzar et al29 2020	Yes	271	256 (94%)	15	Exhibited evidence of bot automation; provided a duplicate or unusual response to one or more open-ended items and exhibited evidence of inattention.
16. Pratt-Chapman et al20 2021	Yes	1977	1081 (54%)	569	Open-ended suggestion responses are duplicates; Respondents indicate a survey source prior to dissemination of the survey from that source; email/address was suspicious or duplicate.
17. Singh and Sagar23 2021	NA	NA	NA	NA	Inconsistent (eg, age and date of birth responses not consistent) or implausible response patterns (eg, average daily smartphone use of greater than 24 hours, ‘all or none’ response pattern).
18. Stuart and Moore30 2021	NR	704	168 (23%)	536	Inconsistency in the open-ended questions responses; incomplete entries.
19. Wang et al21 2023†	Yes	950	NR	NR	Survey time stamps and locations; unusual email format; response to optional open-ended question text identical to another respondent; Email/address/telephone number already reported by another respondent.
20. Wilkerson et al19 2015	Yes	543	107 (19%)	427	Duplicates among the IP addresses-mail addresses, and key demographic characteristics.
21. Willis et al38 2023	Yes	1241	268 (21%)	NR	Duplicates emails; short time completion (>5 min).
22. Wood and Bindler36 2023	Yes	109	43 (39%)	NR	Inconsistencies in the responses provided.
23. Young et al15 2020	Yes	410	67 (16%)	234	Ineligible geolocation; incompletion of open-ended questions.

^*863 study entries were identified as bots during data screening in the first 2 months of recruitment. Out of the 5956 participants (excluding bots) who enrolled, 298 (5%) were found to be false participants.

^†After Facebook advertisements for an hour and reaching approximately 125 individuals, 2578 screeners were attempted and 950 survey responses were completed, however, it is not clear how many indicated possible fraud. Hammond, 2022: Total sample 276 from the community and 1227 from the panel; valid entries:114 from the community and 1065 from the panel.

IP, internet protocol; NA, not applicable; NR, not reported; URL, uniform resource locator; VPN, virtual private network.

Identifying fraudulent responses

A summary of the fraud detection strategies employed and their evaluation is presented in table 2.

Table 2. Overview of fraud detection and mitigation methods and effectiveness across studies.

Strategy	Specific features	Evidence for effect
Detecting fraud
Survey completion Time	Short completion time identifies suspicious responses (eg, completion time less than 5 min)	Not evaluated
Non-confirming responses	Incorrect response to an open-ended question; including specific directives (eg, in 5 words describe what this survey is about) identifies suspicious responses Inconsistent responses to pairs of questions (when the same question is asked twice) identify suspicious responses Incorrect response to a question that requires specific knowledge identifies suspicious responses Inconsistent responses when compared with publicly available information identify suspicious responses Anonymous email format (eg, 12458@hotmail.com) identifies suspicious responses	These measures may reduce automated attacks and reduce time spent manually checking responses but do not prevent invalid responses27
Temporal events (odds night hours responses)	Survey time response outside of daytime hours when compared with current time zone identifies suspicious responses	Not evaluated
Geographical incongruity	Geographical verification: the address provided by the participant does not exist in the real-world, does not match the expected local neighbourhood’s name or the ZIP code is outside the target location identifies suspicious responses The IP address is outside the expected country, or the participants are using a VPN identifies suspicious responses	Not evaluated
Counteracting fraud
Incentives	Decrease incentive value or deliver incentive to physical addresses only. Do not advertise the amount or type of compensation. Do not provide incentives or clearly state that fraudulent responses will not be compensated.	Higher incentives led to participants providing fraudulent responses.18 35
In-built survey features	Activate the ‘prevent ballot box stuffing’ option in survey platforms like Qualtrics to ensure that participants cannot take the survey more than once. This feature works by placing a cookie on the participant’s browser when they submit a response. Search engine blocking: To prevent unauthorised access, features that block search engines from indexing the survey can be activated, ensuring the survey does not appear in search results	Not evaluated
CAPTCHA feature	Add CAPTCHA field (eg, text-based, 3D CAPTCHA, image) can block automated bots, it cannot stop fraudulent responses from actual people who may be incentivised to manipulate the survey and provide false information.	CAPTCHA, attention checks, can minimise bots, however, alone is unlikely to prevent fraudulent responses in internet-based research promoted on social media18 20
Manual verification	Perform regular checks of survey responses including data audits, IP addresses and time stamps Apply dual-method contact, for example, phone and email for respondent verification. Review responses to optional open-ended question text to ensure that it is not identical to that of another respondent Include video checks (eg, video conferencing with the participant) or add a public service announcements and computer webcam to record their facial expressions to verify identify.	Recommended but not evaluated17 36

CAPTCHA, Completely Automated Public Turing test to tell Computers and Humans Apart; VPN, virtual private network.

Detecting fraudulent responses

Survey completion time

This indicator was defined as inadequate survey completion time to complete the survey accurately and can be estimated by pilot testing.¹² This method was reported in six studies (6/23, 26%) included in this scoping review.2224 26,28 In their qualitative review, Singh and Sagar²³ also reported response time as a method of identifying fraud from online surveys. One study²² reported that the same start and stop as an indicator of fraud, and one²⁴ reported using a response time of less than 5 min as part of a 10-point algorithm to detect fraud anticipated to require a maximum of 45 min to complete. Two studies (8%)^{27 28} reported using the embedded date/time stamps to identify quick completions but did not state a specific time frame that would indicate fraud.

Non-confirming responses

This indicator was defined as text responses with unusual and repeated combinations that were perceived as implausible (eg, reporting an average daily smartphone use of more than 24 hours or exhibiting an ‘all or none’ response pattern). 12 studies (12/23, 52%) included in this scoping review reported on methods to identify and remove non-confirming responses.

Two studies^{27 29} reported asking respondents specific open-ended questions, such as where they heard about the study or included directives (eg, select the third option below).²⁹ Other studies included pairs of questions²⁹ or reported collecting the same information at multiple points to detect inconsistencies in responses that might indicate fraudulent or careless participation.^{19 22 27}

Four studies attempted to identify fraudulent responses by including questions that required insider knowledge.^{27 29} Stuart and Moore³⁰ asked participants to self-disclose inclusion criteria. However, Levi et al²⁸ reported that false respondents were able to ascertain highly specific study eligibility requirements. Another strategy reported was to include an open-ended question to identify the possible use of bot automation,^{29 30} where responses were reviewed for English language misuse, incoherent or nonsensical phrases, duplicate responses, and content copied verbatim from existing websites.

Glazer et al³¹ reported using TransUnion’s TLOxp (www.tlo.com) to verify publicly available information, including names, address history, date of birth, social security number, phone and email information, and possible relatives. These verification strategies were also highlighted in the qualitative results. For instance, Singh and Sagar²³ recommended checking for inconsistencies and implausible response patterns. Additionally, Pratt-Chapman et al²⁰ emphasised the value of specific questions (eg, where a respondent heard about the survey), open-ended questions, checking question pairs and directive questions.

Temporal events

This indicator was defined as survey time stamps outside the target population’s expected norms. Four studies included in this scoping review reported on this indicator. Loebenberg et al,¹⁸ based in the UK, reported that bots were identifiable because they usually joined the study at odd night hours. Wang et al,²¹ from the USA, used checking the time zone as a strategy to identify fraudulent responses. Similarly, Pozzar et al²⁹ included survey time stamp checks as part of various strategies used to identify fraudulent responses. Pratt-Chapman et al²⁰ also included this in their qualitative report on identifying fraudulent responses.

Geographical incongruity

This indicator was defined as geographic stamps that sit outside the expected norms for the target population. Seven studies (30%) reported on this indicator. Three verified internet protocol (IP) addresses,^{24 27 29} whereas one included location screening.³² Bonett et al²² included four components to check for geographical incongruity, including (1) checking that the address matched the colloquial name of a neighbourhood, (2) verifying the ZIP code was inside the target location, (3) verifying that the IP address was not from outside USA, or that (4) identifying whether participants were using a virtual private network (VPN).²² In addition, they did not permit more than three people to use the same residential address.²² One study²⁹ reported using a virtual private server (VPS) with the Routing Information Protocol or Shiny tool from the R open-source programming language. This approach involves analysing the IP addresses of survey respondents and seeing if there are signs that they might be using a VPS.

Leobenberg et al¹⁸ checked for geographical incongruity by checking postcodes against the first line of the street address. Levi et al²⁸ reported that they geo-located each participant using survey meta-data and cross-checked with partner databases. In addition, qualitative reports by Singh and Sagar,²³ Godinho et al³³ and Chandler et al³⁴ included IP checks as a recommendation for detecting fraud.

Counteracting fraudulent responses

Incentives

17 (73%) of the included studies reported offering incentives to participants. The incentives, in the form of gift cards, ranged in value from US$5 to US$200. In one study, researchers varied the rate of financial incentives to determine their influence on the rate of deception. They observed that increasing the incentive value heightened the likelihood of fraudulent responses.³⁵ In another study, the researchers deliberately did not advertise the amount or type of compensation,²⁷ while another study purposefully offered no incentive.³⁰ Another strategy to detect fraud was to lower the value of the remuneration and only send it to a physical address.²⁹ Habib and Jha¹⁶ reported using the anonymous incentive method to randomly select participants for a reward. In their qualitative review, Pratt-Chapman et al²⁰ recommended that researchers explicitly state that fraudulent responses will not be compensated.

In-built survey features

Eight studies (34%) reported using in-built survey options to prevent fraud. Ballard et al²⁴ reported that they ensured that the ‘prevent ballot box stuffing’ option in Qualtrics was activated. This option prevents participants from taking a survey more than once by placing a cookie on their browser when they submit a response. Other features are activated in Qualtrics to prevent indexing, which blocks search engines from finding the survey and presenting it in search results.²⁴

Eight studies (34%)20,2225 27 29 32 36 reported using a CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) feature. The Q_RecaptchaScore is a specific type of CAPTCHA developed by Google. Both serve as an indicator in reports to determine the likelihood that a response was generated by a bot rather than a human.³⁷ Bonett et al²² set cut-off levels for fraud, including (1) Q_RecaptchaScore >0.5 (score ranging from 0 to 1, greater than or equal to 0.5 means the respondent is likely a human) and (2) Q_RelevantIDFraudScore >30 (score ranging from 0 to 100, with higher values indicating a greater likelihood of fraud). Others reported using specific survey features to identify fraudulent responses, such as speed bump questions (ie, difficult answer questions designed to screen out bots and automatic responders)²¹ or honey pot questions (ie, hidden questions only detectable by bots).20,22

In addition, two studies (8%)^{22 27} checked the uniform resource locator (URL) to ensure that it matched the URL distributed by the study team. In contrast, Campbell et al²⁵ reported that failure to verify via a unique web link was an indication of fraud.

Manual verification

Manual verification refers to when a researcher contacts a participant to verify their identity or manually checks survey responses for inconsistencies. Manual verification in the included studies was considered an additional step in the recruitment process and data-quality check despite being more laborious compared with automated strategies.^{33 34}

Five studies (21%)25,2729 32 reported manual verification to identify fraud, and three studies (13%)^{23 33 34} included manual checking in their literature review as a recommended strategy to detect fraud. Campbell et al²⁵ described a manual review of screening results to identify potential fraudulent responses, with suspected cases followed up by phone. Guest et al²⁶ reported manual checks of age, IP address, completion scores and time stamps for data inconsistencies. Hohn et al³² advocated for a data integrity plan outlining manual checks to prevent fraudulent activity. Kramer et al²⁷ suggested dual-method contact (phone and email) for respondent verification. Pozzar et al²⁹ described quality assurance checks by the research team for completed surveys. One study³⁶ employed videoconferencing (eg, Zoom) scheduled at the participant’s convenience to verify their identity.

Automated facial response analysis

One study¹⁷ embedded public service announcements of 30–40 s within a survey and a computer webcam recorded their facial expressions. The records were analysed using automated facial coding engines.

Response when fraud was suspected

For most studies (78%), the evaluation of the fraudulent responses was intertwined with detection. If fraud was detected, then the participant’s responses were removed. Some authors recommended utilising a fraud detection algorithm^{16 20 24 27 29} and advising participants that fraudulent responses would be monitored. Others conducted monthly data checks,^{17 18 26} and one asked participants to use their professional email to verify participant identity.³⁸

Discussion

This scoping review identified 23 studies focusing on strategies to detect and counteract fraudulent responses in online health research recruitment. While no single method prevented invalid or fraudulent responses combining multiple strategies can enhance data integrity. Most of these studies (83%) were conducted in the USA, published within the last 5 years (2019–2024) and primarily used Qualtrics as an online survey platform and a mix of recruitment channels (eg, Facebook, Instagram and web-based platforms). This review uncovered various methods for detecting fraud in online recruitment, but their evaluation was inconsistent, failing to outline how researchers could counteract the risk.

Although online recruitment offers many opportunities for health research,^{2 39} it also carries the risk that research participants may exaggerate or distort their experience. Without initial direct contact, detecting dishonest responses becomes more challenging for researchers.⁴⁰ A multilayered approach is likely necessary given the variety and evolving nature of fraudulent activities, ranging from simple to complex schemes involving sophisticated techniques. Relying on a single method may fail to detect certain types of fraud. Consequently, included studies have reported a combination of approaches to counteract fraudulent responses in the context of increasing online recruitment demand.^{12 17 19 21 25 35}

In this scoping review, manual verification was used by 34% of included studies, and monthly data checks, along with the use of professional emails, are recommended.⁴¹ However, these strategies have undergone limited evaluation, leading to ambiguity in their effectiveness. For example, about temporal events, there is no consensus on what constitutes ‘late’ survey entries. Furthermore, higher monetary compensation appears to correlate with an increased risk of fraudulent responses.^{20 32} While fraud prevention measures like telephone calls, repeated questions or reduced incentives may mitigate this risk, they can also increase participant burden, reduce accessibility and potentially compromise the generalisability of the sample.^{13 42}

There are several automated features to detect fraud. In-built survey features were common, with 34% of studies using options like the cookies option and reCAPTCHA in Qualtrics to prevent fraud.20,2225 27 29 32 36 Specific tactics included speed bump questions, honey pot questions and URL verification to ensure survey integrity. CAPTCHA schemes (eg, text-based, image-based, audio-based, video-based or cognitive-based CAPTCHAs) offer several advantages: they are easy to implement, consistently applied across studies and often more sophisticated than other manual strategies.⁴³ However, they can still be vulnerable to automated and human-assisted fraud.⁴⁴ While CAPTCHA schemes offer certain benefits, their overall effectiveness remains poorly documented, and the lack of transparency in their evaluation highlights a critical need for continuous assessment and refinement. Currently, it is challenging to provide evidence-based recommendations to researchers regarding their effectiveness.

Fraud mitigation strategies have also been explored in non-health research, and the approaches to preventing fraudulent responses differ significantly between health and non-health surveys, reflecting the unique challenges each domain faces.⁴⁵ Some researchers consider transitioning to prevetted participants through third-party companies as a solution to these issues.⁴⁶ However, while prevetting can reduce the likelihood of fraudulent responses, it introduces new challenges, such as higher costs and potential biases that may limit the generalisability of findings. The use of pre-vetted participants can skew the sample towards individuals recruited multiple times for different studies, thus possibly affecting the diversity and representativeness of the research population.⁴⁷

Health surveys are known to require more rigorous methods due to the sensitive nature of data, ethical considerations and the vulnerability of participants, often compounded by incentives like monetary compensation or potential medical treatment. Despite this need for rigour, challenges such as ethical concerns about privacy, reduced participation due to intrusive methods and balancing data integrity with participant burden persist.^{28 46} Non-health surveys, on the other hand, could typically employ simpler fraud detection methods like CAPTCHA or IP address verification, which are less intrusive but also less robust. This contrast underscores the importance of context-specific approaches, where health surveys must carefully balance rigorous fraud detection with maintaining participant trust and engagement.

Our results mirror those of Ziyi et al⁴⁸ who systematically evaluated the effectiveness of tests in preventing and detecting fraudulent responses across two different online surveys. Those authors tested 28 strategies, including reCAPTCHA, VPN detection, checking for consistency (by asking participants for their occupation twice with the question written differently both times) and checking IP address. Their results show that inconsistent knowledge about the subject was the most useful indicator of fraud, although reported that tests related to consistency and attention were also useful for detecting fraud. They concluded that there is no single perfect strategy to prevent and detect invalid responses in online surveys.⁴⁸

Lawlor et al⁴⁹ proposed the REAL framework (Reflect, Expect, Analyse, Label) to guide fraud prevention in online surveys. While studies included in this review generally adopted analysis strategies, the framework’s emphasis on reflection, expectation setting, and labelling of fraudulent responses was less evident in the included studies. Lawlor et al⁴⁹ also highlight the potential for fraud to occur postrecruitment, which was not addressed by the studies included in this review. This broader perspective, which advocates for reflection, expectation setting, and labelling as critical components of fraud prevention, remains underused. The limited adoption of these practices suggests that further refinement and application of such comprehensive strategies are necessary to more effectively address the persistent challenge of survey fraud.

Strengths and limitations of the study

This scoping review has several strengths. Among its strengths, addressing the strategies to identify and counteract fraudulent responses collected through social media is timely and relevant, given the increasing use of these platforms in health research. This study focused on quantitative studies in health research and compiled strategies including studies from comprehensive databases. Additionally, the review highlights the critical need for more rigorous evaluation and development of fraud detection strategies in the context of social media recruitment, drawing attention to an underexplored area in the existing literature. By addressing this gap, the review emphasises the importance of providing clear guidance for researchers to improve data integrity, ensuring that online studies can yield reliable and valid results.

However, the study also has some limitations. Initially, our inclusion criteria were limited to English-language studies. However, given the increasing accessibility of language translation tools and the potential for valuable insights from international research, we acknowledge this as a limitation. While no non-English studies were identified in the current review, future research may benefit from a more inclusive approach. Additionally, we did not report on other serious biases such as selection bias, as it was not explicitly addressed in the included studies.

Implications for future research and policy

In the absence of evidence for one effective strategy, researchers should implement multiple fraud detection strategies, conduct pilot tests to establish benchmarks, and continuously monitor and update their recruitment methods. Collaboration between researchers, platform providers and policy-makers is necessary to develop robust fraud detection and prevention frameworks, guidelines and best practices. Future research should explore under-researched indicators, such as unexpected language settings and conduct comparative studies to evaluate the effectiveness of fraud detection methods. Additionally, emerging technologies like machine learning and artificial intelligence could enhance fraud detection capabilities, leading to more effective online health research recruitment solutions.

Conclusion

Effective strategies for detecting online research fraud remain elusive. This review outlines commonly employed methods and highlights the need for rigorous evaluation. Given the absence of a gold standard, researchers should implement a multifaceted approach to safeguard data integrity.

Data availability statement

No data are available.