Recognizing and Preventing Participant Deception in Online Nicotine and Tobacco Research Studies: Suggested Tactics and a Call to Action

Introduction

Nicotine and tobacco researchers have been increasingly using online methods to conduct research, but this became a practical necessity following the COVID-19 pandemic and the coinciding restrictions on in-person data collection. Online research has many potential advantages to participants and scientists, including greater convenience of and access to research participation and greater diversity and geographic distribution of study samples. Centralized recruitment, data collection, and intervention may also translate into increased staffing efficiencies, lower research costs, and can allow information to be collected in real-time, under real-world conditions, thereby increasing validity and generalizability. At the same time, online studies introduce new challenges, particularly regarding deception among potential participants. This includes misrepresenting critical information about eligibility criteria or study outcomes. Individually and collectively, these practices can significantly undermine research integrity.

Participant deception occurs frequently in research, whether conducted online or in person. Across all research formats, it is estimated that anywhere from a small minority to a vast majority (~75%) of participants engage in some form of deception.^1,2 The potential for deception is significantly greater in online studies that have little to no face-to-face interaction, with deception likelihood increased when monetary incentives are provided.^1,3

Several recent reviews^4,5 have addressed this issue, but none to date have focused explicitly on deception seen in nicotine and tobacco research. Thus, the aims of this commentary are to: (1) raise awareness of this important topic by sharing examples of participant deception the authors have experienced in our collective online nicotine and tobacco studies, (2) offer recommendations for addressing these issues, and (3) call for action to prevent and mitigate deception-related threats to the integrity of the work conducted in our field through planning, transparency, and establishment of best-practice anti-deception methods.

Examples of Deception in Online Nicotine and Tobacco Research

Deception in online research can take several forms. Some deception is indicative of an attempt to amass monetary incentives, often in large quantities. An example is the use of a bot—ie, software that impersonates human users—to systematically identify online paid research opportunities, complete surveys to “learn” eligibility criteria and then enroll and complete large numbers of online surveys using fraudulent information. Such deception does not require advanced programming skills, as survey-filling software is readily available for purchase online (eg, ultimatesurveybot.com, advertised with the tag line, “Making $$$ has never been this easy!”).⁶ Other forms of deception occur for nonfinancial reasons. For example, individuals who are ineligible for a clinical trial may provide false personal information to gain access to a no-cost treatment, or eligible participants may misreport smoking status outcomes to be “helpful” to the researchers or out of shame of not quitting smoking. The following are a few potential indicators of participant deception we have noted in our studies:

1. During screening: Web-based eligibility screens and surveys were received in clusters with minimal response differences, were submitted within minutes of one another, from the same IP address, and had similarly formatted email addresses (eg, firstname@gmail.com).

2. During survey data collection: Online surveys were accessed and completed prior to distribution of the survey link. Completed surveys were also linked to IP addresses clearly outside of the recruitment area.

3. When providing post-study compensation: Names provided to receive electronic study incentives (eg, e-gift cards) differed from names participants provided at study entry.

Developing and Implementing an Anti-Deception Protocol

Drawing from previously published guidance for addressing deception in online research,^4,6 as well as anti-deception protocols employed by the authors of this commentary, we provide a sampling of practical suggestions for prevention and detection of deception in Supplementary Table 1. There is limited empirical research evaluating the effectiveness of these methods, and level of effectiveness can vary across time and context. Given that any single method will have largely unknown effectiveness within a specific context and is unlikely to prevent all forms of deception, the authors, as well as other researchers,^4,6 typically use several strategies as part of their anti-deception protocols. Although these protocols are often not published with the study findings, some reports in the nicotine and tobacco research area⁷ as well as in other fields, provide detailed descriptions of how researchers implemented multiple deception prevention and mitigation strategies, including use of algorithms⁸ or evidence-grading⁹ to evaluate the probability of deception using multiple indicators.

The study context—aims, target population, budget, recruitment methods, intervention logistics, participant incentives, etc.—informs the risk for deception and what mitigation strategies are feasible. Investigators encounter several tensions when developing an anti-deception protocol. Among them are balances between cost and data validity, scrutiny and generalizability, and individual privacy and anti-deception effectiveness. Many anti-deception techniques require additional programming for automated checks or staff time for manual scrutiny, which reduces some of the efficiency of online recruitment. Additionally, some techniques may inadvertently screen out legitimate candidates, making it more difficult to reach recruitment goals. More intrusive measures, such as requiring photo IDs, may discourage or prevent some individuals from participating (eg, due to privacy concerns, anxiety over potential identify theft, or lack of photo ID), and may differentially impact marginalized groups.¹⁰ Techniques such as using cookies, looking up IP addresses, researching contact information, and creating or searching databases of study participants who have been flagged for potential deception in past studies raise privacy concerns. Informing potential participants that such measures will be used could deter fraud, but may discourage legitimate participants from volunteering and provide bad actors with insight on how to foil anti-deception methods. Investigators and IRBs must determine where to set the fulcrum between transparency and effectiveness.

In addition to describing the techniques used to identify potentially deceptive behavior, anti-deception protocols should specify the study-specific evidential thresholds for actions such as removing a candidate from enrollment consideration or removing a participant’s data from the study (eg, see Refs. ^8,9]. Some techniques result in false positives (ie, labeling a behavior as deceptive when it is not), with different techniques having different false positive rates. Using the same email address as another participant in the study might be conclusive evidence of an attempt to enroll in a study more than once, while use of the same IP address as another participant or use of a virtual private network (VPN) would be less conclusive. Generally, evidence from multiple techniques should be used before taking action.

The protocol should also describe how deception identification is communicated to participants, if at all. During recruitment it may make sense to simply inform the participant that they are ineligible without revealing the specific reason. Removing an enrolled participant may require a different approach if there are additional study activities planned. Some studies have included language in the consent that participants determined to have engaged in fraud will be withdrawn from the study.¹¹ Others have contacted individuals with suspected deception to offer an opportunity to verify data.¹²

Given its often central role in the determination of tobacco use status in nicotine and tobacco research, the issue of biochemical verification to detect deception in self-report warrants specific comment here. While verification may enhance accuracy of self-report, many commonly employed methods of implementing remote biochemical verification in online studies (eg, unobserved, mailed cotinine tests returned for laboratory analysis, without verification of the identity of the person providing the sample) are incomplete solutions that have unknown effectiveness at deterring or detecting deception. However, more rigorous methods such as video-recorded sampling¹¹ or verification of participant identity via photo ID¹³ can produce less-than-optimal adherence to the protocol (~40%–60%), as noted in a recent review.¹⁴ Because of this tension between rigor and adherence, as well as other tensions noted above, we recommend decisions about biochemical verification in online studies be guided by the balance between the relative risk to study integrity of misreporting and equally important considerations such as feasibility, adherence, burden, privacy concerns, and potential for inequitable impact on marginalized groups. This recommendation is generally consistent with the recent SRNT expert consensus recommendations, which highlight the importance of considering context (eg, study population) and feasibility in deciding whether and how to include biochemical verification in nicotine and tobacco trials.¹⁵

A Call to Action: Maximizing Transparency and Enhancing Capacity to Prevent and Mitigate Deception in Online Nicotine and Tobacco Research

Without broad awareness and efforts to prevent and mitigate deception in online nicotine and tobacco research studies, our science remains susceptible to both small- and large-scale deception that may result in inaccurate findings. To increase transparency and to ensure planning for and implementation of deception prevention and mitigation strategies in online studies, we strongly recommend inclusion of anti-deception protocols in grant applications, IRB submissions, study registries, protocol papers, and published supplemental materials with main outcome papers. While this is not standard practice in nicotine and tobacco research, it is consistent with recommendations in other fields.^4,6,9 By publishing on anti-deception methods employed and their effectiveness, we can collectively strengthen the rigor and integrity of our science and begin to elucidate best practices for the field.

Supplementary Material

A Contributorship Form detailing each author’s specific involvement with this content, as well as any supplementary data, are available online at https://academic.oup.com/ntr.

Funding

Dr Heffner’s work on the manuscript was supported by grants from the National Cancer Institute (NCI) and National Institute on Drug Abuse (NIDA) (R34DA050967, R21CA236980). Dr Watson’s work on the manuscript was supported by a grant from the National Institute on Drug Abuse (K23DA048181). Dr Dahne’s work was supported by grants from NCI and NIDA (K23 DA045766, R21 CA241842). Dr Kelly’s work is supported by the Department of Veterans Affairs VISN 1 MIRECC. Dr McClure’s work was supported by a grant from the National Cancer Institute (R21CA234003). Dr Thrul’s work on the manuscript was supported by a grant from the National Cancer Institute (R01CA246590). Funding sources had no role in the writing of this report or the decision to submit this manuscript for publication.

Declaration of Interests

Dr Heffner has received research support from Pfizer. Dr Thrul reports membership on the scientific advisory board of MindCotine, Inc., which offers a smoking cessation program. This arrangement has been reviewed and approved by the Johns Hopkins University in accordance with its conflicts of interest policies. Matthew Bars is a founder and CEO of IntelliQuit, a cloud-based nicotine biomonitoring smartphone platform. Mr Meier’s company develops web-based study management systems that employ anti-deception methods. None of the other authors have financial conflicts to disclose.