Abstract
Background
The internet is increasingly being used to conduct randomized controlled trials (RCTs). Knowledge of the types of interventions evaluated and the methodological quality of these trials could inform decisions about whether to conduct future trials using conventional methods, fully online or a mixture of the two.
Objective
To identify and describe the scope of internet-based RCTs for human health condition interventions and evaluate their methodological quality.
Methods
A systematic review of RCTs of any health intervention conducted fully or primarily on the internet was carried out.
Results
23 fully and 27 primarily internet-based RCTs were identified. The first was conducted in 2000. The majority of trials evaluated interventions that involved providing health information to participants, but a few evaluated self-administered interventions (eg, valerian, stretching). Methodological quality was variable and the methods were generally poorly reported. The risk of bias was low in only a small number of trials; most had substantial methodological shortcomings. Only one trial was identified as meeting all criteria for adequate methodological quality. A particular problem was high rates of loss to follow-up (fully online: mean 47%; primarily online: mean 36%).
Conclusions
It is theoretically possible but perhaps difficult to test the effectiveness of health interventions rigorously with RCTs conducted fully or primarily over the internet. The use of the internet to conduct trials is more suited to pragmatic rather than explanatory trials. The main limitation of these trials is that they typically experience high rates of loss to follow-up. Methodological standards now accepted for traditional RCTs needs to be evident for online RCTs as well, especially in reporting of their methods.
Keywords: systematic review, Internet, randomized controlled trial, methods
Introduction
Randomized controlled trials (RCTs) have become the gold standard for evaluating the effectiveness of health interventions.1 2 Hundreds of thousands of trials have been conducted in the last few decades.3
In conventional RCTs, participants are recruited in person, often from clinical facilities. Once participants have consented to participate in the trial, they are allocated to one of the trial arms, often by telephoning a central facility or with the use of sealed envelopes. Outcome data are collected by an investigator, usually with paper-based case record forms. These data are later entered into a computer database.
With the rise in availability and reliability of the internet, some or all of these steps can now be conducted over the internet. In particular, it is possible to recruit participants, allocate interventions, measure outcomes and enter data using the internet. Several trials have combined internet-based trial methods with traditional trial methods.4 5 Others have abandoned traditional trial methods altogether and conducted trials fully online.6–8 In a wholly internet-based randomized trial, the investigators never meet participants—neither the application of the intervention nor the assessment of the outcomes requires face-to-face interaction between the subjects and the investigators. There are potential advantages and disadvantages of conducting part or all of a randomized trial over the internet. The internet potentially provides researchers with the ability to reach potential participants who would otherwise be unreachable, to access non-clinical populations and to reduce research costs. As noted by Murray et al,9 the advantages of conducting an online trial include the potential to improve the trial's external validity when an online intervention is being evaluated, and users have the ability to self-refer without having to participate via a health professional; the randomization process has the potential to be more secure, with allocation being concealed. Despite these advantages, Murray et al note that online recruitment could result in unrepresentative samples and multiple registrations and increases the difficulty of verifying identity. There is also the potential loss of power in the study if participants assigned to the intervention do not receive or fail to complete the intervention, resulting in the true effect of the intervention being underestimated. Loss to follow-up is of particular concern in online trials, with Murray et al noting 3 month follow-up rates of 15% and 11% in two online trials. Data quality is another potential area of concern, as many of the standard patient-completed outcome measures have been designed for paper-based completion and a change in the mode of delivery may change the psychometric properties. Other problems include the potential for deception. Participants could be deceived into participating in research that is not being conducted by legitimate research institutes. Likewise, researchers could be deceived by participants who are not who they claim to be, or who provide dishonest or misleading data.
To our knowledge, there have not been any reviews of published reports of internet-based randomized trials. It is important to know how many internet-based trials have been conducted, the sorts of interventions they investigate, and the methods used in these trials and whether these trials are particularly prone to certain sources of bias. This could inform decisions about whether to conduct future trials using conventional methods, fully online or a mixture of the two.
We conducted a systematic review of randomized trials that were conducted fully or primarily online. The aims were to identify and describe the scope of these trials and evaluate their methodological quality.
Methods
Selection criteria
RCTs testing an intervention for people with or at risk of a health condition were eligible for inclusion if they were conducted either fully or primarily online.
Online status was determined by evaluating whether the trial had the following characteristics:
The study was open to the general public (anybody with access to the internet had the potential to participate in the study, provided he or she met the specific inclusion/exclusion criteria).
Some element of online recruitment was utilized (eg, banner advertisements, internet search engines).
Information about the study was available to potential participants online or electronically (on the study webpage or via email from researchers) and not exclusively via a health professional or face-to-face/telephone contact with researchers.
Informed consent was obtained online (eg, by clicking ‘I consent’ on the webpage).
Registration to participate and screening to determine eligibility was carried out online.
Randomization of participants was completed automatically by the web server.
Baseline data were collected online.
Outcome data were collected online.
Attempts to contact non-responders, if any, were made online or electronically.
Trials were classified as fully online if all criteria were met. They were classified as primarily online if all criteria except for any or all of Criteria 4, 6 or 9 were met. All other trials were excluded from the review but were classified as either partially online (at least five criteria were met but the study was not considered to be fully online or primarily online), some online elements (between one and four of the criteria were met) or not online (none of the criteria were met).
Search strategy
We searched the Cochrane Central Register of Controlled Trials (3rd Quarter 2011) to September 28 2011. We used the search strategy defined in the Cochrane Handbook of Systematic Reviews of Interventions, which was developed to identify all RCTs (sensitivity and precision maximizing version 2008 revision),10 and combined this with a second search produced by exploding the MeSH term ‘Internet’ and multi-purpose terms ‘online’, ‘web$’ or ‘www’. (See online supplementary appendix 1 for full search strategy). One reviewer (EM) screened all titles and abstracts identified by the search strategy. Papers were excluded based on title/abstract if it was apparent that any of the inclusion criteria were not met or if there was no indication that the study was conducted online. Full papers of the remaining records were screened (EM). The reference lists of these papers were examined for additional studies.
Data extraction
One reviewer (EM) extracted all data as detailed in online supplementary appendix 2. Other investigators were consulted if there were any areas of concern regarding clarification of the eligibility criteria or study design. If details regarding specific aspects of the study design were omitted from the paper, authors were emailed for clarification.
Risk of bias was assessed using a modified version of the Cochrane Collaboration's risk of bias tool.11 The full version of this tool covers six domains: selection bias, performance bias, detection bias, attrition bias, reporting bias and other biases. For the purposes of this review, we felt that selection bias (random sequence generation and allocation concealment), performance bias (blinding of participants and personnel), other biases (analysis by intention to treat) and attrition bias (incomplete outcome data) were important in evaluating the risk of bias associated with internet-based trials. Detection bias (the blinding of outcome assessment) was not considered, as outcomes were self-reported in all trials, and bias arising from self-reporting is considered under the heading of performance bias. It is difficult to evaluate the presence of reporting bias (selective reporting of results) because few trials prospectively registered their protocols. We would expect that the prevalence of reporting bias in internet-based trials would be at least as high as in conventional trials.12 Decisions regarding the risk of bias for included trials were made by all investigators.
Results
The search strategy yielded 1922 unique titles (figure 1) but 1750 were excluded based on their titles or abstracts, leaving 172 full papers to be reviewed. Forty-three papers were excluded as they did not meet inclusion criteria (n=30), or because they did not present trial results (n=13). Twenty-three papers were second or third reports of a single trial, and one paper reported the results of two trials. Online status was therefore evaluated for 107 trials, of which 60 were neither primarily nor fully online. Checks of reference lists identified two more primarily online trials, and one fully online trial was known to the authors, resulting in a total of 23 fully online6–8 13–32 and 27 primarily online4 5 33–56 trials.
Fully online trials
The first fully online trials were conducted in 2002 and published in 2003 and 2004 (See online supplementary appendices 3 and 4). In all of these trials, the interventions were delivered electronically (via the website or email). The interventions included tailored messages, information/education materials, online counseling, interactive programs, workbooks, search features, a decision aid and stretching programs; details are provided in online supplementary appendix 3. Participants were recruited from more than one country in seven of the trials (see online supplementary appendix 5). The mean age of participants in the trials ranged from 26 years in two trials (interventions to promote safe sex and to increase the uptake of cervical cancer screening) to 77 years (an intervention to prevent falls). The smallest trial randomized 90 participants and the largest randomized 12 434 (median=714). Duration of follow-up ranged from immediate to 13 months.
Trials varied in methodological quality (Tables 1; see online supplementary appendix 6). Reports of 16 trials described an adequate process for generation of the allocation sequence. Alternation, a process whereby participants were allocated to treatment groups in an alternate manner in the order in which they accessed the website, was reported in three trials. In standard trials, this method of allocation is not appropriate because the order of recruitment could be related to subject characteristics and because alternation threatens concealment of the allocation sequence. However, in internet-based trials, the order in which participants access the trial website is likely to be random. Consequently, this method of allocation, although not ideal, is likely to achieve the goals of randomization. Allocation concealment was described in reports of only four trials, but given that the allocation was web-based (a feature of all fully online trials), it seems reasonable to assume the allocation sequence was concealed in the remaining 19 trials. Levels of blinding were poorly described. The reports of six trials indicated that an attempt was made to blind participants, and reports of a further three trials noted that participants were not blinded. We assumed that the participants of a further five trials were not blinded, as control group participants were allocated to a waiting list or no intervention. No report of any trial indicated whether an attempt was made to blind researchers. The reports of only three trials described analysis by intention to treat (analysis of participants in the groups which they were allocated to, regardless of whether or not participants completed the intervention as instructed). Loss to follow-up ranged from 0% to 85% (mean 47%). The rate of follow-up was generally insufficient to avoid a significant risk of bias, as only four trials analyzed >85% of randomized participants. The methodological quality of the fully online trials is summarized in table 1 and detailed further in online supplementary appendix 6.
Table 1.
Reference | Random sequence generation (selection bias) | Allocation concealment (selection bias) | Blinding* | ITT analysis (selection bias) | Complete outcome data (attrition bias) |
---|---|---|---|---|---|
Fully online randomized controlled trials | |||||
Etter13 | |||||
Bull14 | |||||
Etter15 | |||||
Christensen16 | |||||
Bowen17 | |||||
Yardley18 | |||||
Bowen19 | |||||
Wanner6 | |||||
Roberto20 | |||||
Etter21 | |||||
Evans22 | |||||
Lin23 | |||||
Te Poel24 | |||||
Geraghty25 | |||||
Carpenter26 | |||||
van der Houwen27 | |||||
Overberg28 | |||||
Mathieu7 | |||||
Jamtvedt8 | |||||
Yardley29 | |||||
Luszczynska30 | |||||
Wallace31 | |||||
Dilorio32 | |||||
Summary n (%)—Fully online randomized controlled trials N=23 | |||||
19 (83) | 23(100) | 6 (26) | 3 (13) | 4 (17) | |
0 (0) | 0 (0) | 3 (13) | 0 (0) | 19 (83) | |
4 (17) | 0 (0) | 14 (61) | 20 (87) | 0 (0) | |
Primarily online randomized controlled trials | |||||
Lorig33 | |||||
Andersson34 | |||||
Devineni35 | |||||
Andersson36 | |||||
Swartz37 | |||||
Etter38 | |||||
Oxman5 | |||||
Riper39 | |||||
Rabius40 | |||||
Severson41 | |||||
Brendryen42 | |||||
Brendryen43 | |||||
McKay44 | |||||
Bull45 | |||||
Hwang46 | |||||
Meyer4 | |||||
Hunt47 | |||||
van Lankveld48 | |||||
Zibell (a)49 | |||||
Zibell (b)49 | |||||
Munoz50 | |||||
Rosser51 | |||||
Postel52 | |||||
Lancee53 | |||||
Spijkerman54 | |||||
Kelders55 | |||||
Webster56 | |||||
Summary n (%)—Primarily online randomized controlled trials N=27 | |||||
15 (56) | 12 (44) | 5 (19) | 3 (11) | 7 (26) | |
0 (0) | 0 (0) | 9 (33) | 1 (4) | 20 (74) | |
12 (44) | 15 (56) | 13 (48) | 23 (85) | 0 (0) |
*Blinding status of participants reported.
+: Method determined to be adequately administered (ie, bias minimized).
–: Method determined to be inadequately administered.
?: Unable to determine if method was adequately administered.
Primarily online trials
The first primarily online trials were published in 2002. It is not stated when these trials were conducted. However, a report of one trial conducted in 2000 was published in 2006 (see online supplementary appendices 3 and 4). The interventions were delivered electronically for 21 trials and by post for three trials. One trial delivered part of the intervention electronically and part by post. Two trials used both the internet and mobile phones to deliver the intervention. Interventions included discussion groups, videos, information booklets, online counseling, interactive programs, drugs, role model stories, tailored advice and a request to change (or maintain) gum chewing behavior. The mean age of participants ranged from 18 years (a binge drinking intervention) to 56 years (cholesterol lowering information) (see online supplementary appendix 5). The smallest trial randomized 54 participants and the largest randomized 6,451 participants (median: 417). Duration of follow-up ranged from two weeks to 12 months.
As with fully online trials, the methodological quality varied (table 1, see online supplementary appendix 6). Reports of 15 trials described an adequate process to generate the allocation schedule: 14 of these used random allocation and one used alternation. As online randomization was not required for primarily online trials, allocation concealment cannot be automatically assumed. Reports of five trials described allocation concealment, and we assumed that an additional seven trials concealed the allocation sequence, as an online randomization process was used. As with fully online trials, levels of blinding were not well described. Reports of 14 studies described the blinding status of participants (five studies blinded participants, nine did not blind participants). A further seven studies were assumed to have not blinded participants, as control group participants were allocated to a waiting list. Three studies described the blinding status of investigators. Intention to treat analysis was reported by three trials, and one trial reported that intention to treat analysis was not undertaken. Loss to follow-up ranged from 8% to 83% (mean=36%). Loss to follow-up was slightly lower (mean=30%) in the 14 trials that described alternate methods to contact participants who did not complete outcome data via the internet compared to those trials that did not describe alternate methods of contact (mean=42%), although this difference was not significant (t25=1.51, p=0.14). Seven trials analyzed >85% of randomized participants (five of which used alternate methods to contact participants). The methodological quality of primarily online trials is summarized in table 1 and detailed further in online supplementary appendix 6.
Discussion
Summary of results
It is possible but perhaps difficult to test the effectiveness of health interventions rigorously with randomized trials conducted over the internet. We identified 50 trials conducted fully or primarily over the internet. It appears that the first of these was conducted in 2000. A diverse spectrum of interventions have been evaluated, many of which are internet-based interventions. Trials have ranged in size and many have recruited participants from around the world. One trial randomized 7935 participants from 73 countries. Methodological quality was variable and generally poorly reported. Only a small number of trials were exposed to little risk of bias, but most had substantial methodological shortcomings and the methods were often poorly reported. Loss to follow-up was a particular issue for the majority of trials.
Range of interventions
Internet-based trials are particularly suited to self-administered interventions (eg, stretching) and trials in which the primary outcome is self-reported (eg, muscle soreness).5 Internet-based trials are not appropriate when the intervention must be delivered in person (eg, surgery) or when there is the possibility of serious adverse events and close, accurate monitoring of participants is required. Even when a trial is not internet-based, it may be possible and beneficial to utilize the internet for some aspects of these trials. For example, the recruitment of participants for conventional trials could be carried out via the internet.
Sample size and recruitment—large and fast
In some circumstances, internet-based trials may be more able to recruit effectively from the target population. A particular strength of internet trials is that they make it possible to obtain large samples quickly. Studies included in this review achieved recruitment rates ranging from six to 6480 potential participants per month (median=231, IQR=75–594 per month). Another strength of recruiting via the internet is that it enables sampling from populations that might otherwise not be accessible. Such populations may include rural men who have sex with men17 19 and those with social anxiety disorders.
Online panels were used in six trials. Online panels or ‘epanels’ consist of panelists who have agreed to participate in online surveys. They are generally not paid for their involvement in these surveys but often receive points which can be used in loyalty programs or to receive credits to online gift stores. In one online panel, the database of potential participants was created using national demographic data, resulting in a database which was representative of the population. This panel was used to recruit parents of adolescent children.22 Another panel used a variety of methods to ensure it had a large pool of members from traditionally hard-to-reach populations. This database was used to recruit English-speaking Hispanics living in America.20 The trials which utilized epanels generally achieved high recruitment rates. However, in some of these trials, a large number of potential participants were invited in order to gain a sufficient sample size.
Multiple registrations
Multiple registrations are an issue in internet-based trials. Participants may register multiple times for a variety of reasons. Some may do so inadvertently, thinking that their first registration did not work. Others may do so intentionally; for example, to receive incentives, if they are unhappy with their initial treatment allocation or if they want to increase their chances of receiving the intervention. It is also possible that participants could be registered by another person who thinks they should participate (eg, a smoking cessation trial).
Only one study included in this review45 undertook validation checks of all participants using both electronic and human processes. Participants were flagged if they had the same IP address and/or a similar name, address, email address and telephone number. Participants who were suspected to have enrolled multiple times were sent an email requiring further verification (eg, by faxing a copy of their driver's license). As a result of this process, 20% of registered participants were excluded, as verification was not achieved.
Generalizability
The external validity of samples recruited via the internet is often criticized57 58 and, in some cases, the criticism is warranted. The digital divide has been well described: internet users tend to be younger and more educated than those that do not use the internet.59 However, as internet access around the world continues to increase, and with the rise of mobile internet devices and wireless technology, the sub-group of the population without internet access will continue to decrease.
Most trials attempted to demonstrate that their sample was representative of the population in which the intervention would be used and, in some cases, more generalizable than would have been possible if recruitment had occurred locally. Internet-based trials allow access to populations that are not accessible or difficult to access via conventional trial methods. The internet greatly increases geographic and demographic reach. Using traditional methods, it is difficult to reach participants who live in rural or isolated communities, participants who are well or participants who have a health condition that does not require medical attention. The internet provides a mechanism for researchers to reach these participants. This is particularly the case in countries where internet penetration is high.
Depending on the context, samples obtained using internet-based sampling will be more or less representative of the target populations than conventional trials. For example, they may be more representative because they can sample subsets of a population that are unable to travel to a clinic, but they may be less representative because they can only sample people with internet access.
Self-selection of participants into any trial is an important issue as, in most cases, self-selection of participants will result in bias being introduced. In conventional trials, although the investigators find potential participants and invite them to participate in the study, the participants have the right to choose whether or not they participate. In internet-based trials, there is generally no formal invitation into the study—the participants tend to ‘invite’ themselves. In both cases, bias is introduced, as neither sample is entirely representative of the population from which they are drawn.
In an attempt to minimize self-selection bias, conventional trial methods may be preferred for trials of clinical populations. Internet-based trials may be more suited for populations that do not often attend clinics.
Methodological quality
The issues that impact upon methodological quality of internet-based trials are no different to issues that affect all trials, regardless of the way in which participants are recruited, the intervention is delivered or outcomes are collected. What does differ is the way in which researchers address these issues and attempt to minimize the bias introduced as a result of these issues.
It is difficult to draw conclusions about the methodological quality of internet-based RCTs, primarily due to inadequate reporting in published papers, particularly with regards to analysis by intention to treat. This issue is no different than conventional trials, and all reports of RCTs should conform to the Consolidated Standards of Reporting Trials recommendations60 regardless of the medium used to conduct the trial.
Sources of internal bias: randomization
Most internet-based randomized trials appear to have used adequate processes to generate the randomization schedule. Some trials used alternation. The use of alternation is less likely to cause bias in internet-based trials than in conventional trials due to the nature in which participants are recruited. Nonetheless, there can be little justification for using alternation. Whenever it is ethically possible to allocate participants using alternation, it is also possible to allocate participants using truly random procedures.
Sources of internal bias: allocation concealment
One of the benefits of using the web server to randomize participants automatically is that this conceals allocation from investigators. Although there is no reason why allocation concealment cannot be achieved for all internet-based trials, we were unable to determine what proportion of primarily online trials used concealed allocation due to poor reporting.
Sources of internal bias: blinding
As with conventional trials, researchers conducting internet-based trials can blind participants and investigators in an attempt to minimize biased outcome reporting, contamination and co-intervention. In internet-based RCTs, blinding of participants is particularly important, as outcomes are predominantly self-reported. Also, there can be little oversight of participants by investigators and health care professionals, so in some trials, the potential for contamination and co-intervention may be high. Blinding of participants in internet-based trials can be implemented in several ways, including the use of a placebo intervention or a standard intervention. For example, in drug trials,5 participants can be sent either the active drug or the placebo drug. In trials of a web-based intervention, participants in the control group can be given access to a standard website while participants in the intervention group can be given access to a more comprehensive interactive website.29 Only 11 of the included 50 trials attempted to blind participants, one of which evaluated the success of blinding by measuring participants’ belief about treatment allocation.
Blinding of investigators is less important in internet-based trials than in conventional trials due to the limited contact that investigators have with participants. In fully automated trials where there is no personal contact between investigators and participants, blinding of investigators is probably unnecessary. However, if members of the research team do have contact with research participants, particularly if researchers are attempting to collect outcome data from participants who have failed to complete data collection via the internet, steps should be taken to blind the researchers involved.
Sources of internal bias: intention to treat analysis
Only six of the 50 trials reported analysis by intention to treat (analyzing participants in the group to which they were randomized, regardless of their exposure to the intervention). A large number of the trials reported conducting an analysis with imputed data for those participants who failed to provide outcome data. This was often, incorrectly, referred to as an intention to treat analysis. Methods of imputing data rely on assumptions regarding the reasons for the data being missing. Data can be assumed to be missing completely at random (the reason for being missing is unrelated to any observed or unobserved patient characteristic), missing at random (the reason for being missing is related to observed characteristics) and missing not at random (the reason for being missing is related to unobserved characteristics of the patient such as the outcome of interest). Contemporary imputation methods can remove attrition bias if certain assumptions (such as the assumption of missing at random) are met.61 However, as the data are missing, it is often not possible to determine which of the assumptions are correct using the data that are available.62
Sources of internal bias: follow-up
Retention of participants is potentially problematic in all trials but is particularly likely to be a problem in internet-based trials. Few fully online trials have successfully followed up a sufficient proportion of participants to avoid bias, with only four of the 23 trials following up >85% of randomized participants. The average rate of follow-up for fully online trials was 53%.
Several of the primarily online trials utilized non-internet-based methods, including telephone and post, to follow participants up. This had varying success, with rates of follow-up ranging from 38% to 93%. There is good evidence that a range of strategies can improve follow-up in conventional trials63 64 but, to our knowledge, the effects of these strategies have not yet been investigated in the context of internet-based trials.
Sources of internal bias: overall
Overall, of the 50 trials, only one trial56 adequately met the five standards (described above), in which the internal biases of the trial are minimized. One trial8 met four of the standards. Due to the nature of the intervention (stretching), blinding was not possible in this trial. For the remaining nine trials with adequate follow-up,17 23 30 38 42 43 46 48 51 we were unable to conclude how methodologically sound these trials were, due to poor reporting.
Use of internet-based trials
Researchers carrying out internet-based trials have limited control over the delivery of the intervention and the measurement of outcomes. This makes internet-based trials more suited to pragmatic trials than explanatory trials. A pragmatic trial is primarily used when researchers wish to evaluate the effects of an intervention under the usual conditions in which it is to be applied. On the other hand, an explanatory trial aims to determine the effectiveness of the intervention under ideal circumstances.65 Very few trials would be classified as purely pragmatic or purely explanatory, and fall into a continuum somewhere between these two extremes.66
Strengths and limitations of this review
We attempted to identify all fully and primarily online RCTs. As there is no published literature on what constitutes a fully or primarily online trial, we set the definitions based on key features of RCTs. As such, some trials have been excluded that others may have included. For example, one trial evaluating the effect of various types of baby bottles on infant colic was excluded, as it allowed some women who did not have access to the internet to complete the study by mail and telephone.67
We only searched the Cochrane Central Register of Controlled Trials. Before making this decision, we conducted a check by searching, for the year 2006, three other databases (Medline, Embase and Cinahl) and compared the results with those obtained by searching only the Cochrane database. Three additional articles were found in the combined databases. However, these three articles would have been identified from the reference lists of papers included in the search of the Cochrane database. As such, we are confident that the Cochrane database of RCTs contains details of most published fully online and primarily online RCTs.
Conclusion
It is theoretically possible to test the effectiveness of health interventions rigorously with randomized trials conducted partly or wholly over the internet. A pragmatic evaluation of the intervention is more appropriate, given the limited control the investigators have over the intervention delivery and outcome measurement. There are serious problems with the quality of reporting of internet-based trials. The main limitation of internet-based trials is that they typically experience high rates of loss to follow-up. To improve the internal validity of trials, researchers must emphasize to potential participants the importance of participating fully in trials. Where possible, multiple methods should be used to follow participants up. Further research is required to determine the best methods to encourage greater retention of participants in internet-based trials. In the meantime, researchers conducting internet-based trials need to understand the importance of using Consolidated Standards of Reporting Trials guidelines when reporting their results.
Footnotes
Contributions: All authors contributed to the conception and design of the study. EM conducted the review and extracted all data. EM drafted the manuscript, and all authors revised it critically for important intellectual content and provided final approval of the version to be published. EM is the guarantor.
Funding: Erin Mathieu and Robert Herbert are supported by grants by the National Health and Medical Research Council of Australia. The funding body had no role in the study.
Competing interests: None.
Ethics approval: As this study reports the results of a systematic review and no original data were collected, ethical approval was not required.
Provenance and peer review: Not Commissioned; externally peer reviewed.
References
- 1.Lachin JM. Statistical properties of randomization in clinical trials. Control Clin Trials 1988;9:289–311 [DOI] [PubMed] [Google Scholar]
- 2.Horwitz RI. Complexity and contradiction in clinical trial research. Am J Med 1987;82:498–510 [DOI] [PubMed] [Google Scholar]
- 3.Paul J, Seib R, Prescott T. The internet and clinical trials: background, online resources, examples and issues. J Med Internet Res 2005;7:e5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Meyer B, Berger T, Caspar F, et al. Effectiveness of a novel integrative online treatment for depression (Deprexis): randomized controlled trial. J Med Internet Res 2009;11:e7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Oxman AD, Flottorp S, Havelsrud K, et al. A televised, web-based randomised trial of an herbal remedy (valerian) for insomnia. PLoS One 2007;2:e1040. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Wanner M, Martin-Diener E, Braun-Fahrlander C, et al. Effectiveness of active-online, an individually tailored physical activity intervention, in a real-life setting: randomized controlled trial. J Med Internet Res 2009;11:e23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Mathieu E, Barratt A, McGeechan K, et al. Helping women make choices about mammography screening: an online randomized trial of a decision aid for 40 year old women. Patient Educ Couns 2010;8:63–72 [DOI] [PubMed] [Google Scholar]
- 8.Jamtvedt G, Herbert RD, Flottorp S, et al. A pragmatic randomised trial of stretching before and after physical activity to prevent injury and soreness. Br J Sports Med 2010;44:1002–9 [DOI] [PubMed] [Google Scholar]
- 9.Murray E, Khadjesari Z, White IR, et al. Methodological challenges in online trials. J Med Internet Res 2009;11:e9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Higgins JPT, Green S. ed. Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration; 2011. http://www.cochrane-handbook.org (accessed 27 March 2012)
- 11.Higgins JPT, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Mathieu S, Boutron I, Moher D, et al. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 2009;302:977–84 [DOI] [PubMed] [Google Scholar]
- 13.Etter JF, le Houezec J, Landfeldt B. Impact of messages on concomitant use of nicotine replacement therapy and cigarettes: a randomized trial on the internet. Addiction 2003;98:941–50 [DOI] [PubMed] [Google Scholar]
- 14.Bull SS, Lloyd L, Rietmeijer C, et al. Recruitment and retention of an online sample for an HIV prevention intervention targeting men who have sex with men: the Smart Sex Quest Project. AIDS Care 2004;16:931–43 [DOI] [PubMed] [Google Scholar]
- 15.Etter JF. Comparing the efficacy of two internet-based, computer-tailored smoking cessation programs: a randomized trial. J Med Internet Res 2005;7:e2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Christensen H, Griffiths KM, Mackinnon AJ, et al. Online randomized controlled trial of brief and full cognitive behaviour therapy for depression. Psychol Med 2006;36:1737–46 [DOI] [PubMed] [Google Scholar]
- 17.Bowen AM, Horvath K, Williams ML. A randomized control trial of internet-delivered HIV prevention targeting rural MSM. Health Educ Res 2007;22:120–7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Yardley L, Nyman SR. Internet provision of tailored advice on falls prevention activities for older people: a randomized controlled evaluation. Health Promot Int 2007;22:122–8 [DOI] [PubMed] [Google Scholar]
- 19.Bowen AM, Williams ML, Daniel CM, et al. Internet based HIV prevention research targeting rural MSM: feasibility, acceptability, and preliminary efficacy. J Behav Med 2008;31:463–77 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Roberto AJ, Krieger JL, Beam MA. Enhancing web-based kidney disease prevention messages for Hispanics using targeting and tailoring. J Health Commun 2009;14:525–40 [DOI] [PubMed] [Google Scholar]
- 21.Etter JF. Comparing computer-tailored, internet-based smoking cessation counseling reports with generic, untailored reports: a randomized trial. J Health Commun 2009;14:646–57 [DOI] [PubMed] [Google Scholar]
- 22.Evans WD, Davis KC, Ashley OS, et al. Efficacy of abstinence promotion media messages: findings from an online randomized trial. J Adolesc Health 2009;45:409–16 [DOI] [PubMed] [Google Scholar]
- 23.Lin ZC, Wang SF. A tailored web-based intervention to promote women's perceptions of and intentions for mammography. J Nursing Res 2009;17:249–60 [DOI] [PubMed] [Google Scholar]
- 24.Te Poel F, Bolman C, Reubsaet A, et al. Efficacy of a single computer-tailored e-mail for smoking cessation: results after 6 months. Health Educ Res 2009;24:930–40 [DOI] [PubMed] [Google Scholar]
- 25.Geraghty AW, Wood AM, Hyland ME. Attrition from self-directed interventions: investigating the relationship between psychological predictors, intervention content and dropout from a body dissatisfaction intervention. Soc Sci Med 2010;71:30–7 [DOI] [PubMed] [Google Scholar]
- 26.Carpenter KM, Stoner SA, Mikko AN, et al. Efficacy of a web-based intervention to reduce sexual risk in men who have sex with men. AIDS Behav 2010;14:549–57 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.van der Houwen K, Schut H, van den Bout J, et al. The efficacy of a brief internet-based self-help intervention for the bereaved. Behav Res Ther 2010;48:359–67 [DOI] [PubMed] [Google Scholar]
- 28.Overberg R, Otten W, de Man A, et al. How breast cancer patients want to search for and retrieve information from stories of other patients on the internet: an online randomized controlled experiment. J Med Internet Res 2010;12:e7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Yardley L, Joseph J, Michie S, et al. Evaluation of a web-based intervention providing tailored advice for self-management of minor respiratory symptoms: exploratory randomized controlled trial. J Med Internet Res 2010;12:e66. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Luszczynska A, Goc G, Scholz U, et al. Enhancing intentions to attend cervical cancer screening with a stage-matched intervention. Br J Health Psych 2011;16(Pt 1):33–46 [DOI] [PubMed] [Google Scholar]
- 31.Wallace P, Murray E, McCambridge J, et al. On-line randomized controlled trial of an internet based psychologically enhanced intervention for people with hazardous alcohol consumption. PLoS One 2011;6:e14740. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.DiIorio C, Bamps Y, Walker ER, et al. Results of a research study evaluating WebEase, an online epilepsy self-management program. Epilepsy Behav 2011;22:469–474. [DOI] [PubMed] [Google Scholar]
- 33.Lorig KR, Laurent DD, Deyo RA, et al. Can a back pain e-mail discussion group improve health status and lower health care costs? A randomized study. Arch Intern Med 2002;162:792–6 [DOI] [PubMed] [Google Scholar]
- 34.Andersson G, Stromgren T, Strom L, et al. Randomized controlled trial of internet-based cognitive behavior therapy for distress associated with tinnitus. Psychosom Med 2002;64:810–16 [DOI] [PubMed] [Google Scholar]
- 35.Devineni T, Blanchard EB. A randomized controlled trial of an internet-based treatment for chronic headache. Behav Res Ther 2005;43:277–92 [DOI] [PubMed] [Google Scholar]
- 36.Andersson G, Bergstrom J, Hollandare F, et al. Internet-based self-help for depression: randomised controlled trial. Br J Psychiatry 2005;187:456–61 [DOI] [PubMed] [Google Scholar]
- 37.Swartz LH, Noell JW, Schroeder SW, et al. A randomised control study of a fully automated internet based smoking cessation programme. Tob Control 2006;15:7–12 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Etter JF. Informing smokers on additives in cigarettes: a randomized trial. Patient Educ Couns 2007;66:188–91 [DOI] [PubMed] [Google Scholar]
- 39.Riper H, Kramer J, Smit F, et al. Web-based self-help for problem drinkers: a pragmatic randomized trial. Addiction 2008;103:218–27 [DOI] [PubMed] [Google Scholar]
- 40.Rabius V, Pike KJ, Wiatrek D, et al. Comparing internet assistance for smoking cessation: 13-month follow-up of a six-arm randomized controlled trial. J Med Internet Res 2008;10:e45. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Severson HH, Gordon JS, Danaher BG, et al. ChewFree.com: evaluation of a web-based cessation program for smokeless tobacco users. Nicotine Tobacco Res 2008;10:381–91 [DOI] [PubMed] [Google Scholar]
- 42.Brendryen H, Kraft P. Happy ending: a randomized controlled trial of a digital multi-media smoking cessation intervention. Addiction 2008;103:478–84 [DOI] [PubMed] [Google Scholar]
- 43.Brendryen H, Drozd F, Kraft P. A digital smoking cessation program delivered through internet and cell phone without nicotine replacement (happy ending): randomized controlled trial. J Med Internet Res 2008;10:e51. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.McKay HG, Danaher BG, Seeley JR, et al. Comparing two web-based smoking cessation programs: randomized controlled trial. J Med Internet Res 2008;10:e40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Bull S, Pratte K, Whitesell N, et al. Effects of an Internet-based intervention for HIV prevention: the Youthnet trials. AIDS Behav 2009;13:474–87 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Hwang KO, Hamadah AM, Johnson CW, et al. Screening for obstructive sleep apnea on the internet: randomized trial. Am J Med 2009;122:e1–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Hunt MG, Moshier S, Milonova M. Brief cognitive-behavioral internet therapy for irritable bowel syndrome. Behav Res Ther 2009;47:797–802 [DOI] [PubMed] [Google Scholar]
- 48.van Lankveld JJDM, Leusink P, van Diest S, et al. Internet-based brief sex therapy for heterosexual men with sexual dysfunctions: a randomized controlled pilot trial. J Sex Med 2009;6:2224–36 [DOI] [PubMed] [Google Scholar]
- 49.Zibell S, Madansky E. Impact of gum chewing on stress levels: online self-perception research study. Curr Med Res Opin 2009;25:1491–500 [DOI] [PubMed] [Google Scholar]
- 50.Munoz RF, Barrera AZ, Delucchi K, et al. International Spanish/English internet smoking cessation trial yields 20% abstinence rates at 1 year. Nicotine Tobacco Res 2009;11:1025–34 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Rosser BR, Oakes JM, Konstan J, et al. Reducing HIV risk behavior of men who have sex with men through persuasive computing: results of the Men's INTernet Study-II. AIDS 2010;24:2099–107 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Postel MG, de Haan HA, ter Huurne ED, et al. Effectiveness of a web-based intervention for problem drinkers and reasons for dropout: randomized controlled trial. J Med Internet Res 2010;12:e68. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Lancee J, Spoormaker VI, van den Bout J. Cognitive-behavioral self-help treatment for nightmares: a randomized controlled trial. Psychother Psychosom 2010;79:371–7 [DOI] [PubMed] [Google Scholar]
- 54.Spijkerman R, Roek MA, Vermulst A, et al. Effectiveness of a web-based brief alcohol intervention and added value of normative feedback in reducing underage drinking: a randomized controlled trial. J Med Internet Res 2010;12:e65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Kelders SM, van Gemert-Pijnen JE, Werkman A, et al. Usage and effect of a web-based intervention for the prevention of overweight; a RCT. Stud Health Tech Informatics 2010;160:28–32 [PubMed] [Google Scholar]
- 56.Webster R, Li SCH, Sullivan DR, et al. Effects of internet-based tailored advice on the use of cholesterol-lowering interventions: a randomized controlled trial. J Med Internet Res 2010;12:e42. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Couper MP. Issues of representation in ehealth research (with a focus on web surveys). Am J Prev Med 2007;32:S83–9 [DOI] [PubMed] [Google Scholar]
- 58.Braithwaite D, Emery J, de Lusignan S, et al. Using the internet to conduct surveys of health professionals: a valid alternative? Fam Prac 2003;20:545–51 [DOI] [PubMed] [Google Scholar]
- 59.Willis S, Tranter B. Beyond the ‘digital divide’: internet diffusion and inequality in Australia. J Sociology 2006;42:43–59 [Google Scholar]
- 60.Schulz KF, Altman DG, Moher D, et al. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med 2010;152:726–32 [DOI] [PubMed] [Google Scholar]
- 61.White IR, Horton NJ, Carpenter J, et al. Strategy for intention to treat analysis in randomised trials with missing outcome data. BMJ 2011;342:d40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 62.Sterne JA, White IR, Carlin JB, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ 2009;338:b2393. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 63.Sackett DL. The principles behind the tactics of performing therapeutic trials. In Haynes RB, Sackett DL, Guyatt GH, Tugwell P (Eds). Clinical Epidemiology: How to do clinical practice research. Philadelphia: Lippincott, Williams and Wilkins; 2006:173–243. [Google Scholar]
- 64.Thorpe KE, Zwarenstein M, Oxman AD, et al. A pragmatic–explanatory continuum indicator summary (PRECIS): a tool to help trial designers. CMAJ 2009;180:E47–57 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65.Edwards P, Roberts I, Clarke M, et al. Methods to increase response rates to postal questionnaires. Cochrane Database Syst Rev 2007;(2):MR00008 doi:10.1002/14651858.MR000008.pub3 [DOI] [PubMed] [Google Scholar]
- 66.Kiezebrink K, Crombie I, Irvine L, et al. Strategies for achieving a high response rate in a home interview survey. BMC Med Res Methodol 2009;9:46. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 67.Cirgin Ellett Ml, Perkins SM. Examination of the effect of Dr. Brown's Natural Flow Baby Bottles on infant colic. Gastroenterol Nurs 2006;29:226–31 [DOI] [PubMed] [Google Scholar]