A systematic review of eHealth cancer prevention and control interventions: new technology, same methods and designs?

Abstract

There has been a recent surge of eHealth programs in cancer and other content areas, but few reviews have focused on the methodologies and designs employed in these studies. We conducted a systematic review of studies on eHealth interventions on cancer prevention and control published between 2001 and 2010 applying the Pragmatic Explanatory Continuum Indicator Summary (PRECIS) criteria and external validity components from the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) framework. We identified 113 studies that focused on cancer prevention and control of eHealth interventions. Most studies fell midway along the explanatory/pragmatic trial continuum, but few reported on various practical feasibility criteria for translation. Despite vast interest in cancer eHealth and the applied nature of this field, few studies considered key external validity issues. There is a need for use of alternative pragmatic study designs and transparent reporting of external validity components to produce more rapid and generalizable results.

BACKGROUND

Reductions in cancer morbidity and mortality are partly attributable to interventions addressing modifiable risk factors and screening behaviors [1]. However, such advancements in cancer outcomes are not observed among all populations, specifically those with limited access to cancer care and efficacious behavior interventions [2, 3]. The recent surge of eHealth interventions (EHIs) presents unique opportunities to enhance cancer prevention and control by increasing intervention reach, adapting to various contextual conditions, being readily available where users live, work, and play, and tailoring information to patients' needs [4–6].

eHealth research is a relatively young field that is rapidly growing. EHIs have been found to be effective in promoting change in behaviors, knowledge, self-efficacy, and clinical outcomes [4–8]. While substantial progress has been made, few efficacious EHIs are adopted or sustained in real-world settings beyond the scope of the research project [5]. This lack in translation may be due, in part, to the use of predominantly explanatory (efficacy) research methods, which do not usually evaluate external validity, and to issues with limited reporting of intervention details (e.g., intervention cost and contextual factors of implementation setting), which would allow for replication [9, 10]. Moreover, the extent to which eHealth studies have addressed both effectiveness and generalizability is unknown.

To address these issues, two Consolidated Standards of Reporting Trials (CONSORT) statements have been developed describing reporting criteria for EHIs [10] and pragmatic trials [11] that provide guidance for study designs and evaluation methods, and reporting to inform decisions pertaining to both effectiveness and practical implementation of EHIs [12]. The Pragmatic Explanatory Continuum Indicator Summary (PRECIS) developed by the CONSORT Work Group on pragmatic trials is designed to assist researchers in study design and assess a study along the pragmatic–explanatory continuum, a multidimensional continuum displaying varying levels of pragmatism across study dimensions of a particular trial [11, 12]. PRECIS has been used to develop intervention trial design [13, 14], to assess interventions with common goals and measures [15], and to draw contextual information from systematic reviews [16]. Additional ratings of practical feasibility and other implementation factors related to CONSORT criteria for EHIs [10] may be useful for study design and have been applied in a similar assessment of intervention trials [15].

The purposes of this paper are as follows: (1) review and summarize the literature to assess the extent to which EHI studies in cancer prevention and control have utilized pragmatic trial design features and reported on issues related to translation, generalizability, and feasibility measures; (2) apply practical feasibility and generalizability reporting criteria to EHI trials; and (3) describe our experiences, including review procedures and inter-rater reliability in applying both rating criteria.

METHODS

Identification of studies

For the purpose of this review, the description of eHealth interventions put forth by Eng et al. was used [17]. eHealth interventions are defined as “the use of emerging information and communication technology, especially the Internet, to improve or enable health and health care,” which includes internet, email, mobile phone text or applications, interactive voice response, automated and electronic programs, CD-ROMs, and computer-tailored print but exclude telemedicine targeted solely at clinicians that do not have a patient- or consumer-facing interface [17]. This review was based on a larger systematic review of the EHI literature focused on health promotion and disease management published in English between 1980 and mid 2010 [18]. Methods of the original systematic review are described first to provide a context for the current review.

The original systematic review by Rabin et al. [18] included 467 papers. An inclusion criterion for the review was evaluation of an EHI as the main intervention component primarily directed toward patients and/or their caregivers rather than healthcare providers. We excluded studies of telemedicine interventions, and those that were expressly described as feasibility, preliminary, or pilot studies were excluded since it was not considered appropriate to hold these latter studies to the methodological standards used for intervention trials. Studies were classified as T1 (efficacy studies) or T2 and later stage (T2+), operationalized as effectiveness, dissemination and implementation, and scale-up studies using the translational research stages [19]. Given T1 studies focus mostly on efficacy of basic biological discovery to candidate health application, T1 studies had to use an experimental design using randomized or nonrandomized comparison condition(s) and had to include at least one broadly defined behavioral-oriented outcome (e.g., change in health behavior, knowledge, and self-efficacy) or biological outcome (e.g., BMI and HbA1c). Given the broad range of T2+ studies focus on effectiveness, dissemination, and outcomes research of existing health applications to practice and population impact, T2+ studies were included regardless of their study design or measured outcomes. A more detailed description of methods is provided elsewhere [18].

For the current review, we selected a subsample of 149 papers related to cancer prevention and control from the original review described above. Studies from the original review were included if they had relevance to any stage of the cancer control continuum, specifically EHIs that applied behavioral, social, and populations sciences to study new approaches that address cancer-related issues including primary prevention, screening, treatment/disease management, survivorship, and end-of-life care [20]. Studies focusing on management of diabetes and cardiovascular disease were included if the main focus was management of cancer-related risk behaviors (e.g., diet, physical activity, etc.). Interventions had to have an interactive component (e.g., we excluded tailored print and phone interventions unless the provider used them to counsel patients). We limited our review to studies published between 2001 and 2010. These years reflect a period in which there was a large increase in peer-reviewed publications of both T1 and T2+ EHI study results. We only included studies that focused on adults, since eHealth modalities for adults and adolescents are quite different within the context of cancer prevention and control.

After articles were identified, we grouped them into studies. Since study results could be published in multiple papers, we assigned reviewers by study groups rather than individual papers so reviewers would have the full range of information provided about the study across the various papers. After assignments were made, each reviewer was instructed to conduct a brief web search of publications (using the combination of the intervention name and first author) within PubMed to identify any additionally relevant papers to the main study paper and and/or group of papers and include those in the rating process.

Rating of studies for pragmatism and practical feasibility

Reviewers abstracted general information (i.e., citation, topic area, eHealth modality, setting, study design, and target audience) from each study and rated them for pragmatism and practical feasibility using two rating scales. The pragmatic rating scale was adapted from the PRECIS review tool with PRECIS criteria as a means to identify the extent to which a trial was widely applicable (pragmatic) or more mechanistic (explanatory) [12, 16]. PRECIS criteria includes 10 domains related to participant eligibility criteria, flexibility of the experimental and comparison conditions, follow-up intensity, primary trial outcomes, participant compliance, practitioner adherence to study protocol, and primary analysis. A more detailed description illustrating the extremes of explanatory and pragmatic approaches to each PRECIS domain and the PRECIS review tool is provided elsewhere [12, 16].

The practical feasibility rating scale was previously developed by members of the study team (BR, RG, and BG) based on the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) framework [15, 21]. We adapted this rating scale to include translational aspects of CONSORT reporting criteria of eHealth intervention trials [10, 21]. Similar to PRECIS, the practical feasibility rating criteria were adapted to assess the extent to which a trial was pragmatic or explanatory in addressing issues important to potential eHealth adopters that were not included within the PRECIS criteria. Practical feasibility domains focused on setting and participant representativeness (i.e., how typical are recruited settings/participants of target population), participant engagement with all parts of the intervention, intervention adaptation during study, program sustainability, unintended effects (i.e., harmful or beneficial consequences), monetary costs of intervention, and intervention resources (i.e., extent to which minimal intervention resources were reported). Both rating scales used a five-point Likert scale for responses, where “5” indicated a completely pragmatic and “1” a completely explanatory approach. The rating form is available upon request from the lead author.

Seven reviewers participated in the rating process. All reviewers were trained on both the PRECIS and practical feasibility criteria. Training sessions also served to develop consensus on all rater domains. The rating form was pilot-tested and refined based on the ratings of a subsample of four papers by all reviewers. For studies in which EHI replaced practitioners with no personal or phone contact, ‘not applicable’ ratings were applied to relevant PRECIS domains on practitioner expertise and practitioner compliance to study protocol. After refinements and clarification of rating process, the entire group reviewed two additional papers to pilot the revised criteria. For the remainders of the papers, two reviewers were assigned to each study.

Analyses

A total of 113 studies covered by 149 individual publications were independently rated by two reviewers (Table 1). Average scores were calculated for each study on PRECIS and practical feasibility domain for areas in which ratings were appropriate. Overall composite mean scores across all studies were calculated for PRECIS and practical feasibility domains (Table 2), and individual scores are available upon request. These scores were aggregated by study characteristics including intervention settings, intervention topic, eHealth modalities, target audience, cancer control continuum, year published, and translational phase. Analyses of variance (ANOVA) with Tukey's post hoc corrections were conducted to compare average scores across study characteristics (Table 3). A trend analysis of composite mean scores by domain across all publication years was conducted using a nonparametric test for trends based on the Wilcoxon–Mann–Whitney test.

Table 1.

List of studies reviewed

Article title	Journal	Year of publication	Lead author
A randomized clinical trial evaluating online interventions to improve fruit and vegetable consumption	Am J Public Health	2010	Alexander GL
A computerized social cognitive intervention for nutrition behavior: direct and mediated effects on fat, fiber, fruits, and vegetables, self-efficacy, and outcome expectations among food shoppers	Ann Behav Med	2001	Anderson ES
Text-message reminders to improve sunscreen use: a randomized, controlled trial using electronic monitoring	Arch Dermatol	2009	Armstrong AW
Web-based weight loss in primary care: a randomized controlled trial	Obesity	2010	Bennett GG
An online lifestyle diary with a persuasive computer assistant providing feedback on self-management	Technol Health Care	2009	Blanson Henkemans OA
A randomized trial of the Little by Little CD-ROM: demonstrated effectiveness in increasing fruit and vegetable intake in a low-income population	Prev Chronic Dis	2004	Block G
Smoking cessation using mobile phone text messaging is as effective in Maori as non-Maori	N Z Med J	2005	Bramley D
A digital smoking cessation program delivered through internet and cell phone without nicotine replacement (happy ending): randomized controlled trial	J Med Internet Res	2008	Brendryen H
Happy ending: a randomized controlled trial of a digital multi-media smoking cessation intervention	Addiction	2008	Brendryen H
Understanding the effects of printed health education materials: which features lead to which outcomes?	Health Commun	2001	Bull FC
Randomized trial on the 5 a day, the Rio Grande Way Website, a web-based program to improve fruit and vegetable consumption in rural communities	J Health Commun	2008	Buller DB
Improving multiple behaviors for colorectal cancer prevention among African American church members	Health Psychol	2004	Campbell MK
Randomized trial of a tailored nutrition education CD-ROM program for women receiving food assistance	J Nutr Educ Behav	2004	Campbell MK
Comparisons of tailored mammography interventions at two months post intervention	Ann Behav Med	2002	Champion VL
Comparison of three interventions to increase mammography screening in low income African American women	Cancer Detect Prev	2006	Champion VL
The effect of telephone versus print tailoring for mammography adherence	Patient Educ Couns	2007	Champion VL
A field test of a web-based workplace health promotion program to improve dietary practices, reduce stress, and increase physical activity: randomized controlled trial	J Med Internet Res.	2007	Cook RF
Pressing the key pad: trial of a novel approach to health promotion advice	Prev Med	2005	Corkrey R
Interactive voice response reminder effects on preventive service utilization	Am J Med Qual	2005	Crawford AG
Maintenance of weight loss in overweight middle-aged women through the Internet	Obesity	2008	Cussler EC
Evaluation of an interactive computer-tailored nutrition intervention in a real-life setting	Ann Behav Med	2006	De Bourdeaudhuij I
Short- and long-term effects of tailored information versus general information on determinants and intentions related to early detection of cancer	Prev Med	2004	De Nooijer J
Randomized trial of a "talking computer" to improve adults' eating habits	Am J Health Promot	2001	Delichatsios HK
Comparison of two email-delivered, pedometer-based interventions to promote walking among insufficiently active women	J Sci Med Sport	2007	Dinger MK
Solar UV forecasts: a randomized trial assessing their impact on adults' sun-protection behavior	Health Educ Behav	2007	Dixon HG
A tailored Internet-plus-email intervention for increasing physical activity among ethnically-diverse women	Prev Med	2008	Dunton GF
Health-related quality of life and physical recovery after a critical illness: a multi-centre randomized controlled trial of a home-based physical rehabilitation program	Crit Care	2004	Emmons KM
Comparing the efficacy of two Internet-based, computer-tailored smoking cessation programs: a randomized trial	Med Internet Res	2005	Etter JF
Randomized trial of a neighborhood environment-focused physical activity website intervention	Prev Med	2009	Ferney SL
A randomized controlled trial comparing internet and video to facilitate patient education for men considering the prostate specific antigen test	J Gen Intern Med	2009	Frosch DL
Associations of internet website use with weight change in a long-term weight loss maintenance program	J Med Internet Res	2010	Funk Kl
Reach, engagement, and retention in an Internet-based weight loss program in a multisite randomized controlled trial	Med Internet Res	2007	Glasgow RE
Effect of computer support on younger women with breast cancer	J Gen Intern Med	2001	Gustafson DH
Effect of computer support on younger women with breast cancer	J Commun	2008	Gustafson DH
Weight loss by mobile phone: a 1-year effectiveness study	Public Health Nutr	2009	Haapala I
Evaluation of an Internet, Stage-Based Physical Activity Intervention	Am J Health Educ	2002	Hager RL
Does using the Internet facilitate the maintenance of weight loss?	Int J Obes Relat Metab Disord	2002	Harvey-Berino J
Effect of internet support on the long-term maintenance of weight loss	Obes Res	2004	Harvey-Berino J
Effect of Internet peer-support groups on psychosocial adjustment to cancer: a randomized study	Br J Cancer	2010	Hoybye MT
The effects on saturated fat purchases of providing internet shoppers with purchase-specific dietary advice: a randomized trial	PLoS Clin Trials	2006	Huang A
The effect of an internet-based, stage-matched message intervention on young Taiwanese women's physical activity	J Health Commun	2009	Huang SJ
Weight management using the internet a randomized controlled trial	Am J Prev Med	2008	Hunter CM
Using internet and mobile phone technology to deliver an automated physical activity program: randomized controlled trial	J Med Internet Res	2007	Hurling R
The effectiveness of an interactive multimedia program to influence eating habits	Health Educ Res	2004	Irvine AB
Effects of a tailored follow-up intervention on health behaviors, beliefs, and attitudes	J Womens Health	2006	Jacobs AD
Smoking cessation via the internet: a randomized clinical trial of an internet intervention as adjuvant treatment in a smoking cessation intervention	Nicotine Tob Res	2006	Japunitch SJ
Changes in diabetes self-care behaviors make a difference in glycemic control: the Diabetes Stages of Change (DiSC) study	Diabetes Care	2003	Jones H
Utility of a Web-based intervention for individuals with type 2 diabetes: the impact on physical activity levels and glycemic control	Comput Inform Nurs	2006	Kim CJ
Ongoing physical activity advice by humans versus computers: the Community Health Advice by Telephone (CHAT) trial	Health Psychol	2007	King AC
A randomized trial of an Internet weight control resource: the UK Weight Control Trial	BMC Health Serv Res	2003	Kirk SF
The efficacy of Web-based and print-delivered computer-tailored interventions to reduce fat intake: results of a randomized, controlled trial	J Nutr Educ Behav	2008	Kroeze W
Effects of a Web-based food portion training program on food portion estimation	J Nutr Educ Behav	2007	Riley WT
Effects of a Web-based food portion training program on food portion estimation	J Nutr Educ Behav	2007	Riley WT
The short-term impact of tailored mammography decision-making interventions	Patient Educ Couns	2001	Rimer BK
Do u smoke after txt? Results of a randomized trial of smoking cessation using mobile phone text messaging	Tob Control	2005	Rodgers A
Enhancing theoretical fidelity: an e-mail-based walking program demonstration	Am J Health Promot	2005	Rovniak LS
Increasing nutrition literacy: testing the effectiveness of print, web site, and game modalities	J Nutr Educ Behav	2008	Silk KJ
Do brief online planning interventions increase physical activity amongst university students? A randomized controlled trial	Psychol Health	2010	Skar S
Evaluation of a website-delivered computer-tailored intervention for increasing physical activity in the general population	Prev Med	2007	Spittaels H
Effectiveness of an online computer-tailored physical activity intervention in a real-life setting	Health Educ Res	2007	Spittaels H
Randomized trial of a brief dietary intervention to decrease consumption of fat and increase consumption of fruits and vegetables	Am J Health Promot	2002	Stevens VJ
Randomized controlled trial of a web-based computer-tailored smoking cessation program as a supplement to nicotine patch therapy	Addiction	2005	Strecher V
The role of engagement in a tailored web-based smoking cessation program: randomized controlled trial	J Med Internet Res	2008	Strecher VJ
The PRO-AGE study: an international randomized controlled study of health risk appraisal for older persons based in general practice	BMC Med Res Methodol	2007	Stuck AE
A randomized control study of a fully automated internet based smoking cessation program	Tob Control	2006	Swarz LH
Using Internet technology to deliver a behavioral weight loss program	JAMA	2001	Tate DF
Effects of Internet behavioral counseling on weight loss in adults at risk for type 2 diabetes: a randomized trial	JAMA	2003	Tate DF
A randomized trial comparing human email counseling, computer-automated tailored counseling, and no counseling in an Internet weight loss program	Arch Intern Med	2006	Tate DF
Efficacy of a single computer-tailored e-mail for smoking cessation: results after 6 months	Health Educ Res	2009	Te Poel F
Effectiveness of a nutrition intervention with rural low-income women	Am J Health Behav	2007	Tessaro I
Multicenter randomized evaluation of a nutritional education software in obese patients	Diabetes Metab	2001	Turnin MC
Randomized controlled trial of a computer-based, tailored intervention to increase smoking cessation counseling by primary care physicians	J Gen Intern Med	2007	Unrod M
Using internet technology to deliver a home-based physical activity intervention for patients with rheumatoid arthritis: a randomized controlled trial	Arthritis Rheum	2006	Van den Berg MH
Investigating message-framing effects in the context of a tailored intervention promoting physical activity	Health Educ Res	2010	Van't Riet J
Efficacy of sequential or simultaneous interactive computer-tailored interventions for increasing physical activity and decreasing fat intake	Ann Behav Med	2005	Vandelanotte C
A randomized trial of sequential and simultaneous multiple behavior change interventions for physical activity and fat intake	Prev Med	2008	Vandelanotte C
Evaluating nicotine replacement therapy and stage-based therapies in a population-based effectiveness trial	J Consult Clin Psychol	2006	Velicer WF
Web-based targeted nutrition counseling and social support for patients at increased cardiovascular risk in general practice: randomized controlled trial	J Med Internet Res	2004	Verheijden M
Effect of a web site intervention on physical activity of college females	Am J Health Behav	2010	Wadsworth DD
Comparison of trial participants and open access users of a web-based physical activity intervention regarding adherence, attrition, and repeated participation	J Med Internet Res	2010	Wanner M
A randomized comparison of two motivationally enhanced Internet behavioral weight loss programs	Behav Res Ther	2008	Webber KH
Guide to health: nutrition and physical activity outcomes of a group-randomized trial of an Internet-based intervention in churches	Ann Behav Med	2007	Winnett RA
Effect of emailed messages on return use of a nutrition education website and subsequent changes in dietary behavior	J Med Internet Res	2006	Woolf SH
Computerized weight loss intervention optimizes staff time: the clinical and cost results of a controlled clinical trial conducted in a managed care setting	J Am Diet Assoc	2001	Wylie-Rosett J

Table 2.

Average ratings (and standard deviations) by domains and composite mean scores for PRECIS and practical feasibility

PRECIS domains
Participant eligibility	Experimental intervention flexibility	Practitioner expertise (experimental)	Comparison intervention flexibility	Practitioner expertise (comparison)	Follow-up intensity	Primary trial outcomes	Participant compliance	Practitioner adherence to protocol	Primary analysis	Composite mean scores
3.2 (0.7)	2.8 (0.9)	2.7 (0.9)	2.9 (1.2)	2.8 (1)	3.2 (0.9)	2.9 (0.7)	3.2 (0.8)	3.6 (1)	3.5 (0.8)	3.12 (0.5)*
Practical feasibility domains
Setting representativeness	Participant representativeness	Participant engagement	Adaptation/change	Program sustainability	Unintended effects	Monetary costs of existing treatment	Intervention resources			Composite mean scores
2.8 (1)	2.5 (0.7)	2.7 (0.7)	1.5 (0.6)	1.6 (0.7)	1.6 (1)	1.6 (0.7)	1.7 (0.7)			1.98 (0.4)*

^* p = .000 (statistically significant difference between PRECIS and practical feasibility composite mean scores)

Table 3.

Summary of PRECIS and practical feasibility composite mean scores (and standard deviations) by study characteristics

Scoring criteria	Intervention topic	Cancer control continuum	eHealth modality	Tailored	Intervention setting	Study design	Target population	Year published	Translation phase
	Cancer screening (n = 12)a	Primary prevention (n = 78)a	Computer-based, CD-ROM and multimedia (n = 10)	No (n = 65)	Healthcare (n = 35)	RCT (n = 99)	At risk (n = 58)	2001–2005 (n = 44)	T1 (n = 97)
PRECIS	3.2 (0.6)	3.1 (0.5)	3.2 (0.7)	3.1 (0.6)	3.1 (0.3)	3.1 (0.5)	3.1 (0.5)	3.2 (0.6)	3.1 (0.5)
Feasibility	2.1 (0.5)	1.9 (0.3)	2.3 (0.6)	2 (0.5)	2.1 (0.5)*	2 (0.4)	2.1 (0.4)	2.1 (0.5)*	1.9 (0.4)*
	Diet/nutrition (n = 19)	Screening (n = 12)a	CT-phone/IVR (n = 7)	Yes (n = 48)	Community (n = 51)	Quasi (n = 9)	Healthy (n = 44)	2006–2010 (n = 69)	T2+ (n = 16)
PRECIS	3.1 (0.7)	3.2 (0.6)	3.0 (0.9)	3.1 (0.5)	3.1 (0.3)	3.4 (0.6)	3.2(0.6)	3.1 (0.5)	3.1 (0.6)
Feasibility	1.9 (0.4)	2.1 (0.5)	2.1 (0.5)	2 (0.4)	2 (0.4)*	2.1 (0.3)	1.8 (0.3)*	1.9 (0.4)*	2.2 (0.5)*
	Multicomponent (n = 13)a	Disease management (n = 21)	CT-print (n = 7)		Schools (n = 7)	GRT (n = 5)	Diseased (n = 11)
PRECIS	3.2 (0.6)	3.2 (0.5)	3.3 (0.4)		3 (0.6)	3.1 (0.4)	3.2 (0.5)
Feasibility	1.8 (0.3)	2.1 (0.4)	2.1 (0.2)		1.6 (0.2)*	2.2 (0.5)	2.2 (0.6)*
	Obesity/overweight (n = 19)	Survivorship (n = 3)	CT-web/email (n = 31)		Workplace (n = 20)
PRECIS	3.1 (0.5)	3.3 (0.7)	3.2 (0.5)		3.2 (0.6)
Feasibility	2.1 (0.4)	2.4 (1.1)	1.9 (0.4)		1.8 (0.4)
	Physical activity (n = 23)		Mobile phone/text (n = 7)
PRECIS	3.1 (0.4)		3.2 (0.4)
Feasibility	1.8 (0.3)		1.9 (0.5)
	Smoking (n = 18)		Multiple (n = 13)
PRECIS	2.9 (0.6)		3.1(0.5)
Feasibility	2 (0.3)		2 (0.4)
	Other (n = 10)		Web-based/email (n = 38)
PRECIS	3.2 (0.6)		3 (0.5)
Feasibility	2.3 (0.7)		1.9 (0.4)

IVR Interactive voice response, CT computer-tailored, GRT group randomized trial, Quasi quasiexperimental, RCT randomized-control trial, T1 efficacy studies, effectiveness, dissemination and implementation, and scale-up studies

^aStudy counted in multiple areas within study characteristic

*p = 0.05 (statistically significant difference within study characteristic)

RESULTS

Of the 113 studies, 23 (20 %) were physical activity interventions, 78 (69 %) focused on primary prevention, 69 (61 %) were delivered through web-based or computer-tailored (CT) web-based modalities, and 68 (60 %) were not tailored. The majority of the studies were implemented in community settings (51, 45 %), with at-risk populations (58, 51 %), conducted as randomized-controlled trials (99, 88 %) and as T1 studies (97, 86 %). More details about study characteristics are summarized in Table 3.

Each study was rated by two reviewers with reasonable reliability on almost all ratings for individual PRECIS and practical feasibility domains. Weighted percent agreement scores for PRECIS domains ranged from 63.9 to 78.5 %, with a median of 73.9 %, and for practical feasibility domains, these ranged from 63.7 to 84.7 %, with median of 78.9 %.

PRECIS scores

Average ratings by PRECIS domains were fairly uniform and ranged between 2.7 and 3.6 across all 10 PRECIS domains (Table 2). Lower domain average ratings (indicating less pragmatic approach) were observed on experimental intervention flexibility (2.8, SD = 0.9), practitioner expertise (experimental) (2.7, SD = 0.9), and practitioner expertise (comparison) (2.8, SD = 1), while participant adherence to protocol (3.6, SD = 1) and primary analysis (3.5, SD = 0.8) had higher mean scores. Composite mean scores varied by some study characteristics (Table 3). Noticeable differences were seen within study characteristics for intervention topic including multicomponent (3.2, SD = 0.5), cancer screening (3.2, SD = 0.6), and other (3.2, SD = 0.5) compared to smoking studies (2.9. SD = 0.6); eHealth modality including CT print (3.3, SD = 0.4) compared to web-based/email (3.0, SD = 0.5); and study design including quasiexperimental (3.4, SD = 0.6) and group-randomized trial (3.1, SD = 0.4). However, these differences were not statistically significant. Interestingly, a trend analysis showed a significant increase in composite mean scores within the experimental intervention flexibility domain across all publication years (p = 0.02). However, no differences were seen on PRECIS scores across publication years (2001–2005 vs. 2006–2010) or translation phase (T1 vs. T2+).

Practical feasibility scores

Average ratings for practical feasibility domains ranged from 1.5 to 2.8 using the same five-point scale as for the PRECIS ratings (Table 2). Lower domain average ratings (indicating lower levels of reporting or less pragmatic approach on practical characteristics) were observed on domains related to adaptation/change (1.5, SD = 0.6), program sustainability (1.6. SD = 0.7), and monetary cost of existing treatment [1.6, SD = 0.7)], while setting representativeness (2.8, SD = 1) and participant engagement (2.7, SD = 0.7) received higher average ratings. Composite means scores varied by study characteristics (Table 3). Noticeable differences by study characteristics were seen across intervention setting including schools (1.6, SD = 0.2) compared to both healthcare (2.1, SD = 0.5) and community (2, SD = 0.4), target population including interventions targeted to healthy (1.8, SD = 0.3) vs. diseased individuals (2.2, SD = 0.5), year of publication when comparing 2001–2005 (2.1, SD = 0.5) to 2006–2010 (1.9, SD = 0.4), and translational phase when comparing T1(1.9, SD = 0.4) to T2+ (2.2, SD = 0.5). These differences were all statistically significant (p = < 0.05). Additionally, a trend analysis revealed a significant decrease in composite mean scores within the intervention resource domain across all publication years (p = 0.05).

Overall, PRECIS composite mean scores across all studies were significantly higher than practical feasibility mean scores. In addition, the range of these scores by domain was consistently larger for PRECIS than for practical feasibility across all studies. Figure 1 uses two “spoke and wheel diagrams” to illustrate where PRECIS and practical feasibility domain ratings fell within the pragmatic–explanatory continuum for the highest and lowest scored studies on each scale [12].

Fig 1.

Pragmatic Explanatory Continuum Indicator Summary (PRECIS) and practical feasibility “spoke and wheel” diagrams: a PRECIS lowest versus highest scored studies*; b practical feasibility lowest versus highest scored studies *Illustrates lowest and highest scores of studies for which all domains where scored

DISCUSSION

To our knowledge, there have been no published systematic reviews using PRECIS review criteria or practical feasibility criteria to evaluate studies of EHIs. The purpose of this systematic review was to assess the extent to which studies of EHI in cancer control and prevention published between 2001 and 2010 utilized pragmatic trial design features and reported on issues related to translation, generalizability, and feasibility measures. Our analyses revealed two noteworthy findings. First, there was little variability in PRECIS scores across all studies. PRECIS overall composite mean score was 3.12 (domain range, 2.7–3.6) and is consistent with composite mean score of the original application of the PRECIS review criteria [16]. However, firm conclusions concerning a meaningful magnitude of score has not been feasible given the limited application of PRECIS within the systematic review literature. There were small, statistically nonsignificant differences of overall PRECIS composite mean scores by study characteristics including intervention topic, eHealth modality, and study design. No differences were observed for year of publication and translation phase. However, a trend analysis did reflect a significant increase in composite mean scores within the experimental intervention flexibility domain that may be reflective of the adaptation capabilities of eHealth technology. From prior reports of PRECIS scores in obesity interventions [15], we were surprised to see so little variability in PRECIS scores both across study characteristics and over time within a larger sample of intervention trials. This may be partly attributable to the fact that PRECIS domains specific to practitioner delivery were often not scored (i.e., missing data), since in many cases, EHI replaced the role of the practitioner. Additionally, the majority of studies reviewed (88 %) were conducted as randomized-controlled trials. One would expect such homogeneity of study design to affect average PRECIS scores towards a more explanatory end. However, randomized-controlled trials may not be appropriate for testing all aspects of EHI components, especially attributes such as implementation strategies, intervention reach, and multiple intervention settings, for which pragmatic trial features would be more appropriate [22, 23].

Second, studies consistently rated lower on practical feasibility scores than on PRECIS scores. This finding held true when comparing scores across study characteristics. Average ratings for practical feasibility domains (ranged from 1.5 to 2.8) were much closer to the explanatory or efficacy end of the rating scale than PRECIS domain ratings. This finding was due to the low frequency at which studies reported on factors related to costs, setting representativeness, adaptation flexibility, and program sustainability. This lack of reporting on practical feasibility measures and especially the relative absence of cost and setting representativeness has been reported in other content areas and is a considerable challenge to translation [24]. EHI trials seem to be the ideal context to address several of these measures. Furthermore, adaptation/change scored highly explanatory (average of 1.5), meaning that EHI studies rarely reported adapting an intervention during a trial. This notion assumes EHIs must be completed and static throughout the implementation period or at least indicates that no such adaptations were reported in the published articles. Static EHI contradicts common practice by eHealth industry and rapid learning approaches for which iterative testing of intervention components are integral to eHealth research and development [22, 25, 26]. Similarly, experimental methods such as Sequential Multiple Assignment Randomized Trial (SMART) and Multiphase Observation Strategy (MOST) systematically test components of EHIs, including implementation components, while addressing both rigor and relevance of EHI trials [27] that were seldom applied in these eHealth studies.

Our review has several limitations. We reviewed only the published literature between 2000 and 2010 and realize that more recent publications exist on EHIs in cancer control and prevention. However, we feel our current review spans an extensive EHI literature, and therefore, we do not expect these additional publications would change the conclusion of our current review. We also realize reporting constraints imposed by peer-reviewed journals may not be reflective of other aspects of EHIs, especially practical feasibility characteristics. Additionally, PRECIS and eHealth CONSORT criteria were developed post 2005, meaning that applying such criteria retrospectively may not be reflective of study design decisions in response to these criterions or, in some sense, “fair” to evaluate earlier investigations by these standards. However, our paper reviews a relatively large number of studies over a 10-year span and applies two innovative scoring frameworks to identify gaps in the reporting of key translational issues. We demonstrated that such frameworks could be productively and reliably applied to eHealth studies, including review procedures and inter-rater reliability.

CONCLUSION

Cancer eHealth interventions have made great, provocative use of cutting edge technology and are uniquely positioned to study a broader level intervention impact and test new behavior theories based on interactive technologies [28]. Despite vast interest in cancer eHealth and the applied nature of this field, our findings suggest that few studies used innovative designs to address key translation issues or reported transparently on issues central to dissemination. Given the surge of EHIs, health technology, and the lack of evidence-based interventions readily available to consumers [29, 30], there is a need for use of alternative pragmatic study designs, transparent reporting of external validity components to produce more rapid and generalizable results, and comparison of intervention effects assessed along the pragmatic–explanatory continuum by both PRECIS domains and practical feasibility criteria. We encourage investigators to utilize PRECIS and practical feasibility criteria used in this review to design, test, and evaluate EHIs in the future. Such research can lead to both interventions that work and that can be translated more rapidly into practice.