Build It, and Will They Come? Unexpected Findings from a Study on a Web-Based Intervention to Improve Colorectal Cancer Screening

Abstract

Given the extensive utilization of the Internet for health information, web-based health promotion interventions are widely perceived as an effective communication channel. This study was conducted to determine utilization of a web-based intervention intended to improve colorectal cancer screening in a population of women who are at average risk and non-compliant to current screening recommendations. The study was a randomized controlled trial designed to compare the effectiveness of colorectal cancer screening educational materials delivered via the Internet versus a printed format. In three years, 391 women seen for routine obstetrics/gynecology follow-up at two academic centers provided relevant survey information. Of these, 130 were randomized to the web intervention. Participants received voluntary access to a password protected, study specific web site that provided information about colorectal cancer and colorectal cancer screening options. The main outcome measures were self-reported and actual website utilization. Only 24.6% of women logged onto the website. Age was the only variable that differentiated users from non-users (p = .03). In contrast, 16% of participants self-reported web use. There was significant discordance between the veracity of actual and self-reported use (p = .004). Among true users, most (81%) logged on once only. These findings raise questions about how to increase utilization of important health communication interventions.

Worldwide internet usage is growing rapidly, increasing by 300% since 2008 (http://www.internetworldstats.com/stats.htm). In the United States, 3 of 4 adults report using the Internet, with more than half logging on to seek health information (Fox & Jones, 2009; Fox & Rainie, 2002; Hesse et al., 2005; Murray et al., 2003; Risk & Dzenowagis, 2001; Ybarra & Suman, 2006). The most current data from the Pew Internet and American Life Project suggest that 61% of adults look online for health information. The Internet is the third most cited source of information or assistance for dealing with a health or medical issue (Fox & Jones, 2009). These trends are not consistent for all adults. For example, a recent study suggests that nearly 70% of adults between 50 and 64 years of age go online, whereas about 38% of adults 65 and older go online, a significantly lower adoption rate than the general population (Fox, 2010). A descriptive study examining computer use among elderly populations (65 years and older) revealed that frustration, physical and mental limitations, mistrust, and time issues were barriers to use (Tak & Gatto, 2008). Given this expansive, growing use of the Internet among the general adult population, a host of health related commercial and research websites have been developed (Impicciatore, Pandolfini, Casella, & Bonati, 1997). Research has shown that the Internet may empower patients to improve their health behaviors and to take a more active role in their health care (Bass et al., 2006). Web-based educational interventions may represent an important way to educate relevant populations about critical health issues and to spur the uptake of recommended disease prevention behaviors. Target populations include not only those with an elevated risk for a specific disease, but those at average risk where prevention or early detection is of known value.

Because of the potential impact, a vast amount of research dollars are invested in the development of a portfolio of web-based interventions addressing a broad array of health issues. A recent (2009) search of the federal database of funded biomedical research projects (National Institutes of Health as well as multiple other Federal agencies) provided over 2,000 studies involving the Internet and the web for health promotion and/or health behavior change. Over 100 of these projects are in the cancer domain. Although these research efforts focus on the effectiveness of interventions in controlled settings, few report the actual uptake or utilization of the interventions, despite the availability of tracking software to validate actual usage (Evers, 2006). The issue of engagement, not only from the perspective of the health communication messages and approach, but from the initial step of taking action to “open” or access the web-based intervention, is critical to the ultimate test of effectiveness.

For the study reported here, we conducted a randomized, controlled, prospective trial comparing the effect of a web versus a printed letter intervention on colorectal cancer (CRC) screening adherence among women who have not completed colorectal screening in accordance to screening guidelines. Both interventional arms contained identical educational content, however the delivery channel and format differed; for example, the web version is purposefully more visually appealing and media rich. We hypothesized that the web intervention would, compared to print or control, represent a preferred communication channel and that participants exposed to that intervention would demonstrate greater uptake of colorectal cancer screening.

Our preliminary findings reported here highlight an emerging, and unanticipated issue in web-based health promotion interventions; namely, the underutilization of these interventions in “real world” settings. When this project was in development, we implicitly assumed, as did consultants and reviewers, that study participants would embrace the use of the web intervention as an easily accessible, highly engaging communication channel. However, results to date indicate surprisingly modest use of the web intervention, raising important questions about the utilization, effectiveness and impact of this type of health communication tool. A review of the current literature provides little insight, as actual web usage by study participants is often not reported, especially when web use occurs outside a directly observed research environment or in more natural settings. While unexpected, these findings regarding web usage patterns are important to more fully understand, as they could inform planning and implementation of web-based health interventions and encourage ongoing dialogue among health communication researchers regarding research methodology.

METHODS

Overview.

This NCI funded study, conducted by Fox Chase Cancer Center, was designed to test the impact of a print vs. web-based intervention to improve colorectal cancer screening among women seeking care through ob/gyn practices. Eligible women who consented to the study were randomized to either the control or one of the two intervention arms. Baseline and follow-up interviews were conducted with all study participants. A secondary goal of the study was to determine the usage of each of the interventions, through web-tracking software and self-report.

Eligibility.

Study participants for this IRB approved trial were drawn from Ob/Gyn practices at Geisinger Health System and Emory University. Both institutions have sophisticated electronic medical record (EMR) systems that integrate clinical scheduling with searchable clinical data repository capabilities. Using electronic screens reflecting the study’s eligibility requirements (below), we searched both EMRs for routine appointments 4-6 weeks in advance.

Eligibility criteria included being: 1) female; 2) 50 years of age or older; 3) at average risk for CRC defined as no personal history of colorectal polyps, CRC, inflammatory bowel disease, or CRC in more than one first-degree relative; 4) non-adherent with standard CRC screening recommendations at the time of index appointment (to be deemed non-adherent meant that all of the following were true: no at home fecal occult blood testing in the last 12 months, and no barium enema, flexible sigmoidoscopy or colonoscopy in the last 5 years); and 5) reported Internet access at home and/or work.

Participant Randomization.

Potentially eligible women were contacted initially by telephone. Upon contact, eligibility was confirmed and verbal consent obtained. All participants completed a baseline survey which included demographic and past medical history information and data for several accepted psychometric scales, (Radloff, 1977) anxiety, (Spielberger, 1983) and information seeking preferences (Miller, 1987). Participants were also queried about their knowledge and expectations regarding CRC and CRC screening as well as their beliefs about the risk of developing CRC and intention to participate in CRC screening (J.O. Prochaska, 1986; J. O. Prochaska & DiClemente, 1992; J. O. Prochaska & Velicer, 1997; Rakowski, 1999; Rakowski, Clark, & Ehrich, 1999).

After baseline data were collected, participants were randomized 1:1:1 to the usual care (control) arm or one of the two interventional arms (web and print). The web and print interventions were identical with regard to health-related informational content. Each contained essential information about CRC screening (i.e., rationale, description of recommended screening regimens and associated benefits/risks, sources of additional CRC related information). Message construction was based on information gleaned from our survey of average risk women (Weinberg, Turner, Wang, Myers, & Miller, 2004). Control participants completed the same baseline and follow-up telephone surveys (4 months and 12 months) as other participants; however, no additional CRC related information was provided.

Web-based Intervention.

The development of the web-based intervention included a systematic approach to pre-testing of messages, layout and usability testing as recommended by NCI’s Making Health Communications Work and www.usability.gov, the NCI-based website that includes human factors approaches and usability guidelines. Pre-testing was performed on approximately 50 women who fit eligibility criteria but were not included in the study. Based on our usability testing, we anticipated that participants could review the information on the site in about five minutes. In addition to educational material developed by the research team, hot links were provided for several carefully reviewed external websites devoted to CRC and CRC screening maintained by the American Cancer Society, National Cancer Institute, the Centers for Disease Control, the JAMA Patient Page, Medlineplus and the Colon Cancer Alliance. The website content was refined based on this formative research prior to study initiation to assure usability, medical accuracy and literacy level (7^th - 8^th grade reading level as per established guidelines for health communications).

As part of the initial recruitment, participants randomized to the web intervention were assisted over the telephone in developing a username/password at the completion of the baseline survey. Each received the study site URL and log-in instructions for the secure, private section of the Fox Chase extranet web portal system. With their unique username/password, these participants could privately view the portal site as often and for as long as they wished. Of note, a follow-up letter was mailed to these participants by standard post within 3 business days following the phone call. This letter contained the website use information and their username/password for later reference. Access to the web intervention was available from any computer with internet access.

Follow-up Telephone Survey.

Per study design, up to 15 attempts were made to contact participants by telephone 4-5 months after their index appointment. In addition to providing psychometric information similar to the baseline survey, participants were also asked about their receipt and use of any study materials, including web site use. Participants were asked first if they recalled receiving study related information about web access to CRC screening materials. Next, they were asked if they logged onto the study website. They were also queried about their perception of, and satisfaction with, the CRC screening information they received. For comparison, participants in the print intervention were also asked about their use of study materials at the 4 month telephone survey.

Web-based Intervention Assessment.

In addition to self-reported use, participant use of the web-based intervention was electronically tracked to determine if the intervention was accessed by the study participant and, if so, how often and how much time was spent viewing the information.

Website usage statistics and hyperlink tracking were accomplished with the NetTracker Professional software package. This package captures and stores longitudinal usage data at the level of the individual user. Importantly, the package functions as a server-only solution that does not rely on client licenses or hidden javascript downloads to the participant’s web browser. This capacity ensures accurate data collection by avoiding browser and user specific issues that might result from the use of differing web browsers, browser versions, and browser-computer or network security parameter settings.

Each time a participant clicked on a link within the website, a request was sent to the server. The server processed the request and recorded in a database the following information: the unique participant username; the page requested; current date/time of the request and the amount of time passed since they last clicked a link. From this information we determined which pages the participant viewed, how frequently and approximately how long they spent on each page. This security information also allowed us to monitor the number of times an individual logged in, the amount of time spent on the site, and the number and frequency of embedded hyperlinks to other CRC information sites used.

Statistical Methods.

Participants randomized to the web intervention arm who had completed their 4-month telephone survey were the primary focus of analysis. Self-reported web use was determined from responses to questions on the 4-month telephone survey. Actual web use for each participant, including web pages visited and time spent on each page, was tracked as described above. If a participant had visited the website for any length of time, she was classified as a web user.

The strength of the agreement between self-reported web use and actual web use was assessed with Kendall’s Tau-b, a non-parametric correlation statistic with values between −1 and + 1. The discordance between self-report and actual web use was tested using McNemar’s test. In an attempt to identify predictors of three endpoints, actual web use, self-reported web use, and accuracy in reporting web use, we selected numerous variables from the baseline questionnaires. The potential predictors included demographics, information seeking preferences (MBSS), risk related knowledge, expectancies/beliefs, and stage of change. A significance level of 0.05 was used to determine predictors of these endpoints. Differences were assessed using non-parametric tests including the Wilcoxon rank sum test for continuous variables and Fisher’s exact test for categorical variables. All analyses were done using SAS statistical software, version 9.1.

RESULTS

From June 2006 to August 2009, 740 women were enrolled in the overall study. Of this group, 391 had completed a 4-month follow-up survey. The randomization process assigned 170 of these women to the web arm. Due to a technical error, 35 of these women were not granted immediate access to the study website; therefore, they are not included in these analyses.

Of the 135 women able to log onto the web intervention at will, 5 provided non-interpretable answers to items on the 4 month telephone survey and were excluded from analysis (e.g. Question: “did you receive information about how to obtain CRC screening information on the web?” Response: “no”. Follow-up question: “Did you go to/use the website listed in the information you received?” Response: “yes”).

All subsequent results pertain to the remaining 130 women. The majority were white and between the ages of 50 and 59 years consistent with the predominance of participants from the rural Pennsylvania site (Table 1). Approximately 65% (n = 86) had at least some college education and nearly 75% (n = 96) were married. Nearly 70% (n = 88) had part or full-time employment. Of those willing to provide a report of income, 13% (n = 17) described an annual income of less or equal to $30,000, 24% (n = 31) between $30,000-$60,000 and 29% (n = 37) more than $60,000. Nearly one-third of participants declined to report their income.

Table 1.

Demographics of Web Use Arm Participants (n=130)

Variable	N (%)
Race
White	126 (96.9)
Non-White	3 (2.3)
Missing	1 (0.8)
Age
50-59	95 (73.1)
60-69	23 (17.7)
70-79	9 (6.9)
80-94	3 (2.3)
Education
<= HS	44 (33.9)
Some College/Voc	29 (22.3)
College Grad	57 (43.8)
Employment
Part Time	21 (16.2)
Disabled/Retired	33 (25.4)
Full Time	67 (51.5)
Student/Unemployed	9 (6.9)
Marital Status
Married/Cohab	96 (73.9)
Single/Div/Wid	33 (25.4)
Missing	1 (0.7)
Income
≤ $30K	17 (13.1)
$30K – 60K	31 (23.8)
> $60K	37 (28.5)
Did not answer	41 (31.5)
Missing	4 (3.1)

Tracked Web Use.

Of participants randomized to the web-based intervention (n = 130), 32 (24.6%) actually logged onto the website based on tracking data. For these 32 participants, the majority logged on only once (n=26). The amount of time spent on the site ranged from less than a minute to 22 minutes, with a median of 5 minutes. There were seven pages (one introductory page and six pages with CRC content) in the site, and the range of total requested pages (the total number of pages the participant visited, counting multiple visits per page if opened more than once) was one to eighteen pages, with a median of six pages. The most frequented pages were “Who Should Get Screened?”, “Screening Methods” and “What Can You Do?”. The final page of the site has a list of recommended websites for additional information on colon cancer screening. Twenty-eight percent (n=9) visited an external site. The two most frequently recorded sites were the NCI page on Colorectal Cancer Screening (n=5) and the Cancer.org page “Can colon and rectum cancer be prevented?” (n=4). A few participants linked to the JAMA site and the CDC fact sheets. One participant linked out to the Medlineplus site; no one linked to the Colon Cancer Alliance site. The time between consent to participate in the study and the first visit to the site ranged from 0 to 154 days, with a mean of 20. Twenty-five percent of those who used the site went on the same day as the telephone consent.

Self-reported web use.

We found discrepancies in self-report and actual usage in both directions. As shown in Table 2, the discordance between self-reported and actual website use was significant (p=0.004), with those who used the web less likely to self-report usage. In addition, 6% of those who did not use the web reported they did on the follow-up interview.

Table 2.

Actual Web Use and Self-Reported Use

Actual Web Visit and Self-report: N (%)
	No Web Used N (%)	Web Used N (%)	Total
Self-Reported no Web use	92 (94)	22 (69)	114 (88)
Self-Reported Web use	6 (6)	10 (31)	16 (12)
Total	98	32	130 (100.0)

p=.0.004

Predictors of Web Use. We examined differences among those who did and did not use the website.

The only significant demographic differences among those who used the website and those who did not was age, with users between the age of 50-59 years more likely to use the web than those 60 and older (p=.03). We did not find any significant relationships between information seeking preference scores, CRC knowledge, perceived CRC risk, intention to participate in CRC screening, attitudes about CRC screening nor perceptions about screening and disease related anxiety with actual or self-reported web use. In an effort to identify predictors of accurate web use reporting, the study population was divided into three groups: 1) actual web users; 2) those with a mismatch of self-reported web use and actual use; and 3) those whose self-report was consistent with actual use. There were no significant demographic factors that discriminated between these three groups including education or income.

Self-reported use of print materials.

For comparison purposes, we asked women randomized to the “print” intervention arm (n=171) if they had read the intervention materials. Approximately 25% reported they didn’t look at the materials, while 42% said they looked at it once, and more than 30% stated they reviewed the material at least twice.

DISCUSSION

Despite the common perception that health-related internet use is broad and increasing, only 1 in 4 women logged onto the website in this research study. Although this was not a controlled setting where all participants are automatically logged into the intervention, this is a significantly low uptake rate. In addition to the underutilization of the site, actual and reported web use was quite discordant. Nearly 40% reported using the website, but did not. Conversely, 20% of women who reported no web use in reality logged on. These findings of considerable web under-utilization are surprising. They call into question many assumptions about the real world impact of web resources for health promotion, especially patient-centered interventions that depend upon voluntary use.

Easy access to the highly interactive, multi-media driven web has created a perception that if we “build” inviting websites the public will “come”. Although it is reasonable to predict that women historically non-compliant with CRC screening will demonstrate a lower usage rate in a study about CRC screening, these same women fit demographic profiles typically associated with high health prevention utilization. Among this group of women, nearly 90% indicated that they accessed the Internet and used email and nearly 83% reported spending on average an hour a day on the Internet. Even younger participants, while more likely to log on than older women, still visited the web site less than 30% of the time. In addition, while by no means a guarantee of web use, all participants voluntarily agreed to a research project explicitly studying the Internet as a means of health promotion. Based on some of our preliminary data, it would seem reasonable to assume that the majority of participants would utilize the intervention website when asked to do so.

There is a growing body of literature to suggest that true web utilization is less than perceived. For example, a self-help web program for panic disorder reported a 99% non-usage rate (Farvolden, Denisoff, Selby, Bagby, & Rudy, 2005) and fewer than 1% of participants completed the 12-week program. Another study using a 5-module depression program, “Moodgym,” found that only 97 out of 19,607 (0.5%) participants spontaneously completed all 5 web modules. Despite the subsequent introduction of a more proactive, directive approach, only 22.5% completed the modules (Christensen, Griffiths, & Jorm, 2004; Christensen, Griffiths, Korten, Brittliffe, & Groves, 2004). Another study, focused on a physical activity website, showed only 76% of those randomized to the web-based intervention actually visited the site (Leslie, Marshall, Owen, & Bauman, 2005). A recent study (Silvestre, Sue, & Allen, 2009) reported the discrepancy between the number of persons who registered to use a secure health portal and subsequent usage. Fewer than 30% of those who registered accessed the site two or more times in a six month period. The authors did find that registration and usage increased over time, especially as new functionality was added to the site.

In contrast to measured use, self-report may misrepresent true use for a variety of reasons including evaluation apprehension, social desirability and cognitive limitations, as well as mere forgetfulness (Gosling, Vazire, Srivastava, & John, 2004). This is highlighted in our findings where almost 70% of those who used the website did not remember using it four months later. Descriptions of the impact of various health interventions, delivered through any channel, which require participant-initiated action, should be viewed cautiously. While not a specific goal of this study, it is reasonable to revisit the question of why many patient-oriented interventions, web, print or other, have had modest or no effect on improving health behaviors. In our study, nearly 75% of women randomized to the print (as opposed to the web) intervention claimed to have read the study materials at least once. Viewed in the context of the web tracking data, the veracity of these responses is uncertain. When interventions reported in the literature are found to have less positive effect than desired, it is unclear if this represents a failure of the intervention content or rather a lack of use of the intervention. Although the number of publications focused on web-based interventions is growing, few report actual usage data. Tracking as a means of corroborating Internet use is necessary when ultimately trying to design and evaluate how “effective” a web tool is for mainstream use. To date, no published systematic reviews use web-tracking as a criterion to judge methodological quality of Internet interventions relying on self-report. Study designs adequate to investigate these interventions in a variety of settings hinge upon realistic enrollment and follow-up rates.

If collected, tracking data would help to determine the real versus the expected use of these interventions. More needs to be learned about why some individuals are proactive and seek out the information on the internet, how individuals approach and process web-based information, how that information subsequently informs behavior, and what individual differences influence these outcomes. Drawbacks do exist however, with over reliance on tracking. For example, a lengthy duration on a particular webpage does not necessarily translate into an individual being actively engaged for that length of time.

Although it is clear that the Internet is a mainstream communication channel, we have much to learn about how it is actually used and by whom. In the realm of disease prevention, including cancer prevention, what are the most effective ways to increase the initial engagement with “e-health” tools? Typically, at home web utilization is voluntary and unsupervised and health promotion rather than disease treatment and management may be less compelling. Emerging technology provides a fertile ground for growth, but also presents new challenges. More research is needed to explore innovative ways to integrate health messages in other types of venues, such as social networking sites. Interventions may need to be marketed to increase usage, while more reliable navigational cues to direct viewers to higher quality sites are required to maximize positive impact. Our findings and others raise questions about the potential impact of web-based interventions to improve health behavior when targeted at healthy or at risk individuals, as opposed to those aimed at populations already affected by a specific disease where they may be more likely to seek information. Innovative marketing strategies may be needed to increase the salience and relevancy of these web-based interventions to those more recalcitrant individuals. Our findings show that 25% of those who did access the site, did so on the first day they were contacted. Future research could explore the benefit of more proactive approaches, such as telephone reminders or other cues to action, to increase usage among those who do not log on immediately.

Our unexpected findings and similar findings in other emerging research raise questions about research methodology, marketing of interventions, and additional questions for future research (see Appendix). Although our study was initially designed to answer questions regarding the effectiveness of the web as a health communication channel, our disappointing findings on utilization raise new questions about the challenges posed by web-based interventions. From the perspective of health communication research, investigators have been responding to increased pressure to provide more informed and interactive resources to patients (Kirsch & Lewis, 2004). Funding has significantly increased; in 2008 alone, the National Institutes of Health provided over 26 million dollars in funding for prevention projects that utilized the Internet (NIH Office of Extramural Research, 2009). It is not clear that when new interventions are built, target audiences will come.

This study has several important limitations. Most importantly, we assumed that web use would be widespread. In retrospect, greater attention to understanding barriers at the practical level, for example challenges related to unique password utilization to ensure privacy, or the speed of web connectivity, modem versus broadband, would have been useful. These and related issues should be considered in future research. In addition, there may be specific barriers related to colorectal screening, particularly in a historically non-compliant group that are not generalizable to other health domains.

Rather than simply continue to develop new technology-based interventions, reallocation of resources towards programs designed to understand how the Internet can best be exploited to improve health may have greater immediate impact. And for those currently funded research projects, more data regarding actual utilization should be reported to fully describe and elucidate this phenomenon. In addition, successful strategies to increase engagement and utilization should also be reported. For example, we added a reminder letter to encourage those in the web arm to log into the website. Of those who remembered the letter, 46% logged on in comparison to 12% of those who did not remember the letter. As in other industries where the initial goal is to market the website, these types of strategies may be required, especially for health promotion among less compliant populations. As these web-based interventions are used in community and practice-based settings, additional efforts will be required to ensure patients “log-in” and take the first step. Personalized letters from their physician and follow-up calls may be necessary (Toobert, Strycker, Glasgow, & Bagdade, 2002). Health promotion and disease prevention interventions have limited impact when left to passive diffusion. Therefore, documented active strategies and their outcomes need to be reported to expand our knowledge and arsenal of approaches to improve the uptake of these new and emerging web-based interventions in order to reduce future unexpected results.

Appendix: Questions about web based intervention research

Research Methodology

• How is actual usage tracked and reported?

• How many participants actually use the intervention?

• Since self-report of usage may be unreliable, is it a sufficient measure in web-based interventions?

• What is the expected non-usage to inform study design and sample size?

• How does non-usage vary in populations and interest in health topic or issue?

• Should all studies use tracking software and report both actual and self-reported usage?

Web-based Interventions

• Are those people less likely to participate in prevention behaviors also less likely to be engaged by e-health tools?

• What kinds of innovative approaches are needed to integrate health education and motivational messages into other uses of online information?

• Are web sites an appropriate way of reaching people for prevention and screening, including those who are non-compliant?

Research Questions

• Does the web only reach people already in the “action stage” of taking health protection steps, similar to past experience with “health fairs”?

• What research is needed to determine innovative ways to encourage and increase usage among print and web-based interventions, especially those that address more difficult health behaviors or are self-navigated?

• How do we consistently track usage of online interventions and reporting those data to generate new approaches and research questions?

• Are limited usage and over-reporting common problems across other online research interventions?