Online Research in Older Adults: Lessons Learned From Conducting an Online Randomized Controlled Trial

Abstract

The Internet has revolutionized healthcare delivery. With the increasing number of online users and the advancement of eHealth technologies, many healthcare studies have been conducted online. However, online research is still a relatively new field, and many methodological issues still need to be investigated. Over the years, the authors have conducted studies on various aspects of online health intervention research, including development and usability testing of online health interventions, web surveys, and an online randomized controlled trial employing older adult online users. The purpose of this article was to describe lessons learned from conducting an online randomized controlled trial designed to improve older adults' health behaviors focusing on methodological issues and strategies to overcome them.

Recent advances in information and communication technology (ICT) have transformed healthcare significantly and offer a great potential to improve the health of the public (Cronquist Christensen & Remler, 2007; Strehle & Shabde, 2006). Various technologies, such as the Internet and telemonitoring devices, have been used to provide up-to-date health information and to manage specific health problems (Kleinpell & Avitall, 2005; Napolitano et al., 2003). In particular, the popularity of the Internet has been exploding, with a penetration rate of 73% for American adults in 2006 (Madden, 2006). Although the increase has been slower than in younger age groups, 32% of adults age 65 and older use the Internet (Madden, 2006).

To many online users, the Internet serves as a resource for health information. In a 2006 survey of a national representative sample, half (51%) of individuals living with some kind of a disability or chronic disease were online users, and among those, approximately 86% had reviewed online health information (Fox, 2007). With the rapid growth of the Internet, various health studies have been conducted online (Curry, 2007; Steele, Mummery, & Dwyer, 2007). Most major methodological issues (e.g., sampling, attrition, etc.) in research are similar in both face-to-face and online settings. The addition of technologies in online trials, however, directly affects the entire research process, and must be considered for each phase of research, including sampling, protection of human subjects, development and implementation of interventions, recruitment, data collection (e.g., web survey), data management, as well as technical aspects such as software and hardware platform and confidentiality and security issues (Ahern, 2007; Dillman & Smyth, 2007; Nahm, Mills, & Resnick, 2004; Strecher, 2007).

Over the years, the authors have conducted studies on various aspects of online health intervention research. These studies have focused on health behavior change, including web surveys, the development and usability testing of online health interventions, and online randomized controlled clinical trials (RCT) employing older adult online users (Nahm et al., In Press; Nahm et al., 2004; Nahm, Resnick, & Covington, 2006). The purpose of this article is to describe lessons learned from conducting a theory-based online RCT designed to improve older adults' health behaviors focusing on methodological issues and strategies to overcome them.

Online Randomized Controlled Trials in Older Adults

Prior Online Health Behavior Trials

Although many online studies have been conducted in the area of health behavior change (Spittaels, De Bourdeaudhuij, Brug, & Vandelanotte, 2007; Strecher, 2007), only a small number used a randomized controlled design to evaluate the impact of Internet-delivered (mostly web-based) interventions on selected health behaviors (we define these as “online health RCTs”). Furthermore, only a few of them employed behavior change theories in the development and delivery of online interventions.(Lorig, Ritter, Laurent, & Plant, 2006; Patten et al., 2007) For instance, a Medline search of English articles published from 2004 to 2007 using the keywords “Internet,” “randomized controlled trial,” and “behavior” yielded 159 articles. Among those, after excluding editorial/commentarial articles as well as studies solely focused on healthcare providers, telemonitoring devices, cell phones, or e-mails, only 53 of the studies were online RCTs. Interventions in those online RCTs encompassed a range of technology components, including simple static web pages, interactive learning modules, video clips/films, e-mails, and/or online discussions. The intervention delivery methods (e.g., length, frequency, etc.) also varied. Only one-third (n = 16) of the studies used an online format for both data collection and intervention delivery. The majority of the studies used a face-to-face method for participant recruitment and data collection.

Most online RCTs have focused on younger adults, as only 23.1% (n = 12) of the studies we reviewed included adults age 50 and older, and only one study (Hageman, Walker, & Pullen, 2005) exclusively focused on older adults (age 50-69 years). Currently, older adults are one of the fastest growing online user groups. These individuals tend to have the most interest in health compared with other age groups, as older adults are more likely to live with chronic conditions and face other health problems.(Centers for Disease Control and Prevention, 2008) Thus, more studies focusing on this important subgroup are needed. Conducting online studies employing older adults, however, requires special considerations. In addition, the publications primarily focused on reporting the effects of online interventions and lacked information about specific methodological aspects, such as intervention development and its online delivery methods. Considering that online studies are still an evolving area, a body of the literature in this area must be further established.

A Social Cognitive Theory Based Online Trial for Older Adults

In our study, an interdisciplinary team of investigators developed a social cognitive theory-based, structured hip fracture prevention website (TSW) for older adults (age 55 and older) and assessed the feasibility of using this type of intervention (specific outcomes were reported elsewhere (Nahm et al., In Press). The TSW was comprised of structured learning modules and a moderated discussion board. Preliminary effects of the 2-week TSW intervention were tested as compared to those of a control group web intervention (CW) including only learning modules mainly comprised of links to existing health websites. Participants (N=245) recruited from two websites and a newspaper advertisement were randomized into the TSW and the CW conditions. Outcomes were assessed at baseline, end-of-treatment (EOT), and 3-month follow-up using online surveys.

Over the course of the study, our research team identified unique aspects of conducting social cognitive theory (SCT) based online RCTs. The next section discusses these aspects focusing on each phase of the trial.

Methodological Issues and Strategies in Online Trials

Online Recruitment

Specific demographic characteristics of most online users and the nature of online communication present unique methodological challenges in online studies.

Representativeness

The Internet affords several important strengths in promoting representativeness. It offers an excellent opportunity to recruit demographically and culturally diverse participants, as well as groups with particular characteristics and/or health conditions (Witham, Willard, Ryan-Woolly, & O'Dwyer, 2008). For instance, while it may take considerable time and resources to recruit individuals with rare illnesses face-to-face, the Internet can reach out to large numbers of people instantaneously at a fairly low cost.

Access to the Internet has recently increased across more diverse demographic groups (Fox, 2007; Madden, 2006). Online users, however, still tend to be more educated, of higher socioeconomic status, younger, and White compared to nonusers (Dillman & Smyth, 2007). This trend is even more prevalent in older adults (Fox, 2004). Such differences must be taken into consideration when the results are interpreted.

To alleviate the lack of representativeness in online study samples, a few strategies can be used. The first strategy is to employ diverse online settings (Ahern, 2007; Im, Chee, Tsai, Bender, & Lim, 2007). For instance, currently, most members of many older adult-focused web communities are White. In our study, more than 90% of the members of one recruitment site were White. Thus, we selected a second site that particularly focused on a minority population. Researchers also suggest inclusion of ethnic group-specific websites using quota sampling methods (Im et al., 2007). The second strategy is to use a mixed recruitment method, such as direct online recruitment (e.g., web page, eNewsletters) along with newspaper advertisement or standard postal-delivered mail (Dillman & Smyth, 2007). Participants recruited via different venues, however, may have different demographics and tendencies in using online interventions; thus information must be retained to permit evaluation of any differences in intervention or outcome performance attributable to recruitment methods.

Trust issues

Frequent press reports of Internet fraud and identity theft have made consumers very cautious about giving out personal information, such as their names and e-mail addresses (Horrigan, 2008b). This awareness may make potential participants, particularly older adults, reluctant to respond to online recruitment approaches. Thus it is important that online research projects demonstrate the credibility of the research site to potential participants. Conversely, researchers may have some doubts about the accuracy of demographic information provided by participants when the recruitment process occurs online (Lenert & Skoczen, 2002).

Several strategies are available to alleviate many of the issues concerning trust during online recruitment. The first strategy is related to selection of recruitment settings (e.g. websites or mailing lists) and the researchers' close collaboration with the owners of these sites. Participants of well-established websites (e.g., active organizational websites or healthcare groups' online portals) tend to trust the information from those sites. Participants know if they have doubts and/or questions about the study, they can consult the website owners who serve as gatekeepers and strive to protect their members. In our study, when we had the opportunity to communicate with participants via phone (e.g., help desk calls, etc.) or e-mail, several of them expressed their trust in and/or rapport with the web community in which the study was advertised. Additionally, identities of those web community members are likely to be more trustworthy than those of participants who found the research website randomly during an online search.

Second, provision of a toll free number and contact e-mail addresses of the investigators on the study website also help to build trust. In our study, several participants who reviewed the recruitment information on the web called the project team to confirm the legitimacy of the site and/or to ask specific questions. In addition, some researchers use telephone follow-up strategies to validate participants' identity (Ahern, 2007; Lenert & Skoczen, 2002). In our project, there was often a period of time between consenting and intervention initiation. Within 48 hours from the online sign-up, we contacted participants to provide them with more specific information about the intervention (e.g., start date). During this time, we also confirmed their basic demographic information (e.g., address, age, etc.). If an online study uses follow-up phone calls, a question regarding the participant's preferred time of day to receive calls must be asked.

Recruitment method (use of eNewsletters)

Website announcements of research participation opportunities may not be sufficiently robust to attract participants' attention. With the popularity of the Internet and e-mail, eNewsletters have become a powerful communication medium and have been used to deliver health information (Richards, Kattelmann, & Ren, 2006). These eNewsletters can be a very useful recruitment tool. Compared to printed versions of newsletters, eNewsletters with embedded hyperlinks serve as portals to much larger amounts of information. When users read an eNewsletter, they can conveniently click on the link to the study website and read about the study. In fact, once online users are familiar with certain websites, they may not regularly visit the same sites and just review eNewsletters for new information. In our study, we noted that the release of eNewsletters with hyperlinks to our study resulted in a marked increase in participant enrollment.

Informed Consent

Online research is relatively new to the health and behavioral sciences, and application of conventional informed consent processes to this environment also warrants further investigation (Ahern, 2007). Although ethical issues in online research are similar to those in face-to-face studies, special concerns arise in online studies due to the unique nature of the online population and recruitment processes. In particular, important demographic information, such as age (e.g., children vs. adults) and gender, are not transparent in online studies. Thus, researchers may need to incorporate specific procedures (e.g., telephone follow-up) to ensure the accuracy of such information (Ahern, 2007).

Over the last several years, many Institutional Review Boards (IRBs) have established procedures for obtaining informed consent for online studies. Depending on the purpose and design of the study and the particular IRB, procedures may vary. Researchers need to work with their own IRBs to ensure that ethical considerations to protect subjects are properly met. Generally, however, it is critical to ensure that participants have the opportunity to ask questions about the study via telephone or e-mail communication options (Ahern, 2007). Some IRBs recommend including questions that can demonstrate participants' understanding of the information on the consent form as a part of the online consent process. Most IRBs request some evidence that network pathways are secure and that data are encrypted to prevent intrusion from unwanted parties during data transmission.

Retention of Participants

As compared to face-to-face trials (e.g., drug trials, exercise trials with physical therapists), online intervention trials rely more on the participant's own intention to use the program, and it is easier for them to opt not to use the online intervention (Ahern, 2007; Eysenbach, 2006). High attrition rates have been addressed as a particular issue in online studies (Ahern, 2007; Eysenbach, 2006). The rates, however, varied across studies, and a few trials including older adults showed relatively low attrition rates (Lorig et al., 2006; Napolitano et al., 2003). For instance, an online physical activity study using a website in combination with 12 weekly e-mail tip sheets (N = 65; mean age, 43; range, 18-65) reported a 20% attrition rate at 3 months. In another study using a 6-week web-based chronic disease self-management program (N = 958; mean age, 57; range, 22-89), the attrition rate at 12 months was 22.5% (Lorig et al., 2006).

Our study with a mean age of 69.3 years (range, 55-92) resulted in an 11.8% attrition rate at 3 months. This rate is considerably lower than in other studies, and even more significant considering the age of the older adults participating in this study. We found a number of strategies helpful in achieving participant retention. First, our website was specifically developed to be older adult friendly (Nahm et al., 2006; The National Institute on Aging and the National Library of Medicine, 2002). For example, use of an easy navigation method and large fonts prevented some potential challenges for older adults who may lack Internet skills and be facing changes in vision and memory functions (Nahm et al., 2006).

Second, availability of a toll-free number for questions and technical support was helpful to many of our older adult participants. Although there is some controversy over the use of phone contacts in online studies (Patten et al., 2007), our study showed that it is especially helpful to many older adults. For instance, common challenges reported by participants were Internet and e-mail setups installed by the Internet service provider's (ISP) program. Our participants used different ISPs, and each ISP employed unique, but ever changing, configurations to log on to the Internet and maintain web security. Some of our participants found that their original configurations (for both computers and the Internet) prevented the opening of certain web pages and/or blocked or redirected e-mail messages from the study team to the participants' junk e-mail folder. Most of these issues were easily resolved by the support desk via simple telephone calls. The issue with participants' specific e-mail set-ups (Strecher, 2007) highlights the importance of confirming electronic communications via phone if the project team does not receive a response from participants.

Maintaining contact with participants and responding promptly to their queries also assisted retention. For instance, when participants signed up for our project, the instructions on the web stated that the project manager would contact participants within 48 hours to provide them with further instructions about the project, and we adhered to that timeline. During the 3-month waiting period, a monthly e-mail was sent to participants to let them know about the upcoming survey dates, as well as to deliver a personal “thank you” for participating in the study. This e-mail notification also served as a tool to confirm the participants' current e-mail addresses. We received several responses informing us of participants' upcoming changes in e-mail addresses or travel schedules.

Implementation of the Online Social Cognitive Theory Based Intervention

General logistical aspects of implementation

Although the Internet currently affords highly sophisticated technologies, it is critical that researchers select technology set-ups that are accessible to most of the target population and that the program is user friendly. For instance, prior findings suggest that many older adults still use a modem to access the Internet (Horrigan, 2008a; Nahm et al., 2006). Thus, when we needed to use video clips, we made an effort to use short segments with transcripts. In our online RCT, 23.7% (n = 51) of participants used modem access, and 69.2% (n = 148) used either cable or digital subscriber line (DSL) access. We also tried to offer options for technologies, such as the use of both PDF and rich text format documents.

Generally, participants in online trials have few opportunities to ask questions to the research staff. Thus, it is important that researchers anticipate potential questions and proactively provide sufficient information in appropriate web sections. For instance, although participants were apprised at the beginning of the study that approximately 10 individuals would follow the program together as a group, by the time they began using the online discussion board, this information was often forgotten, and a few expected to see responses from the entire study sample on the discussion board. Reminding participants of the specific group size again at the beginning of the online discussion would have mitigated this confusion.

Researchers must also take some unexpected situations into consideration in the implementation of online health interventions, including changes in e-mail addresses, power outages, and travel. Older adults in particular seem to have more frequent hospitalizations and seasonal travels than younger adults. Availability of telephone communication has been particularly helpful in those cases.

Social learning environment using group-based intervention

In changing one's behavior, the Social Cognitive Theory (SCT) emphasizes goal setting, motivation, self-efficacy, and outcome expectations (Bandura, 1998). To apply this theory to an intervention, a group approach can be effective because group members can motivate each other toward achieving their goals.

In our study, a virtual social learning environment was formulated using the discussion board and the group approach. Once participants signed up for our study, they were placed on a waiting list until there were approximately 22 to 24 individuals available, at which point we contacted the first 20 and administered the baseline data collection (one week was given). Upon completion, these individuals were randomized into two groups (each group with approximately 10 participants) and then started the intervention (either TSW or CW) as a group. Learning modules were assigned for each week, and specific behavior-oriented discussion questions associated with those modules were posted weekly. This approach worked very well, as evidenced by participants' qualitative comments about their experiences with the discussion board (Nahm, Resnick, & Brothemakle, 2009). Many participants reported that they enjoyed reading others' postings and that the discussion board helped their learning, as well as served as a barometer for their health behaviors. Group members also offered positive comments to each other.

Moderator's role

In moderated discussion boards, the moderator's role is particularly important to engage participants in the discussions and to direct the discussions toward the goal of the discussion board. In a report on computer-based support for women managing a long-term illness, Cudney and Weinert (2000) explained that the presence of the nurse monitor was helpful in maintaining structure in the discussions and participants' sense of security about the discussions. The findings from our study also showed consistent results. In particular, the moderator of our study facilitated discussions following SCT, offering encouraging comments to participants and inviting others' comments. Several participants specifically reported that positive experience with the moderator. There is, however, a paucity of research on the moderator's role in online intervention studies, and further research is needed.

Treatment fidelity

As in any trial, treatment fidelity is critical to assure that the intervention is implemented as intended (Bellg et al., 2004; Nguyen, Cuenco, Wolpin, Benditt, & Carrieri-Kohlman, 2007). Challenges to the evaluation of treatment fidelity in online studies are different than those encountered during face-to-face settings (Ahern, 2007). Prior studies have incorporated specific monitoring devices, such as activity monitors (van den Berg, Schoones, & Vliet Vlieland, 2007); however, using self-report of registered values is still an issue. Another option is using self-monitoring devices of which values can be automatically downloaded, but the high costs of this method is often a hindrance.

Many online studies monitored participants' website usage either by using simple login and logout times or using various web tracking programs (Brennan, Casper, Kossman, & Burke, 2007). In our study, we used a web tracking program (WebTrends Inc., 2009) because this type of program provides researchers with more robust information (e.g., frequently visited web pages, time spent on each section, etc.). Several web tracking programs are available commercially or without charge. Each tracking program offers different types of reports, and researchers must carefully review those reports before selecting a tracking program. While evaluating participants' online activities, researchers must take into consideration Internet traffic and idle times (Choi, Moon, Cruz, Zhang, & Diot, 2007). Additionally, the amount of time participants were logged into an online program may not accurately reflect the amount of intervention exposure because the participants could have been engaged in other activities while they were logged on.

Web-Based Data Collection

Web-based data collection method (web surveys) offers many advantages, as well as methodological and technical issues (Couper, 2007; Dillman & Smyth, 2007; Nahm et al., 2004). Some major advantages include convenience and time and cost efficiency. Methodological issues include reliability and validity of data and ethical issues (Couper, 2007; Dillman & Smyth, 2007; Nahm et al., 2004).

It is, however, important to recognize that advantages and issues of web surveys are relative appraisals based on the scope of the studies and available technologies (Couper, 2007). For instance, some vendors offer free online survey programs. The number of items and functionalities of these surveys, however, are quite limited. Furthermore, for studies including individuals' health data, researchers must ensure the security and privacy of the data submitted from the participants (Nahm et al., 2004; U.S. Department of Human and Human Services, 2008). Data storage servers must be located in a locked secure place and protected by network firewalls. During data transmission, the data must be encrypted to avoid unwanted intrusion. Thus, researchers in these trials tend to work with web developers to either develop a web survey or to use a high quality commercial program, either of which could be quite costly.

The investigators and web developers must carefully consider the structure of the database that will store the data (Nahm et al., 2004). This is especially important in longitudinal studies with multiple data collection points because eventually the data stored in the databases will need to be exported to a statistical program for planned analyses. The researchers must ensure that the structure of the exported data are workable (e.g., for multiple data points, addition of rows vs. columns).

For successful online survey implementation, collaboration between researchers and web developers is vital. For researchers to be able to work effectively with the web developer, they must have appropriate knowledge about web technology and web usability (Nahm et al., 2004). For instance, web developers may be familiar with developing websites for business but not for research projects. The research team therefore may need to offer its Web development staff brief inservices on important aspects of conducting research.

Compared to most web surveys we often encounter while surfing the web for information, the length of surveys in most large scale RCTs tends to be long. There has been a lack of online RCTs in older adults, and we were concerned about whether older adults could complete a lengthy online survey. Although most items were either simple check-off or multiple-choice options, the maximum number of items in our baseline survey was 121 (the follow-up survey was shorter than the baseline survey). The findings from our user testing of 10 older adults, however, showed that they did not have much difficulty completing a survey of this length, usually taking 20 to 30 minutes. For the trial, we offered the participants the opportunity to complete the survey in a single sitting or multiple sittings. Most participants, however, completed the survey in one sitting and required little support. Only a few completed it in two sittings, and most of those individuals forgot to complete the survey at a later time and needed reminder e-mails or calls.

Further, we noted that the quality of the survey data entered by older adults was excellent. When the data were screened, we did not find any evidence of invalid data patterns (e.g., all “0”s or “1”s), and missing data were less than 2% throughout surveys. Overall findings from our study showed that online surveys can be successfully used by older adults.

Conclusion

In this article, our research team reported specific lessons learned from conducting an SCT-based online RCT. Online health intervention research is a still an evolving field with a great potential for future research. Although similar principles of research methods (e.g., sampling, recruitment, implementation of interventions, data collection, and ethical issues) in face-to-face settings can be applied to online research, execution of these methods requires distinctly different approaches. Furthermore, older adults may respond differently to online studies than their younger counterparts. Considering the rapid growth of older adult online users and their interest in health, these individuals must be actively included in online health trials.

One of the most important differences between face-to-face and online trials is the addition of technologies that can directly affect entire aspects of the research. Thus, technology developers and researchers must work as a harmonized team, and each party must have sufficient knowledge about the other. In addition, the research team must have sufficient knowledge of the technological infrastructure used by the majority of the target sample. Older adults tend to use less up-to-date technologies than younger adults, and their interaction with online programs may differ. While many online studies have been conducted, most publications (except online surveys) have focused on reporting outcomes rather than the process of conducting these studies themselves. To advance online research and to replicate studies, a more extensive body of knowledge in online research methods needs to be accumulated.