Methodologic and Design Issues in Patient-Centered e-Health Research

Introduction

Over the past decade, dozens of studies have demonstrated the impact of e-health interventions across a variety of health domains.^1–6 These include targeted and tailored interventions delivered through various channels such as the web, print, CD/DVD, and more recently PDAs and cell phones. These mostly positive findings have fueled considerable growth in e-health program development. Such programs have tremendous potential for public health impact as they can reach large numbers of participants at a relatively low cost.⁷

Whereas the first generation of patient-centered e-health studies have focused largely on answering the question of whether such programs are efficacious; it is proposed that the next generation of e-health research begin to better address the questions of why, how, and for whom they work.^5,8 This brief article discusses some design and analytic approaches for determining what components of e-health programming work (i.e., active ingredients), how they worked (i.e., mechanisms of action), and for whom (i.e., moderator effects).

Did It Work? What Worked?

Let us begin with an example of a hypothetic e-health Internet-based program designed to help patients quit smoking. The program uses three primary motivational components or factors: outcome expectation messages focusing on reasons for quitting and consequences of continued smoking (Factor A); efficacy expectation messages addressing confidence and barriers to quitting (Factor B); and a narrative testimonial of a patient's specific reasons for, and methods of, quitting (Factor C). This example is a simplified version of an e-health intervention recently developed and evaluated by a number of the authors.⁹

Based on prior empirical studies and theory, factors a, b, and c are all considered important to the outcome of smoking cessation (Y). Most existing e-health interventions have typically examined the aggregate effect of these factors (denoted as abc) in comparison to a control treatment (denoted as 0). Yet, an obvious limitation of such a design lies in the inability to tease out specific contribution of each intervention factor or possible interaction of factors.

The doubly randomized preference trial represents a research design where patients are randomly assigned to a “randomized arm” (where treatments are randomized) and a “choice arm” where patients choose their treatment combination to potentially disentangle intervention factors. However, this approach can yield unmatched control group members with differential exposure to message dose and content. Other options include additive and factorial designs. For example, in the factorial design, the sample of subjects can be randomly divided into eight equal sized groups, and then assign one group to each of the eight treatment combinations, 0, A, B, C, AB, AC, BC, and ABC. Three-way ANOVA can be applied to assess the effects of each factor and their potential interactions. The random allocation to the eight groups ensures that the comparisons among the various ingredients have a valid causal interpretation. Factorial experiments are useful in assessing the main effects of the individual factors A, B, and C, as well as all two- and higher-way interactions.

Full factorial experiments have several drawbacks. A practical difficulty is that the number of treatment combinations (i.e., cells) increases rapidly with the number of factors. For example, if two additional factors d and e could be added to the three-component smoking study, 32 treatment combinations would be required, even with just two levels (e.g., presence/absence) of each factor. A second issue is that there are a large number of statistical comparisons, increasing the number of Type I errors. Additionally, they are less efficient in comparing any two particular treatment combinations, say, ABC against 0. For example, in the original “kitchen sink” approach, ABC and 0 are each assigned ½ of the subjects while only 1/8 of the subjects are assigned in the full factorial experiment with 8 combinations. This reduces the power in comparing specific treatment combinations. In other words, a larger sample size is needed in order to simultaneously assess the existence of an overall effect while also identifying which ingredients and their combinations are most effective.

To address these and other issues, the current research group has begun to use fractional factorial designs to systematically dismantle active ingredients of e-health interventions. These designs were used within a sequential framework which identifies the intervention effects and optimizes their levels before comparing the “optimal” treatment combination against some reference group. Collins et al.¹⁰ call this a multiphase optimization strategy or MOST. This strategy is adapted from a similar framework that has been successfully used in engineering applications for many years (10) and consists of three phases involving separate randomized trials:

Phase I. Screening

The goal in this phase is to efficiently “screen” a larger set of potentially important treatment components and identify those with promise. Two-level fractional factorial designs, judiciously chosen to estimate selected higher-order interactions of interest, allow a reduction in the number of experimental groups required. The Pareto principle underlies the screening phase— identifying a relatively small number of active treatment components that account for the greatest variance in the outcome. Since the goal here is to identify potentially important effects, liberal levels were used for significance testing and less consideration is given to multiple comparisons or spurious effects.

Phase II. Refining

This phase aims to refine current understanding of the effects of the “candidate” components that are identified in Phase I. This is done using follow-up experiments to untangle important effects that may have been “aliased” in Phase I (i.e., groups of intervention combinations that were not isolated) and to determine optimal “dosage” levels of factors. Phase II results lead to the formulation of an optimal combination of treatment components which can be tested in a fully powered Phase III trial.

Phase III. Confirming

The final phase is a confirmation trial to examine the efficacy of the aggregate program built in Phases I and II. While this phase is similar to classic RCTs with two or more arms, the multiphase approach leads to the inclusion of only important components at their optimized levels.

To date, this approach has been successfully implemented in two e-health studies: Project Quit to study the active components of a smoking cessation program⁹ and Guide to Decide to study decision aids among women at high risk for breast cancer.¹¹ Since most behavioral and decision-making interventions in clinical settings rightfully involve multiple treatment components, the MOST approach can serve as an efficient method for “unpacking” the efficacy of each component, thereby maximizing the impact of the overall intervention. A discussion of the use of fractional factorial designs in this context is available elsewhere.¹²

Limitations of fractional factorial designs

In designing fractional studies, investigators are required to identify, a priori, the key interactions that are likely to be ‘active’. This leads to the aliasing of various other combinations of intervention components, so these interaction effects cannot be isolated. The decisions on which interactions are active and which are inert, while ideally based on prior empirical studies, are often instead based on theoretic assumptions as well as research priorities delineated by the investigators. Potentially important interactions that might not have been anticipated may be excluded from the model, so it is possible to miss small to moderate effects.

Similarly, given that the effect sizes of interactions are rarely known, power estimates are largely hypothetic. This can lead to either underpowered or overpowered studies. Finally, similar to full factorial designs, fractional approaches also involve multiple comparisons and the possibility of Type I error. These limitations can be offset in part by the multiphase confirmatory approach used in the MOST framework whereby even borderline significant effects identified in Phase II are tested again with a new sample and intervention combinations omitted in Phase I can be tested in Phase II.

How Did It Work?

Program use data (paradata)

One of the powerful features of e-health interventions is that they lend themselves to highly specified process analyses. In nonelectronic interventions, process measurements are often limited to constructs such as self-reported use, and user satisfaction (e.g., “Did you read all of the materials?”); however, in most e-health applications, more granular-level details regarding what a participant did and how they interacted with the materials can be ascertained. These data on intervention use, which elsewhere have been called “paradata”^13,14, can be used to help identify on the micro-level which elements of a e-health intervention are used, and to identify participants who were more or less engaged with the materials. This information can then be used to guide the development and refinement of the intervention, subject to the caveat that paradata is observational and subject to the limitations of observational data for making causal inferences.

Several types of paradata can be captured from online interventions. For example in terms of technical specifications, information can be gathered, such as participant browser type, connection speed, and available plug-ins, that may be useful in evaluating feasibility and usability issues. In terms of intervention dose and exposure, information can be obtained regarding number of log-ins, which pages are viewed and for how long, which links are clicked, which applications are launched, and so on. At a finer level of detail, client-side scripts¹⁴ can be run to measure actions within a web page, such as clicking on different objects, scrolling, order of selection of survey response options, among other actions.

In addition to elucidating the consumer uptake of intervention components these data can serve as indicators of engagement which can then be used in “dose–response” analyses.¹⁵ As Danaher et al.¹⁶ note, “a key ingredient in determining the impact of any web-based behavior change program is the extent to which participants are exposed to the program.” Such measures are useful mediators in understanding the effectiveness of online programs. For example, the HMO-based Project Quit program consisted of 5 sections (The program included: an introduction section, a section focusing on outcome expectations, 2 sections focusing on efficacy expectations, and a section with a narrative success story); it was found that each additional section opened by the participant contributed to an 18% higher likelihood of quitting, on average.¹⁷

Message processing data

One method of better understanding how participants process e-health information is to study how they read and attend to the materials. Recent advances in biometrics have expanded the ability to objectively measure, in real time, how materials and messages are processed. For example, tracking eye movements is commonly used in the fields of psychology, engineering, and marketing. People generally have limited awareness and control of how their eyes move under normal viewing conditions. More importantly, eye movements reflect real-time allocation of attention. Eye tracking collects information about patterns of eye movements by providing information about where an individual looks and the duration of their fixations. Fixation positions are important because regions that fall in the eye's fovea are likely to be encoded in greater detail than peripheral regions. Information that is encoded in greater detail is more likely to be remembered later. Some eye tracking devices also collect information about blinks as well as pupil dilation.

Recently, the research group for the current study completed a study examining how testimonials are read as a function of testimonial images that appeared with them.¹⁸ Cigarette smokers were asked to read testimonials of ex-smokers that were paired with images either matched to the participant's age, race, and gender, mismatched to the participant, or neutral with respect to the match. Participants reported that they found the testimonial messages more persuasive when it was paired with a matched image. Moreover, eye-tracking data showed that the smokers viewed the matched image more times, and made more fixations between the matched image and text as if connecting the text and the image. Thus the eye movements appeared to support the persuasion data.

Eye tracking is also useful for testing hypotheses.¹⁹ For instance, suppose a patient is given information about calculating BMI as well as messages about eating fruits. The patient remembers 1 month later reading about the BMI calculation but not the fruit-related messages. In situations like this, it is unclear whether the memory difference was due to a reporting bias or to differences in perception, encoding, or memory consolidation. Eye movement data could potentially distinguish among these hypotheses. For example, if the patient made more eye fixations to BMI calculation than to information about fruits, then the memory difference was probably driven by attention/encoding differences. If the number and duration of fixations were the same in the two conditions, then it is likely that the memory difference was driven by processes that occur after information encoding.

Another promising method for understanding how e-health messages achieve their effects is through functional magnetic resonance or fMRI. fMRI has existed for over 1 decade and has been increasingly utilized in investigating neural and cognitive mechanisms underlying human behavior. This technique can be used to study how e-health messages are processed by patients and how these messages might influence health-related behavior. fMRI investigates neural activity non-invasively and with relatively good spatial and temporal resolution. Participants lie in an MRI scanner and perform tasks while the scanner collects images of blood-oxygen level– dependent (BOLD) signals throughout the brain every few seconds. Because blood-oxygen levels are strongly correlated with neural activity, fMRI can identify brain areas in which neural activity changes as independent variables are changed.

Better understanding patients' neural processing of e-health messages can be used as converging evidence to support existing e-health findings and related theories. Recent research demonstrates that personal relevance appears to partially mediate the effect of tailored messages in smoking abstinence.^17,20 A recent study examined how smokers process high- versus low-depth tailored smoking cessation messages.¹⁸ It was found that smokers activated areas of the brain related to self-relevance processing more when they were processing high-depth tailored messages. Neuroimaging studies can also offer insights into how individuals process information. For example, if other studies find evidence that patients activate a region of the brain when they are engaged in reward processing, and the same area of the brain is activated when a patient is reading a gain-framed motivational health message, it suggests that reward processing may be involved in processing that type of message. Finally, neuroimaging studies are useful in understanding differential cognitive processing based on characteristics of the patient. These characteristics could be genetic, physiologic, behavioral, or psychosocial and could be the basis for new factors used in message tailoring, which is discussed in the next section.

For Whom Did It Work and Why?

Whereas the previous sections have focused on methodologies for determining the effect of specific intervention factors and mechanisms by which they work, it is also important to determine for whom they work and why. One of the most powerful advantages of e-health interventions is the ability to tailor messages and programs to individual characteristics.⁵ Traditionally this task falls under the rubric of moderator analyses (e.g., examining baseline participant characteristics that interact with treatment condition). Typically, in RCTs, moderators include variables such as race, gender, age, and possibly health indicators. However, in e-health interventions, moderators extend beyond these sociodemographic variables. This is in part enabled by the fact that in order to delivery highly tailored interventions, detailed assessments of demographic and psychosocial variables are required. Several tailored and e-health programs^21–23 (including smoking interventions^1,8,17,20,24) have been shown to interact with a wide range of personality and motivational variables.

One interpretation of these complex interaction patterns is that individual response to e-health interventions may be “sensitive to initial conditions^25,26.” That is, small initial individual differences (i.e., participant characteristics) may generate large differences in intervention response. If so, this has significant implications for the design, analysis, and replicability of such programs. Although interventions are generally evaluated by comparing overall group differences to a comparison/control group, there may be multiple response patterns that have to be elucidated to fully understand why the intervention worked and for whom. For example, spontaneous quitting may yield more sustained cessation than planned quit attempts²⁷and transformative motivation may be more effective than rational motivation.²⁸

In addition to executing standard interaction effects, other techniques are available to identify individuals and subgroups of individuals for whom an e-health intervention may be particularly effective.²⁹ One technique is Signal Detection Methodology (SDM).^29–32 This nonparametric method, which uses Receiver Operator Curves (ROC) and recursive partitioning, identifies homogeneous subgroups (with distinct sets of characteristics) who are higher versus lower on an outcome parameter (e.g., success with a smoking cessation e-health intervention). In contrast to standard interaction effects which tend to look at one or two levels, SDM can convey higher-order interactions, providing a potentially richer characterization of those for whom an intervention was successful and not. Incorporating a broader range of traditional moderator variables, as well as paradata, in SDM analyses, can enable e-health researchers to identify for whom interventions are more and less effective and ultimately to create more effectively tailored interventions.

The more frequent interactions patients can have with e-health programs, often entailing repeated assessment of psychosocial and behavioral processes (occasionally in real time), lend themselves to more highly powered and granular analyses of why behavior change may occur. Thus, the ability to examine potential mediators or mechanisms of behavior change can be significantly enhanced with fine-grained, time-stamped data obtained in e-health programs.³³ Taken together, through both designing e-health interventions to assess theoretic mechanisms and examine paradata, researchers can develop a better understanding of the “black box” of why and how these programs work.

Conclusion

This article proposes some novel design and analytic approaches to determining how, why, and for whom aggregate and subcomponents of patient-centered e-health program work. Each of the approaches to addressing the questions described in this article has been used by the current research group as well as by a number of others.⁸ A better understanding of what is inside the black box of e-health interventions will lead to more empirically informed and ultimately more effective programs. Furthermore, innovative analytic techniques should be employed to determine for whom e-health interventions are most effective. The refinement of current and future e-health interventions promises to increase the effectiveness of these interventions for specific populations and future research employing the analytic strategies discussed above is warranted to examine this possibility.