Monetary Incentives Improve Recall of Research Consent Information: A Randomized Pilot Study

Abstract

Research participants often fail to recall substantial amounts of informed consent information after delays of only a few days. Numerous interventions have proven effective at improving consent recall; however, virtually all have focused on compensating for potential cognitive deficits and have ignored motivational factors. In this pilot study, we randomly assigned 31 drug court clients participating in a clinical research trial to a standard consent procedure or to the same procedure plus incentives for correctly recalling consent information. The incentive group was told they would receive $5 for each of the 15 consent items they could answer correctly 1-week later. At the follow-up, the incentive group recalled a significantly greater percentage of consent information overall than the standard group (65% vs. 42%; p < .01). Similar findings were observed for specific categories of consent information, including study purpose and design, risks and benefits, and human subject protections. Effect sizes were all large (d = 0.89 to 1.25). Findings suggest that motivation plays a key role in recall of consent information and should be considered in the development of future interventions.

Introduction

Participants in clinical research studies often fail to recall much of the information presented during the informed consent process after delay intervals of only a few days. A recent review article concluded that most standard informed consent procedures elicited recall scores below 60% on post-consent quizzes and few trials reported recall scores exceeding 75% (Flory and Emanuel, 2004). These findings are particularly disturbing considering that many of the trials measured recall shortly after the informed consent procedures were completed.

Importantly, the failure to recall consent information is not restricted to minute details in the consent document or technical aspects of the study protocol. In some studies, research participants were unaware they were part of a research study, failed to recall study-related risks, were unable to describe randomization procedures or placebo interventions, and were unaware they could withdraw from the study without negative consequences (e.g., Edwards et al., 1998; Festinger et al., 2007; Verheggen and van Wijmen, 1996).

A number of interventions have been developed to improve recall of consent information. These include simplifying the reading level (e.g., Bjorn et al., 1999), providing post-consent telephone reminders (e.g., Aaronson et al., 1996) and giving corrected feedback of inaccurate recall over multiple learning trials (e.g., Stiles et al., 2001). Although some studies have found these approaches to significantly enhance recall of consent information, the improvements were generally small to moderate in magnitude and a substantial minority of participants did not exhibit improvements.

The consent interventions that have been studied to date are essentially remedial in nature and assume that deficits in recall stem from cognitive or educational limitations on the part of participants. In fact, numerous studies have found cognitive variables such as vocabulary level, full-scale IQ, verbal IQ, educational attainment and neuropsychological measures of memory and attention to be positively correlated with recall of consent information (e.g., Festinger et al, 2007; Taub et al., 1986). However, in statistical combination these variables account for less than half of the variance in recall (Festinger et al., 2007). This suggests that cognitive remediation strategies might be addressing only part of the problem and perhaps other relevant factors are being neglected. Some research participants might be insufficiently motivated to learn the elements of informed consent. They may not view it as worth the time or effort to attend to the information presented during the consent process and commit that information to memory. If so, this could point to the need for interventions aimed at enhancing participants’ motivation to learn consent information.

Barring any cognitive impairments most individuals have the ability to concentrate, but one’s level of concentration may vary substantially across different contexts. For example, when someone is engrossed in a captivating novel, playing a musical instrument, or engaged in a sports event, they may engage in nearly total concentration. But at other times, for example during a boring lecture, concentration may be less than optimal, thoughts may be less focused, and one may be easily distracted by other stimuli. This phenomenon is not difficult to explain. In fact over a half-century of research on operant conditioning provides a very detailed explanation of how and why this occurs (see Mackintosh, 1977; Pauli and O’Reilly, 2008 for review). Put simply, a person’s attention and concentration like any other behavior will either increase or decrease as a function of behavioral consequences or contingencies. That is, if ones’ concentration on or attention to a particular stimulus is rewarded it will naturally increase; if it is punished or not rewarded it will decrease. Despite the substantive role that operant conditioning has been shown to play in learning, it has yet to be used to improve research participants’ learning of their human subject protections and consent information.

Nevertheless, decades worth of research reveals that performance on memory tests can be significantly enhanced by rewarding participants for successfully completing recall tasks (e.g., Brown and White, 2005; Croft et al., 2007; Nevin, 1974). The current study examined whether recall of research consent information could be similarly improved by offering participants tangible payment incentives for each element of informed consent they correctly recalled after a one-week delay interval. This consent study was conducted within the context of a larger experimental trial involving drug-abusing criminal offenders in a drug court program. The participants were being asked to consent to be randomly assigned to different intensities of supervision by a judge and different intensities of clinical case-management services. Because closer supervision by the court or a case manager could potentially lead to a greater detection of infractions, participants risked the possibility of increased legal consequences by consenting to be in the study. Therefore, it was important to ensure they could recall the implications of the study and their right to withdraw without negative consequences. As such, this larger study provided a meaningful context for studying approaches to enhancing recall of consent information.

Importantly, this pilot study was not intended to be sufficiently powered to detect statistically significant effects. Rather, the goal was limited to estimating effect sizes for the intervention to determine whether it would be worth the cost and effort to proceed with a fully powered experimental trial.

2. Methods

2.1 Sample

The study was approved and monitored by the Institutional Review Boards (IRB’s) of the Treatment Research Institute and the Delaware State Department of Health and Social Services. Participants (N = 31) were recruited from a misdemeanor drug court located in the urban city of Wilmington, DE. Eligibility criteria for this drug court require participants to be at least 18 years of age, charged with a misdemeanor crime involving possession of cannabis, possession of drug paraphernalia or possession of hypodermic syringes, and have no history of a violent crime or drug dealing or manufacturing. The program is scheduled to be a minimum of 4 months in length and provides a combination of psycho-educational group counseling, court hearings, case management sessions and random, weekly urine drug screens. Successful graduation results in the criminal charges being dropped and an opportunity to have the conviction expunged or erased from the participant’s record.

Participants in the study were predominantly young adults (M = 27.90, SD = 11.57 years), male (75%), Caucasian (48%) or African American (42%), unmarried (87%), high school educated (M = 11.77, SD = 1.93 years) and employed full or part time (65%). They reported their primary substance of abuse was cannabis (81%), cocaine (13%) or opiates (6%).

2.2 Recruitment Procedures

Upon entering the drug court program, all defendants were instructed by the judge to report to the Brandywine Counseling, Inc. (BCI) treatment program on a specified date the following week for an intake appointment. Immediately following a group orientation session conducted by BCI treatment staff, a research technician provided a brief oral description of the study, including participation requirements, payment incentives, confidentiality protections and the right to refuse or withdraw from the study at any time. Individuals who indicated a potential interest in the study were then scheduled for an individualized informed consent procedure. Thirty-one consecutive clients who indicated a potential interest in the study ultimately consented to participate in the main study.

Prior to undergoing the individualized informed consent procedure, participants were block randomized into one of two conditions: (1) incentivized consent (n = 14) or (2) consent as-usual (n = 17). Individuals in the incentivized condition were informed prior to starting the consent procedure that they would be quizzed on their recall of the consent information at their one week follow-up appointment and would earn a $5.00 money order for each item they answered correctly, for a possible total of $75.00. Individuals in the consent-as-usual condition were also informed they would be quizzed on their recall of the consent information at the one week follow-up appointment; however, they were not offered payment incentives.

Participants were not informed that there were two different consent conditions and did not provide consent to be randomized to one of those conditions. As a practical matter, it would have been confusing to ask participants to provide informed consent to participate in one study with the goal of improving their consent to yet a second study. More importantly, this could have had the effect of seriously skewing the sample because only those individuals who were already inclined to give consent would have been exposed to the experimental consent procedures. For these reasons, both IRBs granted a waiver of informed consent for the consent pilot study pursuant to 45 CFR §46.116(d)(1)-(4) because the research involved no more than minimal risk and the study could not have been practicably carried out without a waiver.

Participants in both conditions underwent a manualized informed consent procedure to ensure the consent process was equivalent across groups apart from the availability of payment incentives. Each participant was asked to read the consent document, which was written at or below the 6^th grade level, silently to him or herself while the interviewer read it aloud. This procedure avoids clients experiencing embarrassment by having to admit difficulties with reading. The technician read and subsequently paraphrased each section of the consent document according to a standardized script. When the technician was finished with each section, the participant was given an opportunity to ask questions and was then asked to paraphrase the information just presented. Errors were immediately corrected and this process continued until the participant correctly paraphrased each section of the consent document. This manualized consent procedure reflects a reasonably intensive standard of research practice.

2.3 Follow-up Procedures

All 31 participants attended their follow-up assessment and completed the post-consent quiz. The follow-up appointment was held an average of 12.42 days (SD = 9.8 days) following the consent procedure. There was no significant difference in the average time delay between the consent procedure and the post-consent quiz for the incentivized (M = 12.71, SD = 11.44 days) vs. the as-usual (M = 12.18, SD = 8.50 days) conditions, t (29) = 1.50, p = .54.

The post-consent quiz was constructed from the Understanding Scale of the MacArthur Competence Assessment Tool for Clinical Research (MacCAT-CR; Appelbaum and Grisso, 2001). This scale assesses a participant’s ability to recall important facts about a research study that are necessary for making an informed decision about participation. Applying the procedures developed by Appelbaum and Grisso (2001), the post-consent quiz was created by translating each of the main elements of the consent document into a scoreable question format resulting in 15 items. The questions varied in the number of answers that were called for. For example, one question concerning the duration of the study had only one correct response (6 months) whereas another question concerning the schedule of follow-up appointments had seven responses corresponding to each of seven follow-up interviews. Participants were not cued in advance about the number of required responses.

The research technicians followed a structured script for administering the consent quiz and achieved greater than 95% agreement in the scoring of responses on inter-rater reliability trials. The 15-item consent quiz inquired about three broad content areas in the consent document: (1) participants’ general understanding of the research protocol (8 items), (2) the risks and benefits associated with the study (2 items) and (3) human subject protections (5 items). Table 1 presents the specific items contained within each content area.

Table 1.

Post-consent quiz scores by condition giving partial credit for incomplete responses

Variable	Partial Credit Score: M (SD)		Avg. Proportion of Material
	Incentivized	As-Usual	Incentivized	As-Usual
TOTAL SCORE (range = 0 to 15)	9.74 (2.91)**	6.23 (2.68)	65%	42%
Understanding of Protocol (range = 0 to 8)	5.58 (1.43)**	4.01 (1.52)	70%	50%
Items (range = 0 to 1)
Purposes of study (3 answers)	.29 (.43)	.18 (.24)
Duration of study (1 answer)	.93 (.27)	.47 (.51)
Follow-up appointments (7 answers)	.87 (.31)	.62 (.42)
Follow-up payments (7 answers)	.93 (.19)	.75 (.34)
What follow-ups involve (1 answer)	1.00 (.00)	1.00 (.00)
Study conditions (3 answers)	.48 (.31)	.35 (.30)
Random assignment (1answer)	.72 (.47)	.59 (.51)
Reasons one might be removed from the study (3 answers)	.38 (.49)	.06 (.24)
Risks and Benefits (range = 0 to 2)	1.09 (0.65)**	0.29 (0.34)	55%	15%
Items (range = 0 to 1)
Potential benefits (5 answers)	.34 (.43)	.18 (.22)
Potential risks (2 answers)	.75 (.38)	.12 (.22)
Human Subject Protections (range = 0 to 5)	3.06 (1.40)*	1.92 (1.15)	61%	38%
Items (range = 0 to 1)
Collateral data to be collected (2 answers)	.75 (.43)	.53 (.48)
Consequences for declining or withdrawing (1 answer)	.93 (.27)	.94 (.24)
Who to contact with questions (1 answer)	.57 (.51)	.18 (.39)
Who to contact if harmed (1 answer)	.43 (.51)	.12 (.33)
Who will have access to the data (HIPAA disclosure) (3 answers)	.38 (.47)	.16 (.27)

^*p < .05;

^**p < .01. = HIPAA = Health Insurance Portability and Accountability Act.

2.4 Data Analysis

A randomization check compared the groups at baseline by gender, race, age, primary substance(s) of abuse, current criminal charges and prior bench warrants. A liberal p-value of < .10 was used to detect possible differences due to the small cell sizes in this pilot study. The groups differed only by age, with the incentivized group being significantly older than the consent-as-usual group (M = 32.71, SD = 14.19 vs. M = 23.94, SD = 7.11). However, because age did not correlate significantly with any of the outcome measures, it was not included as a covariate in subsequent analyses.

Because some items required multiple responses, the consent quiz was tabulated both by scoring items as correct only when they were answered in the entirety, and also by giving partial credit for incomplete responses. For example, a participant who correctly reported three out of seven follow-up interviews received a partial credit score of 0.43 (3/7). The results using both of these scoring methods were virtually identical; therefore, only the partial credit scores are reported in Table 1. To control Type I error resulting from multiple comparisons, outcomes were statistically compared only on the total scores and the three content areas relating to participants’ general understanding of the research protocol, risks and benefits of the study and human subject protections.

3. Results

Participants in the incentivized condition had significantly higher total post-consent quiz scores than participants in the as-usual condition, t (29) = 3.50, p < .01 (see Table 1). Participants in the incentivized condition recalled an average of 65% of the consent material whereas those in the as-usual condition recalled an average of only 42% of the material.

The same was true for each of the three content areas in the consent document. Participants in the incentivized condition recalled significantly more information relating to the general requirements of the research protocol, t (29) = 2.94, p < .01; the anticipated risks and benefits of the study, t (29) = 4.42, p < .01; and human subjects protections, t (29) = 2.48, p < .05. The same pattern was also found for the large majority of the 15 individual items, with participants in the incentivized condition generally recalling substantially more consent information.

The estimated effect sizes (ES’s) for the between-group comparisons were d = 1.25 for the total score, d = 1.06 for participants’ general understanding of the research protocol, d = 1.54 for the risks and benefits of the study, and d = 0.89 for human subjects protections. Each of these ES’s is in the “large” range as specified by Cohen (1988).

4. Discussion

The results of this pilot study provide strong preliminary support for the efficacy of using contingent incentives to improve research participants’ recall of consent information. Compared to a standard informed consent procedure, participants who were offered additional payment incentives for learning consent information had significantly better recall one week following their entry into the study, including better recall of study procedures, the foreseeable risks and benefits of participation, and human subject protections. Although this small pilot study was not intended to be adequately powered to detect significant differences, statistical significance was reached for all of the planned comparisons due to the large effects of the intervention.

Prior research (e.g., Festinger et al., 2007) indicates that less than half of the variance in recall of consent information may be due to largely static factors, such as participants’ current intellectual functioning and educational achievement; however, more than half of the variance in recall remains unexplained. The current findings suggest participants’ motivation to learn the elements of informed consent might be an equally important factor for investigators to consider during the informed consent process. Of course, as a practical matter, many investigators are unlikely to have sufficient resources to pay research participants to learn consent information; however, other forms of motivational enhancement interventions, such as motivational interviewing techniques (e.g., Miller and Rollnick, 1991), could turn out to be similarly effective and should be examined in future studies.

Ideally, investigators should combine motivational strategies aimed at improving participants’ interest in learning consent information with remedial measures aimed at simplifying the consent process and compensating for cognitive or educational deficits. In the current study, participants receiving payment incentives recalled an average of approximately 65% of the material in the consent document. Perhaps combining this approach with telephonic reminders or corrected feedback could further enhance recall to levels exceeding 80% or 90%. Particularly in high-risk studies in which participants face potentially serious study-related adverse events, such a combined strategy might be deemed necessary by IRB’s or other oversight boards to ensure adequate ethical protections.

4.1 Limitations

The limitations of this study are largely self-evident. The small sample size raises questions about whether the results are likely to be stable in a larger cohort and whether these 31 individuals were sufficiently representative of the target population of clients in the drug court program. Moreover, there is no way to know whether the results would generalize to other populations of research participants, such as mentally ill clients, juveniles or the elderly.

In addition, recall was measured over a relatively brief one-week delay interval. It is unknown whether the effects can be maintained over the full course of a research study or whether the effects might decline precipitously once the incentives have been discontinued.

The consent quiz was administered in an open-ended format and measured participants’ free recall of consent information. This places fairly high cognitive demands on participants, who might have difficulty verbally describing what they remember. In fact, the use of a recognition test, which asked participants to select the correct response from a list of possible answers, has resulted in substantially higher post-consent quiz scores (Stiles et al., 2001). However, a free-recall procedure more closely approximates what typically occurs in standard research practice. Research participants are not ordinarily asked to identify their ethical protections from a list, but rather must often act on their own volition to advance or protect their rights and interests.

Some scholars distinguish among various levels or degrees of knowledge about consent information (Appelbaum and Grisso, 2001; Flory and Emanuel, 2004). For example, simply recalling the elements of a consent document does not necessarily mean a participant appreciates the implications of the study for his or her personal well-being. Concrete knowledge about consent provisions might not translate into rational decision-making about whether to enter a study or to a sensible balancing of the potential risks and benefits. Regardless, it is reasonable to assume that accurately recalling consent information would be a minimum requirement for engaging in such a decision-making process. If participants do not correctly recall the basic elements of a study, it would be difficult for them to rationally consider the implications of the study for their personal interests.

Findings from this study have important implications for ethics research. First, the incentivized consent procedure may be a useful strategy to improve consent recall in research studies, particularly in studies in which participants face a realistic threat of serious side effects or injury from the research procedures. Second, the results of this study provide an important “proof-of-concept” regarding whether motivational procedures are required to obtain an acceptable level of mastery of consent information. The doctrine of informed consent is among our most basic and important legal and ethical protections; therefore, additional work is critical for improving research participants’ recall of consent information beyond currently unacceptable levels.