Achieving New Levels of Recall in Consent to Research by Combining Remedial and Motivational Techniques

Abstract

Introduction

Research supports the efficacy of both a remedial consent procedure (corrected feedback) and a motivational consent procedure (incentives) for improving recall of informed consent to research. Although these strategies were statistically superior to standard consent, effects were modest and not clinically significant. This study examines a combined incentivized consent and corrected feedback that simplifies the cognitive task and increases motivation to learn consent information.

Methods

We randomly assigned 104 individuals consenting to an unrelated host study to a consent as usual (CAU) condition (n = 52) or an incentivized corrected feedback (ICF) condition (n = 52). All participants were told they would be quizzed on their consent recall following their baseline assessment and at 4 monthly follow-ups. ICF participants were also informed that they would earn $5.00 for each correct answer and receive corrected feedback as needed.

Results

Quiz scores in the two conditions did not differ at the first administration (p = 0.39, d = 0.2), however ICF scores were significantly higher at each subsequent administration (second : p = 0.003, Cohen’s d = 0.6; third: p < 0.0001, d = 1.4; fourth: p < 0.0001, d = 1.6; fifth: p < 0.0001, d = 1.8.)

Conclusions

The ICF procedure increased consent recall from 72% to 83%, compared to the CAU condition in which recall decreased from 69% to 59%. This supports the statistical and clinical utility of a combined remedial and motivational consent procedure for enhancing recall of study information and human research protections.

More than 60 years following the Nuremberg Code¹, it remains unclear whether the average research participant is truly informed about the nature of their research participation. Results from research in many scientific disciplines indicate that people have poor understanding and recall of the studies to which they have consented, resulting in the consent process being far from “informed.” Participants in clinical research studies often fail to recall much of the information presented during the informed consent process even after only a few days. A comprehensive review concluded that most informed consent procedures elicited recall scores below 60% on post-consent quizzes.²

Failure to recall consent information is not restricted to minute study-related details or technical content. Participants are often unaware that they were participating in a research study, fail to recall study-related risks, are unable to describe randomization procedures or placebo interventions, and are unaware that they can withdraw from a study without negative consequences.^3–5 Research in a population of substance abusing offenders^6,7 revealed that participants failed to recall 60% of consent information just two weeks after their initial consent, calling into question whether participants can make informed decisions about their initial or continued involvement in research.

A broad array of interventions have been developed to improve understanding and recall of consent information.^2,8,9 The most successful strategies can be conceptualized as two general categories: (1) the form of the consent document and (2) the process of presenting consent information.

Much of the work examining ways of improving understanding of consent information has focused on revising the basic form of printed consent materials to improve readability. Readability is often enhanced by writing materials at a 6–8^th grade reading level which is important given that 14% of U.S. adults have marginal literacy skills and 29% can only perform simple literacy activities.¹⁰

Other methods for improving readability of consent materials include summarizing key points; utilizing headings and bulleted points; increasing font sizes; clustering similar content in the document; using simpler sentence structures; adding illustrations and graphics; and presenting consent information in a streamlined booklet.^11–14 These modifications are most beneficial for individuals with lower reading and comprehension levels.^11,9 Regrettably, strategies designed to improve readability have received mixed results in improving comprehension.^2,15,12 Researchers have begun to recognize the need to focus more attention on the process and not simply the structure of informed consent.¹⁶ A 2001 report by the National Bioethics Advisory Commission re-emphasized the importance of the informed consent process in research, stating that “Federal policy should emphasize the process of informed consent rather than the form of its documentation…” and “ensure that participants continue to make informed and voluntary decisions throughout their involvement in the research” (p. 101).¹⁷ Corrected feedback is an example of a process focused procedure. The procedure has been consistently associated with improvements in both initial comprehension and long-term recall of consent information.^18–20 The procedure typically involves assessing participants’ knowledge and comprehension of informed consent information following initial consent and providing corrected feedback on incorrect responses. Studies have demonstrated its efficacy in both clinical and non-clinical settings.^19–22

In a series of studies, Taub and colleagues examined the efficacy of the corrected feedback (CF) procedure among elderly participants. In their initial study¹⁹, participants in the CF condition read the consent form, answered multiple-choice questions about key points, and were provided with a single trial of corrected feedback. Control participants read the consent form but did not receive a quiz or CF. The CF procedure significantly improved recall of consent information for all age and vocabulary levels. In a second study, Taub and Baker¹⁸ varied the number of trials of CF that participants received. The multi-trial CF approach improved comprehension scores at all vocabulary levels but had a more limited effect on recall two to three weeks later. These results have been extended to other clinical and non-clinical settings.^20–22

Festinger et al.⁶ evaluated the efficacy of using a CF procedure with substance abusers participating in a host clinical trial by examining recall of study-related information on a monthly basis. Participants completed a consent quiz two weeks after consenting to the study and again at months 1, 2, and 3. Participants who received corrected feedback recalled significantly more consent information over the course of the study than controls. Although the CF procedure was superior to CAU in improving recall, the gains were modest with rates reaching only 55% after several repetitions.

The strategies discussed above are remedial in nature as their goal is to simplify the cognitive tasks or compensate for cognitive deficits. Although numerous studies have found cognitive variables (IQ, memory, attention) to be positively correlated with consent recall,^1,2,6,23 in statistical combination, these variables account for less than half of the variance.²⁴ This suggests that remedial strategies may address only part of the problem and that other factors may be involved. One such factor may be motivation. Some participants might be uninterested in learning the elements of informed consent or may not view it as worth the time or effort to attend to or commit the information to memory.

A small two-group pilot study (N=30) used incentives to examine the role of motivation in consent recall.⁷ At the time of consent, experimental participants were told that they would receive $5 for every correct response on a consent quiz administered one week later, while control participants were quizzed but not offered the incentives. Incentivized participants recalled significantly more information after one week than controls. This difference was found for the total score (65% vs. 42%) and for specific content areas addressing study procedures (70% vs. 50%), human subjects protections (5% vs. 15%), and risks/benefits (61% vs. 38%). This suggests that incentives, and perhaps other motivational strategies, may be useful in improving the consent process. Given these promising findings, we hypothesized that combining a remedial (corrected feedback) and a motivational (incentives) approach would elicit greater recall because it would both simplify the cognitive task and increase motivation to learn the consent information.

METHODS

PARTICIPANTS

The study was approved and monitored by the Treatment Research Institute Institutional Review Board and the Delaware Human Subjects Review Board. Study participants were 104 misdemeanor drug court clients who were participating in a host clinical trial examining different schedules of judicial hearings and treatment in drug court. To be eligible for the host study participants had to: be at least 18 years old, be charged with a misdemeanor drug crime, and have no history of violent or felony crimes. Clients were randomly assigned to a CAU or to an incentivized corrected feedback (ICF) condition.

Participants were predominantly young adults (M=23.4 years, SD=6.1), male (77%), never married (91%), employed (79%), Caucasian (59%), with an average of 12.2 years of education (SD=1.63). Using independent t-tests for continuous variables (age, years of education, onset of substance abuse) and chi-square analyses for categorical variables (gender, marital status, employment, race, drug/alcohol dependence), no between group baseline differences were identified (p range=.09–.84). The groups also did not differ on verbal IQ as measured by the WASI vocabulary subscale test (p=0.78).

RECRUITMENT

At intake into the drug court program, the judge instructed clients to report to an outpatient treatment program for orientation and initial assessment. Following this group orientation and assessment, a research assistant (RA) described the host study to the clients. Of the 254 clients approached, 104 (41%) indicated interest and consented to participate. Prior to the consent procedure, participants were randomly assigned to the ICF (n=52) or CAU condition (n=52).

PROCEDURES

Prior to the consent procedure all participants were informed that they would be quizzed on their recall of consent information as a part of the baseline assessment and again at each of the follow-ups. However, ICF participants were also informed that they would earn $5 for each item answered correctly on the 15-item post-intake consent quiz (possible total=$75) and would receive corrected feedback on incorrect responses at each assessment.

All participants received a manualized, standard informed consent procedure by a trained RA. They were informed of their study condition and asked to read the consent form silently to themselves, as it was read aloud. The RA then administered each section of the informed consent document after which the participant was given an opportunity to ask questions. Participants were then asked to paraphrase each section of the consent and the RA corrected errors until the participant could paraphrase each section correctly.

All participants then completed a 90-minute baseline assessment consisting of the assessments for the host drug court study and the current study. They were then asked to complete the first consent quiz. ICF participants received corrections to erroneous responses and $5 for each correct response. Finally, all participants were scheduled to complete a consent quizzes at follow-up appointments scheduled over the next four months. Consequently, participants had the opportunity to complete five consent quizzes (baseline and months 1, 2, 3, 4). All participants received $40 for completing the baseline assessment and $25 for each monthly quiz.

Consent quiz

The 15-item consent quiz assessed the 15 principal topics covered in the host study’s consent form including study purpose, procedures, remuneration, human rights protections (i.e., confidentiality, and recourse in the event of being harmed, (i.e., whom to contact with additional questions or concerns). This quiz has been used in several previous studies.^7,24 Quiz items were open-ended to elicit recall rather than recognition. The coding of responses as correct or incorrect followed a highly objective scoring procedure in which the coding forms clearly specified the content necessary for items to be considered correct. Although RAs could not be blinded to the study conditions, this highly standardized procedure reduced the likelihood of RA bias. RAs achieved greater than 95% agreement during pre-study interrater reliability trials.

The questions varied in the number of answers that were called for, with the number of necessary responses ranging from one to eight. For example, one question regarding the duration of the study had only one response (i.e., 12 months), whereas another question addressing the schedule of follow-up interviews had eight responses, corresponding to each of the scheduled study interviews. Questions were delivered in an open-ended manner; therefore, participants did not know the number of responses required for each question.

Because each item varied in the number of responses, item scores were weighted by calculating the proportion of correct responses provided (out of the total of necessary responses) for each item. For instance, if a person provided one correct response to an item that required two components, they received a score of 0.50 for that item. Providing the two correct responses would result in a score of 1. Consent quiz scores were calculated by summing these 15 weighted scores (total score range=0–15). In addition, each item could be classified into one of four domains: understanding of protocol (8 items), human subject protections (5 items), risks of participation (1 item), and benefits of participation (1 item). Scores were calculated for each of these domains by summing their total weighted scores.

DATA ANALYSIS

Groups were compared on the number of consent quizzes completed using a t-test, and correlations between the number of quizzes completed and baseline demographics (age, race, gender, employment status, marital status, and education) were examined. Group differences in time from consent to each quiz were examined using a linear mixed effects model and included terms for group, administration number, and their interaction. A main effect for group or a group x administration number interaction would indicate that time between each administration varied systematically between conditions.

The primary analyses examined group differences in recall of the consent information. A series of linear mixed effects models were used to examine differences between the CAU and ICF conditions on recall of consent information. The primary model examined differences in total consent quiz scores administrations 1–5 and included terms for group, administration number, and their interaction. Models used a maximum likelihood estimation strategy and specified a compound symmetry covariance structure. This approach was repeated for each of the four domain scores. Analyses were conducted using SAS v. 9.3.

RESULTS

QUIZ COMPLETION

Sample sizes for the CAU group were 52, 50, 45, 44, and 37 at administrations 1, 2, 3, 4, and 5, respectively. Corresponding sample sizes for the ICF group were 52, 50, 48, 41, and 31. CAU participants completed an average of 4.38 (SD=1.12) consent quizzes and ICF participants completed an average of 4.27 (SD=1.12) consent quizzes (p=.60). The number of consent quizzes completed was not related to any baseline demographic or status variables (p range=0.09–0.63). In the CAU group, the number of days from consent for quizzes 2 through 5 were 44.12 (SD=20.37), 69.17 (SD=14.51), 99.68 (SD=12.88), and 124.20 (SD=9.70), respectively. In the ICF group, the corresponding number of days from consent was 40.94 (SD=11.43), 69.77 (SD=13.10), 100.70 (SD=15.95), and 123.30 (SD=11.93). There was no group, F(1,102)=.12, p=.72, or group x administration interaction, F(1,338)=.51, p=.73, for the number of days from consent.

Total Scale Score

Quiz scores for CAU and ICF groups at each assessment point are presented in Table 1. The mixed effects model examining total scales score revealed a significant main effect for group, F(1,104)=39.92, p<0.0001, administration number, F(4,336)=53.1, p<0.0001, and the group by administration number interaction, F(4,336)=25.9, p<0.0001. Specific contrasts indicated that total scale scores in the ICF and CAU conditions did not differ after the first administration (p=0.39, d=0.2) but scores in the ICF condition were significantly higher than those in the CAU condition after the second (p=0.003, d=0.6), third (p<0.0001, d=1.4), fourth (p<0.0001, d=1.6), and fifth (p<0.0001, d=1.8) administrations. ICF participants increased their average total recall of the material from 72% to 83% at the fifth administration, whereas CAU participants’ average recall decreased from 69% to only 59% after five administrations.

Table 1.

Quiz scores for each group.

Measure	Group	Quiz 1: M (SD)	Quiz 2: M (SD)	Quiz 3: M (SD)	Quiz 4: M (SD)	Quiz 5: M (SD)
Total (15 items)	As Usual	10.36 (2.05)	7.79 (1.78)	8.43 (1.83)	8.70 (1.89)	8.84 (2.0)
	Incentivized	10.74 (1.95)	9.05 (2.26)	11.18 (2.19)	11.71 (1.83)	12.45 (1.99)
Understanding of Protocol (8 items)	As Usual	5.79 (1.18)	4.41 (1.12)	4.83 (1.08)	5.04 (1.15)	5.03 (1.2)
	Incentivized	5.74(1.17)	4.93 (1.33)	6.07 (1.28)	6.2 (1.14)	6.69 (1.05)
Human Subject Protections (5 items)	As Usual	3.83 (0.91)	2.91 (0.97)	3.11 (0.86)	3.18 (0.84)	3.36 (0.77)
	Incentivized	4.05 (0.85)	3.38 (0.89)	4.05 (0.74)	4.3 (0.54)	4.46 (0.84)
Risks of Participation (1 item)	As Usual	0.40 (0.37)	0.24 (0.32)	0.21 (0.33)	0.24 (0.29)	0.24 (0.33)
	Incentivized	0.51 (0.40)	0.40 (0.38)	0.61 (0.45)	0.73 (0.33)	0.74 (0.36)
Benefits of Participation (1 item)	As Usual	0.33 (0.26)	0.23 (0.18)	0.28 (0.25)	0.24 (0.23)	0.19 (0.20)
	Incentivized	0.43 (0.30)	0.33 (0.27)	0.44 (0.29)	0.49 (0.33)	0.55 (0.34)

Understanding of Protocol Score

The model examining the understanding of protocol domain scores indicated a significant effect of group, F(1,104)=23.9, p<0.0001, administration number, F(4,336)=31.1, p<0.0001, and a group by administration number interaction, F(4,336)=14.6, p<0.0001. Specific contrasts indicated that the understanding of protocol scores in the ICF condition did not differ at the first administration (p=0.76, d=0.03) but the scores in the ICF condition were significantly higher than those in the CAU condition at the second (p=0.04, d=0.4), third (p<0.0001, d=1.0), fourth (p<0.0001, d=1.0), and fifth (p<0.0001, d=1.5) administrations. ICF participants increased their average recall of understanding the protocol from 72% to 84% at the fifth administration, whereas CAU participants decreased their recall from 72% to 63% at the fifth administration.

Human Subjects Protections Score

Examination of the human subject protections domain scores indicated significant effects of group, F(1,104)=32.4, p<0.0001, administration number, F(4,336)=27.8, p<0.0001, and a group by administration number interaction, F(4,336)=9.8, p<0.0001. Specific contrasts indicated that the human subjects protections scores in the ICF and CAU conditions did not differ significantly after the first administration (p=0.20, d=0.2) but the scores in the ICF condition were significantly higher than those in the CAU condition at the second (p=0.006, d=0.5), third (p<0.0001, d=1.2), fourth (p<0.0001, d=1.6), and fifth (p<0.0001, d=1.4) administrations. ICF participants increased their average recall of human subject protection information from 81% to 89% after five administrations, whereas CAU participants decreased their recall from 77% to 67% at the fifth administration.

Risks of Participation Score

Examination of the risks domain scores indicated a significant main effect of group, F(1,104)=32.1, p<0.0001, administration number, F(4,336)=5.26, p=0.0004, and a group by administration number interaction, F(4, 336)=8.3, p<0.0001. Specific contrasts indicated that the risk scores in the ICF and CAU groups did not differ at the first administration (p=0.16, d=0.3) but the scores in the ICF condition were significantly higher than those in the CAU condition at the second (p=0.04, d=0.5), third (p<0.0001, d=1.0), fourth (p<0.0001, d=1.6), and fifth (p<0.0001, d=1.5) administrations. ICF participants increased their average recall of risk information from 51% to 74% at the fifth administration, whereas CAU participants decreased their recall from 40% to 24% at the fifth administration.

Benefits of Participation Score

Examining the benefit domain scores indicated a main effect of group, F(1,104)=25.04, p<0.0001, administration number, F(4,336)=3.66, p=0.006, and a group by administration number interaction, F(4,336)=5.3, p=0.0004. Specific contrasts indicated scores in the ICF and CAU groups did not differ significantly at the first (p=0.05, d=0.4) or second (p=0.05, d=0.4) administrations but the scores in the ICF condition were significantly higher than those in the CAU condition at the third (p=0.003, d=0.6), fourth (p<0.0001, d=0.9), and fifth (p<0.0001, d=1.3) administration. ICF participants increased their average recall of benefit information from 43% to 55% at the fifth administration, whereas CAU participants decreased their recall from 33% to 19% at the fifth administration.

DISCUSSION

Despite the recent emphasis on viewing informed consent as an ongoing process,¹⁷ the primary focus of many IRBs remains on the consent document. Apart from occasionally requiring investigators to administer brief consent quizzes, IRBs rarely attend to what transpires in the informed consent process or require proof that consent was obtained knowingly and voluntarily.

This study demonstrated the efficacy of the ICF procedure in improving recall of consent information over the course of a longitudinal study. Participants who received the ICF procedure displayed improved recall over time while CAU participants showed a decline in recall rates. These findings provide promising initial support for the statistically and clinically significant efficacy of this remedial and motivational consent procedure in helping participants to recall essential human subject protections. In practice, the ICF procedure may be quite useful in its current form. Intermittently administering a brief consent quiz coupled with small incentives for correct responses may appear impracticable on the surface, but it is actually a relatively straightforward way to ensure human subject protections. This becomes more apparent when one considers the many human subject protections researchers and their organizations regularly undertake and budget for (e.g., IRB reviews, data safety monitoring board reviews, HIPAA requirements) that, in many ways, assume the integrity of the consent process.

Conceptually, these findings provide additional support for the importance of motivation in the consent process. Although remedial procedures such as shortening the consent, simplifying reading levels, and providing corrected feedback may compensate for certain intellectual or cognitive deficits, they are unlikely to motivate research participants to attend to and encode consent information it into long-term memory. Acknowledging the importance of motivation in the consent process, as was demonstrated through the use of incentives in this study, may open the door for developing other strategies for increasing participant motivation to recall consent information. Increasing participants’ appreciation of consent information, viewed by some as necessary for true informed consent,²⁵ may increase their recall, and finding alternative ways to enhance motivation without monetary incentives may increase the feasibility and utility of the ICF procedure.

The present study has three potential limitations. First, the two group experimental design compared consent as usual to a combined incentivized and corrected feedback procedure. This prevents us from dismantling the individual effects of the incentives and the corrected feedback. Our prior studies examining corrected feedback and incentives independently were conducted in the same drug court using a similar population, host study, and consent quiz^6,7. Although these studies demonstrated the statistical efficacy of each procedure over consent as usual, the rates of recall did not exceed 65%, suggesting that neither procedure alone produced clinically significant improvements. While the current study indicated that the combined procedure produced recall rates exceeding 80%, the design prevents us from confidently identifying the relative contribution of incentives and corrective feedback.

Second, the generalizability of the findings is somewhat limited because it was conducted using a single population, a single set of study procedures, and a single consent form. The study was conducted in the context of a drug court host study involving individuals who abused or were dependent on illicit substances. Although prior findings²⁴ found that this population’s intelligence, reading, and other cognitive abilities were in the normative range, the efficacy of the ICF procedure may be different when applied to other populations. Similarly, the host study involved a randomized trial of a single behavioral intervention. It is unclear whether the findings generalize to studies involving different levels of risk or complexity (e.g., biomedical trials). Finally, related to the unique nature of the host trial, the current study involved a single consent form with a very specific set of procedures, risks and benefits, and human subject protections. Although the ICF process could presumably be applied to virtually any consent form, we do not know if changes to the consent form content would affect study findings.

Finally, the open-ended consent quiz used in the study has not been standardized or psychometrically validated. Our prior literature review failed to identify a single structured, psychometrically validated consent quiz. Nevertheless, we found dozens of versions of consent quizzes in the literature.⁸ One common limitation of many of these quizzes was that their use of brief, overly simplistic, and often leading questions that relied on recognition (e.g., true and false, multiple choice) rather than recall. Such assessments are likely to overestimate participants’ recall. To avoid this shortcoming, the open-ended format of our consent quiz relied on recall rather than recognition. We believed this was critical considering that participants are more likely to rely on recall (not recognition) when they need to recall study-related information throughout or following their study participation. Importantly, the ICF procedure could readily be used with validated consent quizzes.

The current study provides strong support for the efficacy of the ICF procedure. Although previous strategies have been able to increase recall, increases of this magnitude have not previously been achieved, let alone using a comprehensive, open-ended consent quiz. Our findings have important implications for improving the consent process and setting new standards in human subject protections.

Future research should examine the generalizability of the ICF procedure in other contexts and populations. For example, the ICF may be particularly useful and effective in improving recall in high risk studies or those that involve long and complicated consent forms. In addition, the procedure may be particularly beneficial to individuals that have particular types of vulnerabilities. Having established the efficacy of the ICF procedure in improving recall of consent information, developing strategies to facilitate the translation of this novel and useful procedure into research practice is warranted.