Methods and Practices of Investigators for Determining Participants’ Decisional Capacity and Comprehension of Protocols

Abstract

Ethicists recommend that investigators assess subjects’ comprehension prior to accepting their consent as valid. Because children represent an at-risk population, ensuring adequate comprehension in pediatric research is vital. We surveyed all corresponding authors of research articles published over a six-month period in five leading adult and pediatric journals. Our goal was to assess how often subject’s comprehension or decisional capacity was assessed in the consent process, whether there was any difference between adult and pediatric research projects, and the rate at which investigators use formal or validated tools to assess capacity. Responses from 102 authors were analyzed (response rate 56%). Approximately two-thirds of respondents stated that they assessed comprehension or decisional capacity prior to accepting consent, and we found no difference between adult and pediatric researchers. Nine investigators used a formal questionnaire, and three used a validated tool. These findings suggest that fewer than expected investigators assess comprehension and decisional capacity, and that the use of standardized and validated tools is the exception rather than the rule.

Since the nuremburg code of 1947 (Nuremberg Military Tribunals, 1947) the research community has steadfastly held that investigators must obtain the voluntary consent of potential subjects prior to including them in research. In the Helsinki Declaration of 1964 (World Medical Association, 1964) provisions were accepted to allow the inclusion of pediatric subjects in research, with the permission of a parent or guardian substituting for consent. In 1977 the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research expanded on this concept, and required an additional safeguard: the assent of the child himself or herself (National Commission, 1978). These protections continue today, embodied in US federal regulations governing the ethical conduct of research involving human subjects (Federal Regulations, 2005).

In order to accept a potential subject’s consent as valid, the subject must have adequate comprehension of the research protocol, otherwise any consent is meaningless (Faden & Beauchamp, 1986). Further work in this field has demonstrated that potential subjects must meet several other criteria as well in order to have decisional capacity to give consent that is valid and informed. Potential subjects must also have the ability to appreciate how study participation will affect them, be able to reason through making a decision regarding participation, and clearly express their choice. (Appelbaum & Roth, 1982) Requirements for competent parental permission are the same, however there is no clear consensus regarding the requirements for meaningful assent (Kon, 2006).

Unfortunately, many subjects appear to have a poor understanding of research protocols when they consent to participate (Criscione, Sugarman, Sanders, Pisetsky, & St Clair, 2003; Joffe, Cook, Cleary, Clark, & Weeks, 2001a; Yuval et al., 2000). Similarly, many parents lack adequate understanding of research protocols. (Mason & Allmark, 2000; Rogers, Tyson, Kennedy, Broyles, & Hickman, 1998) Finally, while the requirements for valid assent are still in the formative stages, many children have an insufficient understanding of research to qualify their assent as meaningful (Miller, Drotar, & Kodish, 2004).

Children represent an at-risk population. As such, in Subsection D of the federal code, special protections are articulated to ensure protection for potential pediatric research subjects. (Federal Regulations, 2005) Because there are serious ethical differences between consenting for one’s own participation and giving permission to include another, research on children may only be undertaken in specific circumstances. (Federal Regulations, 2005) Recently, with new FDA and NIH policies encouraging the inclusion of children in research, (Food and Drug Administration Modernization Act (FDAMA), 1997; National Institutes of Health, 1998; Food & Drug Administration, 1998) investigators are more motivated to recruit children as research subjects.

Because the research community insists on substantial protection for children as research subjects, and because data suggest that both informed consent and permission often do not meet minimum criteria to be considered valid, (Criscione et al., 2003; Joffe et al., 2001a; Mason & Allmark, 2000; Rogers et al., 1998; Yuval et al., 2000) we hypothesized that investigators who recruit pediatric research subjects would be more likely to assess parental comprehension of research protocols and their decisional capacity than investigators who recruit only adult research subjects. Further, because the research community has long noted that standardization is important, and investigators have developed validated tools that researchers can use to assess comprehension and decisional capacity, (Appelbaum & Grisso, 2001; Joffe, Cook, Cleary, Clark, & Weeks, 2001b; Saks et al., 2002) we hypothesized that investigators would readily use such tools. Further, we hypothesized that researchers would be more likely to assess comprehension of study protocols and decisional capacity in the consent process for studies involving interventions or alterations in medical care when compared to studies involving only surveys or interviews.

Method

We reviewed the January through June 2004 issues of Pediatrics, the Journal of Pediatrics, Archives of Pediatric and Adolescent Medicine, the Journal of the American Medical Association, and the New England Journal of Medicine, entering the name and contact information of the corresponding author for each publication of research on human subjects into a database. Authors were excluded if their report was a reanalysis of previously existing data; informed consent for study participation was waived under an emergency treatment/research waiver; consent was given by the community, school, or other body and individual consent was not required; or no current email address was available for the author. Further, only one survey was sent to authors in cases when multiple publications were based on a single study. All corresponding authors were sent a copy of the survey via email, and all non-responders were sent a second email copy.

The survey included similar questions regarding the assessment of comprehension of study protocols and/or the decisional capacity of adult subjects, parents/guardians of pediatric subjects, and pediatric subjects themselves. The survey was pre-tested with ten investigators who had published in these journals in the past but who did not meet inclusion criteria. Survey questions were refined as needed to assure comprehension by pre-test volunteers. Questions included: Did you assess the subject’s level of understanding of the research protocol and/or their competence prior to the subject consenting to participate? If yes, did you use a formal questionnaire or structured method for evaluating the subject’s level of understanding of the research protocol and/or their competence? If yes, was this questionnaire or method validated? To the best of your recollection, was any potential subject NOT included in your study solely because the researchers felt that the subject did not adequately understand the research protocol or was not competent to consent? (Survey available from the author upon request)

Studies were stratified into three categories for analysis. Studies that included only questionnaires or interviews (e.g., survey research, assessment of novel educational programs, cognitive or neurodevelopmental testing) were classified as “interview studies.” Studies that included invasive testing or procedures (e.g., blood draws, CT scans, MRI scans, endoscopy) were classified as “procedure studies.” Studies that included alteration in care (e.g., drug trials) were categorized as “treatment studies.” Responses were analyzed using descriptive statistics and two by two table Pearson’s Chi-squares.

Response rate calculations were performed using methods published by the American Association for Public Opinion Research (AAPOR) (American Association for Public Opinion Research, 2000) and the Institute for Social and Economic Research (ISER). (Lynn, Beerten, Laiho, & Martin, 2001) These calculations were performed using the Simple Interactive Statistical Analysis (SISA) website (http://home.clara.net/sisa/).

This study was approved by the University of California Davis IRB. Potential subjects were sent a complete informed consent packet, approved by the IRB, as an email attachment with the survey. The portions of the consent documents that pertained directly to this study were written at an eighth-grade reading level, however standardized sections of the documents were more complex. Subject response was taken as consent for inclusion. We did not assess subjects’ comprehension of our informed consent document.

Results

Many publications did not detail their informed consent procedures; therefore we sent surveys to authors of all publications that we believed might have required the consent of individual subjects (n = 265). Using AAPOR and ISER definitions and classifications, 102 respondents completed all questions and are the subject of our analysis, 2 respondents completed too few questions to be included in analysis (AAPOR/ISER classification “eligible but refused participation”), 44 stated that they declined to participate and answered no questions (AAPOR/ISER classification “contacted however eligibility unknown”), 69 did not respond in any way (AAPOR/ISER classification “not contacted”), and 48 responded stating that they did not require the consent/permission of individual research subjects (AAPOR/ISER classification “ineligible”). Using the SISA online response-rate calculation tool, response rate was calculated to be 56%.

Table 1 presents the characteristics of the 102 subjects included in our analyses. In general, respondents were well distributed in regards to journal of publication. About two-thirds of respondents’ research was performed in the United States, 20% was performed in Europe, and 16% was performed in other sites. Of the 102 respondents, 46 included only adult research subjects, 29 included only minors, and 27 included both adults and minors. As such, 73 studies included adult subjects and 56 included minors. Using the study classification system described above, 35 of the studies were classified as treatment studies, 25 as procedure studies, and 42 as interview studies.

TABLE 1.

Characteristics of respondents.

Characteristic	n (%)
Journal
JAMA	28 (27)
Pediatrics	27 (26)
Archives	20 (20)
J Peds	16 (16)
NEJM	11 (11)
Site
USA	64 (63)
Europe	21 (21)
Australia	5 (5)
Canada	3 (3)
Middle East	3 (3)
Asia	2 (2)
Africa	1 (1)
South America	1 (1)
Multiple sites	2 (2)
Study subjects
Adults only	46 (45)
Minors only	29 (28)
Both adults and minors	27 (26)
Study classification
Interview studies	42 (41)
Procedure studies	25 (25)
Treatment studies	35 (34)

There were no statistically significant differences in assessment rates comparing adult and pediatric studies. Table 2 presents the rates of assessment for adult and pediatric studies with comparisons by study classification. Among the 102 respondents, 9 investigators stated that they used a formal instrument to assess comprehension of study protocols and/or decisional capacity, and 3 investigators responded that they used a tool that had been validated to make such assessments. Due to the relatively low rate at which formal and validated tools were used, we were unable to perform analyses comparing rate of use in adult versus pediatric studies or comparing rate of use by study classification.

TABLE 2.

Rates of assessment of adult subject, parent, and pediatric subject comprehension of study protocols and/or their decisional capacity in the informed consent process.

	Adults subjects		Parents		Pediatric subjects
	Total n	Assess n (%)	Total n	Assess n (%)	Total n	Assess n (%)
All studies*	73	44 (66)	56	35 (63)	27†	17 (63)
Interview studies	34	16 (47)	21	12 (57)	11	7 (64)
Procedure studies	13	10 (77)	20	14 (70)	10	7 (70)
Treatment studies	26	22 (85)§	15	9 (60)	6	3 (50)

^*27 studies overlapped, included both adults and children

^†Of the 56 studies involving children, 27 required assent

^§Pearson’s Chi-squares 8.95, p < 0.01 compared to adult interview studies

Among the 73 investigators that included adult subjects, 27 (37%) recalled excluding some potential subjects because it was believed that these subjects lacked adequate comprehension of study protocols and/or capacity to provide informed consent. Among the 56 investigators who included children, 7 (13%) recalled excluding some children because their parents lacked adequate comprehension of study protocols and/or capacity to provide informed permission. Interestingly, among the 27 respondents for studies where minors’ assent was sought, 5 (19%) stated that some children were excluded because the children themselves were felt to have inadequate comprehension of study protocols and/or capacity to provide meaningful assent.

Discussion

We found that approximately one-third of investigators did not assess comprehension of study protocols and/or decisional capacity prior to accepting consent or permission as informed, and that those who enrolled pediatric research subjects were no more likely to make such assessments than were those who enrolled adult subjects. Moreover, in the treatment study category there was a trend towards lower rates of assessment in pediatric studies (60% compared to 85% in adult treatment studies). Among studies of adults, those who performed procedure and treatment studies were more likely to assess subjects’ comprehension of study protocols and/or decisional capacity when compared to those conducting interview studies, however there was no such trend in the studies that recruited pediatric subjects. We also found that few investigators used a formal questionnaire to assess comprehension of study protocols and/or decisional capacity, and only three used a validated assessment instrument.

There is a large body of evidence indicating that many adult research subjects, parents of pediatric research subjects, and pediatric research subjects themselves have a poor understanding of research protocols. (Criscione et al., 2003; Mason & Allmark, 2000; Miller et al., 2004; Rogers et al., 1998; Yuval et al., 2000) It should be noted that most of the research conducted to date investigating participants’ comprehension of study protocols have relied on unvalidated assessment tools, and therefore interpretation of their conclusions is somewhat difficult. A few investigators, however, have used validated tools and have demonstrated high rates of significant misunderstanding on the part of participants. Using a validated assessment tool in a three-centered study, Joffe et al. found that while 90% of respondents who had consented to participate in a clinical cancer trial felt that they were well informed, many failed to appreciate important aspects of the trial. For example, 74% did not recognize non-standard treatment, 63% did not recognize incremental risk of participation, 70% did not understand the unproven nature of the treatment, and 25% did not realize that the study was performed mainly for the benefit of future patients. (Joffe, 2001a).

Clearly, the adequacy of any informed consent or permission process is multifactorial. The ability of research subjects to adequately understand study protocols relies not only on their own capacity to comprehend such information but also on the ability of those obtaining consent to clearly explain complex issues. When subjects or surrogates are stressed, in unfamiliar environments, fearful of illness or death, or overwhelmed by the many decisions that they must make, it may become exceedingly difficult for them to comprehend what they are told. Further, researchers who have the task of recruiting subjects may have little expertise in communicating with ill patients or anxious surrogates. Finally, because researchers may be on a tight schedule, consent conversations may be rushed. Given these issues, it is perhaps not surprising that data suggests that many subjects have relatively low rates of comprehension of complex issues.

One response to the criticism that many subjects may inadequately comprehend study protocols would be to formally assess the decisional capacity of all potential subjects prior to accepting their consent as valid. Several validated tools for assessing comprehension or decisional capacity have been published and are widely available. The Quality of Informed Consent tool (QuIC) is designed to assess a subject’s understanding of a proposed clinical trial using 20 questions to which the respondent answers “agree”, “disagree”, or “unsure”. (Joffe, 2001b) The California Scale of Appreciation (CSA) is an 18-question tool validated in the schizophrenic population to assess the appreciation component of capacity to consent to study participation consisting of 13 close-ended and 5 open-ended questions. (Saks, 2002) The most comprehensive tool available is the Macarthur Competence Assessment Tool for Clinical Research (MacCAT-CR), which tests all four aspects of decisional capacity: understanding, appreciation, reasoning, and choice-making ability. (Appelbaum, 2001) The MacCAT-CR consists of four distinct sections corresponding to the four aspects of capacity, with a total of 21 questions scored on a 0 to 2 scale.

The main limitations to using these validated instruments are the lack of clear scores necessary to qualify one’s consent as valid and the time necessary to administer the tools. Without standards against which to compare individual subjects’ scores, such tools may be useful in research projects that investigate the informed consent process itself but are less applicable when assessing the decisional capacity for individual potential subjects. It would seem reasonable, however, to expect all subjects to comprehend some key issues: the primary purpose of the study, the major risks and benefits of participation, possible alterations in care if one agrees to participate, alternatives to study participation, the research nature of the project, a clear understanding that the main purpose of the research is not to benefit the study participants themselves, the voluntary nature of participation, and the ability to withdraw at any time. Because the QuIC, CSA, and MacCAT-CR include specific questions regarding these key elements, it might be reasonable to use such tools to ensure that all subjects at least understand these key elements prior to accepting their consent as informed.

The time necessary to ensure the adequacy of consent can be a serious barrier. In the research arena, one must balance the need to protect human subjects with the time and resources necessary to formally test the decisional capacity of potential subjects or their surrogate decision-makers. When studies pose a risk that is commensurate with daily life, it may be reasonable to forego lengthy assessments. When, however, the potential risk of study participation is substantial, it would seem prudent to ensure that subjects or their surrogates understand study protocols and have sufficient decisional capacity to make well-reasoned choices.

The finding that 5 investigators recalled excluding children from research participation solely because these children were deemed incompetent to provide meaningful assent was unexpected. Researchers who enroll minors are required to obtain assent when possible, and children have a right to refuse to participate, particularly when the research does not present a prospect of direct benefit to the subjects. (Federal Regulations, 2005; American Academy of Pediatrics, 1995) However, it is unclear how investigators should approach the problem of older minors who are unable to comprehend research protocols. These data, which demonstrate that many investigators make no attempt to assess minors’ comprehension of study protocols or their decisional capacity in the assent process while others not only make such assessments, but exclude minors if they are unable to give assent that is meaningful, demonstrates the broad interpretation of the role of assent in research settings.

We based our response rate calculations on methods recommended by the AAPOR and the ISER, which is appropriate for survey studies of this type, however our calculated response rate of 56% was lower than anticipated. While this response rate is suboptimal, other investigator have reported results of physician surveys with similar response rates, (Keating, Zaslavsky, & Ayanian, 1998; Mangus, Dipiero, & Hawkins, 1999; Meier et al., 1998) and many have reported results with significantly lower response rates. (Kim, Galustyan, Sato, Bergholte, & Hennes, 2003; Lacasse, Plante, Martel, & Raby, 2003; Peshkin, Isaacs, Finch, Kent, & Schwartz, 2003; Zachry, Dalen, & Jackson, 2003) Given our relatively low rate of response, there may have been a selection bias in our sample, and due to study design, it is not possible to compare the responder and non-responder groups. One might expect, however, that researchers who assess potential subjects’ comprehension of study protocols and their decisional capacity might have been more likely to respond to a survey on informed consent for research, which would skew our findings to over-represent this population. As such, actual percentage of researchers who assess comprehension of study protocols and/or decisional capacity may be lower than our findings suggest. Further, as with any survey in which respondents might feel certain responses are more acceptable than others, respondents are more likely to answer questions with what they believe to be the “right” answer. As such, our findings may overestimate the number of researchers who assess comprehension of study protocols and capacity.

It is also important to note that we surveyed corresponding authors, many of whom may not have been the individuals who actually obtained informed consent for study participation. As such, these investigators may have been unaware of the actual practices of those obtaining consent, however if study protocols required assessment of comprehension or decisional capacity in the consent process, these authors should have been aware of such requirements. Further, while three investigators stated that they used validated assessment tools, we did not determine whether these respondents had a clear understanding of the requirements of tool validation and whether the tools they used were in fact previously validated. Lastly, it is important to note that our sample was not representative of a broad base of research. We specifically targeted high quality journals assuming that studies published in these journals would have been the most likely to require valid informed consent. We have no data, however, to support this assumption, and are therefore unable to predict the generalizability of our findings. These data can serve, however, as the impetus for further, broader-based projects to assess the current practices of investigators and measure changes over time.

The finding that most investigators who assess comprehension or decisional capacity do not use formal or validated tools raises several questions. Our study was not designed to ascertain how such informal assessments are made, however clearly many investigators rely on such informal methods and indeed exclude subjects if they “fail” such tests. Future work to describe the methods used by investigators could shed greater light on this area, and such information could be used to inform those who develop validated tools. Integration of validated expert practices with the real-world experiences of investigators could yield valid tools that may gain broad acceptance.

Given the high incidence of inadequate comprehension among research subjects and their surrogates demonstrated in previous work, (Criscione et al., 2003; Joffe et al., 2001a; Mason & Allmark, 2000; Rogers et al., 1998; Yuval et al., 2000) we believe it would be prudent for investigators to develop standardized tools to assess potential subjects’ comprehension of study protocols and their decisional capacity. Particularly in cases where research participation entails greater risk, investigators should assess the level of comprehension of study protocols and the decisional capacity of potential subjects, or surrogates of potential subjects, prior to accepting their consent or permission as valid. Such tools must balance the need for comprehensive assessment with the real-world limitations of time and resources. Because validated instruments are readily available, those who recruit research subjects should consider using them, however further work must define “minimum scores” for decisional capacity so that those who recruit subjects may use such tool during study recruitment. Further work should explore how to best balance the need to ensure adequately informed consent with the burdens of assessment on both investigators and research subjects themselves before policy changes can be considered.

Best Practices

Given the importance of ensuring the validity of the consent of research subjects, and the validity of the permission of parents, we recommend that assessments of comprehension and decisional capacity should become a standard requirement during the consent process for all research posing greater than minimal risk to subjects. Further, investigators should consider either using currently available validated tools, or developing novel validated instruments for this purpose.

Research Agenda

Further work should focus on improving consent discussions to better facilitate subjects’ comprehension. Because there will be instances where subjects, or their surrogates, are not capable of comprehending complex research protocols, further work should investigate how the research community should handle such situations. Should these potential subjects be excluded from research? If so, will this limit the generalizability of research findings? When research has the potential to directly benefit an individual, should we allow their participation regardless of the ability to comprehend study protocols? Others have questioned whether comprehension of study protocols is indeed necessary for consent to be deemed valid, (Sreenivasan, 2003) and further work should investigate this argument to resolve the requirement for valid and meaningful consent in the context of research.

Educational Implications

Prior research has demonstrated that many research subjects have incomplete understanding of study protocols and are often unaware of study implications when they volunteer for participation. Our data suggests that many investigators do not attempt to ensure that potential subjects have adequate comprehension and decisional capacity when their consent is accepted. IRB members should consider methods to ensure that research subjects are adequately informed, and should consider requiring documentation of their comprehension in the consent process, particularly when research poses greater than minimal risk and the individual subjects are unlikely to benefit from participation.

Biographies

Dr. Alexander A. Kon is an Associate Professor of Pediatrics and Bioethics at the University of California Davis, and a Greenwall Faculty Scholar in Bioethics. Dr. Kon’s creative work centers on informed consent and decision-making in clinical pediatrics and research involving children, as well as on pediatric palliative care, and he has authored multiple publications on these topics. He is the chair of the Hospital Ethics Committee at the University of California Davis Medical Center, and the ethicist for the University of California Davis Clinical and Translational Research Center. Dr. Kon’s clinical work focuses on the care of critically ill children, and he is the associate director of the Pediatric Intensive Care Unit at the University of California Davis Children’s Hospital.

Mr. Michael Klug completed his undergraduate education at the University of California Davis, and was a volunteer research assistant for this project. Mr. Klug is currently a medical student at St. George’s University.