Abstract
Background
Ethical principles obligate researchers to maximize study participants’ comprehension during the informed consent process for clinical trials. A pilot evaluation of the consent process was conducted during an international clinical trial of treatment for pulmonary tuberculosis to assess the feasibility of conducting an evaluation in a larger population and to guide these future efforts.
Methods
Study staff administered an informed consent assessment tool (ICAT) to a convenience sample of trial participants, measuring comprehension of consent components as derived from the Common Rule and FDA Title 21 Part 50, and satisfaction with the process. Participating site staff completed a consent process questionnaire about consent practices at their respective sites and provided improvement recommendations. ICAT scores and corresponding practices were compared where both were completed.
Results
ICATs (n = 54) were submitted from one site in Spain (n = 10), one in Uganda (n = 30), and five in the United States (n = 14). Participants were primarily male (76%), born in Africa (n = 31, 57%), and had a median age of 27 years (interquartile range [IQR]: 24–42). Median ICAT scores were 80% (IQR: 67–93) for comprehension and 89% (IQR: 78–100) for satisfaction. Ugandan participants scored higher than participants from other sites on comprehension (87% vs. 64%) and satisfaction (100% vs. 78%). Staff from 14 sites completed consent process questionnaires. Median ICAT scores for comprehension and satisfaction were higher at sites that utilized visual aids. Practice recommendations included shorter forms, simpler documents, and supplementary materials.
Conclusions
Participants achieved high levels (≥80%) of comprehension and satisfaction with their current consent processes. Higher ICAT scores at one site suggest an additional evaluation may identify approaches to improve comprehension and satisfaction in future trials. Through this pilot evaluation, complexities and challenges were identified in obtaining consent in a large, international multicenter trial and provided insights for a more robust assessment of the consent process in future trials.
Keywords: Comprehension, informed consent, satisfaction, tuberculosis
The informed consent process is a fundamental part of research and is intended to explain and communicate the risks and rights of participants in clinical trials (U.S. Department of Health and Human Services 2009a; 2011a). Ethical guidelines pertaining to the protection of human subjects, as enumerated in the Belmont Report, include three principles to guide the conduct of clinical trials: respect for persons, beneficence, and justice (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research 1979). U.S. federal regulations under the Department of Health and Human Services, Title 45 Part 46 of the Code of Federal Regulations (45 CFR 46), subpart A, known as the “Common Rule,” and the Food and Drug Administration (FDA) Title 21 Part 50 (21 CFR 50) specify required elements that investigators must describe when they recruit and seek consent from a prospective study participant (Chanaud 2008; U.S. Department of Health and Human Services 2009; 2011a; 2013).
Presentation of these components should occur
under circumstances that provide the prospective subject or the representative sufficient opportunity to consider whether or not to participate and that minimize the possibility of coercion or undue influence. (U.S. Department of Health and Human Services 2009)
Nevertheless, studies of informed consent in clinical trials have that found some patients have limited understanding of the content presented during the consent process (Breese et al. 2007; Crepeau et al. 2011; Hill et al. 2008; Joffe et al. 2001). In response, researchers have proposed alternative methods, including cognitive interviewing techniques, visual aids, culturally appropriate resources, and attention to the oral presentation of the informed consent process’s key elements (Breese et al. 2007; Yates et al. 2009).
The Tuberculosis Trials Consortium (TBTC) is sponsored by the Division of Tuberculosis Elimination of the U.S. Centers for Disease Control and Prevention (CDC) (Sandman et al. 2006). TBTC’s informed consent process has been evaluated previously (Breese et al. 2007; Burman et al. 2003). In light of the U.S. Department of Health and Human Services (DHHS) proposed changes to the Common Rule to accommodate multicenter studies and address the expansion of research involving human participants in new scientific disciplines and venues as well as technical adjustments proposed by the Federal Advance Notice of Proposed Rulemaking to improve the forms and processes used for informed consent (U.S. Department of Health and Human Services and Office of the Secretary 2011), TBTC committed itself to further evaluation of the consent process across the consortium.
Advocating on behalf of research participants, TBTC’s Community Research Advisory Group (CRAG) supported the proposed evaluation efforts. Specifically, CRAG advised that from the perspective of participants, the process for obtaining informed consent was just as important as the statements and specific wording in the consent forms. They suggested TBTC should further evaluate participant comprehension and satisfaction of the process and explore site-specific processes. Additionally, CRAG members hypothesized that the trial site in Kampala, Uganda, provided a useful comparison group because (1) the site has been the consortium’s highest enrolling site for a number of studies, enrolling approximately half of the participants in the clinical trial in which this evaluation was embedded, and (2) the site’s two-step informed consent process, which entailed introducing general research principles to a prospective study participant on one day and presenting study-specific details on another day, may be an innovative approach to maximizing participant comprehension and satisfaction. Comparing the practices at the Uganda site with those at other sites may help to demonstrate which consent practices are associated with differences in comprehension and satisfaction.
The objectives of this pilot evaluation were to measure participant comprehension and satisfaction of the informed consent process in a large, international, multicenter site research study, using an adapted informed consent assessment tool (ICAT) (Breese et al. 2007); gather information about the consent processes at the study sites in a clinical trial; and compare ICAT scores with corresponding site consent practices.
Limited empirical research exists on the administration, comprehension, and satisfaction of informed consent (Kass, Maman, and Atkinson 2005) as part of international clinical trials. Much of the published consent research is derived from studies that have employed hypothetical scenarios or from studies conducted in developed countries and among study participants with relatively high levels of education (Montalvo and Larson 2014; Nishimura et al. 2013). Breese et al. (2007) helped demonstrate the importance of evaluating the effects of education level on informed consent comprehension and satisfaction (Breese et al. 2007). The project described in this report collected data from persons with an infectious disease of major public health importance (World Health Organization 2014) that largely affects marginalized groups (Centers for Disease Control and Prevention 2013) and evaluated participants enrolled in a multisite, international trial. In addition to identifying strengths and weakness of existing consent practices, this project aimed to assess this evaluation approach so as to guide evaluation efforts in future studies.
METHODS
Study Population
From December 2008 through November 2010, TBTC sites enrolled participants into an international, open-label, randomized Phase 2 clinical trial assessing the safety and activity of rifapentine, a long-acting rifamycin antibiotic, in comparison to the standard drug rifampin, during the first 2 months of treatment for pulmonary tuberculosis (TB) (Dorman et al. 2012).
As required for all clinical trials, participants who agreed to enroll were asked to provide informed consent, which was based on the elements of 45 CRF 46 (Common Rule) and 21 CFR 50 (U.S. Department of Health and Human Services 2009; 2013). Between August 2009 and November 2010, participants enrolled in the clinical trial were asked to answer ICAT questions to measure comprehension and satisfaction with the informed consent process. Site staff chose study participants for ICAT administration by convenience. ICATs were administered only to persons who already had enrolled in the clinical trial. A volunteer staff member at participating sites also completed an open-ended questionnaire about their site-specific consent process. All site staff members enrolling in the clinical trial were invited to participate in this evaluation.
Ethics Statement
TBTC Study 29, the clinical trial within which this project was conducted, was approved in May 2008 by the institutional review boards (IRBs) of the CDC and participating institutions, and was registered with ClinicalTrials.gov (Study 29: NCT00694629). Some local IRBs ceded oversight of the TBTC protocols to the CDC’s IRB (U.S. Department of Health and Human Services 2009). For this project, the CDC’s National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention determined that the assessment of the consent process constituted a program evaluation and did not require IRB review (Centers for Disease Control and Prevention 2010). Additionally, each site followed local review policies and procedures.
Informed Consent Assessment Tool
The original Informed Consent and Authorization Assessment Tool (ICAAT) was developed and validated by Breese et al. (2007) to assess participants’ comprehension and satisfaction with the consent process. The ICAAT contained 28 items to determine how persons from diverse backgrounds experience the consent process. It included 21 items to assess comprehension of representative required elements of informed consent and seven items to assess satisfaction with the process (Breese et al. 2007). The ICAAT was designed to be self-administered with three response options for each question (“Agree,” “Not Sure,” and “Disagree”) and included one set of paired questions that required opposite answers to assess the consistency of responses. For subsequent uses, questions pertinent to the Health Insurance Portability and Accountability Act (HIPAA) were deleted (Breese, personal communication) and the tool was renamed the Informed Consent Assessment Tool (ICAT) (Breese et al. 2007).
For this evaluation, the ICAT questions were adapted from a written, self-administered format to a verbal, interviewer-administered format in an effort to reduce barriers related to low literacy (Supplement 1). The ICAT was simultaneously translated into the participant’s language by the ICAT administrator, if needed. Study staff members, other than those who had administered informed consent, administered ICATs.
The adapted ICAT used in this evaluation contained 15 questions assessing comprehension of required consent components and nine evaluating participant satisfaction with the consent process (Supplement 1). Three response options (“Yes,” “No,” and “Not Sure”) were provided for each item. “Not sure” responses were grouped with the incorrect/unfavorable responses for analysis. Three questions were repeated in slightly modified versions to measure response consistency (Q.3 and Q.11; Q.6 and Q16; Q.10 and Q.17). ICATs were administered once during the 8-week intensive study phase when participants underwent clinical evaluations every 2 weeks. Participant scores were calculated based on the percentage of correct responses for comprehension (out of 15) or favorable responses for satisfaction (out of nine). ICAT scores equal to or greater than 80% were considered high.
Consent Process Questionnaire
To assess the informed consent process at each site, researchers created a 27-item, open-ended questionnaire, based on the informed consent process used at the TBTC study site in Kampala, Uganda (Supplement 2). Questionnaire items included inquiries about site staff who administered the consent process, the use of interpreters, time spent administering consent, the number of visits to complete the consent process, and the approaches for obtaining consent from persons with low literacy. Additional items addressed approaches to assess participant understanding such as the use of short forms or translated forms, the use of visual aids, the presence or absence of family and friends during the consent process, and protocols related to a participant signing the form. Suggestions for improving the informed consent process were also solicited.
Data Analysis
Comprehension and satisfaction scores at the site in Kampala, Uganda, were compared to other sites with the Wilcoxon rank-sum test (Hollander, Wolfe, and Chicken 2013); a p-value <.05 was considered statistically significant (SAS v. 9.3, SAS Institute, Cary, NC). Medians and nonparametric tests were used for a more conservative approach in this evaluation.
Answers to the site consent process questionnaire were evaluated by identifying common themes among the responses, tabulated for comparison, and reviewed by three of the investigators. Finally, among sites where both informed consent practices and ICATs were reported, median ICAT scores were compared by reported consent practices to determine which practices were related to higher comprehension and satisfaction scores.
RESULTS
The clinical trial included 24 TBTC sites and 531 enrolled patients (Figure 1). Fifteen of the 24 sites participated by completing one or both of the data collection tools. ICATs were administered to 54 participants at seven (29%) of the 24 sites. These 54 participants represent 76% of the 71 potential participants from the seven sites during the ICAT collection period. There were differences in the distribution of age, birthplace, and screening country between the 17 participants who were eligible for ICAT administration but did not respond and the 54 participants who did respond (data not shown). Information regarding the reasons for these individuals’ nonparticipation was not available, and the small numbers limit the value of additional formal comparisons.
Figure 1.
Analysis groups for pilot informed consent evaluation.
Site process questionnaires were completed at 14 sites. Six sites participated in the collection of both instruments by completing the consent process questionnaire and administering 51 ICATs. At one site, ICATs were administered to three patients but a questionnaire was not completed. While all site staff were invited to participate, some declined because the site did not have additional capacity to participate in activities outside of the initial protocol procedures or because local IRBs required additional approval for the evaluation.
Demographics
ICATs were collected by site staff in 2010 at one site in Spain (10 ICATs), one site in Uganda (30 ICATs), and five sites in the United States (14 ICATs). Participants (Table 1) were primarily male (76%) with a median age of 27 years (interquartile range [IQR]: 24–42), with the majority having a 12th-grade education or less (n = 42, 78%). Seventeen percent (n = 9) reported that they had at least some college education; 6% (n = 3) reported education as “other,” with no further information.
Table 1.
Demographics by Informed Consent Assessment Tool (ICAT) Scores
| All
|
Kampala, Uganda
|
Other sites
|
||||
|---|---|---|---|---|---|---|
| n (%) | Median (IQR)* | n (%) | Median (IQR)* | n (%) | Median (IQR)* | |
| Comprehension Scores | ||||||
|
| ||||||
| Total | 54 | 80 (67–93) | 30 (56) | 87 (80–93) | 24 (44) | 64 (57–87) |
| Sex | ||||||
| Female | 13 (24) | 87 (67–93) | 5 (17) | 87 (80–87) | 8 (33) | 77 (64–97) |
| Male | 41 (76) | 80 (67–93) | 25 (83) | 87 (73–93) | 16 (67) | 60 (50–73) |
| Age | ||||||
| Median Age [IQR] | 54 | 27 (24–42) | 30 | 25 (22–33) | 24 | 41 (26–53) |
| 0 to ≤27 years | 28 (52) | 87 (73–93) | 20 (67) | 87 (77–93) | 8 (33) | 73 (60–93) |
| 28 years & older | 26 (48) | 73 (60–87) | 10 (33) | 84 (80–87) | 16 (67) | 60 (44–80) |
| Race | ||||||
| American Indian/Alaskan Native | 1 (2) | 60 N/A | 1 (4) | 60 N/A | ||
| Asian/Hawaiian/Pacific Islander | 1 (2) | 100 N/A | 1 (4) | 100 N/A | ||
| Black | 36 (67) | 87 (73–93) | 30 (100) | 87 (80–93) | 6 (25) | 80 (73–93) |
| White | 12 (22) | 64 (44–80) | 12 (50) | 64 (44–80) | ||
| Not Reported/Unknown | 4 (7) | 60 (50–60) | 4 (17) | 60 (50–60) | ||
| Birth Place | ||||||
| US/Canada | 4 (7) | 87 (67–100) | 4 (17) | 87 (67–100) | ||
| Americas-Other | 11 (20) | 60 (60–87) | 11 (46) | 60 (60–87) | ||
| Africa | 31 (57) | 87 (73–93) | 30 (100) | 87 (80–93) | 1 (4) | 73 N/A |
| Europe | 7 (13) | 53 (40–67) | 7 (29) | 53 (40–67) | ||
| Western Pacific | 1 (2) | 100 N/A | 1 (4) | 100 N/A | ||
| Screening Country | ||||||
| Spain | 10 (19) | 60 (40–67) | 10 (42) | 60 (40–67) | ||
| USA | 14 (26) | 73 (60–93) | 14 (58) | 73 (60–93) | ||
| Uganda | 30 (56) | 87 (80–93) | 30 (100) | 87 (80–93) | ||
| Education | ||||||
| ≤8th grade | 18 (33) | 67 (60–87) | 10 (33) | 87 (73–87) | 8 (33) | 54 (40–60) |
| >8–≤12th grade | 24 (44) | 80 (70–93) | 15 (50) | 87 (80–93) | 9 (38) | 67 (60–73) |
| >12th grade | 9 (17) | 87 (73–100) | 2 (7) | 77 (73–80) | 7 (29) | 93 (73–100) |
| Other | 3 (6) | 93 (80–93) | 3 (10) | 93 (80–93) | ||
|
| ||||||
| Satisfaction Scores | ||||||
|
| ||||||
| Total | 54 | 89 (78–100) | 30 (56) | 100 (89–100) | 24 (44) | 78 (67–100) |
| Sex | ||||||
| Female | 13 (24) | 89 (78–100) | 5 (17) | 89 (89–100) | 8 (33) | 89 (73–100) |
| Male | 41 (76) | 89 (78–100) | 25 (83) | 100 (89–100) | 16 (67) | 78 (67–95) |
| Age | ||||||
| Median Age [IQR] | 54 | 27 (24–42) | 30 | 25 (22–33) | 24 | 41 (26–53) |
| 0 to ≤27 years | 28 (52) | 95 (78–100) | 20 (67) | 100 (89–100) | 8 (33) | 78 (50–95) |
| 28 years & older | 26 (48) | 89 (78–100) | 10 (33) | 100 (78–100) | 16 (67) | 84 (73–100) |
| Race | ||||||
| American Indian/Alaskan Native | 1 (2) | 89 N/A | 1 (4) | 89 N/A | ||
| Asian/Hawaiian/Pacific Islander | 1 (2) | 100 N/A | 1 (4) | 100 N/A | ||
| Black | 36 (67) | 100 (78–100) | 30 (100) | 100 (89–100) | 6 (25) | 62 (44–100) |
| White | 12 (22) | 78 (73–100) | 12 (50) | 78 (73–100) | ||
| Not Reported/Unknown | 4 (7) | 84 (78–89) | 4 (17) | 84 (78–89) | ||
| Birth Place | ||||||
| US/Canada | 4 (7) | 100 (100–100) | 4 (17) | 100 (100–100) | ||
| Americas-Other | 11 (20) | 78 (67–89) | 11 (46) | 78 (67–89) | ||
| Africa | 31 (57) | 100 (89–100) | 30 (100) | 100 (89–100) | 1 (4) | 22 N/A |
| Europe | 7 (13) | 78 (67–100) | 7 (29) | 78 (67–100) | ||
| Western Pacific | 1 (2) | 100 N/A | 1 (4) | 100 N/A | ||
| Screening Country | ||||||
| Spain | 10 (19) | 78 (67–100) | 10 (42) | 78 (67–100) | ||
| USA | 14 (26) | 84 (67–100) | 14 (58) | 84 (67–100) | ||
| Uganda | 30 (56) | 100 (89–100) | 30 (100) | 100 (89–100) | ||
| Education | ||||||
| ≤8th grade | 18 (33) | 89 (78–100) | 10 (33) | 100 (89–100) | 8 (33) | 78 (73–89) |
| >8–≤12th grade | 24 (44) | 89 (78–100) | 15 (50) | 100 (89–100) | 9 (38) | 78 (67–89) |
| >12th grade | 9 (17) | 100 (100–100) | 2 (7) | 100 (100–100) | 7 (29) | 100 (78–100) |
| Other | 3 (6) | 100 (67–100) | 3 (10) | 100 (67–100) | ||
IQR: Interquartile range.
ICATs were completed in the same language in which participants had discussed informed consent, with the possible exception of one participant for whom the language used to administer the ICAT was not specified. The majority of ICATs from the Kampala, Uganda, site were completed on the day of enrollment, whereas at other sites, ICATs were completed an average of 15 days after enrollment.
Comprehension and Satisfaction of Consent Process: ICAT Scores
The median ICAT scores were 80% (IQR: 67–93) on the comprehension scale and 89% (IQR: 78–100) on the satisfaction scale. The median comprehension score for participants in Kampala (87%, IQR: 80–93) was higher than the median for those enrolled at other sites (64%, IQR: 57–87) (p = .002 by the Wilcoxon rank-sum test); this was also true for satisfaction scores with 100% (IQR: 89–100) as the median score for those enrolled in Kampala and 78% (IQR: 67–100) for those enrolled at other sites (p = .017). For the three paired ICAT questions, 67% (36/54) of the responses were consistent for comprehension questions 3 and 11, and 82% (44/54) of the responses were consistent for comprehension questions 6 and 16. For the satisfaction questions 10 and 17, 87% (47/54) of the responses were consistent.
For seven of the 15 ICAT comprehension questions, the overall proportion of correct or favorable responses was less than 80% (Table 2). Fifty-seven percent of individuals were unclear about whether their private information could be seen by any medical person (ICAT Q.9), and 22% were unclear that their names could not be used in presentations of the study findings (Q.14). In responding to the paired questions (Q.6 and 16) about whether study staff could contact friends or family without their approval, 48% and 33% did not reply with a correct or favorable response. At least a quarter of participants did not demonstrate a clear understanding that they could terminate study participation at any time (Q.11: 26%) and that they could receive medical care elsewhere to treat their disease (Q.22: 35% and Q.24: 28%). For the satisfaction questions, almost all questions had a favorable response (≥80%), except for one question where one-third of the participants were unsure or agreed that there was too much information in the consent forms (Q.2).
Table 2.
ICAT* Questions: Proportion of Correct/Favorable Responses by Common Rule
| Correct/Favorable Response | Total (n = 54)/n (%) |
Common Rule Component | ||
|---|---|---|---|---|
| Comprehension | ||||
|
| ||||
| 1 | Are you taking part in a research study? | Yes | 52 (96) | (1) Research |
| 3 | Now that you are in the study, do you have to stay in it even if you do not want to? | No | 43 (80) | (8) Voluntariness |
| 5 | Did you have to sign the papers because your doctor gave them to you? | No | 43 (80) | (8) Voluntariness |
| 6 | Can the study staff contact your family or friends without your approval? | No | 28 (52) | (5) Confidentiality |
| 9 | May your private information from this study be seen by any medical person? | No | 23 (43) | (5) Confidentiality |
| 11 | Can you stop taking part in this study at any time? | Yes | 40 (74) | (8) Voluntariness |
| 13 | Will you be told about any important changes in this study? | Yes | 49 (91) | (b) Additional Elements: (5) Significant new findings |
| 14 | Can your name be used for presentations of this study? | No | 42 (78) | (5) Confidentiality |
| 16 | Does the study staff need your approval to contact your family and friends? | Yes | 36 (67) | (5) Confidentiality |
| 18 | Do you know whom you can contact if you have a question about this study? | Yes | 50 (93) | (7) Questions |
| 19 | Can you still get care for your illness, even if you don’t want to be in this study? | Yes | 47 (87) | (4) Alternative Procedures |
| 21 | Could the treatment you are receiving cause you to have side effects? | Yes | 46 (85) | (2) Risks |
| 22 | Must you be in the study to get medical care for tuberculosis? | No | 35 (65) | (4) Alternative Procedures |
| 23 | Will this study be six to nine months long? | Yes | 50 (93) | (1) Research |
| 24 | Is the treatment you are receiving in this study your only option? | No | 39 (72) | (4) Alternative Procedures |
|
| ||||
| Satisfaction | ||||
|
| ||||
| 2 | Was there too much information in the papers you signed? | No | 36 (67) | |
| 4 | Did someone explain the study to you in words that you understood? | Yes | 52 (96) | |
| 7 | Were there questions you wanted to ask about this study but did not ask? | No | 43 (80) | |
| 8 | Did you sign the papers to be part of this study without completely reading them? | No | 43 (80) | |
| 10 | Did you have enough time to read the papers you signed to be part of this study? | Yes | 47 (87) | |
| 12 | Did the study staff answer your questions about this study? | Yes | 48 (89) | |
| 15 | Were you confused by the papers you signed for this study? | No | 51 (94) | |
| 17 | Did you need more time to read the papers you signed to be part of this study? | No | 47 (87) | |
| 20 | Do you have a copy of all the papers you signed? | Yes | 51 (94) | |
Bolded questions indicate questions with <80% of the TOTAL responding with correct or favorable response.
ICAT: Informed Consent Assessment Tool.
The overall median ICAT score for comprehension for those reporting an eighth-grade education or less was 67% (IQR: 60–87); participants with more than an eighth-grade education but no college scored 80% (IQR: 70–93); and those with at least some college scored 87% (IQR: 93–100). For satisfaction, the median score for those with at least some college education (IQR: 100–100) was 100%, versus 89% (IQR: 78–100) for those with a high school education or less.
Characteristics of the Consent Process
Staff members from 14 sites in five countries (Canada [n = 1], South Africa [n = 2], Spain [n = 1], Uganda [n = 1], and the United States [n = 9]) completed the consent process questionnaire (Table 3). At 13 of the sites, the study coordinator was reported to be one of the persons who administered informed consent. Staff members from 12 sites reported not using a generic short consent form, and two of these stated that they are not permitted to do so by their respective IRBs (U.S. Department of Health and Human Services 2009). Nearly all site staff surveyed (n = 13) reported that they allowed patients to defer their enrollment decision until a later follow-up visit.
Table 3. Consent Process Questionnaire & ICAT*.
This table shows overall numbers and median ICAT scores, demonstrating comprehension and satisfaction according to whether the practice identified in the question is utilized at the site. Results are shown separately for Kampala and other sites.
| Questionnaire Responses & Corresponding Median ICAT Scores‡
|
|||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Questionnaire† | Yes
|
No
|
|||||||||
| Questionnaire | Sites (Yes) n = 14 |
Sites | # Sites | # Patients | Comp % | Sat % | # Sites | # Patients | Comp % | Sat % | |
| 3 | Do you introduce principles of informed consent separately from discussion of the Study ##-specific informed consent form? | 8 | OVERALL | 4 | 44 | 80 | 100 | 2 | 7 | 60 | 89 |
| Kampala | 1 | 30 | 87 | 100 | … | … | … | … | |||
| Other Sites | 3 | 14 | 64 | 78 | 2 | 7 | 60 | 89 | |||
| 4 | Do you use the short consent form? | 2 | OVERALL | 3 | 18 | 60 | 78 | 3 | 33 | 87 | 100 |
| Kampala | … | … | … | … | 1 | 30 | 87 | 100 | |||
| Other Sites | 3 | 18 | 60 | 78 | 2 | 3 | 100 | 100 | |||
| 6 | Are interpreters (professional/non-professional) used? | 10 | OVERALL | 5 | 21 | 60 | 78 | 1 | 30 | 87 | 100 |
| Kampala | … | … | … | … | 1 | 30 | 87 | 100 | |||
| Other Sites | 5 | 21 | 60 | 78 | … | … | … | … | |||
| 9 | Are patients allowed to defer their decision until a later time? | 13 | OVERALL | 6 | 51 | 80 | 89 | … | … | … | … |
| Kampala | 1 | 30 | 87 | 100 | … | … | … | … | |||
| Other Sites | 5 | 21 | 60 | 78 | … | … | … | … | |||
| 17 | Are materials used during the consent to help explain the topics? | 8 | OVERALL | 4 | 36 | 87 | 100 | 2 | 15 | 60 | 78 |
| Kampala | 1 | 30 | 87 | 100 | … | … | … | … | |||
| Other Sites | 3 | 6 | 73 | 89 | 2 | 15 | 60 | 78 | |||
| 18 | May friends/family members accompany the patient during the consent process? | 14 | OVERALL | 6 | 51 | 80 | 89 | … | … | … | … |
| Kampala | 1 | 30 | 87 | 100 | … | … | … | … | |||
| Other Sites | 5 | 21 | 60 | 78 | … | … | … | … | |||
| 23 | Does your site require a witness to the signature in any or all cases? | 10 | OVERALL | 6 | 51 | 80 | 89 | … | … | … | … |
| Kampala | 1 | 30 | 87 | 100 | … | … | … | … | |||
| Other Sites | 5 | 21 | 60 | 78 | … | … | … | … | |||
| 8 | Length of informed consent process | 15–45 minutes
|
60–120 minutes
|
2–3 hrs
|
|||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| # Sites | # Patients | Comp % | Sat % | # Sites | # Patients | Comp % | Sat % | # Sites | # Patients | Comp % | Sat % | ||
| Kampala | … | … | … | … | … | … | 1 | 30 | 87 | 100 | |||
| Other | 2 | 7 | 60 | 89 | 2 | 4 | 73 | 78 | … | … | |||
ICAT: Informed Consent Assessment Tool.
Sites that answered “Yes” on the consent process questionnaire.
Sites that answered the consent process questionnaire and had participants complete ICATs.
Comp D Comprehension & Sat D Satisfaction.
Site staff reported using different procedures for obtaining consent. For example, staff members from two sites read the consent form along with the patient; one had the patients silently read the form to themselves. Depending on the patients’ needs, staff members at three sites would read the consent form aloud to the patient. The length of time site staff members reported for administering consent ranged from less than 1 hour to up to 3 hours.
When asked to provide their overall impressions of the consent process, staff members from 13 of the 14 sites reported on concerns expressed by study participants. Staff members from six sites shared 11 different concerns related to risks and safety (e.g., possible drug side effects and toxicity). Staff members from three sites reported six different concerns related to study and treatment procedures.
When asked to provide suggestions for improving the consent process, staff members from seven sites recommended shorter documents and three suggested creating simpler, more understandable consent forms. Additional suggestions included ensuring that the administration of the consent process includes adequate time for patients to consider their decisions from three site staff members, and the use of supplementary materials such as explanation booklets, brochures, posters, or videos by eight site staff members. Site staff members also proposed utilizing a questionnaire to assess participant understanding at the time of requesting consent, providing a setting free from interruptions during consent administration, and providing on-going education on consent topics.
ICAT Scores Related to Consent Processes
Six of the sites completed both the consent process questionnaire and the administration of ICATs (n = 44). Three of the consent process questionnaire items and the corresponding ICAT scores from Table 3 are provided here. The first explores the question of whether the Kampala site’s two-step process might be responsible for their higher comprehension and satisfaction results. The other two explore approaches that might be most modifiable in the informed consent process.
A. Do You Introduce Principles of Informed Consent Separately from Discussion of the Study 29-Specific Informed Consent form? (CRAG Hypothesis)
Staff from four sites that administered 44 ICATs reported they discussed concepts of informed consent separately from the study-specific informed consent (overall comprehension score: 80%, overall satisfaction score: 100%). The Uganda site was one of the sites that engaged in this practice. Participants at the Uganda site (n = 30) had a median score of 87% on the comprehension scale; however, patients at sites other than Uganda (sites [n = 3], patients [n = 14]), where consent principles were discussed separately from the study-specific consent form, had a median score of 64% on the comprehension scale. Similarly, satisfaction in Uganda was 100%, compared to 78% at other sites with a two-step process. Participants at two sites (patients [n = 7]) where principles of consent were not reported to be discussed separately from the discussion of the study-specific consent form scored 60% for comprehension and 89% for satisfaction.
B. Are Materials Used During the Consent Process to Help Explain the Topics?
The median comprehension score for participants at sites (n = 4) where staff used visual aids was 87% (patients [n = 36]), whereas the median comprehension score for participants at sites (n = 2) where staff did not use visual aids was 60% (patients [n = 15]). Uganda site staff (patients n = 30) used visual aids; participants at this site attained 87% for comprehension and 100% for satisfaction. Participants from sites outside of Uganda (n = 3) (patients [n = 6]) had a median comprehension score of 73% and a satisfaction score of 89% where visual aids were used. At sites where visual aids were not used (sites [n = 2], patients [n = 15]), comprehension was 60% and satisfaction was 78%. Visual aids included informal written instructions, diagrams, pictures, flyers, a sample package of TB medication, and pills on a sheet to demonstrate differences between treatments.
C. How Long Does the Informed Consent Process Take?
The median comprehension scores for participants at sites where less than 45 minutes was spent on informed consent (sites [n = 2], patients [n = 7]) was 60%, 73% for sites where 60 to 120 minutes was spent (sites [n = 2], patients [n = 4]), and 87% for the one site (Uganda) where 120 to 180 minutes was spent (patients [n = 30]). Satisfaction scores varied across time spent administering informed consent. The median satisfaction score for participants at sites where less than 60 minutes was spent was 89%. Sites where 60 to 120 minutes was spent had a median score of 78%, while the score was 100% at the site where 120 to 180 minutes was spent.
DISCUSSION
Evaluation of informed consent practices may offer insight for investigators to improve consent administration and maximize the alignment of clinical trials’ conduct with the principles of respect, beneficence, and justice enunciated in the Belmont Report (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research 1979). Our pilot evaluation gathered information about the consent process at study sites of an international, multisite clinical trial and explored the implementation of an adapted ICAT interview tool.
Among the 54 participants in TBTC Study 29 to whom ICATs were administered, median scores were high, with 80% (IQR: 67–93) for comprehension and 89% (IQR: 78–100) for satisfaction. On average, participants at the Uganda TBTC site scored higher than those at other sites on comprehension (87% vs. 64%) and satisfaction (100% vs. 78%). Our evaluation found that median comprehension and satisfaction scores were higher among persons with more education, consistent with the findings of Breese et al. (2007).
Seven of the ICAT questions on the comprehension scale were of particular interest for the purposes of this study. These questions each had an overall score of less than 80% correct or favorable response, suggesting potential areas for improvement (Table 2). These seven questions are rooted in three components of the Common Rule and FDA 21 CFR 50: (1) confidentiality, (2) voluntariness, and (3) alternative procedure options (U.S. Department of Health and Human Services 2009; 2013). Not all participants understood that their names would not be used for presentations or that study staff could not contact their family and friends without approval. Additionally, it was not well understood that they could stop taking part in the study at any time or that medical care for TB could be obtained separately from the study, similar to findings from a previous report (Kiguba et al. 2012). It is important to note that a participant’s misunderstanding of the right to confidentiality would not undermine the effectiveness of this protection; however, the right to withdraw from a study is under the volitional control of the participant. As such, it is essential for participants to understand that they have rights and can take steps to exercise them.
With one exception, all ICAT satisfaction questions were 80% or higher. The only exception was for a question where 33% were either unsure or felt that the papers they signed had too much information, which supports the call for shorter informed consent documents. Recently, researchers reported the importance of using a “context-specific” approach where consent was described using words and analogies that are locally understood (Corneli et al. 2012). Our evaluation also found that site-specific approaches, including visual aids, facilitated understanding of selected topics. Such methods could be evaluated further to refine strategies that are culturally appropriate and most effective at communicating the consent information and study participants’ rights.
Limitations
Limitations of this study included low and uneven participation in the consent evaluation across study sites. This can be attributed to some sites not being staffed for activities peripheral to the primary study protocol, investigators being skeptical of the value of evaluating their informed consent processes, and differences in TB prevalence and research capacity.
The Uganda site enrolled approximately 35% of the participants in the clinical trial (Dorman et al. 2012). Of the 54 participants who volunteered to complete an ICAT, 30 (56%) were from the Uganda site. The participants from Uganda are not representative of participants at other sites.
The revised ICAT used in this evaluation was based on a self-administered form validated by Breese et al. (2007). After adapting the statements to be interviewer-administered, the tool was not revalidated, and participants may have responded to satisfaction questions in a socially desirable manner. In addition, the ICAT was only provided to the participating sites in English and study staff simultaneously translated questions for some patients, which had the potential to introduce variability.
Additional biases may have been introduced through the use of convenience sampling, variability among the staff members who administered consent, the possibility that staff members inadvertently led participants in responding to ICAT questions, and the time between when informed consent was conducted and when the ICAT was administered. Furthermore, with each site submitting a single questionnaire, variability in administrative practices amongst multiple staff members at a site may not have been fully captured. Finally, not all sites that reported consent practices administered ICATs, limiting opportunities to correlate practices with ICAT scores. These factors, along with a small sample size, limit the useful depth of analysis, precluding, for example, multivariate analysis.
Implications
Clinical trial sites expend considerable effort, striving for high comprehension of informed consent principles among trial participants. The findings of our pilot evaluation present opportunities to improve the consent process and the assessment of the process itself, particularly more focused training, including the need to reemphasize specific components of the Common Rule and FDA 21 CFR 50 multiple times throughout a study (U.S. Department of Health and Human Services 2009; 2013).
Observers have speculated that a tool similar to the ICAT could be used throughout the study to reemphasize the rights and protections afforded to study participants. Adapting an ICAT for patient-level, real-time use would require a development and validation process. A more immediate course of action could include training site staff members to routinely assess participants’ comprehension both during and after presentation of specific components of the Common Rule and FDA 21 CFR 50 and evaluating the impact of the educational intervention.
Our pilot evaluation provides insights for a more robust evaluation of the consent process in future multicenter clinical trials. Specifically, it may be beneficial to incorporate the evaluation of the consent process into the main study protocol to ensure all sites’ participation; dedication of resources to evaluation activities; and the systematic collection of data. Additionally, it may be advantageous to develop a tool to quantify site-specific consent practices to help pinpoint specific practices associated with higher comprehension and satisfaction.
As seen in this pilot evaluation, further investigation is needed to identify reasons for the Uganda site’s higher comprehension and satisfaction scores. While this evaluation supported CRAG’s hypothesis of comprehension and satisfaction being higher at the TBTC site in Uganda, it did not support the additional hypothesis that a two-part informed consent process would result in higher comprehension among participants outside of Uganda. A median satisfaction score of 100% from participants surveyed at the Ugandan site is impressive, yet further evaluation is needed to determine whether the following influenced the score: cultural factors, the influence of a staff-administered satisfaction survey on patient responses, patients’ perceptions of the care available in a research setting compared to regular program care, the time period between enrollment and ICAT administration, length of consent process, and median age of the participants.
The IRBs at many sites declined to take advantage of the short consent form as permitted by the Code of Federal Regulations (U.S. Department of Health and Human Services 2009; 2013). Future evaluation efforts of the consent process could include using a long version of the consent form as the “written summary of what is to be said” coupled with use of the short form for all participants. The short form could also be used throughout the study to reinforce important topics. When considering the benefits that may be derived from use of the short consent form, researchers must also consider additional resources that are needed. For example, per regulations, another individual may be needed to witness the oral presentation (bU.S. Department of Health and Human Services 2011b).
Finally, while ethical regulations stress the importance of comprehension, participant satisfaction with the informed consent process also deserves consideration. The ICAT’s satisfaction questions focus on actions that, in theory, contribute to comprehension. This calls attention to the challenge of balancing these two domains in order to achieve a “successful” informed consent from both the participants’ and investigators’ perspectives.
CONCLUSIONS
This project presents an approach to how the quality of the informed consent process may be systematically evaluated and illustrates logistical issues to consider when working with persons who possess varying levels of education. Overall, this pilot evaluation found high levels of participant comprehension and satisfaction with the consent process at sites participating in an international clinical trial.
In addition, both comprehension and satisfaction were higher in Uganda, which was consistent with CRAG’s original hypothesis. However, with only small numbers of ICATs, the underlying reasons for these higher scores could not be determined. Based on the scores of individual ICAT questions, three components of the Common Rule and FDA 21 CFR 50 may need additional emphasis when sites review consent information with potential participants: confidentiality, voluntary participation, and alternative procedure options. Furthermore, re-review of rights and procedures throughout study participation could prove beneficial.
The staff at study sites also helped identify opportunities to improve the informed consent process, such as creating educational materials to facilitate explanation of consent concepts, especially those for which the study participants demonstrated lower comprehension. Additionally, greater acceptance of the short consent process form may address participant concerns that there was too much information in the papers they signed.
Findings from this pilot evaluation will assist in the design of a larger study and in the revision of data collection tools. Continuing this work may provide insights to improve comprehension and satisfaction with the informed consent process.
Supplementary Material
Acknowledgments
We thank all of the study participants who contributed their time to this study. We greatly appreciate all TBTC site study staff members who participated, as well as those who were willing to participate yet were constrained by logistical issues (please see full list of site staff in Dorman et al. 2012). We appreciate the invaluable guidance offered by TBTC’s Community Research Advisory Group (CRAG). We acknowledge Nigel Scott, Pei-Jean Feng, and Charles Heilig for their outstanding statistical and epidemiological support. We appreciate the technical and editorial review provided by Andrew Vernon, John L. Johnson, Amera Khan, and Wanda Walton.
FUNDING
This work was supported by the U.S. Centers for Disease Control and Prevention (CDC), Division of Tuberculosis Elimination, National Center for HIV, Viral Hepatitis, STD, and TB Prevention. Sanofi donated the rifapentine used in this study. Since 2007, Sanofi has donated $2.26 million to the CDC Foundation to supplement available funding for TBTC rifapentine research.
ROLE OF THE FUNDER/SPONSOR
Sanofi had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the article; and decision to submit the article for publication.
Footnotes
SUPPLEMENATAL MATERIAL
Supplemental data for this article can be accessed on the publisher’s website.
PREVIOUS PRESENTATIONS
Preliminary results of this evaluation were presented at a poster presentation at the 2011 Advancing Ethical Research Conference, Public Responsibility in Medicine & Research, Poster No. 31, Programmatic Submission, December 1–4, 2011, National Harbor, MD.
AUTHOR CONTRIBUTIONS
Kimberley N. Chapman: design, analysis, writing, editing, review; Eric Pevzner: analysis, writing, editing, review; Joan M. Mangan: writing, editing, review; Peter Breese: analysis, writing, editing, review; Dorcas Lamunu: analysis, writing, editing, review; Robin Shrestha-Kuwahara: design, editing, review; Joseph G. Nakibali: editing, review; and Stefan V. Goldberg: design, analysis, writing, editing, review.
DISCLAIMER
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
CONFLICTS OF INTEREST
Kimberley N. Chapman is employed by the CDC Foundation.
ETHICAL APPROVAL
TBTC Study 29, the clinical trial within which this project was conducted, was approved in May 2008 by the institutional review boards (IRBs) of the CDC and participating institutions, and was registered with ClinicalTrials.gov (Study 29: NCT00694629). Some local IRBs ceded oversight of the TBTC protocols to the CDC’s IRB (U.S. Department of Health and Human Services 2009). For this project, the CDC’s National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention determined that the assessment of the consent process constituted a program evaluation and did not require IRB review (Centers for Disease Control and Prevention 2010). Additionally, each site followed local review policies and procedures.
Contributor Information
Kimberley N. Chapman, Division of Tuberculosis Elimination, Centers for Disease Control and Prevention and CDC Foundation
Eric Pevzner, Division of Tuberculosis Elimination, Centers for Disease Control and Prevention.
Joan M. Mangan, Division of Tuberculosis Elimination, Centers for Disease Control and Prevention
Peter Breese, Denver Public Health and University of Colorado Health Sciences Center.
Dorcas Lamunu, Uganda–Case Western Reserve University Research Collaboration.
Robin Shrestha-Kuwahara, Division of Tuberculosis Elimination, Centers for Disease Control and Prevention.
Joseph G. Nakibali, Uganda–Case Western Reserve University Research Collaboration and Community Research Advisory Group, Tuberculosis Trials Consortium
Stefan V. Goldberg, Division of Tuberculosis Elimination, Centers for Disease Control and Prevention and Tuberculosis Trials Consortium (TBTC)
References
- Breese PE, Burman WJ, Goldberg S, Weis SE. Education level, primary language, and comprehension of the informed consent process. Journal of Empirical Research on Human Research Ethics. 2007;2(4):69–79. doi: 10.1525/jer.2007.2.4.69. http://dx.doi.org/10.1525/jer.2007.2.4.69. [DOI] [PubMed] [Google Scholar]
- Burman WJ, Weis Breese S, Bock N, Bernardo J, Vernon A, the Tuberculosis Trials Consortium The effects of local review on informed consent documents from a multicenter clinical trials consortium. Controlled Clinical Trials. 2003;24(3):245–255. doi: 10.1016/s0197-2456(03)00003-5. http://dx.doi.org/10.1016/S0197-2456(03)00003-5. [DOI] [PubMed] [Google Scholar]
- Centers for Disease Control and Prevention. Distinguishing public health research and public health nonresearch. 2010 Available at: http://www.cdc.gov/od/science/integrity/docs/cdc-policy-distinguishing-public-health-research-nonresearch.pdf.
- Centers for Disease Control and Prevention. Tuberculosis— United States, 1993–2010. Morbidity and Mortality Weekly Report. 2013;62(suppl. 3):149–154. [Google Scholar]
- Chanaud CM. Determination of required content of the informed consent process for human participants in biomedical research conducted in the U.S. A practical tool to assist clinical investigators. Contemporary Clinical Trials. 2008;29(4):501–506. doi: 10.1016/j.cct.2007.11.006. http://dx.doi.org/10.1016/j.cct.2007.11.006. [DOI] [PubMed] [Google Scholar]
- Corneli AL, Sorenson JR, Bentley ME, et al. Improving participant understanding of informed consent in an HIV-prevention clinical trial: A comparison of methods. AIDS and Behavior. 2012;16(2):412–421. doi: 10.1007/s10461-011-9977-z. http://dx.doi.org/10.1007/s10461-011-9977-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Crepeau AE, McKinney BI, Fox-Ryvicker M, Castelli J, Penna J, Wang ED. Prospective evaluation of patient comprehension of informed consent. Journal of Bone and Joint Surgery. 2011;93(19):e114, 1–7. doi: 10.2106/JBJS.J.01325. http://dx.doi.org/10.2106/JBJS.J.01325. [DOI] [PubMed] [Google Scholar]
- Dorman SE, Goldberg S, Stout JE, et al. Substitution of rifapentine for rifampin during intensive phase treatment of pulmonary tuberculosis: Study 29 of the tuberculosis trials consortium. Journal of Infectious Diseases. 2012;206(7):1030–1040. doi: 10.1093/infdis/jis461. http://dx.doi.org/10.1093/infdis/jis461. [DOI] [PubMed] [Google Scholar]
- Hill Z, Tawiah-Agyemang C, Odei-Danso S, Kirkwood B. Informed consent in Ghana: What do participants really understand? Journal of Medical Ethics. 2008;34(1):48–53. doi: 10.1136/jme.2006.019059. http://dx.doi.org/10.1136/jme.2006.019059. [DOI] [PubMed] [Google Scholar]
- Hollander M, Wolfe DA, Chicken E. Nonparametric statistical methods. Hoboken, NJ: John Wiley & Sons; 2013. [Google Scholar]
- Joffe S, Cook EF, Cleary PD, Clark JW, Weeks JC. Quality of informed consent in cancer clinical trials: A cross-sectional survey. Lancet. 2001;358(9295):1772–1777. doi: 10.1016/S0140-6736(01)06805-2. http://dx.doi.org/10.1016/S0140-6736(01)06805-2. [DOI] [PubMed] [Google Scholar]
- Kass NE, Maman S, Atkinson J. Motivations, understanding, and voluntariness in international randomized trials. IRB. 2005;27(6):1–8. http://dx.doi.org/10.2307/3563534. [PMC free article] [PubMed] [Google Scholar]
- Kiguba R, Kutyabami P, Kiwuwa S, Katabira E, Sewankambo NK. Assessing the quality of informed consent in a resource-limited setting: A cross-sectional study. BMC Medical Ethics. 2012;13:21. doi: 10.1186/1472-6939-13-21. http://dx.doi.org/10.1186/1472-6939-13-21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Montalvo W, Larson E. Participant comprehension of research for which they volunteer: A systematic review. Journal of Nursing Scholarship. 2014;46(6):423–431. doi: 10.1111/jnu.12097. http://dx.doi.org/10.1111/jnu.12097. [DOI] [PubMed] [Google Scholar]
- National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. The Belmont report: Ethical principles and guidelines for the protection of human subjects of research. 1979 Available at: http://www.hhs.gov/ohrp/humansubjects/guidance/belmont.html. [PubMed]
- Nishimura A, Carey J, Erwin PJ, Tilburt JC, Murad MH, McCormick JB. Improving understanding in the research informed consent process: A systematic review of 54 interventions tested in randomized control trials. BMC Medical Ethics. 2013;14:28. doi: 10.1186/1472-6939-14-28. http://dx.doi.org/10.1186/1472-6939-14-28. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sandman L, Mosher A, Khan A, et al. Quality assurance in a large clinical trials consortium: The experience of the Tuberculosis Trials Consortium. Contemporary Clinical Trials. 2006;27(6):554–560. doi: 10.1016/j.cct.2006.06.003. http://dx.doi.org/10.1016/j.cct.2006.06.003. [DOI] [PubMed] [Google Scholar]
- U.S. Department of Health and Human Services. 45 CFR 46 Protection of human subjects. 2009 Available at: http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.html.
- U.S. Department of Health and Human Services. Federal policy for the protection of human subjects (’Common Rule’) 2011a Available at: http://www.hhs.gov/ohrp/humansubjects/commonrule/index.html (accessed February 17, 2011)
- U.S. Department of Health and Human Services. Informed consent—FAQs: Who must sign the informed consent or parental permission document? 2011b Available at: http://www.hhs.gov/ohrp/policy/faq/informed-consent/who-must-sign-informed-consent-document.html (accessed July 7, 2014)
- U.S. Department of Health and Human Services. 21 CFR 50 Protection of human subjects. 2013 Available at: http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPartD50&showFRD1&subpartNodeD21:1.0.1.1.20.2.
- U.S. Department of Health and Human Services and Office of the Secretary. Human subjects research protections: Enhancing protections for research subjects and reducing burden, delay, and ambiguity for investigators. Federal Register. 2011;76(143):44512–44531. Available at: http://www.gpo.gov/fdsys/pkg/FR-2011-07-26/pdf/2011-18792.pdf. [Google Scholar]
- World Health Organization. Global tuberculosis report 2014. Geneva, Switzerland: World Health Organization; 2014. [Google Scholar]
- Yates BC, Dodendorf D, Lane J, et al. Testing an alternate informed consent process. Nursing Research. 2009;58(2):135–139. doi: 10.1097/NNR.0b013e31818c3df5. http://dx.doi.org/10.1097/NNR.0b013e31818c3df5. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.