Abstract
Objective
Methods to improve informed consent efficiency and effectiveness are needed for pragmatic clinical trials. We compared informed consent using a tablet computer to a paper approach to assess comprehension and satisfaction of patients and clinic staff for a future osteoporosis clinical trial.
Methods
Nine community-based practices identified and recruited patients to compare the informed consent processes (tablet vs. paper) in a mock osteoporosis clinical trial. The tablet informed consent included an animation summarizing the trial, complete informed consent document, and questions to assess and reinforce comprehension of the study. Participants were women age ≥55 years with ≥1 year of alendronate use. We surveyed participants to assess comprehension and satisfaction and office staff for satisfaction and perceived time demands.
Results
The nine practices enrolled 33 participants. There was not a significant difference in comprehension between the tablet vs. paper informed consent [mean (SD) tablet: 12.2 (1.0) vs. paper: 11.4 (1.7)]. Office staff preferred the tablet to the paper informed consent for identifying potential study participants (two-sided t-test p = 0.02) despite an increased perceived time spent to complete the tablet process [tablet: 28.3 min (SD 16.3) vs. paper: 19.0 min (SD 6.9); p = 0.08].
Conclusions
Although, there were no significant differences in participant satisfaction and comprehension with the tablet informed consent compared to a paper informed consent, patients and office staff trended towards greater satisfaction with the tablet informed consent. Larger studies are needed to further evaluate the utility of electronic informed consent in pragmatic clinical trials.
Keywords: Informed consent, Pragmatic clinical trials, Osteoporosis
1. Introduction
Pragmatic clinical trials (PCTs) evaluate the real world effectiveness of interventions in the general population rather than the more homogenous populations used in traditional randomized controlled trials [1], [2]. Including participants from diverse community settings is needed to maximize the generalizability typically required of PCTs. However, ensuring efficient and effective participant informed consent represents a challenging component of PCTs in settings where the conduct of research is not a primary function [3], [4], [5], [6].
Due to time and resource requirements, the informed consent process is a barrier to performing PCTs in many practice settings [7], [8]. This is particularly true for many community practices; especially among practices not well-versed in research procedures or without dedicated research staff. The traditional informed consent process using a paper form requires dedicated personnel to explain the study, clarify details, answer questions, and guide patients through the informed consent process. This requires time and effort from staff members and clinicians who may be in a busy practice setting focused on high throughput clinical care.
As recognized by the Office of Human Research Protections, there are also concerns about whether the current informed consent process leads to fully informed study participants [9]. Thus, there is a need to improve the efficiency of the informed consent process and improve patient comprehension of clinical research benefits and risks [10], [11]. Electronic tools that provide audiovisual enhancements to improve patient comprehension can potentially reduce time demands of busy office staff, might alleviate some of the informed consent barriers, and could improve comprehension, but they have not been shown to have clear benefit in all studies [11], [12], [13], [14], [15], [16].
To address these evidence gaps in the informed consent process, we developed a patient self-administered tablet informed consent tool and conducted a mock study of a future osteoporosis PCT of bisphosphonate discontinuation versus continuation. We compared patient comprehension and satisfaction and provider satisfaction with the electronic informed consent process compared to traditional paper informed consent process in community practices.
2. Methods
2.1. Overview
We identified nine community-based practices and asked them to enroll three participants per informed consent process type (tablet informed consent process versus traditional paper informed consent process). In collaboration with a software developer experienced in direct-to-patient studies (Mytrus, Inc, San Francisco), we developed an interactive tablet informed consent tool that included an animated audiovisual summary of a future PCT, the complete informed consent document, and comprehensive multiple choice questions with feedback to assess and reinforce the key study consent elements. We surveyed participants' comprehension and satisfaction immediately following both informed consent processes. We also surveyed practice staff to assess satisfaction and perceived time demands following each type of informed consent process and again at the end of the study to compare the two processes. All study procedures were approved by the University of Alabama at Birmingham Institutional Review Board.
The mock study used for this evaluation of the informed consent processes was a future osteoporosis PCT. The future study does not include the use of any new treatments or interventions and, consistent with PCT methodology, the inclusion criteria are minimal. Women enrolling in the mock study were required to provide social security numbers, which would be used to link to the participants' administrative claims data in the future PCT.
2.2. Site recruitment and selection strategy
We identified community-based practice sites (n = 9: n = 7 “solo”, n = 2 “group”) from the American Academy of Family Physicians National Research Network (AAFP NRN), the Alabama Practice Based Research Network (APBRN), and the South Texas Ambulatory Research Network (STARNet). We selected a convenience sample of eligible sites based on their desire to participate, their perceived ability to recruit a sufficient number of eligible participants, and resources available. While the AAFP NRN, APBRN, and STARNet membership differed slightly in their demographic characteristics, members of practice based research networks (PBRNs) have been shown to be representative of community practices at large [17]. Practices were randomized to start with either the tablet informed consent process or the paper informed consent process and then were switched to the alternate informed consent process after completing 3 participant enrollments, for a total of 6 participants per practice in order to best assess each clinic's satisfaction with the both methods.
2.3. Patient recruitment and informed consent process
Practice sites initially identified eligible participants from their specific practice based on the inclusion criteria of the future osteoporosis study, i.e. ≥55 years old and ≥1 year alendronate use. When women presented to the clinic, if they were willing to consider participation, they were first provided a paper screening form, which included 3 questions: (1) Are you currently taking Alendronate (Fosamax® or Binosto™)?, (2) If yes, have you been taking Alendronate (Fosamax® or Binosto™) for 1 or more years?, and, (3) Are you willing to use a tablet computer to give and receive medical information? If the women answered “yes” to all three questions, the clinic staff would then provide them with either the tablet or paper informed consent. We included the requirement of willingness to use a tablet computer in the screening questionnaire in an effort to include women with similar comfort levels with technology in the two informed consent study groups.
2.4. Electronic informed consent process
The tablet informed consent tool initially presented screening questions for participants to complete, first re-verifying the basic inclusion criteria, including the participant's age and use of alendronate for at least 1 year. If the inclusion criteria were confirmed, the tool then presented an audiovisual description of the research study using an animated video with avatars describing the important details of the study, including the randomization process and the potential risks and benefits associated with each arm of the osteoporosis PCT. Participants were provided with earbuds to assure only they could hear the audio and a stylus to help navigate the tablet touchscreen. After the animated video, a complete IRB-approved informed consent document was presented on the tool. The informed consent also contained seven comprehension multiple choice questions. The questions focused on the key components of informed consent: study purpose, randomization, medication risks, medication benefits, withdrawal from the study, patient compensation, and confidentiality. If the participant did not answer a question correctly, she was provided a pertinent “Hint”. The participant was also offered the option to return to the informed consent document to review the related information in greater detail prior to having the question re-asked. There was no limit to the number of attempts a participant had to answer a question correctly but questions must be answered correctly before the participant could continue in the informed consent document. After completely reviewing the informed consent document and answering the questions correctly, the participant was asked to provide her social security number and date of birth and was then prompted to sign the informed consent document using the touch screen (with the provided stylus or her finger). After completion of the participant component, the staff then verified all inclusion/exclusion criteria and co-signed the informed consent document. Participants were then asked to complete the brief comprehension and satisfaction assessment (see below).
The number of questions, their content, and how many tries participants were given to answer questions correctly within the tablet tool was under the investigators' control. A summary report from the tablet tool, which included any questions which were answered wrong, was provided in real time for review by the enrolling practice staff before co-signing the informed consent. After the clinic staff signed off on the informed consent, dynamic randomization occurred and the clinic staff and participant were made aware of the study group to which the participant was randomized. The total amount of time spent on the tablet informed consent process was collected electronically. The tool was developed in full compliance with the HIPAA, FDA: Part 11 Compliance 21 CFR 11 (1997), which imposes certain requirements on an entity when it chooses to maintain FDA-required records and signatures in electronic form, and uses 256-bit data encryption.
2.5. Paper informed consent process
During the paper informed consent process, clinic staff met one-on-one with women who qualified for the study based on the initial screening criteria and the screening form completed upon the participant's presentation in clinic. After discussing the study with a clinic staff member, women were provided additional time to review the informed consent document and ask questions regarding the study. After review, women agreeing to participate in the study signed the consent and provided their social security number and date of birth. As in the tablet consent process, the staff verified all inclusion/exclusion criteria and then co-signed the informed consent document. Participants then also completed the brief comprehension and satisfaction assessment as the tablet participants did after completing the entire informed consent process (see below).
2.6. Survey assessment of patient comprehension and satisfaction
We assessed all participants upon completion of assigned informed consent process for comprehension of the informed consent elements/study aims, satisfaction with the mode of informed consent, perceived time requirement for completion of the informed consent, ease of completion, and importance of the informed consent in determining a willingness to participate in a future research study through surveys. The survey comprehension questions were multiple choice and were based on the validated Health-Information Technology Usability Evaluation Scale (Health-ITUES) [18], [19] and Quality of Informed Consent (QuIC) [20] surveys. Seven of the 38 survey questions in the brief survey administered at the conclusion of the consent process were the same questions included in the tablet informed consent tool. Additional survey questions measured each of the basic components of informed consent: who to contact with questions/problems regarding the study, coverage of potential study-related injuries, and procedures specific to the study. We also assessed basic knowledge/experience with electronic tools through questions regarding use and possession of smart phones and home computers. All questions other than the first 13 multiple choice and true/false comprehension questions were administered using a 7-point Likert response set. Participants in each process were also asked to provide their perceived time to complete the informed consent process in minutes.
In both informed consent processes, participants were recruited and presented the informed consent as if they were enrolling in the future osteoporosis clinical trial. They were initially “blind” to the actual purpose of this pilot study, namely to compare informed consent methodologies. Following completion of the informed consent process for this mock study, all participants were debriefed to the true nature of the experiment by practice staff and the participants were offered re-consent to complete the informed consent comprehension and satisfaction surveys. None of the participants had access to the Informed Consent document at the time of completion of these surveys.
2.7. Survey assessment of physician and practice staff satisfaction
Practice sites were surveyed by phone three times during the study: after enrolling 3 participants using the paper process, after enrolling 3 participants using the tablet process, and after completing both processes/end of the study. The practice site surveys included 10 questions using a 7-point Likert response set and covered site satisfaction, perceived time requirements for the informed consent process, confidence in administering the informed consent, clinic disruption, ease of the informed consent process for the clinic staff and participants, and the willingness to use a similar informed consent process for future clinical research. An additional question was also included regarding the perceived amount of time required for participants to complete the informed consent process.
The end of study survey was administered immediately after completion of the second informed consent process and the applicable process survey. The end of study survey consisted of five questions directly comparing the tablet informed consent process to the traditional paper informed consent process. The questions included ease of the informed consent process, efficiency of each process, helpfulness of each process, time required to complete each process, and the appeal of each process to participants, using a 7-point Likert response set.
Given the limited participant enrollment period of this pilot, not all sites completed recruitment using both informed consent methods, and thus did not complete both sets of surveys. Only staff that participated in both phases of this pilot were administered all surveys.
2.8. Data analysis
We used descriptive statistics to summarize the characteristics of participants and practices at baseline. Participant characteristics were compared between those who completed the tablet informed consent process to those who completed the paper informed consent process. An overall summary score for actual patient comprehension was calculated from the cumulative number of correct responses on the first 10 survey questions and mean scores were compared by group. Likert-scale responses on participant and practice surveys were treated as interval-level data and mean responses to the seven-point questions were compared. We determined statistical significance using two sided t-test (unless otherwise stated), or primarily Fisher's exact test due to small cell sizes. All analyses were conducted in SAS v.9.3 (Carey, NC).
3. Results
Nine practices participated, with clinical sites located in three states (Alabama, Colorado and Texas). The majority (78%) of sites were solo practices (Table 1). All practices were recruited from practice based research networks and the primary specialty of the recruitment sites was Family Practice, except for one Gynecology practice site. All sites had been involved in some form of clinical research in the past.
Table 1.
Participant and practice characteristics by method of informed consent administration.
| Tablet (n = 15) | Paper (n = 18) | |
|---|---|---|
| Participant characteristics | ||
| Age, Mean (SD) | 69.1 (7.1) | 71.4 (8.8) |
| Race/ethnicity, n (%) | ||
| Black | 2 (13.3) | 2 (11.1) |
| White | 13 (86.7) | 16 (88.9) |
| Employment, n (%) | ||
| Full-time | 4 (26.7) | 4 (22.2) |
| Not employed | 10 (66.7) | 13 (72.2) |
| Part time | 1 (6.7) | 1 (5.6) |
| Education, n (%) | ||
| Less than high school | 1 (6.7) | 1 (5.6) |
| High school or GED | 4 (26.7) | 5 (27.8) |
| Some college | 7 (46.7) | 5 (27.8) |
| 4 year college or higher | 3 (20.0) | 7 (38.9) |
| Marital status | ||
| Single or widowed | 3 (20.0) | 4 (22.2) |
| Separated | 0 (0.0) | 1 (5.6) |
| Married or cohabitating | 6 (40.0) | 9 (50.0) |
| Divorced | 5 (33.3) | 4 (22.2) |
| Missing | 1 (6.7) | 0 (0.0) |
| Use a personal computer, *n (%) | 12 (80.0) | 8 (44.4) |
| Use a smartphone, n (%) | 9 (60.0) | 6 (33.3) |
| Practice characteristics (n = 9) | n (%) | |
| Clinic size | ||
| Group practice | 3 (33.3) | |
| Solo practice | 6 (66.7) | |
| Practice type | ||
| Family practice | 8 (88.9) | |
| Gynecology/Obstetrics | 1 (11.1) | |
*Non-significant trends (p-value = 0.07; using Fisher's exact test) in the proportion of women with experience with a home computer between those randomized to tablet consent versus those assigned to paper consent.
The sites enrolled 33 women (n = 15 tablet informed consent; n = 18 paper informed consent) (Table 1). There were an additional 10 women who completed the initial screening questionnaire based on age/medication use eligibility but who were ineligible for the mock study based on an unwillingness to use a tablet for informed consent (n = 7), or an unwillingness to provide social security number (n = 3) (Fig. 1). Mean (SD) age of the women enrolled was 69.1 (SD 7.1) years in the tablet group and 71.4 (SD 8.8) years in the paper group and 68.7 (SD 7.7) years in the 10 women who were ineligible based on additional screening. Approximately one-third of the women were employed and educational status was similar among both groups. There was a trend, but not a significant difference, in the proportion of women with experience with a home computer between those randomized to tablet consent versus those assigned to paper consent (80% vs. 44%; p = 0.07) (Table 1). There was a non-significant difference in smartphone users among the same group (60% vs. 33%, respectively, p = 0.17). We found no significant differences in overall demographic characteristics (shown in Table 1) between the 10 women who began the process but were not enrolled and the women who completed the full informed consent process (data not shown).
Fig. 1.
Consort diagram showing the number of patients recruited into the study and reasons for dropout.
Overall, participant comprehension and satisfaction with both informed consent processes was high. Based on the results from the 13 comprehension questions, there was a non-significant difference in comprehension between the two groups. The tablet informed consent group had a mean of 12.2 (SD 1.0) correct answers compared to the paper informed consent group mean of 11.4 (SD 1.6) correct answers out of 13 possible. Participants using the tablet trended toward higher mean perceived comprehension in all categories (Fig. 2A: comprehension). Although not statistically significant, mean satisfaction in all categories was slightly greater among participants using the tablet consent process (Fig. 2B: satisfaction). Among the participants, the mean perceived time to complete the informed consent process (to the point of randomization) for the mock study using both methods was nearly identical: 22.9 min (SD 9.4) for the tablet group and 23.1 min (SD 14.6) for the paper group, (two-sided t-test p = 0.97). The actual time for completion was not available for the paper informed consent process; on average, women interacted with the tablet for 13 min and 7 s (SD 8.8).
Fig. 2.
A. Perceived Participant Comprehension (n = 33). Differences detected were not significant. PHI, Protected Health Information. B. Participant Reported Satisfaction by Mode of Informed Consent.
Provider and staff satisfaction was measured in the six practices (n = 5 “solo” practices and n = 1 “group” practice) that were able to enroll participants using both of the informed consent processes and completed the final surveys. Similar trends in greater satisfaction with the tablet informed consent process that were seen in patient participants were seen from practice site staff. For physician and practice staff, the mean perceived time for patients to complete the informed consent process was 28.3 min (SD 16.3) for the tablet group and 19.0 min (SD 6.9) for the paper group (two-sided t-test p = 0.08). Physician and staff reported the tablet tool was more beneficial in helping to recruit and enroll appropriate study participants (Fig. 3). Specifically, providers and clinic staff significantly preferred the tablet (mean 6.6, SD 0.5) to the paper informed consent (mean 6.0, SD 0.8) (two-sided t-test p = 0.02) for identifying potential study participants, based on responses to a 7-point Likert scale, with 7 being “Extremely Satisfied” and 1 being “Not Satisfied”. Other questions of satisfaction including the integration of the informed consent process into clinic workflow; the level of understanding patients had; the time required for patients to complete the informed consent process for the osteoporosis research study; and, their satisfaction with the informed consent process trended towards favoring the tablet process but results were not significant (Fig. 3).
Fig. 3.
Provider Overall Preference for Informed Consent Completion in Tablet vs. Traditional Paper Consent (n = 12). Favors tablet = 7, Favors Paper = 1, Neutral = 4, Mean (SD).
4. Discussion
We conducted a pragmatic randomized study of a tablet informed consent process versus traditional paper informed consent to determine overall efficiency of the two approaches and to determine if participant comprehension and participant and practice site satisfaction were similar or different using a mock future osteoporosis PCT. Despite similar or greater perceived time spent completing the tablet informed consent process, we found a non-significant trend towards participant satisfaction and provider and clinic staff satisfaction with the tablet informed consent process compared to the paper informed consent. Physicians and clinic staff noted significantly greater satisfaction in identifying potential study participants with the tablet informed consent process.
Most studies of multimedia informed consent processes were completed for an anticipated surgical procedure, consisting of a single uncontrolled medical intervention rather than a clinical trial [21], [22]. A systematic review of 12 trials using multimedia informed consent reported that significant improvements in study comprehension were not seen immediately after completion of the informed consent process [9]. The addition of a video to the paper informed consent in two of four studies showed that sustained comprehension was improved when participants were re-surveyed a week following the informed consent completion [9]. With the use of tablet technology, participants have the ability to not only read the consent but see and hear about study details using the animated narrative of study procedures.
Recently, a study on the use of a tablet informed consent tool, similar to the one we developed for our study, showed significantly improved comprehension of study procedures and risks for a clinical trial of cancer chemotherapy when compared to a paper consent in a randomized study involving 55 patients [14]. In that study, participants were clinical research personnel and other volunteers from a single institution; all were aware that they were not being recruited for an actual study but were reviewing materials to assess their comprehension and recall of the informed consent details after use of a new tool [14]. Moreover, the participants were excluded if they had the disease of interest for the study. In this cancer treatment study, the addition of audiovisual aids led to improved participant comprehension and satisfaction when compared to a paper informed consent despite increased time to complete the informed consent process with the tablet [14]. This study differed from our study in which we specifically recruited participants who were eligible and interested in enrolling in the trial. Notwithstanding, the improved comprehension seen in this study that also used tablet technology supports findings from our more generalizable study, in which we observed mainly non-significant trends between the two informed consent groups but we were limited by a smaller sample size.
Use of an electronic or tablet-based informed consent has several potential advantages with regard to standardizing informed consent administration and conduct in multi-site trials in addition to participant-related benefits. With the use of tablet technology, participant knowledge can be assessed in real time to determine whether informed consent materials are understood through comprehension questions, as we did in our study. If a lapse in understanding is noted, reinforcement of specific details can occur in real-time. Electronic informed consent also allows for real-time monitoring of participant recruitment with immediate recording of site/participant interactions; enables more efficient initial screening; all collected information obtained has an electronic time and date stamp; and, the process can be managed from any location. There is also the opportunity to have access to the informed consent in multiple languages within a single device.
Our study has several limitations. This was a mock trial to determine the best mode of informed consent to use in a future osteoporosis clinical trial. Therefore, during the participant recruitment process, participants were recruited and presented the informed consent as if they were enrolling for the future osteoporosis clinical trial, keeping them “blind” to the true nature of the experiment. This was done to test and obtain the least contrived participant input during an informed consent process. Although such an approach should improve generalizability, it is possible that the mock nature of our study could have become contaminated, if patients became unblinded to our true study hypothesis on consent methodology through discussions with clinic staff or otherwise. We did not directly assess anxiety to our methods of informed consent, but it is an important component that possibly contributed, in either direction, to overall satisfaction with our tool compared to the traditional approach to obtaining informed consent. Although there were non-significant differences between informed consent groups with regard to prior home computer use, there was a trend and proportionately more women had home computers in the tablet informed consent group than the paper informed consent group (80% vs. 44.4%). It is unclear the degree to which this potential imbalance in computer experience may have impacted participants' preferences for the tablet informed consent.
The number of patient participants was limited to include only those who would agree to use a tablet (an inclusion criteria for all study participants, with 16% (n = 7) refusing). As with home computer exposure, this potentially limits generalizability to a younger or more affluent patient population who may be more comfortable with newer technologies. However, our data do not support this concern since the mean ages in both groups ranged from 69 to 71, which was similar in range to those who did not participate in the study due to an unwillingness to use a tablet for consent. Although not a limitation of our specific study, a tablet-based informed consent, in general, has the potential limitation of the expense of developing the initial infrastructure and technology to manage and validate online documents. Additionally, there may be challenges regarding the need for a stable secure Wi-Fi network or cellular data coverage to allow for data delivery and updates to electronic consents. However, there is growing use of electronic technology in most clinical practices and a general movement toward the use electronic informed consent, as demonstrated by the FDA report guiding the use of e-consent [23].
An additional limitation is the method used for site selection and recruitment. Convenience sampling may have limited the generalizability (and influenced the external validity) of the study. Although our study was conducted among community practices well distributed among rural, urban, and suburban locations and across several U.S. regions, the number of “solo” practices may have impacted the ability to meet recruitment goals, and the number of “group” practices was relatively small and therefore may not be generalizable to all practice sites. Additionally, each “solo” practice was different in clinic size and operation, which may have impacted timely recruitment of participants as all practices were given the freedom to operationalize the trial to best fit their clinic workflow. While the variability between these methods and how they were implemented may have impacted recruitment in this pilot study, allowing busy clinical practices to have the freedom to conduct clinical trials as they see fit minimized disruption to patient care and created a more pragmatic design. While future study designs could incorporate more rigid recruitment and operational methods, further research into the workflow/operations of successful community based practices is needed to better understand how to best implement recruitment for pragmatic clinical trials into every day practice. Future studies with larger sample sizes will be needed to further evaluate electronic informed consent for PCTs.
In summary, we found good feasibility and satisfaction of using electronic technology aiming towards more efficient PCT patient recruitment and randomization and more efficient and effective informed consent processes. Despite the clinic staffs having a longer perceived time requirement for participants to complete the informed consent process with a tablet computer, there was significantly greater satisfaction by practices for identifying potential study participants with the tablet computer; and while not significant, participant mean satisfaction in all categories was slightly greater among participants using the tablet consent process. Use of electronic informed consent may not be beneficial for all clinical studies; but in certain settings, electronic informed consent has the potential to improve participant comprehension and research site satisfaction.
Funding
Funding for this project was provided by grants from the Agency for Healthcare Research and Quality (U18HS10389) and the National Institute of Arthritis and Musculoskeletal and Skin Diseases (R21AR062300).
Acknowledgments
The authors would like to extend their most sincere appreciation to the physicians and staffs of the participating community practices for their diligent efforts and support of this project. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality or the National Institute of Arthritis and Musculoskeletal and Skin Diseases.
Footnotes
Supplementary data related to this article can be found at http://dx.doi.org/10.1016/j.conctc.2016.02.003.
Appendix A. Supplementary data
The following is the supplementary data related to this article:
References
- 1.Tunis S.R., Stryer D.B., Clancy C.M. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003;290(12):1624–1632. doi: 10.1001/jama.290.12.1624. [DOI] [PubMed] [Google Scholar]
- 2.Maclure M. Explaining pragmatic trials to pragmatic policymakers. J. Clin. Epidemiol. 2009;62(5):476–478. doi: 10.1016/j.jclinepi.2008.06.021. [DOI] [PubMed] [Google Scholar]
- 3.Chalkidou K., Tunis S., Whicher D., Fowler R., Zwarenstein M. The role for pragmatic randomized controlled trials (pRCTs) in comparative effectiveness research. Clin. Trials. 2012;9(4):436–446. doi: 10.1177/1740774512450097. [DOI] [PubMed] [Google Scholar]
- 4.Eapen Z.J., Lauer M.S., Temple R.J. The imperative of overcoming barriers to the conduct of large, simple trials. JAMA. 2014;311(14):1397–1398. doi: 10.1001/jama.2014.1030. [DOI] [PubMed] [Google Scholar]
- 5.Godwin M., Ruhland L., Casson I., MacDonald S., Delva D., Birtwhistle R. Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity. BMC Med. Res. Methodol. 2003;3:28. doi: 10.1186/1471-2288-3-28. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Warner E.T., Glasgow R.E., Emmons K.M., Bennett G.G., Askew S., Rosner B. Recruitment and retention of participants in a pragmatic randomized intervention trial at three community health clinics: results and lessons learned. BMC Public Health. 2013;13:192. doi: 10.1186/1471-2458-13-192. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Mattson M.E., Curb J.D., McArdle R. Participation in a clinical trial: the patients' point of view. Control. Clin. Trials. 1985;6(2):156–167. doi: 10.1016/0197-2456(85)90121-7. [DOI] [PubMed] [Google Scholar]
- 8.Rosenthal G.E. The role of pragmatic clinical trials in the evolution of learning health systems. Trans. Am. Clin. Climatol. Assoc. 2014;125:204–218. [PMC free article] [PubMed] [Google Scholar]
- 9.Flory J., Emanuel E. Interventions to improve research participants' understanding in informed consent for research: a systematic review. JAMA. 2004;292(13):1593–1601. doi: 10.1001/jama.292.13.1593. [DOI] [PubMed] [Google Scholar]
- 10.(US) IoM . 2012. Envisioning a Transformed Clinical Trials Enterprise in the United States: Establishing an Agenda for 2020: Workshop Summary.http://www.ncbi.nlm.nih.gov/books/NBK92275/ [cited 2014 September 17]; Available from: [PubMed] [Google Scholar]
- 11.Tait A.R., Voepel-Lewis T. Digital multimedia: a new approach for informed consent? JAMA. 2015;313(5):463–464. doi: 10.1001/jama.2014.17122. [DOI] [PubMed] [Google Scholar]
- 12.Hutchison C., Cowan C., McMahon T., Paul J. A randomised controlled study of an audiovisual patient information intervention on informed consent and recruitment to cancer clinical trials. Br. J. Cancer. 2007;97(6):705–711. doi: 10.1038/sj.bjc.6603943. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Moseley T.H., Wiggins M.N., O'Sullivan P. Effects of presentation method on the understanding of informed consent. Br. J. Ophthalmol. 2006;90(8):990–993. doi: 10.1136/bjo.2006.092650. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Rowbotham M.C., Astin J., Greene K., Cummings S.R. Interactive informed consent: randomized comparison with paper consents. PloS One. 2013;8(3):e58603. doi: 10.1371/journal.pone.0058603. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Schenker Y., Fernandez A., Sudore R., Schillinger D. Interventions to improve patient comprehension in informed consent for medical and surgical procedures: a systematic review. Med. Decis. Mak. Int. J. Soc. Med. Decis. Mak. 2011;31(1):151–173. doi: 10.1177/0272989X10364247. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Synnot A., Ryan R., Prictor M., Fetherstonhaugh D., Parker B. Audio-visual presentation of information for informed consent for participation in clinical trials. Cochrane Database Syst. Rev. 2014;5:CD003717. doi: 10.1002/14651858.CD003717.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Galliher J.M., Bonham A.J., Dickinson L.M., Staton E.W., Pace W.D. Representativeness of PBRN Physician practice patterns and related beliefs: the case of the AAFP National Research Network. Ann. Fam. Med. 2009;7(6):547–554. doi: 10.1370/afm.1015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Yen P.Y., Sousa K.H., Bakken S. Examining construct and predictive validity of the Health-IT usability evaluation scale: confirmatory factor analysis and structural equation modeling results. J. Am. Med. Inf. Assoc. JAMIA. 2014;21(e2) doi: 10.1136/amiajnl-2013-001811. e241–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Yen P.Y., Wantland D., Bakken S. AMIA Annual Symposium Proceedings/AMIA Symposium AMIA Symposium. 2010. 2010. Development of a customizable health IT usability evaluation scale; pp. 917–921. [PMC free article] [PubMed] [Google Scholar]
- 20.Joffe S., Cook E.F., Cleary P.D., Clark J.W., Weeks J.C. Quality of informed consent: a new measure of understanding among research subjects. J. Natl. Cancer Inst. 2001;93(2):139–147. doi: 10.1093/jnci/93.2.139. [DOI] [PubMed] [Google Scholar]
- 21.Fenner Y., Garland S.M., Moore E.E., Jayasinghe Y., Fletcher A., Tabrizi S.N. Web-based recruiting for health research using a social networking site: an exploratory study. J. Med. Internet Res. 2012;14(1):e20. doi: 10.2196/jmir.1978. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Nehme J., El-Khani U., Chow A., Hakky S., Ahmed A.R., Purkayastha S. The use of multimedia consent programs for surgical procedures: a systematic review. Surg. Innov. 2013;20(1):13–23. doi: 10.1177/1553350612446352. [DOI] [PubMed] [Google Scholar]
- 23.FDA . Use of electronic informed consent in clinical investigations. In: Services DoHaH, editor. Federal Register: Department of Health and Human Services. 2015. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.