Abstract
Screening and tracking subjects and data management in clinical trials require significant investments in manpower that can be reduced through the use of web-based systems. To support a validation trial of various dietary assessment tools that required multiple clinic visits and eight repeats of online assessments, we developed an interactive web-based system to automate all levels of management of a biomarker-based clinical trial. The “Energetics System” was developed to support 1) the work of the study coordinator in recruiting, screening and tracking subject flow, 2) the need of the principal investigator to review study progress, and 3) continuous data analysis. The system was designed to automate web-based self-screening into the trial. It supported scheduling tasks and triggered tailored messaging for late and non-responders. For the investigators, it provided real time status overviews on all subjects, created electronic case reports, supported data queries and prepared analytic data files. Encryption and multi-level password protection were used to insure data privacy. The system was programmed iteratively and required six months of a web programmer's time along with active team engagement. In this study the enhancement in speed and efficiency of recruitment and quality of data collection as a result of this system outweighed the initial investment. Web-based systems have the potential to streamline the process of recruitment and day-to-day management of clinical trials in addition to improving efficiency and quality. Because of their added value they should be considered for trials of moderate size or complexity. Grant support: NIH funded R01CA105048.
Keywords: Web-based trial management, screening, tracking, data security
Introduction
Recruitment, screening, tracking, and data management in clinical trials are complex processes essential to the efficient collection of high-quality data. For multifaceted clinical trials involving multiple measures and repeat visits, maintenance of up-to-the-minute quality control and timely compliance to all components can be a logistical nightmare. Establishing a coordinated system to support screening and tracking of subjects, visit scheduling, data entry, data monitoring and quality control checks is a prerequisite to beginning any clinical trial. Although clinical researchers are beginning to take advantage of current information technology to improve access to data [1], reduce errors [2, 3], enhance consistency across centers[4, 5] as well as using the web for recruitment [6] and interventions [7, 8] there are few reports of systems designed for the management of clinical trials[5, 9]. Despite evidence that using a web-based system for any of these purposes increased time efficiency, improved data management and reduced overall costs, there are still countless trials relying on inefficient methods of print advertising and manual subject tracking and data management.
The value of using web-based systems in clinical trial research has been increasingly recognized in the last several years [10]. Recruitment remains one of the most challenging aspects of clinical trials, and automated systems hold the potential to dramatically alter the playing field. More often than not, clinical trials are unsuccessful in recruiting subjects in the desired time frame and budget [11]. A survey of recruitment among 41 randomized controlled trials found that 66% of the trials never attained their planned sample size, primarily due to logistical and administrative difficulties of screening and enrolling large numbers of participants [12]. Tracking and scheduling subjects also entails a great deal of time and resources that could be greatly minimized using an automated web-based system. Integrating the system to encompass automatic data entry and management would further reduce potential error and increase study efficiency and quality. Data entry from paper/pencil-based questionnaires adds unnecessary layers of cost, editing and potential error, even when scanned forms are used, and could be avoided with a completely automated system.
We developed an automated interactive web-based system designed to take advantage of current information technology and public websites, automate scheduling tasks and reminders, and provide real time up-to-date status overview of each subject enrolled in a complex, biomarker-based clinical trial validating several dietary assessment approaches aimed at examining the feasibility and validity of web-based versus paper-pencil approaches. Three dietary assessment tools were examined: a self-administered, online 24-hr recall, a computer-based diet history and a paper-pencil food frequency questionnaire. The system was also designed for automatic data entry and secure data storage and efficient access to all raw data for analysis.
Methods
Design
Six months before the scheduled start of the study a team was assembled to conceptualize, design and develop a web-based support system that would assist in patient scheduling and data collection and management. A web designer employed half time and the study coordinator, the principal investigator, and a support staff person trained in laboratory methods, were involved in the process. Central functions were defined and in an iterative process, the system was expanded to incorporate multiple additional features that reduced the manpower demands of the study. Critical to the design was the development of a system that had various levels of access and addressed strictly data privacy and data security (Figure 1). The system was designed around three websites: a public website that gathered study information from study participants, the complete informed consent forms for participant preview, and automated screening of subject eligibility. This system was linked to an administration site that was password-protected such that the staff of the Clinical Research Center could not access the features available at the study center and could not change or edit any information. This component supported automated entry of all questionnaire-based data by the subjects themselves and provided quality control of the information capture at the clinic. The study coordinator had tailored programs that provided real-time updates on subject status, actions needing to be taken, and calendars. This site also provided data entry menus for the clinical forms that required remote entry, and study overviews of throughput, screenings and completion quotas for the principal investigator.
Figure 1.
Overview of information flow
Recruitment
Targeted subjects included African Americans and Caucasians aged 21–69 years residing within 50 miles of the Westwood (UCLA) neighborhood of Los Angeles. Subjects were required to meet a total of 21 eligibility criteria in order to be enrolled. Recruitment was initially largely through free public announcements of the study posted on websites such as the volunteer section of Craigslist Los Angeles, a widely used webpage featuring free online classified advertisements. Interested subjects were directed to the Energetics Study website (https://brs.ucla.edu/energetics), followed by the eligibility screener (https://brs.ucla.edu/energetics/screen). The flow chart for the screener is presented in Figure 2. Contact information on eligible subjects only is acquired at the end of successful eligibility screening, when a file is created, and the subject is identified in the tracking system as an eligible subject. They are then immediately added to the next day’s To-Do list of the study coordinator for scheduling a consent visit. Recruitment of 15 subjects per month was planned to ensure consent, scheduling and study completion of 12 subjects per month, allowing for three drop-out participants per month.
Figure 2.
Eligibility screener flow diagram
Eligibility Screening
The Energetics website (https://brs.ucla.edu/energetics) provides the aims of the study, a complete copy of the consent form, a self-screener, contact information and directions, and a list of Frequently Asked Questions. The self-screener (https://brs.ucla.edu/energetics/screen) was approved by the UCLA Medical Institutional Review Board. It was carefully designed so that criteria were combined and responses were stored in a fashion that would not allow recognition of individual medical conditions or behaviors of the respondents. The flow diagram presented in Figure 2 illustrates the eligibility screening steps. Subjects were notified immediately of ineligibility at the first inappropriate response and the questionnaire terminated at that point. Those subjects proving eligible were prompted to enter their contact information including phone number, address, and preferred mode of contact regarding the study. The tracking system at that point automatically created a file, registered the subject, including all contact information, in the subject database and triggered a call reminder on the study coordinator’s “To-Do List” to contact the interested subject.
Subjects were also given the option of screening via telephone through the use of a computer-assisted telephone interview (CATI) if they were unable or unwilling to screen-in via the Internet. Reasons for ineligibility were automatically logged into the system in categorical form. However, no subject identifiers (name, address or contact information) were entered until after eligibility had been ascertained.
Consent
The study coordinator contacted all eligible subjects to schedule a consent visit and answer any remaining questions about the study. The consent visit was scheduled within 10 days of the initial contact and was conducted in person by the study PI per the requirements of the UCLA Medical IRB. The study participants also completed two computer assisted self-interviews online at this visit.
Tracking and Data Management System
The tracking system was a highly interactive program designed by the study web designer, principal investigator (PI) and study coordinator to fully automate all possible scheduling tasks and reminders and to provide an up-to-date status overview of each subject enrolled in the study. Figure 3 presents an overview of the automated system designed for this study.
Figure 3.
An overview of the screening, tracking and data management system. (Text details the numbered items)
The system included a calendar feature that supported scheduling of three clinical visits and one consent visit of 12 subjects per month (48 visits per month) and coordinated these with the availability of the General Clinical Research Center, the site of the study. A color-coded matrix of all study subjects in each stage of study completion was continually available to the study coordinator. The system also provided options to review the screening status and study completion rates by race. The features of the tracking system are described in detail below with each subtitle cross-referenced to Figure 3.
Demographic Subject Information: All personal information on study subjects was collected either through the web-based questionnaires or through entry using electronic case report forms stored securely on the website. A variety of queries could be accessed easily and efficiently under the queries in the link “Show all Subjects.” Under this link, there was an alphabetical listing of subjects including basic demographic and contact information (gender, race, address, and phone number). The site also included a search function that could be used to quickly pull up additional participant information.
Monthly Calendar and To-Do List: After logging on to the administrative website, the first page to appear is the “To-Do List.” This screen provided a listing of all study tasks to be completed in order of priority, including all follow-up and reminder calls. These are in addition to extensive automated messaging and reminders. It also featured a real-time color-coded confirmation of scheduling and completion status of each subject (Figure 4). Scheduling information was automatically entered into a monthly calendar accessible from this page.
Automated Messaging: Subjects were required to conduct five independent web-based DietDays (24-hour dietary recall) on their own time, as well as one at each visit. Automatically generated email messages were sent to trigger subjects to conduct the recalls on nonadjacent days spread over a three-month time period. The link to the web-based DietDay was included in the automatically generated email message so that participants could easily log-in and complete the questionnaire. This system allowed for the full tracking of the subject’s progress and identified when participants did not complete the questionnaire on time. Symbols were used to represent the status (i.e. reminder sent, reminder read, visited dietary site, and dietary data found) were posted on the administrative website for the study coordinator to track the subject’s progress in completing the DietDay requirement.
Web Questionnaires: The study involved five web-based questionnaires including DietDay (a 24-hour dietary recall), a General Questionnaire, the validated International Physical Activity Questionnaire [12], a second comparison physical activity questionnaire, and an Exit Questionnaire. While the DHQ food frequency questionnaire developed by the NCI has been validated in other studies [13, 14], the online 24-hr recall and computer-based diet history used were to be specifically validated by this study. The questionnaires were completed online and featured automatic range checks to prevent implausible answers as well as real-time data uploading to the analytical database. Timestamps on all procedures allowed a quantitative estimate of the amount of time subjects spent conducting each of the web-supported questionnaires. The questionnaires could be printed out and completed manually in case of a computer malfunction during completion. The selection choices on each of these questionnaires did not allow for missing data as subjects could not move to the next screen without responding fully. There were however options among the choices allowing them to opt out of sharing personal information such as income.
Automatic Data Entry and Form Preparation: Data from all five web questionnaires were automatically entered into the analytical database where they were available for later analysis by the study PI and study statistician. The system also included automatic completion and print-out of forms required for the study, such as the Hospital Medical ID Form and shipping inventories of lab specimens. Self-tracking of study biological samples shipped and those ready to be shipped was also incorporated into the system. The system also automatically generated shipping inventories of samples to be sent to externals lab in batches at various time intervals.
Subject Overviews: Queries were built into the system to permit the efficient retrieval of real-time information on recruitment, scheduling, and up-to-date statistics on the progress of the study. Possible queries included: a) recruitment and scheduling overview, b) the number of potential participants screened and reasons for ineligibility, and c) a subject overview. The completion status for clinic visits, 24-hour recalls completed and clinical data entered into the website was provided for each subject using a color-coded scheme. These features allowed the study coordinator to monitor and address missing data.
Analytical Dataset: All data from demographic and dietary questionnaires completed online were automatically stored and immediately accessible to be downloaded to the analytical dataset for use by the study PI and statistician. Clinical data results were entered for each subject under their individual profile by the research assistants and study coordinator. Once downloaded, an analysis request based on variable name and data set could be entered by the PI and statistician to retrieve entire datasets for analysis.
Figure 4.
An overview of participants with scheduling and completion status of each subject
Data Security
The tracking system was designed to be secure and in compliance with HIPAA requirements. All subject information was stored on an internal key-protected administrative website available only to study staff through secure username and password IDs. A second level of security requiring two sets of keys protected the analytical datasets, which were only accessible to the PI and the study statistician.
Security System: The first tier of security entailed encryption of all communications between the user’s browser and the server. The user-browser connection to the web-application server was encrypted by industry-standard Secure Sockets Layer (SSL) using 256-bit SSL certificates acquired from GeoTrust, Inc (http://geotrust.com/). The network connection between the web-application server and the database server was encrypted in a similar manner. The second tier of security involved authentication with a username and password. An authorized username and password combination was required by all users to identify themselves to the system and log in. Users were encouraged to change their password frequently. Controlled access to different areas of the website provided a third level of security. Different user accounts could only access certain sections of the site. For example, the study coordinator used the “Admin”-user account to log in to the central interface of the website, including personal information on study subjects, calendar and To-Do lists, and subject status overviews. The nutritionist at the General Clinical Research Center (GCRC) used the “Nutrition”-user account to access the web questionnaires for study subjects to complete at each clinic visit. The study PI and statistician used the “Data”-user account to access the raw data for analysis. Lastly, all sensitive data were encrypted prior to being stored in the database via the AES/Rijndael algorithm (http://csrc.nist.gov/CryptoToolkit/aes/rijndael). This encryption was keyed via a passphrase known only to the study administrators and did not exist anywhere within the web application. Even in the event of a security breach, the data would remain encrypted.
Server: The web application resided on a server housed in the data center of the UCLA Department of Human Genetics. The server ran Red Hat Enterprise Linux 4 (http://redhat.com/) as an operating system. Web-based requests were handled by Apache 2 (http:/httpd.apache.org/), serving application code written with PHP 5 (http://php.net/). The data were stored in a separate database server, using the PostgreSQL relational database management system (http://postgresql.org/). The database server was housed in the same data center, but did not have a direct connection to the Internet and was only accessible indirectly via the web server.
Results
Recruitment
By utilizing Craigslist, the recruitment costs were kept low. The original budget projected $20,000 for newspaper advertisement. Placing the Craigslist ad provided widespread distribution to a diverse population with little effort and expense. Hits on the study website ranged from 14,517 to 33,136 hits per month. Out of 1692 individuals screened through the website, 780 screened positive, and 333 consented to participate in the study.
Eligibility Screening
The use of web-based screening reduced the projected number of phone calls required of the study staff for recruitment, based on prior experience in the challenges of contacting individuals by phone and the documented rate of screening ineligible, from an estimated 3000 attempts, yielding 1000 contacts and 300 consents to less than 50 screening calls. To address the concern that people who were motivated to participate but did not screen eligible would return to the site and change their responses, repeat attempts from the same Internet protocol address were not allowed repeat access to the screener. Subsequent verification of eligibility criteria confirmed the self-administered screener to be 99 percent accurate. The number screened, the number ineligible, and reasons for ineligibility were automatically logged onto the website. This information was useful for monitoring study balance and ensuring that screening and recruitment were on schedule.
Tracking system
As a result of the automated tracking system, the size of the personnel remained low. Based on a projected budget, it is estimated that the use of the automated tracking system reduced the needed personnel for this size study by half an administrative staff. The system was used constantly, and rapidly became a much appreciated and valued resource that was often updated to provide new features. The study coordinator spent approximately six hours/day on the system and relied on it heavily for all tasks related to running the study. Without the tracking system, the study coordinator would have had to schedule 48 visits per month while manually keeping track of the subjects’ schedules, the clinical center’s schedule, and the study schedule. The use of the tracking system not only greatly reduced the burden on the study coordinator saving an immense amount of time, but also minimized the number of scheduling mistakes by substituting automatic procedures for manual recordkeeping. Furthermore, it was accessible via the Internet from anywhere in the world, making it possible for the PI to view real-time updates on study recruitment and subject status, and access the data for analysis when traveling.
Data Management
Automatic data entry and secure storage not only saved valuable staff time and resources but also greatly reduced the inevitable data entry errors associated with manual entry. Automatic secure storage facilitated efficient use of the data from anywhere in the world. The system was programmed to include automated analysis request features that provided up-to-date lists of variables and their source files, documented and archived requests, emailed them to the programmer and study PI, and allowed continual creation of newly generated variables that were subsequently available through the request form. This standardized and greatly simplified the process of pulling up raw data through the use of web-based queries.
Discussion
Arguably, all the clinical trials could benefit from a web-based real time management system. Due to the demands for intense security, exquisite timing, and high data quality, clinical trials are logical targets for automated support systems. Such systems could well be considered an important component to budget in as part of the trial infrastructure. However the development of such systems may represent a burden on the investigator. Knowledge of such systems, the burden of selecting and hiring the programmer and the design time are barriers to the creation of such a system. On the other hand, utilization of such technology promises to improve efficiency, data quality and to reduce the logistics burden on the study coordinator. Our experience was that the system presented here required an investment of time and manpower to set up that was well worth the effort in a study of this size and complexity. After the involvement of a half-time web designer at the start of the study to handle design, programming, and security issues, maintenance at 10% of an FTE allowed subsequent updates, revision of forms and tailoring. This interactive collaborative process led to the implementation of a system that was tailored to the needs of this complex biomarker-based trial.
Given the current state of information technology, barriers to setting up such systems need to be reduced. One of these might be the recycling of modules that are common to most trials. The screening and tracking features for example of the system described addresses needs common to almost all trials. Web-based data entry by subjects on the other hand requires considerably more effort in creating access, security and programming questionnaires.
The online queries that might be desired by staff and researchers are likely to be study specific. However, creating these automated queries and reports is among the simplest of the programming tasks in system design.
The limitations to the generalizability of the findings on this system include the fact that the study population was a healthy convenience sample, which might be easier to identify and recruit than a more specific patient population. In addition, generalizability from the use of a convenience sample is always of concern as subjects may not be representative of the broader population. There is always concern about who is self-selecting into the trial.
This trial aimed at testing feasibility and validity in a ’best case scenario’ of motivated subjects. The population was, however, diverse in their age, gender and socioeconomic status. Table 2 provides the characteristics of the study population, which was limited to African-Americans and Caucasians. This was not, however, a group selected by Internet access at home. Not all could access the Internet in their homes and resolved this need by going to friends’ homes, Internet cafes or the public library.
Table 2.
Characteristics of all subjects who completed the 8 online 24-hr recalls and 2 clinic visits
| N = 247 | Percent |
|---|---|
| Gender | |
| Females | 64.8 |
| Males | 35.2 |
| Race | |
| Caucasian | 48.6 |
| African-American | 51.4 |
| Age | |
| <30 | 39.3 |
| [30, 40) | 18.2 |
| [40, 50) | 19.0 |
| [50, 60) | 19.4 |
| <70 | 4.1 |
| Educational Level | |
| Less Than High School | 0.4 |
| High School Graduate | 2.8 |
| Some College | 39.7 |
| College Graduate | 42.1 |
| Post Graduate | 15.0 |
| Body Mass Index Status | |
| Under weight <18.5 | 2.8 |
| Normal weight 18.5–24.9 | 43.3 |
| Overweight 25–29.9 | 29.6 |
| Obese >=30 | 24.3 |
Although the automated screener saved time and effort, as in any study there is the possibility that people who want to participate may falsify their responses, such as smokers saying they were not. As with any trial, we were challenged to decide the degree to which we would verify their results (for example using biomarkers of urinary cotinine). Therefore, systems can prevent repeat access to and altering of screener responses, and questionnaire can be designed to test the consistency of response, but the possibility that an individual who wants to become a subject might be dishonest in his or her responses remains an issue.
One potential concern for studies dependent on the Internet for independent offsite data entry by subjects might be the need for high-speed Internet in areas of lower economic development. It is possible that the reduced availability to the Internet might bias the pool of subjects who can utilize the system. In Los Angeles our subjects found ample opportunity to access the Internet, with many of those who did not have access at home turning to a public library. However the distance to and opening hours of libraries can be a consideration, especially when library hours are being reduced for economic reasons.
In moving to any web system that involves patient identifiers (in this case, names, addresses, phone numbers, birthdates, and medical data), protection of the site from external compromise and internal misuse is essential. Several levels of security were applied in this study including encryption of all web traffic so it can not be intercepted between our server and client browsers or end users, password protection of all admin items, and separation of admin from analytic data through additional passwords. In addition, additional decryption keys are included to keep the IDs blinded and a HIPPA key applied to separate identifying information from medical information. Working closely with extremely knowledgeable technical staff is essential to insure that these security needs are being met in an ever-changing environment.
In this study the enhancement in speed and efficiency of recruitment and quality of data collection as a result of this system outweighed the initial investment of training and set-up. Web-based systems have the potential to streamline the process of recruitment and day-to-day management of clinical trials in addition to improving their efficiency and quality. Because of their added value they should be budgeted into proposals for trials that are of moderate size or complexity. In addition, clinics should seek to encourage and support researchers in developing such systems. This can include providing technical support and facilitating the sharing of design modules.
Table 1.
Features of the Energetics Web-based Support System
| Features of the Energetics Web-based Trial Management System |
| 1. Recruiting, screening and subject tracking features |
| a. Automatic web-based eligibility screening |
| b. Tracking of subject visits and milestones |
| c. Automated e-mail reminders to subjects, tailored to their compliance level |
| d. Automated reminders to staff regarding scheduling tasks |
| e. Real Time world wide accessible status overview of each subject enrolled |
| f. Real time access to recruitment and monthly throughput statistics |
| g. Support of Monthly calendaring of all scheduled visits and available slots |
| h. Automated daily “To-Do” lists for study coordinator and staff |
| i. Color-coded monitoring of completion status lists of all active subjects |
| 2. Recruitment Status Reports |
| a. Real time statistics on screening and enrollment accessible worldwide |
| a. Status overviews of targets and achieved subject throughput on a monthly basis |
| b. Breakdowns of participants by gender, race and age to insure balance |
| 3. Automated data entry, secure data storage and efficient access to raw data for analysis |
| a. Electronic Case Report Forms |
| b. Automatic shipping inventory lists and updates |
| c. Real time Data Access through Website |
| d. Quality control of data entered and range checking |
| e. Programmer and analyst access for continuous data analyses throughout the trial |
| 4. Online Query Management Features |
| a. Status Overview |
| b. Individual data screens |
| c. Individual Subjects |
| d. Data and Reports |
| e. Dropouts and Incompletes |
| f. Analyses requested |
Acknowledgements
We would like to acknowledge grant support from NIH funded R01CA105048 for the described trial and the efforts of Elaine Quiter and Jasmine Chen in formatting and submitting this manuscript for publication.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Contributor Information
Lenore Arab, David Geffen School of Medicine at UCLA, Los Angeles, CA.
Harry Hahn, David Geffen School of Medicine at UCLA, Los Angeles, CA.
Judith Henry, Phoenix Agency, Tampa, FL.
Sara Chacko, UCLA School of Public Health, Los Angeles, CA.
Ashley Winter, USC, Los Angeles, CA.
Mary C Cambou, David Geffen School of Medicine at UCLA, Los Angeles, CA.
References
- 1.Pavlović I, Kern T, Miklavcic D. Comparison of paper-based and electronic data collection process in clinical trials: costs simulation study. Contemp Clin Trials. 2009 Jul;30(4):300–316. doi: 10.1016/j.cct.2009.03.008. Epub 2009 Apr 2. PubMed PMID: 19345286. [DOI] [PubMed] [Google Scholar]
- 2.Pavlović I, Miklavcic D. Web-based electronic data collection system to support electrochemotherapy clinical trial. IEEE Trans Inf Technol Biomed. 2007 Mar;11(2):222–230. doi: 10.1109/titb.2006.879581. PubMed PMID: 17390992. [DOI] [PubMed] [Google Scholar]
- 3.Santoro E, Nicolis E, Franzosi MG, Tognoni G. Internet for clinical trials: past, present, and future. Control Clin Trials. 1999 Apr;20(2):194–201. doi: 10.1016/s0197-2456(98)00060-9. PubMed PMID: 10227418. [DOI] [PubMed] [Google Scholar]
- 4.Winget M, Kincaid H, Lin P, Li L, Kelly S, Thornquist M. A web-based system for managing and co-ordinating multiple multisite studies. Clin Trials. 2005;2:42–49. doi: 10.1191/1740774505cn62oa. [DOI] [PubMed] [Google Scholar]
- 5.Smith KS, Eubanks D, Petrik A, Stevens VJ. Using web-based screening to enhance efficiency of HMO clinical trial recruitment in women aged forty and older. Clin Trials. 2007;4:102–105. doi: 10.1177/1740774506075863. [DOI] [PubMed] [Google Scholar]
- 6.McNeill LH, Viswanath K, Bennett GG, Puleo E, Emmons KM. Feasibility of using a web-based nutrition intervention among residents of multiethnic working-class neighborhoods. Prev Chronic Dis. 2007 Jul;4(3):A55. Epub 2007 Jun 15. PubMed PMID: 17572959; PubMed Central PMCID: PMC1955411. [PMC free article] [PubMed] [Google Scholar]
- 7.Park A, Nitzke S, Kritsch K, Kattelmann K, White A, Boeckner L, Lohse B, Hoerr S, Greene G, Zhang Z. Internet-based interventions have potential to affect short-term mediators and indicators of dietary behavior of young adults. J Nutr Educ Behav. 2008 Sep-Oct;40(5):288–297. doi: 10.1016/j.jneb.2008.02.001. PubMed PMID: 18725147. [DOI] [PubMed] [Google Scholar]
- 8.Unützer J, Choi Y, Cook IA, Oishi S. A web-based data management system to improve care for depression in a multicenter clinical trial. Psychiatr Serv. 2002 Jun;53(6):671–673. doi: 10.1176/ps.53.6.671. 678. PubMed PMID: 12045303. [DOI] [PubMed] [Google Scholar]
- 9.Marks R, Bristol H, Conlon M, Pepine CJ. Enhancing clinical trials on the Internet: lessons from INVEST. Clin Cardiol. 2001 Nov;24(11 Suppl):V17–V23. doi: 10.1002/clc.4960241707. PubMed PMID: 11712772. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Lovato LC, Hill K, Hertert S, Hunninghake DB, Probstfield JL. Recruitment for controlled clinical trials: literature summary and annotated bibliography. Control Clin Trials Aug. 1997;18:328–352. doi: 10.1016/s0197-2456(96)00236-x. [DOI] [PubMed] [Google Scholar]
- 11.Charlson ME, Horwitz RI. Applying results of randomised trials to clinical practice: impact of losses before randomisation. Br Med J (Clin Res Ed) 1984;289:1281–1284. doi: 10.1136/bmj.289.6454.1281. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Maddison R, Ni Mhurchu C, Jiang Y, Vander Hoorn S, Rodgers A, Lawes CM, Rush E. International Physical Activity Questionnaire (IPAQ) and New Zealand Physical Activity Questionnaire (NZPAQ): A doubly labelled water validation. Int J Behav Nutr Phys Act. 2007 Dec 3;4:62. doi: 10.1186/1479-5868-4-62. PubMed PMID: 18053188. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Subar AF, Thompson FE, Kipnis V, Midthune D, Hurwitz P, McNutt S, McIntosh A, Rosenfeld S. Comparative validation of the Block, Willett, and National Cancer Institute food frequency questionnaires: the Eating at America's Table Study. Am J Epidemiol. 2001 Dec 15;154(12):1089–1099. doi: 10.1093/aje/154.12.1089. PubMed PMID: 11744511. [DOI] [PubMed] [Google Scholar]
- 14.Millen AE, Midthune D, Thompson FE, Kipnis V, Subar AF. The National Cancer Institute diet history questionnaire: validation of pyramid food servings. doi: 10.1093/aje/kwj031. [DOI] [PubMed] [Google Scholar]