Electronic Data Capture for Registries and Clinical Trials in Orthopaedic Surgery: Open Source versus Commercial Systems

Abstract

Background

Collection and analysis of clinical data can help orthopaedic surgeons to practice evidence based medicine. Spreadsheets and offline relational databases are prevalent, but not flexible, secure, workflow friendly and do not support the generation of standardized and interoperable data. Additionally these data collection applications usually do not follow a structured and planned approach which may result in failure to achieve the intended goal.

Questions/purposes

Our purposes are (1) to provide a brief overview of EDC systems, their types, and related pros and cons as well as to describe commonly used EDC platforms and their features; and (2) describe simple steps involved in designing a registry/clinical study in DADOS P, an open source EDC system.

Where are we now?

Electronic data capture systems aimed at addressing these issues are widely being adopted at an institutional/national/international level but are lacking at an individual level. A wide array of features, relative pros and cons and different business models cause confusion and indecision among orthopaedic surgeons interested in implementing EDC systems.

Where do we need to go?

To answer clinical questions and actively participate in clinical studies, orthopaedic surgeons should collect data in parallel to their clinical activities. Adopting a simple, user-friendly, and robust EDC system can facilitate the data collection process.

How do we get there?

Conducting a balanced evaluation of available options and comparing them with intended goals and requirements can help orthopaedic surgeons to make an informed choice.

Electronic supplementary material

The online version of this article (doi:10.1007/s11999-010-1469-3) contains supplementary material, which is available to authorized users.

Introduction

The transition from benchtop to bedside has long been a point of discussion and exploration in biomedical research [23, 42]. Although drug discovery forms a major part of continuing efforts to achieve better health care, effectiveness and practice variations are more practical and equally important topics [29]. Orthopaedic surgeons routinely face critical questions and dilemmas in their clinical practice. These dilemmas are usually related to choice of treatment, medical versus surgical management, patient outcomes, care quality, and drug resistance, to name a few. Experience and expertise also provide insights on possible alternatives to practice guidelines. These clinical ideas and dilemmas form the other end of the channel—“bedside to benchtop.”

A review of previous literature indicates that data collected in parallel to routine medical practice can be used to answer clinical questions [9, 12]. For example, collecting data on all the patients undergoing total knee arthroplasty can assist in evaluating effectiveness of different surgical techniques; yet, such activities are less prevalent among orthopaedic surgeons. Planned and structured data collection aligning with their routine workflow can enable orthopaedic surgeons to explore a multitude of research questions. Methods such as paper-based case report forms and direct data entry into spreadsheets and offline relational databases have long been utilized for this purpose. Although simple and inexpensive, they are not suitable for large registries and clinical trials as they are not secure, customizable, and accessible to geographically distributed users. Electronic data capture (EDC) systems, in addition to addressing these limitations, reduce a substantial amount of workload, time, and cost, as well as enhance the quality of data collected [17, 41]. They have also been found to be more efficacious than paper based systems in clinical trials in terms of trial duration, number of queries and time to locked database [3]. Due to these benefits, EDC systems are encouraged in clinical studies and trials.

Based on the development and distribution model followed, EDC systems fall into commercial and open source categories. Apart from the difference in features, cost for access, support and maintenance form the major difference between the two categories. Both commercial and open source EDC systems have a variety of features and have their own pros and cons which is a cause of frequent confusion and indecision.

The purpose of this article is to provide a brief overview of EDC systems, their types, and related pros and cons, as well as to describe commonly used EDC platforms and their features. This information will help orthopaedic surgeons make a balanced evaluation against the backdrop of their requirements and goals. It will also facilitate their decision and subsequent adoption of EDC systems. We also describe simple steps involved in designing a registry/clinical study in DADOS P, an open source EDC system to demonstrate the simplicity of using EDC systems.

EDC Systems - An Overview

Traditionally, paper-based case report forms or questionnaires have been used for collecting data as a part of clinical practice, research studies, and quality control exercises. Despite being simple, manual data entry into spreadsheets and subsequent verification is time-consuming, tedious, and prone to data errors. Further, scanning paper questionnaires to populate a database is time-consuming and affected by data recognition errors [44]. With the advent of information technology in health care, EDC systems have emerged as an alternative to paper-based systems.

EDC systems are computerized systems designed to collect and manage clinical and laboratory data in an electronic format [25]. Although the use of electronic media to capture data into an online database is a basic defining feature, there exists a diverse variation in EDC systems based on the technology used and the intended target audience. In most cases information collected from the subject is directly fed into an electronic CRF (eCRF) through a browser. In other cases electronic data entry is subsequent to a paper based CRF data entry. The definition of EDC systems also extends to interactive voice response systems (IVR) where a subject can report information through a phone, electronic diaries that capture subject reported outcomes and point of contact data collection systems that use devices such as digital pen and tablet computers to capture data. Although they primarily complement data collection activities in clinical research studies, they are also used by clinicians for populating registries and databases. Phase 3 and Phase 4 clinical trials, pharmaco-vigilance studies, and safety surveillance activities are some other examples where EDC systems are used [24]. Usually an EDC system has components that support and validate data entry as well as conduct single-multiple field cross checks [35]. Other typical features range from remote data entry, security, multi-level user access, discrepancy review and resolution, the ability to reuse case report forms and data export in a variety of formats. Some variants also include the ability to analyze data and generate reports. Despite being similar to clinical data management systems in terms of data collection and discrepancy management features, they differ on account of the following features: (1) they support multiple studies, multi-site studies and numerous data streams; (2) they have a unique workflow for study setup; (3) they store data centrally and handle coding differently [35].

Numerous benefits have been associated with the use of an EDC. Online data collection forms, support for real time multisite data capture and edits as well as automated workflow processes are some advantages [35]. By facilitating direct entry of data into a computerized database, electronic questionnaires ensure data quality and accuracy characterized by low incidence of missing or problematic values [44]. Discrepancy management is another major advantage with EDC systems. Discrepancies have been reduced by 60–80% on account of preprogrammed edit checks and screen level checks in EDC systems that minimize common errors and missing values. When used in a clinical trial, use of an EDC accelerates study startup and substantially reduces the time to study lock thus reducing the overall study cost [35]. It also reduces the time involved in collecting and processing data [18].

Types of EDC Systems: Commercial and Open Source

EDC systems can either be standalone databases on a desktop computer/server supporting a single site or they can be Web based with the ability to support multisite studies [1, 3, 27]. Based on the business model utilized and the licensing-distribution method followed, they can be broadly categorized as commercial and open-source EDC systems. Commercial EDC applications are usually developed by a for-profit company or developer group. They charge for user licenses with or without annual support contracts while the source code is not published. Some examples include Oracle® Clinical (Oracle, USA) [31], Clinsys® (Jubilant Organosys, USA) [6], InForm™ (Phase forward, USA) [22] DATATRAK Electronic Data Capture (DATATRAK, USA) [8]. On the other hand, free and open-source software are applications developed by a single or group of developers, often as a voluntary effort. The application and its source code are published online and users can download them without any cost. Some examples include DADOS P (Research on Research group, Duke University, USA) [7] OpenClinica® (Akaza Research, USA) [30], Redcap (Vanderbilt University, USA) [34] and TrialDB (Yale University, USA) [43] .

Commercial EDC Systems

Commercial EDC systems can either be purchased as a software package or through a license with periodic support either included in the package or charged separately. In some cases, users have to pay a one-time license fee while in other cases users can renew their license periodically. Troubleshooting and guidance are usually provided under support plans while bugs in the application are rectified and released under regular updates.

Commercial EDC applications are usually easy to use in light of the quality of documentation and customer support extended to users. Well-designed interfaces are responsible for their user-friendly design. Multiple levels of user access, security, adherence to industry and regulatory standards, support for the design of eCRF’s, data entry, and management are features common to many commercial EDC applications. Some of them may also be able to generate reports, for example, DATATRAK One™ (DATATRAK international Inc, USA) [8], Entrypoint Plus® (Phoenix Software International Inc, USA) [10], and CliniProteus (Roskamp Bioinformatics Core, USA) [5]. Although standalone EDC systems are prevalent, they are increasingly being offered as a part of a complete clinical trial management system (CTMS). For example Oracle Clinical and InForm are offered as a part of a larger CTMS.

Despite the benefits, commercial EDC systems are expensive [13] and frequently non-customizable. They have been considered inadequate in context to the needs of healthcare stakeholders (clinicians, administrators, and patients) [46]. Given the variety of clinical practices and research methods used, commercial EDC systems may not fit into the workflow at each clinical/research site, thus having implications on their effective and efficient use. Many of them do not support interoperable data standards, i.e. it may not be possible to merge data exported out of one EDC with data exported from another EDC system or research/clinical application. As a result data from multiple studies or sources cannot be merged and utilized for answering research questions. Since the source code is not released, users must depend on the developer group or vendor for customization and support-related requirements. This limitation is frequently referred to as “vendor lock-in,” which makes further development and maintenance of commercial EDC an expensive effort.

We reviewed the web to identify some of the prominent and popular commercial EDC systems. They include Oracle® Clinical, InForm™ and Rave® (Table 1). Among themselves they hold a larger share of the EDC market [3]. Other examples include DATATRAK and eTrials EDC (Merge Healthcare) [8, 11].

Table 1.

Comparison of EDC systems

EDC system	Oracle clinical	InForm	Rave	DADOS P	Openclinical
Type (Commercial/Open source)	Commercial	Commercial	Commercial	Open source	Open source
Vendor	Oracle	Phase Forward	Medidata solutions	Research on Research group	Akaza Research
Features	A core component of an eClinical suite that integrates adverse event report reporting, thesaurus management, trial management and remote data capture features	It has separate EDC and CDMS databases	It has a single platform that supports both EDC and CDMS	EDC platform	A single platform that has both EDC and CDMS
License	Paid	Paid	Paid	GNU GPL	GNU LGPL
Safety reports	Y	Y	Y	N	N
Standards	Y	Y	Y	Can be added	Y
Security (21 CFR Part 11)	Y	Y	Y	Y	Y
Data import/export	Y	Y	Y	Y	Y

Oracle® Clinical [31] is a commercial EDC system for conduction of clinical studies and trials. It is a core component of an integrated eClinical research solution that integrates adverse event report reporting, thesaurus management, trial management and remote data capture features in a single application.

InForm™ [22] global trial management system delivers a variety of features essential for the effective and streamlined implementation of clinical studies and trials. It sports an impressive study setup page, scalability and has separate EDC and CDMS databases. Other features include an intuitive interface, streamlined workflow, reporting and analysis tools that help the study team to work more efficiently. It can also be seamlessly integrated with randomization, trial design, and medical coding modules.

The Rave® (Medidata solutions) [36] platform is another industry leader in EDC systems. It has an impressive study design tool that nullifies the need for programming skills. It offers a single, flexible and scalable platform that captures, manages and reports clinical research data. It also undertakes a balanced approach between ease of use, features and functionalities. It has the flexibility to interface with legacy systems, adheres to CDISC clinical data standards and sports a plugin for modifying the interface and functionality.

DATATRAK Electronic Data Capture has considerable market presence [8]. Its features range from patient data management, electronic forms, supports queries, alerts and visit scheduling and generation of custom reports. It is also equipped with custom checklists and workviews, configurable tools for data cleaning, real time statistical support and an integrated medical coding package.

eTrials EDC [11] is a web based application that collects, manages and analyzes clinical trial information in real time. It is a user-friendly yet robust application with an in-built workflow. It can generate reports and can easily integrate with data from other applications.

Despite being feature rich, scalable, secure and compliant to industry standards, these EDC systems are prohibitively expensive [13], thus limiting their use by individual investigators and users from developing countries that do not having adequate funding but are interested in research participation.

Open-source EDC Systems

Open and freely available source code released under open distribution licenses like general public license [14] is the defining feature in this category of EDC systems. Open source code generates a large community of users and developers that interact, modify, and enrich the source code over a period of time and report bugs and solutions, thereby enhancing the quality, features, and value of the EDC system. In order to sustain an open source license, developers usually charge for customization requests. The same is applicable to troubleshooting and support, which are usually delivered through a yearly contract or support plan. Thus, by providing free access to the source code and annulling restrictions on use, modification, and distribution, open-source EDC systems form an attractive alternative for users.

The availability of inexpensive (or free) open source EDC applications for individual physicians/researchers, departments, and institutes has the potential to improve clinical and research activities and enhance academic standards by reaching a wider audience. Since further development/customization and ownership costs are lower, institutional administrators can modify and adapt open-source EDC systems to suit their environment and workflow, thus ensuring the success of EDC implementation. User-friendly and simple interfaces, adherence to industry standard security protocols, customizability, interoperability, and low maintenance costs are some of the major benefits of open-source EDC systems. Additionally, the presence of user support groups and communities ensures continuing support.

Although open-source EDC systems have a unique mix of features, their adoption in healthcare organizations has been slow. Software cost and maintenance are not the only features that influence decision makers. For an institution/organization-wide implementation, decision makers usually prefer and opt for EDC systems that are easy to deploy, manage, and support. Not all open-source EDC systems qualify for the same. In addition, dependence on the developer community for support and updates may cripple the organization if the community stops being productive. It is possible to address this issue by hiring programmers who could work on further development and maintenance, But locating and training relevant workforce is a challenge as in many cases the background technology has a steep learning curve.

DADOS Prospective, OpenClinica® and Redcap are examples of open source EDC systems. DADOS Prospective is a Web-based application developed by the research on research group (RoR) [37] to support data collection activities among researchers, research groups, and research networks [7]. It enables users to replicate any case report form into an eCRF, collect data in single/multisite studies, and extract data in an interoperable format. It is compliant with Chapter 11, Title 21, Code of Federal Regulations [4] and Health Privacy and Accountability Act (HIPAA) guidelines for EDC. It can be used to streamline and support individual/departmental/institutional databases, registries, and single/multisite clinical/nonclinical studies and clinical at a low cost [28].

OpenClinica® [30] is an open source application that has both EDC and data management capabilities. As an EDC system, it not only facilitates collection, validation, and annotation of clinical data but also has features that allow study audits, reporting and data extraction.

Research electronic data capture—‘Redcap’ is an open source metadata driven application designed for the clinical and translational research target audience by Vanderbilt. Its features include: a streamlined process for building a database, an intuitive and secure data collection interface that supports data validation and automated data export in multiple formats. It also supports other advanced features such as branch logic and file upload. It is freely available to its consortium partners with a modest personnel investment of < 0.5 FTE that covers training and support activities [16, 34].

Choosing an EDC System

There are a wide variety of available EDC systems in terms of underlying platforms, business models, costs and features. Orthopaedic surgeons can evaluate the offerings of each system against the backdrop of important features described below and make an informed choice.

Cost

Cost is inarguably an important factor to be considered while choosing an EDC system. Although the nature of licensing and support required to maintain an EDC system are the main predictors of cost, the purpose of data capture and workflow at the site of implementation also have a major influence on the cost. Implementation at an institutional, departmental or individual level, number and type of users, single site or multi site data collection and workflow complexity are some examples of the latter. Commercial EDC systems like Oracle Clinical, InForm and Rave are expensive in comparison to open source EDC systems like DADOS prospective, OpenClinica and TrialDB. While, the former have a higher presence in industry sponsored clinical trials, the latter are more common in academic settings.

Licensing

The types of ownership and user license have an impact on the costs associated with implementing an EDC system. Both commercial and open source EDC systems are available under a variety of license types. Commercial EDC systems may offer an annual/perpetual license or individual/site specific license. A maintenance and support package may be bundled with the license or offered at a separate annual fee. Open source EDC systems are usually released under General Public License (GPL) and Lesser General Public License (LGPL) licenses which impart the freedom to use and varying levels of modification and distribution rights [14, 15].

Implementation and Maintenance

Most electronic data capture systems are browser based while some of them operate using a Software as a Service (SaaS) model with remote hosting abilities. Implementation in both cases needs thoughtful planning in terms of staff and resources available to ensure success. In terms of staff, both commercial and open source EDC systems usually need: (1) support of a database manager to install and maintain the EDC system; and (2) clinical research coordinator to design an eCRF, enter and manage data. While a database manager can manage multiple studies inside an EDC system, a clinical research coordinator can manage a fixed number of studies at a given point in time. In terms of resources, secure and hi-speed network connectivity, secure data entry consoles (desktops/laptops/tablets/mobile devices) and a web server are bare essentials to support EDC systems. Although, users (clinicians, clinical research coordinators) do not need to have programming knowledge to design an eCRF, they need varying amounts of user training according to the EDC system involved. Both commercial and open source EDC systems provide extensive documentation and support for user training.

Workflow

Success of a data collection initiative depends on the quality of data captured. Workflow efficiency has been reported as a predictor of data quality [26] since inefficient operations result in errors related to data collection and transcription. Aligning EDC systems with the workflow in a clinic or research study can enhance data quality. Although in many cases, both commercial and open source EDC systems are pre-equipped with a general workflow, it is easier and cheaper to customize the latter as per the requirements.

Data Input

Although direct input of data into an EDC system is widely prevalent, in many cases data are first entered into a paper-based CRF followed by entry into an online system. Additionally, other modes of data input such as table PC based, mobile based and voice based systems are also becoming popular. The choice of input method is usually based on the nature and type of study and availability of resources.

Data Management, Analysis and Reports

While data handling (data entry, data cleaning and other such activities) by multiple individuals may lead to the introduction of errors, data captured through standalone systems is difficult to merge and leverage on. Accordingly, if resources permit, single point access systems and all-in-one suites offering data capture, data management, analysis and report generation are preferable over standalone systems. Oracle Clinical [31] and InForm [22] are examples of data management suites.

Security and Backup

Data security is a major point of concern for all electronic data. Ideally, EDC systems should be compliant with 21 Code of Federal Regulations (CFR) chapter 11 [4] and Health Insurance Portability and Accountability act (HIPAA). Additionally, all data should be stored in an encrypted and password protected server. Web access to the EDC system should be through a secure internet protocol. All changes in the database should be tracked and monitored through an audit trail. Data backup and recovery should ideally include a combination of RAID (redundant array of independent discs) backup, remote backup through mirror servers and tape based backup.

Standards

Given the wide diversity of EDC platforms and suites it is important that they support data interchange standards thus allowing the ability to merge disparate sources of data. Consequently data from different studies and sources can be merged and analyzed to yield generalizable results.

Discussion

Data collection in parallel with clinical practice can complement evidence-based orthopaedic practice. EDC systems are the method of choice for collecting data in single/multisite clinical and nonclinical studies. The various types, platforms, features as well as relative pros and cons make it difficult to choose among different EDC systems. We summarize the basic know-how related to EDC systems in order to facilitate the choice of a suitable EDC. Equipped with this information, orthopaedic surgeons can make informed decisions while choosing an EDC system.

Despite the benefits and value proposition of EDC systems, their widespread adoption has been slow [45]. A large chunk of users either continue to use paper based questionnaires or use a combination of paper and electronic data capture. One reason for this trend is that switching from paper based to EDC system is a challenging and expensive task. Additionally, EDC systems have disadvantages such as hardware and network related constraints as well as maintenance costs [33]. Other reasons reported for this delayed adoption include: (1) Most of the products and vendors in the market are relatively new. As a result, potential users are unsure about the quality of the products and dependability of the vendors. (2) Administrators and policy makers in government and industry are averse to taking risks related to adoption and implementation of a system wide technology which has the potential to fail [3]. Other problems with EDCs are related to site management, technical issues or both. The former include electronic signatures, password assignment and regular password resets, 24x7 support, study specific validation of the EDC system and study staff trained in appropriate procedures (in accordance with FDA code of federal regulations—chapter 11, title 21) for using the EDC system. Technical issues include instances when changes are needed in the EDC in accordance to significant changes in the study protocol and software upgrades. The former may substantially impact database integrity while the latter may result in data loss due to data migration as well as considerable system downtime [43]. As a result, careful planning and validation are essential for EDC systems.

Where are we now?

Realizing the potential of data collection and the availability of electronic data capture systems, clinicians in many parts of the world have undertaken systematic initiatives like high quality clinical databases [2] and have met with great success. Similar initiatives have been undertaken in orthopaedics in the form of international databases like the International Documentation and Evaluation System database (IDES) [32, 38], national registries like Swiss orthopaedic registry [40], Swedish hip registry [20], European spine registry [39] and other regional databases. The contribution of these registries to the improvement of patient outcomes and understanding of practice variation has also been demonstrated on multiple occasions [19–21]. Despite these facts, research participation through data collection at an individual level is lacking in orthopaedic surgery. Apart from the lack of time, resources and funding to carry out these activities, the lack of a simple, intuitive and user-friendly EDC system is noteworthy.

Where do we need to go?

It is imperative for individual orthopaedic surgeons to populate their own clinical database/registry or participate in regional/national/international efforts in parallel to their clinical practice thereby enriching evidence based orthopaedic care.

How do we get there?

Choosing the right infrastructure and tools for data collection can save time, cost and efforts. There is a wide variation among existing EDC systems in terms of features, underlying platforms, business models, support and management structures as well as cost. It is essential for new users to be aware of the EDC landscape and terms to be able to make an informed decision. Cost, simplicity, and workflow alignment usually predict the success of an EDC system. An overload of features not only increases the cognitive burden for users but also substantially reduces data quality and performance of the system, while an EDC with a simple and intuitive workflow can enhance adoption and successful implementation of a data collection initiative. We have provided an example of simple workflow involved in setting up a database/registry in DADOS Prospective. (Supplemental materials are available with the online version of CORR.) A careful evaluation of existing EDC systems against the backdrop of these factors can help busy orthopaedic surgeons answer clinical questions by adopting EDC systems and thus practice evidence-based medicine.