Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2026 Apr 15.
Published in final edited form as: Int J Med Inform. 2024 Dec 4;195:105749. doi: 10.1016/j.ijmedinf.2024.105749

Error rates of data processing methods in clinical research: A systematic review and meta-analysis of manuscripts identified through PubMed☆

Maryam Y Garza a,b,*, Tremaine Williams a, Songthip Ounpraseuth b, Zhuopei Hu c, Jeannette Lee c, Jessica Snowden c,d, Anita C Walden e, Alan E Simon f, Lori A Devlin g, Leslie W Young h, Meredith N Zozus b
PMCID: PMC13078370  NIHMSID: NIHMS2153678  PMID: 39647291

Abstract

Background:

In clinical research, prevention of data errors is paramount to ensuring reproducibility of trial results and the safety and efficacy of the resulting interventions. Over the last 40 years, empirical assessments of data accuracy in clinical research have been reported, however, there has been little systematic synthesis of these results. Although notable exceptions exist, little evidence exists regarding the relative accuracy of different data processing methods.

Methods:

A systematic review of the literature identified through PubMed was performed to identify studies that evaluated the quality of data obtained through data processing methods typically used in clinical trials. Quantitative information on data accuracy was abstracted from the manuscripts and pooled. Meta-analysis of single proportions based on the Freeman-Tukey transformation method and the generalized linear mixed model approach were used to derive an overall estimate of error rates across data processing methods used in each study for comparison.

Results:

A total of 93 papers (published from 1978 to 2008) meeting our inclusion criteria were categorized according to their data processing methods. The accuracy associated with data processing methods varied widely, with error rates ranging from 2 errors per 10,000 fields to 2,784 errors per 10,000 fields. MRA was associated with both high and highly variable error rates, having a pooled error rate of 6.57% (95% CI: 5.51, 7.72). In comparison, the pooled error rates for optical scanning, single-data entry, and double-data entry methods were 0.74% (0.21, 1.60), 0.29% (0.24, 0.35) and 0.14% (0.08, 0.20), respectively.

Conclusions:

Data processing methods may explain a significant amount of the variability in data accuracy. MRA error rates, for example, were high enough to impact decisions made using the data and could necessitate increases in sample sizes to preserve statistical power. Thus, the choice of data processing methods can likely impact process capability and, ultimately, the validity of trial results.

Keywords: Medical record abstraction, Data quality, Clinical research, Clinical data management, Data collection

Graphical Abstract

graphic file with name nihms-2153678-f0001.jpg

1. Background

In regulated clinical trials, investigators must rely on research data acquired to (1) ensure the safety and efficacy of medical treatments (to protect research participants and the general population at large), and (2) ensure the reliability and reproducibility of study results. High quality data provide the foundation from which study conclusions may be drawn,[1] and, in contrast, poor data quality threatens the validity and generalizability of study findings.[1,2] In general, quality refers to “a product or service free of deficiencies”[1,3] – some experts also using terms like “fitness for use”[4] and “conformance to requirements.”[5] Within the context of clinical research and the practice of clinical data management, the Institute of Medicine defines data quality as data that “support the same conclusions as error free data.”[6] There are several attributes tied to quality, but, for this project, we focused primarily on data accuracy – data that accurately represent data points collected directly from study participants.[1].

Authors in the clinical research arena lament the scarcity of published information regarding data quality.[6–18] While many authors point out that conclusions drawn from studies depend on data quality (and the underlying data collection and management methods), others consider the associated tasks clerical or even unnecessary.[19–22] This perception has resulted in minimal investigation and a small number of publications on the topic of data collection and management compared with other areas of clinical research and informatics methodology. With the current rapid influx of new technology into clinical research – starting with electronic data capture (EDC) and clinical trial management systems (CTMSs) shortly after the turn of the century, and followed by electronic patient reported outcomes (ePRO) systems, mobile health (mHealth), a myriad of digital health technologies (DHTs), and direct electronic health record-to-electronic case report form (EHR-to-eCRF) tools – understanding the quality of data from different available capture and processing methods has become even more important.[23,24] Many unresolved issues exist with respect to data quality in clinical research, including a thorough understanding of the accuracy and variability of current data processing methods[24–28] – a primary objective of this manuscript. A thorough review and synthesis of the relevant published literature is an initial step in providing guidance to investigators and clinical research teams. Accordingly, we aimed to address this gap through the systematic review and meta-analysis described in this manuscript.

Common options in data processing methods identified in the literature include: (1) chart review and abstraction versus direct electronic acquisition from electronic medical records (i.e., both types of medical record abstraction, or MRA); (2) use of vended or commercial data collection systems by local healthcare facilities (e.g., data entry and cleaning in local systems versus web-based data entry and cleaning in a centrally hosted system); (3) use of paper data collection forms with central processing versus local processing with data transfer to a central coordinating center; and (4) single- versus double-data entry (with or without programmed edit checks). Data cleaning methods also vary greatly, from use of reports to identify irregularities in the data, to on-screen checks (OSCs) during data entry (e.g., programmed edit checks), to post-entry batch data processing. We define the 4 major processing methods considered in this review (MRA, optical scanning, single-data entry, and double-data entry with or without programmed edit checks) in Table 1.

Table 1.

Definitions of Data Processing Methods in Clinical Research

Data Processing Method Definition
Medical Record Abstraction (MRA) A data processing method that involves the review and abstraction of data from patient records, often referred to as chart review or chart abstraction. Traditional MRA is a manual process, which may or may not involve paper forms. For this review, any study that utilized the medical record (i.e., EHR) as the source of data collection – meaning a study coordinator (or equivalent) accessed the medical record system directly and manually abstracted the data (either directly to the study data capture system or on to paper first and then to the study data capture system) – was classified as MRA.
Optical Scanning (OCR/OMR)* A data processing method that relies on software packages to “recognize characters from paper forms or faxed images, and these data are placed directly into the database.”1 Examples include optical character recognition (OCR) and optical mark recognition (OMR).
Single-Data Entry (SDE)* With respect to classification of data processing in the included manuscripts, single-data entry involves 1 person who enters data from a structured form into the study data capture system. SDE can be implemented with and without programmed edit checks (or OSCs)*. For this review, we distinguished SDE from MRA based on the source of the data. A study was classified as using SDE if the source was a structured form (often paper) that existed outside of the medical record system (for example, a study-specific questionnaire or clinical forms, such as concomitant medication logs or study vitals not documented in the EHR) and only involved 1 person for data entry.
Double-Data Entry (DDE)* Double-data entry involves 2 people (e.g., clinical research coordinator, data entry personnel) who independently enter data from a structured form to the study data capture system with a third, independent adjudicator to review and resolve any discrepancies. DDE can be implemented with and without programmed edit checks (or OSCs)*. For this review, we distinguished DDE from MRA based on the source of the data. A study was classified as using DDE if the source was a structured form (often paper) that existed outside of the medical record system (for example, a study-specific questionnaire or clinical forms, such as concomitant medication logs or study vitals not documented in the EHR) and involved 2 people for data entry.
Open in a new tab
*

Note. Hereinafter, we use a single acronym, OMR, to encompass all optical scanning methods discussed in the literature. Both single- and double-data entry methods can be conducted with or without programmed edit checks. Programmed edit checks – also referred to as discrepancy checks, edit checks, OSCs, or query rules – are electronic data quality checks that are programmed into the study data collection system and are triggered by data entry, either in real-time as data is entered or in batches.

Complicating comparisons of different data processing methods are the significant variability in quantitative methods for assessing data accuracy across clinical research and other secondary data uses. [1,29,30] Data accuracy has often been measured in terms of database error rates, although, registries commonly assess percent completeness as well. To standardize, the Society for Clinical Data Management’s (SCDM) Good Clinical Data Management Practices (GCDMP) document has defined the error rate as the “number of errors divided by the number of data values inspected.”[1,31].

As described in the GCDMP,[1] there are significant differences in the way errors and values are inspected and counted across different clinical research studies, even across those conducted by the same institution. Based on these counting differences, the error rates obtained can differ by a factor of 2 or more.[1,30] In addition, differences in how error rates are reported (e.g., as raw counts, errors per record, errors per fields inspected, or errors per 10,000 fields), necessitate scaling and normalization of the values reported in the literature before comparisons can be made. Due to variability in counting, such comparisons may still not be meaningful. Here, we undertook a systematic review of the relevant literature identified through PubMed to characterize data collection and processing methods utilized in clinical studies and registries. Additionally, we conducted a meta-analysis to calculate and compare error rates across the various data processing methods described.

2. Methods

2.1. Literature review

A PubMed search on the Medical Subject Heading (MeSH) terms “data quality” AND (registry OR “clinical research” OR “clinical trial”) through 2008 was conducted to identify relevant citations (see Additional File 1, Appendix A, Item A1 for the full PubMed Search Strategy and Table A2 for the PRISMA Checklist). Once an initial list of manuscripts was generated via PubMed, duplicates were excluded. The abstracts of the de-duplicated set of citations were screened for relevance against the eligibility criteria and those not meeting the criteria were also excluded. A search using PubMed related links and secondary and tertiary references was then conducted to identify additional manuscripts. The full-text of included manuscripts was reviewed against the eligibility criteria to generate the final set of manuscripts for inclusion in analysis (see Additional File 1, Appendix A, Reference ListA3 and Table A4).

2.2. Criteria for manuscript inclusion

The goal of this search was to identify quantitative reports of data quality in clinical studies, and the search terms and logic were selected to optimize that. Manuscripts were included in the analysis if: (1) they were published in peer reviewed journals indexed for retrieval or referenced by such and were obtainable; (2) they had a focus on secondary data use of healthcare data (e.g., clinical research, quality improvement, surveillance, research registries); (3) the database error rate was presented or resolvable (e.g., via number of errors identified and number of fields inspected, or contained sufficient information to calculate); (4) they described how the data were processed (e.g., MRA, optical scanning, single- or double-data entry); (5) they were written in the English language; and (6) the manuscript was the primary source for the error rate. Manuscripts not meeting 1 or more of these inclusion criteria were excluded.

2.3. Information gathered from manuscripts

Three types of data were collected from each manuscript: (1) information about how data were processed; (2) information about how data quality was measured; and (3) the number of errors and number of fields inspected. Concepts of interest of the data processing and quality measurement methods reported were noted as each manuscript was read. Prior to quantitative data analysis, factors identified from items (1) and (2) were developed in a qualitative, iterative manner during the review of the manuscripts. As such, concepts of interest, such as OSCs versus batch data discrepancy identification were added to the data collection form as they were identified, and previously reviewed manuscripts were re-reviewed for presence of the newly identified concepts of interest. Natural groupings were organized into categories. These categories were later explored in the analysis to ascertain which (if any) of the factors might affect data quality.

The following parameters were also collected but were considered supplemental: data cleaning method (i.e., batch data cleaning), location of data processing (central data center vs. local healthcare facility), gold standard used, and scope of method of comparison.

Quantitative data accuracy information, including the number of errors identified and the number of fields inspected, was abstracted from the manuscripts. Manuscripts were categorized by type of secondary data use, data processing method, and data accuracy assessment. Information on the number of errors identified and the number of fields inspected was collected for each manuscript. We abstracted the number of errors reported and the total number of data fields (values) inspected. The number of errors and number of fields inspected were used to calculate normalized error rates (number of errors per 10,000 fields) based on the recommendations in the GCDMP.[1] In cases where the authors presented only normalized error rates, such as errors per 10,000 fields, the normalized denominator was assumed for the total number of fields inspected. For example, if the normalized error rate presented was 100 per 10,000 fields, we took 100 to be the total number of errors (numerator) and 10,000 to be the total number of fields (denominator). Where error rates for more than 1 database were provided in a manuscript, each individual assessment was included in this analysis. Where error rates for multiple data processing steps were provided, we included each.

For consistency, 1 rater was used to abstract the error rate information from the manuscripts. A sample of the manuscripts included in the analysis, comprising 10 % of the total (standard for the domain), was re-evaluated by the primary rater following the initial abstraction to assess reliability. For the sample, the time between the initial and intra-rater reliability review was at least 1 year. Intra-rater reliability, calculated as percent difference, was used to gauge reliability of the data. In addition, a second rater reviewed the same intra-rater reliability sample.

2.4. Statistical analysis

Meta-analysis of single proportions[32,33] based on the Freeman-Tukey transformation method[34] and the generalized linear mixed model approach[35]of studies from the literature were used to derive an overall estimate of error rates across data processing methods for comparison. We also performed subgroup analyses where the data allowed. All statistical tests were performed at a two-sided significant level of 0.05, and all analyses were carried out using the R package ‘metafor’ and ‘meta’.[36,37] For each of the data processing methods, we used an inverse variance weighted meta-analytical method with Freeman-Tukey transformation[34] to calculate the pooled effect size and corresponding 95% confidence interval (CI). In the analysis, records with studentized residuals greater than an absolute value of 3 were considered outliers and subsequently removed. This threshold is frequently used because studentized residuals greater than an absolute value of 3 suggest that the data point is unusual or influential in the model.[38,39] The degree of heterogeneity between studies were examined based on the Q-statistic and Higgins and Thompson’s I2 statistic. The I2 statistic can be interpreted approximately as ≤ 25%, indicating low heterogeneity; 25% to 75% indicating moderate heterogeneity; and > 75%, indicating considerable heterogeneity.[40] The Q-statistic is typically underpowered for detecting true heterogeneity when the number of studies is small; therefore, we pooled data using a random effects model. The inter-study variance was evaluated by computing tau-squared, which provides the estimated standard deviation of the underlying effects across studies. Finally, to evaluate the consistency of our study, a sensitivity analysis was conducted using a leave-one-out model.[41] Also, a meta-regression with mixed-effect model with Freeman-Tukey transformation was implemented to compare the pooled effect among data processing methods.

3. Results

3.1. Manuscripts included for analysis

An initial search of the literature identified 350 citations. After excluding duplicates and performing the initial screen of abstracts, 54 manuscripts remained. A search using PubMed related links and secondary and tertiary references identified an additional 70 manuscripts, yielding 124 manuscripts for full-text review. Through the full-text review, we identified the final set of 93 manuscripts (see Additional File 1, Appendix A, Reference List A3 and Table A4), which were included in the pooled literature analysis (Fig. 1).

Figure 1.

Figure 1.

Open in a new tab

PRISMA Diagram: Identification of Data Quality Literature for Pooled Analysis

Four manuscripts [42–45] presented only normalized error rates as errors per 10,000 fields; for these, the denominator (10,000) was assumed for the total number of fields inspected. Each manuscript described a data quality assessment of 1 or more databases. Likewise, in some manuscripts, error rates were reported for more than 1 process step; for example, medical record-to-CRF or source-to-CRF, CRF-to-first entry, first entry-to-second entry, or CRF-to-clean file. A total of 22 manuscripts reported results for more than 1 processing step or database,[14,29,42,44,46–63] providing a total of 124 data points normalized as number of errors per 10,000 fields and demonstrating increasing dispersion over time of the health-related research literature with respect to data accuracy. The data processing methods, as reported in the literature, were not mutually exclusive; thus, some articles appear in more than 1 category (see Additional File 1, Appendix A, Table A5).

3.2. Meta-Analysis

During the meta-analysis, 9 records with absolute studentized residuals values greater than an absolute value of 3 were identified as outliers and, consequently, excluded from the analysis. To ensure robustness of these findings, a sensitivity analysis was performed by including the flagged records and re-running the analysis. The results remained stable, which suggested that the exclusion of these records did not bias the outcomes.

Thus, 84 manuscripts remained, which were categorized by data processing method and were included in the final analysis. Database error rates ranged from 2 – 2,784 errors per 10,000 fields (having excluded outliers) across 4 data processing methods: MRA, optical scanning, single-data entry, and double-data entry. This 3 orders-of-magnitude range necessitated a logarithmic display. There appeared to be no pattern in the year-to-year reporting. The data processing method with the highest error rates was MRA, having a pooled error rate of 6.57% (95% CI: 5.51, 7.72) (Table 2). The 3 other processing methods (optical scanning, single-data entry, and double-data entry) had much lower pooled error rates at 0.74% (0.21, 1.60), 0.29% (0.24, 0.35) and 0.14% (0.08, 0.20), respectively (Table 2). Heterogeneity was observed in all 4 data processing methods (see Additional File 2, Appendix B, Figures B1–B4). The sensitivity analysis did not indicate the extreme influence of any particular study (see Additional File 3, Appendix C, Tables C1–C4).

Table 2.

Pooled Effect Size of Error Rates (errors per 10,000) by Data Processing Method

Data Processing Method Range Pooled Proportion (95% CI) p-value*
Abstraction (MRA) 70 – 2,784 6.57 (5.51, 7.72) Ref
Optical Scanning (OCR/OMR) 2 – 358 0.74 (0.21, 1.60) <0.0001
Single-Data Entry (SDE) 4 – 650 0.29 (0.24, 0.35) <0.0001
Double-Data Entry (DDE) 4 – 33 0.14 (0.08, 0.20) <0.0001
Open in a new tab
*

Note. Ref: Reference. p-values were calculated through meta-regression. Source-to-database and source-to-CRF (case report form) were combined into a single category labeled abstraction, or MRA, based on: (1) some of the manuscripts reported error rates for abstraction directly to an electronic data collection form; i.e., no separate data entry step, and (2) the central tendency and dispersion of the 2 processes being similar. The source-to-database error rates ranged from 82–2,784 errors per 10,000 fields; the source-to-CRF error rates ranged from 70–2,424 errors per 10,000 fields.

3.3. Subgroup analysis

In exploring subgroups of the 4 main data processing methods, there was insufficient information in the literature about the MRA methods employed to further investigate possible causes for the variability in a subgroup analysis. Similarly, there were too few optical methods data points to support a subgroup analysis. For single- and double-data entry, a review of the literature surfaced different variations on key entry, including single-data entry (1 person enters the data), single-data entry with on-screen data checks (1 person enters the data within a system employing programmatic, OSCs), and double-data entry (2 people independently enter data with a third, independent adjudicator to review and resolve discrepancies). Further variations on single-data entry found in the literature included use of batch data cleaning and the location of data processing. These results are provided in Additional File 4, Appendix D, Table D1. Due to the importance of this particular model, manuscripts reporting data accuracy from similar data processing configurations (e.g., central versus distributed data processing in the presence of OSC), were examined (see Additional File 4, Appendix D, Table D2). Sixty-eight studies (across 49 manuscripts) versus 49 studies (across 39 manuscripts) reported central versus distributed processing; while 7 studies (across 5 manuscripts) did not report the location of data processing (noted in Table A4, see Additional File 1, Appendix A).

The intra-rater reliability for number of errors, number of fields, and error rate were 85%, 97%, and 86%, respectively. In addition, a second rater reviewed the same intra-rater reliability sample, with comparable results. In light of the underlying variability in the data, the variability in error rate calculation methods currently in use, and the aims of this study, these were considered reasonable. In addition, they were comparable to those in a similar review paper of errors in EHRs.[64].

4. Discussion

This study calculated and compared error rates across the various data processing methods described in the literature. The results indicated that the accuracy associated with data processing methods varied widely. Error rates ranged from 2 to 2,784 errors per 10,000 fields within the 4 most common data processing methods, strengthening our understanding of the influence of data processing and cleaning methods on data accuracy.

4.1. Medical record abstraction

Ordered by the mean, MRA was associated with the highest error rate. Importantly, abstraction was also associated with significant variability. Notably, the error rates reported for MRA methods spanned 3 orders of magnitude, with error rates ranging from 70 to 2,784 errors per 10,000 fields. These results support claims that MRA, which remains the dominant method of data collection in retrospective and prospective research, is the most significant source of error across data processing methods.[13,65].

4.2. Optical scanning

Although optical scanning methods such as OCR and OMR have been touted as a faster, higher-quality or less resource-intensive substitute for manual data entry,[19,55,66–72] others have reported error rates with optical methods that were 3 times higher than manual keyboard data entry.[73] Based on the pooled literature, we found optical scanning error rates ranged from 2 to 358 errors per 10,000 fields. Optical methods were associated with a variability of 2 orders of magnitude in accuracy. Such variability may be influenced by: (1) the presence and type of data cleaning employed in processing the optical scans; (2) use of post-entry visual verification or pre-entry manual review; (3) training of form completers on handwriting; (4) differences in form compatibility with the software; (5) software configuration (e.g., recognition engine); and (6) variations in data quality assessment methods. In particular, based on the available error in human inspection in other disciplines ranging from 16.4% to 30.0%,[74–78] using manual visual verification is likely less effective than OSCs.

4.3. Single- vs. double-data entry

Overall, single-data entry error rates ranged from 4 to 650 errors per 10,000 fields, and double-data entry error rates ranged from 4 to 33 errors per 10,000 fields. Great variability was observed between different sub-types of single-data entry, which provided a plausible explanation for the high level of variability observed in single-data entry as a whole. This is an important finding because large amounts of data are collected through single-data entry from research sites via web-based systems, including entry of abstracted data into web-based systems, clinicians entering data in EHRs, and data collected directly from patients via hand-held devices. Due to the problem of “alert fatigue,” however, OSCs may not be feasible in EHRs, where clinical alerts will often be a higher priority. The question of alert fatigue in these systems is an important topic for further research.

4.4. Measuring data accuracy

Claiming to have measured data accuracy (or error) is a statement implying that the measurer has compared the data to something, identified differences, and, in the case of a difference, was able to discern whether the data value from the assessed dataset was in error or not. In other words, a gold standard exists. In addition to aforementioned differences in counting errors and data values inspected, there was also variability in the literature with respect to the comparison made to measure data accuracy. In some cases, the comparator was the medical record; in other cases, it was an upstream recording of the data; in other cases, it was another dataset supposed to contain the same observations on the same individuals; and still in other cases, it was independent collection of the same information, such as a repeat interview or test. As evidenced by the literature and practice standards[79] the error rate has historically been the accuracy metric used. However, use of sensitivity and specificity have been recommended in draft regulatory guidance as the preferred measures of accuracy in the case of EHR and claims real-world data (RWD).[80] Sensitivity and specificity are preferred over overall accuracy or error rates because they are not dependent on prevalence.[81] These measures were not often used in the included manuscripts, probably due to a long history of using accuracy (the sum of true positives and true negatives divided by the total number of data values inspected) or error rate (the sum of false positives and false negatives divided by the same denominator) metrics. Where a gold standard is not available, errors cannot be determined in the case of a difference, and the difference or discrepancy rate is tallied instead. In this case, only measures of agreement such as inter-rater reliability and chance-adjusted agreement are appropriate. There are many such measures.[82] These measures, along with measures of agreement, were far more common in the included manuscripts than sensitivity and specificity. It is important to note that, while agreement may correlate with accuracy, agreement measures are not measures of data accuracy and, in many cases, may differ substantially from measures of accuracy.

As web-based EDC leads as the predominant method of future clinical research data collection, we anticipate heavier reliance on programmed edit checks to reduce error rates. Additionally, the role and process of programmed edit checks could serve as a model for addressing data quality checks of error rates within the more automated, standards-based processes of future data exchange, such as direct EHR-to-eCRF methods using the Health Level Seven (HL7®) Fast Healthcare Interoperability Resources (FHIR®) standard.[83–88].

5. Limitations

This study was a secondary, pooled analysis of database error rates in the published literature. Although it constitutes an important contribution in synthesizing the very fragmented historical literature, there are significant and inherent limitations. Very few of the included papers were controlled studies. Most of the included manuscripts merely stated the observed error rate and described data handling methods as part of reporting research results from a clinical study. With the exception of 8 included manuscripts (manuscripts 15, 36, 39, 42, 45, 71, 88, 92 from Appendix A, Table A4), the included studies were observational in nature (a “one shot” design) and lacked a comparator, i.e., “low quality evidence”.

As a secondary analysis, this work relies on data that were collected for other purposes. Although we used error and field counts reported in the literature, prior work has shown that even these have significant variability.[1,30] There likely is a bias toward counting rules that yield a larger denominator and smaller numerator. These represent a potential bias in measurement of outcome, and in handling missing data. Taken together, the risks of bias in included studies would tend toward lower reported error rates and less difference between data processing methods, since the ideal in all cases is low error rates.

Most of the manuscripts in our review were from academic organizations and government or foundation-funded endeavors that employ different data collection and management methodologies. Although over the time span of the literature we reviewed, those methods have tended to converge, our results may be less applicable to industry funded studies. Though our results are relevant to EDC data collection and cleaning processes, having to exclude the EDC (no manuscripts past the year 2008) literature from this review is a limitation. Authors did not consistently report the processes undertaken for collection and processing, nor did they include the error rate. For example, as reported by Nahm and colleagues in 2008,[23] some sites used paper worksheets to record data abstracted from medical records, while others charted source data directly in such worksheets, versus others that abstracted directly from the medical record into the EDC system without a paper intermediary. Because these aspects often could not be resolved in published manuscripts, the review was truncated to account for the onset of EDC adoption, with the latest included manuscript published in 2008.

For this study, we used a meta-regression with mixed-effect model with Freeman-Tukey transformation to compare the pooled effect among data processing methods. While the Freeman-Tukey transformation is often used in meta-analysis to stabilize variance, especially when individual studies have effect sizes close to the boundaries (e.g., very small or very large proportions), there is the potential to introduce bias related to the study selection process.

5.1. Future direction

As data (increasingly captured electronically) are used to support clinical research, the effects of data quality on decision-making need thorough exploration. Potential effects of system usability and data processing methods on data quality should also be characterized to guide data management and planning choices. In particular, the 2018 revision of Good Clinical Practices (GCP) calls for risk-based prioritization of study activities that focus resources on activities that impact human safety and research results.[89] Use of the word ensure rather than assure in the guidance strongly suggests that quality management systems be in place to prospectively design capable processes and to control error rates within acceptable limits. We found very few reports of prospective prediction of process capability or of implementation of process control for the data error rate. Quality management system (QMS) design and implementation with respect to data accuracy remains an area for further exploration. The variability and the magnitude of error rates reported in the literature should encourage quantitative evaluation of the impact of new technology and processes on data accuracy and subsequent decisions regarding whether the accuracy of the data is acceptable for the intended use.

6. Conclusion

Based on the pooled analysis of error rates from the published literature, we conclude that data processing and cleaning methods used in clinical trials research may explain a significant amount of the variability in data accuracy. In general, error rates reported in the literature were well within ranges that could necessitate increases in sample sizes from 20% or more in order to preserve statistical power for a given study design.[90,92] Data errors have also been shown to change p values[93] and attenuate correlation coefficients to the null hypothesis;[94–96] in other words, a given clinical trial may fail to reject the null hypothesis because of data errors rather than because of a genuine lack of effect for the experimental therapy.[97] In the presence of large data error rates, a researcher must then choose to either (1) accept unquantifiable loss of statistical power and risk failure to reject the null hypothesis due to data error; or (2) measure the error rate and increase the sample size to maintain the original desired power.[91,92,96] The adverse impact of data errors has also been demonstrated in registries and performance measurements,[56,97–101] as has failure to report data.[102] Thus, the choice of data processing methods can likely impact process capability and, ultimately, the validity of trial results. Our findings suggest that reporting the results of a clinical study without specifying (1) the error rate, (2) the uncertainty in the error rate, and (3) the method used to measure the error rate limits the ability to interpret study findings.

Supplementary Material

Supplementary Data 1
Supplementary Data 2
Supplementary Data 3

Summary Table.

Problem: Data processing methods may explain a significant amount of the variability in the accuracy of data collected for clinical trials.
What is already known: Over the last 40 years, empirical assessments of data accuracy in clinical research have been reported in the literature. Although there have been reports of data error and discrepancy rates in clinical studies, there has been little systematic synthesis of these results. Further, although notable exceptions exist, little evidence exists regarding the relative accuracy of different data processing methods.
What this paper adds: Many unresolved issues exist with respect to data quality in clinical research, including a thorough understanding of the accuracy and variability of current data processing methods. We identify and discuss the 4 major data processing methods used in clinical research and present the amount of variability in data accuracy across methods. These results can guide clinical researchers as they look to operationalize studies, make decisions on the most appropriate data processing methods, and define comprehensive data management plans for their projects.
Open in a new tab

Acknowledgments

We thank Phyllis Nader, BSE for her assistance with this project.

Funding

Research reported in this publication was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under award numbers UL1TR003107 and KL2TR003108, and by the IDeA States Pediatric Clinical Trials Network of the National Institutes of Health under award numbers U24OD024957, UG1OD024954, and UG1OD024955. The content is solely the responsibility of the authors and does not represent the official views of the NIH.

List of Terms & Abbreviations.

Terms / Abbreviation Meaning / Definition
EDC

Electronic Data Capture
CTMS

Clinical Trial Management System
ePRO

Electronic Patient Reported Outcomes
mHealth

Mobile Health
DHT

Digital Health Technology
EHR

Electronic Health Record
CRF

Case Report Form
eCRF

Electronic Case Report Form
MRA

Medical Record Abstraction: A data processing method that involves the review and abstraction of data from patient records, often referred to as chart review or chart abstraction. Traditional MRA is a manual process, which may or may not involve paper forms. For this review, any study that utilized the medical record (i.e., EHR) as the source of data collection – meaning a study coordinator (or equivalent) accessed the medical record system directly and manually abstracted the data (either directly to the study data capture system or on to paper first and then to the study data capture system) – was classified as MRA
OSC

On-screen checks (i.e., programmed edit checks)
Optical Scanning

A data processing method used in clinical research that relies on software packages to “recognize characters from paper forms or faxed images, and these data are placed directly into the database.”1
OCR

Optical Character Recognition; an example of optical scanning
OMR

Optical Mark Recognition; an example of optical scanning
SDE

Single-Data Entry: With respect to classification of data processing in the included manuscripts, single-data entry involves 1 person who enters data from a structured form into the study data capture system. For this review, we distinguished SDE from MRA based on the source of the data. A study was classified as using SDE if the source was a structured form (often paper) that existed outside of the medical record system (for example, a study-specific questionnaire or clinical forms, such as concomitant medication logs or study vitals not documented in the EHR) and only involved 1 person for data entry. SDE can be implemented with and without programmed edit checks (or OSCs). When on-screen checks are employed, a series of programmatic edit checks are actively running during data entry and will “fire” when a discrepancy is identified during data entry. The data entry person is then able to review and address discrepancies during data entry.
Programmed Edit Checks

A data processing method during which electronic data quality checks are programmed into the study data collection system and are triggered by data entry, either in real-time as data is entered field-by-field or upon the form being saved, or in batch based on some pre-determined criteria. Programmed Edit Checks are also referred to as Discrepancy Checks, Edit Checks, On-Screen Checks, or Query Rules
DDE

Double-Data Entry: Double-data entry involves 2 people (e.g., clinical research coordinator, data entry personnel) who independently enter data from a structured form to the study data capture system with a third, independent adjudicator to review and resolve any discrepancies. For this review, we distinguished DDE from MRA based on the source of the data. A study was classified as using DDE if the source was a structured form (often paper) that existed outside of the medical record system (for example, a study-specific questionnaire or clinical forms, such as concomitant medication logs or study vitals not documented in the EHR) and involved 2 people for data entry. DDE can be implemented with and without programmed edit checks (or OSCs)*.
SCDM

Society for Clinical Data Management
GCDMP

Good Clinical Data Management Practices
MeSH

Medical Subject Heading
CI

Confidence Interval
RWD

Real-World Data
HL7®

Health Level Seven
FHIR®

Fast Healthcare Interoperability Resources87
GCP

Good Clinical Practice
QMS

Quality Management System

Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.ijmedinf.2024.105749.

Footnotes

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethics Approval and Consent to Participate.

The literature review and meta-analysis received a determination of not human subjects research as defined in 45 CFR 46.102 by the UAMS IRB (IRB#239826). All methods were carried out in accordance with relevant guidelines and regulations.

Consent for Publication

Not Applicable.

CRediT authorship contribution statement

Maryam Y. Garza: Writing – review & editing, Writing – original draft, Methodology, Formal analysis, Data curation, Conceptualization. Tremaine Williams: Writing – review & editing, Validation, Formal analysis. Songthip Ounpraseuth: Writing – review & editing, Validation, Formal analysis, Conceptualization. Zhuopei Hu: Writing – review & editing, Validation, Formal analysis. Jeannette Lee: Writing – review & editing, Formal analysis, Conceptualization. Jessica Snowden: Writing – review & editing, Conceptualization. Anita C. Walden: Writing – review & editing, Conceptualization. Alan E. Simon: Writing – review & editing, Methodology, Formal analysis, Conceptualization. Lori A. Devlin: Writing – review & editing, Conceptualization. Leslie W. Young: Writing – review & editing, Conceptualization. Meredith N. Zozus: Writing – review & editing, Writing – original draft, Methodology, Data curation, Conceptualization.

☆

This work was conducted while Dr. Garza was at the University of Arkansas for Medical Sciences (UAMS) and while Dr. Simon was at the ECHO Program at the NIH. Dr. Garza is now at the University of Texas for Health Sciences (UTHealth) at San Antonio. Dr. Simon is now at the National Center for Health Statistics (NCHS), Centers for Disease Control and Prevention (CDC).

Availability of Data and Materials.

The dataset(s) supporting the conclusions of this manuscript is(are) included within the manuscript (and its additional file(s)).

References

  • [1].SCDM. Good clinical data management practices (GCDMP). Published online 2013. https://scdm.org/wp-content/uploads/2019/10/21117-Full-GCDMP-Oct-2013.pdf. [Google Scholar]
  • [2].Kahn MG, Brown JS, Chun AT, et al. , Transparent reporting of data quality in distributed data networks, eGEMs. 3 (1) (2015) 1052, 10.13063/2327-9214.1052. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [3].American Society for Quality. Quality. In: American Society for Quality Glossary. American Society for Quality; 2008. [Google Scholar]
  • [4].Juran JM, Godfrey AB, Juran’s quality handbook, 5th Edition., McGraw Hill Professional, 1999. [Google Scholar]
  • [5].Crosby PB, Let’s Talk Quality, 1st Ed., McGraw Hill Professional, 1989. [Google Scholar]
  • [6].Division of Health Sciences Policy, Institute of Medicine (IOM). Assuring Data Quality and Validity in Clinical Trials for Regulatory Decision Making: Workshop Report. (Davis JR, Nolan VP, Woodcock J, Estabrook RW, eds.). The National Academies Press; 1999. doi: 10.17226/9623. [DOI] [Google Scholar]
  • [7].Knatterud GL, Rockhold FW, George SL, et al. , Guidelines for quality assurance in multicenter trials: a position paper, Control Clin Trials. 19 (5) (1998) 477–493, 10.1016/s0197-2456(98)00033-6. [DOI] [PubMed] [Google Scholar]
  • [8].Bagniewska A, Black D, Molvig K, et al. , Data quality in a distributed data processing system: the SHEP Pilot Study, Control Clin Trials. 7 (1) (1986) 27–37, 10.1016/0197-2456(86)90005-x. [DOI] [PubMed] [Google Scholar]
  • [9].van der Putten E, van der Velden JW, Siers A, Hamersma EA, A pilot study on the quality of data management in a cancer clinical trial, Control Clin Trials. 8 (2) (1987) 96–100, 10.1016/0197-2456(87)90034-1. [DOI] [PubMed] [Google Scholar]
  • [10].Knatterud GL , Methods of quality control and of continuous audit procedures for controlled clinical trials, Control Clin Trials. 1 (4) (1981) 327–332, 10.1016/0197-2456(81)90036-2. [DOI] [PubMed] [Google Scholar]
  • [11].Freeland B, Moving toward continuous quality improvement, J Intraven Nurs. 15 (1992) 278–282. [PubMed] [Google Scholar]
  • [12].Christian MC, McCabe MS, Korn EL, Abrams JS, Kaplan RS, Friedman MA, The national cancer institute audit of the national surgical adjuvant breast and bowel project protocol B-06, N Engl J Med. 333 (22) (1995) 1469–1475, 10.1056/NEJM199511303332206. [DOI] [PubMed] [Google Scholar]
  • [13].Blumenstein BA, Verifying keyed medical research data, Stat Med. 12 (17) (1993) 1535–1542, 10.1002/sim.4780121702. [DOI] [PubMed] [Google Scholar]
  • [14].Neaton JD, Duchene AG, Svendsen KH, Wentworth D, An examination of the efficiency of some quality assurance methods commonly employed in clinical trials, Stat Med. 9 (1–2) (1990) 115–123, 10.1002/sim.4780090118, discussion 124. [DOI] [PubMed] [Google Scholar]
  • [15].Day S, Fayers P, Harvey D, Double data entry: what value, what price? Control Clin Trials. 19 (1) (1998) 15–24, 10.1016/s0197-2456(97)00096-2. [DOI] [PubMed] [Google Scholar]
  • [16].Steward WP, Vantongelen K, Verweij J, Thomas D, Van Oosterom AT, Chemotherapy administration and data collection in an EORTC collaborative group–can we trust the results? Eur J Cancer Oxf Engl 29A (7) (1990. 1993,) 943–947, 10.1016/s0959-8049(05)80199-6. [DOI] [Google Scholar]
  • [17].Van den Broeck J, Mackay M, Mpontshane N, Kany Kany Luabeya A, Chhagan M, Bennish ML , Maintaining data integrity in a rural clinical trial, Clin Trials Lond Engl. 4 (5) (2007) 572–582, 10.1177/1740774507084106. [DOI] [Google Scholar]
  • [18].Hall HI, Gerlach KA, Miller DS, Methods of quality management: accuracy, J Regist Manag. 29 (2002) 72–77. [Google Scholar]
  • [19].Nies MA, Hein L, Teleform: a blessing or burden? Public Health Nurs Boston Mass. 17 (2) (2000) 143–145, 10.1046/j.1525-1446.2000.00143.x. [DOI] [Google Scholar]
  • [20].Crewson PE, Applegate KE, Fundamentals of clinical research for radiologists: data collection in radiology research, Am J Roentgenol. 177 (4) (2001) 755–761, 10.2214/ajr.177.4.1770755. [DOI] [PubMed] [Google Scholar]
  • [21].Keralis SD, Stark S, Halbert M, Moen WE. Research data management in policy and practice: the DataRes project. In: Research Data Management: Principles, Practices, and Prospects. Council on Library and Information Resources (CLIR); 2013:16–38. http://www.clir.org/pubs/reports/pub160. [Google Scholar]
  • [22].Williams M, Bagwell J, Zozus MN, Data management plans, the missing perspective, J Biomed Inform. 71 (2017) 130–142, 10.1016/j.jbi.2017.05.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [23].Nahm ML, Pieper C, Cunningham M, Quantifying data quality for clinical trials using electronic data capture, PLoS One. 3 (2008) e3049. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [24].Zozus MN, Sanns W, EL EDC . Beyond, J Soc Clin Data Manag. 1 (1) (2021), 10.47912/jscdm.33. [DOI] [Google Scholar]
  • [25].Garza MY, Williams TB, Myneni S, et al. Measuring and controlling medical record abstraction (MRA) error rates in an observational study. Published online March 11, 2022. doi: 10.21203/rs.3.rs-1225727/v1. [DOI] [Google Scholar]
  • [26].Helms RW, A distributed flat file strategy for managing research data, in: Proceedings of the ACM 1980 Annual Conference. ACM ‘80. Association for Computing Machinery, 1980, 10.1145/800176.809982. [DOI] [Google Scholar]
  • [27].Collen MF , Clinical research databases–a historical review, J Med Syst. 14 (6) (1990) 323–344, 10.1007/BF00996713. [DOI] [PubMed] [Google Scholar]
  • [28].Nahm M, Clinical data management: why we do what we do? Data Basics. Published Online 13 (2007). [Google Scholar]
  • [29].Arts DGT, De Keizer NF, Scheffer GJ, Defining and improving data quality in medical registries: a literature review, case study, and generic framework, J Am Med Inform Assoc JAMIA. 9 (6) (2002) 600–611, 10.1197/jamia.m1087. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [30].Nahm M, Dziem G, Fendt K, Freeman L, Masi J, Ponce Z, Data quality survey results, Data Basics. 10 (2004) 13–19. [Google Scholar]
  • [31].Caballero I, Verbo E, Calero C, Piattini M. MMPRO: Amethodology based on ISO/IEC 15939 to draw up data quality measurement processes. In: Proceedings of the 13th International Conference on Information Quality, MIT.; 2008:340. [Google Scholar]
  • [32].Barker TH, Migliavaca CB, Stein C, et al. , Conducting proportional meta-analysis in different types of systematic reviews: a guide for synthesisers of evidence, BMC Med Res Methodol. 21 (1) (2021) 189, 10.1186/s12874-021-01381-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [33].Schwarzer G, Rücker G, Meta-analysis of proportions, Methods Mol Biol Clifton NJ. 2345 (2022) 159–172, 10.1007/978-1-0716-1566-9_10. [DOI] [Google Scholar]
  • [34].Chen Y, Chen D, Wang Y, Han Y, Using Freeman-Tukey double arcsine transformation in meta-analysis of single proportions, Aesthetic Plast Surg. 47 (1) (2023) 83–84, 10.1007/s00266-022-02977-6. [DOI] [PubMed] [Google Scholar]
  • [35].McCulloch CE. An introduction to generalized linear mixed models. Conf Appl Stat Agric. Published online April 28, 1996. doi: 10.4148/2475-7772.1314. [DOI] [Google Scholar]
  • [36].Balduzzi S, Rücker G, Schwarzer G, How to perform a meta-analysis with R: a practical tutorial, Evid Based Ment Health. 22 (4) (2019) 153–160, 10.1136/ebmental-2019-300117. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [37].Viechtbauer W, Conducting meta-analyses in R with the metafor Package, J Stat Softw. 36 (2010) 1–48, 10.18637/jss.v036.i03. [DOI] [Google Scholar]
  • [38].Kutner N, Neter Li, Applied linear statistical models, 5th ed., McGraw-Hill Education, 2004. [Google Scholar]
  • [39].Montgomery DC, Peck EA, Vining GG, Introduction to linear regression analysis, 6th ed., Wiley, 2012. [Google Scholar]
  • [40].Higgins JPT, Thompson SG, Deeks JJ, Altman DG, Measuring inconsistency in meta-analyses, BMJ. 327 (7414) (2003) 557–560, 10.1136/bmj.327.7414.557. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [41].Gareth J, Witten D, Hastie T, Tibshirani R, An introduction to statistical learning, 2nd ed., Springer, 2021, 10.1007/978-1-0716-1418-1. [DOI] [Google Scholar]
  • [42].Prud’homme GJ, Canner PL, Cutler JA, Quality assurance and monitoring in the hypertension prevention trial. hypertension prevention trial research group, Control Clin Trials. 10 (3 Suppl) (1989) 84S–94S, 10.1016/0197-2456(89)90044-5. [DOI] [Google Scholar]
  • [43].Fabri J, Beller E, An assessment of the value of double data entry and interactive checking [abstract], Control Clin Trials. 14 (1993) 405. [Google Scholar]
  • [44].Kantonen I, Lepäntalo M, Salenius JP, et al. , Auditing a nationwide vascular registry–the 4-year finnvasc experience. finnvasc study group, Eur J Vasc Endovasc Surg Off J Eur Soc Vasc Surg. 14 (6) (1997) 468–474, 10.1016/s1078-5884(97)80126-8. [DOI] [Google Scholar]
  • [45].McEntegart DJ, Jadhav SP, Brown T, Channon EJ, Checks of case record forms versus the database for efficacy variables when validation programs exist, Drug Inf J. 33 (1) (1999) 101–107, 10.1177/009286159903300113. [DOI] [Google Scholar]
  • [46].Wahi MM, Parks DV, Skeate RC, Goldin SB, Reducing errors from the electronic transcription of data collected on paper forms: a research data case study, J Am Med Inform Assoc JAMIA. 15 (3) (2008) 386–389, 10.1197/jamia.M2381. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [47].Crombie IK, Irving JM, An investigation of data entry methods with a personal computer, Comput Biomed Res Int J. 19 (6) (1986) 543–550, 10.1016/0010-4809(86)90028-5. [DOI] [Google Scholar]
  • [48].Birkett NJ, Computer-aided personal interviewing. A new technique for data collection in epidemiologic surveys, Am J Epidemiol. 127 (3) (1988) 684–690, 10.1093/oxfordjournals.aje.a114843. [DOI] [PubMed] [Google Scholar]
  • [49].Jasperse DM, Ahmed SW, The mid-atlantic oncology program’s comparison of two data collection methods, Control Clin Trials. 10 (3) (1989) 282–289, 10.1016/0197-2456(89)90068-8. [DOI] [PubMed] [Google Scholar]
  • [50].McKinlay SM, Carleton RA, McKenney JL, Assaf AR, A new approach to surveillance for acute myocardial infarction: reproducibility and cost efficiency, Int J Epidemiol. 18 (1) (1989) 67–75, 10.1093/ije/18.1.67. [DOI] [PubMed] [Google Scholar]
  • [51].Reynolds-Haertle RA, McBride R, Single vs. double data entry in CAST, Control Clin Trials. 13 (6) (1992) 487–494, 10.1016/0197-2456(92)90205-e. [DOI] [PubMed] [Google Scholar]
  • [52].Gibson D, Harvey AJ, Everett V, Parmar MK, Is double data entry necessary? The CHART trials. CHART steering committee. continuous, hyperfractionated, accelerated radiotherapy, Control Clin Trials. 15 (6) (1994) 482–488, 10.1016/0197-2456(94)90005-1. [DOI] [PubMed] [Google Scholar]
  • [53].Horbar JD, Leahy KA, An assessment of data quality in the vermont-oxford trials network database, Control Clin Trials. 16 (1) (1995) 51–61, 10.1016/0197-2456(94)00019-y. [DOI] [PubMed] [Google Scholar]
  • [54].Lancaster S, Hallstrom A, McBride R, Morris M, A comparison of key data entry versus fax data entry, accuracy and time [abstract], Control Clin Trials 16 (1995) 75–76. [Google Scholar]
  • [55].Diener-West M, Connor PB, Newhouse MM, Hawkins BS, Feasibility of keying data from screen-displayed facsimile images in an ongoing trial: the collaborative ocular melanoma study, Control Clin Trials. 19 (1) (1998) 39–49, 10.1016/s0197-2456(97)00094-9. [DOI] [PubMed] [Google Scholar]
  • [56].Goldhill DR, Sumner A, APACHE II, data accuracy and outcome prediction, Anaesthesia. 53 (10) (1998) 937–943, 10.1046/j.1365-2044.1998.00534.x. [DOI] [PubMed] [Google Scholar]
  • [57].Velikova G, Wright EP, Smith A, et al. , Automated collection of quality-of-life data: a comparison of paper and computer touch-screen questionnaires, J Clin Oncol off J Am Soc Clin Oncol. Published Online (1999), 10.1200/JCO.1999.17.3.998. [DOI] [Google Scholar]
  • [58].Håkansson I, Lundström M, Stenevi U, Ehinger B, Data reliability and structure in the swedish national cataract register, Acta Ophthalmol Scand. 79 (5) (2001) 518–523, 10.1034/j.1600-0420.2001.790519.x. [DOI] [PubMed] [Google Scholar]
  • [59].Arts D, de Keizer N, Scheffer GJ, de Jonge E, Quality of data collected for severity of illness scores in the Dutch National Intensive Care Evaluation (NICE) registry, Intensive Care Med. 28 (5) (2002) 656–659, 10.1007/s00134-002-1272-z. [DOI] [PubMed] [Google Scholar]
  • [60].Kawado M, Hinotsu S, Matsuyama Y, Yamaguchi T, Hashimoto S, Ohashi Y, A comparison of error detection rates between the reading aloud method and the double data entry method, Control Clin Trials. 24 (5) (2003) 560–569, 10.1016/s0197-2456(03)00089-8. [DOI] [PubMed] [Google Scholar]
  • [61].Rostami R, Data quality in a global environment, Society for Clinical Data Management (2004). [Google Scholar]
  • [62].Pan L, Fergusson D, Schweitzer I, Hebert PC, Ensuring high accuracy of data abstracted from patient charts: the use of a standardized medical record as a training tool, J Clin Epidemiol. 58 (9) (2005) 918–923, 10.1016/j.jclinepi.2005.02.004. [DOI] [PubMed] [Google Scholar]
  • [63].Fontaine P, Mendenhall TJ, Peterson K, Speedie SM, The “Measuring Outcomes of Clinical Connectivity” (MOCC) trial: investigating data entry errors in the electronic primary care research network (ePCRN), J Am Board Fam Med JABFM. 20 (2) (2007) 151–159, 10.3122/jabfm.2007.02.060069. [DOI] [PubMed] [Google Scholar]
  • [64].Hogan WR, Wagner MM, Accuracy of data in computer-based patient records, J Am Med Inform Assoc. 4 (5) (1997) 342–355. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [65].Jansen ACM, van Aalst-Cohen ES, Hutten BA, Büller HR, Kastelein JJP, Prins MH, Guidelines were developed for data collection from medical records for use in retrospective analyses, J Clin Epidemiol. 58 (3) (2005) 269–274, 10.1016/j.jclinepi.2004.07.006. [DOI] [PubMed] [Google Scholar]
  • [66].Smyth ET, Emmerson AM, Survey of infection in hospitals: use of an automated data entry system, J Hosp Infect. 34 (2) (1996) 87–97, 10.1016/s0195-6701(96)90133-4. [DOI] [PubMed] [Google Scholar]
  • [67].Jørgensen CK, Karlsmose B, Validation of automated forms processing. A comparison of Teleform with manual data entry, Comput Biol Med. 28 (6) (1998) 659–667, 10.1016/s0010-4825(98)00038-9. [DOI] [PubMed] [Google Scholar]
  • [68].O’Rourke MK, Fernandez LM, Bittel CN, Sherrill JL, Blackwell TS, Robbins DR, Mass data massage: an automated data processing system used for NHEXAS, arizona. national human exposure assessment survey, J Expo Anal Environ Epidemiol. 9 (5) (1999) 471–484, 10.1038/sj.jea.7500043. [DOI] [PubMed] [Google Scholar]
  • [69].Dyck PJ, Turner DW, Davies JL, et al. , Electronic case-report forms of symptoms and impairments of peripheral neuropathy, Can J Neurol Sci J Can Sci Neurol. 29 (3) (2002) 258–266, 10.1017/s0317167100002043. [DOI] [Google Scholar]
  • [70].Jinks C, Jordan K, Croft P, Evaluation of a computer-assisted data entry procedure (including Teleform) for large-scale mailed surveys, Comput Biol Med. 33 (5) (2003) 425–437, 10.1016/s0010-4825(03)00012-x. [DOI] [PubMed] [Google Scholar]
  • [71].Quan KH, Vigano A, Fainsinger RL, Evaluation of a data collection tool (TELEform) for palliative care research, J Palliat Med. 6 (3) (2003) 401–408, 10.1089/109662103322144718. [DOI] [PubMed] [Google Scholar]
  • [72].Hardin JM, Woodby LL, Crawford MA, Windsor RA, Miller TM, Data collection in a multisite project: teleform, Public Health Nurs Boston Mass. 22 (4) (2005) 366–370, 10.1111/j.0737-1209.2005.220410.x. [DOI] [Google Scholar]
  • [73].Guerette P, Robinson B, Moran WP, et al. , Teleform scannable data entry: an efficient method to update a community-based medical record? Community care coordination network Database Group, Proc Symp Comput Appl Med Care (1995) 86–90. Published Online. [Google Scholar]
  • [74].Harris D, Effect of equipment complexity on inspection performance, J Appl Psychol. 50 (3) (1966) 236–237, 10.1037/h0023419. [DOI] [PubMed] [Google Scholar]
  • [75].Yerushalmy J, The statistical assessment of the variability in observer perception and description of roentgenographic pulmonary shadows, Radiol Clin North Am. 7 (3) (1969) 381–392, 10.1016/S0033-8389(22)01511-1. [DOI] [PubMed] [Google Scholar]
  • [76].Schoonard JW, Gould JD, Field of view and target uncertainty in visual search and inspection, Hum Factors. 15 (1) (1973) 33–42, 10.1177/001872087301500105. [DOI] [Google Scholar]
  • [77].Drury CG, Fox JG, Human reliability in quality control, Taylor & Francis Ltd, 1975. [Google Scholar]
  • [78].Kirwan B, A guide to practical human reliability assessment, 1st Edition, CRC Press, 1994. [Google Scholar]
  • [79].Nahm M Measuring data quality. In: Good Clinical Data Management Practices (GCDMP) (Version 2000 - Present). Society for Clinical Data Management; 2012. https://www.scdm.org/gcdmp. [Google Scholar]
  • [80].FDA. Real-World Data: Assessing Electronic Health Records and Medical Claims Data to Support Regulatory Decision Making for Drug and Biological Products: Draft Guidance for Industry. Health and Human Services, United States Food and Drug Administration; 2021. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/real-world-data-assessing-electronic-health-records-and-medical-claims-data-support-regulatory. [Google Scholar]
  • [81].Alberg AJ, Park JW, Hager BW, Brock MV, Diener-West M, The use of “overall accuracy” to evaluate the validity of screening or diagnostic tests, J Gen Intern Med. 19 (5 Pt 1) (2004) 460–465, 10.1111/j.1525-1497.2004.30091.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [82].Gwet KL, Handbook of inter-rater reliability: the definitive guide to measuring the extent of agreement among raters, 4th Edition, Publisher: Advanced Analytics, LLC, 2014. [Google Scholar]
  • [83].Garza M, Myneni S, Nordo A, et al. , eSource for standardized health information exchange in clinical research: a systematic review, Stud Health Technol Inform. 257 (2019) 115–124. [PubMed] [Google Scholar]
  • [84].Garza M, Myneni S, Fenton SH, Zozus MN, eSource for standardized health information exchange in clinical research: a systematic review of progress in the last year, J Soc Clin Data Manag. 1 (2) (2021), 10.47912/jscdm.66. [DOI] [Google Scholar]
  • [85].Garza MY, Prior F, Sanford JA, Sexton KW, Zozus MN, Innovative solutions to streamline data collection, exchange, and utilization in translational research, J Clin Transl Sci. 6 (1) (2022) 94, 10.1017/cts.2022.277. [DOI] [Google Scholar]
  • [86].Garza MY, Rutherford M, Myneni S, et al. , Evaluating the coverage of the HL7® FHIR® standard to support eSource data exchange implementations for use in multi-site clinical research studies, AMIA Annu Symp Proc AMIA Symp. 2020 (2020) 472–481. [PMC free article] [PubMed] [Google Scholar]
  • [87].Garza MY, Rutherford MW, Adagarla B, et al. , Evaluating site-level implementations of the hl7 fhir standard to support esource data exchange in clinical research, Stud Health Technol Inform. 281 (2021) 397–401, 10.3233/SHTI210188. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [88].HL7. HL7 FHIR Release 4 (R4). Health Level Seven (HL7) International. Published November 1, 2019. http://hl7.org/fhir/R4/index.html. [Google Scholar]
  • [89].FDA. E6(R2) Good Clinical Practice: Integrated Addendum to ICH E6(R1). Published online 2018. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/e6r2-good-clinical-practice-integrated-addendum-ich-e6r1. [Google Scholar]
  • [90].Freedman LS, Schatzkin A, Wax Y, The impact of dietary measurement error on planning sample size required in a cohort study, Am J Epidemiol. 132 (6) (1990) 1185–1195, 10.1093/oxfordjournals.aje.a115762. [DOI] [PubMed] [Google Scholar]
  • [91].Rostami R, Nahm M, Pieper CF, What can we learn from a decade of database audits? The Duke Clinical Research Institute experience, 1997–2006, Clin Trials Lond Engl. 6 (2) (2009) 141–150, 10.1177/1740774509102590. [DOI] [Google Scholar]
  • [92].Perkins DO, Wyatt RJ, Bartko JJ, Penny-wise and pound-foolish: the impact of measurement error on sample size requirements in clinical trials, Biol Psychiatry. 47 (8) (2000) 762–766, 10.1016/s0006-3223(00)00837-4. [DOI] [PubMed] [Google Scholar]
  • [93].Eisenstein EL, Lemons PW, Tardiff BE, Schulman KA, Jolly MK, Califf RM, Reducing the costs of phase III cardiovascular clinical trials, Am Heart J. 149 (3) (2005) 482–488, 10.1016/j.ahj.2004.04.049. [DOI] [PubMed] [Google Scholar]
  • [94].Liu K, Measurement error and its impact on partial correlation and multiple linear regression analyses, Am J Epidemiol. 127 (4) (1988) 864–874, 10.1093/oxfordjournals.aje.a114870. [DOI] [PubMed] [Google Scholar]
  • [95].Mullooly JP, The effects of data entry error: an analysis of partial verification, Comput Biomed Res Int J. 23 (3) (1990) 259–267, 10.1016/0010-4809(90)90020-d. [DOI] [Google Scholar]
  • [96].Stepnowsky CJ, Berry C, Dimsdale JE, The effect of measurement unreliability on sleep and respiratory variables, Sleep. 27 (5) (2004) 990–995, 10.1093/sleep/27.5.990. [DOI] [PubMed] [Google Scholar]
  • [97].Williams SC, Watt A, Schmaltz SP, Koss RG, Loeb JM, Assessing the reliability of standardized performance indicators, Int J Qual Health Care J Int Soc Qual Health Care. 18 (3) (2006) 246–255, 10.1093/intqhc/mzi098. [DOI] [Google Scholar]
  • [98].Hospital quality data: CMS needs more rigorous methods to ensure reliability of publicly released data | U.S. GAO. Accessed July 19, 2023. https://www.gao.gov/products/gao-06-54. [Google Scholar]
  • [99].Braun BI, Kritchevsky SB, Kusek L, et al. , Comparing bloodstream infection rates: the effect of indicator specifications in the evaluation of processes and indicators in infection control (EPIC) study, Infect Control Hosp Epidemiol. 27 (1) (2006) 14–22, 10.1086/498966. [DOI] [PubMed] [Google Scholar]
  • [100].Jacobs R, Goddard M, Smith PC, How robust are hospital ranks based on composite performance measures? Med Care. 43 (12) (2005) 1177–1184, 10.1097/01.mlr.0000185692.72905.4a. [DOI] [PubMed] [Google Scholar]
  • [101].Pagel C, Gallivan S, Exploring potential consequences on mortality estimates of errors in clinical databases, IMA J Manag Math. 20 (4) (2009) 385–393, 10.1093/imaman/dpn034. [DOI] [Google Scholar]
  • [102].Brien SE, Ghali WA, Public reporting of the hospital standardized mortality ratio (HSMR): implications for the Canadian approach to safety and quality in health care, Open Med Peer-Rev Indep Open-Access J. 2 (3) (2008) e70–e73. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary Data 1
NIHMS2153678-supplement-Supplementary_Data_1.docx (70.4KB, docx)
Supplementary Data 2
NIHMS2153678-supplement-Supplementary_Data_2.docx (1.2MB, docx)
Supplementary Data 3
NIHMS2153678-supplement-Supplementary_Data_3.docx (27.5KB, docx)

Data Availability Statement

The dataset(s) supporting the conclusions of this manuscript is(are) included within the manuscript (and its additional file(s)).

RESOURCES