Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Feb 1.
Published in final edited form as: J Biomed Inform. 2012 Sep 10;46(1):139–141. doi: 10.1016/j.jbi.2012.08.007

Publication Bias in Clinical Trials of Electronic Health Records

David K Vawdrey 1, George Hripcsak 1
PMCID: PMC3662474  NIHMSID: NIHMS407083  PMID: 22975314

Abstract

Objective

To measure the rate of non-publication and assess possible publication bias in clinical trials of electronic health records.

Methods

We searched ClinicalTrials.gov to identify registered clinical trials of electronic health records and searched the biomedical literature and contacted trial investigators to determine whether the results of the trials were published. Publications were judged as positive, negative, or neutral according to the primary outcome.

Results

76% of trials had publications describing trial results; of these, 74% were positive, 21% were neutral, and 4% were negative (harmful). Of unpublished studies for which the investigator responded, 43% were positive, 57% were neutral, and none were negative; the lower rate of positive results was significant (p<0.001).

Conclusion

The rate of non-publication in electronic health record studies is similar to that in other biomedical studies. There appears to be a bias toward publication of positive trials in this domain.

Background

Publication bias refers to the selective publication or suppression of research results according to outcome. In multiple scientific domains, there has been a greater likelihood for publication when studies have positive results [13]. A recent review assessed the effect of health information technology on care quality, efficiency, and provider satisfaction, finding that 92% of articles reached conclusions that were positive [4]. Although the positive findings associated with health information technology adoption are encouraging for those implementing electronic health records, we hypothesized that the true rate of positive results in trials of electronic health records may differ from the published rate due to publication bias.

Advance registration of clinical trials has been introduced as a method to reduce publication bias by making available a catalog of the trials that are less likely to be published due to neutral or negative results [5]. ClinicalTrials.gov is an information resource maintained by the United States National Library of Medicine that provides a registry of both federally and privately funded clinical trials since February 2000. Journals whose editors belong to the International Committee of Medical Journal Editors (ICMJE) will only publish clinical trial results if the trial is registered with ClinicalTrials.gov or another ICMJE approved trial registry before the first patient is recruited [6].

When the ICMJE adopted the policy requiring registration of clinical trials, they noted that only ClinicalTrials.gov met their standards for free public access, completeness and validity of the registration data, electronic search capability [7]. While other registries exist (notably the International Standard Randomised Controlled Trial Number [8]), ClinicalTrials.gov appears to be the preferred Web site for posting clinical trials [9]. A U.S. federal law enacted in 2007 mandates ClinicalTrials.gov registration and basic results reporting for interventional studies of drugs, biological products, and devices, regardless of sponsor or funding source [10].

We hypothesized that ClinicalTrials.gov could be used to examine potential publication bias in studies of electronic health records (EHRs), based on the number of completed clinical trials that were never published. We recognize that not all EHR studies are registered in ClinicalTrials.gov. For example, ICMJE guidelines may not apply where the unit of randomization in a trial is the healthcare system or care delivery location as opposed to an individual human subject. Moreover, while any study can be registered with ClinicalTrials.gov, studies that are observational, or that lack a control group, are less likely to be registered than other trials. Nevertheless, ClinicalTrials.gov is a convenient, publicly available repository of trial information that can provide an estimate of publication rates related to EHR studies.

Methods

We queried ClinicalTrials.gov for clinical trials of electronic health records using the following search phrases: “electronic health records,” “electronic medical records,” “electronic documentation,” “electronic prescribing,” “electronic reminders,” and “CPOE” (computerized provider order entry) for all trials registered with ClinicalTrials.gov from 2000 to 2008. Trials were reviewed and excluded if they did not involve a primary intervention that was use of an electronic health record, use of an electronic prescribing or ordering system, or delivery of electronic reminders. Those found to be in scope were reviewed by the authors to ascertain whether the trial was completed and whether it was published in a peer-reviewed journal. Publication was determined by reviewing the ClinicalTrials.gov record, which occasionally listed publications, by searching PubMed, and by contacting trial investigators. The authors (DKV and GH) searched PubMed for all publications by all personnel listed in the ClinicalTrials.gov record. If no relevant publication was found, then secondary search terms were used, such as the name of the trial, the sponsoring consortium, or the location of the trial. If still no publication was identified for a trial, the authors attempted to contact the principal investigator listed in ClinicalTrials.gov to determine the trial results and whether there was a resulting publication.

Trials were considered “completed” if the Recruitment or Completion Date fields of the ClinicalTrials.gov record indicated that they were completed by 2009 or if there was a definitive publication by the investigators. Completed here implies that a sufficient portion of the trial was completed to allow for publication of results, even though aspects of the trial may have been ongoing. If trial status was not recorded by December 2009 and there was no publication through July 2012, they were considered “missing publication.” If trials were designated as completed or expected to be completed after July 2012, then they were considered “ongoing” (even if the intervention was over, there may have been insufficient time to publish results).

Trials with one or more publications were then categorized using the framework of Buntin et al. [4] as being positive (including trials with mixed results that were predominantly positive), neutral (no effect), or negative (meaning actually harmful). Broadly speaking, most studies involving health information technology have mixed results. For example, an EHR reminder might improve adherence for a quality measure at the expense of requiring extra time or additional mouse clicks for clinicians. In our categorization, positive outcomes were recorded when health information technology was associated with improvement in one or more aspects of care, and in the case of mixed results, the overall conclusion of the study authors was that the positive effects of technology outweighed the negative effects. Negative outcomes were recorded when the negative effects of the technology intervention outweighed the positive effects. A neutral rating was given when the study reported no demonstrable change in care.

Results

The results are shown in Table 1. One hundred twelve trials were identified in ClinicalTrials.gov using the search phrases; 85 were found to be within scope, and 62 were completed. Of completed trials, 76% had publications that described trial results. Most publications (74%) reported positive results, with 21% reporting neutral results and 4% reporting negative (harmful) results. Of the 24% of completed trials that had no publications, 8 were studies where the principal investigator (PI) listed in ClinicalTrials.gov did not respond to our repeated requests for information (in one instance, the PI was deceased), 3 were studies where the PI reported positive results, and 4 were studies where the PI reported neutral results. PIs identified several reasons for the lack of publications about a trial, such as: key members had left the project team, they were too busy or had not had sufficient time to publish, and their manuscript was rejected by journal editors.

Table 1.

Disposition of completed trials of electronic health records registered in ClinicalTrials.gov from 2000–2008.

Trials (%) Published Trials (%) Unpublished Trials (%)
Negative result   2   (4) *   2  (4)   0   (0)*
Neutral result 14 (26) * 10 (21)   4 (57)*
Positive result 38 (70) * 35 (74)   3 (43)*
Unknown result   8   0   8
TOTAL 62 47 15
*

Computed percentage omits trials with “Unknown” results.

Discussion

Two particular issues may affect our results. First, in the initial years of ClinicalTrials.gov, the resource was not well known so few trials were registered. Moreover, the few trials that were registered may have been unusual (e.g., investigators may have registered trials that were more likely to be published), and the data were frequently unreliable (e.g., in several cases, the completion date preceded the start date). Before 2005, there were only zero to three registered EHR studies per year; after that there were five or more. The second concern is that the recent trials may not have had enough time to be published. The mean and median time to publication were 4.3 years and 4.1 years, respectively. Based on these data and on knowledge of the publication process, most trials completed by 2007 should have had enough time to be published. Taking only trials that completed between 2005 and 2007 (inclusive), 75% were published and 25% were not, corroborating our main result. Therefore, despite concerns on either end, our overall non-publication result of 24% appears to be reasonable.

The classification of study results was performed by the two authors together; the study may be limited by the fact that no formal calculation of inter-observer agreement was made. Nevertheless, our analysis does provide bounds to the proportion of registered trials with positive results. Depending on the outcome of trials with missing results, the number of positive trials ranged from 61–74%. In 7 unpublished trials for which trial investigators reported results, 3 studies were positive and 4 were neutral. Thus, unpublished trials were less likely to have positive results (3/7) compared to published trials (35/47, p<0.001). In other evaluations of publication bias, trials with missing results were more likely to be negative or neutral than positive [11,12].

Assuming that the disposition of trials with missing results is similar to other unpublished trials, our best estimate is that approximately 67% of EHR trials had positive results. This is somewhat less than the 92% reported by Buntin et al. [4]. Other than publication bias, several reasons may account for the difference. Our study focused specifically on electronic health records, while Buntin’s review also included telemedicine, administrative functions, information retrieval, patient registries, health information exchange, and personal health records. Moreover, our study examined only trials registered in ClinicalTrials.gov. This repository does not contain a comprehensive listing of all studies of health information technology interventions. For example, studies where individual subjects are not considered to be “at risk” from the intervention may not require trial registration according to ICMJE or other policies. It is conceivable that the sample of EHR studies that were registered in ClinicalTrials.gov had a greater likelihood of being published than the larger population of all studies of EHR from the same time period. Thus, our findings may overestimate the publication rate for HER interventions.

Our study complements that of Ammenwerth and de Keizer [13], who used a survey to assess non-publication of medical informatics studies. While the response rate was low, they found that over one-third of studies are not published, a finding that is close to our results. The authors elicited several reasons for non-publication, including the results not being of interest for others, the publication being in preparation, not having time for publication, limited scientific quality of the study, political or legal reasons, and the study only being conducted for internal use. Several of these reasons would not be relevant to trials registered in ClinicalTrials.gov. Machan et al. [14] reported a similar rate of positive published study results (69.8%) to ours (74%).

Other investigators have used ClinicalTrials.gov to study publication outcomes [15]. In 2007, Ross et al. examined a cross-section of all trials registered in ClinicalTrials.gov, finding that of trials that ended by 2004 had a 61% publication rate [16]. Bourgeois and colleagues reported a 66.3% publication rate for drug trials registered in Clinical Trials.gov [17]. These findings suggest that the rate of non-publication for studies of EHRs in this data set is comparable to other types of studies. Bourgeois and colleagues found a statistically significant difference in the outcomes of studies that were funded by industry (85.4%) compared with government (50.0%) and nonprofit or nonfederal organizations (71.9%) [17]. We did not have a large enough sample to stratify our results according to funding source, but it is possible that published outcomes of EHR studies may be influenced by this parameter.

There is no simple answer to the question of how many studies "ought" to be positive. Even if electronic health records are generally beneficial, as researchers develop innovative interventions and employ clinical equipoise in designing trials, one may expect some substantial number to be positive, another substantial number to be neutral (due to failed innovations, unsuccessful implementations of efficacious interventions, and underpowered studies), and a small number to be negative (harmful). This is consistent with our results.

Implications

Our results demonstrate that there is a moderate amount of non-publication of EHR trials, and that the rate is in line with that of other areas. Such non-publication may lead to publication bias. This finding has several implications.

Investigators and publishers should continue to be encouraged to disseminate neutral and negative results [18]. In the still-nascent field of health information technology, it is just as important to understand why systems fail as it is to learn about why they succeed. Research investigators who also happen to be the developers or implementers of health information technology should recognize the subtle ways that they may be influenced by their preexisting belief in the technology’s value [19] and how this may affect their own likelihood to publish the results. Publishers should also be mindful of the developer-evaluator bias, perhaps recommending manuscript authors to provide a disclosure of such “personal” conflicts of interest in the same way financial conflicts are currently revealed. Additional research could be conducted to assess differences in publication rates and outcomes for studies based on whether the evaluator of a system or application also plays a key role in its development or implementation.

Policymakers should be aware that publication bias likely has some effect on meta-analyses of studies, such as the recent evaluation performed by Buntin et al. [4]. Health information technology meta-analyses should comment specifically on the possibility of non-publication and should estimate the range of its likely effect on the results of the analysis.

Our main implication, however, is that the rate of non-publication of EHR studies does not appear to be larger than in other domains. There may be a small inflation of the proportion of positive trials published, but the potential difference probably is not large enough to warrant undue concern. Based on our data, it is likely that the majority of EHR studies produce positive results. Nevertheless, publication of negative results remains essential; indeed, such studies have provided substantial benefit by warning would-be adopters of implementation pitfalls as well as alerting the EHR software development community of problems that need to be addressed [2022].

Supplementary Material

01

Highlights.

  • We assessed publication bias in clinical trials of electronic health records.

  • Of 62 trials, 76% had publications describing trial results; of these, 74% had positive results.

  • Unpublished trials were less likely to have positive results compared to published trials.

  • Publication bias could affect meta-analyses of EHR studies.

Acknowledgment

Funding was provided by the Agency for Healthcare Research and Quality (R03 HS018250) and by the National Library of Medicine (R01 LM06910).

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

The authors have no financial interests, activities, relationships, or affiliations related to this work.

References

  • 1.Olson CM, Rennie D, Cook D, et al. Publication bias in editorial decision making. JAMA. 2002;287:2825–2828. doi: 10.1001/jama.287.21.2825. [DOI] [PubMed] [Google Scholar]
  • 2.Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ. 1997;315:640–645. doi: 10.1136/bmj.315.7109.640. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Friedman CP, Wyatt JC. Publication bias in medical informatics. J Am Med Inform Assoc. 2001;8(2):189–191. doi: 10.1136/jamia.2001.0080189. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff (Millwood) 2011;30:464–471. doi: 10.1377/hlthaff.2011.0178. [DOI] [PubMed] [Google Scholar]
  • 5.Rasmussen N, Lee K, Bero L. Association of trial registration with the results and conclusions of published trials of new oncology drugs. Trials. 2009;10:116. doi: 10.1186/1745-6215-10-116. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.International Committee of Medical Journal Editors. Uniform requirements for manuscripts submitted to biomedical journals: writing and editing for biomedical publication. [Accessed May 4, 2011]. http://www.icmje.org/urm_full.pdf. [PubMed] [Google Scholar]
  • 7.De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJ, Schroeder TV, Sox HC, Van Der Weyden MB International Committee of Medical Journal Editors. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Lancet. 2004 Sep 11-17;364(9438):911–912. doi: 10.1016/S0140-6736(04)17034-7. [DOI] [PubMed] [Google Scholar]
  • 8.Tonks A. A clinical trials register for Europe: Stop talking and start making it happen. BMJ. 2002 Dec 7;325(7376):1314–1315. doi: 10.1136/bmj.325.7376.1314. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Foote M. Clinical trial registries and clinical trial result posting: new paradigm for medical writers. Biotechnol Annu Rev. 2006;12:379–386. doi: 10.1016/S1387-2656(06)12011-6. [DOI] [PubMed] [Google Scholar]
  • 10.Tse T, Williams RJ, Zarin DA. Reporting "basic results" in ClinicalTrials.gov. Chest. 2009;136:295–303. doi: 10.1378/chest.08-3022. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008 Jan 17;358(3):252–260. doi: 10.1056/NEJMsa065779. [DOI] [PubMed] [Google Scholar]
  • 12.Smyth RM, Kirkham JJ, Jacoby A, Altman DG, Gamble C, Williamson PR. Frequency and reasons for outcome reporting bias in clinical trials: interviews with trialists. BMJ. 2011 Jan 6;342:c7153. doi: 10.1136/bmj.c7153. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Ammenwerth E, de Keizer N. A viewpoint on evidence-based health informatics, based on a pilot survey on evaluation studies in health care informatics. J Am Med Inform Assoc. 2007;14(3):368–371. doi: 10.1197/jamia.M2276. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Machan C, Ammenwerth E, Bodner T. Publication bias in medical informatics evaluation research: is it an issue or not? Stud Health Technol Inform. 2006;124:957–962. [PubMed] [Google Scholar]
  • 15.Mathieu S, Boutron I, Moher D, Altman DG, Ravaud PJ. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009;302:977–984. doi: 10.1001/jama.2009.1242. [DOI] [PubMed] [Google Scholar]
  • 16.Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM. PLoS Med. Trial publication after registration in ClinicalTrials.Gov: a cross-sectional analysis. 2009;6(9):e1000144. doi: 10.1371/journal.pmed.1000144. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Bourgeois FT, Murthy S, Mandl KD KD. Outcome reporting among drug trials registered in ClinicalTrials.gov . Ann Intern Med. 2010;153:158–166. doi: 10.1059/0003-4819-153-3-201008030-00006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Tierney W, McDonald C. Testing informatics innovations: the value of negative trials. J Am Med Inform Assoc. 1996;3(5):358–359. doi: 10.1136/jamia.1996.97035027. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Friedman C, Wyatt JC. Evaluation methods in medical informatics. 2nd Edition. New York: Springer; 2006. [Google Scholar]
  • 20.Bloomrosen M, Starren J, Lorenzi NM, Ash JS, Patel VL, Shortliffe EH. Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting. J Am Med Inform Assoc. 2011;18(1):82–90. doi: 10.1136/jamia.2010.007567. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Han YY, Carcillo JA, Venkataraman ST, Clark RSB, Watson RS, Nguyen TC, Bayir H, Orr RA. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry. Pediatrics. 2005;116:1506–1512. doi: 10.1542/peds.2005-1287. [DOI] [PubMed] [Google Scholar]
  • 22.Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, Strom BL. Role of computerized physician order entry systems in facilitating medication errors. JAMA. 2005;293(10):1197–1203. doi: 10.1001/jama.293.10.1197. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

01

RESOURCES