Authors report lack of time as main reason for unpublished research presented at biomedical conferences: a systematic review

Abstract

Objective

To systematically review reports that queried abstract authors about reasons for not subsequently publishing abstract results as full length articles.

Study Design and setting

Systematic review of Medline, EMBASE, The Cochrane Library, ISI Web of Science and study bibliographies for empirical studies in which investigators examined subsequent full publication of results presented at a biomedical conference and reasons for non-publication.

Results

The mean full publication rate was 55.9% (95% CI, 54.8% to 56.9%) for 24 of 27 eligible reports providing this information, and 73.0% (95% CI, 71.2% to 74.7%) for 7 reports of abstracts describing clinical trials. 24 studies itemized 1,831 reasons for non-publication, and 6 itemized 428 reasons considered the most important reason. Lack of time was the most frequently reported reason (weighted average = 30.2% (95% CI, 27.9% to 32.4%)) and the most important reason (weighted average = 38.4% (95% CI, 33.7% to 43.2%)). Other commonly stated reasons were lack of time and/or resources, publication not an aim, low priority, incomplete study and trouble with co-authors.

Conclusions

Across medical specialties, the main reasons for not subsequently publishing an abstract in full lies with factors related to the abstract author rather than with journals.

1. Introduction

The number of peer-reviewed full-text scientific articles in biomedical journals has increased exponentially recently [1], as has the number of scientific conferences and correspondingly the number of conference abstracts. Results from abstracts presented at conferences allow for early evidence of completed or ongoing research. However, abstracts are accepted for presentation without peer review or are reviewed less thoroughly than is typical of a journal article. Investigators reviewing the quality of reporting of abstracts show that adherence to CONSORT guidelines for abstracts is poor [2–5]. In addition, the limit in the number of words typically allowed for an abstract means that there may be sparse information about the details of study design or results presented.

Less than half of all studies, and only about 60% of controlled clinical trials, presented as abstracts at biomedical conferences are eventually published in full. Furthermore, studies that have positive results are subsequently published as full-length journal articles more often than studies with negative results [6]. The consequence is that the information presented in an abstract is not generally available to the scientific community. Because conference abstracts are an intermediate step in sharing study results, they may contain preliminary results, report results following intermediate follow-up, or report unusual or unexpected findings. This information is frequently available only to those persons who attended the conference or who have access to the conference abstracts.

Because of this risk of dissemination bias (publication based on the direction or significance of study results), it is important to understand why authors do not publish their results in full. It has been suggested that conference abstract authors do not prepare and submit full length papers because they believe that journals will reject manuscripts with negative or null findings. To examine this assumption, we aimed to review systematically cohort studies that follow-up abstracts and which assessed authors reasons for non-publication, and determine the number and types of reasons provided by abstract authors for not publishing their abstract results in full.

2. Methods

We included all reports that examined the subsequent proportion of full publication of results at least 2 years after presentation as an abstract at a biomedical conference. In addition, the report authors must have contacted abstract authors of studies that were not published, and asked authors of unpublished studies to provide reasons why they did not subsequently publish the study presented in the abstract as a full length journal article.

We searched electronically for reports in MEDLINE, EMBASE, all databases within The Cochrane Library, and ISI Web of Science in November 2012. The search strategy (available online) was developed by the investigators of the OPEN (To Overcome failure to Publish nEgative fiNdings; www.open-project.eu) project for their systematic review) [7, 8], which aims to determine the proportion and/or rate of non-publication of studies by following cohort of studies that were presented as abstracts at conferences. There was no restriction by date or language. Citations identified by the search strategy were independently screened for the OPEN project by at least two authors for possibly eligible reports. References cited in eligible studies were also searched. Two authors then reviewed all reports classified by the OPEN investigators as eligible or possibly eligible for the OPEN review for inclusion in this systematic review.

Information extracted included total number of abstracts followed, whether the abstracts described randomized or controlled clinical trials (yes/no), proportion of abstracts subsequently published as a full journal article, methods used to contact authors (letter, email, telephone), and methods used to query authors (open or close-ended). We extracted the number of authors contacted, number of authors who responded, total number of reasons for not publishing abstract results in full, and if reported, the most important reason. Information was independently extracted by two reviewers and discrepancies resolved by discussion between the two reviewers until consensus was achieved.

Reasons were recorded verbatim and grouped into common categories. For example, the following reasons were coded as “lack of time” (“not enough time”, “lack of time”, “no time to prepare paper for publication”, “insufficient time to conform to editorial policies”, “no time”, “main problem identified…was time”, “time restriction”, and “too little time”). For non-English papers, we translated reason into English before coding (verbatim responses and codes are available online).

Data were entered into Microsoft Excel 2011 (Microsoft Inc., US) for all analyses. We assessed proportions of categorical variables, and calculated means for continuous data. To calculate the proportion of each type of reason, we calculated a weighted average of the proportion of type of reason, weighted by total number of responses in each report. We calculated the weighted average and 95% confidence intervals (95% CI) of each type of reason reported for all included reports and for two report subgroups: those where investigators asked for the most important reason, and those where investigators followed abstracts that presented clinical trials.

3. Results

3.1

The electronic and reference searches yielded 3,955 unique citations; 367 were obtained for full text evaluation, and 29 citations comprising 27 reports fulfilled our eligibility criteria (see PRISMA flowchart, Figure 1). The characteristics of included reports [3, 9–36] are presented in Table 1. Most abstract authors investigated all abstracts presented at a conference (n = 13); but 3 authors selected a random sample of all types of abstracts and the remainder (n = 11) selected abstracts, usually by study design, to investigate. Among the reports not selected by study design, 8 investigators reported that the abstracts described clinical and basic science studies, or only clinical studies; while 8 did not report the types of studies described in the conference abstracts. The median number of abstracts followed in the various reports was 313 (IQR: 102 to 492). We could not calculate the proportion of abstracts published as a full length journal article for 3 reports either because the proportion of abstracts subsequently published was not reported [35] or quantitative information was unavailable [14, 36]. For the remaining 24 reports the weighted mean proportion of abstracts published was 55.9% (95% CI, 54.8% to 56.9%). The proportion of abstracts published for the 7 reports that only included abstracts of clinical trials [3, 11, 18, 21, 23, 28, 32] was 73.0% (95%CI, 71.2% to 74.7%).

Figure 1.

PRISMA Flow Chart

Table 1.

Characteristics of reports investigating reasons for non-publication

Report	Published/Total Abstracts (%)	Type sample	Type studies described in abstracts	Conference Focus	Country of corresponding author
Bhandari 2002 [9,10]	231/465 (49.7)	All abstracts	Clinical and basic science	Orthopedic Surgery	Canada
Camacho 2005 [11]	185/275 (67.3)	All abstracts describing Phase I trials	Phase I trials	Oncology	USA
De Bellefeuille 1992 [12]	63/81 (77.8)	Random sample of all abstracts	Not reported	Oncology	Canada
Dirk 1996 [13]	80/147 (54.4)	All abstracts by faculty of a single department	Not reported	Anesthesiology	USA
Drury 2012 [15]	606/909 (66.7)	All abstracts	Not reported	Cardiovascular Surgery	UK
Dyson 2006 [14]	208/283 (73.5)	All abstracts	Not reported	Veterinary anesthesiology	Canada
Harvey 2010 [16]	122/442 (27.6)	All abstracts	Not reported	Medical library science	USA
Hashkes 2003 [17]	134/331 (40.5)	All abstracts	Clinical and basic science	Rheumatology	USA
Hoeg 2009 [18]	361/559 (64.6)	All abstracts desriibing Phase II Trials	Phase II Trials	Oncology	USA
Hopewell 2001 [19]	39/91 (42.8)	All abstracts	Methods studies	Methodology	UK
Jourbert 2004 [20]	NR/102	Abstracts classified as dompetition winners	Clinical and basic science	General Medicine	South Africa
Kottachchi 2010 [3]	64/82 (78.0)	All abstracts describing RCTs	RCTs	Gastroenterology	Canada
Klassen 2002 [21,22]	248/447 (55.4)	All abstracts describing RCTs	RCTS	Pediatrics/Child Health	Canada
Krzyanowska 2003 [23]	415/510 (81.4)	All abstractrs describing RCTs	RCTS	Oncology	Canada
Montane 2007 [24]	71/248 (28.6)	All abstracts	Clinical	Pharmacology	Spain
Nader 2009 [25]	34/125 (27.2)	Random sample of all abstracts	Clinical	Geographic Medicine	USA
Oliveira 2009 [26]	122/313 (39.0)	All abstracts	Clinical and basic science	Urology	Brazil
Sanossian 2006 [27]	220/353 (62.3)	All abstracts	Clinical	Neurology	USA
Scherer 1994 [28]	61/93 (65.6)	All abstracts describing RCTs	RCTs	Vision	USA
Seaton 1983 [29]	202/625 (32.3)	All abstracts	Not reported	Communication disorders	USA
Snedecker 2010 [30]	70/149 (47.0)	All abstracts describing trials	Trials	Food safety research	Canada
Stöcker 2009 [31]	200/368 (54.3)	All abstracts	Not reported	General Practice	Germany
Tam 2011 [32]	643/709 (90.7)	All abstracts describing RCTs	RCTS	Oncology	Canada
Timmer 2002 [33]	392/863 (45.4)	Random sample of all abstracts	Clinical and basic science	Gastroenterology	Canada
Vuckovic 2001 [34]	42/63 (66.7)	All	Not reported	Oncology	Serbia
Weber 1998 [35]	NR/492	All	Clinical and basic science	Emergency Medicine	USA
Wong 2010 [36]	NR/8	Identified for a systematic review	Clinical	Urology	UK

3.2. Characteristics of Reports

Nineteen investigators described abstracts presented at a single conference with a clear medical specialties [3, 9, 11, 12, 15, 17, 18, 21, 23, 24, 26–29, 32–36]. Five reports covered the American Society of Clinical Oncology [11, 12, 18, 23, 32] and 2 covered Digestive Diseases Week [3, 33]. Three reports covered conferences that are ancillary to medical specialties including geographic medicine [25], medical library science [16], and methodology [19]; 2 were from conferences associated with animal science areas [14, 30]. Corresponding authors from 4 reports were from non-English speaking countries [24, 26, 31, 34].

The main reason why most investigators contacted abstract authors was to determine whether an abstract had been published, either all abstracts (n = 13) or only those not found by an electronic search (n= 10); investigators of 4 reports specifically contacted authors to determine the reason for non-publication. When stated (n = 14), investigators sent a simple close-ended questionnaire (n = 4), a close-ended questionnaire with free text options (n = 6), or an open-end questionnaire (n = 4) to abstract authors. The questionnaire was sent to a single author (n = 7), multiple authors (n = 15), or not reported (n = 5). Investigators approached authors via email (n = 15), postal mail (n = 13), telephone (n = 5), fax (n = 2), or by a face-to-face interview (n =1); 18 investigators used a single method to contact authors, 7 used 2 methods, and 1 used 3 methods. The median response rate of authors to these questionnaires was 65.5% (IQR: 49.7% to 80.2%).

Of 24 studies that itemized reported reasons (the remaining 3 reports only presented qualitative information), 6 collected information on the most important reason ([16, 27, 28, 31, 33, 34] with 2 investigators collecting information both on all reasons and on the most important reason for failure to publish a study as a full journal article [16, 33]. Seven investigators collected information from authors of clinical trial abstracts, including phase I trials [11], phase II trials [18] or randomized clinical trials [3, 21–23, 28, 32].

3.3. Reasons for non-publication

Lack of time comprised about a third of all reasons from reports that had included this as a reason, whether as the proportion of all reasons (30.2% (95% CI, 27.9% to 32.4%)) from all reports; the single most important reason (38.4% (95% CI, 33.7 to 43.2)) for the subset of reports that included this item; or the reason reported by authors of clinical trials (31.9% (95% CI, 34.5 to 59.5)) for the subset of reports that investigated clinical trials. Lack of time and/or resources was also a commonly stated reason for all reports 22.2 (95% CI, 17.2 to 27.3)) and clinical trials (47.0 (95% CI, 34.5 to 59.5)), but was not included as one of the most important reasons. Together these two reasons made up about half of all reported reasons. All reported reasons for non-publication are shown in Figures 2 to 4. Details are available online.

Figure 2.

Weighted average of reasons for not fully publishing the results reported in a conference abstract as recorded in 24 reports. The total number of reasons reported in all reports was 2,307. Each bar represents the weighted average with 95% confidence intervals of a specific type of reason. The number of reports that included a specific reason is noted within parenthesis.

Figure 4.

Weighted average of the reason for non-publication as noted in 7 reports investigating reasons for not publishing the results of clinical trials. The total number of reasons reported in all reports was 322. Each bar represents the weighted average with 95% confidence intervals of a specific type of reason with number of reports including that reason. The number of reports that included a specific reason is noted within parenthesis. Not included in the graph was loss of support which was reported as 2 of 3 reasons from a single report.

Other reasons frequently stated were trouble with co-authors, publication not an aim, low priority, and incomplete study. Of note, reasons having to do with journal editors were infrequently mentioned, with expect journal rejection being mentioned by abstract authors only 11.2 % (95% CI, 9.3 to 13.2) of the time and as the most important reason 3.2% (95% CI. 0.8 to 6.6) of the time. Among reports following clinical trials, it was mentioned only once in a single report as a reason for non-publication. Perhaps the most disturbing reasons for not fully publishing results presented in abstracts as a journal article are negative results or publication not permitted by sponsor. This latter reason was reported as a reason by 16 abstract authors in 4 reports [11, 13, 18, 36] and comprised a weighted average of 7.5 % (95% CI, 3.6 to 11.3) of all reasons reported for non-publication.

4.0 Discussion

This systematic review showed that across biomedical scientific areas the responsibility for not publishing abstract results as full journal articles remains primarily with abstract authors and not journals or journal editors. The expectation of journal rejection was less frequently stated as a reason for non-publication than reasons related to inadequate time, inadequate resources, trouble with coauthors or other factors related to the authors perception of the importance of study results. Lack of time or lack of time and/or resources were clearly the most frequently stated reasons, whether across all studies, as the most important reason, or among reports examining abstracts that described clinical trials. Lack of time is somewhat difficult to interpret, however, in that it could represent many things, or it may be that clinical, academic or other responsibilities intrude on time commitments. Lack of time could also be a euphemism for low priority, lack of interest in the current project, or work on other projects. The concept of an uninteresting study or lack of time has also been cited as a reason for not publishing initiated studies [37, 38]. Studies that have compared the acceptance rate of positive to negative or null studies find little or no difference in the acceptance rate by journals even though the proportion of positive to negative studies submitted is skewed, implying that negative studies are often not being submitted by authors [39, 40].

We listed lack of time and lack of time and/or resources separately reason for two reasons. First, some investigators collected both reasons and we could not determine when both reasons had been reported by the same abstract author and combining the two categories could result in double-counting. Second, separating these as two reasons highlights the role of funding in the publication process. Usually the scientific process begins with development of a proposal and proceeds to grant writing. If funded, the investigator then develops and conducts the research and often presents the results at a conference following submission of an abstract. By then, however, the next proposal and grant are due, with little time for full text writing. The process of writing up the results in full, especially when the results are null or negative, then becomes less of a priority [41]. We recommend that, when possible, funding agencies provide adequate funds to allow time for dissemination of study results as peer reviewed journal articles.

Another reason cited frequently was trouble with co-authors, either re-location or shifting responsibilities for writing the final manuscript. The existence of a dissemination plan before a study begins may ameliorate problems related to relocation or disagreements between authors.

On the other hand, it is unrealistic to expect that all studies reported at conferences should be published as full journal articles. Authors noted limitations in methods, problems with equipment, recruitment, software, or ethical problems as contributing to the lack of publication. In addition, authors sometimes presented preliminary results or otherwise incomplete study data and had not written up the results in full at the time of the investigator query. It would be interesting to note when, or if, authors presenting incomplete study data transitioned to full journal publication. In addition, some authors had no reason for not publishing study results as full journal articles or stated that publication was not an aim. Presentation of a study at a meeting may have been an end in itself. Sometimes, especially with funding constraints, presentation of a paper at a meeting is required to allow attendance at a meeting, so that there is no impetus to subsequently publish study results in full. Although not noted frequently, there were some instances of constraints on publication of study results in that the sponsor refused to allow subsequent publication. This constraint represents an ethical problem; study participants take part in research studies to “further science”. When the research results are not published, this implicit contract is violated because only the sponsor has access to the data.

An important issue is the availability of conference abstracts for the general scientific community. Only about half of abstracts are subsequently published in full and only about 60 to 70% of clinical trials, with positive results associated with full text publication [6]. Searching for abstracts is time-consuming and it is a matter of debate whether results presented in abstracts should be included in systematic reviews [42]. Typically important aspects of the study design are not reported or not reported in enough detail in an abstract to include the results in a systematic review [43, 44]. However, notwithstanding the poor reporting and the paucity of information, the results from abstracts should be considered as part of the evidence base, even if not published as full journal articles, to inform systematic reviews, avoid duplication of research, and improve future studies.

One possible solution to the problem of data not being available outside of the conference is open access of study results [45]. The National Institutes of Health, Medical Research Council, the Bill and Melinda Gates Foundation, and Wellcome Trust already have a policy of open access for journal articles. Clinical trials results should be posted in ClinicalTrials.gov, if registered, or can be made available via other open access sources. Increased accessibility of abstracts in an electronic database may also be useful, as would including a citation in the relevant trials register record to any abstract describing a clinical trial.

There are some limitations of our study. First, our search strategy was not designed specifically for our systematic review; rather, it was designed to identify reports that evaluated the proportion of studies presented as abstracts and subsequently published as journal articles. We identified reports that additionally provided reasons for non-publication from among the reports that were classified as eligible or possibly eligible for the original systematic review, including at least one report that was not eligible for the original systematic review [35]. Another limitation is the heterogeneity across reports: investigators approached abstract authors differently or requested reasons in different ways. The list of reasons for non-publication that investigators included in their questionnaires was not the same across all reports, making comparisons across studies hard to interpret when only one or a few reports included a specific reason. Our sample may also be biased in that not all investigators who followed up the publication of abstract results gathered information on reasons for non-publication. In addition, not all authors of abstracts responded to the investigator queries. Despite this heterogeneity, reports provided sufficient and consistent enough information to allow us to have some confidence in our finding that the main reason for authors not submitting abstract results for full publication was the authors perception of lack of time or lack of time and/or resources and not related to journals or journal editors. Our study is also limited by the information provided by the investigators about the abstracts. For example, there was no information about the academic status or gender of the respondents, whether the abstracts were presented orally or as posters, or the study design described in the abstracts in many reports. The lack of representativeness of the reports in our review is another limitation: the majority of the included reports are based on US and European conferences, leading to a possible bias in our results. For example, lack of fluency in English was a noted as a reason for non-publication from an international conference [25], but not for any of the reports where conferences were held in Western countries. This reason may be a more prevalent reason for international conferences, or conferences held in LMIC or non-English speaking countries, but we would not have been able to identify it as such given our sample. In addition, although many clinical specialties are represented in our study, it is not clear if the summary results are relevant for each individual specialty.

In conclusion, we found that the author and not the editorial process was the main barrier to subsequent full journal publication of results presented in abstracts at conferences. To address this problem, authors and funding agencies should negotiate some protected time within the grant funding period to allow publication of the majority of the abstracts who do not publish results presented in full journal articles. If publication as a full journal article is not possible, then authors should deposit study results in an open access format to ensure that study results are available to all with the aim to increase transparency in biomedical research and reduce the impact of dissemination bias.

Figure 3.

Weighted average of the most important reason for not fully publishing the results reported in an abstract as recorded in 6 reports. The total number of reasons reported in all reports was 428. Each bar represents the weighted average with 95% confidence intervals of a specific type of reason. The number of reports that included a specific reason is noted within parenthesis.