Publication rate for funded studies from a major UK health research funder: a cohort study

Abstract

Objectives

This study aimed to investigate what percentage of National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme-funded projects have published their final reports in the programme's journal HTA and to explore reasons for non-publication.

Design

Retrospective cohort study.

Setting

Failure to publish findings from research is a significant area of research waste. It has previously been suggested that potentially over 50% of studies funded are never published.

Participants

All NIHR HTA projects with a planned submission date for their final report for publication in the journal series on or before 9 December 2011 were included.

Primary and secondary outcome measures

The projects were classified according to the type of research, whether they had been published or not; if not yet published, whether they would be published in the future or not. The reasons for non-publication were investigated.

Results

628 projects were included: 582 (92.7%) had published a monograph; 19 (3%) were expected to publish a monograph; 13 (2.1%) were discontinued studies and would not publish; 12 (1.9%) submitted a report which did not lead to a publication as a monograph; and two (0.3%) did not submit a report. Overall, 95.7% of HTA studies either have published or will publish a monograph: 94% for those commissioned in 2002 or before and 98% for those commissioned after 2002. Of the 27 projects for which there will be no report, the majority (21) were commissioned in 2002 or before. Reasons why projects failed to complete included failure to recruit; issues concerning the organisation where the research was taking place; drug licensing issues; staffing issues; and access to data.

Conclusions

The percentage of HTA projects for which a monograph is published is high. The advantages of funding organisations requiring publication in their own journal include avoidance of publication bias and research waste.

Article summary.

Article focus

• It has previously been suggested that potentially over 50% of biomedical studies funded are never published. Currently, the literature on publication rates for funded studies is sparse.

Key messages

• This paper supplies data from a major UK funder of clinical trials (the NIHR HTA Programme) showing that 98% of its funded studies will publish in its own MEDLINE indexed journal.

• Benefits of a journal series run by the funder including high percentages of studies publishing findings, the opportunity for complete reporting and avoidance of publication bias are highlighted.

Strengths and limitations of this study

• We considered a large sample of projects from a major UK research funder, over a period of 18 years.

• Studies from earlier phase trials were not represented in this study.

Introduction

It was stated by Chalmers and Glasziou¹ that worldwide over US$100 billion is invested per year in biomedical research. They went on to describe four stages at which waste of this resource may occur: choosing the wrong questions for research; doing studies that are unnecessary or poorly designed; failure to publish promptly or not at all; and biased or unusable reports of research. This project responds primarily to the third stage of research waste identified, enabling accessible full publication. In their paper, Chalmers and Glasziou¹ suggested that potentially over 50% of clinical trials funded are never published in full. These data were obtained from a Cochrane review² which stated that “Less than half of all studies, and about 60% of randomized or controlled clinical trials, initially presented as summaries or abstracts at professional meetings are subsequently published as peer-reviewed journal articles.”

It is vitally important that studies report in order to provide evidence to clinicians to inform practice, and policy makers to support them in decision-making. There is currently a move towards an open access to the data from publically funded research^{3 4} in order to increase the returns on public investment; to increase transparency; to prevent duplication in research commissioning; to allow public scrutiny of the research process and inform patient and public decision-making; and to make the results of trials available to the public including participants who have given their time to the study for public benefit.

It was also noted by Chalmers and Glasziou¹ that publication bias leads to a systematic under-reporting of studies with disappointing results, and that public access to the full results of all research remains an aspiration. Other investigators have also found lower publication rates for studies with negative results or indefinite conclusions.^{2 5–8} The National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme commissions and funds primary research and evidence synthesis on the effectiveness, costs and broader impact of healthcare treatments and tests for those who plan, provide or receive care in the National Health Service (NHS). It aspires to maximise the return on investment by enabling, where possible, all funded projects to complete and publish and maximising the impact for the money spent.⁹

The HTA Programme publishes a journal (HTA, known colloquially as the monograph series) which is available to all via the web and aims to publish a report for each project funded. The monograph is unique in that each publication focuses exclusively the final report of one study. Not only is publication encouraged but also the agreement for the team to write and submit this final report is written into the contractual arrangement at the time of funding. The report is typically much longer than peer-reviewed journal articles as teams are expected to publish full details of the studies (such as a full description of the intervention)—essentially as an archive of the study (irrespective of whether the results are positive, negative or indefinite), without limits on word count or length, in a high impact factor journal which is publically and freely available. Authors are also encouraged to publish elsewhere to broaden the dissemination of their findings, and there are other processes for the dissemination of Technology Assessment Reports (TARs). This project aims to investigate the performance of the HTA Programme by assessing what percentage of HTA projects are published in the monograph series, and if they are not published what are the reasons?

Methods

For this study, we selected a cohort of HTA projects for which the planned date for submission of their draft final report (DFR) for monograph publication was on or before 9 December 2011. We identified these projects from a proprietary database system used to manage the HTA and other NIHR research programmes.

We excluded from the sample projects for which the reports were supplementing monographs that were already published; projects that were prospectively not considered suitable for the publication of a monograph, for example, working papers for the National Institute for health and Clinical Excellence (NICE) or short briefing papers; and projects for which certain criteria needed to be met before the project commences, for example, projects relating to possible future H₁N₁ pandemics.

To assure data quality, NIHR Evaluation, Trials and Studies Coordinating Centre staff with responsibility for the publication process independently checked the records for studies where there was no publication. Similarly, where data indicated that no DFR had been received, this information was again checked with the team which should have received it.

All projects were categorised as either primary research (typically randomised controlled trials); secondary research (mainly systematic reviews); HTA TARs (which identify, assess and synthesise research evidence from a number of healthcare interventions, providing estimates of relative effectiveness and cost effectiveness of a range of interventions) or NICE TARs (similar to HTA TARs but prepared specifically for NICE). Projects were also categorised as (1) Projects for which a monograph has been published; (2) projects for which the DFR had been received, but as yet there was no published monograph; (3) No DFR received; and (4) project discontinued. The data were further subdivided into those projects where the commissioning process started within the last 10 years (ie, after 2002), and those where it began in 2002 or before.

When projects are published in the monograph series, the DFRs go through an editorial review process which is conducted between the editors, reviewers and authors. Reports are published in the HTA journal series if they are of a sufficiently high scientific quality as assessed by the referees and editors.¹⁰ For projects which had not been published yet, we needed to know whether a report would eventually be produced. Projects in this category were considered by a staff member (LT) with experience in editorial processes, detailed knowledge of the projects concerned and knowledge of editorial decisions. They designated projects as either ‘will publish’ or ‘will not publish’. This judgement was made using the following criteria. If at the end of the editorial process the editorial board of the HTA journal had deemed the report to be of ‘insufficient quality’ to publish, this report was recorded as ‘will not publish’. If a report has been deemed as of sufficient quality to publish this was recorded as ‘will publish’. Data for ‘published’, ‘will publish’ or ‘will not publish’ were originally obtained in July 2012 and updated on 8 March 2013. Each project was counted as one entity and the data were analysed by the calculation of percentages.

For projects which were not expected to be published (or had been discontinued, or where no DFR had been received), we further investigated the reasons by interrogating in-house electronic records and by referring to hard copy project files which contained detailed records of correspondence with the authors, at the time when the report was due. For projects where no DFR had been received, a web search (searching Medline via Ovid, and Google Scholar; using authors’ names and key words) and a search of internal records were conducted to see if the results of the studies had been published elsewhere.

Results

Initial searches identified 642 projects (see figure 1). Of these, one was excluded because it was a supplementary project following a monograph which had already been published; three because they related to potential future H₁N₁ flu pandemics and required particular circumstances to occur before the project would begin; one as the report had been superseded by another; five as they had been included with another report under a different identification number. Four projects were not suitable for publication as monographs as they were very small and not suitable for publication alone; they had been commissioned to report by a different route; or were working papers for NICE. This left a cohort with a final total of 628 projects (201 primary research, 169 systematic reviews, 110 HTA TARs and 148 NICE TARs).

Figure 1.

Flow diagram of projects included in study.

For 31 projects, a DFR had been received, but as yet there was no publication. After consultation with staff expert in this area, it was deemed that 19 of these would eventually be published and 12 would not (see table 1). By March 2013, all of the 19 reports expected to be published were with the publisher and had been assigned dates by which it was anticipated that they would be published.

Table 1.

DFR received, not yet published: ‘will publish’ or ‘will not publish’

Type of research	Will publish	Will not publish	Total number for which DFR received but not yet published
Primary research	9	10	19
Systematic reviews	2	2	4
HTA TARs	4	0	4
NICE TARs	4	0	4
TOTALS	19	12	31

DFR, draft final report; HTA, Health Technology Assessment; NICE, National Institute for health and Clinical Excellence; TAR, Technology Assessment Report.

In total, 582 projects had published a monograph, two studies had no DFR and 13 studies had been discontinued (see table 2). For primary research studies, the reasons for discontinuation were mainly failure to recruit; for example, in one case, it was due to difficulties for the principal investigator (PI) caused by reorganisation within NHS institutions. For the systematic reviews and TARs, the reasons were either to do with drug licensing (NICE often requests TARs anticipating drug licensing, to inform future guidance; if subsequently the drug is not licensed, there is no need for a review, and NICE will cancel its request consequently there will be no publication). Other reasons for discontinuation of the studies were reliance being placed on access to data being allowed by a third party who then would not release the data; issues around key staff leaving; or being unwell. A summary of results is given in table 2.

Table 2.

Numbers of studies published and research type

	PR	SR	HTA TAR	NICE TAR	Totals
Published	172	163	105	142	582 (92.7%)
No DFR received	2	0	0	0	2 (0.3%)
Discontinued studies	8	2	1	2	13 (2.1%)
DFR received—will publish	9	2	4	4	19 (3.0%)
DRF received—will not publish	10	2	0	0	12 (1.9%)
Totals	201	169	110	148	628 (100%)

DFR, draft final report; HTA, Health Technology Assessment; NICE, National Institute for health and Clinical Excellence; TAR, Technology Assessment Report.

It was noticeable from the data that the majority of projects that did not publish were those commissioned early on in the history of the HTA Programme. The data show that the vast majority of projects for which there will be no publication in the HTA monograph series were commissioned in 2002 or before (see figure 2). There is a difference over time, where the percentage of projects that are published rises from 94% to 98% and the numbers of projects not completing or not publishing fall from 6% to 2% after 2002.

Figure 2.

Percentages of projects commissioned either in 2002 or before or after 2002, which do or do not publish in the HTA monograph series.

More than half of the projects which will not be published (6 of the 10 primary research studies and 1 of the 2 systematic reviews), and both projects that did not submit a DFR were commissioned in 1993. This was before the HTA Programme had the current processes and procedures in place which have developed as the programme matured. The results of the investigations, as to why no monograph was to be published for projects for which a DFR had been submitted, are shown in table 3. In the majority of cases (75%), this was because the draft report was of insufficient quality to be published as a monograph. Currently, the HTA Programme operates the editorial processes which work together with authors to bring reports up to a publishable quality. For one project commissioned in 1993, we were unable to locate the paper files and so were unable to determine the reasons for non-publication. Most of the projects (83%) which were not published as monographs were primary research projects.

Table 3.

Reasons why no monograph is to be published for projects for which a DFR had been submitted

	Draft final report (DFR) of insufficient quality	Study was only a pilot and was not therefore published as a monograph	Project commissioned in 1993, no records available	Totals
Primary research	8 (67%)	2 (17%)	0	10 (83%)
Systematic reviews	1 (8%)	0	1 (8%)	2 (17%)
Totals	9 (75%)	2 (17%)	1 (8%)	12

DFR, draft final report.

Considering the two projects where no DFR had been received, searches identified one peer-reviewed paper. Eight of the primary research studies, where the DFRs were received, were of insufficient quality to be published as monographs; five of these projects had also been published elsewhere in peer-reviewed journals. Whether or not a DFR is deemed to be of ‘insufficient quality’ to be published is a judgement made by the editorial board of the HTA journal series. A monograph is expected to cover all the aspects of the study concerned in detail (average word count approx. 50 000 words); in contrast, journal articles are much shorter (approximately 3000 words), less detailed and cover only certain aspects of the study. The judgment concerning whether a pilot study can be published as a ‘stand alone’ monograph or possibly together with another study as a combined monograph is made by the editorial board.

Discussion

Overall, the percentage of projects commissioned by the HTA Programme which are published in its journal is high, for those commissioned in 2002 or before 94% is published; for those commissioned after 2002, the figure rises to 98%. This number is well in excess of the figure of 50% quoted by Chalmers and Glasziou¹ although it must be born in mind that their data related to studies initially presented as summaries or abstracts at professional meetings (which may include pilot or feasibility studies which do not progress to full studies), rather than commissioned projects, and so are likely to overestimate the publication rate. This rate is also higher than some other funders, for the National Institutes of Health in the USA, after a median of 51 months after trial completion, a third of trials remained unpublished.¹¹

The strengths of this study were that it considered a large sample of projects from a major UK research funder, over a period of 18 years, encompassing a variety of research methodologies. Additionally, in depth data were available on most of the projects to enable us to understand the reasons behind the statistical evidence. Weaknesses include primary research projects considered within the cohort all related to a certain stage in health research and had to be within the remit of the HTA Programme which typically funds late-phase clinical trials, investigating the effectiveness and cost effectiveness of a diverse range of health technologies (which may include drugs, devices, physical therapies, talking therapies, public health interventions, surgical procedures, etc). Consequently, the studies from earlier phase trials were not represented in this study. This work does not specifically consider the length of time to publication which is also pertinent¹²; however, this question is being addressed in another study currently underway.

It is highly desirable that projects should publish the final results for the completed studies and the data presented here demonstrate a high level of project completion and publication. This is likely to be attributable to three particular elements of the programme: (1) selection of the right projects at the beginning using a ‘needs led’ process to identify research questions of the most pressing interest to clinicians in the NHS. This also encourages buy-in from investigators and participants who are committed to answering important questions. (2) A robust monitoring process which assists with timely delivery, budgets, etc; and which can anticipate which projects might fail and help to correct problems as they arise. The majority of projects which have not been published or will not be published were commissioned very early in the history of the HTA Programme. Current monitoring processes carefully monitor progress of projects, and action is taken to assist the studies struggling with problems such as recruitment. It is likely that the current processes of the HTA Programme for both commissioning and monitoring have had a positive effect on the monograph publication rate.

Element (3) is the existence of the HTA journal. The high publication rate—a proxy for converting research funding into useful and accessible knowledge—demonstrates the benefits of such a system. The teams are offered not only the opportunity but also the space to publish studies in full; it is part of the contractual arrangement for funding and a proportion of funds is with-held until the report has been received. The journal publishes almost all projects regardless of results, thus minimising publication bias. Authors are also encouraged to publish in other peer-reviewed journals to increase dissemination; however, the shorter length of these articles does not allow for the reporting of the detail presented in the monographs, for example, detailed descriptions of the intervention. Some studies elect to publish interim results in peer-reviewed journals; however, it has been noted that the direction of effects reported in interim analyses and subsequent final analyses can vary,^{8 13} the monograph series publishes final results in full. Teams associated with projects for which no monograph is to be published are strongly encouraged by the HTA Programme to publish in other journals. Of the two projects for which no DFR was submitted, one had published a peer-reviewed paper elsewhere; of the eight primary studies for which no monograph was to be published, five had published peer-reviewed papers elsewhere. This would indicate that potential waste of resource had been minimised as at least some of the findings had been disseminated. The generalisability of these findings would only relate directly to another funding system with an in-house journal, but the general principles of encouraging and facilitating publication would be generalisable to all funders.

Interesting areas for future research could be to compare the findings of this study which has used data from the HTA Programme with data from other funding streams or organisations, both within the UK and internationally. Additionally, an investigation of the dissemination profile of HTA projects in terms of journal publications and publically accessible reports would be informative.

Recommendations for future commissioning would include funders making it a requirement for funded projects to publish reports of final findings, and for the funders to facilitate this process.