Increasing value and reducing waste: addressing inaccessible research

Abstract

The study protocol, publications, full study report detailing all analyses, and participant-level dataset constitute the main documentation of methods and results for health research. However, journal publications are available for only half of all studies and are plagued by selective reporting of methods and results. The protocol, full study report, and participant-level dataset are rarely available. The quality of information provided in study protocols and reports is variable and often incomplete. Inaccessibility of full information for the vast majority of studies wastes billions of dollars, introduces bias, and has a detrimental impact on patient care and research. To help improve this situation at a systemic level, three main actions are warranted. Firstly, it is important that academic institutions and funders reward investigators who fully disseminate their research protocols, reports, and participant-level datasets. Secondly, standards for the content of protocols, full study reports, and data sharing practices should be rigorously developed and adopted for all types of health research. Finally, journals, funders, sponsors, research ethics committees, regulators, and legislators should implement and enforce policies supporting study registration and availability of journal publications, full study reports, and participant-level datasets.

“When I had to decide whether to have a second bone-marrow transplant, I found there were four trials that might have answered my questions, but I was forced to make my decision without knowing the results because, although the trials had been completed some time before, they had not been properly published! This should not happen. I believe that research results must be seen as a public good that belongs to the community – especially patients.” Alessandro Liberati ¹

The benefits of health research can only be realised if the study methods and results are fully disseminated in a timely and unbiased manner.² Availability of full information on study methods facilitates critical appraisal, interpretation of study results, and appropriate replication. Proper reporting of results can improve clinical practice and policy, prevent unnecessary duplication, and help to inform ongoing and future research. Availability of participant-level data enables ancillary research and independent re-analysis of study results.

Despite advances in the dissemination of study information, half of health-related studies remain unpublished,³ and few study protocols and participant-level datasets are accessible. Inaccessibility of research is detrimental to patient care and wastes much of the $240 billion in annual worldwide health research expenditure.⁴

This fifth article in the Lancet series documents the extent and impact of non-dissemination and selective reporting of health research, and examines the options for reducing the waste and harms arising from inaccessible study information.

ACCESS TO PRIMARY REPORTS

Non-publication

Journal publication is traditionally the primary means of communicating research results to the scientific community. By failing to contribute to knowledge, unpublished studies represent a complete lack of return on the investment of research resources and the contributions of study participants. For example, among health-related studies funded by the European Union from 1998-2006 at a cost of 6 billion Euros, only half produced detectable publications.⁵ In the case of oseltamivir (Tamiflu), the influenza drug stockpiled by governments for over $3 billion in 2009 alone, unpublished phase 3 clinical trials accounted for 60% of patient data (Table 1), including the largest known trial.

Table 1.

Examples of estimated impact of selective study publication and selective outcome reporting (see Web Appendix 2 for supporting references).

Drug	Type of biased dissemination	Impact
Rosiglitazone	■ Non-publication of unfavourable trials and company meta-analysis. ■ Increased risk of myocardial infarction confirmed by independent meta-analysis of 56 rosiglitazone trials, which included 36 unpublished trials whose data were obtained from the sponsor's trial registry. ■ Concerns over ascertainment of cardiovascular events in the RECORD trial based on review of case report forms, with problematic cases favouring rosiglitazone 80% of the time.	Number needed to harm of 37-52 over 5 years translates into over 6,000-8,000 additional myocardial infarctions among 325,000 patients taking rosiglitazone in US and UK in 2010. About 83,000 excess myocardial infarctions attributable to rosiglitazone from 1999-2006 in US.
Oseltamivir	■ Non-publication of trials containing 60% of patient data. ■ Inacccessible full study reports for 29% of trials. ■ Missing modules for 16 of 17 available full study reports. ■ Discrepancies observed between published articles and full study reports.	Billions of dollars ($3.3 billion in 2009 alone) spent worldwide stockpiling a drug with questionable benefits for reducing hospitalisation and pulmonary complications from pandemic influenza, and with unclear harms.
Gabapentin	■ Non-publication and delayed publication of negative trials for off-label indications. ■ Selective reporting of positive primary outcomes within trial publications for off-label usage, with suppression of negative outcomes.	In 2002, $2.1 billion (94% of total sales) spent in US alone on prescriptions for off-label uses promoted by sponsor despite lack of evidence of efficacy.
TGN1412	■ Non-publication of phase 1 trial in 1994 that showed serious adverse effects from a similar antibody.	Serious adverse effects from TGN1412 – leading to hospitalisation of 6 healthy volunteers – could have been prevented if prior evidence of harms from similar antibodies had been disseminated.
Paroxetine	■ Selective reporting of 4 positive, post hoc outcomes and suppression of 4 negative, protocol-specified outcomes within trial publication involving children with depression. ■ Non-publication of 2 trials and 2 observational extension studies showing increased harms (e.g., suicidal ideation) and lack of efficacy in children. ■ Systematic review revealed that risk:benefit balance no longer favoured the drug when unpublished trials were included.	In 2002, about 900,000 prescriptions (costing $55 million) written for children with mood disorders in US for a drug with potential harms and lack of evidence of efficacy.
Lorcainide and Class I anti-arrhythmic drugs	■ Non-publication of trial conducted in 1980 showing increased mortality with lorcainide (19%, 9/48) versus placebo (2%, 1/47). ■ Mortality risk for this class of drugs remained unknown until subsequent trials with similar findings were published in 1989 and 1992.	20,000-70,000 preventable deaths every year in US alone due to widespread use of harmful anti-arrhythmic drugs in 1980s.
Rofecoxib	■ Non-publication of sponsor's internal meta-analysis showing increased mortality from 2 trials in Alzheimer's disease, with 2-year delay in reporting the results to regulators. ■ Selective exclusion of placebo-controlled trials from 3 published, sponsor-conducted meta-analyses that showed no overall increase in cardiovascular events, in contrast to subsequent independent meta-analysis that included all trials (made available through litigation). ■ Selective omission of cardiovascular harms within arthritis trial publication.	88,000-144,000 additional myocardial infarctions among 107 million prescriptions filled in the US from 1999-2004. About 400,000 users in UK in 2004.
Celecoxib	■ Selective reporting of favourable 6-month harms data within trial publication, with suppression of unfavourable 12 to 15-month data (identified via publicly-accessible regulatory documents) that no longer showed benefit in reducing gastrointestinal ulcers. ■ Discrepant reporting of cardiovascular mortality data between regulatory report and two publications for same trial.	600,000 users in UK and over 14 million prescriptions filled in US in 2004 for an expensive drug with questionable benefit over cheaper alternatives.
Ezetimibe/simvastatin	■ Two-year publication delay for randomised trial showing no benefit of combination drug (ezetimibe/simvastatin) versus simvastatin alone.	Billions of dollars spent globally during publication delay ($2.8 billion in 2007) for costly combination drug with no evidence of benefit over cheaper alternatives.
Vitamin A and albendazole (deworming)	■ Five-year publication delay for clinical trial of 2 million children showing no benefit of Vitamin A and deworming on mortality.	Millions of children dewormed (>300 million in 2009) and given Vitamin A supplementation (77% of preschool children in 103 countries) based on global policies with uncertain benefit.

Overall, only half of completed clinical and preclinical studies are published, and publication rates have not changed substantially over time (Web Appendix 1).³ Non-publication is common among studies approved by research ethics committees (N=15 cohorts, pooled publication rate 45%; 95% CI 40% to 50%), or studies defined by funding sources, trial registries, institutions, and research groups (N=16 cohorts, 54%; 95% CI 44% to 63%). Studies presented as conference abstracts have similarly low publication rates (N=92 cohorts, 46%; 95% CI 43% to 50%).³

Selective publication

As the most important predictor of publication status, studies with positive or statistically significant results are more likely to be published than studies with negative or statistically non-significant results.³ Selective publication has been found in cohorts of studies followed from the time of their inception, abstract presentation, and regulatory submission (Figure 1). This bias exists in both clinical and preclinical research, although selective publication of animal experiments has undergone less evaluation.^3,6,7 Other factors such as industry funding and sample size are not consistently associated with journal publication (Figure 2, Web Appendix 1).

Figure 1.

Publication rates for positive studies versus null/negative studies, by type of study cohort (12 inception cohorts of 2,531 protocols; 4 cohorts of 855 regulatory agency submissions; 27 cohorts of 10,289 conference abstracts; and 4 cohorts of 2,636 manuscripts submitted to journals).^3,115

Figure 2.

Publication rates for random sample of 677 completed trials registered on ClinicalTrials.gov from 2000-2007, by study characteristic. Adapted from Ross J et al.¹¹⁶

When published, clinical trials with positive results appear in journals about one year earlier than non-positive trials.⁸ Negative trial results for a combination cholesterol-lowering drug (ezetimibe/simvastatin) were suppressed by the sponsor for two years, leading to class action lawsuits (Table 1). Publication of the largest randomised trial ever conducted was delayed for five years after not finding any significant effect of Vitamin A or deworming on mortality in 2 million children in India (Table 1) – results that carry substantial global health implications.

Though widely suspected, there is no empirical evidence that journals preferentially publish manuscripts with positive results over those with non-positive results (Figure 1),³ indicating that non-publication of negative studies arises primarily from investigators’ failure to submit. Investigators report that a lack of time, priority, or importance of results are their most common reasons for non-publication– all of which may be related to a lack of statistical significance.³

Consequences of non-publication and selective publication

Overall, the literature represents an incomplete and biased subset of research findings. Selective study publication prevents fully informed decisions about patient care,¹ resource allocation, prioritisation of research questions,⁹ and study design.¹⁰ Our ignorance can lead to the use of ineffective or harmful interventions, and to a waste of limited healthcare resources (Table 1).^11-13 For example, in a meta-analysis that included unpublished trials, reboxetine was more harmful and no more efficacious than placebo for treatment of major depression (Figure 3), in sharp contrast to the published trials.¹⁴

Figure 3.

Results of published versus unpublished randomised trials of reboxetine versus placebo for acute treatment of major depression. Adapted from Eyding D et al.¹⁴

Selective publication of positive preclinical or observational research is a potential explanation for why the published results of only 11%-25% of promising preclinical studies could be independently replicated to drive drug development;^15,16 why clinical trials have often failed to confirm the benefit reported in preceding publications of animal or clinical studies;^17,18 and why many published studies reporting new epidemiological and genetic associations are subsequently refuted.^19,20 Inaccessible research can also facilitate redundant, misguided, or potentially harmful research evaluating similar interventions.

Barriers to accessing published research

Even when published, access to research reports is restricted. Journal subscriptions are costly,²¹ particularly for low income settings but even for leading private academic institutions.^22,23 Although the number of open access articles has been increasing, access to 78% of published medical research remained restricted to journal subscribers in 2009.²⁴

Language barriers represent another obstacle to accessing published research. Most high-profile scientific journals are published in English, but a large body of literature is published in other languages. In China alone, there are over 2,500 biomedical journals; less than 6% of these are indexed in MEDLINE.²⁵ Publications in languages other than English are often excluded from systematic reviews due to limited resources or inaccessibility. There is conflicting evidence on whether the quality and results of research differ systematically between studies published in English versus other languages,^26,27 and recent data are limited. The impact and quality of studies published in languages other than English is likely to be context-dependent,²⁶ and the blanket exclusion of these studies from systematic reviews can lead to substantial waste of research data.

ACCESS TO ALL STUDY METHODS AND RESULTS

Although the publication of all studies has a major role in reducing bias and improving transparency, journal publication alone is insufficient. Evidence of frequently incomplete and selective reporting of methods and results in published articles challenges their traditional role as the sole source of research information.^28,29

Access to key study documents

Produced by industry sponsors, a Clinical Study Report represents the most complete final report of study conduct and results, and contains the study protocol as an appendix.^30,31 While Clinical Study Reports are familiar to those involved in industry-sponsored drug or device trials, we use the general term ‘full study report’ to encompass unabridged final reports for all clinical and preclinical studies (Figure 4).

Figure 4.

Key sources of information on study methods and results, with associated information loss and potential for selective reporting.

The study protocol and full study report provide detailed information that is not available in the published articles.^32,33 Their availability can help to clarify unclear reporting and identify selective reporting within publications, as well as inform clinical practice and future research. For example, published eligibility criteria reported in publications often differ from those listed in the protocol.^34,35 Among trials conducted by two HIV research networks, the published eligibility criteria gave the perception of 40% greater inclusivity compared with the protocol-defined criteria.³⁵ These discrepancies provide journal readers with the misperception of a broader study population with greater generalisability.

Despite their importance, protocols and full study reports have generally not been publicly accessible.^36,38 In a systematic review of oseltamivir, discrepancies between the trial publications and full study reports prompted investigators from the Cochrane Collaboration to question the validity of the published literature. Only a subset of full study reports (with missing modules) could be obtained from the sponsor and European Medicines Agency (Table 1).

Selective reporting within journal publications

Recent discussion of full study reports from drug trials submitted to regulators provides insight into the mechanism of selective outcome reporting,^36,37 which refers to the biased reporting of some results but not others within a published article.³⁹ While the full study report can number in the thousands of pages, this information must be compressed into a few journal pages (Figure 4).

The decisions about what to report or exclude are rarely transparent and often lead to selective outcome reporting in published clinical trials,²⁹ systematic reviews,⁴⁰ and observational research.⁴¹ On average, one third to one half of efficacy outcomes are fully reported in the journal publication of a randomised trial, with statistically significant outcomes being more than twice as likely to be fully reported than non-significant ones.^39,42,43 Selective reporting of outcomes amplifies the bias arising from selective publication of entire studies, and can have a substantial impact on the results of systematic reviews.^13,44

Furthermore, comparisons of protocols and registry records with journal publications have identified discrepancies in the definition of primary outcomes in one to two thirds of randomised trials and systematic reviews.^29,40,45 Similar problems have been found when comparing publications with full study reports.^33,46-48 Frequent discrepancies have also been identified for important aspects of trial methods.^29,49 These changes are not transparently reported in publications, precluding a full understanding of the trial's validity.

Selective outcome reporting can lead to significant patient harm and waste of resources (Table 1). For example, a placebo-controlled trial of paroxetine for depression in adolescents did not find a difference for any of the eight protocol-specified efficacy outcomes, but in the publication, four of these negative outcomes were replaced with four new positive ones.⁵⁰ The highly-cited publication also suppressed serious adverse events associated with paroxetine, yet concluded that the drug was generally well-tolerated and efficacious. Over the subsequent year, almost a million paroxetine prescriptions were written for children with mood disorders in the United States.

Quality of protocols

A lack of transparent description of key methodological elements in protocols impairs critical appraisal,⁵¹ and can raise concerns about the quality of study design, conduct, and reporting.¹⁰ If the analysis plan or primary outcome is not pre-specified, then investigators have free rein to select any result they wish to report. While a lack of pre-specification may be appropriate for exploratory studies, the post hoc nature of such analyses is often not transparently described in publications of clinical trials and systematic reviews.^29,39,40

A substantial proportion of randomised trial protocols fail to adequately address important aspects of study methodology,^29,51 such as the primary outcomes, sample size calculations, allocation concealment mechanism, and blinding procedures. To our knowledge, the quality of study protocols for other types of clinical and preclinical research, as well as the quality of full study reports, have not been examined.

ACCESS TO PARTICIPANT-LEVEL DATA

Beyond the compelling rationale for disseminating publications, protocols, and full study reports, there are also numerous benefits to sharing of participant-level data.

Independent re-analysis of key findings

Errors, selective reporting, and fraud can be identified and deterred when others can verify statistical properties and calculations using participant-level data. A substantial proportion of published studies have statistical errors,^52,53 and willingness to share data has been positively correlated with methodological quality and statistical robustness.⁵⁴

There are notable examples where re-analysis of participant-level data by independent researchers raised serious questions about the validity of high-profile papers.^55,56 Promising results from gene expression microarray studies published by one researcher led to the launch of three clinical trials.⁵⁷ However, independent re-analyses failed to reproduce the published findings and identified multiple concerns that prompted the retraction of at least ten articles.

Testing of secondary hypotheses

Leveraging existing datasets to examine new questions broadens the impact of the original data and saves the costs of unnecessarily compiling new datasets.⁵⁸ For example, re-analysis of data from a radical prostatectomy trial demonstrated substantial heterogeneity of treatment effect.⁵⁹ In another example, re-analysis of data obtained through the US National Institutes of Health Data Sharing Policy found that compared with men, women had significantly higher mortality rates with digoxin.⁶⁰

Increased power and reliability of meta-analysis

Pooled effect estimates can be calculated and more easily interpreted when the outcome definitions from the pooled studies are comparable. For example, it can be difficult to combine trials that report absolute decrease in systolic blood pressure with those reporting the proportion experiencing a certain percentage reduction in blood pressure. Access to participant-level data can harmonise such outcome definitions and yield more powerful meta-analyses.

Promotion of well-annotated datasets

In an empirical study, authors unwilling to share data often stated that doing so would involve too high a workload.⁶¹ This suggests that researchers do not always develop a clean, well-annotated dataset in a format that is easily understood by others. Along with facilitating routine data sharing, proper annotation could help the researchers themselves to easily understand and use their datasets in the future.

Inaccessible data

Despite the benefits, participant-level data from health-related studies are rarely made available to outside researchers.⁶² Although public archiving of microarray datasets has been widely accepted, the data remain unavailable for many gene expression studies.⁶³ Those involving cancer or human participants – arguably among the most valuable for their potential impact on health – were found to be least likely to have archived their data.⁶⁴

Investigators and sponsors too often deny requests for access to data.⁶⁵ In a typical study, data were made available on request for only one of 29 medical research papers.⁶⁶ Even when medical journals mandate data sharing, only 10-27% of authors provided their dataset upon request from external academic researchers.^61,67

Several practical barriers contribute to the widespread lack of data sharing. The current reality is that researchers are usually rewarded for answering their main study questions, but given little credit or funding for data sharing practices that in some instances can incur substantial time, effort, and costs. Guidance is also lacking on the practicalities of preparing datasets for re-use by others.

RECOMMENDATIONS

We propose three main recommendations for improving accessibility to full information from preclinical and clinical studies (Box 1).

Box 1. Summary of key recommendations and proposed measures for monitoring implementation.

Recommendation 1: Institutions and funders should adopt performance metrics that recognise full dissemination of research and re-use of original datasets by external researchers.

Monitoring: Proportion of institutional and funding agency policies that explicitly reward dissemination of study protocols, reports, and participant-level data.

Recommendation 2: Investigators, funders, sponsors, regulators, research ethics committees, and journals should systematically develop and adopt standards for the content of study protocols and full study reports, as well as for data sharing practices.

Monitoring: Adoption rates of international standards by stakeholders.

Recommendation 3: Journals, funders, sponsors, research ethics committees, regulators, and legislators should endorse and enforce the following for all health research:

• ■ Adherence to and expansion of study registration policies to enable tracking of all clinical trials (regardless of intervention or trial type), systematic reviews, observational research, and preclinical experiments;

• ■ Accessible publication of well-conducted research, regardless of the strength and direction of associations observed;

• ■ Public availability of the study protocol, amendments, and full study report, regardless of intervention type, market approval, or journal publication status;

• ■ Sharing of participant-level data.

Monitoring: Proportion of stakeholder policies that endorse dissemination activities; proportion of studies that are registered and published with available protocols, full study reports, and participant-level data.

Recommendation 1

Institutions and funders should adopt performance metrics that recognise full dissemination of research.

Because non-publication often arises from investigators failing to submit manuscripts for publication, incentives are needed to encourage manuscript completion and submission. Rather than focusing on total numbers of publications, reviews of academic performance should explicitly take into account the proportion of a researcher's initiated studies (e.g., those receiving ethics approval or funding) that have led to publications, sharing of protocols, or re-use of data by other researchers. Funding agencies should instruct review panels to strongly consider the applicants’ dissemination output from previously awarded funds. Journals can also encourage manuscript submissions by making an explicit statement that they will publish well-conducted studies regardless of the magnitude or direction of their results, as done by 12% of a sample of 107 medical journals.⁶⁸

To encourage data sharing, academic institutions and funders should make clear that they regard publication of participant-level datasets and their re-use by other researchers as a metric of research impact. The efforts of the original investigators should be acknowledged in publications that arise from secondary analyses, along with citation of the datasets and the original publication. In microarray research, data sharing is associated with increased citation rates.⁶⁹ Some journals now provide the opportunity to publish descriptions of datasets, producing a citable publication.⁷⁰

Recommendation 2

Investigators, funders, sponsors, regulators, research ethics committees, and journals should systematically develop and adopt standards for the content of key study documents and for data sharing practices.

Content of protocols and full study reports

Protocols and full study reports are most useful to researchers and external reviewers when they provide complete details on study methods and results. To address observed deficiencies in protocol content, the recent SPIRIT and upcoming PRISMAP statements define the key elements to address in the protocol of a clinical trial and systematic review, respectively.^51,71 Protocol standards should also be systematically developed for other study designs. High-quality protocols can facilitate transparency, rigorous study implementation, and efficiency of research and external review.⁷²

While protocols are standard for most types of studies,^73-75 full study reports are uncommon outside industry-sponsored trials. We encourage the creation of a full study report that documents all analyses performed and any modification to analysis plans and study conduct (Figure 4). This report can serve as the basis for, and in the case of small studies with few analyses, may be the same document as the manuscript submitted to journals.

For regulated drug trials, the 1995 ICH guidance E3 outlines the key elements of a full study report.³⁰ This guidance, along with other relevant reporting guidelines for primary reports of specific study designs (e.g., CONSORT, STROBE, STARD, PRISMA, ARRIVE),⁷⁶ could serve as the basis for guidelines for full study reports that are applicable to trials of non-drug interventions and to other types of clinical and pre-clinical research.

In order to be widely used by investigators and sponsors, these standards must be enforced by funders as a condition of grant payment; research ethics committees as a condition of ethics approval; and journal editors as a condition of publication.

Best practices for data sharing

Defining best practices will enable researchers and sponsors to better prepare for and participate in data sharing. Consultation with researchers, patients, privacy experts, funders, sponsors, regulators, journal editors, and data curators is needed to establish international standards and processes. An authoritative global body such as the World Health Organization should take the lead in this effort, as it did for trial registration. Multiple scientific, ethical, and technical considerations need to be clarified for implementation of routine data sharing:^77,78

Privacy issues

In the vast majority of cases, patient privacy can be protected by following anonymisation guidelines that are neither technically complex nor time consuming.⁷⁹ For clinical trials, current European legislation already requires that industry sponsors anonymise any patient data contained in the regulatory submission.⁸⁰ In some cases, additional steps beyond anonymisation are needed to prevent the identification of individuals.⁷⁷ The low privacy risk of an anonymised dataset with appropriate safeguards is usually outweighed by the public interest of good research.

Scope

Exactly which participant-level data would be subject to a data sharing policy should be defined. There is a spectrum of granularity, ranging from the original case report forms to a clean dataset that is ready for final analysis (Figure 4). Access to data from case report forms and other source documents in the RECORD trial was essential for fully understanding the cardiac risks of rosiglitazone and identifying inappropriate primary outcome assessment practices that favoured the drug (Table 1).

Method of access

There are several possibilities for how datasets can be accessed, ranging from full publication of anonymised participant-level data for unrestricted use, to restricted access based on some mechanism for evaluating the data request and the new study proposal.⁸¹

Timing of access

Researchers should be given sufficient time to explore the datasets that they developed, but this must be balanced by the public interest of timely access. The defined time period should be as short as possible and may vary by research field. For example, genomic data are usually subject to immediate release, with a period of exclusivity for publication by the original researchers.⁷⁷

Academic input

Datasets are often complex, and a good understanding of the conditions under which the data were obtained and missed can be essential for ensuring appropriate analysis. An investigator from the original research team that produced the dataset could be invited to join the new study, or if independence is preferred, could be offered a commentary on publications that arise from secondary analyses.⁶⁵

Data format and archiving

Formatting standards should be developed to define what constitutes a clean, well-annotated dataset, so that researchers can better prepare their datasets for sharing. There are numerous options for storing participant-level data. Several journals now give authors the option of uploading participant-level data as supplemental material. However, journal staff may have limited expertise in data curation. Approved archives would appear to be a preferable solution, such as those developed for microarray data.⁸² Datasets should be linked to the protocol, full study report, registry record, and journal publication – creating a series of ‘threaded’ electronic documents that form the core components of a study (Figure 4).⁸³

Recommendation 3

Journals, funders, sponsors, research ethics committees, regulators, and legislators should endorse and enforce study registration, availability of full study information, and sharing of participant-level data for all health research.

Identification of unpublished research

Important progress has been made in recent years to improve access to unpublished studies.⁸⁴ Prospective, public registration of all studies at their inception is the key mechanism for tracking existing studies. Since 2005, the International Committee of Medical Journal Editors has required that clinical trials be registered prospectively in an approved registry as a condition of publication.⁸⁵ Subsequent legislation in several countries has solidified the mandate for trials included in submissions to regulators,⁸⁶ and several government funders require registration of trials as a condition of grant approval.^87,88 But many published trials remain unregistered, retrospectively registered, or registered with poor quality information, in violation of the journals’ policies.^45,89-91 It is thus vital that research ethics committees, journals, funders, institutions, governments, regulators, and sponsors adopt and enforce comprehensive registration policies for all trials, including those that fall outside the current adherence mechanisms.

The compelling need to document existing studies is not limited to clinical trials. The registration of systematic reviews,⁷⁴ observational research,⁹² and preclinical experiments^7,18 can be promoted through an expansion of registration requirements. The registry infrastructure already exists to record systematic reviews and observational research.^92,93 Registration of exploratory observational research and preclinical experiments has its challenges,⁹⁴ including potential lack of a formal pre-specified protocol, but a key benefit of registration would be to transparently distinguish hypothesis-generating versus confirmatory studies.

Ultimately, to encompass the greatest breadth of studies, registration requirements need to be firmly enforced by research ethics committees or institutional review boards.^95,96 Since October 2013, the Health Research Authority has required registration of all clinical trials in the UK as a condition of ethics approval.⁹⁷ This important step should be taken in other countries so that the potential risks and costs of research are balanced by its dissemination and contribution to knowledge.⁹⁵ The added workload on overburdened committees could be minimised by automatically withholding final approval for any annual renewals or applications that do not provide a study registration number.

Access to published reports

An increasing number of funding agencies, academic institutions, and legislators have adopted policies to support open access to journal publications, particularly for publicly-funded research.^23,98 For example, grant submissions to the US National Institutes of Health are required to include the PubMed Central open access archive numbers for any papers arising from federally supported research. Public-private partnership programs that provide free access to lower income countries can be helpful if publishers maintain a long-term commitment to participate.²²

To avoid potential waste due to exclusion of studies published in languages other than English, investigators conducting systematic reviews should attempt to identify and screen these studies to determine their number and potential relevance. For fields where a large number of relevant publications are known to exist in languages other than English, international collaboration helps to reduce the cost of translation. Further research is needed to evaluate the relevance of a recent cohort of these studies, weighed against the resources needed to identify and review them.

Access to key study documents

Enforceable solutions are needed to resolve the untenable status quo where certain groups (e.g., regulators, sponsors) have access to complete information while those directly using, evaluating, or paying for the intervention (e.g., patients, clinicians, researchers, policymakers) have access to only a potentially biased subset of information. To address this wasteful imbalance, detailed documents on all studies must be made publicly accessible – including the study protocol with any amendments, and the full study report detailing all analyses and results (Figure 4).

The full protocol is inseparable from the study results, which cannot be properly interpreted without a detailed understanding of the study methods.⁹⁹ Since study registries already record basic protocol information, they have the infrastructure to serve as a logical repository for full protocols and full study reports. Several journals such as Trials and BMJ Open publish study protocols, serving as another important means of public access.

Stakeholders with enforcement capacity, including regulators, legislators, journal editors, and funders, should facilitate access to protocols and full study reports.^36,37,99 The European Medicines Agency recently committed to providing access to full study reports that are routinely submitted for market approval.^37,100 Individual companies have also committed to disclosing, with conditions, full study reports for their published trials.¹⁰¹

Since 2007, US legislation has required the posting of main results on ClinicalTrials.gov for non-exploratory trials of licensed drugs and devices, and similar legislation is being implemented in Europe.⁸⁶ In 2012, additional US legislation was proposed to include early phase 1 trials, trials without a US site, and trials of unapproved drugs or devices.¹⁰² The proposed legislation also calls for availability of the full protocol, and this has become increasingly accepted by some pharmaceutical companies.^103,104 Comprehensive legislation should also be introduced and enforced in other countries.

Since current legislative and regulatory policy efforts are limited to trials of regulated drugs and devices, additional measures by journals and funders are needed to encompass trials of unregulated interventions (e.g., surgery), or other clinical and preclinical study designs. Half of the highest-impact biomedical journals require that authors make the study protocol available upon request,⁶² but the extent of adherence to and enforcement of this policy is unclear. Journals should routinely require submission of the protocol and full study report along with the manuscript, and provide links to them as a web supplement upon publication of the journal report. Peer reviewers and others who appraise studies should also be encouraged to routinely compare journal articles with protocols, full study reports, and study registries in order to identify any unacknowledged discrepancies. Only a third of journal peer reviewers routinely compare trial registry entries with manuscripts.¹⁰⁵

To maximise the return on investment of public funds, funding agencies should promote rigorous reporting practices by adopting policies requiring public posting of the protocol and full study report for all funded studies. For example, the Health Technology Assessment Programme in England requires a detailed full study report to be submitted, peer reviewed, and published in its own journal, with the advantages of having no space restriction and the ability to also publish abbreviated reports in other journals. The HTA Programme withholds 10% of funds until the report has been submitted – leading to a 98% publication rate for research that it has funded.¹⁰⁶ This policy has now been extended to all research funded by the National Institute for Health Research.

Fostering support for data sharing

Data sharing practices differ markedly between and within disciplines. Whereas it is commonly accepted that microarray data should be publicly deposited, clinical trial datasets are rarely available. A survey of trial investigators revealed broad support for mandatory data sharing in principle, but also widespread concerns over sharing in practice.¹⁰⁷ A cultural shift that recognises the benefits and addresses the barriers is needed for data sharing to become a routine part of research practice.

Journals, industry, funders, regulators, and legislators should facilitate and enforce access to participant-level data for all research. Several journals, including Science, Nature, BMJ, and PLoS Medicine, make publication conditional on providing access to participant-level data in an approved database or upon request.^12,62,108 Recent industry efforts have committed to increase the availability of certain study datasets.^109-111

In 2010, a consortium of medical research funders committed to increase the availability of data generated by the research they fund.¹¹² Since 2003, the US National Institutes of Health has required that grant applications requesting more than $500,000 per year submit a plan for data sharing, although the extent of enforcement is unclear. The impact of datasets shared under this policy can be substantial, such as for the Women's Health Initiative.

Funders should mandate that researchers make available participant-level data from prior grants before they are eligible to receive new funds. It is also important that funders allow grant budgets to include sufficient funds to pay personnel for preparation of datasets and associated documentation for sharing. This investment, which in some instances can be substantial in absolute terms, is usually minimal relative to the time and costs needed to collect new data.^77,113

To avoid waste from redundant datasets, funders should also ask grant applicants to explain why new proposed datasets are needed. For example, the UK Economic and Social Research Council will not fund any dataset creation unless applicants confirm that no appropriate dataset is already available for re-use.

In addition, regulatory agencies could require that participant-level data and protocols from drug or device trials be made publicly available once the market authorisation process has ended. The public health benefit of providing access to study data should outweigh any commercial interests.¹¹⁴ Independent review by academic researchers provides regulators with a second set of eyes and has the potential to improve regulatory decision-making.³⁷

If publication, funding, and licensing were contingent on providing access to participant-level data, data sharing would rapidly become a routine part of medical research. Ultimately, legislation with significant penalties for violation is the inevitable option when self-regulation fails.¹² Legislation alone is not sufficient, however, if its scope continues to be limited to clinical trials of regulated drugs and devices, rather than being more broadly applicable.

The overwhelming evidence of substantial waste and harms due to inaccessible research illustrates the need for urgent action. The time has come for all stakeholders to develop and implement policies that reduce waste in health research and promote its unbiased translation to optimal patient care.