Does the medical literature remain inadequately described despite having reporting guidelines for 21 years? – A systematic review of reviews: an update

Abstract

Purpose

Reporting guidelines (eg, Consolidated Standards of Reporting Trials [CONSORT] statement) are intended to improve reporting standards and enhance the transparency and reproducibility of research findings. Despite accessibility of such guidelines, researchers are not required to adhere to them. Our goal was to determine the current status of reporting quality in the medical literature and examine whether adherence of reporting guidelines has improved since the inception of reporting guidelines.

Materials and methods

Eight reporting guidelines, such as CONSORT, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), STrengthening the Reporting of OBservational studies in Epidemiology (STROBE), Quality of Reporting of Meta-analysis (QUOROM), STAndards for Reporting of Diagnostic accuracy (STARD), Animal Research: Reporting In Vivo Experiments (ARRIVE), Consolidated Health Economic Evaluation Reporting Standards (CHEERS), and Meta-analysis of Observational Studies in Epidemiology (MOOSE) were examined. Our inclusion criteria included reviews published between January 1996 to September 2016 which investigated the adherence to reporting guidelines in the literature that addressed clinical trials, systematic reviews, observational studies, meta-analysis, diagnostic accuracy, economic evaluations, and preclinical animal studies that were in English. All reviews were found on Web of Science, Excerpta Medical Database (EMBASE), MEDLINE, and Cumulative Index to Nursing and Allied Health Literature (CINAHL).

Results

Among the general searching of 26,819 studies by using the designed searching method, 124 studies were included post screening. We found that 87.9% of the included studies reported suboptimal adherence to reporting guidelines. Factors associated with poor adherence included non-pharmacological interventions, year of publication, and trials concluding with significant results. Improved adherence was associated with better study designs such as allocation concealment, random sequence, large sample sizes, adequately powered studies, multiple authorships, and being published in journals endorsing guidelines.

Conclusion

We conclude that the level of adherence to reporting guidelines remains suboptimal. Endorsement of reporting guidelines by journals is important and recommended.

Introduction

Medical science is an evolving and dynamic field of research that impacts health care, disease outcomes, and health care systems in general. The evidence generated from millions of medical publications is meant to inform these dynamic changes and therefore has to be presented in a clear, consistent, and transparent fashion. There are more than 26 million citations for biomedical literature in the PubMed¹ database alone. To understand and evaluate the evidence presented in these citations, a harmonized method of reporting the research findings is needed to ensure clarity, consistency, and the uptake and dissemination of knowledge.² Tremendous efforts have been made to provide guidelines for different types of research designs to assist in the process of transparent and clear reporting, eg, Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network website.³ However, despite the wide availability of such guidelines since the inception of the Consolidated Standards of Reporting Trials (CONSORT⁴) statement in 1996, the uptake remains suboptimal in the face of the exponential volume of medical literature leaving the readers confused. For example, some studies show positive harmful results from eating red meat on the risk of having colorectal cancer,⁵ while others are showing inconsistent effect marked by substantial methodological differences, type of red meat investigated, and the population selection limitations.⁶ Therefore, the reader is unable to decide whether red meat has an effect on bowel cancer risk. Poor reporting without using well-designed guidelines in primary studies may lead to a bias in the treatment effects found in systematic reviews. In addition, poorly conducted systematic reviews may not be able to detect the bias effect that the studies included. In a previous study, we conducted a scoping review and examined the level of adherence to six reporting guidelines and found the level of adherence to be suboptimal in 86% of the included studies.⁷

The aim of this review was to conduct a systematic review of reviews to update the state of adherence to guidelines since 2012 and to identify factors associated with improved adherence. Our hypothesis was that the reporting standards have improved since our last examination in 2012 given that a longer period has passed after guideline statements were first introduced for researchers and more journals started to endorse the guidelines. Our search was looking at reviews published between January 1, 1996, and September 30, 2016.

Materials and methods

This systematic review was performed and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.⁸ A protocol for a series of three reviews including the current systematic review has been peer reviewed and published elsewhere.⁹

Study inclusion and exclusion criteria

Systematic reviews which investigated the adherence to commonly used reporting guidelines in medical literature that addressed clinical trials, systematic reviews, observational studies, meta-analysis, diagnostic accuracy, economic evaluations, and preclinical animal studies that have been reported in English were selected. Eight guidelines included in this review were as follows: CONSORT,⁴ PRISMA,⁸ STrengthening the Reporting of OBservational studies in Epidemiology (STROBE),¹⁰ Quality of Reporting of Meta-analysis (QUO-ROM),¹¹ STAndards for Reporting of Diagnostic accuracy (STARD),¹² Animal Research: Reporting In Vivo Experiments (ARRIVE),¹³ Consolidated Health Economic Evaluation Reporting Standards (CHEERS),¹⁴ and Meta-analysis of Observational Studies in Epidemiology (MOOSE).¹⁵

The exclusion criteria included studies that 1) were not systematic reviews; 2) did not explore adherence to the aforementioned reporting guidelines; 3) did not provide data on guideline adherence; 4) were subsets of the included studies; 5) published abstracts, letters, editorials, or commentaries; and 6) reviews in languages other than English for feasibility and resource purposes.

Search strategy

The search strategy was based on the previously published review⁷ and was updated for this systematic review. We searched four databases (Excerpta Medical Database [EMBASE], MEDLINE, Cumulative Index to Nursing, and Allied Health Literature [CINAHL], and Web of Science) from 1996 (CONSORT inception – first created guideline among all eight included guidelines) to September 30, 2016.

We used the following search terms for each of the four databases: (Systematic reviews OR reviews OR quality of reporting OR completeness of reporting) AND (CONSORT OR STROBE OR QUOROM OR PRISMA OR MOOSE OR STARD OR ARRIVE OR CHEERS) OR adherence. Detailed search terms have been reported in the published protocol.⁹ All stages of search, inclusion, exclusion, and data abstraction were performed independently in duplicate, and agreement was reached through team discussion and consensus.

Outcome measures

The primary outcome was the level of adherence to reporting guidelines and their checklists as reported in the systematic reviews. The secondary outcome included the factors that were associated with improved adherence to guidelines.

Data extraction

A specific data abstraction form was designed to include the following data: 1) general characteristics of the included studies (first author, publication year, country, journal, study field, search time frame, data sources, numbers of included primary studies, and study design), 2) main findings from the included studies, 3) authors’ summaries and conclusions, and 4) factors reported to be related to improved guideline reporting adherence. Each assessment of the systematic reviews was conducted in duplicate. Calibration was performed on the data extraction form. If the pair of evaluators was unable to come to a conclusion, a third-party reviewer would have settled the dispute.

Quality evaluation

We used the modified Assessing the Methodological Quality of Systematic Reviews/Overview of Quality Assessment Questionnaire (Assessment of Multiple Systematic Reviews [AMSTAR]/Overview Quality Assessment Questionnaire [OQAQ]), a 10-item scale,⁷ to assess the quality of the systematic reviews included in this review. We assigned a number out of a maximum of 20 points for each included study. The higher the number assigned, the better the quality of the systematic review.

Data synthesis

We provided a qualitative summary and characteristics of the included studies. We summarized the factors associated with adherence based on the included study results; no quantitative analysis was possible in this review. We also reported the percentage of studies in which the level of adherence to reporting each guideline was suboptimal. This was calculated by dividing the number of studies with this finding by the total number of studies evaluating the guideline.

Results

Our search resulted in a total of 9,123 publications, of which 124 systematic reviews that included 26,819 primary studies were included in this systematic review of reviews. Figure 1 shows the PRISMA flowchart for the included studies.

Figure 1.

PRISMA flow diagram.

Abbreviation: PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

The characteristics of the included studies are described in Table 1. The majority of the studies (65% of the total 124 included studies) investigated the adherence to the CONSORT statement as expected since it is the first and oldest guideline. The second most commonly investigated guideline is the PRISMA with 19 studies (15%; Table 1).

Table 1.

Characteristics of the included studies

Study	Year	Journal	Country	Statement assessed	Number of studies
Adie28	2013	Annals of Surgery	Australia	CONSORT	150
Adie et al29	2015	Annals of Surgery	Australia	PRISMA	150
Agha et al30	2015	Annals of Plastic Surgery	UK	STROBE	94
Agha et al31	2016	International Journal of Surgery	UK	CONSORT PRISMA STROBE	193
Aguiar et al32	2014	Annals of Pharmacotherapy	Brazil	PRISMA	7
Aguiar et al33	2016	Journal of Clinical Pharmacy and Therapeutics	Brazil	CHEERS	8
Al Faleh and Al-Omran34	2009	BMC Pediatrics	Saudi Arabia	QUOROM	61
Al-Namankany et al35	2009	International Journal of Pediatric Dentistry	UK	CONSORT	173
Alvarez et al36	2009	British Journal of Dermatology	France	CONSORT	98
Anttila et al37	2006	Pediatrics	Finland	CONSORT	15
Areia et al38	2010	Endoscopy	Portugal	CONSORT	120
Augestad et al39	2012	Journal of the American Medical informatics Association	Norway	CONSORT	32
Balasubramanian et al40	2006	Annals of Surgery	UK	CONSORT	69
Bath and Bath41	2000	Stroke	UK	CONSORT	114
Bereza et al42	2008	Annals of Pharmacotherapy	Canada	QUOROM	16
Bian et al43	2006	Journal of Chinese Integrative Medicine	People’s Republic of China	CONSORT	66
Biondi-Zoccai et al44	2006	BMJ	Italy	QUOROM	10
Borg Debano et al45	2012	BMC Anesthesiology	Canada	CONSORT	23
Bousquet et al46	2011	Journal of Allergy and Clinical Immunology	France	CONSORT	94
Bramhall et al47	2015	Inflammatory Bowel Diseases	UK	ARRIVE	58
Cairo et al48	2012	Journal of Clinical Periodontology	Spain	CONSORT	276
Capili et al49	2010	Clinical Journal of Pain	USA	CONSORT	10
Cavadas et al50	2011	International Urogyn J	Portugal	CONSORT	41
Choi et al51	2014	Trials	South Korea	CONSORT	29
Chowers et al52	2009	Journal of Antimicrobial Chemotherapy	Israel	CONSORT	49
Cook et al53	2011	Medical Education	USA	STROBE	130
Daitch et al54	2016	Journal of Pediatric Gastroenterology and Nutrition	Israel	CONSORT	51
Dasi et al55	2012	Journal of Clinical Pharmacology	Spain	CONSORT	40
Delaney et al56	2010	Transfusion	USA	STROBE, CONSORT	47
DeMauro et al57	2011	Pediatrics	USA	CONSORT	179
de Vries and van Roon58	2010	Archives of Diseases in Childhood	The Netherlands	CONSORT	107
Dias et al59	2006	Human Reproduction	UK	CONSORT	164
Ethgen et al60	2009	BMC Medical Research Methodology	France	CONSORT	132
Eyawo et al61	2008	Trials	Canada	CONSORT	47
Fan et al62	2014	PLoS One	China	CONSORT	21
Farrokhyar et al63	2007	Canadian Journal of Surgery	Canada	CONSORT	50
Fidalgo et al64	2015	Ophthalmic and Physiological Optics	UK	STARD	58
Fleming et al65	2013	Angle Orthodontist	UK	PRISMA	109
Fontela et al66	2009	PLoS One	Canada	STARD	90
Freeman et al67	2009	European Journal of Obstetrics & Gynecology and Reproductive Biology	UK	STARD	27
Froud et al68	2012	Community Dentistry and Oral Epidemiology	UK	CONSORT	23
Fung et al69	2009	Ophthalmology	USA	CONSORT, STROBE	36
Gagnier et al70	2006	American Journal of Medicine	Canada	CONSORT	206
Gao et al71	2015	Trials	China	CONSORT	98
Gianola et al72	2013	Physical Therapy	Italy	PRISMA	88
Gohari et al73	2016	Journal of Diabetes and Metabolic Disorders	Iran	CONSORT	185
Gulin et al74	2015	PLoS Neglected Tropical Diseases	Argentina	ARRIVE	83
Halpern et al75	2004	International Journal of Obstetric Anesthesia	Canada	CONSORT	99
Hemels et al76	2004	Current Medical Research and Opinion	France	QUOROM	32
Herdan et al77	2011	Gynecological Surgery	Germany	CONSORT	37
Huang et al78	2015	Expert Review of Anticancer Therapy	China	CONSORT	40
Hui et al79	2012	Support Care Cancer	USA	CONSORT	44
Junhua et al80	2007	The Journal of Complementary and Alternative Medicine	China	QUOROM	107
Karpouzis and Bonello81	2016	Chiropractic and Manual Therapies	Australia	CONSORT	35
Kiehna et al82	2010	Journal of Neurosurgery	USA	CONSORT	27
Kim et al83	2014	BMJ Open	South Korea	CONSORT	146
Kober et al84	2006	Journal of the National Cancer Institute	Australia	CONSORT	142
Ladd et al85	2010	Addictive Behaviors	USA	CONSORT	127
Lee et al86	2013	Trauma Acute Care Surgery	UK	CONSORT	83
Lee et al87	2016	JAMA Facial Plastic Surgery	UK	PRISMA	79
Li et al88	2011	Evidence-based Complementary and Alternative Medicine	USA	CONSORT	42
Li et al89	2014	Systematic Reviews	China	PRISMA	487
Li et al90	2014	BMC Complementary and Alternative Medicine	China	CONSORT	6994
Liu et al91	2015	PLoS One	China	PRISMA	72
Liu et al92	2013	Transplant International	UK	CONSORT	290
Liu et al93	2015	Journal of Evidence-based Medicine	China	CONSORT	76
Liu et al94	2014	PLoS One	China	PRISMA	476
Liu et al95	2016	PLoS One	China	ARRIVE	396
Lu et al96	2015	Archives of Physical Medicine and Rehabilitation	USA	CONSORT	105
Lu et al97	2011	Expert Review of Anticancer Therapy	China	CONSORT	46
Ma et al98	2011	PLoS One	China	PRISMA	369
Ma et al99	2012	The Journal of Alternative and Complementary Medicine	China	PRISMA	88
Marshman and Farid100	2010	Community Dental Health	UK	CONSORT	48
McCormick et al101	2013	Journal of Shoulder and Elbow Surgery	USA	CONSORT	54
Miller et al102	2009	Academic Radiology	Canada	STARD	18
Moberg-Mogren and Nelson103	2006	American Journal of Occupational Therapy	USA	CONSORT	14
Moher et al104	2002	BMC Pediatrics	Canada	CONSORT	251
Montané et al105	2010	BMC Clinical Pharmacology	Spain	CONSORT	92
Montgomery et al106	2011	Trials	UK	CONSORT	76
Nicolau et al107	2013	The International Journal of Tuberculosis and Lung Disease	Canada	PRISMA	137
Norton-Mabus and Nelson108	2008	OTJR: Occupation, Participation and Health	USA	CONSORT	30
Ntala et al109	2013	Primary Care Respiratory Journal	Greece	CONSORT	35
Panic et al110	2013	PLoS One	Italy	PRISMA	90
Parsons et al111	2011	Journal of Bone and Joint Surgery, British Volume	UK	CONSORT STROBE	100
Patel et al112	2014	Psychological Medicine	UK	CONSORT	31
Piggott et al113	2004	Palliative Medicine	UK	CONSORT	93
Péron et al114	2012	Journal of the National Cancer Institute	France	CONSORT	357
Peters et al115	2015	PLoS One	The Netherlands	PRISMA	80
Plint et al116	2006	Medical Journal of Australia	Canada	CONSORT	8
Prady et al117	2008	PLoS One	UK	CONSORT	90
Pratoomsoot et al118	2015	PLoS One	Thailand	CONSORT	71
Rao et al119	2016	PLoS One	UK	STROBE	37
Rice et al120	2016	Journal of Psychosomatic Research	Canada	PRISMA	21
Rios et al121	2008	Journal of Clinical Endocrinology and Metabolism	Canada	CONSORT	89
Rikos et al122	2016	Multiple Sclerosis and Related Disorders	Greece	CONSORT	102
Schwarz et al123	2012	Journal of Clinical Periodontology	Germany	ARRIVE	75
Scott et al124	2012	The Pediatric Infectious Disease Journal	Switzerland	CONSORT	70
Shawyer et al125	2015	Journal of Pediatric Surgery	Canada	STROBE	48
Shea et al126	2006	BMC Medical Research Methodology	Canada	QUOROM	53
Shea et al127	2006	The Journal of Rheumatology	The Netherlands	QUOROM	57
Stevely et al128	2015	PLoS One	UK	CONSORT	68
Strech et al129	2011	Journal of Clinical Psychiatry	Germany	CONSORT	105
Tan et al130	2014	International Journal of Surgery	UK	PRISMA	37
Thabane et al131	2007	International Journal of Obesity	Canada	CONSORT	63
Tunis et al132	2013	Radiology	Canada	PRISMA	130
Turner et al133	2012	Cochrane Database of Systematic Reviews	Canada	CONSORT	45
Vigna-Taglianti et al134	2006	Annals of Oncology	Italy	QUOROM	80
Walleser et al135	2011	Journal of Clinical Epidemiology	Switzerland	CONSORT	106
Wang et al136	2007	Clinical Therapeutics	China	CONSORT	7422
Wang et al137	2013	PLoS One	China	CONSORT	27
Wangge et al138	2010	PLoS One	The Netherlands	CONSORT	232
Weingärtner et al139	2016	Expert Review of Clinical Pharmacology	Germany	CONSORT	117
Weir et al140	2012	International Journal of Medical Informatics	USA	PRISMA QUOROM	13
Wen et al141	2008	Journal of Clinical Epidemiology	China	QUOROM	161
Willis and Quigley142	2011	BMC Medical Research Methodology	UK	PRISMA	236
Yao et al143	2014	Eye	UK	CONSORT	65
Zafar et al144	2008	Clinical and Experimental Ophthalmology	Pakistan	STARD	76
Zhang145	2015	BMJ Open	China	MOOSE	607
Zhao et al146	2016	Medicine	China	CONSORT	68
Zheng et al147	2016	Open Heart	UK	CONSORT	33
Zhong et al148	2011	European Journal of Integrated Medicine	China	CONSORT	153
Zintzaras et al149	2010	Clinical Therapeutics	Greece	CONSORT	18
Zintzaras et al150	2012	BMC Musculoskeletal Disorders	Greece	STARD	103
Ziogas and Zintzaras151	2009	Annals of Epidemiology	Greece	CONSORT	261

Abbreviations: ARRIVE, Animal Research: Reporting In Vivo Experiments; BMC, BioMed central; BMJ, British Medical Journal; CHEERS, Consolidated Health Economic Evaluation Reporting Standards; CONSORT, Consolidated Standards of Reporting Trials; International Urogyn J, International Urogynecology Journal; JAMA, The Journal of the American Medical Association; MOOSE, Meta-analysis of Observational Studies in Epidemiology; OTJR, Occupational Therapy Journal of Research; PLoS, Public Library of Science; PRISMA, Preferred Reporting Items for Systematic reviews and Meta-Analyses; QUOROM, Quality of Reporting of Meta-analysis; STARD, Standards for Reporting of Diagnostic Accuracy; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology.

The majority of studies used the guideline checklist to evaluate the level of adherence and generated a mean score as summarized in Table S1. Table S1 summarizes the studies’ findings by guideline with authors’ conclusions for each study. Most studies described the adherence to the different guidelines using the following qualitative descriptors:

deficient, not adequately reported, generally poor, suboptimal, poor, medium, low, poor to moderate, lack of CONSORT adherence, bad, far from satisfactory, lack of standard reporting, improvement over the years has been minor, weak, quality of the articles varied substantially, insufficient, missed reporting some important factors, deficiencies in reporting, inconsistent, needs to be improved, inadequate, there is a need for improvement in quality of reporting, overall adherence is low.

A summary of the quantitative assessment of adherence to guidelines is presented in Table 2.

Table 2.

Summary of the included studies’ conclusions

Type of guideline	Total number of studies	Studies reporting inadequate adherencea
CONSORT	81 (three combined studies with both CONSORT and STROBE; one combined study with STROBE, CONSORT, and PRISMA)	71 (88%)
PRISMA	19 (one combined study with both PRISMA and QUOROM; one combined study with STROBE, CONSORT, and PRISMA)	16 (84%)
STROBE	8 (three combined studies with both CONSORT and STROBE; one combined study with STROBE, CONSORT, and PRISMA)	7 (88%)
QUOROM	10 (one combined study with both PRISMA and QUOROM)	5 (50%)
STARD	6	5 (83%)
ARRIVE	4	4 (100%)
CHEERS	1	1 (100%)
MOOSE	1	1 (100%)
All guidelines	124 (distinct studies)	109 (87.9%)

Note:

^aThe number of studies concluding that “some improvements are needed, reporting inadequate, poor, medium, suboptimal, etc.”

Abbreviations: ARRIVE, Animal Research: Reporting In Vivo Experiments; CHEERS, Consolidated Health Economic Evaluation Reporting Standards; CONSORT, Consolidated Standards of Reporting Trials; MOOSE, Meta-analysis of Observational Studies in Epidemiology; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; QUOROM, Quality of Reporting of Meta-analysis; STARD, Standards for Reporting of Diagnostic Accuracy; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology.

The level of adherence to all included reporting guidelines was 87.9% of all guidelines combined showing a need for improvement in reporting. Factors associated with poor adherence to CONSORT guideline included trials with significantly positive results, trials with the categorical outcome, trials conducted in North America compared to Europe, and trials funded by nonindustry source. A summary of factors associated with adherence standards is summarized in Table 3. Several factors were associated with better reporting standards relating to authors, study design, outcome specifications, year of publication (recent years of publications are associated with better reporting standards), journal, funding source, and study/author country.

Table 3.

Factors associated with reporting quality of articles using the CONSORT guideline

Study	Year	Sample size	Factors associated with adherence (↑↓)
Adie23	2013	150	Outcome specification (↑)a
			At least one author with a degree in epidemiology (↑)a
			Length of article in words (↑)a
			Allocation concealment (↑)a
			Random sequence (↑)a
			Power calculation (↑)
Agha et al31	2016	193	Greater details on study design (↑)
			Detailed outcome definitions and measurements (↑)
			Indication of how quantitative variables were handled during analyses (↑)
			Discussion of limits and potential sources of bias (↑)
Al-Namankany et al35	2009	173	Year of publication (↑)
Alvarez et al36	2006	98	Pharmaceutical industry funding (↑)a
			Year of publication (↑)a
			Sample size (↑)a
Areia et al38	2010	120	Publication in CONSORT-endorsing journals (↑)
			Year of publication (↑)
Balasubramanian et al40	2006	69	Number of authors (↑)a
			Multicenter studies (↑)a
			Declared funding sources (↑)a
			Reporting in medical journals (↑)a
Bath and Bath41	2000	114	Trial quality (↑)a
			Trials with positive outcome (↓)a
			Year of publication (↑)a
Borg Debano et al45	2012	23	Impact factor (↑)
			Funding reported (↑)
			Journal adopted CONSORT statement at the time of data collection (↑)
			Sample size (↑)
Cairo et al48	2012	64	Year of publication (↑)a
			Statistically significant clinical outcomes – positive study results (↓)a
Capili et al49	2010	10	Journal requiring the use of CONSORT (↑)
Chowers et al52	2009	49	Industry-sponsored trials (industry-sponsored vs. nonindustry-sponsored trial) (↑)
			Year of publication (↑)a
de Vries and van Roon58	2010	107	Sponsoring (↑)
DeMauro et al57	2011	179	Time trend (↑)a
			Journal type – general medical journals vs. pediatric journals (↑)a
Ethgen et al60	2009	132	Impact factor (↑)a
			Publication in CONSORT-endorsing journals (↑)a
Farrokhyar et al63	2007	50	Sample size (↑)a
			Year of publication (↑)a
			Location of the study (↑)a
			Source of funding (↓)
			Type of primary outcome in the study (categorical) (↓)
Gao et al71	2015	98	Supported by funding (↑)a
Herdan et al77	2011	37	Year of publication (↑)a
Karpouzis and Bonello81	2016	35	Year of publication (↑)a
			Larger sample size (↑)a
Kiehna et al82	2010	27	Publication in CONSORT-endorsing journals (↑)a
Kim et al83	2014	146	Year of publication (↑)a
Ladd et al85	2010	127	Year of publication (↑)a
Lee et al86	2013	83	Higher impact factor of journal (↑)a
			Journals requiring submission of CONSORT checklist (↑)a
Liu et al92	2013	290	Reporting of funding (↑)
			Journal endorses CONSORT (↑)a
			Good-quality RCTs (high Jadad scores) (↑)a
			Allocation concealment (↑)a
			Data analysis by randomized group (↑)a
			Sample size>100 (↑)a
Liu et al93	2015	76	Journal adopting CONSORT guidelines (↑)a
			Later publication year (↑)a
Lu et al96	2015	105	Year of publication (1976–2001, 2002–2010, 2011–2013) (↑)a
McCormick et al101	2013	54	High Jadad score (↑)a
Moberg-Mogren and Nelson103	2006	14	Year of publication (↑)a
Montané et al105	2010	92	Impact factor (↑)a
			Year of publication (↑)a
Montgomery et al106	2011	76	Year of publication (↑)a
Ntala et al109	2013	35	Impact factor (↑)
			Country with high income (↑)a
Péron et al114	2012	357	Trials with positive results (↓)
			Year of publication (↑)a
			Impact factor (↑)a
			Geographic region – North American compared to European trials (↓)a
			Sample size (↑)
Plint et al116	2006	8	Overall consort items (↑)
			Reporting method of sequence generation (↑)a
			Allocation concealment (↑)a
Prady et al117	2008	90	Standardized page length (↑)
			Year of publication (↑)a
Pratoomsoot et al118	2015	71	Country of publication (ASEANb vs. plus six) (↑ for some factors for ASEAN; ↑ for some factors for plus six)
Rikos et al122	2016	102	After the publication of CONSORT (↑)
			Impact factor (↑)a
			Year of publication (↑)a
Rios et al121	2008	89	Sample size (↑)a
			Industrial funding (↑)a
			Journal of publication (publication in JCEM) (↑)a
Scott et al124	2012	70	Trial registration (↑)
			Year of publication (↑)
			Trial size (↑)
Thabane et al131	2007	63	Type of intervention (pharmacological intervention vs. non-pharmacological intervention) (↑)a
			Sample sizes (↑)a
			Year of publication (↑)a
Turner et al133	2012	45	Time trend (↑)a
Yao et al143	2014	65	Number of authors (↑)
			Impact factor (↑)
Zhao et al146	2016	68	Year of publication (↑)
			Reporting of funding (↑)
			Reporting of informed consent form (↑)
			Reporting of ethical approval (↑)
Zheng et al147	2016	33	Number of authors (↑)a
			Number of patients (↑)a
			Impact factor (↑)a
			Time trend (↑)a
			Number of participants (↑)
			Treatment duration (↑)
			Reporting of funding (↑)
Zhong et al148	2011	153	Non-Chinese reports (compared to those published in mainland China) (↑)a
			Publication in CONSORT-endorsing journals (↑)a
Ziogas and Zintzaras151	2009	261	Year of publication (↑)a
			Impact factor (↑)a

Notes:

^aStatistically significant increase/decrease, p≤0.05; (↑), positively associated with adherence; (↓), negatively associated with adherence. The number of studies concluding that “some improvements are needed, reporting inadequate, poor, medium, suboptimal, etc”.

^bAssociation of Southeast Asian nations, Association of Southeast Asian Nations (ASEAN) plus six groups, which composed of the members of the ASEAN plus Australia, China, India, Japan, New Zealand, and South Korea.

Abbreviations: CONSORT, Consolidated Standards of Reporting Trials; JCEM, The Journal of Clinical Endocrinology and Metabolism; RCT, randomized control trial.

Factors associated with improved adherence to reporting guidelines

Author factors

The included studies reported that the expertise of the author team, for example, an epidemiologist, improved the quality of reporting the study. In addition, having multiple authors also improved reporting quality.

Study factors

Study design with detailed methods including allocation concealment, randomization, specific outcome measures, sample size and power calculations, acknowledgment of limitations and sources of bias, larger sample size, registration of clinical trials, pharmacological interventions, and detailed statistical analysis plan were associated with better reporting and adherence to reporting guidelines. Year of publication was also associated with adherence in which the more recently published articles had increased adherence.

Journal factor

Publications in journals endorsing reporting guidelines have better adherence to these guidelines than articles published in journals that do not endorse such guidelines. In addition, journals’ impact factor, medical journals, and journals with restriction on the number of words per article also had articles with better reporting standards. Publication in a general medical journal was associated with better reporting quality than a specialty journal.

Ethics and funding factors

Articles that reported ethical approval, participants’ consent, and the source of funding were associated with improved adherence to reporting guidelines.

Country of study factors

Geographic location of the study has an impact on the quality of reporting and adherence to reporting guidelines, for example, studies reported from Europe had better reporting standards compared to studies from North America. Studies reported from China had lower adherence to guidelines than elsewhere indicating geographical variations may directly or indirectly impact the level of adherence to reporting guidelines in the medical literature.

Quality assessment of included studies

For each included systematic review, we performed a quality assessment using the modified AMSTAR/OQAQ score. Table 4 provides the total score out of 20 for each study. The scores varied from 9 to 20. The average score for all the included studies is 16.14. The lowest scores were related to items 5 and 6 of the quality assessment related to the availability of the primary studies’ characteristics similar to a previously reported study.⁷ Items 5 and 6 were evaluated if there was information on included and excluded studies provided and if the characteristics of included studies provided, respectively.

Table 4.

Reporting quality of the 124 included systematic reviews, assessed by the modified AMSTAR/OQAQ (10 items, score out of 20)

Study	Global score
Adie28	17
Adie et al29	18
Agha et al30	15
Agha et al31	14
Aguiar et al32	14
Aguiar et al33	19
Al Faleh and Al-Omran34	16
Al-Namankany et al35	15
Alvarez et al36	10
Anttila et al37	15
Areia et al38	18
Augestad et al39	20
Balasubramanian et al40	16
Bath and Bath41	16
Bereza et al42	20
Bian et al43	15
Biondi-Zoccai et al44	15
Borg Debano et al45	9
Bousquet et al46	18
Bramhall et al47	10
Cairo et al48	19
Capili et al49	15
Cavadas et al50	17
Choi et al51	17
Chowers et al52	12
Cook et al53	18
Daitch et al54	17
Dasi et al55	19
Delaney et al56	14
DeMauro et al57	17
de Vries and van Roon58	18
Dias et al59	17
Ethgen et al60	13
Eyawo et al61	18
Fan et al62	18
Farrokhyar et al63	19
Fidalgo et al64	18
Fleming et al65	15
Fontela et al66	17
Freeman et al67	11
Froud et al68	16
Fung et al69	17
Gagnier et al70	16
Gao et al71	13
Gianola et al72	12
Gohari et al73	15
Gulin et al74	14
Halpern et al75	14
Hemels et al76	19
Herdan et al77	15
Huang et al78	12
Hui et al79	18
Junhua et al80	13
Karpouzis and Bonello81	16
Kiehna et al82	16
Kim et al83	16
Kober et al84	17
Ladd et al85	19
Lee et al86	16
Lee et al87	17
Li et al88	18
Li et al89	15
Li et al90	14
Liu et al91	19
Liu et al92	16
Liu et al93	14
Liu et al94	17
Liu et al95	19
Lu et al96	18
Lu et al97	18
Ma et al98	19
Ma et al99	16
Marshman and Farid100	14
McCormick et al101	16
Miller et al102	17
Moberg-Mogren and Nelson103	16
Moher et al104	14
Montané et al105	15
Montgomery et al106	17
Nicolau et al107	16
Norton-Mabus and Nelson108	10
Ntala et al109	18
Panic et al110	11
Parsons et al111	17
Patel et al112	13
Piggott et al113	14
Péron et al114	15
Peters et al115	17
Plint et al116	18
Prady et al117	19
Pratoomsoot et al118	15
Rao et al119	18
Rice et al120	19
Rios et al121	20
Rikos et al122	17
Schwarz et al123	10
Scott et al124	16
Shawyer et al125	15
Shea et al126	13
Shea et al127	19
Stevely et al128	18
Strech et al129	18
Tan et al130	14
Thabane et al131	19
Tunis et al132	18
Turner et al133	20
Vigna-Taglianti et al134	15
Walleser et al135	19
Wang et al136	15
Wang et al137	17
Wangge et al138	12
Weingärtner et al139	17
Weir et al140	20
Wen et al141	18
Willis and Quigley142	20
Yao et al143	16
Zafar et al144	16
Zhang145	18
Zhao et al146	17
Zheng et al147	18
Zhong et al148	17
Zintzaras et al149	18
Zintzaras et al150	14
Ziogas and Zintzaras151	15

Abbreviations: AMSTAR, Assessment of Multiple Systematic Reviews; OQAQ, Overview Quality Assessment Questionnaire.

Discussion

The medical literature is paramount to the progression of the understanding of health and disease and the establishment of priorities and recommendations for prevention, diagnosis, treatment, and measurement of outcomes. To implement research findings, transparent and consistent reporting standards are needed to help make informed decisions. Such standards have been set by the CONSORT working group and others for the past 2 decades with the aim of improving the reporting standards in biomedical research. It is expected that the introduction of new change to the current practice will take time to adopt and disseminate. However, the uptake of the widely available guidelines has been less than ideal. We define suboptimal and less than ideal as <100%. The whole idea of a systematic review is to have completely transparent methods reported, so everyone can follow and reproduce the results. Inherently, systematic reviews are meant to be a more rigorous study design. This allows them to produce meaningful results than individual studies. Thus, when reviews fail to adhere to reporting guidelines, it calls into question the consistency of their results. Given the weight that systematic reviews have in the scientific community, it is imperative that we hold reviews to a high standard.

Five years ago, we investigated the level of adherence to reporting standards in the medical literature, and we identified 86% of the systematic reviews conducted on the level of adherence to reporting guidelines of the medical literature to be less than ideal.⁷ Since our previous scoping review, many new revisions and updates to reporting guidelines have been introduced. Currently, there are 358 reporting guidelines on the EQUATOR Network website¹⁶ for many study types that are freely available. However, endorsement of reporting guidelines by journals still remains low.

Among all the factors that can improve the reporting quality, such as author factors, study factors, journal factors, ethics and funding factors, and country of study factors, author factors as well as their limitations have been studied in other researches. The author factors were the number of the authors of the publication and the level of expertise in the different research methods. Multiple authorships were shown to be an important determinant of the impact of the research being produced and its likelihood of being cited.¹⁷ The complexity and cost of medical research today requires multiple levels of expertise in various disciplines as well as accountability and oversight by study team members, institutions, and funding bodies. It is known that the number of authors per article has increased over the past few decades^18,19 with a concern posed to question the roles of multiple authors and the most senior academics holding senior authorship at the expense of others in the team.²⁰ Other studies have reported that the research produced by teams rather than single authors was impactful and more frequently cited, at least in certain fields.²¹ It is likely that multiple authorships arising from collaborative efforts have advantages of producing good quality impactful research; however, multiple authorships also have limitations and may not be feasible at every setting due to geographical limitations or strict timeline to follow as bringing more authors is time-consuming.²² In this review, we found that having multiple authorships is important to have publications with better adherence to reporting guidelines. However, the role of each author and the hierarchy of authorship should be clarified for successful collaborations and research impact as discussed earlier.

Study factors that improved adherence to reporting guidelines included well-designed, detailed study methods and adequately powered studies. Study results could be altered regarding trial designs, qualities, and methods.²³ Therefore, guidelines such as CONSORT statement that is designed for randomized control trials (RCTs), STROBE guideline for observational studies, and PRISMA guideline for systematic reviews were invented accordingly based on different study designs. RCTs are also considered as the highest level of primary evidence in the clinical practice, and therefore it is vital that these trials are reported according to the expected standards.²⁴

Other factors reported that might improve the level of adherence to reporting guidelines included journals endorsing these guidelines. The Internal Committee of Medical Journal Editors (ICMJEs) recognized the importance of reporting guidelines in ensuring study details that are described adequately to be evaluated appropriately and encouraged journals to request these reporting standards from authors.²⁵ The EQUATOR Network has valuable resources and tool kits to assist authors and journal editors to adopt the reporting guidelines and provide case studies of journals endorsing the guidelines. Since journals that endorsed reporting guidelines often ask authors to submit a completed checklist regarding the guidelines, it improves the quality of reporting for those journals endorsing these guidelines. Yet, not all journals currently endorse the guidelines. According to the CONSORT website, there are 585 journals that endorse CONSORT,²⁶ while there are about 30,000 journals indexed in PubMed.²⁷ While not all of these indexed journals publish RCTs, many of them do publish them, but do not adhere to CONSORT guidelines.²⁷

The EQUATOR Network also has tool kits for ethics boards and study sponsors to ensure that the reporting guidelines are considered when these agencies review research submissions for ethical approval or funding requests. It is therefore important that all stakeholders take part in the use and dissemination of the reporting guidelines to enhance the quality of medical research and biomedical literature.

Limitations

The included studies are limited to only eight of the reporting guidelines, and therefore the current study lacks the generalizability to other guidelines that may have a better adherence standard. In addition, there was no comparison between studies to ensure that they are using qualitative descriptors such as “inadequate” or “suboptimal” with the same operational definition. The studies do not provide sufficient information regarding the operationalization of qualitative descriptors to allow us to adequately compare descriptors across studies.

In addition, the study was limited to systematic reviews that present with its own set of limitations. The most notable limitation is the low mean score on the quality assessment since each systematic review follows different reporting guidelines or does not follow guidelines at all and the lack of detailed data on the included studies’ characteristics. Furthermore, a quantitative analysis was not conducted, as not all included studies provided relevant data. Strict inclusion criteria may have allowed a quantitative analysis. However, for the sake of a more representative sample, such criteria were not implemented.

The inclusion of studies in English only is also a limitation to a selected section of the medical literature and did not include other reporting guidelines that may be in use in other languages.

Despite the limited scope of inclusion criteria and quality limitation of the included studies, this review provides an insight into the limited uptake of reporting guidelines and calls for exploring barriers to such uptake. Future studies may include broad surveys of authors, journal editors, funding agencies, ethics boards, and readers to solicit opinions and understanding of the role of reporting guidelines in the medical research and literature.

Conclusion

Current adherence to reporting guidelines in the medical literature is suboptimal. However, there are factors associated with better reporting upon which we can develop strategies for better reporting. Reporting guidelines are an imperative tool in the endeavor to improve the consistency of reporting in the medical literature. However, the suboptimal uptake and correct usage of reporting guidelines demonstrate the need for further emphasis in the scientific community to encourage the use of reporting guidelines. The responsibility for improving the transparency, quality, and reproducibility of medical literature lies with all stakeholders from the research participants to regulatory authorities and everyone in between including authors, readers, educators, funders, academic and health care institutions, editors, peer reviewers, and guideline developers. Future studies may include broad surveys of authors, journal editors, funding agencies, ethics boards, and readers to solicit opinions and understanding of the role of reporting guidelines in the medical research and literature.

Data sharing statement

Unpublished study data are available upon request.