Quality of Reporting of Neonatal and Infant Trials in High-Impact Journals

Abstract

OBJECTIVES:

To perform a systematic review of the quality of reporting for randomized controlled trials (RCTs) with infants and neonates that were published in high-impact journals and to identify RCT characteristics associated with quality of reporting.

METHODS:

RCTs that enrolled infants younger than 12 months and were published in 2005–2009 in 6 pediatric or general medical journals were reviewed. Eligible RCTs were evaluated for the presence of 11 quality criteria selected from the Consolidated Standards of Reporting Trials guidelines. The relationships between quality of reporting and key study characteristics were tested with nonparametric statistics.

RESULTS:

Two reviewers had very good agreement regarding the eligibility of studies (κ = 0.85) and the presence of quality criteria (κ = 0.82). Among 179 eligible RCTs, reporting of the individual quality criteria varied widely. Only 50% included a flow diagram, but 99% reported the number of study participants. Higher quality of reporting was associated with greater numbers of study participants, publication in a general medical journal, and greater numbers of centers (P < .0001 for each comparison). Geographic region and positive study outcomes were not associated with reporting quality.

CONCLUSIONS:

The quality of reporting of infant and neonatal RCTs is inconsistent, particularly in pediatric journals. Therefore, readers cannot assess accurately the validity of many RCT results. Strict adherence to the Consolidated Standards of Reporting Trials guidelines should lead to improved reporting.

WHAT'S KNOWN ON THIS SUBJECT:

Variations in how randomized trials are reported make it difficult for readers to assess the quality of a trial and therefore the validity of the results. Adoption of the Consolidated Standards of Reporting Trials guidelines improves the quality of reporting.

WHAT THIS STUDY ADDS:

Neonatal and infant trial reports frequently omit important methodologic details. Therefore, readers cannot assess accurately the validity of the trial results. Authors and editors have a responsibility to provide readers with high-quality reports of randomized controlled trials.

Randomized controlled trials (RCTs) are the standard method for the evaluation of treatments and technologies in clinical medicine. Inadequate trial design and inadequate reporting are associated with biased estimates of effect. ^{1 – 3} Incomplete descriptions of important methodologic details in RCT publications make it difficult to assess the quality of the design, conduct, and analysis of the trial and limit readers' ability to interpret the results.

At least 25 scales and 9 checklists to assess the methodologic quality of RCTs have been developed. ^1,2,4,5 The Consolidated Standards of Reporting Trials (CONSORT) statement, which was first published in JAMA in 1996, is the most widely used checklist to assess the quality of reporting of clinical trials. ^6,7 The CONSORT statement is composed of items that are essential for high-quality design, conduct, and reporting of RCTs. Since the initial publication of the CONSORT statement, the guidelines have been expanded and explained in detail. ^{7 – 10} The CONSORT statement was designed to serve as a guide for authors, reviewers, and editors, as well as readers. Journals that adopt the CONSORT guidelines have a higher quality of RCT reporting. ^11,12

Many publications have examined the quality of reporting for adult specialties and diseases. ^{13 – 18} One study used the CONSORT guidelines to evaluate the quality of reporting in pediatric trials sampled from the Cochrane registry. ³ However, no study has explored whether there is a gap between the CONSORT guidelines for reporting and articles published in the infant and neonatal literature. This systematic review aimed to evaluate the quality of reporting for neonatal and infant RCTs published in high-impact journals and to identify study factors that are associated with the quality of reporting.

METHODS

Data Sources and Study Selection

We identified neonatal and infant studies that were classified by PubMed as randomized clinical trials and were published in English between January 2005 and December 2009. The search was limited to 6 high-impact journals that regularly publish pediatric studies, namely, Pediatrics; The Journal of Pediatrics; Archives of Disease in Childhood, Fetal and Neonatal Edition; The New England Journal of Medicine; JAMA, the Journal of the American Medical Association; and The Lancet. The specific search terms were as follows: “infant, newborn”[MeSH Terms] and “1/1/2005”[Publication Date]: “12/31/2009”[Publication Date] and “humans”[MeSH Terms] and randomized controlled trial[Publication Type] and English[Lang] and jsubsetaim[Text Word] and “infant”[MeSH Terms] and (“The New England journal of medicine”[Jour] or “Lancet”[Jour] or “JAMA”[Jour] or “Pediatrics”[Jour] or “The Journal of pediatrics”[Jour], or “Archives of disease in childhood. Fetal and neonatal edition”[Jour]).

Articles were included in the analysis if they met the following criteria: (1) subjects receiving the intervention were infants younger than 12 months; (2) the article was the first full-length publication of the methods and results of a trial, not a short report, follow-up report, or subgroup analysis; (3) the study design was a RCT; and (4) the study evaluated the effect of an intervention on patients and not on simulated subjects, parents, or clinicians. Two neonatologists (Drs DeMauro and Giaccone) independently performed the literature search, evaluated each article for inclusion, and extracted data. All disagreements were resolved through consensus.

Data Extraction

The methods and results sections of eligible trials were analyzed for the presence or absence of 11 quality criteria from the most recent CONSORT statement. ¹⁰ These items represent the elements of study design and reporting that are most useful for the assessment of bias and validity. Quality criteria, listed with the corresponding item number from the 2010 CONSORT checklist, were as follows. (1) Eligibility criteria (item 4a). The eligibility criteria of study participants, including specific inclusion and exclusion criteria, were listed. (2) Primary outcome (item 6a). A primary outcome was defined and ascertained at a specified time point. Credit was given if an outcome was used to perform a sample size calculation, even if that outcome was not identified specifically as primary. (3) Target sample size (item 7a). Any sample size calculation was given credit, because it was difficult to differentiate clearly between a priori and posthoc power calculations. (4) Method of randomization (item 8a). The method for generating the randomization sequence (eg, random number table or computerized randomization) was defined. Credit also was given if randomization was described as being performed by a centralized data coordinating center, even if the methods used by the data coordinating center were not described explicitly. (5) Method of allocation concealment (item 9). The method by which future group assignments were concealed from caregivers and the study team (eg, sealed opaque envelopes) was described. (6) Blinding (item 11a). Credit was given for any description of blinding methods, even if blinding was not described for all team members. (7) Number of participating centers (item 4b). The number of centers was explicitly stated or otherwise could be determined with the information provided. Studies clearly described as single center also received credit. (8) Study flow diagram (item 13a). A diagram depicting the number of infants screened, assigned randomly, and evaluated for the primary outcome was included. (9) Number of subjects enrolled (items 13a and 13b). Credit was given if the total number of subjects enrolled was stated. (10) Number of subjects analyzed (item 16). The total number of subjects analyzed either was stated in the text or could be determined from a table or study diagram. (11) Primary outcome result (item 17a). The results of the primary outcome were reported. Credit was not given if no primary outcome was identified.

The primary goal of this review was to identify the total number of quality criteria that were fulfilled by each RCT report. We also examined the proportion of studies that reported each criterion. Finally, we tested the correlations between the quality of RCT reporting and the following study characteristics: year of publication, nations in which the study was conducted, number of participating centers, exact number of subjects enrolled, and whether the authors reported a statistically significant primary outcome result based on the standard for significance set in the individual study.

Data Analysis

We planned nonparametric analyses because it was unlikely that the number of quality criteria fulfilled by each report, sample size, and number of centers would be normally distributed. Correlations were analyzed with Spearman's test, 2-sample comparisons were performed with the Wilcoxon rank-sum test, and multiple-sample comparisons were performed with the Kruskal-Wallis, equality-of-populations, rank-sum test. Significance was measured at the α = .05 level. All analyses were performed with Stata 11/IC (Stata Corp, College Station, TX).

RESULTS

The search revealed 350 potentially relevant articles (Fig 1). We identified 179 articles that met all eligibility criteria, with 92% agreement (κ = 0.85). We then reviewed all included articles for the presence of the 11 quality criteria, with 92% agreement (κ = 0.82). All disagreements were resolved through consensus.

FIGURE 1.

Flow diagram of included and excluded studies. ^a Thirty-nine studies did not meet 2 inclusion criteria, and 3 studies did not meet 3 inclusion criteria.

The proportions of studies that fulfilled each quality criterion ranged from 50% to 99% (Table 1). The median number of criteria fulfilled by each article was 9 (interquartile range [IQR]: 8–10). Only 79% of studies reported a primary outcome. Moreover, occasionally it was difficult to determine whether the primary outcome was developed a priori and often it was difficult to determine whether the sample size calculation was performed a priori. When the primary outcome was a hypothesis of equivalence, this rarely was reflected in the sample size calculation. Although description of any blinded study participant resulted in credit, blinding most often was described only at the level of the intervention and not at the level of the outcome assessment.

TABLE 1.

RCTs Reporting Each Quality Criterion

	n/N (%)
Eligibility criteria are listed	150/179 (84)
Primary outcome is stated	142/179 (79)
Sample size estimate is provided	138/179 (77)
Method of randomization is described	106/179 (59)
Method of allocation concealment is described	123/179 (69)
Blinding of study team members is mentioned, unless study is described as unblinded	96/115 (83)
If study is multicenter, the number of participating centers is stated	75/80 (94)
Study flow diagram is included	89/179 (50)
Number of study participants is stated	177/179 (99)
Number of subjects analyzed is stated	168/179 (94)
Result of the primary outcome is included, if a primary outcome is identified	137/142 (96)

Most of the 179 studies were small. The median number of patients enrolled was 89 (IQR: 42–201) (Table 2). Ninety-nine RCTs (55.3%) were performed in a single center. Five multicenter studies did not report the number of participating centers. The median number of centers in the remaining 75 multicenter RCTs was 5 (IQR: 3–15). There was a highly significant correlation between the quality of reporting and both the number of subjects and the number of centers (P < .0001).

TABLE 2.

Characteristics of 179 Neonatal and Infant RCTs

No. of centers (if reported and multicenter; N = 75), median (IQR)	5 (3–15)
No. of participants enrolled (if reported; N = 177), median (IQR)	89 (42–201)
Journal, n (%)
The New England Journal of Medicine	15 (8.4)
JAMA, the Journal of the American Medical Association	5 (2.8)
The Lancet	8 (4.5)
Pediatrics	73 (40.8)
The Journal of Pediatrics	43 (24.0)
Archives of Disease in Childhood, Fetal and Neonatal Edition	35 (19.6)
Year published, n (%)
2005	40 (22.4)
2006	39 (22.8)
2007	22 (12.3)
2008	45 (25.1)
2009	33 (18.4)
Region, n (%)
International	19 (10.6)
United States	42 (23.5)
Western Europe	27 (15.1)
Southern Europe/Israel	21 (11.7)
United Kingdom	12 (6.7)
Australia/New Zealand	11 (6.2)
Canada	8 (4.5)
Central/South America	8 (4.5)
Scandinavia	8 (4.5)
Southeast/South Asia	8 (4.5)
East Asia	8 (4.5)
Africa	7 (3.9)
Primary outcome statistically significant (if result was reported), n/N (%) a	64/136 (47.1)

^aThe significance of the primary outcome result could not be determined for 1 study because of the statistical methods used.

There were 28 infant or neonatal RCTs (15.6%) published in general medical journals and 151 (84.4%) published in pediatric journals (Fig 2). The quality of the studies that were published in general medical journals was significantly higher than the quality of those that were published in pediatric journals (P < .0001).

FIGURE 2.

Distribution of total number of quality criteria fulfilled by RCT reports in pediatric (left) (n = 151) and general medical (right) (n = 28) journals. The median quality score (hatched bars) was 9 (IQR: 7–10) in pediatric journals and 10 (IQR: 10–11) in general medical journals. The number of criteria fulfilled was significantly different between journal types (P < .0001).

The 5-year time period was divided into 1-year epochs (Table 2). The number of publications did not change but the quality of reporting seemed to increase over time (P = .03). There was no association between the trial results being positive (ie, statistically significant) and the quality of reporting (P = .20). The 179 RCTs represented research that was performed throughout the world. There were 19 international multicenter trials (10.6%) and 160 trials (89.4%) performed in individual nations. The single-nation studies were categorized according to the world region in which they were performed. There was no correlation between region and the quality of reporting (P = .14).

DISCUSSION

We reviewed the quality of reporting for 179 RCTs that enrolled infants younger than 12 months and were published in 6 general medical and pediatric journals between 2005 and 2009. We found large variations in the quality of reporting. It is concerning that 21% of the included RCTs did not identify a primary outcome. Single-center studies, studies with small numbers of participants, and trial reports in pediatric journals were more likely to be of lower quality. Although smaller studies may be of inherently inferior quality, they also may receive less rigorous review and editing. In the adult literature, general medical journals enforce the CONSORT guidelines more consistently than do adult subspecialty journals. ¹⁹ We suspect that variations in enforcement of CONSORT guidelines between pediatric and general medical journals may contribute to the significant difference in the quality of reporting we observed.

Our findings are consistent with the few studies that have examined the quality of pediatric RCT publications. Two 2010 reviews applied the Cochrane Collaboration Risk of Bias tool and found high risk of bias in a large proportion of pediatric RCTs. ^3,20 Another recent review identified poor compliance with CONSORT guidelines for reporting of adverse drug reactions in pediatric RCTs. ²¹ A 2001 report identified poor reporting of blinded outcome assessment and randomization methods in the pediatric surgery literature. ²² A review of all RCTs published in a single pediatric journal over a 15-year period found incomplete reporting of randomization methods, blinding of outcome assessment, and data in primary analyses, with only 43% of the RCTs reporting their method of randomization. ²³ Thirteen years later, we found that only 40% of infant RCTs in the same journal provided this information. Although this review was limited to high-impact journals, we found that the reporting quality of neonatal and infant RCTs was lower than the overall reporting quality of RCTs published in leading general medical journals that endorse the CONSORT statement. ²⁴

This study has several strengths. It is the first evaluation of the quality of trial reports in the neonatal and infant population. The design allowed us to explore associations between trial characteristics and quality of reporting. There are also limitations. For ease of analysis, we weighed all of the quality criteria equally. However, the relative importance of each item may not be the same for every study. Another limitation of this study is that we were unable to differentiate between items not reported and items not performed. Failure to report a bias-reducing procedure may not mean that the procedure was not performed. ²⁵ A study comparing trial registry data with published reports may help answer this question. However, most readers do not routinely access registration data, and pediatric studies have a low rate of registration. ³ Regardless of whether methodologic details were included in a trial registry, low quality of published reports is associated with exaggerated estimates of effect. ^{1 – 3} Both authors and editors, therefore, have a responsibility to publish trial reports that contain all of the information necessary to assess the validity of the results. Finally, our study was limited to a select group of high-impact journals. As a result, it is likely that we have underestimated the problem of poor reporting quality in neonatal and infant studies. This is supported by the fact that some previous reports found lower quality of reporting in pediatric studies than we have reported. ³

CONCLUSIONS

The high quality of RCT reports in general medical journals is encouraging. However, general medical journals publish only a small number of neonatal or pediatric RCTs. In addition, although the number of adult RCTs published in general medical journals is increasing, the number of pediatric RCTs published in the same journals is nearly stagnant. ²⁶ Therefore, the influence of general medical journals on the quality of reporting for pediatric RCTs may be limited. With consistent enforcement of the CONSORT guidelines in pediatric journals, investigators would be compelled to design and to publish studies of higher quality.

As practitioners seek more RCT-based evidence to guide pediatric care, this evidence must come from studies with high-quality design and reporting. ^27,28 Only then will clinicians have access to the best possible evidence with which to make decisions about the care of their neonatal and pediatric patients.