Skip to main content
Journal of Pharmacy & Bioallied Sciences logoLink to Journal of Pharmacy & Bioallied Sciences
. 2025 Jan 30;16(Suppl 5):S4593–S4598. doi: 10.4103/jpbs.jpbs_556_24

Assessment of Reporting Quality in Orthodontic Systematic Reviews: An Observational Study

Fahad Alharbi 1,, Rawda O Alghabban 1
PMCID: PMC11888721  PMID: 40061652

ABSTRACT

Objective:

This study aimed to assess the reporting quality of orthodontic systematic reviews published between 2018 and 2022.

Methods:

A retrospective observational study was conducted, analyzing 2114 articles from selected orthodontic journals. Descriptive statistics were used to summarize the characteristics of the included systematic reviews. Compliance scores were calculated for each review, and the overall compliance score was determined.

Results:

The European Journal of Orthodontics (EJO) contributed the most systematic reviews (49%), followed by the Journal of Orthodontics (JO, 8%). Europe had the highest first-author proportion (42%), while Asia accounted for 26%. The majority of reviews involved 4-6 authors (59%). The overall compliance score was 79.2% (95% CI: 78.1-80.3). EJO had the highest compliance (79.9%, 95% CI: 78.3-81.5), followed by Angle Orthodontist (78.9%, 95% CI: 76.8-81.1). Compliance scores varied across regions, with Europe scoring the highest (79.7%, 95% CI: 77.9-81.6). Reporting key Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist items, such as meta-analysis methods and reporting bias assessment, was inadequate.

Conclusion:

The reporting quality of orthodontic systematic reviews varied, highlighting the need for improvement in areas such as reporting meta-analysis methods and addressing reporting bias. Adherence to the PRISMA guidelines should be emphasized to enhance transparency and strengthen the credibility of orthodontic systematic reviews, supporting evidence-based decision-making in orthodontic practice.

KEYWORDS: Evidence-based medicine, orthodontics, PRISMA guidelines, reporting quality, systematic reviews

INTRODUCTION

Systematic reviews are crucial in evidence-based medicine, rigorously synthesizing existing evidence to answer specific research questions.[1] Their meticulous and transparent methodologies minimize bias, ensuring a comprehensive evaluation of relevant studies, which enhances the reliability of the synthesized evidence. By systematically searching and evaluating studies, they reduce biases like selective reporting and publication bias, increasing the validity and reliability of the evidence.[2] Including a meta-analysis allows for quantitative synthesis and statistical analyses, providing precise estimates of intervention effects, thus strengthening the evidence base and facilitating evidence-based decision-making.[3] Robust reporting of systematic reviews is vital for transparency, credibility, and reliability, enhancing evidence-based decision-making and patient outcomes.[4] Good reporting practices ensure trust among clinicians, policymakers, and researchers, enabling them to assess the quality, methodology, limitations, and potential biases of the review. This transparency is crucial for informed clinical decisions, research design, and policy formulation.

A widely recognized guideline for reporting systematic reviews is the PRISMA.[5] PRISMA provides a comprehensive checklist of essential items to be included in systematic review reports, covering various domains and steps in the review process. Adhering to PRISMA guidelines enhances the clarity and consistency of reporting, facilitating the replication and appraisal of the review by others.[6] However, compliance with PRISMA guidelines across medical and dental fields is often suboptimal.[7,8,9,10,11,12,13,14,15,16,17] Despite journal endorsements, studies show the need for improved reporting quality. Regular assessments of reporting practices are essential. This study aimed to evaluate systematic review reporting quality according to PRISMA guidelines, identifying areas for improvement and highlighting the importance of robust reporting. The findings will enhance transparency, credibility, and reliability, supporting evidence-based decision-making and improving patient outcomes.

MATERIALS AND METHODS

A retrospective observational study was carried out from January 2018 to December 2022, and systematic reviews were identified. Four leading orthodontic journals were the source of the papers. These publications included the American Journal of Orthodontics and Dentofacial Orthopedics (AJODO), Angle Orthodontist (AO), European Journal of Orthodontics (EJO), and the Journal of Orthodontics (JO). Articles with “systematic review” or “meta-analysis” in the title or abstract satisfied the inclusion criteria. If the report contained one or more of these keywords, the full text was retrieved to confirm that the article described a systematic review. Exclusion criteria included narrative reviews, historical reviews, scoping reviews, and case reports with in-depth literature reviews. The two authors independently and in duplicate conducted the screening. Disagreements were resolved between the authors through discussion until a consensus was reached.

Assessment of systematic reviews

One of the authors conducted an analysis to determine the level of reporting accuracy present in the full texts of the included systematic reviews (RA). The degree to which each of the included reviews complied with the PRISMA guidelines served as the criterion for determining the reviews’ overall quality as reported. The title, abstract, introduction, methods, results, discussion, conclusion, and funding are among the 27 items included in these guidelines for each section of the review. Each item was scored as “Yes” if it had been reported and “No” if it had not been reported. If an item could not be included, such as synthesis of results in a systematic review without a meta-analysis, a ‘NA’ was assigned to it, and it was removed, and the denominator was adjusted accordingly[18]. Finally, an overall mean score of compliance with PRISMA-A guidelines was calculated for each included systematic review and converted to a percentage. Each review received an overall percentage score based on the sum of the applicable items.

For every included systematic review, the following details were also retrieved: the publishing journal, the year of publication, the number of authors, the continent and country of the first author, the affiliation of the first author, and the involvement of a statistician.

To assess the inter-rater reliability between the authors, the first author (FA) assessed a 10% random sample. In addition, the second author (RA) reevaluated the reporting quality of an additional 10% of the random sample following a 3-month washout period in order to determine intraexaminer reliability.

Statistical analysis

For included systematic reviews, the descriptive data and percentage of compliance with PRISMA checklist items were assessed. To determine the relationships between the overall mean compliance score and other independent variables, a linear regression analysis was performed. The assessment of inter- and intraexaminer reliability was conducted through the application of intercorrelation coefficient (ICC) tests. Statistical analysis was performed using IBM SPSS Statistics for Macintosh, Version 22.0 (IBM Corp, Armonk, NY).

RESULTS

The reliability of scoring the reporting in the study was assessed using ICC tests, which indicated high levels of inter- and intrarater reliability at 0.8 and 0.89, respectively. Out of the 2114 articles analyzed from selected orthodontic journals published between 2018 and 2022, only 5.2% (n = 111) met the criteria for systematic reviews. The EJO contributed 49% of the included reviews, while the JO provided 8% of the total. Regarding first authorship, 42% of the systematic reviews originated from Europe, with Asia accounting for 26%. Furthermore, 96% of the included reports had a first author affiliated with an academic institution. The majority (59%) of the systematic reviews involved a team of 4-6 authors, representing over half of the analyzed reviews. Table 1 provides a summary of the characteristics of the systematic reviews that were included in the study. The reporting of PRISMA checklist items varied across the included papers, ranging from 33% to 100%. Out of the 27 items on the PRISMA checklist, 17 items were reported adequately in less than 75% of the included papers. Specifically, items related to meta-analysis methods (item 13d), methods to assess heterogeneity (item 13e), and reporting bias assessment due to missing data (item 14) were among the items that were reported inadequately in a significant proportion of the papers. The detailed percentages of checklist item reporting can be found in Table 2.

Table 1.

Characteristics of the 111 SRs

Characteristic Number of publication Percentage Mean Score SD 95% CI
Journals
 AJODO 20 18% 78.4 7.1 75.1 to 81.8
 JO 9 8% 78.8 5.1 74.9 to 82.7
 EJO 54 49% 79.96 5.9 78.3 to 81.5
 AO 27 24% 78.9 5.3 76.8 to 81.1
Year
 2018 23 21% 79.3 6.03 76.6 to 82.01
 2019 18 16% 77.9 4.6 75.6 to 80.2
 2020 25 23% 79.7 6.5 77.01 to 82.4
 2021 26 23% 79.6 5.2 77.4 to 81.7
 2022 19 17% 79.8 7.2 76.3 to 83.3
Authors
 <4 29 26% 79.5 6.2 77.1 to 81.9
 4-6 66 59% 78.4 5.7 77.01 to 79.8
 >6 16 14% 81.8 6.01 78.6 to 85.1
Work in Academia
 No 4 4% 74.4 11.7 47.9 to 85.4
 Yes 107 96% 79.5 5.9 78.4 to 80.7
Continent
 ASIA 29 26% 79.3 5.9 77.0 to 81.5
 AFRICA 2 2% 75.0 . .
 NORTH AMERICA 9 8% 79.6 5.9 75.0 to 84.2
 SOUTH AMERICA 21 19% 78.1 5.1 75.7 to 80.4
 EUROPE 47 42% 79.7 6.4 77.9 to 81.6
 Australia 3 3% 78.6 6.3 62.9 to 94.2
 Overall 111 100% 79.2 5.9 78.1 to 80.3

Table 2.

Calculated score value of PRISMA checklist

Item All journals (%) AJODO JO EJO AO
Identify the report as a systematic review. 100% 100% 100% 100% 100%
Abstract according to Prisma for abstract 52% 65% 33% 52% 50%
Rationale/Context of existing knowledge 100% 100% 100% 100% 100%
Objective/Questions addressed 92% 95% 89% 91% 93%
Eligibility criteria 94% 100% 89% 91% 96%
Information sources (databases/registers) 96% 90% 100% 96% 100%
Search strategy 90% 75% 89% 93% 96%
Selection process/screening 94% 75% 100% 98% 96%
Data collection process 94% 90% 100% 98% 86%
Define all outcomes 65% 75% 44% 65% 64%
Define other variables/missing data 58% 60% 56% 69% 36%
Risk of bias tool 97% 95% 100% 100% 93%
Effect measures 92% 70% 89% 98% 96%
Process used to include studies for synthesis 68% 60% 56% 69% 79%
Methods for preparing data for synthesis 64% 40% 56% 67% 79%
Presentation of included studies findings 91% 80% 100% 93% 93%
meta analysis methods 51% 50% 44% 54% 50%
Methods to assess heterogeneity 54% 35% 67% 56% 61%
methods of sensitivity analyses 94% 95% 100% 94% 89%
Reporting bias assessment due to missing data 33% 45% 44% 31% 25%
Certainty assessment 77% 90% 67% 74% 75%
Results of the search/number of records 95% 95% 100% 96% 89%
Excluded results/reasons 95% 85% 89% 100% 96%
Study characteristics 92% 90% 89% 93% 93%
Risk of bias report 100% 100% 100% 100% 100%
Results of individual studies 71% 85% 78% 70% 61%
characteristics/risk of bias for each synthesis 94% 75% 100% 100% 93%
Results of all statistical syntheses 95% 85% 100% 98% 93%
results of heterogeneity 75% 75% 89% 72% 75%
Results of sensitivity analyses 72% 60% 67% 72% 82%
Reporting biases in each synthesis due to missing data 76% 70% 78% 76% 79%
Certainty of evidence report for each outcome 59% 75% 56% 59% 46%
general interpretation 68% 85% 67% 65% 64%
Limitation of evidence included in the review 60% 80% 78% 56% 50%
Limitation of the review 59% 70% 67% 48% 71%
Implication of the results 52% 55% 44% 46% 64%
Registration number/or stated that it was not registered 100% 100% 100% 100% 100%
Access to protocol/or stated it was not prepared 88% 100% 89% 94% 68%
Explain amendments to protocol/registration 60% 60% 56% 56% 71%
Sources of financial or non financial support 93% 95% 89% 98% 82%
Declare any competing interests 100% 5% 11% 2% 4%
Which of the data are publicly available and where 67% 70% 56% 69% 64%

In this study, the compliance score was determined for all 111 systematic reviews evaluated, resulting in an overall score of 79.2% (95% CI: 78.1-80.3). Among the evaluated journals, the EJO achieved the highest compliance score, with a mean of 79.9% (95% CI: 78.3-81.5). The AO followed with a mean score of 78.9.1% (95% CI: 76.8-81.1). The AJODO obtained a mean score of 78.4% (95% CI: 75.1-81.8), while the JO achieved a mean score of 78.8% (95% CI: 74.9-82.7), both lower than EJO. However, the differences in compliance scores between the journals were not statistically significant, as indicated by the univariate analysis presented in Table 3. The mean compliance score varied across regions, with systematic reviews from Europe scoring the highest at 79.7% (95% CI: 77.9-81.6). Asia and South America achieved scores of 79.3% (95% CI: 77.0-81.5) and 78.1% (95% CI: 75.7-80.4), respectively. Although Europe had a higher compliance score compared to other continents, the difference was not statistically significant according to the univariate analysis.

Table 3.

Univariate linear regression derived coefficients (B) and 95% confidence interval with mean score of compliance with PRISMA as dependent variable for 111 SRs

Predictor variables Univariate analysis


Variable Category or unit B 95% CI
Journals EJO Baseline (reference)
AJODO -0.93 0.36-4.5
JO -0.49 0.62-5.3
AO -0.97 0.35-4.1
Continents Europe Baseline (reference)
Asia -0.03 -3.3-2.3
Africa -0.11 -13.5-13.44
North America -0.01 -4.5-4.2
South America -0.11 -4.8-1.4
Australia -0.33 -8.3-5.9
Year 2018 Baseline (reference)
2019 -0.06 -4.7-2.8
2020 0.05 -2.6-4.2
2021 0.04 -2.8-4.03
2022 0.06 -2.7-4.6
Number of authors 4-6 authors Baseline (reference)
Fewer than 4 0.08 -1.4-3.7
More than 6 0.2 0.156-6.68*

DISCUSSION

Systematic reviews serve the purpose of identifying gaps in scientific knowledge and delivering the most reliable evidence to address pertinent research questions. Maintaining the integrity of scientific inquiry requires following a series of steps that ensure the dependability of the findings.[1] Consequently, it is crucial to adhere to high reporting standards when documenting these steps. Reporting guidelines, such as PRISMA, have been devised to enhance the reporting quality of systematic reviews.[5] Furthermore, previous studies evaluating the reporting quality of research reports in various disciplines have contributed to this objective of evaluating and attempting to encourage reporting systematic reviews. The present study shares a similar aim: assessing the reporting status of systematic reviews in the orthodontics field.

The findings of this study suggest that the quality of reporting in systematic reviews has shown improvement over time. Currently, adherence to the PRISMA 2020 guidelines stands at nearly 80%. In comparison, a previous study conducted in 2013 examined 109 systematic reviews published between 2000 and 2011, evaluating them against the PRISMA guidelines from that time, and reported a score of 64%.[18] A closer examination of the current study and the study of Fleming et al. (2013)[18] work reveals substantial enhancements in specific areas, such as the inclusion of a systematic review description in the title (increasing from 63% to 100%) and the assessment of bias risk (improving from 26% to 100%). Despite disparities between the two studies, such as the publication dates of the included studies and the endorsement status of PRISMA by journals at the time of data collection, as well as variances stemming from the subjective nature of scoring the items, it is reasonable to observe an improvement in the reporting quality of systematic reviews in journals renowned for their stringent peer-review process in the orthodontics field.

Similarly, in a larger sample size, a previous study evaluated all published dental systematic reviews in 2017 and found a positive association between the use of PRISMA and improved reporting of clinical trials.[19] However, the authors emphasized the need to further elevate the standards of reporting for systematic reviews. Despite evident dissimilarities with the present study, such as the sample size and the scope of research for identifying relevant systematic reviews (limited to orthodontics and four prominent journals), as well as differences in methods for assessing reporting quality, it is evident that there is still room for enhancing the reporting quality of systematic reviews. Therefore, we propose conducting periodic assessments of reporting quality to inform stakeholders involved in scientific projects, including clinicians, editors, policymakers, and funding bodies. It is worth noting that there is a lack of studies that assess the quality of reporting the full text of systematic review reports in Orthodontics.

This study is constrained by the publication dates of the included studies (2018-2022) and the selection of four notable orthodontics journals as the publication platform. Consequently, the findings of this study may not be universally applicable to all systematic reviews in the field. Nevertheless, the observations made in this study contribute to an understanding of the current state of reporting in orthodontics. Another limitation of this study is the subjectivity involved in scoring each item. However, this potential bias was minimized by calibrating the authors before commencing the study and by directly referencing the PRISMA guidelines for clarification while scoring the included reviews.

CONCLUSION

The assessment of reporting quality in orthodontic systematic reviews published between 2018 and 2022 revealed a relatively high overall compliance with reporting guidelines (79.2%). The EJO obtained the highest compliance score, while certain items on the PRISMA checklist were reported inadequately in a significant portion of the papers. These findings underscore the importance of improving reporting transparency and completeness among authors, reviewers, and editors in order to enhance the reliability and usability of systematic reviews in orthodontics and support evidence-based practice.

Conflicts of interest

There are no conflicts of interest.

Acknowledgements

The authors appreciate the support from Prince Sattam Bin Abdulaziz University in conducting this research.

Funding Statement

Nil.

REFERENCES

  • 1.Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: Synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126:376–80. doi: 10.7326/0003-4819-126-5-199703010-00006. [DOI] [PubMed] [Google Scholar]
  • 2.Siddaway AP, Wood AM, Hedges LV. How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annu Rev Psychol. 2019;70:747–70. doi: 10.1146/annurev-psych-010418-102803. [DOI] [PubMed] [Google Scholar]
  • 3.Gurevitch J, Koricheva J, Nakagawa S, Stewart G. Meta-analysis and the science of research synthesis. Nature. 2018;555:175–82. doi: 10.1038/nature25753. [DOI] [PubMed] [Google Scholar]
  • 4.Hutton B, Salanti G, Caldwell DM, Shamseer L. Reporting guidance considerations from a statistical perspective: Overview of tools to enhance the rigour of reporting of randomised trials and systematic reviews. Evid Based Ment Health. 2017;20:46–52. doi: 10.1136/eb-2017-102666. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. Int J Surg. 2021;88:105906. doi: 10.1016/j.ijsu.2021.105906. [DOI] [PubMed] [Google Scholar]
  • 6.Sewell KA, Schellinger J, Bloss JE. Effect of PRISMA 2009 on reporting quality in systematic reviews and meta-analyses in high-impact dental medicine journals between 1993–2018. PLoS One. 2023;18:e0295864. doi: 10.1371/journal.pone.0295864. doi:10.1371/journal.pone. 0295864. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Nawijn F, Krijnen P, Huijsman R, Veen EJ, van der Waal RA, de Groot V. Quality of reporting of systematic reviews and meta-analyses in emergency medicine based on the PRISMA statement. BMC Emerg Med. 2019;19:1–8. doi: 10.1186/s12873-019-0233-6. doi:10.1186/s12873-019-0233-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Croitoru D, Asher G, Neagu N, Ioannides D, Nast A. Quality of reporting in systematic reviews published in dermatology journals. Br J Dermatol. 2020;182:1469–76. doi: 10.1111/bjd.18528. [DOI] [PubMed] [Google Scholar]
  • 9.Maticic K, Krnic Martinic M, Puljak L. Assessment of reporting quality of abstracts of systematic reviews with meta-analysis using PRISMA-A and discordance in assessments between raters without prior experience. BMC Med Res Methodol. 2019;19:1–7. doi: 10.1186/s12874-019-0675-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Pussegoda K, Turner L, Garritty C, Mayhew A, Skidmore B, Stevens A, et al. Systematic review adherence to methodological or reporting quality. Syst Rev. 2017;6:131. doi: 10.1186/s13643-017-0527-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Jin Y, Sanger N, Shams I, Luo C, Shahid H, Li G, et al. Does the medical literature remain inadequately described despite having reporting guidelines for 21 years?–A systematic review of reviews: An update. J Multidiscip Healthc. 2018;11:495–10. doi: 10.2147/JMDH.S155103. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Blanco D, Biggane AM, Cobo E, Moher D, Kirkham JJ. Scoping review on interventions to improve adherence to reporting guidelines in health research. BMJ Open. 2019;9:e026589. doi: 10.1136/bmjopen-2018-026589. doi:10.1136/bmjopen-2018-026589. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Peters JP, Hooft L, Grolman W, Stegeman I. Reporting quality of systematic reviews and meta-analyses of otorhinolaryngologic articles based on the PRISMA statement. PLoS One. 2015;10:e0136540. doi: 10.1371/journal.pone.0136540. doi:10.1371/journal.pone. 0136540. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Innocenti T, Giagheddu A, Peruzzi E, Casazza G, Agostini F, Tofani M. Adherence to the PRISMA statement and its association with risk of bias in systematic reviews published in rehabilitation journals: A meta-research study. Braz J Phys Ther. 2022;26:100450. doi: 10.1016/j.bjpt.2022.100450. doi:10.1016/j.bjpt. 2022.100450. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Tam WW, Lo KK, Khalechelvam P, Sunderland B. Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: A cross-sectional study. BMJ Open. 2017;7:e013905. doi: 10.1136/bmjopen-2016-013905. doi:10.1136/bmjopen-2016-013905. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Pidgeon TE, Lyu H, Mackenzie TA, Slavin SA, Lee BT. An assessment of the compliance of systematic review articles published in craniofacial surgery with the PRISMA statement guidelines: A systematic review. J Craniomaxillofac Surg. 2016;44:1522–30. doi: 10.1016/j.jcms.2016.07.018. [DOI] [PubMed] [Google Scholar]
  • 17.Lee S-Y, Park JH, Nam SW, Lee SH, Kim Y-H, Park SJ. Compliance of systematic reviews in ophthalmology with the PRISMA statement. BMC Med Res Methodol. 2017;17:1–7. doi: 10.1186/s12874-017-0450-1. doi:10.1186/s12874-017-0450-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Fleming PS, Seehra J, Polychronopoulou A, Fedorowicz Z, Pandis N. A PRISMA assessment of the reporting quality of systematic reviews in orthodontics. Angle Orthod. 2013;83:158–63. doi: 10.2319/032612-251.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Bassani R, Candido-Silva JA, Santiago-Júnior JF, Fronza F, Fatturi-Parolo CC. Systematic reviews in dentistry: Current status, epidemiological and reporting characteristics. J Dent. 2019;82:71–84. doi: 10.1016/j.jdent.2019.01.014. [DOI] [PubMed] [Google Scholar]

Articles from Journal of Pharmacy & Bioallied Sciences are provided here courtesy of Wolters Kluwer -- Medknow Publications

RESOURCES