The quality of systematic reviews about interventions for refractive error can be improved: a review of systematic reviews

Abstract

Background

Systematic reviews should inform American Academy of Ophthalmology (AAO) Preferred Practice Pattern® (PPP) guidelines. The quality of systematic reviews related to the forthcoming Preferred Practice Pattern® guideline (PPP) Refractive Errors & Refractive Surgery is unknown. We sought to identify reliable systematic reviews to assist the AAO Refractive Errors & Refractive Surgery PPP.

Methods

Systematic reviews were eligible if they evaluated the effectiveness or safety of interventions included in the 2012 PPP Refractive Errors & Refractive Surgery. To identify potentially eligible systematic reviews, we searched the Cochrane Eyes and Vision United States Satellite database of systematic reviews. Two authors identified eligible reviews and abstracted information about the characteristics and quality of the reviews independently using the Systematic Review Data Repository. We classified systematic reviews as “reliable” when they (1) defined criteria for the selection of studies, (2) conducted comprehensive literature searches for eligible studies, (3) assessed the methodological quality (risk of bias) of the included studies, (4) used appropriate methods for meta-analyses (which we assessed only when meta-analyses were reported), (5) presented conclusions that were supported by the evidence provided in the review.

Results

We identified 124 systematic reviews related to refractive error; 39 met our eligibility criteria, of which we classified 11 to be reliable. Systematic reviews classified as unreliable did not define the criteria for selecting studies (5; 13%), did not assess methodological rigor (10; 26%), did not conduct comprehensive searches (17; 44%), or used inappropriate quantitative methods (3; 8%). The 11 reliable reviews were published between 2002 and 2016. They included 0 to 23 studies (median = 9) and analyzed 0 to 4696 participants (median = 666). Seven reliable reviews (64%) assessed surgical interventions.

Conclusions

Most systematic reviews of interventions for refractive error are low methodological quality. Following widely accepted guidance, such as Cochrane or Institute of Medicine standards for conducting systematic reviews, would contribute to improved patient care and inform future research.

Electronic supplementary material

The online version of this article (10.1186/s12886-017-0561-9) contains supplementary material, which is available to authorized users.

Background

Systematic reviews of interventions are used to inform clinical guidelines [1], to set research priorities [2], and to help patients and clinicians make healthcare decisions. Well-conducted systematic reviews focus on clear research questions and use reproducible methods to identify, select, describe, and synthesize information from relevant studies [3]. Such reviews can reduce uncertainty about the effectiveness of interventions, lead to faster adoption of safe and effective interventions, and identify research needs. On the other hand, poorly-conducted systematic reviews may be harmful if they contribute to suboptimal patient care or promote unnecessary research.

Each year, the number of published systematic reviews continues to increase throughout medicine, yet the quality of systematic reviews is highly variable [2, 4]. To be considered reliable, systematic reviews must be conducted using methods that minimize bias and error in the review process, and must be reported completely and transparently [3].

Because refractive error is the leading cause of visual impairment globally [5], effective interventions to correct refractive error are important to patients and to optometrists and ophthalmologists. The American Academy of Ophthalmology (AAO) updated their Preferred Practice Pattern (PPP) for Refractive Errors & Refractive Surgery in 2017. To assist the AAO in this task, Cochrane Eyes and Vision @ United States (CEV@US) investigators sought to identify reliable systematic reviews about interventions for refractive error using a database of systematic reviews in eyes and vision [6].

Methods

Systematic reviews were eligible for this project if they evaluated the effectiveness or safety of interventions for refractive error that were included in the 2012 AAO’s PPP Refractive Errors & Refractive Surgery [7]. We excluded systematic reviews related to cataract removal with implantation of an intraocular lens for correcting refractive error because this topic is covered in another PPP. We included all reports that claimed to be systematic reviews. Otherwise, we defined a systematic review as “a scientific investigation that focuses on a specific question and uses explicit, pre-specified scientific methods to identify, select, assess, and summarize similar but separate studies [8].” Consistent with this definition, eligible studies were not required to include meta-analyses. Whenever a systematic review had been updated since the initial publication, we reviewed the most recent update.

CEV@US maintains a database of systematic reviews related to vision research and eye care (see Online Additional file 1 for search strategies used to identify reviews). We conducted an initial search of PubMed and Embase in 2007, and we updated the search in 2009, 2012, 2014, and on May 15, 2016 [9]. Two individuals identified systematic reviews related to vision research and eye care. For the 2007, 2009, and 2012 searches, two people identified relevant refractive error systematic reviews independently. For the 2014 and 2016 searches, one person identified potentially eligible refractive error reviews, then a second person verified that the reports were eligible. Differences were resolved through discussion that sometimes included another team member.

We developed a data abstraction form that included components of the Critical Appraisal Skills Programme (CASP) [10], the Assessment of Multiple Systematic Reviews (AMSTAR) [11], and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [12]. We adapted a version of the form used in previous studies [4, 13] (see Online Additional file 2 for a copy of the electronic form). In addition to the results presented in this paper, we recorded other descriptive information about the reviews. We entered data electronically using the Systematic Review Data Repository (SRDR) [14].

Two authors (EMW and SN) abstracted data independently from eligible systematic reviews and resolved discrepancies through discussion. We classified systematic reviews as “reliable” when the systematic reviewers had (1) defined criteria for the selection of studies, (2) conducted comprehensive literature searches for eligible studies, (3) assessed the methodological quality (risk of bias) of the included studies, (4) used appropriate methods for meta-analyses (which we assessed only when meta-analyses were reported), and (5) presented conclusions that were supported by the evidence provided in the review. We considered a systematic review “unreliable” when one or more of these criteria were not met.

Results

Of 124 systematic reviews in our database and classified as related to refractive error, 39 met our eligibility criteria (Fig. 1). Systematic reviews that met our inclusion criteria were published between 1996 and 2014 (median = 2011). They included between 0 and 309 studies (median = 9.5). In 13/39 (33%) systematic reviews, we could not ascertain how many participants contributed data; the 26 remaining reviews included data from 0 to 9336 participants.

Fig. 1.

Flow chart showing the identification of systematic reviews of interventions for refractive error

Of the 39 systematic reviews, 22 (56%) assessed surgical interventions, including reviews of LASEK or LASIK (18; 45%) and intraocular lenses (4; 10%). Other reviews assessed orthokeratology (10; 26%), atropine (2; 5%), monovision (1; 3%), contact lenses (1; 3%), spectacles (1; 3%), and multiple interventions (2; 5%). Participants with myopia were included in 30 (77%) reviews. Reviews also included participants with astigmatism (15; 38%), hyperopia (10; 26%), and presbyopia (3; 8%).

Of the 39 eligible systematic reviews, 11 (28%) were classified as reliable [15–25], and 28 (72%) were classified as unreliable (Fig. 2) [26–53]. One of the reliable reviews did not include any studies. All 11 reliable systematic reviews assessed interventions for myopia, either to slow progression of myopia in children (5 reviews), to correct myopia surgically (5 reviews), or to treat choroidal neovascularization secondary to pathologic myopia (1 review). The 11 reliable systematic reviews were published between 2002 and 2016, included data from 0 to 23 studies (median = 9), and analyzed data for 0 to 4696 participants (median = 666). Of the 11 systematic reliable reviews, 7 (64%) assessed surgical interventions, and included some of the same studies. Different groups of reviewers reached contradictory conclusions about the safety and effectiveness of LASEK compared with PRK (Table 1).

Fig. 2.

Assessment of reliability criteria for 40 systematic reviews on interventions for refractive error

Table 1.

Objectives, Participants, Interventions, Outcomes and Conclusions of the Reliable Systematic Reviews on Interventions for Refractive Error and Refractive Surgery (N = 11)

Study ID	Objective(s)	Condition(s)	Intervention Comparisons	Outcome	Number of Studies; Participants; Eyes	Conclusion(s) from the abstract
Pharmaceuticals
Li 2014 [17]	“To conduct a meta-analysis on the effects of atropine in slowing myopia progression and to compare Asian and white children and randomized controlled trials (RCTs) and observational studies.”	Myopia (children)	Atropine compared with placebo or non-atropine treatment	Refractive error	11; 1815; Not reported	“Atropine could significantly slow myopia progression in children, with greater effects in Asian than in white children. Randomized controlled trials and cohort studies provided comparable effects.”
Spectacles
Li 2011 [22]	“Multifocal lenses (MLs) are advocated as a substitute for single vision lenses (SVLs) to slow myopia progression in children, but results vary greatly across studies.”	Myopia (children)	Multifocal lenses compared with single vision lenses	Visual acuity, axial length	9; 1464; Not reported	“A meta-analysis of nine of these trials showed that MLs with powers ranging from þ1.50 to þ2.00D were associated with a statistically significantly decrease in myopia progression in school-aged children compared with SVLs. The benefit was greater in children with a higher level of myopia at baseline and sustained for a minimum of 24 months. Asian children appeared to have greater benefit from intervention with MLs than white children.”
Surgery
*Barsam 2014 [15]	“To compare excimer laser refractive surgery and phakic IOLs for the correction of moderate to high myopia by evaluating postoperative uncorrected visual acuity, refractive outcome, potential loss of best spectacle corrected visual acuity (BSCVA) and the incidence of adverse outcomes.”	Myopia	Phakic intraocular lenses compared with excimer laser surgical	Visual acuity (UCVA); Need for correction; Patient satisfaction; Quality of life; Cost	3; 132; 228	“The results of this review suggest that, at one year post surgery, phakic IOLs are safer than excimer laser surgical correction for moderate to high myopia in the range of −6.0 to −20.0 D and phakic IOLs are preferred by patients. While phakic IOLs might be accepted clinical practice for higher levels of myopia (greater than or equal to 7.0 D of myopic spherical equivalent with or without astigmatism), it may be worth considering phakic IOL treatment over excimer laser correction for more moderate levels of myopia (less than or equal to 7.0 D of myopic spherical equivalent with or without astigmatism). Further RCTs adequately powered for subgroup analysis are necessary to further elucidate the ideal range of myopia for phakic IOLs. This data should be considered alongside comparative data addressing long-term safety as it emerges.”
*Li 2016 [16]	“The objective of this review is to compare LASEK versus PRK for correction of myopia by evaluating their efficacy and safety in terms of postoperative uncorrected visual acuity, residual refractive error, and associated complications.”	Myopia; Astigmatism	LASEK compared with PRK	Visual acuity (UCVA, BCVA); Refractive error; Adverse events	11; 428; 866	“Uncertainty surrounds differences in efficacy, accuracy, safety, and adverse effects between LASEK and PRK for eyes with low to moderate myopia. Future trials comparing LASEK versus PRK should follow reporting standards and follow correct analysis. Trial investigators should expand enrollment criteria to include participants with high myopia and should evaluate visual acuity, refraction, epithelial healing time, pain scores, and adverse events.”
*Settas 2012 [19]	“The objectives of this review were to determine whether PRK or LASIK leads to more reliable, stable and safe results when correcting a hyperopic refractive error.”	Hyperopia; Astigmatism	LASIK compared with PRK	Visual acuity (UCVA, BCVA); Refractive error; Adverse events	0; 0; 0	“No robust, reliable conclusions could be reached, but the non-randomised trials reviewed appear to be in agreement that hyperopic-PRK and hyperopic-LASIK are of comparable efficacy. High quality, well-planned open RCTs are needed in order to obtain a robust clinical evidence base.”
Shortt 2006 [23]	“The aim of this review was to compare the effectiveness and safety of PRK and LASIK for correction of myopia.”	Myopia; Astigmatism	LASIK compared with PRK	Visual acuity (UCVA); Refractive error; Quality of life; Adverse events	6; 666; 417	“LASIK gives a faster visual recovery than PRK but the effectiveness of these two procedures is comparable. Further trials using contemporary techniques are required to determine whether LASIK and PRK are equally safe.”
*Shortt 2013 [18]	“To compare the effectiveness and safety of LASIK and PRK for correction of myopia by examining post-treatment uncorrected visual acuity, refractive outcome, loss of best spectacle-corrected visual acuity, pain scores, flap complications in LASIK, subepithelial haze, adverse events, quality of life indices and higher order aberrations.”	Myopia; Astigmatism	LASIK compared with PRK	Visual acuity (UCVA, BCVA); Refractive error; Quality of life; Adverse events	13; 1135; 1923	“LASIK gives a faster visual recovery and is a less painful technique than PRK. The two techniques appear to give similar outcomes one year after surgery. Further trials using contemporary techniques are required to determine whether LASIK and PRK as currently practised are equally safe. Randomising eyes to treatment is an efficient design, but only if analysed properly. In future trials, more efforts could be made to mask the assessment of outcome.”
Virgili 2005 [24]	“The primary objective of this review was to examine the effects of laser photocoagulation for CNV associated with pathologic myopia. A secondary objective was to compare the effects of different photocoagulation techniques.”	Myopia	Laser photocoagulation compared with no treatment or sham treatment	Visual acuity; Quality of life; Functioning; Adverse events	2; 96; 97	“Despite its use over several years the effectiveness of laser photocoagulation for myopic CNV has not been established. Although there was a suggestion of short-term effectiveness in one small study on non-subfoveal CNV the results were potentially biased. Observational studies suggest that the enlargement of the atrophic laser scar after laser treatment of non-subfoveal CNV could be a potentially vision- threatening long-term complication, even in eyes free of CNV recurrence.”
Multiple interventions
Saw 2002 [25]	“To evaluate the efficacy of interventions such as eyedrops, bifocal lenses, or contact lenses in retarding the progression of myopia in myopic children.”	Myopia (children)	“Interventions to retard the progression of myopia”	Adverse events	10; 1612; Not reported	“The latest evidence from randomized clinical trials does not provide sufficient information to support interventions to prevent the progression of myopia. Long-term large-scale double-masked randomized clinical trials, including cycloplegic refraction, are needed before any recommendations about interventions in clinical practice to prevent high myopia in myopic children are considered.”
*Walline 2011 [20]	“To assess the effects of several types of interventions, including eye drops, undercorrection of nearsightedness, multifocal spectacles and contact lenses, on the progression of nearsightedness in myopic children younger than 18 years. We compared the interventions of interest with each other, to single vision lenses (SVLs) (spectacles), placebo or no treatment.”	Myopia (children)	Bifocal soft contact lenses (BSCLs), rigid gas permeable contact lenses (RGPCLs) and corneal reshaping (orthokeratology) contact lenses; Bifocal lenses (spectacles), progressive addition lenses (PALs) and undercorrection of myopia; Pharmaceutical agents	Refractive error; Quality of life; Adverse events; Cost	23; 4696; Not reported	“The most likely effective treatment to slow myopia progression thus far is anti-muscarinic topical medication. However, side effects of these medications include light sensitivity and near blur. Also, they are not yet commercially available, so their use is limited and not practical. Further information is required for other methods of myopia control, such as the use of corneal reshaping contact lenses or bifocal soft contact lenses (BSCLs) with a distance center are promising, but currently no published randomized clinical trials exist.”
Other interventions
*Wei 2011 [21]	“To assess the effectiveness and safety of acupuncture in slowing the progression of myopia in children and adolescents.”	Myopia (children and adolescents)	Acupuncture	Refractive error; Adverse events; Cost; Axial length; Corneal radius	2; 131; Not reported	“Two trials are included in this review but no conclusions can be drawn for the benefit of co-acupressure for slowing progress of myopia in children. Further evidence in the form of RCTs are needed before any recommendations can be made for the use of acupuncture treatment in clinical use. These trials should compare acupuncture to placebo and have large sample sizes. Other types of acupuncture (such as auricular acupuncture) should be explored further as well as compliance with treatment for at least six months or longer. Axial length elongation of the eye should be investigated for at least one year. The potential to reduce/eliminate pain from acupuncture experienced by children should also be reviewed.”

*Cochrane review

The 28 systematic reviews that we classified as unreliable were published between 1996 and 2016 (see Online Additional file 3 for the data extracted for all included systematic reviews) [26–53]. Unreliable systematic reviews did not define the criteria for selecting studies (5; 13%), did not assess methodological quality of the individual studies (10; 26%), did not conduct comprehensive literature searches for eligible studies (17; 44%), or used inappropriate quantitative methods, such as combining randomized and non-randomized studies for meta-analysis (3; 8%). All systematic reviews that presented conclusions not supported by the data were also classified as unreliable for another reason (20; 51%).

Discussion

Our investigation revealed that most systematic reviews about interventions for refractive error published to date have been of low methodological quality; consequently, these reviews may result in inappropriate decisions about clinical care and future research. Some of the shortcomings we identified were related to reporting deficiencies. For example, authors of most reviews did not report criteria for selecting studies. Also, many authors did not report essential characteristics of the included studies, such as sample sizes. Other shortcomings were related to both poor reporting and poor conduct. Our findings are thus consistent with evidence that most systematic reviews published to date have been neither conducted nor reported following best practices [54, 55].

Although visual impairment due to uncorrected refractive error is particularly prevalent in low-income countries [5], most of the systematic reviews included in our study addressed interventions not widely available in those countries. Systematic reviewers may have focused on interventions of interest to decision-makers in high-income countries, and there may be few randomized trials of interventions suitable for use in low-income countries and thus available for inclusion in systematic reviews. Although we classified the systematic review reported by Pearce [29] as unreliable, it is the exception among the 40 systematic reviews for having dealt with an intervention with wide applicability.

A comprehensive and reproducible literature search lays the foundation for a high-quality systematic review. To be transparent and reproducible, systematic reviews should list all information sources searched, the exact search terms used, the operators used to combine terms, and the dates of the searches [56]; however, many systematic reviews of interventions for refractive error did not describe the search methods for identifying eligible studies. Moreover, the number of citations retrieved in many systematic reviews was so small that a knowledgeable reader might suspect the search methods were not sensitive. As a rule of thumb, systematic reviewers can expect to retrieve and review about 200 citations (titles and abstracts) and 5 full-text reports to identify one study eligible for inclusion in a systematic review [57].

Reviewers should describe the methodological quality of included studies using a method such as the Cochrane Risk of Bias Tool to identify potential sources of bias that could influence the credibility of results of individual studies and hence systematic reviews and meta-analysis [58, 59]. Many reviews did not mention having assessed the methodological quality of individual studies, and some reviews specified methods that are known to be unreliable (e.g., the Jadad Scale) [60]. Quality scores are problematic because they are inconsistent and lack validity; the same study may be assessed as excellent using one scoring method but poorly using another scoring method [61].

Finally, authors of seven systematic reviews used methods for data synthesis that were inappropriate. Of particular concern are reviews in which outcomes from randomized controlled trials were combined with outcomes from observational studies in the same meta-analysis. For a research question on the effectiveness of an intervention, randomized controlled trials, by design, have more protection against bias than observational studies. When both types of studies are available, systematic reviewers should justify the reasons for relying on observational data as a substitute for or a complement to data from randomized trials. It is important to understand and to appraise the strengths and weaknesses of observational data when they are used to assess the effectiveness of interventions. When dissimilar studies are combined in meta-analysis, the resulting estimates of intervention effects might be meaningless.

Systematic reviews about interventions for refractive error assessed only a few distinct interventions, a finding consistent with other evidence that systematic reviews are duplicative throughout medicine [62]. After completing our review and sharing the results with the AAO PPP, the panel informed us that one reliable systematic review was not relevant because it examined choroidal neovascularization and laser treatment in high myopia, which is considered a retina topic [24]. Furthermore, some of the reliable reviews included only a few small studies and thus provided inconclusive evidence about the effectiveness and safety of the targeted interventions. Although it is possible for reviews to use pre-specified methods and reach different conclusions (e.g., because of differences in outcomes or inclusion criteria), we find it concerning that some of the reliable reviews reached inconsistent conclusions about the same interventions, even though they included some of the same studies.

We searched for systematic reviews using PubMed and Embase, and we might have missed systematic reviews that are not indexed in these databases. It seems unlikely that systematic reviews in non-indexed journals would be of better quality than the reviews we identified, and their inclusion in our study would have been unlikely to lead us a different conclusion. More high-quality evidence is needed to inform clinical guidelines and to improve patient care for this common problem.

Journal editors could help improve the quality of systematic reviews by adopting four requirements. First, editors should require that authors of systematic reviews publish or provide protocols for their reviews that adhere to current best practices (e.g., the Preferred Reporting Items for Systematic Review and Meta-analysis Protocols (PRISMA-P) statement [63]). Second, editors should require that all reports of systematic reviews include the information in the PRISMA statement [64], which should be documented in a completed PRISMA checklist that accompanies every systematic review manuscript and is confirmed by a peer reviewer or a member of the editorial team. By considering only those manuscripts that adhere to these requirements, editors could ensure that readers have the information needed to assess the methodological rigor of systematic reviews and to decide whether to apply their findings to practice. Third, to reduce the publication of reviews that are essentially redundant (i.e., reviews that include the same studies), editors should require that authors explain how their systematic reviews differ from other reviews of the same interventions or, if the systematic reviews are not different, editors should require that authors reference existing systematic reviews about the same topic. Finally, manuscripts that report systematic reviews should be reviewed by people knowledgeable about both systematic reviews and the clinical specialties that the reviews concern. Eight ophthalmology and optometry journals now have editors for systematic reviews, which may improve the quality of reviews in those journals (http://eyes.cochrane.org/associate-editors-eyes-and-vision-journals).

Conclusions

In conclusion, systematic reviews about interventions for refractive error could benefit patient care and avoid redundant research if the authors of such reviews followed widely accepted guidance, such as Cochrane or Institute of Medicine standards for conducting systematic reviews [3, 65]. The reliable systematic reviews we identified may be useful in updating the AAO PPP and in future updates of the American Optometric Association’s Evidence-Based Clinical Practice Guidelines.

Authors’ contributions

TL conceived the study. TL and EMW developed the data extraction form. EMW and SN extracted the data. EMW analyzed the data and drafted the manuscript. EMW, SM, RC, and TL contributed to interpreting the results, reviewed the manuscript, provided critical revisions, and approved the manuscript for publication.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

EMW, SMN, TL have no financial or other interests. RSC chairs the refractive error PPP panel and is a member on the AAO PPP steering committee.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.