Interventions for Age-Related Macular Degeneration: Are Practice Guidelines Based on Systematic Reviews?

Abstract

Topic

Are existing systematic reviews of interventions for age-related macular degeneration incorporated into clinical practice guidelines?

Clinical relevance

High-quality systematic reviews should be used to underpin evidence-based clinical practice guidelines and clinical care. We have examined the reliability of systematic reviews of interventions for age-related macular degeneration (AMD) and described the main findings of reliable reviews in relation to clinical practice guidelines.

Methods

Eligible publications are systematic reviews of the effectiveness of treatment interventions for AMD. We searched a database of systematic reviews in eyes and vision and employed no language or date restrictions; the database is up-to-date as of May 6, 2014. Two authors independently screened records for eligibility and abstracted and assessed the characteristics and methods of each review. We classified reviews as “reliable” when they reported eligibility criteria, comprehensive searches, appraisal of methodological quality of included studies, appropriate statistical methods for meta-analysis, and conclusions based on results. We mapped treatment recommendations from the American Academy of Ophthalmology Preferred Practice Patterns (AAO PPP) for AMD to the identified systematic reviews and assessed whether any reliable systematic review was cited or could have been cited to support each treatment recommendation.

Results

Of 1,570 systematic reviews in our database, 47 met our inclusion criteria. Most of the systematic reviews targeted neovascular AMD and investigated anti-vascular endothelial growth factor (anti-VEGF) interventions, dietary supplements or photodynamic therapy. We classified over two-thirds (33/47) of the reports as reliable. The quality of reporting varied, with criteria for reliable reporting met more often for Cochrane reviews and for reviews whose authors disclosed conflicts of interest. Although most systematic reviews were reliable, anti-VEGF agents and photodynamic therapy were the only interventions identified as effective by reliable reviews. Of 35 treatment recommendations extracted from the AAO PPP, 15 could have been supported with reliable systematic reviews; however, only one recommendation had an accompanying intervention systematic review citation, which we assessed as a reliable systematic review. No reliable systematic review was identified for 20 treatment recommendations, highlighting areas of evidence gaps.

Conclusions

For AMD, reliable systematic reviews exist for many treatment recommendations in the AAO PPP and should be used to support these recommendations. We also identified areas where no high-level evidence exists. Mapping clinical practice guidelines to existing systematic reviews is one way to highlight areas where evidence generation or evidence synthesis is either available or needed.

Introduction

Age-related macular degeneration (AMD) is the leading cause of severe vision loss in people over age 65 in industrialized countries.^1,2 This disease can be divided into two basic subtypes: neovascular (“wet AMD”) and non-neovascular (“dry AMD”). Neovascular AMD is characterized by choroidal neovascularization (CNV), in which formation of abnormal blood vessels leads to sub- and intra-retinal macular edema, hemorrhage, and/or fibrosis causing rapid central vision loss. In non-neovascular AMD, because of the gradual loss of photoreceptors and development of geographic atrophy, vision decreases slowly over many years. With no effective treatment available, patients with non-neovascular AMD are usually followed to detect and treat complications, such as development of neovascular AMD.

For decades, laser photocoagulation was the only available treatment for neovascular AMD, yet other treatments have been the subject of research, including radiotherapy, interferon alpha, and photodynamic therapy, of which photodynamic therapy received regulatory approval in April 2000.³ More recently, treatments focusing on the neutralization of vascular endothelial growth factor (VEGF) by injecting antibodies (bevacizumab), antibody fragments (ranibizumab), or fusion proteins (aflibercept) into the vitreous of the eye have become the current standard of care for neovascular AMD.⁴

Systematic reviews are summaries of the best research evidence available to address a specific question and follow explicit eligibility criteria and methods.⁵ Because systematic reviews underpin evidence-based clinical practice guidelines, it is important that they are trustworthy and at low risk of bias, yet we know that this is not always the case.⁶ For example, an author who has a potential conflict of interest may influence research conclusions,⁷ or multiple reviews on the same topic may represent unnecessary duplication of effort and prove confusing if the review authors reach different conclusions. Some reasons for differing conclusions are understandable, for example when the studies synthesized in systematic reviews were conducted at dissimilar time periods or included different types of study designs.⁸ But sometimes differing conclusions can be ascribed to use of systematic review methods that are potentially subject to bias.⁹

Best practice for the development of clinical practice guidelines involves the integration of high quality systematic reviews.⁶ To accomplish this goal, guideline developers can elect to undertake a systematic review in-house, commission a third party to conduct a systematic review, use results from previously completed systematic reviews, or implement a combination of these methods.

The objectives of this study were to 1) identify all published systematic reviews in the area of eyes and vision that had examined the treatment of AMD, 2) assess the reliability of existing reviews, and 3) map clinical practice guideline recommendations to reliable systematic reviews in order to encourage the integration of reliable systematic reviews and clinical practice guideline recommendations.

Methods

Identification of systematic reviews of interventions for AMD

The search strategies and definition used for systematic reviews have been published.^10,11 Our searches employed no language or date restrictions and were up-to-date as of May 6, 2014. Systematic reviews eligible for the current study had examined interventions for AMD; we excluded reviews concerned only with AMD etiology, diagnosis, prognosis, and cost-effectiveness of treatment. Furthermore, to be eligible, reports of systematic reviews had to be full-text journal articles representing “a scientific investigation that addressed a focused question and used explicit, pre-specified scientific methods to identify, select, assess, and summarize similar but separate studies.”^5,12 Systematic reviews were eligible regardless of whether meta-analyses were performed; however, we considered articles that described a meta-analysis only, without a systematic review component, ineligible, because we could not be sure they were based on a systematic review. For eligible reviews with multiple published versions, such as updated or co-published Cochrane reviews, we included the most recent publication.

We used a two-stage screening process to identify eligible systematic reviews. First, two individuals independently screened the titles and abstracts of all 1,570 reviews listed in our database of systematic reviews in eyes and vision as of May 6, 2014. Next, for all records classified as potentially relevant, two individuals reviewed each full-text report independently for eligibility. We resolved discrepancies at each stage through discussion.

Assessment of systematic reviews of interventions for AMD

For each eligible systematic review, two individuals independently abstracted data from the review onto an electronic data collection form developed, pilot-tested, and maintained in the Systematic Review Data Repository (SRDR).¹³ This form was adapted from components of the Critical Appraisal Skills Programme (CASP),¹⁴ the Assessment of Multiple Systematic Reviews (AMSTAR),¹⁵ and the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA);¹⁶ we have used it in other studies.^9,17 We extracted data related to review objectives, populations, interventions, outcomes, methods (e.g., eligibility criteria for selection of studies for the systematic review, search strategies for eligible studies, assessment of risk of bias in included studies), results, conclusions, and financial support. When a meta-analysis was conducted, we also abstracted data on the statistical methods used. We resolved any discrepancy in data abstraction through discussion.

Based on previously published criteria,⁹ and standard systematic review methodology,^5,6,14–16 we classified reviews as reliable when they reported (1) defined criteria for selection of studies, (2) comprehensive searches for eligible studies, (3) assessment of risk of bias in included studies, (4) appropriate statistical methods for meta-analysis, and (5) agreement between the results and conclusions. We considered searches to be comprehensive when three or more bibliographic databases were searched, at least one method of other searching was employed (e.g., handsearching conference abstracts, identifying ongoing trials, screening reference lists of included studies), and search results were not limited to English-language only.⁵ When one or more of these criteria were not met, we classified reviews as being unreliable.

We conducted descriptive analyses of review characteristics and estimated proportions of reliable reviews. We conducted a pre-specified subgroup analysis by whether the systematic review was a Cochrane review. Further, we explored characteristics of systematic reviews when more than one addressed the same research question.

Mapping clinical practice guideline recommendations to systematic review evidence

We extracted treatment recommendations from the 2015 American Academy of Ophthalmology Preferred Practice Patterns (AAO PPP) on management of AMD.¹⁸ We included only recommendations related to the effectiveness of treatment interventions (i.e., recommendations related to diagnosis and follow-up were excluded) and recorded the section of the AAO PPP where we found each recommendation.

We mapped the treatment recommendations to systematic reviews identified by our study and assessed whether reliable systematic reviews were available to address each treatment recommendation and, if so, whether they were cited by the AAO PPP. We also assessed whether sources of evidence were provided with each treatment recommendation and, when provided, categorized each cited reference as a systematic review, randomized controlled trial, or other study type.

Results

Description of search results

Of 1,570 systematic reviews in our database as of May 6, 2014, 47 systematic reviews met our eligibility criteria (Figure 1).^19–65

Figure 1. Identification of systematic reviews (SRs) of interventions for age-related macular degeneration (AMD) as of 6 May 2014.

Characteristics of systematic reviews of AMD

The earliest eligible AMD systematic review identified was published in 2001 (Table 1). More than half (26/47; 55%) of the AMD systematic reviews focused on neovascular disease. The most commonly investigated interventions were anti-VEGF agents (15/47; 32%), dietary supplements (9/47; 19%), and photodynamic therapy (6/47; 13%). A majority of systematic reviews examined the effect of treatment on visual acuity (32/47; 68%) and safety (37/47; 79%); almost half assessed quality of life as outcomes of interest (23/47; 49%).

Table 1.

Characteristics of systematic reviews on interventions for age-related macular degeneration (AMD)

	Reliability of review
	All systematic reviews (N=47)		Reliable (n=33)		Unreliable (n=14)
Year published (median; range)	2009	(2001–2014)	2009	(2002–2014)	2009	(2001–2013)
A. Eligibility criteria	n	%	n	%	n	%
Participants
Neovascular AMD	26	55.3	20	60.6	6	42.9
Any AMD	11	23.4	6	18.2	5	35.7
General population	8	17.0	5	15.2	3	21.4
Non-neovascular	1	2.1	1	3.0	0	0.0
Early AMD	1	2.1	1	3.0	0	0.0
Interventions examined
Anti-VEGF1 vs. anti-VEGF or PDT or placebo	15	31.9	10	30.3	5	35.7
Dietary supplement vs. dietary supplement or placebo	9	19.1	7	21.2	2	14.3
Photodynamic therapy vs. placebo or no treatment	6	12.8	5	15.2	1	7.1
Submacular surgery vs. no treatment	3	6.4	2	6.1	1	7.1
Health or rehabilitation intervention vs. no intervention	3	6.4	1	3.0	2	14.3
Other comparison	11	23.4	8	24.2	3	21.4
Outcomes examined2
Visual acuity	32	68.1	25	75.8	7	50.0
Safety (e.g., cardiovascular events)	37	78.7	28	84.8	9	64.3
Quality of life	23	48.9	20	60.6	3	21.4
Contrast sensitivity	14	29.8	13	39.4	1	7.1
Visual function	8	17.0	7	21.2	1	7.1
Cost	11	23.4	7	21.2	4	28.6
Development or progression of AMD	10	21.3	7	21.2	3	21.4
Studies designs examined2
Randomized controlled trials	40	85.1	32	97.0	8	57.1
Controlled clinical trials	10	21.3	8	24.2	2	14.3
Other study designs	15	31.9	7	21.2	8	57.1
B. Systematic review publication characteristics
Publication type
Cochrane Library	15	31.9	15	45.5	0	0.0
Other peer review journal	25	53.2	14	42.4	11	78.6
Government or insurance agency report	7	14.9	4	12.1	3	21.4
Language
English	41	87.2	31	93.9	10	71.4
Non-English	6	12.8	2	6.1	4	28.6
Number of authors
1	4	8.5	3	9.1	1	7.1
2	9	19.1	7	21.2	2	14.3
3 or more	34	72.3	23	69.7	11	78.6
C. Search for studies
Databases searched2
MEDLINE (PubMed)	47	100.0	33	100.0	14	100.0
Cochrane Central Register	40	85.1	31	93.9	9	64.3
EMBASE	38	80.9	32	97.0	6	42.9
LILACS	11	23.4	11	33.3	0	0.0
Other databases	27	57.4	18	54.5	9	64.3
Median # of databases searched (interquartile range)	4	(3–5)	4	(3–5)	3	(1.75–5)
Search restrictions
No restriction in language of studies	28	59.6	23	69.7	5	35.7
No restriction in years searched for at least one database	31	66.0	25	75.8	6	42.9
Other sources searched2
Reference lists, reports that cited the study, or both	36	76.6	30	90.9	6	42.9
Experts in the field and/or study authors	22	46.8	17	51.5	5	35.7
Hard-to-access or unpublished studies	24	51.1	20	60.6	4	28.6
Ongoing studies	19	40.4	19	57.6	0	0.0
D. Results of systematic reviews2
Median number of studies included (interquartile range)	7	(2–14)	5	(2–12.75)	11	(7.25–35)
Median number of participants included (interquartile range)	1,480	(505–4,414)	948	(339–2,505)	4,052	(1,560–82,941)
Qualitative synthesis performed3	38	88.4	28	96.6	10	71.4
One or more meta-analysis presented3	22	51.2	16	55.2	6	42.9
Statistical heterogeneity assessed4	19	86.4	16	100.0	3	50.0
E. Funding and conflicts of interest	n	%	n	%	n	%
Funding sources
Funding reported2	31	66.0	22	66.7	9	64.3
Government	18	58.1	13	39.4	5	35.7
Department/Institution	10	32.3	9	27.3	1	7.1
Industry	4	12.9	0	0.0	4	28.6
Foundation	3	9.7	3	9.1	0	0.0
Other sources	2	6.5	2	6.1	0	0.0
Explicitly stated no funding	1	3.2	1	3.0	0	0.0
Funding not reported	16	34.0	11	33.3	5	35.7
Conflict of interest
Conflict of interest reported	31	66.0	25	75.8	6	42.9
Explicitly stated no conflict of interest	19	40.4	19	57.6	0	0.0
Any conflict of interest reported	12	25.5	6	18.2	6	42.9
Conflict of interest not reported	16	34.0	8	24.2	8	57.1

¹Anti-vascular endothelial growth factor (anti-VEGF) medications; photodynamic therapy (PDT)

²Systematic reviews may be counted in more than one category, so totals may add to greater than 100%

³Denominator = 43 systematic reviews with ≥ 2 included studies (4 reliable reviews included fewer than two studies)

⁴Denominator = 22 systematic reviews that performed ≥ 1 quantitative synthesis (i.e., meta-analysis)

About one-third (15/47; 32%) of AMD systematic reviews were published in The Cochrane Library,^19–33 with 25/47 (53%) published in other journals,^34–58 and 7/47 (15%) as agency reports (e.g. French National Authority for Health).^59–65 Most systematic reviews had at least two authors (43/47; 91%). The median number of bibliographic databases searched for systematic reviews was four; 31/47 (66%) groups of authors searched all possible years of at least one database. Only 28/47 (60%) review groups searched for non-English language articles. The number of included intervention studies in each systematic review ranged from 0 to 88 (median 7). Review findings were synthesized qualitatively in most (38; 88%) and quantitatively (“meta-analyses”) in about half (22; 51%) of the 43 systematic reviews that included two or more studies.

Almost two-thirds of AMD systematic reviews provided information on funding (31/47; 66%), with government (18/31; 58%) and department or institution (10/31; 32%) as the most common funding sources. Less than half of systematic review author teams stated that they had no conflicts of interest (19/47; 40%), with 12/47 (26%) disclosing that at least one author had a potential conflict of interest; 16/47 (34%) did not report information on conflicts of interest.

Assessment of the reliability of AMD systematic reviews

We classified the majority (33/47; 70%) of AMD systematic reviews as reliable (Figure 2). The most common reason for classifying a review as unreliable was not reporting a comprehensive search for eligible studies (Table 2 available at http://aaojournal.org). Compared with unreliable systematic reviews, reliable systematic reviews were more likely to have been funded by departments or institutions and produced by review authors who explicitly stated they had no conflicts of interest; all four systematic reviews that reported industry funding were assessed as unreliable (Table 2). Areas needing improvement across all reviews were the need for explicit statements regarding 1) pre-specification of eligibility criteria for studies to be included and 2) limitations of the review. In addition, review authors seldom performed independent evaluation of study eligibility and methodological quality, or independent data abstraction, by two or more reviewers (Figure 2).

Figure 2. Assessment of reliability criteria for 47 systematic reviews on interventions for age-related macular degeneration.

Green, yes; red, no; yellow, can’t tell/not reported; gray, not applicable

*Five criteria used for classifying reliability of systematic reviews

†Denominator = 22 systematic reviews with ≥ 1 quantitative synthesis

All 15 Cochrane systematic reviews were classified as reliable compared with 18/32 (56%) non-Cochrane systematic reviews (Figure 3 available at http://aaojournal.org). All 15 Cochrane systematic reviews specified pre-defined eligibility compared with 16/32 (50%) non-Cochrane systematic reviews, and were more likely to have reported independent selection of studies by two or more review authors, assessment of risk of bias, and extraction of data compared with non-Cochrane systematic reviews. However, fewer Cochrane systematic reviews (27%) discussed limitations at the review level (e.g., incomplete retrieval of relevant studies, the potential effect of reporting bias on the review findings) than non-Cochrane systematic reviews (53%).

Main findings of reliable AMD systematic reviews

Reliable AMD systematic reviews of anti-VEGF agents and photodynamic therapy reported favorable results (Table 3 available at http://aaojournal.org). For other interventions, including antioxidant vitamins and/or minerals, complement inhibitors, interferon alpha, laser photocoagulation, radiotherapy, rheophoresis, statins, submacular surgery, and steroids, reliable AMD systematic reviews reported findings that were either inconclusive or that demonstrated no evidence of an intervention effect.

Among reliable AMD systematic reviews that had addressed the same research question, the conclusions were in good agreement with the exception of the comparative effectiveness and safety of ranibizumab versus bevacizumab for neovascular AMD. Ten reliable systematic reviews published between 2007 and 2014 included 17 distinct randomized controlled trials published between 2004 and 2011^66–82 (Figure 4 available at http://aaojournal.org). Reasons for discordance among systematic reviews all related to the studies included which, in turn, were due to variations in search dates, eligibility criteria, and minimum lengths of follow-up time. Authors of earlier systematic reviews that had compared ranibizumab versus bevacizumab cautioned against using bevacizumab as an off-label alternative to ranibizumab,^41–43 whereas the more recent reviews, which included additional randomized controlled trials, suggested no appreciable difference between the anti-VEGF agents in terms of effectiveness or safety.^34,38 The eligibility criteria of the systematic reviews changed over time, in accordance with completion and publication of findings from new randomized controlled trials. For example, earlier systematic reviews evaluated pegaptanib or ranibizumab versus sham treatment, but more recent systematic reviews evaluated head-to-head comparison of bevacizumab versus ranibizumab.

Mapping of clinical practice guidelines to existing systematic review evidence

We extracted 35 treatment recommendations from the 2015 AAO PPP for AMD (Table 4). Treatment recommendations appeared in five sections of the AAO PPP document: 1) Highlighted findings and recommendations for care table; 2) Background text; 3) Care Process text; 4) Treatment recommendations and follow-up for AMD (Table 4 of PPP); and 5) PPP recommendation grading (Appendix 3 of PPP). Twenty-five of 35 recommendations were reported within the section of the PPP specific to the management of AMD, and 4 of the 35 recommendations were stated in all five sections of the PPP that reported recommendations. Most evidence cited by the AAO PPP to support recommendations were RCTs rather than systematic reviews: 18/35 recommendations were accompanied by citations to randomized controlled trials, whereas 1/35 recommendations was accompanied by citation to a reliable systematic review (Table 5 available at http://aaojournal.org). The PPP cited one other reliable systematic review identified by our study, but it was cited in the background section and not in direct support of a recommendation. No citation was provided to support 12/35 recommendations and 4/35 recommendations cited other reference types (e.g., AAO policy statements, insurance company documents, non-AMD intervention systematic reviews).

Table 4.

Treatment recommendations from the American Academy of Ophthalmology (AAO) Preferred Practice Pattern (PPP) Guideline Statement (2015) and systematic reviews (SRs) of age-related macular degeneration (AMD)

Recommendation made		Relevant and eligible SRs identified in our study		Intervention SRs cited with recommendation in AAO PPP
		Any SRs	Reliable SRs	Any SRs	Reliable SRs
1	"Patients who are currently smoking should be advised to stop."	Sin55	None	None	N/A
2	"In light of all the available information on the subject of aspirin use and AMD, the current preferred practice is for patients who have been instructed to use aspirin by a physician to continue their aspirin therapy as prescribed."	None	N/A	None	N/A
3	"The routine use of genetic testing is not supported by the existing literature and is not recommended at this time."	None	N/A	None	N/A
4	"Patients with early AMD and/or a family history of AMD should be encouraged to assess their own visual acuity using monocular vision testing (i.e., Amsler grid) and have scheduled dilated eye examinations for detecting the intermediate stage of AMD."	None	N/A	None	N/A
5	"Patients with a high-risk AMD phenotype are at increased risk of progression to advanced AMD and should be educated about methods of detecting new symptoms of CNV, including self-monitoring. They should also be educated about the need for promptly reporting new symptoms to an ophthalmologist who can confirm if the new symptoms are from CNV and who can begin any necessary treatment."	None	N/A	None	N/A
6	“The risks, benefits, complications, and alternatives of the treatment should be discussed with the patient and informed consent obtained.”	None	N/A	None	N/A
Antioxidants and minerals
7	"Treatment with antioxidants and minerals as described previously in the original AREDS and AREDS2 trials is recommended for patients who have progressed to intermediate or advanced AMD in at least one eye."	Evans22; Evans36	Evans22; Evans36	None	N/A
8	"There is no evidence to support the use of these supplements for patients who have less than intermediate AMD."	Chong35; Evans21; Evans36	Chong35; Evans21; Evans36	None	N/A
9	“Additional vitamin E supplementation above the AREDS levels should be avoided.”	Evans36	Evans36	None	N/A
10	"A lower zinc dose (25 mg) in the AREDS2 formulation could be considered"	Vishwanathan56	None	None	N/A
11	"The final results of AREDS2 support the recommendation for substitution of beta-carotene with lutein (10 mg) and zeaxanthin (2 mg)."	Zhang57	None	None	N/A
12	"When considering long-term supplementation, some people may have reason to avoid one of more of the supplements evaluated in the original AREDS or AREDS2. Because of the potential adverse effects, such as increased rate of genitourinary conditions that may require hospitalizations, the high doses of antioxidant vitamins and minerals recommended by the original AREDS and AREDS2 should be reviewed by the patient's primary care physician."	Evans21; Evans36	Evans21; Evans36	None	N/A
Anti-VEGF therapy
13	"Intravitreal injection therapy using anti-vascular endothelial growth factor (VEGF) agents (e.g., aflibercept, bevacizumab, and ranibizumab) is the most effective way to manage neovascular AMD and represents the first line of treatment."	Brown59; Colquitt62; Ip52; Jiang38; Lanzetta53; Oliva61; Schouten44; Takeda45; Vedula30; Ziemssen47	Colquitt62; Jiang38; Schouten44; Takeda45; Vedula30; Ziemssen47	Vedula 2008	Vedula 2008
14	"Current practice patterns support the use of anti-VEGF monotherapy for patients with newly diagnosed neovascular AMD, and suggest that these other therapies [verteporfin PDT and thermal laser photocoagulation surgery] are rarely needed yet may be used in unresponsive cases."	Brown59; Colquitt62; Ip52; Jiang38; Lanzetta53; Oliva61; Schouten44; Takeda45; Vedula30; Ziemssen47	Colquitt62; Jiang38; Schouten44; Takeda45; Vedula30; Ziemssen47	None	N/A
15	“Aflibercept intravitreal injection 2.0 mg as described in published reports”	None	N/A	None	N/A
16	“Bevacizumab intravitreal injection 1.25 mg as described in published reports”	Jiang38; Lanzetta53; Mitchell54; Schouten44; Ziemssen47	Jiang38; Schouten44; Ziemssen47	None	N/A
17	“The ophthalmologist should provide appropriate informed consent with respect to the off-label status”	Schmucker41; Schmucker42; Schmucker43	Schmucker41; Schmucker42; Schmucker43	None	N/A
18	“Caution should be used when dosing PRN bevacizumab, as it may be slightly less effective than other monthly anti-VEGF regimens.”	Lanzetta53	N/A	None	N/A
19	“Ranibizumab intravitreal injection 0.5 mg as recommended in literature”	Brown59; Colquitt62; Ip52; Jiang38; Lanzetta53; Mitchell54; Oliva61; Takeda45; Vedula30	Colquitt62; Jiang38; Takeda45; Vedula30	None	N/A
20	“Small retinal hemorrhages are a sign of active CNV or polypoidal choroidal vasculopathy and may be managed with anti-VEGF therapy.”	None	N/A	None	N/A
21	"Most juxtafoveal lesions that may have been previously treated using laser photocoagulation are currently managed using the anti-VEGF agents."	None	N/A	None	N/A
22	"The current trend is to use anti-VEGF agents in preference to laser photocoagulation” for extrafoveal lesions	None	N/A	None	N/A
23	“Symptoms suggestive of postinjection endophthalmitis or retinal detachment require prompt evaluation.”	None	N/A	None	N/A
Verteporfin photodynamic therapy
24	“PDT with verteporfin as recommended in the TAP and VIP reports”	Cruess49; Husereau63; Oliva61; Oliva65; Meads40; Meads64; Wormald33	Husereau63; Oliva65; Meads40; Meads64; Wormald33	None	N/A
25	“Photosensitivity reaction (<3% of patients)…The stated, current recommendations are to avoid direct sunlight for the first 5 days after a treatment.”	None	N/A	None	N/A
26	“Careful consideration should be given to patients with liver dysfunction and to patients who are pregnant, breast-feeding, or of pediatric age, because these patients were not studied in published reports.”	None	N/A	None	N/A
27	"Patients with juxtafoveal lesions may also be considered eligible for the off-label use of PDT with verteporfin."	Husereau63; Oliva65; Meads40; Meads64; Wormald33	Husereau63; Oliva65; Meads40; Meads64; Wormald33	None	N/A
Thermal laser photocoagulation surgery
28	“Thermal laser photocoagulation surgery as recommended in the MPS reports”	Parodi28; Virgili31	Parodi28; Virgili31	None	N/A
29	Thermal laser photocoagulation surgery: “These realities must be emphasized to the patient and family before treatment.” These realities = “Introduction or enlargement of pre-existing scotoma, with or without visual acuity loss, is not a complication of thermal laser photocoagulation; rather, it is an anticipated side effect of the treatment. Similarly, recurrence or persistence of CNV, or the development of new CNV and further visual deterioration after adequate thermal laser surgery, is usually a result of the disease process and is not a complication.”	Parodi28; Virgili31	Parodi28; Virgili31	None	N/A
30	"Thermal laser photocoagulation surgery is no longer recommended for subfoveal CNV treatment."	Parodi28; Virgili31	Parodi28; Virgili31	None	N/A
31	"Laser surgery for extrafoveal lesions remains a less-commonly used, yet reasonable, therapy."	None	N/A	None	N/A
Other treatment recommendations
32	"The data do not currently support the use of combination therapy [intravitreal corticosteroids and/or anti-VEGF agents in various drug combinations or with verteporfin PDT] at this time, especially with the long-term side effects of glaucoma and cataract that are associated with corticosteroid use."	Zhou58	N/A	None	N/A
33	“Observation with no medical or surgical therapies” recommended for early, non-neovascular AMD	None	N/A	None	N/A
34	"Current therapies that have insufficient data to demonstrate clinical efficacy include radiation therapy, acupuncture, electrical stimulation, macular translocation surgery, and adjunctive use of intravitreal corticosteroids with verteporfin PDT. Therefore, at this time, these therapies are not recommended."	Eandi19; Evans23; Falkner50; Giansanti26	Eandi19; Evans23; Giansanti26	None	N/A
35	“The data on management of these larger [submacular] hemorrhages are inadequate to make a recommendation at this time.”	None	N/A	None	N/A

We identified existing reliable systematic reviews of interventions for AMD for 15 of the 35 treatment recommendations (Table 4). For example, additional reliable systematic reviews of anti-VEGF agents, vitamins and minerals, photodynamic therapy, laser photocoagulation, submacular surgery, and radiotherapy could have been referenced by the AAO PPP guideline to inform their recommendations but were not (Table 4). There were 20 treatment recommendations for which we identified no existing reliable systematic review, which highlights evidence gaps. The treatment recommendations and findings from reliable systematic reviews were generally consistent (Table 3 available at http://aaojournal.org).

Discussion

Reliability of SRs

We classified 14 (30%) of 47 systematic reviews describing intervention effectiveness for AMD as unreliable according to standard methodological criteria. Lack of reporting a comprehensive search strategy was the most common reason for classifying a review as unreliable. We found that Cochrane reviews comprise about one-third of all AMD systematic reviews. We assessed all 15 Cochrane reviews as reliable compared with 18 (56%) of 32 non- Cochrane reviews. This finding is in keeping with other investigations that have shown the high quality of Cochrane reviews and methodology.^83–90 Because we are affiliated with the Cochrane Eyes and Vision Group, the criteria we set for assessing review methods and reporting are Cochrane-oriented. Our perspectives may partially explain the judgements we made and the discrepancies we found.

Studies evaluating the reporting quality of systematic reviews of other topics have found systematic reviews to be of disappointing quality, many finding 20% to 65% of the systematic reviews as being poor or low quality.^{83,84,91–95} Yet with the availability and promotion of methodological and reporting standards for systematic reviews,^16,96–98 we expect reliable conduct and reporting of systematic reviews published in the literature to increase. Well-reported methods may not accord with methods actually used to conduct the review, however. For example, an investigation of studies described as randomized controlled trials in Chinese-language journals found that 93% (95% confidence interval (CI) 92.3% to 94.1%) of the studies actually used non-random methods to allocate treatment groups.⁹⁹ A limitation of our study is that we evaluated systematic review reporting and did not contact review authors for supplemental information when methods were not reported or were reported unclearly. Furthermore, authors of reports from studies included in systematic reviews may not report methods clearly and accurately.

The uncoordinated fashion in which many systematic reviews currently are conducted and reported appears to result in unnecessary duplication of effort and varying results.^100,101 In some cases, existing reviews were unreliable because of the lack of adherence to reporting standards and use of systematic review methodology aimed at minimizing selection and reporting biases. Publication of unreliable reviews represents a waste of resources. Journal editors should set standards for systematic reviews they publish and refer authors and peer reviewers to the PRISMA reporting standards.^96,97 To conserve resources, we recommend that future systematic reviews should address unanswered clinical questions. Further, systematic reviews should be undertaken by individuals trained in systematic review methodology. Manuscripts that report systematic reviews should be reviewed by editors and peer reviewers knowledgeable in methodological and reporting standards in order to produce reliable research that can be used by guideline developers, patients, clinicians, and others.

Usefulness of SRs for informing clinical practice guidelines

The risk of producing reviews that are not relevant to clinical users is made tangible by the fact that many treatments for AMD summarized in reliable reviews included in our study were not mentioned in the 2015 AAO PPP. Many systematic reviews, including Cochrane reviews, undergo a long publication process that on one hand ensures high quality, but on the other hand may render them out of date or unavailable to users and guidelines producers in a rapidly emerging therapeutic area, such as anti-VEGF therapy for neovascular AMD. Collaboration between systematic reviewers and guideline developers could facilitate relevancy of topics and communication of results in a timely manner.

Six types of treatments for AMD were evaluated by two or more systematic reviews. In the case of five types of interventions (antioxidants, omega-3 fatty acids, photodynamic therapy, laser photocoagulation, and submacular surgery), reviews addressing the same topic yielded the same conclusions and initially appear to indicate a waste of resources. However, in the case of anti-VEGF therapy, the research question and eligibility criteria addressed by the systematic reviews changed over time as treatment availability and potential outcomes changed. The first systematic reviews included only RCTS that had compared pegaptanib or ranibizumab with control. The more recent systematic reviews of anti-VEGF therapy also included case series and non-randomized studies, specifically to address the issue of effectiveness and safety of the off-label drug bevacizumab. Since the time the searches were conducted for the current study, Cochrane authors have updated an earlier review of anti-VEGF effectiveness and also have published a review comparing the systemic safety of ranibizumab versus bevacizumab.^102,103 Unlike other research that has found duplication of systematic reviews on the same topic to be wasteful^101,104 or lead to discordant findings,^105,106 we conclude that sequential systematic reviews that at first glance appear to cover similar topics instead may represent evolution in the research question with increased clinical experience and serve as an indication of a rapidly developing field.

Despite summarizing the available evidence, systematic reviews may not meet the needs of clinicians, patients, and guideline panels. Reviews with narrow scopes, i.e., those that split a clinician’s “real world” question into answerable research questions, may not provide all information needed by guidelines panelists. Nor do traditional pairwise comparisons address the question of “what works best”? “Multiple treatment comparisons” utilize network meta-analysis methodology, and increasingly are used when head-to-head trials of multiple interventions are not available or are insufficient to address the research question.¹⁰⁷

Integration of SRs in clinical practice guidelines

Literature searches for the 2015 AAO PPP on AMD were updated 11 June 2013. The AAO PPP cited two systematic reviews that were rated as reliable in our study, with many recommendations citing only evidence from individual studies or no citation at all; the AAO PPP did not cite any unreliable systematic review. However, evidence from 22 additional reliable systematic reviews underpinning 15 of the 35 recommendations could have been incorporated into the AAO PPP. There were nine existing Cochrane reviews that directly supported 12 of the treatment recommendations. In accordance with best practice standards outlined by the Institute of Medicine,⁶ we suggest that interaction between systematic review teams and clinical practice guideline groups be encouraged to provide a comprehensive view of the evidence at a point in time and to illuminate evidence gaps. For example the AAO PPP panel for AMD could collaborate with the Cochrane Eyes and Vision Group to identify existing Cochrane reviews for their guidelines and highlight evidence gaps where Cochrane reviews should be given high priority. Cochrane authors would need to act promptly to provide timely development or updating of reviews.

The majority of treatment recommendations in the AAO PPP for AMD were supported by evidence from only randomized controlled trials or non-randomized studies. We acknowledge that a number of studies supporting some recommendations on treatments for AMD were well-designed, landmark RCTs, and these studies may have been well known to experts preparing recommendations. However, by transparently filtering and summarizing evidence in one place, systematic reviews provide an evidence base more extensive and comprehensive than looking at individual studies alone; they include structured assessment of trial methodology and the overall certainty of the evidence, providing the opportunity to evaluate all the evidence addressing a question to determine the current best answer. Systematic reviews and meta-analyses also are likely to be more useful than individual studies for providing information about rare adverse events, as even large RCTs often are not adequately powered to detect differences between treatments for infrequently observed outcomes.¹⁰⁸

Although systematic reviews are important underpinnings of trustworthy treatment recommendations, they are not intended to serve in place of clinical practice guidelines. Clinical practice guidelines should be clear in stating unambiguously what is recommended, or not recommended, and should provide the evidence in support of each recommendation. In fact, frameworks such as GRADE (www.gradeworkinggroup.org), have tools that use complementary methods and presentation graphics to support the work of both guideline developers and systematic reviewers. These are especially important for recommendations for which no high-quality evidence exists so that guideline developers must rely on lower level sources of evidence and clinical expertise. For clarity, when preparing clinical practice guidelines it would be helpful to have all recommendations with supporting citations clearly reported in one place in the guideline document.

Conclusions

Ideally, reliable systematic reviews underpin evidence-based clinical practice guidelines. For AMD,reliable systematic reviews exist for many treatment recommendations in the AAO PPP and should be used to support these recommendations. Mapping clinical practice guidelines to existing systematic reviews is a useful way to highlight areas where evidence generation or evidence synthesis is either available or needed.