What Comparative Effectiveness Research Is Needed? A Framework for Using Guidelines and Systematic Reviews to Identify Evidence Gaps and Research Priorities

Abstract

The authors developed and tested a framework for identifying evidence gaps and prioritizing comparative effectiveness research by using a combination of clinical practice guidelines and systematic reviews. In phase 1 of the project, reported elsewhere, 45 clinical questions on the management of primary open-angle glaucoma were derived from practice guidelines and prioritized by using a 2-round Delphi survey of clinicians. On the basis of the clinicians′ responses, 9 questions were classified as high-priority. In phase 2, reported here, systematic reviews that addressed the 45 clinical questions were identified. The reviews were classified as at low, high, or unclear risk of bias, and evidence gaps (in which no systematic review was at low risk of bias) were identified. The following comparative effectiveness research agenda is proposed: Two of the 9 high-priority questions require new primary research (such as a randomized, controlled trial) and 4 require a new systematic review. The utility and limitations of the framework and future adaptations are discussed.

Comparative effectiveness research (CER) is a key element of current efforts in health care reform in the United States (1). Before initiating new research, investigators must ascertain gaps in the evidence by identifying the important clinical questions in a topic area and then determining whether existing research has answered these questions. Prioritizing new CER to address identified gaps presents an ongoing challenge (2). The Patient-Centered Outcomes Research Institute (PCORI) has committed funds to develop, test, refine, and evaluate methods that can inform the process of establishing and updating national priorities for CER and patient-centered outcomes research (3, 4).

We propose and assess a general framework for prioritizing CER, using primary open-angle glaucoma (POAG) as a topic area. Worldwide, glaucoma of all types is a major source of morbidity, decreased quality of life, and increased health care costs (5, 6). It has been estimated to be the second-leading cause of blindness and visual impairment in 2010 (5, 6). In the United States, open-angle glaucoma accounts for more than 90% of all glaucoma cases; it affected more than 2.25 million Americans aged 40 years or older in 2000, and this number is expected to increase to 3.36 million by 2020 because of the aging population (5, 6).

We aimed to develop, implement, and evaluate a framework for identifying evidence gaps and prioritizing CER. Because clinical practice guidelines (CPGs) reflect questions of interest to clinicians, we considered them a starting point for building our framework to prioritize CER. Once the clinical questions addressed in a CPG are identified, one can search for relevant systematic reviews to determine evidence gaps. For questions of intervention effectiveness, a systematic review of high-quality randomized, controlled trials (RCTs) can show whether research has answered the question; uncertainty reflected by systematic reviews indicates a potential candidate area for primary CER (2, 7, 8). Opinions on topics and outcomes can be sought from stakeholders, such as practicing clinicians, patients, and evidence users, and incorporated during the prioritization process (for example, when framing or ranking the research questions). Our objective for this report was to identify systematic reviews that addressed clinical questions derived from the CPGs, classify their methodological quality, propose a CER agenda, and assess the utility of our framework.

Methods

For practical purposes, we implemented and evaluated the utility of our framework in 2 phases (Figure 1). In phase 1, reported elsewhere (9), we used the 2005 American Academy of Ophthalmology (AAO) Preferred Practice Patterns for the management of POAG (10) to derive 45 answerable clinical questions that could be addressed by RCTs and systematic reviews of RCTs. We conducted a 2-round Delphi survey of 620 members of the American Glaucoma Society to prioritize the questions for research that would inform good patient care. Using responses from 169 participating clinicians, we classified 9 clinical questions as high priority (9).

Figure 1.

Key steps and stakeholder involvement in a framework that combines practice guidelines and systematic reviews to identify evidence gaps and prioritize comparative effectiveness research.

* Patients and consumers were not involved in this project but could be engaged in future adaptations.

In phase 2, we determined whether up-to-date systematic reviews were available for each of the 45 clinical questions derived from the AAO CPGs. To identify evidence gaps, we appraised the methodological quality of the reviews and mapped evidence from the reviews to the 45 questions. We derived the CER agenda for the management of POAG by combining the prioritized questions from phase 1 with the evidence gaps.

Identifying Existing Systematic Reviews

We prespecified the following criteria for including systematic reviews: full-text articles that described a systematic review, defined as “a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize similar but separate studies” (11, 12); inclusion of patients with POAG, as defined by the individual review authors; comparison of any intervention described in the 45 clinical questions with another intervention, placebo, or no intervention; and reporting of any outcome. We excluded reports that concerned only health economic evaluation and meta-analyses with no systematic review.

We worked with information specialists at the William H. Welch Medical Library, Johns Hopkins University School of Medicine, and developed a search strategy that combined eye and vision terms with a validated search filter designed to identify systematic reviews (13). We originally searched PubMed in November 2006; EMBASE in March 2007; and The Cochrane Library, Issue 4, 2006, with no language restrictions, to build a register of systematic reviews addressing eye and vision research questions (Appendix 1, available at www.annals.org, lists our search strategies). We updated our searches in September 2009.

We imported all citations from our electronic searches into a ProCite, version 5.0, database (Thomson Scientific, Philadelphia, Pennsylvania) and removed duplicate citations. Two investigators independently reviewed titles and abstracts, and we retrieved full-text articles of all records classified by at least 1 investigator as possibly eligible. Two investigators independently reviewed the full-text reports for final eligibility and resolved discrepancies through discussion.

Extracting Data and Appraising Methodological Quality

We extracted data from eligible full-text systematic reviews for study objective, population, interventions compared, outcomes examined, eligibility criteria for studies in the systematic review, search methods used to identify studies, and other study characteristics.

We assessed methodological quality by using a pilot-tested form that adapted items from related instruments (14–18). Our assessment included the 13 quality criteria listed in Figure 2. When a meta-analysis was conducted as part of a systematic review, we abstracted data on the statistical methods used to calculate the meta-analytic estimate and its variance and on the assessment of statistical heterogeneity (the variability in the intervention effects across studies).

Figure 2.

Systematic reviews on the management of primary open-angle glaucoma that satisfied each methodological quality criterion.

* Not applicable because the reviews did not assess methodological quality or did not perform a meta-analysis,

† The denominator was the 32 systematic reviews with ≥1 meta-analysis.

We selected 4 key deficiencies from the 13 quality items to classify the findings from a review as at low, high, or unclear risk of bias. Findings from a systematic review were classified as at high risk of bias if it contained a non-comprehensive literature search, did not assess the methodological quality of included studies, used inappropriate statistical methods for meta-analysis, or presented conclusions inconsistent with the review findings (for example, if the review finding was based on 1 statistically significant meta-analysis out of 26 meta-analyses conducted [19]). Findings were classified as at low risk of bias if none of these deficiencies was observed and as at unclear risk of bias if insufficient details were provided to permit judgment on 1 or more of the 4 deficiencies (Table 1) (20).

Table 1.

Criteria for Classifying Systematic Review Findings as at Low Risk of Bias^*

Criterion	Definition
Comprehensive literature search	An electronic search of at least 2 bibliographic databases was conducted by using a search strategy comprising a mixture of Medical Subject Headings and keywords, and this search was supplemented by searching reference lists of included studies or other reports that cited the study under evaluation; contacting experts in the field or study authors; or searching for unpublished studies, ongoing studies, or studies published in hard-to-access literature, such as letters to the editor or conference abstracts.
Assessment of methodological quality of included trials	Investigators could use any method to assess the methodological quality or risk of bias of included trials, such as a scale, in which various components of quality are scored and combined to give a summary score; a checklist, in which specific questions are asked; or a domain-based evaluation, in which critical assessments are made separately for different domains, such as allocation concealment or masking (12).
Appropriate statistical methods of meta-analysis	Statistical methods that preserve the randomized nature of the trial were used (20), and the individual trial effect estimate was weighted by using appropriate variance estimates (12, 16, 17).
Conclusions consistent with review findings	Conclusions were consistent with findings from valid statistical analyses, provided a balanced consideration of the benefit and harms of the intervention, and were not overly optimistic (for example, the conclusions drawn did not depend on results from tests of multiple hypotheses or chance findings) (16, 17).

^*In selecting the 4 potential deficiencies, we recognized that other factors could determine the methodological quality of a systematic review.

One reviewer abstracted data, assessed methodological quality, and classified risk of bias for the included systematic reviews. A second reviewer verified the abstracted data against the original publication; discrepancies were resolved through discussion.

Identifying Evidence Gaps and Prioritizing CER

We mapped systematic reviews that met our eligibility criteria to the 45 clinical questions; a systematic review could be associated with more than 1 question. We classified evidence cited in the AAO Preferred Practice Patterns to support the recommendations by study design (systematic review or RCT).

We compared the systematic reviews that were at low risk of bias with the results of the Delphi survey to identify evidence gaps and CER priorities. We considered evidence from systematic reviews to be most useful for decision making if it was at low risk of bias, up to date (the literature search was done within 2 years of publication of the review), and conclusive. We considered evidence to be conclusive if further research was unlikely to change our confidence in the estimate of effect and not conclusive if the reviews indicated uncertainty or unresolved questions and further research was likely to change our confidence in the estimate of effect (18).

Statistical Analysis

We summarized the characteristics of included systematic reviews and tabulated the number and proportion of reviews that fulfilled each methodological quality criterion. All data analyses were performed with Stata, version 10 (StataCorp, College Station, Texas). We estimated the resources required and staff needed to complete the project.

Role of the Funding Source

This phase of our study was funded by the National Eye Institute, National Institutes of Health. Phase 1 was supported by the Cochrane Prioritization Fund, The Cochrane Collaboration. The sponsors had no role in the design or conduct of the study; the collection, management, analysis, or interpretation of the data; or the preparation, review, or approval of the manuscript.

Results

Searches and Study Selection

We identified 5990 unique records from our electronic searches, of which 547 systematic reviews addressed eye and vision research questions. Of these, 39 systematic reviews (42 articles) addressed questions related to management of POAG and met our eligibility criteria (Appendix Figure 1, available at www.annals.org) (19, 21–61). Eight of the 39 were Cochrane reviews, 18 were found in the Database of Abstracts of Reviews of Effects in The Cochrane Library, and the remaining 13 were found only in PubMed or EMBASE. The 39 systematic reviews were published between 2000 and 2009, with more than half published after 2007 (Appendix Table 1, available at www.annals.org). The literature search was done within 2 years of publication of the review in 35 of 39 cases (89.7%).

Appendix Figure 1.

Summary of evidence search and selection.

POAG = primary open-angle glaucoma; SR = systematic review.

* One systematic review is associated with 4 articles. A review could address >1 clinical question depending on the breadth of the review topic. Consequently, the number of clinical questions does not match the number of reviews.

Appendix Table 1.

Characteristics of the 39 Systematic Reviews on the Management of Primary Open-Angle Glaucoma

Characteristic	Value
Median year published (range)	2007 (2000–2009)
Type of interventions examined, n(%)*
Medical intervention vs. medical intervention or placebo	24 (61.5)
Laser trabeculoplasty vs. medical intervention	1 (2.6)
Laser trabeculoplasty vs. filtering surgery or medical or no intervention	1 (2.6)
Filtering surgery vs. medical intervention	1 (2.6)
Filtering surgery vs. filtering surgery	11 (28.2)
Cyclodestructive surgery vs. cyclodestructive surgery	1 (2.6)
Outcome examined, n(%)*
Intraocular pressure	34 (87.2)
Visual field	7 (18.0)
Safety	30 (76.9)
Other outcomes	19 (48.7)
Type of studies included, n(%)*
Randomized, controlled trials	36 (92.3)
Non-randomized, controlled trials or observational studies	5 (12.8)
Search
Databases searched, n (%)*
MEDLINE (PubMed)	39 (100)
EMBASE	31 (79.5)
Cochrane Central Register of Controlled Trials	27 (69.2)
LILACS	6 (15.4)
Other databases	14 (35.9)
Median databases searched (IQR), n	3 (2–4)
Reported searching non-English-language studies, n (%)	26 (66.7)
Searched all possible years for ≥1 database, n(%)	34 (87.2)
Other sources searched, n (%)*
Searched reference lists or reports that cited the study	29 (74.4)
Contacted experts in the field or study authors	11 (28.2)
Searched hard-to-access or unpublished studies	14 (35.9)
Searched ongoing studies	10(25.6)
Searches completed within 2 y of review publication, n (%)	35 (89.7)
Searches updated after review publication, n(%)	5 (12.8)
Results of systematic reviews*
Median studies included (IQR), n	11 (8–19)
Median participants included (IQR), n	1310 (758–2187)
Synthesized evidence qualitatively, n (%)	26 (66.7)
Performed ≥1 meta-analysis, n (%)	32 (82.1)
Assessed statistical heterogeneity, n (%)	22 (68.9t)
Funding sources, n(%)*
No funding	14 (35.9)
Industry	6 (15.4)
Government	8 (20.5)
Other sources of funding	2 (5.1)
Not reported	11 (28.2)

IQR = interquartile range.

^*Totals may be >39 because systematic reviews could be counted in ≥1 category.

^†The denominator was the 32 systematic reviews with ≥1 meta-analysis.

Figure 2 illustrates the findings from our appraisal of methodological quality of the 39 systematic reviews. Nearly 40% of the reviews did not report having prespecified eligibility criteria or performing a comprehensive literature search. For nearly half of the reviews, independent evaluation by more than 1 review author was not reported for eligibility assessment, data abstraction, or quality assessment of the included studies.

Thirteen of the 39 reviews (33.3%) included in this study were classified as at low risk of bias, 22 (56.4%) at high risk of bias, and 4 (10.3%) at unclear risk of bias. Reviews at high risk of bias were classified as such for 1 or more of the following reasons: noncomprehensive literature search (12 reviews), no assessment of the methodological quality of the included studies (11 reviews), inappropriate statistical analyses (13 reviews), or conclusions that were inconsistent with the review findings (1 review). Fifteen of 22 reviews were classified as at high risk of bias because of at least 2 reasons.

We identified 2 major types of inappropriate statistical analyses in 13 of 32 reviews (40.6%) with a meta-analysis (3 reviews had both types of inappropriate analyses) (Appendix 2, available at www.annals.org). Nine reviews pooled data from similar groups across studies, resulting in a nonrandom comparison (for example, data from the timolol treatment groups from 21 trials and latanoprost treatment groups from 33 trials were pooled separately and compared in a review [39]). In addition, 7 reviews that reported percentage change in intraocular pressure (IOP) from baseline as the effect estimate used an incorrect formula to calculate the variance of the effect estimate. Appendix 2 summarizes the incorrect formulas used in the reviews to calculate the variance for the percentage of change in IOP from baseline.

Identifying Evidence Gaps and Prioritizing CER

Table 2 shows the results of applying our method to identify evidence gaps in the context of the overall framework that we evaluated. We identified at least 1 systematic review for 23 of 45 clinical questions (51.1%) and at least 1 review at low risk of bias for 13 of 45 questions (28.8%) (Table 2 and Appendix Table 2, available at www.annals.org). Five of the 9 questions identified by clinicians as high priority for research were associated with at least 1 systematic review that was at low risk of bias. Of these, 2 had reviews with findings that we classified as not conclusive; therefore, these 2 questions need new primary research (such as an RCT). The remaining 4 high-priority questions require a new systematic review because no systematic review at low risk of bias was found. Of the 36 lower-priority questions, 28 require a new systematic review (Appendix Table 2). The 2005 AAO Preferred Practice Patterns cited only 5 of 10 systematic reviews published before the guidelines were prepared (32–35, 37, 49, 59, 60), and we classified 3 of the 5 as at low risk of bias (37, 59, 60).

Table 2.

Results of Using Our Framework to Set the CER Agenda for High-Priority Clinical Questions

Question and Study, Year (Reference)	Date of Search	Main Findings on the Effectiveness and Safety of Interventions	CER Agenda
Questions with ≥1 available review at low risk of bias (5 questions)*
Is medical therapy an effective initial treatment for lowering IOP in patients with POAG?		4 of 4 reviews were at low risk of bias, and main findings were classified as not conclusive.	New primary CER (such as an RCT)
Maier et al, 2005 (46)	2004	In patients with OHT or manifest glaucoma, lowering IOP with medications is more effective than placebo or no treatment in reducing the risk for visual field loss.
Rolim de Moura et al, 2007 (50)	June 2007	In patients with newly diagnosed OAG, laser trabeculoplasty is more effective than medication used before the 1990s in reducing the risk for uncontrolled IOP at 6 mo and 2 y of follow-up.
		In patients with newly diagnosed OAG, laser trabeculoplasty is associated with a higher risk for peripheral anterior synechiae formation than medications used before the 1990s.
		In patients with newly diagnosed OAG, no statistically significant effect was noted between laser trabeculoplasty and medications used before the 1990s in reducing the risk for visual field deterioration or optic neuropathy progression.
		No evidence is available to determine the effectiveness of laser trabeculoplasty compared with contemporary medication (prostaglandin analogues, topical anhydrase inhibitors, or α2-agonists).
Vass et al, 2007 (55)	May 2007	In patients with POAG who had first onset of a visual field defect, no evidence is available to determine whether topical hypotensive medical therapy used for ≥1 y reduces progression of POAG.
Burr et al, 2005 (24)†	April 2007	In patients with mild OAG, initiating treatment with medication or trabeculectomy does not result in different risk for glaucoma progression up to 5 y.
		In patients with mild OAG, initiating treatment with trabeculectomy is associated with a higher risk for reduced vision, cataract formation, and eye discomfort than medication.
		In patients with more severe OAG, initiating treatment with medication (pilocarpine, which is rarely used now as a first-line medication) is associated with a higher risk for glaucoma progression than surgery.
		In patients with more severe OAG, surgery is more effective than pilocarpine in lowering IOP.
		In severe OAG and in persons of “black ethnicity,” no evidence is available to determine the effectiveness of contemporary medication (prostaglandin analogues, α2-agonists, and topical carbonic anhydrase inhibitors) compared with surgery.
		No evidence is available to determine the effectiveness of medication compared with nonpenetrating surgery, drainage devices, or transscleral cyclophotocoagulation.
Are prostaglandin-analogue eyedrops effective in lowering IOP in patients with POAG?		4 of 17 reviews were at low risk of bias, and main findings were classified as conclusive.	Resolve discrepant findings from 2 reviews that were at low risk of bias
Li et al, 2006 (44)	August 2005	In patients with OAG or OHT, travoprost 0.004% is more effective than timolol 0.5% in lowering IOP.
		In patients with OAG or OHT, travoprost 0.004% seems equivalent to bimatoprost 0.03% and latanoprost 0.005% in lowering IOP.
		In patients with OAG or OHT, travoprost 0.004% seems more effective than travoprost 0.0015% in lowering IOP.
		Travoprost 0.004% is associated with a higher risk for ocular hyperemia and eyelash changes than timolol 0.5%, latanoprost 0.005%, or travoprost 0.0015%.
		Travoprost 0.004% is associated with a higher risk for iris pigmentation than timolol 0.5% but not latanoprost 0.005% or travoprost 0.0015%.
Fung et al, 2007 (36)	March 2006	In patients with OAG, OHT, and NTG, monotherapy with latanoprost 0.005% once daily is more effective than brimonidine 0.2% twice daily in lowering IOP up to 1 y.
		Brimonidine is associated with a higher risk for fatigue than latanoprost.
Aptel et al, 2008 (23)	July 2006	In patients with POAG and OHT, bimatoprost is more effective than latanoprost and travoprost in lowering IOP.
		Bimatoprost is associated with a higher risk for hyperemia than latanoprost and travoprost.
Hodge et al, 2008 (38)	October 2006	In patients with POAG and OHT, latanoprost is more effective than dorzolamide in lowering IOP but not more effective than brimonidine.
		Latanoprost induces fewer ocular adverse events (excluding hyperemia) than brimonidine.
		Neither travoprost nor bimatoprost has been compared with dorzolamide or brimonidine in the present literature.
Are β-adrenergic antagonist eyedrops effective for lowering IOP in patients with POAG?		2 of 8 reviews were at low risk of bias, and main findings were classified as conclusive.	Not included‡
Li et al, 2006 (44)	August 2005	In patients with OAG or OHT, travoprost 0.004% is more effective than timolol 0.5% in lowering IOP.
		In patients with OAG or OHT, travoprost 0.004% seems equivalent to bimatoprost 0.03% and latanoprost 0.005% in lowering IOP.
		In patients with OAG or OHT, travoprost 0.004% seems more effective than travoprost 0.0015% in lowering IOP.
		Travoprost 0.004% is associated with a higher risk for ocular hyperemia and eyelash changes than timolol 0.5%, latanoprost 0.005%, or travoprost 0.0015%.
		Travoprost 0.004% is associated with a higher risk for iris pigmentation than timolol 0.5% but not latanoprost 0.005% or travoprost 0.0015%.
Vass et al, 2007 (55)	May 2007	In patients with POAG, timolol is more effective than carteolol in reducing the progression of visual field defects.
		In patients with OHT and POAG, levobunolol is more effective than timolol in reducing the onset and progression of visual field defects.
		In patients with OHT and POAG, patient dropouts due to drug-related adverse events were similar for levobunolol and timolol.
Is intraoperative mitomycin C effective and safe for improving the success rate of primary and repeated filtering surgery?		1 of 1 review was at low risk of bias, and main findings were classified as conclusive.	Not included‡
Wilkins et al, 2005 (59)†	January 2010	In eyes that have undergone no previous surgery and those at high risk for surgery failure, intraoperative mitomycin C is more effective than placebo in reducing the risk for surgical failure.
		Intraoperative mitomycin C is more effective than placebo in lowering mean IOP at 12 mo.
		Intraoperative mitomycin C is associated with a higher risk for cataract formation than placebo.
		Power was insufficient to detect any increase in other serious side effects, such as endophthalmitis.
Is the use of drainage devices effective and safe for management of POAG in patients with failed filtering surgery, scarred conjunctiva, or poor prognosis of filtration surgery?		1 of 1 review was at low risk of bias, and main findings classified as not conclusive.	New primary CER (such as an RCT)
Minckler et al, 2006 (47)	January 2006	Trabeculectomy is more effective than the Ahmed implant in lowering IOP at 11- to 13-mo follow-up, but the difference is small.
		In patients with complicated glaucoma, evidence is insufficient to conclude that clinical outcomes of trabeculectomy differ from those of aqueous shunts.
		Evidence is insufficient to conclude that any specific aqueous shunt is superior to the others.
Questions with no available reviews at low risk of bias (4 questions)
Is combination medication effective in lowering IOP in patients with POAG?	NA	0 of 8 reviews was at low risk of bias	New systematic review
Does postoperative care (such as topical corticosteroids, repair of bleb leaks, bleb massage, or suture lysis) result in better outcomes in patients who have filtering surgery?	NA	No reviews available	New systematic review
Does the use of topical corticosteroids in the postoperative period improve patient outcomes?	NA	No reviews available	New systematic review
Do the recommended indications for adjusting therapy (such as target IOP not achieved) improve outcomes in patient with POAG?	NA	No reviews available	New systematic review

CER = comparative effectiveness research; IOP = intraocular pressure; NA = not applicable; NTG = normal-tension glaucoma; OAG = open-angle glaucoma; OHT = ocular hypertension; POAG = primary open-angle glaucoma; RCT = randomized, controlled trial.

^*A systematic review could be mapped to ≥1 clinical question depending on the breadth of the review topic.

^†The Cochrane systematic review included an updated search after its initial publication.

^‡Research questions with conclusive findings from systematic reviews were not included in the CER agenda.

Appendix Table 2.

Existence of and Findings From Systematic Reviews That Relate to Nonpriority Clinical Questions

Question and Study, Year (Reference)	Date of Search	Main Findings on the Effectiveness and Safety of Interventions
Question with ≥1 available review at low risk of bias (8 questions) *
Are α2-adrenergic agonist eyedrops effective in lowering IOP in patients with POAG?
Fung et al, 2007 (36)	March 2006	In patients with OAG, OHT, and NTG, monotherapy with latanoprost 0.005% once daily is more effective than brimonidine 0.2% twice daily in lowering IOP up to 1 y.
		Brimonidine is associated with a higher risk for fatigue than latanoprost.
Hodge et al, 2008 (38)	July 2006	In patients with POAG and OHT, latanoprost is more effective than dorzolamide in lowering IOP but not more effective than brimonidine.
		Latanoprost induces fewer ocular adverse events (excluding hyperemia) than brimonidine.
		Neither travoprost nor bimatoprost has been compared with dorzolamide or brimonidine in the present literature.
Are topical and oral carbonic anhydrase inhibitors effective in lowering IOP in patients with POAG?
Vass et al, 2007 (55)	May 2007	The comparison of dorzolamide to placebo failed to demonstrate a protective effect.
Hodge et al, 2008 (38)	July 2006	In patients with POAG and OHT, latanoprost is more effective than dorzolamide in lowering IOP but not more effective than brimonidine.
		Latanoprost induces fewer ocular adverse events (excluding hyperemia) than brimonidine.
		Neither travoprost nor bimatoprost has been compared with dorzolamide or brimonidine in the present literature.
Are parasympathomimetic eyedrops effective in lowering IOP in patients with POAG?
Vass et al, 2007 (55)	May 2007	Brimonidine, pilocarpine, and epinephrine were not superior to timolol in visual field protection (which should not be interpreted as evidence of equivalence).
Are interventions effective for improving adherence to and efficacy of medical therapy in patients with POAG?
Gray et al, 2009 (37)	January 2009	Interventions involving simplified dosing regimens, reminder devices, education, and individualized care planning improve adherence.
		No particular intervention can be advocated at this time because of inadequate methodological quality and heterogeneity of study design.
Is laser trabeculoplasty an effective initial treatment in lowering IOP in patients with POAG?
Maier et al, 2005 (46)	2004	In patients with POAG or NTG, laser trabeculoplasty or surgery is more effective than no treatment in reducing the risk for visual field loss or deterioration of the optic disc.
Rolim de Moura et al, 2007 (50)	June 2007	Trabeculoplasty is less effective than trabeculectomy in controlling IOP at 6 mo and 2 y of follow-up.
		Argon laser trabeculoplasty decreases the risk for uncontrolled IOP in participants already receiving maximal tolerated medical antiglaucoma therapy.
		More evidence is necessary to determine whether different laser technology and protocol modalities (e.g., 360 degrees in 1 session or 180 degrees in 2 sessions, selective laser trabeculoplasty, diode laser, and monochromatic wave laser) are as effective as the traditional laser trabeculoplasty.
Is filtering surgery an effective and safe initial treatment in lowering IOP in patients with POAG?
Maier et al, 2005 (46)	2004	In patients with POAG or NTG, laser trabeculoplasty or surgery is more effective than no treatment in reducing the risk for visual field loss or deterioration of the optic disc.
Rolim de Moura et al, 2007 (50)	June 2007	Trabeculoplasty is less effective than trabeculectomy in controlling IOP at 6 mo and 2 y of follow-up.
Burr et al, 2005 (24)†	April 2007	In patients with mild OAG, initiating treatment with medication or trabeculectomy does not result in different risk for glaucoma progression up to 5 y.
		In patients with mild OAG, initiating treatment with trabeculectomy is associated with a higher risk for reduced vision, cataract formation, and eye discomfort than medication.
		In patients with more severe OAG, surgery is more effective than pilocarpine in lowering IOP.
		In severe OAG and in persons with “black ethnicity,” no evidence is available to determine the effectiveness of contemporary medication (prostaglandin analogues, α2-agonists, and topical carbonic anhydrase inhibitors) compared with surgery.
		No evidence is available to determine the effectiveness of medication compared with nonpenetrating surgery, drainage devices, or transscleral cyclophotocoagulation.
Is postoperative 5-fluorouracil therapy effective and safe in improving the success rate of primary and repeated filtering surgery?
Wormald et al, 2001 (60)†	October 2008	In eyes at high risk for failure and those undergoing surgery for the first time, postoperative injection of 5-fluorouracil seems effective in reducing the risk for failure of trabeculectomy.
		Injection of low doses of 5-fluorouracil may not be effective.
		No evidence is available of an increased risk for serious sight-threatening complications.
		No evidence is available on the participants’ perspective of care.
Do additional surgical procedures, such as flat anterior-chamber correction, repair of bleb leaks, bleb massage, suture lysis, or bleb needling, improve the long-term result for patients who have filtering surgery?
Feyi-Waboso and Ejere, 2004 (31)†	September 2008	Needling of encapsulated trabeculectomy blebs is no more effective than medical treatment in reducing IOP (on the basis of 1 small trial).
Questions with no available reviews at low risk of bias (28 questions)
Does discussion of the treatment options for lowering IOP (medical, laser, filtering, or cyclodestructive surgery) affect the patient’s choice of initial therapy?	NA	No reviews available
Is treating one eye initially and comparing it with the initially untreated eye a useful way of determining the pressure- lowering efficacy of a topical ocular hypotensive agent?	NA	No reviews available
Does discussion of the benefits and harms of medical treatment with patients affect patient satisfaction?	NA	No reviews available
Does routine assessment help reduce systemic side effects, toxicity, and possible interactions of glaucoma medication?	NA	No reviews available
Does patient education about eyelid closure or nasolacrimal occlusion help reduce systemic absorption when applying topical medications?	NA	No reviews available
What is the relative effectiveness of selective laser trabeculoplasty compared with other trabeculoplasty techniques in lowering IOP in patients with POAG?	NA	No reviews available
Is laser trabeculectomy effective in patients with POAG who cannot or will not use medications reliably?	NA	No reviews available
Is laser trabeculoplasty effective in reducing the amount of medical treatment required in patients with POAG?	NA	No reviews available
Is repeated laser trabeculoplasty effective and safe in lowering IOP in patients with POAG who have not responded adequately to the first laser surgery?	NA	No reviews available
Does preoperative care (such as performing preoperative evaluation, measuring IOP, and obtaining informed consent) result in better outcomes in patients scheduled to undergo laser trabeculoplasty?	NA	No reviews available
Does postoperative care (such as checking IOP within 30–120 min of surgery or follow-up examination within 6 wk of surgery) result in better outcomes in patients who undergo laser trabeculoplasty?	NA	No reviews available
Are perioperative medications that are not received long-term effective in prevent- ing temporary IOP elevations in patients who undergo laser trabeculoplasty?	NA	No reviews available
Are ≥1 repeated filtering surgeries effective and safe for patients with POAG who do not respond to an initial filtering procedure?	NA	No reviews available
Is filtering surgery effective in eyes that have had cataract surgery involving the conjunctiva?	NA	No reviews available
Is intraoperative 5-fluorouracil therapy effective and safe in improving the success rate of primary and repeated filtering surgery?	NA	No reviews available
Is postoperative mitomycin C effective and safe in improving the success rate of primary and repeated filtering surgery?	NA	No reviews available
Is viscocanalostomy effective in lowering IOP in patients with POAG?	NA	No reviews available
Is nonpenetrating deep sclerectomy effective and safe in lowering IOP in patients with POAG?	NA	No reviews available
What is the relative effectiveness of viscocanalostomy compared with trabeculectomy in lowering IOP in patients with POAG?	NA	No reviews available
What is the relative effectiveness of nonpenetrating deep sclerectomy compared with trabeculectomy in lowering IOP in patients with POAG?	NA	No reviews available
Is sequential filtration and cataract surgery more effective than combined procedures in lowering IOP in patients with POAG and cataracts?	NA	No reviews available
Is mitomycin C effective in lowering IOP in combined glaucoma and cataract procedures?	NA	No reviews available
Are combined glaucoma and cataract procedures with separate incisions more effective than a 1-site incision?	NA	No reviews available
Is cataract surgery alone effective in lowering IOP in patients with POAG and cataracts?	NA	No reviews available
Does preoperative care by the surgeon (such as performing preoperative evaluation or obtaining informed consent) result in better outcomes in patients scheduled to undergo filtering surgery?	NA	No reviews available
Are cyclodestructive procedures effective and safe for treating patients with POAG who are poor incisional surgical candidates, have limited visual potential, or have undergone several previous glaucoma operations?	NA	No reviews available
In patients with stable optic nerve status and low IOP, is discontinuing medication (or upward adjustment of IOP) effective and safe in terms of disease progression?	NA	No reviews available
What is the optimal interval for follow-up visits to assess the response and side effects from washout of the old medication and onset of maximum effect of the new medication?	NA	No reviews available

IOP = intraocular pressure; NA = not applicable; NTG = normal-tension glaucoma; OAG = open-angle glaucoma; OHT = ocular hypertension; POAG = primary open-angle glaucoma.

^*A systematic review could be mapped to ≥1 clinical question depending on the breadth of the review topic.

^†The Cochrane systematic review included an updated search after its initial publication.

Resource Requirements to Complete the Prioritization Project

We report the approximate time associated with each step and staff needed to complete the prioritization project (Figure 1 also lists stakeholder involvement).

For phase 1, deriving and refining clinical questions from CPGs took 80 hours. Systematic review methodologists, epidemiologists, and clinical trialists derived the questions from the CPG. Glaucoma specialists with expertise in clinical trials and systematic reviews and methodologists from the CPG development committee verified and refined the questions. Prioritizing the clinical questions took 200 hours, including the time for creating the survey questionnaire, conducting pilot testing, and administering the survey. In addition, we estimate that it took 30 minutes for each participating member of the American Glaucoma Society to complete our prioritization survey. The online survey interface and underlying database were created with donated service from a data management specialist. The survey was administered and data were managed by a graduate student.

For phase 2, we cannot estimate the time for identifying existing systematic reviews because this step was part of another project, for which we searched for and identified systematic reviews covering all eye and vision topics for a subject-specific database. Systematic review methodologists worked together with information specialists to complete this step. Extracting data and appraising the methodological quality of the systematic reviews took 3 hours per included review for data extraction, verification, and adjudication of discrepancies. A senior epidemiologist oversaw the process. Identifying the evidence gaps and prioritizing CER took 120 hours. Systematic review methodologists, epidemiologists, clinical trialists, and glaucoma specialists worked together in identifying evidence gaps. Finally, the time for drafting and revising the final report or manuscript will vary depending on the ultimate goal of the prioritization project, the sponsor’s requirements, and the publication process.

Discussion

Our overall goal was to test a framework for prioritizing clinical questions and identifying evidence gaps by using existing systematic reviews and CPGs. The framework we tested includes the following steps: deriving clinical research questions from CPGs to reflect issues that clinicians encounter frequently, asking clinicians to prioritize the 45 questions for research to incorporate opinions from evidence users, and determining whether high-quality systematic reviews of all previous research exist for each clinical question and identifying evidence gaps. By mapping evidence gaps to clinicians’ priorities, we proposed a CER agenda.

Our experience demonstrated the feasibility of engaging key stakeholders in setting priorities at several junctures. We collaborated with practicing clinicians, researchers, CPG developers (AAO), and a relevant professional association (American Glaucoma Society) (9). In phase 1, we asked clinicians which questions they needed answered to practice effectively. Patient views could easily be incorporated into this phase. In phase 2, we identified the questions for which research is needed. Clinicians and patients may be in the best position to decide what is needed in clinical practice but may not be aware of or may discount existing research for various reasons (for example, because the patient population differs from the clinician’s own). Thus, a 2-phase framework that engages stakeholders with different perspectives and expertise is essential for setting CER priorities.

Our framework for setting CER priorities provides a clear guide for the type of research needed to address each clinical question (such as studies to synthesize existing evidence or to generate new evidence). We found that high-quality systematic reviews were needed to address 32 of the 45 clinical questions on the management of POAG, including 4 of the 9 high-priority clinical questions. We also identified 2 clinical questions for which evidence from new RCTs is needed because the existing reviews were not conclusive. However, a good practice would be to update the existing high-quality systematic reviews to include recent and ongoing research as a confirmatory step before initiating any new RCTs.

The framework we tested is a robust addition to existing approaches for setting CER priorities (2, 62–70). It differs from nomination-based methods, such as those used by the Agency for Healthcare Research and Quality (62, 63) and the Institute of Medicine (64), in which research topics are suggested by clinicians, payers, members of the public, or others, and a set of rules are applied to determine the final priorities. These other methods are quite broad and have produced overall priorities for health care research (71). In contrast, our framework focuses on a specialty area and thus addresses the entire range of questions specialists face.

Our framework has limitations. When translating CPG recommendations into answerable clinical questions, we relied on interventions and outcome measures that were stated explicitly in the CPG. This may be of concern when guideline recommendations are out of date or do not address emerging research, techniques, practices, or outcomes considered important and relevant to clinicians and patients. We addressed this concern by asking stakeholders to supplement and refine the list of derived clinical questions as part of survey development. For our ongoing project in which we are applying our framework to diabetic retinopathy, a condition in which new technologies and drugs are developing rapidly, we are seeking the involvement of both clinical experts and patients to develop clinical questions to supplement those derived from CPGs. In contrast, when we applied our framework to interventions for POAG, we found no new interventions in the 2010 update of the 2005 guidelines by the AAO. In addition, we have involved stakeholders from regions where the condition is prevalent to ensure broad experience with the topic (Yu T, Li T, Friedman DS, Puhan MA, Dickersin K. Setting priorities for comparative effectiveness research on the management of primary angle closure: a survey of Asia-Pacific clinicians. In preparation). In a third example, we are planning to seek input from patients for research questions on dry eye, for which patient-reported outcomes are of primary interest. We encourage others to test our framework by extending our methods according to the topic and their needs.

Our framework relies on published systematic reviews; thus, the underlying trial data may not be completely up to date. We believe timeliness is a problem regardless of the prioritization method used; this is addressed to some extent when prioritized questions are first reported in a clinical journal because the community has an opportunity to assess the applicability of the question to current practice and knowledge. Our prioritization framework is less sensitive to timeliness than the questions themselves. For example, in our study of diabetic retinopathy, involving clinicians and patients will almost certainly result in adding clinical questions about the effectiveness of antivascular endothelial growth factor interventions for treating diabetic macular edema to those derived from CPGs. The trials on this topic are fairly recent, and existing systematic reviews may not yet include them. Thus, we suggest that CER researchers update existing systematic reviews before undertaking a new trial.

We focused on RCTs and systematic reviews of RCTs because we were interested in clinical questions of intervention effectiveness. Other study types may be equally or more appropriate in some instances, such as for questions about intervention harms. In addition, for questions that involve both effectiveness and harm, existing data may be conclusive for benefit but not for harm or conclusive for intervention effectiveness on some but not all outcomes. For example, we classified the main review findings about effectiveness as conclusive for our priority question on the effectiveness and safety of intraoperative mitomycin C; however, we also concluded that additional data may be needed to clarify certain serious but rare side effects, such as endophthalmitis (Table 2). The clinical and research community will need to decide when unresolved questions of harm are sufficiently important to warrant undertaking a new clinical trial or observational study.

Although we searched 3 major databases with a comprehensive and validated search strategy to identify systematic reviews, we did not search for unpublished systematic reviews, which may have a different methodological quality profile and findings from those published (72). In addition, our assessment of methodological quality relied on what the authors reported in the article, which may not reflect the actual methods used. Finally, if systematic review authors made errors when abstracting trial data, we could have underestimated the number of unreliable systematic reviews (73–78).

Our findings show that although systematic reviews of RCTs are considered the highest level of evidence for questions about intervention effectiveness (7), their findings should not be accepted without critical evaluation of their methodological rigor. Consistent with others (79– 83), we identified a large proportion of systematic reviews that did not use rigorous methods, such as those suggested by the Institute of Medicine and others (84). We classified the findings of two thirds of the reviews as at high risk of bias for either having a noncomprehensive search for relevant studies or using inappropriate statistical methods. The negative impact of these methodological problems has been well-described in the literature (20, 85–94).

The clinical questions we prioritized are restricted to pairwise comparisons and did not directly answer the question, “Which intervention works best?” because of the way clinical questions are addressed in most RCTs. Because this question must be answered to facilitate optimal health care decision making, a network meta-analysis should be considered once data from all high-quality systematic reviews and trials are identified for a given question (95). Our framework, which identifies all high-quality systematic reviews related to a health topic generally (96, 97), has an important scoping role in this regard.

We believe that our framework, which uses CPGs and systematic reviews to prioritize CER involving both new primary research and systematic reviews, is broadly applicable and could be applied in conjunction with other methods (for example, by PCORI) to formulate future research agendas.

Appendix 1: Search Strategies for Identifying Systematic Reviews on Eye and Vision Research Questions

PubMed Search Strategies

(ABNORMAL ACCOMMODATION[tiab] OR Abnormal color vision[tiab] OR ABNORMAL LACRIMATION[tiab] OR Abnormal vision [tiab] OR accommodative disorders [tiab] OR Amblyopia [tiab] OR Ametropia [tiab] OR ANISOCORIA-[tiab] OR ANOPHTHALMIA[tiab] OR Anterior CHAMBER hemorrhage [tiab] OR Aphakia[tiab] OR aqueous outflow obstruction [tiab] OR Asthenopia [tiab] OR Balint’s syndrome [tiab] OR Bilateral visual field constriction [tiab] OR Binocular Vision Disorder[tiab] OR BLEPHARITIS [tiab] OR BLEPHAROSPASM-[tiab] OR BLINDNESS [tiab] OR blurred vision [tiab] OR CATARACT [tiab] OR Cataracts [tiab] OR Chorioretinal disorder [tiab] OR Chorioretinitis [tiab] OR Choroid Diseases [tiab] OR Choroidal [tiab] OR Choroiditis [tiab] OR CHROMATOPSIA-[tiab] OR Color Blindness[tiab] OR Color Vision Defects [tiab] OR Color vision deficiency [tiab] OR Colour blindness [tiab] OR Conjunctival Diseases [tiab] OR CONJUNCTIVAL HAEMOR-RHAGE[tiab] OR Conjunctival Injury[tiab] OR CONJUNCTIVAL ULCERATION [tiab] OR CONJUNCTIVITIS [tiab] OR CORNEAL DEPOSITS [tiab] OR Corneal Diseases [tiab] OR Corneal Disorder [tiab] OR Corneal injuries [tiab] OR Corneal Injury[tiab] OR CORNEAL OEDEMA[tiab] OR CORNEAL OPACITY[tiab] OR CORNEAL ULCERATION [tiab] OR decreased Lacrimation [tiab] OR Decreased vision [tiab] OR defective vision [tiab] OR Delayed visual maturation [tiab] OR Difficulty seeing[tiab] OR difficulty with vision [tiab] OR Dim vision [tiab] OR Diminished Vision [tiab] OR DIPLOPIA[tiab] OR disturbed vision [tiab] OR ENDOPHTHALMITIS [tiab] OR EPIPHORA[tiab] OR Episcleritis [tiab] OR Equatorial staphyloma [tiab] OR EXOPHTHALMOS [tiab] OR Eye Abnormalities [tiab] OR EYE ABNORMALITY[tiab] OR Eye Burns-[tiab] OR Eye disease [tiab] OR Eye Diseases [tiab] OR Eye dis-order[tiab] OR Eye disorders [tiab] OR Eye edema [tiab] OR Eye Foreign Bodies[tiab] OR Eye Hemorrhage [tiab] OR EYE INFECTION [tiab] OR Eye Infections [tiab] OR Eye Injuries [tiab] OR EYE MALFORMATION[tiab] OR Eye Manifestations-[tiab] OR EYE MUSCLE PARALYSIS [tiab] OR Eye Neoplasm-[tiab] OR Eye Neoplasms [tiab] OR EYE PAIN[tiab] OR Eye swelling [tiab] OR Eyelid Disease[tiab] OR Eyelid Diseases [tiab] OR Eyelid Disorder[tiab] OR Eyelid Disorders [tiab] OR Eyelid pain[tiab] OR EYELID RETRACTION [tiab] OR FIXED PUPILS [tiab] OR GLAUCOMA[tiab] OR Glaucomas [tiab] OR Hazy vision [tiab] OR HEMIANOPIA[tiab] OR Hemianopsia-[tiab] OR Hepatolenticular Degeneration [tiab] OR Horner’s syndrome [tiab] OR HYPOPYON [tiab] OR Impaired vision-[tiab] OR impaired visual acuity[tiab] OR Interference with vision [tiab] OR IRITIS [tiab] OR KERATITIS [tiab] OR KERATOCONJUNCTIVITIS [tiab] OR Keratoconus[tiab] OR Lacrimal Apparatus Diseases [tiab] OR lacrimal disorder [tiab] OR LACRIMAL DUCT OBSTRUCTION [tiab] OR legally blind[tiab] OR Lens Diseases [tiab] OR Lens Disorder[tiab] OR Lens Opacities [tiab] OR Lens Subluxation [tiab] OR Localized anterior staphyloma [tiab] OR Low vision [tiab] OR MACULAR DEGENERATION [tiab] OR MEIBOMIANITIS[tiab] OR Metastases to eye[tiab] OR MIOSIS[tiab] OR MYDRIASIS-[tiab] OR MYOPIA[tiab] OR Night Blindness[tiab] OR Nystagmus [tiab] OR Ocular degeneration [tiab] OR Ocular discomfort [tiab] OR Ocular disease [tiab] OR OCULAR HAEMORRHAGE [tiab] OR OCULAR HEMORRHAGE [t-iab] OR OCULAR HERPES [tiab] OR Ocular Hypertension-[tiab] OR Ocular Hypotension [tiab] OR Ocular infections [tiab] OR Ocular inflammations[tiab] OR Ocular injuries [tiab] OR Ocular Injury[tiab] OR Ocular Motility Disorders[tiab] OR Ocular neoplasms [tiab] OR oculopathy [tiab] OR Open wound of ocular adnexa[tiab] OR OPHTHALMIC DISORDERS [tiab] OR Ophthalmological disorder [tiab] OR Ophthalmopathy-[tiab] OR OPTIC ATROPHY[tiab] OR Optic Nerve Diseases-[tiab] OR optic nerve disorder [tiab] OR Optic nerve injury[tiab] OR OPTIC NEURITIS [tiab] OR Orbital Diseases [tiab] OR PAPILLOEDEMA[tiab] OR Partial sight [tiab] OR Partial vision loss[tiab] OR Partially Sighted[tiab] OR Penetrating Eye Injuri-es[tiab] OR Periorbital fat herniation [tiab] OR Photalgia[tiab] OR PHOTOPHOBIA[tiab] OR PHOTOPSIA[tiab] OR PIGMENT PRECIPITATION[tiab] OR Poor vision[tiab] OR Posterior dislocation of lens[tiab] OR Posterior synechiae[tiab] OR Problem seeing [tiab] OR Proliferative Vitreoretinopathy[tiab] OR Pupil Disorders[tiab] OR Redness[All Fields] OR discharge of eye[tiab] OR Reduced ability to see[tiab] OR Reduced Vision-[tiab] OR Refraction Errors [tiab] OR Refractive disorders [tiab] OR Refractive Errors[tiab] OR Retinal defects [tiab] OR RETINAL DEPOSITS [tiab] OR RETINAL DETACHMENT [tiab] OR Retinal detachments [tiab] OR Retinal Disease[tiab] OR Retinal Diseases[tiab] OR RETINAL DISORDER[tiab] OR RETINAL EDEMA[tiab] OR RETINAL HAEMORRHAGE [tiab] OR RETINAL HEMORRHAGE [tiab] OR RETINAL OEDE-MA[tiab] OR RETINITIS [tiab] OR RETINOBLASTOMA-[tiab] OR retinopathy[tiab] OR RETROBULBAR NEURITIS-[tiab] OR Scleral Diseases [tiab] OR Scleral staphyloma [tiab] OR SCLERITIS[tiab] OR SCOTOMA[tiab] OR Sight impaired-[tiab] OR Staphyloma posticum[tiab] OR STRABISMUS [tiab] OR Subnormal Vision [tiab] OR SUNKEN EYES [tiab] OR SYMBLEPHARON[tiab] OR THROMBOSIS RETINAL VEIN [tiab] OR Tunnel Vision [tiab] OR Uveal Diseases [tiab] OR Uveal Disorder[tiab] OR UVEITIS [tiab] OR vision defects-[tiab] OR vision disorder [tiab] OR Vision Disorders [tiab] OR vision disturbance [tiab] OR vision impaired[tiab] OR VISION IMPAIRMENT[tiab] OR Vision problem[tiab] OR Visual Ag-nosia[tiab] OR visual defect[tiab] OR Visual difficulty[tiab] OR Visual disorders[tiab] OR Visual disturbance [tiab] OR VISUAL FIELD DEFECT[tiab] OR Visual Field Disorder[tiab] OR Visual field disorders [tiab] OR Visual impairment [tiab] OR Visual loss[tiab] OR Visual Pathway Disorder[tiab] OR Visual pathway disorders [tiab] OR Visual system disorder [tiab] OR VITREOUS DETACHMENT[tiab] OR VITREOUS HAEMORRHAGE-[tiab] OR Vitreous Hemorrhage [tiab] OR Vitreous membranes and strands [tiab] OR Vitreous prolapse [tiab] OR Vitreous sy-neresis[tiab] OR Wavefront aberration [tiab] OR Weak vision-[tiab] OR Wilson’s Disease[tiab] OR XEROPHTHALMIA[tiab]

OR

((accommodation[tiab] OR aqueous humor[tiab] OR binocular movement [tiab] OR Capsule [tiab] OR choroid [tiab] OR ciliary body[tiab] OR conjunctiva[tiab] OR Cornea[tiab] OR (eye[tiab] AND adnexa[tiab]) OR eye[tiab] OR eyelid[tiab] OR globe[tiab] OR intraocular pressure [tiab] OR Iris[tiab] OR lacrimal passage [tiab] OR lacrimal system [tiab] OR lens [tiab] OR ocular adnexa[tiab] OR optic nerve[tiab] OR orbit[tiab] OR refraction [tiab] OR retina[tiab] OR sclera[tiab] OR trochlear nerve[tiab] OR uvea[tiab] OR Uveal Tract [tiab] OR vision [tiab] OR visual pathway[tiab] OR vitreous body[tiab] OR vitreous humor [tiab])

AND

(“disease”[MeSH Terms] OR Disease[Text Word]) OR ((“disease”[TIAB] NOT Medline[SB]) OR “disease”[MeSH Terms] OR Diseases [Text Word]) OR ((“disease” [TIAB] NOT Medline[SB]) OR “disease”[MeSH Terms] OR Disorder[Text Word]) OR ((“disease”[TIAB] NOT Medline[SB]) OR “disease-”[MeSH Terms] OR Disorders [Text Word]) OR ((“contusion-s”[TIAB] NOT Medline[SB]) OR “contusions”[MeSH Terms] OR Contusion [Text Word]) OR (“contusions” [MeSH Terms] OR Contusions [Text Word]) OR (“syndrome” [MeSH Terms] OR Syndrome[Text Word]) OR ((“syndrome”[TIAB] NOT Med-line[SB]) OR “syndrome” [MeSH Terms] OR Syndromes [Text Word]) OR (“dislocations”[MeSH Terms] OR Dislocation-s[Text Word]) OR ((“dislocations”[TIAB] NOT Medline[SB]) OR “dislocations” [MeSH Terms] OR Dislocation [Text Word]) OR (((“blood vessels”[TIAB] NOT Medline[SB]) OR “blood vessels”[MeSH Terms] OR Vascular [Text Word]) AND Occlusion [All Fields]) OR (((“blood vessels”[TIAB] NOT Medline[SB]) OR “blood vessels”[MeSH Terms] OR Vascular [Text Word]) AND Occlusions [All Fields]) OR ((“wounds and injuries” [TIAB] NOT Medline [SB]) OR “wounds and injuries” [MeSH Terms] OR Injury[Text Word]) OR (“injuries”[Subheading] OR (“wounds and injuries”[TIAB] NOT Medline[SB]) OR “wounds and injuries”[MeSH Terms] OR Injuries[Text Word]) OR (“coloboma”[MeSH Terms] OR Coloboma[Text Word]))

OR

“amaurosis fugax”[MeSH Terms] OR “amblyopia”[MeSH Terms] OR “asthenopia”[MeSH Terms] OR “blindness”[MeSH Terms] OR “blindness, cortical” [MeSH Terms] OR “color vision defects” [MeSH Terms] OR “conjunctival diseases” [MeSH Terms] OR “corneal diseases” [MeSH Terms] OR “diplopi-a”[MeSH Terms] OR “eye abnormalities”[MeSH Terms] OR “eye burns”[MeSH Terms] OR “eye diseases”[MeSH Terms] OR “eye diseases, hereditary” [MeSH Terms] OR “eye foreign bodies” [MeSH Terms] OR “eye hemorrhage” [MeSH Terms] OR “eye infections”[MeSH Terms] OR “eye injuries”[MeSH Terms] OR “eye injuries, penetrating” [MeSH Terms] OR “eye manifestations” [MeSH Terms] OR “eye neoplasms” [MeSH Terms] OR “eyelid diseases” [MeSH Terms] OR “hemianopsia” [MeSH Terms] OR “lacrimal apparatus diseases” [MeSH Terms] OR “lens diseases”[MeSH Terms] OR “night blindness”[MeSH Terms] OR “ocular hypertension”[MeSH Terms] OR “ocular hypotension”[MeSH Terms] OR “ocular motility disorders” [MeSH Terms] OR “optic nerve diseases”[MeSH Terms] OR “orbital diseases”[MeSH Terms] OR “photophobia”[MeSH Terms] OR “pupil disorders” [MeSH Terms] OR “refractive errors” [MeSH Terms] OR “retinal diseases” [MeSH Terms] OR “scleral diseases” [MeSH Terms] OR “scotoma” [MeSH Terms] OR “uveal diseases”[MeSH Terms] OR “vision disorders”[MeSH Terms] OR “vision disorders”[MeSH Terms] OR “vitreoreti-nopathy, proliferative” [MeSH Terms] OR “vitreous detachment” [MeSH Terms])

AND

(“therapy” [Subheading] OR (“therapeutics” [TIAB] NOT Medline[SB]) OR “therapeutics”[MeSH Terms] OR treatment-[Text Word]) OR ((“therapeutics”[TIAB] NOT Medline[SB]) OR “therapeutics” [MeSH Terms] OR treatments [Text Word]) OR (“diagnosis” [Subheading] OR “diagnosis” [MeSH Terms] OR diagnosis [Text Word]) OR intervention [All Fields] OR interventions [All Fields] OR (“prevention and control” [Subheading] OR prevention [Text Word])

AND

(Humans[MeSH] OR (Humans[Mesh] NOT Animals [MeSH])

AND

cochrane database syst rev[ta] OR search[tiab] OR metaanalysis [pt] OR MEDLINE [tiab] OR (systematic [tiab] AND re-view[tiab])

EMBASE Search Strategies

((*’asthenopia’*/de OR *’asthenopia’*) OR (*’cerebral blind-ness’*/de OR *’cerebral blindness’*) OR (*’diplopia’*/de OR *’diplopia’*) OR (*’eye malformation’*/de OR *’eye malformation’*) OR (*’eye burn’*/de OR *’eye burn’*) OR (*’eye injury’*/de OR *’eye injury’*) OR (*’perforating eye injury’*/de OR *’perforating eye injury’*) OR (*’hemianopia’*/de OR *’hemianopia’*) OR (*’lacrimal gland disease’*/de OR *’lacrimal gland disease’*) OR (*’intraocular hypotension’*/de OR *’intra-ocular hypotension’*) OR (*’photophobia’*/de OR *’photopho-bia’*) OR (*’pupil disease’*/de OR *’pupil disease’*) OR (*’sclera disease’*/de OR *’sclera disease’*) OR (*’scotoma’*/de OR *’scotoma’*) OR (*’vitreoretinopathy’*/de OR *’vitreoretinopathy’*) OR (*’vitreous body detachment’*/de OR *’itreous body detachment’*) OR (*’ransitional blindness’*/de OR *’ransitional blindness’*) OR (*’mblyopia’*/de OR *’mblyopia’*) OR (*’blindness’*/de OR *’blindness’*) OR (*’color vision defect’*/de OR *’color vision defect’*) OR (*’conjunctiva disease’*/de OR *’conjunctiva disease’*) OR (*’cornea disease’*/de OR *’cornea disease’*) OR (*’eye disease’*/de OR *’eye disease’*) OR (*’intraoc-ular foreign body’*/de OR *’intraocular foreign body’*) OR (*’intraocular hemorrhage’*/de OR *’intraocular hemorrhage’*) OR (*’eye infection’*/de OR *’eye infection’*) OR (*’eye tu-mor’*/de OR *’eye tumor’*) OR (*’eyelid disease’*/de OR *’eye-lid disease’*) OR (*’lens disease’*/de OR *’lens disease’*) OR (*’night blindness’*/de OR *’night blindness’*) OR (*’intraocular hypertension’*/de OR *’intraocular hypertension’*) OR (*’eye movement disorder’*/de OR *’eye movement disorder’*) OR (*’optic nerve disease’*/de OR *’optic nerve disease’*) OR (*’or-bit disease’*/de OR *’orbit disease’*) OR (*’refraction error’*/de OR *’refraction error’*) OR (*’retina disease’*/de OR *’retina disease’*) OR (*’uvea disease’*/de OR *’uvea disease’*) OR (*’visual disorder’*/de OR *’visual disorder’*)) AND ((*search*) OR (*’meta analysis’* OR *’systematic review’*))

The Cochrane Library Search Strategies

We searched the Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, Health Technology Assessment database, and the United Kingdom National Health Service Economic Evaluation Database in the Cochrane Library by using the following Medical Subject Heading terms as terms and as keywords in “Title, Abstract, and Keyword”: amaurosis fugax; amblyopia; blindness; color vision defects; conjunctival diseases; corneal diseases; eye diseases; eye diseases, hereditary; eye foreign bodies; eye hemorrhage; eye infections; eye injuries; eye manifestations; eye neoplasms; eyelid diseases; lens diseases; night blindness; ocular hypertension; ocular motility disorders; optic nerve diseases; orbital diseases; refactive errors; retinal diseases; uveal diseases; and vision disorders.

We also searched the following non—Medical Subject Heading terms in “Title, Abstract, and Keyword”: vision, visual, eye diseases, eye disorders, macular degeneration, cataract, glaucoma, strabismus, hyphema, conjunctivitis, eye, optic nerve, choroiditis, chorioretinitis, blepharitis, blindness, ocular, lacrimal, iris, uveitis, pupil, and retina.

Appendix 2: Statistical Issues and Impact on Systematic Review Findings

Thirteen of the 32 systematic reviews with at least 1 metaanalysis that we included in our sample applied at least 1 inappropriate statistical analysis. We identified 2 major types of inappropriate analysis among the 13 reviews. Nine reviews pooled data from similar groups across studies, resulting in a nonrandom comparison, and 7 used an incorrect formula to calculate the variance of the effect estimate (3 reviews had both types of inappropriate analyses).

Using a Naive Approach to Conduct an Indirect Comparison and Network Meta-analysis

Nine of the 32 systematic reviews with a meta-analysis that we included in our sample pooled data from treatment groups (for example, 1 review [39] pooled data separately from the timolol treatment groups from 21 trials and latanoprost treatment groups from 33 trials and compared them [Appendix Figure 2]). In all 9 cases, the reviews examined at least 3 interventions for a given condition. The analysis that pooled individual treatment groups may have been an attempt to indirectly compare interventions; that is, comparison of 2 interventions through a common comparator (20, 95). For example, indirect evidence on A versus C can be obtained from RCTs of either A or C versus a common comparator B.

Appendix Figure 2.

Example of pooling data from treatment groups.

IOP = intraocular pressure. Reproduced from Holmstrom and colleagues (39) with permission of LibraPharm Ltd.

The consensus among methodologists is that pooling treatment groups is an incorrect way of conducting a meta-analysis (20, 98). Pooling treatment groups has been described by Glenny and colleagues (20) as the “naive method” for indirect comparison. This approach has been criticized for negating the randomized nature of the comparison between interventions and thereby increasing the likelihood of selection and confounding biases and falsely precise estimates. The correct way of comparing multiple treatments for the same condition is through a network metaanalysis (20, 95, 98, 99).

Using an Incorrect Formula to Calculate the Variance of the Effect Estimate

Many systematic reviews included in our sample described differences between the intervention and comparison groups in mean percentage of change in IOP from baseline to the end point as a measure of treatment effect and to gauge the success of IOP-lowering treatments. Mean percentage of change in IOP from baseline can be expressed as follows:

$$\frac{{\sum }_{i=1}^n\left(\frac{{\mathit{\text{IOP}}}_{\mathit{\text{baseline}}i}-{\mathit{\text{IOP}}}_{\mathit{\text{follow}}-\mathit{\text{up}}i}}{{\mathit{\text{IOP}}}_{\mathit{\text{baseline}}i}}\right)}{n}\times 100\%$$

Where i indicates individual participant and n indicates the total number of participants.

We found that incorrect formulas, in different forms, were used for estimating the SD for the percentage of change in IOP from baseline to the end point (Appendix Figures 3 and 4). Incorrect formulas for estimating the SD can invalidate results and conclusions from a review, because the SD directly relates to the SE through sample size. The SE is used to determine the weight of each study in the meta-analysis and to calculate the CIs for the pooled treatment effect.

Appendix Figure 3.

Example of an incorrect formula used to estimate the SD of the percentage of change in IOP from baseline.

IOP = intraocular pressure. Reproduced from Zhang and colleagues (61) with permission from the BMJ Publishing Group Ltd.

Appendix Figure 4.

Example of transcription and use of an incorrect formula to estimate the SD of the percentage of change in IOP from baseline.

IOP = intraocular pressure. Reproduced from Cheng and colleagues (19) with permission. The publisher for this copyrighted material is Mary Ann Liebert, Inc. Publishers.

The formulas to calculate the SD of the absolute change in IOP from baseline to the end point were incorrect in Appendix Figures 3 (61) and 4 (19) because the analyses ignored the correlation between the repeated IOP measurements from the same individual at baseline and the end point. In Appendix Figure 3, the authors used the SE instead of the SD, which could be a notation error. The incorrect formulas used to calculate the SD of change in IOP from baseline to the end point were expressed as

$${\mathit{\text{SE}}}_{\mathit{\text{IOPR}}}=\sqrt{{\mathit{\text{SE}}}_{\mathit{\text{baseline}}}^2+{\mathit{\text{SE}}}_{\mathit{\text{endpoint}}}^2}$$

and

$${\mathit{\text{SD}}}_{\mathit{\text{IOPR}}}=\sqrt{{\mathit{\text{SD}}}_{\mathit{\text{baseline}}}^2+{\mathit{\text{SD}}}_{\mathit{\text{endpoint}}}^2-{\mathit{\text{SD}}}_{\mathit{\text{baseline}}}\times {\mathit{\text{SD}}}_{\mathit{\text{endpoint}}}}$$

in Appendix Figures 3 and 4, respectively. The correct formula could be written as

$${\mathit{\text{SD}}}_{\mathit{\text{IOPR}}}=\sqrt{{\mathit{\text{SD}}}_{\mathit{\text{baseline}}}^2+{\mathit{\text{SD}}}_{\mathit{\text{endpoint}}}^2-2\times \rho \times {\mathit{\text{SD}}}_{\mathit{\text{baseline}}}\times {\mathit{\text{SD}}}_{\mathit{\text{endpoint}}}}$$

in which p is the correlation between the repeated IOP measurements from the same individual at baseline and the end point.

When individual-patient data are available, that is, a data set with intervention, outcome, and covariate information for each patient, another way to compute SD_IOPR is to take the differences (end point minus baseline) for each individual and compute the SD of these values.

The formulas to calculate the SD of the percentage of change in IOP from baseline to the end point were also incorrect in Appendix Figures 3 and 4 because the percentage of change in IOP from baseline to the end point is a ratio of 2 quantities, and estimating the SD for a ratio is mathematically complex (100). The incorrect formulas used to calculate the SD of the percentage of change in IOP from baseline to the end point were expressed as

$${\mathit{\text{SE}}}_{\mathit{\text{IOPR}}\%}={\mathit{\text{SE}}}_{\mathit{\text{IOPR}}}/{\mathit{\text{IOP}}}_{\mathit{\text{baseline}}}$$

and

$${\mathit{\text{SD}}}_{\mathit{\text{IOPR}}\%}={\mathit{\text{SD}}}_{\mathit{\text{IOPR}}}/{\mathit{\text{IOP}}}_{\mathit{\text{baseline}}}$$

in Appendix Figures 3 and 4, respectively.

If all trials included in the meta-analysis had reported the mean percentage of change in IOP from baseline to the end point and SD for each treatment group, or for between-group comparisons, then using these data to conduct a meta-analysis would have been straightforward. However, because 1 or more of the included trials did not report mean percentage of change in IOP from baseline and its SD, accurate estimation of these values requires individual-patient data.

Using percentage of change in IOP from baseline as an outcome in a meta-analysis is recommended only when individual-patient data are available and only with advice from an experienced statistician because of the myriad potential problems associated with the assumptions and the realities of available data. The clinical community should reach a consensus on whether percentage of change in IOP from baseline is an appropriate measure of treatment effect for IOP-lowering interventions. Such a consensus would inform design, analyses, and reporting for clinical trials of interventions that affect IOP. This discussion has been initiated through the National Eye Institute/U.S. Food and Drug Administration symposium on glaucoma clinical trial design and end points (101).