Need for Quality Improvement in Renal Systematic Reviews

Abstract

Background and objectives: Systematic reviews of clinical studies aim to compile best available evidence for various diagnosis and treatment options. This study assessed the methodologic quality of all systematic reviews relevant to the practice of nephrology published in 2005.

Design, setting, participants, & measurements: We searched electronic databases (Medline, Embase, American College of Physicians Journal Club, Cochrane) and hand searched Cochrane renal group records. Clinical practice guidelines, case reports, narrative reviews, and pooled individual patient data meta-analyses were excluded. Methodologic quality was measured using a validated questionnaire (Overview Quality Assessment Questionnaire). For reviews of randomized trials, we also evaluated adherence to recommended reporting guidelines (Quality of Reporting of Meta-Analyses).

Results: Ninety renal systematic reviews were published in year 2005, 60 of which focused on therapy. Many systematic reviews (54%) had major methodologic flaws. The most common review flaws were failure to assess the methodologic quality of included primary studies and failure to minimize bias in study inclusion. Only 2% of reviews of randomized trials fully adhered to reporting guidelines. A minority of journals (four of 48) endorsed adherence to consensus guidelines for review reporting, and these journals published systematic reviews of higher methodologic quality (P < 0.001).

Conclusions: The majority of systematic reviews had major methodologic flaws. The majority of journals do not endorse consensus guidelines for review reporting in their instructions to authors; however, journals that recommended such adherence published systemic reviews of higher methodologic quality.

Physicians make better clinical decisions when they understand the circumstances and preferences of their patients and combine their personal experience with clinical evidence supporting the available options (1). The public also expects their physicians to take up new research findings in a timely way (2). Systematic reviews of clinical studies aim to compile the best available evidence for various diagnosis and treatment options (3), and the number of systematic reviews has increased in the past decade (4). Physicians rate systematic reviews as more relevant than original articles for clinical practice (5), presumably because reading a systematic review is a more efficient way of keeping up to date.

A well-conducted review summarizes the best available evidence for a specific question, highlights the strengths and weaknesses of the evidence, and identifies new focuses for future research (3). A meta-analysis, when used appropriately, can achieve statistical power to detect an overall effect that was not observed in each of the smaller primary studies.

The quality of a systematic review depends on the quality of the primary studies cited in the review. The results can also be influenced by the methods used to compile and interpret the primary studies and by inappropriately pooling the data. The review team can potentially arrive at the wrong conclusion (6,7). In an attempt to prevent this problem, guidelines for the reporting of systematic reviews have been published. Quality of Reporting of Meta-Analyses (QUOROM) guidelines were published in 1999, describing recommended methods for conducting and reporting a systematic review of randomized, controlled trials (8), and Meta-analysis of Observational Studies in Epidemiology (MOOSE) consensus guidelines were published in 2000 (9); however adherence to these guidelines has been variable. Previous studies in the fields of anesthesia (10), general surgery (11), critical care (12), emergency medicine (13), and oncology (14) showed that a majority of systematic reviews contained major methodologic flaws.

To date, no study has examined whether systematic reviews of higher methodologic quality are more likely to be published in journals that endorse reporting guidelines. We examined this issue in this analysis of recently published systematic reviews relevant to the practice of nephrology. We also determined where renal systematic reviews are published and their quality and other factors associated with high-quality reviews.

Materials and Methods

Reviews Eligible for Analysis

We defined a systematic review as a full-text article that attempted to answer a focused clinical question by examining previously published evidence identified through a protocol-driven process. Case reports, clinical practice guidelines, narrative reviews, and pooled individual patient data meta-analyses in the absence of a comprehensive review of the literature all were excluded.

Systematic reviews were deemed relevant to nephrology when they described one of the following content areas: Kidney failure; renal transplantation; chronic kidney disease; glomerular disease; reflux nephropathy; metabolic, acid/base, or water disturbances; procedures involving patients with kidney disease; or care of patients with kidney disease. To provide information on the current state of systematic reviews, we considered only reviews that were published in 2005.

When the same review was published by the authors in two or more different sources (e.g., Cochrane Database and a journal), we abstracted the data from both sources separately (occurred three times). Although we recognize that these are not completely independent observations, this was done for three reasons. First, in this study, we wanted to characterize where renal systematic reviews are published. Excluding a review because of duplication would counter this purpose. Second, we wanted to investigate whether the source of publication influenced the reporting and/or the methodologic quality of a systematic review, and duplicate reviews offered the potential to examine this issue. Finally, choosing to exclude one of the duplicate sources would have been an arbitrary decision.

Finding Renal Reviews

A systematic literature search of Medline (OVID), Experta Medica (Embase), American College of Physicians Journal Club, Cochrane Database of Systematic Reviews, and Database of Abstracts of Reviews of Effectiveness for the year 2005 (DARE) for the year 2005 was performed by an experienced librarian and was supplemented by a hand search of Cochrane Renal Group records of non-Cochrane meta-analyses. Language restrictions were not used.

Duplicates, protocols, and conference abstracts were excluded. The search strategy included the terms renal, kidney, dialysis, hemodialysis, hemofiltration, continuous ambulatory peritoneal dialysis, continuous cycler-assisted peritoneal dialysis, glomerulonephritis, GFR, proteinuria, end-stage renal disease, nephritis, nephrosis, nephropathy, nephrotic, and nephrotoxicity as combination of keywords and corresponding subject headings in thesaurus tree structure (MeSH and EMTREE) and was refined and tested for each database.

Two reviewers independently evaluated the eligibility of each citation, and the full-text article was retrieved when either reviewer considered the citation potentially relevant. Two reviewers independently evaluated the eligibility of each full-text article, with disagreements resolved by consensus.

Data Abstraction

Two reviewers independently abstracted data from all reviews that met the eligibility criteria, and disagreements were resolved by consensus. The source of publication was classified into three groups: Journals specializing in nephrology or transplant medicine, journals not specializing in nephrology or transplant medicine, and Cochrane reviews. The US National Library of Medicine Medline journal subject terms were used to determine the primary discipline of each journal. The journal impact factor was obtained from the 2005 edition of the Institute for Scientific Information Journal Citation Reports.

We abstracted the type of analytic methods used in the study (meta-analysis, meta-regression, qualitative synthesis), study type (diagnosis, prognosis, treatment, etiology/harm, economics), search methods used to identify primary literature, language restrictions used, and the sources of funding. The topic of the systematic reviews was classified into one of five specified renal categories: Acute kidney injury, chronic kidney disease or glomerulonephritis, chronic hemodialysis or peritoneal dialysis, kidney transplantation, and pediatric nephrology. The conclusions reached were classified into three groups: Majority (≥50%) of included studies demonstrated sufficient evidence of significant effect, majority (≥50%) demonstrated sufficient evidence of no significant effect, or majority (≥50%) demonstrated insufficient evidence to make a conclusion.

The quality of each systematic review was evaluated using a previously validated instrument: Overview Quality Assessment Questionnaire (OQAQ) (15). Systematic reviews of randomized, controlled trials were further assessed for adherence to QUOROM. We then searched the website of each journal that published at least one relevant systematic review to establish whether it recommended the use of any consensus guidelines for reporting of systematic reviews (e.g., QUOROM, MOOSE) in its instructions to potential authors. The date of implementation of guidelines was confirmed via correspondence with editors of each journal. The data abstractors were blinded to journal endorsement status.

Overview Quality Assessment Questionnaire

The OQAQ is the only validated tool for assessing the methodologic quality of systematic reviews (15). It was designed to evaluate whether the authors of a systematic review conducted a comprehensive search, minimized bias in the selection of primary studies to be included for analysis, evaluated the primary literature, and pooled the results appropriately.

OQAQ is composed of 10 questions. The first nine questions have set answers of “yes,” “no,” or “partially/can't tell” and are designed to assess different aspects of methodologic quality. The 10th item is an assessment of the overall methodologic quality of the systematic review on a scale of 1 to 7. A score of ≤3 indicates major methodologic flaws that have a strong potential for creating bias in the results of a systematic review.

To reduce interrater variability, we used the modified version to the OQAQ as described in past studies (12,13,16,17). This addendum does not change the questions in the OQAQ; rather, it adds guidelines for each question to standardize interpretation.

QUOROM Consensus Statement

The QUOROM statement was published in 1999 (7). It is a set of consensus guidelines developed to standardize and improve reporting quality of meta-analyses of randomized, controlled trials to minimize bias. It has also been used in the past to evaluate the quality of systematic reviews (12,14,16–18).

The QUOROM checklist is composed of 18 points. It evaluates the reporting quality of all parts of a systematic review, including the title, abstract, introduction, methods, results, and the conclusion. Much like the CONSORT guidelines for randomized, controlled trials, the QUOROM guidelines ensure adequate reporting in all aspects of a systematic review, thereby allowing the reader to judge the quality of the presented evidence.

Data Analysis

Reviewer agreement was quantified by using the κ statistic. To determine which factors were associated with higher methodologic quality, an ANOVA was used for continuous outcomes and a χ² used for categorical outcomes. Factors that were significant in univariate analysis were examined in linear and logistic regression. P ≤ 0.01 was considered significant. Citations were managed using Reference Manager 11.0 (ResearchSoft, Carlsbad, CA). Data were abstracted using an electronic form on Microsoft Excel 2000 (Microsoft Corp., Redmond, WA). Data analysis was performed using SPSS 14.0 (SPSS, Chicago, IL) and SAS 9.1 (SAS Institute, Cary, NC).

Results

Study Selection

We screened a total of 591 citations. A total of 109 full-text articles were retrieved, and the eligibility of each full-text article was evaluated (Figure 1). The chance-corrected agreement was good between two independent reviewers who evaluated the eligibility of each review (κ = 0.71). We included two reviews that initially seemed nonrelevant from the title (19,20), because they included important outcomes that influence renal care. We excluded three reviews published by the Cochrane Renal Group (21–23); two reviews were not deemed relevant to our definition of renal care, because one review examined the treatment of uncomplicated urinary tract infection (21), and the other examined treatment of acute renal colic (22); the third review studied emergency interventions for acute hyperkalemia from all causes (23) but did not perform any analysis of the data in the context of a population with renal disease.

Figure 1.

Selection of systematic reviews.

Characteristics of Nephrology-Related Systematic Reviews

A total of 90 renal systematic reviews were analyzed (all reviews presented in Table 1) (19,20,24–111). The majority of reviews were published in journals that do not specialize in nephrology or transplant medicine (n = 47; 52%), and a substantial proportion were published as Cochrane reviews (n = 18; 20%). Non-nephrology nontransplant journals and the Cochrane Database published the majority of systematic reviews in all five renal subject areas (Table 2).

Table 1.

Included studies^a

Title	Authors	Source	No. of Studies Included	Reference
Contrast-medium-induced nephropathy correlated to the ratio between dose in gram iodine and estimated GFR in ml/min	Nyman U, Almen T, Aspelin P, Hellstrom M, Kristiansson M, Sterner G	Acta Radiologica	16	68
Improving clinical outcomes in coronary artery bypass graft surgery	Sedrakyan A	American Journal of Health-System Pharmacy	35	20
Causes of reversible nephrogenic diabetes insipidus: A systematic review	Garofeanu CG, Weir M, Rosas-Arellano MP, Henson G, Garg AX, Clark WF	American Journal of Kidney Diseases	155	45
Medication-related problems in ambulatory hemodialysis patients: A pooled analysis	Manley HJ, Cannella CA, Bailie GR, St Peter WL	American Journal of Kidney Diseases	7	59
Interventions for preventing bone disease in kidney transplant recipients: A systematic review of randomized controlled trials	Palmer SC, Strippoli GF, McGregor DO	American Journal of Kidney Diseases	23	71
Comparison of hemodialysis, hemofiltration, and acetate-free biofiltration for ESRD: Systematic review	Rabindranath KS, Strippoli GF, Roderick P, Wallace SA, MacLeod AM, Daly C	American Journal of Kidney Diseases	18	79
A systematic review of psychological interventions for the treatment of nonadherence to fluid-intake restrictions in people receiving hemodialysis	Sharp J, Wild MR, Gumley AI	American Journal of Kidney Diseases	16	86
Multinutrient oral supplements and tube feeding in maintenance dialysis: A systematic review and meta-analysis	Stratton RJ, Bircher G, Fouque D, Stenvinkel P, De Mutsert R, Engfer M, Elia M	American Journal of Kidney Diseases	18	90
Has mortality from acute renal failure decreased? A systematic review of the literature	Ympa YP, Sakr Y, Reinhart K, Vincent J	American Journal of Medicine	80	107
Exercise training in patients receiving maintenance hemodialysis: A systematic review of clinical trials	Cheema BSB, Fiatarone Singh MA	American Journal of Nephrology	29	31
Hepatitis C virus antibody status and survival after renal transplantation: Meta-analysis of observational studies	Fabrizi F, Martin P, Dixit V, Bunnapradist S, Dulai G	American Journal of Transplantation	8	39
Post-transplant diabetes mellitus and HCV seropositive status after renal transplantation: Meta-analysis of clinical studies	Fabrizi F, Martin P, Dixit V, Bunnapradist S, Kanwal F, Dulai G	American Journal of Transplantation	10	40
HBsAg seropositive status and survival after renal transplantation: Meta-analysis of observational studies	Fabrizi F, Martin P, Dixit V, Kanwal F, Dulai G	American Journal of Transplantation	6	41
Calcineurin inhibitor withdrawal from sirolimus-based therapy in kidney transplantation: A systematic review of randomized trials	Mulay AV, Hussain N, Fergusson D, Knoll GA	American Journal of Transplantation	6	61
Meta-analysis: Low-dose dopamine increases urine output but does not prevent renal dysfunction or death	Friedrich JO, Adhikari N, Herridge MS, Beyene J	Annals of Internal Medicine	61	44
Meta-analysis: The efficacy of strategies to prevent organ disease by cytomegalovirus in solid organ transplant recipients	Kalil AC, Levitsky J, Lyden E, Stoner J, Freifeld AG	Annals of Internal Medicine	17	52
Ruboxistaurin, a protein kinase C beta inhibitor, as an emerging treatment for diabetes microvascular complications	Joy SV, Scates AC, Bearelly S, Dar M, Taulien CA, Goebel JA, Cooney MJ	Annals of Pharmacotherapy	NR	51
The effect of creatine intake on renal function	Pline KA, Smith CL	Annals of Pharmacotherapy	NR	74
Glycoprotein IIb/IIIa inhibitors in patients with end-stage renal disease	Villeneuve E, Sunderji R	Annals of Pharmacotherapy	3	97
Retarding the progression of diabetic nephropathy in type 2 diabetes mellitus: Focus on hypertension and proteinuria	Lee GS	Annals of the Academy of Medicine, Singapore	NR	55
Role of urinary screening programmes in children in the prevention of chronic kidney disease	Yap HK, Quek CM, Shen Q, Joshi V, Chia KS	Annals of the Academy of Medicine, Singapore	NR	106
Risk of hypertension with multicystic kidney disease: A systematic review	Narchi H	Archives of Disease in Childhood	29	64
Risk of long term renal impairment and duration of follow up recommended for Henoch-Schonlein purpura with normal or minimal urinary findings: A systematic review	Narchi H	Archives of Disease in Childhood	12	65
Theophylline for prevention of contrast-induced nephropathy: A systematic review and meta-analysis	Bagshaw SM, Ghali WA	Archives of Internal Medicine	9	25
Dopexamine and its role in the protection of hepatosplanchnic and renal perfusion in high-risk surgical and critically ill patients	Renton MC, Snowden CP	British Journal of Anaesthesia	21	80
Risk factors for pre-eclampsia at antenatal booking: Systematic review of controlled studies	Duckitt K, Harrington D	British Medical Journal	52	36
Tacrolimus versus ciclosporin as primary immunosuppression for kidney transplant recipients: Meta-analysis and meta-regression of randomised trial data	Webster AC, Woodroffe RC, Taylor RS, Chapman JR, Craig JC	British Medical Journal	30	101
N-acetylcysteine prophylaxis significantly reduces the risk of radiocontrast-induced nephropathy: Comprehensive meta-analysis	Duong MH, MacKenzie TA, Malenka DJ	Catheterization and Cardiovascular Interventions	14	37
Effect of nonionic radiocontrast agents on the occurrence of contrast-induced nephropathy in patients with mild-moderate chronic renal insufficiency: Pooled analysis of the randomized trials	Sharma SK, Kini A	Catheterization and Cardiovascular Interventions	11	85
The effectiveness of tacrolimus and cyclosporine A after renal transplantation: A systematic review.	Li X, Wu T, Su B, Yang Y, Li Y, Wang L, Lu Y	Chinese Journal of Evidence-Based Medicine	18	56
Safety of mycophenolate mofetil versus azathioprine in renal transplantation: A systematic review	Wang K, Zhang H, Li Y, Lu Y, Wei Q, Li H, Yang Y	Chinese Journal of Evidence-Based Medicine	24	99
Prognostic value of troponin T and I among asymptomatic patients with end-stage renal disease: A meta-analysis	Khan NA, Hemmelgarn BR, Tonelli M, Thompson CR, Levin A	Circulation	28	53
Risk of worsening renal function with nesiritide in patients with acutely decompensated heart failure	Sackner-Bernstein JD, Skopicki HA, Aaronson KD	Circulation	5	82
New therapeutic strategies for systemic sclerosis: A critical analysis of the literature	Zandman-Goddard G, Tweezer-Zaks N, Shoenfeld Y	Clinical and Developmental Immunology	NR	109
Use of protein:creatinine ratio measurements on random urine samples for prediction of significant proteinuria: A systematic review	Price CP, Newall RG, Boyd JC	Clinical Chemistry	16	75
Antibiotic dosing in critically ill adult patients receiving continuous renal replacement therapy	Trotman RL, Williamson JC, Shoemaker DM, Salzer WL	Clinical Infectious Diseases	NR	95
Cinacalcet: An oral calcimimetic agent for the management of hyperparathyroidism	Dong BJ	Clinical Therapeutics	NR	34
Biocompatible hemodialysis membranes for acute renal failure	Alonso A, Lau J, Jaber BL	Cochrane Database of Systematic Reviews	9	24
Antibiotics for acute pyelonephritis in children	Bloomfield P, Hodson EM, Craig JC	Cochrane Database of Systematic Reviews	18	27
Recombinant human erythropoietin for chronic renal failure anaemia in pre-dialysis patients	Cody J, Daly C, Campbell M, Donaldson C, Khan I, Rabindranath K, Vale L, Wallace S, Macleod A	Cochrane Database of Systematic Reviews	15	33
Frequency of administration of recombinant human erythropoietin for anaemia of end-stage renal disease in dialysis patients	Cody J, Daly C, Campbell M, Donaldson C, Khan I, Vale L, Wallace S, Macleod A	Cochrane Database of Systematic Reviews	11	32
Non-corticosteroid treatment for nephrotic syndrome in children	Durkan A, Hodson EM, Willis NS, Craig JC	Cochrane Database of Systematic Reviews	20	38
Low protein diets for chronic kidney disease in non diabetic adults	Fouque D, Laville M, Boissel JP	Cochrane Database of Systematic Reviews	8	43
Antiviral medications for preventing cytomegalovirus disease in solid organ transplant recipients	Hodson EM, Barclay PG, Craig JC, Jones C, Kable K, Strippoli GF, Vimalachandra D, Webster AC	Cochrane Database of Systematic Reviews	32	47
Corticosteroid therapy for nephrotic syndrome in children	Hodson EM, Knight JF, Willis NS, Craig JC	Cochrane Database of Systematic Reviews	19	49
Cellulose, modified cellulose and synthetic membranes in the haemodialysis of patients with end-stage renal disease	MacLeod AM, Campbell M, Cody JD, Daly C, Donaldson C, Grant A, Khan I, Rabindranath KS, Vale L, Wallace S	Cochrane Database of Systematic Reviews	32	58
Interventions for preventing bone disease in kidney transplant recipients	Palmer S, McGregor DO, Strippoli GF	Cochrane Database of Systematic Reviews	23	70
Physical measures for treating depression in dialysis patients	Rabindranath KS, Butler JA, MacLeod AM, Roderick P, Wallace SA, Daly C	Cochrane Database of Systematic Reviews	1	77
Psychosocial interventions for depression in dialysis patients *	Rabindranath KS, Daly C, Butler JA, Roderick PJ, Wallace S, MacLeod AM	Cochrane Database of Systematic Reviews	0	78
Calcium channel blockers for preventing acute tubular necrosis in kidney transplant recipients	Shilliday IR, Sherif M	Cochrane Database of Systematic Reviews	10	87
Antihypertensive agents for preventing diabetic kidney disease	Strippoli GF, Craig M, Craig JC	Cochrane Database of Systematic Reviews	16	91
Tacrolimus versus cyclosporin as primary immunosuppression for kidney transplant recipients	Webster A, Woodroffe RC, Taylor RS, Chapman JR, Craig JC	Cochrane Database of Systematic Reviews	30	101
Routine intraoperative ureteric stenting for kidney transplant recipients	Wilson CH, Bhatti AA, Rix DA, Manas DM	Cochrane Database of Systematic Reviews	7	104
Interventions for protecting renal function in the perioperative period	Zacharias M, Gilmore IC, Herbison GP, Sivalingam P, Walker RJ	Cochrane Database of Systematic Reviews	37	108
Sevelamer as a phosphate binder in adult hemodialysis patients: An evidence-based review of its therapeutic value	Nadin C	Core Evidence	53	63
Bench-to-bedside review: Preventive measures for contrast-induced nephropathy in critically ill patients	van den BG, Tonino S, de Fijter C, Smit W, Schultz MJ	Critical Care (London)	16	96
Mechanical ventilation and acute renal failure	Kuiper JW, Groeneveld AB, Slutsky AS, Plotz FB	Critical Care Medicine	NR	54
Angiotensin-I converting enzyme insertion/deletion polymorphism and its association with diabetic nephropathy: A meta-analysis of studies reported between 1994 and 2004 and comprising 14,727 subjects	Ng DP, Tai BC, Koh D, Tan KW, Chia KS	Diabetologia	47	67
Anaemia as an early predictor for diabetic renal disease? A review of the literature	Balshaw-Greer A, Davies J, Casey J	EDTNA-ERCA Journal	10	26
Efficacy of therapies targeted at treating patients with membranous glomerulonephritis	Zhuo DX, Hanna-Johnson M	Ethnicity and Disease	NR	110
Adverse effects of antifungal therapies in invasive fungal infections: Review and meta-analysis	Girois SB, Chapuis F, Decullier E, Revol BG	European Journal of Clinical Microbiology & Infectious Diseases	54	46
The management of abdominal aortic aneurysms in patients with concurrent renal impairment	Bown MJ, Norwood MG, Sayers RD	European Journal of Vascular and Endovascular Surgery	74	28
Prevalence of anemia and effects on mortality in patients with heart failure	Lindenfeld J	European Journal of Vascular and Endovascular Surgery	26	19
A review of the current treatment of renal artery stenosis	Mwipatayi BP, Beningfield SJ, White LE, Irish A, Abbas M, Sieunarine K	European Journal of Vascular and Endovascular Surgery	NR	62
Systematic review on urine albumin testing for early detection of diabetic complications	Newman DJ, Mattock MB, Dawnay AB, Kerry S, McGuire A, Yaqoob M, Hitman GA, Hawke C	Health Technology Assessment	NR	66
Clinical and cost-effectiveness of newer immunosuppressive regimens in renal transplantation: A systematic review and modelling study	Woodroffe R, Yao GL, Meads C, Bayliss S, Ready A, Raftery J, Taylor RS	Health Technology Assessment	60	105
Chronic viral hepatitis in hemodialysis patients	Tang S, Lai KN	Hemodialysis International	NR	93
Systematic review of combined angiotensin-converting enzyme inhibition and angiotensin receptor blockade in hypertension	Doulton TW, He FJ, MacGregor GA	Hypertension	14	35
Epoetin alfa's effect on left ventricular hypertrophy and subsequent mortality	Jones M, Schenkel B, Just J	International Journal of Cardiology	18	50
Psycho-educational strategies to promote fluid adherence in adult hemodialysis patients: A review of intervention studies	Welch JL, Thomas-Hawkins C	International Journal of Nursing Studies	9	102
N-acetylcysteine for the prevention of contrast-induced nephropathy: A systematic review and meta-analysis	Liu R, Nair D, Ix J, Moore DH, Bent S	Journal of General Internal Medicine	9	57
Association between the GLUT1 gene polymorphism and the risk of diabetic nephropathy: A meta-analysis	Zintzaras E, Stefanidis I	Journal of Human Genetics	6	111
Antihypertensive agents for primary prevention of diabetic nephropathy	Strippoli GF, Craig M, Schena FP, Craig JC	Journal of the American Society of Nephrology	16	92
Effects of fish oil supplementation on kidney transplantation: A systematic review and meta-analysis of randomized, controlled trials	Tatsioni A, Chung M, Sun Y, Kupelnick B, Lichtenstein AH, Perrone R, Chew P, Lau J, Bonis PA	Journal of the American Society of Nephrology	16	94
The literature increasingly supports expectant (conservative) management of renal trauma: A systematic review	Santucci RA, Fisher MB	Journal of Trauma: Injury Infection and Critical Care	110	84
Is benign prostatic hyperplasia a risk factor for chronic renal failure?	Rule AD, Lieber MM, Jacobsen SJ	Journal of Urology	NR	81
Acute renal failure independently predicts mortality after myeloablative allogeneic hematopoietic cell transplant	Parikh CR, McSweeney P, Schrier RW	Kidney International	6	72
The role of osmolality in the incidence of contrast-induced nephropathy: A systematic review of angiographic contrast media in high risk patients	Solomon R	Kidney International	17	89
A systematic review of the effect of nocturnal hemodialysis on blood pressure, left ventricular hypertrophy, anemia, mineral metabolism, and health-related quality of life	Walsh M, Culleton B, Tonelli M, Manns A	Kidney International	14	98
Effect of inhibitors of the renin-angiotensin system and other antihypertensive drugs on renal outcomes: Systematic review and meta-analysis	Casas JP, Chua W, Loukogeorgakis S, Vallance P, Smeeth L, Hingorani AD, MacAllister RJ	Lancet	127	30
Antiviral medications to prevent cytomegalovirus disease and early death in recipients of solid-organ transplants: A systematic review of randomized controlled trials	Hodson EM, Jones C, Webster AC, Strippoli GF, Barclay PG, Kable K, Vimalachandra D, Craig JC	Lancet	32	48
Vascular access complications in daily dialysis: A systematic review of the literature	Shurraw S, Zimmerman D	Minerva Urologica e Nefrologica	12	88
Dose-reducing H2 receptor antagonists in the presence of low glomerular filtration rate: A systematic review of the evidence	Manlucu J, Tonelli M, Ray JG, Papaioannou A, Youssef G, Thiessen-Philbrook HR, Holbrook A, Garg AX.	Nephrology Dialysis Transplantation	45	60
Measured creatinine clearance from timed urine collections substantially overestimates glomerular filtration rate in patients with liver cirrhosis: A systematic review and individual patient meta-analysis	Proulx NL, Akbari A, Garg AX, Rostom A, Jaffey J, Clark HD	Nephrology Dialysis Transplantation	7	76
Cost-effectiveness of irbesartan in diabetic nephropathy: A systematic review of published studies	Palmer AJ, Tucker DM, Valentine WJ, Roze S, Gabriel S, Cordonnier DJ	Nephrology Dialysis Transplantation	7	69
Peritoneal catheters and exit-site practices toward optimum peritoneal access: A review of current developments	Flanigan M, Gokal R	Peritoneal Dialysis International	NR	42
Cinacalcet: A new treatment for secondary hyperparathyroidism in patients receiving hemodialysis	Byrnes CA, Shepler BM	Pharmacotherapy	NR	29
The clinical spectrum of primary renal vasculitis	Samarkos M, Loizou S, Vaiopoulos G, Davies KA	Seminars in Arthritis and Rheumatism	NR	83
Routine intraoperative stenting for renal transplant recipients	Wilson CH, Bhatti AA, Rix DA, Manas DM	Transplantation	7	103
Updated metaanalysis of steroid withdrawal in renal transplant patients on calcineurin inhibitor and mycophenolate mofetil	Pascual J, Quereda C, Zamora J, Hernandez D	Transplantation Proceedings	6	73

Table 2.

Number of systematic reviews by publication type and by topic in nephrology

Publication Type	Topic					Total (n [%]; N = 90)
	Acute kidney injury (n[%]; n = 20)	Chronic kidney disease or glomerulonephritis (n [%]; n = 32)	Chronic hemodialysis or peritoneal dialysis (n [%]; n = 21)	Kidney transplantation (n [%]; n = 19)	Pediatric Nephrology (n [%]; n = 5)
Nephrology or transplant journals	3 (12)	5 (20)	9 (36)	8 (32)	0 (0)	25 (28)
Non-nephrology, nontransplant journals	16 (84)	22 (73)	7 (33)	5 (26)	2 (40)	47 (52)
Cochrane reviews	1 (5)	5 (17)	5 (24)	6 (32)	3 (60)	18 (20)

Journals that specialize in transplant medicine or nephrology focused on topics relating to dialysis and kidney transplantation and published few reviews on acute kidney injury, chronic kidney disease, and glomerulonephritis. No systematic reviews related to pediatric nephrology were published by journals that specialize in nephrology or transplant medicine.

On average, 23 primary studies were included in the final analysis of each systematic review, with a range of 0 to 155 studies; however, a significant proportion of systematic reviews (n = 16; 11%) did not clearly indicate the number of primary studies included in the analysis. One systematic review included no articles in its analysis (87); however, because it was designated as a systematic review and not a protocol by the publishing source, it was included in the analysis.

Quality of Nephrology-Related Systematic Reviews

Interrater agreement on the presence of major flaws was good (κ = 0.71). The OQAQ scores were also poor with an average score of 6.0 out of 9.0 (SD 2.9; range 0.0 to 9.0), indicating incomplete reporting of methodologic techniques used in the review. The majority of nephrology-related systematic reviews (49 [54%] of 90) contained major methodologic flaws (Table 3).

Table 3.

Quality of systematic reviews

Parameter	No. of Systematic Reviews	OQAQ Score of Methodologic Quality (9 Points)		Overall Quality Score as per OQAQ Question 10 (Scale 1 to 7)		OQAQ Methodologic Errors
		Mean (SD)	P	Mean (SD)	P	Major (n [%])a	Minor (n [%])a	P
All systematic reviews	90	6.0 (2.9)	3.4 (2.2)	49 (54)	41 (46)
Systematic reviews published in
Journals	72	5.3 (2.8)	<0.001	2.8 (1.9)	<0.001	49 (68)	23 (32)	<0.001
Cochrane Reviews	18	8.8 (0.6)	5.9 (0.8)	0 (0)	18 (100)
Systematic reviews published in
journals that require adherence to consensus guidelines for reporting of systematic reviews	11	8.0 (1.7)	<0.001	4.7 (1.6)	<0.001	2 (18)	9 (82)	<0.001
journals that do not require adherence to consensus guidelines for reporting of systematic reviews	61	4.8 (2.7)	2.4 (1.7)	47 (77)	14 (23)
Systematic reviews published in
journals that require adherence to consensus guidelines for reporting of systematic reviews	11	8.0 (1.7)	0.650	4.7 (1.6)	0.140	2 (18)	9 (82)	0.420
Cochrane reviews	18	8.8 (0.6)	5.9 (0.8)	0 (0)	18 (100)
Systematic reviews published in
nephrology or transplant journals	25	6.2 (2.8)	0.090	3.3 (2.0)	0.140	14 (56)	11 (44)	0.180
non-nephrology, non-transplant journals	47	4.9 (2.7)	2.5 (1.8)	35 (74)	12 (26)
Language restrictions
no	32	8.4 (1.1)	<0.001	5.5 (1.3)	<0.001	4 (13)	28 (87)	<0.001
yes	24	5.6 (2.4)	2.6 (1.7)	17 (71)	7 (29)
not reported	34	4.0 (2.6)	1.9 (1.4)	28 (82)	6 (18)
Primary study authors contacted
yes	41	7.8 (1.9)	<0.001	4.8 (1.8)	<0.001	10 (24)	31 (76)	<0.001
no	49	4.5 (2.7)	2.1 (1.7)	39 (80)	10 (20)
Funding
reported	52	7.3 (2.4)	<0.001	4.3 (1.9)	<0.001	18 (35)	34 (65)	<0.001
not reported	38	4.3 (2.6)	2.1 (1.8)	31 (82)	7 (18)
Impact factorb
0.00 to 2.99	29	4.0 (2.4)	<0.001	1.8 (1.2)	<0.001	26 (90)	3 (10)	<0.001
≥3.00	35	6.5 (2.5)	3.6 (2.0)	17 (49)	18 (51)

^aStudies were deemed to contain major methodological errors if the overall OQAQ score was less than or equal to 3. Studies were deemed to contain minor methodological errors if the overall OQAQ score was greater than or equal to 4.

^bInstitute for Scientific Information impact factor was not available for 26 systematic reviews as ISI does not calculate an impact factor for the Cochrane database and nine journals (Chinese Journal of Evidence-Based Medicine, Clinical & Developmental Immunology, Core Evidence, Edtna-Erca Journal, Health Technology Assessment, Hemodialysis International and Minerva Uroloica E Nefrologica).

There were two common methodologic flaws. The majority of reviews (44 of 90 reviews) failed to assess the methodologic quality of the studies that they included in their analysis. An even bigger proportion of reviews (51 of 90 reviews) failed to show evidence that they attempted to minimize their bias during the process of selection of studies for analysis by having only one person select eligible studies, or they lacked a priori criteria for inclusion. Factors that were independently associated with higher methodologic quality were a lack of language restriction in the literature search, an attempt to contact primary study authors, and an explicit statement regarding sources of funding (Table 3).

The number of authors (P = 0.17) or funding by industry (P = 0.30) did not significantly correlate with study quality. The final conclusions reached (presence of association, absence of association, or insufficient evidence as judged independently by M.M. and H.T.-P.) did not have any relationship to the quality of the systematic reviews (P = 0.35). Affiliation with a nephrology department showed a trend toward an association with a higher methodologic quality (P = 0.05), but it did not reach our predetermined significance level of P = 0.01.

Overall, many nephrology-related systematic reviews of randomized, controlled trials did not adhere to QUOROM guidelines (Table 4), with an average score of 11.9 out of 18.0 (SD 4.6; range 0.0 to 18.0). A minority (four of 48) of journals recommended adherence to consensus guidelines for reporting of systematic reviews (e.g., QUOROM, MOOSE) in the instructions to authors (112–115); however, these journals published systematic reviews of randomized, controlled trials that had a significantly higher methodologic quality as measured by OQAQ when compared with journals that did not recommend adherence to guidelines (see Table 4).

Table 4.

Quality of systematic reviews of randomized, controlled trials^a

Parameter	No. of Systematic Reviews of RCT	OQAQ Score of Methodologic Quality (9 Points)		Overall Quality Score as per OQAQ Question 10 (Scale 1 to 7)		OQAQ Methodologic Errors			Adherence to QUOROM Guidelines (18 Points)
		Mean (SD)	P	Mean (SD)	P	Major (n [%])b	Minor (n [%])b	P	Mean (SD)	P
Systematic reviews of RCT published in
Journals	41	5.4 (3.10)	<0.001	2.9 (2.10)	<0.001	27 (66)	14 (34)	<0.001	10.6 (4.9)	<0.001
Cochrane reviews	18	8.8 (0.60)	5.9 (0.80)	0 (0)	18 (100)	15.1 (1.0)
Systematic reviews of RCT published in
journals that require adherence to consensus guidelines for reporting of systematic reviews	8	8.5 (0.53)	<0.001	5.0 (1.40)	<0.001	1 (13)	7 (87)	<0.001	16.1 (2.1)	<0.001
journals that do not require adherence to consensus guidelines for reporting of systematic reviews	34	4.6 (3.00)	2.4 (2.00)	26 (76)	8 (24)	9.3 (4.4)
Systematic reviews of RCT published in
journals that require adherence to consensus guidelines for reporting of systematic reviews	8	8.5 (0.53)	0.960	5.0 (1.40)	0.420	1 (13)	7 (87)	0.680	16.1 (2.1)	0.720
Cochrane reviews	18	8.8 (0.60)	5.9 (0.80)	0 (0)	18 (100)	15.1 (1.0)

^aRCT, randomized, controlled trials.

^bStudies were deemed to contain major methodological errors if the overall OQAQ score was less than or equal to 3. Studies were deemed to contain minor methodological errors if the overall OQAQ score was greater than or equal to 4.

Discussion

Well-conducted systematic reviews aim to compile the best available evidence for various diagnosis and treatment options. We assessed the methodologic quality of 90 systematic reviews in the field of nephrology published in 2005. Only one quarter of these reviews were published by journals that specialize in nephrology or transplant medicine, another half were published in other journals, and the remainder were published as Cochrane reviews. The majority of systematic reviews had major methodologic flaws as measured by a validated instrument. Consensus guidelines for reporting of data in systematic reviews (QUOROM and MOOSE) were published in 1999 and 2000, respectively, but a majority of journals still do not endorse these guidelines in their instructions to authors; however, journals that recommended adherence to these guidelines published systemic reviews of higher methodologic quality.

Findings and Implications for Clinicians, Researchers, Publishers, and Librarians

The methodologic quality of many nephrology-related systematic reviews was poor. This could indicate that the methods used in systematic reviews were inappropriate and the conclusions of such reviews possibly biased. At the same time, the reviews did not adhere strictly to reporting guidelines, indicating suboptimal reporting quality. Clinicians and researchers should carefully appraise the methods used in a scientific review before giving any weight to the results.

Our study indicates that journals that endorsed reporting guidelines published systematic reviews of higher methodologic quality. Adoption of such guidelines by all journal publishers could help to improve the quality of published reviews.

The conclusion of any systematic review is limited by the reliability of estimates provided by the included primary articles; however, a majority of the systematic reviews failed to acknowledge this fact with no assessment of the methodologic quality of included studies. If the readers are to rely on review conclusions, then it is essential that they be given enough information to evaluate the quality of the supporting data.

Scientific research articles must include enough information to allow an independent party to repeat and verify the results. The method by which relevant information is compiled can be subjective. If there are no a priori inclusion criteria and if only a single person arbitrarily determines the entire selection process of primary evidence, the results may not replicate if the process was to be repeated independently. This was the most common critical flaw of the systematic reviews that we examined.

Independent factors that were associated with higher methodologic quality were a lack of language restriction in the literature search, an attempt to contact primary study authors, and an explicit statement regarding sources of funding. These points are not part of the OQAQ, but they are labor-intensive (e.g., translation of articles published in a foreign language). This may indicate that study authors who were willing to commit more effort to their project generally produced higher quality research.

Reviews published in the Cochrane Database were of significantly higher quality compared with the reviews published in journals that did not endorse consensus guidelines. The difference in quality may be due to RevMan software provided by Cochrane specifically for the creation of systematic reviews. This software has a built-in methodologic framework for conducting reviews, which may prompt review teams to adopt better methodologic quality.

Only a minority of reviews were published in journals that specialize in nephrology or transplant medicine, and the nonspecialty journals published articles on a broad range of topics. This is an important point for clinicians, researchers, and librarians, because a review of evidence based solely on subspecialty journals will result in missed studies.

Strengths, Limitations, and Future Directions

Although reporting guidelines such as QUOROM were published more than 7 years ago, this is the first study to demonstrate that the endorsement of such guidelines by journal editors is associated with significantly higher methodologic quality. This is also the first study to evaluate the sources and the quality of systematic reviews relevant to the clinical practice of nephrology. We used a comprehensive method to identify all renal systematic reviews for the most recent complete year. The inclusion of studies and the abstraction of data were performed in duplicate to minimize any bias in these tasks.

With respect to the quality of systematic reviews, our findings concur with previous articles that examined the quality of systematic reviews in the fields of anesthesia (10), general surgery (11), critical care (12), and emergency medicine (13). All of them found that a majority of systematic reviews contained major methodologic flaws. It is interesting that another common conclusion was that a large proportion of systematic reviews in a given specialty are not published in the subspecialty-related journals (10,12); however, none of these studies examined the association between the endorsement of reporting guidelines by a journal and the quality of the published articles.

The search was performed by an experienced librarian, but it remains conceivable that we missed a systematic review relevant to nephrology, particularly studies in which renal outcomes were secondary to the main question. Many important quality measures for the rigorous conduct of a systematic review (e.g., statistical tests for quantitative measurement of publication bias) are not considered in the OQAQ; however, currently, it is the only validated tool for the assessment of methodologic quality of systematic reviews.

Although the quality of the reviews was measured, we did not assess whether the flaws present in the studies were due to poor reporting at the level of the study authors or journal publishers or to poor methods used by the study authors. This could be investigated in the future by contacting the publishers and study authors. Other studies previously indicated that reporting often differs from the actual study method (116); however, regardless of the reason, the poor quality of the systematic reviews as judged by the reader makes these reports less credible for use in clinical practice. Finally, we assessed the quality of systematic reviews published in the course of a single year to provide an assessment of the current state of affairs. Although there may be changes in the quality of literature from year to year, that there were large deficiencies in methodologic quality of reviews published during a recent period of time should serve as a call to action.

It is not known whether poor methodologic quality influences the final conclusions reached by a nephrology-related systematic review, and this would be an important area of further study. A minority of reviews were published by renal journals. This may reflect a possible publishing bias and warrants further consideration.

Conclusions

Although there are examples of high-quality systematic reviews in nephrology, most could be improved. Funding bodies and authors need to make efforts to improve the quality of published reviews, to raise confidence for using these reports in patient care. Journal endorsement of consensus guidelines such as QUOROM may also be an important tool for improving the quality of systematic reviews.

Disclosures

None.