Reporting, handling and assessing the risk of bias associated with missing participant data in systematic reviews: a methodological survey

Abstract

Objectives

To describe how systematic reviewers are reporting missing data for dichotomous outcomes, handling them in the analysis and assessing the risk of associated bias.

Methods

We searched MEDLINE and the Cochrane Database of Systematic Reviews for systematic reviews of randomised trials published in 2010, and reporting a meta-analysis of a dichotomous outcome. We randomly selected 98 Cochrane and 104 non-Cochrane systematic reviews. Teams of 2 reviewers selected eligible studies and abstracted data independently and in duplicate using standardised, piloted forms with accompanying instructions. We conducted regression analyses to explore factors associated with using complete case analysis and with judging the risk of bias associated with missing participant data.

Results

Of Cochrane and non-Cochrane reviews, 47% and 7% (p<0.0001), respectively, reported on the number of participants with missing data, and 41% and 9% reported a plan for handling missing categorical data. The 2 most reported approaches for handling missing data were complete case analysis (8.5%, out of the 202 reviews) and assuming no participants with missing data had the event (4%). The use of complete case analysis was associated only with Cochrane reviews (relative to non-Cochrane: OR=7.25; 95% CI 1.58 to 33.3, p=0.01). 65% of reviews assessed risk of bias associated with missing data; this was associated with Cochrane reviews (relative to non-Cochrane: OR=6.63; 95% CI 2.50 to 17.57, p=0.0001), and the use of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology (OR=5.02; 95% CI 1.02 to 24.75, p=0.047).

Conclusions

Though Cochrane reviews are somewhat less problematic, most Cochrane and non-Cochrane systematic reviews fail to adequately report and handle missing data, potentially resulting in misleading judgements regarding risk of bias.

Strengths and limitations of this study.

• This is the first study to assess the reporting, handling and assessment of risk of bias associated with missing participant data among systematic reviews.

• The study used systematic, rigorous and transparent approaches to define eligibility criteria, search for eligible studies, select studies and abstract data.

• The broad eligibility criteria and inclusion of both Cochrane and non-Cochrane systematic reviews make the results more generalisable.

• One limitation of the study is the restriction of our search to MEDLINE database.

• The study did not analyse the individual trials that contributed to eligible systematic reviews.

Background

Although clinical trial investigators may strive to reduce the amount of missing data, they will in most instances fail to achieve complete follow-up.^{1 2} In a recent survey of the top five general medical journals, 87% of published trials reported participants with missing data for the primary outcome.³ The median percentage of participants with missing data was 6% (IQR 2–14%).³ Moreover, the way the trials handled missing participant data varied and was unclear in about a fifth of reports.³

The Cochrane Collaboration Handbook and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement recommend that the systematic review authors provide a description of incomplete reporting of data in the included trials.^{4 5} Furthermore, they recommend reporting how missing data are incorporated into the review findings. Although the Cochrane Handbook encourages the reanalysis of a study's effect estimate by including all randomised participants, it lacks a detailed guidance on how to handle missing data.⁴

The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to judging the confidence in pooled effect estimates takes into account the risk of bias associated with missing data.⁶ The Cochrane Collaboration Handbook and PRISMA statement recommend that the authors of systematic reviews clearly describe how they assessed the risk of bias associated with missing data.^{4 5} The Cochrane Handbook describes situations in which an analysis can be judged to be at low or high risk of bias. For example, exclusion of participants due to ‘inefficacy’ or ‘failure to improve’ would be judged as introducing a high risk of bias.

There is a need to better understand how systematic review authors handle missing participant data in systematic reviews. Therefore, the primary objective of this methodological survey was to describe how systematic review authors report, handle and judge the risk of bias associated with missing data for dichotomous outcomes. Given the evidence that Cochrane reviews tend to be of higher methodological quality compared with other reviews,^{7 8} our secondary objective was to investigate differences in findings between Cochrane and non-Cochrane reviews.

Methods

Design overview

This study is part of a larger project (the ARROW project) examining methodological issues in systematic reviews; we have reported the full details of the methodology elsewhere.⁹ We used standard systematic review methods to survey how Cochrane and non-Cochrane systematic reviews report, handle and judge the risk of bias associated with missing participant data. We defined Cochrane systematic reviews as systematic reviews published in the Cochrane Database of Systematic Reviews. We considered all other systematic reviews as non-Cochrane systematic reviews. Since no human participants were involved, ethical approval was not required. The Cochrane Methods Innovation Fund funded this project.

Eligibility criteria

Eligible studies met the following inclusion criteria: described as a meta-analysis or a systematic review; describes a search strategy of at least one electronic database; published in the Cochrane Database of Systematic Reviews or in a journal indexed in MEDLINE;⁴ includes randomised controlled trials comparing an intervention with another intervention or with no intervention (or placebo) in humans; and reports measures of effect for at least one dichotomous outcome either from a single study or from a pooled analysis.

Search strategy

We conducted the searches in the MEDLINE database through the OVID interface (see online supplementary appendix S1). For non-Cochrane systematic reviews, we employed a systematic review filter designed by the Health Information Research Unit of McMaster University.¹⁰ For Cochrane systematic reviews, we restricted the search results to the Cochrane Database of Systematic Reviews as the journal type. We limited the searches to the year 2010. We used no language restrictions.

Selection process

We conducted the selection process for Cochrane and non-Cochrane reviews separately. Of the search results, we screened consecutive citations in a random order until we included the target sample size, with approximately equal representation of Cochrane and non-Cochrane reviews.

Teams of two reviewers conducted title and abstract screening independently and in duplicate. We obtained the full texts of citations judged as potentially eligible by at least one reviewer. The same teams of reviewers conducted full text screening. They resolved discrepancies by consensus, and when unsuccessful, with the help of a third reviewer. Reviewers participated in calibration exercises and used standardised and pilot-tested forms with detailed written instructions. They selected for each study a pairwise comparison and the most patient-important outcome.⁹ We defined a patient-important outcome as an outcome for which one would answer with ‘yes’ the following question: ‘If the patient knew that this outcome was the only thing to change with treatment, would the patient consider receiving this treatment if associated with side effects or cost?’.¹¹ The patient-important outcome did not necessarily have to be the primary outcome.⁹ We used an online systematic review software application (DistillerSR, Evidence Partners, Ottawa, Canada; http://systematic-review.net/) to facilitate screening and data abstraction.

Data abstraction

Teams of two reviewers abstracted data using DistillerSR. They resolved discrepancies by consensus, and when unsuccessful, with the help of a third reviewer. They participated in calibration exercises, and used standardised and pilot-tested forms with detailed instructions.

We abstracted information about the following general characteristics of the systematic review:

• Number of included trials;

• Number of included participants in intervention and control arms;

• Quality of the systematic review using the ‘assessment of multiple systematic reviews’ (AMSTAR) tool.¹² The tool consists of 11 questions with 4 answer options (yes, no, can't answer, not applicable). The score for each study consisted of the number of ‘yes’ answers, with higher values indicating better scores (0–4: low quality; 5–8: moderate quality; 9–11 high quality);

• Type of intervention (pharmacological vs surgery/invasive procedure);

• Type of meta-analysis (standard meta-analysis vs meta-regression vs individual participant data meta-analysis);

• Evaluation of the risk of bias (using Cochrane Risk of Bias (RoB) tool vs by dimensions vs point system scale);

• Use of the GRADE approach to rate confidence in effect estimates;¹³

• Source of funding (for profit source vs source other than for profit vs not funded);

• Whether any of the authors reports industry ties.

We abstracted the following information regarding the collection and reporting of information about missing participant data within each systematic review:

• Plan to collect the number of participants with missing data;

• Plan to collect the reasons for missing participant data;

• Reporting of the number of participants with missing data;

• Reporting of categories for participants with missing data (eg, withdrawal of consent, cross-over, dropouts, non-adherence);

• Reporting on missing participant data as a separate outcome;

• Inclusion of information on missing data in tabular format.

We abstracted data on how each systematic review handled missing participant data:

• Plan for handling missing categorical data;

• Plan for sensitivity analysis using different methods for handling missing participant data;

• Provision of a justification for the method(s) for handling missing participant data.

We also assessed whether the authors judged the risk of bias associated with missing participants using a specific tool as such as the Cochrane RoB tool. Finally, we assessed whether the implications of missing participant data were incorporated in the Results or Discussion sections.

Analysis

We conducted a descriptive analysis of all variables. We used frequencies and percentages for categorical variables, and median and IQR for continuous variables as these data were not normally distributed. We analysed the data combined as well as stratified by type of review (Cochrane vs non-Cochrane). For this stratification, we compared continuous variables using the independent t test or Wilcoxon test depending on the distribution of the data. We compared categorical variables using the χ² test or Fisher's exact test if the expected event number is less than 5.

We conducted two multivariable logistic regression analyses and prespecified the following dependent variables: (1) whether the systematic reviewers used complete case analysis; and (2) whether the systematic reviewers judged the risk of bias associated with missing participant data. We included the following independent variables in our adjusted models: type of review (Cochrane vs non-Cochrane); number of included trials; type of intervention; use of the GRADE approach to rate confidence in effect estimates; and whether any of the authors reported ties to industry. Following descriptive analyses, we omitted the latter two independent variables from the first regression analysis due to the low number of events. We hypothesised that Cochrane reviews and the use of the GRADE approach, but not industry ties, would be associated with the use of complete case analysis and judgement of the risk of bias associated with missing data. We used SPSS statistical software, V.18.0 (SPSS INC, Chicago, Illinois, USA).

Sample size calculation

We originally calculated the sample size for the purpose of evaluating the association of study characteristics with the reporting of absolute effects (the ARROW project).⁹ As we used the same sample (N=202) to conduct the current study, we estimated whether its size would be appropriate for a regression analysis to study the association of study characteristics with using complete case analysis. The regression analysis includes five independent variables, and would require 10 events per variable. As we were not aware of studies providing an estimate for the dependent variable, we considered a conservative estimate of 30%. The resulting sample size is 167, making the 202 systematic reviews sample we have large enough for our purpose.

Results

Out of 2328 citations identified by the search strategy, we included a total of 202 systematic reviews: 98 Cochrane and 104 non-Cochrane reviews (figure 1). Table 1 reports the general characteristics of included studies, with a p value for the test of difference between these two types. Cochrane reviews included fewer trials and participants, less frequently conducted meta-regression, had a higher AMSTAR score, but more frequently addressed pharmacological interventions, assessed risk of bias, and used the GRADE approach for rating certainty in estimates.

Figure 1.

Flow chart of the screening literature process (RCT, randomised controlled trial).

Table 1.

General characteristics of the included SRs

	Overall (N=202)	Cochrane SR (N=98)	Non-Cochrane SR (N=104)	p Value*
Number of included trials; median (IQR)	5 (2–9)	3 (2–8)	6 (4–9)	<0.0001
Number of participants in intervention group; median (IQR)	426.5 (127–1141)	289 (103–794)	660 (221–1924)	0.006
Number of participants in control group; median (IQR)	418 (117–1026)	271 (85–657)	646 (212–1773)	0.002
AMSTAR score; median (IQR)†	9 (7–10)	10 (9–10)	7 (6–8.5)	<0.0001
Intervention
Pharmacological	130 (64.4%)	70 (71.4%)	60 (57.7%)	0.074
Surgery/invasive procedure	33 (16.3%)	15 (15.3%)	18 (17.3%)
Other	39 (19.3%)	13 (13.3%)	26 (25%)
Type of meta-analysis
Standard meta-analysis	186 (92.1%)	88 (89.8%)	98 (94.2%)	0.30
Metaregression	18 (8.9%)	3 (3%)	15(14.4%)	0.006
Individual participant data meta-analysis	7 (3.4%)	2 (2%)	5 (4.8%)	0.45
Other	10 (5%)	7 (7.1%)	3 (2.9%)	0.204
Evaluation of the risk of bias				<0.0001
Using the Cochrane Risk of Bias tool	94 (46.5%)	84 (85.7%)	10 (9.6%)
By dimensions (eg, blinding)	39 (19.3%)	8 (8.2%)	31 (29.8%)
Using Jadad's or other point system scale	37 (18.3%)	4 (4.1%)	33 (31.7%)
Not done	18 (8.9%)	0 (0%)	18 (17.3%)
Used the GRADE approach to rate confidence in effect estimates	32 (15.8%)	27 (27.6%)	5 (4.8%)	<0.0001
Funding
For profit source	7 (3.5%)	2 (2%)	5 (4.8%)	0.45
Source other than for profit	99 (49%)	63 (64.3%)	36 (34.6%)	<0.0001
Not funded	23 (11.4%)	9 (9.2%)	14 (13.5%)	0.34
Not reported	72 (35.6%)	23 (23.5%)	49 (47%)	0.0005
Reported industry ties by authors				<0.0001
Yes	37 (18.3%)	19 (19.4%)	18 (17.3%)
No	95 (47.0%)	60 (61.2%)	35 (33.7%)
Not reported	68 (33.7%)	19 (19.4%)	49 (47.1%)
Unclear	2 (1.0%)	0 (0%)	2 (1.9%)

*p Value for the difference between Cochrane and non-Cochrane SRs.

†AMSTAR interpretation: 0–4: low quality; 5–8: moderate quality; 9–11 high quality.

AMSTAR, assessment of multiple systematic reviews; SR, systematic review.

Reporting missing participant data

Table 2 reports the collection and reporting of information about missing participant data in the included systematic reviews. The percentages of reviews that reported plans for collecting the number and reasons for missing data were 34% and 17%, respectively. The percentages that actually reported the number and categories with potentially missing participant data were 26% and 19%, respectively. Five per cent of systematic reviews reported missing participant data as an outcome measure. About half of the reviews included information about missing participant data in a tabular format. Cochrane reviews compared favourably to non-Cochrane reviews for all these reporting variables.

Table 2.

The collection and reporting of information about missing participant data in Cochrane and non-Cochrane SRs

	Overall (N=202)	Cochrane SR (N=98)	Non-Cochrane SR (N=104)	p Value*
Reported a plan to collect number of participants with missing data	68 (33.7%)	51 (52%)	17 (16.3%)	<0.0001
Reported a plan to collect reasons for missing data	35 (17.3%)	26 (26.5%)	9 (8.7%)	0.0008
Reported number of participants with missing data
No	149 (73.8%)	52 (53.1%)	97 (93.3%)	<0.0001
Across comparisons, but not for each comparison	14 (7.0%)	11 (11.2%)	3 (2.9%)	0.03
For each comparison, but not for each study	3 (1.5%)	2 (2.0%)	1 (1.0%)	0.61
For each study but not for each outcome	34 (16.8%)	31 (31.6%)	3 (2.9%)	<0.0001
For each outcome	2 (1.0%)	2 (2.0%)	0 (0%)	0.23
Reported on the following participant categories with potentially missing data
No reporting of any category	163 (80.7%)	66 (67.4%)	97 (93.3%)	<0.0001
Mistakenly randomised	6 (3.0%)	6 (6.1%)	0 (0%)	0.01
Did not receive intervention	4 (2.0%)	4 (4.1%)	0 (0%)	0.05
Withdrew consent	12 (6.0%)	11 (11.2%)	1 (1.0%)	0.002
Crossed over	0 (0%)	0 (0%)	0 (0%)	–
Dropped out	21 (10.4%)	16 (16.3%)	5 (4.8%)	0.007
Non-adherent	7 (3.5%)	7 (7.1%)	0 (0%)	0.006
Lost contact	14 (6.9%)	13 (13.3%)	1 (1%)	<0.0001
‘Other reasons’ not otherwise specified	6 (3.0%)	5 (5.0%)	1 (1.0%)	0.11
Other specified reason	10 (5.0%)	10 (10%)	0 (0%)	<0.0001
SR authors reported that included studies did not report on missing data	3 (1.5%)	3 (3.1%)	0 (0%)	0.11
Reported on missing data as distinct outcome	9 (4.5%)	8 (8.2%)	1 (1.0%)	0.02
Reported information on missing data in
Cochrane RoB tool (table format)	61 (30.2%)	59 (60.2%)	2 (1.9%)	<0.0001
Cochrane RoB tool (graph format)	46 (22.77%)	45 (45.92%)	1 (1.0%)	<0.0001
GRADE SoF table	9 (4.5%)	9 (9.2%)	0 (0.0%)	0.001
GRADE EP	0 (0.0%)	0 (0.0%)	0 (0.0%)	–
Table describing characteristics of included studies	70 (34.7%)	61 (62.2%)	9 (8.7%)	<0.0001
Other table	9 (4.5%)	2 (2.0%)	7 (6.7%)	0.17
None of the above	102 (50.5%)	17 (17.4%)	85 (81.7%)	<0.0001

*p Value for the difference between Cochrane and non-Cochrane SRs.

EP, Evidence Profile; GRADE, Grading of Recommendations Assessment, Development and Evaluation; RoB, Risk of Bias; SoF, Summary of Findings; SR, systematic review.

Handling missing participant data

Table 3 shows the characteristics of systematic reviews in terms of handling missing participant data. The percentage of systematic reviews reporting a plan for handling missing categorical data was 25%. The two most reported approaches were using complete case analysis and assuming no participants with missing data had the event (8.5% and 4%, respectively, of all 202 reviews). A small percentage of reviews (11%) reported a planned sensitivity analysis using different method(s) for handling missing participant data, with only 2% providing a justification for those methods.

Table 3.

Handling of, and assessing the risk of bias associated with missing participant data in Cochrane and non-Cochrane SRs

	Overall (N=202)	Cochrane SR (N=98)	Non-Cochrane SR (N=104)	p Value*
Reported plans for handling missing categorical data				<0.0001
Using complete case analysis	17 (8.5%)	15 (15.5%)	2 (2.0%)
Assuming no participants with missing data had the event	8 (4.0%)	7 (7.2%)	1 (1.0%)
Assuming all participants with missing data had the event	5 (2.5%)	4 (4.1%)	1 (1.0%)
Assuming participants with missing data had same event rate as those followed up in respective randomisation groups	2 (1.0%)	1 (1.0%)	1 (1.0%)
Using worst case scenario†	0 (0.0%)	0 (0.0%)	0 (0.0%)
Using best case scenario‡	1 (0.5%)	0 (0.0%)	1 (1.0%)
Using whatever assumptions the included trials used	1 (0.5%)	1 (1.0%)	0 (0.0%)
Using other assumption(s)	15 (7.5%)	12 (12.4%)	3 (3.0%)
No method described	150 (75.4%)	57 (58.8%)	93 (91.2%)
Reported a planned sensitivity analyses for handling missing participant data
Yes, clearly for categorical outcomes	12 (5.9%)	10 (10.2%)	2 (1.9%)	0.02
Yes, but unclear as to whether for categorical or continuous outcomes	11 (5.4%)	9 (9.2%)	2 (1.9%)	0.03
Provide a justification for any of the method(s) for handling missing participant data	4 (2.0%)	4 (4.2%)	0 (0.0%)	<0.0001
Assessed the risk of bias associated with missing participant data
Yes, using the Cochrane RoB tool (incomplete outcome data (attrition bias))	85 (42.1%)	76 (77.6%)	9 (8.7%)	<0.0001
Yes, using a tool other than the Cochrane RoB tool (eg, Jadad’s scale)	47 (23.3%)	6 (6.1%)	41 (39.4%)
Discussed the implications of missing participant data
Yes, in the Results section	23 (11.4%)	19 (19.4%)	4 (3.9%)
Yes, in the Discussion section	20 (10.0%)	16 (16.3%)	4 (3.9%)

*p Value for the difference between Cochrane and non-Cochrane SRs.

†Worst case scenario: assuming all participants with missing data in the intervention group had the event but none in the control group did.

‡Best case scenario: assuming that all participants with missing data in the control group had the event but none in the intervention group did.

RoB; Risk of Bias; SR, systematic review.

In our multivariable logistic regression, whether the systematic review used complete case analysis was associated only with the type of review (Cochrane vs non-Cochrane: OR=7.25; 95% CI 1.58 to 33.3, p=0.01). The variables for which no statistically significant associations were identified were: number of included trials and type of intervention.

Assessing risk of bias associated with missing participant data

Table 3 describes the assessment of the risk of bias associated with missing participant. Of all included systematic reviews, 65% reported assessing risk of bias associated with missing data: 42% used the Cochrane RoB tool; 23% used a tool other than the Cochrane RoB tool. Differences between Cochrane and non-Cochrane reviews were statistically significant (p<0.0001). Eleven per cent of included reviews discussed the implications of missing participant data in the Results section, and 10% addressed them in the Discussion section.

In our multivariable logistic regression, whether the systematic review judged the risk of bias associated with missing participant data was associated with Cochrane versus non-Cochrane type (OR=6.63; 95% CI 2.50 to 17.57, p=0.0001), and whether the systematic review used the GRADE approach to rate the confidence in effect estimates (OR=5.02; 95% CI 1.02 to 24.75, p=0.047). The variables for which no statistically significant associations were identified were: number of included trials, type of intervention and whether any of the authors reported ties to industry.

Discussion

Although Cochrane systematic reviews are less problematic, most Cochrane and non-Cochrane reviews do not adequately report and handle missing data. In general, 50% or less of Cochrane reviews met the criteria we explored, versus 20% or less of non-Cochrane reviews. Only 13% of reviews reported planned sensitivity analyses for handling missing participant data. Reporting on risk of bias associated with missing participant data was the least problematic item (65%). Better performance was associated with Cochrane versus non-Cochrane type, and whether the systematic review used the GRADE approach to rate confidence in effect estimates.

Our study has a number of strengths. This is the first study that assesses the reporting, handling and assessment of risk of bias associated with missing participant data among systematic reviews. We used explicit eligibility criteria, and sensitive search strategies to identify eligible studies. We also employed systematic, rigorous and transparent approaches to study selection and data abstraction, including calibration exercises, duplicate processes and use of standardised pilot-tested forms with detailed instructions. We included both Cochrane and non-Cochrane systematic reviews and used broad eligibility criteria to make our results more generalisable.

The main limitation of our study is that we did not review the individual trials that contributed to eligible systematic reviews, and the very low percentage of systematic reviews reporting on potential reasons for missing data might be due to the poor reporting of such information at the trial level. While the restriction of our search to MEDLINE database could have affected the representativeness of systematic reviews, we believe the included reviews represent those typically accessed by clinicians.

Although all reviews performed poorly in terms of reporting, Cochrane reviews performed better than non-Cochrane reviews. Likely explanations include the availability of a methodological guidance (ie, the Cochrane Handbook), and the use of standardised tables in Cochrane (ie, the Cochrane RoB tool). Another potential explanation is the lack of space constraints for the publication of Cochrane systematic reviews. It is important to note that non-Cochrane reviews are likely to be a highly heterogeneous in terms of methodological and reporting characteristics. Unfortunately, our study lacked the power to explore that hypothesis.

A recent study examined how 100 Cochrane systematic reviews and 100 non-Cochrane systematic reviews published in 2012 assessed risk of bias of primary studies and incorporated them into their statistical analysis and overall findings.⁷ The investigators found that incomplete outcome data, defined as missing outcome data due to attrition, were reported in 95% of Cochrane reviews compared with 61% of non-Cochrane reviews. We found substantially lower figures that are probably related to lower percentage of reviews using the GRADE approach in our sample (16% vs 45%).

That study found that 8% and 1% of Cochrane and non-Cochrane reviews, respectively, assessed incomplete outcome data for more than one outcome. In our study, we assessed whether the systematic reviews reported the number of participants with missing data for each outcome. The percentages were similarly very low at 2% and 0% for Cochrane and non-Cochrane reviews, respectively. This is problematic because missingness of data may actually vary across outcomes (eg, due to different follow-up times for different outcomes).

We recently suggested a simple guidance for addressing dichotomous data for participants excluded from analyses of randomised trials and for assessing the associated risk of bias.¹⁴ Briefly, the guidance suggests for the primary analysis, either a complete case analysis or making plausible assumptions about the outcomes of participants with missing data. When the primary analysis suggests important benefit, the guidance recommends sensitivity meta-analyses using relatively extreme assumptions that may vary in plausibility to assess the associated risk of bias. We have developed a similar guidance for continuous data, including situations where continuous data are measured with different instruments.^{15 16} Other authors have similarly published recommendations for addressing missing data in meta-analysis.^17–20 Our findings highlight the need for better compliance with and enforcement of the Cochrane Collaboration Handbook and the PRISMA statement's recommendations related to reporting of missing participant data.^{4 5} This would be facilitated by better reporting of missing data at the trial level,²¹ given the systematic review authors can typically rely on what the authors of primary studies report, as well as easing of space constraints in reporting, of non-Cochrane reviews in particular. Systematic reviewers also need to assess the risk of bias associated with missing participant data, and how it affects the confidence in the effects estimates.

The findings have also implications for methodological research. A number of approaches for handling missing data in meta-analysis are available.^14–20 We now need to compare the impact of these methods on the statistical significance of pooled effect estimates and on the associated quality of evidence.