Methods and characteristics of published network meta-analyses using individual patient data: protocol for a scoping review

Abstract

Introduction

Individual patient data (IPD) meta-analysis (MA) offers advantages over aggregate MA of using standardised criteria for patient characteristics across trials, and allowing reliable investigation of subgroup effects of interventions. Network meta-analysis (NMA) allows for the comparison of multiple treatments in a comprehensive analysis and the determination of the best treatment among several competing treatments, including those that have never been compared in a head-to-head study. Including IPD in NMA may enable the prevention of misleading inferences due to several biases, such as aggregation bias. Application of IPD-NMA methods in healthcare have begun to appear in medical journals. Our objective is to conduct a scoping review of existing IPD-NMA methods, and summarise their properties. We also aim to describe the characteristics of empirical IPD-NMAs, and examine how their results are reported.

Methods and analysis

We will search relevant electronic databases from inception until October 2014 (eg, MEDLINE), grey literature, and Google. The scoping review will consider published and unpublished papers that report completion of an IPD-NMA, describe a method, or report the methodological quality of IPD-NMA. We will include IPD-NMA of any quantitative study (eg, experimental, quasiexperimental, observational studies). Two reviewers will independently screen titles, abstracts and full-text articles, and will complete data abstraction. The anticipated outcome will be a collection of all the IPD-NMAs completed to date, and a description of their methods and reporting of results. We will create summary tables providing the characteristics of the included studies, and the various methods. Quantitative data (eg, number of patients) will be summarised by medians and IQRs, and categorical data (eg, type of effect size) by frequencies and percentages.

Ethics and dissemination

Ethical approval is not required as our study will not include confidential participant data and interventions. We will disseminate our results through an open access, peer-reviewed publication.

Strengths and limitations of this study.

• Network meta-analysis (NMA) using individual patient data can increase power and identify interactions between treatment effect and a covariate not detected with aggregated data.

• This study will be the first scoping review that will provide a comprehensive overview and description of the specific steps of the methods for completing an individual patient data NMA, as well as an insight into the characteristics of NMAs with individual patient data in healthcare research.

• This scoping review will be limited only to English language publications.

• This review focuses on the presentation and description of the methods and characteristics and reporting of individual patient data NMAs, but does not assess the quality of papers or methods themselves.

Introduction

Healthcare providers, policymakers, and consumers of healthcare services make decisions regarding alternative healthcare options, such as choosing from antiemetic medications used to prevent nausea for patients undergoing chemotherapy. Many organisations, including the Canadian Agency for Drugs and Technologies in Health (CADTH), National Institute for Health and Care Excellence, Agency for Healthcare Research and Quality, and WHO, consider knowledge synthesis and meta-analysis (MA) as the ‘base unit’ for knowledge translation activities, which provide the most reliable and valid evidence on which to base healthcare decisions.^{1 2}

Meta-analyses can be conducted using two distinct sources of data. Aggregated data (AD) MA utilises summary point estimates derived from all participants enrolled in a trial. Individual patient data (IPD) MA, by contrast, utilises patient-level data (ie, data collected from each participant in the trial). IPD is usually obtained directly from trial authors. Although most meta-analyses have used AD to date, AD-MA may suffer from relatively low statistical power for detecting a treatment by covariate interaction, and introduces potential aggregation bias.

Aggregation bias is also known as ecological fallacy in the epidemiological literature, and this bias may occur if one (incorrectly) assumes that relationships observed at the group level hold at the individual level as well.^3–5 IPD-MA is considered the ‘gold-standard’ approach for synthesising evidence across clinical trials, as it has numerous advantages. IPD-MA is particularly valuable when exploring phenomena which tend to be inconsistently analysed or reported; when there is a need for adjustment due to confounding, such as in observational studies, or when evaluating interactions between treatment and a covariate, such as sex (men and women), and geographical location.⁶ Knowledge about effectiveness of interventions in different subgroups is particularly important for decision-making. For example, while oral anticoagulants are effective in reducing stroke in all patients with non-valvular atrial fibrillation, we know that older patients (ie, 75 years of age and older) are at the highest risk of a stroke and achieve greater benefit than patients less than 65 years.⁷ Similarly, we know that this older age group is at a higher risk of bleeding with these agents.⁷ By contrast with AD-MA, IPD-MA has greater statistical power to detect participant-treatment relationships, as it allows participant-level covariates to be directly modelled.^8–10

Several surveys have shown that the use of IPD has increased significantly over the last decade, however, researchers often do not take into account the study cluster, but instead, analyse the data as a large database resulting in invalid results.^{9 11 12} It has been shown that most researchers apply a two-step analysis method for MA by first producing aggregate data for each study, and then synthesising the study results using AD-MA, whereas a one-stage analysis is commonly applied to model the individual effects clustered within a particular study.⁶ In the two-stage approach for IPD-MA, the associations between treatment and participant characteristics can be investigated via subgroup analysis or meta-regression analysis. However, it has been shown that these methods lack statistical power, and using trial-level data fails to detect such interactions.¹³ Subgroup analyses are optimally informed by the conduct of one-stage IPD, especially when a small number of studies with small sample size are included in the MA.

Pairwise MA methods for clinical trials focus on comparing two interventions, such as a drug versus placebo, or a new intervention versus standard practice. Often we lack evidence from head-to-head trials on active treatments. Moreover, there are rarely only two interventions under consideration in clinical practice, and a plethora of analyses would be needed to draw conclusions. Extensions of MA to compare three or more treatments have been the subject of substantial methodological research in recent years.¹⁴ The simplest extension is indirect comparison MA, which can be performed in ways that respect the randomisation within each clinical trial.¹⁵ More complex extensions are network meta-analysis (NMA) models that have been widely used during the last decade, which allow the simultaneous analysis of clinical trials involving different treatments.^{16 17} NMA is more advantageous than both pairwise MA and indirect comparison MA, as it provides the ability to (1) increase precision of point estimates, (2) draw inferences on the comparability between interventions that have never been compared in a clinical trial, and (3) rank the interventions according to their efficacy and safety.^16–18

NMA is commonly performed using AD, and the value of IPD in NMA is currently unknown. Although the use of IPD has been extensively evaluated in pairwise MA, little is known about the value of IPD in the evaluation of the consistency assumption, and particularly in the presence of substantial heterogeneity. Simulation studies have shown that the available approaches to assess the consistency assumption in NMA using AD have low power to detect inconsistency.^{19 20} However, since NMA is an extension of pairwise MA and IPD, MA has been shown to be advantageous,^8–10 we expect that this will hold true for NMA models. IPD-NMA models can increase precision in the results, as both within-study and across-study data are taken into account, decreasing heterogeneity.^{21 22} A key advantage of IPD-NMA is the ability to identify interactions that cannot be detected when using AD due to patient-level effect modifiers across treatment comparisons.^{3–5 9} The imbalance in patient-level effect modifiers across treatment comparisons can lead to misleading results.²¹ For instance, Donegan et al²¹ compared four interventions for treating malaria, using both IPD-NMA and AD-NMA models. The IPD-NMA models suggested that dihydroartemisinin piperaquine was the best drug for all patients, whereas the AD-NMA models suggested that the best drug varied depending on patients’ mean age. The difference in the ranking of the drugs between the two models was due to the differences in the distribution of the covariates within and across studies. Incorporating IPD in NMA makes it possible to use advanced modelling strategies to explore subject-level covariates as potential treatment-effect modifiers reducing statistical heterogeneity across the network.²³ Jansen²⁴ showed that combining IPD with AD minimises the chances of confounding bias being evident in indirect comparison and NMA. These methods have begun to appear in publications^{25 26} and in clinical practice guidelines.^{27 28}

Recently, several researchers have recognised that the use of IPD in NMA may provide the most trustworthy evidence, and hence they have been developing statistical methods to complement and enhance IPD-NMA.^{21–23 29} We aim to conduct a comprehensive scoping review of the available methods to apply IPD-NMA or combine IPD with AD in NMA, and summarise the properties of these methods. In particular, our objectives are to:

1. Identify and describe the process and properties of each IPD-NMA method;

2. List the advantages and disadvantages of each IPD-NMA method;

3. Describe any similarities and differences between the IPD-NMA methods;

4. Describe how the IPD-NMA methods differ from the AD-NMA methodology.

We also aim to provide a comprehensive description of the empirical IPD-NMAs, and examine how the results are reported in IPD-NMAs. We will further identify which key elements should be reported when conducting an IPD-NMA.

Methods and analysis

This project will facilitate the identification of gaps and methodological deficiencies in the existing literature. We will (1) conduct a systematic search of the literature for IPD-NMA methods across multidisciplinary fields and (2) describe the specific steps to conducting the IPD-NMA using the scoping review methods of Arksey and O'Malley.³⁰

Search strategy

We intend to search health-related databases such as MEDLINE, EMBASE, The Cochrane Library and CINAHL from inception until the end of October 2014 for potentially relevant articles. A search of the ‘grey’ literature, including difficult-to-locate, or unpublished material, will be conducted by searching conference abstracts, as well as general internet searches using several web search engines (eg, Google), and the approach outlined by CADTH.³¹ The search will be carried out by an experienced librarian (Ms Becky Skidmore), and a second librarian (Ms Heather MacDonald) will peer review the main (MEDLINE) electronic search strategy using the Peer Review of Electronic Search Strategies (PRESS) checklist.³² A draft literature search for the search strategy for MEDLINE can be found in online supplementary appendix A. The search strategy will be modified as necessary for EMBASE, The Cochrane Library, and CINAHL. References from included studies will be scanned for additional relevant articles. We will use our networks of professional collaborations to contact methodological experts in the field and identify further articles. An updated search will also be performed when we are close to the completion of the review to look for any new and important reports meeting our eligibility criteria that may have recently been published.

Eligibility criteria

We will include all studies that report the development, comparison, use or methodological quality of IPD-NMA. IPD-NMAs that include any quantitative study (eg, experimental studies (randomised controlled trials (RCTs), quasi-RCTs, non-RCTs), quasiexperimental studies (controlled before and after studies, interrupted time series) and observational studies (cohort, case control studies)) related to health will be eligible for inclusion. We will use the WHO definition for health which includes complete mental, physical and social well-being (http://www.who.int/about/definition/en/print.html). We will exclude commentaries, as well as reviews not involving human participants or not pertaining to healthcare. Published papers, protocols, abstracts, or unpublished studies (eg, dissertations) that compare the clinical efficacy of three or more interventions will be eligible for inclusion. A draft eligibility form is presented in online supplementary appendix B. There will be no restrictions on publication status or date of publication. Only studies written in English will be eligible for inclusion due to resource limitations. However, a list of IPD-NMAs published in other languages will be included as an online supplementary appendix in the final paper.

Study selection

We will use the Synthesi.SR Tool to import the search results and to screen citations and full-text articles (http://knowledgetranslation.ca/sysrev/login.php). To ensure reliability, a training exercise will be conducted prior to commencing screening. Using the eligibility criteria, a random sample of 10% of citations from the search will be screened by all reviewers. Inter-rater agreement for study inclusion will be calculated using per cent agreement; if it is >90% across the team, we will proceed to the next stage. If poor agreement is found, inclusion and exclusion criteria will be modified and clarified with the team. A second test with another 10% of citations will be completed, and screening will only begin when agreement is >90%. Two reviewers (CS or MJE, and AAV) will screen each title and abstract for inclusion, independently (level 1). They will then independently review the full text of potentially relevant articles to determine inclusion using the inclusion and exclusion criteria (level 2). A training exercise will be conducted if the eligibility criteria for level 2 screening (based on refinements to criteria following level 1) differ from level 1 screening. Studies that do not fulfil the eligibility criteria in level 1 screening, will be excluded and not considered for inclusion at level 2 screening. Conflicts will be resolved by discussion or involvement of a third reviewer (SES or ACT). The process of the study selection will be reported using the PRISMA flow diagram.³³

Data abstraction

Abstracted data will include general study characteristics, for example, authorship and publication-related information, as well as characteristics according to the type of article. For example, we will provide the general description of methods including steps, the required data type (eg, both AD and IPD or IPD alone), the outcome data type (eg, dichotomous, continuous), the advantages and disadvantages of the approach as well as the similarities and differences with the AD methods as reported by the authors, but also from our own perspective, for a methodological paper. For an application paper (ie, a paper that reports a completed IPD-NMA), we will describe the general characteristics of the network (eg, number of patients and studies), rationale for using IPD-NMA, methodological considerations to ensure transparency of methods (eg, statistical methods and assumptions), IPD-NMA methods applied along with their advantages and/or disadvantages that might be reported, as well as approaches to summarise results from IPD-NMA (eg, presentation of figures and tables). For an IPD-NMA review, we will extract the process used to identify relevant studies (eg, literature review), describe if the study is an overview of systematic reviews or intervention review of primary studies, and the study design of the included studies (eg, RCTs, observational). If the review assesses the quality of IPD-NMA, we will also abstract the tool that was used for the assessment, and the criteria that were assessed. A draft data abstraction form is presented in online supplementary appendix C. Each example presented in a methodological paper will be treated as a separate application and we will abstract all relevant data as necessary (online supplementary appendix C). The data abstraction form will be piloted on a random sample of 10% of included articles and modified as required. Data abstraction will only begin when sufficient agreement is noted (ie, per cent agreement >90%). To ensure accuracy, two reviewers (CS or MJE, and AAV) will independently abstract all data; discrepancies will be resolved by discussion.

Risk of bias appraisal

We will not appraise methodological quality or risk of bias of included articles, since this is a scoping review.³⁰

Synthesis

Data analysis will involve quantitative (ie, frequency analysis) and qualitative (ie, content analysis) methods. The approach is based on methods we used to complete three scoping reviews funded by the Canadian Institutes of Health Research of knowledge synthesis methods, rapid reviews, and scoping reviews, which used the methods of Arksey and O'Malley as a framework.^{30 34 35} Specifically, we will collect information on the methods used for IPD-NMA and extract the specific steps for each method. Two people (CS and AAV) will conduct the initial categorisation independently using NVivo V.10 (http://www.qsrinternational.com/products_nvivo.aspx). The anticipated outcome will be a collection of all the IPD-NMAs completed to date, and a description of their methods and their results. We will also create summary tables providing the characteristics of the included studies, and matrix tables to compare and contrast the IPD-NMA methods themselves as well as with AD-NMA approaches. Quantitative data from the retrieved IPD-NMA (eg, number of patients, studies and treatments in the network) will be summarised by medians and IQRs, and categorical data (eg, effect measure, outcome data type, reference treatment type (eg, placebo, control, active) by frequencies and percentages.

Ethics and dissemination

To date, there has been an increase in publication of systematic reviews that apply NMA, but the related statistical methodology, including the use of IPD-NMA is continuously evolving.¹⁴ IPD-NMA has the potential to provide modelling flexibility (eg, including baseline characteristics), help reduce heterogeneity across the network, or resolve possible inconsistencies which cannot be explicitly explored through AD, and produce unbiased results that would otherwise be affected by aggregation bias. The key strength of using IPD is that it can lead to more precise estimates of treatment effects even in the absence of treatment-covariate interactions. However, accessing and analysing IPD can be time-consuming and may cause delay, and detailed information on individual-level data is rarely available from all eligible trials. If IPD are available for a subset of studies and AD for the remaining studies, IPD may be combined with AD^{6 11 36 37} although the impact of this approach on IPD-NMA is not yet known.

The proposed scoping review will be the first study that will provide a comprehensive overview of the methods for completing an IPD-NMA, as well as an insight into the characteristics of the IPD-NMAs in healthcare research. This study does not require formal ethical assessment and informed consent, as no confidential participant data and interventions will be included. Our findings will provide guidance for an appropriate application of the IPD-NMA technique and, thereby, strengthen the validity of this research. We will also identify gaps where methodology is lacking, and we will be able to highlight the potential for novel statistical advances necessary to evaluate the key assumptions in NMA.

Further research will be necessary though to establish the benefits of IPD in various settings, as well as the properties of the IPD-NMA models in networks of interventions with complex evidence structure. Simulation and empirical studies will be needed to evaluate the assumptions and the properties of the IPD-NMA and provide evidence on whether IPD-NMA is more valuable than AD-NMA.

Our research will be useful to statistical researchers, health professionals and methodologists who aim to compare multiple interventions. In order to ensure that our results have wide dissemination and uptake, we will publish our results in open-access journals, present and discuss them at local, national and international conferences (eg, Cochrane Colloquium) with various audiences.