Standards in the Prevention and Handling of Missing Data for Patient Centered Outcomes Research – A Systematic Review and Expert Consensus

Abstract

Objectives

To recommend methodology standards in the prevention and handling of missing data for primary patient-centered outcomes research (PCOR).

Study Design and Setting

We searched National Library of Medicine Bookshelf and Catalog, regulatory agencies’ and organizations’ websites in January 2012 for guidance documents that had formal recommendations regarding missing data. We extracted the characteristics of included guidance documents and recommendations. Using a two-round modified Delphi survey, a multidisciplinary panel proposed mandatory standards on the prevention and handling of missing data for PCOR.

Results

We identified 1,790 records and assessed 30 as having relevant recommendations. We proposed 10 standards as mandatory, covering three domains. First, the single best approach is to prospectively prevent missing data occurrence. Second, use of valid statistical methods that properly reflect multiple sources of uncertainty is critical when analyzing missing data. Third, transparent and thorough reporting of missing data allows readers to judge the validity of the findings.

Conclusions

We urge researchers to adopt rigorous methodology and promote good science by applying best practices to the prevention and handling of missing data. Developing guidance on the prevention and handling of missing data for observational studies and studies that utilize existing records is a priority for future research.

Introduction

Patients and health care providers need complete, accurate, reliable, and timely information to make health care decisions. Patient-centered outcomes research (PCOR) focuses on comparative clinical effectiveness research that “helps people and their caregivers communicate and make informed health care decisions, allowing their voices to be heard in assessing the value of health care options (http://www.pcori.org/research-we-support/pcor/; accessed on April 5, 2013)” with special emphasis of a patient-centered perspective. The scientific integrity of the design, conduct, analyses, and reporting of PCOR can be threatened by missing data. Missing data refers to unrecorded values, which, if recorded, would be meaningful for analysis and interpretation of a study [1–3]. When missing data are related to an intervention, to patient outcomes, or to patient prognosis, inferences about treatment effects may suffer from bias induced by nonrandom selection of data into the analysis [1–3]. For example, if patients experiencing adverse events are more likely to cease participation in the trial and that adverse events occur more frequent with the new treatment than the standard of care, then inappropriately accounting for the missing data may lead to biased inference of the comparative effectiveness of the two treatments. Even when missing data occur completely randomly, for example mishandling of tissue samples, the smaller sample size due to the randomly missing data leads to decreased statistical power.

Missing data can occur in any type of primary PCOR, regardless of study design (e.g., randomized controlled trials (RCTs) or observational studies such as cohort, case-control, and cross-sectional studies), type of data source (e.g., registry data, electronic health records), or whether the study is prospective or retrospective in data collection (e.g., data collected from clinical records). When investigators are in control of data collection, strategies can be built into the design to minimize the occurrence of missing data. However, when PCOR is based on existing records such as electronic health records, patient registries, and administrative datasets, prevention may be more difficult to implement, leading to greater reliance on analytic methods to account for missing data.

Understanding good practices and creating standards for the prevention and handling of missing data can help to improve the translation of the research into complete, accurate, and reliable evidence for health care decision-making. In this project, we conducted a systematic review of existing guidance documents on the prevention and handling of missing data in primary PCOR. As part of the project, we also used a 2-round Delphi consensus survey of experts to propose minimum methodological standards in the prevention and handling of missing data for use in assessing grant applications to the Patient-Centered Outcomes Research Institute (PCORI).

Methods

Systematic review of guidance documents on missing data

Eligibility criteria

For our systematic review, we defined eligible “guidance documents” as those containing research guidelines, recommendations, principles, requirements, regulations, standards, statements, and relevant literature on the prevention and handling of missing data for any type of primary research in humans. An eligible guidance document had to include at least one formal recommendation about how missing data should be prevented or handled.

Searches and study selection

We worked with an information specialist to develop a search strategy, which was peer reviewed by two independent information specialists (Appendix lists our search strategies). We searched the National Library of Medicine (NLM) Bookshelf on January 9, 2012 and NLM Catalog on January 14, 2012 without any language or publication date restrictions. To supplement the electronic searches, we searched major relevant regulatory agencies’ and organizations’ websites in January 2012 (Appendix lists websites we searched). We also hand-searched the reference lists of included guidance documents we identified.

Because our searches identified mostly documents on the prevention and handling of missing data for RCTs, we made special efforts to identify guidance related to observational studies. We contacted 47 experts who perform research that does not routinely include clinical trials to suggest additional guidance documents. These experts included key persons working with the National Cancer Institute’s Surveillance Epidemiology and End Results (SEER) program, senior epidemiologists at health maintenance organizations (e.g., Kaiser Permanente Division of Research), investigators following large AIDS cohorts, and experts who have focused on using claims data and electronic medical records in their research.

Two team members independently reviewed the titles and abstracts identified by the electronic searches. We retrieved the full text of all guidance documents that were classified by one or more person as possibly eligible. At least two team members independently reviewed full text reports for determining final eligibility, resolving discrepancies through discussion. We documented reasons for exclusion.

Data extraction

We extracted the characteristics of included guidance documents, as well as recommendations within each guidance document on the prevention and handling of missing data (see Table 1 and Appendix Table 3 for items we extracted). One person extracted the data and a second person verified the extracted data by comparing to the original text.

Table 1.

Description and characteristics of the 30 included guidance documents

Title of document	Description	Year of publication	Organization, program, or authors	Pertinent standards in Box 1
A. Randomized controlled trial (RCT)
The Prevention and Treatment of Missing Data in Clinical Trials (1)	The purpose of the document is to provide 18 recommendations to address missing data in clinical trials through careful trial design, conduct, and analysis. This document was created at request of the US Food and Drug Administration (FDA) for a future FDA guidance report on missing data.	2010	National Research Council, Committee on National Statistics	1,2,3,4,6,7,8
Statistical Principles for Clinical Trials; Step 5: Note for Guidance on Statistical Principles for Clinical Trials. International Conference on Harmonisation (ICH) Topic E9 (2)	The purpose of the guidance is to harmonize the principles of statistical methodology applied to clinical trials for marketing applications submitted in Europe, Japan and the United States. The focus of the guidance is on statistical principles rather than the use of specific statistical procedures or methods. The principles outlined in this guidance are primarily relevant to clinical trials conducted in the later phases of development, many of which are confirmatory trials of efficacy.	1998	European Medicines Agency; Expert Working Group (Efficacy) of ICH	1,2,3,7,8,10
Guideline on Missing Data in Confirmatory Clinical Trials (3)	The purpose of the guideline is to provide advice on how the presence of missing data in confirmatory clinical trials should be addressed and reported in a dossier submitted for regulatory review and provides an insight into the regulatory standards that will be used to assess confirmatory clinical trials with missing data.	2010	European Medicines Agency	2,4,8
Guidance for industry: E9 statistical principles for clinical trials (8)	The purpose of the guidance is to give direction to sponsors in the design, conduct, analysis, and evaluation of clinical trials of an investigational product in the context of its overall clinical development. The document will also assist scientific experts charged with preparing application summaries or assessing evidence of efficacy and safety, principally from clinical trials in later phases of development. This guidance is written primarily to attempt to harmonize the principles of statistical methodology applied to clinical trials for marketing applications submitted in Europe, Japan and the United States.	1998	Expert Working Group (Efficacy) of ICH	1,2,3,4,5,8,9
Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (9)	The purpose of the guidance is to describe how the FDA reviews and evaluates existing, modified, or newly created patient-reported outcome instruments used to support claims in approved medical product labeling. A patient-reported outcome instrument (i.e., a questionnaire plus the information and documentation that support its use) is a means to capture patient-reported data used to measure treatment benefit or risk in medical product clinical trials.	2009	FDA	1,2,3,4,8
Guidance for Industry: E6 Good Clinical Practice Consolidated Guidance (13)	The purpose of the ICH Good Clinical Practice guidance is to provide a unified standard for the European Union, Japan, and the United States to facilitate the mutual acceptance of clinical data by the regulatory authorities in these jurisdictions. The guidance was developed with consideration of the current good clinical practices of the European Union, Japan, and the United States, as well as those of Australia, Canada, the Nordic countries, and the World Health Organization. This guidance should be followed when generating clinical trial data that are intended to be submitted to regulatory authorities. The principles established in this guidance may also be applied to other clinical investigations that may have an impact on the safety and well-being of human subjects.	1996	Expert Working Group (Efficacy) of the ICH	2,3,9
Guideline for Industry: Structure and Content of Clinical Study Reports (14)	The purpose of the guideline is to facilitate the compilation of a single core clinical study report acceptable to all regulatory authorities of the ICH regions. The clinical study report described in this guideline is an "integrated" full report of an individual clinical trial of any therapeutic, prophylactic, or diagnostic agent (referred to herein as drug or treatment) conducted in patients.	1996	Expert Working Group (Efficacy) of the ICH	2,3,8,9,10
Guidance for Sponsors, Clinical Investigators, and IRBs. Data Retention When Subjects Withdraw from FDA-Regulated Clinical Trials (17)	The purpose of the guidance is to describe the FDA's longstanding policy that already-accrued data, relating to individuals who cease participating in a clinical trial, are to be maintained as part of the study data. This guidance is intended for sponsors, clinical investigators and institutional review boards (IRBs).	2008	FDA	4
Guidance for Clinical Trial Sponsors: Establishment and Operation of Clinical Trial Data Monitoring Committees (20)	The purpose of the guidance is to discuss the roles, responsibilities, and operating procedures of Data Monitoring Committees (also known as Data and Safety Monitoring Boards or Data and Safety Monitoring Committees) that may carry out important aspects of clinical trial monitoring. This guidance is intended to assist clinical trial sponsors in determining when a Data Monitoring Committee may be useful for study monitoring, and how such committees should operate.	2006	FDA	5
Clinical Investigations of Medical Devices for Human Subjects; Part 1: General Requirements (19)	The purpose of the standard is to specify requirements for the conduct of a clinical trial to establish the performance of medical devices intended to mimic normal clinical use, reveal adverse events under normal conditions of use, and permit assessment of the acceptable risks.	2004	This Australian Standard was prepared by Committee HE-012, Surgical Implants. It was approved on behalf of the Council of Standards Australia; The following are represented on Committee HE-012: Australian College of Operating Room Nurses, Australian Dental Association, Australian Industry Group, Australian Orthopaedic Association, Commonwealth Department of Health and Ageing, Department of Defense (Australia), Medical Industry Association of Australia Inc, Neurosurgical Society of Australasia, Royal Australasian College of Surgeons, Royal Perth Hospital, The Australian Society for Biomaterials, University of New South Wales, University of Sydney	5,10
Good Research Practices for Cost-Effectiveness Analysis Alongside Clinical Trials: The ISPOR RCT-CEA Task Force Report (21)	The purpose of the guidance document is to provide good research practices for designing, conducting, and reporting cost-effectiveness analyses conducted as a part of clinical trials.	2005	International Society for Pharmacoeconomics and Outcomes Research (ISPOR)	6
CONSORT Statement 2010 (28)	The purpose of the Consolidated Standards of Reporting Trials (CONSORT) statement is to help to improve the reporting of randomized controlled trials.	2010	CONSORT Group	9,10
B. Non-RCT
Good Research Practices for Comparative Effectiveness Research: Analytic Methods to Improve Causal Inference from Nonrandomized Studies of Treatment Effects Using Secondary Data Sources: The ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report--Part III (10)	The purpose of the report is to review recent developments in statistical control of confounding to improve causal inference of comparative treatment effects from nonrandomized studies using secondary databases.	2009	ISPOR	1
Registries for Evaluating Patient Outcomes: A User's Guide. 2nd ed. (11)	The purpose of the document is to serve as a reference for establishing, maintaining, and evaluating the success of registries created to collect data about patient outcomes.	2010	Agency for Healthcare Research and Quality (AHRQ), the Centers for Medicare & Medicaid Services	2,7,10
Prospective Observational Studies to Assess Comparative Effectiveness: ISPOR Good Research Practices Task Force Report (Draft) (15)	The purpose of the report is to recommend good practices in the design, conduct, analysis and reporting of prospective observational studies to address a comparative effectiveness or relative effectiveness research question.	2011 (Draft)	ISPOR	2
Guidance for Industry and FDA Staff. Best Practices for Conducting and Reporting Pharmacoepidemiologic Safety Studies Using Electronic Healthcare Datasets (Draft) (16)	The purpose of the guidance is to describe best practices pertaining to conducting and reporting on pharmacoepidemiologic safety studies that use electronic healthcare data sets, which include administrative claims data and electronic medical record data. The guidance includes recommendations for documenting the design, analysis, and results of pharmacoepidemiologic safety studies to optimize FDA’s review of protocols and final reports that are submitted to the Agency for these types of studies. For purposes of this guidance, the term pharmacoepidemiologic safety study refers to an observational study designed to assess the risk attributed to a drug exposure and to test pre-specified hypotheses.	2011 (Draft)	FDA	2
Good Research Practices for Comparative Effectiveness Research: Approaches to Mitigate Bias and Confounding in the Design of Nonrandomized Studies of Treatment Effects Using Secondary Data Sources: The International Society for Pharmacoeconomics and Outcomes Research Good Research Practices for Retrospective Database Analysis Task Force Report--Part II (29)	The purpose of the guidance document is to recommend methodological approaches for making causal inference with observational studies using secondary data source, with the goal to mitigate bias and confounding.	2009	ISPOR	Not included in the final standards
A Checklist for Retrospective Database Studies—Report of the ISPOR Task Force on Retrospective Databases (31)	The purpose of the report is to assist decision makers in evaluating the quality of published studies that use health related retrospective databases. A checklist was developed that focuses on issues that are unique to database studies or are particularly problematic in database research. Although the checklist was developed primarily for the commonly used medical claims or encounter-based databases, it could potentially be used to assess retrospective studies that employ other types of databases, such as disease registries and national survey data.	2003	ISPOR	Not included in the final standards
STROBE Statement (27)	The purpose of the statement is to improve reporting of observational studies. It is a checklist of 22 items that considered essential for good reporting of observational studies.	2008	Strobe Initiative; Erik von ElmPeter C. Gøtzschea,g, Douglas G. Altmanc, Matthias Eggera,b,*, Stuart J. Pocockd,e, Jan P. Vandenbrouckeffor the STROBE Initiative	10
Modeling Discrete Event Simulation: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group - Part 6 (Draft) (32)	The purpose of the paper is to provide consensus setting, covering the many forms of analysis to which discrete event simulation (DES) can be applied. The paper works through the different stages of the modeling process: structural development, parameter estimation, model implementation, model analysis, and representation and reporting. At each stage, a brief description of the relevant DES processes are provided, followed by consideration of issues that are of particular relevance to the application of DES in a healthcare setting.	2011 (Draft)	ISPOR	Not included in the final standards
Comparative Effectiveness Review Methods: Clinical Heterogeneity (22)	The purpose of the document is to summarize studies on how to handle clinical heterogeneity and subgroup analyses for producing comparative effectiveness reviews.	2010	RTI International - University of North Carolina Evidence-based Practice Center, prepared for Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services	8
Methods Guide for Effectiveness and Comparative Effectiveness Reviews (Draft) (23)	The purpose of the document is to provide guidance on how to conduct effectiveness and comparative effectiveness reviews.	2011	A collaborative efforts from the Agency for Healthcare Research and Quality, the Scientific Resource Center, and the Evidence-based Practice Centers	8
Interpreting Indirect Treatment Comparisons and Network Meta- Analysis for Health-Care Decision Making: Report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: Part 1 (26)	The purpose of the report is to recommend good research practices for indirect treatment comparisons and network meta-analysis.	2011	ISPOR	10
C. No specific study design
Conceptual Modeling: A Report of the ISPOR- SMDM Modeling Good Research Practices Task Force Working Group - Part 2 (Draft) (7)	The purpose of the paper is to summarize good research practices regarding the development of a model from the conceptualization of the problem that the model is expected to inform, through the representation of the disease or health care processes of concern in the form of a decision model. The recommendations apply most directly to models whose explicit purpose is to structure evidence on clinical and economic outcomes in a form that can help decision makers to choose from among competing courses of action and to allocate limited resources in health and medicine.	2011 (Draft)	ISPOR	1,8
Handbook for Good Clinical Research Practice (GCP): Guidance for Implementation (12)	The purpose of the handbook is to assist national regulatory authorities, sponsors, investigators and ethics committees in implementing GCP for industry sponsored, government-sponsored, institution-sponsored, or investigator-initiated clinical research.	2002	World Health Organization	2,3,9
Guidance on Important Considerations for When Participants of Human Subjects in Research is Discontinued (Draft) (18)	The purpose of the document, when finalized, will represent Office for Human Research Protections' considerations for when participation of human subjects in research is discontinued, either because a subject voluntarily choose to discontinue participation during the course of the research, or because an investigator terminates a subject's participation in the research without regard to the subject's consent. The guidance applies to non-exempt human subjects research conducted or supported by the Department of Health and Human Services.	2008 (Draft)	Office for Human Research Protections, Department of Health and Human Services	4
Reviewer Guidance: Conducting a Clinical Safety Review of a New Product Application and Preparing a Report on the Review: Good Review Practices (24)	The purpose of the good review practice guidance is to assist reviewers conducting the clinical safety reviews as part of the new drug application and biological license application review process, provide standardization and consistency in the format and content of safety reviews, and ensure that critical presentations and analyses will not be inadvertently omitted.	2005	FDA	8,10
Using Real-World Data for Coverage and Payment Decisions: The ISPOR Real-World Data Task Force Report (25)	The purpose of the paper is to develop a framework to assist health- care decision makers in dealing with real-world data and information in health-care decision-making, especially related to coverage and payment decisions.	2007	ISPOR	8
The Learning Healthcare System: Workshop Summary (30)	The purpose of the document is to summarize a workshop held in July 2006. The aim of the workshop is to identify and discuss the broad range of issues in research if we are to meet the ever-growing demand for evidence that will help bring better health and economic value for our sizable investment in health care.	2007	Institute of Medicine Roundtable on Evidence-based Medicine	Not included in the final standards
Selecting Quality and Resource Use Measures A Decision Guide for Community Quality Collaboratives (33)	Selecting quality of care and resource use measures is an important and challenging task for organizations striving to improve the quality of health care in their communities. The decision guide is designed to inform readers about the most critical issues to consider when selecting and adopting such performance measures.	2010	Romano PS, Hussey P, Ritley D. Prepared for AHRQ	Not included in the final standards

Delphi survey to identify minimum standards for PCORI grant applications

We categorized recommendations identified and four team members (TL, SH, DS, and KD) independently reviewed, refined, and condensed the list. We then used a two-round modified Delphi survey online to achieve consensus on recommended minimum standards across a panel of 10 experts between February 17 and February 28, 2012. The panel members were multidisciplinary and have extensive experience in missing data research. The list of panel members can be found in the Acknowledgement. The Delphi approach is a structured process of obtaining opinion and information from a group of experts by means of a series of consultation and questionnaires, each one refined based on the feedback from previous responses [4].

To ensure clarity and consistent understanding among those surveyed, we communicated to the consensus panel that the recommended standards will “inform investigators requesting Patient-Centered Outcomes Research Institute (PCORI) funding and assist grant reviewers in evaluating research proposal so as to ensure methodological rigor in PCOR (http://www.pcori.org/funding-opportunities/past-funding-opportunities/methods-review/; accessed March 11, 2012).” We provided the consensus panel with descriptive information for each potential standard, including the research phase and study design the potential standard is applicable to, and the number and title of guidance documents that described the potential standard. We provided space for comments, questions, and nomination of items not included in the list.

In Round One, we asked panel members to rate whether each potential standard was applicable to one or more study designs, and if applicable, to rate each potential standard as either (a) “mandatory”(i.e., a project must adhere to the standard); (b) “highly desirable”(i.e., a project should generally adhere to the standard, but that there are justifiable exceptions); or (c) “other - not mandatory or highly desirable” (i.e., it is unimportant whether a project adhere to the standard). The consensus survey was designed, administered, and analyzed using Survey Monkey. Details on how scoring was achieved over the two rounds can be found in in the PCORI Draft Methodology Report (http://www.pcori.org/assets/Minimal-Standards-in-the-Prevention-and-Handling-of-Missing-Data-in-Observational-and-Experimental-Patient-Centered-Outcomes-Research.pdf; accessed on March 29, 2013). After conclusion of two rounds of the consensus survey, we conducted a structured discussion via teleconference with the consensus panel in which results and comments from both rounds were discussed and the final set of mandatory standards agreed.

Results

Systematic review of guidance documents on missing data

We identified 1,790 records: 787 from NLM Bookshelf, 969 from NLM Catalog, 13 from searching regulatory agencies’ websites, and 21 from hand searching the reference lists of relevant documents. We did not identify any additional reports through contacting the 47 experts. After the initial screening of titles and abstracts, we retrieved and reviewed 190 full text documents, and classified 30 as meeting our eligibility criteria [1–3, 7–33] (Figure1).

Figure. Summary of guidance documents search and selection.

NLM: National library of medicine

The characteristics of the 30 included guidance documents are described in Table 1 and Appendix Table 3. In brief, they were published between 1996 and 2011 (5 were in draft form), with more than half published after 2008. We found more guidance documents written for RCTs (n=12) than for any other study design. Almost one third (9/30, 30%) were prepared by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), followed by 5/30 (17%) prepared by the US Food and Drug and Administration (FDA), and 4/30 (13%) by the Expert Working Group (Efficacy) of the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH).

Minimum standards for the prevention and handling of missing data

We extracted 39 potential standards from the 30 guidance documents (Table 2). The two-round consensus process and discussion yielded 10 mandatory standards: 3 on study design, 2 on conduct, 3 on analysis, and 2 on reporting (Box 1). Detailed explanation, elaboration, rationale, examples, and empirical and theoretical support for each standard can be found in Appendix Box 2.

Table 2.

Thirty-nine potential standards abstracted from 30 included guidance documents on the prevention and handling of missing data

Potential standards	Minimum standards?
1. The study protocol should explicitly define (i) the objective(s) of the study; (ii) the intervention or interventions of interest; (iii) the associated primary outcome(s) that quantify the impact of interventions for a defined period of time; (iv) how, when, and on whom the outcome(s) will be measured; (v) potential confounders if relevant, and (vi) the measures of intervention effects, that is the causal estimands (“parameters”) of primary interest. The estimands should be meaningful for all study participants, and estimable with minimal assumptions.	Yes, item 1 in Box 1
2. Investigators, sponsors, and regulators should design clinical trials consistent with the goal of maximizing the number of participants who are maintained on the protocol-specified intervention until the outcome data are collected.	No
3. Trial sponsors should continue to collect information on key outcomes on participants who discontinue their protocol specified intervention in the course of the study, except in those cases for which a compelling cost-benefit analysis argues otherwise, and this information should be recorded and used in the analysis.	Yes, item 4 in Box 1
4. Data collection and information about all relevant treatments and key covariates should be recorded for all initial study participants, whether or not participants received the intervention specified in the protocol.	No
5. The trial design team should consider whether participants who discontinue the protocol intervention should have access to and be encouraged to use specific alternative treatments. Such treatments should be specified in the study protocol.	No
6. Investigators should explicitly anticipate potential problems of missing data. The study protocol should contain a section that addresses missing data issues and steps taken in study design and conduct to monitor and limit the impact of missing data. As relevant, the protocol should include the anticipated amount of and reasons for missing data, and plans to follow up participants.	Yes, item 2 in Box 1
7. All trial protocols should recognize the importance of minimizing the amount of missing data, and, in particular, they should set a minimum rate of completeness for the primary outcome(s), based on what has been achievable in similar past trials.	No
8. Informed consent documents should emphasize the importance of collecting outcome data from individuals who choose to discontinue treatment during the study, and they should encourage participants to provide this information whether or not they complete the anticipated course of study treatment. When a subject withdraws from a study, the data collected on the subject to the point of withdrawal remains part of the study database and may not be removed.	No
9. Attrition should be considered with regard to both patients and study sites, as results may be biased or less generalizable if only some sites (e.g., teaching hospitals) remain in the study while others discontinue participation.	No
10. When deciding on data definitions and data coding on data collection tools, consideration should be given to accounting for missing or unknown data. Missing values should be distinguishable from the value zero or characteristic absent. For example, including "unknown" as an option will allow the person completing the form to provide a response to each question rather than leaving it blank.	No
11. When planning data elements for registries during the planning phase, pilot testing will help to identify the missing data rate.	No
12. For studies that include a data and safety monitoring board, the board should review plans for and the implementation of the prevention and handling of missing data. The board should review completeness and timeliness of data and recommend modifications as appropriate.	Yes, item 5 in Box 1
13. The crossover design should generally be restricted to situations where losses of subjects from the trial are expected to be small.	No
14. The equivalence (or noninferiority) trial is not conservative in nature, so that many flaws in the design or conduct of the trial will tend to bias the results towards a conclusion of equivalence. For these reasons, it is especially important to minimize the incidence of losses to follow-up and missing data, and also to minimize their impact on the subsequent analyses.	No
15. Sample size estimation should reflect and (when appropriate) account for the loss of power from missing data.	No
16. If protocol changes occur and they affect handling of missing data, the modifications to the planned analysis should be documented in a protocol amendment. Changes should be made without breaking the masking.	No
17. All participants who enter the study should be accounted for in reporting the results, whether or not they are included in the analysis. Describe and justify any planned reasons for excluding participants from analysis.	Yes, item 9 in Box 1
18. Statistical methods for handling missing data should be pre-specified in the study protocol, and their associated assumptions stated in a way that can be understood by all stakeholders.	Yes, item 3 in Box 1
19. When abstracting data from medical chart or using data from electronic medical records, use of abstraction and strict coding standards (including handling of missing data) and use of data transfer (transfer data electronically from one source to another, e.g., from a regional registry to a national registry) increased the quality and interpretation of data abstracted. When using data from electronic medical records, it is crucial to employ and describe methods to ensure complete observation and capture of patient care over time to facilitate the likelihood that all exposures and safety outcomes of interest will be captured.	No
20. It is important to consider approaches that will distinguish patients who are lost to followup from those who have missing data for other reasons (such as a patient who missed a visit but is still in the study). In claims data, a missing element may indicate that a test was not done; yet depending on the nature of the data source, it may also reflect that those tests are not covered under by the health insurance provider that provided the data for the study.	Yes, item 3 in Box 1
21. Trained study personnel should run data queries, generate reports to identify missing data issues, and provide clinical site(s) with immediate feedback. Additional site(s) training may be required.	No
22. Single imputation methods, such as last observation carried forward and baseline observation carried forward, generally should not be used as the primary approach for handling missing data in the analysis.	Yes, item 7 in Box 1
23. Analysis that includes only those participants with no missing data (complete case analysis) cannot be recommended as the primary analysis in a study.	No
24. Parametric models in general, and random effects models in particular, should be used with caution, with all their assumptions clearly spelled out and justified. Models relying on parametric assumptions should be accompanied by goodness-of-fit procedures.	No
25. Statistical inference of intervention effects or measures of association should account for statistical uncertainty attributable to missing data. This means that under the stated missing data assumptions of the methods used for imputing missing data, the associated significance tests should have valid type I error rates and that confidence intervals should have the nominal coverage properties.	Yes, item 6 in Box 1
26. Weighted generalized estimating equations methods should be more widely used in settings when missing at random can be well justified and a stable weight model can be determined, as a possibly useful alternative to parametric modeling.	No
27. When substantial missing data are anticipated, auxiliary information should be collected that is believed to be associated with reasons for missing values and with the outcomes of interest. This could improve the primary analysis through use of a more appropriate missing at random model or help to carry out sensitivity analyses to assess the impact of missing data on estimates of treatment differences. In addition, investigators should seriously consider following up all or a random sample of trial dropouts, who have not withdrawn consent, to ask them to indicate why they dropped out of the study, and, if they are willing, to collect outcome measurements from them.	No
28. When substantial missing data are anticipated, auxiliary information should be collected that is Discrete Event Simulation (DES) models facilitate complex model structures, and hence, they often require extensive data to populate them. In the absence of empirical data for some parameters, one option is to eliminate the sections of the model that require the parameters for which information is missing. This means restructuring the perspective of the model to the appropriate level of detail, for example, higher level models may provide insight into the problem and still have available valid data that a more micro model may not. Alternatively, the original model structure may be maintained by deriving the missing parameter values via calibration, by identifying sets of input parameter values that produce output values that are similar to target values (i.e. observed estimates of the output parameters).	No
29. Data completeness and methods for handling missing data should be considered when analyzing and interpreting study data. Before analyzing the study data, the database should be "cleaned" and attempts should be made to obtain as much missing data as realistically possible from source documents. How missing values are handled should be explained following the data management guidelines. The degree of data completeness should be summarized.	No
30. A framework for the applicability of the different methods to handle missingness is based on a classification according to the following missingness mechanisms: Missing Completely At Random (MCAR), Missing At Random (MAR), and Missing Not At Random (MNAR). To gain insight into which of the three categories of missing data (MCAR, MAR, MNAR) are in play, one can compare the distribution of observed variables for patients with specific missing data to the distribution of those variables for patients for whom those same data are present.	No
31. Examining sensitivity to the assumptions about the missing data mechanism (i.e., sensitivity analysis) should be a mandatory component of the study protocol, analysis, and reporting.	Yes, item 8 in Box 1
32. Report on data completeness and how missing data were handled in the analysis to facilitate interpretation of study results. The potential influence of missing data on the study results should be described.	Yes, item 10 in Box 1
33. When using electronic trial data handling and/or remote electronic trial data systems, the sponsor should ensure that the systems are designed to permit data changes in such away that the data changes are documented and that there is no deletion of entered data (i.e., maintain an audit trail, data trail, and edit trail).	No
34. The reported trial data should be accurate, complete, and verifiable from source documents (e.g., case report forms). Checking the accuracy and completeness of the Case Report Form (CRF) entries, source data/documents, and other trial-related records against each other. Visits that the subjects fail to make, tests that are not conducted, and examinations that are not performed are clearly reported as such on the CRFs. All withdrawals and dropouts of enrolled subjects from the trial are reported and explained on the CRFs.	No
35. Whenever a participant discontinues some or all types of participation in a research study, the investigator should document the following: (i) the reason for discontinuation; (ii) who decided that the participant would discontinue; (iii) whether the discontinuation involves some or all types of participation. Investigators should continue to collect information on key outcomes for participants who discontinue the protocol-specified intervention.	Yes, item 4 in Box 1
36. The regulatory agencies, funders (e.g., NIH), drug, device, and biologic companies that sponsor clinical trials, academic investigators, researchers should carry out continued training of their analysts to keep abreast of up-to-date techniques for missing data analysis. Regulatory agencies should also encourage continued training of their clinical reviewers to make them broadly familiar with missing data terminology and missing data methods.	No
37. For analysis of registry data, consideration should be given to implementation of routine followup of all registry patients for key adverse events to ensure that analyses of the occurrence of adverse events among the registry population are not hampered by extensive missing data. During safety review of a new drug application, detailed comments on the quality and completeness of the data should be provided. The reviewer should be concerned about patients with abnormal clinical or laboratory findings who are lost to follow up, particularly if there are significant numbers of such patients. In these situations, the reviewer may consider asking the applicant to attempt to obtain the needed follow up information. A separate file may be needed for any missing data.	No
38. Applicants sometimes segregate certain clinical trials from their primary source data especially foreign data. This may be appropriate, especially if there is a basis for believing that these data differ substantially in quality and/or completeness or in critical aspects of investigator practice from the data included in the primary source database. This is a matter of judgment, however, and cannot be assumed to be valid. An explanation should be provided in the review describing the basis for decisions about what data were included and what excluded from the primary source data.	No
39. Users of electronic medical records should ensure that they are aware of the quality assurance and quality control procedures used by the data holders and how the procedures could affect data integrity and the study. FDA recommends that investigators address the general procedures used by the data holders to ensure completeness.	No

Box 1. Minimum standards in the prevention and handling of missing data in patient-centered outcomes research.

Standards on study design

1. Define research question, in particular, the outcome(s)

The study protocol should explicitly define (i) the objective(s) of the study; (ii) the intervention or interventions of interest; (iii) the associated primary outcome(s) that quantify the impact of interventions for a defined period of time; (iv) how, when, and on whom the outcome(s) will be measured; (v) potential confounders if relevant, and (vi) the measures of intervention effects, i.e., the parameters (“estimands”) that capture the causal effect of the intervention of primary interest. The parameters should be meaningful for all study participants, and estimable with minimal assumptions. This standard applies to all study designs that aim to assess intervention effectiveness.

Defining outcome(s) precisely and accurately requires careful attention because the choice of outcome may have important implications for study design, implementation, expected amount of and reason for missing data, as well as methods for handling missing data. For example, the outcome could be defined in the population that includes all participants randomized to the study intervention(s), regardless of the intervention participants actually received (i.e., intention-to-treat estimand (1)); or the outcome could be defined in a more restricted population that includes only those who can tolerate the intervention for a given period. Outcome(s) could be measured after a short follow-up time period or assessed after a long follow-up period, measured at one point in time or measured repeatedly over time. At a minimum, the primary outcome must be decided and adequately described in the study protocol. Imprecise and vague definition may lead to a lack of clarity in how to prevent and handle missing data.

2. Take steps in design and conduct to minimize missing data

Investigators should explicitly anticipate potential problems of missing data. The study protocol should contain a section that addresses missing data issues and steps taken in study design and conduct to monitor and limit the impact of missing data. As relevant, the protocol should include the anticipated amount of and reasons for missing data, and plans to follow up participants. This standard applies to all study designs for any type of research question.

3. Pre-specify statistical methods for handling missing data

Statistical methods for handling missing data should be pre-specified in the study protocol, and their associated assumptions stated in a way that can be understood by all stakeholders. The reasons for missing data should be considered in the analysis. This standard applies to all study designs for any type of research question.

Standards on study conduct

4. Continue collecting information on key outcomes

Whenever a participant discontinues some or all types of participation in a research study, the investigator should document the following: (i) the reason for discontinuation; (ii) who decided that the participant would discontinue; (iii) whether the discontinuation involves some or all types of participation. Investigators should continue to collect information on key outcomes for participants who discontinue the protocol-specified intervention. This standard applies to prospective study designs that aim to assess intervention effectiveness.

5. Monitor missing data

For studies that include a data and safety monitoring board, the board should review plans for and the implementation of the prevention and handling of missing data. The board should review completeness and timeliness of data and recommend modifications as appropriate.

Standards on analysis

There is no universal statistical method to cope with missing data because each study has its own design and measurement characteristics, and different assumptions about missing data mechanism [1]. Standards 6–8 cover the basic principles that can be applied in a wide range of settings. Interested readers should refer to the statistical literature or The Prevention and Treatment of Missing Data in Clinical Trials for guidance on commonly used methods for analysis (e.g., multiple imputation, inverse probability weighting, likelihood-based methods, and Bayesian approaches). Understanding the advantages and limitations of each method and its underlying assumptions is key for appropriate application in practice, as is working with an experienced statistician.

6. Account for uncertainty in handling missing data in the analysis

Statistical inference of intervention effects or measures of association should account for statistical uncertainty attributable to missing data. This means that under the stated missing data assumptions of the methods used for imputing missing data, the associated significance tests should have valid type I error rates and that confidence intervals should have the nominal coverage properties. This standard applies to all study designs for any type of research question.

7. Discourage single imputation methods

Single imputation methods, such as last observation carried forward and baseline observation carried forward, generally should not be used as the primary approach for handling missing data in the analysis. This standard applies to all study designs for any type of research question.

8. Conduct sensitivity analysis

Examining sensitivity to the assumptions about the missing data mechanism (i.e., sensitivity analysis) should be a mandatory component of the study protocol, analysis, and reporting. This standard applies to all study designs for any type of research question.

Standards on reporting

9. Account for all participants entered in the study in reporting the results

All participants who enter the study should be accounted for in reporting the results, whether or not they are included in the analysis. Describe and justify any planned reasons for excluding participants from analysis. This standard applies to all study designs for any type of research question.

10. Report on data completeness and strategies applied to handle missing data

Report on data completeness and how missing data were handled in the analysis to facilitate interpretation of study results. The potential influence of missing data on the study results should be described. This standard applies to all study designs for any type of research question.

Gaps in guidance identified

We found that not only did more guidance documents apply to RCTs than other designs, the recommendations in the documents were more detailed for RCTs. Standards for analyzing existing records such as claims data and electronic health records, and specifically how to prevent and handle missing data with specific reference to patient-centered outcomes, have not been fully addressed. For example, the Prevention and Treatment of Missing Data in Clinical Trials (1) report, prepared at the request of FDA, focuses primarily on missing data in Phase III confirmatory clinical trials that serve as the basis for the approval of drugs and devices. In another example, ICH, which represents the consensus views of the regulatory authorities and pharmaceutical industry of Europe, Japan, and the US (http://www.ich.org/; accessed on September 11, 2012), describes missing data-related issues in sections within ICH’s good practice documents. In contrast, ISPOR, an international organization with members from 100 countries has manuscript-length reports that provide recommendations for a wide range of observational study designs and topics, but it has no documents devoted to missing data exclusively (http://www.ispor.org/about-ispor.asp; accessed on September 11, 2012). We did not find any formal guidance documents written exclusively about missing data for observational study designs even after contacting 47 experts in the field.

Discussion

We extracted 39 recommendations on the prevention and handling of missing data from 30 guidance documents identified. Using a Delphi consensus approach, we proposed 10 standards as indispensible for the prevention and handling of missing data in research studies, covering the following broad areas. First, we recommend that the single best approach is to prospectively prevent missing data occurrence, through carefully designing and implementing of a research study [1, 34–36]. Prevention requires defining in advance the primary objective, outcome, and population effect of interest, as well as specifying procedures in the protocol that aimed at minimizing any irregularities in study conduct. This recommendation represents a major challenge for secondary analysis of administrative data sets and electronic health records where the main purpose of the record is not research.

Second, because many large-scale studies that are used to inform clinical practice have at least some missing data, use of valid statistical methodology for analysis and decision making is also of utmost importance. Drawing inferences from incomplete data relies on assumptions that can never be tested or verified from the observed data [37]. For this reason, sensitivity analyses for assessing the potential effects of missing data on a study’s conclusions should be pre-specified in the study protocol and carefully followed in the analysis [1–3, 7–9, 14, 22–25, 38–41]. In addition, and of particular note, single imputation approaches such as last or baseline observation carried forward are strongly discouraged [1, 2, 11].

Third, transparent reporting of the amount of and reasons for missing data, and the assumptions underlying analyses, allows users of study findings to judge the validity of the work, which in turn contributes to improving the reproducibility of research and reliability of the findings [2, 3, 8, 11–14,16, 19, 21, 24, 26–30].

We found that most guidance on missing data was prepared specifically for RCTs, one of several possible study designs that can be used for testing the effectiveness and safety of interventions in PCOR. This is no surprise, as RCTs are required by regulatory authorities worldwide for approval of drugs and biologics for specific indications. Observational designs are of increasing importance to PCOR in part because RCTs are typically run on patients with fewer co-morbidities and they provide estimates of effectiveness of interventions. Electronic health records, for example, provide information about health outcomes in patients with multiple chronic conditions on a variety of medications. These patients may not be eligible for RCTs, yet understanding how interventions work in these patients in a real-world setting is vitally important, especially in conditions where the majority of patients have comorbidities. As analysis of Medicare and Medicaid datasets is a longstanding approach to health services and outcomes research, it is somewhat surprising that standards for analyzing claims data and electronic health records, and specifically how to handle missing data, have not been more fully addressed for these resources. Even in the case of traditional epidemiological studies such as cohort and case-control studies in which investigators are in control of data collection, few standards exist. This finding is striking to us because the threat to validity from missing data is potentially greater for observational studies than RCTs: in observational studies, data can be missing for exposures, known confounders, and outcomes compared with the situation for well-conducted RCTs where data are likely to be missing primarily for outcomes alone.

Although almost all of our recommended standards are also applicable to observational studies, we believe an in-depth and thorough discussion of challenges facing observational studies, and especially studies that utilize existing records and databases, is urgently needed. Some unique challenges include missing or incomplete confounder measures which could distort the inference; difficulty interpreting missing data in secondary data analyses (e.g., does absence of a specific symptom in patient registers and electronic health records indicate the symptom was not present or that the physician did not actively inquire about or document this symptom?); data not available for some important confounders (e.g., smoking, over-the-counter medications, body mass index); losses to follow up (e.g., due to turnover in healthcare plans); and nonstandard follow-up time points based on need for care rather than a research protocol [11, 30, 34]. Detailed strategies for using data from resources not originally established for research purposes need to be worked out so that the rich information provided in such data are meaningfully gathered, analyzed, and interpreted.

A major strength of our approach is that we conducted a comprehensive search for existing guidance documents on missing data, which was combined with a subsequent multidisciplinary expert consensus, to recommend mandatory standards. Our review highlights gaps in the literature, and that guidance on the prevention and handling of missing data for observational studies and studies that utilize existing records are needed urgently. A limitation of our approach is that guidance documents published in manuscript format were omitted due to the scope of our contracted work. To counteract this limitation we searched websites and queried experts, which yielded no additional guidance documents for inclusion. Also, consensus panel was naturally limited to what was out there in recommending the minimal standards. With the rapid development in research on missing data with specific reference to PCOR and patient-reported outcomes, we urge researchers to distill the information and provide them in a guidance format for dissemination and implementation.

The problems associated with missing data deserve our focused attention. In particular, some well-described approaches in the large and growing missing data literature [34–48] are not being implemented broadly and it is not clear that investigators and evidence users are aware of the problems missing data introduce or how best to handle it [49–50]. For example, discontinuation of a study intervention does not require discontinuation of follow-up for relevant clinical outcomes. We urge PCOR researchers to adopt rigorous methodology and promote good science. After all, trustworthy research evidence and valid inference depend on application of best practices to the prevention and handling of missing data in the primary studies; and the flaws in the primary studies will only be amplified in the systematic reviews and clinical practice guidelines [51, 52]. The standards recommended herein, and by PCORI, provide a starting point.

What is new?

• We performed a systematic review and identified 30 guidance documents and extracted 39 recommendations on the prevention and handling of missing data. Using a Delphi consensus approach, we proposed 10 standards as indispensible for the prevention and handling of missing data in research studies.

• We found that not only did more guidance documents apply to randomized controlled trials (RCTs) than other designs, the recommendations in the documents were more detailed for RCTs. Standards for analyzing existing records such as claims data and electronic health records, and specifically how to prevent and handle missing data with reference to patient-centered outcomes, have not been fully addressed in the documents we retrieved. We did not find any formal guidance documents written exclusively about missing data for observational study designs.

• We urge researchers to adopt rigorous methodology and promote good science by applying best practices to the prevention and handling of missing data. Developing guidance on the prevention and handling of missing data for observational studies and studies that utilize existing records is a priority for future research.