Real-world evidence for neovascular age-related macular degeneration: a practical approach

Evidence based medicine is the integration of the best evidence to guide management decisions for patients. Randomised control trial (RCT) data provide gold-standard evidence on treatment efficacy and form the basis for the widespread usage of vascular endothelial growth factor (VEGF) inhibitors in treating neovascular age-related macular degeneration. However, VEGF inhibitor therapy outcomes in real-world practice are variable, often reporting poorer results compared to RCTs [1]. Thus the difference between RCT and real-world evidence (RWE) data reflect different but complementary aspects of therapeutic outcomes. In contrast to RCTs which aim to evaluate efficacy by controlling as many variables as possible, RWE measures overall effectiveness, which incorporates the multitude of variables in clinical practice [2]. In contrast to RCTs, which have well-established design analysis and reporting structure, the rigour of RWE has often been questioned, due to inherent challenges in study design due to the lack of randomisation, variability in the quality of data and non-uniform analytical methods. In this editorial, several principles are recommended for the analysis and reporting of RWE. These are summarised in Table 1.

Table 1.

Strategies for the analysis and interpretation of real-world data sets.

Stage	Steps	Details/method
Pre analysis	Formulation of clinical question and hypothesis	This should include comparators, time frame, endpoint, patient subset and clinical relevance
	Primary outcome selection	Develop a statistical analysis plan (SAP) according to primary outcome Choose appropriate statistical method which are dependent on nature of design (comparator, cohort, or survival analysis) and outcome of interest (continuous or binary)
	Population and dataset selection	Set inclusion exclusion criteria based on SAP Ensure sample size can meaningfully correct for covert bias Comparator groups may require balancing with methods such as matching or the use of propensity scores
Analysis	Address heterogeneity of baseline characteristics	Determine covariates that may confound primary outcome These covariates which contribute to overt bias can be corrected with multivariate analysis
	Address heterogeneity of treatment exposure or variable of interest	Associations can be corrected statistically. For example, mixed effects longitudinal generalised additive model with presentation of adjusted means can be used for longitudinal data Ascertain linear or non-linear relationship based on distribution of results using either logistic regression analysis (in a linear relationship) or LOESS regression analysis (in non-linear relationship)
	Address selection bias related to non completers	A few strategies include last observation carried forwards analysis, Kaplan–Meier statistics to correct for dropouts in a time-to-event analysis and mixed effects longitudinal generalised additive model to utilise all data available. Suggest using both to present thorough data. A sensitivity analysis for completers versus non completers or other treatment groups that may confound the primary analysis should be performed
Interpretation of results	Present all available data in addition to aggregated results	Results can be graphically represented to illustrate association using either linear regression or non-linear regression methods Scatter plots can provide more information in addition to summarised data (means). These plots allow for individual variations, clustering effects and longitudinal change to be appreciated
	Present findings and conclusion	Discussion should include a summary of key findings, generalisability to populations beyond the study dataset, and clinical utility Limitations and weaknesses of analysis should be acknowledged and methods to mitigate these should be highlighted

Principle 1: Clearly defined clinical question

A clear clinically relevant question with a specific outcome should be pre-defined. This question should be hypothesis driven comprising: (1) the comparison to be made, (2) study population, (3) clinically impactful measure and (4) the duration of the study or intervention. These should be set out clearly in the final report to ensure that the reader is aware of the main clinical question.

Principle 2: Interrogate the data source

The highly variable nature of real-world data requires that the data source be reviewed thoroughly before embarking on any analysis. The incompleteness of data is a major pitfall of RWE. Completeness should be assessed to ensure no systematic bias in under-reporting. Some strategies to address completeness include the use of a disease registry which mandates inclusion of all patients and all relevant data. This avoids selecting only patients with favourable outcomes. Participating physicians should also conform to uniform reporting standards to ensure consistency data entered [3].

Once the overall dataset has been standardised, inclusion criteria related to the clinical question in principle 1 should be applied. Applying criteria to restrict the study population may seem counterintuitive to the generalisability of RWE but this may be necessary as practice patterns may change. For example, analysing a treat and extend regimen prior to 2014 is fruitless as the first large study to describe the efficacy of this regimen was only reported in 2015 [4].

Principle 3: A pre-specified statistical analysis plan and addressing bias

A pre-specified analysis plan is important to avoid picking a method post hoc that gives a ‘desirable’ result. This plan should identify relevant independent, or predictor, variables that are available from the dataset. The plan should also aim to address biases inherent to real-world data due to the lack of randomisation.

Covert and overt biases exist in RWE. Covert biases are results of imbalances between groups due to factors that are not observed by investigators. Methods to address this type of bias have limited success [5, 6]. Disease registries, which consist of a large heterogenous data pool, can help to address this by ensuring enough variability to mitigate the effects of covert bias. Overt biases, which are apparent and identifiable differences, can be corrected through matching or statistical adjustments. Selection bias which can manifest as high dropout rates are a major concern that can invalidate RWE [7, 8]. A method of addressing selection bias from dropout is to take the last observation and carry it forwards (LOCF) to the pre-specified time point. This method ensures that patients that may have dropped out from poor outcomes are still included in the analysis. A sensitivity analysis comparing completers and non completers should be included if a LOCF strategy is used to address any confounding of the primary outcome by the non-completer group [9]. Statistical methods such as a generalised additive model (GAM) can further help to address bias from dropout. These models analyse all data accrued including those by subjects that do not fulfil the pre-specified time point.

Statistical methods in RWE require careful deliberation as comparators are not as straightforward as in RCT. Comparator groups in RWE may not always be categorical (treatment versus untreated groups in RCT), and an accepted method is the grouping of the population by quartiles (or tertiles) by the continuous variable if interest. This categorisation method is an easy way to communicate findings to the wider scientific community but require careful consideration to ensure that groupings are clinically significant [10].

Traditional hypothesis testing methods applied to RCT may not be applicable to RWE where relationships may often be non-linear [10]. Advanced non-linear techniques such as GAM or locally weighted regression (‘lowess’) may be preferred, as it allows for the use of all available data to be ‘fitted’ to establish a predictor-outcome relationship [11, 12], This allows for the visualisation of relationships and can often reveal insightful patterns that are missed if linear models are applied [13].

Principle 4: Structured interpretation of results

The interpretation of the results should be targeted toward answering the clinical question defined in principle 1. The discussion should highlight the primary outcome and contextualise this outcome for stakeholders and decision-makers. The plausibility of the comparison should be discussed as well as the magnitude of the relative benefits (or harm). Careful attention should be paid to the accurate reporting of causal inference or correlation avoiding over-generalisation. The STrengthening the Reporting of OBservational studies in Epidemiology statement provides guidance on transparent reporting of observational studies and should be applied to real-world data analysis.

Conclusion

Real-world evidence is an important form of research in informing medical and health care decisions in clinical practice. While it can help answer questions that RCT cannot, it is vital to apply thoughtful methodology to improve its scientific rigour.

Funding

KYCT is supported by grants from the National Medical Research Council, (Open Fund Large Collaborative grant no: NMRC/LCG/0042018) and MOH-TA19nov-0002.

Competing interests

The author declares no competing interests.