Guidelines for reporting meta-epidemiological methodology research

Abstract

Published research should be reported to evidence users with clarity and transparency that facilitate optimal appraisal and use of evidence and allow replication by other researchers. Guidelines for such reporting are available for several types of studies but not for meta-epidemiological methodology studies. Meta-epidemiological studies adopt a systematic review or meta-analysis approach to examine the impact of certain characteristics of clinical studies on the observed effect and provide empirical evidence for hypothesised associations. The unit of analysis in meta-epidemiological studies is a study, not a patient. The outcomes of meta-epidemiological studies are usually not clinical outcomes. In this guideline, we adapt items from the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) to fit the context of meta-epidemiological studies.

Published research should be reported to evidence users with clarity and transparency to facilitate optimal appraisal and use of evidence and allow replication by other researchers. However, for a variety of reasons, the literature is commonly viewed as poorly reported with critical information missing from published reports. This necessitated the development of reporting guidelines. Currently, there are guidelines for reporting randomised trials (CONSORT), observational studies (STROBE), systematic reviews (Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA)), case reports (CARE), qualitative research (SRQR), diagnostic and prognostic studies (STARD), quality improvement studies (SQUIRE), economic evaluations (CHEERS), animal preclinical studies (ARRIVE), study protocols (SPIRIT) and clinical practice guidelines (AGREE). These guidelines are catalogued on the website of the EQUATOR Network (Enhancing the QUAlity and Transparency Of health Research).¹ The EQUATOR Network is an international initiative that seeks to improve the reliability and value of health research literature by promoting transparent and accurate reporting and wider use of robust reporting guidelines. Some of these reporting guidelines have extensions for subtypes of research (eg, PRISMA-Equity is used for equity-focused systematic reviews). Some function as a tool for reporting and a tool for assessing the methodological quality (eg, AGREE-II is used to evaluate the rigour and the reporting of clinical practice guidelines).

Meta-epidemiological studies are a fairly new type of study that has been increasingly published in the last two decades,^2–4 with no clear reporting standards. Sometimes called methodology research, meta-epidemiological research adopts a systematic review or meta-analysis to provide data for methodological analysis.⁵ The goal is generally broad but often focuses on examining the impact of certain characteristics of clinical studies on the observed effect, describing the distribution of research evidence in a specific setting, examining heterogeneity and exploring its causes, identifying and describing plausible biases and providing empirical evidence for hypothesised associations. Unlike classic epidemiology, the unit of analysis for meta-epidemiological studies is a study, not a patient. The outcomes of meta-epidemiological studies are usually not clinical outcomes.^6–8

In this guide, we adapt the items used in the PRISMA statement⁹ for reporting systematic reviews and meta-analysis to fit the setting of meta- epidemiological studies. The rationale for starting with PRISMA is that meta- epidemiological research requires a systematic approach (ie, a systematic review) to select data (the input for methodological analysis). The overarching goal of this guideline is to improve the transparency and clarity of healthcare research.

Examples of meta-epidemiological studies

Analysis of Cochrane systematic reviews showed that trials with inadequate or unclear allocation concealment yielded larger estimates of effect relative to controlled trials with adequate allocation concealment.¹⁰ In a second meta-epidemiological study, non-blinded trials of complementary medicine exaggerated the effect size by an average of 0.56 SD (95% CI –0.71 to –0.41) compared with blinded trials.¹¹ A third example is a study that explored the effect of trials stopped early for benefit on future research.¹² In a fourth example, it was shown that in almost 30% of endocrinology studies, the largest treatment effect was observed in the first or second published studies (ie, earlier studies, when studies were ordered chronologically using the date of publication).¹³

Proposed reporting guidelines

We have adapted items of the PRISMA statement to fit a meta-epidemiological study. The items are categorised according to the title, abstract, introduction, methods, results, conclusion and funding and are supported by rationale and examples.

Title and abstract

Both the title and abstract should identify the report as a meta-epidemiological study, for example, ‘Early studies reported extreme findings with large variability: a meta-epidemiologic study in the field of endocrinology’.¹³ The rationale for this identification is to improve indexing and facilitate retrieval. It also helps clinicians searching for systematic reviews about clinical topics to quickly find relevant evidence and avoid reviewing additional non-clinical meta-epidemiological studies. The abstract should also include a structured summary with subheadings that provide the context, hypothesis, methods, data sources, results, limitations and conclusions of meta-epidemiological studies. Variations of these subheadings are acceptable. The rationale for having this information in the abstract relates to the fact that many readers only read the abstract. These requirements are similar to PRISMA.

Introduction

The introduction should include the rationale and objectives of the study. This requirement is similar to PRISMA. However, it is more important to explicitly state the hypothesis being tested in meta-epidemiological studies compared with a systematic review summarising evidence about the effectiveness of an intervention (which may or may not have a clear hypothesis and primarily aim at empirically evaluating a clinical outcome).

Methods

The methods section should specify if the meta-epidemiological study follows a priori established protocol and clarify the eligibility criteria, information sources, search methods, study selection and data collection process, data items collected, summary measures and synthesis methods (including assumptions and computational approximations and methods of additional analyses such as sensitivity, subgroup or meta-regression analyses). These requirements are similar to PRISMA. However, registration of protocols may not be needed, particularly since systematic reviews registration sites are geared more towards clinical questions. Search strategy in a meta-epidemiological study is different because it is not driven by a clinical question. Evaluating and reporting the risk of bias in individual studies may or may not be relevant to the question at hand. The type of bias can also be different. Data extraction is often more complex in meta-epidemiological studies and should be described in details sufficient to allow replication. The methods section of a meta-epidemiological study should describe in details the effect measure used in statistical analysis because it is likely that a clinical audience will not be familiar with such measure (eg, ratio of OR, standardised measures and regression coefficients). Such measures are not commonly used in studies that report on treatment effects (ie, clinical audience would be familiar with OR but not familiar with ratio of OR). Quantifying interagreement between investigators is more critical in a meta-epidemiological study than in a clinical systematic review because of the judgements commonly made to classify data in a meta-epidemiological study. For example, a study evaluated the association between industry affiliation and researchers’ position on the cardiovascular risk of rosiglitazone. Reviewers had to assess how favourable was the researchers’ opinion about rosiglitazone using an ordinal scale.¹⁴ This task is fairly subjective and requires judgement. In this case, agreement between reviewers is critical to report and affects the reliability of this meta-epidemiological study (arguably more critical than in a systematic review of clinical outcomes).

Results

The results section should include the results of the study selection process (how many studies, how many patients, etc). Different from PRISMA, study and patient characteristics need to be tailored to the meta-epidemiological question at hand. Therefore, it may not be important to know patients' age, comorbidities and settings in some questions (although such characteristics may be important in other questions). For example, the meta-epidemiological study showing that trials with inadequate allocation concealment yielded larger estimates of effects (relative to trials with adequate allocation concealment) does not necessarily have to describe patients’ characteristics or comorbidities in each individual trial.¹⁰ Publication bias should be evaluated and reported similar to PRISMA recommendations.

Discussion and conclusions

As with PRISMA, the discussion section should include summary of the main findings and comparison with existing knowledge and should also include the limitations and strengths of the meta-epidemiological study. Limitations at a study level may not be important (as in clinical systematic reviews); however, limitations at the level of the meta-epidemiological study are important. The quality of evidence is a highly important step in a systematic review reporting on a treatment effect¹⁵ and is critical for decision making but also may not be relevant to a meta-epidemiological study. Investigators should, however, inform readers of their certainty in the results, even if this was determined narratively¹⁶ because formal approaches¹⁵ are intended to rate the certainty of evidence about interventions. Lastly, systematic reviews of clinical questions should include in their discussion the implication of their findings on future research and current clinical practice. Meta-epidemiological studies will have profound implications for future research and should report such implications. Meta-epidemiological studies will likely have minimal direct impact on clinical practice. Nevertheless, indirect impact on clinical practice is possible and should be reported when present. For example, a meta-epidemiological study showed that in endocrinology, the largest treatment effect was observed in the first or second published studies.¹³ Clinically, this should make endocrinologists sceptical about new drugs, laboratory tests or genetic markers and not rush into recommending them to their patients until more studies confirm findings observed in these earlier studies.¹³

In table 1, we present PRISMA items with adaptations to fit the setting of meta-epidemiological studies.

Table 1.

Proposed items to be used for reporting methodology research, adapted from the PRISMA Checklist (http://prisma-statement.org/PRISMAStatement/Checklist.aspx)

Section/topic	Proposed item to be used in methodology research
Title
Title	Identify the report as a meta-epidemiologic study.
Abstract
Structured summary	Provide a structured summary that includes the background of the topic, goal of the study, data sources, method of data selection, appraisal and synthesis methods, results, limitations, conclusions and implications of key findings.
Introduction
Rationale	Describe the rationale for the meta-epidemiological study in the context of what is already known.
Objectives	Provide an explicit statement of the goal of the meta-epidemiological study and the hypothesis being empirically tested.
Methods
Protocol	Indicate if a protocol exists, if and where it can be accessed (eg, Web address). Registration of a protocol is not mandatory.
Eligibility criteria	Specify study characteristics used as criteria for eligibility with a rationale.
Information sources	Describe all information sources (eg, databases with dates of coverage, contact with experts to identify additional studies, Internet searches) and search date.
Search	Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated. Search is commonly not driven by a clinical question.
Study selection	Describe the process for selecting studies for inclusion (ie, how many reviewers selected studies, reviewing in duplicate or by single individuals).
Data collection process	Describe method of data extraction from reports (eg, piloted forms, independently, in duplicate) and any processes used for manipulating data or obtaining and confirming data from investigators.
Data items	List and define all variables for which data were sought and any assumptions and imputations made.
Risk of bias in individual studies	If risk of bias assessment of individual studies was relevant to the analysis, describe the items used and how this information is to be used during data synthesis.
Summary measures	State the principal summary measures (eg, ratio of risk ratios, difference in means) and explain its meaning and direction to readers.
Synthesis of results	Describe the statistical or descriptive methods of synthesis including measures of consistency if relevant. If applicable, describe the development of statistical or simulation modelling based on theoretical background. Describe and justify assumptions and computational approximations. Describe methods of additional analyses (eg, sensitivity or subgroup analyses, meta-regression), if done, indicating which were prespecified.
Results
Study selection	Give numbers of studies assessed for eligibility and included in the study, with reasons for exclusions at each stage, ideally with a flow diagram. Present a measure of inter-reviewer agreement (eg, kappa statistic).
Study characteristics	For each study, present characteristics for which data were extracted and provide the citations. Clinical characteristics may not always be relevant.
Risk of bias within studies	If risk of bias assessment of individual studies was used in the meta-epidemiological analysis, report risk of bias indicators of each study to allow replication of findings.
Results of individual studies	Present data elements used in the meta-epidemiological analysis from each study (results of clinical outcomes may not be relevant).
Synthesis of results	Present results of statistical analysis done, including measures of precision and measures of consistency. Present validity of assumptions and fit of statistical or simulation modelling, if applicable.
Additional analysis	Give results of additional analyses, if done (eg, sensitivity or subgroup analyses, meta-regression).
Discussion
Summary of evidence	Summarise the main findings and compare them with existing knowledge about the topic. The quality of evidence may not be relevant; however, investigators should describe their certainty in the results to readers.
Limitations	Discuss limitations at research methodology level (eg, likelihood of reporting or publication bias).
Conclusions	Provide general interpretation of the results and implications for future research. Provide any plausible impact on clinical practice.
Funding
Funding	Describe sources of funding for the methodology research and role of funders.

Discussion and limitations

Reporting guidelines have three goals: to enhance the applicability and use of research results, to allow readers to critically appraise the results and to permit reproducibility and replication of the work, which is essential for accountability and transparency purposes. Meta-epidemiological studies should have similar standards and require reporting guidelines that achieve these three goals.

In this guideline, we propose adaptation to PRISMA items to fit meta-epidemiological studies. Meta-epidemiological studies are inherently heterogeneous in terms of their approach and execution, which is a challenge for current reporting guidelines of systematic reviews (such as PRISMA). For example, a meta-epidemiological study used trial registries as a source of its data.¹⁷ Registries are not searched using standard search terms and methods and may not fit the reporting requirement needed for systematic reviews. Entries in trial registries are also dynamic, just like Internet searches. In this case, authors of such study should make every possible effort to describe their work in a reproducible fashion but acknowledge these limitations.¹⁷ In addition, reporting guidelines are frequently underused and are not always emphasised by journal editors. A descriptive analysis of published randomised trials in the four highest impact general medical journals shows inconsistencies and non-adherence to the CONSORT statement (eg, allocation concealment and details of blinding were reported only in 12% and 21%, respectively).¹⁸ Adherence across all reporting guidelines was estimated to be 86% in another study.¹⁹ Evaluation of the instructions given to authors shows that PRISMA guidance was referenced only in 13% of gastroenterology and hepatology journals.²⁰

Here, we have proposed a reporting guideline to be used when publishing meta-epidemiological research. We hope that these guidelines will be used, modified and enhanced and lead to more clarity and transparency of healthcare research.