ISPOR Health Policy Council proposed Good Research Practices For Comparative Effectiveness Research: Benefit or Harm?

There are increasing calls for better understanding of “what works” in health care [1]. One of the means for assessing what works is through “comparative effectiveness research” (CER) [2]. Ideally, the needed data would come from randomized controlled trials (RCTs) or from natural experiments. RCTs would need to be large, practical clinical trials that compare interventions head-to-head in real clinical settings [3,4], using novel approaches to assess clinically relevant outcomes.

Non-experimental studies of intended drug effects have been criticized because confounding by indication (selective channeling of patients to treatment modalities based on outcome predictors such as severity of disease) can almost never be ruled out [5,6]. Recent developments in pharmacoepidemiologic methods limit the potential for bias and thus increase the value of non-experimental comparisons of intended drug effects. These methods include instrumental variable methods [7,8], the new user design [9], the use of a comparator drug with a similar indication to that of the index drug [10], propensity scores [11], and simple improvements such as eliminating immortal person-time [12] and reducing selection bias by not censoring follow-up when a person stops taking a drug [13,14]. Much remains to be done, however, including the study of heterogeneity of treatment effects at the intersection between personalized medicine and pharmacoepidemiology. In addition, there remains an unresolved tension between emulating RCTs (increasing internal validity based on increasing restrictions [15]) and enhancing generalizability (external validity).

Comparative effectiveness research (CER) is an interdisciplinary endeavor in which the disciplines are linked by the need for information and the development of methods. The involvement of ISPOR in this enterprise is welcome. Like drugs, the Good Research Practices proposed by the working group of the ISPOR Health Policy Council and published in this issue of the journal [16–18] need to be evaluated by their potential benefits and harms. The potential benefits are obvious. Someone unfamiliar with performing non-experimental comparisons of drugs and their outcomes will find valuable discussion in these documents of issues to be considered. Common to all such documents, however, there is the potential for harm when the recommendations are used as a cookbook without understanding their interplay. References to standard textbooks of pharmacoepidemiology [e.g., 19] could help alleviate this problem.

It is inevitable for any detailed overview to contain questionable or outmoded recommendations. Recommendations of the proposed documents that some experienced pharmacoepidemiologists might find arguable include:

• The requirement to report the results from all ex ante analyses. Some such analyses will have been abandoned because the researchers discovered that they are biased.

• The assessment of the importance of biases based on how they affect the acceptance or rejection of the null hypothesis. Biases are best measured by their effects on the magnitude, direction and precision of effect-measure estimates.

• The recommendation for propensity score models to include variables that are only weakly related to treatment selection (but unrelated to the outcome per the following recommendation). It is unclear why any variable that is unrelated to the outcome should be included in a propensity score [20].

Some core issues in the design and analysis of non-experimental comparisons are not addressed in enough detail. One is the importance of the role of various ‘stakeholders’ in CER. Another is the distinction between confounding (e.g., by indication) and selection bias (due, for instance, to non-adherence, drop-out of “sick stoppers,” etc. [13]). The potential to separate these forms of bias is one of the main advantages of the new user design [9].

Given the expected continuation of the rapid development of pharmacoepidemiologic methods over the past 5 years, the proposed Good Research Practices may become outdated very rapidly [21]. We found no indication of how ISPOR intends to keep these guidelines up to date. In an era of guideline proliferation, one might ask what the proposed ISPOR document will add to the existing ones in this field, especially the Good Pharmacoepidemiologic Practice document published and continuously updated by the International Society for Pharmacoepidemiology (ISPE) [22]. Finally, harmonization of the ISPOR documents with others, including those proposed by ISPE, the US Institute of Medicine (IOM), the US Agency for Healthcare Research and Quality (AHRQ), the UK National Institute for Health and Clinical Excellence (NICE) should be considered. Such harmonization would prevent confusion and nit picking by groups opposed to CER.

In our view, the benefit-to-harm balance of ISPOR’s proposed documents on Good Research Practices favors the benefit side. It will help to spread the news that non-experimental treatment comparisons are possible given careful design, analysis, and interpretation. We congratulate ISPOR for providing guidelines for CER that emphasize the potential benefits without giving CER a black box warning for its potential harms.