Addressing Multimorbidity in Evidence Integration and Synthesis

Abstract

To minimize bias, clinical practice guidelines (CPG) for managing patients with multiple conditions should be informed by well-planned syntheses of the totality of the relevant evidence by means of systematic reviews and meta-analyses. However, deficiencies along the entire evidentiary pathway hinder the development of evidence-based CPGs. Published reports of trials and observational studies often do not provide usable data on treatment effect heterogeneity, perhaps because their design, analysis and presentation is seldom geared towards informing on how multimorbidity modifies the effect of treatments. Systematic reviews and meta-analyses inherit all the limitations of their building blocks and introduce additional of their own, including selection biases at the level of the included studies, ecological biases, and analytical challenges. To generate recommendations to help negotiate some of the challenges in synthesizing the primary literature, so that the results of the evidence synthesis is applicable to the care of those with multiple conditions. Informal group process. We have built upon established general guidance, and provide additional recommendations specific to systematic reviews that could improve the CPGs for multimorbid patients. We suggest that following the additional recommendations is good practice, but acknowledge that not all proposed recommendations are of equal importance, validity and feasibility, and that further work is needed to test and refine the recommendations.

INTRODUCTION

Most current clinical practice guidelines (CPG) are typically developed for managing individuals with a single disease, and also apply to patients with multiple conditions, provided that each can be managed separately from the others.^1,2 Some patients, however, have conditions that must be considered jointly, perhaps because the management of one affects the management or the course of other. Uncritical adherence to guideline-recommended management of their isolated conditions can be impractical or even harmful for such patients.³ Tailored guidance is needed at least for specific combinations of conditions. Very few examples of CPGs providing such guidance exist,^1,2 perhaps because developing such guidance is hindered by deficiencies along the entire evidentiary pathway. Pivotal trials or observational studies often exclude individuals with multimorbidity.^4,5 Even when these individuals are included, the treatment effect is typically averaged over all participants, without report of whether and how it is modified in those with important multimorbid conditions.⁶ Thus, systematic reviews that inform CPG recommendations often do not or cannot describe the pertinent evidence base.⁷

This manuscript describes the outcomes of a working session conducted with the goal of generating recommendations to help overcome some of the challenges in synthesizing the primary literature so that the results of the evidence synthesis are applicable to the care of those with multiple conditions that must be considered jointly, or in other words, cannot be considered separately from each other. Here, we focus on improving evidence synthesis and integration; companion papers discuss the generation, analysis and reporting of primary data,⁸ and the development of CPG recommendations themselves.⁹

METHODS

We used an informal group process. Two investigators with substantial expertise in systematic review and meta-analysis compiled an initial list of challenges for systematic reviews in the development of clinical practice guidelines for multimorbid patients; this list was discussed in a meeting of the Improving Guidelines for Multimorbid Patients Study Group investigators. The revised list was discussed in a dedicated session attended by methodologists attending the 2010 Spring Directors’ meeting of the Agency for Healthcare Research and Quality Evidence-based Practice Center program. Attendees included the senior leadership of the Centers comprising a group with extensive collective expertise in systematic reviews, meta-analysis, epidemiology, and decision science. Feedback was subsequently incorporated, and a set of draft recommendations was developed. The draft recommendations were discussed with respect to their importance, scientific and face validity, and feasibility at a conference on Improving Guidelines for Multimorbid Patients (Baltimore, MD, October 2010) that was attended by researchers from various disciplines (medicine, public health, biostatistics), and stakeholders from government, other payors and industry. Stakeholders reviewed a set of preliminary recommendations and suggested revisions. The herein described final set of issues and recommendations were created with incorporation of ideas discussed at the conference.

All systematic reviews, including those described in the current paper, should adhere to existing good practice recommendations. Examples include the Institute of Medicine’s 21 standards and 82 elements of performance for publicly funded systematic reviews,¹⁰ as well as guidance from programs and initiatives that perform systematic reviews worldwide, such as the Cochrane Collaboration,¹¹ Agency for Healthcare Research and Quality Effective Healthcare Program,^12–19 and the Center for Reviews and Dissemination.²⁰ We build upon the established general guidance with recommendations that are specific to systematic reviews that aim to inform CPGs for patients with multiple conditions that must be managed jointly. Not all of the proposed recommendations are of equal importance, validity and feasibility. Not every CPG has a formal systematic review process, although this is recommended by the Institute of Medicine.²¹ We recognize that the personnel tasked with developing CPGs and the team tasked with performing systematic reviews can overlap, in which case some of the recommendations below should be modified in the obvious way. While this work has focused on systematic reviews conducted to inform CPG development, the recommendations and elaborations we provide may be of interest to all systematic reviews dealing with multimorbidity.

RESULTS

Table 1 outlines recommendations for systematic reviews that inform CPGs for individuals with multimorbidity, organized according to the systematic review step they refer to (steps A to H in the first column of Table 1). Such reviews will have to summarize evidence on interactions between treatments and comorbidities. Meta-analyses of individual patient data are most suitable for synthesizing evidence on such interactions, but present substantial logistical challenges, and are very resource intensive.^22,23 Until access to individual patient data from trials becomes more routine, it is unrealistic to recommend meta-analysis of individual patient data to address comorbidity; instead, best efforts should focus on making reviews of aggregate data as informative as possible. The following sections provide explanations and elaboration on our additional recommendation statements, organized by systematic review step.

Table 1.

Issues and Corresponding Recommendations for Systematic Reviews Aiming to Inform Guidelines in Patients with Comorbidity

Systematic review step [Description]	Issues of particular importance to systematic reviews of multimorbidity	Specific recommendations
A. Prepare the topic
This includes:   a. Refining key questions with stakeholder engagement   b. Developing an analytic framework   c. Assessing feasibility and creating time lines	When the CPG workgroup and the systematic review team do not overlap, there is risk of miscommunication of: • The focus and scope of the CPGs. • The strengths and limitations of systematic reviews that are based on literature data. • The cost and effort involved in performing a systematic review. Systematic reviews of multimorbidity will have to address the presence and magnitude of interactions between conditions and treatments. To become manageable, they may have to be organized into several key questions. The logic and assumptions behind the key questions should be transparent.	1. Establish early communication between the CPG workgroup and the systematic review team to refine scope and goals for the systematic review, and educate CPG workgroup members who lack Evidence-Based Medicine (EBM) expertise. 2. Decide on the feasibility of performing an informative systematic review within time line and resource constraints.
B. Establish a protocol
This includes defining the:
a. Population of interest	Most studies will not enroll exactly the population that is the focus of the CPG. One has to operationalize which studies are applicable enough to the CPG. For, example, is it necessary that all patients enrolled in a study have all conditions of interest? If not, how prevalent should the each condition be to make a study eligible for inclusion?	3. Engage the CPG workgroup to define: a. The conditions that are most likely to lead to heterogeneity of treatment effect b. Population eligibility criteria that are as inclusive as meaningful
b. Comparison of interest (intervention, comparator)	Multimorbid patients are likely to receive many treatments in addition to the intervention and comparator of interest. For example, if the comparison of interest is A vs. B, the background treatments in two studies may be X 1 and X 2. Combining studies comparing A + X 1 vs. B + X 1 with studies comparing A + X 2 vs. B + X 2 is not meaningful if the background treatment interacts with the interventions (A, B) or populations of interest.	4. Engage the CPG clinical experts to help evaluate the likelihood for interaction between any background interventions and the examined intervention or comparator.
c. Outcome of interest	The relative importance of outcomes may differ for patients with multimorbidity compared to patients with a single condition. Outcomes can be classified in order of importance to patients into: • Critical (e.g., mortality, disabling stroke), • Important clinical, often representing competing risks (e.g., myocardial infarction) • Intermediate clinical (e.g., hypertension), and • Minor/other (e.g., anemia).	5. Engage the CPG workgroup to identify critical or important clinical outcomes.
d. Study designs of interest	We anticipate scarcity of data on interactions between the comorbidities and interventions of interest. Using a best-available-evidence approach, we would consider both randomized and nonrandomized studies.	6. Consider including well-designed, well-conducted and well-analyzed nonrandomized comparative data.
C. Identify studies
	[Same considerations as in all systematic reviews]	[Follow standard systematic review guidance]
D. Extract data
	[Same considerations as in all systematic reviews]	[Follow standard systematic review guidance]
E. Assess the risk of bias
	Treatment-by-comorbidity interactions analyses are not commonly reported in primary studies. It is possible that the reporting of interaction analyses is dependent on the findings (analysis reporting bias).	7. Assess the likelihood of selective reporting biases (including publication bias).
F. Synthesize information
	As above, treatment-by-comorbidity interactions may be incompletely and selectively reported, limiting ability for quantitative analyses. Further, there is lack of statistical methods for the joint meta-analysis of main and interaction effects accounting for their covariance.	8. Perform a nonquantitative synthesis of the available information. 9. If applicable, perform quantitative analysis of the main treatment effects and treatment-by-comorbidity interaction effects using methods that allow for between-study heterogeneity.
G. Evaluate the strength of the evidence base
	Follow a systematic approach to assess the strength of the body of evidence with respect to each outcome of interest.	10. Assess the strength of the evidence for each outcome of interest.
H. Report findings
	[Same considerations as in all systematic reviews]	[Follow standard systematic review guidance]

Systematic Review Step A: Prepare the Topic

1. Early communication between the CPG workgroup and the systematic review team to refine scope and goals for the systematic review, and educate CPG workgroup members who lack Evidence-Based Medicine expertise.

Although ostensibly obvious, this interaction between the CPG workgroup and the systematic review team can be complex. Presumably, the CPG workgroup has identified the topic at hand as one where interactions between treatments and comorbidities should be accounted for. The third paper⁹ elaborates on these interactions. Briefly, it is important to prioritize the comorbidities that will be included in the systematic review, i.e., comorbidities likely to modify the effect of treatments, and that are not too rare. A preliminary, scoping literature review and consultation with clinical experts may help identify which (if any) comorbidities to prioritize. For example, depression, chronic obstructive pulmonary disease, osteoarthritis, hypertension or glaucoma are unlikely to change the effect of tyrosine kinase inhibitors in endothelial growth factor receptor positive lung cancer. By contrast, heart failure should be included as a treatment effect modifier in studies examining the effect of beta-agonists on dyspnea in patients with asthma; advanced kidney disease might modify the effectiveness of heart failure treatments; and mild cognitive impairment (or alcoholism) might modify the effectiveness of virtually any medication, especially those with low therapeutic index (e.g. warfarin, digoxin), and so forth.

• 2.Decide on the feasibility of performing an informative systematic review within time line and resource constraints.

The rationale for this recommendation is clear.

Systematic Review Step B: Establish a Protocol

Developing and using a protocol is strongly recommended for all systematic reviews. The study eligibility criteria are the part of the protocol that merit special mention for systematic reviews of multimorbidity. The Population, Intervention, Comparator, Outcomes and Study design formalism is often used to describe study eligibility criteria for reviews of interventions.²⁴

• 3.Engage the CPG workgroup to define the conditions that are most likely to lead to heterogeneity of treatment effect, and population eligibility criteria that are as inclusive as meaningful.

The comorbidities of interest will probably include conditions that frequently co-occur with the index condition and whose presence or absence is likely to lead to differential response to treatment. In reality, no two comorbidities are completely independent; however, for some comorbidities, interactions with treatments can be clinically important (e.g., beta-agonists for asthma in those with ischemic heart failure), whereas for others they can be negligible (e.g., imatinib’s effect on the course of chronic myelogenous leukemia is probably unmodified by the presence or absence of heart failure).

Operationalizing the eligibility criteria for study populations presents a very practical challenge. For example, studies enrolling patients with comorbidities are needed, if the objective is to synthesize information on treatment effect heterogeneity across comorbidity categories.⁸ In light of sparse data, the systematic review should follow a best-available-evidence approach and be as inclusive as practical—we call attention to the inclusion of nonrandomized trials and epidemiological studies in the synthesis in recommendation #6. When relevant subgroup analyses were not done and only overall results are available, the review team should consider defining a lower boundary for the prevalence of each condition in the enrolled populations, such that results might be relevant to patients with the comorbid condition(s). Because such boundaries will be arbitrary, sensitivity analysis using lower or higher prevalence cutoffs should be planned. Average results from studies including people with comorbidities in high proportions (e.g., above 60 %) are, other things being equal, more likely to generalize to patients with comorbidity, even in the presence of treatment by comorbidity interactions compared to studies where said proportions are low (e.g., less than 50 %).

• 4.Engage the CPG clinical experts to help evaluate the likelihood for interaction between any background interventions and the examined intervention or comparator.

Individuals with multimorbidity often receive other treatments (background treatments) along with the intervention that is the subject of the review. Where interaction between background interventions and the index condition or treatment of the index condition is highly unlikely, the background interventions can be effectively ignored. Otherwise, the definitions of the interventions and comparators should ideally include the background treatment as well, although the latter are rarely uniform in the trial and are typically not well described.²⁵ For example, consider patients with schizophrenia and diabetes, in whom antipsychotic drugs can worsen diabetes outcomes (but nevertheless benefit patients in terms of other outcomes, including quality of life). Considering such patients together with patients not on antipsychotics introduces heterogeneity in the treatment effect. Additionally, some interactions may be clinically plausible, but may have unclear practical impact. For example, nonsteroidal anti-inflammatory drugs for arthritis or lower back pain might or might not alter the effectiveness of heart failure interventions appreciably.

• 5.Engage the CPG workgroup to identify critical or important clinical outcomes.

This recommendation is discussed in detail in the third paper.⁹ Briefly, one can reasonably anticipate that most patients would prioritize “critical” outcomes such as mortality, and “important clinical” outcomes such as myocardial infarction or major depression over surrogates such as cholesterol levels. However, in general, choice of outcomes should be informed by patients’ preferences and values.²⁶ The range of outcomes for consideration in a review is wider when the population has important comorbidities or multimorbidity, because several competing outcomes may be relevant.²⁷ For example, if the systematic review is focused on interventions to reduce osteoporotic fractures and the population is a multimorbid, frail population, mortality is an essential outcome to include in the review in order to evaluate the absolute benefit of the intervention. Obviously, expanding the list of outcomes broadens the scope of the review, and requires more resources.

• 6.Consider including well-designed, well-conducted and well-analyzed nonrandomized comparative data.

All studies that explicitly enrolled individuals with the comorbidity cluster of interest in addition to other patients should be considered. The most informative studies would report analyses for effect modification by comorbidity in the form of treatment-by-comorbidity interactions, or sufficient data to calculate these interactions (see Step F and Recommendations 8 and 9). If results from an interaction analysis are not presented, studies should report outcomes stratified by comorbidity. In our experience, interaction analyses or stratified results are rarely reported, and this is a major obstacle to a meaningful synthesis.⁷

Given the anticipated scarcity of randomized controlled trials (RCTs) enrolling individuals with multimorbidity, nonrandomized studies represent a practical alternative to estimate the effectiveness and safety of treatments in real-world settings.^4,5 Several CPGs have followed this route.²⁸ Empirical evidence suggests that data from nonrandomized and randomized designs are not in profound disagreement,^29–32 but nonrandomized studies are generally more susceptible to biases.³³ For example, propensity score methods attempt to emulate randomized comparisons by making contrasts between patient groups that are on average similar on all observed confounders.³⁴

Systematic Review Step C: Identify Studies

With respect to study identification, we did not identify specific considerations beyond those generally applicable to all systematic reviews other than the potential expansion of the scope or work.⁹

Systematic Review Step D: Extract Data

Data to be extracted include information on the provenance of the paper (such as citation information); the Population, Interventions, Comparator and Outcomes elements with numerical data on the effect of intervention and its modification by comorbidity; and methodological items that will help them assess the risk of bias of the extracted studies. This step should be managed with great care, and as resources allow, should be done independently and in duplicate. Detailed recommendations are found elsewhere.^{10–16,19,20} We did not identify specific considerations for systematic reviews of multimorbid patients, beyond those generally applicable to all systematic reviews.

Systematic Review Step E: Assess the Risk of Bias

This happens concurrently with data extraction. Current thinking on assessing the risk of bias for clinical trials is summarized in the Cochrane manual for systematic reviews, and includes evaluating the likelihood for selection bias induced by the patients included and analyzed, performance bias (systematic differences in administering interventions), detection bias (systematic differences in determining outcomes), attrition bias (differential attrition rates), and reporting (selective reporting of analyses and results) and other biases.¹¹ The actual assessments are based on study and design characteristics for which there is empirical support (such as allocation concealment, blinding of patients, outcome assessors or analysts), and on theory and methodological principles.^35–40 Although such characteristics may be appropriate and sufficient for assessing treatment-by-subgroup interactions, empirical data on their association with the magnitude of subgroup-by-treatment interaction effects do not exist.

• 7.Assess the likelihood of selective reporting biases (including selective analysis reporting bias).

We hypothesize that selective analysis reporting is a prevalent challenge for systematic reviews of multimorbidity. Primary studies do not routinely report analyses informing on the interaction between treatment effect and comorbidity.^7,41 It is conceivable that such analyses are reported based on their results, e.g., may be more likely to be reported if they are statistically significant. Because the power to detect interaction effects in many studies is low, most interaction tests are expected to be nonsignificant. Thus, if reporting bias exists, a synthesis of reported results of interactions can be highly misleading. Reporting bias is exceedingly difficult to document. Careful review of the publication record, along with content knowledge⁴² or cross-comparison of published reports and prospectively registered protocols, may suggest whether selective analysis reporting has happened. However, in practice, assessments of the likelihood of bias in an evidence base are very difficult to perform, and are often based on conjecture. Because of the above, we emphasize recommendations 7, 8, and 9.

Systematic Review Step F: Synthesize Information

Every systematic review should include at least a nonquantitative (qualitative) synthesis. When appropriate and possible, a quantitative synthesis (meta-analysis) is encouraged. The same general principles that apply to all systematic reviews are relevant here.¹⁹

• 8.Perform a nonquantitative synthesis of the available information.

Because the treatment-by-comorbid condition interactions are unlikely to be reported in all studies, or to be analyzed in the same way (e.g., using similar definitions for subgroups for comorbid conditions), nonquantitative syntheses are expected. Nonquantitative syntheses present study characteristics and results succinctly, in tabular or graphical form. More than a simple listing, the presentation aims to “summarize” overall trends, make evidence gaps obvious, and alert on the likelihood of biases that operate at the study level, such as publication bias, selective outcome and analysis reporting bias, and time-lag bias.⁴³ Common pitfalls when performing nonquantitative analyses include unwarranted reliance on the number of statistically significant results (“vote counting”) or claiming associations between treatment effects and study characteristics when none exist.⁴⁴ Unfortunately, nonquantitative analyses rarely lead to strong, specific and actionable conclusions.

• 9.If applicable, perform quantitative analyses of the main treatment effects and treatment-by-comorbidity interaction effects using methods that allow for between-study heterogeneity.

The standard guidance is to perform quantitative analyses whenever possible and informative.¹⁹ The premise, role and methodology of meta-analysis and meta-regression, the impact of biases (including publication bias) on quantitative results, and the pitfalls in the interpretation of quantitative results have been discussed extensively in the literature.⁴⁵

When individual participant data are not available, there are at least two ways to quantify whether treatment effects are systematically different between those with a single condition and those with multiple conditions. In the more common case, each study reports only overall results, and one can only explore associations of the overall treatment effect with the proportion of patients with the comorbidities of interest in each study in meta-regression analyses.^45–48 In the best case, treatment by comorbidity interaction analyses have been performed (and are adequately reported) in each study and can be quantitatively summarized.

Relating the Treatment Effect to the Proportion of Patients with Comorbid Conditions

Meta-regressions are particularly useful when examining the effects of study-level factors that apply equally to all patients in a study, such as the duration of follow-up or country of study conduct.⁴⁹ However, they are often less useful in examining the effects of patient-level factors, such as comorbidities,⁵⁰ across studies. Patient-level factors are captured by aggregate data (e.g., percentage of patients with diabetes), and ecological fallacy can obscure the true relationship between individual patient characteristics and treatment effect.^50,51

Synthesizing Study-Level Analyses of Treatment-by-Comorbidity Interactions

The goal is to synthesize two pieces of information, namely, the main effect of the treatment in patients with an index condition, and the treatment-by-comorbidity interaction effect. Because this is a multivariate problem, multivariate meta-analysis methods may be best suited to address it. Instead of performing separate meta-analyses for the main and interaction effects across studies, multivariate meta-analysis would analyze both quantities jointly, in the same model. Methods for multivariate meta-analysis are being developed for the joint analysis of multiple outcomes,^52–57 multiple follow-ups^58,59 and multiple treatments.^60–67 In particular, methods for the meta-analysis of regression models may be especially relevant.⁶⁸ This would require reporting of the covariance matrices of risk prediction models, which is not common practice.

Systematic Review Step G: Evaluate the Strength of the Evidence Base

A rating of the strength of the body of evidence communicates to the CPG workgroup the level of confidence that the reviewers have about the results in the literature.

• 10.Assess the strength of the evidence for each outcome of interest.

Frequently used is the “strength of evidence” system developed by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group, although other systems with modest differences from GRADE are used as well.⁶⁹ In the GRADE and other systems, the body of evidence for a given intervention and given outcome are graded on the likelihood that additional evidence will change the conclusion about the results.⁶⁹ The considered domains are: 1) study risk of bias, 2) consistency of results across studies, 3) directness of results to the question of interest, and 4) the precision of the estimates within each individual study. We expect that the proponents of the GRADE system will find it suitable for grading the strength of the body of evidence as pertains to multimorbid patients. The consideration of multimorbid patients may affect the evaluation of all of these domains. Systems such as GRADE are meant to be tools for transparency and communication, but may fall short of their goals if they are employed in an uncritical way.

Systematic Review Step H: Report Findings

Typical recommendations on reporting of findings for systematic reviews are applicable. These include following statements such as the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines.⁷⁰

CONCLUDING REMARKS AND ROADMAP FOR THE FUTURE

Those tasked with developing CPGs have a very tall order. Especially for questions on the management of people with multimorbidity, their challenges are multiplied when pertinent primary data do not exist, or are not reported in a useable and informative way. Therefore, notwithstanding the transparency and rigor that systematic reviews bring to CPG development, we expect that most reviews will not be directly informative to the CPG development.¹⁰ The Institute of Medicine (IOM) recommendations for developing CPGs outline a process that has to be followed, but that will most likely have a “low yield” for systematic reviews that address multimorbidity.

Substantial change can come only from a paradigm shift towards (a) prioritizing pragmatic clinical research with special attention to effect modification, (b) enabling large scale cross-institutional collaborations, and (c) informing CPG recommendations by rigorous analyses of the trade-offs of benefits, harms, and burdens accounting for their uncertainties. The first point is expanded in the accompanying reference.⁸ On the second point, individual-patient data meta-analyses can utilize studies that have already been performed, and repurpose them to address questions for which they are otherwise uninformative. Thus, though resource intensive compared to reviews of aggregate data, they should be undertaken for priority questions within a coherent and rational health system. Third, CPG recommendations address decisional problems, and their development would benefit from formal methods for making decisions under uncertainty. Mathematical modeling-based decision and economic analysis examines and compares all meaningful alternatives, makes assumptions explicit, distinguishes choices from chance, promotes transparency, incorporates preferences, and helps navigate the tradeoffs of intervention benefits, risks and burdens in the presence of uncertainty.

In sum, unless the global clinical research community adopts a drastically different approach to knowledge generation, data sharing and attribution of credit, the expected payoff of efforts to generate evidence-based CPGs for people with multimorbidity is modest at best. Lessons may be learned from molecular medicine—the research community understood the futility of using small individual studies to dissect the genetic components of complex diseases, and has embraced large-scale collaborative meta-analyses of individual participant data.^71,72 It is time that important questions in clinical medicine followed a similar approach.