Summaries of harms in systematic reviews are unreliable Paper 1: An introduction to research on harms

Abstract

Objective:

Most systematic reviews of interventions focus on potential benefits. Common methods and assumptions that are appropriate for assessing benefits can be inappropriate for harms. This paper provides a primer on researching harms, particularly in systematic reviews.

Study Design and Setting:

Commentary describing challenges with assessing harm.

Results:

Investigators should be familiar with various terminologies used to describe, classify, and group harms. Published reports of clinical trials include limited information about harms, so systematic reviewers should not depend on these studies and journal articles to reach conclusions about harms. Visualizations might improve communication of multiple dimensions of harms such as severity, relatedness, and timing.

Conclusion:

The terminology, classification, detection, collection, and reporting of harms create unique challenges that take time, expertise, and resources to navigate in both primary studies and evidence syntheses. Systematic reviewers might reach incorrect conclusions if they focus on evidence about harms found in published reports of randomized trials of a particular health problem. Systematic reviews could be improved through better identification and reporting of harms in primary studies and through better training and uptake of appropriate methods for synthesizing evidence about harms.

Incomplete knowledge concerning intervention harms threatens public health. Systematic reviews of randomized controlled trials (RCTs) are often thought to provide the most reliable evidence of benefits and harms. Systematic reviews, however, focus largely on potential benefits and typically include trials designed to assess potential benefits. Methods and assumptions that are appropriate for assessing benefits in trials and reviews may be inappropriate for assessing harms. Furthermore, reviews based on publicly available information might be limited because reports often include only a subset of harms, which may be biased.

The sources referenced in this commentary have been collected from regulatory and methods guidance documents, textbooks, non-systematic searching and snowball sampling of journal articles, communication with experts in the field, and discussion among the authors. This paper is the first in a series of three articles (Table 1) dedicated to harms and provides a primer for researching harms and it identifies changes needed to ensure that systematic reviews and clinical practice guidelines consider the trade-offs among all outcomes that matter to patients.

Table 1.

Synthesis of harms: Challenges and opportunities

Series paper	Description
Qureshi R, Mayo-Wilson E, Li T. Summaries of harms in systematic reviews are unreliable Paper 1: An introduction to research on harms.[1]	This paper describes key aspects of assessing harms. First, it identifies the various ways that harms are classified, including the terminology used to describe grouping of harms and different data dimensions such as rarity, severity, relatedness, and timing. Second, it explores different methods for assessing harms in clinical research. Third, it describes the challenges associated with analyzing, reporting, and synthesizing harms.
Qureshi R, Mayo-Wilson E, Rittiphairoj T, McAdams-DeMarco M, Guallar E, Li T. Summaries of harms in systematic reviews are unreliable Paper 2: Methods used to assess harms are neglected in systematic reviews of gabapentin.[2]	Part 1 of an overview of gabapentin systematic reviews that serves as a methodologic study for how harms are synthesized. This paper examines the methods used in systematic reviews and meta-analyses to assess harms across all stages of the review process and compared these with recommendations for assessing harms in reviews. We found review methods are focused on addressing questions of benefit and the tokenistic inclusion of harms in reviews means that the methods may not meet the needs to produce a valid synthesis of harms.
Qureshi R, Mayo-Wilson E, Rittiphairoj T, McAdams-DeMarco M, Guallar E, Li T. Summaries of harms in systematic reviews are unreliable Paper 3: Given the same data sources, systematic reviews of gabapentin have different results for harms.[3]	Part 2 of an overview of gabapentin systematic reviews that serves as a methodologic study for how harms are synthesized. This paper evaluates the results for harms across these reviews, with a particular interest in the consistency of results given similar supporting data. We found significant challenges in the selection of harms to assess and report, largely driven by reviewer preferences as opposed to standardized approaches, leading to different summaries of harms across reviews.

1. What are harms?

When patients “weigh the balance of benefits and harms”, they hope that potential positive effects of an intervention will outweigh any negative effects, such that the intervention would be worth taking.[4] “Harms” is an umbrella term for unwanted effects of interventions. This use follows recommendations in the Consolidated Standards of Reporting Trials (CONSORT) Statement Harms extension[5] and the Preferred Reporting Items in Systematic Reviews and Meta-Analyses (PRISMA) Harms extension,[6] which discourage using euphemisms that might obfuscate the importance of these effects. For different interventions and settings, harms might be described as “adverse event(s)”, “adverse effect(s)”, “adverse drug event(s)”, “adverse reaction(s)”, “adverse drug reaction(s)”, “risk(s)”, “safety”, “sequalae”, “toxicity”, “complication(s)”, or “side effect(s).”[7-11]

There are some important distinctions among terms used to describe harms. Box 1 provides some examples. Defining a “harm” also depends on context. That is, an event might be considered a harm in one setting but a beneficial outcome in another; for example, increased weight might be a harm in a trial of an antipsychotic medications or a positive outcome in a trial of an intervention for eating disorders. The same event might also be both a harm and a benefit; for example, a surgical trial might observe both that some people are harmed because they die during surgery and that the surgery is beneficial because it increases overall survival time. Relatedly, not all events are of equal importance to all people; for example, some people might be upset to gain 5% of their body weight while other people might be indifferent or even glad.

Box 1. Different definitions within the umbrella of “harms”.

Adverse event

United States (US) Food and Drug Administration (FDA) defines “adverse event” as: “Any untoward medical occurrence associated with the use of a drug in humans, whether or not considered drug related.”[12]

Similar definitions are shared by other agencies around the world such as the International Conference on Harmonization (ICH), European Medical Association (EMA), and World Health Organization (WHO).

Side effect

On the other hand, “side effects” are defined very differently by the WHO as:

“Any unintended effect of a pharmaceutical product occurring at doses normally used in humans which is related to the pharmacological properties of the medicine. Such effect may be either positive or negative. Such effects may be well-known and even expected and may require little or no change in patient management.”[7]

Whereas “adverse events” do not have to be causally related to interventions and are always negative, “adverse effects” are also negative but are related to an intervention, and “side effects” are related to the intervention and can be either negative or positive.[10,11] For example, a potentially desirable side effect was found for sildenafil (Viagra), which was originally intended as a blood pressure medication and had a strong unintended “side effect” that led to its use for erectile disfunction.

2. How are harms classified by type and frequency?

Binary or dichotomous harms are often reported as counts or frequencies of events. In regulatory sciences, harms may be collected as events and mapped to standardized terminology before they are analyzed. Examples of standardized terminologies include Common Terminology Criteria for Adverse Events (CTCAE) and Medical Dictionary for Regulatory Activities (MedDRA).[13-15] The Coding Symbols for a Thesaurus of Adverse Reaction Terms (COSTART) preceded MedDRA and was used in trials conducted from 1985 to the late 1990s. MedDRA remains in common use. Both MedDRA and COSTART are hierarchical systems of categorizing harms that (1) establish common vernacular to facilitate analysis and interpretation, and (2) group physiologic events to better describe patterns at higher, aggregated levels.[13,15-19]

Patients and clinicians might use multiple terms to describe a single event. For example, an investigator seeking to evaluate whether an intervention makes people drowsy would want to combine similar terms used by patients and clinicians to describe the same feeling.[16-19] MedDRA identifies a number of “lowest level terms” that are considered synonyms for the same “preferred term.” The latter is the standardized code for a specific harm, which is used for the lowest level of analysis.[19] For example, “drowsiness,” “lethargy,” “sedation,” and “somnolence” are ways of referring to the preferred term “somnolence.”[13] Fig. 1 shows an example of the MedDRA path from lowest to higher-level terms for the preferred term “nausea.”[13,19]

Fig. 1.

MedDRA hierarchy for nausea.

Specific events might be related biologically or physiologically, so investigators might want to conduct broader evaluations of harms that share common characteristics.[16-19] For example, assessing harms at higher levels—i.e., above the preferred term—might reveal which body systems are affected (e.g., nervous, genitourinary, cardiac, cardiovascular). Grouping harms in this way has the additional effect of increasing statistical power because, unless individual and specific harms are very common, trials might not observe enough events to precisely estimate harms.[18,20-22] Although different organizations use different cut-offs, Box 2 includes common thresholds for describing the frequency or rarity of harms.[23]

Box 2. Council for International Organizations of Medical Sciences thresholds of occurrence.

Very common ≥ 1/10

Common (frequent) < 1/10 AND ≥ 1/100

Uncommon (infrequent) < 1/100 AND ≥ 1/1000

Rare < 1/1000 AND ≥ 1/10000

Very rare < 1/10000

2.1. Seriousness and Severity of harms

Related harms might occur at several different levels of severity. For example, allergic reactions can be mild to life-threatening or deadly. The Common Terminology Criteria for Adverse Events (CTCAE) defines five grades of severity (Box 3).[14] “Seriousness” is related to “severity,” but has a specific meaning in regulatory research. That is, a “Serious Adverse Event” is an event that results in (1) death or is life-threatening, (2) hospitalization or prolongation of existing hospitalization, (3) persistent or significant incapacity or substantial disruption of the ability to conduct normal life functions (i.e., disability), or (4) congenital anomaly or birth defect.[12] Legal requirements apply to “serious” harms, which may trigger subsequent actions or reporting requirements to ethics committees and regulatory bodies. While these regulatory and standardized definitions exist, “serious” and “severe” are also commonly used in a non-standardized and informal way, especially outside in studies of unregulated interventions (e.g., clinical psychology, exercise science).

Box 3. Grades to classify severity of harms.

Grade 1 – “Mild”: Asymptomatic or minor symptoms; clinical or diagnostic observations only; no intervention needed

Grade 2 – “Moderate”: Minimal, local, or non-invasive intervention indicated

Grade 3 – “Severe”: Medically significant but not immediately life-threatening; hospitalization or prolongation of hospitalization indicated; disabling

Grade 4 – Life-threatening consequences (i.e., immediate risk of death); urgent intervention indicated

Grade 5 – Death related to adverse event

In addition to classifying severity, harms are often classified according to how likely they are to be caused by the intervention. Relatedness is challenging to determine and the approach is not standardized.[24] The challenge of classifying harms data is further exacerbated by the multi-dimensionality of the data as the aspects of timing and repeated events are often ignored when studies analyze harms, although these can play a role in helping to assess relatedness and importance to stakeholders.[25-27] See APPENDIX for more details on relatedness and multi-dimensionality of harms.

3. How are harms detected?

Harms can be detected in trials, observational studies, and in clinical care. Different sources of evidence for harms have different strengths and weaknesses with regard to the harms signal that they identify and the certainty of the evidence that they provide for the relationships between the harms and interventions.

3.1. Randomized controlled trials

Before a new drug or biologic can be approved for marketing, regulators such as the FDA may require that several studies be conducted, including RCTs. Trials done before marketing approval typically evaluate a drug’s benefits and look for common and clinically important harms.[28] RCTs can provide strong evidence of causality and would be the ideal design to assess harms if designed to do so. In reality, however, RCTs conducted to support regulatory approval often have limitations for harms.[29] For example, harms are rarely chosen as primary outcomes, so RCTs are typically not designed to assess differences in harms between groups. Additionally, RCTs to support regulatory approval often have short duration of follow up with small sample size and observe too few events to detect all but the most common harms (i.e., they are “underpowered”).[20,25,30] Box 4 presents an example to illustrate how many trials are underpowered to detect the true effect size for even common harms.

Box 4. Example of inadequate power for common harms in Phase 3 trials.

Assume that a randomized trial (e.g., conducted to seek marketing approval) has a sample size of 300 patients, a 1:1 allocation, and treatment duration of 6 months.

Assume a “common” harm has a true annual incidence of 1/50 among those receiving Intervention. The harm is three times as common in the Intervention group compared with the Control group (i.e., the true Risk Ratio = 3).

In this hypothetical trial, on average, we could expect 1.5 events in the Intervention group and 0.5 events in the Control group. It would take a trial twice the size before we could expect to see a single “common” event in the Control group.

In a trial such as this, in which 0 to 2 events might be expected in either group, the observed Risk Ratio will often be 1, infinitely positive, or infinitely negative.

Following conventions for estimating statical power for benefit outcomes, a trial approximately eight times this size (i.e., over 2300 patients) would be needed to precisely estimate a “common” harm such as this.

3.2. Observational studies of harms

There are also many different sources of observational data on harms (e.g., cohort studies, case-control studies, case-series, electronic medical records). Compared with RCTs, observational studies face greater challenges stemming from selection bias and information bias. That is, the types of interventions people take for particular health problems might affect the harms that occur for a given intervention in a given population. Despite these limitations, observational studies may be able to achieve larger sample sizes, use less restrictive eligibility criteria (e.g., include people with comorbidities and receiving concomitant interventions), include greater representation of people belonging to minority and underserved populations, and observe longer periods of exposure. These features make observational studies especially useful for detecting unexpected, rare, or long-term harms.[20,29,31-33] Observational studies may be most useful when they are designed to address research questions that RCTs cannot address. Because they can also be very misleading, evidence from RCTs may be more informative when both observational studies and RCTs have addressed the same question. For example, observational studies of hormone replacement therapy for coronary heart disease among postmenopausal women suggested no negative effects of therapy, however large RCTs of the same population, intervention, and outcomes found increased cardiovascular events and a survivorship bias that was missed in the observational studies.[34,35]

Spontaneous reporting systems (i.e., pharmacovigilance) can be valuable for detecting rare or long-term harms of drugs after they have been approved for marketing.[36-38] Examples of these systems include the FDA’s Adverse Event Reporting System (FAERS), Vaccine Adverse Event Reporting System (VAERS), and Sentinel initiative.[39-41] Social media has also been used as a potential source of spontaneously reported harms.[42] FAERS and VAERS are publicly accessible, government-run databases of harms that are reported for any drug and vaccine approved for use in the US, although reports from non-US countries are accepted.[39,40] The majority of reports comprise mandated reporting by manufacturers who must submit any harms reported back to them.[39,40] Reports can also be submitted voluntarily by patients and clinicians. There are several important limitations to this voluntary approach that lead to these surveillance systems being used primarily for hypothesis generation and for identifying potential harms that might warrant further evaluation (Box 5).[36,39,40,43]

Box 5. Limitations of assessing harms using spontaneous reporting systems.

• Information biases (such as recall biases and sub-optimal reporting quality);

• Confounding by indication: Harms might be associated a drug because they are causally associated with the condition for which the drug was prescribed;

• Polypharmacy: Harms might be caused by concomitant medications that are associated with the drug under investigation;

• Issues with data accuracy and completeness in reports (e.g., missing demographics or timing);

• The possibility of duplicate reports (i.e., reports of the same event submitted by both a patient and their doctor);

• Information in the reports has not been verified (i.e., submission of a report does not mean the information has been medically confirmed); and

• Common measures of effect such as rate and incidence cannot be determined because there is no denominator for the total number of people or person time exposed to a drug.

In contrast to FAERS and VAERS, which are passive surveillance systems that can be mined for potential drug-harms pairs of interest, Sentinel uses active surveillance of electronic healthcare and administrative claims data. Sentinel is a new program–launched as a pilot in 2008 and fully implemented in 2014–utilizing many databases and surveillance systems. It applies standardized formatting to allow FDA to monitor potential harms of interventions post approval.[41,44,45] Although both types of pharmacovigilance databases have limitations, they are valuable as potential supplemental sources of data for harms in reviews of pharmacologic interventions.[32,46-50]

4. How are harms collected?

Unlike benefit outcomes which are typically prespecified in studies, it can be challenging to anticipate every harm that may occur. Harms may be collected systematically (i.e., assessed for all participants using the same scales and instruments at planned times) or non-systematically (i.e., relying on self-reporting by participants).[51-54] See APPENDIX for more details about harms collection.

5. How are harms analyzed and reported?

Harms data that have been collected in studies may be insufficient for analyses if they do not include relevant dimensions such as date of onset, duration and severity of the event, and concomitant medications. Additionally, the lack of use of standardized terminology when collecting harms, as described in section 2, can lead to incompleteness and inconsistency in the data which further threatens the validity of harms analyses.[55-60] Interviews with trialists have revealed many reasons for not analyzing and reporting harms, including: an unawareness of the importance of “negative” results, data being considered “uninteresting”, having “too few events worth reporting”, and a perception of limited space in publications”.[55] Indeed, there may even be a purposeful suppression of the evidence of harm; an anonymous trialist interviewed by Smyth and colleagues in 2010 said:

“When we looked at that data, it actually showed an increase in harm amongst those who got the active treatment, and we ditched it because we weren’t expecting it and we were concerned that the presentation of these data would have an impact on people’s understanding of the study findings.” [85]

Improving the reporting of harms in trials is an ongoing challenge that remains unsolved, despite the recommendations of the CONSORT-Harms extension being 16 years old.[61]

5.1. Use of selection criteria to report harms

Most reports about interventions–from journal articles to FDA prescribing information–do not enumerate all harms that are recorded in trials and observational studies.[62] Instead, authors “selection criteria,” which are rules to determine which harms will be presented in a given report.[62] Authors might choose a subset of all harms based on frequency or based on perceived importance to patients and clinicians.[62] Selection criteria might be used to limit the number of harms to be reported because of space constraints or because readers would find it difficult to interpret hundreds of effect estimates.[5,55,62] Because selection criteria restrict what is reported to a small fraction of what was found in a study, they can be used to purposely supress certain findings. Biased reporting could be mitigated by pre-specifying which harms will be reported, however, investigators do not always know which important harms will occur before the data are collected. Moreover, selection criteria are variable between trial reports and within multiple data sources about the same trial (e.g., trial registration, drug label, publication).[62] Thresholds for reporting create a specific challenge for synthesis because reported harms are typically those that are “very common” (Box 2); because studies with null or small effects are systematically missing from the literature, meta-analyses will tend to be biased.

Fig. 2 presents several examples of selection criteria from different sources (i.e., drug label, trial registration, and journal article) with multiple thresholds for a single trial of Aristada (aripiprazole lauroxil).[63-65] These different rules in reports that rely on data from the same trials mean that different harms are presented to evidence users depending on where they look.[62,66,67]

Fig. 2.

Inconsistent selection criteria applied across different sources for the same trial.

The extent to which harms are underreported using various selection criteria can be examined by comparing different public sources for trials with the individual participant data: the more restrictive the selection criteria, the less of the overall picture consumers, clinicians, and systematic reviews have when they do not have access to the full, unpublished, data.[62,68,69] Individual participant data (IPD) for trials can sometimes be obtained through data sharing sites such as Vivli or YODA (Yale Open Data Access), government repositories such as European Medicine Agency Clinical Data or Health Canada Clinical Information on Drugs and Health Products, or from the clinical study reports supplied by the pharmaceutical manufacturers.[70-73] Clinical study reports are the most comprehensive study documents often created by a pharmaceutical manufacturer for submission to a regulator.[74,75] They detail the design, methods, analyses, and results of a study.[74-77] Appendices of CSRs sometimes contain tables of individual patient data, also called “patient data listings”, and study protocols.[74-77]

6. How do systematic reviews assess harms?

The limitations of harms assessment, analysis, and reporting in individual studies affect evidence syntheses and clinical practice guidelines. The evidence provided by a systematic review can only be as good as the evidence in the included studies. If the limitations of available data are not considered in the review process, then a review might reach incorrect conclusions concerning harms and the trade-offs between benefits and harms. Although the current paradigm for conducting systematic reviews of interventions recommends that harms be assessed so that there can be a balanced discussion of potential benefits and harms, most reviews are not designed to assess harms rigorously[6,11,78-80] and might reach conclusions that are misleading or wrong. Early assessments of systematic review methods and reporting of harms revealed limitations in the approaches taken by reviewers to assessing harms at each stage of the review process, including: restrictions on the sources that are searched for evidence and the types of studies included, limitations in the analyses of harms, and poor reporting of methods used to assess harms (Box 6).[58,78-86]

Box 6. Common limitations in harm assessment in systematic reviews.

• Reviews designed to assess the potential benefits of an intervention for a particular health problem (e.g., sertraline for adolescents with depression) tend to exclude evidence about potential harms from relevant studies, including preclinical studies (e.g., studies in non-human animals), clinical studies of related interventions (e.g., studies of similar antidepressants), studies of the same intervention for different health problems (e.g., sertraline for anxiety), and studies in related populations (e.g., adults with depression).

• Systematic reviewers limit evidence searches to bibliographic databases, and searches are poorly constructed and unlikely to return evidence for harms.

• Unpublished data are rarely used in reviews, and reviewers often restrict sources of evidence to the primary publications of RCTs.

• With limited research on the comparability and validity of models and assumptions for meta-analysis of rare events, reviewers commonly use default methods of meta-analysis.

• The reporting of how reviewers assessed harms is generally poor and inconsistent.

7. What are the recommendations for systematic reviewers in assessing harms?

Based on the early assessments of how systematic reviews synthesized harms, guidance has been developed for searching for evidence, performing analysis, and reporting results. A 2008 report by the Agency for Health-care Research and Quality[32] was followed by the Institute of Medicine’s Finding What Works in Health Care: Standards for Systematic Reviews and the first edition of the Cochrane Handbook for Systematic Reviews of Interventions, both published in 2011.[87,88] These provided guidance for incorporating harms in the review process. Later, the 2015 PRISMA-Harms extension and a 2018 FDA Guidance for Industry document provided more specific guidance on reporting harms in reviews and for conducting meta-analyses of harms.[6,89] The 2nd edition of the Cochrane Handbook, published in 2020, includes more detailed recommendations about including harms in reviews.[11,90] Although these sources focus on different aspects of the review process, there are several common themes that are consistent in their recommendations (Box 7).

Box 7. Recommendations for methods to assess harms in systematic reviews.

• Assess multiple measures for harm including multiple specific harms (i.e., preferred terms), general harms (e.g., “risk of any harm”), and proxies for harm (e.g., “drop out due to harm”)

• Focus on specific harms (as opposed to general assessments of harm)

• Search multiple sources of evidence (e.g., bibliographic databases, grey literature, etc.) to identify information about harms.

• Do not restrict the search for evidence to RCTs: look for harms data in observational studies, case reports, and adverse event reporting systems and assess the quality of the data appropriately.

• Utilize any unpublished data that can be obtained for included studies (e.g., Clinical Study Reports).

• Consider absolute measures (e.g., risk difference) over relative measures (e.g., risk ratio and odds ratio) when conducting a meta-analysis because they may give a better indication of public health impact, however they have wider confidence intervals and may have lower statistical power for rare events.

• For rare harms, including those with zero events in one or more arms, avoid inverse-variance and Dersimonian and Laird methods when selecting a model for meta-analysis and give preference to Bayesian or select frequentist models (e.g., Peto one-step odds ratio or Mantel-Haenszel odds ratio without zero-cell corrections).

• Report methods for assessing harms in systematic reviews, including details of how the primary studies assessed harms.

Implementing these recommendations requires a substantial investment, both in time and resources. For example, the time required to obtain and make use of unpublished data can be prohibitive for reviewers because harmonizing IPD is labor intensive and because important details may be buried in thousands of pages of reports.[75] Additionally, the expertise necessary to implement some recommendations, such as appropriately incorporating observational data, may be beyond what some review teams possess.

8. Conclusions

Assessing harms is a demanding process that requires much work and careful thought, above and beyond that required to synthesize evidence of efficacy or effectiveness. The terminology, classification, detection, collection, and reporting of harms create unique challenges that take time, expertise, and resources to yield valid summaries of harms. Investigators of primary studies should be mindful of these issues as their implementation can greatly impact subsequent evidence synthesis. To avoid tokenism and incorrect conclusions for harms, systematic reviewers should carefully consider how to handle each of these aspects if they plan to draw conclusions about trade-offs between benefits and harms. More systematic reviews are needed that focus exclusively on harms, include appropriate study designs, populations, and data sources, and that apply valid methods for synthesis.

What is new?

• Valid and reliable syntheses of evidence about harms requires different types of data, and different methods for synthesis, compared with evidence about benefits.

• The terminology, classification, detection, collection, and reporting of harms create unique challenges which take time, training and expertise, and resources to navigate.

• Systematic reviewers should carefully consider how to handle each of these aspects if they plan to synthesize harms or to draw conclusions about trade-offs between benefits and harms.

• In systematic reviews that focus on potential intervention benefits, methods to identify and synthesize evidence about harms are often tokenistic.

• Many interventions (e.g., drugs) are used for multiple indications (i.e., health problems), and harms might be related to interventions rather than indications. Consequently, systematic reviews focused exclusively on harms might be the most comprehensive and valid approach to synthesize evidence across indications using multiple study designs and data sources.