Biases in reporting of adverse effects in clinical trials, and potential impact on safety assessments in systematic reviews and therapy guidelines

Abstract

Background

Clinical trials are an important source of adverse effects data, including analyses in systematic reviews and recommendations in therapy guidelines. Trial publication bias may have profound effects on safety perceptions. This MiniReview presents and discusses biases in reporting of safety data in clinical trials and the implications for systematic reviews and guidelines.

Objectives

The objectives of this work are to analyse risk of gastrointestinal bleeding in systemic corticosteroid trials and to assess adverse effects reporting in a fluoxetine trial in depression (Treatment for Adolescents With Depression Study [TADS]) and descriptions of adverse effects in adolescent depression therapy guidelines.

Methods

We performed literature reviews and descriptive analyse of clinical trials with corticosteroids, and publications from the TADS trial. Risk of gastrointestinal bleeding from corticosteroids was analysed by meta‐analysis.

Findings

Gastrointestinal bleeding definitions varied considerably between trials. The incidence was significantly increased in hospitalized, but not in ambulant, patients compared to placebo. We identified several biases concerning TADS safety reporting, including severity thresholds and nonpublication of most adverse effects data beyond the initial 12 weeks. Therapy guidelines on adolescent depression mentioned suicidality risk, but many failed to mention other adverse effects.

Conclusions

We identified several pitfalls in adverse effects reporting in clinical trials. These include heterogeneous disease definitions, reporting thresholds, and incomplete reporting. Trial bias may have great impact on risk assessments in systematic reviews and meta‐analyses.

1. INTRODUCTION

Despite the popular belief that clinical trials will identify risks associated with new medications, flaws and biases in reporting adverse effects in clinical trial publications have been identified, in particular for antidepressant medications. ¹ , ² , ³ , ⁴ , ⁵ , ⁶ , ⁷ , ⁸ , ⁹ In several cases, reassessments of data have resulted in considerably increased risk estimates for serious adverse effects. ³ , ⁴ , ¹⁰

In this MiniReview, we give an overview of biases and weaknesses in the reporting of adverse effects in clinical trials, exemplified by our own research. We discuss how trial protocols and performance may influence identification and reporting of adverse effects. Trial publications are a major source of adverse event data and are essential for systematic reviews on drug safety. ¹¹ , ¹² Biases in trial publications will be continued along the evidence chain and have an impact on risk–benefit beliefs. For systematic reviews and therapy guidelines, there are no recommendations on the extent of safety information.

1.1. Identification and reporting of adverse effects in clinical trials

Randomized clinical trials (RCTs) are central to drug development, as randomization is assumed to offer a nonbiased comparison of treatment and control groups for both efficacy and risk. Consequently, RCTs have a high ranking in the hierarchy of evidence. ⁵ There are, however, several sources of bias throughout the complicated process of performing an RCT. These include study design, patient monitoring, data analysis, and manuscript preparation (Table 1).

TABLE 1.

Aspects of adverse effects reporting in assessed trials

Trial phase	Examples of variables	Corticosteroid trials	TADS
Planning	Patient group Inclusion criteria Exclusion criteria Study duration	Different diseases and all ages Different study durations Exclusion criterion ongoing or previous peptic ulcer applied in some studies Varying study duration	Multiple exclusion criteria, primarily psychiatric diseases Double blind 12 weeks Duration max. 88 weeks.
	Medication and dose Control group	Different corticosteroids and doses	Adjunctive treatment allowed
	Specified adverse effects All adverse effects	Not specified in many studies	Adverse event criteria threshold
Data collection	General or specific examinations Laboratory tests Questionnaires or interviews Doctor or patient reporting	Differences in monitoring for adverse effects	Interview setting with parents and assessments by investigators
	Criteria Severity thresholds	Heterogeneity in definitions and severity thresholds of gastrointestinal bleeding	Severity threshold Varying criteria for mania diagnosing
Assessments	Grouping of adverse effects Classification system Translation to medical codes	Different terminology	Ambiguous terminology for some symptoms Not considering symptoms from discontinuation
	Criteria for causality Assessment of causality	Not always described	Suicidality assessment described in study manual. Reanalysis of causality on suicidal events. Variations in assessment of mania
	Intention‐to‐treat Per protocol As treated	Zero events in several studies	ITT analysis possibly unsuitable due to supplemental therapy ITT underestimating true AE risk?
Publication	Most frequent adverse effects Most serious adverse effects Adverse effects in specific organs	Specific information missing in many studies	Several publications Focus on suicidality Not published all adverse effects for entire trial time
	Absolute numbers Relative numbers General statements	General statements in some studies Both absolute and relative numbers reported	Varying between adverse effects, publications and time periods. Suicidality: Absolute numbers, mean scores, score changes, proportion of patients over threshold values

1.1.1. Planning and collection of data

Most clinical trials are performed to assess treatment efficacy. Incidence of adverse effects is generally not a primary outcome, and trials are not powered to detect adverse effects unless very common. ¹³ , ¹⁴ A priori definitions of adverse effects is unrealistic due to the large number of possible adverse reactions. Consequently, few adverse effects are predefined with regard to severity criteria, diagnosis, identification, and classification. ⁵ , ⁶ , ¹³ Procedures for safety monitoring may vary considerably between trials and include general or specific physical examinations, laboratory tests, questionnaires, patient interviews, checklists, or application of diverse scoring tools. ¹⁵ Safety findings will largely depend on procedures for identifying adverse effects. Reporting will be subject to individual judgement, inaccuracies, and spontaneity if definitions are missing, probably with increased risk in multicentre, multinational trials with greater numbers of investigators. In addition, risks may be affected by dosages, treatment duration, and use of concomitant medications, such as use of additional antidepressants in antidepressant trials. Patient selection and exclusion criteria may reduce the risk of adverse effects in trial patients compared to real‐life populations. ⁵ , ¹⁴

In trials where adverse effects have been defined as a primary or secondary outcome, and randomization, blinding, patient monitoring, data collection, events, assessment methods, and analyses are described and fully reported, the risk of bias and errors will clearly be reduced.

Even for approved indications and patient groups, safety data may be surprisingly limited, as illustrated by selective serotonin reuptake inhibitors (SSRIs) for depression in children and adolescents. For the most extensively studied SSRI, fluoxetine, the pivotal trial is the “Treatment for Adolescents With Depression Study (TADS)” where 216 of 439 patients received fluoxetine. ¹⁶ , ¹⁷

1.1.2. Assessment and data registration

The chain of events, from a patient experience of an adverse effect until the event has been registered in the study files, offer opportunities for individual judgements. Basically, events must be acknowledged as possible adverse effects and not dismissed as chance findings or caused by the condition being treated (e.g., suicidality and depression). Reanalysis of primary, individual patient data from a paroxetine and imipramine trial caused new cases of serious adverse effects to be identified. ⁴ For antidepressants, new criteria and reassessment of suicidality cases resulted in identification of new cases and elimination of previously reported cases. ¹⁸ For the antidiabetic drug rosiglitazone, reanalysis of individual patient data, as opposed to summary level data, identified additional cases of myocardial infarction, and the drug was withdrawn several years after marketing. ¹⁰

Events (symptoms, verbal descriptions, laboratory findings) must be translated into diagnostic codes, using complex medical vocabularies. Interpretations and judgements during this process may give rise to variations and misclassifications and have been found to vary between individuals. ¹⁹ Coding omissions, where not all events are coded, have been identified through analysis of individual patient data. ⁴ Choice of classification terms may lessen the apparent severity of an event, as exemplified by using the coding term “emotional lability” in cases of suicidal behaviour. ⁴ Trials of checkpoint inhibitors in cancer treatment used 24 different terms for the adverse effect of colitis, and trials of orlistat in weight reduction treatment used 11 different terms for describing diarrhoea. ²⁰ , ²¹ Other examples include difficulties in classifying cases of self‐harm for antidepressants. ¹⁹

In some cases, adverse effects that originally were recorded quantitatively, for example, liver enzyme levels or symptom scores, are converted to dichotomous values (criteria fulfilled or not). ⁶ , ¹⁴ Consequently, the threshold value for qualifying for an adverse event will have a large impact on the number of cases.

1.1.3. Publication

To improve the reporting of adverse effects in clinical trial publications, the CONSORT recommendations (Consolidated Standards of Reporting Trials) were extended with more detailed recommendations for harms reporting (CONSORT Harms) in 2004 in order to improve quality and reduce risk of bias and errors. ²² Despite the CONSORT Harms recommendations, deviations to complete safety reporting have been identified. Many trials are not published, or safety data are not presented in full. ¹ , ⁴ , ⁷ , ²³ , ²⁴ Some papers merely state that no major adverse effects were observed, fail to mention serious adverse effects, or limit reporting to serious or severe reactions, adverse effects with incidence above a threshold value, or statistically significant risk increases. ¹ , ⁴ , ⁶ , ⁷ , ²¹ , ²⁵

Reporting of adverse effects data is customarily summed up by organ systems, with varying levels of subspecifications. The level of grouping adverse effects for analyses is often not specified a priori, ²⁶ and each alternative carry an inherent risk of errors. Grouping by major terms, for example, “gastrointestinal disorders,” will increase group size and statistical power. However, grouping may easily combine highly disparate events with different aetiologies, and the increase in numbers may mask cases of rare but significant adverse reactions. Analysis by single, specific diagnoses will provide more information of certain risks but will be subject to errors and differences in classifications, as in “emotional lability” versus “suicidality,” and may render categories too narrow to identify relevant risks. ¹⁸

1.2. Adverse effects in systematic reviews

Systematic reviews aim to answer specific research questions through comprehensive analyses of relevant, high‐quality trials. ²⁷ Most reviews focus on efficacy, and many have a limited mention of risks, or fail to address treatment risk altogether. ²⁸ , ²⁹ , ³⁰ , ³¹ Recommendations for harms reporting in systematic reviews, the PRISMA Harms checklist, were published in 2016, ³² but the impact is still unknown.

Systematic reviews of risk are at risk of bias for many reasons. In contrast to high quality efficacy data, risk assessments are generally not based on primary outcomes in multiple trials, but on poorer quality heterogeneous data on secondary outcomes across trials, or spontaneous reports of varying quality. RCTs generally do not have the size and statistical power to identify or draw conclusions on potential risks. Trials may differ in their design, monitoring, and descriptions of risk, thereby affecting the quality of risk assessments. ²⁹ , ³⁰ , ³³ Systematic reviews on identical topics may reach different conclusions, as illustrated by the question of corticosteroid‐induced gastrointestinal bleeding, which has been debated for decades. Some researchers have found the risk to be significantly increased, ³⁴ while others have not. ³⁵ , ³⁶ This may be the reason why databases and handbooks describe the risk association either as unlikely or weak ³⁷ or as increased. ³⁸

Systematic reviews on risk do not necessarily address all relevant adverse effects, as illustrated by SSRI safety reviews. Due to warnings on suicidality risk, ³⁹ several reviews and meta‐analyses have focused on suicidality, ⁴⁰ , ⁴¹ without addressing other adverse effects.

1.3. Adverse effects in clinical guidelines

Clinical therapy guidelines are expected to assess therapy benefits and risks, based on systematic literature reviews. ¹² , ⁴² There is no standard for risk descriptions of adverse effects of different treatment options in clinical guidelines and there is little research on the subject. The quality instrument for evaluating therapy guidelines, the AGREE II tool, does not comment on level of risk information. ⁴² Clinical guidelines are known to focus on benefits and be liable to biases, including conflicts of interest. ⁴³ It is not known to what extent safety concerns and adverse effects information is described in guidelines on antidepressant therapy in children and adolescents.

In our research, we aimed to assess publications of clinical trials with regard to reporting of adverse effects. The evaluations were performed in two model areas: Risk of gastrointestinal bleeding or perforation in trials of systemic corticosteroid therapy and descriptions of adverse effects from fluoxetine in a pivotal trial (the TADS study) in children and adolescents with depression. We also aimed to assess clinical therapy guidelines on depression in children and adolescents for their information on adverse effects.

2. METHODS

For assessment of adverse effects reporting in clinical trials, we performed literature searches for trials of corticosteroids and publications arising from the TADS antidepressant trial, as described elsewhere. ⁴⁴ , ⁴⁵ Corticosteroid trial publications were analysed with regard to inclusion and exclusion criteria, extent and indication for corticosteroid use, concomitant medications, ambulant or hospital treatment, and definitions or criteria for gastrointestinal bleeding. Risk of gastrointestinal bleeding or perforation was analysed through meta‐analysis. ⁴⁴ Fulfilment of criteria for harms reporting ²² were analysed quantitatively. ²⁶

TADS trial publications were analysed descriptively with regard to information about adverse effects.

Therapy guidelines on treatment of depression in children and adolescents were identified through literature searches in PubMed, EMBASE, guideline collections, and manual searches, as described elsewhere. ⁴⁶ Presentation of adverse effect data and risk profiles were analysed descriptively.

3. RESULTS

3.1. Descriptions of gastrointestinal bleeding or perforation in corticosteroid trials

To assess how clinical trials had addressed the question of gastrointestinal bleeding from corticosteroids, we analysed the reporting of this adverse reaction in 159 published clinical trials of corticosteroid therapy. ⁴⁴ Most trials addressed treatment efficacy. Monitoring procedures, and definitions of gastrointestinal bleeding or perforation, varied considerably between trials, from general, unspecific examinations to blood tests and faecal analysis. Overall, 37 terms had been used to describe gastrointestinal bleeding. Criteria varied greatly with regard to disease severity, ranging from “guaiac‐positive aspirate” to “hematemesis” or “melena requiring transfusion.” The variations and potential biases identified in the corticosteroid trials are described in Table 1. Overall, we found an increased risk of gastrointestinal bleeding or perforation of 40% (OR 1.43, 95% CI 1.22 to 1.66) in patients treated with systemic corticosteroids. The risk increase was statistically significant for hospitalized, and presumably sicker, patients. Few cases were reported for ambulant patients.

Assessment of publications according to the CONSORT Harms criteria showed that many studies did not state an intention of identifying adverse effects or describe plans for risk presentation or analysis. ²⁶ We found the criteria ambiguous and unsuitable for retrospective trial assessment for quality, as publications that do not fulfil all criteria might still present relevant safety data despite low assessment scores.

3.2. Descriptions of adverse effects in the TADS trial

To assess how a single, pivotal antidepressant trial monitored patients and reported on safety results, we identified and analysed TADS trial publications. In the TADS trial, patients underwent several interviews and assessments through screening tools, which are described in the trial protocol. ⁴⁷ These include Affective Disorder Screening (ADS), Clinical Global Impressions (CGI), and Children's Global Assessment Scale (CGAS), with guidelines for rating major depression, suicidality, mania, and overall functioning, but do not provide definitions or specific monitoring of other potential adverse effects. In addition, patients and parents were asked about any new health problems at assessments, but the trial protocol specified that only adverse events that fulfilled threshold criteria would be recorded in an adverse event form. For adverse events that were recorded in an adverse events form, the therapist would indicate an opinion with regard to causality.

We identified 48 publications with data from the TADS trial, ⁴⁵ of which eight presented adverse effects data to some extent. All eight publications described cases of suicidal behaviour. The TADS study was performed as a randomized, double‐blinded, placebo‐controlled trial for the first 12 weeks, but many patients continued on open, noncontrolled treatment for up to 88 weeks. Data on adverse effects other than suicidality were only published for the controlled, double‐blinded 12‐week trial phase I. The variations and potential biases identified in the TADS trial are described in Table 1. We did not identify any publications describing other psychiatric, or somatic, adverse effects occurring after the first 12 weeks. Comparison of adverse effects reported in the TADS trial and the list of fluoxetine adverse effects in the Summary of Product Characteristics (SmPC) ⁴⁸ showed that many well‐known risks were not mentioned in publications from the TADS trial.

3.3. Descriptions of adverse effects in guidelines on antidepressants in children and adolescents

Nineteen clinical guidelines on treatment of depression in children and adolescents were assessed and found to vary considerably with regard to risk descriptions. ⁴⁶ All guidelines mentioned risk of suicidality, and many described other psychiatric reactions, but most guidelines failed to describe somatic adverse effects.

4. DISCUSSION

4.1. Reporting of adverse effects in clinical trials

4.1.1. Definitions of adverse effects

The lack of clear and uniform definitions of what constitutes an adverse effect, and the large variations in monitoring methods, will result in the summing up of different entities. The lack of common definitions and monitoring methods for adverse effects is likely to be valid across all medical areas and is potentially a large source of bias. ⁵ , ¹³ As exemplified by the diagnosis of gastrointestinal bleeding, monitoring for occult faecal blood must be expected to identify more cases than counting patients needing blood transfusion.

4.1.2. Limitations, thresholds, and filters

The use of reporting thresholds will have a profound effect on the number of cases registered as an adverse effect. The TADS trial illustrates this point, as the trial protocol describes thresholds and limitations on adverse effects reporting that must have influenced results. A major threshold that has received little attention in the literature citing the trial was the fact that adverse events would not be recorded unless they caused a clinically significant interference with functioning, required medical attention, or caused a need to take medication. ¹⁷

4.1.3. Salami publications

The 48 publications from the TADS trial illustrate the problem of salami publications, ⁴⁹ that is, publications of several papers from a single trial. The sheer number of publications may easily give readers the impression of extensive research, while in reality the published adverse effects data cover only 216 patients treated with fluoxetine for 12 weeks.

4.1.4. Statistical analysis of adverse outcomes

Choice of statistical methodology may influence risk estimates in clinical trials. In many trials, reporting is descriptive, giving the number of cases in treatment and control groups (incidence rate). For statistical analyses, recommendations state that adverse effects should be analysed as intention‐to‐treat (ITT). ²² Potential biases, and erroneous risk estimates inherent to ITT analyses of adverse effect data, have been little discussed. If many patients leave the study early, or receive additional treatment that differ from their assigned group, ITT analysis will tend to reduce group differences. ¹⁴ The adverse effects incidence will be underestimated if the denominator includes nonadherent patients, as was the case in the TADS trial. Conversely, risk estimates patients will exclude events in patients who withdrew from treatment due to adverse events.

4.1.5. What is missing in harms reporting?

As described in Table 1, risks of bias may arise from a number of causes during performance and publication of clinical trials. For adverse effects, these include nonpublication, skewed presentations, and lack of statistical strength. Bias may be expected if study design is suboptimal, patient monitoring not clearly defined, outcomes diffuse, and reporting selective. For readers, absence of relevant data may be difficult to notice unless an article is subjected to close scrutiny. Publication of safety data according to the CONSORT Harms recommendations ²² will potentially improve descriptions of trial performance with regard to any specific adverse reactions being addressed, monitoring methods, disease definitions, data collection, withdrawals, and risk analysis. It will not, however, reduce bias due to heterogeneity in patient groups, disease definitions, or monitoring methods.

Adverse effects are usually presented as number of cases, or percent incidence, for each adverse effect, by organ systems. There are few descriptions of event severity, ⁵ latency and duration of adverse effects, ⁵ , ²¹ and overall impact on patients. Obviously, a light and passing headache will differ from a debilitating, enduring headache in terms of acceptability. Furthermore, the present descriptive methods do not assess benefit and harms together for each patient, and current methods do not assess skewedness in distribution of benefits and risks within the patient groups.

There is currently great interest in the possibilities of in‐depth analysis in individual patient data from trials, but issues regarding data availability and methodology are still pending.

4.2. Adverse effects in systematic reviews

Systematic reviews will include heterogeneous data, given the differences in included trials. A systematic review addressing adverse effects of a specific medication may appear to be conclusive, but biases and limitations in the underlying data may not be taken sufficiently into account by researchers and readers. There are no clear recommendations as to when heterogeneity is large enough to preclude performance of systematic reviews. ⁵ The Cochrane Handbook for performance of systematic reviews discusses several sources of heterogeneity and states that “review authors must recognize the possibility of poor case definition, inadequate monitoring and incomplete reporting when synthesizing data.” ⁵⁰ This raises the question of whether systematic reviews can be more misleading than valid. Heterogeneity may be the reason for different conclusions in systematic reviews. In the case of corticosteroid‐induced gastrointestinal bleeding, systematic reviews, including our own, have included different trial publications, used different inclusion criteria and limitations, and included trials with different monitoring and definitions of gastrointestinal bleeding. Even if review authors describe selection criteria and biases, the take‐home message from a systematic review will probably be limited to the main findings for most readers.

4.3. Descriptions of adverse effects in therapy guidelines

Our assessment of therapy guidelines on depression in children and adolescents showed that many guidelines mentioned only selected adverse effects and failed to mention several common risks. Analysis of therapy guidelines is challenging, partly due to identification and collection difficulties. Any professional body or organization is free to develop guidelines, and they are in many cases published locally, outside peer‐reviewed journals. The extent of adverse effects descriptions in therapy guidelines has been little studied and is, at present, to be decided by guideline authors, with no clear expectations from society or readers as to the level of information.

5. CONCLUSION

Our findings confirm the results of other researchers and show considerable pitfalls in identification and reporting of adverse effects in clinical trials. Some biases are caused by methods choices including study design and patient monitoring, and others are caused by use of thresholds, filters, and selective reporting. In corticosteroid trials, the main issue was the highly heterogeneous definitions and diagnostic criteria for gastrointestinal bleeding, as well as heterogeneous patient groups with regard to underlying disease severity, risk factors, and concomitant medications. In the TADS trial, the main issue was the reporting threshold with regard to event severity, and failure to publish adverse effects data beyond the initial 12 weeks of treatment for other adverse effects than suicidality. Trial publication biases have potentially great implications for attempts to assess adverse effects risk in systematic reviews and meta‐analyses based on trial publications.

CONFLICTS OF INTEREST

The authors declare no conflict of interest.

Westergren T, Narum S, Klemp M. Biases in reporting of adverse effects in clinical trials, and potential impact on safety assessments in systematic reviews and therapy guidelines. Basic Clin Pharmacol Toxicol. 2022;131(6):465‐473. doi: 10.1111/bcpt.13791