Potential Implicit Bias in Attribution of Adverse Events in Randomized Controlled Trials in Cystic Fibrosis

Abstract

Introduction:

Although work to date in cystic fibrosis (CF) has elucidated frequencies and characteristics of adverse events, the accuracy of attribution of relatedness to study drug by investigators has not been assessed. We aimed to determine whether there was an association of attribution by group allocation in CF clinical trials.

Methods:

We conducted a secondary analysis from 4 CF trials of all persons who experienced an AE. Our primary outcome was the odds of an AE related to active study drug and predictor of interest was the treatment allocation. We constructed a multivariable generalized estimating equation model allowing for repeated measures.

Results:

A total of 785 subjects (47.5% female, mean age 12 years) had 11,974 AEs, of which 430 were serious. AE attribution was greater with receipt of active study drug as compared to placebo but did not reach statistical significance (OR 1.38, 95% CI 0.98-1.82). Significantly associated factors included female sex (OR 0.58, 95% 0.39-0.87), age (OR 1.24, 95% CI 1.06-1.46) and baseline lung function (per 10%, OR 1.16, 95% CI 1.05-1.28).

Conclusion:

In our large study, there was a non-significant but greater odds of AE attribution (a key element of clinical trial reporting) to active study drug based on assigned treatment to study drug or control which suggests that there is a trend in physicians to attribute blinded safety data to the active drug. Interestingly, females were less likely to have AE attribution to study drug and warrants further work in development and validation of monitoring guidelines and processes.

Introduction

Clinical trials are considered a gold standard for evaluating interventional efficacy and safety in populations. Although cystic fibrosis (CF) is an inherited fatal disease that has orphan disease status, clinical trials have greatly improved our understanding of disease management in this population and have led to significant improvements in survival. Clinical trial conduct is highly rigorous and methods to evaluate efficacy are well established, however efforts are needed to ensure the consistent monitoring of harm outcomes across age and sex- are similarly robust^1–4.

Adverse events (AEs) are an important safety metric to capture in therapeutic clinical trials. An AE is defined as any untoward medical occurrence associated with the use of drug in a research participant and may be indicated by laboratory or physical measurements⁵. AEs are reviewed by medical and safety monitors and by data safety monitoring boards (DSMBs) during the conduct of a clinical trial, with the potential to stop a trial early for excess harm possible. With increasing awareness to better understand the events and experiences of harm in the clinical trial context and in relation to baseline characteristics in the population, some work has been done to evaluate the rates and nature of AEs in select CF clinical trials. Specifically, rates of respiratory AEs with inhaled therapies in the short and long-term were assessed with data recorded from multiple CF trials enabling a better understanding of expected occurrences of respiratory events in people with CF (PwCF)⁶. Additional work by our group has described the frequency and severity of AEs experienced in pediatric trials in CF⁷. As well, rates of biochemical abnormalities inclusive of liver function tests were evaluated to better understand variability and clinical implications during the conduct of clinical trials in CF^8,9.

Although the preceding studies have enabled an improved understanding of rates and characteristics of AEs in CF clinical trials, there is a paucity of data on the accuracy of AE coding. The act by site investigators and medical monitors to delineate a severity grade and relationship of AEs to an investigational agent remains subjective. Guidance documents outline AE grading and attribution criteria^10,11, but no formal validated guidelines or processes exist. Further, time requirements and complexities on AE delineation and attribution may negatively impact the accuracy of AE coding in clinical trials yet have a dramatic impact on AE reporting to regulatory bodies. The subjectivity needed to assess attribution in particular is at risk of introducing implicit bias. Serious AEs that are both attributed to study drug and unexpected (a SUSAR) demand rapid reporting and mandatory unblinding of study drug (placebo or treatment) based on current Food and Drug Administration regulations⁵. This process in and of itself could introduce bias in an ongoing trial though enhancing safety. We aimed to evaluate whether the attribution of an AE as related to active study drug is associated with treatment group allocation or other factors in CF clinical trials. We hypothesized that AE attribution would differ and be greater in the treatment group despite blinding and be influenced directly or indirectly by clinical or demographic factors¹².

Methods

Study population

A secondary analysis of four interventional randomized clinical trial (RCT) datasets conducted in the Cystic Fibrosis Foundation Therapeutic Development Network (CFF TDN) was conducted¹³. The datasets were from the EPIC Clinical (NCT00097773), AZ0001, AZ0004 (NCT00431964), and N-acetyl cysteine (NAC) (NCT00809094) trials. The data were anonymized and included in the CFF TDN archive with ethics approval from the Seattle Children’s Hospital, Seattle, Washington (ethics approval: PIROSTUDY15817).

Briefly, EPIC was a multicenter placebo-controlled RCT of 18 months duration in PwCF aged 1-12 years assigned to one of four regimens of cyclical versus culture response based anti-pseudomonal therapy (inhaled tobramycin combined with either oral ciprofloxacin or placebo) following initial isolation of Pseudomonas aeruginosa from the airways¹⁴. The trial concluded that inhaled tobramycin led to a high rate of Pseudomonas aeruginosa eradication regardless of treatment arm. AZ001 was a multicenter, double-blind, placebo-controlled RCT in persons aged 6 and older infected with P. aeruginosa to evaluate the effects of azithromycin use over a 168 day period on pulmonary function¹⁵. This trial demonstrated that oral azithromycin in P. aeruginosa newly infected CF patients led to reduced exacerbation rate, improved lung function and decreased inflammation. AZ004, similar to AZ001 in design and duration, was a multicenter double-blind placebo-controlled RCT to assess the effects of azithromycin on pulmonary outcomes in PwCF aged 6-18 years without P. aeruginosa infection¹⁶. It demonstrated that oral azithromycin decreased exacerbation rate without improving lung function. Lastly, the phase II NAC RCT used a multicenter placebo-controlled double-blind proof of concept approach over a 24 week period to assess the effects of oral N-Acetylcysteine (NAC) on pulmonary inflammation and function in PwCF¹⁷. They noted a small significant improvement in lung function associated with oral NAC use. The EPIC and AZ004 included children and the N-acetyl cysteine and AZ001 trials included both adult and children.

All participants who experienced at least one AE were included for analysis encompassing the vast majority of all participants in the trials. AEs in each trial were categorized based on the Medical Dictionary for Regulatory Activities (MedDRA) coding classification, batched into common descriptors where appropriate and grouped by organ class. For the purposes of this analysis, each AE was considered discrete with equal weighting given regardless of the nature of the event. Sex was defined using biological terms of male or female and gender was utilized as a reference to the social construct inclusive of identity, roles and behaviors.

Statistical analysis

The clinical trial cohorts were descriptively summarized. Between group differences in AE frequency were calculated with the Chi-squared test. The primary outcome of interest was the physicians’ determination of the adverse event as related or unrelated to study drug. Our exposure of interest was the receipt of active study drug versus placebo in the clinical trial. For the primary outcome, we constructed a generalized estimating equation (GEE) model with a logit link allowing for repeated measures by subject. Relevant demographic and clinical confounders of sex, age (years), genotype (F508 status), body mass index (BMI), baseline lung function (forced expiratory volume in 1 second % predicted (FEV₁pp), and CF related diabetes were included a priori into the multivariable models. We pre-specified two-way interactions to test in our model. To ensure parsimonious selection, three variables were included: sex, age and lung function. A GEE model similar to above for the subgroup who experienced a serious AE was also constructed. We conducted a sensitivity analysis to account for within clinical trial effects by adjusting for each clinical trial as a stratification variable. All outcomes were pre-determined at a two-sided α of 0.05 and analyses were conducted using STATA 17 (College Station, Texas).

Results

A total of 785 out of 819 participants (96%) experienced 11,974 AEs across the four RCTs. Females compared to males experienced a greater number of AEs overall and in all but the AZ004 trial. A total of 34 participants were not included for analysis as they did not experience any AEs; these subjects were most frequently from the AZ004 trial (18/34, 53%), a study with patients with very mild CF based on inclusion and exclusion criteria. Of all AEs, 916 (7.7%) were deemed to be related to the drug and attribution was greater in the active study drug group compared to placebo/control group (8.5% vs 6.7%, p<0.001). There were 430 (3.6%) serious AEs that occurred across the studies. The overall cohort was 47.5% female with a mean age of 12 years and mild lung disease (mean FEV1pp of 86.2%) (Table 1). Of the total cohort, 46% were homozygous for ΔF508 and 34% were heterozygous for the same. The two trials including adult participants – AZ001 and NAC – had a lower mean lung function at baseline and greater prevalence of CF related diabetes (Table 1).

Table 1:

Participant characteristics in the cohort and by clinical trial

Parameter	EPIC	AZ001	AZ004	Oral N-acetyl cysteine	Overall
Number of subjects	301	179	242	63	785
Study duration	18 months	168 days	168 days	24 weeks	n/a
Number of AEs	7,151	2,353	1,772	698	11,974
Number of AEs (%) in females	3,711 (51.9%)	1,198 (50.9%)	828 (46.7%)	385 (55.2%)	6,122 (51.1%)
Mean Age in years (range)	5.3 (1-13)	20.0 (6-50)	10.6 (6-18)	26.4 (9-59)	12.0
% Female	50.5%	48.0%	42.2%	52.4%	47.5%
% Genotype
Homozygous ΔF508	49.5%	39.7%	45.9%	47.6%	46.0%
Heterozygous ΔF508	37.9%	31.3%	30.2%	34.9%	33.8%
Other	12.6%	29.1%	24.0%	17.5%	20.3%
Mean FEV1 % Predicted	96.2%	69.1%	98.4%	63.1%	86.2%
Mean Body Mass Index (kg/m2)	16.5	20.1	17.8	20.7	18.2
% Pancreatic Insufficiency	46.5%	28.5%	39.7%	41.3%	39.9%
% Diabetes	1.7%	12.9%	2.5%	25.4%	6.4%

For the primary outcome, the unadjusted odds ratio (OR) of attribution of the AE was associated with treatment assignment and greater for active study drug compared to placebo (OR 1.34 (95% CI 1.00 – 1.79, p = 0.05). In a multivariable model adjusting for baseline characteristics of the trial participants, the OR for study drug compared to placebo was slightly greater at 1.36 (95% CI, 0.92-2.01) but no longer reached statistical significance (Table 2). Factors that were significantly associated with greater odds of attribution of the AE to drug included age (per 5 years, OR 1.24, 95% CI 1.06-1.46), and lung function (per 10%, OR 1.16, 95% CI 1.05-1.28). Female sex was associated with a significantly lesser odds of AE attribution to drug (OR 0.58, 95% 0.39-0.87). We also ran models where only clinical or demographic covariates respectively were included and there was no significant difference for the primary predictor for AE attribution (data not shown). Additionally, we ran a stratified analysis to account for study specific effects and found similar results (Table 3).

Table 2:

Multivariable model of AE association with drug allocation

Covariate	Odds Ratio	95% CI
Receiving study drug vs placebo	1.36	0.92-2.01
Age (per 5 years)	1.24	1.06-1.46
Gender (female)	0.58	0.39-0.87
FEV1 % Predicted (per 10%)	1.16	1.05-1.28
Genotype
Homozygous ΔF508	Ref	Ref
Heterozygous ΔF508	1.24	0.80-1.92
Other	1.61	1.00-2.57
BMI (per 1 point change)	0.98	0.92-1.05
Diabetes	1.00	0.45-2.24
Pancreatic Insufficiency	1.30	0.88-1.91

Table 3:

Multivariable model of AE association with drug allocation accounting for trial effects

Covariate	Odds Ratio	95% CI
Receiving study drug vs placebo	1.38	0.96-1.99
Study
AZ001 (ref)
AZ004	0.73	0.41-1.30
EPIC	0.88	0.48-1.63
NAC	1.26	0.64-2.44
Age (per 5 years)	1.21	1.04-1.40
Gender (female)	0.59	0.40-0.87
FEV1 % Predicted	1.18	1.07-1.31
Genotype
Homozygous ΔF508	Ref	Ref
Heterozygous ΔF508	1.19	0.78-1.84
Other	1.54	0.98-2.42
BMI (per 1 point change)	0.97	0.90-1.04
Diabetes	1.00	0.45-2.24
Pancreatic Insufficiency	1.30	0.88-1.91

In our subgroup analysis of serious AEs, active study drug again suggested a greater odds of AE attribution but did not reach statistical significance in a multivariable model (OR 2.66, 95% CI 0.66 – 10.78). Based on our study hypothesis we analyzed interactions between key demographic and clinical variables sex and age as well as lung function in the overall AE population, but these were not significant.

Discussion

In our analysis of large CF clinical trial datasets, most participants experienced at least one AE consistent with prior observations^7,8. There was a greater frequency of AEs in females compared to males overall and in three of four trials which is consistent with data in non-CF populations with a suggestion that this may relate to sex differences in pharmacokinetics¹⁸. In our multivariable model, we identified that attribution of an AE to active drug receipt was greater but not statistically significant compared to a participant assigned to placebo after adjustment for baseline clinical characteristics, although there was a suggestion of concordance between the models. This study provides some reassurance that physicians are more likely to attribute AEs to active study drug versus placebo when an individual is actually randomized to active study drug, even though the physicians (investigators) are blinded to this information. Whether or not the AE is actually related to study drug may be discordant to a physician’s attribution, but in this case, although not definitive, our results suggest that a physician may be possibly erring on the side of caution.

When we examined other factors associated with AE attribution, lung function and age were associated with increased odds of attribution to study drug independent of study drug receipt. Pooled data from clinical trials have previously demonstrated that PwCF have relatively high AE rates even in the placebo arm and appear to be greater in short-term versus long-term studies⁶, creating complexity in our ability to interpret and correctly attribute AEs to receipt of a study drug. With lung function, it is possible that an AE in a person with milder disease would be more likely to be attributed to study drug as they may have fewer symptoms at baseline. Increased age also positively associated with AE attribution to active study drug and may be multifactorial be it related to improved health based on the specific age group (younger patients have improved clinical status) or ability to communicate about symptoms at an older age not relying on surrogates such as parents and care givers for small children. As our trials encompassed both children and adults, it would be interesting to repeat this evaluation in children and adults separately with larger datasets to assess if age remained associated.

The ‘CF gender gap’ has described the phenomenon of survival in men exceeding that in women and the converse for morbidity in the last decades despite advancements in therapeutics^19–22. In a clinical trial context, gender differences have been noted in AE rates in clinical trials of persons with Human Immunodeficiency Virus infection²³. Our analysis demonstrated that females had significantly lower odds of having AE attribution to study drug and is consistent with differences identified outside of the trial context in disease management and lived experience of females with CF^24–26 or other diseases^27,28. When examined separately by trial, results were similar for females and the lack of statistical significance in three of the trials may have been due to underpowering. The differences in attribution of AEs identified by clinical and demographic factors and particularly so by sex highlight potential for implicit biases and subjectivity in clinical trial assessment and decision making that may lead to an incomplete understanding of health and disease. Perhaps more pressing, recent reviews of drug and vaccine trials have highlighted that there was a significant lack of sex-specific reporting of data for outcomes²⁹ or adverse events³⁰. Although not specifically examined here, sex, gender and their inter-relations need to be considered in all aspects of trial design and conduct as well as safety monitoring to minimize the harm and move towards equity and advancement^31,32. The impact of sex on AE reporting could diminish our understanding of ongoing safety issues during the conduct of a clinical trial. Eventually AEs will be described in relation to the differences noted in the final study population after unblinding, but this could be too late for demonstrating important safety issues.

Our study has limitations that must be considered. We included both pediatric and adult participants, but trial populations may not reflect all PwCF and may not be generalizable to those with more advanced disease. Although the NAC trial had considerably less subjects, our agnostic approach and use of all blinded interventional trials likely minimized a scenario where one trial skewed the results and increased our power to interpret the results. Our use of analyses stratified on study also mitigated this potential concern. There were some differences in comorbidities between the trials but as these were not significantly associated with AE attribution and were used as adjustment covariates in the model. Thus, comorbidities are less likely to have affected the results. It is possible that some may have participated in more than one trial, but this was not possible to account for as the data were anonymized. The study dataset did not include information relating to participant race or ethnicity or investigator demographic characteristics and their concordance with participants which could also have influenced AE attribution. As we analyzed persons within their treatment assignment groups, some of the confounding effects may have been mitigated. We assumed an equal distribution of AEs which may have underestimated the true population variance. However, the high rate of AEs reported across CF clinical trials and use of a binomial distribution increased our confidence in our results. To note, the trials included in the analysis used a double-blind approach with exception of the EPIC trial where participants and personnel were blinded to oral therapy but not to the cyclical or culture based therapy assignment¹⁴.

Despite our limitations, this is one of the first studies to our knowledge to evaluate AE attribution by group allocation in a clinical trial context and identified potential biases that may exist based on clinical and demographic variables. Beyond AE rates, participant safety can be significantly impacted based on the ability to accurately classify AEs and manage them appropriately including discontinuing of a study drug or unblinding. Work is being done to advance how AEs are analyzed beyond the traditional display tables^2,33, and collation of multiple trials enables powered and robust evaluations of the same. Similar approaches by sponsors and regulators will need to be undertaken in redesigning classification systems of AEs and harmonization of these systems³⁴ such that the safety of vulnerable populations can be ensured. Our study also speaks to the immense potential of clinical trial registries which enable granular and powered evaluations of AEs, mitigating the challenges involved with accessing industry-sponsored trial datasets.

Conclusion

In summary, our study of multiple CF clinical trials identified that AE attribution was greater in participants receiving active study drug compared to placebo. Despite a lack of statistical significance, this provides some reassurance that even in the context of blinding, physicians tend to err on the side of caution when attributing AEs to active study drug. We also observed that AE attribution was significantly associated with female sex as well as lung function and age. This approach to evaluate for the potential for implicit bias in reporting of safety and AE attribution data can be readily applied to other settings. This data is highly relevant at the site, sponsor and regulator levels for training and approaches to both current and future trials in CF as well as in other disease spaces to optimize participant safety.