Leveraging analytics to assure quality during the Covid-19 pandemic - The COVACTA clinical study example

Abstract

The world has seen a shift in the ways of working during the Covid-19 pandemic. Routine activities performed at the clinical investigator sites (e.g. on-site audits) that are a part of Quality Assurance (QA) have not been feasible at this time. Analytics has played a huge role in contributing to our continued efforts of ensuring quality during the conduct of a clinical trial. Decisions driven through data, now more than ever, heavily contribute to the efficiency of QA activities. In this report, we share the approach we took to conduct QA activities for the COVACTA study (to treat Covid-19 pneumonia) by leveraging analytics.

1. Background

Compliance with the fundamental principles of good clinical practice (GCP) ensures the rights, safety and well-being of research subjects and ensures the integrity of clinical research data. Trial sponsors are required by the International Conference on Harmonization guidelines to implement and maintain quality assurance (QA) and quality control systems to achieve these objectives [1]. Traditional QA practices heavily rely on audits to detect sites or studies with quality issues [2]. Current site monitoring strategies, which rely on on-site visits with source data verification (SDV) and on risk-based approaches, are also attempting to mitigate the risk of occurrence of clinical quality issues [3,4].

The COVACTA study was a phase III, randomized, double-blind, placebo-controlled study to evaluate the safety and efficacy of tocilizumab in patients with severe Covid-19 pneumonia [5]. As COVACTA was being conducted under the circumstances of the Covid-19 pandemic that forbade the possibility of on-site monitoring and on-site investigator audits, an alternative approach to ensure the clinical quality of the study was required. Roche/Genentech Product Development Quality (PDQ) had implemented a set of data-driven solutions to complement and augment traditional QA approaches to improve the quality and oversight of GCP regulated activities [[6], [7], [8], [9], [10]]. These analytics had been developed since 2018 and the COVACTA study provided the opportunity to fully implement their use. We performed remote quality reviews, using the data collected in electronic systems at the site and at sponsors and applying descriptive and statistical analysis.

This was the first clinical trial sponsored by Roche/Genentech where quality oversight was performed 100% remotely using analytics. Future clinical trials, whether or not conducted under exceptional circumstances, could benefit from a similar approach. In this analytical report, we are sharing the insights gained from this experience and are proposing a strategy to assure clinical quality by leveraging analytics.

2. Methods

Our objective was to demonstrate that patient safety and the integrity of data collected for the primary endpoint were protected throughout the course of the COVACTA trial. To do so, analysis and quality reviews have been conducted on a daily basis. Quality reviews were performed by Quality Program Leads (QPLs), who have QA/GCP relevant subject matter expertise to support the study team. Of note, continuous reviews of protocol deviations were performed by QPLs, as this was a well-established process to identify potential quality signals. Roche/Genentech Quality Analytics and Insights (QA&I) was responsible for designing, executing and verifying the analysis described in this paper.

While the study was being initiated, QPLs and QA&I data analysts/scientists first reviewed the protocol [5] to define the key data elements that would serve as quality evidence.

2.1. Patient safety

The main areas of focus with respect to patient safety were assessing the risk for Adverse Events (AE) under-reporting (including Serious Adverse Events (SAE) and Adverse Events of Special Interests (AESI)) and ensuring patients had been dosed properly with tocilizumab. Individual investigator sites have been monitored for potential AE under-reporting, using descriptive analytics, complemented by a machine-learning approach [6,[7], [10]]. Due to the uniqueness of this study (short timelines, high mortality) the statistical models trained on past study data could not always be applied. We therefore relied on the monitoring of ongoing AE reporting rates and defined a minimum AE per visit rate of 0.05. This rate was the minimal AE reporting level based on the observation of sites with continuous AE reporting over time (proportionally to their number of patients and visits).

Additionally, the following summary statistics were reviewed on ongoing basis:

• -Number of AE per site
• -Number of AE per site/number of patient visits on site
• -Number of AE per site/number of days on study for all patients on site
• -Number of AE per patient/number of days on study of patient
• -Time of entry of AE in the electronic Case Report Form (eCRF)
• -SAE/AESI reporting timelines

The following calculations using eCRF data were performed to verify for safe and correct dosing:

• -Correct dosage according to weight was administered
• -Subjects with deteriorating symptoms were allowed a second dose which should have been administered according to the protocol
• -Dose administration times were according to the protocol

2.2. Protecting the Primary endpoint

To provide quality evidence to demonstrate that the primary endpoints were protected, the following indicators were calculated using eCRF data. Outliers and patterns were then detected through thresholds defined by the protocol, data visualizations and/or summary statistics (e.g. Standard Deviation, Mean, Median, etc.) These quality indicators had been assessed by QPLs for potential risks:

• -Ordinal scale of clinical status

• -Mortality rate
• -Time to hospital discharge or “ready for discharge” (as evidenced by normal body temperature and respiratory rate, and stable oxygen saturation on ambient air or supplemental oxygen)
• -Schedule of assessments
• -Inclusion and exclusion criteria

The following indicators had also been calculated and were available for review, when needed:

• -Clinical status assessed using a 7-category ordinal scale
• -Time to clinical improvement (TTCI) defined as a National Early Warning Score 2 (NEWS2)
• -Incidence of mechanical ventilation
• -Ventilator-free days
• -Organ failure-free days to Day 28
• -Incidence of intensive care unit (ICU) stay # times in ICU
• -Duration of ICU stay ICU length of stay
• -Time to clinical failure, defined as the time to death, mechanical ventilation, ICU admission, or withdrawal (whichever occurs first)
• -Duration of supplemental oxygen

2.3. Data and tools

The data used to conduct the analysis came from the following sources: the eCRF and the Clinical Trial Management System. The infrastructure used to collect, store and analyze data was based on a file distributed system that had been in place for Roche/Genentech QA&I. Descriptive analysis had been performed using Microsoft Excel and R. For statistical analysis, we used R and Python. For visualization, we used Tableau. All these tools had been implemented and in use since 2018 at Roche/Genentech QA&I.

3. Discussion

As the COVACTA study was not formally closed at the time we submitted this paper, the figure and example were taken at a cut-off point during the study; hence they are provided here as illustrations and do not reflect the current quality status of the trial.

Quality reviews started two weeks after the first patient was enrolled and immediately began identifying potential quality issues. For example, one issue that had been detected early was low AE reporting rates. This was flagged and fed back by the QPLs to the study team for remediation. Additionally, potential root causes for low AE reporting rates were identified:

• -Given the circumstances of the Covid-19 pandemic and its impact on resources, site staff had entered data in the eCRF with delay
• -Overlap between AEs and Covid-19 symptoms, which were not to be reported as AEs

Due to the early detection of low AE reporting rates and diligent follow up by the site monitors, AEs collection improved, and AE under-reporting was resolved for the majority of the study sites (see Fig. 1).

Fig. 1.

Improvement of AE reporting rates for individual study sites after being identified through data analytics.

Other quality issues were identified almost in real time through the analysis described in the Methods section and were addressed immediately. As the study was not closed at the time of submission, we cannot disclose further details in this paper. All analysis performed by Roche/Genentech QA&I, their interpretation and related corrective actions have been summarized in a Quality Analytics Review Report. The conclusions will be available for use in the drug application package, should Roche/Genentech decide to file at the end of the clinical trial. The full Quality Analytics Review Report and the related analysis will be made available upon request for Health Authority Inspectors.

Our approach can enhance routine/on-site QA activities by the addition of the advanced analytics to provide evidence for the quality state of a clinical trial. The use of advanced analytics has been very well received by our internal stakeholders. First, business continuity for clinical quality assurance activities can be provided. Real-time detection and resolution of quality issues are enabled. Last, but certainly not least, the analytics can generate evidence of the assurance of the quality of the trial's safety and data (for example for AE under-reporting as demonstrated in Fig. 1). We hope that the COVACTA experience can serve as a model for ensuring quality of future clinical trials - even those conducted in a “normal” setting. The trials would benefit from real-time quality monitoring supported by advanced-analytic detection methods. This approach would have the potential to reduce the need for on-site audits and thereby shift the focus away from source data validation and verification towards pre-identified, higher risk areas.

3.1. Challenges

As the analyses were conducted on a daily basis, it required the equivalent of 2.5 Full Time Employees (FTEs) for a period of several weeks. As we expand on this approach, the overall process we described should be streamlined and, where possible, automated. The frequency of the analysis can likely be decreased, especially when trials are conducted outside of urgent circumstances, such as the Covid-19 pandemic.

The quality monitoring needed to be adjusted to the unique specifics of the study protocol and quality thresholds needed to be carefully defined. This required a significant experience with data analytics in the context of clinical trials. This effort would need to be repeated for new studies that will be monitored as closely.

An IT infrastructure that was up and running, access to the required data, and the right set of tools were prerequisites to enable quick set-up and execution of these activities. Last, but not least, having quality professionals with advanced data literacy plus data analysts/scientists with sufficient business/GxP knowledge was essential to ensure the right data elements were reviewed and the outputs of the analytics were interpreted correctly. We benefited from having an internal data analytics training program at Roche/Genentech PDQ. Since most of our staff participated in the training, they had the necessary skills to apply their learnings to this project.

4. Conclusion

We used the COVACTA study as the first example of clinical QA activities being conducted fully remotely. We believe that the learnings could be applied for other clinical trials, as advanced analytics enable holistic and almost real-time issue detection. Leveraging analytics and integrating the outputs and their interpretation in a Quality Analytics Review Report could serve as quality evidence and reduce the burden of on-site audits and inspections. As a next step, we will continue to engage with other pharmaceutical sponsors, industry associations and Health Authorities [8] to accelerate the use of advanced analytics for clinical QA [[6], [7], [8], [9], [10]] - the overall goal being to improve quality and compliance throughout the trial and thereby contribute to an accelerated drug approval process for the benefit of patients.

Declaration of competing interest

Timothé Ménard, Rich Bowling, Pooja Mehta, Björn Koneswarakantha and Eileen Magruder were employed by Roche/Genentech at the time this research was completed.