Improving Risk Assessment in Clinical Trials: Toward a Systematic Risk-Based Monitoring Approach

Abstract

Regulatory authorities have encouraged the usage of a monitoring (RBM) system in clinical trials before trial initiation for detection of potential risks and inclusion of a mitigation plan in the monitoring strategy. Several RBM tools were developed after the International Council for Harmonization gave sponsors the flexibility to initiate an approach to enhance quality management in a clinical trial. However, various studies have demonstrated the need for improvement of the available RBM tools as each does not provide a comprehensive overview of the characteristics, focus, and application.

This research lays out a rationale for a risk methodology assessment (RMA) within the RBM system. The core purpose of RMA is to deliver a scientifically based evaluation and decision of any potential risk in a clinical trial. Thereby, a monitoring plan can be developed to elude prior identified risk outcome.

To demonstrate RMA's theoretical approach in practice, a Shiny web application (R Foundation for Statistical Computing) was designed to describe the assessment process of risk analysis and visualization tools that eventually aid in focusing monitoring activities.

RMA focuses on the identification of an individual risk and visualizes its weight on the trial. The scoring algorithm of the presented approach computes the assessment of the individual risk in a radar plot and computes the overall score of the trial. Moreover, RMA's novelty lies in its ability to decrease biased decision making during risk assessment by categorizing risk influence and detectability; a characteristic pivotal to serve RBM in assessing risks, and in contributing to a better understanding in the monitoring technique necessary for developing a functional monitoring plan.

Future research should focus on validating the power of RMAs to demonstrate its efficiency. This would facilitate the process of characterizing the strengths and weaknesses of RMA in practice.

Introduction

Clinical trials are conventionally monitored by source data verification that is costly, requires ample resources, and exhibits several limitations.^1,2 The International Council for Harmonization (ICH) has provided sponsors with the flexibility to initiate a novel approach called risk-based monitoring (RBM) to enhance quality management in a clinical trial.³ Regulatory authorities such as European Medicines Agency (EMA) define RBM as a systematic process that involves identification, assessment, controlling, communicating, and reviewing the risks in a clinical trial before its initiation.⁴ With this methodology, not only would the occurrence of the assessed risk be prevented, but it would also minimize onsite monitoring duties to some extent. Following the ICH recommendation for approach utilization, several RBM tools were developed. The available RBM tools have been identified and summarized based on their structural approaches, similarities, and differences.⁵ Additionally, noncommercial RBM tools were compared in their application on real clinical trial protocols to assess the overall risk level of each protocol by each tool; furthermore, each noncommercial RBM tool was compared directly with the Transcelerate RBM tool (commonly accepted as the standard in pharmaceutical industry) to investigate the risk category and risk coverage in each.⁶

These studies reveal distinct approaches employed by the available RBM tools to assess a certain risk, demonstrate the unique assessment of each RBM Tool to the same clinical trial protocol, and exhibit the different risks investigated within each RBM tool. The Food and Drug Administration (FDA) encourages all clinical trials regardless of the phase to implement RBM. Currently a standardized RBM approach for clinical trials is lacking,⁷ which presents a challenge to implement RBM by the industry.⁸ Our objective is to fill the gap by presenting a systematic risk analysis in clinical trials to standardize RBM. To have an efficient RBM tool, a potent risk assessment has to be performed first. For this reason, we propose a novel methodology and a robust algorithm to assess any risk in a clinical trial. The methodology can be implemented on any clinical trial regardless of the phase and complexity. Moreover, the algorithm aids the assessor in the decision-making process of monitoring technique needed and monitoring level required during the development of the monitoring plan.

Risk Identification Process

The quality feature of an RBM system entails risk assessment of a study and a mitigation plan that details a monitoring strategy for the concerned trial. However, the crucial question arising is how to define a certain risk.

The presence of varying risk criteria covered and examined within the risk assessment by each RBM tool suggests the need to restructure the definition of a certain risk. A risk is defined as the unsolicited outcome of a certain process. Any event that is likely to have a negative influence on the trial should be counted as a risk. The identified risk must be assessed through its influence on the safety of the human participant, trial integrity, the chance of its occurrence, and the ease by which it can be detected. Several systems such as Delphi⁹ or SWOT analysis¹⁰ can be oriented toward identifying risks in clinical trials. The Delphi method is a process that utilizes a questionnaire circulated among experts such as clinical research associates, statisticians, clinical investigators, sponsors, and any member involved in a clinical trial stage.⁹ SWOT analysis is yet another strategy that aids organizations to pinpoint strengths, weaknesses, opportunities, and threats to a business or a project planning, in this case a clinical trial.¹⁰ The application of both methods is simple, and their outcome is highly dependent on the diverse groups involved.¹¹ Another approach is utilizing risk summaries from monitoring reports of completed clinical trials; however, it is unlikely to access those reports as they are only accessible by the sponsors.¹¹

An Ideal RBM System

Clinical trial sponsors along with the involved clinical trial members are responsible for guaranteeing the safety and well-being of the human participants, their rights, and the data quality.¹² The regulatory authorities require sponsors to ensure proper monitoring during the initiation and progress of a clinical trial.¹³ RBM is expected to be an imperative tool in guiding the sponsor to identify and mitigate risks.¹⁴ Similarly, EMA's reflection article concerning risk-based management demonstrates that a risk-based approach is needed to enhance quality management of clinical trials.¹⁵ To date, FDA's guidance on RBM approach is divided into 3 parts, the detection of critical data and processes, the risk assessment categorization tool, and developing an appropriate monitoring plan following the risk-based approach.¹⁶ Such a revolutionized technology played a huge role in achieving RBM in the field of mitigation monitoring techniques developed as remote monitoring.¹⁷ The focus of any mitigation plan is shaped by the outcome of a risk assessment. Although 100% source data verification can certainly be reduced by the available mitigation plans, it does not reflect the focus of the personnel carrying out onsite monitoring activities, as the FDA entailed.¹⁶

Proposed Risk Methodology Assessment in Clinical Trials

An RBM tool that covers risks in any clinical trial including a monitoring plan of appropriate technique is still missing.¹⁸ Additionally, there still exists ambiguity in the assessment methodology behind a certain risk. In this study we propose a novel risk methodology assessment (RMA) that enables the user to visualize the assessment of individual or overall risks present in a specific trial. RMA follows the concept of failure mode and effect analysis, specifically a systematic failure mode and effect analysis.¹⁹ The focus is on system-related deficiencies in which hazards are identified, studied, and prevented.

The fundamental process is to initially focus on the most common faults detected in previous trials. For this reason, the RMA approach includes the frequent findings detected by Good Clinical Practice- Inspectors Working Group (EMA GCP-IWG) report.²⁰ The EMA GCP-IWG objective is to harmonize and coordinate GCP activities in the European Union. The annual report, which emphasizes GCP practice in the European Union, can be used as a reference for risk identification. The report sheds light on the number of inspections done routinely and non-routinely to active clinical trial sites and reports deficiencies detected in the trials.

Our article follows the recommendation of the ICH to favor risk based monitoring by providing a methodology of risk assessment that evaluates the occurrence likelihood of a risk, summarizes the extent of monitoring required with the help of a radar plot-based visualization of said risk and hence aids in the decision making of the mitigation step to be put forth. RMA does not suggest a prevention strategy due to the miscellaneous outcome of a certain risk in an individual trial. For instance, a risk associated with investigational medicinal products in a Phase I trial might have a higher impact than a Phase III trial. The anticipation step and the overall mitigation plan should be developed by the stakeholders responsible for the planning procedure. The FDA specifically highlights the sponsors’ responsibility to have a mitigation approach for defined risks irrespective of the implemented risk assessment technique.¹⁴ Figure 1 shows RMA's approach to identify, assess, and form a mitigation plan.

Figure 1.

Flowchart of risk methodology assessment (RMA) risk assessment process before and after trial initiation. This flowchart shows the methodological approach of risk-based monitoring (RBM). Following risk identification, each risk is evaluated and assigned a mitigation technique. Following the assessment, stakeholders develop the monitoring plan based on the assessment. The assessment must be repeated if any amendments were established to the protocol or when unidentified faults are discovered.

Theoretical Implementation of RMA Methodology

Each clinical trial is based on an explicit study protocol outlining the study end point(s), study procedures, medical investigations, and so on, which necessitate appropriate consideration during risk identification. The results presented by the GCP-IWG annual report signify the definite complications that a monitoring team can detect during a routine site visit. For this reason, the identification process of potential risks could be derived from GCP-IWG report as a starting point. Accordingly, a risk assessment should reflect the detected faults as risks that must be assessed before trial initiation.

A risk assessment system should consist of components in which a risk is identified, assessed, visualized for its monitoring level, and classified into the type of monitoring required. The assessment process is classified based on the FDA's recommendation of impact, probability, and detectability.⁷ Nonetheless it does not indicate standards each category should be assessed on. It is left to the stakeholders to decide the appropriate decision process. In the presented methodology we propose defined standards required for impact and detectability measurements.

According to the ICH-GCP guidelines,²¹ monitoring is conducted to ensure the well-being/safety of participants, the reliability of data and compliance with GCP/protocol guidelines. A risk that does not affect at least 1 of these criteria must not be deliberated as a risk that can be covered by RBM monitoring. The individual criteria should be differentially weighted based on the critical aspect resulting from each separately. For instance, a risk affecting the well-being/safety of participants alone will have a higher impact than a risk affecting GCP/protocol compliance. The detectability and probability should be assessed by the stakeholders based on their decision process. However, probability is weighed based on the likelihood of a risk occurrence and detectability is evaluated based on the monitoring detection technique either as remote monitoring or onsite monitoring. We propose a score measure for the category of each criterion (Table 1).

Table 1.

Risk assessment criteria. Following risk identification, each risk is evaluated based on the category it impacts, the probability of risk occurring, and the monitoring technique required for detection.

Criteria	Assessment category	Score
Impact	1. Well-being/safety of subjects	3
	2. Reliability of data	2
	3. Compliance with GCP/protocol guidelines	1
Probability	1. Very likely	5
	2. Likely	4
	3. Even chance	3
	4. Unlikely	2
	5. Very unlikely	1
Detectability	1. Onsite monitoring	2
	2. Remote monitoring	1

GCP = Good clinical practice.

Scoring Method

The scoring algorithm of RMA allows the stakeholders a unique prospect to visualize the risk size and quantify it. The goal of risk communication is to guide the stakeholders in the risk assessment in a transparent manner and to assist them in the decision plan to mitigate its occurrence by an effective measure.²² Visual representation can help stakeholders observe the assessment of the risk and understand its needed monitoring level. The visualization process can be achieved by radar charts as they enhance comparisons of quality measurements.²³

With the defined scaling system, the area would reflect the extent of how critical a risk is, which subsequently hints to the extent of monitoring required. The larger the area, the more monitoring is required; however, it does not reflect the type of monitoring technique needed as this must be decided by the stakeholders themselves (Figure 2). Following the assessment, a monitoring technique should be assigned. According to regulatory agencies, the main techniques can either be traditional onsite monitoring, remote monitoring, or a combination of both.

Figure 2.

The area under the radar chart. This figure shows the total area of the radar chart. Each area of the subtriangles is calculated based on the conventional formula.

Area Under the Radar Chart

The aim of radar chart is to present multivariate data, the main advantage is to translate the data to a meaningful sense. The area under the radar is equivalent to the cumulative area of the separate triangles (Figure 2). The area under the radar chart is then reported as a percentage of the maximum score possible.

Each area is detected by the sides of the respective triangle input

$$\begin{array}{ccc}\hfill AreaA1& =& 0.5\times input\left(Impact\right)\times input\left(Detectability\right)\hfill \\ & & \times \text{sin}\left(120\right)\hfill \end{array}$$

$$AreaA2=0.5\times input\left(Impact\right)\times input\left(Probability\right)\times \text{sin}\left(120\right)$$

$$\begin{array}{ccc}\hfill AreaA3& =& 0.5\times input\left(Probability\right)\times input\left(Detectability\right)\hfill \\ & & \times \text{sin}\left(120\right)\hfill \end{array}$$

$$TotalRadarArea=AreaA1+AreaA2+AreaA3$$

Practical Implementation of RMA

A shiny web application was formed to illustrate the theoretical approach of RMA. The application includes risks that could be assessed and visualized under the radar plot (Figure 3).

Figure 3.

The individual risk assessment presented by the radar chart. This figure shows the criteria of risk assessment that should be completed by the assessor. The individual risk is assessed by its impact, probability, and detectability. Accordingly, the total area of the risk is presented by the radar chart.

Following the assessment of the individual risks, the input scores provided by the assessor and the subsequent score areas are documented. The following process can aid stakeholders in comparing the assessment report with monitoring reports after trial initiation to get a better understanding of the faults/weaknesses and strengths of the performed assessment (Figure 4).

Figure 4.

Assessment score of each individual risk with corresponding input. This figure shows the documentation of the individual risks assessed with its input criteria score and the computed overall score.

The score of the distinct risks assessed allows stakeholders to distinguish high score risks that necessitate more extensive monitoring in the monitoring plan (Figure 5a). Consequently, based on the profile input of each risk (Figure 5b) and its relation to the threshold for maximum score, represented by dashed lines, stakeholders can decide on the extent of monitoring visits/checks required in the monitoring plan. Finally, an overview of the sum of risks to be monitored by each technique (Figure 5c) imparts a clearer understanding of the type of monitoring plan needed, which is highly essential in the application of RBM.

Figure 5.

Risk methodology assessment. (A) Overall scores (area under the radar) of each risk. (B) Risks based on the input of the assessment; red, green, and blue points are compared with their respectively colored dashed lines representing the maximum score. (C) Overall counts of risks covered by each monitoring technique.

The assessment process should repeated as soon as amendments are made to the trial protocol or when identifying new risks during monitoring process after trial initiation. This would require the stakeholders to conduct a new risk assessment to engage a proper mitigation action in the monitoring plan. It is essential to act on a new identified risk to understand its direct effect on the overall score of the risk assessment as a whole and on the monitoring technique required to prevent its occurrence.

Generally, the monitoring activities of the clinical research coordinator/monitoring team should focus on the requirements, responsibilities, and hazards that can carry potential liabilities to the trial assurances. The final assessment report will stipulate the potential risks to be monitored and frequency of monitoring needed.

Because RBM is becoming a principle stage in clinical trials,²⁴ both RMA's strategy and approach have the potential to improve data quality and reduce clinical costs. Undoubtedly, the risk assessment within other RBM systems can also identify certain risks; however, the assessment methodology of the individual risk criteria is either not reported or vague. As for their systems, they are fixed on prespecified risks lacking the ability of tallying new ones. For this reason, RMA's scoring system provides a means to facilitate confirmation of a certain risk and assess its outcome measure. Additionally, it incorporates flexibility in directly, including an additional risk area in the assessment report. Finally, once the entire risk assessment is completed, risks could be grouped based on the monitoring technique to assist the stakeholders in the trial monitoring plan development. The established method can be considered a primary step toward a practical monitoring guidance in which a monitoring plan form will be based on different risks in a trial, individual process, and required monitoring.²⁵

The innovative approach of RBM will facilitate establishment of adequate and focused monitoring activities, reduce 100% source data verification activities, and enhance the quality of the trial and patient safety.²⁶ This goal should be clearly communicated to stakeholders and clinical trials to prevent misconceptions among clinical research coordinators regarding RBM's outcome in increasing workload, a concern that has been previously reported, despite its capacity to do the opposite. RBM is a continued improvement process that requires all stakeholders and clinical trial staff to initiate the risk assessment before and during the trial period. An effective monitoring plan can only be achieved after a successful implementation of RBM.²⁷ We believe the RMA approach can aid stakeholders in distinguishing and evaluating any potential risk. Future investigation should focus on validating the power of RMAs to demonstrate efficiency in practice.

RMA could be further developed to software that utilizes existing data to forecast a certain risk outcome and provide a mitigation plan based on the risk score. Further work is required to achieve the desired prediction. Classification models may be employed to predict the existence of a specific risk and measure its individual score; however, numerous factors such as data quality and model fit variability require consideration during the utilization of such models.²⁸ Artificial intelligence algorithms should be the next phase of any risk assessment. Transparent risk methodologies such as RMA should be made available to both regulatory authorities and the public. The prospect of being able to estimate a risk outcome and potential mitigation serves as a continuous incentive for future research. We believe the efficiency of RBM has been well established and proven; yet the ultimate design of RBM development will be a challenge for us for years.