A risk-based monitoring approach to source data monitoring and documenting monitoring findings

Abstract

Background

Clinical trial monitoring is evolving from labor-intensive to targeted approaches. The traditional 100% Source Data Monitoring (SDM) approach fails to prioritize data by significance, diverting attention from critical elements. Despite regulatory guidance on Risk-Based Monitoring (RBM), its widespread implementation has been slow.

Methods

Our study teams assess the study’s overall risk, document heightened and critical risks, and create a study-specific risk-based monitoring plan, integrating SDM and Central Data Monitoring (CDM). SDM combines a fixed list of pre-identified variables and a list of randomly identified variables to monitor. Identifying variables follows a two-step approach: first, a random sample of participants is selected, second, a random set of variables for each participant selected is identified. Sampling weights prioritize critical variables. Regular team meetings are held to discuss and compile significant findings into a Study Monitoring Report.

Results

We present a random SDM sample and a Study Monitoring Report. The random SDM output includes a look-up table for selected database elements. The report provides a holistic view of the study issues and overall health.

Conclusions

The proposed random sampling method is used to monitor a representative set of critical variables, while the Study Monitoring Report is written to summarize significant monitoring findings and data trends. The report allows the sponsor to assess the current status of the study and data effectively. Communicating and sharing emerging insights facilitates timely adjustments of future monitoring activities, optimizing efficiencies, and study outcomes.

Introduction

Clinical trial monitoring has undergone significant transformation, progressing from labor-intensive methods to more streamlined and targeted approaches.[1] In the traditional monitoring model, a comprehensive review compares all reported data elements to the original source documents (referred to as 100% source data monitoring (SDM)). SDM encompasses source data review and verification. This traditional monitoring approach does not differentiate data based on its significance or potential impact to participant safety or study results. The broad scope distracts monitors from concentrating efforts on critical elements. Additionally, 100% SDM results in intensive time and resource endeavors to rectify minor errors.

Clinical research has seen continued escalating costs over the past several decades [2–4]; and both regulatory agencies and pharmaceutical companies have recognized the need for enhanced monitoring efficacy and the necessity for focused monitoring practices, leading to the development of risk-based monitoring (RBM) principles. In 2013, the European Medicines Agency (EMA) [5], the United States Food and Drug Administration (FDA) [6], and TransCelerate [7], an industry consortium, all released recommendations in support of RBM approaches. In 2016, the International Council for Harmonization (ICH) of Technical Requirements for Pharmaceuticals for Human Use Good Clinical Practice (GCP) E6 (R2) introduced a systematic framework for evaluating and managing study risks to help reduce implementation challenges.[8] Two additional FDA guidance documents have been released since then (2019 and 2023), reinforcing the agency’s commitment to RBM.[9, 10]

Despite regulatory guidance on RBM, its widespread adoption has been slow.[11] To better understand the barriers to adoption, the Tufts Center for the Study of Drug Development conducted an online survey in 2022 and 2023 among pharmaceutical, biotechnology, and contract research organizations, revealing several primary obstacles with RBM.[4] These include the lack of organizational knowledge and awareness, mixed perceptions of the value proposition of RBM and quality management, and poor change management planning and execution. The Association of Clinical Research Organization (ACRO) also conducted surveys to quantify the use of RBM activities in trials conducted across several Clinical Research Organizations (CROs) prior to and during the COVID-19 pandemic.[12, 13] Findings indicate that, while studies are increasingly implementing RBM activities, an area that requires further development is the implementation of centralized data monitoring (CDM) corresponding to a reduction in SDM.[12, 13] Authors have recognized that reluctance from sponsors to perform less than 100% SDM might stem from a concern for potentially missing safety signals and compromising data quality, which ultimately may result in regulatory agencies challenging results.[1] However, a comparison between 100% SDM and RBM across six studies revealed that only two (1.8%) of 112 serious adverse events (SAEs) were missed with RBM compared to no missed SAEs with 100% SDM.[14] In another comparison study, the implementation of CDM paired with targeted remote monitoring successfully identified all critical items that were found during on-site 100% SDM, suggesting the effectiveness of RBM.[15] Agrafiotis et. al. delineated an integrative approach to monitoring techniques, outlining how SDM can adapt over time based on findings throughout the study as a site’s risk level changes.[16]

The University of Utah Data Coordinating Center (Utah DCC) is an academic research organization that provides clinical operations, data management, statistical, and regulatory support for several government, philanthropic, and industry funded clinical studies. While findings from the Tufts Center for the Study of Drug Development and the ACRO resonated with our experiences, we identified the lack of a universally accepted, straightforward RBM methodology as a significant hurdle to our RBM implementation. To address the gap in existing literature on participant selection for SDM and the interaction between SDM and CDM, as identified by the ACRO [12, 13], this paper describes the use of random variable sampling for SDM, discusses how we allocate SDM and CDM to study risks and their interaction, and advocates for a cross-functional, collaborative team-based approach for the distribution of monitoring findings to inform and agilely adapt monitoring activities. Sharing our approach aims to advance the field and promote wider adoption of RBM in the research community.

Methods

Monitoring is an ongoing and iterative process, starting with the first participant enrolled through database lock. Comprehensive scrutiny of findings from each monitoring method informs and guides adjustments to monitoring procedures, potentially leading to the identification of new study risks and protocol amendments. Monitoring activities remain dynamic and perpetually evolve, adapting in response to new information and emerging risks. Consequently, study documents may undergo periodic revisions to account for insights gained during monitoring activities. A visual representation of the interactive RBM concepts is depicted in Figure 1. This figure illustrates how Study Documentation drives which data and processes need to be monitored through SDM and CDM (outgoing arrows from Study Documentation). Findings from CDM can trigger adjustments in SDM activities, and vice versa (bidirectional arrows). Based on findings from both SDM and CDM, study teams may need to revise study documentation, which could involve amending the protocol or consent forms, updating instructions, or modifying the monitoring strategy itself (outgoing arrows from SDM and CDM towards Study Documentation).

Figure 1:

Lifecycle of Study Monitoring – An Iterative Process

Development of a Risk-Based Monitoring Strategy for an Individual Study

The Utah DCC RBM strategy begins by assessing the study’s overall risk using our standard Study Risk Level tool, which incorporates FDA guidance key aspects [6, 10] such as study design, blinding status, and the investigational product’s and intervention’s relative safety. The study risk level tool classifies each study aspect as low, medium, or high risk. This risk assessment process culminates in an overall study risk determination as low, medium, or high risk. As recommended by TransCelerate [7] and ICH GCP E6 (R2) [8], we tailor our SDM and CDM monitoring efforts in proportion to the overall study risk determination.

Following determination of the study risk level, the study team transitions to the Utah DCC Risk Assessment and Risk Management (RARM) tool [17] during protocol development to identify and address key risks to participant safety and data integrity. The protocol undergoes iterative revisions prior to beginning enrollment to eliminate or minimize risks during the study. Metrics and mitigation plans are documented in the RARM tool for all remaining key study risks. Continuous risk assessment throughout the study ensures proper risk management and the identification of previously unforeseen risks.

We create a study-specific risk-based monitoring plan (RBMP), integrating SDM and CDM according to the overall risk of the study and key risks of the study documented on the RARM tool. The RBMP outlines all planned monitoring activities throughout the study. CDM monitoring descriptions include information about the individual(s) responsible and frequency for reviewing each CDM report. Additionally, the RBMP differentiates which variables will be monitored for all participants and which variables are monitored for only a subset of participants.

Figure 2 presents a visual representation of our Risk-Based Quality Management (RBQM) approach, which encompasses our RBM strategy for individual studies. The process commences with the identification, evaluation, and implementation of a risk control plan for study risks. Subsequently, as monitoring activities commence, identified findings are communicated among key stakeholders for resolution, potentially leading to the identification of new risks, adjustments to monitoring activities, and revision to study documents when applicable. The figure also specifies the tools developed for documentation purposes.

Figure 2.

Risk-Based Quality Management Cycle for a Study

Apportioning SDM and CDM to Identified Study Risks

The Utah DCC uses the study’s risk level to guide the overall monitoring strategy. CDM is used to enhance and guide our SDM activities. We program data checks in the electronic data capture system to identify missing or potentially erroneous data, protocol deviations, and timeliness of data entry or corrections, reducing the need for extensive SDM. In order to identify outliers and systematic data, we develop analytics and visualizations to summarize variables in aggregate. Aggregate summaries also assess quality metrics and allow for visual comparisons of data across sites including summaries of protocol non-adherence and potentially unreliable data. We rely on metrics such as enrollment rates, follow-up rates, and missingness rates to identify individual sites or personnel who need retraining. These metrics often serve as proxies for larger issues that might be occurring at the site (e.g., inefficient screening methods). We also use previously described CDM methods, including summaries about supervised and unsupervised monitoring [18] and statistical techniques [19] such as statistically comparing distributions of data between sites that can be documented in various RBM tools.[20]

Source Data Monitoring Sampling Schema

CDM activities can detect missing or logically erroneous data and identify site or studywide systemic issues. However, ensuring data accuracy ultimately hinges on a monitor comparing the data reported in the database with the original source document through SDM. For SDM, the Utah DCC identifies data that should be monitored based on a combination of a pre-identified fixed list of variables (e.g., date of consent) and a two-step random sampling of participants and variables. In the first step, a random sample of participants is selected. In the second step, we identify a random set of variables to SDM for each participant. Participants selected for SDM are no longer eligible for subsequent rounds of SDM sampling at future monitoring visits. Sampling weights for variables are assigned so that critical variables, such as eligibility and primary efficacy/safety outcomes, are more likely to be selected compared to other variables. Variables that are not assigned weights are defaulted to a sampling weight of 0 and therefore are not considered for monitoring. For example, a patient reported outcome where the data are entered directly into the database would have a weight of 0 since the database is the source document. For non-zero weight variables, our goal is to monitor the primary, secondary, and safety outcomes more frequently than exploratory outcomes. Assigning a sampling weight of 10 to critical variables intensifies their monitoring because the variables will be randomly selected more often than a descriptive variable which might be assigned a sampling weight of 1. At the beginning of a study, we utilize increased SDM activities for the first few participants enrolled at each site and during site staff transitions. This initial intensified monitoring strengthens our RBM approach by enabling early error detection and swift corrective actions such as retraining.

During the SDM sampling process, a mandatory minimum set of variables is selected, along with an extended list of additional variables for each selected participant. The minimum and extended number of sampled variables is based on the study risk level and pre-specified suggested thresholds delineated in the study RBMP. During SDM, the monitor reviews each participant’s minimum set of variables. If a significant issue is identified based on the criteria outlined in the RBMP, the monitor then continues to review all additional variables on the extended list for that participant. Significant issues include errors related to the informed consent process, participant eligibility, primary or secondary outcomes, SAEs or safety outcomes, study compliance (such as protocol deviations or regulatory/GCP non-compliance), or any other relevant findings identified by the monitor.

Depending on the severity of the issues, SDM might be expanded to include additional randomly selected participants at the same site. When necessary, a root cause analysis is conducted, and a corrective and preventive action plan (CAPA) is implemented. Establishing precise monitoring criteria to trigger additional monitoring activities ensures that the monitoring process remains adaptive and responsive to new findings.

Visually, the process of random variable sampling can be seen for a hypothetical example in Figure 3. In this hypothetical example, the team wanted to ensure the primary and secondary outcomes were sampled approximately 10 times more than model covariates. If the team wanted a more even distribution of sampling selection, the primary and safety outcomes could have used a smaller weight value instead. The study team pre-specified that consent forms and eligibility would be monitored for all sampled participants. Ten variables were additionally considered for SDM random sampling based on what was proposed for analyses in the Statistical Analysis Plan, and up to 70% of these variables were eligible for sampling per the RBMP (of which 30% would be minimum and 40% would be extended if issues were observed). Of the sampled participants, each participant has a different random set of variables that are monitored. In this example, among the 10 variables considered for monitoring, 4 (40%) are high-risk, indicated by a weight of 10, 3 (30%) are medium risk, indicated by a weight of 5, and 4 (40%) are low risk, indicated by a weight of 1. As mentioned earlier, if errors were found for the minimum set of monitored variables, the monitor would then proceed with monitoring the extended variables randomly selected for that specific participant.

Figure 3:

Hypothetical example of source data monitoring sampling schema

When outcomes are randomly sampled for SDM using our described method, there is a statistical phenomenon where higher-risk variables are undersampled and lower-risk variables are oversampled in comparison to the sampling weight due to sampling without replacement. When there is only 1 variable sampled for monitoring, no bias is introduced in variable sampling. However, when more than 1 variable is sampled, bias is introduced. For a simple illustration, assume there are 3 potential variables that can be selected for monitoring with weights 10, 5, and 1. If only one variable is sampled for monitoring, the outcome associated with a weight of 10 has the highest probability of being selected. However, if three variables are sampled, all three variables are selected for monitoring thus eliminating the concept of weighting (as the same outcome for a given participant would only be monitored once [i.e., sampled without replacement]). Sampling larger proportions of variables within a participant mimics the concept of 100% SDM for the considered set of variables.

If it is critical to ensure that high-risk variables are monitored more often relative to medium/low risk variables, and random sampling is used, we recommend implementing the statistical SDM sampling using the following guidelines, which depend on the relative proportion of high-risk variables monitored compared to medium/low risk variables. While the proportion of variables monitored (i.e., the 30% minimum and 40% extended in the example above) is independent of the weights assigned to the variables (i.e., the weights of 10, 5, and 1 in the example above), different combinations of these aspects produce a better sampling framework. In general, we recommend the following:

• Use higher sampling weights for high-risk variables when the proportion of high-risk variables is small compared to all variables eligible for monitoring. For instance, if we had options of assigning weights of 10 and 5 to high sample variables, it is advisable to use the higher weight of 10 when there are fewer high-risk variables compared to the entire dataset being submitted for sampling.

• Increase the proportion of variables monitored, such as monitoring 70% of variables on a participant instead of 30% of variables, when the number of high-risk variables is large compared to the number of medium/low risk variables. This essentially begins to mirror 100% SDM for the list of considered variables on the sampled participants as higher rates are selected.

Understanding and addressing the impact of sampling without replacement is crucial for maintaining the integrity of variables monitoring in clinical studies.

Documentation and Dissemination of Monitoring Findings

We employ a systematic approach to documenting and sharing monitoring findings. In the case of SDM, the monitor compiles their observations and findings into a site-specific Site Monitoring Visit Report at the completion of each site’s SDM visit. Subsequently, an SDM visit letter is sent to the individual site, outlining all issues that require necessary corrective actions. In the case of CDM activities, the reviews and date of review of all reports are entered in a structured database (e.g., REDCap) to provide documentation that the RBMP plan was followed for CDM activities.

The Utah DCC study team compiles significant findings identified through SDM and CDM activities into a single document referred to as the Study Monitoring Report. To finalize the Study Monitoring Report, the internal DCC study team conducts regularly scheduled meetings (e.g., monthly or quarterly) where each team member presents the report(s) that led to their respective SDM and CDM findings. The documentation serves as concrete evidence that identified issues have undergone investigation. Identified data trends lead to team discussions to identify the root causes of issues and outline subsequent steps. The Study Monitoring Report is updated during the meeting based on team discussion, and the document is then sent to the sponsor. The Study Monitoring Report provides a holistic review of findings from all monitoring methods.

Results

We have applied these methods to several studies. The SDM random sampling and documentation processes allow all team members to contribute to RBM activities and effectively implement the study. In this section, we provide example random SDM sampling results and a Study Monitoring Report.

Example output from random SDM sampling is shown in Figure 4. Figure 4.A shows 5 randomly selected participants for SDM. For this example, three variables were required for minimum monitoring while an additional four variables should be evaluated (extended monitoring) if monitoring criteria were not met, if significant issues were found on the minimum set of variables, or per monitor’s judgement. The monitor uses a look-up table in the output (Figure 4.B) to identify which database elements should be reviewed and confirmed from the site’s source documentation that corresponds to each variable selected. The project lead provides the statistician with the selection parameters, which consist of the date range for data selection and the proportion of variables that should be selected for monitoring. The study statistician then generates the sampling output using a sampling function created by the Utah DCC.

Figure 4:

In this example, Part A represents a random sample of participant outcomes selected for monitoring for 5 participants. Part B lists the corresponding database elements that match each outcome.

An example Study Monitoring Report finding is shown in Figure 5. The Study Monitoring Report is created using the team approach where individual report reviewers add findings based on their frequent assessment of accumulating data. Monitors include high-level summaries in the Study Monitoring Report to provide a holistic view of the study results. Findings are summarized into overall study aspects followed by individual site findings. Depending on the complexity of the study, this process typically involves one to four hours of preparation for team members, followed by a one to two hours meeting to review and analyze findings for each report created. Subsequently, the project lead spends a couple hours consolidating findings and finalizing the Study Monitoring Report.

Figure 5:

Example of a Study Monitoring Report

Conclusions

We present a risk-based monitoring approach that includes random sampling of participants and variables for SDM and describe a method for centralizing documentation of SDM and CDM activities. In our proposed two-step SDM method, sampling occurs at the participant level followed by the variable level. If critical issues are found during monitoring, additional variables are monitored, which allows for an adaptive SDM approach. The proposed random sampling method is used to monitor a representative set of critical variables for accuracy. All CDM activities are implemented as outlined in the RBMP. High-level SDM findings and data trends identified from CDM reviews are all summarized in a central Study Monitoring Report, which is then provided to the sponsor.

Our risk-based monitoring approach shares conceptual similarities with approaches used by others, which assess sites based on both SDM and CDM activities. [16, 21] We expand on how SDM can be adapted during an ongoing monitoring session. Additionally, our approach provides a holistic view on study data quality, facilitating timely adjustments of our monitoring activities. Agrafiotis et. al. demonstrate how an individual site’s risk score can change over time as performance is evaluated.[16] Grading systems for the sites to quantify the risk level can potentially allow adaption of the SDM frequency based on CDM findings.[16, 21] Our Study Monitoring Report document could be extended to fit site categorization by applying a grading system to the findings and systematically categorizing sites into risk levels. Diani et al. describe 20 risk factors that leads to an overall site-level risk score, although details about other qualitative assessments are not provided.[21]

Effective communication and collaboration are at the core of RBM approaches. It is critical to share potential issues identified through various monitoring methods. When findings are collectively considered, overall study and site-level conclusions are identified that can subsequently influence the course of monitoring actions. Our cross-functional team science approach facilitates swift implementation of preventive actions. Detailed CDM documentation can signal the need for focused SDM or, conversely, indicate when SDM frequency should change at a site. Collaborative sharing not only facilitates timely adjustments but also enables proactive utilization of emerging insights and accumulated data, resulting in a continuously refined monitoring approach tailored for optimal efficiency and outcome.

Despite its merits, our proposed SDM sampling scheme and the creation of the Study Monitoring Report process have several limitations. The sampling schema does not easily extend to longitudinal analyses, especially when a participant may only have a baseline visit that needs monitoring yet the follow-up visits have not yet been observed. It also requires more complex statistical code to perform the sampling. While the Utah DCC typically acts as both a DCC and a clinical coordinating center (CCC), scheduling meetings with separate entities (one functioning as a DCC and the other as a CCC) can be challenging when monitoring activities are split across organizations. Furthermore, in projects with limited budgets, holding additional internal meetings to compile a Study Monitoring Report may not seem to be financially feasible. However, our experience from piloting this approach in several studies over the past several years indicates that identifying emerging issues early on through these meetings enables more efficient issue resolution. This proactive approach prevents issues from escalating into unrecoverable problems, potentially requiring significant operational overhauls if addressed at a later stage. While we have not conducted a controlled evaluation of our RBM methodology to directly assess its impact, we have found investing in these meetings upfront can save resources and prevent budget overruns, proving to be a necessary investment, especially for studies with tight budgets. We have observed study team members use data from the study monitoring report to objectively communicate issues to a sponsor. Moving forward, conducting a formal review of studies where our method has been fully implemented is necessary to validate our claims.

The ongoing evolution from a comprehensive approach (100% SDM) to RBM reflects the increasing recognition that monitoring all source data elements is labor-intensive, costly, and less effective than a well-planned and executed targeted monitoring strategy. Implementing a robust RBM approach ensures targeted and efficient clinical trial oversight. The Utah DCC’s implementation of a study-specific RBM strategy, grounded in established regulatory guidance, emphasizes the importance of identifying critical data and study risks. It involves the use of dynamic and complimentary monitoring methods, along with the sharing of observations and findings.

Abbreviations:

ACRO

Association of Clinical Research Organization

CAPA

Corrective And Preventive Action

CCC

Clinical Coordinating Center

CDM

Central Data Monitoring

CRO

Clinical Research Organization

DCC

Data Coordinating Center

EMA

European Medicines Agency

FDA

Food and Drug Administration

GCP

Good Clinical Practice

ICH

International Council for Harmonization

RARM

Risk Assessment and Risk Management

RBM

Risk Based Monitoring

RBMP

Risk-Based Monitoring Plan

RBQM

Risk-Based Quality Management

SAE

Serious Adverse Event

SDM

Source Data Monitoring