Cross-validation of pharmacokinetic assays post-ICH M10 is not a pass/fail criterion

ABSTRACT

The ICH M10 guideline establishes global standards for bioanalytical method validation for pharmacokinetic assays, focusing on data reliability and accuracy across studies. A significant component is cross-validation, which should be performed to ensure data comparability when multiple methods or laboratories are involved in a single study or across studies where comparison will be performed. However, ICH M10 does not specify acceptance criteria for cross-validation, creating challenges for the industry because traditionally many laboratories have always utilized acceptance criteria to “pass” or “fail” the study. This editorial discusses how bioanalytical labs should conduct cross-validation for PK assays post-ICH M10, highlighting the role of statistical methods and the need for close collaboration with clinical pharmacology and biostatistics departments. Proper implementation and strategic focus on relevant studies are essential for effective cross-validation.

1. Introduction

The ICH M10 guideline [1] provides comprehensive recommendations for the validation and implementation of bioanalytical methods for PK assays used in drug development. It focuses on ensuring the reliability, consistency and accuracy of bioanalytical assays used to analyze pharmacokinetic (PK) samples from nonclinical and clinical trials. ICH M10 harmonizes the requirements for bioanalytical method validation across different regulatory regions, making it easier for pharmaceutical companies to meet global regulatory expectations.

One of the testing parameters where guidance is provided in the ICH M10 is cross-validation. A cross-validation is required to demonstrate how the reported data are related when multiple bioanalytical methods and/or multiple bioanalytical laboratories are involved.

This can be when:

• Data are obtained from different fully validated methods within a study.

• Data are obtained within a study from different laboratories with the same bioanalytical method.

• Data are obtained from different fully validated methods across studies that are going to be combined or compared with support special dosing regimens, or regulatory decisions regarding safety, efficacy and labeling.

However, unlike other validation parameters or past industry norms, the ICH M10 guideline does not specify acceptance criteria for cross-validation. This has left the industry with a challenge, which is to implement a robust and scientifically sound approach. The current editorial is focused on how bioanalytical laboratories should conduct cross-validation post-ICH M10 implementation, focusing on the challenges posed by the lack of explicit acceptance criteria and the necessity of statistical assessments.

2. The importance of cross-validation in bioanalysis

Cross-validation in bioanalysis serves to ensure that results generated by different methods, laboratories, or analytical platforms can be correlated when the objective of the clinical study(ies) requires comparison across the data generated from these various methods. This process is crucial when bioanalytical methods are transferred from one laboratory to another, when different methods are used to analyze the same drug compound, or when data from multiple studies using different methods are combined for a pharmacokinetic assessment.

Without proper cross-validation, discrepancies between methods or laboratories could lead to incorrect conclusions about a drug's pharmacokinetics, efficacy, or safety. The integrity of bioanalytical data is paramount in drug development, as these data support critical decisions regarding dosing, safety margins and efficacy evaluations. Therefore, ensuring the consistency and comparability of bioanalytical results through cross-validation is a fundamental aspect of the bioanalytical method validation process.

ICH M10 [1] specifies that fully validated methods should be cross-validated when necessary. It is the purpose of the data that defines if a cross-validation should be performed. Cross-validation should be performed in the studies where the data are combined or compared in support of a special dosing regimen or if the data are compared or combined for a regulatory decision regarding safety, efficacy and labeling. It is common for multiple bioanalytical methods to be utilized in the course of the development of a drug. It is not always required to cross-validate these methods even if different methods are utilized within a single submission. Cross-validation should only be conducted if the previously mentioned criteria are met.

3. Previous guidelines from the FDA & EMA on cross-validation

Before the implementation of ICH M10, bioanalytical method validation was primarily guided by the guidelines from the US Food and Drug Administration (FDA) [2,3] and the European Medicines Agency (EMA) [4].

The FDA's 2018 Bioanalytical Method Validation (BMV) Guidance [3] emphasized the importance of cross-validation when two or more bioanalytical methods are used to generate data within the same study or across different studies. It also stipulated that a cross-validation study should be performed when sample analysis within a single study is conducted at more than one site or more than one laboratory. However, while the FDA BMV 2001 [2] and 2018 [3] provided general recommendations on when cross-validation should be conducted, it did not define specific acceptance criteria for assessing the comparability of results between methods, just that the acceptance criteria should be defined a priori.

Similarly, the EMA 2011 Guideline on Bioanalytical Method Validation [4] provided guidance on cross-validation, particularly when data are obtained from different methods within and across studies or when data are obtained within a study from different laboratories. EMA describes that QCs or study samples can be used and that the acceptance criteria are that mean accuracy between QCs of each method should be <15% and when study samples are used at least 2/3 of the study samples should be within 20%. The EMA guideline goes on to say that the outcome of the cross-validation is critical in determining whether the obtained data are reliable and whether they can be compared and used. In the context of the new recommendation from ICH M10 it is also relevant to note that the EMA defines cross-validation as a “Comparison of validation parameters of two bioanalytical methods.” It doesn't state that it is a pass-fail activity but rather a comparison study which the ICH M10 built upon as we describe later.

4. Previous use of incurred sample reanalysis acceptance criteria in cross-validation

In the absence of explicit cross-validation criteria from the FDA BMV, and as acceptance criteria defined in EMA BMV is similar to the incurred sample reanalysis (ISR) criteria, the industry generally adopted the acceptance criteria established for ISR as a surrogate benchmark for cross-validation acceptance criteria and cross-validation became a pass/fail activity [5,6].

According to both FDA and EMA guidelines, ISR results are considered acceptable if at least 67% of the reanalyzed samples fall within ±20% or ±30% of the original results for chromatographic or ligand binding assay (LBA) methods respectively. This criterion became standard for assessing cross-validation in bioanalytical laboratories despite knowing that EMA did not differentiate between Chromatographic or LBA and did not mention the 30% acceptance criteria. This also created confusion as companies would adopt different practices.

By applying the ISR acceptance criteria, the industry established a practical approach for how to handle cross-validation in the bioanalytical laboratory by defining that methods or platforms are comparable if the results of cross-validation meet the same criteria as ISR.

Utilization of ISR acceptance criteria is thought of as a useful tool to establish a pass/fail approach to cross-validation but doesn't answer the central statistical question of the exercise of a cross-validation study which is “can study data be compared intra- or inter-study?” and presumably, how should it be compared? So is it a useful tool? Using ISR criteria can establish a passed cross-validation even when there is significant bias between the methods that potentially should be accounted for. One simple example to consider is if the methods have an average performance bias of 12%, the ISR criteria approach would most likely establish that the cross-validation passes and then data would generally be reported as if identical from both methods. The bioanalytical scientists are not usually expected to make the determination if there is significant bias and therefore utilization of a more statistically robust approach is appropriate. Additionally, if an assay to measure the drug is established in two laboratories and both meet the acceptance criteria, yet a cross-validation “fails”, more questions are raised. Does it mean the new assay which is often the more modern and technically sophisticated assay is no good? Or does this automatically mean the original assay is not good because it is an old assay? What does that say about the validity of the data? These are difficult questions that that now need be answered. Furthermore, utilizing the ISR criteria approach is a regulatory risk since the global regulators and ICH expert committee have determined that it is not the preferred way to conduct a cross-validation.

5. Challenges in the absence of defined acceptance criteria

The lack of defined acceptance criteria for cross-validation under ICH M10 presents several challenges for the bioanalytical industry. These challenges include moving away from the previous pass/fail determinations, such as the so-called ‘ISR acceptance criteria’, and having the conversation with the sponsor and/or stakeholders regarding the responsibility for the statistical calculations and interpretation of cross-validation results [6]. While there can be a concern regarding the regulatory expectations of cross-validation results, ICH M10 provides straightforward, statistically sound guidance on best practice approaches to cross-validation for modern bioanalysis.

6. ICH M10 & the lack of explicit cross-validation criteria

The ICH M10 guideline provides a comprehensive framework for bioanalytical method validation covering aspects such as: accuracy, precision, selectivity, sensitivity, specificity, dilution linearity and stability. However, – M10 – very deliberately and intentionally omits acceptance criteria for cross-validation. This omission presents a significant change to bioanalytical scientists. It is an abrupt but important change from past practice. The bioanalytical industry is generally more comfortable with clearly delineated pass/fail exercises, but a new and more appropriate way of thinking needs to be applied. Cross-validation is a study of assay comparability between two validated, acceptable assays. There is no universally accepted benchmark for determining whether the results of different methods or laboratories are sufficiently comparable. Future papers presenting real-world examples following the ICH M10 approach will help guide the industry toward new and more comfortable best-practices. The ICH M10 guideline recommends assessing the bias of the method via the use of statistical approaches to assess the comparability of bioanalytical results and provides two-such approaches.

7. Statistical approaches to cross-validation

Given the changes to processes associated with not having defined acceptance criteria, statistical approaches to cross-validation provide the appropriate way to assess the outcome of a cross-validation study. It is important to note that these statistical approaches typically lie outside the purview of the bioanalytical laboratory. Instead, the responsibility for implementing and interpreting statistical analysis often falls on the clinical pharmacology or biostatistics departments, which are accountable for the overall data interpretation. These groups should understand and endorse the application of descriptive statistics as a tool over a simple pass/fail.

Since the end user of the bioanalytical data, often within the clinical pharmacology or biostatistical department owns the data, they are better positioned to understand the nuances of bias between studies or discrepancies between laboratories. This deeper understanding enables more informed decisions regarding the comparability of data from different sources, whether it involves different methods, instruments, or even entirely separate laboratories. Thus, the outcome of a cross-validation study should be either a set of data for this team to determine a bias between assays or can be a bias calculated by the bioanalytical team.

Statistical methods offer a rigorous, objective means of assessing this comparability, thereby providing a transparent and reproducible framework for cross-validation. Below are several statistical approaches presented in ICH M10 that can be employed to evaluate the comparability of bioanalytical methods:

8. Bias assessment using Bland-Altman plots & Deming regression

Assessing bias is crucial in cross-validation to identify systematic differences between methods or laboratories. Bland-Altman plots and Deming regression are statistical approaches suggested for this in the ICH M10 document. Other methods are also acceptable and the use of concordance correlation coefficient for assessing agreement between methods is one example provided.

Bland-Altman analysis plots the difference between methods against their mean, with limits of agreement typically calculated as the mean difference ±1.96-times the standard deviation. It helps detect systematic bias and random variability. A significant mean difference indicates consistent bias, while data points within the limits suggest acceptable agreement. Deviations or trends in the plot can reveal systematic errors, though the method assumes normally distributed differences, which may not always hold.

Deming regression is ideal for comparing methods with measurement errors in both variables. Unlike ordinary regression, it accounts for errors in both methods, assessing proportional error and constant error (bias). The slope and intercept of the regression line reveals biases, with alignment to the equality line indicating minimal discrepancies.

A Deming regression plot can visually demonstrate the degree of agreement between methods. If the regression line closely aligns with the line of equality (where results from both methods are identical), this suggests minimal bias. Deviations from this line highlight areas where the methods diverge.

The Concordance Correlation Coefficient (CCC) is used to assess method agreement by combining the Pearson correlation coefficient with a bias correction factor. Ranging from -1 to 1, a CCC close to 1 indicates strong agreement, reflecting both precision and accuracy and results that are consistent with true values.

9. Sample selection in cross-validation

The selection of samples for cross-validation is crucial to ensure robust statistical analysis. It is important to use samples that represent the full range of concentrations and matrices encountered in real-world applications. QC samples at low, medium and high concentrations are essential for covering the method's analytical range, while statistical power analysis can determine the necessary number of cross-validation QC samples. Including incurred samples from actual study subjects helps identify real-world variability and matrix effects. A balanced design, with equal numbers of samples analyzed by each method or laboratory, minimizes bias and ensures accurate comparability.

10. Implementing cross-validation in the post-ICH M10 environment

Given the challenges and opportunities presented by the ICH M10 guideline, implementing cross-validation in the post-ICH M10 environment requires a structured and scientifically sound approach.

The following steps outline a recommended framework for conducting cross-validation under ICH M10:

10.1. Develop a cross-validation plan

A cross-validation plan should be developed at the outset of the study process. This plan should outline the methods or laboratories to be compared, the specific data sets to be included, the statistical methods to be used and the criteria for determining comparability. The plan should also consider the specific regulatory expectations for the study or submission in question, and in addition include that the statistical calculation is done by the statistical department within a sponsor or contracted CRO. It is not recommended that these calculations are performed within a bioanalytical laboratory unless they have the statistical expertise required. Even in many statistics departments in Pharma or CROs the statistical expertise will be available but these are not standard tools used in Pharmaceuticals so the approaches may need statistical analysis packages such as SAS or R.

10.2. Define statistical methods

There is an intentional absence of acceptance criteria from ICH M10, an upfront definition of which statistical approach such as setting equivalence margins, defining limits of agreement, or establishing thresholds for acceptable differences in means and variances, should be described. The criteria should be scientifically justified and aligned with the specific context of the bioanalytical data.

10.3. Document & report cross-validation results

The data from the cross-validation study is delivered to the sponsor statistical or clinical pharmacology department. The documentation at the bioanalytical lab should include a description of the methods used, and description of samples and the bioanalytical results.

The sponsor should prepare a document which describes the statistical results of the cross-validation, which statistical analysis was conducted, the % Bias observed, and the conclusions drawn regarding the comparability of the results. Transparent and comprehensive reporting is essential to ensure that the cross-validation process is understood and accepted by regulators and other stakeholders.

The authors recommend a separate cross-validation report be written as opposed to other options which may include an amendment to a validation report or a study report. There are several reasons for this. The simplest argument might be that it is simply cleaner. The cross-validation report or data may need to be included in multiple clinical study reports and may need to be on file with a sponsor and one or more CROs. Having a stand-alone report enables much more flexibility. Additionally, the cross-validation report will often be developed as a collaboration between CROs. It may be desirable to have sign-off across CROs and sponsors which may not be deemed acceptable due to confidentiality concerns. Also, if the statistical analysis is included in the cross-validation report it becomes a much more straightforward approach to have a stand-alone report to collaborate on.

10.4. Utilize the cross-validation outcome

Cross-validation is a study to understand the statistical bias between methods. Upon completion of the cross-validation study the outcome of this should generate a bias among other descriptive statistics. This bias can be utilized over time by the biostatisticians to interpret the outcomes of studies that are being compared with the different assays now or in the future.

11. Conclusion

The implementation of ICH M10 has advanced bioanalytical PK method validation through harmonizing global recommendations but has also introduced challenges as any new guideline may do. In the case of ICH M10 one such challenge is the change in thinking about cross-validation as being an assessment study versus a test requiring the application of defined pass/fail acceptance criteria. Effective cross-validation should focus on studies involving combined or compared data for special dosing regimens. To address the absence of explicit criteria, a statistical approach is essential, utilizing methods such as Bland-Altman analysis, Deming regression and the Concordance Correlation Coefficient (CCC).

It should be recognized that this change will be difficult for many in the industry. Those that choose to continue to utilize an acceptance criteria pass/fail approach to a cross-validation assessment study need to be prepared to answer hard questions as previously discussed. It is strongly recommended that a decision tree for these questions be decided a priori. To effectively implement cross-validation in accordance with ICH M10, the bioanalytical industry must foster a closer collaboration within all participating bioanalytical laboratories as well as with the clinical pharmacology and biostatistics departments. This partnership is crucial to avoid unnecessary cross-validation for every minor change in the bioanalytical laboratory. Instead, cross-validation should be strategically focused on studies where pharmacokinetic (PK) data are combined or compared, particularly in support of special dosing regimens. By aligning efforts with clinical pharmacology, cross-validation can be more efficiently targeted, ensuring that it is performed only when it directly impacts the interpretation of critical PK data.

Financial disclosure

The authors have no financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

Competing interests disclosure

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

Writing disclosure

No writing assistance was utilized in the production of this manuscript.