Summary of Confirmation Cut Point Discussions

Abstract

A subgroup of AAPS NBC Immunogenicity Workshop attendees met to discuss the current recommendations in white papers and guidance documents, to describe and discuss current practices, and to resolve concerns as to the biologically and statistically appropriate approaches to determining a confirmatory cut point for immunogenicity assays. This is a summary of our discussions and recommendations.

INTRODUCTION

Many of us who are responsible for assessing the immunogenicity of biotherapeutics have been reading the regulatory guidance documents and white paper recommendations with continued interest. When Mire-Sluis et al. (1) was published in 2004, we compared what was recommended to our own current practices, and if different, questioned which approach was “right,” or “most scientifically valid”. We performed internal experiments and held discussions with our colleagues across the industry. Six years later, the biopharmaceutical industry appears to have synergized around the Mire-Sluis recommendations, with additional clarity provided by Koren et al. (2) and the help of several AAPS workshops. That’s not to say that we are all in agreement with every recommendation. However, we appear to have agreed upon the overarching strategies of immunogenicity assessment, and we have learned how to be flexible on other points. For example, there are those of us who see value in reporting anti-drug antibody results in mass unit concentrations, while the recommendations and guidance support the reporting of titers. Scientific caveats have been included in the documents to allow for this difference of opinion, and some may report both to ensure completeness.

The confirmatory assay and establishment of a confirmation cut point (or specificity cut point) have also been topics of considerable discussion. Mire-Sluis et al. recommended the confirmation assay as an assurance of assay sensitivity and as a “characterization step” in study sample analysis. “A screening assay that picks up 5% positives that are subsequently shown to be due to non-specific binding in a confirmatory (immunodepletion) assay provides assurance that true low positives can be detected.” (1).

Although we all agree with the goal of developing a specific and sensitive assay, again, we vary on the approach. Even the term “immunodepletion” is interpreted differently, with some viewing it as removal of immunoglobulins, others as removal of drug-specific binding, and still others as demonstration that the signal is not nonspecific binding. A majority of our colleagues have adopted the process of competitive drug inhibition as the confirmatory assay. This is not to be confused with the demonstration of specificity during the validation of the screening assay, although it is similar in concept. To avoid confusion, we will use the term Confirmation Cut Point (CCP). Multiple white paper authors have recommend competitive drug inhibition for the confirmatory assay (1–3), and Wakshull and Coleman (4) have spent considerable effort assessing competitive drug inhibition as an orthogonal assay for confirmation.

There are acknowledged weaknesses in the competitive drug inhibition approach, such as the incomplete inhibition properties of low-affinity IgM and the inability to confirm that the response is specifically immunoglobulin binding to drug (“false” immunodepletion due to presence of target in the sample which is inhibited with the addition of excess drug). However, the alternatives also have advantages and disadvantages which are comprehensively reviewed by Swanson and Chirmule (5) and are outside the scope of this Commentary. Those who choose to use other approaches are advised to do so with scientific justification.

THE ISSUE

The issue surrounding the competitive drug inhibition approach is: What degree of signal change constitutes confirmation that a screen assay-positive sample is specifically positive for antibodies to the drug?

Our discussions focused on the determination of an appropriate confirmation cut point; the threshold at which a screen-positive is confirmed to be a “true” positive as opposed to a nonspecific binding “false” positive. There are two widely recognized approaches for determination of a CCP using competitive drug inhibition. One involves characterizing a positive response while the other characterizes the negative population. The discussions revolved around which of these two methods was more appropriate for its intended purpose.

THE DISCUSSION

Characterizing a “Positive”

This method involves the spiking of a known concentration of drug into low, medium, and high anti-drug antibody (ADA)-positive controls and a statistical evaluation of the signal difference observed between non-spiked and spiked samples in the screening assay format. The results are considered to be predictive of the degree of inhibition characteristic of a “true positive”.

From a bioanalytical standpoint, we are trained to prove we can measure a positive response by demonstrating the expected result with a known positive or standard. In the confirmatory assay, we can demonstrate an inhibition of the assay signal of the ADA-positive control through the addition of soluble drug. At issue with this approach is the selection of the positive control. It is impossible to generate a single positive control that can reflect all of the variable characteristics of a positive anti-drug antibody sample, such as proportion of isotypes and subclasses, affinity and avidity for the drug, etc. Because of these variables, the calculated CCP using monoclonal antibodies will differ from each other as well as differ from the CCP of polyclonal antibodies of individual subjects. Therefore, the calculation of a CCP using a positive control will always be relative to the characteristics of that positive control, generating an artificially low or high CCP that may not be relevant to the ADA-positive samples within a study.

Characterizing a “Negative”

This approach utilizes the naïve population, rather than the ADA-positive control, to establish the drug inhibition characteristics of a “treatment naïve” sample. This method is a variation of the approach used for determining the screening assay cut point which is specifically addressed in Shankar et al. (3). The screening assay uses the drug-naïve population to identify a threshold above which “positives” are identified with 95% confidence of a 5% false positive rate. It seems counterintuitive to also use the drug-naïve population to establish a threshold for confirmation of ADA positives, but it may add a level of consistency in selection of cut points during tiered testing (screen, confirm, and characterize). This approach involves the calculation of signal change for drug-naïve population samples with a known quantity of drug added as compared to the same samples without drug. The signal difference is statistically evaluated, and a “specificity cutpoint” is determined. Examples of these calculations are published by Shankar et al. (3). The purpose of this exercise is to characterize a “normal” drug-inhibited response in a naïve population and to establish a signal change threshold (cut point). If a sample inhibition exceeds this normal variation, it is considered “positive”.

There are limitations associated with this approach as well. It is acknowledged that the assessment of inhibition of signal is occurring at or below the ADA screening assay lower limit of sensitivity. The variation of signal that is observed is a combination of assay and biologic variation. That is actually the intent of the approach; to determine what falls outside of this normal variation.

The selection of the appropriate “drug naïve” population is important. So, how does one establish the appropriate sample set? Confirmatory cut points calculated from healthy versus disease state naïve samples may be similar or very different. The CCP should be calculated from the same population type as the study samples. A drug-naïve population is not necessarily a “negative” population. Many anti-drug antibody assays have detected pre-existing drug-reactive antibodies that are confirmed by competitive drug inhibition. If there is a low incidence of ADA-specific “positives” in the drug-naïve population, they may be removed as biological outliers from the CCP calculation. However, there are instances where a large percentage of the drug-naïve population is positive as determined by drug inhibition. In this situation, scientific judgment is necessary in order to set the most appropriate CCP.

If the naïve population is used to establish the CCP, it is recommended that during CCP validation, at a minimum, a low ADA-positive control be evaluated against the calculated CCP to determine if known ADA positives will be confirmed as “true positives.” This also helps to gain confidence in the confirmatory assay sensitivity and avoids the risk of overlooking clinically relevant pre-existing or treatment-emergent ADA.

An additional step for confirming a true positive, which has been discussed but not currently encouraged by many experts in the field, is the comparison of signal inhibition of a pre-treatment sample to the post-treatment ADA-positive sample. Declaration of a confirmed positive would require that a sample exceeds the CCP threshold AND exceeds pre-treatment sample signal inhibition values. This comparison addresses the issue of pre-existing ADA and confirmation of drug-induced ADA-positive samples. However, one should not assume that pre-existing ADA have the same characteristics as treatment-emergent ADA nor should we assume that pre-existing ADA are not clinically relevant.

THE OUTCOME

At this time, the approach recommended by Shankar et al. (3) appears most relevant. The group also advises the use of ADA-positive controls to confirm the performance of the CCP during validation of the confirmatory assay. Scientific arguments for alternate approaches may exist, and the recommended approach may not be appropriate in all situations. It is, therefore, recommended that regulatory authorities be consulted prior to initiation of these activities.

Confirmatory cut point estimation is only one of the many challenging aspects of understanding immunogenicity, and there is a real need for more publications exploring the implementation of appropriate and valid methodologies. It must be remembered that ultimately, all data need to be evaluated in the context of other study parameters such as pharmacokinetics, pharmacodynamics, and unexpected adverse events to verify the presence of ADA and to evaluate the impact on the study, and ultimately, patient safety.

DISCUSSION PARTICIPANTS:

• Leslie Abad, ImClone

• Dan Baltrukonis, Pfizer

• Ron Bowsher, B2S Consulting

• Dan Coleman, Genentech

• Viswanath Devanarayan, Abbott

• Deborah Finco, Pfizer

• Boris Gorovits, Pfizer

• Ralf Loebbert, Abbott

• Jeremy Ma, XOMA

• Robin Marsden, Ambrx

• Mike Moxness, Amgen

• Gopi Shankar, Centocor Research & Development, Inc

• Holly W. Smith, Eli Lilly and Company

• Eric Wakshull, Genentech