MOVER approximated CV: a tool for quantifying precision in ratiometric droplet digital PCR assays

Abstract

Droplet digital PCR is a particularly valuable tool for ratiometric assays because it provides simultaneous absolute quantification of two target sequences in a single assay. This manuscript addresses a challenge in establishing a new ratiometric droplet digital PCR assay for use in sputum, the rRNA synthesis ratio. In principle, the methods established to evaluate precision and determine the limit of quantification for a single measurand cannot be applied to a ratiometric assay. The precision of a ratio depends on precision in both the numerator and denominator. Here, we evaluated the MOVER approximated coefficient of variation as indicator of assay precision that does not require technical replicates. We estimated the MOVER approximated coefficient of variation in dilution series and routine assays and evaluated its agreement with the traditional coefficient of variation. We found that the MOVER approximated coefficient of variation was able to recapitulate the traditional coefficient of variation without the requirement for replicate assays. We also demonstrated that the MOVER approximated coefficient of variation threshold can be used to define the limit of quantification of the rRNA synthesis Ratio. In conclusion, the MOVER approximated coefficient of variation may be useful not only for the rRNA synthesis ratio but for other assays that measure ratios via droplet digital PCR.

1. Introduction

Quantification of the ratio between two nucleic acid sequences is the basis of several clinical or research assays [1-5]. An advantage of ratiometric assays is that ratios are inherently “self-normalizing,” thereby controlling for between-sample variability in inconsistent samples like sputum.

Droplet digital polymerase chain reaction (dPCR) is a particularly valuable tool for ratiometric assays because it provides simultaneous absolute quantification of two target sequences in a single reaction. Unlike qPCR, dPCR enables direct calculation of the ratio between two targets without dependence on external standard curves. Additionally, dPCR is more sensitive than qPCR and less susceptible to PCR inhibition in complex matrices such as human or environmental samples [5-9]. The dPCR technique is currently used in an FDA-approved clinical assay that quantifies the ratio of BCR-ABL to ABL transcripts in human blood to monitor treatment response and relapse risk in patients with chronic myeloid leukemia (CML) [10].

This manuscript addresses the challenge of quantifying precision and establishing thresholds for data quality control in a ratiometric dPCR assay applied to a heterogenous sputum matrix. We have developed an assay called the RS ratio for use as a pharmacodynamic (PD) marker in tuberculosis (TB). By quantifying the abundance of M. tuberculosis (Mtb) precursor rRNA (pre-rRNA) relative to the burden of 23S rRNA, the RS ratio indicates the degree of ongoing bacterial ribosomal RNA (rRNA) synthesis in the sputum of TB patients [11]. The RS ratio represents a fundamentally new measure of treatment efficacy. Unlike conventional PD markers which enumerate the burden of Mtb capable of growth in culture [12], the RS ratio evaluates the effect of drugs and regimens on a basic cellular process of the pathogen. The RS ratio appears to indicate the treatment-shortening potency of drugs and regimens [11] and therefore promises to accelerate the development of new, shorter and more effective TB treatments [12].

In principle, the methods established to determine the limit of quantification (LOQ) for a single measurand (such as a single target nucleic acid sequence) are not applicable to ratiometric assays [13-16]. For a single measurand, absolute values of the measurand typically correlate with precision of measurements. The lower the value of the measurand, the greater the variability in measurement. A threshold value for the measurand can be established, below which results are insufficiently precise and classified as below the LOQ. By contrast, the value of a ratio alone does not indicate precision. The precision of a ratio depends on precision in both the numerator and denominator. Conceptually, it is possible for a low ratio to be highly precise and a high ratio to be imprecise.

For a ratiometric assay in a more homogenous sample type like blood, this concept may not be a practical concern. For example, in the blood CML assay, the abundance of ABL transcript per microgram of RNA is relatively consistent across samples, irrespective of disease state [17,18]. Because the denominator is functionally static, the value of the blood BCR-ABL/ABL ratio is directly correlated with precision, the same as it would for a single measurand. By contrast, in TB the denominator is highly inconsistent. Even prior to treatment, the sputum burden of culturable Mtb in patients with active disease may range from undetectable to 10⁹ organisms per milliliter [12,19]. Since the denominator of the RS ratio (23S rRNA) is a measure of Mtb burden, variability in bacterial burden influences precision of the RS ratio. As a result, the value of the RS ratio alone is not directly correlated with precision as it is with the BCR-ABL/ABL assay.

Precision in molecular assays is usually expressed as the coefficient of variation (CV) of a series of measurements [16]. Although the level of precision needed can vary depending on the type of assay and intended use, a CV less than 30% is generally considered an acceptable level of precision [20]. One method for quantifying the precision of the RS ratio would be to routinely perform multiple technical replicates for every assay and calculate the conventional CV which this manuscript will refer to as the “True” CV (CV_True). However, dPCR has been shown to be highly repeatable [21-24] and technical replicates are not routinely recommended [25]. Technical replicates add expense and labor and may not be feasible in low-abundance samples. The RS ratio therefore demanded an alternative indicator of assay precision that does not require technical replicates.

In this paper, we establish and evaluate a new precision metric for ratiometric dPCR assays called the method of variance estimates recovery (MOVER) approximated CV (CV_MA). The CV_MA has been chosen to describe precision in a way similar to the traditional CV_True without the requirement for replicate assays. The CV_MA threshold provides a useful new measure of uncertainty around a ratio estimate and can be used to define the LOQ of the RS ratio. The CV_MA may be useful not only for the RS ratio but for other assays that measure ratios via dPCR.

2. Materials and methods

2.1. Conceptual approach

Two distinct approaches were used to evaluate CV_MA as surrogate for CV_True. First, the relationship between CV_MA and CV_True was interrogated in a 10,000-fold dilution series using two sputum RNA samples. Second, the same relationship was evaluated in sputum RNA samples that underwent RS ratio assays in our routine workflow. The RS ratio assay was previously described [11] and has been summarized in Supplemental Material.

2.2. Description of sample set

This analysis used RNA-preserved human sputa collected longitudinally in Study 31, an international, multicenter, randomized, open-label, phase 3, noninferiority trial conducted at sites of the Centers for Disease Control and Prevention Tuberculosis Trials Consortium and the National Institutes of Health AIDS Clinical Trials Group [26]. Participants provided written informed consent for the use of their sputa for research. Details of supervising institutional review boards is provided in Supplemental Material.

2.3. Description of dilution series

Serial dilutions were performed using two Mtb-infected sputum RNA samples, one known to have a high RS ratio, and another known to have a low RS ratio. The low and high RS ratio samples were serially diluted 2 to 200-fold and 4 to 10,000-fold, respectively, generating up nine dilution levels. The RS ratio was assayed in the dilutions in sextuplicate.

2.4. Evaluation in routine workflow

In our routine workflow, we first conducted a screening qPCR assay to estimate the burden of Mtb 23S rRNA then quantified the RS ratio in a singlicate dPCR assay. RNA extraction and quantification of 23S rRNA and RS ratio are described in Supplemental Material. dPCR was conducted with replicates if the screening qPCR showed the burden of rRNA was very low (i.e., < 200 copies) or if the CV_MA in the singlicate assay was >30%. These replicate assays are the basis for our comparison of CV_MA and CV_True in routine use. The usefulness of the CV_MA to define the LOQ of the RS ratio was evaluated in routine samples.

2.5. Description of dPCR statistics for duplex assays

dPCR partitions each sample into ~20,000 droplets that undergo endpoint PCR and are individually classified as having presence or absence of the target sequence. Using these ~20,000 binary events, the QuantaSoft software package (Bio-Rad, AP v1.0) applies Poisson statistics to estimate the absolute copy number of each target in the sample with a 95% confidence interval (CI). QuantaSoft calculates the ratio between the two targets and uses the method of variance estimates recovery (MOVER) Fieller's theorem to calculate a 95% CI around the ratio (Personal Communication, Bio-Rad) [27-29]. The 95% CI alone is of limited usefulness for evaluating precision because it is not “scaled” relative to the ratio. To avoid leading zeros, the ratio provided by QuantaSoft is multiplied by 10,000, resulting in the RS ratio.

2.6. Calculation of CV_True and CV_MA

CV_True was calculated in replicate assays using the mean and standard deviation of the ratios. Calculation of CV_MA uses the ratio and 95% CI as presented in situ in QuantaSoft. To calculate CV_MA the half-width of the 95% CI around the ratio is first divided by the appropriate standard normal value (1.96), resulting in a term that is the conceptual equivalent of the standard deviation. This term is then divided by the value of the ratio to give the CV_MA. Calculation of CV_MA does not require technical replicates. Therefore, the median CV_MA across technical replicates was selected to evaluate the relationship between CV_MA and CV_True.

$${\text{CV}}_{\text{MA}}=\frac{(\text{Upper}95\%{\text{CI}}_{\text{ratio}})-(\text{Lower}95\%{\text{CI}}_{\text{ratio}})}{2\times 1.96\times \text{ratio}}$$

2.7. Relationship between CV_MA and CV_True

Pearson correlation and Bland-Altman analyses were used to evaluate the relationship between CV_MA and CV_True. The practical impact of using CV_MA to define the LOQ of the RS ratio was tested based on the CV_MA rather than the CV_True as follows. Applying a conventional approach, samples with RS ratio CV_True ≤ 30% or > 30% were classified as quantifiable and non-quantifiable, respectively. This classification based on CV_True served as our reference standard. We then tested whether a CV_MA of 30% would provide the same classification as CV_True. Receiver operating characteristic (ROC) analysis was further used to test the practical effect of using CV_MA thresholds other than 30% to classify samples as quantifiable versus non-quantifiable. In the context of ROC analysis, sensitivity was defined as the proportion of samples that were quantifiable based on CV_True and were correctly classified as quantifiable based on CV_MA. Specificity was defined as the proportion of samples that were non-quantifiable based on CV_True and were correctly classified as non-quantifiable based on CV_MA. Both CV_True and CV_MA were reported as percentages. A p-value < 0.05 was considered as sufficient evidence for a real association. Statistical analysis was conducted in R (v 3.5.3; R Development Core Team, Vienna, Austria).

3. Results

3.1. RS ratio in dilution series

RS ratio results in up to 10,000-fold dilution of two sputum specimens are shown in Table 1 and Fig. 1a. For both the low and high RS ratio samples, quantification of RS ratio was highly precise (CV_True <10%) in minimally diluted samples (<10-fold dilution for the low RS ratio sample and <1000-fold for the high RS ratio sample). At progressively higher dilutions, precision decreased (reflected by increasing CV_True). Precision decreased more rapidly with higher dilutions for the low RS ratio sample than for the high RS ratio sample. Consistent with the established performance characteristics of dPCR [25], the CV_True for individual pre-rRNA and 23S rRNA targets increased substantially once the absolute copy number of either target was less than 50. The low RS ratio sample showed that if the CV_True of the denominator was low (i.e., the denominator was estimated with precision), then precision in the RS ratio followed precision of the numerator (i.e., pre-rRNA) (Fig. 1b). By contrast, in the high RS ratio sample, precision depended on precision of both the numerator and the denominator (i.e., pre-rRNA and 23S rRNA, respectively) (Fig. 1c).

Table 1.

Droplet digital PCR results from a dilution series of RNA from two human sputum samples selected for their low and high RS ratio estimates. Mean copy numbers and CV_True. for the pre-rRNA numerator and 23S rRNA denominator illustrate the effect of decreasing target abundance on precision and effect on CV_MA and CV_True for the RS ratio. Six replicates were used for the calculation of each mean and CV_True except for the 200-fold low RS ratio dilution (N=3) and the 10,000-fold high RS ratio dilution (N=5).

Sample	Dilution	Pre-rRNA		23S rRNA		RS ratio
		Mean	CVTrue	Mean	CVTrue	Mean	CVTrue	CVMA
Low RS ratio sample	Ref.	614	7%	79,472	3%	77	6%	5%
	2-fold	301	5%	38,612	2%	78	5%	7%
	6-fold	106	8%	14,234	5%	74	5%	12%
	10-fold	30	23%	5,725	2%	53	23%	21%
	20-fold	18	22%	2,904	2%	62	23%	28%
	60-fold	7	39%	1,081	5%	63	40%	47%
	100-fold	3	33%	432	4%	66	33%	65%
	200-fold	2*	50%	222	5%	80†	57%†	86%†
High RS ratio sample	Ref.	23,921	3%	166,174	4%	1,440	3%	4%
	4-fold	7,153	7%	49,491	6%	1,445	2%	2%
	10-fold	2,780	3%	18,702	2%	1,487	3%	2%
	40-fold	744	3%	4,978	4%	1,495	2%	5%
	100-fold	518	4%	3,220	5%	1,612	7%	5%
	400-fold	161	12%	911	6%	1,768	9%	10%
	1,000-fold	34	15%	177	17%	1,946	17%	20%
	4,000-fold	9	37%	42	22%	2,127	33%	46%
	10,000-fold	3‡	43%	14	54%	2,323§	52%§	58%§

^*3 of 6 samples had zero pre-rRNA copies so RS ratio could not be calculated.

^†Based on the 3 replicates for which RS ratio could be calculated.

^‡1 of 6 samples had zero pre-rRNA copies so an RS ratio could not be calculated.

^§based on the 5 replicates for which RS ratio could be calculated.

Copy number refers to copies per 20uL well.

Fig. 1. Evaluation of CV_MA as alternative for CV_True in dilution series.

A, RS ratio results in a dilution series of a human sputum sample with high RS ratio (black circles) and a sputum sample with low RS ratio (blue triangle). For the low RS ratio sample, variability increased at a lower level of dilution than for the high RS ratio sample. Horizontal lines show median. B, Precision of the low RS ratio sample in dilution series. For the low RS ratio sample, precision of the RS ratio (blue) was primarily driven by precision of the numerator, (i.e., pre-rRNA, red). 23S rRNA (green) was relatively less variable. C, Precision of the high RS ratio sample in dilution series. For the high RS ratio sample, precision of the RS ratio (blue) depended on precision of both the numerator (i.e., pre-rRNA, red) and the denominator (i.e., 23S rRNA, green). D, Correlation between CV_MA and CV_True in both dilution series. A scatterplot showed a strong correlation between the two variables (Pearson correlation = 0.96). The 45-degree diagonal dashed line is a reference line that shows perfect equality between the two variables. E, Agreement between CV_MA and CV_True in both dilution series. A Bland-Altman plot showed a high agreement between CV_MA and CV_True. The middle dashed line represents the bias (mean difference) between CV_MA and CV_True. The upper and lower dashed lines represent the 95% limits of agreement.

3.2. Relationship between CV_MA and CV_True in dilution series

CV_MA and CV_True were strongly correlated (Pearson correlation = 0.96) in dilution series (Fig. 1d), suggesting a strong agreement between the two metrics. At higher dilutions, CV_MA slightly over-estimated variability relative to CVTrue. A Bland-Altman plot further confirmed the agreement between CV_MA and CV_True (Fig. 1e). The Bland-Altman bias (i.e., mean difference between CV_MA and CV_True) was 6% (95% CI: −13% to 25%), again illustrating that on average, CV_MA tends to slightly overestimate CV_True.

3.3. Relationship between CV_MA and CV_True in routine workflow

In our routine testing of 2,239 sputa, 244 (10.9%) samples met quality criteria to be tested in replicate assays and were therefore available for use in this analysis. In this practical evaluation, CV_MA and CV_True were also strongly correlated (Pearson correlation = 0.70) (Fig. 2a). Consistent with the dilution series results, a Bland-Altman analysis also showed a high agreement between CV_MA and CV_True (Fig. 2b). The Bland-Altman bias was 5% (95% CI: −18% to 29%).

Fig. 2. Evaluation of CV_MA as surrogate of CV_True in routine assays.

A, Correlation between CV_MA and CV_True in routine assays. A scatterplot confirmed a strong correlation between the 2 variables (Pearson correlation = 0.70, P < 0.01). The 45-degree diagonal dashed line is a reference line that shows perfect equality between the two variables. B, Agreement between CV_MA and CV_True in routine assays. A Bland-Altman plot confirmed a high agreement between CV_MA and CV_True. The middle dashed line represents the bias (mean difference) between CV_MA and CV_True. The upper and lower dashed lines represent the 95% limits of agreement.

3.4. Determination of LOQ based on CV_MA in routine samples

A CV_MA threshold of 30% roughly recapitulated the reference classification based on CV_True. This means, in a single dPCR assay, samples would be classified as quantifiable and non-quantifiable when CV_MA <=30% and CV_MA >30%, respectively. With this threshold, 91% of non-quantifiable samples based on CV_True were correctly classified as non-quantifiable based on CV_MA (i.e., specificity = 91%, Table 2). The 30% CV_MA threshold also achieved a high classification accuracy (74%) while keeping the false positive rate reasonably low (9%). None of the other CV_MA thresholds tested resulted in a meaningfully better recapitulation of the reference classification. Relative to a CV_MA threshold of 30%, a CV_MA threshold of 35% achieved a slightly greater classification accuracy (76%), but the specificity was 19% lower (72%), and the false positive rate was three times higher (28%).

Table 2.

CV_MA thresholds with their ability to correctly identify quantifiable versus non-quantifiable samples in routine assays.

CVMA thresholds	Sensitivity	FN rate	Specificity	FP rate	Accuracy
20%	42%	58%	99%	1%	62%
25%	51%	49%	95%	5%	67%
30%	65%	35%	91%	9%	74%
35%	78%	22%	72%	28%	76%
40%	89%	11%	45%	55%	73%

FN = false negative; FP = false positive.

Fig. 3 illustrates the implications of applying a CV_MA threshold of 30% in our routine workflow. A small number of samples were false positive (N = 8; 3% of total samples), meaning they were erroneously classified as quantifiable based on CV_MA but were non-quantifiable based on CV_True. Among these eight samples that CV_MA classified as quantifiable, CV_True was only modestly >30% (median = 35%, min = 31%, max = 41%). A larger number of samples were false negative (N = 56; 23% of total samples), meaning that they were erroneously classified as non-quantifiable based on CV_MA but were quantifiable based on CV_True. The majority of samples had consistent classification with both CV_MA and CV_True (N = 180; 74% of total samples).

Fig. 3. Classification of routine assays based on CV_MA and CV_True.

Red and blue dots represent false positives and false negatives, respectively. Gray dots represent true positives (bottom left quadrant) and true negatives (top right quadrant). Dashed lines indicate 30% thresholds for both CV_MA and CV_True. The proportions of false positive, false negative and correctly classified samples are 9%, 35% and 74%, respectively.

4. Discussion

This work addressed the need for a practical measure of precision for a ratiometric dPCR assay in sputum that does not require multiple technical replicates. Our results showed that CV_MA approximates CV_True. CV_MA slightly over-estimated actual variation relative to CV_True, making it a conservative alternative measure. CV_MA offers a practical solution to a challenge in use of a powerful new technology.

We anticipate that CV_MA will be particularly useful for ratiometric assays in sample types such as sputum in which the abundance of targets is highly variable. Our results demonstrated that it is possible to generate imprecise ratios under two circumstances: (1) when the absolute copy number for the numerator approaches single digits, irrespective of the value of the denominator (e.g., in a sample with a low ratio) or (2) when the absolute copy number of both the numerator and denominator values are low (e.g., in a sample with a high ratio but low abundance of both targets). We confirmed that the value of the ratio itself is not a proxy for precision. A low ratio may be measured with a high degree of precision and a high ratio may be measured imprecisely. The CV_MA provides a practical solution to assure that ratiometric assay results meet an acceptable precision threshold. Although calculation of the CV_MA may not be essential for ratiometric assays such as the BCR-ABL test that have highly consistent denominators, it may nonetheless be a useful adjunctive confirmation for all ratiometric dPCR assays.

These findings highlight two types of misclassified results: (1) false positives that the CV_MA erroneously classified as quantifiable and (2) false negatives that the CV_MA erroneously classified as non-quantifiable. False positives are problematic because they over-estimate the precision with which a ratio is estimated. However, analysis of our routine workflow showed that when samples are erroneously classified as quantified, their CV_True is typically only marginally higher than the acceptable 30% variability threshold, suggesting that this misclassification may be of limited consequence.

With a CV_MA threshold of 30%, we found a false negative rate of 35%. Importantly, this does not indicate that 35% of all results are erroneously rejected. The analysis presented here included only the small subset of samples that had features suggesting imprecision (10.9% of all samples tested). We consider it appropriate to be conservative in assessment of this subset of potentially problematic samples. Additionally, erroneous rejection in a singlicate assay does not mean the sample necessarily is unreportable because a “rescue strategy” exists. In routine practice, when a single assay has CV_MA >30%, suggesting insufficient precision, it is repeated with technical replicates which are merged in QuantaSoft to enhance precision [25], frequently resulting in CV_True <30% and acceptance of the result. For the RS ratio, we therefore selected an LOQ (CV_MA =30%) that favored specificity over sensitivity, resulting in very few false positives. This strategy and workflow assure that assays meet a consistent standard for precision while minimizing the number of replicate assays that must be performed.

This work has several limitations. First, while our goal is a practical measure of precision to be used routinely for all samples, the set of routine assays analyzed here had a disproportionate over-representation of samples with very low Mtb abundance. Our routine workflow results in replicate assays primarily for the most variable samples. Despite an unusually variable sample set that makes this a “worst case” analysis, a high agreement was observed between CV_MA and CV_True. Second, the selection of a CV_MA threshold of 30% as the LOQ for the RS ratio requires a judgement about the relative importance of sensitivity and specificity. A conservative approach was used to maximize specificity and minimize false positives, but other thresholds could also be acceptable. Finally, this analysis addresses precision only at the dPCR step of the RS ratio, not variability that might in the process of RNA extraction or reverse transcription. An advantage of a ratiometric assay is that variation in RNA recovery does not systematically affect the RS ratio unless there is differential loss of the pre-rRNA or 23S rRNA which we have not observed.

In summary, we have used the output of a single dPCR assay (i.e., tens of thousands of binary results) to create a proxy for the traditional CV without the requirement for replicate assays. This is an important step for the development of the RS ratio as a marker of treatment response for TB regimen development since it shows that we now have a reliable measure of precision for this important marker. The CV_MA may be a practical tool not only for the RS ratio but for other assays that measure ratios via dPCR.

Highlights.

• The precision of a ratio depends on precision in both the numerator and denominator

• The value of a ratio alone does not indicate precision

• A low ratio value can be highly precise and a high ratio value can be imprecise

• Precision methods for a single measurand cannot be applied to ratio measurements

• Established methods for quantification limit are not applicable to ratio assays