Quantifying the Impact of Nasopharyngeal Specimen Quality on Severe Acute Respiratory Syndrome Coronavirus 2 Test Performance

Abstract

Background

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) reverse-transcription polymerase chain reaction (RT-PCR) cycle threshold (Ct) has been used to estimate quantitative viral load, with the goal of targeting isolation precautions for individuals with coronavirus disease 2019 (COVID-19) and guiding public health interventions. However, variability in specimen quality can alter the Ct values obtained from SARS-CoV-2 clinical assays. We sought to define how variable nasopharyngeal (NP) swab quality impacts clinical SARS-CoV-2 test sensitivity.

Methods

We performed amplification of a human gene target (β-actin) in parallel with a clinical RT-PCR targeting the SARS-CoV-2 ORF1ab gene for 1282 NP specimens collected from patients with clinical concern for COVID-19. We evaluated the relationship between NP specimen quality, characterized by late Ct values for the human gene target β-actin Ct, and the probability of SARS-CoV-2 detection via logistic regression, as well as the linear relationship between SARS-CoV-2 and β-actin Ct.

Results

Low-quality NP swabs are less likely to detect SARS-CoV-2 (odds ratio, 0.607 [95% credible interval {CrI}, .487–.753]). We observed a positive linear relationship between SARS-CoV-2 and β-actin Ct values (slope, 0.181 [95% CrI, .097–.264]), consistent with a reduction in detection of 0.181 cycles for each additional cycle of the β-actin target. COVID-19 disease severity was not associated with β-actin Ct values.

Conclusions

Variability in NP specimen quality significantly impacts the performance of clinical SARS-CoV-2 assays, and caution should be taken when interpreting quantitative SARS-CoV-2 Ct results. If unrecognized, low-quality NP specimens, which are characterized by a low level of amplifiable human DNA target, may limit the successful application of SARS-CoV-2 Ct values to direct infection control and public health interventions.

As the coronavirus disease 2019 (COVID-19) pandemic continues to drive morbidity and mortality around the world, interest has grown in using severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) reverse-transcription polymerase chain reaction (RT-PCR) cycle threshold (Ct) values as a means of quantifying viral load [1, 2]. It has been proposed that SARS-CoV-2 Ct values may correspond with viral burden and infectivity, and that SARS-CoV-2 values may be used to predict disease severity and guide isolation precautions for individuals with coronavirus disease 2019 (COVID-19) [3–7]. SARS-CoV-2 Ct values have been shown to correspond with community COVID-19 burden, and it has also been proposed that community Ct values may help to guide nonpharmaceutical interventions to control COVID-19 [8, 9].

We sought to understand the impact of nasopharyngeal (NP) specimen swab quality on the measurement of SARS-CoV-2 Ct and the sensitivity of virus detection. To collect an NP swab for SARS-CoV-2 testing, health care workers are instructed to advance a synthetic fiber swab with plastic or wire shaft through the nostril until contacting the posterior nasopharynx at a depth equal to the distance from the nostril to the opening of the ear, then to rub and roll the swab, leaving the swab in place for several seconds to collect secretions, before rotating the swab further as it is removed from the nostril [10]. Variability in practice and patient tolerance of the procedure has been observed and may impact the sensitivity of SARS-CoV-2 detection, as well as the Ct value observed when SARS-CoV-2 is detected [11–13].

To measure variability in the quality of NP swab collection, we performed amplification of a human gene target (β-actin) in parallel with RT-PCR targeting the SARS-CoV-2 ORF1ab gene. High β-actin Ct values have been previously validated as a marker of poor NP swab quality [2, 14]. Below we report the relationship between quality of NP swab collection, sensitivity of SARS-CoV-2 detection, and the range of impact we expect substandard NP swab collection may exert on SARS-CoV-2 Ct values. We also examine the possibility of confounding by greater NP epithelial cell damage associated with increasing COVID-19 disease severity.

MATERIALS AND METHODS

Study Design, Setting, and Population

We performed a retrospective cohort study of adult patients presenting to the PennMedicine health system between 26 March and 4 July 2020. The patient population included drive-through symptomatic, asymptomatic preprocedure, and hospital inpatient transfer and discharge patients. As the goal of the study was to evaluate the impact of specimen quality on SARS-CoV-2 assay sensitivity, we included only NP specimens processed by a laboratory-developed emergency use authorization (EUA) SARS-CoV-2 assay that included a human gene target (β-actin) as a specimen quality control [2, 14]. No other SARS-CoV-2 assays in the health system included a human gene target. All specimens were assayed for β-actin and MS2 phage RNA in parallel with SARS-CoV-2. We included all SARS-CoV-2 RT-PCR results for which the positive PCR control analyte (MS2 phage RNA) was detected in the target Ct range (20–25 cycles) [15]. A total of 1282 NP specimens were included.

Patient Consent Statement

A waiver of informed consent was granted by the University of Pennsylvania Institutional Review Board (IRB protocol numbers 843085 and 843274).

Causal Models

We hypothesized that β-actin and SARS-CoV-2 Ct values are related because poor NP specimen collection technique results in reduced capture of NP epithelial cells and SARS-CoV-2 alike. β-actin is a commonly used endogenous reference gene, used as an internal control for PCR reactions involving human specimens. This gene has been previously validated as a marker for the presence of nasal epithelial cells, and prior research has supported its use to assess the quality of self-collected midturbinate swabs [2, 14]. We additionally considered the possibility of confounding by COVID-19 severity of illness because those with more severe infection may have greater NP epithelial cell damage, resulting in greater detection of both PCR targets, irrespective of sampling technique.

Clinical Data Collection

To evaluate the possibility of confounding by disease severity, we measured 2 independent markers of respiratory illness: (1) the minimum room-air oxygen saturation recorded within 2 days of SARS-CoV-2 testing, and (2) whether infiltrates were observed on chest computed tomographic (CT) imaging performed within 7 days of SARS-CoV-2 testing. Per Centers for Disease Control and Prevention (CDC) guidelines [16], we considered room-air oxygen saturation <94% indicative of severe respiratory illness. We chose to use CT imaging rather than radiographic imaging to discriminate findings likely to be related to SARS-CoV-2. Radiology reports for CT imaging that described parenchymal lung disease, including “infiltrates,” “pneumonia,” “ground glass,” or other “opacities,” were considered indicative of severe respiratory illness. The presence of lung nodules, lung masses, chronic airway disease including bronchiectasis, emphysematous changes, or pleural effusions in the absence of parenchymal disease as described above, were not considered indicative of severe acute respiratory illness. We chose to include CTs performed within 7 days of SARS-CoV-2 testing to capture disease severity that could plausibly have confounded the relationship between β-actin and SARS-CoV-2 Ct values.

Specimen Collection, Processing, and RT-PCR Assay

Specimens were collected during routine clinical practice using a nylon flocked mini-tip swab collected in viral transport medium or saline [17–20]. Health care providers obtained samples using CDC guidelines for NP sample collection. Samples were transported to the laboratory at ambient temperature and stored at 4°C if not run immediately. All specimens were analyzed at the Hospital of the University of Pennsylvania’s Clinical Microbiology Laboratory using an EUA-approved laboratory-developed SARS-CoV-2 assay on the BD MAX system. The multiplex assay was designed to include a specimen quality control (β-actin) [2, 14], an internal processing control to monitor the RT and PCR steps (MS2 phage RNA) [15], and a SARS-CoV-2 target (Orf1ab). Exk TNA2 extraction reagent kits (Becton Dickinson) for the BD MAX open system reagent suite were used for the laboratory-developed SARS-CoV-2 assay based on a previously described assay [21, 22]. The BD MAX system was set to run type 1 workflow. PCR conditions consisted of a reverse transcriptase step (600 seconds at 58°C, 1 cycle), denaturation step (60 seconds at 98°C, 1 cycle) and extension steps (10 seconds at 98°C followed by 40 seconds at 63°C, 40 cycles). Ct values for all 3 targets (β-actin, MS2 phage RNA) were recorded. The acceptable Ct range for β-actin in our assay was 24–37. Samples that were negative for SARS-CoV-2 and either negative for β-actin or with a β-actin Ct >37 were reported as invalid.

Definition of Exposures and Outcomes

The primary exposure of interest was the β-actin Ct value, a surrogate for the quality of NP swab collection. The primary outcome of interest was SARS-CoV-2 Ct value.

Statistical Methods

Data were organized using R statistical software version 3.6.1 [23], and plots generated using the “ggplot2” package [24]. Where β-actin and SARS-CoV-2 were not detected, Ct values were imputed as 40 cycles. We examined the linear relationship between β-actin and SARS-CoV-2 Ct values, as well as the impact of β-actin Ct on SARS-CoV-2 detection using Bayesian linear and generalized-linear mixed-effects models, which were fit using Stan Hamiltonian Monte Carlo (HMC) version 2.21, via the “brms” package with default weakly informative priors [25, 26]. Mixed-effects models were fit in the same manner, with random slopes and intercepts according to disease severity (assessed by radiography or oxygen saturation) to evaluate the impact of disease severity on the relationship. Prior predictive modeling was performed, and models were fit with 4 chains of 1000 iterations, confirmed with HMC diagnostics (no divergent iterations, Rhat statistic <1.1 for all parameters, and estimated Bayesian Fraction of Missing Information >0.2) [27–29]. We examined parameter distributions at 50%, 80%, and 95% posterior credible intervals (CrIs) to understand the relationship between exposure and outcome variables. The posterior CrIs estimate the probability that the true parameter value lies within the interval, based on the observed data.

Power and Sample Size

We estimated the necessary cohort size based on the anticipated effect of poor NP swab quantity [30]. We anticipated that approximately 800 subjects would detect a 10% reduction in sensitivity of SARS-CoV-2 detection related to a β-actin Ct increase of 10, with CrI precision ensuring type S error <5% [31, 32]. We targeted enrollment of 10% more subjects to allow for a margin of error in that estimate, and we exceeded our enrollment target.

Availability of Data

Data, analysis scripts, and model code are available at github.com/bjklab.

RESULTS

SARS-CoV-2 Detection and Cycle Threshold Range

Of 1282 tested specimens, 134 were found to have detectable SARS-CoV-2 within 40 cycles of PCR. Among these specimens, median SARS-CoV-2 Ct was 27.9 (interquartile range [IQR], 20.4–32.9). No secular trend was observed between calendar time from local onset of COVID-19 cases and SARS-CoV-2 Ct values during the study period (Spearman correlation = 0.18). Table 1 summarizes the SARS-CoV-2 Ct values, as well as those of the specimen quality control, β-actin, and RT-PCR reaction control, MS2 RNA. Supplementary Figure 1 depicts the distribution of each target and the relationships between their distributions.

Table 1.

Severe Acute Respiratory Syndrome Coronavirus 2 and Control Cycle Threshold Values

Target	Measured Ct, No. (%) of Specimens	Median (IQR) Ct of Measured Values
β-actin	1240 (96.7%)	30.2 (28.7–31.8)
MS2	1282 (100%)	22.5 (22.1–22.9)
SARS-CoV-2	134 (10.5%)	27.9 (20.4–32.9)

A summary of the observed Ct values for SARS-CoV-2 reverse-transcription polymerase chain reaction, with MS2 RNA–positive control and β-actin specimen quality control, over 1282 consecutive clinical assays run between 26 March and 4 July 2020.

Abbreviations: Ct, cycle threshold; IQR, interquartile range; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2.

Relationship Between β-Actin and SARS-CoV-2 Cycle Threshold

We evaluated the relationship between NP specimen quality, measured by β-actin Ct value, and SARS-CoV-2 test sensitivity with logistic regression, and we found that increasing β-actin Ct values are significantly associated with reduced detection of SARS-CoV-2 (odds ratio, 0.607 [95% CrI, .487–.753]). Figure 1 shows the relationship between β-actin and SARS-CoV-2 detection probability. We further evaluated the linear relationship between β-actin and SARS-CoV-2 Ct values with linear regression, and we found that SARS-CoV-2 Ct increases significantly with β-actin Ct (slope, 0.181 [95% CrI, .097–.264]). A linear model restricted to include only the 131 specimens within which both SARS-CoV-2 and β-actin were detectable (ie, Ct <40) also found that SARS-CoV-2 Ct increased with β-actin, but this relationship did not have high posterior certainty (slope, 0.200 [95% CrI, –.295 to .720]), and linear model fit was poor.

Figure 1.

Relationship between β-actin and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) detection probability. Binomial logistic regression relating SARS-CoV-2 detection to β-actin cycle threshold (Ct) value reveals a negative association, with high β-actin Ct (ie, low quality) nasopharyngeal specimens less likely to detect SARS-CoV-2. The absolute probability of SARS-CoV-2 detection is presented in relation to the observed range of β-actin Ct values.

Impact of Poor NP Specimen Quality on SARS-CoV-2 Detection Sensitivity

To understand the potential impact of poor NP specimen quality, we evaluated the change in probability of SARS-CoV-2 detection as β-actin Ct increases. We found that a 4-Ct increase in β-actin, from Ct of 28 to Ct of 32 (roughly from the first quartile of observed β-actin Ct values to the third quartile) results in a 6.34% (95% CrI, 3.35%–9.17%) decreased probability of SARS-CoV-2 detection.

Impact of Disease Severity on Relationship Between β-Actin and SARS-CoV-2 Cycle Threshold

Considering the possibility that the observed association between SARS-CoV-2 and β-actin Ct values is confounded by respiratory illness severity, we evaluated the relationship between β-actin Ct and independent markers of respiratory illness. Oxygen saturation data were available for 425 (33.1%) subjects; chest CT imaging was available for 108 (8.4%) subjects. At least 1 measure of disease severity was available for 38 (28.3%) of the cases of detectable SARS-CoV-2. Linear regression relating β-actin Ct values to oxygen saturation revealed no significant association with linear regression slope –0.04 (95% CrI, –.273 to .182). Similarly, we found that the presence of parenchymal lung disease on chest CT radiography reports had no significant association with β-actin Ct values, with a linear regression slope 0.428 (95% CrI, –.655 to 1.61). Mixed-effects models revealed that the observed association between β-actin Ct values and probability of SARS-CoV-2 detection persists across all levels of respiratory illness, whether assessed by oxygen saturation or chest radiography (Figure 2). These analyses of independent markers of severe respiratory disease suggest that it is NP specimen quality, not disease severity, that drives the association between SARS-CoV-2 and β-actin Ct values. To further corroborate these findings, we repeated the analysis, comparing specimens collected from inpatient (n = 858) vs outpatient (n = 424) locations. We found the same association between β-actin Ct values and probability of SARS-CoV-2 detection across both specimen collection sites.

Figure 2.

Relationship between β-actin and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) detection probability, stratified by illness severity. A mixed-effects binomial logistic regression relating SARS-CoV-2 detection to β-actin cycle threshold (Ct) value reveals a negative association, with high β-actin Ct (ie, low-quality) nasopharyngeal specimens less likely to detect SARS-CoV-2, across all strata of disease severity as assessed by chest radiography (A) or oxygen saturation (B). The absolute probability of SARS-CoV-2 detection is presented in relation to the observed range of β-actin Ct values. Abbreviations: Ct, cycle threshold; CT, computed tomography; O₂, oxygen; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2.

DISCUSSION

In this study, we found that higher β-actin Ct values, which have been previously validated as a marker of low NP swab quality [2, 14], were associated with reduced probability of SARS-CoV-2 detection and with higher SARS-CoV-2 Ct values. This finding has several important implications. First, the correlation between β-actin Ct and SARS-CoV-2 suggests that quantitative interpretation of SARS-CoV-2 human specimens may be significantly impacted by the quality of specimen collection, potentially limiting the ability to compare results. In this cohort, we found that each increase in β-actin Ct was associated with an increased SARS-CoV-2 Ct of 0.181. The quantitative interpretation of SARS-CoV-2 RT-PCR Ct results may be enhanced by adjusting for the β-actin Ct. Such adjustment could be used in studies comparing results of serial testing within subjects or populations, where there is a change in β-actin Ct between specimens. Second, the data support the concern that poor specimen collection may contribute to false-negative results. The concern of false-negative NP SARS-CoV-2 testing has led to the recommendation to retest patients with moderate to high clinical suspicion of COVID-19 [33, 34]. Reporting the β-actin Ct, or a β-actin–adjusted SARS-CoV-2 Ct may allow clinicians to better interpret specimen quality when considering retesting.

We considered the possibility that the observed relationship between SARS-CoV-2 and β-actin Ct values might be confounded by respiratory disease severity, but we found no significant association between independent markers of severe respiratory disease and lower β-actin Ct values. However, several limitations of our analysis must be acknowledged. Measures of disease severity are imprecise, and oxygen saturation data and chest CT radiography reports were only available for a small percentage (32.6% and 8.5%, respectively) of our subjects. Subject demographics and medical comorbidities could not be ascertained for subjects, so unmeasured confounders may contribute to the observed association. The impact of comorbid conditions, immunocompromise, and duration of illness on the observed relationship are important areas for future study.

Despite our study’s limitations, we believe that the observed association between NP specimen quality and SARS-CoV-2 RT-PCR sensitivity is an important finding. From 1282 NP specimens submitted for SARS-CoV-2 testing, we have quantified the variation in specimen quality measured by β-actin Ct value, and we have defined the impact of the observed variation on test sensitivity and SARS-CoV-2 Ct values. The original CDC SARS-CoV-2 assay included a human gene internal control, ribonuclease P (RNAse P). However, most commercial SARS-CoV-2 assays do not include a human gene target. Our findings suggest that caution is required in the clinical interpretation of Ct values from such assays.

SARS-CoV-2 Ct values have shown promise as a means to roughly quantify viral burden and so to guide infection control and public health interventions [1, 2, 4–9]. However, variability in NP specimen collection may exert large effects on observed SARS-CoV-2 Ct values, limiting these useful applications. As testing efforts expand, infrastructure to ensure quality sample collection must expand as well [10]. Concurrent measurement of a β-actin human gene target may provide a means to recognize and adjust for variability in NP specimen quality.

Supplementary Data

Supplementary materials are available at Open Forum Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Notes

Acknowledgments. B. J. K., M. J. Z., and E. L. are supported in part by the Centers for Disease Control and Prevention (cooperative agreement number FOA CK16-004, Epicenters for the Prevention of Healthcare Associated Infections). B. J. K. is supported by the National Institute for Allergy and Infectious Diseases (grant numbers K23 AI121485 and L30 AI120149). M. J. Z. is supported by the National Institute for Allergy and Infectious Diseases (grant number K23 AI143925).

Author contributions. Study design: M. R. G., M. J. Z. Data collection: M. R. G., V. B., E. H., H. A., P. T., L. G., B. J. K. Data analysis: M. J. Z., E. L., B. J. K. All authors participated in the preparation of the manuscript.

Data availability. Data, analysis scripts, and model code are available at github.com/bjklab.

Potential conflicts of interest. All authors: No reported conflicts of interest.

All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.