Test, trace, isolate: evidence for declining SARS-CoV-2 PCR sensitivity in a clinical cohort

Abstract

Real-time reverse transcription-polymerase chain reaction (RT-PCR) on upper respiratory tract (URT) samples is the primary method to diagnose SARS-CoV-2 infections and guide public health measures, with a supportive role for serology. We reinforce previous findings on limited sensitivity of PCR testing, and solidify this fact by statistically utilizing a firm basis of multiple tests per individual. We integrate stratifications with respect to several patient characteristics such as severity of disease and time since onset of symptoms. Bayesian statistical modelling was used to retrospectively determine the sensitivity of RT-PCR using SARS-CoV-2 serology in 644 COVID-19-suspected patients with varying degrees of disease severity and duration. The sensitivity of RT-PCR ranged between 80% − 95%; increasing with disease severity, it decreased rapidly over time in mild COVID-19 cases. Negative URT RT-PCR results should be interpreted in the context of clinical characteristics, especially with regard to containment of viral transmission based on ‘test, trace and isolate’. Keywords: SARS-CoV-2, RT-PCR, serology, sensitivity, public health

1. Introduction

COVID-19 is diagnosed primarily by testing upper respiratory tract (URT) samples with real-time reverse transcription-polymerase chain reaction (RT-PCR) (Bhimraj et al., 2020). Experience with nucleic acid amplification tests for other respiratory viruses, such as influenza virus, granted a high level of confidence in the clinical sensitivity of these types of assays (Eigner et al., 2019). In the SARS-CoV-2 pandemic, however, frequent false-negatives were reported by physicians worldwide (Krumholz, 2020, Woloshin et al., 2020) and were indicated to significantly complicate healthcare organization, hospital admission and isolation capacity (Murk et al., 2020, Reusken et al., 2020).

In several systematic reviews, the false-negative rate of RT-PCR was calculated to range between 22% to 66%, depending on symptom duration (Arevalo-Rodriguez et al., 2020, Kucirka et al., 2020). It is not known whether false-negatives were due to methodological problems, such as sampling error, suboptimal handling of samples or suboptimal assay design, or if they reflect a biological feature of SARS-CoV-2 infections. Clinical samples of lower respiratory tract material such as sputum or broncho-alveolar lavage fluid (BALF) appeared to yield significantly lower false negative rates than URT samples in patients with COVID-19 pneumonia (Liu et al., 2020, Wang et al., 2020, Yang et al., 2020). However, only 13% to 30% of patients produce sputum (Murk et al., 2020) and obtaining BALF is often impractical. URT samples are therefore preferred. Estimates of sensitivity are further complicated by the lack of a reliable gold standard. Although imaging techniques may aid in the diagnosis of COVID-19 (Ai et al., 2020, Pan et al., 2020), their sensitivity and specificity are insufficient to be used as such.

As compared to previous studies on the sensitivity of PCR and serological tests, our approach integrates some features that are not always properly accounted for. Firstly, we do not employ a gold standard, but rather draw information from the results of multiple tests on a single individual. Secondly, severity and duration of symptoms are modelled, as well as sex, age, and immunocompetence. Finally, our use of Bayesian statistics enabled us to retrospectively determine the clinical sensitivity of RT-PCR in URT-samples from a large cohort of COVID-19-suspected patients including a full characterization of uncertainty in the outcomes.

Although complex, it is crucial to reliably assess the sensitivity of RT-PCR in different clinical cohorts as RT-PCR is the foundation for test, trace and isolation policies that are the cornerstone of worldwide pandemic control efforts. Highly sensitive and specific antibody assays are important tools to support patient diagnostics at a later stage of the disease.

2. Methods

2.1. Patient cohort, setting and data collection

The study was performed in the Elisabeth-TweeSteden Hospital, Tilburg, The Netherlands, a 996-bed Dutch teaching hospital and tertiary referral center located in the central part of the Noord-Brabant province. Adults tested for SARS-CoV-2 real-time reverse transcriptase polymerase chain reaction (RT-PCR) on at least one combined naso- and oropharyngeal upper respiratory tract swab (URT-swab) between March 15th - April 15th 2020, were eligible for enrolment. These weeks represent the peak of the coronavirus epidemic in the Noord-Brabant province. Inclusion criteria were a minimum age of 18 years and either respiratory tract or gastrointestinal symptoms compatible with COVID-19 (coughing, sneezing, dyspnea, rhinitis, fever or diarrhea) at the time of URT-swab collection. Some patients had one or multiple follow-up RT-PCR on different clinical samples (either repeated URT-swab, sputum, broncho-alveolar lavage fluid (BALF) or feces) if the first URT-swab was negative and clinical suspicion remained.

Serum samples were collected after a minimum of 12 days post disease onset whenever possible, to determine if specific antibodies to SARS-CoV-2 had developed. For this purpose, two independent tests were used; an in-house developed protein micro-array based on SARS-CoV-2 S1 and N proteins (PMA (Koopmans et al., 2012, Reusken et al., 2013)) and the Wantai total antibody ELISA based on the RBD-domain of S1 (Beijing Wantai Biological Pharmacy Enterprise, Beijing, China; Cat # WS1096). A patient was considered immunocompromised if, at the time of the URT-swab collection, at least one of the following criteria was met: solid malignancy with active chemotherapy (not including hormone-based therapy); hematologic malignancy irrespective of chemotherapy status; auto-immune disease treated systemically with immunosuppressants; HIV-infection with high viral load and CD4 below 450 cells/µL; transplant recipient using systemic immunosuppressants; primary immunodeficiency. Further details on sample collection may be found in the supplementary materials.

2.2. Molecular diagnostics

A duplex PCR for Sars-Cov-2 E-gene was performed as described in the supplementary materials. RT-PCR results were considered positive if the cycle threshold (Ct-) value was ≤50.

2.3. Serology

SARS-CoV-2 antibodies (Beijing Wantai Biological Pharmacy Enterprise, Beijing, China; Cat # WS1096) were detected according to the instructions of the manufacturer. The presence of SARS-CoV-2 antibodies is expressed as a sample/cut-off ratio. Additionally, an in-house protein microarray was performed for the detection of antibody responses to SARS-CoV-2-S1 and SARS-CoV-2-N antigens. Further details on the serological assays may be found in the supplementary materials. In the microarray, the threshold value for the detection of specific antibodies against N or S1 has been set at a titer of >1:20.

2.4. Statistical analysis

Towards the goal of determining clinical sensitivity of the test assays used, we set up a Bayesian statistical framework which allowed us to estimate sensitivities of each test without resorting to a ‘gold standard’. Our definition of clinical sensitivity is: the ability to detect (possibly asymptomatic) infection in a patient. We assumed perfect specificity for both RT-PCR (Chu et al., 2020, Corman et al., 2020, Nalla et al., 2020, van Kasteren et al., 2020), and serology (Lassaunière et al., 2020). This is further justified by noting that all individuals enrolled in the cohort were clinically diagnosed for COVID-19. It is feasible to determine sensitivities without a perfect benchmark to compare with, which can be understood by realizing that each individual must be positive when at least one test scores positive (assuming the 100% specificity as outlined above). In such a case all negative tests for this person must be false negatives; this gives evidence against the sensitivity of those tests. We included time since onset of symptoms, severity, sex and immunocompromised status as covariates. For a detailed description of the Bayesian statistical model that was used, see the Supplementary Text.

3. Results

In the study period, 644 patients that presented at our hospital with clinically suspected COVID-19 were tested for SARS-CoV-2 by RT-PCR on URT samples. In total, sera from 250 patients were obtained and analyzed for the presence of SARS-CoV-2 specific antibodies (Table 1 , Fig. 1 ). The manufacturers of the Wantai assay recommend index values of 1.1 and higher as evidence for the presence of specific antibodies. However, we observed that at this cut-off value, several RT-PCR confirmed patients scored negative and a cut-off of 0.25 was calculated to be more appropriate in our cohort (Fig. 2 , Supplementary Text). The rightmost panel of this figure gives quantitative evidence. Clearly, using the manufacturer's cut-off, many positives would be to the left of the cut-off and would score false-negative in the Wantai test.

Table 1.

Cohort baseline characteristics. Ct-value, cycle threshold value; ICU, intensive care unit; SD, standard deviation; URT, upper respiratory tract; RT-PCR, reverse-transcriptase polymerase chain reaction.

	Serum collected	No serum collected	P	Total
	(N = 250)	(N = 394)		(N = 644)
Age, mean (SD), years	64.5 (14.38)	67.9 (16.33)	0.007	66.6 (15.7)
Female, N (%)	116 (46.4%)	148 (37.6%)	0.026	264 (41%)
Immunocompromised, N (%)	37 (14.8%)	36 (9.12%)	0.027	73 (11.3%)
Interval symptom onset and URT-swab collection, mean (SD), days	8.25 (6.68)	7.49 (7.54)	0.192	7,78 (7.2)
Disease severity categories
1, outpatients, N (%)	58 (23.2%)	67 (17.0%)	0.148	125 (19.4%)
2, hospitalized, non-ICU, N (%)	162 (64.8%)	273 (69.3%)		435 (67.5%)
3, ICU, N (%)	30 (12.0%)	54 (13.7%)		84 (13%)
Deceased, N (%)	16 (6.4%)	120 (30.5%)	<0.001	136 (21.1%)
First URT-swab positive (N/N total)	109 (43.6%)	195 (49.5%)	0.144	304 (47.2%)
Ct-value (first URT-swab)	28.7 (6.59)	28.5 (5.63)	0.795	28.5 (5.99)
RT-PCR positive (any sample, total), N (%)	123 (49.2%)	216 (54.8%)	0.164	339 (52.6%)
Additional samples in first URT-swab negative patients
Second URT-swab positive (N/N total)	1/38 (2.6%)	9/51 (17.6%)		10/89 (11.2%)
Sputum positive (N/N total)	2/9 (22.2%)	4/19 (21.1%)		6/28 (21.4%)
Faeces positive (N/N total)	11/29 (37.8%)	12/29 (41.4%)		23/58 (39.6%)
Interval symptom onset and serum collection, mean (SD), days	26.7 (10.1)	NA		26.7 (10.1)
Wantai ab index value (out of N = 250)	10.38 (10.1)	NA		10.38 (10.1)
Wantai ab positive (out of N = 250)	155 (62.0%)	NA		155 (62.0%)

Fig. 1.

Patient cohort flow-diagram

Fig 2.

(A) Histogram of Wantai total antibody index values on a logarithmic scale. The red vertical line indicates the manufacturer's cut-off value, the green vertical line our adapted cut-off value. (B) Percentage of PCR-positivity within Wantai total antibody index values. Since each bar represents a number of patients, the percentage of PCR positive test can be calculated within each bar. The histogram is scaled to 100% height per bar, and colored according to percentage of PCR positive (purple) and negative (yellow) status of the patients. When no patients existed in a certain range of Wantai index values, the bar is left blank. The red vertical line indicates the manufacturer's cut-off value, the green vertical line our adapted cut-off value.

URT-RT-PCR sensitivity varied according to disease severity: for outpatients, 80% (57% −100% Bayesian credible interval (CI)), for patients admitted to a non-ICU hospital ward 86% (77 − 95 CI), and for patients admitted to ICU 95% (84 −100 CI) (Fig. 3 , Table S1). URT-RT-PCR sensitivity was higher in males than in females (91% vs 81%), higher in deceased than in non-deceased patients (95% vs 83%) and higher in immunocompromised than in immunocompetent patients (93% vs 86%) (Fig. S1, Table S1).

Fig. 3.

(A) Posterior uncertainty distributions of the sensitivities for the microarray assays, serology and RT-PCR assay. Colors indicate severity category. (B) Estimates of prevalence of infection by severity category. ICU = Intensive Care Unit; OUTP = outpatients; WARD, non-ICU hospitalized patients; EC50_CoV-N, SARS-CoV-2 N protein microarray; EC50_CoV-S1, SARS-CoV-2 S1 protein microarray; serum_our, Wantai serology using cut-off value of 0.25; PCR, polymerase-chain reaction

For each disease severity category, URT RT-PCR cycle threshold (Ct)-values displayed an increasing trend by number of days since symptom onset, reflecting a decreasing viral load in time, albeit with considerable inter-individual variation (Fig. 4 , Table S2). Ct-values of outpatients increased rapidly compared to Ct-values of hospitalized patients. This translates in a rapid loss of sensitivity (Fig. 5 ). In contrast, RT-PCR on IC and in-hospital patients retains its initial sensitivity for a prolonged period of time (approximately 15 and 20 days respectively, Fig. 5). However, the point of decline is highly uncertain, due to the absence of patient material longer after onset of symptoms. Therefore, conclusions on the time of decline of sensitivity should not be drawn for IC and in-hospital patients. In contrast, outpatients do show a marked decrease of sensitivity over time, with lower uncertainty: sensitivity could be halved already within 3 weeks.

Fig. 4.

Linear increase of Ct-value in relation to days since onset of symptoms across different disease severity categories. The shaded band indicates 95% Bayesian credible interval. The dots are the original data. Dots positioned at a Ct of 50 were right-censored in the inflated model (i.e. count as either above 50 or a negative individual). ICU = Intensive Care Unit; OUTP = outpatients; WARD, non-ICU hospitalized patients; ct, Ct-value

Fig. 5.

Modeled sensitivities as a function of days since onset of symptoms. Panels and shading indicate disease severity. The bands indicate 95% Bayesian credible interval, and black dots an estimate of sensitivity directly from the original data, by assuming Wantai serology as the gold standard, and calculating the ratio of Wantai positive to PCR positive for each day since disease onset. Note that since only a limited number of individuals contribute to each point, points are clustered around fractions with low denominators (e.g. 0/2, 1/2 2/2). Concordance of the curves with the data points should be assessed by observing that the curve runs on average between the dots. ICU = Intensive Care Unit; OUTP, outpatients; WARD, non-ICU hospitalized patients.

COVID-19-suspected patients with a negative URT RT-PCR and without alternative diagnosis were frequently retested for SARS-CoV-2 after resampling. This increased the total number of RT-PCR-confirmed patients with 35 (Table 1).

Serum samples were analyzed for SARS-CoV-2 antibodies by ELISA and PMA. Bayesian modelling revealed an overall sensitivity of >94% for both ELISA and PMA, which did not change significantly according to disease severity category, sex or mortality (Fig. 3, Fig S1, Table S1). The sensitivity of antibody assays was lower in immunocompromised patients (82% −92%, Table S1) and this was the only group in which antibody assays had a lower sensitivity compared to UTR RT-PCR. The ELISA and PMA results were discrepant in only four of 250 sera (Table S3). All four patients tested RT-PCR positive in URT-swab, and 3 out of 4 were immunocompromised.

4. Discussion

An accurate assessment of the clinical sensitivity of diagnostic tools, in particular of RT-PCR on URT samples on which global test, trace and isolate strategies are based (Critical preparedness, readiness and response actions for COVID-19: WHO/2019-nCoV/Community_Actions/2020.3, 2020), is an absolute requisite for good patient care and adequate infection risk management. We observed a decrease in sensitivity with decreasing disease severity, an increase in sensitivity in immunocompromised patients and a rapid decline of sensitivity in time post onset of symptoms, but only in outpatients. In our study, 5% to 14% of hospitalized COVID-19 cases and 21% of outpatients tested negative in URT-PCR. This finding is in contrast with local practice and guidelines, which are often based on the assumption of near-perfect sensitivity (Grassly et al., 2020).

Comparing our results to other studies that model the sensitivity of SARS-CoV-2 test sensitivity, we find an overall concordance. In a study involving patients from the emergency room, comparable to our ICU group, a sensitivity of 89% is found using a latent class model, which compares well to 95% found in our study (Bisoffi et al., 2020). In another study, a Bayesian spline model yields a PCR sensitivity of 76% at day of onset of symptoms, increasing to 89% after 4 days (Zhang et al., 2020). Furthermore, no difference between sensitivity up to, and after day 7 post symptoms was found, which we also observe in ICU patients (up to at least 2 weeks post symptom start). Another study reports on a group of patients from an outbreak cluster, where the sensitivity of the PCR test was established at 86%. This result relied on a composite ‘golden standard’ by comparing the PCR result with several follow-up tests (Holborow et al., 2020). Such a reliance on direct comparison with additional tests is commonplace. For example, the sensitivity of the PCR assay has been estimated at 79.2% (until day 4 post-symptoms: 100%, after day 4, 50%) using PCR combined with rapid antibody testing (Mlcochova et al., 2020). Other practitioners have relied on clinical diagnosis (Williams et al., 2020), reporting a sensitivity of 82.2% is found. This is a low estimate, which may be explained by the aspecific symptoms of Covid-19 infection.

The analytical sensitivity of PCRs generally approaches 100%, which means tests are able to detect a single viral genome copy in the reaction volume (Chu et al., 2020, Corman et al., 2020, Nalla et al., 2020, van Kasteren et al., 2020), the clinical sensitivity may be substantially lower, due to low quality of samples, presence of inhibiting factors, suboptimal pre-analytic processing or specific biologic features of the viral infection. If one assumes infection of the URT always occurs in SARS-CoV-2 infected persons, a negative URT RT-PCR could be the result of localized early clearance of the virus or low levels of local viral replication. The localized clearance would then have different kinetics than the rate of viral load decline we measured in this study. Alternatively, the possibility should be considered that the URT of these patients was not infected and infection can remain more localized (e.g. to the trachea / lower respiratory tract) (Hou et al., 2020). As can be expected, RT-PCR sensitivity of URT correlates with the SARS-CoV-2 RNA load in these samples. The increase of mean Ct-value over time was significantly slower in hospitalized patients compared to outpatients, which explains why the sensitivity of RT-PCR on URT samples decreased more rapidly in cases with mild infections than in hospitalized patients and suggests hospitalized patients have difficulties clearing the virus. In our cohort, immunocompromised patients were the only category in which the sensitivity of serology was lower (PMA), or comparable (Wantai) to URT-RT-PCR. According to the definition we used, 73 of 644 patients were immunocompromised (Table S4), and most were treated by systemic immunosuppressants that inhibit antibody production. This relative lack of antibody response may explain the lower sensitivity of serology in this patient group, and it may clarify why relatively many immunocompromised patients’ Wantai results fell below the manufacturer's cut-off value. Although antibody assays were shown to have high sensitivity in the rest of our cohort, their use in tracing and isolating strategies is non-existent, due to the fact that antibodies take a while to be produced and may be detectable for several months after initial infection. Therefore, in order to timely diagnose and quarantine COVID-19 patients, URT-PCR will continue to be used as the assay on which initial decisions are based.

Our estimated URT-PCR sensitivity has important consequences for screening, treatment and isolation measures in hospitals. Though a positive PCR does not necessarily signify the presence of viable virus (Wölfel et al., 2020), sensitivities of 86-94% are not sufficient to lift isolation measures in the event of a negative initial URT-PCR. In the early stages of hospital admission, clinical suspicion and local COVID-19 prevalence should guide isolation and treatment decisions. Additionally, timely sputum or feces sampling should take place in an effort to confirm COVID-19 diagnosis. Several studies found that 30% to 50% of COVID-19 patients have detectable SARS-CoV-2 RNA in feces (Chen et al., 2020, Wang et al., 2020). In our study, feces yielded 3.5 times more RT-PCR-confirmed positives than repeated URT-swab: 40% of additionally sampled feces tested positive, as opposed to only 11% of repeated URT-swabs (Table 1).

Our findings also have ramifications for broad molecular testing in the general population, as currently established across the world, based on URT-swabbing in high-throughput testing lanes (Netherlands, 2020). The evidently often absent thorough epidemiological and clinical interpretation of negative results in these settings, in combination with the observed low clinical sensitivity of RT-PCR in the population with mild complaints that typically visit those testing sites, will lead to missed cases. The effect of imperfect test sensitivity and incomplete testing was previously assessed by Bayesian methods (Wu et al., 2020), where it was estimated that the actual number of cases could be 3 to 20 times higher than the number of confirmed cases, and 14% could be attributed to imperfect sensitivity.

In conclusion, our results show that for an accurate diagnosis based on RT-PCR test results and subsequent appropriate clinical management and infection control measures, a thorough understanding of the clinical sensitivity of RT-PCR in URT samples is necessary. The apparent lack of a high clinical sensitivity of this standard diagnostic method in specific situations warrants vigilance for missed cases especially in settings of high-throughput testing lanes where epidemiological and clinical context are often disconnected from negative test results for final interpretation.

Author contributions

Barbara J.M. Bergmans: conceptualization, writing of the manuscript, data collection and curation. Chantal B.E.M. Reusken: conceptualization, writing of the manuscript. Anne J.G. van Oudheusden: laboratory analyses. Gert-Jan Godeke: laboratory analyses. Axel A. Bonačić Marinović: statistics. Esther de Vries: conceptualization. Yvette C.M. Kluiters-de Hingh: laboratory analyses. Ralf Vingerhoets: laboratory analyses. Marvin A.H. Berrevoets: laboratory analyses. Jaco J. Verweij: laboratory analyses. An-Emmie Nieman: laboratory analyses. Johan Reimerink: laboratory analyses. Jean-Luc. Murk: conceptualization, writing of the manuscript. Arno Swart: statistics, programming, data curation, writing of the manuscript. All authors critically reviewed the manuscript.

Acknowledgments

We thank BJ Bosch for providing the S1 antigen for the protein microarray and H van Zundert for providing additional data on immunocompromised patients.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.