Estimated Sensitivity Values of Severe Acute Respiratory Syndrome Coronavirus 2 Tests from Cross-sectional Data Warrant Caution Due to Unvalidated Model Assumptions

TO THE EDITOR—I read with interest the recent study by Yokota et al [1] on polymerase chain reaction (PCR) tests for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) between nasopharyngeal swab (NPS) and saliva samples. The authors used a Bayesian latent-class model to simultaneously estimate the sensitivity and specificity of PCR tests along with the unknown prevalence in the study cohorts (contact tracing and airport screening), and reported a sensitivity of 86% and 92%, respectively, for NPS and saliva samples and over 99.9% specificity for both tests. They also reported high concordance between them and concluded that saliva samples can replace the conventional NPS. Compared with NPSs, saliva tests are low-invasive and require less involvement of healthcare professionals. The equivalent performance of saliva tests, also suggested in other studies [2–5], has a profound impact on the ongoing public health planning against coronavirus disease 2019 (COVID-19) where efficient and safe testing protocols are of paramount importance.

However, I noticed that the authors’ model has limitations due to the following 2 conditions, which rendered the reported sensitivity of 86% and 92% merely reflective of positive agreement rates (PARs) [6] between NPS and saliva tests.

• NPS and saliva test results are assumed to be independent, although the authors showed that their cycle threshold (Ct) values are highly correlated (with a Kendall’s W [7] of 0.87).

• The prevalence p was jointly estimated with other parameters, which caused the sensitivity almost solely determined by PAR in the data.

I showed that posterior distributions of PAR obtained from the original data are, in effect, identical to those of sensitivity obtained from the full dataset: NPS, 86% (90% credible interval, 77–93%); saliva, 90% (82–96%) (Figure 1A). Almost identical distributions were also obtained from data excluding individuals testing negative for both tests (“negative-negatives”; 1872/1924 participants): NPS, 88% (79–94%); saliva, 91% (83–96%). These indicate that the original sensitivity estimates almost solely relied on individuals receiving at least 1 positive result (47 from contact tracing and 5 from airport screening data) and that the rest of the dataset was barely informative, which was caused by the latent variable p freely optimized reflecting PARs and the independent assumption. By using simulation, I also showed that, in the presence of high correlation between (hypothetical) viral loads in NPS and saliva samples (Pearson’s correlation, 0.8), I can reproduce the observed contact-tracing data and Kendall’s W (0.86), which nonetheless suggest lower sensitivity (62% for NPS and 67% for saliva) than the original study (Figure 1B and 1C).

Figure 1.

 Posterior distributions and simulated dataset. A, Estimated sensitivity and PAR for NPS and saliva tests. Top panels: posterior distribution of sensitivity reproduced from the original method. Bottom panels: PARs obtained from only saliva-positive samples for NPS and NPS-positive samples for saliva. The histograms show 10 000 posterior samples. B, Simulated hypothetical viral loads in NPS and saliva samples. Assuming a multivariate-normal distribution with a Pearson’s correlation of 0.8, 61 samples were drawn and plotted. Detection limits for NPS and saliva tests were assumed to be −0.3 and −0.5, respectively (denoted by dashed lines) and each sample was colored according to the test results (blue: both positive; green: 1 positive and 1 negative; red: both negative). Kendall’s W for samples with at least 1 positive test (viral load above the detection limit) was 0.86. C, Simulated and original data in 2-by-2 tables. The simulated viral loads were classified as positive/negative according to the assumed detection limits. Left table: simulated data restricted to truly infected individuals. Middle table: overall simulated data combined with 100 true-negative individuals (assumed to be negative for both tests). Right table: the original contact-tracing data. Abbreviations: CrI, credible interval; NPS, nasopharyngeal swab; PAR, positive agreement rate.

These results highlight the inherent limitations of the original study relying on potentially intercorrelated single-point PCR tests without external validation of true infection statuses. Sensitivity and PAR correspond under the independence assumption, and this study estimated sensitivity based on that assumption. While I support the finding of concordance between NPS and saliva tests, the interpretation of their sensitivity estimates warrants caution. In practice, multiple factors can lead to viral loads of both NPS and saliva below the detectable limit, including samples taken too early or late in the infection course or simply of poor quality. Polymerase chain reaction will remain the de facto gold-standard test; the possibility of a less-invasive sampling method without apparent loss in performance is an invaluable finding. However, any tests should be used with a proper understanding of their limitations in performance—for example, how frequently and, more importantly, when they can fail—which this study was unfortunately not sufficient to inform.

Notes

Code availability. Detailed methods and replication codes are available on GitHub (https://github.com/akira-endo/reanalysis_Yokota2020).

Financial support. A. E. is supported by The Nakajima Foundation.

Potential conflicts of interest. A. E. received a research grant from Taisho Pharmaceutical Co., Ltd. The author has submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.