Prospective clinical validation of 3D printed nasopharyngeal swabs for diagnosis of COVID-19

Abstract

COVID-19 greatly disrupted the global supply chain of nasopharyngeal swabs, and thus new products have come to market with little data to support their use. In this prospective study, 2 new 3D printed nasopharyngeal swab designs were evaluated against the standard, flocked nasopharyngeal swab for the diagnosis of COVID-19. Seventy adult patients (37 COVID-positive and 33 COVID-negative) underwent consecutive diagnostic reverse transcription polymerase chain reaction testing, with a flocked swab followed by one or two 3D printed swabs. The “Lattice Swab” (manufacturer Resolution Medical) demonstrated 93.3% sensitivity (95% CI, 77.9%–99.2%) and 96.8% specificity (83.3%–99.9%), yielding κ = 0.90 (0.85–0.96). The “Origin KXG” (manufacturer Origin Laboratories) demonstrated 83.9% sensitivity (66.3%–94.6%) and 100% specificity (88.8%–100.0%), yielding κ = 0.84 (0.77–0.91). Both 3D printed nasopharyngeal swab results have high concordance with the control swab results. The decision to use 3D printed nasopharyngeal swabs during the COVID-19 pandemic should be strongly considered by clinical and research laboratories.

1. Introduction

COVID-19, the clinical disease caused by the pathogen SARS-CoV-2, has presented unique challenges to the global community. Among these challenges is the massive need for population-wide diagnostic testing to contain the COVID-19 pandemic. In the United States, various expert groups have estimated a national need for anywhere from 500 thousand to 30 million targeted tests on a daily basis to mitigate and to suppress viral transmission (Anonymous, 1 July 2020; Danielle Allen et al., 20 April 2020; Jennifer Kates et al., 27 April 2020, Romer, April 2020, ). This immense need combined with traditional swab manufacturing supply chain interruptions (Thomas, 18 March 2020) presented an opportunity for the additive manufacturing industry (also called the 3D printing industry) to fill the need for large-scale production of nasopharyngeal (NP) swabs.

In the span of days-to-weeks, numerous 3D printing companies went through an iterative design process (Bennett et al., 2020; Callahan et al., 2020) to meet the requirements of the NP swab: (1) to collect an adequate patient specimen from the nasopharyngeal mucosa, and (2) to elute the sample into a reverse transcription polymerase chain reaction (RT-PCR) assay without affecting the quality of results. Quality design controls and early clinical use have demonstrated similar safety profiles to NP swabs currently on the market (Gupta et al., 2020). 3D printed NP swabs are classified as Class I, 510(k) Exempt in vitro diagnostic medical device by the U.S. Food and Drug Administration (FDA) (FDA, 29 June 2020). This designation implies that manufacturers need to follow Good Manufacturing Practices and have a Quality Management System, but it does not dictate the clinical performance standards for the swabs. The clinical validation process is at the discretion of the individual diagnostic laboratory to determine whether a particular NP swab should be considered for diagnostic use for COVID-19.

The rapid emergence of numerous 3D printed swabs options also implies a dearth of widely-published clinical efficacy data to recommend for or against their use. Given how important clarity is in the diagnosis of COVID-19, this study sought to determine the clinical performance of 3D printed nasopharyngeal swabs compared to FDA-approved nasopharyngeal swabs already on the market.

2. Materials and methods

2.1. Swab selection

Many swab designs were considered but 2 swab types were ultimately chosen for this study based on swab tip and shaft design, material, comfort during use, and ability of the company to manufacture large quantities: The Generation 1 “Lattice Swab” by Resolution Medical (FDA, 29 June 2020), and the “Origin KXG” by Origin Laboratories (FDA, 20 June 2020), as shown in Fig. 1 .

Fig. 1.

Swab designs. The Lattice Swab by Resolution Medical (left) and Origin KXG by Origin Laboratories (right).

2.2. Participants

A minimum sample size of 30 COVID-positive and 30 COVID-negative patients for each NP swab type was chosen based on the FDA's recommendation for clinical evaluation of new diagnostics eligible for emergency use authorization (EUA). Inclusion criteria for participants were: 18 years of age and older; an outpatient scheduled for testing in the preprocedural or preoperative setting to confirm COVID-negative status; a hospitalized inpatient having already received a COVID-positive test result in the prior 24 hours. Ambulatory outpatients underwent simultaneous swabbing with the control and experimental swabs. Known COVID-positive inpatients underwent serial swabbing with the experimental swab(s) within the 24-hour timeframe of their initial control swab. Patients assessed for the study were excluded if they had conflicting prior COVID diagnostic results, if they were physiologically unstable for swabbing, had recent facial trauma or nasal surgery, had a platelet count <50,000 cells/μL or absolute neutrophil count <500 cells/μL. All patients underwent a verbal consent process approved by the Institutional Review Board at the University of California, Los Angeles.

2.3. Sample collection and determination of results

Sample collection was performed by trained nurses to ensure a high-quality sample. The swab was inserted into either the right or left naris based on nurse and patient preference until the nasopharynx was reached. The swab was then gently rotated for several seconds and withdrawn. This process was used for both experimental and control swabs. With each subsequent swab, the choice of naris was again at the discretion of the nurse and patient. Collections that were felt to be likely inadequate samples based on the nurses’ prior experience were immediately discarded.

Control samples were collected with the Universal Viral Transport Kit by Becton, Dickinson & Company, which consists of a flocked swab and 3 mL Universal Viral Transport medium (FDA, 29 June 2020). Experimental swab samples were collected consecutively to the control and transported in sterile polypropylene tubes with 3 mL saline (0.9% sodium chloride in water) as transport media. After collection, the samples were promptly delivered to the laboratory and if not immediately processed via RT-PCR, were stored for less than 48 hours at 4°C until processing. The RT-PCR machine used to run the tests was Liaison MDX by DiaSorin Molecular, LLC, using the parameters outlined for use with the Simplexa COVID-19 Direct Kit assay (EUA March 19, 2020 (FDA, 2 July 2020), with amendment for use of saline as transport media on March 26, 2020) which tests for the S gene fragment and the ORF1ab gene fragment of SARS-CoV-2. Laboratory personnel were blinded to the control sample result, as well as to patient identifying information. The maximum number of cycles run for viral detection was a cycle threshold (Ct) of 40. Detection of either the S gene or the ORF1ab gene, or both, yields a COVID-positive clinical result. Detection of neither the S gene nor the ORF1ab gene yields a COVID-negative clinical result.

2.4. Statistical analysis

RT-PCR results are reported categorically as COVID-positive or COVID-negative. To analyze performance differences between swab type, sensitivity, and specificity with 95% confidence intervals (CI) for each experimental swab type was determined in relation to the control swab. Categorical concordance was calculated using Cohen's kappa (κ), which is a measure of nonrandom agreement between 2 different tests, with a k > 0.8 implying excellent agreement. All statistical calculations were performed using SAS version 9.4 (SAS Institute, Cary, NC).

Since the Simplexa assay detects both the S gene and ORF1ab gene, these data were analyzed separately. For quantitative Ct value comparison in the COVID-positive patients, a Wilcoxon Signed-Rank test was performed for paired, dependent, nonparametric data between each experimental swab and control, as well as between the 2 different experimental swabs. In order to preserve data integrity and reduce bias that would result from excluding “non-detected” gene fragments, these results were instead imputed as the value 40, the maximum Ct for the RT-PCR assay. The difference in Ct values between the pairs, always calculated as control Ct minus experimental Ct, is reported as median, first quartile (Q₁), and third quartile (Q₃) values. P values >0.05 imply no systematic bias, meaning that the Ct values among compared swabs are drawn from the same underlying distribution. Also included are scatterplots comparing Ct values for ease of visualization.

3. Results

Of 144 total participants assessed for study inclusion, 125 (87%) met criteria and were approached for consent and 70 (49%) ultimately provided samples for the study. This is further detailed in Supplementary Figure S1. The most common reason cited by patients for not consenting to the study was the perceived discomfort of the NP sample collection, whether or not they had undergone a previous NP swab collection prior to the study.

The COVID-positive patients were approached in the inpatient setting at 2 UCLA-affiliated hospitals, Ronald Reagan University Medical Center and UCLA Medical Center, Santa Monica. Eleven of the 37 patients had their clinical swab (control) and one or two 3D printed swab (experimental) samples collected simultaneously, and thus for these patients the elapsed time between sample collections was effectively zero. The remaining 26 of the 37 patients underwent experimental swabbing within a 24-hour window of their control swab in order to minimize the potential for change in viral load in the nasopharynx.

The COVID-negative patient samples were obtained predominantly in the outpatient “drive-through testing” setting (32 of 33 patients), thereby undergoing the control and experimental swabbing simultaneously. Patient recruitment demographics are represented in Table 1 . No adverse events occurred during sample collection for either group.

Table 1.

Study population demographics. The mean age between the two sampled groups was significantly different (P = 0.004), with the COVID-positive patients mean age 62 years (SD 17), and the COVID-negative patients mean age 50 years (SD 18).

	COVID-positive	COVID-negative
	Participants, No. (%)	Participants, No. (%)
Age, y
18–40	5 (14)	5 (11)
41–60	11 (30)	11 (33)
61–80	15 (41)	11 (33)
81+	6 (16)	6 (18)
Sex
Male	19 (51)	18 (55)
Female	18 (49)	15 (45)
Setting
Floor/Ambulatory	26 (70)	32 (97)
ICU/Intubated	11 (30)	1 (Romer, April 2020)

Performance data of the experimental swabs can be seen in Fig. 2 (raw PCR data is included in Supplemental Table S1). The Lattice Swab displayed 93.3% sensitivity (95% CI, 77.9%–99.2%) and 96.7% specificity (83.3%–99.9%). The 2 false negatives included patients that were near the limit of detection, so-called “low-positives,” with control Ct values of 34.4 and 35.1 cycles. The singular false positive reflected a patient that was intubated for confirmed COVID-positive pneumonia by NP swab 5 days prior to the study, had a low-positive result 5 days later with the experimental Lattice Swab (Ct of 34.0 cycles), but ultimately resulted negative with the study control swab. The Origin KXG displayed 84% sensitivity (66.3%–94.6%), and 100% specificity (88.8%–100%). The 5 false negatives included 3 low-positive controls (Ct of 33.5, 34.4, and 37.2 cycles) and 2 more moderate controls (Ct of 27.3 and 31.8 cycles).

Fig. 2.

Sensitivity, specificity, and Cohen's kappa (κ) results for the Lattice Swab and Origin KXG in comparison to control swab. The Lattice Swab (κ = 0.90; 95% CI, 0.85–0.96) had higher concordance with the control swab than did the Origin KXG (κ = 0.84; 95% CI, 0.77–0.91), however both of these values suggest “excellent” agreement between tests.

When analyzing detection of the S gene fragments and the ORF1ab gene fragments separately, the Wilcoxon-Signed Rank test suggests no significant difference between either experimental swab against control for either gene, nor a significant difference between the 2 experimental swabs for either gene. This is demonstrated by P values in Table 2 ranging from 0.30 to 0.83 that are calculated based off the difference between Ct values paired for an individual across different tests. Comparative data are displayed in Table 2 and visualized in Fig. 3 , which overall shows dense clustering around the 1:1 line.

Table 2.

Comparison of median and quartile Ct value differences between swab type pairs with SARS-CoV-2 RT-PCR positive results using the Wilcoxon Signed-Rank test.

	# Total pairs	# S gene pairs	S gene ΔCt, cycles median [Q1, Q3]	P value	# ORF1ab gene pairs	ORF1ab gene ΔCt, cycles median [Q1, Q3]	P value
Control vs Lattice Swab	31	28	−0.7 (−2.9, 2.9)	0.83	29	−0.6 (−1.8, 2.6)	0.53
Control vs Origin KXG	31	29	−0.8 (−3.1, 3.3)	0.83	29	−0.2 (−2.4, 5.3)	0.73
Lattice Swab vs Origin KXG	26	23	−1.0 (−2.4, 1.1)	0.38	23	−0.3 (−3.8, 1.1)	0.30

Nondetected results were imputed with a value of 40, unless both compared tests exhibited nondetected results, in which case they were removed from analysis. ΔCt was calculated as the control Ct minus the experimental Ct, or in the third case, the Lattice Swab Ct minus the Origin KXG Ct. This data is visualized in more detail in the Table S1. Q1 = first quartile, Q3 = third quartile.

Fig. 3.

Comparison of Ct values between swab type pairs with SARS-CoV-2 RT-PCR positive results. Positivity for either the S gene or ORF1ab gene signified clinical positivity. Nondetected singular gene fragment results for all swabs were imputed with a Ct value of 40. Note the general clustering around the 1:1 dashed line, indicating Ct values were often similar between different swab types. ΔCt >5 cycles between swab types lie outside of the dotted line, and occurred in 16 of 62 gene fragment samples for the Lattice Swab vs control (11 S gene, 5 ORF1ab gene), in 20 of 62 for the Origin KXG vs control (9 S gene, 11 ORF1ab gene), and in 13 of 50 for the Origin KXG vs Lattice Swab (6 S gene, 7 ORF1ab gene).

Twenty-six of 37 COVID-positive swabs were collected with a delay less than 24 hours between the control and the experimental swab. When comparing simultaneous-collection pairs to delayed-collection pairs using the Mann Whitney U test for independent samples, there was no statistically significant difference in swab performance for either the S gene or the ORF1ab gene if using a strict P value of 0.05, as shown in Supplemental Table S2 and visualized in Supplemental Figure S2.

4. Discussion

3D printing can offer unique healthcare solutions due to the capacity for rapid design iteration in conjunction with high-throughput manufacturing. This was demonstrated in April 2020 by Callahan et al. at Beth Israel Deaconess Medical Center, where open-source, collaborative efforts with 3D printing companies resulted in the successful design and validation of multiple new NP swabs in the course of only 22 days (Callahan et al., 2020). More recently the FDA, the National Institutes of Health, the Department of Veterans Affairs, and America Makes have entered a multi-institutional collaborative to make 3D printing data and resources publicly available to accelerate the pace of product development (FDA, 19 June 2020). The transparency of this translational research process can serve as a model for other urgent medical supply shortages that might arise in the future during times of crisis. Yet as time allows, these stop-gap solutions need to be carefully examined to ensure that they indeed stand up to the rigors of safety and efficacy expected during the treatment of human disease.

Compared to other published data directly evaluating 3D printed NP swab performance against the traditional flocked NP swab (Bennett et al., 2020; Callahan et al., 2020), this study offers data on a large sample of COVID-positive patients. In this study the sensitivities of the swabs are robust with a relatively narrow 95% CI. Given the high infectivity of SARS-CoV-2 it is of utmost importance to use swabs and assays with high sensitivity. Because nasopharyngeal swabbing is uncomfortable for the patient and centers may have variable access to large numbers of patients with SARS-CoV-2 infection, it may not be feasible for single healthcare entities to validate multiple 3D printed swab designs. Therefore, clinicians need to select the most promising designs prior to use in a clinical setting, as was done in this study.

The FDA has issued EUAs to COVID-19 molecular diagnostic platforms that show promising early results, with a minimum data requirement of 30 COVID-positive and 30 COVID-negative patients (FDA, 1 July 2020). This small sample size reflects a compromise between demonstrating early convincing data to allow for widespread diagnostic use and getting new technologies to market urgently during a crisis – assuming there is careful ongoing monitoring and re-evaluation of results. While 3D printed NP swabs alone are not eligible for EUA as a Class I 510(k) exempt device, this study and sample size was modeled with similar intent. Data supplementing and verifying early claims of sensitivity and specificity should continue to be collected by clinical laboratories that use new technologies, and 3D printed swabs are no exception.

Determining noninferiority between swab types was not attempted, given that considerable data on test variance and on what quantitative values constitute an appropriate clinically important difference have not been widely proposed and accepted by the COVID-19 molecular diagnostics expert community. This is the case for both the qualitative comparison between COVID-positive and COVID-negative status, as well as for the quantitative comparison between samples with high Ct values and low Ct values.

The Mann-Whitney U test for simultaneous-collection vs delayed-collection pairs demonstrates no difference in performance between the swab samples. However, this difference may ultimately result in statistical significance if a larger sample size was obtained or if there was a longer average mean time elapsed, as indicated by the comparison of control swab to Origin KXG for the ORF1ab gene fragment in Supplemental Table S2 (P = 0.052 with a mean time elapsed of 18.2h in the delayed-collection subgroup). This reinforces the notion that it is important to minimize the time elapsed during specimen collection when comparing results between 2 different devices.

The use of specialized viral transport media for the control swabs compared to saline media for the 3D printed swabs may also cause some variability in results. While data for the Simplexa assay comparing the 2 media has not been published, it can be inferred that they have similar performance given the FDA granted an EUA amendment for this purpose (FDA, 2 July 2020). The use of saline media was unavoidable due to shortages in supplies during the pandemic.

It is clear that 3D printed swabs have an important role to play in responding to the COVID-19 pandemic, yet certain limitations of this study need to be understood to make its findings actionable.

First, there is inherent error in treating any currently-approved NP swab as the provisional “reference standard” for comparison (Deeks and Altman, 1999; Trikalinos TA, 2012; Woloshin et al., 2020). For example, if 2 identical swabs were collected simultaneously from a single patient, those 2 swabs still may not show perfect concordance due to inherent variation in sample collection. Realizing that the comparator reference standard test is imperfect changes the meaning of test sensitivity and specificity, particularly for patients near the limit of detection with a high Ct value.

Second, the clinical sensitivity of NP swabs for diagnosing COVID-19 varies widely in the literature, with some estimates as low as 70% (Woloshin et al., 2020). This is more likely to be related to error in sample collection and handling than to misdiagnosis by RT-PCR. NP swabbing is a medical procedure that requires a detailed understanding of nasal anatomy, and collecting a blind NP sample can be both uncomfortable for the patient and difficult for the clinician. As with any procedure, there is a significant learning curve to overcome prior to achieving proficiency. Disease-specific sources of uncertainty include the timing of testing during disease course and the anatomic location of sampling. Collection-specific sources of uncertainty include significant time elapsing between sample collection and testing, improper storage temperatures, and inappropriate handling during sample collection, transport, and processing. Many variables affect the overall performance of a test and over-reliance on any one single result can lead to misdiagnosis, particularly when it does not complement the patient's history and clinical presentation. It is the authors’ opinion that when clinical diagnosis is in doubt, especially when pretest probability is high but the test returns negative (Woloshin et al., 2020), confirmatory testing should likely be repeated as RT-PCR analytical sensitivity is excellent but NP swab clinical sensitivity is more variable.

Third, the sensitivity and specificity values reported in this study are derived from different patient populations. The sensitivity is derived from a cohort of known COVID-positive patients by laboratory confirmation. The specificity is derived from a cohort of COVID-negative patients that were anticipated to have a low pretest probability of having COVID-19. Some were sent for testing by their primary care provider for suspicious symptoms, while others were sent by their surgeon or proceduralist without symptoms in order to verify COVID-negative status prior to a planned procedure. This spectrum bias (Zhou et al., 2002) was unavoidable, due in part to the urgent need to enroll patients to understand the effectiveness of the swabs during the pandemic, but also because the local incidence of COVID-positivity was <5% for patients tested during this time frame, and thus it would have taken extensive time and resources to collect 30 COVID-positive samples if randomly selected from the broader testing population.

Finally, there are many unanswered questions regarding the interpretation of RT-PCR results and Ct values for SARS-CoV-2. For example, what is the minimal clinically important difference in Ct values between 2 tests, and is that difference consistent across the range of infectiousness? Is the 3-cycle difference between 15 and 18 cycles less clinically relevant than the 3-cycle difference between 34 and 37 cycles? What does a “low-positive” mean in terms of infectiousness (Wölfel et al., 2020)? How comparable are Ct values from different assays? To what extent does viral RNA detection imply replication-competence (Bullard et al., 2020)? Until some of these challenging questions can be definitively answered with further research, clinical laboratories will need to continue making critical decisions based on insufficient evidence.

5. Conclusion

The 3D printed NP swabs evaluated in this study have high concordance with traditional flocked NP swabs and should be strongly considered by clinical laboratories to supplement deficits they may face in their COVID-19 NP swab supply chain.

Authorship statement

Gabriel Oland: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Visualization, Writing – original draft, Writing – review and editing; Omai Garner: Funding Acquisition, Resources, Writing – review and editing; Annabelle De St Maurice: Conceptualization, Methodology, Supervision, Writing – review and editing