SARS-CoV-2/COVID-19 Testing: The Tower of Babel

Abstract

Background and aim:

Testing represents one of the main pillars of public health response to SARS-CoV-2/COVID-19 pandemic. This paper shows how accuracy and utility of testing programs depend not just on the type of tests, but on the context as well.

Methods:

We describe the testing methods that have been developed and the possible testing strategies; then, we focus on two possible methods of population-wide testing, i.e., pooled testing and testing with rapid antigen tests. We show the accuracy of split-pooling method and how, in different pre-test probability scenarios, the positive and negative predictive values vary using rapid antigen tests.

Results:

Split-pooling, followed by retesting of negative results, shows a higher sensitivity than individual testing and requires fewer tests. In case of low pre-test probability, a negative result with antigen test could allow to rule out the infection, while, in case of a positive result, a confirmatory molecular test would be necessary.

Conclusions:

Test performance alone is not enough to properly choose which test to use; goals and context of the testing program are essential. We advocate the use of pooled strategies when planning population-wide screening, and the weekly use of rapid tests for close periodic monitoring in low-prevalence populations. (www.actabiomedica.it)

The SARS-CoV-2/COVID-19 pandemic struck unexpectedly and rapidly spread across the world, with an unprecedented impact on health, economy, and societies. Because the search for effective therapies and vaccines is still in progress, specific public health measures called non-pharmaceutical interventions (NPIs) have been put in place to different degrees across countries to limit the spread of the disease, including face masks, social distancing, sanitization of spaces, prompt isolation of cases, contact tracing, and quarantine of identified contacts (1, 2). Alongside these measures, a robust testing capacity represents one of the main pillars of public health response to COVID-19 (3).

To address this need, a vast array of test types and testing strategies have been developed. For a threat like COVID-19, the purposes of testing are somewhat different than in other public health settings and vary for individuals, organizations, and government actors. This paper shows how the accuracy and ultimately the utility of testing programs depend not just on the tests themselves, but on the use context as well. We begin by describing the testing methods that have been developed, and then describe testing strategies. The next section illustrates how all of these factors together affect the interpretation of tests.

Testing methods

The impressive rate of spread of the disease has urged for procedures to speed up the approval of tests and for providing updated guidance for validation of quality-assured assays. The US Food and Drug Administration (FDA) (4) and the European Commission (5) issued two guidance documents for clinical laboratories, commercial manufacturers and other stakeholders, in order to define the minimum standards required for market authorization of SARS-CoV-2 tests and in vitro diagnostic medical devices (hereinafter SARS-CoV-2 tests). As regard the EU context, Directive 98/79/EC on In-Vitro Diagnostic devices (IVD) (6) currently applies to SARS-CoV-2 tests. In addition, with the guidance (5) the European Commission outlines the regulatory context of SARS-CoV-2 tests for CE-marking during the public health crisis due to the pandemic.

Since the early phases of the pandemic, the collection and testing of an upper respiratory specimen has been recommended for the diagnosis of SARS-CoV-2, the coronavirus that causes COVID-19. Nucleic acid tests, also referred to as molecular tests, are the most used with a diagnostic purpose for the detection of SARS-CoV-2 in patients with COVID-19-like symptoms, being essential for triage, isolation and assistance in healthcare facilities (7). With this purpose, they can also be used in case of asymptomatic individuals with recent suspected or confirmed exposure to SARS-CoV-2 (5). In particular, nasopharyngeal swab, carried out by trained healthcare personnel using adequate personal protective equipment (PPI), and subsequent reverse-transcriptase-polymerase chain reaction (RT-PCR) analysis to detect and amplify viral RNA, has been referred to as the gold standard for initial diagnosis (8, 9). With specific regard to the implementation of this technique, the cycle threshold (Ct) value represents the number of replication cycles required to produce a fluorescent signal, that indicates the presence of viral RNA; therefore, higher viral RNA loads can be detected with a lower number of Ct. Variations of this value in nasopharyngeal swabs collected during different epidemic periods have been reported (10). The analysis carried out in equipped and with a biosafety level 2 laboratories lasts about four hours, with highly variable turnaround times considering the addition of transport time, sample preparation and reporting (8). The shortage of necessary reagents created considerable difficulties in emergency management.

If we test symptomatic individuals, viral RNA is detectable through nasopharyngeal swab since the day of symptom onset, while the peak of the load is reached within seven days (11). Nevertheless, according to a recent review, on the day of symptom onset the median false-negative rate with RT-PCR was 38% (12). This decreased to 20% on day 8 after symptom onset, then began to increase again, from 21% on day 9 to 66% on day 21 (12). Positivity typically starts to decrease in the third week, but in severe hospitalized patients, it may persist for a longer period. However, a positive PCR result only reflects the detection of viral RNA and does not necessarily indicate the presence of viable virus (11). Recent evidence supports the use of saliva as a specimen for detecting viral RNA (13). Being collected by individuals themselves, saliva samples allow to avoid direct contact between healthcare professionals and patients and reduce demands for supplies of swabs and personal protective equipment (13) An example is Yale’s test, Saliva Direct, approved by the FDA with emergency procedure (14, 15).

“Rapid molecular tests” have also been developed over time, eliminating traditional RNA extraction steps. This enables faster turnaround times because the processing times range between 50 and 90 minutes, but an equipped laboratory is still required; however, there is pretty large performance variation among the rapid PCR tests (16).

Other types of tests for SARS-CoV-2 that have been developed over the course of the pandemic include rapid antigenic tests that identify directly the presence of the virus, usually part of a surface protein, and serological tests that detect antibodies versus SARS-CoV-2.

Molecular tests have been highly acclaimed for their analytical sensitivity and specificity, usually greater than 95%, while rapid antigenic tests have lower sensitivity compared to molecular tests and a good specificity. Indeed, we usually think of sensitivity and specificity when we ask ourselves the extent to which we can consider a test reliable. However, this is just a part of the answer. Sensitivity and specificity only refer to the proportion of infected individuals that receive a positive result and the proportion of healthy individuals that receive a negative result, respectively, and represent a feature of the test that is defined as “accuracy”. In the EU, tests’ manufacturers are obliged (5) to explain their choices about performance levels in the instructions for use and to determine the specific purpose of the test in accordance with the choices made. They are also obliged to identify the target population. Validation refers to confirmation that the test achieves the performance levels specified by the manufacturer. Indeed, according to the guidance (5), it is the manufacturer who evaluates the performance of the device in accordance with the intended purpose before placing the device on the market. However, the performance of the device may vary in practice with respect to the performance study the manufacturer has done for the purposes of CE-marking. Therefore, it is highly recommended to carry out additional validation on the clinical performance of SARS-CoV-2 tests with respect to a reference method in a sufficiently large number of target population subjects before introducing the devices into the clinical routine. Scientific peer-reviewed results on the clinical validation of commercial COVID-19 tests are also highly recommended before they can be safely and reliably used for medical or public health decision making.

Testing strategies

Typically, tests are conducted to diagnose individuals with clinical symptoms as well as to track disease prevalence in the population (i.e. surveillance). With a threat like COVID-19, the purposes of testing have become more nuanced and vary for individuals, organizations, and government actors. Testing strategies differ in their purpose, the settings in which they are used. Consequently, who gets tested and the characteristics of the individuals who are tested, including the pre-test probability of infection, vary markedly. Although a bit of a simplification, it is useful to consider three separate purposes of COVID-19 testing: diagnosis, surveillance, and screening.

Diagnosis

In the first instance, tests are used to determine SARS-CoV-2 infection status of individuals who have symptoms to guide the healthcare they receive. Tests are also used for close contacts of known cases. Whether a test is conducted depends on provider perceptions (e.g. what the provider would do differently depending on the results), test availability, and official guidance about who should be tested. In addition, psychological motivations (e.g. perceived susceptibility) and setting (e.g. prevalence in the community) influence patients’ test seeking behavior. As described by Piltch-Loeb and colleagues, because the likelihood of being tested depends on patient and provider factors, the prior probability of infection varies quite widely by setting, creating challenges in interpreting clinical results (17).

Surveillance

COVID-19 tests also provide public health authorities sufficient understanding of the epidemic to impose and relax control measures appropriately. Broadly speaking, testing for viral RNA or antigen is intended to provide information about the current incidence and prevalence and trends over time. To make these decisions, reports of COVID-19 confirmed (i.e. test positive) cases should be coupled with additional population health information, such as evidence about hospitalizations and COVID-19-attributed mortality. In interpreting surveillance data, it is important to remember that the factors that influence test interpretation are all constantly changing (18). For instance, more and different types of tests that are being developed have lower costs and shorter turnaround times, but may be less sensitive and specific. Similarly, there may be an increase in the number of people tested for screening purposes (see below), possibly resulting in a lower proportion of individuals who are positive. As the perceived level of transmission in a community changes, changes in testing may vary across subpopulations based on changing perceived risk, barriers to testing (such as longer lines at testing centers), and other factors (19).

Seroprevalence surveys, in which a representative sample of individuals are tested for SARS-CoV-2 antibodies, are intended to help authorities understand the nature of the outbreak. For example, Rosenberg and colleagues analyzed a convenience sample of New York grocery store customers and estimated that the antibodies to SARS-CoV-2, through March 29, 2020, was 14%. This rate varied substantially by geographic area (reaching 24% in New York City) as well as race and ethnicity. They also estimated that only 8.9% of individuals infected during this period were diagnosed, and that this fraction varied from 6.1% of individuals aged 18–34 years to 11.3% of those 55 years or older (20). Population-based seroprevalence studies have been conducted in Iceland (21), Geneva (Switzerland) (22) and Spain (23). A large Italian study in the period May-July 2020 on a sample of 64,660 population has estimated a mean seroprevalence among the general population of 2.5%, with wide variations across regions, age classes and professional categories (24). Indeed, a cross-sectional study involving 423 workers as they returned to workplace after the national lockdown carried out in the province of Bergamo, Italy, that experienced one of the deadliest COVID-19 outbreak in the world, found 38.5% positive individuals (25). Another alternative is to analyze blood samples obtained for other clinical assessments. Anand and colleagues, for instance, tested blood samples from 28,503 randomly selected US adult patients receiving dialysis in July, 2020, showing how the seroprevalence of SARS-CoV-2 varies regionally (26). Seroprevalence only represents cumulative incidence over the period for which antibody is detectable by the test used, which is often currently unknown (27).

Screening

Routine testing for SARS-CoV-2 may be useful in settings where it is not possible to work remotely in order to quickly isolate infected workers. The rationale for such testing in healthcare settings, where close proximity and physical contact is often unavoidable is to avoid having asymptomatic or pre-symptomatic patients and healthcare workers inadvertently transmitting the virus to others. For similar reasons, other non-healthcare groups that would be high priority for universal testing include prisoners and prison employees and workers at warehouses, factories and food processing plants.

Testing has become an important component of many American universities’ effort to control COVID-19 as they reopened in the Fall of 2020. In addition to symptom-based testing, a variety of testing strategies are being employed in an attempt to reduce transmission. These include (i) universal entry screening: testing all students before arrival on campus; (ii) 2-phased universal screening: pre-arrival testing paired with a follow-up test, typically about 1 week after arrival; (iii) scheduled screening, with repeated testing of the entire campus population (e.g., weekly); (iv) random screening, with testing a random sample of the campus population; and (v) testing on-demand, by making tests available to students on campus on demand but not requiring testing (28). While the effectiveness of these strategies has not been evaluated, one can see that they vary markedly in the types of tests used and the prior probability of infection, so the results will be difficult to interpret. While the rationale for universities’ testing strategies is not always clear, there is an emerging consensus that high-frequency testing, which increases the likelihood of testing at the right moment for rapid isolation, is important. An attractive strategy in some instances is rapid antigen testing, since some antigen tests are cheap enough to make higher frequency testing feasible for more universities and since their short turnaround times outweigh their lower sensitivity compared with most lab-based PCR tests (29, 30).

Test interpretation

Test performance alone is not enough to properly choose which test to use in different scenarios. Our a-priori knowledge of the pre-test probability of the infection, that gives us an indication on how widespread the infection is in the target population, is essential. Bayes’ theorem allows to combine together all these necessary elements in order to get the posterior probability that a patient with a positive result is actually ill (e.g., positive predictive value [PPV]) and that a patient with a negative result is actually healthy (e.g., negative predictive value [NPV]), and accordingly choose which test for which context to use.

If we consider individual molecular assays with 99% sensitivity and 99% specificity to be used to test all individuals, assuming a low pre-test probability of 1% the PPV would be 50% and the NPV would be 99.99%; assuming a pre-test probability of 3.5%, the PPV would be 78.2% and the NPV would be 99.96%. Drawing from this example, we can state that even if the performance of the test is excellent, a low prevalence of the infection affects the reliability of the results. With a prevalence of 1%, only one person in two who receives a positive result is actually ill. Increasing pre-test probability to 3.5% also increases the positive predictive value, which, however, cannot be considered sufficient. On the other side, both NPVs are extremely high, allowing to rule out the infection in case of a negative result.

In the early stages of the pandemic, tests were mainly used as a diagnostic tool in symptomatic people and in high-risk groups, being nucleic acid tests the main used and, to a lower extent, serological tests.

SARS-CoV-2 infection can be detected indirectly by measuring the host immune response. However, the actual development of antibodies in every infected individual (31) and the degree of protective immunity conferred by or correlated with the antibodies detected in subjects with past SARS-CoV-2 infection are still under investigation and serological tests currently have limited diagnostic application. Indeed, the main purposes that these tests should be used for are sero-epidemiological surveys and studies to understand the extent of the infection in the community (11, 32).

This initial symptom-based approach for case detection and subsequent testing was adopted in accordance with interventions used to control severe acute respiratory syndrome (SARS) in 2003, considering many similarities between SARS-CoV and SARS-CoV-2, including respiratory symptoms and incubation period (33). However, as the pandemic progressed, the recognition that a high proportion of infected individuals with SARS-CoV-2 - especially in the younger segments of the population (34) - is actually asymptomatic and that the viral load does not differ significantly between symptomatic and asymptomatic individuals (35) urged for a change in the testing strategy. Indeed, infected people with asymptomatic or pauci-symptomatic disease, not intercepted by healthcare services, could have a great impact on the transmission and diffusion of the infection putting a huge strain on the pandemic containment. Therefore, a shift from diagnostic-oriented and symptom-based testing strategy to a broader population-wide screening is essential in order to break the chain of transmission and prevent the spread of the infection. In this perspective, tests can really become a strong public health tool. Tests can also be used to screen crucial target groups like healthcare and social workers as part of local surveillance programs or as a screening in certain congregate settings such as long-term care facilities or schools. Therefore, from a broader point of view, testing is the starting point of the public health response to COVID-19 pandemic with either a clinical or a public health aim (36). However, when considering the deployment of a complex intervention such as a population-wide testing screening, many factors, such as the epidemiological situation, technical feasibility, logistics and resource should be taken into account, and case isolation and contact tracing should be planned and implemented accordingly in order to increase the effectiveness of the intervention (3).

Molecular tests may detect even small amounts of viral RNA and for a longer period than the transmission window (29), which only indicates that an individual is infected, not necessarily infectious. Because of this characteristic, cost, turnaround time, and resource required, individual molecular tests performed through RT-PCR on nasopharyngeal swabs do not appear as a feasible tool for mass screening.

In case of low pre-test probability, samples from several individuals could be analyzed together through a molecular detection method, possibly reducing the cost of supplies and time required to perform the analysis and expanding testing capacities of existing laboratories (37). This well-established technique in blood banking is called pooled testing (37) and could represent a method through which to conduct population-wide screening. Following the classic approach (Dorfman’s protocol (38)), each sample is analyzed individually if the pooled result is positive. However, variations of this pooling approach have been developed. Considering an initial pool of size 16 and adopting a 4-step halving pooling procedure (39), if the initial pooled sample is positive, it is halved, and each sub-pool is retested. The procedure continues by halving the sub-pools that test positive until the individual positive sample/s is/are identified. This split method would allow to use fewer tests and to improve the specificity, but the sensitivity of the original molecular test would be affected (from 99% to 94.1%). Still, if negative results were retested, the sensitivity would be higher than individual testing (99.92%) and would still require fewer tests. However, it could be personnel consuming for increased manual steps and reduced use of automated processes. Pooled testing was successfully used to screen 22,000 residents and staff at 131 nursing and residential care homes in Saarland, Germany, in order to early identify and isolate asymptomatic infected individuals (37). Whereas young people, such universities’ students, have a greater chance of asymptomatic infection, as previously mentioned, there are recently reported experiences (UK universities) of pooled testing screening programs for SARS-CoV-2 to look for undiagnosed community transmission (40). There are communications that also US schools, like Syracuse University, are using this pooling approach with saliva samples (41).

In the pool-testing method, the analysis process of a laboratory is the same as for standard individual testing. The difference concerns the analyte; instead of analyzing an individual specimen, a mixture of several specimens - a pool - is analyzed. In order to make the process even more efficient some innovative digital solutions have been proposed. For example, an open source software (42) for the management of SARS-CoV-2 pool testing is being developed to test 2 to 30 times more people with the existing resources of a laboratory. This tool may help to reduce the quantity of assays, reagents and time spent on each test, by suggesting the optimal strategies depending on the prevalence, optimization task and operational constraints. In particular, it seems to be useful to help laboratory personnel in improving the processes of specimen mixing plan, tracks specimens and assays, and decoding the results of assays completed into the results of specimens. However, organizations may prefer an invariant pooling strategy to simplify logistics.

Other digital solutions (43) are being developed to speed up PCR (both individual and pooled) interpretation process. They rely on automatic data interpretation algorithms to reduce hands-on analysis time, improve efficiency, and provide better accuracy, thus enhancing the diagnostic process.

Another method for population-wide screening consists in the use of rapid antigen tests, which have been highly advocated as screening tool despite their lower sensitivity compared to molecular tests. These types of tests did not play a primary role at the beginning of the pandemic, but were mainly considered in the following phases; they perform best when carried out in the early stages of infection, when viral load is generally higher. Indeed, they are really good to identify the so-called “transmission window”, the period during which infected individuals are most infectious (29, 44). The vast majority of antigen tests can be used as Point-Of-Care (POC) tests with manual interpretation or via instrument like immunofluorescence analyzer. Others require laboratory equipment like antigen tests based on Chemiluminescence Enzyme Immuno Assay (CLEIA). They are relatively inexpensive and have a short turnaround time, potentially returning results in approximately 15 minutes (7). Still, there has been a huge debate on the use of these tests due to their lower sensitivity compared to RT-PCR. The erroneous belief that poor performance can affect their use is misleading, especially if the stated purpose of the testing, following a public health approach, is to identify asymptomatic infectious individuals in order to break the chain of transmission and prevent possible outbreaks on a large scale. Further, volatility in laboratory-based PCR turnaround times creates a risk that test result delays will increase if there is a spike in testing demand during a spike in cases. A test’s effective sensitivity for the purpose of interrupting transmission drops as turnaround time increases because isolation is pushed farther back into the period of infectiousness.

As demonstrated by the following two hypothetical scenarios that imply the use of these tests, antigen tests can prove to be a valuable tool to counteract the pandemic. In the first scenario, assuming a low pre-test probability of 1%, a sensitivity ranging from 70% to 85% and a specificity ranging from 95% to 99%, the use of antigen test alone would determine a range of extremely high NPVs (99.7%-99.8%), but really low PPVs (20%-40%), as shown in Fig. 1 (A) and (B). The addition of the retesting of positive individuals with highly sensitive (99%) and specific (99%) molecular assays, as depicted in Fig. 1 (C) and (D), dramatically improves the PPVs (94%-98%), leaving the NPVs virtually unchanged.

Figure 1.

Positive predictive value (PPV) and negative predictive value (NPV) of antigen-based rapid tests assuming a pre-test probability of 1%, a sensitivity ranging 70% to 85% and a specificity ranging from 95% to 99%. Panels A and B illustrate the predictive values when a single rapid test is performed; panels C and D illustrate the predictive values when positives undergo confirmatory molecular testing. Sensitivity and specificity of molecular assays are assumed to be 99%.

In a second scenario, using a rapid antigen test with the same performance as above, but with higher pre-test probability (3.5%), the NPVs slightly decrease (99.0%-99.4%), while the PPVs improve (40-70%) compared to the previous scenario, as illustrated in Fig. 2 (A) and (B). Still, this is not sufficient to confirm the diagnosis of infection in case of a positive result and, again, by adding a high-performance confirmatory molecular test in case of positive result, the PPVs strikingly increase (98.5%-99.5%) while the NPVs are virtually unchanged (99.0%-99.4%) [Fig. 2 (C) and (D)].

Figure 2.

Positive predictive value (PPV) and negative predictive value (NPV) of antigen-based rapid tests assuming a pre-test probability of 3.5%, a sensitivity ranging 70% to 85% and a specificity ranging 95% to 99%. Panels A and B illustrate the predictive values when a single rapid test is performed; panels C and D illustrate the predictive values when positives undergo confirmatory molecular testing. Sensitivity and specificity of molecular assays are assumed to be 99%.

In both scenarios (with a considered low pre-test probability as for asymptomatic individuals), a negative result could allow to rule out the infection, while, in case of a positive result, a confirmatory molecular test would be necessary. Because antigen tests are particularly effective in detecting the maximum infectiousness period, and considering their lower cost compared to molecular assays, they should be frequently used to identify the transmission window and break the infection chain, thus representing a strong and needed tool to prevent SARS-CoV-2 spread, together with other fundamental NPIs. In light of these considerations, we advocate and support the weekly use of these rapid tests as a method for close periodic monitoring in a low-prevalence population, such as periodic monitoring of students and schools’ staff and healthcare professionals in high-risk settings such as long-term care settings. Moreover, these tests could be used by general practitioners in their ambulatories for screening a target population or for the differential diagnosis in individuals with less predictive symptoms than the typical anosmia/ageusia, persistent fever and cough (45), in order to rule out SARS-CoV-2 infection in case of a negative result. The calculator published by the BMJ (46) could be a helpful tool in interpreting test results in these settings, for different pre-test probabilities.

One of the limitations is represented by the required use of nasopharyngeal swab to carry out rapid antigenic tests. Currently, there are rapid antigenic tests on saliva that received market authorization in EU, but to date, in the Italian context, we are still waiting for further clinical validation studies (47).

Definitely, the use of saliva as specimen for rapid antigenic test would represent a true game changer in testing strategies.

Conclusions

Alongside non-pharmaceutical measures, SARS-CoV-2 testing represents one of the main pillars of public health response to the pandemic. After all, testing was one of the keys to contrasting the COVID-19 outbreak in Germany, South Korea and other countries, and the lack of testing capacity impeded early efforts in the US. In Italy, the number of molecular tests increased from less than 5,000 in late February to more than 230,000 in early November (48). However, testing per se does not confer health benefits; rather is useful to the extent it forms a critical link to subsequent medical or public health interventions.

Now that several SARS-CoV-2 tests are finally available on the market, it is important to remember that the effectiveness and ultimately the utility of testing programs depend not just on the accuracy of tests themselves, but on the goals and context of the testing program. Pre-test probability represents the main driver of both positive and negative predictive values, and this probability depends on the testing program’s primary goal, which we categorize as diagnosis, surveillance, or screening. Depending on the context and goals, even tests that at first could seem not useful and could arouse skepticism purely due to their performance characteristics can truly represent a fundamental piece of a timely public health response. Therefore, a change of perspective is necessary: choosing the proper type of test, considering its performance and use context, should be regarded as one of many elements that are the basis to develop interventions as effective as possible (29).

Testing strategies, planned and implemented accordingly to the target population and to the specific context and aim, could represent a key element among the interventions that, combined together, currently constitute the most effective way to counteract the COVID-19 pandemic. From this perspective, “the key question is not how well molecules can be detected in a single sample but how effectively infections can be detected in a population by the repeated use of a given test as part of an overall testing strategy – the sensitivity of the testing regimen” (29).

Conflict of Interest:

Each author declares that he or she has no commercial associations (e.g. consultancies, stock ownership, equity interest, patent/licensing arrangement etc.) that might pose a conflict of interest in connection with the submitted article