Machine learning for optimal test admission in the presence of resource constraints

Abstract

Developing rapid tools for early detection of viral infection is crucial for pandemic containment. This is particularly crucial when testing resources are constrained and/or there are significant delays until the test results are available – as was quite common in the early days of Covid-19 pandemic. We show how predictive analytics methods using machine learning algorithms can be combined with optimal pre-test screening mechanisms, greatly increasing test efficiency (i.e., rate of true positives identified per test), as well as to allow doctors to initiate treatment before the test results are available. Our optimal test admission policies account for imperfect accuracy of both the medical test and the model prediction mechanism. We derive the accuracy required for the optimized admission policies to be effective. We also show how our policies can be extended to re-testing high-risk patients, as well as combined with pool testing approaches. We illustrate our techniques by applying them to a large data reported by the Israeli Ministry of Health for RT-PCR tests from March to September 2020. Our results demonstrate that in the context of the Covid-19 pandemic a pre-test probability screening tool with conventional RT-PCR testing could have potentially increased efficiency by several times, compared to random admission control.

Highlights

• We propose that rapid early diagnostic tools for Pandemic-related virus infections can be developed using predictive analytics methods with an emphasis on machine learning algorithms.

• We show how the results of predictive models can be combined with simple decision rules to drive efficiency gains in the presence of constrained testing resources

• We develop an automated modeling pipeline and interactive website implementation to facilitate use of our techniques by medical practitioners

Introduction

One of the main challenges confronting healthcare systems during pandemics is accurate, rapid and efficient diagnostic strategies. In the case of Covid-19, for example, while large-scale testing was implemented by many countries over the course of the pandemic, the supply of test kits was often insufficient to meet the demand, especially in the beginning of the crisis. For example, in the province of Ontario, the largest province in Canada with the population of over 13 million, the health authorities were only able to perform about 20,000 tests per day until May 2020, and, while the testing capacity was increased, it never exceeded 60,000 tests per day [1]. This meant that at the height of several “waves” of the pandemic, access to testing was constrained. In fact, during the Omicron wave in the summer of 2021, access to testing was limited to those exhibiting symptoms and certain designated groups (healthcare providers, etc.).

The testing capacity shortages in the case of Covid-19 were further exacerbated as the economies began to reopen and mass testing was required in a variety of settings ranging from outpatient clinics to airports to manufacturing facilities. Another constraint on testing capacity was that proper administration of tests required training – as noted below, incorrect sample collection is thought to account for the low sensitivity of the real-time reverse transcriptase polymerase chain reaction (RT-PCR), which was used to test for SARS-COV-2 in the field [2, 3]. Also, diminishing virus counts in the nasal passage made it harder to detect in later stages of infection [4]. Finally, there was often a significant turnaround time until the test results are known—a delay of up to 48 h was quite common. Such delays have harmful effects on contact tracing, which in turn make pandemic containment efforts ineffective.

The goal of the current paper is to develop strategies for using limited test resources as effectively as possible, with the primary goal of detecting as many “true positives” (i.e., patients who are infected and whose infection status is correctly identified by the test). The key aspect of our approach is integration of machine learning and decision analysis techniques. The former are based on obtaining a reasonably accurate prediction of the outcome of the test based on patient characteristics, symptoms, as well as the current trajectory of the disease (% of positive tests in the last week, etc.). This prediction is used to derive an optimal admission policy: i.e., decide whether a particular presenting patient should be admitted to the test, given the current resource constraints. Starting with a basic admissions control policy, we extend it to the cases of multiple tests for high-risk patients, as well as pooled testing. In each case we derive necessary conditions on test and predictive model accuracy for our approach to result in efficiency gains.

We assume the availability of a database containing past test outcomes, as well as symptoms and patient characteristics (such as age or gender) observable prior to the test. Our modeling approach has two key components: an automated modeling pipeline designed to predict the outcome of a test if it is administered to a patient with particular symptoms/ characteristics, and model-based decision rules designed to deploy available testing resources as effectively as possible. The automated pipeline consists of a battery of machine learning models (in our implementation we use five common algorithms, but new ones are easy to add) that are automatically calibrated on the training sample, applied to the hold-out sample, the best-performing one is selected and used for scoring. These scores are then used as inputs by the decision rules. The basic decision rule specifies whether a patient displaying certain combination of symptoms and characteristics should be admitted for testing or turned away. It is based on the model score cut-off adapted to the current availability of testing resources and the current estimated baseline infection rate; we prove the optimality of this rule within a class of policies that admit a certain subset of patients for testing.

Our decision rules are meant to be applied in an online fashion (i.e., to one patient at a time, rather than in batch). To account for time-varying disease prevalence, as well as possible differences in patient populations over time, we apply cluster analysis to ensure that each patient’s scores are compared to those of a relatively homogenous group of patients, so that highest-risk patients within each group can be properly identified.

Our automated modeling pipeline is implemented in Python, while the decision rule is implemented via an interactive website easily accessible by practitioners.

Diagnostic tools based on predictive modeling and machine learning to predict the outcome of the viral infection are important for combatting pandemics. Such tools can be clinically useful in several ways. In addition to more efficient use of limited testing resources by prioritizing patients most likely to test positive, which is the main focus of the current paper, pre-test tools can be useful in obtaining instantaneous risk assessment when clinical tests are either unavailable or the results are delayed, during which time appropriate medical intervention, including contact tracing, can be started for high-risk patients (see [5] for a discussion of the importance of delays in contact tracing).

We illustrate the effectiveness of our approach with a case study using a large dataset (over 1.53 M observations) from the Israeli Ministry of Health (IMOH) to train and evaluate a variety of machine learning algorithms to predict the outcome of the RT-PCR test for SARS-Cov2. The resulting predictions are shown to be quite accurate in identifying highest and lowest risk patients, driving significant efficiency gains when combined with our admission control policies. While the dataset we use is specific to Israel, our methodology and the machine learning pipeline are quite general and can be applied to any locality or any pandemic.

As noted earlier, the essential aspect and assumption underlying our approach is that the values of the predictors (e.g., symptoms, gender, age, etc.) for each patient should be known prior to the test being administered; in particular, these values should be available for all patients tested in the past, regardless of the test outcome. This requirement should not, in principle, be hard to satisfy as patients arriving for testing typically have to fill out a number of registration forms that are used for a variety of purposes (insurance filings, patient contacts, contact tracing, etc.). Somewhat surprisingly, based on our review of the literature, data sets that satisfy these requirements appear atypical: many publicly available data sets either contain symptoms data only for patients testing positive, collect symptoms after the test results are known, or re-construct “patient state” at the time of testing manually from patient’s medical files (see, e.g., [6]); the latter, of course, results in a time consuming process and a relatively small data set. We hope that our results will motivate health authorities to systematically collect the data on pre-test status of each patient, including their characteristics and symptoms.

We finish the current section with a brief review of the relevant literature. To the best of our knowledge, no previous studies attempted to combine effective decision rules aimed at maximizing test efficiency with machine learning-based predictions of test outcomes based on patient characteristics observed prior to testing. There are, however, growing bodies of literature examining each of these issues separately. Several previous studies looked at pre-test screening for SARS-COV-2, constructing predictive models of test outcomes, without employing formal decision rules based on them. While a number of these studies have used relatively small samples and utilized a limited set of statistical methods [6–9], machine learning tools were used in Ahamad et al., however the data set used was relatively small [9]. A notable exception was Shmueli et al. [10], who used a large and rich dataset (their set of symptoms and patient characteristics was significantly larger than in our IMOH dataset) and a variety of machine learning algorithms. The accuracy levels they achieved were somewhat lower than ours, possibly due to the lack of predictors tracking the changing infection prevalence rates over time. Model-based decision algorithms were outside the scope of their study.

We note that a general notion that big-data and machine learning tools can help curb pandemics has been gaining traction lately [11–15]. In a systematic review, Wynants el at. have identified 107 studies describing 145 prediction models mostly aimed at early diagnosis and setting prognosis for COVID-19 patients [15]. Yet most prediction models were for patients presenting to the hospital and have included variables such as imaging results and blood tests. Furthermore, [15] noted that proposed models are at a high risk of bias and are based on relatively small study population that might not be representative of the general population. In both, test setting, as well as in study population size, we believe our research is unique.

A separate stream of literature is concerned with the optimal use of limited testing resources. Here the main emphasis is on pooled testing, where samples collected from individual subjects are pooled, the pooled samples are tested, and the individual samples belonging to the pooled samples testing positive are re-tested to identify infected individuals within the pool. This line of research originates with Dorman [16] and includes a large body of literature. Of particular relevance are recent contributions by Aprahamian et al. [17] and Bish et al. [18] who derived optimal pool size both in the deterministic case, and in the presence of uncertainty about infection prevalence. We refer the reader to references in the above-mentioned papers for further literature on pooled testing. Note that the premise and the objective of pooled testing is significantly different from ours: no prior predictive modeling is assumed (i.e., all patients are treated as being equally likely to be infected a priori), and the objective is to minimize the expected number of tests per patient, assuming that all presenting patients must be tested. As noted earlier, our objective is to detect as many true positives as possible, given the available testing resources.

The plan for the rest of the paper is as follows. The key theoretical results underlying our methodology are presented in Section 2. The implementation, including data expectations and preparation, the modeling pipeline, and the decision rule are presented in Section 3. The case study is described in Section 4. Some concluding remarks and open questions are presented in Section 5.

Methodology I: theory

We assume that in a given time period we have limited testing capacity of T (this is the number of tests we can administer) and we have N > T subjects presenting for testing (here and hereafter we will refer to these subjects as “patients” even though the vast majority may not require any medical treatment). We treat the available test capacity as a “perishable inventory”, i.e., the capacity cannot be transferred from period to period. This is a reasonable assumption when the bottleneck capacity constraints involve nursing staff and test processing times, rather than physical resources, such as swabs or reagents. Similarly, we assume that patients presenting for testing cannot be “inventoried”, i.e., transferred from one period to the next. While, in principle, such transfers may be possible, they are generally viewed as undesirable, particularly for patients who are at the highest risk of being infected (as will be seen below, our policies focus on precisely this group of patients).

We assume our test has a known false positive rate of α (this is 1 – specificity of the test) and false negative rate of β (equal to 1 – sensitivity). While the values of these parameters are test-specific, we will use the values corresponding to the “gold standard” RT-PCR test for SARS-COV2 as our reference point throughout the paper. While this test has excellent in vitro sensitivity and specificity, it has been repeatedly documented to suffer from much lower in vivo sensitivity. This is due to several factors, including: (1) the nasal sampling procedure requires deep swab, which, if done incorrectly, can miss virus-bearing mucous; (2) as the infection progresses, the amount of virus present in the nasal passage varies, making it harder to detect during certain stages of the infection [4, 19]. The available estimates of in vivo sensitivity for RT-PCR test fall between 70 and 80% [20, 21]. On the other hand, the in vivo specificity of the test is thought to be high, 99% or above (though some concern about false positives when using high cycle counts has been expressed). We will thus use the values of sensitivity = $1-\beta =$ 80% and specificity = $1-\alpha =99\%$ throughout the paper, unless noted otherwise.

Note that the assumption that true values of test sensitivity and specificity are known may not be reasonable in some cases, especially during the early part of an epidemic before a significant number of test results are accumulated. Since we assume that a large enough data file on prior test results has been accumulated to construct reasonably precise predictive models, the value of sensitivity and specificity are likely to have already been estimated with some precision. Thus, we will largely ignore this issue, but will touch on it again in Section 5.

The four possible outcomes of administering a test to a patient are: True Positive (TP), True Negative (TN), False Positive (FP) and False Negative (FN). Since the primary purpose of testing is to identify patients who are infected, we define test efficiency as the TP rate – the expected number of TP outcomes per conducted test.

If we further assume that (1) the current disease prevalence rate is q – this is the probability that, at a certain point in time, a randomly selected patient is infected, and (2) that tests are administered at random, i.e., tests are administered to a random selection of T out of available N patients (for $T\le$ N), then the resulting test efficiency is

$$TE=q(1-\beta )$$ 1

We refer to this as a “no-screen test admission” policy and treat $\mathit{TE}$ as the “baseline” efficiency level. This policy is illustrated on the left side of Fig. 1.

Fig. 1.

Process models with and without screening

We note that the prevalence rate q cannot be observed directly; what can be observed is the test positivity rate p. However, under the assumptions that tests are administered at random, as well as that $\alpha$ and $\beta$ are known, it is not hard to derive the estimated prevalence rate

$$\widehat{q}=\frac{p-\alpha }{1-\alpha -\beta }.$$ 2

We illustrate with the results for the province of Ontario, Canada. The provincial COVID test results were reported daily [1], starting with Feb 6, 2020. Through the end of our data set (i.e., July 2021), the “test positivity rate” never exceeded 10.9%, with the average value of 3.5%—that is the daily percentage of positive tests outcomes. Using Eq. (2), we obtain

$$\widehat{q}=\frac{p-\alpha }{1-\alpha -\beta }=\frac{.035-.01}{1-.01-.2}=3.16\%,$$

It is interesting to note that the actual prevalence rate is 10% lower than the test positivity rate – the fact missed by most public reporting.

From this we obtain the estimated test efficiency of $TE=0.0316\ast \left(1,-,.,2\right)=2.53\%.$ Thus, while throughout the pandemic there have been well-documented struggles by the government of Ontario to ensure sufficient testing capacity, the average test efficiency has stayed below 3%, indicating that 97% of testing capacity may have been “wasted”. This is typical of the periods of the pandemic when $q$ is low. This is also when contact tracing is particularly effective, and thus identifying as many TP patients as possible is particularly valuable.

How can we improve test efficiency beyond the baseline level? One obvious answer is pre-test admission control (i.e., screening): instead of admitting patients to testing at random, we could focus the testing on those who are most likely to be infected (the remaining patients can either be turned away, or re-directed to “pooled testing” discussed in the following section). While the admission control can take many forms (we will discuss some specific simple policies below), all policies require some ability to identify patients who are more or less likely to test positive. Specifically, we need a mechanism (a predictive model) that, for a given patient i produces a prediction (probability) $P_i$ that this patient will test positive based on patient characteristics observable at the time when the admission control decision is made. In addition to the prediction, we also require the standard measures of accuracy of this predictive mechanism, i.e., the estimates of sensitivity and specificity of the prediction model.

In the following subsections we discuss the basic predictive-model-based admission control mechanism, as well as several extensions.

Threshold-Based Admission Control

To illustrate the basic idea, suppose we consider the following threshold-based pre-screening policy: admit for testing only patients for whom the predicted probability ${\mathit{P}}_{\mathit{i}}$ of a positive outcome is above some threshold $\overline{\mathit{P}}(\mathit{T},\mathit{N})$. The conditions for optimality of such a policy are discussed below. Note that the threshold is a function of the available testing resources, as well as the number of patients presenting for testing.

Any admission policy, including the one based on the threshold $\overline{P}(T,N)$, is associated with sensitivity $p_{\mathit{SE}}$—the proportion of all positively testing patients who are admitted for testing, and specificity $p_{\mathit{SP}}$- the proportion of negatively testing patients who are refused admission (or are re-directed to pool testing, depending on the admission policy). Note that these sensitivity and specificity parameters are different from the ones related to the test introduced earlier: $p_{\mathit{SE}},p_{\mathit{SP}}$ refer to the probability of a patient testing positive, while the earlier parameters refer to the presence of disease given a positive or negative test. The feasible values of $\left(p_{\mathit{SE}},,,p_{\mathit{SP}}\right)$ for a given prediction model are commonly depicted on a ROC curve, such as the one presented on Fig. 4 . Since each threshold corresponds to a specific point on the ROC curve, we can think of any threshold-based decision rule as being specified by the $\left(p_{\mathit{SE}},,,p_{\mathit{SP}}\right)$ pair.

Fig. 4.

ROC Curves for Asymptomatic (“Symptomatics = 0) and Symptomatic (Symptomatic = 1) patients

The basic pre-screening approach associated with sensitivity and specificity values $\left(p_{\mathit{SE}},,,p_{\mathit{SP}}\right)$ is illustrated on the right-hand side of Fig. 1. Each patient belongs to one of two classes: those who would test positive if admitted and those who would not. Which class the patient belongs to is not known prior to testing. Note the “positive if tested” is a combination of TP and FP subgroups. Similarly, the “negative if tested” class is composed of TN and FN subgroups. The decision rule (correctly) admits $p_{\mathit{SE}}$ proportion from the ‘test positive” group (incorrectly) ${1-p}_{\mathit{SP}}$ proportion from the “test negative” one (the remainder are correctly identified as belonging to the negative group and thus not tested). The probabilities of the resulting outcomes are given on the right-hand side of Fig. 1.

Collecting the appropriate terms, the efficiency ${\mathit{TE}}_s$ for the pre-screen procedure, i.e., the TP rate among tested, is thus given by

$${\mathit{TE}}_s\text{=}\frac{q{p}_{\mathit{SE}}(1-\beta )}{1-[q\left(1,-,\beta \right)+\left(1,-,q\right)\alpha ]\left(1,-,p_{\mathit{SE}}\right)-{[\left(1,-,q\right)\left(1,-,\alpha \right)+q\beta ]p}_{\mathit{SP}}}.$$ 3

Comparing with Eq. (1), the efficiency gain EG vs. no pre-test regime given by

$$\text{EG}=\frac{{\mathit{TE}}_s}{\mathit{TE}}=\frac{p_{\mathit{SE}}}{1-[q\left(1,-,\beta \right)+\left(1,-,q\right)\alpha ]\left(1,-,p_{\mathit{SE}}\right)-{[\left(1,-,q\right)\left(1,-,\alpha \right)+q\beta ]p}_{\mathit{SP}}}$$ 4

Thus, the pre-screening approach will outperform no-screen admissions as long as $EG\ge 1.$ The following result provides necessary and sufficient conditions for EG exceeding 1.

Theorem 1

Suppose a prediction model estimates probability of positive outcome $P_i$ for each patient $i$. Consider an admission policy that admits patients for testing only if $P_i\ge \overline{P}$ for some threshold $\overline{P}\ge 0$ and let $\left(p_{\mathit{SE}},,,p_{\mathit{SP}}\right)$ be the sensitivity and specificity values corresponding to this threshold. Then the efficiency gain $EG\ge 1$ if, and only if,$p_{\mathit{SE}}+p_{\mathit{SP}}\ge 1.$

The proof can be found in Appendix A1. The condition in Theorem 1 requires that the selected point $\left(p_{\mathit{SE}},,,p_{\mathit{SP}}\right)$ corresponding to the threshold $\overline{P}$ must fall above the 45° line on the ROC curve of the underlying predictive model. Intuitively, this boils down to “better than random” requirement for the model – a very mild condition (a well-behaved model should have ROC curve that stays above the 45° line at every point). As we will see in Section 4, even with rather limited symptoms data, this condition is easily met, and very substantial efficiency gains can be achieved.

It is important to note that the denominator in (3) and (4) is the proportion of patients admitted to testing. Thus, assuming Theorem 1 holds, the efficiency is automatically increased when more patients are turned away. It follows that when comparing the efficiency of two decision rules (or screen vs no-screen conditions) one needs to ensure that the same number of tests T are conducted under the alternative admission policies.

An easy way to ensure this for a threshold-based admission policy is to set $\overline{P}\left(T,,,N\right)=\mathcal{P}\left(\frac{T}{N}\right),$ where $\mathcal{P}\left(\frac{T}{N}\right)$ is the predicted probability of the positive test corresponding to percentile $\frac{T}{N},$ i.e. such that $\frac{T}{N}\%$ of patients presenting for testing have scores no higher than $\mathcal{P}\left(\frac{T}{N}\right)$ (as before, we assume $T\le N).$ The next result shows that this “constrained” admission policy is optimal under the condition similar to Theorem 1 (see Appendix A1 for the proof).

Theorem 2:

Suppose the test’s false negative and false positive rates satisfy $\alpha +\beta \le 1.$ Consider a class of admission policies relying on the same prediction model to estimate the probability of a positive test, and where the number of tests to be conducted is set to $T\le N$. Let $p_{\mathit{SE}}\left(\frac{T}{N}\right),p_{\mathit{SP}}(\frac{T}{N})$ be the model sensitivity and specificity at percentile $\frac{T}{N}$. Suppose $p_{\mathit{SE}}\left(\frac{T}{N}\right)+p_{\mathit{SP}}(\frac{T}{N})\ge 1$ holds. Then the admission policy using threshold $\mathcal{P}\left(\frac{T}{N}\right)$ achieves optimal efficiency in this policy class.

The intuition behind the preceding result is that as long as the predictive model is consistent in ranking a patient who is more likely to test positive above one who is less likely, it would be suboptimal to admit the latter ahead of the former. The expected ranking order is ensured by the condition of the Theorem.

Remark 1

Note that the decision rule in Theorem 2 is very easy to implement: for a given T and N we first compute $\frac{T}{N}$ and then, from the table of model scores ranked in decreasing order, find the score $\mathcal{P}\left(\frac{T}{N}\right)$ corresponding to this percentile (this table is only needed for a sample of patients that is representative of the population presenting for testing in the current period). Then for any patient under consideration, we simply compute their score and compare it to the threshold. The values $p_{\mathit{SE}}\left(\frac{T}{N}\right),p_{\mathit{SP}}(\frac{T}{N})$ are not required – we only need to know that the conditions of theorem are satisfied; as previously noted this condition holds if the ROC for the predictive model is above the 45° line.

Remark 2

An alternative view of the objective of testing is to “uncover” TP cases in the population available for testing, so that “true” infections can be isolated and treated. Under our assumptions, TP = qN. If testing resources allow for all N patients to be tested (i.e., $T=N$), then the number of detected TP cases will be $\widehat{\mathit{TP}}=qN\left(1,-,\beta \right)=N\ast TE=T\ast TE.$ For $T<N,$ assuming random selection, $\widehat{\mathit{TP}}=T\ast TE.$ Thus maximizing efficiency is equivalent to maximizing $\widehat{\mathit{TP}}.$ It is not hard to see that for the screen-based admission policy with threshold $\mathcal{P}\left(\frac{T}{N}\right)$ we obtain ${\widehat{\mathit{TP}}}_S=T\ast {\mathit{TE}}_S,$ and thus efficiency maximization is equivalent to TP detection.

In the following two sections we extend the results presented above in two ways. First, in Section 2.2 we consider the popular strategy of pooled testing and show how it can be optimally combined with threshold-based admission control. Second, in Section 2.3 we relax the assumption that each patient is only tested once and allow for the possibility of re-testing some high-risk patients whose initial test is negative.

Pooled Testing and Mixed testing Regime

In the previous section we considered the “basic” pre-screening approach under which $T$ patients are admitted for testing and the remaining $N-T$ are turned away. In many settings, medical authorities may be uncomfortable with rejecting any patients at all, thus requiring that all N presenting patients be tested. A popular methodology for stretching the limited $T<N$ resources to all patients is pooled testing, where each patient is randomly assigned to a group (pool) consisting of S patients, the samples from the pool are mixed and only one test is administered to the pooled sample. If this test is positive, then follow-up testing of all S patients in the pool is performed. Please see Aprahamian et al. [17] for more details on this testing methodology.

The implicit assumption behind pooled testing is that the constraining resource is the number of samples that can be tested. This is not always the case – if the constraining resource are the nurses that collect samples from individual patients, or the test swabs used for individual samples, then individual testing described in the previous section is the only option. However, there are many situations when the number of samples to be tested is the bottleneck resource and pooled testing applies.

Note that the goal in pooled testing design is not the maximization of efficiency in the sense defined earlier (i.e., maximizing number of TPs per performed test). This is because the assumption is that all N presenting patients will be tested and thus the number of TPs can be shown to be $Nq{(1-\beta )}^2$ (see Lemma 2 below for details). Rather, the goal is to minimize the number of tests, which in our case is fixed at T. However, efficiency becomes a meaningful objective under the following mixed testing regime:

Assume $N_I$ patients are tested individually and $N_P$ are directed to pooled testing, where

$$N_I+N_P=N$$ 5

(i.e., all patients are tested). Assume further that group size $S\le N_P$ is used for pooled testing. Then, using Eq. (1) in [17], the expected number $E\left[S\right]$ of tests per group is given by

$$E\left[S\right]=1+S\left[1,-,\beta ,-,\left(1,-,\alpha ,-,\beta \right),{\left(1,-,q,\left(N_P\right)\right)}^S\right],$$ 6

where $q\left(N_P\right)$ is the disease prevalence (i.e., probability of infection) within the pooled testing group. The expected total number of tests is given by

$$N_I+E\left[S\right]\frac{N_P}{S}=T$$ 7

(note that in the previous expression we used $\frac{N_P}{S}$ as the number of pooled groups, which ignores the restriction that this number must be integer; enforcing this restriction leads to some messy algebra without additional insight, so we will ignore it).We also assume that the pooled testing group size is constrained from above:

$$S\le S_{\mathit{max}}.$$ 8

This constraint is motivated by the “dilution” effect that occurs when pooled test size becomes too large to detect the presence of a single infected individual within the group; [18] suggests that for PCR tests the commonly used maximum group size is $S_{\mathit{max}}=32.$

Thus, the objective of the mixed testing regime is to select the values of $N_I,$ $N_P,S$ so as to maximize the expected number of True Positives (TPs) detected, while satisfying constraints (5), (7), and (8) above.

We next establish the expected number of true positives for a particular mixed testing policy. Note that the latter is defined by five quantities: $N_I,$ $N_P,S$ above, as well as the prevalence $q_I,q_P$ within each group, where the latter depends on the allocation of patients to the individually and pooled testing regimes. The relationship between within-group prevalences is given by

$$qN=q_I{N}_I+q_P{N}_p.$$ 9

Since every patient in the individually tested group will be tested, the expected number of TPs is given by (1):

$${\mathit{TP}}_I=q_I{N}_I\left(1,-,\beta \right).$$ 10

The expected number of TPs for the pooled-testing group is given by the following result (we note that a similar derivation is given in [17], the proof is provided in the Appendix).

Lemma 1

The expected number of TPs in each pooled-tested group of size $S$ is given by $TP\left(S\right)=S{q}_P{(1-\beta )}^2$, and the total expected number of TPs in the pooled tested group is ${\mathit{TP}}_P=q_P{N}_P{(1-\beta )}^2.$

As noted earlier, it follows from the previous result that the expected number of TPs (and hence the test efficiency, as defined earlier) under pooled testing is independent of the group size. The total expected number of TPs, i.e., the objective we seek to maximize under the mixed testing regime, is thus given by:

$$TP=q_I{N}_I\left(1,-,\beta \right)+q_P{N}_P{\left(1,-,\beta \right)}^2=q_I{N}_I\left(1,-,\beta \right)\beta +qN{(1-\beta )}^2,$$ 11

where the second equality follows by substituting (5) and (9) for $N_P$ and $q_P$ respectively. The following results now follows immediately from (11) and Theorem 2.

Theorem 3

Suppose the conditions of Theorem 2 hold. Then for any $N_I>0$ the optimal policy will allocate $N_I$ patients with highest predicted model scores for individual testing.

The logic behind the previous result is that under Theorem 2, for any fixed value of $N_I,$ the number of TPs in individual testing group is maximized by “highest scores” allocation, which means that $q_I$ is maximized as well, while the right-hand-side of (11) implies that the overall objective increases with $q_I.$

Note that for the individual testing group, Theorem 3 implies using the threshold-based policy corresponding to the score $\mathcal{P}\left(N_I,\%\right),$ where $N_I\%=\frac{N_I}{N}$ is the percentile of the total population directed to individual testing. As discussed previously, this threshold corresponds to a unique model sensitivity and specificity pair $p_{\mathit{SE}}\left(N_I,\%\right),p_{\mathit{SP}}\left(N_I,\%\right).$ In particular, we can express $q_I$ as

$$q_I=q\frac{[p_{\mathit{SE}}\left(N_I,\%\right)+\beta (1-p_{\mathit{SE}}\left(N_I,\%\right)-p_{\mathit{SP}}\left(N_I,\%\right)]}{N_I\%}$$ 12

An immediate consequence of Theorem 3 is that (assuming its conditions hold), for any $N_I,$ $N_P$ satisfying (5), (7) above we must have

$$q_I>q>q_P$$ 13

(since highest-risk patients are allocated to the individual testing group). Observe that by (10) and Lemma 1, the per-test efficiencies (i.e., probability of TP per test) for the individual and pooled testing groups, respectively, are given by

$${\mathit{TS}}_I=q_I\left(1,-,\beta \right)\ge {\mathit{TS}}_P=q_P{\left(1,-,\beta \right)}^2,$$

where the inequality holds by (13) and since $\beta <1.$ This immediately leads to the following corollary of Theorem 3.

Corollary 1

The mixed regime test efficiency is maximized by allocating as many patients as possible to the individual testing group, while satisfying constraint (5) and (7).

We can now derive the optimal values of $N_I,$ $N_P,$ and $S.$ Unfortunately, the exact computation is messy and can only be done numerically. We outline the computation below. First observe that, using $N_p=N-N_I,$ constraint (7) becomes an equation with just two unknowns: $S$ and $N_I.$ Second, $N_I$ is clearly maximized by choosing the group size $S^{\ast }$ that achieves the minimum $E^{\ast }={\mathit{min}}_S\frac{E\left[S\right]}{S},$ i.e., the expected number of tests per patient for each group. This optimization problem is discussed in [17]; we summarize the relevant results below.

Theorem 4

(based on Eqs. (2), (4) and Theorem 1 in [17]).

Let $\underset{\_}{q}=1-\exp \left[-,\left(1,-,\alpha ,-,\beta \right),e^{-1}\right]$ and $q_P=(qN-N_I{q}_I)/(N-N_I).$ Then.

• (i)If $q_P>\underset{\_}{q}$ then $S^{\ast }=S_{\mathit{max}}$ (i.e., optimal group size is the maximal allowed size) and $E^{\ast }=E(S^{\ast })/S^{\ast }\approx (1-\beta )$, where $E\left(S^{\ast }\right)$ is given by (6).
• (ii)If $q_P\le \underset{\_}{q}$ then $S^{\ast }\approx \sqrt{\frac{1}{q_P\left(1,-,\alpha ,-,\beta \right)}},$ with the exact value given by Eq. (4) in [17]. Moreover, $E^{\ast }=E(S^{\ast })/S^{\ast }<1.$

The previous result allows us to express $S^{\ast }$ and $E^{\ast }$ in terms of a single variable $N_I$ (the dependence is via $q_I,$ which determines $q_P)$. Thus, from (5), (7) and (12) we obtain the following two equations that determine $N_I\%$(we express it as a percentage of the patient population N):

$$\left\{\begin{array}{c}{N}_I\%\left(1,-,E^{\ast }\right)+E^{\ast }=T\%\\ q_P=q[1-p_{\mathit{SE}}\left(N_I,\%\right)-\beta (1-p_{\mathit{SE}}\left(N_I,\%\right)-p_{\mathit{SP}}\left(N_I,\%\right)]/(1-N_I\%)\end{array}\right)$$ 14

where $T\%=T/N$ is the testing capacity as a percent of N, $E^{\ast }$ is given in Theorem 4, and quantities ($p_{\mathit{SE}}\left(N_I,\%\right),p_{\mathit{SP}}\left(N_I,\%\right))$ come form the ROC curve of the predictive model for a given value of $N_I\%.$

While Eqs. (14) do not have a closed-form analytical solution, they are easy to solve numerically (e.g., by line search) since only one variable is involved. Note that these are two equations with one variable, thus either a unique solution exists, or the system is infeasible. The infeasibility will occur whenever the testing resources T are insufficient to meet constraint (5), i.e., to ensure that all presenting patients are tested either individually or via pooled testing.

In fact, perhaps a better use of system (14) is for planning: treating $T\%$ as an objective to be minimized, rather than a given constant, the solution tells us what is the minimal testing capacity $T\%$ (as a percentage of patient population available for testing) that is needed to ensure that all patients can be tested in the mixed mode, as well as the percentage of the patient population that should be allocated to the individual testing mode. We illustrate these ideas with an example below.

Example 1.

Consider the data on Table 1. The table has two panels corresponding to two populations (“S” and “A”) differing by the prevalence of the infection: $q=31.3\%$ for “S” and $q=2.9\%$ for “A”. The first three columns come from the ROC curve for the (hypothetical) predictive model for each population, where for each value of $N_I\%$(column 3) the corresponding values of ($p_{\mathit{SE}}\left(N_I,\%\right),p_{\mathit{SP}}\left(N_I,\%\right))$ are given in the first two columns. Note that the first row on each panel corresponds to $N_I\%=0$ (i.e., 100% of patients are directed to pooled testing). The values of $q_P$ in the fourth column are computed via the second equation in (14), from which we compute the optimal pooled group size $S^{\ast }$ via Theorem 4 and then $E^{\ast }=E(S^{\ast })/S^{\ast }$ via Eq. (6). The latter is presented in column 5 of the table, representing the expected number of tests per patient in the pooled patient group. The final column is $T\%,$ the amount of testing resources required relative to the size of patient population presenting for testing. Thus, the top row of Table 1 (upper panel) indicates that in the “S” population, using only pooled testing would require $T=.83N.$ This amount drops to $.78N$ when $N_I\%=26.7\%$, i.e. individual testing is used for 26.7% of patients (row 4 of the table). It is interesting to note that the required resource amount $T\%$ first increases and then drops as $N_I\%$ is increased from $0\%$ to $26.7\%$, before increasing when $N_I\%$ increases above $26.7\%.$ This is because increasing $N_I\%$ has two effects: it reduces the prevalence $q_P$ in the pooled testing group (this can be seen in column 4), which reduces $E^{\ast }$ (column 5), while at the same time directing more patients to the more “expensive” individual testing mode, which requires 1 test per patient, vs. the pooled mode that always requires less than 1 test (as per Theorem 4). Thus $T\%$ is minimized at some optimal value of $N_I\%\ge 0.$

Table 1.

Data for Example 1. Disease prevalence q is 31.3% for “S” and2.9% for “A”

	PSE	PSP	NI%	qp	E*	T%
"S" pop	N/A	N/A	0.0%	31%	83%	83%
	50%	99%	16.3%	22%	82%	85%
	60%	97%	20.8%	20%	73%	79%
	70%	93%	26.7%	18%	70%	78%
	80%	83%	36.7%	16%	67%	79%
	95%	30%	77.8%	14%	63%	92%
"A" pop	N/A	N/A	0%	3%	25%	25%
	50%	99%	2%	2%	18%	20%
	60%	89%	12%	2%	18%	28%
	70%	70%	31%	2%	18%	44%
	80%	51%	50%	1%	18%	59%
	90%	30%	71%	1%	18%	76%

Comparing the top and bottom panels on Table 1 we see similar effects. However, in the “A” population, with much lower disease prevalence (over 10 × less than in “S”), the pooled testing is much more efficient with respect to testing resource usage: the minimum $T=.20N$ is reached at $N_I\%=2\%$ (i.e., 98% of patients are directed to pooled testing). This makes intuitive sense: when disease prevalence is low, individual testing should be reserved for only the highest risk cases.

We also note that Table 1 can be used to find $N_I\%$ for a given value of $T\%$, as long as the latter is restricted to the feasible range $T\%\ge 78\%$. E.g., suppose for the “S” population $T\%=80\%.$ We observe that this corresponds to $N_I\%$ between $16.3\%$ and $20.8\%.$ We can now use ROC curve of the predictive model to obtain additional values of ($p_{\mathit{SE}}\left(N_I,\%\right),p_{\mathit{SP}}\left(N_I,\%\right))$ for $N_I\%$ in that range, or use simple linear interpolation $N_I\%\left(80,\%\right)\approx 20.8\%\frac{.80-.79}{.85-.79}+16.3\%\frac{.85-.80}{.85-.79}=17.4\%.$

To summarize, we have shown how to optimally combine individual and pooled testing – the highest-risk patients (i.e., with highest predicted model scores) should be allocated to the individual testing group, and the remainder to pooled testing. It is important to note that the mixed testing regime achieves higher efficiency both, with respect to the detection of TPs (the concept of efficiency used in the current paper), and the use of the required testing resources (the concept of efficiency commonly used in the pooled testing literature) – the latter effect is illustrated in Example 1 above.

Extension to Multiple Testing

Our threshold-based admission control approach developed in Section 2.1 can also be extended to the case where some patients may be tested repeatedly. Indeed, for a test with relatively low sensitivity, such as RT-PCR for SARS-COV-2, when a patient suspected of being infected has a negative test result, a natural idea is to re-test. This idea is particularly appealing when results of the repeated tests are likely to be independent of each other (i.e., a FN outcome on a given test does not change the probability of the FN outcome on the following test), as opposed to being correlated. As mentioned earlier, low in vivo sensitivity of the RT-PCR test is thought to be mostly due to human errors during specimen collection (swab inserted insufficiently deep, etc.) – conditions that are more likely to lead to independent outcomes on repeated tests.

In this section we discuss the optimal repeated testing regime under constrained testing resources (as before, we assume that at most T tests can be conducted, while the number of patients available for testing is N > T) and independent testing assumption. Consider first a random admission policy and let ${\mathit{TP}}^{(k)}$ be the probability that a TP condition is discovered on the k-th test administered to the same patient for some k. Observe that this is the probability that the following events occur: (a) patient is infected, (b) we observe k-1 FN outcomes on the first k-1 tests, and (c) we observe a TP outcome on the k-th test. Assuming independence of the repeated tests, we obtain

$${\mathit{TP}}^{(k)}=q{\beta }^{k-1}\left(1,-,\beta \right)$$ 15

Since k > 0 we have ${\mathit{TP}}^{(k)}<{\mathit{TP}}^{(k-1)}.$ It follows that the probability of discovering a new TP case is always higher when testing a patient who has not yet been tested (i.e., has k = 0), compared to re-testing a patient who has already been tested k > 0 times. This is summarized in the following observation:

• Observation 1: Under a random-access policy with T < N, it is never optimal to re-test a patient.

Next we assume that using a prediction model it is possible to classify all patients into m groups with group i containing $n_i$ patients with the average positivity rate (i.e., percent testing positive within the group) of $p_i$ where $p_1>p_2>\cdots p_{m.}$ For concreteness, we can think of these groups as deciles, i.e., $n_i=\frac{N}{10}$, ordered by decreasing “lift” $\frac{p_i}{p}$, where p is the average positivity rate. However, we do not generally require the groups to be of equal size.

First, we convert positivity rates to prevalence rates (i.e., percentage of TP’s in each group) using Eq. (2):

$$q_i=\frac{p_i-\alpha }{1-\alpha -\beta }.$$

Note that $q_1>\cdots >q_m$. Following (15) we now define ${\mathit{TP}}_i^{\left(k\right)}$ as the probability of discovering a TP outcome for a patient on the k-th round of testing for some $k\ge 0$ and $i\in \left\{1,,,\cdots ,,,m\right\}:$

$${\mathit{TP}}_i^{(k)}=q_i{\beta }^{k-1}\left(1,-,\beta \right)$$ 16

Comparing ${\mathit{TP}}_i^{\left(k,+,1\right)}$ and ${\mathit{TP}}_j^{\left(k\right)}$ for groups i and j we see that ${\mathit{TP}}_i^{\left(k,+,1\right)}>{\mathit{TP}}_j^{\left(k\right)}$ holds when $i<j$ and

$$q_i\beta >q_j,$$ 17

implying that re-testing a patient from a lower-indexed group i for k + 1-st time is more efficient than testing an individual from a higher-indexed group j for the k-th time as long as inequality (6) holds (note that this statement is independent of the value of k). Applying this result to k = 1 (i.e., the initial test), we obtain the following characterization of conditions under which re-testing is optimal:

Theorem 5:

Assume the number of tests is limited by T and let $j\left(T\right)=\min \left\{j,:,{\sum }_{i=1}^j,n_i,\ge ,T\right\}.$ If inequality (17) holds for $i=1$ and $j=j\left(T\right),$ then an efficiency-maximizing admissions policy must involve some retesting.

Note that j(T) is the index of the highest-numbered group from which a threshold-based policy in Section 2.1 would admit a patient. The result holds since, by (16) and (17), admitting a patient from group j(T) is less efficient than re-testing a patient from group 1.

The condition in Theorem 5 is easy to check and is not too restrictive when (1) the predictive model is accurate, implying a steep drop-off in $p_i$ (and thus $q_i$) values with i, (2) the false negative rate $\beta$ is not too low, and (3) the testing resources are not too limited (implying that without re-testing, patients from groups with low $p_i$ values would have to be admitted). For example, for the case study presented in Section 4 below, using the best predictive model applied to decile-based groups, the conditions in Theorem 5 hold for $j(T)=5$, implying that patients in the top decile testing Negative should be re-tested before any patients from decile 5 or higher are admitted (in fact, patients from decile 1 testing negative twice should be retested for the third time before admitting patients in decile 5 or higher).

To summarize, the threshold-based admission policy developed in Section 2.1 can be extended to allow for multiple rounds of testing.

Methodology II: implementation predictive modeling pipeline and the interactive website

Test admission approaches described in the preceding sections require a sufficiently accurate predictive model (i.e., with ROC curve falling above the 45° line, as required by Theorem 2). In the current section we describe the data requirements and the automated predictive modeling pipeline we use to obtain the best-fitting predictive model for a given dataset. The effectiveness of the proposed approach is illustrated on the Case Study in Section 4.

Our approach consists of three main steps: (1) model development, (2) scoring mechanism and clustering, (3) deployment via an interactive website. Each step is described briefly below; the overall modular structured is presented on Fig. 2.

1. Model development. We assume that a dataset, recording pre-test patient status (symptoms and other patient characteristics) and the test result (positive or negative), is available for a large number of past tests. The goal is to develop an accurate model relating patient status, as well as other available data such as current disease prevalence, to predicted test result. We also assume that each data line belongs to a pre-defined group (in the Case Study, the two groups are “Symptomatic” and “Asymptomatic” patients, but the definition is general), giving us an option to develop one joint model for all groups, or to use group-specific models. We also try a variety of different modeling forms (algorithms); these are listed in the description of Module 2 below. In the end, the best modelling form-group combination is selected.

2. Scoring mechanism and Clustering. The objective of this step is to develop a mechanism capable of making “online” (i.e., real-time) admission decision for a presenting patient. Recall from Section 2.1 that this requires two key components: the predictive model percentile score for a given patient, and the score threshold. While the latter is a simple ratio available test resources and the size of the presenting population, the former is more involved. First, we need the predictive model score – this comes directly from the best model selected in the previous step. Next, we need to convert this score to the model percentile, which should reflect the patient population presenting for testing during the current time period, rather than the total data universe on which the model is built. This is particularly important as disease prevalence may change substantially over time: in our case study, using % of positive tests over the previous 7 days as a proxy for prevalence, we have observed changes of nearly 10 × between different time periods. As a result, the same predicted score may be in a very high percentile (and thus well-above the score threshold) during the low prevalence period, while being in a low percentile (and below the threshold) during the high prevalence period, leading to different admission decisions. Thus, when evaluating a score for a given patient, we also need to make a “prediction” about the disease prevalence in the population of patients presenting during the current time period, in order to estimate the percentile this score corresponds to; since admission decisions must be made in real time, this must be done prior to observing this population. To this end, we cluster all observations, placing particular emphasis on various dynamic measures of disease prevalence, compute model score percentiles within each cluster, and then evaluate which cluster a given patient belongs to, as well as what percentile their score corresponds to within this cluster. Thus, determining the “correct” number of clusters, identifying the clusters, and creating cluster classification rule for a new observation are all part of the scoring mechanism.

3. Model Deployment via the Interactive Website. To facilitate user-friendly model deployment, we developed an interactive website allowing the user to input information about the available test resources, expected size of the presenting population, and characteristics of a given patient. The website uses mechanism from the previous part to compute predicted model score, the cluster-specific percentile, the test threshold and the “policy advice” on whether this particular patient should be tested or not.

Fig. 2.

Modeling Pipeline

Our pipeline is implemented in Python using scikit-learn library [22] for various machine learning algorithms. Each module on Fig. 2 is briefly described below.

Module M1: data processing and partition

The input data set is assumed to be in the form of a table with each row corresponding to an executed test with a known outcome. Since some patients may have been tested more than once over time, there may be several entries in the data corresponding to the same patient.

Data description and directions for data processing are contained in the accompanying “data_info” text file (the metadata). It identifies a binary target column (test results), and the set of columns to be used as features (predictors), divided into two subsets: categorical and numerical (continuous). It also specifies a column identifying groups (e.g. “Symptomatics”), as well as columns to be used for cluster analysis (the latter should include measures reflecting disease prevalence, such as test positivity rates for each group). Finally, it identifies a “data partition” column described below. All other columns that may be present in the data file are ignored, as are all rows for which the target variable value is missing (missing values for the predictors are allowed).

Once Module 1 ingests the data, two main functions are performed: (a) informative missing values encoding, (b) data partition.

“Informative missing” encoding is applied to any predictor column with missing values in some rows. Such column (assume it is named “X”) is converted to two columns: “X_missing” – a binary indicator of whether the value is present (“1”) or missing (“0”), and “X_Mean”, where the value is set to the original value for non-missing rows and to the mean(X) for the missing rows. The two new columns are used as predictors in place of the original column.

Next, the “partition” column is used to split the data into “calibration”, “validation” and “test” subsets (we assume that the specified column contains the corresponding values). If the partition column is missing, the rows are split randomly in 50%, 25%, 25% proportion (the common rule of thumb in the machine learning community). The calibration data is used to tune model parameters (e.g., tree splits or regression coefficients), the validation data is used for tuning meta-parameters (number of trees in a random forest, learning rate, etc.) and the test data is used as a hold-out sample for model evaluation and best model selection.

We note that our pre-specified partition column allows for date-based partitions commonly used in time-series analysis; the latter would be more appropriate if time-series models (e.g., ARIMA or RNN) were added to our model forms; this is not currently the case.

We also note that our data partition is based on “hold-out sample” (out test partition) approach. An alternative is “K-fold cross-validation” where the partition is repeated K > 1 times, and models are assessed based on average performance on the test partition. The advantage of this approach is that all rows of data (rather than only the rows initially allocated to the “calibration” part) are used for model estimation. While this approach could, of course, be used in our implementation, we regard the potential advantage as not crucial when the available data is large (which is our implicit assumptions, as large-scale testing should lead to rapid data accumulation). Moreover, as discussed in Rao et al. [23], cross-validation can lead to significant overestimate of model accuracy.

Module M2: Model Calibration and Meta-Parameter tuning

Module 2 is the “heart” of our pipeline, where different machine learning algorithms (“modeling forms”) are applied to the data. The following supervised learning algorithms were used in our analysis:

1. Nominal Logistic Regression (denoted as LR) with L1 and L2 regularization terms

2. Decision Tree (denoted as DT)

3. Gradient Boosted Decision Trees (denoted as Boosted Tree, or BT)

4. Bootstrap Random Forests (denoted as Random Forest or RF)

5. Artificial Neural Network (with Tanh activation function, denoted as ANN)

Please refer to software descriptions [22] for the details of the implementations of the above algorithms. Each algorithm contains a number of local and meta-parameters, the latter are tuned using the validation data partition via grid search.

As described earlier, we consider two options for each learning algorithm: developing separate model for each group vs a single model covering all groups. Thus, two versions of predicted model scores are constructed for each learning algorithm: group-specific and joint versions.

The output of Module 2 is a scoring rule for each model, i.e., a function that applies the best values of local and meta parameters to produce the predicted probability of a positive test for each observation. Thus, the current implementation results in 10 scoring rules (group-specific and joint). Of course, other machine learning algorithms can easily be added to our list (we note that we also tested Vector Support Machine algorithm, but it performed significantly worse than others on our case study while incurring long running times, thus it was dropped).

Module M3: Out-Of- Sample Evaluation and Model Selection

This module applies all models (i.e., scoring functions) produced by Module M2 to the test subset (the hold-out sample). Since, as explained earlier, the main “link” between our admission policies and the predictive model is the ROC curve, we used the AUC (Area under the ROC Curve) as our primary measure of accuracy; the higher AUC value indicates that a higher sensitivity can be achieved for a given specificity (and vice versa). We also note that AUC can be intuitively interpreted as the concordance rate: i.e., the percentage of all pairs consisting of one row (patient) with a “positive” test outcome and one row with a negative test outcome that are correctly ranked by the algorithm in terms of the predicted probability of the positive test.

The module produces a report of how each model performs on each of the data partitions; an example of this report can be found on Table 3. The model with the best AUC performance on the test partition is selected, and its scoring rule (in the form of Python code) is uploaded to the interactive website module M5. In addition, the predicted scores for the best model for all observations in the data file are passed to the clustering module M4.

Table 3.

Summary statistics for the symptomatic cases sample

		Total Sample	Training	Validation	Test
Total No		108,101 (100)	54,051 (50)	27,025 (25)	27,025 (25)
Gender
	Female	52,180 (48)	25,970 (48)	13,011 (48)	13,199 (49)
	Male	53,902 (50)	27,061 (50)	13,519 (50)	13,322 (49)
	Missing	2,019 (2)	1,020 (2)	495 (2)	504 (2)
Reason for Test
	Contact	36,971 (34)	18,509 (34)	9,212 (34)	9,250 (34)
	Travel	21,277 (20)	10,666 (20)	5,353 (20)	5,258 (20)
	Other	49,853 (46)	24,876 (46)	12,460 (46)	12,517 (46)
Age
	Sixty and under	72,350 (67)	36,219 (67)	18,133 (67)	17,998 (67)
	Over sixty	10,165 ( 9)	5,047 ( 9)	2,493 ( 9)	2,625 (10)
	Missing	25,586 (24)	12,785 (24)	6,399 (24)	6,402 (24)
Symptoms
	Cough
	0	39,706 (37)	19,897 (37)	9,834 (36)	9,975 (37)
	1	68,395 (63)	34,154 (63)	17,191 (64)	17,050 (63)
	Fever
	0	56,363 (52)	28,285 (48)	14,109 (52)	13,969 (52)
	1	51,738 (48)	25,766 (24)	12,916 (48)	13,056 (48)
	Sore Throat
	0	93,795 (87)	46,888 (87)	23,450 (87)	23,457 (87)
	1	14,306 (13)	7163 (13)	3,575 (13)	3,568 (13)
	Shortness of Breath
	0	101,933 (94)	50,977 (94)	25,489 (94)	25,467 (94)
	1	6,168 (6)	3,074 (6)	1,536 (6)	1,558 (6)
	Headache
	0	81,264 (75)	40,510 (75)	20,335 (75)	20,419 (75)
	1	26,837 (25)	13,541 (25)	6,690 (25)	6,606 (25)
Test Results
	Negative	56,357 (52)	28,110 (52)	14,186 (52)	14,061 (52)
	Positive	51,744 (48)	25,941 (48)	12,839 (48)	12,964 (48)

Module M4: Clustering and Within-Cluster Percentile Computation

As discussed earlier, this module groups all observations into clusters consisting of observations coming from time periods with roughly the same disease prevalence (we note that there may be multiple measures of the latter that can all be used for clustering). Once the clusters are constructed, a within-cluster score percentile is computed for the best model passed from module M3.

The variables used for cluster analysis are specified in the data_info file (see Module M1). We use K-means algorithm to identify clusters, varying the number of clusters K from 2 to the user-defined upper bound (set to 20 by default). For each cluster solution we compute the silhouette score (see documentation [22] for details); the number of clusters K* corresponding to the highest silhouette score, and the corresponding cluster assignments are retained.

After computing within-cluster score percentiles using model scores passed from module M3, we upload these percentiles, together with the K*-cluster assignment rule to the interactive website (module M5).

Module M5: The Interactive Website

An important part of making sure the proposed methodology can be implemented is allowing a non-technical user (e.g., a doctor or a public health administrator) to use it in an interactive and intuitive manner. The screenshots of the website inputs and outputs are provided on Fig. 3; this implementation is adapted to the case study described in Section 4; the website can be accessed by following the link in [24].

Fig. 3.

Screenshots of Inputs and Outputs from https://cvdpredictor-app.herokuapp.com/

The user is first asked to provide input values for the sensitivity and specificity of the test, as well as the available test capacity T for the current period, and the estimated number of patients N presenting for testing. This part of the input is invariant for all patients during the current period.

It is important to note that the estimated number of presenting patients N is an important input, considered exogenous by the current model. It may be useful to construct a separate predictive model projecting N for a given day.

The next panel applies to an individual patient. The user is asked to provide values for all predictors in the “best” model identified in Module M3. This list should include only information available at the time the patient is being considered for testing: i.e., symptoms, as well as basic demographic information (other variables, related to disease prevalence in recent periods are invariant for all patients in the current period). The list of the predictors and their possible values are based on the scoring function uploaded by Module 3; this list identifies whether a given predictor is continuous or binary and whether missing values are allowed or not.

Once the inputs are submitted, the scoring function computes (1) the predicted probability of a positive test based on the “best” model, (2) the cluster the current patient belongs to, (3) the within-cluster model percentile, and (4) the testing threshold T/N. The website then displays the “policy advice” panel, summarizing the information above and containing a recommendation of whether the presenting patient should be admitted for testing.

Remark 3

If the data for all presenting patients is available prior to the admission decisions, “online processing” is not necessary. We can compute score percentiles directly based on the presenting patient population, and then use the percentile for each patient for the admission decision; no clustering is needed in this case. While this would be a more accurate computation, the need to have data for all patients ahead of time will likely make the application less practical.

Case study: IMOH data

In order to demonstrate the methodology described in the previous section we applied it to a large dataset obtained from Israel Ministry of Health (IMOH) [25]. We note that a crucial assumption underlying our approach is that values of all predictors used in the model predicting test outcome for an individual patient must be observed prior to the test admission decision. For the IMOH data set used to construct the model this implies that all predictor values must be observed and recorded prior to the test result being known. There are, in fact, serious questions whether the data set satisfies this requirement. These questions, as well as the supplementary analysis we conducted to address them, are discussed in Appendix A2. While our supplementary data analysis gives us some reason to believe that the data set was not seriously corrupted by retro-active revision of the values of the predictors, we cannot be sure that this is the case. Thus, the results in this section should be viewed mainly as an illustration of the potential capabilities and application of our proposed techniques.

The dataset covers 1,532,513 COVID-19 initial standard RT-PCR tests that were conducted by the IMOH from March 22, 2020 to September 8, 2020 [25]. The tests were done using both an oral and a nasal swab. The data that includes the test result (positive = 1 or negative = 0), as well as the following information self-reported by the tested individuals: symptoms (cough, fever, sore throat, shortness of breath, headache; value of 1/ 0 indicates the symptom is present/ absent), gender (male/ female), and sixty (1 if the tested individual was over 60 years old, 0 otherwise). In addition, there is a test indication field with three possible values: “contact” (contact with a known infected individual), “travel” (recently returned from a trip abroad), or “other” [26]. We note that the analysis presented below was approved by the IMOH Data Sharing Institutional Review Board.

Table 2 depicts the summary statistics for the dataset. Note that symptomatic individuals (“symptomatics”) account for 7% of the total, asymptomatics make up the remaining 93%. The overall proportion of positive test results was 7%, differing sharply between symptomatics (48% positive) and asymptomatics (3% positive). The missing data in the sample related mostly to the gender or age (i.e., “sixty” variable) of the tested individuals. In particular, gender was missing for just over 6% of observations (6.3% for asymptomatics and 1.9% for symptomatics), and for 32% of “sixties” indicator (33% for asymptomatics and 23.7% for symptomatics). While some of the missing values may be due to the tested individuals being reluctant to report their age and/or gender, the preponderance appears to be due to data collection issues: during certain time periods all values for the respective variables are missing (5/15 – 5/25 for gender, and 3/11- 3/19, 4/15–4/18, and 4/20–6/17 for sixties; while the data description [26] notes omitted values for these variables during certain dates, the ranges it specifies are different from those in the data). Outside of these date ranges, the percent of missing values is quite low. As noted earlier, informative missing encoding was used for these predictors. Summary statistics for symptomatics are presented on Table 3.

Table 2.

Summary Statistics Total Sample

	Total Sample N0. (%)
		Test Results
	Total Sample	Positive	Negative
Total sample	1,532,513 (100)	100,401 ( 7)	1,432,112 (93)
Demographic information
Gender
Female	749,739 (52)	48,428 (6)	701,311 (94)
Male	698,053 (48)	50,994 (7)	639,544 (93)
Total	1,440,277 (100)	99,422 (7)	1,340,855 (93)
Age
Sixty and under	918,152 (87)	80,165 (9)	837,987 (91)
Over sixty	137,722 (13)	10,769 (8)	126,953 (92)
Total	1,055,874 (100)	90,934 (9)	964,940 (91)
Reason for Test
Contact	69,916 (5)	51,833 (74)	18,083 (26)
Travel	26,450 (2)	1,561 (6)	24,889 (94)
Other	1,436,147 (94)	47,007 (3)	1,389,140 (97)
Total	1,532,513 (100)	100,401 (7)	1,432,112 (93)
Incidence of Symptoms
Symptomatic	108,101 (7)	51,744 (48)	56,357 (52)
Asymptomatic	1,424,412 (93)	48,657 (3)	1,375,755 (97)
Total	1,532,513 (100)	100,401 (7)	1,432,112 (93)
Observations with missing data on
Gender	92,236
Age group	496,659

According to the IMOH [26], symptoms are based on self-reporting by the patients (please see Appendix A2 for further discussion). While individuals presenting with at least one symptom were always tested, the criteria for testing asymptomatic individuals evolved over time. In the early stages of the testing policy, the focus was on those returning from a trip to certain countries or having a confirmed contact with an infected person. Starting from June 2020, as the testing capacity grew, the IMOH opened testing to all comers, encouraging the population to be tested unrelated to symptoms or to risk.

We note that the number of predictors in the data is quite small. The problem is particularly acute for the asymptomatics (comprising 93% of our sample), where the only information available before the test was the “sixties” indicator, gender, and reason for testing (with the “other” category selected 97% of the time); as noted earlier gender and “sixties” information is entirely missing for certain date ranges. For symptomatics at least one of the five symptoms predictors (all binary) was also present (note that [26] states that there were more symptoms, but only the five most prevalent were made available in the data). Several derived predictors were added to the data set, including:

• Number of symptoms: sum of the five symptoms variables for each patient

• “Prevalence” indicators computed as 7-day moving averages for the 7 days preceding the test day for a given patient. These included: overall percent of positive test results, percent of positive test results for symptomatics and for asymptomatics, the number of tests by symptomatic status.

While the overall number of predictors available for each patient group, particularly asymptomatics, is limited, it turns out to be sufficient to achieve reasonably high predictive accuracy.

The modeling pipeline described in Section 3 above was applied to the dataset. The data was randomly split into calibration/ validation/ test subsets according to 50%/25%/25% ratio. The two “grouping” indicators were aymptomatics and symptomatics. For each group, two sets of models were developed: the ones trained on the full data set (designated “trained all”), and the ones trained on the data set comprised only of a given group. In both cases, the models were evaluated using the test (i.e., holdout) sample filtered to include only patients in a given group. The aggregate results, sorted by the AUC score, can be found on Table 4. In addition to AUC and other standard measures of accuracy (precision, accuracy, etc.) we also compute the Standard Error of AUC using the approximation provided in Eq. (2) of [27].

Table 4.

Aggregate Measures of Fit on the Test (Holdout) Sample

rank	name	AUC	SE(AUC)	sensitivity	specificity	accuracy	precision	f1 score	Running Time (Sec)
Asymptomatic Patients
1	trainedallGradientBoostingClassifier	0.86479	0.00211	49.11%	99.64%	97.89%	83.21%	61.77%	4,637.56
2	RandomForestClassifier	0.86433	0.00211	51.28%	99.53%	97.86%	79.53%	62.36%	396.24
3	trainedallRandomForestClassifier	0.86433	0.00211	49.16%	99.62%	97.87%	82.32%	61.56%	694.83
4	DecisionTreeClassifier	0.86408	0.00211	51.39%	99.52%	97.86%	79.50%	62.43%	10.96
5	trainedallDecisionTreeClassifier	0.86401	0.00211	47.20%	99.69%	97.87%	84.37%	60.53%	31.27
6	GradientBoostingClassifier	0.86217	0.00212	49.07%	99.64%	97.89%	83.13%	61.71%	1,860.93
7	NN_Classifier	0.86063	0.00213	46.45%	99.72%	97.87%	85.40%	60.17%	239.24
8	trainedallNN_Classifier	0.85869	0.00214	47.43%	99.69%	97.88%	84.49%	60.76%	151.90
9	LogisticRegression	0.85827	0.00214	49.43%	99.61%	97.87%	82.03%	61.69%	18.73
10	trainedallLogisticRegression	0.85231	0.00218	40.80%	99.82%	97.78%	89.19%	55.98%	44.77
Symptomatic Patients
1	RandomForestClassifier	0.91553	0.00182	87.71%	82.46%	84.99%	82.33%	84.94%	25.85
2	NN_Classifier	0.91503	0.00182	87.01%	82.75%	84.80%	82.45%	84.67%	25.17
3	GradientBoostingClassifier	0.91439	0.00183	88.84%	81.21%	84.89%	81.50%	85.01%	251.00
4	DecisionTreeClassifier	0.91401	0.00183	88.68%	80.84%	84.62%	81.17%	84.76%	1.16
5	trainedallGradientBoostingClassifier	0.91245	0.00185	88.33%	81.45%	84.77%	81.61%	84.84%	4,635.85
6	trainedallNN_Classifier	0.91119	0.00186	85.28%	83.80%	84.52%	83.07%	84.16%	151.45
7	LogisticRegression	0.90006	0.00197	83.90%	83.41%	83.65%	82.49%	83.19%	5.69
8	trainedallRandomForestClassifier	0.89990	0.00197	84.71%	80.68%	82.63%	80.34%	82.47%	690.50
9	trainedallDecisionTreeClassifier	0.88696	0.00209	87.99%	77.30%	82.45%	78.32%	82.87%	30.67
10	trainedallLogisticRegression	0.85063	0.00239	68.19%	80.96%	74.80%	76.94%	72.30%	44.30

AUC – The area uuder the ROC curve (concordance rate)

SE(AUC) – Standard Error of AUC (approx.)

For definitions of all other measures please see https://en.wikipedia.org/wiki/Sensitivity_and_specificity

We observe that, not surprisingly in view of much higher disease prevalence and richer predictor set, the models for symptomatics are substantially more accurate (in terms of the AUC score) than their counterparts for asymptomatics. The best model for symptomatics is the Random Forest classifier trained on symptomatic-specific data sample. The best model for asymptomatics is the Gradient Boosted Tree Classifier trained on the full data set. The worst-performing models in both cases are Logistic Regression models trained on the full data set. In view of the Standard Errors, the AUCs for the top 5 ranked models in each group are not significantly different, thus any one of these models could be chosen as the “best model”; our pipeline automatically selects the top-ranked model for upload to the website (i.e., admission decision support).

The corresponding ROC curves are provided on Fig. 4. It is interesting to note that the curves for all models for asymptomatics show extremely steep rise to the level of about 0.55 on the y-axis (i.e., the sensitivity axis) – indicating that all models are very accurate in identifying approximately 55% of all positively-testing asymptomatics. However, all curves flatten out after this level, indicating that the identification of the remaining 45% of positive asymptomatics is much more difficult. The curves for better-performing models for symptomatics show much more even performance throughout the test sample, reflected in higher AUC rates for this group.

It is also interesting to identify which features have the strongest impact on model performance. We use the permutation feature ranking (see documentation in [22] for further description of this methodology) for the top-performing model for each group. The results are displayed on Fig. 5. For both groups, the most important feature is “Test Reason” (namely, Test Reason = contact, i.e., when patient is being tested due to a close contact with an infected person). Average percent of positive tests among symptomatics in the last 7 days (“Ave_Pos_Sympt_7dMA”) is also very important for both groups (interestingly, for asymptomatics this is more predictive than the 7-day positive average for asymptomatics). Not surprisingly, the five symptom features (particularly headache, shortness of breath and sore throat) are all predictive for the symptomatic group. We emphasize the importance of including the prevalence features (with “7dMA” suffix) in both symptomatic and asymptomatic models: these features allow the models to track the disease prevalence rate that shows very large variations over time (from 10% positive rate in some time periods to around 90% in others). We note that the predictive modeling approach in [10], that was also based on a fairy large dataset, did not include these variables, which may have accounted for a somewhat lower model accuracy in their study.

Fig. 5.

Feature Importance Scores for Asymptomatics (“Symptomatics = 0) and Symptomatics (Symptomatic = 1)

Next, we apply the K-Means clustering method to the 5 running average features (designated with the “7dMA” suffix on Fig. 5) and compute silhouette scores for different cluster solutions; see Fig. 6(a). The highest score is achieved at the 5-cluster solution. The key features of the corresponding clusters are shown on Fig. 6(b). We observe that the key characteristic distinguishing clusters is the 7-day moving average of positive tests among symptomatic patients (“Ave_Pos_Sympt_7dMA”): cluster “0” identifies periods where this level is very high at 85.77% (this covers 34.05% of all data), followed by clusters 3, 4, 1, and 2. Clusters 4 and 1 are quite similar at 37.75% and 35.18% positive rates, so they are distinguished by the positive rates for asymptomatic patients which are almost 3 times higher in cluster 4 than in cluster 1.

Fig. 6.

Cluster Information

As describes in Section 3, the key output of the modeling pipeline that is uploaded to the website app is the file containing model scores and corresponding percentiles within each cluster. While scores for both symptomatic and asymptomatic are included, the former group, due to much higher disease prevalence, tend to have much higher scores. Thus, top deciles of each cluster are overwhelmingly composed of symptomatics: the percent of asymptomatics who fall outside of the top score decile ranges from a high of 98.18% in cluster 2 to a low of 91.5% in cluster 1. Thus, any admission policy described in Section 2 above will tend to prioritize symptomatic patients over asymptomatic ones.

Finally, we demonstrate the application of our optimal admission policies developed in Section 2.1to the IMOH dataset. Recall that the percentage admitted for testing is determined by resource constraint ratio (the ratio of testing resources to expected number of patients on a given day) and all patients with model scores falling above the percentile given by this ratio should be admitted. On Fig. 7 we plot the expected efficiency gain compared to the random admission policy, given by Eq. (4) above, vs % of patients admitted for testing for each of the five clusters. As before, we assume 80% sensitivity and 99% specificity for the RT-PCR test. We observe that the lower the disease prevalence (given by Eq. (2) above and indicated on the Figure next to each cluster), the higher the potential efficiency gain achievable by the optimal admission policy. However, the main discrepancies appear for very low admission levels. Once the admission levels exceed approximately 10%, the efficiency gains are remarkably consistent across all clusters – indicating that the primary driver is predictive model accuracy, not the disease prevalence. The expected efficiency gain is 8 × at 10% and 2 × at 50% for all clusters. This indicates that when model accuracy is high, very significant efficiency gains can be achieved, even when admission rates are quite high.

Fig. 7.

Expected Efficiency Gain vs % Admitted for Testing. Disease prevalence in each cluster indicated in parenthesis

Discussion and conclusions

In this paper we demonstrate how predictive analytics tools can be leveraged to optimize efficiency of the available testing resources. As the on-going COVID-19 pandemic has shown, testing resources are often scarce due to availability of test kits, lab time, and trained nurses. Allocating these resource so as to maximize the number of True Positive cases detected, which is the efficiency objective used in this paper, is important in treating and isolating infected individuals as soon as possible.

We show that a simple admission policy based on the percentile of the predictive model is optimal with respect to efficiency, and derive conditions under which it outperforms the “baseline” policy where patients are admitted for testing randomly. We extend this policy to several settings, including re-testing “high-risk” patients, as well as combining admission control with pooled testing.

Our admission policies rely on having a reasonably accurate prediction of the outcome of the test for a given patient based on current disease prevalence, patient characteristics and symptoms. To obtain these scores we develop a general automated machine learning pipeline and implement it in Python. The decision rule based on the optimal policy is implemented via an interactive website.

We illustrate our approach by applying it to the large data set from Israel’s Ministry Of Health. Even though the data is quite limited in terms of patient symptoms and related characteristics recorded at the time of testing, we are able to construct quite accurate predictive models, which can be used to drive very significant testing efficiency gains vs the random-access admission policy.

It should be noted that the data set that we use in has several limitations. The main ones are discussed in Appendix A2. We also note that, while being one of the largest COVID-19 datasets we are aware of, it is quite shallow, containing a very limited number and simplistic description of symptoms and patient characteristics. Many well-documented symptoms, such as sudden and complete loss of the olfactory function without any nasal obstruction [28], are missing from our data. Our data is particularly limiting for asymptomatic patients who constitute 93% of the original data and account for 48% of positive RT-PCR tests. A richer set of patient characteristics would allow for significant improvement in model accuracy and result in further efficiency gains.

Another important limitation is that we assume that the “true” (i.e., in vivo) medical test sensitivity and specificity are known. While this may be a reasonable assumption for well-established tests, it may not be the case for new testing tools early in the pandemic. Methodology for inferring disease prevalence when test specificity and sensitivity are unknown has been proposed in Cai et al. [29]. It would be interesting to combine this methodology with our optimal admission controls.

Finally, we underscore that our methodology is not limited to RT-PCR testing for SARS-COV2, but is rather quite general and applicable to any testing applications where the primary goal is to identify True Positive cases.

Appendices

Appendix A1: Proofs

Proof of Theorem 1

From (3) we see that $EG\ge 1$ is equivalent to.

$$q\left(1,-,\beta \right)+\left(1,-,q\right)\alpha +p_{\mathit{se}}\left(1,-,q,\left(1,-,\beta \right),+,\left(1,-,q\right),\alpha \right)+\left(q,\beta ,+,\left(1,-,q\right),\left(1,-,\alpha \right)\right)p_{\mathit{sp}}\ge 1$$

Letting $\gamma =\left(q,+,\alpha \right)-q\left(\alpha ,+,\beta \right),$ this is equivalent

$$\gamma +\left(1,-,\gamma \right)\left(p_{\mathit{se}},+,p_{\mathit{sp}}\right)\ge 1,$$

which holds whenever $p_{\mathit{se}}+p_{\mathit{sp}}\ge 1.$ □

Proof of Theorem 2

First consider two patients, 1 and 2, and let $X_i,i=1,2$ represent their test outcomes and $I_{i,i=1,2}$ their actual infection state where “1” represent positive or infected. Suppose $X_1=1$ and $X_2=0$ (i.e., patient 1 tests positive and patient 2 negative). Then that patient 1 is more likely to be infected than patient 2 is.

$$P\left(I_1,=1|,X_1,=,1\right)\ge P\left(I_2,=1|,X_2,=,0\right)<=>1-\alpha \ge \beta <=>\alpha +\beta \le 1.$$

Thus, when the first condition of the Theorem holds, patients who test positive are more likely to lead to TP outcomes, increasing the test efficiency. Now consider a threshold-based decision rule with threshold c that admits patient 1 and rejects patient 2. Letting $p_1,p_2$ be the predicted values of a positive test outcome for our predictive model, this implies $p_1>c\ge p_2.$ Let $p_{\mathit{se}}\left(c\right),p_{\mathit{sp}}(c)$ be the corresponding sensitivity and specificity and note that, by definition $p_{\mathit{se}}(c)=P(X_1=1|p_1>c)$ and ${1-p}_{\mathit{sp}}(c)=P(X_2=1|p_2\le c)$. Thus patient 1 is more likely to test positive compared to patient 2 iff $p_{\mathit{se}}\left(c\right)+p_{\mathit{sp}}\left(c\right)\ge 1.$ This holds for the proposed admission rule. Now consider any alternative admission rule that admits the same number of T patients for testing. Then there exists a pair of patients where 1 is admitted and the proposed rule, 2 is not admitted, and the opposite holds for the alternative rule. As shown above, admitting 1 ahead of 2 increases the expected number of TP outcomes and thus test efficiency. Therefore, the alternative rule cannot be optimal. □

Proof of Lemma 2

Consider a group of containing $0<k\le S$ infected patients. The probability of this group testing positive is $\left(1,-,\beta \right)$, as is the probability of each infected patient testing positive. The probability of observing such as group is given by the binomial distribution $Bin(k;S,q_p)$. Taking expectation, we get expected number of TPs as $TP\left(S\right)=S{q}_P{(1-\beta )}^2$. Since there are $\frac{N_P}{S}$ groups, the second expression in the Lemma follows. □

Algorthm 1: Optimal Testing Strategy with Multiple Tests.

Assume conditions of Theorem 5 hold. After dividing patients into m groups as described in Section 2.3, we let $k_i$ be the number of times members of group $i$ have been tested so far; initially $k_i=0$ for all $i$. At each iteration we apply Eq. (16) to each group and identify the one with the highest ${\mathit{TP}}_i^{(k_i)}.$ Patients in this group are all admitted for testing in order of their predicted scores. If the remaining testing capacity is exhausted before all patients in this group can be served, the process stops. Otherwise, patients who have tested positive are removed from group $i$, we update $k_i=k_i+1,$ and the process is repeated until all T available tests have been allocated. Note that the expected number of patients remaining in group $i$ after the current round of testing can be computed to be $n_i{[q_i\beta }^{k_i-1}+(1-q_i){\left(1,-,\alpha \right)}^{k_i-1}].$ Since at each step we allocate at least some patients for testing, the algorithm has linear running time in T.

Appendix A2: Supplemental Analysis of Potential Issues in IMOH Data

Our case study in Section 4 is based on IMOH data set. As noted there, a crucial assumption underlying our analysis is that the symptoms of each patient be observed before the test results are known. However, it is not clear whether this requirement was always met for this data. In fact, the website information [26] states:

“There is an inherent reporting bias as the data is collected differently from positive and negative results:

• For positive results, a comprehensive epidemiological investigation was made, with specific questions regarding symptoms

• For negative results, the data were not collected consistently and not through direct questioning.”

However, it is not clear whether the results of the “comprehensive epidemiological investigation” in case of the positive tests were retroactively reflected in the same database (note that it takes 1–3 days to obtain test results, thus updates must be retroactive). The supplemental analysis described below does not appear to support these retroactive updates:

1. The average number of symptoms (for symptomatic patients) is 1.39 (st.dev 0.61) for patients testing negative and 1.71 (st.dev. 0.86) for patients testing positive – a relatively small difference, which is entirely consistent with the latter group feeling less well upon arrival. One would expect a post-factum “comprehensive investigation” to uncover significantly more symptoms for the latter group.

2. The data was downloaded on two separate occasions. The first download (“old data”), conducted when the initial work on this paper started, covered the dates through April 9, 2020. The second download (“new data”) covered the dates through September 8, 2020. Thus, any retroactive updating should be reflected in the test records during the data overlap period of April 8–9; we would expect to see more symptoms uncovered for positive-testing symptomatic patients in the later download (after their test results became known and follow-up could be done). For patient confidentiality reasons, no unique patient identifier is included in the data, so direct match of patient records is not possible, thus we are limited to aggregate comparisons. For the two-day period in question we see 347 positive and 2795 negative test results in the “old data” vs 394 positive and 3528 negative test results in the “new data” (all reported values are for symptomatic patients only). Thus, retroactive updates were more prevalent for negatively-testing patients. While the number of recorded symptoms went up slightly for both groups (1.792 vs 1.703 for positively-testing patients, and 1.296 vs 1.285 for negatively-testing ones), neither difference is statistically significant (t-statistics of 1.43 and 0.91, respectively). The percent of symptomatic patients also stayed very consistent between the old and new data. The results remain the same if we expand the overlap period to include April 6, 2020 – April 9, 2020.

3. While the disease prevalence, as reflected by the percentage of positive test results, varied greatly over time (see Fig. 6(b)), the correlations between the average number of symptoms and the average test positivity rate or the number of tests conducted are small and not statistically significant for either positive or negative testing groups. One would expect the resources dedicated to epidemiological follow-up for positive test scores to vary with disease prevalence (i.e., more symptoms may be uncovered when more resources are available), but we see no evidence of that.

4. Our efforts to reach out to IMOH personnel directly to inquire whether retroactive updates were in fact entered into the same database did not result in clear-cut answers; the policies appear to have been inconsistent.

In summary, our analysis suggests that post-test updates of patient records were either not entered into the database, or were only entered on relatively rare occasion, indicating that IMOH data may be suitable for our purposes. However, it is impossible to be sure, and thus the case study presented in Section 4 should be treated as illustrative only – it would be much better to obtain a new data with the specific purpose of supporting admission-related decisions.

Authors' contributions

All authors worked closely to design and write the manuscript. RE led the data management and analysis. DK supervised the design of the Python code and the website, and the theoretical derivations of the decision rules.

Funding

NSERC Discovery Grant (Dmitry Krass).

Data availability

The Israeli Ministry of Health data on the characteristics of tested individuals for SARS-COV-2 is publicly available.

Code availability

The source code for all machine learning algorithms can be found here: https://github.com/nitimkc/covid_tests/blob/master/classification/build.py

Declarations

Conflicts of interest/Competing interest

We declare no competing interests.