Multi-event dynamic capture-recapture model for big data: Estimating undetected COVID-19 cases in British Columbia, Canada

Abstract

The accurate quantification of the impact of COVID-19 pandemic on both public health and the economy is essential for informed policy-making. However, the true scope of the pandemic remains challenging to ascertain due to undetected cases, particularly when relying on reported cases, which rely heavily on test availability and strategies. To accurately quantify COVID-19 cases in British Columbia (BC), we develop a Susceptible-Infectious-Recovered multi-event capture-recapture (SIRMECR) model to capture the dynamics of COVID-19. Specifically, we present a time-varying Markov model to estimate the number of undetected COVID-19 cases in five Health Authority Regions in BC, Canada, during the year 2020. We utilize individual-level information available from Population Data BC database to estimate the case detection probability, infection probability, survival probability, and recovery probability by incorporating testing volumes as covariates that improve the estimate of our parameters. We develop a Markov chain Monte Carlo (MCMC) algorithm to estimate SIRMECR model parameters. However, analyzing this big COVID-19 data set prompts a discussion on the computational challenges encountered. Therefore, we developed divide-and-conquer strategies to address the challenges. Our application provides an estimate of the total COVID-19 burden in year 2020 and found the percentage of undetected varying from 77.4 % to 84.0 %. More specifically, we validate our results through a simulation study and N-mixture model for Northern Health Authority Region of BC.

1. Introduction

The emergence of the Coronavirus pandemic in December 2019 marked the onset of a global health crisis, swiftly permeating every corner of the world and transcending geographical boundaries. The profound implications of this pandemic reached deep into various aspects of human lives and prompted shifts in societal and cultural norms (Béland et al., 2021; Cypress, 2022). Furthermore, its far-reaching impact erode the global economy, giving rise to transformative changes that bear undeniable significance. By March 2022, the pandemic had exacted a devastating toll, with the cumulative global death count surpassing 5 million, from more than 400 million documented cases. Moreover, Canada reported more than 3 million confirmed cases, and more than 30 thousand confirmed deaths (Dong et al., 2020). Nonetheless, the count of confirmed cases merely scratches the surface, unable to capture the full extent of the pandemic's spread. Underlying reasons for this under-reporting include the prevalence of asymptomatic, mildly symptomatic, and pre-symptomatic cases, limited testing for cases with lower severity, and periods marked by reduced testing capacities.

It is crucial to have an accurate estimate of the total infection count to help planners make decisions about economic openness. Knowing the true extent of the infection allows policymakers to make informed decisions on allocating resources and implementing measures to control the spread of the disease. Similarly, public health can make strategic decisions on how to operate amidst ongoing pandemic. However, a considerable challenge faced by policymakers, public health, and the general public in comprehending the trajectory of COVID-19's spread is the fact that many cases go undetected because of testing shortages or infected individuals not seeking a test, for example, asymptomatic individuals may not even consider the need for a test (Wu et al., 2020). Unfortunately, the number of undetected cases, while challenging to enumerate, is much larger than the confirmed cases due to the vast amount of asymptomatic patients, which undermines estimations of the total number of cases. Friedman et al. (2021) showed that most predictive models indicate the actual total to be at least 2 − 3 times greater than the confirmed cases. In particular, Skowronski et al. (2020) found that in May 2020 the number of undetected cases in BC was between 2.25 and 20.5 times greater (a 95 % confidence interval) than the reported number of cases. Low ascertainment rates could potentially have been improved using more effective testing strategies (Lawless & Yan, 2021).

To properly understand the impact of the pandemic, it is necessary to estimate the actual total number of infections. Individual-level data gives more information on disease progression, and thus makes model parameters such as the detection rate, hospitalization rate and recovery rate more identifiable. Methodologies developed after the outbreak of the pandemic has produced reliable estimates. These methods include meta-analyses of asymptomatic prevalence (Alene et al., 2021; He et al., 2021), extensions to susceptible-infectious-recovered (SIR) type modelling (Huo et al., 2021; Li et al., 2021; Subramanian et al., 2021). Alternative models include case fatality rate (CFR) models (Dougherty et al., 2021) and discrete count models for estimating under-reporting cases (Parker et al., 2021). Many research that include intervention strategy analysis in Ontario (Tuite et al., 2020), mental health and well-being of Canadians during the pandemic (Appleby et al., 2022), forecast models for transmission and incidence (Mullah & Yan, 2022), have focused on understanding the course of the pandemic within Canada.

Multi-event capture-recapture (MECR) model is a model that can be used to estimate infection probability and to determine the impact of infection on survival (Gimenez et al., 2012; Pradel, 2005). The model has been used extensively to study animal disease population dynamics (Benhaiem et al., 2018; Marescot et al., 2018). MECR allows the estimation of assignment uncertainty from the testing capacity (Benhaiem et al., 2018), which increases precision when individuals known may be either undetected (never tested) or detected (tested) but their health status may be uncertain e.g., when clinical signs are indecisive or when diagnostic tests include false positive or negative results (Buzdugan et al., 2017). Often, individuals with such “misclassified” or “imperfect” information have been removed from data sets and statistical analysis. Not only does this reduce sample size and hence statistical power, it can also lead to loss of information, and increase bias in the estimation of state-specific demographic and epidemiological parameters (Conn & Cooch, 2009; Desprez et al., 2013). Substantial improvements in addressing these issues have been made by the development and application of multi-state hidden Markov models, also called multi-state capture-recapture (CR) models (Lebreton et al., 2009). Similarly, multi-event CR (MECR) approaches have been used to improve the estimates of demographic and epidemiological parameters in population ecology and disease ecology (Conn & Cooch, 2009; Culina et al., 2013; Desprez et al., 2013).

Maximum likelihood estimation (MLE) is commonly used for model estimation in MECR. This involves the summation over several hidden states and can be computationally expensive. Alternatively, MCMC algorithm, a widely used estimation technique within Bayesian modelling used to sample from the often intractable posterior, instead samples the states from a distribution and ignores the expensive summation. However, one of the potential drawbacks is their lack of scalability. The computational cost of MCMC is typically linear in the amount of data and can be infeasible for large data sets, both in computational cost and storage. In setting involving large data sets, or where the model is computationally expensive, evaluating the posterior at every iteration of the MCMC may be impractical. Some strategies to overcome this include data subsampling (Nemeth & Fearnhead, 2021; Welling & Teh, 2011), where only a subset of the data is used at each MCMC iteration. On the other hand, in situations where it is possible to easily parallelize computation, statistical modelling becomes easily scalable to large data sets.

Our contributions are to (1) propose a novel SIRMECR model to estimate the number of undetected COVID-19 cases in BC. (2) Modify our model dynamics to allow for assignment uncertainty or misclassification in COVID-19 test results. We apply our new model to data set from five Health Authority regions of BC using an individual-level information. (3) Improve our model by incorporating parameter covariates using testing volumes to better estimate disease dynamics such as infection probability, recovery and death probabilities, COVID-19 assignment probability of the testing protocols, and abundance of undetected cases. We incorporate two events/observations matrices such as detection event and assignment event. Often, assignment uncertainty occur when biological samples can only be collected opportunistically or by non-invasive techniques (Marescot et al., 2018) or when clinical results are indeterminate in the case of COVID-19 results. (4) While the enumeration of undetected COVID-19 cases remains our overarching goal, we analyze big COVID-19 datasets comprising individual-level information that comes with computational challenges. For example, COVID-19 datasets in BC has over n = 300, 000 tested individuals. Prior studies on COVID-19 have utilized aggregated online data sets which lack individual information. Moreover, the limitation of capture-recapture, in general, is the lack of focus on how to deal with big capture-recapture data such as COVID-19 dataset. Considering a Fraser Health Authority Region of BC population of size N ∼ 2, 700, 000, estimating undetected cases from n₀ = 2, 200, 000 unobserved individuals becomes a daunting task. To address these challenges, we adopt divide-and-conquer computational approaches (Nemeth & Fearnhead, 2021; Nemeth & Sherlock, 2018; Quiroz et al., 2019). We suggest that a careful application of this method is imperative to avoid double counting which can introduce bias to estimations. Hence, we discuss the computational approach used to achieve our primary goals. The goals of this paper are to (i) enumerate and quantify the abundance of undetected COVID-19 cases, (ii) estimate infection probability, recovery probability, death probability, detection (testing rate) probability, and assignment probability in different states; (iii) discuss the computational approach employed to handle big COVID-19 data; and (iv) make an informed decision for future pandemic preparedness.

The plan of the paper is as follows. Section 2 discusses the background problem formulation of our model and gives a detailed discussion of our model development. Further, we discuss the approach employed to solve the computational challenges. Section 3 discusses the applications of our proposed model to COVID-19 data set. The applications are in two parts: one, we assume the parameters are not time-dependent and two, we assume the detection probabilities are time-dependent with covariates. Further, we discuss the results and the major outcomes of our paper. We demonstrate the accuracy of our divide-and-conquer approach through a simulation study in Section 4 and the concluding remarks are given in Section 5.

2. Model

2.1. Model development

The classical SIRMECR model developed by Benhaiem et al. (2018) allows estimation of parameters depicting Susceptible (S)-Infected (I)-Recovered (R), while accounting for imperfect detection and infection state uncertainty (Gimenez et al., 2012). The model had four infection states (S, I, R, and D), and five underlying events - not detected, detected and diagnosed as S, detected and diagnosed as I, detected and diagnosed as R, detected and undiagnosed (i.e., unknown: U). The model included transition matrix with state-dependent parameters: survival probability ϕ, infection probability α, recovery probability γ. Here, α is the transition probability from S at time t to I at time t + 1, and γ is the transition probability from I at time t to R at time t + 1, detection probability p, and assignment probability of infection states δ.

The construction of SIRMECR models requires the formulation of state process and observation process. The state process accounts for transitions of individuals between the three infection states conditional on the survival in their given state, and the observation process accounts for imperfect detection and state uncertainty. The form of the SIRMECR affords many extensions to state and/or observation processes, making it flexible for studying different sorts of populations. To develop a disease analytic version of the SIRMECR model, several modifications are necessary. Equation (1) (State process) considers the transitions of individuals to three infection states; Susceptible (S), Infected (I), and Recovered (R). The state parameter in Equation (1) include infection probability α, recovery probability γ. The infection probability α is the probability of transition from a susceptible state to an infected state and 1 − α is the probability of not yet infected. Equation (1) also accounts for the survival probability of individuals during the pandemic period where ϕ_S, ϕ_I, and ϕ_R are the survival probabilities in the three infection states. The survival probabilities ϕ_S, ϕ_I, and ϕ_R are dynamic and can be subject to change over the course of the pandemic. It is straightforward to incorporate the evolving nature of these probabilities to capture the dynamics of survival during different phases of the pandemic in our model. Intuitively, at the beginning of the pandemic, we hypothesize ϕ_S to be very high with a value close to 1, while ϕ_I is expected to be relatively lower at the rise of the pandemic. The infection states were assigned using nose swab PCR test for COVID-19 for the presence or absence of ribonucleic acid or RNA of SARS-CoV-2, the virus that causes COVID-19 diseases. Similar to standard SIRMECR model, we make the assumptions that individuals cannot transition from S at time t to R at time t + 1 transition probability. To ascertain death is due to COVID-19, we make assumption that ϕ_S = 1, and individuals in R state is likely to develop immunity against the virus, therefore, individuals in R at time t cannot transition to D at time t + 1, (i.e., ϕ_R = 1).

We let T denote the sampling occasions: t ∈ {1, 2, …, T}. We make several additions and modifications to SIRMECR models. We specify the state process in Equation (1), with three hidden infection states (S, I, and R) and the dead state (D) treated as a known state caused by COVID-19. We made an addition to the observation process in Equation (2). Our model has six underlying events which include “not detected” (ND), “detected and diagnosed as S” (DS), “detected and diagnosed as I” (DI), “detected and diagnosed as R” (DR), “detected and confirmed as D” (DD), “detected and undiagnosed” (i.e., set as Indeterminate/Unknown: U).

Fig. 1 shows the transition diagram. An individual who is currently infected at time t can remain infected, recovered, or die by time t + 1. Thus, infected individuals I_t at time t is partitioned into three categories relative to t + 1: cases who remain infected I_t+1; cases who will recover R_t+1, and cases who will die D_t+1. The transition is done using categorical distribution, with probability of death due to COVID-19 1 − ϕ_I, probability of recovery γ, and probability of remaining infected 1 − γ. Individuals who die at time t will remain in the dead state at time t + 1. Therefore, the population size is partitioned into S_t, I_t, R_t and D_t as $N_t=N_t^S+N_t^I+N_t^R+N_t^D$. Using this categorical model assumes that we have a state matrix with individual-level infection patterns, where 1, 2, 3, 4 represents S, R, I, D across sampling occasions.

Fig. 1.

The hidden states of the COVID-19 based on the compartment model SIR models with corresponding transition probabilities. The event 1 phases are the testing periods where an individual gets tested, and the event 2 phases are the COVID-19 results which include positive, negative, and indeterminate.

The observation process in Equation (2) accounts for both perfect and imperfect detection and the parameters include detection probabilities (p_S, p_I, p_R) and assignment probabilities (δ_S, δ_I, δ_R, δ_U) in the three states given that an individual is alive. Parameters p_S, p_I, and p_R are the detection probabilities of S, I, and R individuals, respectively and 1 − p_S, 1 − p_I, and 1 − p_R, are their respective complements. The rows of Equation (2) (left) corresponds to the infection states S, I, R, and D and the 5 columns correspond to the not detected (ND), detected in S (DS), detected in I (DI), detected in R (DR), and confirmed dead (DD). Equation (2) (right) accounts for the imperfect state assignment process with parameters δ_S, δ_I, and δ_R as the assignment probability of individuals to S, I, and R states, respectively, and 1 − δ_S, 1 − δ_I, and 1 − δ_R as their respective complements. The columns in Equation (2) (left) are the same as the rows in Equation (2) (right) whose columns correspond to the following events: not detected (ND), detected and assigned as S state (AS), detected and assigned as I state (AI), detected and assigned as R state (AR), “detected but unassigned” to any state (AU), and “detected dead and assigned” as dead (AD). We let dead state in observed matrix to be an absorbed state. This is important because we assume that an individual can die only if they are infected and not through underlying illness. The observed data for this model are the COVID-19 PCR results which are either positive, negative, or indeterminate. The observed processes are conditional on the state process. We also use categorical distribution for the observation process. The plate diagram specifying our model is shown in Fig. 1, illustrating the state-event interactions.

3. Bayesian approach

The Bayesian approach to parameter estimation of our COVID-19 model is different from the standard MLE approach employed for the original SIRMECR. Summations over latent variables are not required for parameter updates. Alternatively, the values of the latent variables are re-sampled from appropriate distributions.

We build a Bayesian approach for our SIRMECR model for COVID-19 depicting Susceptible (S) → Infected (I) → Recovered (R) dynamics to estimate survival, infection and recovery probabilities with infection state uncertainty. We let row vector x_i = (x_i1, …, x_iT) be the i^th sequence of underlying states at occasions t = 1, …, T with each $x_{it}\in \mathcal{X}$, where $\mathcal{X}$ is the finite set of latent states; S, I, R, and D. This means that an individual can only be in one state at time t + 1 given the state at time t, therefore we develop a Markov model of transition probability. We assume all individuals are in the susceptible state at t₀, therefore the initial and transition probabilities are given as follows;

$$\begin{array}{ll}{\mathbf{X}}_{i1}& \sim \text{Categorical}(\eta )\\ {\mathbf{X}}_{it}|{\mathbf{X}}_{it-1}& \sim \text{Categorical}(\mathbf{\Gamma }{\mathbf{x}}_{it-1}),t=2,\dots ,T\end{array}$$ (3)

where η = (1, 0, 0, 0). Similarly, we let row vector y_i = (y_i1, …, y_iT) represent the i^th sequence of observations taken over sampling occasions t = 1, …, T conditional on the state at time t. Each $y_{it}\in \mathcal{Y}$, where $\mathcal{Y}$ is the finite set of observable capture events; detection and assignment. Further, an individual can either be undetected or detected and assigned to any of the states at a given time t, (i.e., ND, AS, AI, AR, AU, and AD). We consider a total of n observed sequences, which comprise the rows of the observed data matrix $\mathbf{y}={({\mathbf{y}}_1^{\prime },\dots ,{\mathbf{y}}_n^{\prime })}^{\prime }$ and unobserved sequences n₀ from the augmented data matrix $\mathbf{y}={({\mathbf{y}}_1^{\prime },\dots ,{\mathbf{y}}_{n_0}^{\prime })}^{\prime }$, where N = n + n₀ is taken to be the total population size. The total population size N is divided into four independent states shown in Table 1.

Table 1.

Definitions for each variable used in construction of the multi-state disease analytics model with time in months.

Statistics
N	number of augmented data: combination of observed and unobserved individuals
T	number of sampling occasions
xit	the state of an individual at time t, i ∈ {1, 2, …, M}, t ∈ {1, 2, …, T}
yit	observed cases in any state at time t, i ∈ {1, 2, …, M}, t ∈ {1, 2, …, T}
Latent States
Sit	cases which is susceptible at time t
Iit	cases which remain Infectious in State A from time t to t + 1
Rit	cases which recover in State R from time t to t + 1
Dit	cases which die in State D from time t to t + 1
Parameters
$p_t^I$	probability of detection at Infectious state I
$p_t^R$	probability of detection at state R
${\varphi }_t^S$	probability of survival in Susceptible state S at time t
${\varphi }_t^I$	probability of survival in Infectious state I at time t
${\varphi }_t^R$	probability of survival in state R at time t
αt	Transmission rate at time t
γt	Recovery rate at time t
${\delta }_t^I$	probability to ascertain the status of an individual detected as positive at time t
${\delta }_t^R$	probability to ascertain the status of an individual detected as negative time t
${\delta }_t^U$	probability to ascertain the status of an individual detected but not diagnosed at time t
ψ	occupancy probability
Derived Variables
$N_t^S$	total susceptible cases at time t
$N_t^I$	total infected cases at time t
$N_t^R$	total Recovered cases at time t
$N_t^D$	total dead cases at time t
N	total population size: $N_t^S+N_t^I+N_t^R+N_t^D$

To model our state-dependent probability, we set up the Bayesian analysis of the superpopulation model (i.e. state-space formulation using data augmentation,Kéry & Schaub, 2011; Royle & Dorazio, 2008), where the superpopulation is defined as the group of individuals that are part of the population at any point during the study period. An individual is in the superpopulation with probability ψ or not with probability 1 − ψ. We model the superpopulation as a Bernoulli trial i.e. Z_i ∼ Bernoulli(ψ). In our Bayesian setup, we assume that every individual in the population is susceptible at time t = 0, (i.e., N₀ = S₀). Therefore, Z is a vector of 1's and 0's, where 1 indicates that an individual is in the population and can be available for detection/testing. Conditional on the presence/absence of an individual, we model the observed event matrix as follows;

$${\mathbf{Y}}_{it}|{\mathbf{X}}_{it},Z_i\sim \text{Categorical}(\mathbf{\Omega }{Z}_i{\mathbf{x}}_{it}),t=1,\dots ,T$$ (4)

where Ω = Ω₁ × Ω₂ is the product of the two matrices given in Equation (2). We now provide the full joint distribution f for our Bayesian SIRMECR formulation, which is used in the Bayesian estimation procedures

$$\begin{array}{l}f(\mathbf{X},p_S,p_I,p_R,\alpha ,\gamma ,{\varphi }_S,{\varphi }_I,{\varphi }_R,{\delta }^S,{\delta }^I,{\delta }^R,\psi )=\mathcal{L}\cdot \pi (p_S,p_I,p_R,\alpha ,\gamma ,{\varphi }_S,{\varphi }_I,{\varphi }_R,{\delta }^S,{\delta }^I,{\delta }^R,\psi )\\ \mathcal{L}=\prod _{i=1}^N\left\{\text{Categorical}\left({\mathbf{x}}_{i1};\eta \right)\cdot \left(\prod _{t=2}^T\text{Categorical}\left({\mathbf{x}}_{it};{\mathbf{x}}_{it-1},\mathbf{\Gamma }\right)\right)\right)\\ \left(\cdot \left(\prod _{t=1}^T\text{Categorical}\left(Z_i{\mathbf{y}}_{it};{\mathbf{x}}_{it},\mathrm{\Omega }\right)\right)\cdot \text{Bernoulli}(Z_i;\psi )\right\}\end{array}$$ (5)

Finally, we let Θ be a vector of all model parameters, which may also include time-dependent random effects. Letting i take all values in 1, …, N, the prior distributions for Equation (5) model are given in Equation (6) below;

$$\begin{array}{ll}{\pi }_{p_S}& \sim \text{Uniform}(0,1),{\pi }_{p_I}\sim \text{Uniform}(0,1),{\pi }_{p_R}\sim \text{Uniform}(0,1),{\pi }_{\alpha }\sim \text{Uniform}(0,1),\\ {\pi }_{\gamma }& \sim \text{Uniform}(0,1),{\pi }_{{\varphi }_S}\sim \text{Uniform}(0,1),{\pi }_{{\varphi }_I}\sim \text{Uniform}(0,1),{\pi }_{{\varphi }_R}\sim \text{Uniform}(0,1),\\ {\pi }_{{\delta }^S}& \sim \text{Uniform}(0,1),{\pi }_{{\delta }^I}\sim \text{Uniform}(0,1),{\pi }_{{\delta }^R}\sim \text{Uniform}(0,1)\end{array}$$ (6)

Furthermore, we model the detection probability $p_t^I$ for infected individuals on the logit scale as a linear function of testing volume:

$$\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}(p_t^I)={\beta }_0+{\beta }_{1}testVolum{e}_t,$$

where β₁ quantifies the change in log-odds of detection probability $p_t^I$ per unit increase in testing volume at sampling time t. We also derive the elasticity is as follows

$$\text{elasticity}=\frac{\nabla p_t^I}{\nabla testVolum{e}_t}\cdot \frac{testVolum{e}_t}{p_t^I}={\beta }_1\cdot testVolum{e}_t\cdot (1-p_t^I),$$

which measures the percentage change in detection probability associated with a 1 % change in testing volume.

3.1. Divide-and-conquer approach for COVID-19 data

Computational challenges arise with MCMC algorithms, commonly employed in both state-space and non-state-space models. Despite benefiting from theoretical guarantees of the asymptotic convergence of estimators with an increasing number of MCMC samples, these algorithms can become computationally infeasible, particularly when applied to datasets with a large number of observations. This becomes evident when handling individual-level datasets. For example, the COVID-19 datasets for the Fraser Health region in BC which consists of over n = 300, 000 tested individuals, poses computational hurdles because Population Data BC requires calculations to be run on their servers, which are computationally limited to handle such a big dataset. Notably, the Fraser Health region is the largest among the five health regions in BC. For instance, estimating undetected cases while considering a population size of approximately N = 2, 500, 000, and the number of unobserved individuals n₀ = 2, 200, 000 becomes daunting.

Applying MCMC to this big dataset can be achieved in two streams; utilising one core of the central processing unit (CPU) and distributing the work load across multiple cores. For single processor case, the computational cost of running MCMC on the full data set may be reduced by using a random subsampling of the data at each iteration (Bardenet et al., 2014; Maclaurin & Adams, 2014; Quiroz et al., 2019). However, this method, while potentially reducing computational costs, comes with its challenges. Specifically, the mixing of MCMC chains may suffer as a consequence (Nemeth & Sherlock, 2018). Moreover, there is a concern about the potential for double counting in capture-recapture models and, consequently, the risk of inaccurate enumeration. This arises from the subsampling approach, which may duplicate observations, leading to biased estimates. Such inaccuracies can significantly impact the reliability of the results, especially in scenarios where precise enumeration is crucial, such as estimating undetected COVID-19 cases. Alternatively, modern computer architectures readily utilize multiple cores of the CPU for computers, but MCMC algorithms are serial in implementation. Parallel MCMC, where multiple MCMC chains, each targeting the full posterior, are run on separate cores, can be easily executed (Wilkinson, 2006), however, this does not completely remove the computational burden as each core still needs to store and evaluate the whole data set. To generate a significant computational speed-up, the data set can be partitioned into disjoint batches and run on independent processors where independent MCMC algorithms are executed (Huang & Gelman, 2005). Using only a subset of the entire data means that the MCMC algorithm targets a different posterior distribution conditional only on a subset of the data. The main challenge, however, is how to merge the separate MCMC samples to approximate the full posterior distribution. We further explore the divide-and-conquer approach on our COVID-19 data set.

Consider a full observed data set $\mathcal{Y}=\{{\mathbf{y}}_{it},\dots ,{\mathbf{y}}_{iT}\}$ and state data set $\mathcal{X}=\{x_{it},\dots ,x_{iT}\}$, where we assume that the data are conditionally independent with a likelihood $\mathcal{L}(\mathbf{\Theta }|{\mathbf{y}}_i)$ given in Equation (5). We randomize the full data sets to allow mixing between observed data of size n and augmented data set n₀. This is crucial as capture-recapture enumerates the unobserved individuals from the observed information. Therefore, naive partitioning of the full dataset could result in a subset with only unobserved augmented data which can lead to error in merging methods and produce inaccurate estimates. Furthermore, we partition the data into independent K batches. The choice of K batches is worth exploring to avoid very small sample size yet have feasible computation time for MCMC runs. Hence, our posterior distribution in Equation (5) now becomes a sub-posterior operating on different cores or machines as

$$\begin{array}{l}p(\mathbf{\Theta },\mathcal{X}|\mathcal{Y})\propto \prod _{k=1}^K\left[{\pi }_k(\mathbf{\Theta })\cdot \prod _{i=1}^{n_k}\left\{\text{Categorical}\left({\mathbf{x}}_{ki1;{\eta }_k}\right)\cdot \left(\prod _{t=2}^T\text{Categorical}\left({\mathbf{x}}_{\mathit{kit}};{\mathbf{x}}_{\mathit{kit-1}},{\mathbf{\Gamma }}_k\right)\right)\right)\right)\\ \left(\left(\cdot \left(\prod _{t=1}^T\text{Categorical}\left(Z_{ki}{\mathbf{y}}_{\mathit{kit}};{\mathbf{x}}_{\mathit{kit}},{\mathrm{\Omega }}_k\right)\right)\cdot \text{Bernoulli}(Z_{ki};{\psi }_k)\right\}\right]\end{array}$$ (7)

Again, considering our Fraser region COVID-19 data set, after randomizing, we let K = 250 batches, each with sample size of n_k = 10, 000 using C = 15 cores with each core holding each batch gives 150, 000 parallel MCMC of G runs (i.e., 2, 500, 000/150, 000 ∼ 17). Assuming we have a single core, MCMC will run 250 times with computation time of $\mathcal{O}(n_{k}J)$, where J is the MCMC iterations. This is 40 times faster than $\mathcal{O}(NJ)$. Alternatively, running in parallel on C cores, MCMC will have $\mathcal{O}(n_{k}J/C)$ times faster than the full approach and C time faster than serial runs. To merge our MCMC samples, we sum the $N_g^S,N_g^I,N_g^R$ and $N_g^D$ to make N_g = 150, 000, and ${\sum }_g^G{N}_{gt}=N_t$. To merge the discrete probabilities, we take the average across the G runs, that is $\frac{1}{G}{\sum }_g^G{p}_{gt}^I=p_t^I$. Algorithm 1 shows the step-by-step procedures and how we process using nimble and R.

Algorithm 1

Divide-and-Conquer MCMC for SIRMECM

1: Input: state matrix $\mathbf{X}$ and observed data $\mathcal{Y}$ of size N = n + n0.

2: K is the number of batches

3: nk is the sample size in each batch

4: C is the number of cores

5: G is the number of replicates to run the whole batches using C cores

6: Randomize to allow for mixing between n and n0.

7: Split the input data equally into K batches sequentially to avoid double counting.

8: Set up parallel MCMC as follows;

9: forg = 1, …, Gdo

10: For each g, run C batches in parallel, i.e., nk × C in parallel using C cores.

11: Use likelihood function $\mathcal{L}(\mathbf{\Theta },\mathcal{X}|\mathcal{Y})$ and proposal distribution $\mathcal{Q}({\mathbf{\Theta }}_{1:n_k},{\mathcal{X}}_{1:n_k})$

12: for$c=1,\dots ,\mathcal{C}$do

13: sample $({\mathbf{\Theta }}_{1:n_k},{\mathcal{X}}_{1:n_k})\sim \mathcal{Q}(.)$ according to nimble setup

14: Evaluate each subposterior ${\pi }_c({\mathbf{\Theta }}_{1:n_k},{\mathcal{X}}_{1:n_k})$ using MCMC sampler in nimble.

15: endfor

16: endfor

17: Compute the probabilities $\mathbf{\Theta }=\frac{1}{G}{\sum }_g^G{\mathbf{\Theta }}_g^G$, ${\sum }_g^G{N}_{gt}^S=N_t^S$, ${\sum }_g^G{N}_{gt}^I=N_t^I$, ${\sum }_g^G{N}_{gt}^R=N_t^R$, and ${\sum }_g^G{N}_{gt}^D=N_t^D$

18: Output:Θ, $N_t^S,N_t^I,N_t^R$ and $N_t^D$, where $N_t=N_t^S+N_t^I+N_t^R+N_t^D$.

4. Case study: Covid-19 data

We use our SIRMECR model to estimate COVID-19 data available in the Population Data BC database over the period of 12 months between January to December 2020. The data involves individual information on the COVID-19 testing trajectories and the death. We monitor monthly COVID-19 test results of individuals to account for both incubation period before infection and recovery period after infection. The outcomes of the COVID-19 tests are categorized as “positive” or “negative” based on the presence or absence of viral RNA or “indeterminate” when clinical tests cannot be substantiated. Individuals exhibiting negative results and lacking clinical signs are classified as being in the Susceptible state. However, if clinical signs and/or at least one positive test result are observed within the same month, individuals are categorized as being in the Infectious state. This classification encompasses both the contagious and non-contagious stages of COVID-19 infection. To account for the recovered state without antibody results, we classified individuals with all negative results in a month after positive results. Any negative result between two positive results in a month is treated as positive (i.e., infected). Therefore, individuals with all negative results after positive results in the previous month are classified as Recovered. This is to factor in the 14 days recovery period. Finally, we handle individuals with indeterminate results with care. Contrary to Béland et al. (2021), any indeterminate results in-between two negative results are relabelled negative. Similarly, any indeterminate results in-between positive results are relabelled positive results while indeterminate results between positive and negative results are positive results. For this reason, SARS-Cov-2 infection occurs only once in an individual's life. By this fact, any individual classified as (1) “susceptible” in a given month was classified as susceptible during all previous months when the individual was detected; (2) “infected” in a given month was classified as “susceptible” during all previous months, and “recovered” during all subsequent months when the individual was detected; (3) “recovered” in a given month was classified as “recovered” during all subsequent months and (4) “dead” in a given month with any test results is classified as a known “dead” due to COVID-19 and all subsequent months. We assume that individuals that survived COVID-19 infection i.e., recovered retain their recovery state. Specifically, in the context of a vaccination program, those that were already infected and recovered may have developed “herd immunity” (Brown et al., 2021; Wu et al., 2020). When constructing the cross-sectional dynamic abundance, denoted as $N_t^A$, we standardize all metrics per 100,000 population. This standardization aids in interpretation and allows for comparisons of estimated infection rates across the five health regions in British Columbia. Based on our data cleaning process, the observed cases in the year 2020 are as follows: Fraser Health: 33,150 cases, Northern Health: 1831 cases, Interior Health: 4054 cases, Vancouver Island Health: 1094 cases, and Vancouver Coastal Health: 11,178 cases. We also identified 24,152 individuals who did not belong to any health region. The total number of infected individuals in BC is 51,307. Notably, these figures closely align with the reported cases on the BCCDC dashboard [BCCDC].

We merged 2021 census reports specific to BC from popDataBC to monitor the population count in each health region. For instance, our data processing revealed approximately N = 2, 644, 000 individuals in the Fraser Health region, constituting roughly 60 % of the total BC population. To speed up our MCMC, we adopted a divide-and-conquer approach with parallel computing for our analysis. We partitioned our dataset into $\sim 265$ batches, each comprising 10, 000 observations. To reduce the computation processing time, we used 15 cores, allocating 10, 000 observations to each core. Each round of analysis processed 150, 000 observations which took an average of 12 min to run. We performed a garbage collector to release the memory that is not needed anymore after every three 15, 000-observation runs.

4.1. Results

4.1.1. Model without covariates

To analyze COVID-19 data set, we fitted our base model to individual-level COVID-19 information available in population Data Bc database. First, we fitted our model without covariates to estimate the infection probability and recovery probability. Secondly, we fitted our model with covariates using the test intensity rate to assess the impact of rapid testing measures in BC. To account for the state of individuals before COVID-19, we set $N_0^S=N_0$, that is the initial region abundances equal to the number of individuals at time t = 0 and 0 for all other states before January 2020 (i.e., $N_0^I=0,N_0^R=0$ and $N_0^D=0$).

The fitted model parameters with no covariates are shown in Table 2. We estimated the infection probability α and the recovery probability γ for Northern region to be 0.12 % and 0.5274, respectively. We estimated the probability of remaining active after infection 1 − γ to be 0.473 in Northern Health Region. For our model, the number of death in all the regions was estimated to be essentially zero. We found that probability of survival across all the regions was $\sim 1.0$. This indicates that individuals will remain alive until infected by SARS-cov-2. Note that the estimated parameters derived from our study closely agree with those documented by Parker et al. (2021) in their investigation of the Northern health region. Our analysis of recovery probability 0.527 falls between their reported recovery probabilities of 0.48 (MLE) and 0.62 (MCMC). For the detection probabilities, we estimated the Northern region detection probability in susceptible state p^S as 0.02. This indicates that any individual without symptoms will have a 2 % probability of being tested. However, the detection probability in an infection state was estimated as 0.69. This means that there is a high probability of any individual with symptoms to be tested. Considering the detection probability in any state, we take the average of p^S = 0.02, p^I = 0.69, and p^R = 0.30 to be 0.34. Additionally, we found that our estimated detection probabilities $\overline{p}=0.34$ in any given states agreed with the previously reported detection probability 0.3 of the same region, further validating the consistency of our findings with their research.

Table 2.

Parameter estimates for the base model (with no parameter covariates) fitted for the five Health Authority Regions in B.C.

Parameter	Fraser Health Region	Northern Health Region	Interior Health Region
α	0.0018 (0.0006, 0.0065)	0.0012 (0.0002, 0.0061)	0.0011 (0.0001, 0.0065)
γ	0.6268 (0.0554, 0.9191)	0.5274 (0.0250, 0.9764)	0.5149 (0.0241, 0.9754)
pI	0.7360 (0.2216, 0.9510)	0.6904 (0.1774, 0.9893)	0.6624 (0.1569, 0.9870)
pR	0.1974 (0.0136, 0.6988)	0.2996 (0.0141, 0.8553)	0.3310 (0.0156, 0.8774)
pS	0.0267 (0.0171, 0.0496)	0.0213 (0.0113, 0.0470)	0.0259 (0.0148, 0.0492)
ϕI	0.9141 (0.6903, 0.9979)	0.7591 (0.3057, 0.9929)	0.7291 (0.2661, 0.9911)
ϕR	0.7699 (0.2828, 0.9927)	0.6450 (0.1229, 0.9852)	0.6120 (0.1013, 0.9829)
ϕS	0.9999 (0.9997, 1.0000)	0.9999 (0.9997, 1.0000)	0.9999 (0.9997, 1.0000)
δI	0.9194 (0.7318, 0.9973)	0.8134 (0.4544, 0.9934)	0.7832 (0.3885, 0.9922)
δR	0.5610 (0.0882, 0.9767)	0.5115 (0.0379, 0.9745)	0.5113 (0.0363, 0.9752)
δS	0.9909 (0.9705, 0.9995)	0.9863 (0.9547, 0.9993)	0.9895 (0.9637, 0.9996)

Parameter	Island Health Region	Coastal Health Region	British Columbia
α	0.0009 (0.0000, 0.0062)	0.0013 (0.0002, 0.0064)	0.0013 (0.0002, 0.0063)
γ	0.4975 (0.0249, 0.9734)	0.5893 (0.0445, 0.9767)	0.5512 (0.0348, 0.9642)
pI	0.5531 (0.0692, 0.9783)	0.6849 (0.1694, 0.9870)	0.6654 (0.1589, 0.9785)
pR	0.4174 (0.0202, 0.9303)	0.2330 (0.0143, 0.7634)	0.2957 (0.0156, 0.8250)
pS	0.0304 (0.0179, 0.0530)	0.0351 (0.0215, 0.0579)	0.0279 (0.0165, 0.0513)
ϕI	0.6107 (0.1206, 0.9843)	0.8281 (0.4556, 0.9995)	0.7682 (0.3677, 0.9923)
ϕR	0.5583 (0.0625, 0.9793)	0.7305 (0.2256, 0.9903)	0.6631 (0.1590, 0.9861)
ϕS	0.9999 (0.9997, 1.0000)	0.9999 (0.9997, 1.0000)	0.9999 (0.9997, 1.0000)
δI	0.6168 (0.1401, 0.9825)	0.8328 (0.4956, 0.9942)	0.7931 (0.4421, 0.9919)
δR	0.5056 (0.0304, 0.9751)	0.5418 (0.0661, 0.9771)	0.5262 (0.0518, 0.9757)
δS	0.9898 (0.9645, 0.9996)	0.9893 (0.9637, 0.9995)	0.9892 (0.9634, 0.9995)

In our study, we focused on partial observation of infection states, which occurs when individuals in the population observed alive are assigned to an infection states with some uncertainty. Specifically, we assigned the “Indeterminate” results to susceptible, infected, or recovered states which could result into “state misclassification”. Although, state uncertainty is likely to be widespread at the beginning of COVID-19 (lack of knowledge about the strain) due to false positive and false negative test results. To account for uncertainty in our COVID-19 data set, we estimated the assignment probabilities δ in different states. We found discrepancies in the assignment probabilities in infection state across the regions. We observed that Fraser health region has the highest assignment probability δ^I = 0.92 in infection state and Island health region has the least assignment probability in infection state δ^I = 0.62. This means that 92 % of the COVID-19 tests in Fraser regions were correctly classified while only 62 % of the tests were correctly classified in Island region in 2020. As pointed out by Robardet et al. (2017), we observed that irrespective of any variation in detection probabilities among infection states.

it is likely that infection states will not have equal assignment probabilities. This heterogeneity may have resulted from the incubation periods observed for COVID-19 (i.e., the manifestation of the disease in the host) and/or the diagnostic method employed.

We estimate the number of undetected COVID-19 cases, considering the dynamic changes in the number of susceptible, infected, recovered, and dead individuals. Table 3 shows the number of observed cases in all the regions considered. According to our data cleaning in PopdataBc, there were 51, 307 reported COVID-19 cases in BC. Instead using our model, we estimated the cumulative total number of infected COVID-19 cases in BC to be 96, 639. This indicates that $\sim 47\%$ of BC population remained undetected in 2020. For the number of undetected COVID-19 cases, we estimated that the number of undetected COVID-19 cases is 45, 332 in BC. Further, we found that the monthly number of undetected cases increases rapidly between January to September 2020, while plummeting starting from October 2020. This might be as a result of increased testing which we further explored using testing volume as a covariate in our second model. For better comparison across the regions, we produced the number of infected individuals per 100,000 populations. We found that Fraser has the highest number of undetected cases (i.e., 1179 per 100,000 population) and Vancouver Island has the least number of undetected cases (104 per 100,000). Fig. 2a shows the estimated cumulative number of COVID-19 cases (red line) compared to the observed cases (blue line) for all regions under consideration. In our base model (i.e., without covariates), we found the percentage of under-reporting to range from 44 % to 53 % across all regions. This underscores the significant underestimation of under-reporting in the absence of mitigation measures such as testing measures. Additionally, in Fig. 2a, we observed that the total number of infected cases slightly exceeds the observed count, with the Island region exhibiting nearly identical figures. Moreover, our model accounts for the uncertainty in the detection probabilities. We also displayed the plot that shows the estimated new cases and the observed new cases. This indicates that our model was able to predict the observed cases. Fig. 2b shows the observed cases compared to the estimated new cases derived by our model. Based on the detection probability, our estimated new cases is comparable to the observed new cases. For Coastal, Fraser, and Island regions, there were low number of new cases from January to September 2020. However, in October there was a rapid increase in COVID-19 infection until November, and a huge decrease in December. On the other hand, Northern region remained increasing in number of new COVID-19 cases.

Table 3.

The number of under-reporting of Covid-19 cases without covariates.

Number of Infections
Health Region	Confirmed	Undetected	Total	% Undetected Infections
Fraser	33, 150	31, 182	64, 332	48.47 %
Northern	1, 831	1, 170	3, 001	38.99 %
Interior	4, 054	2, 912	6, 966	41.80 %
Vancouver Island	1, 094	1, 264	2, 358	53.60 %
Vancouver Coastal	11, 178	8, 804	19, 982	44.06 %
Region Sum	51, 307	45, 332	96, 639	46.91 %

Infections per 100,000 Population
Health Region	Confirmed	Undetected	Total	Population
Fraser	1, 254	1, 179	2, 433	2, 643, 958
Northern	436	279	715	412, 928
Interior	334	240	574	1, 211, 376
Vancouver Island	90	104	194	1, 212, 251
Vancouver Coastal	497	391	888	2, 249, 441
Region Sum	2611	2193	4, 804	7, 729, 954

Fig. 2.

(a): The total infection vs the observed cases. (b): The estimated new cases vs observed new cases.

4.2. Average recovery times

To compute the average recovery time $(\overline{\gamma })$, we utilize the monthly probability of recovery (γ) and the monthly probability of remaining infected, denoted as 1 − γ. Directly taking the reciprocal of the recovery probability would lead to inaccuracies due to the consideration of monthly periods. Therefore, following the approach outlined by Parker et al. (2021), we adopted the geometric series. This series captures the repeated likelihood of recovery for each month of remaining infected, expressed as the infinite sum $S_q=\gamma {\sum }_{q=0}^Q{(1-\gamma )}^q$. Here, S_q represents the cumulative probability of recovery up to month Q. For S_q = 0.5, we calculated Q using Q = log(1 − S_q)/log(1 − γ). For the Northern region, where γ = 0.527 and 1 − γ = 0.473, we found Q = 0.93. Consequently, the average days of recovery amount to 93 % of 7 days, resulting in approximately 6.48 (CI: 5.0, 13.8) days. Comparing this with Parker et al.‘s findings for the Northern region, where the recovery probability is reported as 0.62 for MCMC with an average recovery time of 5 days, we note a slight agreement. Fig. 3 illustrates the distribution of recovery days across all regions. We observed that Interior health region had the longest recovery days of 14.8 days, while Fraser had the shortest recovery days of 2.6 days. Notably, we observe that the maximum recovery time was below 15 days.

Fig. 3.

Recovery Times of Health Region in days.

4.3. Model with covariates

In our second model, we incorporated testing volumes as our covariates to better estimate the number of undetected COVID-19 cases. According to Parker et al., 2021, testing volume becomes one of the important covariates for accurate estimation. Fig. 4 shows the number of positive results from the COVID-19 tests and Fig. 5 shows the testing volumes for all the five health Authority regions in BC. From the plots, it was clear that all the regions had significant positive result starting from August, while the number of COVID-19 positive results rose to the peak in November, it plummeted in December 2020. Fig. 5 shows an increase in the number of testing volumes in all the regions which was intensified starting from March 2020. We found the testing volume to be statistically significant in all regions except Vancouver Island. This exception may be due to the low number of infected cases in the area. Consequently, the results for Vancouver Island are very similar to those of the model without covariates.

Fig. 4.

The time series data of the positive results which is measured by the number of positive tests per day in the five health authority regions of BC and BC.

Fig. 5.

The time series data of all the COVID-19 tests which is measured by the sum of positive tests, negative results, and indeterminate results per day in the five health authority regions of BC and BC.

Fig. 6 presents the posterior predictive checks comparing the estimated detected COVID-19 cases with the observed COVID-19 cases. The estimated detected cases agreed with the observed cases. For example, we estimated the number of detected COVID-19 cases in Fraser region to be 33,632.50 (Crl: 25,169.73, 43,745.89) and the number of observed cases is 33,567. In Northern region, our estimated detected COVID-19 cases is 2366.14 (Crl: 1075.18, 4307.07) which is close to 2045 cases observed. In Fig. 7, we compared our estimated cumulative total detected COVID-19 cases with the cumulative observed cases in each regions and BC. We found that our estimated cumulative total detected COVID-19 cases align with the cumulative observed cases. Also, we estimated that 1.96 % (Crl: 1.07 %, 3.31 %) of the BC population had COVID-19 infection in 2020 which agrees with citf seroprevalence of 2.5 % in 2020 (Murphy et al., 2023).

Fig. 6.

The number of detected COVID-19 cases in the five Health Authority regions in BC. Our model produces estimated detected COVID-19 cases that capture the observed.

Fig. 7.

The estimated cumulative total detected COVID-19 cases in the five Health Authority regions in BC. Our model produces estimated the cumulative total detected COVID-19 cases that capture the observed.

Fig. 8 also shows the number of undetected COVID-19 cases in all the regions compared to the detected cases. In this figure, it is obvious that the number of undetected COVID-19 cases grew very fast from January 2020 to September 2020, while the observed COVID-19 cases grew slowly. This is based on the low testing volume at the onset of the pandemic. From October 2020, there was a bit of overlap between the undetected and the detected COVID-19 cases. This is due to a rapid testing that took effect at the later period of the year. Significantly, both Northern and Interior regions had the numbers of undetected and detected cases overlap in December while Coastal and Fraser regions had both numbers overlap in November. This may be the months when the testing volumes were at their peaks. Having accurate estimates of the cumulative infections is important as testing became increasingly rapid in 2020 and as vaccines became increasingly available in the later period of the pandemic. This may suggest what percentage of the population is immune to the virus if those already infected are not susceptible, how many people need to get a vaccine before we reach herd immunity. We estimated the total cases based on the observed data of confirmed cases and our estimated undetected cases. The total infected cases CTI(T) up to date T is CTI(T) = CDI(T) + CUI(T), where CDI(T) is the confirmed cases and CUI(T) is the undetected infected cases. Our data set has the number of positive tests and we use this to measure confirmed cases. Using our model, the estimated undetected cases as of December 2020 are shown in Table 4. Table 4 also shows our estimated percentage of undetected COVID-19 cases out of the total COVID-19 cases to be between 77.4 %and 84.0 % for British Columbia. Our estimates are consistent with Parker et al. (2021), which used N-mixture model for Northern Heath region of BC. Parker et al. (2021) found that the estimated percentage of undetected COVID-19 cases in Northern region ranged between 60.2 % − 84.2 %. Note that our estimate has a narrower credible interval which overlaps with Parker et al. (2021). While, we found that a 95 % credible interval of 2.30–5.14 times greater than the reported number of cases, Parker et al. (2021) found 3.69 − 8.75 using publicly available case count and Skowronski et al. (2020) using serology data found between 2.25 and 20.5 times greater than the reported cases in the lower mainland. Even though, we do not have a direct comparison for other regions, we noticed that the percentage of undetected cases are very similar as shown in Table 4. This is also consistent with Parker et al. (2021). Fig. 9 shows our estimated newly detected cases in BC compared to the observed new cases. We observed that the new cases grew gradually in all the regions and peaked in November to more than 10, 000 in BC. Fig. 10 shows the detection probabilities estimated by our model for all the regions considered and the province of BC at large. We found that the detection probabilities in infectious state $p_t^I$ reached a high of 0.5 between October and November 2020 in BC. The average detection probability in an infectious state in BC is 0.24 (95 %CrI:0.10, 0.35). Based on our hypothesis, the detection probability or the chance of being available for testing is strongly predicated on whether an individual is infected with COVID-19 or not. We found that in BC, the chance of testing an individual in a susceptible state ranges from 2 %to 3 %. We note that time-varying variable can have identifiability issues. However, we did not have any identifiability issue because we only used testing volume as a covariate for detection probability $(p_t^I)$, while we fixed infection probability α.

Fig. 8.

The number of undetected COVID-19 cases in the five Health Authority regions in BC. Our model produces estimated undetected COVID-19 cases that capture the observed.

Table 4.

The number of under-reporting of Covid-19 cases with covariates with credible intervals in bracket.

Number of Infections
Health Region	Confirmed	Undetected	Total	% Undetected Infections
Fraser	33, 567	150, 287	183, 854	81.7 % (78.8 %, 84.0 %)
Northern	2, 045	7, 329	9, 374	78.2 % (69.7 %, 83.7 %)
Interior	4, 087	19, 141	23, 228	82.4 % (77.1 %, 85.7 %)
Vancouver Island	944	919	1, 863	49.3 % (0.00 %, 84.4 %)
Vancouver Coastal	12, 246	48, 230	60, 476	80.0 % (74.4 %, 83.4 %)
Region Sum	52, 889	225, 906	278, 795	81.0 % (77.4 %, 84.0 %)

Infections per 100,000 Population
Health Region	Confirmed	Undetected	Total	Population
Fraser	1, 270	5, 684	6, 954	2, 643, 958
Northern	487	1, 745	2, 232	412, 928
Interior	336	1, 575	1, 911	1, 211, 376
Vancouver Island	78	76	154	1, 212, 251
Vancouver Coastal	544	2, 144	2, 688	2, 249, 441
Region Sum	2, 715	11, 224	13, 939	7, 729, 954

Fig. 9.

The estimated new detected COVID-19 cases in the five Health Authority regions in BC. Our model predicts the new COVID-19 cases in the five Health Authority Regions.

Fig. 10.

The estimated detection probabilities in the five Health Authority regions in BC. Our model produces estimated the detection probabilities of COVID-19 cases that capture the observed.

5. Simulation study

We conducted a simulation study to assess the performance of our model and validate our divide-and-conquer approach to MCMC. Unlike its conventional use in the field of machine learning (ML), our divide-and-conquer approach was adapted to address the issue of double counting in ecological data, as discussed in subsection 2.2.

For the simulation study, we designed the parameters based on the output data from the Northern Health Region of British Columbia, excluding covariates (see Table 2). The total population size, N, was set to 400,000, and the population was stratified into four mutually exclusive states: Susceptible, Infectious, Recovered, and Death. To implement the divide-and-conquer method, we randomly partitioned the dataset into 40 batches, with each batch containing n_k = 10, 000 observations, where k represents the batch index. The MCMC process was then parallelized across ten cores $(\mathcal{C})$, and the entire process was repeated four times to ensure the full population (N) was captured. We used 200, 000 for the number of iterations in each 10 cores. To evaluate computational efficiency, we measured the runtime of our setup. Running n_k observations took approximately 1.8 h. This represents a significant computational advantage, as MCMC is inherently serial and can be prohibitively expensive when applied to large datasets. To combine our results as outlined by Nemeth & Sherlock (2018), we employed a two-step merging process for the outputs from our divide-and-conquer MCMC approach. In the first step, we integrated the results from ten cores, $\mathcal{C}$, and four independent runs by computing the average of all probabilities and summing all counts. Table 5 shows the results from our simulation study. The results demonstrate that the divide-and-conquer MCMC approach does not affect the accuracy of our estimates, as our method produces estimates that closely aligned with the true parameters. Fig. 11 shows the 10 % partial posterior plots, while Fig. 12 presents the trace plots of the parameters estimated using the divide-and-conquer approach combined with our merging strategy. From the figures we found that convergence was achieved after 200,000 MCMC iterations.

Table 5.

Parameter estimates from our simulation study using the base model (with no parameter covariates).

Params	Truth	Estimate	Std Error	95 % CI
α	0.0010	0.0013	0.0043	(0.0007, 0.0016)
γ	0.5000	0.5324	0.0795	(0.3823, 0.6686)
pI	0.6000	0.6692	0.0898	(0.5199, 0.7986)
pR	0.3000	0.2740	0.0316	(0.2155, 0.3333)
pS	0.0300	0.0312	0.0052	(0.0293, 0.0397)
ϕI	0.7000	0.7254	0.0480	(0.6377, 0.8132)
δI	0.8000	0.7497	0.0515	(0.6468, 0.8455)
δR	0.5000	0.5003	0.0601	(0.3845, 0.6185)
δS	0.9900	0.9907	0.0112	(0.9871, 0.9943)

Fig. 11.

The 10 % partial plots to assess convergence for the parameters of the simulated study.

Fig. 12.

The traceplots for the parameters of the simulated study.

6. Discussion

We developed a novel model for disease analytics, accounting for population dynamics and undetected COVID-19 cases by hybridizing SIR models with capture-recapture models. We compare two versions of our model (with or without covariates), to study the impact of time-varying parameters on detection probabilities. From our study, base model without covariates underestimated the parameter values when there are no testing measures. Also, we found that using individual-level data improved parameter estimates compared to studies with aggregated data.

However, one of our challenges was dealing with large amount of data and using PopdataBc server which because of data privacy becomes restrictive. To overcome this big-data problem, we developed divide and conquer techniques for our model estimation to improve the speed of our MCMC methods. Moreover, to avoid double counting problem in enumeration, we randomized our COVID-19 data and sequentially split into K batches as outlined in Algorithm (1).

We estimated the percentage of undetected COVID-19 cases in the BC Health Authority regions, and found that there is a significant level of undetected COVID-19 cases in 2020. Estimates of detection probability from our model was found to closely follow the trend in monthly testing volumes. We also found the overall percentage of undetected COVID-19 cases in BC ranged from 77.4 %to 84 %. We also estimated the probability of death in Northern region for comparison to be 0.008 (95 %CrI:0.0002, 0.034) which also agrees with Parker et al. (2021); Ioannidis (2020); Pezzullo et al. (2023); Sorensen et al. (2022).

Contrary to Parker et al. (2021), our model was able to account for difference between symptomatic and asymptomatic cases. Our model estimated the total number of COVID-19 cases (including both symptomatic (observed positive tests) and asymptomatic cases (undetected COVID-19 cases)). Our study found that the reported cases in BC did not capture the true scope of the COVID-19 infections. Our estimates across five health regions in BC gave similar estimates on percentage of undetected COVID-19 cases.

The limitation of this study is from the assumptions made during the data cleaning phase. Following Benhaiem et al. (2018); Marescot et al. (2018), we treated the sequence of negative results after positive results as a recovered state. This may give a different result if we had used recovery plans. Concerning mobility, while each Health Region Authority was modelled independently, our primary aim was to estimate province-wide parameters by aggregating results across all regions. As such, population movements between regions have limited impact on the overall estimates. Therefore, explicitly modelling migration and importation dynamics would substantially increase model complexity and computational burden. Given current server constraints, this extension was not feasible, though we acknowledge it as a valuable direction for future work. Also, using a more granular computing technique will provide a better speed and accuracy instead of using divide and conquer approach to MCMC. Even though, there was an improvement in speed compared to serial MCMC which uses a whole data set at each MCMC iteration, the choice of K batches for a reasonable speed and proper MCMC mixing becomes a problem.

In future work, we wish to use more effective variational techniques such as Variational Bayes (VB) or hybrid VB and MCMC for our model estimation. Variational techniques have proven to be effective when handling big data. To the best of our knowledge, this type of estimation method has received little to no attention in ecology. Also, we hope to extend our model component with capture-recapture model. For example, taking into account the exposed individuals before infectious state (SEIR-MECR).

CRediT authorship contribution statement

Kehinde Olobatuyi: Writing – review & editing, Writing – original draft, Visualization, Software, Methodology, Funding acquisition, Formal analysis, Data curation, Conceptualization. Junling Ma: Writing – review & editing, Validation, Supervision. Patrick Brown: Writing – review & editing, Validation, Supervision, Funding acquisition. Laura L.E. Cowen: Writing – review & editing, Validation, Supervision, Resources, Funding acquisition, Data curation.

Data availability statement

Due to data privacy, the data sets used and/or analyzed during the current study is not available. However, we have prepared a simulated data and the codes used for the results in the supplemental materials.

Disclaimer

Access to data provided by the Data Stewards is subject to approval but can be requested for research projects through the Data Stewards or their designated service providers. The following data sets were used in this study: consolidation - census geocodes, consolidation - demographics, consolidation - registry, COVID-19 Testing, and VS - deaths. You can find further information regarding these data sets by visiting the PopData project webpage at: https://my.popdata.bc.ca/project_listings/21-016/collection_approval_dates. All inferences, opinions, and conclusions drawn in this publication are those of the author(s), and do not reflect the opinions or policies of the Data Steward(s).

Declaration of generative AI and AI-assisted technologies in the writing process

During the preparation of this work, Kehinde Olobatuyi used Chatgpt in order to check for grammatical errors in some of the paragraphs. After using this tool/service, I reviewed and edited the content as needed and take full responsibility for the content of the publication.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Handling Editor: Dr. Jianhong Wu

Appendix A. Elasticity, Autocorrelation and Traceplots

Fig. 13.

Elasticity of the logit link function.

Fig. 14.

The Autocorrelation plots of parameter estimates in the five Health Authority regions in BC.

Fig. 15.

The Traceplots of alpha, beta, gamma in the five Health Authority regions in BC.

Fig. 16.

The Traceplots of detection probabilities (p_A_1-p_A_3)in the five Health Authority regions in BC.

Fig. 17.

The Traceplots of detection probabilities (p_A_4-p_A_6)in the five Health Authority regions in BC.

Fig. 18.

The Traceplots of detection probabilities (p_A_7-p_A_9)in the five Health Authority regions in BC.

Fig. 19.

The Traceplots of detection probabilities (p_A_10-p_A_12)in the five Health Authority regions in BC.

Fig. 20.

The Traceplots of detection probability in recovered state, survival probabilities in infected and recovered states in the five Health Authority regions in BC.

Appendix B. Supplementary data

The following are the Supplementary data to this article.