Skip to main content
Infectious Disease Modelling logoLink to Infectious Disease Modelling
. 2021 Mar 4;6:490–502. doi: 10.1016/j.idm.2021.02.005

Within-host model of respiratory virus shedding and antibody response to H9N2 avian influenza virus vaccination and infection in chickens

Xiao-Ting Xie a, Alexander Yitbarek b, Jake Astill b, Shirene Singh c, Salah Uddin Khan a, Shayan Sharif b, Zvonimir Poljak a, Amy L Greer a,
PMCID: PMC7966989  PMID: 33778216

Abstract

Avian influenza virus (AIV) H9N2 subtype is an infectious pathogen that can affect both the respiratory and gastrointestinal systems in chickens and continues to have an important economic impact on the poultry industry. While the host innate immune response provides control of virus replication in early infection, the adaptive immune response aids to clear infections and prevent future invasion. Modelling virus-innate immune response pathways can improve our understanding of early infection dynamics and help to guide our understanding of virus shedding dynamics that could lead to reduced transmission between hosts. While some countries use vaccines for the prevention of H9N2 AIV in poultry, the virus continues to be endemic in regions of Eurasia and Africa, indicating a need for improved vaccine efficacy or vaccination strategies. Here we explored how three type-I interferon (IFN) pathways affect respiratory virus shedding patterns in infected chickens using a within-host model. Additionally, prime and boost vaccination strategies for a candidate H9N2 AIV vaccine are assessed for the ability to elicit seroprotective antibody titres. The model demonstrates that inclusion of virus sensitivity to intracellular type-I IFN pathways results in a shedding pattern most consistent with virus titres observed in infected chickens, and the inclusion of a cellular latent period does not improve model fit. Furthermore, early administration of a booster dose two weeks after the initial vaccine is administered results in seroprotective titres for the greatest length of time for both broilers and layers. These results demonstrate that type-I IFN intracellular mechanisms are required in a model of respiratory virus shedding in H9N2 AIV infected chickens, and also highlights the need for improved vaccination strategies for laying hens.

Keywords: Avian influenza, Disease modelling, Innate immunity, Adaptive immunity, Poultry

Abbreviations

AIV

Avian influenza virus

AICc

Corrected Akaike information criterion

CAD

Canadian dollars

dph

Days post-hatch

dpi

Days post-infection

HA

Hemagglutinin

HI

Hemagglutination inhibition

IFN

Interferon

ODE

Ordinary differential equations

ODN

Oligodeoxynucleotides

PRCC

Partial rank correlation coefficient

TCID

Tissue culture infectious dose

1. Introduction

Low pathogenic (LP) H9N2 avian influenza virus (AIV) is the causative agent of an important infectious disease in the poultry population. The importance of this AIV subtype can be attributed to its potential to result in economic losses in the event of an outbreak and production loss (Peacock, James, Sealy, & Iqbal, 2019; Sun & Liu, 2015). Effective control of the virus in domestic poultry requires improved understanding of host-pathogen dynamics, and how interventions such as vaccination can reduce infection and transmission in birds. Type-I interferons (IFNs) have been shown to play a major role in facilitating early anti-viral mechanisms in infected hosts (Evseev & Magor, 2019; Hsu, 2018; Kreijtz, Fouchier, & Rimmelzwaan, 2011; Williams, 2009). H9N2 AIV transmission between birds may results from virus shed through cloacal or respiratory routes (Guan, Fu, Chan, & Spencer, 2013; Jegede, Fu, Lin, Kumar, & Guan, 2019). Respiratory and cloacal virus shedding patterns in H9N2 AIV infected chickens have also been shown to differ, but it is still unclear whether this is due to differences in the route of infection or in localized early immune response. Therefore, it is important to also examine type-I IFN mechanisms which contribute to respiratory virus shedding patterns. Vaccination programs for the control of H9N2 AIV have been implemented in Asian and Middle Eastern countries where outbreaks occur more frequently. Notably, the impact of these programs on the spread of H9N2 AIV has been limited, primarily due to sub-optimal use of vaccines and antigenic drift, implying a need for improved vaccines and vaccination strategies (Alexander, 2007; Gharaibeh & Amareen, 2016; Kilany et al., 2016; Peacock et al., 2019; Sun & Liu, 2015). The experimental testing of vaccine candidates and exploration of novel vaccination strategies in the laboratory is a critical step, but these studies are often limited in the number of animals used and the length of the experimental period. Feasible tools to assess vaccine efficacy at the host and flock level over an extended timeframe are needed to determine why previously implemented vaccination strategies have had limited success on the control of H9N2 AIV in domestic poultry.

Mathematical models of within-host dynamics can improve our understanding of host-pathogen dynamics at the cellular level, aid in the optimization of interventions (such as vaccination) against AIV, and contribute to the development of more targeted experimental studies (Beauchemin et al., 2009; C.; Beauchemin & Handel, 2011; Cao et al., 2015; Handel, Longini, & Antia, 2007; Manchanda et al., 2014; Pawelek et al., 2012; Perelson, 2002; Saenz et al., 2010; Smith & Perelson, 2011). However, little work has been done using these models to examine the cell-level mechanisms of H9N2 AIV infections in chickens (Hagenaars et al., 2016). A model that can reproduce respiratory and cloacal virus shedding patterns, as well as host antibody responses would be a valuable tool for further examining a range of vaccine candidates and novel vaccination strategies. Within-host modelling can also offer the benefit of assessing the longer-term implications of different vaccination strategies, which is difficult to do experimentally. This study aimed to use a within-host model to evaluate the contributions of three different type-I IFN pathways for the control of respiratory shedding from H9N2 AIV infected chickens. Additionally, a model of the antibody response resulting from different vaccination strategies over the average lifespan of broiler chickens and laying hens will be assessed.

2. Materials and methods

2.1. Experimental data

The empirical data describing respiratory virus shedding originated from an experimental study of A/turkey/Wisconsin/1/1966 (H9N2) infection in chickens (Yitbarek et al., 2018). Full details of the experimental design can be found in Yitbarek et al., 2018. Briefly, layer-type chickens were challenged with H9N2 virus via the oral-nasal route with 400 μl of 107 tissue culture infectious dose 50 (TCID50/ml) at 17 days of age (n = 17). Oropharyngeal swabs were collected at 1, 3, 5, 7, and 9-days post-infection (dpi) to determine virus shedding using TCID50 (Table A1). Experimental data on oropharyngeal viral shedding were reported as log10 TCID50/ml. It is assumed that the TCID50/ml measured from oropharyngeal swabs is proportional to virus production at the site of respiratory infection. Type-I interferon (IFN-α and -β) expression in lung and tracheal tissues was measured using quantitative real-time PCR relative to β-actin expression at 12 h, 24 h, and 36 h post-infection (Fig. A1).

Additional experimental data were obtained in order to parameterize the antibody production component of the model. These data came from three studies of a novel candidate vaccine comprised of whole inactivated A/Turkey/Wisconsin/1/1966 H9N2 virus adjuvanted with 2 μg of CpG ODN 2007 (Astill, Alkie, Yitbarek, Taha-Abdelaziz, Bavananthasivam, et al., 2018; Astill, Alkie, Yitbarek, Taha-Abdelaziz, Shojadoost, et al., 2018; Singh, Alkie, Hodgins, NagyShojadoost, & Sharif, 2015). In these experimental studies, 15 μg of inactivated H9N2 virus was administered intramuscularly (IM) at 7 and 21 days post-hatch (dph) (Astill, Alkie, Yitbarek, Taha-Abdelaziz, Bavananthasivam, et al., 2018; Astill, Alkie, Yitbarek, Taha-Abdelaziz, Shojadoost, et al., 2018; Singh et al., 2015). Each vaccine dose corresponds to ~960 hemagglutinin (HA) units (Astill, Alkie, Yitbarek, Taha-Abdelaziz, Bavananthasivam, et al., 2018). Across these three experimental studies, serum hemagglutination inhibition (HI) antibody titres were measured from 7 to 42 dph, every 7 days, with a total of 132 sampled timepoints. The HI titres were expressed as the log2 of the reciprocal of the highest serum dilution resulting in complete HA inhibition of chicken red blood cells (RBCs). Average HI antibody titre for each day sampled was used for model parameter estimation. All experimental procedures listed here were approved by the University of Guelph Animal Care Committee and conducted according to specifications of the Canadian Council on Animal Care (AUP #3284).

2.2. A within-host model of respiratory virus shedding

A previously established system of ordinary differential equations (ODEs) was used to assess the contributions of three proposed type-I IFN pathways on respiratory virus shedding (Appendix A) (Xie et al., 2020). Model equations were solved using RStudio (deSolve package) over a period of 9 days post-infection (dpi) to match experimental virus shedding data (Soetaert, Petzoldt, & Setzer, 2010). Briefly, the model compartments represent host target (T) and infected (I) cells, as well as infectious virus in the respiratory tract (V) (Fig. 1). Virus particles can infect susceptible host cells at a certain rate, after which the host cell becomes actively infected and produces viral progeny. The type-I IFN pathways examined were represented by an additional type-I IFN compartment (FR). We assumed that type-I IFN may affect virus production in three ways (Cao et al., 2015; Saenz et al., 2010). Pathway 1 represents the induction of a refractory cell state in host cells, in which target cells are no longer susceptible to infection but that they may revert back to the susceptible state after a period of time. Pathway 2 involves the activation of Natural Killer (NK) cells which facilitates the destruction of infected target cells. Intracellular anti-viral pathways of type-I IFNs reduce the capacity of viruses to replicate in host cells is represented by Pathway 3.

Fig. 1.

Fig. 1

A simplified model schematic representing three type-I IFN pathways examined in this study. 1) Induction of a refractory cell state, 2) activation of Natural Killer (NK) cells leading to destruction of infected cells, and 3) reduction of virus replication in infected cells (type-I IFN sensitivity). An eclipse phase was also tested in the model in addition to account for the effect of a cellular latent period (blue dotted lines). The effect of vaccination on the production of neutralizing antibodies was also examined (purple dashed lines).

Refractory host cells that were associated with Pathway 1 are represented by a separate compartment (R), and these cells may transition back into susceptible host cells at a given rate. Parameters associated with the destruction of infected cells and virus sensitivity to intracellular type-I IFN pathways are used in the V compartment to represent Pathways 2 and 3. In order to compare the relative contributions of these proposed mechanisms, the three type-I IFN pathways were assessed individually as well as in combination with each other, for a total of 7 models tested with each representing a different hypothesis about the immune response in chickens (Table 1). The parameter values for all models were determined using 10,000 iterations of non-linear least-squares regression to minimize the sum of the squared errors (SSE) between the experimental data and the model predictions of viral TCID50/ml, where all parameter values were constrained to be positive. To avoid over-fitting the model, parameters not expected to vary between human and chicken AIV infections were held constant (Table 2) (Cao et al., 2015). Initial conditions were T = 1010, V = 2900, and I Created by potrace 1.16, written by Peter Selinger 2001-2019 F=0 (Hagenaars et al., 2016; Hylka & Doneen, 1983). This study also examined the effect of a cellular latent period on model fit, which was represented by an eclipse phase compartment (E). Exposed susceptible cells may move into the eclipse phase compartment at a certain rate, and subsequently become actively infected. Details of all the biological pathways examined and the associated equations for each model can be found in Appendix A and Table A. 2. The model with the lowest corrected Akaike Information Criterion (AICc) was considered to be the best-fitting model and explanation for the observed data Eq. (A9) (Holder & Beauchemin, 2011). The nAICc was calculated for all models, where nAICc > 10 indicates sufficient evidence that a model does not fit the experimental data better than the model with lowest AICc, or AICcmin Eq. (1) (Burnham, Anderson, & Huyvaert, 2011). Latin hypercube sensitivity analysis (LHS) was performed for all parameters which were fit to experimental data in this model of respiratory virus shedding. Partial Rank Correlation Coefficients (PRCCs) were used to identify the parameters the model outcome is most sensitive to.

nAICci=AICciAICcmin. (1)

Table 1.

The relative contributions of three type-I interferon (IFN) pathways on respiratory shedding of H9N2 avian influenza (Fig. 1). The best-fit model of respiratory H9N2 avian influenza virus shedding was determined based on the lowest corrected Akaike Information Criterion (AICc). A nAICc >10 indicates that there is no evidence to support better model fit to experimental data.

Model IFN Pathways SSE AICc nAICc
3 Pathway 3 1.34E+07 50.66 /
3E Eclipse Pathway 3 1.08E+07 52.91 2.25
2 Pathway 2 3.84E+07 55.94 5.28
7E Eclipse Pathways 2 + 3 1.54E+07 56.71 6.05
7 Pathways 2 + 3 3.58E+07 58.91 8.25
6 Pathways 1 + 3 2.02E+07 59.39 8.73
5E Eclipse Pathways 1 + 2+3 1.51E+07 59.59 8.93
6E Eclipse Pathways 1 + 3 1.54E+07 61.85 11.19
2E Eclipse Pathway 2 6.47E+07 61.88 11.22
4E Eclipse Pathways 1 + 2 3.23E+07 62.7 12.04
4 Pathways 1 + 2 1.04E+08 67.57 16.91
5 Pathways 1 + 2+3 8.80E+07 67.7 17.04
1 Pathway 1 1.40E+08 67.73 17.07
1E Eclipse Pathway 1 1.41E+08 69.11 18.45

Table 2.

Parameters used in all models, their interpretations, and the method used to obtain parameter values. Ordinary differential equations used in the model can be found in Appendix A. Parameters which were not fit to experimental data were derived from the literature. Antibody clearance rate is based on a chicken IgY antibody half-life of 4.1 days. Parameters expected to differ from human infection kinetics were fit to experimental data.

PARAMETER: RATE INTERPRETATION VALUE REFERENCE
β: infection Fit /
ε: eclipse phase Fit /
α: virus production Fit /
σ: virus clearance 20d−1 Cao et al. (2015)
ρ: virus sensitivity to type-I IFN Fit /
μ: infected cell lysis Fit /
ϕ: refractory cells become susceptible again 0.05d−1 Cao et al. (2015)
ω: susceptible cells become refractory cells Fit /
D: refractory cell death 0.01d−1 Cao et al. (2015)
θ: IFN induced infected cell death 3d−1 Cao et al. (2015)
δ: type-I IFN production Fit /
λ: type-I IFN clearance 2d−1 Cao et al. (2015)
π: antibody clearance 0.17d−1 Härtle et al. (2013)
γ: antibody production post-vaccination Fit /
σV: vaccine HA unit clearance Fit /

2.3. Antibody production in response to vaccination

To simulate the effects of vaccination on the host adaptive immune response, two compartments were used to represent vaccination (V) and neutralizing antibodies (N). Vaccination was introduced into the model system through the addition of 960 HA units/vaccine (based on an experimental vaccine), which is cleared at rate σV Eq. (2). The vaccination event causes the production of virus neutralizing antibodies (γ), which are also cleared over time (π) (Eq. (3), Fig. 1). The parameters related to vaccine clearance and antibody production (σV and γ respectively) were fit to experimental HI antibody titre data post-vaccination using biologically plausible ranges based on results from Cao et al. (2015). The antibody clearance rate π (0.17d−1) was based on the previously reported chicken IgY half-life of 4.1 days (Härtle, Magor, Göbel, Davison, & Kaspers, 2013; Leslie & Clem, 1970; Watanabe, Kobayashi, & Isayama, 1975). For a prime-boost vaccination strategy, 960 HA units were introduced in the model on 7 dph then again on 21 dph. For a single vaccination strategy, 960 HA units were introduced on 7 dph only. Based on the average maximum lifespan of a broiler chicken (49 days), we examined the effect of the timing of the booster administration at 21, 28, and 35 days post-hatch (dph) after a primary vaccination at 7 dph (Bennett et al., 2018).

Laying hens generally have a maximum lifespan of 490 days, therefore antibody titre resulting from booster administration at 21, 150, and 250 dph was assessed (Mcwhorter & Chousalkar, 2018).

dVdt=σVV, (2)
dNdt=γVπN. (3)

2.4. Model assumptions

The initial value of virus titre in the model on 1 dpi was set to the lower range of the observed experimental data, as the measure of titre using an oropharyngeal swab so soon after virus inoculation through the oral-nasal route may result in a higher virus titre than expected. Non-infectious particles were assumed to be negligible and are not considered in the model. Additionally, the model did not include the loss of virus due to the infection of host cells, or the re-generation of target cells since the timescale of infection considered is relatively short (Baccam, Beauchemin, Macken, Hayden, & Perelson, 2006; Ciupe & Heffernan, 2017; Hagenaars et al., 2016). Type-I IFNs play a dominant role in the early anti-viral response, and the model was simplified so that the contributions of other IFNs are not considered (Kreijtz et al., 2011; Sanders, Doherty, & Thomas, 2011; Santhakumar, Rubbenstroth, Martinez-Sobrido, & Munir, 2017). Lymphocytes such as B and T cells play important roles in the adaptive immune response, however, due to a lack of chicken specific experimental data, model complexity was limited to only represent those components for data was available, and which are necessary to describe the antibody response. Hemagglutination-inhibition antibody titre is accepted as an appropriate indicator of vaccine efficacy, therefore the production and clearance of neutralizing antibodies was assumed to be proportional to HI antibody titres. HI antibody titres >5.3 (log2) were considered protective, which is equivalent to a 1:40 dilution in the HI assay (Trombetta & Montomoli, 2018). Although HI titres were used in this study, assessment of neutralizing antibody titres would be a more accurate representation of vaccine efficacy. Infection was not introduced into the model when estimating parameters related to antibody production and clearance, and therefore the adaptive response shown here is due to vaccination alone.

3. Results

3.1. Type-I IFN mechanisms of virus suppression

Three major type-I IFN antiviral pathways were systematically examined based on their effects on virus shedding from the respiratory tract of infected chickens. When comparing between model structures that only included type-I IFN pathways and no eclipse phase (Models 1–7), models of IFN pathways 2, 3, and 2 + 3 had the lowest AICc of all models evaluated (AICc2 = 55.94, AICc3 = 50.66, AICc7 = 58.91) (Table 1). Model 3 had the lowest AICc and therefore, it was used to calculate the nAICc to assess the fit of the other models. A model that included type-I IFN pathway 3 as well as an eclipse phase was the most consistent with the observed experimental data compared to all other eclipse phase models tested (nAICc = 2.25). Model 3E also had the lowest nAICc compared to all other models. Compared to Model 3, Model 3E underestimated respiratory virus titres on 5 and 9 dpi, and overestimated virus titre on 7 dpi (Fig. 2). Despite having a lower SSE compared to Model 3, the AICc of Model 3E was higher as a result of having an additional model parameter (Model 3 SSE = 1.34E+07, Model 3E SSE = 1.08E+07) (Burnham et al., 2011). Seven of the tested models resulted in a calculated nAICc >10, which were Models 1, 1E, 2E, 4, 4E, 5, and 6E. This indicated that there was sufficient evidence to suggest poor model fit to the experimental data compared to Model 3. Models 1 and 4 generated a very poor fit to the data compared to Model 3, even without the penalty of an additional eclipse phase parameter. These models corresponded to type-I IFN pathways 1 and 2, induction of a refractory cell state, and activation of NK cells leading to infected cell death. Based on these results, intracellular type-I IFN anti-viral pathways (virus sensitivity to type-I IFNs) appear to be the most important mechanism to consider in models of H9N2 AIV respiratory infection in chickens (Model 3, Eq. (4)). The best-fit value of the IFN sensitivity parameter was ρ=2.28 (Table 3). Model output demonstrated type-I IFN response rapidly increased post-infection and peaked on 2 dpi, then was progressively cleared until the end of the experimental period (Fig. A1). Unfortunately, experimental type-I IFN response relative to β-actin expression was only measured up until 1.5 dpi and was therefore not directly comparable to our model output. Other parameter values resulting from the best-fit model of respiratory H9N2 virus shedding in chicken hosts can be found in Table 3. Sensitivity analysis of the five parameters fit to experimental data in Model 3 demonstrated that the rate of cell infection by virus (β) has the greatest effect on the resulting virus titre (Fig. 3).

dTdt=βVT, (4)
dIdt=βVTμI,
dVdt=αI1+ρFσV,
dFdt=δIλF.

Fig. 2.

Fig. 2

Estimated virus shedding output of two models with lowest corrected Akaike Information Criterion (AICc) compared to average experimental oropharyngeal virus titres (TCID50/ml). The best fit model incorporates virus sensitivity to intracellular type-I interferon effects (Pathway 3) and does not include the eclipse phase.

Table 3.

Parameter values from the best-fit model of respiratory H9N2 AIV shedding and neutralizing antibody response as a result of vaccination.

Parameter Interpretation Value Units
β Rate of infection of susceptible target cells 9.74e-05 uV−1d−1
α Rate of virus production 0.21 uVuT−1d−1
ρ Sensitivity of virus production to type-I IFN response 2.28 /
μ Death rate of infected cells 1.96 d−1
δ Rate of type-I IFN production 1.13e-07 uFuT−1d−1
γ Rate of antibody production post-vaccination 8.29e-04 uAd−1
σV Rate of vaccine HA unit clearance 3.42e-03 uAd−1

Fig. 3.

Fig. 3

Sensitivity analysis of five parameters used to fit a within-host model of H9N2 avian influenza virus infection in the chicken respiratory tract. Based on the Partial Rank Correlation Coefficient (PRCC), β (rate of infection in host cells) is the most influential parameter in the model.

3.2. Antibody response to single and double vaccination

The best-fit values of parameters related to vaccine clearance and antibody production as a result of vaccination on 7- and 21-dph can be found in Table 3. Based on experimental data resulting from a prime-boost vaccination strategy, the estimated rate of antibody production post-vaccination was 8.29e−4 d−1, and the clearance rate of vaccine HA units was 3.42e−3 d−1. The model slightly overestimated antibody production on 14 and 21 dph, and underestimated antibody production on 28 and 35 dph (Fig. 4). The model showed that antibody production resulting from a prime-boost vaccination strategy reached a protective titre (>5 log2) between 21 and 28 dph, shortly after booster administration. Model predicted antibody production followed the pattern of experimentally measured HI antibody titre and reached a titre of 8.48 by 42 dph. In comparison, the single vaccination strategy produced a peak antibody titre of ~4.3 by 28 dph which was maintained until 35 dph, after which the titre lowered to 4.12.

Fig. 4.

Fig. 4

Comparison between model predicted host antibody response to whole inactivated H9N2 virus vaccine and experimental data. Shown are experimental hemagglutinin inhibition (HI) antibody titre (log2) (standard deviation), antibody response to single vaccination at 7 dph (orange dashed line), and antibody response to a prime-boost vaccination on days 7 and 21 dph (green dot-dash line). A seroprotective antibody titre (~5.3 log2) is represented by the grey dotted line.

For the lifespan of broiler chickens, the model suggested that regardless of the timing of booster administration, the antibody titre resulting from each scenario on 49 dph would be protective (8.28–8.29) (Fig. 5). A single vaccine dose was not able to induce protective antibody titres, and delayed booster administration resulted in a longer period of low, non-protective, antibody titres. In the case of layer hens, similar titres were observed between the three scenarios by 490 dph ranging from 1.89 to 3.07, none of which would be considered seroprotective (Fig. 5). Additionally, later administration of booster vaccine (150 and 250 dph) also resulted in reduced maximum titre compared to early administration of booster vaccine (21 dph). Earlier administration of the booster vaccine appeared to provide the greatest length of time in which the vaccinated chickens had protective antibody titres.

Fig. 5.

Fig. 5

Antibody titre (log2) based on timing of booster administration in broiler chickens and laying hens compared to single vaccination at 7 days post-hatch (dph). A Booster vaccine administered to broiler chickens at 21 (green triangle), 28 (purple diamond), and 35 (grey circle) dph. B Booster vaccine administered to laying hens at 21 (green triangle), 150 (purple diamond), and 250 (grey circle) dph. A seroprotective antibody titre (~5.3 log2) is represented by the black dashed line.

4. Discussion

4.1. Model of respiratory virus shedding

It is important to consider the effect of the host innate immune response on virus shedding to gain a better understanding of H9N2 AIV infection in chickens as a whole. Previous work has focused on gastrointestinal cellular dynamics however, no work has examined the dynamics associated with respiratory infections (Xie et al., 2020). This study investigated the effect of the three main type-I IFN pathways, and a cellular eclipse phase on respiratory virus shedding in LPAI H9N2 infected chickens. As type-I IFNs of the innate immune response are the main source of anti-viral mechanisms in the early stages of infection, three major mechanisms in which type-I IFNs can reduce virus shedding from the host were examined. Based on the model results, respiratory virus shedding is mainly reduced by intracellular type-I IFN effects (Pathway 3) which inhibit virus replication in H9N2 AIV infected respiratory epithelial cells. This does not imply that other type-I IFN pathways do not play important roles in early infection, rather it suggests that when all pathways are considered, the intracellular anti-viral effects may be most effective in controlling virus replication in the cells of the respiratory system. However, the calculated nAICc values suggest that a model that considers only the induction of a refractory cell state, or this mechanism along with the activation of NK cells, is not sufficient to replicate the observed experimental virus shedding data. The present results were compared to those from a previous study that investigated the relative contributions of these three type-I IFN pathways in reducing cloacal virus shedding in infected chicken gastrointestinal cells (Xie et al., 2020). Firstly, the rate at which host cells are infected and the rate of type-I IFN production from infected cells is similar between both respiratory and gastrointestinal models. In contrast, the rate of virus production and type-I IFN sensitivity appear higher in the respiratory model compared to the gastrointestinal model. This implies that although the virus replicates efficiently in chicken respiratory epithelial cells, this effect may be mitigated by increased sensitivity to type-I IFN anti-viral effects (Gu, Xu, Wang, & Liu, 2017; Pusch & Suarez, 2018; Sun & Liu, 2015). Finally, while the inclusion of an eclipse phase was able to better reproduce cloacal virus shedding patterns, it did not improve the model fit to the respiratory virus shedding data. Experimental respiratory shedding patterns generally show a peak in virus shedding on 2–3 dpi, whereas cloacal virus shedding peaks closer to 5 dpi. A comparison of the SSE and nAICc between Models 3 and 3E suggest that although a cellular latent period may be biologically relevant, it is not necessary to replicate the observed pattern of respiratory virus shedding. In addition to Model 3E, several models examined in this study also have nAICc values < 10 compared to Model 3 (Table 1). This result further confirms that although intracellular type-I IFN anti-viral effects are necessary in a model of H9N2 AIV infection in chickens, other biological pathways are also involved in virus clearance and shedding patterns. Model 3 was identified to be the most parsimonious model that best fits the experimental data based on AICc, but these results also demonstrate that other biological pathways may be included in the model without compromising model fit to the data. Model 3 output demonstrates a peak in type-I IFN response on 2 dpi, which is one day prior to peak virus shedding (Fig. A1). Although type-I IFN experimental data was not collected beyond 1.5 dpi, the model output is consistent with previous work indicating that type-I IFN response generally peaks close to the time of peak virus shedding (Baccam et al., 2006). The model was found to be most sensitive to parameter β, which is an expected result as increasing the rate at which susceptible host cells are infected is directly related to the production of new virus particles.

4.2. Antibody production following vaccination

Antibody titre, activation of host adaptive immune system cells, and the overall reduction in virus shedding in infected hosts are all important metrics of vaccine protection. In this study, different vaccination strategies were assessed based on the production of virus neutralizing antibodies post-vaccination and the potential for reducing virus shedding over the average lifespan of broilers and layers. To do so, a simple model simulation of antibody production was fit to HI antibody titres resulting from a prime and boost vaccination strategy using whole-inactivated A/Turkey/Wisconsin/1/1966 H9N2 virus vaccine adjuvanted with CpG ODN 2007. The results of modelling different vaccination strategies suggested that infection and virus shedding may be avoided by early booster administration, however, a single dose of this candidate vaccine without a booster vaccine would be insufficient to provide protection for the entirety of a layer’s lifespan. Although tedious and expensive to implement, multiple boosters may be necessary to maintain long-term seroprotective titres in layers. It must be noted that the model thus far only considers the residual antibody titre post-vaccination and does not incorporate the activation of the adaptive immune response if the host were to be infected, which would lead to an increase in antibody titres. A future model of H9N2 AIV infection in a vaccinated bird should also account for the adaptive response post-infection. However, the residual antibody titre post-vaccination may play an important role in concert with the host innate immune response to prevent infection of host cells and rapid early clearance of the virus from the host. A low residual antibody titre post-vaccination may allow the virus to infect host cells, leading to virus shedding prior to the activation of the primed adaptive immune response. The results of this study demonstrate that a high residual antibody titre post-vaccination would decrease the risk of infection in other hosts.

Due to the simplicity of the antibody response model proposed here, it was assumed that the vaccine clearance parameter σV inherently represents the time delay between vaccine administration and activation of other cells of the adaptive immune response prior to antibody production. A model that includes B cell (naive and activated) and plasma cell (short-lived and long-lived) populations was also examined however, that model structure did not improve the model fit to the experimental data compared to the model presented in this study (results not shown).

4.3. Model limitations and conclusions

This work represents an additional step towards a whole-host model of H9N2 AIV vaccination and infection in chickens. Early booster administration 2 weeks after the primary vaccination may better protect broiler chickens from infection and subsequent virus shedding, but improved strategies are still required to provide layers with a sufficient level of protection. The model presented here is limited by the lack of experimental data for type-I IFN production through the infection period, activation rates of host adaptive immune system cells, and antibody production from these cells. Although additional data could improve model fit, the presented model output was consistent with the experimental data and has provided new insights regarding chicken immune response dynamics following H9N2 virus infection and vaccination. This study focused on one strain of H9N2 (A/Turkey/Wisconsin/1/1966) and therefore the immune response to infection described here may not be completely applicable to different H9N2 AIV strains. Instead, this model provides the necessary framework to investigate the infection dynamics of different H9N2 virus strains. Future work should be focused on introducing H9N2 AIV infection into a model of a vaccinated host to assess the reduction in respiratory and cloacal virus shedding.

Funding

This research has been supported in part by the University of Guelph’s Food from Thought initiative, thanks to funding from the Canada First Research Excellence Fund (SS and ALG), and also from the Canada Research Chairs program (ALG). Xiao Ting Xie is the recipient of an Ontario Veterinary College Scholarship.

CRediT authorship contribution statement

Xiao-Ting Xie: Conceptualization, Formal analysis, Investigation, Writing – original draft. Alexander Yitbarek: Investigation, Writing – review & editing. Jake Astill: Investigation, Writing – review & editing. Shirene Singh: Investigation, Writing – review & editing. Salah Uddin Khan: Conceptualization, Writing – review & editing. Shayan Sharif: Conceptualization, Writing – review & editing. Zvonimir Poljak: Conceptualization, Writing – review & editing. Amy L. Greer: Supervision, Conceptualization, Methodology, Writing – review & editing, Funding acquisition.

Declaration of competing interest

The authors declare no conflicts of interest.

Acknowledgements

We would like to acknowledge Heba Atalla and Leah R Read for their help in the compilation of experimental data (collected by students in the Sharif lab).

Handling editor; Dr. J Wu

Footnotes

Peer review under responsibility of KeAi Communications Co., Ltd.

Appendix A. Respiratory virus shedding model

Base model and type-I interferon response pathways.

The host target cell population was modelled in terms of susceptible cells (T) and infected cells (I), where β represents the rate at which susceptible cells become infected by the virus, and μ represents the rate at which infected cells die (Eq. (A1) and (A2)). The viral load (TCID50/ml) is modelled by compartment V, where virus production is positively proportional to the number of infected cells at rate α, and virus is cleared at rate σ Eq. (A3).

dTdt=βVT, A1
dIdt=βVTμI, A2
dVdt=αIσV. A3

This base model does not consider the interactions between the innate immune response, the target cell population, and H9N2 AI virus. To assess the effects of type-I IFN pathways on the infection dynamics within a population of susceptible GI tract cells, three previously proposed type-I IFN pathways were examined in the model each representing a different hypothesis (Cao et al., 2015). The first type-I IFN pathway tested was the induction of a refractory state in host target cells by rate ω, refractory cells may revert back to a susceptible state at rate φ, and die at a rate d (Eq. (A4) and (A5)). The second type-I IFN pathway is explored in this model through the parameter θ, which represents infected cell death that is positively proportional to the level of type-I IFN in the system which activates the Natural Killer (NK) cell response Eq. (A6).

dTdt=βVT+φRωFT, A4
dRdt=ωFTφRdR, A5
dIdt=βVTμIθIF. A6

The final type-I IFN pathway was modelled through inclusion of a sensitivity parameter, ρ. This represents the sensitivity of the infectious viral particle population to intracellular type-I interferon anti-viral pathways Eq. (A7). An ODE was included to represent type-I IFN response (F), where δand λrepresent type-I IFN production and clearance rates, respectively Eq. (A8).

dVdt=αI1+ρFσV, A7
dFdt=δIλF. A8

Parameters which were held constant included the rate of virus clearance (σ), the rate at which refractory cells become susceptible again (φ), refractory cell death rate (d), IFN induced infected cell death rate (θ), and the rate of type-I IFN clearance (λ) (Cao et al., 2015). The best-fit model was determined using the corrected AICc for small sample size, where k represents the number of parameters, and n is the number of data points Eq. (A9) (Holder & Beauchemin, 2011).

AICc=2k+nlnSSEn+2kk+1/nk1. A9

Cellular eclipse phase.

To determine the effect of a cellular eclipse phase, compartment E was added to the model to account for a time lag induced by a group of infected cells not yet producing virus particles, which become actively infected at a rate ε (Eq. (A10) and (A11)). In the event a model accounts for both the cellular eclipse phase and pathway 2 (IFN activated NK cells acting on infected host cells), Eq. (A12) is used to represent the infected cells in the system. All models examined in this study and the associated equations can be found in Table A.2.

dEdt=βVTεE, A10
dIdt=εEμI, A11
dIdt=εEμIθIF. A12

Table A.1.

Specific-pathogen free (SPF) chickens were experimentally inoculated with low pathogenic H9N2 avian influenza virus. Oropharyngeal swabs were collected at 1, 3, 5, 7, and 9 days post-infection (DPI) and virus shedding was determined using TCID50. This table reports virus shedding data in exponentiated TCID50/ml.

Host ID TCID50/ml
DPI1 DPI3 DPI5 DPI7 DPI9
T2B1 1.22E+03 3.86E+03 6.85E+03 2.17E+02 6.87E+01
T2B2 3.85E+03 2.17E+03 1.22E+04 6.86E+02 2.17E+02
T2B3 2.17E+03 6.86E+02 6.86E+02 6.87E+01 2.17E+02
T2B4 2.17E+03 2.17E+03 2.17E+03 6.86E+02 2.17E+02
T2B5 6.86E+02 6.86E+02 2.17E+04 6.87E+01 6.87E+01
T2B6 2.17E+04 6.84E+04 2.17E+04 6.87E+01 6.87E+01
T2B7 2.17E+03 2.17E+04 2.17E+04 6.86E+02 2.17E+02
T2B8 6.85E+03 6.84E+04 6.85E+03 6.87E+01 6.87E+01

Table A.2.

Models examined in this study and the associated model equations. All equations and descriptions of the biological pathways considered can be found in Appendix A.

Model IFN Pathways Equations
3 Pathway 3 A1, A2, A7, A8
3E Eclipse Pathway 3 A1, A3, A8, A10, A11
2 Pathway 2 A1, A3, A6, A8
7E Eclipse Pathways 2 + 3 A1, A7, A8, A10, A11
7 Pathways 2 + 3 A1, A6, A7, A8
6 Pathways 1 + 3 A2, A4, A5, A7, A8
5E Eclipse Pathways 1 + 2+3 A4, A5, A7, A8, A10, A12
6E Eclipse Pathways 1 + 3 A2, A4, A5, A7, A8
2E Eclipse Pathway 2 A1, A3, A8, A10, A11
4E Eclipse Pathways 1 + 2 A3, A4, A5, A8, A10, A12
4 Pathways 1 + 2 A3, A4, A5, A6, A8
5 Pathways 1 + 2+3 A4, A5, A6, A7, A8
1 Pathway 1 A3, A4, A5, A6, A8
1E Eclipse Pathway 1 A3, A4, A5, A8, A10, A11

Fig. A1.

Fig. A1

A) Type-I IFN (IFN-α and -β) mRNA expression relative to β-actin from lung and tracheal tissues collected from low pathogenic avian influenza (LPAI) H9N2 infected chickens on 0.5, 1, and 1.5 days post-infection (dpi). B) Model output of type-I IFN response as a result of LPAI H9N2 infection in respiratory epithelial cells of chickens. Due to differences in scaling and lack of quantitative data, model output of type-I IFN response is not directly comparable to experimental results.

References

  1. Alexander D.J. An overview of the epidemiology of avian influenza. Vaccine. 2007;25(30):5637–5644. doi: 10.1016/j.vaccine.2006.10.051. [DOI] [PubMed] [Google Scholar]
  2. Astill J., Alkie T., Yitbarek A., Taha-Abdelaziz K., Bavananthasivam J., Nagy É., et al. Examination of the effects of virus inactivation methods on the induction of antibody- and cell-mediated immune responses against whole inactivated H9N2 avian influenza virus vaccines in chickens. Vaccine. 2018;36(27):3908–3916. doi: 10.1016/j.vaccine.2018.05.093. [DOI] [PubMed] [Google Scholar]
  3. Astill J., Alkie T., Yitbarek A., Taha-Abdelaziz K., Shojadoost B., Petrik J.J., et al. A comparison of toll-like receptor 5 and 21 ligands as adjuvants for a formaldehyde inactivated H9N2 avian influenza virus vaccine in chickens. Viral Immunology. 2018;31(9):605–612. doi: 10.1089/vim.2018.0072. [DOI] [PubMed] [Google Scholar]
  4. Baccam P., Beauchemin C., Macken C.A., Hayden F.G., Perelson A.S. Kinetics of influenza A virus infection in humans. Journal of Virology. 2006;80(15):7590–7599. doi: 10.1128/JVI.01623-05. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Beauchemin C., Handel A. A review of mathematical models of influenza A infections within a host or cell culture: Lessons learned and challenges ahead. BMC Public Health. 2011;11(1):S7. doi: 10.1186/1471-2458-11-S1-S7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Beauchemin C.A.A., McSharry J.J., Drusano G.L., Nguyen J.T., Went G.T., Ribeiro R.M., et al. Modeling amantadine treatment of inuenza A virus in vitro. Journal of Theoretical Biology. 2009;254(2):439–451. doi: 10.1016/j.jtbi.2008.05.031.Modeling. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Bennett C.E., Thomas R., Williams M., Zalasiewicz J., Edgeworth M., Miller H., et al. 2018. The broiler chicken as a signal of a human reconfigured biosphere. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Burnham K.P., Anderson D.R., Huyvaert K.P. AIC model selection and multimodel inference in behavioral ecology: Some background, observations, and comparisons. Behavioral Ecology and Sociobiology. 2011;65(1):23–35. doi: 10.1007/s00265-010-1029-6. [DOI] [Google Scholar]
  9. Cao P., Yan A.W.C., Heffernan J.M., Petrie S., Moss R.G., Carolan L.A., et al. Innate immunity and the inter-exposure interval determine the dynamics of secondary influenza virus infection and explain observed viral hierarchies. PLoS Computational Biology. 2015;11(8) doi: 10.1371/journal.pcbi.1004334. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Ciupe S.M., Heffernan J.M. In-host modeling. Infectious Disease Modelling. 2017;2(2):188–202. doi: 10.1016/j.idm.2017.04.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Evseev D., Magor K. Innate immune responses to avian influenza viruses in ducks and chickens. Veterinary Sciences. 2019;6(5) doi: 10.3390/vetsci6010005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Gharaibeh S., Amareen S. Vaccine efficacy against a new avian influenza (H9N2) field isolate from the Middle East (serology and challenge studies) Avian Diseases. 2016;60(1):407. doi: 10.1637/0005-2086-60.01s1.407. [DOI] [PubMed] [Google Scholar]
  13. Guan J., Fu A.Q., Chan M., Spencer J.L. Aerosol transmission of an avian influenza H9N2 virus with a tropism for the respiratory tract of chickens. Avian Diseases. 2013;57(3):645–649. doi: 10.1637/10486-010913-Reg.1. [DOI] [PubMed] [Google Scholar]
  14. Gu M., Xu L., Wang X., Liu X. Current situation of H9N2 subtype avian influenza in China. Veterinary Research. 2017;48(1):1–10. doi: 10.1186/s13567-017-0453-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Hagenaars T.J., Fischer E.A.J., Jansen C.A., Rebel J.M.J., Spekreijse D., Vervelde L., et al. Modelling the innate immune response against avian influenza virus in chicken. PloS One. 2016;11(6) doi: 10.1371/journal.pone.0157816. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Handel A., Longini I.M., Antia R. Neuraminidase inhibitor resistance in influenza: Assessing the danger of its generation and spread. PLoS Computational Biology. 2007;3(12):2456–2464. doi: 10.1371/journal.pcbi.0030240. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Härtle S., Magor K.E., Göbel T.W., Davison F., Kaspers B. Structure and evolution of avian immunoglobulins. Avian Immunology: Second Edition. 2013:103–120. doi: 10.1016/B978-0-12-396965-1.00006-6. [DOI] [Google Scholar]
  18. Holder B.P., Beauchemin C.A.A. Exploring the effect of biological delays in kinetic models of influenza within a host or cell culture. BMC Public Health. 2011;11(1):S10. doi: 10.1186/1471-2458-11-S1-S10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Hsu A.C.-Y. Influenza virus: A Master Tactician in innate immune evasion and Novel Therapeutic interventions. Frontiers in Immunology. 2018;9(9):1–11. doi: 10.3389/fimmu.2018.00743. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Hylka V.W., Doneen B.A. Ontogeny of embryonic chicken lung: Effects of pituitary gland, corticosterone, and other hormones upon pulmonary growth and synthesis of surfactant phospholipids. General and Comparative Endocrinology. 1983;52(1):108–120. doi: 10.1016/0016-6480(83)90163-6. [DOI] [PubMed] [Google Scholar]
  21. Jegede A., Fu Q., Lin M., Kumar A., Guan J. Aerosol exposure enhanced infection of low pathogenic avian influenza viruses in chickens. Transboundary and Emerging Diseases. 2019;66(1):435–444. doi: 10.1111/tbed.13039. [DOI] [PubMed] [Google Scholar]
  22. Kilany W.H., Bazid A.-H.I., Ali A., El-Deeb A.H., El-Abideen M.A.Z., Sayed M. El. Comparative effectiveness of two oil adjuvant–inactivated avian influenza H9N2 vaccines. Avian Diseases. 2016;60(1s):226–231. doi: 10.1637/11145-050815-reg. [DOI] [PubMed] [Google Scholar]
  23. Kreijtz J.H.C.M., Fouchier R.A.M., Rimmelzwaan G.F. Immune responses to influenza virus infection. Virus Research. 2011;Vol. 162:19–30. doi: 10.1016/j.virusres.2011.09.022. [DOI] [PubMed] [Google Scholar]
  24. Leslie G.A., Clem L.W. Chicken immunoglobulins: Biological half-lives and normal adult serum concentrations of IgM and IgY. PSEBM. 1970 doi: 10.3181/00379727-134-34758. [DOI] [PubMed] [Google Scholar]
  25. Manchanda H., Seidel N., Krumbholz A., Sauerbrei A., Schmidtke M., Guthke R. Within-host influenza dynamics: A small-scale mathematical modeling approach. Biosystems. 2014;118(1):51–59. doi: 10.1016/j.biosystems.2014.02.004. [DOI] [PubMed] [Google Scholar]
  26. Mcwhorter A.R., Chousalkar K.K. Vol. 9. 2018. pp. 1–13. (A long-term efficacy trial of a live , attenuated Salmonella typhimurium vaccine in layer hens). [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Pawelek K.A., Huynh G.T., Quinlivan M., Cullinane A., Rong L., Perelson A.S. Modeling within-host dynamics of influenza virus infection including immune responses. PLoS Computational Biology. 2012;8(6) doi: 10.1371/journal.pcbi.1002588. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Peacock T.P., James J., Sealy J.E., Iqbal M. A global perspective on H9N2 avian influenza virus. Viruses. 2019;11(7):620. doi: 10.3390/v11070620. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Perelson A.S. Modelling viral and immune system dynamics. Nature Reviews Immunology. 2002;2(1):28–36. doi: 10.1038/nri700. [DOI] [PubMed] [Google Scholar]
  30. Pusch E., Suarez D. The multifaceted zoonotic risk of H9N2 avian influenza. Veterinary Sciences. 2018;5(4):82. doi: 10.3390/vetsci5040082. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Saenz R.A., Quinlivan M., Elton D., MacRae S., Blunden A.S., Mumford J.A., et al. Dynamics of influenza virus infection and pathology. Journal of Virology. 2010;84(8):3974–3983. doi: 10.1128/JVI.02078-09. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Sanders C.J., Doherty P.C., Thomas P.G. Respiratory epithelial cells in innate immunity to influenza virus infection. Cell and Tissue Research. 2011;343:13–21. doi: 10.1007/s00441-010-1043-z. [DOI] [PubMed] [Google Scholar]
  33. Santhakumar D., Rubbenstroth D., Martinez-Sobrido L., Munir M. Avian interferons and their antiviral effectors. Frontiers in Immunology. 2017;8 doi: 10.3389/fimmu.2017.00049. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Singh S.M., Alkie T.N., Hodgins D.C., Nagy É., Shojadoost B., Sharif S. Systemic immune responses to an inactivated, whole H9N2 avian influenza virus vaccine using class B CpG oligonucleotides in chickens. Vaccine. 2015;33(32):3947–3952. doi: 10.1016/j.vaccine.2015.06.043. [DOI] [PubMed] [Google Scholar]
  35. Smith A.M., Perelson A.S. Influenza A virus infection kinetics: Quantitative data and models. Wiley Interdisciplinary Review System Biology Medicine. 2011;3(4):429–445. doi: 10.1002/wsbm.129.Influenza. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Soetaert K., Petzoldt T., Setzer R.W. Package deSolve : Solving initial value differential equations in R. Journal of Statistical Software. 2010;33(9):1–25. http://www.jstatsoft.org/v33/i09/paper [Google Scholar]
  37. Sun Y., Liu J. H9N2 influenza virus in China: A cause of concern. Protein and Cell. 2015;6(1):18–25. doi: 10.1007/s13238-014-0111-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Trombetta C.M., Montomoli E. Comparison of hemagglutination inhibition , single radial hemolysis , virus neutralization assays , and ELISA to detect antibody levels against seasonal influenza viruses. Influenza and Other Respiratory Viruses. 2018;12:675–686. doi: 10.1111/irv.12591. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Watanabe H., Kobayashi K., Isayama Y. Peculiar secretory IgA system identified in chickens. II. Identification and distribution of free secretory component and immunoglobulins of IgA, IgM, and IgG in chicken external secretions. The Journal of Immunology. 1975;115(4):998–1001. http://www.ncbi.nlm.nih.gov/pubmed/1176771 [PubMed] [Google Scholar]
  40. Williams A.J.S., G B.R. Interferon-inducible antiviral effectors. Nature Reviews Immunology. 2009;8(7):559–568. doi: 10.1038/nri2314.Interferon-inducible. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Xie X.T., et al. A within-host mathematical model of H9N2 avian influenza infection and type-I interferon response pathways in chickens. Journal of Theoretical Biology. 2020 doi: 10.1016/j.jtbi.2020.110320. [DOI] [PubMed] [Google Scholar]
  42. Yitbarek A., Alkie T., Taha-Abdelaziz K., Astill J., Rodriguez-Lecompte J.C., Parkinson J., et al. Gut microbiota modulates type I interferon and antibody-mediated immune responses in chickens infected with influenza virus subtype H9N2. Beneficial Microbes. 2018;9(3):1–12. doi: 10.3920/BM2017.0088. [DOI] [PubMed] [Google Scholar]

Articles from Infectious Disease Modelling are provided here courtesy of KeAi Publishing

RESOURCES