Modeling Antiretroviral Drug Responses for HIV-1 infected Patients Using Differential Equation Models

Summary

We review mathematical modeling and related statistical issues of HIV dynamics primarily in response to antiretroviral drug therapy in this article. We start from a basic model of virus infection and then review a number of more advanced models with considering, e.g., pharmacokinetic factors, adherence and drug resistance. Specifically, we illustrate how mathematical models can be developed and parameterized to understand effects of long-term treatment and different treatment strategies on disease progression. In addition, we discuss a variety of parameter estimation methods for differential equation models that are applicable to either within- or between-host viral dynamics.

1. The basic HIV viral dynamic models

1.1 Introduction

Human immunodeficiency virus (HIV) primarily attacks the CD4+ T helper cells via, e.g., gp120 binding to the CD4 and CXCR4 receptors. HIV infection typically results in a vast replication of virus during the acute phase. The viral load then becomes much lower and approaches a quasi-steady state, called chronic phase. A balance between high rates of virus production and clearance [168] could explain why the viral load remains remarkably stable on the timescale of weeks [36, 124]. After that, the viral load will increase slowly until the onset of acquired immunodeficiency syndrome (AIDS) [112, 136], whereas the number of CD4+ T cells declines steadily. Eventually, the viral load will increase significantly after the development into AIDS, which is defined if the CD4+ T cell counts fall below 200 cells per μL plasma, or specific diseases in association with HIV infection occur. The typical stages of HIV infection can be found in references [21, 63, 106].

Highly active antiretroviral therapies (HAART), which consist of reverse transcriptase inhibitors and protease inhibitors, are currently the most effective treatment regime for HIV patients in terms of rapidly suppressing HIV viral load below the limit of detection. New and more effective antiretroviral therapies (ART) have turned out to be successful in slowing down the progression to AIDS and improving the life quality of HIV patients. However, ART may not work effectively for certain patients, and the suppression of viral replication by ART is not in itself sufficient for eradicating the virus. There are evidences showing that virus persists despite treatment, or the viral load rapidly rebounds shortly after antiretroviral therapy interruption [58, 99]. Latently infected memory CD4+ T cells (as well as macrophages and dendritic cells) actually become a reservoir of virus. These cells spread out almost everywhere (e.g., blood, lymphoid organs, biotissues, and neuro systems), which makes the eradication of HIV virus extremely challenging.

Mathematical models play a vital role in gaining a quantitative insight into HIV dynamics and pathogenesis [104, 115, 116, 117, 168]. The majority of HIV dynamic models, either deterministic or stochastic, describe the interaction between CD4+ T cells and virions [5, 28, 34, 62, 83, 90, 96, 97, 102, 104, 111, 117, 119, 121, 131, 153, 166, 174, 176]. In particular, many models developed before the mid-1990s, focused on understanding of CD4+ T cell decline [117], partially due to the lack of accurate methods that can measure the number of virus particles in blood. With the development of rapid and sensitive polymerase chain reaction (PCR)-based methods that can quantify genomic viral RNA copies (each virus particle contains two RNA strands), HIV-1 viral dynamics can be understood in a more precise manner and the host-pathogen interaction in HIV-1 patients can be studied quantitatively using modeling. Stochastic models [90, 153] are used to describe the early events after infection; when the numbers of infected cells and virions are small, occasional events such as viral blips [154] or the variability among individuals are of the primary interest. Stochastic models have been used to investigate the effects of increasing variability among viral strains, as a way of escaping immune response, in the progression to AIDS [105, 106, 109, 112]; however, such an approach could be problematic [171]. Stochastic models were also employed to capture the random fluctuations as well as the mean behavior of immune systems to gain insight into treatment-outcome variability [154], or to characterize the dynamics of early infection when virus is released from cells either continuously or in a burst. Deterministic models have been considered by a large number of studies ([117] and references therein) to examine the changes in mean cell numbers and viral loads, or to describe the late stages of infection progression. These models typically account for the effects of drug therapy besides CD4+ T cell and HIV virus kinetics, but there are some models that explicitly take more types of immune cells, such as macrophages and CD8+ cells, into consideration. More complex models considered time-dependent drug efficacy [77, 176, 133] or modeled the time between viral entry and the production and release of new virus particles [28, 83, 166, 179]. There exists numerous work on emergence of HIV drug resistance during antiretroviral treatment [98, 131, 133, 142, 170] and on treatment adherence [66, 133, 146, 175]. In particular, a class of hybrid differential equations has been proposed to model drug behavior [144, 146, 147, 148] or scheduled treatment interruptions [4, 40, 72].

1.2 Selected HIV infection models

Ordinary differential equations (ODEs) have been widely applied in describing the dynamics of immune system. By considering the dynamic changes and interactions among multiple biological components (e.g., virus, CD4+ T cells), ODE models can capture essential behavior of dynamic systems such as nonlinearity and delay. A substantial effort has been devoted to mathematical modeling of HIV dynamics [117], for which Nowak and May [104] provided an excellent and comprehensive review. We start with an overview of the general dynamical features of HIV infection [82]. The model, which is widely adopted to describe the plasma viral load changes in HIV infected individuals, has four state variables: T, the concentration of uninfected target T cells; L, the concentration of latently infected T cells; T^*, the concentration of productive infected T cells; and V, the concentration of free virus particles in the blood. The model structure is

$$\begin{array}{l}\frac{d}{dt}T(t)=f(T)-\mathit{kTV}\hfill \\ \frac{d}{dt}L(t)=\eta \mathit{kTV}-d_{L}L-\nu L\hfill \\ \frac{d}{dt}{T}^{\ast }(t)=(1-\eta )\mathit{kTV}+\nu L-\delta T^{\ast }\hfill \\ \frac{d}{dt}V(t)=N\delta T^{\ast }-cV-i\mathit{kTV}\hfill \end{array},$$ (1.1)

where k is the infection rate, η the fraction of latency, ν the transition rate at which latently infected cells become virus-production active, d_L the death rate of latently infected cells, δ the death rate of infected cells, N (burst size) the total number of virus particles released by a productively infected cell over its lifespan, and c the clearance rate of viral particles. Note that the case i=1 in (1.1) accounts for the loss of a free virus particle once it enters the target cell, but this term can be neglected due to its small magnitude in comparison with the clearance term −cV [117, 166]. Furthermore, the function f (T) denotes the growth rate of uninfected target T cells and can take different forms:

$$f(T)=\{\begin{array}{l}{f}_1(T)=\lambda -dT+pT(1-\frac{T}{T_{max}})\hfill \\ f_2(T)=\lambda -dT+pT(1-\frac{T+L+T^{\ast }}{T_{max}}),\hfill \\ f_3(T)=\lambda -dT\hfill \end{array}$$ (1.2)

where λ denotes the rate at which new CD4+ T cells are produced and d is the per capita death rate of uninfected cells. In the case f = f₁ [117, 121], the healthy T cells are assumed to proliferate exponentially at a rate p until reaching the carrying capacity T_max in the absence of virus particles or infected T cells. Perelson et al. [121] considered the case f = f₂ where the proliferation of L and T^* cells were considered although their proportions are very small. Dropping the growth term leads to the case f = f₃ [104].

Perelson et al. [121] investigated Model (1.1) with i = 0 and f = f₁ or f₂, and their results suggest that if the number of infectious virions produced per actively infected T cell is less than a critical value, N_crit, the free-of-infection state is the only steady state in the nonnegative orthant, and this state is stable. For N > N_crit, the free-of-infection state becomes unstable, but the endemic state can be either stable or unstable within a stable limit cycle. Without considering dynamics of latently infected T cells (i.e., η = 0), De Leenheer and Smith [35] examined Model (1.1) with f = f₁ or f₃. Assuming f(T) is a smooth function and there exists a positive steady state T̄ for variable T such that

$$f(T)>0,0\le T<\overline{T},f(\overline{T})=0,f^{\prime }(\overline{T})<0,\text{and}f(T)<0,T>\overline{T}.$$ (1.3)

That is, homeostasis in a healthy individual is maintained at a steady state T̄. Note Model (1.1) with f = f₁ or f₃ is a competitive system with respect to the cone defined by G ={(T, T^*, V) ∈ R³ : T, V ≥ 0, T^* ≤ 0}, and thus solutions with initial states ordered according to the order of G (i.e., their differences are a vector in G) remain ordered for the backward time [143]. Using theories of the three-dimensional competitive dynamical systems, De Leenheer and Smith [35] conducted a global analysis of viral dynamics. If the basic reproduction number R₀<1, virus will be cleared and the infection will be eradicated; if R₀>1, virus persists in the host, solutions approaching either a chronic steady state (f = f₂) or a periodic orbit (f = f₁), and here R₀ = kT̄(N − i)/c, with i = 0 or 1.

HIV dynamic models that consider intracellular delays are more accurate in terms of representing the real biological processes and pharmacokinetics. To account for the time between viral entry into a target T cell and the production of new virus particles, several delay differential models have been proposed [62, 83, 96, 97, 100, 102, 166, 179]. Herz et al. [62] initially assumed that CD4+ T cells became productively infected τ time units after initial infection and formulated a discrete delayed model. When fitting the proposed model to experiment data, they found that including a delay term will change the estimate of the viral clearance rate (c), but not the loss rate (δ) of productively infected T cells. Mittler et al. [96, 97] assumed that the intracellular delay follows a gamma distribution and proposed the model with a continuous but random delay. Fitting the model to experimental data, Mittler et al. obtained a different estimate for the viral clearance rate. When considering imperfect drug efficacy, Nelson et al. [101] formulated a model with a discrete infection delay and a constant target cell density, and found that the estimated values of the viral clearance rate and the loss rate of productively infected T cells were affected by introducing a delay. Nelson and Perelson [100] further investigated the effects of delays on the estimate of the loss rate of productively infected T cells, which became larger than the values estimated from a non-delay model. These facts suggest that intracellular delays may not be neglected.

A notable feature of delay differential equation models is that delays will generally destabilize an otherwise stable equilibrium and cause sustained oscillations through Hopf bifurcations. In recent studies of within-host viral models with intracellular delays and cell divisions [28, 166], it is shown that sustained oscillation can occur for realistic parameter values. However, Li and Shu [83] considered a viral dynamic model with intracellular delays but without cell divisions, and they showed that sustained oscillations are not possible for their model. Let τ₁ be the time between viral entry into a target cell and the production of new virus particles, and τ₂ be a virus production period for new virions to be produced within and released from the infected cells. Without considering the latent infected CD4+ T cells, Zhu and Zou [179] considered the following delay differential equations

$$\begin{array}{l}\frac{d}{dt}T(t)=\lambda -dT-\mathit{kTV}\hfill \\ \frac{d}{dt}{T}^{\ast }(t)={ke}^{-{\mu }_1{\tau }_1}T(t-{\tau }_1)V(t-{\tau }_1)-\delta T^{\ast }\hfill \\ \frac{d}{dt}V(t)=N\delta e^{-{\mu }_2{\tau }_2}{T}^{\ast }(t-{\tau }_2)-cV\hfill \end{array},$$ (1.4)

where μ₁ is a constant death rate for infected cells that are not producing viruses, and μ₂ may differ from μ₁. Model (1.4) is actually a simplified version of the model proposed by Nelson and Perelson [100]. For Model (1.4), Zhou and Zou [179] obtained a ‘global’ stability result for the infection-free equilibrium if R₀ = λNke^{−μ₁τ₁−μ₂τ₂}/(dc) < 1; otherwise, infection can establish and become locally asymptotically stable. Li and Shu [83] further studied the effects of delay by analyzing Model (1.4) in the presence of only one delay (i.e. assuming τ₂ = 0), and showed that the basic reproductive number completely determines the global dynamics of Model (1.4). In particular, if R₀ = λNke^{− μ₁τ₁}/(dc) ≤1, the infection-free equilibrium is globally asymptotically stable, and hence viruses will be cleared completely; if R₀>1, the unique chronic infection equilibrium is locally asymptotically stable and acts as an attractor. Finally, it should be mentioned that experimental data of CD4+ T cells in vivo may not well support the hypothesis of sustained oscillations [15, 164] although mathematical models with intracellular delays have predicted so [28, 166]. This could be due to data sparsity, and further experimental work could resolve this mystery.

Antigen-specific immunity against HIV infection includes cytotoxic T cells (CTLs) that can kill the infected cells. Let Z(t) denote the concentration of CTLs, then Model (1.1) without considering latently infected cells can be modified as

$$\begin{array}{l}\frac{d}{dt}T(t)=\lambda -dT-\mathit{kTV}\hfill \\ \frac{d}{dt}{T}^{\ast }(t)=\mathit{kTV}-\delta T^{\ast }-{pT}^{\ast }Z\hfill \\ \frac{d}{dt}V(t)=N\delta T^{\ast }-cV\hfill \\ \frac{d}{dt}Z(t)=g(T,T^{\ast },Z)-bZ\hfill \end{array},$$ (1.5)

where p denotes the killing rate and b is the death rate of CTLs. The function g(T, T^*, Z) describes the rate of antigen-specific CTL response [32, 33, 104, 116]. Some investigators [66, 163] assume that the production rate of CTLs depends only on the concentration of the infected cells and chose the linear form g(T, T^*, Z) = ρT^*. Based on this simple model, Arnaout et al. [6] explained the biphasic decay of blood viremia in HIV patients under treatment: viral load decreases quickly while CTLs are abundant, but slowly as CTLs are rare. Nowak and Bangham [110] assumed that the production of CTLs is also dependent on the concentration of CTL themselves, yielding g(T, T^*, Z) = ρT^*Z, and explored the effects of between-subject variation in immune responsiveness on virus load and viral strain diversity. The model in [110] has also been analyzed by Liu [86] and Kajiwara and Sasaki [71]. Culshaw et al. [29] further assumed that the production of CTLs is CD4+ T cell-dependent and accordingly chose the form g(T, T^*, Z) = ρTT^*Z. Assuming that the viral load is proportional to the level of infected cells since free virus is thought to be short lived in comparison with infected cells [6, 118], Culshaw et al. [29] considered the following model

$$\begin{array}{l}\frac{d}{dt}T(t)=\lambda -dT-{\mathit{kTT}}^{\ast }\hfill \\ \frac{dt}{dt}{T}^{\ast }(t)=k^{\prime }{TT}^{\ast }-\delta t^{\ast }-{pT}^{\ast }Z\hfill \\ \frac{d}{dt}Z(t)=\rho {TT}^{\ast }Z-bZ\hfill \end{array},$$ (1.6)

where the ratio k′/k is the proportion of infected cells that survive the incubation period. It was shown that Model (1.6) had up to three equilibria and the local stability of these equilibria was analyzed by Culshaw et al. [29]. Also, Culshaw et al. investigated the optimal control problem in which they maximized the benefit in terms of levels of healthy CD4+ T cells and other immune cells and the systemic cost of chemotherapy. In particular, a self-regulating CTL response, g(T, T^*, Z) = ρ, has also been discussed in [104].

HIV viral species that can successfully evade the host immune response are called “escape mutants”. It is challenging to investigate the dynamics of immune escape. However, there are some studies [5, 34, 104, 111] that made an attempt to tackle this problem by describing antigenic escape from CTLs using simple mathematical models. Specifically, Nowak and Bangham [110] proposed the following model to understand the interplay between selection forces in favor of and against viral mutation:

$$\begin{array}{l}\frac{d}{dt}T(t)=\lambda -dT-T{\sum }_{i=1}^m{k}_i{V}_i\hfill \\ \frac{d}{dt}{T}_i^{\ast }(t)=k_i{TV}_i-\delta T_i^{\ast }-{pT}_i^{\ast }{Z}_i\hfill \\ \frac{d}{dt}{V}_i(t)=N_i\delta T_i^{\ast }-{cV}_i\hfill \\ \frac{d}{dt}{Z}_i(t)=\rho T_i^{\ast }{Z}_i-{bZ}_i\hfill \end{array},$$ (1.7)

where $T_i^{\ast }$ and V_i denote the abundance of infected cells and free virus of type i, respectively, and Z_i denotes the concentration of antigen-specific CTLs against mutant i(i =1, ···, n). Viral mutants differ in their antigenic specificity, the rate at which they infect cells (k_i), and the rate of virus production (N_i). De Boer [34] extended Model (1.7) by including a density-dependent infection term which may better describe the dynamics of acute infection of the viral loads and the immune response. Althaus and De Boer [5] added stochastic events of viral mutation and considered the saturated interaction between infected cells and CTLs according to Michaelis-Menten kinetics. Their computational model of HIV/SIV infection has a broad cellular immune response targeting different viral epitopes. Their simulation shows that a higher degree of immunodominance will result in more frequent immune escape, a reduced control of viral replication, and a substantially impaired replicative capacity of the virus.

2. HIV viral dynamic models with antiretroviral intervention

From Zidovudine (AZT) to dual-drug therapy and then to highly active antiretroviral therapy (HAART), the treatment of HIV/AIDS has been exercised for more than 20 years. HAART uses at least three different antiretroviral drugs (ARVs), typically two nucleoside or nucleotide reverse transcriptase inhibitors (NRTI’s) and one non-nucleoside reverse transcriptase inhibitor (NNRTI) or a protease inhibitor (PI) or another NRTI called abacavir (Ziagen). Reverse transcriptase inhibitors (RTIs) can effectively block the infection of target cells by free virus, while PIs prevent HIV protease from cleaving the HIV polyprotein into functional units. Modeling of antiretroviral intervention can significantly advance our understanding of a variety of biological and clinical questions.

2.1 Modelling constant and variant drug efficacy

Let η_RT be the efficacy of RTIs and η_PI be the efficacy of PIs, the two parameters were assumed to be constant in [117, 100]. Based on Model (1.1), the HIV viral dynamic model with HAART treatment is given as

$$\begin{array}{l}\frac{d}{dt}T(t)=\lambda -dT-(1-{\eta }_{RT})\mathit{kTV}\hfill \\ \frac{d}{dt}{T}^{\ast }(t)=(1-{\eta }_{RT})\mathit{kTV}-\delta T^{\ast }\hfill \\ \frac{d}{dt}{V}_I(t)=(1-{\eta }_{PI})N\delta T^{\ast }-cV\hfill \\ \frac{d}{dt}{V}_{NI}(t)={\eta }_{PI}N\delta T^{\ast }-cV\hfill \end{array}$$ (2.1)

where V_I and V_NI denote the concentration of infectious and noninfectious virus, respectively, and V = V_I +V_NI is the total viral load. Model (2.1) is a simplified version of the model in [117], which considered a more complex form of T cell reproduction. Note that variable V_NI does not show up in the first three equations of Model (2.1), and hence the qualitative dynamics of this model is the same as that of model (1.1) with f = f₃ and i = 0. Based on this model, it is easy to show that the virus-free equilibrium (E₀ = (λ/d, 0, 0)) is locally asymptotically stable if R₀ ≤ 1, while the endemic state

$$E^{\ast }=\left(\frac{\lambda }{R_{0}d},\frac{R_{0}dc}{kN\delta (1-{\eta }_{RT})(1-{\eta }_{PI})},\frac{(R_0-)d}{(1-{\eta }_{RT})k}\right),$$

is globally asymptotically stable when R₀ > 1, where

$$R_0=\lambda Nk(1-{\eta }_{RT})(1-{\eta }_{PI})/(dc).$$ (2.2)

Furthermore, the basic reproduction number is related to the threshold of drug efficacy to achieve virus eradication. For example, if we only consider RT inhibitors, then virus can be eradicated if drug efficacy η_RT is greater than a critical value ${\eta }_{RT}^c=1-dc/(\lambda Nk)$.

Note that the drug concentration in HIV patients is not a constant, but can vary over time, especially during the medication intervals. In fact, once a dose is administered, drug concentration increases rapidly and reaches a peak value, then decreases gradually. When another dose is administered, drug concentration is likely to vary in a similar way [144]. We assume that drugs are taken at time t_k (not necessarily equally spaced) and the effect of the drugs is instantaneous, leading to a system of impulsive differential equations which have a solution that is continuous for t ≠ t_k and undergoes a sudden jump at t = t_k (see [8, 9] for more details on the theory of impulsive differential equations). Let C(t) be the intracellular concentration of the drug and we have

$$\begin{array}{l}\frac{d}{dt}C(t)=h(t),t\ne t_k\hfill \\ C(t_k^{+})=C(t_k^{-})+C^k,t=t_k\hfill \end{array},$$ (2.3)

where h(t) is the drug elimination rate and can be parameterized by either first order elimination kinetics or the Michaelis-Menten elimination kinetics. Considering a fixed dose and a constant time-interval T (that is, t_{k +1} − t_k = T), then the solution C(t) to model (2.3) is a periodic and piecewise continuous function of time t, which can also be explicitly expressed [144, 154]. In most viral dynamic studies, either drug efficacy was assumed constant over treatment time, as in Model (2.1) [117, 174], or antiviral regimens are assumed to be perfect for blocking viral replication [63, 118]. However, in reality, the effect of antiviral treatment will change over time, primarily due to pharmacokinetic variation, fluctuating adherence, the emergence of drug resistant mutations and other factors.

Without considering drug adherence or drug resistance, a simple pharmacodynamic E_max model for dose–effect relationship can be given as [49]

$$E=\frac{E_{max}C(t)}{{EC}_{50}+C(t)}$$ (2.4)

where E_max is the maximal effect that can be achieved, C(t) is the drug concentration, and EC₅₀ is the drug concentration that corresponds to the 50% of the maximal effect. Note that many different forms of E_max have been developed in previous studies; for example, the sigmoid E_max model, the ordinary E_max model, and the composite E_max model [49, 141]. It follows from Models (2.3) and (2.4) that the drug efficacy may be a periodic and piecewise continuous function of time and there are studies on HIV dynamic models with periodic drug efficacy [64, 176, 178]. Let drug efficacies η_RT(t) and η_PI(t) be periodic and continuous (or piecewise continuous) function of time. For system (2.1) with ϖ -periodic drug efficacies η_RT(t) and η_PI(t), Yang and Xiao [176] investigated the treatment dynamics by employing the persistent theory for the periodic system [7, 167]. They defined a threshold parameter, similar to the basic reproduction number, which determines the extinction or persistence of the disease. Their main results showed that the disease-free equilibrium of system (2.1) is globally asymptotically stable if r(Φ_{M (·)} (ϖ)) < 1, while the disease is persistent if r(Φ_{M (·)} (ϖ)) > 1, where r(Φ_{M (·)} (ϖ)) is the spectral radius of Φ_{M (·)} (ϖ), and Φ_{M (·)} (ϖ) is the fundamental solution matrix of the linear and ϖ -periodic differential equation

$$x^{\prime }=M(t)x,M(t)=\left(\begin{array}{ll}-\delta \hfill & \hfill \lambda k(1-{\eta }_{RT}(t))/d\\ N\delta (1-{\eta }_{PI}(t))\hfill & \hfill -c\end{array}\right).$$ (2.5)

Note that the threshold parameter cannot be expressed explicitly, but it can be calculated numerically [167, 176].

2.2 Emergence of drug resistance

Although HAART has proved to be extremely effective in suppressing the plasma viral load in most HIV-1 infected patients down to the detection limit (e.g., 50 RNA copies ml⁻¹) of the standard assay to date [25], drug treatment often fails to achieve virus eradication primarily due to the emergence of drug-resistant mutants [37].

It is widely acknowledged that two reasons can account for the development of HIV drug resistance: the transmission of drug-resistant mutants to susceptible individuals, or the adaptive mutations generated during treatment [16, 128]. Ribeiro and Bonhoeffer [129] calculated the probabilities of both reasons and suggested that under a wide range of conditions, treatment failure is most likely due to the pre-existence of drug-resistant virus before therapy. Bonhoeffer and Nowak [21] showed that, given pre-existence of drug-resistant virus, a more efficient therapy could lead to a greater initial reduction of virus load, but would also cause a faster rise of drug-resistant mutants. A number of mathematical models have been developed to study the effect of ARV drugs on the evolution of drug-resistant HIV mutants. McLean and Nowak [98] examined the competition between drug-resistant and wild-type strains to determine which type of virus will eventually dominate the virus population during the course of AZT treatment. Nowak et al. [108] considered a two-strain model and compared the modeling results with experiment data on the development of drug resistance in patients treated with nevirapine. Kirschner and Webb [77] investigated drug resistance for the case of single drug treatment and compared the treatment outcomes for drug therapies initiated at different CD4+ T cell levels. The effect of an immune response on the emergence of drug resistance was investigated in [142, 170]. Rong et al. [133] proposed a mathematical model including both wild-type and drug resistant strains to understand the mechanism of the emergence of drug resistance during therapy. Let T_s(t) be the concentration of cells productively infected by drug sensitive virus, T_r(t) be the concentration of cells productively infected by drug-resistant virus, and V_s(t) and V_r(t) represent concentrations of drug sensitive and drug-resistant virus, then the model can be given as

$$\begin{array}{l}\frac{d}{dt}T(t)=\lambda -dT-k_s{TV}_s-k_r{TV}_r\\ \frac{d}{dt}{T}_s(t)=(1-u)k_s{TV}_s-\delta T_s\\ \frac{d}{dt}{V}_s(t)=N_s\delta T^{\ast }-{cV}_s\\ \frac{d}{dt}{T}_r(t)={uk}_s{TV}_s+k_r{TV}_r-\delta T_r\\ \frac{d}{dt}{V}_r(t)=N_r\delta T^{\ast }-{cV}_r\end{array}$$ (2.6)

where k_s and k_r denote the rate constants at which uninfected cells T(t) are infected by drug sensitive and drug-resistant virus, respectively; also, u(0 ≤ u < 1) is a rate at which cells infected by the drug sensitive virus become drug-resistant due to viral RNA mutation. Both types of infected cells are assumed to have the same death rate δ. Assume that the drug sensitive and resistant strains have different burst sizes, N_s and N_r, while they can have the same virion clearance rate c. Without antiretroviral intervention, the reproductive ratio for each strain

$$R_s=\lambda k_s{N}_s/(dc),R_r=\lambda k_r{N}_r/(dc)$$ (2.7)

can be obtained. The infection-free steady state E₀ = (λ/d, 0, 0, 0, 0) is locally asymptotically stable if R_s < 1/(1−u) and R_r < 1, and it is unstable if R_s > 1/(1−u) or R_r > 1. The steady state with only drug-resistant virus exists if and only if R_r > 1 and it is locally asymptotically stable if R_r > (1−u)R_s, and is unstable if R_r < (1−u)R_s. The co-existence steady state exists and is locally asymptotically stable if and only if R_s > 1/(1−u) and R_r < (1−u)R_s.

After drug intervention, the reproductive ratios for drug sensitive and resistant strains become

$$R_s^{\prime }=(1-{\epsilon }_{RT}^s)(1-{\epsilon }_{PI}^s)R_s,R_r^{\prime }=(1-{\epsilon }_{RT}^r)(1-{\epsilon }_{PI}^r)R_r,$$ (2.8)

where ${\epsilon }_{RT}^s,{\epsilon }_{RT}^r$ are the efficacies of RTIs and ${\epsilon }_{PI}^s,{\epsilon }_{PI}^r$ are the efficacies of PIs for the drug sensitive strain and drug-resistant strains, respectively. An overall treatment effect for each strain can be defined as follows

$${\epsilon }_s=1-(1-{\epsilon }_{RT}^s)(1-{\epsilon }_{PI}^s),{\epsilon }_r=1-(1-{\epsilon }_{RT}^r)(1-{\epsilon }_{PI}^r).$$ (2.9)

Based on the stability results with ART, there are two threshold values ε₁ and ε₂ for ε_s such that: (i) both the wild type and the drug-resistant strains coexist if drug efficacy ε_s is less than ε₁; (ii) only the drug-resistant virus will persist for ε₁ < ε_s < ε₂; and (iii) both strains will be eradicated if ε_s > ε₂. This indicates that drug resistance is more likely to arise for an intermediate level of treatment effectiveness, at which the reproductive ratios of both strains are close. Furthermore, a pharmacokinetic model including both blood and cell compartments is employed to estimate the drug efficacies against the wild-type and the drug-resistant strains. Simulation results suggest that the perfect adherence to the regimen protocol will well suppress the viral load of the wild-type strain while drug-resistant variants develop slowly. However, an intermediate level of adherence may result in the dominance of the drug-resistant virus several months after the initiation of therapy. This result is similar to that of [145].

Experimental measure of adherence remains problematic and challenging. There are lots of evidence showing that suboptimal adherence is associated with a high risk of developing HIV drug resistance [10, 44, 140, 161] and is one of the major causes of treatment failure [63, 175, 113]. A number of mathematical models have considered the effects of imperfect adherence to drug regimens [45, 64, 122, 146, 161, 172] and see Heffernan and Wahl [60] for a comprehensive overview. A standard definition of adherence and reliable measures of adherence are still lacking. Fortunately, there have been substantial progresses in the two areas within the past few years. Wahl and Nowak [161] considered the outcome of a therapy as a function of the degree of adherence to drug regimen and determined the conditions under which a resistant strain would dominate. Phillips et al. [122] proposed a stochastic model to study the resistance development for different drug use patterns.

The effect of time-varying or constant drug concentrations on HIV dynamics has been modeled by a large number of studies [117, 174]. However, only a handful of studies have modeled the direct interaction between drug concentrations and the dynamics of a pathogen population, examining the necessary conditions for the emergence of drug resistance [95, 144, 145, 147]. In such models, the immune cells infected by virus (e.g., CD4+ T cells) are divided into multiple classes, depending on whether a cell has been infected or has absorbed any of the drugs. An impulsive differential equation can be used to model the change in drug concentration when a new dose is administered. Let R(t) denote the intracellular drug concentration that satisfies model (2.2), T(t) be the population of susceptible (uninfected) CD4+ T cells, T_s(T_r) denote the cells infected by the wild-type (mutant) infectious virus, T_Rs denote the uninfected cells which have absorbed sufficient amount of drugs such that the wild-type strain can be inhibited from replication, but not enough to prevent infection by the mutant strain. These cells may come into contact with the wild-type/mutant strain or the drug. T_Rr represents uninfected cells which have absorbed sufficient amount of drugs such that both wild-type and mutant strains are inhibited. Such cells will not become infected while they remain in this state. These cells will eventually revert back to T_Rs cells if the drug effect wears off, or if such cells undergo apoptosis. The model in [145] is thus given as

$$\begin{array}{l}\frac{dT(t)}{dt}=\lambda -dT-k_s{TV}_s-k_r{TV}_r-\theta k_P{k}_{R}TR+m_{Rs}{T}_{Rs}\\ \frac{{dT}_{Rs}(t)}{dt}=\theta k_P{k}_{R}TR-k_r{T}_{Rs}{V}_r-(d+m_{Rs})T_{Rs}+m_{Rr}{T}_{Rs}-\eta k_Q{T}_{Rs}R\\ \frac{{dT}_{Rr}(t)}{dt}=\eta k_Q{T}_{Rs}R-(d+m_{Rr})T_{Rr}\\ \frac{{dT}_s(t)}{dt}=k_s{TV}_s-d^1{T}_s\\ \frac{{dT}_r(t)}{dt}=k_r{TV}_r+k_r{T}_{RI}{V}_r-d^1{T}_r\\ \frac{{dV}_s(t)}{dt}=\omega N_s{d}_s{T}_s-{cV}_s\\ \frac{{dV}_r(t)}{dt}=\omega N_r{d}_r{T}_r-{cV}_r\\ \frac{{dV}_{NI}(t)}{dt}=(1-\omega )N_s{d}_s{T}_s+(1-\omega )N_r{d}_r{T}_r-{cV}_{NI}\end{array}$$ (2.10)

where k_s(k_r) is the rate at which wild-type (drug-resistant) virus infects T cells, N_s(N_r) is the number of virions produced per infected cell and ω is the fraction of infectious virions produced by an infected T cell. Furthermore, m_Rs and m_Rr are drug clearance rates for intracellular compartments with an intermediate or high drug concentration, respectively. d(d¹) is the death rate of uninfected (infected) CD4+ T cells. Smith and Wahl [145] considered three treatment regimens corresponding to a low, intermediate or high drug level, respectively

$$\theta =\{\begin{array}{ll}0,\hfill & ifR<R_1\hfill \\ 1/k_R,\hfill & if{R}_1<R<R_2\hfill \\ 1/k_P,\hfill & ifR>R_2\hfill \end{array}\mathit{and}\eta =\{\begin{array}{ll}0,\hfill & ifR<R_2\hfill \\ 1,\hfill & ifR>R_2\hfill \end{array}$$ (2.11)

where k_P is the rate at which the drug inhibits the wild-type T cells when drug concentrations are intermediate (R₁ < R < R₂), k_R and k_Q are rates at which the drug inhibits the wild-type and drug-resistant T cells, respectively, when drug concentration is high (R > R₂). By analyzing all possible equilibria (or periodic solution) and their stability for low, intermediate or high drug level, Smith and Wahl [145] predicted that drug resistance might arise at both intermediate and high drug concentrations, whereas at low drug levels resistance would not emerge. Smith [146] used a model similar to (2.10) to determine how many doses can be missed before HIV treatment is adversely affected by the emergence of drug resistance. In [146], the dynamics of drug, and hence adherence, is also modeled by the impulsive differential equations. Here the perfect adherence is that the dosage R^k is not zero at each medication time t = t_k. Otherwise, for example, if R^k = 0, k = 2, 3, 4 indicates 3 doses missed after the first dose. Miron and Smith [95] extended the work from a single drug holiday during any given therapy [146] to the case of more than one drug holidays.

2.3 Short-term HIV dynamics and optimal controls

A large number of HIV dynamic models have been proposed by AIDS researchers [119, 174] to provide a theoretical guidance for development of new HIV treatment strategies. For simplicity, some modeling work focused on short-term dynamics while other models with constant or time-dependent drug efficacy are formulated to explore long-term viral dynamics [66, 69, 80, 170, 175]. Specifically, these short-term models either fit only the early segment of the viral load trajectory [72, 90, 128, 153], or design an optimal drug therapy regime in a short time interval [18, 29, 47, 70, 76, 133, 169].

The control-theory approach has been employed to design an optimal treatment strategy. Such investigations employed various types of mathematical models and different objective functions. For example, the studies in [18, 47, 76, 133] for optimal control of HIV chemotherapy used an objective function based on simultaneously maximizing the CD4+ T cell counts and minimizing the systemic cost of chemotherapy. Generally, drugs efficacies η_RT and η_PI in Model (2.1) are replaced by the control variables u₁ and u₂ (0 ≤ u_i ≤ 1, i = 1, 2) which accounts for reverse transcriptase and protease inhibitors actions, respectively. The task then boils down to the determination of optimal control functions u₁(t) and u₂(t). For example, the objective function can be defined as

$$J={\int }_{t_0}^{t_f}\left\{T(t)-\left[B_1{u}_1^2(t)+B_2{u}_2^2(t)\right]\right\}dt,$$ (3.1)

which is maximized subject to model (2.1). The nonnegative constants B₁ and B₂ represent the desired ‘weights’ of the benefit and the cost, respectively. By applying Pontryagin’s Maximum Principle to the constrained control problem, the optimal controls can be obtained. Joshi [70] considered controls representing immune boosting and viral suppressing drugs. Wein et al. [169] proposed a deterministic control problem which is based on a finite number of virus strains and allows virus mutations. Using numerical simulations, they demonstrated a dynamic strategy that can reduce the total free virions, increase the uninfected CD4+ counts, and postpone the emergence of drug-resistant strains. It should be mentioned that the optimal treatment strategies resulted from these studies are for a finite time window so the conclusions may not be applicable to a long-term treatment problem.

Recently, a handful of stochastic models were formulated to characterize the dynamics of early infection when virus is released from cells [55, 72, 129, 157]. For example, Tuckwell and Le Corefec [27, 156] applied the multi-dimensional diffusion process to model early HIV-1 population dynamics. Tan and Wu [152] developed a 4-dimensional stochastic infection model for HIV and studied the system using Monte Carlo simulations. They noted that there was a positive probability that the virus could be eliminated by the process. Monte Carlo approaches were also used by Kamina et al. [72] and Heffernan and Wahl [61] to study the probability that an infection would not be established after exposure to a pre-specified amount of pathogens. Tuckwell et al. [157] investigated the probability of a viral particle infecting one or more target cells before being cleared. A stochastic model of early infection was developed by Haeno and Iwasa [55] to study the generation of drug-resistant virus based on the assumption of exponential growth of virus. Ribeiro and Bonhoeffer [129] also developed a stochastic simulation model of early infection to determine the best timing to initiate antiretroviral therapy with considering the random generation of drug-resistant mutants. Pearson et al. [114] modeled early infection using a discrete random process, in which both the numbers of virions and infected cells are tracked for a gradual or a burst release of virus. They showed that different viral release patterns lead to different early dynamics (e.g., different probabilities of extinction, different distributions of time to establish infection).

3. HIV viral dynamic models with long-term treatment and different treatment strategies

Most HIV dynamic studies for short-term viral dynamics are under assumptions of ideal patient behavior, treatment potency, and drug susceptibility. Although these studies characterized HIV replication during a short period of antiretroviral (ARV) treatment, the effectiveness of ARV therapies and variations of HIV-1 dynamics during chronic treatment of HIV-1 infection in a more realistic setting have not been carefully studied quantitatively. In particular, there are evidences showing that the viral load trajectory may change its shape in the later stage due to variations in drug resistance, noncompliance or other clinical factors [66]. Hence, it is necessary to develop models to account for, e.g., drug susceptibility and drug adherence to quantify long-term dynamics under ARV therapies.

Note that for HIV subspecies within a host, the genetic diversity is usually notable, corresponding to a great diversity in response to various ARV agents. In clinical practice, genotypic or phenotypic tests can be used to measure the sensitivity of HIV-1 to ARV before a treatment regimen is selected. Molla et al. [98] suggested that the phenotype marker, median inhibitory concentration IC₅₀, can be used to quantify agent-specific drug susceptibility. Wu et al. [64, 66] used the following drug resistance model to approximate the within-host changes over time of IC₅₀

$${IC}_{50}=\{\begin{array}{lll}{I}_0+\frac{I_r-I_0}{t_r}t\hfill & \text{for}\hfill & 0<t<t_r\hfill \\ I_r\hfill & \text{for}\hfill & t\ge t_r\hfill \end{array},$$ (3.2)

where I₀ and I_r are respectively the values of IC₅₀ (t) at the baseline and time point t_r, at which resistant mutations dominate. Huang and Wu [64] investigated the relationship between actual failure time (the time at which the viral growth rate changes from negative to positive) and detectable failure time (the time at which viral load rebounds to above the limit of detection), and obtained an approximately linear relationship which could be used to estimate the actual rebound failure time from the detectable rebound failure time. They examined how different patterns of adherence affect antiviral response. Their results suggest that longer sequences of missed doses increase the chance of treatment failure and accelerate the failure.

Patients may occasionally miss doses or multiple consecutive doses for various reasons such as misunderstanding prescription instructions, serious side-effects and others. The deviation from prescribed dosing affects drug exposure in a predictable way. Wu et al. [64, 66, 175] used the following model to describe adherence

$$A_d(t)=\{\begin{array}{ll}1\hfill & \text{for}{T}_k<t\le T_{k+1},\text{if}\text{all}\text{doses}\text{are}\text{taken}\text{in}[T_k,T_{k+1}]\hfill \\ R_d\hfill & \text{for}{T}_k<t\le T_{k+1},\text{if}100R_d\%\text{doses}\text{are}\text{taken}\text{in}[T_k,T_{k+1}]\hfill \end{array},$$ (3.3)

where 0 ≤ R_d <1 (d = 1, 2) with R_d denoting the adherence rate for drug d during the interval, and T_k denotes the adherence evaluation time at the k-th clinical visit. Wu et al. [172] developed a novel HIV-1 dynamic model with consideration of pharmacokinetics, drug adherence and drug susceptibility to link plasma drug concentration to the long-term changes in HIV-1 RNA observation after initiation of therapy. Their results show that any single factor of pharmacokinetic (PK) adherence measured by pill counts and drug susceptibility does not seem to contribute to long-term virologic response, but their combinations in viral dynamic modeling can predict virologic response.

The inherent risks and problems associated with HAART such as adverse effects, imperfect adherence and drug resistance have led to development of strategies of scheduled treatment interruptions (STIs); in particular, CD4+ T cell count-guided STIs might provide a good strategy to address these problems [4, 40, 72]. Several clinical studies have been done to compare STI strategies with continuous antiretroviral therapy, but unfavorable results have been reported [4, 40, 72]. Thus, more careful studies with appropriate quantitative approaches are needed to resolve the problem.

HIV dynamic models and control theory have been used together to study both non-adaptive and adaptive STI strategies [2, 54, 134]. Adams et al. [2] considered a complicated HIV dynamic model and used control theory to design non-adaptive STI strategies that involve several short-term interruptions after infection. They showed that such strategies could lead to a long-term control of virus in some patients. Hadjiandreou et al. [54] formulated a non-adaptive STI therapy as a dynamic programming problem and showed that STIs could control disease progression. Tang et al. [148] proposed models to use CD4+ T cell counts as a guide to start or halt the therapy. The proposed models extended the classical HIV dynamic models [104, 117] to the piecewise dynamic model and were sued to explore potential explanations why the controversial results were obtained from different clinical studies [148]. The recent clinical studies [40, 72] initiated the ARV therapy once the CD4+ T cell counts dropped below a lower threshold (denoted by C_TH, say 200 or 350 cells/ml) and suspended the treatment once the CD4+ T cell counts increased above an upper threshold (denoted by C^TH, say 600 cells/ml or more), where [C_TH, C^TH] is called the threshold window of treatment decision. In the absence of considering dynamics of noninfectious virus or latently infected individuals, we can extend Model (2.1) to include a threshold window as follows

Drug-off state

$$\text{Drug}-\text{off}\text{state}\{\begin{array}{l}\frac{d}{dt}T(t)=\lambda -dT-{\mathit{kTV}}_I\hfill \\ \frac{d}{dt}{T}^{\ast }(t)={\mathit{kTV}}_I-\delta T^{\ast }\hfill \\ \frac{d}{dt}{V}_I(t)=N\delta T^{\ast }-{cV}_I\hfill \end{array}\text{until}T+T^{\ast }\downarrow C_{TH}$$ (3.4)

Drug-on state

$$\text{Drug}-\text{on}\text{state}\{\begin{array}{l}\frac{d}{dt}T(t)=\lambda -dT-(1-{\eta }_{RT}){\mathit{kTV}}_I\hfill \\ \frac{d}{dt}{T}^{\ast }(t)=(1-{\eta }_{RT}){\mathit{kTV}}_I-\delta T^{\ast }\hfill \\ \frac{d}{dt}{V}_I(t)=(1-{\eta }_{PI})N\delta T^{\ast }-{cV}_I\hfill \end{array}\text{until}T+T^{\ast }\uparrow C^{TH}$$ (3.5)

For the drug-off and drug-on states, there are steady states $E_{\mathit{off}}=({\overline{T}}_{\mathit{off}},{\overline{T}}_{\mathit{off}}^{\ast },{\overline{V}}_{\mathit{off}})$ and $E_{on}=({\overline{T}}_{on},{\overline{T}}_{on}^{\ast },{\overline{V}}_{on})$ if the reproduction number for each subsystem is greater than one. And obviously we have ${\overline{T}}_{on}+{\overline{T}}_{on}^{\ast }>{\overline{T}}_{\mathit{off}}+{\overline{T}}_{\mathit{off}}^{\ast }$. Note that according to Models (3.4) and (3.5), for a new patient, if T(t₀) + T^* (t₀)>C^TH, then this patient could be in the drug-off state; if T (t₀) + T^* (t₀)<C_TH, he/she could be in the drug-on state; and if C_TH < T(t₀) + T^* (t₀)<C^TH, he/she may be either in the drug-off state or in the drug-on state, depending on whether the trend of CD4+ T cell counts is increasing or decreasing. Furthermore, if ${\overline{T}}_{on}+{\overline{T}}_{on}^{\ast }>C^{TH}$, the system reverts to the drug-off state before it can reach the equilibrium E_on; while if ${\overline{T}}_{\mathit{off}}+{\overline{T}}_{\mathit{off}}^{\ast }<C_{TH}$, the system must switch to the drug-on state before it can reach the steady state E_off. The two observations together suggest that the system will persistently alternate between drug-off state and drug-on state if ${\overline{T}}_{\mathit{off}}+{\overline{T}}_{\mathit{off}}^{\ast }<C_{TH}<C^{TH}<{\overline{T}}_{on}+{\overline{T}}_{on}^{\ast }$. Generally we assume that ${\overline{T}}_{on}+{\overline{T}}_{on}^{\ast }>C_{TH}$ and ${\overline{T}}_{\mathit{off}}+{\overline{T}}_{\mathit{off}}^{\ast }<C^{TH}$. According to the relationships among ${\overline{T}}_{on}+{\overline{T}}_{on}^{\ast },{\overline{T}}_{\mathit{off}}+{\overline{T}}_{\mathit{off}}^{\ast }$, the lower threshold C_TH and the upper threshold C^TH, we have four possible cases:

$$\begin{array}{l}\text{Case}1:{\overline{T}}_{\mathit{off}}+{\overline{T}}_{\mathit{off}}^{\ast }<C_{TH}<C^{TH}<{\overline{T}}_{on}+{\overline{T}}_{on}^{\ast }\\ \text{Case}2:{\overline{T}}_{\mathit{off}}+{\overline{T}}_{\mathit{off}}^{\ast }<C_{TH}<{\overline{T}}_{on}+{\overline{T}}_{on}^{\ast }<C^{TH}\\ \text{Case}3:C_{TH}<{\overline{T}}_{\mathit{off}}+{\overline{T}}_{\mathit{off}}^{\ast }<{\overline{T}}_{on}+{\overline{T}}_{on}^{\ast }<C^{TH}\\ \text{Case}4:C_{TH}<{\overline{T}}_{\mathit{off}}+{\overline{T}}_{\mathit{off}}^{\ast }<C^{TH}<{\overline{T}}_{on}+{\overline{T}}_{on}^{\ast }\end{array}$$

For Case 1 we can easily see that both equilibria E_on and E_off are local stable in the given range and are also virtual (see definitions in [12, 46]). It is clear that these stable but virtual equilibria can never be actually reached, which indicates that the system may switch between the drug-off state (3.4) and the drug-on state (3.5) forever. It follows that for the given threshold window, the CD4+ T cell counts fluctuate periodically during the whole treatment period. The variations of healthy and infected CD4+ T cells are also given, respectively, in Fig. 1 (B). Fig. 1(C) shows that the durations of each drug-on/off are stabilized at fixed values and the duration of drug-on is much longer than that of drug-off. Numerical simulations show that the effects of the width of threshold window on the durations of drug-on and drug-off states are complex. The duration of the drug-on state is more sensitive to the variation of thresholds compared with that of the drug-off state [148].

Fig. 1.

Simulation of typical solutions of drug on-off state systems (3.4) and (3.5) with C^TH = 700ul⁻¹, C_TH = 200ul⁻¹. (A) Time trajectory of total CD4+ T cell population; (B) Time trajectory of health CD4+ T cell population (dashed line) and infected CD4+ T cell population (solid line); (C) Durations of drug on and drug off for each drug on-off switch; (D) Phase plane plot of health CD4+ T cell and infected CD4+ T cell populations.

For Case 2, the drug-on equilibrium E_on becomes a regular steady state, which is globally stable for the drug-on system (3.5) only. Fig. 2(A, B) shows that one trajectory (pink curve) initiating from an initial point approaches the regular equilibrium E_on for C^TH = 1400 ul⁻¹, whereas another trajectory (blue curve) starting from the same point oscillates periodically if C^TH = 1300 ul⁻¹. Case 4 is similar to Case 2 except that the real drug-off equilibrium E_off is globally stable for the drug-off system (3.4). The CD4+ T cell counts either approach a certain value (green curve), corresponding to free of therapy, or oscillate persistently (blue curve), corresponding to a drug-on/off treatment, as shown in Fig. 2 (E, F). For Case 3, we have that both equilibria E_on and E_off are regular and globally stable for their own systems. Fig.2 (C, D) shows that the CD4+ T cell counts will approach certain levels represented by the equilibrium E_on or E_off, corresponding to a continuous treatment strategy (pink curve) or free of therapy (green curve), under certain thresholds. It is interesting to note that, similar to Case 1, oscillation of the CD4+ T cell counts can also be observed, suggesting an STI strategy with a threshold window is required (blue curve). This indicates that the STI strategy with different thresholds may result in different treatment regimes such as drug-on/off, continuous therapy, and free of ARV treatment.

Fig. 2.

Numerical simulations of solutions of drug on-off state systems (3.4) and (3.5) with different low and upper threshold values. Case 2: (A–B) Blue cure for the solution with C^TH = 1300ul⁻¹, C_TH = 350ul⁻¹, pink curve for the solution with C^TH = 1400ul⁻¹, C_TH = 350ul⁻¹; Case 3: (C–D) Blue cure for the solution with C^TH = 1300ul⁻¹, C_TH = 150ul⁻¹, pink curve for the solution with C^TH = 1400ul⁻¹, C_TH = 350ul⁻¹, and green curve for the solution with C^TH = 1300ul⁻¹, C_TH = 100ul⁻¹; Case 4: (E–F) Blue cure for the solution with C^TH = 700ul⁻¹, C_TH = 150ul⁻¹, and green curve for the solution with C^TH = 700ul⁻¹, C_TH = 100ul⁻¹.

In summary, the results showed that the CD4+ T cell counts can either fluctuate around the two thresholds or stabilize at an equilibrium for drug-on or drug-off state. This implies that, for a fixed threshold window, whether a patient needs a continuous therapy or STI strategy depends on the initial CD4+ T cell counts of the patient at the treatment starting time. Further numerical studies show that the STI strategy is needed for a patient with a relatively high or low initial CD4 T cell counts, while the continuous therapy is required to maintain CD4+ T cell count above a safe level (Fig. 2 (C–D)) if the initial CD4+ T cell count of a patient is in the middle (data not shown here). This further confirms that it is important to personalize the treatment strategy for different patients at different stages of their disease progression with different initial CD4+ T cell counts.

4. Inverse problems for differential equation models

Solving inverse problems is more challenging and usually of great importance to mathematical modeling, especially when the majority of key model parameters are unknown. Although some parameter values found in literature can be plugged into models, biases are usually introduced into the estimates of the rest parameters, which will consequently affect the validity and generality of biological conclusions inferred. It is therefore desirable to rigorously determine all unknown parameter values from available data and given model structures in practice. In this section, we briefly review the main techniques for solving the inverse problems of differential equation models for HIV dynamics, including nonlinear least squares (NLS), time-varying parameter estimation, and the estimation methods for mixed-effect ODE models. More advanced nonparametric smoothing-based methods are also discussed.

4.1 Nonlinear least squares approach

Notice that the ODE identifiability, in particular nonlinear ODE identifiability problem should be carefully addressed before the ODE parameter estimation is carried out. The nonlinear ODE identifiability studies, in particular for HIV viral dynamics models, are recently reviewed by Miao et al. [94], and thus are omitted here. The solutions to even linear ODEs are usually nonlinear such that the nonlinear least squares (NLS) method has been widely used in practice by researchers in various fields. In this section, we review the basic principle of NLS and the associated practical issues.

For a nonlinear ODE system, the measurement model can be written as follows

$$y_{ij}=f\left({\mathit{x}}_i,{\theta }^{\ast }\right){\epsilon }_{ij},\text{for}i=1,2,\dots ,n\text{and}j=1,2,\dots ,J_i,$$

where x_i ∈ R^m a vector of state variables, θ^* ∈ R^q the true parameter vector, y_ij the observed variable, n the number of observation points, and J_i the number of replicates at x_i. For convenience, we introduce the notations y = (y₁₁, y₁₂,…, y_1,J1,…, y_n1, y_n2,…, y_n,Jn)^T and ε = (ε₁₁, ε₁₂,…, ε_1,J1,…, ε_n1, ε_n2,…, ε_n,Jn)^T, and let Σ denote the covariance matrix; then the unbiased estimator θ̂ of the true parameter vector θ^* is given by

$$\widehat{\theta }=\underset{\theta }{argmin}S(\theta )=\underset{\theta }{argmin}{(\mathbf{y}-\mathbf{f})}^T\mathbf{\sum }(\mathbf{y}-\mathbf{f}),$$

where S(θ)is also called the weighted residual sum of squares (WRSS). When Σ is unknown and has to be estimated, the computing details can be found in Seber and Wild [139]. For simplicity, let Σ = σ²V and assume ε ~ MVN(0,σ²V), where MVN stands for multivariate normal distribution. The distribution of θ̂ is thus

$$\widehat{\theta }~\mathit{MVN}\left({\theta }^{\ast },{\sigma }^2{\mathbf{C}}^{-1}\right),$$

where $\mathbf{C}={\left(\frac{\partial \mathbf{f}}{\partial {\theta }^{\ast }}\right)}^T{\mathbf{V}}^{-1}\left(\frac{\partial \mathbf{f}}{\partial {\theta }^{\ast }}\right)$ is also called the Fisher information matrix. The unbiased variance estimator of σ² is given by

$${\widehat{\sigma }}^2=\frac{1}{N-q}{\widehat{\mathbf{\epsilon }}}^T·{\mathbf{V}}^{-1}·\widehat{\mathbf{\epsilon }},$$

where $N={\displaystyle \sum _{i=1}^n}{J}_i$ is the total number of observations.

Although the NLS is conceptually simple, the associated computing issues need to be carefully addressed. Specifically, the NLS regression problem boils down to the determination of the global minima of S(θ), which becomes very challenging for a high-dimensional parameter space or for noisy data. Therefore, the choice of the appropriate optimization algorithms becomes critical to obtaining reliable estimates. Four categories of optimization algorithms have been widely used in practice, including direct search methods, gradient-based methods, global optimization methods, and hybrid (or memetic) algorithms. Briefly speaking, the direct search methods (e.g., the Simplex method [30]) search for local minima and are usually sensitive to the starting point and computationally expensive. The gradient-based methods, such as the Levenberg-Marquardt method and the Gauss-Newton method, can efficiently search for local minima based on the Jacobian and/or Hessian matrix of the objective function [103]. The sequential quadratic programming (SQP) algorithms are among the highly recommend algorithms in the gradient-based category. High-quality implementations of SQP can be found in the fmincon routine in MATLABTM (The MathWorks Inc.), SOLNP by Ye [177], or SNOPT by Gill etc. [50]. Although the gradient-based algorithms have been previously used for parameter estimation of ODE models [41], these methods can be easily trapped by local minima or just fail to converge for nonlinear ODE models. To overcome such problems, global optimization algorithms have been considered for ODE models (e.g., differential evolution [149], particle swarm [74], and scatter search [51]). Although the global optimization algorithms are capable of searching the global minima/maxima for spiny nonlinear objective functions, the associated computing costs are usually prohibitive for such methods to converge. It is thus natural to combine the global methods with gradient-based methods to achieve a better balance between estimation accuracy and computing efficiency, which results in the so-called hybrid or memetic algorithms. Rodriguez-Fernandez et al. [130] combined the scatter search algorithm with the SQP method and applied their algorithm to an example of HIV models for parameter estimation. However, the DESQP algorithm proposed in [84] may have a better performance in comparison with the approach in [130] primarily due to the superiority of the differential evolution algorithm over the scatter search method.

4.2 Parameter estimation for semi-mechanistic models

One common concern about mathematical modeling is that any given model structure can only take a limited number of biological factors into consideration such that the conclusions may become model structure specific. Furthermore, many biological mechanisms may remain unknown or undistinguishable such that the development of mechanistic models becomes infeasible. To address the problems above, semi-mechanistic modeling has been introduced and employed for a variety of biomedical problems [1, 18, 41, 44, 53, 73, 75, 148]. In particular, the work of Liang et al. [84] considered a more general framework for semi-mechanistic modeling and used the long-term HIV dynamics as an example. More specifically, the following model with both constant and time-varying unknown parameters was considered

$$\{\begin{array}{l}\frac{d}{dt}T(t)=\lambda -dT-\eta (t){TV}_I\hfill \\ \frac{d}{dt}{T}^{\ast }(t)=\eta (t){TV}_I-\delta T^{\ast }\hfill \\ \frac{d}{dt}{V}_I(t)=N\delta T^{\ast }-{cV}_I\hfill \end{array},$$ (4.1)

where the biological meaning of state variables and model parameters are the same as those in Model (3.4) except that a time-varying parameter η(t) is introduced to describe the infection rate which may vary due to the change in antiviral drug efficacy (e.g., the development of drug resistance). To estimate both the constant and time-varying parameters in this model, the semi-parametric model was turned into a parametric model by approximating the unknown time-varying parameter η(t) using, e.g., splines. For example, Li et al. [82] used natural cubic splines to approximate the unknown time-varying parameters in a pharmacokinetic model. However, different types of splines can be used for approximation, such as the well-known piecewise polynomial splines and basis splines. Here we use the B-splines to illustrate the idea so Model (4.1) becomes

$$\{\begin{array}{l}\frac{d}{dt}T(t)\approx \lambda -dT-\left(\sum _{i=1}^s{a}_i{b}_{i,k}(t)\right){TV}_I\hfill \\ \frac{d}{dt}{T}^{\ast }(t)\approx \left(\sum _{i=1}^s{a}_i{b}_{i,k}(t)\right){TV}_I-\delta T^{\ast }\hfill \\ \frac{d}{dt}{V}_I(t)=N\delta T^{\ast }-{cV}_I\hfill \end{array},$$ (4.2)

where $\eta (t)={\displaystyle \sum _{i=1}^n}{a}_i{b}_{i,k}(t)$ is approximated by a linear combination of the kth-order basis spline function b_i,k (t). Since b_i,k (t) are uniquely determined once k is given and knots are specified, the unknown model parameters are now (λ, ρ, N,δ, c, a₁, a₂,…, a_s)^T, which are all constant. In this way, the semi-parametric model becomes a parametric model so the NLS method introduced in the above subsection can be applied for parameter estimation.

Two practical issues need to be addressed for this approach. First, the spline order k is usually between 2 and 5. B-splines of order 2 are actually piecewise straight lines. Also, higher order B-splines (e.g., k> 5) can introduce unnecessary dramatic local oscillation called Runge’s phenomenon [135]. Second, the positions of knots can usually be selected by quantile such that the number of data points is nearly equal between any two consecutive knots. However, the knots’ position may have to be manually adjusted for some problems to achieve certain temporal patterns, and unfortunately there are no systematic and rigorous methods so far for automatic determination of optimal knots’ positions. That is the reason Liang et al. [84] did not select knots’ positions but spline orders using Akaike Information Criterion (AIC) [3], Bayesian Information Criterion (BIC) [138] and corrected Akaike Information Criterion (AICc) [19].

4.3 Parameter estimation for mixed-effects ODE models

Longitudinal dynamic systems were first introduced in Han, Chaloner and Perelson [54]; however, this work was formulated in the regular nonlinear mixed-effects model framework instead of general mixed-effects ODE models. Li et al. [82] and Huang, Liu and Wu [65] formally addressed this issue. Specifically, it was suggested in Huang et al. [65] that a general longitudinal dynamic system can be given as follows:

$$\begin{array}{l}\frac{d}{dt}{\mathbf{x}}_i(t)=\mathbf{f}(t,{\mathbf{x}}_i(t),{\mathbf{u}}_i(t),{\mathit{\theta }}_i),{\mathbf{x}}_i(0)={\mathbf{x}}_{0i},\\ {\mathbf{y}}_i\left(t_{ij}\right)=\mathbf{h}\left({\mathbf{x}}_i\left(t_{ij}\right),{\mathbf{x}}_{0i},{\mathit{\theta }}_i\right)+{\mathit{\epsilon }}_i(t_{ij}),i=1,\dots ,n,j=1,\dots ,n_i\end{array}$$

where the subscript i denotes the ith subject, x_i(t) ∈ R^m is a vector of state variables, y_i(t_ij) ∈ Rⁿ the measured output variable vector, u_i(t) ∈ R^p the known system input vector, θ_i ∈ R^q the parameter vector, and x_0i the initial condition vector. In the two equations above, the structure of the dynamic system is the same for all subjects but the values of parameters and initial conditions could be different for different subjects. The between-subject variation of parameters and initial conditions can be generally specified as

$${\mathit{\beta }}_i=\mathbf{g}(\mathit{\mu },{\mathbf{b}}_i)$$

where ${\mathit{\beta }}_i={\left({\mathit{\theta }}_i^T,{\mathbf{x}}_{0i}^T\right)}^T$ is the subject-specific parameter vector, μ the population parameter vector, and b_i are random effects. For simplicity, it is usually assumed b_i ~ MVN(0, D), β_i = μ + b_i, and ε_i (t_ij)|b_i ~ MVN(0, R_i). In nature, the formulation above is still a nonlinear mixed-effects (NLME) model [31]. Guedj, Thiébaut and Commenges [52] proposed a maximum likelihood estimator (MLE) and investigated the associated statistical inference for ODE NLME models. As argued in [52], the advantages of the MLE approach over the Bayesian approach include a much lower computing cost, requiring no prior distributions, and a well-established framework of inference theories. Although the original work of Guedj et al. [52] accounted for the left-censored data, it is not the focus here and thus we only give the complete data log-likelihood function here

$$L(\mathbf{y},\mathbf{b})=-\frac{1}{2}\sum _{i=1}^{n}log\mid {\mathbf{R}}_i\mid -\frac{1}{2}\sum _{i=1}^n{({\mathbf{y}}_i-{\mathbf{h}}_i)}^T{\mathbf{R}}_i^{-1}({\mathbf{y}}_i-{\mathbf{h}}_i)-\frac{n}{2}log\mid \mathbf{D}\mid -\frac{1}{2}\sum _{i=1}^n{\mathbf{b}}_i^T{\mathbf{D}}^{-1}{\mathbf{b}}_i+\text{const}\mathrm{..}$$

The key step to obtain the maximum likelihood estimates in Guedj et al. [52] is to compute the first order derivatives of the log-likelihood in two stages. At the first stage, the score of the full log-likelihood was calculated for given random effects; at the second stage, the score of the observed log-likelihood was calculated by integrating out the random effects as suggested in [87, 24]. In addition, a Newton-Raphson type of iterative method was proposed to reduce the computing burden associated with the calculation of the Hessian matrix. Their simulation studies suggested that the rate of successful convergence by the proposed method is greater than 90% for a specific ODE model. However, there also exist a number of alternative methods for fitting NLME models, among which the stochastic approximation EM algorithm (SAEM) [38] looks a promising solution to general NLME problems due to its accuracy and efficiency. The SAEM algorithm has also been extended and successfully applied in a number of studies [78, 79, 137, 91, 92, 93].

Finally, it should be mentioned that the estimation problem of mixed-effects ODE models has also been addressed using the Bayesian framework [54, 162] and the two-stage smoothing-based approach [43]. But limited by space, the technical details are not described here.

4.4 Nonparametric smoothing-based approach

Varah [158] proposed an alternative parameter estimation technique, based on the earlier work of Swartz and Bremermann [150] and Benson [11], that does not require repeated numerical solutions of ODEs. This method fits the discrete measurements of the measured dynamic variables, say y, empirically using a nonparametric smoothing method such as splines, which are then differentiated with respect to time to obtain the estimated time-derivative curves, dy/dt. This time-derivative information is then substituted into the ODEs, converting the parameter estimation problem from a dynamic optimization problem into a much simpler linear/nonlinear regression problem that can be solved using either the linear or nonlinear least-squares method. For this smoothing-based method, parameter values are selected to minimize squared residuals in the differential form of the model, (dy/dt–dŷ/dt)² rather than the traditional integrated form of the model, (y − ŷ)². Swartz and Bremermann [150] and Varah [158] claimed that the main benefits of such a smoothing-based technique include less computational time than other parameter estimation techniques for dynamic models, and no need for determining the initial conditions for the output variables. The drawback is that the estimate of the derivative curve is usually poor, which may result in poor estimates of interested parameters.

Ramsay [125] proposed another smoothing-based approach called the principal differential analysis (PDA) wherein coefficients (possibly time-varying) in linear ODEs are fitted empirically from the data. PDA has been used to fit linear differential equation models in a variety of applications [126]. However, for the smoothing-based methods, there is always a trade-off between the function being over-smoothed and being under-smoothed. In the spline-based ODE parameter estimation approach, Varah [158] attempted to tackle this trade-off problem by interactively adjusting the number and position of knots by hand until satisfactory smoothing was obtained. Alternatively, the extent of smoothing can be controlled by adding a penalty on higher-order derivatives of the splines [125]. Usually the second-order derivative (curvature) is used, which is similar to the idea of smoothing splines. Heckman and Ramsay [59] also considered the L-spline, i.e., replacing the second-order derivative penalty with the ODE model-based penalty. Recently, Poyton et al. [123] and Varziri et al. [159] proposed an iterative PDA (iPDA) approach for simultaneously minimizing the PDA objective function with the ODE model-based penalty. Ramsay et al. [126] proposed a new algorithm to minimize the penalized spline objective function to fit ODE models.

Liang and Wu [85] adopted a different perspective (the measurement error in regression model) to tackle the smoothing-based estimation problem, called two-stage pseudo least squares (PLS). For simplicity, assume that the X(t) is a univariate state variable which is directly observable, i.e.,

$$y_i=x(t_i)+{\epsilon }_i,i=1,\dots ,n.$$

In this case, the first step is to estimate x(t) and its derivative ẋ(t), say x̂(t) and $\widehat{\stackrel{.}{x}}(t)$, using a smoothing method such as the local polynomial approach, and then at the second step a regression model can be fitted

$$\widehat{\stackrel{.}{x}}(t)=f(x(t),\theta )+\mathrm{\Delta }(t),$$

where $\mathrm{\Delta }(t)=\widehat{\stackrel{.}{x}}(t)-\stackrel{.}{x}(t)$ can be regarded as an error term due to replacement of ẋ(t) by its estimate. Now the least squares approach can be applied, that is, to minimize,

$$S_n(\theta )=\sum _t{[\widehat{\stackrel{.}{x}}(t)-f(\widehat{x}(t),\theta )]}^2$$

with respect to the unknown parameter vector θ. Liang and Wu [85] have also established the consistency and asymptotic normality of the PLS estimator under some regularity assumptions. In constructing consistency, we utilized the asymptotic properties of both the local linear estimator and the nonlinear least squares estimator. We have also shown that the proposed PLS estimator is still asymptotically normal. Chen and Wu [22, 23] also extended the above two-step PLS approach to estimate the time-varying parameters in the ODE models.

5. Summary and discussion

In this article, we reviewed differential equation modeling methods for HIV viral dynamics, especially those for understanding antiretroviral drug responses to HIV infection. Starting from the classic HIV infection models, we illustrated how such models were extended to account for drug efficacy. We then reviewed the studies on short-term and long-term treatment modeling. The combination of mathematical models with control theories has resulted in many interesting findings and predictions, which is also briefly reviewed. In particular, the STI treatment strategies were shown to be mathematically complex but effective; however, clinical evidences are still needed to support the development of treatment strategies along this line of ideas.

The usability of the models in this article is worth of further discussion. In general, model usability depends on the model assumptions, the goal of modeling and sometimes the data availability. More specifically, it is usually not possible to include every relevant biological detail in one model, so models are basically simplified mathematical representation of the real world. Besides the essential components, if some other biological processes are not explicitly included in a model, this model can be deemed as unusable for understanding such specific processes. For example, some HIV viral dynamic models (e.g., Models (1.4)~(1.7)) do not explicitly consider the latently infected T cells, so they cannot be used to investigate the effects of latent infection; however, these models can still be used to understand other aspects of HIV viral dynamics as in [118]. Also, a model can be constructed for different purposes such as simulation or parameter estimation. For simulation, more complex models can be used if the kinetics parameter values are known from literatures or experiments. However, if a model is used for estimation purpose, one has to consider whether sufficient data are available such that simpler models may be favorable. Furthermore, when multiple models are developed for the same biological problem, we can assess and compare them by considering the biological validity, the balance between complexity and parsimony, and the capability of data interpretation. In practice, one can use model evaluation scores such as AIC, BIC and AICc for model evaluation and comparison.

Parameter estimation techniques are of extreme importance in many settings to understand and control HIV infection (e.g., determination of key kinetic/epidemic parameters), we therefore also reviewed the classical and newly developed estimation approaches in this article. Although numerous approaches for parameter estimation of ODE models have been proposed within the last few decades, existing methods need to be further improved to achieve a better balance between accuracy and computing efficiency (e.g., least squares vs. smoothing-based methods). The introduction of semi-mechanistic modeling and the development of mixed-effects ODE models allow investigators to use more flexible structures to deal with more complex data.

In short, modeling of antiretroviral drug responses of HIV infected patients is far from being mature in the sense of designing novel and realistic regimes and generating accurate predictions of treatment outcomes. This review summarized a number of selected models to provide a basis for further discussions and communications.