Evaluating the Efficacy of a Malaria Vaccine

Abstract

Malaria is a major public health problem. An effective vaccine against malaria is actively being sought. We formulate a potential outcomes definition of the efficacy of a malaria vaccine for preventing fever. A challenge in estimating this efficacy is that there is no sure way to determine whether a fever was caused by malaria. We study the properties of two approaches for estimating efficacy: (1) use a deterministic case definition of a malaria caused fever as the conjunction of fever and parasite density above a certain cutoff; (2) use a probabilistic case definition in which the probability that each fever was caused by malaria is estimated. We compare these approaches in a simulation study and find that both approaches can potentially have large biases. We suggest a strategy for choosing an estimator based on the investigator's prior knowledge about the area in which the trial is being conducted and the range of vaccine efficacies over which the investigator would like the estimator to have good properties.

1. Introduction

Malaria is a giant killer of children (over one million deaths per year (World Health Organization, 2005)), imposes financial hardships on poor households and holds back economic growth (World Health Organization, 1999). Symptoms of malaria include fever, vomiting, shivering, anemia, retinal damage and coma (in severe cases). Fever is the most common symptom of malaria and a vaccine that protected against fever caused by malaria would be considered valuable for public health (Moorthy, Reed and Smith, 2007). In this paper, we consider the challenges of evaluating the efficacy of a malaria vaccine for preventing fever and study the properties of proposed approaches. An excellent reference for issues related to measuring malaria vaccine efficacy, which this paper builds on, is the WHO Study Group on Measures of Malaria Vaccine Efficacy report (Moorthy et al., 2007).

For a vaccine against an infectious agent, vaccine efficacy is defined roughly as the reduction in incidence of the disease due to the infectious agent if everyone took the vaccine (as opposed to a placebo) compared to the incidence of the disease due to the infectious agent under the placebo (Hudgens, Gilbert and Self, 2004). We give a precise causal definition of vaccine efficacy in Section 2.

Lachenbruch (1998) showed that if some people who do not have the disease due to the infectious agent are misclassified as having the disease due to the infectious agent (i.e., specificity is less than 1), then vaccine efficacy estimates will be biased downwards. This result poses challenges for estimating the efficacy of a malaria vaccine. In settings in which the malaria parasite is endemic, it is difficult to determine whether a child suffering from a fever is suffering from a fever caused by malaria parasites or a fever caused by a different infectious agent (Smith, Schellenberg and Hayes, 1994). Fevers caused by malaria parasites often cannot be distinguished on the basis of clinical features from fevers caused by other common childhood infections such as the common cold, pneumonia, influenza, viral hepatitis or typhoid fever (Hommel, 2002; Koram and Molyneux, 2007). One aid to deciding whether a fever is caused by malaria parasites or some other infection is to measure the density of malaria parasites in the child’s blood. However, in areas where malaria is endemic, children can develop partial immunity to the toxic effects of the parasites and can tolerate high parasite densities without developing fever (Marsh, 2002; Boutlis, Yeo and Antsey, 2006). Thus, even if a child has fever and has a high parasite density, the fever might still be caused by another infection. In summary, there is no current perfect case definition of a fever caused by malaria that has specificity 1 and sensitivity 1.

Two types of case definitions have been proposed for a fever caused by malaria. One is an imperfect, but deterministic case definition. The WHO Study Group on Measures of Malaria Vaccine Efficacy (Moorthy et al., 2007) recommends using as a case definition the conjunction of fever and a parasite density above a certain cutoff. For example, Alonso et al. (2004) and Bejon et al. (2008) used a cutoff of 2,500 parasites per μl. An alternative to an imperfect, deterministic case definition of a malaria caused fever is to estimate the probability that a fever was caused by malaria and use a probabilistic case definition. Smith, Schellenberg and Hayes (1994) proposed a logistic regression approach for estimating the probability that a child’s fever is malaria caused based on the child’s parasite density.

We compare in a simulation study the two approaches of using an imperfect, deterministic case definition and a probabilistic case definition for estimating the efficacy of a malaria vaccine. Rogers et al. (2006) and Smith (2007) have also compared these two approaches. Our simulation study differs from theirs in that we include in the simulation study model the phenomenon that fever due to an infectious agent other than malaria parasites can reduce malaria parasite density (Kwiatkowski, 1989; Long et al., 2001; Rooth and Bjorkman, 1992).

In Section 2, we provide a causal definition of vaccine efficacy in terms of potential outcomes. In Section 3, we provide a causal diagram for how a vaccine and other factors might affect fever. In Section 4, we discuss deterministic and probabilistic case definitions of malaria caused fever and how they are used to estimate vaccine efficacy. In Section 5, we compare the two approaches in a simulation study. In Section 6, we provide discussion.

2. Potential Outcomes Definition of Vaccine Efficacy

Before defining vaccine efficacy, we first define what we mean by a fever that is caused by malaria parasites. Let $Y_i^{(p)}$ be child i’s potential fever outcome under the placebo, $Y_i^{(p)}=1$ if child i would have a fever if given the placebo and $Y_i^{(p)}=0$ if child i would not have a fever if given the placebo. Let $Y_i^{(e)}$ be child i’s potential fever outcome if malaria parasites were eradicated from the world. We say that a child i who has a fever (under the placebo) has a malaria caused fever if the fever would be eliminated by eradicating malaria, i.e., child i has a malaria caused fever if $Y_i^{(p)}=1$, $Y_i^{(e)}=0$.

Let $Y_i^{(v)}$ denote child i’s potential fever outcome if the child would take the vaccine. We define the vaccine efficacy in a population as the reduction in the number of children who would have fever if all children took the vaccine compared to if all children took the placebo divided by the number of children who would have a malaria caused fever if all children took the placebo:

$$VE=\frac{P\left(Y_i^{(p)}=1\right)-P\left(Y_i^{(v)}=1\right)}{P\left(Y_i^{(p)}=1,Y_i^{(e)}=0\right)}.$$ (1)

VE ranges from (−∞, 1] with a VE of 1 indicating that the vaccine is as effective as eradicating malaria, a VE greater than zero indicating that the vaccine is more effective than the placebo, VE equal to zero indicating that the vaccine is equivalent to the placebo and VE less than zero indicating that the vaccine is worse than the placebo.

Another way to think about what VE is measuring is the following. Let R_P be the risk of fever under the placebo and R_V be the risk of fever under the vaccine. Then,

$$VE=\frac{R_P\text{ by all causes}-R_V\text{ by all causes}}{R_P\text{ by malaria}},$$

where “by all causes,” we mean fever due to any cause, not just malaria. Consider the assumption that the vaccine does not affect the probability of a fever from causes other than malaria, i.e., Assumption (A-1):

$$P\left(Y_i^{(p)}=1,Y_i^{(e)}=1\right)=P\left(Y_i^{(v)}=1,Y_i^{(e)}=1\right).$$

Under (A-1), V E is equal to the effect of the vaccine on preventing malaria caused fevers, which we denote by V E^*,

$$\begin{array}{l}V{E}^{*}=\frac{R_P\text{ by malaria}-R_V\text{ by malaria}}{R_P\text{ by malaria}}\\ \hspace{1em}=\frac{P\left(Y_i^{(p)}=1,Y_i^{(e)}=0\right)-P\left(Y_i^{(v)}=1,Y_i^{(e)}=0\right)}{P\left(Y_i^{(p)}=1,Y_i^{(e)}=0\right)}.\end{array}$$ (2) (2)

The equivalence of V E and V E^* under assumption (A-1) is illustrated in Figure 1. In Figure 1, $VE=\frac{0.7-0.5}{0.5}=0.4$ and $V{E}^{*}=\frac{0.5-0.3}{0.5}=0.4$.

Figure 1:

Illustration of equivalence of VE and VE^* under assumption (A-1).

We prefer to use V E rather than than V E^* as our definition of vaccine efficacy because when assumption (A-1) fails and the vaccine affects the probability of a non-malaria caused fever, we would like to include this effect in the measure of vaccine efficacy. A vaccine may cause fever by a variety of mechanisms (Follmann, Fay and Proschan, 2009): (a) some vaccines are live but weakened pathogens and the weakened pathogen can produce an infection in a weakened host; (b) vaccine induced antibodies may enhance rather than reduce the chance of disease (Burke, 1992); or (c) vaccines might induce an auto-immune reaction which could hamper the ability of the immune system to fight disease. As an example, consider a vaccine that prevents 10% of malaria caused fevers but causes 10{R_P by malaria/(1–R_P by all causes)}% of the people without fevers under the placebo to have a fever. Then V E = 0 but V E^* = 0.1. This vaccine is not preventing fevers so we think V E = 0 is a better measure of the overall public health benefit of the vaccine than V E^* = 0.1.

A problem with both V E and V E^* is that the denominator R_P by malaria is not identifiable without assumptions that go beyond just randomization of vaccine and placebo; we discuss this nonidentifiability in Section 4.2. A quantity that is identifiable from a randomized trial without further assumptions is the proportion of all fevers that are prevented by the vaccine, which we denote by V E.All,

$$VE.All=\frac{R_P\text{ by all causes}-R_V\text{ by all causes}}{R_P\text{ by all causes}}=\frac{P\left(Y_i^{(p)}=1\right)-P\left(Y_i^{(v)}=1\right)}{P\left(Y_i^{(p)}=1\right)}.$$

V E.All measures the impact of the vaccine on the trial population and is hence of considerable interest. However, V E.All does not address the following question that is addressed by V E: Is the vaccine preventing most malaria-caused fevers (V E close to 1) or is there considerable room for improving the vaccine? For example, suppose V E.All is 0.2. This could mean that the vaccine is not that effective and could be considerably improved, or it could mean that the vaccine is very effective, preventing most malaria caused fevers, but that most fevers in the area are not malaria caused. Another question of interest that requires knowing both V E and V E.All is the following: What impact would we expect the vaccine to have on a population in which malaria accounts for a different proportion of fevers than in the trial population? Thus, both V E and V E.All are important quantities in assessing a vaccine.

3. Causal Diagram for the Effect of a Vaccine

To better understand the properties of different estimators of the efficacy of a malaria vaccine, it is helpful to consider a causal diagram for how a vaccine and other causes affect fever. Figure 2 depicts a causal diagram. The bottom row of the figure shows how malaria parasites cause fever. First, a person needs to be bitten by a mosquito carrying malaria parasites. When the mosquito takes its blood meal, it injects the parasites into the person. The parasites then go to the person’s liver where they develop and replicate. While in the liver, the parasites do not cause any clinical symptoms. After several days, the parasites escape from the liver into the blood. The parasites attack red blood cells, replicate within the red blood cells, burst the red blood cells and attack further red blood cells. If uncontrolled, the parasites will keep multiplying and keep destroying red blood cells, eventually causing severe anemia. The parasite density measures the density of parasites in the blood at a given time. As the parasites replicate, they are thought to release toxins which, when built up enough, cause fever. For good discussion of the malaria parasite life cycle and its effects on humans, see Warrell, Turner and Francis (2002) and Sinden and Gilles (2002).

Figure 2:

Causal Diagram of the Effects of Vaccine and Other Causes on Fever

There are several types of vaccines being developed that aim to interrupt various parts of how malaria parasites cause fever and other symptoms. One type of vaccine (pre-erythrocytic) aims to attack the parasites in the liver and prevent them from entering the blood. Another type of vaccine (erythrocytic) aims to reduce the multiplication of parasites once they have entered the blood. A third type of vaccine (anti-toxic) does not aim to reduce the parasites but aims to reduces the toxins created by the parasites.1 Vaccines are also being investigated that combine different components of these three types of vaccines. Figure 2 depicts how the vaccine can affect the parasites’ development in the liver, the parasites’ replication in the blood and the parasites’ release of toxins by the lines between the vaccine and the boxes representing each of these three stages of parasite activity. For a good review of recent work on malaria vaccines, see Girard et al. (2007).

Other infections that causes fever, such as common cold, pneumonia, influenza, viral hepatitis or typhoid fever, are depicted by the box on the top of Figure 2 as non-malaria infections. There may be unmeasured common causes between the non-malaria infections and any stage of parasite development. For example, there may be unmeasured socioeconomic variables that make a child more vulnerable to being bit by a mosquito carrying malaria parasites (e.g., a family with low socioeconomic status may be less likely to purchase a bed net) and more vulnerable to non-malaria infections.

As discussed in Section 2, it is possible that the vaccine causes non-malaria infections and non-malaria caused fevers. This is reflected in Figure 2 by the lines between the vaccine box and the non-malaria infections and fever boxes.

One additional causal relationship that is depicted in Figure 2 is that fever itself causes a reduction in parasite density; this is depicted by the line between the fever box and the parasites replicate box. There is in vitro evidence that high temperatures inhibit parasite growth (Kwiatkowski, 1989; Long et al., 2000) and in vivo evidence that parasite density is suppressed during the febrile illnesses measles and influenza (Rooth and Bjorkman, 1992).

4. Estimators of Vaccine Efficacy

We discuss the properties of the two currently used approaches to estimating vaccine efficacy. Both consist of substituting estimates of R_P by malaria and R_V by malaria into (2) for V E^*. Recall that under assumption (A-1), V E^* is equal to V E. Both approaches estimate R_P by malaria and R_V by malaria by using a “case definition” of a malaria caused fever: the first method uses a deterministic case definition while the second method uses a probabilistic case definition.

4.1. Deterministic Case Definition

The most commonly used deterministic case definition of malaria caused fever is the conjunction of fever and a parasite density above a cutoff c. For example, Alonso et al. (2004) and Bejon et al. (2008) used a cutoff of 2,500 parasites per μl and Sagara et al. (2009) used a cutoff of 3,000 parasites per μl. Because of variation in natural and acquired immunity, there is considerable variation in children’s pyrogenic thresholds, the minimum parasite density at which the parasites will cause a child to have a fever (Marsh, 2002). Consequently a case definition of the conjunction of fever and parasite density above 2,500 μl will include some false positives, children who have a fever and a parasite density above 2,500 μl but who are tolerating the parasites and only have a fever because of another infectious agent. The case definition will also miss some false negatives, children whose pyrogenic threshold and actual parasite density is below 2,500 but who have parasite densities that are greater than their pyrogenic thresholds and thus have fevers that are caused by malaria parasites.2

Rogers et al. (2006) and Smith (2007) study the use of case definitions of the conjunction of fever and parasite density above a certain cutoff c for estimating vaccine efficacy. Both papers show in simulation studies that estimated vaccine efficacies depend heavily on the cutoff. Here, we present a formula for the asymptotic bias of the estimator V̂ E (c) that uses the deterministic case definition of the conjunction of fever and a parasite density above c. As the sample size from a randomized trial converges to ∞, V̂E (c) converges in probability to

$$VE\left(c\right)=1-\frac{P\left(Y^{(v)}=1,D^{\left(v\right)}>c\right)}{P\left(Y^{\left(p\right)}=1,D^{\left(p\right)}>c\right)},$$

where D^(v) and D^(p) denote the potential parasite densities under the vaccine and placebo respectively. Let SE_P be the sensitivity of the cutoff c for detecting malaria-caused fevers under the placebo, SE_P = P(D^(p) > c|Y^(p) = 1, Y^(e) = 0); SP_P be the specificity of the cutoff c for detecting non-malaria caused fevers under the placebo, SP_P = P(D^(p) ≤ c|Y^(p) = 1, Y^(e) = 1); SE_V be the sensitivity under the vaccine, SE_V = P(D^(v) > c|Y^(v) = 1, Y^(e) = 0); and SP_V be the specificity under the vaccine, SP_V = P(D^(v) ≤ c|Y^(v) = 1, Y^(e) = 1). Then,

$$\begin{array}{c}VE\left(c\right)=\\ \hspace{1em}1-\\ \frac{P\left(Y^{\left(v\right)}=1,Y^{\left(e\right)}=0\right)+P\left(Y^{\left(v\right)}=1,Y^{\left(e\right)}=1,D^{\left(v\right)}>c\right)-P\left(Y^{\left(v\right)}=1,Y^{\left(e\right)}=0,D^{\left(v\right)}\le c\right)}{P\left(Y^{\left(p\right)}=1,Y^{\left(e\right)}=0\right)+P\left(Y^{\left(p\right)}=1,Y^{\left(e\right)}=1,D^{\left(p\right)}>c\right)-P\left(Y^{\left(p\right)}=1,Y^{\left(e\right)}=0,D^{\left(p\right)}\le c\right)}=\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}1-\frac{P\left(Y^{\left(v\right)}=1,Y^{\left(e\right)}=0\right)\left(S{P}_V+S{E}_V-1\right)+P\left(Y^{\left(v\right)}=1\right)\left(1-S{P}_V\right)}{P\left(Y^{\left(p\right)}=1,Y^{\left(e\right)}=0\right)\left(S{P}_P+S{E}_P-1\right)+P\left(Y^{\left(p\right)}=1\right)\left(1-S{P}_P\right)}.\end{array}$$

Under assumption (A-1) that the vaccine does not cause fevers, the asymptotic bias of V E(c) is

$$\begin{array}{c}VE\left(c\right)-VE=\\ \frac{P\left(Y^{\left(v\right)}=1,Y^{\left(e\right)}=0\right)}{P\left(Y^{\left(p\right)}=1,Y^{\left(e\right)}=0\right)}-\frac{P\left(Y^{\left(v\right)}=1,Y^{\left(e\right)}=0\right)\left(S{P}_V+S{E}_V-1\right)+P\left(Y^{\left(v\right)}=1\right)\left(1-S{P}_V\right)}{P\left(Y^{\left(p\right)}=1,Y^{\left(e\right)}=0\right)\left(S{P}_P+S{E}_P-1\right)+P\left(Y^{\left(p\right)}=1\right)\left(1-S{P}_P\right)}.\end{array}$$

The WHO Study Group on Measures of Malaria Vaccine Efficacy (Moorthy et al., 2007) recommended choosing the cutoff c so that the specificity is at least 0.8. However, the asymptotic bias depends on the specificity and sensitivity under both the placebo and vaccine arms. Furthermore, the variance of V̂E (c) is a complicated function of θ = (P(Y^(v) = 1, Y^(e) = 0), P(Y^(p) = 1, Y^(e) = 0), SP_V, SE_V, SP_P, SE_P). Therefore, we recommend that if a deterministic cutoff is used, it should be chosen based on considering the estimated root mean squared error and power under a preliminary estimate(s) of θ. This is illustrated in Section 6 (see Figure 8).

Figure 8:

RMSE and proportion biases for different cutoffs for erythrocytic vaccines in a holoendemic area under the assumption of a moderate fever killing effect with a sample size of n = 1,000 per arm.

4.2. Using Probabilistic Case Definitions to Estimate Vaccine Efficacy

Rogers et al. (2006) and Smith (2007) suggest as an alternative to using an imperfect, deterministic case definition of a malaria caused fever to instead estimate the probability that a fever was caused by malaria and use these estimated probabilities to estimate R_P and R_V.

Smith, Schellenberg and Hayes (1994) proposed the following probabilistic case definition of malaria caused fever: if child i on the placebo arm has a fever, then the probability that it is malaria caused is

$$p_i=\frac{P\left(Y_i^{\left(p\right)}=1|D=D_i,\mathbf{\text{X}}={\mathbf{\text{X}}}_i\right)-P\left(Y_i^{\left(p\right)}=1|D=0,\mathbf{\text{X}}={\mathbf{\text{X}}}_i\right)}{P\left(Y_i^{\left(p\right)}=1|D=D_i,\mathbf{\text{X}}={\mathbf{\text{X}}}_i\right)},$$ (3)

where X_i are measured covariates such as age and season and D_i is measured parasite density. The same case definition holds for the vaccine arm with $Y_i^{\left(v\right)}$ replacing $Y_i^{\left(p\right)}$ in (3). The intuition behind (3) is the following: (a) all fevers for children with D = 0 are assumed to come from non-malaria fever causing infections; (b) if parasite densities D_i are independent of non-malaria infections conditional on X_i, then the probability of a non-malaria fever causing infection for a child with D = D_i is the same as for a child with D = 0 and thus equals $P\left(Y_i^{\left(p\right)}=1|D=0,\mathbf{\text{X}}={\mathbf{\text{X}}}_i\right)$; (c) thus, the difference between the probability of a fever given D = D_i and the probability of a fever given D = 0 represents the proportion of children with D = D_i and X = X_i who have malaria caused fever. Small (2009) provides more discussion.

A key assumption for (3) to provide the correct probability of child i having a malaria caused fever is that the parasite densities D_i are independent of non-malaria fever causing infections conditional on X_i. However, Figure 2 suggests that this is not likely to be true. First, there is the problem of unmeasured common causes of parasite densities and non-malaria infections. Second, there is the problem that fever due to any cause kills parasites. The consequence of parasites being killed by fever is that children with non-malaria fever causing infections tend to have lower parasite density than children without non-malaria fever causing infections.

5. Simulation Study

In our simulation study, we will consider four important factors that may affect the performance of estimators of vaccine efficacy:

1. Type of vaccine: The two most common types of vaccines that are being developed are (1) pre-erythrocytic (pre-blood stage) vaccines that seek to prevent the malaria parasites from entering an infected person’s blood; (2) erythrocytic (blood stage) vaccines that seek to slow the replication of malaria parasites after the parasites have entered the blood (Girard et al., 2007). Smith (2007) found that the type of vaccine affected the performance of different estimators. Following Smith, we will consider one pre-erythrocytic stage vaccine and one erythrocytic vaccine:

   • Pre-erythrocytic (pre-blood stage) vaccine. This vaccine eliminates 50% of malaria infections. For simplicity, this is assumed to be reflected in a 50% reduction in the proportion of children who are infected across the range of parasite densities.
   • Erythrocytic (blood stage) vaccine. This vaccine reduces parasite densities by 50% but does not affect the number of children who are infected.

2. Endemicity: Areas differ greatly in how endemic the malaria parasites are, i.e., how high the rate of infection with malaria parsites is. This could have a substantial impact on various estimators, in particular the deterministic case definition estimators since the specificity and sensitivity of the case definition varies with the endemicity. We consider three levels of endemicity: mesoendemic (low to moderate), hyperendemic (medium high) and holoendemic (high). We define specifically the levels of endemicity we consider below.

3. Effect of fever on parasites: One reason that the probabilistic case definition estimators of vaccine efficacy may be biased is that fever due to any cause kills parasites, as discussed in Section 3. Although there is both in vitro and in vivo evidence for this phenomenon, as cited in Section 3, there are no definitive studies of the proportion of parasites that fever kills in field settings. Long et al. (2001) estimated that in vitro, parasite growth was reduced by 50% at 101.8°F and 92% at 104°F. We consider three ranges of effect of fever on parasites: large fever effect (fever kills approximately 90% of parasites), moderate fever effect (fever kills approximately 50% of parasites) and no fever effect (fever kills no parasites). Although we do not think the no effect assumption is realistic, we include it for comparison because most previous work on malaria vaccine efficacy has assumed no effect.

4. Sample size: We consider two sample sizes, 1,000 children on each arm and 10,000 children on each arm. Preliminary explorations indicated that the estimators differed in their finite sample biases.

We conduct a 2 × 3 × 3 × 2 factorial experiment to investigate the effect of these factors on the performance of different estimators of vaccine efficacy.

Figure 3 describes our simulation study in a causal diagram. Specifically, we generate the data as follows:

1. We randomly assign n children to the placebo and n children to the vaccine arm (either the pre-erythrocytic or erythrocytic vaccine), where n = 1,000 or n = 10,000.

2. A child can get fever through two sources: malaria infection (MI) or non-malaria infection (NMI). Let $Y_i^{MI}=1$ if child i has a malaria infection strong enough to cause a fever, 0 if not and $Y_i^{NMI}=1$ if child i has a non-malaria infection strong enough to cause a fever, 0 if not. The non-malaria and malaria infections are assumed to operate like parallel circuits in causing a fever so that child i is observed to have a fever, Y_i = 1, if and only if $Y_i^{MI}+Y_i^{NMI}\ge 1$.

3. Let D_i be child i’s observed parasite density and $D_i^{No.NMI}$ be the parasite density that child i would have if the child did not have a non-malaria infection strong enough to cause a fever. D_i might not equal $D_i^{No.NMI}$ because fever can kill parasites. We can think of $D_i^{No.NMI}$ as the parasitological challenge faced by the child that arises from the amount that malaria-carrying mosquitoes bite the child and the immune response of the child. We assume that $D_i=D_i^{No.NMI}$ unless $Y_i^{NMI}=1$, $Y_i^{MI}=0$. This is because when $Y_i^{NMI}=1$, $Y_i^{MI}=1$ or $Y_i^{NMI}=0$, $Y_i^{MI}=1$, then child i would have a fever even if she did not have a non-malaria infection and so the effect of fever killing parasites on the child would be the same in a world without non-malaria infections as in the actual world so that $D_i=D_i^{No.NMI}$; also, if $Y_i^{NMI}=0$, $Y_i^{MI}=0$, then child i would not have a fever in both a world without non-malaria infections and the actual world so that fever killing parasites has no effect in either world and consequently $D_i=D_i^{No.NMI}$. When $Y_i^{NMI}=1$, $Y_i^{MI}=0$, then in the absence of a non-malaria infection, child i would not have a fever and then parasites would not have been killed by the fever so that $D_i^{No.NMI}$ might be greater than D_i. The effect of fever on parasite density is represented by the arrow from fever to parasite density in Figure 3. We consider three magnitudes of effect of fever on parasite density:

   • No fever effect: $D_i=D_i^{No.NMI}$ when $Y_i^{NMI}=1$, $Y_i^{MI}=0$.
   • Moderate fever effect: $D_i=k_i{D}_i^{No.NMI}$ when $Y_i^{NMI}=1$, $Y_i^{MI}=0$ where k_i ∼ Beta(5, 5), i.e., on average, fever kills 50% of parasites.
   • Large fever effect: $D_i=k_i{D}_i^{No.NMI}$ when $Y_i^{NMI}=1$, $Y_i^{MI}=0$ where k_i ∼ Beta(1,9), i.e., on average, fever kills 90% of parasites.
4. The probability that $D_i^{NMI}=0$ is determined by the endemicity level. We consider three levels of endemicity:

   • Holoendemic: The probability that a child is infected with malaria parasites in the placebo arm is 0.8.
   • Hyperendemic: The probability that a child is infected with malaria parasites in the placebo arm is 0.5.
   • Mesoendemic: The probability that a child is infected with malaria parasites in the placebo arm is 0.2.

   For children infected with malaria parasites, we assume that the distribution of $D_i^{NMI}$ is that shown in Figure 4. This distribution was chosen to approximate the distribution found in a study in the Kilombero District (Morogoro) Region of Tanzania that was analyzed in Smith, Schellenberg and Hayes (1994).
5. Based on the estimates of the probability of malaria caused fever in Smith, Schellenberg and Hayes (1994) for the study mentioned above in the Kilombero District of Tanzania, we set $P\left(Y_i^{NMI}\right)=0.06$ and

   $$P\left(Y_i^{MI}|D_i^{No.NMI}\right)=\frac{\text{exp}\left(.000027D_i^{No.NMI}\right)}{1+\text{exp}\left(.000027D_i^{No.NMI}\right)}.$$

   The $P\left(Y_i^{MI}\right)$ and $P\left(Y_i^{NMI}\right)$ are plotted in Figure 5.

The R code for our simulation study is available from the authors.

Figure 3:

Causal diagram describing the simulation study.

Figure 4:

Probability Density of D^NMI conditional on a child being infected (D^NMI > 0) under the placebo.

Figure 5:

P(Y^MI = 1|D^No.NMI) and P(Y^NMI = 1|D^No.NMI)

We consider five estimators of vaccine efficacy: (1) deterministic case definition set so that the specificity of the case definition under the placebo is 0.7; (2) deterministic case definition set so that the specificity of the case definition under the placebo is 0.8; (3) deterministic case definition set so that the specificity under the placebo is 0.9; (4) deterministic case definition with a cutoff of 2,500 parasites per μl, this is the cutoff used by Alonso et al. (2004) and Bejon et al. (2008); (5) probabilistic case definition using the correct logistic regression model for P(Y_i = 1|D_i) when there is no effect of fever killing parasites.

The simulation results are shown in Figure 6 (pre-erythrocytic vaccine) and Figure 7 (erythrocytic vaccine). Abs. Prop. Bias is is the absolute proportion bias, i.e., the absolute bias of the estimator divided by the true value of V E. The true values of V E ranged from 0.46 to 0.64. In the figures, we only show the bias and RMSE up to a maximum of 0.7. Complete numerical results are available from the authors.

Figure 6:

Simulation results for pre-erythrocytic vaccine. Bias is the absolute proportion bias. If the bias or RMSE was above 0.7, only a maximum of 0.7 is shown in the plots.

Figure 7:

Simulation results for erythrocytic vaccine. Bias is the absolute proportion bias. If the bias or RMSE was above 0.7, only a maximum of 0.7 is shown in the plots.

Our main findings are the following:

• Performance of WHO’s recommended approach: For the pre-erythrocytic vaccine, the WHO’s recommended approach of choosing a deterministic cutoff so that the specificity is 0.8 (under the placebo) generally worked well, with biases of at most 15%, and RMSE less than the probabilistic case definition estimator for all settings. However, for the erythrocytic vaccine, the WHO’s recommended approach sometimes worked very poorly, with biases over 70% for some settings. For the erythrocytic vaccine, the RMSE of the WHO’s recommended approach was sometimes better and sometimes worse than that of the probabilistic case definition estimator.

• Comparison of deterministic case definitions of different specificities: For most settings, the estimators based on deterministic case definitions set to have specificities 0.7, 0.8 and 0.9 performed similarly. For the pre-erythrocytic vaccine in the mesoendemic setting with no or moderate fever effects and n = 1,000, the specificity 0.7 and 0.8 estimators were somewhat better than the specificity 0.9 estimator. However, for these same settings for the erythrocytic vaccine, the specificity 0.9 estimator was somewhat better than the specificity 0.7 and 0.8 estimators (not only for n = 1,000 but also for n = 10,000). The specificity 0.9 estimator was also somewhat better than the 0.7 and 0.8 estimators for the erythrocytic vaccine in the hyperendemic settings with n = 10,000.

• Performance of cutoff of 2,500 parasites per μl: For the pre-erythrocytic vaccine, the deterministic cutoff of 2,500 parasites per μl worked reasonably well in the holoendemic and hyperendemic settings, but did not work well for mesoendemic settings with sample size n = 1,000 per arm, especially when there was a large fever effect. For the erythrocytic vaccine, the deterministic cutoff of 2,500 also performed poorly in these settings. In addition, for the erythrocytic vaccine, the deterministic cutoff of 2,500 was substantially worse than the WHO’s recommended approach for the holoendemic setting with no fever effect for both n = 1,000 and n = 10,000 and for the holoendemic setting with a moderate fever effect for n = 1,000.

• Finite sample bias of probabilistic case definition estimator: The probabilistic case definition estimator sometimes exhibited large finite sample biases for n = 1,000 that were reduced substantially for n = 10,000. When there is no fever effect, the probabilistic case definition estimator is consistent; see Section 4.2. The consistency is apparent for n = 10,000 for which the biases of the probabilistic case definition estimator when there is no fever effect are all less than 5%. However, for n = 1,000, the biases were sometimes large, above 70% for the pre-erythrocytic vaccine in the mesoendemic setting and above 10% for the pre-erythrocytic vaccine in the holoendemic and hyperendemic settings and for the erythrocytic vaccine in the mesoendemic setting.

6. Discussion

We have provided a potential outcomes definition of the efficacy of a malaria vaccine and studied the properties of two methods for estimating the efficacy, (1) using an imperfect deterministic case definition of the conjunction of fever and parasite density above a certain cutoff and (2) using a probabilistic case definition that aims to estimate the probability that a child with fever has a malaria caused fever. We have shown that both methods potentially have large biases.

To choose an estimator that will be used for the primary analysis in a clinical trial, we suggest that the performance of different estimators over a range of a different vaccine efficacies be considered. For example, consider an erythrocytic vaccine for a holoendemic area under the assumption of a moderate fever killing effect with a sample size of n = 1,000 per arm. Figure 8 plots the RMSEs and proportion bias of the the deterministic case definition estimators with cutoffs ranging from 100 to 10,000 parasites per μl for three vaccines of different efficacies: one vaccine reduces parasite density by 20%, another by 50% and another by 80%. For vaccines covering this range of efficacies, the cutoff of 2,500 is a reasonable choice – it has about the smallest RMSE over different cutoffs for the vaccines that reduce parasite density by 20% and 50%, and is only a little above the minimum RMSE for the vaccine that reduces parasite density by 80%.

In addition to choosing one estimator for a primary analysis, a useful secondary analysis is to conduct a sensitivity analysis that displays a range of efficacy estimates over a range of plausible assumptions about how unmeasured common causes affect parasite density and non-malaria infections and how much of an effect fever has on killing parasites. Further work needs to be done in developing such sensitivity analyses

An additional important issue which we have not discussed in this paper is measurement error in parasite density. O’Meara, Hall and McKenzie (2007) discuss the difficulties created by measurement error.