Use of cross-reactive serological assays for detecting novel pathogens in wildlife: assessing an appropriate cutoff for henipavirus assays in African bats

Abstract

Reservoir hosts of novel pathogens are often identified or suspected as such on the basis of serological assay results, prior to the isolation of the pathogen itself. Serological assays might therefore be used outside of their original, validated scope in order to infer seroprevalences in reservoir host populations, until such time that specific diagnostic assays can be developed. This is particularly the case in wildlife disease research. The absence of positive and negative control samples and gold standard diagnostic assays presents challenges in determining an appropriate threshold, or ‘cutoff’, for the assay that enables differentiation between seronegative and seropositive individuals. Here, multiple methods were explored to determine an appropriate cutoff for a multiplexed microsphere assay that is used to detect henipavirus antibody binding in fruit bat plasma. These methods included calculating multiples of ‘negative’ control assay values, receiver operating characteristic curve analyses, and Bayesian mixture models to assess the distribution of assay outputs for classifying seropositive and seronegative individuals within different age classes. As for any diagnostic assay, the most appropriate cutoff determination method and value selected must be made according to the aims of the study. This study is presented as an example for others where reference samples, and assays that have been characterised previously, are absent.

1. INTRODUCTION

Serological assays are a valuable and widely used tool for studying infectious disease ecology in wildlife. However, inferences from assay results often are made based on a number of assumptions that may, or may not, be fully justified (for review, see Gilbert et al, Ecohealth in press). For example, it may be assumed that a diagnostic assay can “discriminate two mutually exclusive states of tested animals” (Greiner et al., 2000) (e.g. individuals are either ‘seropositive’ or ‘seronegative’). In fact, there is likely to be considerable overlap between these two states due to the dynamic nature of infections and antibody responses within individuals and across populations. An assay cutoff therefore must be selected which artificially dichotomises the antibody response observed into positive and negative results and achieves the desired sensitivity and specificity of the assay according to the needs of the study.

The complexities of interpreting serological results are compounded when the agent being studied is novel and unknown and, in the absence of specific diagnostic assays, existing assays often are used outside their original scope. This is particularly the case in wildlife disease research, where serological cross-reactivity to known pathogens may be detected within a new species or a new geographic area well in advance of detection or isolation of the actual pathogen(s). In some cases, it may be many years or decades before the causative agent is definitively isolated and characterised from the wildlife host (e.g. Hendra viruses in Australian bats (Halpin et al., 2000), Ebola virus in African fruit bats (Bossart et al., 2005; Leroy et al., 2005; Bossart et al., 2007; Li et al., 2008)). In the meantime, valuable information can be obtained using existing assays to which there is cross-reactivity and/or cross-neutralisation, providing the limitations of the assay are recognised and inferences based on results are made with caution (for example, Hayman et al., 2012).

Development and validation of diagnostic assays is recommended (Jacobson, 2009), a process which determines “the fitness of an assay, which has been properly developed, optimised and standardised, for an intended purpose”. However, full validation of an assay for use with a novel pathogen is impossible if the pathogen is yet to be definitively identified and known positive and naïve control samples are unavailable. This is also the case when an existing assay is used with samples from alternative species (Gilbert et al., in press). In the meantime, attempts should be made to determine the validity and limitations of using pre-existing assays across the species and pathogen boundaries, including comparison against alternative assays which may detect antibodies in different ways (e.g. antibody binding and neutralisation assays) and assessing assay performance across populations and laboratories.

An appropriate threshold, or cutoff, against which samples can be designated as ‘positive’ or ‘negative’, must be determined by following logical and repeatable methods. Multiple methods are available to determine an appropriate cutoff, however the majority of these assume that known positive and naive reference samples are available. Gardner et al. (2010) reviewed statistical approaches for the evaluation of diagnostic assays in the presence and absence of available gold standard assays (one that assumes near-perfect classification of infection status). In the presence of a gold standard assay, the approaches reviewed included examining diagnostic sensitivity and specificity using receiver operating characteristic (ROC) curves and likelihood ratio tests. In the absence of a gold standard assay, Bayesian or maximum likelihood latent class models were cited as powerful approaches that enable the sensitivity of two assays being compared to be estimated jointly, without the need to assume that one is ‘perfect’. However, latent-class models are not recommended for use in comparing assays for acute infections (Branscum et al., 2005) due to ambiguity in interpreting the latent class. Additionally, if the two assays are conditionally dependent (e.g. both measure similar biological processes), then accurate estimation of the sensitivities and specificities of the tests—when used in combination—require additional parameters (the covariances between the test outcomes) to be accounted for (Gardner et al., 2000). The latent-class model approach therefore still relies on one assay being sufficiently well-characterised to provide informative priors. Where these values are unknown, as is the case when utilising existing assays for novel and unknown pathogens, the relative sensitivities and specificities of the two assays are unidentifiable.

Hendra (HeV) and Nipah (NiV) viruses (genus Henipavirus, family Paramyxoviridae) are highly pathogenic, recently emerged viruses with Chiropteran host reservoirs in Australasia (Wang et al., 2000). HeV and NiV soluble G (sG) proteins have been developed and used in highly sensitive multiplexed microsphere binding and inhibition assays on the Luminex® platform (Luminex, Austin, USA), allowing high-throughput multiplexing and, as with ELISA assays utilising the same sG proteins, allowing detection of HeV and NiV antibodies without the requirement of BSL4 laboratories for neutralisation assays (Bossart et al., 2005; 2007; Li et al., 2008). While related henipa- or henipa-like viruses have been detected serologically or by PCR in mainland Africa (Hayman et al., 2008; Drexler et al., 2009; Hayman et al., 2011; Baker et al., 2012; Drexler et al., 2012; Peel et al., 2012; Weiss et al., 2012), no associated virus has been isolated to date and therefore no specific serological assays have been developed. The HeV and NiV sG proteins were found to elicit highly cross-reactive humoral immune responses to known henipaviruses, and the multiplexed assays have therefore been used to screen African bat serum and plasma samples for henipavirus antibodies (Hayman et al., 2008; Peel et al., 2012). While current assays must be used with caution, they have helped improve understanding of the distribution and dynamics of African henipaviruses (Hayman et al., 2008; Peel et al., 2012) until such time that isolates are obtained and specific diagnostic assays developed.

The output values of microsphere binding assays, Median Fluorescent Intensity (MFI), represent intensity of antibody binding on a continuous scale. A previous study reporting henipavirus antibodies using HeV and NiV microsphere binding assays reported raw data without calculating seroprevalences (Peel et al., 2012). While this avoids the difficulties associated with defining a cutoff, presentation of data in this form can cause difficulties and the ability to simplify the data into seroprevalences has its advantages. In other African studies using these assays, in addition to reporting raw MFI values, three times the mean MFI of negative bat or pig sera was used as a threshold (i.e. cutoff) for positive reactivity for the binding assay and sera with an MFI > 200 were considered positive (Hayman et al., 2008; 2011). The same equipment, assay and calculation for cutoff has been used for serological studies in Australasian Pteropus spp. (Plowright et al., 2008; Breed, 2010), although the MFI values and cutoff values used were not reported. It is unclear whether this ‘three times negative’ cutoff is statistically justified, or whether it is valid to apply it across multiple species or across different cross-reactive viruses.

The choice of cutoff has obvious impacts on calculated seroprevalences and therefore interpretation of the data. Standardised approaches, justification of the cutoff chosen, and/or reporting of raw data are required to allow comparisons across studies. In this study, the ultimate objective of determining a cutoff was to enable estimation of henipavirus seroprevalence in E. helvum across multiple sampling events and locations, and in some cases to determine the probability of an individual animal being seropositive. Here, the cutoffs for henipavirus microsphere binding assays for E. helvum fruit bat plasma, generated from multiple methods, were compared. For each of the different cutoffs generated, a fitted mixture model was used to assess the probability of an individual being seropositive or seronegative at that value. The results indicate that the choice of method used, and cutoff chosen, is context-dependent. This study is presented as an example for other studies where reference samples, and assays that have been characterised previously, are absent.

2. MATERIALS AND METHODS

2.1. Sampling

All fieldwork was undertaken under permits granted by national and local authorities, with ethical approval from the Zoological Society of London Ethics Committee (project reference WLE/0489). Plasma samples were collected from E. helvum populations in Ghana, Tanzania, Uganda, Malawi, Zambia, Bioko, Príncipe, São Tomé and Annobón (Appendix A). In São Tomé, bats were obtained in collaboration with local hunters, who hunted at roost sites during the day or at feeding sites at night. Elsewhere, bats were captured at the roost using mist nets as described previously (Peel et al., 2010). Under manual restraint, 1ml blood samples were collected and processed, and morphometric and demographic details were recorded as described previously (Peel et al., 2010; 2012). Age was assessed by morphological characteristics (body size and the degree of genital and nipple development) and all individuals were allocated into one of four age classes: Neonate (<2 months), Juvenile (J; 2 – <6 months), Sexually Immature (SI; 6 – <24 months) or Adult (A; ≥24 months).

2.2. Serological analyses

Virus neutralisation tests were undertaken on all Tanzanian and Annobónese samples, and a subset of samples from Bioko, São Tomé and Príncipe (those with MFIs >750; Table 1). Samples exhibiting virus neutralising at dilutions of ≥ 1:10 to either HeV or NiV were considered positive, which is equivalent to (Hayman et al., 2008; Breed, 2010; Halpin et al., 2011), or more conservative than (Plowright et al., 2008) other studies performed previously in the same laboratory. The neutralisation tests were conducted in a BSL-4 laboratory.

Table 1:

Sample numbers analysed using the Nipah virus (NiV) sG binding assay and Nipah virus and Hendra Virus (HeV) virus neutralisation tests. A= Adult, SI = Sexually immature, J= Juvenile. * indicates biased sample sets, where only samples with microsphere binding MFI>750 were tested using virus neutralisation tests.

Country	NiV sG binding					HeV/NiV virus neutralization tests
	A	SI	J	Neonate	Total
Annobón	42	31	0	0	73	73
Bioko	17	4	66	18	105	49*
Ghana	640	166	76	2	884
Malawi	12	4	0	0	16
Príncipe	41	20	0	0	61	21*
São Tomé	60	15	0	22	97	39*
Tanzania	157	88	0	0	245	222
Uganda	6	0	0	0	6
Zambia	11	0	0	0	11
Total	986	328	142	42	1498	295

In addition to neutralisation tests, all samples were screened for antibodies against henipavirus soluble glycoproteins (HeV sG and NiV sG) using the Luminex® multiplexed microsphere binding assay as described previously (Bossart et al., 2007; Hayman et al., 2008) (Table 1). Briefly, recombinant HeV and NiV glycoproteins were conjugated to internally coloured and distinguishable microspheres, allowing multiplexing. Antibody binding to each microsphere was detected after conjugation of bound antibodies with biotinylated Protein A and fluorescent streptavidin-R-phycoerythrin. Positive and negative controls included in each assay were from wild Pteropus spp. fruit bats previously found to have extreme positive or negative values in microsphere and neutralisation assays. Binding results are given as the MFI value of ≥100 microspheres for each virus type.

To assess inter-laboratory variation, a subset of samples (n=122 from Tanzania) were run in two different laboratories: firstly using a QIAGEN LiquiChip machine (QIAGEN, Manchester, UK) in laboratory A, and secondly, on a Bio-Plex Protein Array System integrated with Bio-Plex Manager Software (v 3.0) (Bio-Rad Laboratories, Hercules , USA), in laboratory B. The laboratory A assays were run first, and the operator at laboratory B was blinded to these results. Where there was sufficient plasma remaining, repeat analyses were performed on the machine at laboratory A after major components of this machine were replaced during servicing.

Microsphere binding assay MFIs were consistently higher against NiV sG than HeV sG, and only NiV binding results are presented here. The correlation of NiV sG binding MFI values between laboratories and after machine servicing was assessed using linear regression using the R statistical package (R Development Core Team, 2012).

2.3. Statistical analyses

Frequency distributions of MFI values were negatively skewed, therefore they were log-transformed (ln) to reduce the skewness prior to further analyses. Four methods were explored to identify an optimal ln(MFI) value to use as a positive/negative cutoff for the NiV binding assay and to enable the estimation of seroprevalences.

Firstly, receiver operating characteristic (ROC) curve analyses were performed using microsphere binding assay MFI values and virus neutralisation test results to derive optimal cutoffs (with the virus neutralisation test considered the reference test; Gardner and Greiner, 2006). Analyses were performed using the pROC library (Robin et al., 2011) in R. Secondly, a cutoff was calculated based on three times the mean MFI of negative control bat sera, as described previously (Hayman et al., 2008). Thirdly, frequency distributions of ln(MFI) values were plotted to attempt differentiation between ‘seropositive’ and ‘seronegative’ populations. Bimodal ln(MFI) frequency distributions were observed, suggesting that it may be that ln(MFI) values for seronegative and seropositive animals could be identified using separate (distinct) probability distributions.

Finally, a mixture model approach was taken to assess this bimodality in a more sophisticated manner (for similar approaches in different systems, see, for example, Greiner et al., 1994; Baughman et al., 2006; Nielsen et al., 2007; Pai et al., 2008). In this approach, if X is a random variable corresponding to ln(MFI), then the distribution of X in a population, f(x), can be modelled as a mixture of the distributions of X for seronegative animals, f_N(x), and seropositive animals, f_P(x), weighted by the proportions of seropositive and seronegative individuals in the population. Therefore,

$$f(x)=p{f}_N(x)+(1-p)f_P(x),$$

where 0 < p < 1 is the proportion of seronegative animals in the population. Given distributional forms for f_N(x) and f_P(x), which are dependent on parameters θ_N and θ_P respectively, it is possible to produce estimates for the proportion of seronegative animals, p, (and therefore also the seroprevalence, 1−p), as well as the unknown distributional parameters, using samples from a single population (or age class). It is possible to improve inference on θ_N and θ_P by utilising samples from multiple populations (or age classes), under the assumption that f_N(x) and f_P(x) do not vary between populations (and so differences in the shapes of the observed distributions between populations are explained solely by changes in p). The underlying assumption taken here that the ln(MFI) distributions are the same in each age group is based on the expectation that while the level of circulating antibodies will vary according to an individual’s infection history and immune response, and the proportion of antibody-positive individuals within each age class may vary according to an age-dependent likelihood of exposure and antibody waning, the actual detection of antibodies when they are present is independent of age class. In applying this model to other studies, the assumptions made here must be carefully considered on a case-by-case basis.

Here the number of components in the mixture model is fixed to two, representing seropositive and seronegative animals. It would also be possible to allow the number of components of the mixture model to vary, and be estimated as part of the model fitting process (for example, Böhning et al., 1992; Greiner et al., 1994; for maximum likelihood approaches to this problem, and Richardson and Green, 1997 for a Bayesian approach). This could be useful to distinguish multiple peaks in the data, however here the number of components was fixed to two for four main reasons: i) biologically it is of interest to classify the data into two groups: seropositive and seronegative; ii) qualitatively the data support the assumption of a two-component mixture, and the fitted plots look reasonable; iii) it is straightforward to derive estimates for the optimal cut-off to determine seroprevalence based on two groups (Böhning et al., 1992; Greiner et al., 1994); and iv) the probabilities of classification into seropositive and seronegative groups can be explored for a range of different cut-offs, providing a useful means of comparison between choices.

For n samples, across S populations, such that n = n₁ + … + n_S, the likelihood can be written as:

$$L({\theta }_N,{\theta }_P,p_1,\dots ,p_S)={\prod }_{s=1}^S\left\{{\prod }_{i=1}^{n_s}[p_s{f}_N(x_i;{\theta }_N)+(1-p_s)f_P(x_i;{\theta }_P)]\right\}.$$

A Bayesian framework was adopted and models were fitted using Markov chain Monte Carlo (MCMC) (for example, Gilks et al., 1996) in the freely-available WinBUGS software (Lunn et al., 2000) using the R2WinBUGS package (Greiner et al., 2000; Sturtz et al., 2005). The underlying ln(MFI) distributions were modelled as normal distributions. Full details of the models used in this manuscript, along with associated WinBUGS code are given in Appendix B.

It is possible to derive a measurement for the probability of an individual (with a specific ln(MFI) value) belonging to either of the two groups by comparing the weighted probability density functions between the two mixture components. For example, the probability that an individual with a ln(MFI) value of x belongs to the seronegative group, is given by:

$$p_N(x)=\frac{p{f}_N(x)}{p{f}_N(x)+(1-p)f_P(x)}.$$ [Equation 1]

The probability that an individual belongs to the seropositive group is p_P(x) = 1 − p_N(x).

If this model is fitted to the data, then a direct estimate of the seroprevalence in the population can be obtained as 1 − p. To estimate the seroprevalence in other datasets, it is desirable to determine an appropriate cutoff, x₀, such that individuals are classified as seronegative if they have ln(MFI) values ≤ x₀, and seropositive if they have ln(MFI) values > x₀. The seroprevalence can then be estimated as Y/N, where Y is the number of seropositive individuals in a sample of size N, according to the cutoff. One option is to choose a cutoff, x₀, such that

$$p{f}_N(x_0)=(1-p)f_P(x_0).$$ [Equation 2]

However, as noted in various papers (Titterington et al., 1985; for example, Böhning et al., 1992; Greiner et al., 1994), this can produce a biased estimate of the seroprevalence, which is exacerbated when the variances of the two component normal distributions are different. An alternative cutoff, which addresses this bias, is to choose a value of x₀ which satisfies the following equation:

$$p{\int }_{x_0}^{\infty }{f}_N(x)dx=(1-p){\int }_{-\infty }^{x_0}{f}_P(x)dx,$$ [Equation 3]

(Titterington et al., 1985; Böhning et al., 1992; Greiner et al., 1994). Using the mixture model approach also enabled calculation of the probability of belonging to either component of the mixture for a range of different ln(MFI) values. This enabled comparison of the predictive capacity of the model (at the individual level) using cutoffs obtained from each of the competing methods. Since the model is fitted in a Bayesian framework, 95% credible intervals for these quantities can also be calculated.

3. RESULTS

3.1. Serological Assays

A good fit was observed between MFI values from microsphere binding assays run in two different laboratories (R² = 0.84) and on one machine before and after machine servicing (R² = 0.88). For simplicity, only binding results from laboratory A prior to servicing are presented here (as these provided the largest data set).

Overall, 60/442 samples were positive at ≥1:10 dilution using the virus neutralisation tests. The minimum NiV sG MFI at which neutralisation was observed was 501, and 100% of samples with a NiV sG MFI > 9300 neutralised either HeV or NiV. A high degree of overlap in the distribution of MFI values between neutralising and non-neutralising samples was observed (Appendix C), yet the proportion of samples that were neutralising increased with increasing MFI (Figure 1).

Figure 1:

Proportion of 402 E. helvum samples capable of henipavirus neutralisation at dilutions of ≥1:10, by a) NiV sG binding assay MFI result b) NiV sG binding assay ln(MFI) result.

3.2. Receiver Operator Characteristic curve analyses

For the purpose of the ROC curve analyses, the virus neutralisation test was considered a reference test. Analyses performed on the results of unbiased sample sets tested by both microsphere binding assay and virus neutralisation tests (334 individual samples from Tanzania and Annobón) identified two potential cutoff values (Figure 2). Firstly, a NiV sG cutoff was selected such that specificity and sensitivity assume maximum values (79% and 93%, respectively; MFI = 1172). The specificity of virus neutralisation tests using HeV (from Australia) and NiV (from Malaysia), however, is likely compromised when testing the neutralising capability of antibodies from Africa, therefore, a second cutoff was also selected so as to give maximum specificity (61%) while maintaining 100% sensitivity (MFI = 501).

Figure 2:

ROC curve analysis of sensitivity and specificity afforded by NiV sG binding assay MFIs for predicting HeV or NiV neutralisation. Two values are highlighted on the curve, with specificity and sensitivity in brackets. To the left is the ‘best’ value for optimal sensitivity and specificity, and to the right, the point that gives maximum specificity while maintaining 100% sensitivity.

3.3. Three times the mean MFI of negative controls

Samples from two Australian Pteropus spp. fruit bats were used as negative controls across multiple assays, and had mean MFI values of 14 (from 6 independent assay runs) and 92 (from 9 independent assay runs). Cutoffs calculated as three times the mean MFI of these samples were therefore 43 and 277, respectively. If all results for both negative bat sera were considered together, three times the mean MFI gave a cutoff of 184.

3.4. Frequency distributions of microsphere binding assay MFI values

Frequency distributions were plotted for natural log transformed NiV sG binding MFI values (Figure 3). Clear bimodal peaks were observed in juvenile and adult bats and, as described in section 2.3, were assumed to represent mixed distributions of ln(MFI) values for ‘seronegative’ and ‘seropositive’ bats (left and right peaks, respectively). The neonate distribution also appeared to be bimodal, but with a more prominent ‘positive’ than ‘negative’ peak. While the distribution was less clear for sexually immature bats, there was a suggestion of a more prominent ‘negative’ peak, which, interpreted together with other sub-adult age classes, suggested a transition from more ‘positive’ to more ‘negative’ with age (i.e. from left to right in the top row of Figure 3).

Figure 3:

Frequency distribution histograms of NiV sG binding assay ln(MFI) values for each age group (Neonate: n = 42, Juvenile (J): n = 142, Sexually Immature (SI): n = 328, Adult (A): n = 986). The red lines correspond to the predictive posterior means generated from the fitted mixture model, fitted to all age groups simultaneously.

Although results gave an indication of the expected distribution of ‘negative’ and ‘positive’ ln(MFI) values, there was a reasonable degree of overlap between the tails of the two distributions in all age groups. It is worth noting that although the magnitude of the peaks differ between the different age-groups, their mean locations are at similar ln(MFI) values. This is consistent with the idea proposed earlier that the underlying distributions of ln(MFI) values for seronegative and seropositive bats should be the same regardless of age, but that the underlying proportions of seropositive and seronegative bats within each age group may differ.

3.5. Mixture model analyses

In the absence of true positive and negative controls for plasma samples containing antibodies to African henipaviruses (required to establish expected ranges and standard deviations of ln(MFI) distributions), a mixture model approach was applied. Table 2 provides posterior mean estimates and 95% credible intervals (CI) for the parameters of the fitted model, generated from 1,000 posterior samples. The seroprevalences are stratified by age-class, as well as for the population as a whole. Model convergence was assessed by visual inspection of the trace plots and the Gelman-Rubin $\widehat{R}$ values; autocorrelation was assessed by examining the effective sample size of the chains; and for each chain the Monte Carlo error for the posterior mean was less than 5% of the posterior standard deviation. It was concluded that these samples were sufficient to make robust inference from, and as such it was not necessary to run the chain for longer.

Table 2:

Posterior means and 95% credible intervals for means and variances of NiV sG binding assay ln(MFI) distributions for seronegative animals (μ_N and ${\sigma }_N^2$) and seropositive animals (μ_P and ${\sigma }_P^2$), as well as for the proportions of seronegative (p) and seropositive (1-p) animals in each age group (and the population as a whole), generated from the fitted mixture model, fitted to all age groups simultaneously (A=Adult, J=Juveniles, Ne=Neonates and SI=Sexually Immature).

	Mean	95% C.I.
μN	3.3	(3.2-3.5)
μP	6.6	(6.5-6.8)
${\sigma }_{\mathrm{N}}^2$	0.59	(0.43-0.79)
${\sigma }_{\mathrm{P}}^2$	1.5	(1.2-1.7)
pA	0.24	(0.2-0.29)
1−pA	0.76	(0.71-0.80)
pSI	0.62	(0.53-0.70)
1−pSI	0.38	(0.3-0.47)
pJ	0.35	(0.26-0.44)
1−pJ	0.65	(0.56-0.74)
pNe	0.16	(0.059-0.3)
1−pNe	0.84	(0.7-0.94)
p	0.33	(0.29-0.37)
1−p	0.67	(0.63-0.71)

The fitted mixture distributions are shown in Figure 3. Table 3 provides the age-specific estimates for the optimal cutoff based on the criterion described in section 2.3, as well as an estimate for the cutoff after combining information on the demography across the whole population.

Table 3:

Posterior means and 95% credible intervals for the optimal cutoff based on Equation 3, for each age-group and for the population as a whole.

	Mean ln(MFI)	95% C.I.	Mean MFI	95% C.I.
Adults	4.3	(4.0–4.7)	74	(55–110)
Juveniles	4.5	(4.1–4.8)	90	(60–122)
Neonates	4.2	(3.8–4.6)	67	(45–100)
Sexually immature	4.7	(4.3–5.1)	110	(74–164)
Total Population	4.4	(4.1–4.8)	82	(60–122)

Results supported the observation in section 3.4 that the proportion of seropositive animals decreases with age in sub-adult classes (posterior means (and 95% credible intervals) for neonate, juvenile and sexually immature seroprevalences of 0.84 (0.7-0.94), 0.65 (0.56-0.74) and 0.38 (0.3-0.47), respectively). To assess the assumption that f_N(x) and f_P(x) do not vary between age classes, the results from a mixture model fitted to all age groups simultaneously were compared to those obtained from fitting the mixture model to each age group separately. Fitting to each age group separately results in a trade-off between the estimates of the parameters of the underlying ln(MFI) distributions and the mixing proportions p_s. Analyses with the neonate age class failed to converge (likely due to small sample size), but on the whole the results were comparable to the analyses with age groups fitted simultaneously (Appendices D and E). Appendix F shows that fitting the model to each age group independently reduced the variance of the negative distribution (σ_N) within each age group, but increased the variance of the positive distribution (σ_P). While the mean of the positive distribution (μ_P) was consistent for adults and juveniles when they were fitted separately compared to fitting across all samples (ln(MFI) = 6.6–6.7), a shift to the left was observed in the sexually immature age class (ln(MFI) = 5.4), with non-overlapping CIs. The mean of the negative distribution (μ_N) was decreased for sexually immature individuals and juveniles, yet the CIs were overlapping across all estimates. Finally, the proportions of seronegative individuals (p) decreased dramatically in the sexually immature age class when the age group was fitted separately, but was consistent for adults and juveniles.

4. DISCUSSION

Here, in the absence of a validated gold standard serological assay to determine the serostatus of an individual against a particular pathogen, the application of established serological assays across non-target host-species and viruses was assessed. Firstly, henipavirus neutralising and binding assay results were compared, and indicated that E. helvum plasma samples were increasingly more likely to be capable of neutralising HeV and NiV as NiV sG binding MFI values increased, but very high MFI values were required before 100% concordance was observed between microsphere binding and neutralisation assays. This may be accounted for by the different binding mechanisms employed in each assay, particularly to a heterologous virus.

Frequency distributions of natural log-transformed MFI values displayed bimodal peaks, yet there was some overlap between the tails of ‘positive’ and ‘negative’ distributions. This likely reflects a natural population comprising individuals with diverse ages, immunocompetence, and pathogen exposure history. Individual bat antibody levels are expected to be dynamic, reflecting factors such as time since last exposure and the total number of exposures over an individual’s life span. Indeed, the proportion of individuals within the ‘negative’ and ‘positive’ ln(MFI) frequency distributions varied according to age class. Larger ‘positive’ distributions were seen in adults and neonates (consistent with natural infection, and transfer of maternal antibodies, respectively). A transition to a more negative distribution was observed as age increased from neonates to sexually immature bats, at a time when maternally-derived antibodies are expected to wane and result in the individual becoming susceptible to natural infection.

To dichotomise an individual’s assay results into ‘seropositive’ and ‘seronegative’ groups, they must be classified according to pre-determined criteria. Therefore, secondly, a specific aim of this study was to determine an appropriate cutoff for henipavirus binding assays in order to differentiate between seropositive and seronegative E. helvum fruit bats. Despite the inherent limitations and difficulties in defining a single ‘cutoff’, this facilitates the calculation of seroprevalences and is routinely used for all serological assays.

In previous E. helvum studies (Halpin et al., 2000; Hayman et al., 2008; 2011), a cutoff based on three times the mean negative bat MFI was used and equated to an MFI of 200. In the current study, ‘negative’ bat samples had highly variable mean MFI readings, highlighting a potential difficulty with using this method across multiple studies and laboratories, and characterisation of controls to allow standardisation across laboratories is required. Additionally, it is important to note that no negative controls are available for E. helvum, as is often the case in wildlife studies (Gilbert et al., in press) and the control samples used in this and previous studies were from wild Australian Pteropus spp bats with an unknown infection history, but which were designated as negative based on results from multiple Hendra virus microsphere and neutralisation assays. Applying a cutoff based on MFI values from these samples from wild-caught bats to different bat genera and to different cross-reactive viruses is problematic.

An alternative approach, where results from a novel assay are compared with a ‘gold standard’ assay such as a virus neutralisation test using Receiver Operator Characteristic (ROC) curve analyses, usually allows the calculation of a threshold that determines the sensitivity and specificity of the screening assay. While HeV and NiV neutralisation assays have been identified as ‘gold standard’ assays for HeV and NiV, respectively (Daniels et al., 2001; Leroy et al., 2005), due to the likely novel nature of the African henipavirus(es), these assays cannot be considered a gold standard for African bat serology. Additionally, it is recognised that the specificity of HeV and NiV neutralisation assays is low, with significant cross-reactivity between them (Bossart et al., 2007; for example, Hayman et al., 2012). A standard approach of selecting a cutoff from the ROC curve so that specificity and sensitivity assume maximum values is therefore inappropriate in this situation. However, a benefit of ROC curve analyses, is that a cutoff can be selected based on this knowledge and according to the specificity and sensitivity required for the testing purpose (Jacobson, 2009; Gardner et al., 2010). Here, an alternative cutoff was considered to be one that would maximise maximum specificity while maintaining 100% sensitivity in the ROC curve analyses, giving a cutoff MFI of 501.

The mixture model approach has the advantage of assessing the microsphere binding assay output in its own right, on a population level, without the need to compare it to an alternative assay with unknown sensitivity and specificity. The bimodal distributions observed in the frequency distributions could be quantitatively assessed in terms of the shape of the distributions and the proportions of individuals within each distribution in a given population. In the absence of prior information on the shapes of the underlying distributions, this approach depends on the distributions for groups being identifiable from the observed data (such as in the adult and juvenile classes). Under certain assumptions this identifiability can be improved by combining information from different populations. Similar results were observed whether the model was fitted individually or to age groups separately. An exception was the positive distribution of the sexually immature individuals, which was shifted to the left. This illustrates the trade-off between p and the distributional parameters, and highlights the potential improvements in identifiability that can be gained by fitting multiple populations together in one model (provided that the underlying assumptions are valid). Given the expectation that a significant proportion of maternally-derived antibody waning is occurring in the sexually mature age class, and that the other age classes are largely unchanged, it was considered that fitting the model to the entire dataset is reasonable, and that using information from all four age-groups provides a more valuable estimate for the seropositivity cutoff.

This decision must be based on the aims of the study, however. If the aim of the study is to estimate population-wide seroprevalences (with the purpose of understanding the proportion of immune and susceptible individuals present within the population) for predicting disease dynamics, then a cutoff estimated across all age classes is likely to be most useful. If, however, the aim is to investigate antibody dynamics within a particular age class (for example, including waning antibodies), then fitting separately to the age group under consideration may be more likely to provide accurate results.

In summary, the potential MFI cutoffs identified using various methods were: 43, 184 and 277 (three times the mean negative bat MFI), 82 (mixture model analyses), 501 (maximising sensitivity in ROC curve analyses) and 1172 (jointly maximising sensitivity and specificity in ROC curve analyses). Clearly there is a large range of potential or ‘optimal’ cutoff values depending on the choice of method used to generate the cutoff, which should depend on the aims of the specific study. One way in which these can be compared is to use the results from the fitted mixture model to assess the probability of belonging to the seronegative or seropositive groups, for each of the different cutoffs (see Figure 4 and Table 4).

Figure 4:

A plot of the posterior mean (and 95% credible intervals) for the relative probability of a given NiV sG binding assay ln(MFI) value belonging to the seronegative, rather than seropositive groups (generated from the fitted mixture model, fitted to all age groups simultaneously). The coloured lines represents the posterior mean probability of being seronegative at each of the cutoffs identified by the various methods.

Table 4:

Posterior means and 95% credible intervals for the probability of belonging to the seronegative group, generated from the fitted mixture model and evaluated at different cutoffs.

MFI	Mean	2.5%	97.5%
43	0.90	0.81	0.96
82	0.60	0.33	0.80
184	0.08	0.01	0.23
277	0.02	0.001	0.07
501	0.003	0.00002	0.007
1172	0.00003	0.00000001	0.0002

For an assay producing a continuous response, there will likely always be some overlap between these two distributions (though if the effect size is large enough and the variability small enough, this could be negligible in some cases). Theoretical results (Titterington et al., 1985; Böhning et al., 1992; Greiner et al., 1994) suggest that for a mixture of two normal distributions, it is possible to generate a cutoff that provides an unbiased estimate of seroprevalence at the population level (based on the fitted mixture model using Equation 3). If this cutoff (in this case, ln(MFI) = 82) is used to generate a probability of belonging to the seronegative group using Equation 1, then the probability that an individual with this ln(MFI) value belongs to the seronegative group is 0.6 (95% CI: 0.33–0.8). The fact that this value is close to 0.5 makes intuitive sense because this threshold is an adjusted measure based around the point of inflexion of the cumulative distribution functions—hence the largest predictive uncertainty is expected to be around this point (note that if a threshold based on Equation 2 is used then this value would be exactly 0.5). That there is a large degree of variability at this point (and therefore the potential for a high potential for misclassification at the individual level at this point) is reflective of the fact that there is considerable overlap between the two component distributions. A benefit to the mixture model approach, not explicitly exploited here, is that since the model is fitted in a Bayesian framework, 95% credible intervals for the cutoff can be calculated and then utilised.

If the assumptions underlying the mixture model are to be believed (i.e. that the two component distributions reflect seropositives and seronegatives), and the optimal cutoff selected by this method is 82, then it can be seen that most of the other methods tend (with the exception of the value of 43) to produce higher cutoff values that are therefore likely to trade a reduced sensitivity (at the individual level) for an improved specificity when classifying individuals. Consequently, they minimise false positive misclassification at an individual level. However, if these higher cutoff values are used to estimate seroprevalence at the population level, then they will most likely underestimate the true seroprevalence. The choice of the most appropriate cutoff is therefore highly context dependent. Ideally, work of this nature, where seroprevalence to a cross-reacting pathogen is being assessed, should be supplemented by specific serological studies that can follow isolation of a more local isolate; this does not yet exist in the situation considered in this work. More detailed quantitative analysis of cross-reactivity to a range of related pathogens can be very informative in this context, as demonstrated for bat lyssaviruses (Horton et al., 2010).

There is a strong justification for using unbiased mixture model approaches to estimation of seroprevalence and in detailed studies of infection dynamics within reservoir host species. However, in other situations, such as where the possibility of confirmed infection spillover between different species is being evaluated, a higher cutoff value, where individual results can be confirmed by ancillary tests (such as serum neutralisation tests) may be easier to justify to public health and related authorities. In this situation, using the higher value of MFI 501 may be appropriate.

5. CONCLUSION

While considering the artificial nature of a strict cutoff given the dynamic nature of antibody levels within populations, this study addressed issues commonly encountered in serological surveys in wildlife, including unknown cross-reactivity between circulating and assay viruses, interpretation of results in the absence of a well characterised, gold standard assay and determining an appropriate positive/negative cutoff point for substitute serological assays.

Seroprevalences will vary according to the baseline sensitivity of the assay used (for example, neutralising assays versus binding assays), in addition to the cutoff selected within each assay protocol. Results here suggest that cutoffs selected using more ‘traditional’ methods such as ROC curve analyses may be producing biased underestimates of seroprevalence, if the assumptions regarding the use of a two-component mixture model to capture the underlying distributions of ln(MFI) values in seronegative and seropositive individuals is considered reasonable. Full reporting of test methodology and results are therefore critical to allow comparisons with other studies, including a description of precise protocols or modifications employed, cutoffs, controls and uncertainty intervals for antibody prevalence estimates. While MFI values were strongly correlated across laboratories and machines in this study, the absolute MFI values varied. As with any assay, the cutoff is subject to measurement bias in the particular instrument or environment under which the assay is performed, only here it is more obvious since the results are given on a continuous scale rather than as a dilution series. Therefore, if new E. helvum samples were analysed on the same machine as used here, then the cutoff analyses performed here would be applicable. However, we recommend that similar analyses are performed for each new species of bats where the virus concerned is uncharacterised, or for each new machine or laboratory in which the microsphere binding assay is performed. If using a mixture model with data where age groups are identifiable, it is advisable to consider that the underlying distribution of seropositive and seronegative individuals may differ in some cases and, depending on the objectives of the study, fitting to a specific age group may be required.

Abbreviations

HeV

Hendra virus

NiV

Nipah virus

MCMC

Markov chain Monte Carlo

MFI

Median fluorescence intensity

ROC

Receiver operating characteristic