Stereoscopic Video Analysis of Anopheles gambiae Behavior in the Field: Challenges and Opportunities

Abstract

Advances in our ability to localize and track individual swarming mosquitoes in the field via stereoscopic image analysis have enabled us to test long-standing ideas about individual male behavior and directly observe coupling. These studies further our fundamental understanding of the reproductive biology of mosquitoes. In addition, our analysis using stereoscopic video of swarms of the African malaria vector Anopheles gambiae have produced results that should be relevant to any “release-based” method of control including Sterile Insect Technique (SIT) and genetically modified male mosquitoes (GMM). The relevance of the results is primarily due to the fact that any mosquito vectors released for control are almost certainly going to be males; further, for SIT, GMM or similar approaches to be successful, the released males will have to successfully locate swarms and then mate with wild females. Thus, understanding and potentially manipulating the mating process could play a key role in future control programs. Our experience points to special challenges created by stereoscopic video of swarms. These include the expected technical difficulties of capturing usable images of mosquitoes in the field, and creating an automated tracking system to enable generation of large numbers of three dimensional tracks over many seconds of footage. Once the data are collected, visualization and application of appropriate statistical and analytic methods also are required. We discuss our recent progress on these problems, give an example of a statistical approach to quantify individual male movement in a swarm with some novel results, and suggest further studies incorporating experimental manipulation and three dimensional localization and tracking of individual mosquitoes in wild swarms.

1. Introduction

Our understanding of the mating process in swarming mosquitoes such as Anopheles gambiae is limited by the difficulty in quantitatively observing the mating behavior of males and females when and where it occurs naturally. Recent advances in our ability to localize and track individual swarming mosquitoes in the field via stereoscopic image analysis now enable direct observation and quantification of mating behavior (Butail et al., 2012).

Anopheline swarms usually form at dusk (though dawn swarms have also been observed; see Charlwood et al., 2002), composed almost entirely of males. They are considered mating aggregations in which males compete for access to females (Downes 1969; Yuval 2006). An. gambiae swarms often form over a “swarming marker” and nearly all mating occurs in swarms (Dao et al., 2008). Swarms change in size over the period of 10–30 minutes during which they occur, and have a maximum size ranging from a half dozen to thousands of males (Manoukis et al., 2009; Diabate et al., 2011). Markers are usually an object such as a donkey cart or a contrasting area such as the opening of a well (Howell and Knols, 2009). In Mali, where we conducted our work, the sub-species of An. gambiae known as “molecular forms” are known to swarm over different markers: the M molecular form swarms over visually evident markers such as the ones described above, whereas the S form is almost always found over bare ground (Diabate et al., 2009; Manoukis et al., 2009).

Direct observation and quantification of swarming and mating has yielded insight into the fundamental biology of mating in An. gambiae, including information on swarm structure and a mathematical characterization of individual male movement (Manoukis et al., 2009; Butail et al., 2013). Having a measurement-based description of what occurs in nature is especially important because it enables us to form new hypotheses about mating and to test them with or without stereoscopic video analysis. Basic characterization and subsequent experiments will have important implications for any “release-based” method of vector control including sterile insect technique (SIT) (Knipling, 1979), release of insects carrying a dominant lethal gene (RIDL) (Phuc et al., 2007; Thomas et al., 2000), methods involving cytoplasmic incompatibility such as via Wolbachia (Zabalou et al., 2004), and genetic modification of the vector such as to make it refractory to the malaria parasite (Scott et al., 2002). All of these interventions depend on the introduction of laboratory-reared mosquitoes into nature that will be able to locate the appropriate mating site, swarm with wild males and then mate with wild females, but the ability of colony-reared Anopheles males to perform these steps in nature is currently unknown (Boëte and Koella, 2003; Catteruccia et al., 2003; Marrelli et al., 2006). Results from other species indicate that mating behavior may be sufficiently modified in colony insects to severely limit mating with wild females (Riesen et al., 1985).

Consequently, understanding and potentially manipulating the mating process could play a key role in a variety of control programs. Our experience points to special challenges created by stereoscopic video of swarms. These include technical difficulties of capturing usable images of mosquitoes in the field and creating an automated tracking system to enable the generation of large numbers of three dimensional tracks over many seconds of footage. Once the data are collected, visualization and application of appropriate statistical and analytic methods also present special challenges. We will discuss progress on these problems, give an example of an analytical approach to describe the movement of individual males and suggest further studies incorporating experimental manipulation and three dimensional localization and tracking of individual mosquitoes in wild swarms.

2. Challenges and Solutions

Stereoscopically videographing swarms (“Capture”), localizing and tracking individual swarming mosquitoes, and analyzing the position and movement of these mosquitoes pose special challenges. In this section we review some problems and solutions for capture, tracking and analysis using a statistical and frequency domain based approach. For complete details on the first two challenges (capture and tracking), please refer to Butail et al. (2012).

2.1 Capture

The first and most obvious challenge to quantifying the positions and movements of individual mosquitoes in swarms is capturing the data visually. An. gambiae individuals are relatively small, they move at speeds of about 0.5–1 m/s, and exhibit mating behavior primarily at dusk or occasionally at dawn, when light levels are low. These conditions call for careful consideration of videographic equipment to be used.

High-definition Charge-Coupled Device (CCD) sensors today have sufficient resolution and light sensitivity to allow imaging of individual flying mosquitoes while covering a small to medium size swarm (5–40 individuals). Our practice has been to image the swarm from a low vantage point in the direction of the setting sun, using the sky as a background. On a clear day, mosquitoes will appear as dark spots or streaks against a light background. If there are clouds, these should be kept out of the frame if possible, as they require an adaptive background-subtraction technique (Butail et al., 2012).

In order to perform stereoscopic localization, images must be acquired from two cameras simultaneously. Any deviation in the capture time will make correspondence harder (Hartley and Zisserman, 2000) and introduce error in position estimation. Our solution has been to use cameras that can be externally triggered to take an image simultaneously (i.e., an external device sends an electrical signal to the two cameras simultaneously). Most machine-vision cameras connected via a Camera Link (CL) connection can be triggered in this manner by a “frame-grabber”, which also records the image pixel data to disk. A complete capture system with cameras, lenses, frame-grabber and notebook computer can be assembled for around US$10,000.

It is important to note that each camera must be “calibrated”; its optical characteristics measured in order to allow localization of objects in the field of view of both cameras. We used the MATLAB Calibration Toolbox (Bouguet, 2004) each day before filming, and found this to be relatively efficient.

2.2 Localization and Tracking

For localization and tracking we apply quantitative tools from computer vision (Hartley and Zisserman, 2000; Milan et al., 2002) and estimation theory (Bar-Shalom, 1987; Cox, 1993; Veenman et al., 2001) to reconstruct three dimensional tracks of swarming mosquitoes from video footage. The tracking framework takes as input a sequence of video images and produces the estimated position and velocity of individual mosquitoes over the sequence. Complete details on the tracking algorithm are given in Butail et al. (2012), so only an overview is presented here.

Automated processing of sequential video images (Fig. 1a) extracts each mosquito’s three dimensional position and velocity. The first processing step enhances each image by reducing noise and increasing contrast (Fig. 1b) and then applies a threshold over a sliding average of five nearest frames to isolate individual insects (Cavagna et al., 2008) (Fig. 1c). After this step, an insect appears in the field of view as an elongated white blob: the position of the blob in the image corresponds to the position of the insect in three dimensional space; the elongation of the blob corresponds to the component of the insect’s velocity that is parallel to the image plane. We model mosquito motion as a constant-velocity Markov process with random perturbations (Bar-Shalom, 1987), which allows the tracking software to predict where in the next pair of images the blobs are expected to be seen.

Figure 1.

Probabilistic reconstruction of mosquito position and velocity from automated processing of images. a) original raw image; b) enhanced image (contrast/brightness); c) thresholded image.

Based on position and velocity of tracked mosquitoes, missing measurements in the track (if they exist) are sought by lowering the threshold in the region where the blob is expected to occur. The next step validates the position and tracking estimate generated by the epipolar constraint (the geometry of stereo vision) and then updates the mosquito position such that it is consistent with the individual motion model and minimizes the error between the predicted value and the measurement. Blobs that are expected to arise from multiple mosquito motions are split into individual streaks by fitting elongated streak shapes onto the pixels until the blob is completely covered. Our automated process to address the matching problem is not guaranteed to work at all times due to occlusions between mosquitoes that are yet undetected, or fading out of view, so the tracks are later verified by a human analyst. The complete, supervised tracking system is relatively fast; an experienced operator can track an individual mosquito for 60 seconds in under 30 minutes of work.

2.3 Analysis

Analysis of tracking or even position data poses special challenges. The first of these is that the quantity of data can be large. Manoukis et al. (2009) estimated the positions of all visible mosquitoes every 15 seconds for 12 swarms, resulting in about 5,000 data points. Butail et al. (2013) created tracks at 25 frames per second for all males in 10 swarms, a dataset of approximately 62,000 points. As the process of tracking becomes more common and efficient, we expect the dataset size to continue to increase.

A second analytical problem is deciding what methodological approach to use when asking questions of tracking data. There are three dimensional spatial approaches available for studying positions, movement, and correlations between them (for example, pursuit analysis; see Wei et al., 2009), but these are not commonly used by entomologists and so may be unfamiliar. A lack of common methods can make it difficult to evaluate the significance of results generated by the analysis of position or movement data on mosquitoes.

Finally, the data on position and tracking of An. gambiae in the field thus far has been observational, rather than produced in response to a priori hypotheses. The lack of hypotheses was a particularly acute when the fundamental quantitative characteristics of swarms in the field were unknown, but we expect this to be a smaller problem in future studies (see section 4).

3. Statistical Analysis of Male Movement

One of the visually obvious patterns observed in the movement of individual male An. gambiae when they are swarming is that there are oscillations in all three dimensions (Figure 2). It is further apparent that there might be a difference between the period of oscillations in the horizontal directions compared with the vertical. Butail et al., (2013) quantify these oscillations by fitting them to a second-order model of individual movement that includes a linear combination of three forces: an external force (representing the swarming marker or an attraction to the centroid of the swarm), a damping force (representing aerodynamic drag) and a (random) force arising from interaction with some number of neighboring individuals. An alternative analysis is presented here of the movement of males in six swarms filmed in 2010 (Table 1).

Figure 2.

Positions of individual male mosquitoes in the East, North and vertical directions over time. Each row represents a single male filmed in the 21 August 2010 swarm.

Table 1.

Details on swarms analyzed, periods detected by spectral analysis and orientation of the first principal component of the horizontal oscillations.

Swarm	Temp (C)	N	t (min-max); sec	P [PC1 H] (SD); sec	P [V](SD); sec	L [East]	L [North]
21 Aug 2010	26.6	17	28.5 (2.0–59.7)	2.09 (0.84)	6.13 (1.11)	0.53	0.78
25 Aug 2010	16	6	19.4 (16.0–20.0)	3.33 (-)*	5.33 (1.73)	0.41	0.89
26 Aug 2010	28.8	10	22.5 (3.2–32.2)	1.10 (0.45)	3.71 (0.69)	0.45	0.85
28 Aug 2010	28.1	18	22.3 (1.2–34.3)	2.13 (0.69)	4.32 (1.40)	0.77	0.63
29 Aug 2010	24.1	21	28.6 (1.2–33.8)	1.86 (0.74)	4.13 (1.07)	0.40	0.81
01 Sept 2010	-	14	17.5 (7.5–19.1)	2.18 (1.08)	4.22 (3.43)	0.54	0.91

N = number of males tracked; t = mean track length; P [PC1 H] = mean period of the first principal component of the horizontal directions; P[V] = mean period of the vertical direction; SD= Standard Deviation; L [East/North] = mean PCA loadings for East and North directions;

^*horizontal spectral peak detection failed for all but one male track in this swarm due to short track lengths, so SD is not available.

First, although the horizontal directions in Figure 2 are the cardinal directions of the compass, it is not clear that these directions are biologically meaningful. We can collapse the horizontal directions into a single dimension via principal components analysis (PCA) of each male’s track, so that we capture the major axis of horizontal movement for each male. The orientation of this axis can itself be analyzed to test whether there is a particular alignment that is more or less common in swarming males, perhaps in response to the setting sun. For the six swarms analyzed, the average loadings of the Eastern and Northern components for the first principal component (PC1) and its proportion of variance are given in Table 1; these results indicate that in 5 out of 6 cases the Northern direction includes most of the variance in position (oscillations).

Second, the period of the oscillations in PC1 of the horizontal and in the vertical direction can be estimated by spectral analysis. (A good introduction to spectral analysis is given by Venables and Ripley (2002) chapter 13, and a more detailed exposition by Shumway and Stoffer (2006).) We estimated the spectral density of the track (a time series) by a smoothed periodogram, calculated via a Fast Fourier Transform (FFT) plus a modified Daniell smoother as implemented in spec.pgram (Venables and Ripley, 2002) for R (R Core Team, 2012). Figure 3 shows the spectral analysis for four representative males from the swarm filmed on 21 August 2010. We note that this method is statistically sound but demands long tracks (around 20 sec) with relatively little noise in the position estimates.

Figure 3.

Sample spectral density plots for four individual males from the 21 August 2010 swarm. The top row of plots shows period against spectral density for the first principal component of the horizontal movement; the bottom row shows the vertical. Track lengths for each individual are given in the top row.

We observed that the swarm-averaged period of the vertical (P[V]) and PC1 horizontal (P[PC1 H]) oscillations are significantly different (paired t test, t=7.965, df=5, p<0.001) with P[V] longer than P[PC1 H]. In addition, we found that the two periods are positively related by a fixed factor of approximately 2 (linear regression with zero intercept: P[V]=s*P[PC1 H]; s=2.064, SE=0.232, t=8.893, p<0.0003, Residual SE=1.259 on 5 degrees of freedom, adjusted R²=0.929), though there is a lot of variation at the individual level. There is no a priori reason to hypothesize that vertical and horizontal oscillation periods are related across swarms, or that they would be significantly different in this manner. Indeed, swarms of Aedes albopictus filmed in Mauritius show predominantly vertical oscillations and relatively little horizontal displacement – the opposite pattern of what we describe here (J. Gillies, personal communication). The observed relationship between vertical and horizontal oscillations in An. gambiae may be caused by swarm-level characteristics or abiotic drivers, perhaps the overall number of swarming males (Diabaté et al., 2011; Manoukis et al., 2009) or the ambient temperature, though we did not detect either relationship here. Patterns may also be affected by other insect species. The existence of a characteristic oscillatory behavior in swarms may be an important mating characteristic that could be selected for (or engineered into) males slated for eventual release.

4. Prospects for Future Studies

Today we have a quantitative description of several important characteristics of mating aggregations in An. gambiae. We know details about how the swarm is organized, and how males move within it and some information regarding couple formation within naturally occurring swarms. The next step is to conduct experiments with swarming males in the field or under semi-natural conditions, though forming swarms similar to natural ones under controlled conditions is challenging, even in large enclosures (Knols et al., 2002).

Previous studies have included some sort of experimental manipulation, like moving markers or removing males (Charlwood and Jones, 1980; Charlwood et al., 2002), but now that we have the ability to reliably localize and track individuals in the field new possibilities arise. These include introducing artificial wind, as (natural) wind has been shown to have an effect of swarm coordination (Butail et al., 2013). Another possibility is to create an artificial female decoy that produces a “wing beat” acoustic frequency similar to that of a real female mosquito when ejected at a velocity of about 1 m/s. This could be sent through the swarm and the response of individual males observed.

The ability to identify mass-reared males in the video footage would be extremely useful; this would enable researchers to directly watch the “dance” of the released males in the context of a swarm of wild males. It is possible that such identification could be accomplished by means of IR-sensitive dyes. This capability will be especially useful as we begin to test the participation of released males in natural swarms (Hassan et al., 2014, this volume).

Finally, the creation of a synthetic swarm (Butail et al., 2013) opens up the possibility of observing the behavior of a single An. gambiae in a “swarm simulator” (Fry et al., 2008) using virtual reality. An experiment of this type would tether the mosquito being studied and present it with visual and tactile stimuli simulating flight through a swarm, then record the insect’s responses; these responses would in turn drive the simulation parameters.

5. Conclusion

Quantitative study of the swarming behavior of mosquitoes has been successfully conducted by researchers for some time (Gibson, 1985; Ikawa and Okabe, 1997), with recent progress making such studies more accessible than ever before. Despite this, challenges remain for wider access to these tools. In this paper we have outlined three areas of potential difficulty (image capture, localization and tracking, and analysis) and several ways to address them. We have also demonstrated an analytical approach using statistical techniques that can lead to novel insight into swarm organization in An. gambiae. Our hope is that other researchers will use these methods to conduct experimental studies of this vector species in the field or in field enclosures to benefit release-based control methods such as SIT and GMM.

Highlights.

• Tracking individual swarming mosquitoes is important for basic and applied reasons

• Challenges include image capture, tracking methods and analysis of resulting data

• We review approaches to address problems in those areas

• We find that vertical oscillations of males are slower than horizontal

• Results like these will be important for release-based control methods