Automated Monitoring and Analysis of Social Behavior in Drosophila

Abstract

We introduce a method based on machine vision for automatically measuring aggression and courtship in Drosophila melanogaster. The genetic and neural circuit bases of these innate social behaviors are poorly understood. High-throughput behavioral screening in this genetically tractable model organism is a potentially powerful approach, but it is currently very laborious. Our system monitors interacting pairs of flies, and computes their location, orientation and wing posture. These features are used for detecting behaviors exhibited during aggression and courtship. Among these, wing threat, lunging and tussling are specific to aggression; circling, wing extension (courtship “song”) and copulation are specific to courtship; locomotion and chasing are common to both. Ethograms may be constructed automatically from these measurements, saving considerable time and effort. This technology should enable large-scale screens for genes and neural circuits controlling courtship and aggression.

INTRODUCTION

How are innate behaviors programmed into the genome? Answering this question requires identifying the genes that control specific behaviors, the neural circuitry on which they act, and how this circuitry controls behavior^1–5. This may be attempted in model organisms such as the nematode Caenorhabditis elegans and the fruit fly Drosophila melanogaster, thanks to the abundant genetically based tools for marking, mapping and manipulating specific populations of neurons^6,7, thereby enabling large-scale genetic and functional screens^8,9.

Social behaviors, such as courtship and aggression, are of particular interest, because they have strong innate components. Mating has been studied in both Drosophila and C. elegans using a combination of molecular genetic and cellular approaches^5,10. Drosophila is unique amongst invertebrate genetic model organisms in that it exhibits both courtship and aggression^11–14.

Measuring animal behavior is difficult. Both aggression and courtship consist of rich ensembles of stereotyped behaviors, which often unfold in a characteristic sequence^13,15. Currently these behaviors are measured manually, which is slow and laborious. Subjective decisions by the observer may lead to difficulty in reproducing experiments. Furthermore, human observers may fail to detect behavioral events that are too quick or too slow, and may miss events due to flagging attention. These constitute substantial obstacles to conducting large-scale behavioral screens.

These limitations could be overcome through automation. The first step towards measuring behavior is tracking (i.e., measuring the position and orientation of animals over time). Machine vision systems have been designed for tracking houseflies¹⁶, mice¹⁷, ants¹⁸, bees¹⁹, single Drosophila in 3D²⁰ and for measuring Drosophila locomotion^9,21,22 showing promising accuracy and flexibility. However, we do not yet have systems for measuring complex behaviors automatically. A machine vision apparatus that detects lunging, an aggressive behavior in Drosophila, has been developed recently²³. In addition to lunging, it would be desirable to measure other aggressive behaviors, such as chasing, tussling, boxing (fencing) and wing threat¹³, as well as a suite of courtship behaviors. This would allow the study of whether a given mutation or environmental influence exerts a selective effect on aggression, or on social interactions in general²⁴.

Here we describe a machine vision system designed to quantify and analyze a broad spectrum of behaviors associated with both aggression and courtship in Drosophila. We designed our system, first, to allow detailed, accurate and reproducible quantitative measurements of individual component behaviors (“actions”) that are expressed in courtship and/or aggression (“activities”); and second, to enable large-scale genetic and circuit-perturbation screens. The system is simple and inexpensive to build and replicate, functions automatically and permits measurements of multiple fly pairs simultaneously. The application of this approach, together with appropriate multiplex aggression arena configurations²⁵ (see also Supplementary Methods online), should enable large-scale genetic and circuit-based screens of these behaviors.

RESULTS

We used a double-arena adapted from a recently developed system ²³ (Fig. 1a). A consumer camcorder, connected to a PC, filmed one pair of flies per arena. This setup can be adapted to a medium-throughput mode, consisting of four double-arenas (Fig. 1b–d). Additional details in the Supplementary Methods.

Figure 1. Imaging setup for genetic screens in Drosophila. (a).

Lateral cut through our double-chamber (all lengths in millimeter). (b) Example of a high-throughput behavioral screening assay - 4 double-chambers, 4 cameras, 2 PCs, and standard video-acquisition software. (c) Double-chamber with the walls removed to expose the floor. (d) Camera view of the double chamber. Each of the two arenas has food in the centre, surrounded by agarose; walls are coated with Fluon. Bar, 10 mm.

Our software consists of six modules: video import, ground-truthing, calibration, fly detection and tracking, action detection and graphical user interface. Fly detection and tracking, as well as action detection, are described here and in Figure 2, for other modules see Supplementary Methods.

Figure 2. Detection and tracking of fruit flies. (a–c).

Fly segmentation procedure. (a) ‘Foreground image’ F_I, computed by dividing the original image I by (μ_I + 3σ_I) (F_I values in false-colors). (b) The fly body is localized by fitting a Gaussian mixture model²⁶ (GMM) with three Gaussians (black curves; background (dashed), other parts and body (solid)) to the histogram of F_I values (gray curve) using the Expectation Maximization (EM) algorithm²⁶. First (top) and final (bottom) iterations of the GMM-EM optimization. All pixels with brightness values greater than the value at the intersection of the solid black Gaussians (orange areas) are assigned to the body, and are fit with an ellipse. (c) Full fly detection by segmenting the complete fly from the background, with body parts and wings (empirically represented as four segments/colors)²⁷. First iteration (top) and final result (bottom). (d) Head and abdomen are resolved by dividing the fly along the minor axis b and comparing the brightness-value distribution of both halves (head is brighter; see also c). The wings (l_L, l_R, φ_L, φ_R) are measured by detecting, on each side of the fly’s posterior half, the pixel with the furthest distance from the center of the ellipse (wing tip) in the segmented full fly. (e) Definition of fly orientation Θ and moving direction Θ_move. (f) Separation of occluding (pair 1) and touching flies (pair 2). Original image (left), foreground image (center), and final segmentation result (right) with the corresponding ellipses. (g) Definition of additional features. Bars, 1 mm.

Detection and tracking

The first step in tracking the flies is computing their silhouette (body and wings) (Fig. 2a–c). An important and novel feature of our software is the ability to detect and measure the position of the fly’s head and wings, as illustrated in Figure 2c,d. Computing the orientation θ of a fly is described in Figure 2e. The bodies of abutting flies are resolved by fitting a two-component Gaussian mixture model²⁶ simultaneously to pixel location and brightness (Fig. 2f). For details see Supplementary Methods.

At each video frame 25 measurements are computed, characterizing body size, wing pose, and position and velocity of the fly pair (Fig. 2d, e, g and Supplementary Table 1 online). These measurements are the features used to detect actions.

In male-female assays the flies’ identity is directly measurable since the female is larger. In male-male assays a white dot is painted on the back of one fly to allow individual identification. For unlabeled male-male pairs our software computes the most likely fly-specific trajectories (see Supplementary Methods).

Action detection

The twenty five features defined in the previous section are used for detecting fly actions. Lunging, tussling and wing threat are actions specific to aggression; wing extension (courtship “song”), circling and copulation are specific to courtship; locomotion and chasing are common to both.

A lunge is defined as one fly rearing briefly on its hind legs and snapping down onto the other fly²³ (Fig. 3a and Supplementary Fig. 1a online). Lunging is detected automatically by an example-based classifier in a two-step process (Supplementary Methods for details). First, probable lunges are selected amongst all frames by using ranges (intervals) on feature values as listed in Supplementary Table 2 online. Second, probable lunges are accepted/rejected by a k-nearest-neighbor classifier²⁶ using 10 features (see Supplementary Table 3 online). The training examples consist of ≈250 distinct expert-selected lunge events occurring over 8 fly pairs × 20 minutes of recorded video, and of a comparable number of negative examples.

Figure 3. Detectable actions.

(a–d) Single shots of side and top views of (a) lunging, (b) tussling, (c) wing threat, and (d) copulation. High-resolution sequences of (e) wing extension and circling, as well as (f) chasing (see also Supplementary Videos 1–7 online). Time index in seconds is relative to the first frame in each movie clip. Bars, 1 mm. Supplementary Figure 1 online shows the same actions as detected by our system.

We evaluated the performance of our lunge detector on a 20-minute movie containing a large number of lunges (139). Ground-truth, i.e. the accurate identification of all lunges in a movie, was obtained following a two-step process (see Materials and Methods and Supplementary Methods). We compared both the algorithm’s performance, as well as a second expert’s annotations, to ground truth. Figure 4a shows the receiver operating characteristic²⁶ (ROC) representing the fraction of false-negatives (lunges present in the ground truth, but not detected) vs the number of false-positives (detected, but not present in the ground truth), as the number of lunge-labeled nearest neighbors that are necessary to declare a lunge is varied. The threshold we selected for labeling a lunge yields a detection rate of ≈91%, representing 126 correct detections, with 13 false-negatives, and 7 false-positives (encircled). Lowering the detection threshold decreases the probability for false-negatives and increases the number of false-positives.

Figure 4. Performance of action detection. (a).

Performance of our lunge detector, described by the Receiver Operating Characteristic (ROC). Each ROC curve gives the fraction of false-negatives (number of missed lunges divided by the total number of lunges on the ground truth) vs the number of false-positives. Curves are shown for different values of k-nearest neighbors constant k. Best performance is achieved for k = 15. The operating point of the system is shown by a black circle; it is obtained by labeling an action ‘lunge’ when 12 or more of its k = 15 nearest-neighbors are lunges. The performance of an expert human observer (40% missed lunges) is indicated by the gray cross. The expert detected 84 lunges, while our two-step process for establishing ground-truth (see Methods) yielded 139 lunges. (b) Comparison between automatically measured lunges and ground-truth. Each dot represents the number of lunges detected automatically (Y axis) vs the number of lunges in the ground truth (X axis) for each of 56 20-minute movies of fly pairs. Note that automatic counting is very close to ground-truth both when there are many lunges and when there are few.

We applied the lunge detector to 56 additional fly pair movies (Fig. 4b) and found excellent agreement with the ground truth, with a correlation of 0.99, a bias of 1.5 lunges and a standard error of the mean of 0.40 lunges.

The detection of other aggressive and courtship actions could be achieved by using only the first of the two steps in the example-based classifier algorithm described above. Features and ranges for each action were determined empirically by expert analysis of movies containing sample actions. The k-nearest-neighbor classifier step was unnecessary in these cases.

In aggressive tussling, both flies grip each other with their front legs¹³ (Fig. 3b and Supplementary Fig. 1a online). The bodies face each other so that their axes of symmetry are parallel (body alignment) and form a single line. Connected solidly in this configuration, they move about in jerks at high velocity and acceleration. In order to detect and classify aggressive tussling, 8 features are compared to ranges as listed in Supplementary Table 4 online. When all features are within their empirically determined ranges and this configuration is maintained for 0.3 s or longer we flag an aggressive tussling event.

Wing threat is characterized by a lateral extension of both wings by 80°–90° followed by their elevation to a vertical extension of ≈40° (Fig. 3c and Supplementary Fig. 1a online). We observe both rapid, transient elevation of the wings, as well as longer-lasting (≥ 0.3 s) occurrences; the latter are typically, but not always, associated with a reduction in walking speed (velocity ≤ 5 mm/s). In this study we have restricted detection of “wing threat” to longer-lasting occurrences, as they are more easily discriminated from rapid “wing flicking” (see Supplementary Table 5 online for features and ranges).

During courtship, males extend their wings laterally and vibrate them (≈280Hz) to produce a “courtship song” (Fig. 3e and Supplementary Fig. 1b online). Wing extension is detected when the angle between the wing and the long axis of the fly body is greater than 60°, the fly is not standing up (fly length, as viewed from the camera, is maximal), and this configuration is maintained for 1 s or longer (see Supplementary Table 6 online for features and ranges).

Circling is a behavioral action that is part of courtship, and is detected when one fly drifts sideways in a circle with approximately constant velocity around the other fly (Fig. 3e and Supplementary Fig. 1b online). Supplementary Table 7 online lists features and ranges.

Copulation involves a male fly approaching and mounting a female fly (Fig. 3d and Supplementary Fig. 1b online). The beginning and end of copulation are characterized by an abrupt change in the distance between the two flies. During copulation their movements become coupled and locomotion is decreased. Thus, to detect both the starting and ending time-points of copulation, we compute the mean and standard deviation of inter-fly distances Δ_c within a moving 250-frame (8.3 s) window. The earliest frame when the criteria of mean distance < 2 mm and standard deviation < 0.3 mm are simultaneously met is defined as the “copulation start”. The last such frame is defined as the “copulation end”. Supplementary Table 8 online lists features and ranges.

Chasing is detected when the change of the head-center distance δΔ_h-c between both flies (see Fig. 2g) is small, both flies have the same, constant velocity, the distance between the flies is small but not zero, the chasing fly is oriented towards the chased fly, and the head of the chasing fly is behind the chased fly’s abdomen (Fig. 3f and Supplementary Fig. 1c online). This configuration has to be maintained for 1 s or longer. Supplementary Table 9 online lists features and ranges.

The performance of our system in detecting the actions described above was evaluated using the same approach as in the case of lunging. For all of these behaviors we measured detection rates between 90% and 100% (Table 1).

Table 1.

Performance evaluation of action detection.

	# of fly pairs	# of events	correct positives	false negatives	# of false positives	# of false positives	# of false positives
Behavior			in percent			per 20′ movie	per event
Lunginga, b	1	139	90.7	9.3	7	7	0.05
Tussling	40	176	-	-	13	0.33	0.07
Wing Threata,c	40	87	94.3	5.7	4	0.1	0.04
Wing Extensiond,e	10	797	96.7	3.3	35	3.5	0.04
Circlingd,f	10	422	99.8	0.2	18	1.8	0.04
Chasingg	6	400	98.0	2.0	4	0.67	0.01

^awild-type (CS) male-male fly pairs.

^b56 additional pairs were tested and the correlation with ground-truth was 0.99 (see text).

^c118 hand-counted wing threats. 87/118 lasted longer than 0.3s. False-positives are ambiguous situations of wing threat or common wing extension.

^dwild-type (CS) male-female fly pairs.

^e906 hand-counted single wing extensions. 797/906 lasted longer than 1 s. False-positives were due to segmentation errors.

^f422/435 hand-counted circling events had a minimum length of 1 s.

^gCha-Gal4;UAS-tra male-male fly pairs. Minimum duration 1 s.

Automated analysis of genetic and environmental influences on aggressiveness and courtship

To validate the utility of our system for studying experimental perturbations of courtship and aggressive behavior, we first investigated whether it could detect previously described phenotypes produced by gene- or circuit-level manipulations. Octopamine (OA) is an insect biogenic amine, which is closely related to mammalian noradrenaline. It plays a critical role in aggressive behavior in Drosophila¹⁴. A recent study showed that silencing of OA neurons decreases aggressive behavior²³. We performed a similar manipulation, by expressing the inwardly rectifying potassium channel Kir2.1²⁸ in tyrosine decarboxylase-2 (Tdc2)-expressing neurons in order to suppress their electrical activity. Tdc2-Gal4; UAS-Kir2.1 flies showed significant decreases in lunging, tussling and wing-threats (Fig. 5a–d, Supplementary Fig. 2a and 3a,b online). There was no statistically significant change in chasing, or total distance traveled (Supplementary Fig. 2b,c online).

Figure 5. Genetic and environmental influences on aggressive and courtship behavior. (a,g).

Number of lunges, (b,h) tussling, (c,i) lunges per meter, (d,j) wing threats, (e) circling, (f) wing extensions. All panels show mean and standard error of the mean. (a–d) Octopamine control (tdc2/+) and mutant (tdc2/kir). (e–j) CS male-male, male-female, and Cha-Gal4;UAS-tra (“Cha-Tra”) male-male pairs. (k–m) Ethograms, based on transition matrices. The ethograms show transitions where the interval between a fly’s action and the next lasted ≤ 10 seconds. We count intervals > 10 seconds without action as ‘no action’ nodes (not shown). The transition probability is represented by the thickness of the arrows (normalized over all arrows that exit a node including the arrow into ‘no action’). The arrow stumps represent the transition probability from one action into the same action. Circle diameters (logarithmically scaled) and numbers denote the average action frequencies. (n) Frequency of CS male-fly positions while extending a wing towards a decapitated CS female. Left: group-housed flies (GG, n = 10); right: single-housed flies (SS, n = 10). (o)

We also examined flies bearing a mutation in fruitless (Fru), a sex-specifically spliced transcription factor that specifies gender-dimorphic fly behaviors^2,4. Male Fru^F flies, in which the fruitless gene is spliced into a female-specific (inactive) form, exhibited a strong reduction in male-specific patterns of aggressive behavior (Supplementary Fig. 4a–d online), as previously reported⁴.

Further, we studied the behavior of Cha-Gal4;UAS-tra (“Cha-Tra”) flies, in which all cholinergic neurons have been feminized due to mis-expression of the transformer gene²⁹. Cha-Tra males exhibited little or no courtship activity towards females, but a robust increase in courtship towards other males (Fig. 5e,f and Supplemental Fig. 2d online), at a frequency indistinguishable from wild-type Canton-S (CS) male-female pairs.

Cha-Tra male pairs also showed an increase in some aggressive behaviors compared to CS male pairs (Fig. 5g,h, Supplementary Fig. 2e and 3c,d online) and greater locomotor activity (Supplementary Fig. 2f online). Lunging activity was significantly higher than in controls, even after normalizing for distance traveled²³ (Fig. 5i). Surprisingly, wing threat was significantly lower in Cha-Tra flies (Fig. 5j); thus, not all aggressive actions were more frequent in Cha-Tra flies. Nevertheless, the total time spent in aggressive activity (Supplementary Fig. 2g online) and chasing (Supplementary Fig. 2h online) was significantly elevated in Cha-Tra males. Control experiments indicated that copulation could be detected in CS male-female pairs (Supplementary Fig. 2i online).

Our software allows us to compute 2D histograms showing the frequency of actions in each spatial location in order to detect phenotypes with an altered spatial distribution of behaviors (Supplementary Fig. 5 online). The 2D histograms revealed that Cha-Tra males performed a greater proportion of their tussling bouts on the central food patch, in comparison to controls (Supplementary Fig. 5a online). Moreover, the pattern of Cha-Tra chasing was more intense around the perimeter of the arena, while that in controls was more uniformly distributed (Supplementary Fig. 5b online).

We examined ethograms¹³, illustrating the frequency of each action, as well as the frequency with which one action was followed by the same or another action. Both CS and Cha-Tra males exhibited multiple transitions between courtship and aggressive activities (Fig. 5k–m). Cha-Tra males additionally show transitions between aggression and courtship and vice versa. They also show an elevated amount of chasing activity. In CS male pairs, chasing was followed most often by lunging, whereas in Cha-Tra males, chasing was followed with equal probability by either lunging, an aggressive action, or by wing extension, a courtship action (Figs. 5k and 5m, light blue circles). By contrast, in CS male-female pairs, chasing by males was followed most often by wing-extension (Fig. 5l). One interpretation is that in CS male-male pairs, chasing is primarily an aggressive action, while in male-female pairs it is primarily a courtship action. In Cha-Tra male-male pairs, chasing may be indicative of either aggression or courtship, suggesting that these flies may have a deficit in gender recognition or discrimination. This observation is consistent with a recent study³⁰ reporting that feminization of octopaminergic/tyraminergic neurons by misexpression of Tra caused male-directed wing extensions to be followed primarily by male-male courtship.

Finally, to examine the effects of social experience, an environmental influence, on male-female courtship, we used our software to analyze wing-extension and circling, two male-specific courtship actions. The male was in the presence of a decapitated female in the center of the arena. Previous studies have shown that raising post-eclosion Drosophila males in social isolation strongly increases aggressiveness, in comparison to flies raised in groups²⁴. We observed that both wing-extension and circling were significantly elevated in socially isolated (SS) vs. group-housed (GG) flies (Fig. 5n,o). Thus, social isolation increases both male-male aggressiveness and male-female courtship. This poses the question of whether these influences on social behavior are mediated by common or distinct mechanisms.

DISCUSSION

We describe a method for measuring automatically a broad array of social behaviors in Drosophila. Actions associated with courtship, aggression and locomotion are detected from overhead videos of fly pairs. We evaluated the system’s performance by comparing its measurements with carefully established ground truth. For all actions we found detection rates of 90–100% and comparably small false detection rates (see Table 1).

Our software detects wing postures, permitting measurements of wing threat and wing-extension. Wing threat, in particular, is an interesting and important aggressive display because it is independent of locomotor activity. Indeed, we show that a genetic manipulation (Cha-Tra) that strongly increases lunging, tussling and chasing actually decreases wing threat.

The time saved by our software is enormous. In our experience, it takes at least one hour to score manually one type of action in one 20 minute fly-pair movie. If one wished to characterize aggression and courtship in a line of flies (all eight actions in, say, twelve fly pairs), one would have to spend ≈100 hours of manual labor, as opposed to a few minutes to set up and run our system. As an example, the three ethograms presented in Figure 5 would have taken ≈270 person-hours to prepare. This capability affords the opportunity to compare multiple genotypes or wild-type genetic backgrounds, which would be virtually impossible to do manually.

The ability to monitor simultaneously both aggressive and courtship activities allows the computation of ethograms. This may prove to be valuable in determining whether aggression and courtship actions are part of a social behavior “continuum”, or whether these two activities represent discrete “states” with “state transitions”, controlled by different circuits. Genetic or circuit-based screens can be performed to search for phenotypes that affect not only the ability to perform a particular action, but also that affect the probabilities of transition between actions.

Our system (see Supplementary Methods and Software online) is completely automatic and self-calibrating; using it does not require special training, provided that the hardware setup is well reproduced. It has been designed for inexpensive implementation and easy replication. We found that fly behavior may be measured accurately in smaller arenas, allowing us to simultaneously monitor an array of arenas with each camera, thus permitting large-scale genetic screens with high throughput. Together, these features should open up aggression and courtship to powerful genetic screens. This in turn should help to illuminate the genes and neural circuits that control these important social behaviors, and may reveal general principles of the organizational logic or control mechanisms that are evolutionarily conserved.

MATERIALS AND METHODS

Fly stocks and rearing conditions

Flies were reared in plastic vials containing standard fly medium (yeast, corn syrup, agar), at 25°C, 60% humidity with a 12 hours light-dark cycle. Newly eclosed males were single housed or group housed (10 flies per vial) for 4–6 days before performing the behavioral assay. Virgin Canton-S (CS) females were collected shortly after eclosion and raised at 20 flies per vial for 4–6 days before the courtship assay. Cha-Gal4;UAS-tra flies were made by crossing male UAS-transformer to female Cha-Gal4 flies. Fru^F mutant was generated in B. Dickson lab. Tdc2Gal4:UAS-Kir2.1 flies were made by crossing male Tdc2-Gal4 flies to female UAS-Kir2.1 flies.

Aggression assay

We introduced two males of the same age into the double-arena setup (Fig. 1) by gentle aspiration without anesthesia and immediately video-captured them for 20 minutes. The temperature and humidity of the apparatus were set to ≈ 25°C and ≈ 40%, respectively.

Courtship assay

Two types of courtship assays were performed. The apparatus and environmental conditions were as used in the aggression assay. One male and a virgin female were introduced into the apparatus by gentle aspiration without anesthesia and immediately video-captured for 30 minutes (24 pairs) to cover the copulation period. For all other actions only the first 20 minutes were analyzed. In another assay the female was decapitated and placed in the center of the food patch and replaced every hour. After the male was introduced into the apparatus both flies were immediately video-captured for 10 minutes.

Training and ground-truth data

We collected a hand-annotated database of lunging, wing threat, chasing, wing extension, and circling to train our software and to measure its performance. Data used to train the detectors were produced by a human observer without further checks. Data that were used for testing the system’s performance were further processed to obtain a reliable ‘ground truth’. Human experts tend to miss relevant events (in our observations, between 30% and 40% of events are missed) for two reasons: (a) Different human observers will use slightly different criteria, even when they agree on the overall action definition. (b) A human observer’s attention level will change over time during movie annotations. Therefore, in order to obtain reliable ground truth for performance testing, we devised an improved two-step procedure involving two experts (see Supplementary Methods).

Statistical Analyses

Kruskal-Wallis-ANOVA was applied to detect overall differences among the unpaired groups. Significantly different groups were compared pairwise by the Mann-Whitney U-test. In all figures with statistics, one, two, and three asterisks indicate an α -level of 0.05, 0.01, and 0.001, respectively. For all multiple comparisons, Bonferroni correction was applied.

Graphical user interfaces

Our software (Supplementary Fig. 6a,b and Software online) was run from graphical user interfaces allowing the user to visualize tracking and statistical data.

Additional methods

In Supplementary Methods online we provide further information on the software, detection and tracking, as well as method hardware. An executable of the software is provided in the Supplementary Software. A free license of the source code is available on request to academic and non-profit investigators. Commercial entities should contact the Caltech Office of Technology Transfer for licensing arrangements.

Supplementary Material

Supplementary

Supplementary Figure 1 Sequences of automatically detected aggressive actions, chasing, and courtship actions.

Supplementary Figure 2 Frequency and time spent performing actions.

Supplementary Figure 3 Distribution of lunges over time.

Supplementary Figure 4 Frequency and time spent performing actions for Fru^F.

Supplementary Figure 5 Frequency of fly positions for aggressive actions, chasing and courtship actions.

Supplementary Figure 6 Software modules of the “Caltech Automated Drosophila Aggression-Courtship Behavioral Repertoire Analysis (CADABRA)”.

Supplementary Table 1 Features, extracted by the tracking module per frame and fly.

Supplementary Table 2 Lunging feature limits for pre-selection of possible lunges.

Supplementary Table 3 Features used final decision based on pre-selected frames (Supplementary Table 2).

Supplementary Table 4 Tussling feature limits.

Supplementary Table 5 Wing threat feature limits.

Supplementary Table 6 Wing extension feature limits.

Supplementary Table 7 Circling feature limits.

Supplementary Table 8 Copulation feature limits.

Supplementary Table 9 Chasing feature limits.

Supplementary Methods Details on our method and on used equipment.

Supplementary Video 1 High-resolution movie clip of a lunge (top view), complementing Figure 3a.

Supplementary Video 2 High-resolution movie clip of tussling (top view), complementing Figure 3b.

Supplementary Video 3 High-resolution movie clip of a wing threat (side view), complementing Figure 3c.

Supplementary Video 4 High-resolution movie clip of copulation (side view), complementing Figure 3d.

Supplementary Video 5 High-resolution movie clip of copulation with wing extension and circling (top view), complementing Figure 3d.

Supplementary Video 6 High-resolution movie clip of wing extension and circling (top view), complementing Figure 3e.

Supplementary Video 7 High-resolution movie clip of chasing (top view), complementing Figure 3f.

Supplementary Video 8 Movie clip of body and wing tracking during a lunge.

Supplementary Video 9 Movie clip of body and wing tracking during a lunge.

Supplementary Video 10 Movie clip of body and wing tracking during a wing threat.

Supplementary Video 11 Movie clip of body and wing tracking during wing extension and circling.

Supplementary Software Source code of “Caltech Automated Drosophila Aggression-Courtship Behavioral Repertoire Analysis (CADABRA)”