Experimental Assessment of Mouse Sociability Using an Automated Image Processing Approach

Abstract

Mouse is the preferred model organism for testing drugs designed to increase sociability. We present a method to quantify mouse sociability in which the test mouse is placed in a standardized apparatus and relevant behaviors are assessed in three different sessions (called session I, II, and III).

The apparatus has three compartments (see Figure 1), the left and right compartments contain an inverted cup which can house a mouse (called "stimulus mouse").

In session I, the test mouse is placed in the cage and its mobility is characterized by the number of transitions made between compartments. In session II, a stimulus mouse is placed under one of the inverted cups and the sociability of the test mouse is quantified by the amounts of time it spends near the cup containing the enclosed stimulus mouse vs. the empty inverted cup. In session III, the inverted cups are removed and both mice interact freely. The sociability of the test mouse in session III is quantified by the number of social approaches it makes toward the stimulus mouse and by the number of times it avoids a social approach by the stimulus mouse.

The automated evaluation of the movie detects the nose of the test mouse, which allows the determination of all described sociability measures in session I and II (in session III, approaches are identified automatically but classified manually). To find the nose, the image of an empty cage is digitally subtracted from each frame of the movie and the resulting image is binarized to identify the mouse pixels. The mouse tail is automatically removed and the two most distant points of the remaining mouse are determined; these are close to nose and base of tail. By analyzing the motion of the mouse and using continuity arguments, the nose is identified.

Figure 1. Assessment of Sociability During 3 sessions.Session I (top): Acclimation of test mouse to the cage. Session II (middle): Test mouse moving freely in the cage while the stimulus mouse is enclosed in an inverted cup. Session III (bottom): Both test mouse and stimulus mouse are allowed to move freely and interact with each other.

Download video file ^{(23.5MB, mp4)}

Introduction

Impaired sociability is one of the primary symptom domains in a range of neurodevelopmental disorders, including autism spectrum disorders (ASDs)^1,2. It severely limits the patients' ability to develop relationships and greatly affects patients, families, and caregivers^2,3.

In an effort to better understand the mechanism underlying impaired sociability and develop treatments, several mouse models have been studied^1,4–9. One such strain is the BALB/c mouse, an inbred strain that exhibits strikingly low sociability, resembling features seen in patients with ASD5^8–12. For example, the BALB/c mouse displays a lower frequency of social approaches and a usually negative response to social approaches by a stimulus mouse (e.g., social avoidance) than the Swiss Webster comparator strain¹⁰.

To test whether a proposed treatment is effective in increasing sociability, objective measures are needed. A widely accepted framework is to analyze the sociability in a standardized apparatus that has three compartments (see Figure 1).

The left and right compartments each contain an inverted wire cup which houses a socially salient stimulus mouse. The test mouse is released into the apparatus for three intervals of 10 min (sessions I-III); session I assesses the locomotor activity of the test mouse, while sessions II and III assess different aspects of the test mouse's sociability.

In session I, the test mouse is allowed to acclimate to the apparatus. In this session, the number of transitions that the test mouse makes between the different compartments is measured. Importantly, sociability measures obtained in the standard three-compartment apparatus are dependent upon locomotor activity, and reduced locomotor activity can confound interpretation of sociability data.

In session II, the test mouse is moving freely and a stimulus mouse is placed under one of the inverted cups (counterbalanced). The stimulus mice are 4-week old outbred male ICR mice, age- and sex-matched to the test mice. The stimulus mouse is enclosed in an inverted cup in the side designated as the social compartment, and an empty inverted cup is placed in the side designated as the nonsocial compartment. In session II, the sociability of the test mouse is quantified by comparing the time it spends near the “social” inverted cup (less than 2 cm away) vs. the time it spends near the “nonsocial” inverted cup, and by the time it spends in the social compartment vs. the time it spends in the nonsocial compartment. Swiss Webster comparator mice spend significantly more time in the social compartment and near the social inverted cup compared to BALB/c mice that show no significant preference for the social inverted cup or the social compartment¹².

In session III, test and stimulus mice are allowed to interact freely (inverted cups are removed). Measures of sociability, stereotypic behaviors and transitions between compartments are reliably obtained in the third 10 min session of free interaction between test and stimulus mice and analyzed. In this report, the sociability of the test mouse is quantified via two measures: 1) the number of social approaches it makes toward the stimulus mouse, where a social approach is defined as moving forward (nose first) towards the stimulus mouse so that the nose is less than 2 cm away from the stimulus mouse; 2) the number of avoidances of social approaches by the stimulus mouse. The test mouse is considered to be avoiding the approach if it turns or moves away from the stimulus mouse, or if it temporarily stops moving (“freezing behavior”) after the stimulus mouse has approached the test mouse. BALB/c mice show a decrease in number of social approaches made towards the stimulus mouse and an increase in the number of negative responses to social approaches by the stimulus mouse, compared to the Swiss Webster comparator strain¹⁰.

The evaluation of the behavioral movies has traditionally been done manually, i.e., by watching the movie carefully (possibly at reduced speed) and activating stopwatches for the time the mouse spends in a certain compartment or near one of the inverted cups¹⁰. This method requires watching of the movie repeatedly and is, therefore, very time-consuming. Also, the accuracy is limited by the rater’s reaction time for the stopwatch measurements and ability to perceive whether the mouse is within 2 cm of the inverted cup. In recent work, two compounds, (D-serine and D-cycloserine) have proven effective in restoring sociability in BALB/c mice^9,10,12. It is known that both components modulate the NMDA receptor, which is known to play a central role in social behavior^2,12. A large array of compounds is known to modulate NMDA receptor activity in a variety of well-defined ways, and the potential for a breakthrough in ASD treatment makes the expedited evaluation of these compounds a priority and a fast automatic method for evaluation highly desirable.

Described below is an automatic analysis procedure developed for the assessment of mouse sociability. It uses image processing methods on all the frames of the mouse movies to identify the position of the mouse or mice. In sessions I and II, it further detects nose position and uses it to compute sociability measures. In session III, it automatically detects all approaches, but manual classification (social approach, social avoidance, or other) is required.

Protocol

The protocol for the ease of explanation has been divided into 2 parts: 1) Experimental procedure and 2) Automated analysis. All animals were treated according to the Guide for the Care and Use of Laboratory Animals¹⁵, and all procedures were approved by the Institutional Animal Care and Use Committee.

1. Experimental Procedure

1. Set up the cage, which is a black non-reflective Plexiglas rectangular box (52 x 25 x 23 cm³) divided into three compartments: social, nonsocial, and a neutral middle compartment (Figure 1). Place inverted wire cups in each of the end compartments during sessions I and II (discussed below) and house the stimulus mouse during session II.

2. Set up the camera in such a manner that the cage fills the field of view. Since lighting can interfere with mouse behavior, maintain a luminance of no more than 3.5 lux inside the cage, coming from indirect light from an incandescent light bulb and measured on the cage floor in upward direction. Also use an infrared light source, which mice cannot sense but the camera can detect. Record all movies in 720 x 480 standard definition.

3. Record a single frame of an empty cage for later image processing (called the “reference frame”) and save as a grayscale image.

4. Session I: Introduce test mouse into the cage and wave hand over the cage, indicating the start of session (compare Automated Analysis). Record movie for 10 min. At the end of 10 min, put the mouse back into its original cage.

5. Session II: Put the stimulus mouse under one of the inverted cups (counterbalanced).

6. Use the “Social left” or “Social right” button to select whether the stimulus mouse is in the inverted cup on the left or in the inverted cup on the right. NOTE: The inverted cup with the stimulus mouse is called the “social cup”, whereas the other cup the “nonsocial cup”. The compartment in which the social cup is located is called the social compartment, the compartment with the nonsocial cup is the nonsocial compartment, and the middle compartment is the “neutral compartment”.

7. Introduce the test mouse into the cage. Wave hand over the cage to indicate the start of the session. Record movie for 10 min. Importantly, keep mice in the sociability apparatus before removing the inverted cups to begin Session III.

8. Session III: Remove the inverted cups, release both mice and allow the test and stimulus mice to interact freely with each other. Wave hand over the cage to indicate the start of the session. Record movie for 10 min. At the end of 10 min, put both mice back into their respective cages.

2. Automated Analysis

NOTE: The commands in section 2 are issued via a graphical user interface (see Figure 2) that controls our software. Here we explain the step-by-step processing that our software allows; the software is also capable of performing the analysis in batch mode without user input.

1. For every movie, execute the following processing steps:

   1. Load movie (click Load button).
   2. Select cage configuration, which tells the analysis program the location of cage boundaries, compartment boundaries, and cup locations. Select one of four saved configurations or manually adjust all coordinates. NOTE: The configuration usually does not change from one experiment to the next and can therefore be reused.
   3. For each frame of the movie, perform the following processing steps (2.1.3.1-2.1.3.6).

      1. Convert the frame to grayscale by clicking the “Grayscale” button.
      2. Digitally subtract the grayscale reference frame (see step 1.3) by clicking the “Subtract” button. NOTE: The goal of this step is to make the pixel values of the frame very close to zero everywhere except for the mouse position.
      3. Binarize the frame by setting all pixels below a certain threshold level to black, all pixels above that level to white (click the “Binarize” button). Note: The idea is that since the mouse pixels generally have higher values than the remaining pixels, the mouse pixels will become white and all other pixels black. The optimal threshold level depends on the lighting condition and can be determined by assessing the mouse shape throughout a movie. If the threshold is chosen too low, the mouse appears too large, and pixels that do not belong to the mouse can be colored white. If the threshold is too high, the mouse becomes small and can disintegrate into disconnected pieces. In the recorded movies, the optimal threshold was always between 20 and 35 (on a scale from 0 to 255).
      4. Erode¹³ the binarized picture e times, then dilate¹³ 2e times, then erode again e times, by pushing the button “Erod/Dil”. Note: The purpose to this procedure is to remove the tail. For the resolution (720 x 480) that was used, e = 3 worked in all circumstances. Note: The software checks automatically for the presence of a hand in the picture (during the recording of movies, hands are waved through the picture to indicate the beginning of a session). If a hand is present, there will be unusually many white pixels in the binarized frame because the hand is much larger than a mouse. The condition for the presence of a hand is that there are more than 10,000 white pixels in the binarized frame. The software determines the beginnings of sessions I, II, and III, by detecting the range of frames in which hand movements occur.
2. In sessions I and II, continue as follows:

   1. Detect the largest connected component of the set of white pixels (Click “LComp”). Note: The purpose of this step is to eliminate white pixels that do not belong to the mouse, e.g., caused by the motion of the stimulus mouse in session II. The largest connected component is in almost all frames the mouse.
   2. Determine the two most distant white pixels (called the “ends” of the mouse) by clicking “Find Ends”. Note: The software establishes the continuity of the two ends, i.e., for each of the ends in the first frame to which of the ends in the subsequent frames they belong. The general idea is that since frames are recorded at high rate, the ends cannot travel far from one frame to the next, so that for the correct continuation of the ends from one frame to the next is that which minimizes the sum of the distances that the ends move. See the Discussion for more detail.
   3. Determine which end of the mouse is the nose.

      1. Determine the direction in which a mouse moves by observing how the center of gravity (COG) changes from frame to frame (the COG’s motion is the motion of the sum of all mouse pixels). Note: The COG of a set of pixels p₁, ... , p_n  with coordinates x₁, .. , x_n   is defined by: Mice usually move forward (i.e., towards their nose). In a section of several hundred frames, there is always very high confidence in the head/tail detection. Note: The software automatically determines towards which of the detected ends the mouse is moving; this end is identified as the nose and marked with a red circle in the movie. When the detected nose is marked with a red circle in every frame, the user can check the quality of the detection.
3. In session III, continue as follows:

   1. Detect two largest components of white pixels.

      1. If both are of acceptable size for a mouse and their centers of gravity (COGs) are compatible with the mouse COGs of the previous frame, consider these two components the two mice.
      2. Consider mouse sizes acceptable if they have not changed by more than 20% since the previous frame, and consider COGs of two consecutive frames as compatible if their difference is not larger than the distance a mouse can travel in the time between frames (this distance follows from the maximum mouse speed, which can be determined as part of the preparations of an experiment series by analyzing the motion of several individual mice (as in sessions I and II). Note: If the two largest components are not acceptable sizes for a mouse, or the COGs are too different from those in the previous frame, the software automatically increases the binarization threshold until the two largest components are of acceptable size and their COGs are compatible with those of the previous frame.
   2. Click the test mouse in the first frame. Use the analysis in 2.3.1 to identify both test and stimulus mouse in every frame of session III.
   3. Identify the frames in session III in which the one mouse socially approaches the other. Note: The software implements this step by automatically checking that the distance between the mice is less than the typical interaction distance of mice (we use 2 cm).
   4. When the software plays a short segment of the movie with the detected approach to the user, click a button to indicate whether this approach qualifies as a social approach by the test mouse, social avoidance by the test mouse, or neither. Note: Step 2.3.4 can be automatized if it is known which mouse is the test mouse. Since this information gets lost when the mice get too close to each other (or climb on top of each other), and since we have not yet found a way to label the mice in a way that can be detected reliably by our software AND does not affect their behavior, we have reverted to manual classification for now.
4. To evaluate processed movies, load movie using Load button.

   1. Click on Start Session 1 to display the first frame of session 1 (or Start Session 2 to display the first frame of session 2).
   2. Click on Play/Stop to play and stop movies. Note: The processed movies will display mouse with head and tail marked with differently colored circles. All sociability measures are displayed on the screen and updated as the movie plays.
   3. Export sociability data in Excel format by clicking “Export”.
   4. Click on “Compile Data” to compile all the data of all movies in a folder into a single excel file.

Representative Results

Figure 3 shows a color frame of a behavioral movie with an empty cage. The mouse has not been introduced into the cage yet. The position of the cups, (social and nonsocial) and the compartment boundaries are superimposed. The color frame of empty cage is converted to 8 bit grayscale as shown in Figure 4 and used as a reference frame. The reference frame is subtracted from every other frame of the captured movie.

Figure 5 shows one of the frames from the movie during session I, again with the compartment boundaries and cup positions superimposed. The reference frame is subtracted from sample frame and the result, called the difference frame, is shown in Figure 6.

Figure 7 is the binarized version of the difference frame. (see section 2.1.3.3). The figure also shows the position of the nose (red circle) and the base of the tail (gray circle), as determined in section 2.2.4).

Once the nose position is established for every frame of the movie, the trajectory of the mouse can be analyzed in any desired way. Figure 8 shows the positions of the mouse nose (white circles) for a part of the movie that covers approximately 1,000 frames (~ 30 sec).

Figure 9 shows the result of a sociability analysis for session I. The number of transitions between cage compartments is used as a measure of locomotor activity (in the example shown, 59 transitions in 10 min indicate normal locomotor activity). The time spent in different parts of the cage serve as a baseline to judge preference for the social compartment/cup in session II.

Figure 10 shows the result of the sociability analysis for session II. Of particular interest are the times spent near the social vs. nonsocial cup (in the shown example 230.8 vs 72.3 sec) and the social vs. the neutral and nonsocial compartments (314.7 vs. 109.4 vs. 185.5 sec). The clear bias towards the social cup and compartment are typical for control mice, and comparison with the data from session I (Figure 9) clearly demonstrates that the bias really is a result of the presence of a stimulus mouse.

Figure 11 shows the result of the sociability analysis for session III. The high number of social approaches (35 in 10 min) as well as the overwhelmingly positive responses to social approaches by the stimulus mouse (9/10) are typical of mice with normal sociability.

Figure 2. Graphical User Interface. Please click here to view a larger version of this figure.

Figure 3. Empty Cage. This is used as a reference frame after conversion to gray scale. The red circle shows the position of the social cup and nonsocial cup in the opposite side. The vertical white lines divide the cage into different compartments and the rectangle shows the inner boundary of the cage.

Figure 4. Grayscale Version of Figure 3.

Figure 5. Original Frame. A movie frame in grayscale format. The social cylinder is marked by a white circle, and an additional gray circle with a radius 2 cm larger than that of the social cylinder is drawn to demarcate the region in which the test mouse is considered “close” to the social cylinder. In the opposite corner, the nonsocial cylinder is marked by a gray circle and again, an additional circle with a 2 cm larger radius is drawn to demarcate the area that is considered close to the nonsocial cylinder. The vertical white lines divide the cage into different compartments and the rectangle shows the inner boundary of the cage.

Figure 6. Difference Image. Image obtained by subtracting the reference frame from the sample frame.

Figure 7. Binarized Image. The pixels of the difference image that exceed a particular threshold (30 in this example) are considered the mouse pixels and are shown white. The red circle denotes the nose and white circle is the tail.

Figure 8. Mouse Trajectory. The small white circles show the location of the nose from the start of session I until this frame was captured.

Figure 9. Results of Sociability Analysis for Session I. The numbers above the respective cage compartments indicate the times the mouse spent in them (in this case 254.3 sec in the left compartment, 155.2 sec in the neutral compartment, and 237.0 sec in the right compartment). The numbers inside the circles indicate the times the mouse spent close to the respective cup (“close” means less than 2 cm away, or inside the outer of the two circles): 140.0 sec close to the left cup, and 105.6 sec close to the right cup. The number in the bottom center indicates how often the mouse transitioned between the different compartments of the cage (59 times in the example).

Figure 10. Results of Sociability Analysis for Session II. The numbers above the compartments and inside the circles indicate times spent in different part of the cage as in Figure 9 / session I. Note that in session II, there is a stimulus mouse in one of the inverted cups (the left one in this case), so this cup becomes the “social” cup. In addition to the measures of session I, the time the test mouse takes to get close (within 2 cm) to the social cylinder for the first time is given in the bottom right of the center. Please click here to view a larger version of this figure.

Figure 11. Results of Sociability Analysis for Session III. The number after “Approach” states the number of social approaches made by the test mouse, the numbers after “Pos. Resp.” and “Neg. Resp.” state the number of positive and negative responses by the test mouse to social approaches by the stimulus mouse.

Discussion

By combining videotaping mice and automatic movie analysis, we have created an affordable, reliable high-throughput screening technique.

The accuracy of the automated analysis was compared with the results of manual analysis for more than 100 movies. In the overwhelming majority of frames (> 99%) of frames, the nose of the mouse is identified with good precision. Most of the (few) misdetections have no effect on the sociability measures, so that around 80% of the movies do not require any corrections.

In those movies that do contain misdetections, by far the most common problem is that head and tail are confused, so we have included in the software a simple way to switch head and tail for a movie segment. Even if movies are viewed to check for and correct all misdetections, this check can be done at about 3 times real time speed, so that the evaluation time remains very short. The sections of the movie in which the mouse is not close to one of the cylinders can be viewed at even higher speed.

The agreement between all automatically and manually determined sociability measures is excellent (assuming that possible misdetections have been corrected). In fact, the quality of the automatically determined sociability measures is superior to manually determined ones, because objective criteria are consistently used to determine distances and times, while manual evaluation suffers from limits of human perception and reaction time. These limits become more severe with higher viewing speeds, which are desirable to limit the evaluation time.

Note regarding step 2.2.2.: This routine is implemented by reducing the set of white pixels to its boundary using standard image processing tools¹⁴ and computing the distance between any two boundary pixels. Finding the most distant points is useful because the nose and the base of the tail tend to be very close to these two ends.

For completeness, the implementation of the continuity of the ends (step 2.3.4) is described below. In a given frame with ends c and d and a preceding frame with ends a and b, whether c belongs to a and d to b or c belongs to b and d to a (see Figure 12).

Figure 12. Continuity Determination. Mouse in light gray color with nose and tail marked as a and b respectively is the initial position. Mouse in white color with nose tail marked c and d respectively is the position of mouse in the next frame.

The general idea is that for the correct continuation, the sum of the distances traveled by the ends is smaller than for the incorrect continuation, because the mouse travels less than its own length from one frame to the next. In the example (Figure 2), introduce δ₁ = |c - a|, δ₂ = |d - b|, Δ₁ = |c - b|, and Δ₂ = |d - a|, and say that because δ₁ + δ₂ < Δ₁ + Δ₂, the correct continuation is a → c and b → d. A good measure for the confidence of such a determination is the ratio r = max(δ₁ + δ₂, Δ₁ + Δ₂)/min(δ₁ + δ₂, Δ₁ + Δ₂), which describes how much more the ends would travel in one continuation versus the other. By definition, r is larger or equal to 1. For elongated mouse shapes, r is typically larger than 10, only if the mouse shape becomes more circular (e.g., because the mouse is standing on its hind legs), r can get close to one and the continuation is unreliable. For reliable continuation, a threshold of r > 2.5 has been used. With this threshold, the sections of frames over which the continuity of the ends is established, ranges typically from hundreds to thousands of frames. For each section, determine towards which end the mouse is moving within a section. Once continuity in a section is established, the “center of gravity” (COG, the average coordinates of all mouse pixels) for each frame in the section is determined. The difference in COG from one frame to the next defines the velocity of the mouse between the two frames. This velocity is projected on the vector that connects the two ends to see whether and how much the mouse is moving towards the first or towards the second end. The total motion (which is either towards the first or the second end) is the summation of the motions of all frames.

The present protocol can be modified in several ways to address different experimental needs. If different aspects of the mouse's movement (e.g., the average velocity) are of interest, these can be extracted from the computed mouse trajectory. If the lighting conditions are very different from those described here, the threshold in the binarization step that yields the mouse pixels can be adjusted. Even changes like a different cage shape can be implemented by modifying our current cage configuration routine.

The described software occasionally confuses head and tail (in less than 20% of the movies, there is a need for correction because the head/tail flip would affect our evaluation). We wrote a convenient tool that allows the user to identify the first and the last frame of the interval in which head and tail need to be flipped and then execute this correction with one click. Very few movies (a few percent) had problems other than head/tail flip, for these we made a more general correction tool that allows the user to specify in which part of the cage the test mouse nose is during any interval. Note that the need for correction occurs so rarely that the effect on evaluation time is small.

Planned extensions of the protocol are to also detect more socially relevant behaviors, such as pursuing and anogenital sniffing (which can be detected by monitoring the distance from the nose of the test mouse to the body and the anus of the stimulus mouse, respectively). Stereotypical behaviors may also be detectable, for example rearing, which can be detected from the change in the mouse shape. Another interesting extension is the use of mice of different color, which should be straightforward as long as the color of test and stimulus mouse have good contrast with the color of cage walls and cage floor.

In summary, the described protocol has an experimental and an image processing part. In the experiment part, the most important steps are to check the lighting and the positioning of the cage and then to record the mouse behavior during three sessions (I: alone in cage, II: with confined stimulus mouse, III: freely interacting with stimulus mouse). In the analysis part, the critical steps are to load a movie and cage configuration, to perform image processing on each movie frame to determine nose and tail positions and then to compute the statistics necessary to quantify sociability from the nose and tail trajectories.

A limitation of the described method is that it has only been tested for mice with white fur. It stands to reason that the automated analysis will work equally well with other fur colors as long as the apparatus walls and floor provide strong contrast with the fur color, but this has not yet been established.

Another limitation is the lack of automation of the evaluation of session III. The major difficulty for us, as for other groups, is to keep track of which mouse is the test mouse and which the stimulus mouse during the frequent close encounters of the mice. We have found that we can solve this problem with markers such as colored dots between the mice’s ears, but all the marking methods that we have tried so far have irritated at least part of the mice and altered their behavior substantially. Therefore, our current analysis of session III automatically detects all approaches but requires manual classification.

Disclosures

The authors have nothing to disclose.