Rapid antibiotic susceptibility testing based on bacterial motion patterns with long short-term memory neural networks

Abstract

Antibiotic resistance is an increasing public health threat. To combat it, a fast method to determine the antibiotic susceptibility of infecting pathogens is required. Here we present an optical imaging-based method to track the motion of single bacterial cells and generate a model to classify active and inactive cells based on the motion patterns of the individual cells. The model includes an image-processing algorithm to segment individual bacterial cells and track the motion of the cells over time, and a deep learning algorithm (Long Short-Term Memory network) to learn and determine if a bacterial cell is active or inactive. By applying the model to human urine specimens spiked with an Escherichia coli lab strain, we show that the method can accurately perform antibiotic susceptibility testing as fast as 30 minutes for five commonly used antibiotics.

I. INTRODUCTION

Antibiotic-resistant bacteria infections affect over two million people, causing more than 35,000 deaths every year in the US [1]. Overuse and misuse of antibiotics in clinics and in agriculture have contributed to the antibiotic resistance epidemic [1]–[4]. To address this crisis, there is a need to develop an antibiotic susceptibility testing (AST) technology that can rapidly identify antibiotic-resistant bacterial strains and determine the susceptibility of the infecting pathogens to antibiotics. However, the current AST methods, mainly represented by disk diffusion and broth dilution [5], [6], require bacterial isolation and culture, which from sample collection to AST results, takes several days or longer [6]–[8], depending on the bacterial growth rate. For this reason, healthcare practitioners often rely on experience or use a broad-spectrum approach to treat patients.

Many groups have attempted to shorten AST by measuring bacterial metabolic activities with various approaches, including traditional optical microscopy to track the growth of bacterial cells with image processing algorithms [9]–[11], forward light scattering [12], vibrational amplitude changes of magnetic beads [13], phase-shift spectroscopy [14], microfluidic mass sensor arrays [15], or biochemical markers, such as ATP [16] and luciferase [17]. While promising, these approaches still rely on cell culture and detecting large changes in bacterial cell size and/or numbers due to bacterial growth and division, thereby requiring additional time (typically over 3 hours) to yield quantifiable results [18], [19].

Recently our group has developed a deep learning video microscopy approach to achieve rapid AST [20]. This approach condenses videos into images and uses Convolutional Neural Networks (CNN) to classify each bacterium as active or inactive [21]. CNN is a deep neural network model, which can automatically create filters to identify features in images. The advantage of the CNN-based deep learning algorithm is that it does not require manually selected and extracted features, such as size and shape of bacterial cells. However, the “black-box” nature of this algorithm makes it challenging to determine which specific features are detected for rapid AST. The algorithm also requires a large data set to account for different bacterial strains and antibiotics [22]. We present here a method to differentiate active and inactive cells captured by optical images using combined phenotypic features: bacterial growth and motion patterns. Combining information from both growth and motion using state of the art neural networks provides more robust and faster results. There is no need to rely on genetic testing, thereby avoiding detection limitations caused by incomplete resistance genes. Our method is also capable of performing single cell analysis, like other imaging approaches with the advantages of being label-free and immobilization free and use a simple phase contrast microscope. Experiments performed in free solution avoid surface modification, so there is no need to consider the efficiency of bacterial capture. Labeling could also affect bacterial activity and thereby affecting the accuracy of test results. The detection limit is between 10^6 – 10^8 CFU/mL and the throughput is limited to one sample at a time. In contrast to the CNN approach [20], the new method focuses on single bacterial cell motion patterns and changes of the patterns in the presence of different antibiotics.

To extract the motion patterns, we developed a robust segmentation and tracking algorithm to determine the motion pattern of each bacterial cell, and a Long Short-Term Memory (LSTM) neural network to analyze the pattern over time. This approach makes full use of time information and the sequence of events, potentially generating more robust and intuitive models than those created with static images. Moreover, since this model only uses positions of the bacterial cells, it requires much less data than full image-based CNN, thus enabling easier training and application. The LSTM approach also allows prediction of bacterial motion, which can be useful for robust bacterial tracking. In addition to applying this new method to our previous CNN-analyzed AST data [20], we extended the studies to include additional antibiotics.

We validate the method using Escherichia coli (ATCC 43888) and five antibiotics, including polymyxin B, ampicillin, gentamicin, ciprofloxacin and streptomycin. These antibiotics were selected as representative compounds with different mechanistic activities and are inclusive of both bacteriostatic and bactericidal antibiotics. We demonstrate the method’s ability to differentiate active and inactive cells and to obtain accurate AST results as fast as 30 minutes.

II. RESULTS AND DISCUSSIONS

A. Model of image-processing and deep learning algorithms

Phase contrast microscopy reveals individual E. coli cells inside a microfluidic channel in Lysogeny Broth (LB), a rich bacterial growth medium (Fig. 1a). Instead of attaching the cells onto a solid support [23], the bacterial cells freely move within the microfluidic channel, allowing tracking of single bacterial cell motion as an important phenotypic feature. Because the bacterial cells constantly move in and out of the field of view of the optical microscope, this method also allows analysis of a larger number of different bacterial cells. Tracking and analyzing larger numbers of bacterial cells reduce statistical errors associated with a finite number of bacterial cells. In the presence and absence of different antibiotics and multiple antibiotic concentrations, bacteria are imaged for 30 seconds. Subsequently, the image sequence is modeled to generate cell trajectories, which are processed into a deep learning algorithm to classify the cells as either active or inactive (Fig. 1b). From the classified trajectories, the number of active cells is determined and plotted as a function of concentration for each antibiotic, from which the minimum inhibitory concentration (MIC) - the lowest antibiotic concentration at which the bacterial strain does not grow - is obtained (Fig. 1c). We applied the model to five antibiotics: polymyxin B [24]–[26], ampicillin [27], [28], gentamicin [29]–[33], ciprofloxacin [34] and streptomycin [35]. Using two-fold increasing concentrations from 0.5 – 16 μg/mL, we visually observed killing activity (bactericidal effects) for polymyxin B, ampicillin, and gentamicin. Inhibitory activity (bacteriostatic effects) was observed for streptomycin and ciprofloxacin with low concentrations (< 0.5 μg/mL).

Fig. 1.

a) Single bacterial cells are imaged in urine in a microfluidic channel using phase contrast microscopy. The cells are revealed either as bright or dark spots, depending on their relative positions in the focal plane (each cell is marked with a red circle). b) Image sequence over time (2 seconds) shown in panel (a) is fed into an algorithm comprising two parts: image processing and deep learning. The former includes segmentation and tracking algorithms to obtain individual bacterial cell trajectories (motion pattern). The latter processes the trajectories with a deep learning algorithm to determine if a cell is active and inactive and records the number of active and inactive cells. c) AST results are generated from the number of active cells obtained using the deep learning algorithm, thereby extrapolating the minimum inhibitory concentration (MIC) for each tested antibiotic.

B. Image segmentation and cell motion pattern determination

For each image frame, bacterial cells are first segmented with a threshold to generate a binary image, where connected objects represent bacterial cells (Fig. 2). The threshold and the algorithm are evaluated by visual inspection. Then, the centroid of each cell is identified, representing the position of the cell at a given time (image frame). Position changes over time are tracked from adjacent image frames, which are connected and recorded as the cell trajectory. More details about the algorithm can be found in the Algorithms section.

Fig. 2.

Image processing algorithm segments bacterial cells from the optical images and extracts the trajectory of each bacterial cell. a) Phase contrast optical images over time, where bright or dark elongated spots are single E. coli cells. b) After segmentation, the optical images are transformed into binary (black background and white cells) images. c) Centroids of the bacterial cells in the binary are extracted (marked by red dots). d) Tracking algorithm connects the centroid of each bacterial cell over time to generate cell trajectories (red lines).

C. Overview of LSTM algorithm and experimental setup

To utilize the trajectories of bacterial cells tracked with the imaging processing algorithm, a machine learning algorithm that can model time-dependent features is essential. LSTM, developed in this study, is such an algorithm that models long time sequences and retains critical information and features, enabling robust training of the model [36]. Because different cells are located at different positions within the image, all cell trajectories are set to start at positions (0,0) by subtracting the first point from the rest in the time sequence and used as the input for the LSTM algorithm. Since the entire trajectory of each bacterial cell is processed into LSTM, there is no need to manually extract motion features, such as speed and direction of the motion, and their changes over time [37], [38]. Deep learning algorithms typically generate features that have no physical meaning to humans such as the ones mentioned before like speed and direction, however their superior classification accuracy has made them popular over the last years.

We analyzed several thousand cell trajectories in the presence of each antibiotic with the goal of dividing them into two categories: active cells or inactive cells. Active cell trajectories were obtained from the images of control samples, (healthy human urine spiked with E. coli without antibiotics) captured throughout each experiment (Fig. 3). Inactive cell trajectories were captured from the images of samples treated with each antibiotic at its respective highest concentration (4× MIC obtained from the modified broth macrodilution experimental AST), recorded 120 min after the beginning of the experiment. Since there is no standard method for determining single cell activity level, we must assume all the cells in the control videos are active and all the cells in the high drug concentration videos are inactive. Even though this assumption may not be true, the machine learning algorithm should to capture the patterns if the majority of the cells follow the assumption. The upper limit on 4× MIC was determined experimentally as we observed it was sufficiently high to inhibit most cells. The proposed algorithm is ultimately evaluated by comparing the obtained MIC to the one obtained using the clinical standard method.

Fig. 3.

E. coli growth curves and determination of optical LSTM MIC values for a) polymyxin B, b) ampicillin, c) gentamicin, d) streptomycin, and e) ciprofloxacin. The dashed line for each antibiotic marks the threshold (1.5×) used to determine the optical LSTM MIC values. E. coli cells incubated in healthy human urine without antibiotics were included as controls (blue line). Replicates for each time point were obtained by recording non-overlapping, 2-sec windows throughout a set of three 30 sec videos (see Materials and Methods for more details).

To establish an experimental MIC for each antibiotic, we used a modified broth macrodilution AST approach (see Materials and Methods). Table I compares the results obtained using the optical LSTM AST and with the modified broth macrodilution, as well as the expected values from the Clinical and Laboratory Standards Institute (CLSI) reference [39], [40]. With the exception of streptomycin, the optical LSTM MIC results were in good agreement (+/− 1 dilution) with the modified broth macrodilution method (Table I). The small disagreements could be due to expected sample variability and differences in the setup conditions, for example the usage of room temperature instead of 37° C, to ensure better bacterial motion (see Supplemental material).

Table I.

Comparison of the optical LSTM experimental AST with modified broth microdilution AST and (CLSI) AST interpretive standards.

Antibiotic	MIC CLSI reference (clinical standards)	MIC Modified broth macrodilution (experimental AST)	MIC Optical LSTM (experimental AST)
polymyxin B	≤ 2 μg/mL	2 μg/mL	1 μg/mL
ampicillin	≤ 8 μg/mL	2 μg/mL	4 μg/mL
gentamicin	≤ 4 μg/mL	4 μg/mL	4 μg/mL
streptomycin	None	4 μg/mL	16 μg/mL
ciprofloxacin	≤ 1 μg/mL	0.03125 μg/mL	0.0625 μg/mL

D. Optical LSTM AST

To determine MIC with the trained LSTM algorithm, the number of active cells, measured every 30 min, was recorded for each antibiotic at different concentrations (Fig. 3). For each growth curve, the number of active cells was normalized using the initial number of cells determined at 0 min from the first image frame (Fig. 3). This plot both motion and general growth information, as the number of active cells is correlated with both features. The slope of each curve decreases with increasing drug concentrations, showing that higher concentrations have a greater effect on the cells. We set a threshold at 1.5× the initial number of the active cells for each antibiotic (horizontal dashed lines in Fig. 3). If the cells are inhibited, the number of active cells remains at or below the threshold at each time point. Conversely, if the cells are not inhibited, then the number of active cells increases above the threshold over time. The antibiotic concentration that demonstrated consistent growth inhibition throughout 150 min (Gentamicin only needed 120 min) was defined as the optical LSTM MIC. For all time points when the growth control exceeded the 1.5× threshold, the LSTM MIC values for all antibiotics recorded at 150 min (120 for gentamicin) were similar to earlier time points (Fig. 3a–c, e).

We randomly selected 80% of the cell trajectories to train the LSTM model for each antibiotic and used the remaining 20% to test the accuracy of the trained LSTM model (Table II). We chose not to separate the test phase into training and validation as we did not perform hyper-parameter selection or best epoch selection, cases where a validation set is required. The LSTM model produced excellent accuracy for polymyxin B, ampicillin, and gentamicin, but was less accurate for streptomycin and ciprofloxacin (Table II). This observation indicates that motion pattern alone is inadequate to determine if it is susceptible to some antibiotics. However, the LSTM algorithm is capable of determining the total number of active cells in the sample and its change over time. If the bacterial strain is inhibited by an antibiotic at a certain concentration, the cell number will not increase over time, so the number of active cells serves as an additional phenotypic feature for AST.

Table II.

LSTM training results for different antibiotics.

	Active			Inactive
Antibiotic (concentration)	Number of cells		Accuracy (%)	Number of cells		Accuracy (%)
	Training	Testing		Training	Testing
polymyxin B (8 μg/mL)	2886	722	84.49	2915	729	96.30
ampicillin (8 μg/mL)	2886	722	85.87	2454	614	91.86
gentamicin (16 μg/mL)	2886	722	83.66	2480	621	94.69
streptomycin (16 μg/mL)	2886	722	42.94	2962	741	55.87
ciprofloxacin (0.125 μg/mL)	2886	722	46.81	2956	739	80.11

III. EXPERIMENTAL SETUP

A PDMS microfluidic chip with a channel volume less than 100 nL was fabricated by soft lithography and used for the AST experiments described here [42]–[44]. The chip included two layers of PDMS, the upper layer was a pneumatic control layer (RTV 615, the ratio of A/B is 5: 1) and the bottom layer was a fluidic layer (RTV 615, the ratio of A/B is 10: 1). The two layers were aligned by thermopolymerization reaction and bonded on a glass slide with oxygen plasma. The mold of the microfluidic chip was made from photoresist (the positive photoresist AZ-50XT was used for fluidic layer mold and the negative photoresist SU8–2025 was used for control layer mold and the inlet and outlet channels mold). Fig. 4 shows the detailed structure of the microfluidic chip. The chip includes six channels, which are controlled by independent valves and main valves. The function of the valve is to allow each channel to perform experiments with different concentrations or different antibiotics in parallel or separately. For one reaction channel (0.5 cm in length, 25 μm in height, 200 μm in width), the sample solution was injected into the microchannel by a syringe pump. After the injection, the pneumatic valve is closed to stabilize the solution in the channel, which eliminates the effects of the fluid and ensures that all movements are caused by bacterial activity and not by liquid flow. The outlets were connected to a waste bottle by Teflon tubing.

Fig. 4.

Schematic illustrations of designed microfluidic chip. This microfluidic chip has six parallel detection channels, which allows AST detection with different concentration of antibiotics simultaneously.

The microfluidic chip was placed on an inverted microscope (Olympus IX-81) with a 40X phase contrast objective lens and a CCD camera (Pike-032B, Allied Vision Technologies, Newburyport, MA) to record phase contrast images. The recording process was controlled using software AVT Smart View (Version 1.14.1). All imaging sequences were collected at 100 fps at a pixel resolution of 640 × 480. An appropriate exposure time was chosen to maximize image intensity and avoid over exposure. Images were recorded in raw format and batch-converted to 16-bit tiff format using a MATLAB program. The images were subsequently processed using custom-written MATLAB and Python programs. Since bacterial cells are typically resolved with positive contrast (bright) and negative contrast (dark) in phase contrast imaging, we developed algorithms to process positive and negative contrast bacterial images and to track cells that moved towards and away from the focal plane.

IV. MATERIALS AND METHODS

A. Materials

Photoresist SU-8 2025 and photoresist AZ 50 XT were purchased from Microchem and AZ Electronic Materials USA Corp., respectively. Polydimethylsiloxane (PDMS) (RTV-615-044) was purchased from Momentive Specialty Chemicals (NY). The wafer was purchased from Avago Technologies. E. coli ATCC 43888 (Biosafety Level 1 organism that lacks the genes for Shiga-like I and II toxins) was purchased from Fisher Scientific. Unfiltered, pooled human urine specimens from multiple healthy individuals (Lot: BRH1041997) were purchased from BioreclamationIVT Co and were stored at −80°C immediately after receiving and thawed prior to the experiment. Antibiotics, including polymyxin B, ampicillin, gentamicin, ciprofloxacin and streptomycin and all other reagents were purchased from Sigma-Aldrich.

B. Antibiotics preparation

Polymyxin B, ampicillin, gentamicin, and streptomycin were individually prepared as 200 μg/mL stock solutions in ultrapure water. Ciprofloxacin was first dissolved in 0.1 mol/L of HCl (1:60, m/v) prior to generating 200 μg/mL stock solutions in ultrapure water. Antibiotics were stored at −80°C in the dark. Before AST, the antibiotic stock solutions were thawed at room temperature and diluted in ultrapure water to a series of different concentrations.

C. Bacteria cultivation and preparation

E. coli ATCC 43888 was stored at −80°C in 5% glycerol. After aerated and agitated growth at 35°C for approximately 16 h, saturated E. coli (20 μL) was diluted into Lysogeny Broth (LB) broth (5 mL) and cultured for ~ 2 h at 35°C to attain a logarithmic phase of growth. Maeda et al [41] demonstrated that E. coli K-12 exhibited greater swimming motility (straighter paths) at 20°C and greater tumbling motility (circular paths) at 39°C. In order to maximize the straighter path trajectories for the LSTM algorithm and training, the bacteria were maintained at 25°C for 3 h before the beginning of the optical-based AST experiments. Bacterial cells were then collected by centrifugation at 450g for 5 min and resuspended in unfiltered healthy human urine (BioreclamationIVT Co.) to a concentration of 2 × 107 cells/mL. Prior to LSTM algorithm testing and optical AST assessment, the bacteria-spiked urine samples were filtered using a 5 μm syringe filter (EMD Millipore) to remove large particles.

D. LSTM optical AST

After dilution to appropriate cell concentrations in urine, antibiotics were added to the E. coli suspensions for optical-based AST. Briefly, the E. coli-spiked urine samples (200 μL) were mixed with an equal volume (200 μL) of antibiotic solution (in ultrapure water) prior to beginning of each optical-based AST experiment. The antibiotic concentrations ranged from 4 – 0.25× MIC as determined via modified broth dilution testing (described below). The upper limit on 4× MIC was determined experimentally as we observed it was sufficiently high to inhibit most cells. The mixtures were simultaneously injected into different microfluidic channels, and the valves were closed after the microfluidic channels were fully filled. Continuing incubation at 25°C, bacterial status was recorded every 30 min at 100 frame per second (fps) for 35 seconds. Using this method, we analyzed the antibiotics previously described by Yu et al [20] and included gentamicin, a more clinically-relevant aminoglycoside antibiotic.

E. Modified broth macrodilution AST

To compare the optical-based AST to more traditional AST, a modified broth microdilution approach was adopted. After centrifugation at 450g for 5 min (described above), E. coli was resuspended in cation-adjusted Mueller Hinton Broth (CAMHB) (Sigma-Aldrich) and adjusted to 1 × 106 cells/mL after UV-Vis spectrophotometer (NanoDrop 2000, Thermo Fisher) readings at OD600. Thereafter, the E. coli suspension (1 mL) was mixed with an equal volume (1 mL) of antibiotic solution (in ultrapure water) at a specific concentration, resulting in half-strength CAMHB and initial bacterial concentrations of approximately 5 × 10⁵ cells/mL. As a positive control, the E. coli suspension (1 mL) was mixed with an equal volume (1 mL) of ultrapure water without antibiotics. After incubation at 37°C for 16h, the tube with the lowest concentration of each antibiotic that lacked visible growth was recorded as the MIC value. All modified broth microdilution experiments were performed in triplicate.

F. Growth curves and MIC determination

MIC curves were generated in the following manner: 30-second videos were analyzed for each antibiotic, concentration, and time point. Each video was subdivided into fifteen 2-second videos, which were then processed with the LSTM algorithm, generating a count of active cells. The average count over all the sub-videos represented one point in the MIC curve, with the respective standard deviation shown as an error bar.

V. ALGORITHMS

A. Segmentation

We developed a robust segmentation algorithm composed of 4 steps: Initial segmentation, Refined Segmentation, Watershed and Artifact removal. The final segmentation I_seg obtained after all steps and is shown in Supplemental Fig. S1e. The used parameters for our experiments were: T_bright = 800, T_dark = −500, S_F = 10, M_ph = 5, M_pw = 10, M_pp = 0.5, M_pd = 5, N_p = 3, N_spur = 4, E_n = 1, S_t = 0.77, O_t = 70, B_t = 60, A_t = 15. All spatial units are measured in pixels.

1). Initial segmentation

1. Initial segmentation is performed by pre-defined global threshold’s T_bright and T_dark (selected through visual inspection), which give a rough estimate of the bacteria locations.

2. To reduce background noise, the grayscale image I (Supplemental Fig. S1a) needs to go through background subtraction. The background is computed by averaging all the frames in the entire video. The subtracted image results in image I_s (Supplemental Fig. S1b).

3. I_s is transformed into two binary images for any pixels with intensities higher than T_bright and lower than T_dark, generating binary images I_bright and I_dark, respectively (Supplemental Fig. S1c and S1d).

2). Refined Segmentation

1. The initial segmentation is refined for each connected component in I_bright and I_dark using the OTSU’s method[45].

2. A bounding box is calculated being the smallest rectangle which encloses the connected component. The bounding box selects an area corresponding to the connected component on the original grayscale image I generating image I_C1 (Supplemental Fig. S2a). The corresponding bounding box on image I_bright or I_dark, is image I_C2 (Supplemental Fig. S2b).

3. The OSTU algorithm is applied to the bounding box, generating image I_C3 (Supplemental Fig. S2c), which contains 3 threshold levels.

4. We use the middle threshold to generate the final binary image I_C4, for the connected component (Supplemental Fig. S2d).

3). Watershed

As overlapping bacteria are frequent, we developed an algorithm to separate overlapping bacteria using a marker-controlled watershed [46] based on the Radon transform [47]. Supplemental Fig. S3a show an example of two overlapping cells, and Fig S3b shows the result after applying the watershed algorithm. The watershed algorithm is based on the following steps:

1. For each connected component, we perform the thinning operation to transform the cells into lines (Supplemental Fig. S3c). This is convenient since the E. coli cells are rod-shaped.

2. Next, we perform the Radon transform on the thinned object and select the maxima for each angle from 0° to 180°. This generates a length plot for each direction, where the peak represents the bacteria orientation.

3. The length plot is circularly shifted so that the global peak is at the center of the plot. This avoids having two peaks, in case the peak is close to 0° or 180°. Connected components containing two or more bacteria should exhibit two or more local peaks (Supplemental Fig. S3d).

4. The length plot is smoothed by smooth factor S_F to correct noisy peaks.

5. Only peaks that obey certain rules are selected, such as having a minimum peak height M_ph, minimum peak width M_pw and minimum peak prominence M_pp.

6. Peaks that are too close by a distance of M_pd, are merged into a single peak with height as the average of the two peaks, and width as the sum of the two widths.

7. The number of obtained peaks per connected component is limited to N_p, to avoid over-segmentation. Priority is given to peaks with greater height.

8. For each peak, a marker is found at the center of the object, along the specified direction, departing from the position where a maxima is found (Supplemental Fig. S3e).

Since the E coli cells are rod-shaped, this approach showed to be more robust than distance transform [48] and radial transform-based watershed methods [49], which would often lead to oversegmentation and to conditional erosion methods [50], complicated schemes with unexpected results, causing either oversegmentation, undersegmentation, or separating cells at imprecise points. Supplemental Fig. S4 compares the results of all these methods.

4). Artifact removal

One of the biggest challenges is eliminating non-bacteria objects originated from artifacts surrounding the bacteria, after background subtraction (Supplemental Fig. S5). Artifacts are removed by post-processing the segmented images according to the following steps:

1. Artifacts usually consist of sharp objects containing lots of spurs and holes inside. Many artifacts can be removed by calculating the number of spur pixels N_spur in the connected components, the Euler number E_n (1 - number of holes inside the connected component) and the solidity Sol (Area / Area of bounding convex polygon). This way, objects with E_n > E_t, N_spur > N_st and Sol < S_t are removed.

2. Another indication of an artifact is that after OTSU thresholding, very often the resulting image occupies almost the entire bounding box or more than 2 entire borders. Thus, objects that occupy more than O_t % of the bounding box area or B_t % of the bounding box border are also removed.

3. Small objects resulting from noise are eliminated by area threshold A_t.

Artifacts that are connected to real bacterial cells usually vanish after the OTSU refinement. Most of the remaining objects can be removed during tracking, since they are of intermittent nature, and result in very short trajectories. We also noticed more occurrence of artifacts for the light cells, most likely because our optical setup has its focus adjusted for the dark cells.

B. Cell Tracking

1). Overview

We developed a tracking algorithm based on a linear Kalman Filter, capable of tracking cells under different focal planes, undergoing color changes.

Cells are tracked from the first to the last frame over windows of 200 frames (2 seconds) using the centroids of all objects obtained from I_bright and I_dark. We also keep track of bacteria color in each frame (dark or bright) to improve the robustness of the algorithm.

Tracking is performed by a linear Kalman Filter[50], [51] and the system is modeled by the equations:

$$\{\begin{array}{c}{X}_{\mathit{\text{priori}}}(t)=AX(t-1)+q(t)\\ Z(t)=CX(t)+r(t),\end{array}$$ (1)

$$X(t)=X_{\mathit{\text{priori}}}(t)+K(Z(t)-C\left(X_{\mathit{\text{priori}}}(t)\right),$$ (2)

where X(t) is the state vector, Z(t) is the observation vector, A is the state transition matrix and C is the observation matrix, K is the Kalman gain and q(t) andr r(t) are Gaussian random variables with distributions p(q) ~ N(0,Q) and p(r) ~ N(0,R). The state vector is described by X = [x,y,v_x,v_y,a_x,a_y]^T, where (x,y) denote position and v and a denote velocity and acceleration, respectively. The state transition matrix and observation matrix are:

$$A=\left[\begin{array}{cccccc}1& 0& 1& 0& 0.5& 0\\ 0& 1& 0& 1& 0& 1\\ 0& 0& 1& 0& 0.5& 0\\ 0& 0& 0& 1& 0& 1\\ 0& 0& 0& 0& 1& 0\\ 0& 0& 0& 0& 0& 1\end{array}\right],$$

$$C=\left[\begin{array}{cccccc}1& 0& 0& 0& 0& 0\\ 0& 1& 0& 0& 0& 0\end{array}\right].$$

To estimate the observation noise matrix R, the transition noise covariance matrix Q and the error covariance matrix P, we manually segmented 10 cells over 200 hundred frames (2000 points) and calculated the error between the automatic segmentation and the manual segmentation, as well as the error between the predicted positions and the real cell positions. Cells were selected from videos with high bacteria concentration (180 minutes after the beginning of a control experiment), to test the algorithm under a stressful scenario. For R, only errors above 0.5 were taken into account. We also found that the maximum cell displacement between two adjacent frames was smaller than 5 pixels (925 nm).

2). Algorithm

1. At each frame, we compute the distance of each measured centroid to the position of all trajectories in the previous frame. For each measured centroid, only trajectories within the search radius S_r are considered as candidates.

2. The a priori Kalman estimate is used to compute the expected position of each trajectory in the current frame (1).

3. Candidate trajectories are sorted according to the distance of their expected positions to the measured centroid. Higher priority is given to trajectories that are currently the same color as the measured centroid, even if distances are greater. The first connected centroids are the ones with closest matches and higher priority is also given to those whose closest matches maintain the trajectory color.

4. Each measured centroid determines the observation vector, which is connected to the closest match as long as this match is not already taken by another cell in the current frame. In the latter case, an attempt to connect to the second closest is made and so on. If no matches are left within the search radius, this represents a new trajectory. This case covers new cells coming from the borders, from a different focal plane and bacteria divisions.

5. Once the observation vector is connected to a trajectory, the new state estimate for this trajectory is determined by the a posteriori Kalman estimate (2). The Kalman filter helps addressing bad segmentations, missed segmentations, and sorting out trajectories that are too close or overlapping.

6. After all measured centroids are connected to different trajectories, some trajectories might be left unconnected. This might happen when two different trajectories converge to the same point, represented by a single measured centroid. In that case, we connect the trajectory to the closest measured centroid to the a priori estimate, as long as this centroid is within S_r and keeps the trajectory color. We found the trajectory color criterion important since artifacts are often mistaken for this case, as they are inherently close to the real cells.

7. If no measurements are found for a determined trajectory, the new state vector is obtained solely from the a priori Kalman estimate. This keeps the trajectory alive if a segmentation fails for that frame, or if the cell momentarily leaves the field of view in the XY plane or is out of focus. However if this trajectory misses a measurement for M_t frames in a row this trajectory is terminated, since the cell has left the field of view for good. All the last M_t estimates are also eliminated since it is more likely that they were not real.

8. It’s also common to have two trajectories stuck in the exact same path, either because an artifact managed to blend with a cell trajectory or because the algorithm lost track of two close real trajectories. In that case, if the common points exceed C_t, we determinate a base trajectory, which is either the oldest one or the one which was connected first, if they were created in the same frame. The other trajectory is then terminated, eliminating all the common points in this trajectory.

The parameters used on our experiments were: S_r = 15 pixels, M_t = 5, C_t = 20.

C. Machine learning

LSTM models were used for two tasks: 1) Creating profiles for active and inactive cells; 2) Bacterial cell motion prediction. The tracks obtained from the tracking algorithm were used as inputs for both neural network models. The Keras library was used with the Theano backend [52] to generate the LSTM models.

1). Active / inactive cell classification

Only trajectories with at least 50 frames are utilized for model training or classification. Short trajectories are padded with zeros to a sequence of length 200, to make the network dimensions compatible and masking is applied to consider only the relevant part of the sequence. Models use 1 LSTM layer of dimension 128, batch size of 32, the Adam optimizer,binary cross-entropy as the loss function and were trained over 10 epochs. The network structure is shown in Supplemental Fig. S6.

2). Bacterial cell motion prediction

We also tested models to predict the bacteria trajectory using LSTM networks and the same manually segmented trajectories used for estimating the Kalman Filter parameters. The aim is to predict the next position from a sequence of previous positions, and we use as entries the sequence of position errors between adjacent frames (i.e. the x and y velocity sequence). We performed leave-one-out cross validation (each trajectory took turns being left out of the model and used for validation), and tried 3 different models:

• Estimating the x and y parameters with 2 independent networks considering x and y random variables with different distributions

• Estimating the x and y parameters with 2 independent networks considering x and y independent and identically distributed random variables (i.i.d.)

• Estimating x and y jointly in a single network whose output is a vector with two real numbers.

All these networks have a similar structure as those from Supplemental Fig. S6 but use sequence length of 20, 1 LSTM layer of dimension 32, batch size of 8, the RMSprop optimizer, linear activation and mean squared error (MSE) as the loss function. Results on Supplemental Table I show the best results were obtained for LSTM with independent X an Y and different distributions. This method presented smaller average MSE over all tested trajectories and smaller maximum error than the physical model that uses Newton’s traditional equations (1). This reveals this approach’s potential to be used in the a priori Kalman estimate and other future studies.

VI. CONCLUSION

We have developed an optical imaging method to track bacteria, identify changes on motion patterns due to antibiotic activity, and obtain the MIC for different antibiotics. We detected motion differences with four different antibiotic classes and generated MIC values similar to CLSI standards in 30–150 minutes. We also developed image processing algorithms for robust cell segmentation and tracking, integrated the algorithms with state-of-the-art LSTM networks, and designed a LSTM network to predict bacterial trajectories.

The present method focuses on tracking active E. coli cells via motion patterns, naturally filtering out inactive cells or impurities in the sample and improving AST accuracy. The LSTM model can also predict bacterial cell trajectories from a sequence of previous positions with accuracy much better than traditional approaches (see Supporting Information). This capacity could further advance understanding of bacterial physiological responses and environmental adaptation.

Future work includes extending the present study to clinical samples, where bacterial concentrations are low (103–105 colony forming units/mL[53]) and impurities are high. In this scenario, the LSTM could be modified and also used for bacterial identification to discriminate between bacterial cells and other particles.