A comprehensive review of image analysis methods for microorganism counting: from classical image processing to deep learning approaches

Abstract

Microorganisms such as bacteria and fungi play essential roles in many application fields, like biotechnique, medical technique and industrial domain. Microorganism counting techniques are crucial in microorganism analysis, helping biologists and related researchers quantitatively analyze the microorganisms and calculate their characteristics, such as biomass concentration and biological activity. However, traditional microorganism manual counting methods, such as plate counting method, hemocytometry and turbidimetry, are time-consuming, subjective and need complex operations, which are difficult to be applied in large-scale applications. In order to improve this situation, image analysis is applied for microorganism counting since the 1980s, which consists of digital image processing, image segmentation, image classification and suchlike. Image analysis-based microorganism counting methods are efficient comparing with traditional plate counting methods. In this article, we have studied the development of microorganism counting methods using digital image analysis. Firstly, the microorganisms are grouped as bacteria and other microorganisms. Then, the related articles are summarized based on image segmentation methods. Each part of the article is reviewed by methodologies. Moreover, commonly used image processing methods for microorganism counting are summarized and analyzed to find common technological points. More than 144 papers are outlined in this article. In conclusion, this paper provides new ideas for the future development trend of microorganism counting, and provides systematic suggestions for implementing integrated microorganism counting systems in the future. Researchers in other fields can refer to the techniques analyzed in this paper.

Introduction

Basic knowledge of microorganisms

Microorganism is a kind of tiny organism which cannot be observed by naked eyes but can be observed by light microscope or electron microscope (Madigan et al. 1997). There are many different types of microorganisms, and the classification standards are various. Generally, microorganisms are composed of bacteria, viruses, fungi and some algae.

• a Bacteria are unicellular organisms with minimal size, simple structure, lack of nuclei, cytoskeletons, and membranous organelles. It widely distributes in soil and water, and most of them are decomposers at the bottom of the biological chain, such as Escherichia coli. Some bacteria are consumers and producers. For example, sulfur bacteria and iron bacteria are producers. They can use inorganic materials to produce organic substances they need. The rhizobia can consume organic substances produced by the photosynthesis of legumes (Doetsch and Cook 2012).
• b The virus is a kind of microorganism that can spread and infect other organisms. It is small and has a simple structure. It contains only one type of nucleic acid, such as ribonucleic acid(RNA) virus and deoxyribonucleic acid(DNA) virus. It must parasitize in living cells and proliferate in the way of replication. Viruses consist of single and double-stranded RNA virus, single and double-stranded DNA virus (Cui et al. 2019). For example, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is a single-stranded RNA virus (Andersen et al. 2020).
• cThe fungus is one type of eukaryotic microorganism, including mold, yeast and mushroom, that can produce spores through asexual and sexual reproduction. Tinea pedis is a kind of foot skin disease caused by pathogenic fungi, which is widely spread globally. There are no sebaceous glands between the soles of human feet and toes, so the environment lacking fatty acids and poor air circulation is conducive to the growth of filamentous fungi (Perea et al. 2000).
• dAlgae are eukaryotes of the protozoa and most of them are aquatic organisms, which can carry out photosynthesis. Algae can be composed of one or a few cells, or many cells aggregate into tissue-like structures. According to the color, algae can be divided into green algae, brown algae and red algae. Red tide is an abnormal phenomenon in the marine ecosystem. It is caused by the explosive proliferation of red tide algae under specific environmental conditions, which is a signal of marine pollution. During the red tide period, a large number of fish, shrimp, crabs, and shellfish die, causing significant damage to aquatic resources and human health (Kirkpatrick et al. 2004).

Some microorganisms are harmful to human beings by causing food decomposition, infect humans and cause diseases, but some microorganisms are beneficial to human beings. Penicillin is an epoch-making discovery in the medical field, which has saved countless lives. Yeast is widely used in industrial fermentation, ethanol production and food production for human beings (Brill 1981). Some microorganisms can degrade plastics, treat waste-water, gas, and have great potential in renewable resources (Rizzo et al. 2013). There are also many microorganisms in the intestines of healthy people, which can help humans decompose and absorb food and toxic substances. Some microorganisms have adverse effects on the human body and industrial production. For example, the human immunodeficiency virus (HIV) can cause the loss of immune function of patients and cause infection. The disease spreads rapidly, has high mortality and cannot be cured, which has caused a significant threat to world health; SARS-CoV-2 breaks out at the end of December in 2019 (Hui et al. 2020). More than 183,000,000 people have been infected worldwide till July 1st, 2021, which becomes a global malignant epidemic (University 2020). The SARS-CoV-2 is highly infectious and mainly transmitted through close contact and respiratory droplets. Microorganisms play an essential role in human’s daily life and production. Therefore, beneficial microorganisms should be used wildly, and harmful microorganisms should be prevented.

Microorganism counting is an essential part of microbial research, which is widely used in food and drug safety tests, biomedical tests, and environmental monitoring (Liu et al. 2004). At present, there are two main methods for microorganism counting and quantification, one of the methods is manual counting, the other one is computer image analysis counting (Rajapaksha et al. 2019). Manual counting mainly includes the plate counting method, hemocytometry and turbidimetry. In the plate counting method, the bacteria are placed in a suitable medium and then wait for them to grow into colonies. After that, the number of colonies is counted through the microscope. The advantage of the plate counting method is that the number of live bacteria can be estimated. However, the operation is complicated, and it takes a period to culture the microorganisms and gets the results. In general, the number of colonies obtained is lower than the actual number of living bacteria because when more than two living bacteria cells stick together, the observed number is still one colony (Balestra and Misaghi 1997). In the hemocytometry method, the bacteria are diluted and dropped on a blood cell counting plate, which is then observed under a microscope to calculate the average number of bacteria in each compartment. Finally, the total number of bacteria is estimated. However, the hemocytometry method cannot distinguish the dead bacteria from the live bacteria and can only estimate the total number of bacteria by the average value, which carries out the low accuracy (Sambrook and Russell 2006). In the turbidimetry method, a spectrophotometer is applied to measure the optical density of bacterial suspension at a particular wavelength. The cell concentration in bacterial suspension is proportional to the turbidity of bacterial suspension within a specific range, that is, the cell concentration is proportional to the optical density. So, the number of bacteria can be expressed in terms of optical density. However, the turbidimetry method has specific requirements for the wavelength of light in the experimental environment, which should be controlled within the line limit range where the bacterial concentration is proportional to the optical density. Otherwise, the measurement result will have a large error (Dalgaard et al. 1994).

It can be seen that the traditional methods can obtain satisfactory counting results under certain conditions, such as when the number of samples is small and the imaging effect is good under the microscope. However, when the sample becomes larger, it is often encountered that the colony is small, the contrast between the colony and the culture medium is not clear, and it is not easy to detect and count with naked eyes. The detection results have the problems like large errors and poor reliability. The sample image contains many particles, and the workload is heavy and dull, which is easy to cause misjudgment. Moreover, the subjectivity of manual counting is common. Even if the same staff member observes the same sample in different periods, different observation results will be obtained (Chien et al. 2007). With the development of computer image analysis technology, automatic particle image analysis systems based on image processing and visual analysis can automatically, quickly and objectively count the number of particles contained in the image and extract various characteristic parameters of particles, which significantly reduces the workload and improves the analysis accuracy, so it has been widely used. The image analysis system for microorganism counting can improve counting performance if the quantity of sample is large (Thiran et al. 1994). An example image of yeast cells is shown in Fig. 1. The precise boundaries of microorganisms make it possible to separate and count the number of colonies by image analysis.

Fig. 1.

An example of yeast cells image (in (Dietler et al. 2020) proposed dataset)

Motivation of this review

Digital image processing (DIP), also known as computer image processing, refers to converting an image signal into a digital signal and processing it by computer. DIP first appeared in the 1950s, when the electronic computer has developed to a certain level. People can use the computer to process images and improve image quality (Gonzalez et al. 2004). The commonly used digital image processing methods include image enhancement, denoising, restoration, coding and compression. DIP has been widely used in many fields. Agricultural and forestry departments understand the growth of plants through remote sensing images, estimate the yield, and monitor the development and management of diseases and insect pests (Amrita and Kaur 2016). Through remote sensing image analysis, the water conservancy department can obtain the change of water disaster (Sudiana and Rizkinia 2012). The meteorological department is used to analyze the meteorological cloud chart and improve the accuracy of the forecast (Chatterjee and Chaulya 2019). The department of national defense, surveys, and mapping use aerial surveys or satellites to obtain regional landform and ground facilities (Feifei et al. 2017). The mechanical department can use image processing technology to analyze and identify the metallographic diagram automatically (Privezentsev et al. 2019). Medical departments use various digital image analysis technologies to diagnose various diseases automatically (Salvi et al. 2020; Madabhushi and Lee 2016; Li et al. 2020a). Because of the flexibility and universality of DIP, there are no complex measurement steps involved, which means it has low learning cost. In the field of microorganism analysis, expensive equipment is usually needed to ensure the accuracy of the measurement. DIP can save this part of this cost (Ekstrom 2012). Therefore, DIP has been widely used in microbial counting in many types of research. Its development trend is shown in Fig. 2, which has shown a good development trend so far.

Fig. 2.

The total number of related works on microorganism counting approaches

As shown in Fig. 2, the application of digital image processing in the field of microorganism counting has been explored. Since the 1980s, DIP has been applied to microorganism counting. From 1980 to 1995, the application and development of this field is relatively slow, but it is rapidly developed from 1995 to 2010. After 2010, the number of research for microorganism counting increases faster. As for machine learning and deep learning based microorganism counting methods, machine learning is firstly applied in this field in 1987, and then it is slowly developed from 1990 to 1995. Since 1990, microorganism counting methods based on machine learning and deep learning is increasing steadily, and it has a tremendous development since 2015. According to the content of the papers, a possible reason is summed up, that is, the development of the deep learning algorithm can lead to more accurate image segmentation. For example, the segmentation of adherent colonies can lead to more precise microorganism counting.

Related reviews

The microorganism counting is an essential topic in microbial research, and the relevant works are also relatively abundant. Many researchers have written relevant reviews, which are summarized as follows:

Review Gray et al. (2002) outlines several image analysis methods for algal cell estimating, and several image segmentation methods based on thresholding, edge tracking and template matching are compared. There are 32 papers summarized, and only three are about the algal counting method. Review Qiu et al. (2004) describes the development course of bacteria counting and cell size measurement, which contains the classical methods and automated flow analysis technology. More than 33 papers are summarized and 7 of them are about bacteria counting. Review Gracias and McKillip (2004) describes the use of fluorogenic or chromogenic to classify different species of bacteria and impedance technology for enumeration. There are more than 25 papers are about traditional food bacteria counting methods in total 103 papers. Review Daims and Wagner (2007) indicates that the difference between microorganism counting and biovolume measurement is whether or not to identify individual objects (cell or cell clusters) in the biomass. There are six papers about automatic cell counting in total 92 papers. Review Barbedo (2012a) describes the object counting methods using digital image processing. The methods are composed of morphological operation, filtering operation, contrast enhancement, transformation, edge detection and image segmentation. They summarized over 130 papers, among them, 29 papers are used for cell counting and 13 papers are about bacteria counting. Review Dazzo and Niccum (2015) describes the use of CMEIAS for both microorganism counting and biovolume measurement based on image processing. The hierarchical tree classifier and k-Nearest Neighbour classifier are applied for classification. There are 65 papers in total, and more than 30 papers are used for cell counting. Review Li et al. (2019a) describes computer-based microorganism image analysis development and introduces different methods for different microorganism classification. This review is a comprehensive microorganism classification paper. It uses plenty of works of literature for quoting, but there is no significant description for microorganism counting in more than 300 papers in total. Review Puchkov (2019) describes the main quantitative analysis methods of single bacterial and yeast cells at the cellular and subcellular levels. More than 150 papers are summarized. This review mainly introduces several techniques for scanning, but there is no straightforward application of DIP in microorganism quantification.

Although the reviews above are excellent enough and the descriptions about the current situation of microbial research are objective and detailed. However, there is no targeted research about image analysis based microorganism counting, so it is necessary to do additional research on this aspect. For a clear overview, a histogram (Fig.  3) is used to show each of the related survey papers and their contribution to microorganism counting with our proposed studies. Because of the vital role microorganism quantification plays in microbial research, this review focuses on the application of microorganism counting and summarizes each method’s development and prospects. This review has great reference value for microbiological researchers and computer vision researchers. There are more than 136 papers are used for microorganism counting.

Fig. 3.

A comparison among recent survey papers. Number of summarized papers in the existing review in comparison to their contribution to image analysis based microorganism counting methods

Microorganism counting methods

In order to expound the approach of microorganism counting, the organization chart of this review is shown in Fig. 4. The approach contains five steps: microbiological data acquisition, microscopic image, image pre-processing, microorganism counting methods and evaluation methods.

Fig. 4.

The organisation chart of microorganism counting approaches in this paper

Firstly, according to the different application domains, microorganisms are composed of the following seven categories: agricultural microorganism, environmental microorganism, food microorganism, industrial microorganism, medical microorganism, water-borne microorganism and other microorganisms. Then, the samples are stained and sliced. After that, the microscopic images are captured by the imaging equipment, such as a charge-coupled device (CCD) camera (Gmür et al. 2000).

Next, in the pre-processing part, the images are denoised and enhanced to improve the contrast between the object particles and the background. In the process of microorganism microscopic image scanning, the loss of information in the process of electronic transmission and the pretreatment process, such as staining may bring some noise to the microscopic image. In order to improve image quality, image processing can be used to reduce or remove these noises. The main methods to remove noise are wire filter (such as Gaussian filter and Mean filter), median filter and so on (Li et al. 2020b).

The next step is microorganism counting (Li et al. 2020b). The objects of the microorganism counting method are separating the adherent colonies and counting. Image segmentation is an essential part of this task, which contains three broad categories: threshold segmentation, edge detection and region extraction. The initial image segmentation method is threshold segmentation, whose core algorithm is the selection of the threshold. At present, there are two main methods, one is based on the iterative method and the other one is Otsu thresholding. For the image with prominent double peaks and deep valley bottom, the iterative method can get satisfactory results quickly, but for images with significant differences in the ratio of target and background, the iterative method cannot segment the target well (Perez and Gonzalez 1987). Another standard method is the maximum inter class variance based Otsu method which can achieve good segmentation results for most images (Otsu 1979). The advantages of Otsu segmentation are fast and straightforward calculation, not affected by brightness and contrast of images, and most of the segmentation results are satisfactory. Nevertheless, it has limitations such as the sensitivity to noise and cannot support semantic segmentation functions (Xu et al. 2011). Edge detection mainly includes gradient and second-order differential operator based methods, Laplace of Gaussian function (LoG) edge detection method and Canny edge detection method. The gradient detection method is the most widely used method among them that usually contains Roberts, Sobel, Prewitt, Kirsch and Robinson (Gonzales and Woods 2002). The Watershed method is one of the most popular methods in the region extraction domain, a closed region signature method based on region growth (Levner and Zhang 2007). The image segmentation is considered according to the composition of the watershed. The calculation process of the watershed is an iterative labeling process, which has an excellent response to weak edges. However, the watershed algorithm may lead to over-segmentation because of the noise or slight gray-level change of object surface (Strahler 1957).

Another critical part of microorganism counting is morphological operations which contain erosion, dilation, open and close. The erosion operation uses structural elements to erode the input image, eliminating the image’s boundary points. It can reduce the size of the object, filter the image interior and eliminate the isolated noise points effectively (Jackway and Deriche 1996). Dilation operation is the dual operation of erosion operation. The dilation operation can merge all the background points contacted by the target object into the object, which can increase the target and the shrink holes (Jackway and Deriche 1996). The open operation is using the erosion operation firstly and then use the dilation operation. The open operation can eliminate the isolated points in the image, eliminate the burr and connect the two domains so that the outer boundary of the image can be polished by the open operation (Chudasama et al. 2015). The close operation is the opposite of the open operation, which means the image is dilated first and then eroded. The close operation can fill small holes, close small cracks, and polish the inner boundary of the image (Chudasama et al. 2015).

After image segmentation, the microorganisms need to be classified and counted respectively. Machine learning is widely applied in image classification, which has been developed rapidly. Principle component analysis (PCA) is an unsupervised machine learning algorithm, which is always applied for exploration and dimension reduction of higher dimensional data (Roweis 1998). More comprehensible features can be extracted, and valuable information of the sample can be processed faster by using dimension reduction. In addition, dimension reduction can also be applied to visualization and denoising. The primary process of PCA is to map n-dimensional features to k-dimensional features, which are new orthogonal features, also known as principal components. Then the k-dimensional features are re-constructed based on the original n-dimensional features. PCA can increase the sampling density by dropping part of the information, which is helpful for the curse of dimensionality. However, PCA retains the primary information, which is only for the training set, but the primary information is not necessarily meaningful. So the overfitting may be exacerbated by using PCA (Karamizadeh et al. 2013). Support vector machine (SVM) is one kind of generalized linear classifier for data classification by using supervised learning (Vishwanathan and Murty 2002). The object of SVM learning is to find the separation hyperplane with the most considerable geometric interval, which can divide the training data set correctly. The learning strategy of SVM is to maximize the interval, which can be formalized into a problem to solve the convex quadratic programming. The selection of SVM kernels can make it to be a nonlinear classifier, such as polynomial kernel, RBF kernel, Laplacian kernel and Sigmoid kernel (Han et al. 2012).

Artificial neural network (ANN) is a mathematical model of distributed and parallel information processing that imitates animal neural networks’ behavior characteristics. The most commonly used ANN is Multilayer perceptron (MLP), which is a feedforward ANN model (Ghate and Dudul 2010). ANN is composed of input layers, hidden layers and output layers, which are fully connected to each other. The structure of ANN is shown in Fig. 5. ANN is composed of many simple neurons, and each neuron receives input from other neurons. In this way, every neuron restricts and influences each other to achieve nonlinear mapping from input state space to output state space. ANN is a combination of many same simple processing units in parallel. Although the function of each unit is simple, the parallel activities and the ability of information processing are unique. ANN can realize the memory of information through its network structure, and the memory information is stored in the weights between neurons. This makes the network has good fault tolerance and can handle the pattern information processing such as clustering analysis, feature extraction and defect pattern restoration (Zupan 1994).

Fig. 5.

The structure of ANN

Back propagation neural network (BPNN) is a supervised learning, which is developed from ANN (Karsoliya 2012). The loss function in BPNN is optimized based on back propagation. In forward-propagating, the data is processed from input layers to output layers. In back propagation, the loss function is transmitted from output layers to input layers, then the weights and biases are optimized based on the gradient descent method. BPNN can carry out the nonlinear mapping from input to output, and can still make the correct mapping for the new non-sample data, which has a specific generalization ability and fault tolerance ability (Dai and MacBeth 1997).

Convolutional neural network (CNN) is one kind of feedforward ANN, that is wildly applied in DIP and computer vision (Li et al. 2016). The convolutional kernels are applied to scan the whole image and the deep features are then extracted. After pooling, the image can be classified through a fully connected layer. The loss function is minimized with back propagation (Chauhan et al. 2018). CNN is developed rapidly after 2010, and it is not only be applied for classification, but segmentation (Unet) and image generation (GAN). VGG-16 is one of the most popular CNN, which is composed of five convolution layers, three pooling layers and three fully connected layers Simonyan and Zisserman (2014). The structure of VGG-16 is shown in Fig. 6. Only 3 $\times$ 3 filters are applied in VGG-16 because the combination of small filters can simulate a larger filter, reducing the parameters and improving the nonlinear ability.

Fig. 6.

The structure of VGG-16 (in Nash et al. (2018) Fig. 3)

The last step is system evaluation, which can help researchers systematically perceive the image processing results. The results of the evaluation can help to prompt the accuracy of the system. Counting accuracy is a standard evaluation method in target counting, which is the ratio of the number of detected targets to the ground truth. Generally, the accuracy can only be used to evaluate the global accuracy, because it cannot show whether the detected target and the ground truth target are one-to-one corresponding (Bloem et al. 1995). Evaluation of image segmentation and image classification can also reflect the performance of the counting system. True positive (TP), false negative (FN), false positive (FP) and true negative (TN) are four basic metrics in image classification (Zhang et al. 2008b). Pixel accuracy (PA) is one of the simplest evaluation methods for image segmentation, which means the ratio of the number of correctly classified pixels and the number of whole pixels. The mean pixel accuracy (MPA) is the improved method of PA, which indicates the mean PA of all classes (Zhang et al. 2008b). Mean intersection over union (MIoU) is the ratio of intersection and union of ground truth and predicted segmentation result. It can be regarded as the mean ratio of TP and the union of TP, FN and FP in the process of image segmentation (Rahman and Wang 2016).

Structure of this review

In this review, a comprehensive overview of microorganism counting using image analysis is presented. The relevant research in the microbial application has been investigated since 1980, and the applications of microorganism counting in different situations are discussed. Furthermore, this paper also summarizes the research motivation and research methods of microorganism counting in the microbial field. The review articles related to this research are also summarized, and the structures of their references are recorded. More than 144 papers are selected from the initial paper dataset and the structure of the systematic review is shown in Fig.  7. The initial papers are searched from Google Scholar, IEEE, ACM, Nature, Science, Cell, Elsevier, Wiley, Hindawi, IOP, PloS, BMC and Springer, and the keywords contain “microorganism counting”, “bacteria counting”, “cell counting”, “algae counting” and “fungus counting”. Then the duplicate and irrelevant papers are deleted. There are 57 papers about microorganism biovolume counting, which do not conform to this review. Finally, 144 papers are about microorganism counting methods, containing 8 review papers, 82 bacteria counting methods, and 54 other microorganism counting methods.

Fig. 7.

The systematic flow chart of paper selection for our work

The review is structured as follows: In Sect. 2, the related works of image analysis based bacteria counting are introduced. In Sect. 3, the related works of other microorganisms counting based on image analysis are introduced. Then in Sect. 4, the commonly used microorganism counting methods are analyzed, and their different application domains are summarized. Finally, in Sect. 5 this review is concluded by summarizing the whole paper. This review structure can help microbiological workers clearly and quickly understand the development status of this field and obtain the relevant content they need.

Bacteria counting methods

Bacteria are one crucial part of the ecosystem because of the cardinal role they play in the carbon and nitrogen cycle, which are closely related to human daily life (Madigan et al. 1997). Therefore, bacteria counting has become one of the most important directions in the field of microorganism counting, including the study of bacteria number and colony size. Thus, the bacteria counting methods are summarized in this chapter. Bacteria counting is of great significance in food safety monitoring and industrial safety detection, but manual counting is tedious and redundant work, that is very subjective. Therefore, the research of computer image analysis based bacteria counting is significant. This chapter is structured as follows: the first part summarizes the classic bacteria counting methods. The second part is bacteria counting methods based on machine learning and deep learning. The last part is third-party tools methods.

Classic counting methods

Bacteria counting method based on image enhancement

In Pettipher and Rodrigues (1982), Niyazi et al. (2007), the gray-level contrast is applied for bacteria counting. The contrast is used for counting bacteria and somatic cells of milk (Pettipher and Rodrigues (1982)). In Niyazi et al. (2007), the maximum size of one colony (defined by the area) and the distribution of the gray color within the colony are further measured beside the gray-level contrast, which has the mistake of less than 3%. The colony counting result is shown in Fig. 8.

Fig. 8.

Screenshot of the program. The left panel is the scanned flask and the right panel is the segmented result (in Niyazi et al. (2007) Fig. 1)

In Shenglang and Yongguang (2005), histogram equalization is used to enhance the contrast of bacteria images as pre-processing. Then a convolutional filter is used to extract the curvature feature of bacteria and remove the background features. After that, a median filter is used to remove noises. The basis of judging whether there are bacteria in the image is whether the maximum connected number of non-zero gray pixels in the image exceeds a certain threshold. Finally, the number of connected domains is calculated as the number of bacteria in the image. The result shows that the accuracy is more than 95% while counting the number of bacteria.

In Buzalewicz et al. (2010), optical transforms are used for the determination of bacteria colony number. Because of the scale invariant of Mellin transform, the combination method of Fourier transform and Mellin transform is applied. First, the Fourier transform of the input objects is calculated and the high pass frequency filter is applied to eliminate the zero-order component of the Fourier spectrum. Then the Mellin transform followed by the log-polar transformation is performed. Moreover, the two-dimensional Fourier transform is computed in order to obtain scale and rotation invariance. Finally, the value of the Mellin spectrum is used to evaluate the number of the analyzed object. A good agreement between calculations and manual counting for twelve samples is achieved (the differences range from 1 to 3% and the standard deviation is equal 4.51).

Bacteria counting method based on thresholding

In Masuko et al. (1991), Trujillo et al. (2001), Chunhachart and Suksawat (2016), Gupta et al. (2012), Sethi and Yadav (2012), Kaur and Sethi (2012), Nayak et al. (2010), Pernthaler et al. (1997), Sotaquira et al. (2009), Maretić and Lacković (2017), Chunhachart and Suksawat (2016), Kaur and Sethi (2012), Payasi and Patidar (2017), global thresholding is applied for bacteria enumeration. The RGB image is firstly converted to YCbCr and Lab color spaces in Sotaquira et al. (2009). An adaptive median filter (Gupta et al. (2012), Kaur and Sethi (2012)), a‘flatten filter’ (Trujillo et al. (2001)) or a top-hat algorithm (Pernthaler et al. (1997)) is applied for denoising. Besides, a combination method of Gaussian low-pass filter, simple symmetric moving average filter and median filter is applied for noise removal in Maretić and Lacković (2017). After thresholding, the morphological operations are applied for image enhancement (Chunhachart and Suksawat (2016), Kaur and Sethi (2012)), and the detected circles are used for counting the number of colonies that appeared on the selected region. The detection result is shown in Fig. 9 and the average percentage error of 2.13% is obtained in comparison to the counting by the expert. In Payasi and Patidar (2017), the RGB image is converted to HSI color space image and the image is then segmented based on thresholding. Then the noises are removed, so the labeling and counting of the bacilli in the image will be possible. Afterward, the boundaries of bacteria are detected and stored, and the area and perimeter are calculated. If there is a clump of bacilli, the count of bacilli is increased by the integer, which is closed to the ratio of the area of the clump to the average area of bacilli. The result is shown in Fig. 10 and accuracy of 90% is obtained.

Fig. 9.

The detection result (in Chunhachart and Suksawat (2016) Fig. 8)

Fig. 10.

The counting result. a Original image. b Image after shape characterization and segmentation (in Payasi and Patidar (2017) Fig. 8)

In Shen et al. (2010); Austerjost et al. (2017), the iterative local threshold is applied for bacteria colony counting. First, a median filter and contrast enhancement are applied to remove noises and enhance images (Shen et al. (2010)). Then the iterative local threshold method is used for image segmentation. After that, the petri dish edge is removed by detecting the connected region with the maxima white pixel. Finally, the number of bacteria is counted based on eight neighborhoods in Shen et al. (2010) and the average relative error of 2.5% is obtained. In Austerjost et al. (2017), after thresholding, the region of interest is examined for objects which will be divided into single colonies and colony clusters by using a classification algorithm based on the previously defined threshold. Afterward, a Hough circle transformation is applied for the segregation of colony clusters into single colonies. The last part of the algorithm is dedicated to finding colonies that could not be detected within the previous steps. For this, the sizes of previously found colonies are compared with other objects found on the plate. If these objects fit into the size range of previously found colonies and have a suitable roundness, they are recognized as a colony. Finally, the detected colonies are all counted with an average accuracy of $86.76\pm 9.76\%$.

In Jun (2010), Clarke et al. (2010), Marotz et al. (2001), feng Hu (2013), Boukouvalas et al. (2018), Matić et al. (2016), Siqueira and de Carvalho (2017), the adaptive threshold is used for bacteria counting. First, a median filter (Jun (2010), Boukouvalas et al. (2018), Siqueira and de Carvalho (2017)), a Gamma correction (Matić et al. (2016)) and a Gaussian filter (Clarke et al. (2010)) are used for noise removal. Then the extended minima function is used to find the center of the colonies (Clarke et al. (2010)). Finally, a saturation based adaptive thresholding is applied for image segmentation. The morphological operations such as opening and closing are used for adherence colonies segmentation and image smoothing. The segmentation procedure is shown in Fig. 11. The results correlate well with the results obtained from manual counting, with a mean difference of less than 3%. Moreover, the distance transform and progressive erosion are applied in feng Hu (2013) to separate connected colonies into a single one. The counting result of Matić et al. (2016) is shown in Fig. 12. In Boukouvalas et al. (2018), the circular area is detected through Hough transform to obtain only the inner area of the dish and a mask is created for the removal of the unwanted area. Afterward, Gaussian adaptive thresholding is performed for image segmentation because of the different lighting conditions in different areas. Then the histogram of vertical projections of the image is analyzed by varying its rotation angle to align the stripe at a ${90}^{\circ }$ angle. Finally, cross correlation-based granulometry is applied for the determination of the amount of bacteria colonies.

Fig. 11.

Illustration of the colony counting procedure. a The initial image. b The image after thresholding. c The extended minima of the original image. d The counted colonies (in Clarke et al. (2010) Fig. 4)

Fig. 12.

Test result (in Matić et al. (2016) Fig. 7)

In Zhang et al. (2008a), Alves and Cruvinel (2016), Sánchez-Femat et al. (2016), Boukouvalas et al. (2019), Otsu thresholding is applied for bacteria counting. First, a linear expansion of the histogram (Sánchez-Femat et al. (2016)) and a multi-directional Sobel operation (Boukouvalas et al. (2019)) is applied for image enhancement and edge detection. In Zhang et al. (2008a), the RGB and achromatic are processed, respectively. For RGB images, the Otsu thresholding method is firstly used for segmentation, then the color similarity in HSV (Hue-Saturation-Value) color space is adopted to assist the colony boundaries detection. For achromatic images, the sizes of all objects detected by the Otsu method from the dish/plate region are collected, and the frequency distribution with log base of those size values is generated. Colonies of similar size should occupy the high frequency segment in this distribution, and the frequencies for those massive artifacts should be very low. By this assumption, the large size objects can be removed. Then the hypothesis testing is used to remove minor artifacts which are very similar to the colonies. After Otsu thresholding, a Laplacian filter is applied for edge detection and circular Hough transform is used to detect circular bacteria colonies in Alves and Cruvinel (2016), Boukouvalas et al. (2019). The mean error between the proposed method and the manual counting method is less than 10%. In Sánchez-Femat et al. (2016), Euler’s method is applied for colony counting. The accuracy of 98% is obtained by comparing with the proposed method and manual counting method.

In Zhang and Chen (2007), Chen and Zhang (2008), the Otsu and watershed are applied for automatic detection and enumeration of bacteria colonies. First, the contrast-limited adaptive histogram equalization (CLAHE) is used on the converted gray-scale images to enhance the dish/plate contour (Zhang and Chen (2007)). Then, the Otsu threshold is used to detect the dish/plate region and binarize the images automatically. After the morphological operation is used to fill holes, the color similarity values between a pixel and its eight neighbors are calculated and the minimum value is used to detect the object boundaries. Moreover, the watershed algorithm is used for clustered colony separation and the number of viable colonies is counted. The proposed counter performs very well on the blue medium dish/plate, which has average precision, recall, and F-measure values of 0.97, 0.96, and 0.96, respectively in Zhang and Chen (2007). In Chen and Zhang (2008), The precision, recall, and F-measure values of the proposed counter are $0.61\pm 0.29$, $0.94\pm 0.06$, and $0.69\pm 0.20$, while the corresponding values of the Clono-Counter are $0.22\pm 0.25$, $1.00\pm 0.00$, $0.29\pm 0.31$, respectively. The segmentation method is shown in Fig. 13.

Fig. 13.

a The original image. b The plate mask. c The colony mask. d Colonies separated from aggregated colony clusters (in Chen and Zhang (2008) Fig. 2)

Bacteria counting method based on edge detection

In Massana et al. (1997), Ogawa et al. (2003), Yamaguchi et al. (2004), Choudhry (2016),edge detection is applied for bacteria counting. A Gauss filter (kernel $5\times 5$), a Laplace filter (kernel $5\times 5$), and a median filter (rank 3) are used for edge detection before thresholding (Massana et al. (1997)), which is shown in Fig. 14. On the contrast, the Sobel and Laplacine filter are used to detect the edges after thresholding (Yamaguchi et al. (2004)). In Choudhry (2016), the edge detection system has six major steps. First, the background is subtracted to enhance contrast and reduce the effects caused by uneven illumination. The radius for background subtraction is determined empirically. A starting number can be the average radius of colonies. The next step is sharpening and enhancing the image, which follows by finding the edges. Sobel filter is used in the macro. Then the image is smoothed using Gaussian blur, and converted to black and white. Alternatively, the image can be smoothed by sequential dilate and erode steps. This is followed by the closing of the edges to form a closed circle. Closed objects are filled black using holes filling resulting in images containing black colonies on a white background. To ensure that all colonies are detected, an additional step of closing and filling holes is performed. Here, the size of each pixel is increased, in order to bring the detected edges closer to each other that allows the detection of colonies whose entire edge along the perimeter fails to be otherwise detected. After filling, the size of the pixels is reduced to return the colony size to their original values. After that, denoising and segmentation are applied to remove small particles and separate clustered colonies using thresholding. Finally, the objects are filtered based on size, circularity and measured. Then a new pipeline is developed for the detection of cells and colonies from images. The background is corrected and then the colonies are detected. After that, the parameters are measured and the number of colonies is counted.

Fig. 14.

Overview of the whole process of image processing (in Massana et al. (1997) Fig. 2)

In Barbedo (2013), five digital processing methods for automatic colony counting are proposed and compared. In the first method, a Gaussian Laplacian filter is applied for edge detection and the connected regions are identified and counted after holes filling. In the second method, the Gaussian Laplacian filter is replaced by the Canny filter. In the third method, three thresholding values are used for image segmentation. In the fourth method, thresholding is used for histogram equalization but not for image segmentation. In the fifth method, the region growing method is applied for segmentation. After that, the concave surface between the connected colonies can be detected to separate the colonies into a single one. Finally, the number of colonies is counted. The accuracy of the first method performs best that obtains the accuracy of 99%.

Bacteria counting method based on watershed

In Ates and Gerek (2009), Selinummi et al. (2005), watershed is applied for bacteria counting. In Ates and Gerek (2009), a median filter is applied first for noise removal and the petri dish boundary is detected and removed. Then the patterns are separated into two groups: colonies and clusters of colonies, based on the classification of circularity ratio. After that, the cluster colonies are segmented based on the watershed (Ates and Gerek (2009)) and marker-controlled watershed(Selinummi et al. (2005)). The watershed segmentation method is shown in Fig. 15. Finally, the number of actual colonies is estimated as the ratio of cluster area to an average colony area.

Fig. 15.

Watershed segmentation for two merged colonies (in Ates and Gerek (2009) Fig. 5)

In Hong et al. (2008), Brugger et al. (2012), Masschelein et al. (2012), Zhu et al. (2018), Wong et al. (2016), Minoi et al. (2016), Yujie (2009), Kan (2008), Fang et al. (2008), Martinez-Espinosa et al. (2016), Mukherjee et al. (1995), the distance transform and watershed are applied for bacteria counting. First, a median filter is used to remove noise and determine the threshold for every single patch in Kan (2008), Fang et al. (2008). After that, the iterative threshold (in Fang et al. (2008)), a gray-scale weighted thresholding method (in Hong et al. (2008)), a combination method of distance transform and region growing (Mukherjee et al. (1995)), and Otsu thresholding (in Brugger et al. (2012), Minoi et al. (2016)) are used to obtain the binary image. Then the objects are detected based on eight neighbor regions in Martinez-Espinosa et al. (2016). Moreover, an adaptive thresholding method is applied in Brugger et al. (2012) for secondary binarization to solve the challenges that come from the fact that bacterial strains from the same species may exhibit different colony phenotypes. In Zhu et al. (2018), image subtraction is carried out to extract the candidate colonies, which are connected to the inner circle of the agar plate and a nonlinear gray transformation is used to enhance the gray-scale. Afterward, a distance transformation is performed on the binarized image and segmentation is done with a watershed transformation. Furthermore, the sharp corners produced by the watershed transformation are removed by using the morphological opening method. After the segmentation algorithm is completed, the Bayes classifier distinguishes the remaining concatenated groups into classes of one, two, three or four containing colonies, and the final colony is counted. Finally, in Masschelein et al. (2012), Yujie (2009), the GLCM is extracted and the SVM is applied for classification. The total number of single and clustered colonies is counted with an average relative error of 0.2% in Zhu et al. (2018). The counting result is shown in Fig. 16. The processing method of Mukherjee et al. (1995) is shown in Fig.  17.

Fig. 16.

Identified colonies displayed in different colors (in Zhu et al. (2018) Fig. 11). (Color figure online)

Fig. 17.

a Original image. b Image after thresholding at gray value 125. c Image after distance transform. d Image after region growing. e Image after component labelling. f Frequency distribution (in Mukherjee et al. (1995) Fig. 3). (Color figure online)

Bacteria counting method based on color segmentation

In Ogawa et al. (2005), a distinctive multicolor segmentation algorithm is applied for the accurate and simultaneous differentiation of triple-stained bacteria. The result has the 95% confidence intervals of the regression.

In Schönholzer et al. (2002), Peitz and van Leeuwen (2010), Mukti et al. (2010), RGB images are separated into three channels. The preliminary detection of both bacteria and debris is based on the green channel, and the differentiation between bacteria and debris is based on the processing of the green and blue channels. Then the debris particles are eliminated by the combination of two output images above. Then a Gaussian filter is applied for noise removal and Otsu thresholding is used to roughly separate the data of the relatively dark electrodes from data belonging to the electrode gaps in Peitz and van Leeuwen (2010). Moreover, the numbers of single and dividing cells and cell agglomerates are determined by a method based on the number of local grey value maxima in Schönholzer et al. (2002). Finally, cell numbers and cell sizes are calculated based on area and perimeter measurements for each single or dividing cell.

Machine learning and deep learning counting methods

In Ishii et al. (1987), Yoon et al. (2015), Chiang et al. (2015), principal-component analysis (PCA) is used to separate the biological pattern with the surrounding area. The type of pattern for selection is identified and the objective biological pattern is counted. Moreover, the nearest neighbor searching algorithm is applied to separate touching colonies after PCA in Yoon et al. (2015), which contains three main steps. First, the local maxima on an absorbance image is found, and a mask image is created in which the locations of the local maxima are marked with 255 and otherwise with 0. Then the local maximal pixels outside the binary segmentation image are masked out by a logical AND operation. Afterward, the clumped blobs are split when the number of local maxima is greater than the number of blobs. The separation result is shown in Fig. 18. Finally, the image of the bacteria colony is segmented and counted. The accuracy of the colony segmentation and counting algorithm is over 99%. However, in Chiang et al. (2015), the Otsu thresholding is applied for segmentation after PCA. Then, the distance transform and waster-shed are applied for the division of overlapping colonies. Afterward, the bottom-hat transformation is applied to extract colonies from the rim image. Comparisons show that the proposed system is an effective method with excellent accuracy with a mean value of absolute percentage error of 3.37%.

Fig. 18.

Local absorbance maxima search and separation of touching colonies (in Yoon et al. (2015) Fig. 14)

In Andreini et al. (2015), Andreini et al. (2016), Zhang et al. (2010), Chen and Zhang (2009), SVM is applied for bacteria counting and classification. In Andreini et al. (2015, 2016), the colonies are segregated from the background by a background removal process based on chromatic information about the specific chromogenic medium used in the culture. Then, a supervised training technique is adopted to obtain a chromatic description of the background and the uncertainty region is obtained as a union of the intersections of some binary masks obtained by imposing a threshold on the probability level of the background and of the infected regions. After that, a mean shift segmentation algorithm is used to associate each image pixel to the corresponding modal density value and a Sobel based edge enhancement is applied to distinguish different classes. Moreover, the considered uncertainty region is divided into two subregions based on the computed thresholding and the histograms of the two sub regions are calculated and compared to establish if a significant separation exists. Finally, SVM is applied for classification and the number of colonies is counted with an accuracy of 99.2%. The segmentation result is shown in Fig. 19. In Zhang et al. (2010), Chen and Zhang (2009), a subtraction operation is applied between the original image and background image to eliminate the background unevenness caused by the light source. Then a median filtering algorithm is applied to smooth the image because it reduces the hot-electron noise and the noise caused by environmental disturbance during image collection, quantify and transmission, and overcomes the blur of the image details created by linear filtering. After that, the gray-level histogram equalization is applied for image enhancement. Then the Otsu thresholding (Zhang et al. (2010)) and watershed (Chen and Zhang (2009)) are used to obtain the binary image. Finally, the shape features are extracted and used for SVM training to identify and count bacteria. It can be seen that the counting results of SVM have a small difference from that of human eye recognition and its relative error is less than 3%, which means that SVM can be used for rod-shaped bacteria counting.The classification results of Chen and Zhang (2009) is shown in Fig. 20.

Fig. 19.

a The original image. b The identified edges within the background. c Candida colonies found on the Petri dish (in Andreini et al. (2015) Fig. 4)

Fig. 20.

The classification results for different sizes of the training set (in Chen and Zhang (2009) Fig. 15)

In Blackburn et al. (1998), the Marr-Hildreth operator is used for edge detection of bacteria image and threshold is used for image binarization. Then a rank 3 filter is applied to remove pixels that have intensities equivalent to the intensities of amplified background noise. An artificial neural network (ANN) is firstly applied for the classification of bacteria. The ANN is composed of 6 input nodes, 5 intermediate nodes, and 3 output nodes, and then the ANN is activated by using a sigmoid activation function. After training, the images can be analyzed automatically at a rate of 100 images per h. Minimal variation in cell counts between filters is observed (5%) with the filtering procedure used. The bacteria counting procedure is shown in Fig. 21.

Fig. 21.

Edge detection procedure. a The original image. b Image after application of the Marr-Hildreth operator with $3\times 3$ kernel. c Image after application of a rank 3 filter. d Binary image after thresholding. e Image after erosion (in Blackburn et al. (1998) Fig. 1)

In Shenglang et al. (2008), Hongwei (2012), back propagation (BP) neural network and DIP are used for analysis and counting for the microscopic image of bacteria. Median filtering and adaptive filtering are used for denoising and background elimination, and then the iterative algorithm is used for image segmentation in Shenglang et al. (2008). In Hongwei (2012), the Otsu thresholding method and the combination of square and circle filter are used for segmentation and edge detection of other microorganisms. After that, morphological operations are applied to smooth the contour of cells and the binary images are obtained. Moreover, the morphology and colorimetry features are extracted and trained in BP neural network for identification and counting. The detection error between the proposed method and the manual counting method is no more than 5%. In Shenglang et al. (2008), the perimeter, area, shape factor, rectangularity, extension length and gray-scale of the object are input to the BP neural network. Then the neural network is activated by using Sigmoid function, which contains six hidden layers and one output layer. The counting system can analyze the sample in less than 10 minutes, whereas the classical manual counting method takes 48 hours.

In Ferrari et al. (2015, 2017); Tamiev et al. (2020), the convolutional neural network (CNN) is applied for bacterial colony counting. The example of the dataset is shown in Fig. 22. Then a horizontal flip is performed on the images to double the training dataset and three different artificial color distortions on RGB color space are applied. After that, another transformation is the conversion of the masked dataset in gray-scale color space and seven different values of spatial rescaling before cropping is performed. Then the images are enhanced through normalization concerning the segment orientation (Ferrari et al. (2015)) and contrast limited adaptive histogram equalization (Ferrari et al. (2017)). Finally, CNN is applied for classification and counting that contains five learned layers, four convolutional and one fully connected as shown in Fig. 23. During the training, the testing accuracy flattens after 15000 iterations. 50,000 iterations have taken approximately 3 hours on an Nvidia Titan Black GPU. The accuracy of 92.8% is obtained. After CNN classification, a watershed algorithm is applied for colony separation in Ferrari et al. (2017). The testing accuracy increases with the number of training iterations and flattens around 30,000 iterations. 50,000 iterations take approximately one hour on an Nvidia Titan X GPU. The accuracy of 92.1% is obtained after data augmentation. In Tamiev et al. (2020), a classification-type convolutional neural network (cCNN) is proposed for automatic bacteria classification and counting, and an efficient method for microscope image preprocessing is presented. First, the raw images are segmented with an adaptive binary thresholding method and images with individual cells or cell clusters are cropped. Then the images are trained using cCNN. The network’s output corresponds to the number of cells in given cell clusters and the individual outputs are then added to find the total cell count. The counting accuracy of 86% is obtained. The workflow is shown in Fig. 24. The result shows a 3.8X increase in processing speed by using an NVIDIA Quadro K620 GPU.

Fig. 22.

Example of dataset images representing a certain number of colonies, from 1 (a) to 6 (f), and two example of outliers (g) and (h) (in Ferrari et al. (2015) Fig. 2)

Fig. 23.

Convolutional Neural Network topology (in Ferrari et al. (2015) Fig. 3)

Fig. 24.

The work flow of image processing. a The image is binarized and annotated manually. b Null Bumper. c Blended. d Masked. e Neural network training. f The input of neural network (in Tamiev et al. (2020) Fig. 2)

Third-party tools

In Jung and Lee (2016), image analysis is used for real-time bacterial counting. First, the time-series high-resolution (HR) images of bacterial microcolonies are reconstructed using sub-pixel sweeping perspective microscopy (SPSM). Then the images are segmented, and the equivalent diameter and number of colonies in each time-lapse image are then calculated. The processed images are shown in Fig. 25.

Fig. 25.

Image processing. a Time-series high-resolution (HR) images. b Following reconstruction of the HR image (in Jung and Lee (2016) Fig. 3)

In Moller et al. (1995), Cellstat image analysis program is developed to determine the biovolume of bacteria. They present a method for simultaneous quantitative staining of RNA and DNA using the metachromatic dye AO and quantify the RNA and DNA. The automated image analysis is not biased by the operator, and it allows the analysis of a number of objects, ensuring good statistics. Choosing the right parameters for cell identification makes it possible to discriminate between single cells and clumps of cells. By using a different set of parameters for object recognition, it is possible to detect and measure the intensities of surface-associated microcolonies and single cells on the surface independently. The result of automatic identification of bacteria with Cellstat is shown in Fig. 26.

Fig. 26.

Automatic identification of bacteria with Cellstat. a The cell between 75 and 700 pixels. b The cell between 350 and 700 pixels. c Identification of single cells between 75 and 175 pixels. d Identification of micro colonies on the surface (in Moller et al. (1995) Fig. 1)

In David and Paul (1989), ‘Model 2000’ (Image Technology Corporation, Deer Park, New York) image analysis system is used for enumeration and sizing of bacteria, which can detect and enhance each individual cell at the same time. There is no statistical difference in cell counts made manually or by the image analysis system.

In Kildesø and Nielsen (1997), ‘Kontron Vidas Plus’ (Kontron Elektronik GmbH, Germany) image analyser system is used for airborne microorganisms counting. Gaussian filter is used to remove noises and the edge is derived using Laplace filter. This work has established a possibility of improving exposure assessment of airborne microorganisms through image processing instead of manual counting.

In Shopov et al. (2000), a program ‘Skidaway Tools’ (Skidaway Institute of Oceanography, 10 Ocean Science Circle, Savannah, USA) is developed based on Marr-Hildreth Gaussian-smoothed Laplacian edge-detection protocol that is proposed in Viles and Sieracki (1992), with added flat-fielding and edge-strength operators. The alpha-channel is applied in bacteria image segmentation, masking the background and providing a count of the attached bacteria cells.

In Gmür et al. (2000), ‘IBAS 2.0’ (Kontron Inc., Eching, West Germany) is used to process the images of dental bacteria. A gradient convolution filter is used to process images firstly, then, any white objects below or above an acceptable size range are excluded and the remaining spots are counted automatically after image binary. In Singleton et al. (2001), ‘IBAS 2.0’ is used for oral microbial quantification. Thresholdiinging is used for images binarization, and the edge-effect rule is used to eliminate the objects of the wrong size. Then the individual bacteria are segmented, and the white spot of images are counted. A close agreement between the automated system and the manual visual counts is observed.

In Nunan et al. (2001), ‘Zeiss KS300 Imaging System 3.0’ is used for bacteria image processing. The RGB images are decomposed into 3 channels that can be processed separately. Sigma smoothing and top-hat transform are used for edge detection and segmentation in green channel images that can detect all features in the bacteria size range. High pass filter and morphological opening are used to remove autofluorescent objects in red channel images. The top hat transform is used to distinguish bacteria from other objects by detecting the blue halos in blue channel images. The binary images are obtained based on the three-channel images above. The number of cells and other parameters such as area is measured. In Stoderegger and Herndl (2005), ‘Zeiss KS300’ is used to quantify the natural bacterial community. The binary images are obtained by adjusting the threshold level. Then the images are corrected by excluding or adding cells originally not detected by the channel settings. The individual cell area is determined, and the total area is calculated. Therefore, the number of cells can be obtained.

In Peña et al. (2002), ‘Image-Pro Plus’ image analysis software (Media Cybernetics, USA) is used for the quantification of bacteria. For aggregates, contour extraction is used for automatic image segmentation. The average equivalent diameter (AED) is used to characterize the size of aggregates, which is used for the automatic elimination of individual cells and debris. A High-Gauss filter is used for image enhancement for individual cells, and a Gaussian filter is used for noise reduction. The roundness value is calculated to select the objects that correspond to individual cells. The binary images based on the two methods above are used for bacteria counting and biovolume measurement.

In O’cleirigh et al. (2003), ‘Optimas 6.5’ (Media Cybernetics Inc., Silver Spring, MD) image processing system is used to quantify bacteria. Firstly, the low-frequency background noises are isolated and removed, then a combination method of binary erosions to point and dilations within image masks is used to separate cells. The parameters such as cell count and cell volume are measured by using the ‘Optimas’ image processing system. The result shows that the deviation of the experimentally measured density from the known density is 3.2%.

In Putman et al. (2005), ‘ProtoCOL’ (Version 4.04 from Synoptics Ltd., Cambridge, UK) is used to count the bacteria colonies. When the ‘ProtoCOL’ software is used to process the digital camera image, the count result is highly correlated with the true count but slightly less than the true count.

In Thiel and Blaut (2005), ‘KS400’ (Carl Zeiss Vision, Hallbergmoos, Germany) is used for automated enumeration of fluorescently labeled bacteria. First, the DAPI images are analyzed to detect single signals at a high spatial resolution. Then, a second analysis system is developed to detect signals with a low signal-to-noise ratio. After, the binary images are obtained by merging the resulting images above. Finally, the third step processes the Cy3 image and thus provides information on the signals that derive from target organisms. The logical ‘AND’ operation of the processed DAPI and Cy3 images ensures that only those signals are counted in both channels. The calculated correlation coefficient of 0.984 indicates that the manual and the automatic counting result are in agreement.

In Wang et al. (2007), a micro-colony auto counting system ‘MACS’ (Chuo Electric Works, Osaka, Japan) is used for bacteria colony counting. The ‘MACS’ has an automatic scanning stage and blue light emitting diode (LED) as a light source. Micro-colonies are captured using a CCD camera and analyzed using ‘Micro-colony V’ software (version 1.504; Chuo Electric Works). SYBR Green II is used to stain the bacteria images, and the green fluorescence is detected clearly. The stained images and counting results are shown in Fig. 27.

Fig. 27.

Fluorescence images of single cells and micro-colonies of bacteria. Bacteria in compost are stained with EtBr (a, c and e) and SYBR Green II (b, d and f). Single bacterial cells in compost suspension before incubation (a and b) and micro-colonies developed after incubation on LB medium (c and d) observed under blue excitation by epifluorescence microscopy. Micro-colonies are also observed using micro-colony auto counting system (e and f) (in Wang et al. (2007) Fig. 3). (Color figure online)

In Hua et al. (2009), ‘Davinci’ technology is used for bacteria counting and area calculating. Image enhancement and median filter are applied to remove noises and local binary fitting (LBF) is used for image segmentation. Finally, the connected region is detected as the number of bacteria and the area of the connected region is measured. The average error between the proposed method and the manual counting method is no more than 1.6%.

In Freitas et al. (2014), the automated enumeration software ‘SigmaScan Pro 5.0’ (Systat Software Inc) is used for the quantification of cells in the biofilm. The intensity thresholding is used for image segmentation. There are no significant differences found using the software thresholding and the manual counting ($\text{r}>0.05$), indicating that the Live/Dead staining is strongly discriminative between bacteria and background, and there is no significant fluorophore bleach effect that could impair the automatic counts.

In Song et al. (2018), ‘ImageJ’ software version 1.52 (NIH, Bethesda, MD, USA) is applied for automatic bacteria counting. First, the threshold is adjusted to enhance the contrast of the objects of interest. Second, the image is binarized to remove the noise by rendering micro-colony regions with clear boundaries as black and the surrounding background as white. After the binarization, filling-holes processing is conducted to ensure each closed region represents one intact micro-colony. Finally, micro-colony regions above the desired size are outlined, and the number of these regions is automatically counted by ‘ImageJ’ software.

Summary of image analysis based counting for bacteria

By reviewing the related work of image analysis for bacteria counting and referring to Table 1, we find that:

• Development trend The bacteria counting using image analysis approaches began in the 1980s and developed quickly in the 2010s. This development trend is due to the government and people attach importance to the bacteria problems in recent years, which play essential roles in the food industry and social hygiene. With the development of computer based image analysis technologies, more explorations and higher accuracies will be achieved in the future.

• Counting techniques The most frequently used pre-processing methods are the median filter and Gaussian filter, image segmentation methods are thresholding, distance transform and watershed, classifier algorithms are SVMs and ANNs.

Table 1.

Summary of image analysis based counting for bacteria

Related work	Microorganism type	Pre-processing	Segmentation	Classification	Evaluation
Pettipher and Rodrigues (1982)	Bacteria		Gray-level contrast
Masuko et al. (1991)	Bacteria		Thresholding
Moller et al. (1995)	Bacteria		Gray-level intensity
Massana et al. (1997)	Bacteria	Edge detection	Thresholding
Trujillo et al. (2001)	Bacteria	Flatten filter	Thresholding
Ogawa et al. (2003)	Bacteria	Color and square measuring	Thresholding
Yamaguchi et al. (2004)	Bacteria	Intensity expanding	Hue segmentation
Ogawa et al. (2005)	Bacteria		Multicolour segmentation		95% confidence intervals
Shenglang and Yongguang (2005)	Bacteria	Histogram equalization, convolutional filter and median filter	Thresholding
Niyazi et al. (2007)	Bacteria		Thresholding		95% accuracy
Gupta et al. (2012)	Bacteria	Adaptive median filter	Thresholding		95%-100% accuracy
Sethi and Yadav (2012)	Bacteria	Adaptive median filter and thinning	Thresholding		95%-100% accuracy
Kaur and Sethi (2012)	Bacteria	RGB adjusting and adaptive median filter	Morphological dilation for edge extraction
Chunhachart and Suksawat (2016)	Bacteria	Morphological operations for image enhancement	Thresholding		97.87% accuracy
Ishii et al. (1987)	Bacteria		PCA
Mukherjee et al. (1995)	Bacteria	Thresholding and distance transform	Region growing and connected component labelling
Pernthaler et al. (1997)	Bacteria		Top-hat and contrast retransformation
Blackburn et al. (1998)	Bacteria	Marr-Hildreth operator	Thresholding	ANN	95% accuracy
Schönholzer et al. (2002)	Bacteria	RGB processing	Local grey value maxima
Marotz et al. (2001)	Bacteria		Local adaptive threshold		0.968 correlation coefficient with manual count
Selinummi et al. (2005)	Bacteria		Marker-controlled watershed segmentation and global thresholding		99% accuracy
Zhang and Chen (2007)	Bacteria	Contrast-limited adaptive histogram equalization	Otsu thresholding and watershed
Zhang et al. (2008a)	Bacteria	Hypothesis testing for denoising	Otsu thresholding and frequency distribution segmentation
Chen and Zhang (2008)	Bacteria		Watershed		61% precision
Chen and Zhang (2009)	Bacteria		Watershed	SVM	80% accuracy
Shenglang et al. (2008)	Bacteria	Median filter and adaptive filter	Iterative thresholding	BP neural network
Hong et al. (2008)	Bacteria	Gray-scale weighted thresholding	Distance transform and watershed		98.2% accuracy
Yujie (2009)	Bacteria	Gray-scale weighted thresholding	Distance transform and watershed	SAGA and SVM	99.67% accuracy
Kan (2008)	Bacteria	Median filter	Distance transform and watershed		13.5% average relative error with manual count
Fang et al. (2008)	Bacteria	Median filter and contrast enhancement	Iterative thresholding		2.5% average relative error with manual count
Sotaquira et al. (2009)	Bacteria		Thresholding		96.3% accuracy
Ates and Gerek (2009)	Bacteria	Median filter	Watershed
Jun (2010)	Bacteria	Median filter	Adaptive thresholding and morphological operations	BP neural network
Mukti et al. (2010)	Bacteria		Color thresholding
Peitz and van Leeuwen (2010)	Bacteria	Gaussian filter	Otsu thresholding and erosion operation
Buzalewicz et al. (2010)	Bacteria	Fourier transform	Mellin transform		4.51 standard deviation with manual counting
Zhang et al. (2010)	Bacteria	Median filter and gray-level histogram equalization	Otsu thresholding	SVM	97% accuracy
Nayak et al. (2010)	Bacteria		HSI color segmentation and thresholding
Shen et al. (2010)	Bacteria	Median filter	Iterative local thresholding		97.5% accuracy
Clarke et al. (2010)	Bacteria	Gaussian filter	Adaptive thresholding		97% accuracy
Hongwei (2012)	Bacteria	Background marking and wavelet method	Otsu thresholding and square and circle filter	BP neural network	95% accuracy
Ferrari et al. (2015)	Bacteria		Thresholding	CNN	92.8% accuracy
Brugger et al. (2012)	Bacteria	Top-hat transform	Otsu thresholding, adaptive
thresholding, distance transform and watershed	Bayes classifier
Masschelein et al. (2012)	Bacteria		Distance transform and watershed	SVM
feng Hu (2013)	Bacteria		Adaptive thresholding, distance transform and erosion
Barbedo (2013)	Bacteria	Gaussian filter and Canny filter	Region growing and thresholding		99% accuracy
Ferrari et al. (2017)	Bacteria	Contrast limited adaptive histogram equalization	Watershed	CNN	92.1% accuracy
Yoon et al. (2015)	Bacteria	PCA and wavelength band selection	Thresholding		99% accuracy
Andreini et al. (2015)	Bacteria	Sobel operator and chromatic information selection	Mean shift segmentation and thresholding	SVM	99.2% accuracy
Andreini et al. (2016)	Bacteria	Morphological filter	Random Hough circle transform and thresholding	SVM	92.1% accuracy
Chiang et al. (2015)	Bacteria	PCA and Sobel masks	Otsu thresholding, watershed and distance transoform		96.63% accuracy
Martinez-Espinosa et al. (2016)	Bacteria	Edge detection	Distance transoform		0.994 correlation coefficient with manual count
Minoi et al. (2016)	Bacteria	Edge detection	Otsu thresholding, watershed and distance transform
Wong et al. (2016)	Bacteria	Contrast enhancement	Watershed and distance transform		90.3% accuracy
Choudhry (2016)	Bacteria	Sobel filter, Gaussian blur and holes filling	Thresholding
Alves and Cruvinel (2016)	Bacteria	Laplacian filter and circular Hough transform	Otsu thresholding		90% accuracy
Jung and Lee (2016)	Bacteria	Image reconstruction	Thresholding
Matić et al. (2016)	Bacteria	Gamma correction and Gaussian filter	Thresholding, Canny operator and Hough transform		97% precision and 82% recall
Siqueira and de Carvalho (2017)	Bacteria	Median filter	Thresholding, Canny operator and Hough transform		92.31% accuracy
Sánchez-Femat et al. (2016)	Bacteria	Histogram liner expansion	Otsu thresholding		98% accuracy
Maretić and Lacković (2017)	Bacteria	Gaussian filter, average filter and median filter	Thresholding
Austerjost et al. (2017)	Bacteria		Iterative thresholding and Hough transform		86.76 ± 9.76% accuracy
Payasi and Patidar (2017)	Bacteria	HSI color space processing	Thresholding		90% accuracy
Boukouvalas et al. (2018)	Bacteria	Median filter and Hough transform	Gaussian adaptive thresholding
Boukouvalas et al. (2019)	Bacteria	Multidirectional Sobel operator	Otsu thresholding, circularity filter and inertia filter		99.8% accuracy for high definition dataset and 95.9% for low definition dataset
Zhu et al. (2018)	Bacteria	Nonliner gray transform	Image subtraction and watershed		99.8% accuracy
Tamiev et al. (2020)	Bacteria		Adaptive thresholding	Classification-type convolutional neural network
David and Paul (1989)	Bacteria	Image enhancement	Model 2000 (Image Technology Corporation, Deer Park, New York) image analysis system
Kildesø and Nielsen (1997)	Bacteria	Gaussian filter and Laplacian filter	Kontron Vidas Plus (Kontron Elektronik GmbH, Germany) image analyser system
Shopov et al. (2000)	Bacteria	Marr-Hildreth Gaussian-smoothed Laplacian edge-detection protocol, flat-fielding and edge-strength operators	Skidaway Tools (Skidaway Institute of Oceanography, 10 Ocean Science Circle, Savannah, USA)
Gmür et al. (2000)	Bacteria	Gradient convolution filter	IBAS 2.0 (Kontron Inc., Eching, West Germany)
Singleton et al. (2001)	Bacteria	Thresholding and edge-effect rule	IBAS 2.0 (Kontron Inc., Eching, West Germany)
Nunan et al. (2001)	Bacteria	Sigma smoothing, top hat transform, high pass filter and morphological opening	Zeiss KS300 Imaging System 3.0
Stoderegger and Herndl (2005)	Bacteria	Thresholding	Zeiss KS300 Imaging System 3.0
Peña et al. (2002)	Bacteria	Contour extraction, average equivalent diameter, high-Gauss filter and Gaussian filter	Image-Pro Plus image analysis software (Media Cybernetics, USA)
O’cleirigh et al. (2003)	Bacteria	Morphological erosions and dilations	Optimas 6.5 (Media Cybernetics Inc., Silver Spring, MD) image processing system		96.8% accuracy
Putman et al. (2005)	Bacteria		ProtoCOL (Version 4.04 from Synoptics Ltd., Cambridge, UK)
Thiel and Blaut (2005)	Bacteria		KS400 (Carl Zeiss Vision, Hallbergmoos, Germany)		0.984 correlation coefficient with manual counting
Wang et al. (2007)	Bacteria		Micro-colony auto counting system MACS (Chuo Electric Works, Osaka, Japan)
Hua et al. (2009)	Bacteria	Image enhancement and median filter	Davinci technology and local binary fitting		98.4% accuracy
Freitas et al. (2014)	Bacteria		SigmaScan Pro 5.0 (Systat Software Inc) and thresholding
Song et al. (2018)	Bacteria	Thresholding and filling-holes	ImageJ software version 1.52 (NIH, Bethesda, MD, USA)

Other microorganism counting methods

Classic counting methods

Counting methods based on image enhancement

In Zalewski and Buchholz (1996), color filtering and contour enhancement are used to separate the yeast cells and compare the automatically detected cell concentration and the traditional cell counting using a counting chamber.

In Barbedo (2012b), the histogram equalization is applied for microorganism counting. First, the images are converted to gray-scale images, and a median smoothing filter is applied for noise removal. Then, the histogram equalization is used for image enhancement, and the enhanced images are submitted to top-hat morphological filtering. Finally, the images are converted to binary images, and the connected regions are counted. The accuracy in correctly identifying the objects is more than 90% and the overall deviation is 8%.

In Dazzo and Gross (2013), the center for microbial ecology image analysis system (CMEIAS) is developed for understanding microbial ecology at single-cell resolution and spatial scales relevant to the individual microbes and their ecological niches in situ. An optimization method of quadrat size is proposed to reduce the complexity of calculating. Four types of quadrat sizes are proposed, that is $4\times 4$, $6\times 6$, $8\times 8$ and $10\times 10$ grid. After that, the construction of 2-dimensional scatter plots based on Cartesian coordinates of object centroids can solve serious edge effects when the best possible grid-lattice on the landscape index image still significantly overlaps foreground objects. The index image of the dot map representation derived from the original biofilm landscape image with the optimized grid raster overlay is shown in Fig. 28.

Fig. 28.

a The original image. b All of the microoragnisms in dot map(in Dazzo and Gross (2013) Fig. 5)

Counting methods based on thresholding

In Costello and Monk (1985), Brown et al. (1989), Dias et al. (2003), Shijing et al. (2012), Mazzei et al. (2014), Cross and Kenerley (2004), Packer and Thomas (1990), Tucker et al. (1992), Zeder et al. (2010), thresholding is applied for fungi (Cross and Kenerley (2004), Packer and Thomas (1990)), mycelia (Tucker et al. (1992)), yeast (Costello and Monk (1985)) and protozoan (Brown et al. (1989), Dias et al. (2003)) counting. In Costello and Monk (1985), the yeast strains are counted by adjusting the gray-level of the images with the presence of high cell destinies. In Brown et al. (1989), Dias et al. (2003), the minimum size of picoplankton and protozoan are set as the filters to remove the noises. Finally, the connected regions are counted as the number of chlorella in Shijing et al. (2012), and the flood fill algorithm is applied for labeling and tracking in Mazzei et al. (2014). In Zeder et al. (2010), the gray-level intensities of pixels determine the threshold. ‘Fixels’ are defined as the primary component of filaments that are can cover small parts of a filament. Once the orientation of the fixel is established, a rectangular field with the same orientation is moved along a line perpendicular to the orientation of the fixel, and the precise placement of the fixel is locally optimized by maximizing the sum of the gray-level intensities in the rectangle. Finally, the filaments are annotated and the lengths of filaments are measured. The accuracy of the method is more than 85%. In Hamid et al. (2013), thresholding is applied for pus cell counting and feature extraction. The objects that are less than 20 pixels are eliminated after thresholding. Then, the morphological and shape features are extracted for criteria selection. Finally, the single cells and overlap cells are classified and counted, respectively. The identified pus cells are marked on the initial image that is shown in Fig. 29. It is shown that the reliability of the proposed system is above 80% from the validation results.

Fig. 29.

The original image of pus cells (in Hamid et al. (2013) Fig. 8)

In Jones et al. (1992), local threshold and morphological transformation are used to calculate spore numbers accurately. Red and blue signals are sampled for pixel segmentation. In all cases, the counts are elevated compared with those obtained by the manual method.

In Robinson et al. (1998), Xianjiu et al. (2012), Kim and Cho (2013), Saur et al. (2014), Otsu thresholding is applied for the counting of microorganisms. First, the hue, lightness and saturation (HLS) model is applied to transform RGB image into the gray image. Then, the global smoothing (Robinson et al. (1998)) and wavelet shrinkage method (Xianjiu et al. (2012)) are used for noise removal. The wavelet transform is applied in the signal, and the wavelet coefficients are shrunk by thresholding. After that, the inverse wavelet transform is applied. Then, in Robinson et al. (1998), the binary image consisting of the regional maxima is used as the marker image for the watershed algorithm, and the Sobel filter is used to detect the edge. Finally, Otsu thresholding is used for the accurate estimation of each cell colony area. Total cell number is achieved following identification of cells by application of the shape-independent watershed algorithm. In Xianjiu et al. (2012), the image dilation is used for image enhancement, and the combination method of Otsu thresholding and the morphological opening is used for image segmentation. Finally, the number of algae is counted based on eight neighborhood regions. The mean accuracy of the proposed method is more than 94% comparing with the manual counting method. In Kim and Cho (2013), the morphological features are extracted after Otsu thresholding for classification and the number of the object is used for hatching rate measurement. It is shown that the maximum difference is about 19.7%, and the average root-mean squared difference is about 10.9% as the difference between the results using automatic counting (this study) and manual counting is compared. The result of object detection is shown in Fig. 30. In Saur et al. (2014), Otsu thresholding is used to quantify moving predators in biofilm. The Otsu thresholding is applied for image binarization, and the noises are removed by filtering. Then the global displacement response and the number of moving objects corresponding to the number of detected individual objects on the processed image are calculated. The calculating results of the two parameters above are shown in Fig. 31.

Fig. 30.

The shape features and boundary tracks of the detected objects (in Kim and Cho (2013) Fig. 3)

Fig. 31.

The counting result. a The correlations between the manual counting result and the automatically calculated global displacement response. b The number of objects per $\text{c}{m}^2$ (in Saur et al. (2014) Fig. 3)

In Song et al. (2006), the HSI thresholding is used to count the number of algae. A median filter is used to remove noises and reduce the fuzzy edges. Hue-Saturation-Intensity (HSI) threshold is used for image segmentation, and the flood fill method is used to fill the connected region. Area threshold is used to remove debris, and then the images are thinned for central point searching. Finally, the number of algae cells is counted. The accuracy of the method for identifying and counting is more than 90%.

In Zhonglei and Peng (2012), the histogram thresholding is applied for automatic fungi counting. First, the median filter and linear gray-scale transformation are applied to reduce uneven illumination and noise. Then, the image is segmented based on histogram thresholding. After that, an adaptive smooth filter is used for image enhancement, and morphological operations are applied for smoothing and hole filling. Finally, the connected regions are labeled and counted as the number of fungi to be counted. The average relative error between the proposed method and the manual counting method is 2.56%.

In Sharma (2015), two methods based on image processing are proposed for microorganisms counting in medical. The first one is based on object recognition, which means the microorganisms’ shape is considered to find out the total number of microbes in the image sample. The histogram equalization is applied for image enhancement that can help to separate the objects from the background. Then the circular hough transformation technique is used to determine the circular objects in the image. The second one is based on thresholding. First, the image is enhanced with histogram equalization and converted to a binary image. Then the Moore neighbor tracing algorithm is applied to detect objects that have close boundaries. Finally, the combination method of thresholding, object recognition and morphological operation is used for counting, and the accuracy of 93% is obtained. The segmentation result is shown in Fig. 32.

Fig. 32.

The results after thresholding and morphological operations. a The original image. b Image after thresholding and opening operation. c The identified microorganisms (in Sharma (2015) Fig. 8)

In Fang et al. (2019), a multi-threshold image counting method based on improved particle swarm optimization (PSO) is proposed for automatic microbial counting. The two-dimensional maximum entropy algorithm is extended to design the objective function using exponential entropy and an improved PSO algorithm to acquire its maximum value and the best image segmentation effect. Furthermore, the breadth-first search (BFS) algorithm is applied to complete the microorganism marker and counting in the segmented images. Finally, the number of image target segmentation is determined according to the histogram peak searching method. The comparisons of target segmentation results are shown in Fig. 33.

Fig. 33.

Comparisons of target segmentation effects (in Fang et al. (2019) Fig. 1)

Counting methods based on edge detection

In Viles and Sieracki (1992), Sieracki et al. (1995), the Marr-Hildreth method is used for edge detection and image segmentation of picoplankton (in Viles and Sieracki (1992)) and heterotrophic bacteria (in Sieracki et al. (1995)). Combinations of edge strength and minimum and maximum cell sizes allow the user to count specific cell populations. The images are captured using a charge-coupled device (CCD) imaging system, and the result shows an accurate performance.

In Kocak et al. (1999), the Snake model is used for low-level interaction, which is a deformable parametric curve with its corresponding energy function, the closed curve with the minimum energy is the target contour. The edge image is produced by subtracting the eroded image from a dilated image. An extermination algorithm is used to examine the image gradient on both sides of the snakelet by computing the directional derivative orthogonal to each of its nodes. The Snake model is used for plankton counting in this research, and the accuracy of 94.12% is achieved. The example of segmentation is shown in Fig. 34.

Fig. 34.

Example of steps in segmentation (In Kocak et al. (1999) Fig. 12)

In Barber et al. (2000), the Sobel operator is used to find the edges, and the binary images are obtained using thresholding. A compact Hough transform is used to highlight the centers of circular objects. The local area can be processed to determine a colony boundary, and so the colony area and the colony number can be calculated. An example of colony boundary determination is shown in Fig. 35.

Fig. 35.

Example of steps in segmentation (In Barber et al. (2000) Fig. 5)

In Pernthaler et al. (2003), the gradient transformation is applied for microorganism edge detection. First, a mean background gray level is determined for remapping the image gray values to the total gray value range. Edge detection is performed by using gradient transformation, and a neighborhood median filter is applied to smooth the resulting image. The stained plankton cells are calculated and counted.

In Tsechpenakis et al. (2008), the probabilities are applied for image segmentation. First, a nonlinear morphological filter, that is, an alternating sequential filter (ASF) is used to preserve the line-type image structures in predefined orientations while filtering random noise. Then the region of animals is segmented based on the probabilities. The isolated pixels or small groups of pixels with probabilities higher than the threshold are eliminated. After that, a nonrigid is used to recover a global transformation that brings the pose of a source shape as close as possible to that of a target shape. The shape estimation of animals is based on the maximum likelihood (ML) approach, and the animal region is extracted based on a probability map. Finally, the number of animals is calculated and compared with the manual counting result.

In Barbedo (2012c), a unified framework for counting agriculture microorganisms is proposed. There are five methods listed for object delineating. In the first method, the Laplacian of Gaussian method is applied for edge detection, then the inner regions are filled, and all connected objects are identified. The only difference between the second method and the first one is that the Canny method is used to detect the edges. In the third method, three different thresholds are applied for image binarization. In the fourth method, contrast is modified by the technique of histogram equalization. In the last method, region growing is applied for segmentation. After object delineating, a decision tree is used for classification. Then the contrast limited adaptive histogram equalization is applied, and the image is morphologically opened using as kernel a disk with a radius of 1% of the image width. Finally, the estimate for the number of objects is calculated based on the number of local maxima.

Machine learning and deep learning counting methods

In Shabtai et al. (1996), Embleton et al. (2003), a neural network is applied for fungus (Shabtai et al. (1996)) and phytoplankton (Embleton et al. (2003)) counting. In Embleton et al. (2003), the gray-level is used to separate the regions of interest, and then a median filter is used to smooth images. A skeletonize operator is used to separating the filaments and objects. Then the final binary image is used as a mask to take measurements from the original image. Each region in the binary image is given an identifying number, and the size, shape, color and grey level distribution are measured for that region. A neural network is used for the classification and counting of phytoplankton. The automated imaging system takes 75 images for each sample in seven minutes, and the image processing and classification take thirty to forty minutes.

In Benyon et al. (1999), seven basic features and 17 more complex features are extracted from fungal spores for image analysis. Linear and quadratic discriminant analysis are used for image classification, and then the number of every species of spores is counted based on the results of classification. Genus comparisons using only seven basic features resulted in 98% accuracy.

In Motta et al. (2001), Akiba and Kakui (1997), PCA is applied for protozoa (Motta et al. (2001)) and plankton (Akiba and Kakui (1997)) counting and classification. In Motta et al. (2001), the histogram local equalization is used to enhance the contours of protozoa images, the opening operation and closing operation are used to remove halo. Euclidian Distance Map is used for semi-automated segmentation. A series of erosion and reconstruction are used to eliminate the flocs of the protozoa silhouette. Finally, PCA is used to classify different protozoa, and the number of the various species of protozoa is counted. The main steps of segmentation are shown in Fig. 36, (a) shows the initial image of protozoa, (b) shows the contour enhancement by histogram local equalization, (c) shows the background suppression by opening (2 iterations) and closing (55 iterations) to remove the halo, (d) shows the semi-automated segmentation based on the Euclidian Distance Map, (e) shows part of the flocs is eliminated by a border-killing routine, (f) shows the hole-filling of the silhouette and semi-automated segmentation based on the Euclidian Distance Map, (g) shows the elimination of flocs by a series of erosion and reconstruction of the protozoa silhouette, (h) shows the localization of flagella and stalk.

Fig. 36.

Main steps of segmentation (In Motta et al. (2001) Fig. 1)

In Grosjean et al. (2004), the random forest and discriminant vector forest are applied for zooplankton image processing. A threshold is used to eliminate the background and enhance the contrast, then the objects are detected, contoured, and labeled by the image analysis system. The combination method of random forest and discriminant vector forest is used for classification, and the number of each species is counted. The result of object detection is shown in Fig. 37.

Fig. 37.

The objects are detected, contoured, and labelled by the image analysis (In Grosjean et al. (2004) Fig. 3)

In Rong et al. (2006), BP neural network and image processing are used for classification and counting of zooplankton. Otsu thresholding is used for initial image segmentation, and region growing is used to fill holes. The noises of debris are removed by detecting the particles that are smaller than the set threshold area. Then the features such as gray level co-ocurence matrices (GLCM) and some shape features are measured and used for classification and counting by using a back propagation (BP) neural network. There are 5 nodes in the input layer, 20 nodes in the hidden layer, and 1 node in the output layer, and the Sigmoid function is applied for activation.

In Albaradei et al. (2020), deep transfer learning is applied for automatic pluripotent stem cell colony counting. First, the RGB image is converted to a binary image by using thresholding. Then, some augmentation techniques are applied to expand the training dataset. The augmentation techniques include color jitter to randomly alter brightness, contrast, saturation, and hue of each image, horizontal/vertical flip, and random rotation. Moreover, the trained SRNetDL model is applied for training. The first 10 layers are frozen to remain the pre-trained network. Then the last 6 layers are fine-tuned. The stochastic gradient descent is applied for optimization. Finally, the number of cell colonies is counted. The overview of transfer learning is shown in Fig. 38.

Fig. 38.

Processing of transfer learning for counting (in Albaradei et al. (2020) Fig. 2)

Third-party tools

In Ogawa et al. (2012), a time-lapse shadow image analysis system is designed for microbial colony counting. First, an agar plate containing many clusters of microbial colonies is trans-illuminated to project their 2-dimensional (2D) shadow images on a color CCD camera. Then the 2D shadow images of every cluster distributed within a 3-mm thick agar layer are captured in focus simultaneously through a multiple focusing system and then converted to 3-dimensional (3D) shadow images. It is possible to determine whether each cluster comprised single or multiple colonies by time-lapse analysis of the 3D shadow images. Finally, the recognized colonies are counted, and the result is compared with the manual counting method, and an excellent value of correlation efficiency is obtained (r = 0.999). The colony detection method is shown in Fig. 39.

Fig. 39.

Detection of 2 overlapping colonies of E.coli. a 2D shadow image of agar plate. The arrow points to the cluster of overlapping colonies. b Magnified image. c 3D shadow image. d 2 colonies recognized in the cluster (in Ogawa et al. (2012) Fig. 6)

In Rolke and Lenz (1984), ‘Quantimet 720’ image analysis system (Leica Cambridge Ltd., Cambridge, United Kingdom) is used to detect the zooplankton, and then the total number of objects is measured. The detector automatically selects the mean grey-level between image and background within the present range to ensure the optimal detection of the image contours. ‘Quantimet 570’ is developed in 1990 that is used in Bloem et al. (1995), the images are sharpened with maxima and minimum filter, and then all local maxima values are detected to determine the number of particles. A 5 by 5 convolution filter is used to remove noises, and a skeleton operation is used to separate the particles precisely. The mean differences between the visual and automated method are not significantly different from zero. The first method in Grivet et al. (1999) is the usage of the ‘Quantimet 570’ image analysis system for scanning and counting the adherent microorganisms. The second method shows the use of thresholding and image skeleton for enumeration. The correlation between the two enumeration methods is highly significant. The example of the second method for counting is shown in Fig. 40.

Fig. 40.

The processing of the actinomyces image. a Original image. b The binary image after a two-step thresholding. c The skeleton selection. d The whole skeleton with branches. e The whole pruned skeleton. f The triple points and end points on the skeletons (in Grivet et al. (1999) Fig. 3)

In Sieracki et al. (1985), ‘Artek 810’ image analyzer (Artek Systems Corp., Farmingdale, N.Y.) is used to detect, count and size the picoplankton. The threshold destiny level of detection can be set at 1 to 256 gray-level. Comparisons between visual and image analyzed counts show that none of the mean counts are significantly different at the 95% significance level by the paired t test.

In Estep and MacIntyre (1989), the ‘Zeus’ image analysis system (Institute of Marine Research, Bergen) is used for algae counting, sizing and identification. Images are enhanced, smoothed, shadowed, and then ‘Zeus’ system is used for automated counting. Comparison counts show no significant difference between the manual method and automatic identification and counting with the image-analysis system.

In Wright et al. (1991), a constant threshold is used to segment the images instead of an adaptive threshold, and then the Sobel operator is applied to detect the edge of micro plants and animals. Finally, ‘ImageMeasure 5100’ (Microscience, Div., Phoenix Trade, Inc. Seattle, Washington) is used for counting. The developed script reduces the time required to count and measure marine fouling tube worms by at least one order of magnitude over manual counts, with an error of five percent or less.

In Corkidi et al. (1998), a commercial program ‘IMAGENIA 2000’ (Biocom, Les Ulis, France) is used for image processing and object counting based on the multi-level threshold, and the total number of bright spots over the dark background is counted. The confluent and various sizes image analysis method (COVASIAM) is proposed, which estimates an average of 95.47% ($\sigma$ = 8.55%) of the manually counted colonies, while an automated method based on a single-threshold segmentation procedure estimates an average of 76% ($\sigma$ = 16.27%) of the manually counted colonies. Fig. 41h shows the segmentation result of Fig. 41g, and i shows the superimposed images of Fig. 41g and (h). COVASIAM gave 135 CFU, representing 97.8% of the manual counts.

Fig. 41.

Detection of colony. a The image is digitized and filtered through a binary mask. b Enhanced image. c Resulting image after adding images at T1 and T2 thresholds. d Colony segmentation at a single threshold level of data. e Colony segmentation by using COVASIAM. f Overlap of Fig. 41a and e. g Digitized image of colony in various size. h Colony segmentation result. i Overlap of Fig. 41g and h. (In Corkidi et al. (1998) Fig. 2)

In Rodenacker et al. (2001), a program ‘IDL’ (Research System Inc., Boulder, USA) is used for image analysis. Thresholding is used to segment the images. The opening operation and closing operation are used to clean the masks and fill holes. A neural network is designed to classify the microorganisms based on shape features that can help count each microorganism species in water. The ‘IDL’ is also used for identification and quantification of phytoplankton in Rodenacker et al. (2002), and the threshold is used for image segmentation. The segmentation result is shown in Fig. 42. The morphological and some intensity features are used for identification, and a neural network is designed for classification, then the number of each species is counted.

Fig. 42.

Segmentation of one image, central field with masks, right field with marked featured objects (In Rodenacker et al. (2002) Fig. 2)

In Sándor et al. (2001), ‘Quanitmet 570’ computer system (Leica, Cambridge, United Kingdom) is used for image processing. The proportion of the clumps is determined as the mean value of their projected areas, and the mean total hyphal length and the mean number of tips are determined for the freely dispersed mycelia.

In Nishimura et al. (2006), an automatic cell counting system, ‘Bioplorer’ (BP) (Matsushita Ecology Systems Co. Ltd, Kasugai, Aichi-ken, Japan), is used for the enumeration of yeast cells. BP system is used to exclude yeast cells measuring 5 $\mu \text{m}$ in diameter from the count when cultured yeast. The intensity is used for image segmentation and bright points are counted. In Nishimura et al. (2008), BP is used for the quantification of eukaryotic and prokaryotic cells. The cells are stained and captured using a CCD camera that can capture photons emitted from bacteria or yeast cells. Bright points are visualized on display and enumerated automatically. The threshold brightness value is optimized for each set of measurements by using BP.

In Eickhorst and Tippkötter (2008), an image analysis software, ‘AnalySIS’ (Soft Imaging) is used for soil microorganisms counting and detection. First, the images are optimized, such as the contrast of enhancement and gradation. Then the threshold value is set for the color of fluorescent probes. After that, the parameters are set to detect the microorganism pixels and the cell number is automated counted. The ratio of automated to manual counting is 97.7% (± 1.0) for bacteria and 92.2% (± 2.4) for archaea in the investigated paddy soils.

In Ghită et al. (2013), ‘ImageJ’ is applied for analysis of cyanobacteria from the marine sample. First, the background is separated from the objects based on the intra-class variance threshold method. Then the mathematical morphology operations are used to remove noises produced by specks of staining color in the image. Finally, the clustered objects are separated and counted. The same image is then analyzed with the program ‘CellC’ to count the cells in the filament of cyanobacteria. The result of the proposed method is shown in Fig. 43. In Stolze et al. (2019), ‘ImageJ’ is applied for yeast colony counting based on automatic image analysis. First, the RGB image is converted to an 8-bit image, and the thresholding value is adjusted for image binarization. After that, the watershed is applied for splitting merging colonies, and finally, the number of colonies is counted. The proposed method is shown in Fig. 44.

Fig. 43.

a Digital image of cyanobacteria. b Delimitation A panel edges using Image J software. c The total number of cells in A panel using CellC software (In Ghită et al. (2013) Fig. 9)

Fig. 44.

ImageJ automated cell analysis of a Petrifilm image. a Petrifilm colony forming area outlined with the oval ROI tool. b The area outside the ROI is cleared. c Image is converted to 8-bit. d Threshold is set to highlight colonies as black particles. e Merging colonies split by a single pixel line via the Watershed tool. f Particles included in the total count, highlighted and numbered in an overlay on the image (In Stolze et al. (2019) Fig. 1)

In Bennke et al. (2016), ‘Automated Cell Measuring and Enumeration tool 2.0’ program is applied for the enumeration of the microbial cell. First, the sample is stained using DAPI and captured using a CCD camera. Then the program is applied for cell determination and enumeration. The parameter values like object area, circularity, mean gray value and signal-to-background ratio are measured. Automated enumeration result is highly correlated with manual counts ($r^2>0.9$).

Summary of image analysis based counting for other microorganisms

By reviewing the related work of image analysis for other microorganisms counting and from Table 2, we find that:

• Development trend The counting for other microorganisms using image analysis approaches began in the 1980s and developed in the 2000s. By comparing with the related research on bacteria, the development speed of research about other microorganisms counting is relatively slow, and the research is relatively limited. There are two main reasons to cause this situation, firstly, the structures of bacteria are relatively simple, and most of them are circular, which are more visualized and accessible to evaluated for segmentation results. By contrast, some other microorganisms, such as alga and fungi, are relatively complex in structure and with plenty of hyphae, challenging to be segmented precisely. Secondly, microorganism counting systems are designed but not for one specific type. A dataset is necessary when evaluating systems’ performances, but the number of bacteria datasets is relatively abundant, so the researchers tend to test their systems with bacteria datasets.

• Counting techniques The most frequently used pre-processing methods are the medial filter and Gaussian filter, image segmentation methods are thresholding and Otsu thresholding, classifier algorithms are PCA and neural networks.

Table 2.

Summary of image analysis based counting for other microorganisms

Related work	Microorganism type	Pre-processing	Segmentation	Classification	Evaluation
Costello and Monk (1985)	Yeast		Gray-level adjusting
Brown et al. (1989)	Picoplankton	Minimize size filter	Gray-level adjusting		7 times faster than manual counting
Dias et al. (2003)	Protozoan	Wiener filter	Thresholding and size filter
Cross and Kenerley (2004)	Fungi	Square root transformation	Line detection, thinning and thresholding
Shijing et al. (2012)	Chlorella		Thresholding and connected region detection
Mazzei et al. (2014)	Bioluminescence organisms		Thresholding and flood filling		92.24% accuracy
Packer and Thomas (1990)	Fungi	Circularity test	Thresholding and skeletonization
Tucker et al. (1992)	Fungi	Circularity test	Thresholding and skeletonization
Viles and Sieracki (1992)	Picoplankton		Edge strength filter, minimum and maximum cell size filter and Marr-Hildeth method
Sieracki et al. (1995)	Heterotrophic bacteria		Edge strength filter, minimum and maximum cell size filter and Marr-Hildeth method
Jones et al. (1992)	Spores	RGB processing	Local thresholding and morphological transform
Zalewski and Buchholz (1996)	Yeast cells	Color filtering and contour enhancement	Thresholding
Shabtai et al. (1996)	Fungi		Self-organizing multilayer neural network
Akiba and Kakui (1997)	Plankton	Local auto-correlational masks	PCA	Auto-correlational masks and discrimination analysis	90% accuracy
Robinson et al. (1998)	Cell	Global smoothing and Sobel operator	Otsu thresholding and watershed
Kocak et al. (1999)	Plankton	Morphological erosion and dilation	Snake model		94.12% accuracy
Shabtai et al. (1996)	Fungi		Self-organizing multilayer neural network
Akiba and Kakui (1997)	Plankton	Local auto-correlational masks	PCA	Auto-correlational masks and discrimination analysis	90% accuracy
Robinson et al. (1998)	Cell	Global smoothing and Sobel operator	Otsu thresholding and watershed
Kocak et al. (1999)	Plankton	Morphological erosion and dilation	Snake model		94.12% accuracy
Benyon et al. (1999)	Fungi	Feature extraction		Linear and quadratic discriminant analysis	98% accuracy
Barber et al. (2000)	Mammalian cellular	Sobel operator and Hough transform	Thresholding
Motta et al. (2001)	Protozoa	Histogram local equalization and morphological operations	Euclidian distance map	PCA
Embleton et al. (2003)	Phytoplankton	Median filter	Gray level and skeletonize operator	Neural network
Pernthaler et al. (2003)	Plankton	Gradient transformation and neighbourhood median filter	Thresholding
Grosjean et al. (2004)	Zooplankton		Thresholding	Random forest and discriminant vector forest
Rong et al. (2006)	Zooplankton		Otsu thresholding and region growing	BP neural network
Song et al. (2006)	Alga	Median filter	Hue-Saturation-Intensity thresholding, flood fill method and area thresholding		90% accuracy
Tsechpenakis et al. (2008)	Animal	Alternating sequential filter	Nonrigid local registration and thresholding	Maximum likelihood
Zeder et al. (2010)	Cyanobacteria		Thresholding and gray-level intensities maximizing		85% accuracy
Ogawa et al. (2012)	Colony		Time-lapse shadow approach		0.999 correlation efficiency with manual counting
Xianjiu et al. (2012)	Alga	Wavelet shrinkage image dilation	Otsu thresholding and morphological opening		94% accuracy
Zhonglei and Peng (2012)	Fungi	Median filter, linear gray-scale transformation and adaptive smooth filter	Histogram thresholding		97.44% accuracy
Barbedo (2012b)	Colony	Median filter, histogram equalization and top-hat filter	Thresholding		90% accuracy
Barbedo (2012c)	Agriculture microorganism	Laplacian filter, Canny filter and histogram equalization	Thresholding and region growing	Decision tree
Kim and Cho (2013)	Cysts	Morphological operation	Otsu thresholding		10.9% average root-mean squared difference
Hamid et al. (2013)	Pus cell	Image enhancement	Thresholding		80% reliability
Dazzo and Gross (2013)	Colony		Quadrat size segmentation
Saur et al. (2014)	Moving predator	Thresholding	Global displacement response
Sharma (2015)	Medical microorganism	Histogram equalization and circular hough transformation	Thresholding and Moore neighbour tracing		93% accuracy
Fang et al. (2019)	Microorganism	Histogram peak searching	Particle swarm optimization, breadth-first search and exponential entropy
Albaradei et al. (2020)	Stem cell	Augmentation techniques include color jitter to randomly alter brightness, contrast, saturation, and hue of each image, horizontal/vertical flip, and random rotation.	Thresholding	SRNetDL
Rolke and Lenz (1984)	Zooplankton	Mean grey-level selection	Quantimet 720 image analysis system (Leica Cambridge Ltd., Cambridge, United Kingdom)
Bloem et al. (1995)	Microorganism	Image sharpen and convolutional filter	Quantimet 570
Grivet et al. (1999)	Microorganism		Quantimet 570, thresholding and image skeleton
Sieracki et al. (1985)	Picoplankton	Thresholding	Artek 810 image analyzer (Artek Systems Corp., Farmingdale, N.Y.)
Estep and MacIntyre (1989)	Alga		Zeus image analysis system (Institute of Marine Research, Bergen)
Wright et al. (1991)	Micro plant and animal	Sobel operator and thresholding	ImageMeasure 5100 (Microscience, Div., Phoenix Trade, Inc. Seattle, Washington)
Corkidi et al. (1998)	Microorganisms		IMAGENIA 2000 (Biocom, Les Ulis, France) and multilevel thresholding		95.47% accuracy
Rodenacker et al. (2001)	Microorganisms in water	Morphological operation and hole filling	IDL (Research System Inc., Boulder, USA) and thresholding	Neural network
Rodenacker et al. (2002)	Phytoplankton		IDL (Research System Inc., Boulder, USA) and thresholding	Neural network
Sándor et al. (2001)	Fungi		Quanitmet 570 computer system (Leica, Cambridge, United Kingdom)
Nishimura et al. (2006)	Yeast		Bioplorer (BP) (Matsushita Ecology Systems Co. Ltd, Kasugai, Aichi-ken, Japan) and intensity
Nishimura et al. (2008)	Eukaryotic and prokaryotic cell		Bioplorer (BP) (Matsushita Ecology Systems Co. Ltd, Kasugai, Aichi-ken, Japan) and thresholding
Eickhorst and Tippkötter (2008)	Microorganisms in soil	Contrast enhancement and gradation	AnalySIS (Soft Imaging) and thresholding		97.7% accuracy for bacteria and 92.2% for archaea
Ghită et al. (2013)	Cyanobacteria	Morphological operation	ImageJ
Stolze et al. (2019)	Yeast	Thresholding	ImageJ and watershed
Bennke et al. (2016)	Cell		Automated Cell Measuring and Enumeration tool 2.0

Analysis of image processing based counting methods

The image processing methods based microorganism counting are summarized from Sects. 3 to 4. It can be seen that the most effective approaches for microorganism image counting are image pre-processing, image segmentation, image classification, connected region detection, and feature extraction. In order to find out the reasons why they are widely used and reveal the potential future direction, in this section, the properties of these methods with their application domains are analyzed and summarized. In order to illustrate the correlation between methods and their applications, some representative works are selected as examples.

Image pre-processing methods

Because many kinds of microorganisms are colorless, staining methods are necessary to apply before image capture. Different staining methods lead to different color images, so the color feature is not appropriate for automatic microorganism counting. Moreover, due to the illumination and image noise’s inhomogeneity, pre-processing methods should be applied to solve the problems and prepare for image segmentation.

Firstly, to reduce the effect of different colors for image segmentation, the RGB images are usually converted to gray-scale images by adjusting the proportions of red, green, and blue channels. The RGB can also be converted to HSI (Hue-Saturation-Intensity) color space to assist the colony boundaries detection, such as the works in Song et al. (2006), Nayak et al. (2010), Payasi and Patidar (2017). HSI color space can adjust the Intensity but does not change the color type of the original image when processing colored images. Furthermore, it can ultimately reflect the primary attribute of color perception and corresponds to the result of color perception, which is helpful for the following segmentation.

Secondly, the uneven illumination can result in shading and a nonuniform background, which can usually be corrected using background subtraction, linear gray-scale transformation and low pass filtering, such as the works in Choudhry (2016) and Zhonglei and Peng (2012) (see Fig. 45).

Fig. 45.

Contrast of the yeast preprocessing image. a Original image. b Filtered image (In Zhonglei and Peng (2012) Fig. 1)

Thirdly, noise removal is also one of the most necessary parts of pre-processing. Median filter and Gaussian filter are applied for denoising that are easy to approach and perform well in this part. The morphological open and close operations can be used to remove halos that appear while imaging, such as the work in Motta et al. (2001).

Finally, the contrast of images may not be striking and need to be enhanced for image segmentation. The gray-level histogram equalization is the most acclaimed method that is easy to operate and can enhance the contrast in a global field, such as the work in Zhang et al. (2010) and the image after enhancement is shown in Fig. 46. A contrast limited adaptive histogram equalization (CLAHE) is proposed in Ferrari et al. (2017) for local contrast enhancement, and a linear histogram expansion method is applied in Sánchez-Femat et al. (2016) that is based on a transformation of the gray levels, a linear distribution of the values that are within the range of 0 to 255 is performed. The performance is shown in Fig. 47.

Fig. 46.

Comparison of original image and enhanced image. a Original image. b Image obtained by histogram equalization (In Zhang et al. (2010) Fig. 2, Fig. 3)

Fig. 47.

Comparison of original image and enhanced image. a Original image with its gray-level histogram. b Resulting image with its gray-level histogram (In Sánchez-Femat et al. (2016) Fig. 7, Fig. 8). (Color figure online)

Image segmentation methods based on thresholding

Image segmentation is the most significant part of microorganism counting methods. The extraction of a region of interest can be regarded as the segmentation of the colony part. Segmentation based on thresholding is the basic technique widely applied for microorganism counting, while many new segmentation methods are proposed to segment the area for counting accurately.

Firstly, the segmentation technique based on thresholding is applied in many works for microorganism counting. Global thresholding is the easiest method for image segmentation when it has strong contrast, and an excellent result can be obtained, such as the works in Gupta et al. (2012) and Chunhachart and Suksawat (2016). Most of the segmentation methods are developed from the thresholding method and can improve performance in complex environments.

Secondly, Otsu thresholding is applied in many works such as Zhang and Chen (2007), Zhang et al. (2008a) and Peitz and van Leeuwen (2010). The Otsu thresholding is simple and easy to calculate. It can be used to segment the image effectively when the area difference between the target and the background is negligible. Nevertheless, the target and background can not be separated accurately when the gray-scale of the target and the background have a large overlap because the gray-scale distribution is used as the basis of image segmentation. It is also sensitive to noises, so denoising processing is usually applied first. The example of Otsu thresholding based image binarization is shown in Fig. 48.

Fig. 48.

Comparison of gray-scale image and binary image. a Gray-scale image. b Binary image based on Otsu thresholding. (In Sánchez-Femat et al. (2016) Figs. 8, 9). (Color figure online)

Finally, the segmentation result of the single threshold method is not satisfied when the gray-level of the image may be unevenly distributed, resulting in the influence of illumination. An iterative local threshold method is applied in Shen et al. (2010), the point with a local maximum threshold is obtained using a Laplacian operator, that is, the initial local thresholds. The microorganism images captured by microscope can be affected by lighting distribution, so the idea of the algorithm is not to calculate the global image threshold, but to calculate the local threshold according to the brightness distribution of different regions of the image, which means different thresholds can be calculated adaptively for different regions of the image. Another method to improve the segmentation performance when traditional thresholding does not work well is the multi-level threshold that is applied in Corkidi et al. (1998). The performance of multi-level thresholding for segmentation is shown in Fig. 49. Thresholding based on multi-level method is divided into multi-spatial-level and multi-threshold-level, it can help segment more detailed pieces of information that may be lost using global thresholding.

Fig. 49.

Comparison of gray-scale image and binary image. a Enhanced image. b Multi-level thresholding image (In Corkidi et al. (1998) Fig. 2)

Other image segmentation methods

First of all, edge detection is applied in a mass of works that performs well in image segmentation. Sobel operator is a classical method that is easy to be applied, such as the works in Andreini et al. (2015), Chiang et al. (2015) and Choudhry (2016). Laplacian operator is a second order differential operator that is isotropic but sensitive to noises, such as the works in Barbedo (2013), and Laplacian operator can be combined with Sobel operator that can obtain a better detection result, such as the work in Ogawa et al. (2003). Canny operator is a multi-stage optimization operator with filtering, enhancement and detection that performs best but is relatively complex to use, such as the works in Matić et al. (2016) and Barbedo (2012c). Another popular method is Marr-Hildreth operator, that is, a Gaussian filter is applied first for smoothing and a Laplacian filter is applied for image enhancement, such as the works in Viles and Sieracki (1992) and Blackburn et al. (1998), that performs well for images with the low signal-to-noise ratio.

Secondly, the method combined with distance transform and watershed is applied in colony segmentation. Distance transform can extract the distance between a non-zero pixel and the nearest zero pixel, that is, the gray-scale value of each pixel in the image is the distance between the pixel and the nearest background pixel. The distance transform is usually applied for image segmentation with watershed, such as the works in Hong et al. (2008), Yujie (2009) and Masschelein et al. (2012). The performance of distance transform and watershed is shown in Fig. 50.

Fig. 50.

Sample step-sequence for an automated colony counting system (In Masschelein et al. (2012) Fig. 15)

Thirdly, watershed segmentation is always applied for the separation of connected colonies. The original watershed algorithm performs well in the segmentation process for adherent colonies, such as the works in Zhang et al. (2008a), Ates and Gerek (2009) and Stolze et al. (2019). The segmentation method is shown in Figs. 51 and 15. However, the original watershed may get the results of over-segmentation because of the noise and local discontinuity of the images, the marker-controlled watershed algorithm is applied in Selinummi et al. (2005). The Hough transformation is applied to extract the object’s marker, and the background is marked and eliminated separately.

Fig. 51.

The concept of watershed algorithm (In Zhang et al. (2008a) Fig. 5)

Finally, top-hat transform and bottom-hat transform are morphological operations applied for image segmentation when the illumination is uneven. The top-hat transformation is the difference between the image and the image after the open operation, such as the works in Pernthaler et al. (1997), Brugger et al. (2012) and Barbedo (2012b), while the bottom-hat transform is the difference between the image after the close operation and the image, such as the work in Chiang et al. (2015). The top hat transformation is used for light objects on a dark background, while the bottom hat transformation is used for the opposite.

Image classification methods

Classification is a necessary operation when the microorganisms need to be counted respectively. Firstly, a decision tree is a supervisor learning that is widely used based on probability analysis. In Barbedo (2012c), a decision tree is applied for the classification of agriculture microorganisms. The decision tree is easy to understand and explain, and can make possible and practical results for large data sources in a relatively short period, but the overfitting problem while classification needs to be solved.

Secondly, support vector machine (SVM) is a kind of linear classifier that classifies data in a binary way according to supervised learning, such as the works in Chen and Zhang (2009) and Masschelein et al. (2012). In Yujie (2009), the shape invariant moment and gray level co-ocurence matrices (GLCM) are extracted for SVM training, and the classification accuracy of bacteria is 99.67%. SVM performs well with small sample and can be trained to solve the problem of high dimensional, but it is sensitive to missing data, and the choice of features has enormous implications for classification results.

Thirdly, artificial neural network (ANN) is a network with self-learning, self-organization, self-adaptation and strong nonlinear function approximation ability, that has strong fault tolerance. In Blackburn et al. (1998), an ANN is trained for classification and quantification of bacteria, and about 95% of all objects are classified in each image. The classification result is shown in Fig. 52. ANN has high classification accuracy and strong robustness to noise nerves, but the learning process is unobservable and the output is hard to interpret. Moreover, back propagation (BP) neural network is a multi-layer feedforward network trained by error back propagation, that is the most widely used ANN. In Jun (2010), BP neural network is applied for bacteria classification, and in Rong et al. (2006), BP neural network is used for zooplankton classification and counting. BP neural network has strong nonlinear mapping ability and flexible network structure, but the convergence rate is slow. Moreover, it is easy to fall into local minima. Furthermore, the convolutional neural network (CNN) is a feedforward neural network with deep structure and convolution computation representing learning. In Ferrari et al. (2015), CNN is applied for bacteria colony counting, and the accuracy of 92.8% is obtained. In Tamiev et al. (2020), a classification-type convolutional neural network (cCNN) is designed for bacteria classification and counting.CNN can automatically extract the features of images and process high-dimensional data quickly, but the pooling layer may lose much valuable information while training.

Fig. 52.

The classification result (In Blackburn et al. (1998) Fig. 4)

By reviewing all the existing deep learning based microorganism counting methods, the classification can be achieved automatically, but the segmentation part still needs to be adjusted manually. In order to show the development of deep learning and the time for microorganism counting, the training time and counting time of deep learning methods in this review are summarized in Table 3.

Table 3.

The training time and counting time of deep learning methods

Related work	Model	GPU	Training time	Counting time	Accuracy
Blackburn et al. (1998)	ANN	–	–	100 images per hour	95%
Shabtai et al. (1996)	ANN	PC-Vision Plus	–	–	–
Embleton et al. (2003)	ANN	Pentium processor	–	75 images per minutes	>90%
Hongwei (2012)	BPNN	–	–	< 1 hour	95%
Shenglang et al. (2008)	BPNN	SDK-2000	–	1 image in 3 seconds	Student’s t test p>0.05
Rong et al. (2006)	BPNN	–	–	–	–
Ferrari et al. (2015)	CNN	Nvidia Titan Black	50000 iterations in 3 hours	–	92.8%
Ferrari et al. (2017)	CNN	Nvidia Titan X	50000 iterations in 1 hours	–	92.1%
Tamiev et al. (2020)	cCNN	NVIDIA Quadro K620	–	3.8 times faster	86%
Albaradei et al. (2020)	SRNet	–	–	–	RMSE = 22.38

Analysis of potential methods

Through summarizing the work of image analysis based microorganism counting, it can be found that the accuracy of microorganism counting is continuously improving with the development of computer vision and deep learning technologies, which indicates that computer vision based microorganism counting methods will completely replace the traditional manual counting methods. However, the deep learning methods are mainly applied for microorganism classification, while the microorganism segmentation methods are still adopted the traditional techniques, such as thresholding or watershed, resulting in a huge gap with the state-of-the-art technology. The application of the latest semantic segmentation technology can classify microorganisms at the same time of segmentation, which will be the trend of future development.

According to the existing microorganism counting work, the work of imaging and image analysis are often separated. Therefore, it is difficult to obtain real-time microorganism counting information, leading to time-consuming and workforce waste. BiSeNet is one of the real-time semantic segmentation networks (Yu et al. 2018). However, an extra encoding path is applied for spatial information, which is time-consuming. In Fan et al. (2021), a novel architecture is designed as a Short-Term Dense Concatenate network (STDC network), which is shown in Fig. 53. Multiple contiguous layers of response maps are connected, and each layer encodes the input image at different scales and in its own field to achieve multi-scale feature representation. Then the Detail Guidance is applied for decoding, which can guide the low-level layers to learn spatial details. Finally, the spatial information and segmentation of deep layers are combined to show the final results. The 71.9% of mIoU is obtained, and the computing speed is 45.2% faster than the original method.

Fig. 53.

a The original architecture of STDC network. b The proposed architecture of STDC network (In Fan et al. (2021) Fig. 3)

The work of deep learning based counting methods also provides a new direction for the field. In Yang et al. (2020), the scale variations of images are solved based on a reverse perspective network, which is shown in Fig. 54. The reverse perspective networks can reduce the scale variances of images before regression, reducing the complexity of the network. The original is sampled firstly, and then the number of objects can be evaluated by a regression network. The reverse perspective networks can evaluate perspective distortion precisely, which can be correct by uniformly distorting the image. Finally, the images with similar scales are transmitted to the regressor, and 61.2 of mean average error (MAE) is obtained.

Fig. 54.

The architecture of the reverse perspective network(In Yang et al. (2020) Fig. 4)

In Bai et al. (2020), an adaptive dilated convolution and a novel supervised learning framework is proposed for self-correlation counting works, which is shown in Fig. 55. In classical counting methods, the models are optimized by comparing the ground truth and predicted image, and the density map is not precise because of the labeling deviation. First, the image is input into the model for feature extraction, and then the density map is output by using six adaptive convolutions. After that, the sample locations are calculated by dilation rates. The result can adapt the scale variation of the images, and the MAE of 66.5 is obtained.

Fig. 55.

The overview of the adaptive dilated convolution network (In Bai et al. (2020) Fig. 2)

Deep learning frameworks have quickly become the primary method for analyzing microscopic images. We can infer and predict machine and deep learning based methods will be applied in microorganism counting as well as in other researches such as digital pathology, which is summarized in Salvi et al. (2020), Madabhushi and Lee (2016). However, there are two main limitations during the surveying. First, the number of research based on deep learning is limited, proving the vast development potential in this field. Second, most of the deep learning methods are applied for classification but not for segmentation, and the deep learning based segmentation can carry out more precise segmentation results by comparing with classic methods. These two limitations also show the development directions and opportunities in the future. First, traditional manual counting methods will be replaced by deep learning based microorganism counting methods, which cannot only be used for classification, but also for precise segmentation. Second, in the future, the microorganism counting systems will be integrated with sampling, imaging and analyzing systems using deep learning, helping researchers monitor the microorganism timely.

Conclusion and future work

In this paper, a comprehensive review of image analysis methods for microorganism counting is proposed. The counting methods are summarized and grouped based on the types of microorganisms, including bacteria counting and other microorganisms counting. Then the methods are separated based on segmentation approaches, such as thresholding methods, edge detection methods, third-party tools and deep learning based methods. By reviewing all the related works, we can find that the classic methods in Sects. 2.1 and 3.1 are developed from the 1980s to 2000s, such as the Otsu thresholding method, watershed algorithm and edge detection methods, which shows a blooming development of digital image processing for microorganism analysis. Since the 2010s, the development of deep learning carries out the microorganism counting results with high accuracy. Furthermore, the development of professional microorganism counting systems is summarized in Sects. 2.3 and 3.3, such as ‘ImageJ’ and ‘CellC’ show people pay more and more attention to microorganism counting. In summary, the successful development of image analysis based microorganism counting methods shows vast research potential in this field. Moreover, the most frequently used microorganism counting approaches of image preprocessing, image segmentation and image classification are analyzed in Sect. 4.

The image analysis based microorganism counting methods discussed in this paper can be referred to in other digital image analysis fields. For example, microorganism classification is a significant application field of the microorganism analysis, referring to environmental microorganism classification (Kosov et al. 2018), cervical cell classification (Mamunur Rahaman et al. 2021), blood cell classification (Su et al. 2014), classification for different types of microorganisms (Li et al. 2019a). Furthermore, the segmentation methods for microorganisms can be referred to by digital image processing workers, such as stem cell segmentation (Huang et al. 2016), cancer cell segmentation (Chen et al. 2006), environmental microorganism segmentation (Zhang et al. 2021). Moreover, microscopic image processing performs an essential role in industrial analysis, such as the monitoring for waste water (Amaral and Ferreira 2005), beef carcass evaluation (Cross et al. 1983), monitoring of bacteria in milk (Pettipher and Rodrigues 1982), monitoring flames in an industrial boiler (Yu and MacGregor 2004), softwood lumber grading (Bharati et al. 2003) and so on. Finally, the summarized counting methods can also be applied in small object detection, such as sperm counting (Peng et al. 2015), crowd counting (Zhang et al. 2015) and vehicle counting (Li et al. 2019b).

In the future, deep learning based microorganism counting methods are promising. Since the COVID-19 broke out in 2019, people pay increasing attention to microorganism analysis. There is still a considerable limitation and opportunity in microorganism research. This review can contribute a lot to the research of microorganism counting for future researchers.