Modernized Crowd Counting Strategies for Mass Gatherings—A Review

Abstract

Estimation of crowd size for large gatherings is an indispensable metric for event planners, local authorities, and emergency management. Currently, most crowd counting relies on dated methods such as people counters, entrance sensors, and ticket sales. Over the past decade, there has been rapid development in crowd counting techniques and related technology. Despite progress, theoretical advances in crowd counting technology have outpaced practical applications. The emergence of the vast array of crowd counting techniques has added to the challenge of determining those advances that can be most readily implemented. This article aims to provide an overview of promising crowd counting strategies and recent developments applied within the disaster medicine context along with the best use cases and limitations.

Introduction

Crowd size estimation is essential for mass gathering event planning and preparation. Information on the anticipated size of a crowd in attendance serves as an indispensable metric which can guide resource allocation for event planners, local authorities, and emergency management. Any large gathering, whether planned or unplanned, is subject to the harrowing reality that crowds can turn deadly at a moment’s notice. On November 5, 2021, a fatal crush during the first night of the Astroworld Festival held at NRG Park in Houston, Texas led to 10 deaths and over 300 injuries. The stampede tragedy at Astroworld underscores the importance of accurately counting and preventatively managing a crowd. Anticipation of crowd movement patterns and real-time crowd density estimation could avert potential catastrophes by predictive recognition of dangerous crowding thresholds. At the present time, most crowd counting is based on dated methods such as people counters, entrance sensors, and ticket sales. While manual counting techniques may work adequately for controlled situations, they are labor intensive, lack accuracy and at times, not possible with large multi-directional or turbulent crowd movements. During the past decade, rapid development of newer crowd counting techniques has shown promise with recent technology incorporating computer vision and artificial intelligence (AI). To contextualize the developments, we first trace historical computerized crowd counting efforts. These efforts fit into three categories: detection, regression, and convolutional neural networks (CNNs).

Discussion

Taxonomies

Due to the great influx in crowd counting technology, crowd analysis has grown increasingly difficult to categorize. Every paper presents its own understanding of the works before it. Among the fractal understanding of the crowd counting technology, Tripathi et al. 1 distill four main categories: crowd behavior analysis, people counting, abnormality detection, and people detection and tracking. Other papers further divide these categories into smaller groups: pixel-level analysis, texture-level analysis, line counting, density mapping, etc. 2 However, as the taxonomies grow in nuance, lines are blurred as crowd counting technologies more often than not fall into several if not all of these categories. While this paper touches on counting technologies from all four categories, it is primarily focused on people counting and crowd behavior analysis as these provide the most direct relevance to emergency management.

Counting by Detection

In an effort to improve the accuracy of crowd counting and alleviate resource strain, researchers began exploration of computerized crowd counting in the early 2000s. These early crowd counting methods focused on individual detection of persons captured in picture and video segments. Detection counting methods teach a classifier to detect target objects (i.e., human faces) to obtain a count. 3 Detection-based methods can further be sub-categorized into monolithic detection, part-based detection, shape match detection, and multi-view detection. However, regardless of the detection method, such counts encounter certain problems. Low-resolution images, intense crowding, foreground obstruction, and difficulties judging perception may lead to largely inaccurate and inconsistent counts across the board. 4 When images reach a certain level of complexity and density, crowd counting by detection is no longer possible (see Fig. 1 5 ). Since these environments are precisely where crowd counting is most needed, detection-based methods have been largely discarded. Nonetheless, these pioneering approaches set the stage for higher sophistication crowd counting. 6 , 7

Fig. 1 . Crowd at the Hajj too dense to be counted by detection 5 .

Counting by Regression

Regression-based counting was developed to avoid the difficulties with data clutter. Counting by regression involves segmenting an image by features. Regression can be broken into two categories: individual regression and density-based regression. 3

Individual regression seeks to abstract features from the foreground of the image (i.e., edge count and texture features) and apply the pattern to the rest of the image to generate a count. 8 This form of regression improved upon the accuracy of detection-based counting and provided a practical application for a range of crowd scenarios. 9 However, as image resolution decreases and occlusion increases, this method encounters similar problems to that of detection. First, it is often unclear which features to extract. This can lead to errors and time-consuming training that requires extensive datasets. Second, spatial data are not considered, meaning the model can only be fed images from a certain perspective. Performance can fall when presented with a brand-new image and incomplete image features can lead to incorrect counts. 10 As such, regression as a standalone method has also largely fallen out of practice.

Density based regression sought to overcome these issues. Ushered by Lempitsky and Zisserman 11 , density-based regression employs a density map to generate a count. Counting by density combines spatial data with counts of low-level features by segments to determine density of an area. Once a density map is created, the crowd count is provided by its integral. This method can be highly accurate in certain cases and can be generated more quickly than individual regression because it classifies groups rather than individuals. 12 Further, density-based methods hold the significant advantage of preserving spatial information (i.e., density methods can relay the parts of an image that are the densest). However, while density-based crowd counting methods often provide reliable counts in uniformly dense crowds, accuracy decreases with increasingly variable density. Further, density maps struggle with specific person localization. Though a density count may quickly provide the number of people present in an image, it does not reliably characterize their individual localization. 13 Most importantly, density maps still require significant adaptation and training for each specific task at hand which severely limits the applicability of purely density-based methods. 3

Counting by CNN

CNN based crowd counting is considered the state-of-the-art counting technique. CNN is a deep learning neural network model with multiple layers of neural nodes. It has the ability to extract important features from raw images, and based on those features, it can detect different objects in the images. CNNs for crowd counting may be considered in four categories: basic CNNs, scale-aware CNNs, context-aware CNNs, and multi-task CNNs. 10

Basic CNNs utilize simply-layered CNN modulation to improve upon and train standard detection, regression, and density models. Among the first were CNN methods such as those proposed by Wang et al. 14 and Fu et al. 15 . While these held advantages over previous non-CNN methods, they continued to face difficulties with images of changing scale and perspective.

Attempts to overcome these difficulties led to the introduction of scale-aware CNNs. Most notably, the introduction of a multi-column CNN (MCNN) revolutionized the crowd counting world. 16 This method utilizes a three-columned CNN structure in which each column is defined by the size of its receptive field (large, medium, small). The proposed MCNN was a major advancement in crowd counting as it drastically improved in accuracy compared to previous methods by allowing variation in perspective and image resolution between photos. Notably, MCNN also creates a density map while generating the count ( Fig. 2 8 ). This additional information is useful as it can be used to recognize the buildup of unusual density in crowd situations. 16 Given suitable time and technical ability for training, the scale-aware CNNs, such as Zhang et al. 16 ’s MCNN, structure can also be used on new data sets. Other influential scale-aware CNNs include: RSAR 17 , ACM-CNN 18 , SaCNN 19 , CrowdNET 20 , and MMNet 21 .

Fig. 2 . Examples of convolutional neural network density crowd counting 8 .

Within the realm of scale-aware CNNs, there has been a general shift towards attention-based mechanisms. 3 These attention-based systems seek to further alleviate the burden of varying scale by weighting features to account for varying density and scale. For instance, the scale-aware CNN created by Zou et al. 18 generates an initial coarse density map, which is subsequently clarified by finely tuned micro-density maps. The two sets of maps are then fused together by a “count attention” algorithm generating a final density map and count.

Context-aware CNNs sought to enhance accuracy and reduce the complexity of CNN-based crowd counting techniques by incorporating information from local and global features from the entire image. 10 Instead of breaking the input image into patches, the context-based CNNs treat the whole image at once, first with a pre-trained CNN to capture local and global features and subsequently with recurrent network layers. 22 These CNNs are easier to tune and require less training time due to the lower level of complexity. While they rival scale-aware CNNs in pure crowd counting capability, context-based CNNs do not generate a density map, limiting their functionality. 23

Finally, the aptly named multi-task CNNs seek to generate crowd counts while simultaneously retaining the information for a variety of tasks. 10 These CNNs generally have a higher complexity and generally do not equal the broad applicability of scale and context that characterizes aware CNNs for pure count generation. Nonetheless, these CNNs may be well-suited for specific tasks. For instance, multi-task CNNs have been described to generate count in the scenarios of people crossing a certain line of interest 24 and to generate accurate crowd count across multiple scenes. 4

Method Comparison

Comparing the success of different counting methods is inherently difficult. While results among similar datasets have been compared, the gold standard datasets and the methods of evaluation have changed over time and differ by counting method. For instance, the original detection papers typically used percent accuracy, while the preferred current method of evaluation is the mean absolute error and mean squared error (MSE). Nonetheless, insofar as the results are comparable, CNNs vastly outperform all previous methods 6 , 8 , 9 , 11 , 14 , 16 - 18 , 22 - 24 ( Table 1 ).

Table 1 . Comparison of crowd counting methods based on datasets of known ground truth. Though CNNs perform better than traditional methods, CNN methods have significant variability in an application. Selecting the best suited crowd counting method depends on the specific application .

CNN: convolutional neural network; MAE: mean absolute error; MSE: mean squared error; UCSD: UC San Diego.

	Dataset	UCSD			UCF-CC-50			Shanghai A			Shanghai B
Traditional	Paper	MAE	MSE		MAE	MSE		MAE	MSE		MAE	MSE
Detection	Wu and Nevatia 6
Regression	Chen et al. 8	2.29	8.08
Regression	Xu and Qiu 9	1.90	6.01
Density	Lempitsky and Zisserman 11	1.7			493.4	487.1
Density	Wang et al. 14	1.98	1.82
CNN	Paper	MAE	MSE		MAE	MSE		MAE	MSE		MAE	MSE
Scale-aware	Zhang et al. 16	1.07	1.35		377.6	509.1		110.2	173.2		26.4	41.3
Scale-aware	Liu et al. 17				219.2	250.2		69.3	96.4		11.1	18.2
Scale-aware	Zou et al. 18	1.01	1.29		291.6	337		72.2	103.5		16.2	25.8
Context-aware	Shang et al. 22
Context-aware	Liu et al. 23				212.2	243.7		62.3	100.0		7.8	12.2
Multi-task	Zhao et al. 24	1.18

One of the most well sought-after applications of crowd counting is the prevention of human stampedes and specifically, crowd crush injuries. Crush injuries typically occur when crowd density exceeds a certain critical threshold. CNN models that provide density maps could help mitigate risks by monitoring crowd density and alerting authorities as it approaches critical density. While density-based scale-aware CNNs have had significant success in improving accurate counts on the ground-truth data sets, these CNNs have largely been tailored for these ground-truth data sets, limiting their real-world applicability. 25 Nevertheless, this technology appears fairly promising and with additional tailoring and training could prove instrumental in improving crowd safety.

Recent advances in CNN crowd counting technology also have useful applications in security contexts. For instance, Sam et al. 13 developed a CNN crowding counting algorithm that both generates a count and bounds each notable head in a bounding box even in extremely dense crowd ( Fig. 3 ). In sparser crowds (i.e., airports, malls, etc.), there exists a plethora of deep-learning-based counting mechanisms that not only count and bound each person, but also automatically track people between scenes. 2 For instance, Li et al. 26 proposed region-based CNN which utilizes a deep-learning trained head-shoulder detector to detect people in multiple poses, obtain their trajectory, and track them within a set bounding box. Other CNNs, such as that of Japar et al. 27 , have taken it a step further and not only track individuals, but “coherent groups” across multiple scenes ( Fig. 4 ). Such crowd analysis could help authorities track and find suspect individuals in an emergency situation and better understand crowd movement to aid the design of future event settings.

Fig. 3 . Example of bounding boxes from Sam et al. 13 .

Fig. 4 . Example of coherent group detection from Japar et al. 27 .

Additionally, real-time action analysis in crowds has become a viable option. You and Jiang 28 developed an Action4D analysis program that has the ability to abstract individuals from their surrounding and predict their actions. This method saw significant success even in scenarios with many people and significant clutter. Currently, its widespread usage is primarily limited by the limited implementation of RGBD cameras in many scenarios. 2 On the macro level, Wei et al. 29 developed a CNN that analyzes crowd behavior. By monitoring and analyzing the coherency of actions and trajectories of people within the crowd, the program determines if the crowd is heterogeneous (without unity), homogeneous (with unity of purpose or prevailing motive for action), or violent (exhibiting extreme emotional excitement). Furthermore, it can automatically notify authorities as the crowd behavior begins to shift, thus helping authorities react before the onset of a violent or panicked crowd.

With the success that CNNs have seen, traditional methods still retain certain advantages. Primarily, traditional methods require less training data and less training time to achieve reliable counting in new scenarios. In quick deployment scenarios, detection-based counting is still likely to dominate for the time being. The key differences and advantages of each method are summarized in Table 2 .

Table 2 . Summary of advantages and disadvantages of counting methods .

Method	Advantage	Disadvantage
Detection	• Quick deployment • Minimal training required • Can be used in real-time	• Low performance in dense scenes • Performance falls quickly with low quality footage
Density regression	• Generates density map • Accurate in homogeneously dense crowds	• Requires high quality data • Accuracy falls in crowds with varying density • Real-time analysis not available
Individual regression	• Works well with limited data • Not affected by density and image clutter • Takes time and pre-labeled data to train	• Does not generate a density map • Real-time analysis generally not possible • Requires significant training
Deep learning (convolutional neural network)	• Most accurate form of crowd counting • Flexible to specific needs • Works with dense crowds • Can generate other useful data and detect anomalies	• Accuracy varies widely with different deep learning models • Application varies significantly between models. Can be more difficult to set up • Real-time analysis not always possible

Collecting Data for Crowd Counting Using Unmanned Aerial Vehicles

Given the ubiquity of unmanned aerial vehicles (UAVs), their role in crowd counting is worthy to mention. UAVs, also referred to as drones, are often proposed to collect data in the application of crowd counting methods. The use of drones has proven to be successful in very specific instances. For instance, images captured by UAVs were processed by a color based-crowd segmentation method to monitor crowds at the Hajj. 30 This approach demonstrated as high as 80% success in real-time crowd count and density monitoring. 31 However, this approach may lack generalizability as it relies on the ability to distinguish participants by their white clothing from their background. Against a lighter background, color-based segmentation resulted in significant crowd counting inaccuracies. While useful in this particular context, the color-based approach may not be applicable elsewhere.

Apart from this notable Hajj example, crowd counting using data obtained from UAVs has enjoyed limited success largely because its development is hampered by the lack of UAV datasets with annotated ground truth. Aerial datasets are limited in number and in scope. Prior to 2020, the largest aerial dataset contained a max count of 188. 31 Yet, signs point to further development in this sector that could have powerful implications. Several developments within the past year have begun to open the door for UAVs for collecting data for crowd counting. For instance, Wen et al. 31 introduced a dataset with 3,360 frames and max count of 421. In competitions for crowd counting using drone-based data, several CNN models achieved an MSE of below 15. This suggests that while using UAVs to collect data for crowd counting has current limitations, there is significant promise for the next generation of crowd counting.

Conclusion

In this paper, we provide a brief review of the history of crowd counting and the importance of its modernization. Some of the most cutting-edge developments and potential applications with regards to event planning and management are examined. To acknowledge an important limitation, not every noteworthy crowd counting technique could be reviewed due to the sheer breadth of technological advancements. Given the rapid developments taking place in the field, including proprietary or classified work, new and promising crowd counting approaches may not have been captured or described here. As AI grows and becomes increasingly seamlessly incorporated into crowd counting, it is anticipated that more accurate counting methods as well as counting methods that can perform with minimal training time will become more widely available. In particular, as the datasets for drone crowd counting grow and the ability to capture quality aerial images improves, drone crowd counting is likely to become integrated into crowd management as it can be readily deployed to situations without pre-installed and trained camera networks. As with most computational advancements, advanced crowd counting techniques will continue to develop at a rapid pace and ways to best incorporate this into practical event planning and management should be strongly considered.