A dynamic light image enhancement algorithm using generative adversarial network for group activity recognition

Abstract

Video-based or image-based human activity recognition (HAR) via machine learning algorithms helps track, detect, and categorize users’ daily activities. It is usually formulated as specific research problems, such as fall detection, gait recognition, posture recognition, and gesture recognition. In practice, HAR is extended to a more generic group activity recognition (GAR) problem, focusing on multiple user (group) activities for various purposes, such as the recognition of social activities, autonomous driving, surveillance, and pedestrian crossing management. In this research, the formulation of the GAR problem aims to enhance the performance of the GAR model by tackling the research limitations in three aspects: (i) image quality may not always be guaranteed, resulting bias and inaccurate models; (ii) the working environments, in both indoor and outdoor, of the GAR model are dynamic, such as varying light conditions; and (iii) the number of people in the covered area may vary and the number of class labels for activities is large, leading to an extensive computing time to train a complex GAR model architecture. Therefore, a dynamic light image enhancement generative adversarial network has been proposed to enhance the image quality significantly. A multi-input image-enhanced generative adversarial network (MIIEGAN) has been proposed for generating high-quality additional training images, and a guided asymmetric depthwise separable convolution (GA-DSC) contributed to the model complexity and performance. These algorithms were evaluated in dynamic light environments to investigate their robustness and performance in varying light conditions that meet the nature of GAR in indoor and outdoor environments. Two benchmark datasets (The Volleyball Dataset and The Collective Dataset) were selected for the performance evaluation and comparison. Five backbones (AlexNet, VGG-16, Inception-v3, ResNet-50, and EfficientNetV2) were chosen as deep learning models for analysis. Nine existing methods were compared. Five variants of GANs and five variants of convolutional techniques were compared in the ablation experiments. Through experiments and ablation studies, the proposed work outperforms existing works (0.931–9.29% and 1.05–9.83%), five variants of GANs (on average 2.71% and 2.62%), and five variants of convolutional techniques (on average 1.91% and 1.75%).

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-026-49326-0.

Introduction

Background and motivation

Human activity recognition (HAR) using artificial intelligence (mainly deep learning algorithms) is one of the essential elements in understanding human mobility, health status, and quality of life, which are two key smart areas specified in the Hong Kong Smart City Blueprint 2.0. A recent report, Leveraging Human Mobility to Rescue the 2030 Agenda, stated, “Well-managed human mobility is key to achieving the 2030 Agenda”¹. Recognizing the activities of individual users, particularly for elderly living alone, may help to identify potential diseases, such as Alzheimer’s disease, Parkinson’s disease, and muscle damage that affect their quality of life². A recent survey³ was conducted on the market size of computer vision, estimating that the worldwide annual market volume will reach USD 29.9 billion (2025) and grow by 16.0% in the timeframe of 2025–2031. Generally, people live in groups, where HAR is extended to group activity recognition (GAR). Some existing studies segmented the images of multiple individuals into images of individuals if individuals are sufficiently separated from each other, i.e., formulating the GAR problem as numerous HAR problems after image segmentation. The recognition of the activities of a group of people can be applied to various applications, including analysis of social interactions⁴, autonomous driving⁵, security surveillance⁶, and pedestrian crossing management⁷.

Our research contributions

The research contributions are threefold:

• (i)A multi-input image-enhanced generative adversarial network (MIIEGAN) has been proposed for generating high-quality additional training images. The formulation used semi-supervised learning by retaining the labelled group activities while ignoring the labelled individual activities. As a GAR problem, individual activities were considered unlabelled, supplementing the data in a semi-supervised learning setting. Compared to fully-supervised learning, the semi-supervised learning strategy generates more reliable data, enhances image quality, and denoises images. Our work achieved an overall accuracy of 95.3–97.6% and 92.3–95%, respectively, using two benchmark datasets and five backbone deep learning models. The proposed method also outperformed existing studies by 0.931–9.29% and 1.05–9.83%, respectively;
• (ii)An investigation into the robustness of the MIIEGAN on low-light and varying-light conditions, examining the model performance in dynamic light environments. Having a GAR model serving in dynamic light environments, i.e., indoor/outdoor, different light intensity levels, and background information, is desired. Image synthesis was performed by considering three aspects: exposure, contrast, and noise insertion. The performance of the GAR model using MIIEGAN was compared with five variants of generative adversarial networks (GANs). The average percentage of improvement by MIIEGAN is 2.71% and 2.62%, respectively, using two benchmark datasets and five backbone deep learning models; and.
• (iii)A guided asymmetric depthwise separable convolution (GA-DSC) has been proposed to optimally design the number and extent of depthwise separable convolutions (DSC), thereby achieving a tradeoff between computation time, model complexity, and model accuracy, satisfying different conflicting objectives. This helps suggest optimal trade-off solutions to the different needs and requirements of GAR problems. The performance of the GAR model using GA-DSC was compared with five variants of convolutional techniques. The average percentage of improvement by GA-DSC is 1.91% and 1.75%, respectively, using two benchmark datasets and five backbone deep learning models.

Organization of the paper

The rest of the paper is organized as follows. The second section details the methods and formulations of the proposed MIIEGAN and a GA-DSC mechanism. The performance evaluation of the proposed methods is presented in the third section. The results of the ablation experiments will be illustrated in the fourth section to reveal the contributions of individual components to the GAR model. The statistical significance between different models will be compared in the fifth section. In the last section, potential future research directions are discussed before concluding remarks.

Related work

A literature review is first presented on three major categories of GAR approaches, attention-based^8–11, relationship-based^12–15, and ensemble learning-based approaches^16–19. It is followed by a discussion of recent methods for low-light image enhancement and lightweight convolutional architectures.

Attention-based GAR methods

The discussion started with attention-based approaches. In⁸, researchers utilized the merge function and cross-attention mechanism for GAR using RGB modality. Structural and temporal features were aggregated using an inter-frame gating fusion mechanism. Performance evaluation of the model using four benchmark datasets yielded an accuracy range of 84.5–96.8%. Huan et al.⁹ proposed a joint attention mechanism that built two networks in the data and semantic domains. It achieved 83.3–95.0% accuracy for random group size (number of individuals from two to five) and 99.8–99.9% for fixed group size in two benchmark datasets. A self-supervised transformer network was proposed to leverage the traditional weakly supervised learning approach¹⁰. It ensured that the spatio-temporal dimensions remain consistent. The results outperformed seven existing studies using weakly supervised learning; however, the proposed model tied with the performance of one existing work and was less accurate than two existing studies, which used fully supervised learning. It suggested that the model takes advantage of datasets that are partially labelled and suffer from noisy environments. Zhu et al.¹¹ proposed a dynamical attention hypergraph convolutional network. Results revealed that the model enhanced the accuracy of GAR; however, a statistical test was not performed to check if the proposed model outperformed all existing studies, and it is likely not to pass the test because the percentage of enhancement is less than 0.5% in various studies.

Relationship-based GAR methods

Regarding relationship-based approaches, several studies have been reviewed. The spatio-temporal correlations and interaction between individuals were considered in the GAR problem¹². Differing from existing works using graph convolutional networks and recurrent neural networks, researchers proposed graph parsing networks. The model achieved a mean average precision of 59.7, comparable to that of an existing study, and outperformed eight existing studies by 34.8–59.2%. Wang et al.¹³ employed the unified spatio-temporal graph to investigate the spatio-temporal relationship between individuals. Two benchmark datasets were used to test the model, achieving 91.1% and 93.2% accuracy. A clustering-based spatio-temporal interactive reasoning model was proposed for GAR¹⁴. It was characterized by the feature extraction of individuals, graph reasoning of inter-actor relationships, extraction of the nearest neighbor, clustering of spatio-temporal dimensions, and group interaction reasoning. The accuracy enhancement by the proposed model in three benchmark datasets was: a slight enhancement from 69% to 69.6% compared to one work; a significant enhancement from 55% to 76.7% compared to one work; and a moderate to significant enhancement from 8.7 to 25.5% to 29% compared to six works. A sensor-based relation gating and spatiotemporal attention was proposed¹⁵. The relationship of the group was selected via a deep reinforcement learning strategy. Results confirmed that the model outperformed the existing works using two benchmark datasets.

Ensemble learning-based GAR methods

Attention is also drawn to the ensemble learning-based approach. An adaptive random forest was designed for the GAR of multiple residents¹⁶. A weighted ensemble learning was introduced to leverage the model performance. A performance comparison was made with seven existing methods to confirm the performance enhancement in terms of F1 score, recall, precision, and accuracy. Five models were ensembled to enhance the performance of the GAR model in public regions¹⁷. A dual-image stream was used to reduce the dimensions and thus the computational power and model complexity. Ablation studies revealed the benefits of the loss function, CNN, and context module, yielding about 1–3% improvement in the model performance. Another work¹⁸ utilized ensemble learning to extend the YOLOv5 model. It was validated by a dataset that covered scenarios from different weather conditions in the crowd and mall environments. Ensemble learning improved the model performance by 2%. Attention mechanism and multi-modalities were formulated for GAR¹⁹. Ensemble layers aggregated the features of multi-modalities. Apart from the typical model evaluation metric, accuracy, Zaman et al.²⁰ aimed to enhance the real-time capability, effectiveness, and efficiency by ensemble learning of three algorithms, including CNN, gated recurrent unit, and multilayer perception. Another work²¹ proposed a faster regional CNN approach to speed up the detection with a rectangular region scheme.

Low-light image enhancement algorithms

Improving low-light images aims to reduce noise and enhance brightness and contrast. A recent review-type article²² analyzed more than 160 low-light image enhancement algorithms, concluding that the top two categories were the Retinex model (29%) and deep learning (18.5%).

Yin and Yang²³ proposed to combine the Retinex theory with iterative learning mechanisms for the enhancement of details and colours, ultimately, the feature engineering. Feature loss was minimized using a global feature attention module. Compared to four existing methods, the improvement in multiscale structural similarity was 0.65–4.99%, and in peak signal-to-noise ratio was 3.5–36.1%. A texture and structure guided variation technique was proposed to enhance the Retinex model formulation²⁴. Performance evaluation and comparison indicated a tradeoff between five performance evaluation metrics, including contrast-changed image quality measure, natural image quality evaluator, learned perceptual image patch similarity, structural similarity, and peak signal-to-noise ratio, for some existing methods yield better results in some metrics.

Regarding deep learning algorithms, a lightweight CNN with exposure fusion was proposed to reduce the computational cost (6.8–8.3%) and noise level (0.48–1.48 dB) of low-light images²⁵. Another work combined the autoencoder, attention guidance, and dense convolutional network to achieve low-light image enhancement²⁶. The model performance was evaluated and achieved a slight improvement in peak signal-to-noise ratio (2.55%), multiscale structural similarity (0.488%), and earned perceptual image patch similarity (1.83%).

Lightweight convolutional architectures

A lightweight convolutional block (attaching to a depth-separable CNN) was used to address the issue of sophisticated decisions for image detection problems²⁷. Performance evaluation showed that the proposed method outperformed three existing methods in terms of recall and F1-score, and was comparable in terms of precision. Another work²⁸ proposed a frozen random-convolution scheme, which comprised two branches for freezing random convolutional feature extraction. It reduced the training time by up to 90%. An efficient downsampling convolution was proposed to address the trade-off between feature preservation and spatial compression²⁹. It reduced the computational cost to 2.9 GFLOPs and increased mAP50 by 1.1%. Jiang et al.³⁰ proposed 1D and 2D complex-valued convolutions for image classification. Results indicated that the method significantly reduced FLOPs and Params compared to six existing methods. In various studies, CNN has demonstrated superiority in noise reduction in images^31,32.

Research gaps and objectives

This research emphasizes three research aspects (research problems) of GAR models. It begins with data quality management and enhancement to ensure reliable and high-quality data is available for deep learning model training. The data type is usually videos and images because it is expensive to ensure all target participants wear sensors. The rationale for designing and formulating an image enhancement algorithm is to enhance the video quality of the spatial-temporal metrics³³, subject to an in-depth investigation of tradeoff solutions (involving factors such as acquisition time, resolution, and signal-to-noise ratio). In essence, generating additional training data based on high-quality samples can further enhance the dataset quality. Many research studies have revealed data generation’s effectiveness in improving the deep learning model’s performance^34,35.

The next concern is that the applications of GAR can be in indoor or outdoor areas that experience different (mainly two) light conditions: (i) low-light³⁶: the weak visibility and low contrast may challenge the restoration of visual information, particularly in blocked and dark regions (with hidden information), i.e., others may block some bodies. The indoor environments are often low-light; and (ii) varying light³⁷: outdoor areas during daytime are highly affected by varying sunlight intensities. Reflective materials under high light intensities may lower the image quality. Different weather conditions, such as sunshine, lightning, fog, rain, cloudy, and sun, will also affect the light intensities of images. Therefore, the research question is to design a GAR model to maintain performance in both light conditions (such as dynamic light).

The third research aspect is about a high computational power to train a deep learning-based GAR model, equivalent to a heavy model complexity. The former requires high-performance computing services to proceed with a large number of computing tasks. The time for model training and hyperparameter tuning is generally lengthy because of the nature of large-scale datasets and a large number of hyperparameters. The latter describes large-scale deep learning models, which are inappropriate to deploy in edge devices to support real-time monitoring and recognition of group activities. Fog devices and cloud platforms are capable of large-scale deep learning models; however, they increase the latency, which does not fulfill the real-time requirements of GAR applications. To summarize, a lightweight deep learning model is desired to be deployed in edge devices.

Before highlighting the research contributions of this work, a review of related work is presented to provide the latest developments of recent GAR studies, including the methodologies and results.

Methodology

This section is organized based on four aspects: (i) a brief summary of the key information of two benchmark datasets; (ii) the design of MIIEGAN for generating high-quality additional training images; (iii) image synthesis for creating dynamic light environments; and (iv) the design of the GA-DSC.

Benchmark GAR datasets

A recent review-type article^38–40 summarized that the benchmark GAR datasets could be categorized into (i) Sports video datasets, and (ii) Surveillance video datasets. In our research, two benchmark datasets are selected, one from each category and both received the highest number of Google Scholar citations, namely, The Volleyball Dataset⁴¹(in the first category and 692 citations) and The Collective Dataset⁴²(the second category and 482 citations), representing indoor and outdoor environments, respectively. It is also worth noting that many of the benchmark datasets in the sports video category did not provide ground truth labels for individual activities. Table 1 summarizes the details (environments, GAR applications, number of videos, number of frames, number of group activities, and number of individual activities).

Table 1.

Summary of benchmark GAR datasets for the performance evaluation of algorithms.

Datasets	Environments	GAR applications	Number of videos	Number of frames	Number of group activities	Number of individual activities
The Volleyball Dataset41	Indoor	Sports	55	4830	8	9
The Collective Dataset42	Outdoor	Surveillance	44	2500	5	6

Design of the multi-input image-enhanced generative adversarial network (MIIEGAN)

Images can be easily degraded in low-light and indoor environments, which require performing image enhancement to restore intricate details. The formulations of image enhancement algorithms can be categorized as supervised (all data are labeled), semi-supervised (some data are labeled), and unsupervised (all data are not labeled). Intuitively, supervised learning is chosen because both group and individual activities are labeled in the benchmark datasets^41,42. Inspired by a highly cited research study⁴³ using cGAN as the core GAN architecture, semi-supervised learning using a conditional generative adversarial network can be used as a framework for image enhancements. In a typical semi-supervised learning, the semi-supervised learning will be using labeled group activities and unlabeled group activities. In our formulation, we have formulated the image enhancement algorithm for GAR as semi-supervised learning by retaining the labeled group activities while ignoring the labeled individual activities. The group activities and individual activities serve as multi-input. Different arguments can lead to different formulations: (i) As a GAR problem, ignoring individual activities is straightforward because GAR is different to individual activity recognition. This is a fully supervised learning problem. However, model performance can be enhanced if more useful information, i.e., individual activities, is provided; (ii) With the availability of labelled group activities and individual activities, including all labelled data as a fully-supervised learning problem, will extend the GAR to GAR and individual activity recognition. This will significantly enhance the complexity of the model, and feature extraction becomes more difficult; (iii) Employing labelled group activities and unlabeled individual activities maintains the application to only GAR, and fully supervised learning is reduced to semi-supervised learning. The benefits of unlabeled individual activities are that they provide additional information, as they help enhance the feature extraction process for group activities. As a data preprocessing step, all images were normalized to stabilize the training process of MIIEGAN. The MIIEGAN framework takes advantage of generating more reliable data, enhancing the image quality, and denoising images.

Figure 1 shows the architecture of the proposed conditional generative adversarial networks. The generator G takes both group and individual images. Feature extraction will be carried out for each group and individual. G generates synthetic image samples (with the additional description of the activity information related to individual images as the overall activity). This will be passed into the discriminator D and the real labelled activity images. The training of G aims to generate information on activities (mainly related to individual images) that are indistinguishable from real activities. In contrast, the discriminator distinguishes between real and fake group activities. A detailed explanation of the formulation is presented in the remaining paragraphs.

Fig. 1.

An architecture of the proposed multi-input image-enhanced generative adversarial network model for image generation and image enhancement.

The benchmark dataset is considered to possess unlabeled samples and labeled samples . The total number of group activities is so that the labeled class labels are . The aim of the formulation is to learn a conditional label distribution based on and to estimate the class label of test samples. Referring to the prior distribution for some noisy samples , generates fake samples. serves as a classifier to estimate group activities with a possible pseudo-class label, false. In other words, MIIEGAN characterizes the input distribution and classifies group activities using the modeled discriminated information.

The discriminator loss is defined as:

1

2

where the first term in Eqs. (1) and (2), , is the expected input under the class labels, classifying group activities. It is worth noting that provides the probability of the input of the fake class label that is identical to the probability . As a result, there are two formulations in the second term of Eqs. (1) and (2).

To precisely reflect the utilization of unlabeled and labelled samples, the discriminator loss () is updated using a loss with unsupervised learning () and a loss with supervised learning ().

3

4

5

where and are control factors to control the importance of labelled and unlabelled samples.

Here are the details of various key components in the MIIEGAN framework: (i) generator: Skip connections are used to retain important features from earlier layers, which can help improve the quality of generated images. Attention mechanisms are also employed to allow the generator to focus on the relevant parts of the input images corresponding to the desired activity; (ii) discriminator: A multi-scale discriminator is used to analyze the generated images at different resolutions, improving its ability to distinguish between real and fake images. Spectral normalization is also included to prevent the discriminator from overfitting and to stabilize training; (iii) Learning rate: Adam and RMSprop are used as the adaptive learning rate schemes to adjust the learning rate based on the training progress. This can lead to better convergence properties; and (iv) fine-tuning hyperparameter: A grid search is used to optimize the hyperparameters, including learning rates, and coefficients for the mixing of losses. Consider using grid search.

Figure 2 shows some selected images, including original images and generated images using MIIEGAN.

Fig. 2.

Selected images for The Volleyball Datase from subfigures (a) to (l) and for The Collective Dataset from subfigures (m) to (v): (a) left pass (original), (b) left pass using MIIEGAN, (c) right pass (original), (d) right pass using MIIEGAN, (e) left set (original), (f) left set using MIIEGAN, (g) right set (original), (h) right set using MIIEGAN, (i) left spike (original), (j) left spike using MIIEGAN, (k) right spike (original), (l) right spike using MIIEGAN and (m) crossing (original), (n) crossing using MIIEGAN, (o) waiting (original), (p) waiting using MIIEGAN, (q) queueing (original) (r) queueing using MIIEGAN, (s) walking (original) (t) walking using MIIEGAN, (u) talking (original) and (v) talking using MIIEGAN.

Algorithm 1 presents the formulations of MIIEGAN.

Algorithm 1.

MIIEGAN.

Image synthesis for the creation of dynamic light environments

Although the benchmark datasets comprised GAR data from the indoor⁴¹ and outdoor environments⁴², the available samples do not cover the full range of light environments. The available benchmark GAR datasets were not focused on collecting data in varying light conditions. The resources required for the synthesis of dynamic lighting environments are considerable. They include not only substantial data storage capabilities but also significant investment in time and the deployment of multiple devices equipped with advanced sensors and imaging technology. These resources are fundamentally necessary to capture comprehensive data reflective of different light sources, angles, and conditions. By creating a controlled yet diverse set of images that represent various lighting environments, we can evaluate the performance of our proposed approach more thoroughly. This allows for a more effective assessment of how well our model adapts and responds when exposed to real-world complexities, ultimately leading to improved accuracy in applications such as computer vision, autonomous driving, and augmented reality. The ability to effectively navigate and interpret dynamic lighting situations will greatly enhance the robustness and reliability of image processing systems, paving the way for advancements across multiple technology domains. To evaluate the robustness and applicability of the proposed algorithms in different (we call it dynamic) light environments, image synthesis is performed to generate images for dynamic light environments. Three aspects: exposure, contrast, and noise insertion have been considered. Figure 3 summarizes the workflows of image synthesis.

Fig. 3.

Workflows of image synthesis for varying light environments.

The exposure and contrast of images can be adjusted using:

6

where is the synthetic image based on the original image . , , and are transformation factors, with extended range from the existing study⁴⁴. A step size of 0.1 is used for all transformation factors.

The synthetic images are further transformed with the insertion of Gaussian-Poisson noise as follows:

7

where is the operator for uniform sampling, is the operator for RGB to Bayer image conversion⁴⁵, is the operator for Poisson noise insertion, and AWGN is the additive white Gaussian noise with variance .

Design of the guided asymmetric depthwise separable convolution (GA-DSC)

Before illustrating the design of the proposed GA-DSC, the foundation of DSC^46,47 is recalled. The nature of DSC divides a convolution layer into two convolutional layers: (i) a depthwise convolution applies a convolutional filter to each input channel; and (ii) the output is then combined using a pointwise convolution (i.e., a 1 × 1 convolution). It facilitates the reduction of computing power and model complexity.

Considering the number of input channels , number of output channels , size of the convolutional kernel , and size of the feature map . The costs of the standard convolutional layer () and that of DSC () are:

Standard convolutional layer:

8

DSC:

9

The asymmetric DSC (A-DSC) strategy further breaks the convolutional layer (size of ) into one depthwise convolutional layer and one depthwise convolutional layer. Therefore, the cost of A-DSC () is reduced to:

A-DSC:

10

There are two reasons for a further modification of the design, as a guided A-DSC (GA-DSC): (i) Regarding the formulation of A-DSC, many convolutional layers can be used in the design of the deep learning model. In the extension of A-DSC, an investigation is needed on whether A-DSC takes advantage of both model complexity reduction and model accuracy retention. Precisely, whether some convolutional layers require designing as DSC or standard convolutional layers, as a mixture of convolutional strategies instead of a full A-DSC; and (ii) performing pointwise convolution after two asymmetric depthwise convolutional layers is a more straightforward design to complete one process before the next. The consideration of the functionalities of depthwise convolution and pointwise convolution can enhance the design of the formulation. Yet, the information (features) follows a 1-to-1 channel flow between a convolutional layer and a convolutional layer. Some important features may be ignored because the pointwise convolution combines the features after performing the and convolutional layers, which is a three-step process. Therefore, an alternating depthwise-pointwise strategy is used in a four-step process, a convolutional layer (Step 1), a pointwise convolution (Step 2), a convolutional layer (Step 3), and a pointwise convolution (Step 4). These two pointwise convolutions can efficiently combine the features from all output channels of the depthwise convolutional layer.

Considering the hyperparameter settings and fine-tuning, various aspects are considered: (i) kernel sizes: Selecting appropriate kernel sizes is pivotal for capturing relevant features from the input data. The nature of asymmetric convolutions allows for versatility in design, meaning that experimenting with various configurations is not only beneficial but often necessary. Different kernel sizes can focus on different scales of the input, enhancing the model’s ability to recognize patterns and details within varying contexts.; (ii) stride value: A smaller stride facilitates fine-grained feature extraction as it moves the kernel pixel by pixel, while larger strides can lead to a reduction in the output dimensionality and computational cost. Testing different stride values allows for the discovery of an optimal balance between feature retention and efficiency, directly impacting the model’s accuracy and performance; (iii) activation function: ReLU and Leaky ReLU are used as non-linear activation functions after each convolutional operation, which can significantly enhance the learning capability of the network. By introducing nonlinearities after each convolutional operation, these functions help the network make informed decisions based on its learned features, significantly enhancing the training process and overall learning capability.; and (iv) batch normalization: Implementing batch normalization after each convolution step helps in normalizing the output, which leads to more stable and faster training. It mitigates internal covariate shift by ensuring that inputs to each layer maintain a consistent distribution. This not only accelerates convergence during training but also diminishes the dependency on careful initialization and regularization methods. It contributes to a more robust model that can generalize better to unseen data. Figure 4 shows the architecture of the GA-DSC.

Fig. 4.

Architecture of GA-DSC.

Performance evaluation of the proposed method

The research focus is not intended to train a new deep learning model from scratch, but rather to provide a GA-DSC for the convolutional part. Therefore, five pre-trained models, AlexNet⁴⁸, VGG-16⁴⁵, Inception-v3⁴⁹, ResNet-50⁵⁰, and EfficientNetV2⁵¹, are selected as the backbones for the performance evaluation of the proposed algorithms in dynamic light environments.

A k-fold cross-validation with k = 5 is chosen for the iterative training and validation process, which is a widely used strategy in evaluating deep learning models^52,53. 5-fold cross-validation is an effective resampling method for assessing model performance by dividing the dataset into 5 equal segments, or folds. This approach helps mitigate overfitting and delivers a more precise evaluation of data that the model has not encountered before. In this process, the model is trained five times, each time using four folds for training and reserving one for testing. The final performance metrics are calculated by averaging the results from each iteration. It is noted that the overall accuracy is the most common performance evaluation metric used in the literature. The evaluation will be divided into two parts: (i) individual group activities and overall; (ii) dynamic light environments.

Performance evaluation towards individual group activities and overall

To extend the performance evaluation, we include the accuracy of individual group activities and the overall group activities. The analysis will examine whether the proposed model biases towards individual group activities. Ideally, the applicability of the model in GAR is related to satisfactory performance across all individual group activities. Table 2 summarizes the results of the proposed algorithms for GAR using five backbone models and two benchmark datasets. Accordingly, Fig. 5 is presented as a visual comparison.

Table 2.

Results of the proposed method across five backbone models and two benchmark datasets.

	Accuracy (%)
The Volleyball Dataset
	Individual group activities												Overall
	Right set	Right spike		Right pass	Right winpoint		Left winpoint		Left pass	Left spike		Left set
AlexNet	96.1	95.9		96.3	94.5		94.3		96.8	95.7		96	95.9
VGG-16	96.4	96.7		97.7	95		94.7		97.5	96.8		96.6	96.7
Inception-v3	97.6	97.2		98.1	95.6		96.1		98.6	97.5		97.8	97.6
ResNet-50	96.5	96.6		97.7	95.2		95.6		98	97.2		96.8	97
EfficientNetV2	95	95.2		95.7	94.1		93.8		96.1	95.3		95.5	95.3
The xollective dataset
	Individual group activities												Overall
	Crossing		Waiting			Queueing		Walking			Talking
AlexNet	91.8		92.7			93.3		92.4			94		92.9
VGG-16	93.1		93.8			94.3		93.6			94.7		93.9
Inception-v3	93.4		94.5			95		94.2			94.6		94.4
ResNet-50	94.1		94.8			95.5		94.5			95.9		95
EfficientNetV2	91.4		92.4			92.9		92			92.5		92.3

Fig. 5.

Visual comparison of the proposed method across five backbone models and two benchmark datasets: (a) The Volleyball Dataset and (b) The collective dataset.

Regarding the benchmark dataset, The Volleyball Dataset, the performance of models, in descending order, is Inception-v3, ResNet-50, VGG-16, AlexNet, and EfficientNetV2. The average accuracy of five models in the individual group activities in descending order is left pass, right pass, left set, left spike, right spike, right set, left winpoint, and right winpoint. The average mutual deviation between individual group activities, in ascending order, is EfficientNetV2 (0.965%), AlexNet (1.04%), ResNet-50 (1.20%), Inception-v3 (1.21%), and VGG-16 (1.29%), showing that the issue of biased models is mild.

Considering another benchmark dataset, The Collective Dataset, the performance of models, in descending order, is ResNet-50, Inception-v3, VGG-16, AlexNet, and EfficientNetV2. The average accuracy of five models in the individual group activities, in descending order, is talking, queueing, waiting, walking, and crossing. The average mutual deviation between individual group activities, in ascending order, is EfficientNetV2 (0.757%), Inception-v3 (0.761%), VGG-16 (0.827%), ResNet-50 (0.964%), and AlexNet (1.13%).

The reasons for the achievements in different models are analyzed as follows. Inception-v3 utilizes a deep convolutional neural network structure with multiple filter sizes in its inception modules, allowing it to capture both fine and coarse features from the input images. The architectural design facilitates effective utilization of computational resources through the use of filter concatenation, which improves the model’s ability to extract relevant visual information without increasing the overall complexity significantly. Its efficient use of computational resources and ability to handle varying types of features likely contributed to its superior performance. Residual connections were used in ResNet-50, which allows gradients to flow more easily during the training of very deep networks. This mitigates the vanishing gradient problem that often hampers deep learning models, enabling the training of deeper architectures without significant degradation in performance. The residual connections help in maintaining network performance as depth increases, resulting in better feature representation and classification accuracy. AlexNet and VGG-16 have simpler architectures that may not capture the complexity of features as effectively as the more advanced architectures can. EfficientNetV2, while designed to optimize performance with fewer parameters and less computational cost, might not have been tuned for the same level of complexity compared to Inception-v3 and ResNet-50 in GAR problems.

Investigation of the model performance in dynamic light environments

The aim is to obtain a robust GAR model, which is capable of supporting dynamic light environments in indoor and outdoor areas. The rationale is explained as follows: Indoor settings typically have more controlled lighting conditions, often characterized by artificial light sources. This can lead to less variability in light intensity and color temperature compared to the outdoors. Outdoor environments are more variable and subject to fluctuations in natural light due to changing weather conditions, time of day, and seasonality. This variability necessitates a GAR model that can adapt quickly and efficiently. The results of the overall accuracy (performance of individual group activities is not reported to avoid excessive details of numeric results, as many scenarios are analysed).

Table 3 presents the results of the models in varying , , and , respectively, as the analysis of the model robustness towards light exposure and light contrast. In addition, Fig. 6 is presented as a visual comparison. In both benchmark datasets, it can be seen from the results that degradation of model performance is observed. Considering using The Volleyball Dataset, the degradation details are (0.417,0.834,2.29)% in AlexNet, (0.414,0.724,1.96)% in VGG-16, (0.204,0.512,1.33)% in Inception-v3, (0.206,0.412,1.86)% in ResNet-50, and (0.420,0.630,2.41)% in EfficientNetV2. On the other hand, benchmarking with The Collective Dataset, the degradation details are (0.646,1.18,2.91)% in AlexNet, (0.532,1.06,2.34)% in VGG-16, (0.530,0.953,1.91)% in Inception-v3, (0.316,0.947,1.89)% in ResNet-50, and (0.433,0.867,2.49)% in EfficientNetV2. Therefore, the investigation reveals the robustness of the proposed algorithms in dynamic light environments.

Table 3.

Results of the proposed method across varying transformation factors , , and .

	(Overall) Accuracy (%)
The volleyball dataset
	Varying	Varying	Varying
AlexNet	95.5	95.1	93.7
VGG-16	96.3	96.0	94.8
Inception-v3	97.4	97.1	96.3
ResNet-50	96.8	96.6	95.2
EfficientNetV2	94.9	94.7	93.0
The collective dataset
	Varying	Varying	Varying
AlexNet	92.3	91.8	90.2
VGG-16	93.4	92.9	91.7
Inception-v3	93.9	93.5	92.6
ResNet-50	94.7	94.1	93.2
EfficientNetV2	91.9	91.5	90

Fig. 6.

Visual comparison of the proposed method across varying transformation factors , , and : (a) The Volleyball Dataset and (b) The Collective Dataset.

Tables 4 and 5 compare the performance between the proposed algorithms and existing methods using The Volleyball Dataset and The Collective Dataset, respectively. There are several observations:

Table 4.

Performance comparison between the proposed algorithms and existing methods using The Volleyball Dataset.

Method	Backbone	Accuracy (%)
A two-branch factor graph network54	Inception-v3	96.7
A coherence-constrained graph with a long short-term memory network55	AlexNet	89.3
A position distribution network and an appearance relation network56	Inception-v3	92.2
A graph long short-term memory network57	Inception-v3	93.0
A spatial-temporal bilinear pooling and a transformer-based context encoding58	VGG-16	94.7
A transformer encoder59	ResNet-18	93.3
A dual attention-based spatial-temporal inference network60	Inception-v3	89.5
F-formation modeling, tri-coupling transformer, and graph neural networks39	VGG-16	96.5
	Inception-v3	97.5
	ResNet-18	95.8
Ours: MIIEGAN and GA-DSC	AlexNet	95.9
	VGG-16	96.7
	Inception-v3	97.6
	ResNet-50	97
	EfficientNetV2	95.3

Table 5.

Performance comparison between the proposed algorithms and existing methods using The Collective Dataset.

Method	Backbone	Accuracy (%)
A two-branch factor graph network54	Inception-v3	94.9
A coherence-constrained graph with a long short-term memory network55	AlexNet	93.0
A position distribution network and an appearance relation network56	Inception-v3	90.3
A graph long short-term memory network57	Inception-v3	94.9
A dual attention-based spatial-temporal inference network60	Inception-v3	86.5
Progressive relation learning61	VGG-16	93.8
Detector-Free62	ResNet-18	90.5
F-formation modeling, tri-coupling transformer, and graph neural networks39	VGG-16	93.0
	Inception-v3	93.2
	ResNet-18	91.7
Clustered spatial-temporal transformer40	Inception-v3	93.4
Ours: MIIEGAN and GA-DSC	AlexNet	92.9
	VGG-16	93.9
	Inception-v3	94.4
	ResNet-50	95
	EfficientNetV2	92.3

• In both benchmark datasets, the proposed method outperforms the existing methods. Considering The Volleyball Dataset, the percentage of improvement is 0.931–9.29% (on average 5.39%). Regarding The Collective Dataset, the percentage of improvement is 0.105–9.83% (on average 3.38%).

• Although the improvement in some works^54,57 may not be too significant, our work takes advantage of dynamic light environment verification, revealing that the model performance can be maintained at a good level in different light conditions.

• Different methods share the same observation that the model performance is better in The Volleyball Dataset compared to The Collective Dataset.

Ablation experiments

The ablation experiments have been conducted to reveal the contributions of the MIIEGAN and GA-DSC to the GAR model. Several famous variants of GAN and convolution techniques were also chosen in the analysis.

Variants of GANs

Five additional types of GANs, including Vanilla GAN (vGAN)⁶³, conditional GAN (cGAN)⁶⁴, information maximizing GAN (InfoGAN)⁶⁵, Wasserstein GAN (WGAN)⁶⁶, and deep convolutional GAN (dcGAN)⁶⁷, were implemented and compared. All five backbones remained in the analysis. Tables 6 and 7 provide a summary of the results of six GANs across five backbones using two benchmark datasets, respectively. In addition, Fig. 7 is presented as a visual comparison of the results in Tables 6 and 7. The observations are:

Table 6.

Performance comparison between six GANs using the Volleyball dataset.

Method	Backbone	Accuracy (%)
vGAN	AlexNet	92.8
	VGG-16	92.9
	Inception-v3	93.1
	ResNet-50	92.6
	EfficientNetV2	92
cGAN	AlexNet	94.4
	VGG-16	95
	Inception-v3	95.2
	ResNet-50	94.4
	EfficientNetV2	93.4
InfoGAN	AlexNet	94.2
	VGG-16	94.4
	Inception-v3	93.5
	ResNet-50	94
	EfficientNetV2	94.1
WGAN	AlexNet	95.1
	VGG-16	95.8
	Inception-v3	96.1
	ResNet-50	95.3
	EfficientNetV2	93.8
dcGAN	AlexNet	93.3
	VGG-16	93.8
	Inception-v3	94
	ResNet-50	93.6
	EfficientNetV2	92.2
MIIEGAN	AlexNet	95.9
	VGG-16	96.7
	Inception-v3	97.6
	ResNet-50	97
	EfficientNetV2	95.3

Table 7.

Performance comparison between six GANs using The Collective Dataset.

Method	Backbone	Accuracy (%)
vGAN	AlexNet	89.2
	VGG-16	90.1
	Inception-v3	90.5
	ResNet-50	90.1
	EfficientNetV2	89.6
cGAN	AlexNet	90.6
	VGG-16	91.4
	Inception-v3	91.8
	ResNet-50	92.2
	EfficientNetV2	90.1
InfoGAN	AlexNet	91.1
	VGG-16	92.7
	Inception-v3	92.6
	ResNet-50	93.3
	EfficientNetV2	90.7
WGAN	AlexNet	91.6
	VGG-16	92.2
	Inception-v3	92.9
	ResNet-50	92.9
	EfficientNetV2	91.5
dcGAN	AlexNet	90.2
	VGG-16	91.9
	Inception-v3	91.1
	ResNet-50	91.3
	EfficientNetV2	91.1
MIIEGAN	AlexNet	92.9
	VGG-16	93.9
	Inception-v3	94.4
	ResNet-50	95
	EfficientNetV2	92.3

Fig. 7.

Visual comparison of the proposed MIIEGAN and other five variants of GANs: (a) The Volleyball Dataset and (b) The Collective Dataset.

• The results demonstrated that MIIEGAN outperforms the other five variants of GAN in both benchmark datasets, confirming the need for a new design of the GAN algorithm, i.e., MIIEGAN.

• The Volleyball Dataset: The ranking, in descending order, using (i) AlexNet is MIIEGAN, WGAN, cGAN, InfoGAN, dcGAN, and vGAN, with the range of accuracy improvement 0.841–3.34% (on average 2.07%); (ii) VGG-16 is MIIEGAN, WGAN, cGAN, InfoGAN, dcGAN, and vGAN, with the range of accuracy improvement 0.939–4.09% (on average 2.47%); (iii) Inception-v3 is MIIEGAN, WGAN, cGAN, dcGAN, InfoGAN, and vGAN, with the range of accuracy improvement 1.56–4.83% (on average 3.43%); (iv) ResNet-50 is MIIEGAN, WGAN, cGAN, InfoGAN, dcGAN, and vGAN, with the range of accuracy improvement 1.78–4.75% (on average 3.22%); and (v) EfficientNetV2 is MIIEGAN, InfoGAN, WGAN, cGAN, dcGAN, and vGAN, with the range of accuracy improvement 1.28–3.59% (on average 2.37%). The overall average improvement is 2.71% across all five backbone deep learning models.

• The Collective Dataset: The ranking, in descending order, using (i) AlexNet is MIIEGAN, WGAN, InfoGAN, cGAN, dcGAN, and vGAN, with the range of accuracy improvement 1.44–4.08% (on average 2.58%); VGG-16 is MIIEGAN, InfoGAN, WGAN, dcGAN, cGAN, and vGAN, with the range of accuracy improvement 1.29–4.22% (on average 2.45%); (iii) Inception-v3 is MIIEGAN, WGAN, InfoGAN, cGAN, dcGAN, and vGAN, with the range of accuracy improvement 1.61–4.31% (on average 2.86%); (iv) ResNet-50 is MIIEGAN, InfoGAN, WGAN, cGAN, dcGAN, and vGAN, with the range of accuracy improvement 1.82–5.44% (on average 3.32%); and (v) EfficientNetV2 is MIIEGAN, WGAN, dcGAN, InfoGAN, cGAN, and vGAN, with the range of accuracy improvement 0.874–3.01% (on average 1.88%). The overall average improvement is 2.62% across all five backbone deep learning models.

The proposed MIIEGAN presents several advantages and merits when compared to traditional approaches. It focuses on improving image quality specifically in low-light and indoor environments, addressing common deficiencies where traditional GANs may struggle. For instance, vGAN and even dcGAN generally produce images that lack clarity in challenging lighting conditions. WGAN is known for its stable training and better convergence properties, but it may not directly address the nuances of image degradation caused by noise. infoGAN can struggle with output variability, but MIIEGAN’s capacity to consider both group and individual inputs allows it to capture a broader representation of the data distribution, leading to more consistent and realistic image generations. VGAN, which simply generates images without leveraging such discriminative features. By improving the conditioning on both group and individual activities, MIIEGAN can produce outputs that better reflect real-world scenarios and variations, providing a level of detail and context that traditional GANs may miss.

Variants of convolutional techniques

The proposed GA-DSC approach has been compared with five variants of convolutional techniques, including DSC, A-DSC, traditional convolution (Conv), dilated convolution (DC)⁶⁸, and transposed convolution (TC)⁶⁹. It is noted that DSC, A-DSC, and GA-DSC belong to the DSC family. Tables 8 and 9 provide a summary of the results of six convolutional techniques across five backbones using two benchmark datasets, respectively. In addition, Fig. 8 is presented as a visual comparison of the results in Tables 8 and 9.

Table 8.

Performance comparison between six convolutional techniques using The Volleyball Dataset.

Method	Backbone	Accuracy (%)
DSC	AlexNet	93.9
	VGG-16	94.3
	Inception-v3	95.1
	ResNet-50	94.8
	EfficientNetV2	93
A-DSC	AlexNet	95.4
	VGG-16	95.6
	Inception-v3	96.8
	ResNet-50	96
	EfficientNetV2	93.9
Conv	AlexNet	93.3
	VGG-16	93.5
	Inception-v3	94.8
	ResNet-50	94.1
	EfficientNetV2	92.5
DC	AlexNet	94.6
	VGG-16	95
	Inception-v3	95.7
	ResNet-50	95.4
	EfficientNetV2	93.4
TC	AlexNet	95.3
	VGG-16	96.2
	Inception-v3	96.2
	ResNet-50	96.4
	EfficientNetV2	94.2
GA-DSC	AlexNet	95.9
	VGG-16	96.7
	Inception-v3	97.6
	ResNet-50	97
	EfficientNetV2	95.3

Table 9.

Performance comparison between six convolutional techniques using The Collective Dataset.

Method	Backbone	Accuracy (%)
DSC	AlexNet	91.1
	VGG-16	91.4
	Inception-v3	92.6
	ResNet-50	93.7
	EfficientNetV2	90.3
A-DSC	AlexNet	91.8
	VGG-16	92.4
	Inception-v3	93.6
	ResNet-50	94.5
	EfficientNetV2	91.2
Conv	AlexNet	90.7
	VGG-16	91
	Inception-v3	91.8
	ResNet-50	92.9
	EfficientNetV2	89.6
DC	AlexNet	91.6
	VGG-16	91.8
	Inception-v3	92.1
	ResNet-50	93.5
	EfficientNetV2	89.8
TC	AlexNet	92.3
	VGG-16	93
	Inception-v3	93
	ResNet-50	94.1
	EfficientNetV2	90.8
GA-DSC	AlexNet	95.9
	VGG-16	96.7
	Inception-v3	97.6
	ResNet-50	97
	EfficientNetV2	95.3

Fig. 8.

Visual comparison of the proposed GA-DSC and the other five variants of convolutional techniques: (a) The Volleyball Dataset and (b) The Collective Dataset.

The analyses are:

• The results demonstrated that GA-DSC outperforms the other five variants of convolutional techniques in both benchmark datasets across five backbone deep learning models, confirming the need for the enhancement of DSC and A-DSC to GA-DSC.

• The Volleyball Dataset: The ranking, in descending order, using (i) AlexNet is GA-DSC, A-DSC, TC, DC, DSC, and Conv, with the range of accuracy improvement 0.524–2.79% (on average 1.49%); (ii) VGG-16 is GA-DSC, TC, A-DSC, DC, DSC, and Conv, with the range of accuracy improvement 0.520–3.42% (on average 1.89%); (iii) Inception-v3 is GA-DSC, A-DSC, TC, DC, DSC, and Conv, with the range of accuracy improvement 0.826–2.95% (on average 1.97%); (iv) ResNet-50 is GA-DSC, TC, A-DSC, DC, DSC, and Conv, with the range of accuracy improvement 0.622–3.08% (on average 1.75%); and (v) EfficientNetV2 is GA-DSC, TC, A-DSC, DC, DSC, and Conv, with the range of accuracy improvement 1.17–3.03% (on average 2.04%). The overall average improvement is 1.91% across all five backbone deep learning models.

• The Collective Dataset: The ranking, in descending order, using (i) AlexNet is GA-DSC, TC, A-DSC, DC, DSC, and Conv, with the range of accuracy improvement 0.646–2.40.646.40% (on average 1.53%); (ii) VGG-16 is GA-DSC, TC, A-DSC, DC, DSC, and Conv, with the range of accuracy improvement 0.968–3.19% (on average 2.16%); (iii) Inception-v3 is GA-DSC, A-DSC, TC, DSC, DC, and Conv, with the range of accuracy improvement 0.854–2.83% (on average 1.93%); (iv) ResNet-50 is GA-DSC, A-DSC, TC, DSC, DC, and Conv, with the range of accuracy improvement 0.529–2.26% (on average 1.35%); and (v) EfficientNetV2 is GA-DSC, A-DSC, TC, DSC, DC, and Conv, with the range of accuracy improvement 1.21–3.01% (on average 2.17%). The overall average improvement is 1.75% across all five backbone deep learning models.

One key benefit of GA-DSC is its ability to effectively balance model complexity reduction and accuracy retention. While A-DSC allows for the integration of multiple convolutional layers, it does not always ensure optimal performance across all layers. GA-DSC addresses this by allowing for a strategic selection of when to employ traditional DSC or standard convolutional layers alongside A-DSC. This mixture of convolutional strategies optimally tailors the architecture to maximize performance without compromising model efficiency. In addition, GA-DSC simplifies the design process by implementing pointwise convolution after two asymmetric depthwise convolutional layers. This approach not only streamlines the overall architecture but also sequentially integrates the functionalities of depthwise and pointwise convolutions, enhancing the operational flow and performance of the model. DC can increase the computational load compared to standard convolutions, especially as the dilation rate increases. This can lead to slower training and inference times. While DC allow for a larger receptive field without increasing the number of parameters, it may result in sparse feature maps, which can lead to loss of important spatial information. On the other hand, TC can be sensitive to the choice of kernel size and stride. If not properly configured, they may produce unexpected results or degrade image quality. Transposed convolutions can be memory-intensive, especially in deeper networks, leading to increased resource requirements. This may limit the size of the batch that can be processed effectively.

Fully supervised and semi-supervised learning strategies

Table 10 summarizes the performance comparison between the proposed model (with fully labelled group activities and unlabelled individual activities), fully-supervised cGAN (with only labelled group activities), and semi-supervised cGAN (with some labelled and unlabelled group activities) to reveal the effectiveness of the semi-supervised learning strategy. The semi-supervised strategy with solely group activities reduces the class size. The results reveal that the proposed MIIEGAN outperformed the fully-supervised cGAN and semi-supervised cGAN, suggesting that introducing unlabelled individual activities data benefits the model enhancement for GAR.

Table 10.

Performance comparison between the proposed model (with fully labelled group activities and unlabelled individual activities), fully-supervised cGAN (with only labelled group activities), and semi-supervised cGAN (with some labelled and unlabelled group activities).

Method	Backbone	Accuracy (%)
The Volleyball Dataset
Fully-supervised cGAN	AlexNet	93.3
	VGG-16	94.1
	Inception-v3	93.8
	ResNet-50	93.1
	EfficientNetV2	92.0
Semi-supervised cGAN	AlexNet	92.6
	VGG-16	93.2
	Inception-v3	93.0
	ResNet-50	92.7
	EfficientNetV2	91.2
MIIEGAN	AlexNet	95.9
	VGG-16	96.7
	Inception-v3	97.6
	ResNet-50	97
	EfficientNetV2	95.3
The Collective Dataset
Fully-supervised cGAN	AlexNet	90.0
	VGG-16	90.6
	Inception-v3	91.1
	ResNet-50	91.3
	EfficientNetV2	89.7
Semi-supervised cGAN	AlexNet	89.5
	VGG-16	90.2
	Inception-v3	90.9
	ResNet-50	91.6
	EfficientNetV2	90.0
MIIEGAN	AlexNet	92.9
	VGG-16	93.9
	Inception-v3	94.4
	ResNet-50	95
	EfficientNetV2	92.3

Testing for significant differences between multiple GAR models

Many models were investigated, analyzed, and compared in Sects. 3 and 4. Statistical tests are needed to test the significant difference in any performance evaluation metric between two models using various non-parametric statistical approaches, including McNemar’s test⁷⁰, DeLong’s test⁷¹, and Wilcoxon signed-rank test⁷². Regarding the performance comparison of multiple models (more than two), these approaches can be repeated to test different combinations of two models. However, the false positive (also known as Type I error) increases with the number of combinations⁷³. Applying to this research study, the more existing works being compared and the more in-depth ablation experiments will experience Type I error using McNemar’s test, DeLong’s test, and the Wilcoxon signed-rank test.

Instead, Friedman’s test has been used to test the significant differences between multiple GAR models. Assume M GAR models and N benchmark datasets. The average rank of the m-th GAR model (), is given by:

11

where is the rank of the n-th value of the metric for the m-th GAR model. Refer to the null hypothesis, -distribution () with the degrees of freedom M-1 forms the foundation of the statistic:

12

The null hypothesis has been rejected. It is suggested that the difference between GAR model performance is significant. A follow-up Wilcoxon signed-rank test has been carried out, the results of which reveal that the proposed model outperforms the existing methods as well as the variants of GANs and convolutional techniques.

Concluding remarks and potential future research directions

A multi-input image-enhanced generative adversarial network (MIIEGAN) has been proposed for generating high-quality additional training images, and a guided asymmetric depthwise separable convolution (GA-DSC) contributed to the model complexity and performance. These algorithms were evaluated in dynamic light environments to investigate their robustness and performance in varying light conditions that meet the nature of GAR in indoor and outdoor environments. Two benchmark datasets (The Volleyball Dataset and The Collective Dataset) were selected for the performance evaluation and comparison. Five backbones (AlexNet, VGG-16, Inception-v3, ResNet-50, and EfficientNetV2) were chosen as deep learning models for analysis. Nine existing methods were compared. The proposed work outperforms existing works by 0.931–9.29% and 1.05–9.83%, respectively. Five variants of GANs and five variants of convolutional techniques were compared in the ablation experiments. The performance evaluation, analysis, comparison, and ablation experiments reveal the effectiveness of the proposed algorithms and their contribution to the development of GAR models.

Although the research team properly designed and formulated the GAR models, there are some key future research directions: (i) combine multiple models using ensemble learning: Combining multiple models using ensemble learning can enhance the robustness and accuracy of GAR systems. This approach leverages the strengths of diverse models, reducing the likelihood of overfitting while improving generalization across varying contexts. Techniques such as bagging, boosting, or stacking could be explored to integrate predictions from different architectures (multi-branch modeling) effectively. By assessing the performance of individual models before integration, researchers can identify the most complementary pairings. Moreover, ensemble methods can be adapted to utilize models trained on both indoor and outdoor datasets, maximizing the use of available data and improving overall prediction reliability; (ii) borrow knowledge between indoor and outdoor GAR datasets via transfer learning: Applying transfer learning techniques to harness knowledge between indoor and outdoor GAR datasets can significantly benefit model performance, especially when one dataset is larger or better annotated than the other. By pre-training models on a comprehensive outdoor dataset, researchers can fine-tune those models on the smaller indoor dataset, leveraging shared features and patterns in activity recognition. This approach not only accelerates model convergence but also mitigates issues related to scarce indoor data. Future research should delve into GANs or domain adaptation methods to refine transfer learning methodologies, ensuring effective application across varying contexts; (iii) investigate strategies to select appropriate backbones: Investigating strategies to select appropriate backbones is crucial for optimizing GAR model performance. The choice of backbone architecture can greatly influence the model’s ability to extract relevant features from data. Future research should focus on evaluating various architectures, such as convolutional neural networks (CNNs) or transformers, by analyzing their performance in both indoor and outdoor settings. Additionally, methods like neural architecture search (NAS) can automate backbone selection, ensuring the chosen framework aligns well with the specific characteristics of the dataset. Comprehensive performance metrics and cross-validation techniques should be employed to guide this selection process; (iv) further evaluate the performance of the proposed method with more benchmark datasets and other emergent GAR dataset categories. Particularly, the impact of data generation across datasets of different sizes, different numbers of group activities, different numbers of individual activities, and other factors can be investigated; and (v) inspired by deep learning approaches for aerial image-based object detection, such as an enhanced graph convolutional remora dual-attentive network⁷⁴ and a pyramid generative adversarial network⁷⁵, the GAR formulations would benefit from the aspect of individual activities, the unlabelled data in this research paper.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

Kwok Tai Chui: Designed and implemented the experiments, analysed the data, and drafted the manuscript.Brij B. Gupta: Analysed the data, and drafted the manuscript.Miguel Torres-Ruiz: Analysed the data, and drafted the manuscript.Varsha Arya: Analysed the data, and drafted the manuscript.All authors reviewed and approved the submission of the manuscript.

Funding

The work described in this paper was fully supported by a grant from Hong Kong Metropolitan University (Ref. no. RD/2023/2.3).

Data availability

The benchmark datasets analysed during the current study are available in: (i) The Volleyball Dataset, [https://github.com/mostafa-saad/deep-activity-rec? tab=readme-ov-file#dataset](https:/github.com/mostafa-saad/deep-activity-rec? tab=readme-ov-file); and (ii) The Collective Dataset, https://cvgl.stanford.edu/projects/collective/collectiveActivity.html.

Declarations

Competing interests

The authors declare no competing interests.