Multimodal and multiscale feature fusion for weakly supervised video anomaly detection

Abstract

Weakly supervised video anomaly detection aims to detect anomalous events with only video-level labels. In the absence of boundary information for anomaly segments, most existing methods rely on multiple instance learning. In these approaches, the predictions for unlabeled video snippets are guided by the classification of labeled untrimmed videos. However, these methods do not account for issues such as video blur and visual occlusion, which can hinder accurate anomaly detection. To address these issues, we propose a novel weakly supervised video anomaly detection method that fuses multimodal and multiscale features. Firstly, RGB and optical flow snippets are input into pre-trained I3D to extract appearance and motion features. Then, we introduce an Attention De-redundancy (AD) module, which employs an attention mechanism to filter out task-irrelevant redundancy in these appearance and motion features. Next, to mitigate the effects of video blurring and visual occlusion, we propose a Multi-scale Feature Learning module. This module captures long-term and short-term temporal dependencies among video snippets to provide global and local guidance for blurred or occluded video snippets. Finally, to effectively utilize the discriminative features of different modalities, we propose an Adaptive Feature Fusion module. This module adaptively fuses appearance and motion features based on their respective feature weights. Extensive experimental results demonstrate that our proposed method outperforms mainstream unsupervised and weakly supervised methods in terms of AUC. Specifically, our proposed method achieves 97.00% AUC and 85.31% AUC on two benchmark datasets, i.e., ShanghaiTech and UCF-Crime, respectively.

Introduction

With the increasing focus on security, surveillance cameras are becoming more prevalent in public areas like shopping malls, banks, and intersections. Surveillance videos primarily aim to detect abnormal events¹, including road accidents and criminal activities. Traditional methods involve personnel monitoring these videos to detect anomalies. While these methods are effective to an extent, they often require substantial human resources. Moreover, fatigue work or fluke mentality of the staff can lead to wrong detection. Therefore, developing intelligent algorithms capable of automatically detecting anomalous events in monitoring videos is crucial for reducing manual labor, alleviating economic losses, and enhancing social security.

Surveillance videos contain various types of anomalous events, such as motion anomalies and appearance anomalies. These anomalies can differ significantly in visual features and temporal patterns, making it challenging for the detection models to generalize across diverse types of anomalies in videos. In response to these issues, prior research^2–4 primarily focused on extracting the most discriminative features from RGB snippets to effectively detect anomalies. However, these approaches solely rely on unimodal RGB data for anomaly detection. RGB data offers detailed appearance and texture information, enabling effective detection of appearance anomalies. However, its performance in detecting motion anomalies can be relatively limited. For instance, in the case of a high-speed vehicle abruptly halting, it is highly probable that this vehicle has encountered a breakdown or collision. Such an event might not exhibit any visible anomalies in terms of appearance (i.e., RGB), yet the corresponding changes in motion (i.e., optical flow) become distinctly evident. Optical flow and RGB represent motion and appearance features, respectively, which allow them to provide complementary information for video anomaly detection. Some researchers proposed multimodal video anomaly detection approaches by fusing the information from two modalities: RGB and optical flow. These two modalities complement each other and effectively mitigate the limitations presented by insufficient information within a single modality, enabling the detection of diverse types of anomalies in videos.

Current multimodal approaches detecting video anomaly are primarily categorized into two distinct classes: unsupervised methods and weakly supervised methods. Unsupervised approaches^5–10 train the model solely on normal training samples, detecting events deviating from the learned normal patterns as anomalies. These methods alleviate the need for artificially labeled data, resulting in significant reduction in labor costs. Nonetheless, these approaches are vulnerable to mistakenly detecting new, unlearned normal events, leading to decreased accuracy in detecting anomalies. Moreover, by concentrating exclusively on acquiring the feature representation of normal events, these methods overlook effective optimization for abnormal event detection.

Weakly supervised methods train the network by using both video-level labeled abnormal and normal videos. These approaches can acquire more discriminative representations between abnormal and normal events, thus improving the detection performance. Weakly supervised methods consist of two main stages: feature extraction and feature fusion. In the feature extraction stage, certain approaches^11,12 employed pre-trained Inflated 3D ConvNet (I3D) or 3D convolution (C3D) networks to extract motion and appearance representations from optical flow and RGB clips. However, aforementioned feature extraction methods suffer from the following limitations: (1) The extracted features involve redundant information. Previous studies detect abnormal events by directly leveraging representations acquired from pre-trained C3D¹³ or I3D¹⁴. However, C3D or I3D are trained for the task of classifying video actions, rather than specifically for video anomaly detection task, leading to inevitable redundancy in the extracted features. (2) These methods do not account for issues such as video blur and visual occlusion. The employed datasets include video scenes captured under low-light conditions, resulting in video blur. The examples of video blur are shown in Fig. 1. Additionally, the datasets contain scenes with visual occlusion. The examples of visual occlusion are shown in Fig. 2. Regarding video snippets that are blurred or occluded, the inability to extract their complete features hinders the detection of abnormal snippets, leading to inferior detection performance. In the feature fusion stage, Wan et al.¹¹ and Dubey et al.¹² integrated motion and appearance features with equal weight and subsequently fed these features into the anomaly detection classifiers. Wei et al.¹⁵ designed a multimodal fusion framework. This framework comprises two unimodal streams, namely RGB and optical flow, along with their fused stream. The feature representation is enhanced by aligning information from these three streams. Subsequently, the improved features from different streams are concatenated. However, the aforementioned feature fusion methods simply integrate appearance and motion features with equal weight, treating these features equally. However, the significance of appearance and motion features may vary for different types of abnormalities. To effectively leverage features from different modalities, it is crucial to allocate diverse weights to the representations of respective modality.

Fig. 1.

The examples of video blur. The parts highlighted by the red lines indicate blurred anomaly regions.

Fig. 2.

The examples of visual occlusion. The parts highlighted by the red lines indicate occluded anomaly regions.

To cope with the above problems in weakly supervised multimodal video anomaly detection, namely redundancy in extracted appearance and motion representations, the neglect of video blur and visual occlusion, and the equal treatment of appearance and motion features, we propose a novel weakly supervised video anomaly detection method that fuses multimodal and multiscale features. In the feature extraction stage, to enhance feature representation, we introduce an Attention De-redundancy (AD) module. This module captures both local and global features specific to each modality and fuses these local and global features with an attention mechanism to exclude redundancy irrelevant to the task in both motion and appearance features. Intuitively, AD can use the global information to determine whether a certain part is task-irrelevant information redundancy. Furthermore, we devise a Multiscale Feature Learning (MFL) module. This module leverages a dilated convolutional network with different dilation rates to obtain short-term dependency among video clips and a self-attention mechanism to acquire long-term dependencies among video clips, effectively addressing challenges associated with video blur and visual occlusion. In the feature fusion stage, to effectively leverage the representations from different modalities, we propose an Adaptive Feature Fusion (AFF) module. This module leverages global information to capture feature weights for different modalities. It assigns higher weights to the features of the more important modality and lower weights to the features of the less significant modality, enabling adaptive fusion of diverse modalities.

The primary contributions of our work are as follows:

1. We propose a novel weakly supervised video anomaly detection method that fuses multimodal and multiscale features. By combining optical flow and RGB modalities, our method effectively addresses the limitations arising from by insufficient information within a single modality, enabling the detection of various types of video anomalies.

2. We devise a MFL module to effectively handle issues related to video blur and visual occlusion. This module employs dilated convolutional networks and a self-attention mechanism to obtain short-term and long-term dependencies among video snippets. Consequently, it provides guidance on global and local information for blurred or occluded video snippets.

3. We design an AFF module to effectively utilize features of different modalities. This module leverages global information to obtain the importance weights for appearance and motion features. By assigning higher weights to the features of significant modality and lower weights to the features of less significant modality, the AFF module adaptively fuses the motion and appearance features.

4. We conduct comprehensive experiments on the UCF-Crime and ShanghaiTech datasets. The experimental results demonstrate that our method surpasses mainstream video anomaly detection algorithms.

Literature review

RGB-based Video anomaly detection methods

Hasan et al.² extracted regular features of normal patterns in long-term videos by employing a Convolutional AutoEncoder (CAE) and detected abnormal events based on reconstruction errors. Lower reconstruction errors characterize normal events, while higher reconstruction errors characterize abnormal events. However, in certain cases, the AutoEncoder (AE) can effectively reconstruct abnormal events, leading to missed detection of abnormal events. Gong et al.³ designed the Memory enhanced deep AutoEncoder (MemAE) to address this problem. The MemAE reconstructed the most similar item retrieved from the Memory module, thereby enhancing the reconstruction error associated with anomalies. Liu et al.⁴ detected video anomaly through predicting future frame. The prediction network utilizes the U-Net network architecture to detect abnormal events by comparing the disparity between the predicted future frame and the actual future frame. Sultani et al.¹⁶ applied C3D to extract features from normal and abnormal snippets, subsequently calculating anomaly scores for each snippet. By imposing the constraint that the maximum anomaly score among the abnormal snippets exceeds that among the normal snippets, an iterative training process yields a multi-instance anomaly ranking model for effective anomaly detection. Tian et al¹⁷ detected video anomaly through feature magnitude learning. They utilized the Inflated 3D ConvNet (I3D) network to extract snippet representations and trained a classifier with the top-k representations scores, distinguishing abnormal videos from normal videos more effectively. Since video-level annotated anomalous videos may contain normal segments, Zhong et al.¹⁸ pre-cleared label noise and employed supervised action classifiers to detect abnormal events. By maximizing the utilization of well-established classifiers, their method achieves effective abnormal behavior detection. Zaheer et al.¹⁹ proposed a normal suppression mechanism to minimize the abnormal score of normal snippets and designed a loss function based on clustering distance to mitigate label noise. To address real-time anomaly detection tasks, Zhang et al.²⁰ encoded adjacent snippets through a Temporal Convolutional Network (TCN). Peng et al.²¹ devised an anomaly detection method consisting of four modules: causal temporal relation, classifier, compactness, and dispersion. They utilized I3D to extract snippet representations. Subsequently, they designed the compactness and dispersion modules. The compactness module ensures intra-class compactness of normal features, while the discreteness module amplifies the discrepancy between normal and abnormal videos. Additionally, the causal temporal relationship module obtains local text features. Finally, the classifier module employs causal convolutions to obtain segment-level anomaly scores. Feng et al.²² designed the Multiple Instance Self-Training (MIST) framework. This framework utilized a generator to produce segment-level pseudo-labels for videos, thereby enabling the supervised training of a feature encoder. However, these aforementioned methods employ unimodal RGB data for anomaly detection. Unimodal data may lack comprehensive information, making it challenging to accurately detect anomalies.

Video anomaly detection methods based on multi-modality

Data from different modalities can enhance each other. Numerous computer vision models^23–31 have employed multimodal data to achieve superior performance. Ngiam et al.³² employed deep AE to learn general representation of multimodal data, which yields promising results in speech and vision tasks. Visual and audio events often co-occur. Video and audio provide distinct information. Afouras et al.³³ fused visual and audio data to address specific tasks. Hong et al.²⁴ employed audio data to assist visual data in localizing video highlight. Some studies on video anomaly detection have fused the information from two modalities: RGB and optical flow. Optical flow represents motion information, while RGB represents appearance information. These modalities complement each other, effectively mitigating the limitations presented by insufficient information within a single modality and advancing the anomaly detection performance. Lonescu et al.³⁴ designed a feature learning framework that utilizes an object-centric convolutional AE to capture both motion and appearance information. Moreover, they proposed a classification method for training sample clusters, using an abnormal event classifier to distinguish each normal cluster from the others. Nguyen et al.³⁵ combined a convolutional AE with the U-Net architecture. They utilized the convolutional AE to reconstruct the appearance information of the frames and the U-Net structure to predict the motion information of the frames. The anomaly scores of these frames were computed by sharing the same encoder. Wan et al.¹¹ designed an Anomaly Regression Net (AR-Net) framework to detect anomaly. They leveraged a pre-trained I3D network to extract motion and appearance representations from optical flow and RGB snippets. These features were subsequently concatenated and input into the anomaly detection classifier. Dubey et al.¹² designed the Deep-network with Multiple Ranking Measures (DMRMs) model to detect abnormal events. They used optical flow and RGB snippets as input and employed a pre-trained C3D network to extract motion and appearance representations. These representations were integrated and input into the anomaly detection classifier. To widen the gap in anomaly scores between anomalous and normal snippets, a multi-ranking metric loss was utilized during the training process. Wei et al.¹⁵ designed an anomaly detection framework called Multimodal Supervise-Attention Enhanced Fusion (MSAF). This framework comprised two unimodal streams, namely RGB and optical flow, along with their integrated stream. The features were enhanced by aligning information from these three streams. Subsequently, the improved features from different streams were concatenated. However, the above methods have several challenges including redundancy in extracted motion and appearance features, the neglect of video blur and visual occlusion, and the equal treatment of motion and appearance features.

Temporal action detection

Temporal action detection focuses on identifying the start and end timestamps as well as action categories. Anomaly detection can be regarded as the coarse temporal action detection. Our focus is on weakly supervised action localization, which closely relates to weakly supervised anomaly detection. They share many similarities, such as using only video-level labels for training and the goal of detecting the start and end timestamps of actions. In recent years, several innovative approaches have been proposed, including MHCS³⁶, TSM³⁷, Densernet³⁸, and NGPR³⁹. However, these previous methods do not adequately address challenges like video blur and visual occlusion.

Method

Figure 3a depicts the architecture of our method. A given pair of normal video $B_n$ and abnormal video $B_a$ is divided into N RGB segments ${\left\{v_i^{\mathit{RGB}}\right\}}_{i=1}^N$ and optical flow segments ${\left\{v_i^{\mathit{Flow}}\right\}}_{i=1}^N$. By using a pre-trained I3D network, motion features ${\left\{f_i^{\mathit{motion}}\right\}}_{i=1}^N$ and appearance features ${\left\{f_i^{\mathit{apperance}}\right\}}_{i=1}^N$ are exacted from optical flow snippets ${\left\{v_i^{\mathit{Flow}}\right\}}_{i=1}^N$ and RGB snippets ${\left\{v_i^{\mathit{RGB}}\right\}}_{i=1}^N$, respectively. After that, the AD module filters out the task-irrelevant information in both appearance features ${\left\{f_i^{\mathit{apperance}}\right\}}_{i=1}^N$ and motion features ${\left\{f_i^{\mathit{motion}}\right\}}_{i=1}^N$. Then, the MFL module obtains the long-term and short-term dependencies among segments to address the challenges of video blurring and visual occlusion. In the feature fusion stage, the AFF module adaptively fuses motion features ${\left\{{\overline{f}}_i^{\mathit{motion}}\right\}}_{i=1}^N$ and appearance features ${\left\{{\overline{f}}_i^{\mathit{apperance}}\right\}}_{i=1}^N$. Then, these fused features are input into fully connected layers to calculate the anomaly scores for each snippets in both abnormal and normal videos, i.e. ${\left\{f\left(v_a^i\right)\right\}}_{i=1}^N$ and ${\left\{f\left(v_n^i\right)\right\}}_{i=1}^N$. To enhance the distance between anomalous snippet scores ${\left\{f\left(v_a^i\right)\right\}}_{i=1}^N$ and normal snippets scores ${\left\{f\left(v_n^i\right)\right\}}_{i=1}^N$, we train the network by utilizing the Multiple Instance Learning (MIL) ranking loss.

Fig. 3.

The architecture of our method. a A given pair of normal and abnormal videos is divided into N RGB segments and optical flow segments. In the feature extraction stage, RGB and optical flow segments are fed into pre-trained I3D to extract appearance and motion features. Subsequently, the AD module filters out task-irrelevant redundancy in these appearance and motion features. Furthermore, the MFL module alleviates challenges such as video blurring and visual occlusion. In the feature fusion stage, the AFF module adaptively fuses appearance and motion features. These fused features are then input into fully connected layers to compute the anomaly scores for each segment in both the normal and anomalous videos. These parts highlighted by the orange and blue dashed lines are our key contributions. b Multi-scale feature learning module leverages a dilated convolutional network and a self-attention mechanism to capture long-term and short-term dependencies among clips. By providing global and local information guidance for snippets with blurred or occluded regions, this module effectively mitigates the effects of video blur and visual occlusion. c Adaptive feature fusion module utilizes global information to calculate the importance weights of the features from different modalities. It assigns higher weights to the features of the more important modality and lower weights to the features of the less significant modality, enabling adaptive fusion of diverse modalities.

Attention de-redundancy module

The optical flow segments ${\left\{v_i^{\mathit{Flow}}\right\}}_{i=1}^N$ and RGB segments ${\left\{v_i^{\mathit{RGB}}\right\}}_{i=1}^N$ are fed into a pre-trained I3D network to acquire motion and appearance representations. Nevertheless, it’s worth noting that the I3D network is pre-trained on the Kinetics dataset, primarily designed for classifying video actions, rather than specifically for detecting video anomalies. Consequently, the extracted motion and appearance representations may inevitably contain redundancies. To address this challenge, the Attention De-redundancy (AD) module is introduced, as shown in Fig. 4. We input the representations of appearance and motion modality into AD module to capture both local and global features specific to each modality. Subsequently, local and global features are fused to extract task-independent redundancy. Finally, we employ an attention mechanism to filter out these redundancy from the motion and appearance features. To facilitate clarity, we utilize the appearance modality as the illustration. The identical method is applicable to the motion modality as well.

Fig. 4.

Attention de-redundancy module. The module contains Global and Local Context units to distinguish the information redundancy and employs an attention mechanism to filter out these redundancy from the motion and appearance features.

Firstly, to acquire global features of the appearance modality, we input the appearance features $F^{\mathit{appearance}}={\left\{f_i^{\mathit{apperance}}\right\}}_{i=1}^N$ into the average pooling layer. Then, a convolutional layer $F^G$ is employed to produce global perceptual descriptor $M_G\in R^{D\times 1}$, which D denotes the feature dimensions.

$$\begin{array}{cc}\hfill X_g=& AvgPool\left(F^{\mathit{appearance}}\right)\hfill \end{array}$$ 1

$$\begin{array}{cc}\hfill M^G=& Conv\left(X_g\right)\hfill \end{array}$$ 2

We utilize the local features of the appearance modality to detect task-independent redundancy within the appearance features. To generate a local perception descriptor $M_L\in R^{D\times T}$, which T denotes the number of snippets, the appearance representations are input into a convolutional layer, formulated as follows:

$$\begin{array}{c}\hfill M^L=Conv\left(F^{\mathit{appearance}}\right)\end{array}$$ 3

The channel descriptor M is obtained by multiplying global perceptual descriptor $M^G$ with local perception descriptor $M^L$. The Sigmoid function is adopted to generate optimized weights of the appearance features $F^{\mathit{appearance}}$. Therefore, we filter out the redundancy through attention mechanism, formulated as follows:

$$\begin{array}{cc}\hfill M=& M^G\otimes M^L\hfill \end{array}$$ 4

$$\begin{array}{cc}\hfill {\overline{F}}^{\mathit{appearance}}=& \sigma (M)\otimes F^{\mathit{appearance}}\hfill \end{array}$$ 5

Where $\otimes$ represents the multiplication operator, $\sigma \left(·\right)$ denotes the Sigmoid function.

Multi-scale feature learning module

The employed datasets include video scenes captured under low-light conditions, resulting in video blur. Additionally, these datasets contain scenes with visual occlusions. Regarding video snippets that are blurred or occluded, the inability to extract their complete features hinders the identification of abnormal snippets, leading to inferior detection performance. To address the challenges posed by video blur and visual occlusion, we propose the Multi-scale Feature Learning (MFL) module, as shown in Fig. 3b. Unlike the multi-scale features extraction methods proposed by Guo et al.^40,41, Yan et al.⁴², Zhang et al.⁴³ and Zhou et al.^44,45, this module leverages a dilated convolutional network and a self-attention mechanism to capture short-term and long-term dependencies among clips. By providing global and local information guidance for snippets with blurred or occluded regions, this module effectively mitigates the effects of video blur and visual occlusion. To facilitate clarity, we utilize the appearance modality as the illustration. The identical method is applicable to the motion modality as well.

The MFL module employs dilated convolutions with dilation rates of 1, 2, and 4 respectively to capture multi-scale temporal dependencies among video clips. Consequently, the MFL module provides local information guidance for snippets with blurred or occluded regions.

To begin with, the optimized appearance features ${\overline{F}}^{\mathit{appearance}}$ are fed into a convolutional layer to compress the number of feature channels, thereby reducing computational requirements. The specific process is formulated as follows:

$$\begin{array}{c}\hfill {\overline{F}}_c^{\mathit{appearance}}=\mathit{Re}LU(Conv({\overline{F}}^{\mathit{appearance}}))\end{array}$$ 6

Where ${\overline{F}}^{\mathit{appearance}}$ is the appearance feature after compression.

Next, to fuse multi-scale local information, the compressed appearance features ${\overline{F}}_c^{\mathit{appearance}}$ are passed through dilated convolutional layer, where dilation rate of this layer is 1. The output of this layer ${\overline{F}}_1^{\mathit{appearance}}$ is then concatenated with its input ${\overline{F}}_c^{\mathit{appearance}}$. The specific processes are formulated as follows:

$$\begin{array}{cc}\hfill {\overline{F}}_1^{\mathit{appearance}}=& \mathit{Re}LU(D1\_Conv({\overline{F}}_c^{\mathit{appearance}}))\hfill \end{array}$$ 7

$$\begin{array}{cc}\hfill \overline{F}_{c,1}^{\mathit{appearance}}=& cat({\overline{F}}_c^{\mathit{appearance}},{\overline{F}}_1^{\mathit{appearance}})\hfill \end{array}$$ 8

Where $D1\_Conv$ is dilated convolutional layers with expansion rates of 1, respectively. ${\overline{F}}_1^{\mathit{appearance}}$ is the output features of the dilated convolutional layers, where expansion rate of this layer is 1.

Similarity, the concatenated result $\overline{F}_{c,1}^{\mathit{appearance}}$ is fed into another dilated convolutional layer $D2\_Conv$, where dilation rate of this layer is 2. Subsequently, the output of this layer ${\overline{F}}_2^{\mathit{appearance}}$ is concatenated with its input, and the concatenated result $\overline{F}_{c,1,2}^{\mathit{appearance}}$ is input into a dilated convolutional layer $D4\_Conv$, where dilation rate of this layer is 4. Finally, to capture multi-scale temporal dependencies, the compressed appearance features are concatenated with the output information obtained from dilated convolutional layers of varying scales. The specific process is as follows:

$$\begin{array}{c}\hfill \overline{F}_{c,1,2,4}^{\mathit{appearance}}=cat({\overline{F}}_c^{\mathit{appearance}},{\overline{F}}_1^{\mathit{appearance}},{\overline{F}}_2^{\mathit{appearance}},{\overline{F}}_4^{\mathit{appearance}})\end{array}$$ 9

Where $\overline{F}_{c,1,2,4}^{\mathit{appearance}}$ represents the multi-scale appearance features.

The MFL module employs a self-attention mechanism to generate a correlation matrix among video clips, thereby capturing global temporal dependencies. Consequently, the MFL module provides global information guidance for video snippets with blurry or occluded regions.

First, the multi-scale features are passed through three separate $1\times 1$ convolutional layers to acquire three feature matrices, namely $A_1$, $A_2$, and $A_3$.

$$\begin{array}{cc}\hfill A_1=& Conv(\overline{F}_{c,1,2,4}^{\mathit{appearance}})\hfill \end{array}$$ 10

$$\begin{array}{cc}\hfill A_2=& Conv(\overline{F}_{c,1,2,4}^{\mathit{appearance}})\hfill \end{array}$$ 11

$$\begin{array}{cc}\hfill A_3=& Conv(\overline{F}_{c,1,2,4}^{\mathit{appearance}})\hfill \end{array}$$ 12

$A_1$, $A_2$, and $A_3$ can be treated as “Query”, “Key” and “Value” in the self-attention module. The attention score is computed by measuring the similarity between Query vectors and Key vectors, determining the relevance of each snippet to others in the sequence. The attention weights are used to perform a weighted sum of the Value vectors, creating global context representations for each video. The specific processes are as follows:

Next, by multiply the feature matrices $A_1$ and $A_2$ element by element, the correlation matrix among different snippets are obtained.

$$\begin{array}{c}\hfill A_F=A_1·A_2\end{array}$$ 13

Where $A_F$ is the correlation matrix.

Finally, the correlation matrix $A_F$ is multiplied element by element with the feature matrix $A_3$ to obtain the global representations. These representations are then input into a convolutional layer. Additionally, the multi-scale features and the global features are summed element-wise. Thus global features are incorporated into the multi-scale appearance features.

$$\begin{array}{c}\hfill \overline{F}_{c,1,2,4,g}^{\mathit{appearance}}=Conv(A_3·A_F)+\overline{F}_{c,1,2,4}^{\mathit{appearance}}\end{array}$$ 14

Where $\overline{F}_{c,1,2,4,g}^{\mathit{appearance}}$ represents the output feature obtained through a residual connection.

Adaptive feature fusion module

For various types of anomalies, the significance of appearance and motion modalities may vary. To effectively leverage the features from different modalities, we propose the Adaptive Feature Fusion (AFF) module, as shown in Fig. 3c. Unlike the feature fusion methods CTL-AFFS⁴⁶ proposed by Shi et al. and MSFF-Net⁴⁷, CFANet⁴⁸ proposed by Zhang et al., this module utilizes global information to calculate the importance weights of the features from different modalities. It assigns higher weights to the features of the more important modality and lower weights to the features of the less significant modality, enabling adaptive fusion of diverse modalities.

Firstly, the appearance features and motion features are passed through global average pooling separately. Subsequently, 1D convolutions are utilized to obtain the global information for both the appearance and motion features, namely $G^{\mathit{appearance}}$ and $G^{\mathit{motion}}$.

$$\begin{array}{cc}\hfill G^{\mathit{appearance}}=& Conv\left(GAP\left(\overline{F}_{c,1,2,4}^{\mathit{appearance}}\right)\right)\hfill \end{array}$$ 15

$$\begin{array}{cc}\hfill G^{\mathit{motion}}=& Conv\left(GAP\left(\overline{F}_{c,1,2,4}^{\mathit{motion}}\right)\right)\hfill \end{array}$$ 16

Next, the appearance vectors $G^{\mathit{appearance}}$ and motion vectors $G^{\mathit{motion}}$ are stacked, as well as the Softmax function is applied to obtain the adaptive weight vectors, ${\omega }_{\mathit{appearance}}$ and ${\omega }_{\mathit{motion}}$, corresponding to the appearance and motion features, respectively.

$$\begin{array}{cc}\hfill {\omega }_{\mathit{appearance}}=& \frac{e_{}^{G_{}^{_{}^{\mathit{appearance}}}}}{e_{}^{G_{}^{_{}^{\mathit{appearance}}}}+e^{\mathit{motion}}}\hfill \end{array}$$ 17

$$\begin{array}{cc}\hfill {\omega }_{\mathit{motion}}=& 1-{\omega }_{\mathit{appearance}}\hfill \end{array}$$ 18

The feature weight vectors are multiplied with the features of corresponding modality element by element. Then, to obtain the fused feature, the weighted appearance and motion features are added together.

$$\begin{array}{c}\hfill F={\omega }_{\mathit{appearance}}\times {\overline{F}}_{c,1,2,4}^{\mathit{appearance}}+{\omega }_{\mathit{motion}}\times {\overline{F}}_{c,1,2,4}^{\mathit{motion}}\end{array}$$ 19

MIL ranking loss

To enhance the distinction between anomaly scores of anomalous snippets and those of normal snippets, the Multiple Instance Learning (MIL) ranking loss is employed to train the model¹⁶. This loss is formulated below:

$$\begin{array}{c}\hfill l\left(B_a,B_n\right)=max\left(0,1-\underset{i\in B_a}{max}f\left(v_a^i\right)+\underset{i\in B_n}{max}f\left(v_n^i\right)\right)\end{array}$$ 20

Where $v_a$ and $v_n$ represent the fused features of the abnormal video and normal video, $\underset{i\in B_a}{max}f\left(v_a^i\right)$ and $\underset{i\in B_n}{max}f\left(v_n^i\right)$ denote the highest scores of snippets in the abnormal and normal videos, respectively.

Given that videos are comprised of sequential segments, the anomaly score ought to exhibit smooth variation between these segments. Furthermore, as abnormal events are infrequent and transient in real-world situations, the abnormal scores of segments within abnormal videos are expected to be highly sparse. Therefore, smoothness item a and sparsity item b are introduced.

$$\begin{array}{cc}\hfill a=& \sum _i^{n-1}{\left(f\left(v_a^i\right)-f\left(v_a^{i+1}\right)\right)}^2\hfill \end{array}$$ 21

$$\begin{array}{cc}\hfill b=& \sum _i^{n}f\left(v_a^i\right)s.t.f\left(v_a^i\right)\ne max\left(f\left(v_a^i\right)\right)\hfill \end{array}$$ 22

To mitigate overfitting during network training, we incorporate a regularization term ${∥W∥}_F$. The ultimate formulation of the loss function is presented below:

$$\begin{array}{c}\hfill L\left(W\right)=l\left(B_a,B_n\right)+{\lambda }_{1}a+{\lambda }_{2}b+{∥W∥}_F\end{array}$$ 23

Where ${\lambda }_1$ denotes the weights of the smoothness item, ${\lambda }_2$ denotes the weights of the sparsity item.

Experiments

Dataset and evaluation metric

Dataset. To verify the efficacy of the proposed method, we perform comprehensive experiments on two benchmark datasets, i.e., ShanghaiTech and UCF-Crime. The ShanghaiTech dataset, originating from stationary-angle street video monitoring, is of medium size. This dataset comprises 437 videos, consisting of 307 labeled as normal and 130 labeled as abnormal. The original dataset⁴ was designed for the task of detecting video anomalies unsupervisedly, where the training set exclusively comprises normal videos. To adapt this dataset for weakly supervised algorithms, Zhong et al¹⁸ restructured this dataset by incorporating a portion of the abnormal test videos into the training data. As a result, both the training and the test sets of the ShanghaiTech dataset contain both normal and anomalous videos. We utilize the same processing approach above to make the original ShanghaiTech dataset suitable for weakly supervised algorithms. The UCF-Crime dataset, an extensive anomaly detection dataset, includes 1900 untrimmed videos totaling 128 hours collected from real-world indoor and street monitoring. Differing from the static scene of the ShanghaiTech dataset, the UCF-Crime dataset contains intricate and varied scenes. It covers 13 types of anomalies, including road accidents, vandalism, shoplifting, and others. The training set of UCF-Crime comprises 810 abnormal videos and 800 normal videos, all annotated with video-level labels. The test set consists of 140 abnormal videos and 150 normal videos, all annotated with frame-level labels.

Evaluation metric. The evaluation metric used for all datasets is the area under the ROC curve (AUC). AUC evaluates the classifier’s ability to classify both positive and negative instances. Especially in case of an imbalance between positive and negative bags, it makes more reasonable evaluation for the classifier. The effectiveness of the classifier improves as the AUC value increases.

Implementation details

We implement and train the proposed method on a single GTX 2080Ti GPU using PyTorch. We re-size each video frame to $240\times 320$ pixels. The optical flow maps are generated utilizing the TV-L1 algorithm⁴⁹. To extract both RGB and optical flow features, we employ the I3D network pre-trained on the Kinetics dataset. For every 16-frame video clip, we compute I3D features and then apply $l_2$ normalization. To obtain the features of a video segment, we average all 16-frame clip features within that segment. The 1024D motion and appearance representations extracted from the pre-trained I3D network are input into the AD module. In the AD module, we use the 3 $\times$ 1 Conv1D for each convolutional layer. Each convolutional layer is followed a ReLU activation function. In the MFL module, we use the 3 $\times$ 1 Conv1D for each dilated convolutional layer. Each convolutional layer is followed a LeakyReLU activation function. For the self-attention block, we use the 1 $\times$ 1 Conv1D. The 2048D motion and appearance representations extracted from the MFL module are input into the AFF module. In the AFF module, we use the 1 $\times$ 1 AdaptiveAvgPool1d for the pooling layer and 3 $\times$ 1 Conv1D for each convolutional layer. The 2048D fusion representations obtained from the AFF module are input into the fully connected layers. The nodes of three fully connected layers depicted in the model are 512, 128 and 1, respectively. A ReLU activation function follows each fully connected layer. Both the weights of the sparsity item and the smoothness item in the MIL ranking loss, namely ${\lambda }_2$ and ${\lambda }_1$, are set to $8\times {10}^{-5}$. The Adam optimizer is employed to train the proposed method. The learning rate is fixed at 0.001, with a weight decay of 0.0005. During training, a batch size of 16 is utilized, and the training process runs for 300 iterations.

Performance comparison with competing approaches

To assess the performance of the proposed approach, we conduct comparisons with competing approaches on ShanghaiTech and UCF-Crime. These approaches include Conv-AE², Stacked-RNN⁶, GCL⁵⁰, GCN¹⁸, CLAWS¹⁹ and MIST²².

Performance comparison with competing approaches on ShanghaiTech

We conduct comparative experiments between the proposed method and competing algorithms on ShanghaiTech, as presented in Table 1.

Table 1.

Performance comparison with competing methods on ShanghaiTech.

Supervision	Method	AUC(%)
Unsupervised	Conv-AE2	60.85
	Stacked-RNN6	68.00
	MNAD8	70.50
	Mem-AE3	71.20
	Frame-Pred4	73.40
	Deng et al.51	75.00
	GCL50	78.93
	FRD52	80.14
Weakly Supervised	Zhang et al.20	82.50
	GCN18	84.44
	Sultani et al.16	85.33
	CLAWS19	89.67
	AR-Net11	91.24
	CLAWS Net+53	91.46
	MSAF15	93.46
	MIST22	94.83
	VPE54	96.88
	Ours	97.00

The best scores are highlighted in bold.

Hasan et al.² used Convolutional Auto-Encoder (Conv-AE) to learn regular characteristics in long-term videos and performed anomaly detection based on reconstruction error. Lower reconstruction errors indicate normal behaviors, while higher reconstruction errors indicate abnormal behaviors. This method achieved an AUC score of 60.85%. Luo et al.⁶ developed the Temporally-coherent Sparse Coding (TSC) to detect abnormal events. This method used the Stacked Recurrent Neural Network (stacked-RNN) to map TSC, and then performed anomaly detection based on prediction error. This method achieved an AUC score of 68.00%. Liu et al.⁴ detected abnormal events by comparing predicted frames with ground truth and achieved an AUC score of 73.40%. Gong et al.³ designed the Mem-AE to detect abnormal events. The Mem-AE reconstructs the most similar item retrieved from the Memory module. This method achieved an AUC score of 71.20%. Park et al.⁸ introduced a Memory module between the decoder and encoder to enhance the model ability which remembers normal information, and achieved an AUC score of 70.50%. Zaheer et al.⁵⁰ developed Generative Cooperative Learning (GCL) to detect video anomaly unsupervisedly. The generator and discriminator perform cross-supervised training in a cooperative manner. The pseudo-label generated by the generator is utilized to calculate the loss of the discriminator, while the pseudo-label generated by the discriminator is utilized to calculate the loss of the generator. This method achieved an AUC score of 78.93%. The above approaches utilize unsupervised video anomaly detection. Nonetheless, these approaches are vulnerable to mistakenly detecting new, unlearned normal events, leading to decreased accuracy in detecting anomalies. Moreover, by concentrating exclusively on acquiring the features of normal events, these methods overlook effective optimization for abnormal event detection. Consequently, the anomaly detection accuracy is low, with the highest achieved AUC value being 78.93%. In contrast, our proposed method achieves a significantly higher AUC value, surpassing the previous value by 18.07%.

Zhang et al.²⁰, GCN¹⁸, Sultani et al.*¹⁶, and CLAWS¹⁹ employ weakly supervised methods jointly train the network by using both video-level labeled abnormal and normal videos. These approaches can acquire more discriminative representations between abnormal and normal events, thus improving the detection performance. Zhong et al¹⁸ proposed GCN to detect abnormal events, achieving an AUC score of 84.44%. Zhang et al.²⁰ proposed a novel intrinsic loss to effectively constrain the function space of the anomaly detection problem, achieving an AUC score of 82.50%. Sultani et al.¹⁶ detected abnormal events by multi-instance learning, obtaining an AUC score of 85.33%. Zaheer et al.¹⁹ proposed a normal suppression mechanism to minimize the abnormal score of normal instances and designed the clustering distance loss to reduce label noise, achieving an AUC score of 89.67%. Feng et al.²² proposed the MIST framework to detect abnormal events, achieving an impressive AUC score of 94.83%.

Some studies on video anomaly detection have fused the information from two modalities: RGB and optical flow. These two modalities complement each other and effectively mitigate the limitations presented by insufficient information within a single modality, consequently enhancing the effectiveness of anomaly detection.

However, the above methods utilize unimodal RGB data for anomaly detection. Unimodal data lacks sufficient information. Relying on a single modality can result in suboptimal detection performance. To overcome this limitation, Wan et al.¹¹ and Wei et al.¹⁵ fused information from both RGB and optical flow modalities. These modalities complement each other and effectively mitigate the limitations presented by insufficient information within a single modality, thus improving the detection performance. Wan et al.¹¹ proposed the AR-Net framework to detect abnormal events, achieving an AUC score of 91.24%. Wei et al.¹⁵ developed an anomaly detection framework called MSAF, yielding an AUC score of 93.46%. However, despite these advancements, several challenges still exist. These challenges include redundancy in extracted optical flow and RGB representations, the neglect of video blur and visual occlusion, and the equal treatment of RGB and optical flow features. Addressing these challenges can further improve the detection performance. We extract appearance representations from RGB clips and motion representations from optical flow clips, respectively. To address the issue of task-independent redundant information in these features, we introduce the AD module. Additionally, we propose the MFL module to mitigate challenges such as video blur and visual occlusion. Moreover, we propose the AFF module to adaptively fuse the representations from diverse modalities. Our method is superior to mainstream approaches, acquiring the best result with an impressive AUC of 97%.

Performance comparison with competing approaches on UCF-Crime

We further conduct comparative experiments between the proposed method and mainstream algorithms on UCF-Crime, as presented in Table 2.

Table 2.

Performance comparison with competing methods on UCF-Crime.

Supervision	Method	AUC(%)
Unsupervised	Conv-AE2	50.60
	Sohrab et al.55	58.50
	Lu et al.56	65.51
	GODS57	70.46
	GCL50	71.04
	FRD52	77.56
	DyAnNet58	79.76
	C2FPL59	80.65
Weakly Supervised	Sultani et al.16	77.92
	Zhang et al.20	78.66
	GCN18	82.12
	MIST22	82.30
	CLAWS19	83.03
	Ullah et al.60	84.00
	CLAWS Net+53	84.16
	RTFM17	84.30
	Thakare et al.61	84.48
	Peng et al.21	84.89
	Ours	85.31

The best scores are highlighted in bold.

Hasan et al.² proposed the Conv-AE algorithm to detect abnormal events, achieving an AUC score of 50.60%. Sohrab et al.⁵⁵ proposed the subspace support vector data description to detect abnormal events, achieving an AUC score of 58.50%. Lu et al.⁵⁶ detected abnormal events by sparse combination learning, yielding an AUC score of 65.51%. Wang et al.⁵⁷ proposed the Generalized One-class Discriminative Subspaces (GODS). This method flexibly constrains a class of data distribution by learning a pair of complementary classifiers, achieving an AUC score of 70.46%. The GCL algorithm, proposed by Zaheer et al.⁵⁰, achieved an AUC score of 71.04%.

The aforementioned methods employ unsupervised anomaly detection, resulting in low AUC score. The GCN algorithm proposed by Zhong et al.¹⁸ obtained an AUC score of 82.12%. The approach devised by Zhang et al.²⁰ achieved an AUC score of 78.66%. The approach designed by Sultani et al.¹⁶ achieved an AUC score of 77.92%. The MIST devised by Feng et al.²² achieved an AUC score of 82.30%. The approach proposed by Thakare et al.⁶¹ achieved an impressive AUC score of 84.48%. Tian et al.¹⁷ devised the feature magnitude learning (RTFM) to detect abnormal events, achieving an AUC score of 84.30%. Peng et al.²¹ devised an anomaly detection method consisting of four modules: causal temporal relation, classifier, compactness, and dispersion, achieving an AUC score of 84.89%. Zaheer et al.¹⁹ proposed clustering-assisted weakly supervised (CLAWS) learning. This method designed a normal suppression mechanism to minimize the abnormal score of normal snippets and a loss function based on clustering distance to mitigate label noise, achieving an AUC score of 83.03%.

The aforementioned weakly supervised methods enhance detection performance in contrast to unsupervised methods. However, these methods rely solely on unimodal RGB data for anomaly detection, leaving room for further improvement in detection performance. In contrast, our proposed method outperforms mainstream methods, attaining superior results with an impressive AUC score of 85.31%.

Sample performance evaluation

To evaluate the sample performance of our proposed approach, we train the model with varying numbers of abnormal videos on ShanghaiTech. Specifically, we decrease the number of anomalous training videos from 63 to 25, while keeping the number of the test set and normal training videos unchanged. We compare our proposed method with Sultani et al.¹⁶, presenting the results in Fig. 5.

Fig. 5.

The detection results in different numbers of abnormal videos.

Figure 5 illustrates that as the number of anomalous training videos decreases, the AUC scores of both Sultani et al.¹⁶ and our proposed method also decrease. The method of Sultani et al, which uses all abnormal training videos to train the model, achieves an AUC score of 87.03%. Our proposed method, which only uses 25 abnormal training videos to train the model, achieves an AUC score of 87.46%. Despite using fewer abnormal training videos, the proposed method can effectively utilize training data and acquire superior detection performance. This is due to our method’s superior recognition of positive instances in abnormal videos, allowing it to utilize the same training data more effectively than the approach proposed by Sultani et al.

Subtle anomaly discriminability

To evaluate the validity of the proposed method in detecting each anomaly class, the AUC score is calculated for individual anomaly class on UCF-Crime. Additionally, we compare our proposed method with Sultani et al.¹⁶, presenting the results in Fig. 6.

Fig. 6.

The detection results in each individual anomaly class.

Figure 6 illustrates that the proposed method outperforms Sultani et al. in detecting 10 classes of anomalies, namely abuse, arson, assault, road accidents, explosion, fighting, robbery, shooting, stealing, vandalism and shoplifting. However, our proposed method underperforms Sultani et al. in detecting two types of anomalies, namely arrest and burglary. The proposed method significantly improves the accuracy of detecting multiple types of anomalies in complex scenarios by integrating cross-modal feature fusion, multi-scale feature learning, and attention de-redundancy. This approach effectively captures both long-term and short-term temporal dependencies while fully leveraging complementary information from multi-modal data. Nevertheless, the proposed method exhibits distinguished performance in detecting individual anomaly classes.

Ablation studies

Effectiveness of proposed modules. To evaluate the contribution of each module in our method, we conduct ablation experiments on the two anomaly benchmarks, i.e., UCF-Crime and ShanghaiTech datasets. We design some comparison experiments. (1) The baseline is the approach devised by Sultani et al.¹⁶. This approach adopts the I3D network to capture representations from RGB snippets and subsequently calculates anomaly scores by passing these features through three fully connected layers. (2) We incorporate the optical flow snippets in addition to the RGB snippets. (3) Building upon experiment (2), we add the AFF, AD or MFL module. (4) Building upon experiment (2), we add the AFF and AD modules. (5) Building upon experiment (2), we add the AFF, AD and MFL modules. The results of these five comparison experiments are presented in Table 3.

Table 3.

Contribution of each module in our method.

Number	Baseline(RGB)	Optical flow	AFF	AD	MFL	AUC(%) ShanghaiTech	AUC(%) UCF
1	$✓$					85.33	77.92
2	$✓$	$✓$				88.10	81.1
3	$✓$	$✓$	$✓$			91.49	82.56
4	$✓$	$✓$		$✓$		90.51	81.69
5	$✓$	$✓$			$✓$	92.29	83.05
6	$✓$	$✓$	$✓$	$✓$		93.88	83.73
7	$✓$	$✓$	$✓$	$✓$	$✓$	97.00	85.31

The best scores are highlighted in bold.

As shown in Table 3, when the baseline incorporates optical flow snippets in addition to the RGB snippets, there is a notable increase in the AUC score. Specifically, the AUC scores increase by 2.77% and 3.18% on ShanghaiTech and UCF-Crime, respectively. This set of experiments validates the effectiveness of combining RGB snippets and optical flow snippets. These two modalities complement each other and effectively mitigate the limitations presented by insufficient information within a single modality, consequently enhancing detection effectiveness. Based on Experiment (2), the addition of AFF, AD or MFL module yields further enhancement in the AUC score. Specifically, the AUC scores increase by 3.39%, 2.41%, 4.19% on ShanghaiTech, as well as 1.46%, 0.59%, 1.95% on UCF. These experiments validate the effectiveness of the proposed AFF, AD and MFL modules. Additionally, combining AD with AFF yields further enhancement in the AUC score. Specifically, the AUC scores increase by 2.39% and 1.17%, respectively. This set of experiments validates the efficacy of the proposed AD module. This module can filter out task-independent redundancy in both optical flow and RGB representations. Based on this, the addition of the MFL module yields further enhancement in the AUC score. Specifically, the AUC scores increase by 3.12% and 1.58%, respectively. This set of experiments validates the efficacy of the proposed MFL module. This module can obtain long-term and short-term dependencies among segments to effectively mitigate issues of video blur and visual occlusion. The above experimental results demonstrate that the proposed modules effectively enhance detection performance.

Effectiveness of dual branches. Table 4 illustrates the impact of applying the AD and MFL modules to branches with RGB and optical flow inputs. We observe the following: 1) Adding the AD module to either the RGB or optical flow branch yields similar results, with AUC scores of 89.76%, 89.69% on ShanghaiTech, as well as 91.14%, 91.23% on UCF-Crime, respectively. Likewise, the addition of the MFL module shows similar effectiveness. 2) When adding the AD or MFL module to both branches simultaneously, these modules achieve the best performance. This demonstrates that all the two branches have irreplaceable roles, and adding the AD or MFL module to both branches simultaneously can further boost the performance.

Table 4.

Effectiveness of dual branches.

Number	AD	MFL	RGB	Optical flow	AUC(%) ShanghaiTech	AUC(%) UCF
1	$✓$		$✓$		89.76	81.32
2	$✓$			$✓$	89.69	81.29
3	$✓$		$✓$	$✓$	90.51	81.69
4		$✓$	$✓$		91.14	81.89
5		$✓$		$✓$	91.23	81.95
6		$✓$	$✓$	$✓$	92.29	83.05

The best scores are highlighted in bold.

Compare with other redundancy reduction methods. To demonstrate that our proposed AD module is more effective for weakly supervised video anomaly detection compared to other redundancy reduction methods, we conduct the comparative analysis with approaches like BSP, SGA, and STGA, as shown in Table 5. BSP primarily reduces redundant information by utilizing knowledge transferred from a teacher model. SGA enhances the learning of task-specific representations by optimizing attention map generation through pseudo-label supervision. STGA, which employs a spatial correlation graph and a temporal dependence graph to eliminate redundancy, shows improvement over earlier methods. However, our method still outperforms STGA. The results presented in Table 5 confirm that our AD module achieves the best performance compared with other redundancy reduction methods.

Table 5.

Compare with other redundancy reduction methods.

Method	BSP62	SGA63	STGA64	AD
AUC(%) ShanghaiTech	87.23	89.06	89.68	90.51
AUC(%) UCF	80.23	80.90	81.02	81.69

The best scores are highlighted in bold.

Hyperparameter analysis. We perform experiments on the ShanghaiTech dataset to analyze the impact of various parameters on the overall performance of our model. Our model includes three key parameters: 1) the number of fully connected layers L; 2) sample number k for Top-k selection; and 3) the loss weights ${\lambda }_1$ and ${\lambda }_2$. Specifically, L controls the depth of the fully connected layer, while k determines the number of samples used to train the proposed model in a bag. The weights ${\lambda }_1$ and ${\lambda }_2$ balance the smoothness and sparsity terms. As illustrated in Fig. 7a, our method can achieve the optimal detection performance on the ShanghaiTech dataset when L is set to 3. Similarly, Fig. 7b shows that our method can obtain the best performance on ShanghaiTech when k = 3. Moreover, as depicted in Fig. 7c and d, our method achieves the optimal detection performance on ShanghaiTech when ${\lambda }_1$ and ${\lambda }_2$ are both set to $8\times {10}^{-5}$.

Fig. 7.

Influences of the hyperparameters. a Layer number, b Sample number, c Weight of loss ${\lambda }_1$, d Weight of loss ${\lambda }_2$.

Generalizability analysis. To validate the generalization capability of the proposed approach, we perform domain generalization analysis on the ShanghaiTech and UCF-Crime datasets. Specifically, we train the model on the source domain and assess the performance of this model on the target domain. We utilize ShanghaiTech and UCF-Crime datasets in turn as the source and the target domains. The performance comparisons with the above cross-domain settings are presented in Table 6. The following conclusions are obtained from Table 6. 1) The effectiveness of current methods lowers significantly in the cross-domain settings. 2) Our method has strong cross-domain generalization capabilities. In ShanghaiTech->UCF-Crime and UCF-Crime->ShanghaiTech cross-domain settings, Sultani et al. achieves 33.30% and 42.49% AUC accuracies, respectively. Our DSC method outperforms Sultani et al. by 7.44% and 6.14% on AUC accuracy. The effectiveness gain of our method in the cross-domain setting demonstrates that cross-modal feature fusion, multi-scale feature learning, and attention de-redundancy enhance the effectiveness of video anomaly detection on unseen domains.

Table 6.

Experimental results on domain generalization task.

Method	ShanghaiTech->UCF	UCF->ShanghaiTech
AUC(%) Sultani et al.	33.30	42.49
AUC(%) Ours	40.74	48.63

The best scores are highlighted in bold.

Complexity analysis

The entire model can run at 44 FPS with an input frame resolution of $240\times 320$. Specifically, the MFL module, extracting local and global features, runs at 45 FPS. Following feature extraction, predicting anomaly scores takes just 1.81 ms, equivalent to a speed of 550 FPS. Overall, our model is suitable for online applications.

Qualitative results

To assess the performance of the proposed approach, we compare qualitative performance with four baselines on Shanghai and UCF-Crime by using qualitative descriptions. The qualitative performance is shown in Figs. 8 and 9.

Fig. 8.

Qualitative performance of our method and another three baselines on Shanghai. a Normal, b Skateboard, c Bicycle, d Run, e Car, f Motorcycle. Additionally, “GT” represents the ground truth at the frame-level. “w/o MFL” indicates a baseline without the Multi-scale Feature Learning module compared to our method, “w/o MFL &AD” indicates a baseline without a Multi-scale Feature Learning module and an Attention De-redundancy module compared to our method, and “w/o MFL &AD &AFF” indicates a baseline without a Multi-scale Feature Learning module, an Attention De-redundancy module and an Adaptive Feature module compared to our method, meaning that only optical flow is incorporated. Best viewed in high resolution.

Fig. 9.

Qualitative performance of our method and another three baselines on UCF-Crime. a Normal, b Abuse, c Arrest, d Vandalism, e Explosion, f Robbery.

Specifically, Figs. 8a and 9a represent the visualization performance of normal videos. Our method generates low abnormal scores for normal videos, almost approaching 0. Figs. 8b-f and 9b-f respectively represent the qualitative results for various types of video anomalies. An effective algorithm would assign higher scores to abnormal segments and lower scores to normal segments. It can be seen from Figs. 8b-e and 9b-e that our method generates high abnormal scores when the abnormal events occur, while generating low abnormal scores during normal states. The proposed method accurately localizes the time periods in which the abnormal events occur. The appearance of the explosive event in Fig. 9e changes significantly. The motion of the robbery event in Fig. 9c and the running event in Fig. 8d change significantly. The appearance and motion of the abnormal events in Fig. 8b, c, and e change significantly. These qualitative results demonstrate that fusing RGB and optical flow modalities effectively compensates for the limitations of a single modality. Additionally, the proposed AFF module adaptively fuses RGB and optical flow features to effectively utilize discriminative features, enabling precise detection of various types of video anomalies. Fig. 9b shows a scenario with visual occlusion, while Fig. 9d depicts video blurring caused by low lighting conditions during nighttime capture. The qualitative performance illustrates the efficacy of the proposed MFL module. This module effectively captures the long-term and short-term dependencies among clips to provide global and local information guidance for video clips containing blurred or occluded areas. Thus, the MFL module effectively mitigates challenges associated with video blurring and visual occlusion. The experimental results obtained across various scenes present the superior detection capability of the proposed method. Figs. 8f and 9f illustrate examples of failures due to the lack of guidance from high-level semantic information such as text. In the future, we will focus on incorporating text information to enhance the accuracy and robustness of anomaly.

Conclusion

We propose a weakly supervised video anomaly detection method that fuses multimodal and multiscale features. Firstly, we introduce an AD module to reduce the task-irrelevant redundancy in appearance and motion features extracted from pre-trained I3D. After that, we propose a MFL module. This module incorporates a dilated convolutional network and a self-attention mechanism to capture long-term and short-term temporal dependencies among video clips, effectively addressing challenges associated with video blur and visual occlusion. Finally, to effectively leverage the features from different modalities, we propose an AFF module. This module can obtain feature weights for different modalities to adaptively fuse appearance and motion features. Comprehensive experimental results demonstrate that our approach surpasses mainstream methods on ShanghaiTech and UCF-Crime. In the future, we will consider incorporating semantic information and active learning to improve video anomaly detection efficiency and accuracy.

Author contributions

L. C. contributed to conceptualization, methodology, writing - review & editing. W. S. contributed to methodology, software, writing - original draft, investigation. Y. G. contributed to formal analysis, investigation, Data curation. K. D. contributed to writing - review & editing, supervision.

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.