Overview of deep learning based pedestrian attribute recognition and re-identification

Abstract

Pedestrian attribute recognition (PAR) and re-identification (ReID) are important works in the area of computer vision, which are widely used in intelligent surveillance and are of great significance to the creation of smart life. The purpose of this article is to focus on organizing a review of ReID based on deep learning and analyze the associations between PAR and ReID. Firstly, we summarize the major ideas of Attribute-Assisted ReID and compare the differences in datasets and algorithmic concerns between the two areas. Secondly, we introduce a wide range of representative ReID methods. By analyzing some cutting-edge researches, we summarize their specific network structure, loss function design, and effective training tricks. Reference methods and solutions are provided for the main challenges of ReID, such as cloth-changing, domain adaptation, occlusion condition, resolution changes, etc. Finally, we conclude the performance and characteristics of the SOTA methods, obtain inspiration and prospects for future research directions, and demonstrate the effectiveness of Attribute-Assisted ReID.

Computer vision; Pedestrian attribute recognition; Person re-identification

1. Introduction

Pedestrian re-identification can be considered as a research area for image retrieval, which uses computer vision to judge whether the target pedestrian is present in specified image or in consecutive frames. Given a monitored pedestrian image, the purpose is to retrieve the pedestrian image across devices. Pedestrian re-recognition technology can effectively use multiple camera views, often complementing pedestrian detection and pedestrian tracking, which is of great importance in the fields of intelligent surveillance and security.

General pedestrian re-identification can be divided into four steps, as shown in Fig. 1, including pedestrian detection, feature extraction, similarity measurement and image retrieval respectively. The input of pedestrian re-recognition is a specific pedestrian image called probe. Feature extraction is to learn the diverse features of pedestrians on different cameras. Metric learning is to map the features to a new space, so that people with similar features are clustered together. Image retrieval is to sort and reranking pictures according to the similarity distance between features.

Figure 1.

System composition of pedestrian recognition and similarity ranking method.

The traditional algorithm of pedestrian ReID is mainly based on manual design features combined with distance measurement. The research of feature extraction mainly focuses on obtaining robust and discriminative low-level visual features, including color features or histograms in RGB and HSV color spaces. [1], [2], texture features (Gabor) [3] Shape features (HOG features) [4] and key points (SIFT) [5] and so on. Metric learning finds feature mappings in different spaces, so that the feature vector of the same kind of sample is closer than the feature vector of the non-same kind of sample. Commonly used metric learning methods include Mahalanobis distance [1], local adaptive decision function [6], and saliency weighted metric learning [2]. Due to the pedestrian re-recognition problem, the scene (background, posture, etc.) changes are complex and other challenges.

The above-mentioned traditional low-level visual feature extraction methods are difficult to obtain effective and invariant features, which directly affect the accuracy and effectiveness of metric learning methods. In this paper, we mainly explore pedestrian ReID based on deep learning, which use more advanced algorithms to extract discriminative features. The preprint [7] is available at SSRN.

2. Association between APR and ReID

2.1. Attribute-Assisted re-identification

From the perspective of the composition of the ReID system, the realization of ReID requires feature extraction and then similarity measurement, attribute recognition also needs to extract pedestrians features. Attribute recognition network usually uses CNN to extract the features. Generally, the output of the model is consistent with the number of attributes needed to be recognized. The output of pedestrian ReID is generally not directly equal to the extracted features. Most methods use attribute to assist re-recognition [8], [9], [10], [11], [12] as shown in Fig. 2. Yin et al. [13] firstly presented extensive investigation on the attribute-image ReID problem under an adversarial framework.

Figure 2.

Attributes are consistent with human recognition mechanism for identifying people. [11].

One application of ReID is finding a suspect of interest in a criminal investigation. Witnesses often indicate personal traits of the suspect as seen at the moment when the crime was committed, including hair color, clothing type, etc. Based on that description, the police manually scan the entire video archive looking for a person with similar characteristics. Since people are less likely to change clothes or shave when walking in two different places in a short period of time, so attributes can be used for tracking and communication between two related cameras to achieve re-identification.

Human attributes usually stand as robust visual properties when condition changes and have been investigated in many works. This part analyzes several methods that leverage features from human attributes for person ReID. The algorithms are summarized in Fig. 4

Figure 4.

Attribute Assisted Re-Identification Algorithm.

2.1.1. Attribute attention mechanism

Attention models have been used in fine-grained recognition to crop the key parts. In Comparative Attention Network (CAN) [14], attention mechanism was used in person re-identification by integrating a recurrent attention with the Siamese model. Attribute-Aware Attention Model ($A^{3}M$) [8] explored attention mechanism to select better attribute features. To generate attentions for global category features fusion and local attribute features fusion simultaneously, it consists of two components: attribute-guided attention module and category-guided attention module. The final feature is the addition of the weighted category feature and attribute feature $f^{final}=f^{region}+f^{attr}$.

2.1.2. Attribute feature fusion

Layne et al. [9] proposed Attribute Interpreted ReID (AIR) which Assigned weights to semantic attributes to serve as a representation of pedestrian features. The model learned an attribute-centric, parts-based feature representation, by selecting those attributes that are most effective for re-identification, the attribute-level information and standard low-level features are fused. Attributes-aided Part Detection and Refinement model (APDR) [11] designed an effective attribute fusion network, extracted local attribute features to handle the body part misalignment problem, which is another major challenge for ReID. Experiments show that the learned local features, along with holistic image-level feature, can further improve the accuracy on ReID task. In Fig. 3, Fariborz et al. [15] used a tensor to non-linearly fuse identity and attribute features, and then forced the parameters of the tensor in the loss function to generate discriminative fused features for ReID. Lin et al. [16] proposed the ARM module to calculate the weighting of M individual attributes, which effectively complements the global features for ID loss.

Figure 3.

Attribute based Tensor fusion diagram for Person ReID [15].

2.1.3. Attribute search / zero-shot ReID

Attributes provide a low-dimensional mid-level representation which variability in appearance. Although individual attributes vary in robustness and informativeness, they provide a strong cue for identity. Semantic attributes could potentially be used to constrain or permute a search for a particular person, for example by specifying invariance to whether or not they have removed or added a hat. Given an attribute description in the form of a binary vector, a person can be found by NN matching against the attribute profiles $A(x_i)$ of each person i in the dataset [10]. In the $short-term$ $recognition$, Parts-and-Attributes Search Framework (PASF) [17] distinguished people based on parsing the partial human body attributes, visual attributes include hair type (beards, mustaches, absence of facial hair), type of eye-wear (sunglasses, eyeglasses, absence of glasses), hair type (baldness, hair, wearing a hat), and clothing color. Face detection result was matched to its neighboring regions based on correlation, each person who leaves the scene is assigned a unique identifier (“track ID”). Lampert et al. [18] collected judgments on the “relative strength of association” between 85 attributes and 48 animal classes, showed that description of high-level attributes makes sense for person ReID on untrained images. Optimized Attribute Re-identification (OAR) [19] showed mid-level attributes can be an effective representation for ReID and zero-shot identification.

2.1.4. Attribute fine-tuning

Su et al. [12] proposed SSDAL algorithm (Semi-supervised Deep Attribute Learning) and further extended [20]. Firstly train a deep Convolutional Neural Network on an independent dataset $T=t_1,t_2,...,t_N$, in which each sample is labeled with a binary attribute label. Then it was fine-tuned on the dataset with person ID labels as $U=u_1,u_2,...,t_M$ using triplet loss, where N, M represent sample size. The third stage fine-tuned the model using sigmoid cross entropy loss on the dataset T&U. Experiments showed that fine-tuning the model trained on PAR dataset acquires more discriminative power for ReID task. Attribute-complementary re-id net (ACRN) [21] included automatically predicted attribute features into the training stage of the model. This allowed the CNN to focus on learning information for ReID which is complementary contained in the attribute predictions. Tetsu [22] et al. improved the CNN features by conducting a fine-tuning on a pedestrian attribute dataset, proposed a loss function for classifying combination attributes to increase discriminative ability.

2.1.5. Attribute embedding / space mapping

Attribute-Consistent Matching (ACM) [23] projected the input image into a lower dimensional subspace where the matches exhibit consistency to changing illumination. Su et al. [24] studied the tracklet-to-tracklet identification, proposed a novel discriminative model to exploit low-level features and map attributes to a discriminative latent prototypes space (LPSM). MTL-LORAE framework [25] used semantic attributes in addition to low-level features, and transforms the attributes in low rank space. Adaptive Semantic Margin Regularizer (ASMR) [26] conducted person search by finding images that are closest to the query in a co-embedding space of person images and categories. Wang et al. [27] introduced TJ-AIDL and IIA Space for achieving the knowledge fusion learning on attribute and ID labels.

2.1.6. Clothing attributes assisted

Li et al. [28] studied clothing attributes assisted person reidentification, presented latent support vector machine (c-LSVM)-based person ReID approach to describes the integration of low-level, mid-level and high-level attributes, specifically, it learns the relationship between body parts, clothing information and ID labels. Yang et al. [29] presented a discriminative latent model (DLM) for joint modeling of object classes and their visual attributes. Wu et al. [30] suggested the usefulness of clothing color in surveillance scenarios, and evaluated at the pixel level and frame level respectively. Dual Attribute-aware Ranking Network (DARN) [31] could retrieve clothing items that have the same or similar attributes as a given clothing image.

2.2. Differences between attribute recognition and ReID

2.2.1. Label annotation

Person ReID and attribute recognition usually use different datasets separately to train the network, which have different annotations for the dataset labels. Datasets with pedestrian attribute tags such as the Market-1501 [32] can be used universally. What needs to be paid attention to is that the specific usage may be different for the same dataset. Specifically, PAR focuses on the correspondence between images and tags, while pedestrian ReID focuses on personal identity information, in particular, divide the train and test based on ID.

2.2.2. Image feature

Many ReID algorithms combine local and global features such as PIE [33], PDC [34], GLAD [35], or merge local features such as Spindle Net [36]. These local features may be extracted through local blocks or according to methods such as posture key points, and not necessarily attribute features. Typically, the APR [16] network combined attribute loss and ID loss, which is also a typical example of Attribute-Assisted ReID.

2.2.3. Main focus of the model

Pedestrian re-identification is to find the image of the target pedestrian in the galley data, so the model focuses on the similarity arrangement of the recognition results and the Average Precision of each query. The commonly used evaluation metrics mainly include CMC and mAP.

Attribute recognition aims to identify the attribute characteristics of pedestrian images as accurately as possible, so it focuses on the difference between the true label and the predicted value evaluate the model. Accuracy, precision, recall and f1 score are commonly used evaluation indicators.

In summary, Table 1, Table 2 shows the connection and difference between the two research fields of attribute recognition and pedestrian re-identification.

Table 1.

Comparisons of datasets and evaluations between PAR and ReID.

Comparison	ReID	PAR
Datasets	Market-1501 [32]	PETA [37]
	Duke-MTMC [38]	PA-100K [39]
	CUHK03 [40]	Market-1501-Attribute [16]
Evaluation	CMC	Precision
	mAP	Recall

Table 2.

Comparisons of labels and focus between PAR and ReID.

Comparison	ReID	PAR
labels	pedestrian ID	pedestrian attributes
Focus	arrangement of the recognition results	difference between true labels and predicted value

3. Datasets and evaluation

3.1. ReID datasets

Pedestrian re-identification datasets can be divided into image-based and video-based datasets according to whether it is a single frame or a sequence as shown in Table 5, Table 6, respectively. The former category has Market-1501 [32], CUHK03 [40], DukeMTMC-reID [38], MSMT-17 [41], VIPeR [42], MARS [43], etc. Typical of the latter are iLIDS [44], Duke-video [45], LPW [46], etc. Besides, there are other scene-based datasets, such as PRW [47]. Table 3 and Table 4 provide the examples and statistical summaries of these datasets.

Table 5.

Summary of Image-based ReID Datasets.

Dataset	Time	IDs	Cams	Images	Label method	Size
VIPeR [42]	2007	632	2	1264	Hand	128 × 48
GRID [48]	2009	1025	8	1275	Hand	Vary
CUHK01 [49]	2012	971	2	3884	Hand	160 × 60
CUHK02 [50]	2013	1816	2	7264	Hand	160 × 60
CUHK03 [40]	2014	1360	2	13164	DPM [51]	Vary
RAiD [52]	2014	43	4	6920	Hand	128 × 64
Market1501 [32]	2015	1501	6	32668	DPM	128 × 64
DukeMTMC [38]	2017	1812	8	36441	Hand	Vary
PKU-Reid [53]	2016	114	2	1824	Hand	128 × 64
MARS [43]	2016	1261	6	20715	DPM [51]+GMMCP [54]	256 × 128
Airport [55]	2017	9651	6	39902	ACF [56]	128 × 64
MSMT17 [41]	2018	4101	15	126441	Faster RCNN	Vary
PRW [41]	2016	932	6	34304	Hand	vary

Table 6.

Summary of Video-based ReID Datasets.

Dataset	Time	IDs	Cams	Traklets (BBox)	Label source
DukeMTMC4ReID [57]	2017	1852	8	-(46k)	Doppia [58]
Duke-Video [45]	2018	1812	8	4832(-)	Hand
LPW [46]	2018	2,731	4	7694(590k)	Detector + Hand
Ilids-VID [59]	2014	300	2	600(44k)	Hand
LS-VID [60]	2019	3772	15	14943(3M)	Faster RCNN

Table 3.

Sample of Pedestrian Image Attribute Annotation of DukeMTMC-Attribute and Market-1501-Attribute.

DukeMTMC-Attribute		Market-1501-Attribute
attribute	label	attribute	label
top	1	hair	2
backpack	2	up	2
bag	1	down	2
handbag	1	clothes	1
boots	2	hat	1
shoes	1	backpack	1
upblack	2	bag	1
downblue	2	handbag	2
		age	2
		downred	2

Table 4.

Summary of PAR Datasets.

Dataset	Pedestrians	Attribute	Source
PETA [37]	19000	61 binary and 4 multi-class	outdoor + indoor
RAP	41585	69 binary and 3 multi-class	indoor
RAP2.0	84928	69 binary and 3 multi-class	indoor
PA-100K [39]	10w	26 binary	outdoor
Market1501-Attribute [16]	32668	26 binary and 1 multi-class	outdoor
DukeMTMC-Attribute [16]	34183	23 binary	outdoor

As for the division of datasets, in the research of pedestrian re-identification, the datasets are usually obtained through manual annotation or detection algorithm. The pictures of pedestrians will be divided into training sets and validation sets, as well as query and gallery. After training, the model calculates the similarity for the extracted features in the query and the gallery separately. Generally, in the process of training and testing, the identity of the pedestrian is not repeatedly selected.

3.2. Attribute datasets

Compared with re-recognition datasets, attribute recognition focuses more on marking the appearance characteristics of pedestrians, such as gender, age, clothing, accessories, etc. Table 4 summarizes several commonly used datasets for attribute recognition. Fig. 5 concludes the annotation information of several APR datasets. Generally the datasets are labeled with arabic numerals, PETA [37] and PA-100K [39] using 1 and 0, existing attributes are represented by 1, labels in Market-1501 Attribute [16] and DukeMTMC Attribute [16] use 1,2,3,4 to represent each type of attribute. Table 3 gives an example of annotation.

Figure 5.

Labeling annotation of several APR datasets. (a) PA-100K dataset. (b) PETA dataset. (c) Market-1501 Attribute. (d) DukeMTMC-Attribute.

3.3. Metrics to measure ReID experiment results

To evaluate an attribute recognition system, Precision (P) and Recall (R) are two widely used measurements. Moreover, mAP and CMC are two indicators commonly used to measure re-identification and retrieval capabilities

The attribute recognition task mainly focuses on the differences between the true labels and predicted values. Both Precision (P) and Recall (R) can reflect the exact numbers of items retrieved, but they are actually different. Specifically, Precision is the number of correct information in the extracted information. Recall is the ratio of the extracted correct information to the information in all samples

3.3.1. mAP (mean Average Precision)

Because more than one person is detected in the pedestrian re-identification task, we use mAP as the final evaluation index, which reflects the property of the correct search results arranged at the top of the tested list. The area under the PR curve (Precision-Recall) is defined as AP (Average Precision). It reflects the relationship between the horizontal coordinate R and the vertical coordinate P. Furthermore, the larger the area under the PR curve, the better the model performance. The Precision rate and Recall rate are calculated as eq. (1) and eq. (2):

$$Precision=\frac{TP}{TP+FP}$$ (1)

$$Recall=\frac{TP}{TP+FN}$$ (2)

Among them, TP (True Positives) means the number of detection frames with IoU >0.5 (Intersection over Union, IoU); FP (False Positives) is the number of detection frames that indicate IoU ≤ 0.5, which can also be said to be the number of redundant detection frames that detect the same real label; FN (False Negatives) indicates the number of detection frames that have not detected real labels.

3.3.2. CMC (Cumulative Matching Characteristics)

Another metric is CMC (Rank-k matching accuracy). Rank-n indicates that in the descending order of similarity of recognition results, the first n results contain the target.

The recognition rate is the ratio of the number of Rank-n to the number of query samples; it is the same as the commonly used top1 err or top5 err evaluation index, but here Rank-1 recognition rate represents the correct rate rather than the error rate. The relationship between the two is eq. (3).

$$rank1=1-top1err$$ (3)

3.3.3. mINP (mean inverse negative penalty)

In passenger retrieval tasks, the ranking of the correct target in the search list should be considered, especially for multi-view data. The ranking of hard-samples further determines the measurement of model evaluation. It may happen that the AP obtained by Rank List 1 under the same CMC is better than Rank List 2, but finding all the right matches takes much more effort. To solve this problem, Ye [61] designed negative penalty (NP), in which the hardest sample with the correct match is considered.

$$N{P}_i=\frac{R_i^{hard}-G_i}{R_i^{hard}}$$ (4)

where $R_i^{hard}$ in eq. (4) represents the sort location of the hardest sample, and $G_i$ indicates the number of pairs matched correctly. The inverse of NP is defined as INP, the calculation of mINP is given by eq. (5).

$$mINP=\frac{1}{n}\sum _i(1-N{P}_i)=\frac{1}{n}\sum _i\frac{G_i}{R_i^{hard}}$$ (5)

3.3.4. RS (Robustness Score)

To further evaluate the robustness for retrieval objects in long time span. Huang [62] defined a Robustness Score (RS) which is defined as eq. (6):

$$RS(\mathbb{P})=\frac{\sqrt{Scor{e}_L(\mathbb{P})\times Scor{e}_L(\mathbb{P})}}{|Scor{e}_L(\mathbb{P})-Scor{e}_L(\mathbb{P})|+1}$$ (6)

where $Scor{e}_L$ and $Scor{e}_S$ represent the long-term and short-term re-ID accuracy respectively; $\mathbb{P}$ represents different evaluation indexes. For a given ReID model, the closer $Scor{e}_L$ and $Scor{e}_S$ are to each other, the greater the value of the RS indicator.

4. Pedestrian reidentification based on deep learning

Deep learning based Pedestrian ReID including feature learning and distance metric. For the feature learning approach, the model treats ReID as a classification or validation task, rather than directly determining the similarity between images.

Feature representation learning mainly involves classification loss and verification loss. Specifically, ID is often used as the basis for classification loss. ID Embedding network (IDE) [47],[63] is a widely used baseline model. Through using ID of a pedestrian as a training label to train the model, it only needs to input one picture at a time. While the input of verification loss is a pair of pictures of pedestrians, the network learns whether they are the same person, which is equivalent to a binary classification problem. The verification model aims to learn the similarity of two pictures. Generally it uses end-to-end models such as Siamese network [64], Fig. 6 summarizes various ReID algorithms and potential future directions. In this part, we will introduce cutting-edge ReID methods mainly from the perspective of global and local features-based, video sequence-based, GAN-based and attention mechanism-based.

Figure 6.

Summary of deep learning-based ReID algorithm and its challenges and future research directions.

4.1. Global and local features-based method

4.1.1. Global features

Global feature refers to the extraction of the overall information of each picture, it does not have any spatial information. While Local feature refers to the extraction of a certain area in pedestrian image, and finally multiple local information is merged as the final feature.

In the early years of incorporating advanced deep learning methods into the area of PAR and pedestrian ReID, global feature was the well known choice. Global features can usually be extracted end-to-end, and the advantage lies in simple calculations. But the limitations include that noise regions may cause great interference to global features, and the misalignment of poses will also lead to poor global feature matching.

4.1.2. Horizontal slices

The partial feature extraction method based on deep learning includes image block [65], [66], [67]and the posture-based local feature [36],[68]. There are many blocking methods of many images, such as uniform block [65],[67]and context sensing block [69], where uniform block can be divided into grid block [65] and horizontal slice [66]. This part will introduce several levels of horizontal slice.

In Gate Siamese [70], each image is characterized by a CNN, and local features are input to the LSTM network in order, automatically express the image final feature. Deep-Person [71] regarded the pedestrian as a sequence of body parts, and apply LSTM to take into account the contextual information between body parts. AlignedReID [72] introduced Dynamically Matching Local Information (DMLI), using shortest path to find the most effective dynamic connection (Dynamic Time Warping). PCB [66] horizontally cut the input image into 6 pieces, extracts a partial feature for each line and finally concatenate the local characteristics. Integration convolutional neural network (ICNN) [73] jointly learned global and local features. Fan et al. [74] proposed SCPNet in which discriminative features are learned using the space-channel relationship, each channel is assigned to focus on a specific space of the body.

4.1.3. Posture information

The motivation of local features based on gesture extraction is to make full use of pedestrian key points [36],[68]. First we migrate the trained gesture estimation model to the pedestrian dataset, so obtain the positioning of the human body posture skeletal point, then extract local characteristics based on key points. Fig. 7 shows an effective approach under pose changes, i.e., fusing global and local features based on extracting key points of the human body.

Figure 7.

Fusion method of local features and global features based on the extraction of key points of human posture.

Spindle Net [36] firstly used human body structure information to facilitate feature learning, which employed feature decomposition and feature fusion based on body region information. Zheng et al. [33] introduced the PIE to describe pedestrian, the Pose-Box structure was used to align pedestrians, which is generated through pose estimation followed by affine transformations. Pose-driven Deep Convolutional (PDC) [34] proposed an end-to- end architecture that use partial human cues to mitigate pose changes. Wei et al. [35] explicitly leveraged the local and global information of body to generate more robust representation. PABP [75] learned part-aligned representation, the body feature maps are connected to a bilinear ensemble layer and then fused into a single image descriptor.

4.1.4. Semantic segmentation

Semantic segmentation of the image is also applied to the weight recognition, the representative methods are as follows. Generating realistic images of persons is a task full of challenge as the complex interplay between the different image factors, such as the foreground, background and pose information. Ma et al. [88] aimed at generating such images based on a novel, two-stage reconstruction pipeline, they give a new idea for image prospect extraction. SPReID [89] used additional semantic parsing branches to generate probability maps associated with different regions. Besides, the method of semantic segmentation can also be referred to DeepLabV2 [90], Mask RCNN [91], etc.

4.1.5. Grid characteristics

IDLA [92] simultaneously learned a similarity metric for person ReID. The input to the network is pairs of images, higher layers compute relationships by calculating the difference of the eigenvalues of a certain size grid. PersonNet [93]employed a layer of computing neighborhood range differences across two input images to capture local relationship between patches. For the problem of people images of different sizes, DSR [94] used FCN to make the pixel-level features consistent.

In general, the grid feature is a relatively fine-grained physical area feature. Some early works extended the grid feature to a part feature to calculate the feature map difference of two images. The above typical researches used the grid feature to solve ReID Work, but due to the large amount of calculation and the lack of obvious performance, the grid feature is not very commonly used.

4.2. Video sequence-based method

Video re-identification (Video ReID) is also called as sequence re-identification, which refers to the use of a continuous sequence of pedestrian pictures for pedestrian re-identification tasks. Generally, the characteristics of Video ReID include rich pedestrian posture changes, common occlusion phenomena, difficulty in fusion frame information, and inconsistent quality between frames, etc.

The earlier method of video ReID is to extract per-frame features, and then directly obtain the final feature through average pooling or maximum pooling. This is relatively simple, and the performance depends on the single frame. The difficulty of sequence re-identification lies in how to judge the quality of each frame and perform feature fusion of multiple frame features, how to generate the motion features for the sequence, and improve the computational efficiency.

Accumulative Motion Context (AMOC) [95] contained spatial motion sub-network and employed RNN to fuse the characteristic information of all frames. Aiming at how to integrate features, Deep Feature Guided Pooling (DFGP) [96] used PCA-based convolutional network (PCN) to extract features of each frame and give a new idea to take full advantage of the space-time information. Region-based Quality Estimation Network (RQEN) [97] mainly focused on the case of occlusion. Besides, Saha et al. [98]used residual network (ResNet) along with LSTM for feature extraction. Zhang et al. [99] introduced the concept of “mean-body” and defined an intra-video loss. Yu et al. [100] exploited unlabeled tracklets, Li et al. [101] proposed a new spatio-temporal attention model that automatically discovers adverse set of distinctive body parts. TCA-Net [102]considered the temporal alignment problem.

Research on video ReID across time and space has received increasing attention in recent studies. Table 7 compares several cross-temporal video re-recognition algorithm indicators published at the top conference.

Table 7.

Performance comparison of several cross-temporal video re-recognition algorithms on MARS dataset.

Method	Source	MARS
		Rank1	Rank5	Rank20	mAP
CTL [76]	CVPR 21	91.4	96.8	86.7
BiCnet-TKS [77]	CVPR 21	90.2	-	-	86.0
GRL [78]	CVPR 21	91.0	96.7	98.4	84.8
ST-GCN [79]	CVPR 20	89.95	96.41	98.28	83.70
MGH [80]	CVPR 20	90.0	96.7	98.5	85.8
MG-RAFA [81]	CVPR 20	88.8	97.0	98.5	85.9
AP3D [82]	ECCV 20	90.7	-	-	85.6
AFA [83]	ECCV 20	90.2	96.6	-	82.9
TCLNet [84]	ECCV 20	89.8	-	-	85.1
STRF [85]	ICCV 21	90.30	-	-	86.10
STMN [86]	ICCV 21	90.5	-	-	84.5
STEP-Emb [87]	ICCV 21	90.8	97.1	98.8	87.0

4.3. GAN based method

Generally, the performance of training models on different datasets has certain differences. The changes in experimental effect caused by changes in the training datasets are likely to be caused by overfitting of the model. To solve such problems, GAN is widely used in data augmentation and expansion. GAN [105] could randomly generate sample images, CGAN [106] added conditional constraints to generate images on the basis of GAN, Pix2pix [107] and CycleGAN [108] could convert paired images and arbitrary images in the A and B domain respectively. Zheng et al. [109]applied GAN to create unlabeled samples to expand data, and used the label smoothing regularization for outliers (LSRO) to smooth ID tags. StyleGAN [110] and CamStyle [111] could achieve style transfer between any two cameras.

PTGAN [112], SPGAN [113] solved the obvious deviation between the data collected in different scenarios, improved the network trained on the A dataset in the B scene by generating the image of the former into the style of the latter. PNGAN [114] generated the image of the desired posture according to the input image and the target posture, the features extracted from multiple fixed poses of the same pedestrian make the model more robust. Several GAN-based methods are compared and summarized in Table 8. Fig. 8 shows the images generated by several GAN networks, in which CamStyle transfers between two cameras, PNGAN generates a variety of poses, and PTGAN realizes style conversion in different domains.

Table 8.

The realization methods and targets of a typical GAN network.

Method	Basis	Tricks	Target
DCGAN	GAN	Label smooth	Extended data
CamStyle	CycleGAN	Label smooth	Camera style deviation
PTGAN	CycleGAN + PSPNet [103]	Background segmentation	Domain dissimilarity
SPGAN	CycleGAN + Sianet	Constrain mapping	Domain dissimilarity
PNGAN	GAN+openpose [104]	Extract body pose	pose-normalization

Figure 8.

Examples of the transferred images by Camstyle, DCGAN, PNGAN and PTGAN.

4.4. Attention mechanism-based method

Attention mechanism [115] is similar to the human method of selective signal processing by focusing attention on specific areas while ignoring other information. It can be divided into time attention mechanism and spatial attention mechanism. The former mainly focuses on which sequence is more important, while the latter mainly focuses on specific part. Song [116]used a mask to divide the complete image into background and pedestrian body regions, and used the triple loss supervised network to learn human body region features and ignore background features. Li et al. [101] solved the problems of pedestrian occlusion and misalignment in video sequences by new proposed spatio-temporal attention model. HA-CNN [117] jointly learned soft pixel as well as hard regional attention. Xu et al. [118] introduced Attention-Aware Compositional Network (AACN) for ReID, which consists of PPA and AFC, which represent pose-guided and attention-aware feature, respectively. There is also a typical HydraPlus-Net [39], that multi-directionally provide the multi-level maps with attention to different layers.

5. Challenges and frontier research

With increasing public safety requirements, pedestrian re-identification often faces the task of cross-camera retrieval. The primary difficulty in solving this problem is the appearance change caused by the difference in viewing angle or cloth-changing in different scenes. At the same time, some fine-grained attribute information needs to be considered to distinguish different people. In summary, pedestrian re-identification is challenging in learning a discriminative and robust visual representation for changes in perspective. Specifically, pedestrian re-identification may face problems and difficulties such as limited training data, algorithm performance, including effectiveness and efficiency, domain gap, unconstrained environment, etc. There are also some frontier researches to explore new structure, loss function design, and effective training tricks, etc.

5.1. Network structure design

5.1.1. ReID specific architecture

Existing ReID methods usually make some architectural modifications on the basis of image classification, mostly follow the CaffeNet [119], VGG-16 [120] and ResNet-50 backbone, for example, SVDNet [121] used the Eigenlayer as the second last FC layer, PCB [66] set the last convolutional stripe to 1 to increase feature layer pixels. Researchers have also proposed some new networks designed specifically for ReID. Noting that the metric and the classification loss are inconsistent in an embedding space, BNNeck [122] added a BN layer after global pooling to separate them into two different feature spaces. Omni-Scale Network (OSNet) [123] designed a residual block consists of feature stream that detects features of certain size. Auto-ReID [124] involved a new reID search space and a new retrieval-based model, it combined a typical classification search space and a novel part-aware module. Faster-ReID [125] gave an All-in-One framework which learns multiple codes of different lengths in a single framework with a code pyramid structure and self-distillation learning. Besides, there are some other typical networks including FPNN [65], MLFN [126], BraidNet [127] and Siamese network [128]. AGW [129] provided a baseline and achieves competitive performance on cross-modality ReID tasks.

5.1.2. Loss function design

To search for pedestrians with the same probe ID in the gallery, pedestrian re-identification often needs to achieve similarity-based clustering effects. Therefore, unlike the frequently used loss functions such as cross entropy, the commonly used metric learning methods and loss functions in pedestrian re-identification research are ID loss [16],[121], verification loss [92],[130],[131], triplet Loss [132], [133], [134], [135] as shown in Fig. 9, and contrastive loss [136], quadruplet loss [137], fusion of part loss and global loss [36], multiple loss combinations [138], circle loss [139], etc. In unsupervised learning, other non-parametric classification losses such as InfoNCE [140], [141], ClusterNCE loss [142], OIM loss [143].

Figure 9.

Typical widely used loss functions [61]. (a) Identity Loss, (b) Verification Loss, (c) Triplet Loss.

Classification loss: It is also well known as ID loss. The ID number of pedestrians in the train set is consistent with the network categories. The feature layer is connected with a FC layer for classification, and the cross entropy loss is calculated through the softmax activation function.

$$L_{id}=\frac{-1}{n}\sum _{i=1}^{n}log(p(y_i|x_i))$$ (7)

As shown in eq. (7), where the label of the input image $x_i$ is represented as $y_i$, p is the probability of network prediction, n represents the count of training samples in per batch.

Verification Loss: It fuses two feature information to calculate a contrastive loss [70], and each time a pair of pictures are input into the same Siamese network to extract features. The contrastive loss optimizes the comparison of paired sample distances, it can be formulated by eq. (8)

$$L_{con}=y{d}_{I_a,I_b}^2+(1-y){(\alpha -d_{I_a,I_b})}_{+}^2$$ (8)

Where $y=1$ when $I_a$ and $I_b$ are the same, and $y=0$ on the contrary. α is a margin parameter. ${(a)}_{+}$ + represents max(z,0), $d_{I_a,I_a}$ indicates the Euclidean distance between pairs of input features of $I_a$ and $I_b$, which can be expressed by eq. (9)

$$d_{I_a,I_b}={\Vert f_{I_a}-f_{I_b}\Vert }_2$$ (9)

Triplet Loss: Arising in the context of nearest neighbor classification [144], x is an image that is embedded into Euclidean space and constrained to a d -dimensional hypersphere ${\Vert f_x\Vert }_2$. For a specific person, to ensure that the anchor $x_i^a$ is closer to the positive sample $x_i^p$ than the negative $x_i^n$, define the loss function as eq. (10).

$$L=\sum _i^N{[{\Vert f(x_i^a)-f(x_i^p)\Vert }_2^2-{\Vert f(x_i^a)-f(x_i^n)\Vert }_2^2+\alpha ]}_{+}$$ (10)

Unified Loss: Maximizing intra-class similarity $s_p$ and minimizing inter-class similarity $s_n$ are main purposes of deep feature learning. In the feature space, given a sample x, assume that the similarity scores of within-class and between-class are K and L. γ is a scale factor and m is a margin for better similarity separation. We denote these similarity scores as $\{s_p^i\}(i=1,2,\cdots ,K)$ and $\{s_n^j\}(j=1,2,\cdots ,L)$, respectively. $L_{uni}$ iterates through every similarity pair to reduce $s_n^j-s_p^i$. It degenerates to triplet loss or classification loss through slight modifications. To minimize each sjn as well as to maximize each $s_n$ as well as to maximize $s_p$, unified loss function is defined by:

$$L_{uni}=log[1+\sum _{i=1}^K\sum _{j=1}^{L}exp(\gamma (s_n^j-s_p^i+m))]$$ (11)

Circle Loss: It allows each similarity score to be learned at their own speed, thus increasing the flexibility of optimization, depending on its current optimization status. ${\alpha }_n^j$ and ${\alpha }_p^i$ are non-negative weighting factors. Firstly neglect the margin item m in eq. (11) and transfer the unified loss function into the proposed circle loss and get eq. (12).

$$L_{circle}=log[1+\sum _{i=1}^K\sum _{j=1}^{L}exp(\gamma ({\alpha }_n^j{s}_n^j-{\alpha }_p^i{s}_p^i))]$$ (12)

Softened Cross-entropy Loss: Lin et al. [145] proposed softened similarity learning and relieved the error of the hard quantization loss. Unlike original one-hot labels, an image is encouraged to be associated with k possible related classes, whose labels are treat as a distribution. The softened cross-entropy loss is formulated as eq. (13)

$$L=-\sum _{j=1}^{N}log(p(y_j|x_i,V))t(y_j)$$ (13)

Where the probability of the $i_{th}$ class is defined as $p(y_i|x,V)$, V is the lookup table that stores the feature of each class. Hyper-parameter λ balances the effect of the ground truth and the reliable classes.

5.1.3. Bag of tricks

Research on deep learning-based person ReID has reaped tremendous progress and remarkable performance, while most of them rely on complex structural design. Luo [122] has proved through experiments that some tricks can be used to improve model performance significantly without increasing training costs. These tricks include preprocessing of the data as well as minor modifications to the network design.

Learning rate has a non-negligible impact of a ReID model. Warmup strategy [146] in Fig. 10(a) can be applied to bootstrap the network for better performance. To address the occlusion issue and Optimize the generalization ability, REA [147] was proposed to enhance data. To prevent overfitting for a classification task, Label smoothing (LS) [148], [149] in Fig. 10(b) is widely used. To obtain feature maps with higher resolution and richer granularity, Sun et al. [66] deleted the spatial down-sampling, more conveniently, the last stride of which in the backbone network was set to 1. Besides, Luo et al. added a new layer BNNeck [122] and put the triple loss and ID loss in different embedding spaces which has improved performance of the model to a large extent.

Figure 10.

Training tricks. (a). With label smoothing [149], the distribution centers at the theoretical value. (b). With warmup strategy [146], there is gradual increase in learning rate over the first 10 epochs.

5.2. Cloth-changing

Most current researches assume that no change in the dress of the same person. While in real-world scenarios, pedestrians are likely to reappear in the surveillance network after a long-time interval and change their wearing [150]. Existing researches on re-identification in changing clothes scenarios can be broadly classified into two categories according to whether pedestrians' dresses are known. Given what clothes pedestrians are wearing, Yu [151] designed Biometric-Clothes Network (BC-Net) which could effectively integrate biometric and clothes feature, it was promising in tracking suspects and finding lost children/elders in real-world scenarios. For general re-identification in the case of changing clothes with a long time span, Wu [152] exploited depth information to provide more invariant body shape and skeleton information regardless of illumination and color change. Contour sketch [153] of person image was proposed to take advantage of the shape of the human body for extracting features. 3D information caught by RGB-D Sensors [154] was considered to be a very fruitful research direction because of its main advantage of a soft biometric policy. Huang et al. [62] adopted Feature Sparse Representation and Soft Embedding Attention, integrated capsules to deal with the person re-ID task [155]. There are also Cloth-Clothing Change Aware Network (CCAN) [156] which performed identity recognition and clothing change detection simultaneously. The above methods also established some typical datasets, including COCAS [151], RGB-D [154], PRCC [153], Celeb [157], VC-Clothes [150], DG-Net [158], some samples of which are shown in Fig. 11.

Figure 11.

Pedestrian re-identification dataset samples in cloth-changing scene. (a). Celeb [157], obtained by crawling celebrity images. (b). VC-Clothes [150], generated by the game engine. (c). DG-Net [158], expanded by style transfer.

5.3. Domain adaptation

After the application scenario changes, the performance of the algorithm trained with a specific dataset could vary greatly. It is well known that there are large domain gaps among different datasets [64],[159]. For example, in the data collected by the same camera in summer, short-sleeved shorts will account for more pedestrian attributes, and backpack attributes will be more common in the data collected by outdoor cameras. For the problem of insufficient data and limitation of scene area, the data can be augmented by synthesizing images through style transfer methods such as GAN [105].

Training across datasets to achieve domain adaptation is a common method used in existing researches. In addition, it is possible to train a domain generalized model that contains multiple source datasets, so it can be transferred to new datasets without additional learning. Domain Invariant Mapping Network [160]was proposed to enable a ReID model to be deployed out-of-the-box for any new camera network domain. There are also PUL [161] methods that don't label the data of the new scene, but assign pseudo-labels to the new data through the method of clustering, and finally fine-tune the existing model. HHL [162] and Tracklet Association [163] used prior knowledge and soft label to further mine information in the target domain. Some progresses have also been made on more general classification and semantic segmentation [164] issues. Kang [165] aligned the attention maps of different domain so that the information is better adapted from source to the target network, which inspired a new way and made contributions to unsupervised domain adaptation (UDA).

5.4. Occlusion conditions

In actual scenes, occlusion is often impossible to avoid, only a portion of the human body is available [67], resulting in a decrease in the accuracy of the model, which requires the model to have a high generalization ability. The general idea to solve this problem in person recognition can be divided into two types. One is to use the key point detection [166] to extract the non-occluded part of the characterization for similarity matching, and another is to build a large-scale occlusion datasets, among which representative ones are Occluded-ReID [167], Partial-ReID [67] and P-ETHZ [168].

Using the posture information to extract the key points is a better approach to locate the occluded part of the body. OG-Net [169] map the human body to 3D space, used the point cloud to integrate the structure to learn the human body expression and obtain robust features. PGFA [166] exploited marks of pose to disentangle the useful information from the occlusion noise. HO-Net [170] learned high-order relation information assisted by posture for robust alignment.

For partial re-identification, Visibility-aware Part Model (VPM) [168] perceived the visibility of regions through self-supervision, which was capable to estimate the shared regions between two images and thus suppresses the spatial mis-alignment. Deep Spatial feature Reconstruction DSR [94] used half-length images to search for partially occluded full-body images, and generated spatial feature maps with fixed size to remain features consistent at pixel level. FPR [171] was proposed for Alignment-free Occluded Person ReID. Aligned-ReID [172],[173] used Dynamically Matching Local Information (DMLI) dynamic connection to solve horizontal block misalignment.

The performances of typical methods under various occlusion datasets are shown in Table 9. Besides, multi-modal input can also improve model robustness in unrestricted case, which makes the model closer to practical use, such as searching by verbal description. Textual-visual matching [197] measured similarities between sentence descriptions and images.

Table 9.

Performance comparisons with the occluded methods on the reported datasets.

Method	Occluded-REID		Partial-REID		P-DukeMTMC
	Rank1	mAP	Rank1	mAP	Rank1	mAP
IDE [63]	52.6	46.4	51.7	52.4	36.0	19.7
PCB [66]	59.3	53.2	66.3	63.8	43.	24.7
PGFA [166]	57.1	56.2	68.0	56.2	44.2	23.1
PVPM [174]	66.8	59.5	75.3	71.4	50.1	29.4
Occluded-ReID [167]	68.14	-	78.52	-	46.15	-
VPM [175]	-	-	67.7	-	-	-
DSR [94]	72.80	62.83	73.67	68.07	-	-
FPR [171]	78.30	68.00	81.00	76.60	-	-

5.5. Model efficiency

Lightweight ReID model is getting more and more attention in current research. For higher accuracy, most of the existing methods utilize a large deep network to learn high-dimensional features for computing similarities. But the query time would increase massively with the expand of the tested gallery size.

The most direct idea to reduce model complexity is mainly focused on network pruning for image retrieval acceleration. Compared with the conventional global center-based methods, which only keep the filters far away from the geometric center, Progressive Local Filter Pruning (PLFP) [198] pruned filters according to the local relationships to the neighbors to preserve the representation ability of the model. Based on a new search space and a new retrieval algorithm, Auto-ReID [124] enabled the automated approach to find an efficient and effective CNN architecture.

To search person images quickly and accurately. The main idea of recent fast ReID methods is hashing algorithm which learns a binary code instead of real value. Several fast ReID methods [199], [200], [201], [162], [202] have been proposed to increase algorithm speed while maintaining competitive accuracy, a typical example is CtF [125] search strategy. Another research direction is to design a lightweight network by modifying the model. [123],[203].

5.6. Limited training data

Existing pedestrian datasets are generally obtained for common scenarios. In most cases, in order to generate more effective models, training specific networks require specific data sets. However, collecting data and labeling is a very time-consuming and laborious work. In order to expand the limited dataset, the most popular approach in the research is GAN [105] and its derivative models including Camstyle [111], PTGAN [112], SPGAN [113], PNGAN [114] etc. Besides, APR [16] added annotated appearance attributes for the Market-1501 dataset and the Duke-MTMC dataset, making the dataset originally used for pedestrian ReID can be applied to attribute recognition. DG-Net [158] used clothes-changing pedestrian images to enhance data. SOMAnet [204], PersonX [205] and VC-Clothes [150] used 3D game engine to generate human data in different views. Huang [157] crawled the street snap-shots of celebrities from the Internet and built Celeb-ReID.

5.7. Resolution changes

Due to changes in camera angles and inconsistent target distances, the resolution of pedestrian pictures varies greatly. SING [206] simultaneously optimized image super-resolution and person ReID matching, which use different methods to extract features of different resolutions. DSPDL [207] designed a Discriminative Semi-coupled Projective Dictionary Learning framework for matching pedestrian images of great resolution divergences. SDALF [208] (Symmetry-Driven Accumulation of Local Features) consisted in the extraction of features that model three complementary aspects of the human appearance, in this way, robustness to viewpoint and illumination variations is achieved. A new work in CVPR2020 named as INTACT [209] (Inter-Task Association Critic) gave an effective solution to use an image super-resolution model to improve the low resolution of query images, which can reduce unaligned feature distributions and promote identity matching performance.

6. State-of-the-arts

6.1. Analysis of ReID on SOTA

This part mainly reviews the ReID works published in top conferences in computer vision and artificial intelligence, and makes a comparative analysis with some other representative works. As Table 10 shown, we summarize the outcomes mainly on Market-1501 and DukeMTMC datasets. Correspondingly, the visualized scatter plot is depicted in Fig. 12.

Table 10.

Comparison with state-of-the-arts for ReID on Market-1501 and DukeMTMC.

Method	Source	Market-1501		DukeMTMC
		mAP	Rank1	mAP	Rank1
Supervised Learning
Deep-Person [71]	Pattern Recognition	79.58	92.31	64.80	80.90
AlignedReID++ [173]	Pattern Recognition	77.6	91.0	68.0	80.7
FMN [176]	Pattern Recognition Letters	67.12	85.99	56.88	74.51
MGN [177]	ACM Multimedia	86.9	95.7d	78.4	88.7
PAN [178]	TCSVT	63.5	82.81	51.51	71.59
GP-reID [179]	ArXiv 18	81.2	92.2	72.8	85.2
OG-Net [169]	ArXiv 20	59.97	80.94	50.81	71.77
ACRN [21]	CVPR 17	62.60	83.61	51.96	72.58
SVDNet [121]	CVPR 17	62.1	82.3	56.8	76.7
AACN [118]	CVPR 18	66.87	85.90	-	41.37
SPreID [89]	CVPR 18	79.67	91.45	68.78	83.3
HA-CNN [117]	CVPR 18	75.7	91.2	63.8	80.5
HOReID [170]	CVPR 20	84.9	94.2	75.6	86.9
CamStyle [111]	CVPR 18	71.55	89.49	57.61	78.32
PCB [66]	ECCV 18	77.3	92.4	65.3	81.9
PISNet [180]	ECCV 20	87.1	95.6	78.7	88.8
Unsupervised Learning
UnityStyle [181]	CVPR 20	95.8	98.5	93.6	95.1
M3[182]	CVPR 20	82.6	5.4	68.5	84.7
Unsupervised re-ID [183]	CVPR 20	37.8	71.7	28.6	52.5
SADA [184]	CVPR 20	59.8	83	55.8	74.5
Generalizing REID [185]	ECCV 20	71.5	88.1	65.2	79.5
MEB-Net [186]	ECCV 20	76.0	89.9	66.1	79.6
CrowdReID [187]	ECCV 20	84.7	95.3	74.4	88.3
Light-ReID [125]	ECCV 20	84.9	93.7	74.8	87.6
UNRN [188]	AAAI 21	78.1	91.9	69.1	82.0
Transfer Learning
SPGAN [113]	CVPR 18	26.9	58.1	26.4	46.9
VPM [175]	CVPR 19	80.8	93.0	72.6	83.6
HCT [189]	CVPR 20	56.4	80.0	50.7	69.6
SCSN [190]	CVPR 20	88.5	95.7	79	91
AD-Cluster [191]	CVPR 20	68.3	86.7	54.1	72.6
HHL [162]	ECCV 18	31.4	62.2	27.2	46.9
BUC [192]	AAAI 18	38.3	66.2	27.5	47.4
ARN [193]	CVPR 18	39.4	70.3	33.4	60.2
SSG [194]	ICCV 19	58.3	80.0	53.4	73.0
MMCL [195]	CVPR 20	60.4	84.4	51.4	72.4
NRMT [196]	ECCV 20	71.7	87.8	62.2	77.8

Figure 12.

State-of-the-arts (SOTA) of ReID on Market-1501 datasets.

Analyzing the improvement effect of the above new algorithms on the model performance, it can be seen that some tricks have brought objective efficiency, as discussed below. Applying of GAN models (e.g., DI-ReID [210] and UnityStyle [181]) could obtain more style-robust depth features for querying. Domain adaptive person ReID, including SADA [184], AD-Cluster [191], MEB-Net [186] and Generalizing ReID [185] gradually become a research hotspot. Besides, unsupervised ReID [183] adopt iterative training. PRI [211] solved low-resolution problem. Ahmed et al. [212] used transfer learning that aims to transfer the knowledge using only source models and limited labeled data. HCT [189] conducted training with hard-batch triplet loss. There are also novel methods that focus on solving some challenges, representatives are HOReID [170], PISNet [180] and CrowdReID-GASM [187], which studied ReID under occlusion, pedestrian interference and crowded conditions respectively.

For cross-domain re-identification, as Table 10 displayed, we evaluated some cross-domain ReID methods published on top conferences on two public person ReID datasets: Market-1501 and DukeMTMC. Quantitatively, we compared these state-of-the-art domain adaptation person ReID approaches. The comparison shows the influences on Map and Rank-1,5,10(%), with Market-1501 and DukeMTMC-ReID as source and target datasets, respectively.

Most work published on 2020 ECCV focused on unsupervised ReID, especially unsupervised domain adaptation (UDA), as the methods listed in the table. Typically, GDS [213] and JVTC [214] designed new training strategies. DG-Net++ [215] segregated the id-unrelated noises to get more effective adaptation from the adaptation process. AD-Cluster [191] introduced adaptive sample augmentation to generate more diverse samples.

6.2. Potential research direction of ReID

Domain Adaptation: According to Table 10, it can be seen that domain adaptive re-recognition has been extensively studied in recent years, but it is still full of challenges to achieve rank-1 above 80% when the model trained under a specific dataset is transferred to others or applications in real scenarios or even unseen domains. Therefore, ReID in rich scenarios is a promising research direction. In particular, most datasets are collected in scenes with a wide field of view, so “partial” pedestrian re-recognition technology is very important.

Visible-Infrared ReID: Pedestrian re-identification at night or under poor light conditions is also a problem worthy of study, which is an important task in night-time surveillance applications. The main idea to solve this problem is to use infrared cameras to collect pictures. Hi-CMD [216], cm-SSFT [217], CMAlign [218] and DDAG [219] provided new ideas for cross- modality person re-identification respectively.

Unsupervised and transfer learning: Since the collected data is limited in most cases, the workload of manual labeling is huge and costly. Semi-supervised [220], unsupervised learning [221] methods and transfer learning [222] have achieved widespread attention in recent studies, which also reflected by the increasing amount of work published in top venues. There has been some research works in this direction [183], [223], [224], but there is still a lot of rooms for improvement and full of challenges.

High-quality datasets: Most of the current pedestrian re-identification approaches are evaluated based on well-defined datasets. In practical scenarios, the ReID data might be collected from a variety of heterogeneous modalities. Due to domain differences, only the establishment of a large enough high-quality standard dataset can better train the robustness of the algorithm. For some clothing-change occasions, the author has created new datasets [150], [151], [153], [12], human interaction or active learning [225], [226] also provides another possible way to reduce the dependence on manual annotation.

6.3. Analysis of Attribute-Assisted ReID on SOTA

Pedestrian can be described by a combination of various different kinds of attributes, which can be a decisive clue to distinguish people with very similar global appearance. Through the Attribute-Assisted ReID algorithms mentioned in chapter 2.1, it can be drawn that person attribute recognition plays an auxiliary role in re-identification.

PAR and ReID can transfer information by attributes sharing. There are two ways to integrate attribute and ID information: (a) Independent Supervision [20]: Independently train a deep CNN for either attribute or identity label then use the concatenated feature for ReID matching. (b) Joint Supervision [27]: Multi-task joint learning CNN subjecting the identity and attribute supervision to a shared feature representation in the end-to-end model training.

Table 11, Table 12 show the performance of several Attribute Assisted ReID algorithms on Market-1501/DukeMTMC dataset and VIPeR dataset, respectively. Experimental results demonstrated the effective ReID performance gain obtained by attribute assisting [11],[15]. At the same time, the integration of ID effectively introduces complementary information to some extent and helps to obtain the improvement of overall attribute identification [19],[16].

Table 11.

Comparison of several state-of-art Attribute-Assisted ReID methods on Market-1501 and DukeMTMC-ReID datasets.

Method	Market-1501		DukeMTMC
	Map	Rank1	Map	Rank1
TJ-AIDL [27]	26.5	58.2	23.0	44.3
ASMR [26]	31.0	49.6	-	-
ACRN [21]	62.60	83.61	51.96	72.58
APR [16]	66.89	87.04	55.56	-
A3M[8]	68.97	86.54	-	-
APDR [11]	80.1	93.1	-	-
Tensor Fusion [15]	90.16	98.38	84.74	95.42

Table 12.

Comparison of the accuracy and re-identification performance of several state-of-art Attribute-Assisted ReID methods on VIPeR datasets.

Method	VIPeR
	Acc	R1	R5	R10	R20
c-LSVM [28]	-	17.09	38.61	52.53	68.35
SSDAL [12]	58.6	37.9	65.5	75.6	85.4
OAR [19]	66.9	41.5	69.0	82.5	-
CAN [14]	-	54.1	83.1	91.8	96.4
FT-CNN [22]	-	52.1	79.6	89.2	95.0

Analyzing the attribute-assisted ReID method, we can draw the following conclusions:

1) Attribute attention model generate features by “seeing” and “comparing” people images to locate the most discriminative parts [14], which provide new clues for searching for target persons given appearance descriptions [17].

2) Attribute consistency principle can be exploited to achieve a clear advantage in unsupervised fusion of multiple supervisions for cross-domain ReID [27], as well as in addressing human misalignment caused by pose changes [11].

3) Attribute semantic integration model jointly learns the discriminative projection of the appearance attribute subspace can effectively assist ReID [23].

4) Attribute dataset fine-tuning improves CNN features, combined with metric learning on ReID dataset can further improve the discriminative ability. [12],[21],[22].

7. Conclusion

Through the analysis and comparison of the different methods mentioned in this article, we can draw the following conclusions. Firstly, attributes are consistent with human recognition mechanisms for identifying people. Referring to perceptual ability, person attributes can be exploited as mid-level human semantic information to help promote the performance of person re-identification task. Secondly, the fusion of global and local features can more comprehensively represent pedestrian images and enhance the robustness of the extracted features. Thirdly, the method based on posture and key point extraction of the human body is effective for occlusion and partial re-recognition. Fourthly, training datasets can be expanded using pose or style transfer methods, which can effectively solve uncontrollable environmental factors such as domain deviation and clothing change. In general, with the booming development of deep learning technology and the urgent pace of building an intelligent society, re-identification and attribute recognition are facing many challenges and have significant potential research value.

Declarations

Author contribution statement

All authors listed have significantly contributed to the development and the writing of this article.

Declaration of interests statement

The authors declare no conflict of interest.

Data availability statement

Data will be made available on request.

Funding statement

Jie Hu was supported by National Social Science Fund of China [17ZDA020], National Natural Science Foundation of China [51975360 & 52035007]. This work was supported by Cross Fund for medical and Engineering of Shanghai Jiao Tong University (YG2021QN118).

Additional information

No additional information is available for this paper.