Categorical Attention: Fine-grained Language-guided Noise Filtering Network for Occluded Person Re-Identification

1 Introduction

Person Re-Identification (ReID) aims to identify and match the same target individual across different and nonoverlapping camera views [Ye et al., 2021]. However, people and objects often move randomly, and surveillance devices typically cover wide areas in the real-world scenario, which leads to a high likelihood of individuals being partially occluded. This occlusion creates a major challenge for person re-identification, as it introduces significant noise during feature extraction and feature matching. To cope with occluded ReID [Zheng et al., 2015b; Zhuo et al., 2018;

∗Corresponding Author

Hou et al., 2021], various strategies have been proposed to mitigate the effects of noisy information caused by occlusion. Common approaches, such as auxiliary models [Hou et al., 2021; Wang et al., 2022a; Dou et al., 2023] and attention mechanisms [He et al., 2021; Tan et al., 2022; Jia et al., 2023] help the model distinguish key information from occlusion-induced noise. PFD [Wang et al., 2022a] is a notable method that integrates the auxiliary model and attention mechanism, employing pose-guided feature disentangling to reduce occlusion noise by associating features with human body parts. While auxiliary models provide external information and attention mechanisms adaptively focus on unobstructed regions, their reliance solely on visual data often fails to completely filter out occlusion noise. This leaves occlusions still being misinterpreted as identity-relevant features, yielding less discriminative representations. As shown in Fig. 1, the attention maps reveal the varying performance of different methods under diverse occlusion scenarios. It can be observed from Fig. 1(a) that the attention maps generated by PFD highlight many irrelevant regions.

Inspired by the potential of vision-language models, recent works have explored the integration of textual descriptions with visual features to enhance feature extraction. As a representative work, CLIP-ReID [Li et al., 2023] deploys the vision-language model CLIP [Radford et al., 2021] to enhance the learning of visual features by training a set of tokens for each pedestrian ID. However, existing languageguided ReID methods are not well-suited for the occluded ReID task, as their prompts tend to capture global information from images, failing to provide the necessary contextual information in occluded scenarios. Furthermore, textual information in these methods is not fully utilized, as it is primarily employed for contrastive loss with images rather than explicitly guiding feature extraction. As shown in Fig. 1(b), although CLIP-ReID helps reduce noise in occlusion scenarios, its attention maps still frequently emphasize occluding objects and irrelevant background regions, retaining significant noise in the extracted features.

To address these limitations, we propose the Fine-grained Language-guided Noise Filtering Network (FLaN-Net) for Occluded Person Re-Identification. FLaN-Net innovatively employs a categorical attention mechanism to generate adaptive tokens that capture three distinct types of visual information: comprehensive descriptions of individuals, detailed visible attributes, and characteristics of occluding objects. Specifically, our method transforms image information into subject and detail tokens, and then represents each image with a occlusion-aware fine-grained textual prompt in the format:"A photo of a [S∗ ] person with [A∗ ] partially occluded by [O∗ ]." Here, S∗ denotes the primary subject token, which encapsulates the essential identity of the pedestrian. The attribute token A∗ and the context token O∗ serve as detail tokens, capturing identity-relevant attributes and occluding objects respectively. FLaN-Net learns detailed information for each pedestrian through the guidance of multiple learnable queries. Moreover, to ensure effective utilization of these prompts, FLaN-Net incorporates a cross-attention mechanism that dynamically aligns textual tokens with image patch tokens. This alignment enables the image encoder to focus on semantically relevant and visible regions of the pedestrian, filtering out noise caused by occlusions. Finally, FLaN-Net combines the features from the image encoder, text encoder, and cross-attention in a dynamic weighting fusion module to generate a robust feature representation for each occluded pedestrian. The dynamic fusion module assigns weights to each feature based on its uncertainty, highlighting the influence of more reliable features. As shown in Fig. 1(c), our method effectively highlights the visible regions of the pedestrian while filtering out noise from occluding objects, demonstrating improved focus and robustness in occluded scenarios. We summarize our contributions as follows:

• We introduce FLaN-Net, a novel method that employs a fine-grained language-guided mechanism to construct adaptive occlusion-aware prompts for occluded ReID. This technique enables meticulously detailed descriptions of the visible aspects of an individual, while effectively filtering out noise caused by occlusions.
• We propose an advanced multi-modal feature enhancement paradigm that combines a cross-attention mechanism with a dynamic weighting fusion module. This integration is designed to produce robust and discriminative representations of occluded pedestrians.
• Experimental results demonstrate that FLaN-Net significantly improves retrieval performance on both occluded and holistic ReID benchmarks, outperforming state-ofthe-art methods.

2 RelatedWork

2.1 Occluded Person Re-identification

Occluded Person Re-Identification (Occluded ReID) presents significant challenges due to the noise introduced by occlusions, which hinder the model's ability to extract and match features accurately. One common approach to mitigate this issue is the incorporation of auxiliary information, such as pose estimation [Hou et al., 2021; Wang et al., 2022a] and human parsing models [Gao et al., 2020a; Dou et al., 2023]. Another widely used approach involves attention mechanisms [He et al., 2021; Tan et al., 2022; Jia et al., 2023; Li et al., 2024], which enhance robustness by adaptively focusing on relevant regions of an image. To further aid the attention learning process, various data augmentation strategies, such as random erasing [Wang et al., 2022b] and artificially generated occlusions [Chen et al., 2021; Xia et al., 2024; Tan et al., 2024], have been incorporated to help the model better handle occlusions across diverse scenarios. However, these methods rely only on visual features, neglecting the potential of text information to help filter out noise. In recent years, language-guided methods [Li et al., 2023; Yang et al., 2024] have been explored to address the challenges of ReID. CLIP-ReID [Li et al., 2023] is the pioneering work that uses the CLIP [Radford et al., 2021] model to integrate textual prompts with visual features.

2.2 Image-to-WordMapping

In the field of text-to-image generation, [Gal et al., 2022] was the first to use novel pseudo-words in the word embedding space to represent an object or a style. Recently, this technique, known as textual inversion, has been widely applied to zero-shot compositional image retrieval tasks [Saito et al., 2023; Suo et al., 2024]. These methods map a reference image to a pseudo-token in the CLIP embedding space, which is then combined with a descriptive query to facilitate text-to-image retrieval. However, a limitation of these models is that they map the entire image to a single pseudo-token, which can introduce noise and overlook important details relevant to the retrieval task. Recent studies have proposed using learnable queries to capture fine-grained features within an image. For example, in instance segmentation task, learnable queries are used to explicitly represent an object's class, location, and mask [Dong et al., 2021]. In object detection task, learnable queries are used to capture object relationships and global context for parallel predictions [Carion et al., 2020]. Inspired by these works, this paper proposes incorporating learnable queries into occluded ReID tasks to perform finegrained textual inversion.

3 Methodology

We introduce the Fine-grained Language-guided Noise Filtering Network (FLaN-Net) for Occluded Person ReIdentification, which consists of two key components, as shown in Fig. 2. The first component, Occlusion-aware Fine-grained Prompt Construction, generates adaptive tokens that capture three types of visual information: comprehensive descriptions of individuals, detailed visible attributes, and characteristics of occluding objects. These tokens are used to generate fine-grained textual descriptions, which provide a comprehensive representation of the pedestrian. The second component, Multi-modal Feature Enhancement, integrates visual and textual information to improve feature extraction. This is achieved through a cross-attention mechanism that aligns textual prompts with specific image regions, and a dynamic weighting fusion module, which adaptively combines the features to ensure that the most relevant and reliable features are given larger weight.

3.1 Occlusion-aware Fine-grained Prompt Construction

To effectively represent occluded images, our method projects the image into pseudo-word tokens, including a subject-focused token S∗ for the individual's essential identity and detail-focused tokens A∗ and O∗ for visible attributes and occluding objects. These tokens are then combined to construct a personalized prompt:"A photo of a [S∗ ] person with [A∗ ] partially occluded by [O∗ ]." This occlusion-aware prompt enhances the model's ability to distinguish individuals in occluded ReID tasks.

Subject-focused Token Generation

In order to get the subject-focused token, we leverage the image encoder fv of the pre-trained CLIP model. Specifically, given an image I, the visual encoder extracts visual feature v = fv(I) = {vg, v1 l , . . . , vn l }. Here vg ∈ Rd×1 denotes the global visual feature and {vi l }ni=1 ∈ Rd×n represent the local patch features, where d is the feature dimension and n is the number of patches. Then we apply a simple mapping network ϕs to transform the global image feature vg into a subject-focused pseudo-word token. Formally, we define:

where ϕs is a three-layered fully-connected network and S∗ serves as a comprehensive description of the main subject.

Detail-focused Token Generation

The detail tokens are divided into two types: attribute tokens A∗ , which describe identity-relevant visible attributes, and context tokens O∗ , which represent characteristics about the occluding object. Both A∗ and O∗ are generated using the same network architecture, leveraging learnable queries to extract relevant features from the input image. To illustrate this process, we take the generation of A∗ as an example and provide a detailed explanation of its network structure.

Let Vl = {vi l }ni=1 ∈ Rd×n represent the local patch features. Next, these patch features are fed into a fine-grained noise filtering network. This network interacts with a set of m learnable queries X = {xi}mi=1 ∈ Rd×m through crossattention, allowing these queries to capture attribute information from the corresponding semantic regions in the image. For A∗ , this process enables these queries to focus on specific visual details, capturing exposed attribute features such as clothing colors, accessory types, and hairstyles. Similarly, for O∗ , these queries could extract information about the occluding objects. Specifically, we compute the query Qd, key Kd, and value Vd matrices as follows:

where WQ, WK, WV are different linear transformations and [X, Vl] denotes the concatenation of the learnable queries and local patch features, facilitating their interaction. The crossattention output Hi for the learnable queries in the i-th attention block is then computed as:

Subsequently, we feed Hi into a two-layer feed-forward network FFN(·), producing ˜Xi , which represents the updated features of the learnable queries after the i-th attention block. This process is formulated as:

Afterwards, we perform average pooling over the refined query embeddings after multiple transformer blocks. The result is then passed through a simple mapping network ϕa to derive the final attribute token A∗ . It can be formulated as:

where ˜Xfinal is the final output query embeddings frommultiple transformer blocks, AvgPool(·) denotes average pooing, and ϕa denotes a three-layer feed-forward network.

The above network produces the refined attribute token A∗ . Similarly, a parallel network with the same structure but different parameters is employed to generate the corresponding context token O∗ . In this network, ϕo serves as the associated mapping function.

3.2 Multi-modal Feature Enhancement

To effectively capture the intricate relationships between multi-modal information, we integrate a cross-attention mechanism with a dynamic fusion strategy, enhancing the model's capacity to represent nuanced identity features.

Cross-Attention Mechanism

The cross-attention mechanism allows the model to interactively align specific textual cues with corresponding visual regions, focusing on subject and detail tokens in the constructed prompt. Let T represent the constructed prompt "A photo of a [S∗ ] person with [A∗ ] partially occluded by [O∗ ]", which is then fed into the frozen CLIP text encoder ft to obtain the textual representation tp. Formally, this process can be expressed as follows:

To implement cross-attention, we treat the textual embedding tp as the query Qc, while the visual feature v extracted from the image serves as both the key Kc and value Vc. The output of the cross-attention mechanism can be expressed as:

By leveraging the cross-attention mechanism, the textual tokens dynamically guide the image encoder to focus on the visible and discriminative features of the pedestrian while suppressing noise from occluded regions.

Dynamic Weighting Fusion Module

To further enhance feature representation, we introduce a dynamic weighting fusion module that assigns adaptive weights to each feature based on prediction uncertainty, allowing the model to prioritize more reliable features. Specifically, the global visual feature vg, the textual feature tp, and the crossattention feature satt are fed into the fusion module. For each feature fm (m = 1, . . . ,M, where M = 3, representing the visual, textual, and cross-attention features), the associated weight λm is determined according to the uncertainty of feature fm, which is quantified by the entropy of its prediction distribution. The uncertainty Em is computed as follows:

where pm(k) is the softmax probability assigned to the k-th individual's feature fm, and C represents the total number of individuals in the dataset. A lower Em reflects less uncertainty in the prediction, resulting in an increased weight for that feature during the fusion process. The weight λm for feature fm is then determined as follows:

The final fused representation ˆ f is formulated as:

where fm represents each prediction feature. This adaptive fusion approach enables balanced contributions from the visual, textual, and cross-attention features.

3.3 Loss Function and Inference

Our framework incorporates four loss functions: Crossmodal Contrastive Loss LAlign [Radford et al., 2021], Triplet Loss LTriplet [Hermans et al., 2017], ID Classification Loss LID [Zheng et al., 2017], and ArcFace Loss LArcFace [Deng et al., 2019].

Cross-modal Contrastive Loss

To align visual and textual representations of each individual, we employ a cross-modal contrastive loss that encourages high similarity between images and their corresponding prompts in the embedding space. It is formulated as:

where P(n) denotes the set of positive samples that correspond to the same identity i and τ is a temperature parameter.

Triplet Loss

The triplet loss is employed to increase the distinction between identities by minimizing the distance between positive pairs while maximizing the distance between negative pairs. This loss is formulated as:

where dp and dn are the distances of the positive and negative pairs respectively and m is the margin.

ID Classification Loss

The ID classification loss ensures correct identification of each individual and is defined as:

where yk is the predicted probability of an individual, and qk is the corresponding ground truth label.

ArcFace Loss

ArcFace loss improves feature discrimination by introducing an angular margin to enhance the separation between different individuals and compact intra-individual representations:

where θj denotes the angle between a feature and the weight vector of the j-th identity. The angular margin m improves discrimination, and the scale factor s stabilizes optimization.

Total Loss

Overall, the loss function used in FLaN-Net is defined as:

where λ is a hyper-parameter that balances the contribution of LAlign to the total loss.

Inference

During inference, following CLIP-ReID [Li et al., 2023], we rely solely on features extracted from the image encoder for person retrieval. The Euclidean distance is computed between the feature of the query image and those in the gallery set to identify the closest matches. The proposed components work together to enhance the model's ability to accurately identify the target subject, ultimately optimizing the feature representation generated by the image encoder.

4 Experiments

4.1 Experimental Settings

Datasets and Evaluation Protocols

We evaluate the proposed FLaN-Net method on two categories of datasets: occluded datasets, including OccludedDuke [Miao et al., 2019] and Occluded-REID [Zhuo et al., 2018], and holistic datasets, comprising Market-1501 [Zheng et al., 2015a], DukeMTMC-reID [Zheng et al., 2017] and CUHK03-NP [Li et al., 2014]. As the Occluded-REID dataset lacks a dedicated training set, we utilize Market1501 for training, consistent with other methods to maintain a fair basis for comparison. To assess the effectiveness of our approach, we adopt Cumulative Matching Characteristic (CMC) curves and the mean Average Precision (mAP).

Implementation Details

In this work, we employ the ViT-B/16 pretrained on CLIP as the visual encoder, and the pre-trained CLIP text transformer as the text encoder. We use two independent fine-grained noise filtering networks to get A∗ and O∗ , each consisting of 3 learnable queries and 6 cross-attention blocks. The model is trained using a batch size of 64, consisting of 16 identities, each with 4 images. All input images are resized to 256 × 128 pixels. For optimization, we use the Adam optimizer with a base learning rate of 5e-5 for the randomly initialized modules and 1e-5 for the visual encoder. The model is trained for 60 epochs, with the learning rate decaying by a factor of 0.1 at epochs 20 and 40. The Triplet Loss uses the margin m = 0.3, and the ArcFace Loss is configured with the margin m = 0.5 and the scale factor s = 30. The optimizer for the Arcface Loss function is separately initialized with SGD, using a learning rate of 0.1 and a weight decay of 5e-4. The λ in Eq.17 is set to 0.5 for all datasets. All components are trained on a single NVIDIA RTX3090 GPU.

4.2 Comparison with State-of-the-Art Methods Experimental Results on Occluded ReID Datasets

To evaluate the effectiveness of our proposed FLaN-Net, we conducted extensive comparisons with various state-ofthe-art methods on the occluded ReID datasets, including Occluded-Duke and Occluded-REID, and show the results in Tab. 1. SOTA methods are divided into four mainstreams:

• Auxiliary model-based methods: PVPM [Gao et al., 2020b]; HOReID [Wang et al., 2020]; RFCnet [Hou et al., 2021]; HCGA [Dou et al., 2023].
• Transformer-based methods: PAT [Li et al., 2021]; TransReID [He et al., 2021]; DRL-Net [Jia et al., 2022];

PFD [Wang et al., 2022a]; DPM [Tan et al., 2022]; SAP [Jia et al., 2023]; OAT [Li et al., 2024].

• Data augmentation methods: OAMN [Chen et al., 2021]; FED [Wang et al., 2022b]; CAAO [Zhao et al., 2023]; ADP [Xia et al., 2024]; DPM-SPT [Tan et al., 2024].
• Language-guided methods: CLIP-ReID [Li et al., 2023]. The experimental results demonstrate that FLaN-Net achieves outstanding performance on both occluded ReID datasets. For the Occluded-Duke dataset, FLaN-Net achieves a Rank-1 accuracy of 75.2% and an mAP of 65.5%, outperforming the second best method, DPM-SPT, by +0.5% and +2.5%, respectively. On the Occluded-REID dataset, FLaN-Net achieves the best performance with at least +3.4% Rank-1 accuracy and +4.4% mAP compared to other methods. These results indicate that FLaN-Net effectively addresses the noise caused by occlusions.

Experimental Results on Holistic ReID Datasets

We also experiment our proposed method on holistic person ReID datasets, including Market-1501, DukeMTMCreID and CUHK03-NP. Tab. 2 shows the results on Market1501 and DukeMTMC-reID datasets. We compare FLaN-Net with two categories of methods:

• Holistic ReID methods: SAN [Jin et al., 2020]; TransReID; HAT [Zhang et al., 2021]; DCAL [Zhu et al., 2022]; AAformer [Zhu et al., 2023]; PHA [Zhang et al., 2023]; CLIP-ReID.
• Occluded ReID methods: RFCnet; PFD; FED; DPM; CAAO; HCGA; SAP; ADP; DPM-SPT; OAT.

We observe that our FLaN-Net achieves competitive results on the Market1501 dataset and achieves SOTA performance on the DukeMTMC-reID dataset. Compared with language-guided method CLIP-ReID, our method surpasses it by +0.2%/+0.9% Rank-1 accruacy/mAP on Market-1501 and +2.1%/+1.1% Rank-1 accruacy/mAP on DukeMTMC.

Additionally, we evaluate FLaN-Net's performance on the CUHK03-NP dataset under both manually labeled and auto-detected bounding box settings. Several methods are compared, including RGA-SC [Zhang et al., 2020], HAT, NetVLAD-M [Zhang et al., 2022], MPN [Ding et al., 2022], AAformer, PHA, OAT. As shown in Tab. 3, FLaN-Net still surpasses all other methods with significant margins. Specifically, it achieves at least +3.1%/+3.7% improvements in Rank-1 accuracy/mAP on the labeled setting and at least +3.7%/+4.5% on the detected setting. Though FLaN-Net is not explicitly designed for holistic ReID tasks, it still guarantees a comparable performance with most of holistic methods, underscoring its robustness and generalization capabilities.

4.3 Ablation Study

Ablation Study on Prompt Variations

We design a series of prompts to evaluate their impact on model performance, as shown in Tab. 4. From index 1 to 2, the addition of subject-specific S∗ improves Rank-1 accuracy from 72.6% to 73.4% and mAP from 63.7% to 64.4%, demonstrating the benefit of incorporating identity-specific information. Expanding the prompts with attribute details A∗ (index 3) further enhanced Rank-1 accuracy to 74.6% and mAP to 65.2%, underscoring the importance of fine-grained identity attributes. Finally, introducing occlusion details O∗ (index 4) yields the highest performance, with Rank-1 reaching 75.2% and mAP 65.5%. These results validate the effectiveness of occlusion-aware fine-grained prompts in guiding feature extraction under occluded scenarios.

Ablation Study on Model Components

In Tab. 5, we evaluate the contribution of occlusion-aware fine-grained prompt (F), cross-attention mechanism (C), and dynamic weighting fusion module (D) on Occluded-Duke. From index 1 to 2, the performance improves by +2.9% in rank-1 accuracy and +1.8% in mAP, demonstrating the importance of utilizing fine-grained descriptions to capture identity-relevant features. From index 1 to 3, cross-attention also shows its effectiveness. For index 4, combining F and C further boosts performance by +4.3% in rank-1 accuracy and +3.3% in mAP, highlighting the necessity of utilizing fine-grained prompts through cross-attention. Notably, Using fine-grained prompts alone (index 2) or applying crossattention with simple prompts (index 3) proves insufficient for optimal performance. Finally, from index 4 to 5, the addition of D results in an additional improvement of +0.9% in rank-1 accuracy and +0.6% in mAP, demonstrating the effectiveness of dynamically fusing features.

4.4 Model Analysis

Explore the Optimal Number of Learnable Queries

We investigate the impact of varying the number of learnable queries on model performance, as illustrated in Fig. 3. The results indicate that using 3 learnable queries yields the best performance, with both mAP and Rank-1 reaching their highest values. The performance initially drops as the number of learnable queries increases from 3 to 5. However, the performance begins to improve again as the number rises from 5 to 7 queries. Despite this, the performance gain remains modest compared to the increase in computational cost. Consequently, selecting 3 learnable queries strikes the optimal balance between performance and computational efficiency.

Effectiveness of Dynamic Weighting Fusion Module

To assess the effectiveness of our proposed dynamic weighting fusion module, we compare it against three alternative fusion methods: average weighting, summation, and concatenation. In the average weighting approach, each of the three features is assigned an equal weight of 1/3, disregarding their individual importance. The summation method combines the three features through element-wise addition, while the concatenation approach merges features along a specified axis. As shown in Tab. 6, our dynamic fusion method outperforms all these methods by adaptively assigning weights to each feature based on the entropy of its prediction distribution.

5 Visualization

To evaluate the model's ability to handle occluded images, we visualize attention maps generated by different methods, as shown in Fig. 4. The PFD is heavily distracted by occlusions, leading to less effective attention. CLIP-ReID demonstrates improved attention but still captures irrelevant areas in some cases. In contrast, our proposed FLaN-Net focuses precisely on the visible and identity-relevant regions of the pedestrian, avoiding interference from occlusions. This visualization highlights the robustness of FLaN-Net in handling diverse occlusion scenarios and its ability to focus on the most discriminative features for person re-identification.

6 Conclusions

In this paper, we propose FLaN-Net, an innovative framework specifically designed to address the challenges of occluded person re-identification. By employing a categorical attention mechanism, FLaN-Net generates fine-grained prompts capturing individual descriptions, visible attributes, and occluding object characteristics. The integration of crossattention mechanisms and a dynamic weighting fusion module enables the model to focus on core identity while mitigating the impact of occlusions. Experimental results across multiple datasets demonstrate that FLaN-Net achieves stateof-the-art performance, underscoring its robustness and effectiveness in challenging real-world scenarios. This work paves the way for the development of more advanced occlusionaware ReID solutions and highlights the potential of integrating vision and language for robust identity recognition.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 62472420.