A Solution to Co-occurence Bias: Attributes Disentanglement via Mutual Information Minimization for Pedestrian Attribute Recognition

1 Introduction

Pedestrian attribute recognition (PAR), as a key component of the pedestrian analysis stemming from development of the ubiquitous video surveillance, targets to determine the softbiometrics of local or semantic attributes of a person given its captured main-body image. To date, researches investigate PAR basically along the analogous routine of discriminative deep multi-label classification. They can be basically abstracted as emphasizing on better attribute localization to mitigate the accuracy drop from inferring on irrelevant area [Liu et al., 2018; Jia et al., 2022], or involving additional supervision or information, like pose keypoints [Liu et al., 2018] and pedestrian video clip [Chen et al., 2019; Ji et al., 2020], to guide PAR under explicit assumptions regarding body topological structure or temporal context, etc..

For work striving to enhance the attribute localization, [Fabbri et al., 2017; Li et al., 2017] split the body image vertically, by a manually defined fixed strategy, into three parts and feeds each part into an individual network for feature extracting. As a further step, [Liu et al., 2017] proposed the HydraPlus-Net, learning to locate attributes of different scale Cosine similarity between attribute anchor features with a multi-directional attention modules. Instead of learning to generate the attention map, [Liu et al., 2018] produces attribute-specific features by means of the class activation map (CAM) [Zhou et al., 2016], which is initially designed to build a generic localizable deep representation. However, same technique is applied in [Jia et al., 2021b] to demonstrate that, even without explicitly modeling the attribute-specific area, network is still able to locate attributes precisely, implying that the fundamental task for PAR should be instead to explore better feature learning.

Figure 1: Left: Illustration of a prediction failure resulting from biased attributes interdependency. Right: The Pearson correlation coefficient (Upper) of training set labels, and the cosine similarity between anchor features (Lower), for hat and short/long sleeve vs. other annotated attributes in PA100k. It reveals that the network precisely captured the data selection bias that Glass & ShortSleeve, Viewpoint-Back & ShortSleeve and Hat & Female, etc. are not prone to present simultaneously, and uses such biased correlations for attributes inference. The weights of the last fully-connected (FC) layer in a Resnet-50 are applied as attribute anchor features.

Seeking for better feature learning of information exchanges, a thread of literature [Wang et al., 2016; Wang et al., 2017; Zhu et al., 2017; Fan et al., 2020; Li et al., 2022] aims to enhance the modeling of interdependencies among attributes, under the hypothesis that such correlations provide a contextual constraint complementary to visual attributes recognition. To this aim, Graph-based methods are often used to explicitly model the attributes co-occurrence and estimate the joint label probability. However, the strong variability and unpredictability exhibited by attributes co-occurrence actually cast doubts on the robustness of these methods for scenarios out of the datasets, which are typically in practical applications. Worse, given the observation in Figure.1 that, such unreliable interdependency can be memorized by network, in a form of bias, even without explicit modeling of it, current methods would suffer in generalizing well onto realistic PAR, as evidenced in [Jia et al., 2021b].

Equipped with such perspective, this paper evolves in a novel spirit that we resort to infer attributes while entirely discarding their relations. Actually, it is a mechanism of PAR that accords with us human, say, we do not infer one's hat color by referring to even a single clue from its gender, instead, we look just at the hat for robustness, so should intelligent models. To embody this philosophy in stark contrast to the fragile mechanism of deep models for PAR, our attribute-disentangled learning for PAR is formalized as that the attribute-specific feature learned for predicting one attribute should not use information pertinent to other attributes, i.e., the mutual information between one attribute and other attributes' specific feature should be minimized.

To cope with the non-triviality of a direct mutual information minimization, an equivalent optimization-friendly training objective of it is deduced, which guarantees that the variation of other attributes' information results in no shift on the estimated posterior of a given attribute. Both mathematical insights and experimental evidence are provided, indicating that under this setting of the posterior-invariant learning, the proposed disentangled attribute learning can be attained, and thus tackle the issue of attribute co-occurrence bias.

Sequentially, we also propose an efficient training strategy for the posterior-invariant learning, which enables it implemented in a manner of fast convergence. Practically, unlike most work in this field building on large networks that deteriorates the applicability, our method's plug-and-play nature makes it cater to various models, with almost no extra computational burden thereon. Albeit lightweight, our method establishes the-state-of-art performance on various realistic datasets like the PETAzs and RAPzs [Jia et al., 2021b], with considerable margins over previous approaches.

Our contribution is summarized as three folds:

• With solid experimental evidences, we establish a novel perspective for understanding the learned attributes interdependency bias as the current bottleneck of PAR from achieving robustness. It is not covered by existing work and can serve as a mindset of future researches for rethinking PAR.
• We propose one direction of improvement as to infer the attributes by disregarding their correlations. From it, a lightweight prescription of information-theoretic attributes-disentangled feature learning is developed.
• Along with ablation studies, we present analytical experiments on various realistic PAR datasets to demonstrate our generic proposal's efficacy and a spectrum of superiorities, validating that the proposed method might serve as a foothold of robust PAR.

2 On the Attributes Co-occurence Bias

Fundamentally, although there exists certain pattern of interdependencies among attributes, such a phenomenon eslabel Index diff. of label correlation coeff.

Figure 2: Left: For the 10 attributes common between PA100k and PETA, the absolute difference of labels Pearson correlation coefficients computed on these two datasets. For better viewing, all values are arranged in an increasing order. Right: On the PETAzs dataset of zero-shot setting (no overlapped identities of training set in test set), distribution of the difference between attributes Pearson correlation coefficients on training set and test set.

sentially roots from conditioned statistical relevance rather than causality, implying that it can vary from scene to scene, individual to individual drastically. Thus, such variability and unpredictability inherent to the attributes co-occurrence make it hardly a universal known a prior to safely rely on, and could be immensely biased regarding the statistics of limited scenarios, e.g., some datasets exclusively involve indoor or outdoor scenes and constant season or whether condition, race or culture, etc. [Li et al., 2016; Liu et al., 2017].

Such a claim is supported by the observations in Figure.2 that, the correlations among attributes are likely to fluctuate between different datasets like PA100k [Liu et al., 2017] and PETA [Deng et al., 2014], or even mutate on different groups of identities from the same scenes. For instances, ShortSleeve and Trouser lean to not co-occure in PA100k, as images in this dataset are mostly captured during summer time. While for PETA, ShortSleeve and Trouser appear simultaneously with a significantly larger ratio, resulting in the interdependency discrepancy of these two attributes. Also, even for two groups of pedestrians from a same dataset PETA, about 1/6 attributes correlations exhibit distinct characteristic (absolute difference between Pearson correlation coefficients ≥ 0.1).

As a result, empirical risk dominates in the way that, existing work would have difficulty in generalizing the explicitly or implicitly leveraged attributes co-occurrence to other circumstances with different pattern of attributes interdependencies. Since an ideal dataset collection of pedestrian images, which captures global facets of the non-static myriad population distribution of attributes co-occurence, to soften such underlined bias, can be intractable, to enhance a robust PAR, a disentangled and discriminative feature learning for each attribute can be indispensable and consequential.

3 Method

Attributes Disentanglement by Mutual Information Minimization. Here, we establish the theoretical framework of our methods. Formally, it is supposed that there is a distribution X characterized by all of the pedestrian images. Under certain conditions, some data points {xi}Ni=1

are sampled from X, with their corresponding annotations {ai}Ni=1 of some predefined attributes, to form the train set D = {xi, ai}Ni=1 , where ai ∈ {0, 1}C and C is the total number of attributes. Specifically, if the latent embedding output from the feature extractor is denoted as f ∈ RK , we hope to decompose it as f = f 1 + f 2 + ... + f C , wheref s ∈ RK , s = 1, 2, ...,C, is the attribute-specific feature learned for predicting the attribute indicating random variable ys . From the information theory point of view, the mutual information I(yk ; f s ) measures the knowledge that could be told from the random variable f s about the random variable yk . Thus, to ensure the prediction of each attribute independent to the existence of others, for every f s , it should be satisfied that the mutual information I(yk ; f s ) = 0, for any k = 1, 2, ..., C, k ̸= s, which can be factorized as

where F s is the marginal distribution of f s . Noticably, if P(yk |f s ) = P(yk ), Eq.1 naturally holds since ∫ P(yk ) logP(yk ) dyk is independent of f s . Obviously, for any f s drawn formF s , if it satisfies that P(yk |f s ) = Q and Q is a constant, we have

Eq.2 reveals that, to satisfy P(yk ) = P(yk |f s ) is equivalent to ensure that for any f sa and f sb sampled from F s , it holds that P(yk |f s = f sa ) = P(yk |f s = f sb ). Note that

With this formulation, if the estimate of P(yk |f) = P(yk | ∑Cl=1 f l ) is invariant of f's component f s , for any s ̸= k, then we have I(yk ; f s ) = 0. Thus, a statement can be declared as follows

Goal: attribute-disentangled learning can be enabled as long as the probability estimate of P(yk |f) varies only with the yk 's specific feature f k .

Approach of Posterior-invariant Leanring. In practice, for the feature of attributes hybrid, extracted from a given input xi, we translate it first to obtain each attribute-specific components f si , s = 1, 2, ...,C, by a FC layer followed with nonlinearities, and fi is obtained by adding them up as fi = ∑Cs=1 f si . Sequentially, we randomly build C mappings G s i (·) : F s ↦→ F s to transform each f si into an arbitrary data point ˜ f si within f s 's domain of distribution F s , and generate a new feature vector of ˜ fi = ∑Cs=1 ˜ f si , where ˜ f si = G s i (f si ). We require P(ys = ys i | ˜ fi - ˜ f si + f si ) = P(ys = ys i |fi) to meet the goal of the proposed attributedisentangled learning that P(ys |f) should be tolerant to the variation of other attributes' specific features, and therefore to ensure no factors for decreasing the uncertainty of ys are conveyed by f k , ∀k ̸= s. Under the context of supervised PAR, P(ys = ys i |fi) is simply the ground truth label asi coupled with the input xi, and thereby the underlined attribute-disentangled learning can be enabled by leveraging the information-theoretic regularizer of

̂ P is the estimate of P(yk |f) given by a discriminative classifier, which is a common technique for approximating a posterior, and plugged right onto f with MLP operations in it. This framework is graphed in Figure.3a.

To specify the form of G s i (·), we opt for simplicity by adopting the convex linear combination between f si and f sr(i,s) , which is the extracted feature from a randomly picked training sample xr(i,s), as a plausible form among possible variants that satisfy the closureness of a transform, i.e., the mapped feature still lies within the domain of F s . Here, r(i, s) is simply a function that maps different i&s to a different sample index. It might be noticed that, with the convex combination of two attribute-specific features from training samples, points off of the training distribution can be generated. However, [Mikolov et al., 2013] has shown that this linear interpolation between hidden states is an effective way of transiting between learned factors to produce new indistribution semantics, making the combined features still informative of attributes thus reside within the F s . Specifically,

αs and βs are independently drawn from the uniform distribution U(0, 1) for each attribute. To fully explore Fi and produce more of the points with semantics potentially to be presented in the capricious test environment, for attributespecific features with identical label, we do the linear combinations to their norms and direction vectors, respectively. The rationale is that, since feature norm has been validated to be an effective measure of data uncertainty [Shalev et al., 2018;

Meng et al., 2021; Zhou, 2022] that is prone to be fluctuated by bad image quality (often in PAR), the style semantics of attribute (color, size, shape, etc.) might be mostly encoded into feature's direction, i.e., it's normalized unit feature [Wang et al., 2018]. Hence, such a norm-direction-separated linear combination can generate features from attributes of one style but shot under various uncertainties, exploiting more of the domain of Fs, exemplified in Figure.3b.

Efficient MixUp-based Alternative. Notably, directly optimizing over Eq.3 necessitates C inferences though the classifier for each sample in every update, which can be computationally inefficient when the number of attributes grows larger. Regardless of the ease for implementing G(·) as linear combination, such a design enjoys another merit as it actually supplies us with an approach to integrate the training objective of Eq.3 into a desirable single-inference-every-instance manner of

Such a simplification from Eq.3 to Eq.4 is inspired by the well-known data augmentation method MixUp [Zhang et al., 2017], which shows that the linear combination of images is actually aligned with exactly the same linear combination of their ground truth in the label space. Here, we adopt an analogous idea into Eq.4 to reduce the training cost.

Please note that Eq.4 differs from MixUp in two aspects. First, during every update, MixUp uses a single α and a single sample-to-interpolate for each input, purely for the sake of data augmentation. Whereas in Eq.4, for each attribute's specific feature, we use feature interpolation of different αs , βs and f sr(i,s) to guarantee that the transform imposed over it is totally independent thus differentiated from others. By requiring the variation of a certain attribute's posterior aligned exclusively to the variation of its specific feature, we enforce other attribute's specific feature acquires no factors informative to the given attribute's posterior, exactly the same way for realizing attribute-disentangled learning by Eq.3. Second, instead of pixel space, the linear combination is conducted in the space of decoupled attribute-specific features. One might argue that such a strategy can bring non-trivial boost in performance by somewhat working as a data augmentation of MixUp. We admit this concern. However, we will present in the experiment section that it is actually not the case here for achieving SOTA performance as MixUp can not foster any accuracy increase over the baseline. More importantly, as will be presented in the experiment section, the equivalent Eq.3 suffices to deliver similar performance w.r.t Eq.4, which does not optimize over the interpolated feature by its correspondingly interpolated label.

In a nutshell, attribute-disentangled learning is secured by our method from two sides. One, given the specific feature of all attributes, we render the posterior estimate of a certain attribute exclusively associated to the feature component of its own, tasking the rest specific features, on which other attributes are inferred, to preserve no clues informative about this given attribute. Second, serving as an additional virtue of Eq.4, the biased attribute interdependencies are strongly dismissed from classifier learning owing to the scheme that the ground truth employed in Eq.4 is randomly generated for each attribute, and therefore can be of any possible pattern in term of attributes co-occurence, making classifier no way to trace on the attribute correlations embedded in the limited dataset, and thus generalize better. Efficiently, both sides are achieved in a holistic, end-to-end manner.

4 Experiments

Data and EvaluationMetric. For the benchmark datasets, PETA [Deng et al., 2014], along with the two largest public pedestrian attribute datasets RAP [Li et al., 2016] and PA100k [Liu et al., 2017], are adopted for evaluation. Detailed dataset information and usage are consistent to those in [Jia et al., 2021b]. We also test our methods on two realistic datasets of RAPzs and PETAzs stated and released in [Jia et al., 2021b]. As for the evaluation metrics, the label-based metric mean Accuracy (mA), which takes an average over all attributes' classification accuracy on the positive and negative samples, and two instance-based metrics Recall and F1-score (F1) are considered. we do not present Precision since it can be basically inferred when Recall and F1 are told.

Network and Training Details. We adopt Resnet-50 [He et al., 2016] and ConvNeXt-base [Liu et al., 2022] as backbones to study the efficacy and compatibility of our method under feature extractors of both classical and up-to-date designs. For the training details, Adam solver is applied without Nesterov momentum. The learning rate starts at 1e-4 and decays by a factor of 10 in a manner of multistep. If not specifically stated, the results in this section are produced by Eq.4 with efficiency. We refer readers to the code of this work in the supplementary material for further details.

Benchmark Results. Following the benchmark protocol provided in [Jia et al., 2021b], our method is compared with recent notable SOTA approaches MsVAA [Sarafianos and Kakadiaris, 2018], MTMS [Gao et al., 2019], ALM [Tang et al., 2019], MTA-Net [Ji et al., 2020], AR-BiFPN [Tan et al., 2020], VAC [Guo et al., 2020], JLAC [Tan et al., 2020], SSC [Jia et al., 2021a], DAFL [Jia et al., 2022] and Label2Label [Li et al., 2022]. Since we start at a training setup that involves less data augmentation methods, the baseline performances on these datasets could be about 1% mA inferior to some of the compared methods. To mitigate this gap, we also present experimental results of applying the data augmentation settings akin to JLAC (additional random scaling, rotation, translation, cropping, erasing and adding random gaussian blurs). The overall results are reported in Table.1. It highlights that across all settings, our prescription achieves performance at least comparable to others. Please note that our method is highly efficient since for test samples, the extra operations over the baseline models, are only one single FC layer applied to produce attribute-specific features, in stark contrast to the prior arts paying a premium in term of computational cost. Therefore, one potential of it is that, one could just make full use of the model parameters being largely saved by our lightweight framework, to plug a better-but-wider backbone, like ConvNeXt for further significant improvements.

Basically, one might notice that our proposal works on PA100k better than RAP and PETA. It is not surprising, since, as pointed out by [Jia et al., 2021b], about 31.5% and 57.7% of pedestrian identities in the test set of RAP and PETA are identical to those in their respective training set. Thus, for PETA and RAP, memorizing pattern of the biased attributes co-occurence in training set can be conducive for model's test-set performance, making existing approaches overestimated on them. For this, we also report in Table.1 the experimental results on the PETAzs and RAPzs, which are respectively formed from PETA and RAP to follow the zero-shot setting of pedestrian identities, i.e., no overlapped identities between their training and test sets. Therefore, the results on PETAzs and RAPzs, along with the results on PA100k, are much convincible [Jia et al., 2021a], and thus should be attached of more practical implications, on which our method outperforms others with considerable margins. Also, it should be noticed that the concepts of existing works only fuel trivial boost on these realistic datasets, implying that previous development of PAR on PETA and RAP may partly come from better modeling of the common bias in training and test sets. Overall, our method excels in both the recognition performance and practical applicability of realistic PAR.

Ablation Studies. In this sub-section, we investigate the effectiveness and characteristics of each technical contribution. To better discern between attributes classifiers, we use PA100k and ResNet-50 in the following ablation studies.

Posterior-invariant learning vs. MixUp-based efficient learning. In this paper, we present two pipelines for attributes-disentangled feature learning, the one is directly derived from the goal of posterior-invariant learning to minimize the mutual information between one certain attribute and its irrelevant features specific to other attributes, as described in Eq.3. While effective as shown in the first row of Table.2, it requires multiple inferences for a single sample, which hinders the training efficiency and increases the memory usage when the number of attributes is large. To address this issue, a second pipeline for efficient single-inferenceevery-instance is introduced in Eq.4. It extends the concept behind MixUp to significantly lower the training cost, and achieves performance similar to Eq.3, as can be seen in the third row of Table.2. During our multiple rounds of experiments, we find that Eq.4 can always yield slightly better mA, and we credit such improvement into that the biased attribute correlation within dataset is totally discarded during the training of classifier in Eq.4, since the labels used in it are attribute-wise randomly generated thus can form any possible pattern of attribute co-occurence, making classifier hardly to memorize the limited attribute co-occurence presenting in the dataset, and thereby generalize better, as shown in Figure.4.

Norm-direction separated interpolation vs. plain feature interpolation. We apply the proposed norm-direction separated interpolation in Eq.3 and Eq.4 to fully exploit the possible variations of attribute-specific features, as graphed in Figure.3b. The results in Table.2 clearly demonstrate that this trick, in tandem with both pipelines, outperforms the plain interpolation used in previous work and thus can be applied in other related fields of research like recognition with unstable image quality for enhanced robustness and performance.

Mixup vs. Proposed Methods. Eq.4 directly optimizes over the interpolated features with their correspondingly interpolated labels, in a similar way as that of Mixup. Therefore, it might be questioned that our method's superiority can fundamentally come from tailoring on MixUp. However, as shown in Table.2, sample interpolation is not the case here for bringing significant performance boost, since Mixup delivers even negative increase in mA. Moreover, our pipeline of Eq.3, which does not optimize over the augmented samples, can score comparatively with Eq.4. It implies that, what essentially works for Eq.4 is the strategy that we use feature interpolation to enable a random and differentiated transform over each attribute's specific feature, and empower model with the consequential mutual information minimization learning by Eq.3 or Eq.4. On a higher level, data augmentations are methods creating data points beyond the training set to reduce the various bias embedded in dataset. Under such perspective, it brings no bad to comprehend our method as a data augmentation trick that augments the pattern of attributes co-occurence, out of the less representative dataset.

Analysis on Improvements. In Figure.5, we draw the perattribute accuracy results on PA100k dataset of the proposal and the baseline work. We find that our work is capable of sizably raising scores on almost all attributes. It might serve as a relief to the likely concern that discarding totally the use of attributes correlation could be detrimental to the recognition of some certain type of attributes, for which a further study is presented in the next parts. Generally, it can be seen that improvements are considerable on attributes with positive ratio closer to 1 or 0. It is expectable as attributes with inadequate chances to appear or disappear are likely to be limited in the captured or non-captured scenes and identities in term of the attributes interdependency. By our framework, on one side the inference dependence of an imbalanced attribute to its frequently or hardly co-occurred attributes is greatly undermined, reducing the bias within feature learning. On another side, aforementioned for Figure.4, the memorization effect of attributes correlation in classifier is also repressed. Thanks to these two factors, the recognition of attributes with most potential to be correlated can be facilitated utterly, whilst for attributes appearing evenly, the improvements can be trivial.

It also accounts for the reason that our method works better on label-based metric than instance-based metric, since the growth of mA is actually bottlenecked by the imbalanced attributes, while instance-based metrics are relatively not.

On the Attribute InterdependencyModeling. Intuitively, for attributes not causally independent to each other, discovering and utilizing the relations among them might be conducive for robust information exchanges and propagation in PAR. For this, turning back to seek help from previous work that precisely models the attributes interdependency seems suboptimal, since without elaborate expert knowledge use, there is no feasibilities of learning, for these methods to discern the welcomed attributes interdependency from the intricate attributes co-occurence bias, and exclusively dismiss use of the latter. Noting in Figure.5, the baseline model is error-prone not only for attributes appearing barely, but those occurring often like 'Age18-60' (positive ratio above 0.9) so, such an indiscriminative modeling of all attributes correlations is overall detrimental to most imbalanced attributes, making PAR strongly bottlenecked.

Moreover, there is another drawback of learning with attribute interdependencies - even the causally robust ones like the mutual exclusiveness among ages groups ( 'age < 18', '18 ≤ age ≤ 60' and 'age > 60' in PA100k are of a multi-class relation rather than that of multi-label), for which we call the infestation of attribute independency. As shown in Figure.4, two attributes tending to (not to) co-occur with each other would also prefer to (not to) co-occur with other attributes, representing as the dark or light stripes spanning across the anchor feature similarity matrix. Whereas these traversing lines do not show up in the Pearson correlation coefficient matrix of labels, it is actually by-produced during learning of the attributes correlations. This inclination of hallucinating new interdependency bias from the dataset attributes interdependency is smoothed out by ours, greatly yet not totally, as there are still lines can be eyed in the heatmap.

Towards Robust PAR. Ideally, a PAR of robustness should be capable of inferring on a-prior concrete attributes interdependencies while disregarding the others. Solutions of ease towards this aim can stand on the base of attributesdisentangled feature learning. Taking the age groups in PA100k as example, on a trained model of attributesdisentangled learning, one could train an additional classifier head of multi-class only for age prediction (namely, RobustA), or conduct certain back-end processings onto the outputs to rectify or unify the age predictions (Robust-B). Here, we realize both in Table.3. For Robust-B, we output only the age attribute of highest confidence score. Compared to our method, these robust PARs yield negligible accuracy increase in test set. However, we still emphasize on the importance of such designs since they secure the reliability of PAR, and might make difference in realistic conditions.

5 Conclusion

We present a novel method of attribute-disentangled feature learning to enhance the robustness of PAR. Without any sophisticated or time-consuming frameworks, leveraging a simple information-theoretic regularizer, it is ensured that the pedestrian attributes are inferred without considering the biased attributes interdependency inherent to dataset, in order to enable an attribute recognition mechanism respecting us human. The comprehensive experiments demonstrate that our proposal reaches SOTA performance with appealing merits like better generalizability and applicability in realistic scenarios. Importantly, our theorized perspectives of attribute disentanglement learning differs from the paradigms of previous methods, even advancing in an opposite direction, heuristically exploring a promising avenue for future work.

Acknowledgments

This work was partially supported by the "Pioneer" and "Leading Goose" R&D Program of Zhejiang (Grant No. 2023C01030), the National Natural Science Foundation of China (No.62122011, U21A20514), and the Fundamental Research Funds for the Central Universities. (Corresponding Author: Hai-Miao Hu)