Absorbing Gradient Conflicts: Modeling Semantic Variance via Kent Distributions for Cross-Modal Hashing

1 Introduction

Driven by the surge in multimodal data [Xu et al., 2024], Cross-Modal Hashing Retrieval has emerged as a standard method for large-scale information retrieval [Cao et al., 2022; Singh and Gupta, 2022; Yan et al., 2025; ?]. By projecting high-dimensional features into a compact Hamming space, this approach circumvents the prohibitive computational and storage costs of conventional Euclidean methods, enabling efficient similarity approximation via bitwise XOR operations [Wang et al., 2022; Lu et al., 2019; Sun et al., 2022]. Despite rapid progress towards Supervised Deep CrossModal Hashing Retrieval-leveraging deep neural networks and semantic supervision for end-to-end representation learning-fundamental challenges still persist. Real-world data are inherently multi-faceted, with single instances often associated with multiple overlapping labels. This multi-label characteristic poses severe difficulties for existing supervised frameworks, which struggle to capture complex, non-exclusive semantic correlations, necessitating more robust modeling strategies.

∗Corresponding Author

Despite the dominance of proxy-based strategies in deep cross-modal hashing [Tu et al., 2023; Huo et al., 2024b], fundamental limitations persist in multi-label scenarios. The prevailing paradigm typically abstracts each semantic category as a singular, deterministic point [Teh et al., 2020;

Kim et al., 2020; Movshovitz-Attias et al., 2017; Qian et al., 2019] within the embedding space. While effective for singlelabel tasks, this rigid zero-variance assumption provokes severe gradient contention when confronted with the complex semantic conflicts inherent in label co-occurrence. Specifically, in a point-proxy framework, a multi-label sample simultaneously exerts pulling forces on divergent proxies. Modeled as rigid points, these proxies lack the spatial capacity to accommodate such conflicting semantic signals. Consequently, this unresolved contention drives the model towards optimization collapse. These two limitations manifesting in two degenerate states: (1) Chaotic Proxy Oscillation, where the proxy, unable to satisfy conflicting gradients, is forced to constantly shift its position, preventing the model from anchoring to a stable semantic center; and (2) Proxy Collapse, where the optimization process forcibly pulls proxies of distinct categories together to minimize aggregate loss-particularly causing lowfrequency tail classes to collapse onto high-frequency head classes-thereby eliminating fine-grained decision boundaries and impairing discriminative power.

To address these challenges, we propose the Kent-based Distributed Proxy Hashing (KDPH) framework. We depart from the single-point paradigm by modeling each class proxy as a learnable Kent distribution-an anisotropic spherical probability model defined by a mean direction and an elliptical covariance structure. Unlike rigid point proxies that must drastically shift position to satisfy conflicting gradients, this distributed representation resolves conflicts by changing specific directional variances, thereby endowing each class with a semantic shape and volume to flexibly absorb the contextual variance introduced by label co-occurrence. Specifically, observing the anisotropic geometry of the Kent distribution, we design a tailored loss function to achieve gradient decoupling mechanism by stretching shape rather than moving centroid. This approach effectively resolves conflict, preventing chaotic oscillation and proxy collapse. Furthermore, by incorporating the Cayley transform to enforce strict orthogonality for the principal axes and utilizing this probabilistic module solely to guide the learning process, KDPH achieves state-of-the-art performance with zero additional computational overhead during inference, successfully resolving the persistent trade-off between retrieval accuracy and efficiency.

In summary, our contributions are as follows:

• We introduce KDPH, a novel framework that reformulates proxy-based hashing by substituting rigid point proxies with anisotropic Kent distributions. To the best of our knowledge, KDPH is the first work to demonstrate that modeling proxy uncertainty through directional distributions can eliminate the optimization collapse caused by gradient contention in multi-label cross-modal retrieval.
• We derive a manifold-constrained loss function leveraging the Cayley transform, which enables proxies to adaptively absorb gradient conflicts via directional variance rather than centroid displacement, thereby effectively eradicating Proxy Collapse and suppressing Chaotic Proxy Oscillation during the learning process.
• Extensive experiments demonstrate that KDPH achieves

state-of-the-art performance on three benchmarks, empirically validating the efficacy of our gradient decoupling strategy and showing superior robustness particularly in complex multi-label scenarios.

2 RelatedWork

2.1 Supervised Deep Cross-Modal Hashing and Proxy Learning

Supervised deep cross-modal hashing has advanced significantly, positioned such as CNN-RNN and adversarial baselines [Jiang and Li, 2017; Li et al., 2018; Bai et al., 2020]. Recently, graph and transformer-based models [Xu et al., 2019; Tu et al., 2022; Liu et al., 2023; Chen et al., 2024; Liu et al., 2024] have further pushed the boundaries of performance. In parallel, these models improving their algorithmic design from simple pairwise constraints [Qin et al., 2024] to adaptive gradient-triplet losses [Zhu et al., 2025a] or sophisticated geometric [Qin et al., 2025b] strategies which ranging from spherical mutual information to boundarysensitive mining [Qin et al., 2025a]-to refine the Hamming space structure. To address the scalability bottlenecks of pairwise comparisons, proxy-based metric learning has been effectively incorporated. By representing categories as distinct point proxy, this paradigm allows the model to efficiently organize the embedding space [Tu et al., 2023; Huo et al., 2024b], and has recently explored hyperbolic geometries for hierarchical modeling [Huo et al., 2024c]. However, a major challenge persists that the deterministic point formulation's inherent zero-variance assumption fundamentally limits its applicability to multi-label scenarios with high contextual variance.

2.2 Probabilistic Embeddings and Non-Isotropy

Probabilistic embedding learning has emerged as a robust alternative to deterministic point estimates [Lin et al., 2018]. To respect the unique geometry of hyperspherical feature spaces, the field has evolved from early Euclidean Gaussian models [Shi and Jain, 2019] to spherical probability density functions, such as the von Mises-Fisher (vMF) distribution [Zhe et al., 2018; Park et al., 2019; Li et al., 2021]. However, the efficacy of vMF-based approaches remains constrained by their inherent isotropy. One major challenge is the implicit assumption of uniform variance in all directions and this rigid simplification fails to accommodate the complex, anisotropic semantic manifolds inherent in multi-label data. Although recent metric learning approaches have explored non-isotropy through lightweight mechanisms, such as variance scaling or geometric regularization [Roth et al., 2022; Kirchhof et al., 2022], these low-parameter heuristics often lack the sufficient geometric flexibility required for large-scale multi-label cross-modal hashing, where semantic manifolds are densely entangled. To bridge this gap, we propose the KDPH, which introduces the parameter-rich Kent distribution to deliberately increase geometric capacity. We further facilitate stable training by resolving gradient contention and optimization collapse via the Cayley transform and a tailored loss function.

3 Method

3.1 Formulation and Architecture

Given a training dataset O = {(in, tn, ln)}Nn=1 with images in, texts tn, and multi-label vectors ln ∈ {0, 1}C , we extract D-dimensional semantic features f in and f tn via a dual-stream Transformer backbone initialized with pre-trained CLIP. Modality-specific hashing networks HI,HT : RD → RK then project these features into K-dimensional continuous representations hin and htn . The final binary hash codes are obtained via the sign function:

3.2 Kent-based Distributed ProxyMechanism

Conventional hashing methods typically abstract each class as deterministic point whose rigid zero-variance assumption fundamentally unsuitable for multi-label scenarios, and conflicting semantic signals from co-occurring labels lead to gradient contention and optimization collapse. To resolve this, KDPH shifts the paradigm from static points to distributional proxies on the hypersphere [Wang and Isola, 2020]. By modeling each class as a flexible Kent distribution with learnable centroid and variance, our framework effectively absorbs the gradients variance arising from label co-occurrence.

Kent Distribution for Proxy Modeling

To instantiate this distributional paradigm, we employ the generalized Kent distribution. This strategic choice addresses the inherent limitations of simpler spherical models, such as the von Mises-Fisher (vMF) distribution. While vMF offers mathematical convenience, its isotropic nature rigidly imposes uniform variance across all dimensions, rendering it ill-suited for capturing the complex, directional manifolds typical of multi-label features. In contrast, the Kent distribution is intrinsically anisotropic. It provides the necessary geometric flexibility to model elliptical contours on the hypersphere SK-1 , where K denotes the hash code dimension, distinct from the feature dimension D.

Formally, the probability density function (PDF) for a class c is defined as:

where c(κ, B) denotes the normalizing constant. The distribution is parameterized by Θc = {µc, κc, Γc, Bc}, in which each component serves a distinct semantic role: the mean direction µc ∈ SK-1 acts as the semantic centroid representing the stable core identity of the class, while κc governs the global concentration of samples around this centroid. Crucially, to model non-uniform variance, Γc defines the orthogonal principal axes, and Bc = {βc,i} controls the anisotropic shaping along these dimensions. By learning distinct βc,i values, the proxy can dynamically stretch or compress along specific semantic directions. This mechanism allows the distribution to accommodate the high semantic variance introduced by multilabel co-occurrences without drastically shifting the centroid µc.

3.3 Hash learning

To jointly optimize the deep networks and proxy parameters Θc while respecting manifold geometry, we introduce a differentiable reparameterization technique. This mechanism is integrated with a Kent-based log-likelihood scoring function and a multi-task objective to strictly enforce both semantic discrimination and quantization constraints.

Orthogonal Frame Construction via Cayley Transform

Directly optimizing µc and Γc is challenging due to the strict orthogonality required by the Kent distribution. To address this, we construct a unified orthonormal basis using the Cayley Transform. Specifically, for each class c, we learn a skewsymmetric parameter matrix Ac ∈ RK×K . By exploiting the mapping between the Lie algebra so(K) and the Lie group SO(K), we generate a strict rotation matrix Qc:

This transformation ensures that Qc remains orthogonal regardless of the updates to Ac. We then define the proxy components by slicing Qc: the first column serves as the centroid µc, while the following columns form the shape axes Γc. This parameterization naturally enforces geometric constraints, guaranteeing orthogonality not only between the centroid and the axes but also pairwise among all principal axes within Γc, effectively bypassing the need for expensive orthogonalization procedures.

Anisotropic Energy Scoring and Parameterization

To evaluate the semantic affinity between a query sample and the class proxies, we first project the continuous hash code h onto the unit hypersphere, z = h/∥h∥2 ∈ SK-1 , ensuring geometric consistency with the directional statistics.

While the canonical Kent distribution defines a rigorous probability density function, its optimization is hindered by a computationally intractable normalization constant dependent on high-dimensional Bessel functions. Since our primary objective in hashing is discriminative ranking rather than exact density estimation, we circumvent this bottleneck by reinterpreting the log-density formulation as a geometric Compatibility Score. We define the anisotropic energy score S(z, c) for a class c as follows:

Here, we employ a logarithmic regularization term R(κ, β) = log(κc + ∥βc∥1). This term serves as a soft magnitude constraint, effectively transforming the probabilistic objective into a flexible quadratic energy metric on the hypersphere, preserving the topological benefits of Kent geometry while avoiding the gradient instability of Bessel approximations.

To strictly enforce the definition of concentration and magnitude, the parameters κc and βc,i are constrained to be positive via a Softplus activation on learnable scalars ˆκc and ˆ βc,i. Crucially, regarding the shape constraints, while the canonical Kent distribution requires 2|βc,i| < κc for uni-modality, we adopt a generalized Fisher-Bingham formulation by removing this upper bound. By allowing the network to learn βc freely, the proxy gains the flexibility to model complex, potentially multi-modal distributions. This relaxation is particularly advantageous for capturing the high variance and distinct semantic manifolds inherent in multi-label data.

Distribution-Aware Proxy Triplet Loss

To strictly enforce discriminative margins within the hyperspherical manifold, we employ a structure-preserving Distribution-Aware Proxy Triplet Objective. Unlike standard cross-entropy which treats classes as independent points, this loss explicitly optimizes the anisotropic energy alignment between samples and learnable Kent distributions. By maximizing the energy score for relevant classes while minimizing it for irrelevant ones, the objective ensures that an instance resides firmly within the high-density acceptance region of its ground-truth semantic distributions, effectively pushing decision boundaries away from the ambiguous overlapping zones.

Formally, given a normalized query sample z, a positive proxy distribution cp ∈ P (where P is the set of ground-truth labels), and a negative proxy cn ∈ N, we enforce a strict separation constraint based on the energy score S(·):

where m is a fixed margin hyperparameter governing the minimum semantic gap.

To accommodate the multi-label setting where a sample is associated with multiple semantic centroids, the final loss aggregates these constraints over all valid triplets. This formulation allows the gradients to dynamically adjust the shape parameters βc of the Kent distributions, absorbing the conflict from hard negatives:

Multi-Modal Irrelevance Regularization

To strictly separate semantically disjoint samples, we incorporate the Multi-Modal Irrelevance Loss [Huo et al., 2024b]. This objective penalizes high similarities between irrelevant pairs-defined as instances sharing no common labels (i.e., li · lj = 0). Taking the image intra-modal regularization as an example, the loss is defined as the average cosine similarity over all such pairs in a batch:

where I(·) is the indicator function and B is the batch size. The text intra-modal loss Lreg t and the cross-modal loss Lreg it are computed analogously by substituting the continues hash feature pairs with (hy , hy ) and (hx , hy ), respectively. The final regularization term aggregates these three components: LReg = Lreg i +Lreg t +Lreg it.

Cross-Modal Contrastive Alignment

To bridge the heterogeneity between modalities, we employ the symmetric InfoNCE [He et al., 2020] objective to project representations into a shared, modality-invariant hypersphere. This contrastive loss maximizes the mutual information between matched image-text pairs. The image-to-text alignment loss is formulated as:

where τ is a learnable temperature parameter scaling the distribution sharpness. The text-to-image loss Lt→i is computed symmetrically by swapping the anchor and positive samples. The final cross-modal alignment objective is the sum of both directions: LCM = Li→t + Lt→i.

Uniform Distribution Constraint for Hashing

To maximize code entropy and prevent bit-collapse, we enforce a Uniform Distribution Constraint via Optimal Transport. Specifically, we minimize the Wasserstein-1 distance between the learned batch codes Bx and a sampled ideal uniform distribution A ∈ {-1, 1}B×K . This is formulated as finding the optimal permutation matrix Ox that minimizes the transport cost:

where O is the set of all B×B permutation matrices. The constraint is applied symmetrically to both modalities, yielding the total loss LU = LU,x +LU,y.

Overall Objective Function

Our comprehensive training objective LTotal is a multi-task function, designed to integrate the four distinct components discussed previously. It simultaneously optimizes the model for (1) proxy learning, (2) cross-modal alignment, (3) semantic irrelevance separation, and (4) high-entropy hash code generation.The final objective is a weighted summation of these components:

where ϵ, ζ, and η are scalar hyperparameters that balance the contribution of each term.

4 Experiment

To demonstrate the validity of our proposed KDPH framework, we conducted comprehensive experiments on three publicly available cross-modal multi-label datasets MIRFLICKR25K, NUS-WIDE and MS COCO. Besides, we introduce the datasets for the experiments and explain the details of the implementation of KDPH and evaluate metrics.

4.1 Experimental Settings

Datasets

We evaluate KDPH on three standard multi-label datasets. MIRFLICKR-25K [Huiskes and Lew, 2008] contains 24,581 image-text pairs across 24 classes. NUS-WIDE [Chua et al., 2009] comprises 269,648 pairs; following standard protocols, we utilize a subset of 195,834 pairs from 21 frequent categories. MS COCO [Lin et al., 2014] consists of 123,289 images, each with 5 captions, spanning 80 categories. Across all datasets, we randomly sample 5,000 pairs as the query set and use the remainder as the database, from which 10,000 pairs are uniformly sampled to construct the training set.

Baseline Methods

In our experiments, we selected 9 state-of-the-art deep cross-modal hashing methods for comparison, which contain DCPH[Tu et al., 2023], nivMF[Kirchhof et al., 2022], DCHMT[Tu et al., 2022], MIAN[Zhang et al., 2023], DSPH[Huo et al., 2024b], MITH[Liu et al., 2023], DNPH[Huo et al., 2024a], DNpH[Qin et al., 2024], DDBH[Qin et al., 2025a], DNcH[Zhu et al., 2025b] and DAGtH[Zhu et al., 2025a].

Experimental Details

We implemented KDPH in PyTorch on an NVIDIA A800 GPU, optimizing via Adam (LR=0.001, batch size=128). The

• (a) Retrieval Performance vs. Semantic Complexity
• (b) Precision of 'Hard-to-Find' Labels

Figure 3: Impact of Semantic Complexity on Model Performance.

hyper-parameters are set as ϵ = 40, ζ = 0.3, η = 0.1. Regarding the anisotropic geometry, we set the number of principal axes as Naxes = K/8, where K denotes the target hash code length (e.g., Naxes = 8 for 64-bit codes). To ensure a fair comparison, we follow all baselines that use the identical CLIP backbone and data splits.

4.2 Main Results

To evaluate the effectiveness of our KDPH framework, we conduct a comprehensive comparison with state-of-the-art baselines on three benchmark datasets. The comparative results are summarized in Table 1. The proposed KDPH method outperforms all state-of-the-art baselines with significant performance improvements across all hash code lengths. This phenomenon demonstrates that the anisotropic Kent distributions possess stronger feature extraction and discriminative capabilities than the deterministic point-based baselines.In particular, our KDPH yields superiority over nivMF, a representative approach employing non-isotropic von Mises-Fisher distributions. It is observed that such methods suffer from the limitations of geometric inflexibility, particularly in scenarios enriched with complex semantic entanglements. This limitation is empirically evidenced by the substantial margin of over 7.92% on the challenging MS COCO dataset (I2T @ 16 bits).Despite the reliable performance achieved by previous methods, our KDPH benefits from the Kent-based distributional proxy and modeling elliptical semantic variance, demonstrates superior performance.

Furthermore, to verify the performance of our KDPH under highly compressed spaces, we investigate the Bit-Efficiency. In comparison to the state-of-the-art methods (e.g., DCPH, DSPH) which suffer from sharp performance degradation under limited encoding capacity, KDPH maintains high discriminative power at 16 bits. Specifically, our KDPH yields an improvement of approximately 6.07% on MS COCO over the second-best method. These findings suggest that the meticulously designed Kent distributions offer superior information density, effectively absorbing gradient contentions even within highly compressed Hamming spaces. KDPH also achieves good balance between modalities, indicating that the optimization strategy successfully harmonizes visual and textual distributions within the shared latent space, thereby preventing the modality domination often observed in existing baselines.

4.3 Analysis

Mitigation of Chaos Proxy Oscillation via KDPH

We first investigate the robustness of our KDPH approach in preventing optimization collapse. So we monitor the track of Proxy Flip Rate R(t) flip [Xie et al., 2016], defined as the ratio of samples changing their nearest proxy assignment between consecutive epochs. As illustrated in Figure 4, it can be observed that in Low Co-occurrence Classes (Fig. 4b), both the baseline and KDPH exhibit stable convergence, confirming that rigid point proxies suffice for simple, unimodal manifolds.In contrast, regarding High Co-occurrence Classes (Fig. 4a), the baseline suffers from severe oscillation and jagged precision curves, leading to a deficiency in optimization stability. Benefiting from the advantages of anisotropic Kent distributions, KDPH maintains exceptional stability with a flip rate < 1%. Specifically, our framework is designed to dynamically absorb gradient variance through shape deformation rather than through straightforward centroid shifting. In this way, the proposed method effectively mitigates the chaotic oscillations that plague traditional point-based methods, ensuring better adaptability to complex semantic environments.

Prevention of Proxy Collapse via the KDPH

To comprehensively evaluate the effectiveness in prevention of proxy collapse, we track the Subordinate Label as the category with the least frequency in an sample. As illustrated in Figure 3, Baseline suffers from the limitations in high-complexity situations where rigid point proxies degrade rapidly under the pressure of dominant classes. In contrast, KDPH method consistently demonstrates the superiority and robustness over state-of-the-art competitors, yielding a significant improvement of 41.1% in scenarios with 5 labels. This promising performance confirms that the meticulously designed anisotropic distributions effectively mitigate the collapse issue by reserving sufficient semantic variance for tail concepts.

Despite the reliable performance achieved by the Baseline in single-label regimes due to the advantage of point proxies, it is noted that this advantage vanishes in multi-label scenarios. This slight redundancy introduced by KDPH serves as a necessary trade-off, providing the parametric flexibility required to bridge the semantic gaps between conflicting manifolds.

Ablation Study on Model Components

We conduct ablation studies by systematically evaluating the mAP of each component in our framework on the MS COCO dataset. The comparative results are summarized in Table 2. The performance metric exhibits a distinct hierarchy. Specifically, mAP decreases from 0.7693 with the isotropic vMF distribution (β = 0) to 0.7615 with rigid point proxies, suggesting only a marginal advantage for distributional proxies. Crucially, our anisotropic Kent model surpasses all state-of-the-art variants, owing to its capacity to capture anisotropic semantic conflicts, enabling effective recalibration of semantic interactions to resolve gradient contentions.Furthermore, the Multi-Modal Irrelevance Regularization LReg and Cross-Modal Alignment LCM are designed to collaboratively enhance the contextual awareness and semantic capabilities, ensuring better adaptability to nuanced semantic variations while Uniform Constraint LU provide essential complementary gains for modality invariance and code entropy.

Visualizing Distributions Across Multi-label Scenarios

To empirically verify the topological superiority of KDPH, we visualized the embedding space of the complex Person category on MS COCO using t-SNE (Figure 5). The Baseline (Fig. 5a) exhibits severe proxy collapse, where proxies of distinct co-occurring classes (e.g., Car, Bicycle) physically collapse onto the central anchor. This visually confirms that rigid point proxies suffer from gradient contention, eliminating decision boundaries. In sharp contrast, KDPH (Fig. 5b) reveals a clear semantic-adaptive topology. The embedding space disentangles into coherent radial clusters based on highlevel affinity-grouping animate entities, vehicles, and food along distinct directional axes. This structured layout demonstrates that our anisotropic Kent proxies successfully capture latent semantic hierarchies, accommodating contextual diversity without destabilizing the core identity.

5 Conclusion

In this paper, we address the critical bottleneck of gradient contention in multi-label cross-modal hashing with the Kentbased Distributional Proxy Hashing (KDPH) framework. By shifting the paradigm from point proxies to anisotropic Kent distributions, our method enables proxies to absorb contextual conflicts by dynamically adjusting directional variance. This geometric flexibility allows the model to maintain stable semantic centroids while stretching to accommodate diverse label correlations. Extensive experiments on three benchmarks demonstrate that KDPH outperforms state-of-the-art methods.

Acknowledgments

This work was supported by the Institute of Information Engineer ing, Chinese Academy of Sciences, under Grant E4101311F3, E4V06811F3. The reimbursement for this work was covered by this grant.