Endogenous Recovery via Within-modality Prototypes for Incomplete Multimodal Hashing

1 Introduction

Hashing [Wu et al., 2019; Zhang12 et al., 2019; Zhang et al., 2020; Zhang et al., 2021; Zhang et al., 2022; Wu et al., 2022a;

∗Corresponding author

Wu et al., 2023; Wu et al., 2024] aims to transform highdimensional data into compact low-dimensional hash codes, thereby reducing retrieval complexity. As the scale of multimodal data rapidly expands, multimodal hashing [Shen et al., 2018; Lu et al., 2019b; Zheng et al., 2020; An et al., 2022] has garnered significant attention for its remarkable capability to enable large-scale multimodal retrieval with low storage costs and high retrieval efficiency. Unlike cross-modal hashing [Shen et al., 2020; Kang et al., 2023; Chen et al., 2024; Liu et al., 2024; Sun et al., 2024] which focuses on learning a shared space for different modalities to enable crossmodal search, multimodal hashing aims to create a complementary space by fusing different modalities for concise retrieval. However, multimodal hashing often assumes complete modalities for each sample, which is often unavailable in real-world scenarios. For example, in social networks, the privacy or security constraints on uploaded images and texts make it challenging to access multimodal data.

The primary challenge of this incomplete multimodal hashing is how to recover the miss modality data. As shown in Fig. 1(a), previous method like NCH [Tan et al., 2023] directly recovers the missing modality features from available ones of another modality, which usually introduces a optimized neural network in a generative manner. However, this inevitably leads to the distribution gap between the generated data and the original data of miss modality, which is also verified by the empirical results of T-SNE visualizations in Fig. 1(c). Clearly, NCH shows distinct distribution gap between the representations of the recovered text features (depicted as brown points) and the ground-truth text features (depicted as green points).

To address the aforementioned challenges, we design a novel modality recovery paradigm for multimodal hashing, named Prototype-based Modality Completion Hashing (PMCH). PMCH performs data recovery by adaptively aggregating learned within-modality prototypes. Specifically, PMCH consists of two key components: Within-modality Prototype Learning Module (PLM), and Weight-Net based Prototype Aggregating Module (PAM). The PLM first designs a set of learnable prototypes for each modality, then utilizes a relation-driven prototype loss to transfer the modalityand category-specific information from available features. The PAM utilizes a modality-specific Weight-Net to dynamically learn the coefficients from the available modality data, enabling adaptive aggregation of missing modality prototypes. Since the within-modality prototypes reside in the same representation space as their corresponding features, the domain gap issue empirically observed in previous methods can be alleviated. Employing prototypes within the same modality for recovering missing data significantly minimizes the semantic discrepancy between the reconstructed and original data. The main contributions of this paper are summarized as follows:

• We propose a novel incomplete multimodal hashing framework called PMCH, that completes missing modalities by adaptively aggregating learned withinmodality prototypes. PMCH can effectively mitigate the domain discrepancy between the recovered data and the authentic one.
• We meticulously devise a PLM module and a PAM module for modality and category-specific prototype learning and aggregation, respectively. The two modules can work seamlessly to directly reconstruct missing modality data with within-modality prototypes, thereby enhancing the authenticity of the recovered data.
• Extensive experiments conducted on three widely used benchmark datasets demonstrate that our PMCH achieves state-of-the-art performance in both complete and incomplete multimodal retrieval scenarios.

2 RelatedWork

The multimodal hashing methods aim to learn unified and semantic-rich binary representations by fusing different modalities [Liu et al., 2012; Song et al., 2013; Shen et al., 2015; Liu et al., 2015; Wang et al., 2015; Yang et al., 2017; Liu et al., 2018; Lu et al., 2019b; Lu et al., 2020; Tan et al., 2022; Zheng et al., 2022; Zhu et al., 2023; Zhu et al., 2024].

Until now, numerous attempts have been made to handle incomplete multimodal hashing retrieval scenarios. Depending on whether to restore missing modalities, existing solutions can be broadly categorized into non-recovery and recovery methods [Wang et al., 2023]. Non-recovery methods primarily aim to enhance the fusion model's robustness against missing modalities. For instance, Flexible Online Multimodal Hashing (FOMH) [Lu et al., 2019a] and Flexible Graph Convolutional Multimodal Hashing (FGCMH) [Lu et al., 2021] employ a self-weighted strategy to seamlessly fuse heterogeneous multimodal data. Supervised Adaptive Partial Multi-View Hashing (SAPMH) [Zheng et al., 2020] employs multimodal matrix factorization to learn shared and view-specific hash codes for complete and incomplete modalities respectively. While these non-recovery methods exhibit less sensitivity to modality missing compared to traditional complete multimodal hashing methods, there still exists a persistent fusion gap between complete-modality and partialmodality data. Recovery methods focus on explicitly estimating and reconstructing the missing modality data from the available modalities. Graph Convolutional Incomplete Multi-modal Hashing (GCIMH) [Shen et al., 2023] develops Graph Convolutional Autoencoder to complete partialmodality data with effective exploitation of its semantic structure. Neighbor-aware Completion Hashing (NCH) [Tan et al., 2023] designs a cross-modal generator implemented by Multi-layer perceptron (MLP) to directly generate missing modalities from available ones, and incorporates a neighboraware completion learning module to guide the learning of the recovery procedure.

Current recovery methods effectively bridge the semantic gap between complete and partial modality data, but overlook the domain gap between modalities. This results in recovered features occupying a distinct representation space from ground truth. NCH's neighbor-aware completion learning module generates domain-consistent representations for incomplete training data by randomly selecting complete training samples as anchors and aggregating them based on the similarity between the available modalities of the samples and anchors. However, it faces two issues: 1) Random anchors may miss categories, leading to recovered features deviating from ground truth [Tan et al., 2023]. 2) It overlooks semantic inconsistencies across modalities [Li et al., 2024] by directly transferring similarity relationships from the available modality to the missing modality. In this paper, we propose to aggregate within-modality prototypes for modality recovery. On the one hand, our learned within-modality prototypes ensure the recovered features align with authentic ones in the same space and integrates valuable information across all categories. On the other hand, our proposed modality specific Weight-Net mitigates the semantic inconsistency problem by adaptively learning the coefficients for aggregation from the available modality of data.

3 Methodology

3.1 ProblemFormulation

Similar to previous multimodal hashing methods [Tan et al., 2023; Shen et al., 2023], we primarily focus on two modalities: image and text. Suppose that our training dataset contains incomplete image and text multimodal data, their embeddings extracted by VGGNet [Simonyan and Zisserman, 2014] and Bag-of-Words (BoW) are denoted respectively as X = [Xp ,Xu ] ⊆ Rnx×dx and Y = [Y p , Y u ] ⊆ Rny×dy , where np samples have complete modalities, i.e., Xp ⊆ Rnp×dx and Y p ⊆ Rnp×dy , nxu samples have only image modality, i.e., Xu ⊆ Rnxu×dx and nyu samples have only text modality, i.e., Y u ⊆ Rnyu×dy . dx and dy are dimensions of image and text feature, and we have nx = np +nxu, ny = np + nyu. Additionally, the multi-label annotation is represented as L = [Lp ,Lxu ,Lyu ] ⊆ {0, 1}N×C , where N = np + nxu + nyu, C denotes the total number of categories. We utilize the Frobenius norm denoted as || · ||F for various computations. The objective of our proposed PMCH framework is to learn compact and fused H-bit hash codes bi ∈ {-1, 1}1×H for both complete-modality and incomplete-modality data, as shown in Fig. 2.

3.2 Within-modality Prototype Learning

This module learns within-modality prototypes by correlating multimodal data, involving feature projection, fusion, and relation-driven prototype learning.

Multimodal feature projection and fusion. We first map the extracted visual embedding X and textual embedding Y to the feature representation that shares the same dimension through visual encoders Hv: V = Hv(X; θHv ) and textual encoders Ht: T = Ht(Y ; θHt ), where V = [V p , V u ] ⊆ Rnx×K denotes the visual feature, T = [T p , T u ] ⊆ Rny×K represents the textual feature. K is the feature dimension. Then we fuse the visual and textual feature of complete training data to generate the complete fusion hash code as follows:

where Hf is the common hash function for hash projection, Hp ⊆ Rnp×H represents the fusion hash code of complete training data, H is the length of hash code.

Relation-driven prototype learning. Firstly, we devise a set of learnable prototypes tailored for each modality, denoted as P v = {pvc }Cc=1 ⊆ RC×K for the visual modality and P t = {ptc }Cc=1 ⊆ RC×K for the textual modality. Here, pvc ∈ R1×K and ptc ∈ R1×K represent learnable vectors that correspond to each category for their respective modalities, C denotes the class number. The dimension of the prototypes matches the dimension of their corresponding features, ensuring compatibility and efficient interactions during the learning process. Subsequently, we integrate prototypes from different modalities but belonging to the same category to create fusion prototypes. These fusion prototypes are then projected from their distinct representation space into a common hash space in a manner analogous to the projection of fusion features. Given learnable within-modality prototypes P v and P t , the fusion hash prototypes are obtained as follows:

where P h ⊆ Rnp×H represents the fusion hash prototype.

We further propose a relation-driven prototype loss to refine the hash prototypes. This loss function serves to pull relevant hash codes toward their corresponding prototypes while pushing away irrelevant hash codes, which is calculated in the following manner:

where I(·) is an indicator function. The relation-driven prototype loss facilitates the effective transfer of semantic information from fusion hash features to fusion hash prototypes. As these fusion prototypes are constructed through the integration of within-modality prototypes, category information is subsequently propagated to the within-modality prototypes during the backpropagation process.

3.3 Weight-Net based Prototype Aggregating

This module focuses on aggregating within-modality prototypes for modality recovery, encompassing TPAM for text and IPAM for image prototype aggregation.

Text Prototype Aggregating Module (TPAM). This module completes patrial samples containing only image modality by aggregating text prototypes. The coefficients required for this aggregation process are dynamically generated by the image Weight-Net, leveraging the accessible image features. During the training stage, this process can be expressed as follows:

where V u i denotes the incomplete training data with only image modality, ωtc ∈ R1×C represents the generated coefficient corresponding to each text prototype. Wv refers to the image Weight-Net parameterized by θwv . In our implementation, we adopt a MLP model with two hidden layers as Wv.

Once we have obtained the coefficients, we then utilize them to combine the text prototypes for feature recovery. The feature of the text modality can be reconstructed as follows:

where ptc ∈ R1×K represents the learnable text prototypes, ˜ T u i denotes the recovered text feature.

Image Prototype Aggregating Module (IPAM). This module is designed to handle the missing of image modality. Similarly, the text Weight-Net Wt is initially employed to generate the coefficients ωv c . These coefficients are then utilized to aggregate the image prototypes, thereby reconstructing the image feature, denoted as ˜ V u i

Modality reconstruction loss. We further propose a modality reconstruction loss to facilitate the learning of the modality specific Weight-Net and within-modality prototypes using the complete training data [V p , T p ], formulated as follows:

where V p and T p are the authentic visual and textual feature, ˜ V p and ˜ T p are the recovered visual and textual feature.

As the modality reconstruction loss aims to minimize the distance between the true features and those recovered through an adaptive aggregation of their respective prototypes, it can also implicitly ensure that the within-modality prototypes and their corresponding features reside in the same representation space.

3.4 Multimodal Hashing Learning

After restoring the missing modality features, we then integrate them with their associated available modality features to generate the hash code as follows: Hyu = Hm(˜ V u +

T u ; θHm ) and Hxu = Hm(V u + ˜ T u ; θHm ). Then the hash codes for the entire training set can be denoted as H = [Hp ,Hxu ,Hyu ] ⊆ RN×H , where Hp represents the hash codes of paired training data computed using Eq 1. Subsequently, we employ the pairwise similarity loss to learn discriminate hash codes as follows:

where Sij denotes the semantic similarity between the i-th sample and j-th sample. Sij is constructed as follows:

To minimize the quantization errors caused by sign(·) operator, the discrete hash code sign(Hi) is used to guide the learning process of the continuous hash code Hi as follows:

Finally, the overall objective function of our proposed PMCH architecture can be formulated as follows:

where α, β, and γ serve as hyper-parameters.

3.5 Out-of-Sample Extension

Our proposed PMCH architecture shown in Fig 2 can effectively handle both queries with complete modalities and those with missing modalities. Specifically, given a query containing both visual modality vq and textual modality tq , its corresponding hash code can be generated by the trained multimodal hashing network Hf as follows:

When confronted with a query containing only the visual modality, i.e., (vq , ∗) or solely the textual modality, i.e., (∗, tq ), PMCH employs modality specific Weight-Net to generate the coefficients for prototype aggregation from the available modality :

where m represents the missing modality, a represents the available modality.

Subsequently, the missing modality is reconstructed by aggregating its corresponding prototypes with the generated coefficients, as outlined below:

Finally, we integrate the recovered missing modality feature with the available modality feature to produce the final hash code, as detailed below:

niques, including five complete multimodal hashing methods which limited to handling complete multimodal data, i.e., DCMVH [Zhu et al., 2020], OASIS [Wu et al., 2022b], BSTH [Tan et al., 2022], SFISMH [Zhu et al., 2024], STBMH [Tu et al., 2024] and five incomplete approaches which are capable of processing not only complete but also incomplete multimodal data, i.e., FOMH [Lu et al., 2019a], FGCMH [Lu et al., 2021], SAPMH [Zheng et al., 2020], NCH [Tan et al., 2023], and GCIMH [Shen et al., 2023]. For the complete multimodal hashing methods, we directly model the incomplete multimodal data for training and query without recovery. We carefully reproduce these methods using their publicly available codes and adhere to the parameter settings specified in the original papers.

Evaluation Metric. Consistent with established multimodal hashing retrieval methods [Tan et al., 2022; Shen et al., 2023], we consider the widely-used metrics, i.e., mean Average Precision (mAP) to quantitatively assess the retrieval performance of our proposed approach. The mAP is calculated using all the samples in the database.

4.2 Comparison with State-of-the-Art Methods

We compare the proposed PMCH with state-of-the-arts on both incomplete and complete multimodal retrieval tasks.

Incomplete Query Set. This section evaluates the performance of PMCH when query data is incomplete. Following [Shen et al., 2023], we consider two missing-modality conditions: missing visual modality and missing textual modality. According to Table 1, we can obtain the following observations: 1) Generally, recovery methods show better performance than non-recovery methods. This demonstrates that reconstructing the missing modality data could bridge the semantic gap with complete modality data, thereby obtaining higher performance. 2) Our PMCH outperforms all baselines

4 Experiments

4.1 Experimental Settings

Datasets. In this study, we conduct experiments on three widely used benchmark datasets, i.e., MIR Flickr, NUSWIDE, and MSCOCO. The multimodal data includes image and text modalities. The images are extracted using VGGNet, while the text modality is represented using bag-ofwords vectors [Shen et al., 2023; Tan et al., 2023]. MIR Flickr [Huiskes et al., 2010] comprises 20,015 image-text pairs annotated with one or more of 24 categories, which are crawled from the Flickr website. NUS-WIDE [Chua et al., 2009] comprises 195,834 web images-text pairs associated with the 21 most prevalent concept labels. MS COCO [Lin et al., 2014] comprises 82,783 training samples and 40,504 validation samples, each associated with at least one of the 80 categories. We follow Tan [Tan et al., 2023] to split the three datasets into the training, validation, and test sets.

Implementation. In our proposed PMCH, the modalityspecific Weight-Net comprises two fully-connected (FC) layers, with the latent feature dimension set to 2048. Additionally, the visual and textual encoders are implemented using FC layers, with latent dimensions of 2,048 and 1,024 respectively. The outputs of these encoders have a dimension of 512. Furthermore, the hash fusion model incorporates a linear projection layer that takes in an input dimension of 512 and outputs a hash code length as specified. During the training stage, we set the batch size to 512 and run the iterations for 50 epochs. For optimization, we employ the Adam optimizer [Kingma and Ba, 2014]. Empirically, we set the learning rate to 0.05 for prototype learning and 0.001 for the remaining components.

Baselines. We compare our proposed method against ten state-of-the-art (SOTA) multimodal hashing techin both missing cases. This proves that PMCH could recover more authentic features for incomplete query data, as it alleviates the domain gap issue by adaptively aggregating withinmodality prototypes.

Incomplete Training Set. This section section evaluates the performance of PMCH when training data is incomplete. Following [Tan et al., 2023], we construct incomplete training data using five Partial Data Ratios (PDR). For example, "90%" indicates 90% of the data misses a modality, with visual and textual modalities each having a PDR of 45%. The mAP results of all baselines with respect to 32 bits are shown in Fig. 3. From Fig. 3, we can observe that PMCH obviously outperforms the baselines across all PDRs. In addition, as PDR increases, PMCH achieves relatively stable performance. These empirical results imply that the proposed PMCH can effectively handle the training of incomplete multimodal hashing by minimizing the semantic discrepancy between the reconstructed and original data.

Complete Multimodal Retrieval. This section evaluates the performance of PMCH on complete multimodal retrieval. We present comparison results on MIR Flickr with the top four baselines in Table 2. According to Table 2, although PMCH

Table 5: Ablation study on Prototype Aggregating Module.

is not specifically designed for complete multimodal retrieval, it still shows competitive performance in this setting. The reason lies in the joint learning of hash codes and their corresponding prototypes in PLM module, which not only learns within-modality prototypes, but also effectively draws relevant data closer, thereby improving complete multimodal retrieval performance.

4.3 Ablation Study

In this section, we perform ablation studies by substituting the modality completion strategies and key components with their respective variants. The experiment is conducted on Mir Flickr with 50% PDR applied to the training set, and either image or text modality is missing in the query set.

Variants on Modality Completion Method. In this section, we design three variants to prove the superiority of the proposed modality completion paradigm, including: 1) w/o completion. In this variant, we remove our completion learning and directly model incomplete multimodal data. 2) +GCAE w/o PMCH In this variant, following GCIMH [Shen et al., 2023], we use Graph Convolutional Autoencoder (GCAE) to complete the partial modality data. 3) +MLP w/o PMCH In this variant, following NCH [Tan et al., 2023], we introduce a Multi-layer perceptron (MLP) to generate the missing modality from the available one. From the retrieval performance shown in Table 3, two observations can be obtained: 1) Removing total completion learning will result in incomplete multimodal semantics and thus reduce retrieval performance. 2) Compared with the generationbased modality recovery method, our proposed prototypebased completion method performs the best. The main reason is that compared with the cross-modal generation, PMCH can avoid the separated representation space with true features when recovering missing modalities.

Effectiveness of Prototype Learning Module (PLM). In our PMCH, we design PLM to learn category-specific prototypes for each modality. In order to evaluate its effectiveness, following the setting of [Tan et al., 2023], we replace the learnable within-modality prototypes with complete anchors randomly selected from the training set. The number of anchors is set to 300. According to the results shown in Table 4, our PMCH performs better. The reason is that random anchors may miss categories. If all the anchors are dissimilar to the arrived incomplete samples, the recovered features will be distinct from the ground truth. While our within-modality prototypes learned by PLM integrate information across all categories, enabling the generation of appropriate features for partial samples belonging to any category.

Effectiveness of Prototype Aggregating Module (PAM). PAM employs the modality specific Weight-Net to dynamically learn the coefficients for aggregating missing modality prototypes. In order to evaluate its effectiveness, we replace it with three similarity-based aggregation methods mentioned in [Tan et al., 2023], which respectively employ KNearest Neighbors (KNN), Graph Attention Network (GAT) and Transformer Encoders (TEs) to compute similarities between the available modality of samples and the available modality prototypes. These similarities are then utilized to aggregate missing modality prototypes. As shown in Table 5, we can find that our PAM shows better performance than the other similarity-based prototype aggregating variants, which implies that the modality specific Weight-Net could dynamically learn the coefficients from the available modality of samples, enabling adaptive aggregation of missing modality prototypes, thus recovering more authentic features.

4.4 Further Analysis

Visualization. We visualize the recovered features and ground truth to qualitatively compare the generation-based recovery method NCH, GCIMH, and our proposed PMCH. To achieve this, we randomly select one label from each of the datasets, with a code length set to 16. We then project the features of the selected samples into a 2D space using tSNE [Van der Maaten and Hinton, 2008]. Additionally, we visualize the learned within-modality prototypes for the chosen label. The results on MIR Flickr are presented in Fig. 4. By analyzing the t-SNE results, we make two key observations: 1) The prototypes learned by PMCH reside in the same representation space as their associated modality features. 2) The representation space of features recovered by PMCH is significantly closer to the ground truth compared to NCH and GCIMH. These results underscore the effectiveness of our proposed PMCH method in avoiding the domain gap during the feature recovery process.

Parameter Sensitivity Analysis. We investigate the impact of three crucial hyper-parameters in our proposed PMCH method: α, β, and γ, using a hash code length of 32-bits on the MIR Flickr dataset. Fig. 5 depicts the performance variation curves on image modality, text modality and both modalities, respectively. Overall, our method demonstrates robustness as the mAP performance remains relatively stable within a certain range of hyper-parameter variations.

5 Conclusion

In this paper, we introduce a prototype-based modality completion method called PMCH for incomplete multimodal hashing learning. Specifically, we devise an Within-modality Prototype Learning Module that learns prototypes for each modality. Subsequently, we delicately design a modality specific Weight-Net to dynamically generate coefficients from available modality, enabling adaptive aggregation of missing modality prototypes. PCMH ensures recovered features align with authentic ones in the same space and adaptively integrates valuable information across all categories for precise recovery. Extensive experiments on three widely used multimodal retrieval datasets underscore the superiority of our proposed method from various aspects in both complete and incomplete multimodal retrieval scenarios.

Acknowledgments

This work was supported by the National Key R&D Program of China under Grant 2022YFB3103500, the National Natural Science Foundation of China under Grant 62472420, the Natural Science Foundation of Guangdong, 2025A1515011224.