Negative Prompt Driven Complementary Parallel Representation for Open-World 3D Object Retrieval

1 Introduction

With the gradual increase in 3D data, 3D object retrieval (3DOR) has emerged as a central area of interest within computer vision [Krause et al., 2013; Gao et al., 2012]. The essence of 3DOR lies in establishing the relationship between the query and target datasets through training. Although recent advancements have significantly propelled the development of 3DOR, most existing methods operate under a closed-set assumption. This suggests that all object categories encountered in the testing phase have been previously seen during training [Chen et al., 2022a]. However, in practical open-world applications, training sets often fail to encompass all potential categories, owing to limitations of data volume and insufficient labeling [Vaze et al., 2021]. In openworld scenarios, the abundance of data coupled with incomplete labels makes it challenging to acquire sufficient and precise positive information for 3DOR. Against this backdrop, negative information, being more readily accessible than positive labels, can serve as effective hints or guidance [Wang et al., 2023; Chen et al., 2022b] within retrieval tasks. However, there is still a lack of research on the utilization of negative information for open-set retrieval.

∗Corresponding author: Yue Gao

Prompts, as essential components in representation learning [Liu et al., 2023] and generative models [Wu et al., 2022], have the capability to represent information, whether it is positive or negative. Recently, there has been an increasing application of prompt-driven methods in the realm of computer vision tasks [Jia et al., 2022; Bahng et al., 2022; Sung et al., 2022]. However, most current methods for computer vision generate the prompt embeddings only from positive labels unidirectionally, which is a challenging path to implement in an open-world environment. Besides, typical open-world learning approaches are designed to generalize invariant knowledge from seen to unseen categories directly, employing specific approaches such as predicting the Outof-Distribution (OOD) and In-Distribution (ID) scores [Yu et al., 2020], or structured-aware knowledge learning [Feng et al., 2024]. However, these methods still struggle to overcome the substantial distribution gaps [Zhou, 2022; Parmar et al., 2023]. In this paper, we explore the use of negative prompts to bridge the gap of distributions between seen and unseen categories, aiming to enhance the generalization performance for open-world 3D object retrieval.

Distinct from direct open-set retrieval methods, the prompt-driven open-set retrieval method focuses more on fine-grained similarity in feature representations and demands greater ability to constrain and guide object embeddings using prompt information. This leads to several challenges for prompt-driven open-set retrieval, including: First, the difficulty of achieving sufficient embeddings from limited prompt information. Prompts for retrieval serve as cues for coarse categories in the open-world space, rather than simply classifying between OOD and ID [Wang et al., 2023], thus offering broader possibilities. Hence, there is a strong motivation to extract more information from a limited set of prompts. Second, the difficulty in achieving embedding unification across different spaces, which involves aligning and fusing the features of prompts with 3D object representations from diverse modalities into a unified space. Third, the difficulty in generalizing the prompt-based embeddings to unseen categories, which requires achieving spatial generalization of feature representations under the guidance of prompt information.

Addressing the aforementioned challenges, we explore a method for open-world retrieval tasks through the lens of Negative Prompts. As shown in Figure 1, we introduce the Negative Prompt Driven Complementary Parallel Representation framework (NPCP) for open-world 3D object retrieval. On one hand, to tackle the challenge of sufficient prompt embedding, we utilize the Parallel Exclusive Embedding (PEE) to fully leverage the prompt information from both explicit negative and implicit positive directions. On the other hand, to overcome the difficulty in embedding unification and generalization, we construct a complementary hypergraph based on the bi-directional and cross-category correlations. This hypergraph captures the mutually exclusive and complementary structure between negative and positive information. Besides, we adopt the Complementary Structure Tuning (CSL) approach to exploit high-order correlations among objects for category generalization by the complementary hypergraph structure. Our contributions are summarized as follows:

• We explore a method to navigate the complexities of open-world 3D object retrieval through the lens of Negative Prompts, and we construct four datasets with multiple negative prompts for benchmarking.
• We propose the NPCP framework for prompt-based open-world 3D object retrieval tasks, including the Parallel Exclusive Embedding (PEE) and the Complementary Structure Tuning (CST) modules, which are designed to fully leverage the prompt information in both positive and negative directions and overcome the distribution deviation of categories.
• We propose a complementary hypergraph structure to capture high-order correlations among objects, guided

by the bi-directional and cross-category correlations.

• Extensive experiments are conducted on the four benchmarks for evaluation, demonstrating the superiority of NPCP over current state-of-the-art 3D object retrieval methods.

2 RelatedWork

2.1 3DObject Retrieval

Traditional 3D object retrieval methods are based on the close-set assumption, which means the training set and testing set share the same category distribution space. In the retrieval phase, all categories of objects in the testing (query and target) set have been seen in the training phase. Current closeset 3D object retrieval methods can be divided into two categories according to the modality representation of objects: i.e., single-modal retrieval and multi-modal retrieval.

Single-modal 3D object retrieval refers to detecting similar objects within a single modality of 3D data. [Su et al., 2019] and [Wei et al., 2020] construct a view-based graph model to generate the aggregated embeddings for retrieval frommultiview. [He et al., 2018] propose a triplet-center loss to pull objects from the same category closer together and push objects from different categories farther away. HGNN [Feng et al., 2019] proposes a hypergraph-based structure-aware method to capture the high-order correlations among objects and generate better embeddings. As for multi-modal retrieval, existing methods [Nie et al., 2019; Liang et al., 2021; You et al., 2018; You et al., 2019; Dong et al., 2020; Bai et al., 2021] propose the weighted fusion or feature fusion network to generate the aggregation embeddings from different modality-specific basic features. Besides, CMCL [Jing et al., 2021] designs the cross-modal center loss to reduce the difference across different 3D modalities by the common center embeddings.

2.2 Open-World Learning

Open-world (open-set) learning aims to do the machine learning research in open-world scenarios where important factors are subject to change [Zhou, 2022]. Most existing methods focus on the recognition problem, [Vaze et al., 2021] introduce a benchmark for open-set recognition, named Semantic Shift Benchmark (SSB). [Zhou et al., 2021] proposes a "none-of-above" classifier to detect whether the sample belongs to the seen categories or not. [Chen et al., 2021] introduces an adversarial-based method to minimize the overlap of known distributions and unknown distributions without loss of known classification accuracy. Furthermore, more open-set recognition methods are proposed for 3D object learning [Bendale and Boult, 2016; Joseph et al., 2021; Alliegro et al., 2022; Zhu et al., 2023]. Different from the recognition task, retrieval tasks in the open-world are more practical. While only a few methods [Feng et al., 2024; Liu et al., 2024] address the open-set 3DOR task, they just focus on structure learning networks and overlook the specific conditions in the open-world.

Parallel Exclusive Embedding (PEE)

Complementary Structure Tuning (CST)

2.3 Prompt Learning

Prompt learning aims to justify pre-trained models to downstream tasks [Radford et al., 2021; Li et al., 2022] by modus operandi methods, and it has achieved remarkable success in Natural Language Processing (NLP) [Radford et al., 2021; Shin et al., 2020] tasks such as GPT series. Recently, prompt learning has been increasingly applied in computer vision studies [Jia et al., 2022; Bahng et al., 2022; Li et al., 2022; Hegde et al., 2023], which is treated as the task-specific embedding instead of model parameters. In this paper, we focus on extracting sufficient information carried by prompts, and the generalization for open-world scenarios.

3 Problem Setup

3.1 Open-World Retrieval

Given the query set Dq of 3D objects, the 3D object retrieval (3DOR) task is to find the matched or similar objects from the target set Dt. The key for the 3DOR task is to find the relationship between the query and the target database through the training set Dtrn. Each 3D object is represented by (oi, yi), the oi = {mr}Mr=1 denotes a 3D object represented by M modalities, such as multi-view, point cloud, voxel and others. The yi ∈ Y = {cj}Yj=1 indicates the category label associated with the 3D object oi.

In the open-set retrieval assumption, the training set Dtrn and testing set Dtes = {Dq,Dt} are drawn from the different distributions, which means during the retrieval phase, every category of objects present in the query set has not been encountered and learned in the training phase. Specifically, for the training set Dtrn = {(oi, yi)}Li=1 and testing set Dtes = {(oi, ˆ yi)}Ri=1 = {Dq,Dt}, the category spaces of the training set and the retrieval set are not the same indicating yi ∈ Y = {cj}Yj=1 , ˆ yi ∈ ˆ Y = {ˆ cj } ˆ Yj=1 , and Y ̸= ˆ Y.

3.2 Negative Prompt

Each 3D object can be classified according to the basic geometric forms bi ∈ B = {c∗j }Bj=1 it contains, which is inherently independent of the dataset. Face the emergence of the open-world environment, it is much easier to determine the object category from a negative perspective, i.e., geometric forms that the object must not contain or does not belong to. We term this negative cue as negative prompt:

where ni denotes a negative prompt of object oi, which is in the form of one-hot. {1, · · · ,B} \ c∗j denotes the set of category labels whose correct labels are removed. Sr(·) is proposed to select elements randomly from the set.

Consequently, the negative-prompt driven open-world 3DOR task aims to design a method using the training set Dtrn = {(oi, ni, yi)}Li=1 and then use to search similar objects of the query in the testing set Dret = {(oi, ni, ˆ yi)}Ri=1 = {Dq,Dt}. The negative-prompt driven open-world 3DOR aims to minimize the expected risk:

where Di = (oi, ni, ˆ yi) and Dj = (oj, nj, ˆ yj) are object instance sampled from the query set Dq and target set Dt, 1{·} is the indicator function, which returns 1 if the expression is true and 0 otherwise. f := oi|ni → zi is the model that maps the 3D object oi into a fusion embedding zi ∈ Rd under the condition of negative prompt ni. H is the hypothesis space of map f(·|·). ∥∥ · ∥∥2 is the L2 norm function for distance metric, which could measure the Euclidean distance between two embeddings.

4 Methodology

4.1 Framework Architecture

As shown in Figure 2, the architecture of NPCP consists of two modules: Parallel Exclusive Embedding (PEE) and Complementary Structure Tuning (CST). The framework takes basic features of the different modalities of 3D objects as input.

The PEE module is designed to generate the prompt embeddings bilaterally from both the explicit negative and implicit positive directions. Next, in the CST stage, the complementary hypergraph structure is constructed based on the exclusive and cross-category correlations. Guided by this structure, hypergraph convolution is adopted to leverage the high-order correlations based the bi-directional and cross-category correlations. Finally, the tuned embeddings are distilled and generated by the memory bank for retrieval or other downstream tasks.

4.2 Parallel Exclusive Embedding

To fully leverage the information conveyed by negative prompts, the PEE module is designed here. Specifically, the PEE is composed of a multi-modal auto-encoder and two parallel prompt auto-encoders (negative and positive). The multi-modal auto-encoder is employed for integrating features of different modalities to obtain the object embeddings of 3D objects. The two prompt auto-encoders are utilized to get the bi-directional (negative and positive) embeddings parallelly, from the explicit negative and implicit positive information carried by prompts.

Bi-Directional Embedding

As shown in Figure 2, the PEE module takes the extracted basic features {rk i }Mk=1 (ri ∈ RN×dr ) of N instances and M modalities. Object embeddings ui are generated by the multimodal auto-encoder Am({rk i }Mk=1 ), where ui ∈ RN×du and Am(·) denotes the fusion function of the multi-modal autoencoder. Given a negative prompt ni for the i-th 3D object oi, we represent it as the index of form that the object does not contain. The PEE module first encodes it as a negative encoding eni with the same size as ui.

For each object oi, the negative auto-encoder An first compresses negative encoding eni aligned with object embedding ui into the negative-prompt space Sn , then does the reverse reconstruction while mapping it to a prompt mask for positive auto-encoder Ap through Multilayer Perceptron (MLP) layers. Specially,

where cni ∈ Rdu and ˆuni ∈ Rdu denotes the negative prompt embedding and reconstructed feature, and xni ∈ Rdc denotes the negative mask for positive auto-encoder. The encoder and decoder are defined as λn := Snx → Sn and ωn := Sn → Snx , which map the representation between negative-prompt space Snx and negative-mixed space Snm. Γ(·) denotes the MLP layers for mask generation.

During the embedding for negative information, the positive auto-encoder Ap also leverages the opposite information using a negative mask. Specifically, Ap compresses the object embedding ui to positive-prompt embedding under the guidance of prompt mask xni and does the reverse reconstruction. For better representation,

where cpi ∈ Rdc and ˆupi ∈ Rdu denote the negative prompt embedding and reconstructed feature, the encoder and decoder are defined as λp := Spx → Sp and ωp := Sp → Spx , which map the representation between positive-prompt space Spx and positive-mixed space Spx .

Finally, the PEE module generates the object embedding ui, negative embedding cni , and positive embedding cpi for each object oi.

Loss Function

To better construct the prompt embeddings for both negative and positive auto-encoders, we adopt the Dual BinaryEntropy loss Lde for An and Ap .

Dual Binary-Entropy Loss. Given a negative prompt ni = { ¯ c∗j }Bj=1 of the 3D object oi, the object may contain or belong to any other forms other than negative prompt as described in Section 3.2. The implicit positive label can be denoted as ¯ ni = {1, ..,B} \ ni, which means all forms except negative prompt. For each 3D object, the explicit negative and implicit positive labels are mutually exclusive and complementary, and together constitute a complete form label set ni ∪ ¯ ni = {1, ..,B}. To guide the negative and positive embedding upon prompt information, the dual binary-entropy loss can be defined as follows:

where pni,k = e ˆuni,k

∑Am=1 e ˆuni,m indicates the prediction score of that the 3D object oi does not contain the k-th form (negative label), and ppi,k = e ˆupi,k ∑Am=1 e ˆupi,m denotes the positive prediction. Both predictions are classified from the reconstruction feature ˆuni and ˆupi . ni,k is the k-th value of the one-hot encoded explicit negative label, and ¯ni,k is the k-th value of multi-hot encoded implicit positive labels, and B is the number of basic forms.

JointOptimization. In the PEE stage, the overall loss function is given:

where α are the hyper-parameter to trade-off between the loss of object and prompt embeddings, and Lmm denotes the multi-modal fusion loss defined by Homology Loss Lhm and Bi-Reconstruction Loss Lbr following [Feng et al., 2024], which can be calculated by Lmm = Lhm + Lbr .

4.3 Complementary Structure Tuning

Although the PEE module generates the negative and positive embeddings from the prompts, the open-world learning paradigm is frequently affected by the distribution gaps across seen and unseen categories. As shown in Figure 2, we proposed the CST module for generalization. Specifically, the complementary hypergraph is constructed to model the high-order correlations among objects in terms of category observability and prompt commonality. After structure construction, the combination of hypergraph convolution and memory bank is adopted for structure-aware smoothing and distilling to get the tuned embedding of each object.

Complementary Hypergraph Generation

To get the most out of potential correlation information from category observability and prompt commonality, the complementary hypergraph structure is designed here. As shown in Figure 3, we use a complementary hypergraph from two perspectives: bi-directional correlations and cross-category correlations.

A hypergraph can be represented as G = {V, E}, where V and E are the vertex set and the hyperedge set, respectively. In our complementary hypergraph, each vertex is composed of the object embedding ui and bi-directional prompt embeddings cni and cpi of each object. For better representation, we define the vertex of our complementary hypergraph as:

where cni and cpi denote the negative and positive embedding of the prompt, respectively. ui denotes the object embedding and δ is the hyper-parameter for trade-off.

Bi-Directional Hyperedge. As discussed in Section 4.2, each negative prompt encapsulates dual information: the object is excluded from this specific form and concurrently included in one of the alternative forms. As shown in Figure 3, bi-directional hyperedges are constructed based on this mutually exclusive information. Specially,

where Pv(n) denotes the vertex subset that shares the same negative prompt n. For each negative edge enj ∈ En, there is a corresponding mutually exclusive and complementary edge epj in Ep, these two hyperedges have no overlapping vertices and together constitute all the vertices in the hypergraph. N denote the space of the negative prompts and their implicit positive prompts. In this way, we got B hyperedges for both negative and positive, where B is the number of basic forms.

Cross-Category Hyperedge. Followed [Gao et al., 2022], cross-category hyperegdes are constructed through the knearest neighbors (KNN) algorithm. For each vertex, we construct a hyperedge Ei to link it and its K-1 neighbor vertices:

where MKNNk (v) denotes the k-nearest neighbors of vertex v.

In this way, we construct cross-category hyperedges with the same number of vertices. By combining these two kinds of hyperedges, we get the final complementary hypergraph G = {V, En ∪ Ep ∪ Ec}.

Structure-Aware Tuning

To leverage the potential collaborative information, we utilize the modified hypergraph convolution from [Gao et al., 2022] to smooth the complementary structure:

where H denotes the incidence matrix of the hypergraph G. Dv and De are the diagonal degree matrices for vertice and hyperedges, respectively.

After getting the tuned embedding ˜ vi of the 3D object oi, we construct a memory bank M that contains L invariant memory anchors for knowledge distillation. We rebuild the embedding of each object oi by zi = ∑Lj=1 s′ij hj , zi ∈ Rdc , where s′ij denotes the normalization of activation score and calculated by sij = Dm(˜ vi, hj), hj denotes the anchor and Dm(·, ·) denotes the distance metric function.

Loss Function

To get better embeddings through hypergraph convolution and distillation, we adopt Memory Reconstruction Loss Lmr and the common Cross-entropy Loss Lce:

where ˜ vi and zi denote the structure-aware embedding and memory reconstruction embedding, ∥∥ · ∥∥2 is the L2 norm function. ˜ pi,k = e˜ vi,k ∑Ym=1 e˜ vi,m and pi,k = ezi,k ∑Ym=1 ezi,m is the predicted probability score of the 3D object oi in k-th category. yi,k is the k-th value of the one-hot encoded ground truth label of oi, and Y is the number of categories.

In the Complementary Structure Tuning stage, the overall loss function is given by combining Eq. 11 and Eq. 12:

where β is the hyper-parameter for trade-off.

5 Experiments

5.1 Experimental Settings

NPOR Datasets. We generate four negative-prompt driven open-world 3D object retrieval (NPOR) datasets, including NPMN, NPAB, NPNT, NPES based on the public datasets ModelNet40 [Wu et al., 2015], ABO [Collins et al., 2022], NTU [Chen et al., 2003], and ESB [Jayanti et al., 2006]. As shown in Table 1, we construct three negative prompts for each object in both the training set and retrieval set according to the basic geometric forms, and we remove some objects that are difficult to categorize under this rule. These datasets consist of seen and unseen categories, each object has three modalities including multi-view, voxel, and point cloud.

Implemental Details. For a fair comparison, we fix the random seed as 2022 for all experiments in this paper. The basic features of multi-view, point cloud, and voxel are extracted by MVCNN [Su et al., 2015], PointNet [Qi et al., 2017], and 3DShapeNet [Wu et al., 2015], respectively. We set α = 0.5, δ = 0.8, β = 0.9 for the hyper-parameters of NPCP in Eq. 6, Eq. 7, and Eq. 13, respectively. The two modules are trained separately with 40 epochs on learning rate lr = 0.1 and 120 epochs on lr = 0.001.

5.2 Retrieval Performance

Compared Methods. As no methods are specifically designed for the prompt-driven open-world 3DOR, we refine the current state-of-the-art methods of close-set 3DOR (MMJN [Nie et al., 2019], TCL [He et al., 2018], CMCL [Jing et al., 2021], MMSAE [Wu et al., 2019]), and open-world 3D recognition or recognition (PROSER [Zhou et al., 2021], HGM2R [Feng et al., 2024]), then we added a prompt tuning module for each methods following [Wang et al., 2023; Li et al., 2022].

MMJN [Nie et al., 2019]: MMJN is a multi-modal joint network that employs weighted fusion to integrate features across multiple modalities for retrieval.

TCL [He et al., 2018]: TCL is a method based on metric learning, combining triplet and center loss to get unified fusion embeddings from different modalities.

CMCL [Jing et al., 2021]: CMCL designs an adversarial center loss to minimize the distances of features from objects belonging to the same class across all modalities.

MMSAE [Wu et al., 2019]: MMSAE is a multi-modal retrieval method using auto-encoders. It trains encoders with a reconstruction loss function to align embeddings from various modalities into a unified latent space.

PROSER [Zhou et al., 2021]: PROSER is an open-world recognition method that extends the closed-set classifier to determine if a sample belongs to seen categories or not.

HGM2R [Feng et al., 2024]: HGM2R is an open-world 3D multi-modal retrieval method, which retrieves the objects from unseen categories through structure-aware learning.

Evaluation Metric. For a fair comparison, we employ the commonly used retrieval metrics, including Mean Average Precision (mAP), Normalized Discounted Cumulative Gain (NDCG), Average Normalized Modified Retrieval Rank (ANMRR), and the Precision-Recall Curve (PR-Curve).

Comparison Analysis. As shown in Table 2, we evaluate the prompt-based open-world retrieval results from NPCP framework and other sate-of-the-art methods. Comparison results show that the proposed method outperforms the other methods on all four datasets. In particular, on the NPNT and NPMN datasets, our method achieves 0.5316/0.6443 mAP with about 18.2%/9.6% improvements compared with the second-best method. We also provide the Precision-Recall (PR) Curve to evaluate the performance of the proposed NPCP framework and other compared methods, as illustrated in Figure 4. The larger area below the curve indicates better performance. From the results, we can observe that our method outperforms all other compared methods. The better performance indicates that by the PEE and CST modules, the proposed method can take full advantage of negative prompts and has the generalized capability for unseen categories. Besides, as shown in Table 2 and Figure 4, our method presents a more significant improvement on the NPNT dataset. Specifically, the NPNT dataset has the maximum of categories and the lowest average number of objects per category. The proposed method can provide sufficient information for openworld retrieval with limited data to the greatest extent.

5.3 Ablation Study

We conduct ablation studies to verify the effectiveness of the proposed modules. For the PEE module, we remove the positive auto-encoder (PEE w/o Pos) and dual binary-entropy loss (PEE w/o Lde) for comparison. As shown in Table 3 and Figure 5, the performance of PEE degrades after positive autoencoder and dual binary-entropy loss are removed, respectively. Results show that direct embedding from only one direction of the negative prompts cannot provide sufficient auxiliary information for open-world retrieval. The proposed bi-directional embedding can take full advantage of prompt from both explicit negative and implicit positive directions.

As for the CST module, we compared the hypergraph structure without complementary structure (CST w/o Ep or CST w/o En&Ep), also we replace the hypergraph-based structure learning with MLP and GCN. Table 3 and Figure 5 show that the proposed CST outperforms all other structure learning methods, and the combination of PEE and CST yields the best performance. These results demonstrate the proposed framework can effectively utilize the high-order structure-aware correlations among seen and unseen categories under the open-world setting, which has the generalized capability for unseen categories with the help of prompts.

6 Conclusion

In this paper, we introduce the Negative Prompt Driven Complementary Parallel Representation (NPCP) framework, which navigates the complexities of open-world retrieval through the lens of Negative Prompts. To tackle the challenge of sufficient prompt embedding, we utilize the Parallel Exclusive Embedding (PEE) to fully leverage the prompt information from both explicit negative and implicit positive directions. Besides, we adopt the Complementary Structure Tuning (CSL) approach to exploit high-order correlations among objects for category generalization by the complementary hypergraph structure. This module constructs a complementary hypergraph based on bi-directional and cross-category correlations. We have developed four multimodal datasets for open-world 3D object retrieval with negative prompts, i.e., NPMN, NPAB, NPNT, and NPES. Extensive experiments and ablation studies on these four benchmarks demonstrate the superiority of our method over current state-of-the-art approaches. We believe that this paper will offer innovative perspectives for the research of open-world retrieval.

Acknowledgments

This work was supported by the National Natural Science Funds of China (No. 62088102, 62021002), and the Beijing Natural Science Foundation (No. 4222025).