Cross-modal Representation Learning and Relation Reasoning for Bidirectional Adaptive Manipulation

1 Introduction

Generating natural and meaningful outputs from one modality that semantically matches given supervision information from the other modalities is a challenging problem with vast potential applications, including image editing, language style transfer, and computer-aided design. Considering that a cross-modal model can receive inputs from multiple modalities simultaneously, it is a natural idea that the outputs of these models should also cover all the input modalities simultaneously. However, the current research focuses on the output of a specific modality [Nam et al., 2018; Li et al., 2019; Dhamo et al., 2020]. For example, text-guided image manipulation generates natural-looking images from language descriptions [Li et al., 2020a; Li et al., 2020b], and imageguided text manipulation produces language conforming to grammar rules [Cornia et al., 2019] from a conditional image. However, cross-modal adaptive manipulation for bidirectional generation on both modalities has received less attention.

∗ Corresponding author

While one can introduce additional supervision to manipulate images or texts, it is challenging for a non-expert user to design appropriate rules, specific priors or adopt professional tools for single-modal manipulation. Additionally, the generation from single-modal manipulation has been mostly restricted to object-centric scenarios [Dhamo et al., 2020] and the limited functionality of fixed or mobile devices. By comparison, cross-modal adaptive manipulation aims to achieve cross-modal mutual supervision with no need for specific priors, adaptively exchanging attributes, relations, or objects in input modalities, and generating outputs for both inputs by multi-task learning. Therefore, CAM can better fit crossmodal scenarios and inspire related research.

We specify the introduced task, i.e., consider the two critical modalities of vision and language. Given an image and a sentence, on the one hand, it allows modifying visual presentations (e.g., texture, color, and category) of the image according to the input sentence; on the other hand, it supports adjusting textual contexts (e.g., noun, adjective, and verb) of the sentence by the input image. The differences between our task and previous works are listed in Figure 1.

Our contributions are summarized as follows: (1) The weaknesses of single-modal controllable manipulation are addressed, and a novel task, namely cross-modal adaptive manipulation (CAM), is introduced, which has excellent potential for cross-modal applications. (2) A strong baseline is proposed to solve the CAM task. The HRL module performs cross-modal representation learning for semantic alignment, and the CRR module implements cross-modal relation reasoning in input modalities. (3) Experiments on CUB and Visual Genome verify that our approach outperforms the leading methods of single-modal controllable manipulation.

2 RelatedWork

Single-modal ControllableManipulation. Distinct froma portion of generative networks that are typically uncontrollable, another line of research can be summarized as singlemodal controllable manipulation, which mainly includes textguided image manipulation and image-guided text manipulation. For text-guided image manipulation, [Nam et al., 2018] utilized a text-adaptive generative adversarial network to semantically manipulate images while preserving independent contents in the original image. [Li et al., 2020b] proposed a word-level discriminator to facilitate training a lightweight generator for image manipulation using natural language descriptions. For image-guided text manipulation, which is different from image caption (no additional text input) and rarely paid attention to, many applications can benefit from this task, such as the children's early teaching robot dialogue system.

However, the existing works only perform single-modal controllable manipulation for visual or language generation, thus only obtaining biased models which can only be applied in restricted applications. In this work, we introduce the novel CAM task for multi-modal generations, simultaneously benefiting the modalities of both inputs.

Graph Representation Learning. The latest developments in graph representation learning have revolutionized various applications as they are well suited for realistic scenarios. Generally, the main ways to apply graph networks for non-structural scenarios include visual and language are: (1) incorporate structural information from other modalities to improve the performance of the current modality. For example, using information from knowledge graphs to guide zeroshot recognition classification [Kampffmeyer et al., 2019], or performing more delicate relation exploration for a more interpretable reasoning process [Wang et al., 2018]; (2) infer or assume the relational structure defined on graphs in a single modality. For instance, [Yao et al., 2019] regarded the documents and words as nodes to construct the corpus graph and used the Text GCN to guide the representation learning of nodes. [Liu et al., 2019] conducted evidence aggregating and reasoning based on a fully connected evidence graph.

However, these approaches implement graph representation learning by introducing external supervision or only performing single-modal relation reasoning. In contrast, we explore cross-modal graph representation learning and bidirectional relation reasoning in the way of mutual supervision, leading to a novel multi-task learning framework.

3 ProposedMethod

Task Definition. We introduce the cross-modal adaptive manipulation task: given an image Ii and a sentence Si as inputs, on the one hand, the job semantically manipulates Ii according to Si so that the visual presentation of the manipulated output Io match the linguistic descriptions of Si while preserving Si-independent information in Ii; on the other hand, it adaptively manipulates Si according to Ii so that language presentation of the manipulated output So match the visual contents of Ii while keeping Ii-independent information in Si. Notably, the above two procedures are performed simultaneously.

An overview of our solution is shown in Figure 2, which could be split into two parallel pipelines. For controllable image manipulation, the pipeline is based on GAN [Goodfellow et al., 2020] framework, in which the generator G is trained to produce Io = G(Ii, Si), and the discriminator D is adopted to ensure that realistic images are generated to semantically match the conditional text. The pipeline adopts the Encoder-Decoder framework for controllable text manipulation, including the encoder En to capture the distribution of manipulated sentences and the decoder De to output the desired words sequentially. In the following, we first introduce the main contributions of this work, i.e., the HRL module and CRR module, which are shared as the first part of both G and En. After that, we will briefly describe the other features of GAN and the Encoder-Decoder framework, which are mainly based on the existing works.

3.1 Heterogeneous Representation Learning

Given both inputs, We first construct an image graph GI for Ii and a sentence graph GS for Si, respectively. Specifically, we adopt a pre-trained model to discretize the input image to obtain multiple objects and treat each object as a visual node, eventually getting the GI with its adjacency matrix AI . During this period, we use a similar approach to build the GS with its AS.

We need to find bridges to link both input modalities to perform representation learning for cross-modal semantic alignment. Specifically, we search for objects co-occurrence in both modalities to serve as bridge nodes. As shown in Figure 3, the visual object 'bird' and the word 'bird' appear in the input modalities, respectively. In this case, we treat 'bird' as a bridge object and create four kinds of nodes, including birdI , birdS, birdIS and birdSI, which indicate the visual node of GI, the language node of GS, and the copies of birdI and birdS, respectively. The first two nodes (denoted as reserved nodes) are kept in the built graphs of their original modalities, while the last two nodes will serve as bridge nodes to create heterogeneous graphs for cross-modal representation learning. Significantly, the node birdIS derived from visual modality is added to GS, while the node birdSI generated from language modality is incorporated into GI. We define F (birdIS ) = F (birdI ), F (birdSI ) = F (birdS ) where F (·) returns the initial node features. A multi-layer perceptron (MLP) layer is then utilized to map the feature dimension of visual nodes to the same as the accordingly word embeddings. After that, we extend the adjacency matrix under each modality by introducing the bridge nodes from the other modality to obtain the extended version AEI of the extended image graph GEI and the corresponding AES of GES. In this way, we obtain two heterogeneous graphs consisting of nodes from both input modalities.

Heterogeneous representation learning is then performed in the extended graphs of GEI and GES in two steps. First, the representation aggregations based on regular graph convolutions [Kipf and Welling, 2016] is performed independently on each heterogeneous graph, i.e., this step in GEI could be implemented by a layer-wise propagation rule:

where ˆ AEI is the normalization of AEI, W l is a layerspecific trainable weight matrix, and (; ) means the row-wise concatenation. HlI indicates the feature matrix of visual nodes in the l-th layer, and Hlbg is the feature vectors of introduced bridge nodes from the language modality. The same propagation process is performed in GES.

After one-layer propagation of both graphs, we update the representation of all reserved nodes in both graphs periodically with the help of the copy mechanism. Take the reserved node birdI for example, in the second step, we merge the representation of related nodes through a linear transformation:

where HbirdI(,IS,S) denotes the feature vectors of node birdI(,IS,S), and λ1 and λ2 are the weighting factors. After that, the representation of node birdI is updated by: HbirdI = ¯HbirdI . The above two-step learning process is iterated by L graph layers and finally, we obtain HL I and HL S . In this way, the introduced bridge nodes can learn the representation of both modalities and spread information in the heterogeneous graphs, progressively aligning the semantics of nodes from both input modalities while providing modalconsistent representation for subsequent relation reasoning.

3.2 Cross-modal Relation Reasoning

We introduce the bidirectional graph masking mechanism to implement cross-modal relation reasoning parallelly. Based on the updated representation of cross-modal nodes, we remove all the bridge nodes and predict two soft masks MI and MS for the original GI and GS, which are formulated as:

where WI,WS are learnable matrices, and (·) indicates matrix multiplication. The process simultaneously performs global relations reasoning on both graphs. Each generated masking matrix enables the exchange of corresponding graph nodes in both modalities by masking some nodes in one modality and highlighting the corresponding (semantically similar) nodes in the other modality. For instance, consider the red node in the visual modality and the corresponding blue node in the language modality, MI tries to mask the red node and highlights the blue node for generating the image with a 'blue' object. While MS attempts to do the opposite operation for generating the sentence with a 'red' word by masking the blue node and highlighting the red node. By masking a portion of nodes and performing the representation aggregation of all highlighted nodes in both graphs, two advantages are obtained in this way: (1) the exchange of nodes is achieved across modalities without destroying the graph structure of each modality; (2) bidirectional relation reasoning is implemented through learning MI and MS in parallel on both modalities. Finally, the obtained soft masks will be multiplied by the node features and processed by a linear transformer to get the output:

where ⊗ denotes the element-wise product operation.

Other Parts for Manipulations. For text-guided image manipulation, GAN is adopted while its generator mainly contains the two modules mentioned above, one additional feature fusion part and a light decoder. We adopt the VGG16 network to encode the input image, and the obtained features are combined with the representation of the node for fusion. The decoder, which consists of a few upsampling layers, is adopted for image generation. Additionally, we adopt the text-adaptive discriminator introduced in [Nam et al., 2018] to force the generator to receive feedback from each local discriminator for each visual attribute.

We adopt the Encoder-Decoder framework for imageguided text manipulation, whose encoder contains the two modules mentioned earlier and a representation aggregation part and whose decoder is composed of a visual attentionbased network. Specifically, we perform the representation aggregation process to obtain the attention-guided representation of each node in the language modality. The decoder generates one word at each time step conditioned on a visual vector, a previous hidden state, and a previously generated word.

Loss. During training, we adopt multiple losses to optimize the whole framework and effectively highlight the optimal graph nodes for manipulating attributes, relations, or objects in each modality with a global perspective. Overall, two commonly used loss functions in generation tasks, namely the GAN loss LGAN and the Encoder-Decoder loss LED, as well as the designed cross-modal loss LCM are adopted. Significantly, we introduce the cross-modal loss LCM which further helps to shape the feature space:

We have LCM = LH + LM, where |·| indicates the first norm of the matrix, and WS[a, b] computes the Wasserstein distance between a and b. For LH, the first term encourages the bridge nodes to sufficiently learn the representation from other modalities, while the second term constrains the joint distribution of the representation from both modalities to stay close, maintaining the stability of the learning process. For LM, The first two items are adopted to force MI and MS to be sparse so that only a portion of cross-modal nodes are highlighted for the graph-based aggregation. The third term with the weighting factor γ makes the highlighted nodes by MI , and MS tend to be distinctive after the relation reasoning process, thus facilitating the exchange of cross-modal nodes and avoiding the overfitting during training. The final loss becomes: L = LGAN + LED + LCM .

4 Experiments

Experimental Setup. We verified our baseline in two aspects, including the manipulation of both attributes and relations of objects. For cross-modal attribute manipulation, we evaluated our method on the CUB dataset [Wah et al., 2011]. For cross-modal relation manipulation, we validated on the Visual Genome (VG) [Krishna et al., 2017].

Quantitative Results. The quantitative evaluations are conducted on both synthesized images and generated sentences. For synthesized images, we conducted a human evaluation and compared to relevant generative models including SISGAN [Dong et al., 2017], AttnGAN [Xu et al., 2018], TaGAN [Nam et al., 2018], ManiGAN [Li et al., 2020a], and Lightweight GAN [Li et al., 2020b].

We randomly selected 20 images and 10 texts from the test set of the CUB dataset and produced 200 image-sentence pairs for each method. We invited 80 workers to compare the results after looking at the input image, the input sentence, and both outputs based on two criteria: (i) Accuracy: whether the visual attributes (colors, textures) of the manipulated image match the text, and the background-independent to the text is preserved, and (ii) Naturalness: whether the manipulated image looks natural and visually pleasing. Both criteria are categorized into three ranks (1, 2, and 3), and the lower, the better. We collected a total of 16000 results. Besides, we computed L2 reconstruction error by forwarding images with the ground-truth text descriptions, while Inception Score (IS) and Fr´echet Inception Distance (FID) are also evaluated on a large number of modified samples produced from mismatched pairs, i.e., randomly chosen input images edited by randomly selected text descriptions.

As shown in Table 1, our method consistently achieves the highest average ranking on both Accuracy and Naturalness, the lowest reconstruction error, and better IS and FID values. It indicates that our method generates more realistic images, where the visual attributes are manipulated accurately with the given descriptions and effectively preserve the text-independent contents. We computed the parameters and recorded the runtime for training a single optimization epoch (RPE) and the inference time (IT) for generating 100 new modified images to evaluate the efficiency. The results indicate that our method is more friendly to memory-limited devices.

In addition, we evaluated our method on the Visual Genome to verify its effectiveness in generating natural images. Since there is no ground truth for manipulations, we formulated the quantitative evaluation as image reconstruction. In this case, we manipulated the relations between objects from the language descriptions and measured the reconstruction quality given an input image. Table 2 shows the reconstruction errors of comparative methods, showing that our method significantly outperforms Cond-sg2im [Johnson et al., 2018] and SIMSG [Dhamo et al., 2020]. Notably, our method achieves better results on all frequently-used reconstruction metrics (MAE, SSIM, LPIPS) [Dhamo et al., 2020] and dominates for both inception score and FID, indicating higher visual quality. Compared to SIMSG, which changes the relations among objects by manipulating the generated scene graphs, our method provides a way to directly manipulate the image by inputting simple language descriptions that contain the desired modifications, improving efficiency with less manual effort from the user.

For generated sentences, BLEU [Papineni et al., 2002] score, the TextRank criteria (TR s) [Mihalcea and Tarau, 2004], and the Levenstein distance ratio (LD r) [Yujian and Bo, 2007] are used for measuring the image-text similarity scores of our method. Furthermore, we compared the top-1 image-to-text retrieval accuracy (Top-1), and the percentage of the matching images in the top-50 text-to-image retrieval results (AP@50) with AttnGAN, TaGAN, and ManiGAN on the CUB dataset. As shown in Table 3, our proposal outperforms other methods on the image-text matching task, fully maintaining the image-independent contents from the input sentences to guide the sentence generation.

Qualitative Results. Figure 4 shows the qualitative results for cross-modal attribute manipulations on the CUB dataset, verifying that the attributes of visual modality are accurately exchanged with the corresponding attributes in language modality. This indicates our method: (1) effectively disentangles visual attributes invariant to pose, shape, and background in the visual modality; explicitly locating and exchanging the focused attributes with the language modality to reconstruct for obtaining images with detailed textures; (2) efficiently captures linguistic attributes by sufficiently analyzing the syntactic structure and content of the text in the language modality; accurately locating and exchanging the focused attributes with the visual modality to produce highquality sentences.

To verify the effectiveness of our method for cross-modal relation manipulations, we configured three different settings, including object removal, replacement, and reposition, and evaluated on the Visual Genome. All the image manipulations are controlled by changing the corresponding text descriptions. As shown in Figure 5, Our method successfully removes the focused target (mirror) in the synthesized image based on the text description, while text-independent visual features from the input image can be successfully transferred to the output. In object replacement, we provided diverse replacements from foreground objects to background components. In Figure 5, our method removes the old object ('giraffe') very naturally from the input image while the substituted object ('elephant') adapts well to the synthesized image. A more challenging scenario is to change the relations among objects, which typically involves object reposition. The result in Figure 5 verifies that our method can differentiate between semantic concepts such as 'on' and 'holding', and the focused object ('clock') are re-arranged meaningfully according to the relation change indicated in the input sentence. In addition to the relation manipulations of focused objects, our method performs well for the background with uniform texture and complex structures. Moreover, the generated sentences are high-quality and meaningful, and the exchanges of target relations with the input images are effectively accomplished.

Components Ablation. We ablated the major components of our method qualitatively in Figure 6. First, removing the HRL module is not conducive to cross-modal representation learning for semantic alignment, leading to difficulties in implementing transformations of both visual and linguistic information from the inputs to the outputs; second, removing the CRR module damages the bidirectional information control in cross-modal relation reasoning, generating uncontrolled and unsatisfactory results on both modalities.

5 Conclusion

We have presented a strong baseline for cross-modal graph representation learning and relation reasoning and applied our proposal to the novel cross-modal adaptive manipulation task. Experimental results verify the superior performance of our method in cross-modal applications. We leave it for future work to extend our research to more modalities.

Acknowledgements

• This work was supported by NSFC project Grant No. U1833101, SZSTC Grant No. JCYJ20190809172201639, ZDSYS20210623092001004 and WDZC20200820200655001.