Who Looks like Me: Semantic Routed Image Harmonization

1 Introduction

Image editing technology is extensively utilized across various aspects of our daily lives, encompassing areas such as commercial promotion, social sharing, digital entertainment, and even the emerging realm of the Metaverse [Kaur et al., 2023; Ren and Liu, 2022]. Notably, AIGC [Ho et al., 2020; Kim et al., 2022] technology empowers the direct generation of a diverse array of images, although many synthetic images require subsequent editing to enhance realism. However, individuals lacking professional photo-editing expertise may find that composited images face challenges in terms of evaluation credibility, stemming from issues such as inharmonious color, texture, or illumination. Consequently, the process of image harmonization becomes imperative for elevating the overall quality of composite images.

∗Correspongding author

Numerous methods have been developed with the objective of harmonizing composite images, addressing the discordance between foreground and background [Cong et al., 2020; Liang and Pun, 2022; Ren and Liu, 2022; Zhu et al., 2022; Chen et al., 2022; Niu et al., 2023]. Zhu et al. [Zhu et al., 2022] proposed a technique to align the representation of each foreground location with corresponding background elements. In a different approach, Tsai et al. [Tsai et al., 2017] introduced an end-to-end learning method for image harmonization, primarily focusing on constraining semantic information learning in the encoder. Cun et al. [Cun and Pun,

2020] integrated a spatial-separated attention module to compel the network to learn foreground and background features separately, but this approach falls short in ensuring style consistency between the two components. However, these existing methods predominantly emphasize visual style consistency between foreground and background regions, lacking realism derived from instance similarity.

Based on the human perception process for image harmonization, the appearance characteristics and semantic similarity of foreground and background objects are highly relevant. As illustrated in Figure 1, the little girl could be related to multiple instances in the background, including the man on the left and the woman on the right, with varying degrees of semantic similarity, When the appearance characteristics are influenced by semantic similarity, the resulting harmonization exhibits distinct characteristics. A stronger influence from the left-side instance leads to a more subdued color profile, whereas a stronger influence from the right-side instance results in a more vibrant color profile.

To alleviate the ambiguity derived from different semantic information, we propose an image harmonization model by measuring semantic similarity explicitly to enhance the consistency of appearance characteristics. As the saying goes, "who looks like me". We approach image harmonization as a semantic routed style transfer problem, focusing on refining the appearance of foreground objects using the style guidance of the most similar instance. Specifically, an Instance Similarity Evaluation Module (ISEM) is designed to compute the similarity matrices of components between the composed foreground object and the background instances. To harness analogous semantic information more effectively, we further introduce the Style Transfer Block (STB). On one hand, this module is specifically crafted to query the most akin background instance. On the other hand, corresponding style characteristics are seamlessly transferred onto the composed foreground object, enhancing the overall harmonization process. Extensive experiments including human perception experiments demonstrate the superior performance of our proposed method in improving image harmonization.

In summary, our contributions are given as follows:

• We design an image harmonization framework by evaluating the instance-similarity
• We propose an instance similarity evaluation module (ISEM), designed to assess the similarity of components within both the semantic and stylistic domains of instances in the foreground and background.
• We introduce a style transfer block(STB) that captures the global style information of the input image and transfers it to the latent space of the style encoder.

2 RelatedWork

Most early studies on image harmonization aimed to design and match low-level color statistical information of foreground and background, such as color histograms [Xue et al., 2012], gradient information [P´erez et al., 2003] and image pyramids [Sunkavalli et al., 2010]. The utilization scenarios of these methods are significantly constrained due to limitations in representing high-level features. Paired images and harmonized training data [Tsai et al., 2017; Cong et al., 2020] have been constructed by adjusting the color and illumination of the foreground objects in real images. Based on these datasets, large numbers of image harmonization models based on supervised deep learning models have been proposed and achieved more reliable results using these datasets. DIH [Tsai et al., 2017] and Sofiiuk et al. [Sofiiuk et al., 2021] use semantic information to capture image context, which aids in harmonizing the composite foreground. RainNet[Ling et al., 2021] treats the mean and variance of the deep features as appearance information and adjusts the mean and variance of the foreground to match those of the background. In addition, several endeavors have attempted to apply models that have achieved outstanding performance in other domains, such as Transformer [Guo et al., 2021a] and diffusion models [Lu et al., 2023; Li et al., 2023], to address the task of image harmonization.

Furthermore, in the pursuit of context consistency, recent notable works have approached image harmonization as a style transfer problem [Song et al., 2023]. These endeavors aim to precisely transfer the global features of the background onto the composed foreground object. Hao et al. [Hao et al., 2020] align the standard deviation of the foreground features with that of the background features, capturing global dependencies in the entire image. BargainNet [Cong et al., 2021] uses a domain code extractor to capture background domain information, guiding the foreground's harmonization. Recently, Hang et al. [Hang et al., 2022] has achieved state-ofthe-art results by incorporating background and foreground style consistency constraints and dynamically sampling negative examples in a contrastive learning paradigm. These methods leverage network models to learn the relationship between foreground and background feature representations implicitly.

In this paper, the background elements that exert a more pronounced influence on the appearance characteristics of foreground objects are concerned. We explicitly extract the semantic relationship between the background and foreground elements, and employ this information to guide and inform the image harmonization process.

3 Methods

3.1 Overall Pipeline

The objective of our paper is to maintain consistent appearance characteristics between the foreground and background of synthetic images. Consequently, forming a substantial association between the composite foreground instance and other background instances is vital for crafting harmonious appearance uniformity. As depicted in Figure 2, we initially deploy a pre-trained SAM model to divide the synthetic image into a semantic space, with the mask of the foreground functioning as the model's prompt. Subsequently, semantic mapping takes place to transform the SAM model's output into the semantic and location data of the background instances. We introduce the Instance Similarity Evaluation Module (ISEM), designed to compute a similarity matrix between the composite foreground instance and the various background instances. As part of the harmonization procedure, we utilize a semantic routing technique that utilizes semantic similarity, which incorporates instance location and a semantic similarity matrix to deliberately adjust the feature representations within the image. To bolster the influence of analogous semantics, we employ an encoder-decoder network architecture. Here, the composite image is subject to convolutional encoding and then processed through three strata of the STB encoder. During decoding, to leverage the semantic similarity matrix in guiding the harmonization process, we introduce the Style Transfer Block (STB). This block shares a similar framework with STB, with a distinction in the attention mechanism where the Key-value matrix is modulated by the corresponding scale instance similarity matrix. This adjustment ensures alignment with semantic similarity and the subsequent refinement of the harmonization results. We apply a feature transformation function to ensure feature dimension consistency following each multiplication process. The process is formulated as:

Where K and K′ are the input and output feature map, same to V and V ′ , S is the same scale instance similarity metrix obtained from the semantic routing module. Finally, following the traversal of a convolutional layer, we can get the harmonized image.

3.2 Instance Similarity EvaluationModule

We employ the pre-trained Segment Anything Model (SAM) [Kirillov et al., 2023] on a comprehensive dataset for decomposing the composite image. SAM leverages foreground/background points, bounding boxes, or masks as prompts to produce segmentation results. It incorporates three primary components: an image encoder, a prompt encoder, and a mask decoder. Utilizing a pre-trained mask selfencoder based on the Vision Transformer (ViT), SAM processes the image into intermediary features while transforming the prompts into embedding tokens. The mask decoder's cross-attention mechanism then enables interactions between image features and prompt embeddings, culminating in the generation of the mask output. This process can be expressed as:

where Fi is the image feature, Fp is the prompt feature, ˆM is the mask output, Tout and Tprompt are the output and prompt embedding tokens, respectively.

To derive the semantic representation of each instance, we initially employ the "full image" mode of SAM for segmenting all possible instance targets within the image. Subsequently, we introduce a semantic mapping module that ascertains the location and semantic details of instances, drawing from the image embedding produced by the SAM decoder.

Specifically, following the SAM decoder, the image embedding undergoes an up-sampling by a factor of 4× via two transposed convolutional layers. The image tokens, labeled as Eim and incorporating prompt and output tokens, engage with the image embedding. The refreshed token embedding is then directed through three-layer MLP (Multi-Layer Perceptron) [Riedmiller and Lernen, 2014] modules to yield the instance embedding, represented as Ein. A spatial point-wise product is performed between the up-scaled image embedding and the instance embedding to predict the position of the instance, signified as P. This process can be expressed as:

Furthermore, we use a cross-similarity module to calculate the similarity between N instances. We use global average pooling to generate mean query feature ¯ F(Ein). Then we copy it and make it have the same shape with the target feature Ei in . The cross similarity map S has the same width/height with the number of instances detected. Mathematically, the similarity metric can be expressed as

where cos(·) indicates the cosine similarity.

3.3 Semantic Routing

To preserve the pronounced impact of background regions with analogous semantics on the foreground object, we introduce a semantic routing strategy predicated on assessing semantic similarity within the semantic space. As depicted in Figure 3, the semantic similarity matrix coupled with instance location data is employed to identify all feasible instances. By aligning semantic information with spatial location indices, we compute the correlation coefficient between background instances and foreground objects, subsequently generating a spatial importance map. In detail, the instance index of the position embedding is denoted as i and the corresponding value as Si, it can be formulated as:

where M is the semantic similar value from the semantic similarity matrix.

Upon finalizing the semantic-location mapping, the semantic similarity matrix is transformed into an instance similarity matrix. This matrix not only embeds instance location information but also encompasses correlation coefficients between background instances and foreground targets. To align with the Key-Value pairing mechanism in the multi-level STB, the similarity matrix is subject to interpolation operations, which yield a multi-scale matrix pyramid mirroring the scale structure of the STB.

3.4 Style Transfer Block

Style Transfer Block(STB) aims to integrate the spatial semantic and similarity information, which involves applying Self-Attention (SA) across channels instead of the spatial dimension. This enables the calculation of cross-covariance among channels, facilitating the creation of an attention map that innately represents the global context. We augment the STB by integrating depth-wise convolutions as recommended by Zamir et al. [Zamir et al., 2022], which accentuate the local context prior to the computation of feature

From a layer-normalized tensor Y ∈ RH×W×C , our STB initially produces query (Q), key (K), and value (V ) projections that are imbued with local contextual information. This is accomplished by implementing 1 × 1 convolutions, which aggregate the cross-channel context at the pixel level, followed by 3 × 3 depth-wise convolutions that encode the spatial context within the channel, resulting in Q = W Q d W Q p Y , K = W K d W K p Y , and V = W V d W V p Y . Here, W (·) p signifies the 1 × 1 point-wise convolution, while W (·) d represents the 3 × 3 depth-wise convolution.

Subsequently, we reconfigure the query and key projections so that the dot-product interaction between them produces a transposed-attention map A with dimensions RC×C , as opposed to the substantially larger standard attention map sized RHW×HW . Overall, the STB process is defined as:

where X and ˆX are the input and output feature maps, Q,K, V matrices are obtained after reshaping tensors from the original size RH×W×C . Echoing traditional multi-head self-attention mechanisms, we partition channel dimensions into discrete heads, concurrently computing distinct attention maps to learn style feature representations . For the stylization of features, a uniform feed-forward network (FN) independently processes each pixel location. The first convolution augments feature channel capacity, typically quadrupling it, and the second convolution restores the channel count to match the original input dimensionality. Interposed between these convolutions is the application of a non-linearity within the hidden layer to facilitate complex feature transformations.

In thie work, we harness a gating mechanism and depthwise convolutions within the feed-forward network (FN) to enhance representation learning. The gating mechanism is implemented as an element-wise product of two linear transformation pathways, with one pathway undergoing activation by GELU non-linearity. Depth-wise convolutions are incorporated to capture information from spatially adjacent pixels, instrumental in learning the local imagery structure crucial for effective restoration. Given an input tensor X ∈ RH×W×C , it is formulated as:

where (·) denotes element-wise multiplication, ϕ represents the non-linearity, and LN is the layer normalization. Overall, STB effectively captures the global stylistic attributes of the input image and conveys them to the latent space of the style encoder.

4 Experiments

4.1 Datasets

Our experiments use the iHarmony4 dataset, a publicly available synthesized dataset referenced by Cong et al. [Cong et al., 2020], which includes four sub-datasets: HCOCO, HAdobe5k, HFlickr, and Hday2night. These sub-datasets encompass synthesized composite images, foreground masks for these images, and their corresponding real images. We employed the same processing method as HDNet [Chen et al., 2022] for the dataset. Additionally, to validation the performance of our methods in real-world scenarios, we employed 100 real-world images from CDTNet [Cong et al., 2022], which are processed in the format of the iHarmony4 dataset.

We evaluated the performance of our method using MSE, PSNR, fMSE, as suggested by [Cong et al., 2020; Ling et al., 2021; Niu et al., 2023], in which fMSE means MSE within the foreground region. To illustrate performance, we qualitatively compare our method with following harmonization methods, including DoveNet [Cong et al., 2020], Intrinsic [Guo et al., 2021b], Bargainnet [Cong et al., 2021], RainNet [Ling et al., 2021], D-HT [Guo et al., 2021a], Harmonizer [Ke et al., 2022], SCS-Co [Hang et al., 2022], CDTNet [Cong et al., 2022], HDNet [Chen et al., 2022], GKNet [Shen et al., 2023], and LEMaRT [Liu et al., 2023].

4.2 Implementation Details

Our model is trained by AdamW optimizer with β1 = 0.9, β2 = 0.999, and weight decay 1e-4 . We train the model for 200 epochs with input images resized to 256×256 and batch size set to 8. The initial learning rate is set to 3e-4 and gradually reduced to 1e-6 with the cosine annealing [Loshchilov and Hutter, 2017]. We use PyTorch to implement our models with NVIDIA GeForce RTX 4090.

4.3 Comparison with ExistingMethods

Quantitative comparison. Table 1 shows the quantitative results of previous image harmonization methods as well as our method. It is evident that our method surpasses the comparative methods across all datasets with the exception of MSE and fMSE on HCOCO. Furthermore, when contrasted with the second-best performing method, ours realizes a substantial average enhancement of 0.52dB in PSNR, a 0.55 reduction in MSE, and an improvement of 77.26 in fMSE.

Influence of fore-ground ratios. Following [Cong et al., 2020], we examine the influence of different fore-ground ratios on the harmonization models, i.e., 0% to 5%, 5% to 15%, 15% to 100%, and overall results. The comparative results of previous methods and our method are tabulated in Table 2. Upon scrutiny, it is evident that our method exhibits superior performance, outperforming all other approaches.

Qualitative comparison. In Figure 4, Additionally, we provide a qualitative comparison of results on the iHarmony4 dataset. It is readily apparent that our method secures a more uniform visual style across the entire composite image, resulting in a more photorealistic outcome. For example, as shown in the second row of Figure 4, the visual style of the foreground and the background are quite different, resulting in obvious image distortion. The other three methods cannot adjust the style of the foreground, especially the overall tone and the contrast of lighting and shadows. Unlike them, our method produces a more photo-realistic result and is closer to the ground-truth real image.

Overall inference time. In Table 4, we present the inference time, parameter count, and FLOPs required for harmonizing a single image during testing. our approach does not show efficiency advantages, as indicated in the last row of Table 4, due to utilizing the pretrained SAM model for instance information retrieval. Yet, when relying solely on pixel domain architecture without ISEM, our model demonstrates comparable inference speed, with each step taking 20.4ms and a parameter count of 25.28M, as shown in the third row of Table 4. In this study, we intentionally sacrificed some speed advantages to prioritize the realism of the harmonized images. Nonetheless, there is significant potential to enhance both the speed and parameter count of the SAMmodel, a direction we aim to pursue in future research.

4.4 Ablation Study

Effectiveness of each component. In this section, we investigate the effectiveness of each component in our model.

The results of ablating each component are reported in Table 3. Our ISEMmodule enables assess the similarity of components within both the semantic and stylistic domains of instances in the foreground and background. In Table 3, we can see that adding ISEM to the baseline brings 0.56 dB and 5.12 average performance improvement in terms of PSNR and MSE.

The STB effectively learns global style features and applies them to foreground objects. The addition of the STB enhances the overall coherence between foreground objects and background images. However, it also introduces a limitation in the form of excessive reliance on the background, which limits the effectiveness of improvement. In Table 3, we can see that adding STB to the baseline brings 0.71dB and 5.24 average performance improvement in terms of PSNR and MSE.

By concurrently incorporating the ISEM and STB modules, our method effectively establishes correlations between various components of the target object and background instances, thus enhancing overall coherence. Consequently, the improvement is significantly pronounced. In Table 3, we can see that adding both ISEM and STB to the baseline brings 2.45 and 12.85 average performance improvement in terms of PSNR and MSE.

Visual comparison. To further illustrate the effectiveness of our mothods, we show some output results of ablation experiments in Figure 5. It can be found that compared with the distortion results produced by the module, the full model's results performe more consistent in lighting and color with background regions.

4.5 User Study

We extend our evaluation by comparing various methods using a dataset of 100 real composite images provided by CDTNet [Cong et al., 2022]. To gauge the performance against competitive baselines, we conduct a user study. This study involves the construction of 600 image pairs, in which we randomly select two images from each composite image and its 3 corresponding harmonized results across the 100 real composite images. Subsequently, we allocate 60 pairs for each of the 20 participants, who are tasked with viewing one image pair at a time and selecting the image they perceive as more harmonious. This process generates a total of 1200 pairwise results. Following the methodology adopted in GiftNet [Niu et al., 2023], we computed the Bradley-Terry(B-T) scores for all methods, as detailed in Table 5. Notably, our approach emerges with the highest B-T score (which is 0.413) concerning realism, underscoring the efficacy of the method proposed in this paper. The visualization results pertaining to real composite images are presented in Figure 6. Compared to previous methods, our results demonstrate enhanced realism, particularly evident when similar instances are present in the background, as are N(N > 0) related instances in the background, the model constructs an N-dimensional similarity matrix to represent the degree of similarity between instances. These instances affect the foreground through weighted accumulation across the matrix, and the foreground maintains good consistency with the most relevant instances, such as the color of sunflowers in the 3rd row of Figure 6. Furthermore, in the absence of similar instances, the proposed STB and ISTB, which can capture and transfer global color information into the foreground, can maintain overall appearance consistency throughout the image, as illustrated in the 4th row of Figure 6.

Figure 6: The visualization of different methods on real composite images.

5 Conclusion

In this paper, we propose a image harmonization model utilizing instance similarity to maintain consistency uniformity in global and similar regions. We propose an instance similarity evaluation module (ISEM), which can assess the similarity of components within both the semantic and stylistic domains of instances in the foreground and background. We introduce a style transfer block (STB) that captures the global style information of the input image and transfers it to the latent space of the style encoder. Our method has achieved excellent experimental results on existing datasets and has more significant advantages in user visual reality evaluation.

Acknowledgments

This work is supported in part by the National Science and Technology Major Project of China under Grant 2022ZD0118001; and in part by the National Science Foundation of China under Grant 62332017, Grant 62303043, Grant U22A2022, Grant 62372036, Grant 62120106009.