Matting Moments: A Unified Data-Driven Matting Engine for Mobile AIGC in Photo Gallery

1 Introduction

Precise matting of salient subjects in an image becomes pivotal in phone applications. And matting results can facilitate secondary creation including intelligent editing and Artificial Intelligence Generated Content (AIGC) [Du et al., 2023]. As a key capability of visual systems, matting substantially impacts the editing efficiency and user experience of mobile phone products. To realize diverse AIGC applications on mobile terminals, matting serves as the foundation for translating technical capabilities into commercial value [Sun et al., 2022; Chen et al., 2022; Hu et al., 2023; Li et al., 2022]. Integrating the visual interface enables users to select and extract desired elements by holding, copying and dragging them. This expedites and simplifies editing or generation.

Within the broader context of AIGC, matting functions and capabilities appear to be in high demand among users. They can be extended in several ways, such as through scene parsing and portrait editing. Combining matting abilities with super-resolution and stylization models could enable diverse intelligent creations. To maximize the effectiveness of matting functions, it would be useful to automatically identify matting subjects based on visual and contextual content. This could enable recommending relevant generated galleries to users. This would not only help users showcase the beauty of life's moments, but also improve the user experience by recommending recent and relevant content.

Advances in AI have enabled tremendous potential for automatic generation and creation. A wide range of innovative applications have emerged in this field, particularly AIGC applications on mobile devices. Matting technology can help users quickly and accurately isolate parts of images for editing, such as people, objects, and scenes. This makes it easier for users to perform various editing operations including beautification, retouching, and compositing of specific objects, thereby enhancing entertainment and visual effects. However, automatic recommendation of AIGC in conjunction with matting engines remain underdeveloped. This could improve the overall user experience and promote the development of more efficient and effective creation and generation applications on mobile devices.

In this demonstration, we present a photo gallery application called "Matting Moments". It is a unified system based on diverse matting models and AIGC functions, allowing the consumers to obtain intelligent recommendations with beautifully generated content tailored to their photos. Specifically, our system comprises three modules: 1) photo content structuring, 2) data-driven matting engine, and 3) AIGC functions. Unlike previous works, we focus on editing photos that conform to the moment generation, helping users respond to sharing demands quickly and emotionally.

2 SystemFramework

We present an overview of our system framework in Figure 1, where users' photo gallery is used as input, and AIGCgenerated contents are integrated as output. The workflow involves several steps: Firstly, the photo content structuring module extracts basic and semantic information from each image. Then, the data-driven matting engine guides users to highlight and matte diverse subjects. Finally, the AIGC functions module collects and presents beautiful moments. We will introduce each component in detail below.

2.1 Photo Content Structuring

Photo content structuring aims to parse photos in a gallery using both basic and semantic information. The proposed process requires foundational information about a photo, as shown in Figure 1. Data synchronization is used to obtain file path, size, time, and GPS. Additionally, each photo is fed into a 1000-class tagging model and an OCR model to categorize scenes, objects, and OCR texts through scanning. After categorization, images are scanned for content relevance and aesthetic quality. Quality assessment is conducted based on factors such as sizes, formats, composition, and aesthetic filters. These assessments determine the generation strategy based on the content and scene of each photo.

2.2 Data-drivenMatting Engine

We introduce our data collection and training pipeline for the basic matting engine. Due to the diverse types of samples, we employ multiple training strategies [Liang et al., 2022] to incorporate various available data into our training loop. First, we conduct supervised training with pixelwise labeled samples. Then, we incorporate semi-supervised techniques cited in [Bao et al., 2022; Lv et al., 2022; Piao et al., ] and multi-modal learning capabilities to utilize unlabeled samples and text-image samples. As a result, our model can be continuously updated as new data arrives.

Pixel-Wise Supervised Training

Encoder-decoder structure is adopted for the matting model which is well-suited for dense prediction tasks such as U2Net [Qin et al., 2020]. With the labeled samples, we propose a two-step training protocol. a) First stage: we train the model on a large dataset containing 160k simple samples to achieve accurate segmentation results. The large number of simple examples facilitates training and improves the model's robustness. b) Second stage: we fine-tune the model on a smaller set of 20k hard samples to refine the matting results. This stage focuses on training the model to handle complex scenes and ambiguous boundaries using the robust visual features learned in the first stage. The resulting model has a high capacity for generalization and can handle challenging corner cases.

Semi-Supervised and Multi-Modal Learning

We extend our framework to incorporate these expanding data into our training. Notably, the vast majority of these data do not include pixel-wise annotations and are typically labeled only with text-based tags or left unlabeled [Shin et al., 2022; Cong et al., 2022]. We aim to increase the trained model capacity to process these continuously arriving samples as in Figure 2, incorporating semi-supervised learning and multimodal processing capabilities into our matting model.

Unlabeled data: We construct our semi-supervised framework following [Ouali et al., 2020]. In addition to the encoder and the main decoder, multiple auxiliary decoders are considered and their inputs are perturbed feature maps of encoder in Figure 2. For an unlabeled sample, we generate a prediction from the main decoder as the pseudo target, and then apply various augmentations to each auxiliary decoder to obtain auxiliary predictions. The consistency between the auxiliary predictions and the pseudo-target is enforced. For training stability, we have regularized the mean squared error (MSE), and the loss is not back-propagated through the main decoder. The shared representation of encoder is improved through the use of a perturbed auxiliary decoder and the consistencies derived from unlabeled samples. Only the main decoder is employed during inference.

Text-image data: We also collect a substantial number of text-image samples. These texts can be easily extracted from manual annotations, web sources, and image caption models available on the market. Thus, we enable our framework further to process text-image samples. With a pretrained multimodal model [Radford et al., 2021], we propose multi-modal consistency learning and text-aware contrastive learning. For a text-image pair, the prediction is initially made using the image through the main decoder. The text feature intuitively complements the visual feature with semantically significant concepts. Then, we regularize the distance between pureimage prediction and text-image prediction to encourage the model to predict semantically significant salient objects [Liu et al., 2022; Zhang et al., 2022]. In addition, we propose a contrastive learning scheme to better couple image-text relationships. We consider the image-text feature to be the anchor, and encourage the pure-image feature to be adjacent to it. Then, we sample some negative texts in order to generate negative image-text features, which are then pushed away from the anchor. Thus, the image encoder is encouraged to explore semantically significant entities.

Hierarchical Training Strategy

Given the diverse data sources, we integrate different training strategies to actively update the model with incoming data. For the images with pixel-wise labels, we categorize them into two groups by difficulties. A two-stage supervised training strategy is adopted. A group of simple samples ensures the quality of the features and the capacity to generalize, and the other group of fewer challenging examples handles corner cases and ambiguous boundaries. Such method ensures an optimal utilization of the fully labeled data. In addition, a semi-supervised method is used to include unlabeled data. We incorporate the multi-decoder consistency technique [Ouali et al., 2020]. In particular, multiple feature perturbations are incorporated into the decoders, and the image encoder is encouraged to provide enhanced image representations. Finally, we propose the multi-modal learning method to leverage the image-text samples. The multi-modal consistency learning and text-aware contrastive learning enable our model to be aware of the semantically prominent objects in the images.

In summary, our matting engine training pipeline incorporates labeled, unlabeled, and text-image samples. Specially, we design the basic encoder with three specific decoders for saliency, portrait and scenery category to improve the accuracy, as shown in Figure 1(b).

2.3 Functions of AIGCGeneration

The recommended content is generated by functions of AIGC as "Matting moments" from the perspective of time and space according to the category. To guarantee quality and computation efficiency of AIGC, our system presents a compact content selection and beautification functions with four matting schemes in Figure 3. a) Flash matting: to discover dominant subjects with easily copy and drag operations, flash matting is designed to make users easily edit the subjects. b) Sky matting: to achieve content beautification for foggy or smog weather, we apply blending and recoloring after sky matting to beautify the sky region, which greatly increases quality of scenery. c) Magic eraser: To meet with the demand of removing pedestrians in the scenery, we use the inpainting algorithm [Zeng et al., 2021] on the accurately extracted masks to achieve eraser capability. d) Stylization and colorization: With prominent subject of portrait mask, we apply lightweight GAN [Chiu and Gurari, 2022] to humanpreserving background stylizing and achieve portrait-only recoloring. With content selection and automatic beautification, the system presents the beauty of life moments with matting engine.

In addition, the whole system provides real-time mobile online services for a large number of photos. We develop a heterogeneous computing inference engine with hardware and software adaptation for real-time matting, inpainting and stylization for mobile AIGC application.

3 Experiments

We investigate the contributions of several learning components in "Data-driven Matting Engine" and implement alternative models of learning components. The experiments are conducted on the dataset containing 3K images of six categories including portrait, multi-person, pets, accessories, cards, and delicacy. MSE, Maxf, BoundaryIoU (BIoU), Connectivity (Conn) are adopted as evaluation metric. The contributed components in "Matting Engine" are composed of visual modality with pixel-wise supervised training (Pixel Sup), semi-supervised training (Semi Sup), and multiple modalities with text-image (Mlt Sup). We observe that Mlt Sup achieves the continuous better results along all the categories in Table 1, which demonstrates the superior performance for data-driven strategy of accurate matting.

4 Conclusion

In this paper, we propose a unified data-driven matting engine for exact subject matting for mobile AIGC application. The system included three aspects: 1) photo content structuring based on semantic parsing, 2) data-driven matting engine, 3) functions of AIGC generation. Our system achieves favourable performance in mobile AIGC products, which provides satisfying user experience.