TeSTNeRF: Text-Driven 3D Style Transfer via Cross-Modal Learning

1 Introduction

Given a collection of artworks, learning their internal expressions of art (such as color tones, strokes) and applying them to 3D scenes is a meaningful yet challenging task. From a practical point of view, artistic scene creations resembling style of various artists can be toured on visual reality (VR) and augmented reality (AR) devices, which provides users with a more intuitive understanding of artists' styles. One possible solution to 3D style transfer is to directly apply image/video stylization techniques to 3D scenes. However, these methods lack 3D scene perception. Without considering the underlying 3D structure, such methods may cause short-range flickering or long-range discontinuity when changing viewpoints.

The major challenge for 3D style transfer is to maintain consistency among different viewpoints to produce coherent results. Recently, Neural Radiance Field (NeRF) [Mildenhall et al., 2020] has shown superior performance in reconstructing 3D objects and scenes and novel view synthesis. An implicit neural scene can be estimated from a few image observations. NeRF uses MLP to regress both volume density and color. Some subsequent works [Schwarz et al., 2020; Niemeyer and Geiger, 2021; Gu et al., 2022] focus on disentangling shape and appearance to manipulate them. Nevertheless, they are mainly designed for objects, which have difficulty in stylizing complex 3D scenes.

∗Corresponding authors.

To address the problem, a number of works [Chiang et al., 2022; Huang et al., 2022; Zhang et al., 2022; NguyenPhuoc et al., 2022] investigate adopting NeRF for style transfer. Once trained, [Chiang et al., 2022] is capable to transfer arbitrary style to 3D scenes, while [Zhang et al., 2022] and [Nguyen-Phuoc et al., 2022] need to re-train the network every time given a new style image. These methods use an image for style reference. However, to achieve artist-aware stylization, it is more convenient to use a text for reference, as one reference image is not representative enough. Thus, we tend to guide the stylization with texts instead of images.

In this paper, we propose TeSTNeRF, a novel method for text-driven 3D style transfer. On the basis of a pre-trained NeRF where the 3D scene has already been reconstructed, we introduce latent codes to represent styles of different artists. Moreover, we leverage CLIP [Radford et al., 2021], a large cross-modal vision-language model, to align the input text and output stylized images. We also enhance the stylization by learning style feature statistics from image collection corresponding to the text. To generate more harmonious stylized scenes, we utilize results from 2D stylization approaches to alleviate abrupt color spill. Our experiments show that the proposed approach can produce different results by using distinct texts, which are also view-consistent.

In summary, our main contributions are threefold:

• To the best of our knowledge, we for the first time propose a novel approach to stylize 3D scene according to a given text via cross-modal learning, which can produce consistent novel views of high visual quality.
• We introduce latent codes to control styles of different artists, simplifying the representation of style domain. To establish the connection between texts and latent codes better, we adopt CLIP as the encoder.
• We conduct rectification of abrupt color spill utilizing 2D stylization results, which helps generate more harmonious visual results.

2 RelatedWork

2.1 Representing Scenes with Neural Field

In the past few years, rendering 3D scenes implicitly via neural networks [Jiang et al., 2020; Genova et al., 2020; Park et al., 2019; Riegler and Koltun, 2020; Mildenhall et al., 2020] has gained much concern. Among them, NeRF [Mildenhall et al., 2020] has achieved incredibly highquality results in reconstructing 3D objects and scenes. Given a few images, NeRF encodes a continuous neural radiance field to render photo-realistically novel views. The success of NeRF has inspired various follow-up works, extending NeRF to generative models [Schwarz et al., 2020; Niemeyer and Geiger, 2021; Gu et al., 2022], decomposing rendering of scenes [Boss et al., 2021; Martin-Brualla et al., 2021; Yang et al., 2021], etc. We leverage NeRF as our backbone to learn volume density and view-dependent color for the scene, which can also be extended to better-quality NeRFs [Zhang et al., 2020; Barron et al., 2021].

2.2 2D Style Transfer

2D style transfer includes image style transfer and video style transfer. Image style transfer is to combine style feature from an image and content feature from another image, whose style is similar to the former and content the latter. Many works [Huang and Belongie, 2017; Li et al., 2017] investigate approaches to perform the combination. To represent the full scope of artistic style, learning style from a collection of artworks, which forms a style domain, is another perspective of image style transfer. AST [Sanakoyeu et al., 2018] manages to extract shared qualities among a group of artworks. DualAST [Chen et al., 2021] develops a scheme to learn simultaneously both the holistic artist-style and the specific artwork-style via its proposed Style-Control Block. StyleBank [Chen et al., 2017b] utilizes multiple convolution filter banks, each of which explicitly represents one style domain. Moreover, image translation approaches, like CycleGAN [Zhu et al., 2017], can also deal with collection style transfer. Some other works [Kwon and Ye, 2022; Fu et al., 2021]learn styles from text descriptions. TxST [Liu et al., 2022] proposes to embed an image-text model.

Directly applying image style transfer techniques to video frame-by-frame usually causes instability and flickering. Video style transfer [Chen et al., 2017a; Gao et al., 2018] tackles this problem by introducing optical flow or aligning intermediate feature to constrain nearby video frames. MCCNet [Deng et al., 2021] rearranges style representations based on content representations to make style patterns suitable for content structures. With the help of temporal regularization, ReReVST [Wang et al., 2020] reconciles the contradiction between style transfer and temporal consistency.

2.3 3D Style Transfer

3D style transfer has higher requirements compared to video style transfer, since novel view synthesis is required in 3D scenes. Previous methods extend stylization to 3D scenes by representing 3D scenes with point clouds [Huang et al., 2021] or meshes [Yin et al., 2021]. Due to the success of NeRF, some approaches explore 3D style transfer based on NeRF. Stylizing-3D-Scene [Chiang et al., 2022] develops a hypernetwork to control the appearance-related weights of the NeRF model. StylizedNeRF [Huang et al., 2022] proposes a mutual learning strategy for NeRF and 2D stylization method. SNeRF [Nguyen-Phuoc et al., 2022] deals with the memory limitation of training with a whole image in NeRF and improves the visual quality by alternating between stylization and NeRF training. ARF [Zhang et al., 2022] aims to transfer detailed style features via matching features between the style image and the scene. These methods guide the stylization with images. Instead, we focus on realizing 3D style transfer under the guidance of texts.

2.4 CLIP-based NeRFs

Recently, a model based on Contrastive Language-Image Pretraining (CLIP) [Radford et al., 2021] learns a latent space, which may be used to estimate the similarity between a text and an image and promote the development of cross-modal learning. The powerful representation learned by CLIP narrows the gap between texts and images. CLIP-NeRF [Wang et al., 2022] leverages CLIP to flexibly control the 3D content through texts or images, providing an interactive approach to manipulate the shape and appearance of 3D objects. Dream Fields [Jain et al., 2022] generates objects via NeRF with the help of CLIP. LaTeRF [Mirzaei et al., 2022] uses CLIP to remove artifacts while extracting objects that is partly occluded from a scene. DFFs [Kobayashi et al., 2022] distills the knowledge of CLIP-LSeg, a model for segmentation, into a 3D feature field to semantically decompose 3D scenes.

We propose to utilize CLIP for stylization in 3D scenes. Given texts, we use the text embeddings encoded by CLIP to manipulate which style domain the scene should be transferred to and align the input text and output stylized images in CLIP latent space.

3 Preliminary

NeRF [Mildenhall et al., 2020] adopts multi-view images of a 3D scene to optimize the underlying continuous radiance field, using MLP to output volume density σ and color c given a point coordinate x and its view direction d. The whole networks can be disentangled into three parts: Fθ, Fσ, and Fc. Practically, volume density and color can be calculated separately as:σ = Fσ(Fθ(x)), c = Fc(Fθ(x), d). (1)

During volume rendering, a ray r(t) = o + td is cast from the center o of the camera along the direction d through a pixel in the image. According to the volume rendering equation, each pixel's color is integrated along the ray as:

where tn and tf represent the near and far bounds of the ray. The image taken from a specific viewpoint is finally generated by aggregating its pixels according to Eq. 2 and 3.

4 Method

Our goal is to stylize a scene generated by a set of photos from different viewpoints. We carry out the implementation based on pre-trained implicit reconstruction of the scene using NeRF. During stylization, we fix the parameters of Fθ and Fσ to maintain the geometry of the scene, and only optimize Fc to change the appearance. Existing methods [Chiang et al., 2022; Huang et al., 2022; Zhang et al., 2022; Nguyen-Phuoc et al., 2022] achieve stylization under the guidance of style images. Our proposed TeSTNeRF manages to stylize the scene with texts as conditional inputs. Please refer to Fig. 2 for an overview of our proposed framework.

4.1 Cross-modal Text-driven Style Transfer

Given paintings from different artists, it is not a difficult task for experienced people to distinguish which painting belongs to whom. Thus, the artists' name can be used as a high-level representation, denoting different style domains. Inspired by TxST [Liu et al., 2022], we leverage a latent space with different latent codes to control different style domains. Although it is possible to distinguish style domains with one-hot encoding, one-hot encoding may lead to poor performance since its representation lacks concrete meaning in latent space. To directly handle a text t as input, we project t into CLIP text latent space. With the powerful representation of CLIP's text embedding, the latent code z guides the optimization of hypernetwork Ψ, which predicts the parameters W c to update Fc. In this way, we utilize CLIP embedding as style condition to manipulate the appearance of a scene.

To ensure the generated images containing the same content as the origin images of the scene, we introduce content loss. Features produced by pre-trained VGG-19 network [Simonyan and Zisserman, 2014] ϕ can effectively capture intrinsic representation of an image, which is often used in image style transfer. Denoting Icontent as the ground truth image and Iout as the stylized image, we compute the content loss as:

where ϕ4 denotes the relu4 1 layer in VGG-19.

Cross-modal learning aims to effectively utilize the correlation of different modal contents for modeling. Since CLIP has made a significant breakthrough in evaluating the similarity of image-text pairs, we carry out our cross-modal learning based on CLIP. For an image and a text, the similarity of their features is proportional to the probability of the text being associated with the image. Thus. we define a text-image crossmodal loss LTIC to shorten the CLIP-space distance between the input text t and the output stylized image Iout:

where ζi and ζt are pre-trained CLIP image encoder and text encoder, respectively, and ⟨, ⟩ denotes cosine similarity. Through training with Lcontent and LTIC, we are capable to generate stylized scene corresponding to the input text.

4.2 Style Supervision

Although cross-modal learning can produce stylized results, the results are not visually-pleasant enough, as shown in Fig. 6. To tackle this problem, we introduce style loss to assist in better learning the style feature, which is usually used in image style transfer. We utilize a collection of style images corresponding to the text, where a style image is randomly chosen each iteration. Style loss Lstyle measures the error between feature statistics of the style image Istyle and the stylized image Iout as:

where µ and s are channel-wise mean and standard deviation, respectively. ϕi denotes a layer in VGG-19 used to compute the style loss. In our experiments, we use relu1 1, relu2 1, relu3 1, and relu4 1 layers.

Previous domain-based 2D style transfer approaches [Sanakoyeu et al., 2018; Zhu et al., 2017] generate globally color harmonious results due to their effective learning from a large scale of images. They learn a mapping between a source domain and a target domain, generating results that are indistinguishable from the target domain. To rectify color spill when training with limited data in a scene, we leverage 2D stylization to supply additional supervision. We adopt CycleGAN [Zhu et al., 2017] to provide the 2D stylization supervision. Specifically, we utilize their pre-trained collection style transfer models. It should be noted that CycleGAN is a representative domainbased style transfer method, which may be replaced by other advanced methods. 2D stylization supervision is introduced via computing L1 loss between the corresponding result generated by 2D style transfer method I2D and 3D output stylized image Iout:

The loss L2D constrains the output of 3D style transfer to alleviate the influence of color spill from irrelevant object in the scene, thus leads to better visual quality, as we will later demonstrate in Fig. 6.

4.3 Total Loss

In summary, the content loss Lcontent is to make sure that the generated images contain the same content as the origin images of the scene, while LTIC is utilized to align the input text and output images in a pre-trained latent space. Meanwhile, the style loss Lstyle is to guide the model to learn from a specific collection of images corresponding to the input text, and L2D is leveraged to mitigate the abrupt color spill, yielding high-quality stylization results. The final loss function L

used to train our model is:

where the constants λcontent, λTIC, λstyle, and λ2D are hyper-parameters of the model.

5 Experiment

5.1 Implementation Details

Following NeRF [Mildenhall et al., 2020], we use separate coarse and fine sampling strategies, both sampling 64 points each ray. To represent a larger region with a limited amount of rays, we generate a K ×K patch P containing pixel (u, v) using stride c:

Specifically, we set the patch size K = 100 for all experiments. The loss weights in Eq. 8 are set to λcontent = 0.1, λTIC = 1, λstyle = 1, λ2D = 0.3. Our proposed method is optimized using Adam optimizer with β1 = 0.9, β2 = 0.999, and its learning rate starts from 1e-4 . The proposed TeSTNeRF is trained for 1500 iterations on a single NVIDIA RTX 3090 GPU.

Datasets

We utilize scenes provided by NeRF [Mildenhall et al., 2020] and FVS [Riegler and Koltun, 2020]. All of them are captured using handheld cameras in real-scenes.

Baselines

We compare TeSTNeRF to SOTA image stylization methods, video stylization methods and 3D stylization methods. Since implementing image style transfer first and then reconstructing the scene is not stable and fails sometimes, which is caused by inconsistent stylization across different views (we also provide the failure case in our supplementary video), we compare our method to the following 3 categories of schemes:

• Novel View Synthesis→Image Style Transfer: we perform novel view synthesis using NeRF, and stylize each new view using CycleGAN, AST, and DualAST.
• Novel View Synthesis→Video Style Transfer: we perform novel view synthesis along a smooth camera path using NeRF, gather the results as a video, and then stylize the video using ReReVST and MCCNet.
• 3D Style Transfer→Novel View Synthesis: we perform 3D scene stylization and then synthesize novel view. Specifically, we conduct a point-cloud-based method LSNV, NeRF-based methods StylizedNeRF and ARF.

For CycleGAN and AST, we use their provided models pre-trained on the corresponding collection of artworks. For the rest methods, we choose a style image from the artist's domain corresponding to the given text.

5.2 Qualitative Results

In Fig. 3, we show our stylization results based on different texts in various scenes. It can be seen that the stylized scenes contain both view-consistent visual quality and satisfying text-driven style controllability.

To validate the superiority of our method, we compare our results with baselines, as shown in Fig. 4. We show results generated by different methods on the scene, Sandbox.

Image style transfer approaches, e.g., CycleGAN, AST, and DualAST, are able to produce high quality stylized results each viewpoint. However, they produce obvious inconsistency artifacts, as highlighted in red boxes.

Although video style transfer approaches, e.g., ReReVST and MCCNet, are able to maintain short-range consistency, they struggle to deal with long-range instability. Moreover, video style transfer methods lack perception of the spatial information in the scene, resulting in violation of geometry of the scene. Please see their failures as reconstructing the sandbox with jagged edges in blue boxes in Fig. 4, while their inconsistencies are highlighted in red boxes as well.

3D style transfer approaches, e.g., LSNV, StylizedNeRF, ARF, and ours, aim to stylize the holistic scene, therefore they all generate view consistent results. However, LSNV cannot capture the reference style well, which affects its visual quality. StylizedNeRF and ARF utilize NeRF as their geometry representation, which is the same as our method. StylizedNeRF and ARF learn style from a reference image, while our method takes in a text as the style reference.

In comparison, we utilize a text instead of an image as the reference style. We produce both view-consistent and highquality visual results. We encourage readers to have a look at the supplementary video, which shows more intuitive differences between our method and baselines, especially in consistency and stability when changing viewpoints.

We also conduct experiments on more texts than just the artist's name. In this case, we only use Lcontent and LTIC in Eq. 8 to train the network. The result is shown in Fig. 5.

5.3 Quantitative Results

Consistency Measurement

Following the measurement in LSNV, we use a warped LPIPS metric [Zhang et al., 2018] to measure the consistency across different views. Firstly, we utilize FlowNetS [Dosovitskiy et al., 2015] to compute the optical flow from a ground truth image Ix to another Iy. Then a warped mask M is generated according to the optical flow. Finally, we warp the corresponding stylized images ˆ Ix to ˆ Iy and calculate their distance along with M. The distance score is formulated as:

where ⊙ denotes element-wise multiplication.

We compare our method with baselines on three scenes, Fern, Orchids, and Vasedeck, reporting average warped distance score on texts "Van Gogh" and "Monet". For shortrange consistency, we randomly choose 20 adjacent novel views (ˆ It, ˆ It+1) from each scene, as shown in Tab. 1. In Tab. 2, we show the long-range consistency score, randomly choosing 20 frame pairs (ˆ It, ˆ It+7) from each scene. From these two tables, we observe that TeSTNeRF outperforms baselines in short-range consistency in all scenes, and performs the best or second best as for long-range consistency.

User Study

We also conduct a user study for subjective evaluation. We invite 27 male and 23 female participants to score stylized scenes in visual quality and temporal consistency. We first show a text denoted an artist with three representative artworks from the artist as reference. We carry out the study on scene Orchids, providing its stylized result of each method together with a ground truth video for easy comparison. The participants are required to score the visual quality and temporal consistency, where 10 denotes excellent performance, and 1 denotes poor performance. The results are shown in Tab. 3. We observe that our method outperforms other methods both in visual quality and temporal consistency.

5.4 Ablation Studies

In this section, we explore each component's effect in TeSTNeRF and validate their importance by ablation studies.

With and without style loss. In this study, we explore the effect of including style loss Lstyle. We introduce Lstyle to learn the style feature statistics. The model trained with only Lcontent and LTIC shows stylized result, but its visual quality is dissatisfying, as shown in Fig. 6(a). It demonstrates that introducing style feature statistics is necessary.

With and without text-image cross-modal loss. We introduce LTIC to align the CLIP-space embeddings between the text and the stylized image. In Fig. 6(b), the model trained with only Lcontent and Lstyle presents color supersaturation, with color not existing in the artist domain and in this image especially on flowers highlighted in the red box. Regarding the benefits of aligning texts and images, we conclude that text-image cross-modal loss is worth adopting.

With and without 2D loss. As shown in Fig. 6(c), when training without L2D, there exits color spill from the desk to the vase, highlighted in the red box. To rectify the color spill, we leverage results from 2D stylization to supply additional supervision. From Fig. 6(d), we can see that applying this supervision helps resist with the color spill and provide better visual results.

6 Conclusion

In this paper, we investigate text-driven 3D style transfer and propose TeSTNeRF to stylize scenes. Specifically, we utilize CLIP to encode the input text to a latent code, guiding a hypernetwork to predict the parameters related to appearance of the scene in NeRF model and conduct crossmodal learning by aligning the input text and output results in CLIP latent space. Moreover, adopting style supervision improves the stylized visual quality. The experiments on 3D scenes demonstrate the effectiveness of TeSTNeRF for 3D style transfer according to given texts.

Acknowledgements

This work was supported in part by the projects No. 2021YFF0900604, 2020YFC1523202-2, 19ZDA197, LY21F020005, 2021009, 2019011, 2022C01222, Research and Application Demonstration of Key Technologies in Digital Archaeology of Ningbo Prehistoric Sites, Key Technologies and Product Research and Development for Cultural Relics Protection and Trading Circulation, MOE Frontier Science Center for Brain Science & Brain-Machine Integration (Zhejiang University), National Natural Science Foundation of China (62172365).