Eye-See-You: Reverse Pass-Through VR and Head Avatars

1 Introduction

Augmented Reality (AR) and Virtual Reality (VR) have become critical technological advancements, transforming industries such as gaming, remote collaboration, education, and healthcare [Al-Ansi et al., 2023; Rambach et al., 2020; Kanschik et al., 2023]. As immersive technologies, they enable new forms of interaction and engagement, reshaping human-computer interfaces and digital experiences. While VR headsets have become mainstream consumer technology, they inherently isolate users from their surroundings, limiting their integration into shared environments and public spaces [Hobbs, 2017; Gugenheimer et al., 2019]. Eye contact is a cornerstone of human connection and emotional communication, yet current VR headsets obscure users' eyes and facial expressions, severing visual interaction with the real world. This lack of transparency not only diminishes social presence but also leaves bystanders unaware of the user's engagement with VR content or their attentiveness.

Addressing this fundamental limitation requires transformative AI-driven solutions to bridge the gap between virtual and physical environments. One such approach is reverse pass-through technology, which reconstructs and displays a user's eyes and facial expressions on the outward-facing surface of the headset. This technique enables real-time interaction, allowing bystanders to perceive eye movements and emotional expressions, effectively bridging the gap between virtual and physical environments.

Efforts to mitigate VR isolation made effort to maintain social presence, while they suffer from inauthentic eye movements [Chan and Minamizawa, 2017; Bozgeyikli and Gomes, 2022], hardware requirements, limited facial reconstruction [Matsuda et al., 2021], and underwhelming performance [Apple, 2024], as shown in Table 1. Besides, photorealistic avatar generation has significantly improved realism [Feng et al., 2021; Danecek et al., 2022; Zheng et al., 2023; Grassal et al., 2022; Li et al., 2023], but limited by the need for multi-view images or specialized VR headsets. Personspecific avatar generation struggles with high customization needed, hindering widespread adoption [Lombardi et al., 2018; Wei et al., 2019; Schwartz et al., 2020].

To address the limitations of current VR systems, we propose RevAvatar, a framework to generalize across VR headsets with minimal device-specific fine-tuning. Unlike existing solutions, RevAvatar leverages advanced AI techniques to overcome the isolation caused by VR headsets by reconstructing and displaying the user's eyes and facial expressions in real-time, enabling seamless interaction between virtual and physical environments. Additionally, RevAvatar facilitates the creation of full-head 3D avatars, enhancing immersive experiences for applications like virtual meetings. The main pipeline combines real-time 2D face restoration for "reverse pass-through" and a one-shot 3D avatar generation model, achieving 0.008-second inference on mobile SoCs like Apple M2. It is compatible with consumer mixed-reality devices such as the Apple Vision Pro and upcoming Samsung and Google VR headsets. Crucially, it eliminates the need for 3D scans, requiring only a selfie-like Digital Persona (DP) image, improving accessibility and convenience.

Developing generalized solutions for diverse VR headsets is challenging due to variability in camera specifications and placements across brands like Apple, Samsung, Meta, Vive, and Varjo. To address this, we introduce VR-Face, a novel dataset comprising 200,000 samples that simulate diverse VR conditions, including occlusions, lighting variations, and distortions. VR-Face not only supports RevAvatar's development but also provides a foundational resource for advancing research in AI-driven VR technologies.

Our contributions are: ① RevAvatar Framework: We introduce RevAvatar, an AI-driven framework for real-time reverse pass-through and 3D avatar generation in VR. This solution enhances VR immersion by eliminating the need for user-specific models or custom hardware. ② Efficient AI for Mobile SoCs: Our 2D face reconstruction model operates efficiently on mobile SoCs, such as the Apple M2 in Apple Vision Pro, achieving an inference time of just 0.008 seconds. This demonstrates its scalability across diverse VR devices. ③ VR-Face Dataset: We present VR-Face, a public dataset with 200,000 samples simulating challenging VR scenarios. It supports the development of AI technologies adaptable to various headset specifications and advances research in VR. ④ AI-Enabled VR Advancements: Through RevAvatar and VR-Face, we drive significant AI innovations that address key VR challenges like user isolation and hardware diversity, setting a new standard for AI-driven progress in VR.

2 RelatedWork

Eye tracking based animation FrontFace [Chan and Minamizawa, 2017] and Googly Eyes [Bozgeyikli and Gomes, 2022] aimed to represent user attention and gaze using animated eye movements via eye tracking and Head-Mounted Displays (HMDs). However, these approaches focus on displaying animated eye states, such as whether the eyes are open or closed, and the gaze direction, without conveying genuine emotions or expressions.

Eye and Face Reconstruction: Reverse Pass-Through A "reverse pass-through" prototype headset [Matsuda et al., 2021] reconstructs and displays users' eyes on an external screen but requires costly and custom hardware inaccessible to most users. Apple's Vision Pro with "EyeSight" projects eyes onto an external display, but its functionality remains unclear [Apple, 2024], and early reviews suggest underwhelming performance [Chokkattu, 2024; Patel, 2024]. Other face-restoration methods from partial VR headset data often require multiple views or customized headsets, making them impractical for widespread use. They also rely on user-specific avatars, requiring costly individualized training and limiting real-world use [Lombardi et al., 2018; Wei et al., 2019; Schwartz et al., 2020].

Face image composition and one-shot Avatar generation Several methods were proposed for face composition and synthesis. PixelStyle2Pixel (PSP) [Richardson et al., 2021] performs image-to-image translation using StyleGAN's latent space, while failing to preserve the identity of unseen individuals. StyleMapGAN [Kim et al., 2021] faces similar identity preservation challenges during inference. One-shot photo-realistic avatar generation made significant strides, like ROME [Khakhulin et al., 2022] generating mesh-based avatars from single images, and CVTHead [Ma et al., 2024] using transformers and point-based neural rendering. Portrait4D [Deng et al., 2024] employs a part-wise 4D generative model for synthesizing multi-view images and leverages transformers to create highly detailed, animatable avatars. They are closest to our task of one-shot avatar generation.

VR simulation dataset Despite growing interest in VR simulations, publicly available datasets remain scarce. Eye images captured by IR cameras in VR headsets suffer from occlusions and limited fields of view. MEAD data [Chen et al., 2024] was used for VR simulations but lacked realworld scenario complexity, particularly in occlusions like eyebrow obstruction. Other works [Lombardi et al., 2018; Wei et al., 2019] used custom VR headsets with IR cameras for optimal eye capture, but these setups are not generalizable to commercial headsets, and datasets are publicly inaccessible.

3 VR-Face Dataset

We develop the VR-Face dataset (Figure 2), containing 200,000 samples, each comprising a full-face image, left and right eye images from various angles, and a lower-face image. The dataset captures a wide range of facial expressions and perspectives, with pre-processing to simulate visual effects observed in VR headset imagery (details in Table 2). VR-Face is designed to be inclusive, representing diverse skin tones, sex, race, and ethnicity. While it serves as a benchmark for reverse pass-through and analysis, it is not intended to replace real-world VR headset data but can be adapted to specific devices with suitable datasets.

4 Methodology

RevAvatar comprises an initial setup stage and the main pipeline, which consists of image alignment, 2D face restoration, and 3D avatar generation for reverse pass-through and immersive VR experience, as shown in Figure 3. and 4.

During the initial VR headset setup, the user is prompted to capture a selfie using the headset's external camera or upload an image via their online account. Manufacturers provide detailed instructions to ensure the image captures the entire face without occlusions and under good lighting. The image is then processed using a facial landmark model [Bulat and Tzimiropoulos, 2017] for cropping and alignment in subsequent stages. This selfie, referred to as the Digital Persona (DP) image, serves as the reference for 2D and 3D reconstruction.

The main pipeline aligns or frontalizes the processed eye and face-tracking images. A lightweight GAN-based model [Goodfellow et al., 2014; Dash et al., 2024] is then used for full face restoration. For Avatar generation, we combine 3DMM [Egger et al., 2020] models with neural networks for tri-plane representation and volume rendering to achieve accurate 3D reconstruction.

4.1 Eye and Face Alignment

We use VR-Face dataset, consisting of non-aligned images of left and right eyes, with the goal of aligning or frontalizing these images (Figure 3-A). To achieve this, we adopt a CycleGAN-based framework [Zhu et al., 2017], which learns bidirectional mappings between tilted eye images (domain A) and frontal eye images (domain B) through CycleConsistency. We use two distinct alignment models, CGLE and CGRE for each eye, and use a separate model CGFace to frontalize the lower face image. To preserve gaze, we use eye landmark detection [Bulat and Tzimiropoulos, 2017] and compute Gaze Estimation Error as the angular difference between aligned and ground truth images, achieving errors below one degree. Aligned eye and lower face images are then pasted onto the DP image using facial landmarks. Dataset details are in Section 5.1, with examples in Figure 3-B.

4.2 2D Full Face Restoration

To restore a full face image from frontalized eye and face tracking images, we develop a lightweight GAN-based restoration model with a Generator G and Discriminator D. It takes aligned grayscale left and right eye images, along with a lower face image pasted onto the color DP image, and generates a restored output where the grayscale images blend seamlessly with the rest of the face, capturing facial expressions. Figure 4. shows our 2D Face Restoration Framework. To handle high occlusion of input eye images, the model uses a reference image for reconstructing facial features, providing context for occluded areas like eyebrows. During training, random reference images from various users are selected to improve robustness. During deployment, the DP image serves as a reference for reconstructing occluded areas.

The Generator G comprises an Input Encoder EI and a Reference Encoder ER, which share a similar structure. EI takes the DP image with partial VR observations pasted on top of the base DP image x as the input, and the reference DP image z is given as input to ER. To enhance the model's ability to capture both global and fine-grained details, the encoders are extended to operate at multiple scales, where each scale extracts features at different levels of resolution. The Generator is built on multiple ResNet blocks, with each block processing features at different resolutions. We use cross-attention module to align and integrate features from the reference image with those of the input image, enhancing context-aware and guided image generation. The crossattention module operates at multiple scales, where each scale learns to focus on different levels of detail, from coarse structures to fine textures. This multi-scale feature fusion ensures the generation of high-fidelity images by combining both global context and local detail. The architecture begins with initial convolutional layers for downsampling through EI and ER, extracting multi-scale features. The multi-scale features are then processed through cross-attention and residual block processing. Finally, the decoder reconstructs the synthesized image G(x, z), utilizing the multi-scale features to improve the quality of the generated image at all levels. We use a Multiscale Discriminator, which employs multiple instances of a PatchGAN [Isola et al., 2017] discriminator, each responsible for evaluating the image at a specific scale.

The Generator G is trained to minimize the following loss functions. (1) The adversarial loss Ladv ensures that the generated image G(x, z) is indistinguishable from real images y by the Discriminator D. (2) The L1 loss LL1 ensures the generated image G(x, z) is close to the ground truth image y. (3) The LPIPS (Learned Perceptual Image Patch Similarity) [Zhang et al., 2018] loss LLPIPS assesses perceptual similarity between the generated image G(x, z) and the target image y. The total loss is defined as Ltotal = LG +LD.

where the weights λadv, λL1, and λLPIPS balance the contributions of each loss term.

The Discriminator D is trained to distinguish between real images and generated images using the adversarial loss:

LD = Ey [log D(y)] + Ex [log(1 - D(G(x, z)))] (3) where D(y) is the probability that the image y is real, and D(G(x, z)) is the probability that the generated image G(x, z) is real.

4.3 One-shot 3DHead Avatar Model

We extend our reverse pass-through system to generate fullhead avatars for immersive VR, building on recent one-shot facial avatar generation advancements. This approach overcomes the limitations of requiring specialized models for each subject or multi-view inputs, addressing challenges that hinder practical real-world applications.

Our Framework uses the user's DP or selfie image as the source image Is and the reconstructed image from 2D fullface restoration serves as the driving or target image It. The source image is used to extract the identity, and the target image is responsible for providing the pose and expression information. Our framework comprises three main branches which include the Global Branch EG, Detail Branch ED and the Expression Branch EE. The output is then up-scaled and refined using a super-resolution module.

The EG branch uses a hybrid transformer model with a series of convolutional and transformer blocks along with SegFormer [Xie et al., 2021] to generate a tri-plane representation. We use SegFormer as it allows for effective mapping from 2D space to 3D space. This is achieved by predicting a tri-plane Tg that represents the neutral expression of the human face in a canonical 3D space. To ensure that the generated tri-plane Tg aligns with the identity of Is and maintains a neutral expression, we incorporate a 3D Morphable Model (3DMM) to render a face with the same identity and camera pose as the source image, but with a neutral expression. The Detail Branch ED builds on the geometry provided by the Global Branch by capturing and reconstructing intricate facial details from the source image Is. The features of the Detail Branch are transferred to the global triplane, creating an appearance triplane Td that improves the initial reconstruction with fine-grained details, such as texture and surface features. The Expression Branch focuses on modeling and transferring the expression from the target image It onto the reconstructed 3D avatar. This branch utilizes a 3DMM to predict the shape and expression coefficients for both the source image Is and target image It. The expression coefficients of It are used to render a frontal-view expression image Ie, which is then encoded into an expression tri-plane Te. This expression triplane is added to the canonical tri-plane Tg along with the appearance tri-plane Td to generate the final 3D reconstruction. The integration of these three branches allows the model to combine the identity from the source image with the expression and head pose from the target image, effectively transferring the desired expression onto the source image while maintaining high fidelity in both appearance and geometry. Given the high computational demands of volumetric rendering, we first render low-resolution images and then use a super-resolution [Wang et al., 2021] module to produce the final high-quality output.

We use a two-stage training schedule for multi-view consistency and efficiency. In the first stage, the model trains at a lower resolution without an upscaling module, optimizing L1 and LLPIPS losses between the Global Branch feature rendering and the 3DMM rendering of the source image, as well as the combined tri-plane features and target image.

In the second stage of training, we only fine-tune the upscaling module using L1, LLPIPS, and GAN loss objective. Additionally, we use an eye region loss which calculates the L1 between only the eye region rendering the output image and the target ground truth image to ensure accurate gaze.

4.4 Reverse Pass-through and Avatar Outputs

The output from our Full Face Restoration model can be cropped to display the eye region and realize reverse passthrough on mainstream VR headsets(Figure 5). This allows users to maintain eye contact and convey expressions. Additionally, the outputs from our head avatar model can be leveraged for immersive applications such as VR meetings and the metaverse, providing visually accurate and expressive 3D avatars that enhance virtual interactions and communication.

5 Experiments

5.1 Datasets

We utilize VR-Face as the main dataset to train and test our framework, and three additional datasets for training to enhance the generalization. (1) The Eye and Face Alignment model is exclusively trained and tested on VR-Face dataset. (2) For 2D face restoration model, we integrate CelebHQ [Karras et al., 2017] and FFHQ [Karras et al., 2019] datasets, which contain images only, in conjunction with VR-Face dataset for training, allowing for better generalization across different skin tones and facial attributes. (3) For 3D avatar generation, besides VR-Face, we further leverage FFHQ, CelebV-HQ [Zhu et al., 2022], VFHQ [Xie et al., 2022] datasets to provide a rich set of facial attributes and emotional variations.

5.2 Implementation

All models are trained on 512 × 512 images. The Alignment and 2D face restoration models use a single A100 GPU, while the 3D head avatar model is trained on 4 A100 GPUs. We train three alignment models: two for the eyes and one for the lower face. Batch sizes are set to 32 for Alignment, 16 for 2D face restoration, and 4 for the 3D avatar model. All models use the Adam optimizer with a 0.0001 learning rate. During inference, the alignment models run in 0.004s, the face restoration model in 0.006s per image, and the avatar model achieves 22 FPS. All inferences are performed on a single A100 GPU including mobile GPUs such as Apple M2 GTX 1050, and MX350.

5.3 Baseline

Given the lack of well-established baseline models for reverse pass-through VR, a direct holistic comparison of our entire framework is not feasible. Instead, we evaluate our 2D full-face restoration model by comparing it with state-ofthe-art GAN-based approaches in image composition and reconstruction: CycleGAN, PSP [Richardson et al., 2021], and SMG [Kim et al., 2021]. Additionally, we include DifFace [Yue and Loy, 2024] as a diffusion-based baseline. Although we considered diffusion models such as OSDFace [Wang et al., 2024], DifFace [Yue and Loy, 2024], OSEDiff [Wu et al., 2024], and DiffBIR [Lin et al., 2024], their inference times of 0.1, 6.1, 0.12, and 8.01 seconds, respectively, on an A100 GPU make them unsuitable for real-time applications. In contrast, CycleGAN, PSP, SMG, and our model achieve real-time performance with inference times of 0.016, 0.041, 0.038, and 0.006 seconds, respectively. For 3D avatar reconstruction, we benchmark our model against leading oneshot approaches: ROME [Khakhulin et al., 2022], CVTHead [Ma et al., 2024], and Portrait-4D [Deng et al., 2024].

5.4 Results

For our 2D face reconstruction model and avatar, we generate images at 512x512 resolution and evaluate them against ground truth images using three metrics: SSIM [Wang et al., 2004], PSNR, and LPIPS. These metrics assess visual accuracy and perceptual quality by considering structural similarity, pixel-level differences, and perceptual relevance.

2D Face Restoration Figure 6. and Table 3. presents the qualitative and quantitative comparison between our face restoration model and baselines. Despite its lightweight design, our model performs better than other baselines. While CycleGAN performs comparably to other GAN-based models, it struggles to effectively colorize and blend eye and lower face, leading to severe artifacts, as shown in Figure 6. PSP

and SMG, which rely on StyleGAN-based generators, map inputs to a latent space, resulting in a loss of identity and inaccurate face restorations. During testing, SMG tends to output similar images from its training set but with altered expressions, while PSP produces outputs that often diverge significantly from the ground truth, as highlighted in Figure 6. This exposes a critical limitation of StyleGAN-based models: poor generalization on unseen data. DifFace, as a diffusionbased model, achieves high SSIM and PSNR for full-face reconstruction, outperforming other baselines in preserving global facial structure. However, it struggles to retain individual identity, leading to subtle yet noticeable shifts in facial features. Additionally, the iterative nature of the diffusion process results in significantly higher inference time, making DifFace less suitable for real-time applications. In contrast, our model excels at handling unseen face images, demonstrating superior generalization capabilities. It achieves the highest PSNR and the lowest LPIPS, indicating better perceptual quality and sharpness. Moreover, its inference time is orders of magnitude faster than DifFace, making it highly efficient for real-time applications. Figure 5 depicts how the output of our face restoration model enables real-time reverse passthrough capabilities in VR applications.

3D Head Avatar For 3D avatar generation, we compare rendered images with ground truth images. Additional videos showing different views for the reconstructed 3D avatars, as well as the appendix, are available at 1 . Our model shows significant improvements in key metrics, achieving the highest SSIM and PSNR, along with the lowest LPIPS, as shown in Table 5. It indicates that our model excels in maintaining both structural integrity and perceptual quality. The high SSIM reflects our model's ability to accurately capture fine details and facial features, while the superior PSNR highlights its robustness in minimizing reconstruction noise and artifacts. Moreover, the lower LPIPS suggests that our method produces image reconstructions that are perceptually closer to the ground truth, ensuring high-fidelity 3D avatars with realistic texture. Although Portrait4D-v2 performs competitively and ranks slightly behind our model, the noticeable jittering in the output faces during eye blinks affects the overall realism. ROME and CVTHead exhibit more challenges in preserving facial identity, reflected in higher LPIPS and lower SSIM. CVTHead, in particular, struggles to maintain identity consistency across different poses, as shown in Figure 7. In contrast, our approach preserves identity more effectively, yielding visually accurate avatars that are faithful to the subject's original appearance. This underscores our model's ability to generate high-quality, realistic 3D avatars with improved generalization across diverse inputs.

1 https://github.com/ankan2709/eye-see-you-vr

Realtime FaceReconstruction To assess the real-time performance of our 2D face reconstruction model for reverse pass-through on VR headsets like the Apple Vision Pro (with Apple M2 chip) and Meta Quest 3, we tested the model and compared inference times on the A100 GPU (used for training) with the Apple M2 SoC (in the Vision Pro), tested on a MacBook Air (8-core CPU, 4 performance cores, 4 efficiency cores, 10-core GPU, 16-core Neural Engine, and 16GB unified memory), as well as on the NVIDIA MX350 and GTX1050. Table 4 shows that the Apple M2, with MPS acceleration, delivers inference times comparable to the A100, demonstrating its ability to efficiently run complex models in real-time. The NVIDIA MX350, though slower due to its older architecture, and the GTX1050, showed promising performance for model inference.

Ablation Study Table 6 presents the quantitative results of ablation experiments. Eye alignment ablation: We compare the performance of Cycle-GAN versus Auto Encoder (AE) [Bank et al., 2021] for eye and face alignment, as described in section 4.1. The results show that Cycle-GAN outperforms AE in alignment tasks, leading to better reconstruction quality. Cross attention ablation: The model's performance significantly degrades when the reference and input features are concatenated, rather than using cross attention. This highlights the importance of cross attention in capturing fine-grained details for accurate reconstruction. Reference image ablation: Omitting the reference image results in lower performance, as it limits the model's ability to accurately reconstruct occluded areas, which are critical for realistic face restoration. The absence of this contextual information hinders the model's ability to recover facial features that are obstructed in the input. LPIPS ablation: Excluding LPIPS loss degrades the perceptual quality of the generated images, as evidenced by increased LPIPS score. Including it helps the model generate more visually accurate and perceptually consistent reconstructions by optimizing for human perception rather than pixel-wise similarity alone.

6 Conclusion

We introduce RevAvatar, an AI-driven solution to mitigate social isolation induced by VR headsets by restoring full-face images from tracking cameras using a user's DP image, enabling real-time eye movement display on an outward-facing VR screen. Additionally, RevAvatar generates realistic oneshot full-head avatars for VR meetings and interactions. As AR/VR continues to revolutionize digital interaction, we support this advancement with VR-Face, a dataset designed to simulate real-world VR scenarios and drive research in this field. Through RevAvatar and VR-Face, we aim to set new benchmarks for AI-driven VR experiences, enhancing social presence and immersion.