CostFormer:Cost Transformer for Cost Aggregation in Multi-view Stereo

1 Introduction

Given a series of calibrated images from different views in one scene, Multi-view Stereo (MVS) aims to recover the 3D information of the observed scene. It is a fundamental problem in computer vision and widely applied to robot navigation, autonomous driving, augmented reality, and etc. Recent learning-based MVS networks [Yao et al., 2018; Gu et al., 2020; Wang et al., 2021] have achieved inspiring success both in the quality and the efficiency of 3D reconstruction. Generally, deep MVS approaches consist of the following five steps: feature extraction from multi-view images via CNN network with shared weights, differentiable warping to align all source features to the reference view, matching cost computation from reference features and aligned source features, matching cost aggregation or regularization, depth or disparity regression.

† Corresponding authors.

‡ Appendix is presented in: https://arxiv.org/abs/2305.10320.

Current progresses in learning-based MVS primarily concentrate on the limitation of reconstruction quality [Wei et al., 2021; Yang et al., 2020a], memory consumption [Yan et al., 2020; Wei et al., 2021], and efficiency [Wang et al., 2021; Wang et al., 2022a]. The basic network architecture of these works is based on the pioneering backbone network called MVSNet [Yao et al., 2018], which provides an elegant and stable baseline. However, instead of taking the inheritance of network design principle in MVSNet [Yao et al., 2018] for granted, we can rethink the task of MVS problem as a dense correspondence problem [Hosni et al., 2012] alternatively. The core of MVS is a dense pixelwise correspondence estimation problem that searches the corresponding pixel of a specific pixel in the reference image along the epipolar line in all warped source images. No matter which task this correspondence estimation problem is applied to, the matching task can be boiled down to a classical matching pipeline [Scharstein and Szeliski, 2002]: (1) feature extraction, and (2) cost aggregation. In learning-basedMVSmethods, the transition from traditional hand-crafted features to CNN-based features inherently solves the former step of the classical matching pipeline via providing powerful feature representation learned from large-scale data. However, handling the cost aggregation step by matching similarities between features without any prior usually suffers from the challenges due to ambiguities generated by repetitive patterns or background clutters [Cho et al., 2021]. Consequently, a typical solution in MVSNet and its variants [Yao et al., 2018; Gu et al., 2020;

Wang et al., 2021] is to apply a 3D CNN or an RNN to regularize the cost volume among reference and source views, rather than directly rely on the quality of the initial correlation clues in cost volume. Although formulated variously in previous methods, these methods either use hand-crafted techniques that are agnostic to severe deformations or inherit the limitation of CNNs, e.g. limited receptive fields, unable to discriminate incorrect matches that are locally consistent.

In this work, we focus on the cost aggregation step of cost volume and propose a novel cost aggregation Transformer (CostFormer) to tackle the issues above. Our CostFormer is based on Transformer [Vaswani et al., 2017], which is renowned for its global receptive field and long-range dependent representation. By aggregating the matching cost in the cost volume, our aggregation network can explore global correspondences and refine the ambiguous matching points effectively with the help of the self-attention (SA) mechanism in Transformer. Though the promising performances of Vision Transformers have been proven in many applications [Dosovitskiy et al., 2020; Sun et al., 2021], the time and memory complexity of the key-query dot product interaction in conventional SA grow quadratically with the spatial resolution of inputs. Hence, replacing 3D CNN with Transformer may result in unexpected extra occupancy in memory and latency in inference. Inspired by [Wang et al., 2021], we further introduce the Transformer architecture into an iterative multi-scale learnable PatchMatch pipeline. It inherits the advantages of the long-range receptive field in Transformers, improving the reconstruction performance substantially. Meantime, it also maintains a balanced trade-off between efficiency and performance, which is competitive in the inference speed and parameters magnitude compared with other methods.

Our main contributions are as follows:

(1) In this paper, we propose a novel Transformer-based cost aggregation network called CostFormer, which can be plugged into learning-based MVS methods to improve cost volume effectively. (2) CostFormer applies an efficient Residual Depth-Aware Cost Transformer to cost volume, extending 2D spatial attention to 3D depth and spatial attention. (3) CostFormer applies an efficient Residual Regression Transformer between cost aggregation and depth regression, keeping spatial attention. (4) The proposed CostFormer brings benefits to learning-based MVS methods when evaluating DTU [Aanæs et al., 2016], Tanks&Temples [Knapitsch et al., 2017] ETH3D [Sch¨ops et al., 2017], BlendedMVS [Yao et al., 2020] and YFCC[Thomee et al., 2016] datasets.

2 RelatedWork

2.1 Learning-basedMVSMethods

Powered by the great success of deep learning-based techniques, many learning-based methods have been proposed to boost the performance of Multi-view Stereo. MVSNet [Yao et al., 2018] is a landmark for the end-to-end network that infers the depth map on each reference view for the MVS task. Feature maps extracted by a 2D CNN on each view are reprojected to the same reference view to build a variance-based cost volume. A 3D CNN is further used to regress the depth map. Following this pioneering work, lots of efforts have been devoted to boosting speed and reducing memory occupation. To relieve the burden of huge memory cost, recurrent neural networks are utilized to regularize the cost volume in AA-RMVSNet [Wei et al., 2021]. Following a coarse-to-fine manner to develop a computationally efficient network, a recent strand of works divide the single cost volume into several cost volumes at multiple stages, like CasMVSNet [Gu et al., 2020], CVP-MVSNet [Yang et al., 2020a], UCSNet [Cheng et al., 2020], and etc. Inspired by the traditional PatchMatch stereo algorithm, PatchMatchNet [Wang et al., 2021] inherits the pipeline in PatchMatch stereo in an iterative manner and extend it into a learning-based end-to-end network.

2.2 Vision Transformer

The success of Transformer [Vaswani et al., 2017] and its variants [Dosovitskiy et al., 2020; Liu et al., 2021] have motivated the development of Neural Language Processing in recent years. Borrowing inspiration from these works, Transformer has been successfully extended to vision tasks and proven to boost the performance of image classification [Dosovitskiy et al., 2020]. Following the pioneering work, many efforts are devoted to boosting the development of various vision tasks with the powerful representation ability of Transformer.

In [Li et al., 2021], the application of Transformer in the classic stereo disparity estimation task is investigated thoughtfully. Swin Transformer [Liu et al., 2021] involves the hierarchical structure into Vision Transformers and computes the representation with shifted windows. Considering Transformer's superiority in extracting global content information via attention mechanism, many works attempt to utilize it in the task of feature matching. Given a pair of images, CATs [Cho et al., 2021] explore global consensus among correlation maps extracted from a Transformer, which can fully leverage the self-attention mechanism and model long-range dependencies among pixels. LoFTR [Sun et al., 2021] also leverages Transformers with a coarse-to-fine manner to model dense correspondence. STTR [Li et al., 2021] extends the feature matching Transformer architecture to the task of stereo depth estimation task in a sequence-to-sequence matching perspective. TransMVSNet [Ding et al., 2021] is the most relevant concurrent work compared with ours, which utilizes a Feature Matching Transformer (FMT) to leverage self-attention and cross-attention to aggregate long-range context information within and across images. Specifically, the focus of TransMVSNet is on the enhancement of feature extraction before cost aggregation, while our proposed CostFormer aims to improve the cost aggregation process on cost volume.

3 Methodology

In this section, we introduce the detailed architecture of the proposed CostFormer which focuses on the cost aggregation step of cost volume. CostFormer contains two specially designed modules called Residual-Depth Aware Cost Transformer (RDACT) and Residual Regression Transformer (RRT), which are utilized to explore the relation between pixels within a long range and the relation between different depth hypotheses during the evaluation process. In Section Preliminary, we give a brief preliminary on the pipeline of our method. Then we show the construction of RDACT and RRT respectively. Finally, we show experiments.

3.1 Preliminary

In general, the proposed RDACT and RRT can be integrated with arbitrary cost volume of learning-based MVS networks. Based on the patch match architecture [Wang et al., 2021], we further explore the issue of cost aggregation on cost volume. As shown in Figure 2, CostFormer based on PatchMatchNet [Wang et al., 2021] extracts feature maps from multi-view images and performs initialization and propagation to warp the features maps in source views to reference view. Given a pixel p at the reference view and its corresponding pixel pi,j at the i-th source view under the j-th depth hypothesis dj is

where R0,i and t0,i denote the rotation and translation between the reference view and i-th source view. K0 and Ki are the intrinsic matrices of the reference and i-th source view. The warped feature maps at the i-th source view Fi(pi,j) are bilinearly interpolated to remain the original resolution. Then, a cost volume is constructed from the similarity of feature maps, and 3D CNNs are applied to regularize the cost volume. Warped features from all source views are integrated into a single cost for each pixel p and depth hypothesis dj by computing the cost per hypothesis Si(p, j)g via group-wise correction as follows:

where G is the group number, C is the channel number, < ·, · > is the inner product, F0(p)g and Fi(pi,j)g are grouped reference feature map and grouped source feature map at the i-th view respectively. Then they aggregate over the views with a pixel-wise view weight wi(p) to get S(p, j).

Taking no account of Transformer at the cost aggregation (CA) step, a CA module firstly utilizes a small network with 3D convolution with 1×1×1 kernels to obtain a single cost, C ∈ RH×W×D . For a spatial window of Ke pixels {pk}Ke k=1 can be organized as a grid, per pixel additional offsets {∆pk}Ke k=1 can be learned for spatial adaptation. The aggregated spatial cost ˜ C(p, j) is defined as:

where wk and dk weight the cost C based on feature and depth similarity. Given the sampling positions (p + pk + ∆pk)Ke k=1 , corresponding features from F0 are extracted via bilinear interpolation. Then group-wise correlation is applied between the features at each sampling location and p. The results are concatenated into a volume on which 3D convolution layers with 1×1×1 kernels and sigmoid non-linearities are applied to output normalized weights {wk}Ke k=1 . The absolute difference in inverse depth between each sampling point and pixel p with their j-th hypotheses are collected. Then a sigmoid function on the inverted differences is applied to obtain {dk}Ke k=1 .

The remarkable thing is that such cost aggregation inevitably suffers from challenges due to ambiguities generated by repetitive patterns or background clutters. The local mechanisms in ambiguities exist in many operations, such as local propagation and spatial adaptation by small learnable slight offset. CostFormer significantly alleviates these problems through RDACT and RRT. The original CA module is also repositioned between RDACT and RRT.

After RRT, soft argmin is applied to get the regressed depth. Finally, a depth refinement module is designed to refine the depth regression.

For CascadeMVS and other cascade architectures, CostFormer can be plugged into similarly.

3.2 Residual Depth-Aware Cost Transformer

In this section, we explore the details of the Residual DepthAware Cost Transformer (RDACT). Each RDACT consists of two parts. The first part is a stack of Depth-Aware Transformer layer (DATL) and Depth-Aware Shifted Transformer layer (DASTL), which deal with the cost volumes to explore the relations sufficiently. The second part is the ReEmbedding Cost layer (REC) which recovers the cost volume from the first part.

Given a cost volume C0 ∈ RH×W×D×G, temporary intermediate cost volumes C1,C2,...,CL ∈ RH×W×D×E are firstly extracted by DATL and DASTL alternatively:

where DATLk is the k-th Depth-Aware Transformer layer with regular windows, DASTLk is the k-th Depth-Aware Transformer layer with shifted windows, E is the embedding dimension number of DATLk and DASTLk.

Then a Re-Embedding Cost layer is applied to the last Ck, namely CL, to recover G from E. The output of RDACT is formulated as:

where REC is the Re-Embedding Cost layer, and it can be a 3D convolution with G output channels. If E = G, Cout can be simply formulated as:

This residual connection allows the aggregation of different levels of cost volumes; Cout instead of C0 is then aggregated by the original aggregation network described in section 3.1. The whole RDACT is shown in the red window in Figure 2.

Before introducing the construction of DATL and DASTL, we dive into the details of core constitutions called DepthAware Multi-Head Self-Attention (DA-MSA) and DepthAware Shifted Multi-Head Self-Attention (DAS-MSA). Both DA-MSA and DAS-MSA are based on Depth-Aware SelfAttention Mechanism. In order to explain Depth-Aware Self-Attention Mechanism, we supply the knowledge about Depth-Aware Patch Embedding and Depth-Aware Windows as preliminary.

Depth-Aware Patch Embedding

Obviously, directly applying the attention mechanism for feature maps at pixel-wise level is quite costly in terms of GPU memory usage. In order to tackle this issue, we propose a Depth-Aware Patch Embedding to reduce the high memory cost and get an additional regularization. Specifically, given a grouped cost volume before aggregation C ∈ RH×W×D×G, a depth-aware patch embedding is firstly applied to C to get tokens. It consists of a 3D convolution with kernel size h×w×d and a layer normalization.To downsample the spatial sizes of cost volume and keep the depth hypotheses, we set h and w to more than 1 and d as 1. So the sample ratio is adaptive for memory cost and run time. Before convolution, cost volume will be padded to fit the spatial sizes and downsampling ratio. After layer normalization(LN), these embedded patches are further partitioned by depth-aware windows.

Depth-Aware Windows

Beyond the nonlinear and linear global self-attention, local self-attention within a window has been proven to be more effective and efficient. As an example of 2D windows, Swin Transformer [Liu et al., 2021] directly applies multi-head self-attention mechanisms on non-overlapping 2D windows to avoid the big computation complexity of global tokens.

Extended from the 2D spatial window, an embedded cost volume patch ∈ RH∗×W ∗×D∗×G with depth information is partitioned into non-overlapping 3D windows. These local windows are then transposed and reshaped to local cost tokens. Assuming the sizes of these windows are hs × ws × ds, the total number of tokens is ⌈ H∗ hs ⌉×⌈ W ∗ ws ⌉×⌈ D∗ ds ⌉. These local tokens are further processed by the multi-head self-attention mechanism.

Depth-Aware Self-Attention Mechanism

For a cost window token X ∈ Rhs×ws×ds×G, the query, key, and value matrices Q, K and V ∈ Rhs×ws×ds×G are computed as:

where PQ, PK, and PV ∈ RG×G are projection matrices shared across different windows. By introducing depth and spatial aware relative position bias B1 ∈ R(hs×hs)×(ws×ws)×(ds×ds) for each head, the depth-aware self-attention(DA-SA1) matrix within a 3D local window is thus computed as:

Where Q1, K1 and V 1 ∈ Rhswsds×G are reshaped from Q, K and V ∈ Rhs×ws×ds×G. The process of DATL with LayerNorm(LN) and multi-head DA-SA1 at the current level is formulated as:

By introducing depth-aware relative position bias B2 ∈ Rds×ds for each head, the depth-aware self-attention(DASA2) matrix along the depth dimension is an alternative module to DATL and thus computed as:

Where Q2, K2 and V 2 ∈ Rhsws×ds×G are reshaped from Q, K and V ∈ Rhs×ws×ds×G. B1 and B2 will be along the depth dimension and lie in the range of [-ds + 1, ds - 1]. Along the height and width dimension, B1 lies in the range of [-hs + 1, hs - 1] and [-ws + 1,ws - 1]. In practice, we parameterize a smaller-sized bias matrix B1 ∈ R(2hs-1)×(2ws-1)×(2ds-1) from B1 and perform the attention functionfor f times in parallel, and then concatenate the depth-aware multi-head self-attention (DA-MSA) outputs. The process of DATL with LayerNorm(LN), multi-head DA-SA1, and DA-SA2 at the current level is formulated as:

Then, an MLP module that has two fully-connected layers with GELU non-linearity between them is used for further feature transformations:

Compared with global attention, local attention makes it possible for computation in high resolution.

However, there is no connection across local windows with fixed partitions. Therefore, regular and shifted window partitions are used alternately to enable cross-window connections. So at the next level, the window partition configuration is shifted along the height, width, and depth axes by ( hs 2 , ws 2 , ds 2 ). Depth-aware self-attention will be computed in these shifted windows(DAS-MSA); the whole process of DASTL can be formulated as:

DAS-MSA1 and DAS-MSA2 correspond to multi-head Attention1 and Attention2 within a shifted window, respectively. Assuming the number of stages is n, there are n RDACT blocks in CostFormer.

3.3 Residual Regression Transformer

After aggregation, the cost ˜ C ∈ RHXWXD will be used for depth regression. To further explore the spatial relation under some depth, a Transformer block is applied to ˜ C before softmax. Inspired by the RDACT, the whole process of Residual Regression Transformer(RRT) can be formulated as:

where RTk is the k-th Regression Transformer layer with regular windows, RSTk is the k-th Regression Transformer layer with shifted windows, RER is the re-embedding layer to recover the depth dimension from ˜ CL, and it can be a 2D convolution with D output channels.

RRT also computes self-attention in a local window. Compared with RDACT, RRT focuses more on spatial relations. Compared with regular Swin [Liu et al., 2021] Transformer block, RRT treats the depth as a channel, the number of channels is actually 1 and this channel is squeezed before the Transformer. The embedding parameters are set to fit the cost aggregation of different iterations. If the embedding dimension number equals D, ˜ Cout can be simply formulated as:

As a stage may iterate many times with different depth hypotheses, the number of RRT blocks should be set the same as the number of iterations. The whole RRT is shown in the yellow window in Figure 2.

3.4 Training

Loss Function

Final loss combines with the losses of all iterations at all stages and the loss from the final refinement module:

where Lki is the regression or unification loss of the i-th iteration at k-th stage. Lref is the regression or unification loss from refinement module. If refinement module does not exist, the Lref loss is set to zero.

Common Training Settings

CostFormer is implemented by Pytorch [Paszke et al., 2019]. For RDACT, we set the depth number at stages 3, 2, 1 as 4, 2, 2; patch size at height, width and depth axes as 4, 4, 1; window size at height, width and depth axes as 7, 7, 2. If the backbone is set as PatchMatchNet, embedding dimension number at stages 3, 2, 1 are set as 8, 8, 4. For RRT, we set the depth number as 2 at all stages, patch size as 1 at all axes; window size as 8 at all axes. If the backbone is set as PatchMatchNet, embedding dimension number at iteration 2, 2, 1

at stages 3, 2, 1 as 32, 64, 16, 16, 8. All models are trained on Nvidia GTX V100 GPUs. After depth estimation, we reconstruct point clouds similar to MVSNet [Yao et al., 2018].

4 Experiments

4.1 ComparedMethods

In the experiments, we compare our proposed method with following state-of-the-art MVS methods, including: 1) Traditional MVS methods, like Furu [Furukawa and Ponce, 2010], Tola [Tola et al., 2012], Gipuma [Galliani et al., 2015], Colmap [Sch¨onberger and Frahm, 2016], MVE [Fuhrmann et al., 2014], PMVS [Furukawa and Ponce, 2010]; 2) Learningbased MVS methods, like SurfaceNet [Ji et al., 2017], MVSNet [Yao et al., 2018], R-MVSNet [Yao et al., 2019], PMVSNet [Luo et al., 2019], Point-MVSNet [Chen et al., 2019], Fast-MVSNet [Yu and Gao, 2020], CasMVSNet [Gu et al., 2020], UCS-Net [Cheng et al., 2020], CVP-MVSNet [Yang et al., 2020b], PVA-MVSNet [Yi et al., 2020], PatchmatchNet [Wang et al., 2021], AA-RMVSNet [Wei et al., 2021], UniMVSNet [Peng et al., 2022], PVSNet [Xu and Tao, 2020], IterMVS [Wang et al., 2022a], PatchMatch-RL [Lee et al., 2021], MVSNet-s [Darmon et al., 2021] , VisMVSNet [Zhang et al., 2020], and EPPMVSNet [Ma et al., 2021]. Furthermore, we also conduct the comparison with Transformer based MVS networks: MVSTR [Zhu et al., 2021], TransMVSNet [Ding et al., 2022], MVSTER [Wang et al., 2022b], and WT-MVSNet [Liao et al., 2022].

4.2 Main Settings

Our proposed CostFormer can utilize arbitrary MVS network as backbone. We name the framework as CostFormer-PM, CostFormer-Cas, and CostFormer-Uni, when PatchMatchNet, CasMVSNet, and UniMVSNet are respectively used as backbone.

4.3 Datasets

The datasets used in the evaluation are DTU [Aanæs et al., 2016], BlendedMVS [Yao et al., 2020], ETH3D [Sch¨ops et al., 2017], Tanks&Temples [Knapitsch et al., 2017], and YFCC-100M [Thomee et al., 2016]. The DTU dataset is an indoor multi-view stereo dataset with 124 different scenes, there are 49 views under seven different lighting conditions in one scene. Tanks&Temples is collected in a more complex and realistic environment, and it's divided into the intermediate and advanced set. ETH3D benchmark consists of calibrated high-resolution images of scenes with strong viewpoint variations. It is divided into training and test datasets. While the training dataset contains 13 scenes, the test dataset contains 12 scenes. BlendedMVS dataset is a large-scale synthetic dataset, consisting of 113 indoor and outdoor scenes and split into 106 training scenes and 7 validation scenes.

4.4 Results on Tanks&Temples Benchmark

For the evaluation on Tanks&Temples, we use the DTU dataset and the Blended MVS dataset. We compare our method to those recent learning-based MVS methods, including PatchMatchNet and UniMVSNet which are also set as backbones of CostFormer. The quantitative results on the Tanks&Temples set are summarized in Table 1 and 2, which indicate the robustness of CostFormer. Note that the UniMVSNet- in the table only uses BlendedMVS for training which uses less data (no DTU) than the UniMVSNet baseline. The qualitative results are shown in Figure 4.

4.5 Results on DTU Benchmark

Comparisons with Other Methods

For the evaluation on the DTU evaluation set, we only use the DTU training set. During the training phase, we set the image resolution to 640 × 512. We compare our method to recent learning-based MVS methods, including CasMVSNet and PatchMatchNet which are also set as backbones of CostFormer. We follow the evaluation metrics provided by the DTU dataset. The quantitative results on the DTU evaluation set are summarized in Table 3, which indicates that the plugand-play CostFormer improves the cost aggregation. Partial visualization results of Table 3 are shown in Figure 3.

Complexity Analysis

For the complexity analysis of CostFormer, we plug it into PatchMatchNet and first compare the memory consumption and run-time with this backbone. For a fair comparison, a fixed input size of 1152 × 864 is used to evaluate the computational cost on a single GPU of NVIDIA Telsa V100. Memory consumption and run-time of PatchMatchNet are 2323MB and 0.169s. They are only increased to 2693MB and 0.231s by the plug-in.

Based on the reports of PatchMatchNet, we then get the comparison results of other state-of-the-art learning-based methods. Memory consumption and run-time are reduced by 61.9% and 54.8% compared to CasMVSNet, by 48.8% and 50.7% compared to UCSNet and by 63.5% and 77.3% compared to CVP-MVSNet. Combining the results(lower is better) are shown in Table 4 and Figure 1, GPU memory and run-time of CostFormer are set as 100%.

Comparison with Other Transformer-based Methods

We also compare CostFormer with other Transformers-based networks which are used in MVS methods and not plug-andplay. For a fair comparison, only direct improvements(higer is better) and incremental cost of run time(low is better) from pure Transformers under similar depth hypotheses are summarized in Table 5.

4.6 Results on ETH3D Benchmark

We use the PatchMatchNet as backbone and adopt the trained model used in the Tanks&Temples dataset to evaluate the ETH3D dataset. As shown in Table 6, our method outperforms others on both the training and particularly challenging test datasets(higher is better).

4.7 Results on BlendedMVS Benchmark

We use the model used in ETH3D. On BlendedMVS evaluation set, we set N = 5 and image resolution as 576 × 768. End point error (EPE), 1 pixel error (e1), and 3 pexels error (e3) are used as the evaluation metrics. Quantitative results(lower is better) of different methods are shown in Table 7.

4.8 Results on YFCCDataset

Following the depth evaluation protocal in previous work, we utilize the YFCC-100M [Thomee et al., 2016] to test the performance. YFCC100M is the largest public multimedia collection that has ever been released. The dataset contains a total of 100 million media objects, of which approximately 99.2 million are photos and 0.8 million are videos. We set backbone as PatchMatchNet, N = 5, and image resolution as 512x512. End point error (EPE), 1 pixel error (e1), and 3 pixels error (e3) are also used as the evaluation metrics. Quantitative results of different methods are shown in Tabisle 8.

5 Conclusion

In this work, we explore whether cost Transformer can improve the cost aggregation and propose a novel CostFormer with the cascade RDACT and RRT modules. The experimental results on DTU , Tanks&Temples, ETH3D, BlendedMVS, and YFCC show that our method is competitive, efficient, and plug-and-play. Cost Transformer can be your need for better cost aggregation in multi-view stereo.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No.61976095).

Contribution Statement

Weitao Chen and Hongbin Xu contribute equally to this work.