Spatially Covariant Lesion Segmentation

1 Introduction

Convolutional neural networks (CNNs) have been the dominant approach across a variety of medical image computing applications such as magnetic resonance imaging (MRI) reconstruction [Zhang et al., 2021b; Zhang et al., 2020c], brain lesion segmentation [Zhang et al., 2021a; Zhang et al., 2019a], disease identification [Khosla et al., 2019; Zhang et al., 2022], and low-dose CT denoising [Wang et al., 2022; Fan et al., 2019]. Two main properties [Elsayed et al., 2020] of CNNs are believed to be the key to their success: 1) The local receptive field increases as the network goes deeper [Huang et al., 2017; He et al., 2016], providing richer contextual information for down-stream tasks; 2) Spatially invariant convolution filters regularize the network training, serving as a good inductive bias for grid-based image data [Simoncelli and Olshausen, 2001]. However, medical images scanned from human body exhibit stronger patterns than natural images and therefore the question of whether the spatial invariance is overly restrictive in certain clinical applications remains to be answered.

Unlike natural images, many medical images, obtained through non-invasive techniques like MRI, display structured patterns due to their commonalities across the population. Despite variations in size, disease type, or other attributes, brains illustrate similar tissue structures, like ventricles, white matter, and cortex location/shape (see Fig. 1). However, existing lesion segmentation studies often use CNN architectures, intended for natural image segmentation, leading to loss of structural information. Based on three observations on brain lesion segmentation, we propose that enforcing spatial invariance in all CNN layers is excessively restrictive for tasks with distinct tissue structures. Relaxing this invariance may enhance computational efficiency and maintain or boost segmentation accuracy, fitting resource-limited clinical needs.

Firstly, CNN-based methods are more likely to missegment brain lesions near cortices or ventricles compared to those within normal-appearing white matter (NAWM) [Moll et al., 2011]. This inconsistency, visible in Fig. 1, could be due to varying image contrast and tissue structures, questioning the strict use of spatially invariant convolution filters. Secondly, despite individual variation, lesion density distribution in a patient cohort is structured, with higher density around the peri-ventricular area, as shown in Fig. 2. Lastly, a pixel's

0

Figure 2: A visual example of lesion distribution in human brain. Images and binary lesion masks from a private dataset with 176 subjects are linearly registered to a standard MNI space using FSL neuroimaging toolbox [Fischl, 2012]. All registered lesion masks were then summed up to form a unified heat map. The heat map was projected into orthogonal planes as above using Nilearn library [Abraham et al., 2014].

effective receptive field varies based on its location. Pixels nearer the brain center have larger receptive fields than those on the periphery, mainly due to the brain boundary introducing more zero values in the latter.

In general, better performance can be achieved by enlarging the network size to accommodate changes of the tissue context, lesion density distribution, and receptive field; however, simple enlargement of the network without considering domain-specific information may suffer from a higher computational cost or even overfitting due to the limited amount of medical data. To test our hypothesize, a viable way is replacing the sharing filters with spatially covariant filters. However, this is not applicable as it adds up a huge amount of computational resources, deviating from our original intention. Therefore in this paper, we propose spatially covariant pixel-aligned classifier (SCP) to improve the computational efficiency and meantime maintain or increase accuracy for lesion segmentation. SCP relaxes the spatial invariance constraint imposed by convolutional operations and optimizes an underlying implicit function that maps image coordinates to network weights, the parameters of which are obtained along with the backbone network training and later used for generating network weights to capture spatially variant contextual information. SCP is intriguingly simple and can be plugged into most existing segmentation networks.

We apply SCP to two lesion segmentation tasks, white matter hyperintensity (WMH) segmentation in magnetic resonance imaging (MRI) and liver tumor (LiT) segmentation in contrast-enhanced abdominal computerized tomography (CT), to verify our hypothesis. The main findings of the paper are in three folds: 1) Relaxing spatial variance for tasks with datasets having structured representation or spatially inhomogeneous features are beneficial. 2) SCP works better for WMH segmentation than LiT segmentation as brains are more structured than liver across different individuals. 3) The experimental results suggest that with SCP relaxing the spatial invariance to a certain degree, 23.8%, 64.9% and 74.7% reduction in GPU memory usage, FLOPs, and network parameter size can be achieved without compromising any segmentation accuracy.

2 RelatedWorks

2.1 Lesion Segmentation Networks

Lesion segmentation is one of the most important and difficult tasks in clinical applications, because lesions such as WMH and LiT vary greatly in terms of shape, size, and location. Numerous automated approaches have been proposed to handle the problem, and usually, methods with prior information injected perform better than those using plain neural networks. The Tiramisu network [Zhang et al., 2019b] uses 2.5D stacked DenseNet [Huang et al., 2017] to capture broader brain structural information, [Aslani et al., 2019] uses multi-branch residual network [He et al., 2016] to fuse multi-contrast information, and [Zhang et al., 2019a; Zhang et al., 2020b] applies regularized self-attention networks for aggregation of richer contextual information. nnUNet [Isensee et al., 2021] takes one step further to condense and automate the key decisions for designing a successful segmentation network. Despite their success, little domainspecific structure for the network has been developed by these methods, which may lead to sub-optimal results.

The distance transformation mapping has been integrated into neural networks through the development of edgeaware loss functions [Kervadec et al., 2019; Zhang et al., 2020a; Karimi and Salcudean, 2019] and network layers with anatomical coordinates as prior information [Zhang et al., 2021a], improving lesion segmentation performance. These edge-aware loss functions reduces the class imbalance between teh background and lesions, helping segmenting small lesions. Interestingly, All-Net [Zhang et al., 2021a] has outperformed its pioneers [Zhang et al., 2019b; Aslani et al., 2019; Isensee et al., 2021] and achieved the state-of-the-art performance on multiple sclerosis lesion segmentation by explicitly integrating domain-specific information into the neural network. All these methods have achieved a reasonably good result, and we make one step further to provide the possibility and elasticity to find a better trading-off between the segmentation accuracy and computation efficiency by relaxing spatial invariance.

2.2 Invariant Representations of CNNs

While few efforts have been made to study the spatial invariance of CNNs in the context of medical images, many researchers has explored this in natural images. One line of works seek to learn invariant [Yi et al., 2016; Cheng et al., 2016; Liu et al., 2019] or equivariant representations [Cohen and Welling, 2016; Esteves et al., 2018; Esteves et al., 2018; Tai et al., 2019] for CNNs. The performance gain from these methods is usually accompanied with linearly scaled computational cost. Another line of works exploits the hidden information behind the invariant representations learned from CNNs, and our SCP falls in this line. The most related work to our SCP is a low-rank locally connected (LRLC) layer [Elsayed et al., 2020] that can parametrically adapt the spatial invariance degree for each convolotion layer by controlling its rank parameter; however, it's non-trivial to adapt LRLC for lesion segmentation and there is no apparent way of finding a trading-off between accuracy and computation efficiency. Interestingly, [Kayhan and Gemert, 2020] empirically shows that CNNs consisting of spatially invariant filters can naturally encode absolute spatial location information, reducing the translation invariance in CNNs, and a removal of such encoding can impose stronger inductive bias on the translation invariance, benefiting image classification tasks with small datasets. Further, [Islam et al., 2020] analyzes howmuch spatial information CNNs encode and how CNNs encode, particularly the authors show that the zero-padding near the image border is the key to deliver the spatial location information; however, this implicit spatial information comes at the price of using deeper CNNs or larger convolution filters.

[Kayhan and Gemert, 2020] and [Islam et al., 2020] show that some part of the CNN capacity is occupied by implicit spatial location encoding, meaning that it is possible to preserve this part of capacity for feature learning by feeding directly the spatial location information such as coordinates [Liu et al., 2018] to the network. In addition, considering the hierachical structure, boundary shape and spatially inhomogeneous contrast features of human brain, our proposed SCP can further exploit these information and reduce computational complexity.

3 Methodology

3.1 Preliminaries

Let X ∈ RC×H×W be an input feature map with C channels (H and W denote the input size, and C is the number of input channels), Xij be the feature vector at location (i, j), and Y ∈ RN×W×H (N is the number of segmentation categories) be the output of the segmentation head of the network, we can then describe the output logit at position (i, j) as follows:

where w is the trainable weight of the filters for the segmentation head and f transforms the feature vector in every location in an identical way using w. The spatial invariance refers to sharing the same set of filters across all spatial locations. Usually a practical implementation of Eq. (1) for an encoderdecoder network is a 1 × 1 convolution layer, which can be written as follows:

where Yij ∈ RN×1×1 , w ∈ RC×N and Xij ∈ RC×1×1 .

3.2 Spatially Covariant Segmentation Head

Let S ∈ RM×H×W (M is the number of elements for the encoded position vector) be the position encoding tensor, to relax the spatial invariance, we can replace the shared w with a function of location:Wij = Φ(Sij ; θ), (3)

where Wij is the generated spatially covariant weight, θ is the trainable parameter for function Φ. Basically, function Φ takes in an encoded position vector Sij and uses its parameter determined by θ to generate weights for convolution filters. With Eq. (3), we can derive the spatially covariant segmentation head as follows:

Eq. (4) can be seen as a generalization of Eq. (1), as the former reduces to the latter when g equals f and Φ(Sij; θ) outputs the θ ∈ RC×N regardless of the location information. Thus, by optimizing the function Φ to generate spatially covariant weights across all locations, the spatial invariance is relaxed.

3.3 Position Encoding

Let X ∈ RC×H×W be an output feature map with C channels from a encoder-decoder network, we encode the position at (i, j) of X as follows:

For the rest of the paper, we will refer to Sij instead of (i, j) as the pixel location. The Fourier representation [Rahaman et al., 2019] of the Cartesian coordinates as adopted by neural scene representation [Mildenhall et al., 2020] or Transformer architecture [Vaswani et al., 2017] is not adopted, because in our application, the surface of generated filter weights is smooth among adjacent pixels, containing no highfrequency information, and enforcing high-frequency variation introduces noise and degrades performance. Also, please note that we use four distances (i.e. the distance to the left, right, top, and bottom) to represent a pixel coordinate, because the input size (H,W) of the image may vary among different individuals. Compared to encoding position vector as Sij = (i, j,H,W), our four-distance encoding method helps expand a smoother surface for the implicit function Φ.

3.4 Modeling SCP with Neural Networks

It has been shown that the effective receptive field of a CNN has a Gaussian distribution [Luo et al., 2016]. Without loss of generality, we hypothesize that feature vector at each position Sij follows a multi-variate Gaussian distribution (the dimension is the number of channels C) as follows:

where µij and Σij are the corresponding mean vector and covariance matrix in location Sij . While the normal CNN assumes that µij and Σij stay the same across all spatial locations, we relax this restriction by optimizing an implicit function that maps the pixel location Sij to the parameter space of µ and Σ as follows:

where Φ represents the implicit mapping function which outputs a continuous parameter space of µ and Σ to generate spatial covariant filter weights. In practice, we use 1 × 1 convolution layers to approximate this continuous parameter space and optimize the weight θ of Φ along with other network weights to map each location Sij to the corresponding mean vector and covariance matrix.

3.5 Derivation of SCP

It has been shown that deep CNNs can linearize [Bengio et al., 2013; Upchurch et al., 2017] the manifold of image features into a globally Euclidean subspace of deep features, meaning that the elements of the deep feature vector have minimal interactions with each other. As the input feature map for SCP has passed through an entire encoder-decoder network, the output of the implicit function can be further simplified. More specifically, the covariance matrix Σij can be reduced to a variance vector σij . We can can then normalize and output the logit of the feature vector at position Sij as follows:

Where µij ∈ RC×1×1 and σij ∈ RC×1×1 are estimated from the implicit function Φ with θ as the trainable parameter, Yij is the output logit at position Sij, and w is the weight of the final 1×1 convolution classifier. Note that as we use Z-score to normalize the input image, there is no bias term for the classifier. To be numerically stable, in practice, we estimate Pij = 1/σij and Qij = -µij/σij instead, and the equation for normalization and classification of the feature vectorcan be altered as:

where ◦ is the Hadamard product. We have derived the spatially covariant segmentation head using the implicit function Φ in Eq. (10). However, in practice, using Eq. (10) directly may generate unnecessary GPU memory usage due to redundant intermediate computation. Looking closer into Eq. (10), we can reformulate it as:

where the first termw◦pij in Eq. (11) can be replaced with a singleWij generated by the implicit function Φ directly. The second term in Eq. (13) becomes a scalar, serving as a pixelwise bias term, but as aforementioned, the bias term is not necessary. Upon removing the bias term, we add one residual term to stabilize the network training, and our SCP is derived as follows:

whereWij ∈ RN×C is estimated from the implicit function Φ with θ as the trainable parameter, wr ∈ RN×C denotes the weight of the residual 1×1 convolution, serving as the shared term for pixels across all spatial locations.

3.6 Overall Framework Implementation with SCP

The overall framework consists of an encoder-decoder structured feature extraction network and the proposed SCP module for pixel-wise classification. As shown in Fig. 3, the input image is passed through an encoder-decoder CNN to obtain the intermediate feature map X ∈ RC×H×W , and in the meantime, pixel-wise position encoding is obtained by concatenating matrices with distances to the left, right, top and bottom of the images. The spatially covariant weight tensor W ∈ RN×C×H×W is then generated by the implicit function ϕ consisting of two layers of a 1×1 convolution followed by a ReLU activation function and a single 1 × 1 convolution. With Wij for each pixel position, we apply Eq. (13) to output the spatial variant logits for the feature vector of each position. The final segmentation can be obtained by the sum of the output from the residual term and the output from the implicit function, and a Sigmoid function.

4 Experiments and Results

4.1 Comparators and EvaluationMetrics

We use PyTorch [Paszke et al., 2019] for all network implementations, and train them with an Nvidia RTX 2080Ti GPU. We carefully carry out a set of experiments to show both strengths and useful characteristics of the proposed SCP. We use MRI and CT datasets, which are the most commonly used modalities for medical imaging, to show that SCP generalizes across different imaging techniques. Meanwhile, brain lesions and liver tumors, two commonly seen diseases, are used to demonstrate SCP's generalization across different human organs.

Baselines and Comparators

All-Net [Zhang et al., 2021a] is a recently developed network for brain lesion segmentation, outperforming other highly competitive networks such as nnU-Net [Isensee et al., 2021], Tiramisu network [Zhang et al., 2019b] and multi-branch residual network [Aslani et al., 2019] on the lesion segmentation task. Without loss of generality, we adopt the backbone network from All-Net as baseline network.

To illustrate the effectiveness of SCP utilizing spatially variant information rather than taking additional coordinates, two network variants with coordinate convolution [Liu et al.,

2018] are further compared: 1) baseline network with coordinates as an additional input channel (denote as Base F); 2) baseline network with coordinate convolution replacing the final convolution layer (denote as Base B). We also deploy three variants with different model capacity by varing the number of channels Nc in the encoder-decoder network (see Fig. 4 for more details about how Nc changes the network capacity), in which Nc denotes the maximum number of channels in the center layer of the network (e.g. the Base B network with Nc = 256 is denoted as Base B (256)).

Evaluation Metrics

To quantify the models' performance, we use the Dice similarity coefficient [Dice, 1945], lesion-wise Dice (L-Dice), lesion-wise true positive rate (L-TPR), lesion-wise positive predictive value (L-PPV), and lesion-wise F1 score (L-F1) as metrics. L-Dice, LTPR, and LPPV are defined as L-Dice = TPR GL+PL , L-TPR = TPR GL , and L-PPV = TPR PL , where TPR denotes the number of lesions in ground-truth segmentation that overlap with a lesion in the produced segmentation, and GL, PL are the numbers of lesions in ground-truth segmentation and produced segmentation, respectively. L-F1 is the harmonic mean of L-TPR and L-PPV: L-F1 = 2L-TPR×L-PPV L-TPR+L-PPV . In addition, we use the two-tailed paired t-test to compare Dice and L-Dice metrics for the proposed SCP with other network variants to demonstrate statistical significance for the improved computational efficiency.

To assess computational complexity, we have chosen the base model with Nc = 128 as our reference model because it demonstrates the lowest GPU memory usage, FLOPs, and network size among models using the same input image. This model is used to compute the ratios of Floating Point Operations Per Second (FLOPs), memory usage, and network parameter size relative to each network variant. Note that while the parameter size remains consistent across both tasks, FLOPs and memory usage vary with the input image size. Therefore, we utilize a fixed input size of 3 × 160 × 224 to display these ratios in Table 1.

4.2 White Matter Hyperintensity Segmentation

Our first experiment is WMH segmentation in human brain. WMH is an important type of brain lesion whose precise tracing can provide useful biomarkers for clinical diagnosis and assessment of disease progression. We use a publicly available 1 WMH segmentation challenge dataset [Kuijf et al., 2019] for our experiments. The dataset contains 60 subjects acquired from different scanners of three institutes. For each subject, a 3D T1-weighted (T1-w) image and a 2D multi-slice fluid-attenuated inversion recovery (FLAIR) image are supplied. The manual reference segmentation is accomplished based on the consensus of four observers. In the experiments, the dataset is randomly split into three subsets for model training (42), validation (6), and testing (12). The final results of each model are the average of three independent runs with different random seeds (random seed uses the system time and is not sensitive).

Implementation Details

We slice the original volumes into a stack of consecutive images. Each volume is center-cropped to a fixed size of 160 × 224 to accommodate batch training, followed by Zscore based intensity normalization. T1-w and FLAIR images of each subject are concatenated through the channel dimension before being used as the input to the network. To train the network, we use Adam [Kingma and Ba, 2014] optimizer, with an initial learning rate of 10-3 (weight decay of 10-6 ), and a batch size of 14. The learning rate is halved at 50%, 70%, and 90% of the total training epoch (90) for optimal convergence.

Quantitative Results

We compare the proposed SCP with the baseline network [Zhang et al., 2021a] and other variants. Three different levels of network capacities (Nc = 128 for low-capacity, Nc = 256 for moderate-capacity, and Nc = 512 for highcapacity) of the backbone networks are adopted. Pixel-wise and lesion-wise metric scores as well as ratios of floating point operations per second (FLOPs), memory usage and net-

1 https://wmh.isi.uu.nl

(a) WMH on Dice.

SCP (256)

Base (256)

Base (512)

Base F (512)

Base B (512)

(b) WMH on L-Dice.

(c) LiTS on Dice.

(d) LiTS on L-Dice.

Figure 5: Performance comparison of different network architectures with Dice and L-Dice evaluated on the testing set of WMH and LiTS datasets. The number in the bracket indicates the number of channels Nc for the network. Statistical significance test between of the proposed SCP and other comparators are evaluated using a paired t-test. The threshold of the significance is α= 0.05, and the p-values in the figure are annotated as: * for p < 0.05, ** for p < 0.01, *** for p < 0.001, **** for p < 0.0001, and ns for non-significant.

work parameter size between each network variant and the Base (128) network are shown in Table 1.

In the low-capacity group (Nc = 128), it is observed that SCP (128) outperforms Base (128), Base F (128), and Base B (128) in all metrics except for L-PPV. Though Base B (128) has the highest L-PPV (90.95%), the L-TPR (24.21%) is much lower than SCP (128), resulting in worse L-Dice. Importantly, SCP (128) has at least 7.1% and 9.8% improvement in Dice and L-Dice compared to other network variants with the same compacity. Similar trends in performance are observed among the moderate- and high-capacity groups (Nc = 256 or Nc = 512), where SCP shows a consistent improvement compared to those networks without the proposed method. Interestingly, both Base F and Base B which take additional coordinate information, perform consistently worse than SCP in all three levels of model capacity, demonstrating the effectiveness of SCP utilizing spatially variant contextual information. In addition, SCP with moderate-capacity backbones achieves better performance compared to other network variants in a higher-capacity backbone. This results in 23.8%, 64.9% and 74.7% reduction in GPU memory usage, FLOPs, and network size respectively without sacrificing any segmentation accuracy.

4.3 Liver Tumor Segmentation

The second experiment is liver tumor segmentation using images scanned from contrast-enhanced abdominal computerized tomography. It is a challenging task as the image contrast is highly heterogeneous and shape of the liver is diffusive. An open challenge 2 dataset LiTS [Bilic et al., 2019] from MICCAI 2017 is adopted in our experiment. The dataset is collected from six clinical centers around the world using different scanners and acquisition protocols. There are in total 131 scans, where the in-plane resolution of each scan varies from 0.55 mm to 1.0 mm with a slice spacing ranging from 0.45 mm to 6.0 mm. The horizontal shape of each 3D scan is 512×512, and the number of slices varies from 74 to 987. In the image pre-processing step, we truncate the intensity values of each scan to [-200, 250] in order to exclude the nonliver information. Following previous work [Ma et al., 2022], we resize the in-plane image from 512×512 to 256×256. In the experiments, the dataset is split into three subsets for model training (91), validation (13), and testing (27). Similar to the WMH segmentation task, we report the average of three independent runs with different random seeds for each model.

Quantitative Results

As shown in Table 2, we compare SCP with other network variants Base, Base F, and Base B using different network capacities (Nc = 128, Nc = 256 or Nc = 512). Please note that FLOPs, GPU memory usage and network parameter size are excluded in Table 2 because LiTS datasets are trained with the same network as done in WMH. In the lowcapacity group (Nc = 128), Base F has a slightly better L-F1 and L-PPV, but the proposed scp improves the Dice and LDice scores by 11.0% and 2.8% respectively. SCP (128) has a 5.7% increment for Dice, a 4.4% increment for L-Dice, and a 7.6% increment for L-TPR compared to the Base (128). In addition, SCP (256) consistently outperforms other variants from the moderate-capacity group in terms of Dice and LDice. However, similar performance gain as in the WMH segmentation task is not observed in LiTS segmentation in the high-capacity group. Further, the scores of L-F1 and LDice are lower in the high-capacity group than those in the moderate-capacity group, indicating that these networks may have encountered overfitting problem.

2 https://competitions.codalab.org/competitions/17094

0

Figure 6: Computational comparison of GPU memory, FLOPs, and parameter size for four networks. SC2 indicates SCP (256), BF5 indicates Base F (512), BB5 indicates Base B (512), and BA5 indicates Base (512). All the numbers above are obtained from the same machine used for network training. Each module is tested using an input feature tensor with size (3 × 160 × 224).

4.4 Statistical &Complexity Analysis

Paired t-tests are used compare performance metrics of Dice and L-Dice between the proposed SCP (256) and other network variants Base (256), Base (512), Base F (512), Base B (512). SCP (256) from the moderate-capacity group is used to compare other variants from the high-capacity group, because the goal for SCP is to improve computational efficiency and in the meantime maintain or achieve better segmentation accuracy by relaxing the spatial invariance. As we can see from Fig. 5a and Fig. 5b, in WMH segmentation task, except for Base B (512) on Dice, SCP (256) outperforms all other network variants with statistical significance (p < 0.05) on both Dice and L-Dice. Fig. 5c and Fig. 5d show that in LiTS segmentation task, SCP outperforms all other network variants in terms of Dice and L-Dice scores; however, SCP (256) only outperforms Base F (512) on Dice and Base (512) and Base B (512) on L-Dice with statistical significance (p < 0.05).

Combining results from paired t-tests and Fig. 6, we can conclude that the proposed SCP (256), relaxing spatial invariance, achieves a significant reduction on computational cost (23.8%, 64.9% and 74.7% reduction in GPU memory usage, FLOPs, and network size) without losing any segmentation accuracy on both WMH and LiTS segmentation tasks. The main advantage achieved by SCP is the computation efficiency, favorably making the image analysis tool more accessible in clinical environments without GPU support.

4.5 Discussions

To understand how SCP relaxes the spatial invariance and how datasets affect the learning of the spatially covariant weight W ∈ RN×C×H×W , we have plotted five key statistics of W as shown in Fig. 7. To make value color consistent, we use min-max to normalize each statistic image with value ranging from 0 to 1. Please note that as both WMH and LiTS segmentation tasks are binary, N is set to 1. From left to right in Fig. 7 are the maximum values, mean values, minimum values, L2-Norm and standard deviation (STD) of W ∈ RC×H×W along the channel dimension respectively.

As shown in Fig. 7, the weight W learned from the WMH segmentation shows a stronger pattern than that learned from the LiTS segmentation, where all five statistics of the weight delineate an average image of the shape of human brain. The statistic images from the LiTS segmentation instead delineate the shape of human breast rather than the target liver. The boundary of the max value map is clearer in WMH than in LiTS, and the value distribution across five statistics is more consistent in WMH than LiTS. Moreover, in the max, min and mean value maps from the WMH, value decreases with distance from the center, and especially in the max value map, there is a clear boundary between averaged brain area and background. The max value map from WMH segmentation shows consistency with the receptive field where pixels closer to the brain center has a larger receptive field than those farther away. However, as the shape and location of human liver varies greatly from patient to patient, such phenomenon is not observed from the LiTS segmentation.

Interestingly, we have found out that SCP is more sensitive to small lesions, and more details and qualitative results can be found in the supplementary materials. In conclusion, SCP can help relax the spatial variance and benefit lesion segmentation tasks with datasets having structured representation or spatially inhomogeneous features, e.g. aligned human brains with hierachical tissue structure.

5 Conclusions

In this paper, we present a novel network module, called spatially covariant pixel-aligned classifier (SCP), to trade-off between computational efficiency and accuracy for lesion segmentation. SCP relaxes the spatial invariance constraint imposed by convolutional operations and optimizes an underlying implicit function that maps image coordinates to network weights, the parameters of which can be obtained along with the backbone network training and later used for generating network weights to capture spatially variant contextual information. We find out that SCP suits lesion segmentation tasks with datasets having structured representation or spatially inhomogeneous features. We believe that the proposed SCP technique will contribute to more medical imaging applications in the future.