Fluid Dynamics-Inspired Network for Infrared Small Target Detection

1 Introduction

Infrared small target detection (ISTD) has been extensively applied in remote sensing and military tracking system. ISTD is challenging because the targets are always small and ambiguous when surrounded by similar background areas. Besides, infrared images are of low contrast and low SNR. Generic segmentation methods cannot achieve expected performance on this task, so we devote to precise and robust ISTD. ISTD methods can be classified into traditional methods and deep-learning-based methods. In early stages, due to lack of public data set, traditional methods [Sun et al., 2020;

∗Corresponding Author.

Marvasti et al., 2018; Zhang and Peng, 2019; Han et al., 2019; Rivest and Fortin, 1996] take the dominance. However, fully relying on prior knowledge and handcraft features, these methods suffer limited accuracy on images with characteristics inconsistent with model assumptions.

In recent years, deep-learning-based methods have lifted the detection performance by a large margin. These methods can be categorized into CNN-based methods [Dai et al., 2021b; Dai et al., 2021a; Wang et al., 2019; Li et al., 2022a; Zhang et al., 2021; Zhang et al., 2022c; Chen et al., 2023] and transformer-based methods[Wang et al., 2022; Qi et al., 2022; Liu et al., 2021; Zhang et al., 2022a]. Despite different network design concept, these methods mainly focus on devising novel neural blocks or feature fusion modules to adapt to ISTD task, but rarely study the ISTD process from the perspective of image evolution, which is significant for constructing an effective and explainable ISTD network and proposes a potential future research direction.

In fluid dynamics, the fluid elements on the fluid intersecting surface exhibit different density distribution at different time. Similarly, the ISTD process can be perceived as a series of intermediate images that change over time. The intuitive analogy between ISTD and fluid dynamics is presented in Fig. 1. The upper part offers the change of feature maps corresponding to the target in the ISTD process, which is to perform from-coarse-to-fine target extraction using the pixel information in the adjacent area. During this process, the edge and shape of targets become more and more clearly visible, and detection results tend to be more precise. The three consecutive images in the upper right part present this process. Specifically, in convolution operations, pixels are jointly determined by multiple adjacent pixels in the previous layer. The lower part describes the density change of a certain intersecting surface in the fluid. The bottom right image shows the density distribution change of the intersecting surface formed by the mutual influence of fluid elements.

In this paper, we explore a novel research routine by abstracting the movement of pixels in the ISTD process as the flow of fluid in fluid dynamics (FD) and propose FDI-Net. On the one hand, most ISTD networks use ResNet as backbone, which is based on one-order Euler method. Intuitively, some mathematical theories (e.g., the definite difference method used for solving FD issues) can be borrowed to devise or improve the ISTD network backbone structure. Therefore, we adopt Taylor Central Difference (TCD) method and propose TCD feature extraction block to locate target pixel from irrelevant background. On the other hand, since small targets in infrared images are of ambiguous shapes and a slight mistake will significantly impact evaluation index, precise detection of edge areas matters to the detection quality of small targets. In fluid dynamics, the Navier-Stokes (N-S) equation is a collection of motion equations describing the momentum conservation of a viscous incompressible fluid and generates a spatial-temporal constraint to guide the flow direction. Thus, we introduce this equation to ISTD and devise N-S Refinement Module to refine the coarse features extracted from TCD block with fine edge details.

Our contributions can be summarized in three folds:

• We are the first to draw an analogy between the dynamic process in the fluid field and the image field in ISTD, where the change of fluid element distribution on the fluid intersecting surface is used to describe the change of pixels in consecutive feature map series. This analogy interprets the changes of microscopic pixels in the ISTD process.
• We propose FDI-Net based on the rule of hydrodynamic fluid element motion that consists of a Taylor Central Difference (TCD) convolution-transformer hybrid structure and an N-S refinement module. Using TCD method, the former structure can serve as the backbone structure to more effectively extract global and local feature. In addition, the latter module converts the spatial information in infrared images into time information and complements the extracted features with finer edge details.
• Our method outperforms others on IRSTD-1k and SIRST in terms of evaluation metrics.

2 RelatedWork

2.1 Infrared Small Target Detection Networks

Generally speaking, ISTD networks can be classified to two categories: CNN-based and transformer-based networks.

CNN-based networks mainly focus on local feature extraction. Dai et al. [Dai et al., 2021a] proposed asymmetric contextual modulation (ACM) for cross-layer information exchange to improve ISTD performance. They also designed AlcNet [Dai et al., 2021b], including a local attention module and a cross-layer fusion module to preserve local features of small targets. Wang et al. proposed MDvsFA [Wang et al., 2019], which applied GAN to ISTD, and achieved a trade-off between miss detection and false alarm. AGPCNet [Zhang et al., 2021] divided the infrared image into patches for better extraction of global and local information and uses cross layer feature fusion to recover lost low-level details. DNANet [Li et al., 2022a] progressively interacted high-level and lowlevel features. ISNet [Zhang et al., 2022c] designed a simple Taylor finite difference-inspired block and a two-orientation attention aggregation module to detect targets. BAUENet [Chen et al., 2023] introduced uncertainty to ISTD by enhancing contexts with target uncertain area maps.

Only extracting local features is not enough because of ubiquitous target ambiguity caused by low contrast and quality of infrared images. Therefore, some transformer-based methods have been put forward to complement local details with global information by adding transformer to CNN structures. IAANet [Wang et al., 2022] just concatenated local patch outputs from a simple CNN with the original transformer, causing limited feature extraction especially in ambiguous scenarios. RKformer [Zhang et al., 2022a] applied Runge-Kutta method to build coupled CNN-Transformer blocks to highlights infrared small targets and suppresses background interference.

The above ISTD networks focus on building either effective neural blocks or feature fusion modules, none of them describe the ISTD mechanism from the image evolution perspective during the from-coarse-to-fine detection process. In this paper, we open a novel research perspective by abstracting the movement of pixels in the ISTD process as the flow of fluid in fluid dynamics.

2.2 Neural Ordinary Differential Equation

Recently, ODE-inspired networks have gained wide attention since it can deliver higher-accuracy feature extraction performances. Weinan [Weinan, 2017] firstly discovered the link between and discrete ODE and ResNet [He et al., 2016]. Some insightful networks [He et al., 2019; Li et al., 2021; Lu et al., 2019; Zhang et al., 2022b] can also be interpreted from the ODE perspective. To the best of our knowledge, [Zhang et al., 2022c; Zhang et al., 2022a] are the only two works that design ISTD network with ODE theories. From a dynamic system perspective, the two works adopt finite difference method and Runge-Kutta method respectively for network design. Inspired by the two works, we create a new research routine by borrowing the useful idea from the field of FD, where the movement of pixels in the ISTD process is analogous to the flow of fluid. Contrary to the above networks, we devise FDI-Net following the explicit central difference method and Navier-Stokes equation in fluid dynamics, which presents a superb capacity of preserving clear details in ISTD.

2.3 Fluid Dynamics and Neural Network

As a branch of mechanics, fluid dynamics studies the interaction and flow laws of the static and moving state of the fluid itself and the relative motion between the fluid and the solid boundary wall under the action of various forces. Typically, fluid micro element is the basic unit in fluid dynamics and its behaviors (fluid motion) are described by the Navier-Stokes equations, which consist of multiple differential equations derived from three basic physical principles: Newton's second law, the law of mass conservation and the law of energy conservation. Common variables like velocity v, pressure p, density ρ and temperature T can be expressed as functions of position (x, y, z) and time t. The Navier-Stokes equations describe the link between the properties of fluid elements over time and space, and thus makes it possible for information to get transformed to different dimensions. Analogously, in the ISTD process pixels in different areas of an infrared image differ in values, whose properties are described by functions of position (x, y) and depth location i of the feature map where the pixel is in the network.

In recent years, researchers have begun to try linking fluid dynamics with neural networks for specific tasks. SPNets framework [Schenck and Fox, 2018] was introduced to integrate fluid dynamics with deep networks for liquid manipulation. Zhang et al.[Zhang et al., 2022b] proposed a secondorder finite difference residual network for super resolution. In [Deng et al., 2019], a general super-resolution reconstruction strategy for turbulent velocity fields was proposed using a GAN framework. In [Belbute-Peres et al., 2020], differentiable PDE solvers and GNNs were combined for fluid flow prediction.

Based on the analogy between fluid field and image field, we model the ISTD process to extract more useful feature information and achieve better detection performance. The variables in the Navier-Stokes equation can be determined by the thermodynamic equation f (ρ, p, t ) and the material properties of the medium, which can be solved given the initial and boundary conditions. Because this equation has secondorder derivative terms, it is difficult to find an exact solution except under some special conditions. Therefore, we turn to numerical solutions and notice that finite difference method is commonly used in solving fluid dynamics problems. Inspired by this, we use high-order central difference method to model the ISTD process.

3 ProposedMethod

3.1 Analogy of ISTD and Fluid Dynamics

The paper is inspired by the analogy between the ISTD process and the fluid dynamics process, because the pixel movement during ISTD can be analogous to the movement of fluid flow in fluid dynamics. The analogy can be interpreted from two aspects. From the microscopic view, the fluid element studied in fluid dynamics as the basic unit moves under the joint constraints of temperature, pressure, and surrounding fluid elements. The image pixel changes in a targeted manner and gets influenced by basic operations such as convolution, pooling and some surrounding pixels. From the macroscopic perspective, the distribution of an intersecting surface in fluid changes regularly over time. In the ISTD process, after multi-step feature extraction and image reconstruction, the 2-D feature maps change towards the final prediction where only target pixels are highlighted.

The particle movement processes in the fluid field and the image field are described respectively as follows. In fluid dynamics, interaction forces exist between particles, so each particle moves under the combined constraints of internal and external forces. For each particle, forces exist in pairs, such as friction, pressure and viscosity, so each particle is influenced by its surrounding particles. The ISTD process is similar, with each pixel depending on other pixels in its vicinity. Therefore, we can borrow the basic theory of fluid dynamics to devise new ISTD network structure. Present ISTD algorithms use 2-D convolution kernels for filtering. The value of each pixel is updated based on the products of its neighboring pixels and the convolution kernel weights. Different image filtering effects, for example edge extraction, can be achieved by applying corresponding convolution kernels. The pixel movement under various convolution operations and loss functions finally leads to the final detected result with clear edges. The ISTD process between the original image and the final result can be interpreted as interpolating the target pixels (i.e., movement of pixels) using surrounding pixels while generating clear edges. The inherent structure of edges serves as a constraint on the pixel movement process.

The consistency of the particle motion law in image and fluid fields inspires us to use the fluid dynamics theory for novel ISTD network construction. Considering that the problems in fluid dynamics are continuity problems, the exact solution to related equations is difficult to find, so we turn to numerical solver and choose Taylor Central Difference (TCD) method with third-order accuracy to build TCD feature extraction block to preliminarily locate target areas from ambiguity. Besides, the N-S equation in fluid dynamics builds the relationship between fluid element properties over time and space. We can apply the N-S equation to the ISTD process to convert the space information of image pixels into time dimension, and this can bring edge refinement to the TCD body part extraction result and achieve better detection performance.

3.2 Overall Architecture

We propose our FDI-Net based on Taylor Central Difference method and the Navier-Stokes equation for ISTD. As shown in Fig. 2, the encoder contains a backbone network composed of a cascaded TCD feature extraction branch (Section 3.3) and a parallel N-S Refinement module (Section 3.4). In the backbone, the TCD feature extraction branch generally determines the primary location of target pixels, which helps distinguishing ambiguous targets from irrelevant background interference. Besides, we devise a parallel N-S Refinement module to refine coarse predictions from TCD blocks with fine edge details. The two branch features at each stage are added before sent to decoder for upsampling by skip connections. Sobel operator is applied to get the initial coarse edge map of input image before fed into the N-S branch and the gradient value of the masks are used as labels to deep supervise this branch to guarantee target edges are explored. Specific losses (Section 3.6) are adopted to supervise the generation of TCD and N-S side outputs and the final prediction.

3.3 TCD Feature Extraction Block

Due to the low quality and low contrast of infrared images, in many cases targets are in ambiguity and not easy to get distinguished from surroundings, leading to many miss detection and false alarms. To tackle this issue, we propose a specific residual block to explore effective target features from irrelevant background interference. To this end, the residual block is implemented as an ODE solver based on an efficient numerical scheme, Taylor Central Difference (TCD), i.e., we devise a basic TCD feature extraction block to serve as part of the backbone network. We choose TCD due to its better accuracy in solving ODE than Euler method [Anderson and Wendt, 1995].

To further describe target pixels from ambiguity, spatial information is required to eliminate uncertainty. Hence, we also design the Spatial Information Integration (SII) across the basic Uniformer [Li et al., 2022b] block to constitute basic forwarding block (BFB) and embody it into our TCD feature extraction block to further promote information exchange while encoding more precise spatial information. Specifically, BFB consists of four key modules: Dynamic Position Embedding (DPE), Multi-Head Relation Aggregator (MHRA), Feed-Forward Network (FFN) and Spatial Information Integration (SII):

where Xin is the input token tensor. SII mines the relationship between pixels, not just patch tokens, by introducing attention in two spatial dimensions, making our TCD feature extraction block more suitable for ISTD task. The structures of SII and TCD feature extraction block are illustrated in Fig. 4.

Specifically, we discretize the ODE using the Taylor central finite difference equations with third-order accuracy, i.e.,

where u is a target depending on the input variable x. The above formula can be rewritten as:

The left side of the equation can be defined as the change of u from step j - 2 to step j + 1, which can be approximated with a specific block, denoted as δf(x). Therefore, the above equation can be rewritten as:

In this paper, we design the TCD feature extraction block to implement Equation (4), where δf(x) is implemented as our proposed basic forwarding block.

Perceiving the pixels of the infrared image as a collection of points following a certain distribution, the proposed TCD structure can determine the primary movement direction of pixels, leading the diffused points to the main body areas of small targets. However, during this process, the pixels cannot avoid a certain degree of dispersion, causing uncertainty in edge areas in the final prediction. To address this problem, additional constraints should be applied to the ISTD process so that edge information can be enriched. Thereby, we consider devising an edge module from the perspective of dynamic pixel-value distribution to refine the residual extraction process with finer edge details.

3.4 N-S Refinement Module

The continuity equation, which describes the transport behavior of a conserved quantity, is included in N-S equations in fluid dynamics. Continuity equation can be divided into four forms as shown in Table 1 depending on different object of study and computation method. Since each of these forms can be mutually converted, we only consider the PDE in conservative form

where ρ denotes the fluid density, ⃗v is the velocity vector, and ⃗v = (u⃗i, w⃗j). According to the multiplication law of the Hamiltonian operator, equation (5) can be rewritten as

During ISTD process, the extracted feature map will change as the network goes deep, and this process can be viewed as the change on an intersecting surface of 3-D fluid. Therefore, we can transfer the motion equation in the fluid field to the 2-D image domain. Equation (5) can be rewritten as follows:

Where the divergence change of ⃗v can be understood as the change of pixel number per unit image area. In the encoder stage, there is no fusion of images, which means that pixels do not flow in or out. Therefore, we assume that ∂u ∂x = 0 and ∂w ∂y = 0, meaning that the partial derivatives of two velocity components u and w along their corresponding directions are 0. Then, equation (7) is converted to

In the image field, ρ stands for the pixel value, t stands for the layer depth of the feature map where the pixel is, and ∂ρ ∂x and ∂ρ ∂y are the gradients of the pixel value in different directions. Specifically, the gradients of edge areas are larger, while for smooth areas the gradients are smaller. The objective variable in equation (8) is ∂ρ ∂t , we convert this variable to the difference form as follows

Equation (9) builds the relationship between the change of pixel value in time and spatial domains during feature extraction. We choose a residual block layer to calculate the partial derivative term and rewrite equation (9) as

On the right side of equation (10), ⃗v stands for the forward extraction direction, ∇ρi is the partial derivatives of pixel values in the x-and y-directions, which is the gradient term. As we know, the gradients of edge areas are larger, while for smooth areas the gradients are smaller. Thus, we place basic residual block to N-S Refinement Module to extract edge feature, the direction of which serves as the ⃗v term; and we use the Sobel operator to explore the gradients of each infrared image, which serve as the ∇ρi term. Combining the residual block and Sobel operator together constitutes our N-S Refinement Module. Inspired by the N-S control equation, the N-S Refinement Module converts the space information (along the x, y directions) to time information (along the i direction), thereby refining coarse TCD block predictions with edge details.

3.5 Overall Structure of FDI-Net

Combining the N-S Refinement branch with the TCD Feature Extraction structure, we can obtain the final expression of the overall feature extraction of the encoder as

where NS(uj) denotes the N-S Refinement function defined in equation (10). The final prediction is obtained after the decoder. Therefore, the final training objective is

where we set I as the input and GT as the ground truth, and θ refers to all the learnable parameters in the proposed FDI-Net.

3.6 Loss Function

ISTD is regarded as a pixel-level classification problem, with each pixel in the image classified as target or background. Generally, the problem is solved by using binary crossentropy loss. However, the pixels of targets only account for a small proportion, which means that the pixel distributions of target and background are very uneven. In this case, if the loss function treats target pixels and background pixels equally, the model constrained by the loss function is optimized toward background categories to fall into the local optima. Furthermore, the pixel accuracy of target is reduced under the constraint of the loss function. To address sample imbalance, we tailor a hybrid loss Lhyb for ISTD based on binary cross-entropy loss and Dice loss [Sudre et al., 2017].

where λ is a coefficient to balance between LBCE and LDice and is set to 1. From equation (13), the similarity between the predicted result and the label only determines LDice, which is independent of the number of training samples and can mitigate the sample imbalance problem. However, Dice loss has strong gradient changes, which causes unstable training. Considering the above, we design a new loss function Lhyb, which not only maintains gradient stability but also prevents falling into the local optimum.

Our total loss function LTotal includes LOut as the main loss function and TCD interior area loss (LTCD) and N-S edge area loss (LN-S) as two auxiliary loss functions. All of these losses are in our hybrid loss form.

4 Experiment

4.1 Datasets, Metrics and Implementation Details

We choose IRSTD-1k and SIRST as benchmarks. IRSTD1k consists of 1,000 real infrared images of 512×512 in size, containing various kinds of small targets and scenes. SIRST contains 427 infrared images.

For evaluation metrics, since there has not been any single set of evaluation metrics get widely accepted for ISTD task, we choose the most commonly used IoU and nIoU (Normalized Intersection over Union) as our evaluation metrics, which are defined as:

where Ai and Au are the areas of intersection region and union region. N is the total number of samples, TP[.] is the number of true positive pixels, T[.] and P[.] denotes the number of ground truth and predicted positive pixels. The algorithm is implemented in Pytorch, with Adaptive Gradient (AdaGrad) as optimizer, initial learning rate set to 0.05 and weight decay coefficient to 0.0004. A Titan Xp GPU is used for training, with batch size set to 4. Training on SIRST and IRSTD-1k takes 600 epochs and 400 epochs respectively.

4.2 Quantitative Results

We select some most well-performed ISTD methods for comparison. Among them, Tophat [Rivest and Fortin, 1996], PSTNN [Zhang and Peng, 2019] and MSLSTIPT [Sun et al., 2020] are traditional methods; MDvsFA [Wang et al., 2019], ACM [Dai et al., 2021a], AlcNet [Dai et al., 2021b], DNANet [Li et al., 2022a], UIUNet [Wu et al., 2022] and ISNet [Zhang et al., 2022c] are CNN-based methods; IAANet [Wang et al., 2022] and RKformer [Zhang et al., 2022a] are transformerbased methods. The results of DNANet, IAANet, UIUNet are implemented by ourselves with source codes, the rest are referred from [Zhang et al., 2022c]. As shown in Table 1, the proposed FDI-Net achieves the SOTA performance in terms of all the evaluation metrics compared with all other methods on both benchmarks in IoU and nIoU. Traditional methods perform poorly for highly depending on hand-crafted features. Nevertheless, most CNN-based and transformer-based methods lay insufficient emphasis on target edges, causing lower IoU and nIoU. Our FDI-Net delivers promising results owing to the designed TCD block to effectively extract useful features from disturbance and the N-S refinement module to further refine extracted features with edge details.

4.3 Visual Results

Some visual results obtained by different methods and intermediate branch outputs on two benchmarks are shown in Fig. 4. As can be seen, even in low contrast and ambiguous scenarios, our FDI-Net can not only accurately locate the targets but also segment out complete and precise target shapes.

4.4 Ablation Study

To investigate the effectiveness of each component in our FDI-Net, we perform several ablation studies on SIRST and IRSTD-1k dataset. The ablation study results of TCD Feature Extraction Block (TCD) and N-S Refinement Modules (NS) are shown in Table 3. As can be seen, each of them enhance the performance of the UNet baseline and combining both of them brings the best results, implying that they are complementary to each other. This is because the TCD branch plays a diffusion role to guide the primary movement of pixels for locating target main areas; the N-S module constrains the diffusion process to curtail dispersion so that target edge information can be enriched.

The ablation study of TCD Feature Extraction Block is shown in Table 4, where our TCD Block outperforms other block designs by a large margin, showing its superiority in distinguishing target pixels from ambiguity. RK2+BFB means combining second-order Runge-Kutta method with BFB to substitute TCD Block. The ablation also demonstrates the effectiveness of TCD method and our SII tailored for ISTD.

The ablation study of N-S Refinement Module is shown in Table 5. During data pre-processing, the Sobel operator is used to extract the coarse edge of the target from the input image to serve as the gradient term in equation (10). We try replacing ResBlock with simple Conv/ReLu/Conv block and the Sobel operator with other the Canny operator. Results show that the combination of the Sobel operator and ResBlock delivers the best result.

5 Conclusion

Motivated by the analogy of pixel movement during ISTD process and flow of fluid in fluid dynamics, we propose FDINet for ISTD. Specifically, we design a TCD Feature Extraction Block based on central difference method and a N-S Refinement Module based on Navier-Stokes equations in fluid dynamics. The TCD block guides the primary movement of pixels like diffusion to locate target main areas while the N-S module constrains the diffusion process to curtail dispersion to refine main area features of targets with edge details. Experiments on SIRST and IRSTD-1k validate the effectiveness of our network.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 62272430, No. U20B2047) and the Fundamental Research Funds for the Central Universities.