RF-DTR: A Multi-Stage DCT Token Regression Network for Progressive Rib Fracture Mask Refinement

1 Introduction

Rib fracture detection presents a significant challenge due to the need to accurately identify small, hollow lesions with intricate geometries in large 3D voxel spaces. The scarcity of positive samples further complicates tasks such as classification [Lindsey et al., 2018; Cheng et al., 2019; Huang et al., 2023], segmentation [Yao et al., 2021; Wu et al., 2021], and object detection [Yao et al., 2021; Yu et al., 2022]. While generative anomaly detection models trained exclusively on healthy samples have demonstrated potential in clinical applications, their reliance on one-class setting often undermines their robustness. These models identify high reconstruction errors from out-of-distribution (OOD) data [Fernando et al., 2021] as pixel-level anomalies. However, recent studies [Lu et al., 2023; You et al., 2022; Zhang et al., 2023] indicate that generative models can inadvertently reconstruct OOD samples with high fidelity, leading to false negatives in anomaly detection. Although semantic segmentation-based methods typically achieve higher accuracy, their ability to capture spatial details remains limited.

∗Corresponding author. Email: lianghu@tongji.edu.cn

Despite the progress in rib fracture detection, existing methods inadequately leverage the high-frequency characteristics of fractures in the frequency domain. For instance, FracNet [Jin et al., 2020] employs a sliding window strategy with 3D U-Net variants for patch-wise learning, establishing a foundational pipeline for subsequent works [Wu et al., 2021; Yao et al., 2021]. However, as illustrated in Figure 1, our empirical analysis demonstrates that rib fractures exhibit greater discriminability in the frequency domain, particularly in the high-frequency components. This observation underscores the potential advantages of frequency-domain modeling. Clinically, annotating small and hollow rib fractures is an iterative process that relies on human expertise for refinement and quality assurance. While this well-established method produces reliable results, it is often labor-intensive and susceptible to human error. An automated workflow that accurately replicates this process would be highly desirable, offering the dual benefits of ensuring robust quality and maintaining interpretability.

Our study targets two fundamental challenges: (1) detecting small and hollow fractures and (2) designing interpretable models that emulate clinicians' annotation process, ensuring a more transparent and intuitive decision-making framework. Recent research [Xu et al., 2020; Wen et al., 2022] suggests that frequency-domain representations can serve as effective feature embeddings. Since fine structural details in CT images primarily manifest as high-frequency signals, accurately capturing these components is crucial. However, previous studies have demonstrated that deep neural networks (DNNs) [Xu et al., 2019], convolutional neural networks (CNNs) [Xu et al., 2020], and Transformers [Wang et al., 2022; Piao et al., 2024] often exhibit insensitivity to high-frequency information, limiting their ability to learn fine-grained structures. Empirical results in Figure 3(a) further illustrate this challenge in rib fracture detection. To mitigate this issue, we introduce a frequency-domain regularization to enhance high-frequency learning. Last but not least, we propose a multi-stage penalty mechanism that progressively refines predictions, closely mimicking expert annotation workflows.

We formulate rib fracture detection as an instance segmentation task and propose a model that learns from spatialfrequency transformed input images and corresponding output masks. Our model integrates the discrete cosine transform (DCT) [Ahmed et al., 1974] and employs a progressive mask refinement strategy. Inspired by hierarchical designs in computer vision, we introduce frequency-domain regression modules and a conditional penalty term to improve mask prediction. Our key contributions are as follows:

• We conduct a frequency-domain analysis revealing that existing rib fracture detection models suffer from a critical limitation: insufficient high-frequency learning.
• To address this issue, we propose an encoder-only DCT token regression network that operates entirely in the frequency domain, significantly enhancing sensitivity to fine-grained structures.
• We introduce a novel Mahalanobis regularization to enhance high-frequency learning. Moreover, we improve our method's interpretability for a transparent decisionmaking process by a unified cross-stage penalty.
• We validate our approach through experiments on a public CT benchmark and our curated dataset, demonstrating superior performance in segmentation and detection tasks, surpassing state-of-the-art (SOTA) methods.

2 RelatedWorks

2.1 Classical Methods: Challenges and Advances

Deep learning models achieve high recall but often exhibit higher false positive rates than radiologists [Zhang et al., 2021]. Existing methods [Chen et al., 2017; Jin et al., 2020] struggle to capture fine-grained fracture features, limiting their effectiveness in precise localization. To mitigate this issue, some approaches integrate detection and segmentation techniques. For instance, [Wu et al., 2021] combines 2D Faster R-CNN for detection with 3D U-Net for segmentation, while a three-stage pipeline [Yao et al., 2021] sequentially performs rib segmentation, localization, and fracture classification. Cascade-based framework [Zhang et al., 2021] leverages the Foveal network [Brosch and Saalbach, 2018] and Faster R-CNN to refine rib masks and detect fracture candidates. To better capture rib morphology, SA-FracNet [Cao et al., 2023] employs contrastive learning to address the elongated and inclined rib structure, while CCE-Net [Gao et al., 2022] adopts feature fusion. Additionally, SA-FracNet introduces a shape-aware loss function based on Signed Distance Maps to improve fracture delineation. In contrast, our method employs an encoder-only architecture that effectively captures detailed fracture features by frequency modeling.

2.2 Frequency-Informed Learning

Spatial voxel details correspond to high-frequency components. Recent studies have revealed inherent learning biases in neural networks when analyzed from a frequency perspective. The Frequency Principle [Xu et al., 2019] suggests that DNNs inherently prioritize low-frequency signals. CNNs exhibit a strong preference for low-frequency components [Xu et al., 2020], and similar tendencies have been observed in Transformers [Wang et al., 2022; Park and Kim, 2022; Tian et al., 2023; Guo et al., 2023; Piao et al., 2024]. Fast Fourier Transform (FFT) has gained widespread application. GFNet [Rao et al., 2023] utilizes FFT for global feature extraction, while Frequency-Adaptive Dilated Convolutions [Chen et al., 2024] dynamically adjust dilation rates based on local frequency characteristics. FFT-based token mixers provide a computationally efficient alternative to selfattention [Tatsunami and Taki, 2024], and Fourier regularization mitigates high-frequency artifacts [Xu et al., 2019]. Additionally, MDTNet [Zhao et al., 2024] enforces predictionground truth alignment via Fourier constraints, improving tasks such as image reconstruction [Wang et al., 2018; Jiang et al., 2021] and enhancement [Greenspan et al., 2000; Fuoli et al., 2021]. Building on these insights, we propose an efficient DCT-based architecture incorporating a multi-stage penalty mechanism to refine mask predictions hierarchically.

2.3 DCT-Based Frequency-Domain Analysis

DCT is a real-valued frequency transformation that provides greater computational efficiency than FFT [Pan et al., 2022]. It has been integrated into neural networks to replace convolutions with DCT-based perceptrons, reducing computational costs [Pan et al., 2022]. Additionally, a frequency-channel selection method [Xu et al., 2020] eliminates non-salient DCT components without compromising accuracy. In instance segmentation, DCT-Mask [Shen et al., 2021] encodes binary masks into compact vectors, reducing training costs, while PatchDCT [Wen et al., 2022] refines this approach for precise boundary segmentation. In channel attention, DCT frequency analysis [Qin et al., 2021] models channel representation as a compression process. Furthermore, DCTNet [Zhao et al., 2022] reconstructs high-resolution images from lowresolution depth maps by capturing both shared and modalityspecific features. Despite these advances, existing methods such as DCT-Mask and PatchDCT [Wen et al., 2022] rely on L1 loss and overlook critical frequency-domain properties such as scale and correlation. To address this, we introduce a metric learning approach based on DCT that de-correlates frequency features while ensuring scale alignment.

3 Method

3.1 ProblemDescription

We aim to detect and segment rib fractures in CT volumes using a sliding window approach, framing the problem as a 3D instance segmentation task. Formally, the training dataset is defined as D = {(Vi,Mi)}Ni=1 , where each image window Vi ∈ RD×H×W represents a cropped 3D sub-volume extracted from the CT volume, and its corresponding ground truth maskMi ∈ {0, 1}D×H×W indicates the presence of rib fractures at a voxel level. During inference, the model takes Vi as input and predicts a segmentation mask ˆMi, where each of its voxels represents the probability of belonging to a fractured region, as illustrated in Figure 2(a).

3.2 DCT Tokenizer

Spatial-Frequency Transformation. Inspired by the JPEG compression standard [Wallace, 1992] and related methods [Shen et al., 2021; Wen et al., 2022] in 2D computer vision, we design a DCT Tokenizer tailored for volumetric V and M, as illustrated in Figure 2(c). The tokenizer projects spatial patches into the frequency ones. For t ∈ {V,M}, this process is defined as Ft = Tokenizer(t). Specifically, the data is first split into non-overlapping patches along all dimensions:

Here, B denotes the patch size, set to 8 to align with the JPEG standard. The terms DB , HB , WB represent the number of patches along each axis. Each patch undergoes DCT-II encoding as Ft = DCT-II(Bt) (details in Supplementary Materials §B). Notably, B = 8 aligns with the configuration of the 3D ViT encoder, allowing FV to be used seamlessly as input to the ViT's projection layer without any modifications.

The task is then reformulated as patch-level regression, targeting FM = {FM,p}p, FM,p denotes a DCT mask patch, and p = (x, y, z) specifies the patch indices along each axis: x ∈ {1, · · · , DB }, y ∈ {1, · · · , HB }, z ∈ {1, · · · , WB }.

Gating Label Definition. Each patch is labeled as state 0, 1, or 2, representing non-fractured, partially fractured, and fully fractured samples. Following PatchDCT [Wen et al., 2022], the label is determined by the DCT tensor's direct current component (DCC), which reflects its overall intensity: a DCC of 0 corresponds to state 0, B2 √2 to state 2, and otherwise to state 1. The states for all patches of current M are collectively denoted as G = {Gp}p.

3.3 Multi-Stage Token Prediction

The ViT encoder produces hierarchical hidden features represented as h1 , h2 , h3 , h4 = ViT(FV). For each hierarchical level l ∈ {1, 2, 3, 4}, hl ∈ RB3×dmodel , dmodel is the dimension of the ViT patch vector. RF-DTR is composed of three core components: (1) a DCT Token Regressor with RegressorAfter-Geting (RAG) mechanism, (2) a frequency-domain regularization, and (3) a multi-stage conditional penalty. Patch indices are omitted in the subsequent descriptions for clarity.

DCT Token Regressor. As illustrated in Figure 2(b), the output hl from the l-th ViT stage is reshaped first to restore its spatial dimensions:

We present the RAG mechanism, designed to successively predict the gating state and the DCT token for each patch, denoted as ˆ G l , ˆ F lM = RAG(hl ). The module begins by applying stacked 3D convolutions (S-Conv3D) for feature extraction, resulting in hlshare = S-Conv3D(hl ). Subsequently, the gating and regression tasks are executed upon hlshare :

For ˆ F lM, the last dimension corresponds to a flattened cubic structure encoding DCT tokens. As shown in Table 4, the proposed RAG is an effective and important design. However, relying solely on h4 may lead to blurred predictions, particularly in complex boundaries. To compensate for this limitation, we propose a multi-stage design optimized using the gating loss LGat and regression loss LReg. The hierarchical loss function at a single stage l is defined as follows:

Here, ˆ vlp = vec( ˆ F lp ) and vlp = vec(Fp) represent the flattened predicted and ground truth DCT tokens at p in ˆ FM and FM. The set ˆ GPF = {p | ˆ G lp = 1, ˆ G lp ∈ ˆ G l } identifies the indices of patches classified as partially-fractured. We employ shared weights for RAG modules across all stages, supported by the fact that the hierarchical features hl maintain identical spatial scale and target. Under this design, deeper features hl can be viewed as conditional encodings of hl-1 , capturing progressively refined representations.

Mahalanobis-Regularized Regression Loss. Figure 4(a) empirically reveals that the naive hybrid Transformer-CNN architecture is disproportionately sensitive to low-frequency signals, leading to the loss mainly concentrated in highfrequency components. To mitigate this imbalance, we improve the L1 regression loss by introducing the Mahalanobis regularization, enabling scale normalization and feature decorrelation across different frequencies:

Σ ∈ RB3×B3 is a learnable symmetric and positive definite matrix, and its inverse Σ-1 can be factorized as:

Q ∈ RB3×B3 is an orthogonal matrix representing rotation, and Λ is a diagonal matrix representing scaling. The DCT token discrepancy between the prediction and ground truth, projected onto the positive definite cone, is defined as ∆F = ∆v⊤Q, where ∆v = ˆ vlp - vp. The self-adaptive weight factor is subsequently noted as follows:

sg denotes the stop-gradient operation. This formulation leads to the Mahalanobis-regularized loss, as shown below, l and p in Equation 6 are omitted for simplicity:

Unified Cross-Stage Penalty. We propose a cross-stage penalty to encourage deeper stages to generate progressively more accurate prediction masks. An uncertainty function, fUnc, quantifies the discrepancy between prediction and the shared ground truth at each stage. The function is applicable for both gating and regressor:

where fUnc is instantiated by DM in Equation 6 for the regressor and cross-entropy for gating module, ensuring alignment with the primary loss functions. The penalty that exists between consecutive stages l and l-1 at position p is formally defined as follows:

For stage l ≥ 2, the penalty term is computed as:

where |G+| denotes the cardinality of valid positions that are predicted as partially fractured by both consecutive stages. Similarly, ψlG for the gating module can be computed by applying cross-entropy as fUnc over all patches. The overall multi-stage penalty term across all stages is then defined as:

This formulation ensures a progressive contribution from each stage, encouraging the outputs of deeper stages to achieve consistently greater accuracy than their shallower counterparts. The overall loss function incorporates the regularization and multi-stage penalty as follows:

where LGat denotes the cross-entropy loss for the gating module, and LM represents the Mahalanobis-regularized regression loss. The weighting factors α and β control the relative contributions of these terms. This formulation enables multi-stage refinement by mitigating the inherent limitations of L1 loss in frequency-sensitive tasks, enhancing both robustness and precision-particularly in scenarios demanding fine-grained recognition.

4 Experiments and Results

4.1 Dataset and Preprocessing

We conducted experiments using the publicly available RibFrac Challenge dataset [Jin et al., 2020]. In our slidingwindow framework, the patch size was set to D×H ×W = 64×64×64 voxels, following [Jin et al., 2020]. To enhance rib visibility, bone window normalization was applied using a window width of 1200 Hounsfield units (HU) and a window level of 400 HU. The CT intensities were then linearly scaled to [-1, 1] via min-max normalization. Spatial augmentations were applied to training patches, including random perturbation, flipping, and axis permutation. To standardize pixel spacing across different CT scanners, we set the spacing parameter to (0.6, 0.6, 0.6), corresponding to a voxel size of 0.63mm3 . Figure S1 in Supplementary Materials illustrates the coarse nature of rib fracture annotations in RibFrac, which do not fully capture the complexity of real-world fractures. To address this limitation, we refined RibFrac using rib annotations from RibSeg v2 [Jin et al., 2023], producing a higher-quality dataset with reduced label noise. Detailed information on the refinement process is provided in Supplementary Materials §D.

4.2 Settings

Performance Metrics. We evaluate the model's performance in fracture detection and instance segmentation. Following the FracNet workflow [Jin et al., 2020], we report sensitivity, specificity, and F1-score for detection. For segmentation, we use Intersection over Union (IoU) and Dice coefficient, consistent with previous studies [Jin et al., 2020; Yu et al., 2022;

Zhao et al., 2021; Wu et al., 2021]. To assess computational efficiency, we report floating point operations (FLOPs).

Network Configuration and Training Protocol. Our model consists of a DCT token regressor and a four-stage ViT encoder, resulting in a compact and computationally efficient design. The model is randomly initialized and trained for 200 epochs using the AdamW optimizer [Loshchilov, 2017]. The learning rate is set to 1 × 10-4 , with a batch size of 4 and a gradient accumulation factor of 2. The loss function incorporates weighting factors α = 0.3 and β = 0.1. We sample 8 patches per volume to ensure class balance, evenly split between fractured and healthy patches. Fracture patches are extracted around the centroid of each lesion, while healthy patches are selected from symmetrical regions corresponding to the fracture locations and from the spine. During validation and testing, all patches are sampled using a sliding window with a fixed stride. For a fair comparison, we implemented 3D adaptations of the competitor methods DCTNet, DCT-Mask, and PatchDCT, initially designed for 2D inputs. Additional architectural details and implementation specifics are provided in Supplementary Materials §C.

4.3 Quantitative Comparison with SOTAs

Table 1 compares our method against SOTA approaches in spatial and frequency domains. The comparison includes CNN-based architectures and Transformer-CNN hybrid models for the detection task. Our results demonstrate consistent superiority over existing methods on the RibFrac dataset. Notably, the performance gap widens on the more challenging RibFrac-GTRefined dataset, which better reflects real-world clinical scenarios, highlighting our model's robustness in handling complex anatomical structures. Additionally, frequency-domain-based methods generally achieve higher F1 scores than their spatial-domain counterparts, except DCT-Mask, underscoring the advantage of frequencydomain modeling in capturing high-frequency details. Table 3 further corroborates this trend in segmentation results, where our method consistently outperforms others. The superior performance on RibFrac-GTRefined reinforces the effectiveness of our approach in tackling challenging clinical segmentation tasks. Beyond accuracy, our method demonstrates superior computational efficiency, as summarized in Table 2. By adopting an encoder-only architecture with a shared DTR module, our design significantly reduces the overall parameter count while preserving segmentation accuracy. This architectural choice optimizes the trade-off between accuracy and computational cost, making our model more feasible for real-world medical applications.

4.4 Visualization

Figure 3 provides a qualitative comparison of segmentation results between our method and SOTA approaches. Our method exhibits superior segmentation accuracy, particularly in detecting small, hollow structures that pose significant challenges for existing models. Figure 5 illustrates our proposed multi-stage refinement strategy, where penalty terms between consecutive stages enforce consistency and guide deeper stages toward more confident predictions. This design mirrors the iterative refinement process employed by clinicians during manual annotation, enhancing both segmentation reliability and interpretability.

4.5 Ablation Study

Different Input and Output Domains. We explore the effect of varying the data domains for model inputs and outputs. As shown in Table 4, replacing voxel representations with DCT tokens in the input space consistently improves model performance, with larger patch sizes yielding better results. However, substituting the segmentation mask with DCT tokens and using L1 loss on the output side significantly degrades performance. This is likely due to the dominance of healthy voxels, which overshadow the gradients and impair the model's ability to capture fracture features effectively. In contrast, the proposed RAG mechanism substantially enhances model performance, demonstrating its effectiveness in modeling small and hollow targets.

Mahalanobis Regularization. Table 5 presents a quantitative comparison between the L1 loss and the proposed Mahalanobis-regularized loss. We evaluate two self-adaptive weighting strategies for Mahalanobis loss, and the results consistently show its superiority in improving model performance. This improvement is further supported by Figure 4, which illustrates that Mahalanobis regularization alleviates the known deficiency of L1 loss in capturing high-frequency signals, leading to more robust feature learning.

Number of Stages. We investigate the effect of varying the number of stages, as shown in Table 6. While a hierarchical design intuitively suggests performance gains, our results reveal a non-monotonic trend as the number of stages increases from 1 to 4. Performance initially improves with additional stages but declines when the number of stages exceeds three. Nevertheless, hierarchical architectures consistently outperform their non-hierarchical counterparts, emphasizing their effectiveness in structured feature learning.

Unified Multi-Stage Penalty. Table 7 evaluates the impact of the proposed hierarchical penalty, which imposes constraints on both the gating module and the regressor, either independently or in combination. The results indicate that penalizing both components simultaneously yields the highest performance, highlighting the importance of jointly optimizing these modules for optimal results.

5 Conclusions

We propose a novel approach for rib fracture analysis by developing a full-frequency model, which integrates Mahalanobis-regularized frequency loss and a unified crossstage penalty within the RAG module. This method improves interpretability and achieves high-performance fracture detection, bridging the gap between clinical annotation workflows and deep learning model design. Future iterations are expected to improve performance by strategically selecting signals from input DCT tokens and developing a more taskoriented frequency-domain model.

Acknowledgments

This work is partially supported by the National Natural Science Foundation of China (Granted No. 62276190).