Streamlining Long-Chain Reasoning via Differentiable Hierarchical Fusion

1 Introduction

The evolution of Large Language Models (LLMs) has recently shifted toward Large Reasoning Models (LRMs) like DeepSeek-R1 [DeepSeek-AI et al., 2025] and OpenAI o3 [OpenAI, 2025], which utilize reinforcement learning to elicit sophisticated Chain-of-Thought (CoT) behaviors [Wei et al., 2022]. However, the pursuit of enhanced logical depth often comes at a significant cost, manifesting as high computational overhead and protracted latency [Wu et al., 2025]. A primary driver of this inefficiency is the tendency for models to engage in "overthinking" [Chen et al., 2025], a phenomenon where the model generates redundant or unnecessarily exhaustive reasoning paths for tasks that could be resolved with far less cognitive effort. This lack of adaptive computation not only inflates inference costs but also hinders the deployment of LRMs in real-time applications, necessitating a more nuanced balance between reasoning rigor and efficiency.

∗Corresponding author

†Corresponding author

Model fusion [Ilharco et al., 2023] has emerged as a promising paradigm for balancing accuracy and efficiency by synthesizing the strengths of base LLMs and reasoning LLMs. However, existing techniques encounter three critical bottlenecks when distilling long-form reasoning into concise outputs. First, established metric-based approaches, such as TIES-Merging [Yadav et al., 2023] and DARE [Yu et al., 2024], and even activation-informed methods like AIM [Nobari et al., 2025] often suffer from performance collapse due to weight and activation noise. These methods rely on static heuristics or noisy statistics derived from limited calibration data, which frequently results in destructive interference that degrades the model's core logic during the fusion process. Second, current frameworks overlook the heterogeneous weight distributions inherent in transformer architectures. By applying uniform merging coefficients across the Attention and MLP modules, they overlook the divergent variance patterns exhibited by these components when transitioning from base to reasoning-heavy weights, resulting in suboptimal parameter integration. Third, traditional merging lacks a dynamic feedback mechanism to refine parameters based on downstream performance. Relying on fixed global coefficients or hand-crafted importance scores prevents these methods from navigating the complex Pareto frontier between reasoning depth and response conciseness, ultimately failing to achieve a truly adaptive fusion.

In this paper, we introduce Differentiable Hierarchical

Fusion (DHF), a framework that casts the merging of longto-short reasoning models as a differentiable optimization problem. As illustrated in Figure 1, our approach is motivated by two critical empirical insights derived from a systematic analysis of weight distributions in reasoning versus base models. First, we observe a significant disparity in variance profiles within transformer blocks: Attention mechanisms exhibit substantially higher weight variance (σ2 att ) between base and reasoning models compared to MLP sections (σ2mlp ), with variance ratios frequently exceeding 2.5. This suggests that reasoning-specific knowledge is primarily localized within attention parameters, necessitating finer-grained control during the fusion process. Second, we find a hierarchical functional divergence across the model architecture: shallow layers provide the structural foundation essential for linguistic coherence, whereas deeper layers govern complex reasoning logic and verification steps. Leveraging these insights, we establish a dual-factor adaptive weighting scheme that optimizes intra-block coefficients βl,k for component-level fusion and inter-block coefficients αl for layer-wise importance. This is achieved through a differentiable learning framework that minimizes cross-entropy loss on a multi-domain calibration set, following the optimization efficiency established in recent literature [Yang et al., 2024]. To ensure robust convergence, we employ an activation outlier-aware initialization strategy, seeding coefficients based on the concentration of high-magnitude activations to preserve critical reasoning pathways. Furthermore, we employ multi-calibration set construction by sampling diverse reasoning problems frommathematics, coding, and logical domains to enhance the generalizability of learned fusion coefficients.

Experimental Results: We evaluate the performance of DHF on different mathematical reasoning benchmarks, including GSM8K, MATH 500, and AIME24. On the Qwen2.5 series, the DHF shows a mean accuracy improvement of 2.8% over four competitive merging baselines, including Task Arithmetic, TIES, DARE, and AIM. Over LLaMA-3.18B, the mean accuracy improvement reaches 4.4%, while simultaneously reducing average response lengths by 55%. Leading results are obtained on the large-scale OlympiadBench and CollegeMath datasets. This effectively bridges the gap between efficiency and performance, enabling practical deployment of reasoning capabilities in resource-constrained environments.

2 RelatedWork

2.1 Efficient Long-to-Short Reasoning

The emergence of LRMs has introduced the "overthinking" bottleneck, where models expend excessive tokens on trivial logic [Chen et al., 2025]. Current mitigation strategies generally fall into three categories. Inference-time heuristics, such as prompt-based constraints (Concise CoT [Renze and Guven, 2024]) or explicit token budgets [Han et al., 2025], offer immediate but brittle solutions that often compromise accuracy on non-trivial tasks. Architectural and routing methods, including RouteLLM [Ong et al.,

2025] and Sketch-of-Thought [Aytes et al., 2025], dynamically allocate resources but introduce additional latency via auxiliary models or symbolic processing. Training-centric approaches utilize length-penalized reinforcement learning [Luo et al., 2025; Aggarwal and Welleck, 2025] or supervised fine-tuning on distilled concise datasets [Kang et al., 2025; Munkhbat et al., 2025]. While effective, these require prohibitive computational resources. DHF departs from these paradigms by internalizing the efficiency-accuracy trade-off within the model's weight space. Unlike prompt-based or routing methods, DHF requires no inference-time overhead. Unlike RL-based methods, it achieves competitive conciseness through a brief, differentiable optimization on a tiny calibration set, effectively bridging the gap between training-free merging and full-scale fine-tuning.

2.2 Model Fusion

Model fusion (also known as model merging) enables the fusion of specialized capabilities into a single backbone without the cost of multi-task training [Wortsman et al., 2022]. Foundational techniques like Task Arithmetic [Ilharco et al., 2023] utilize linear combinations of task vectors, while subsequent works like TIES-Merging [Yadav et al., 2023] and DARE [Yu et al., 2024] focus on mitigating parameter interference through sign election and delta-pruning. However, these weight-centric methods ignore the runtime activation dynamics that define model behavior. To address this, activation-informed methods [Li et al., 2025a; Li et al., 2025b] ACM [Yao et al., 2025] use calibration data to weight important parameters. AdaMerging [Yang et al., 2024] further advanced this by introducing differentiable layer-wise coefficients. Despite their progress, these methods treat transformer layers as monolithic entities, ignoring the significant variance between attention and feed-forward components. DHF extends the differentiable merging framework by introducing a hierarchical dual-factor weighting scheme. By optimizing both inter-block (layer) and intra-block (component) coefficients, and utilizing outlier-aware initialization [Zhu et al., 2025], DHF captures the fine-grained structural nuances of reasoning models that uniform merging methods overlook.

3 Methodology

3.1 Formulation for ReasoningModel Merging

We begin by formalizing the long-to-short reasoning merging problem. Given a base model θbase (efficient but less capable in complex reasoning) and a reasoning model θreason (capable but generating verbose outputs), we define the task vector as:

The standard merging approach in Task Arithmetic [Ilharco et al., 2023] produces a fused model through a global linear interpolation:θfused = θbase + λ∆ (2)

where λ ∈ [0, 1] is a scalar coefficient controlling the strength of reasoning capability integration. When λ = 0, we recover the base model with concise outputs; when λ = 1, we obtain the full reasoning model with verbose chains.

In the context of long-to-short reasoning, our goal is to identify configurations of merging coefficients that simultaneously minimize output length L while maximizing task accuracy A. This can be formulated as a bi-objective optimization:min

where µ is a trade-off hyperparameter. However, uniform global λ fails to capture the heterogeneous importance of different model components, leading to sub-optimal solutions.

3.2 Dual-Factor Differentiable Framework

Through systematic analysis of weight distributions in reasoning versus base models, we observe two critical structural patterns that motivate our hierarchical approach.

Intra-block Weight Variance. To quantify the structural distribution of reasoning capabilities, we analyze the variance of the task vectors within each transformer block l. For the attention (att) and feed-forward (mlp) components, the variance is defined as:

Empirical analysis of Qwen2.5-7B and DeepSeek-R1-DistillQwen-7B reveals a consistent disparity: σ2 att /σ2mlp > 2.5. This indicates that reasoning-specific knowledge is predominantly concentrated in the attention mechanisms. Consequently, uniform merging within a block is suboptimal; while attention weights require an aggressive integration of reasoning logic, MLP weights should remain closer to the base model to preserve efficiency.

Inter-block Importance Hierarchy. Analyzing layer-wise activation statistics on reasoning tasks reveals a non-uniform distribution of reasoning-critical operations. Shallow layers (first 25% of depth) exhibit higher activation stability and serve primarily structural roles in token representation, while middle-to-deep layers (50%-90% depth) show pronounced outlier activations correlated with reasoning steps. The final layers perform answer extraction and formatting, requiring less reasoning depth. This hierarchical importance pattern indicates that different layers should contribute differently to the merged model.

Based on these observations, we decompose the global merging coefficient into a hierarchical structure. For each layer l ∈ {1, . . . , L} and component k ∈ {att,mlp}, we define component-specific coefficients through factorization:

where αl ∈ [0, 1] represents the inter-block factor capturing layer-wise importance, and βl,k ∈ [0.5, 1.5] represents the intra-block factor modulating component-specific sensitivity relative to the layer average. The constraint Ek[βl,k] = 1 ensures that intra-block factors act as multiplicative refinements rather than global scaling.

Algorithm 1 Differentiable Hierarchical Fusion (DHF)

• 1: Input: Base model θbase, Reasoning model θreason 2: Calibration set Dcal, Initialization set Dcal 3: Output: Optimized fused model θfused 4: 5: // Stage 1: Delta Computation 6: Compute task vectors: ∆ = θreason - θbase 7: Compute component variances: σ2 att (l), σ2mlp (l) for all l 8: 9: // Stage 2: Outlier-Aware Initialization 10: for each layer l = 1, . . . ,L do 11: Compute activation outlier ratio Rl using Dinit 12: Initialize: αinit l = λmin +λscale · Rl/maxj Rj 13: Initialize: βinit l,att , βinit l,mlp using variance ratios 14: end for 15: Set Λ(0) = {αinit l , βinit l,k } 16: 17: // Stage 3: Differentiable Optimization 18: for iteration t = 1 to Tmax do 19: Sample mini-batch (x, y) ∼ Dcal 20: Construct θfused(Λ(t) ) using Eq. 4 21: Compute loss: L = ℓCE(f(x; θfused), y)+γ1Rinit+ γ2Rsmooth 22: Compute gradients: ∇ΛL 23: Update: Λ(t+1) = Λ(t) -η∇ΛL 24: Project αl ∈ [0, 1], βl,k ∈ [0.5, 1.5] 25: if |L(t) -L(t-5) | < ϵ then 26: Break // Converged 27: end if 28: end for 29: 30: // Stage 4: Final Model Construction 31: Construct final θfused with optimized Λ
• 32: return θfused

The fused weights for component k in layer l are then computed as:W (l,k) fused = W (l,k) base +λl,k · (W (l,k) reason - W (l,k) base ) (5)

This dual-factor formulation provides fine-grained control: αl modulates overall reasoning depth integration across layers, while βl,k adjusts the balance between attention and MLP within each block. Together, they enable the merged model to selectively absorb reasoning logic in high-variance attention components while maintaining the base model's structural efficiency in MLP components.

3.3 Outlier-Aware Initialization

The convergence speed and final quality of differentiable optimization depend critically on initialization. Random or uniform initialization often requires hundreds of iterations and may converge to poor local minima. We leverage recent insights on activation outliers in LLMs [Zhu et al., 2025; Lin et al., 2024] to develop an informed initialization strategy.

Activation Outlier Analysis. We define activation outliers relative to a threshold τ, which is proportional to the mean activation magnitude within a layer. For each layer l, we calculate the outlier ratio Rl as follows:

This metric serves as a proxy for information density; layers with higher outlier ratios typically exhibit more pronounced reasoning-specific patterns where the base and reasoning models diverge. By identifying these critical layers, we can prioritize the preservation of specialized characteristics that are essential for maintaining logical integrity.

Initialization Strategy. We initialize the inter-block factors αl proportionally to the normalized outlier ratios to balance reasoning preservation and structural efficiency:

where λmin and λscale are constants that bound the initial fusion strength. This ensures that layers with high outlier concentrations begin the optimization closer to the reasoning model's weight space, while layers with lower outlier density favor the efficiency of the base model.

For intra-block factors, we initialize based on the relative variance of the components. This allows us to assign more aggressive initial merging coefficients to components that exhibit higher variance, as these typically require more nuanced integration. The initial factors for the attention and feedforward components are formulated as:

where η modulates the sensitivity to variance shifts and ζ maintains a constant expected value for the coefficients within each block. This warm-start strategy guides the differentiable framework toward a stable and high-performing region of the parameter space from the outset.

Multi-Domain Calibration Protocol. To ensure that the learned fusion coefficients generalize across diverse reasoning scenarios, we construct a multi-domain calibration set Dcal. We curate this set by sampling problems from a broad spectrumof domains, includingmathematical reasoning, coding tasks, and logical inference. We employ stratified sampling across multiple difficulty levels to ensure a balanced representation of reasoning patterns. This diversity prevents the optimization from overfitting to specific task patterns and ensures that the resulting fusion generalizes well to unseen scenarios. By focusing on a balanced distribution of data, we achieve a stable hierarchy of layer importance that enhances the overall reliability of the fused model.

3.4 Differentiable Optimization

Unlike traditional merging methods that set coefficients through heuristics or grid search, we treat the complete set of fusion parameters Λ = {αl, βl,k}k∈{att,mlp} l=1,...,L as learnable variables optimized through gradient descent. We define a task-specific loss function over a multi-calibration set Dcal containing diverse reasoning problems:

where ℓCE denotes cross-entropy loss, f(·; θ) represents the model's output distribution, and θfused(Λ) is constructed according to Equation 4. This formulation directly optimizes for task performance rather than proxy metrics [Liu et al., 2018].

The optimization proceeds through gradient descent with respect to the fusion coefficients:

where η is the learning rate. The gradient ∇ΛL is computed through backpropagation by treating the weight construction process as a differentiable operation. Specifically, for a given coefficient λl,k, the gradient is:

This differentiable path ensures that fusion coefficients are adjusted based on their actual contribution to task loss reduction. Layers and components that improve reasoning accuracy when merged more aggressively will receive gradient signals encouraging higher λ values, while those that maintain accuracy with conservative merging will be naturally down-weighted.

To prevent overfitting to the calibration set and ensure stable convergence, we incorporate two regularization terms. First, we add an L2 penalty encouraging coefficients to remain close to initial values:

Second, we enforce smoothness across adjacent layers to prevent abrupt transitions:

The complete optimization objective becomes:

where γ1 and γ2 are regularization coefficients set to 0.01 and 0.005 respectively based on validation performance.

3.5 Overall DHFWorkflow

Algorithm 1 presents the complete DHF procedure, integrating all components into a coherent workflow. The method begins with delta computation and outlier analysis for initialization, followed by iterative optimization of fusion coefficients through gradient descent on the calibration set.

4 Experiments

4.1 Experimental Setup

Evaluations. We evaluate DHF using two prominent model families. For the Qwen family, we merge Qwen2.5-Math-7B (base model optimized for mathematical problem-solving) with DeepSeek-R1-Distill-Qwen-7B (reasoning model with long CoT capabilities). For the LLaMA family, we merge LLaMA-3.1-8B-Instruct with DeepSeek-R1-Distill-LLaMA8B. Evaluation is performed across six diverse reasoning benchmarks: GSM8K [Cobbe et al., 2021] (grade school math), MATH 500 [Lightman et al., 2024] (challenging competition mathematics), Minerva Math [Lewkowycz et al., 2022] (STEM problems), OlympiadBench [He et al., 2024] (olympiad-level problems), CollegeMath [Tang et al., 2024] (undergraduate-level mathematics), and AIME24 (American Invitational Mathematics Examination 2024). These benchmarks span difficulty levels from elementary to expert and cover diverse mathematical domains. We compare DHF against state-of-the-art merging methods, including Task Arithmetic (TA) [Ilharco et al., 2023], TIES-Merging [Yadav et al., 2023], DARE [Yu et al., 2024], Twin Merge [Lu et al., 2024], AIM [Nobari et al., 2025], and ACM [Yao et al., 2025].

Implementation Details. The calibration set Dcal contains 256 examples sampled from multiple domains (Sec. 3.4) to ensure generalizability. We employ an outlier-aware initialization strategy: inter-block factors αl are initialized based on layer-wise outlier ratios Rl with constants λmin = 0.4 and λscale = 0.5, while intra-block factors βl,k are scaled according to the relative variance of component activations. Optimization uses the Adam optimizer with learning rate η = 1 × 10-3 , batch size 8, and runs for a maximum of 100 iterations with early stopping when loss change falls below ϵ = 1×10-4 . Regularization coefficients are set to γ1 = 0.01 and γ2 = 0.005. All experiments use greedy decoding for fair comparison; response length is measured as the number of generated tokens before the final answer. Further details are provided in the Appendix.

4.2 Main Results

Table 1 presents comprehensive results on Qwen2.5-7B models across all six benchmarks. DHF consistently achieves the best balance between accuracy and response length reduction. The results demonstrate several important findings: (1) Superior accuracy-efficiency trade-off: DHF achieves 52.6% average accuracy, outperforming the best baseline (ACM) by 3.1 absolute points while simultaneously providing greater length reduction (55% vs. 47%). This represents a 6.3% relative improvement in accuracy over ACM and demonstrates that differentiable optimization successfully identifies superior points in the accuracy-length trade-off space. (2) Consistent improvements across benchmarks: DHF shows gains across all six benchmarks, with particularly notable improvements on challenging datasets like MATH (+4.4 over ACM), AIME (+3.3), and OlympiadBench (+3.4). This indicates that the learned coefficients effectively preserve complex reasoning capabilities while eliminating verbosity. (3) Bridging the gap to reasoning models: DHF recovers 90.2% of the full reasoning model's accuracy (52.6% vs. 58.3%) while using only 45% of its output length. In contrast, the best baseline (ACM) recovers only 84.9% of accuracy with 53% of length. This demonstrates DHF's effectiveness in maintaining reasoning depth with dramatically improved efficiency. Table 2 shows results on LLaMA-3.1-8B models, confirming that DHF's benefits generalize across architectures. DHF maintains its advantage on LLaMA models, achieving 60.1%

average accuracy (2.8% absolute improvement over AIM) with 58% length reduction. This confirms that the dual-factor framework and differentiable optimization generalize beyond specific model families.

4.3 Ablation Studies

Component Analysis. Table 3 examines the impact of removing key components from the full DHF framework. The results reveal several insights: (1) Intra-block factors are crucial: Removing βl,k (treating all components uniformly within blocks) causes a 2.8-point accuracy drop. This validates our observation that attention and MLP components require different merging strategies due to their distinct weight variance patterns. (2) Inter-block hierarchy matters: Removing αl (using uniform layer importance) degrades accuracy by 4.3 points, the largest single-component impact. This confirms that different layers contribute heterogeneously to reasoning capabilities and require adaptive weighting. (3) Outlier initialization accelerates convergence: Without outlier-aware initialization, convergence requires 2.3× more iterations (156 vs. 68) while achieving slightly lower accuracy. This demonstrates that leveraging activation statistics provides an effective warm start for optimization. (4) Multi-calibration improves generalization: Using a single-domain calibration set reduces average accuracy by 2.1 points, indicating that diversity in calibration data is essential for learning coefficients that generalize across reasoning tasks.

Coefficient Visualization. Figure 2 visualizes the learned coefficients αl and βl,k for Qwen2.5-7B. The visualization reveals clear patterns: (1) αl follows a U-shaped curve with lower values in shallow layers (0.4-0.5), peaking in middle layers (0.8-0.9), then decreasing slightly in final layers (0.60.7). This aligns with the expectation that reasoning logic is primarily encoded in middle-to-deep layers. (2) βl,att consistently exceeds βl,mlp across all layers (average ratio 1.15:0.85), confirming that attention mechanisms carry more reasoning information and benefit from stronger integration.

Hyperparameter Sensitivity. We analyze sensitivity to key hyperparameters: (1) Calibration Set Size. Figure 3

shows performance as calibration set size varies from 64 to 512 examples. Accuracy improves rapidly from 64 (50.8%) to 256 examples (52.6%), then plateaus with minimal gains beyond 256 (52.7% at 512). This demonstrates that DHF is highly data-efficient, requiring only a small calibration set for effective coefficient learning. (2) Learning Rate. Figure 4 examines the sensitivity of our method to different learning rates. The optimal learning rate of η = 1 × 10-3 provides the best balance, achieving high validation accuracy (Fig. 4a) with fast and stable convergence, while also yielding the best final performance (Fig. 4b). Higher rates risk instability and reduced accuracy, while lower rates converge too slowly without gains in final performance.

Computational Efficiency. The results in Table 5 demonstrate that DHF provides a superior efficiency-to-performance ratio compared to existing baselines. While heuristic methods like TIES and DARE are faster, they fail to capture the complex reasoning capabilities of the Long-Thought model, resulting in a significant accuracy gap. Conversely, while fullparameter SFT distillation achieves respectable accuracy, it requires nearly seven times the optimization time and double the VRAM of our approach. Notably, DHF recovers approximately 99.5% of the reasoning accuracy of the Long-Thought model while reducing the output verbosity by 84.3%. This suggests that the 35-minute optimization process successfully identifies the critical "thought-dense" layers, allowing the model to retain high-level logic without the overhead of redundant tokens. The 38 GB VRAM usage further indicates that our implementation is accessible for researchers with limited compute resources, bridging the gap between simple weight averaging and expensive fine-tuning.

Analysis of Length-Accuracy Trade-off. We investigate how different merging strengths affect the accuracy-length trade-off by varying the initialization of αl and re-optimizing. Table 6 shows that DHF achieves a superior Pareto frontier compared to baseline methods. At any target response length, DHF consistently delivers higher accuracy, and conversely, for any target accuracy level, DHF requires fewer tokens. This confirms that differentiable optimization successfully identifies better trade-off points than heuristic methods.

5 Conclusions

We present Differentiable Hierarchical Fusion (DHF) for streamlining long-chain reasoning. DHF addresses the critical challenge of reducing inference verbosity while maintaining reasoning accuracy by treating fusion as a differentiable optimization problem. Our key innovation is the dualfactor adaptive weighting scheme that captures both intrablock component variance and inter-block layer importance, enabling fine-grained control over the merging process. Our DHF achieves significant reductions in response length while improving accuracy. The efficiency and effectiveness of DHF make it particularly suitable for scenarios where both computational cost and response quality matter [Li et al., 2024a; Li et al., 2024c; Li et al., 2024d; Li et al., 2024b].

Acknowledgments

This work is supported by the Shandong Provincial Natural Science Foundation under No. ZR2022QF010, and also supported by the "Jinan Science and Technology Plan 'Open Competition' Project" (Grant No. 202527031)