Diverge to Converge: Mutual Heterogeneous Learning for Robust Pruning

1 Introduction

Deep neural networks (DNNs) have achieved remarkable success across many vision tasks [Saeed et al., 2017; Saeed et al., 2019; Gao et al., 2025], yet their deployment for various edge-based applications [Mostafa et al., 2026; Mostafa et al., 2024; Essameldin et al., 2020; Abdelnasser et al., 2020], on resource-constrained devices remains challenging due to substantial compute and memory demands [Fontanesi et al., 2025; Zha et al., 2025; Bartl et al., 2025]. Structured pruning is a practical compression technique that removes redundant channels/filters to reduce latency and model size [Cheng et al., 2024; Zhong et al., 2025]. However, this efficiency gain Single-model fine-tuning Adv Corr

8

6

2

4

0

2

6

3.0

1.5

Model

3.0 1.5 0.0 1.5 3.0

1.5

3.0

8

6

4

2

0

2

4

6

is often achieved by sacrificing the network's resilience. [Piras et al., 2025] As the model becomes leaner, it often suffers significant robustness degradation: they become substantially more vulnerable to both adversarial perturbations and natural corruptions than their unpruned counterparts. This trade-off poses a severe challenge for safety-critical applications, where efficiency cannot come at the expense of trustworthiness.

To mitigate this degradation, current approaches predominantly rely on adversarial training on pruned models, robustness-aware pruning, or specialized fine-tuning [Zhou et al., 2025; Saxena et al., 2024; Zhang et al., 2023]. However, we argue that these methods fundamentally follow a singlemodel paradigm, which is limited by their dependence on a single optimization trajectory and struggles to simultaneously recover multiple robustness properties after compression. As illustrated in Figure 1(a), fine-tuning on non-convex landscapes restricts a lone network to a single trajectory, often resulting in sharp local minima. Crucially, such constrained optimization on one pruned model fails to simultaneously maximize both internal feature smoothness and external boundary separation, inevitably yielding a suboptimal trade-off and recovery in multiple robustness properties.

To bridge the gap between efficiency and robustness, we propose a paradigm shift from single-trajectory optimization to a collaborative mutual learning framework. Our key insight is diverge to converge, instead of forcing one pruned model to be universally robust, we recover robustness through designed complementarity. As visualized in Figure 1(b), we instantiate two heterogeneous models driven by comple-

0.0

mentary objectives. For "diverge", two pruned models are intentionally driven toward different robustness behaviors, their failure regions can become less correlated. For "converge", with proper mutual supervision, they can collaboratively cover each other's vulnerabilities and yield stronger robustness. This strategy effectively expands the search space and synthesizes multifaceted robustness into a single model without incurring extra inference costs.

Concretely, we propose Mutual Heterogeneous Learning (MHL), a dual-model co-training framework for robust pruning with single-model inference, strictly adhering to edge application constraints. For "diverge" in MHL, heterogeneous objectives have two complementary mechanisms: layer-wise Lipschitz regularization (feature smoothness under local perturbations) to enforce intermediate feature smoothness, and an adaptive margin objective (difficulty-aware logit margin) for difficulty-aware boundary separation. Crucially, this design bridges the gap between internal feature stability and external decision separability, ensuring that the pruned model is not only insensitive to input noise but also decisive in its classifications. For "converge" in MHL, we employ entropy-based mutual distillation under a dynamic schedule. By adaptively regulating the optimization focus across training stages, this strategy actively shifts the trajectory from initially exploring diverse feature subspaces to consolidating a unified consensus, thereby restoring multiple robustness properties while strictly maintaining inference efficiency by deploying only a single pruned model at test time.

We conduct extensive experiments under structured pruning on four benchmarks, evaluating clean accuracy (CIFAR10/100), adversarial robustness (FGSM/PGD/PGD-20), and corruption robustness (CIFAR-10/100-C). Our results show MHL outperforming post-pruning baselines across robustness metrics while maintaining competitive clean accuracy. Ablation studies further validate the necessity of heterogeneity, mutual distillation, and feature diversity regularization.

Our main contributions are summarized as follows:

• We propose MHL, a novel structured pruning framework that shifts the robust pruning paradigm from single-trajectory fine-tuning to a "diverge-to-converge" strategy. This approach avoids the dependency on a single optimization trajectory, mitigates local optima, and synthesizes a robust consensus from diverse optimization directions.
• We instantiate heterogeneity via twomechanisms: layerwise Lipschitz regularization to enforce internal feature stability, and an adaptive margin objective to ensure external decision separability. We synthesized these via entropy-based mutual distillation and feature diversity regularization that dynamically transitions from subspace exploration to robust consensus.
• We demonstrate consistent improvements on adversarial robustness and corruption robustness under structured pruning, with ablations confirming the role of each component. MHL improves adversarial robustness by an average of 5% and corruption robustness by 2.6% over prior methods.

2 RelatedWorks

2.1 Adversarial TrainingMethods

Adversarial Training (AT) [Madry et al., 2018] is the most effective defense against adversarial examples [Goodfellow et al., 2014]. To improve the robustness-accuracy tradeoff, early variants like TRADES [Zhang et al., 2019] and MART [Wang et al., 2019] introduced regularization-based objectives. More recently, the focus has shifted to featurespace manipulations, such as explicit feature disentanglement in AFD [Zhou et al., 2025] and asymmetric representation regularization in AR-AT [Waseda et al., 2025]. Despite their effectiveness, these methods train dense models, which limits their deployment efficiency. Our approach extends adversarial robustness to the pruned model regime through mutual heterogeneous learning.

2.2 Robust Neural Network Pruning

Standard pruning strategies [Han et al., 2015; Filters'Importance, 2016; Liu et al., 2017; Frankle and Carlin, 2019], though effective for clean accuracy, suffer from severe robustness degradation [Ye et al., 2019; Gui et al., 2019]. To address this, robustness-aware pruning integrates adversarial objectives into the compression process. Prominent approaches focus on learning robust importance scores or saliency rankings [Sehwag et al., 2020; Zhao and Wressnegger, 2024; Lee et al., 2022]. Alternatively, some works explore dynamic sparse training [Chen et al., 2022] or weight reparameterization [Li et al., 2023] to mitigate robust overfitting and maintain representational capacity. However, these methods strictly adhere to a single-model paradigm, limiting their ability to escape sharp local optima or recover multifaceted robustness. Our MHL overcomes this via heterogeneous mutual learning, effectively synthesizing complementary robustness properties without incurring additional inference costs.

2.3 Ensemble Methods and Architectural Defenses

Ensemble methods leverage diversity to mitigate attack transferability [Kariyappa and Qureshi, 2019; Pang et al., 2019]. Recent approaches maximize this via explicit diversity regularization [Yang et al., 2020; Jung and Song, 2025] or layer-wise orthogonalization [Ebrahimpour Boroojeny et al., 2025]. However, these strategies typically incur prohibitive inference costs. Parallel efforts explore architectural modifications, such as input filtering layers [Suresh et al., 2025] or neuron-level monosemanticity constraints [Zhang et al., 2024]. While promising, these architectural defenses are often specific to certain backbones and lack generalizability.

Unlike ensembles that multiply inference costs, MHL adopts a Two-model training, single-model deployment strategy. By distilling heterogeneous experts via a diverge-toconverge trajectory, we condense multifaceted robustness into a single pruned model, achieving ensemble-level resilience with zero additional overhead.

3 Method

3.1 Preliminaries and Problem Setting

Let D = {(xi, yi)}Ni=1 denote a labeled dataset with inputs x ∈ Rd and labels y ∈ {1, . . . ,C}. A classifier fθ : Rd → RC parameterized by θ produces logits z = fθ(x), and the predictive distribution is p = softmax(z). We consider structured pruning-removing entire filters or channelswhich yields models directly accelerable on commodity hardware without specialized sparse libraries.

Three-stage pipeline. Following the standard pipeline (Pre-train → Prune → Fine-tune), our work focuses exclusively on the final stage: given a pruned checkpoint ˜ fθ = P(fθ), we aim to maximize its adversarial and corruption robustness without increasing inference cost.

3.2 Mutual Heterogeneous Learning (MHL)

We propose Mutual Heterogeneous Learning (MHL), a cotraining framework that recovers robustness of pruned models through designed complementarity. The core insight is diverge to converge: rather than forcing a single compact model to satisfy all robustness requirements simultaneously, we train two pruned models with deliberately different inductive biases, then leverage their complementary strengths through mutual supervision.

Two-model mutual training, single-model deployment. Starting from the same pruned-and-finetuned checkpoint ˜ fθ , we create two branches with identical architecture: ˜ fθ1 and ˜ fθ2 . Both models share the same pruned structure (e.g., 50% filters removed) but diverge during training due to heterogeneous regularization. Crucially, only one model is retained at deployment-the better-performing branch on a held-out validation set-thus preserving the efficiency benefits of pruning with zero additional inference cost.

Base classification objective. Both models are supervised with ground-truth labels via cross-entropy:

MHL augments L(1) CE + L(2) CE with three synergistic components: (i) heterogeneous constraints to induce complementary robustness behaviors, (ii) entropy-gated mutual distillation to transfer reliable knowledge, and (iii) feature diversity regularization to prevent representation collapse.

3.3 Heterogeneous Constraints

The key to MHL is imposing different regularizers on the two branches: layer-wise Lipschitz on ˜ fθ1 (encouraging smooth internal representations) and adaptive margin on ˜ fθ2 (enforcing confident logit separation). This asymmetric design is intentional-the two constraints target orthogonal aspects of robustness, producing complementary failure modes that can be exploited via mutual learning.

Layer-wise Lipschitz Regularization

Adversarial perturbations exploit the sensitivity of neural network activations. We encourage ˜ fθ1 to have stable intermediate features under small input perturbations. Let ϕ(1) ℓ (x) denote the feature map at stage ℓ of a four-stage ResNet (ℓ ∈ {1, 2, 3, 4}). For each input x, we sample K isotropic Gaussian perturbations δk ∼ N(0, ρ2 I) and penalize feature displacement across all layers:

where wℓ = (0.25, 0.25, 0.25, 0.25) are equal layer weights, K = 3 samples, and ρ = 0.1 is the perturbation radius. Importantly, the center features ϕ(1) ℓ (x) are computed without gradients (stop-gradient), so only the perturbed path contributes to the regularization. Compared to end-to-end Lipschitz bounds [Ciss´e et al., 2017], layer-wise control offers finer granularity, particularly valuable for pruned networks where reduced redundancy amplifies sensitivity.

Adaptive Margin Regularization

To complement the representation-level smoothness of ˜ fθ1 , we enforce decision-level robustness on ˜ fθ2 via logit margin. Given logits z(2) = ˜ fθ2 (x), let zy be the correct-class logit and zmax = maxc̸=y z(2) c the largest incorrect logit. We define sample difficulty as

and set a sample-adaptive target margin:

where m0 is a base margin hyperparameter (default m0 = 10). The loss penalizes insufficient margins:

This adaptive design imposes larger margins on easy (highconfidence) samples while relaxing constraints on hard ones, avoiding gradient explosion on inherently ambiguous inputs and stabilizing optimization.

3.4 Entropy-GatedMutual Distillation

Heterogeneous constraints alone may lead to excessive divergence without transferring complementary knowledge. We bridge the two branches via entropy-gated mutual distillation, where the more confident model on each sample acts as the teacher.

For model i, we measure prediction uncertainty via entropy:

Lower entropy indicates higher confidence. We define binary teaching indicators: I1→2 = I[H(p(1) ) < H(p(2) )] and I2→1 = 1 - I1→2. Using temperature τ (default τ = 3) and softened probabilities p(i) τ = softmax(z (i) /τ), the distillation losses are:

The teacher's logits are detached (stop-gradient) to prevent symmetric updates. The total distillation loss is LKD = L1←2 KD +L2←1 KD .

This sample-wise gating mechanism differs from standard mutual learning [Zhang et al., 2018], which transfers knowledge bidirectionally on all samples. Our entropy gating ensures that only reliable soft targets are transferred, reducing error propagation and stabilizing co-training dynamics.

3.5 Feature Diversity Regularization

Mutual distillation encourages output agreement, but may inadvertently collapse the two branches to identical internal representations, defeating the purpose of heterogeneous training. We introduce a feature diversity loss to explicitly maintain representational differences.

For each stage ℓ, we extract global-average-pooled features ¯ ϕ(i) ℓ (x) ∈ Rdℓ and ℓ2-normalize them to unit vectors ˆ ϕ(i) ℓ (x). The diversity loss penalizes high cosine similarity:

where αℓ = (0.5, 1.0, 1.0, 0.5) are layer weights that emphasize middle layers (which capture most semantic content). Minimizing Ldiv encourages orthogonal feature directions between the two branches, preserving complementarity throughout training.

3.6 Overall Objective and Optimization

Combining all components, the MHL training objective is:

We optimize Eq. (10) end-to-end using SGD with momentum (0.9) and cosine annealing learning rate schedule. Gradient clipping (max norm 1.0) is applied to both branches for training stability. Default hyperparameters are λLL = λAM = 0.1, λKD = 0.5, λdiv = 0.1.

The optimization concludes by discarding the auxiliary branch and retaining only the primary pruned model for deployment, ensuring zero additional inference cost compared to standard pruning.

Design rationale. The four loss terms form a balanced system: (i) LLL and LAM induce complementary inductive biases-one enforces representation smoothness, the other enforces decision confidence; (ii) LKD transfers reliable knowledge from the stronger model to the weaker one on each sample; (iii) Ldiv prevents the distillation from collapsing both models to identical solutions.

3.7 Analysis of Robustness Components

We analyze why the two heterogeneous training components in our implementation (layer-wise Lipschitz constraints and adaptive margin) are complementary, and how mutual supervision further enlarges the robust radius of each pruned model. Crucially, while this analysis establishes the static theoretical lower bound, our dynamic scheduling which is driven by the feature diversity loss, serves as the practical mechanism to prevent representational collapse, ensuring that the final model effectively approximates this optimal robustness limit.

Theoretical Motivation and Instantiation. The certifiable robust radius of a classifier is lower-bounded by the ratio of its decision margin to its Lipschitz constant: rf(x) ≥ mf (x) L [Tsuzuku et al., 2018]. This inequality reveals two complementary levers for robustness enhancement: increasing the numerator mf(x) (via boundary separation) and decreasing the denominator L (via feature smoothness). To explicitly target these complementary levers, we instantiate MHL by training two pruned models {f (1) , f (2) } jointly with heterogeneous objectives. Specifically, we assign distinct regularizers to each model, one focusing on Adaptive Margin to maximize the numerator, and the other on Layer-wise Lipschitz Regularization to minimize the denominator. So, we can decouple the complex robustness objective into two manageable sub-problems.

Layer-wise Lipschitz constraints as local smoothing (denominator control) Formally, for a network f(x) = Wlϕ(...W1x), the global Lipschitz constant L is upperbounded by the product of layer-wise spectral norms: L ≤ ∏li=1 ∥Wi∥2.

Proposition 1: Minimizing the layer-wise spectral norms explicitly decreases the global Lipschitz constant L. According to the robust radius bound rf (x) ≥ mf (x)/L, this operation effectively performs "denominator control," enlarging the certified robust radius by suppressing feature amplification.

Mutual supervision transfers the two gains across models. While Layer-wise Lipschitz constraints create diverse models (one with high margin mf , one with low Lipschitz L), we need to unify these gains. We employ an entropybased mutual distillation loss LKD.

Proposition 2 (Robustness Transfer). We employ an entropy-gated mutual distillation loss LKD that dynamically designates the lower-entropy agent as the teacher. By minimizing the KL divergence KL(p(student) ||p(teacher) ) directionally, MHL explicitly transfers the superior geometric property (e.g., larger margin or smoother features) from the stronger agent to the weaker one, thereby enlarging the robust radius of the ensemble consensus.

Synergy: robust radius of the smooth branch increases beyond using a single component. Finally, we analyze how the interaction between the two agents surpasses the limits of independent training. While the smooth branch minimizes the denominator L, it risks collapsing the numerator mf (decision margin) due to over-regularization.

Proposition 3 (Synergy of Heterogeneity). Let r(1) base be the certified robust radius of f (1) when trained in isolation. Under MHL, the effective radius of the smooth agent r(1) MHL

satisfies:

where ∆m2→1 > 0 represents the margin gain transferred from the margin-aware teacher f (2) via entropy-gated mutual distillation, and L(1) is the low Lipschitz constant explicitly maintained by f (1) .

Remark 1: This inequality mathematically formalizes the "Diverge-to-Converge" benefit. Crucially, the explicit Lipschitz constraint on f (1) acts as a geometric anchor, ensuring the denominator L(1) remains suppressed even as the numerator increases via distillation. This allows the smooth agent to "borrow" the margin advantage (∆m) from f (2) without compromising its own stability, achieving a robust radius impossible for either model in isolation.

Remark 2 (Diverge-Converge Principle). Simultaneously maximizing the margin m (numerator) and minimizing the Lipschitz constant L (denominator) is often an ill-posed optimization problem for a single sparse network with limited capacity. MHL circumvents this by decoupling these conflicting objectives across heterogeneous agents and subsequently aggregating their specific advantages (high separability from f (2) and high smoothness from f (1) ) via mutual supervision, effectively realizing the theoretical synergy described in Proposition 2.

4 Experiments

Datasets. Following the edge application constraint in [Ebrahimpour Boroojeny et al., 2025], we evaluate the performance of MHL on CIFAR-10 and CIFAR-100, including clean and adversarial samples. For corruption robustness, we also evaluate CIFAR-10-C and CIFAR-100-C, and illustrate category-wise and average accuracy.

Implementation Details. All methods follow the same pipeline: dense models are pretrained first, followed by structured pruning and fine-tuning for 20 epochs to obtain a common initialization, and finally trained for 50 epochs using MHL or a baseline training strategy. We consider three structured pruning rates: 25%, 50%, and 75%. All methods at each pruning rate share the same initialization. We use the ℓ∞ threat model with perturbation budget ϵ = 1/255 and report clean, FGSM, PGD-10/20 accuracy.

4.1 Baselines

We compare against robust training and pruning-related baselines, all the methods are implemented in the final training phase, same as our MHL. The baselines include vanilla finetuning (FT), AT [Madry et al., 2018], SWA, ASAM [Kwon et al., 2021], AdaSAP [Bair et al., 2023], AFD [Zhou et al., 2025], EED [Jung and Song, 2025], ANF [Suresh et al., 2025], LOTOS [Ebrahimpour Boroojeny et al., 2025], and MONO [Zhang et al., 2024]. For methods originally designed for ensembles or different training regimes, we use their adapted variants under the same pipeline. Most baseline use single-model inference, for ensemble methods (EED, LOTOS), we set 3 sub-models for their evaluation.

4.2 Main Results on CIFAR-10/100

Results on CIFAR-10. Table 1 shows that MHL consistently improves adversarial robustness of pruned models while keeping clean accuracy competitive. At 50% pruning, MHL boosts PGD-20 accuracy from 51.29% (FT) and 60.28% (MONO) to 71.13%, indicating that complementary constraints plus mutual learning recover robustness properties that single-model fine-tuning fails to restore. Compared to optimization-centric baselines (SWA/ASAM), MHL yields markedly higher PGD robustness. While adversarial training (AT) achieves stronger attack accuracy, it incurs a substantial clean-accuracy drop, whereas MHL retains high clean performance (92.81%) and delivers a stronger robustness-accuracy trade-off under structured pruning. Under more aggressive compression, MHL still improves PGD-20 over FT by a large margin, demonstrating robustness recovery in the most challenging regime.

Results on CIFAR-100. Table 2 further confirms the benefit of MHL on a harder dataset. At 50% pruning, MHL improves PGD-20 from 19.43% (FT) and 27.37% (MONO) to 32.63%, and at 75% pruning it more than doubles FT's PGD-20 accuracy. These gains are achieved with substantially higher clean accuracy than AT, suggesting that MHL is particularly effective for robustness recovery when pruning amplifies capacity constraints.

4.3 Corruption Robustness on CIFAR

Results on CIFAR-10-C. Table 3 shows that MHL achieves strong average corruption robustness across pruning rates and exhibits clear category-wise complementarity. In particular, MHL substantially improves robustness on Weather and Digital corruptions, which aligns with our design that combines feature-space smoothing and decision-boundary shaping. Although some baselines (e.g., AT) can be stronger on Noise and Blur, MHL maintains competitive averages and better overall balance when considering both adversarial and corruption robustness under single-model deployment.

Results on CIFAR-100-C. As shown in Table 4, MHL yields consistent improvements over fine-tuning and robustness recovery baselines, especially under heavier pruning. For instance, at 75% pruning, MHL improves the average corruption accuracy by +3.78 points over FT (41.00 vs. 37.22), demonstrating that heterogeneous co-training remains beneficial when the pruned model's robustness is severely degraded.

4.4 Ablation Studies

Analysis of pruning methods. Table 5 shows that MHL is robust to the choice of structured pruning criterion; all three pruning rules yield similar clean and corruption accuracy, indicating that MHL is the dominant factor for robustness recovery. Meanwhile, the pruning criterion can affect adversarial robustness under heavy compression: at 25% pruning, Taylor pruning yields higher PGD-20 than Magnitude-L1 and Magnitude-L2; at 75% pruning, Taylor again achieves the best PGD-20 while maintaining strong corruption robustness. Given this mild sensitivity, we adopt Magnitude-L1 as the default for its simplicity and stability, and emphasize that MHL consistently improves robustness across pruning protocols.

Analysis of mutual learning schemes. Table 6 validates that how we couple the heterogeneous branches is critical. Our default entropy-gated distillation achieves the best overall balance (Clean 92.81%, Corrupt 76.84%, PGD20 71.13%), outperforming alternative couplings such as confidence gating and alignment. Notably, disagreementdriven coupling can boost corruption robustness (78.51%) but severely hurts adversarial robustness, suggesting that encouraging output mismatch may increase robustness to distribution shift while weakening worst-case robustness. In contrast, consistency coupling degrades both clean and attack performance, indicating that overly strong agreement can suppress the benefits of heterogeneity. These results suggest that entropy gating transfers only reliable soft targets, enabling effective knowledge sharing while preserving complementary robustness behaviors.

5 Conclusion

In this work, we identified the reliance on a single optimization trajectory as the fundamental bottleneck in current robust pruning methods, which inevitably leads to local minima and suboptimal robustness trade-offs. To bridge this limitation, we proposed Mutual Heterogeneous Learning (MHL), a novel framework governed by the core philosophy of "diverge to converge". By orchestrating heterogeneous models to explicitly decouple robustness into internal feature stability and external decision separability, and subsequently synthesizing them via dynamic consensus, MHL successfully recovers multiple robustness properties. Extensive experiments on 4 benchmarks and under diverse adversarial attacks demonstrate that MHL consistently outperforms strong single-model baselines, yielding substantial gains in both adversarial robustness (+5%) and corruption robustness (+2.6%) while maintaining competitive clean accuracy. Crucially, our approach demonstrates that leveraging mutual heterogeneity learning is a potent strategy to enhance the trustworthiness of compact models while maintaining singlemodel inference efficiency.

Acknowledgments

Zikai Zhang (zikaiz2@cmu.edu) and Yidong Li (ydli@bjtu.edu.cn) are the corresponding authors. This work was supported in part by ARG02-0429-240389 from the Qatar National Research Fund/Qatar Research, Development and Innovation Council; the National Science Foundation of China under Grant U2268203, 62306027, 62402031 and 62372436; the Beijing Natural Science Foundation 4264109; the ZTE Corporation project Sim-to-Real Technology Exploration for Robotics (K25L01090); and the Beijing Nova Program (No. 20240484620).