Manifold-Constrained Adversarial Training for Long-Tailed Robustness via Geometric Alignment

1 Introduction

Deep neural networks have achieved remarkable success in visual recognition tasks, yet their vulnerability to adversarial perturbations remains a fundamental concern. Among existing defenses, adversarial training, formulated as a minmax optimization problem, is widely regarded as one of the most effective and principled approaches. However, the evaluation of adversarial robustness has largely focused on balanced benchmarks, whereas real-world data are often characterized by long-tailed class distributions [Wu et al., 2021; Zhang et al., 2023; Zhang et al., 2025]. Under such imbalance, tail classes not only suffer from degraded clean accuracy, but also exhibit disproportionately weaker adversarial robustness, raising serious concerns about the reliability and fairness of robust models in practice.

Motivated by this gap, recent studies have begun to investigate adversarial robustness under long-tailed distributions. RoBal [Wu et al., 2021] pioneers this line of work by introducing margin rebalancing combined with classifier adjustment. Subsequent methods further improve tail robustness through loss reweighting, multi-stage training strategies, or class-aware regularization [Ren et al., 2020; Li et al., 2021; Liu et al., 2022; Zhang and Feng, 2024; Zhang et al., 2022; Ahn et al., 2023; Du et al., 2023; Xu et al., 2021; Li et al., 2023; Yu-Hang et al., 2025; Yue et al., 2024; Gupta et al., 2025]. Despite the progress achieved, most existing approaches primarily operate at the level of loss design or optimization heuristics, and do not explicitly regulate the geometry of learned representations or the semantic validity of adversarial examples.

∗Corresponding author

In this paper, we attribute the failure of adversarial training under long-tailed distributions to two closely coupled mechanisms: imbalance-induced geometric misalignment and offmanifold adversarial drift (Figure 1). First, head-classdominated optimization distorts the representation geometry, compressing inter-class margins associated with tail classes and rendering their decision boundaries fragile and unstable. Second, due to the scarcity of tail samples, unconstrained adversarial optimization is prone to exploit low-density and semantically unsupported regions of the feature space, diverting robustness away from the true data support. Together, these effects result in severe robust-margin collapse and unreliable predictions for tail classes.

Several complementary approaches based on knowledge transfer, such as long-tailed adversarial self-distillation [Cho et al., 2025], attempt to alleviate data scarcity at the decision level. While effective to some extent, these methods remain largely orthogonal to the representation-space issues discussed above, as they neither explicitly correct geometric misalignment nor constrain adversarial examples to lie within semantically meaningful regions.

We argue that achieving adversarial robustness under longtailed distributions fundamentally requires addressing both the geometry of the learned decision space and the location of adversarial examples. To this end, we propose Manifold-Constrained Adversarial Training (MCAT), a unified framework grounded in a twofold geometric principle. First, MCAT constrains adversarial perturbations to remain close to class-conditional semantic manifolds in feature space, ensuring that robustness is learned within high-density and semantically valid regions. Notably, although tail classes are sparsely sampled in pixel space, their representationspace structure is substantially more regular and lowerdimensional, which makes such manifold constraints feasible even with limited tail data. Second, MCAT promotes balanced inter-class geometry by aligning classifier weight vectors toward a simplex Equiangular Tight Frame (ETF) structure [Papyan et al., 2020], thereby restoring margin-balanced decision boundaries. Our theoretical analysis shows that manifold constraints effectively control robust risk within the semantic support, while geometric alignment induces provable lower bounds on adversarially robust margins. By jointly enforcing semantic validity and geometric alignment, MCAT stabilizes adversarial decision boundaries and substantially improves robustness for both head and tail classes.

Our main contributions are summarized as follows:

• We identify two fundamental mechanisms underlying the degradation of adversarial robustness under long-tailed distributions: geometric misalignment and off-manifold adversarial drift.
• We propose MCAT, a unified adversarial training framework that integrates manifold-constrained perturbations with ETF-inspired geometric alignment.
• We provide theoretical guarantees that link geometric separation to adversarially robust margins and show that manifold constraints control robust risk on semantic support.
• Extensive experiments demonstrate consistent improvements in overall, balanced, and tail-class adversarial robustness.

2 Preliminaries

Let D = {(x, y)} denote a long-tailed dataset with class prior πy, where x ∈ Rd and y ∈ {1, . . . ,C}. Let fΘ : Rd → RC be a classifier parameterized by Θ, and let φΘ(x) ∈ Rm corresponds to the output of the feature extractor before the final linear classification layer of f. The final linear classifier is parameterized by weights W ∈ RC×m, whose y-th row wy corresponds to class y. We denote by sΘ(x) = fΘ(x) the logit vector, where sk(x) is the logit associated with class k.

Let ℓ(·, ·) denote a standard classification loss, such as crossentropy. Expectations E(x,y)∼D[·] are taken with respect to the empirical training distribution induced by D. We consider an ℓ∞ threat model: Bϵ(x) = {x′ | ‖x′ - x‖∞ ≤ ϵ}. The robust risk is defined as

3 Method

3.1 Overview

As illustrated in Figure 2, under long-tailed distributions, standard adversarial training tends to shape decision boundaries in low-density regions while being dominated by headclass geometry. This leads to unstable decision boundaries and severely reduced margins for tail classes. MCAT addresses these issues by jointly enforcing the semantic validity of adversarial examples and a balanced geometry of the decision space.

Concretely, MCAT consists of two complementary components. First, adversarial perturbations are constrained to remain close to class-conditional semantic manifolds in the feature space (Section 3.2), thereby guiding adversarial optimization toward high-density and semantically meaningful regions. Second, the classifier weight vectors are regularized toward a simplex Equiangular Tight Frame (ETF) structure (Section 3.3), which encourages uniform angular separation between classes. These two components are combined into a unified min-max training objective (Section 3.4) and optimized using a manifold-aware PGD procedure (Section 3.5).

3.2 Class-Conditional Semantic Manifolds in Feature Space

We assume that features of each class y concentrate around a low-dimensional semantic support in representation space, denoted as My. Rather than explicitly recovering My, we employ a class-conditional generator Gy as a proxy to characterize off-manifold deviation.

Let z ∼ N(0, I) be a latent code. Each generator Gy : Rk → Rm is a lightweight MLP mapping latent codes to the feature space, z ∼ N(0, I), ˜ φy = Gy(z). The generators {Gy} are pretrained using features extracted by a classifier fΘ by minimizing

Learning Gy directly in representation space substantially reduces the intrinsic complexity of tail classes compared to pixel-space generation, making class-conditional manifold approximation feasible even with limited samples. After pretraining, all generators are frozen throughout adversarial training. Although φΘ continues to evolve during robust optimization, its class-conditional structure changes gradually, allowing Gy to act as a stable semantic reference that regularizes adversarial drift rather than enforcing exact reconstruction.

We measure off-manifold deviation of an embedding u = φΘ(x) by

Algorithm 1 MCAT Training

Require: Dataset D, classifier fΘ, generators {Gy}, steps T , budget ϵ, step size η, weights λ, β

• 1: for each iteration do 2: Sample mini-batch {(xi, yi)}ni=1 3: for each sample i do 4: Initialize δi,0 ∼ [-ϵ, ϵ] 5: for t = 0 to T - 1 do 6: Update δi,t+1 according to Equation (6) 7: end for 8: xadv i ← xi + δi,T 9: end for 10: Update θ according to Equation (5) 11: end for

3.5 Manifold-Constrained Inner Maximization

The inner maximization is solved using Manifold Supported PGD (MS-PGD):

where Π‖δ‖∞≤ϵ denotes projection by clipping. MS-PGD preserves the standard ℓ∞ threat model while biasing adversarial search toward semantically supported regions.

3.6 Training Algorithm

Algorithm 1 summarizes the MCAT training procedure.

4 Theoretical Analysis

We provide two complementary results that connect MCAT to (i) margin-based robustness induced by balanced representation geometry, and (ii) robust-risk control through suppressing off-manifold adversarial drift. Proofs are deferred to the appendix A.

where the inner minimization is approximated by Tz steps of gradient descent on z, warm-started from a per-sample cache. We report a sensitivity analysis with respect to Tz in Appendix Table 6, and further verify the validity of frozen generators by tracking reconstruction error over training epochs, which remains stable across classes, including tail classes (Appendix Figure 10a).

3.3 ETF-Inspired Geometry Regularization

To counteract imbalance-induced geometric compression, we regularize classifier weights W toward a simplex Equiangular Tight Frame (ETF) structure by penalizing deviations of the Gram matrix:

where α and β are scalar parameters, I is the identity matrix, and 1 is the all-ones vector.

This regularizer promotes approximately equal-norm and equiangular classifier weights, thereby enlarging and stabilizing the minimum inter-class angle θmin. By Theorem 1, a larger θmin directly implies a larger certifiable robust margin. Since the simplex ETF maximizes the minimum pairwise angle among class vectors, it represents an optimal geometry for robustness under fixed dimensionality. Under long-tailed adversarial training, head-dominated optimization distorts balanced geometry, which our regularization counteracts to prevent tail-class margin collapse.

3.4 Unified Objective

We combine manifold constraints (Eq. (3)) and geometric regularization (Eq. (4)) into a single objective:

where λ controls semantic consistency and β controls geometric balancing.

4.1 Theorem 1: Geometric Separation Implies Robust Margin Lower Bound

We assume ‖φΘ(x)‖2 = 1 for all x, and that the feature map φΘ is L-Lipschitz under ℓ∞ perturbations, i.e., ‖φΘ(x+δ)- φΘ(x)‖2 ≤ Lϵ for all ‖δ‖∞ ≤ ϵ. Let wy denote the classifier weight vector for class y, and define the minimum inter-class angle as

Theorem 1 (Robust Margin from Geometric Separation). If ϵ < sin(θmin/2)/L, then the predicted label of x remains invariant to all perturbations in Bϵ(x).

Corollary 1 (Sample-wise Robust Radius). Let sy(x) = w⊤ y φΘ(x) denote the logit of the true class y, and define the logit margin γ(x) = sy(x) - maxk̸=y sk(x). Then the sample-wise robust radius satisfies r(x) ≥ γ(x) 2L .

Remark 1. Theorem 1 shows that adversarial robustness is governed by the minimum inter-class angle θmin. In particular, ETF maximizes θmin among normalized class weights, and thus attains the largest robust margin implied by the theorem. Theorem 1 establishes a sufficient robustness condition governed by the minimum inter-class angle, while Corollary 1 yields a sample-dependent lower bound via the logit margin. Together, they illustrate why geometry matters in long-tailed robust training: when head-dominated updates compress tail margins, the guaranteed robust radius for tail samples can quickly vanish.

4.2 Theorem 2: Manifold Constraint and Robust Risk Control

Theorem 2 (Manifold-Constrained Training Controls Robust Risk). Assume that, for each class y, the data distribution is supported on a semantic manifold My, and that regions far fromMy have negligible probability mass. Then, for the MCAT objective RMCAT(θ),

Remark 2. Theorem 2 shows that robust risk is controlled by the MCAT objective up to a residual term O(λ-1 ), which arises from off-manifold adversarial drift under finite λ. While such drift cannot be fully eliminated, it is the uncontrolled drift-common in standard adversarial training-that is especially harmful for tail classes due to sparse sampling and weak manifold support. The manifold penalty in MCAT significantly suppresses this drift and confines it to a limited range, preventing large semantic deviations and spurious decision boundaries in low-density regions, without restricting the adversary's ℓ∞ budget.

5 Experiments

5.1 Goals and Research Questions

We evaluate MCAT on standard long-tailed adversarial robustness benchmarks to answer the following research questions.

• RQ1 (Overall robustness): Does MCAT improve adversarial robustness on long-tailed data compared with standard adversarial training and long-tailed robust baselines?
• RQ2 (Tail and balanced robustness): Does MCAT improve robustness for tail classes and class-balanced metrics without sacrificing head-class performance?
• RQ3 (Component contribution and sensitivity): How do the individual components of MCAT (manifold constraint and geometric alignment) and their associated hyperparameters contribute to robustness under long-tailed distributions?
• RQ4 (Mechanism verification and theory consistency): Can we empirically verify the two failure mechanisms in Figure 1 (geometry compression and off-manifold adversarial drift), and observe empirical trends consistent with our theoretical analysis?

5.2 Experimental Settings

Datasets and Long-Tailed Construction

Benchmarks. We conduct experiments on CIFAR-10-LT, CIFAR-100-LT, and Tiny-ImageNet-LT following prior longtailed robustness protocols [Wu et al., 2021; Yue et al., 2024; Cho et al., 2025].

Imbalance ratio (IR). Given a dataset with C classes, we construct a long-tailed training set by exponentially decaying the number of samples per class. Let nmax be the maximum class size (head) and nmin the minimum class size (tail), then IR = nmax/nmin. We evaluate multiple imbalance levels, e.g., IR ∈ {10, 20, 50, 100}, and report the default setting for each benchmark consistent with prior work [Wu et al., 2021; Yue et al., 2024; Cho et al., 2025].

Baselines

Standard adversarial training baselines. We consider commonly used adversarial training methods including TRADES [Zhang et al., 2019], MART [Wang et al., 2020], AWP [Wu et al., 2020], LAST-AT [Jia et al., 2022] as widely adopted in recent studies [Yue et al., 2024].

Long-tailed adversarial training baselines. We compare to representative long-tailed robustness methods such as RoBal [Wu et al., 2021], REAT [Li et al., 2023], TAET [Yu-Hang et al., 2025], and long-tailed adversarial self-distillation [Cho et al., 2025]. We follow the official implementations or reproduce their reported settings under a unified protocol whenever possible.

MCAT ablations. To isolate the effects of each component, we evaluate: (i) Base AT (or the strongest common baseline), (ii) Base AT + manifold constraint only, (iii) Base AT + geometric alignment only, (iv) MCAT (full) (manifold constraint + geometric alignment + MS-PGD).

Architectures and Training Details

Backbones. For CIFAR-10/100-LT, we use ResNet-18 as the default backbone and optionally include WideResNet-3410 for stronger capacity comparisons, following prior longtailed robustness evaluations [Cho et al., 2025; Yue et al., 2024]. For Tiny-ImageNet-LT, we use a standard residual backbone (e.g., PreActResNet-18) consistent with previous protocols [Cho et al., 2025].

Adversarial training. Unless otherwise specified, we consider the ℓ∞ threat model with perturbation budget ϵ (e.g., ϵ = 8/255 on CIFAR). For training, we use a multi-step PGD inner maximization (e.g., T = 10 steps) with step size η and random initialization in [-ϵ, ϵ], following standard practice. For MCAT, the inner maximization is replaced by MS-PGD (Section 3.5) with the manifold penalty weight λ.

Generators for class-conditional manifolds. We train the class-conditional generators {Gy} on clean features (Section 3.2) with gradients stopped to θ and keep {Gy} fixed during robust training. We report generator architecture, latent dimension, and training iterations in Appendix B.

The hyperparameter settings of MCAT, the baselines, and the training process are provided by Table 8 in Appendix D.

Evaluation Protocol

Attacks. We evaluate robustness under a suite of increasingly strong white-box attacks: FGSM, multi-step PGD (e.g., PGD-20 and optionally PGD-100), and AutoAttack (AA). We keep ϵ consistent with training and use standard step sizes and iterations as in prior work [Wu et al., 2021; Yue et al., 2024; Yu-Hang et al., 2025].

Model selection and robust overfitting. To mitigate robust overfitting effects, we report both: (i) best checkpoint selected by validation PGD robustness, and (ii) last checkpoint at the final epoch, following robust training evaluation conventions [Yue et al., 2024; Cho et al., 2025].

Metrics

Standard accuracy and robustness. We report clean accuracy (Clean Acc) and robust accuracy (Robust Acc) under each attack (PGD-20/AA).

Tail and group-wise robustness. We report head/tail robust accuracy under PGD-20 and AA. We also report tail-only robustness (e.g., Tail-PGD, Tail-AA) to directly quantify tail reliability.

Balanced metrics. To measure fairness under class imbalance, we report: Balanced Accuracy (BA) and Balanced Robustness (BR) [Yu-Hang et al., 2025], defined as the average per-class accuracy under clean and adversarial evaluation, respectively. Concretely, letting Ac denote accuracy on class c, we compute BA = 1C ∑Cc=1 Aclean c , BR = 1C ∑Cc=1 Aadv c . Reporting. We report mean and standard deviation over multiple runs with different random seeds.

5.3 RQ1: Overall Adversarial Robustness

We evaluate overall adversarial robustness under longtailed distributions. Table 1 reports clean accuracy and robust accuracy under PGD-20 and AutoAttack (AA) on CIFAR-10LT, CIFAR-100-LT, and Tiny-ImageNet-LT with imbalance ratio IR = 100.

Across all benchmarks, MCAT consistently achieves the strongest adversarial robustness. In particular, MCAT substantially improves AutoAttack robustness over standard adversarial training baselines (PGD-AT, TRADES, MART, AWP) and recent long-tailed robust methods, with larger gains on CIFAR-100-LT and Tiny-ImageNet-LT. Notably, these improvements are achieved without sacrificing clean accuracy, indicating a favorable robustness-accuracy trade-off under severe imbalance.

(a) Overall AA robustness.

(b) Tail-class AA robustness.

Figure 4: Sensitivity analysis of MCAT hyperparameters on CIFAR100-LT (IR=100). Increasing λ suppresses off-manifold adversarial drift and improves robustness, while increasing β enlarges interclass angular separation and enhances tail robustness.

5.4 RQ2: Tail and Balanced Robustness

Robustness under increasing imbalance severity. We evaluate robustness under increasing imbalance severity by varying the imbalance ratio on CIFAR-100-LT. Figure 3 summarizes the results on CIFAR-100-LT under AutoAttack.

As imbalance becomes more severe, all methods experience performance degradation. However, MCAT degrades substantially more gracefully. In particular, MCAT maintains higher overall AA robustness and preserves substantially stronger Tail-AA robustness even under severe imbalance. Complete numerical results are reported in Tables 4 and 5 in Appendix C.

Class-balanced and tail-class robustness. Overall robustness metrics can obscure failures on tail classes. To assess robustness fairness, Table 2 reports balanced accuracy (BA), balanced robustness (BR), and tail-class robustness on CIFAR-100-LT (IR=100).

MCAT achieves the highest BA and BR among all compared methods and yields substantial gains in tail robustness under both PGD-20 and AA, indicating that its improvements

• (a) Minimum inter-class angle.
• (b) ETF alignment error.

Figure 5: Geometry under long-tailed adversarial training on CIFAR-100-LT (IR=100). Left: minimum inter-class angle θmin. Right: deviation from a margin-balanced ETF geometry. MCAT preserves larger angular margins and more balanced geometry.

are not driven solely by head classes but instead mitigate imbalance-induced bias. Results on Tiny-ImageNet-LT are deferred to Table 7 in Appendix C, where MCAT again shows the best results.

5.5 RQ3: Component Contribution and Sensitivity

Component ablations. Table 3 reports ablation results on CIFAR-100-LT. Adding the manifold constraint alone yields pronounced improvements in tail robustness, highlighting the importance of suppressing off-manifold adversarial drift. Geometric alignment alone substantially improves BA and BR, reflecting its role in alleviating imbalance-induced geometric bias. Combining both components yields the strongest and most consistent gains across all metrics.

Effect of λ (manifold constraint). Figures 4a and 4c show that λ controls a clear robustness-validity trade-off. With λ =

0, adversarial examples drift far off the class manifold and tail robustness drops. Increasing λ consistently suppresses drift and improves both overall and tail robustness, with gains saturating beyond a moderate range.

Effect of β (geometric alignment). Figures 4b and 4d show that increasing β enlarges the minimum inter-class angle θmin and improves Tail-AA robustness, with smooth and saturating trends consistent with the margin-robustness relationship in Theorem 1. This indicates that geometric alignment acts as a stable inductive bias rather than brittle tuning. As shown in Appendix Figure 10b, moderate β improves tail robustness without degrading head-class performance, while overly large β leads to over-regularization.

• (a) Inter-class angle vs. tail robustness

Base-Head Base-Tail MCAT-Head MCAT-Tail

• (b) Sample-wise robustness proxy

(c) Manifold constraint effects

Figure 7: Theory-aligned empirical evidence on CIFAR-100-LT (IR=100). (a) Larger minimum inter-class angle θmin correlates with stronger tail robustness. (b) MCAT shifts the tail-class distribution of the sample-wise robustness proxy ˆ r(x) toward larger values. (c) Increasing the manifold constraint weight λ jointly suppresses off-manifold drift and improves robust accuracy.

5.6 RQ4: MechanismVerification and Theory Consistency

Imbalance-induced geometric bias and Off-manifold adversarial drift. Figure 5 reports geometry diagnostics. Baseline methods exhibit reduced inter-class angular separation and increased deviation from a margin-balanced ETF geometry. In contrast, MCAT preserves larger angular margins and maintains more balanced decision geometry. Figure 6 shows distributions of off-manifold drift. Standard adversarial training induces pronounced drift, especially for tail classes, whereas MCAT substantially suppresses drift across all class groups. In addition to quantitative diagnostics, we provide a qualitative case study by Figure 8 in Appendix C. Compared to Base AT, MCAT yields noticeably tighter and better-separated tail-class embeddings while preserving compact head-class structure, offering intuitive evidence of improved geometric balance.

Theory-aligned empirical evidence. Figure 7 provides theory-consistent observations: (i) larger minimum interclass angles correlate with stronger tail robustness, (ii) MCAT shifts tail-class distributions of the sample-wise robustness proxy toward larger values, and (iii) increasing λ jointly suppresses drift and improves robustness. These results align with Theorems 1, 2, and Corollary 1. Additional per-class results are deferred to Figure 9 in Appendix C, where MCAT again shows the best results.

6 RelatedWork

General Long-Tailed Learning. Long-tailed learning in the standard setting has been widely studied through data augmentation, training paradigms, and representation rebalancing. Recent work leverages generative models for tail data synthesis [Zhao et al., 2024a; Shao et al., 2024], controllable expert-based training [Zhao et al., 2024b], and feature-space analyses that attribute tail failures to geometric distortion and representation collapse [Yi et al., 2025; Sun et al., 2025; Zhou et al., 2024].

Long-Tailed Adversarial Robustness. Prior studies show that adversarial training disproportionately harms tail classes under imbalance. Existing solutions rely on margin or sampling rebalancing [Wu et al., 2021; Liu et al., 2022], staged or reweighted optimization [Li et al., 2023; Yu-Hang et al., 2025], and robustness distillation [Cho et al., 2025], but do not explicitly regulate feature geometry or semantic validity of adversarial examples.

Geometry and Manifold Structure. Neural collapse and simplex ETF analyses highlight the role of balanced geometry in robustness [Papyan et al., 2020; Cao et al., 2025; Kothapalli, 2022; Zhu et al., 2022], while manifold-based studies link adversarial vulnerability to off-manifold perturbations [Li et al., 2025; Satou et al., 2025; Zhang et al., 2024]. Our work integrates these perspectives by jointly enforcing geometric balance and class-conditional manifold constraints for long-tailed adversarial training.

7 Conclusion

We proposed MCAT for long-tailed adversarial robustness that combines manifold-constrained adversarial training with ETF-inspired geometry regularization. We provided theoretical results connecting balanced geometry to robust margins and showing the benefit of constraining adversarial drift away from semantic low-density regions. Experiments on standard long-tailed benchmarks validate improved balanced robustness and tail performance under standard adversarial attacks.

Acknowledgments

This work is supported by Natural Science Foundation of Sichuan Province under grant 2024NSFSC0516 and National Natural Science Foundation of China under grant 61972270.