DANCE: Resource-Efficient Neural Architecture Search with Data-Aware and Continuous Adaptation

1 Introduction

Neural Architecture Search (NAS) has revolutionized deep neural network design by automating the architecture optimization process [Chen et al., 2022]. Despite its recent success, existing NAS methods face significant challenges in effectively modeling three key aspects: architecture adaptability [Zhao, 2022], deployment efficiency [Zhu et al., 2023], and performance consistency across diverse scenarios [Ren et al., 2021; Zhang et al., 2023]. While current approaches can address certain aspects, they often fail to comprehensively tackle all these challenges simultaneously [Cai et al., 2019].

∗Corresponding authors.

In real-world deployment scenarios, neural architectures naturally exhibit three key characteristics: (1) ComputeConstraints: architectures need to adapt to varying computational constraints (e.g., an architecture suitable for GPU servers may be impractical on mobile devices) [Howard, 2017; Liu et al., 2020; Zhao et al., 2021b], (2) Varied Requirements: deployment scenarios vary significantly in their requirements (e.g., real-time applications demand low latency while offline tasks prioritize accuracy) [Wang et al., 2019; Zhu et al., 2023], and (3) Performance Variance: model performance patterns often differ across deployment contexts (e.g., models optimized for one domain may underperform in others) [Tan and Le, 2019; Gao et al., 2023]. These observations highlight the critical need for dynamic and adaptable neural architectures that can efficiently handle diverse deployment scenarios while maintaining robust performance [Zhao, 2022].

Based on these observations, neural architecture search systems need to address three fundamental challenges that require innovative solutions: First, architecture flexibility needs to be effectively modeled. Current discrete search methods lack the ability to smoothly adapt architectures as deployment requirements change, necessitating expensive re-search for each scenario [Zoph, 2016; Zhao, 2022; Zhu et al., 2023; Lin et al., 2022]. Second, deployment scenarios exhibit diverse computational constraints. While recent methods have shown promise in handling specific deployment contexts [Wu et al., 2019; Liu et al., 2020], they struggle to maintain consistent performance across varying scenarios due to their rigid architecture designs [Gao et al., 2023; Li et al., 2023; Song et al., 2022]. Third, architecture search naturally involves multiple competing objectives. Despite their advantages in finding optimal architectures, existing methods typically process these objectives uniformly, failing to capture the complex trade-offs between performance metrics [Elsken et al., 2019; Zhao et al., 2021a; Zhang et al., 2023; Jin et al., 2021]. These limitations of current approaches are particularly evident in practice: Pure evolutionary methods demonstrate sub-optimal efficiency in large-scale deployments [Real et al., 2019; Liu et al., 2024], gradient-based approaches face significant scalability challenges [El Bsat et al., 2017], and while oneshot methods offer improved search efficiency [DONG et al., 2023; Zhaok et al., 2021], they not only show limited transfer capability but also lack mechanisms to effectively capture diverse deployment requirements [Bender et al., 2018; Chen et al., 2022; Li et al., 2023]. This creates a clear need for a unified solution that can simultaneously address architecture adaptability and deployment efficiency while maintaining consistent performance.

To address these challenges, we propose DANCE (Dynamic Architectures with Neural Continuous Evolution), a novel framework that synergistically combines continuous architecture evolution and distribution learning through a unified optimization approach. Unlike previous methods that focus on block-level components, DANCE operates directly on feature dimensions, providing a unified perspective that bridges the gap between NAS and pruning. The continuous evolution mechanism enables smooth architecture adaptation across different deployment scenarios, while the distribution learning component efficiently captures and models deployment patterns. By integrating data-aware dynamic selection and importance-based sampling at the feature level, this unified framework effectively leverages both architectural flexibility and deployment efficiency while maintaining computational tractability through selective sampling and adaptive optimization strategies.

Our main contributions are summarized as follows:

• We propose DANCE, a novel neural architecture search framework that learns continuous distributions over architectural components. This enables efficient architecture adaptation under varying computational budgets while maintaining high performance.
• We design a unified architecture space with dynamic select gates that integrate batch-level features and layerwise importance metrics. This mechanism enables flexible component selection and smooth architecture adaptation across different hardware constraints and deployment contexts.
• Extensive experiments on five image datasets demonstrate that DANCE achieves consistent accuracy improvements over state-of-the-art methods while significantly reducing search costs. The results validate its effectiveness for practical applications with diverse computational requirements and resource constraints.

2 Methodology

Neural architecture search (NAS) has emerged as a promising approach for automating deep neural network design. However, current NAS methods face two key challenges in realworld deployments: (1) the static architectures lack flexibility to adapt to varying computational constraints, and (2) separate expensive searches are required for each deployment scenario, leading to prohibitive costs. To address these limitations, we propose DANCE (Dynamic Architectures with Neural Continuous Evolution), which reformulates architecture search as a continuous evolution problem. Instead of finding a single fixed architecture, DANCE learns a distribution over architectural components that can be efficiently sampled and adapted to specific deployment requirements. This enables rapid architecture derivation while maintaining performance across diverse scenarios.

2.1 ProblemFormulation

Traditional NAS methods typically operate in a discrete search space, optimizing architectures under the fixed resource constraints:

where A denotes a candidate architecture and C specifies the computational budget constraint. This discrete optimization approach presents several challenges. First, it requires running separate searches for different deployment scenarios, leading to substantial computational overhead. Second, the discrete nature of the search space makes it difficult to adapt architectures smoothly as requirements change. Third, there is no guarantee of consistency between the selected components across different layers, resulting in suboptimal results.

2.2 Unified Architecture Space Design

Building upon the limitations of traditional NAS methods, DANCE introduces a unified architecture space design that accommodates diverse deployment requirements while significantly enhancing search efficiency. As shown in Figure 1, for each N-th SuperNet layer, we introduce an data-aware dynamic select gate to dynamically generate a pruning layer for component selection.

The key insight is learning a continuous distribution over architectural components that factorizes across layers while maintaining dependencies:

This formulation expresses the probability of selecting a specific architecture A as a product of layer-wise selection probabilities, where each layer's selection depends on the layer features Fl, input data batch D, and resource constraints Cl (where Cl represents the number of parameters to retain in layer l). Specifically, we implement this through a data-aware dynamic selection mechanism:

For each n-th layer in the supernet, the SelectGate system combines concurrent layer feature and batch embedding through feature combination, which then passes through gumbel-softmax to generate select gate for layer selection.

This process can be formulated as:

where FeatureCombl is the combined feature and gl is the final select gate output. Here MLP is a two-layer neural network with ReLU activation in between, which learns to map the combined features to selection probabilities. Given the layer-wise resource budget Cl, the selection gate mechanism employs a combination of soft and stochastic selection to determine the final gate value.

Here, ϵ is sampled from a Gumbel(0, 1) distribution to introduce randomness in the softmax operation, τ is the temperature parameter controlling the sharpness of the softmax distribution, ˜ gl represents the soft selection probabilities after gumbel-softmax. pl normalizes the importance-weighted probabilities through softmax to ensure valid sampling probabilities. The BernoulliSample function performs stochastic binary sampling where components are sampled according to their probabilities while ensuring exactly Cl components are selected, maintaining the resource constraint while introducing controlled randomness in architecture exploration. Score(Al) comprehensively evaluates component importance by considering static importance, dynamic importance, feature importance, and correlation penalty metrics. The resulting g(l) f determines which components in the layer are retained or pruned, effectively generating a sublayer structure. Detailed design and mathematical equations of score(Al) are provided in Appendix Sec.A.

2.3 Multi-Stage Training Process

To effectively learn and utilize the unified architecture space, DANCE employs a three-stage training process (as shown in Figure 2) that progressively optimizes both the architectural distribution and model parameters at the same time.

First Stage: SuperNet Pre-training

We first construct and pre-train a SuperNet containing N stacked layers as the backbone architecture. For each layer, we incorporate multiple parallel branches with different operators and channel dimensions to provide architectural flexibility. The SuperNet training process involves sampling different paths and optimizing them jointly to learn robust and transferable feature representations. To maintain stability during training, we employ progressive path dropping and knowledge distillation between different architectural configurations. Additionally, we introduce architecture-aware regularization to encourage diversity in the learned features across different components. This pre-training stage establishes a strong foundation that can effectively generalize across varying computational environments while ensuring consistent feature extraction capabilities. The pre-trained SuperNet serves as the starting point for subsequent distribution-guided architecture adaptation.

Second Stage: Distribution-Guided Learning

For each layer l, in each batch iteration t, our SelectGate takes three key inputs: the batch-level distribution features Dt, concurrent layer features Fl, and layer-wise resource constraints Cl, generating binary masks g(l,t) to indicate components:

f

The training process employs a dual-loop optimization strategy that enables comprehensive architecture exploration while ensuring stable performance. In the outer loop across epochs E, the network progressively refines both its model weights and SelectGate parameters.

In each batch iteration t, we sample T different architectures using SelectGate to explore the architectural space while adhering to resource constraints. These sampled architectures are used to train the network weights, with SelectGate parameters being updated based on their performance feedback. This inner loop process enables thorough architecture exploration with efficient parameter optimization.

Specifically, for each batch iteration t, we first obtain T different architecture samples {A(t) i }Ti=1 following SelectGate's probability distribution under given resource constraints C. We then update both the network weights θ and SelectGate parameters ϕ using the training feedback from these sampled architectures. This dual-loop mechanism achieves both efficient architecture exploration and effective parameter optimization while satisfying deployment constraints.

Third Stage: Scenario-Specific Optimization

In the final stage, DANCE optimizes SubNets for specific deployment scenarios. Given the input query data D and resource constraints C, we obtain the optimal architecture A∗ by inheriting weights from the SuperNet W and selecting features at each layer:

where gl is the selection gate output at layer l based on distribution D and features Fl, and Wl represents the weights from layer l of the SuperNet. The final architecture A∗ is constructed by applying these pruning layers g(l) to the SuperNet weights. We then fine-tune this architecture through a multi-objective optimization process task performance: minθ Ltotal(A∗ ) = Ltask. Through this carefully

Algorithm 1 Three-Stage Training Process

Input: Training data D, Resource constraints C, Initial parameters θ

Output: Optimized SubNet A∗

• 1: // Stage 1: SuperNet Pre-training
• 2: for each training iteration do
• 3: Sample batch data fromD
• 4: Update SuperNet parameters θ with standard training
• 5: end for
• 6: // Stage 2: Distribution-Guided Learning
• 7: for each batch iteration t do
• 8: Extract features: F t l = BatchReprl (Dt)
• 9: for each layer l do
• 10: g(l,t) f = SelectGate(Dt,F t l ,Cl)
• 11: end for

12: Update parameters with sampled architectures

• 13: end for
• 14: // Stage 3: Scenario-Specific Optimization
• 15: for deployment scenario do
• 16: Sample batch data fromD
• 17: Generate architecture via SelectGate
• 18: A∗ = {g(l) f ⊙Wl}Ll=1
• 19: Fine-tune A∗ by minimizing Ltotal
• 20: end for
• 21: return A∗

designed three-stage process, DANCE achieves continuous adaptation to varying computational constraints and data distributions, resource-efficient architecture search guided by distribution awareness, robust performance across diverse deployment scenarios, and end-to-end trainability without discrete sampling or complex optimization procedures.

2.4 Framework Analysis

Given a network with L layers and maximum width W per layer, processing a batch of size B, DANCE achieves O(TBL(W + |C| + P)) computational complexity, where T is the number of training iterations, |C| is the dimension of constraint embeddings, and P denotes the number of trainable parameters in the network. Through distribution learning and continuous evolution, DANCE enables efficient architecture sampling at O(1) cost per scenario with the learned distribution p(A|D, C). While traditional NAS methods require separate searches for each deployment scenario leading to substantial computational overhead, DANCE learns a unified distribution that enables efficient architecture sampling for new scenarios. Compared to methods that directly evaluate architectures without training, DANCE's distribution learning captures both data characteristics and deployment constraints through Fl(D) for more accurate architecture generation. The discrete nature of traditional search spaces makes it difficult to adapt architectures smoothly as requirements change, but DANCE's continuous evolution through the selection gate gl enables smooth architecture adaptation via distribution sampling. Unlike methods requiring manual tuning with separate optimizations per scenario, DANCE automates this through the comprehensive scoring mechanism Score(A) that maintains consistency between selected components across layers. The continuous evolution through the learned distribution p(A|D,C) enables dynamic adaptation as constraints change, making DANCE particularly effective for deployment with varying computational requirements.

3 Experiment

To demonstrate the effectiveness of our proposed DANCE, we conduct extensive experiments on five image datasets.

3.1 Datasets and Backbones

To systematically evaluate DANCE's adaptive capabilities and validate its technical innovations, we conduct comprehensive experiments across five datasets using two classic backbone architectures: The evaluation framework employs CIFAR-10 [Krizhevsky et al., 2009] and CIFAR100 [Krizhevsky et al., 2009] as foundational benchmarks to establish baseline architectural distributions. Three challenging fine-grained datasets-Stanford Cars [Kramberger and Potoˇcnik, 2020], CUB-200-2011 [Wah et al., 2011], and Food-101 [Bossard et al., 2014]-are used to rigorously test the precision of dynamic selection mechanisms and architectural sampling under high-resolution visual discrimination requirements. These datasets test DANCE on several dimensions, including resolution scalability, architecture efficiency, and task complexity progression (basic categorization to finegrained differentiation). It demonstrates how DANCE can adapt to constantly changing task requirements while balancing resource constraints. For more details about datasets and backbones, please refer to Appendix Sec. B.

3.2 EvaluationMetrics

We establish a rigorous evaluation framework to assess the effectiveness of our proposed pruning method across multiple dimensions of performance and efficiency. The primary performance indicator is Top-1 accuracy measured across five datasets (CIFAR-10/100, Food-101, Stanford Cars, CUB200), complemented by performance retention ratios relative to pretrained models. To quantify computational efficiency, we track both parameter reduction metrics and FLOPs measurements, analyzing pruning granularity through perlayer parameter changes and channel configuration modifications across network stages. We also performing detailed ablation studies to examine the contributions of static, dynamic, and feature importance components. The framework incorporates cross-platform evaluation across different model scales (ResNet-18 [He et al., 2016], VGG-16 [Simonyan, 2014]) and progressive assessment under varying resource constraints (target sparsity from 0.5 to 0.9), enabling comprehensive analysis of the accuracy-efficiency trade-off. The stability and adaptability of our method are further validated through fine-grained resource constraint variations (0.1 to 0.5 with 0.02 step size), providing insights into performance consistency across diverse deployment scenarios. This comprehensive evaluation methodology enables us to rigorously validate our method's capability in achieving optimal balance between model performance and resource efficiency while maintaining robust adaptability.

3.3 Baselines

To demonstrate the effectiveness and efficiency of our approach, we compare our approach with different state-ofthe-art NAS methods. OFA [Cai et al., 2019] uses progressive shrinking to train a supernet that supports diverse architectural configurations, enabling flexible deployment under different resource constraints. ProxylessNAS [Cai et al., 2018b] directly trains specialized architectures on target hardware while addressing resource constraints through gradient-based optimization. RecNAS [Peng et al., 2022] employs recursive channel pruning to find efficient architectures, achieving a better trade-off between accuracy and model size. SPOS [Guo et al., 2020] adopts a single-path one-shot architecture to train a supernet, allowing efficient architecture sampling and evaluation. FixMatch [Sohn et al., 2020] leverages a semi-supervised learning [Zhu, 2005] strategy to improve model performance with limited labeled data through consistency regularization. The key difference in our approach is the integration of dynamic channel selection and correlationaware pruning, which enables more effective identification of redundant parameters while maintaining model accuracy across different sparsity levels. We also implement a Scorebased Pruning baseline that directly applies our architecture evaluation metric Score(A) (defined in Appendix Sec.A and Sec 2.2) for the direct parameter.

3.4 Implementation Details

When implementing the training process, we adopt a twostage approach with carefully selected hyperparameters. Stage 1(Pre-training)runs for epochs with frozen SelectGate modules, while Stage 2 (Distribution-Guided Architecture Learning) continues for epochs with all components activated. The learning rates are selected from [0.0001, 0.0005, 0.001] for different components. We employ the AdamW optimizer with OneCycleLR scheduler using 30% warm-up period and early stopping patience of 15. All components are implemented using PyTorch and monitored through comprehensive metrics, including accuracy, loss components, and gate statistics. The backbone network uses ResNet-18 and VGG-16 architecture with customized SelectGate modules integrated at different layers. Training is performed on standard GPU hardware with automatic mixed precision (AMP) enabled for efficiency. For more details about datasets and backbones, please refer to Appendix Sec. C.

3.5 Overall Performance Comparison

As shown in Table 1, we observe three remarkable phenomena across five datasets. First, our Stage 2 results achieve strong performance through direct sampling without any retraining - notably outperforming several baseline methods that require full fine-tuning. For instance, on CIFAR-100, our Stage 2 sampling achieves 70.92% accuracy, surpassing RecNAS (66.50%) and SPOS (60.76%). Second, with minimal parameters (2.1M-3.0M for ResNet-18), our method maintains competitive accuracy across diverse tasks. Third, while retraining further boosts performance, the strong Stage 2 results already demonstrate the effectiveness of our approach.

These phenomena stem from our distribution-guided architecture learning strategy. Unlike traditional methods that require separate, expensive fine-tuning for each deployment scenario, our SelectGate mechanism learns to directly generate effective architectures by comprehensively evaluating both the static and dynamic importance of components. The data-aware nature of our approach enables the SelectGate to capture task-specific requirements and generate suitable architectures through simple sampling from the learned distribution p(A|D, C). The exceptional Stage 2 performance validates our core insight of reformulating architecture search as a continuous evolution problem. Our approach enables rapid architecture derivation through direct sampling while maintaining strong performance by learning a distribution over architectural components rather than searching for fixed architectures. This represents a significant advantage over existing methods that rely heavily on costly fine-tuning procedures.

3.6 Effectiveness of Distribution-Guided Learning vs Simple Score-Based Sampling

As shown in Figure 3, we observe several key advantages of DANCE over score-based pruning, despite both methods utilizing the same importance scoring mechanism. First, DANCE achieves substantially better efficiency-accuracy trade-off, with a much steeper slope in the accuracy-FLOPs curve. While score-based pruning requires 1200-1500M FLOPs to reach 60% accuracy, DANCE achieves similar or better accuracy with only 400-500M FLOPs, demonstrating superior efficiency. Second, score-based pruning shows scattered and inconsistent performance (blue dots widely spread), while DANCE demonstrates stable and predictable behavior with a clear trend (red dots closely following the trend line). Random architecture selection on SuperNet performs significantly worse, validating the importance of structured pruning strategies. These phenomena validate that merely using importance scores for selection is insufficient. Although both methods leverage the same scoring mechanism, our SelectGate's distribution-guided learning strategy enables more effective architecture generation by learning to combine and utilize the importance scores through the continuous distribution p(A|D,C). The substantial performance gap demonstrates that the key advantage comes from our distribution-guided learning framework rather than the scoring mechanism itself.

To further validate this finding, we conduct experiments on sub-networks with different constraints. As shown in Figure 4, DANCE consistently outperforms score-based pruning across different scales. This consistent superior performance validates the effectiveness of our distribution-guided learning framework, showing its ability to maintain high accuracy while satisfying various computational constraints.

3.7 Ablation Study

Our sampling strategy incorporates four essential and complementary dimensions of importance metrics. Static importance measures the inherent significance of nodes by reflecting their fundamental structural contributions, while dynamic importance captures runtime activation patterns that reveal operational importance during inference. Feature importance evaluates the discriminative power of different features to preserve critical information, and the correlation penalty balances these metrics by reducing redundancy for efficient sampling. As shown in Table 2, using only a single importance metric leads to substantial performance drops across all resource constraints, validating that these dimensions must work in synergy to achieve optimal sampling effectiveness. Hyperparameter analysis can be found at Appendix Sec . D.

4 RelatedWork

Neural Architecture Search. Neural Architecture Search (NAS) aims to automate the discovery of optimal neural network architectures [Elsken et al., 2019]. Early NAS methods relied on reinforcement learning or evolutionary algorithms to explore discrete architecture spaces, but suffered from prohibitive computational costs due to training each candidate architecture from scratch [Zoph, 2016]. One-shot NAS methods [Pham et al., 2018] improved efficiency by training a supernet containing all possible architectures and sharing weights between candidates. While this reduced search costs, these methods still struggle with effectively adapting architectures across different deployment scenarios due to their discrete nature, as they primarily focus on block-level architecture components rather than feature dimensions.

Recent works have explored more flexible approaches. Progressive NAS [Liu et al., 2018] gradually increases architecture complexity during search but lacks mechanisms for handling varying computational constraints. Differentiable NAS methods [Wang et al., 2021] enable end-to-end architecture optimization but face challenges in maintaining performance consistency across diverse deployment contexts. Although these advances have improved search efficiency, they do not fully address the need for dynamic architecture adaptation under varying computational requirements.

DANCE demonstrates consistent accuracy gains and reduced search costs across diverse datasets, proving its effectiveness for practical applications with varying computational demands and resource limitations.

Efficient Architecture Design. The growing demand for deploying deep neural networks across different computational environments has sparked interest in efficient architecture design [Cai et al., 2018a]. Traditional network pruning approaches focus on removing redundant parameters or channels based on importance metrics [Han et al., 2015], while NAS methods typically search over predefined block-level components. Our work bridges this gap by reformulating both pruning and NAS from a unified feature dimension perspective, enabling more flexible architecture adaptation.

More recent approaches have focused on automated, efficient architecture design. Resource-aware NAS methods [Yang and Sun, 2021] explicitly incorporate hardware constraints during search but struggle with smooth architecture adaptation as requirements change. While these methods have shown promise for specific deployment contexts, they lack mechanisms for effectively modeling the complex trade-offs between multiple competing objectives across diverse scenarios. Additionally, their block-based search spaces limit the granularity of architecture optimization compared to our feature-dimension based approach.

Our DANCE framework addresses these limitations by unifying NAS and pruning through continuous evolution over feature dimensions rather than discrete blocks. This novel perspective enables more efficient and flexible architecture adaptation while maintaining performance consistency across diverse deployment scenarios.

5 Conclusion

This paper presents DANCE, a novel framework that reformulates neural architecture search through continuous evolution over feature dimensions rather than traditional blocklevel components. Our approach uniquely bridges the gap between NAS and pruning, providing a unified perspective for flexible architecture adaptation across diverse deployment scenarios. Through extensive experiments, we demonstrate that DANCE successfully addresses several key limitations of existing methods by enabling smooth architecture adaptation, effectively handling diverse computational constraints, and efficiently balancing multiple competing objectives during the search process. Our results show that DANCE achieves superior accuracy compared to state-of-the-art methods while significantly reducing search costs across various real-world deployment scenarios. These findings validate the effectiveness of viewing architecture search through the lens of feature-dimension optimization and continuous evolution, suggesting a promising direction for future research in automated neural architecture design.

Acknowledgements

This research was partially supported by Research Impact Fund (No.R1015-23), Collaborative Research Fund (No.C1043-24GF), Huawei (Huawei Innovation Research Program, Huawei Fellowship), Tencent (CCF-Tencent Open Fund, Tencent Rhino-Bird Focused Research Program), Alibaba (CCF-Alimama Tech Kangaroo Fund No. 2024002), Ant Group (CCF-Ant Research Fund), and Kuaishou.

Contribution Statement

Maolin Wang and Tianshuo Wei conceptualized the research idea, designed the methodology, and conducted the primary experiments. Maolin Wang led the software implementation, manuscript writing and data analysis. Tianshuo Wei contributed to software implementation and validation. Ruocheng Guo (corresponding author) provided critical insights on the experimental design and data interpretation. Wanyu Wang (corresponding author) supervised the research direction and provided overall guidance. Shanshan Ye (corresponding author) contributed to the methodology verification and oversaw the experimental validation. Sheng Zhang assisted with data collection and analysis, and contributed to manuscript preparation. Lixin Zou assisted with data analysis and visualization. Xuetao Wei provided valuable feedback on the manuscript and contributed to the discussion section. Xiangyu Zhao supervised the project, secured funding, and provided critical revision of the manuscript. All authors reviewed and approved the final version of the manuscript.