Info-Driven Zero-Cost Proxy: Rethinking Vision Transformer Architecture Evaluation via Information Quantification

1 Introduction

In recent years, Vision Transformer (ViT)[Dosovitskiy et al., 2021; Liu et al., 2021] has demonstrated exceptional modeling capabilities in computer vision tasks, gradually becoming a mainstream architecture in the field. However, the performance of ViT relies heavily on complex architectural designs [Touvron et al., 2021b; Zhai et al., 2022; Alabdulmohsin et al., 2023], including the selection of hyperparameters such as embedding dimension, number of heads, MLP expansion ratio, and network depth. The search space formed by these design dimensions is vast, where even subtle structural differences can lead to drastic fluctuations in model performance. Traditional manual design approaches are not only time-consuming and labor-intensive but also rely heavily on expert experience, making it difficult to systematically explore optimal structures [Ren et al., 2021; Gong and Wang, 2022].

∗Corresponding author.

To automate the design of efficient ViT architectures, traditional Neural Architecture Search (NAS) [Elsken et al., 2019; Wei et al., 2024b] emerged as an early solution. However, this approach requires training every candidate model, consuming massive computational resources. To reduce costs, researchers proposed supernet [Guo et al., 2020; Chen et al., 2021b] methods, which involve training a single colossal network and subsequently selecting different sub-structures from it. Although faster than traditional methods, training a supernet still requires hundreds of GPU hours. Consequently, Training-free NAS [Abdelfattah et al., 2021; Chen et al., 2021c; Mellor et al., 2021] has become a research hotspot. The core of this approach lies in designing a proxy metric capable of predicting the final performance of an architecture at Zero-Cost, enabling the rapid evaluation of untrained models and reducing search costs.

However, the search efficiency of mainstream zero-cost proxy [Lee et al., 2019; Tanaka et al., 2020; Ha et al., 2023; Jiang et al., 2023], including ViT-specific approaches such as TF-TAS [Zhou et al., 2024] and Auto-Prox [Wei et al., 2024a], remains suboptimal. Since these methods rely on forward or backward propagation, they incur non-negligible computational overhead when applied to large-scale search spaces. Furthermore, these methods often overlook the structural characteristics of ViT and rely on single metrics, thereby failing to provide a comprehensive evaluation of the architectures. Finally, although some studies have attempted to establish benchmark datasets [Ho et al., 2024; Wei et al., 2024a] for ViT architecture search, they remain insufficient in terms of dataset types and the number of architectures. This limitation directly hinders the rigorous evaluation of zero-cost proxy performance.

To address these issues, this paper proposes Info-NAS, a zero-cost proxy method based on information quantification that deeply models the unique structural characteristics of ViT by quantifying their information features. This represents the first structural modeling and quantification of the information processing efficiency of ViT architectures, providing a new physical perspective for understanding the intrinsic quality of neural network architectures. Specifically, we define the concept of Layer-wise Modulated Information and evaluate the potential of ViT architectures using three core components: (1) Global Information Volume (V): Measures the inherent information processing capability of network layers. (2) Local Information Gradient (G): Evaluates the smoothness of information flow between layers. (3) Global Consistency (C): Analyzes the overall consistency of the architecture. Finally, we fuse the information features from these three dimensions to obtain the final proxy score. Additionally, we constructed the ViT-Info-Bench dataset, which contains diverse ViT architectures and their ground-truth performance on CIFAR-10, CIFAR-100, Mini-ImageNet, and Tiny-ImageNet, providing a reliable benchmark for evaluating zero-cost proxy research on ViT.

Our core contributions can be summarized as follows:

• We propose Info-NAS, a zero-cost proxy that achieves the first structured quantification of information transmission in ViT. This method evaluates the architecture through three core components: Global Information Volume, Local Information Gradient, and Global Consistency, without requiring any forward or backward propagation.
• We construct a benchmark dataset based on the AutoFormer [Chen et al., 2021b] search space, which includes various ViT architectures and their performance across multiple datasets. This establishes an evaluation
• platform for training-free methods and provides data support for the correlation analysis of zero-cost proxies.
• Through extensive experiments, we comprehensively validate the superiority of Info-NAS across multiple benchmark datasets and search spaces. The results demonstrate that Info-NAS outperforms advanced zerocost proxies in terms of the correlation between proxy scores and ground-truth accuracy. More importantly, as shown in Figure 1, while maintaining high performance, Info-NAS significantly reduces the computational overhead of proxy scoring and the overall search time, achieving search efficiency far surpassing existing training-free algorithms.

2 RelatedWorks

2.1 Neural Architecture Search for ViT

Early ViT NAS adopted the One-Shot [Guo et al., 2020; Peng et al., 2020] paradigm from CNN. The core strategy is to train a supernet that contains all candidate sub-networks. AutoFormer [Chen et al., 2021b] pioneered this direction by introducing a weight entanglement mechanism. It achieved superior performance by evaluating sub-networks sampled from the trained supernet. Subsequently, ViTAS [Su et al., 2022] proposed cyclic weight sharing and identity transformation mechanisms. Independent class tokens were assigned to different patch sizes to reduce weight interference between sub-networks. Later methods focused on more complex search spaces and optimization strategies. For example, GLiT [Chen et al., 2021a] introduced convolutional operations to search for hybrid architectures. NASViT [Gong and Wang, 2022] addressed gradient conflicts during supernet training and used hardware constraints to guide the search. Although these methods automate architecture design, supernet training remains extremely expensive. The high computational cost hinders the application of NAS in scenarios requiring rapid iteration.

2.2 Zero-Cost Proxy

Training-free architecture search has emerged as a promising direction to mitigate the computational costs of supernet training. These methods utilize a zero-cost proxy to predict final performance at initialization, bypassing expensive training processes. Early proxies [Lee et al., 2019; Tanaka et al., 2020; Abdelfattah et al., 2021] primarily targeted CNN and relied on weight magnitudes or gradient norms. Recently, proxies tailored for ViT have been developed. TF-TAS [Zhou et al., 2024], the pioneering trainingfree method for ViT, assesses architectural potential by measuring synaptic diversity in MSA and synaptic saliency in MLP. Auto-Prox [Wei et al., 2024a] introduced an automated discovery framework to enhance robustness using a Joint Correlation Metric (JCM). AZ-NAS [Lee and Ham, 2024] improved ranking consistency by integrating metrics for expressivity, progressivity, trainability, and complexity. However, existing ViT proxies typically require a forward or backward pass, preventing them from being truly zero-cost. In this paper, we model the architectural information flow based on ViT components and propose a simpler, effective proxy. This approach drastically reduces evaluation time while significantly improving NAS performance.

3 Methodology

Figure 2 illustrates the overall framework of the proposed information-driven zero-cost proxy method. First, we introduce the concepts of Layer-wise Inherent Information, Information Modulation Coefficient, and Layer-wise Modulated Information. Next, we define three components by quantifying the connection between architectural parameters and information efficiency. Finally, we normalize these metrics to formulate the comprehensive proxy score.

3.1 Layer-wise InformationModeling

Within a discrete ViT architecture search space S, any candidate architecture A is uniquely determined by the following tuple of hyperparameters: A = {L, (El, Dl, Hl, Rl)Ll=1 } where L denotes the total number of layers in the Transformer encoder. For the l-th layer, El represents the embedding dimension; Dl indicates the dimension of the Q-K-Vmatrices; Hl signifies the number of attention heads; and Rl denotes the expansion ratio of MLP.

Layer-wise Inherent Information

In deep neural networks, information transmission often decreases layer by layer after multiple non-linear transformations [He et al., 2016; Liu et al., 2022]. This information dissipation extends to ViT [Noci et al., 2022; Dong et al., 2021; Zhou et al., 2021], where deep features become homogenized, implying severe information loss during transmission. Motivated by this, we introduce Layer-wise Inherent Information. Treating the network as a transmission channel with inherent loss, we employ a depth-based exponential decay model to simulate natural information attenuation in an idealized state. Specifically, for the l-th layer of architecture A, it is defined as:

where l represents the layer depth, and α, β, γ are hyperparameters controlling the inherent information decay curve. We set α = 12 and β = 2 through experiments. Given the maximum search depth of 16, we assign γ = 17, where this slight margin ensures γ functions as a valid asymptotic boundary.

Information Modulation Coefficient

While Cl serves as a theoretical prior, the effective information transmission is governed by specific architectural configurations that actively modulate the information flow. Consequently, we introduce the Information Modulation Coefficient η to quantify this effect for the l-th layer. Considering that a ViT layer comprises MSA and MLP blocks operating within a defined embedding dimension, η is designed to jointly evaluate the efficacy of these structural components:

where Edim l , Emsa l , and Emlp l represent the embedding dimension factor, the MSA factor, and the MLP factor of the l-th layer, respectively; wdim, wmsa, and wmlp are hyperparameters used to regulate the sensitivity of each factor in the overall score.

Dimension Factor. Serving as the carrier of information flow, the embedding dimension El defines the width of the inter-layer channel and determines the capacity upper bound of the feature state space. To quantify its impact on processing efficiency, we analyze how model performance responds to varying dimensions. As shown in Figure 3, widening the channel significantly enhances feature representation in the low-dimension regime. However, performance quickly saturates with further expansion and approximates a logarithmic trend. Consequently, we quantify the gain of El logarithmically and define the factor as:

MSA Factor. To quantify the efficacy of the MSAmodule, we model it through two dimensions: Feature Transformation Scale and Multi-head Subspace Diversity. First, we analyze the feature transformation scale. As the search space allows for decoupled embedding dimensions El and Q-K-V dimensions Dl, a linear projection is required within the MSA module. Specifically, the mapping from the embedding space REl to the Q-K-V subspaces RDl is performed by a weight matrixWl ∈ REl×Dl . The scale of this transformation is determined by the product of the input and output dimensions, which quantifies both computational complexity and parameter volume. Therefore, the feature transformation scale Sl of the l-th layer is defined as:

Theoretically, a restricted scale may cause an information bottleneck, leading to semantic loss. Conversely, an excessive scale introduces redundancy with minimal gain. To balance representation capacity and parameter efficiency, we propose an adaptability metric P (Sl) utilizing the Sigmoid function.

where σ1 and τ1 are tunable hyperparameters that control the steepness and threshold of the function, respectively.

Next is Multi-head Subspace Diversity. MSA projects inputs into Hl parallel subspaces to capture multi-perspective feature representations. While increasing Hl initially enriches semantics, the attentional foci of distinct heads inevitably overlap as the head count grows. This redundancy leads to diminishing marginal returns in non-redundant information gain. To quantify this saturation effect, we define multi-head subspace diversity as:

where σ2 and τ2 are hyperparameters that adjust the sensitivity and offset of the logarithmic function, respectively.

Synthesizing these dimensions, we define the MSA factor Emsa l for the l-th layer as the product of the feature transformation scale and multi-head subspace diversity:

MLP Factor. The MLP performs nonlinear transformations on the information flow across channel dimensions. The expansion ratio Rl determines the capacity of the hidden feature space, directly reflecting the information reconstruction capability of that layer. To quantify the specific impact of Rl on performance, we conducted experiments. As shown in Figure 3, within the commonly used search range of [3, 5], model accuracy demonstrates a monotonic ascent with increasing Rl. Therefore, we directly adopt Rl as the MLP factor:

Layer-wise Modulated Information

Finally, we define the information of the l-th layer processed by architectural modulation as the Layer-wise Modulated Information. This metric is formed by coupling the layer-wise inherent information with the information modulation coefficient, defined as:

3.2 Zero Proxy of Info-NAS

Next, we further analyze and process the Layer-wise Modulated Information and specifically design three core evaluation components.

Global Information Volume. In constructing the InfoNAS comprehensive proxy metric, the primary task is to quantify the architecture's capacity for information flow carriage and processing at the global scale. Since each network layer performs nonlinear mapping on input features, its contribution to the global representation is cumulative. Therefore, we define the Global Information Volume V by summing the modulated information of all layers.

Local Information Gradient. In training-free NAS research for ViT [Wei et al., 2024a; Zhou et al., 2024], gradient distribution and magnitude are considered direct indicators of model diversity. These metrics reflect the network's sensitivity to input features and its processing potential. Inspired by this, we introduce the Local Information Gradient component to capture feature transformation activity from the perspective of local dynamic evolution. To quantify the dynamic variations in inter-layer information flow, we define this metric as the cumulative sum of the change intensity of weighted information between adjacent layers:

Global Consistency. To ensure macro structural robustness and prevent pathological architectures, we introduce the Global Consistency metric. Designed to enforce global equilibrium, it is defined as the sum of pairwise differences in weighted information between any two network layers:

We formulate C as a structural penalty term. A value of C closer to zero implies a more stable distribution of information flow across the network depth, indicating superior global consistency.

Comprehensive Proxy Score. Based on the aforementioned research, we can directly compute the raw scores {Vi, Gi, Ci} for the three components using the architectural parameters. To eliminate dimensional differences between metrics of varying scales, we standardize each component's score using Z-scores. Finally, we obtain the comprehensive proxy score by linearly weighting and aggregating the three components:

where λV, λG, λC are adjustable weighting hyperparameters used to adjust the contribution weights among different components.

3.3 Training-free ViT NAS

We randomly sample 100,000 architectures from the constrained search space and compute the raw scores {Vi, Gi, Ci} for each architecture. Based on the composite proxy score derived from standardizing and weighting these three components, we select the network with the highest proxy score as the optimal architecture. Finally, we retrain it to obtain its test accuracy on the target dataset.

4 Experiments

In this section, we first outline the experimental setup, followed by an introduction to the benchmark datasets: ViTInfo-Bench, OoD-ViT-NAS, and ViT-Bench-101. Finally, we compare Info-NAS with existing methods and present a performance evaluation across multiple benchmarks.

4.1 Experimental Settings

We tuned the hyperparameters of Info-NAS on a small dataset containing 50 real architectures. To ensure fair evaluation, this dataset does not overlap with any subsequent benchmarks. The determined parameters were fixed for all experiments and were not fine-tuned for specific datasets. We evaluated our method on ViT-Info-Bench, ViT-Bench-101 [Wei et al., 2024a], and OoD-ViT-NAS [Ho et al., 2024]. These experiments cover different model complexities, with parameter counts ranging from 4-9M, 14-34M, and 42-75M. For evaluation, we select the architecture with the highest zero-cost proxy score and report its ground-truth accuracy. The details of these benchmark datasets are introduced below.

ViT-Info-Bench. ViT-Info-Bench records the ground-truth accuracy of ViTs on four datasets: CIFAR-10, CIFAR-100, Mini-ImageNet, and Tiny-ImageNet. It is built upon the AutoFormer search space. We first trained the AutoFormer-T and AutoFormer-S supernets using unified settings for the optimizer, learning rate, and regularization to ensure parameter reliability. Subnets were then sampled to inherit pretrained weights directly from the supernet for accuracy evaluation. This benchmark includes 500 evaluated architectures per dataset, providing a basis for zero-cost proxy analysis.

ViT-Bench-101. ViT-Bench-101 provides both distillation and vanilla accuracies for ViT on small datasets, including CIFAR-100, Flowers, and Chaoyang, and the large-scale ImageNet-1K. It supports the evaluation of zero-cost proxies by analyzing score-accuracy correlations across different scenarios.

OoD-ViT-NAS. OoD-ViT-NAS is the first comprehensive benchmark for the OOD generalization of ViT in NAS. It comprises 3,000 architectures, with 1,000 sampled from each of the AutoFormer Tiny, Small, and Base search spaces. These architectures are evaluated on eight mainstream OOD datasets, such as ImageNet-C, providing core metrics including test accuracy and parameter counts.

4.2 Experimental Results On ViT-Info-Bench

We evaluated the rank correlation of various NAS methods on ViT-Info-Bench using Kendall's τ [Abdi, 2007] and Spearman's ρ [Stephanou and Varughese, 2021]. As shown in Table 1 and Figure 4, Info-NAS consistently outperforms competitors across most metrics. These results demonstrate that Info-NAS accurately identifies high-performance ViT architectures and exhibits strong robustness across diverse datasets. Further analysis reveals that methods such as Jacobian and Meco primarily focus on local features or gradient sensitivity. While effective for CNN, these approaches struggle in the ViT search space due to gradient noise and signal attenuation during initialization. In contrast, Info-NAS quantifies discrete architectural parameters as information transmission capabilities, directly evaluating performance through these parameters and deeply modeling the structural characteristics of ViT across the V, G, and C dimensions. This result not only validates the comprehensiveness and precision of Info-NAS in capturing ViT performance attributes but also underscores the significant potential and rationality of constructing efficient zero-cost proxies centered on information flow.

Table 2 compares the Top-1 validation accuracy and computational overhead of various NAS methods. Data-driven approaches like GraSP and SNIP suffer from high complexity, requiring over 800 ms per evaluation, rendering them impractical for large-scale search. While advanced methods such as Auto-Prox reduce latency to approximately 90 ms, they remain constrained by the necessity of forward propagation. Conversely, Info-NAS bypasses forward propagation entirely, reducing the proxy time to just 1.8 ms. This efficiency allows Info-NAS to maximize search speed while preserving high accuracy, underscoring its superiority in ViT architecture search and providing a fresh perspective on fully zero-cost proxies.

4.3 Experimental Results On ViT-Bench-101

To validate the generalization capability of Info-NAS across diverse data distributions, we compared its performance on the ViT-Bench-101 benchmark, as presented in Table 3. The results indicate that Info-NAS achieves the highest Spearman correlation coefficients on both the CIFAR-100 and Flowers datasets, maintaining remarkable ranking consistency. On the Chaoyang dataset, Info-NAS also demonstrates highly competitive performance. This consistent stability confirms that Info-NAS is a robust and efficient zero-cost proxy applicable to a wide range of ViT architecture search scenarios. Moreover, these findings further reveal a universal intrinsic connection between the internal information flow patterns of neural networks and their performance.

4.4 Experimental Results On OoD-ViT-NAS

To further validate the generalizability of Info-NAS, we evaluated it on the challenging OoD-ViT-NAS benchmark. Table 4 reports the correlation results on ImageNet-1K, while Table 5 compares the best Info-NAS model against manually designed networks and advanced NAS methods. The results demonstrate that Info-NAS significantly outperforms existing zero-cost proxies across three correlation metrics and achieves the highest Top-1 accuracy of 75.7%. Notably, InfoNAS achieves superior search efficiency, requiring only 0.005 GPU days. This represents an order-of-magnitude speedup over comparable methods, significantly reducing computational overhead.

4.5 Ablation Study

As shown in Table 6, V plays a dominant role in performance evaluation, and its individual correlation significantly outperforms the other components across all datasets. This finding strongly corroborates that the total information volume of the architecture is the primary determinant of feature representation richness. In contrast, G and C function primarily as auxiliary components: G captures the activity of inter-layer information flow, while C constrains the global consistency of information transmission paths. Together, they effectively refine the capacity estimation provided by V. Ultimately, the fusion of all components (V + G + C) yields optimal performance across all benchmarks, demonstrating that our method comprehensively captures the complex architectural characteristics of ViT and enables a precise information quantification of model potential.

4.6 Visualization of Feature Space

To validate the representational power of Info-NAS, we visualize the architecture embeddings using t-SNE [Maaten and Hinton, 2008]. Specifically, we construct feature vectors by aggregating layer-wise modulated information {MIi}Lmax i=1 , padding them with zeros to the maximum network depth to ensure alignment, followed by standardization and dimensionality reduction. As shown in Figure 5, unlike the chaotic distribution of the random features, Info-NAS exhibits distinct clustering where similarly performing architectures are spatially proximal. This comparison strongly validates it as a robust feature representation for ViT performance evaluation.

5 Conclusion

In this paper, we propose Info-NAS, a zero-cost proxy approach based on information quantification, presenting the first information quantification of ViT architectures. We establish an efficient proxy evaluation framework spanning three components to facilitate the rapid assessment of ViT performance. Furthermore, we introduce the ViT-Info-Bench dataset to provide data support for the correlation analysis of zero-cost proxies. Extensive experiments across multiple benchmarks demonstrate that Info-NAS significantly outperforms existing zero-cost proxy methods in both ranking correlation and computational efficiency. It maintains high performance while substantially reducing proxy calculation overhead and search time, with each evaluation requiring only 1.8 ms. We hope this perspective, grounded in information flow, offers a promising direction for future research on efficient, training-free NAS methodologies.

Acknowledgments

This work was supported in part by the Regional Innovation and Development Joint Fund of the National Natural Science Foundation of China under Grant U25A20435, in part by the Key Research and Development Program of Hubei Province, China under Grant 2025BAB051, and in part by the Provincial Key Science and Technology Project of Hubei Province, China under Grant 2025BEA006.

Contribution Statement

†Yue Yang and Jiacheng Wang contributed equally.