AC2-VLA: Action-Context-Aware Adaptive Computation in Vision-Language-Action Models for Efficient Robotic Manipulation

1 Introduction

Recent progress in vision-language foundation models and large-scale robot datasets such as Open X-Embodiment [O'Neill et al., 2023] have accelerated the development of generalist Vision-Language-Action (VLA) models. In closed-loop embodied tasks, these models must deliver lowlatency decisions with stable closed-loop control over longhorizon tasks. Representative methods such as RT-2 [Brohan et al., 2023] and OpenVLA [Kim et al., 2024] demonstrate that large multimodal backbones can follow language instructions and generalize to diverse tasks. More recent policies such as CogACT [Li et al., 2024a] further improve control by generating expressive action trajectories via diffusionbased modeling. However, deploying these models remains challenging because inference repeatedly executes a computationally expensive vision-language backbone at every control step, resulting in high latency and compute cost, reducing control frequency, and compromising real-time responsiveness in dynamic environments.

∗Corresponding author.

To mitigate these deployment challenges, recent work has explored various efficiency mechanisms for VLA models. Static compression methods such as pruning [Yang et al., 2025] and quantization [Fang et al., 2025] reduce model size but cannot adapt to changing task complexity. Dynamic computation techniques, including token pruning [Tan et al., 2025] and layer skipping [Zhang et al., 2025], adjust compute online, while caching approaches such as VLA-Cache [Xu et al., 2025] exploit temporal redundancy by reusing features across adjacent timesteps. Despite these advances, most existing methods make compute allocation decisions primarily based on visual cues, which can be suboptimal for robot manipulation. In embodied tasks, visual complexity does not necessarily correlate with control difficulty: visually simple scenes may require full-capacity reasoning for precise interactions, while visually complex transit phases may allow more aggressive pruning.

Based on this insight, we present Action-Context-aware Adaptive Computation for VLA models (AC2 -VLA), as illustrated in Fig. 1. AC2 -VLA dynamically allocates computation along the temporal, spatial, and depth dimensions, guided by the action-centric context that is directly relevant to embodied tasks. Specifically, we introduce a lightweight action-prior router that conditions on the previous action state together with multimodal embeddings, and predicts a unified sparsity strategy for the current timestep. The router orchestrates three complementary mechanisms: (i) cognition caching, which reuses backbone features across adjacent timesteps when the action context suggests stable state transitions; (ii) action-context-aware token pruning, which removes visual tokens that are irrelevant to the current manipulation stage; and (iii) conditional layer skipping, which bypasses redundant transformer blocks when the action context indicates lower reasoning demand. To preserve the robustness of the original dense policy under structured sparsification, we train the router using an action-guided self-distillation scheme that encourages consistent action predictions while enabling adaptive computation. Results show that AC2 -VLA significantly reduces inference cost while maintaining strong manipulation success rates, highlighting the effectiveness of action-guided adaptive computation for efficient closed-loop control. In summary, our contributions are as follows:

• We identify that computation redundancy in VLA models aligns more with action context than visual cues, and propose action-context-aware adaptive computation for efficient robotic manipulation.
• We propose AC2 -VLA with an action-prior router that adaptively coordinates cognition caching, token pruning, and layer skipping, supported by action-guided selfdistillation for robust sparsification.
• Experiments on robotic manipulation benchmarks show substantial latency and FLOPs reductions with minimal performance degradation, and confirm consistent gains over baselines with extensive ablations.

2 RelatedWork

2.1 Vision-Language-ActionModels

The rapid progress of VLMs, together with large-scale robot data collections such as Open X-Embodiment [O'Neill et al., 2023], has accelerated the emergence of generalist embodied policies that unify perception, instruction following, and action prediction. Among modern VLA systems, the RT series, such as RT-1 and RT-2, demonstrates that tokenbased autoregressive decoding can scale to broad task sets via large multimodal backbones [Brohan et al., 2022; Brohan et al., 2023]. Building on similar foundations, OpenVLA [Kim et al., 2024] further systematizes training and evaluation for vision-language-action modeling at scale. In parallel, diffusion- and flow-based policies have become a compelling alternative for continuous control, where action generation is formulated as conditional denoising or flow matching and sampled as coherent trajectories, such as CogACT [Li et al., 2024a] and the π0/π0.5 line [Black et al., 2024; Black et al., 2025]. Despite their strong generalization and stable closed-loop behaviors, these models share a common deployment bottleneck: regardless of paradigm, VLA inference is dominated by repeatedly executing a large multimodal backbone, leading to high latency in real-time control.

2.2 Efficient VLA Strategies

Prior work on efficient VLA strategies broadly falls into four categories: lightweight model design, dynamic routing and conditional execution, compression via pruning and quantization, and temporal reuse through caching or compute switching. Lightweight designs reduce per-step cost by scaling down backbones or streamlining training and inference, such as TinyVLA [Wen et al., 2024], SmolVLA [Shukor et al., 2025], and FLOWER [Reuss et al., 2025]. Dynamic routing methods reduce computation by activating only a subset of the model, such as MoLe-VLA [Zhang et al., 2025] and instruction-driven routing with structured sparsification in CogVLA [Li et al., 2025a]. Compression-oriented approaches aim to remove redundant tokens and layers or co-design pruning with quantization, such as LightVLA [Jiang et al., 2025], FlashVLA [Tan et al., 2025], and SQAP-VLA [Fang et al., 2025]. Temporal reuse exploits redundancy across adjacent control steps by reusing cognition or switching computation modes, such as VLA-Cache [Xu et al., 2025], SP-VLA [Li et al., 2025b], and VOTE [Lin et al., 2025]. However, existing methods often target only one redundancy axis or rely on static heuristics without modeling action context, limiting robustness in closed-loop control. In contrast, AC2 -VLA uses action context for routing and unifies temporal reuse, spatial sparsification, and depth-wise conditional execution within an action-prior router.

3 Method

3.1 Overview

Given a visual observation xt and a language instruction u, a VLA model predicts an action chunk at:t+H with horizon H. We consider a generic VLA pipeline that factorizes action generation into a multimodal backbone and an action head:

Here, pφ can be instantiated as an autoregressive decoder or a diffusion/flow-based trajectory generator, depending on the underlying VLA policy.

In real-time deployment, inference is bottlenecked by repeatedly executing the VLM backbone at every control step. We observe structured computation redundancies along three complementary axes:

• (i) Temporal redundancy: backbone representations can be reused across adjacent timesteps;
• (ii) Spatial redundancy: only a subset of vision tokens is necessary for action prediction;
• (iii) Depth redundancy: executing fewer backbone layers often suffices with minimal performance loss.

To exploit these redundancies within a unified mechanism, AC2 -VLA introduces an action-prior router, as shown in Fig. 2. At each timestep, conditioned on the action-centric context ct, an action-prior router generates a set of computation gates for temporal reuse, spatial token selection, and depth-wise conditional execution:

where Nv and L denote the numbers of vision tokens and transformer layers, respectively. The cache gate pcache t determines whether to reuse cached backbone representations, while ptopk t and play t control token pruning and conditional layer execution. We train the router via teacher-student distillation with lightweight regularization to preserve densepolicy behavior under structured sparsification.

Overall, AC2 -VLA jointly optimizes temporal reuse, token selection, and conditional layer execution, enabling efficient inference for closed-loop robotic manipulation.

3.2 Action-Context-aware Unified Routing

AC2 -VLA is driven by an action-prior condition vector ct that explicitly encodes the robot's action context. In closed-loop control, the next action distribution is strongly shaped by the ongoing motion state, making the previous action at-1 a natural and inexpensive prior for allocating computation. We therefore use at-1, parameterized consistently with the action head, as the primary routing signal. When no previous action is available at the first step, we set at-1 = 0 and rely on lightweight visual and instruction summaries to generate initial gates.

Let Vt ∈ RNv×dv denote per-token vision features, and we summarize them with a mean-max mixture:

For language, we avoid an additional full forward by pooling embedded instruction tokens Et ∈ RT×d :

where ℓt denotes the last valid token index under the attention mask when available, otherwise we use mean pooling.

We embed the action-head step index τt with a sinusoidal encoder e(τt), which captures the internal generation progress of the action head. When reuse is enabled, we additionally include a cache-state cue sct encoding the quantized action-delta proxy used for cache keying, together with compact cache statistics and an availability probe. All inputs are projected to a shared hidden size and fused by an MLP:

In implementation, vision tokens and pooled summaries are detached before entering the router to prevent gradients from flowing into heavyweight backbone components through the routing pathway.

Given ct, the router predicts three gate families: a reuse probability pcache t , token keep scores ptopk t , and layer execution gates play t . We detail each gate in the following.

Cache reuse gate. We predict a scalar reuse probability:

where Tcache controls gate sharpness. A high pcache t indicates a reuse request, while an actual reuse occurs only when the cache lookup succeeds.

Token pruning gate. For each vision token vt,i, we predict its keep score via action-conditioned matching:

where gt,i is an optional lightweight bias such as a geometric prior derived from the current observation. During inference, tokens are compacted by keeping the top-ranked ones according to ptopk t,i .

Layer skipping gate. We predict per-layer execution probabilities:

with bias initialization that favors near-dense execution early in training. At runtime, transformer blocks with low play t,ℓ are conditionally bypassed to reduce depth-wise computation.

3.3 FromUnified Gating to Practical Speedups

We next describe how the unified gates translate into practical inference speedups in AC2 -VLA, through feature reuse across timesteps, spatial token pruning with compaction, and depth-wise conditional execution. Algorithm 1 summarizes the resulting inference-time procedure.

Cache reuse. When the router predicts a high reuse probability pcache t , we attempt to bypass the expensive multimodal backbone forward by querying a cognition cache. Specifically, we build a compact and robust cache key that captures both motion continuity and visual consistency. We first pool the vision tokens:

and form the cache key as

where Quant(‖∆at‖) is an action-delta norm proxy, and Hash(¯vt) is a lightweight vision hash for state matching. The robust hash normalizes ¯vt, applies a fixed random projection, and quantizes before hashing.

We distinguish a reuse request from an actual cache hit ht ∈ {0, 1}. When a hit occurs, we directly reuse the cached multimodal representation zt and skip the VLM backbone forward, otherwise we compute zt = fVLM(xt, u) as usual. To keep cache population consistent with the router's intent, we only write back the newly computed zt when reuse was requested but the lookup missed.

Token Pruning. Token gating determines which vision tokens should be retained. To obtain real wall-clock speedups beyond attention masking, we perform token pruning by physically removing pruned tokens and shortening the transformer sequence. Let mt ∈ {0, 1}Nv denote the keep mask. Compaction produces

where πt maps each kept token to its original patch index. We always keep at least one token to prevent degenerate empty sequences.

For Rotary Position Embedding (RoPE)-based backbones, naive reindexing after compaction would distort the original patch coordinates. We therefore preserve RoPE-consistent patch positions using πt:

Algorithm 1 AC2 -VLA inference for a single control step t

Input: visual observation xt, instruction u, previous action at-1, step index τt, cache C

Output: action chunk at:t+H

• 1: Vt ← fvis(xt); ¯vt ←MeanPool(Vt)
• 2: svt ← 12 (MeanPool(Vt) +MaxPool(Vt))
• 3: sut ←EmbedPool(u)
• 4: ct ← ffuse(ψa(at-1), ψv(svt ), ψu(sut ), ψτ(e(τt)), ψc(sct ))
• 5: (pcache t , ptopk t , play t ) ←R(ct)
• 6: ht ← 0
• 7: if REUSEREQ(pcache t ) then
• 8: ∆at ←DeltaProxy(at-1)
• 9: kt ← (Quant(‖∆at‖), Hash(¯vt))
• 10: (ht, zt) ← C.Get(kt)
• 11: end if
• 12: if ht = 0 then
• 13: mt ←TopKMask(ptopk t ); gt ←BinGate(play t )
• 14: ( ˜Vt,πt,N orig v ) ←Compact(Vt,mt)
• 15: post ←RoPEAlign(πt,N orig v )
• 16: zt ← fVLM(xt, u; ˜Vt, post, gt)
• 17: if REUSEREQ(pcache t ) then
• 18: C.Put(kt, zt) {write back only on request & miss}
• 19: end if
• 20: end if
• 21: at:t+H ← pφ(a | zt; τt)
• 22: return at:t+H

and assign text positions to start after the original patch span N orig v recorded during compaction:

During training, to stabilize optimization and maintain differentiability, we adopt a soft relaxation by scaling projected patch embeddings with token keep scores:

Hard compaction is applied only at inference time for maximum speedup.

Layer Skipping. We implement conditional depth execution by wrapping each transformer block with a lightweight gating mechanism. Given the hidden state h(ℓ) at layer ℓ, the gated residual update is defined as

where Fℓ(·) denotes the ℓ-th transformer block and αt,ℓ ∈ [0, 1] is the layer gate derived from play t,ℓ . During training, αt,ℓ remains soft; at inference, it is binarized so that inactive samples bypass the layer entirely.

For efficient execution, active samples are dynamically grouped into a sub-batch to run Fℓ(·), and the results are scattered back to the full batch.

3.4 Optimization

We train the router to preserve dense-policy behavior under sparse execution with:

Table 1: Google Robot success rates on SIMPLER under two evaluation settings. Most baseline results are reported by CogACT, and we add the AC2 -VLA row by evaluating our method under the same protocol.

Action-guided self-distillation. We use a teacher-student scheme, where the teacher runs the dense policy and the student executes routed sparse inference, including cache reuse, token pruning, and layer skipping. We distill both action outputs and cognition features:

where ˆϵ denotes the action prediction, and zt is the backbone representation. λϵ and λz control the relative weights, and D(·, ·) is a feature-matching distance.

Regularization and temporal smoothing. We add Lreg to enforce target token/layer budgets and supervise the reuse gate, and Ltemp to penalize abrupt changes in gating decisions across timesteps for stable closed-loop control.

4 Experiment

4.1 Experimental Setup

Backbones. We build AC2 -VLA on CogACT [Li et al., 2024a], a diffusion-based VLA model with a Prismatic-7B vision-language backbone and a DiT-Base action head. To isolate routing effects, we freeze the pre-trained vision and language backbones and optimize only the lightweight routing modules, while keeping the action head unchanged with 8 denoising steps.

Implementation Details. All experiments are conducted on a node with NVIDIA RTX 5090 GPUs. We initialize from the CogACT-Base checkpoint [Li et al., 2024a] and train on the Bridge subset of Open X-Embodiment [O'Neill et al., 2023] for 3,000 steps using AdamW with batch size 48 and learning rate 1 × 10-6 . We use an action horizon of H = 15 with 8 diffusion steps. Unless noted otherwise, AC2 -VLA enables action distillation, sets the maximum token pruning ratio to 0.6, and uses a cache reuse threshold of 0.2.

Benchmarks. We evaluate on SIMPLER [Li et al., 2024b], a high-fidelity simulation benchmark for robotic manipulation that aims to narrow the real-to-sim gap. We report results on two robot embodiments under three protocols:

• Google Robot Visual Matching: Tests generalization in visually matched real-world conditions on tasks including Pick Coke Can, Move Near, Open/Close Drawer, and Place Apple.
• Google Robot Variant Aggregation: Introduces variations in background, lighting, and distractors, providing a more challenging robustness setting.
• WidowX Visual Matching: Evaluates fine-grained manipulation on WidowX with tasks including Put Spoon on Towel, Put Carrot on Plate, Stack Cube, and Put Eggplant in Basket.

Following standard SIMPLER protocols, we use 3 Hz control with 513 Hz simulation for Google Robot and 5 Hz control with 500 Hz simulation for WidowX. Episodes are capped at 80 steps for Google Robot and 120 steps for WidowX to penalize inefficient or stalled behaviors.

Baselines. We compare AC2 -VLA with two groups of baselines: generalist dense VLA policies and efficiency-oriented methods.

Generalist VLAModels: We report results for RT-1 [Brohan et al., 2022], RT-2-X [Brohan et al., 2023], Octo [Octo Model Team et al., 2024], OpenVLA [Kim et al., 2024], and our backbone CogACT [Li et al., 2024a] in full precision, which serve as dense upper bounds on task success.

Efficiency-Oriented Methods: We include representative acceleration approaches, including VLA-Cache [Xu et al., 2025] for temporal reuse, EfficientVLA [Yang et al., 2025] for static pruning, MoLe-VLA [Zhang et al., 2025] for conditional layer skipping via mixture-of-layers routing, and FastV [Chen et al., 2024] as a lightweight pruning baseline. These comparisons characterize the trade-off between compute efficiency, measured by FLOPs and latency, and manipulation success across diverse tasks.

4.2 Comparison with State-of-the-Art

We compare AC2 -VLA on SIMPLER, reporting task success in Tables 1 and 2 and the speed-accuracy trade-off in Table 3.

Task Performance. AC2 -VLA achieves strong control performance across evaluation protocols. On Google Robot Visual Matching, it reaches 76.8% average success, outperforming the dense CogACT baseline at 74.8% and larger models such as RT-2-X at 46.3%. Gains are most pronounced on precision-critical tasks, e.g., Drawer Opening improves from 71.8% to 80.6%, suggesting that action-prior-guided sparsification helps suppress distractors and stabilizes decision making. On Variant Aggregation, AC2 -VLA matches the fullprecision baseline with 61.6% versus 61.3%, while consistently surpassing RT-1 and OpenVLA.

Efficiency and Computational Cost. AC2 -VLA substantially reduces the inference cost of CogACT. As shown in Table 3, it uses 29.4% of the original FLOPs, yielding a 1.79× wall-clock speedup. Notably, this acceleration does not compromise performance and even improves success over dense CogACT, indicating that the removed computation largely corresponds to redundant or distracting features for closedloop control.

4.3 Ablation Study

We ablate AC2 -VLA on SIMPLER Google Robot Visual Matching to validate key design choices. We focus on the three efficiency axes controlled by the router, namely token pruning, layer skipping, and cognition reuse, and analyze their interaction under comparable budgets. Ablation results are summarized in Table 4. We observe that each component provides complementary benefits, and the full model achieves the best speed-accuracy trade-off when all three axes

are jointly enabled:

• Cache reuse. Without cognition reuse, success drops to 70.5% and speedup decreases to 1.66×, suggesting that temporal reuse improves both efficiency and closed-loop stability. Fig. 3 illustrates adaptive layer execution and cache hit behavior over time.
• Token pruning. Removing token pruning reduces speedup to 1.52×, showing that spatial sparsification contributes most to acceleration. Fig. 4 visualizes the token-level routing patterns.
• Layer routing. Disabling layer routing drops the success rate to 67.4% under similar FLOPs, indicating that conditional depth execution helps retain high-level reasoning while reducing compute.
• Full model. AC2 -VLA attains 76.8% success while delivering a 1.79× speedup, demonstrating the effective-

ness of jointly leveraging spatial, depth-wise, and temporal redundancies.

4.4 More Exploration

Beyond component ablations, we further analyze the hyperparameter space and emergent behaviors of AC2 -VLA, focusing on the joint sparsity trade-off and the effect of cognition caching on closed-loop stability.

Token-layer sparsity and the efficiency-accuracy tradeoff. We perform a grid search over the token keep ratio rtopk and the executed layer count Nlay, as shown in Fig. 5. The results exhibit a clear Pareto frontier, where rtopk = 0.4 and Nlay = 28 achieves the best trade-off, reaching 1.79× speedup with 76.8% success. This suggests that many visual tokens are dispensable, while sufficient depth remains important for reasoning over the retained tokens.

When cache reuse improves robustness beyond speed. Interestingly, cache reuse can improve robustness in addition to reducing compute. As shown in Table 5, setting the cache threshold to τcache = 0.2 yields 87.1% success, outperforming the dense baseline by +12.3%. We attribute this gain to improved temporal consistency: standard per-frame inference can amplify high-frequency visual noise and induce action jitter, whereas reusing zt when the action context is stable effectively smooths decision making and stabilizes control.

Sensitivity to key hyper-parameters. We next vary the token keep ratio and the maximum executed depth to examine how accuracy degrades under more aggressive sparsification.

• Token sparsity. Performance remains stable down to rtopk = 0.4, but collapses at rtopk = 0.2 with 33.3% success, indicating a minimum visual information requirement for manipulation.
• Depth. The policy remains competitive with Nlay = 28 at 77.3% success, while reducing below 24 layers causes a sharp drop, suggesting that sufficient depth is critical for complex tasks.

5 Conclusion

We present AC2 -VLA, an action-context-aware framework for efficient Vision-Language-Action inference. By introducing a unified router that allocates computation across spatial, depth, and temporal dimensions based on the robot's manipulation state, AC2 -VLA addresses the limitations of efficiency methods driven solely by visual complexity and enables adaptive closed-loop control. Experiments on the SIMPLER benchmark demonstrate a superior efficiency-accuracy tradeoff, achieving a 1.79× speedup and reducing FLOPs to 29.4% of the dense baseline while improving task success. These results indicate that action-guided sparsification acts as both an efficiency mechanism and a regularizer, suppressing visual distractors and promoting temporal consistency. Overall, AC2 -VLA suggests that aligning computation with action is a more effective paradigm for embodied intelligence than static compression, and points toward adaptive inference as a key ingredient for scalable generalist robot policies.

Ethical Statement

There are no ethical issues.

Acknowledgments

This work was supported in part by the New Generation Artificial Intelligence-National Science and Technology Major Project (No. 2025ZD0123000), the Central Guidance on Local Science and Technology Development Fund of Shanghai City (No. YDZX20253100002004), the China Postdoctoral Science Foundation (No. 2025M781504), the New Cornerstone Science Foundation through the XPLORER PRIZE, and the Fundamental Research Funds for the Central Universities.