VLAs Are Confined yet Capable of Generalizing to Novel Tasks

VLAs Are Confined yet Capable of Generalizing to Novel Tasks

Quanyi Li

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence
AI and Robotics. Pages 7583-7591. https://doi.org/10.24963/ijcai.2026/843

Vision-language-action models (VLAs) often achieve high performance on demonstrated tasks but struggle significantly when required to extrapolate, recombining skills used in different tasks in novel ways. For instance, VLAs might successfully put the cream cheese in the bowl and put the bowl on top of the cabinet, yet still fail to put the cream cheese on top of the cabinet. This motivates us to investigate whether VLAs merely overfit to demonstrated tasks or still hold the potential to extrapolate. Our study uses text latent as the ingredient; it is a task-specific vector derived from the models’ hidden states. It thus encodes semantics necessary for completing a task and can be used to reconstruct the associated task behavior by writing it to the model’s residual stream. Furthermore, we find that skills used in distinct tasks can be combined to produce novel behaviors by blending their respective text latent. Applying this to π0, we increase its success rate from 9% to 83% on the proposed libero-ood benchmark, which features 20 tasks extrapolated from standard LIBERO tasks. This reveals that the skill representations encoded in text-latent are individual yet composable, while π0 fails to autonomously combine these representations for extrapolation. This also validates the design of libero-ood; it comprises tasks that the model fails, yet should be able to complete. We then tested other VLAs on libero-ood, and none of them achieved a success rate higher than 21%. Further analysis reveals VLAs share a common pattern to exhibit spatial overfitting, associating object names with where the object is spatially located in the demonstrated scene rather than achieving true object and goal understanding.
Keywords:
Safety, trustworthiness, generalizability, and evaluation: Verification and validation methods for AI-powered robotic systems
AIR: Safety, trustworthiness, generalizability, and evaluation
Intentional, causal, and intuitive physics reasoning: Representations and inference for intentions, goals, and affordances
Learning to understand, generalize, and explain actions: Robust generalization and transfer across tasks, objects, environments, embodiments, and long-horizon scenarios