Empowering Precise Embodied Agents with Executable Analytic Concepts as Semantic-Physical Blueprints

1 Introduction

The realization of embodied agents capable of autonomous reasoning and high-fidelity physical interaction remains a fundamental objective of artificial intelligence research. Recent advancements in Large Language Models (LLMs) [Achiam et al., 2023] and Multimodal VisionLanguage Models (VLMs) [Zhang et al., 2024; Hurst et al.,

∗Corresponding Author

2024] have endowed these agents with a sophisticated substrate of world knowledge, enabling them to interpret openvocabulary instructions and infer complex goal states without the need for exhaustive task-specific demonstrations. This shift has catalyzed several core cognitive faculties in contemporary agentic research, most notably hierarchical task planning [Ahn et al., 2022; Driess et al., 2023], reflexive error detection [Duan et al., 2024a], and spatial-semantic grounding [Huang et al., 2025; Huang et al., 2023]. Consequently, agent architectures have converged toward a dominant hierarchical paradigm, where a high-level "reasoning layer" leveraging VLMs provides semantic abstractions and tasklevel guidance to a low-level "execution layer" responsible for physical interaction [Ma et al., 2024; Shi et al., 2025].

However, a fundamental theoretical gap persists in the agent's cognitive model: the "semantic-to-physical gap". While VLMs excel at 2D image understanding and linguistic dialogue, they lack an inherent formal model of physical intelligence. Existing agent engineering methods often attempt to bridge this by either fine-tuning models into endto-end controllers-which suffers from high data costs and poor generalization-or by feeding semantic features directly into control policies. The latter is particularly inefficient, as it forces the agent to relearn fundamental physical principles from scratch without a structured representation language to link symbolic concepts to metric-space actions. Consequently, embodied agents still struggle with the precise numerical reasoning and kinematic awareness required for tasks demanding high accuracy.

To bridge the semantic knowledge inferred by VLMs and the physical realm in which robots operate, we leverage a novel knowledge representation paradigm for embodied agents based on the notion of Analytic Concepts [Sun et al., 2024]. An analytic concept serves as a procedural, mathematically defined schema that captures the generalized physical commonality of an object or task. We introduce GRACE (From VLM-based Grounding to Robotic manipulation through Analytic Concept Execution), an agent framework that operationalizes these concepts as semantic-physical blueprints. In this paradigm, the agent does not merely recognize an object; it annotates the environment by selecting and instantiating a mathematical blueprint from a concept library. This transforms the agent's internal reasoning from abstract chain-of-thought (COT) into an Executable Analytic Concept

(EAC)-a physics-grounded plan whose parameters are directly compatible with motion planners. Unlike traditional scene graphs or point clouds, an EAC provides a structured, programmable representation encoding constituent parts, spatial topology, and kinematic primitives. Crucially, EACs are not just static annotations but dynamic computational templates that that undergo parameter reification-automatically estimating physical attributes from sensory inputs to translate high-level semantics into actionable physical constraints.

By mediating between symbolic reasoning and physical execution through this structured paradigm, our approach provides a systematic architectural framework for developing embodied agents with inherent interpretability and highprecision capabilities. Our main contributions are:

• A Unified Conceptual Framework for Embodied Cognition: We introduce GRACE, a plug-and-play agent architecture that elicits the physical intelligence of VLMs through structured, physics-aware representations. This framework provides a formalized interface between high-level cognitive reasoning and low-level physical execution, offering a new perspective on how agents internalize world knowledge.
• The EAC Paradigm for Embodied Cognition: We develop a systematic methodology that adopts EAC as a knowledge annotation paradigm. The policy scaffolding pipeline enables the agent to translate semantic knowledge into "Semantic-Physical Blueprints," effectively grounding symbolic reasoning into a mathematically rigorous and executable domain.
• Extensive Empirical Validation: We demonstrate our approach's outstanding performance in a wide range of manipulation tasks, showcasing the remarkable success rate in both simulated and real-world environments. We also highlight the compatibility of our EAC-based approach with VLA architecture.

2 ProblemFormulation

The fundamental objective of an embodied agent is the reification of semantic intent into metric-space execution. We formalize this as a mapping problem where an agent must translate high-level, potentially under-specified human language into precise physical interactions.

2.1 Input and Output Spaces

At any decision epoch t, the agent is provided with:

• Language Instruction (l): A high-level, often abstract or under-specified natural language command.
• Visual Observation (Vt): A snapshot of the environment (e.g., an RGB-D images or point cloud) providing the spatial context of the relevant objects.
• Robot Proprioception (St): The robot state, including current joint configurations, end-effector pose, and gripper status. This ensures the generated actions are kinematically feasible and context-aware.

We define the aggregate observation at time t as Ot = {Vt, St}. The agent's goal is to generate an Executable Trajectory: τ = {a0, a1, . . . , aH}, where each action ak ∈ R7

represents the robot's Cartesian displacement and gripper configuration. The first six dimensions encode translation and rotation vectors, while the last component gk ∈ {0, 1} indicates the binary gripper state.

2.2 The Semantic-to-Physical Mapping

The core challenge of high-precision embodied interaction lies in operationalizing the mapping functionM:

where C denotes a library of structured prior knowledge, and ΩΦ represents the motion manifold constrained by the object's physical properties Φ. Unlike generic endto-end approaches, M transitions the task from a highdimensional regression problem to a knowledge-driven instantiation process. Central to this work is that by incorporating C as a repository of Analytic Concepts, the agent leverages pre-defined object-centric physical schemas to disambiguate under-specified instructions l and anchor them into the metric-geometric context of Ot. A task is successful only if the agent navigates ΩΦ with high precision, ensuring semantic fulfillment while strictly adhering to the mechanical invariants of the environment.

3 Executable Analytic Concept

To bridge the representational gap between symbolic reasoning and metric execution, we leverage the Analytic Concept (AC) [Sun et al., 2024] as a structured knowledge annotation paradigm that allows the agent to interpret the world through "Semantic-Physical Blueprints."

3.1 Formal Definition

An AC, denoted as E, is an explicit, programmatic and hierarchical representation that formalizes the agent's internal model of an object or part. Unlike flat representations, an AC is defined recursively to capture the compositional nature of articulated systems:

• Constituent Parts (P): A set of lower-level Part ACs {E1, E2, . . . , En}. At the lowest level of this hierarchy, the part is grounded as a Geometric Concept in Fig. 1a, defined by parametric surfaces S(p; θg) = 0 (e.g., a cylinder for a handle or a plane for a cabinet face).
• Spatial Topology (T ): A set of transformations {Tij} that define the spatial configuration and relative poses between constituent parts in P. This ensures the structural integrity of the composite object (e.g., the handle's fixed offset relative to the door).
• Kinematic Primitive (Φ): A library of IntentionConditioned Guidance Functions that instantiate specific control laws u(t) based on the interaction semantics. For complex E, the root-level Φ is generated by inheriting and composing part-level functions and their associated parameters.

✞ # Structural Blueprint Code class Curve_Handle: def __init__(self, curveL, radian, innerRadius, outerRadius , rotation, position, ...): self.shape = Ring(curveL, radian, innerRadius, outerRadius) ... class Sunken_Door(): def __init__(self, size, sunken_size, handle_size, handle_pos, rotation, position, ...): self.bottom_shape = Cuboid(size) self.top_shape = Rectangular_Ring(sunken_size) self.handle = Curve_Handle(handle_size, handle_pos) ✝

✞ # Manipulation Blueprint Code def get_grasp_pose(obj, initial_pose): if (isinstance(obj, Curve_Handle)): local_pose = translate_world2local(pose, init_pose) grasp_pose = obj.apply_pose(local_pose) if (isinstance(obj, Sunken_Door)): ... return translate_local2world(grasp_pose) def push(obj): force_direction = get_direction(obj.axis...) def pull(obj): force_direction = get_direction(obj.axis...) ✝

✆

Figure 1: Example of analytic concepts. (a) Geometric Concept Assets. Each asset exposes its free parameters (top), canonical structure (mid), and partial affordance cues (bottom). (b) Structural Blueprint: higher-level objects or parts are procedurally composed by wiring multiple geometric assets together, forming a parametric graph with their spatial layout and structural relationships. (c) Manipulation Blueprint: parameterised routines compute grasp poses and force directions that exploit the affordances encoded in the underlying structure.

3.2 Blueprint Instantiation

The AC serves as a programmatic template. To bridge the gap between abstract knowledge and physical interaction, the agent performs Knowledge Annotation to instantiate a taskspecific Semantic-Physical Blueprint B as an Executable Analytic Concept (EAC). Unlike static templates, the instantiation of B is a dynamic mapping conditioned on the current task l and visual observation Vt:

To achieve this, the agent need to identifies the task-relevant constituent parts PB ⊆ P (see Fig. 1b ) and specific procedural guidance function uB (see Fig. 1c). By extracting metric features from Vt via neural regressor tools, the agent instantiates the Geometric Parameters and Transition Parameters. This blueprint acts as a policy scaffold, providing the necessary geometric and physical "rails" that guide the agent's low-level motion planner to generate the final trajectory τ. Specifically, by generalizing the concepts towards certain objects, various knowledge associated with the concepts can be automatically propagated to all these objects.

4 The Agent Framework

4.1 AgenticWorkflowOverview

The GRACE agent orchestrates a hierarchical decisionmaking pipeline that bridges abstract intent with deterministic physical control. As illustrated in Fig. 2, the workflow is divided into three cognitive stages:

• Task Parsing: The agent resolves linguistic ambiguity by grounding instructions in the visual scene.
• Policy Scaffolding: The agent performs a sequence of specialized tool calls to construct an Executable Ana-

☎

✆

(b) Structural Blueprint

☎

lytic Concept (EAC). This transforms "insight" into a structured, parameter-driven blueprint.

• Policy Execution: The agent exports the EAC's computational parameters to a downstream policy model for low-level motor control.

We give the pseudocode of the overall process in the appendix.

4.2 Spatial-Aware Task Parsing

Object Centric Symbolic Graphing. As the foundational stage for perception and language grounding, the agent first interprets the l by constructing a structured graph G = (V, E), where V denotes the set of object nodes-each represented as a structured dictionary containing id, name, and state-and E constitutes a set of directed spatial relationships between objects, each expressed as a triple eij = (vi, r, vj). This object-centric symbolic graph provides a semantically rich and structurally explicit representation for subsequent reasoning stages. This process distills the "what" and "where" from the command, delivering a clean symbolic input for downstream reasoning.

Hierarchical Task Decomposition. For long-horizon tasks, the agent functions as a high-level planner, decomposing the primary task into discrete sub-tasks {(l1, d1), . . . , (ln, dn)}. Each sub-task is associated with a computable verification condition di, ensuring the agent can autonomously validate successful completion before proceeding. Each sub-task then undergoes an execution loop, as depicted in Fig. 2.

4.3 Policy Scaffolding

The Policy Scaffolding stage represents the core cognitive layer of the GRACE agent. In this phase, the agent transitions from semantic reasoning to physical grounding by executing a series of inter-tool protocols that construct the Executable Analytic Concept (EAC). This process is governed by a structured workflow that transforms raw observation into instantiated physical blueprints.

Target Identification. Upon receiving the structured object graph G = (V, E), the agent initiates the Perceptual Grounding Protocol. The agent ground its geometric structure in a formalized representation by interfacing with a Library of Analytic Concepts. This library acts as an Analytic Knowledge Base containing parametric models of structural archetypes. This involves calling a sequence of Visual Foundation Tools:

• Semantic Concept Retrieval: The agent interfaces with a Library of Analytic Concepts, which serves as an organized repository of common structural archetypes (e.g., primitive geometries, specific handle designs). Each concept is paired with a short natural-language synopsis. Based on the sub-task instructions identified in the task parsing stage, the agent prunes the library and performs a VLM-driven query to match the visual part to its corresponding symbolic archetype.
• Open-Vocabulary Localization: The agent invokes GroundingDINO [Liu et al., 2024] to identify and localize target part or object nodes within the scene.
• Instance Segmentation Tool: The Segment Anything Model (SAM) [Kirillov et al., 2023] is triggered to gen-

erate high-fidelity 2D masks M for task-relevant entities.

• 3D Back-Projection: Using the depth map, the agent executes a geometric tool call to back-project M into the 3D space, producing object-centric point clouds P .

Structural Parameter Estimation. Once the symbolic concept is retrieved and the point cloud is segmented, the agent transitions to Parameter Reification. This process transforms the abstract concept into an executable program by estimating concrete metric variables from the point cloud P. These parameters are of two types:

• Structural parameters encode the concept's intrinsic geometry of the analytic concept (e.g., the size l, w, h of a sunken door). To estimate them, we encode the point cloud P into a deep feature vector using a Point Transformer [Zhao et al., 2021]. This feature vector is then fed into multiple specialized MLP heads, each regressing a specific structural parameter.
• 6-DoF pose parameters locate the concept's global position and orientation. We leverage the FoundationPose [Wen et al., 2024] to estimate the 6-DoF pose of the target part in the global coordinate system. These are recovered analytically by combining the object's known simulation pose with the newly estimated structural variables.

ManipulationBlueprinting. While the structural blueprint defines the geometry, the Manipulation Blueprinting protocol specifies the functional "how-to". The agent treats manipulation knowledge as an Executable API. Firstly, the agent retrieves candidate Manipulation Functions (e.g., "pull-type grasp") from the knowledge base. The VLM selects the module that best satisfies the high-level intent, mapping semantic perception directly onto a mathematical interaction law.

With the parameters, a physically grounded grasp pose G can be calculated according to the initial grasp pose G∗ by solving the analytic equations of the selected module. For instance, a curved handle's interaction is governed by the transformation:

Once G is fixed, the force-direction formula-conditioned by the verb or manipulation type chosen by the VLM (e.g., pull vs. push)-is invoked to produce the vector F, ensuring that the applied force is semantically aligned with the selected action and correctly oriented on the target part. The final output of the scaffolding process is an instantiated EAC, exported as a lightweight Python Object. This serves as the Standard Execution Protocol for the downstreammotion planner, closing the loop from abstract language to deterministic control.

4.4 Low-Level Motion Execution

Blueprint Execution. The structural and manipulation blueprints jointly output two quantities in the local frame of the target part: a grasp pose Glocal =(tlocal, rlocal), and a force direction Flocal. Running the blueprint therefore reduces to transforming the local descriptors into the world frame and then feeding them to a standard motion planning stack.

Transformation to World Coordinates. Let M ∈ R4×4 denote the homogeneous transform of the target part with respect to the world frame, obtained from perception or simulation. For every point-set or inequality description F in the blueprint we apply F((x, y, z, 1)⊤) ≤ 0 =⇒ F(M-1 (x, y, z, 1)⊤) ≤ 0, thereby re-expressing all structural constraints globally. The grasp pose is mapped by Gworld = MGlocal. For rotationally symmetric geometries we additionally enforce a minimal-rotation constraint on rlocal to obtain a unique orientation. The force vector is transformed analogously: Fworld = RFlocal, where R is the rotational part of M.

Motion Planning Interface. The instantiated EAC parameters are forwarded to a policy model (e.g., a standard motionplanning stack or a learned neural controller). This model realizes the final joint-space command sequence by synthesizing a collision-free path that strictly adheres to the force vector Fworld and pose Gworld. By decoupling the analytic "what" from the motor "how", GRACE enables high-precision execution across diverse hardware platforms.

5 Experiments

In this section, we conduct comprehensive experiments and analyses to answer the following key questions: (1) How does the GRACE framework perform against state-of-the-art baselines across diverse manipulation tasks? (2) To what extent can the agent accurately discover the analytic concept and ground their functional affordances in complex articulated objects? (3) Can the integrated agentic pipeline successfully execute long-horizon, multi-stage tasks in real-world settings?

5.1 Manipulation Evaluation in Simulation

To rigorously evaluate the execution efficacy of the GRACE framework, we leverage SimplerEnv [Li et al., 2024c] as our primary simulation benchmark. SimplerEnv is chosen for its high-fidelity physical modeling and its standardized suite of real-world robotic manipulation tasks, which provide a reliable proxy for physical hardware constraints. We conduct extensive quantitative comparisons on Google Robot and Widow-X task suites, benchmarking GRACE against stateof-the-art Vision-Language-Action (VLA) models including Octo [Ghosh et al., 2024], OpenVLA [Kim et al., 2024], and several concurrent spatial-aware baselines [Qi et al., 2025; Qu et al., 2025; Li et al., 2024b]. This evaluation specifically targets the agent's ability to transcend the precision limitations of end-to-end black-box models by utilizing the EACbased decision framework.

On the four Widow-X tasks (Table 1), GRACE powered by GPT-4o achieves an average success rate of 86.1%, clearly outperforming the strongest published baseline, SoFar (58.3%). Although it is not the best on every single task, GRACE never performs poorly, maintaining consistently high scores across the entire suite. The pattern repeats on the Google-robot tasks (Table 2): GRACE(GPT-4o) attains 90.1%mean success, exceeding the best prior result by almost 30 pp. Notably, on the articulated Open/Close Drawer task the jump is the largest, rising from 29.7% (SoFar) and

36.2% (SpatialVLA) to 90.3% with GRACE for "Variant Aggregation", highlighting the advantage of EACs when precise kinematic reasoning is required.

To isolate the contribution of analytic concepts, we retrofit SpatialVLA by replacing its native, end-to-end action output with EAC-guided motion planning when the gripper approaches the target; this variant is denoted SpatialVLA-EAC. The simple swap boosts SpatialVLA's average success to 69.8% on Widow-X and to 83.4% on the Google robot, demonstrating that EACs can be used as a plug-and-play module to substantially enhance existing VLA architectures. Finally, GRACE's performance is insensitive to the underlying VLM. The fully open-source Qwen2.5-VL backend trails GPT-4o by only 1-2 pp on both robot families, yet still outperforms every external baseline, confirming that the bulk of the gain comes from the analytic-concept layer rather than the choice of language model.

5.2 Evaluation of Concept Grounding

This experiment evaluates the agent's capacity to perform analytic concept discovery and functional affordance grounding within complex articulated objects. Unlike general manipulation, success in this context requires the agent to correctly identify the structural blueprints and adhere to the resulting kinematic manifolds. We conduct these evaluations within the SAPIEN environment, utilizing the standardized settings established by Where2Act [Mo et al., 2021]. The task requires the agent to transition an articulated object (e.g., cabinets, faucets, drawers) from an initial state to a target configuration. We compare our method against three baselines, i.e., Where2Act, Where2Explore [Ning et al., ] and ManipLLM [Li et al., 2024a], each representative of a distinct modelling paradigm for articulated-object manipulation. To isolate the contribution of VLM reasoning, we also report an ablated variant, GRACE-w/o-VLM, in which the concept-selection step is replaced by ground-truth ontology labels.

Quantitative Results. Table 3 demonstrates that GRACE(GPT-4o) achieves the highest scores across all categories. For instance, it attains 0.65 for "faucet" objects and 0.91 on "cabinet" doors, significantly outperforming ManipLLM, which scores 0.26 and 0.71, respectively.

These results decisively surpass both pixel-level affordance methods and the LLM-based ManipLLM. The substantial numerical margins underscore the advantage of integrating VLM-based reasoning with analytically grounded control.

Replacing the oracle concept labels with GPT-4o's automatic selection reduces performance only marginally-from an average of 0.80 to 0.77. This negligible 3% drop indicates that the bottleneck for the remaining failure cases is primarily localized within occasional semantic misclassifications by the VLM, rather than inherent limitations of the analytic concepts themselves. Once the correct concept is retrieved, the resulting execution remains highly deterministic and reliable.

Qualitative Insights in Real World. To validate our approach qualitatively, we illustrate the execution success and generated trajectories for five distinct articulated objects in Fig. 3. These real-world outcomes demonstrate the effectiveness of EAC for physics-grounded planning. The visualizations affirm that the agent's cognitive framework functions as intended: the Visual Language Model successfully completes Part Identification and EAC Construction, while the underlying manipulation blueprints manage the complexities of physical compliance. This modular design enables the robot to perform high-precision interactions consistently.

5.3 Long-Horizon Tasks in Real-world

To validate the deployment-readiness and generalization of the GRACE framework, we conducted real-world experiments in a tabletop environment using a Realman RM75 robotic arm equipped with a parallel gripper. To rigorously test the system's integration, we designed a long-horizon manipulation tasks involving diverse objects. This task requires the agent to maintain coherent semantic reasoning while executing a series of precise, multi-stage physical interactions. As illustrated in Fig. 4 , the agent autonomously decomposed the high-level goal into a sequence of 19 distinct instructions. The successful execution of this long-horizon chain underscores the efficacy of our decoupled architecture: the VLM ensures inter-stage semantic continuity, while the EACs guarantee intra-stage kinematic adherence.

6 RelatedWork

Foundation Model Paradigms in Robotics. Current research leveraging foundation models for manipulation can be categorized into two primary paradigms. The first paradigm employs monolithic end-to-end Vision-LanguageAction (VLA) models [Kim et al., 2024; Ghosh et al., 2024; Shao et al., 2025] that directly map multimodal observations to low-level actions, though they are often bottlenecked by massive data requirements and limited zero-shot precision. The second paradigm adopts a hierarchical structure where a high-level VLM generates intermediate representations-such as symbolic code, sparse keypoints, or discrete skill selections-to guide a separate low-level policy [Ahn et al., 2022; Huang et al., 2023; Duan et al., 2024b; Huang et al., 2024; Pan et al., 2025]. GRACE focuses on how an agent can establish a reliable intermediate structured representation through Executable Analytic Concepts. By formalizing these concepts, our framework provides the deterministic, numerically computable geometric and physical "rails" necessary to bridge fluid semantic reasoning with precise, kinematically compliant execution. This structured alignment ensures that the agent's high-level intent is grounded in the object's inherent mechanical manifold, overcoming the sparsity and lack of physical depth in existing intermediate representations.

7 Conclusion

In this paper, we introduced GRACE, a plug-and-play agentic framework that bridges the gap between high-level semantic reasoning and high-precision physical interaction. By grounding visual observations into Executable Analytic Concepts, GRACE effectively translates open-vocabulary instructions into kinematically compliant robot actions. Extensive experiments on simulation and real world demonstrate marked gains in manipulation success rates, particularly on kinematically challenging tasks. In future work, we plan to extend GRACE to multi-fingered hands and to explore onthe-fly concept refinement from real-world interaction data.

Acknowledgments

The authors declare that this research received no external funding.