Advancing Embodied Agent Security: From Safety Benchmarks to Input Moderation

1 Introduction

Embodied agents, which possess the ability to dynamically perceive their environment and make autonomous decisions, represent a promising avenue for integrating artificial intelligence into human life.

∗Corresponding Author.

This technology exhibits significant potential across diverse fields, such as military operations and domestic services. However, a fundamental prerequisite for the successful deployment of embodied agents in real-world applications is ensuring the behavioral safety of these intelligent agents.

In general, for embodied agents, large language models (LLMs) often serve as the core component for task planning and decision-making, with their outputs subsequently executed by unmanned systems [Vemprala et al., 2024; Singh et al., 2023; Song et al., 2023; Huang et al., 2022]. In this process, LLMs generate task plans and action command codes based on the input requirements provided to them. Consequently, the inputs to embodied agents play a pivotal role in determining whether the subsequent actions of the intelligent system will be safe or potentially harmful. This underscores the critical importance of effective input moderation mechanisms for ensuring the safety and reliability of these systems.

Existing research on input moderation can be broadly categorized into two approaches: external intervention and internal blocking. External approaches typically employ standalone detection models like Moderation API 1 and Llama Guard [Inan et al., 2023] to filter input before LLM processing, but incur high computational costs due to complex detection models and external API calls. Instead, internal blocking methods leverage the hidden states generated during model inference to detect and block malicious input, as demonstrated by Legilimens [Wu et al., 2024] and ToxicDetector [Liu et al., 2024]. These methods strike an exceptional balance between effectiveness and efficiency, making them ideal for real-time applications, such as embodied agents.

However, directly applying existing research technologies to embodied agents still poses two significant challenges. The first challenge is the lack of a dedicated safety benchmark tailored for embodied agents to support safety behavior learning and testing. Unlike general-purpose LLMs, embodied agents operate in dynamic real-world environments, necessitating the development of a safety benchmark that integrates realistic safety classifications and task-specific instruction sets tailored to their unique operational requirements. The second challenge lies in the limitations imposed by existing functional prompt schemes on input moderation performance. Functional prompts, which are crucial for activating the planning and decision-making capabilities of LLMs [Sarch et al., 2023; Zeng et al., 2023], are dynamically configured based on the agent's form, task, and execution environment. However, current methods embed inputs into fixed prompts (e.g., "You are a helpful assistant") for training and testing, rendering them unreliable in dynamically changing contexts. Additionally, the diversity of prompts will complicate effective data sharing across different agents. Therefore, considering the unique characteristics and operational demands of embodied agents, it is essential to develop a dedicated safety benchmark and an internal blocking-based moderation method that effectively mitigates the impact of functional prompts.

1 https://platform.openai.com/docs/guides/moderation/overview

In this paper, we propose a comprehensive moderation framework designed to ensure input safety for embodied agents. This framework spans the entire pipeline, including taxonomy definition, dataset construction, moderator design, and model training and evaluation. Specifically, it introduces EAsafetyBench, a safety benchmark tailored to support the training and evaluation of moderators for embodied agents. Furthermore, it incorporates Pinpoint, a lightweight and efficient input moderation method that leverages a masked attention mechanism to extract features from hidden states generated during LLM inference and classify inputs. This approach ensures robust and reliable performance, even in the dynamically changing contexts faced by embodied agents.

We summarize our key contributions as follows:

• We present a comprehensive moderation framework designed to ensure the input safety for embodied agents, encompassing the entire pipeline from taxonomy definition and dataset construction to moderator design, model training, and evaluation.
• We propose a safety benchmark designed for embodied agents to address the gaps in risk taxonomy and the lack of safety-aware datasets.
• We develop a prompt-decoupled input moderation method tailored for embodied LLMs. This method effectively addresses the complex and dynamic interferences caused by functional prompts in embodied agents.
• We conduct extensive experiments to validate the effectiveness and efficiency of the proposed method.

2 RelatedWork

2.1 Behavioral Safety of Embodied Agents

Research on the behavioral safety of embodied agents primarily evaluates whether their planning, decision-making, and actions pose risks to humans or the environment, and explores methods to enhance security. [Zhang et al., 2024] introduced three novel JailBreak attack methods, demonstrating that these attacks can induce hazardous behaviors in LLMbased embodied agents in both simulated and real-world environments. RiskAwareBench [Zhu et al., 2024] assessed the physical risks in high-level planning by embodied agents, exposing a widespread lack of physical risk awareness in existing LLMs. SafeAgentBench [Yin et al., 2024] established a universal simulation environment to test LLMs' refusal capabilities when given hazardous tasks, evaluating safety awareness from both execution and semantic perspectives.

However, the aforementioned research on the behavioral safety of embodied agents relies solely on general-purpose LLMs' inherent understanding of hazardous behaviors. In practical application scenarios, numerous hazardous behaviors fall outside the comprehension or coverage of these general LLMs, leading to inadequate behavioral safety moderation capabilities in existing studies. In contrast, this work introduces a specifically designed input moderation mechanism tailored to the unique characteristics of embodied agents, significantly enhancing the performance and reliability of behavioral safety moderation.

2.2 Input Moderation for LLMs

Input moderation for LLMs is crucial for ensuring the generated content meets ethical, legal, and security standards. The moderation methods can be categorized into external intervention and internal blocking approaches, depending on whether moderation occurs before or after LLM inference.

External intervention methods review inputs using external models before they are processed by the host LLM. OpenAI's Moderation API can provide online content moderation services for other LLM applications. [Kumar et al., 2024] employed a predefined definition of toxicity to prompt commercial LLMs like GPT-4 to rate input toxicity. Meta's LlamaGuard [Inan et al., 2023] integrated a safety risk taxonomy into model parameters through instruction fine-tuning, transforming the LLM into a dedicated moderator.

Internal blocking methods perform moderation using the model's intrinsic reasoning states after inputs have been processed. [Wu et al., 2024] identified toxic inputs by leveraging the LLM's hidden states as feature vectors, combined with a lightweight multi-layer perceptron (MLP) classifier. [Liu et al., 2024] extracted and augmented toxic concepts, constructing similarity metrics between user inputs and known toxic concepts to assess toxicity.[Xie et al., 2024] achieved toxicity detection by computing gradient differences between the current input and unsafe example inputs.

Nevertheless, these external intervention methods often incur excessive computational and communication overhead, rendering them unsuitable for real-time systems, while existing internal blocking methods are typically limited to a single functional prompt. Unlike these existing methods, we propose Pinpoint, a moderation approach that utilizes a straightforward masked attention mechanism to extract features for classification. It preserves efficiency while decoupling functional prompts from the moderation process, thereby improving both adaptability and performance.

3 EAsafetyBench

The proposed EAsafetyBench aims to bridge the gap in existing security research concerning safety-aware datasets for embodied agents. Specifically, we first develop a typical risk taxonomy using drones as an example to address the unique safety risk scenarios faced by embodied agents. Based on this taxonomy, we construct a comprehensive safety-aware dataset of instructions. Subsequently, we collect function prompts highly relevant to embodied agents from existing embodied agent research and open-source code repositories.

3.1 EAsafetyBench-Taxonomy

A well-defined safety risk taxonomy helps security researchers better understand the nature of risks and develop effective identification and mitigation strategies. It serves as a crucial guide for both methodological advancements and practical applications.

To address the lack of a unified standard for safety risks in embodied agents, we develop a taxonomy based on public perceptions of risk factors in drone applications [Wang et al., 2023; Sabino et al., 2022; Smith et al., 2022; Rubagotti et al., 2022]. While our taxonomy does not claim to exhaustively cover all potential risks associated with embodied agents, we believe it is general enough to be easily adapted to other robotic systems. Our work is influenced by stateof-the-art taxonomies, particularly those from Llama Guard [Inan et al., 2023] and AEGIS [Ghosh et al., 2024]. For clarity, we categorize existing taxonomies as "generative risks" and define our work as "behavioral risks".

We begin by identifying categories of generative risks that align with behavioral risks, such as violent crimes involving physical harm. These categories encompass the most direct and prevalent safety threats. Next, we highlight key differences between the taxonomies. For example, in generative risks, privacy invasion occurs when responses disclose unpublished personal information, while in behavioral risks, it involves actions like using onboard cameras to record private activities. Finally, we expand the taxonomy to address safety concerns unique to embodied agents. While existing taxonomies do not consider the security of the agent itself, this remains an essential, albeit lower-priority, aspect of embodied agent safety-even in the context of Isaac Asimov's Three Laws of Robotics [Asimov, 2004]. The seven risk categories defined in the taxonomy and their corresponding few examples are illustrated in Figure 1.

3.2 EAsafetyBench-Drone

A well-defined safety risk taxonomy enables the systematic creation of comprehensive safety-aware datasets for training and evaluating moderators of embodied agents. Based on the EAsafetyBench-Taxonomy, we leverage data synthesis and augmentation techniques facilitated by LLMs [Wang et al., 2024; Zheng et al., 2023] to develop EAsafetyBench-Drone, a safety-aware instruction dataset for drone agents.

Instruction Generation We leverage GPT-4 [Achiam et al., 2023] as the core tool for efficient and accurate data generation, supplemented by human interaction and supervision.

To guide GPT in generating malicious instructions for specific risk categories or safety instructions for particular applications, we design and refine a set of prompts. For malicious instructions, the input includes detailed risk category definitions, a chain-of-thought framework for step-by-step analysis, and requirements for diversity. For safety instructions, GPT evaluates safety application scenarios for embodied agents based on the defined risk categories and then generates tailored safety instructions. Human supervisors provide feedback to refine prompts and improve generation quality.

Augmentation and Filtering Given the generation preferences of a single model, we utilize LLama3.1-70B-Instruct [Dubey et al., 2024] to semantically rephrase the candidate dataset generated by GPT-4. The rephrasing process employs two approaches: the first enhances expression diversity by converting between active and passive voice, adjusting tone, and replacing synonyms. The second approach, inspired by the deception attack method in BADROBOT [Zhang et al., 2024], subtly replaces concepts to lead embodied agents into performing harmful actions without recognizing their intent. This increases the subtlety of malicious instructions at the semantic level. However, some malicious instructions may lose their harmfulness after rephrasing. To address this, we employ Llama as a judge, utilizing techniques from [Zheng et al., 2023] to assess the consistency of rephrased instructions with the originals, specifically regarding malicious intent and harmful consequences. Instructions that deviate significantly from the originals are filtered based on the consistency score. Dataset Analysis Following the above steps, we construct a safety-aware dataset containing 9,435 instructions, with 4,875 posing safety threats. We present a metric-based analysis of the length distribution and self-BLEU score [Zhu et al., 2018] for EAsafetyBench-Drone. The results show that instruction lengths are mostly between 10 and 20 words, with a balanced distribution across categories, and the self-BLEU score is 0.292, indicating a relatively high text diversity.

3.3 EAsafetyBench-Prompt

We collect 48 prompts from existing research on embodied agents [Zawalski et al., 2024; Chen et al., 2023; Huang et al., 2023; TAZIR et al., 2023], covering various agent forms and different formats. These prompts are key components that endow embodied agents with powerful functionalities. Additionally, we gather 52 system prompts related to tasks such as planning or code generation from the open-source repository PromptCraft 2 , enabling a more accurate evaluation of detection methods across diverse prompt contexts.

4 Pinpoint

Although the benchmark in Section 3 contributes to the model's understanding of the unique risk scenarios faced by embodied agents, limitations in existing methods constrain their application in these contexts. To address this, we propose Pinpoint, an efficient and lightweight input moderation method tailored for embodied agents. By decoupling functional prompts from moderation tasks, Pinpoint ensures reliable performance even in dynamically changing contexts.

2 https://github.com/LouisShark/chatgpt system prompt

4.1 Overview of Pinpoint

Figure 2 illustrates the workflow of Pinpoint, designed to detect malicious instructions in embodied LLMs. Solid arrows represent the standard inference process of the LLM, while dashed arrows indicate the detection process of Pinpoint.

In the context of embodied agents, user instructions are embedded within functional prompts to help the LLM execute them more effectively. Before this process, we insert special markers, <|begin_of_instruction|> and <|end_of_instruction|>, to define the instruction's boundaries. Based on these markers, we create a special mask sequence, where only the instruction is marked as valid (with a value of 1). The tokens corresponding to the special markers are then removed from both the encoded sequence and the mask sequence before the processed sequence is fed into the LLM for inference.

When the inference reaches the middle layer (e.g., the 10th layer), the transformer block in this layer performs an additional attention operation based on the instruction mask sequence, generating classification features. The trained classifier then evaluates the instructions, classifying them as either "malicious" or "safe". If malicious, the generation process is immediately halted and returns a predefined refusal response.

4.2 External Instruction Localization

Before inference, LLM first converts the continuous text sequence into discrete units (tokens) that the model can understand and process. Each token is then mapped to an index corresponding to its entry in a predefined vocabulary.

We represent the input instruction as I, define a tokenization function T that maps I into a sequence of tokens T, and a mapping function ϕ that maps each token to its index in the vocabulary. This process can be formalized as:

where n represents the number of tokens, and Φ denotes the final sequence that is input to the model's embedding layer.

Typically, a mask sequence M is generated alongside Φ, with the same length n, and it marks the non-padding portions of the sequence with a valid value of 1. This mask M guides the attention mechanisms in the model to ignore the padding components, which do not carry any semantic information.

Based on the aforementioned process, we add two special tokens, <|begin_of_instruction|> and <|end_of_instruction|>, to the vocabulary, assigning them indices ts and te, respectively. Before embedding the instruction within the functional prompts, we insert the corresponding special tokens at the start and end of the instruction. This ensures that after converting the input into a token sequence T and a mask sequence M, the start and end positions of the instruction can still be accurately identified:

Utilizing these positions, we create a mask sequence Mi, which marks only the instruction portion enclosed by the two special tokens with 1. This mask guides the model to focus its attention on the instruction content during computation.

Finally, we remove the special tokens from both T and M to restore the original input content. The sequences T, M, and Mi are then jointly input into the model for inference, where T and M correspond to the standard inference process.

4.3 Intrinsic Feature Extraction

LLMs are composed of l layers of stacked transformer blocks [Vaswani, 2017]. During autoregressive generation, the complete input is first transformed into semantic embeddings and then progressively processed through each layer's decoder.

where Hl denotes the l-th hidden layer of the host LLM, and hl represents the hidden state output by the l-th layer.

Each decoder block primarily consists of a multi-head selfattention module and a position-wise feedforward network (FFN). The multi-head attention module employs a masked attention mechanism to handle the input sequence.

We observe that during the inference, multi-layer attention mechanisms may excessively blend the semantics of instructions and prompts. To address this, we extract features from the middle layers Hmiddle of the LLMs. In contrast, deep features lead to model overfitting to specific functional prompts, while shallow features are not suitable for classification as they have yet to sufficiently capture the semantic distinctions required for the task.

Furthermore, based on the previously constructed instruction mask sequence Mi, we create the corresponding mask matrix Mmiddle and utilize it to guide the model in performing a masked attention operation at the middle layer using the hidden state output from the preceding layer, hmiddle-1. This approach further focuses the feature semantics on the instructions themselves, i.e.,

where Q, K, V are all derived from the same input hmiddle-1 by mapping through different projection layers into the corresponding query, key, and value matrix, dk represents the feature dimension of the hidden state, and the softmax function is used to transform the scores into the attention weight.

The final features for classification are obtained by performing a weighted fusion of matrix V with the weight matrix, followed by normalization. Specifically, we extract features only from the final token of the instruction sequence, based on the consensus in existing research [Liu et al., 2024; Wu et al., 2024] that the final token effectively represents the entire sequence through attention mechanisms.

4.4 Malicious Instruction Detection

Leveraging the powerful representational capabilities of LLMs, Pinpoint only requires a lightweight classifier to perform input moderation. Specifically, we design a three-layer MLP model with approximately 4 million parameters. This classifier frames the task as a binary classification problem and is trained using cross-entropy as the loss function.

Once the classifier is trained, it moderates the input during LLM inference. At a designated layer in the model, we extract features using the aforementioned method and perform rapid moderation. If the classifier detects malicious instructions, the generation process is immediately halted, and a predefined refusal response is returned. To improve detection accuracy, we introduce a masked attention mechanism, which helps the classifier focus on the instruction-specific features while avoiding interference from the functional prompts used by the embodied agents. This method maintains efficiency while significantly enhancing accuracy. Experiments show that the overhead introduced by instruction localization and attention computation is minimal, yet the resulting improvements in detection performance are substantial.

5 Experiments

5.1 Setup

Datasets In addition to EAsafetyBench-Drone, our experiments also utilize data from SafeAgentBench [Yin et al., 2024]. To the best of our knowledge, these are the only two safety-aware datasets specifically dedicated to embodied agents. SafeAgentBench focuses on potential hazards to humans or property in household scenarios. It comprises 750 instructions, including 450 tasks with safety risks.

We partition the combined dataset from EAsafetyBenchDrone and SafeAgentBench into a training set and test set based on semantic similarity to ensure distinction for each set. For this, we employ NV-Embed-v2 [Lee et al., 2024] as the embedding model. The training set is allocated 70% of the data. Similarly, we split EAsafetyBench-Prompt into visible and in-the-wild prompts. The training set is embedded with visible prompts for training, while the test set uses both visible prompts and in-the-wild prompts for evaluation. This allows us to assess the model's fit to visible prompts and its generalization to real-world scenarios.

Baselines To evaluate the effectiveness of Pinpoint, we employ three state-of-the-art internal methods as baselines.

• Gradsafe [Xie et al., 2024] conducts moderation by calculating the gradient differences between the current input and unsafe example inputs.
• Legilimens [Wu et al., 2024] leverages the hidden states of the first token in the generated sequence from the model's last few layers as features for classification.
• ToxicDetector [Liu et al., 2024] performs element-wise product of the embedding for toxic concept and user input at each layer of LLMs. The results from all layers are concatenated to form the input features for the classifier.

Host Model We evaluate the Pinpoint on 10 open-source LLMs. These LLMs encompass various architectures and cover a range of hidden layers from 16 to 40.

• Falcon [Penedo et al., 2023] follows a causal decoder architecture. We select the 1B version with 18 layers.
• Llama [Dubey et al., 2024] follows a causal decoder architecture. The Llama-3.2-1B with 16 layers, Llama3.2-3B with 28 layers, Llama-2-7B with 32 layers, and Llama-3.1-8B with 32 layers are selected.
• ChatGLM [GLM et al., 2024] follows a prefix decoder architecture. ChatGLM3-6B with 28 layers and ChatGLM4-9B with 40 layers are tested to evaluate performance across different architectures.
• Qwen [Yang et al., 2024] is currently the most powerful open-source model under 10 billion parameters, following a causal decoder architecture with 28 layers.
• Mistral [Jiang et al., 2023] follows a causal decoder architecture. We select the Mistral-7B and the latest version, Ministral-8B. They both have 36 layers.

Metrics The metrics we adopted include Accuracy, F1 Score, False Negative Rate (FNR), and False Positive Rate (FPR). Accuracy and F1 Score indicate general classification performance. FNR measures the proportion of malicious instructions misclassified as safe, which could pose safety risks. FPR measures the proportion of safe instructions misclassified as malicious, which could impact user experience.

Experimental Settings All experiments are conducted on Ubuntu 22.04 using four NVIDIA RTX A6000 GPUs. We build the entire experimental environment on the PyTorch platform and configure all baselines and host LLMs strictly following their respective specifications. We train a fully connected MLP classifier with 3 layers and 4 million parameters using the Adam optimizer. The training parameters are set as follows: a batch size of 16, 50 epochs, a learning rate of 1e-3, and a weight decay (ℓ2 penalty) of 2e-4.

5.2 Performance Evaluation

Effectiveness In this subsection, we evaluate the effectiveness of Pinpoint. We select the layer m exhibiting the highest overall performance for the extraction of features to perform the experimental presentation. Specifically, for LLMs with 28 or fewer layers, m=10, and for LLMs with more than 28 layers, m=17. To comprehensively validate Pinpoint's performance, we compare its results on multiple host LLMs against baselines. Experimental results are presented in Table 1.

For visible prompts, Pinpoint demonstrates effective fitting across mainstream models with varying layer structures, achieving an average F1 score of 0.9493. Additionally, it maintains a low FPR and FNR, balancing both system usability and security. In contrast, baseline methods exhibit inferior performance. Specifically, Legilimens, which relies on the hidden states of the last few layers, suffers from excessive semantic fusion of functional prompts due to repetitive attention computations. This fusion diminishes its ability to discern classification differences in contaminated features. Legilimens achieves an F1 score of only 0.8359. ToxicDetector, which performs detection by aggregating embeddings from all layers, introduces excessive noise during processing. Although it uses a contrastive classification strategy to improve performance, its F1 score remained at only 0.7461. For GradSafe, the functional prompts introduce significant interference in the computation of its gradient differences, resulting in an F1 score of only 0.4916. Overall, all baseline methods exhibit poor performance, especially in terms of FNR, which could significantly compromise the safety of embodied agents. This is primarily because functional prompts are generally safe, and excessive semantic fusion obscures the inherent malicious intent of the instructions.

For in-the-wild prompts, our method maintains strong detection performance, achieving an average F1 score of 0.9478, with a negligible difference of only 0.0015 compared to visible prompts. This result highlights the exceptional generalization capability of our method and further demonstrates its effectiveness in mitigating semantic interference from functional prompts. In contrast, the performance of Legilimens and ToxicDetector shows significant degradation, with F1 scores decreasing by 0.0795 and 0.0295, respectively. As for GradSafe, although the scores improve slightly, its overall performance remains the lowest.

Notably, Pinpoint exhibits relatively low metrics on the ChatGLM which is based on a prefix decoder architecture. This primarily stems from the bidirectional attention mechanism in prefix decoders, which inherently incorporates more semantic information from functional prompts. We preliminarily investigate multilayer feature fusion as a viable solution. Specifically, we extract and fuse features frommultiple hidden layers (e.g., concatenating features from layers 9, 10, and 11 for LLMs with 28 or fewer layers, concatenating features from layers 16, 17, and 18 for LLMs with more than 28 layers). This approach enables Pinpoint to achieve an F1 score of 0.9498 on prefix-decoder architecture models, which is comparable to that of the causal architecture model.

Efficiency In this subsection, we measure the average time overhead for different methods in completing detection tasks on the test set which is embedded within in-the-wild prompts. To simulate real-world applications, we set the batch size to 1 for all tests. The measurement results are shown in Table 4. Compared to the baselines, Pinpoint outperforms all other methods in terms of efficiency, requiring only 0.002 seconds per moderation. Furthermore, when deployed on different host LLMs, Pinpoint maintains the same level of efficiency, meeting the real-time requirements of the embodied agents.

5.3 Ablation Study

We conduct ablation experiments on both the layer selection strategy and the masked attention mechanism to objectively evaluate their effectiveness. Figure 3 presents the comprehensive results of the ablation experiments.

Layer Selection Strategy Figure 3 illustrates the performance of LLMs with varying numbers of layers in feature extraction for detection at each layer. The trends in each subplot are consistent, with performance gradually declining or stabilizing starting from the middle layers. This aligns with our previous analysis: deeper layers integrate more semantics of functional prompts through attention mechanisms, which may introduce additional noise and impact detection performance. This effect is particularly pronounced in ChatGLM, which is based on a prefix decoder architecture.

InstructionMasked Attention In Figure 3, the orange and gray curves represent classification performance on visible and in-the-wild prompts, respectively, using a method without masked attention (directly using the final layer's last token hidden state as classifier input). Across all host LLMs, incorporating masked attention significantly improves performance, particularly in generalizing to in-the-wild prompts. By leveraging this, Pinpoint effectively handles the complex and diverse prompts encountered in real-world applications.

6 Conclusion

In this work, we propose a comprehensive input moderation framework to address the critical challenge of ensuring the behavioral safety of embodied agents. The framework comprises two key contributions: (1) EAsafetyBench, a tailored safety benchmark designed to capture the unique safety risks inherent to embodied agents, providing a robust safety-aware instruction dataset for training and evaluating moderators; and (2) Pinpoint, an innovative input moderation method that leverages a masked attention mechanism to efficiently extract features decoupled from functional prompts, thereby enhancing both safety and computational efficiency. Extensive experiments conducted across various LLMs demonstrate that Pinpoint consistently outperforms existing baseline methods in terms of safety performance and operational efficiency.

Acknowledgments

This work was supported in part by the National Key R&D Program of China under Grant 2022YFB3103500, in part by the National Natural Science Foundation of China under Grant 62101079, in part by the Venture and Innovation Support Program for Chongqing Overseas Returnees under Grant cx2021012.