A Survey on Value Alignment in Agentic AI Systems

1 Introduction

Agentic AI systems, leveraging capabilities such as autonomous decision-making, tool usage, and dynamic task decomposition, are emerging as a frontier in artificial intelligence research. With the rapid advancement of large language models, they have become the core reasoning engines powering these agentic AI systems, enabling agents to handle increasingly open and diverse real-world tasks.

∗Hengshu Zhu and Yihang Cheng are the corresponding authors.

However, widespread agentic AI deployment introduces systemic risks alongside efficiency gains. In societal domains like finance and healthcare, agents may prioritize efficiency over fairness or misinterpret ethical guidelines, thereby exacerbating discrimination and privacy breaches [Tennant et al., 2025]. Furthermore, complex inter-agent interactions and the often ambiguous task boundaries frequently cause difficulties in accountability and conflicts among stakeholder objectives, severely undermining system trustworthiness without robust institutional design [Carichon et al., 2025]. Addressing these challenges necessitates moving beyond the technical alignment of single models toward systematic value alignment [Gabriel, 2020]-shifting from mere algorithmic optimization to a governance issue balancing diverse interests and societal values. Although extensive research explores LLM value alignment methods [Zhou et al., 2025], few works systematically examine the complete technical stack and hierarchical relationships for agent alignment. Existing research remains scattered across disparate pathways, lacking a unified analytical framework that spans foundational models, single agents, and multi-agent interactions, alongside an integrated multi-level evaluation system.

To this end, this paper proposes a systematic three-tiered classification framework for organizing value alignment research based on the locus of optimization: namely, alignment at the level of the foundational LLM, at the level of a single instantiated agent, and at the level of multi-agent interaction. The main contributions of this research are as follows: (1) A tentative multi-level taxonomy that groups value principles into L0, L1, and L2 tiers, suggesting possible mappings to LLMs, single agents, and multi-agent systems respectively.

(2) Systematically reviewing core alignment methods along the technical stack: at the LLM level, analyzing value shaping mechanisms separately during pretraining and post-training stages; at the single-agent level, focusing on value representation and injection, long-term alignment via profiles and memory, and safety constraints at the planning and action levels; at the multi-agent level, summarizing collaborative alignment paradigms such as communication strategy optimization and multi-objective reinforcement learning. (3) Systematically reviewing existing datasets and methods for multi-level value alignment evaluation. (4) Outlining key future research directions, including inter-agent value coordination mechanisms, high-quality scenario data sharing, and alignment at the communication protocol level, and advocating for the establishment of a more systematic and dynamic evaluation framework to promote robust and trustworthy value consensus in agentic AI systems within societal collaboration. The rest of this paper is structured as follows: Section 2 defines value and alignment; Sections 3-5 examine alignment mechanisms across three technical levels; Section 6 reviews evaluation methods and datasets; and Section 7 outlines future research directions.

2 Preliminaries

2.1 Value and Value Alignment

Values serve as stable cognitive standards for human behavior, shaping individual attitudes, social interactions, and the establishment of collective goals [Catton Jr, 1959]. Characterized by inherent contradictions and dynamism, values are not rigid hierarchical structures but rather flexible systems contingent upon context [Sagiv et al., 2017]. This paper frames values as dynamic, multi-level systems not exclusive to human bearers.

The fundamental core of value alignment lies in ensuring that AI behaviors conform to human values [Russell and Norvig, 2016], spanning interdependent normative (which values to uphold) and technical (how to align via datasets and reinforcement learning) dimensions [Gabriel, 2020]. Although these two dimensions are interdependent, a unified framework for values remains absent. Consequently, this study reviews and explores the values that agentic AI systems might follow across different levels of operation.

2.2 Principles of Value Alignment

Value alignment aims to delineate the value principles to which the behaviors of agents must adhere. This study categorizes and reviews values across three distinct levels: Level 0 (L0), Level 1 (L1), and Level 2 (L2), see Figure 1. L0 encompasses the general ethical baselines and fundamental value consensus with which LLM-based models must align. L1 represents values influenced by cultural nuances and industry-specific characteristics, while L2 consists of value principles further refined according to specific scenarios.

Level 0 - Universal Values. A comprehensive review of existing literature identifies three representative core theoretical frameworks for general AI values. The first is the HHH principles (Helpful, Honest, Harmless), which serve as one of the most widely adopted standards in current alignment tasks for LLMs [Yao et al., 2025]. The second is Moral Foundations Theory (MFT) [Graham et al., 2013]. This theory posits that human morality is constructed upon several innate foundational values and has been widely applied to analyze the latent moral tendencies of LLMs [Abdulhai et al., 2024]. Finally, the Schwartz Theory of Basic Human Values [Schwartz, 2007], established and validated through extensive empirical research, is widely incorporated into various datasets.

Level 1 - Cultural and Industry Values. This study defines L1 as the "Cultural and Industry Value" tier. Values at this level are not universally uniform but are embedded within specific cultural and industrial contexts. Notably, cross-national cultural differences often manifest as directional preferences along specific value dimensions; that is, distinct cultural groups may exhibit diametrically opposed tendencies within the same dimension. Existing research regarding the quantification of such directional disparities is primarily grounded in Hofstede's Cultural Dimensions Theory [Hofstede, 2011]. Studies have sought to verify whether model performance along these dimensions aligns with human values within specific cultural contexts [Kharchenko et al., 2025]. Furthermore, the GLOBE framework from social psychology [House et al., 2004] has been introduced as a crucial supplementary tool, offering a more granular evaluative perspective. Building on this, researchers proposed the LLM-GLOBE benchmark [Karinshak et al., 2024]. By comparing Chinese and American large models, they revealed significant directional differences across multiple cultural dimensions, thereby quantifying the cultural value systems implicit in model outputs.

As AI penetrates vertical domains, different industries exhibit distinct priorities and requirements regarding values. In the legal domain, mere compliance with general morality is insufficient. The concept of "Legal Alignment" was introduced to emphasize the necessity for AI to comprehend and adhere to legal reasoning, authority, and procedural justice, thereby ensuring safety and compliance in judicial assistance [Kolt et al., 2026]. In the field of education, scholars noted that when integrating LLMs into higher education, it is essential to prioritize the potential impact on students' mental health while optimizing their academic engagement [Zhang, 2025]. In the healthcare domain, ethical requirements are directly correlated with life safety and privacy. Through a systematic review, it highlighted that LLMs in medical applications face unique challenges, such as accountability in clinical decisionmaking, and must strictly adhere to medical ethical standards (e.g., non-maleficence and patient autonomy) rather than mere helpfulness [Fareed et al., 2025].

Level 2 - Context-specific Values. L2 represents the most granular, context-specific tier, refining value criteria through two dimensions: execution environments and task goals. Environment variability implies that identical models deployed in distinct operational settings, must identify context-specific causal logics and adhere to differing value standards. In scientific experiments, agents must follow standard operating procedures (SOPs), such as safety protocols, to strictly control high-risk tool usage (e.g., chemical synthesis or pathogen handling). This is essential to prevent their autonomous capabilities from being maliciously exploited or inducing unforeseen biochemical hazards [Peng et al., 2025]. When constructing enterprise-internal chatbots, models must not only comply with general laws and regulations but also strictly align with the company's specific internal business conduct guidelines [Achintalwar et al., 2024]. Task objectives dictate the value priorities of the model when executing specific instructions. In biosafety laboratories, agents are required to prioritize the prevention of biological hazards and ethical risks over experimental efficiency [Tang et al., 2024]. Specifically within the context of enterprise applications, corporate strategic planning and core values must serve as essential critical anchors for such trade-offs [Broestl et al., 2025].

2.3 Taxonomy

Figure 2 uses a multidisciplinary medical consultation as an illustrative case to characterize multi-level value alignment in agentic AI systems. At the foundational LLM level, alignment targets L0 universal values (e.g., helpful, honest, harmless) together with L1 general medical ethical principles (e.g., nonmaleficence and medical objectivity), providing a common normative baseline. At the single-agent level, specialized agents operationalize L1 role-independent clinical values (e.g., patient autonomy and confidentiality) and L2 role-specific constraints (e.g., diagnostic scope compliance and interactional empathy). At the multi-agent level, alignment primarily concerns L2 coordination values, such as inter-departmental consensus formation, cross-verification of diagnostic recommendations, and hierarchical compliance. This example illustrates that value alignment in agentic AI systems is inherently hierarchical and must be addressed consistently across L0-L2, from foundation models to individual agents and collective coordination. Accordingly, here we propose a taxonomy for value alignment methods in agentic AI system, organized by different optimization focus (see Figure 3). Specifically, alignment can be pursued (i) at the level of the underlying LLM, (ii) at the level of a single instantiated agent, or (iii) at the level of multi-agent interaction. In the following sections, we examine these three categories in detail, highlighting the distinct mechanisms through which each approach achieves value alignment in agentic Al.

3 LLMOptimization

LLMs are core components of agentic AI systems. As foundational engines, current LLMmethodologies are specifically tailored for handling critical tasks such as adaptive query generation, recruitment cognitive diagnosis, and open-set recognition [Chen et al., 2025; Ma et al., 2025; Tong et al., 2025; Zhu et al., 2025a];meanwhile, the broad applications of LLM have revolutionized fields like organizational management, scientific research, healthcare, traffic forecasting [Qin et al., 2025a; Jiang et al., 2024; Qin et al., 2025b; Qin et al., 2025c; Shen et al., 2026; Liu et al., 2026a; Liu et al., 2026b; Long et al., 2024; Long et al., 2026a; Zhu et al., 2025b; Long et al., 2026b; Wang et al., 2026]. While alignment primarily focuses on L0 universal values to establish a shared ethical baseline for all agents, practical deployment in specific domains or open, dynamic environments necessitates extending alignment objectives to L1 and L2 values. Methodologically, approaches can be grouped into pretraining- and post-training-based value alignment.

3.1 Value Alignment in LLMPretraining

Pretraining Data Optimization. Because multiple encapsulated agents often inherit the same foundation model, corpus optimization is a high-leverage mechanism for injecting shared value priors, especially for L0 and L1 values. (1) Curation and filtering (harm reduction, quality control, documentation): open-data pipelines emphasize principled deduplication, quality filtering, and transparent mixture ablations to clarify how curation choices shape downstream behavior [Li et al., 2024a]. Safety resources further stress that detoxification must be documented and monitored since it can shift both exposure to and measurement of toxicity [Luong et al., 2024], anchoring L0 safety/fairness baselines. (2) Rights- and compliance-aware sourcing and normative-source mixing: using permissibly licensed/public-domain sources with clearer provenance and mixing in legal/regulatory corpora better encodes jurisdictional and sectoral constraints [Niklaus et al., 2024], operationalizing L1 compliance while remaining grounded in L0 rights protection. In summary, data optimization forms an engineering control stack: curation stabilizes core L0 values, compliant sourcing and normative mixing implement domainspecific L1 values, and privacy/integrity safeguards become pivotal for applied L2 values.

Prompt Preconditioning. Prompt preconditioning injects value priors by augmenting pretraining inputs with repeated prefixes (directives, tags, or control codes) that become part of the learned context distribution; this suits agentic AI systems because the conditioning interface can be shared across agents and reused at inference. (1) Instruction- and tag-augmented pretraining: safety-relevant instructions or categorical tags encourage aligned framing; instruction injection and toxicityaware meta-conditioning can reduce toxic generations while preserving utility [Prabhumoye et al., 2023]. Harmfulness tags in unsafe passages provide a structured channel for trainingtime and downstream controllability [Maini et al., 2025]. (2) Metadata/control-code conditioning: metadata prefixes offer scalable handles over style/domain attributes and improve steerability [Gao et al., 2025]; conditioning on preferencederived scores integrates the conditioning variable during learning rather than only post hoc [Korbak et al., 2023]. In summary, prompt preconditioning is a lightweight alignment interface that can encode L0 safety values, express L1 values via jurisdiction/sector codes, and support micro policies via organization-specific prefixes, provided the channel is consistently exercised and audited.

Objective-Level Value Shaping. Objective-level value shaping modifies the pretraining objective so alignment signals are internalized via gradients, complementing data selection and prompt formatting while discouraging reproduction of undesirable-but-informative content. (1) Preferenceaware objectives: coupling next-token learning with humanfeedback-derived scores (e.g., score-conditioning, filtering objectives, unlikelihood penalties) often improves alignmentcapability trade-offs relative to applying feedback only after standard MLE pretraining [Korbak et al., 2023]. (2) Selective/constrained losses: masking or altering loss on high-risk tokens while keeping them in context improves safe handling without increasing generation propensity [Wang et al., 2025a]. (3) Attribution-guided interventions: influence/attribution methods identify toxicity-driving training signals and suppress their impact through integrated objectives, reducing reliance on explicit preference datasets [Coalson et al., 2025]. In summary, objective shaping serves as an inner-loop control mechanism that not only supports L0 values (such as harm reduction and privacy preservation) but also enables fine-grained, auditable L2 values (such as multi-stakeholder trade-offs) through explicit loss design.

Curriculum-Based Pretraining Strategy. Curriculum-based strategies control data order, pacing, and mixing so models acquire capabilities and constraints in stages, shaping when (and how strongly) alignment signals enter representation learning. (1) Difficulty-ordered curricula and pacing: studies comparing easy-to-hard ordering, pacing schedules, and interleaving show improved convergence and durable gains when used as structured warmup [Zhang et al., 2026b]. (2) Learnabilityand influence-driven curricula: learnability proxies (e.g., lossgap trajectories) prioritize informative examples early [Fan and Jaggi, 2023], while influence-based ordering provides another scalable curriculum signal beyond domain heuristics [Schoenegger et al., 2025]. In summary, curriculum design is an outer-loop knob complementing data- and objective-level alignment, governing the timing and persistence of reinforcement. Targeting stable, long-horizon constraints in agentic AI systems, it prioritizes L0/L1 value embedding.

3.2 Value Alignment in LLMPost-Training

Value-aware Supervised Fine-tuning. Value-aware supervised fine-tuning (SFT) is a loss-level post-training approach that injects L0/L1 value principles (e.g., safety, rights protection, non-discrimination) into a foundation model via curated demonstrations and value-sensitive objective shaping, yielding a reusable normative prior across encapsulated agents. Given demonstrations Ddemo = {(x, y)}, SFT minimizes LSFT(θ) = -E(x,y)∼Ddemo logπθ(y|x). Value-aware variants further apply loss shaping (e.g., reweighting, contrastive supervision, auxiliary value-attribute regularizers) to bias behavior toward value-consistent regions. Rule-/constitutionguided supervision can also generate or revise targets under explicit principles (e.g., Constitutional AI self-critique/revision), improving auditability and top-down controllability [Bai et al., 2022]. However, SFT depends on the integrity of instruction and demonstration pipelines and is susceptible to trainingtime manipulation: instruction-tuning backdoors show that a small set of malicious instructions can implant triggers that redirect behavior at inference [Xu et al., 2024], and gradientguided trigger learning demonstrates stealthy poisoning that preserves benign performance while enabling backdoored behaviors [Qiang et al., 2024].

Reinforcement Learning from Feedback. Reinforcement learning from feedback is a policy-level post-training alignment paradigm that optimizes the LLM to internalize preference trade-offs among competing value dimensions (e.g., helpfulness vs. harmlessness) beyond demonstration imitation. In practice, this pipeline can be extended toward L0/L1 principle control through approaches like (i) principle-based feedback (RLAIF/Constitutional AI) that replaces human preference labels with AI preferences conditioned on rules [Bai et al., 2022], (ii) preference optimization without RL sampling such as DPO that directly optimizes an equivalent KL-regularized preference objective for stability and simplicity [Rafailov et al., 2023], and (iii) feedback decomposition into interpretable rule dimensions (as in Sparrow) to improve governance-grade auditability [Glaese et al., 2022].

4 Single-Agent Optimization

At the single-agent level, methods for representing and injecting values largely address L1 cultural and industrial values, whereas alignment within profile, memory, planning, and action is crucial for adhering to L2 context-specific values.

4.1 Representing and Injecting Values to Agents

A core challenge in single-agent value alignment is the mismatch between human intent (often expressed as high-level values and policies) and an LLM agent's concrete behavior (plans, tool calls, and environment-interactive actions). Given practical alignment cost considerations, this stage may focus on L1 and L2 values like role ethics, industry codes, and environmental rules. Effectively bridging this gap requires operational value representations and injection mechanisms that remain effective under distribution shift. Existing work can be broadly grouped by where values are encoded and how they enter the agent loop.

Prompt-Conditioned Norm Specification. Values can be written as natural-language specifications and injected via policy prompts or constitution-style constraints [Zhou et al., 2025]. While expressive, this route can be fragile: changing an agent's persona through system prompts may substantially and unpredictably shift misalignment tendencies (e.g., avoiding oversight or resisting shutdown), suggesting limited robustness from prompts alone [Naik et al., 2025].

Preference-Based Alignment Post-Training. Values can be learned implicitly from human preferences and injected through preference-based post-training (e.g., RLHF-style reward modeling or DPO) [Jiang et al., 2025]. To improve reliability, agentic reward modeling augments preference rewards with verifiable correctness signals (e.g., factuality and instruction following), turning partial norms into executable constraints through verification-oriented reward components [Peng et al., 2025].

Explicit Normative Reward Shaping. Normative objectives can be formally encoded as reward functions. Specifically, moral alignment work designs intrinsic rewards grounded in ethical frameworks and applies RL fine-tuning in controlled environments, thereby enabling more transparent optimization [Tennant et al., 2025].

4.2 Value Alignment in Profile andMemory

Agents must maintain behavioral consistency across sessions and contexts. Profiles and memory provide the core infrastructure to meet these demands. In this process, profiles can offer more refined definitions of identity based on L2, while the memory component allows for greater consideration of persistent contextual adaptation at the L2.

Profile-based Alignment. Profile defines identity or roles, stable preferences, long-term objectives, and hard constraints (privacy, safety, compliance), supplying explicit value boundaries for planning and tool use rather than leaving values implicit in prompts [Li et al., 2024b]. Recent work also argues for structured profiles (traits, emotions, goals) with behavioral tests, and shows that alignment should be judged by observable consistency over multi-turn discourse, not profile text alone [Cerqueira and Bezerra, 2025].

Memory-based Alignment. Memory complements profile by storing interaction history, preference evidence, decisions, and feedback, enabling retrieval that grounds abstract principles in the current situation and reduces value drift. MemOS promotes OS-style abstractions and lifecycle management for heterogeneous memories to support writing, retrieval, updates, and auditing [Li et al., 2025]. Cognitively inspired memory structures can further improve interpretability and long-horizon coherence [Vishwakarma et al., 2025], while "digital me" work highlights governance needs when persistent personal memory is coupled with identity [Coll et al., 2025].

4.3 Value Alignment in Planning and Action

Planning-action safety alignment can be understood along two complementary dimensions. The planning dimension constrains what the agent proposes to execute (e.g., multi-step plan structure and feasibility), corresponding to L2 values such as safety protocols and procedural compliance. The action dimension constrains how the agent performs under uncertainty and long-horizon error accumulation (e.g., tool invocation, procedure adherence, and runtime safeguards), corresponding to L2 values such as risk aversion and procedural compliance.

Planning-level Alignment. At the planning layer, the goal is to rule out unsafe or infeasible plans before execution by integrating hazard knowledge, constraint checks, and uncertainty signals into plan scoring and search. SafePlan-Bench benchmarks safety-critical embodied planning and couples it with an alignment approach that injects physical hazard knowledge, effectively translating "be safe" into constraints over plan steps and action sequences [Huang et al., 2025]. Complementarily, HEAL demonstrates that instruction-scene inconsistency can trigger actionable hallucinations in embodied agents, motivating planning-time inconsistency detection, uncertainty-aware replanning, and conservative fallbacks when the world model is unreliable [Chakraborty et al., 2025]. More generally, planning improvements such as atomic-fact augmentation with limited lookahead search provide a natural interface to incorporate safety facts and constraint checks during plan generation and evaluation [Holt et al., 2025].

Action-level Alignment Even well-formed plans fail due to tool misuse, unexpected side effects, or deviation from prescribed procedures. Tool-learning surveys emphasize that safety requires governance over when tools are invoked and how arguments and side effects are controlled, not merely selecting the correct tool [Xu et al., 2025b]. In high-stakes workflows, SOP-Bench operationalizes action-level constraints via SOP-constrained industrial tasks and reports substantial failure rates and wrong-tool calls, highlighting the fragility of current agent execution policies and the need for procedure-aware execution control [Nandi et al., 2025]. Practically, this layer is implemented via controlled invocation policies, argument validation, side-effect-aware execution, and runtime monitoring that can trigger interruption, clarification, or rollback when violations are detected [Xu et al., 2025b].

Hybrid Alignment. SafePlan-Bench, HEAL, and SOPBench assess hazard-aware plan validity, robustness to scenetask inconsistency, and tool or procedure-constrained execution, respectively [Huang et al., 2025; Chakraborty et al., 2025; Nandi et al., 2025]. Together, effective planning-action alignment combines (i) constraint-based plan filtering [Huang et al., 2025], (ii) inconsistency and uncertainty detection to trigger replanning or safe fallback [Chakraborty et al., 2025], and (iii) constraint-governed execution for tools and procedures at runtime [Xu et al., 2025b; Nandi et al., 2025].

5 Multi-Agent Optimization

LLM-based multi-agent systems (MAS) are designed to solve complex tasks by specifying how multiple agents interact through coordination strategies and communication protocols. To ensure that agent behaviors remain consistent with human values and system-level objectives, with a focus on L2 value specifications, existing value alignment methods for LLMbased MAS can be broadly classified into two categories: communication strategy optimization and multi-objective reinforcement learning.

5.1 Communication Strategy Optimization

Debate and Adversarial Learning. Recent work has increasingly explored debate-based methods as a promising approach for enhancing value alignment in LLM-based MAS. In these frameworks, agents adopt opposing or complementary roles to engage in structured argumentation, enabling cross-verification and error correction. For example, [Yang et al., 2025] combine adversarial debate with voting mechanisms to improve decision reliability in high-stakes domains such as medicine and law. Furthermore, [Choi et al., 2025] employ identity-agnostic response anonymization to reduce sycophancy and self-bias in multi-agent debates. Beyond these approaches, frameworks such as GVIC optimize debate outcomes by explicitly balancing usefulness and harmlessness-two core dimensions of value alignment-via structured vigilance and communication protocols [Zou et al., 2024].

Hierarchical Communication Workflows. While prior approaches primarily leverage agent disagreement and crossverification to improve alignment, an alternative line of work emphasizes hierarchical orchestration and structured supervision among LLM-based agents to embed value alignment directly into collaborative workflows. For example, the Enforcement Agent framework introduces dedicated supervisory agents that monitor and correct misaligned behaviors in real-time, helping to maintain safety and coherence in joint outcomes [Tamang and Bora, 2025]. TalkHier combines a structured communication protocol with hierarchical refinement stages, enabling agents to iteratively clarify, validate, and refine shared outputs through stratified interactions [Wang et al., 2025b]. More broadly, recent architectural taxonomies and social-structural analyses suggest that organizational hierarchies not only enhance system performance but also shape how agents align with human preferences at both individual and collective levels [Carichon et al., 2025].

5.2 Multi-Objective Reinforcement Learning

In LLM-based MAS, individual agents' performance objectives often conflict with system-level values such as fairness and equity. Multi-objective reinforcement learning (MORL) addresses this tension by enabling agents to balance task efficiency with normative criteria rather than optimizing a single scalar reward. For example, recent work on moral decisionmaking in multi-agent social dilemmas defines intrinsic ethical rewards grounded in moral theories (e.g., consequence- and norm-based ethics) and shows that agents trained in repeated Prisoner's Dilemma-style games can balance self-interest with collective welfare by jointly optimizing performance and moral objectives [Tennant et al., 2023]. Fairness-aware policy optimization methods further operationalize this idea. Approaches such as Fairness-Aware Proximal Policy Optimization jointly optimize welfare-based fairness objectives and task performance, yielding decentralized policies that improve equity among agents while preserving coordination effectiveness [Zhang et al., 2026a]. Beyond agent-centric MORL, environment-centered approaches achieve value alignment by shaping the learning environment itself. In particular, work on designing ethical environments uses an approximate ethical embedding process to transform multi-objective ethical Markov games into environments that inherently incentivize value-aligned policies, allowing agents to pursue individual goals while reliably satisfying ethical constraints [MayoralMacau et al., 2025].

6 Value Alignment Evaluation

6.1 Methods for Value Alignment Evaluation

Current value alignment evaluation is predominantly grounded in explicit value frameworks and relies on datasets composed of subjective and objective questions to assess system alignment across L0, L1, and L2 value dimensions, ranging from abstract normative principles to context-sensitive social norms and fine-grained behavioral judgments. To reflect the decisionmaking characteristics of agentic AI systems operating in complex tasks and real-world environments, existing evaluations have begun to extend beyond purely static response assessment, incorporating limited forms of scenario-based evaluation, value-aware judgment, and constraint-sensitive task assessment. Concretely, most benchmarks remain organized around question formats and metric design, including multiple-choice questions (e.g., BBQ [Parrish et al., 2022]), binary judgments (e.g., ETHICS [Hendrycks et al., 2021]), ranking tasks (e.g., StereoSet [Nadeem et al., 2021]), rating scale questions (e.g., Moral Integrity Corpus [Ziems et al., 2022]), and open-ended questions (e.g., BeaverTails [Ji et al., 2023]), which primarily evaluate isolated, short-horizon value judgments at the response level. Despite these advances, current evaluation paradigms lack agent-centric frameworks that systematically assess long-horizon decision-making, multistep interaction, and value consistency under dynamic constraints, limiting their ability to fully capture value alignment risks in real-world agentic AI systems.

6.2 Datasets for Value Alignment Evaluation

Due to space constraints, we summarize all surveyed opensource datasets for value alignment and evaluation in a public GitHub repository 1 . These datasets are organized according to the proposed multi-level framework to clarify their coverage at different levels of value alignment. Despite this breadth, important limitations remain. At the L0, widely used datasets such as ETHICS [Hendrycks et al., 2021] are largely centered on western cultural norms and provide limited support for modeling diverse or evolving value systems. At the L1, datasets including CultureSPA [Xu et al., 2025a] capture national or cultural perspectives but offer limited global and domain-specific coverage. L2 datasets are especially scarce, and many remain closed-source, such as VITAL [Shetty et al., 2025], constraining fine-grained value alignment evaluation in multi-agent settings.

The datasets are primarily constructed using three paradigms: manual, automatic, and hybrid approaches. Manual construction, including expert-driven design, crowdsourcing, and dataset integration, yields high-quality and interpretable benchmarks but is limited by cost, scalability, and subjective bias (e.g., BBQ [Parrish et al., 2022]). Automatic construction leverages LLMs to efficiently generate large-scale and context-rich data (e.g., CultureSPA [Xu et al., 2025a]), while raising concerns about hallucination and inherited model biases. Hybrid approaches combine expert-curated seeds, LLM-based expansion, and human verification to balance scalability and data quality, and are increasingly adopted in value alignment research (e.g., CBBQ [Huang and Xiong, 2024]).

7 Future Directions

With the deepening integration of artificial intelligence and social infrastructure, agentic AI systems are experiencing explosive growth. However, the rapid expansion in the scale of these agents and their complex interactive behaviors have

1 https://github.com/Wei-ZENG1020/

Value-Alignment-Agentic-AI-Papers-Survey-Taxonomy

triggered multidimensional value conflicts and potential systemic risks that have yet to receive sufficient attention. These challenges involve not only the tension between technical implementation and ethical norms but also extend to behavioral alignment across cultures, industries, and specific contexts. Against this backdrop, value alignment has evolved from a purely technical optimization issue into a core concern for the reliable operation of agent systems and the modernization of social governance. To address this challenge, coordinated efforts are needed across the following four dimensions.

7.1 Multi-Level Value Alignment Evaluation

This system is designed to resolve "goal conflicts" from a social governance perspective by synchronizing L0, L1, and L2 values, thereby balancing individual optimization with collective welfare. Its construction requires the participation of diverse stakeholders: the government establishes benchmarks for core values such as public safety and privacy; industry organizations develop sector-specific standards; and enterprises define internal values for specific scenarios. Future research should focus on a governance ecology that integrates these tiers, utilizing cross-cultural mapping for algorithmic compatibility and value evolution models based on social big data to achieve proactive alignment.

7.2 High-Quality Scenario Data Sharing

From a multi-level perspective, scenario datasets at the L2 are the most urgently needed compared to those at the L0 and L1, as they are particularly critical for defining evolving scenario labels, open-world benchmarks, and value judgments of agents in specialized domains [Chen et al., 2026; Xu et al., 2026; Qin et al., 2026].From a multi-level perspective, scenario datasets at the L2 are the most urgently needed compared to those at the L0 and L1, as they are particularly critical for defining the value judgments of agents in specialized domains like medicine and finance. These datasets represent the "last mile" between universal ethics and practical action. Future research should prioritize extracting ethical decision-making logic from proprietary data and transforming it into securely shareable formats with embedded ethical safeguards. Ultimately, the sharing model must evolve from data ownership toward value-added services, fostering a sustainable infrastructure that balances commercial viability with ethical objectives.

7.3 Value Alignment for Muti-Agent Interaction

The effective collaboration of LLM-based MAS relies on value alignment during their interactions. Future research can focus on three key activities in games: agent identification (involving individualism or collectivism), goal setting (concerning equilibrium vs. distribution, fairness vs. efficiency), and boundary constraints (considering information symmetry, interaction rounds, and short-term vs. long-term orientation), to systematically align the values in interactions.

7.4 Value Alignment in Communication Protocols

Communication protocols, including the Model Context Protocol (MCP) and Agent-to-Agent (A2A) interfaces, are essential for agent coordination but present scalability challenges to alignment. In MCP, value conflicts in external tool integration can lead to bias, while in A2A environments, a single agent's deviation can propagate throughout the network, made worse by the traceability issues of black-box decision-making. Future inquiry must address the prevention of external "pollution" at the protocol layer and develop methods to supervise complex emergent behaviors. As agents evolve into open ecosystems, regulatory granularity must expand from individual systems to entire networks to ensure that interaction remains robust, secure, and controllable.

8 Conclusion

Research on value alignment in agentic AI systems is gaining critical importance. This survey introduces a multi-level framework that structures value principles into a three-tiered system: L0 (universal), L1 (cultural/industrial), and L2 (contextspecific). We systematically review alignment methods across the technical stack, covering LLMs, single agent, and multiagent, and further synthesize existing evaluation benchmarks while highlighting their limitations. Finally, this paper discusses current challenges and outlines future directions, aiming to provide a structured foundation for developing trustworthy and socially-aligned agentic AI.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under the grant number 72472048 and 62506352, the Postdoctoral Fellowship Program of CPSF under GZC20250519, Youth Innovation Project of Computer Network Information Center, Chinese Academy of Sciences under CNICYIP2025-16.