Normative Testimony and Belief Functions: A Formal Theory of Norm Learning

1 Introduction

It is widely accepted that our AI systems need to know what we humans value to effectively participate in our society. Research on modeling such evaluative phenomena is relatively new, but helpful analogies have been made to philosophical research in descriptive ethics [Jiang, L. et al., 2021; Olson, T., 2024]. The descriptive ethicist asks, "what does this population believe is good/bad/neutral?" To answer this, they gather relevant evidence and infer the population's normative beliefs. We work within this analogy here to construct a theoretically grounded formalism for norm learning.

We specifically consider learning from normative testimony, i.e., natural language expressions such as "you should help others". Independent of whether one is an optimist or pessimist about normative testimony [Hills, A., 2009] , these acts relay the norms of a speaker (i.e., their evaluative judgments) to a hearer. Such knowledge allows the hearer to better predict the speaker's behavior and, in cases where they disagree, make attempts to correct their beliefs. Therefore, having a strong formalism for learning from normative testimony is necessary to build socially competent AI systems. However, existing approaches have been mostly empirical, concerned with learning external responses (e.g., praise) [Hamid, N. H. A., et al., 2015; Sarathy, V. et al., 2017] rather than internal beliefs, and/or do not consider vital features like deontic ambiguity and speaker reliability (though Andrighetto, G. et al. [2010] do briefly discuss reliability).

This paper makes such a theoretical contribution and provides a philosophically justified formalism for learning a populations' normative beliefs from their normative testimony, considering both deontic ambiguity and speaker reliability. This work may be categorized as a novel approach to norm synthesis via group consensus [Kadir, M. R. A. and Selamat, A., 2018]. We start with a running example to build intuition around the process of learning from normative testimony. We then describe our representation scheme for normative beliefs and testimony. We then define a set of intuitive axioms of norm learning. Next, we construct our formal theory of norm learning, including philosophical justification and proof that it satisfies each axiom. Lastly, we theoretically evaluate its complexity and deontic consistency. We conclude with a discussion of related and future work.

2 Learning from Normative Testimony

Imagine you are visiting your friend Mary who just moved to a remote island, one which you have never visited. Let's call this society the Flarps. When you meet up with Mary at the Flarp library she leans over to you and whispers, "just so you know, you don't have to wear shoes in public." Nevertheless, you leave them on. Later that day at Mary's favorite coffee shop you witness the barista, Demarcus, get scolded by his boss. He then turns and yells at a customer, "put on your shoes!" And as you go about your day, you receive more normative testimony about wearing shoes in public.

Back at Mary's place and in for the night, you realize that you are quite certain that shoes are not required here, but you are unsure whether they are prohibited or just optional. Though Demarcus disagreed, he was under a bit of pressure, and many others said you could take them off. You have thus collected a set of normative testimony, fused this evidence in some fashion, and concluded the population's normative belief. This is the evidential process of descriptive ethics.

Many interesting questions arise about this process. Should the order in which you fuse the normative testimony matter? Should we weigh one more than another? We explore such questions here as we attempt to formalize norm learning. We start by detailing our formal representation scheme.

2.1 Formalizing Normative Concepts

Our language for representing deontic statuses is the ThreeFold Classification (TTC) of Deontic Logic [McNamara, P., 1996]. This consists of three mutually exclusive deontic statuses: {Obligatory, Impermissible, and Optional}, taken as "must be done", "must not be done", and "neither must nor must not be done" respectively. We also consider the weaker deontic concepts of Permissible as {Obligatory, Optional}, Omissible as {Optional, Impermissible}, and NonOptional as {Obligatory, Impermissible}.

We consider a norm as an internal normative belief. We modify existing formal representations for such mental attitudes [Olson, T. & Forbus, K., 2023; 2021; Santos, J. S. et al., 2017] to represent a set of agents, rather than individuals:

Definition 1 (Normative Belief). A normative belief is a fourwhere: ۧܦ,ܥ ,ܤ ,ܣۦ tuple

• Demarcus}; {Mary, e.g., agents of set a is ܣ ·
• WearingShoes); (e.g., behavior the represents ܤ ·
• InPublic); (e.g., context the represents ܥ ·
• Omissible). (e.g., status deontic a is TTC ∈ ܦ ·

evaluate(s) ܣ agent(s) the ,ܥ This can be read as "in context testimony. normative define we Next, ".ܦ as ܤ behavior

Definition 2 (Normative Testimony). An instance of normawhere: ۧܶ ,ܦ,ܥ ,ܤ ,ܵۦ tive testimony is a five-tuple

• Mary); (e.g., speaker the is ܵ ·
• WearingShoes); (e.g., behavior the represents ܤ ·
• InPublic); (e.g., context the represents ܥ ·
• Omissible); (e.g., status deontic a is TTC ∈ ܦ ·
• AtLibrary). (e.g., testimony the of context the is ܶ ·

Given these structures, learning a population's normative beliefs can be formally viewed with a function that aggregates a set of normative testimony to yield a normative belief: ,ۧD C, B, ,Aۦ → ሻ}ۧTn Dn, C, B, ,nSۦ …,ۧTͳ Dͳ, C, B, ,ͳSۦ{ሺܨ where each agent Sn ∈ A.

Our main contribution here is formally characterizing this fash- mathematical a in so do We .ܨ norm learning function ion and start with the axioms.

2.2 Norm Learning Axioms

recur- a into ܨ We can decompose our aggregation function ݕ and ݔ sion with a binary fusion operator as below. Where are relevant sets of normative testimony,

The binary function ⊚ essentially takes in two normative testimony and fuses them in some fashion to yield a combined normative belief. Considering our Flarps example, we present five intuitive axioms governing this function.

Axiom 1 (Conjunctive Pooling). Fusing normative testimony .ݕת ݔ ≅ ݕ⊚ݔ considers the agreement in opinions:

When merging normative testimony to learn a population's beliefs we must consider the agreement between the claims of its individuals. The disagreement between Mary and Demarcus would make us less confident that the Flarps believe wearing shoes in public is omissible or obligatory individually. But if they agreed, then they would strengthen each other's claims. Thus, ⊚ should be viewed as a conjunctive pooling operation [Dubois, D., & Prade, H., 1992].

Axiom 2 (Commutative). The result of fusing two normative .ݔ⊚ݕ = ݕ⊚ݔ testimony is the same, regardless of order:

A temporal ordering of events is created while considering Mary and Demarcus's normative testimony. But we argue that, once the evidence is evaluated by the learner (interpreted, assessed for reliability, etc.), the order in which it is fused should not fundamentally matter. For example, all else being equal, encountering Demarcus before meeting up with Mary should yield the same normative belief of the Flarps.

Axiom 3 (Associative). The result of repeatedly fusing normative testimony is the same regardless of how the body of .)ݖ⊚ݕሺ⊚ݔ = ݖ⊚ሻݕ⊚ݔሺ evidence is partitioned:

Norm learning is an online process and thus fusing a body of normative testimony should be associative (once the evidence is evaluated). This implies that fundamentally batch processes (e.g., LLMs like Delphi [Jiang, L. et al., 2021]) will not suffice for building true artificial social agents.

Axiom 4 (Idempotent). Fusing an instance of normative tes.ݔ = ݔ⊚ݔ timony with itself has no effect:

Repeatedly considering the same instance of normative testimony should not have any effect on our certainty in the corresponding normative belief. For instance, ruminating on Mary's testimony should not make us more certain that the Flarps believe wearing shoes in public is omissible.

Axiom 5 (Non-Dictatorial). Fusing two instances of normative testimony should preserve the opinion of both speakers (a formal definition is provided in later sections).

When merging evidence from different sources, the hearer should not ignore any set of speakers. Because we wish to know what the Flarps as a whole believe, Mary's normative testimony should not completely override Demarcus's nor any other of the citizens. Thus, ⊚ should be non-dictatorial [Dalla Pozza, G. et al., 2011]. We note that this axiom does interact with conjunctive pooling, as the intersection ignores disagreement. We discuss how to handle this in later sections.

3 Normative Testimony as Evidence

With these axioms in place, we now construct our evidence fusion operator. We start by considering how to formally represent normative testimony as evidence.

The simplest option would be to view normative testimony as a frequentist would and consider the distribution of the explicit statements. If every single Flarp that day said, "you must wear shoes in public", then we would simply claim that the Flarps believe it is obligatory. Such a view seems to reduce normative testimony to the speaker's normative belief. However, this is clearly not the case as we can lie with our normative testimony. Demarcus's normative testimony seemed to arise purely due to his boss. A remorseless meateater might say, "eating animals is wrong" only because they are with vegans.

Our semantics for normative testimony instead seem to be that an instance of normative testimony probabilistically bears on the corresponding belief. Presumably, all of the features-the speaker's bodily expressions, context of the discourse-come into play when determining this measure of reliability. E.g., if Demarcus's boss was not around, then his normative testimony would hold much more weight. We thus need to represent such reliability measures in our function testimony, the of reliability the is [Ͳ,ͳ] ∈ 𝑛ܲ like so: where → ሻ}ሻۧTn C,Dn, B, ,nSۦሺ𝑛ܲ … ,ሻۧTͳ Dͳ, C, B, ,ͳSۦሺଵܲ{ሺܨ .ۧD C, B, ,Aۦ

It seems intuitive that these probabilities are at least determined by the freedom of expression of the discourse context. However, it is unclear how to ground such measures. How do we determine when others are telling the truth about their internal mental states? We cannot perceive minds like we can perceive dice rolls. We examine this question next.

3.1 Probability in Normative Testimony

We could ground our reliability measures in behavior and reduce normative testimony to the overt performance of demand actions. This would then be confirmable via perception in the same way that regular testimony is. For example, you could determine Mary's reliability by observing if she praises or condemns people who wear shoes. However, our normative beliefs are what we should really wish to teach our AI systems. And our beliefs and overt behavior are not always consistent [Borg J.S. et al., 2006; Buckholtz, J.W. et al., 2015; Gold, N. et al., 2015]. An agent may believe an action is impermissible and still not scold others for it (and vice versa).

So, if we stick with our intended cognitivist interpretation of normative testimony as assigning some degree of support for the claim "I believe this action is good/bad/neutral", then must the hearer verify this claim in some way? If so, how?

We argue that such probability measures for reliability can be grounded in meta-cognition and theory of mind. Under this interpretation, the higher the probability assigned to a speaker's reliability, the more the hearer correctly relays their own normative beliefs when in the speaker's position. Thus, each of us have probability measures for "how often I, the hearer, say my true normative beliefs in situations like this." We then assume "other people are like me" and project these probabilities as the reliability of the speaker (for a similar hypothesis, see Meltzoff [2007] and for a computational model of theory of mind, see Rabkina et al. [2017]). We formally define our hypothesis below.

Hypothesis 1 (As Reliable as Me). Given a discourse context T, hearer H takes the chance that speaker S's normative tesbelief normative internal their relays ۧܶ ,ଵܦ,ଵܥ ,ଵܤ ,ܵۦ timony tes- normative own their that chance the as ,ۧ ଵܦ,ଵܥ ,ଵܤ ,}ܵ{ۦ own their relay would ,ۧܶ ,ଶܦ,ଶܥ ,ଶܤ ,ܪۦ timony in context T .ۧ ଶܦ,ଶܥ ,ଶܤ ,}ܪ{ۦ normative belief

By this hypothesis, our reliability measures for Mary and Demarcus' testimony are determined by how often we, the hearer, would be truthful about our normative beliefs in the Flarp library and when scolded by a boss respectively. With this philosophical grounding for such probabilities, we move on to continue formalizing our norm learning function.

4 Normative Testimony and Belief Functions

probabil- in takes ܨ At this point our norm learning function ities on normative testimony and computes normative beliefs, likely with some measure of certainty. Such a framework of course aligns with a Bayesian approach. However, such a view requires data sources we do not have. How could we ever get prior and conditional probabilities for normative beliefs? Again, we cannot perceive people's mental states like we can observe dice rolls. We argue that epistemics do not fit into such a chance picture, and thus Bayesian semantics do not fit here either.

Instead, we argue for a relaxation of subjective probability theory, specifically belief functions [Shafer, G., 1976]. Belief functions emerge when we cannot fit our question into traditional probability theory, but we can justify probabilities for a related question. We then extend these subjective probabilities to get degrees of belief for the question we are truly interested in. And this framing fits quite nicely here. While we may only have probabilities for how reliable an agent's normative testimony may be, this is of course related to the question of what their normative beliefs actually are.

To illustrate, under this interpretation we would first consider the chance that Mary is providing truthful normative is ݏ Again, .ݏ testimony in the Flarp library. Denote this as determined by reflecting on the self and determining how truthful we, the hearer, would be in this context. This then serves as a premise along a chain of reasoning to the Flarp's normative belief. Thus, we get the syllogism below (inspired by Shafer's [1981] example due to J.H. Lambert).

• P1: A belief of an agent is a belief of their group.
• P2: Mary is a Flarp.
• P3: Mary believes wearing shoes in public is omissible.
• C: Flarps believe wearing shoes in public is omissible.

above. three premise to ݏ We, the hearer, assign probability Therefore, our argument that "Flarps believe wearing shoes unsound and ݏ in public is omissible" is sound with chance semantics suitable are these that argue We .ݏͳ- with chance for normative testimony because while epistemics cannot fit into a chance picture, truth-telling can.

4.1 Background on Dempster-Shafer Theory

Here we specifically build upon Dempster-Shafer's theory of belief functions. AI researchers have applied this mathematical theory of evidence for sensor fusion [Premaratne, K. et al., 2009], learning indirect speech acts [Wen, R. et al., 2020], and more empirical norm learning work [Olson, T. and Forbus, K., 2021; 2023; Sarathy, V. et al., 2017]. We define the concepts of DS theory below.

Definition 3 (Frame of Discernment). DS theory considers an exhaustive set called the frame of discernment (FOD), denoted as Θ, of elements that are mutually exclusive. We can interpret each element of Θ as an answer to our question.

Definition 4 (Mass Assignment). A mass assignment, or basic that ,݉ belief assignment (BBA), is a function, denoted as maps each subset of Θ to a real number in [Ͳ,ͳ], such that ͳ = ሻܣሺ݉ ∑ and Ͳ = ሻ׎ሺ݉ ஺∈ଶ𝛩 . A mass assignment represents a judgment of the degree to which the evidence supports the set of propositions. Mass is assigned to non-singleton sets in cases of ambiguity or ignorance.

Definition 5 (Focal Element). The focal elements of a mass assignment are those sets with non-zero mass.

Definition 6 (Belief Function). Given set A, the belief funcሻܤሺ݉ =∑ ሻܣሺ݈݁ܤ = ܣ tion of ஻|஻⊆஺ .

Definition 7 (Plausibility Function). Given set A, the plausi∑ = ሻܣሺ݈ܲ = ܣ bility of ሻܤሺ݉ ׎஺≠ ת஻|஻ . We can also compute ሻ. ஼ܣሺ݈݁ܤͳ- = ሻܣሺ݈ܲ this from belief as:

5 Formalizing Norm Learning

Our approach to representing normative testimony as evidence in DS theory involves: 1) given the frame of discernment, convert normative testimony into an answer on this frame and 2) given a reliability measure, assign belief measures to this evidence, yielding a corresponding mass assignment. We define concepts relevant to this process below.

Definition 8 (Deontic Frame of Discernment). A deontic frame of discernment 𝛩௔,௕,௖ is the set of possible normative beliefs for a set of agents, a behavior, and a context: .}ܥܶܶ ∈ ݀ |ۧ݀ ,ܿ ,ܾ ,ܽۦ{

We use a few abbreviated notations for a frame. First, unless necessary, we omit certain features and write Θ. We denote specific subsets of a frame with a functional notation: ab- also We .ܥܶܶ ⊆ ܦ where }ܦ ∈ ݀ |ۧc,d b, ,aۦ{ = ሻܦΘሺ breviate non-singleton subsets with corresponding weaker .ሻ}݈ܾ݁𝑖ݏݏ𝑖ܱ݉Θሺ{ = ሻ}݌݉ܫ ,ݐ݌ܱΘሺ{ deontic status e.g.,

Definition 9 (Deontic Mass Assignment). A deontic mass assignment is a mass assignment on the subsets of a deontic frame. Formally, a deontic mass assignment is a mass assigndis- of frame deontic a is 𝛩 where [Ͳ,ͳ] → ʹ𝛩 :}௓…஺{݉ ment testimony. normative of instances are ܼ…ܣ cernment and

Definition 10 (Body of Evidence). Given deontic frame 𝛩, its deontic a is ௑݉ | ௑݉{ = ሻ𝛩ሺܧ݋ܤ body of evidence (BoE) is mass assignment on ʹ𝛩 }.

The ability to assign mass to subsets in DS theory, rather than just propositions, is powerful for norm learning as it naturally handles deontically ambiguous claims like Mary's. In without ,ሻ}݈ܾ݁𝑖ݏݏ𝑖ܱ݉ሺ{ this case, we assign mass to Θ஺,஻,஼ needing to decide how to distribute the mass to each disjunct.

We have formalized normative testimony as evidence. We now discuss how to aggregate this evidence to learn from .ܨ multiple sources i.e., we finally define our novel function Again, this involves chaining our binary fusion operator ⊚ over the body of evidence and we have argued that this operator must satisfy five axioms. Thankfully, Dempster has provided us with a binary operator that comes quite close.

Definition 11 (Dempster's Rule of Combination). Dempster's rule of combination (Dempster, 1967) computes the sum of the mass product intersections. It is defined as:

ሻܾሺ ଶ݉ሻܽሺ ଵ݉ ∑ = ሻܿሺ ଶ݉⊕ ଵ݉ ௖=௕ת௔ 𝛩 ⊆ ܿ∀ ሻܭ/ሺͳon assignments mass independent two are ଶ݉ and ଵ݉ where as: computed is ܭ the same frame 𝛩, and conflict measure ሻܾሺ ଶ݉ሻܽሺ ଵ݉ ∑ = ܭ ׎=௕ת௔ . We use the following set notation .ሻܿሺ ௒݉⊕ ௑݉ ≡ ሻܿሺ ௒׫௑݉ for a fused assignment:

5.1 Example: The Norms of the Flarps

To illustrate Dempster's rule in our formalism, we revisit the Flarps. We wish to know how they evaluate wearing shoes in = Θ where )ሻΘሺܧ݋ܤ(ܨ public. Thus, we are computing ,ۧObligatory InPublic, ,seohSgniraeW,spralFۦ{ ,ۧOptional InPublic, ,seohSgniraeW,spralFۦ

.}ۧImpermissible InPublic, ,seohSgniraeW,spralFۦ

as- ,ͳ݁ Because Mary is a Flarp, her normative testimony signs evidence on this frame, abbreviated as Θ. Assessed with as- mass deontic the produces ͳ݁ a Ͳ.ͻ degree of reliability, Ͳ.ͳ. = ሻΘሺ}ଵ𝑒{݉Ͳ.ͻ, = )ሻ}݈ܾ݁𝑖ݏݏ𝑖ܱ݉{ሺΘ(}ଵ𝑒{݉ signment:

De- when ,ʹ݁ We have another relevant piece of evidence, marcus yelled, "put on shoes!" Given the perceived pressure from his boss, we evaluated this as being much less reliable: Ͳ.͹. = ሻΘሺ}ଶ𝑒{݉Ͳ.͵, = )ሻ}݈ܾܱ{ሺΘ(}ଶ𝑒{݉

To determine the Flarps' normative belief, we then fuse these two pieces of evidence with Dempster's rule. This results in the mass product intersections shown in Table 1. Because Mary and Demarcus disagreed, there is mass assigned result- Ͳ.ʹ͹, = ܭ to the empty set. This is normalized out via ing in the fused mass assignment described below.

"Flarps believe wearing shoes in public is omissible"

Ͳ.ͺ͸͵ͳ; = Ͳ.͸͵/ሺͳ-Ͳ.ʹ͹ሻ = )ሻ}ܱ݉{ሺΘ(}ଶ𝑒,ଵ𝑒{݉ =

"Flarps believe wearing shoes in public is obligatory"

• Ͳ.ͲͶͳʹ; = Ͳ.Ͳ͵/ሺͳ-Ͳ.ʹ͹ሻ = )ሻ}݈ܾܱ{ሺΘ(}ଶ𝑒,ଵ𝑒{݉ =

"Flarps believe it is of some deontic status" (ignorance)

Ͳ.Ͳͻͷ͹. = Ͳ.ʹ͹ሻ Ͳ.Ͳ͹/ሺͳ- = ሺΘሻ }ଶ𝑒,ଵ𝑒{݉ =

Because Mary's testimony was taken as much more reliable than Demarcus's, the model is still quite certain that the Flarps believe wearing shoes in public is omissible:

Ͳ.ͺ͸͵ͳ; = )ሻ}ܱ݉{ሺΘ(}ଶ𝑒,ଵ𝑒{݉= )ሻ}ܱ݉{ሺΘ(}ଶ𝑒,ଵ𝑒{݈݁ܤ = ሻΘሺ}ଶ𝑒,ଵ𝑒{݉ + )ሻ}ܱ݉{ሺΘ(}ଶ𝑒,ଵ𝑒{݉ = )ሻ}ܱ݉{ሺΘ(}ଶ𝑒,ଵ𝑒{݈ܲ

Ͳ.ͺ͸͵ͳ+ Ͳ.Ͳͻͷ͹ = Ͳ.ͻͷͺͺ.

This example illustrates how norms are synthesized considering both deontic ambiguity and speaker reliability. Next we examine Dempster's rule with respect to our axioms.

5.2 A Modified Fusion Rule

Dempster's rule satisfies most of our axioms. It is a conjunctive pooling operation, and it is both commutative and associative [Sentz, K. & Ferson, S., 2002]. However, as noted previously, being a conjunctive pooling operation, Dempster's rule is dictatorial in cases where one mass assignment is definitive, and another non-agreeing mass assignment is uncertain. It is also not idempotent by default. Here we provide a modified fusion rule that satisfies these two conditions. We first provide a formal definition for non-dictatorial.

Definition 12 (Non-Dictatorial). We say that a fusion rule Ṙ Ͳ, > ሻ݂ሺ ଶ݉Ṙଵ݉ ሻ݂ ,ଶ݉,ଵ݉ሺ∀ is non-dictatorial when: frame same the on assignments mass are ଶ݉ and ଵ݉ where .ଵ݉ of elements focal ∈ ݂ and

To illustrate that Dempster's rule is dictatorial, imagine Mary's testimony was taken to be definitive as testimony Demarcus's Because ͳ.Ͳ. = )ሻ}ܱ݉{ሺΘ(}ଵ𝑒{݉ the is }ܱ݉{ and }݈ܾܱ{ does not agree (the intersection of empty set), it would be ignored. To avoid such a dictatorship, we must instead define our mass assignments correctly by assigning a non-zero mass to the frame. That is, a hearer must always leave a small portion of doubt in a speaker's claim. This is formalized with the following function, denoted with an overline. Where the amount of doubt 𝜀 ∈ ሺͲ,ͳሻ (default of 𝛩, ⊆ ݀ 𝛩, frame on assignment mass deontic a is ݉ Ͳ.ͳ),

Before fusion, a deontic mass assignment must now first be processed by this function. Not only does this ensure there can be no dictator, but it also fixes counterintuitive cases that result from normalization like that provided by Zadeh [1984].

We then use set-theoretic operations to ensure fusion is idempotent. Given two deontic mass assignments, if one has already been fused with the other (conditions 1 and 2 in equation 2), then the mass assignment with the most accumulated evidence is returned. If the two are disjoint (condition 3), then they are fused with Dempster's rule. However, if they intersect but one does not subsume the other (condition 4), then fusion is undefined. With these modifications, our updated is )"׫" fusion rule ⊚ (or in short form with the set union ,௑݉ formalized below. Given two deontic mass assignments Θ, frame deontic on ௒݉

Our fusion rule ⊚ is non-dictatorial, in short, because each deontic mass assignment now has a non-zero mass assigned to the entire frame, and thus all focal elements have a nonzero mass product intersection when fused, as the intersection of a focal element and the entire frame is itself. It is idempotent because of conditions 1 and 2 of equation 2.

This fusion rule now satisfies all of our norm learning axioms. It is a conjunctive pooling operation (axiom 1) and is commutative (axiom 2), associative (axiom 3), idempotent (axiom 4), and non-dictatorial (axiom 5). We have thus axiodefined ,ܨ matically constructed our norm learning function BoE, frame's deontic a of subsets are ݕ and ݔ below. Where

We move on to discuss truth-conditions within our semantics and introduce a final axiom along with proof that our formalism satisfies it.

5.3 The Semantics of Normative Belief

While it is useful to have degrees of belief, when acting or predicting based on learned norms, a definitive claim needs to be made. For example, if we wish to not upset anyone, then to decide whether or not to wear shoes in Flarpville we must convert our degree of belief to a definitive norm of the Flarps. We thus need to define when a normative belief is true.

Intuitively, a normative belief is true when its body of evidence sufficiently supports it. We formally define this below as the mean of belief and plausibility (estimating the true support) being greater than or equal to Ͳ.ͻ. We arbitrarily chose this high threshold, but one can imagine defining a personality spectrum from gullible (→ Ͳ.Ͳ) to skeptical (→ ͳ.Ͳ).

deontic with ۧ݀ ,ܿ ,ܾ ,ܽۦ =ࣨ Given a normative belief fused the from stemming functions ݈ܲ and ݈݁ܤ frame Θ, and as: ࣨ of truth the compute we ,ሻሻΘሺܧ݋ܤሺܨ mass assignment

For example, based on its body of evidence, the normative true, is ۧInPublic,Omissible ,seohSgniraeW,spralFۦ belief as the center of its belief and plausibility is greater than this threshold: ሺͲ.ͺ͸͵ͳ +Ͳ.ͻͷͺͺሻ/ʹ ൒ Ͳ.ͻ.

Under these semantics, inferences between deontic statuses should be sound. That is, our truth function above should not produce two deontically inconsistent normative beliefs (e.g., Mary can't believe wearing shoes in public is both omissible and obligatory). Disagreement should instead be represented with underlying uncertainty measures. This idea yields our sixth and final axiom of deontic consistency.

Axiom 6 (Deontically Consistent). Given deontically inconۧܦ,ܥ ,ܤ ,ܣۦ beliefs normative the ,ܧ and ܦ sistent statuses true. be both not should ۧܧ ,ܥ ,ܤ ,ܣۦ and

Next, we analyze the complexity of computing normative beliefs and prove that it is deontically consistent.

6 Theoretical Evaluation

Querying for the truth of a normative belief involves gathering its BoE by A) looping through all evidence and for each, B) comparing its agents, behavior, and context with those Lastly, .ሻܧ݋ܤሺܨ queried for. Then, C) fusing evidence with D) computing belief and plausibility values. The runtime of For .ሻDሺࣩ+ ሻCሺࣩ+ ]ሻBሺࣩ× ሻAሺࣩ[ this algorithm is thus: the as ݊ analyzing this complexity, consider the parameters relevant the of magnitude the as ݉ magnitude of all evidence, frame. deontic our of magnitude the as ݀ BoE, practically (though unbounded are ݉ and ݊ Theoretically, .ሻnሺࣩ = ሻAሺࣩ limited by the lifetime of the organism). Thus, The complexity of comparing a mass assignment's agents, behavior, and context with those queried for is bounded and the distilling of runtime the Therefore, .ሻͳሺࣩ = ሻBሺࣩ thus .ሻnሺࣩ = ሻͳሺࣩ× ሻnሺࣩ = ]ሻBሺࣩ× ሻAሺࣩ[ body of evidence is ͵, = ݀ As the deontic frame is bounded by the size of TTC at the complexity of fusing two deontic mass assignments is op- fusion pair-wise our Thus, .ܿ bounded by some constant operations. ͳሻ -݉ሺ × ܿ involves C) (step ሻܧ݋ܤሺܨ eration summa- a involves merely D Step .ሻ݉ሺࣩ = ሻCሺࣩ Therefore, .ሻͳሺࣩ = ሻDሺࣩ tion over a finite set and thus, Considering these bounds, computing the truth value of a normative belief (equation 4) has a linear time complexity of: is ݊ where ,ሻmሺࣩ+ ሻnሺࣩ = ሻͳሺࣩ+ ሻCሺࣩ+ ]ሻͳሺࣩ× ሻAሺࣩ[ amount the is ݉ the total amount of normative testimony and of normative testimony relevant to our query.

Next, we show that by utilizing DS theory our semantics are deontically consistent. We also show how this can slightly reduce the runtime of computing truth values.

6.1 Deontic Consistency of Normative Belief

leave we ۧܦ,ܥ ,ܤ ,ܣۦ To save space, for a normative belief We .ሻܦሺܤ as belief the abbreviate and implicit ܥ and ,ܤ ,ܣ .ሻ}ܦΘሺ{ abbreviate the subset of its deontic frame as

defined is ሻܦሺܤ We first prove that if the threshold for above Ͳ.ͷ, then no two non-intersecting subsets of its deontic frame can both be believed. We note that this also holds in the other direction but omit the proof to save space.

Theorem 1 (With a truth value threshold > 0.5, two inconbe ሻܦሺܤ sistent normative beliefs cannot both be true). Let If 𝛼. > ʹ/ሻሻ}ܦ{ሺ𝛩ሺ݈ܲ+ ሻሻ}ܦ{ሺ𝛩ሺ݈݁ܤ true if and only if 𝛩, frame deontic for true is ሻܦሺܤ 𝛼 ൒ Ͳ.ͷ, then given true. is ሻܧሺܤ and ׎ = ሻ}ܦ{ሺ𝛩ת ሻ}ܧ𝛩ሺ{ s.t. ሻ}ܧ{ሺ𝛩׍

if only and if true be ሻܺሺܤ Proof. Let α ൒ Ͳ.ͷ. Let + ሻሻ}ܺ{ሺΘሺ݈݁ܤ when or α, > ʹ/ሻሻ}ܺ{ሺ𝛩ሺ݈ܲ+ ሻሻ}ܺ{ሺΘሺ݈݁ܤ ת ሻ}ܦΘሺ{ s.t. Θ ⊆ ሻ}ܧΘሺ{ ,ሻ}ܦΘሺ{ Let ʹα. > ሻሻ}ܺ{ሺΘሺ݈ܲ Because true. are ሻܧሺܤ and ሻܦሺܤ Assume . ׎ = ሻ}ܧΘሺ{ we relation, ordinal the in plausibility rewriting true, is ሻܦሺܤ So, ͳ. ൒ ʹα > ]ሻ ஼ሻ}ܦ{ሺΘሺ݈݁ܤ - [ͳ + ሻሻ}ܦ{ሺΘሺ݈݁ܤ get: > ሻሻ}ܧ{ሺΘሺ݈݁ܤ Similarly, ሻ. ஼ሻ}ܦ{ሺΘሺ݈݁ܤ > ሻሻ}ܦ{ሺΘሺ݈݁ܤ ⊆ ሻ}ܦΘሺ{ , ׎ = ሻ}ܧ{ሺΘת ሻ}ܦΘሺ{ Because ). ஼ሻ}ܧ{ሺΘሺ݈݁ܤ belief the of definition By . ஼ሻ}ܦΘሺ{ ⊆ ሻ}ܧΘሺ{ and ஼ሻ}ܧΘሺ{ ൑ ሻሻ}ܦ{ሺΘሺ݈݁ܤ function and transitivity of subsets, ). ஼ሻ}ܦ{ሺΘሺ݈݁ܤ ൑ ሻሻ}ܧ{ሺΘሺ݈݁ܤ Similarly, ). ஼ሻ}ܧ{ሺΘሺ݈݁ܤ Lemma 1.1 follows.

.ሻܻሺ݈݁ܤ ൑ ሻܺሺ݈݁ܤ , 𝛩 ⊆ ܻ ⊆ ܺ Lemma 1.1 For any

> ሻሻ}ܦ{ሺΘሺ݈݁ܤ We now have the following ordinal relations: ሻሻ}ܧ{ሺΘሺ݈݁ܤ ൒ ሻ ஼ሻ}ܦ{ሺΘሺ݈݁ܤ > ሻሻ}ܧ{ሺΘሺ݈݁ܤ and > ሻሻ}ܦ{ሺΘሺ݈݁ܤ both Thus, .ሻሻ}ܦ{ሺΘሺ݈݁ܤ ൒ ሻ ஼ሻ}ܧ{ሺΘሺ݈݁ܤ

a is which ,ሻሻ}ܦ{ሺΘሺ݈݁ܤ > ሻሻ}ܧ{ሺΘሺ݈݁ܤ and ሻሻ}ܧ{ሺΘሺ݈݁ܤ □ true. be both cannot ሻܧሺܤ and ሻܦሺܤ contradiction. Thus,

To illustrate, theorem 1 holds that with a threshold of truth above Ͳ.ͷ, two normative beliefs like "Flarps believe wearing shoes in public is obligatory" and "Flarps believe wearing shoes in public is impermissible" cannot both be true.

Utilizing theorem 1, we now prove that certain relations of deontic logic are sound. For each theorem below, let the truth (equation previously as defined be ሻܦሺܤ of a normative belief 4) with a threshold of Ͳ.ͻ.

Theorem 2. The following syntactic entailments between normative beliefs in contrary deontic statuses are sound:

ሻܲܯܫሺܤ¬ ⊢ ሻܮܤܱሺܤ ሻܱܶܲሺܤ¬ ⊢ ሻܮܤܱሺܤ .ሻܱܶܲሺܤ¬ ⊢ ሻܲܯܫሺܤ

ת ሻ}ܱܶܲΘሺ{ Because true. be ሻܮܤܱሺܤ Proof. Let thresh- our and ,׎ = ሻ}ܮܤܱ{ሺΘת ሻ}ܲܯܫΘሺ{ ,׎ = ሻ}ܮܤܱΘሺ{ are ሻܲܯܫሺܤ¬ and ሻܱܶܲሺܤ¬ old Ͳ.ͻ > Ͳ.ͷ, by theorem 1, true. Without loss of generality, the same inference holds between all core deontic statuses. □

Theorem 3. The following syntactic entailments between normative beliefs in contradictory deontic statuses are sound:

ሻܯܱሺܤ¬ ⊢ ሻܮܤܱሺܤ ሻܱܱܶܲܰܰሺܤ¬ ⊢ ሻܱܶܲሺܤ .ሻܯܴܧܲሺܤ¬ ⊢ ሻܲܯܫሺܤ

= ሻ}ܯܱΘሺ{ definition, By true. be ሻܮܤܱሺܤ Proof. Let fol- It .׎ = ሻ}ܮܤܱ{ሺΘת ሻ}ܯܱΘሺ{ thus and ሻ}ܲܯܫ ,ܱܶܲΘሺ{ of loss Without true. is ሻܯܱሺܤ¬ lows from theorem 1 that and ,ܯܴܧܲ to ܲܯܫ generality, the same relation holds from .ܱܱܶܲܰܰ to ܱܶܲ □

Theorem 4. The following syntactic entailments between sub-

sumed normative beliefs are sound: ሻܱܱܶܲܰܰሺܤ ⊢ ሻܮܤܱሺܤ

ሻܯܴܧܲሺܤ ⊢ ሻܮܤܱሺܤ ሻܯܱሺܤ ⊢ ሻܲܯܫሺܤ ሻܱܱܶܲܰܰሺܤ ⊢ ሻܲܯܫሺܤ ሻܯܴܧܲሺܤ ⊢ ሻܱܶܲሺܤ .ሻܯܱሺܤ ⊢ ሻܱܶܲሺܤ

To aid in proving theorem 4, we first prove theorem 5.

Theorem 5 (If normative belief with deontic status D is true, then given E subsumes D, normative belief with deontic status .ሻܧሺܤ then ,ሻܦሺܤ if ,ሻ}ܧΘሺ{ ⊆ ሻ}ܦΘሺ{ E is true). Where

lemma By .ሻ}ܧΘሺ{ ⊆ ሻ}ܦΘሺ{ where true, be ሻܦሺܤ Proof. Let ⊆ ሻ}ܦΘሺ{ Because .ሻሻ}ܧ{ሺΘሺ݈݁ܤ ൑ ሻሻ}ܦ{ሺΘሺ݈݁ܤ 1.1, intersect also ሻ}ܦΘሺ{ with intersect that sets all ,ሻ}ܧΘሺ{ follows. 5.1 Lemma .ሻ}ܧΘሺ{ with

.ሻܻሺ݈ܲ ൑ ሻܺሺ݈ܲ 𝛩, ⊆ ܻ ⊆ ܺ Lemma 5.1. For any

is ሻܦሺܤ Because .ሻሻ}ܦ{ሺΘሺ݈ܲ ൒ ሻሻ}ܧ{ሺΘሺ݈ܲ By lemma 5.1, Ͳ.ͻ. ൒ ʹ/ሻሻ}ܦ{ሺΘሺ݈ܲ+ ሻሻ}ܦ{ሺΘሺ݈݁ܤ true, Thus, Therefore, well. as Ͳ.ͻ ൒ ʹ/ሻሻ}ܧ{ሺΘሺ݈ܲ+ ሻሻ}ܧ{ሺΘሺ݈݁ܤ true. is ሻܧሺܤ □

We now utilize theorem 5 to prove theorem 4.

⊆ ሻ}ܮܤܱΘሺ{ defined, As true. be ሻܮܤܱሺܤ Proof. Let 5, theorem by Thus, .ሻ}ܱܱܶܲܰܰ{ሺΘ,ሻ}ܯܴܧܲΘሺ{

of loss Without true. also are ሻܱܱܶܲܰܰሺܤ and ሻܯܴܧܲሺܤ generality, the same inference holds from belief in all deontic statuses to deontic statuses that subsume them. □

We have shown that certain syntactic inferences of standard deontic logic are sound given our semantics. This means that no two inconsistent normative beliefs can be learned and thus our formal theory is deontically consistent (satisfies axiom 6). This is practically useful as it ensures consistency when using learned beliefs to predict behavior. This also means that unless there is new evidence, we can simply prove normative beliefs rather than computing their truth value, reviewed is ܺ" ducing complexity. For instance, if it is true that permissi- as viewed is ܺ" as obligatory", then we can prove ble" via theorem 4, rather than calculating its truth value.

Note that we omit inheritance principles OB-RM (as described by Ross [1941]) or, in conditional form, CPI [Olson, T. and Forbus, K., 2023], from this calculus as they are unsound. To illustrate, consider a population that believes harming an agent is impermissible and that war entails harm. Thus, via inheritance it could be proven that the population also believes that war is impermissible. However, one can clearly imagine a model in which this is not true. We humans are not always consistent across such entailments and thus such inheritance principles should be unsound in epistemic formalisms. However, because they aim at ideal practical reasoning, such rules could be used to point out and then correct the nonrational normative beliefs of a population.

7 Related Work

The work presented here differs from existing norm learning/synthesis approaches that are grounded in external factors. That a sufficient number of individuals demand compliance with a norm, that it comes from an authority, has negative consequences, and so on (as in [Sarathy, V. et al., 2017] and [Hamid, N. H. A., et al., 2015]), are empirical conditions that can easily be verified via scientific method. However, verifying internal mental states is not as simple (As Reliable as Me hypothesis). This idea has been of particular interest in law, with respect to expert testimony on mental states and insanity defenses [Levy, K. M., 2019; Slobogin, C., 2008]. Recent neural approaches [Freitas dos Santos, T. et al., 2023; Jiang et al., 2021] differ for similar reasons. Also, though they can perform quite well empirically on tasks, being black boxes, their formal properties are difficult to examine.

In previous work [Olson and Forbus, 2021] we presented a norm learning implementation that also utilized DS theory. However, we left a lot of theoretical work undone. By working from axioms and formal definitions here, we have made fundamental improvements including a new fusion rule and a sound deontic calculus. But the biggest difference again lies in our probability semantics. Previously, probabilities represented "moral trust". However, such a model could not learn the norms of an immoral population, as their trust measures would be too low. If the goal is to learn prescriptive ethics (moral truths), then this is quite valuable. But here we are concerned with descriptive ethics. We wish to learn norms whether or not they are morally correct. We note that this should not be used prescriptively to guide the actions of AI

systems without explicit moral guardrails (e.g., Olson and Forbus [2023]). Instead, this can be used descriptively to predict behavior and correct immoral beliefs.

8 Conclusion and Future Work

While it is recognized that the ability to learn the normative beliefs of others is necessary for social competence, there exists few formal models of this process. Those approaches that do exist learn only external demand responses (e.g., praise), have been mostly empirical, and/or have ignored the uncertainty of internal mental states. This paper thus makes a theoretical contribution by providing a philosophically justified formalism for learning internal normative beliefs from normative testimony. We have provided a set of axioms for this process. We have provided an interpretation of probability measures as being grounded in the hearer's mental model. We have built an improved binary fusion operator and shown that it satisfies the axioms. Lastly, we have provided a complexity analysis of the norm learning algorithm and a sound deontic calculus for reasoning about learned norms.

We admit that hypothesis 1 introduces a sincere "cold start problem", as artificial agents lacking such knowledge cannot estimate reliability measures. This may be solvable by starting agents with needs and desires similar to ours, but this remains an open research question.

Though our work here is theoretical, it is also of practical use. As we have discussed, this could be implemented to learn norms of a population to then predict their behavior or change their beliefs. This will be an important capability for future artificial social agents. This could also be implemented for learning and comparing beliefs of subpopulations (e.g., left vs right wing ideals) considering reliability. In such applications, the reliability of each opinion depends on the context in which it is provided (hence the appeal of secret ballots).

In future work we plan to examine the potential for prejudice in our model, as it currently clumps agents who have not provided evidence in with others that have. To remove this feature, the algorithm would ensure each agent in the query has provided normative testimony. We also plan to explore strengthening the independence requirement and only consider each agent's most recent normative testimony. This would make the model fairer as a single agent could no longer flood it with evidence.

Acknowledgments

This research was sponsored by the US Air Force Office of Scientific Research under award number FA95550-20-10091. Taylor Olson was supported by an IBM Fellowship. We are grateful to Kyla Ebels-Duggan for discussions that helped shape ideas in this work. We also thank the anonymous reviewers for their feedback and suggestions.