What Is Best for Students, Numerical Scores or Letter Grades?

1 Introduction

Student evaluations and grading play an integral and influential role in every individual's academic experience. Naturally, there has been widespread debate among researchers and policy-makers about the efficacy of various grading systems such as letter v.s. number grades. For instance, coarsegrained grading schemes (i.e., letter grades) are considered to be less noisy indicators of performance, and stronger signals of status, and consequently, are the norm in North American universities. At the same time, there is also growing awareness that the grade itself affects performance independent of student ability, i.e., the grades are "not just an output of the educational process, they may also be an input" [Gray and Bunte, 2022]. For example, empirical evidence suggests the disclosure of midterm grades may motivate or demotivate students to perform better in a future exam, controlling for other effects. In light of this evidence, it is clear that the design of a grading system must be a deliberate choice that takes into account student welfare in addition to other extraneous factors [Guskey, 2011]. In this work, we take an analytical approach and study the design of an optimal grading system with a particular focus on numeric v.s. uniform letter grades. 1 As far as we are aware, this work is among the first to look at the problem of designing a grading scheme with the explicit objective of improving student performance in future tests. Our model captures the impact of grades on future performance via two well-motivated effects:

1. Anchoring: In any given test, students anchor themselves to (i.e., in expectation perform as well as) a specific score or performance level based on their intrinsic ability. We refer to this anchor as the intrinsic quality.
2. (De)Motivation: When the students' actual score falls above (below) their intrinsic quality, they get (de)motivated and subsequently, their expectation increases (decreases) for future tests. This is a phenomenon that has been widely noticed in practice [Deci et al., 1999; Dev, 1997; Cameron and Pierce, 1994].

In this regard, our work departs from other papers in this area, where students are often modelled as statusmaximizers [Dubey and Geanakoplos, 2010], i.e., their intrinsic motivation for a better grade stems from a desire to rank above their fellow students. Our model does not induce any artificial scarcity (status) and instead the fundamental friction is a result of noisy performance and how the same grading rule affects different students differently.

To better illustrate how different grading schemes impact student performance under our model, consider a student with an intrinsic quality of q1 = 85. Suppose that the student scores s1 = 81 in the midterm exam. Disclosing this numeric score may demotivate the student, which may reduce her effective intrinsic quality for the final exam. This adverse effect may be prevented if a (coarser) letter grading scheme is used, in which (say) all students (including the student under consideration) whose scores lie in [80, 90] are assigned a letter grade of A-. However, consider another student whose intrinsic quality is q2 = 91 and whose midterm score is s2 = 89. Receiving the same letter grade A- as everyone who scored in [80, 90] may be more demotivating to her than receiving her numerical score of s2 = 89. Hence, the overall effect of using a letter grading scheme remains unclear.

1 We use the term uniform letter grades to refer to letter grading schemes where each letter grade corresponds to an equal sized score range, e.g., [90, 100] →A+, [80, 90] →A-, and so on.

There is another subtle issue to be considered. While the comparison made by a student between her numerical score and intrinsic true quality is straightforward, 2 it is not obvious how a student should compare her intrinsic true quality to a letter grade received (such as A-). This depends on how the student perceives the letter grade. To that end, we use a scheme for mapping letter grades back to representative numerical scores: each letter grade is mapped to the midpoint of the interval containing all the scores that were mapped to that letter grade. For example, if all scores in the range [80, 90] are mapped to the letter grade A-, then A- is mapped back to (i.e., considered worth) a score of 85, which is what any student receiving A- would compare to her intrinsic quality.

This midpoint scheme has three in-built advantages. First, it reflects how the letter grades may truly be perceived in the outside world (and thus by the students) as it is actually used in the real world [University of Western Ontario, 2022; Victoria University of Wellington, 2022]. Second, the association of a letter grade to the midpoint of its score range accurately conveys the (average) performance of a student receiving that letter grade. Third, in the absence of any structure on how the grades are perceived, the question we ask in this work - which letter grading schemes would lead to the maximum average student performance? - would have a trivial and rather unsatisfactory answer: assign all students a grade worth 100 to maximally motivate them. The midpoint scheme makes this impossible: if the same grade (say A+) was assigned to all the students with scores anywhere in [0, 100], it would only be worth 50.

Building on the ideas presented in this example, we develop a framework to compare various grading systems in an environment with sequential testing. This includes evaluations within a course, e.g., a midterm followed by a final exam, but also grading across related courses, e.g., a student taking Calculus 101 followed by Calculus 102. Since a student's intrinsic quality increases after a test if the grade received is higher than her intrinsic quality and decreases otherwise, our aim is

to compare different grading schemes and choose the one that provides a higher quality improvement (or a lower quality degradation).

Our results. In this work, we compare the numerical scoring scheme, where the student learns her exact score in an evaluation, to uniform letter grading schemes, where the interval of scores is partitioned into T equal-length intervals mapping to different letter grades (and each interval is represented by its midpoint). While uniform letter grading is not completely realistic, we view our work as a starting point for the curiously unaddressed problem of quantitatively optimizing letter grading schemes and a stepping stone for future work to build on. That said, we note that real-world letter grading schemes (at least those used in North American universities) are close to uniform, once a very large interval mapped to the failing grade and a somewhat large interval mapped to very top grade are omitted. Since very few students fall in these two intervals, this omission does not significantly affect the overall analysis.

2 This assumes that students is aware of their own intrinsic true qualities, but it may suffice for them to have noisy estimates.

First, we theoretically study the case where two sequential evaluations take place, such as midterm and final exam. We show that under natural conditions, numerical scoring and all uniform letter grading schemes have equal performance when the motivational and demotivational effects are equally strong, and otherwise, either numerical scoring outperforms all uniform letter grading schemes or the opposite happens. Analytically identifying when each scheme outperforms the other turns out to be far from obvious and subtly dependent on properties of the distributions of intrinsic true qualities and scores, even for this limited setting. Using carefully constructed bijections between students, we are able to identify additional conditions under which numerical scoring outperforms all uniform letter grading schemes when the demotivational effect is stronger than the motivational effect, and the opposite happens when the demotivational effect is weaker than the motivational effect. Since there is significant evidence that negative events have a greater impact than positive events [Baumeister et al., 2001; Coleman et al., 1987], we expect the demotivational effect to be stronger than the motivational effect; thus, our results are in favour of numerical scoring.

Next, we empirically compare numerical scoring to uniform letter grading schemes. Under two sequential evaluations, we observe that numerical scoring continues to outperform uniform letter grading when the demotivational effect is stronger (and the opposite continues to hold when the motivational effect is stronger), even under more realistic conditions than in our theoretical analysis, such as when the true qualities of the students follow a (truncated) normal distribution. However, surprisingly, when more than two evaluations take place, the effect is reversed. Even after just six sequential evaluations, uniform letter grading begins to outperform numerical scoring when the demotivational effect is stronger (and the opposite holds when the motivational effect is stronger). In the intermediate stage between these two regimes, there is another surprising effect: with four sequential evaluations, numerical scoring outperforms uniform letter grading regardless of which effect is stronger!

Our results indicate that the choice of the grading scheme depends on the application at hand: with fewer evaluations (e.g., courses with just a few tests or shorter education programs with just a few semesters), numerical scoring may be better, while with many evaluations (e.g., courses with weekly tests or longer education programs), uniform letter grading may be better. At a high level, although our work draws on literature from fields such as economics and psychology, it provides a fundamental perspective on the question of student grading within the framework of multi-agent systems, i.e., where each student is modeled as an agent whose behavior depends on the decisions made by the system. Our results open up the possibility of designing grading systems that are easy to implement, approximately-optimal, and take into account students' incentives.

Related work. There is a rich literature on comparing grading schemes using various objectives. However, to the best of our knowledge, none of these papers study the objective of improving student quality that we focus on.

Several works have studied, both theoretically and empirically, how the effort exerted by students for an evaluation depends on the grading scheme to be used [Paredes, 2017; Brownback, 2018; Main and Ost, 2014; Czibor et al., 2020]. For example, when using pass/fail grading, a student may try hard enough to pass (with high probability), but not any harder. Our work is orthogonal to this: we focus on effect of the outcome of one evaluation on the student motivation in subsequent evaluations.

Another related work is that of Sikora [2015], who also compares grading schemes, but his goal is to study the tradeoff between conveying the most information about the student's true quality and minimizing noise due to factors unrelated to the true quality, not the (de)motivational effects of the grading scheme in subsequent evaluations. In our work, the task of keeping the grades "consistent" with the actual performance is indirectly performed by the midpoint scheme.

Rohe et al. [2006] and Bloodgood et al. [2020] also study how the grading scheme used may impact students' psychological well-being and stress levels, but do not focus on the impact of this in subsequent evaluations.

2 Model

Define [k] = {1, . . . , k} for k ∈ N. We introduce a model in which the grading scheme used in one evaluation can motivate or demotivate students, affecting their performance in future evaluations.

True qualities. A student begins with an intrinsic (true) quality q drawn from a (nonatomic) prior Q with probability density function (PDF) fQ(·). For simplicity, let the support of Q be [0, 1].

Scores. There is a score model S such that the numerical performance (score) of a student with true quality q in the first evaluation, denoted s ∈ [0, 1], is drawn from the (nonatomic) distribution S(q) with PDF fS(·; q). We focus on score models in which the expected score of a student is equal to their true quality, i.e., Es∼S(q)[s] = q for all q ∈ [0, 1].

Grades. A grading scheme is a function B : [0, 1] → [0, 1] that maps the score to a grade.

Letter grading. A letter grading scheme B⃗c is specified by a vector ⃗c = (c0 = 0, c1, . . . , cT-1, cT = 1), for some T ∈ N (referred to as the number of grades) and ci ⩾ ci-1 for all i ∈ [T], and is given by B⃗c(s) = ci-1+ci 2 for all i ∈ [T] and s ∈ [ci-1, ci). That is, it partitions [0, 1) into finitely many disjoint intervals (one for each grade) and maps a score to the midpoint of the interval containing it.

Uniform letter grading. We are particularly interested in the uniform letter grading (ULG) scheme. For a given number of grades T ∈ N, uniform letter grading with T grades, denoted ULGT , is specified by ci = i/T for each i ∈ [T]. In other words, it partitions [0, 1) into T equal-length intervals. We will use ∆(T) = 1/T to denote the length of the interval, dropping T from the argument when it is clear from the context. Formally, we have that for all s ∈ [0, 1), 3

For instance, ULG10 maps all scores in [0, 0.1) to 0.05, all scores in [0.1, 0.2) to 0.15, and so on. We restrict our focus to uniform grading schemes for two reasons: a) it is straightforward and easy to implement in practice; b) given that different institutions following different grading schemes, this allows us to broadly compare letter and number grading without getting lost in the minutiae. Further our assumption that each letter grade maps to the midpoint of an interval is common practice across universities [University of Western Ontario, 2022; Victoria University of Wellington, 2022] as well as the literature [McEwan et al., 2021; Nisbet, 1975]. More generally, it is consistent with the practice of assigning a score or gradepoint to each letter grade.

Numerical scoring. We will compare (uniform) letter grading to numerical scoring (NS), given by NS(s) = s for all s ∈ [0, 1]. Under numerical scoring, scores are not rounded to any grades. This can also be viewed as the limit of uniform letter grading with T → ∞ grades.

(De)motivation. The grades affect students' level of motivation in subsequent evaluations. Under grading scheme B, a student compares their true quality q to the obtained grade B(s). If the grade is higher than the true quality, the student experiences a motivational boost, but in the converse case, gets demotivated. We model this by assuming that the effective true quality of the student for the next evaluation changes to q′ = q + h(q, B(s)), where

We refer to αm, αd ∈ R⩾0 as motivation and demotivation coefficients, respectively. Note that the amount of (de)motivation is proportional to the difference between the obtained grade and the true quality. In the next evaluation, the student obtains a score s′ drawn from S(q′ ). We remark that when αm, αd ∈ [0, 1], we automatically have q′ ∈ [0, 1]; thus, we focus on this range of parameters. 4 Our choice of a linear model for demotivation follows from studies showing that student performance is linearly dependent on both external [Christensen and Menzel, 1998] and internal stimuli [Latham and Locke, 2007]. Additionally, even when the actual behaviour is more complex, our model serves as a firstorder approximation when (B(s) - q) is small.

Goal. Intuitively, we are interested in choosing grading schemes that achieve a higher increase (or a lower decrease) in the average student quality. Thus, we define the performance of a grading scheme B as:

where q′ = q + h(q, B(s)). Due to linearity of expectation,

3 Because we assume nonatomic distributions, it does not matter what ULGT (1) is. We will use the convention that ULGT (1) = 1.

4 In principle, one can also use larger coefficients and truncate q′ to lie in [0, 1].

Thus, we compare Eq∼Q,s∼S(q)[h(q, B(s))] under numerical scoring and uniform letter grading. Hereinafter, we omit q ∼ Q and s ∼ S(q) from an expression of expectation, whenever it is clear from the context.

Note that for our theoretical analysis, we focus on the case of two evaluations. Later, we empirically study the case of more than two evaluations.

3 Theoretical Results

In this section, we derive theoretical results for the performance of uniform letter grading schemes and numerical scoring, when students participate in two sequential evaluations and identify conditions under which numerical scoring outperforms every uniform letter grading scheme, and conditions under which the converse holds. Let us begin by introducing two useful definitions.

Definition 1 (Jointly Symmetric Distributions). We say that the true quality prior Q and the score model S are jointly symmetric if fQ(q) · fS(s; q) = fQ(1 - q) · fS(1 - s; 1 - q) for all s, q ∈ [0, 1].

Joint symmetry requires that true qualities and scores are symmetric across [0, 1]. That is, the probability of having true quality q and receiving score s should be the same as the probability of having true quality 1 - q and receiving score 1 - s. If the true quality prior is uniform, then this means the score distribution S(q) should be the mirror image of the score distribution S(1 - q). Note that joint symmetry does not necessarily require symmetry of the "noise" contained in the score compared to the true quality. For example, we do not need fS(s = 0.4; q = 0.5) = fS(s = 0.6; q = 0.5).

Definition 2 (Symmetric Grading Scheme). We say that a grading scheme B is symmetric if B(1 - s) = 1 - B(s) for all s ∈ [0, 1].

The reader can check that numerical scoring (NS) and uniform letter grading schemes (ULGT for any T ∈ N) are symmetric. Our first result shows that under such symmetry, the performance of the grading scheme is linear in the difference between the motivation and demotivation coefficients. As we later show in Corollary 1, this allows us to compare numerical scoring to uniform letter grading.

Theorem 1. When the true quality prior Q and the score model S are jointly symmetric, and the grading scheme B is symmetric, then we have

Proof. Note that due to Q and S being jointly symmetric, the pairs (q, s) and (1-q, 1-s) are sampled with equal density. Hence, we have that

Due to the symmetry of the grading scheme, we have B(1 - s) = 1 - B(s), which implies that the two terms h(q,B(s)) and h(1 - q,B(1 - s)) are motivation and demotivation by the same amount. Hence, we have that E[h(q, B(s)) + h(1 - q, B(1 - s))] = (αm - αd) · E[|q - B(s)|]. Plugging this into Equation (2), we get the result.

Corollary 1. Assume that the true quality prior Q and the score model S are jointly symmetric. Then, all symmetric grading schemes have equal performance if αm = αd. Further, if αm ̸= αd, for every T ∈ N one of the following conditions holds.

1. Uniform letter grading with T grades is at least as good as numerical scoring if αm > αd, and the converse holds if αm < αd.
2. Uniform letter grading with T grades is at least as good as numerical scoring if αm < αd, and the converse holds if αm > αd.

Proof. The first claim regarding αm = αd follows immediately from Equation (1). For the second claim regarding αm ̸= αd, note that the comparison between numerical scoring and uniform letter grading with T buckets reduces to the sign of E[|q - NS(s)| - |q - ULGT (s)|], and depending on this sign, one of the two statements in the corollary holds.

Corollary 1 tells us that having equal motivation and demotivation coefficients (αm = αd) is the turning point: between uniform letter grading with a fixed number of grades and numerical scoring, one is better when αm < αd but the other becomes better when αm > αd. But it does not tell us which one is better in each case.

Our next result identifies a sufficient condition under which this dilemma is settled: uniform letter grading is better when αm > αd and numerical scoring is better when αm < αd. To introduce this sufficient condition, we need to define the following natural property of the score model.

Definition 3 (Ex-Ante Single-Peaked Score Model). We say that the score model S is ex-ante single-peaked if, for every q ∈ [0, 1], fS(·; q) is single-peaked with the peak at q, i.e., fS(s; q) ⩽ fS(s′ ; q) for all s ⩽ s′ ⩽ q and s ⩾ s′ ⩾ q.

Intuitively, in an ex-ante single-peaked score model, scores closer to the true quality are more likely than scores farther from the true quality.

For a fixed T, we also denote with D the set of all pairs of true qualities and scores that belong to the same letter grade interval, i.e., D = {(q, s) : ULGT (q) = ULGT (s)}. For example, if T = 10, (q = 0.51, s = 0.59) ∈ D but (q = 0.51, s′ = 0.49) /∈ D.

Theorem 2. Fix any T ∈ N. Assume that the true quality prior Q and the score model S satisfy the following.

1. Q and S are jointly symmetric;
2. S is ex-ante single-peaked; and

Then, the first implication of Corollary 1 holds. That is, uniform letter grading with T grades is at least as good as numerical scoring if αm > αd, the converse holds if αm < αd, and the two have equal performance if αm = αd.

Before diving into the proof, let us make a remark regarding the third technical condition in Theorem 2. The technical condition states that, averaged over all such pairs, the true quality is closer to the score than to the midpoint of the interval that they both belong to. Later, we show that this condition is satisfied in two natural cases. Intuitively, if the score distribution is sufficiently concentrated near the true quality, the expected distance between the score and the true quality will be sufficiently small, satisfying the condition. Let us now turn to the proof of Theorem 2.

Proof sketch. Given Theorem 1, we simply need to show that E[|q - s|] ⩽ E[|q - ULGT (s)|]. We already assume that this holds conditioned on (q, s) ∈ D. Hence, we only need to show that it also holds conditioned on (q, s) /∈ D. We show this given the additional single-peakedness property. In fact, we show that conditioned on (q, s) /∈ D, the desired equation actually holds for every q ∈ [0, 1], and, thus, in expectation over q ∼ Q too. To see why this is true, fix any q ∈ [0, 1] and consider two cases based on whether the letter grade mapping for score s (i.e., ULGT (s)) is smaller or larger than the corresponding grade for q (i.e., ULGT (q)).

In the first case where ULGT (s) < ULGT (q), the singlepeakedness property implies that |q - s| = q - s ⩽ q - ULGT (s) in expectation over s. Conversely, when ULGT (s) > ULGT (q), one can infer that |q - s| = s - q ⩽ ULGT (s) - q = |q -ULGT (s)| in expectation over s, based on the same argument. Putting these two cases together, we get the third statement in the theorem. A formal proof can be found in the full version.

In Theorem 2, we argued that single-peakedness of S establishes the desired inequality of E[|q - s|] ⩽ E[|q - ULGT (s)|] at least conditioned on (q, s) /∈ D, leaving only the case of (q, s) ∈ D, which was stated as an assumption in. Next, we show that if the true quality prior Q is uniform over [0, 1], and it satisfies two natural assumptions then the desired inequality also holds conditioned on (q, s) ∈ D.

Definition 4 (Ex-Post Single-Peaked Score Model). We say that the score model S is ex-post single-peaked if, for every s ∈ [0, 1], fS(s; ·) is single-peaked with the peak at s, i.e., fS(s; q) ⩽ fS(s; q′ ) for all s ⩽ q′ ⩽ q and q ⩽ q′ ⩽ s.

Definition 5 (Probabilistic Single-Dipped Score Model). We say that the score model S is probabilistic single-dipped if, for every x ∈ [0, 1], Pr[s ∈ [q, x] ∪ [x, q] ∣∣∣ q] (let us call this p(x, q)) is single-dipped in q with the dip at q = x, i.e., p(x, q) ⩽ p(x, q′ ) for all x ⩽ q′ ⩽ q and q ⩽ q′ ⩽ x.

Before we state the next theorem, we further partition D into two sub-spaces, Dsame and Dopp , such that Dsame contains the set of all pairs of true qualities and scores such that either both are at most or both are at least the midpoint of their common letter grade interval, i.e.

and Dopp = D \ Dsame . For example, when T = 10, (q = 0.54, s = 0.51) ∈ Dsame , but (q = 0.54, s′ = 0.56) ∈ Dopp . We are now ready to state the result.

Theorem 3. Fix arbitrary T ∈ N. Assume the following regarding the true quality prior Q and the score model S.

1. Q is uniform over [0, 1];
2. Q and S are jointly symmetric;
3. S is ex-ante and ex-post single-peaked, and probabilistic single-dipped; and

Then, the first implication of Corollary 1 holds. That is, uniform letter grading with T grades is at least as good as numerical scoring if αm > αd, the converse holds if αm < αd, and the two have equal performance if αm = αd.

The proofs of Theorems 3 and 4 are our most intricate proofs. However, due to space constraints, we have deferred them to the full version. 5 Let us now understand the assumptions in Theorem 3. A natural choice of S under which Assumptions 3 and 4 in Theorem 3 are satisfied is when S(q) is a symmetric distribution around q, i.e., the noise in the score follows a symmetric zero-mean distribution. Further, for such a score model, we have γ = 1, so Assumption 4 becomes Pr[(q, s) ∈ Dsame ] ⩾ 4 · Pr[(q, s) ∈ Dopp ]. More general, from the definitions of Dsame and Dopp , when the variance of the score distribution is sufficiently small, we can expect Pr[(q, s) ∈ Dsame ] to be much higher than Pr[(q, s) ∈ Dopp ]. The full version includes a figure providing further intuition.

Ex-ante single-peakedness, ex-post single-peakedness, and probabilistic single-dippedness can be subsumed into a single property that captures a stronger form of symmetry, in which the noise in the score is symmetric and zero-mean.

Definition 6 (Strongly Symmetric Score Model). We say that the score model S is strongly symmetric if fS(s; q) = ℓ(|s - q|) for some non-increasing function ℓ : R⩾0 → R⩾0.

Under a strongly symmetric score model, we have γ = 1 in Assumption 4 of Theorem 3, which means a constant of 2(γ + 1) = 4 would be needed. However, using different techniques, we can show that even a constant of 3 suffices to obtain the same result under strong symmetry. This broadens the scope to include less concentrated score models.

Theorem 4. Fix arbitrary T ∈ N. Let D, Dsame , and Dopp be defined as in Theorem 3. Assume the following regarding the true quality prior Q and the score model S.

1. Q is uniform over [0, 1];
2. S is strongly symmetric; and

Then, the first implication of Corollary 1 holds. That is, uniform letter grading with T grades is at least as good as numerical scoring if αm > αd, the converse holds if αm < αd, and the two have equal performance if αm = αd.

We remark that in the proof of Theorem 4, we only really need strong symmetry for pairs of true qualities and scores that belong to the same letter grade interval.

5 Full version: www.cs.toronto.edu/∼nisarg/papers/grading.pdf

4 Experiments

In the previous section, we proved that when Q is uniformly distributed and the variance of the score model is small, we can conclude that the first implication of Corollary 1 holds. In this section, we empirically compare numerical scoring and uniform letter grading while relaxing these assumptions.

First, it is widely believed that students' true qualities, at least in large classes, are normally distributed based on the evidence that "exam scores tend to be normally distributed for well-constructed, norm-referenced, multiple choice tests" [Wedell et al., 1989]. Hence, we empirically study the case where Q is normally distributed, truncated to [0, 1]. We also consider cases where the score is not necessarily concentrated around the true quality. Finally, our analysis was limited to two evaluations; in our experiments, we also consider more than two evaluations. When a student participates in r sequential evaluations, after each evaluation the student compares her "current" true quality to the obtained grade, and experiences (de)motivation that affects her effective true quality in the next evaluation. Formally, for j ∈ [r], let qj and sj denote her effective true quality and score in evaluation j, respectively. Then, sj ∼ S(qj) for each j ∈ [r], and for j ∈ [r - 1], we have:

We measure the performance of a grading scheme by comparing the final true quality, qr, to the initial true quality q1, which extends the performance measure introduced in preliminaries for two evaluations:

Data generation. For all the simulations, we compare numerical scoring (NS) to uniform letter grading (ULGT ) with T ∈ {4, 8, 12, 16, 20} grades. We scale the interval of grades to [0, 100] to resemble percentage grades. We simulate n = 5000 students (average results are plotted with 95% confidence intervals), where the initial true quality q1 of each student is drawn i.i.d. from a truncated normal distribution capped to [0, 100], with the underlying normal distribution characterized by mean µ and standard deviation σ. Given a true quality q in an evaluation, the score s is drawn from another truncated normal distribution capped to [0, 100], with the underlying normal distribution characterized by mean q and standard deviation γ.

Results. Figure 1 shows how the final quality improves (or degrades) with respect to the motivation coefficient (top) and the number of evaluations (bottom). In Figure 1a, the motivation coefficient takes values in {0, 0.1 . . . , 0.9, 1}, the demotivation coefficient is set to 0.5 and the number of evaluations is set to r = 2. We see that when αm < αd, numerical scoring is better than any uniform letter grading (and uniform letter grading with more grades is better than uniform letter grading with fewer grades), whereas when αm > αd, the opposite is true. Hence, it seems that the first implication of Corollary 1 continues to hold, even when the true quality is drawn from more realistic distributions. The comparison between uniform letter grading schemes with different numbers of grades is intuitive: uniform letter grading essentially converges to numerical scoring when T goes to infinity, so larger T should resemble numerical scoring more. The experiments show that this holds even with small values of T .

Going beyond our theoretical analysis for r = 2 evaluations, we consider the case where students participate in more than two evaluations. Surprisingly, as seen in Figures 1c and 1d, the comparison between numerical scoring and uniform letter grading flips completely with large values of r: numerical scoring becomes worse than uniform letter grading (and ULGT becomes worse than ULGT ′ for T > T ′ ) when αm < αd, but better when αd < αm. This shows that the choice of the grading scheme depends not only on the comparison between the strengths of motivational and demotivational effects (αm vs αd) but also, crucially, on the number of evaluations r. When αm < αd, with fewer evaluations (e.g., courses with fewer tests or curricula with fewer semesters), use of numerical scoring may be recommended, whereas with many evaluations (e.g., courses with frequent tests or curricula with many semesters), use of uniform letter grading with fewer letters may be more appropriate.

The transition between the regimes of few evaluations and many evaluations is even more surprising. As seen in Figure 1b, with r = 4 evaluations, numerical scoring seems to outperform uniform letter grading schemes regardless of the comparison between αm and αd. Hence, in general, it is always best to simulate different grading schemes under the model and the number of evaluations of interest in order to pick a suitable grading scheme.

Finally, we observe that under numerical scoring, as the number of evaluations increases, the average student quality declines linearly when αm < αd (Figure 1c) and improves linearly when αm > αd (Figure 1d). This is expected because it can be shown that under numerical scoring, every evaluation changes the expected student quality by the same amount, which is proportional to αm-αd, leading to a linear decline or growth. In contrast, under uniform letter grading schemes with very few grades (small T), the average student quality seems to converge and remain stable as the number of evaluations increases, regardless of the comparison between αm and αd. This can be explained due to the following stabilizing effect. Let [ℓ, h] be a letter grade interval and m be its midpoint. Consider a student who starts with a true quality q ∈ [ℓ, h]. The student is likely to receive a score s in the same interval [ℓ, h] (so that (q, s) ∈ D), and thus, a grade of m. This causes the true quality to update in a manner so that it gets closer to m after which the student experiences very little motivation or demotivation due to receiving a grade that is almost equal to her true quality. Of course, the effect is more pronounced when T is small, so letter grade intervals are large compared to the variance of the score model.

Due to the space constraints, we have presented only the most striking observations, deferring the rest to the full version.

5 Discussion

Our work takes the first step towards proposing a statistical model of the impact of letter grading schemes on student per-

(c) αm = 0.2

(d) αm = 0.8

Figure 1: Performance of numerical scoring and different uniform letter grading schemes, with µ = 65, σ = 12, γ = 1.5 and αd = 0.5 over different motivation coefficients (top) and number of evaluations (bottom). 95% confidence intervals are shown.

formance in sequential evaluations and using it to compare uniform letter grading schemes to numerical scoring. We view our work as a stepping stone and outline appealing extensions below.

Beyond midpoint grading. In our model, we assume that if all the scores from an interval [ℓ, u] are mapped to the same grade, they are effectively mapped to the midpoint grade (ℓ + u)/2. While this is a common method in practice of converting letter grades to percentages [University of Western Ontario, 2022; Victoria University of Wellington, 2022], other values within the range [ℓ, u] are also sometimes used [University of Waterloo, 2022].

Non-uniform letter grading. It would be interesting to extend our analysis to non-uniform letter grading schemes. More broadly, how can our model be extended to incorporate truly non-numeric grades (e.g., A, B, etc.) without converting them to numeric grades somehow (e.g., 4, 3.7, etc.)?

Non-linear (de)motivation. Evidence from prospect theory suggests that motivational effects from positive outcomes are typically concave (diminishing rewards) while demotivational effects from negative outcomes are typically convex

(increasing losses) [Kahneman and Tversky, 1979]. It would be interesting to study such nonlinear effects.

Exploring implications to pedagogy and beyond. There is a growing literature on optimizing design choices in AI-based learning systems, e.g., algorithmically deciding which explanations to show to students [Zavaleta-Bernuy et al., 2021]. Our insights may inform the design of personalized grading schemes in such systems; they can adjust grade disclosure by learning over time whether students respond more strongly to motivation or demotivation.

More broadly, insights from our work can be explored in other multi-agent systems, such as contest design [Levy et al., 2017] and crowdsourcing [Han et al., 2020], where agents participate in rounds, and feedback from earlier rounds can influence the effort in subsequent rounds. For example, under the right conditions, Theorem 3 may suggest a leaderboard design where teams are grouped into buckets (analogously to letter grading) and their exact performance is not revealed.