An Empirical Game-Theoretic Analysis of Price Discovery in Prediction Markets

1 Introduction

Prediction markets offer a marketplace in which traders can buy or sell securities whose payoffs are based on the realizations of unknown future events [Wolfers and Zitzewitz, 2004]. Whereas traditional financial markets exist to facilitate investment, speculation, and hedging, the express purpose of a prediction market is to forecast future events. Prices in a prediction market have direct interpretations as event probabilities, and therefore price discovery in this context becomes synonymous with information aggregation.

The most common prediction market structure in the literature relies on a centralized market maker (MM) to provide liquidity, or the prevalence of opportunities to trade at current market prices. Central among these designs is the Logarithmic Market Scoring Rule (LMSR) devised by Hanson [2007], which uses a cost function to assign charges to trades and prices to securities. The cost function is parametrized by a liquidity parameter to control the sensitivity of prices to trade volume, and can be adapted to implement a spread between buy and sell prices. These parameters have an important effect on prediction performance and MM loss but can be difficult to set in practice, especially given the challenge of anticipating trader behavior [Brahma et al., 2012; Othman and Sandholm, 2010; Othman et al., 2013].

In this paper, we combine agent-based simulation, equilibrium computation, and real-world data to characterize the David M. Pennock Microsoft Research New York, NY dpennock@microsoft.com conditions that promote price discovery in a prediction market. We introduce the use of empirical game-theoretic analysis (EGTA) in the context of prediction markets to uncover trader strategies in equilibrium, which allows us to quantify the impact of different market parametrizations and environments. EGTA has been used to study multi-agent systems in financial markets [Wah and Wellman, 2015], and related approaches have examined continuous double auctions [Phelps et al., 2004; Walsh et al., 2002]. Our study is the first to employ EGTA to study prediction markets, and offers a proofof-concept of the technique in this context.

In our model, there is a single security traded in a market with an LMSR market maker. Our prediction market is populated by informed and less informed traders that each receive a lagged signal of a ground truth, as well as noise traders who create profit opportunities. The ground truth we use is based on prediction market data from Betfair on NBA games in the 2014-2015 season. Strategic traders formulate a belief on the value of the security by forming a convex combination of the MM's quoted price and their own information. We focus on trader behavior in equilibrium, where market participants are strategically responding to each other. We compare two different modes of market entry: one where traders enter only once, and one where they are allowed multiple entries.

Our methodology is as follows. We first verify that our ground-truth data is well-calibrated-that is, for any probability level, the proportion of price series at that level matches the proportion of realized outcomes. This confirms that the ground truth is a consistent signal of payout. To obtain payoff data for EGTA equilibrium computations, we simulate our prediction market model using a discrete-event simulation system that samples the ground truth with replacement. Once we identify equilibria, we evaluate market performance in terms of MM loss and price discovery on a hold-out sample of the ground truth, after confirming that in- and out-ofsample metrics such as player profits closely agree.

As expected, our results show that price discovery is sensitive to the liquidity and spread settings, as noted in practice. Our analysis also reveals that the mode of trader entry (single or multiple) impacts not only sensitivity, but also the overall relationships between liquidity, spread, price discovery, and MM loss. Informed traders prefer to trade as late as possible to fully exploit their information advantage-which is more valuable near the end of the trading horizon-but do not have this option in the single-entry setting. Price discovery therefore benefits from increased liquidity only in the multi-entry setting; otherwise, low liquidity improves price discovery by rapidly incorporating the information from trades.

2 RelatedWork

Much prior work examines different prediction market making mechanisms [Abernethy et al., 2011; Hanson, 2003], including extensions to ordinary prediction markets such as incorporating limit orders [Chakraborty et al., 2015; Heidari et al., 2015], employing pari-mutuel techniques for payoffs [Pennock, 2004] and predicting combinations of events [Dud´ık et al., 2012; 2013].

Price discovery is of primary interest in prediction markets, but the true probability of an event is unknown, rendering evaluation of real-world information aggregation over time near impossible. To address this, several studies employ agent-based modeling (ABM) and simulation to study price discovery, most using a random process as a basis for the ground truth probability [Brahma et al., 2012; Chakraborty et al., 2015; Slamka et al., 2013]. Jumadinova and Dasgupta [2011] use prediction market data from the Iowa Electronic Market to run simulations characterizing the role of information on trader behavior. Slamka et al. [2013] compare price discovery under four automated MMs, including LMSR and the dynamic pari-mutuel market [Pennock, 2004].

Other prior work focuses on deriving equilibria for heterogeneously informed traders, but many of these studies [Hanson and Oprea, 2009] have been restricted to simple models extending the classic model by Kyle [1985]. Pennock and Sami [2007] describe how a market employing LMSR converges to the rational expectations equilibrium price. Chen et al. [2007] study incentives to bluff strategically when trader information is conditionally dependent on the ground truth. Dimitrov and Sami [2008] construct a model with partially informed traders observing independent signals, showing that the myopically optimal strategy profile is not a weak perfect Bayesian equilibrium for an LMSR MM. Ostrovsky [2012] demonstrates that information about separable securities in certain markets with partially informed strategic traders is always aggregated in equilibrium.

To our knowledge, prior work does not explicitly compare equilibrium outcomes under different entry schemes. The theoretical literature typically relies on simple finiteperiod models [Dimitrov and Sami, 2008; Gao et al., 2013]; previous ABM studies (which do not compare equilibrium outcomes) generally implement single-entry schemes, with agents arriving randomly [Chakraborty et al., 2015], sequentially [Slamka et al., 2013], or on every time step [Jumadinova and Dasgupta, 2011].

3 Market Maker

In our model, there is a single security traded in a prediction market that operates as a continuous trading mechanism. The security pays off $1 if event E occurs and $0 if it does not. There are two possible outcomes: one where event E occurs and one where it does not (i.e., event E). There is a single market maker in our prediction market who acts as a counterparty for all transactions. The MM uses the standard logarithmic market scoring rule (LMSR) to charge for trades and to price securities [Hanson, 2007]. The MM has two available parameters: spread δ and liquidity parameter b. Both spread and the amount of liquidity are set a priori and are fixed for the duration of trading. Smaller b reflects lower liquidity, as prices will increase faster as traders buy shares from the market maker. Larger b reflects higher liquidity, as prices will increase slower as traders buy shares from the MM.

Cost Function We implement our model under a no-selling scheme in order to prevent traders from arbitraging the MM by selling a security paying out a guaranteed $1 for more than $1 [Othman et al., 2013]. The two-element vector qt = (qE↪t↪ qE↪t ) represents the total number of shares traders have bet at time t on event E and on E, respectively. From the market maker's perspective, qE↪t is the number of shares the MM has sold, whereas qE↪t is the number of shares of the security the MM has bought. All elements of qt are positive.

In the single-security setting where the LMSR market maker can set a nonnegative spread δ, the cost function is

The cost charged to a trader to buy x > 0 shares is

Cost is negative for sellers, as the trader will be compensated for its shares. The cost of selling x shares is

Due to the spread, a trader cannot buy some quantity and sell it back for zero cost: it pays (1+δ) per share to buy from and sell back to the MM, receiving a guaranteed payout of $1.

Quoted Prices At time t, BIDt corresponds to the price at which the MM offers to buy the security on event E, and ASKt is the quoted price at which the MM offers to sell the security. The quote on complementary event E is comprised of BIDt and ASKt, with BIDt < ASKt and BIDt < ASKt. The MM sets a spread δ ≥ 0, which is the difference between the BID and ASK. The midpoint of the spread can be interpreted as the market probability of event E. The current instantaneous price at which a trader arriving to the market can buy a share (i.e., bet on event E) is the cost of buying an infinitesimal amount from the MM:

Similarly, ASK is the price at which an entering trader can bet on event E (or equivalently, on event E not occurring):

The BID price is defined as BID = ASK - δ, and BID is defined similarly, with BID = ASK - δ. The ASK prices for the two outcomes (E and E) are given by vector pASK = ⇥ASKt↪ASKt ⇤, with ASKt + ASKt = 1 + δ▷ The BID prices are given by pBID = ⇥BIDt↪BIDt ⇤, with BIDt + BIDt = 1 - δ▷ Because these are complementary events, offering to buy (sell) from incoming traders on event E at price ASKt (BIDt) is equivalent to offering to sell (buy) on event E at BIDt (ASKt). Therefore, the following conditions hold for all times t: ASKt = 1 - BIDt and ASK t = 1 - BIDt↪ and the no-arbitrage condition holds.

The MM liquidates its inventory at time T , which denotes the end of the trading horizon. Its payoff is the revenue from trading, minus the value of liquidation:

Here q0 = (0↪ 0) is the initial quantity, and C(qT ) - C(q0) is the total amount paid to the MM.

4 Traders

We include strategic traders, whose payoffs are used in equilibrium computation, and noise traders in our model. Trader profit is the cash flow from trading.

4.1 Strategic Traders

Each strategic trader has an individual valuation for the security that is based on its private belief, which is a lagged signal of the ground truth. We include two types of traders in our model: informed traders receive information that is more recent, on average, than less informed traders.

Ground Truth The ground truth vt is the underlying true value of the security. It represents the probability at time t that the underlying event E will ultimately occur at time T. The probability of event E not happening is given by v0t , and the true values of the two complementary events always sum to 1: v0t = 1 - vt. At the end of the trading horizon, one of the two states of the world will be realized, so vT 2 ¶0↪ 1♦.

Private Values Each trader j has an individual private belief wj↪t regarding the probability of event E, with wj↪t 2 (0↪ 1)▷ An agent's belief wj↪t is a lagged signal of the true value vt at time t, that is, wj↪t = vt-∆t with lag drawn from an exponential distribution, or ∆t ⇠ Exp(λ). Less informed traders have a lower lag rate (and higher mean lag time) than informed traders. The lag rate of informed and less informed traders is λINF and λLESS, respectively, with λINF > λLESS.

Utility Function The traders in our model are risk neutral. The utility ⇡j of a trader j who pays (receives) ⇢ to buy (sell) qj shares is the surplus when liquidating at value vT :

At time T , a trader who bought (sold) the security will receive $1 ($0) per share if E occurs and $0 ($1) if it does not.

We impose a budget constraint c on the traders: the total cost of any buy order and the total redemption value (or liability) of any sell order must not exceed c. More precisely, the following conditions must hold for quantity qj > 0:

The quantity for a seller is analogous.

Single Entry vs. Multiple Entry There are two modes of market entry: single and multiple. Regardless of entry mode, each agent only participates in one trade. In the single-entry mode, traders enter the prediction market once, with time of entry te ⇠ U[0↪ T - 1]↪ where T is the length of the trading horizon in time steps. In the multi-entry mode, traders arrive to the prediction market over time, with time of arrival (and subsequent reentries) determined by an exponential distribution with rate λr. If a potential trade may not be profitable (i.e., BIDt < wj < ASKt), the agent does not submit an order and waits until its next reentry to reevaluate market conditions. It continues to reenter until it has successfully traded.

Strategies Traders in our model play a parametrization that we call the weighted belief update strategy, similar to that described by Jumadinova and Dasgupta [2011]. Trader j computes its belief as a convex combination of the current market price (the midpoint of the BID-ASK spread) and its individual private signal of the ground truth:

Traders optimize the weighting parameter ✓ (which is selected a priori and fixed for the entire trading duration) to maximize their payoffs. If xj↪t ≥ ASKt, the trader submits a buy order; if xj↪t  BIDt, the trader submits a sell order. If BIDt < xj↪t < ASKt, then the trader does not submit an order (but may have the opportunity to reenter later to trade, depending on the type of market entry). To move the ASK to its belief xj↪t, the optimal quantity q j > 0 for a buyer j is:

The optimal quantity for a seller is analogous. The trader either exhausts its budget or the price exceeds its private belief, so the actual quantity submitted is q = min¶q⇤ j ↪ q j ♦.

4.2 Noise Traders

Noise traders determine a priori whether or not to buy or sell, each with equal probability. They arrive into the market only once, with entry times uniformly distributed across the trading horizon. They buy or sell as much as permitted by their budget, so a buyer j computes quantity q⇤ j as above.

5 Empirical Game-Theoretic Analysis

Empirical game-theoretic analysis (EGTA) is a methodology for performing strategy selection by comparing the payoffs of different combinations of trader-strategy assignments. EGTA allows one to compare trader behavior in equilibrium under different market conditions. We give a high-level overview of EGTA here and refer to Wellman [2006] for complete details.

EGTA is an iterative process that involves discretizing the strategy space and analyzing the empirical game model induced by payoff data from simulations. Players are divided into roles and strategies are symmetric within roles. We generate equilibrium candidates by analyzing complete subgames, which are sets of strategies (one per role) for which we have collected data for all profile combinations. This gives us role-symmetric Nash equilibria (RSNE) of the subgames, which are equilibrium candidates for the full game. A subgame RSNE is confirmed as a full-game equilibrium if there are no beneficial deviations in the full strategy set. If deviations are found, they are added to the candidate's support, creating a new subgame. The process repeats until quiescence.

There are limitations that must be taken into consideration given this methodology. As game size grows exponentially with the number of players, we rely on deviation-preserving reduction (DPR) to construct a reduced-game approximation of the full game. DPR preserves payoffs from single-player deviations and has been shown to produce good approximations in other problems [Wiedenbeck and Wellman, 2012], but equilibrium estimates from DPR are not guaranteed. In addition, as we are unable to evaluate all profiles given the size of the game, we cannot guarantee that we have found all equilibria (even in the reduced game), although our process seeks to evaluate all promising equilibrium candidates.

There are two roles in our prediction market game, representing the informed and less informed traders, respectively. For our simulations we used 42 players, with 21 in each role. These specific numbers were chosen because they conveniently reduce via DPR to a (3↪ 3)-reduced game. We simulate our prediction market model using a discrete-event simulation system [Wah and Wellman, 2013] and we manage our experiments via the EGTAOnline infrastructure [Cassell and Wellman, 2013]. We mitigate sampling error by collecting payoff data over many simulation runs: a minimum of 10,000 samples per strategy profile evaluated, with 20,000 for the majority of profiles and at least 19,587 samples per profile on average. We use the RSNE computed to characterize the conditions under which the MM promotes price discovery.

5.1 Market Environment

We utilize real-world prediction market data from Betfair, an Internet betting exchange based in the U.K., as a basis for the ground truth, or the underlying true value of the security. We include data on NBA games from the 2014-2015 season, including the post-season. The price series for a given game is constructed by querying Betfair prices approximately every 2 to 3 minutes for the duration of the game. Excluding the price series with incomplete data, we have price information from 1126 NBA games. Each basketball game is associated with two price series: one reflecting the price of the security on the event the home team wins, and one for the security on the event the home team loses. Each simulation run entails drawing a sample ground truth (with replacement) from a pool of ground-truth price series.

We assess calibration of our data by binning forecasts to the nearest 10% mark according to the price at each time step (Figure 1). For each bin, we determine the percentage of time series for which the team in question wins. Since each bin represents a forecast, the data is perfectly calibrated when the average fraction of teams that win for every bin is equal to the average forecast probability of winning. We observe that the Betfair NBA data is well calibrated over time, which is typical of real betting markets. This confirms that traders in our model have a reason to take into account their signals when trading, since the ground truth correctly reflects the probability of a payoff at different time steps.

There are 14 noise traders in our prediction market, each with a budget of $0.10, which permits a maximum change in price of $0.05 when the market midquote price is $0.50 and the liquidity parameter is 1. Strategic traders (informed and less informed) have a budget of $0.50. There are 21 informed traders and 21 less informed traders. In the multi-entry setting, strategic traders arrive to the market according an exponential distribution with rate λr = 0▷125. We set the trading horizon T to 80 time steps to incorporate price data from the full duration of the game. The informed-trader lag rate is λINF = 0▷5; the less-informed lag rate is λLESS = 0▷1.

5.2 Experiments

We characterize performance of our prediction market primarily via out-of-sample price discovery, which we measure using the root-mean-square deviation (RMSD) between the midquote price from the MM and the ground truth at every time step [Brahma et al., 2012; Chakraborty et al., 2015]. Lower RMSD indicates better price discovery. We also examine how price discovery changes relative to MM loss, total welfare, and trader profit. We use four liquidity settings b 2 ¶1↪ 2↪ 5↪ 10♦ and four spread settings δ 2 ¶0↪ 0▷01↪ 0▷05↪ 0▷1♦ for the MM, giving a total of 16 games for each market entry mode, or 32 games total.

We separate the Betfair data into an in-sample dataset of 845 games, or approximately 75% of the full dataset, and an out-of-sample dataset of 281 games. We use data from the in-sample set for the empirical game-theoretic analysis. We analyze price discovery and welfare for the out-of-sample dataset via 10,000 simulation runs over the mixture probabilities in the role-symmetric equilibria found. Mixed-strategy RSNE are approximated by profiles with trader population proportions corresponding to the strategy probabilities. Our out-of-sample profit results are very close to those achieved on the in-sample dataset. The mean absolute error (MAE) between the in- and out-of-sample informed trader profits is 0.068 and 0.038 for the single and multiple entry cases, respectively (with profit values ranging from 0.158 to 2.247); the less-informed MAE is 0.063 for single-entry and 0.060 for multiple entries (with values between 0.026 and 0.944). The MAE in RMSD is 0.002 for both types of market entry; single-entry RMSD ranges from 0.099 to 0.158 and multientry RMSD ranges from 0.123 to 0.184. As such, we focus our evaluation on out-of-sample profit and RMSD.

EquilibriumAnalysis We find at least one and up to three RSNE 1 within each game. In cases with multiple RSNE, differences in trader and MM performance, as well as price discovery, are generally not significant. Figure 2 provides a visual summary of the regret for each strategy, which is defined as a player's loss in utility if deviating from a Nash equilibrium to the given strategy. We see that informed traders in the single-entry case generally minimize regret with ✓ = 0▷75, whereas those in the multi-entry case weigh their own information less heavily, with ✓ in equilibrium of 0.25. These values correspond to the most common pure-strategy RSNE. In both the single- and the multi-entry cases, the regret is higher for informed than less informed traders, indicating that the informed traders stand to lose more by deviating from equilibrium. The equilibrium strategies of less informed traders are similar across the two market entry scenarios.

The computed equilibria reveal substantially different behavior depending on the possibility of reentry. When traders only enter once, informed traders almost universally weigh their own lagged signal more than or equivalent to the less informed traders. This makes sense, as informed traders receive a more accurate signal of the ground truth than less informed traders. When traders can reenter, however, informed traders do not weigh their own information more heavily in equilibrium. This is because informed traders do not have a significant information advantage over the less informed traders early on: the ground truth price series tend to be most volatile near the end of the trading horizon, and in many cases, the price is fairly stable at the beginning. Consequently, informed traders are incentivized to trade as late as possible, and they avoid trading on their first entry into the market by weighing the market quote more heavily than their own information.

Trader Profit, MM Loss, and Welfare Figure 3 shows trader profits, MM loss, and overall welfare at spread 0.01; the qualitative trends here as liquidity increases are the same for the other spread settings. In all games, regardless of market entry type, informed traders have higher aggregate profit in equilibrium than less informed traders. Traders generate the highest profits when liquidity is high (i.e., for high b). Multi-entry traders obtain the highest profits when spread is 0.05; single-entry traders do the best when spread is 0. For both entry settings, the MM incurs the greatest losses at high levels of liquidity, and maximum welfare is obtained with the widest spread and the lowest liquidity.

1 Full details of all RSNE found are available in an online appendix (http://hdl.handle.net/2027.42/117580).

Aggregate trader profit and welfare is similar across the two market entry modes when the MM spread is large. In the single-entry case, trader profits decline as spread increases, for fixed liquidity. This is consistent with what might be expected, as a wider spread results in less trade. In the multientry setting, however, aggregate trader profits improve as the spread widens. This is because the larger the spread, the more likely agents will be able to defer trading to a later entry into the market, and information closer to the end of the trading horizon is more valuable, as previously mentioned.

Price Discovery We find that market conditions and trader reentry play a significant role in price discovery. Figure 4 illustrates the relationship between price discovery and MM profit (which is negative as the MM incurs a loss in all RSNE found). MM loss is typically around 10% of the maximum possible subsidy (b log 2), which increases with liquidity. In the multi-entry case, improvement in price discovery (i.e., lower RMSD) comes at additional cost to the MM, as traders are compensated for providing information to the market. The single-entry setting runs counter to this intuition, however: price discovery improves as MM profit increases.

In the multi-entry case, price discovery improves as liquidity increases. This follows intuition since higher liquidity means that prices do not change as quickly, so market prices are not as sensitive to noise trader activity. Optimal price discovery is achieved with spread δ = 0▷05 and with liquidity setting b = 5, as we see from Figure 5(b). In fact, δ = 0▷05

is the optimal spread at all liquidity levels. Price discovery is particularly sensitive to spread: RMSD deteriorates by 15% to 35% when moving from δ = 0▷05 to δ = 0 (for all liquidity settings). In contrast, Figure 5(a) shows that spread has little impact on price discovery in the single-entry case. Nonetheless, RMSD remains sensitive to the liquidity setting, with an improvement of up to 6% when moving from b = 2 to b = 5.

In the single-entry setting, price discovery is improved when liquidity is low. RMSD worsens with liquidity, regardless of spread setting. To understand why, note that when traders only enter once, they are forced to trade on their information irrespective of entry time. As a result, price discovery is best when liquidity is low and prices change quickly to reflect the information incorporated by any new trades.

6 Conclusions

In this paper, we employed empirical game-theoretic analysis to compare price discovery, as well as trader and market maker performance, in equilibrium. Our model was comprised of informed and less informed traders, who submit orders to a prediction market mediated by an LMSR market maker. We considered two market entry schemes: one where traders arrive once, and one where traders can reenter.

Our results demonstrate that trader reentry and market conditions are pivotal in promoting price discovery in a prediction market. When traders are allowed multiple entries, price discovery improves (up to a point) as liquidity increasesbut contrary to what might be expected, this relationship is inverted when traders can only enter once. Forecasting accuracy only benefits from higher liquidity when traders can be strategic about when they submit an order, as an information advantage near the end of the trading horizon is more valuable. The equilibria found also reflect this phenomenon: single-entry informed traders weigh their private information more, but are incentivized to postpone trading to a later entry in the multi-entry scenario.

The simulation-based methodology we use to elucidate the drivers behind price discovery offers a promising approach for evaluating other market design choices, such as a liquidity-adaptive MM [Othman et al., 2013] or a market making mechanism that learns from historical activity [Brahma et al., 2012; Das, 2005]. While our results depend on our specific modeling choices, in general we based our model on prior works in the literature. We explored a limited set of trader strategies in this study, and our equilibrium findings could be altered by the inclusion of additional strategies such as Zero-Intelligence [Gode and Sunder, 1993] or other types of traders and utility functions. Another open question regards the market maker parameters in equilibrium, and the impact of a strategic MM on price discovery.

Acknowledgments

We are grateful to Dean Foster and Michael Wellman for valuable discussions, David Rothschild for providing the BetFair data, and the anonymous reviewers for helpful feedback.