Compact Representation of Value Function in Partially Observable Stochastic Games

1 Introduction

Partially Observable Stochastic Games (POSGs) are a very general model of dynamic multi-agent interactions under uncertainty, and they can be used for modeling dynamic problems where players can react to other players based on limited, imperfect observations. Examples include patrolling [Basilico et al., 2009; Vorobeychik et al., 2014; Brazdil et al., 2018] where a defender protects a set of targets against an attacker, pursuit-evasion games [Chung et al., 2011] where a pursuer is trying to find and apprehend an evader. Security games including green security games [Fang et al., 2015; 2016] and cybersecurity games [Nguyen et al., 2017]) can also be generalized so that the defender can observe and react to attackers during the game.

Unfortunately, computing optimal strategies in POSGs is highly intractable from the algorithmic perspective, even in the two-player zero-sum setting. When both players have partial information about the environment, the players may need to reason not only about their belief over possible states, but also about the belief the opponent has over the possible states, beliefs over beliefs, and so on. Restricting to subclasses of POSGs where this issue of nested beliefs does not arise allows us to design and implement algorithms that are guaranteed to converge to optimal strategies [Hor´ak et al., 2017; Hor´ak and Boˇsansk´y, 2019]. However, the scalability of the current algorithms even for this case is limited.

One of the fundamental problems is the complexity of representing and reasoning about uncertainty over a potentially very large state space. In these POSGs (as in single-agent Partially Observable Markov Decision Processes (POMDPs)), beliefs are probability distributions over the possible states. This is a well-known disadvantage of these models, since memory and computation time grow rapidly due to the curse of dimensionality. Taking a related approach to previous work on POMDPs (e.g., in [Roy et al., 2005; Li et al., 2010; Zhou et al., 2010]), we address this problem by introducing a compact representation of the uncertainty in POSGs, and we develop a novel algorithm based on this representation that dramatically improves the scalability.

As a motivating domain, we consider a cybersecurity example where an attacker uses lateral movement actions to expand his control in a network without being detected. The defender tries to observe the attacker and take actions to protect the network by reconfiguring honeypots. A perfectinformation version of this problem was proposed in [Kamdem et al., 2017], however, we focus on a more realistic model where the defender has a limited information about the attacker. This version of the lateral movement game with uncertainty can be modeled as a one-sided POSG (OSPOSG) [Hor´ak et al., 2017]. Originally, the belief of the defender is defined over the possible subsets of resources that the attacker may currently control in the network. This representation scales exponentially in the size of the network so it is intractable for all but the smallest examples. We return to our motivating domain in Section 4 where key steps of our novel algorithm are discussed specifically for this domain.

Our main technical contribution is replacing the representation of beliefs over the exponential number of possible states using a compact characteristic vector that captures key information but reduces the dimensionality of the beliefs. Specifically, we propose using the marginal probability of each resource being infected as a characteristic vector instead of explicitly considering all possible subsets of infected resources. While this offers a path to scaling to much larger problems, it has consequences for the solution quality as well as the construction of the solution algorithm. Many components of state-of-the-art POSG solvers are based on manipulating the full belief distribution and we show how these can be redesigned to operate on more compact characteristic vectors. We formally define the fixed-point equation, show that the value function in the compact representation is still a convex function, and that solving a compact representation of the game yields a lower bound on the value of the original game. For our motivating domain, the novel algorithm operating on the compact representation scales orders of magnitude better with a negligible loss in quality (less than 1%) compared to the state of the art algorithm.

2 One-sided POSGs

A one-sided partially observable stochastic game [Hor´ak et al., 2017], or OS-POSG, is an imperfect-information twoplayer zero-sum infinite-horizon game with perfect recall represented by a tuple (S, A1, A2, O, T, R). The game is played for an infinite number of stages. At each stage, the game is in one of the states s ∈ S and the players choose their actions a1 ∈ A1 and a2 ∈ A2 simultaneously. An initial state of the game is drawn from a probability distribution b0 ∈ ∆(S) over states termed the initial belief. The one-sided nature of the game translates to the fact that while player 1 lacks detailed information about the course of the game, player 2 is able to observe the game perfectly (i.e., his only uncertainty is the action a1 player 1 is about to take in the current stage).

The choice of actions determines the outcome of the current stage: player 1 gets an observation o ∈ O and the game transitions to a state s′ ∈ S with probability T(o, s′ | s, a1, a2), where s is the current state. Furthermore, player 1 gets a reward R(s, a1, a2) for this transition, and player 2 receives -R(s, a1, a2). The rewards are discounted over time with discount factor γ < 1.

OS-POSGs can be solved by approximating the optimal value function V ∗ : ∆(S) → R, mapping beliefs b ∈ ∆(S) of player 1 to his expected utility in belief b, using a pair of value functions V and V (lower and upper bound on V ∗ ). The algorithm from [Hor´ak et al., 2017] refines these bounds by solving a sequence of stage games and is guaranteed to converge to an ϵ-approximation of the optimal solution. In each of these stage games, the algorithm finds the optimal strategies of the players in this stage (i.e., π1 ∈ ∆(A1) for player 1 and π2 : S → ∆(A2) for player 2) while assuming that the current belief of player 1 is b and the play in the subsequent stages yields values represented by value functions V or V , respectively. Solving the stage games also defines the fixed point equation for V ∗ w.r.t. the dynamic operator H,

where τ(b, a1, π2, o) denotes the Bayesian update of the belief of player 1 when he played a1 and observed o. Recall that observations of player 1 depend both on the states of the game and actions of player 2 who has complete information-hence the dependency of Prb,π1,π2 [a1, o] on b and π2.

For piecewise-linear and convex V and V , a stage game can be solved using linear programming. We show this linear program for V (represented as a point-wise maximum over a set Γ of linear functions αi : ∆(S) →R).

In this linear program, player 2 seeks a strategy π2 for the current stage of the game to minimize the utility V of player 1. Here π2(s ∧ a2) stands for the joint probability (ensured by (2b) and (2f)) that the current state of the game is s and player 2 plays the action a2. Constraint (2c) stands for player 1 choosing the best-responding action a1. Constraint (2d) expresses the belief ba1o in the subsequent stage of the game when a1 was played by player 1 and observation o was seen (multiplied by the probability of seeing that observation), and finally constraint (2e) represents the value of V in the belief ba1o . Such a linear program can be then combined with point-based variants of the value-iteration algorithm, such as Heuristic Search Value Iteration (HSVI) [Smith and Simmons, 2004; Hor´ak et al., 2017].

A different approximation scheme is used for the upper bound V on V ∗ . Instead of using the point-wise maximum over a set of linear functions αi ∈ Γ, V is expressed using a set Υ = {(b(i) , y(i) ) | 1 ≤ i ≤ |Υ|} of points (where b(i) is the coordinate of a point in the belief space, and y(i) is its associated value). The lower convex hull of this set of points is then formed to obtain the value of V (b),

Here, λ stands for the coefficients of a convex combination of points in Υ (requiring that the convex combination of the coordinates b(i) of the points matches the belief b). Constraint (2e) can then be adapted to use this representation of V as detailed in [Hor´ak and Boˇsansk´y, 2016].

3 Compact Representation of V ∗

The dimension of the value function V ∗ is the number of states, which is exponential in the size of the network for our motivating domain. We propose an abstraction scheme called summarized abstraction to decrease the dimensionality of the problem by creating a simplified representation of the beliefs over the state space.

We associate each belief b ∈ ∆(S) in the game with its characteristic vector χ(b) = A · b (for some fixed matrix A ∈ Rk×|S| where k ≪ |S|) and we define an (approximate) value function ˜ V ∗ : Rk → R over characteristic vectors.

The main goal is to adapt algorithms based on value iteration to operate over the more compact space Rk instead of the original belief space ∆(S). First, we adapt the fixed point equation (1) for OS-POSGs to work with compact ˜ V ∗ (instead of original V ∗ ). We let the player 2 choose any belief that is consistent with the current characteristic vector χ (by adding an extra minimization term), and we replace the value of belief b, V ∗ (b), by the value of its characterization ˜ V ∗ (A · b). We also denote this compound function ˜ V ∗ ◦ A.

Next, we show that the value function ˜ V ∗ satisfying the fixed point equation (4) (and obtained as a limit of applying operator ˜H) provides a valid lower bound on the solution of the original game representation over the belief space.

Proof sketch. Let ˜ V0 be arbitrary and V0(b) = ˜ V0(χ(b) ). Construct sequences {˜ Vt}∞t=0 and {Vt}∞t=0 such that ˜ Vt+1 = ˜H ˜ Vt and Vt+1 = HVt. Assume ˜ Vt(χ(b) ) ≤ Vt(b) for every b (which holds for t = 0). Now H(˜ Vt ◦ A) ≤ HVt. The extra minimization over beliefs b in ˜H ˜ Vt can only decrease the utility and hence ˜ Vt+1(χ(b) ) = ˜H ˜ Vt(χ(b) ) ≤ H( ˜ Vt ◦ A)(b) ≤ HVt(b) = Vt+1(b). This extends to the limits ˜ V ∗ and V ∗ .

Note that the equation (4) can also be solved using linear programming. Consider a lower bound ˜ VLB on ˜ V ∗ formed as a point-wise maximum over linear functions αi(χ) = (a(i) )T χ+z(i) . We modify the linear program (2) by considering that the belief b is a variable (constrained by χ),

and we replace the constraint (2e) to account for different representation of ˜ VLB by

where qa1o = 1T · ba1o is the probability that o is generated when player 1 uses action a1.

Observe that the only constraints with non-zero right-hand side in this new linear program are constraints (5b) and (5c). This is critical in the following proof that ˜ V ∗ is convex.

Algorithm 1: HSVI algorithm for OS-POSGs when summarized abstraction is used.

• 1 Initialize ˜ VLB and ˜ VUB to lower and upper bound on ˜ V ∗ 2 while ˜ VUB(χ0 ) - ˜ VLB(χ0 ) > ϵ do 3 Explore(χ0 , ϵ, 0) 4 procedure Explore(χ, ϵ, t) 5 (b, π2) ← optimal belief and strategy of player 2 in ˜H ˜ VLB(χ) 6 (a1, o) ← select according to heuristic 7 χ′ ← τ(χ, a1, π2, o) 8 Update Γ and Υ based on the solutions of ˜H˜ VLB(χ) and ˜H˜ VUB(χ) 9 if ˜ VUB(χ′ ) - ˜ VLB(χ′ ) > ϵγ-t then 10 Explore(χ′ , ϵ, t + 1) 11 Update Γ and Υ based on the solutions of

˜H ˜ VLB(χ) and ˜H˜ VUB(χ)

Theorem 2. Value function ˜ V ∗ is convex.

Proof. Consider a dual formulation of the linear program (2) updated according to equations (5). Since the only non-zero right-hand side terms in the primal are 1 and χ, the objective of the dual formulation is o(χ) = χT · a + z. Moreover, this is the only place where the characteristic vector χ occurs. Hence the polytope of the feasible solutions of the dual problem is the same for every characteristic vector χ ∈ Rk and o(χ) (after fixing variables a and z) forms a lower bound on the objective value of the solution for arbitrary χ. Since we maximize over all possible o(χ) in the dual, the objective value of the linear program (and also H ˜ VLB) is convex in the parameter χ.

Now, starting with an arbitrary convex (e.g., linear) ˜ V 0 LB, the sequence of functions {˜ V t LB}∞t=0 , where ˜ V t+1 LB = ˜H ˜ V t LB, is formed only by convex functions. Therefore, the fixed point ˜ V ∗ is also a convex function.

3.1 HSVI Algorithm for Compact POSGs

The Algorithm 1 we propose for solving abstracted games is a modified version of the original heuristic search value iteration algorithm (HSVI) for solving unabstracted OSPOSGs [Hor´ak et al., 2017]. The key difference is that we use the value functions ˜ VLB and ˜ VUB (instead of V and V ) and we have modified all parts of the algorithm to use the abstracted representation of the beliefs.

First, we initialize bounds ˜ VLB and ˜ VUB (line 1) to valid piecewise linear and convex lower and upper bounds on ˜ V ∗ (using sets Γ of linear functions αi = (a(i) )T χ + z(i) and Υ of points (χ(i) , y(i) )). Then, we perform a sequence of trials (lines 2-3) from the initial characteristic vector χ0 = Ab0 until the desired precision ϵ > 0 is reached.

In each of the trials, we first compute the optimal optimistic strategy of player 2, which in this case is the selection of the belief b and strategy π2 (line 5). Next, we choose the action a1 of player 1 and the observation o (lines 6-7) so that the excess approximation error ˜ VUB(χ′ ) - ˜ VLB(χ′ ) - ϵγ-t in the subsequent stage (where the belief is described by a characteristic vector χ′ = τ(χ, a1, π2, o)) is maximized. If this excess approximation error is positive, we recurse to the characteristic vector χ′ (line 10).

Before and after the recursion we update the bounds ˜ VLB(χ) and ˜ VUB(χ) using the dynamic operator ˜H (lines 8 and 11). The update of ˜ VUB is straightforward and a new point (χ, ˜H ˜ VUB(χ)) is added to Υ. To obtain a new linear function to add to the set Γ, we use the objective function o(χ) = χT · a + z (after fixing variables a and z) of the dual linear program to (5) that forms a lower bound on ˜ V ∗ .

4 The Lateral Movement POSG

We now specify the approach for a cybersecurity problem of detecting and mitigating lateral movement of an attacker in a network. The game is played on a directed acyclic graph G = (V,E), where the vertices V = {v1, . . . , v|V |} are sorted in topological order. The goal of the attacker is to reach vertex v|V | from the initial source of the infection v1 by traversing the edges of the graph, while minimizing the cost to do so.

Initially, the attacker only controls the vertex v1, i.e., the initial infection is I0 = {v1}. In every stage of the game, the attacker chooses a directed path P = {P(i)}ki=1 (where k is the length of the path) from any of the infected vertices to the target vertex v|V |. Unless the defender takes countermeasures, the attacker infects all of the vertices on the path, including the target vertex v|V |, which ends the game. The attacker pays the total cost of traversing the edges on the path,

where C(P(i)) is a cost associated with taking edge P(i).

The defender tries to discover the attacker and increase his costs by deploying a honeypot in the network. The honeypot is deployed on an edge (denote this edge h) of the graph, and is able to detect if the attacker traverses that specific edge. Furthermore, it also increases the cost for using the edge h to C(h). If the attacker observes that he has traversed a honeypot, he may decide to change his plan and therefore does not execute the rest of his originally intended path P . The cost of playing a path P against a honeypot placement h is therefore

where P≤h is the prefix of the path P until the interaction with the honeypot edge h,

Since the attacker does not necessarily continue to execute his selected path P (because the defender can reconfigure the position of the honeypot), the new infection Ih,P (i.e., state of the game) becomes

This problem can be formalized as a OS-POSG where the states are possible infections (S = 2V ), defender actions are honeypot allocations (A1 = E), and actions A2 of the attacker are paths in G reaching v|V |. Observations denote whether the defender detected the attacker (i.e., h ∈ P ) or the attacker has reached the target v|V | while avoiding the detection, O = {det, ¬det}1 . Transitions follow the equation (9) and observation det is generated iff the honeypot edge h has been traversed. The reward of the defender is the negative value of the cost of the attacker (R(h, P) = -ch,P ), the discount factor of the game is γ = 1 2 , and the initial belief b0 of the game satisfies b0 (I0) = 1.

4.1 Characteristic Vectors

The number of states in the game is exponential in the number of vertices of the graph, |S| = 2|V |-2 + 1 (we consider that v1 is always infected and we treat all states where v|V | is infected as a single terminal state of the game). We propose using the marginal probabilities of a vertex being infected as characteristic vectors χ ∈ R|V | , i.e.

where I corresponds to some subset of possibly infected vertices of the graph and b(I) denotes the original belief over this subset of vertices being infected. Note that the linear projection introduced in equation (10) defines a projection matrix A ∈ R|V |×|S| from Section 3.

4.2 Value Function Representation

The algorithm from Section 3.1 approximates ˜ V ∗ using a pair of value functions, the lower bound ˜ VLB and the upper bound ˜ VUB. While we have discussed representing the lower bound using a set Γ of linear functions αi(χ) = (a(i) )T χ+z(i) , representing the upper bound is more challenging. In the original algorithm, the value function V is defined (see equation (3)) over the probability simplex ∆(S). In that case, it suffices to consider |S| points in Υ to define V for every belief. In contrast, the space of characteristic vectors here (i.e., marginal probabilities) is formed by a hypercube [0, 1]|V | with 2|V | vertices, which makes the straightforward point representation (using equation (3)) impractical.

However, we can leverage the fact that in this domain infecting an additional node can only decrease the cost to the target (and hence ˜ V ∗ is decreasing). Consider the dual formulation of the optimization problem (3). In this formulation, the projection of χ to the lower convex hull of a set of points is represented by the optimal linear function aT χ+z defining a facet of the convex hull (see Figure 1). Since ˜ V ∗ is decreasing in χ, we can also enforce that aT χ + z is decreasing in χ (i.e., add the constraint a ≤ 0 to the dual formulation).

1 Since we assume a perfect-recall game, the defender knows the current allocation of the honeypot and is able to infer the edge where the attacker has been detected upon receiving det observation.

2 While the original HSVI algorithm for OS-POSGs has been defined and proven for problems with γ < 1, the convergence properties translate even to the undiscounted case in this case since the game is essentially finite (in a finite number of steps, all vertices, including v|V |, get infected and the game ends).

This additional constraint translates to changing the equality ∑1≤i≤|Υ| λiχ(i) = χ in the primal problem to an inequality.

Now it is sufficient that the set Υ contains just one point (χ(i) , y(i) ) where χ(i) = 0|V | (instead of 2|V | points) to make the constraint ∑1≤i≤|Υ| λiχ(i) ≤ χ satisfiable.

It is possible to adapt the constraint (5g) to use the representation from (11), similarly to the original (unabstracted) OS-POSGs [Hor´ak and Boˇsansk´y, 2016], and obtain a linear program to solve the stage games w.r.t. the upper bound.

4.3 UsingMarginalized Strategies in Stage Games

The linear program formed by modifications from equations (5) still requires solving the stage game for the original, unabstracted problem. In this section, we show that it is possible to avoid expressing the belief b explicitly, and to compute the stage game directly using the characteristic vectors and marginalized strategies of the attacker.

First, we present the representation of the stage-game strategies of the attacker. Instead of representing joint probabilities π2(I ∧ P) of choosing path P in state I, we only model the marginalized probability ˜π2(P) of choosing path P aggregated over all states I ∈ S. Furthermore, we allow the attacker to choose the probability ξ(P ∧ vi) that vertex vi is infected while he opts to follow path P .

To ensure that the strategy represented by variables ˜π2 and ξ is feasible it must be consistent with the characteristic vector χ, where χi is the probability that the vertex vi is infected at the beginning of the stage. By marginalizing the variables ξ(·) representing the joint probabilities we get

Furthermore, the path P must start in an already infected vertex (denoted as Pre(P)), i.e., the conditional probability Pr[Pre(P ) ∈ I | P ] of Pre(P ) being infected when path P is chosen has to be 1. Now, since ξ(P ∧v) is the joint probability, ξ(P ∧ v) = Pr[Pre(P) ∈ I | P] · ˜π2(P),

This representation of attacker strategies is sufficient to express the expected immediate reward of the strategy ˜π2, hence the constraint (2c) can be changed to use the marginalized strategies,

Importantly, we can also skip the computation of the belief bho and compute the characteristic vector formed by the marginals χho directly from the variables ˜π2 and ξ. We now present the equation to compute the updated marginal χh,det given that the attacker has been detected while traversing the honeypot edge h.

The first sum stands for the probability that the attacker is detected while traversing edge h, but he infected vi beforehand. The second sum represents the probability that the attacker used edge h as well, but this time he has not infected vi using path P, however, the vertex vi has already been infected before starting to execute path P .

Analogously, we can obtain the probability qh,det that the attacker got detected while traversing edge h as

We need not consider the subsequent stages where the attacker has not been detected (i.e., ¬det observation has been generated) or the honeypot edge h reaches the target vertex v|V |. In both of these cases, the target vertex has been reached and thus the value of the subsequent stage is zero.

4.4 Initializing Bounds

We now describe how we initialize bounds ˜ VLB and ˜ VUB (line 1 of Algorithm 1). Denote by C ∗ (vi) and C ∗ (vi) the costs of the shortest (i.e., cheapest) paths from vi to the target v|V | when costs C and C, respectively, are used for all edges.

We initialize the lower bound ˜ VLB using two linear functions α1(χ) = 0 and α2(χ) = (a(2) )T χ + z(i) such that z(i) = C ∗ (v1) and a(2) i =min{C ∗ (vi) -C ∗ (v1), 0}.

To initialize the upper bound ˜ VUB, we consider that exactly one vertex vi is infected and we consider the most expensive path C ∗ (vi) from vi to the target v|V |. We use the set Υ = {(χ(j) , y(j) ) | 1 ≤ j ≤ |V |} of points to initialize ˜ VUB where

5 Experimental Results

In this section we experimentally evaluate the scalability and properties of our proposed abstraction technique based on the model from Section 4. Unless otherwise stated, we consider directed acyclic graphs generated from the Erd˝os-R´enyi model with probability p = 0.5 that each of the possible edges is included. Furthermore, we make sure to include edges (vi, vi+1) for 1 ≤ i ≤ |V | - 1. We set the costs C(vi, vj) = j - i for every edge (vi, vj) ∈ E when the honeypot is not deployed to (vi, vj). We do, however, apply a significant penalty for traversing a honeypot edge where C(vi, vj) = j(j - i). This setting is used to model layered networks typical of critical infrastructure [Kuipers and Fabro, 2006]. The attacker can either proceed to the subsequent layer (using edge (vi, vi+1)), or use a shortcut (if available). The costs C reflect the fact that the closer the attacker is to the target, the more secure the network is (i.e., he has to use a more expensive exploit to proceed).

All of the experiments were run on a machine with an Intel i7-8700K and 32GB of RAM. We used CPLEX 12.7.1 to solve the linear programs used in the algorithms.

5.1 Comparison with the Original Approach

We used a set of randomly generated graphs (varying the number of vertices) and we attempted to solve these instances using both the original (unabstracted) approach and the algorithm we present in this paper. The parameters used for the original algorithm follow the parameters proposed in [Hor´ak et al., 2017]. We modified the initialization of the bounds to make it valid for the undiscounted problem in question. The target precision ϵ = 0.1 was used for both algorithms.

Figure 2 shows the runtimes of the original algorithm applied to the unabstracted game and our proposed approach. For small instances, the original approach is competitive and outperforms our proposed approach. For larger instances, however, the compact representation used in our approach turns out to be significantly better. Moreover, the original approach was unable to solve 50% of the 11-vertex instances due to memory requirements. In contrast, our approach based on marginal probabilities did not exceed 1GB of memory even when applied to the most challenging 17vertex instances.

5.2 Abstraction Quality

We now focus on the abstraction quality since the summarized abstraction may lose information needed to derive the optimal behavior. However, we show that with properly designed characteristic vectors the quality loss can be minimal.

Due to the insufficient scalability of the original unabstracted approach, we can do the comparison only for graphs with 5 ≤ |V | ≤ 10 vertices where the original approach solved all of the instances. In Table 1, we present the empirical bound on the relative distance from the equilibrium of the unabstracted bound. We compare the upper bound V (b0 ) computed by the original approach for the unabstracted game with the lower bound ˜ VLB(χ0 ) on the quality of the strategy when the abstraction is used. We depict the maximum relative error based on the 100 randomly generated instances for each size of a graph. In all of the cases, the empirical upper bound on the relative error (i.e., the worst-case possible quality loss due to abstraction) is below 1.0%. Note that in all of the instances the quality of the strategy found by our novel algorithm is within the bounds of the original approach (i.e., the empirical bounds are likely to be overestimated).

6 Conclusions

We focus on solving partially observable stochastic games (POSGs) and the representation of partial information in these games. In the existing algorithms, the dimension of the belief space is equal to the number of possible states. This fact limits the scalability since both required memory and computation time grow rapidly. We introduce a novel abstraction method and an algorithm relying on a compact representation of the uncertainty in these POSGs. Our methodology is domain-independent and we demonstrate it on a motivating example from cybersecurity where the defender protects a computer network against an attacker who uses lateral movement to reach the desired target. Experimental results show that our novel algorithm scales several orders of magnitude better compared to the existing state of the art with only a negligible loss in quality (less than 1%). Our paper demonstrates practical aspects of algorithms for solving subclasses of POSGs and opens the possibility of using similar compact representation for other domains.

Acknowledgments

This research was supported by the Czech Science Foundation (grant no. 19-24384Y) and by the Army Research Laboratory and was accomplished under Cooperative Agreement Number W911NF-13-2-0045 (ARL Cyber Security CRA). The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.